Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(metric): add metric system (#1357)
baojun-zhangb441622ee621
diff --git a/docs/design/metric-design.md b/docs/design/metric-design.mdnew file mode 100644index 000000000..8cacfeaec--- /dev/null+++ b/docs/design/metric-design.md@@ -0,0 +1,826 @@+# OpenViking 指标体系设计方案++## 背景++本方案讨论的是 OpenViking 的“指标体系(metrics)”,目标是把 `/metrics` 做成一个可持续抓取的 Prometheus 导出端点,并与 `/api/v1/observer/*`(瞬时状态)和 `/api/v1/stats/*`(分析统计)形成清晰边界。++### 现状入口与实现特征++OpenViking 当前已经存在三类与“观测”相关的入口:++| 入口 | 当前定位 | 当前实现特征 |+| --- | --- | --- |+| `/api/v1/observer/*` | 组件瞬时状态查询 | `ObserverService` 组装 queue/vikingdb/models/lock/retrieval 状态 |+| `/api/v1/stats/*` | 业务统计与内容质量分析 | `StatsAggregator` 动态查询 memory 分类、热度、陈旧度、session extraction 等 |+| `/metrics` | Prometheus 指标导出 | 当前依赖 `PrometheusObserver.render_metrics()` 输出文本 |++结合代码现状可以归纳出关键事实:++| 模块 | 位置 | 当前职责 | 问题摘要 |+| --- | --- | --- | --- |+| `BaseObserver` | `openviking/storage/observers/base_observer.py` | `get_status_table/is_healthy/has_errors` | 瞬时状态语义清晰 |+| `PrometheusObserver` | `openviking/storage/observers/prometheus_observer.py` | 进程内存储 Counter/Histogram + 渲染 `/metrics` | 与 Observer 体系职责不一致,属于“异类” |+| 业务埋点(retrieval/embedding/vlm 等) | 多处 | 直接回调 observer 写指标 | 采集层与导出层强耦合,难扩展/难测试 |+| `/metrics` 路由 | `openviking/server/routers/metrics.py` | 直接取 `app.state.prometheus_observer` | 路由层绑定具体实现对象,而非 exporter 抽象 |++当前 `/metrics` 已打通的指标主要集中在 retrieval/embedding/vlm/cache 的少量 counter/histogram 指标族,覆盖面偏窄。++### 关键问题++- `PrometheusObserver` 破坏 Observer 一致性:它本质是“注册中心 + 导出器”混合体,而非瞬时状态读取器。+- 采集与导出强耦合:业务代码需要感知 Prometheus 是否启用,未来接入其他 exporter 会反向侵入业务埋点。+- 指标类型与语义不完整:缺 gauge/health、缺统一命名与标签边界,难扩展到 queue/task/lock/vikingdb 等运行态。+- 多租户可观测性不足但风险高:缺 `account_id` 切片能力,同时必须防止演化为高基数(user/session/resource 等维度严禁进入 `/metrics`)。+- `/metrics` 与 `/api/v1/stats` 易混淆:分析型统计不应迁入高频抓取的 Prometheus 模型。++### 设计目标与非目标++设计目标:++1. 让 `Observer` 回归“瞬时状态观测”本位。+2. 建立独立 metrics 核心层:采集(DataSource)/语义分流(Collector)/存储(Registry)/导出(Exporter)解耦。+3. 把零散埋点统一接入同一套事件与状态契约(Event/State/DomainStats/Probe)。+4. 为 `/metrics` 提供系统化的 Gauge / Counter / Histogram,并明确失败语义(best-effort、valid/stale)。+5. 明确 `/metrics` 与 `/api/v1/stats` 边界,避免抓取放大成本。++非目标:++- 不重写现有 operation telemetry 响应结构。+- 不要求把所有业务统计都变为 Prometheus 指标。+- 不引入分布式聚合层,也不解决跨进程汇总问题。+- 首版不强制落地 OTel exporter。++## 1. 架构概览++### 1.1 核心原则++本节明确指标体系设计中需要持续成立的几项基础原则。这些原则用于约束后续的抽象分层、模块边界、Telemetry 关系以及对外观测接口的职责划分。++#### 原则 A:先抽象职责,再落实现类型++概览层保留以下四个核心抽象:++- `MetricDataSource`+- `BaseMetricCollector`+- `MetricRegistry`+- `BaseMetricExporter`++#### 原则 B:数据源与指标体系解耦++现有 Observer、Telemetry、TaskTracker、业务事件埋点,本质上都属于“数据来源”,但不等同于指标系统本身。++因此:++- 数据源负责提供原始状态或原始事件;+- Collector 负责把这些原始语义转换为指标;+- Registry 负责保存指标;+- Exporter 负责把指标转换为外部协议。++这种拆分可以避免 retrieval、embedding、vlm 等业务代码直接耦合某一个 exporter。++#### 原则 C:Registry 是唯一真实指标存储++`MetricRegistry` 作为进程内指标的唯一真实来源,负责承接统一的读写与约束。++职责范围如下:++- 指标定义注册;+- 标签规范校验;+- 当前值读写;+- 并发安全;+- 向 exporter 提供可读取的当前视图。++注意:这里的“当前视图”仅指读取时获取 registry 内部状态,不引入独立的 Snapshot 架构层。++#### 原则 D:Exporter 只负责协议导出++Exporter 只负责协议导出,不负责指标语义生成。++Exporter 不负责:++- 业务埋点;+- 状态采集;+- 指标语义转换。++Exporter 只负责:++- 读取 registry;+- 根据目标协议格式化;+- 输出给 `/metrics` 或其他监控后端。++这意味着 Prometheus 只是首个落地实现,而不是整个 metrics 架构的中心。++#### 原则 E:保留三类观测入口的职责边界++`/metrics`、`/api/v1/observer/*`、`/api/v1/stats/*` 三类入口继续并存,但必须保持清晰分工。++- `/metrics` 面向机器抓取,强调低基数、低成本、可持续聚合;+- `/api/v1/observer/*` 面向人工诊断,强调瞬时状态可读性;+- `/api/v1/stats/*` 面向业务分析,允许更重的查询与统计逻辑。++这三个入口共享部分数据来源,但不共享同一种输出模型。++### 1.2 抽象分层设计++本节给出指标体系的抽象主链路,只保留最小且稳定的四类角色,不提前展开任何具体实现类。++抽象主链路如下:++```mermaid+graph LR+ A["MetricDataSource"]+ B["BaseMetricCollector"]+ C["MetricRegistry"]+ D["BaseMetricExporter"]++ A --> B+ B --> C+ C --> D+```++四类抽象角色的职责如下:++| 抽象角色 | 作用 | 只负责什么 | 不负责什么 |+| --- | --- | --- | --- |+| `MetricDataSource` | 提供原始状态或原始事件 | 产生可观测输入 | 不直接生成 Prometheus 文本 |+| `BaseMetricCollector` | 将输入转为统一指标语义 | 采集、归一化、写 registry | 不直接暴露 HTTP |+| `MetricRegistry` | 统一存储进程内指标 | 注册、校验、读写、并发控制 | 不关心数据源来自哪里 |+| `BaseMetricExporter` | 按协议导出指标 | 格式化与导出 | 不直接理解业务链路语义 |++抽象主链路对应的数据流顺序如下:++1. `MetricDataSource` 提供瞬时状态、请求结束摘要或运行时事件;+2. `BaseMetricCollector` 将这些输入映射为 Counter / Gauge / Histogram 等统一指标;+3. `MetricRegistry` 保存当前进程内全部指标状态;+4. `BaseMetricExporter` 在需要时读取 registry 并输出给外部系统。++### 1.3 设计边界++抽象分层确定之后,还需要从设计边界上进一步保证:++* 指标体系可以观测业务主链路,但不成为业务主链路的组成部分;+* 指标链路可以缺失或降级,但不能反向改变 retrieval、embedding、VLM、resource、session、task 与 observer 等核心功能的行为语义。++因此,需要对相应角色施加明确约束:++* **DataSource**:业务接触面收敛到 DataSource。业务对象、业务状态、业务事件、已有统计快照与探针执行结果,只允许由 `MetricDataSource` 接触和读取。`Collector`、`MetricRegistry`、`Exporter` 不再直接访问业务服务、业务存储或业务上下文。+* **Collector**:Collector 禁止重新参与业务流程。Collector 的职责仅为把标准化输入映射为指标语义,并写入 `MetricRegistry`,不拥有业务真实状态。+ * 事件类(如 HTTP API 调用)链路由业务事件发生点触发,DataSource 到 Collector 之间只传递轻量事件;+ * 状态类(如系统状态)链路由采集时机触发,例如 `/metrics` 抓取前刷新,对应读取只能是轻量快照、已有聚合结果或轻量探针结果。++### 1.4 目录与模块边界++在边界约束成立之后,模块划分也需要与之保持一致,避免目录结构重新把已经划清的职责边界混回同一层。为避免新指标体系继续与 `storage/observers/` 的职责混杂,集中放入 `openviking/metrics/`,并按“注册中心 / 采集器 / 导出器 / 规则 / 启动装配”进行分组。++逻辑分组如下:++| 模块分组 | 职责 |+| --- | --- |+| `registry` | 指标存储、指标定义、标签校验 |+| `collectors` | retrieval / embedding / vlm / queue / task / observer / telemetry 等采集器 |+| `exporters` | Prometheus / 未来 OTel / InfluxDB |+| `naming` | 指标命名规则、label 规则、bucket 约定 |+| `bootstrap` | app 启动时初始化 registry / exporters / collector manager |++### 1.5 与现有 telemetry 的关系++operation telemetry 与 metrics 并不是两套彼此替代的系统。前者继续作为请求级结构化摘要存在,服务于单次调用的解释与排障;后者则只对白名单字段做低基数抽取,用于持续抓取、聚合与告警。++operation telemetry 已经拥有很多有价值的数据字段,如:++- `duration_ms`+- `tokens.*`+- `vector.*`+- `queue.*`+- `semantic_nodes.*`+- `memory.extract.*`+- `errors.*`++但并非所有字段都适合直接进入 `/metrics`,因此这里只对白名单字段进行指标化抽取:++| telemetry 分组 | 是否指标化 | 原因 |+| --- | --- | --- |+| `duration_ms` | 是 | 低基数、高通用性 |+| `tokens.total / llm / embedding` | 是 | 有明确容量与成本价值 |+| `vector.searches / scored / returned` | 是 | 检索链路核心运行指标 |+| `queue.*` | 是 | 适合形成任务吞吐与错误指标 |+| `semantic_nodes.*` | 有条件 | 更适合资源导入链路,不应无限扩散标签 |+| `memory.extract.*` | 部分保留在 stats / telemetry | 更偏业务分析,首版不全面指标化 |+| `errors.message` | 否 | 高基数、可能泄漏上下文 |++### 1.6 `/metrics`、`/api/v1/observer`、`/api/v1/stats` 的职责边界++三类对外观测接口共享部分数据来源,但并不共享同一种输出模型,也不应追求由同一套接口承担全部观测需求。明确这一边界,是为了避免后续设计在机器抓取、人工诊断与业务分析之间发生职责漂移。++| 接口 | 定位 | 数据特征 | 输出风格 |+| --- | --- | --- | --- |+| `/metrics` | 机器消费型监控接口 | 低基数、可持续抓取、可聚合 | Prometheus exposition |+| `/api/v1/observer/*` | 人工诊断型瞬时状态接口 | 组件当前状态、表格或结构化描述 | JSON + 可读状态文本 |+| `/api/v1/stats/*` | 业务分析型接口 | 可能昂贵、可扫描、可聚合但非高频 | JSON 统计结果 |++## 2. 核心设计细节++### 2.1 `MetricDataSource` 设计++在实现层,抽象角色 `MetricDataSource` 统一落为基类 `BaseMetricDataSource`,并在其下进一步划分 `EventMetricDataSource`、`StateMetricDataSource`、`DomainStatsMetricDataSource`、`ProbeMetricDataSource` 四类中间抽象。这四类中间抽象并非单纯的逻辑标签,而是分别对应不同的数据访问契约与刷新方式,因此在架构层被明确区分。++这些输入可能是:++- 某个组件的瞬时状态;+- 某个运行时事件;+- 某个领域内部维护的累计统计;+- 某个探针执行结果。++在 OpenViking 中,`MetricDataSource` 采用“统一基类 + 中间契约层 + 具体实现类”的三层结构。具体继承关系如下:++```mermaid+graph LR+ A["BaseMetricDataSource"]++ B["EventMetricDataSource"]+ C["StateMetricDataSource"]+ D["DomainStatsMetricDataSource"]+ E["ProbeMetricDataSource"]++ F["HttpRequestLifecycleDataSource"]+ G["ResourceIngestionEventDataSource"]+ H["SessionLifecycleDataSource"]+ O["EncryptionEventDataSource"]++ I["QueuePipelineStateDataSource"]+ J["TaskStateDataSource"]++ K["RetrievalStatsDataSource"]+ L["ModelUsageDataSource"]+ M["ObserverStateDataSource"]++ N["ServiceProbeDataSource"]+ P["StorageProbeDataSource"]+ Q["RetrievalBackendProbeDataSource"]+ R["ModelProviderProbeDataSource"]+ S["AsyncSystemProbeDataSource"]+ T["EncryptionProbeDataSource"]++ A --> B+ A --> C+ A --> D+ A --> E++ B --> F+ B --> G+ B --> H+ B --> O++ C --> I+ C --> J++ D --> K+ D --> L+ D --> M++ E --> N+ E --> P+ E --> Q+ E --> R+ E --> S+ E --> T+```++这四类中间抽象对应的数据访问契约如下:++| 中间抽象 | 契约语义 | 典型读取方式 | 典型指标类型 |+| --- | --- | --- | --- |+| `EventMetricDataSource` | 提供增量事件流或生命周期事件 | 读取事件批次、消费新事件、按游标增量拉取 | Counter、Histogram |+| `StateMetricDataSource` | 提供当前状态快照 | 按需读取当前状态、重复读取返回最新值 | Gauge |+| `DomainStatsMetricDataSource` | 提供领域内部已经维护好的累计统计 | 读取累计计数、汇总结果、统计快照 | Counter、Gauge、部分 Histogram 输入 |+| `ProbeMetricDataSource` | 提供探针执行结果或健康检查结果 | 执行探测、读取最近一次探测结果 | Gauge、Health |++各类具体数据源的职责如下:++| 实现 | 分类 | 主要对接对象 | 输出形态 | 适用场景 |+| --- | --- | --- | --- | --- |+| `HttpRequestLifecycleDataSource` | `EventMetricDataSource` | FastAPI middleware、路由响应 | 请求生命周期事件 | request total、duration、status code、in-flight |+| `ResourceIngestionEventDataSource` | `EventMetricDataSource` | ResourceProcessor、ResourceService、watch / wait 流程 | 资源处理事件与阶段摘要 | parse / finalize / summarize / wait / watch duration |+| `SessionLifecycleDataSource` | `EventMetricDataSource` | session create / used / commit / archive | 会话生命周期状态与事件 | commit 生命周期、contexts_used、archive 状态 |+| `EncryptionEventDataSource` | `EventMetricDataSource` | Encryptor、API Key 验证路径、KDF / Key Loader | 加密操作事件与密钥处理事件 | encrypt / decrypt / verify count、duration、bytes、kdf / key_load 耗时、auth_failed |+| `QueuePipelineStateDataSource` | `StateMetricDataSource` | `QueueManager`、Semantic DAG、request queue stats | 队列与流水线状态 | pending、in_progress、processed、error_count、semantic_nodes |+| `TaskStateDataSource` | `StateMetricDataSource` | `TaskTracker` | 当前任务状态 | pending、running、completed、failed 数量 |+| `RetrievalStatsDataSource` | `DomainStatsMetricDataSource` | `RetrievalStatsCollector` | 检索累计统计 | query count、zero result、latency、rerank 情况 |+| `ModelUsageDataSource` | `DomainStatsMetricDataSource` | VLM / Embedding / Rerank token tracker | 模型使用统计 | 调用次数、耗时、token 消耗 |+| `ObserverStateDataSource` | `DomainStatsMetricDataSource` | `ObserverService`、`LockManager`、`VikingDBManager` | 诊断聚合视图 | component health、lock、vikingdb、models、retrieval |+| `ServiceProbeDataSource` | `ProbeMetricDataSource` | 服务初始化、组件装配状态 | 服务探针结果 | startup completion、service readiness |+| `StorageProbeDataSource` | `ProbeMetricDataSource` | AGFS / VikingFS、系统表访问检查 | 存储探针结果 | storage readability、storage writability、system table readiness |+| `RetrievalBackendProbeDataSource` | `ProbeMetricDataSource` | VikingDB、检索后端最小能力检查 | 检索后端探针结果 | backend readiness、collection availability |+| `ModelProviderProbeDataSource` | `ProbeMetricDataSource` | VLM / Embedding / Rerank provider 可用性检查 | 模型依赖探针结果 | provider readiness、credential availability |+| `AsyncSystemProbeDataSource` | `ProbeMetricDataSource` | Queue、TaskTracker、后台消费线程检查 | 异步系统探针结果 | queue readiness、worker liveness |+| `EncryptionProbeDataSource` | `ProbeMetricDataSource` | Root Key、KMS / Vault Provider、加密组件检查 | 加密探针结果 | root key readiness、kms availability、encryption component health |++设计边界如下:++- 同一个业务模块可以暴露多个 DataSource,例如 session 相关能力既可以贡献 `SessionLifecycleDataSource`,也可能间接贡献 `TaskStateDataSource`。+- 加密相关监控优先建模为 `EncryptionEventDataSource`;涉及 Root Key readiness、KMS / Vault Provider 可用性、加密组件健康度时,则由 `EncryptionProbeDataSource` 承接。+- `SessionLifecycleDataSource` 只支持聚合级监控,不支持 `session_id` 级细粒度监控,也不允许把 `session_id` 作为指标标签。+- `ObserverStateDataSource` 属于诊断视图适配结果,适合承接当前 Observer 体系的聚合输出,但不应继续向更高层堆叠新的 summary。+- `operation telemetry` 属于请求级链路汇总能力,不作为一级 `MetricDataSource` 建模;如需复用其结果,应通过 collector 或兼容适配层接入。+- memory health、category、staleness 等分析型统计继续通过 `/api/v1/stats` 暴露,不在本节中抽象为独立 DataSource。+- `ProbeMetricDataSource` 采用方案 B:不再设置总的 `SystemProbeDataSource` 聚合父类,而是直接按依赖类型细分为 Service / Storage / Retrieval Backend / Model Provider / Async System / Encryption 六类 probe 子类。++Observer、Telemetry、TaskTracker、HTTP Router 与各类业务服务继续保持原有职责;指标系统只把它们视为数据源,而不将其改造成 exporter 或 collector。++### 2.2 `BaseMetricCollector` 设计++Collector 位于指标体系中的语义转换层,负责接收不同类型的可观测输入,并将其稳定映射为对 `MetricRegistry` 的统一写入操作。随着 `MetricDataSource` 被进一步划分为 Event、State、DomainStats 与 Probe 四类,Collector 侧采用对应的分层组织,而不再停留在仅区分 Event / State 的简化模型。++在这一设计下,Collector 不再只是“埋点写入器”,而是承担统一收口职责:一方面屏蔽上游数据源在访问方式与更新节奏上的差异,另一方面对下游 registry 暴露一致的写入语义。这样可以保证新增指标链路时,扩展点仍然集中在 Collector 层,而不会把 source-specific 逻辑扩散到 registry 或 exporter。++Collector 采用“基类 + 四类子抽象 + 多个具体实现”的组织方式:++```mermaid+graph LR+ A["BaseMetricCollector"]+ B["EventMetricCollector"]+ C["StateMetricCollector"]+ D["DomainStatsMetricCollector"]+ E["ProbeMetricCollector"]+ F["RetrievalCollector(event)"]+ G["EmbeddingCollector"]+ H["VLMCollector"]+ I["CacheCollector"]+ J["EncryptionCollector"]+ K["QueueCollector"]+ L["LockCollector"]+ M["VikingDBCollector"]+ N["ObserverHealthCollector"]+ O["TaskTrackerCollector"]+ P["RetrievalCollector(stats)"]+ Q["ModelUsageCollector"]+ R["ObserverStateCollector"]+ S["ServiceProbeCollector"]+ T["StorageProbeCollector"]+ U["RetrievalBackendProbeCollector"]+ V["ModelProviderProbeCollector"]+ W["AsyncSystemProbeCollector"]+ X["EncryptionProbeCollector"]++ A --> B+ A --> C+ A --> D+ A --> E+ B --> F+ B --> G+ B --> H+ B --> I+ B --> J+ C --> K+ C --> L+ C --> M+ C --> N+ C --> O+ D --> P+ D --> Q+ D --> R+ E --> S+ E --> T+ E --> U+ E --> V+ E --> W+ E --> X+```++抽象层次的分工如下:++| 抽象层次 | 职责 | 典型触发方式 |+| --- | --- | --- |+| `BaseMetricCollector` | 定义 collector 统一行为与 registry 写入约束 | 所有 collector 共用 |+| `EventMetricCollector` | 处理“事件发生一次,就写一次”的场景 | HTTP 请求完成、资源处理完成、VLM 调用完成、加密操作完成 |+| `StateMetricCollector` | 处理“读取当前状态并刷新 gauge”的场景 | `/metrics` 抓取前刷新 |+| `DomainStatsMetricCollector` | 处理“读取已有累计统计并写入 registry”的场景 | 检索统计、模型使用统计、Observer 聚合视图刷新 |+| `ProbeMetricCollector` | 处理“执行探针并映射为健康类指标”的场景 | readiness / dependency probe 刷新 |++Collector 与 DataSource 的主映射关系如下:++| DataSource 分类 | 优先对应的 Collector 分类 | 说明 |+| --- | --- | --- |+| `EventMetricDataSource` | `EventMetricCollector` | 事件型输入天然适合 Counter / Histogram |+| `StateMetricDataSource` | `StateMetricCollector` | 当前状态快照天然适合 Gauge |+| `DomainStatsMetricDataSource` | `DomainStatsMetricCollector` | 已聚合统计应直接桥接到 registry |+| `ProbeMetricDataSource` | `ProbeMetricCollector` | 健康检查结果应统一映射为 readiness / health 指标 |++具体 collector 分工如下:++| Collector | 分类 | 数据输入 | 写入指标 |+| --- | --- | --- | --- |+| `RetrievalCollector` | `EventMetricCollector` | retrieval 完成事件 | 请求数、零结果数、结果数、耗时、rerank 使用情况 |+| `EmbeddingCollector` | `EventMetricCollector` | embedding 完成事件 | 请求数、耗时、错误数 |+| `VLMCollector` | `EventMetricCollector` | token / duration 事件 | 调用数、耗时、token 统计 |+| `CacheCollector` | `EventMetricCollector` | cache 命中或未命中事件 | hit / miss |+| `EncryptionCollector` | `EventMetricCollector` | encrypt / decrypt / verify / kdf / key_load 事件 | 加密次数、认证失败、耗时、字节量、密钥处理统计 |+| `QueueCollector` | `StateMetricCollector` | QueueManager / DAG 当前状态 | pending、in_progress、processed、errors |+| `LockCollector` | `StateMetricCollector` | LockManager 当前状态 | active、waiting、stale |+| `VikingDBCollector` | `StateMetricCollector` | VikingDB collection 当前状态 | health、vectors、collections |+| `ObserverHealthCollector` | `StateMetricCollector` | ObserverService 结果 | component health、component errors |+| `TaskTrackerCollector` | `StateMetricCollector` | TaskTracker 当前状态 | pending、running、completed、failed |+| `RetrievalCollector` | `DomainStatsMetricCollector` | RetrievalStatsDataSource | 检索累计统计桥接 |+| `ModelUsageCollector` | `DomainStatsMetricCollector` | ModelUsageDataSource | 模型使用累计统计桥接 |+| `ObserverStateCollector` | `DomainStatsMetricCollector` | ObserverStateDataSource | 诊断聚合视图桥接 |+| `ServiceProbeCollector` | `ProbeMetricCollector` | ServiceProbeDataSource | service readiness、startup completion |+| `StorageProbeCollector` | `ProbeMetricCollector` | StorageProbeDataSource | storage readiness、system table availability |+| `RetrievalBackendProbeCollector` | `ProbeMetricCollector` | RetrievalBackendProbeDataSource | backend readiness、collection availability |+| `ModelProviderProbeCollector` | `ProbeMetricCollector` | ModelProviderProbeDataSource | provider readiness、credential availability |+| `AsyncSystemProbeCollector` | `ProbeMetricCollector` | AsyncSystemProbeDataSource | queue readiness、worker liveness |+| `EncryptionProbeCollector` | `ProbeMetricCollector` | EncryptionProbeDataSource | root key readiness、kms availability、encryption component health |++设计理由:在引入四类 Collector 之后,整个体系的语义边界更清晰:++- EventCollector 偏向增量写入,天然适合 Counter / Histogram;+- StateCollector 偏向覆盖写入,天然适合 Gauge。+- DomainStatsCollector 偏向桥接已有累计统计,避免把已聚合结果重新拆回事件。+- ProbeCollector 偏向健康检查与 readiness 映射,避免把探针逻辑塞进 StateCollector 或 EventCollector。++对于 retrieval 链路,统一保留 `RetrievalCollector` 命名:它既可以接收 retrieval 完成事件,也可以读取 `RetrievalStatsDataSource` 的累计快照。两类输入在实现上可以走不同分支,但不再拆出单独的 `RetrievalStatsCollectorAdapter` 名称,以避免把“同一指标域的两条输入路径”误写成两套 collector。++### 2.3 `MetricRegistry` 设计++`MetricRegistry` 是整个体系的稳定中心,负责统一注册、校验、存储和读取指标,但不负责采集触发,也不承担协议导出职责。它对外维持统一接口,不针对 Event / State / DomainStats / Probe 四类 Collector 再拆分多套写入 API,语义分流应由 Collector 自身完成。++`MetricRegistry` 必须满足以下能力:++| 能力 | 要求 |+| --- | --- |+| 统一注册 | 每个指标在进程内只注册一次,防止重复定义 |+| 类型约束 | 同名指标不能一会儿是 Counter、一会儿是 Gauge |+| 标签约束 | 每个指标的 label key 集合固定,运行时只允许填充固定维度 |+| 并发安全 | 允许 FastAPI 协程、队列线程、后台清理线程同时读写 |+| 低开销读取 | `/metrics` 抓取不应持有长时间大锁 |+| 可扩展性 | 不耦合 Prometheus 专属概念,避免 registry 层直接出现 exposition 文本逻辑 |+| 统一写入接口 | 对外维持统一的 counter / gauge / histogram 写入接口,不按 source 类型暴露分裂 API |++Registry 只解决“如何统一保存指标”,不解决:++- 数据从哪里来;+- 谁来触发采集;+- 以什么协议导出。++统一接口策略如下:++- EventCollector 决定何时调用 `inc_counter` 或 `observe_histogram`;+- StateCollector 决定何时调用 `set_gauge`;+- DomainStatsCollector 决定如何把已有累计统计映射到统一写接口;+- ProbeCollector 决定如何把 probe 结果翻译成 readiness / health gauge。++也就是说,Registry 不感知“当前写入的是事件、状态、统计还是探针”,它只感知“要写入哪种指标类型、哪个指标名、哪些标签和值”。++由此,registry 可以作为整个指标系统的稳定核心,而不随 exporter 或 collector 的演化频繁变动。++### 2.4 `BaseMetricExporter` 设计++Exporter 位于指标体系最下游,只负责把 registry 中的当前指标状态转换成外部协议。与 Registry 相同,Exporter 也遵循统一接口原则:它只依赖统一的读接口,不感知指标来自 Event / State / DomainStats / Probe 中的哪一路。++Exporter 采用如下继承关系:++```mermaid+graph TB+ A["BaseMetricExporter"]+ B["PrometheusExporter"]+ C["OtelExporter"]+ D["InfluxDBExporter"]++ A --> B+ A --> C+ A --> D+```++各 exporter 的定位如下:++| Exporter | 定位 | 首版状态 |+| --- | --- | --- |+| `PrometheusExporter` | 负责 `/metrics` exposition 输出 | 首版必须落地 |+| `OtelExporter` | 对接未来 OTel 指标导出 | 预留 |+| `InfluxDBExporter` | 对接未来 InfluxDB | 预留 |++统一读取策略如下:++- Exporter 通过统一的 registry 读取接口获取当前样本集合;+- Exporter 不关心某个样本来自哪类 DataSource 或哪类 Collector;+- Exporter 只关心指标定义、标签集合、当前值与目标协议的序列化方式;+- source/category 语义在 Collector 写入 registry 前已经完成收敛,不应渗透到 exporter 层。++### 2.5 Prometheus 方式数据路径++本节讨论 `/metrics` 请求从进入服务到返回结果之间的完整运行时序,并明确导出前刷新策略的边界。Prometheus 抓取 `/metrics` 时,先执行必要的 collector 刷新,再统一读取 registry 快照,最后由 exporter 完成协议序列化。++Prometheus 抓取 `/metrics` 时,执行以下流程:++```mermaid+sequenceDiagram+ participant P as Prometheus+ participant R as /metrics Router+ participant E as PrometheusExporter+ participant C as StateCollectors+ participant G as MetricRegistry++ P->>R: GET /metrics+ R->>E: export()+ E->>C: refresh()+ C->>G: set Gauge values+ E->>G: read unified metric snapshot+ G-->>E: metric samples+ E-->>R: text exposition+ R-->>P: 200 text/plain+```++设计理由:采用“导出前刷新 StateCollector”的原因是:++- 不需要额外后台采样协程;+- 指标与抓取时间点一致;+- 避免无人抓取时持续消耗资源;+- 与现有 Observer 的按需读取模式一致。+- Exporter 保持协议层纯度,不因 source/category 增长而扩展额外分支逻辑。++### 2.6 State / Probe 刷新控制策略++对于 Queue、Lock、VikingDB、Task、Observer health 以及各类 probe 而言,设计目标不应是“每次 `/metrics` 抓取都严格同步拿到瞬时最新值”,而应是“在保证抓取路径稳定、延迟可控的前提下,提供足够新的状态视图”。换句话说,`/metrics` 首先是一个可持续抓取的监控接口,其次才是状态刷新触发点。++基于这一点,State / Probe 刷新统一采用“默认 scrape-triggered refresh + 基于 TTL 的刷新控制 + stale-while-revalidate” 组合策略:++- 默认情况下,State / Probe Collector 仍在 Prometheus 抓取 `/metrics` 时触发刷新;+- 对读取成本高、依赖外部系统或容易抖动的 Collector,允许引入短 TTL 的刷新控制窗口;+- 当当前时间仍处于 TTL 控制窗口内时,不重复触发刷新;+- 当 TTL 控制窗口结束后,允许先返回最近一次成功值,再触发一次受控刷新;+- 刷新策略属于 Collector 运行时能力,而不是 `MetricRegistry` 的职责。++这里需要进一步明确两类信息的边界:++- `MetricRegistry` 保存的是对外可见的当前指标值,是 Exporter 读取并序列化的唯一真实来源;+- State / Probe Collector 不保存另一份对外指标结果,而只负责在合适的时机触发刷新。++因此,TTL、超时、是否允许返回旧值、是否触发后台刷新等策略不下沉到 `MetricRegistry`。Registry 只负责统一存储指标,不负责刷新调度,也不在其读取接口上附带“读取即触发刷新”的副作用。刷新时机由 `/metrics` 导出路径中的执行层显式控制,例如由 Exporter 或独立的 CollectorManager 在导出前调用 `refresh_if_needed()`,随后再统一读取 Registry。++对于返回旧值(stale value),本方案默认允许该行为,而不是把它当作异常分支。原因在于大多数状态类与探针类指标本身就不是强瞬时一致性的观测对象;相比“为了获取最新值而阻塞整个 `/metrics` 抓取路径”,继续导出 `MetricRegistry` 中最近一次成功写入的指标值更符合监控系统的使用语义。++为了避免“旧值长期存在但不可见”,本方案采用更明确的可观测语义:对于无法定义失败默认值的状态类指标,指标设置之初即包含一个最小的有效性标签(如 `valid=1/0`),以便让告警与看板显式识别“当前值是否可用”。当刷新失败时,不强行覆盖写入默认值,而是将该 `valid` 标签置为 `0`,并继续导出 `MetricRegistry` 中最近一次成功写入值。++- 对启用基于 TTL 的刷新控制的 State / Probe 指标,指标定义中统一包含 `valid` 标签;Collector 在刷新成功时写入 `valid=1`,刷新失败时写入 `valid=0`;+- 对具有明确失败语义的探针类指标(如 readiness / dependency probe),刷新失败时应显式覆盖写入失败值(例如 UP -> DOWN),而不依赖“值消失”或隐式过期;++在 Collector 分类上,以下对象优先视为“启用基于 TTL 的刷新控制”的候选:++- 需要访问外部依赖的 Probe Collector,例如 retrieval backend、model provider、storage、encryption provider;+- 需要做聚合、扫描或多对象汇总的 State Collector,例如 collection 级状态或复杂 observer 聚合视图;+- 在运行中已观测到刷新耗时抖动明显、超时概率较高的 Collector。++相对地,Queue、Task 等纯内存态或低成本状态读取,仍应优先保持同步 scrape-triggered refresh,不必默认引入基于 TTL 的刷新控制。++### 2.7 HTTP Metrics Middleware 语义++当前实现中,HTTP 请求指标并不只是“某个 EventMetricCollector 的普通上游事件源”,而是由一个专门的 middleware 负责把请求生命周期翻译为低基数、best-effort 的 HTTP metrics 事件。其运行时语义如下:++#### 2.7.1 路由忽略与低基数路径归一化++- middleware 必须先按 **raw path** 忽略内部接口:+ - `/metrics`+ - `/health`+ - `/ready`+- 这样可以保证即使当前请求尚未绑定 Starlette route template,也不会把内部健康接口错误纳入业务 HTTP 指标。+- 对普通业务请求,优先使用 route template(例如 `/api/v1/sessions/{session_id}`)作为 `route` 标签;+- 当 route template 不可用时,禁止回退到原始 URL path,而统一回落到固定低基数字符串(例如 `"/__unmatched__"`),以避免把动态 ID、UUID、资源路径直接带入标签,造成基数爆炸。++#### 2.7.2 inflight 语义是“进程内近似值”,不是强一致计数++- HTTP inflight 计数可以通过进程内字典按 `(route, account_id)` 维持;+- 该值的目标是为实时看板和排障提供“足够合理的当前并发近似视图”,而不是全局严格正确性机制;+- 为避免多线程运行时的读改写竞态,应使用轻量锁保护该近似计数的更新过程;+- 当 inflight 计数归零后,应及时删除对应键,避免零值键长期残留并放大内存占用。++#### 2.7.3 middleware 必须保持 best-effort++- HTTP metrics middleware 不能影响主请求链路;+- 任何指标写入、inflight 调整、request 计时上报失败,都只能影响 observability side-channel,而不能影响业务响应;+- 统一将此类异常收敛为 debug 级日志,并在日志中带上 `route` 与 `account_id` 等上下文字段;+- 默认不开启高频错误日志,仅在 debug 诊断场景下用于排查 metrics 自身问题。++#### 2.7.4 请求链路中的 `account_id` 解析分两阶段完成++- middleware 进入请求时,可以先尝试从 request state 或 header 提取一个 provisional account id,用于 inflight 早期打点;+- 请求结束后,应重新读取最终的 request-scoped account identity,并在必要时对 inflight 指标做一次 rebalance;+- 这样可兼顾:+ - 早期 inflight 可见性;+ - 认证完成后的最终租户归属正确性。++## 3. 指标策略++### 3.1 指标映射总览++在展开具体指标清单之前,先把“指标族 → DataSource → Collector → Metric Type”的关系收敛成统一视图,可以避免后续列表只见指标名、不见来源链路。对于首版范围内的所有指标,都应能够映射回明确的 DataSource 与 Collector;如果某个指标无法说明其输入来源或采集责任,就不应直接进入首版清单。++| 指标族 | 主要 DataSource | 主要 Collector | 主要指标类型 | 代表指标 |+| --- | --- | --- | --- | --- |+| HTTP 请求监控 | `HttpRequestLifecycleDataSource` | `HTTPCollector` | Counter、Histogram | request total、duration、status code |+| 检索链路监控 | `RetrievalStatsDataSource`、retrieval 完成事件 | `RetrievalCollector` | Counter、Histogram | retrieval requests、zero result、latency |+| 模型链路监控 | `ModelUsageDataSource`、模型调用事件 | `VLMCollector`、`EmbeddingCollector`、`ModelUsageCollector` | Counter、Histogram | model calls、tokens、duration |+| 资源导入监控 | `ResourceIngestionEventDataSource` | `ResourceIngestionCollector` | Counter、Histogram | parse / finalize / summarize / wait duration |+| Session 与异步任务监控 | `SessionLifecycleDataSource`、`TaskStateDataSource`、`QueuePipelineStateDataSource` | `TaskTrackerCollector`、`QueueCollector` | Gauge、Counter | task pending、queue backlog、session lifecycle count |+| 诊断与状态监控 | `ObserverStateDataSource`、`QueuePipelineStateDataSource` | `ObserverHealthCollector`、`LockCollector`、`VikingDBCollector` | Gauge | component health、lock active、collection health |+| 加密监控 | `EncryptionEventDataSource`、`EncryptionProbeDataSource` | `EncryptionCollector`、`EncryptionProbeCollector` | Counter、Histogram、Gauge | encrypt count、decrypt duration、root key readiness |+| 系统探针监控 | 各类 `*ProbeDataSource` | 各类 `*ProbeCollector` | Gauge、Health | service readiness、storage readiness、kms availability |+| 操作级 Telemetry 指标化 | telemetry adapter / bridge | `TelemetryBridgeCollector` | Counter、Histogram、Gauge | operation requests、vector scanned、memory extracted |++### 3.2 指标对象模型++当前指标对象模型收敛为三类基础指标:Counter、Gauge 与 Histogram。`Summary` 不纳入当前范围,以避免在聚合语义、实现复杂度与使用收益之间引入不必要的失衡。++当前统一支持三种指标类型:++| 类型 | 用途 | 典型场景 |+| --- | --- | --- |+| Counter | 单调递增计数 | 请求量、错误数、cache 命中数、任务完成数 |+| Gauge | 当前值 | 队列 backlog、运行中任务数、锁持有数、健康状态 |+| Histogram | 分布统计 | 请求耗时、embedding 耗时、VLM 调用耗时、资源处理耗时 |++当前不支持 `Summary`,原因如下:++- 对 Prometheus 多实例聚合不友好;+- 与 Histogram 的职责边界重叠;+- 会增加 registry 与 exporter 实现复杂度;+- 当前 OpenViking 真实缺的是 Gauge,不是 Summary。++### 3.3 标签策略++标签策略的核心,不在于“尽可能表达更多业务信息”,而在于在可观测性价值与高基数风险之间取得稳定平衡。为此,标签设计必须遵守“低基数优先”原则,默认只允许有限、可枚举、可控的标签集合。++#### 允许的常用标签++| 标签 | 说明 | 默认策略 |+| --- | --- | --- |+| `operation` | 操作名,如 `search.find`、`resources.add_resource` | 默认启用 |+| `status` | `ok` / `error` | 默认启用 |+| `queue` | `Embedding` / `Semantic` / 其他队列名 | 默认启用 |+| `level` | cache level,如 `L0` / `L1` / `L2` | 默认启用 |+| `context_type` | retrieval context,如 `memory` / `resource` | 默认启用 |+| `component` | queue / models / lock / retrieval / vikingdb | 默认启用 |+| `task_type` | 如 `session_commit` | 默认启用 |+| `account_id` | 租户 ID | 条件启用 |+| `provider` | 模型供应商,如 openai / volcengine | 条件启用 |+| `model_name` | 模型名 | 谨慎启用,需归一化后有限枚举 |++#### 租户标签策略++`account_id` 仅用于:++- 请求量;+- 请求耗时;+- 任务堆积;+- 资源导入吞吐。++运行时保护规则如下:++1. 默认关闭;+2. 仅对明确列入白名单的指标启用;+3. 提供最大活跃租户数保护,超出后回退到无标签聚合。++#### `account_id` 运行时解析与回退语义++在当前实现中,`account_id` 已不再只是“是否在指标定义里声明一个标签”的静态设计问题,而是一个运行时解析过程。其落地语义如下:++1. **支持集(support set)比 allowlist 更窄**+ - 只有被明确标记为“支持多租户维度”的指标族,才允许在运行时注入 `account_id`;+ - 即使配置上开启了 `account_id`,不在支持集中的指标也不得携带该标签;+ - allowlist 只用于在“支持集内部”进一步决定哪些指标真正启用租户维度。++2. **最终标签值不是简单的“真实租户 ID 或空”**+ - 运行时解析结果允许出现三类稳定标签值:+ - 真实租户 ID;+ - `__unknown__`:当前请求/事件无法解析出可信租户;+ - `__overflow__`:已启用活跃租户数上限保护,当前租户超出上限,统一回落到溢出桶。++3. **解析路径按“显式值 > 请求上下文 > 所有权信息”收敛**+ - 对事件类指标,优先使用事件调用方显式传入的 `account_id`;+ - 对 HTTP / 请求链路类指标,优先使用 request-scoped context 中已认证/已解析出的租户身份;+ - 对资源、任务等存在 owner 语义的场景,可回退到 owner account 信息;+ - 无法得到可信结果时统一落到 `__unknown__`,而不是直接省略标签。++4. **运行时策略作为独立配置对象**+ - 配置项至少应包含:+ - `enabled`+ - `metric_allowlist`+ - `max_active_accounts`+ - Collector 在写指标时不自行实现租户裁剪逻辑,而统一调用 account-dimension runtime helper 解析最终标签值。++5. **文档层面应明确:`account_id` 是“有损但可控”的可观测维度**+ - 其目标是为低基数、在线监控场景提供有限的租户切片;+ - 它不是审计维度,也不保证所有租户都能长期以真实 ID 形式保留在 `/metrics` 中。++#### 应避免的高基数标签++- `user_id`+- `session_id`+- `resource_uri`+- 原始 `error_message`+- 原始 `query`+- 文件路径+- request id / telemetry id++### 3.4 命名规范++命名规范的目标,是让同一类指标在不同 collector、不同链路中保持一致的表达方式,减少命名漂移与语义歧义。为此,指标命名统一采用 `openviking_<domain>_<metric>_<unit>` 模板,并对 Counter / Histogram / Health 指标施加额外约束。++指标命名统一采用:++`openviking_<domain>_<metric>_<unit>`++示例:++- `openviking_retrieval_requests_total`+- `openviking_queue_pending`+- `openviking_task_running`+- `openviking_operation_duration_seconds`++设计要求:++- Counter 以 `_total` 结尾;+- Histogram / duration 统一使用 `_seconds`;+- Gauge 不强制带单位后缀,但应显式表达;+- health 指标统一使用 `0/1` 数值。++### 3.5 明确不纳入 `/metrics` 的指标++以下指标保留在 `/api/v1/stats` 或 telemetry JSON 中更合适:++| 数据项 | 原因 |+| --- | --- |+| memory category 分布 | 需要扫描或查询聚合,更接近业务分析 |+| hotness / staleness 分布 | 不适合高频抓取 |+| 单 session extraction 明细 | 高基数、偏诊断 |+| 原始错误文本 | 高基数且可能包含敏感信息 |+| 任意 resource / URI 级统计 | 高基数 |++### 3.6 指标桶策略++所有耗时类 Histogram 统一使用秒级桶,便于跨模块比较:++| 桶值(秒) | 适用场景 |+| --- | --- |+| `0.005` / `0.01` / `0.025` | 极短本地操作 |+| `0.05` / `0.1` / `0.25` | retrieval / cache / 轻量 API |+| `0.5` / `1.0` / `2.5` | embedding / 中等资源操作 |+| `5.0` / `10.0` / `30.0` | VLM / wait 模式 / 重处理任务 |++如果某条链路需要独立桶配置,应通过指标定义层配置,而不是在业务埋点中写死。++### 3.7 Telemetry 指标化细则++对于 operation telemetry,只抽取“结束时就已具备、并且不会导致高基数”的字段。结合 `docs/zh/guides/05-observability.md`、`docs/zh/guides/07-operation-telemetry.md` 与 `openviking/telemetry/operation.py` 的当前能力。++| telemetry 字段 | 映射指标 |+| --- | --- |+| `summary.operation` + `summary.status` | `openviking_operation_requests_total` |+| `summary.duration_ms` | `openviking_operation_duration_seconds` |+| `summary.tokens.total` | `openviking_operation_tokens_total{token_type="all"}` |+| `summary.tokens.llm.input` | `openviking_operation_tokens_total{token_type="llm_input"}` |+| `summary.tokens.llm.output` | `openviking_operation_tokens_total{token_type="llm_output"}` |+| `summary.tokens.embedding.total` | `openviking_operation_tokens_total{token_type="embedding"}` |+| `summary.vector.searches` | `openviking_vector_searches_total` |+| `summary.vector.scored` | `openviking_vector_scored_total` |+| `summary.vector.passed` | `openviking_vector_passed_total` |+| `summary.vector.returned` | `openviking_vector_returned_total` |+| `summary.vector.scanned` | `openviking_vector_scanned_total` |+| `summary.semantic_nodes.{total|done|pending|running}` | `openviking_semantic_nodes_total{status=...}` |+| `summary.memory.extracted` | `openviking_memory_extracted_total` |++### 3.8 多租户支持范围++当前代码中,只有进入 `ACCOUNT_DIMENSION_SUPPORTED_METRICS` 支持集的指标族允许注入 `account_id`。支持集如下:++| 指标域 | 当前支持 `account_id` 的指标 |+| --- | --- |+| HTTP | `openviking_http_requests_total`、`openviking_http_request_duration_seconds`、`openviking_http_inflight_requests` |+| Retrieval | `openviking_retrieval_requests_total`、`openviking_retrieval_results_total`、`openviking_retrieval_zero_result_total`、`openviking_retrieval_latency_seconds`、`openviking_retrieval_rerank_used_total`、`openviking_retrieval_rerank_fallback_total` |+| Resource | `openviking_resource_stage_total`、`openviking_resource_stage_duration_seconds`、`openviking_resource_wait_duration_seconds` |+| Session | `openviking_session_lifecycle_total`、`openviking_session_contexts_used_total`、`openviking_session_archive_total` |+| Operation Telemetry | `openviking_operation_requests_total`、`openviking_operation_duration_seconds`、`openviking_operation_tokens_total` |+| VLM | `openviking_vlm_calls_total`、`openviking_vlm_call_duration_seconds`、`openviking_vlm_tokens_input_total`、`openviking_vlm_tokens_output_total`、`openviking_vlm_tokens_total` |+| Embedding | `openviking_embedding_requests_total`、`openviking_embedding_latency_seconds`、`openviking_embedding_errors_total` |++除上述指标外,其他指标族即使在配置上开启了 `account_id`,当前实现也不会为其注入租户标签。++## 5. 兼容性策略++Openviking 指标体系采用如下兼容性策略:++- 保持现有 retrieval / embedding / vlm / cache 指标名尽量不变;+- `/metrics` 路由地址保持不变;+- `telemetry.prometheus.enabled` 配置保持不变;+- `/api/v1/observer/*` 与 `/api/v1/stats/*` 行为保持不变。++## 6. 风险与缓解++| 风险 | 描述 | 缓解策略 |+| --- | --- | --- |+| 指标高基数 | 租户、模型、错误码维度无限增长 | 严格标签白名单 + 上限保护 |+| 抓取放大成本 | StateCollector / ProbeCollector 在 `/metrics` 时做重查询 | 限制为轻量瞬时状态源与轻量 probe,不扫描业务大表 |+| 迁移期间重复计数 | 老 PrometheusObserver 与新 registry 并存 | 引入过渡期开关,避免双写 |+| 并发锁争用 | 高频写入与高频抓取互相影响 | registry 采用细粒度结构与最小持锁时间 |+| 指标语义漂移 | 同一指标在不同 collector 中含义不一致 | 统一命名文档与 contract test |+| Probe 副作用 | 外部依赖 probe 超时或失败拖慢 `/metrics` | 为 probe 设置超时、隔离失败并限制 probe 数量 |+| Adapter 语义漂移 | telemetry bridge 或 domain stats bridge 与原始事件语义不一致 | 明确 bridge 边界并增加 contract test |+| 双路径重复上报 | 一级事件链路与 bridge 链路同时写入同类指标 | 以指标所有权清单约束单一写入责任 |diff --git a/docs/en/api/01-overview.md b/docs/en/api/01-overview.mdindex 8085e4cfc..80982a5f7 100644--- a/docs/en/api/01-overview.md+++ b/docs/en/api/01-overview.md@@ -280,6 +280,7 @@ Compact JSON with status wrapper (when `--compact` is true, which is the default| Method | Path | Description ||--------|------|-------------|| GET | `/health` | Health check (no auth) |+| GET | `/metrics` | Prometheus metrics export || GET | `/api/v1/system/status` | System status || POST | `/api/v1/system/wait` | Wait for processing |@@ -372,4 +373,5 @@ Compact JSON with status wrapper (when `--compact` is true, which is the default- [Sessions](05-sessions.md) - Session management- [Skills](04-skills.md) - Skill management- [System](07-system.md) - System and monitoring API+- [Metrics](09-metrics.md) - Prometheus metrics export and scraping guide- [Admin](08-admin.md) - Multi-tenant management APIdiff --git a/docs/en/api/09-metrics.md b/docs/en/api/09-metrics.mdnew file mode 100644index 000000000..82e882020--- /dev/null+++ b/docs/en/api/09-metrics.md@@ -0,0 +1,87 @@+# Metrics++OpenViking exposes a `/metrics` endpoint for Prometheus, Grafana Agent, and other monitoring systems that scrape Prometheus exposition text.++Unlike `/api/v1/observer/*` and `/api/v1/stats/*`, `/metrics` is intended for:++- machine scraping rather than human-oriented inspection+- Prometheus exposition text rather than the standard JSON API wrapper+- runtime health and service quality signals rather than business analytics++## API Reference++### metrics()++Export Prometheus metrics for the current process.++This endpoint is typically scraped by Prometheus on a schedule, but it is also useful for local debugging and manual inspection.++**Authentication**++- In the current implementation, `/metrics` is not wired to `get_request_context` or other auth dependencies.+- In other words, based on the current server code, `/metrics` is effectively a public scrape endpoint.+- If access control is tightened later through a gateway, reverse proxy, or deployment policy, follow the actual deployment configuration.++**HTTP API**++```+GET /metrics+```++```bash+curl -X GET http://localhost:1933/metrics+```++If your deployment requires gateway- or proxy-level authentication, attach the required headers there, for example:++```bash+curl -X GET http://localhost:1933/metrics \+ -H "Authorization: Bearer your-key"+```++**Response Format**++On success, the endpoint returns `text/plain; version=0.0.4; charset=utf-8` with Prometheus exposition text, for example:++```text+# HELP openviking_http_requests_total Total number of HTTP requests+# TYPE openviking_http_requests_total counter+openviking_http_requests_total{method="GET",route="/api/v1/system/status",status="200"} 12++# HELP openviking_http_inflight_requests Number of inflight HTTP requests+# TYPE openviking_http_inflight_requests gauge+openviking_http_inflight_requests{route="/api/v1/system/status"} 0+```++When the metrics system is disabled, the endpoint returns:++- HTTP status code: `404`+- Response body:++```text+Prometheus metrics are disabled.+```++**Example: Prometheus Scrape Config**++```yaml+scrape_configs:+ - job_name: openviking+ metrics_path: /metrics+ static_configs:+ - targets: ["localhost:1933"]+```++If your deployment protects `/metrics` at the gateway layer, configure the scrape job with the required auth settings through your proxy, service discovery, or Prometheus auth options.++**Notes**++- `/metrics` is meant for frequent scraping, so the exported metrics should remain low-cardinality and low-cost.+- `/metrics` returns Prometheus text, not the standard OpenViking `{status, result, time}` JSON response format.+- Analytics-oriented statistics such as memory health, staleness, or session extraction should remain under `/api/v1/stats/*`, not `/metrics`.+- For human-readable component snapshots, prefer `/api/v1/observer/*`.++## Related Documentation++- [System and Monitoring](07-system.md) - Health checks, system status, and Observer APIs+- [API Overview](01-overview.md) - Shared conventions for all API endpointsdiff --git a/docs/en/concepts/01-architecture.md b/docs/en/concepts/01-architecture.mdindex f82439f1d..af91be90d 100644--- a/docs/en/concepts/01-architecture.md+++ b/docs/en/concepts/01-architecture.md@@ -173,3 +173,7 @@ curl http://localhost:1933/api/v1/search/find \- [Retrieval Mechanism](./07-retrieval.md) - Retrieval process details- [Context Extraction](./06-extraction.md) - Parsing and extraction process- [Session Management](./08-session.md) - Session and memory management+- [Transaction Model](./09-transaction.md) - Write and consistency model+- [Data Encryption](./10-encryption.md) - At-rest encryption and key architecture+- [Multi-Tenant](./11-multi-tenant.md) - Account / user / agent isolation model+- [Metrics](./12-metrics.md) - `/metrics` usage and key metric explanationsdiff --git a/docs/en/concepts/12-metrics.md b/docs/en/concepts/12-metrics.mdnew file mode 100644index 000000000..e01fd21fd--- /dev/null+++ b/docs/en/concepts/12-metrics.md@@ -0,0 +1,313 @@+# Metrics++OpenViking provides a machine-oriented metrics system for exposing runtime health, request quality, model usage, resource processing throughput, and probe health states.++Unlike the human-facing `/api/v1/observer/*` endpoints and the analytics-oriented `/api/v1/stats/*` endpoints, Metrics are designed for:++- high-frequency scraping by Prometheus, Grafana Agent, and similar systems+- low-cardinality, aggregatable metric models+- monitoring, alerting, capacity observation, and regression diagnosis++## Overview++### Why Metrics++Metrics are well suited to answer questions like:++- Has HTTP traffic increased abnormally over the last few minutes?+- Are resource ingestion, retrieval, or model calls getting slower?+- Is there queue backlog?+- Are key dependencies such as storage, model providers, VikingDB, encryption, and async systems currently healthy?+- Is a specific tenant showing abnormal traffic or error rates?++Compared with logs and observer snapshots, metrics are better for:++- continuous scraping+- time-series aggregation+- dashboard visualization+- alert rules++### How Metrics Differ from Observer and Stats++| Capability | Best For | Output Format | Typical Usage |+|------------|----------|---------------|---------------|+| `/metrics` | online monitoring, alerting, trend aggregation | Prometheus exposition text | Grafana dashboards, Prometheus scraping |+| `/api/v1/observer/*` | human inspection of component snapshots | JSON / status tables | debugging, health checks |+| `/api/v1/stats/*` | analytics-oriented statistics | JSON | memory health, staleness, session extraction |++The boundary is:++- `/metrics` only carries **low-cardinality, low-cost** metrics+- `/api/v1/stats/*` continues to carry analytics-oriented statistics without being constrained by the Prometheus scraping model++## Metrics Architecture++The current metrics stack in OpenViking has four layers:++```text+Business logic / HTTP requests / background tasks+ │+ ▼+ DataSource+ (event emission / state reads)+ │+ ▼+ Collector+ (semantic routing + labels)+ │+ ▼+ MetricRegistry+ (in-process metric store)+ │+ ▼+ Exporter+ (Prometheus text rendering)+ │+ ▼+ /metrics+```++### DataSource++DataSources provide inputs to the metrics system in two main forms:++- **Event-based**: business code emits events at key points, such as retrieval completion, successful model calls, or resource ingestion stage completion+- **Read-based**: current state is read before `/metrics` export, such as queue state, lock state, or probe state++### Collector++Collectors turn inputs into metric semantics:++- choose which metric to write+- choose which labels to attach+- define how failure is exposed, such as `valid=1/0`++### MetricRegistry++The MetricRegistry is the in-process metric store that keeps the current metric values and serves them to the exporter.++### Exporter++The first exporter implementation is the Prometheus exporter, which renders registry contents into Prometheus exposition text.++## Usage++### Accessing `/metrics`++In the current implementation, `/metrics` is not wired to `get_request_context` or other auth dependencies, so from the code-path perspective it currently behaves as a public scrape endpoint.++```bash+curl http://localhost:1933/metrics+```++If your deployment protects `/metrics` at the gateway, reverse proxy, or service discovery layer, attach auth according to the deployment environment.++### Prometheus Scrape Example++```yaml+scrape_configs:+ - job_name: openviking+ metrics_path: /metrics+ static_configs:+ - targets: ["localhost:1933"]+```++### Understanding Common Labels++| Label | Meaning | Example |+|-------|---------|---------|+| `account_id` | tenant dimension label | `test-account`, `__unknown__`, `__overflow__` |+| `route` | HTTP route template | `/api/v1/search/find` |+| `method` | HTTP method | `GET`, `POST` |+| `status` | request or stage status | `200`, `ok`, `error` |+| `operation` | structured operation name | `search.find`, `resources.add_resource` |+| `context_type` | retrieval context type | `resource` |+| `provider` | model or external service provider | `volcengine` |+| `model_name` | model name | `doubao-seed-1-8-251228` |+| `stage` | resource processing stage | `parse`, `persist`, `process` |+| `valid` | whether the current sample is fresh and valid | `1` / `0` |++Notes:++- `account_id` is only enabled on controlled allowlisted metric families to prevent high-cardinality growth+- `valid=0` means the current state/probe sample is a fallback or stale value, not that the label itself is malformed++## Key Metric Families++The metric summaries below are based on representative metrics currently exposed in `.vscode/.workdir/metric/METRIC_res.md`.++### Requests and Operations++| Metric Family | Type | Common Labels | Meaning |+|---------------|------|---------------|---------|+| `openviking_http_requests_total` | Counter | `account_id, method, route, status` | total HTTP requests |+| `openviking_http_request_duration_seconds` | Histogram | `account_id, method, route, status` | HTTP latency distribution |+| `openviking_http_inflight_requests` | Gauge | `account_id, route` | current inflight requests (in-process approximation) |+| `openviking_operation_requests_total` | Counter | `account_id, operation, status` | total structured operations |+| `openviking_operation_duration_seconds` | Histogram | `account_id, operation, status` | structured operation duration distribution |++Typical usage:++- inspect whether `/api/v1/search/find` or `/api/v1/resources` is slowing down+- inspect whether a specific `operation` has elevated error rates++### Retrieval and Resource Processing++| Metric Family | Type | Common Labels | Meaning |+|---------------|------|---------------|---------|+| `openviking_retrieval_requests_total` | Counter | `account_id, context_type` | retrieval request count |+| `openviking_retrieval_results_total` | Counter | `account_id, context_type` | total retrieved results |+| `openviking_retrieval_latency_seconds` | Histogram | `account_id, context_type` | retrieval latency distribution |+| `openviking_resource_stage_total` | Counter | `account_id, stage, status` | count of resource ingestion stages |+| `openviking_resource_stage_duration_seconds` | Histogram | `account_id, stage, status` | duration distribution of ingestion stages |++Typical `stage` values include:++- `request`+- `parse`+- `summarize`+- `persist`+- `finalize`+- `process`++### Model Calls and Tokens++| Metric Family | Type | Common Labels | Meaning |+|---------------|------|---------------|---------|+| `openviking_model_calls_total` | Counter | `model_type, provider, model_name` | unified model call count |+| `openviking_model_tokens_total` | Counter | `model_type, provider, model_name, token_type` | unified model token count |+| `openviking_vlm_calls_total` | Counter | `account_id, provider, model_name` | VLM call count |+| `openviking_vlm_tokens_input_total` | Counter | `account_id, provider, model_name` | VLM input tokens |+| `openviking_vlm_tokens_output_total` | Counter | `account_id, provider, model_name` | VLM output tokens |+| `openviking_vlm_tokens_total` | Counter | `account_id, provider, model_name` | VLM total tokens |+| `openviking_vlm_call_duration_seconds` | Histogram | `account_id, provider, model_name` | VLM call duration distribution |+| `openviking_embedding_requests_total` | Counter | `account_id, status` | embedding request count |+| `openviking_embedding_latency_seconds` | Histogram | `account_id, status` | embedding latency distribution |++Notes:++- `openviking_model_*` gives a unified cross-model view for embedding and VLM usage+- `openviking_vlm_*` and `openviking_embedding_*` are better suited for workload-specific dashboards++### Queues, Locks, and Runtime State++| Metric Family | Type | Common Labels | Meaning |+|---------------|------|---------------|---------|+| `openviking_queue_processed_total` | Counter | `queue` | total processed items per queue |+| `openviking_queue_pending` | Gauge | `queue` | pending queue items |+| `openviking_queue_in_progress` | Gauge | `queue` | in-progress queue items |+| `openviking_lock_active` | Gauge | none | current active locks |+| `openviking_lock_waiting` | Gauge | none | locks currently waiting |+| `openviking_lock_stale` | Gauge | none | potentially stale locks |++These help answer:++- Is there queue backlog?+- Is there lock contention or stale locking?++### Probes and Health State++| Metric Family | Type | Common Labels | Meaning |+|---------------|------|---------------|---------|+| `openviking_service_readiness` | Gauge | may include `valid` | main service readiness |+| `openviking_api_key_manager_readiness` | Gauge | may include `valid` | API key manager readiness |+| `openviking_storage_readiness` | Gauge | `probe, valid` | storage probe, for example `agfs` |+| `openviking_model_provider_readiness` | Gauge | `provider, valid` | model provider readiness |+| `openviking_async_system_readiness` | Gauge | `probe, valid` | async system readiness |+| `openviking_retrieval_backend_readiness` | Gauge | `probe, valid` | retrieval backend readiness |+| `openviking_encryption_component_health` | Gauge | `valid` | overall encryption component health |+| `openviking_encryption_root_key_ready` | Gauge | `valid` | whether the root key is ready |+| `openviking_encryption_kms_provider_ready` | Gauge | `provider, valid` | KMS provider readiness |++Meaning of `valid`:++- `valid="1"`: the sample was produced by a successful refresh+- `valid="0"`: the sample is a fallback or stale value and should be treated with caution++### Component and Observer Aggregate Metrics++| Metric Family | Type | Common Labels | Meaning |+|---------------|------|---------------|---------|+| `openviking_component_health` | Gauge | `component, valid` | component health state |+| `openviking_component_errors` | Gauge | `component, valid` | component error state |+| `openviking_observer_components_total` | Gauge | `valid` | number of observed components |+| `openviking_observer_components_unhealthy` | Gauge | `valid` | number of unhealthy components |+| `openviking_observer_components_with_errors` | Gauge | `valid` | number of components with errors |++Typical `component` values include:++- `queue`+- `models`+- `lock`+- `retrieval`+- `vikingdb`++### VikingDB and Model Usage Statistics++| Metric Family | Type | Common Labels | Meaning |+|---------------|------|---------------|---------|+| `openviking_vikingdb_collection_health` | Gauge | `collection, valid` | collection health |+| `openviking_vikingdb_collection_vectors` | Gauge | `collection, valid` | current vector count per collection |+| `openviking_model_usage_available` | Gauge | `model_type, valid` | whether model usage statistics are currently available |++Possible `model_type` values include:++- `vlm`+- `embedding`+- `rerank`++## Configuration Example++### Enabling Metrics++In `ov.conf`, the new metrics subsystem can be explicitly enabled through `server.metrics`:++```json+{+ "server": {+ "telemetry": {+ "prometheus": {+ "enabled": true+ }+ },+ "metrics": {+ "enabled": true,+ "account_dimension": {+ "enabled": true,+ "max_active_accounts": 100,+ "metric_allowlist": [+ "openviking_http_requests_total",+ "openviking_http_request_duration_seconds",+ "openviking_http_inflight_requests",+ "openviking_operation_requests_total",+ "openviking_operation_duration_seconds",+ "openviking_vlm_calls_total",+ "openviking_vlm_call_duration_seconds"+ ]+ }+ }+ }+}+```++Recommended mental model:++- `server.metrics.enabled`: master switch for the new metrics subsystem+- `server.metrics.account_dimension`: controls whether `account_id` labels are enabled and where they are allowed+- `server.telemetry.prometheus.enabled`: compatibility path for the older configuration entry; the current implementation still reads it++### Recommended `account_id` Usage++- enabled by default, but only allowlisted metric families will receive tenant ids (empty allowlist still yields `__unknown__`)+- do not turn `user_id`, `session_id`, or `resource_uri` into labels+- only enable tenant dimensions on a small set of critical dashboard and alert metrics+- `metric_allowlist` supports a limited wildcard syntax: only trailing `*` prefix matches (e.g. `openviking_rerank_*`)+- a standalone `*` is not supported, nor full glob/regex patterns++## Related Documentation++- [Architecture Overview](./01-architecture.md) - overall OpenViking architecture+- [Multi-Tenant](./11-multi-tenant.md) - `account/user/agent` isolation model+- [Data Encryption](./10-encryption.md) - storage-layer encryption and isolation+- [Metrics API](../api/09-metrics.md) - `/metrics` endpoint usage+- [Metrics Design](../../design/metric-design.md) - metrics system design detailsdiff --git a/docs/en/guides/05-observability.md b/docs/en/guides/05-observability.mdindex b4705f242..581c8b919 100644--- a/docs/en/guides/05-observability.md+++ b/docs/en/guides/05-observability.md@@ -6,6 +6,7 @@ This guide collects the current OpenViking observability entry points in one pla- request-level `telemetry`- terminal-side `ov tui`- web-side `OpenViking Console`+- `/metrics` time-series metricsIf you just want to know where to look first, start with the table below.@@ -17,6 +18,7 @@ If you just want to know where to look first, start with the table below.| `ov tui` | `viking://` trees, directory summaries, file content, vector records | development debugging, verifying that data actually landed || `OpenViking Console` | web UI for browsing, search, resource import, tenants, and system state | interactive investigation without typing every command || `telemetry` | per-request duration, token usage, vector retrieval, ingestion stages | debugging one specific slow or unexpected call |+| `/metrics` | request trends, error rates, latency distribution, queue and probe state | Prometheus scraping, Grafana dashboards, alert rules |## Service health and component status@@ -230,9 +232,120 @@ For the full field reference, supported operations, and more examples, see:- [Operation Telemetry Reference](07-operation-telemetry.md)+## Use `/metrics` for time-series observability++`/metrics` is OpenViking's time-series metrics endpoint for the Prometheus scraping model. It is well suited for questions like:++- Has HTTP traffic increased abnormally over the last few minutes?+- Is the error rate for a route or operation continuing to rise?+- Is latency distribution getting worse?+- Is queue backlog starting to build up?+- Have key dependencies, probes, or model providers become unhealthy?++Compared with `observer/*`, `/metrics` is better for **trends, aggregation, and alerting**. `observer/*` is better for inspecting the current point-in-time state by hand.++Compared with `telemetry`, `/metrics` focuses on **aggregated time series**, while `telemetry` focuses on **what happened inside one specific request**.++### Access `/metrics` directly++In the current implementation, `/metrics` is not wired to `get_request_context` or other auth dependencies, so from the code-path perspective it currently behaves as a public scrape endpoint:++```bash+curl http://localhost:1933/metrics+```++If your deployment protects `/metrics` at the gateway, reverse proxy, or service discovery layer, attach auth according to the deployment environment.++### Prometheus scrape example++The most common setup is to let Prometheus scrape it on a schedule:++```yaml+scrape_configs:+ - job_name: openviking+ metrics_path: /metrics+ static_configs:+ - targets: ["localhost:1933"]+```++### Import and view the dashboard in Grafana++Once Prometheus is successfully scraping `/metrics`, the next common step is to import the OpenViking demo dashboard into Grafana.++**Step 1: Confirm that Prometheus is already scraping `/metrics`**++Before importing the dashboard, make sure the Prometheus data source can already query OpenViking metrics. The quickest checks are:++- run `openviking_http_requests_total` in the Prometheus UI+- or run `openviking_service_readiness`+- if either query returns time series, Grafana should be able to render panels afterwards++If there is no data yet, go back to the Prometheus scrape configuration above and verify `targets`, `metrics_path`, and network connectivity first.++**Step 2: Import the official demo dashboard into Grafana**++The OpenViking repository already includes a ready-to-import dashboard JSON:++- [openviking_demo_dashboard.json](../../../examples/grafana/openviking_demo_dashboard.json)++You can import it with the following steps:++1. Sign in to Grafana.+2. Open `Dashboards` from the left-side menu.+3. Click `New` or `Import` in the top-right corner.+4. Upload the JSON file, or paste the contents of the linked file.+5. Select Prometheus as the data source on the import screen.+6. Click `Import` to finish.++If the dashboard imports but panels are empty, the first two things to verify are:++- whether Grafana is bound to the correct Prometheus data source+- whether Prometheus is actually scraping `openviking_*` metrics++**Step 3: What to look at after the dashboard opens**++After import, the most useful panels usually come from these metric families:++- `openviking_http_*`: HTTP request volume, latency, and inflight requests+- `openviking_operation_*`: structured operation success rates and latency+- `openviking_queue_*`: queue throughput, backlog, and in-progress work+- `openviking_*_readiness`: dependency and probe health state++A beginner-friendly viewing order is:++1. Start with readiness and health panels to confirm that the system and dependencies are generally healthy.+2. Move to HTTP and operation panels to see whether traffic, error rates, or latency have changed.+3. Then inspect queue panels to determine whether async work is backing up.+4. Finally, narrow the scope with labels such as `account_id`, `route`, and `status`.++**Step 4: What the final result looks like**++After a successful import, you should see a dashboard centered on OpenViking requests, queues, probes, model calls, and overall system state. For a visual reference, see:++- [grafana-demo-dashboard.png](../../images/grafana-demo-dashboard.png)++This screenshot helps you quickly verify whether the imported layout looks correct. If the dashboard structure matches but some panels are empty, it usually means the corresponding metrics have not produced samples yet, or the filters do not match the current traffic.++### Understanding common labels++The most common labels you will use while investigating dashboards are:++- `account_id`: tenant dimension label. It is only enabled on controlled allowlisted metric families. Unidentified requests fall into `__unknown__`, and values beyond the active-tenant budget fall into `__overflow__`+- `route`: HTTP route template, for example `/api/v1/search/find`+- `status`: request or stage status, such as `200`, `ok`, or `error`+- `valid`: whether the sample came from a successful refresh; `valid="0"` usually indicates a fallback or stale sample++### When to use `/metrics` vs. other entry points++- To check whether the service is healthy and which component is currently unhealthy, start with `/health` and `observer/*`+- To verify whether resources really landed and vectors were actually written, use `ov tui`+- To inspect why one specific request was slow, where tokens went, or which stage blocked resource processing, use `telemetry`+- To inspect whether request volume, error rates, or latency are degrading over time, use `/metrics`+## Related Documentation- [Deployment](03-deployment.md) - server setup- [Authentication](04-authentication.md) - API key setup- [Operation Telemetry Reference](07-operation-telemetry.md) - request-level structured tracing- [System API](../api/07-system.md) - system and observer API reference+- [Metrics](../concepts/12-metrics.md) - time-series metrics and configurationdiff --git a/docs/images/grafana-demo-dashboard.png b/docs/images/grafana-demo-dashboard.pngnew file mode 100644index 000000000..59f18d03aBinary files /dev/null and b/docs/images/grafana-demo-dashboard.png differdiff --git a/docs/zh/api/01-overview.md b/docs/zh/api/01-overview.mdindex 436572193..7b5949e6d 100644--- a/docs/zh/api/01-overview.md+++ b/docs/zh/api/01-overview.md@@ -281,6 +281,7 @@ openviking -o json ls viking://resources/| 方法 | 路径 | 说明 ||------|------|------|| GET | `/health` | 健康检查(无需认证) |+| GET | `/metrics` | Prometheus 指标导出 || GET | `/api/v1/system/status` | 系统状态 || POST | `/api/v1/system/wait` | 等待处理完成 |@@ -373,4 +374,5 @@ openviking -o json ls viking://resources/- [会话管理](05-sessions.md) - 会话管理- [技能](04-skills.md) - 技能管理- [系统](07-system.md) - 系统和监控 API+- [指标与 Metrics](09-metrics.md) - Prometheus 指标导出与抓取说明- [管理员](08-admin.md) - 多租户管理 APIdiff --git a/docs/zh/api/09-metrics.md b/docs/zh/api/09-metrics.mdnew file mode 100644index 000000000..ff3a38582--- /dev/null+++ b/docs/zh/api/09-metrics.md@@ -0,0 +1,87 @@+# 指标与 Metrics++OpenViking 提供 `/metrics` 端点,用于向 Prometheus、Grafana Agent 等监控系统导出运行时指标。++与 `/api/v1/observer/*` 和 `/api/v1/stats/*` 不同,`/metrics` 的定位是:++- 面向机器抓取,而不是面向人工阅读+- 返回 Prometheus exposition 文本,而不是统一 JSON 包装+- 偏系统运行态与服务运行质量,不承担业务分析接口职责++## API 参考++### metrics()++导出当前进程内的 Prometheus 指标文本。++该端点通常被 Prometheus 定时抓取,也可以用于本地调试或手工排查。++**认证**++- 当前实现中,`/metrics` 未接入 `get_request_context` 等鉴权依赖,因此可直接访问。+- 也就是说,从代码实现角度看,`/metrics` 当前等价于公开抓取端点。+- 如果后续通过网关、反向代理或服务端策略收紧访问控制,应以实际部署配置为准。++**HTTP API**++```+GET /metrics+```++```bash+curl -X GET http://localhost:1933/metrics+```++如果你的部署环境在网关或代理层要求鉴权,可以按网关要求附加请求头,例如:++```bash+curl -X GET http://localhost:1933/metrics \+ -H "Authorization: Bearer your-key"+```++**响应格式**++成功时返回 `text/plain; version=0.0.4; charset=utf-8`,内容为 Prometheus exposition 格式文本,例如:++```text+# HELP openviking_http_requests_total Total number of HTTP requests+# TYPE openviking_http_requests_total counter+openviking_http_requests_total{method="GET",route="/api/v1/system/status",status="200"} 12++# HELP openviking_http_inflight_requests Number of inflight HTTP requests+# TYPE openviking_http_inflight_requests gauge+openviking_http_inflight_requests{route="/api/v1/system/status"} 0+```++当指标系统未启用时,返回:++- HTTP 状态码:`404`+- 响应体:++```text+Prometheus metrics are disabled.+```++**示例:Prometheus 抓取配置**++```yaml+scrape_configs:+ - job_name: openviking+ metrics_path: /metrics+ static_configs:+ - targets: ["localhost:1933"]+```++如果你的部署环境对 `/metrics` 做了网关鉴权,可以通过反向代理、service discovery,或 Prometheus 支持的鉴权方式为该抓取任务配置请求头。++**注意事项**++- `/metrics` 适合高频抓取,因此其中的指标应保持低基数、低成本。+- `/metrics` 返回的是 Prometheus 文本,不是标准 OpenViking API 的 `{status, result, time}` JSON 结构。+- 业务分析类统计(如 memory health、staleness、session extraction)应继续使用 `/api/v1/stats/*`,而不是迁移到 `/metrics`。+- 人工查看组件瞬时状态更适合使用 `/api/v1/observer/*`。++## 相关文档++- [系统与监控](07-system.md) - 健康检查、系统状态与 Observer API+- [API 概览](01-overview.md) - 所有 API 端点共享约定diff --git a/docs/zh/concepts/01-architecture.md b/docs/zh/concepts/01-architecture.mdindex ba21f2a00..d060a3352 100644--- a/docs/zh/concepts/01-architecture.md+++ b/docs/zh/concepts/01-architecture.md@@ -172,3 +172,7 @@ curl http://localhost:1933/api/v1/search/find \- [检索机制](./07-retrieval.md) - 检索流程详解- [上下文提取](./06-extraction.md) - 解析和提取流程- [会话管理](./08-session.md) - 会话和记忆管理+- [事务模型](./09-transaction.md) - 写入与一致性模型+- [数据加密](./10-encryption.md) - 静态数据加密与密钥架构+- [多租户](./11-multi-tenant.md) - account / user / agent 隔离模型+- [指标](./12-metrics.md) - `/metrics` 使用方式与关键指标说明diff --git a/docs/zh/concepts/12-metrics.md b/docs/zh/concepts/12-metrics.mdnew file mode 100644index 000000000..b34308105--- /dev/null+++ b/docs/zh/concepts/12-metrics.md@@ -0,0 +1,314 @@+# 指标与 Metrics++OpenViking 提供一套面向机器抓取的指标体系,用于暴露系统运行态、请求质量、模型调用情况、资源处理吞吐、探针健康状态等信息。++与人类排障用的 `/api/v1/observer/*` 和业务分析用的 `/api/v1/stats/*` 不同,Metrics 的目标是:++- 供 Prometheus、Grafana Agent 等系统**高频抓取**+- 使用低基数、可聚合的指标模型+- 服务于监控、告警、容量观察与回归排查++## 概述++### 为什么需要 Metrics++Metrics 适合回答这类问题:++- 最近一段时间 HTTP 请求是否异常升高?+- 资源导入、检索、模型调用是否变慢?+- 队列是否堆积?+- 关键依赖(存储、模型、VikingDB、加密、异步系统)当前是否可用?+- 某些租户是否出现异常流量或异常错误率?++相比日志和 observer 状态,metrics 更适合做:++- 持续抓取+- 时间序列聚合+- Dashboard 展示+- 告警规则++### 与 Observer / Stats 的区别++| 能力 | 适合什么 | 输出形式 | 典型使用场景 |+|------|----------|----------|--------------|+| `/metrics` | 在线监控、告警、聚合趋势 | Prometheus exposition 文本 | Grafana 看板、Prometheus 抓取 |+| `/api/v1/observer/*` | 人工查看组件瞬时状态 | JSON / 状态表 | 排障、健康检查 |+| `/api/v1/stats/*` | 分析型统计 | JSON | memory health、staleness、session extraction 等 |++设计边界是:++- `/metrics` 只承载**低基数、低成本**指标+- `/api/v1/stats/*` 继续承载分析型统计,不为了 Prometheus 抓取模型牺牲表达能力++## 指标体系架构++OpenViking 当前的 metrics 体系由四层组成:++```text+业务逻辑 / HTTP 请求 / 后台任务+ │+ ▼+ DataSource+ (事件发射 / 状态读取)+ │+ ▼+ Collector+ (语义分流、标签决定)+ │+ ▼+ MetricRegistry+ (进程内指标注册中心)+ │+ ▼+ Exporter+ (Prometheus 文本导出)+ │+ ▼+ /metrics+```++### DataSource++DataSource 负责提供指标输入,主要有两种方式:++- **事件型**:业务代码在关键路径发射事件,例如检索完成、模型调用成功、资源导入阶段完成+- **读取型**:在 `/metrics` 抓取前读取当前状态,例如队列状态、锁状态、探针状态++### Collector++Collector 负责把输入转成指标语义:++- 决定写哪个指标+- 决定携带哪些标签+- 决定失败时如何暴露(例如 `valid=1/0`)++### MetricRegistry++MetricRegistry 是进程内的指标注册中心,用于保存当前指标值,并在导出时统一读取。++### Exporter++当前首个落地导出器是 Prometheus Exporter,用于把 registry 中的指标渲染成 Prometheus exposition 文本。++## 使用方式++### 访问 `/metrics`++当前实现中,`/metrics` 未接入 `get_request_context` 等鉴权依赖,因此从代码行为上看,它当前等价于公开抓取端点。++```bash+curl http://localhost:1933/metrics+```++如果你的部署环境通过网关、反向代理或服务发现层对 `/metrics` 做了保护,则应按部署方式附加鉴权。++### Prometheus 抓取示例++```yaml+scrape_configs:+ - job_name: openviking+ metrics_path: /metrics+ static_configs:+ - targets: ["localhost:1933"]+```++### 如何理解常见标签++| 标签 | 含义 | 示例 |+|------|------|------|+| `account_id` | 租户维度标签 | `test-account`、`__unknown__`、`__overflow__` |+| `route` | HTTP 路由模板 | `/api/v1/search/find` |+| `method` | HTTP 方法 | `GET`、`POST` |+| `status` | 请求或阶段状态 | `200`、`ok`、`error` |+| `operation` | 操作名称 | `search.find`、`resources.add_resource` |+| `context_type` | 检索上下文类型 | `resource` |+| `provider` | 模型或外部服务提供方 | `volcengine` |+| `model_name` | 模型名称 | `doubao-seed-1-8-251228` |+| `stage` | 资源处理阶段 | `parse`、`persist`、`process` |+| `valid` | 当前样本是否为有效新鲜值 | `1` / `0` |++其中:++- `account_id` 只在受控白名单指标上启用,避免高基数失控+- `valid=0` 表示该状态/探针的当前样本是失败回退值或 stale fallback,不代表标签本身错误++## 关键指标说明++下面的指标说明基于当前实际暴露的代表性指标输出(整理自 `.vscode/.workdir/metric/METRIC_res.md`)。++### 请求与操作++| 指标族 | 类型 | 常见标签 | 含义 |+|--------|------|----------|------|+| `openviking_http_requests_total` | Counter | `account_id, method, route, status` | HTTP 请求总量 |+| `openviking_http_request_duration_seconds` | Histogram | `account_id, method, route, status` | HTTP 请求耗时分布 |+| `openviking_http_inflight_requests` | Gauge | `account_id, route` | 当前 inflight 请求数(进程内近似值) |+| `openviking_operation_requests_total` | Counter | `account_id, operation, status` | 结构化操作总量 |+| `openviking_operation_duration_seconds` | Histogram | `account_id, operation, status` | 结构化操作耗时分布 |++适用场景:++- 看 `/api/v1/search/find`、`/api/v1/resources` 是否异常变慢+- 看某个 `operation` 是否错误率升高++### 检索与资源处理++| 指标族 | 类型 | 常见标签 | 含义 |+|--------|------|----------|------|+| `openviking_retrieval_requests_total` | Counter | `account_id, context_type` | 检索请求次数 |+| `openviking_retrieval_results_total` | Counter | `account_id, context_type` | 检索返回结果数量累计 |+| `openviking_retrieval_latency_seconds` | Histogram | `account_id, context_type` | 检索耗时分布 |+| `openviking_resource_stage_total` | Counter | `account_id, stage, status` | 资源导入各阶段执行次数 |+| `openviking_resource_stage_duration_seconds` | Histogram | `account_id, stage, status` | 资源导入阶段耗时分布 |++典型 `stage` 包括:++- `request`+- `parse`+- `summarize`+- `persist`+- `finalize`+- `process`++### 模型调用与 Token++| 指标族 | 类型 | 常见标签 | 含义 |+|--------|------|----------|------|+| `openviking_model_calls_total` | Counter | `model_type, provider, model_name` | 模型调用总量(统一视角) |+| `openviking_model_tokens_total` | Counter | `model_type, provider, model_name, token_type` | 模型 token 累计量 |+| `openviking_vlm_calls_total` | Counter | `account_id, provider, model_name` | VLM 调用次数 |+| `openviking_vlm_tokens_input_total` | Counter | `account_id, provider, model_name` | VLM 输入 token |+| `openviking_vlm_tokens_output_total` | Counter | `account_id, provider, model_name` | VLM 输出 token |+| `openviking_vlm_tokens_total` | Counter | `account_id, provider, model_name` | VLM 总 token |+| `openviking_vlm_call_duration_seconds` | Histogram | `account_id, provider, model_name` | VLM 调用耗时分布 |+| `openviking_embedding_requests_total` | Counter | `account_id, status` | embedding 请求数 |+| `openviking_embedding_latency_seconds` | Histogram | `account_id, status` | embedding 耗时分布 |++说明:++- `openviking_model_*` 是统一模型视角,便于同时看 embedding / vlm+- `openviking_vlm_*` 和 `openviking_embedding_*` 更适合业务侧针对性看板++### 队列、锁与系统运行态++| 指标族 | 类型 | 常见标签 | 含义 |+|--------|------|----------|------|+| `openviking_queue_processed_total` | Counter | `queue` | 队列累计处理量 |+| `openviking_queue_pending` | Gauge | `queue` | 队列待处理数 |+| `openviking_queue_in_progress` | Gauge | `queue` | 队列执行中数量 |+| `openviking_lock_active` | Gauge | 无 | 当前活跃锁数量 |+| `openviking_lock_waiting` | Gauge | 无 | 当前等待中的锁数量 |+| `openviking_lock_stale` | Gauge | 无 | 可能 stale 的锁数量 |++这些指标适合回答:++- 是否有队列堆积?+- 是否有锁竞争或 stale lock?++### 探针与健康状态++| 指标族 | 类型 | 常见标签 | 含义 |+|--------|------|----------|------|+| `openviking_service_readiness` | Gauge | 可含 `valid` | 服务主 readiness |+| `openviking_api_key_manager_readiness` | Gauge | 可含 `valid` | API Key Manager readiness |+| `openviking_storage_readiness` | Gauge | `probe, valid` | 存储探针,例如 `agfs` |+| `openviking_model_provider_readiness` | Gauge | `provider, valid` | 模型提供方 readiness |+| `openviking_async_system_readiness` | Gauge | `probe, valid` | 异步系统 readiness |+| `openviking_retrieval_backend_readiness` | Gauge | `probe, valid` | 检索后端 readiness |+| `openviking_encryption_component_health` | Gauge | `valid` | 加密组件总体健康 |+| `openviking_encryption_root_key_ready` | Gauge | `valid` | 根密钥是否就绪 |+| `openviking_encryption_kms_provider_ready` | Gauge | `provider, valid` | KMS provider readiness |++`valid` 的意义:++- `valid="1"`:当前样本是本次成功刷新得到的结果+- `valid="0"`:当前样本是失败回退值或 stale fallback,说明该探针/状态当前不可完全信任++### 组件与 Observer 聚合指标++| 指标族 | 类型 | 常见标签 | 含义 |+|--------|------|----------|------|+| `openviking_component_health` | Gauge | `component, valid` | 组件健康状态 |+| `openviking_component_errors` | Gauge | `component, valid` | 组件错误状态 |+| `openviking_observer_components_total` | Gauge | `valid` | observer 观测到的组件数量 |+| `openviking_observer_components_unhealthy` | Gauge | `valid` | 不健康组件数量 |+| `openviking_observer_components_with_errors` | Gauge | `valid` | 有错误组件数量 |++典型 `component` 包括:++- `queue`+- `models`+- `lock`+- `retrieval`+- `vikingdb`++### VikingDB 与模型使用统计++| 指标族 | 类型 | 常见标签 | 含义 |+|--------|------|----------|------|+| `openviking_vikingdb_collection_health` | Gauge | `collection, valid` | collection 健康状态 |+| `openviking_vikingdb_collection_vectors` | Gauge | `collection, valid` | collection 当前向量数 |+| `openviking_model_usage_available` | Gauge | `model_type, valid` | 模型使用统计是否可用 |++其中 `model_type` 可能包括:++- `vlm`+- `embedding`+- `rerank`++## 配置示例++### 启用 Metrics++在 `ov.conf` 中,可以通过 `server.metrics` 显式启用新的 metrics 子系统:++```json+{+ "server": {+ "telemetry": {+ "prometheus": {+ "enabled": true+ }+ },+ "metrics": {+ "enabled": true,+ "account_dimension": {+ "enabled": true,+ "max_active_accounts": 100,+ "metric_allowlist": [+ "openviking_http_requests_total",+ "openviking_http_request_duration_seconds",+ "openviking_http_inflight_requests",+ "openviking_operation_requests_total",+ "openviking_operation_duration_seconds",+ "openviking_vlm_calls_total",+ "openviking_vlm_call_duration_seconds"+ ]+ }+ }+ }+}+```++推荐理解方式:++- `server.metrics.enabled`:新指标体系总开关+- `server.metrics.account_dimension`:控制 `account_id` 标签是否启用以及启用范围+- `server.telemetry.prometheus.enabled`:兼容旧配置入口,当前实现仍兼容读取++### `account_id` 标签的使用建议++- 默认开启,但仅对白名单指标启用(`metric_allowlist` 为空时仍会输出为 `__unknown__`)+- 不要把 `user_id`、`session_id`、`resource_uri` 这类高基数字段做成标签+- 对于看板和告警,只对少量关键指标族打开租户维度+- `metric_allowlist` 支持有限通配符:仅支持**末尾 `*` 的前缀匹配**(例如 `openviking_rerank_*`、`openviking_embedding_*`)+- 不支持单独的 `*`(空前缀),也不支持中间通配、完整 glob 或正则+++## 相关文档++- [架构概述](./01-architecture.md) - OpenViking 总体架构+- [多租户](./11-multi-tenant.md) - `account/user/agent` 隔离模型+- [数据加密](./10-encryption.md) - 存储层加密与隔离+- [Metrics API](../api/09-metrics.md) - `/metrics` 端点用法+- [指标体系设计](../../design/metric-design.md) - 指标体系设计细节diff --git a/docs/zh/guides/05-observability.md b/docs/zh/guides/05-observability.mdindex 8bd8c8578..266d790af 100644--- a/docs/zh/guides/05-observability.md+++ b/docs/zh/guides/05-observability.md@@ -6,6 +6,7 @@- 请求级 `telemetry`- 终端侧 `ov tui`- Web 侧 `OpenViking Console`+- `/metrics` 时序指标如果你只想快速判断“该看哪里”,先看下面这张表。@@ -17,6 +18,7 @@| `ov tui` | `viking://` 文件树、目录摘要、文件正文、向量记录 | 开发调试、核对资源是否真正落库 || `OpenViking Console` | Web UI 里的文件浏览、检索、资源导入、租户与系统状态 | 不想手敲命令时做交互式排查 || `telemetry` | 单次请求耗时、token、向量检索、资源处理阶段 | 排查一次具体调用为什么慢、为什么结果异常 |+| `/metrics` | 请求量趋势、错误率、时延分布、队列与探针状态 | Prometheus 抓取、Grafana 看板、告警规则 |## 服务健康与组件状态@@ -230,9 +232,114 @@ curl -X POST http://localhost:1933/api/v1/search/find \- [操作级 Telemetry 参考](07-operation-telemetry.md)+## 用 `/metrics` 做时序观测++`/metrics` 是 OpenViking 面向 Prometheus 抓取模型提供的时序指标端点,适合回答这类问题:++- 最近一段时间 HTTP 请求量是不是突然升高了+- 某个接口或操作的错误率是不是在持续上升+- 请求耗时分布是否变差+- 队列是否开始堆积+- 关键依赖、探针或模型提供方是否进入不健康状态++和前面的 `observer/*` 相比,`/metrics` 更适合看**趋势、聚合和告警**;而 `observer/*` 更适合人工查看某一时刻的瞬时状态。++和前面的 `telemetry` 相比,`/metrics` 关注的是**聚合后的时间序列**;`telemetry` 关注的是**某一次请求内部到底发生了什么**。++### 直接访问 `/metrics`++当前实现中,`/metrics` 未接入 `get_request_context` 等鉴权依赖,因此从代码行为上看,它当前等价于公开抓取端点:++```bash+curl http://localhost:1933/metrics+```++如果你的部署环境通过网关、反向代理或服务发现层对 `/metrics` 做了保护,则应按部署方式附加鉴权。++### Prometheus 抓取示例++最常见的使用方式是让 Prometheus 定时抓取:++```yaml+scrape_configs:+ - job_name: openviking+ metrics_path: /metrics+ static_configs:+ - targets: ["localhost:1933"]+```++### 在 Grafana 中导入和查看 Dashboard++如果你已经让 Prometheus 成功抓取 `/metrics`,下一步最常见的做法就是在 Grafana 中导入 OpenViking 的 demo dashboard。++**第 1 步:先确认 Prometheus 已经抓到 `/metrics`**++在导入 Grafana dashboard 之前,先确认 Prometheus 数据源里已经能查到 OpenViking 指标。最简单的判断方式是:++- 在 Prometheus UI 里执行 `openviking_http_requests_total`+- 或执行 `openviking_service_readiness`+- 如果已经能返回时间序列,说明 Grafana 后续就能正常出图++如果这一步没有数据,先回到上面的 Prometheus 抓取配置,确认 `targets`、`metrics_path` 和网络连通性。++**第 2 步:在 Grafana 导入官方 demo dashboard**++OpenViking 仓库里已经提供了一个可直接导入的 dashboard JSON:++- [openviking_demo_dashboard.json](../../../examples/grafana/openviking_demo_dashboard.json)++导入步骤可以按下面做:++1. 登录你的 Grafana。+2. 在左侧菜单进入 `Dashboards`。+3. 点击右上角的 `New` 或 `Import`。+4. 选择上传 JSON 文件,或把上面链接对应文件的内容粘贴进去。+5. 在导入页面选择 Prometheus 作为数据源。+6. 点击 `Import` 完成导入。++如果导入后面板为空,通常优先检查两件事:++- Grafana 绑定的数据源是不是正确的 Prometheus+- Prometheus 里是否真的已经抓到了 `openviking_*` 指标++**第 3 步:打开 dashboard 后重点看什么**++接入之后,通常就可以在 Grafana 里重点观察这些指标族对应的面板:++- `openviking_http_*`:HTTP 请求量、耗时、inflight+- `openviking_operation_*`:结构化操作的成功率和耗时+- `openviking_queue_*`:队列处理量、积压和执行中数量+- `openviking_*_readiness`:依赖与探针健康状态++**第 4 步:最终效果长什么样**++导入成功后,你最终会看到一个以 OpenViking 请求、队列、探针、模型调用和系统状态为主的总览 dashboard。效果示意可以参考:++- [grafana-demo-dashboard.png](../../images/grafana-demo-dashboard.png)++这张图可以帮助你快速确认“导入后的面板布局是不是正常”。如果你的 dashboard 基本结构和它一致,但局部面板没有数据,通常说明是对应指标当前没有产生样本,或者筛选条件与实际流量不匹配。++### 如何理解常见标签++排查看板时,最常见的几个标签是:++- `account_id`:租户维度标签。只在受控白名单指标上开启,未识别请求会被归到 `__unknown__`,超出活跃租户预算时会落到 `__overflow__`+- `route`:HTTP 路由模板,例如 `/api/v1/search/find`+- `status`:请求或阶段状态,例如 `200`、`ok`、`error`+- `valid`:当前样本是否是本次成功刷新得到的有效值;`valid="0"` 通常表示失败回退值或 stale fallback++### 什么时候看 `/metrics`,什么时候看别的入口++- 看服务是否整体健康、哪个组件当前不通:先看 `/health` 和 `observer/*`+- 看资源是否真的落库、向量是否真的写进去:看 `ov tui`+- 看某一次具体请求为什么慢、token 花在哪、资源处理卡在哪个阶段:看 `telemetry`+- 看一段时间内请求量、错误率、时延是否持续恶化:看 `/metrics`+## 相关文档- [部署](03-deployment.md) - 服务器设置- [认证](04-authentication.md) - API Key 设置- [操作级 Telemetry 参考](07-operation-telemetry.md) - 请求级结构化追踪- [系统 API](../api/07-system.md) - 系统与 observer 接口参考+- [指标](../concepts/12-metrics.md) - 时序指标与配置+diff --git a/examples/grafana/openviking_demo_dashboard.json b/examples/grafana/openviking_demo_dashboard.jsonnew file mode 100644index 000000000..77da8533e--- /dev/null+++ b/examples/grafana/openviking_demo_dashboard.json@@ -0,0 +1,5117 @@+{+ "annotations": {+ "list": [+ {+ "builtIn": 1,+ "datasource": {+ "type": "grafana",+ "uid": "-- Grafana --"+ },+ "enable": true,+ "hide": true,+ "iconColor": "#00FFE5",+ "name": "Annotations & Alerts",+ "type": "dashboard"+ }+ ]+ },+ "editable": true,+ "fiscalYearStartMonth": 0,+ "graphTooltip": 2,+ "links": [],+ "panels": [+ {+ "collapsed": false,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 0+ },+ "id": 1,+ "panels": [],+ "title": "Executive KPI",+ "type": "row"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#FF3B30",+ "value": 0+ },+ {+ "color": "#FFD60A",+ "value": 0.5+ },+ {+ "color": "#00FFE5",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 5,+ "w": 3,+ "x": 0,+ "y": 1+ },+ "id": 2,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "percentChangeColorMode": "standard",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 12,+ "valueSize": 28+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "openviking_service_readiness",+ "legendFormat": "Service Readiness",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Svc Ready",+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#FF3B30",+ "value": 0+ },+ {+ "color": "#FFD60A",+ "value": 0.5+ },+ {+ "color": "#00FFE5",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 5,+ "w": 3,+ "x": 3,+ "y": 1+ },+ "id": 3,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "percentChangeColorMode": "standard",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 12,+ "valueSize": 28+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "openviking_api_key_manager_readiness",+ "legendFormat": "API Key Manager",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "API Keys",+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#00FFE5",+ "value": 0+ },+ {+ "color": "#00FFE5",+ "value": 1000000+ },+ {+ "color": "#FFD60A",+ "value": 10000000+ },+ {+ "color": "#FF3B30",+ "value": 100000000+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 5,+ "w": 3,+ "x": 6,+ "y": 1+ },+ "id": 4,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "percentChangeColorMode": "standard",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 11,+ "valueSize": 20+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum(rate(openviking_http_requests_total[$__rate_interval]))",+ "legendFormat": "HTTP RPS",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "HTTP RPS",+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#00FFE5",+ "value": 0+ },+ {+ "color": "#FFD60A",+ "value": 1+ },+ {+ "color": "#FF3B30",+ "value": 2+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 5,+ "w": 3,+ "x": 9,+ "y": 1+ },+ "id": 5,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "percentChangeColorMode": "standard",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 12,+ "valueSize": 28+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "histogram_quantile(0.95, sum by (le) (rate(openviking_http_request_duration_seconds_bucket[$__rate_interval])))",+ "legendFormat": "HTTP P95",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "HTTP P95",+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#00FFE5",+ "value": 0+ },+ {+ "color": "#00FFE5",+ "value": 1000000+ },+ {+ "color": "#FFD60A",+ "value": 10000000+ },+ {+ "color": "#FF3B30",+ "value": 100000000+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 5,+ "w": 3,+ "x": 12,+ "y": 1+ },+ "id": 6,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "percentChangeColorMode": "standard",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 12,+ "valueSize": 28+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum(rate(openviking_retrieval_requests_total[$__rate_interval]))",+ "legendFormat": "Retrieval RPS",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Retrieval",+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#00FFE5",+ "value": 0+ },+ {+ "color": "#00FFE5",+ "value": 1000000+ },+ {+ "color": "#FFD60A",+ "value": 10000000+ },+ {+ "color": "#FF3B30",+ "value": 100000000+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 5,+ "w": 3,+ "x": 15,+ "y": 1+ },+ "id": 7,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "percentChangeColorMode": "standard",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 12,+ "valueSize": 28+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum(rate(openviking_resource_stage_total[$__rate_interval]))",+ "legendFormat": "Resource RPS",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Resource",+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#7C3AED",+ "value": 0+ },+ {+ "color": "#7C3AED",+ "value": 1000000+ },+ {+ "color": "#FFD60A",+ "value": 10000000+ },+ {+ "color": "#FF3B30",+ "value": 100000000+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 5,+ "w": 3,+ "x": 18,+ "y": 1+ },+ "id": 8,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "percentChangeColorMode": "standard",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 12,+ "valueSize": 28+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum(rate(openviking_vlm_calls_total[$__rate_interval]))",+ "legendFormat": "VLM RPS",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "VLM RPS",+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#7C3AED",+ "value": 0+ },+ {+ "color": "#7C3AED",+ "value": 1000000+ },+ {+ "color": "#FFD60A",+ "value": 10000000+ },+ {+ "color": "#FF3B30",+ "value": 100000000+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 5,+ "w": 3,+ "x": 21,+ "y": 1+ },+ "id": 9,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "percentChangeColorMode": "standard",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 12,+ "valueSize": 28+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum(rate(openviking_embedding_requests_total[$__rate_interval]))",+ "legendFormat": "Embedding RPS",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Embedding",+ "type": "stat"+ },+ {+ "collapsed": false,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 6+ },+ "id": 10,+ "panels": [],+ "title": "Platform Health",+ "type": "row"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [+ {+ "options": {+ "0": {+ "text": "Down"+ },+ "1": {+ "text": "Ready"+ },+ "0.5": {+ "text": "Warn"+ }+ },+ "type": "value"+ }+ ],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#FF3B30",+ "value": 0+ },+ {+ "color": "#FFD60A",+ "value": 0.5+ },+ {+ "color": "#00FFE5",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 7+ },+ "id": 11,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "percentChangeColorMode": "standard",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 12,+ "valueSize": 28+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "max by (probe) (openviking_storage_readiness{valid=\"1\"}) or on(probe) max by (probe) (openviking_storage_readiness{valid=\"0\"})",+ "legendFormat": "{{probe}} storage",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "max by (probe) (openviking_retrieval_backend_readiness{valid=\"1\"}) or on(probe) max by (probe) (openviking_retrieval_backend_readiness{valid=\"0\"})",+ "legendFormat": "{{probe}} retrieval",+ "range": true,+ "refId": "B"+ },+ {+ "editorMode": "code",+ "expr": "max by (provider) (openviking_model_provider_readiness{valid=\"1\"}) or on(provider) max by (provider) (openviking_model_provider_readiness{valid=\"0\"})",+ "legendFormat": "{{provider}} model",+ "range": true,+ "refId": "C"+ },+ {+ "editorMode": "code",+ "expr": "max by (probe) (openviking_async_system_readiness)",+ "legendFormat": "{{probe}} async",+ "range": true,+ "refId": "D"+ }+ ],+ "title": "Backend Readiness Matrix",+ "transparent": true,+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [+ {+ "options": {+ "0": {+ "text": "Down"+ },+ "1": {+ "text": "Ready"+ },+ "0.5": {+ "text": "Warn"+ }+ },+ "type": "value"+ }+ ],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#FF3B30",+ "value": 0+ },+ {+ "color": "#FFD60A",+ "value": 0.5+ },+ {+ "color": "#00FFE5",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 7+ },+ "id": 12,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "percentChangeColorMode": "standard",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 12,+ "valueSize": 28+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "max(openviking_encryption_component_health{valid=\"1\"}) or max(openviking_encryption_component_health{valid=\"0\"})",+ "legendFormat": "component health",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "max(openviking_encryption_root_key_ready{valid=\"1\"}) or max(openviking_encryption_root_key_ready{valid=\"0\"})",+ "legendFormat": "root key",+ "range": true,+ "refId": "B"+ },+ {+ "editorMode": "code",+ "expr": "max by (provider) (openviking_encryption_kms_provider_ready{valid=\"1\"}) or on(provider) max by (provider) (openviking_encryption_kms_provider_ready{valid=\"0\"})",+ "legendFormat": "{{provider}} kms",+ "range": true,+ "refId": "C"+ }+ ],+ "title": "Core Security & Encryption",+ "transparent": true,+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [+ {+ "options": {+ "0": {+ "text": "Down"+ },+ "1": {+ "text": "Ready"+ },+ "0.5": {+ "text": "Warn"+ }+ },+ "type": "value"+ }+ ],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#FF3B30",+ "value": 0+ },+ {+ "color": "#FFD60A",+ "value": 0.5+ },+ {+ "color": "#00FFE5",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": [+ {+ "matcher": {+ "id": "byRegexp",+ "options": ".* errors$|^observer unhealthy$|^observer errors$"+ },+ "properties": [+ {+ "id": "mappings",+ "value": []+ },+ {+ "id": "thresholds",+ "value": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#00FFE5",+ "value": 0+ },+ {+ "color": "#FFD60A",+ "value": 1+ },+ {+ "color": "#FF3B30",+ "value": 5+ }+ ]+ }+ }+ ]+ },+ {+ "matcher": {+ "id": "byName",+ "options": "observer total"+ },+ "properties": [+ {+ "id": "mappings",+ "value": []+ },+ {+ "id": "color",+ "value": {+ "fixedColor": "#00FFE5",+ "mode": "fixed"+ }+ }+ ]+ }+ ]+ },+ "gridPos": {+ "h": 4,+ "w": 12,+ "x": 0,+ "y": 15+ },+ "id": 13,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "percentChangeColorMode": "standard",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 11,+ "valueSize": 20+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "max(openviking_observer_components_total{valid=\"1\"}) or max(openviking_observer_components_total{valid=\"0\"})",+ "legendFormat": "observer total",+ "range": true,+ "refId": "C"+ },+ {+ "editorMode": "code",+ "expr": "max by (component) (openviking_component_health{valid=\"1\"}) or on(component) max by (component) (openviking_component_health{valid=\"0\"})",+ "legendFormat": "{{component}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Component Health Signals",+ "transparent": true,+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#00FFE5",+ "value": 0+ },+ {+ "color": "#FFD60A",+ "value": 100+ },+ {+ "color": "#FF3B30",+ "value": 1000+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 4,+ "w": 12,+ "x": 12,+ "y": 15+ },+ "id": 60,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "percentChangeColorMode": "standard",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 11,+ "valueSize": 20+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "max by (component) (openviking_component_errors{valid=\"1\"}) or on(component) max by (component) (openviking_component_errors{valid=\"0\"})",+ "legendFormat": "{{component}}",+ "range": true,+ "refId": "B"+ },+ {+ "editorMode": "code",+ "expr": "max(openviking_observer_components_unhealthy{valid=\"1\"}) or max(openviking_observer_components_unhealthy{valid=\"0\"})",+ "legendFormat": "observer unhealthy",+ "range": true,+ "refId": "D"+ },+ {+ "editorMode": "code",+ "expr": "max(openviking_observer_components_with_errors{valid=\"1\"}) or max(openviking_observer_components_with_errors{valid=\"0\"})",+ "legendFormat": "observer",+ "range": true,+ "refId": "E"+ }+ ],+ "title": "Component Error Signals",+ "transparent": true,+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [+ {+ "options": {+ "0": {+ "text": "Down"+ },+ "1": {+ "text": "Ready"+ },+ "0.5": {+ "text": "Warn"+ }+ },+ "type": "value"+ }+ ],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#FF3B30",+ "value": 0+ },+ {+ "color": "#FFD60A",+ "value": 0.5+ },+ {+ "color": "#00FFE5",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 4,+ "w": 12,+ "x": 0,+ "y": 19+ },+ "id": 14,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "percentChangeColorMode": "standard",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 12,+ "valueSize": 28+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "max by (collection) (openviking_vikingdb_collection_health{valid=\"1\"}) or on(collection) max by (collection) (openviking_vikingdb_collection_health{valid=\"0\"})",+ "legendFormat": "{{collection}} health",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "VikingDB Health",+ "transparent": true,+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#00FFE5",+ "value": 0+ },+ {+ "color": "#00FFE5",+ "value": 1000+ },+ {+ "color": "#FFD60A",+ "value": 100000+ },+ {+ "color": "#FF3B30",+ "value": 1000000+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 4,+ "w": 12,+ "x": 12,+ "y": 19+ },+ "id": 59,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "percentChangeColorMode": "standard",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 12,+ "valueSize": 28+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "max by (collection) (openviking_vikingdb_collection_vectors{valid=\"1\"}) or on(collection) max by (collection) (openviking_vikingdb_collection_vectors{valid=\"0\"})",+ "legendFormat": "{{collection}} vectors",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Vector Count",+ "transparent": true,+ "type": "stat"+ },+ {+ "collapsed": false,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 23+ },+ "id": 15,+ "panels": [],+ "title": "HTTP & API",+ "type": "row"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "fixedColor": "#00FFE5",+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 24+ },+ "id": 16,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (route, status) (rate(openviking_http_requests_total[$__rate_interval]))",+ "legendFormat": "{{route}} / {{status}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "HTTP Requests / Overall",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 24+ },+ "id": 17,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (account_id) (rate(openviking_http_requests_total[$__rate_interval]))",+ "legendFormat": "{{account_id}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "HTTP Requests / By Account",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#00FFE5",+ "value": 0+ },+ {+ "color": "#FFD60A",+ "value": 1+ },+ {+ "color": "#FF3B30",+ "value": 2+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 32+ },+ "id": 18,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "histogram_quantile(0.95, sum by (le, route) (rate(openviking_http_request_duration_seconds_bucket[$__rate_interval])))",+ "legendFormat": "{{route}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "HTTP Request Duration / Overall",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#00FFE5",+ "value": 0+ },+ {+ "color": "#FFD60A",+ "value": 1+ },+ {+ "color": "#FF3B30",+ "value": 2+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 32+ },+ "id": 19,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "histogram_quantile(0.95, sum by (le, account_id) (rate(openviking_http_request_duration_seconds_bucket[$__rate_interval])))",+ "legendFormat": "{{account_id}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "HTTP Request Duration / By Account",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 40+ },+ "id": 20,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "max by (route) (openviking_http_inflight_requests)",+ "legendFormat": "{{route}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "HTTP Inflight Requests / Overall",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 40+ },+ "id": 21,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (account_id) (openviking_http_inflight_requests)",+ "legendFormat": "{{account_id}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "HTTP Inflight Requests / By Account",+ "type": "timeseries"+ },+ {+ "collapsed": false,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 48+ },+ "id": 22,+ "panels": [],+ "title": "Retrieval & Search",+ "type": "row"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 49+ },+ "id": 23,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (context_type) (rate(openviking_retrieval_requests_total[$__rate_interval]))",+ "legendFormat": "requests {{context_type}}",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "sum by (context_type) (rate(openviking_retrieval_results_total[$__rate_interval]))",+ "legendFormat": "results {{context_type}}",+ "range": true,+ "refId": "B"+ }+ ],+ "title": "Retrieval Throughput / Overall",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 49+ },+ "id": 24,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (account_id) (rate(openviking_retrieval_requests_total[$__rate_interval]))",+ "legendFormat": "requests {{account_id}}",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "sum by (account_id) (rate(openviking_retrieval_results_total[$__rate_interval]))",+ "legendFormat": "results {{account_id}}",+ "range": true,+ "refId": "B"+ }+ ],+ "title": "Retrieval Throughput / By Account",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 57+ },+ "id": 25,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "histogram_quantile(0.95, sum by (le, context_type) (rate(openviking_retrieval_latency_seconds_bucket[$__rate_interval])))",+ "legendFormat": "{{context_type}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Retrieval Latency / Overall",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 57+ },+ "id": 26,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "histogram_quantile(0.95, sum by (le, account_id) (rate(openviking_retrieval_latency_seconds_bucket[$__rate_interval])))",+ "legendFormat": "{{account_id}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Retrieval Latency / By Account",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 65+ },+ "id": 27,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (operation, status) (rate(openviking_operation_requests_total[$__rate_interval]))",+ "legendFormat": "{{operation}} / {{status}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Operation Requests / Overall",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 65+ },+ "id": 28,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (account_id) (rate(openviking_operation_requests_total[$__rate_interval]))",+ "legendFormat": "{{account_id}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Operation Requests / By Account",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 73+ },+ "id": 29,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "histogram_quantile(0.95, sum by (le, operation) (rate(openviking_operation_duration_seconds_bucket[$__rate_interval])))",+ "legendFormat": "{{operation}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Operation Duration / Overall",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 73+ },+ "id": 30,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "histogram_quantile(0.95, sum by (le, account_id) (rate(openviking_operation_duration_seconds_bucket[$__rate_interval])))",+ "legendFormat": "{{account_id}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Operation Duration / By Account",+ "type": "timeseries"+ },+ {+ "collapsed": false,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 81+ },+ "id": 31,+ "panels": [],+ "title": "Resource Pipeline",+ "type": "row"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 82+ },+ "id": 32,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (stage, status) (rate(openviking_resource_stage_total[$__rate_interval]))",+ "legendFormat": "{{stage}} / {{status}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Resource Stage / Overall",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 82+ },+ "id": 33,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (account_id) (rate(openviking_resource_stage_total[$__rate_interval]))",+ "legendFormat": "{{account_id}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Resource Stage / By Account",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "fixedColor": "#00FFE5",+ "mode": "fixed"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 90+ },+ "id": 34,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "histogram_quantile(0.95, sum by (le, stage) (rate(openviking_resource_stage_duration_seconds_bucket[$__rate_interval])))",+ "legendFormat": "{{stage}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Resource Stage Duration / Overall",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 90+ },+ "id": 35,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "histogram_quantile(0.95, sum by (le, account_id) (rate(openviking_resource_stage_duration_seconds_bucket[$__rate_interval])))",+ "legendFormat": "{{account_id}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Resource Stage Duration / By Account",+ "type": "timeseries"+ },+ {+ "collapsed": false,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 98+ },+ "id": 36,+ "panels": [],+ "title": "Model & AI Runtime",+ "type": "row"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#FF3B30",+ "value": 0+ },+ {+ "color": "#FFD60A",+ "value": 0.5+ },+ {+ "color": "#00FFE5",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 8,+ "x": 0,+ "y": 99+ },+ "id": 37,+ "options": {+ "displayMode": "gradient",+ "legend": {+ "calcs": [],+ "displayMode": "list",+ "placement": "bottom",+ "showLegend": false+ },+ "maxVizHeight": 300,+ "minVizHeight": 16,+ "minVizWidth": 8,+ "namePlacement": "auto",+ "orientation": "horizontal",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showUnfilled": true,+ "sizing": "auto",+ "valueMode": "color"+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "max by (model_type) (openviking_model_usage_available{valid=\"1\"})",+ "legendFormat": "{{model_type}} ",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Model Availability",+ "type": "bargauge"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 16,+ "x": 8,+ "y": 99+ },+ "id": 38,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (model_type, provider) (rate(openviking_model_calls_total[$__rate_interval]))",+ "legendFormat": "calls {{model_type}} / {{provider}}",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "sum by (model_type, token_type) (rate(openviking_model_tokens_total[$__rate_interval]))",+ "legendFormat": "tokens {{model_type}} / {{token_type}}",+ "range": true,+ "refId": "B"+ }+ ],+ "title": "Model Calls & Tokens",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 107+ },+ "id": 39,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (provider, model_name) (rate(openviking_vlm_calls_total[$__rate_interval]))",+ "legendFormat": "{{provider}} / {{model_name}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "VLM Calls / Overall",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 107+ },+ "id": 40,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (account_id) (rate(openviking_vlm_calls_total[$__rate_interval]))",+ "legendFormat": "{{account_id}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "VLM Calls / By Account",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 115+ },+ "id": 41,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (provider, model_name) (rate(openviking_vlm_tokens_input_total[$__rate_interval]))",+ "legendFormat": "input {{provider}} / {{model_name}}",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "sum by (provider, model_name) (rate(openviking_vlm_tokens_output_total[$__rate_interval]))",+ "legendFormat": "output {{provider}} / {{model_name}}",+ "range": true,+ "refId": "B"+ },+ {+ "editorMode": "code",+ "expr": "sum by (provider, model_name) (rate(openviking_vlm_tokens_total[$__rate_interval]))",+ "legendFormat": "total {{provider}} / {{model_name}}",+ "range": true,+ "refId": "C"+ }+ ],+ "title": "VLM Tokens / Overall",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 115+ },+ "id": 42,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (account_id) (rate(openviking_vlm_tokens_total[$__rate_interval]))",+ "legendFormat": "total {{account_id}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "VLM Tokens / By Account",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 123+ },+ "id": 43,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "histogram_quantile(0.95, sum by (le, provider, model_name) (rate(openviking_vlm_call_duration_seconds_bucket[$__rate_interval])))",+ "legendFormat": "{{provider}} / {{model_name}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "VLM Call Duration / Overall",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 123+ },+ "id": 44,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "histogram_quantile(0.95, sum by (le, account_id) (rate(openviking_vlm_call_duration_seconds_bucket[$__rate_interval])))",+ "legendFormat": "{{account_id}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "VLM Call Duration / By Account",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 131+ },+ "id": 45,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (status) (rate(openviking_embedding_requests_total[$__rate_interval]))",+ "legendFormat": "{{status}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Embedding Requests / Overall",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 131+ },+ "id": 46,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (account_id) (rate(openviking_embedding_requests_total[$__rate_interval]))",+ "legendFormat": "{{account_id}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Embedding Requests / By Account",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 139+ },+ "id": 47,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "histogram_quantile(0.95, sum by (le, status) (rate(openviking_embedding_latency_seconds_bucket[$__rate_interval])))",+ "legendFormat": "{{status}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Embedding Latency / Overall",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 139+ },+ "id": 48,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "histogram_quantile(0.95, sum by (le, account_id) (rate(openviking_embedding_latency_seconds_bucket[$__rate_interval])))",+ "legendFormat": "{{account_id}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Embedding Latency / By Account",+ "type": "timeseries"+ },+ {+ "collapsed": false,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 147+ },+ "id": 49,+ "panels": [],+ "title": "Queue, Task & Lock",+ "type": "row"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "fixedColor": "#00FFE5",+ "mode": "fixed"+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 148+ },+ "id": 50,+ "options": {+ "displayMode": "gradient",+ "legend": {+ "calcs": [],+ "displayMode": "list",+ "placement": "bottom",+ "showLegend": false+ },+ "maxVizHeight": 300,+ "minVizHeight": 16,+ "minVizWidth": 8,+ "namePlacement": "auto",+ "orientation": "horizontal",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showUnfilled": true,+ "sizing": "auto",+ "valueMode": "color"+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "max by (queue) (openviking_queue_pending)",+ "legendFormat": "pending {{queue}}",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "max by (queue) (openviking_queue_in_progress)",+ "legendFormat": "in_progress {{queue}}",+ "range": true,+ "refId": "B"+ }+ ],+ "title": "Queue State",+ "type": "bargauge"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "fixedColor": "#00FFE5",+ "mode": "fixed"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "barWidthFactor": 0.6,+ "drawStyle": "line",+ "fillOpacity": 22,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 3,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "showValues": false,+ "spanNulls": false,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 148+ },+ "id": 51,+ "options": {+ "legend": {+ "calcs": [],+ "displayMode": "table",+ "placement": "bottom",+ "showLegend": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "sum by (queue) (rate(openviking_queue_processed_total[$__rate_interval]))",+ "legendFormat": "{{queue}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Queue Processed",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#00FFE5",+ "value": 0+ },+ {+ "color": "#00FFE5",+ "value": 1000000+ },+ {+ "color": "#FFD60A",+ "value": 10000000+ },+ {+ "color": "#FF3B30",+ "value": 100000000+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 6,+ "w": 24,+ "x": 0,+ "y": 156+ },+ "id": 52,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "percentChangeColorMode": "standard",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "textMode": "value_and_name",+ "wideLayout": true+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "openviking_lock_active",+ "legendFormat": "active",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "openviking_lock_waiting",+ "legendFormat": "waiting",+ "range": true,+ "refId": "B"+ },+ {+ "editorMode": "code",+ "expr": "openviking_lock_stale",+ "legendFormat": "stale",+ "range": true,+ "refId": "C"+ }+ ],+ "title": "Lock State",+ "type": "stat"+ },+ {+ "collapsed": false,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 162+ },+ "id": 53,+ "panels": [],+ "title": "Cache & Encryption",+ "type": "row"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#FF3B30",+ "value": 0+ },+ {+ "color": "#FFD60A",+ "value": 0.5+ },+ {+ "color": "#00FFE5",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 24,+ "x": 0,+ "y": 163+ },+ "id": 54,+ "options": {+ "displayMode": "gradient",+ "legend": {+ "calcs": [],+ "displayMode": "list",+ "placement": "bottom",+ "showLegend": false+ },+ "maxVizHeight": 300,+ "minVizHeight": 16,+ "minVizWidth": 8,+ "namePlacement": "auto",+ "orientation": "horizontal",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showUnfilled": true,+ "sizing": "auto",+ "valueMode": "color"+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "max by (valid) (openviking_encryption_component_health)",+ "legendFormat": "component {{valid}}",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "max by (valid) (openviking_encryption_root_key_ready)",+ "legendFormat": "root_key {{valid}}",+ "range": true,+ "refId": "B"+ },+ {+ "editorMode": "code",+ "expr": "max by (provider, valid) (openviking_encryption_kms_provider_ready)",+ "legendFormat": "kms {{provider}} / {{valid}}",+ "range": true,+ "refId": "C"+ }+ ],+ "title": "Encryption Readiness",+ "type": "bargauge"+ },+ {+ "collapsed": false,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 171+ },+ "id": 55,+ "panels": [],+ "title": "Tenant Drilldown",+ "type": "row"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "fixedColor": "#00FFE5",+ "mode": "fixed"+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "ops"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 24,+ "x": 0,+ "y": 172+ },+ "id": 56,+ "options": {+ "displayMode": "gradient",+ "legend": {+ "calcs": [],+ "displayMode": "list",+ "placement": "bottom",+ "showLegend": false+ },+ "maxVizHeight": 300,+ "minVizHeight": 16,+ "minVizWidth": 8,+ "namePlacement": "auto",+ "orientation": "horizontal",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showUnfilled": true,+ "sizing": "auto",+ "valueMode": "color"+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "topk(10, sum by (account_id) (rate(openviking_http_requests_total[$__rate_interval])))",+ "legendFormat": "http {{account_id}}",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "topk(10, sum by (account_id) (rate(openviking_retrieval_requests_total[$__rate_interval])))",+ "legendFormat": "retrieval {{account_id}}",+ "range": true,+ "refId": "B"+ },+ {+ "editorMode": "code",+ "expr": "topk(10, sum by (account_id) (rate(openviking_operation_requests_total[$__rate_interval])))",+ "legendFormat": "operation {{account_id}}",+ "range": true,+ "refId": "C"+ }+ ],+ "title": "Tenant Traffic Hotspots",+ "type": "bargauge"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "fixedColor": "#7C3AED",+ "mode": "fixed"+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#73BF69",+ "value": 0+ },+ {+ "color": "#FADE2A",+ "value": 0.3+ },+ {+ "color": "#F2495C",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 24,+ "x": 0,+ "y": 180+ },+ "id": 57,+ "options": {+ "displayMode": "gradient",+ "legend": {+ "calcs": [],+ "displayMode": "list",+ "placement": "bottom",+ "showLegend": false+ },+ "maxVizHeight": 300,+ "minVizHeight": 16,+ "minVizWidth": 8,+ "namePlacement": "auto",+ "orientation": "horizontal",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showUnfilled": true,+ "sizing": "auto",+ "valueMode": "color"+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "topk(10, sum by (account_id) (rate(openviking_vlm_calls_total[$__rate_interval])))",+ "legendFormat": "vlm calls {{account_id}}",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "topk(10, sum by (account_id) (rate(openviking_vlm_tokens_total[$__rate_interval])))",+ "legendFormat": "vlm tokens {{account_id}}",+ "range": true,+ "refId": "B"+ },+ {+ "editorMode": "code",+ "expr": "topk(10, sum by (account_id) (rate(openviking_embedding_requests_total[$__rate_interval])))",+ "legendFormat": "embedding req {{account_id}}",+ "range": true,+ "refId": "C"+ },+ {+ "editorMode": "code",+ "expr": "topk(10, histogram_quantile(0.95, sum by (le, account_id) (rate(openviking_embedding_latency_seconds_bucket[$__rate_interval]))))",+ "legendFormat": "embedding p95 {{account_id}}",+ "range": true,+ "refId": "D"+ }+ ],+ "title": "Tenant AI Hotspots",+ "type": "bargauge"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "fixedColor": "#00FFE5",+ "mode": "fixed"+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#F2495C",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 0.5+ },+ {+ "color": "#73BF69",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 24,+ "x": 0,+ "y": 188+ },+ "id": 58,+ "options": {+ "displayMode": "gradient",+ "legend": {+ "calcs": [],+ "displayMode": "list",+ "placement": "bottom",+ "showLegend": false+ },+ "maxVizHeight": 300,+ "minVizHeight": 16,+ "minVizWidth": 8,+ "namePlacement": "auto",+ "orientation": "horizontal",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showUnfilled": true,+ "sizing": "auto",+ "valueMode": "color"+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "topk(10, sum by (account_id) (rate(openviking_resource_stage_total[$__rate_interval])))",+ "legendFormat": "resource {{account_id}}",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "topk(10, sum by (account_id) (openviking_http_inflight_requests))",+ "legendFormat": "inflight {{account_id}}",+ "range": true,+ "refId": "B"+ }+ ],+ "title": "Tenant Pipeline Hotspots",+ "type": "bargauge"+ }+ ],+ "preload": false,+ "refresh": "30s",+ "schemaVersion": 42,+ "tags": [+ "openviking",+ "metrics",+ "prometheus",+ "demo",+ "hud",+ "ai-cockpit",+ "deep-space"+ ],+ "templating": {+ "list": [+ {+ "current": {+ "text": "Prometheus",+ "value": "prometheus"+ },+ "includeAll": false,+ "label": "datasource",+ "name": "DS_PROMETHEUS",+ "options": [],+ "query": "prometheus",+ "refresh": 1,+ "regex": "",+ "type": "datasource"+ },+ {+ "current": {+ "text": "All",+ "value": [+ "$__all"+ ]+ },+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "definition": "label_values(openviking_http_requests_total, account_id)",+ "includeAll": true,+ "label": "account_id",+ "multi": true,+ "name": "account_id",+ "options": [],+ "query": {+ "query": "label_values(openviking_http_requests_total, account_id)",+ "refId": "account_id-var"+ },+ "refresh": 2,+ "regex": "",+ "regexApplyTo": "value",+ "sort": 1,+ "type": "query"+ }+ ]+ },+ "time": {+ "from": "2026-04-10T02:22:10.488Z",+ "to": "2026-04-10T02:47:21.858Z"+ },+ "timepicker": {},+ "timezone": "browser",+ "title": "Openviking-Demo",+ "uid": "openviking-demo",+ "version": 39,+ "weekStart": ""+}\ No newline at end of filediff --git a/examples/ov.conf.example b/examples/ov.conf.exampleindex 4769db20e..73bc41ee2 100644--- a/examples/ov.conf.example+++ b/examples/ov.conf.example@@ -3,7 +3,22 @@"host": "0.0.0.0","port": 1933,"root_api_key": null,- "cors_origins": ["*"]+ "cors_origins": ["*"],+ "telemetry": {+ "prometheus": {+ "enabled": false+ }+ },+ "metrics": {+ "enabled": true,+ "account_dimension": {+ "enabled": true,+ "max_active_accounts": 100,+ "metric_allowlist": [+ "openviking_*"+ ]+ }+ }},"storage": {"workspace": "./data",diff --git a/openviking/crypto/encryptor.py b/openviking/crypto/encryptor.pyindex cf99a78e3..9af1b95e7 100644--- a/openviking/crypto/encryptor.py+++ b/openviking/crypto/encryptor.py@@ -11,6 +11,7 @@ Implements Envelope Encryption pattern:import secretsimport struct+import timefrom typing import Tuplefrom openviking.crypto.exceptions import (@@ -78,24 +79,45 @@ class FileEncryptor:Returns:Encrypted content (Envelope format)"""- # 1. Generate random File Key- file_key = secrets.token_bytes(32)-- # 2. Encrypt file content- data_iv = secrets.token_bytes(12)- encrypted_content = await self._aes_gcm_encrypt(file_key, data_iv, plaintext)-- # 3. Encrypt File Key (all providers now return (encrypted_key, iv)- encrypted_file_key, key_iv = await self.provider.encrypt_file_key(file_key, account_id)-- # 4. Build Envelope- return self._build_envelope(- self._provider_type,- encrypted_file_key,- key_iv,- data_iv,- encrypted_content,- )+ start = time.perf_counter()+ # Metrics must be best-effort: encryption must succeed/fail solely based on crypto logic,+ # not on observability availability. Therefore all metrics emissions are wrapped in try/except.+ try:+ from openviking.metrics.datasources.encryption import EncryptionEventDataSource++ EncryptionEventDataSource.record_payload_size(+ operation="encrypt", size_bytes=len(plaintext)+ )+ EncryptionEventDataSource.record_bytes(operation="encrypt", size_bytes=len(plaintext))+ except Exception:+ pass++ status = "ok"+ try:+ file_key = secrets.token_bytes(32)+ data_iv = secrets.token_bytes(12)+ encrypted_content = await self._aes_gcm_encrypt(file_key, data_iv, plaintext)+ encrypted_file_key, key_iv = await self.provider.encrypt_file_key(file_key, account_id)+ return self._build_envelope(+ self._provider_type,+ encrypted_file_key,+ key_iv,+ data_iv,+ encrypted_content,+ )+ except Exception:+ status = "error"+ raise+ finally:+ elapsed = time.perf_counter() - start+ try:+ from openviking.metrics.datasources.encryption import EncryptionEventDataSource++ EncryptionEventDataSource.record_operation(+ operation="encrypt", status=status, duration_seconds=elapsed+ )+ except Exception:+ passasync def decrypt(self, account_id: str, ciphertext: bytes) -> bytes:"""@@ -130,17 +152,50 @@ class FileEncryptor:except Exception as e:raise CorruptedCiphertextError(f"Failed to parse envelope: {e}")+ start = time.perf_counter()+ status = "ok"try:- # 3. Decrypt File Key (all providers now use (encrypted_key, iv, account_id)file_key = await self.provider.decrypt_file_key(encrypted_file_key, key_iv, account_id)except Exception as e:+ status = "error"raise KeyMismatchError(f"Failed to decrypt file key: {e}")try:- # 4. Decrypt file content- return await self._aes_gcm_decrypt(file_key, data_iv, encrypted_content)+ plaintext = await self._aes_gcm_decrypt(file_key, data_iv, encrypted_content)+ try:+ from openviking.metrics.datasources.encryption import EncryptionEventDataSource++ EncryptionEventDataSource.record_payload_size(+ operation="decrypt", size_bytes=len(ciphertext)+ )+ EncryptionEventDataSource.record_bytes(+ operation="decrypt", size_bytes=len(ciphertext)+ )+ except Exception:+ pass+ return plaintext+ except AuthenticationFailedError as e:+ status = "error"+ try:+ from openviking.metrics.datasources.encryption import EncryptionEventDataSource++ EncryptionEventDataSource.record_auth_failed()+ except Exception:+ pass+ raise AuthenticationFailedError(f"Authentication failed: {e}")except Exception as e:+ status = "error"raise AuthenticationFailedError(f"Authentication failed: {e}")+ finally:+ elapsed = time.perf_counter() - start+ try:+ from openviking.metrics.datasources.encryption import EncryptionEventDataSource++ EncryptionEventDataSource.record_operation(+ operation="decrypt", status=status, duration_seconds=elapsed+ )+ except Exception:+ passdef _build_envelope(self,diff --git a/openviking/crypto/providers.py b/openviking/crypto/providers.pyindex 7c0e8daed..beb39c147 100644--- a/openviking/crypto/providers.py+++ b/openviking/crypto/providers.py@@ -13,6 +13,7 @@ import abcimport asyncioimport osimport secrets+import timefrom abc import ABCfrom pathlib import Pathfrom typing import Any, Dict, Optional, Tuple@@ -73,6 +74,8 @@ class RootKeyProvider(ABC):Returns:Derived key"""+ start = time.perf_counter()+ status = "ok"try:from cryptography.hazmat.primitives import hashesfrom cryptography.hazmat.primitives.kdf.hkdf import HKDF@@ -85,7 +88,24 @@ class RootKeyProvider(ABC):)return hkdf.derive(root_key)except ImportError:+ status = "error"raise ConfigError("cryptography library is required for encryption")+ except Exception:+ status = "error"+ raise+ finally:+ elapsed = time.perf_counter() - start+ try:+ from openviking.metrics.datasources.encryption import EncryptionEventDataSource++ EncryptionEventDataSource.record_key_derivation(+ status=status, duration_seconds=elapsed+ )+ except Exception:+ logger.debug(+ "Failed to record encryption key derivation metrics",+ exc_info=True,+ )class BaseProvider(RootKeyProvider):@@ -160,9 +180,45 @@ class LocalFileProvider(BaseProvider):async def get_root_key(self) -> bytes:"""Get Root Key."""- if self._root_key is None:+ if self._root_key is not None:+ try:+ from openviking.metrics.datasources.encryption import EncryptionEventDataSource++ EncryptionEventDataSource.record_key_cache_hit(provider="local")+ except Exception:+ pass+ return self._root_key++ try:+ from openviking.metrics.datasources.encryption import EncryptionEventDataSource++ EncryptionEventDataSource.record_key_cache_miss(provider="local")+ except Exception:+ pass++ start = time.perf_counter()+ status = "ok"+ try:self._root_key = await self._load_or_create_root_key()- return self._root_key+ return self._root_key+ except Exception:+ status = "error"+ raise+ finally:+ elapsed = time.perf_counter() - start+ try:+ from openviking.metrics.datasources.encryption import EncryptionEventDataSource++ EncryptionEventDataSource.record_key_load(+ status=status, provider="local", duration_seconds=elapsed+ )+ EncryptionEventDataSource.record_key_version_usage(key_version="local")+ except Exception:+ logger.debug(+ "Failed to record encryption key metrics for provider=%s",+ "local",+ exc_info=True,+ )async def _load_or_create_root_key(self) -> bytes:"""Load or create Root Key."""@@ -232,7 +288,7 @@ class VaultProvider(BaseProvider):self.kv_version = kv_versionself.root_key_name = root_key_nameself.encrypted_root_key_key = encrypted_root_key_key- self._client = None+ self._client: Any = Noneself._root_key: Optional[bytes] = Noneasync def _get_client(self):@@ -424,7 +480,47 @@ class VaultProvider(BaseProvider):Returns:Root key"""- return await self._get_or_create_root_key()+ if self._root_key is not None:+ try:+ from openviking.metrics.datasources.encryption import EncryptionEventDataSource++ EncryptionEventDataSource.record_key_cache_hit(provider="vault")+ except Exception:+ pass+ return self._root_key++ try:+ from openviking.metrics.datasources.encryption import EncryptionEventDataSource++ EncryptionEventDataSource.record_key_cache_miss(provider="vault")+ except Exception:+ pass++ start = time.perf_counter()+ status = "ok"+ try:+ self._root_key = await self._get_or_create_root_key()+ return self._root_key+ except Exception:+ status = "error"+ raise+ finally:+ elapsed = time.perf_counter() - start+ try:+ from openviking.metrics.datasources.encryption import EncryptionEventDataSource++ EncryptionEventDataSource.record_key_load(+ status=status, provider="vault", duration_seconds=elapsed+ )+ EncryptionEventDataSource.record_key_version_usage(+ key_version=str(self.root_key_name)+ )+ except Exception:+ logger.debug(+ "Failed to record encryption key metrics for provider=%s",+ "vault",+ exc_info=True,+ )async def derive_account_key(self, account_id: str) -> bytes:"""@@ -481,7 +577,7 @@ class VolcengineKMSProvider(BaseProvider):self.key_file = Path(key_file).expanduser()else:self.key_file = Path.home() / ".openviking" / self.ROOT_KEY_FILENAME- self._kms_client = None+ self._kms_client: Any = Noneself._root_key: Optional[bytes] = Noneasync def _get_kms_client(self):@@ -651,7 +747,45 @@ class VolcengineKMSProvider(BaseProvider):Returns:Root key"""- return await self._get_or_create_root_key()+ if self._root_key is not None:+ try:+ from openviking.metrics.datasources.encryption import EncryptionEventDataSource++ EncryptionEventDataSource.record_key_cache_hit(provider="volcengine_kms")+ except Exception:+ pass+ return self._root_key++ try:+ from openviking.metrics.datasources.encryption import EncryptionEventDataSource++ EncryptionEventDataSource.record_key_cache_miss(provider="volcengine_kms")+ except Exception:+ pass++ start = time.perf_counter()+ status = "ok"+ try:+ self._root_key = await self._get_or_create_root_key()+ return self._root_key+ except Exception:+ status = "error"+ raise+ finally:+ elapsed = time.perf_counter() - start+ try:+ from openviking.metrics.datasources.encryption import EncryptionEventDataSource++ EncryptionEventDataSource.record_key_load(+ status=status, provider="volcengine_kms", duration_seconds=elapsed+ )+ EncryptionEventDataSource.record_key_version_usage(key_version=str(self.key_id))+ except Exception:+ logger.debug(+ "Failed to record encryption key metrics for provider=%s",+ "volcengine_kms",+ exc_info=True,+ )async def derive_account_key(self, account_id: str) -> bytes:"""diff --git a/openviking/metrics/__init__.py b/openviking/metrics/__init__.pynew file mode 100644index 000000000..39f5b6a89--- /dev/null+++ b/openviking/metrics/__init__.py@@ -0,0 +1,54 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+OpenViking metrics subsystem.++High-level architecture:+- DataSource: emits events or exposes read APIs; must not write MetricRegistry directly.+- Collector: the only writer to MetricRegistry; may be Event/State/Probe/DomainStats.+- CollectorManager: orchestrates refresh before each Prometheus scrape with TTL + SWR.+- PrometheusExporter: refreshes collectors (best-effort) and renders registry to text format.+"""++from .core.base import MetricCollector, MetricDataSource, MetricExporter+from .core.registry import MetricRegistry+from .exporters.prometheus import PrometheusExporter+++def get_metrics_registry():+ """Lazily import and return the process-global metrics registry."""+ from .global_api import get_metrics_registry as _get_metrics_registry++ return _get_metrics_registry()+++def init_metrics_from_server_config(config, *, registry=None, app=None, service=None):+ """Lazily initialize metrics from server configuration."""+ from .global_api import init_metrics_from_server_config as _init_metrics_from_server_config++ return _init_metrics_from_server_config(+ config,+ registry=registry,+ app=app,+ service=service,+ )+++def shutdown_metrics(*, app=None):+ """Lazily clear global metrics objects and detach exporter from the application."""+ from .global_api import shutdown_metrics as _shutdown_metrics++ return _shutdown_metrics(app=app)+++__all__ = [+ "MetricDataSource",+ "MetricCollector",+ "MetricExporter",+ "MetricRegistry",+ "PrometheusExporter",+ "get_metrics_registry",+ "init_metrics_from_server_config",+ "shutdown_metrics",+]diff --git a/openviking/metrics/account_context.py b/openviking/metrics/account_context.pynew file mode 100644index 000000000..81d0b2591--- /dev/null+++ b/openviking/metrics/account_context.py@@ -0,0 +1,80 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Request-scoped account context for metrics label injection.++This module keeps metrics-specific request context separate from business-layer request+objects so collectors can resolve `account_id` without depending on FastAPI internals.+"""++from __future__ import annotations++import contextvars+from dataclasses import dataclass+++@dataclass(frozen=True, slots=True)+class MetricAccountContext:+ """+ Store request-scoped account information used during metrics label resolution.++ The context currently keeps the HTTP-derived account identifier that collectors and+ account-dimension policy code can consult without reaching back into framework-specific+ request objects.+ """++ http_account_id: str | None = None+++_CURRENT_METRIC_ACCOUNT_CONTEXT: contextvars.ContextVar[MetricAccountContext | None] = (+ contextvars.ContextVar(+ "openviking_metric_account_context",+ default=None,+ )+)+++def get_metric_account_context() -> MetricAccountContext:+ """+ Return the metrics account context bound to the current execution flow.++ When no context has been bound yet, the function returns an empty `MetricAccountContext`+ instance so callers can treat the result as a stable object instead of handling `None`.+ """+ context = _CURRENT_METRIC_ACCOUNT_CONTEXT.get()+ if context is None:+ return MetricAccountContext()+ return context+++def bind_metric_account_context(+ *, account_id: str | None+) -> contextvars.Token[MetricAccountContext | None]:+ """+ Bind a new metrics account context and return the token needed to restore the prior one.++ This helper is intended for request or worker entry points that need scoped lifetime+ management through `contextvars.Token`, typically via `try/finally`.+ """+ return _CURRENT_METRIC_ACCOUNT_CONTEXT.set(MetricAccountContext(http_account_id=account_id))+++def set_metric_account_context(*, account_id: str | None) -> None:+ """+ Overwrite the current metrics account context for the active execution flow.++ Unlike `bind_metric_account_context`, this helper does not provide a restoration token and+ is therefore best suited to one-way updates where the caller owns the full scope lifetime.+ """+ _CURRENT_METRIC_ACCOUNT_CONTEXT.set(MetricAccountContext(http_account_id=account_id))+++def reset_metric_account_context(token: contextvars.Token[MetricAccountContext | None]) -> None:+ """+ Restore the previous metrics account context captured by `bind_metric_account_context`.++ Callers should use the exact token returned by the matching bind operation so nested context+ updates unwind in the correct order.+ """+ _CURRENT_METRIC_ACCOUNT_CONTEXT.reset(token)diff --git a/openviking/metrics/account_dimension.py b/openviking/metrics/account_dimension.pynew file mode 100644index 000000000..3bae0fb84--- /dev/null+++ b/openviking/metrics/account_dimension.py@@ -0,0 +1,263 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Account-dimension policy and runtime configuration for metrics.++This module keeps account label resolution outside of the registry so label injection remains a+collector-side concern while still being configurable process-wide.+"""++from __future__ import annotations++import threading+from dataclasses import dataclass++from .account_context import get_metric_account_context++UNKNOWN_ACCOUNT_ID = "__unknown__"+OVERFLOW_ACCOUNT_ID = "__overflow__"+ACCOUNT_DIMENSION_SUPPORTED_METRICS = frozenset(+ {+ "openviking_embedding_requests_total",+ "openviking_embedding_latency_seconds",+ "openviking_embedding_errors_total",+ "openviking_http_requests_total",+ "openviking_http_request_duration_seconds",+ "openviking_http_inflight_requests",+ "openviking_resource_stage_total",+ "openviking_resource_stage_duration_seconds",+ "openviking_resource_wait_duration_seconds",+ "openviking_retrieval_requests_total",+ "openviking_retrieval_results_total",+ "openviking_retrieval_zero_result_total",+ "openviking_retrieval_latency_seconds",+ "openviking_retrieval_rerank_used_total",+ "openviking_retrieval_rerank_fallback_total",+ "openviking_session_lifecycle_total",+ "openviking_session_contexts_used_total",+ "openviking_session_archive_total",+ "openviking_operation_requests_total",+ "openviking_operation_duration_seconds",+ "openviking_operation_tokens_total",+ "openviking_vlm_calls_total",+ "openviking_vlm_call_duration_seconds",+ "openviking_vlm_tokens_input_total",+ "openviking_vlm_tokens_output_total",+ "openviking_vlm_tokens_total",+ }+)+++@dataclass(frozen=True, slots=True)+class MetricAccountDimensionConfig:+ """+ Describe the process-wide settings that govern `account_id` label injection.++ The config is intentionally narrow: enablement, per-metric allowlisting, and the active+ account cap that protects Prometheus from unbounded tenant cardinality.+ """++ # Enabled by default, but still allowlist-gated at write time.+ enabled: bool = True+ max_active_accounts: int = 100+ metric_allowlist: frozenset[str] = frozenset()+++class MetricAccountContextResolver:+ """+ Resolve candidate account identifiers from the supported metrics context sources.++ Resolution follows the project-wide source priority so all collectors interpret account+ provenance consistently.+ """++ def resolve(+ self,+ *,+ explicit_account_id: str | None = None,+ owner_account_id: str | None = None,+ ) -> str | None:+ """+ Resolve one account identifier in priority order: explicit > HTTP context > task owner.++ The method normalizes blank values away and returns `None` when no source can supply a+ stable account id for the current metric write.+ """+ if explicit_account_id and str(explicit_account_id).strip():+ return str(explicit_account_id).strip()++ http_account_id = get_metric_account_context().http_account_id+ if http_account_id and str(http_account_id).strip():+ return str(http_account_id).strip()++ if owner_account_id and str(owner_account_id).strip():+ return str(owner_account_id).strip()+ return None+++class MetricAccountDimensionPolicy:+ """+ Decide the final `account_id` label value for an individual metric write.++ The policy layer is responsible for the allowlist gate and active-account limiting. It does+ not discover account ids itself; it only decides whether a resolved candidate becomes a real+ tenant label, `__unknown__`, or `__overflow__`.+ """++ def __init__(+ self,+ *,+ enabled: bool,+ metric_allowlist: set[str] | frozenset[str],+ max_active_accounts: int,+ ) -> None:+ """+ Initialize policy state for allowlist filtering and active-account limiting.++ The active-account set is held in-memory and guarded by a lock because collectors may+ resolve account labels concurrently from request threads and exporter refresh paths.+ """+ self._enabled = bool(enabled)+ exact: set[str] = set()+ prefixes: set[str] = set()+ for item in metric_allowlist:+ normalized = str(item).strip()+ if not normalized:+ continue+ if normalized.endswith("*"):+ # Limited wildcard syntax: trailing '*' means prefix match.+ # '*' alone (empty prefix) is ignored to reduce accidental broad exposure.+ prefix = normalized[:-1].strip()+ if prefix:+ prefixes.add(prefix)+ continue+ exact.add(normalized)+ self._metric_allowlist_exact = frozenset(exact)+ # Tuple keeps iteration deterministic and slightly faster than set in hot path.+ self._metric_allowlist_prefixes = tuple(sorted(prefixes))+ self._max_active_accounts = max(0, int(max_active_accounts))+ self._lock = threading.Lock()+ self._active_accounts: set[str] = set()++ def _is_metric_allowlisted(self, metric_name: str) -> bool:+ name = str(metric_name)+ if name in self._metric_allowlist_exact:+ return True+ for prefix in self._metric_allowlist_prefixes:+ if name.startswith(prefix):+ return True+ return False++ def resolve(self, *, metric_name: str, account_id: str | None) -> str:+ """+ Resolve the stored label value as a real id, `__unknown__`, or `__overflow__`.++ A metric only receives a real tenant id when account-dimension support is enabled, the+ metric name is explicitly allowlisted, and the account passes the active-account cap.+ """+ if not self._enabled:+ return UNKNOWN_ACCOUNT_ID+ if not self._is_metric_allowlisted(metric_name):+ return UNKNOWN_ACCOUNT_ID++ normalized = str(account_id or "").strip()+ if not normalized:+ return UNKNOWN_ACCOUNT_ID++ with self._lock:+ if normalized in self._active_accounts:+ return normalized+ if (+ self._max_active_accounts > 0+ and len(self._active_accounts) >= self._max_active_accounts+ ):+ return OVERFLOW_ACCOUNT_ID+ self._active_accounts.add(normalized)+ return normalized+++_RUNTIME_LOCK = threading.Lock()+_RUNTIME_POLICY = MetricAccountDimensionPolicy(+ enabled=False,+ metric_allowlist=set(),+ max_active_accounts=0,+)+_RUNTIME_RESOLVER = MetricAccountContextResolver()+++def configure_metric_account_dimension(+ *,+ enabled: bool | None = None,+ metric_allowlist: set[str] | list[str] | tuple[str, ...] | None = None,+ max_active_accounts: int | None = None,+ policy: MetricAccountDimensionPolicy | None = None,+ resolver: MetricAccountContextResolver | None = None,+) -> None:+ """+ Configure the process-global account-dimension runtime used by collector write helpers.++ Callers may replace the policy and resolver explicitly for tests, or provide primitive+ configuration fields that are converted into the default runtime objects.+ """+ global _RUNTIME_POLICY, _RUNTIME_RESOLVER+ with _RUNTIME_LOCK:+ if policy is not None:+ _RUNTIME_POLICY = policy+ else:+ _RUNTIME_POLICY = MetricAccountDimensionPolicy(+ enabled=bool(enabled),+ metric_allowlist=set(metric_allowlist or ()),+ max_active_accounts=int(max_active_accounts or 0),+ )+ if resolver is not None:+ _RUNTIME_RESOLVER = resolver+++def reset_metric_account_dimension() -> None:+ """+ Reset the process-global account-dimension runtime to the disabled default state.++ This helper is mainly used by tests and shutdown paths that need deterministic process-wide+ cleanup between runs.+ """+ configure_metric_account_dimension(+ enabled=False,+ metric_allowlist=set(),+ max_active_accounts=0,+ resolver=MetricAccountContextResolver(),+ )+++def resolve_metric_account_label(+ *,+ metric_name: str,+ explicit_account_id: str | None = None,+ owner_account_id: str | None = None,+) -> str:+ """+ Resolve the final `account_id` label for one metric write using the global runtime objects.++ The helper snapshots the current resolver and policy under the runtime lock, then performs+ the actual resolution outside the lock to keep the synchronized section small.+ """+ with _RUNTIME_LOCK:+ resolver = _RUNTIME_RESOLVER+ policy = _RUNTIME_POLICY+ return policy.resolve(+ metric_name=metric_name,+ account_id=resolver.resolve(+ explicit_account_id=explicit_account_id,+ owner_account_id=owner_account_id,+ ),+ )+++def metric_supports_account_dimension(metric_name: str) -> bool:+ """+ Return whether a metric family participates in account-dimension label injection at all.++ This support set is narrower than the allowlist: unsupported metrics never receive an+ `account_id` label even if configuration would otherwise enable account-dimension writes.+ """+ return str(metric_name) in ACCOUNT_DIMENSION_SUPPORTED_METRICSdiff --git a/openviking/metrics/bootstrap.py b/openviking/metrics/bootstrap.pynew file mode 100644index 000000000..7bf8eca18--- /dev/null+++ b/openviking/metrics/bootstrap.py@@ -0,0 +1,110 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Default metrics bootstrap.++This module wires DataSources and Collectors into a CollectorManager that is invoked+before each Prometheus scrape (`/metrics`).++The registered collectors are intentionally split into two categories:+- Per-scrape refresh (no TTL): cheap, purely in-process reads (e.g., queue/lock counters).+- TTL/SWR protected: collectors that may touch slower or less reliable subsystems+ (e.g., service/model probes, vikingdb state, aggregated model usage).++If you add a new collector:+- Prefer low cardinality labels.+- Decide whether it needs TTL refresh control.+- Add/extend tests under `tests/metrics`.++Notes:+- Default-enabled collectors must keep refresh best-effort: `/metrics` should remain available+ even if one subsystem is down.+- For aggregated collectors where a "failure default" is not meaningful, export `valid=1/0`+ (or a similar validity gauge) so dashboards/alerts can distinguish stale data.+"""++from __future__ import annotations++from openviking.metrics.collectors import (+ AsyncSystemProbeCollector,+ CollectorManager,+ EncryptionProbeCollector,+ LockCollector,+ ModelProviderProbeCollector,+ ModelUsageCollector,+ ObserverHealthCollector,+ ObserverStateCollector,+ QueueCollector,+ RetrievalBackendProbeCollector,+ ServiceProbeCollector,+ StorageProbeCollector,+ TaskTrackerCollector,+ VikingDBCollector,+)+from openviking.metrics.datasources.encryption import EncryptionProbeDataSource+from openviking.metrics.datasources.model_usage import ModelUsageDataSource+from openviking.metrics.datasources.observer_state import (+ LockStateDataSource,+ ObserverStateDataSource,+ VikingDBStateDataSource,+)+from openviking.metrics.datasources.probes import (+ AsyncSystemProbeDataSource,+ ModelProviderProbeDataSource,+ RetrievalBackendProbeDataSource,+ ServiceProbeDataSource,+ StorageProbeDataSource,+)+from openviking.metrics.datasources.queue import QueuePipelineStateDataSource+from openviking.metrics.datasources.task import TaskStateDataSource+from openviking_cli.utils.config.open_viking_config import get_openviking_config+++def create_default_collector_manager(*, app=None, service=None) -> CollectorManager:+ """+ Build the default `CollectorManager` used by the server's Prometheus endpoint.++ The default manager mixes lightweight state collectors, probe collectors, and a few+ aggregated collectors that rely on TTL/SWR behavior inside `CollectorManager`.++ Args:+ app: FastAPI application instance needed by certain readiness probes.+ service: Service object that exposes in-process state and subsystem handles preferred by+ several collectors and datasources.++ Returns:+ A `CollectorManager` with the standard OpenViking collector set registered in the order+ expected by the metrics bootstrap path.+ """+ manager = CollectorManager()+ manager.register(QueueCollector(data_source=QueuePipelineStateDataSource()))+ manager.register(TaskTrackerCollector(data_source=TaskStateDataSource()))+ manager.register(ObserverHealthCollector(data_source=ObserverStateDataSource(service=service)))+ manager.register(ObserverStateCollector(data_source=ObserverStateDataSource(service=service)))+ manager.register(LockCollector(data_source=LockStateDataSource()))+ manager.register(VikingDBCollector(data_source=VikingDBStateDataSource(service=service)))+ manager.register(+ ModelUsageCollector(+ data_source=ModelUsageDataSource(config_provider=get_openviking_config, service=service)+ )+ )+ manager.register(+ ServiceProbeCollector(data_source=ServiceProbeDataSource(app=app, service=service))+ )+ manager.register(StorageProbeCollector(data_source=StorageProbeDataSource()))+ manager.register(+ RetrievalBackendProbeCollector(data_source=RetrievalBackendProbeDataSource(service=service))+ )+ manager.register(+ ModelProviderProbeCollector(+ data_source=ModelProviderProbeDataSource(config_provider=get_openviking_config)+ )+ )+ manager.register(AsyncSystemProbeCollector(data_source=AsyncSystemProbeDataSource()))+ manager.register(+ EncryptionProbeCollector(+ data_source=EncryptionProbeDataSource(config_provider=get_openviking_config)+ )+ )+ return managerdiff --git a/openviking/metrics/collectors/__init__.py b/openviking/metrics/collectors/__init__.pynew file mode 100644index 000000000..042d06b07--- /dev/null+++ b/openviking/metrics/collectors/__init__.py@@ -0,0 +1,61 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""Collector entry points for event-driven, state, probe, and exporter-facing metrics writes."""++from .async_system_probe import AsyncSystemProbeCollector+from .base import (+ CollectorConfig,+ DomainStatsMetricCollector,+ EventMetricCollector,+ ProbeMetricCollector,+ Refreshable,+ StateMetricCollector,+)+from .cache import CacheCollector+from .embedding import EmbeddingCollector+from .encryption import EncryptionCollector+from .encryption_probe import EncryptionProbeCollector+from .lock import LockCollector+from .manager import CollectorManager, RefreshResult+from .model_provider_probe import ModelProviderProbeCollector+from .model_usage import ModelUsageCollector+from .observer_health import ObserverHealthCollector+from .observer_state import ObserverStateCollector+from .queue import QueueCollector+from .retrieval import RetrievalCollector+from .retrieval_backend_probe import RetrievalBackendProbeCollector+from .service_probe import ServiceProbeCollector+from .storage_probe import StorageProbeCollector+from .task_tracker import TaskTrackerCollector+from .vikingdb import VikingDBCollector+from .vlm import VLMCollector++__all__ = [+ "CollectorConfig",+ "Refreshable",+ "EventMetricCollector",+ "StateMetricCollector",+ "DomainStatsMetricCollector",+ "ProbeMetricCollector",+ "RetrievalCollector",+ "EmbeddingCollector",+ "VLMCollector",+ "CacheCollector",+ "EncryptionCollector",+ "QueueCollector",+ "LockCollector",+ "VikingDBCollector",+ "ObserverHealthCollector",+ "TaskTrackerCollector",+ "ModelUsageCollector",+ "ObserverStateCollector",+ "ServiceProbeCollector",+ "StorageProbeCollector",+ "RetrievalBackendProbeCollector",+ "ModelProviderProbeCollector",+ "AsyncSystemProbeCollector",+ "EncryptionProbeCollector",+ "CollectorManager",+ "RefreshResult",+]diff --git a/openviking/metrics/collectors/async_system_probe.py b/openviking/metrics/collectors/async_system_probe.pynew file mode 100644index 000000000..89fcae2c8--- /dev/null+++ b/openviking/metrics/collectors/async_system_probe.py@@ -0,0 +1,73 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from dataclasses import dataclass+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector+from openviking.metrics.datasources.probes import AsyncSystemProbeDataSource++from .base import CollectorConfig, ProbeMetricCollector+++@dataclass+class AsyncSystemProbeCollector(ProbeMetricCollector):+ """+ Export readiness for individual async-system probe checks.++ The datasource returns a bounded mapping of probe names to booleans. This collector converts+ that mapping into one readiness gauge series per probe so dashboards can distinguish which+ async subsystem is currently unhealthy.+ """++ DOMAIN: ClassVar[str] = "async_system"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_readiness+ # e.g.: openviking_async_system_readiness+ READINESS: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "readiness")++ data_source: AsyncSystemProbeDataSource+ config: CollectorConfig = CollectorConfig(timeout_seconds=0.5)+ # Keep the last-known probe set so failures can still emit `valid="0"` series.+ _known_probes: list[str] = None # type: ignore[assignment]++ def __post_init__(self) -> None:+ """Initialize the last-known probe set for deterministic failure output."""+ if self._known_probes is None:+ self._known_probes = ["queue"]++ def read_metric_input(self):+ """Read the latest async-system probe mapping from the datasource."""+ return self.data_source.read_probe_state()++ def collect_hook(self, registry, metric_input) -> None:+ """+ Publish one readiness sample per async-system probe name.++ The helper normalizes probe names to strings and writes a fixed 0/1 readiness gauge using+ `probe` as the only series label.+ """+ normalized = {str(k): bool(v) for k, v in metric_input.items()}+ self._known_probes = list(normalized.keys())+ for probe, is_ready in normalized.items():+ self.replace_gauge(+ registry,+ self.READINESS,+ 1.0 if bool(is_ready) else 0.0,+ match_labels={"probe": str(probe)},+ labels={"probe": str(probe), "valid": "1"},+ label_names=("probe", "valid"),+ )++ def collect_stale_hook(self, registry, error: Exception) -> None:+ """Emit invalid readiness series for the last known probe set on failure."""+ for probe in self._known_probes:+ self.replace_gauge(+ registry,+ self.READINESS,+ 0.0,+ match_labels={"probe": str(probe)},+ labels={"probe": str(probe), "valid": "0"},+ label_names=("probe", "valid"),+ )diff --git a/openviking/metrics/collectors/base.py b/openviking/metrics/collectors/base.pynew file mode 100644index 000000000..939dc57eb--- /dev/null+++ b/openviking/metrics/collectors/base.py@@ -0,0 +1,603 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Shared collector-side type definitions.++This module separates two concerns:+- Collection semantics, which belong to `MetricCollector` in `metrics.base`+- Refresh-management semantics, which belong to `Refreshable` here++The split keeps the public collector API simple (`collect` / `receive`) while still allowing+the scrape manager to recognize which collectors support timeout, TTL, and SWR-style control.+"""++from __future__ import annotations++from abc import ABC, abstractmethod+from dataclasses import dataclass+from typing import ClassVar++from openviking.metrics.account_dimension import (+ metric_supports_account_dimension,+ resolve_metric_account_label,+)+from openviking.metrics.core.base import MetricCollector, ReadEnvelope+++class DataSourceReadError(RuntimeError):+ """+ Describe a normalized datasource failure surfaced through a collector read path.++ Collectors raise this wrapper when a `ReadEnvelope` reports failure so stale/fallback hooks+ can receive a regular exception object with preserved datasource error metadata.+ """++ def __init__(+ self,+ *,+ source_name: str,+ error_type: str | None = None,+ error_message: str | None = None,+ ) -> None:+ text = f"{source_name} read failed"+ if error_type:+ text = f"{text}: {error_type}"+ if error_message:+ text = f"{text}: {error_message}"+ super().__init__(text)+++@dataclass(frozen=True, slots=True)+class CollectorConfig:+ """+ Runtime refresh-control configuration for refresh-managed collectors.++ Attributes:+ ttl_seconds: Optional refresh time-to-live. When set, `CollectorManager` may skip+ repeated collections within the TTL window and apply SWR-style background refresh+ after expiry when previous data exists.+ timeout_seconds: Per-collection time budget enforced by `CollectorManager` when the+ collector is executed as part of the scrape pipeline.+ """++ ttl_seconds: float | None = None+ timeout_seconds: float = 0.5+++class Refreshable(ABC):+ """+ Marker interface for collectors that participate in refresh-controlled scrape execution.++ A `Refreshable` collector is still a normal `MetricCollector`, but it additionally exposes+ a `config` attribute so `CollectorManager` can apply timeout, TTL, and SWR policies.+ Event-only collectors intentionally do not implement this interface.+ """++ config: CollectorConfig+++class CollectorMetricWriter:+ """+ Collector-scoped registry writer that injects the resolved `account_id` label.++ The writer centralizes account-dimension support checks and final label resolution so+ concrete collectors do not need to duplicate that policy logic.+ """++ def __init__(self, registry, *, owner_account_id: str | None = None) -> None:+ """+ Bind the writer to one registry instance and an optional owner-account fallback value.++ `owner_account_id` is mainly used by background or task-based collectors that cannot rely+ on an active HTTP request context.+ """+ self._registry = registry+ self._owner_account_id = owner_account_id++ def __getattr__(self, name: str):+ """Forward unsupported attributes to the underlying registry instance transparently."""+ return getattr(self._registry, name)++ def inc_counter(+ self,+ name: str,+ *,+ amount: float = 1.0,+ labels: dict[str, str] | None = None,+ label_names: tuple[str, ...] = (),+ account_id: str | None = None,+ ) -> None:+ """Increment a counter after normalizing any account-aware labels for the target metric."""+ final_labels, final_label_names = self._with_account_labels(+ metric_name=name,+ labels=labels,+ label_names=label_names,+ explicit_account_id=account_id,+ )+ self._registry.inc_counter(+ name,+ amount=amount,+ labels=final_labels,+ label_names=final_label_names,+ )++ def set_gauge(+ self,+ name: str,+ value: float,+ *,+ labels: dict[str, str] | None = None,+ label_names: tuple[str, ...] = (),+ account_id: str | None = None,+ ) -> None:+ """Set a gauge after normalizing any account-aware labels for the target metric."""+ final_labels, final_label_names = self._with_account_labels(+ metric_name=name,+ labels=labels,+ label_names=label_names,+ explicit_account_id=account_id,+ )+ self._registry.set_gauge(+ name,+ value,+ labels=final_labels,+ label_names=final_label_names,+ )++ def observe_histogram(+ self,+ name: str,+ value: float,+ *,+ labels: dict[str, str] | None = None,+ label_names: tuple[str, ...] = (),+ buckets=(),+ account_id: str | None = None,+ ) -> None:+ """Observe a histogram sample after normalizing account-aware labels for the metric."""+ final_labels, final_label_names = self._with_account_labels(+ metric_name=name,+ labels=labels,+ label_names=label_names,+ explicit_account_id=account_id,+ )+ kwargs = {+ "labels": final_labels,+ "label_names": final_label_names,+ }+ if buckets:+ kwargs["buckets"] = buckets+ self._registry.observe_histogram(name, value, **kwargs)++ def gauge_delete_matching(+ self,+ name: str,+ *,+ match_labels: dict[str, str],+ account_id: str | None = None,+ ) -> None:+ """Delete matching gauge series while preserving the current account partition."""+ if not metric_supports_account_dimension(name):+ self._registry.gauge_delete_matching(name, match_labels=dict(match_labels))+ return+ final_account_id = resolve_metric_account_label(+ metric_name=name,+ explicit_account_id=account_id,+ owner_account_id=self._owner_account_id,+ )+ labels = dict(match_labels)+ labels["account_id"] = final_account_id+ self._registry.gauge_delete_matching(name, match_labels=labels)++ def _with_account_labels(+ self,+ *,+ metric_name: str,+ labels: dict[str, str] | None,+ label_names: tuple[str, ...],+ explicit_account_id: str | None,+ ) -> tuple[dict[str, str], tuple[str, ...]]:+ """+ Return labels and label names augmented with the resolved account dimension when allowed.++ Unsupported metrics bypass `account_id` injection entirely so their exported label shape+ stays identical to the original family contract.+ """+ final_labels = dict(labels or {})+ if not metric_supports_account_dimension(metric_name):+ final_label_names = tuple(sorted(set(label_names) | set(final_labels.keys())))+ return final_labels, final_label_names+ if "account_id" in final_labels:+ explicit_account_id = final_labels.pop("account_id")+ final_labels["account_id"] = resolve_metric_account_label(+ metric_name=metric_name,+ explicit_account_id=explicit_account_id,+ owner_account_id=self._owner_account_id,+ )+ final_label_names = tuple(sorted(set(label_names) | set(final_labels.keys())))+ return final_labels, final_label_names+++class EventMetricCollector(MetricCollector, ABC):+ """+ Base category for event-driven collectors.++ These collectors primarily implement `receive(...)` and translate dispatched events into+ registry writes. They do not need refresh-control metadata because they are not scheduled+ by `CollectorManager`.+ """++ KIND: ClassVar[str] = "event"+ SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset()++ @classmethod+ def kind(cls) -> str:+ """Return the collector category identifier."""+ return cls.KIND++ def receive(self, event_name: str, payload: dict, registry) -> None:+ """+ Normalize and gate one incoming event before delegating to the subclass hook.++ The shared event-collector entrypoint performs the logic that all event collectors+ should agree on:+ - normalize the event name into its string form+ - ignore events this collector does not claim to support+ - ignore malformed payloads that are not dictionaries+ - delegate supported events to the concrete collector hook+ """+ normalized_event = str(event_name)+ if normalized_event not in self.SUPPORTED_EVENTS:+ return+ if not isinstance(payload, dict):+ return+ self.receive_hook(+ normalized_event,+ payload,+ CollectorMetricWriter(+ registry,+ owner_account_id=str(payload.get("owner_account_id") or "") or None,+ ),+ )++ @abstractmethod+ def receive_hook(self, event_name: str, payload: dict, registry) -> None:+ """Handle one supported event payload after the shared receive-path checks succeed."""+++class StateMetricCollector(MetricCollector, Refreshable, ABC):+ """+ Base category for pull-based state collectors.++ State collectors read current in-process or subsystem state during `collect(...)` and are+ typically executed in the scrape pipeline. Because they participate in that pipeline, they+ also implement `Refreshable` and therefore expose `CollectorConfig`.+ """++ KIND: ClassVar[str] = "state"+ STALE_ON_ERROR: ClassVar[bool] = False++ @classmethod+ def kind(cls) -> str:+ """Return the collector category identifier."""+ return cls.KIND++ def collect(self, registry) -> None:+ """+ Read state input and delegate metric emission through subclass hooks.++ Concrete state collectors define how state is fetched and how the fetched input is+ translated into registry writes. Read failures are delegated to `collect_error_hook(...)`+ so collectors can either propagate or publish stale/failure metrics.+ """+ writer = CollectorMetricWriter(registry)+ try:+ metric_input = self.read_metric_input()+ except Exception as error:+ self.collect_error_hook(writer, error)+ return+ if isinstance(metric_input, ReadEnvelope):+ if not metric_input.ok:+ self.collect_error_hook(+ writer,+ DataSourceReadError(+ source_name=self.collector_name(),+ error_type=metric_input.error_type,+ error_message=metric_input.error_message,+ ),+ )+ return+ metric_input = metric_input.value+ self.collect_hook(writer, metric_input)++ @abstractmethod+ def read_metric_input(self):+ """Read the current state snapshot or intermediate input needed for metric emission."""++ @abstractmethod+ def collect_hook(self, registry, metric_input) -> None:+ """Translate the successfully-read state input into registry writes."""++ def collect_error_hook(self, registry, error: Exception) -> None:+ """Handle state-input read failures; the default behavior is to re-raise the error."""+ if self.STALE_ON_ERROR:+ self.collect_stale_hook(registry, error)+ return+ raise error++ def collect_stale_hook(self, registry, error: Exception) -> None:+ """Handle stale-on-error fallback; subclasses may override when stale export is allowed."""+ raise error++ def replace_gauge_series(+ self,+ registry,+ metric_name: str,+ value: float,+ *,+ match_labels: dict,+ labels: dict | None = None,+ label_names: tuple[str, ...] = (),+ ) -> None:+ """Replace the full gauge series selected by `match_labels` with one fresh value."""+ registry.gauge_delete_matching(str(metric_name), match_labels=match_labels)+ if labels is None:+ registry.set_gauge(str(metric_name), float(value))+ return+ registry.set_gauge(+ str(metric_name),+ float(value),+ labels=labels,+ label_names=label_names,+ )+++class DomainStatsMetricCollector(MetricCollector, Refreshable, ABC):+ """+ Base category for pull-based domain-statistics collectors.++ These collectors compute higher-level aggregate metrics from existing subsystem state,+ often including validity semantics such as stale-or-invalid output on failure. They are+ refresh-managed because their collection cost and failure behavior may need TTL/SWR control.+ """++ KIND: ClassVar[str] = "domain_stats"++ @classmethod+ def kind(cls) -> str:+ """Return the collector category identifier."""+ return cls.KIND++ def collect(self, registry) -> None:+ """+ Read domain-statistics input and delegate metric emission through subclass hooks.++ Concrete domain collectors define how aggregate input is read and how it becomes+ registry output. Read failures are delegated to `collect_error_hook(...)` so collectors+ can emit stale/invalid gauges when that matches their contract.+ """+ writer = CollectorMetricWriter(registry)+ try:+ metric_input = self.read_metric_input()+ except Exception as error:+ self.collect_error_hook(writer, error)+ return+ if isinstance(metric_input, ReadEnvelope):+ if not metric_input.ok:+ self.collect_error_hook(+ writer,+ DataSourceReadError(+ source_name=self.collector_name(),+ error_type=metric_input.error_type,+ error_message=metric_input.error_message,+ ),+ )+ return+ metric_input = metric_input.value+ self.collect_hook(writer, metric_input)++ @abstractmethod+ def read_metric_input(self):+ """Read the aggregate domain-statistics input required for one collection cycle."""++ @abstractmethod+ def collect_hook(self, registry, metric_input) -> None:+ """Translate the successfully-read domain-statistics input into registry writes."""++ def collect_error_hook(self, registry, error: Exception) -> None:+ """Handle domain-statistics read failures; the default behavior is to re-raise."""+ raise error++ def replace_global_gauges(+ self,+ registry,+ *,+ metric_names: list[str],+ values: list[float],+ labels: dict,+ label_names: tuple[str, ...],+ ) -> None:+ """Replace all existing global gauge series for the provided metric names in one pass."""+ for metric_name, value in zip(metric_names, values, strict=True):+ registry.gauge_delete_matching(str(metric_name), match_labels={})+ registry.set_gauge(+ str(metric_name),+ float(value),+ labels=labels,+ label_names=label_names,+ )++ def inc_counter_from_cumulative(+ self,+ *,+ registry,+ metric_name: str,+ key: tuple,+ current_value: int,+ labels: dict | None = None,+ label_names: tuple[str, ...] = (),+ ) -> None:+ """Convert cumulative counters into monotonic Prometheus increments for one key."""+ last_seen = getattr(self, "_cumulative_last_seen", None)+ if last_seen is None:+ last_seen = {}+ # Lazily create per-collector state; stored on the instance to avoid shared globals.+ self._cumulative_last_seen = last_seen++ prev = last_seen.get(key)+ current = int(current_value)+ if prev is None:+ delta = current+ elif current >= int(prev):+ delta = current - int(prev)+ else:+ delta = current+ last_seen[key] = current+ if delta > 0:+ registry.inc_counter(+ str(metric_name),+ amount=float(delta),+ labels=labels,+ label_names=label_names,+ )+++class ProbeMetricCollector(MetricCollector, Refreshable, ABC):+ """+ Base category for readiness/health probe collectors.++ Probe collectors convert subsystem readiness checks into metrics. They inherit+ `Refreshable` because probes may touch slower or less reliable dependencies and therefore+ benefit from centralized timeout and TTL handling.+ """++ KIND: ClassVar[str] = "probe"++ @classmethod+ def kind(cls) -> str:+ """Return the collector category identifier."""+ return cls.KIND++ def collect(self, registry) -> None:+ """+ Read probe input and delegate metric emission through subclass hooks.++ Probe collectors frequently need failure/staleness fallbacks, so the shared collect path+ centralizes read/dispatch flow while allowing concrete collectors to override+ `collect_error_hook(...)` for probe-specific validity behavior.+ """+ writer = CollectorMetricWriter(registry)+ try:+ metric_input = self.read_metric_input()+ except Exception as error:+ self.collect_error_hook(writer, error)+ return+ if isinstance(metric_input, ReadEnvelope):+ if not metric_input.ok:+ self.collect_error_hook(+ writer,+ DataSourceReadError(+ source_name=self.collector_name(),+ error_type=metric_input.error_type,+ error_message=metric_input.error_message,+ ),+ )+ return+ metric_input = metric_input.value+ self.collect_hook(writer, metric_input)++ @abstractmethod+ def read_metric_input(self):+ """Read the readiness/health probe input required for one collection cycle."""++ @abstractmethod+ def collect_hook(self, registry, metric_input) -> None:+ """Translate the successfully-read probe input into registry writes."""++ def collect_error_hook(self, registry, error: Exception) -> None:+ self.collect_stale_hook(registry, error)++ def collect_stale_hook(self, registry, error: Exception) -> None:+ """Ignore probe read failures by default after `collect_error_hook` delegates here."""+ return None++ def replace_gauge(+ self,+ registry,+ metric_name: str,+ value: float,+ *,+ match_labels: dict,+ labels: dict | None = None,+ label_names: tuple[str, ...] = (),+ ) -> None:+ """Replace the selected gauge series with one fresh readiness-style value."""+ registry.gauge_delete_matching(str(metric_name), match_labels=match_labels)+ if labels is None:+ registry.set_gauge(str(metric_name), float(value))+ return+ registry.set_gauge(+ str(metric_name),+ float(value),+ labels=labels,+ label_names=label_names,+ )++ def set_readiness_flags(+ self,+ registry,+ *,+ metrics: dict[str, str],+ state: dict,+ ) -> None:+ """Set a fixed mapping of readiness metrics from boolean values in the probe state."""+ for metric_name, key in metrics.items():+ registry.set_gauge(+ str(metric_name),+ 1.0 if bool(state.get(key, False)) else 0.0,+ )++ def set_readiness_flags_on_error(+ self,+ registry,+ *,+ metric_names: list[str],+ ) -> None:+ """Drive the provided readiness metrics to zero when the probe itself fails."""+ for metric_name in metric_names:+ registry.set_gauge(str(metric_name), 0.0)++ def set_labeled_readiness(+ self,+ registry,+ metric_name: str,+ *,+ items: dict[str, bool],+ label_key: str,+ label_names: tuple[str, ...],+ valid_label: str | None = None,+ ) -> None:+ """Emit one readiness gauge series per named item, optionally attaching a validity flag."""+ for name, ok in items.items():+ labels = {label_key: str(name)}+ if valid_label is not None:+ labels["valid"] = str(valid_label)+ registry.set_gauge(+ str(metric_name),+ 1.0 if ok else 0.0,+ labels=labels,+ label_names=label_names,+ )++ def delete_gauge_series_for_labels(+ self,+ registry,+ metric_name: str,+ *,+ label_key: str,+ values: list[str],+ ) -> None:+ """Delete all gauge series whose label values are listed in `values`."""+ for value in values:+ registry.gauge_delete_matching(str(metric_name), match_labels={label_key: str(value)})diff --git a/openviking/metrics/collectors/cache.py b/openviking/metrics/collectors/cache.pynew file mode 100644index 000000000..7cd7c0fbb--- /dev/null+++ b/openviking/metrics/collectors/cache.py@@ -0,0 +1,69 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Event collector: CacheCollector.++Records cache hit/miss counters by cache level (L0/L1/L2).+The cache level label is intentionally bounded to avoid cardinality issues.+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector++from .base import EventMetricCollector+++@dataclass+class CacheCollector(EventMetricCollector):+ """+ Translate cache hit and miss events into bounded cache-level counters.++ The collector is purely event-driven and only accepts the small set of cache levels emitted by+ the cache datasource, which keeps the resulting Prometheus series count stable.+ """++ DOMAIN: ClassVar[str] = "cache"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_hits_total+ # e.g.: openviking_cache_hits_total+ HITS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "hits", unit="total")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_misses_total+ # e.g.: openviking_cache_misses_total+ MISSES_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "misses", unit="total")++ SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset({"cache.hit", "cache.miss"})++ def collect(self, registry=None) -> None:+ """Implement the collector interface as a no-op because cache metrics are push-driven."""+ return None++ def receive_hook(self, event_name: str, payload: dict, registry) -> None:+ """+ Route one cache event payload to the matching hit or miss counter update.++ The payload is expected to contain a normalized `level` field supplied by the datasource.+ """+ if event_name == "cache.hit":+ self.record_hit(registry, level=str(payload["level"]))+ return+ self.record_miss(registry, level=str(payload["level"]))++ def record_hit(self, registry, *, level: str) -> None:+ """Increment the cache-hit counter for one normalized cache level value."""+ registry.inc_counter(+ self.HITS_TOTAL,+ labels={"level": str(level)},+ label_names=("level",),+ )++ def record_miss(self, registry, *, level: str) -> None:+ """Increment the cache-miss counter for one normalized cache level value."""+ registry.inc_counter(+ self.MISSES_TOTAL,+ labels={"level": str(level)},+ label_names=("level",),+ )diff --git a/openviking/metrics/collectors/embedding.py b/openviking/metrics/collectors/embedding.pynew file mode 100644index 000000000..bd5869cb0--- /dev/null+++ b/openviking/metrics/collectors/embedding.py@@ -0,0 +1,118 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Event collector: EmbeddingCollector.++Tracks embedding request outcomes and latency:+- Requests counter by status+- Latency histogram by status+- Error counter by normalized error code++It is fed by EmbeddingEventDataSource events emitted from embedding call sites.+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector++from .base import EventMetricCollector+++@dataclass+class EmbeddingCollector(EventMetricCollector):+ """+ Translate embedding success and error events into request, latency, and error metrics.++ The collector treats embedding outcomes as event-driven writes because the interesting facts+ are known at completion time and do not require scrape-time state inspection.+ """++ DOMAIN: ClassVar[str] = "embedding"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_requests_total+ # e.g.: openviking_embedding_requests_total+ REQUESTS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "requests", unit="total")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_latency_seconds+ # e.g.: openviking_embedding_latency_seconds+ LATENCY_SECONDS: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "latency", unit="seconds")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_errors_total+ # e.g.: openviking_embedding_errors_total+ ERRORS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "errors", unit="total")++ SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset({"embedding.success", "embedding.error"})++ def collect(self, registry=None) -> None:+ """Implement the unified collector interface as a no-op for this event-driven collector."""+ return None++ def receive_hook(self, event_name: str, payload: dict, registry) -> None:+ """+ Translate one supported embedding event into the corresponding metric writes.++ Success and error events intentionally diverge into separate write paths so the collector+ can emit different label sets without overloading one payload shape.+ """+ if event_name == "embedding.success":+ self.record_success(+ registry,+ latency_seconds=float(payload["latency_seconds"]),+ account_id=(+ None if payload.get("account_id") is None else str(payload.get("account_id"))+ ),+ )+ return+ if event_name == "embedding.error":+ self.record_error(+ registry,+ error_code=str(payload["error_code"]),+ account_id=(+ None if payload.get("account_id") is None else str(payload.get("account_id"))+ ),+ )++ def record_success(+ self, registry, *, latency_seconds: float, account_id: str | None = None+ ) -> None:+ """+ Record a successful embedding request and its latency histogram sample.++ The collector emits both a request counter and a latency observation so success volume and+ latency remain queryable from the same event stream.+ """+ labels = {"status": "ok"}+ registry.inc_counter(+ self.REQUESTS_TOTAL,+ labels=labels,+ label_names=("status",),+ account_id=account_id,+ )+ registry.observe_histogram(+ self.LATENCY_SECONDS,+ float(latency_seconds),+ labels=labels,+ label_names=("status",),+ account_id=account_id,+ )++ def record_error(self, registry, *, error_code: str, account_id: str | None = None) -> None:+ """+ Record an errored embedding request and increment its normalized error-code counter.++ Error outcomes still increment the request-total family so total embedding attempt counts+ remain comparable with successful request counts.+ """+ registry.inc_counter(+ self.REQUESTS_TOTAL,+ labels={"status": "error"},+ label_names=("status",),+ account_id=account_id,+ )+ registry.inc_counter(+ self.ERRORS_TOTAL,+ labels={"error_code": str(error_code or "unknown")},+ label_names=("error_code",),+ account_id=account_id,+ )diff --git a/openviking/metrics/collectors/encryption.py b/openviking/metrics/collectors/encryption.pynew file mode 100644index 000000000..4726cf108--- /dev/null+++ b/openviking/metrics/collectors/encryption.py@@ -0,0 +1,266 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Event collector: EncryptionCollector.++This collector is fed by encryption DataSources (and by crypto code paths emitting those events).+It exports operational metrics for:+- encrypt/decrypt operation count and latency+- processed bytes and payload size histogram+- auth-failure count+- key derivation/load/cache hit/miss signals++All labels are intentionally low-cardinality:+- operation/status/provider/key_version are expected to be stable and bounded.+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector++from .base import EventMetricCollector+++@dataclass+class EncryptionCollector(EventMetricCollector):+ """+ Translate encryption lifecycle events into low-cardinality operational metrics.++ The collector receives normalized events from crypto instrumentation and expands them into+ counters and histograms that describe operation throughput, latency, byte volume, and+ key-management health signals without exposing high-cardinality payload data.+ """++ DOMAIN: ClassVar[str] = "encryption"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_operations_total+ # e.g.: openviking_encryption_operations_total+ OPERATIONS_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "operations", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_duration_seconds+ # e.g.: openviking_encryption_duration_seconds+ DURATION_SECONDS: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "duration", unit="seconds"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_bytes_total+ # e.g.: openviking_encryption_bytes_total+ BYTES_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "bytes", unit="total")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_payload_size_bytes+ # e.g.: openviking_encryption_payload_size_bytes+ PAYLOAD_SIZE_BYTES: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "payload_size", unit="bytes"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_auth_failed_total+ # e.g.: openviking_encryption_auth_failed_total+ AUTH_FAILED_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "auth_failed", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_key_derivation_total+ # e.g.: openviking_encryption_key_derivation_total+ KEY_DERIVATION_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "key_derivation", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_key_derivation_duration_seconds+ # e.g.: openviking_encryption_key_derivation_duration_seconds+ KEY_DERIVATION_DURATION_SECONDS: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "key_derivation_duration", unit="seconds"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_key_load_duration_seconds+ # e.g.: openviking_encryption_key_load_duration_seconds+ KEY_LOAD_DURATION_SECONDS: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "key_load_duration", unit="seconds"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_key_cache_hits_total+ # e.g.: openviking_encryption_key_cache_hits_total+ KEY_CACHE_HITS_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "key_cache_hits", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_key_cache_misses_total+ # e.g.: openviking_encryption_key_cache_misses_total+ KEY_CACHE_MISSES_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "key_cache_misses", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_key_version_usage_total+ # e.g.: openviking_encryption_key_version_usage_total+ KEY_VERSION_USAGE_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "key_version_usage", unit="total"+ )++ SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset(+ {+ "encryption.operation",+ "encryption.bytes",+ "encryption.payload_size",+ "encryption.auth_failed",+ "encryption.key_derivation",+ "encryption.key_load",+ "encryption.key_cache_hit",+ "encryption.key_cache_miss",+ "encryption.key_version_usage",+ }+ )++ def collect(self, registry=None) -> None:+ """Implement the collector interface as a no-op because encryption metrics are event-driven."""+ return None++ def receive_hook(self, event_name: str, payload: dict, registry) -> None:+ """+ Dispatch one normalized encryption event to the matching metric writer helper.++ Each supported event has a fixed payload shape defined by the datasource layer, so this+ hook only performs lightweight coercion before updating the registry.+ """+ if event_name == "encryption.operation":+ self.record_operation(+ registry,+ operation=str(payload["operation"]),+ status=str(payload["status"]),+ duration_seconds=float(payload["duration_seconds"]),+ )+ return+ if event_name == "encryption.bytes":+ self.record_bytes(+ registry,+ operation=str(payload["operation"]),+ size_bytes=int(payload["size_bytes"]),+ )+ return+ if event_name == "encryption.payload_size":+ self.record_payload_size(+ registry,+ operation=str(payload["operation"]),+ size_bytes=int(payload["size_bytes"]),+ )+ return+ if event_name == "encryption.auth_failed":+ self.record_auth_failed(registry)+ return+ if event_name == "encryption.key_derivation":+ self.record_key_derivation(+ registry,+ status=str(payload["status"]),+ duration_seconds=float(payload["duration_seconds"]),+ )+ return+ if event_name == "encryption.key_load":+ self.record_key_load(+ registry,+ status=str(payload["status"]),+ provider=str(payload["provider"]),+ duration_seconds=float(payload["duration_seconds"]),+ )+ return+ if event_name == "encryption.key_cache_hit":+ self.record_key_cache_hit(registry, provider=str(payload["provider"]))+ return+ if event_name == "encryption.key_cache_miss":+ self.record_key_cache_miss(registry, provider=str(payload["provider"]))+ return+ if event_name == "encryption.key_version_usage":+ self.record_key_version_usage(+ registry,+ key_version=str(payload["key_version"]),+ )++ def record_operation(+ self, registry, *, operation: str, status: str, duration_seconds: float+ ) -> None:+ """+ Record one encryption or decryption attempt and its observed latency.++ Both the counter and histogram share the same `(operation, status)` label set so success+ and error paths can be aggregated consistently in PromQL.+ """+ labels = {"operation": str(operation), "status": str(status)}+ registry.inc_counter(+ self.OPERATIONS_TOTAL,+ labels=labels,+ label_names=("operation", "status"),+ )+ registry.observe_histogram(+ self.DURATION_SECONDS,+ float(duration_seconds),+ labels=labels,+ label_names=("operation", "status"),+ )++ def record_bytes(self, registry, *, operation: str, size_bytes: int) -> None:+ """Increment the processed-bytes counter when the observed payload size is positive."""+ if size_bytes <= 0:+ return+ registry.inc_counter(+ self.BYTES_TOTAL,+ labels={"operation": str(operation)},+ label_names=("operation",),+ amount=int(size_bytes),+ )++ def record_payload_size(self, registry, *, operation: str, size_bytes: int) -> None:+ """Observe the payload-size distribution for one logical encryption operation."""+ if size_bytes < 0:+ return+ registry.observe_histogram(+ self.PAYLOAD_SIZE_BYTES,+ float(size_bytes),+ labels={"operation": str(operation)},+ label_names=("operation",),+ buckets=(64, 256, 1024, 4096, 16384, 65536, 262144, 1048576),+ )++ def record_auth_failed(self, registry) -> None:+ """Increment the authentication-failure counter for failed payload verification."""+ registry.inc_counter(self.AUTH_FAILED_TOTAL)++ def record_key_derivation(self, registry, *, status: str, duration_seconds: float) -> None:+ """Record the outcome and latency of one key-derivation attempt."""+ registry.inc_counter(+ self.KEY_DERIVATION_TOTAL,+ labels={"status": str(status)},+ label_names=("status",),+ )+ registry.observe_histogram(+ self.KEY_DERIVATION_DURATION_SECONDS,+ float(duration_seconds),+ labels={"status": str(status)},+ label_names=("status",),+ )++ def record_key_load(+ self, registry, *, status: str, provider: str, duration_seconds: float+ ) -> None:+ """Record key-load latency grouped by provider and final status."""+ registry.observe_histogram(+ self.KEY_LOAD_DURATION_SECONDS,+ float(duration_seconds),+ labels={"status": str(status), "provider": str(provider)},+ label_names=("status", "provider"),+ )++ def record_key_cache_hit(self, registry, *, provider: str) -> None:+ """Increment the key-cache hit counter for one normalized provider identifier."""+ registry.inc_counter(+ self.KEY_CACHE_HITS_TOTAL,+ labels={"provider": str(provider)},+ label_names=("provider",),+ )++ def record_key_cache_miss(self, registry, *, provider: str) -> None:+ """Increment the key-cache miss counter for one normalized provider identifier."""+ registry.inc_counter(+ self.KEY_CACHE_MISSES_TOTAL,+ labels={"provider": str(provider)},+ label_names=("provider",),+ )++ def record_key_version_usage(self, registry, *, key_version: str) -> None:+ """Increment the usage counter for the normalized key-version label."""+ registry.inc_counter(+ self.KEY_VERSION_USAGE_TOTAL,+ labels={"key_version": str(key_version)},+ label_names=("key_version",),+ )diff --git a/openviking/metrics/collectors/encryption_probe.py b/openviking/metrics/collectors/encryption_probe.pynew file mode 100644index 000000000..87f1f7583--- /dev/null+++ b/openviking/metrics/collectors/encryption_probe.py@@ -0,0 +1,110 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from dataclasses import dataclass, field+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector+from openviking.metrics.datasources.encryption import EncryptionProbeDataSource++from .base import CollectorConfig, ProbeMetricCollector+++@dataclass+class EncryptionProbeCollector(ProbeMetricCollector):+ """+ Export encryption bootstrap and provider readiness as probe-style gauges.++ Successful refreshes publish `valid="1"` series for overall encryption health, root-key+ availability, and the currently active KMS provider. Failed refreshes preserve the last known+ provider label and re-publish the families with `valid="0"` so stale states remain observable.+ """++ DOMAIN: ClassVar[str] = "encryption"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_component_health+ # e.g.: openviking_encryption_component_health+ COMPONENT_HEALTH: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "component_health")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_root_key_ready+ # e.g.: openviking_encryption_root_key_ready+ ROOT_KEY_READY: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "root_key_ready")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_kms_provider_ready+ # e.g.: openviking_encryption_kms_provider_ready+ KMS_PROVIDER_READY: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "kms_provider_ready")++ data_source: EncryptionProbeDataSource+ config: CollectorConfig = CollectorConfig(ttl_seconds=10.0, timeout_seconds=0.5)+ _last_provider: str = field(default="unknown", init=False, repr=False)++ def read_metric_input(self):+ """Read the current encryption bootstrap readiness tuple from the datasource."""+ return self.data_source.read_probe_state()++ def collect_hook(self, registry, metric_input) -> None:+ """+ Refresh encryption probe gauges.++ On success, gauges are exported with `valid="1"`. On failure, the exporter emits+ `valid="0"` using the last known provider so dashboards can detect staleness.+ """+ ok_component, provider = metric_input+ provider = str(provider)+ self._last_provider = provider+ self.replace_gauge(+ registry,+ self.COMPONENT_HEALTH,+ 1.0 if ok_component else 0.0,+ match_labels={},+ labels={"valid": "1"},+ label_names=("valid",),+ )+ self.replace_gauge(+ registry,+ self.ROOT_KEY_READY,+ 1.0 if ok_component else 0.0,+ match_labels={},+ labels={"valid": "1"},+ label_names=("valid",),+ )+ self.replace_gauge(+ registry,+ self.KMS_PROVIDER_READY,+ 1.0 if ok_component else 0.0,+ match_labels={"provider": provider},+ labels={"provider": provider, "valid": "1"},+ label_names=("provider", "valid"),+ )++ def collect_error_hook(self, registry, error: Exception) -> None:+ """+ Export invalid encryption readiness gauges after a failed probe refresh.++ The collector falls back to the last successful provider label so the stale replacement+ series targets the same Prometheus label set.+ """+ provider = self._last_provider+ self.replace_gauge(+ registry,+ self.COMPONENT_HEALTH,+ 0.0,+ match_labels={},+ labels={"valid": "0"},+ label_names=("valid",),+ )+ self.replace_gauge(+ registry,+ self.ROOT_KEY_READY,+ 0.0,+ match_labels={},+ labels={"valid": "0"},+ label_names=("valid",),+ )+ self.replace_gauge(+ registry,+ self.KMS_PROVIDER_READY,+ 0.0,+ match_labels={"provider": provider},+ labels={"provider": provider, "valid": "0"},+ label_names=("provider", "valid"),+ )diff --git a/openviking/metrics/collectors/http.py b/openviking/metrics/collectors/http.pynew file mode 100644index 000000000..1479ff580--- /dev/null+++ b/openviking/metrics/collectors/http.py@@ -0,0 +1,128 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Event collector: HTTPCollector.++This collector is fed by the HTTP middleware via EventCollectorRouter events:+- http.request: records request count and duration histogram.+- http.inflight: records inflight requests gauge per route template.++Labels are chosen to be stable and low-cardinality:+- route is the Starlette route template when available (e.g., "/sessions/{id}").+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector++from .base import EventMetricCollector+++@dataclass+class HTTPCollector(EventMetricCollector):+ """+ Translate HTTP lifecycle events into request counters, latency histograms, and inflight gauges.++ The collector expects already-normalized route templates and status codes from the middleware+ datasource, then applies account-dimension policy before writing registry series.+ """++ DOMAIN: ClassVar[str] = "http"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_requests_total+ # e.g.: openviking_http_requests_total+ REQUESTS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "requests", unit="total")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_request_duration_seconds+ # e.g.: openviking_http_request_duration_seconds+ REQUEST_DURATION_SECONDS: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "request_duration", unit="seconds"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_inflight_requests+ # e.g.: openviking_http_inflight_requests+ INFLIGHT_REQUESTS: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "inflight_requests")++ SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset({"http.request", "http.inflight"})++ def collect(self, registry=None) -> None:+ """Implement the unified collector interface as a no-op for this event-driven collector."""+ return None++ def receive_hook(self, event_name: str, payload: dict, registry) -> None:+ """+ Translate one supported HTTP lifecycle event into the corresponding metric writes.++ The hook assumes payload validation has already happened in the shared event-collector+ entrypoint and only branches on the normalized event name.+ """+ if event_name == "http.request":+ self.record_request(+ registry,+ method=str(payload["method"]),+ route=str(payload["route"]),+ status=str(payload["status"]),+ duration_seconds=float(payload["duration_seconds"]),+ account_id=(+ None if payload.get("account_id") is None else str(payload.get("account_id"))+ ),+ )+ return+ if event_name == "http.inflight":+ self.record_inflight(+ registry,+ route=str(payload["route"]),+ value=float(payload["value"]),+ account_id=(+ None if payload.get("account_id") is None else str(payload.get("account_id"))+ ),+ )++ def record_request(+ self,+ registry,+ *,+ method: str,+ route: str,+ status: str,+ duration_seconds: float,+ account_id: str | None = None,+ ) -> None:+ """+ Record a completed HTTP request as both a counter increment and a latency sample.++ Counter and histogram labels intentionally share the same route/method/status tuple so+ downstream PromQL can aggregate them consistently.+ """+ labels = {"method": str(method), "route": str(route), "status": str(status)}+ registry.inc_counter(+ self.REQUESTS_TOTAL,+ labels=labels,+ label_names=("method", "route", "status"),+ account_id=account_id,+ )+ registry.observe_histogram(+ self.REQUEST_DURATION_SECONDS,+ float(duration_seconds),+ labels=labels,+ label_names=("method", "route", "status"),+ account_id=account_id,+ )++ def record_inflight(+ self, registry, *, route: str, value: float, account_id: str | None = None+ ) -> None:+ """+ Set the current inflight request gauge value for one normalized route template.++ The caller provides the absolute inflight value, allowing middleware to keep the stateful+ accounting logic outside the collector itself.+ """+ registry.set_gauge(+ self.INFLIGHT_REQUESTS,+ float(value),+ labels={"route": str(route)},+ label_names=("route",),+ account_id=account_id,+ )diff --git a/openviking/metrics/collectors/lock.py b/openviking/metrics/collectors/lock.pynew file mode 100644index 000000000..06af3e149--- /dev/null+++ b/openviking/metrics/collectors/lock.py@@ -0,0 +1,48 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from dataclasses import dataclass+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector+from openviking.metrics.datasources.observer_state import LockStateDataSource++from .base import CollectorConfig, StateMetricCollector+++@dataclass+class LockCollector(StateMetricCollector):+ """+ Translate lock-manager snapshots into lock-related gauge families.++ The datasource returns a point-in-time view of active, waiting, and stale locks. This+ collector mirrors that snapshot directly as gauges because the values represent current state+ rather than cumulative activity.+ """++ DOMAIN: ClassVar[str] = "lock"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_active+ # e.g.: openviking_lock_active+ ACTIVE: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "active")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_waiting+ # e.g.: openviking_lock_waiting+ WAITING: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "waiting")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_stale+ # e.g.: openviking_lock_stale+ STALE: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "stale")++ data_source: LockStateDataSource+ config: CollectorConfig = CollectorConfig(timeout_seconds=0.5)++ def read_metric_input(self):+ """Read the latest lock-manager state counters from the datasource."""+ return self.data_source.read_lock_state()++ def collect_hook(self, registry, metric_input) -> None:+ """Publish the current lock snapshot as active, waiting, and stale gauges."""+ active, waiting, stale = metric_input+ registry.set_gauge(self.ACTIVE, float(active))+ registry.set_gauge(self.WAITING, float(waiting))+ registry.set_gauge(self.STALE, float(stale))diff --git a/openviking/metrics/collectors/manager.py b/openviking/metrics/collectors/manager.pynew file mode 100644index 000000000..5a1f3ed60--- /dev/null+++ b/openviking/metrics/collectors/manager.py@@ -0,0 +1,256 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Collector orchestration for Prometheus exposition.++This module implements the "scrape-triggered collection" workflow:+- `/metrics` export calls `CollectorManager.refresh_all(...)` before rendering.+- Each collector may have TTL-based refresh control (RefreshGate).+- When TTL is expired but we have a previous successful refresh, we apply SWR:+ return immediately (do not block scrape) and refresh in background.++The goal is to keep `/metrics` fast and reliable even when some collectors+depend on slow/unstable subsystems.+"""++from __future__ import annotations++import asyncio+import time+from dataclasses import dataclass+from typing import cast++from openviking.metrics.core.base import MetricCollector+from openviking.metrics.core.refresh import RefreshGate++from .base import Refreshable+++@dataclass(frozen=True, slots=True)+class RefreshResult:+ """+ Execution outcome for a single collector refresh attempt.++ Attributes:+ collector: Collector name associated with this refresh result.+ attempted: Whether the manager actually started a collection attempt.+ success: Whether the attempt or scheduling decision is considered successful.+ reason: Machine-readable outcome such as `ok`, `timeout`, `ttl_valid`, or+ `swr_triggered`.+ """++ collector: str+ attempted: bool+ success: bool+ reason: str+++class CollectorManager:+ """+ Manages a set of collectors and executes collection with timeouts and refresh control.++ - Collectors with `config.ttl_seconds is None` always attempt refresh on each scrape.+ - Collectors with TTL use RefreshGate to avoid repeated refresh within TTL.+ - When TTL is expired and we have a previous successful refresh, `refresh_all`+ triggers a background refresh and returns a `swr_triggered` result immediately.+ """++ def __init__(self) -> None:+ """Initialize an empty collector set, its refresh gates, and tracked background tasks."""+ self._collectors: list[MetricCollector] = []+ self._gates: dict[str, RefreshGate] = {}+ self._background_tasks: set[asyncio.Task[RefreshResult]] = set()++ def register(self, collector: MetricCollector) -> None:+ """+ Register a collector into the refresh pipeline.++ If the collector has `config.ttl_seconds`, a gate is created to enforce TTL + SWR.+ """+ if not isinstance(collector, Refreshable):+ raise TypeError("CollectorManager only accepts Refreshable collectors")+ self._collectors.append(collector)+ collector_name = self._collector_name(collector)+ ttl_seconds = getattr(cast(Refreshable, collector).config, "ttl_seconds", None)+ if ttl_seconds is not None:+ self._gates[collector_name] = RefreshGate(ttl_seconds=float(ttl_seconds))++ async def refresh_all(self, registry, *, deadline_seconds: float = 1.0) -> list[RefreshResult]:+ """+ Refresh all registered collectors under a global deadline.++ Args:+ registry: MetricRegistry instance to be written by collectors.+ deadline_seconds: Total time budget for the entire refresh stage.++ Returns:+ A list of RefreshResult items, including "ttl_valid" / "swr_triggered"+ decisions made by the RefreshGate.+ """+ if not self._collectors:+ return []++ start = time.monotonic()+ results: list[RefreshResult] = []+ tasks: list[asyncio.Task[RefreshResult]] = []++ for c in self._collectors:+ collector_name = self._collector_name(c)+ gate = self._gates.get(collector_name)+ timeout_seconds = float(getattr(cast(Refreshable, c).config, "timeout_seconds", 0.5))+ if gate is None:+ tasks.append(+ asyncio.create_task(+ self._run_collector(c, registry, timeout=timeout_seconds),+ name=collector_name,+ )+ )+ continue++ decision = gate.decide(now=time.monotonic())+ if not decision.do_refresh:+ results.append(+ RefreshResult(+ collector=collector_name,+ attempted=False,+ success=True,+ reason=decision.reason,+ )+ )+ continue+ if not gate.mark_inflight():+ results.append(+ RefreshResult(+ collector=collector_name,+ attempted=False,+ success=True,+ reason="inflight",+ )+ )+ continue+ if decision.reason == "ttl_expired" and decision.last_success_at is not None:+ task = asyncio.create_task(+ self._run_collector_with_gate(c, registry, gate=gate, timeout=timeout_seconds),+ name=collector_name,+ )+ self._track_background(task)+ results.append(+ RefreshResult(+ collector=collector_name,+ attempted=False,+ success=True,+ reason="swr_triggered",+ )+ )+ continue++ tasks.append(+ asyncio.create_task(+ self._run_collector_with_gate(c, registry, gate=gate, timeout=timeout_seconds),+ name=collector_name,+ )+ )++ if not tasks:+ return results++ remaining = max(0.0, deadline_seconds - (time.monotonic() - start))+ done, pending = await asyncio.wait(tasks, timeout=remaining)+ for t in done:+ results.append(t.result())+ for t in pending:+ # Deadline means we stop waiting for refresh; we do NOT cancel because collectors run+ # in a thread and cancellation would not stop that work anyway. Keeping the task alive+ # avoids leaving gates stuck in inflight state while also preventing overlapping+ # refresh attempts across scrapes.+ self._track_background(t)+ results.append(+ RefreshResult(+ collector=t.get_name() or "unknown",+ attempted=True,+ success=False,+ reason="deadline_exceeded",+ )+ )+ return results++ def _track_background(self, task: asyncio.Task[RefreshResult]) -> None:+ """+ Track a background refresh task until it completes.++ The task set exists only to keep background SWR refreshes strongly referenced and to+ allow their lifecycle to be cleaned up automatically on completion.+ """+ self._background_tasks.add(task)++ def _done(t: asyncio.Task[RefreshResult]) -> None:+ """Remove a completed background task from the tracked task set."""+ self._background_tasks.discard(t)++ task.add_done_callback(_done)++ async def _run_collector_with_gate(+ self, collector: MetricCollector, registry, *, gate: RefreshGate, timeout: float+ ) -> RefreshResult:+ """+ Run one collector while ensuring the associated refresh gate is completed correctly.++ The gate is marked complete on both success and failure so future refresh decisions do+ not remain stuck in the inflight state.+ """+ success = False+ try:+ res = await self._run_collector(collector, registry, timeout=timeout)+ success = bool(res.success)+ return res+ finally:+ gate.mark_complete(success=success)++ async def _run_collector(+ self, collector: MetricCollector, registry, *, timeout: float+ ) -> RefreshResult:+ """+ Execute one collector's `collect(...)` call under a timeout in a worker thread.++ Returns:+ A `RefreshResult` describing the collector outcome without propagating exceptions to+ the caller.+ """+ try:+ await asyncio.wait_for(asyncio.to_thread(collector.collect, registry), timeout=timeout)+ return RefreshResult(+ collector=self._collector_name(collector),+ attempted=True,+ success=True,+ reason="ok",+ )+ except asyncio.TimeoutError:+ return RefreshResult(+ collector=self._collector_name(collector),+ attempted=True,+ success=False,+ reason="timeout",+ )+ except Exception:+ return RefreshResult(+ collector=self._collector_name(collector),+ attempted=True,+ success=False,+ reason="exception",+ )++ def _collector_name(self, collector: MetricCollector) -> str:+ """+ Resolve the display name used for refresh results and refresh-gate bookkeeping.++ The manager prefers the collector's explicit `collector_name()` method and falls back to+ a `name` attribute or the concrete class name for compatibility.+ """+ try:+ return str(collector.collector_name())+ except AttributeError:+ name = getattr(collector, "name", None)+ if name is not None:+ return str(name)+ return collector.__class__.__name__diff --git a/openviking/metrics/collectors/model_provider_probe.py b/openviking/metrics/collectors/model_provider_probe.pynew file mode 100644index 000000000..5ed71392e--- /dev/null+++ b/openviking/metrics/collectors/model_provider_probe.py@@ -0,0 +1,70 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from dataclasses import dataclass, field+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector+from openviking.metrics.datasources.probes import ModelProviderProbeDataSource++from .base import CollectorConfig, ProbeMetricCollector+++@dataclass+class ModelProviderProbeCollector(ProbeMetricCollector):+ """+ Export model-provider readiness for the currently selected backend provider.++ The datasource reports which provider is active and whether it is currently healthy. The+ collector keeps the last successful provider label so failures can still emit a stale+ `valid="0"` replacement series for the same provider dimension.+ """++ DOMAIN: ClassVar[str] = "model_provider"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_readiness+ # e.g.: openviking_model_provider_readiness+ READINESS: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "readiness")++ data_source: ModelProviderProbeDataSource+ config: CollectorConfig = CollectorConfig(ttl_seconds=10.0, timeout_seconds=0.5)+ _last_provider: str = field(default="unknown", init=False, repr=False)++ def read_metric_input(self):+ """Read the latest model-provider readiness snapshot from the datasource."""+ return self.data_source.read_probe_state()++ def collect_hook(self, registry, metric_input) -> None:+ """+ Refresh model provider probe gauges.++ On success, the current provider readiness is exported with `valid="1"`. On failure,+ the last known provider is exported with `valid="0"` so monitoring can detect staleness.+ """+ state = metric_input+ provider, ok = state.get("provider", ("unknown", False))+ provider = str(provider)+ self._last_provider = provider+ base = {"provider": provider}+ self.replace_gauge(+ registry,+ self.READINESS,+ 1.0 if ok else 0.0,+ match_labels=base,+ labels={"provider": provider, "valid": "1"},+ label_names=("provider", "valid"),+ )++ def collect_error_hook(self, registry, error: Exception) -> None:+ """Export invalid model-provider readiness using the last known provider label."""+ provider = self._last_provider+ base = {"provider": provider}+ self.replace_gauge(+ registry,+ self.READINESS,+ 0.0,+ match_labels=base,+ labels={"provider": provider, "valid": "0"},+ label_names=("provider", "valid"),+ )diff --git a/openviking/metrics/collectors/model_usage.py b/openviking/metrics/collectors/model_usage.pynew file mode 100644index 000000000..b21fc1d45--- /dev/null+++ b/openviking/metrics/collectors/model_usage.py@@ -0,0 +1,196 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+DomainStats collector: ModelUsageCollector.++This collector exports aggregated model usage in a Prometheus-friendly way:+- Input source is cumulative usage (calls/tokens) from model instances or shared token trackers.+- Output is Prometheus Counters, which must be monotonic.++To bridge cumulative -> monotonic, we store the last seen cumulative values in memory and+increment Prometheus Counters by deltas on each refresh.++Failure semantics:+- For each model_type (vlm/embedding/rerank), we export an availability gauge with a `valid`+ label describing whether the sample is fresh (`valid="1"`) or stale fallback (`valid="0"`).+"""++from __future__ import annotations++from dataclasses import dataclass, field+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector+from openviking.metrics.datasources.model_usage import ModelUsageDataSource++from .base import CollectorConfig, DomainStatsMetricCollector+++@dataclass+class ModelUsageCollector(DomainStatsMetricCollector):+ """+ Export aggregated model-usage snapshots as availability gauges and monotonic counters.++ The datasource provides cumulative usage grouped by model type, model name, and provider. This+ collector translates those cumulative values into Prometheus counters by storing the last seen+ totals and incrementing only the positive delta for each label tuple. Availability is exported+ separately so dashboards can distinguish "usage unavailable" from "stale fallback sample".+ """++ DOMAIN_MODEL_USAGE: ClassVar[str] = "model_usage"+ DOMAIN_MODEL: ClassVar[str] = "model"+ # rule: <METRICS_NAMESPACE>_<DOMAIN_MODEL_USAGE>_available+ # e.g.: openviking_model_usage_available+ USAGE_AVAILABLE: ClassVar[str] = MetricCollector.metric_name(DOMAIN_MODEL_USAGE, "available")+ # rule: <METRICS_NAMESPACE>_<DOMAIN_MODEL>_calls_total+ # e.g.: openviking_model_calls_total+ CALLS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN_MODEL, "calls", unit="total")+ # rule: <METRICS_NAMESPACE>_<DOMAIN_MODEL>_tokens_total+ # e.g.: openviking_model_tokens_total+ TOKENS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN_MODEL, "tokens", unit="total")++ data_source: ModelUsageDataSource+ config: CollectorConfig = CollectorConfig(ttl_seconds=10.0, timeout_seconds=0.8)+ _last_available_by_type: dict[str, float] = field(+ default_factory=lambda: {"vlm": 0.0, "embedding": 0.0, "rerank": 0.0},+ init=False,+ repr=False,+ )++ def read_metric_input(self):+ """Read the aggregated model-usage snapshot from the datasource."""+ return self.data_source.read_model_usage()++ def collect_hook(self, registry, metric_input) -> None:+ """+ Refresh model usage metrics from one fresh aggregated snapshot.++ For counters, we apply deltas based on the last seen cumulative usage.+ If the cumulative value decreases (e.g., process restart / reset), we treat the+ current cumulative value as the delta to keep exported counters monotonic.++ Availability is exported as `openviking_model_usage_available`, where the metric value+ describes whether usage data is currently available and `valid="1"` marks the sample as+ fresh.+ """+ usage = metric_input+ for model_type in ("vlm", "embedding", "rerank"):+ per_type = None if not isinstance(usage, dict) else usage.get(model_type)+ available = 0.0+ usage_by_model = {}+ if isinstance(per_type, dict):+ available = 1.0 if bool(per_type.get("available")) else 0.0+ raw_usage_by_model = per_type.get("usage_by_model") or {}+ if isinstance(raw_usage_by_model, dict):+ usage_by_model = raw_usage_by_model++ self._last_available_by_type[model_type] = available+ self._write_availability(+ registry,+ model_type=model_type,+ available=available,+ valid="1",+ )++ for model_name, model_usage in usage_by_model.items():+ if not isinstance(model_usage, dict):+ continue+ usage_by_provider = model_usage.get("usage_by_provider") or {}+ if not isinstance(usage_by_provider, dict):+ continue+ for provider, provider_usage in usage_by_provider.items():+ if not isinstance(provider_usage, dict):+ continue+ self._apply_provider_usage(+ registry,+ model_type=str(model_type),+ provider=str(provider),+ model_name=str(model_name),+ provider_usage=provider_usage,+ )++ def collect_error_hook(self, registry, error: Exception) -> None:+ """+ Re-emit the last known availability state as stale fallback samples.++ Failed refreshes downgrade the availability samples to `valid="0"` but do not mutate the+ monotonic usage counters, because no fresh cumulative snapshot was obtained.+ """+ for model_type in ("vlm", "embedding", "rerank"):+ self._write_availability(+ registry,+ model_type=model_type,+ available=float(self._last_available_by_type.get(model_type, 0.0)),+ valid="0",+ )++ def _write_availability(+ self,+ registry,+ *,+ model_type: str,+ available: float,+ valid: str,+ ) -> None:+ """+ Replace the availability gauge series for one model type with a fresh validity label.++ The replacement deletes any prior series for the same `model_type`, ensuring only one of+ `valid="1"` or `valid="0"` remains visible at a time.+ """+ registry.gauge_delete_matching(+ self.USAGE_AVAILABLE,+ match_labels={"model_type": str(model_type)},+ )+ registry.set_gauge(+ self.USAGE_AVAILABLE,+ float(available),+ labels={"model_type": str(model_type), "valid": str(valid)},+ label_names=("model_type", "valid"),+ )++ def _apply_provider_usage(+ self,+ registry,+ *,+ model_type: str,+ provider: str,+ model_name: str,+ provider_usage: dict,+ ) -> None:+ """+ Apply one provider-specific usage snapshot as counter deltas.++ The helper keeps counter monotonicity local to one `(model_type, provider, model_name)`+ tuple and then repeats the same delta conversion for each token subtype.+ """+ call_count = int(provider_usage.get("call_count", 0) or 0)+ self.inc_counter_from_cumulative(+ registry=registry,+ metric_name=self.CALLS_TOTAL,+ key=(self.CALLS_TOTAL, model_type, provider, model_name),+ current_value=call_count,+ labels={"model_type": model_type, "provider": provider, "model_name": model_name},+ label_names=("model_name", "model_type", "provider"),+ )++ tokens = {+ "prompt": int(provider_usage.get("prompt_tokens", 0) or 0),+ "completion": int(provider_usage.get("completion_tokens", 0) or 0),+ "total": int(provider_usage.get("total_tokens", 0) or 0),+ }+ for token_type, value in tokens.items():+ self.inc_counter_from_cumulative(+ registry=registry,+ metric_name=self.TOKENS_TOTAL,+ key=(self.TOKENS_TOTAL, model_type, provider, model_name, token_type),+ current_value=value,+ labels={+ "model_type": model_type,+ "provider": provider,+ "model_name": model_name,+ "token_type": token_type,+ },+ label_names=("model_name", "model_type", "provider", "token_type"),+ )diff --git a/openviking/metrics/collectors/observer_health.py b/openviking/metrics/collectors/observer_health.pynew file mode 100644index 000000000..281289ad5--- /dev/null+++ b/openviking/metrics/collectors/observer_health.py@@ -0,0 +1,117 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from dataclasses import dataclass, field+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector+from openviking.metrics.datasources.observer_state import ObserverStateDataSource++from .base import CollectorConfig, StateMetricCollector+++@dataclass+class ObserverHealthCollector(StateMetricCollector):+ """+ Export per-component observer health and error gauges with stale-on-error semantics.++ Unlike the summary-oriented observer collector, this class preserves one series per component+ so dashboards can pinpoint which subsystem degraded. Failed refreshes re-publish the last known+ component set with `valid="0"` instead of dropping the series entirely.+ """++ STALE_ON_ERROR: ClassVar[bool] = True++ DOMAIN: ClassVar[str] = "component"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_health+ # e.g.: openviking_component_health+ HEALTH: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "health")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_errors+ # e.g.: openviking_component_errors+ ERRORS: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "errors")++ data_source: ObserverStateDataSource+ config: CollectorConfig = CollectorConfig(ttl_seconds=10.0, timeout_seconds=0.8)+ _known_components: list[str] = field(+ default_factory=lambda: ["queue", "models", "lock", "retrieval", "vikingdb"],+ init=False,+ repr=False,+ )++ def read_metric_input(self):+ """Read the latest observer component-state mapping from the datasource."""+ return self.data_source.read_component_states()++ def collect_hook(self, registry, metric_input) -> None:+ """+ Refresh component health metrics from the observer state datasource.++ On success, component health and error flags are exported with `valid="1"`. On failure,+ the last known component list is exported with `valid="0"` while preserving last values+ when present.+ """+ states = metric_input+ components = [str(k) for k in states.keys()]+ if components:+ self._known_components = components+ for component, state in states.items():+ if state is None:+ continue+ base = {"component": str(component)}+ labels = {"component": str(component), "valid": "1"}+ self.replace_gauge_series(+ registry,+ self.HEALTH,+ 1.0 if getattr(state, "is_healthy", False) else 0.0,+ match_labels=base,+ labels=labels,+ label_names=("component", "valid"),+ )+ self.replace_gauge_series(+ registry,+ self.ERRORS,+ 1.0 if getattr(state, "has_errors", False) else 0.0,+ match_labels=base,+ labels=labels,+ label_names=("component", "valid"),+ )++ def collect_error_hook(self, registry, error: Exception) -> None:+ """Delegate failure handling to the stale hook by re-raising the datasource error."""+ raise error++ def collect_stale_hook(self, registry, error: Exception) -> None:+ """Export stale observer component health values under `valid=0` on refresh failure."""+ for component in self._known_components:+ base = {"component": str(component)}+ last_health = registry.gauge_get(+ self.HEALTH,+ labels={"component": str(component), "valid": "1"},+ )+ last_errors = registry.gauge_get(+ self.ERRORS,+ labels={"component": str(component), "valid": "1"},+ )+ if last_health is None:+ last_health = 0.0+ if last_errors is None:+ last_errors = 0.0+ labels = {"component": str(component), "valid": "0"}+ self.replace_gauge_series(+ registry,+ self.HEALTH,+ float(last_health),+ match_labels=base,+ labels=labels,+ label_names=("component", "valid"),+ )+ self.replace_gauge_series(+ registry,+ self.ERRORS,+ float(last_errors),+ match_labels=base,+ labels=labels,+ label_names=("component", "valid"),+ )diff --git a/openviking/metrics/collectors/observer_state.py b/openviking/metrics/collectors/observer_state.pynew file mode 100644index 000000000..bf0bc8ae9--- /dev/null+++ b/openviking/metrics/collectors/observer_state.py@@ -0,0 +1,112 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+DomainStats collector: ObserverStateCollector.++This collector converts the observer's component status table into a small set of+low-cardinality gauges, suitable for dashboards and alerting.++Why not export the full debug status table?+- The debug output is verbose and not stable as a Prometheus label/value.+- Prometheus works best with bounded series counts and stable labels.++Failure semantics:+- On refresh success: export values with `valid="1"`.+- On refresh failure: export the last successful values with `valid="0"` (SWR-style),+ so `/metrics` stays available and alerting can use `valid` to detect staleness.+"""++from __future__ import annotations++from dataclasses import dataclass, field+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector+from openviking.metrics.datasources.observer_state import ObserverStateDataSource++from .base import CollectorConfig, DomainStatsMetricCollector+++@dataclass+class ObserverStateCollector(DomainStatsMetricCollector):+ """+ Summarize observer component states into three low-cardinality gauges.++ This collector intentionally collapses the detailed component table into global counts so+ high-level dashboards can alert on observer degradation without depending on per-component+ labels. A `valid` label distinguishes fresh snapshots from stale fallback values.+ """++ DOMAIN: ClassVar[str] = "observer"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_components_total+ # e.g.: openviking_observer_components_total+ COMPONENTS_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "components", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_components_unhealthy+ # e.g.: openviking_observer_components_unhealthy+ COMPONENTS_UNHEALTHY: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "components_unhealthy"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_components_with_errors+ # e.g.: openviking_observer_components_with_errors+ COMPONENTS_WITH_ERRORS: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "components_with_errors"+ )++ data_source: ObserverStateDataSource+ config: CollectorConfig = CollectorConfig(ttl_seconds=10.0, timeout_seconds=0.8)+ _last_total: float = field(default=0.0, init=False, repr=False)+ _last_unhealthy: float = field(default=0.0, init=False, repr=False)+ _last_with_errors: float = field(default=0.0, init=False, repr=False)++ def read_metric_input(self):+ """Read the latest observer component-state mapping from the datasource."""+ return self.data_source.read_component_states()++ def collect_hook(self, registry, metric_input) -> None:+ """Refresh observer summary gauges using a successfully-read component state mapping."""+ states = metric_input+ total = 0.0+ unhealthy = 0.0+ with_errors = 0.0+ for _, state in states.items():+ if state is None:+ continue+ total += 1.0+ if not bool(getattr(state, "is_healthy", False)):+ unhealthy += 1.0+ if bool(getattr(state, "has_errors", False)):+ with_errors += 1.0++ self._last_total = total+ self._last_unhealthy = unhealthy+ self._last_with_errors = with_errors+ self._write(registry, valid="1", total=total, unhealthy=unhealthy, with_errors=with_errors)++ def collect_error_hook(self, registry, error: Exception) -> None:+ """Re-publish the last observer summary under `valid=0` when refresh fails."""+ self._write(+ registry,+ valid="0",+ total=self._last_total,+ unhealthy=self._last_unhealthy,+ with_errors=self._last_with_errors,+ )++ def _write(+ self, registry, *, valid: str, total: float, unhealthy: float, with_errors: float+ ) -> None:+ """Replace the observer summary families with one consistent validity-labelled snapshot."""+ self.replace_global_gauges(+ registry,+ metric_names=[+ self.COMPONENTS_TOTAL,+ self.COMPONENTS_UNHEALTHY,+ self.COMPONENTS_WITH_ERRORS,+ ],+ values=[total, unhealthy, with_errors],+ labels={"valid": str(valid)},+ label_names=("valid",),+ )diff --git a/openviking/metrics/collectors/queue.py b/openviking/metrics/collectors/queue.pynew file mode 100644index 000000000..b819dbf07--- /dev/null+++ b/openviking/metrics/collectors/queue.py@@ -0,0 +1,96 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from dataclasses import dataclass, field+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector+from openviking.metrics.datasources.queue import QueuePipelineStateDataSource++from .base import CollectorConfig, StateMetricCollector+++@dataclass+class QueueCollector(StateMetricCollector):+ """+ Export queue pipeline depth gauges and monotonic throughput/error counters.++ Pending and in-progress counts represent instantaneous queue state, while processed and error+ totals arrive from the datasource as cumulative counters that must be converted into positive+ Prometheus increments.+ """++ DOMAIN: ClassVar[str] = "queue"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_pending+ # e.g.: openviking_queue_pending+ PENDING: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "pending")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_in_progress+ # e.g.: openviking_queue_in_progress+ IN_PROGRESS: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "in_progress")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_processed_total+ # e.g.: openviking_queue_processed_total+ PROCESSED_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "processed", unit="total")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_errors_total+ # e.g.: openviking_queue_errors_total+ ERRORS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "errors", unit="total")++ data_source: QueuePipelineStateDataSource+ config: CollectorConfig = CollectorConfig(timeout_seconds=0.5)+ _last_processed: dict[str, int] = field(default_factory=dict)+ _last_errors: dict[str, int] = field(default_factory=dict)++ def read_metric_input(self):+ """Read the latest queue pipeline status snapshot from the datasource."""+ return self.data_source.read_queue_status()++ def collect_hook(self, registry, metric_input) -> None:+ """+ Refresh queue gauges and apply deltas to monotonic queue counters.++ The datasource provides cumulative `processed` and `error_count` values. This collector+ converts them into Prometheus counters by incrementing by positive deltas and ignoring+ decreases.+ """+ statuses = metric_input+ for queue_name, status in statuses.items():+ labels = {"queue": str(queue_name)}+ registry.set_gauge(+ self.PENDING,+ float(status.pending),+ labels=labels,+ label_names=("queue",),+ )+ registry.set_gauge(+ self.IN_PROGRESS,+ float(status.in_progress),+ labels=labels,+ label_names=("queue",),+ )++ processed = int(status.processed)+ prev_processed = int(self._last_processed.get(queue_name, 0))+ if processed >= prev_processed:+ delta = processed - prev_processed+ if delta:+ registry.inc_counter(+ self.PROCESSED_TOTAL,+ labels=labels,+ label_names=("queue",),+ amount=delta,+ )+ self._last_processed[queue_name] = processed++ errors = int(status.error_count)+ prev_errors = int(self._last_errors.get(queue_name, 0))+ if errors >= prev_errors:+ delta = errors - prev_errors+ if delta:+ registry.inc_counter(+ self.ERRORS_TOTAL,+ labels=labels,+ label_names=("queue",),+ amount=delta,+ )+ self._last_errors[queue_name] = errorsdiff --git a/openviking/metrics/collectors/resource.py b/openviking/metrics/collectors/resource.pynew file mode 100644index 000000000..f2594b6cf--- /dev/null+++ b/openviking/metrics/collectors/resource.py@@ -0,0 +1,115 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Event collector: ResourceIngestionCollector.++Exports stage-level metrics for resource ingestion and processing:+- stage counters by stage/status+- stage duration histogram+- wait duration histogram (operation-level)++This collector is fed by ResourceIngestionEventDataSource, which emits events from the+resource processing pipeline.+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector++from .base import EventMetricCollector+++@dataclass+class ResourceIngestionCollector(EventMetricCollector):+ """+ Translate resource-ingestion pipeline events into counters and latency histograms.++ The collector records both stage-level outcomes and queue wait time so the resource ingestion+ path can be observed from enqueue delay through each named processing stage.+ """++ DOMAIN: ClassVar[str] = "resource"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_stage_total+ # e.g.: openviking_resource_stage_total+ STAGE_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "stage", unit="total")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_stage_duration_seconds+ # e.g.: openviking_resource_stage_duration_seconds+ STAGE_DURATION_SECONDS: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "stage_duration", unit="seconds"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_wait_duration_seconds+ # e.g.: openviking_resource_wait_duration_seconds+ WAIT_DURATION_SECONDS: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "wait_duration", unit="seconds"+ )++ SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset({"resource.stage", "resource.wait"})++ def collect(self, registry=None) -> None:+ """Implement the collector interface as a no-op because resource metrics are push-driven."""+ return None++ def receive_hook(self, event_name: str, payload: dict, registry) -> None:+ """+ Dispatch one normalized resource-ingestion event to the matching metric writer.++ `resource.stage` emits both a counter increment and duration sample, while `resource.wait`+ emits only a wait-duration histogram observation.+ """+ if event_name == "resource.stage":+ self.record_stage(+ registry,+ stage=str(payload["stage"]),+ status=str(payload["status"]),+ duration_seconds=float(payload["duration_seconds"]),+ account_id=payload.get("account_id"),+ )+ return+ if event_name == "resource.wait":+ self.record_wait(+ registry,+ operation=str(payload["operation"]),+ duration_seconds=float(payload["duration_seconds"]),+ account_id=payload.get("account_id"),+ )++ def record_stage(+ self,+ registry,+ *,+ stage: str,+ status: str,+ duration_seconds: float,+ account_id: str | None = None,+ ) -> None:+ """Record one named resource stage outcome together with its execution latency."""+ labels = {"stage": str(stage), "status": str(status)}+ registry.inc_counter(+ self.STAGE_TOTAL,+ labels=labels,+ label_names=("stage", "status"),+ account_id=None if account_id is None else str(account_id),+ )+ registry.observe_histogram(+ self.STAGE_DURATION_SECONDS,+ float(duration_seconds),+ labels=labels,+ label_names=("stage", "status"),+ account_id=None if account_id is None else str(account_id),+ )++ def record_wait(+ self, registry, *, operation: str, duration_seconds: float, account_id: str | None = None+ ) -> None:+ """Record one wait-duration sample for a resource-ingestion operation."""+ registry.observe_histogram(+ self.WAIT_DURATION_SECONDS,+ float(duration_seconds),+ labels={"operation": str(operation)},+ label_names=("operation",),+ account_id=None if account_id is None else str(account_id),+ )diff --git a/openviking/metrics/collectors/retrieval.py b/openviking/metrics/collectors/retrieval.pynew file mode 100644index 000000000..983a7f1d7--- /dev/null+++ b/openviking/metrics/collectors/retrieval.py@@ -0,0 +1,127 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Event/DomainStats collector: RetrievalCollector.++This collector records retrieval outcomes:+- request count, results count, zero-result count+- latency histogram+- rerank usage/fallback counts++It is fed by `RetrievalStatsDataSource.record_retrieval(...)` which emits a single+`retrieval.completed` event per retrieval operation.+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector++from .base import EventMetricCollector+++@dataclass+class RetrievalCollector(EventMetricCollector):+ """+ Translate retrieval completion events into throughput, result-volume, and latency metrics.++ Each completed retrieval emits a single normalized event summarizing the request. The collector+ expands that event into bounded counters and one latency histogram sample keyed by+ `context_type`.+ """++ DOMAIN: ClassVar[str] = "retrieval"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_requests_total+ # e.g.: openviking_retrieval_requests_total+ REQUESTS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "requests", unit="total")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_results_total+ # e.g.: openviking_retrieval_results_total+ RESULTS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "results", unit="total")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_zero_result_total+ # e.g.: openviking_retrieval_zero_result_total+ ZERO_RESULT_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "zero_result", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_latency_seconds+ # e.g.: openviking_retrieval_latency_seconds+ LATENCY_SECONDS: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "latency", unit="seconds")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_rerank_used_total+ # e.g.: openviking_retrieval_rerank_used_total+ RERANK_USED_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "rerank_used", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_rerank_fallback_total+ # e.g.: openviking_retrieval_rerank_fallback_total+ RERANK_FALLBACK_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "rerank_fallback", unit="total"+ )++ SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset({"retrieval.completed"})++ def collect(self, registry=None) -> None:+ """Implement the collector interface as a no-op because retrieval metrics are event-driven."""+ return None++ def receive_hook(self, event_name: str, payload: dict, registry) -> None:+ """+ Translate the retrieval completion payload into retrieval outcome metric updates.++ The event has already been normalized by the datasource, so the hook only coerces field+ types before delegating to `record_completed`.+ """+ self.record_completed(+ registry,+ context_type=str(payload["context_type"]),+ result_count=int(payload["result_count"]),+ latency_seconds=float(payload["latency_seconds"]),+ rerank_used=bool(payload.get("rerank_used")),+ rerank_fallback=bool(payload.get("rerank_fallback")),+ )++ def record_completed(+ self,+ registry,+ *,+ context_type: str,+ result_count: int,+ latency_seconds: float,+ rerank_used: bool,+ rerank_fallback: bool,+ ) -> None:+ """+ Record counters and latency for one completed retrieval operation.++ Result counts are clamped to non-negative counter increments, while rerank usage and+ rerank fallback are tracked as separate global counters for coarse operational visibility.+ """+ labels = {"context_type": str(context_type or "unknown")}+ registry.inc_counter(+ self.REQUESTS_TOTAL,+ labels=labels,+ label_names=("context_type",),+ )+ registry.inc_counter(+ self.RESULTS_TOTAL,+ labels=labels,+ label_names=("context_type",),+ amount=max(0, int(result_count)),+ )+ if int(result_count) == 0:+ registry.inc_counter(+ self.ZERO_RESULT_TOTAL,+ labels=labels,+ label_names=("context_type",),+ )+ registry.observe_histogram(+ self.LATENCY_SECONDS,+ float(latency_seconds),+ labels=labels,+ label_names=("context_type",),+ )+ if rerank_used:+ registry.inc_counter(self.RERANK_USED_TOTAL)+ if rerank_fallback:+ registry.inc_counter(self.RERANK_FALLBACK_TOTAL)diff --git a/openviking/metrics/collectors/retrieval_backend_probe.py b/openviking/metrics/collectors/retrieval_backend_probe.pynew file mode 100644index 000000000..731e20334--- /dev/null+++ b/openviking/metrics/collectors/retrieval_backend_probe.py@@ -0,0 +1,71 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from dataclasses import dataclass, field+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector+from openviking.metrics.datasources.probes import RetrievalBackendProbeDataSource++from .base import CollectorConfig, ProbeMetricCollector+++@dataclass+class RetrievalBackendProbeCollector(ProbeMetricCollector):+ """+ Export readiness for each retrieval backend probe with explicit stale semantics.++ Successful refreshes publish one `valid="1"` gauge per probe name. If the datasource fails,+ the collector preserves the last observed probe set and republishes those series with+ `valid="0"` so monitoring can distinguish stale data from healthy zeros.+ """++ DOMAIN: ClassVar[str] = "retrieval_backend"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_readiness+ # e.g.: openviking_retrieval_backend_readiness+ READINESS: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "readiness")++ data_source: RetrievalBackendProbeDataSource+ config: CollectorConfig = CollectorConfig(ttl_seconds=5.0, timeout_seconds=0.8)+ _known_probes: list[str] = field(default_factory=lambda: ["vikingdb"], init=False, repr=False)++ def read_metric_input(self):+ """Read the latest retrieval-backend probe mapping from the datasource."""+ return self.data_source.read_probe_state()++ def collect_hook(self, registry, metric_input) -> None:+ """+ Refresh retrieval backend probe gauges.++ On success, each probe is exported with `valid="1"`. On failure, the last known probe+ set is exported with `valid="0"` so monitoring can detect staleness.+ """+ state = metric_input+ probes = [str(k) for k in state.keys()]+ if probes:+ self._known_probes = probes+ for probe, ok in state.items():+ base = {"probe": str(probe)}+ self.replace_gauge(+ registry,+ self.READINESS,+ 1.0 if ok else 0.0,+ match_labels=base,+ labels={"probe": str(probe), "valid": "1"},+ label_names=("probe", "valid"),+ )++ def collect_error_hook(self, registry, error: Exception) -> None:+ """Export invalid retrieval backend readiness gauges for the last known probes."""+ for probe in self._known_probes:+ base = {"probe": str(probe)}+ self.replace_gauge(+ registry,+ self.READINESS,+ 0.0,+ match_labels=base,+ labels={"probe": str(probe), "valid": "0"},+ label_names=("probe", "valid"),+ )diff --git a/openviking/metrics/collectors/service_probe.py b/openviking/metrics/collectors/service_probe.pynew file mode 100644index 000000000..0afc85a43--- /dev/null+++ b/openviking/metrics/collectors/service_probe.py@@ -0,0 +1,80 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from dataclasses import dataclass+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector+from openviking.metrics.datasources.probes import ServiceProbeDataSource++from .base import CollectorConfig, ProbeMetricCollector+++@dataclass+class ServiceProbeCollector(ProbeMetricCollector):+ """+ Export top-level service readiness gauges for core service dependencies.++ This probe collector intentionally keeps only two global gauges: overall service readiness and+ API key manager readiness. The values reflect current probe state rather than cumulative+ activity, so failed refreshes downgrade the gauges through the probe base helper.+ """++ DOMAIN_SERVICE: ClassVar[str] = "service"+ DOMAIN_API_KEY_MANAGER: ClassVar[str] = "api_key_manager"+ # rule: <METRICS_NAMESPACE>_<DOMAIN_SERVICE>_readiness+ # e.g.: openviking_service_readiness+ SERVICE_READINESS: ClassVar[str] = MetricCollector.metric_name(DOMAIN_SERVICE, "readiness")+ # rule: <METRICS_NAMESPACE>_<DOMAIN_API_KEY_MANAGER>_readiness+ # e.g.: openviking_api_key_manager_readiness+ API_KEY_MANAGER_READINESS: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN_API_KEY_MANAGER, "readiness"+ )++ data_source: ServiceProbeDataSource+ config: CollectorConfig = CollectorConfig(timeout_seconds=0.5)++ def read_metric_input(self):+ """Read the latest service-level readiness booleans from the datasource."""+ return self.data_source.read_probe_state()++ def collect_hook(self, registry, metric_input) -> None:+ """Translate the service readiness snapshot into validity-labelled 0/1 readiness gauges."""+ state = metric_input+ self.replace_gauge(+ registry,+ self.SERVICE_READINESS,+ 1.0 if bool(state.get("service_readiness", False)) else 0.0,+ match_labels={},+ labels={"valid": "1"},+ label_names=("valid",),+ )+ self.replace_gauge(+ registry,+ self.API_KEY_MANAGER_READINESS,+ 1.0 if bool(state.get("api_key_manager_readiness", False)) else 0.0,+ match_labels={},+ labels={"valid": "1"},+ label_names=("valid",),+ )++ def collect_stale_hook(self, registry, error: Exception) -> None:+ """Mark both service readiness gauges as stale when probe refresh cannot complete."""+ self.replace_gauge(+ registry,+ self.SERVICE_READINESS,+ 0.0,+ match_labels={},+ labels={"valid": "0"},+ label_names=("valid",),+ )+ self.replace_gauge(+ registry,+ self.API_KEY_MANAGER_READINESS,+ 0.0,+ match_labels={},+ labels={"valid": "0"},+ label_names=("valid",),+ )diff --git a/openviking/metrics/collectors/session.py b/openviking/metrics/collectors/session.pynew file mode 100644index 000000000..061563bab--- /dev/null+++ b/openviking/metrics/collectors/session.py@@ -0,0 +1,123 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Event collector: SessionCollector.++Tracks session lifecycle and usage signals emitted from session-related code paths:+- create/get/delete/commit/extract lifecycle outcomes+- contexts and skills usage counts+- archive outcome (ok/skip)++Labels are bounded:+- action/status are small enums controlled by the codebase.+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector++from .base import EventMetricCollector+++@dataclass+class SessionCollector(EventMetricCollector):+ """+ Translate session lifecycle and usage events into bounded session counters.++ The collector receives coarse-grained events from session management code paths and records+ only stable labels such as action and status so the exported series remain suitable for+ long-term dashboarding.+ """++ DOMAIN: ClassVar[str] = "session"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_lifecycle_total+ # e.g.: openviking_session_lifecycle_total+ LIFECYCLE_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "lifecycle", unit="total")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_contexts_used_total+ # e.g.: openviking_session_contexts_used_total+ CONTEXTS_USED_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "contexts_used", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_archive_total+ # e.g.: openviking_session_archive_total+ ARCHIVE_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "archive", unit="total")++ SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset(+ {+ "session.lifecycle",+ "session.contexts_used",+ "session.archive",+ }+ )++ def collect(self, registry=None) -> None:+ """Implement the collector interface as a no-op because session metrics are push-driven."""+ return None++ def receive_hook(self, event_name: str, payload: dict, registry) -> None:+ """+ Dispatch one normalized session event to the matching counter update helper.++ Each branch handles a bounded event family emitted from session services and adapters.+ """+ if event_name == "session.lifecycle":+ action = payload.get("action")+ status = payload.get("status")+ if action is None or status is None:+ return+ self.record_lifecycle(+ registry,+ action=str(action),+ status=str(status),+ )+ return+ if event_name == "session.contexts_used":+ action = payload.get("action")+ delta = payload.get("delta")+ if action is None or delta is None:+ return+ self.record_contexts_used(+ registry,+ action=str(action),+ delta=int(delta),+ )+ return+ if event_name == "session.archive":+ status = payload.get("status")+ if status is None:+ return+ self.record_archive(+ registry,+ status=str(status),+ )++ def record_lifecycle(self, registry, *, action: str, status: str) -> None:+ """Increment the lifecycle counter for one session action/status pair."""+ registry.inc_counter(+ self.LIFECYCLE_TOTAL,+ labels={"action": str(action), "status": str(status)},+ label_names=("action", "status"),+ )++ def record_contexts_used(self, registry, *, action: str, delta: int) -> None:+ """Increase the contexts-used counter by the positive number of contexts consumed."""+ if delta <= 0:+ return+ registry.inc_counter(+ self.CONTEXTS_USED_TOTAL,+ labels={"action": str(action)},+ label_names=("action",),+ amount=int(delta),+ )++ def record_archive(self, registry, *, status: str) -> None:+ """Increment the archive outcome counter for one session-archive attempt."""+ registry.inc_counter(+ self.ARCHIVE_TOTAL,+ labels={"status": str(status)},+ label_names=("status",),+ )diff --git a/openviking/metrics/collectors/storage_probe.py b/openviking/metrics/collectors/storage_probe.pynew file mode 100644index 000000000..95408c833--- /dev/null+++ b/openviking/metrics/collectors/storage_probe.py@@ -0,0 +1,70 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from dataclasses import dataclass, field+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector+from openviking.metrics.datasources.probes import StorageProbeDataSource++from .base import CollectorConfig, ProbeMetricCollector+++@dataclass+class StorageProbeCollector(ProbeMetricCollector):+ """+ Export readiness for each storage probe with explicit stale-data semantics.++ Storage availability may degrade independently by backend, so the collector keeps one series+ per probe name and republishes the last observed probe set with `valid="0"` if refreshes fail.+ """++ DOMAIN: ClassVar[str] = "storage"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_readiness+ # e.g.: openviking_storage_readiness+ READINESS: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "readiness")++ data_source: StorageProbeDataSource+ config: CollectorConfig = CollectorConfig(ttl_seconds=5.0, timeout_seconds=0.8)+ _known_probes: list[str] = field(default_factory=lambda: ["agfs"], init=False, repr=False)++ def read_metric_input(self):+ """Read the latest storage probe mapping from the datasource."""+ return self.data_source.read_probe_state()++ def collect_hook(self, registry, metric_input) -> None:+ """+ Refresh storage probe gauges.++ On success, each probe is exported with `valid="1"`. On failure, the last known probe+ set is exported with `valid="0"` so monitoring can detect staleness.+ """+ state = metric_input+ probes = [str(k) for k in state.keys()]+ if probes:+ self._known_probes = probes+ for probe, ok in state.items():+ base = {"probe": str(probe)}+ self.replace_gauge(+ registry,+ self.READINESS,+ 1.0 if ok else 0.0,+ match_labels=base,+ labels={"probe": str(probe), "valid": "1"},+ label_names=("probe", "valid"),+ )++ def collect_error_hook(self, registry, error: Exception) -> None:+ """Export invalid storage readiness gauges for the last known probes on failure."""+ for probe in self._known_probes:+ base = {"probe": str(probe)}+ self.replace_gauge(+ registry,+ self.READINESS,+ 0.0,+ match_labels=base,+ labels={"probe": str(probe), "valid": "0"},+ label_names=("probe", "valid"),+ )diff --git a/openviking/metrics/collectors/task_tracker.py b/openviking/metrics/collectors/task_tracker.pynew file mode 100644index 000000000..93a270ca9--- /dev/null+++ b/openviking/metrics/collectors/task_tracker.py@@ -0,0 +1,77 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from dataclasses import dataclass+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector+from openviking.metrics.datasources.task import TaskStateDataSource++from .base import CollectorConfig, StateMetricCollector+++@dataclass+class TaskTrackerCollector(StateMetricCollector):+ """+ Export task tracker backlog and completion counts grouped by task type.++ The datasource returns a snapshot of task counts, so all four families are exposed as gauges+ keyed only by `task_type` rather than as cumulative counters.+ """++ DOMAIN: ClassVar[str] = "task"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_pending+ # e.g.: openviking_task_pending+ PENDING: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "pending")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_running+ # e.g.: openviking_task_running+ RUNNING: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "running")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_completed+ # e.g.: openviking_task_completed+ COMPLETED: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "completed")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_failed+ # e.g.: openviking_task_failed+ FAILED: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "failed")++ data_source: TaskStateDataSource+ config: CollectorConfig = CollectorConfig(timeout_seconds=0.5)++ def read_metric_input(self):+ """Read the latest task-count snapshot grouped by task type from the datasource."""+ return self.data_source.read_task_counts()++ def collect_hook(self, registry, metric_input) -> None:+ """Translate the task-count snapshot into pending, running, completed, and failed gauges."""+ counts_by_type = metric_input+ # Snapshot semantics: types can disappear entirely, so clear previous series to avoid+ # exporting stale non-zero gauges forever.+ for metric_name in (self.PENDING, self.RUNNING, self.COMPLETED, self.FAILED):+ registry.gauge_delete_matching(metric_name, match_labels={})+ for task_type, counts in counts_by_type.items():+ labels = {"task_type": str(task_type)}+ registry.set_gauge(+ self.PENDING,+ float(counts.get("pending", 0)),+ labels=labels,+ label_names=("task_type",),+ )+ registry.set_gauge(+ self.RUNNING,+ float(counts.get("running", 0)),+ labels=labels,+ label_names=("task_type",),+ )+ registry.set_gauge(+ self.COMPLETED,+ float(counts.get("completed", 0)),+ labels=labels,+ label_names=("task_type",),+ )+ registry.set_gauge(+ self.FAILED,+ float(counts.get("failed", 0)),+ labels=labels,+ label_names=("task_type",),+ )diff --git a/openviking/metrics/collectors/telemetry_bridge.py b/openviking/metrics/collectors/telemetry_bridge.pynew file mode 100644index 000000000..5e05df572--- /dev/null+++ b/openviking/metrics/collectors/telemetry_bridge.py@@ -0,0 +1,280 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Event collector: TelemetryBridgeCollector.++This collector converts the aggregated telemetry summary of a single operation/request+into Prometheus metrics.++Input:+- A single `telemetry.summary` event per operation (best-effort).++Output:+- Low-cardinality counters/histograms for operation duration, tokens, vector search counts,+ semantic node counts, and resource stage timing.++Design notes:+- The telemetry summary is already aggregated; we avoid adding labels that would cause+ cardinality explosions (no session_id/resource_uri/etc.).+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import Any, ClassVar, Mapping++from openviking.metrics.core.base import MetricCollector++from .base import EventMetricCollector+++@dataclass+class TelemetryBridgeCollector(EventMetricCollector):+ """+ Translate aggregated telemetry summaries into low-cardinality Prometheus metrics.++ The collector expands one summary event into multiple families that represent operation+ throughput, latency, vector retrieval work, semantic-node counts, and resource-stage timing.+ """++ DOMAIN_OPERATION: ClassVar[str] = "operation"+ DOMAIN_VECTOR: ClassVar[str] = "vector"+ DOMAIN_SEMANTIC: ClassVar[str] = "semantic"+ DOMAIN_MEMORY: ClassVar[str] = "memory"+ DOMAIN_RESOURCE: ClassVar[str] = "resource"++ # rule: <METRICS_NAMESPACE>_<DOMAIN_OPERATION>_requests_total+ # e.g.: openviking_operation_requests_total+ OPERATION_REQUESTS_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN_OPERATION, "requests", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN_OPERATION>_duration_seconds+ # e.g.: openviking_operation_duration_seconds+ OPERATION_DURATION_SECONDS: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN_OPERATION, "duration", unit="seconds"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN_OPERATION>_tokens_total+ # e.g.: openviking_operation_tokens_total+ OPERATION_TOKENS_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN_OPERATION, "tokens", unit="total"+ )++ # rule: <METRICS_NAMESPACE>_<DOMAIN_VECTOR>_searches_total+ # e.g.: openviking_vector_searches_total+ VECTOR_SEARCHES_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN_VECTOR, "searches", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN_VECTOR>_scored_total+ # e.g.: openviking_vector_scored_total+ VECTOR_SCORED_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN_VECTOR, "scored", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN_VECTOR>_passed_total+ # e.g.: openviking_vector_passed_total+ VECTOR_PASSED_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN_VECTOR, "passed", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN_VECTOR>_returned_total+ # e.g.: openviking_vector_returned_total+ VECTOR_RETURNED_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN_VECTOR, "returned", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN_VECTOR>_scanned_total+ # e.g.: openviking_vector_scanned_total+ VECTOR_SCANNED_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN_VECTOR, "scanned", unit="total"+ )++ # rule: <METRICS_NAMESPACE>_<DOMAIN_SEMANTIC>_nodes_total+ # e.g.: openviking_semantic_nodes_total+ SEMANTIC_NODES_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN_SEMANTIC, "nodes", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN_MEMORY>_extracted_total+ # e.g.: openviking_memory_extracted_total+ MEMORY_EXTRACTED_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN_MEMORY, "extracted", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN_RESOURCE>_stage_total+ # e.g.: openviking_resource_stage_total+ RESOURCE_STAGE_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN_RESOURCE, "stage", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN_RESOURCE>_stage_duration_seconds+ # e.g.: openviking_resource_stage_duration_seconds+ RESOURCE_STAGE_DURATION_SECONDS: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN_RESOURCE, "stage_duration", unit="seconds"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN_RESOURCE>_wait_duration_seconds+ # e.g.: openviking_resource_wait_duration_seconds+ RESOURCE_WAIT_DURATION_SECONDS: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN_RESOURCE, "wait_duration", unit="seconds"+ )++ SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset({"telemetry.summary"})++ def collect(self, registry=None) -> None:+ """+ Implement the unified collector interface as a no-op for this event-driven collector.++ Telemetry summaries are pushed in from runtime code, so scrape-time collection has no+ additional work to perform.+ """+ return None++ def receive_hook(self, event_name: str, payload: dict, registry) -> None:+ """+ Translate the supported telemetry summary event into derived metric writes.++ The payload is expected to contain one already-aggregated summary for a completed logical+ operation or request.+ """+ summary = payload.get("summary")+ if not isinstance(summary, Mapping):+ return+ self.record_telemetry_summary(registry, summary)++ def record_telemetry_summary(self, registry, summary: Mapping[str, Any]) -> None:+ """+ Record counters, histograms, and gauges derived from one aggregated telemetry summary.++ This method is intentionally low-cardinality: it emits bounded labels and avoids copying+ identifiers such as session ids or resource URIs into Prometheus series.+ """+ operation = str(summary.get("operation", "unknown"))+ status = str(summary.get("status", "ok"))++ registry.inc_counter(+ self.OPERATION_REQUESTS_TOTAL,+ labels={"operation": operation, "status": status},+ label_names=("operation", "status"),+ )++ duration_seconds = float(summary.get("duration_ms", 0.0)) / 1000.0+ registry.observe_histogram(+ self.OPERATION_DURATION_SECONDS,+ duration_seconds,+ labels={"operation": operation, "status": status},+ label_names=("operation", "status"),+ )++ tokens = summary.get("tokens") or {}+ total_tokens = int(tokens.get("total", 0) or 0)+ llm = tokens.get("llm") or {}+ llm_input = int(llm.get("input", 0) or 0)+ llm_output = int(llm.get("output", 0) or 0)+ embedding_tokens = int(tokens.get("embedding", 0) or 0)++ for token_type, value in (+ ("all", total_tokens),+ ("llm_input", llm_input),+ ("llm_output", llm_output),+ ("embedding", embedding_tokens),+ ):+ if value <= 0:+ continue+ registry.inc_counter(+ self.OPERATION_TOKENS_TOTAL,+ labels={"operation": operation, "token_type": token_type},+ label_names=("operation", "token_type"),+ amount=value,+ )++ vector = summary.get("vector") or {}+ for metric_name, key in (+ (self.VECTOR_SEARCHES_TOTAL, "searches"),+ (self.VECTOR_SCORED_TOTAL, "scored"),+ (self.VECTOR_PASSED_TOTAL, "passed"),+ (self.VECTOR_RETURNED_TOTAL, "returned"),+ (self.VECTOR_SCANNED_TOTAL, "scanned"),+ ):+ value = int(vector.get(key, 0) or 0)+ if value <= 0:+ continue+ registry.inc_counter(+ metric_name,+ labels={"operation": operation},+ label_names=("operation",),+ amount=value,+ )++ semantic_nodes = summary.get("semantic_nodes") or {}+ for k, v in semantic_nodes.items():+ value = int(v or 0)+ if value < 0:+ continue+ registry.inc_counter(+ self.SEMANTIC_NODES_TOTAL,+ labels={"status": str(k)},+ label_names=("status",),+ amount=value,+ )++ memory = summary.get("memory") or {}+ extracted = int(memory.get("extracted", 0) or 0)+ if extracted > 0:+ registry.inc_counter(+ self.MEMORY_EXTRACTED_TOTAL,+ labels={"operation": operation},+ label_names=("operation",),+ amount=extracted,+ )++ resource = summary.get("resource") or {}+ if resource:+ self._record_resource_stage(+ registry, resource=resource, operation=operation, status=status+ )++ def _record_resource_stage(+ self, registry, *, resource: Mapping[str, Any], operation: str, status: str+ ) -> None:+ """+ Record resource-stage metrics from the nested resource timing section of a summary.++ The method walks a fixed stage map so dashboards can rely on a stable stage vocabulary+ even when some stages are absent from a particular summary payload.+ """+ stage_defs = [+ ("request", ("request",)),+ ("process", ("process",)),+ ("parse", ("process", "parse")),+ ("finalize", ("process", "finalize")),+ ("summarize", ("process", "summarize")),+ ("wait", ("wait",)),+ ("watch", ("watch",)),+ ]++ for stage, path in stage_defs:+ node: Any = resource+ for p in path:+ node = (node or {}).get(p)+ if not isinstance(node, Mapping):+ continue+ duration_ms = float(node.get("duration_ms", 0.0) or 0.0)+ stage_status = status+ if stage == "parse" and stage_status == "ok":+ warnings = int(node.get("warnings_count", 0) or 0)+ if warnings > 0:+ stage_status = "warning"+ labels = {"stage": stage, "status": stage_status}+ registry.inc_counter(+ self.RESOURCE_STAGE_TOTAL,+ labels=labels,+ label_names=("stage", "status"),+ )+ registry.observe_histogram(+ self.RESOURCE_STAGE_DURATION_SECONDS,+ duration_ms / 1000.0,+ labels=labels,+ label_names=("stage", "status"),+ )++ if stage == "wait" and duration_ms > 0:+ registry.observe_histogram(+ self.RESOURCE_WAIT_DURATION_SECONDS,+ duration_ms / 1000.0,+ labels={"operation": operation},+ label_names=("operation",),+ )diff --git a/openviking/metrics/collectors/vikingdb.py b/openviking/metrics/collectors/vikingdb.pynew file mode 100644index 000000000..0fafbeb86--- /dev/null+++ b/openviking/metrics/collectors/vikingdb.py@@ -0,0 +1,102 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from dataclasses import dataclass, field+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector+from openviking.metrics.datasources.observer_state import VikingDBStateDataSource++from .base import CollectorConfig, StateMetricCollector+++@dataclass+class VikingDBCollector(StateMetricCollector):+ """+ Export VikingDB collection health and vector-count gauges with stale fallback semantics.++ The datasource reports one active collection snapshot at a time. This collector keeps the last+ successful collection label so a failed refresh can still publish `valid="0"` series that+ preserve dashboard continuity.+ """++ STALE_ON_ERROR: ClassVar[bool] = True++ DOMAIN: ClassVar[str] = "vikingdb"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_collection_health+ # e.g.: openviking_vikingdb_collection_health+ COLLECTION_HEALTH: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "collection_health")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_collection_vectors+ # e.g.: openviking_vikingdb_collection_vectors+ COLLECTION_VECTORS: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "collection_vectors")++ data_source: VikingDBStateDataSource+ config: CollectorConfig = CollectorConfig(ttl_seconds=10.0, timeout_seconds=0.8)+ _last_collection: str = field(default="unknown", init=False, repr=False)++ def read_metric_input(self):+ """Read the latest VikingDB collection state from the datasource."""+ return self.data_source.read_vikingdb_state()++ def collect_hook(self, registry, metric_input) -> None:+ """+ Refresh VikingDB gauges from the datasource.++ On success, gauges are exported with `valid="1"`. On failure, the collector falls back+ to the last observed collection name and preserves last vectors count when present,+ emitting `valid="0"` to mark the data as stale.+ """+ collection, ok, vectors = metric_input+ self._last_collection = str(collection)+ base = {"collection": str(collection)}+ labels = {"collection": str(collection), "valid": "1"}+ self.replace_gauge_series(+ registry,+ self.COLLECTION_HEALTH,+ 1.0 if ok else 0.0,+ match_labels=base,+ labels=labels,+ label_names=("collection", "valid"),+ )+ self.replace_gauge_series(+ registry,+ self.COLLECTION_VECTORS,+ float(vectors),+ match_labels=base,+ labels=labels,+ label_names=("collection", "valid"),+ )++ def collect_error_hook(self, registry, error: Exception) -> None:+ """Delegate failure handling to the stale hook by re-raising the datasource error."""+ raise error++ def collect_stale_hook(self, registry, error: Exception) -> None:+ """Export stale VikingDB gauges under `valid=0` when datasource refresh fails."""+ collection = self._last_collection+ base = {"collection": collection}+ last_vectors = registry.gauge_get(+ self.COLLECTION_VECTORS,+ labels={"collection": collection, "valid": "1"},+ )+ if last_vectors is None:+ last_vectors = 0.0+ labels = {"collection": collection, "valid": "0"}+ self.replace_gauge_series(+ registry,+ self.COLLECTION_HEALTH,+ 0.0,+ match_labels=base,+ labels=labels,+ label_names=("collection", "valid"),+ )+ self.replace_gauge_series(+ registry,+ self.COLLECTION_VECTORS,+ float(last_vectors),+ match_labels=base,+ labels=labels,+ label_names=("collection", "valid"),+ )diff --git a/openviking/metrics/collectors/vlm.py b/openviking/metrics/collectors/vlm.pynew file mode 100644index 000000000..e62036629--- /dev/null+++ b/openviking/metrics/collectors/vlm.py@@ -0,0 +1,138 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Event collector: VLMCollector.++Tracks VLM call count, duration, and token usage:+- Calls counter by provider/model+- Duration histogram by provider/model+- Token counters by provider/model++This collector is fed by VLMEventDataSource.record_call(...) events emitted from VLM code paths.+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import ClassVar++from openviking.metrics.core.base import MetricCollector++from .base import EventMetricCollector+++@dataclass+class VLMCollector(EventMetricCollector):+ """+ Translate VLM call events into per-provider/model counters and latency/token metrics.++ The collector consumes one normalized payload per completed VLM call and emits both+ per-call latency data and billing-style token totals.+ """++ DOMAIN: ClassVar[str] = "vlm"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_calls_total+ # e.g.: openviking_vlm_calls_total+ CALLS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "calls", unit="total")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_call_duration_seconds+ # e.g.: openviking_vlm_call_duration_seconds+ CALL_DURATION_SECONDS: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "call_duration", unit="seconds"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_tokens_input_total+ # e.g.: openviking_vlm_tokens_input_total+ TOKENS_INPUT_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "tokens_input", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_tokens_output_total+ # e.g.: openviking_vlm_tokens_output_total+ TOKENS_OUTPUT_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "tokens_output", unit="total"+ )+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_tokens_total+ # e.g.: openviking_vlm_tokens_total+ TOKENS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "tokens", unit="total")++ SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset({"vlm.call"})++ def collect(self, registry=None) -> None:+ """+ Implement the unified collector interface as a no-op for this event-driven collector.++ VLM metrics are updated exclusively through `receive(...)`, so scrape-time pull+ collection does not need to do any work here.+ """+ return None++ def receive_hook(self, event_name: str, payload: dict, registry) -> None:+ """+ Translate the supported VLM call event into counters and histogram samples.++ The hook forwards explicit account context from the payload so background VLM work can+ still land in the correct tenant partition.+ """+ self.record_call(+ registry,+ provider=str(payload["provider"]),+ model_name=str(payload["model_name"]),+ duration_seconds=float(payload["duration_seconds"]),+ prompt_tokens=int(payload["prompt_tokens"]),+ completion_tokens=int(payload["completion_tokens"]),+ account_id=(+ None if payload.get("account_id") is None else str(payload.get("account_id"))+ ),+ )++ def record_call(+ self,+ registry,+ *,+ provider: str,+ model_name: str,+ duration_seconds: float,+ prompt_tokens: int,+ completion_tokens: int,+ account_id: str | None = None,+ ) -> None:+ """+ Record one VLM call as counters for calls and tokens plus a latency histogram sample.++ Input, output, and total token counters are all emitted because different dashboards care+ about cost attribution at different levels of granularity.+ """+ labels = {"provider": str(provider), "model_name": str(model_name)}+ registry.inc_counter(+ self.CALLS_TOTAL,+ labels=labels,+ label_names=("provider", "model_name"),+ account_id=account_id,+ )+ registry.observe_histogram(+ self.CALL_DURATION_SECONDS,+ float(duration_seconds),+ labels=labels,+ label_names=("provider", "model_name"),+ account_id=account_id,+ )+ registry.inc_counter(+ self.TOKENS_INPUT_TOTAL,+ labels=labels,+ label_names=("provider", "model_name"),+ amount=int(prompt_tokens),+ account_id=account_id,+ )+ registry.inc_counter(+ self.TOKENS_OUTPUT_TOTAL,+ labels=labels,+ label_names=("provider", "model_name"),+ amount=int(completion_tokens),+ account_id=account_id,+ )+ registry.inc_counter(+ self.TOKENS_TOTAL,+ labels=labels,+ label_names=("provider", "model_name"),+ amount=int(prompt_tokens) + int(completion_tokens),+ account_id=account_id,+ )diff --git a/openviking/metrics/core/__init__.py b/openviking/metrics/core/__init__.pynew file mode 100644index 000000000..b8247db30--- /dev/null+++ b/openviking/metrics/core/__init__.py@@ -0,0 +1,19 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""Core contracts and runtime primitives shared by the metrics subsystem."""++from .base import MetricCollector, MetricDataSource, MetricExporter+from .refresh import RefreshDecision, RefreshGate+from .registry import MetricRegistry+from .runtime import EventCollectorRouter++__all__ = [+ "MetricDataSource",+ "MetricCollector",+ "MetricExporter",+ "MetricRegistry",+ "RefreshDecision",+ "RefreshGate",+ "EventCollectorRouter",+]diff --git a/openviking/metrics/core/base.py b/openviking/metrics/core/base.pynew file mode 100644index 000000000..70836585f--- /dev/null+++ b/openviking/metrics/core/base.py@@ -0,0 +1,223 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Core abstract interfaces for the metrics subsystem.++These interfaces define the minimal contracts used across the metrics architecture:+- DataSource: produces metrics inputs (events or read APIs), without writing the registry.+- Collector: translates inputs into MetricRegistry writes.+- Exporter: triggers refresh (best-effort) and renders metrics to an exposition format.++The intent of this module is to keep the top-level contracts small and stable:+- `MetricDataSource` describes how metric inputs are produced.+- `MetricCollector` describes the two supported collection styles:+ pull via `collect(...)` and push via `receive(...)`.+- `MetricExporter` describes how metrics are rendered for an external system such as Prometheus.+"""++from __future__ import annotations++from abc import ABC, abstractmethod+from dataclasses import dataclass+from typing import Any, Callable, Generic, TypeVar++T = TypeVar("T")+++@dataclass(frozen=True, slots=True)+class ReadEnvelope(Generic[T]):+ """+ Wrap the result of a datasource read together with normalized failure metadata.++ Datasources use this envelope to return a best-effort value even when the underlying read+ raises, allowing collectors to export stale or default values while still surfacing the+ failure type and message for diagnostics.+ """++ ok: bool+ value: T+ error_type: str | None = None+ error_message: str | None = None+++class MetricDataSource(ABC):+ """+ Base interface for metric DataSources.++ A DataSource is responsible for providing metrics inputs:+ - For event-style metrics, it emits events (handled by an event router).+ - For pull-style metrics, it exposes read methods used by DomainStats/State/Probe collectors.++ DataSources must not write MetricRegistry directly; collectors are the only writers.+ """++ @classmethod+ @abstractmethod+ def kind(cls) -> str:+ """Return the datasource category identifier (e.g. `event`, `state`, `probe`)."""++ def source_name(self) -> str:+ """+ Return a stable human-readable name for the data source.++ The default implementation uses the concrete class name, which is sufficient for+ diagnostics, logging, and lightweight introspection. Subclasses may override this+ when they need a more explicit or externally stable identifier.+ """+ return self.__class__.__name__++ def safe_read(self, fn: Callable[[], T], *, default: T) -> ReadEnvelope[T]:+ """+ Execute a synchronous read function and convert failures into a `ReadEnvelope`.++ The helper lets datasource implementations keep best-effort read semantics concise while+ returning a deterministic default value when the underlying subsystem raises.+ """+ try:+ return ReadEnvelope(ok=True, value=fn())+ except Exception as e:+ return ReadEnvelope(+ ok=False,+ value=default,+ error_type=type(e).__name__,+ error_message=str(e),+ )++ def safe_read_async(+ self,+ coro_fn: Callable[[], Any],+ *,+ default: T,+ runner: Callable[[Any], Any],+ ) -> ReadEnvelope[T]:+ """+ Execute an async-style read through a caller-provided runner and wrap the result safely.++ This keeps async read orchestration outside the datasource while preserving the same+ best-effort error envelope used by `safe_read`.+ """+ return self.safe_read(lambda: runner(coro_fn()), default=default)++ @staticmethod+ def as_int(value: Any, *, default: int = 0) -> int:+ """Convert an arbitrary value to `int`, falling back to a deterministic default on error."""+ try:+ return int(value)+ except Exception:+ return int(default)++ @staticmethod+ def as_dict(value: Any) -> dict:+ """Return the value when it is a dictionary, otherwise return an empty dictionary."""+ return value if isinstance(value, dict) else {}++ @staticmethod+ def normalize_str(value: Any, *, default: str = "unknown") -> str:+ """Normalize arbitrary input into a non-empty string with a caller-provided fallback."""+ text = str(value or "").strip()+ return text if text else str(default)+++class MetricCollector(ABC):+ """+ Base interface for metric collectors.++ A collector is the only layer that is allowed to write into MetricRegistry.+ Collectors may be triggered on-demand (e.g., before `/metrics` export) and should:+ - be fast and resilient (best-effort, do not crash the exporter)+ - avoid high-cardinality labels++ A collector supports two input modes:+ - `collect(...)`: pull-style collection, where the collector actively reads state and+ writes samples into the registry.+ - `receive(...)`: push-style collection, where the collector receives an already+ materialized event and translates it into registry writes.++ Both methods intentionally default to no-op so observability remains a sidecar concern:+ a collector only implements the modes it actually needs.+ """++ METRICS_NAMESPACE = "openviking"++ @classmethod+ @abstractmethod+ def kind(cls) -> str:+ """Return the collector category identifier (e.g. `event`, `state`, `probe`)."""++ @classmethod+ def metric_name(cls, domain: str, metric: str, unit: str | None = None) -> str:+ """+ Build a fully-qualified metric name under the project namespace.++ Args:+ domain: Domain namespace for the metric (may include underscores).+ metric: Metric name within the domain (should not include the namespace).+ unit: Optional unit or suffix segment (e.g. `total`, `seconds`, `bytes`).++ Returns:+ Fully-qualified metric name string such as `openviking_http_requests_total`.+ """+ base = f"{cls.METRICS_NAMESPACE}_{str(domain)}_{str(metric)}"+ if unit:+ return f"{base}_{str(unit)}"+ return base++ def collector_name(self) -> str:+ """+ Return a stable collector name for diagnostics and orchestration.++ The default implementation returns the concrete class name, which keeps call sites+ simple and avoids forcing every collector to repeat the same naming boilerplate.+ """+ return self.__class__.__name__++ def collect(self, registry) -> None:+ """+ Perform pull-style collection and write samples into the registry.++ Args:+ registry: The in-process metric registry that receives the collector output.++ This method is intentionally a no-op by default. Event-driven collectors do not need+ to implement pull behavior, while refresh-managed collectors override this method with+ their actual state-reading logic.+ """+ return None++ def receive(self, event_name: str, payload: dict, registry) -> None:+ """+ Perform push-style collection by consuming a dispatched event.++ Args:+ event_name: Logical event type emitted by a DataSource or bridge layer.+ payload: Event payload already normalized into a dictionary.+ registry: The in-process metric registry that receives the translated samples.++ This method is intentionally a no-op by default. Pull-only collectors never need to+ handle events, while event-driven collectors override it to fan out by `event_name`.+ """+ return None+++class MetricExporter(ABC):+ """+ Base interface for metric exporters.++ Exporters render the current registry state into an output format (e.g., Prometheus text).+ Exporters typically trigger a refresh of registered collectors before rendering.+ """++ @abstractmethod+ async def export(self) -> str:+ """+ Render the current metric state into the exporter-specific output format.++ Returns:+ A serialized metrics payload, such as Prometheus text exposition.++ Exporters may choose to refresh pull-based collectors before rendering, but callers+ should treat this method as best-effort and should not rely on observability to affect+ business logic or request correctness.+ """+ raise NotImplementedErrordiff --git a/openviking/metrics/core/refresh.py b/openviking/metrics/core/refresh.pynew file mode 100644index 000000000..19fc51630--- /dev/null+++ b/openviking/metrics/core/refresh.py@@ -0,0 +1,138 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Refresh control primitives for scrape-triggered metrics refresh.++RefreshGate implements TTL-based refresh control with SWR support:+- If a collector never succeeded, we must refresh synchronously (first data).+- If TTL is still valid, we skip refresh to avoid repeated expensive work.+- If TTL expired and we have a previous success, callers may decide to:+ - refresh synchronously (block scrape), or+ - refresh asynchronously in background and serve stale values (SWR).+"""++from __future__ import annotations++import threading+import time+from dataclasses import dataclass+++@dataclass(frozen=True, slots=True)+class RefreshDecision:+ """+ Decision returned by `RefreshGate.decide()`.++ Attributes:+ do_refresh: Whether the caller should start a collection attempt now.+ reason: Machine-readable explanation for the decision, such as `ttl_valid`,+ `ttl_expired`, `never_succeeded`, or `inflight`.+ last_success_at: Monotonic timestamp of the last successful collection, or `None`+ when the collector has not produced a successful result yet.+ inflight: Whether another collection attempt is currently running for the same gate.+ """++ do_refresh: bool+ reason: str+ last_success_at: float | None+ inflight: bool+++class RefreshGate:+ """+ A small state machine that controls refresh frequency and tracks inflight refresh.++ The gate encapsulates the refresh bookkeeping needed by scrape-triggered collectors:+ whether a prior success exists, whether another refresh is currently running, and whether+ the TTL window has expired enough to justify a new collection attempt.+ """++ def __init__(self, *, ttl_seconds: float) -> None:+ """+ Initialize a refresh gate for a single refresh-managed collector.++ Args:+ ttl_seconds: Minimum interval between two successful refresh-triggered collection+ attempts before the gate considers the cached result expired again.+ """+ if ttl_seconds <= 0:+ raise ValueError("ttl_seconds must be positive")+ self._ttl_seconds = float(ttl_seconds)+ self._lock = threading.Lock()+ self._last_success_at: float | None = None+ self._inflight: bool = False++ def decide(self, *, now: float | None = None) -> RefreshDecision:+ """+ Decide whether a new collection attempt should start at the current time.++ Args:+ now: Optional monotonic timestamp override used mainly for tests. When omitted,+ the current `time.monotonic()` value is used.++ Returns:+ A `RefreshDecision` describing whether collection should proceed immediately and+ why that decision was made.+ """+ t = time.monotonic() if now is None else float(now)+ with self._lock:+ last = self._last_success_at+ inflight = self._inflight+ if inflight:+ return RefreshDecision(+ do_refresh=False,+ reason="inflight",+ last_success_at=last,+ inflight=True,+ )+ if last is None:+ return RefreshDecision(+ do_refresh=True,+ reason="never_succeeded",+ last_success_at=None,+ inflight=False,+ )+ if t - last >= self._ttl_seconds:+ return RefreshDecision(+ do_refresh=True,+ reason="ttl_expired",+ last_success_at=last,+ inflight=False,+ )+ return RefreshDecision(+ do_refresh=False,+ reason="ttl_valid",+ last_success_at=last,+ inflight=False,+ )++ def mark_inflight(self) -> bool:+ """+ Mark the gate as currently running a collection attempt.++ Returns:+ `True` if the caller successfully acquired the inflight slot, or `False` if another+ collection attempt was already running.+ """+ with self._lock:+ if self._inflight:+ return False+ self._inflight = True+ return True++ def mark_complete(self, *, success: bool, now: float | None = None) -> None:+ """+ Finish the inflight state and optionally record a successful completion timestamp.++ Args:+ success: Whether the just-finished collection attempt completed successfully.+ Only successful attempts advance `_last_success_at`.+ now: Optional monotonic timestamp override used mainly for tests. When omitted,+ the current `time.monotonic()` value is used.+ """+ t = time.monotonic() if now is None else float(now)+ with self._lock:+ self._inflight = False+ if success:+ self._last_success_at = tdiff --git a/openviking/metrics/core/registry.py b/openviking/metrics/core/registry.pynew file mode 100644index 000000000..0c96a9ec5--- /dev/null+++ b/openviking/metrics/core/registry.py@@ -0,0 +1,653 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+In-process metric registry used by OpenViking.++Design goals:+1) Keep the registry as a "current state store" for metrics (no snapshot/view layer).+2) Provide a strict label contract per metric name to prevent accidental high-cardinality exports.+3) Be safe under concurrent updates from request threads and exporter refresh threads.+4) Allow bounded memory usage via per-metric series limits, while exposing drop counts for debugging.++Important behaviors:+- Label normalization: label dicts are normalized into a sorted tuple of (key, value) pairs.+- Label contract: once a metric name is registered with a label key set, subsequent writes must+ use the same label keys (same keys, same order). Violations raise ValueError.+- Series limit: when a metric family reaches `max_series_per_metric`, new series are dropped+ and `openviking_metrics_dropped_series_total{metric="<name>"}` will reflect the drops.+"""++from __future__ import annotations++import threading+from bisect import bisect_left+from typing import Mapping, Sequence++from .types import normalize_labels++DEFAULT_LATENCY_BUCKETS: tuple[float, ...] = (0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0)+++def _canonicalize_label_names(label_names: Sequence[str]) -> tuple[str, ...]:+ """+ Return the canonical family label order used by the in-process registry.++ The registry normalizes concrete label mappings into key-sorted tuples, so family-level+ `label_names` must follow the same stable order to avoid false mismatches between writers+ that provide the same key set with different input ordering.+ """+ return tuple(sorted(str(name) for name in label_names))+++def _labels_contains(+ labels: tuple[tuple[str, str], ...],+ match: tuple[tuple[str, str], ...],+) -> bool:+ """+ Return whether a normalized label set contains all requested label pairs.++ This helper is used by partial-delete operations where callers provide only a subset+ of labels and expect every matching series to be removed.+ """+ if not match:+ return True+ label_set = set(labels)+ for item in match:+ if item not in label_set:+ return False+ return True+++class MetricRegistry:+ """+ MetricRegistry holds Counter/Gauge/Histogram families in memory.++ The exporter reads current values via `iter_counters/iter_gauges/iter_histograms`.+ Collectors should use the unified write helpers (`inc_counter/set_gauge/observe_histogram`)+ to keep call sites consistent and reduce accidental label misuse.+ """++ def __init__(self, *, max_series_per_metric: int = 2048) -> None:+ """+ Initialize an empty in-process metrics registry.++ Args:+ max_series_per_metric: Hard cap for the number of distinct label series allowed+ inside one metric family before additional series are dropped.+ """+ self._max_series_per_metric = max_series_per_metric+ self._lock = threading.Lock()+ self._counters: dict[str, _CounterFamily] = {}+ self._gauges: dict[str, _GaugeFamily] = {}+ self._histograms: dict[str, _HistogramFamily] = {}+ self._dropped_series_total: dict[str, int] = {}++ def inc_counter(+ self,+ name: str,+ *,+ amount: float = 1.0,+ labels: Mapping[str, str] | None = None,+ label_names: Sequence[str] = (),+ ) -> None:+ """+ Increment a Counter metric.++ Args:+ name: Prometheus metric name.+ amount: Increment delta (float, will be rendered as int when possible).+ labels: Optional label dict. Keys must exactly match `label_names`.+ label_names: Ordered label key tuple for this metric name.++ Notes:+ This helper enforces the per-metric label contract by delegating to the underlying+ Counter family, which validates label keys.+ """+ self.counter(name, label_names=label_names).inc(labels=labels, amount=float(amount))++ def set_gauge(+ self,+ name: str,+ value: float,+ *,+ labels: Mapping[str, str] | None = None,+ label_names: Sequence[str] = (),+ ) -> None:+ """+ Set a Gauge metric to an absolute value.++ Args:+ name: Prometheus metric name.+ value: Gauge value (float).+ labels: Optional label dict. Keys must exactly match `label_names`.+ label_names: Ordered label key tuple for this metric name.+ """+ self.gauge(name, label_names=label_names).set(float(value), labels=labels)++ def observe_histogram(+ self,+ name: str,+ value: float,+ *,+ labels: Mapping[str, str] | None = None,+ label_names: Sequence[str] = (),+ buckets: Sequence[float] = DEFAULT_LATENCY_BUCKETS,+ ) -> None:+ """+ Observe a value into a Histogram metric.++ Args:+ name: Prometheus metric name (base name; exporter renders _bucket/_count/_sum).+ value: Observation value.+ labels: Optional label dict. Keys must exactly match `label_names`.+ label_names: Ordered label key tuple for this metric name.+ buckets: Histogram bucket boundaries (ascending). "+Inf" bucket is implicit.+ """+ self.histogram(name, label_names=label_names, buckets=buckets).observe(+ float(value), labels=labels+ )++ def counter(self, name: str, *, label_names: Sequence[str] = ()) -> "_Counter":+ """+ Get or create a Counter family handle.++ Args:+ name: Prometheus metric name.+ label_names: Canonical ordered label keys for the family.++ Returns:+ A lightweight `_Counter` wrapper bound to the requested family.+ """+ ln = _canonicalize_label_names(label_names)+ with self._lock:+ if name in self._gauges or name in self._histograms:+ raise ValueError(f"metric {name} already registered with a different type")+ family = self._counters.get(name)+ if family is None:+ family = _CounterFamily(+ name=name,+ label_names=ln,+ max_series=self._max_series_per_metric,+ on_drop=self._on_drop_series,+ )+ self._counters[name] = family+ else:+ family._validate_label_names(ln)+ return _Counter(family)++ def gauge(self, name: str, *, label_names: Sequence[str] = ()) -> "_Gauge":+ """+ Get or create a Gauge family handle.++ Args:+ name: Prometheus metric name.+ label_names: Canonical ordered label keys for the family.++ Returns:+ A lightweight `_Gauge` wrapper bound to the requested family.+ """+ ln = _canonicalize_label_names(label_names)+ with self._lock:+ if name in self._counters or name in self._histograms:+ raise ValueError(f"metric {name} already registered with a different type")+ family = self._gauges.get(name)+ if family is None:+ family = _GaugeFamily(+ name=name,+ label_names=ln,+ max_series=self._max_series_per_metric,+ on_drop=self._on_drop_series,+ )+ self._gauges[name] = family+ else:+ family._validate_label_names(ln)+ return _Gauge(family)++ def histogram(+ self,+ name: str,+ *,+ label_names: Sequence[str] = (),+ buckets: Sequence[float] = DEFAULT_LATENCY_BUCKETS,+ ) -> "_Histogram":+ """+ Get or create a Histogram family handle.++ Args:+ name: Prometheus metric base name.+ label_names: Canonical ordered label keys for the family.+ buckets: Histogram bucket upper bounds.++ Returns:+ A lightweight `_Histogram` wrapper bound to the requested family.+ """+ ln = _canonicalize_label_names(label_names)+ b = tuple(float(x) for x in buckets)+ with self._lock:+ if name in self._counters or name in self._gauges:+ raise ValueError(f"metric {name} already registered with a different type")+ family = self._histograms.get(name)+ if family is None:+ family = _HistogramFamily(+ name=name,+ label_names=ln,+ bucket_bounds=b,+ max_series=self._max_series_per_metric,+ on_drop=self._on_drop_series,+ )+ self._histograms[name] = family+ else:+ family._validate_label_names(ln)+ family._validate_buckets(b)+ return _Histogram(family)++ def iter_counters(self):+ """+ Iterate over all counter families and their current series values.++ The returned family payload is a detached snapshot so exporters can iterate without+ holding the registry lock during rendering.+ """+ with self._lock:+ families = dict(self._counters)+ for name, family in families.items():+ yield name, list(family.copy_values().items())++ def counter_label_names(self, name: str) -> tuple[str, ...]:+ """Return the registered label key tuple for a counter family, if present."""+ with self._lock:+ family = self._counters.get(name)+ return family.label_names if family is not None else ()++ def iter_gauges(self):+ """+ Iterate over all gauge families and their current series values.++ As with counters, the family payload is copied before iteration so rendering stays+ independent from concurrent writers.+ """+ with self._lock:+ families = dict(self._gauges)+ for name, family in families.items():+ yield name, list(family.copy_values().items())++ def gauge_label_names(self, name: str) -> tuple[str, ...]:+ """Return the registered label key tuple for a gauge family, if present."""+ with self._lock:+ family = self._gauges.get(name)+ return family.label_names if family is not None else ()++ def gauge_get(self, name: str, *, labels: Mapping[str, str] | None = None) -> float | None:+ """+ Read the current value of a gauge series.++ Args:+ name: Gauge metric name.+ labels: Optional label dict identifying a single series.++ Returns:+ The current gauge value, or `None` when the family or series does not exist.+ """+ with self._lock:+ family = self._gauges.get(name)+ if family is None:+ return None+ return family.get_value(labels=labels)++ def gauge_delete_matching(self, name: str, *, match_labels: Mapping[str, str]) -> None:+ """+ Delete every gauge series whose labels contain the provided subset.++ Args:+ name: Gauge metric name.+ match_labels: Partial label set used for matching series to remove.+ """+ with self._lock:+ family = self._gauges.get(name)+ if family is None:+ return+ family.delete_matching(match_labels=match_labels)++ def iter_histograms(self):+ """+ Iterate over all histogram families and their materialized series snapshots.++ Each yielded item contains family metadata and a detached list of per-series bucket+ counts, sample counts, and sums ready for exporter rendering.+ """+ with self._lock:+ families = dict(self._histograms)+ for name, family in families.items():+ label_names, bucket_bounds, series = family.copy_series()+ yield name, label_names, bucket_bounds, series++ def iter_dropped_series(self):+ """+ Iterate over metrics that have dropped series because of series-limit enforcement.++ Exporters use this bookkeeping to expose internal pressure signals when a metric family+ hits its cardinality cap and starts rejecting additional series.+ """+ with self._lock:+ for metric_name, dropped in self._dropped_series_total.items():+ yield metric_name, dropped++ def _on_drop_series(self, metric_name: str) -> None:+ """Record that a new series was rejected for the given metric family."""+ with self._lock:+ self._dropped_series_total[metric_name] = (+ self._dropped_series_total.get(metric_name, 0) + 1+ )+++class _CounterFamily:+ """Internal mutable storage for one counter metric family and all of its series."""++ def __init__(+ self,+ *,+ name: str,+ label_names: tuple[str, ...],+ max_series: int,+ on_drop,+ ) -> None:+ """Create an internal counter family with bounded series cardinality."""+ self.name = name+ self.label_names = label_names+ self._max_series = max_series+ self._on_drop = on_drop+ self._lock = threading.Lock()+ self._values: dict[tuple[tuple[str, str], ...], float] = {}++ def inc(self, *, labels: Mapping[str, str] | None, amount: float) -> None:+ """+ Increase one counter series by a positive amount.++ New series are rejected once the family has reached the configured series limit, but+ existing series may continue to grow even after the cap is reached.+ """+ if amount <= 0:+ raise ValueError("counter can only be increased by a positive amount")+ key = self._normalize_and_validate(labels)+ with self._lock:+ if key not in self._values and len(self._values) >= self._max_series:+ self._on_drop(self.name)+ return+ self._values[key] = self._values.get(key, 0.0) + float(amount)++ def copy_values(self) -> dict[tuple[tuple[str, str], ...], float]:+ """Return a detached copy of all series values in this family."""+ with self._lock:+ return dict(self._values)++ def _normalize_and_validate(+ self, labels: Mapping[str, str] | None+ ) -> tuple[tuple[str, str], ...]:+ """Normalize label input and enforce the family label contract."""+ normalized = normalize_labels(labels)+ self._validate_label_names_against(normalized)+ return normalized++ def _validate_label_names(self, label_names: tuple[str, ...]) -> None:+ """Ensure subsequent family lookups use the original declared label keys."""+ if self.label_names != label_names:+ raise ValueError(+ f"metric {self.name} label_names mismatch: {self.label_names} vs {label_names}"+ )++ def _validate_label_names_against(self, normalized: tuple[tuple[str, str], ...]) -> None:+ """Ensure a concrete series write uses exactly the configured label keys."""+ if not self.label_names and normalized:+ raise ValueError(f"metric {self.name} does not accept labels")+ if self.label_names and tuple(k for k, _ in normalized) != self.label_names:+ raise ValueError(f"metric {self.name} label keys mismatch: expected {self.label_names}")+++class _GaugeFamily:+ """Internal mutable storage for one gauge metric family and all of its series."""++ def __init__(+ self,+ *,+ name: str,+ label_names: tuple[str, ...],+ max_series: int,+ on_drop,+ ) -> None:+ """Create an internal gauge family with bounded series cardinality."""+ self.name = name+ self.label_names = label_names+ self._max_series = max_series+ self._on_drop = on_drop+ self._lock = threading.Lock()+ self._values: dict[tuple[tuple[str, str], ...], float] = {}++ def set(self, *, labels: Mapping[str, str] | None, value: float) -> None:+ """+ Set one gauge series to an absolute value.++ As with counters, creating a brand-new series is still subject to the family-level+ cardinality cap.+ """+ key = self._normalize_and_validate(labels)+ with self._lock:+ if key not in self._values and len(self._values) >= self._max_series:+ self._on_drop(self.name)+ return+ self._values[key] = float(value)++ def add(self, *, labels: Mapping[str, str] | None, delta: float) -> None:+ """+ Add a signed delta to one gauge series.++ This is the internal primitive behind the public gauge `inc(...)` and `dec(...)`+ helpers.+ """+ key = self._normalize_and_validate(labels)+ with self._lock:+ if key not in self._values and len(self._values) >= self._max_series:+ self._on_drop(self.name)+ return+ self._values[key] = self._values.get(key, 0.0) + float(delta)++ def copy_values(self) -> dict[tuple[tuple[str, str], ...], float]:+ """Return a detached copy of all gauge series values."""+ with self._lock:+ return dict(self._values)++ def get_value(self, *, labels: Mapping[str, str] | None) -> float | None:+ """Return the current value for one normalized gauge series, if that series exists."""+ key = self._normalize_and_validate(labels)+ with self._lock:+ return self._values.get(key)++ def delete_matching(self, *, match_labels: Mapping[str, str]) -> None:+ """Delete every stored gauge series that contains the requested label subset."""+ normalized = normalize_labels(match_labels)+ with self._lock:+ keys = list(self._values.keys())+ for k in keys:+ if _labels_contains(k, normalized):+ self._values.pop(k, None)++ def _normalize_and_validate(+ self, labels: Mapping[str, str] | None+ ) -> tuple[tuple[str, str], ...]:+ """Normalize label input and enforce the family label contract."""+ normalized = normalize_labels(labels)+ self._validate_label_names_against(normalized)+ return normalized++ def _validate_label_names(self, label_names: tuple[str, ...]) -> None:+ """Ensure subsequent family lookups use the original declared label keys."""+ if self.label_names != label_names:+ raise ValueError(+ f"metric {self.name} label_names mismatch: {self.label_names} vs {label_names}"+ )++ def _validate_label_names_against(self, normalized: tuple[tuple[str, str], ...]) -> None:+ """Ensure a concrete series write uses exactly the configured label keys."""+ if not self.label_names and normalized:+ raise ValueError(f"metric {self.name} does not accept labels")+ if self.label_names and tuple(k for k, _ in normalized) != self.label_names:+ raise ValueError(f"metric {self.name} label keys mismatch: expected {self.label_names}")+++class _HistogramFamily:+ """Internal mutable storage for one histogram metric family and all of its label series."""++ def __init__(+ self,+ *,+ name: str,+ label_names: tuple[str, ...],+ bucket_bounds: tuple[float, ...],+ max_series: int,+ on_drop,+ ) -> None:+ """Create an internal histogram family with fixed buckets and bounded series count."""+ self.name = name+ self.label_names = label_names+ self.bucket_bounds = bucket_bounds+ self._max_series = max_series+ self._on_drop = on_drop+ self._lock = threading.Lock()+ self._series: dict[tuple[tuple[str, str], ...], _HistogramSeries] = {}++ def observe(self, *, labels: Mapping[str, str] | None, value: float) -> None:+ """+ Record one histogram observation for the selected label series.++ A new series is materialized lazily on first use, subject to the same family-level+ series cap used by counters and gauges.+ """+ key = self._normalize_and_validate(labels)+ with self._lock:+ series = self._series.get(key)+ if series is None:+ if len(self._series) >= self._max_series:+ self._on_drop(self.name)+ return+ series = _HistogramSeries(bucket_bounds=self.bucket_bounds)+ self._series[key] = series+ series.observe(float(value))++ def copy_series(+ self,+ ) -> tuple[+ tuple[str, ...],+ tuple[float, ...],+ list[tuple[tuple[tuple[str, str], ...], tuple[int, ...], int, float]],+ ]:+ """Return a detached snapshot of the histogram family and all of its series."""+ with self._lock:+ series_copy: dict[tuple[tuple[str, str], ...], _HistogramSeries] = dict(self._series)+ label_names = self.label_names+ bucket_bounds = self.bucket_bounds+ series: list[tuple[tuple[tuple[str, str], ...], tuple[int, ...], int, float]] = []+ for labels, s in series_copy.items():+ bucket_counts, count, value_sum = s.copy_values()+ series.append((labels, bucket_counts, count, value_sum))+ return label_names, bucket_bounds, series++ def _normalize_and_validate(+ self, labels: Mapping[str, str] | None+ ) -> tuple[tuple[str, str], ...]:+ """Normalize label input and enforce the family label contract."""+ normalized = normalize_labels(labels)+ self._validate_label_names_against(normalized)+ return normalized++ def _validate_label_names(self, label_names: tuple[str, ...]) -> None:+ """Ensure subsequent family lookups use the original declared label keys."""+ if self.label_names != label_names:+ raise ValueError(+ f"metric {self.name} label_names mismatch: {self.label_names} vs {label_names}"+ )++ def _validate_label_names_against(self, normalized: tuple[tuple[str, str], ...]) -> None:+ """Ensure a concrete series write uses exactly the configured label keys."""+ if not self.label_names and normalized:+ raise ValueError(f"metric {self.name} does not accept labels")+ if self.label_names and tuple(k for k, _ in normalized) != self.label_names:+ raise ValueError(f"metric {self.name} label keys mismatch: expected {self.label_names}")++ def _validate_buckets(self, bucket_bounds: tuple[float, ...]) -> None:+ """Ensure a histogram family is never reopened with a different bucket layout."""+ if self.bucket_bounds != bucket_bounds:+ raise ValueError(f"metric {self.name} buckets mismatch")+++class _HistogramSeries:+ """Internal bucket/count/sum accumulator for one concrete histogram label series."""++ def __init__(self, *, bucket_bounds: tuple[float, ...]) -> None:+ """Create one histogram series with counters for all finite buckets plus `+Inf`."""+ self._bucket_bounds = bucket_bounds+ self._lock = threading.Lock()+ self._bucket_counts: list[int] = [0] * (len(bucket_bounds) + 1)+ self._count: int = 0+ self._sum: float = 0.0++ def observe(self, value: float) -> None:+ """+ Record a single observation into the appropriate bucket and aggregate summary totals.++ Bucket selection uses `bisect_left`, so each observation increments the first bucket+ whose upper bound is greater than or equal to the recorded value.+ """+ idx = bisect_left(self._bucket_bounds, value)+ with self._lock:+ self._bucket_counts[idx] += 1+ self._count += 1+ self._sum += value++ def copy_values(self) -> tuple[tuple[int, ...], int, float]:+ """Return immutable copies of bucket counts, sample count, and sum."""+ with self._lock:+ return tuple(self._bucket_counts), self._count, self._sum+++class _Counter:+ """Public lightweight handle used by callers to mutate one counter family."""++ def __init__(self, family: _CounterFamily) -> None:+ """Bind a public counter handle to one internal counter family."""+ self._family = family++ def inc(self, amount: float = 1.0, *, labels: Mapping[str, str] | None = None) -> None:+ """Increment one series in the bound counter family using the public wrapper API."""+ self._family.inc(labels=labels, amount=amount)+++class _Gauge:+ """Public lightweight handle used by callers to mutate one gauge family."""++ def __init__(self, family: _GaugeFamily) -> None:+ """Bind a public gauge handle to one internal gauge family."""+ self._family = family++ def set(self, value: float, *, labels: Mapping[str, str] | None = None) -> None:+ """Set one series in the bound gauge family through the public wrapper API."""+ self._family.set(labels=labels, value=value)++ def inc(self, amount: float = 1.0, *, labels: Mapping[str, str] | None = None) -> None:+ """Increase one series in the bound gauge family by a positive delta."""+ self._family.add(labels=labels, delta=amount)++ def dec(self, amount: float = 1.0, *, labels: Mapping[str, str] | None = None) -> None:+ """Decrease one series in the bound gauge family by a positive amount."""+ self._family.add(labels=labels, delta=-amount)+++class _Histogram:+ """Public lightweight handle used by callers to mutate one histogram family."""++ def __init__(self, family: _HistogramFamily) -> None:+ """Bind a public histogram handle to one internal histogram family."""+ self._family = family++ def observe(self, value: float, *, labels: Mapping[str, str] | None = None) -> None:+ """Record one observation for a series in the bound histogram family wrapper."""+ self._family.observe(labels=labels, value=value)diff --git a/openviking/metrics/core/runtime.py b/openviking/metrics/core/runtime.pynew file mode 100644index 000000000..3d92be25d--- /dev/null+++ b/openviking/metrics/core/runtime.py@@ -0,0 +1,65 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Runtime utilities for the metrics subsystem.++Currently this module defines a small in-process event router:+- DataSources emit events by calling `openviking.metrics.global_api.try_dispatch_event(...)`.+- The global API routes events to the matching Collector handler.++This avoids wiring MetricRegistry into business code and keeps the write-path+restricted to Collector implementations.+"""++from __future__ import annotations++from collections.abc import Callable+from typing import Any+++class EventCollectorRouter:+ """+ A minimal in-process dispatcher for event-driven metrics collection.++ The router is intentionally small: it only maps one event name to one handler and invokes+ that handler when the event is dispatched. This keeps event-driven metrics lightweight and+ preserves the architectural rule that collectors, not business code, translate events into+ registry writes.+ """++ def __init__(self) -> None:+ """+ Initialize an empty event-to-handler mapping.++ Handlers are keyed by the normalized string form of the event name and are expected to+ accept a single dictionary payload.+ """+ self._handlers: dict[str, Callable[[dict[str, Any]], None]] = {}++ def register(self, event_name: str, handler: Callable[[dict[str, Any]], None]) -> None:+ """+ Register or replace the handler for a specific metrics event.++ Args:+ event_name: Logical event name emitted by a DataSource or bridge layer.+ handler: Callable that receives the event payload and performs the corresponding+ collector-side metric translation.+ """+ self._handlers[str(event_name)] = handler++ def dispatch(self, event_name: str, payload: dict[str, Any]) -> None:+ """+ Deliver an event payload to the registered handler, if one exists.++ Args:+ event_name: Logical event name to dispatch.+ payload: Event payload already normalized into a dictionary.++ If no handler is registered, the event is ignored silently. This is intentional because+ metrics are a side-channel and must not affect the correctness of the business flow.+ """+ handler = self._handlers.get(str(event_name))+ if handler is None:+ return+ handler(payload)diff --git a/openviking/metrics/core/types.py b/openviking/metrics/core/types.pynew file mode 100644index 000000000..a2b48c8c2--- /dev/null+++ b/openviking/metrics/core/types.py@@ -0,0 +1,100 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Shared metric utility types.++This module enforces label safety rules that protect Prometheus from cardinality explosions:+- Only allow label keys that match Prometheus label naming rules.+- Explicitly forbid label keys that are known to be high-cardinality in OpenViking+ (e.g., session_id, resource_uri, query, prompt, url).++All labels are normalized into a sorted tuple of (key, value) pairs to provide:+- A stable internal series identifier+- Deterministic output ordering in Prometheus exposition+"""++from __future__ import annotations++import re+from dataclasses import dataclass+from typing import Mapping++_LABEL_KEY_RE = re.compile(r"^[a-zA-Z_][a-zA-Z0-9_]*$")++FORBIDDEN_LABEL_KEYS: set[str] = {+ "user_id",+ "session_id",+ "resource_uri",+ "error_message",+ "query",+ "prompt",+ "path",+ "url",+}+++def normalize_labels(labels: Mapping[str, str] | None) -> tuple[tuple[str, str], ...]:+ """+ Normalize a label mapping into the canonical internal representation.++ Args:+ labels: Optional label dictionary provided by a collector write path.++ Returns:+ A tuple of `(key, value)` pairs sorted by key, suitable for use as a stable internal+ series identifier.++ Raises:+ ValueError: If a label key is invalid under Prometheus naming rules or belongs to the+ forbidden high-cardinality set.+ """+ if not labels:+ return ()+ items: list[tuple[str, str]] = []+ for k, v in labels.items():+ key = str(k)+ if not _LABEL_KEY_RE.match(key):+ raise ValueError(f"invalid label key: {key}")+ if key in FORBIDDEN_LABEL_KEYS:+ raise ValueError(f"forbidden label key: {key}")+ items.append((key, str(v)))+ items.sort(key=lambda x: x[0])+ return tuple(items)+++def render_labels(normalized: tuple[tuple[str, str], ...]) -> str:+ """+ Render normalized labels into Prometheus exposition syntax.++ Args:+ normalized: Canonical normalized label tuple.++ Returns:+ An empty string when no labels are present, or a `{k="v",...}` block ready for text+ exposition output.+ """+ if not normalized:+ return ""+ parts = [f'{k}="{_escape_label_value(v)}"' for k, v in normalized]+ return "{" + ",".join(parts) + "}"+++def _escape_label_value(value: str) -> str:+ """Escape a label value according to Prometheus text exposition rules."""+ return value.replace("\\", "\\\\").replace("\n", "\\n").replace('"', '\\"')+++@dataclass(frozen=True, slots=True)+class MetricSeriesId:+ """+ Immutable identifier for a single metric series in the in-process registry.++ Attributes:+ name: Metric family name.+ labels: Canonical normalized label tuple uniquely identifying one concrete series within+ that metric family.+ """++ name: str+ labels: tuple[tuple[str, str], ...]diff --git a/openviking/metrics/datasources/__init__.py b/openviking/metrics/datasources/__init__.pynew file mode 100644index 000000000..26f2d73af--- /dev/null+++ b/openviking/metrics/datasources/__init__.py@@ -0,0 +1,35 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""Datasource entry points for event, state, probe, and aggregated metrics inputs."""++from .base import (+ DomainStatsMetricDataSource,+ EventMetricDataSource,+ ProbeMetricDataSource,+ StateMetricDataSource,+)+from .cache import CacheEventDataSource+from .encryption import EncryptionEventDataSource+from .http import HttpRequestLifecycleDataSource+from .model_usage import EmbeddingEventDataSource, VLMEventDataSource+from .resource import ResourceIngestionEventDataSource+from .retrieval import RetrievalStatsDataSource+from .session import SessionLifecycleDataSource+from .telemetry_bridge import TelemetryBridgeEventDataSource++__all__ = [+ "EventMetricDataSource",+ "StateMetricDataSource",+ "DomainStatsMetricDataSource",+ "ProbeMetricDataSource",+ "RetrievalStatsDataSource",+ "EmbeddingEventDataSource",+ "VLMEventDataSource",+ "SessionLifecycleDataSource",+ "HttpRequestLifecycleDataSource",+ "CacheEventDataSource",+ "ResourceIngestionEventDataSource",+ "EncryptionEventDataSource",+ "TelemetryBridgeEventDataSource",+]diff --git a/openviking/metrics/datasources/base.py b/openviking/metrics/datasources/base.pynew file mode 100644index 000000000..34a1af850--- /dev/null+++ b/openviking/metrics/datasources/base.py@@ -0,0 +1,123 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from typing import Any, Callable, ClassVar++from openviking.metrics.core.base import MetricDataSource, ReadEnvelope+++class EventMetricDataSource(MetricDataSource):+ """+ Shared base for datasources that emit push-style in-process metrics events.++ Event datasources publish normalized payloads into the metrics event router instead of+ writing to the registry directly, keeping business code decoupled from collector internals.+ """++ KIND: ClassVar[str] = "event"++ @classmethod+ def kind(cls) -> str:+ """Return the datasource category identifier used by diagnostics and tests."""+ return cls.KIND++ @staticmethod+ def _emit(event_name: str, payload: dict) -> None:+ """+ Emit one normalized metrics event through the global best-effort event router.++ Missing metrics initialization is handled by the router path so callers can fire events+ without adding explicit metrics-enabled guards around business logic.+ """+ from openviking.metrics.global_api import try_dispatch_event++ try_dispatch_event(str(event_name), dict(payload))+++class StateMetricDataSource(MetricDataSource):+ """+ Base category for datasources that expose pull-style point-in-time subsystem state.++ These datasources usually snapshot in-process managers or trackers and return read envelopes+ that let collectors export stale or default values when reads fail.+ """++ KIND: ClassVar[str] = "state"++ @classmethod+ def kind(cls) -> str:+ """Return the datasource category identifier."""+ return cls.KIND+++class DomainStatsMetricDataSource(MetricDataSource):+ """+ Base category for datasources that expose aggregated domain-level statistics.++ Domain-stat datasources differ from raw state reads in that they return higher-level summary+ structures already grouped by business concepts such as component, model, or queue family.+ """++ KIND: ClassVar[str] = "domain_stats"++ @classmethod+ def kind(cls) -> str:+ """Return the datasource category identifier."""+ return cls.KIND+++class ProbeMetricDataSource(MetricDataSource):+ """+ Base category for datasources that expose readiness or health probe state.++ Probe helpers wrap subsystem-specific checks into normalized dictionaries so collector code+ can emit consistent readiness metrics without repeating error handling.+ """++ KIND: ClassVar[str] = "probe"++ @classmethod+ def kind(cls) -> str:+ """Return the datasource category identifier."""+ return cls.KIND++ def safe_bool_probe(+ self,+ name: str,+ fn: Callable[[], Any],+ *,+ default: bool = False,+ ) -> ReadEnvelope[dict[str, bool]]:+ """Read one boolean probe and wrap the result under the provided probe name."""+ probe_name = str(name)+ return self.safe_read(lambda: {probe_name: bool(fn())}, default={probe_name: bool(default)})++ def safe_tuple_probe(+ self,+ key: str,+ fn: Callable[[], tuple[str, bool]],+ *,+ default_name: str = "unknown",+ ) -> ReadEnvelope[dict[str, tuple[str, bool]]]:+ """Read one named probe result and normalize it into a keyed `(name, ok)` tuple map."""+ k = str(key)++ def _read() -> dict[str, tuple[str, bool]]:+ name, ok = fn()+ return {k: (str(name), bool(ok))}++ return self.safe_read(+ _read,+ default={k: (str(default_name), False)},+ )++ def safe_value_probe(+ self,+ fn: Callable[[], Any],+ *,+ default: Any,+ ) -> ReadEnvelope[Any]:+ """Read one probe value using the generic `safe_read` fallback behavior."""+ return self.safe_read(fn, default=default)diff --git a/openviking/metrics/datasources/cache.py b/openviking/metrics/datasources/cache.pynew file mode 100644index 000000000..047c7accc--- /dev/null+++ b/openviking/metrics/datasources/cache.py@@ -0,0 +1,45 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Cache event datasource.++This datasource standardizes cache hit/miss metrics production under the datasource layer.+It emits bounded low-cardinality events that are later consumed by `CacheCollector`.+"""++from __future__ import annotations++from .base import EventMetricDataSource++_LEVELS = ("L0", "L1", "L2")+++class CacheEventDataSource(EventMetricDataSource):+ """Emit bounded cache hit/miss events for later translation by `CacheCollector`."""++ @staticmethod+ def record_hit(level: str) -> None:+ """+ Emit a cache-hit event for a supported cache level.++ Invalid levels are ignored silently so instrumentation never breaks business logic and+ never expands metrics cardinality with ad-hoc level names.+ """+ lvl = str(level)+ if lvl not in _LEVELS:+ return+ EventMetricDataSource._emit("cache.hit", {"level": lvl})++ @staticmethod+ def record_miss(level: str) -> None:+ """+ Emit a cache-miss event for a supported cache level.++ Invalid levels are ignored silently for the same bounded-cardinality reasons as+ `record_hit`.+ """+ lvl = str(level)+ if lvl not in _LEVELS:+ return+ EventMetricDataSource._emit("cache.miss", {"level": lvl})diff --git a/openviking/metrics/datasources/encryption.py b/openviking/metrics/datasources/encryption.pynew file mode 100644index 000000000..cab1d72da--- /dev/null+++ b/openviking/metrics/datasources/encryption.py@@ -0,0 +1,140 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Encryption metrics DataSources.++The crypto subsystem emits best-effort metrics through these DataSource APIs.+All functions here either:+- emit in-process events (handled by EncryptionCollector), or+- expose readiness probe state (handled by EncryptionProbeCollector).++Instrumentation policy:+- Must never raise and break encryption logic.+- Labels must remain low-cardinality (operation/status/provider/key_version).+"""++from __future__ import annotations++from openviking.metrics.core.base import ReadEnvelope++from .base import EventMetricDataSource, ProbeMetricDataSource+++class EncryptionEventDataSource(EventMetricDataSource):+ """Emit low-cardinality encryption runtime events for `EncryptionCollector`."""++ @staticmethod+ def record_operation(*, operation: str, status: str, duration_seconds: float) -> None:+ """Emit a timed encryption/decryption operation outcome."""+ EventMetricDataSource._emit(+ "encryption.operation",+ {+ "operation": str(operation),+ "status": str(status),+ "duration_seconds": float(duration_seconds),+ },+ )++ @staticmethod+ def record_bytes(*, operation: str, size_bytes: int) -> None:+ """Emit the number of bytes processed by one encryption operation when positive."""+ if size_bytes <= 0:+ return+ EventMetricDataSource._emit(+ "encryption.bytes",+ {+ "operation": str(operation),+ "size_bytes": int(size_bytes),+ },+ )++ @staticmethod+ def record_payload_size(*, operation: str, size_bytes: int) -> None:+ """Emit the logical payload size associated with one encryption operation."""+ if size_bytes < 0:+ return+ EventMetricDataSource._emit(+ "encryption.payload_size",+ {+ "operation": str(operation),+ "size_bytes": int(size_bytes),+ },+ )++ @staticmethod+ def record_auth_failed() -> None:+ """Emit an authentication-failure event for encryption payload verification."""+ EventMetricDataSource._emit("encryption.auth_failed", {})++ @staticmethod+ def record_key_derivation(*, status: str, duration_seconds: float) -> None:+ """Emit the outcome and latency of a key-derivation attempt."""+ EventMetricDataSource._emit(+ "encryption.key_derivation",+ {"status": str(status), "duration_seconds": float(duration_seconds)},+ )++ @staticmethod+ def record_key_load(*, status: str, provider: str, duration_seconds: float) -> None:+ """Emit the outcome and latency of loading a key from the configured provider."""+ EventMetricDataSource._emit(+ "encryption.key_load",+ {+ "status": str(status),+ "provider": str(provider),+ "duration_seconds": float(duration_seconds),+ },+ )++ @staticmethod+ def record_key_cache_hit(*, provider: str) -> None:+ """Emit a key-cache hit event for the given provider."""+ EventMetricDataSource._emit("encryption.key_cache_hit", {"provider": str(provider)})++ @staticmethod+ def record_key_cache_miss(*, provider: str) -> None:+ """Emit a key-cache miss event for the given provider."""+ EventMetricDataSource._emit("encryption.key_cache_miss", {"provider": str(provider)})++ @staticmethod+ def record_key_version_usage(*, key_version: str) -> None:+ """Emit the normalized key version that was used for an encryption operation."""+ EventMetricDataSource._emit(+ "encryption.key_version_usage", {"key_version": str(key_version)}+ )+++class EncryptionProbeDataSource(ProbeMetricDataSource):+ """Read coarse readiness state for the encryption subsystem and configured provider."""++ def __init__(self, *, config_provider) -> None:+ """Store the config provider used to resolve encryption bootstrap configuration."""+ self._config_provider = config_provider++ def read_probe_state(self) -> ReadEnvelope[tuple[bool, str]]:+ """+ Check whether encryption bootstrap succeeds and report the configured provider name.++ Returns:+ A tuple of `(ok_component, provider)` where `ok_component` indicates whether the+ encryption subsystem could be initialized successfully.+ """+ from openviking.crypto.config import bootstrap_encryption++ def _read() -> tuple[bool, str]:+ cfg = self._config_provider()+ provider = str(getattr(getattr(cfg, "encryption", None), "provider", "") or "unknown")+ bootstrap_encryption()+ return True, provider++ default_provider = "unknown"+ try:+ cfg = self._config_provider()+ default_provider = str(+ getattr(getattr(cfg, "encryption", None), "provider", "") or "unknown"+ )+ except Exception:+ default_provider = "unknown"++ return self.safe_value_probe(_read, default=(False, default_provider))diff --git a/openviking/metrics/datasources/http.py b/openviking/metrics/datasources/http.pynew file mode 100644index 000000000..aebc53d8c--- /dev/null+++ b/openviking/metrics/datasources/http.py@@ -0,0 +1,67 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+HTTP request lifecycle DataSource.++This DataSource emits two event types:+- http.request: for request count and duration metrics.+- http.inflight: for inflight request gauge.++The actual MetricRegistry writes are performed by HTTPCollector.+"""++from __future__ import annotations++from .base import EventMetricDataSource+++class HttpRequestLifecycleDataSource(EventMetricDataSource):+ """+ Emit normalized HTTP lifecycle events for request counters, latency, and inflight gauges.++ The datasource intentionally exposes a small, route-template-based payload shape so HTTP+ collectors can remain agnostic of framework request objects.+ """++ @staticmethod+ def record_request(+ *,+ method: str,+ route: str,+ status: str,+ duration_seconds: float,+ account_id: str | None = None,+ ) -> None:+ """+ Emit a completed-request event with normalized method, route, status, and duration data.++ The payload is plain data so middleware can attach an explicit `account_id` when the+ authenticated tenant is already known.+ """+ EventMetricDataSource._emit(+ "http.request",+ {+ "method": str(method),+ "route": str(route),+ "status": str(status),+ "duration_seconds": float(duration_seconds),+ "account_id": None if account_id is None else str(account_id),+ },+ )++ @staticmethod+ def set_inflight(*, route: str, value: float, account_id: str | None = None) -> None:+ """+ Emit the current inflight request gauge value for one normalized route template.++ Callers are expected to supply the post-delta value so the datasource remains stateless.+ """+ EventMetricDataSource._emit(+ "http.inflight",+ {+ "route": str(route),+ "value": float(value),+ "account_id": None if account_id is None else str(account_id),+ },+ )diff --git a/openviking/metrics/datasources/model_usage.py b/openviking/metrics/datasources/model_usage.pynew file mode 100644index 000000000..cdddf9c7e--- /dev/null+++ b/openviking/metrics/datasources/model_usage.py@@ -0,0 +1,223 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Model usage related DataSources.++This file provides two categories of DataSource APIs:+1) DomainStats (pull): `ModelUsageDataSource.read_model_usage()` reads aggregated cumulative+ usage from in-process model instances / shared token trackers. This is used by+ `ModelUsageCollector` which converts cumulative stats into Prometheus Counters via deltas.+2) Event (push): `VLMEventDataSource` and `EmbeddingEventDataSource` emit per-call events.+ These events are routed to Event collectors via `try_dispatch_event`.++Note: DataSources are not allowed to write into MetricRegistry directly in this architecture.+They only emit events or expose read APIs. Collectors are the only writers.+"""++from __future__ import annotations++from typing import Any, Callable++from openviking.metrics.core.base import ReadEnvelope++from .base import DomainStatsMetricDataSource, EventMetricDataSource+++class ModelUsageDataSource(DomainStatsMetricDataSource):+ """+ DomainStats source for aggregated model usage.++ The return value is intentionally aligned with `TokenUsageTracker.to_dict()`:+ - usage_by_model[model_name].usage_by_provider[provider] contains cumulative counters:+ prompt_tokens, completion_tokens, total_tokens, call_count+ """++ def __init__(self, *, config_provider: Callable[[], Any], service: Any = None) -> None:+ """+ Args:+ config_provider: Fallback provider to obtain OpenVikingConfig when `service` is absent.+ service: Optional server service instance. When provided, we prefer its in-memory+ config to avoid reloading configuration from disk.+ """+ self._config_provider = config_provider+ self._service = service++ def read_model_usage(self) -> ReadEnvelope[dict[str, dict]]:+ """+ Read cumulative model usage for configured model types.++ Returns:+ A dict keyed by model_type: "vlm" / "embedding" / "rerank".+ Each value contains:+ - "available": whether fresh usage data for that type was obtained+ - "usage_by_model": normalized cumulative usage payload, empty when unavailable++ Notes:+ - This method is best-effort: it skips types that are not configured/available.+ - It must not create additional long-lived clients except where unavoidable.+ (Rerank currently uses RerankClient.from_config to access the shared token tracker.)+ """+ config_env = self.safe_read(+ lambda: getattr(self._service, "_config", None) or self._config_provider(),+ default=None,+ )+ if not config_env.ok or config_env.value is None:+ return ReadEnvelope(+ ok=False,+ value={},+ error_type=config_env.error_type,+ error_message=config_env.error_message,+ )+ config = config_env.value++ result: dict[str, dict[str, object]] = {+ "vlm": {"available": False, "usage_by_model": {}},+ "embedding": {"available": False, "usage_by_model": {}},+ "rerank": {"available": False, "usage_by_model": {}},+ }++ try:+ vlm = config.vlm.get_vlm_instance()+ result["vlm"] = {+ "available": True,+ "usage_by_model": _extract_usage_by_model(vlm.get_token_usage(), self),+ }+ except Exception:+ pass++ try:+ embedder = config.embedding.get_embedder()+ result["embedding"] = {+ "available": True,+ "usage_by_model": _extract_usage_by_model(embedder.get_token_usage(), self),+ }+ except Exception:+ pass++ try:+ rerank_cfg = getattr(config, "rerank", None)+ if rerank_cfg is not None and rerank_cfg.is_available():+ from openviking.models.rerank.base import get_shared_rerank_token_usage++ result["rerank"] = {+ "available": True,+ "usage_by_model": _extract_usage_by_model(+ get_shared_rerank_token_usage(),+ self,+ ),+ }+ except Exception:+ pass++ return ReadEnvelope(ok=True, value=result)+++class VLMEventDataSource(EventMetricDataSource):+ """+ Event datasource for per-call VLM usage events.++ The payload emitted here is intentionally narrow and collector-facing: model identity, token+ counts, latency, and optional account context.+ """++ @staticmethod+ def record_call(+ *,+ provider: str,+ model_name: str,+ duration_seconds: float,+ prompt_tokens: int,+ completion_tokens: int,+ account_id: str | None = None,+ ) -> None:+ """+ Emit one VLM call event with model identity, latency, token usage, and account context.++ The caller is expected to provide already-normalized provider/model identifiers and the+ final token counts that should be reflected in Prometheus usage metrics.+ """+ EventMetricDataSource._emit(+ "vlm.call",+ {+ "provider": str(provider),+ "model_name": str(model_name),+ "duration_seconds": float(duration_seconds),+ "prompt_tokens": int(prompt_tokens),+ "completion_tokens": int(completion_tokens),+ "account_id": None if account_id is None else str(account_id),+ },+ )+++class EmbeddingEventDataSource(EventMetricDataSource):+ """+ Event datasource for embedding request outcomes.++ Success and error outcomes are modeled as separate event names so collectors can produce the+ appropriate request, latency, and error-code series without inspecting exception objects.+ """++ @staticmethod+ def record_success(*, latency_seconds: float, account_id: str | None = None) -> None:+ """+ Emit an embedding success event with the observed end-to-end latency.++ The latency is expected to cover the full embedding workflow segment that the caller wants+ reflected in metrics, not just the raw provider round-trip time.+ """+ EventMetricDataSource._emit(+ "embedding.success",+ {+ "latency_seconds": float(latency_seconds),+ "account_id": None if account_id is None else str(account_id),+ },+ )++ @staticmethod+ def record_error(*, error_code: str, account_id: str | None = None) -> None:+ """+ Emit an embedding error event with a normalized error code label.++ Callers should translate provider- or worker-specific failures into bounded error codes+ before invoking this datasource.+ """+ EventMetricDataSource._emit(+ "embedding.error",+ {+ "error_code": str(error_code or "unknown"),+ "account_id": None if account_id is None else str(account_id),+ },+ )+++def _extract_usage_by_model(token_usage: dict, datasource: DomainStatsMetricDataSource) -> dict:+ """+ Extract and normalize a `usage_by_model` mapping from TokenUsageTracker-like payloads.++ The helper keeps only the counters required by `ModelUsageCollector` and normalizes model and+ provider keys into stable non-empty strings.+ """+ token_usage = datasource.as_dict(token_usage)+ usage_by_model = datasource.as_dict(token_usage.get("usage_by_model"))+ normalized: dict[str, dict] = {}+ for model_name, model_payload in usage_by_model.items():+ model_key = datasource.normalize_str(model_name)+ normalized_model = datasource.as_dict(model_payload)+ usage_by_provider = datasource.as_dict(normalized_model.get("usage_by_provider"))+ normalized_providers: dict[str, dict] = {}+ for provider_name, provider_payload in usage_by_provider.items():+ provider_key = datasource.normalize_str(provider_name)+ provider_dict = datasource.as_dict(provider_payload)+ normalized_providers[provider_key] = {+ "prompt_tokens": datasource.as_int(provider_dict.get("prompt_tokens"), default=0),+ "completion_tokens": datasource.as_int(+ provider_dict.get("completion_tokens"), default=0+ ),+ "total_tokens": datasource.as_int(provider_dict.get("total_tokens"), default=0),+ "call_count": datasource.as_int(provider_dict.get("call_count"), default=0),+ }+ normalized[model_key] = {+ "usage_by_provider": normalized_providers,+ }+ return normalizeddiff --git a/openviking/metrics/datasources/observer_state.py b/openviking/metrics/datasources/observer_state.pynew file mode 100644index 000000000..86544ac87--- /dev/null+++ b/openviking/metrics/datasources/observer_state.py@@ -0,0 +1,153 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++import time+from typing import Any++from openviking.metrics.core.base import ReadEnvelope+from openviking.storage.transaction import get_lock_manager+from openviking_cli.utils import run_async++from .base import DomainStatsMetricDataSource, StateMetricDataSource+++class ObserverStateDataSource(DomainStatsMetricDataSource):+ """+ Read observer-backed component objects from the in-memory debug service.++ The datasource does not compute health itself; it exposes the raw observer component objects+ so downstream collectors can derive health and error counts consistently.+ """++ def __init__(self, *, service: Any = None) -> None:+ """+ Store the optional service object used to reach the in-memory debug observer.++ Passing `service=None` keeps the datasource usable in tests and degraded environments+ where the debug service is intentionally absent.+ """+ self._service = service++ def read_component_states(self) -> ReadEnvelope[dict[str, Any]]:+ """+ Read the current observer component objects exposed by the debug service.++ Returns:+ A mapping of component names to observer-backed objects, or an empty mapping when+ the debug observer is not currently available.+ """++ def _read() -> dict[str, Any]:+ observer = None+ if self._service is not None:+ observer = getattr(getattr(self._service, "debug", None), "observer", None)+ if observer is None:+ return {}+ components = {+ self.normalize_str("queue"): observer.queue,+ self.normalize_str("models"): observer.models,+ self.normalize_str("lock"): observer.lock,+ self.normalize_str("retrieval"): observer.retrieval,+ }+ try:+ components[self.normalize_str("vikingdb")] = observer.vikingdb(ctx=None)+ except Exception:+ components[self.normalize_str("vikingdb")] = None+ return self.as_dict(components)++ return self.safe_read(_read, default={})+++class LockStateDataSource(StateMetricDataSource):+ """+ Read lock-manager counters needed by lock-related state collectors.++ The datasource inspects active lock handles and derives a stale-handle count using the+ current in-process timeout heuristic.+ """++ def read_lock_state(self) -> ReadEnvelope[tuple[int, int, int]]:+ """+ Read active and stale lock counts from the global transaction lock manager.++ Returns:+ A tuple of `(active_locks, waiting_locks, stale_handles)` where waiting locks are+ currently not tracked separately and therefore remain `0`.+ """++ def _read() -> tuple[int, int, int]:+ lock_manager = get_lock_manager()+ handles = lock_manager.get_active_handles()+ active = 0+ stale = 0+ now = time.time()+ for handle in handles.values():+ try:+ active += len(getattr(handle, "locks", []) or [])+ last = getattr(handle, "last_active_at", None)+ if last is not None and (now - float(last)) > 600:+ stale += 1+ except Exception:+ continue+ return active, 0, stale++ return self.safe_read(_read, default=(0, 0, 0))+++class VikingDBStateDataSource(StateMetricDataSource):+ """+ Read health and size signals from the active VikingDB manager, when available.++ The datasource composes multiple best-effort reads into a single tuple so collectors can emit+ VikingDB health and vector-count gauges from one normalized snapshot.+ """++ def __init__(self, *, service: Any = None) -> None:+ """Store the optional service object used to resolve the active VikingDB manager."""+ self._service = service++ def read_vikingdb_state(self) -> ReadEnvelope[tuple[str, bool, int]]:+ """+ Read the collection name, health status, and approximate row count for VikingDB.++ Returns:+ A tuple of `(collection_name, healthy, count)`. Missing services or failures fall+ back to safe default values so metrics collection remains best-effort.+ """+ vikingdb = None+ if self._service is not None:+ vikingdb = getattr(self._service, "_vikingdb_manager", None) or getattr(+ self._service, "vikingdb", None+ )+ if vikingdb is None:+ return ReadEnvelope(+ ok=False,+ value=("default", False, 0),+ error_type="NotAvailable",+ error_message="vikingdb manager missing",+ )++ collection = self.normalize_str(+ getattr(vikingdb, "collection_name", "default"), default="default"+ )+ ok_env = self.safe_read_async(+ lambda: vikingdb.health_check(),+ default=False,+ runner=run_async,+ )+ count_env = self.safe_read_async(+ lambda: vikingdb.count(filter=None, ctx=None),+ default=0,+ runner=run_async,+ )+ ok = bool(ok_env.value)+ count = self.as_int(count_env.value, default=0)+ envelope_ok = bool(ok_env.ok and count_env.ok)+ return ReadEnvelope(+ ok=envelope_ok,+ value=(collection, ok, count),+ error_type=ok_env.error_type or count_env.error_type,+ error_message=ok_env.error_message or count_env.error_message,+ )diff --git a/openviking/metrics/datasources/probes.py b/openviking/metrics/datasources/probes.pynew file mode 100644index 000000000..444428b09--- /dev/null+++ b/openviking/metrics/datasources/probes.py@@ -0,0 +1,138 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from typing import Any, Callable++from openviking.metrics.core.base import ReadEnvelope+from openviking.storage.queuefs import get_queue_manager+from openviking.storage.viking_fs import get_viking_fs+from openviking_cli.utils import run_async++from .base import ProbeMetricDataSource+++class ServiceProbeDataSource(ProbeMetricDataSource):+ """Read high-level readiness probes from the service object and FastAPI application."""++ def __init__(self, *, app=None, service=None) -> None:+ """Store the optional app and service objects needed to read readiness indicators."""+ self._app = app+ self._service = service++ def read_probe_state(self) -> ReadEnvelope[dict[str, bool]]:+ """+ Read coarse service and API-key-manager readiness flags.++ Returns:+ A mapping of probe names to boolean readiness values derived from the current+ service initialization state and the FastAPI application's API key manager.+ """++ def _read():+ ready = bool(+ getattr(self._service, "initialized", False)+ or getattr(self._service, "_initialized", False)+ )+ api_key_ready = False+ if self._app is not None:+ api_key_ready = getattr(self._app.state, "api_key_manager", None) is not None+ return {"service_readiness": ready, "api_key_manager_readiness": api_key_ready}++ return self.safe_value_probe(+ _read,+ default={"service_readiness": False, "api_key_manager_readiness": False},+ )+++class StorageProbeDataSource(ProbeMetricDataSource):+ """+ Read readiness state for the AGFS-backed storage subsystem.++ The probe intentionally reduces storage health to a simple boolean readiness signal that can+ be exported consistently across environments.+ """++ def read_probe_state(self) -> ReadEnvelope[dict[str, bool]]:+ """+ Check whether the AGFS-backed VikingFS storage dependency can be resolved.++ Failure to resolve the storage dependency is exported as a best-effort false value rather+ than propagating an exception into metrics collection.+ """+ return self.safe_bool_probe("agfs", lambda: bool(get_viking_fs().agfs))+++class RetrievalBackendProbeDataSource(ProbeMetricDataSource):+ """+ Read readiness state for the retrieval backend currently used by the service.++ The probe currently assumes VikingDB is the active backend and normalizes its health check+ result into a single named readiness metric.+ """++ def __init__(self, *, service=None) -> None:+ """Store the optional service used to resolve the active retrieval backend manager."""+ self._service = service++ def read_probe_state(self) -> ReadEnvelope[dict[str, bool]]:+ """+ Check whether the active VikingDB retrieval backend passes its health check.++ Missing service wiring is treated as a negative readiness result instead of an exception.+ """++ def _check() -> bool:+ vikingdb = None+ if self._service is not None:+ vikingdb = getattr(self._service, "_vikingdb_manager", None) or getattr(+ self._service, "vikingdb", None+ )+ if vikingdb is None:+ return False+ return bool(run_async(vikingdb.health_check()))++ return self.safe_bool_probe("vikingdb", _check)+++class ModelProviderProbeDataSource(ProbeMetricDataSource):+ """+ Read readiness state for the configured model provider client.++ The datasource reports both the configured provider name and whether the client can be+ materialized successfully from the current configuration.+ """++ def __init__(self, *, config_provider: Callable[[], Any]) -> None:+ """Store the config provider used to resolve the current VLM provider configuration."""+ self._config_provider = config_provider++ def read_probe_state(self) -> ReadEnvelope[dict[str, tuple[str, bool]]]:+ """+ Resolve the configured VLM provider name and whether its client can be created.++ Returns:+ A mapping with one `provider` entry containing `(provider_name, healthy)`.+ """++ def _probe() -> tuple[str, bool]:+ config = self._config_provider()+ provider = str(getattr(getattr(config, "vlm", None), "provider", "") or "unknown")+ _ = config.vlm.get_vlm_instance()+ return provider, True++ return self.safe_tuple_probe("provider", _probe, default_name="unknown")+++class AsyncSystemProbeDataSource(ProbeMetricDataSource):+ """+ Read readiness state for shared async-system infrastructure such as queue management.++ The queue manager is used as the coarse-grained readiness indicator for the shared async+ processing subsystem.+ """++ def read_probe_state(self) -> ReadEnvelope[dict[str, bool]]:+ """Check whether the shared async queue subsystem can be resolved successfully."""+ return self.safe_bool_probe("queue", lambda: bool(get_queue_manager()))diff --git a/openviking/metrics/datasources/queue.py b/openviking/metrics/datasources/queue.pynew file mode 100644index 000000000..6077a3cac--- /dev/null+++ b/openviking/metrics/datasources/queue.py@@ -0,0 +1,33 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from openviking.metrics.core.base import ReadEnvelope+from openviking.storage.queuefs import get_queue_manager+from openviking_cli.utils import run_async++from .base import StateMetricDataSource+++class QueuePipelineStateDataSource(StateMetricDataSource):+ """+ Read the current queue-pipeline status snapshot from the shared queue manager.++ The datasource translates the queue manager's async status check into the synchronous+ `ReadEnvelope` contract expected by state collectors.+ """++ def read_queue_status(self) -> ReadEnvelope[dict]:+ """+ Read the current queue-manager status snapshot used by queue-related collectors.++ The underlying queue manager API is async, so the datasource uses `run_async` to bridge+ into the synchronous collector refresh path.+ """+ queue_manager = get_queue_manager()+ return self.safe_read_async(+ lambda: queue_manager.check_status(),+ default={},+ runner=run_async,+ )diff --git a/openviking/metrics/datasources/resource.py b/openviking/metrics/datasources/resource.pynew file mode 100644index 000000000..cb67d97fa--- /dev/null+++ b/openviking/metrics/datasources/resource.py@@ -0,0 +1,62 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from .base import EventMetricDataSource+++class ResourceIngestionEventDataSource(EventMetricDataSource):+ """+ Emit resource ingestion lifecycle events consumed by `ResourceIngestionCollector`.++ Resource processing already knows stage names and wait durations at the instrumentation call+ site, so this datasource simply normalizes those facts into event payloads.+ """++ @staticmethod+ def record_stage(+ *,+ stage: str,+ status: str,+ duration_seconds: float,+ account_id: str | None = None,+ ) -> None:+ """+ Emit the outcome and latency of one named resource-processing stage.++ The event represents a completed stage execution, not a streaming update while the stage+ is still in progress.+ """+ payload = {+ "stage": str(stage),+ "status": str(status),+ "duration_seconds": float(duration_seconds),+ }+ if account_id and str(account_id).strip():+ payload["account_id"] = str(account_id).strip()+ EventMetricDataSource._emit(+ "resource.stage",+ payload,+ )++ @staticmethod+ def record_wait(+ *, operation: str, duration_seconds: float, account_id: str | None = None+ ) -> None:+ """+ Emit the wait time spent on a named resource-ingestion operation.++ This is typically used for queueing or dependency wait segments that are operationally+ useful but distinct from active resource-processing stages.+ """+ payload = {+ "operation": str(operation),+ "duration_seconds": float(duration_seconds),+ }+ if account_id and str(account_id).strip():+ payload["account_id"] = str(account_id).strip()+ EventMetricDataSource._emit(+ "resource.wait",+ payload,+ )diff --git a/openviking/metrics/datasources/retrieval.py b/openviking/metrics/datasources/retrieval.pynew file mode 100644index 000000000..271a4bb92--- /dev/null+++ b/openviking/metrics/datasources/retrieval.py@@ -0,0 +1,54 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Retrieval stats DataSource.++This DataSource provides a single entry point to record retrieval outcomes.+It emits an aggregated `retrieval.completed` event which is mapped to Prometheus+metrics by RetrievalCollector.++Why event-based?+The retrieval pipeline already knows the outcome (result_count, latency, rerank flags)+at the call site. Emitting a single event keeps the instrumentation lightweight while+keeping MetricRegistry writes in collectors.+"""++from __future__ import annotations++from .base import EventMetricDataSource+++class RetrievalStatsDataSource(EventMetricDataSource):+ """+ Emit aggregated retrieval outcome events for later collector-side translation.++ Retrieval code already knows the final result count, latency, and rerank decisions at the+ call site, so an event datasource keeps instrumentation lightweight and registry-free.+ """++ @staticmethod+ def record_retrieval(+ *,+ context_type: str,+ result_count: int,+ latency_seconds: float,+ rerank_used: bool = False,+ rerank_fallback: bool = False,+ ) -> None:+ """+ Emit the final retrieval outcome, latency, and rerank flags for one request.++ The payload intentionally represents one completed retrieval operation rather than a live+ progress stream.+ """+ EventMetricDataSource._emit(+ "retrieval.completed",+ {+ "context_type": str(context_type or "unknown"),+ "result_count": int(result_count),+ "latency_seconds": float(latency_seconds),+ "rerank_used": bool(rerank_used),+ "rerank_fallback": bool(rerank_fallback),+ },+ )diff --git a/openviking/metrics/datasources/session.py b/openviking/metrics/datasources/session.pynew file mode 100644index 000000000..3f0357739--- /dev/null+++ b/openviking/metrics/datasources/session.py@@ -0,0 +1,51 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from .base import EventMetricDataSource+++class SessionLifecycleDataSource(EventMetricDataSource):+ """+ Emit session lifecycle events consumed by `SessionCollector`.++ The datasource keeps session instrumentation at the business boundary by expressing lifecycle+ changes as normalized event payloads instead of direct registry writes.+ """++ @staticmethod+ def record_lifecycle(*, action: str, status: str) -> None:+ """+ Emit the outcome of a session lifecycle action such as create, get, delete, or commit.++ The payload is expected to describe the final outcome of the action rather than+ intermediate progress within the session workflow.+ """+ EventMetricDataSource._emit(+ "session.lifecycle",+ {"action": str(action), "status": str(status)},+ )++ @staticmethod+ def record_contexts_used(*, action: str, delta: int = 1) -> None:+ """+ Emit the number of additional contexts consumed by a session-level action.++ Non-positive deltas are ignored so callers can pass computed increments without needing a+ separate guard.+ """+ if delta <= 0:+ return+ EventMetricDataSource._emit(+ "session.contexts_used",+ {"action": str(action), "delta": int(delta)},+ )++ @staticmethod+ def record_archive(*, status: str) -> None:+ """Emit the normalized outcome of one session archive attempt."""+ EventMetricDataSource._emit(+ "session.archive",+ {"status": str(status)},+ )diff --git a/openviking/metrics/datasources/task.py b/openviking/metrics/datasources/task.pynew file mode 100644index 000000000..ef32af6b3--- /dev/null+++ b/openviking/metrics/datasources/task.py@@ -0,0 +1,28 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++from openviking.metrics.core.base import ReadEnvelope+from openviking.service.task_tracker import get_task_tracker++from .base import StateMetricDataSource+++class TaskStateDataSource(StateMetricDataSource):+ """+ Read task-count snapshots from the shared task tracker service.++ This datasource exposes grouped task totals as a point-in-time state read so task-related+ collectors can publish gauges without depending on tracker internals.+ """++ def read_task_counts(self) -> ReadEnvelope[dict]:+ """+ Read the current task counts grouped by task type from the global task tracker.++ The read is best-effort and returns a `ReadEnvelope` so collectors can export fallback or+ stale values if the tracker lookup fails.+ """+ tracker = get_task_tracker()+ return self.safe_read(tracker.snapshot_counts_by_type, default={})diff --git a/openviking/metrics/datasources/telemetry_bridge.py b/openviking/metrics/datasources/telemetry_bridge.pynew file mode 100644index 000000000..683d8bc7c--- /dev/null+++ b/openviking/metrics/datasources/telemetry_bridge.py@@ -0,0 +1,35 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Telemetry bridge event datasource.++This datasource emits the aggregated request/operation telemetry summary as a single+best-effort event. `TelemetryBridgeCollector` consumes that event and translates it into+Prometheus metrics.+"""++from __future__ import annotations++from typing import Any, Mapping++from .base import EventMetricDataSource+++class TelemetryBridgeEventDataSource(EventMetricDataSource):+ """+ Emit aggregated telemetry summaries for later translation by `TelemetryBridgeCollector`.++ This datasource forms the bridge between the runtime telemetry subsystem and Prometheus+ collectors that turn higher-level summaries into multiple metric family updates.+ """++ @staticmethod+ def record_summary(summary: Mapping[str, Any]) -> None:+ """+ Emit a best-effort telemetry summary event using a plain-dictionary payload copy.++ Copying the summary into a plain dict prevents downstream consumers from depending on the+ mutability or concrete type of the source telemetry object.+ """+ EventMetricDataSource._emit("telemetry.summary", {"summary": dict(summary)})diff --git a/openviking/metrics/exporters/__init__.py b/openviking/metrics/exporters/__init__.pynew file mode 100644index 000000000..118902a47--- /dev/null+++ b/openviking/metrics/exporters/__init__.py@@ -0,0 +1,8 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""Exporter entry points that render the shared metrics registry into external formats."""++from .prometheus import PrometheusExporter++__all__ = ["PrometheusExporter"]diff --git a/openviking/metrics/exporters/prometheus.py b/openviking/metrics/exporters/prometheus.pynew file mode 100644index 000000000..38a97616e--- /dev/null+++ b/openviking/metrics/exporters/prometheus.py@@ -0,0 +1,203 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Prometheus exporter implementation.++Key behaviors:+- On each export, attempts to refresh all registered collectors (best-effort).+- Renders the current registry state into Prometheus text exposition format.++The exporter is intentionally tolerant:+- Collector refresh errors are swallowed (metrics must not break `/metrics`).+- Families with no samples are still exported as a single zero sample when possible,+ to keep dashboards stable.+"""++from __future__ import annotations++from typing import Iterable++from openviking.metrics.core.base import MetricExporter+from openviking.metrics.core.types import render_labels+++class PrometheusExporter(MetricExporter):+ """+ Render the in-process registry into Prometheus text exposition format.++ This exporter optionally refreshes refresh-managed collectors before rendering, then walks+ the registry families and serializes counters, gauges, histograms, and dropped-series+ diagnostics into a single text payload suitable for `/metrics`.+ """++ def __init__(+ self, *, registry, collector_manager=None, refresh_deadline_seconds: float = 1.0+ ) -> None:+ """+ Initialize the Prometheus exporter.++ Args:+ registry: The in-process metric registry that stores current metric values.+ collector_manager: Optional refresh orchestrator invoked before each export.+ refresh_deadline_seconds: Best-effort time budget for the pre-render refresh stage.+ """+ self._registry = registry+ self._collector_manager = collector_manager+ self._refresh_deadline_seconds = float(refresh_deadline_seconds)++ async def export(self) -> str:+ """+ Refresh collectors if configured and return the Prometheus exposition payload.++ Collector refresh remains best-effort: refresh failures are swallowed so observability+ does not break the serving of the metrics endpoint itself.+ """+ if self._collector_manager is not None:+ try:+ await self._collector_manager.refresh_all(+ self._registry,+ deadline_seconds=self._refresh_deadline_seconds,+ )+ except Exception:+ pass+ return self.render()++ def render(self) -> str:+ """+ Serialize the registry into Prometheus text exposition format.++ Returns:+ A newline-terminated text payload containing every registered family in the+ Prometheus exposition layout expected by scrapers and tooling.+ """+ lines: list[str] = []++ # Process counters+ for name, counter in self._registry.iter_counters():+ self._add_help_and_type(lines, name, "counter")+ self._process_counter_series(lines, name, counter)++ # Process gauges+ for name, gauge in self._registry.iter_gauges():+ self._add_help_and_type(lines, name, "gauge")+ self._process_gauge_series(lines, name, gauge)++ # Process histograms+ for name, label_names, bucket_bounds, series_iter in self._registry.iter_histograms():+ self._add_help_and_type(lines, name, "histogram")+ series_list = list(series_iter)+ if not series_list and not label_names:+ self._render_empty_histogram(lines, name, bucket_bounds)+ else:+ for labels, bucket_counts, count, value_sum in series_list:+ self._render_histogram_series(+ lines, name, bucket_bounds, labels, bucket_counts, count, value_sum+ )++ # Process dropped series+ for metric_name, dropped in self._registry.iter_dropped_series():+ lines.append(+ f'openviking_metrics_dropped_series_total{{metric="{metric_name}"}} {dropped}'+ )++ return "\n".join(lines) + "\n"++ def _add_help_and_type(self, lines: list[str], name: str, metric_type: str) -> None:+ """Append the Prometheus `HELP` and `TYPE` prelude for one metric family."""+ lines.append(f"# HELP {name} OpenViking metric.")+ lines.append(f"# TYPE {name} {metric_type}")++ def _process_counter_series(self, lines: list[str], name: str, counter) -> None:+ """+ Render all counter series for one family into the output buffer.++ Empty unlabeled counters are rendered as a single zero sample to keep dashboards and+ alert queries stable before the first real increment occurs.+ """+ if not counter and not self._registry.counter_label_names(name):+ lines.append(f"{name} 0")+ for labels, value in counter:+ lines.append(+ f"{name}{render_labels(labels)} {int(value) if value.is_integer() else value}"+ )++ def _process_gauge_series(self, lines: list[str], name: str, gauge) -> None:+ """+ Render all gauge series for one family into the output buffer.++ Empty unlabeled gauges also emit one zero sample so consumers do not need to special-case+ missing series before initialization.+ """+ if not gauge and not self._registry.gauge_label_names(name):+ lines.append(f"{name} 0")+ for labels, value in gauge:+ lines.append(f"{name}{render_labels(labels)} {value}")++ def _render_empty_histogram(+ self, lines: list[str], name: str, bucket_bounds: tuple[float, ...]+ ) -> None:+ """Render an unlabeled histogram family with zero samples across every bucket."""+ zero_counts = (0,) * (len(bucket_bounds) + 1)+ cumulative = 0+ for bound, c in _iter_histogram_buckets(bucket_bounds, zero_counts):+ cumulative += c+ lines.append(f"{name}_bucket{render_labels((('le', bound),))} {cumulative}")+ lines.append(f"{name}_count 0")+ lines.append(f"{name}_sum 0")++ def _render_histogram_series(+ self,+ lines: list[str],+ name: str,+ bucket_bounds: tuple[float, ...],+ labels: tuple[tuple[str, str], ...],+ bucket_counts: tuple[int, ...],+ count: int,+ value_sum: float,+ ) -> None:+ """+ Render one histogram series with stable label ordering and cumulative bucket counts.++ Histogram buckets are exported cumulatively per the Prometheus exposition rules, while+ `_count` and `_sum` reuse the same sorted label tuple for deterministic output.+ """+ # Sort labels once per series to avoid repeated sorting+ sorted_labels = sorted(labels, key=lambda x: x[0])+ cumulative = 0+ for bound, c in _iter_histogram_buckets(bucket_bounds, bucket_counts):+ cumulative += c+ # Insert 'le' label in correct position without full sort+ merged_labels = self._merge_le_label(sorted_labels, str(bound))+ lines.append(f"{name}_bucket{render_labels(merged_labels)} {cumulative}")+ # Use sorted labels for count and sum to ensure consistent order+ lines.append(f"{name}_count{render_labels(tuple(sorted_labels))} {count}")+ lines.append(f"{name}_sum{render_labels(tuple(sorted_labels))} {value_sum}")++ def _merge_le_label(+ self, sorted_labels: list[tuple[str, str]], le: str+ ) -> tuple[tuple[str, str], ...]:+ """Insert the histogram `le` label into an already sorted label list without resorting."""+ # Find insertion point for 'le' label+ pos = len(sorted_labels)+ for i, (key, _) in enumerate(sorted_labels):+ if key > "le":+ pos = i+ break+ # Create new list with 'le' inserted+ merged = sorted_labels.copy()+ merged.insert(pos, ("le", le))+ return tuple(merged)+++def _iter_histogram_buckets(+ bounds: tuple[float, ...], counts: tuple[int, ...]+) -> Iterable[tuple[str, int]]:+ """+ Yield histogram bucket upper bounds paired with their per-bucket sample counts.++ The final synthetic `+Inf` bucket is always yielded after the configured finite bounds.+ """+ for i, upper in enumerate(bounds):+ yield str(upper), counts[i]+ yield "+Inf", counts[len(bounds)]diff --git a/openviking/metrics/global_api.py b/openviking/metrics/global_api.pynew file mode 100644index 000000000..8269e2575--- /dev/null+++ b/openviking/metrics/global_api.py@@ -0,0 +1,240 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Global metrics bootstrap and event dispatch.++This module provides:+- A process-global MetricRegistry and PrometheusExporter instance for the server.+- A lightweight in-process event router used by DataSources to emit events without+ directly depending on MetricRegistry internals.++Why an event router?+Business code (models, encryption, HTTP middleware, etc.) calls DataSource APIs such as+`VLMEventDataSource.record_call(...)`. DataSources emit events here. Collectors are the+only layer allowed to write into MetricRegistry, keeping the architecture consistent.+"""++from __future__ import annotations++import threading+from typing import Optional++from .account_dimension import (+ configure_metric_account_dimension as _configure_metric_account_dimension_runtime,+)+from .account_dimension import (+ reset_metric_account_dimension,+)+from .bootstrap import create_default_collector_manager+from .collectors.cache import CacheCollector+from .collectors.embedding import EmbeddingCollector+from .collectors.encryption import EncryptionCollector+from .collectors.http import HTTPCollector+from .collectors.resource import ResourceIngestionCollector+from .collectors.retrieval import RetrievalCollector+from .collectors.session import SessionCollector+from .collectors.telemetry_bridge import TelemetryBridgeCollector+from .collectors.vlm import VLMCollector+from .core.registry import MetricRegistry+from .core.runtime import EventCollectorRouter+from .exporters.prometheus import PrometheusExporter++_lock = threading.Lock()+_registry: Optional[MetricRegistry] = None+_exporter: Optional[PrometheusExporter] = None+_event_router: EventCollectorRouter | None = None+++def is_metrics_enabled_from_server_config(config) -> bool:+ """+ Return whether metrics are enabled through either the new or legacy server config path.++ This compatibility helper keeps older `telemetry.prometheus.enabled` deployments working+ while newer code migrates toward `server.metrics.enabled`.+ """+ legacy_enabled = bool(+ getattr(getattr(getattr(config, "telemetry", None), "prometheus", None), "enabled", False)+ )+ metrics_enabled = bool(getattr(getattr(config, "metrics", None), "enabled", False))+ return legacy_enabled or metrics_enabled+++def configure_metric_account_dimension(+ *,+ enabled: bool | None = None,+ metric_allowlist: set[str] | list[str] | tuple[str, ...] | None = None,+ max_active_accounts: int | None = None,+ policy=None,+ resolver=None,+) -> None:+ """+ Configure the process-global account-dimension runtime used by collector write helpers.++ This function is a thin façade over `account_dimension.configure_metric_account_dimension`+ so callers do not need to import policy internals directly from the metrics package.+ """+ _configure_metric_account_dimension_runtime(+ enabled=enabled,+ metric_allowlist=metric_allowlist,+ max_active_accounts=max_active_accounts,+ policy=policy,+ resolver=resolver,+ )+++def init_metrics_from_server_config(+ config, *, app=None, service=None, registry: MetricRegistry | None = None+) -> None:+ """+ Initialize the process-global metrics registry, router, and exporter from server config.++ When metrics are enabled:+ - Creates a MetricRegistry (or uses provided one)+ - Registers default collectors (State/Probe/DomainStats)+ - Builds the in-process event router for Event-style metrics+ - Creates PrometheusExporter and stores it in app.state.metrics_exporter+ When metrics are disabled:+ - Clears all global metrics state+ - Resets account-dimension runtime to its disabled defaults+ - Detaches any exporter reference previously stored on `app.state`+ """+ enabled = is_metrics_enabled_from_server_config(config)+ global _registry, _exporter, _event_router+ with _lock:+ if not enabled:+ _registry = None+ _exporter = None+ _event_router = None+ reset_metric_account_dimension()+ if app is not None:+ app.state.metrics_exporter = None+ return+ metrics_config = getattr(config, "metrics", None)+ account_dimension = getattr(metrics_config, "account_dimension", None)+ configure_metric_account_dimension(+ enabled=bool(getattr(account_dimension, "enabled", False)),+ metric_allowlist=list(getattr(account_dimension, "metric_allowlist", []) or []),+ max_active_accounts=int(getattr(account_dimension, "max_active_accounts", 0) or 0),+ )+ _registry = registry or MetricRegistry()+ collector_manager = create_default_collector_manager(app=app, service=service)+ _event_router = _build_event_router(_registry)+ _exporter = PrometheusExporter(registry=_registry, collector_manager=collector_manager)+ if app is not None:+ app.state.metrics_exporter = _exporter+++def shutdown_metrics(*, app=None) -> None:+ """+ Clear all process-global metrics objects and detach the exporter from the application.++ Shutdown is intentionally idempotent so tests and service teardown paths can call it+ repeatedly without needing to coordinate current metrics initialization state.+ """+ global _registry, _exporter, _event_router+ with _lock:+ _registry = None+ _exporter = None+ _event_router = None+ reset_metric_account_dimension()+ if app is not None:+ app.state.metrics_exporter = None+++def get_metrics_registry() -> MetricRegistry:+ """+ Return the process-global `MetricRegistry`.++ Raises:+ RuntimeError: If metrics have not been initialized yet or were explicitly disabled.+ """+ reg = _registry+ if reg is None:+ raise RuntimeError("metrics registry not initialized")+ return reg+++def try_get_metrics_registry() -> MetricRegistry | None:+ """+ Return the global MetricRegistry if metrics are initialized, otherwise `None`.++ This helper is intended for best-effort call sites that want to avoid raising an exception+ when metrics are disabled.+ """+ return _registry+++def try_dispatch_event(event_name: str, payload: dict) -> None:+ """+ Dispatch an in-process metrics event.++ This is a best-effort API: if metrics are not initialized, it is a no-op.+ Callers must not assume the event is recorded.+ """+ router = _event_router+ if router is None:+ return+ router.dispatch(event_name, payload)+++def _build_event_router(registry: MetricRegistry) -> EventCollectorRouter:+ """+ Build the default in-process event router for event-driven metrics.++ The router binds each supported event name to the corresponding collector instance and+ ensures every handler receives the shared registry object that backs the active exporter.+ """+ router = EventCollectorRouter()+ http_collector = HTTPCollector()+ cache_collector = CacheCollector()+ embedding_collector = EmbeddingCollector()+ vlm_collector = VLMCollector()+ session_collector = SessionCollector()+ resource_collector = ResourceIngestionCollector()+ retrieval_collector = RetrievalCollector()+ encryption_collector = EncryptionCollector()+ telemetry_bridge_collector = TelemetryBridgeCollector()++ def _receiver(collector, event_name: str):+ """+ Create a payload handler bound to a specific collector and event name.++ The returned handler closes over the global registry and forwards the payload to the+ collector's `receive(...)` method, preserving the architectural rule that collectors are+ the only layer allowed to write into `MetricRegistry`.+ """++ def _handle(payload):+ """Forward one event payload into the bound collector without altering the payload."""+ collector.receive(event_name, payload, registry)++ return _handle++ for event_name, collector in (+ ("http.request", http_collector),+ ("http.inflight", http_collector),+ ("cache.hit", cache_collector),+ ("cache.miss", cache_collector),+ ("embedding.success", embedding_collector),+ ("embedding.error", embedding_collector),+ ("vlm.call", vlm_collector),+ ("session.lifecycle", session_collector),+ ("session.contexts_used", session_collector),+ ("session.archive", session_collector),+ ("resource.stage", resource_collector),+ ("resource.wait", resource_collector),+ ("retrieval.completed", retrieval_collector),+ ("encryption.operation", encryption_collector),+ ("encryption.bytes", encryption_collector),+ ("encryption.payload_size", encryption_collector),+ ("encryption.auth_failed", encryption_collector),+ ("encryption.key_derivation", encryption_collector),+ ("encryption.key_load", encryption_collector),+ ("encryption.key_cache_hit", encryption_collector),+ ("encryption.key_cache_miss", encryption_collector),+ ("encryption.key_version_usage", encryption_collector),+ ("telemetry.summary", telemetry_bridge_collector),+ ):+ router.register(event_name, _receiver(collector, event_name))+ return routerdiff --git a/openviking/metrics/http_middleware.py b/openviking/metrics/http_middleware.pynew file mode 100644index 000000000..ee2f9b18a--- /dev/null+++ b/openviking/metrics/http_middleware.py@@ -0,0 +1,214 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+HTTP metrics middleware.++This middleware records request-level metrics in a best-effort manner:+- Inflight gauge per route template (to avoid high cardinality per raw path).+- Request count and duration histogram by method/route/status.++Implementation notes:+- Uses HttpRequestLifecycleDataSource, which emits events; collectors write into MetricRegistry.+- All metrics calls are wrapped in try/except to ensure observability never breaks request handling.+- Route label uses the Starlette route template (e.g., "/sessions/{session_id}") when available.+"""++from __future__ import annotations++import logging+import threading+import time+from typing import Callable++from starlette.requests import Request+from starlette.responses import Response++from openviking.metrics.account_context import (+ bind_metric_account_context,+ reset_metric_account_context,+)+from openviking.metrics.datasources import HttpRequestLifecycleDataSource++logger = logging.getLogger(__name__)++_HTTP_IGNORE_ROUTES = frozenset(+ {+ "/metrics",+ "/health",+ "/ready",+ }+)+++def create_http_metrics_middleware() -> Callable[[Request, Callable], Response]:+ """+ Create a Starlette-compatible middleware that records HTTP request metrics.++ The middleware emits events through `HttpRequestLifecycleDataSource`:+ - `http.inflight` updates on request start/end+ - `http.request` on request completion with status and duration++ Routes listed in `_HTTP_IGNORE_ROUTES` are intentionally skipped so internal endpoints such+ as metrics or health probes do not pollute business-facing HTTP timeseries.+ """++ async def middleware(request: Request, call_next: Callable) -> Response:+ """+ Wrap one request execution and emit best-effort inflight and request metrics events.++ The middleware never lets metrics failures affect request handling and will skip the+ entire HTTP metrics flow for routes that are explicitly ignored.+ """+ # Ignore by raw path first so internal endpoints are skipped even when no route template+ # is bound (e.g. middleware executed outside Starlette routing).+ raw_path = str(request.url.path)+ if raw_path in _HTTP_IGNORE_ROUTES:+ return await call_next(request)++ initial_route = _get_route_template(request)+ if initial_route in _HTTP_IGNORE_ROUTES:+ return await call_next(request)+ provisional_account_id = _extract_request_account_id(request)+ account_token = bind_metric_account_context(account_id=provisional_account_id)+ try:+ HttpRequestLifecycleDataSource.set_inflight(+ route=initial_route,+ value=_inflight_delta(initial_route, provisional_account_id, +1),+ account_id=provisional_account_id,+ )+ except Exception:+ _log_metrics_failure("http.inflight increment failed", route=initial_route)++ start = time.perf_counter()+ status_code = 500+ try:+ response = await call_next(request)+ status_code = int(getattr(response, "status_code", 500))+ return response+ finally:+ elapsed = time.perf_counter() - start+ final_route = _get_route_template(request)+ final_account_id = _extract_request_account_id(request)+ if (final_route, final_account_id) != (initial_route, provisional_account_id):+ try:+ HttpRequestLifecycleDataSource.set_inflight(+ route=initial_route,+ value=_inflight_delta(initial_route, provisional_account_id, -1),+ account_id=provisional_account_id,+ )+ HttpRequestLifecycleDataSource.set_inflight(+ route=final_route,+ value=_inflight_delta(final_route, final_account_id, +1),+ account_id=final_account_id,+ )+ except Exception:+ _log_metrics_failure(+ "http.inflight rebalance failed",+ route=final_route,+ account_id=final_account_id,+ )+ try:+ HttpRequestLifecycleDataSource.record_request(+ method=request.method,+ route=final_route,+ status=str(status_code),+ duration_seconds=elapsed,+ account_id=final_account_id,+ )+ except Exception:+ _log_metrics_failure(+ "http.request recording failed",+ route=final_route,+ account_id=final_account_id,+ )+ try:+ HttpRequestLifecycleDataSource.set_inflight(+ route=final_route,+ value=_inflight_delta(final_route, final_account_id, -1),+ account_id=final_account_id,+ )+ except Exception:+ _log_metrics_failure(+ "http.inflight decrement failed",+ route=final_route,+ account_id=final_account_id,+ )+ reset_metric_account_context(account_token)++ return middleware+++_INFLIGHT: dict[tuple[str, str | None], int] = {}+_INFLIGHT_LOCK = threading.Lock()+++def _inflight_delta(route: str, account_id: str | None, delta: int) -> int:+ """+ Update and return the current in-process inflight count for a route template.++ This local counter is only used to provide a reasonable gauge value for event emission.+ It is not a correctness mechanism and is clamped to never go below zero.+ """+ key = (route, account_id)+ with _INFLIGHT_LOCK:+ v = _INFLIGHT.get(key, 0) + delta+ if v < 0:+ v = 0+ if v == 0:+ _INFLIGHT.pop(key, None)+ else:+ _INFLIGHT[key] = v+ return v+++def _get_route_template(request: Request) -> str:+ """+ Resolve a low-cardinality route identifier for labeling request metrics.++ Prefer the Starlette route template (e.g. `/sessions/{session_id}`) when available, and+ fall back to a fixed sentinel when the request is not bound to a recognized route template.+ """+ route = request.scope.get("route")+ path = getattr(route, "path", None)+ if path:+ return str(path)+ # Never label by raw path here: it may contain IDs and explode cardinality.+ return "/__unmatched__"+++def _extract_request_account_id(request: Request) -> str | None:+ """+ Return the best request-scoped account id currently available for metrics labeling.++ Only trust the authenticated request state (`request.state.metric_account_id`).++ The `/metrics` endpoint is intentionally best-effort and must not let unauthenticated or+ rejected traffic control tenant labels via raw headers. When the authenticated account id+ is not available yet, return `None` so collector-side policy can map it to `__unknown__`.+ """+ state_account_id = getattr(request.state, "metric_account_id", None)+ if state_account_id:+ return str(state_account_id)+ return None+++def _log_metrics_failure(message: str, *, route: str, account_id: str | None = None) -> None:+ """+ Emit a debug log for best-effort metrics failures without affecting request handling.++ This middleware intentionally treats observability as a side channel. The helper keeps+ failure reporting centralized so all swallowed exceptions still leave a trace when debug+ logging is enabled.+ """+ if not logger.isEnabledFor(logging.DEBUG):+ return+ logger.debug(+ "http metrics write failed: %s",+ message,+ extra={+ "route": route,+ "account_id": account_id,+ },+ exc_info=True,+ )diff --git a/openviking/models/rerank/base.py b/openviking/models/rerank/base.pyindex 94e89434b..358a78eef 100644--- a/openviking/models/rerank/base.py+++ b/openviking/models/rerank/base.py@@ -21,6 +21,17 @@ def _get_token_tracker():return _token_tracker_instance+def get_shared_rerank_token_usage() -> Dict[str, Any]:+ """+ Return the process-wide rerank token usage snapshot.++ This reads the shared singleton `TokenUsageTracker` used by all rerank clients.+ It exists so observability paths (e.g. Prometheus scrape) can access rerank usage+ without constructing provider clients that may allocate network resources.+ """+ return _get_token_tracker().to_dict()++class RerankBase:"""Base class for all rerank clientsdiff --git a/openviking/models/vlm/base.py b/openviking/models/vlm/base.pyindex 270444283..20b7aaa61 100644--- a/openviking/models/vlm/base.py+++ b/openviking/models/vlm/base.py@@ -205,11 +205,17 @@ class VLMBase(ABC):# Record the VLM call in Prometheus metrics (if enabled).try:- from openviking.storage.observers.prometheus_observer import get_prometheus_observer-- prom = get_prometheus_observer()- if prom is not None:- prom.record_vlm_call(duration_seconds)+ from openviking.metrics.account_context import get_metric_account_context+ from openviking.metrics.datasources import VLMEventDataSource++ VLMEventDataSource.record_call(+ provider=str(provider),+ model_name=str(model_name),+ duration_seconds=float(duration_seconds),+ prompt_tokens=int(prompt_tokens),+ completion_tokens=int(completion_tokens),+ account_id=get_metric_account_context().http_account_id,+ )except Exception:passdiff --git a/openviking/retrieve/retrieval_stats.py b/openviking/retrieve/retrieval_stats.pyindex 191b9d371..5785285bd 100644--- a/openviking/retrieve/retrieval_stats.py+++ b/openviking/retrieve/retrieval_stats.py@@ -137,13 +137,16 @@ class RetrievalStatsCollector:if latency_ms > self._stats.max_latency_ms:self._stats.max_latency_ms = latency_ms- # Notify the Prometheus observer (if enabled) outside our lock.try:- from openviking.storage.observers.prometheus_observer import get_prometheus_observer-- prom = get_prometheus_observer()- if prom is not None:- prom.record_retrieval(latency_ms / 1000.0)+ from openviking.metrics.datasources import RetrievalStatsDataSource++ RetrievalStatsDataSource.record_retrieval(+ context_type=str(context_type or "unknown"),+ result_count=int(result_count),+ latency_seconds=latency_ms / 1000.0,+ rerank_used=bool(rerank_used),+ rerank_fallback=bool(rerank_fallback),+ )except Exception:passdiff --git a/openviking/server/app.py b/openviking/server/app.pyindex 8ef3d9a35..00a4acdd8 100644--- a/openviking/server/app.py+++ b/openviking/server/app.py@@ -34,8 +34,6 @@ from openviking.server.routers import ()from openviking.service.core import OpenVikingServicefrom openviking.service.task_tracker import get_task_tracker-from openviking.storage.observers import PrometheusObserver-from openviking.storage.observers.prometheus_observer import set_prometheus_observerfrom openviking_cli.exceptions import OpenVikingErrorfrom openviking_cli.utils import get_logger@@ -70,6 +68,7 @@ def create_app(await service.initialize()logger.info("OpenVikingService initialized")+ assert service is not Noneset_service(service)# Initialize APIKeyManager after service (needs VikingFS)@@ -110,11 +109,13 @@ def create_app(config.host,)- app.state.prometheus_observer = None- if config.telemetry.prometheus.enabled:- observer = PrometheusObserver()- app.state.prometheus_observer = observer- set_prometheus_observer(observer)+ from openviking.metrics.global_api import (+ init_metrics_from_server_config,+ is_metrics_enabled_from_server_config,+ )++ init_metrics_from_server_config(config, app=app, service=service)+ if is_metrics_enabled_from_server_config(config):logger.info("Prometheus metrics enabled at /metrics")# Start TaskTracker cleanup loop@@ -129,7 +130,9 @@ def create_app(yield# Cleanup- set_prometheus_observer(None)+ from openviking.metrics.global_api import shutdown_metrics++ shutdown_metrics(app=app)task_tracker.stop_cleanup_loop()if owns_service and service:try:@@ -167,6 +170,14 @@ def create_app(response.headers["X-Process-Time"] = str(process_time)return response+ from openviking.metrics.http_middleware import create_http_metrics_middleware++ http_metrics_middleware = create_http_metrics_middleware()++ @app.middleware("http")+ async def add_http_metrics(request: Request, call_next: Callable):+ return await http_metrics_middleware(request, call_next)+# Add exception handler for OpenVikingError@app.exception_handler(OpenVikingError)async def openviking_error_handler(request: Request, exc: OpenVikingError):diff --git a/openviking/server/auth.py b/openviking/server/auth.pyindex 32159a9f7..005b9936f 100644--- a/openviking/server/auth.py+++ b/openviking/server/auth.py@@ -7,6 +7,7 @@ from typing import Optionalfrom fastapi import Depends, Header, Request+from openviking.metrics.account_context import set_metric_account_contextfrom openviking.server.identity import RequestContext, ResolvedIdentity, Rolefrom openviking_cli.exceptions import (InvalidArgumentError,@@ -145,7 +146,7 @@ async def get_request_context(if auth_mode == "trusted" and not identity.user_id:raise InvalidArgumentError("Trusted mode requests must include X-OpenViking-User.")- return RequestContext(+ ctx = RequestContext(user=UserIdentifier(identity.account_id or "default",identity.user_id or "default",@@ -153,6 +154,9 @@ async def get_request_context(),role=identity.role,)+ request.state.metric_account_id = ctx.account_id+ set_metric_account_context(account_id=ctx.account_id)+ return ctxdef require_role(*allowed_roles: Role):diff --git a/openviking/server/config.py b/openviking/server/config.pyindex 988d7e673..3f95d0502 100644--- a/openviking/server/config.py+++ b/openviking/server/config.py@@ -39,9 +39,29 @@ class TelemetryConfig(BaseModel):model_config = {"extra": "forbid"}-class ServerConfig(BaseModel):- """Server configuration (from the ``server`` section of ov.conf)."""+class MetricsAccountDimensionConfig(BaseModel):+ """Account-dimension configuration for metrics label injection."""++ # Enabled by default, but still allowlist-gated to avoid accidental high-cardinality exposure.+ enabled: bool = True+ max_active_accounts: int = 100+ metric_allowlist: List[str] = Field(default_factory=list)++ model_config = {"extra": "forbid"}+++class MetricsConfig(BaseModel):+ """Metrics subsystem configuration."""+ enabled: bool = False+ account_dimension: MetricsAccountDimensionConfig = Field(+ default_factory=MetricsAccountDimensionConfig+ )++ model_config = {"extra": "forbid"}+++class ServerConfig(BaseModel):host: str = "127.0.0.1"port: int = 1933workers: int = 1@@ -52,6 +72,7 @@ class ServerConfig(BaseModel):bot_api_url: str = "http://localhost:18790" # Vikingbot OpenAPIChannel URL (default port)encryption_enabled: bool = False # Whether API key hashing is enabledtelemetry: TelemetryConfig = Field(default_factory=TelemetryConfig)+ metrics: MetricsConfig = Field(default_factory=MetricsConfig)model_config = {"extra": "forbid"}diff --git a/openviking/server/routers/metrics.py b/openviking/server/routers/metrics.pyindex cfe2ab813..65cb7d863 100644--- a/openviking/server/routers/metrics.py+++ b/openviking/server/routers/metrics.py@@ -11,11 +11,11 @@ router = APIRouter(tags=["metrics"])@router.get("/metrics")async def metrics(request: Request):"""Return Prometheus metrics in text exposition format."""- observer = getattr(request.app.state, "prometheus_observer", None)- if observer is None:+ exporter = getattr(request.app.state, "metrics_exporter", None)+ if exporter is None:return PlainTextResponse(status_code=404, content="Prometheus metrics are disabled.\n")return PlainTextResponse(- content=observer.render_metrics(),+ content=await exporter.export(),media_type="text/plain; version=0.0.4; charset=utf-8",)diff --git a/openviking/service/resource_service.py b/openviking/service/resource_service.pyindex 54b2d52f0..f673effb9 100644--- a/openviking/service/resource_service.py+++ b/openviking/service/resource_service.py@@ -230,6 +230,18 @@ class ResourceService:)raise DeadlineExceededError("queue processing", timeout) from excqueue_wait_duration_ms = round((time.perf_counter() - wait_start) * 1000, 3)+ try:+ from openviking.metrics.datasources.resource import (+ ResourceIngestionEventDataSource,+ )++ ResourceIngestionEventDataSource.record_wait(+ operation="queue_processing",+ duration_seconds=float(queue_wait_duration_ms) / 1000.0,+ account_id=getattr(ctx, "account_id", None),+ )+ except Exception:+ passresult["queue_status"] = statusrecord_resource_wait_metrics(telemetry_id=telemetry_id,diff --git a/openviking/service/session_service.py b/openviking/service/session_service.pyindex ab79a1e44..e81be1b35 100644--- a/openviking/service/session_service.py+++ b/openviking/service/session_service.py@@ -79,13 +79,34 @@ class SessionService:Raises:AlreadyExistsError: If a session with the given ID already exists"""- if session_id:- existing = self.session(ctx, session_id)- if await existing.exists():- raise AlreadyExistsError(f"Session '{session_id}' already exists")- session = self.session(ctx, session_id)- await session.ensure_exists()- return session+ try:+ from openviking.metrics.datasources.session import SessionLifecycleDataSource++ SessionLifecycleDataSource.record_lifecycle(action="create", status="attempt")+ except Exception:+ pass+ try:+ if session_id:+ existing = self.session(ctx, session_id)+ if await existing.exists():+ raise AlreadyExistsError(f"Session '{session_id}' already exists")+ session = self.session(ctx, session_id)+ await session.ensure_exists()+ try:+ from openviking.metrics.datasources.session import SessionLifecycleDataSource++ SessionLifecycleDataSource.record_lifecycle(action="create", status="ok")+ except Exception:+ pass+ return session+ except Exception:+ try:+ from openviking.metrics.datasources.session import SessionLifecycleDataSource++ SessionLifecycleDataSource.record_lifecycle(action="create", status="error")+ except Exception:+ pass+ raiseasync def get(self, session_id: str, ctx: RequestContext, *, auto_create: bool = False@@ -98,13 +119,28 @@ class SessionService:auto_create: If True, create the session when it does not exist.Default is False (raise NotFoundError)."""- session = self.session(ctx, session_id)- if not await session.exists():- if not auto_create:- raise NotFoundError(session_id, "session")- await session.ensure_exists()- await session.load()- return session+ try:+ session = self.session(ctx, session_id)+ if not await session.exists():+ if not auto_create:+ raise NotFoundError(session_id, "session")+ await session.ensure_exists()+ await session.load()+ try:+ from openviking.metrics.datasources.session import SessionLifecycleDataSource++ SessionLifecycleDataSource.record_lifecycle(action="get", status="ok")+ except Exception:+ pass+ return session+ except Exception:+ try:+ from openviking.metrics.datasources.session import SessionLifecycleDataSource++ SessionLifecycleDataSource.record_lifecycle(action="get", status="error")+ except Exception:+ pass+ raiseasync def sessions(self, ctx: RequestContext) -> List[Dict[str, Any]]:"""Get all sessions for the current user.@@ -148,9 +184,21 @@ class SessionService:try:await self._viking_fs.rm(session_uri, recursive=True, ctx=ctx)logger.info(f"Deleted session: {session_id}")+ try:+ from openviking.metrics.datasources.session import SessionLifecycleDataSource++ SessionLifecycleDataSource.record_lifecycle(action="delete", status="ok")+ except Exception:+ passreturn Trueexcept Exception as e:logger.error(f"Failed to delete session {session_id}: {e}")+ try:+ from openviking.metrics.datasources.session import SessionLifecycleDataSource++ SessionLifecycleDataSource.record_lifecycle(action="delete", status="error")+ except Exception:+ passraise NotFoundError(session_id, "session")async def commit(self, session_id: str, ctx: RequestContext) -> Dict[str, Any]:@@ -181,7 +229,19 @@ class SessionService:"""self._ensure_initialized()session = await self.get(session_id, ctx)- return await session.commit_async()+ result = await session.commit_async()+ try:+ from openviking.metrics.datasources.session import SessionLifecycleDataSource++ SessionLifecycleDataSource.record_lifecycle(+ action="commit", status="ok" if result.get("status") else "error"+ )+ SessionLifecycleDataSource.record_archive(+ status="ok" if result.get("archived") else "skip"+ )+ except Exception:+ pass+ return resultasync def get_commit_task(self, task_id: str, ctx: RequestContext) -> Optional[Dict[str, Any]]:"""Query background commit task status by task_id for the calling owner."""@@ -207,9 +267,16 @@ class SessionService:session = await self.get(session_id, ctx)- return await self._session_compressor.extract_long_term_memories(+ memories = await self._session_compressor.extract_long_term_memories(messages=session.messages,user=ctx.user,session_id=session_id,ctx=ctx,)+ try:+ from openviking.metrics.datasources.session import SessionLifecycleDataSource++ SessionLifecycleDataSource.record_lifecycle(action="extract", status="ok")+ except Exception:+ pass+ return memoriesdiff --git a/openviking/service/task_tracker.py b/openviking/service/task_tracker.pyindex 582f0d9f0..824653968 100644--- a/openviking/service/task_tracker.py+++ b/openviking/service/task_tracker.py@@ -356,3 +356,14 @@ class TaskTracker:"""Return total task count."""with self._lock:return len(self._tasks)++ def snapshot_counts_by_type(self) -> Dict[str, Dict[str, int]]:+ """Return a snapshot of task counts grouped by task_type and status."""+ from collections import defaultdict++ grouped: Dict[str, Dict[str, int]] = defaultdict(lambda: defaultdict(int)) # type: ignore[assignment]+ with self._lock:+ tasks = list(self._tasks.values())+ for t in tasks:+ grouped[t.task_type][t.status.value] += 1+ return {k: dict(v) for k, v in grouped.items()}diff --git a/openviking/session/memory/extract_loop.py b/openviking/session/memory/extract_loop.pyindex 296c1b9a5..e2dc33734 100644--- a/openviking/session/memory/extract_loop.py+++ b/openviking/session/memory/extract_loop.py@@ -175,6 +175,7 @@ The final output of the model must strictly follow the JSON Schema format shownfor msg in tool_call_messages:if msg.get("role") == "user" and "tool_call_name" in msg.get("content", ""):import json+try:content = json.loads(msg.get("content", "{}"))if content.get("tool_call_name") == "read":@@ -203,9 +204,7 @@ The final output of the model must strictly follow the JSON Schema format shown# Call LLM with tools - model decides: tool calls OR final operationspretty_print_messages(messages)- tool_calls, operations = await self._call_llm(- messages- )+ tool_calls, operations = await self._call_llm(messages)if tool_calls:has_unknown_tool = await self._execute_tool_calls(messages, tool_calls, tools_used)@@ -268,6 +267,7 @@ The final output of the model must strictly follow the JSON Schema format shownTrue if any tool call returned "Unknown tool" error, indicatingthe model should not receive tools in the next iteration."""+# Execute all tool calls in parallelasync def execute_single_tool_call(idx: int, tool_call):"""Execute a single tool call."""@@ -346,8 +346,7 @@ The final output of the model must strictly follow the JSON Schema format shownraise ValueError(error_msg)async def _call_llm(- self,- messages: List[Dict[str, Any]]+ self, messages: List[Dict[str, Any]]) -> Tuple[Optional[List], Optional[Any]]:"""Call LLM with tools. Returns either tool calls OR final operations.@@ -384,6 +383,15 @@ The final output of the model must strictly follow the JSON Schema format shownif isinstance(usage.get("prompt_tokens_details"), dict)else 0)+ try:+ from openviking.metrics.datasources.cache import CacheEventDataSource++ if int(cached_tokens or 0) > 0:+ CacheEventDataSource.record_hit("L2")+ else:+ CacheEventDataSource.record_miss("L2")+ except Exception:+ passif prompt_tokens > 0:cache_hit_rate = (cached_tokens / prompt_tokens) * 100tracer.info(diff --git a/openviking/session/session.py b/openviking/session/session.pyindex 6692d1732..3bcf276b4 100644--- a/openviking/session/session.py+++ b/openviking/session/session.py@@ -302,6 +302,14 @@ class Session:self._usage_records.append(usage)self._stats.contexts_used += 1logger.debug(f"Tracked context usage: {uri}")+ try:+ from openviking.metrics.datasources.session import SessionLifecycleDataSource++ SessionLifecycleDataSource.record_contexts_used(+ action="context", delta=len(contexts)+ )+ except Exception:+ passif skill:usage = Usage(@@ -314,6 +322,12 @@ class Session:self._usage_records.append(usage)self._stats.skills_used += 1logger.debug(f"Tracked skill usage: {skill.get('uri')}")+ try:+ from openviking.metrics.datasources.session import SessionLifecycleDataSource++ SessionLifecycleDataSource.record_contexts_used(action="skill", delta=1)+ except Exception:+ passdef add_message(self,@@ -466,8 +480,6 @@ class Session:# Snapshot mutable state for Phase 2usage_snapshot = self._usage_records.copy()- first_message_id = messages_to_archive[0].id if messages_to_archive else ""- last_message_id = messages_to_archive[-1].id if messages_to_archive else ""# Create TaskRecord for tracking Phase 2tracker = get_task_tracker()diff --git a/openviking/storage/collection_schemas.py b/openviking/storage/collection_schemas.pyindex d63f1dcc5..bec0b0f11 100644--- a/openviking/storage/collection_schemas.py+++ b/openviking/storage/collection_schemas.py@@ -330,18 +330,26 @@ class TextEmbeddingHandler(DequeueHandlerBase):)_embed_elapsed = _time.monotonic() - _embed_t0try:- from openviking.storage.observers.prometheus_observer import (- get_prometheus_observer,- )+ from openviking.metrics.datasources import EmbeddingEventDataSource- _prom = get_prometheus_observer()- if _prom is not None:- _prom.record_embedding(_embed_elapsed)+ EmbeddingEventDataSource.record_success(+ latency_seconds=float(_embed_elapsed),+ account_id=embedding_msg.context_data.get("account_id"),+ )except Exception:passexcept Exception as embed_err:error_msg = f"Failed to generate embedding: {embed_err}"error_class = classify_api_error(embed_err)+ try:+ from openviking.metrics.datasources import EmbeddingEventDataSource++ EmbeddingEventDataSource.record_error(+ error_code=str(error_class or "unknown"),+ account_id=embedding_msg.context_data.get("account_id"),+ )+ except Exception:+ passif error_class == "permanent":logger.critical(error_msg)@@ -399,6 +407,12 @@ class TextEmbeddingHandler(DequeueHandlerBase):else:error_msg = "Embedder not initialized, skipping vector generation"logger.warning(error_msg)+ try:+ from openviking.metrics.datasources import EmbeddingEventDataSource++ EmbeddingEventDataSource.record_error(error_code="not_initialized")+ except Exception:+ passself._merge_request_stats(embedding_msg.telemetry_id, error_count=1)request_failed_message = error_msgreport_error_args = (error_msg, data)diff --git a/openviking/storage/observers/__init__.py b/openviking/storage/observers/__init__.pyindex c8b643c6b..94448c0d7 100644--- a/openviking/storage/observers/__init__.py+++ b/openviking/storage/observers/__init__.py@@ -3,11 +3,6 @@from .base_observer import BaseObserverfrom .lock_observer import LockObserverfrom .models_observer import ModelsObserver-from .prometheus_observer import (- PrometheusObserver,- get_prometheus_observer,- set_prometheus_observer,-)from .queue_observer import QueueObserverfrom .retrieval_observer import RetrievalObserverfrom .vikingdb_observer import VikingDBObserver@@ -16,9 +11,6 @@ __all__ = ["BaseObserver","LockObserver","ModelsObserver",- "PrometheusObserver",- "get_prometheus_observer",- "set_prometheus_observer","QueueObserver","RetrievalObserver","VikingDBObserver",diff --git a/openviking/storage/observers/prometheus_observer.py b/openviking/storage/observers/prometheus_observer.pydeleted file mode 100644index 35f33e40c..000000000--- a/openviking/storage/observers/prometheus_observer.py+++ /dev/null@@ -1,195 +0,0 @@-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.-# SPDX-License-Identifier: AGPL-3.0-"""PrometheusObserver: Prometheus metrics exporter for OpenViking."""--import threading-from threading import Lock-from typing import Dict, Iterable, List, Optional, Tuple--from openviking.storage.observers.base_observer import BaseObserver---class _Histogram:- """Thread-safe in-memory histogram for Prometheus exposition."""-- def __init__(self, buckets: Iterable[float]):- self._buckets = sorted(float(b) for b in buckets)- self._bucket_counts: List[int] = [0] * len(self._buckets)- self._sum = 0.0- self._count = 0-- def observe(self, value: float) -> None:- self._sum += value- self._count += 1- for idx, upper in enumerate(self._buckets):- if value <= upper:- self._bucket_counts[idx] += 1- # +Inf bucket is implied by _count.-- def snapshot(self) -> Tuple[List[float], List[int], float, int]:- return list(self._buckets), list(self._bucket_counts), self._sum, self._count---class PrometheusObserver(BaseObserver):- """Observer that records and renders Prometheus metrics text format."""-- DEFAULT_LATENCY_BUCKETS = (0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0)- CACHE_LEVELS = ("L0", "L1", "L2")-- def __init__(self):- self._lock = Lock()-- self._retrieval_requests_total = 0- self._embedding_requests_total = 0- self._vlm_calls_total = 0-- self._cache_hits_total: Dict[str, int] = dict.fromkeys(self.CACHE_LEVELS, 0)- self._cache_misses_total: Dict[str, int] = dict.fromkeys(self.CACHE_LEVELS, 0)-- self._retrieval_latency_seconds = _Histogram(self.DEFAULT_LATENCY_BUCKETS)- self._embedding_latency_seconds = _Histogram(self.DEFAULT_LATENCY_BUCKETS)- self._vlm_call_duration_seconds = _Histogram(self.DEFAULT_LATENCY_BUCKETS)-- def get_status_table(self) -> str:- """Return Prometheus exposition for observer status views."""- return self.render_metrics()-- def is_healthy(self) -> bool:- return True-- def has_errors(self) -> bool:- return False-- def record_retrieval(self, latency_seconds: float) -> None:- with self._lock:- self._retrieval_requests_total += 1- self._retrieval_latency_seconds.observe(float(latency_seconds))-- def record_embedding(self, latency_seconds: float) -> None:- with self._lock:- self._embedding_requests_total += 1- self._embedding_latency_seconds.observe(float(latency_seconds))-- def record_vlm_call(self, duration_seconds: float) -> None:- with self._lock:- self._vlm_calls_total += 1- self._vlm_call_duration_seconds.observe(float(duration_seconds))-- def record_cache_hit(self, level: str) -> None:- with self._lock:- if level not in self._cache_hits_total:- self._cache_hits_total[level] = 0- self._cache_hits_total[level] += 1-- def record_cache_miss(self, level: str) -> None:- with self._lock:- if level not in self._cache_misses_total:- self._cache_misses_total[level] = 0- self._cache_misses_total[level] += 1-- def render_metrics(self) -> str:- with self._lock:- retrieval_requests_total = self._retrieval_requests_total- embedding_requests_total = self._embedding_requests_total- vlm_calls_total = self._vlm_calls_total- cache_hits_total = dict(self._cache_hits_total)- cache_misses_total = dict(self._cache_misses_total)- retrieval_hist = self._retrieval_latency_seconds.snapshot()- embedding_hist = self._embedding_latency_seconds.snapshot()- vlm_hist = self._vlm_call_duration_seconds.snapshot()-- lines: List[str] = []- lines.extend(- [- "# HELP openviking_retrieval_requests_total Total retrieval requests.",- "# TYPE openviking_retrieval_requests_total counter",- f"openviking_retrieval_requests_total {retrieval_requests_total}",- "# HELP openviking_embedding_requests_total Total embedding requests.",- "# TYPE openviking_embedding_requests_total counter",- f"openviking_embedding_requests_total {embedding_requests_total}",- "# HELP openviking_vlm_calls_total Total VLM calls.",- "# TYPE openviking_vlm_calls_total counter",- f"openviking_vlm_calls_total {vlm_calls_total}",- ]- )-- self._append_histogram(- lines=lines,- name="openviking_retrieval_latency_seconds",- help_text="Retrieval request latency in seconds.",- histogram=retrieval_hist,- )- self._append_histogram(- lines=lines,- name="openviking_embedding_latency_seconds",- help_text="Embedding request latency in seconds.",- histogram=embedding_hist,- )- self._append_histogram(- lines=lines,- name="openviking_vlm_call_duration_seconds",- help_text="VLM call duration in seconds.",- histogram=vlm_hist,- )-- self._append_labeled_counter(- lines=lines,- name="openviking_cache_hits_total",- help_text="Total cache hits by cache level.",- counters=cache_hits_total,- )- self._append_labeled_counter(- lines=lines,- name="openviking_cache_misses_total",- help_text="Total cache misses by cache level.",- counters=cache_misses_total,- )-- return "\n".join(lines) + "\n"-- @staticmethod- def _append_labeled_counter(- lines: List[str],- name: str,- help_text: str,- counters: Dict[str, int],- ) -> None:- lines.append(f"# HELP {name} {help_text}")- lines.append(f"# TYPE {name} counter")- for level in sorted(counters):- lines.append(f'{name}{{level="{level}"}} {counters[level]}')-- @staticmethod- def _append_histogram(- lines: List[str],- name: str,- help_text: str,- histogram: Tuple[List[float], List[int], float, int],- ) -> None:- buckets, bucket_counts, total_sum, total_count = histogram- lines.append(f"# HELP {name} {help_text}")- lines.append(f"# TYPE {name} histogram")-- for upper, count in zip(buckets, bucket_counts):- lines.append(f'{name}_bucket{{le="{upper:g}"}} {count}')- lines.append(f'{name}_bucket{{le="+Inf"}} {total_count}')- lines.append(f"{name}_sum {total_sum}")- lines.append(f"{name}_count {total_count}")---# Module-level singleton so that data-collection hooks can record metrics-# without requiring access to the FastAPI app state.-_observer: Optional[PrometheusObserver] = None-_observer_lock = threading.Lock()---def set_prometheus_observer(observer: Optional[PrometheusObserver]) -> None:- """Set the global PrometheusObserver instance (called during app startup)."""- global _observer- with _observer_lock:- _observer = observer---def get_prometheus_observer() -> Optional[PrometheusObserver]:- """Return the global PrometheusObserver, or None when metrics are disabled."""- return _observerdiff --git a/openviking/storage/queuefs/semantic_dag.py b/openviking/storage/queuefs/semantic_dag.pyindex 57c4d5e3b..9b4a784ff 100644--- a/openviking/storage/queuefs/semantic_dag.py+++ b/openviking/storage/queuefs/semantic_dag.py@@ -352,6 +352,12 @@ class SemanticDagExecutor:return Noneif parent_uri not in self._overview_cache:+ try:+ from openviking.metrics.datasources.cache import CacheEventDataSource++ CacheEventDataSource.record_miss("L1")+ except Exception:+ passasync with self._overview_cache_lock:if parent_uri not in self._overview_cache:overview_path = f"{parent_uri}/.overview.md"@@ -364,6 +370,13 @@ class SemanticDagExecutor:)else:self._overview_cache[parent_uri] = {}+ else:+ try:+ from openviking.metrics.datasources.cache import CacheEventDataSource++ CacheEventDataSource.record_hit("L1")+ except Exception:+ passexisting_summaries = self._overview_cache.get(parent_uri, {})file_name = file_path.split("/")[-1]diff --git a/openviking/storage/queuefs/semantic_processor.py b/openviking/storage/queuefs/semantic_processor.pyindex dee5f22a6..32031b536 100644--- a/openviking/storage/queuefs/semantic_processor.py+++ b/openviking/storage/queuefs/semantic_processor.py@@ -8,6 +8,10 @@ from contextlib import nullcontextfrom dataclasses import dataclass, fieldfrom typing import Any, Dict, List, Optional, Set, Tuple+from openviking.metrics.account_context import (+ bind_metric_account_context,+ reset_metric_account_context,+)from openviking.parse.parsers.constants import (CODE_EXTENSIONS,DOCUMENTATION_EXTENSIONS,@@ -269,66 +273,70 @@ class SemanticProcessor(DequeueHandlerBase):collector = resolve_telemetry(msg.telemetry_id)telemetry_ctx = bind_telemetry(collector) if collector is not None else nullcontext()with telemetry_ctx:- self._current_msg = msg- self._current_ctx = self._ctx_from_semantic_msg(msg)- logger.info(- f"Processing semantic generation for: {msg.uri} (recursive={msg.recursive})"- )+ metric_account_token = bind_metric_account_context(account_id=msg.account_id)+ try:+ self._current_msg = msg+ self._current_ctx = self._ctx_from_semantic_msg(msg)+ logger.info(+ f"Processing semantic generation for: {msg.uri} (recursive={msg.recursive})"+ )- logger.info(f"Processing semantic generation for: {msg})")+ logger.info(f"Processing semantic generation for: {msg})")- if msg.context_type == "memory":- await self._process_memory_directory(msg)- else:- is_incremental = False- viking_fs = get_viking_fs()- if msg.target_uri:- target_exists = await viking_fs.exists(- msg.target_uri, ctx=self._current_ctx- )- # Check if target URI exists and is not the same as the source URI(避免重复处理)- if target_exists and msg.uri != msg.target_uri:- is_incremental = True- logger.info(- f"Target URI exists, using incremental update: {msg.target_uri}"+ if msg.context_type == "memory":+ await self._process_memory_directory(msg)+ else:+ is_incremental = False+ viking_fs = get_viking_fs()+ if msg.target_uri:+ target_exists = await viking_fs.exists(+ msg.target_uri, ctx=self._current_ctx+ )+ # Check if target URI exists and is not the same as the source URI(避免重复处理)+ if target_exists and msg.uri != msg.target_uri:+ is_incremental = True+ logger.info(+ f"Target URI exists, using incremental update: {msg.target_uri}"+ )++ # Re-acquire lifecycle lock if handle was lost (e.g. server restart)+ if msg.lifecycle_lock_handle_id:+ lock_uri = msg.target_uri or msg.uri+ msg.lifecycle_lock_handle_id = await self._ensure_lifecycle_lock(+ msg.lifecycle_lock_handle_id,+ viking_fs._uri_to_path(lock_uri, ctx=self._current_ctx),)- # Re-acquire lifecycle lock if handle was lost (e.g. server restart)- if msg.lifecycle_lock_handle_id:- lock_uri = msg.target_uri or msg.uri- msg.lifecycle_lock_handle_id = await self._ensure_lifecycle_lock(- msg.lifecycle_lock_handle_id,- viking_fs._uri_to_path(lock_uri, ctx=self._current_ctx),+ executor = SemanticDagExecutor(+ processor=self,+ context_type=msg.context_type,+ max_concurrent_llm=self.max_concurrent_llm,+ ctx=self._current_ctx,+ incremental_update=is_incremental,+ target_uri=msg.target_uri,+ semantic_msg_id=msg.id,+ telemetry_id=msg.telemetry_id,+ recursive=msg.recursive,+ lifecycle_lock_handle_id=msg.lifecycle_lock_handle_id,+ is_code_repo=msg.is_code_repo,)-- executor = SemanticDagExecutor(- processor=self,- context_type=msg.context_type,- max_concurrent_llm=self.max_concurrent_llm,- ctx=self._current_ctx,- incremental_update=is_incremental,- target_uri=msg.target_uri,- semantic_msg_id=msg.id,- telemetry_id=msg.telemetry_id,- recursive=msg.recursive,- lifecycle_lock_handle_id=msg.lifecycle_lock_handle_id,- is_code_repo=msg.is_code_repo,- )- self._dag_executor = executor- if msg.lifecycle_lock_handle_id:- # The DAG owns lifecycle lock release after this point.- release_lock_in_finally = False- await executor.run(msg.uri)- self._cache_dag_stats(- msg.telemetry_id,- msg.uri,- executor.get_stats(),- )- self._merge_request_stats(msg.telemetry_id, processed=1)- logger.info(f"Completed semantic generation for: {msg.uri}")- self.report_success()- self._circuit_breaker.record_success()- return None+ self._dag_executor = executor+ if msg.lifecycle_lock_handle_id:+ # The DAG owns lifecycle lock release after this point.+ release_lock_in_finally = False+ await executor.run(msg.uri)+ self._cache_dag_stats(+ msg.telemetry_id,+ msg.uri,+ executor.get_stats(),+ )+ self._merge_request_stats(msg.telemetry_id, processed=1)+ logger.info(f"Completed semantic generation for: {msg.uri}")+ self.report_success()+ self._circuit_breaker.record_success()+ return None+ finally:+ reset_metric_account_context(metric_account_token)except Exception as e:error_class = classify_api_error(e)@@ -501,6 +509,23 @@ class SemanticProcessor(DequeueHandlerBase):else:pending_indices.append((idx, file_path))+ if file_paths and not pending_indices:+ try:+ from openviking.metrics.datasources.cache import CacheEventDataSource++ CacheEventDataSource.record_hit("L1")+ except Exception:+ pass+ elif file_paths and pending_indices:+ try:+ from openviking.metrics.datasources.cache import CacheEventDataSource++ if len(file_paths) > len(pending_indices):+ CacheEventDataSource.record_hit("L1")+ CacheEventDataSource.record_miss("L1")+ except Exception:+ pass+if pending_indices:logger.info(f"Generating summaries for {len(pending_indices)} changed files "diff --git a/openviking/telemetry/execution.py b/openviking/telemetry/execution.pyindex 18d85dcbd..df1679639 100644--- a/openviking/telemetry/execution.py+++ b/openviking/telemetry/execution.py@@ -100,6 +100,15 @@ async def run_with_telemetry(snapshot = collector.finish(status="ok")_log_telemetry_summary(snapshot)+ try:+ if snapshot is not None:+ from openviking.metrics.datasources.telemetry_bridge import (+ TelemetryBridgeEventDataSource,+ )++ TelemetryBridgeEventDataSource.record_summary(snapshot.summary)+ except Exception:+ passtelemetry_payload = build_telemetry_payload(snapshot,selection,diff --git a/openviking/utils/resource_processor.py b/openviking/utils/resource_processor.pyindex 01aaf75df..18b28df17 100644--- a/openviking/utils/resource_processor.py+++ b/openviking/utils/resource_processor.py@@ -128,7 +128,13 @@ class ResourceProcessor:with telemetry.measure("resource.process"):# ============ Phase 1: Parse source and writes to temp viking fs ============try:+ from openviking.metrics.datasources.resource import (+ ResourceIngestionEventDataSource,+ )+parse_start = time.perf_counter()+ stage_start = time.perf_counter()+ stage_status = "ok"media_processor = self._get_media_processor()viking_fs = get_viking_fs()# Use reason as instruction fallback so it influences L0/L1@@ -151,6 +157,7 @@ class ResourceProcessor:result["errors"].extend(parse_result.warnings or ["Parse failed: no content generated"],)+ stage_status = "error"return resultif parse_result.warnings and kwargs.get("strict", False):@@ -163,6 +170,7 @@ class ResourceProcessor:telemetry.set("resource.parse.warnings_count", len(parse_result.warnings or []))except OpenVikingError:+ stage_status = "error"raiseexcept Exception as e:result["status"] = "error"@@ -172,7 +180,18 @@ class ResourceProcessor:import tracebacktraceback.print_exc()+ stage_status = "error"return result+ finally:+ try:+ ResourceIngestionEventDataSource.record_stage(+ stage="parse",+ status=str(stage_status),+ duration_seconds=float(time.perf_counter() - stage_start),+ account_id=getattr(ctx, "account_id", None),+ )+ except Exception:+ pass# parse_result contains:# - root: ResourceNode tree (with L0/L1 in meta)@@ -181,6 +200,8 @@ class ResourceProcessor:# ============ Phase 3: TreeBuilder finalizes from temp (scan + move to AGFS) ============try:+ stage_start = time.perf_counter()+ stage_status = "ok"finalize_start = time.perf_counter()with get_viking_fs().bind_request_context(ctx):context_tree = await self.tree_builder.finalize_from_temp(@@ -203,6 +224,7 @@ class ResourceProcessor:result["status"] = "error"result["errors"].append(f"Finalize from temp error: {e}")telemetry.set_error("resource_processor.finalize", "PROCESSING_ERROR", str(e))+ stage_status = "error"# Cleanup temporary directory on error (via VikingFS)try:@@ -212,6 +234,16 @@ class ResourceProcessor:passreturn result+ finally:+ try:+ ResourceIngestionEventDataSource.record_stage(+ stage="finalize",+ status=str(stage_status),+ duration_seconds=float(time.perf_counter() - stage_start),+ account_id=getattr(ctx, "account_id", None),+ )+ except Exception:+ pass# ============ Phase 3.5: 首次添加立即落盘 + 生命周期锁 ============root_uri = result.get("root_uri")@@ -222,49 +254,62 @@ class ResourceProcessor:if root_uri and temp_uri:from openviking.storage.transaction import LockContext, get_lock_manager+ stage_start = time.perf_counter()+ stage_status = "ok"viking_fs = get_viking_fs()lock_manager = get_lock_manager()- target_exists = await viking_fs.exists(root_uri, ctx=ctx)-- if not target_exists:- # 第一次添加:锁保护下将 temp 移到 final- dst_path = viking_fs._uri_to_path(root_uri, ctx=ctx)- parent_path = dst_path.rsplit("/", 1)[0] if "/" in dst_path else dst_path-- parent_uri = "/".join(root_uri.rstrip("/").rsplit("/", 1)[:-1])- if parent_uri:- await viking_fs.mkdir(parent_uri, exist_ok=True, ctx=ctx)-- async with LockContext(lock_manager, [parent_path], lock_mode="point"):- if candidate_uri:- with viking_fs.bind_request_context(ctx):- root_uri = await self.tree_builder._resolve_unique_uri(- candidate_uri- )- result["root_uri"] = root_uri- dst_path = viking_fs._uri_to_path(root_uri, ctx=ctx)-- src_path = viking_fs._uri_to_path(temp_uri, ctx=ctx)- await asyncio.to_thread(viking_fs.agfs.mv, src_path, dst_path)-- # 在 POINT 锁内获取 SUBTREE 锁(消除竞态窗口)+ try:+ target_exists = await viking_fs.exists(root_uri, ctx=ctx)++ if not target_exists:+ dst_path = viking_fs._uri_to_path(root_uri, ctx=ctx)+ parent_path = dst_path.rsplit("/", 1)[0] if "/" in dst_path else dst_path++ parent_uri = "/".join(root_uri.rstrip("/").rsplit("/", 1)[:-1])+ if parent_uri:+ await viking_fs.mkdir(parent_uri, exist_ok=True, ctx=ctx)++ async with LockContext(lock_manager, [parent_path], lock_mode="point"):+ if candidate_uri:+ with viking_fs.bind_request_context(ctx):+ root_uri = await self.tree_builder._resolve_unique_uri(+ candidate_uri+ )+ result["root_uri"] = root_uri+ dst_path = viking_fs._uri_to_path(root_uri, ctx=ctx)++ src_path = viking_fs._uri_to_path(temp_uri, ctx=ctx)+ await asyncio.to_thread(viking_fs.agfs.mv, src_path, dst_path)++ lifecycle_lock_handle_id = await self._try_acquire_lifecycle_lock(+ lock_manager, dst_path+ )++ try:+ await viking_fs.delete_temp(parse_result.temp_dir_path, ctx=ctx)+ except Exception:+ pass++ result["temp_uri"] = root_uri+ else:+ resource_path = viking_fs._uri_to_path(root_uri, ctx=ctx)lifecycle_lock_handle_id = await self._try_acquire_lifecycle_lock(- lock_manager, dst_path+ lock_manager, resource_path)-+ except Exception:+ stage_status = "error"+ raise+ finally:try:- await viking_fs.delete_temp(parse_result.temp_dir_path, ctx=ctx)+ ResourceIngestionEventDataSource.record_stage(+ stage="persist",+ status=str(stage_status),+ duration_seconds=float(time.perf_counter() - stage_start),+ account_id=getattr(ctx, "account_id", None),+ )except Exception:pass- result["temp_uri"] = root_uri- else:- # 增量更新:对目标目录加 SUBTREE 锁- resource_path = viking_fs._uri_to_path(root_uri, ctx=ctx)- lifecycle_lock_handle_id = await self._try_acquire_lifecycle_lock(- lock_manager, resource_path- )-# ============ Phase 4: Optional Steps ============build_index = kwargs.get("build_index", True)temp_uri_for_summarize = result.get("temp_uri") or parse_result.temp_dir_path@@ -273,6 +318,8 @@ class ResourceProcessor:skip_vec = not build_indexis_code_repo = parse_result.source_format == "repository"try:+ stage_start = time.perf_counter()+ stage_status = "ok"with telemetry.measure("resource.summarize"):await self._get_summarizer().summarize(resource_uris=[result["root_uri"]],@@ -286,6 +333,17 @@ class ResourceProcessor:except Exception as e:logger.error(f"Summarization failed: {e}")result["warnings"] = result.get("warnings", []) + [f"Summarization failed: {e}"]+ stage_status = "error"+ finally:+ try:+ ResourceIngestionEventDataSource.record_stage(+ stage="summarize",+ status=str(stage_status),+ duration_seconds=float(time.perf_counter() - stage_start),+ account_id=getattr(ctx, "account_id", None),+ )+ except Exception:+ passelif lifecycle_lock_handle_id:# 无下游处理接管锁,主动释放from openviking.storage.transaction import get_lock_managerdiff --git a/tests/metrics/account/test_context.py b/tests/metrics/account/test_context.pynew file mode 100644index 000000000..6dbaf39e1--- /dev/null+++ b/tests/metrics/account/test_context.py@@ -0,0 +1,20 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""Tests for metrics request-scoped account context propagation."""++from openviking.metrics.account_context import (+ bind_metric_account_context,+ get_metric_account_context,+ reset_metric_account_context,+)+++def test_http_account_context_round_trip():+ token = bind_metric_account_context(account_id="acct-1")+ try:+ assert get_metric_account_context().http_account_id == "acct-1"+ finally:+ reset_metric_account_context(token)++ assert get_metric_account_context().http_account_id is Nonediff --git a/tests/metrics/account/test_integration.py b/tests/metrics/account/test_integration.pynew file mode 100644index 000000000..6b8834bd7--- /dev/null+++ b/tests/metrics/account/test_integration.py@@ -0,0 +1,479 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""Integration-oriented tests for account-aware metric writes."""++from types import SimpleNamespace++import pytest+from starlette.requests import Request++import openviking.metrics.http_middleware as http_middleware+from openviking.metrics.account_context import (+ bind_metric_account_context,+ reset_metric_account_context,+)+from openviking.metrics.account_dimension import MetricAccountDimensionPolicy+from openviking.metrics.collectors.embedding import EmbeddingCollector+from openviking.metrics.collectors.http import HTTPCollector+from openviking.metrics.collectors.task_tracker import TaskTrackerCollector+from openviking.metrics.collectors.vlm import VLMCollector+from openviking.metrics.core.registry import MetricRegistry+from openviking.metrics.datasources.base import EventMetricDataSource+from openviking.metrics.datasources.http import HttpRequestLifecycleDataSource+from openviking.metrics.datasources.model_usage import (+ VLMEventDataSource,+)+from openviking.metrics.exporters.prometheus import PrometheusExporter+from openviking.metrics.global_api import configure_metric_account_dimension, shutdown_metrics+from openviking.metrics.http_middleware import (+ _INFLIGHT,+ _extract_request_account_id,+ _get_route_template,+ _inflight_delta,+ create_http_metrics_middleware,+)+from openviking.models.vlm.base import VLMBase+++def test_http_collector_emits_account_label():+ registry = MetricRegistry()+ configure_metric_account_dimension(+ policy=MetricAccountDimensionPolicy(+ enabled=True,+ metric_allowlist={+ HTTPCollector.REQUESTS_TOTAL,+ HTTPCollector.REQUEST_DURATION_SECONDS,+ HTTPCollector.INFLIGHT_REQUESTS,+ },+ max_active_accounts=10,+ )+ )+ collector = HTTPCollector()++ token = bind_metric_account_context(account_id="acct-1")+ try:+ collector.receive(+ "http.request",+ {+ "method": "GET",+ "route": "/items",+ "status": "200",+ "duration_seconds": 0.1,+ },+ registry,+ )+ finally:+ reset_metric_account_context(token)+ shutdown_metrics(app=None)++ text = PrometheusExporter(registry=registry).render()+ assert (+ 'openviking_http_requests_total{account_id="acct-1",method="GET",route="/items",status="200"} 1'+ in text+ )+++def test_state_collector_emits_unknown_when_metric_not_allowlisted(monkeypatch):+ class DummyTracker:+ def snapshot_counts_by_type(self):+ return {+ "session_commit": {"pending": 1, "running": 0, "completed": 0, "failed": 0},+ }++ import openviking.metrics.datasources.task as task_datasource_module+ from openviking.metrics.datasources.task import TaskStateDataSource++ monkeypatch.setattr(task_datasource_module, "get_task_tracker", lambda: DummyTracker())+ configure_metric_account_dimension(+ policy=MetricAccountDimensionPolicy(+ enabled=True,+ metric_allowlist={HTTPCollector.REQUESTS_TOTAL},+ max_active_accounts=10,+ )+ )+ registry = MetricRegistry()+ collector = TaskTrackerCollector(data_source=TaskStateDataSource())++ token = bind_metric_account_context(account_id="acct-9")+ try:+ collector.collect(registry)+ finally:+ reset_metric_account_context(token)+ shutdown_metrics(app=None)++ text = PrometheusExporter(registry=registry).render()+ assert 'openviking_task_pending{task_type="session_commit"} 1.0' in text+++def test_http_request_datasource_propagates_explicit_account_id(monkeypatch):+ captured: list[tuple[str, dict]] = []++ def _fake_emit(event_name: str, payload: dict) -> None:+ captured.append((event_name, dict(payload)))++ monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)++ HttpRequestLifecycleDataSource.record_request(+ method="POST",+ route="/api/v1/resources",+ status="200",+ duration_seconds=0.25,+ account_id="acct-http",+ )+ HttpRequestLifecycleDataSource.set_inflight(+ route="/api/v1/resources",+ value=1.0,+ account_id="acct-http",+ )++ assert captured == [+ (+ "http.request",+ {+ "method": "POST",+ "route": "/api/v1/resources",+ "status": "200",+ "duration_seconds": 0.25,+ "account_id": "acct-http",+ },+ ),+ (+ "http.inflight",+ {+ "route": "/api/v1/resources",+ "value": 1.0,+ "account_id": "acct-http",+ },+ ),+ ]+++def test_vlm_collector_uses_explicit_account_id_from_payload():+ registry = MetricRegistry()+ configure_metric_account_dimension(+ policy=MetricAccountDimensionPolicy(+ enabled=True,+ metric_allowlist={+ VLMCollector.CALLS_TOTAL,+ VLMCollector.CALL_DURATION_SECONDS,+ VLMCollector.TOKENS_INPUT_TOTAL,+ VLMCollector.TOKENS_OUTPUT_TOTAL,+ VLMCollector.TOKENS_TOTAL,+ },+ max_active_accounts=10,+ )+ )++ VLMCollector().receive(+ "vlm.call",+ {+ "provider": "volcengine",+ "model_name": "m1",+ "duration_seconds": 1.2,+ "prompt_tokens": 10,+ "completion_tokens": 5,+ "account_id": "acct-vlm",+ },+ registry,+ )+ text = PrometheusExporter(registry=registry).render()+ shutdown_metrics(app=None)++ assert (+ 'openviking_vlm_calls_total{account_id="acct-vlm",model_name="m1",provider="volcengine"} 1'+ in text+ )+++def test_embedding_collector_uses_explicit_account_id_from_payload():+ registry = MetricRegistry()+ configure_metric_account_dimension(+ policy=MetricAccountDimensionPolicy(+ enabled=True,+ metric_allowlist={+ EmbeddingCollector.REQUESTS_TOTAL,+ EmbeddingCollector.LATENCY_SECONDS,+ EmbeddingCollector.ERRORS_TOTAL,+ },+ max_active_accounts=10,+ )+ )++ collector = EmbeddingCollector()+ collector.receive(+ "embedding.success",+ {"latency_seconds": 0.6, "account_id": "acct-embed"},+ registry,+ )+ collector.receive(+ "embedding.error",+ {"error_code": "timeout", "account_id": "acct-embed"},+ registry,+ )+ text = PrometheusExporter(registry=registry).render()+ shutdown_metrics(app=None)++ assert 'openviking_embedding_requests_total{account_id="acct-embed",status="ok"} 1' in text+ assert (+ 'openviking_embedding_errors_total{account_id="acct-embed",error_code="timeout"} 1' in text+ )+++def test_extract_request_account_id_prefers_authenticated_request_state():+ request = Request(+ {+ "type": "http",+ "method": "GET",+ "path": "/api/v1/resources",+ "headers": [(b"x-openviking-account", b"header-account")],+ "state": {},+ }+ )+ request.state.metric_account_id = "state-account"++ assert _extract_request_account_id(request) == "state-account"+++def test_extract_request_account_id_does_not_trust_raw_header_when_unauthenticated():+ request = Request(+ {+ "type": "http",+ "method": "GET",+ "path": "/api/v1/resources",+ "headers": [(b"x-openviking-account", b"header-account")],+ "state": {},+ }+ )+ assert _extract_request_account_id(request) is None+++def test_get_route_template_uses_low_cardinality_fallback_for_unmatched_route():+ request = Request(+ {+ "type": "http",+ "method": "GET",+ "path": "/api/v1/resources/123e4567-e89b-12d3-a456-426614174000",+ "headers": [],+ "state": {},+ }+ )++ assert _get_route_template(request) == "/__unmatched__"+++def test_inflight_delta_removes_zero_value_entries():+ _INFLIGHT.clear()++ assert _inflight_delta("/api/v1/resources", None, +1) == 1+ assert _INFLIGHT[("/api/v1/resources", None)] == 1++ assert _inflight_delta("/api/v1/resources", None, -1) == 0+ assert ("/api/v1/resources", None) not in _INFLIGHT+++async def test_http_metrics_middleware_emits_authenticated_account_id(monkeypatch):+ captured: list[tuple[str, dict]] = []+ middleware = create_http_metrics_middleware()++ def _fake_emit(event_name: str, payload: dict) -> None:+ captured.append((event_name, dict(payload)))++ monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)++ async def _call_next(request: Request):+ request.state.metric_account_id = "acct-real"+ return SimpleNamespace(status_code=200)++ request = Request(+ {+ "type": "http",+ "method": "POST",+ "path": "/api/v1/resources",+ "route": SimpleNamespace(path="/api/v1/resources"),+ "headers": [],+ "state": {},+ }+ )++ await middleware(request, _call_next)++ request_events = [payload for event_name, payload in captured if event_name == "http.request"]+ assert request_events+ assert request_events[0] == {+ "method": "POST",+ "route": "/api/v1/resources",+ "status": "200",+ "duration_seconds": request_events[0]["duration_seconds"],+ "account_id": "acct-real",+ }+ assert isinstance(request_events[0]["duration_seconds"], float)+ assert request_events[0]["duration_seconds"] >= 0.0+ assert any(+ event_name == "http.inflight" and payload.get("account_id") == "acct-real"+ for event_name, payload in captured+ )+++async def test_http_metrics_middleware_ignores_internal_metrics_route(monkeypatch):+ captured: list[tuple[str, dict]] = []+ middleware = create_http_metrics_middleware()++ def _fake_emit(event_name: str, payload: dict) -> None:+ captured.append((event_name, dict(payload)))++ monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)++ async def _call_next(_request: Request):+ return SimpleNamespace(status_code=200)++ request = Request(+ {+ "type": "http",+ "method": "GET",+ "path": "/metrics",+ "headers": [],+ "state": {},+ }+ )++ await middleware(request, _call_next)++ assert captured == []+++async def test_http_metrics_middleware_still_records_business_route(monkeypatch):+ captured: list[tuple[str, dict]] = []+ middleware = create_http_metrics_middleware()++ def _fake_emit(event_name: str, payload: dict) -> None:+ captured.append((event_name, dict(payload)))++ monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)++ async def _call_next(request: Request):+ request.state.metric_account_id = "acct-real"+ return SimpleNamespace(status_code=200)++ request = Request(+ {+ "type": "http",+ "method": "POST",+ "path": "/api/v1/resources",+ "route": SimpleNamespace(path="/api/v1/resources"),+ "headers": [],+ "state": {},+ }+ )++ await middleware(request, _call_next)++ assert any(event_name == "http.request" for event_name, _payload in captured)+++async def test_http_metrics_middleware_uses_route_bound_during_call_next(monkeypatch):+ captured: list[tuple[str, dict]] = []+ middleware = create_http_metrics_middleware()++ def _fake_emit(event_name: str, payload: dict) -> None:+ captured.append((event_name, dict(payload)))++ monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)++ async def _call_next(request: Request):+ request.scope["route"] = SimpleNamespace(path="/api/v1/sessions/{session_id}")+ return SimpleNamespace(status_code=200)++ request = Request(+ {+ "type": "http",+ "method": "GET",+ "path": "/api/v1/sessions/session_123",+ "headers": [],+ "state": {},+ }+ )++ await middleware(request, _call_next)++ request_events = [payload for event_name, payload in captured if event_name == "http.request"]+ assert request_events+ assert request_events[0]["route"] == "/api/v1/sessions/{session_id}"+++async def test_http_metrics_middleware_logs_debug_when_metrics_write_fails(monkeypatch):+ middleware = create_http_metrics_middleware()+ debug_calls: list[tuple[str, tuple, dict]] = []++ def _boom(**_kwargs):+ raise RuntimeError("metrics write failed")++ def _debug(message, *args, **kwargs):+ debug_calls.append((message, args, kwargs))++ monkeypatch.setattr(HttpRequestLifecycleDataSource, "set_inflight", staticmethod(_boom))+ monkeypatch.setattr(http_middleware.logger, "isEnabledFor", lambda _level: True)+ monkeypatch.setattr(http_middleware.logger, "debug", _debug)++ async def _call_next(request: Request):+ request.state.metric_account_id = "acct-real"+ return SimpleNamespace(status_code=200)++ request = Request(+ {+ "type": "http",+ "method": "POST",+ "path": "/api/v1/resources",+ "headers": [],+ "state": {},+ }+ )++ await middleware(request, _call_next)++ assert any(+ message == "http metrics write failed: %s" for message, _args, _kwargs in debug_calls+ )+ assert any("http.inflight" in str(args) for _message, args, _kwargs in debug_calls)+++class _DummyVLM(VLMBase):+ def get_completion(self, *args, **kwargs):+ return ""++ async def get_completion_async(self, *args, **kwargs):+ return ""++ def get_vision_completion(self, *args, **kwargs):+ return ""++ async def get_vision_completion_async(self, *args, **kwargs):+ return ""+++def test_vlm_base_update_token_usage_propagates_current_account(monkeypatch):+ captured: dict[str, object] = {}++ def _fake_record_call(**kwargs):+ captured.update(kwargs)++ monkeypatch.setattr(VLMEventDataSource, "record_call", staticmethod(_fake_record_call))++ token = bind_metric_account_context(account_id="acct-vlm-callsite")+ try:+ _DummyVLM({"provider": "volcengine", "model": "m1"}).update_token_usage(+ model_name="m1",+ provider="volcengine",+ prompt_tokens=3,+ completion_tokens=2,+ duration_seconds=0.5,+ )+ finally:+ reset_metric_account_context(token)++ assert captured["account_id"] == "acct-vlm-callsite"diff --git a/tests/metrics/account/test_policy.py b/tests/metrics/account/test_policy.pynew file mode 100644index 000000000..6154ca8c7--- /dev/null+++ b/tests/metrics/account/test_policy.py@@ -0,0 +1,73 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""Tests for account-dimension whitelist and overflow policy behavior."""++from openviking.metrics.account_dimension import (+ OVERFLOW_ACCOUNT_ID,+ UNKNOWN_ACCOUNT_ID,+ MetricAccountDimensionPolicy,+)+++def test_account_dimension_disabled_still_emits_unknown():+ policy = MetricAccountDimensionPolicy(+ enabled=False,+ metric_allowlist={"openviking_http_requests_total"},+ max_active_accounts=2,+ )++ assert (+ policy.resolve(metric_name="openviking_http_requests_total", account_id="acct-1")+ == UNKNOWN_ACCOUNT_ID+ )+++def test_account_dimension_non_allowlisted_metric_emits_unknown():+ policy = MetricAccountDimensionPolicy(+ enabled=True,+ metric_allowlist={"openviking_http_requests_total"},+ max_active_accounts=2,+ )++ assert (+ policy.resolve(metric_name="openviking_task_pending", account_id="acct-1")+ == UNKNOWN_ACCOUNT_ID+ )+++def test_account_dimension_overflow_after_limit():+ policy = MetricAccountDimensionPolicy(+ enabled=True,+ metric_allowlist={"openviking_http_requests_total"},+ max_active_accounts=1,+ )++ assert (+ policy.resolve(metric_name="openviking_http_requests_total", account_id="acct-1")+ == "acct-1"+ )+ assert (+ policy.resolve(metric_name="openviking_http_requests_total", account_id="acct-2")+ == OVERFLOW_ACCOUNT_ID+ )+++def test_account_dimension_allowlist_supports_prefix_wildcard():+ policy = MetricAccountDimensionPolicy(+ enabled=True,+ metric_allowlist={"openviking_rerank_*"},+ max_active_accounts=10,+ )++ assert (+ policy.resolve(metric_name="openviking_rerank_calls_total", account_id="acct-1") == "acct-1"+ )+ assert (+ policy.resolve(metric_name="openviking_rerank_tokens_total", account_id="acct-1")+ == "acct-1"+ )+ assert (+ policy.resolve(metric_name="openviking_vlm_calls_total", account_id="acct-1")+ == UNKNOWN_ACCOUNT_ID+ )diff --git a/tests/metrics/collectors/test_category_strategies.py b/tests/metrics/collectors/test_category_strategies.pynew file mode 100644index 000000000..7b73f30ef--- /dev/null+++ b/tests/metrics/collectors/test_category_strategies.py@@ -0,0 +1,119 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++import pytest++from openviking.metrics.collectors.base import (+ CollectorConfig,+ DomainStatsMetricCollector,+ ProbeMetricCollector,+ StateMetricCollector,+)+from openviking.metrics.core.registry import MetricRegistry+++class _FailingProbeCollector(ProbeMetricCollector):+ config = CollectorConfig()++ def __init__(self) -> None:+ self.stale_called = False++ def read_metric_input(self):+ raise RuntimeError("probe read failed")++ def collect_hook(self, registry, metric_input) -> None:+ raise AssertionError("collect_hook must not be called on read failure")++ def collect_stale_hook(self, registry, error: Exception) -> None:+ self.stale_called = True+++class _FailingStateCollector(StateMetricCollector):+ config = CollectorConfig()++ def read_metric_input(self):+ raise RuntimeError("state read failed")++ def collect_hook(self, registry, metric_input) -> None:+ raise AssertionError("collect_hook must not be called on read failure")+++class _StaleOnErrorStateCollector(StateMetricCollector):+ config = CollectorConfig()+ STALE_ON_ERROR = True++ def __init__(self) -> None:+ self.stale_called = False++ def read_metric_input(self):+ raise RuntimeError("state read failed")++ def collect_hook(self, registry, metric_input) -> None:+ raise AssertionError("collect_hook must not be called on read failure")++ def collect_stale_hook(self, registry, error: Exception) -> None:+ self.stale_called = True+++class _DeltaDomainStatsCollector(DomainStatsMetricCollector):+ config = CollectorConfig()++ def read_metric_input(self):+ return None++ def collect_hook(self, registry, metric_input) -> None:+ self.inc_counter_from_cumulative(+ registry=registry,+ metric_name="openviking_test_cumulative_total",+ key=("k",),+ current_value=10,+ )+ self.inc_counter_from_cumulative(+ registry=registry,+ metric_name="openviking_test_cumulative_total",+ key=("k",),+ current_value=15,+ )+ self.inc_counter_from_cumulative(+ registry=registry,+ metric_name="openviking_test_cumulative_total",+ key=("k",),+ current_value=3,+ )+++def test_probe_default_error_strategy_does_not_raise_and_calls_stale_hook():+ registry = MetricRegistry()+ collector = _FailingProbeCollector()++ collector.collect(registry)++ assert collector.stale_called is True+++def test_state_default_error_strategy_raises():+ registry = MetricRegistry()+ collector = _FailingStateCollector()++ with pytest.raises(RuntimeError, match="state read failed"):+ collector.collect(registry)+++def test_state_stale_on_error_strategy_calls_stale_hook_and_does_not_raise():+ registry = MetricRegistry()+ collector = _StaleOnErrorStateCollector()++ collector.collect(registry)++ assert collector.stale_called is True+++def test_domain_stats_delta_helper_applies_non_negative_deltas_and_handles_resets():+ registry = MetricRegistry()+ collector = _DeltaDomainStatsCollector()++ collector.collect(registry)++ counters = dict(registry.iter_counters())+ series = dict(counters["openviking_test_cumulative_total"])+ assert series[()] == 10 + 5 + 3diff --git a/tests/metrics/collectors/test_default_collectors.py b/tests/metrics/collectors/test_default_collectors.pynew file mode 100644index 000000000..886a23ba4--- /dev/null+++ b/tests/metrics/collectors/test_default_collectors.py@@ -0,0 +1,146 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++import time++from openviking.metrics.collectors.lock import LockCollector+from openviking.metrics.collectors.observer_health import ObserverHealthCollector+from openviking.metrics.collectors.queue import QueueCollector+from openviking.metrics.collectors.task_tracker import TaskTrackerCollector+from openviking.metrics.collectors.vikingdb import VikingDBCollector+from openviking.metrics.core.registry import MetricRegistry+from openviking.metrics.datasources.observer_state import (+ LockStateDataSource,+ ObserverStateDataSource,+ VikingDBStateDataSource,+)+from openviking.metrics.datasources.queue import QueuePipelineStateDataSource+from openviking.metrics.datasources.task import TaskStateDataSource+from openviking.metrics.exporters.prometheus import PrometheusExporter+++def test_queue_collector_maps_status(monkeypatch):+ class DummyQueueStatus:+ def __init__(+ self, pending: int, in_progress: int, processed: int, error_count: int+ ) -> None:+ self.pending = pending+ self.in_progress = in_progress+ self.processed = processed+ self.error_count = error_count++ class DummyQueueManager:+ async def check_status(self):+ return {+ "semantic": DummyQueueStatus(3, 1, 10, 2),+ "embedding": DummyQueueStatus(5, 0, 7, 0),+ }++ monkeypatch.setattr(+ "openviking.metrics.datasources.queue.get_queue_manager",+ lambda: DummyQueueManager(),+ )+ registry = MetricRegistry()+ QueueCollector(data_source=QueuePipelineStateDataSource()).collect(registry)+ text = PrometheusExporter(registry=registry).render()+ assert 'openviking_queue_pending{queue="semantic"} 3.0' in text+ assert 'openviking_queue_in_progress{queue="semantic"} 1.0' in text+ assert 'openviking_queue_processed_total{queue="semantic"} 10' in text+ assert 'openviking_queue_errors_total{queue="semantic"} 2' in text+++def test_task_tracker_collector_maps_counts(monkeypatch):+ class DummyTracker:+ def snapshot_counts_by_type(self):+ return {+ "session_commit": {"pending": 1, "running": 2, "completed": 3, "failed": 4},+ }++ import openviking.metrics.datasources.task as task_datasource_module++ monkeypatch.setattr(task_datasource_module, "get_task_tracker", lambda: DummyTracker())+ registry = MetricRegistry()+ TaskTrackerCollector(data_source=TaskStateDataSource()).collect(registry)+ text = PrometheusExporter(registry=registry).render()+ assert 'openviking_task_pending{task_type="session_commit"} 1.0' in text+ assert 'openviking_task_running{task_type="session_commit"} 2.0' in text+ assert 'openviking_task_completed{task_type="session_commit"} 3.0' in text+ assert 'openviking_task_failed{task_type="session_commit"} 4.0' in text+++def test_observer_health_collector_maps_component_status():+ class Status:+ def __init__(self, ok: bool, err: bool) -> None:+ self.is_healthy = ok+ self.has_errors = err++ class Observer:+ queue = Status(True, False)+ models = Status(True, False)+ lock = Status(False, True)+ retrieval = Status(True, True)++ def vikingdb(self, ctx=None):+ return Status(True, False)++ class Debug:+ observer = Observer()++ class Service:+ debug = Debug()++ registry = MetricRegistry()+ ObserverHealthCollector(data_source=ObserverStateDataSource(service=Service())).collect(+ registry+ )+ text = PrometheusExporter(registry=registry).render()+ assert 'openviking_component_health{component="lock",valid="1"} 0.0' in text+ assert 'openviking_component_errors{component="lock",valid="1"} 1.0' in text+ assert 'openviking_component_health{component="vikingdb",valid="1"} 1.0' in text+++def test_lock_collector_counts_active_and_stale(monkeypatch):+ class Handle:+ def __init__(self, locks: int, last_active_at: float) -> None:+ self.locks = [object()] * locks+ self.last_active_at = last_active_at++ class DummyLockManager:+ def get_active_handles(self):+ now = time.time()+ return {+ "a": Handle(locks=2, last_active_at=now - 10),+ "b": Handle(locks=1, last_active_at=now - 1000),+ }++ monkeypatch.setattr(+ "openviking.metrics.datasources.observer_state.get_lock_manager",+ lambda: DummyLockManager(),+ )+ registry = MetricRegistry()+ LockCollector(data_source=LockStateDataSource()).collect(registry)+ text = PrometheusExporter(registry=registry).render()+ assert "openviking_lock_active 3.0" in text+ assert "openviking_lock_stale 1.0" in text+++def test_vikingdb_collector_exports_health_and_count(monkeypatch):+ class DummyVikingDB:+ collection_name = "my_collection"++ async def health_check(self):+ return True++ async def count(self, filter=None, ctx=None):+ return 123++ class Service:+ _vikingdb_manager = DummyVikingDB()++ registry = MetricRegistry()+ VikingDBCollector(data_source=VikingDBStateDataSource(service=Service())).collect(registry)+ text = PrometheusExporter(registry=registry).render()+ assert 'openviking_vikingdb_collection_health{collection="my_collection",valid="1"} 1.0' in text+ assert (+ 'openviking_vikingdb_collection_vectors{collection="my_collection",valid="1"} 123.0' in text+ )diff --git a/tests/metrics/collectors/test_domain_stats.py b/tests/metrics/collectors/test_domain_stats.pynew file mode 100644index 000000000..960feb539--- /dev/null+++ b/tests/metrics/collectors/test_domain_stats.py@@ -0,0 +1,127 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+++def test_observer_state_collector_valid_and_failure_keeps_last_values(registry, render_prometheus):+ from openviking.metrics.collectors.observer_state import ObserverStateCollector++ class S:+ def __init__(self, ok: bool, err: bool):+ self.is_healthy = ok+ self.has_errors = err++ class DS:+ def __init__(self):+ self.fail = False++ def read_component_states(self):+ if self.fail:+ raise RuntimeError("boom")+ return {"a": S(True, False), "b": S(False, True), "c": S(True, True)}++ ds = DS()+ c = ObserverStateCollector(data_source=ds)+ c.collect(registry)+ text = render_prometheus(registry)+ assert 'openviking_observer_components_total{valid="1"} 3.0' in text+ assert 'openviking_observer_components_unhealthy{valid="1"} 1.0' in text+ assert 'openviking_observer_components_with_errors{valid="1"} 2.0' in text++ ds.fail = True+ c.collect(registry)+ text2 = render_prometheus(registry)+ assert 'openviking_observer_components_total{valid="0"} 3.0' in text2+ assert 'openviking_observer_components_unhealthy{valid="0"} 1.0' in text2+ assert 'openviking_observer_components_with_errors{valid="0"} 2.0' in text2+++def test_model_usage_collector_delta_and_available_gauge(registry, render_prometheus):+ from openviking.metrics.collectors.model_usage import ModelUsageCollector++ class DS:+ def __init__(self):+ self.data = {+ "vlm": {+ "available": True,+ "usage_by_model": {+ "m1": {+ "usage_by_provider": {+ "p1": {+ "prompt_tokens": 2,+ "completion_tokens": 3,+ "total_tokens": 5,+ "call_count": 1,+ }+ }+ }+ },+ },+ "embedding": {+ "available": False,+ "usage_by_model": {},+ },+ "rerank": {+ "available": False,+ "usage_by_model": {},+ },+ }++ def read_model_usage(self):+ return self.data++ ds = DS()+ c = ModelUsageCollector(data_source=ds)+ c.collect(registry)+ text = render_prometheus(registry)+ assert 'openviking_model_usage_available{model_type="vlm",valid="1"} 1.0' in text+ assert 'openviking_model_usage_available{model_type="embedding",valid="1"} 0.0' in text+ assert 'openviking_model_usage_available{model_type="rerank",valid="1"} 0.0' in text+ assert "openviking_model_usage_valid" not in text+ assert 'openviking_model_calls_total{model_name="m1",model_type="vlm",provider="p1"} 1' in text+ assert (+ 'openviking_model_tokens_total{model_name="m1",model_type="vlm",provider="p1",token_type="total"} 5'+ in text+ )++ ds.data["vlm"]["usage_by_model"]["m1"]["usage_by_provider"]["p1"]["call_count"] = 2+ ds.data["vlm"]["usage_by_model"]["m1"]["usage_by_provider"]["p1"]["total_tokens"] = 7+ ds.data["vlm"]["usage_by_model"]["m1"]["usage_by_provider"]["p1"]["prompt_tokens"] = 3+ ds.data["vlm"]["usage_by_model"]["m1"]["usage_by_provider"]["p1"]["completion_tokens"] = 4+ c.collect(registry)+ text2 = render_prometheus(registry)+ assert 'openviking_model_calls_total{model_name="m1",model_type="vlm",provider="p1"} 2' in text2+ assert (+ 'openviking_model_tokens_total{model_name="m1",model_type="vlm",provider="p1",token_type="total"} 7'+ in text2+ )+++def test_model_usage_collector_failure_reuses_last_available_state_with_valid_zero(+ registry, render_prometheus+):+ from openviking.metrics.collectors.model_usage import ModelUsageCollector++ class DS:+ def __init__(self):+ self.fail = False++ def read_model_usage(self):+ if self.fail:+ raise RuntimeError("boom")+ return {+ "vlm": {"available": True, "usage_by_model": {}},+ "embedding": {"available": False, "usage_by_model": {}},+ "rerank": {"available": False, "usage_by_model": {}},+ }++ ds = DS()+ collector = ModelUsageCollector(data_source=ds)++ collector.collect(registry)+ ds.fail = True+ collector.collect(registry)++ text = render_prometheus(registry)+ assert 'openviking_model_usage_available{model_type="vlm",valid="0"} 1.0' in text+ assert 'openviking_model_usage_available{model_type="embedding",valid="0"} 0.0' in text+ assert 'openviking_model_usage_available{model_type="rerank",valid="0"} 0.0' in textdiff --git a/tests/metrics/collectors/test_event_routing.py b/tests/metrics/collectors/test_event_routing.pynew file mode 100644index 000000000..7efe399cf--- /dev/null+++ b/tests/metrics/collectors/test_event_routing.py@@ -0,0 +1,46 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from openviking.metrics.collectors.base import EventMetricCollector+from openviking.metrics.collectors.cache import CacheCollector+++class _DummyEventCollector(EventMetricCollector):+ SUPPORTED_EVENTS = frozenset({"demo.hit"})++ def __init__(self) -> None:+ self.seen: list[tuple[str, str]] = []++ def receive_hook(self, event_name: str, payload: dict, registry) -> None:+ self.seen.append(+ (+ str(event_name),+ str(payload["value"]),+ registry.collector_name() if hasattr(registry, "collector_name") else "registry",+ )+ )+++def test_event_metric_collector_uses_supported_events_and_receive_hook(registry):+ collector = _DummyEventCollector()++ collector.receive("demo.hit", {"value": "ok"}, registry)+ collector.receive("demo.unknown", {"value": "skip"}, registry)++ assert collector.SUPPORTED_EVENTS == frozenset({"demo.hit"})+ assert collector.seen == [("demo.hit", "ok", "registry")]+++def test_cache_collector_uses_supported_events_and_receive_hook_routing(+ registry, render_prometheus+):+ collector = CacheCollector()++ collector.receive("cache.hit", {"level": "L0"}, registry)+ collector.receive("cache.miss", {"level": "L1"}, registry)++ text = render_prometheus(registry)++ assert collector.SUPPORTED_EVENTS == frozenset({"cache.hit", "cache.miss"})+ assert 'openviking_cache_hits_total{level="L0"} 1' in text+ assert 'openviking_cache_misses_total{level="L1"} 1' in textdiff --git a/tests/metrics/collectors/test_probe_valid_stale_contracts.py b/tests/metrics/collectors/test_probe_valid_stale_contracts.pynew file mode 100644index 000000000..404aad3f0--- /dev/null+++ b/tests/metrics/collectors/test_probe_valid_stale_contracts.py@@ -0,0 +1,138 @@+"""Contract tests for probe collectors (valid/stale semantics)."""++from __future__ import annotations++from dataclasses import dataclass++import pytest++from openviking.metrics.collectors.async_system_probe import AsyncSystemProbeCollector+from openviking.metrics.collectors.encryption_probe import EncryptionProbeCollector+from openviking.metrics.collectors.model_provider_probe import ModelProviderProbeCollector+from openviking.metrics.collectors.retrieval_backend_probe import RetrievalBackendProbeCollector+from openviking.metrics.collectors.service_probe import ServiceProbeCollector+from openviking.metrics.collectors.storage_probe import StorageProbeCollector+from openviking.metrics.core.base import ReadEnvelope+++@dataclass+class _ProbeDataSource:+ """+ Minimal stub datasource for probe collectors.++ The stub can return a `ReadEnvelope` (ok or failure) or raise an exception to exercise+ probe collector error paths.+ """++ value: object | None = None+ ok: bool = True+ raises: bool = False++ def read_probe_state(self):+ """Return a probe envelope or raise an exception depending on the configured flags."""+ if self.raises:+ raise RuntimeError("probe read failed")+ return ReadEnvelope(ok=self.ok, value=self.value, error_type="probe", error_message="fail")+++def test_service_probe_collector_emits_valid_on_success_and_invalid_on_failure(+ registry, render_prometheus+):+ """ServiceProbeCollector must export readiness as `valid=1` on success and `valid=0` on failure."""+ ds = _ProbeDataSource(+ value={"service_readiness": True, "api_key_manager_readiness": False}, ok=True+ )+ c = ServiceProbeCollector(data_source=ds)+ c.collect(registry)+ text = render_prometheus(registry)+ assert 'openviking_service_readiness{valid="1"} 1.0' in text+ assert 'openviking_api_key_manager_readiness{valid="1"} 0.0' in text++ # Failure via envelope must flip to `valid=0` replacement series.+ ds.ok = False+ c.collect(registry)+ text2 = render_prometheus(registry)+ assert 'openviking_service_readiness{valid="0"} 0.0' in text2+ assert 'openviking_api_key_manager_readiness{valid="0"} 0.0' in text2+++def test_storage_probe_collector_preserves_last_probe_set_and_marks_invalid(+ registry, render_prometheus+):+ """StorageProbeCollector must re-publish last-known probe set under `valid=0` on failure."""+ ds = _ProbeDataSource(value={"agfs": True, "other": False}, ok=True)+ c = StorageProbeCollector(data_source=ds)+ c.collect(registry)+ text = render_prometheus(registry)+ assert 'openviking_storage_readiness{probe="agfs",valid="1"} 1.0' in text+ assert 'openviking_storage_readiness{probe="other",valid="1"} 0.0' in text++ ds.raises = True+ with pytest.raises(RuntimeError):+ ds.read_probe_state() # sanity: ensure exception path is exercised+ c.collect(registry)+ text2 = render_prometheus(registry)+ assert 'openviking_storage_readiness{probe="agfs",valid="0"} 0.0' in text2+ assert 'openviking_storage_readiness{probe="other",valid="0"} 0.0' in text2+++def test_retrieval_backend_probe_collector_preserves_last_probe_set_and_marks_invalid(+ registry, render_prometheus+):+ """RetrievalBackendProbeCollector must re-publish last-known probes under `valid=0` on failure."""+ ds = _ProbeDataSource(value={"vikingdb": True}, ok=True)+ c = RetrievalBackendProbeCollector(data_source=ds)+ c.collect(registry)+ text = render_prometheus(registry)+ assert 'openviking_retrieval_backend_readiness{probe="vikingdb",valid="1"} 1.0' in text++ ds.ok = False+ c.collect(registry)+ text2 = render_prometheus(registry)+ assert 'openviking_retrieval_backend_readiness{probe="vikingdb",valid="0"} 0.0' in text2+++def test_encryption_probe_collector_uses_last_provider_on_failure(registry, render_prometheus):+ """EncryptionProbeCollector must preserve last provider label when emitting `valid=0` series."""+ ds = _ProbeDataSource(value=(True, "volcengine"), ok=True)+ c = EncryptionProbeCollector(data_source=ds)+ c.collect(registry)+ text = render_prometheus(registry)+ assert 'openviking_encryption_component_health{valid="1"} 1.0' in text+ assert 'openviking_encryption_root_key_ready{valid="1"} 1.0' in text+ assert 'openviking_encryption_kms_provider_ready{provider="volcengine",valid="1"} 1.0' in text++ ds.raises = True+ c.collect(registry)+ text2 = render_prometheus(registry)+ assert 'openviking_encryption_component_health{valid="0"} 0.0' in text2+ assert 'openviking_encryption_root_key_ready{valid="0"} 0.0' in text2+ assert 'openviking_encryption_kms_provider_ready{provider="volcengine",valid="0"} 0.0' in text2+++def test_model_provider_probe_collector_uses_last_provider_on_failure(registry, render_prometheus):+ """ModelProviderProbeCollector must preserve last provider label when emitting `valid=0` series."""+ ds = _ProbeDataSource(value={"provider": ("volcengine", True)}, ok=True)+ c = ModelProviderProbeCollector(data_source=ds)+ c.collect(registry)+ text = render_prometheus(registry)+ assert 'openviking_model_provider_readiness{provider="volcengine",valid="1"} 1.0' in text++ ds.ok = False+ c.collect(registry)+ text2 = render_prometheus(registry)+ assert 'openviking_model_provider_readiness{provider="volcengine",valid="0"} 0.0' in text2+++def test_async_system_probe_collector_marks_invalid_on_failure(registry, render_prometheus):+ """AsyncSystemProbeCollector must publish `valid` and emit invalid series on failure."""+ ds = _ProbeDataSource(value={"queue": True}, ok=True)+ c = AsyncSystemProbeCollector(data_source=ds)+ c.collect(registry)+ text = render_prometheus(registry)+ assert 'openviking_async_system_readiness{probe="queue",valid="1"} 1.0' in text++ ds.raises = True+ c.collect(registry)+ text2 = render_prometheus(registry)+ assert 'openviking_async_system_readiness{probe="queue",valid="0"} 0.0' in text2diff --git a/tests/metrics/collectors/test_session_collector.py b/tests/metrics/collectors/test_session_collector.pynew file mode 100644index 000000000..4d79a18d2--- /dev/null+++ b/tests/metrics/collectors/test_session_collector.py@@ -0,0 +1,43 @@+"""Module-level tests for `openviking.metrics.collectors.session`."""++from __future__ import annotations++from openviking.metrics.collectors.session import SessionCollector+++def test_session_collector_records_lifecycle_and_contexts_and_archive(registry, render_prometheus):+ """SessionCollector must translate supported events into bounded counters."""+ c = SessionCollector()+ c.receive("session.lifecycle", {"action": "create", "status": "ok"}, registry)+ c.receive("session.contexts_used", {"action": "create", "delta": 2}, registry)+ c.receive("session.archive", {"status": "ok"}, registry)++ text = render_prometheus(registry)+ assert (+ 'openviking_session_lifecycle_total{account_id="__unknown__",action="create",status="ok"} 1'+ in text+ )+ assert (+ 'openviking_session_contexts_used_total{account_id="__unknown__",action="create"} 2' in text+ )+ assert 'openviking_session_archive_total{account_id="__unknown__",status="ok"} 1' in text+++def test_session_collector_ignores_malformed_payloads_instead_of_raising(+ registry, render_prometheus+):+ """Missing keys must not raise; the collector should best-effort ignore malformed payloads."""+ c = SessionCollector()++ # Guard: base collector should ignore non-dict payloads.+ c.receive("session.lifecycle", "not-a-dict", registry)++ # Malformed dicts: should not raise and should not modify counters.+ c.receive("session.lifecycle", {"action": "create"}, registry)+ c.receive("session.contexts_used", {"action": "create"}, registry)+ c.receive("session.archive", {}, registry)++ text = render_prometheus(registry)+ assert "openviking_session_lifecycle_total" not in text+ assert "openviking_session_contexts_used_total" not in text+ assert "openviking_session_archive_total" not in textdiff --git a/tests/metrics/collectors/test_task_tracker_collector.py b/tests/metrics/collectors/test_task_tracker_collector.pynew file mode 100644index 000000000..c547c73de--- /dev/null+++ b/tests/metrics/collectors/test_task_tracker_collector.py@@ -0,0 +1,22 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from openviking.metrics.collectors.task_tracker import TaskTrackerCollector+from openviking.metrics.core.registry import MetricRegistry+from openviking.metrics.datasources.task import TaskStateDataSource+from openviking.metrics.exporters.prometheus import PrometheusExporter+++def test_task_tracker_collector_clears_disappeared_task_types():+ registry = MetricRegistry()+ collector = TaskTrackerCollector(data_source=TaskStateDataSource())++ collector.collect_hook(registry, {"session_commit": {"pending": 2}})+ text = PrometheusExporter(registry=registry).render()+ assert 'openviking_task_pending{task_type="session_commit"} 2.0' in text++ # When the snapshot no longer includes the task type, previously-exported series must not+ # remain stale forever.+ collector.collect_hook(registry, {})+ text2 = PrometheusExporter(registry=registry).render()+ assert 'task_type="session_commit"' not in text2diff --git a/tests/metrics/collectors/test_telemetry_bridge_collector.py b/tests/metrics/collectors/test_telemetry_bridge_collector.pynew file mode 100644index 000000000..502a7b13c--- /dev/null+++ b/tests/metrics/collectors/test_telemetry_bridge_collector.py@@ -0,0 +1,45 @@+"""Module-level tests for `openviking.metrics.collectors.telemetry_bridge`."""++from __future__ import annotations++from openviking.metrics.collectors.telemetry_bridge import TelemetryBridgeCollector+++def test_telemetry_bridge_collector_records_basic_operation_metrics(registry, render_prometheus):+ """TelemetryBridgeCollector must expand one telemetry summary into core operation metrics."""+ c = TelemetryBridgeCollector()+ c.receive(+ "telemetry.summary",+ {+ "summary": {+ "operation": "resource.process",+ "status": "ok",+ "duration_ms": 250.0,+ "tokens": {"total": 3},+ }+ },+ registry,+ )++ text = render_prometheus(registry)+ assert (+ 'openviking_operation_requests_total{account_id="__unknown__",operation="resource.process",status="ok"} 1'+ in text+ )+ assert (+ 'openviking_operation_tokens_total{account_id="__unknown__",operation="resource.process",token_type="all"} 3'+ in text+ )+++def test_telemetry_bridge_collector_ignores_malformed_payloads_instead_of_raising(+ registry, render_prometheus+):+ """Missing/invalid `summary` payload must not raise; collector should ignore it best-effort."""+ c = TelemetryBridgeCollector()++ c.receive("telemetry.summary", {}, registry)+ c.receive("telemetry.summary", {"summary": "not-a-mapping"}, registry)++ text = render_prometheus(registry)+ assert "openviking_operation_requests_total" not in textdiff --git a/tests/metrics/conftest.py b/tests/metrics/conftest.pynew file mode 100644index 000000000..cd2d322b9--- /dev/null+++ b/tests/metrics/conftest.py@@ -0,0 +1,104 @@+"""+Shared fixtures for `tests/metrics/`.++The metrics test suite is split into feature-aligned subdirectories. This `conftest.py`+provides small, reusable fixtures/helpers that are broadly useful across those areas.+"""++from __future__ import annotations++from collections.abc import Callable, Iterator+from pathlib import Path+from typing import Any++import pytest++from openviking.metrics.core.registry import MetricRegistry+from openviking.metrics.datasources import EventMetricDataSource+from openviking.metrics.exporters.prometheus import PrometheusExporter+from openviking.metrics.global_api import configure_metric_account_dimension, shutdown_metrics+++def project_root() -> Path:+ """+ Return the repository root directory.++ Some contract-style tests read source files directly via absolute paths; this helper+ makes those tests stable even when files are moved across `tests/metrics/*/`.+ """++ return Path(__file__).resolve().parents[2]+++def registry() -> MetricRegistry:+ """+ Return a fresh metric registry for a single test.++ This ensures tests do not share cross-test metric state.+ """++ return MetricRegistry()+++def render_prometheus() -> Callable[[MetricRegistry], str]:+ """+ Return a callable that renders a registry into Prometheus exposition format text.++ This is a fixture (not a plain helper import) so tests across subdirectories can use it+ without importing from `conftest.py`.+ """++ def _render(reg: MetricRegistry) -> str:+ return PrometheusExporter(registry=reg).render()++ return _render+++def patch_event_emit(monkeypatch: pytest.MonkeyPatch) -> list[tuple[str, dict[str, Any]]]:+ """+ Patch the event datasource emission path and capture emitted events.++ Returns:+ A list of `(event_name, payload)` tuples captured during the test.+ """++ captured: list[tuple[str, dict[str, Any]]] = []++ def _emit(event_name: Any = None, payload: Any = None, **kwargs: Any) -> None:+ """+ Capture events emitted through the datasource layer.++ The production `_emit` API is called with positional arguments. Some tests may also+ call it with keyword arguments when patching.+ """+ if event_name is None and "event_name" in kwargs:+ event_name = kwargs["event_name"]+ if payload is None and "payload" in kwargs:+ payload = kwargs["payload"]+ captured.append((str(event_name), dict(payload or {})))++ monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_emit), raising=False)+ return captured+++def configure_account_dimension() -> Iterator[Callable[..., None]]:+ """+ Provide a helper for configuring account dimension policy with guaranteed cleanup.++ The returned function forwards keyword arguments to `configure_metric_account_dimension()`.+ Regardless of test outcomes, `shutdown_metrics(app=None)` is executed to avoid leaking+ global metrics state across tests.+ """++ def _configure(**kwargs: Any) -> None:+ configure_metric_account_dimension(**kwargs)++ try:+ yield _configure+ finally:+ shutdown_metrics(app=None)diff --git a/tests/metrics/contracts/test_architecture_contracts.py b/tests/metrics/contracts/test_architecture_contracts.pynew file mode 100644index 000000000..c2dbe5228--- /dev/null+++ b/tests/metrics/contracts/test_architecture_contracts.py@@ -0,0 +1,117 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from pathlib import Path+++def test_required_core_modules_exist():+ import openviking.metrics.core.base as core_base+ import openviking.metrics.core.refresh as core_refresh+ import openviking.metrics.core.registry as core_registry+ import openviking.metrics.core.runtime as core_runtime+ import openviking.metrics.core.types as core_types++ assert hasattr(core_base, "MetricDataSource")+ assert hasattr(core_base, "MetricCollector")+ assert hasattr(core_base, "MetricExporter")+ assert hasattr(core_refresh, "RefreshGate")+ assert hasattr(core_registry, "MetricRegistry")+ assert hasattr(core_runtime, "EventCollectorRouter")+ assert core_types is not None+++def test_required_architecture_types_exist():+ import openviking.metrics.core.base as base_module++ assert hasattr(base_module, "MetricDataSource")+ assert hasattr(base_module, "MetricCollector")+ assert hasattr(base_module, "MetricExporter")+++def test_required_intermediate_types_exist():+ import openviking.metrics.collectors.base as collector_base_module+ import openviking.metrics.datasources as ds_module+ import openviking.metrics.datasources.base as ds_base_module++ assert hasattr(ds_base_module, "EventMetricDataSource")+ assert hasattr(ds_base_module, "StateMetricDataSource")+ assert hasattr(ds_base_module, "DomainStatsMetricDataSource")+ assert hasattr(ds_base_module, "ProbeMetricDataSource")+ assert hasattr(ds_module, "CacheEventDataSource")+ assert hasattr(ds_module, "TelemetryBridgeEventDataSource")++ assert hasattr(collector_base_module, "EventMetricCollector")+ assert hasattr(collector_base_module, "StateMetricCollector")+ assert hasattr(collector_base_module, "DomainStatsMetricCollector")+ assert hasattr(collector_base_module, "ProbeMetricCollector")+ assert hasattr(collector_base_module, "Refreshable")+ assert not hasattr(collector_base_module, "AbstractMetricCollector")+++def test_prometheus_exporter_inherits_base_metric_exporter():+ import openviking.metrics.exporters as exporters_module+ from openviking.metrics.core.base import MetricExporter+ from openviking.metrics.exporters.prometheus import PrometheusExporter++ assert issubclass(PrometheusExporter, MetricExporter)+ assert not hasattr(exporters_module, "AbstractMetricExporter")+++def test_concrete_datasources_and_collectors_follow_doc_inheritance():+ from openviking.metrics.collectors.base import (+ ProbeMetricCollector,+ Refreshable,+ StateMetricCollector,+ )+ from openviking.metrics.collectors.lock import LockCollector+ from openviking.metrics.collectors.queue import QueueCollector+ from openviking.metrics.collectors.service_probe import ServiceProbeCollector+ from openviking.metrics.collectors.vikingdb import VikingDBCollector+ from openviking.metrics.datasources.base import (+ DomainStatsMetricDataSource,+ ProbeMetricDataSource,+ StateMetricDataSource,+ )+ from openviking.metrics.datasources.observer_state import (+ LockStateDataSource,+ ObserverStateDataSource,+ VikingDBStateDataSource,+ )+ from openviking.metrics.datasources.probes import ServiceProbeDataSource+ from openviking.metrics.datasources.queue import QueuePipelineStateDataSource++ assert issubclass(QueuePipelineStateDataSource, StateMetricDataSource)+ assert issubclass(ObserverStateDataSource, DomainStatsMetricDataSource)+ assert issubclass(LockStateDataSource, StateMetricDataSource)+ assert issubclass(VikingDBStateDataSource, StateMetricDataSource)+ assert issubclass(ServiceProbeDataSource, ProbeMetricDataSource)++ assert issubclass(QueueCollector, StateMetricCollector)+ assert issubclass(LockCollector, StateMetricCollector)+ assert issubclass(VikingDBCollector, StateMetricCollector)+ assert issubclass(ServiceProbeCollector, ProbeMetricCollector)+ assert issubclass(QueueCollector, Refreshable)+ assert issubclass(LockCollector, Refreshable)+ assert issubclass(VikingDBCollector, Refreshable)+ assert issubclass(ServiceProbeCollector, Refreshable)+++def test_metrics_code_has_no_snapshot_api():+ root = Path(__file__).resolve().parents[2] / "openviking" / "metrics"+ for path in root.rglob("*.py"):+ text = path.read_text(encoding="utf-8")+ assert ".snapshot(" not in text, f"snapshot api still found in {path}"+++def test_metric_collector_supports_collect_and_receive():+ from openviking.metrics.core.base import MetricCollector++ class _DummyCollector(MetricCollector):+ @classmethod+ def kind(cls) -> str:+ return "dummy"++ collector = _DummyCollector()+ assert collector.collector_name() == "_DummyCollector"+ assert collector.collect(None) is None+ assert collector.receive("event", {}, None) is Nonediff --git a/tests/metrics/contracts/test_prometheus_observer_removed.py b/tests/metrics/contracts/test_prometheus_observer_removed.pynew file mode 100644index 000000000..eaf33d390--- /dev/null+++ b/tests/metrics/contracts/test_prometheus_observer_removed.py@@ -0,0 +1,19 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from pathlib import Path+++def test_runtime_code_has_no_prometheus_observer_references():+ # tests/metrics/contracts -> repo_root/openviking+ root = Path(__file__).resolve().parents[3] / "openviking"+ banned = (+ "PrometheusObserver",+ "get_prometheus_observer",+ "set_prometheus_observer",+ "prometheus_observer.py",+ )+ for path in root.rglob("*.py"):+ text = path.read_text(encoding="utf-8")+ for needle in banned:+ assert needle not in text, f"{needle} still found in {path}"diff --git a/tests/metrics/contracts/test_pull_collector_templates.py b/tests/metrics/contracts/test_pull_collector_templates.pynew file mode 100644index 000000000..f7029927e--- /dev/null+++ b/tests/metrics/contracts/test_pull_collector_templates.py@@ -0,0 +1,101 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from openviking.metrics.collectors.base import (+ CollectorConfig,+ DomainStatsMetricCollector,+ ProbeMetricCollector,+ StateMetricCollector,+)+from openviking.metrics.core.registry import MetricRegistry+++class _DummyStateCollector(StateMetricCollector):+ config = CollectorConfig()++ def __init__(self) -> None:+ self.seen = []++ def read_metric_input(self):+ return {"value": 3}++ def collect_hook(self, registry, metric_input) -> None:+ self.seen.append(("state", metric_input["value"]))+++class _DummyProbeCollector(ProbeMetricCollector):+ config = CollectorConfig()++ def __init__(self) -> None:+ self.seen = []++ def read_metric_input(self):+ return {"ok": True}++ def collect_hook(self, registry, metric_input) -> None:+ self.seen.append(("probe", metric_input["ok"]))+++class _DummyDomainCollector(DomainStatsMetricCollector):+ config = CollectorConfig()++ def __init__(self) -> None:+ self.seen = []++ def read_metric_input(self):+ return {"total": 7}++ def collect_hook(self, registry, metric_input) -> None:+ self.seen.append(("domain", metric_input["total"]))+++class _DummyFailingStateCollector(StateMetricCollector):+ config = CollectorConfig()++ def __init__(self) -> None:+ self.error_seen = None++ def read_metric_input(self):+ raise RuntimeError("boom")++ def collect_hook(self, registry, metric_input) -> None:+ raise AssertionError("should not be called")++ def collect_error_hook(self, registry, error: Exception) -> None:+ self.error_seen = str(error)+++def test_state_metric_collector_collect_uses_template_hooks():+ registry = MetricRegistry()+ collector = _DummyStateCollector()++ collector.collect(registry)++ assert collector.seen == [("state", 3)]+++def test_probe_metric_collector_collect_uses_template_hooks():+ registry = MetricRegistry()+ collector = _DummyProbeCollector()++ collector.collect(registry)++ assert collector.seen == [("probe", True)]+++def test_domain_stats_metric_collector_collect_uses_template_hooks():+ registry = MetricRegistry()+ collector = _DummyDomainCollector()++ collector.collect(registry)++ assert collector.seen == [("domain", 7)]+++def test_state_metric_collector_collect_error_hook_can_handle_read_failures():+ registry = MetricRegistry()+ collector = _DummyFailingStateCollector()++ collector.collect(registry)++ assert collector.error_seen == "boom"diff --git a/tests/metrics/core/test_metric_name_builder.py b/tests/metrics/core/test_metric_name_builder.pynew file mode 100644index 000000000..e43113c6f--- /dev/null+++ b/tests/metrics/core/test_metric_name_builder.py@@ -0,0 +1,27 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from pathlib import Path++from openviking.metrics.core.base import MetricCollector+++def test_metric_collector_metric_name_includes_namespace_and_optional_unit():+ assert (+ MetricCollector.metric_name("cache", "hits", unit="total") == "openviking_cache_hits_total"+ )+ assert (+ MetricCollector.metric_name("resource", "stage_duration", unit="seconds")+ == "openviking_resource_stage_duration_seconds"+ )+ assert MetricCollector.metric_name("lock", "active") == "openviking_lock_active"+++def test_collectors_do_not_embed_openviking_metric_name_literals():+ root = Path(__file__).resolve().parents[2] / "openviking" / "metrics" / "collectors"+ offenders = []+ for path in root.glob("*.py"):+ text = path.read_text(encoding="utf-8")+ if '"openviking_' in text or "'openviking_" in text:+ offenders.append(path.name)+ assert offenders == []diff --git a/tests/metrics/core/test_refresh_control.py b/tests/metrics/core/test_refresh_control.pynew file mode 100644index 000000000..160259af2--- /dev/null+++ b/tests/metrics/core/test_refresh_control.py@@ -0,0 +1,160 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++import asyncio++import pytest++from openviking.metrics.collectors.base import CollectorConfig, Refreshable+from openviking.metrics.collectors.manager import CollectorManager, RefreshResult+from openviking.metrics.core.refresh import RefreshGate+from openviking.metrics.core.registry import MetricRegistry+++def test_refresh_gate_ttl():+ gate = RefreshGate(ttl_seconds=1.0)+ d0 = gate.decide(now=0.0)+ assert d0.do_refresh is True++ assert gate.mark_inflight() is True+ d1 = gate.decide(now=0.1)+ assert d1.do_refresh is False+ assert d1.reason == "inflight"++ gate.mark_complete(success=True, now=0.2)+ d2 = gate.decide(now=0.9)+ assert d2.do_refresh is False+ assert d2.reason == "ttl_valid"++ d3 = gate.decide(now=1.3)+ assert d3.do_refresh is True+ assert d3.reason == "ttl_expired"+++async def test_collector_manager_requires_refreshable():+ class DummyCollector:+ name = "dummy"+ config = CollectorConfig(ttl_seconds=10.0, timeout_seconds=0.01)++ def __init__(self) -> None:+ self.calls = 0++ def collect(self, registry) -> None:+ self.calls += 1+ registry.counter("openviking_dummy_refresh_total").inc()++ mgr = CollectorManager()+ c = DummyCollector()+ assert not isinstance(c, Refreshable)+ with pytest.raises(TypeError):+ mgr.register(c)+++async def test_collector_manager_ttl_and_timeout():+ class DummyCollector(Refreshable):+ name = "dummy"+ config = CollectorConfig(ttl_seconds=10.0, timeout_seconds=0.01)++ def __init__(self) -> None:+ self.calls = 0++ def collect(self, registry) -> None:+ self.calls += 1+ registry.counter("openviking_dummy_refresh_total").inc()++ registry = MetricRegistry()+ mgr = CollectorManager()+ c = DummyCollector()+ mgr.register(c)++ r1 = await mgr.refresh_all(registry, deadline_seconds=1.0)+ assert len(r1) == 1+ assert r1[0].collector == "dummy"+ assert r1[0].attempted is True++ r2 = await mgr.refresh_all(registry, deadline_seconds=1.0)+ assert len(r2) == 1+ assert r2[0].attempted is False+ assert r2[0].reason == "ttl_valid"+++async def test_collector_manager_swr_when_ttl_expired_and_has_last_success():+ class DummyCollector(Refreshable):+ name = "dummy"+ config = CollectorConfig(ttl_seconds=1.0, timeout_seconds=0.5)++ def __init__(self) -> None:+ self.calls = 0++ def collect(self, registry) -> None:+ self.calls += 1+ registry.inc_counter("openviking_dummy_refresh_total")++ registry = MetricRegistry()+ mgr = CollectorManager()+ c = DummyCollector()+ mgr.register(c)++ r1 = await mgr.refresh_all(registry, deadline_seconds=1.0)+ assert r1[0].attempted is True+ assert c.calls == 1++ r2 = await mgr.refresh_all(registry, deadline_seconds=1.0)+ assert r2[0].attempted is False++ await asyncio.sleep(1.1)+ r3 = await mgr.refresh_all(registry, deadline_seconds=1.0)+ assert r3[0].attempted is False+ assert r3[0].reason == "swr_triggered"+++async def test_collector_manager_deadline_exceeded_does_not_stick_inflight(monkeypatch):+ class DummyCollector(Refreshable):+ name = "dummy"+ config = CollectorConfig(ttl_seconds=10.0, timeout_seconds=10.0)++ def collect(self, registry) -> None:+ raise AssertionError("collect should not be called in this test")++ registry = MetricRegistry()+ mgr = CollectorManager()+ mgr.register(DummyCollector())++ started = asyncio.Event()+ release = asyncio.Event()++ async def _fake_run_collector(self, collector, registry, *, timeout: float):+ started.set()+ await release.wait()+ return RefreshResult(+ collector="dummy",+ attempted=True,+ success=True,+ reason="ok",+ )++ monkeypatch.setattr(CollectorManager, "_run_collector", _fake_run_collector)++ refresh_task = asyncio.create_task(mgr.refresh_all(registry, deadline_seconds=0.01))+ await asyncio.wait_for(started.wait(), timeout=0.5)+ r1 = await refresh_task+ assert r1+ assert any(item.collector == "dummy" and item.reason == "deadline_exceeded" for item in r1)++ # While the long refresh is still running, we should see inflight but it must not be permanent.+ r2 = await mgr.refresh_all(registry, deadline_seconds=1.0)+ assert r2 and r2[0].collector == "dummy"+ assert r2[0].attempted is False+ assert r2[0].reason == "inflight"++ # Let the background refresh finish and confirm the next scrape recovers to ttl_valid.+ release.set()+ await asyncio.wait_for(asyncio.sleep(0), timeout=0.5)+ r3 = await mgr.refresh_all(registry, deadline_seconds=1.0)+ assert r3 and r3[0].collector == "dummy"+ assert r3[0].attempted is False+ assert r3[0].reason == "ttl_valid"diff --git a/tests/metrics/core/test_registry_and_exporter.py b/tests/metrics/core/test_registry_and_exporter.pynew file mode 100644index 000000000..68284ee8f--- /dev/null+++ b/tests/metrics/core/test_registry_and_exporter.py@@ -0,0 +1,80 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++import pytest++from openviking.metrics.core.registry import MetricRegistry+++def test_registry_rejects_type_conflict():+ registry = MetricRegistry()+ registry.counter("openviking_conflict_total")+ with pytest.raises(ValueError):+ registry.gauge("openviking_conflict_total")+++def test_registry_label_keys_must_match_definition():+ registry = MetricRegistry()+ c = registry.counter("openviking_labeled_total", label_names=("a", "b"))+ with pytest.raises(ValueError):+ c.inc(labels={"a": "1"})+ with pytest.raises(ValueError):+ c.inc(labels={"a": "1", "b": "2", "c": "3"})+++def test_registry_canonicalizes_label_name_order_for_same_family(render_prometheus):+ registry = MetricRegistry()++ registry.inc_counter(+ "openviking_ordered_total",+ labels={"status": "ok", "provider": "local"},+ label_names=("status", "provider"),+ )+ registry.inc_counter(+ "openviking_ordered_total",+ labels={"provider": "local", "status": "ok"},+ label_names=("provider", "status"),+ )++ text = render_prometheus(registry)+ assert 'openviking_ordered_total{provider="local",status="ok"} 2' in text+++def test_counter_only_increases():+ registry = MetricRegistry()+ c = registry.counter("openviking_counter_total")+ c.inc()+ with pytest.raises(ValueError):+ c.inc(amount=0)+ with pytest.raises(ValueError):+ c.inc(amount=-1)+++def test_gauge_inc_dec(registry, render_prometheus):+ g = registry.gauge("openviking_gauge")+ g.set(1)+ g.inc()+ g.dec(0.5)+ text = render_prometheus(registry)+ assert "openviking_gauge 1.5" in text+++def test_histogram_boundary_bucket(registry, render_prometheus):+ h = registry.histogram("openviking_latency_seconds", buckets=(0.05, 0.1))+ h.observe(0.05)+ text = render_prometheus(registry)+ assert 'openviking_latency_seconds_bucket{le="0.05"} 1' in text+ assert 'openviking_latency_seconds_bucket{le="0.1"} 1' in text+ assert 'openviking_latency_seconds_bucket{le="+Inf"} 1' in text+++def test_exporter_outputs_help_type_and_empty_metrics(registry, render_prometheus):+ registry.counter("openviking_empty_counter_total")+ registry.gauge("openviking_empty_gauge")+ registry.histogram("openviking_empty_histogram_seconds")+ text = render_prometheus(registry)+ assert "# TYPE openviking_empty_counter_total counter" in text+ assert "openviking_empty_counter_total 0" in text+ assert "# TYPE openviking_empty_histogram_seconds histogram" in text+ assert "openviking_empty_histogram_seconds_count 0" in text+ assert 'openviking_empty_histogram_seconds_bucket{le="+Inf"} 0' in textdiff --git a/tests/metrics/datasources/test_abstraction_reuse.py b/tests/metrics/datasources/test_abstraction_reuse.pynew file mode 100644index 000000000..8adb9bd59--- /dev/null+++ b/tests/metrics/datasources/test_abstraction_reuse.py@@ -0,0 +1,43 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from pathlib import Path+++def test_metric_datasource_owns_safe_read_helpers_and_datasource_subclasses_reuse_them():+ # This test reads source files by absolute path; keep it stable across test subdirectories.+ project_root = Path(__file__).resolve().parents[3]++ core_base = (project_root / "openviking" / "metrics" / "core" / "base.py").read_text(+ encoding="utf-8"+ )+ datasource_base = (+ project_root / "openviking" / "metrics" / "datasources" / "base.py"+ ).read_text(encoding="utf-8")+ probes = (project_root / "openviking" / "metrics" / "datasources" / "probes.py").read_text(+ encoding="utf-8"+ )+ encryption = (+ project_root / "openviking" / "metrics" / "datasources" / "encryption.py"+ ).read_text(encoding="utf-8")+ model_usage = (+ project_root / "openviking" / "metrics" / "datasources" / "model_usage.py"+ ).read_text(encoding="utf-8")+ observer_state = (+ project_root / "openviking" / "metrics" / "datasources" / "observer_state.py"+ ).read_text(encoding="utf-8")++ assert "def safe_read(" in core_base+ assert "def safe_read_async(" in core_base+ assert "def best_effort(" not in datasource_base+ assert "def best_effort_async(" not in datasource_base++ assert "safe_value_probe(" in datasource_base+ assert probes.count("safe_value_probe(") >= 1+ assert encryption.count("safe_value_probe(") >= 1++ assert '"available"' in model_usage+ assert '"usage_by_model"' in model_usage+ assert ".as_dict(" in model_usage+ assert ".normalize_str(" in model_usage+ assert ".as_dict(" in observer_state or ".normalize_str(" in observer_statediff --git a/tests/metrics/datasources/test_encryption.py b/tests/metrics/datasources/test_encryption.pynew file mode 100644index 000000000..439078d2c--- /dev/null+++ b/tests/metrics/datasources/test_encryption.py@@ -0,0 +1,70 @@+"""Module-level tests for `openviking.metrics.datasources.encryption`."""++from __future__ import annotations++from types import SimpleNamespace++import openviking.metrics.datasources.encryption as enc+++def test_encryption_event_datasource_emits_operation_event(patch_event_emit):+ """EncryptionEventDataSource must emit a normalized `encryption.operation` event."""++ enc.EncryptionEventDataSource.record_operation(+ operation="encrypt",+ status="ok",+ duration_seconds=0.25,+ )++ assert (+ "encryption.operation",+ {"operation": "encrypt", "status": "ok", "duration_seconds": 0.25},+ ) in patch_event_emit+++def test_encryption_event_datasource_ignores_non_positive_bytes(monkeypatch):+ """EncryptionEventDataSource must not emit `encryption.bytes` when size is non-positive."""++ calls: list[tuple[str, dict]] = []++ def _emit(event_name: str, payload: dict) -> None:+ calls.append((str(event_name), dict(payload)))+ raise RuntimeError("should not be called for non-positive bytes")++ monkeypatch.setattr(enc.EventMetricDataSource, "_emit", staticmethod(_emit))+ enc.EncryptionEventDataSource.record_bytes(operation="encrypt", size_bytes=0)+ enc.EncryptionEventDataSource.record_bytes(operation="encrypt", size_bytes=-1)+ assert calls == []+++def test_encryption_probe_datasource_ok_and_provider(monkeypatch):+ """EncryptionProbeDataSource returns `(True, provider)` on successful bootstrap."""++ class _Cfg:+ encryption = SimpleNamespace(provider="volcengine")++ def _bootstrap():+ return None++ monkeypatch.setattr("openviking.crypto.config.bootstrap_encryption", _bootstrap)+ ds = enc.EncryptionProbeDataSource(config_provider=lambda: _Cfg())+ env = ds.read_probe_state()+ assert env.ok is True+ assert env.value == (True, "volcengine")+++def test_encryption_probe_datasource_returns_default_on_exception(monkeypatch):+ """EncryptionProbeDataSource must return `(False, provider)` with `ok=False` on failures."""++ class _Cfg:+ encryption = SimpleNamespace(provider="volcengine")++ def _boom():+ raise RuntimeError("bootstrap failed")++ monkeypatch.setattr("openviking.crypto.config.bootstrap_encryption", _boom)+ ds = enc.EncryptionProbeDataSource(config_provider=lambda: _Cfg())+ env = ds.read_probe_state()+ assert env.ok is False+ assert env.value == (False, "volcengine")+ assert env.error_type == "RuntimeError"diff --git a/tests/metrics/datasources/test_envelope.py b/tests/metrics/datasources/test_envelope.pynew file mode 100644index 000000000..7df4ef703--- /dev/null+++ b/tests/metrics/datasources/test_envelope.py@@ -0,0 +1,65 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from openviking.metrics.collectors.base import (+ CollectorConfig,+ ProbeMetricCollector,+ StateMetricCollector,+)+from openviking.metrics.core.base import ReadEnvelope+from openviking.metrics.core.registry import MetricRegistry+++class _EnvelopeStateCollector(StateMetricCollector):+ config = CollectorConfig()+ STALE_ON_ERROR = True++ def __init__(self) -> None:+ self.error_seen = False+ self.value_seen = None++ def read_metric_input(self):+ return ReadEnvelope(ok=False, value=("fallback",))++ def collect_hook(self, registry, metric_input) -> None:+ self.value_seen = metric_input++ def collect_stale_hook(self, registry, error: Exception) -> None:+ self.error_seen = True+++class _EnvelopeProbeCollector(ProbeMetricCollector):+ config = CollectorConfig()++ def __init__(self) -> None:+ self.error_seen = False+ self.value_seen = None++ def read_metric_input(self):+ return ReadEnvelope(ok=False, value={"probe": True})++ def collect_hook(self, registry, metric_input) -> None:+ self.value_seen = metric_input++ def collect_stale_hook(self, registry, error: Exception) -> None:+ self.error_seen = True+++def test_state_collector_unwraps_envelope_and_routes_ok_false_to_stale_hook():+ registry = MetricRegistry()+ collector = _EnvelopeStateCollector()++ collector.collect(registry)++ assert collector.error_seen is True+ assert collector.value_seen is None+++def test_probe_collector_unwraps_envelope_and_routes_ok_false_to_stale_hook():+ registry = MetricRegistry()+ collector = _EnvelopeProbeCollector()++ collector.collect(registry)++ assert collector.error_seen is True+ assert collector.value_seen is Nonediff --git a/tests/metrics/datasources/test_event_emit.py b/tests/metrics/datasources/test_event_emit.pynew file mode 100644index 000000000..4a63308ee--- /dev/null+++ b/tests/metrics/datasources/test_event_emit.py@@ -0,0 +1,25 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from openviking.metrics.datasources.base import EventMetricDataSource+from openviking.metrics.datasources.cache import CacheEventDataSource+from openviking.metrics.datasources.http import HttpRequestLifecycleDataSource+++def test_event_datasources_use_shared_event_metric_datasource_emit(monkeypatch):+ calls: list[tuple[str, dict]] = []++ def _fake_emit(event_name: str, payload: dict) -> None:+ calls.append((str(event_name), dict(payload)))++ monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)++ CacheEventDataSource.record_hit("L0")+ CacheEventDataSource.record_miss("L1")+ HttpRequestLifecycleDataSource.record_request(+ method="GET", route="/demo", status="200", duration_seconds=0.01+ )++ assert calls[0][0] == "cache.hit"+ assert calls[1][0] == "cache.miss"+ assert calls[2][0] == "http.request"diff --git a/tests/metrics/datasources/test_probes.py b/tests/metrics/datasources/test_probes.pynew file mode 100644index 000000000..0127e5a34--- /dev/null+++ b/tests/metrics/datasources/test_probes.py@@ -0,0 +1,153 @@+"""Module-level tests for `openviking.metrics.datasources.probes`."""++from __future__ import annotations++from types import SimpleNamespace++import openviking.metrics.datasources.probes as probes+++def test_service_probe_datasource_reads_service_and_app_state_success():+ """ServiceProbeDataSource returns readiness booleans derived from service/app state."""++ app = SimpleNamespace(state=SimpleNamespace(api_key_manager=object()))+ service = SimpleNamespace(initialized=True)+ ds = probes.ServiceProbeDataSource(app=app, service=service)++ env = ds.read_probe_state()+ assert env.ok is True+ assert env.value == {"service_readiness": True, "api_key_manager_readiness": True}+ assert env.error_type is None+++def test_service_probe_datasource_returns_default_on_exception():+ """ServiceProbeDataSource must wrap exceptions into `ok=False` with deterministic default."""++ class _BadApp:+ @property+ def state(self):+ raise RuntimeError("boom")++ ds = probes.ServiceProbeDataSource(app=_BadApp(), service=SimpleNamespace(initialized=True))++ env = ds.read_probe_state()+ assert env.ok is False+ assert env.value == {"service_readiness": False, "api_key_manager_readiness": False}+ assert env.error_type == "RuntimeError"+ assert "boom" in (env.error_message or "")+++def test_storage_probe_datasource_returns_agfs_probe_success(monkeypatch):+ """StorageProbeDataSource exposes a single `agfs` boolean probe on success."""++ monkeypatch.setattr(probes, "get_viking_fs", lambda: SimpleNamespace(agfs=object()))+ ds = probes.StorageProbeDataSource()++ env = ds.read_probe_state()+ assert env.ok is True+ assert env.value == {"agfs": True}+++def test_storage_probe_datasource_returns_default_on_exception(monkeypatch):+ """StorageProbeDataSource must return `ok=False` with `{agfs: False}` on read failure."""++ def _boom():+ raise RuntimeError("no fs")++ monkeypatch.setattr(probes, "get_viking_fs", _boom)+ ds = probes.StorageProbeDataSource()++ env = ds.read_probe_state()+ assert env.ok is False+ assert env.value == {"agfs": False}+ assert env.error_type == "RuntimeError"+++def test_retrieval_backend_probe_datasource_returns_false_when_no_service():+ """RetrievalBackendProbeDataSource treats missing wiring as a negative readiness (ok=True)."""++ ds = probes.RetrievalBackendProbeDataSource(service=None)+ env = ds.read_probe_state()+ assert env.ok is True+ assert env.value == {"vikingdb": False}+++def test_retrieval_backend_probe_datasource_returns_default_on_exception(monkeypatch):+ """RetrievalBackendProbeDataSource must return `ok=False` on health-check runner failures."""++ class _VikingDB:+ def health_check(self):+ return object()++ service = SimpleNamespace(vikingdb=_VikingDB())+ ds = probes.RetrievalBackendProbeDataSource(service=service)++ def _boom(_coro):+ raise RuntimeError("runner failed")++ monkeypatch.setattr(probes, "run_async", _boom)+ env = ds.read_probe_state()+ assert env.ok is False+ assert env.value == {"vikingdb": False}+ assert env.error_type == "RuntimeError"+++def test_model_provider_probe_datasource_returns_provider_tuple_success():+ """ModelProviderProbeDataSource must return provider name and a boolean ok flag."""++ class _VlmCfg:+ provider = "volcengine"++ def get_vlm_instance(self):+ return object()++ cfg = SimpleNamespace(vlm=_VlmCfg())+ ds = probes.ModelProviderProbeDataSource(config_provider=lambda: cfg)++ env = ds.read_probe_state()+ assert env.ok is True+ assert env.value == {"provider": ("volcengine", True)}+++def test_model_provider_probe_datasource_returns_default_on_exception():+ """ModelProviderProbeDataSource must fall back to `unknown/False` on config failures."""++ class _BadVlmCfg:+ provider = "volcengine"++ def get_vlm_instance(self):+ raise RuntimeError("bad cfg")++ cfg = SimpleNamespace(vlm=_BadVlmCfg())+ ds = probes.ModelProviderProbeDataSource(config_provider=lambda: cfg)++ env = ds.read_probe_state()+ assert env.ok is False+ assert env.value == {"provider": ("unknown", False)}+ assert env.error_type == "RuntimeError"+++def test_async_system_probe_datasource_returns_queue_probe_success(monkeypatch):+ """AsyncSystemProbeDataSource returns a boolean `queue` probe based on queue manager wiring."""++ monkeypatch.setattr(probes, "get_queue_manager", lambda: object())+ ds = probes.AsyncSystemProbeDataSource()++ env = ds.read_probe_state()+ assert env.ok is True+ assert env.value == {"queue": True}+++def test_async_system_probe_datasource_returns_default_on_exception(monkeypatch):+ """AsyncSystemProbeDataSource must return `ok=False` with `{queue: False}` on failures."""++ def _boom():+ raise RuntimeError("no queue")++ monkeypatch.setattr(probes, "get_queue_manager", _boom)+ ds = probes.AsyncSystemProbeDataSource()++ env = ds.read_probe_state()+ assert env.ok is False+ assert env.value == {"queue": False}+ assert env.error_type == "RuntimeError"diff --git a/tests/metrics/datasources/test_retrieval.py b/tests/metrics/datasources/test_retrieval.pynew file mode 100644index 000000000..a74b97c9e--- /dev/null+++ b/tests/metrics/datasources/test_retrieval.py@@ -0,0 +1,43 @@+"""Module-level tests for `openviking.metrics.datasources.retrieval`."""++from __future__ import annotations++import openviking.metrics.datasources.retrieval as retrieval+++def test_retrieval_stats_datasource_emits_completed_event(patch_event_emit):+ """RetrievalStatsDataSource must emit a normalized `retrieval.completed` event payload."""++ retrieval.RetrievalStatsDataSource.record_retrieval(+ context_type="context_1",+ result_count=3,+ latency_seconds=0.12,+ rerank_used=True,+ rerank_fallback=False,+ )++ assert (+ "retrieval.completed",+ {+ "context_type": "context_1",+ "result_count": 3,+ "latency_seconds": 0.12,+ "rerank_used": True,+ "rerank_fallback": False,+ },+ ) in patch_event_emit+++def test_retrieval_stats_datasource_normalizes_unknown_context_type(patch_event_emit):+ """Empty/falsey context types are normalized to `unknown`."""++ retrieval.RetrievalStatsDataSource.record_retrieval(+ context_type="",+ result_count=0,+ latency_seconds=0.0,+ )++ assert any(+ event_name == "retrieval.completed" and payload.get("context_type") == "unknown"+ for event_name, payload in patch_event_emit+ )diff --git a/tests/metrics/datasources/test_session.py b/tests/metrics/datasources/test_session.pynew file mode 100644index 000000000..ac7830981--- /dev/null+++ b/tests/metrics/datasources/test_session.py@@ -0,0 +1,34 @@+"""Module-level tests for `openviking.metrics.datasources.session`."""++from __future__ import annotations++import openviking.metrics.datasources.session as session+++def test_session_lifecycle_datasource_emits_lifecycle_event(patch_event_emit):+ """SessionLifecycleDataSource must emit `session.lifecycle` with bounded labels."""++ session.SessionLifecycleDataSource.record_lifecycle(action="create", status="ok")+ assert ("session.lifecycle", {"action": "create", "status": "ok"}) in patch_event_emit+++def test_session_lifecycle_datasource_ignores_non_positive_context_deltas(monkeypatch):+ """SessionLifecycleDataSource must not emit when delta is non-positive."""++ calls: list[tuple[str, dict]] = []++ def _emit(event_name: str, payload: dict) -> None:+ calls.append((str(event_name), dict(payload)))+ raise RuntimeError("should not be called for non-positive deltas")++ monkeypatch.setattr(session.EventMetricDataSource, "_emit", staticmethod(_emit))+ session.SessionLifecycleDataSource.record_contexts_used(action="create", delta=0)+ session.SessionLifecycleDataSource.record_contexts_used(action="create", delta=-1)+ assert calls == []+++def test_session_lifecycle_datasource_emits_archive_event(patch_event_emit):+ """SessionLifecycleDataSource must emit `session.archive` events."""++ session.SessionLifecycleDataSource.record_archive(status="ok")+ assert ("session.archive", {"status": "ok"}) in patch_event_emitdiff --git a/tests/metrics/wiring/test_bootstrap.py b/tests/metrics/wiring/test_bootstrap.pynew file mode 100644index 000000000..63793c82b--- /dev/null+++ b/tests/metrics/wiring/test_bootstrap.py@@ -0,0 +1,44 @@+"""Module-level tests for `openviking.metrics.bootstrap`."""++from __future__ import annotations++import pytest++import openviking.metrics.bootstrap as bootstrap+++def test_create_default_collector_manager_registers_expected_collectors_in_order():+ """Default bootstrap must register a stable collector sequence for `/metrics` refresh."""+ manager = bootstrap.create_default_collector_manager(app=None, service=None)++ names = [type(c).__name__ for c in manager._collectors] # noqa: SLF001 (intentional white-box)+ assert names == [+ "QueueCollector",+ "TaskTrackerCollector",+ "ObserverHealthCollector",+ "ObserverStateCollector",+ "LockCollector",+ "VikingDBCollector",+ "ModelUsageCollector",+ "ServiceProbeCollector",+ "StorageProbeCollector",+ "RetrievalBackendProbeCollector",+ "ModelProviderProbeCollector",+ "AsyncSystemProbeCollector",+ "EncryptionProbeCollector",+ ]+++def test_create_default_collector_manager_propagates_construction_failures(monkeypatch):+ """+ If a required datasource fails to construct, bootstrap should fail fast.++ This protects the server from silently starting with a partially wired metrics stack.+ """++ def _boom():+ raise RuntimeError("cannot init datasource")++ monkeypatch.setattr(bootstrap, "QueuePipelineStateDataSource", _boom)+ with pytest.raises(RuntimeError, match="cannot init datasource"):+ bootstrap.create_default_collector_manager(app=None, service=None)diff --git a/tests/metrics/wiring/test_optional_instrumentation_call_sites.py b/tests/metrics/wiring/test_optional_instrumentation_call_sites.pynew file mode 100644index 000000000..fd3eb74e6--- /dev/null+++ b/tests/metrics/wiring/test_optional_instrumentation_call_sites.py@@ -0,0 +1,20 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from pathlib import Path+++def test_optional_cache_datasource_instrumentation_is_wired_into_key_call_sites():+ # This test reads source files by absolute path; keep it stable across test subdirectories.+ project_root = Path(__file__).resolve().parents[3]+ targets = [+ project_root / "openviking" / "storage" / "queuefs" / "semantic_dag.py",+ project_root / "openviking" / "storage" / "queuefs" / "semantic_processor.py",+ project_root / "openviking" / "session" / "memory" / "extract_loop.py",+ ]+ missing = []+ for path in targets:+ text = path.read_text(encoding="utf-8")+ if "CacheEventDataSource.record_" not in text:+ missing.append(path.name)+ assert missing == []diff --git a/tests/metrics/wiring/test_resource_ingestion_datasource_wiring.py b/tests/metrics/wiring/test_resource_ingestion_datasource_wiring.pynew file mode 100644index 000000000..86a1d5727--- /dev/null+++ b/tests/metrics/wiring/test_resource_ingestion_datasource_wiring.py@@ -0,0 +1,75 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from openviking.metrics.account_dimension import (+ configure_metric_account_dimension,+ reset_metric_account_dimension,+)+from openviking.metrics.collectors.resource import ResourceIngestionCollector+from openviking.metrics.core.registry import MetricRegistry+from openviking.metrics.datasources.base import EventMetricDataSource+from openviking.metrics.datasources.resource import ResourceIngestionEventDataSource+from openviking.metrics.exporters.prometheus import PrometheusExporter+++def test_resource_ingestion_event_datasource_can_drive_resource_ingestion_collector(monkeypatch):+ registry = MetricRegistry()+ collector = ResourceIngestionCollector()++ configure_metric_account_dimension(+ enabled=True,+ metric_allowlist={+ ResourceIngestionCollector.STAGE_TOTAL,+ ResourceIngestionCollector.STAGE_DURATION_SECONDS,+ ResourceIngestionCollector.WAIT_DURATION_SECONDS,+ },+ max_active_accounts=10,+ )++ def _emit(event_name: str, payload: dict) -> None:+ collector.receive(event_name, payload, registry)++ monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_emit), raising=False)++ ResourceIngestionEventDataSource.record_stage(stage="parse", status="ok", duration_seconds=0.01)+ ResourceIngestionEventDataSource.record_stage(+ stage="parse", status="error", duration_seconds=0.02+ )+ ResourceIngestionEventDataSource.record_stage(+ stage="parse",+ status="ok",+ duration_seconds=0.03,+ account_id="acct-resource",+ )+ ResourceIngestionEventDataSource.record_wait(+ operation="queue_processing", duration_seconds=0.03+ )+ ResourceIngestionEventDataSource.record_wait(+ operation="queue_processing",+ duration_seconds=0.04,+ account_id="acct-resource",+ )++ text = PrometheusExporter(registry=registry).render()+ assert (+ 'openviking_resource_stage_total{account_id="__unknown__",stage="parse",status="ok"} 1'+ in text+ )+ assert (+ 'openviking_resource_stage_total{account_id="__unknown__",stage="parse",status="error"} 1'+ in text+ )+ assert (+ 'openviking_resource_wait_duration_seconds_count{account_id="__unknown__",operation="queue_processing"} 1'+ in text+ )+ assert (+ 'openviking_resource_stage_total{account_id="acct-resource",stage="parse",status="ok"} 1'+ in text+ )+ assert (+ 'openviking_resource_wait_duration_seconds_count{account_id="acct-resource",operation="queue_processing"} 1'+ in text+ )++ reset_metric_account_dimension()diff --git a/tests/metrics/wiring/test_telemetry_bridge.py b/tests/metrics/wiring/test_telemetry_bridge.pynew file mode 100644index 000000000..c27a7ce09--- /dev/null+++ b/tests/metrics/wiring/test_telemetry_bridge.py@@ -0,0 +1,95 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from openviking.metrics.datasources.telemetry_bridge import TelemetryBridgeEventDataSource+from openviking.metrics.global_api import init_metrics_from_server_config, shutdown_metrics+from openviking.server.config import PrometheusConfig, ServerConfig, TelemetryConfig+++def test_telemetry_bridge_records_operation_and_resource_metrics(registry, render_prometheus):+ init_metrics_from_server_config(+ ServerConfig(telemetry=TelemetryConfig(prometheus=PrometheusConfig(enabled=True))),+ app=None,+ registry=registry,+ )+ try:+ TelemetryBridgeEventDataSource.record_summary(+ {+ "operation": "resource.process",+ "status": "ok",+ "duration_ms": 1200,+ "tokens": {"total": 10, "llm": {"input": 3, "output": 7}},+ "vector": {"searches": 1, "scored": 2, "passed": 2, "returned": 1, "scanned": 9},+ "memory": {"extracted": 4},+ "semantic_nodes": {"OK": 12},+ "resource": {+ "process": {+ "parse": {"duration_ms": 10, "warnings_count": 1},+ "finalize": {"duration_ms": 20},+ "summarize": {"duration_ms": 30},+ "duration_ms": 80,+ },+ "wait": {"duration_ms": 40},+ "watch": {"duration_ms": 50},+ },+ }+ )+ text = render_prometheus(registry)+ assert (+ 'openviking_operation_requests_total{account_id="__unknown__",operation="resource.process",status="ok"} 1'+ in text+ )+ assert (+ 'openviking_operation_duration_seconds_count{account_id="__unknown__",operation="resource.process",status="ok"} 1'+ in text+ )+ assert 'openviking_vector_searches_total{operation="resource.process"} 1' in text+ assert 'openviking_memory_extracted_total{operation="resource.process"} 4' in text+ assert (+ 'openviking_semantic_nodes_total{status="OK"} 12' in text+ or 'openviking_semantic_nodes_total{status="OK"} 12.0' in text+ )+ assert (+ 'openviking_resource_stage_total{account_id="__unknown__",stage="parse",status="warning"} 1'+ in text+ )+ assert (+ 'openviking_resource_wait_duration_seconds_count{account_id="__unknown__",operation="resource.process"} 1'+ in text+ )+ finally:+ shutdown_metrics(app=None)+++def test_telemetry_bridge_semantic_nodes_total_is_cumulative(registry, render_prometheus):+ init_metrics_from_server_config(+ ServerConfig(telemetry=TelemetryConfig(prometheus=PrometheusConfig(enabled=True))),+ app=None,+ registry=registry,+ )+ try:+ TelemetryBridgeEventDataSource.record_summary(+ {+ "operation": "resource.process",+ "status": "ok",+ "duration_ms": 1,+ "tokens": {"total": 1, "llm": {"input": 1, "output": 0}},+ "semantic_nodes": {"OK": 12},+ }+ )+ TelemetryBridgeEventDataSource.record_summary(+ {+ "operation": "resource.process",+ "status": "ok",+ "duration_ms": 1,+ "tokens": {"total": 1, "llm": {"input": 1, "output": 0}},+ "semantic_nodes": {"OK": 12},+ }+ )+ text = render_prometheus(registry)+ assert (+ 'openviking_semantic_nodes_total{status="OK"} 24' in text+ or 'openviking_semantic_nodes_total{status="OK"} 24.0' in text+ )+ finally:+ shutdown_metrics(app=None)diff --git a/tests/server/test_bot_proxy_auth.py b/tests/server/test_bot_proxy_auth.pyindex bf5b580f0..291326c5b 100644--- a/tests/server/test_bot_proxy_auth.py+++ b/tests/server/test_bot_proxy_auth.py@@ -3,10 +3,8 @@"""Regression tests for bot proxy endpoint auth enforcement."""-import httpximport pytest-import pytest_asyncio-from fastapi import FastAPI, Request+from fastapi import Requestimport openviking.server.routers.bot as bot_router_modulediff --git a/tests/server/test_prometheus_metrics.py b/tests/server/test_prometheus_metrics.pyindex 2568cfe51..1d82d08a0 100644--- a/tests/server/test_prometheus_metrics.py+++ b/tests/server/test_prometheus_metrics.py@@ -1,97 +1,81 @@# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.# SPDX-License-Identifier: AGPL-3.0-"""Tests for the Prometheus metrics exporter."""+"""Tests for the Prometheus metrics endpoint and exposition output."""import httpximport pytest+from openviking.metrics.core.registry import MetricRegistry+from openviking.metrics.exporters.prometheus import PrometheusExporter+from openviking.metrics.global_api import (+ get_metrics_registry,+ init_metrics_from_server_config,+ shutdown_metrics,+)from openviking.server.app import create_app-from openviking.server.config import PrometheusConfig, ServerConfig, TelemetryConfig-from openviking.storage.observers.prometheus_observer import (- PrometheusObserver,- get_prometheus_observer,- set_prometheus_observer,+from openviking.server.config import (+ MetricsAccountDimensionConfig,+ MetricsConfig,+ PrometheusConfig,+ ServerConfig,+ TelemetryConfig,)-class TestPrometheusObserver:- """Unit tests for PrometheusObserver recording and rendering."""-- def test_record_retrieval(self):- obs = PrometheusObserver()- obs.record_retrieval(0.05)- obs.record_retrieval(0.12)- text = obs.render_metrics()- assert "openviking_retrieval_requests_total 2" in text- assert "openviking_retrieval_latency_seconds_count 2" in text-- def test_record_embedding(self):- obs = PrometheusObserver()- obs.record_embedding(0.3)- text = obs.render_metrics()- assert "openviking_embedding_requests_total 1" in text- assert "openviking_embedding_latency_seconds_count 1" in text-- def test_record_vlm_call(self):- obs = PrometheusObserver()- obs.record_vlm_call(1.5)- text = obs.render_metrics()- assert "openviking_vlm_calls_total 1" in text- assert "openviking_vlm_call_duration_seconds_count 1" in text-- def test_cache_hit_miss(self):- obs = PrometheusObserver()- obs.record_cache_hit("L0")- obs.record_cache_hit("L0")- obs.record_cache_miss("L1")- text = obs.render_metrics()+class TestPrometheusExposition:+ def test_counter_histogram_and_labels(self):+ registry = MetricRegistry()+ labels = {"context_type": "memory"}+ registry.counter(+ "openviking_retrieval_requests_total",+ label_names=("context_type",),+ ).inc(labels=labels)+ registry.counter(+ "openviking_retrieval_requests_total",+ label_names=("context_type",),+ ).inc(labels=labels)+ registry.histogram(+ "openviking_retrieval_latency_seconds",+ label_names=("context_type",),+ ).observe(0.02, labels=labels)++ registry.counter("openviking_cache_hits_total", label_names=("level",)).inc(+ labels={"level": "L0"}+ )+ registry.counter("openviking_cache_hits_total", label_names=("level",)).inc(+ labels={"level": "L0"}+ )+ registry.counter("openviking_cache_misses_total", label_names=("level",)).inc(+ labels={"level": "L1"}+ )++ exporter = PrometheusExporter(registry=registry)+ text = exporter.render()++ assert 'openviking_retrieval_requests_total{context_type="memory"} 2' in text+ assert 'openviking_retrieval_latency_seconds_count{context_type="memory"} 1' in text+ assert (+ 'openviking_retrieval_latency_seconds_bucket{context_type="memory",le="0.05"} 1' in text+ )+ assert (+ 'openviking_retrieval_latency_seconds_bucket{context_type="memory",le="+Inf"} 1' in text+ )assert 'openviking_cache_hits_total{level="L0"} 2' in textassert 'openviking_cache_misses_total{level="L1"} 1' in text- def test_render_empty_metrics(self):- obs = PrometheusObserver()- text = obs.render_metrics()- assert "openviking_retrieval_requests_total 0" in text- assert "openviking_embedding_requests_total 0" in text- assert "openviking_vlm_calls_total 0" in text-- def test_histogram_buckets(self):- obs = PrometheusObserver()- obs.record_retrieval(0.02)- text = obs.render_metrics()- assert 'openviking_retrieval_latency_seconds_bucket{le="0.05"} 1' in text- assert 'openviking_retrieval_latency_seconds_bucket{le="+Inf"} 1' in text---class TestPrometheusObserverSingleton:- """Tests for the module-level singleton accessor."""-- def test_default_is_none(self):- set_prometheus_observer(None)- assert get_prometheus_observer() is None-- def test_set_and_get(self):- obs = PrometheusObserver()- set_prometheus_observer(obs)- assert get_prometheus_observer() is obs- set_prometheus_observer(None)-- def test_clear_singleton(self):- obs = PrometheusObserver()- set_prometheus_observer(obs)- set_prometheus_observer(None)- assert get_prometheus_observer() is None--class TestRetrievalStatsPrometheusIntegration:- """Test that RetrievalStatsCollector notifies the PrometheusObserver."""-- def test_record_query_notifies_prometheus(self):+class TestRetrievalStatsMetricsIntegration:+ def test_record_query_updates_metrics_registry(self):from openviking.retrieve.retrieval_stats import RetrievalStatsCollector- obs = PrometheusObserver()- set_prometheus_observer(obs)+ registry = MetricRegistry()+ shutdown_metrics(app=None)+ init_metrics_from_server_config(+ ServerConfig(telemetry=TelemetryConfig(prometheus=PrometheusConfig(enabled=True))),+ app=None,+ registry=registry,+ )try:collector = RetrievalStatsCollector()collector.record_query(@@ -100,11 +84,22 @@ class TestRetrievalStatsPrometheusIntegration:scores=[0.8, 0.7, 0.6],latency_ms=42.5,)- text = obs.render_metrics()- assert "openviking_retrieval_requests_total 1" in text- assert "openviking_retrieval_latency_seconds_count 1" in text+ exporter = PrometheusExporter(registry=get_metrics_registry())+ text = exporter.render()+ assert (+ 'openviking_retrieval_requests_total{account_id="__unknown__",context_type="memory"} 1'+ in text+ )+ assert (+ 'openviking_retrieval_results_total{account_id="__unknown__",context_type="memory"} 3'+ in text+ )+ assert (+ 'openviking_retrieval_latency_seconds_count{account_id="__unknown__",context_type="memory"} 1'+ in text+ )finally:- set_prometheus_observer(None)+ shutdown_metrics(app=None)@pytest.mark.asyncio@@ -122,11 +117,61 @@ class TestMetricsEndpoint:async def test_metrics_enabled_returns_200(self):config = ServerConfig(telemetry=TelemetryConfig(prometheus=PrometheusConfig(enabled=True)))app = create_app(config=config, service=None)- # Simulate lifespan setting the observer on app.state- obs = PrometheusObserver()- app.state.prometheus_observer = obs+ init_metrics_from_server_config(config, app=app)transport = httpx.ASGITransport(app=app)- async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:- resp = await client.get("/metrics")- assert resp.status_code == 200- assert "openviking_retrieval_requests_total" in resp.text+ try:+ registry = get_metrics_registry()+ registry.counter(+ "openviking_retrieval_requests_total",+ label_names=("context_type",),+ ).inc(labels={"context_type": "memory"})+ async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:+ resp = await client.get("/metrics")+ assert resp.status_code == 200+ assert "openviking_retrieval_requests_total" in resp.text+ finally:+ shutdown_metrics(app=app)++ async def test_metrics_enabled_by_new_server_metrics_flag(self):+ config = ServerConfig(metrics=MetricsConfig(enabled=True))+ app = create_app(config=config, service=None)+ init_metrics_from_server_config(config, app=app)+ transport = httpx.ASGITransport(app=app)+ try:+ async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:+ resp = await client.get("/metrics")+ assert resp.status_code == 200+ finally:+ shutdown_metrics(app=app)++ async def test_metrics_enabled_by_legacy_or_new_flag(self):+ config = ServerConfig(+ telemetry=TelemetryConfig(prometheus=PrometheusConfig(enabled=False)),+ metrics=MetricsConfig(enabled=True),+ )+ app = create_app(config=config, service=None)+ init_metrics_from_server_config(config, app=app)+ assert getattr(app.state, "metrics_exporter", None) is not None+ shutdown_metrics(app=app)++ async def test_metrics_account_dimension_config_is_loaded(self):+ config = ServerConfig(+ metrics=MetricsConfig(+ enabled=True,+ account_dimension=MetricsAccountDimensionConfig(+ enabled=True,+ max_active_accounts=3,+ metric_allowlist=["openviking_http_requests_total"],+ ),+ )+ )+ app = create_app(config=config, service=None)+ init_metrics_from_server_config(config, app=app)+ try:+ assert config.metrics.account_dimension.enabled is True+ assert config.metrics.account_dimension.max_active_accounts == 3+ assert config.metrics.account_dimension.metric_allowlist == [+ "openviking_http_requests_total"+ ]+ finally:+ shutdown_metrics(app=app)diff --git a/tests/storage/test_collection_schemas.py b/tests/storage/test_collection_schemas.pyindex 8047a9d15..3a29b11ef 100644--- a/tests/storage/test_collection_schemas.py+++ b/tests/storage/test_collection_schemas.py@@ -56,6 +56,20 @@ def _build_queue_payload() -> dict:return {"data": json.dumps(msg.to_dict())}+def _build_queue_payload_for_account(account_id: str) -> dict:+ msg = EmbeddingMsg(+ message="hello",+ context_data={+ "id": "id-1",+ "uri": "viking://resources/sample",+ "account_id": str(account_id),+ "abstract": "sample",+ },+ telemetry_id="telemetry-1",+ )+ return {"data": json.dumps(msg.to_dict())}++def test_embedding_handler_builds_circuit_breaker_from_config(monkeypatch):class _DummyVikingDB:is_closing = False@@ -199,6 +213,61 @@ async def test_embedding_handler_treats_shutdown_write_lock_as_success(monkeypatassert status["error"] == 0+async def test_embedding_handler_propagates_account_id_on_success(monkeypatch):+ class _DummyVikingDB:+ is_closing = False++ async def upsert(self, _data, *, ctx):+ return None++ captured: dict[str, object] = {}+ embedder = _DummyEmbedder()+ monkeypatch.setattr(+ "openviking_cli.utils.config.get_openviking_config",+ lambda: _DummyConfig(embedder),+ )+ monkeypatch.setattr(+ "openviking.metrics.datasources.EmbeddingEventDataSource.record_success",+ staticmethod(lambda **kwargs: captured.update(kwargs)),+ )++ handler = TextEmbeddingHandler(_DummyVikingDB())+ await handler.on_dequeue(_build_queue_payload_for_account("acct-embed-success"))++ assert captured["account_id"] == "acct-embed-success"+++async def test_embedding_handler_propagates_account_id_on_error(monkeypatch):+ class _DummyVikingDB:+ is_closing = False+ has_queue_manager = False++ class _BrokenEmbedder:+ def embed(self, text: str) -> EmbedResult:+ raise RuntimeError("boom")++ captured: dict[str, object] = {}+ monkeypatch.setattr(+ "openviking_cli.utils.config.get_openviking_config",+ lambda: _DummyConfig(_BrokenEmbedder()),+ )+ monkeypatch.setattr(+ "openviking.metrics.datasources.EmbeddingEventDataSource.record_error",+ staticmethod(lambda **kwargs: captured.update(kwargs)),+ )+ monkeypatch.setattr(+ "openviking.storage.collection_schemas.classify_api_error",+ lambda _err: "unknown",+ )++ handler = TextEmbeddingHandler(_DummyVikingDB())+ await handler.on_dequeue(_build_queue_payload_for_account("acct-embed-error"))++ assert captured["account_id"] == "acct-embed-error"++@pytest.mark.asyncioasync def test_embedding_handler_preserves_parent_uri_for_backend_upsert_logic(monkeypatch):captured = {}diff --git a/tests/test_server_config_loader.py b/tests/test_server_config_loader.pyindex 56d7af866..5f85b232f 100644--- a/tests/test_server_config_loader.py+++ b/tests/test_server_config_loader.py@@ -67,3 +67,36 @@ def test_load_server_config_preserves_supported_fields(tmp_path):assert config.bot_api_url == "http://localhost:19999"assert config.telemetry.prometheus.enabled is Trueassert config.encryption_enabled is True+++def test_load_server_config_preserves_metrics_account_dimension_fields(tmp_path):+ config_path = tmp_path / "ov.conf"+ config_path.write_text(+ json.dumps(+ {+ "server": {+ "metrics": {+ "enabled": True,+ "account_dimension": {+ "enabled": True,+ "max_active_accounts": 5,+ "metric_allowlist": [+ "openviking_http_requests_total",+ "openviking_task_pending",+ ],+ },+ }+ }+ }+ )+ )++ config = load_server_config(str(config_path))++ assert config.metrics.enabled is True+ assert config.metrics.account_dimension.enabled is True+ assert config.metrics.account_dimension.max_active_accounts == 5+ assert config.metrics.account_dimension.metric_allowlist == [+ "openviking_http_requests_total",+ "openviking_task_pending",+ ]diff --git a/tests/test_telemetry_runtime.py b/tests/test_telemetry_runtime.pyindex 967ce205a..27ba89fab 100644--- a/tests/test_telemetry_runtime.py+++ b/tests/test_telemetry_runtime.py@@ -8,6 +8,7 @@ from types import SimpleNamespaceimport pytest+from openviking.metrics.account_context import get_metric_account_contextfrom openviking.models.embedder.base import EmbedResultfrom openviking.server.identity import RequestContext, Rolefrom openviking.service.resource_service import ResourceService@@ -180,6 +181,46 @@ async def test_semantic_processor_binds_registered_operation_telemetry(monkeypatassert "embedding" not in summary["tokens"]+async def test_semantic_processor_binds_metric_account_context(monkeypatch):+ processor = SemanticProcessor()+ ran = {"value": False}++ class FakeVikingFS:+ async def ls(self, uri, ctx=None):+ return []++ class _FakeDagExecutor:+ def __init__(self, **kwargs):+ pass++ async def run(self, root_uri):+ ran["value"] = True+ assert get_metric_account_context().http_account_id == "acct-semantic"++ def get_stats(self):+ return DagStats()++ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_processor.get_viking_fs",+ lambda: FakeVikingFS(),+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor",+ lambda **kwargs: _FakeDagExecutor(**kwargs),+ )++ await processor.on_dequeue(+ SemanticMsg(+ uri="viking://resources/demo",+ context_type="resource",+ recursive=False,+ account_id="acct-semantic",+ ).to_dict()+ )+ assert ran["value"] is True++@pytest.mark.asyncioasync def test_embedding_handler_binds_registered_operation_telemetry(monkeypatch):telemetry = MemoryOperationTelemetry(operation="resources.add_resource", enabled=True)diff --git a/tests/unit/crypto/test_providers_mock.py b/tests/unit/crypto/test_providers_mock.pyindex d045b5d58..192489db1 100644--- a/tests/unit/crypto/test_providers_mock.py+++ b/tests/unit/crypto/test_providers_mock.py@@ -20,6 +20,7 @@ from unittest.mock import AsyncMock, Mock, patchimport pytest+import openviking.crypto.providers as providers_modulefrom openviking.crypto.encryptor import FileEncryptorfrom openviking.crypto.providers import (PROVIDER_LOCAL,@@ -29,6 +30,7 @@ from openviking.crypto.providers import (VaultProvider,VolcengineKMSProvider,)+from openviking.metrics.datasources.encryption import EncryptionEventDataSource@pytest.fixture@@ -273,3 +275,73 @@ class TestCrossProviderEnvelope:finally:if os.path.exists(temp_path2):os.unlink(temp_path2)+++async def test_hkdf_derive_logs_debug_when_metrics_recording_fails(+ local_file_provider, monkeypatch: pytest.MonkeyPatch+):+ """HKDF derivation should log metrics side-effect failures without changing the result."""++ debug = Mock()++ def _boom(*_args, **_kwargs):+ raise RuntimeError("metrics write failed")++ monkeypatch.setattr(providers_module.logger, "debug", debug)+ monkeypatch.setattr(EncryptionEventDataSource, "record_key_derivation", staticmethod(_boom))++ result = await local_file_provider._hkdf_derive(+ b"a" * 32,+ "acct-1",+ b"salt",+ b"info:",+ )++ assert isinstance(result, bytes)+ assert len(result) == 32+ debug.assert_called_once()+++ ("provider_factory", "provider_name"),+ [+ ("local_file_provider", "local"),+ ("vault_mock_provider", "vault"),+ ("volcengine_mock_provider", "volcengine_kms"),+ ],+)+async def test_get_root_key_logs_debug_when_metrics_recording_fails(+ request: pytest.FixtureRequest,+ provider_factory: str,+ provider_name: str,+ monkeypatch: pytest.MonkeyPatch,+):+ """Provider get_root_key should log metrics side-effect failures and still return the key."""++ provider = request.getfixturevalue(provider_factory)+ provider._root_key = None+ debug = Mock()++ if provider_name == "local":+ provider._load_or_create_root_key = AsyncMock(return_value=b"k" * 32)+ else:+ provider._get_or_create_root_key = AsyncMock(return_value=b"k" * 32)++ def _boom(*_args, **_kwargs):+ raise RuntimeError("metrics write failed")++ monkeypatch.setattr(providers_module.logger, "debug", debug)+ monkeypatch.setattr(EncryptionEventDataSource, "record_key_load", staticmethod(_boom))++ result = await provider.get_root_key()++ assert result == b"k" * 32+ if provider_name == "local":+ provider._load_or_create_root_key.assert_awaited_once()+ else:+ provider._get_or_create_root_key.assert_awaited_once()+ debug.assert_called_once()+ (message,) = debug.call_args.args[:1]+ assert "Failed to record encryption key metrics" in str(message)