Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(trace): add request-level trace metrics and API support (#640)
zhoujiahuib280b56b308a
diff --git a/docs/design/operation_telemetry.md b/docs/design/operation_telemetry.mdnew file mode 100644index 000000000..d20fd00eb--- /dev/null+++ b/docs/design/operation_telemetry.md@@ -0,0 +1,520 @@+# 操作级 Telemetry 设计++## 1. 背景与目标++OpenViking 需要一套统一的 telemetry 机制,用来描述一次操作在执行过程中的关键观测信息。当前已经落地的是操作级 telemetry,主要覆盖:++- 请求耗时统计+- token 消耗统计+- 检索、队列、内存提取等阶段的摘要指标++这里统一使用 `telemetry`,而不是 `trace`,原因是这套抽象未来不只服务于“单次操作链路”,还要能承载非操作级数据,例如:++- 服务整体 token 消耗+- 各类后端能力的延迟与错误率+- 存储、向量库、队列等组件级指标+- 基于 OpenTelemetry 的 exporter / backend 对接++当前实现只对“操作级 telemetry”提供正式接口,但抽象命名和结构已经为后续扩展预留空间。++## 2. 设计原则++### 2.1 详细信息显式按需返回++详细 telemetry 由调用方通过 `telemetry` 参数显式请求,当前对外协议只返回结构化 summary,不返回事件流。++### 2.3 字段名直接面向用户++内部打点名与对外 summary 字段名保持一致,避免额外的“内部名 -> 外部名”转换层。++### 2.4 缺失分组不返回++如果某类操作天然不会产出某个 summary 分组,则该分组直接省略,不返回空对象或全 `null` 字段。++例如:++- `resources.add_resource` 不一定有 `memory`+- `session.commit` 一般没有 `semantic_nodes`+- 某些操作没有向量检索,就不返回 `vector`++## 3. 当前支持范围++### 3.1 HTTP 接口++当前已接入 operation telemetry 的接口:++- `POST /api/v1/search/find`+- `POST /api/v1/search/search`+- `POST /api/v1/resources`+- `POST /api/v1/skills`+- `POST /api/v1/sessions/{session_id}/commit`++说明:++- `session.commit` 仅在 `wait=true` 的同步模式下支持返回 telemetry+- `wait=false` 的异步任务模式当前不支持 telemetry,请求时会返回 `INVALID_ARGUMENT`++### 3.2 SDK 接口++当前已接入 operation telemetry 的 SDK 方法:++- `add_resource`+- `add_skill`+- `find`+- `search`+- `commit_session`++本地嵌入式 client 和 HTTP client 都遵循同一套 telemetry 请求语义。++## 4. 响应模型++服务端仍使用统一响应包裹结构:++```json+{+ "status": "ok",+ "result": { "...": "..." },+ "time": 0.031,+ "telemetry": {+ "id": "tm_9f6f4d6b0d0c4f4d93ce5adf82e71c18",+ "summary": {+ "operation": "search.find",+ "status": "ok",+ "duration_ms": 31.224,+ "tokens": {+ "total": 24,+ "llm": {+ "input": 12,+ "output": 6,+ "total": 18+ },+ "embedding": {+ "total": 6+ }+ },+ "vector": {+ "searches": 3,+ "scored": 26,+ "passed": 8,+ "returned": 5,+ "scanned": 26,+ "scan_reason": ""+ }+ }+ }+}+```++说明:++- `telemetry` 只在调用方显式请求时返回+- `telemetry.id` 是不透明标识,只用于关联,不要求调用方解析语义++## 5. telemetry 请求语义++`telemetry` 字段支持两种形态:++### 5.1 布尔形态++```json+{+ "telemetry": true+}+```++语义:++- 返回 `telemetry.id + telemetry.summary`++### 5.2 对象形态++```json+{+ "telemetry": {+ "summary": true+ }+}+```++语义:++- `summary` 默认值为 `true`+- 适合只看结构化摘要++当前支持的合法组合如下:++| 请求值 | 语义 |+| --- | --- |+| `false` | 不返回 `telemetry` |+| `true` | 返回 `id + summary` |+| `{"summary": true}` | 返回 `id + summary` |+| `{"summary": false}` | 不返回 `telemetry` |++以下请求非法:++```json+{+ "telemetry": {+ "events": true+ }+}+```++原因是当前对外 telemetry 已收敛为 summary-only,不再接受事件流选择参数。++## 6. telemetry 的职责划分++### 6.1 `telemetry.summary`++`summary` 是结构化的操作摘要,用于:++- 调试+- 排障+- 离线分析+- 上报到外部观测系统++当前 summary 的核心字段包括:++- `operation`+- `status`+- `duration_ms`+- `tokens`+- `queue`+- `vector`+- `semantic_nodes`+- `memory`+- `errors`++其中:++- `tokens` 始终存在+- 其余分组按是否有产出决定是否返回++### 6.3 `telemetry.id`++`telemetry.id` 是请求级关联标识,用于把一次操作的 summary 与内部异步链路统计关联起来。++## 7. summary 字段约定++### 7.1 顶层公共字段++所有 summary 至少包含:++- `operation`+- `status`+- `duration_ms`+- `tokens`++### 7.2 tokens++示例:++```json+{+ "tokens": {+ "total": 19,+ "llm": {+ "input": 11,+ "output": 7,+ "total": 18+ },+ "embedding": {+ "total": 1+ }+ }+}+```++说明:++- `llm` 统计输入、输出与总量+- `embedding` 当前只统计总量++### 7.3 queue++队列相关摘要示例:++```json+{+ "queue": {+ "semantic": {+ "processed": 1,+ "error_count": 0+ },+ "embedding": {+ "processed": 1,+ "error_count": 0+ }+ }+}+```++### 7.4 vector++向量检索摘要示例:++```json+{+ "vector": {+ "searches": 2,+ "scored": 5,+ "passed": 3,+ "returned": 2,+ "scanned": 5,+ "scan_reason": ""+ }+}+```++### 7.5 semantic_nodes++语义检索 DAG / 节点级摘要示例:++```json+{+ "semantic_nodes": {+ "total": 4,+ "done": 3,+ "pending": 1,+ "running": 0+ }+}+```++### 7.6 memory++会话提交等内存提取类操作示例:++```json+{+ "memory": {+ "extracted": 4+ }+}+```++### 7.7 errors++发生错误时可返回:++```json+{+ "errors": {+ "stage": "resource_processor.parse",+ "error_code": "PROCESSING_ERROR",+ "message": "..."+ }+}+```++无错误时,该分组可以省略。++## 8. 缺失字段裁剪策略++summary 采用“按分组裁剪”的策略,而不是固定返回整套字段。++这样做有几个直接收益:++- 避免返回大量与当前操作无关的空字段+- 降低调用方理解成本+- 更适合未来扩展新的 telemetry 分组++示例:++### 8.1 `resources.add_resource`++可能返回:++```json+{+ "operation": "resources.add_resource",+ "status": "ok",+ "duration_ms": 152.3,+ "tokens": { "...": "..." },+ "semantic_nodes": { "...": "..." },+ "queue": { "...": "..." }+}+```++这里不应强行返回 `memory`。++### 8.2 `session.commit`++可能返回:++```json+{+ "operation": "session.commit",+ "status": "ok",+ "duration_ms": 48.1,+ "tokens": { "...": "..." },+ "memory": {+ "extracted": 4+ }+}+```++这里不应强行返回 `semantic_nodes`。++## 9. 成本模型++当前 collector 只采集 summary 所需的数据:++- 采集 counters / gauges+- 记录 error 状态+- 构造最终 summary+- 不保留事件列表++## 10. 实现结构++### 10.1 核心类型++核心实现位于:++- `openviking/telemetry/operation.py`+- `openviking/telemetry/request.py`+- `openviking/telemetry/context.py`+- `openviking/telemetry/registry.py`++主要对象包括:++- `OperationTelemetry`+- `TelemetrySnapshot`+- `TelemetrySelection`++### 10.2 请求解析++`openviking/telemetry/request.py` 负责统一解析 `telemetry` 请求参数:++- 支持 `bool | object`+- 归一化为 `TelemetrySelection`+- 校验非法字段,例如 `events`++这样 server、local client、HTTP client 都共享同一套语义。++### 10.3 服务端集成++`openviking/server/telemetry.py` 负责:++- 根据请求创建 collector+- 根据 selection 决定是否附带 `summary`++router 层的职责是:++1. 创建 collector+2. 绑定 operation 上下文+3. 执行实际业务逻辑+4. 按请求返回 `telemetry`++### 10.4 本地与 HTTP client++本地 client 和 HTTP client 都暴露同样的 `telemetry` 参数语义:++```python+await client.find("memory dedup", telemetry=True)+await client.find("memory dedup", telemetry={"summary": True})+```++其中:++- local client 在本地生成 telemetry 并拼回结果+- HTTP client 负责参数校验并透传给服务端++## 11. 异步链路与跨组件聚合++当前 operation telemetry 不只覆盖同步请求栈,也支持部分异步处理链路的数据回流。++典型场景包括:++- 请求线程触发语义队列处理+- 请求线程触发 embedding 处理+- 后台处理线程继续向同一个 operation collector 记录指标++实现方式是:++- collector 生成 `telemetry.id`+- 后续消息携带该 `id`+- 后台组件通过 registry 找回原 collector+- 在新的执行上下文中重新绑定 collector++这样一次操作的最终 summary 可以覆盖:++- 请求入口逻辑+- 检索过程+- embedding 处理+- semantic queue 处理+- memory 提取结果++## 12. 与 OpenTelemetry 的关系++当前方案不是直接把 OpenTelemetry 暴露为业务接口,而是先定义 OpenViking 自己的 telemetry 抽象。++这样做的好处是:++- 对调用方暴露稳定、简单的产品接口+- 不把业务接口和具体观测框架强绑定+- 后续可以新增 OpenTelemetry backend,而不影响现有 SDK / HTTP 语义++可以把 OpenTelemetry 看作未来的一种底层实现或导出方式,而不是当前对外协议本身。++## 13. 未来扩展方向++当前文档描述的是 operation telemetry,但未来需要兼容更广义的 telemetry 数据源。++推荐的扩展方向:++- 服务级 token 消耗聚合+- 存储、向量库、模型服务的接口耗时+- 队列吞吐、失败率、积压长度+- 与 OpenTelemetry exporter 的桥接+- 更长期的指标聚合、采样和导出++这些扩展不要求沿用完全相同的 summary schema,但应复用统一的 telemetry 抽象和运行时。++## 14. 使用示例++### 14.1 返回 telemetry summary++```bash+curl -X POST http://localhost:8080/api/v1/search/find \+ -H 'Content-Type: application/json' \+ -d '{+ "query": "memory dedup",+ "limit": 5,+ "telemetry": true+ }'+```++### 14.2 只返回 summary++```bash+curl -X POST http://localhost:8080/api/v1/search/find \+ -H 'Content-Type: application/json' \+ -d '{+ "query": "memory dedup",+ "limit": 5,+ "telemetry": {+ "summary": true+ }+ }'+```++### 14.4 Python SDK++```python+result = await client.find("memory dedup", telemetry={"summary": True})++print(result.telemetry["summary"]["tokens"]["total"])+```++## 15. 新接口接入规范++新接口如果需要接入 operation telemetry,建议遵循以下规则:++1. 为该操作创建 `OperationTelemetry` collector。+2. 用上下文绑定覆盖整个操作生命周期。+3. 在内部关键阶段记录 counters、gauges 和错误状态。+4. 仅在调用方请求时返回 `telemetry`。+5. summary 只返回本次操作真实产出的分组。++这样可以保持默认低成本,同时为调用方提供稳定、可分析的结构化摘要。diff --git a/examples/memory_demo.py b/examples/misc/memory_demo.pysimilarity index 100%rename from examples/memory_demo.pyrename to examples/misc/memory_demo.pydiff --git a/openviking/async_client.py b/openviking/async_client.pyindex 67dfa6966..294d15daf 100644--- a/openviking/async_client.py+++ b/openviking/async_client.py@@ -13,6 +13,7 @@ from typing import Any, Dict, List, Optional, Unionfrom openviking.client import LocalClient, Sessionfrom openviking.service.debug_service import SystemStatus+from openviking.telemetry import TelemetryRequestfrom openviking_cli.client.base import BaseClientfrom openviking_cli.session.user_id import UserIdentifierfrom openviking_cli.utils import get_logger@@ -164,10 +165,12 @@ class AsyncOpenViking:session_id=session_id, role=role, content=content, parts=parts)- async def commit_session(self, session_id: str) -> Dict[str, Any]:+ async def commit_session(+ self, session_id: str, telemetry: TelemetryRequest = False+ ) -> Dict[str, Any]:"""Commit a session (archive and extract memories)."""await self._ensure_initialized()- return await self._client.commit_session(session_id)+ return await self._client.commit_session(session_id, telemetry=telemetry)# ============= Resource methods =============@@ -182,6 +185,7 @@ class AsyncOpenViking:timeout: float = None,build_index: bool = True,summarize: bool = False,+ telemetry: TelemetryRequest = False,**kwargs,) -> Dict[str, Any]:"""@@ -196,10 +200,10 @@ class AsyncOpenViking:parent: Target parent URI (must already exist).build_index: Whether to build vector index immediately (default: True).summarize: Whether to generate summary (default: False).+ telemetry: Whether to attach operation telemetry data to the result."""await self._ensure_initialized()- # Validate that only one of 'to' or 'parent' is setif to and parent:raise ValueError("Cannot specify both 'to' and 'parent' at the same time.")@@ -213,6 +217,7 @@ class AsyncOpenViking:timeout=timeout,build_index=build_index,summarize=summarize,+ telemetry=telemetry,**kwargs,)@@ -246,6 +251,7 @@ class AsyncOpenViking:data: Any,wait: bool = False,timeout: float = None,+ telemetry: TelemetryRequest = False,) -> Dict[str, Any]:"""Add skill to OpenViking.@@ -258,6 +264,7 @@ class AsyncOpenViking:data=data,wait=wait,timeout=timeout,+ telemetry=telemetry,)# ============= Search methods =============@@ -271,6 +278,7 @@ class AsyncOpenViking:limit: int = 10,score_threshold: Optional[float] = None,filter: Optional[Dict] = None,+ telemetry: TelemetryRequest = False,):"""Complex search with session context.@@ -295,6 +303,7 @@ class AsyncOpenViking:limit=limit,score_threshold=score_threshold,filter=filter,+ telemetry=telemetry,)async def find(@@ -304,6 +313,7 @@ class AsyncOpenViking:limit: int = 10,score_threshold: Optional[float] = None,filter: Optional[Dict] = None,+ telemetry: TelemetryRequest = False,):"""Semantic search"""await self._ensure_initialized()@@ -313,6 +323,7 @@ class AsyncOpenViking:limit=limit,score_threshold=score_threshold,filter=filter,+ telemetry=telemetry,)# ============= FS methods =============diff --git a/openviking/client/local.py b/openviking/client/local.pyindex 506041e82..a6bcf061b 100644--- a/openviking/client/local.py+++ b/openviking/client/local.py@@ -9,6 +9,11 @@ from typing import Any, Dict, List, Optional, Unionfrom openviking.server.identity import RequestContext, Rolefrom openviking.service import OpenVikingService+from openviking.telemetry import TelemetryRequest+from openviking.telemetry.execution import (+ attach_telemetry_payload,+ run_with_telemetry,+)from openviking_cli.client.base import BaseClientfrom openviking_cli.session.user_id import UserIdentifierfrom openviking_cli.utils import run_async@@ -62,23 +67,35 @@ class LocalClient(BaseClient):instruction: str = "",wait: bool = False,timeout: Optional[float] = None,+ build_index: bool = True,+ summarize: bool = False,+ telemetry: TelemetryRequest = False,**kwargs,) -> Dict[str, Any]:"""Add resource to OpenViking."""- # Validate that only one of 'to' or 'parent' is setif to and parent:raise ValueError("Cannot specify both 'to' and 'parent' at the same time.")- return await self._service.resources.add_resource(- path=path,- ctx=self._ctx,- to=to,- parent=parent,- reason=reason,- instruction=instruction,- wait=wait,- timeout=timeout,- **kwargs,+ execution = await run_with_telemetry(+ operation="resources.add_resource",+ telemetry=telemetry,+ fn=lambda: self._service.resources.add_resource(+ path=path,+ ctx=self._ctx,+ to=to,+ parent=parent,+ reason=reason,+ instruction=instruction,+ wait=wait,+ timeout=timeout,+ build_index=build_index,+ summarize=summarize,+ **kwargs,+ ),+ )+ return attach_telemetry_payload(+ execution.result,+ execution.telemetry,)async def add_skill(@@ -86,13 +103,22 @@ class LocalClient(BaseClient):data: Any,wait: bool = False,timeout: Optional[float] = None,+ telemetry: TelemetryRequest = False,) -> Dict[str, Any]:"""Add skill to OpenViking."""- return await self._service.resources.add_skill(- data=data,- ctx=self._ctx,- wait=wait,- timeout=timeout,+ execution = await run_with_telemetry(+ operation="resources.add_skill",+ telemetry=telemetry,+ fn=lambda: self._service.resources.add_skill(+ data=data,+ ctx=self._ctx,+ wait=wait,+ timeout=timeout,+ ),+ )+ return attach_telemetry_payload(+ execution.result,+ execution.telemetry,)async def wait_processed(self, timeout: Optional[float] = None) -> Dict[str, Any]:@@ -196,15 +222,24 @@ class LocalClient(BaseClient):limit: int = 10,score_threshold: Optional[float] = None,filter: Optional[Dict[str, Any]] = None,+ telemetry: TelemetryRequest = False,) -> Any:"""Semantic search without session context."""- return await self._service.search.find(- query=query,- ctx=self._ctx,- target_uri=target_uri,- limit=limit,- score_threshold=score_threshold,- filter=filter,+ execution = await run_with_telemetry(+ operation="search.find",+ telemetry=telemetry,+ fn=lambda: self._service.search.find(+ query=query,+ ctx=self._ctx,+ target_uri=target_uri,+ limit=limit,+ score_threshold=score_threshold,+ filter=filter,+ ),+ )+ return attach_telemetry_payload(+ execution.result,+ execution.telemetry,)async def search(@@ -215,20 +250,33 @@ class LocalClient(BaseClient):limit: int = 10,score_threshold: Optional[float] = None,filter: Optional[Dict[str, Any]] = None,+ telemetry: TelemetryRequest = False,) -> Any:"""Semantic search with optional session context."""- session = None- if session_id:- session = self._service.sessions.session(self._ctx, session_id)- await session.load()- return await self._service.search.search(- query=query,- ctx=self._ctx,- target_uri=target_uri,- session=session,- limit=limit,- score_threshold=score_threshold,- filter=filter,++ async def _search():+ session = None+ if session_id:+ session = self._service.sessions.session(self._ctx, session_id)+ await session.load()+ return await self._service.search.search(+ query=query,+ ctx=self._ctx,+ target_uri=target_uri,+ session=session,+ limit=limit,+ score_threshold=score_threshold,+ filter=filter,+ )++ execution = await run_with_telemetry(+ operation="search.search",+ telemetry=telemetry,+ fn=_search,+ )+ return attach_telemetry_payload(+ execution.result,+ execution.telemetry,)async def grep(self, uri: str, pattern: str, case_insensitive: bool = False) -> Dict[str, Any]:@@ -284,9 +332,19 @@ class LocalClient(BaseClient):"""Delete a session."""await self._service.sessions.delete(session_id, self._ctx)- async def commit_session(self, session_id: str) -> Dict[str, Any]:+ async def commit_session(+ self, session_id: str, telemetry: TelemetryRequest = False+ ) -> Dict[str, Any]:"""Commit a session (archive and extract memories)."""- return await self._service.sessions.commit(session_id, self._ctx)+ execution = await run_with_telemetry(+ operation="session.commit",+ telemetry=telemetry,+ fn=lambda: self._service.sessions.commit(session_id, self._ctx),+ )+ return attach_telemetry_payload(+ execution.result,+ execution.telemetry,+ )async def add_message(self,diff --git a/openviking/client/session.py b/openviking/client/session.pyindex 51ccaf237..6289decad 100644--- a/openviking/client/session.py+++ b/openviking/client/session.py@@ -9,6 +9,7 @@ from dataclasses import asdictfrom typing import TYPE_CHECKING, Any, Dict, List, Optionalfrom openviking.message.part import Part+from openviking.telemetry import TelemetryRequestfrom openviking_cli.session.user_id import UserIdentifierif TYPE_CHECKING:@@ -57,13 +58,13 @@ class Session:return await self._client.add_message(self.session_id, role, parts=parts_dicts)return await self._client.add_message(self.session_id, role, content=content)- async def commit(self) -> Dict[str, Any]:+ async def commit(self, telemetry: TelemetryRequest = False) -> Dict[str, Any]:"""Commit the session (archive messages and extract memories).Returns:Commit result"""- return await self._client.commit_session(self.session_id)+ return await self._client.commit_session(self.session_id, telemetry=telemetry)async def delete(self) -> None:"""Delete the session."""diff --git a/openviking/console/app.py b/openviking/console/app.pyindex ea1aba648..c28eda572 100644--- a/openviking/console/app.py+++ b/openviking/console/app.py@@ -4,6 +4,7 @@from __future__ import annotations+import jsonimport refrom contextlib import asynccontextmanagerfrom pathlib import Path@@ -46,6 +47,36 @@ _ALLOWED_FORWARD_RESPONSE_HEADERS = {}+def _is_json_content_type(content_type: str) -> bool:+ value = (content_type or "").lower()+ return "application/json" in value or "+json" in value+++def _should_default_telemetry(upstream_path: str) -> bool:+ if upstream_path in {"/api/v1/search/find", "/api/v1/resources"}:+ return True+ return upstream_path.startswith("/api/v1/sessions/") and upstream_path.endswith("/commit")+++def _with_default_telemetry(request: Request, upstream_path: str, body: bytes) -> bytes:+ if request.method.upper() != "POST":+ return body+ if not _should_default_telemetry(upstream_path):+ return body+ if not _is_json_content_type(request.headers.get("content-type", "")):+ return body++ try:+ payload = json.loads(body.decode("utf-8")) if body else {}+ except (json.JSONDecodeError, UnicodeDecodeError):+ return body+ if not isinstance(payload, dict):+ return body++ payload.setdefault("telemetry", True)+ return json.dumps(payload).encode("utf-8")++def _error_response(status_code: int, code: str, message: str, details: Optional[dict] = None):return JSONResponse(status_code=status_code,@@ -80,6 +111,7 @@ async def _forward_request(request: Request, upstream_path: str) -> Response:"""Forward the incoming request to OpenViking upstream."""client: httpx.AsyncClient = request.app.state.upstream_clientbody = await request.body()+ body = _with_default_telemetry(request, upstream_path, body)try:upstream_response = await client.request(method=request.method,diff --git a/openviking/console/bootstrap.py b/openviking/console/bootstrap.pyindex 02a9cae1e..aca731129 100644--- a/openviking/console/bootstrap.py+++ b/openviking/console/bootstrap.py@@ -40,7 +40,7 @@ def _build_parser() -> argparse.ArgumentParser:parser.add_argument("--request-timeout-sec",type=float,- default=30.0,+ default=3600.0,help="Upstream request timeout in seconds",)parser.add_argument(diff --git a/openviking/models/embedder/openai_embedders.py b/openviking/models/embedder/openai_embedders.pyindex 61c5c9983..032d766d0 100644--- a/openviking/models/embedder/openai_embedders.py+++ b/openviking/models/embedder/openai_embedders.py@@ -12,6 +12,7 @@ from openviking.models.embedder.base import (HybridEmbedderBase,SparseEmbedderBase,)+from openviking.telemetry import get_current_telemetryclass OpenAIDenseEmbedder(DenseEmbedderBase):@@ -79,6 +80,25 @@ class OpenAIDenseEmbedder(DenseEmbedderBase):# Use default value, text-embedding-3-small defaults to 1536return 1536+ def _update_telemetry_token_usage(self, response) -> None:+ usage = getattr(response, "usage", None)+ if not usage:+ return++ def _usage_value(key: str, default: int = 0) -> int:+ if isinstance(usage, dict):+ return int(usage.get(key, default) or default)+ return int(getattr(usage, key, default) or default)++ prompt_tokens = _usage_value("prompt_tokens", 0)+ total_tokens = _usage_value("total_tokens", prompt_tokens)+ output_tokens = max(total_tokens - prompt_tokens, 0)+ get_current_telemetry().add_token_usage_by_source(+ "embedding",+ prompt_tokens,+ output_tokens,+ )+def embed(self, text: str) -> EmbedResult:"""Perform dense embedding on text@@ -97,6 +117,7 @@ class OpenAIDenseEmbedder(DenseEmbedderBase):kwargs["dimensions"] = self.dimensionresponse = self.client.embeddings.create(**kwargs)+ self._update_telemetry_token_usage(response)vector = response.data[0].embeddingreturn EmbedResult(dense_vector=vector)@@ -126,6 +147,7 @@ class OpenAIDenseEmbedder(DenseEmbedderBase):kwargs["dimensions"] = self.dimensionresponse = self.client.embeddings.create(**kwargs)+ self._update_telemetry_token_usage(response)return [EmbedResult(dense_vector=item.embedding) for item in response.data]except openai.APIError as e:diff --git a/openviking/models/embedder/volcengine_embedders.py b/openviking/models/embedder/volcengine_embedders.pyindex dd6dc1ae6..90d82d0b9 100644--- a/openviking/models/embedder/volcengine_embedders.py+++ b/openviking/models/embedder/volcengine_embedders.py@@ -14,6 +14,7 @@ from openviking.models.embedder.base import (exponential_backoff_retry,truncate_and_normalize,)+from openviking.telemetry import get_current_telemetryfrom openviking_cli.utils.logger import default_logger as logger@@ -125,6 +126,25 @@ class VolcengineDenseEmbedder(DenseEmbedderBase):except Exception:return 2048 # Default dimension+ def _update_telemetry_token_usage(self, response) -> None:+ usage = getattr(response, "usage", None)+ if not usage:+ return++ def _usage_value(key: str, default: int = 0) -> int:+ if isinstance(usage, dict):+ return int(usage.get(key, default) or default)+ return int(getattr(usage, key, default) or default)++ prompt_tokens = _usage_value("prompt_tokens", 0)+ total_tokens = _usage_value("total_tokens", prompt_tokens)+ output_tokens = max(total_tokens - prompt_tokens, 0)+ get_current_telemetry().add_token_usage_by_source(+ "embedding",+ prompt_tokens,+ output_tokens,+ )+def embed(self, text: str) -> EmbedResult:"""Perform dense embedding on text@@ -144,10 +164,12 @@ class VolcengineDenseEmbedder(DenseEmbedderBase):response = self.client.multimodal_embeddings.create(input=[{"type": "text", "text": text}], model=self.model_name)+ self._update_telemetry_token_usage(response)vector = response.data.embeddingelse:# Use text embeddings APIresponse = self.client.embeddings.create(input=text, model=self.model_name)+ self._update_telemetry_token_usage(response)vector = response.data[0].embeddingvector = truncate_and_normalize(vector, self.dimension)@@ -187,9 +209,11 @@ class VolcengineDenseEmbedder(DenseEmbedderBase):response = self.client.multimodal_embeddings.create(input=multimodal_inputs, model=self.model_name)+ self._update_telemetry_token_usage(response)data = response.dataelse:response = self.client.embeddings.create(input=texts, model=self.model_name)+ self._update_telemetry_token_usage(response)data = response.datareturn [diff --git a/openviking/models/vlm/base.py b/openviking/models/vlm/base.pyindex 4b602f780..66b859ba1 100644--- a/openviking/models/vlm/base.py+++ b/openviking/models/vlm/base.py@@ -80,6 +80,14 @@ class VLMBase(ABC):prompt_tokens=prompt_tokens,completion_tokens=completion_tokens,)+ # Operation-level telemetry aggregation (no-op when telemetry is disabled).+ try:+ from openviking.telemetry import get_current_telemetry++ get_current_telemetry().add_token_usage(prompt_tokens, completion_tokens)+ except Exception:+ # Telemetry must never break model inference.+ passdef get_token_usage(self) -> Dict[str, Any]:"""Get token usagediff --git a/openviking/parse/tree_builder.py b/openviking/parse/tree_builder.pyindex c8409f414..9a2543ce4 100644--- a/openviking/parse/tree_builder.py+++ b/openviking/parse/tree_builder.py@@ -30,6 +30,7 @@ from openviking.parse.parsers.media.utils import get_media_base_uri, get_media_tfrom openviking.server.identity import RequestContextfrom openviking.storage.queuefs import SemanticMsg, get_queue_managerfrom openviking.storage.viking_fs import get_viking_fs+from openviking.telemetry import get_current_telemetryfrom openviking.utils import parse_code_hosting_urlfrom openviking_cli.utils.uri import VikingURI@@ -284,6 +285,7 @@ class TreeBuilder:user_id=ctx.user.user_id,agent_id=ctx.user.agent_id,role=ctx.role.value,+ telemetry_id=get_current_telemetry().telemetry_id,)await semantic_queue.enqueue(msg)diff --git a/openviking/retrieve/hierarchical_retriever.py b/openviking/retrieve/hierarchical_retriever.pyindex 906795909..cddc5fea1 100644--- a/openviking/retrieve/hierarchical_retriever.py+++ b/openviking/retrieve/hierarchical_retriever.py@@ -19,6 +19,7 @@ from openviking.retrieve.retrieval_stats import get_stats_collectorfrom openviking.server.identity import RequestContext, Rolefrom openviking.storage import VikingDBManager, VikingDBManagerProxyfrom openviking.storage.viking_fs import get_viking_fs+from openviking.telemetry import get_current_telemetryfrom openviking.utils.time_utils import parse_iso_datetimefrom openviking_cli.retrieve.types import (ContextType,@@ -102,9 +103,7 @@ class HierarchicalRetriever:grep_patterns: Keyword match pattern listscope_dsl: Additional scope constraints passed from public find/search filter"""-t0 = time.monotonic()-# Use custom threshold or default thresholdeffective_threshold = score_threshold if score_threshold is not None else self.threshold@@ -234,6 +233,10 @@ class HierarchicalRetriever:extra_filter=scope_dsl,limit=limit,)+ telemetry = get_current_telemetry()+ telemetry.count("vector.searches", 1)+ telemetry.count("vector.scored", len(results))+ telemetry.count("vector.scanned", len(results))return resultsdef _rerank_scores(@@ -388,6 +391,10 @@ class HierarchicalRetriever:extra_filter=scope_dsl,limit=pre_filter_limit,)+ telemetry = get_current_telemetry()+ telemetry.count("vector.searches", 1)+ telemetry.count("vector.scored", len(results))+ telemetry.count("vector.scanned", len(results))if not results:continue@@ -409,6 +416,7 @@ class HierarchicalRetriever:)continue+ telemetry.count("vector.passed", 1)# Deduplicate by URI and keep the highest-scored candidate.previous = collected_by_uri.get(uri)if previous is None or final_score > previous.get("_final_score", 0):diff --git a/openviking/server/models.py b/openviking/server/models.pyindex 4cb7f967d..a97b086bb 100644--- a/openviking/server/models.py+++ b/openviking/server/models.py@@ -2,7 +2,7 @@# SPDX-License-Identifier: Apache-2.0"""Response models and error codes for OpenViking HTTP Server."""-from typing import Any, Optional+from typing import Any, Dict, Optionalfrom pydantic import BaseModel@@ -15,21 +15,13 @@ class ErrorInfo(BaseModel):details: Optional[dict] = None-class UsageInfo(BaseModel):- """Usage information."""-- tokens: Optional[int] = None- vectors_scanned: Optional[int] = None--class Response(BaseModel):"""Standard API response."""status: str # "ok" | "error"result: Optional[Any] = Noneerror: Optional[ErrorInfo] = None- time: float = 0.0- usage: Optional[UsageInfo] = None+ telemetry: Optional[Dict[str, Any]] = None# Error code to HTTP status code mappingdiff --git a/openviking/server/routers/resources.py b/openviking/server/routers/resources.pyindex e1c63bc33..254d291bd 100644--- a/openviking/server/routers/resources.py+++ b/openviking/server/routers/resources.py@@ -14,6 +14,8 @@ from openviking.server.auth import get_request_contextfrom openviking.server.dependencies import get_servicefrom openviking.server.identity import RequestContextfrom openviking.server.models import Response+from openviking.server.telemetry import run_operation+from openviking.telemetry import TelemetryRequestfrom openviking_cli.exceptions import InvalidArgumentErrorfrom openviking_cli.utils.config.open_viking_config import get_openviking_config@@ -37,6 +39,7 @@ class AddResourceRequest(BaseModel):exclude: Optional[str] = Nonedirectly_upload_media: bool = Truepreserve_structure: Optional[bool] = None+ telemetry: TelemetryRequest = False@model_validator(mode="after")def check_path_or_temp_path(self):@@ -52,6 +55,7 @@ class AddSkillRequest(BaseModel):temp_path: Optional[str] = Nonewait: bool = Falsetimeout: Optional[float] = None+ telemetry: TelemetryRequest = Falsedef _cleanup_temp_files(temp_dir: Path, max_age_hours: int = 1):@@ -98,12 +102,10 @@ async def add_resource(_ctx: RequestContext = Depends(get_request_context),):"""Add resource to OpenViking."""- # Validate request: only one of 'to' or 'parent' can be set+ service = get_service()if request.to and request.parent:raise InvalidArgumentError("Cannot specify both 'to' and 'parent' at the same time.")- service = get_service()-path = request.pathif request.temp_path:path = request.temp_path@@ -120,18 +122,26 @@ async def add_resource(if request.preserve_structure is not None:kwargs["preserve_structure"] = request.preserve_structure- result = await service.resources.add_resource(- path=path,- ctx=_ctx,- to=request.to,- parent=request.parent,- reason=request.reason,- instruction=request.instruction,- wait=request.wait,- timeout=request.timeout,- **kwargs,+ execution = await run_operation(+ operation="resources.add_resource",+ telemetry=request.telemetry,+ fn=lambda: service.resources.add_resource(+ path=path,+ ctx=_ctx,+ to=request.to,+ parent=request.parent,+ reason=request.reason,+ instruction=request.instruction,+ wait=request.wait,+ timeout=request.timeout,+ **kwargs,+ ),)- return Response(status="ok", result=result)+ return Response(+ status="ok",+ result=execution.result,+ telemetry=execution.telemetry,+ ).model_dump(exclude_none=True)@router.post("/skills")@@ -141,15 +151,22 @@ async def add_skill():"""Add skill to OpenViking."""service = get_service()-data = request.dataif request.temp_path:data = request.temp_path- result = await service.resources.add_skill(- data=data,- ctx=_ctx,- wait=request.wait,- timeout=request.timeout,+ execution = await run_operation(+ operation="resources.add_skill",+ telemetry=request.telemetry,+ fn=lambda: service.resources.add_skill(+ data=data,+ ctx=_ctx,+ wait=request.wait,+ timeout=request.timeout,+ ),)- return Response(status="ok", result=result)+ return Response(+ status="ok",+ result=execution.result,+ telemetry=execution.telemetry,+ ).model_dump(exclude_none=True)diff --git a/openviking/server/routers/search.py b/openviking/server/routers/search.pyindex 73d7309da..efde8dc02 100644--- a/openviking/server/routers/search.py+++ b/openviking/server/routers/search.py@@ -11,6 +11,8 @@ from openviking.server.auth import get_request_contextfrom openviking.server.dependencies import get_servicefrom openviking.server.identity import RequestContextfrom openviking.server.models import Response+from openviking.server.telemetry import run_operation+from openviking.telemetry import TelemetryRequestrouter = APIRouter(prefix="/api/v1/search", tags=["search"])@@ -24,6 +26,7 @@ class FindRequest(BaseModel):node_limit: Optional[int] = Nonescore_threshold: Optional[float] = Nonefilter: Optional[Dict[str, Any]] = None+ telemetry: TelemetryRequest = Falseclass SearchRequest(BaseModel):@@ -36,6 +39,7 @@ class SearchRequest(BaseModel):node_limit: Optional[int] = Nonescore_threshold: Optional[float] = Nonefilter: Optional[Dict[str, Any]] = None+ telemetry: TelemetryRequest = Falseclass GrepRequest(BaseModel):@@ -63,18 +67,26 @@ async def find("""Semantic search without session context."""service = get_service()actual_limit = request.node_limit if request.node_limit is not None else request.limit- result = await service.search.find(- query=request.query,- ctx=_ctx,- target_uri=request.target_uri,- limit=actual_limit,- score_threshold=request.score_threshold,- filter=request.filter,+ execution = await run_operation(+ operation="search.find",+ telemetry=request.telemetry,+ fn=lambda: service.search.find(+ query=request.query,+ ctx=_ctx,+ target_uri=request.target_uri,+ limit=actual_limit,+ score_threshold=request.score_threshold,+ filter=request.filter,+ ),)- # Convert FindResult to dict if it has to_dict method+ result = execution.resultif hasattr(result, "to_dict"):result = result.to_dict()- return Response(status="ok", result=result)+ return Response(+ status="ok",+ result=result,+ telemetry=execution.telemetry,+ ).model_dump(exclude_none=True)@router.post("/search")@@ -85,26 +97,35 @@ async def search("""Semantic search with optional session context."""service = get_service()- # Get session if session_id provided- session = None- if request.session_id:- session = service.sessions.session(_ctx, request.session_id)- await session.load()-- actual_limit = request.node_limit if request.node_limit is not None else request.limit- result = await service.search.search(- query=request.query,- ctx=_ctx,- target_uri=request.target_uri,- session=session,- limit=actual_limit,- score_threshold=request.score_threshold,- filter=request.filter,+ async def _search():+ session = None+ if request.session_id:+ session = service.sessions.session(_ctx, request.session_id)+ await session.load()+ actual_limit = request.node_limit if request.node_limit is not None else request.limit+ return await service.search.search(+ query=request.query,+ ctx=_ctx,+ target_uri=request.target_uri,+ session=session,+ limit=actual_limit,+ score_threshold=request.score_threshold,+ filter=request.filter,+ )++ execution = await run_operation(+ operation="search.search",+ telemetry=request.telemetry,+ fn=_search,)- # Convert FindResult to dict if it has to_dict method+ result = execution.resultif hasattr(result, "to_dict"):result = result.to_dict()- return Response(status="ok", result=result)+ return Response(+ status="ok",+ result=result,+ telemetry=execution.telemetry,+ ).model_dump(exclude_none=True)@router.post("/grep")diff --git a/openviking/server/routers/sessions.py b/openviking/server/routers/sessions.pyindex 871932e57..0c27e45a6 100644--- a/openviking/server/routers/sessions.py+++ b/openviking/server/routers/sessions.py@@ -6,7 +6,7 @@ import asyncioimport loggingfrom typing import Any, Dict, List, Literal, Optional-from fastapi import APIRouter, Depends, Path, Query+from fastapi import APIRouter, Body, Depends, Path, Queryfrom pydantic import BaseModel, model_validatorfrom openviking.message.part import TextPart, part_from_dict@@ -14,7 +14,10 @@ from openviking.server.auth import get_request_contextfrom openviking.server.dependencies import get_servicefrom openviking.server.identity import RequestContextfrom openviking.server.models import ErrorInfo, Response+from openviking.server.telemetry import resolve_selection, run_operationfrom openviking.service.task_tracker import get_task_tracker+from openviking.telemetry import TelemetryRequest+from openviking_cli.exceptions import InvalidArgumentErrorrouter = APIRouter(prefix="/api/v1/sessions", tags=["sessions"])logger = logging.getLogger(__name__)@@ -73,6 +76,12 @@ class AddMessageRequest(BaseModel):return self+class CommitSessionRequest(BaseModel):+ """Request model for session commit."""++ telemetry: TelemetryRequest = False++def _to_jsonable(value: Any) -> Any:"""Convert internal objects (e.g. Context) into JSON-serializable values."""to_dict = getattr(value, "to_dict", None)@@ -144,6 +153,7 @@ async def delete_session(@router.post("/{session_id}/commit")async def commit_session(+ request: CommitSessionRequest = Body(default_factory=CommitSessionRequest),session_id: str = Path(..., description="Session ID"),wait: bool = Query(True,@@ -173,8 +183,20 @@ async def commit_session(message=f"Session {session_id} already has a commit in progress",),)- result = await service.sessions.commit_async(session_id, _ctx)- return Response(status="ok", result=result)+ execution = await run_operation(+ operation="session.commit",+ telemetry=request.telemetry,+ fn=lambda: service.sessions.commit_async(session_id, _ctx),+ )+ return Response(+ status="ok",+ result=execution.result,+ telemetry=execution.telemetry,+ ).model_dump(exclude_none=True)++ selection = resolve_selection(request.telemetry)+ if selection.include_payload:+ raise InvalidArgumentError("telemetry is not supported when wait=false for session.commit")# Atomically check + create to prevent race conditionstask = tracker.create_if_no_running("session_commit", session_id)diff --git a/openviking/server/telemetry.py b/openviking/server/telemetry.pynew file mode 100644index 000000000..196f09ebd--- /dev/null+++ b/openviking/server/telemetry.py@@ -0,0 +1,33 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: Apache-2.0+"""HTTP router helpers for operation telemetry."""++from __future__ import annotations++from typing import Any, Awaitable, Callable++from openviking.telemetry import TelemetryRequest, TelemetrySelection+from openviking.telemetry.execution import (+ TelemetryExecutionResult,+ parse_telemetry_selection,+ run_with_telemetry,+)+++def resolve_selection(telemetry: TelemetryRequest) -> TelemetrySelection:+ """Validate a router telemetry request without starting execution."""+ return parse_telemetry_selection(telemetry)+++async def run_operation(+ *,+ operation: str,+ telemetry: TelemetryRequest,+ fn: Callable[[], Awaitable[Any]],+) -> TelemetryExecutionResult[Any]:+ """Execute a router operation with request-scoped telemetry."""+ return await run_with_telemetry(+ operation=operation,+ telemetry=telemetry,+ fn=fn,+ )diff --git a/openviking/service/resource_service.py b/openviking/service/resource_service.pyindex 36cf80f8f..44b0b6b82 100644--- a/openviking/service/resource_service.py+++ b/openviking/service/resource_service.py@@ -6,12 +6,20 @@ Resource Service for OpenViking.Provides resource management operations: add_resource, add_skill, wait_processed."""+import timefrom typing import Any, Dict, List, Optionalfrom openviking.server.identity import RequestContextfrom openviking.storage import VikingDBManagerfrom openviking.storage.queuefs import get_queue_managerfrom openviking.storage.viking_fs import VikingFS+from openviking.telemetry import get_current_telemetry+from openviking.telemetry.resource_summary import (+ build_queue_status_payload,+ record_resource_wait_metrics,+ register_wait_telemetry,+ unregister_wait_telemetry,+)from openviking.utils.resource_processor import ResourceProcessorfrom openviking.utils.skill_processor import SkillProcessorfrom openviking_cli.exceptions import (@@ -93,50 +101,72 @@ class ResourceService:Processing result"""self._ensure_initialized()+ request_start = time.perf_counter()+ telemetry_id = register_wait_telemetry(wait)- # add_resource only supports resources scope- if to and to.startswith("viking://"):- parsed = VikingURI(to)- if parsed.scope != "resources":- raise InvalidArgumentError(- f"add_resource only supports resources scope, use dedicated interface to add {parsed.scope} content"- )- if parent and parent.startswith("viking://"):- parsed = VikingURI(parent)- if parsed.scope != "resources":- raise InvalidArgumentError(- f"add_resource only supports resources scope, use dedicated interface to add {parsed.scope} content"- )+ try:+ # add_resource only supports resources scope+ if to and to.startswith("viking://"):+ parsed = VikingURI(to)+ if parsed.scope != "resources":+ raise InvalidArgumentError(+ f"add_resource only supports resources scope, use dedicated interface to add {parsed.scope} content"+ )+ if parent and parent.startswith("viking://"):+ parsed = VikingURI(parent)+ if parsed.scope != "resources":+ raise InvalidArgumentError(+ f"add_resource only supports resources scope, use dedicated interface to add {parsed.scope} content"+ )- result = await self._resource_processor.process_resource(- path=path,- ctx=ctx,- reason=reason,- instruction=instruction,- scope="resources",- to=to,- parent=parent,- build_index=build_index,- summarize=summarize,- **kwargs,- )+ result = await self._resource_processor.process_resource(+ path=path,+ ctx=ctx,+ reason=reason,+ instruction=instruction,+ scope="resources",+ to=to,+ parent=parent,+ build_index=build_index,+ summarize=summarize,+ **kwargs,+ )- if wait:- qm = get_queue_manager()- try:- status = await qm.wait_complete(timeout=timeout)- except TimeoutError as exc:- raise DeadlineExceededError("queue processing", timeout) from exc- result["queue_status"] = {- name: {- "processed": s.processed,- "error_count": s.error_count,- "errors": [{"message": e.message} for e in s.errors],- }- for name, s in status.items()- }+ if wait:+ qm = get_queue_manager()+ wait_start = time.perf_counter()+ try:+ status = await qm.wait_complete(timeout=timeout)+ except TimeoutError as exc:+ get_current_telemetry().set_error(+ "resource_service.wait_complete",+ "DEADLINE_EXCEEDED",+ str(exc),+ )+ raise DeadlineExceededError("queue processing", timeout) from exc+ queue_wait_duration_ms = round((time.perf_counter() - wait_start) * 1000, 3)+ result["queue_status"] = build_queue_status_payload(status)+ record_resource_wait_metrics(+ telemetry_id=telemetry_id,+ queue_status=status,+ root_uri=result.get("root_uri"),+ )+ get_current_telemetry().set("queue.wait.duration_ms", queue_wait_duration_ms)- return result+ get_current_telemetry().set(+ "resource.request.duration_ms",+ round((time.perf_counter() - request_start) * 1000, 3),+ )+ return result+ except Exception as exc:+ get_current_telemetry().set_error(+ "resource_service.add_resource",+ type(exc).__name__,+ str(exc),+ )+ raise+ finally:+ unregister_wait_telemetry(telemetry_id)async def add_skill(self,@@ -165,18 +195,21 @@ class ResourceService:if wait:qm = get_queue_manager()+ wait_start = time.perf_counter()try:status = await qm.wait_complete(timeout=timeout)except TimeoutError as exc:+ get_current_telemetry().set_error(+ "resource_service.wait_complete",+ "DEADLINE_EXCEEDED",+ str(exc),+ )raise DeadlineExceededError("queue processing", timeout) from exc- result["queue_status"] = {- name: {- "processed": s.processed,- "error_count": s.error_count,- "errors": [{"message": e.message} for e in s.errors],- }- for name, s in status.items()- }+ get_current_telemetry().set(+ "queue.wait.duration_ms",+ round((time.perf_counter() - wait_start) * 1000, 3),+ )+ result["queue_status"] = build_queue_status_payload(status)return resultdiff --git a/openviking/service/search_service.py b/openviking/service/search_service.pyindex 7dc13ca92..0e6744414 100644--- a/openviking/service/search_service.py+++ b/openviking/service/search_service.py@@ -64,7 +64,7 @@ class SearchService:if session:session_info = await session.get_context_for_search(query)- return await viking_fs.search(+ result = await viking_fs.search(query=query,ctx=ctx,target_uri=target_uri,@@ -73,6 +73,7 @@ class SearchService:score_threshold=score_threshold,filter=filter,)+ return resultasync def find(self,@@ -96,7 +97,7 @@ class SearchService:FindResult"""viking_fs = self._ensure_initialized()- return await viking_fs.find(+ result = await viking_fs.find(query=query,ctx=ctx,target_uri=target_uri,@@ -104,3 +105,4 @@ class SearchService:score_threshold=score_threshold,filter=filter,)+ return resultdiff --git a/openviking/session/memory_deduplicator.py b/openviking/session/memory_deduplicator.pyindex e63166882..b04f37067 100644--- a/openviking/session/memory_deduplicator.py+++ b/openviking/session/memory_deduplicator.py@@ -17,6 +17,7 @@ from openviking.models.embedder.base import EmbedResultfrom openviking.prompts import render_promptfrom openviking.server.identity import RequestContextfrom openviking.storage import VikingDBManager+from openviking.telemetry import get_current_telemetryfrom openviking_cli.utils import get_loggerfrom openviking_cli.utils.config import get_openviking_config@@ -122,6 +123,7 @@ class MemoryDeduplicator:ctx: RequestContext,) -> List[Context]:"""Find similar existing memories using vector search."""+ telemetry = get_current_telemetry()if not self.embedder:return []@@ -156,6 +158,9 @@ class MemoryDeduplicator:limit=5,ctx=ctx,)+ telemetry.count("vector.searches", 1)+ telemetry.count("vector.scored", len(results))+ telemetry.count("vector.scanned", len(results))# Filter by similarity thresholdsimilar = []@@ -173,6 +178,7 @@ class MemoryDeduplicator:result.get("abstract", ""),)if score >= self.SIMILARITY_THRESHOLD:+ telemetry.count("vector.passed", 1)# Reconstruct Context objectcontext = Context.from_dict(result)if context:diff --git a/openviking/session/session.py b/openviking/session/session.pyindex 243069a14..554180272 100644--- a/openviking/session/session.py+++ b/openviking/session/session.py@@ -14,6 +14,7 @@ from uuid import uuid4from openviking.message import Message, Partfrom openviking.server.identity import RequestContext, Role+from openviking.telemetry import get_current_telemetryfrom openviking.utils.time_utils import get_current_timestampfrom openviking_cli.session.user_id import UserIdentifierfrom openviking_cli.utils import get_logger, run_async@@ -229,6 +230,7 @@ class Session:"stats": None,}if not self._messages:+ get_current_telemetry().set("memory.extracted", 0)return result# 1. Archive current messages@@ -270,6 +272,7 @@ class Session:logger.info(f"Extracted {len(memories)} memories")result["memories_extracted"] = len(memories)self._stats.memories_extracted += len(memories)+ get_current_telemetry().set("memory.extracted", len(memories))# 3. Write current messages to AGFSself._write_to_agfs(self._messages)@@ -305,6 +308,7 @@ class Session:"stats": None,}if not self._messages:+ get_current_telemetry().set("memory.extracted", 0)return result# 1. Archive current messages@@ -344,6 +348,7 @@ class Session:logger.info(f"Extracted {len(memories)} memories")result["memories_extracted"] = len(memories)self._stats.memories_extracted += len(memories)+ get_current_telemetry().set("memory.extracted", len(memories))# 3. Write current messages to AGFSawait self._write_to_agfs_async(self._messages)diff --git a/openviking/storage/collection_schemas.py b/openviking/storage/collection_schemas.pyindex 4d1100137..944f121c2 100644--- a/openviking/storage/collection_schemas.py+++ b/openviking/storage/collection_schemas.py@@ -10,7 +10,10 @@ similar to how init_viking_fs encapsulates VikingFS initialization.import asyncioimport hashlibimport json-from typing import Any, Dict, Optional+import threading+from contextlib import nullcontext+from dataclasses import dataclass+from typing import Any, Dict, List, Optionalfrom openviking.models.embedder.base import EmbedResultfrom openviking.models.embedder.volcengine_embedders import is_429_error@@ -19,6 +22,7 @@ from openviking.storage.errors import CollectionNotFoundErrorfrom openviking.storage.queuefs.embedding_msg import EmbeddingMsgfrom openviking.storage.queuefs.named_queue import DequeueHandlerBasefrom openviking.storage.viking_vector_index_backend import VikingVectorIndexBackend+from openviking.telemetry import bind_telemetry, resolve_telemetryfrom openviking_cli.session.user_id import UserIdentifierfrom openviking_cli.utils import get_loggerfrom openviking_cli.utils.config.open_viking_config import OpenVikingConfig@@ -26,6 +30,12 @@ from openviking_cli.utils.config.open_viking_config import OpenVikingConfiglogger = get_logger(__name__)+@dataclass+class RequestQueueStats:+ processed: int = 0+ error_count: int = 0++class CollectionSchemas:"""Centralized collection schema definitions.@@ -130,6 +140,11 @@ class TextEmbeddingHandler(DequeueHandlerBase):Supports both dense and sparse embeddings based on configuration."""+ _request_stats_lock = threading.Lock()+ _request_stats_by_telemetry_id: Dict[str, RequestQueueStats] = {}+ _request_stats_order: List[str] = []+ _max_cached_stats = 1024+def __init__(self, vikingdb: VikingVectorIndexBackend):"""Initialize the text embedding handler.@@ -149,6 +164,32 @@ class TextEmbeddingHandler(DequeueHandlerBase):"""Initialize the embedder instance from config."""self._embedder = config.embedding.get_embedder()+ @classmethod+ def _merge_request_stats(+ cls, telemetry_id: str, processed: int = 0, error_count: int = 0+ ) -> None:+ if not telemetry_id:+ return+ with cls._request_stats_lock:+ stats = cls._request_stats_by_telemetry_id.setdefault(telemetry_id, RequestQueueStats())+ stats.processed += processed+ stats.error_count += error_count+ cls._request_stats_order.append(telemetry_id)+ if len(cls._request_stats_order) > cls._max_cached_stats:+ old_telemetry_id = cls._request_stats_order.pop(0)+ if (+ old_telemetry_id != telemetry_id+ and old_telemetry_id in cls._request_stats_by_telemetry_id+ ):+ cls._request_stats_by_telemetry_id.pop(old_telemetry_id, None)++ @classmethod+ def consume_request_stats(cls, telemetry_id: str) -> Optional[RequestQueueStats]:+ if not telemetry_id:+ return None+ with cls._request_stats_lock:+ return cls._request_stats_by_telemetry_id.pop(telemetry_id, None)+@staticmethoddef _seed_uri_for_id(uri: str, level: Any) -> str:"""Build deterministic id seed URI from canonical uri + hierarchy level."""@@ -168,38 +209,63 @@ class TextEmbeddingHandler(DequeueHandlerBase):if not data:return None+ embedding_msg: Optional[EmbeddingMsg] = None+ collector = Nonetry:queue_data = json.loads(data["data"])# Parse EmbeddingMsg from dataembedding_msg = EmbeddingMsg.from_dict(queue_data)inserted_data = embedding_msg.context_data+ collector = resolve_telemetry(embedding_msg.telemetry_id)+ telemetry_ctx = bind_telemetry(collector) if collector is not None else nullcontext()- if self._vikingdb.is_closing:- logger.debug("Skip embedding dequeue during shutdown")- self.report_success()- return None-- # Only process string messages- if not isinstance(embedding_msg.message, str):- logger.debug(f"Skipping non-string message type: {type(embedding_msg.message)}")- self.report_success()- return data-- # Initialize embedder if not already initialized- if not self._embedder:- from openviking_cli.utils.config import get_openviking_config-- config = get_openviking_config()- self._initialize_embedder(config)+ with telemetry_ctx:+ if self._vikingdb.is_closing:+ logger.debug("Skip embedding dequeue during shutdown")+ self._merge_request_stats(embedding_msg.telemetry_id, processed=1)+ self.report_success()+ return None- # Generate embedding vector(s)- if self._embedder:- try:- # embed() is a blocking HTTP call; offload to thread pool to avoid- # blocking the event loop and allow real concurrency.- result: EmbedResult = await asyncio.to_thread(- self._embedder.embed, embedding_msg.message- )+ # Only process string messages+ if not isinstance(embedding_msg.message, str):+ logger.debug(f"Skipping non-string message type: {type(embedding_msg.message)}")+ self._merge_request_stats(embedding_msg.telemetry_id, processed=1)+ self.report_success()+ return data++ # Initialize embedder if not already initialized+ if not self._embedder:+ from openviking_cli.utils.config import get_openviking_config++ config = get_openviking_config()+ self._initialize_embedder(config)++ # Generate embedding vector(s)+ if self._embedder:+ try:+ # embed() is a blocking HTTP call; offload to thread pool to avoid+ # blocking the event loop and allow real concurrency.+ result: EmbedResult = await asyncio.to_thread(+ self._embedder.embed, embedding_msg.message+ )+ except Exception as embed_err:+ error_msg = f"Failed to generate embedding: {embed_err}"+ logger.error(error_msg)++ if is_429_error(embed_err) and self._vikingdb.has_queue_manager:+ try:+ await self._vikingdb.enqueue_embedding_msg(embedding_msg)+ logger.info(+ f"Re-enqueued embedding message after rate limit: {embedding_msg.id}"+ )+ self.report_success()+ return None+ except Exception as requeue_err:+ logger.error(f"Failed to re-enqueue message: {requeue_err}")++ self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)+ self.report_error(error_msg, data)+ return None# Add dense vectorif result.dense_vector:@@ -208,6 +274,7 @@ class TextEmbeddingHandler(DequeueHandlerBase):if len(result.dense_vector) != self._vector_dim:error_msg = f"Dense vector dimension mismatch: expected {self._vector_dim}, got {len(result.dense_vector)}"logger.error(error_msg)+ self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)self.report_error(error_msg, data)return None@@ -217,81 +284,69 @@ class TextEmbeddingHandler(DequeueHandlerBase):logger.debug(f"Generated sparse vector with {len(result.sparse_vector)} terms")- except Exception as e:- error_msg = f"Failed to generate embedding: {e}"- logger.error(error_msg)-- if is_429_error(e) and self._vikingdb.has_queue_manager:- try:- await self._vikingdb.enqueue_embedding_msg(embedding_msg)- logger.info(f"Re-enqueued embedding message: {embedding_msg.id}")- self.report_success()- return None- except Exception as requeue_err:- logger.error(f"Failed to re-enqueue message: {requeue_err}")-+ else:+ error_msg = "Embedder not initialized, skipping vector generation"+ logger.warning(error_msg)+ self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)self.report_error(error_msg, data)return None- else:- error_msg = "Embedder not initialized, skipping vector generation"- logger.warning(error_msg)- self.report_error(error_msg, data)- return None-- # Write to vector database- try:- # Ensure vector DB has deterministic IDs per semantic layer.- uri = inserted_data.get("uri")- account_id = inserted_data.get("account_id", "default")- logger.debug(- f"[TextEmbeddingHandler] Preparing to upsert, uri={uri}, account_id={account_id}, inserted_data={inserted_data}"- )-- if uri:- seed_uri = self._seed_uri_for_id(uri, inserted_data.get("level", 2))- id_seed = f"{account_id}:{seed_uri}"- inserted_data["id"] = hashlib.md5(id_seed.encode("utf-8")).hexdigest()-- # Create RequestContext from account_id- user = UserIdentifier(account_id=account_id, user_id="default", agent_id="default")- ctx = RequestContext(user=user, role=Role.ROOT)- logger.debug(- f"[TextEmbeddingHandler] Created ctx for upsert, ctx.account_id={ctx.account_id}, ctx.user={ctx.user}"- )-- record_id = await self._vikingdb.upsert(inserted_data, ctx=ctx)- if record_id:- logger.debug(- f"Successfully wrote embedding to database: {record_id} abstract {inserted_data['abstract']} vector {inserted_data['vector'][:5]}"++ # Write to vector database+ try:+ # Ensure vector DB has deterministic IDs per semantic layer.+ uri = inserted_data.get("uri")+ account_id = inserted_data.get("account_id", "default")+ if uri:+ seed_uri = self._seed_uri_for_id(uri, inserted_data.get("level", 2))+ id_seed = f"{account_id}:{seed_uri}"+ inserted_data["id"] = hashlib.md5(id_seed.encode("utf-8")).hexdigest()++ user = UserIdentifier(+ account_id=account_id,+ user_id="default",+ agent_id="default",)- except CollectionNotFoundError as db_err:- # During shutdown, queue workers may finish one dequeued item.- if self._vikingdb.is_closing:- logger.debug(f"Skip embedding write during shutdown: {db_err}")- self.report_success()+ ctx = RequestContext(user=user, role=Role.ROOT)+ record_id = await self._vikingdb.upsert(inserted_data, ctx=ctx)+ if record_id:+ logger.debug(+ f"Successfully wrote embedding to database: {record_id} abstract {inserted_data['abstract']} vector {inserted_data['vector'][:5]}"+ )+ except CollectionNotFoundError as db_err:+ # During shutdown, queue workers may finish one dequeued item.+ if self._vikingdb.is_closing:+ logger.debug(f"Skip embedding write during shutdown: {db_err}")+ self._merge_request_stats(embedding_msg.telemetry_id, processed=1)+ self.report_success()+ return None+ logger.error(f"Failed to write to vector database: {db_err}")+ self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)+ self.report_error(str(db_err), data)return None- logger.error(f"Failed to write to vector database: {db_err}")- self.report_error(str(db_err), data)- return None- except Exception as db_err:- if self._vikingdb.is_closing:- logger.debug(f"Skip embedding write during shutdown: {db_err}")- self.report_success()+ except Exception as db_err:+ if self._vikingdb.is_closing:+ logger.debug(f"Skip embedding write during shutdown: {db_err}")+ self._merge_request_stats(embedding_msg.telemetry_id, processed=1)+ self.report_success()+ return None+ logger.error(f"Failed to write to vector database: {db_err}")+ import traceback++ traceback.print_exc()+ self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)+ self.report_error(str(db_err), data)return None- logger.error(f"Failed to write to vector database: {db_err}")- import traceback-- traceback.print_exc()- self.report_error(str(db_err), data)- return None- self.report_success()- return inserted_data+ self._merge_request_stats(embedding_msg.telemetry_id, processed=1)+ self.report_success()+ return inserted_dataexcept Exception as e:logger.error(f"Error processing embedding message: {e}")import tracebacktraceback.print_exc()+ if embedding_msg is not None:+ self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)self.report_error(str(e), data)return Nonediff --git a/openviking/storage/queuefs/embedding_msg.py b/openviking/storage/queuefs/embedding_msg.pyindex 19b8381ea..c100d569f 100644--- a/openviking/storage/queuefs/embedding_msg.py+++ b/openviking/storage/queuefs/embedding_msg.py@@ -10,11 +10,18 @@ from uuid import uuid4class EmbeddingMsg:message: Union[str, List[Dict[str, Any]]]context_data: Dict[str, Any]+ telemetry_id: str = ""- def __init__(self, message: Union[str, List[Dict[str, Any]]], context_data: Dict[str, Any]):+ def __init__(+ self,+ message: Union[str, List[Dict[str, Any]]],+ context_data: Dict[str, Any],+ telemetry_id: str = "",+ ):self.id = str(uuid4())self.message = messageself.context_data = context_data+ self.telemetry_id = telemetry_iddef to_dict(self) -> Dict[str, Any]:"""Convert embedding message to dictionary format."""@@ -30,6 +37,7 @@ class EmbeddingMsg:obj = EmbeddingMsg(message=data["message"],context_data=data["context_data"],+ telemetry_id=data.get("telemetry_id", ""),)obj.id = data.get("id", obj.id)return objdiff --git a/openviking/storage/queuefs/embedding_msg_converter.py b/openviking/storage/queuefs/embedding_msg_converter.pyindex b174b9411..133d0de6b 100644--- a/openviking/storage/queuefs/embedding_msg_converter.py+++ b/openviking/storage/queuefs/embedding_msg_converter.py@@ -9,6 +9,7 @@ to EmbeddingMsg objects for asynchronous vector processing.from openviking.core.context import Context, ContextLevelfrom openviking.storage.queuefs.embedding_msg import EmbeddingMsg+from openviking.telemetry import get_current_telemetryfrom openviking_cli.utils import get_loggerlogger = get_logger(__name__)@@ -71,5 +72,6 @@ class EmbeddingMsgConverter:embedding_msg = EmbeddingMsg(message=vectorization_text,context_data=context_data,+ telemetry_id=get_current_telemetry().telemetry_id,)return embedding_msgdiff --git a/openviking/storage/queuefs/semantic_msg.py b/openviking/storage/queuefs/semantic_msg.pyindex 5f7bd7304..59c4ba94e 100644--- a/openviking/storage/queuefs/semantic_msg.py+++ b/openviking/storage/queuefs/semantic_msg.py@@ -37,6 +37,7 @@ class SemanticMsg:role: str = "root"# Additional flagsskip_vectorization: bool = False+ telemetry_id: str = ""def __init__(self,@@ -48,6 +49,7 @@ class SemanticMsg:agent_id: str = "default",role: str = "root",skip_vectorization: bool = False,+ telemetry_id: str = "",):self.id = str(uuid4())self.uri = uri@@ -58,6 +60,7 @@ class SemanticMsg:self.agent_id = agent_idself.role = roleself.skip_vectorization = skip_vectorization+ self.telemetry_id = telemetry_iddef to_dict(self) -> Dict[str, Any]:"""Convert object to dictionary."""@@ -93,6 +96,7 @@ class SemanticMsg:agent_id=data.get("agent_id", "default"),role=data.get("role", "root"),skip_vectorization=data.get("skip_vectorization", False),+ telemetry_id=data.get("telemetry_id", ""),)if "id" in data and data["id"]:obj.id = data["id"]diff --git a/openviking/storage/queuefs/semantic_processor.py b/openviking/storage/queuefs/semantic_processor.pyindex 597007831..2e861bcc3 100644--- a/openviking/storage/queuefs/semantic_processor.py+++ b/openviking/storage/queuefs/semantic_processor.py@@ -3,6 +3,9 @@"""SemanticProcessor: Processes messages from SemanticQueue, generates .abstract.md and .overview.md."""import asyncio+import threading+from contextlib import nullcontext+from dataclasses import dataclassfrom typing import Any, Dict, List, Optional, Tuplefrom openviking.parse.parsers.constants import (@@ -24,6 +27,7 @@ from openviking.storage.queuefs.named_queue import DequeueHandlerBasefrom openviking.storage.queuefs.semantic_dag import DagStats, SemanticDagExecutorfrom openviking.storage.queuefs.semantic_msg import SemanticMsgfrom openviking.storage.viking_fs import get_viking_fs+from openviking.telemetry import bind_telemetry, resolve_telemetryfrom openviking_cli.session.user_id import UserIdentifierfrom openviking_cli.utils import VikingURIfrom openviking_cli.utils.config import get_openviking_config@@ -32,6 +36,12 @@ from openviking_cli.utils.logger import get_loggerlogger = get_logger(__name__)+@dataclass+class RequestQueueStats:+ processed: int = 0+ error_count: int = 0++class SemanticProcessor(DequeueHandlerBase):"""Semantic processor, generates .abstract.md and .overview.md bottom-up.@@ -43,6 +53,14 @@ class SemanticProcessor(DequeueHandlerBase):4. Enqueue to EmbeddingQueue for vectorization"""+ _stats_lock = threading.Lock()+ _dag_stats_by_telemetry_id: Dict[str, DagStats] = {}+ _dag_stats_by_uri: Dict[str, DagStats] = {}+ _dag_stats_order: List[Tuple[str, str]] = []+ _request_stats_by_telemetry_id: Dict[str, RequestQueueStats] = {}+ _request_stats_order: List[str] = []+ _max_cached_stats = 256+def __init__(self, max_concurrent_llm: int = 100):"""Initialize SemanticProcessor.@@ -55,6 +73,61 @@ class SemanticProcessor(DequeueHandlerBase):self._current_ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)self._current_msg: Optional[SemanticMsg] = None+ @classmethod+ def _cache_dag_stats(cls, telemetry_id: str, uri: str, stats: DagStats) -> None:+ with cls._stats_lock:+ if telemetry_id:+ cls._dag_stats_by_telemetry_id[telemetry_id] = stats+ cls._dag_stats_by_uri[uri] = stats+ cls._dag_stats_order.append((telemetry_id, uri))+ if len(cls._dag_stats_order) > cls._max_cached_stats:+ old_telemetry_id, old_uri = cls._dag_stats_order.pop(0)+ if old_telemetry_id:+ cls._dag_stats_by_telemetry_id.pop(old_telemetry_id, None)+ cls._dag_stats_by_uri.pop(old_uri, None)++ @classmethod+ def consume_dag_stats(+ cls,+ telemetry_id: str = "",+ uri: Optional[str] = None,+ ) -> Optional[DagStats]:+ with cls._stats_lock:+ if telemetry_id and telemetry_id in cls._dag_stats_by_telemetry_id:+ stats = cls._dag_stats_by_telemetry_id.pop(telemetry_id, None)+ if uri:+ cls._dag_stats_by_uri.pop(uri, None)+ return stats+ if uri and uri in cls._dag_stats_by_uri:+ return cls._dag_stats_by_uri.pop(uri, None)+ return None++ @classmethod+ def _merge_request_stats(+ cls,+ telemetry_id: str,+ processed: int = 0,+ error_count: int = 0,+ ) -> None:+ if not telemetry_id:+ return+ with cls._stats_lock:+ stats = cls._request_stats_by_telemetry_id.setdefault(telemetry_id, RequestQueueStats())+ stats.processed += processed+ stats.error_count += error_count+ cls._request_stats_order.append(telemetry_id)+ if len(cls._request_stats_order) > cls._max_cached_stats:+ old_telemetry_id = cls._request_stats_order.pop(0)+ if old_telemetry_id != telemetry_id:+ cls._request_stats_by_telemetry_id.pop(old_telemetry_id, None)++ @classmethod+ def consume_request_stats(cls, telemetry_id: str) -> Optional[RequestQueueStats]:+ if not telemetry_id:+ return None+ with cls._stats_lock:+ return cls._request_stats_by_telemetry_id.pop(telemetry_id, None)+@staticmethoddef _owner_space_for_uri(uri: str, ctx: RequestContext) -> str:"""Derive owner_space from a URI.@@ -152,6 +225,8 @@ class SemanticProcessor(DequeueHandlerBase):async def on_dequeue(self, data: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any]]:"""Process dequeued SemanticMsg, recursively process all subdirectories."""+ msg: Optional[SemanticMsg] = None+ collector = Nonetry:import json@@ -164,61 +239,77 @@ class SemanticProcessor(DequeueHandlerBase):# data is guaranteed to be not None at this pointassert data is not Nonemsg = SemanticMsg.from_dict(data)- self._current_msg = msg- self._current_ctx = self._ctx_from_semantic_msg(msg)- logger.info(- f"Processing semantic generation for: {msg.uri} (recursive={msg.recursive})"- )-- if msg.recursive:- executor = SemanticDagExecutor(- processor=self,- context_type=msg.context_type,- max_concurrent_llm=self.max_concurrent_llm,- ctx=self._current_ctx,+ collector = resolve_telemetry(msg.telemetry_id)+ telemetry_ctx = bind_telemetry(collector) if collector is not None else nullcontext()++ with telemetry_ctx:+ self._current_msg = msg+ self._current_ctx = self._ctx_from_semantic_msg(msg)+ logger.info(+ f"Processing semantic generation for: {msg.uri} (recursive={msg.recursive})")- self._dag_executor = executor- await executor.run(msg.uri)- logger.info(f"Completed semantic generation for: {msg.uri}")- self.report_success()- return None- else:- # Non-recursive processing: directly process this directory- children_uris = []- file_paths = []- # Collect immediate children info only (no recursion)- viking_fs = get_viking_fs()- try:- entries = await viking_fs.ls(msg.uri, ctx=self._current_ctx)- for entry in entries:- name = entry.get("name", "")- if not name or name.startswith(".") or name in [".", ".."]:- continue-- item_uri = VikingURI(msg.uri).join(name).uri-- if entry.get("isDir", False):- children_uris.append(item_uri)- else:- file_paths.append(item_uri)- except Exception as e:- logger.warning(f"Failed to list directory {msg.uri}: {e}")-- # Process this directory- await self._process_single_directory(- uri=msg.uri,- context_type=msg.context_type,- children_uris=children_uris,- file_paths=file_paths,- )+ if msg.recursive:+ executor = SemanticDagExecutor(+ processor=self,+ context_type=msg.context_type,+ max_concurrent_llm=self.max_concurrent_llm,+ ctx=self._current_ctx,+ )+ self._dag_executor = executor+ await executor.run(msg.uri)+ self._cache_dag_stats(+ msg.telemetry_id,+ msg.uri,+ executor.get_stats(),+ )+ else:+ children_uris = []+ file_paths = []++ viking_fs = get_viking_fs()+ try:+ entries = await viking_fs.ls(msg.uri, ctx=self._current_ctx)+ for entry in entries:+ name = entry.get("name", "")+ if not name or name.startswith(".") or name in [".", ".."]:+ continue++ item_uri = VikingURI(msg.uri).join(name).uri++ if entry.get("isDir", False):+ children_uris.append(item_uri)+ else:+ file_paths.append(item_uri)+ except Exception as e:+ logger.warning(f"Failed to list directory {msg.uri}: {e}")++ await self._process_single_directory(+ uri=msg.uri,+ context_type=msg.context_type,+ children_uris=children_uris,+ file_paths=file_paths,+ )+ self._cache_dag_stats(+ msg.telemetry_id,+ msg.uri,+ DagStats(+ total_nodes=1,+ pending_nodes=0,+ in_progress_nodes=0,+ done_nodes=1,+ ),+ )+ self._merge_request_stats(msg.telemetry_id, processed=1)logger.info(f"Completed semantic generation for: {msg.uri}")self.report_success()return Noneexcept Exception as e:logger.error(f"Failed to process semantic message: {e}", exc_info=True)+ if msg is not None:+ self._merge_request_stats(msg.telemetry_id, error_count=1)self.report_error(str(e), data)return Nonefinally:diff --git a/openviking/storage/viking_fs.py b/openviking/storage/viking_fs.pyindex 2449f9b93..62cf15fa7 100644--- a/openviking/storage/viking_fs.py+++ b/openviking/storage/viking_fs.py@@ -24,6 +24,7 @@ from typing import TYPE_CHECKING, Any, Dict, List, Optional, Unionfrom openviking.pyagfs.exceptions import AGFSHTTPErrorfrom openviking.server.identity import RequestContext, Role+from openviking.telemetry import get_current_telemetryfrom openviking.utils.time_utils import format_simplified, get_current_timestamp, parse_iso_datetimefrom openviking_cli.exceptions import NotFoundErrorfrom openviking_cli.session.user_id import UserIdentifier@@ -606,6 +607,7 @@ class VikingFS:Returns:FindResult"""+ telemetry = get_current_telemetry()from openviking.retrieve.hierarchical_retriever import HierarchicalRetrieverfrom openviking_cli.retrieve import (ContextType,@@ -665,11 +667,13 @@ class VikingFS:elif ctx.context_type == ContextType.SKILL:skills.append(ctx)- return FindResult(+ find_result = FindResult(memories=memories,resources=resources,skills=skills,)+ telemetry.set("vector.returned", find_result.total)+ return find_resultasync def search(self,@@ -693,6 +697,7 @@ class VikingFS:Returns:FindResult"""+ telemetry = get_current_telemetry()from openviking.retrieve.hierarchical_retriever import HierarchicalRetrieverfrom openviking.retrieve.intent_analyzer import IntentAnalyzerfrom openviking_cli.retrieve import (@@ -763,6 +768,7 @@ class VikingFS:)for ctx_type in [ContextType.MEMORY, ContextType.RESOURCE, ContextType.SKILL]]+ telemetry.set("search.typed_queries_count", len(typed_queries))# Concurrent executionstorage = self._get_vector_store()@@ -799,13 +805,15 @@ class VikingFS:elif ctx.context_type == ContextType.SKILL:skills.append(ctx)- return FindResult(+ find_result = FindResult(memories=memories,resources=resources,skills=skills,query_plan=query_plan,query_results=query_results,)+ telemetry.set("vector.returned", find_result.total)+ return find_result# ========== Relation Management ==========diff --git a/openviking/sync_client.py b/openviking/sync_client.pyindex 149c29c8d..bb98cf20f 100644--- a/openviking/sync_client.py+++ b/openviking/sync_client.py@@ -12,6 +12,7 @@ if TYPE_CHECKING:from openviking.session import Sessionfrom openviking.async_client import AsyncOpenViking+from openviking.telemetry import TelemetryRequestfrom openviking_cli.utils import run_async@@ -73,9 +74,11 @@ class SyncOpenViking:"""return run_async(self._async_client.add_message(session_id, role, content, parts))- def commit_session(self, session_id: str) -> Dict[str, Any]:+ def commit_session(+ self, session_id: str, telemetry: TelemetryRequest = False+ ) -> Dict[str, Any]:"""Commit a session (archive and extract memories)."""- return run_async(self._async_client.commit_session(session_id))+ return run_async(self._async_client.commit_session(session_id, telemetry=telemetry))def add_resource(self,@@ -88,6 +91,7 @@ class SyncOpenViking:timeout: float = None,build_index: bool = True,summarize: bool = False,+ telemetry: TelemetryRequest = False,**kwargs,) -> Dict[str, Any]:"""Add resource to OpenViking (resources scope only)@@ -98,6 +102,8 @@ class SyncOpenViking:**kwargs: Extra options forwarded to the parser chain, e.g.``strict``, ``ignore_dirs``, ``include``, ``exclude``."""+ if to and parent:+ raise ValueError("Cannot specify both 'to' and 'parent' at the same time.")return run_async(self._async_client.add_resource(path=path,@@ -109,6 +115,7 @@ class SyncOpenViking:timeout=timeout,build_index=build_index,summarize=summarize,+ telemetry=telemetry,**kwargs,))@@ -118,9 +125,12 @@ class SyncOpenViking:data: Any,wait: bool = False,timeout: float = None,+ telemetry: TelemetryRequest = False,) -> Dict[str, Any]:"""Add skill to OpenViking."""- return run_async(self._async_client.add_skill(data, wait=wait, timeout=timeout))+ return run_async(+ self._async_client.add_skill(data, wait=wait, timeout=timeout, telemetry=telemetry)+ )def search(self,@@ -131,11 +141,12 @@ class SyncOpenViking:limit: int = 10,score_threshold: Optional[float] = None,filter: Optional[Dict] = None,+ telemetry: TelemetryRequest = False,):"""Execute complex retrieval (intent analysis, hierarchical retrieval)."""return run_async(self._async_client.search(- query, target_uri, session, session_id, limit, score_threshold, filter+ query, target_uri, session, session_id, limit, score_threshold, filter, telemetry))@@ -145,9 +156,20 @@ class SyncOpenViking:target_uri: str = "",limit: int = 10,score_threshold: Optional[float] = None,+ filter: Optional[Dict] = None,+ telemetry: TelemetryRequest = False,):"""Quick retrieval"""- return run_async(self._async_client.find(query, target_uri, limit, score_threshold))+ return run_async(+ self._async_client.find(+ query,+ target_uri,+ limit,+ score_threshold,+ filter,+ telemetry,+ )+ )def abstract(self, uri: str) -> str:"""Read L0 abstract"""diff --git a/openviking/telemetry/__init__.py b/openviking/telemetry/__init__.pynew file mode 100644index 000000000..985beda4d--- /dev/null+++ b/openviking/telemetry/__init__.py@@ -0,0 +1,24 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: Apache-2.0+"""OpenViking telemetry runtime and operation telemetry helpers."""++from .context import bind_telemetry, get_current_telemetry+from .operation import OperationTelemetry, TelemetrySnapshot+from .registry import register_telemetry, resolve_telemetry, unregister_telemetry+from .request import TelemetryRequest, TelemetrySelection, normalize_telemetry_request+from .runtime import get_telemetry_runtime, set_telemetry_runtime++__all__ = [+ "OperationTelemetry",+ "TelemetryRequest",+ "TelemetrySelection",+ "TelemetrySnapshot",+ "bind_telemetry",+ "get_current_telemetry",+ "get_telemetry_runtime",+ "normalize_telemetry_request",+ "register_telemetry",+ "resolve_telemetry",+ "set_telemetry_runtime",+ "unregister_telemetry",+]diff --git a/openviking/telemetry/backends/__init__.py b/openviking/telemetry/backends/__init__.pynew file mode 100644index 000000000..2e8ec32ee--- /dev/null+++ b/openviking/telemetry/backends/__init__.py@@ -0,0 +1,7 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: Apache-2.0+"""Telemetry backend exports."""++from .memory import MemoryOperationTelemetry++__all__ = ["MemoryOperationTelemetry"]diff --git a/openviking/telemetry/backends/memory.py b/openviking/telemetry/backends/memory.pynew file mode 100644index 000000000..93c133f81--- /dev/null+++ b/openviking/telemetry/backends/memory.py@@ -0,0 +1,12 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: Apache-2.0+"""Memory telemetry backend."""++from openviking.telemetry.operation import OperationTelemetry+++class MemoryOperationTelemetry(OperationTelemetry):+ """In-process operation telemetry collector."""+++__all__ = ["MemoryOperationTelemetry"]diff --git a/openviking/telemetry/context.py b/openviking/telemetry/context.pynew file mode 100644index 000000000..1384c95bc--- /dev/null+++ b/openviking/telemetry/context.py@@ -0,0 +1,35 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: Apache-2.0+"""Telemetry context helpers."""++from __future__ import annotations++import contextvars+from contextlib import contextmanager+from typing import Iterator++from .operation import OperationTelemetry++_NOOP_TELEMETRY = OperationTelemetry(operation="noop", enabled=False)+_CURRENT_TELEMETRY: contextvars.ContextVar[OperationTelemetry] = contextvars.ContextVar(+ "openviking_operation_telemetry",+ default=_NOOP_TELEMETRY,+)+++def get_current_telemetry() -> OperationTelemetry:+ """Get current operation telemetry or disabled no-op collector."""+ return _CURRENT_TELEMETRY.get()+++@contextmanager+def bind_telemetry(handle: OperationTelemetry) -> Iterator[OperationTelemetry]:+ """Bind operation telemetry to current context."""+ token = _CURRENT_TELEMETRY.set(handle)+ try:+ yield handle+ finally:+ _CURRENT_TELEMETRY.reset(token)+++__all__ = ["bind_telemetry", "get_current_telemetry"]diff --git a/openviking/telemetry/execution.py b/openviking/telemetry/execution.pynew file mode 100644index 000000000..32bf39dc4--- /dev/null+++ b/openviking/telemetry/execution.py@@ -0,0 +1,115 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: Apache-2.0+"""Shared helpers for telemetry-wrapped operation execution."""++from __future__ import annotations++from dataclasses import dataclass+from typing import Any, Awaitable, Callable, Generic, Optional, TypeVar++from openviking_cli.exceptions import InvalidArgumentError++from .context import bind_telemetry+from .operation import OperationTelemetry+from .request import TelemetryRequest, TelemetrySelection, normalize_telemetry_request++T = TypeVar("T")+++@dataclass+class TelemetryExecutionResult(Generic[T]):+ """Executed operation result plus telemetry payloads."""++ result: T+ telemetry: Optional[dict[str, Any]]+ selection: TelemetrySelection+++def parse_telemetry_selection(telemetry: TelemetryRequest) -> TelemetrySelection:+ """Validate and normalize a telemetry request for public API usage."""+ try:+ return normalize_telemetry_request(telemetry)+ except ValueError as exc:+ raise InvalidArgumentError(str(exc)) from exc+++def build_telemetry_payload(+ collector: OperationTelemetry,+ selection: TelemetrySelection,+ *,+ status: str = "ok",+) -> dict[str, Any] | None:+ """Build a telemetry payload from a finished collector."""+ snapshot = collector.finish(status=status)+ if snapshot is None:+ return None++ if not selection.include_payload:+ return None++ return snapshot.to_dict(+ include_summary=selection.include_summary,+ )+++def attach_telemetry_payload(+ result: Any,+ telemetry_payload: Optional[dict[str, Any]],+) -> Any:+ """Attach a telemetry payload to a dict result."""+ if telemetry_payload is None:+ return result++ if result is None:+ payload: dict[str, Any] = {}+ payload["telemetry"] = telemetry_payload+ return payload++ if isinstance(result, dict):+ result["telemetry"] = telemetry_payload+ return result++ return result+++async def run_with_telemetry(+ *,+ operation: str,+ telemetry: TelemetryRequest,+ fn: Callable[[], Awaitable[T]],+ error_status: str = "error",+) -> TelemetryExecutionResult[T]:+ """Execute an async operation with a bound operation-scoped collector."""+ selection = parse_telemetry_selection(telemetry)+ collector = OperationTelemetry(+ operation=operation,+ enabled=True,+ )++ try:+ with bind_telemetry(collector):+ result = await fn()+ except Exception as exc:+ collector.set_error(operation, type(exc).__name__, str(exc))+ collector.finish(status=error_status)+ raise++ telemetry_payload = build_telemetry_payload(+ collector,+ selection,+ status="ok",+ )+ return TelemetryExecutionResult(+ result=result,+ telemetry=telemetry_payload,+ selection=selection,+ )+++__all__ = [+ "TelemetryExecutionResult",+ "attach_telemetry_payload",+ "build_telemetry_payload",+ "parse_telemetry_selection",+ "run_with_telemetry",+]diff --git a/openviking/telemetry/operation.py b/openviking/telemetry/operation.pynew file mode 100644index 000000000..82a204918--- /dev/null+++ b/openviking/telemetry/operation.py@@ -0,0 +1,244 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: Apache-2.0+"""Operation-scoped telemetry primitives."""++from __future__ import annotations++import time+from collections import defaultdict+from dataclasses import dataclass+from threading import Lock+from typing import Any, Dict, Optional+from uuid import uuid4+++@dataclass+class TelemetrySnapshot:+ """Final operation telemetry output."""++ telemetry_id: str+ summary: Dict[str, Any]++ def to_usage_dict(self) -> Dict[str, Any]:+ return {+ "duration_ms": self.summary.get("duration_ms", 0),+ "token_total": self.summary.get("tokens", {}).get("total", 0),+ }++ def to_dict(+ self,+ *,+ include_summary: bool = True,+ ) -> Dict[str, Any]:+ payload: Dict[str, Any] = {"id": self.telemetry_id}+ if include_summary:+ payload["summary"] = self.summary+ return payload+++class TelemetrySummaryBuilder:+ """Build normalized summary metrics from collector data."""++ @staticmethod+ def _i(value: Any, default: int = 0) -> int:+ if value is None:+ return default+ try:+ return int(value)+ except (TypeError, ValueError):+ return default++ @classmethod+ def _has_metric_prefix(+ cls, prefix: str, counters: Dict[str, float], gauges: Dict[str, Any]+ ) -> bool:+ needle = f"{prefix}."+ return any(key.startswith(needle) for key in counters) or any(+ key.startswith(needle) for key in gauges+ )++ @classmethod+ def build(+ cls,+ *,+ operation: str,+ status: str,+ duration_ms: float,+ counters: Dict[str, float],+ gauges: Dict[str, Any],+ error_stage: str,+ error_code: str,+ error_message: str,+ ) -> Dict[str, Any]:+ llm_input_tokens = cls._i(counters.get("tokens.llm.input"), 0)+ llm_output_tokens = cls._i(counters.get("tokens.llm.output"), 0)+ llm_total_tokens = cls._i(counters.get("tokens.llm.total"), 0)+ embedding_total_tokens = cls._i(counters.get("tokens.embedding.total"), 0)+ vector_candidates_scored = cls._i(counters.get("vector.scored"), 0)+ vectors_scanned = gauges.get("vector.scanned")+ if vectors_scanned is None:+ vectors_scanned = cls._i(counters.get("vector.scanned"), 0)++ memories_extracted = gauges.get("memory.extracted")+ if memories_extracted is None and counters.get("memory.extracted") is not None:+ memories_extracted = cls._i(counters.get("memory.extracted"), 0)+ summary = {+ "operation": operation,+ "status": status,+ "duration_ms": round(float(duration_ms), 3),+ "tokens": {+ "total": cls._i(counters.get("tokens.total"), 0),+ "llm": {+ "input": llm_input_tokens,+ "output": llm_output_tokens,+ "total": llm_total_tokens,+ },+ "embedding": {"total": embedding_total_tokens},+ },+ }++ if cls._has_metric_prefix("queue", counters, gauges):+ summary["queue"] = {+ "semantic": {+ "processed": cls._i(gauges.get("queue.semantic.processed"), 0),+ "error_count": cls._i(gauges.get("queue.semantic.error_count"), 0),+ },+ "embedding": {+ "processed": cls._i(gauges.get("queue.embedding.processed"), 0),+ "error_count": cls._i(gauges.get("queue.embedding.error_count"), 0),+ },+ }++ if cls._has_metric_prefix("vector", counters, gauges):+ summary["vector"] = {+ "searches": cls._i(counters.get("vector.searches"), 0),+ "scored": vector_candidates_scored,+ "passed": cls._i(counters.get("vector.passed"), 0),+ "returned": cls._i(+ gauges.get("vector.returned", counters.get("vector.returned")), 0+ ),+ "scanned": vectors_scanned,+ "scan_reason": gauges.get("vector.scan_reason", ""),+ }++ if cls._has_metric_prefix("semantic_nodes", counters, gauges):+ summary["semantic_nodes"] = {+ "total": gauges.get("semantic_nodes.total"),+ "done": gauges.get("semantic_nodes.done"),+ "pending": gauges.get("semantic_nodes.pending"),+ "running": gauges.get("semantic_nodes.running"),+ }++ if cls._has_metric_prefix("memory", counters, gauges):+ summary["memory"] = {+ "extracted": memories_extracted,+ }++ if error_stage or error_code or error_message:+ summary["errors"] = {+ "stage": error_stage,+ "error_code": error_code,+ "message": error_message,+ }++ return summary+++class OperationTelemetry:+ """Operation-scoped telemetry collector with low-overhead disabled mode."""++ def __init__(+ self,+ operation: str,+ enabled: bool = False,+ ):+ self.operation = operation+ self.enabled = enabled+ self.telemetry_id = f"tm_{uuid4().hex}" if enabled else ""+ self._start_time = time.perf_counter()+ self._counters: Dict[str, float] = defaultdict(float)+ self._gauges: Dict[str, Any] = {}+ self._error_stage = ""+ self._error_code = ""+ self._error_message = ""+ self._lock = Lock()++ def count(self, key: str, delta: float = 1) -> None:+ if not self.enabled:+ return+ with self._lock:+ self._counters[key] += delta++ def increment(self, key: str, delta: float = 1) -> None:+ self.count(key, delta)++ def set(self, key: str, value: Any) -> None:+ if not self.enabled:+ return+ with self._lock:+ self._gauges[key] = value++ def set_value(self, key: str, value: Any) -> None:+ self.set(key, value)++ def add_token_usage(self, input_tokens: int, output_tokens: int) -> None:+ self.add_token_usage_by_source("llm", input_tokens, output_tokens)++ def record_token_usage(self, source: str, input_tokens: int, output_tokens: int = 0) -> None:+ self.add_token_usage_by_source(source, input_tokens, output_tokens)++ def add_token_usage_by_source(+ self, source: str, input_tokens: int, output_tokens: int = 0+ ) -> None:+ if not self.enabled:+ return++ normalized_input = max(input_tokens, 0)+ normalized_output = max(output_tokens, 0)+ normalized_total = normalized_input + normalized_output++ self.count("tokens.input", normalized_input)+ self.count("tokens.output", normalized_output)+ self.count("tokens.total", normalized_total)+ self.count(f"tokens.{source}.input", normalized_input)+ self.count(f"tokens.{source}.output", normalized_output)+ self.count(f"tokens.{source}.total", normalized_total)++ def set_error(self, stage: str, code: str, message: str) -> None:+ if not self.enabled:+ return+ with self._lock:+ self._error_stage = stage+ self._error_code = code+ self._error_message = message++ def record_error(self, stage: str, code: str, message: str) -> None:+ self.set_error(stage, code, message)++ def finish(self, status: str = "ok") -> Optional[TelemetrySnapshot]:+ if not self.enabled:+ return None++ duration_ms = (time.perf_counter() - self._start_time) * 1000+ with self._lock:+ summary = TelemetrySummaryBuilder.build(+ operation=self.operation,+ status=status,+ duration_ms=duration_ms,+ counters=dict(self._counters),+ gauges=dict(self._gauges),+ error_stage=self._error_stage,+ error_code=self._error_code,+ error_message=self._error_message,+ )+ return TelemetrySnapshot(+ telemetry_id=self.telemetry_id,+ summary=summary,+ )+++__all__ = [+ "OperationTelemetry",+ "TelemetrySnapshot",+ "TelemetrySummaryBuilder",+]diff --git a/openviking/telemetry/registry.py b/openviking/telemetry/registry.pynew file mode 100644index 000000000..8cd7b0186--- /dev/null+++ b/openviking/telemetry/registry.py@@ -0,0 +1,36 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: Apache-2.0+"""Telemetry registry helpers."""++from __future__ import annotations++import threading++from .operation import OperationTelemetry++_REGISTERED_TELEMETRY: dict[str, OperationTelemetry] = {}+_REGISTERED_TELEMETRY_LOCK = threading.Lock()+++def register_telemetry(handle: OperationTelemetry) -> None:+ if not handle.enabled or not handle.telemetry_id:+ return+ with _REGISTERED_TELEMETRY_LOCK:+ _REGISTERED_TELEMETRY[handle.telemetry_id] = handle+++def resolve_telemetry(telemetry_id: str) -> OperationTelemetry | None:+ if not telemetry_id:+ return None+ with _REGISTERED_TELEMETRY_LOCK:+ return _REGISTERED_TELEMETRY.get(telemetry_id)+++def unregister_telemetry(telemetry_id: str) -> None:+ if not telemetry_id:+ return+ with _REGISTERED_TELEMETRY_LOCK:+ _REGISTERED_TELEMETRY.pop(telemetry_id, None)+++__all__ = ["register_telemetry", "resolve_telemetry", "unregister_telemetry"]diff --git a/openviking/telemetry/request.py b/openviking/telemetry/request.pynew file mode 100644index 000000000..87e26c1a7--- /dev/null+++ b/openviking/telemetry/request.py@@ -0,0 +1,49 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: Apache-2.0+"""Telemetry request parsing helpers."""++from __future__ import annotations++from collections.abc import Mapping+from dataclasses import dataclass+from typing import Any, TypeAlias++TelemetryRequest: TypeAlias = bool | dict[str, bool]++_ALLOWED_TELEMETRY_KEYS = frozenset({"summary"})+++@dataclass(frozen=True)+class TelemetrySelection:+ """Normalized telemetry payload selection."""++ include_summary: bool++ @property+ def include_payload(self) -> bool:+ return self.include_summary+++def normalize_telemetry_request(+ request: TelemetryRequest | Mapping[str, Any] | None,+) -> TelemetrySelection:+ """Normalize a telemetry request into explicit response selection flags."""+ if request is None or request is False:+ return TelemetrySelection(include_summary=False)+ if request is True:+ return TelemetrySelection(include_summary=True)+ if not isinstance(request, Mapping):+ raise ValueError("telemetry must be a boolean or an object")++ unknown_keys = set(request) - _ALLOWED_TELEMETRY_KEYS+ if unknown_keys:+ joined = ", ".join(sorted(unknown_keys))+ raise ValueError(f"Unsupported telemetry options: {joined}")++ include_summary = request.get("summary", True)+ if not isinstance(include_summary, bool):+ raise ValueError("telemetry.summary must be a boolean")+ return TelemetrySelection(include_summary=include_summary)+++__all__ = ["TelemetryRequest", "TelemetrySelection", "normalize_telemetry_request"]diff --git a/openviking/telemetry/resource_summary.py b/openviking/telemetry/resource_summary.pynew file mode 100644index 000000000..c578c270d--- /dev/null+++ b/openviking/telemetry/resource_summary.py@@ -0,0 +1,132 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: Apache-2.0+"""Resource-specific telemetry summary helpers."""++from __future__ import annotations++from typing import Any, Dict++from .context import get_current_telemetry+from .operation import OperationTelemetry+from .registry import register_telemetry, unregister_telemetry+++def _consume_semantic_request_stats(telemetry_id: str):+ try:+ from openviking.storage.queuefs.semantic_processor import SemanticProcessor++ return SemanticProcessor.consume_request_stats(telemetry_id)+ except Exception:+ return None+++def _consume_embedding_request_stats(telemetry_id: str):+ try:+ from openviking.storage.collection_schemas import TextEmbeddingHandler++ return TextEmbeddingHandler.consume_request_stats(telemetry_id)+ except Exception:+ return None+++def _consume_semantic_dag_stats(telemetry_id: str, root_uri: str | None):+ try:+ from openviking.storage.queuefs.semantic_processor import SemanticProcessor++ return SemanticProcessor.consume_dag_stats(telemetry_id=telemetry_id, uri=root_uri)+ except Exception:+ return None+++def register_wait_telemetry(wait: bool) -> str:+ """Register current telemetry collector for async queue consumers when needed."""+ handle = get_current_telemetry()+ if not wait or not handle.enabled:+ return ""+ register_telemetry(handle)+ return handle.telemetry_id+++def unregister_wait_telemetry(telemetry_id: str) -> None:+ """Unregister request-scoped telemetry handle."""+ unregister_telemetry(telemetry_id)+++def build_queue_status_payload(status: Dict[str, Any]) -> Dict[str, Dict[str, Any]]:+ """Convert queue status objects to response payload format."""+ return {+ name: {+ "processed": s.processed,+ "error_count": s.error_count,+ "errors": [{"message": e.message} for e in s.errors],+ }+ for name, s in status.items()+ }+++def _resolve_queue_group(+ *,+ explicit_stats: Any,+ fallback_status: Any,+) -> Dict[str, int]:+ if explicit_stats is not None:+ return {+ "processed": explicit_stats.processed,+ "error_count": explicit_stats.error_count,+ }+ if fallback_status is None:+ return {"processed": 0, "error_count": 0}+ return {+ "processed": fallback_status.processed,+ "error_count": fallback_status.error_count,+ }+++def record_resource_wait_metrics(+ *,+ telemetry: OperationTelemetry | None = None,+ telemetry_id: str,+ queue_status: Dict[str, Any],+ root_uri: str | None,+) -> Dict[str, Dict[str, int]]:+ """Apply queue and DAG metrics to a resource operation collector."""+ telemetry = telemetry or get_current_telemetry()+ if not telemetry.enabled:+ return {+ "semantic": {"processed": 0, "error_count": 0},+ "embedding": {"processed": 0, "error_count": 0},+ }++ semantic = _resolve_queue_group(+ explicit_stats=_consume_semantic_request_stats(telemetry_id),+ fallback_status=queue_status.get("Semantic"),+ )+ embedding = _resolve_queue_group(+ explicit_stats=_consume_embedding_request_stats(telemetry_id),+ fallback_status=queue_status.get("Embedding"),+ )++ telemetry.set("queue.semantic.processed", semantic["processed"])+ telemetry.set("queue.semantic.error_count", semantic["error_count"])+ telemetry.set("queue.embedding.processed", embedding["processed"])+ telemetry.set("queue.embedding.error_count", embedding["error_count"])++ dag_stats = _consume_semantic_dag_stats(telemetry_id, root_uri)+ if dag_stats is not None:+ telemetry.set("semantic_nodes.total", dag_stats.total_nodes)+ telemetry.set("semantic_nodes.done", dag_stats.done_nodes)+ telemetry.set("semantic_nodes.pending", dag_stats.pending_nodes)+ telemetry.set("semantic_nodes.running", dag_stats.in_progress_nodes)++ return {+ "semantic": semantic,+ "embedding": embedding,+ }+++__all__ = [+ "build_queue_status_payload",+ "record_resource_wait_metrics",+ "register_wait_telemetry",+ "unregister_wait_telemetry",+]diff --git a/openviking/telemetry/runtime.py b/openviking/telemetry/runtime.pynew file mode 100644index 000000000..f6b6db5df--- /dev/null+++ b/openviking/telemetry/runtime.py@@ -0,0 +1,79 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: Apache-2.0+"""Telemetry runtime entrypoints."""++from __future__ import annotations++from dataclasses import dataclass, field+from threading import Lock+from typing import Any, Dict, Tuple+++class MemoryTelemetryMeter:+ """Lightweight in-process telemetry meter.++ This is intentionally simple for now; it provides the global telemetry hook+ points required by the design without forcing immediate broad adoption.+ """++ def __init__(self):+ self._counters: Dict[Tuple[str, Tuple[Tuple[str, Any], ...]], float] = {}+ self._gauges: Dict[Tuple[str, Tuple[Tuple[str, Any], ...]], Any] = {}+ self._histograms: Dict[Tuple[str, Tuple[Tuple[str, Any], ...]], list[float]] = {}+ self._lock = Lock()++ @staticmethod+ def _key(metric: str, attrs: Dict[str, Any] | None) -> Tuple[str, Tuple[Tuple[str, Any], ...]]:+ normalized = tuple(sorted((attrs or {}).items()))+ return metric, normalized++ def increment(self, metric: str, value: float = 1, attrs: Dict[str, Any] | None = None) -> None:+ key = self._key(metric, attrs)+ with self._lock:+ self._counters[key] = self._counters.get(key, 0) + value++ def record_histogram(+ self, metric: str, value: float, attrs: Dict[str, Any] | None = None+ ) -> None:+ key = self._key(metric, attrs)+ with self._lock:+ self._histograms.setdefault(key, []).append(value)++ def set_gauge(self, metric: str, value: Any, attrs: Dict[str, Any] | None = None) -> None:+ key = self._key(metric, attrs)+ with self._lock:+ self._gauges[key] = value++ def record_event(+ self, name: str, attrs: Dict[str, Any] | None = None, scope: str | None = None+ ) -> None:+ # Event capture is intentionally disabled for summary-only telemetry.+ _ = (name, attrs, scope)+++@dataclass+class TelemetryRuntime:+ meter_instance: MemoryTelemetryMeter = field(default_factory=MemoryTelemetryMeter)++ def meter(self) -> MemoryTelemetryMeter:+ return self.meter_instance+++_RUNTIME = TelemetryRuntime()+++def get_telemetry_runtime() -> TelemetryRuntime:+ return _RUNTIME+++def set_telemetry_runtime(runtime: TelemetryRuntime) -> None:+ global _RUNTIME+ _RUNTIME = runtime+++__all__ = [+ "MemoryTelemetryMeter",+ "TelemetryRuntime",+ "get_telemetry_runtime",+ "set_telemetry_runtime",+]diff --git a/openviking/telemetry/snapshot.py b/openviking/telemetry/snapshot.pynew file mode 100644index 000000000..ed7755123--- /dev/null+++ b/openviking/telemetry/snapshot.py@@ -0,0 +1,7 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: Apache-2.0+"""Telemetry snapshot model."""++from .operation import TelemetrySnapshot++__all__ = ["TelemetrySnapshot"]diff --git a/openviking/utils/resource_processor.py b/openviking/utils/resource_processor.pyindex 23834eacd..668ecb0ff 100644--- a/openviking/utils/resource_processor.py+++ b/openviking/utils/resource_processor.py@@ -7,12 +7,14 @@ Handles coordinated writes and self-iteration processesas described in the OpenViking design document."""+import timefrom typing import TYPE_CHECKING, Any, Dict, List, Optionalfrom openviking.parse.tree_builder import TreeBuilderfrom openviking.server.identity import RequestContextfrom openviking.storage import VikingDBManagerfrom openviking.storage.viking_fs import get_viking_fs+from openviking.telemetry import get_current_telemetryfrom openviking.utils.embedding_utils import index_resourcefrom openviking.utils.summarizer import Summarizerfrom openviking_cli.utils import get_logger@@ -119,9 +121,11 @@ class ResourceProcessor:"errors": [],"source_path": None,}+ telemetry = get_current_telemetry()# ============ Phase 1: Parse source (Parser generates L0/L1 and writes to temp) ============try:+ parse_start = time.perf_counter()media_processor = self._get_media_processor()viking_fs = get_viking_fs()# Use reason as instruction fallback so it influences L0/L1@@ -149,10 +153,17 @@ class ResourceProcessor:if parse_result.warnings:result["errors"].extend(parse_result.warnings)+ telemetry.set(+ "resource.parse.duration_ms",+ round((time.perf_counter() - parse_start) * 1000, 3),+ )+ telemetry.set("resource.parse.warnings_count", len(parse_result.warnings or []))+except Exception as e:result["status"] = "error"result["errors"].append(f"Parse error: {e}")logger.error(f"[ResourceProcessor] Parse error: {e}")+ telemetry.set_error("resource_processor.parse", "PROCESSING_ERROR", str(e))import tracebacktraceback.print_exc()@@ -166,6 +177,7 @@ class ResourceProcessor:# ============ Phase 2: Pass to and parent directly to TreeBuilder ============# ============ Phase 3: TreeBuilder finalizes from temp (scan + move to AGFS) ============try:+ finalize_start = time.perf_counter()with get_viking_fs().bind_request_context(ctx):context_tree = await self.tree_builder.finalize_from_temp(temp_dir_path=parse_result.temp_dir_path,@@ -179,9 +191,14 @@ class ResourceProcessor:)if context_tree and context_tree.root:result["root_uri"] = context_tree.root.uri+ telemetry.set(+ "resource.finalize.duration_ms",+ round((time.perf_counter() - finalize_start) * 1000, 3),+ )except Exception as e:result["status"] = "error"result["errors"].append(f"Finalize from temp error: {e}")+ telemetry.set_error("resource_processor.finalize", "PROCESSING_ERROR", str(e))# Cleanup temporary directory on error (via VikingFS)try:diff --git a/openviking/utils/skill_processor.py b/openviking/utils/skill_processor.pyindex 1d6e93f7a..1aed2d0d9 100644--- a/openviking/utils/skill_processor.py+++ b/openviking/utils/skill_processor.py@@ -7,6 +7,7 @@ Handles skill parsing, LLM generation, and storage operations."""import tempfile+import timeimport zipfilefrom pathlib import Pathfrom typing import Any, Dict, List, Optional@@ -18,6 +19,7 @@ from openviking.server.identity import RequestContextfrom openviking.storage import VikingDBManagerfrom openviking.storage.queuefs.embedding_msg_converter import EmbeddingMsgConverterfrom openviking.storage.viking_fs import VikingFS+from openviking.telemetry import get_current_telemetryfrom openviking_cli.utils import get_loggerfrom openviking_cli.utils.config import get_openviking_config@@ -62,8 +64,13 @@ class SkillProcessor:raise ValueError("Skill data cannot be None")config = get_openviking_config()+ telemetry = get_current_telemetry()+ parse_start = time.perf_counter()skill_dict, auxiliary_files, base_path = self._parse_skill(data)+ telemetry.set(+ "skill.parse.duration_ms", round((time.perf_counter() - parse_start) * 1000, 3)+ )context = Context(uri=f"viking://agent/skills/{skill_dict['name']}",@@ -84,10 +91,16 @@ class SkillProcessor:)context.set_vectorize(Vectorize(text=context.abstract))+ overview_start = time.perf_counter()overview = await self._generate_overview(skill_dict, config)+ telemetry.set(+ "skill.overview.duration_ms",+ round((time.perf_counter() - overview_start) * 1000, 3),+ )skill_dir_uri = f"viking://agent/skills/{context.meta['name']}"+ write_start = time.perf_counter()await self._write_skill_content(viking_fs=viking_fs,skill_dict=skill_dict,@@ -103,12 +116,18 @@ class SkillProcessor:skill_dir_uri=skill_dir_uri,ctx=ctx,)+ telemetry.set(+ "skill.write.duration_ms", round((time.perf_counter() - write_start) * 1000, 3)+ )+ index_start = time.perf_counter()await self._index_skill(context=context,skill_dir_uri=skill_dir_uri,)-+ telemetry.set(+ "skill.index.duration_ms", round((time.perf_counter() - index_start) * 1000, 3)+ )return {"status": "success","uri": skill_dir_uri,diff --git a/openviking/utils/summarizer.py b/openviking/utils/summarizer.pyindex a7477ba3e..3c6a39c68 100644--- a/openviking/utils/summarizer.py+++ b/openviking/utils/summarizer.py@@ -1,18 +1,19 @@# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.# SPDX-License-Identifier: Apache-2.0-"""-Summarizer for OpenViking.+"""Summarizer for OpenViking.Handles summarization and key information extraction."""-from typing import TYPE_CHECKING, Any, Dict, List, Optional-from openviking_cli.utils import get_logger+from typing import TYPE_CHECKING, Any, Dict, List+from openviking.storage.queuefs import SemanticMsg, get_queue_manager+from openviking.telemetry import get_current_telemetry+from openviking_cli.utils import get_loggerif TYPE_CHECKING:- from openviking.server.identity import RequestContextfrom openviking.parse.vlm import VLMProcessor+ from openviking.server.identity import RequestContextlogger = get_logger(__name__)@@ -40,6 +41,7 @@ class Summarizer:semantic_queue = queue_manager.get_queue(queue_manager.SEMANTIC, allow_create=True)enqueued_count = 0+ telemetry = get_current_telemetry()for uri in resource_uris:# Determine context_type based on URIcontext_type = "resource"@@ -56,6 +58,7 @@ class Summarizer:agent_id=ctx.user.agent_id,role=ctx.role.value,skip_vectorization=skip_vectorization,+ telemetry_id=telemetry.telemetry_id if telemetry.enabled else "",)await semantic_queue.enqueue(msg)enqueued_count += 1diff --git a/openviking_cli/client/base.py b/openviking_cli/client/base.pyindex ddc9dc709..8eb8e3f30 100644--- a/openviking_cli/client/base.py+++ b/openviking_cli/client/base.py@@ -8,6 +8,8 @@ Defines the abstract base class that both LocalClient and AsyncHTTPClient implemfrom abc import ABC, abstractmethodfrom typing import Any, Dict, List, Optional, Union+from openviking.telemetry import TelemetryRequest+class BaseClient(ABC):"""Abstract base class for OpenViking clients.@@ -39,6 +41,7 @@ class BaseClient(ABC):instruction: str = "",wait: bool = False,timeout: Optional[float] = None,+ telemetry: TelemetryRequest = False,) -> Dict[str, Any]:"""Add resource to OpenViking."""@@ -48,6 +51,7 @@ class BaseClient(ABC):data: Any,wait: bool = False,timeout: Optional[float] = None,+ telemetry: TelemetryRequest = False,) -> Dict[str, Any]:"""Add skill to OpenViking."""...@@ -138,6 +142,7 @@ class BaseClient(ABC):limit: int = 10,score_threshold: Optional[float] = None,filter: Optional[Dict] = None,+ telemetry: TelemetryRequest = False,) -> Any:"""Semantic search without session context."""...@@ -151,6 +156,7 @@ class BaseClient(ABC):limit: int = 10,score_threshold: Optional[float] = None,filter: Optional[Dict] = None,+ telemetry: TelemetryRequest = False,) -> Any:"""Semantic search with optional session context."""...@@ -205,7 +211,9 @@ class BaseClient(ABC):...@abstractmethod- async def commit_session(self, session_id: str) -> Dict[str, Any]:+ async def commit_session(+ self, session_id: str, telemetry: TelemetryRequest = False+ ) -> Dict[str, Any]:"""Commit a session (archive and extract memories)."""...diff --git a/openviking_cli/client/http.py b/openviking_cli/client/http.pyindex 52e7070a9..5b0689bd7 100644--- a/openviking_cli/client/http.py+++ b/openviking_cli/client/http.py@@ -13,6 +13,7 @@ from typing import Any, Dict, List, Optional, Unionimport httpx+from openviking.telemetry import TelemetryRequest, normalize_telemetry_requestfrom openviking_cli.client.base import BaseClientfrom openviking_cli.exceptions import (AlreadyExistsError,@@ -196,8 +197,8 @@ class AsyncHTTPClient(BaseClient):# ============= Internal Helpers =============- def _handle_response(self, response: httpx.Response) -> Any:- """Handle HTTP response and extract result or raise exception."""+ def _handle_response_data(self, response: httpx.Response) -> Dict[str, Any]:+ """Handle HTTP response and return the decoded response envelope."""try:data = response.json()except Exception:@@ -206,7 +207,7 @@ class AsyncHTTPClient(BaseClient):f"HTTP {response.status_code}: {response.text or 'empty response'}",code="INTERNAL",)- return None+ return {}if data.get("status") == "error":self._raise_exception(data.get("error", {}))if not response.is_success:@@ -214,7 +215,33 @@ class AsyncHTTPClient(BaseClient):data.get("detail", f"HTTP {response.status_code}"),code="UNKNOWN",)- return data.get("result")+ return data++ def _handle_response(self, response: httpx.Response) -> Any:+ """Handle HTTP response and extract result or raise exception."""+ return self._handle_response_data(response).get("result")++ @staticmethod+ def _validate_telemetry(telemetry: TelemetryRequest) -> TelemetryRequest:+ normalize_telemetry_request(telemetry)+ return telemetry++ @staticmethod+ def _attach_telemetry(result: Any, response_data: Dict[str, Any]) -> Any:+ telemetry = response_data.get("telemetry")+ if telemetry is None:+ return result++ if result is None:+ payload: Dict[str, Any] = {}+ payload["telemetry"] = telemetry+ return payload++ if isinstance(result, dict):+ result["telemetry"] = telemetry+ return result++ return resultdef _raise_exception(self, error: Dict[str, Any]) -> None:"""Raise appropriate exception based on error code."""@@ -296,9 +323,10 @@ class AsyncHTTPClient(BaseClient):exclude: Optional[str] = None,directly_upload_media: bool = True,preserve_structure: Optional[bool] = None,+ telemetry: TelemetryRequest = False,) -> Dict[str, Any]:"""Add resource to OpenViking."""- # Validate that only one of 'to' or 'parent' is set+ telemetry = self._validate_telemetry(telemetry)if to and parent:raise ValueError("Cannot specify both 'to' and 'parent' at the same time.")@@ -314,6 +342,7 @@ class AsyncHTTPClient(BaseClient):"include": include,"exclude": exclude,"directly_upload_media": directly_upload_media,+ "telemetry": telemetry,}if preserve_structure is not None:request_data["preserve_structure"] = preserve_structure@@ -339,15 +368,18 @@ class AsyncHTTPClient(BaseClient):"/api/v1/resources",json=request_data,)- return self._handle_response(response)+ response_data = self._handle_response_data(response)+ return self._attach_telemetry(response_data.get("result"), response_data)async def add_skill(self,data: Any,wait: bool = False,timeout: Optional[float] = None,+ telemetry: TelemetryRequest = False,) -> Dict[str, Any]:"""Add skill to OpenViking."""+ telemetry = self._validate_telemetry(telemetry)request_data = {"wait": wait,"timeout": timeout,@@ -375,9 +407,10 @@ class AsyncHTTPClient(BaseClient):response = await self._http.post("/api/v1/skills",- json=request_data,+ json={**request_data, "telemetry": telemetry},)- return self._handle_response(response)+ response_data = self._handle_response_data(response)+ return self._attach_telemetry(response_data.get("result"), response_data)async def wait_processed(self, timeout: Optional[float] = None) -> Dict[str, Any]:"""Wait for all processing to complete."""@@ -522,8 +555,10 @@ class AsyncHTTPClient(BaseClient):node_limit: Optional[int] = None,score_threshold: Optional[float] = None,filter: Optional[Dict[str, Any]] = None,+ telemetry: TelemetryRequest = False,) -> FindResult:"""Semantic search without session context."""+ telemetry = self._validate_telemetry(telemetry)if target_uri:target_uri = VikingURI.normalize(target_uri)actual_limit = node_limit if node_limit is not None else limit@@ -535,9 +570,11 @@ class AsyncHTTPClient(BaseClient):"limit": actual_limit,"score_threshold": score_threshold,"filter": filter,+ "telemetry": telemetry,},)- return FindResult.from_dict(self._handle_response(response))+ response_data = self._handle_response_data(response)+ return FindResult.from_dict(response_data.get("result") or {})async def search(self,@@ -549,8 +586,10 @@ class AsyncHTTPClient(BaseClient):node_limit: Optional[int] = None,score_threshold: Optional[float] = None,filter: Optional[Dict[str, Any]] = None,+ telemetry: TelemetryRequest = False,) -> FindResult:"""Semantic search with optional session context."""+ telemetry = self._validate_telemetry(telemetry)if target_uri:target_uri = VikingURI.normalize(target_uri)actual_limit = node_limit if node_limit is not None else limit@@ -564,9 +603,11 @@ class AsyncHTTPClient(BaseClient):"limit": actual_limit,"score_threshold": score_threshold,"filter": filter,+ "telemetry": telemetry,},)- return FindResult.from_dict(self._handle_response(response))+ response_data = self._handle_response_data(response)+ return FindResult.from_dict(response_data.get("result") or {})async def grep(self,@@ -659,10 +700,17 @@ class AsyncHTTPClient(BaseClient):response = await self._http.delete(f"/api/v1/sessions/{session_id}")self._handle_response(response)- async def commit_session(self, session_id: str) -> Dict[str, Any]:+ async def commit_session(+ self, session_id: str, telemetry: TelemetryRequest = False+ ) -> Dict[str, Any]:"""Commit a session (archive and extract memories)."""- response = await self._http.post(f"/api/v1/sessions/{session_id}/commit")- return self._handle_response(response)+ telemetry = self._validate_telemetry(telemetry)+ response = await self._http.post(+ f"/api/v1/sessions/{session_id}/commit",+ json={"telemetry": telemetry},+ )+ response_data = self._handle_response_data(response)+ return self._attach_telemetry(response_data.get("result"), response_data)async def add_message(self,diff --git a/openviking_cli/client/sync_http.py b/openviking_cli/client/sync_http.pyindex 1a735d821..503c26abc 100644--- a/openviking_cli/client/sync_http.py+++ b/openviking_cli/client/sync_http.py@@ -7,6 +7,7 @@ Wraps AsyncHTTPClient with synchronous methods.from typing import Any, Dict, List, Optional, Union+from openviking.telemetry import TelemetryRequestfrom openviking_cli.client.http import AsyncHTTPClientfrom openviking_cli.utils import run_async@@ -105,9 +106,11 @@ class SyncHTTPClient:"""return run_async(self._async_client.add_message(session_id, role, content, parts))- def commit_session(self, session_id: str) -> Dict[str, Any]:+ def commit_session(+ self, session_id: str, telemetry: TelemetryRequest = False+ ) -> Dict[str, Any]:"""Commit a session (archive and extract memories)."""- return run_async(self._async_client.commit_session(session_id))+ return run_async(self._async_client.commit_session(session_id, telemetry=telemetry))# ============= Resource =============@@ -125,8 +128,11 @@ class SyncHTTPClient:include: Optional[str] = None,exclude: Optional[str] = None,directly_upload_media: bool = True,+ telemetry: TelemetryRequest = False,) -> Dict[str, Any]:"""Add resource to OpenViking."""+ if to and parent:+ raise ValueError("Cannot specify both 'to' and 'parent' at the same time.")return run_async(self._async_client.add_resource(path,@@ -141,6 +147,7 @@ class SyncHTTPClient:include,exclude,directly_upload_media,+ telemetry=telemetry,))@@ -149,9 +156,12 @@ class SyncHTTPClient:data: Any,wait: bool = False,timeout: Optional[float] = None,+ telemetry: TelemetryRequest = False,) -> Dict[str, Any]:"""Add skill to OpenViking."""- return run_async(self._async_client.add_skill(data, wait=wait, timeout=timeout))+ return run_async(+ self._async_client.add_skill(data, wait=wait, timeout=timeout, telemetry=telemetry)+ )def wait_processed(self, timeout: Optional[float] = None) -> Dict[str, Any]:"""Wait for all processing to complete."""@@ -169,6 +179,7 @@ class SyncHTTPClient:node_limit: Optional[int] = None,score_threshold: Optional[float] = None,filter: Optional[Dict] = None,+ telemetry: TelemetryRequest = False,):"""Semantic search with optional session context."""return run_async(@@ -181,6 +192,7 @@ class SyncHTTPClient:node_limit=node_limit,score_threshold=score_threshold,filter=filter,+ telemetry=telemetry,))@@ -192,10 +204,19 @@ class SyncHTTPClient:node_limit: Optional[int] = None,score_threshold: Optional[float] = None,filter: Optional[Dict] = None,+ telemetry: TelemetryRequest = False,):"""Semantic search without session context."""return run_async(- self._async_client.find(query, target_uri, limit, node_limit, score_threshold, filter)+ self._async_client.find(+ query,+ target_uri,+ limit,+ node_limit,+ score_threshold,+ filter,+ telemetry=telemetry,+ ))def grep(diff --git a/tests/client/test_resource_management.py b/tests/client/test_resource_management.pyindex 294ce8d41..98302fd7e 100644--- a/tests/client/test_resource_management.py+++ b/tests/client/test_resource_management.py@@ -45,13 +45,11 @@ class TestAddResource:observer = client.observerassert observer.queue is not None- async def test_add_resource_with_target(- self, client: AsyncOpenViking, sample_markdown_file: Path- ):+ async def test_add_resource_with_to(self, client: AsyncOpenViking, sample_markdown_file: Path):"""Test adding resource to specified target"""result = await client.add_resource(path=str(sample_markdown_file),- target="viking://resources/custom/",+ to="viking://resources/custom/",reason="Test resource",)diff --git a/tests/models/test_embedding_telemetry_usage.py b/tests/models/test_embedding_telemetry_usage.pynew file mode 100644index 000000000..374128fde--- /dev/null+++ b/tests/models/test_embedding_telemetry_usage.py@@ -0,0 +1,104 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: Apache-2.0++from __future__ import annotations++from types import SimpleNamespace++from openviking.models.embedder.openai_embedders import OpenAIDenseEmbedder+from openviking.models.embedder.volcengine_embedders import VolcengineDenseEmbedder+from openviking.telemetry.backends.memory import MemoryOperationTelemetry+from openviking.telemetry.context import bind_telemetry+++def _usage(prompt_tokens: int, total_tokens: int):+ return SimpleNamespace(prompt_tokens=prompt_tokens, total_tokens=total_tokens)+++def test_openai_dense_embedder_reports_embedding_telemetry_usage(monkeypatch):+ response = SimpleNamespace(+ data=[SimpleNamespace(embedding=[0.1, 0.2, 0.3])],+ usage=_usage(prompt_tokens=9, total_tokens=9),+ )++ fake_client = SimpleNamespace(embeddings=SimpleNamespace(create=lambda **kwargs: response))+ monkeypatch.setattr("openai.OpenAI", lambda **kwargs: fake_client)++ telemetry = MemoryOperationTelemetry(operation="search.find", enabled=True)+ with bind_telemetry(telemetry):+ embedder = OpenAIDenseEmbedder(+ model_name="text-embedding-3-small",+ api_key="test",+ dimension=3,+ )+ result = embedder.embed("hello")++ assert result.dense_vector == [0.1, 0.2, 0.3]+ summary = telemetry.finish().summary+ assert summary["tokens"]["embedding"] == {"total": 9}+ assert summary["tokens"]["total"] == 9+++def test_volcengine_dense_embedder_reports_embedding_telemetry_usage(monkeypatch):+ response = SimpleNamespace(+ data=SimpleNamespace(embedding=[0.4, 0.5, 0.6]),+ usage=_usage(prompt_tokens=16, total_tokens=16),+ )++ fake_client = SimpleNamespace(+ multimodal_embeddings=SimpleNamespace(create=lambda **kwargs: response),+ )+ monkeypatch.setattr(+ "volcenginesdkarkruntime.Ark",+ lambda **kwargs: fake_client,+ )++ telemetry = MemoryOperationTelemetry(operation="resources.add_resource", enabled=True)+ with bind_telemetry(telemetry):+ embedder = VolcengineDenseEmbedder(+ model_name="doubao-embedding-vision-250615",+ api_key="test",+ input_type="multimodal",+ dimension=3,+ )+ result = embedder.embed("hello")++ assert result.dense_vector == [0.4, 0.5, 0.6]+ summary = telemetry.finish().summary+ assert summary["tokens"]["embedding"] == {"total": 16}+ assert summary["tokens"]["total"] == 16+++def test_volcengine_dense_embedder_reports_embedding_telemetry_usage_from_dict_usage(+ monkeypatch,+):+ response = SimpleNamespace(+ data=SimpleNamespace(embedding=[0.4, 0.5, 0.6]),+ usage={+ "prompt_tokens": 16,+ "prompt_tokens_details": {"image_tokens": 0, "text_tokens": 16},+ "total_tokens": 16,+ },+ )++ fake_client = SimpleNamespace(+ multimodal_embeddings=SimpleNamespace(create=lambda **kwargs: response),+ )+ monkeypatch.setattr(+ "volcenginesdkarkruntime.Ark",+ lambda **kwargs: fake_client,+ )++ telemetry = MemoryOperationTelemetry(operation="search.find", enabled=True)+ with bind_telemetry(telemetry):+ embedder = VolcengineDenseEmbedder(+ model_name="doubao-embedding-vision-250615",+ api_key="test",+ input_type="multimodal",+ dimension=3,+ )+ result = embedder.embed("hello")++ assert result.dense_vector == [0.4, 0.5, 0.6]+ summary = telemetry.finish().summary+ assert summary["tokens"]["embedding"] == {"total": 16}diff --git a/tests/server/conftest.py b/tests/server/conftest.pyindex c38c7eab4..627798b4a 100644--- a/tests/server/conftest.py+++ b/tests/server/conftest.py@@ -15,11 +15,13 @@ import pytest_asyncioimport uvicornfrom openviking import AsyncOpenViking+from openviking.models.embedder.base import DenseEmbedderBase, EmbedResultfrom openviking.server.app import create_appfrom openviking.server.config import ServerConfigfrom openviking.server.identity import RequestContext, Rolefrom openviking.service.core import OpenVikingServicefrom openviking_cli.session.user_id import UserIdentifier+from openviking_cli.utils.config.embedding_config import EmbeddingConfig# ---------------------------------------------------------------------------# Paths@@ -44,6 +46,27 @@ This is a sample markdown document for server testing."""+def _install_fake_embedder(monkeypatch):+ """Use an in-process fake embedder so server tests never hit external APIs."""+ dimension = 2048++ class FakeEmbedder(DenseEmbedderBase):+ def __init__(self):+ super().__init__(model_name="test-fake-embedder")++ def embed(self, text: str) -> EmbedResult:+ return EmbedResult(dense_vector=[0.1] * dimension)++ def embed_batch(self, texts: list[str]) -> list[EmbedResult]:+ return [self.embed(text) for text in texts]++ def get_dimension(self) -> int:+ return dimension++ monkeypatch.setattr(EmbeddingConfig, "get_embedder", lambda self: FakeEmbedder())+ return FakeEmbedder++# ---------------------------------------------------------------------------# Core fixtures: service + app + async client (HTTP API tests, in-process)# ---------------------------------------------------------------------------@@ -69,12 +92,14 @@ def sample_markdown_file(temp_dir: Path) -> Path:@pytest_asyncio.fixture(scope="function")-async def service(temp_dir: Path):+async def service(temp_dir: Path, monkeypatch):"""Create and initialize an OpenVikingService in embedded mode."""+ fake_embedder_cls = _install_fake_embedder(monkeypatch)svc = OpenVikingService(path=str(temp_dir / "data"), user=UserIdentifier.the_default_user("test_user"))await svc.initialize()+ svc.viking_fs.query_embedder = fake_embedder_cls()yield svcawait svc.close()@@ -118,14 +143,16 @@ async def client_with_resource(client, service, sample_markdown_file):@pytest_asyncio.fixture(scope="function")-async def running_server(temp_dir: Path):+async def running_server(temp_dir: Path, monkeypatch):"""Start a real uvicorn server in a background thread."""await AsyncOpenViking.reset()+ fake_embedder_cls = _install_fake_embedder(monkeypatch)svc = OpenVikingService(path=str(temp_dir / "sdk_data"), user=UserIdentifier.the_default_user("sdk_test_user"))await svc.initialize()+ svc.viking_fs.query_embedder = fake_embedder_cls()config = ServerConfig()fastapi_app = create_app(config=config, service=svc)diff --git a/tests/server/test_api_resources.py b/tests/server/test_api_resources.pyindex 013c6baa2..48e214e8f 100644--- a/tests/server/test_api_resources.py+++ b/tests/server/test_api_resources.py@@ -5,8 +5,6 @@import httpx-from tests.server.conftest import SAMPLE_MD_CONTENT-async def test_add_resource_success(client: httpx.AsyncClient, sample_markdown_file):resp = await client.post(@@ -20,6 +18,9 @@ async def test_add_resource_success(client: httpx.AsyncClient, sample_markdown_fassert resp.status_code == 200body = resp.json()assert body["status"] == "ok"+ assert "time" not in body+ assert "usage" not in body+ assert "telemetry" not in bodyassert "root_uri" in body["result"]assert body["result"]["root_uri"].startswith("viking://")@@ -39,6 +40,70 @@ async def test_add_resource_with_wait(client: httpx.AsyncClient, sample_markdownassert "root_uri" in body["result"]+async def test_add_resource_with_telemetry_wait(client: httpx.AsyncClient, sample_markdown_file):+ resp = await client.post(+ "/api/v1/resources",+ json={+ "path": str(sample_markdown_file),+ "reason": "telemetry resource",+ "wait": True,+ "telemetry": True,+ },+ )+ assert resp.status_code == 200+ body = resp.json()+ assert body["status"] == "ok"+ telemetry_summary = body["telemetry"]["summary"]+ assert telemetry_summary["operation"] == "resources.add_resource"+ assert "usage" not in body+ semantic = telemetry_summary["semantic_nodes"]+ assert semantic["total"] is None or semantic["done"] == semantic["total"]+ assert semantic["pending"] in (None, 0)+ assert semantic["running"] in (None, 0)+ assert "memory" not in telemetry_summary+++async def test_add_resource_with_summary_only_telemetry(+ client: httpx.AsyncClient, sample_markdown_file+):+ resp = await client.post(+ "/api/v1/resources",+ json={+ "path": str(sample_markdown_file),+ "reason": "summary only telemetry resource",+ "wait": True,+ "telemetry": {"summary": True},+ },+ )+ assert resp.status_code == 200+ body = resp.json()+ assert body["status"] == "ok"+ assert "summary" in body["telemetry"]+ assert "usage" not in body+ assert "events" not in body["telemetry"]+ assert "truncated" not in body["telemetry"]+ assert "dropped" not in body["telemetry"]+++async def test_add_resource_rejects_events_only_telemetry(+ client: httpx.AsyncClient, sample_markdown_file+):+ resp = await client.post(+ "/api/v1/resources",+ json={+ "path": str(sample_markdown_file),+ "reason": "events only telemetry",+ "wait": False,+ "telemetry": {"summary": False, "events": True},+ },+ )+ assert resp.status_code == 400+ body = resp.json()+ assert body["status"] == "error"+ assert body["error"]["code"] == "INVALID_ARGUMENT"+ assert "events" in body["error"]["message"]++async def test_add_resource_file_not_found(client: httpx.AsyncClient):resp = await client.post("/api/v1/resources",@@ -50,12 +115,12 @@ async def test_add_resource_file_not_found(client: httpx.AsyncClient):assert "errors" in body["result"] and len(body["result"]["errors"]) > 0-async def test_add_resource_with_target(client: httpx.AsyncClient, sample_markdown_file):+async def test_add_resource_with_to(client: httpx.AsyncClient, sample_markdown_file):resp = await client.post("/api/v1/resources",json={"path": str(sample_markdown_file),- "target": "viking://resources/custom/",+ "to": "viking://resources/custom/","reason": "test resource",},)diff --git a/tests/server/test_api_search.py b/tests/server/test_api_search.pyindex 834c04375..05d313fb5 100644--- a/tests/server/test_api_search.py+++ b/tests/server/test_api_search.py@@ -4,6 +4,18 @@"""Tests for search endpoints: find, search, grep, glob."""import httpx+import pytest++from openviking.models.embedder.base import EmbedResult++[email protected](autouse=True)+def fake_query_embedder(service):+ class FakeEmbedder:+ def embed(self, text: str) -> EmbedResult:+ return EmbedResult(dense_vector=[0.1, 0.2, 0.3])++ service.viking_fs.query_embedder = FakeEmbedder()async def test_find_basic(client_with_resource):@@ -16,6 +28,8 @@ async def test_find_basic(client_with_resource):body = resp.json()assert body["status"] == "ok"assert body["result"] is not None+ assert "usage" not in body+ assert "telemetry" not in bodyasync def test_find_with_target_uri(client_with_resource):@@ -81,6 +95,80 @@ async def test_search_with_session(client_with_resource):assert resp.json()["status"] == "ok"+async def test_find_telemetry_metrics(client_with_resource):+ client, _ = client_with_resource+ resp = await client.post(+ "/api/v1/search/find",+ json={"query": "sample document", "limit": 5, "telemetry": True},+ )+ assert resp.status_code == 200+ body = resp.json()+ summary = body["telemetry"]["summary"]+ assert summary["operation"] == "search.find"+ assert "duration_ms" in summary+ assert {"total", "llm", "embedding"}.issubset(summary["tokens"].keys())+ assert "vector" in summary+ assert summary["vector"]["searches"] >= 0+ assert "queue" not in summary+ assert "semantic_nodes" not in summary+ assert "memory" not in summary+ assert "usage" not in body+ assert body["telemetry"]["id"]+ assert body["telemetry"]["id"].startswith("tm_")+++async def test_search_telemetry_metrics(client_with_resource):+ client, _ = client_with_resource+ resp = await client.post(+ "/api/v1/search/search",+ json={"query": "sample document", "limit": 5, "telemetry": True},+ )+ assert resp.status_code == 200+ body = resp.json()+ summary = body["telemetry"]["summary"]+ assert summary["operation"] == "search.search"+ assert summary["vector"]["returned"] >= 0+ assert "queue" not in summary+ assert "semantic_nodes" not in summary+ assert "memory" not in summary+++async def test_find_summary_only_telemetry(client_with_resource):+ client, _ = client_with_resource+ resp = await client.post(+ "/api/v1/search/find",+ json={+ "query": "sample document",+ "limit": 5,+ "telemetry": {"summary": True},+ },+ )+ assert resp.status_code == 200+ body = resp.json()+ assert body["telemetry"]["summary"]["operation"] == "search.find"+ assert "usage" not in body+ assert "events" not in body["telemetry"]+ assert "truncated" not in body["telemetry"]+ assert "dropped" not in body["telemetry"]+++async def test_find_rejects_events_telemetry_request(client_with_resource):+ client, _ = client_with_resource+ resp = await client.post(+ "/api/v1/search/find",+ json={+ "query": "sample document",+ "limit": 5,+ "telemetry": {"summary": False, "events": True},+ },+ )+ assert resp.status_code == 400+ body = resp.json()+ assert body["status"] == "error"+ assert body["error"]["code"] == "INVALID_ARGUMENT"+ assert "events" in body["error"]["message"]++async def test_grep(client_with_resource):client, uri = client_with_resourceparent_uri = "/".join(uri.split("/")[:-1]) + "/"diff --git a/tests/server/test_api_sessions.py b/tests/server/test_api_sessions.pyindex 413907d3a..370e71cf8 100644--- a/tests/server/test_api_sessions.py+++ b/tests/server/test_api_sessions.py@@ -142,7 +142,55 @@ async def test_compress_session(client: httpx.AsyncClient):resp = await client.post(f"/api/v1/sessions/{session_id}/commit")assert resp.status_code == 200- assert resp.json()["status"] == "ok"+ body = resp.json()+ assert body["status"] == "ok"+ assert "usage" not in body+ assert "telemetry" not in body+++async def test_compress_session_with_telemetry(client: httpx.AsyncClient):+ create_resp = await client.post("/api/v1/sessions", json={})+ session_id = create_resp.json()["result"]["session_id"]+ await client.post(+ f"/api/v1/sessions/{session_id}/messages",+ json={"role": "user", "content": "Trace this commit"},+ )++ resp = await client.post(+ f"/api/v1/sessions/{session_id}/commit",+ json={"telemetry": True},+ )+ assert resp.status_code == 200+ body = resp.json()+ assert body["status"] == "ok"+ summary = body["telemetry"]["summary"]+ assert summary["operation"] == "session.commit"+ assert {"total", "llm", "embedding"}.issubset(summary["tokens"].keys())+ assert summary["memory"]["extracted"] is not None+ assert "semantic_nodes" not in summary+ assert "usage" not in body+++async def test_compress_session_with_summary_only_telemetry(client: httpx.AsyncClient):+ create_resp = await client.post("/api/v1/sessions", json={})+ session_id = create_resp.json()["result"]["session_id"]+ await client.post(+ f"/api/v1/sessions/{session_id}/messages",+ json={"role": "user", "content": "Summary only telemetry"},+ )++ resp = await client.post(+ f"/api/v1/sessions/{session_id}/commit",+ json={"telemetry": {"summary": True}},+ )+ assert resp.status_code == 200+ body = resp.json()+ assert body["status"] == "ok"+ assert body["telemetry"]["summary"]["operation"] == "session.commit"+ assert "usage" not in body+ assert "events" not in body["telemetry"]+ assert "truncated" not in body["telemetry"]+ assert "dropped" not in body["telemetry"]async def test_extract_session_jsonable_regression(client: httpx.AsyncClient, service, monkeypatch):diff --git a/tests/server/test_http_client_sdk.py b/tests/server/test_http_client_sdk.pyindex 3fb88cdd1..c757c2ce5 100644--- a/tests/server/test_http_client_sdk.py+++ b/tests/server/test_http_client_sdk.py@@ -43,6 +43,8 @@ async def test_sdk_add_resource(http_client):f.write_text(SAMPLE_MD_CONTENT)result = await client.add_resource(path=str(f), reason="sdk test", wait=True)+ assert "usage" not in result+ assert "telemetry" not in resultassert "root_uri" in resultassert result["root_uri"].startswith("viking://")@@ -121,6 +123,38 @@ async def test_sdk_find(http_client):assert hasattr(result, "total")+async def test_sdk_find_telemetry(http_client):+ client, _ = http_client+ f = TEST_TMP_DIR / "sdk_search_telemetry.md"+ f.parent.mkdir(parents=True, exist_ok=True)+ f.write_text(SAMPLE_MD_CONTENT)+ await client.add_resource(+ path=str(f), reason="telemetry search test", wait=True, telemetry=True+ )++ result = await client.find(query="sample document", limit=5, telemetry=True)+ assert not hasattr(result, "telemetry")+++async def test_sdk_find_summary_only_telemetry(http_client):+ client, _ = http_client+ f = TEST_TMP_DIR / "sdk_search_summary_only.md"+ f.parent.mkdir(parents=True, exist_ok=True)+ f.write_text(SAMPLE_MD_CONTENT)+ await client.add_resource(+ path=str(f),+ reason="summary only telemetry search test",+ wait=True,+ )++ result = await client.find(+ query="sample document",+ limit=5,+ telemetry={"summary": True},+ )+ assert not hasattr(result, "telemetry")++# ===================================================================# Full workflow# ===================================================================diff --git a/tests/telemetry/test_execution.py b/tests/telemetry/test_execution.pynew file mode 100644index 000000000..12606553f--- /dev/null+++ b/tests/telemetry/test_execution.py@@ -0,0 +1,98 @@+from types import SimpleNamespace++import pytest++from openviking_cli.exceptions import InvalidArgumentError+from openviking_cli.retrieve.types import FindResult+++async def test_run_with_telemetry_returns_usage_and_payload():+ from openviking.telemetry.execution import run_with_telemetry++ async def _run():+ return {"status": "ok"}++ execution = await run_with_telemetry(+ operation="search.find",+ telemetry=True,+ fn=_run,+ )++ assert execution.result == {"status": "ok"}+ assert execution.telemetry is not None+ assert execution.telemetry["summary"]["operation"] == "search.find"+++async def test_run_with_telemetry_raises_invalid_argument_for_bad_request():+ from openviking.telemetry.execution import run_with_telemetry++ async def _run():+ return {"status": "ok"}++ with pytest.raises(InvalidArgumentError, match="Unsupported telemetry options: invalid"):+ await run_with_telemetry(+ operation="search.find",+ telemetry={"invalid": True},+ fn=_run,+ )+++async def test_run_with_telemetry_rejects_events_selection():+ from openviking.telemetry.execution import run_with_telemetry++ async def _run():+ return {"status": "ok"}++ with pytest.raises(InvalidArgumentError, match="Unsupported telemetry options: events"):+ await run_with_telemetry(+ operation="search.find",+ telemetry={"summary": True, "events": False},+ fn=_run,+ )+++def test_attach_telemetry_payload_adds_telemetry_to_dict_result():+ from openviking.telemetry.execution import attach_telemetry_payload++ result = attach_telemetry_payload(+ {"root_uri": "viking://resources/demo"},+ {"id": "tm_123", "summary": {"operation": "resources.add_resource"}},+ )++ assert result["telemetry"]["summary"]["operation"] == "resources.add_resource"+++def test_attach_telemetry_payload_does_not_mutate_object_result():+ from openviking.telemetry.execution import attach_telemetry_payload++ result = SimpleNamespace(total=1)++ attached = attach_telemetry_payload(+ result,+ {"id": "tm_123", "summary": {"operation": "search.find"}},+ )++ assert attached is result+ assert not hasattr(result, "telemetry")+++def test_find_result_ignores_usage_and_telemetry_payload_fields():+ result = FindResult.from_dict(+ {+ "memories": [],+ "resources": [],+ "skills": [],+ "telemetry": {"id": "tm_123", "summary": {"operation": "search.find"}},+ }+ )++ assert not hasattr(result, "telemetry")+ assert result.to_dict() == {+ "memories": [],+ "resources": [],+ "skills": [],+ "total": 0,+ }diff --git a/tests/telemetry/test_layering_rules.py b/tests/telemetry/test_layering_rules.pynew file mode 100644index 000000000..98dced6f7--- /dev/null+++ b/tests/telemetry/test_layering_rules.py@@ -0,0 +1,30 @@+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+DISALLOWED_PREFIXES = (+ "telemetry.event(",+ "collector.event(",+)++ALLOWED_FILES = {"openviking/session/memory_deduplicator.py"}++CHECK_DIRS = (+ "openviking/service",+ "openviking/session",+ "openviking/retrieve",+)+++def test_core_layers_do_not_directly_call_telemetry_collectors():+ offenders: list[str] = []+ for check_dir in CHECK_DIRS:+ for path in (ROOT / check_dir).rglob("*.py"):+ rel = path.relative_to(ROOT).as_posix()+ if rel in ALLOWED_FILES:+ continue+ text = path.read_text()+ for needle in DISALLOWED_PREFIXES:+ if needle in text:+ offenders.append(f"{rel}: {needle}")++ assert offenders == []diff --git a/tests/telemetry/test_resource_summary.py b/tests/telemetry/test_resource_summary.pynew file mode 100644index 000000000..1d28e8b82--- /dev/null+++ b/tests/telemetry/test_resource_summary.py@@ -0,0 +1,58 @@+from types import SimpleNamespace++from openviking.telemetry.operation import OperationTelemetry+++def test_record_resource_wait_metrics_collects_queue_and_dag_stats(monkeypatch):+ from openviking.telemetry.resource_summary import record_resource_wait_metrics++ telemetry = OperationTelemetry(operation="resources.add_resource", enabled=True)+ telemetry_id = telemetry.telemetry_id+ queue_status = {+ "Semantic": SimpleNamespace(processed=3, error_count=1, errors=[]),+ "Embedding": SimpleNamespace(processed=5, error_count=0, errors=[]),+ }++ class _SemanticStats:+ processed = 7+ error_count = 2++ class _EmbeddingStats:+ processed = 11+ error_count = 1++ class _DagStats:+ total_nodes = 9+ done_nodes = 8+ pending_nodes = 1+ in_progress_nodes = 0++ monkeypatch.setattr(+ "openviking.telemetry.resource_summary._consume_semantic_request_stats",+ lambda _tid: _SemanticStats(),+ )+ monkeypatch.setattr(+ "openviking.telemetry.resource_summary._consume_embedding_request_stats",+ lambda _tid: _EmbeddingStats(),+ )+ monkeypatch.setattr(+ "openviking.telemetry.resource_summary._consume_semantic_dag_stats",+ lambda _tid, _uri: _DagStats(),+ )++ record_resource_wait_metrics(+ telemetry=telemetry,+ telemetry_id=telemetry_id,+ queue_status=queue_status,+ root_uri="viking://resources/demo",+ )++ summary = telemetry.finish().summary+ assert summary["queue"]["semantic"]["processed"] == 7+ assert summary["queue"]["semantic"]["error_count"] == 2+ assert summary["queue"]["embedding"]["processed"] == 11+ assert summary["queue"]["embedding"]["error_count"] == 1+ assert summary["semantic_nodes"]["total"] == 9+ assert summary["semantic_nodes"]["done"] == 8+ assert summary["semantic_nodes"]["pending"] == 1+ assert summary["semantic_nodes"]["running"] == 0diff --git a/tests/test_session_task_tracking.py b/tests/test_session_task_tracking.pyindex abeb513d9..8a61fe4d6 100644--- a/tests/test_session_task_tracking.py+++ b/tests/test_session_task_tracking.py@@ -72,6 +72,40 @@ async def test_commit_wait_false_returns_task_id(api_client):await asyncio.wait_for(done.wait(), timeout=2.0)+async def test_commit_wait_false_rejects_full_telemetry(api_client):+ """wait=false should reject telemetry payload requests."""+ client, _ = api_client+ session_id = await _new_session_with_message(client)++ resp = await client.post(+ f"/api/v1/sessions/{session_id}/commit",+ params={"wait": False},+ json={"telemetry": True},+ )+ assert resp.status_code == 400+ body = resp.json()+ assert body["status"] == "error"+ assert body["error"]["code"] == "INVALID_ARGUMENT"+ assert "wait=false" in body["error"]["message"]+++async def test_commit_wait_false_rejects_summary_only_telemetry(api_client):+ """wait=false should also reject summary-only telemetry requests."""+ client, _ = api_client+ session_id = await _new_session_with_message(client)++ resp = await client.post(+ f"/api/v1/sessions/{session_id}/commit",+ params={"wait": False},+ json={"telemetry": {"summary": True}},+ )+ assert resp.status_code == 400+ body = resp.json()+ assert body["status"] == "error"+ assert body["error"]["code"] == "INVALID_ARGUMENT"+ assert "wait=false" in body["error"]["message"]++# ── Task lifecycle: pending → running → completed ──@@ -196,6 +230,35 @@ async def test_duplicate_commit_rejected(api_client):await asyncio.sleep(0.1)+async def test_wait_true_rejected_while_background_commit_running(api_client):+ """wait=true must also reject duplicate commits for the same session."""+ client, service = api_client+ session_id = await _new_session_with_message(client)++ gate = asyncio.Event()++ async def slow_commit(_sid, _ctx):+ await gate.wait()+ return {"session_id": _sid, "status": "committed", "memories_extracted": 0}++ service.sessions.commit_async = slow_commit++ resp1 = await client.post(f"/api/v1/sessions/{session_id}/commit", params={"wait": False})+ assert resp1.json()["result"]["status"] == "accepted"++ resp2 = await client.post(+ f"/api/v1/sessions/{session_id}/commit",+ params={"wait": True},+ json={"telemetry": True},+ )+ assert resp2.status_code == 200+ assert resp2.json()["status"] == "error"+ assert "already has a commit in progress" in resp2.json()["error"]["message"]++ gate.set()+ await asyncio.sleep(0.1)++# ── GET /tasks/{id} 404 ──diff --git a/tests/test_telemetry_runtime.py b/tests/test_telemetry_runtime.pynew file mode 100644index 000000000..eecdbf04b--- /dev/null+++ b/tests/test_telemetry_runtime.py@@ -0,0 +1,296 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: Apache-2.0++from __future__ import annotations++import json+from types import SimpleNamespace++import pytest++from openviking.models.embedder.base import EmbedResult+from openviking.server.identity import RequestContext, Role+from openviking.service.resource_service import ResourceService+from openviking.storage.collection_schemas import TextEmbeddingHandler+from openviking.storage.queuefs.semantic_msg import SemanticMsg+from openviking.storage.queuefs.semantic_processor import SemanticProcessor+from openviking.telemetry import (+ get_current_telemetry,+ get_telemetry_runtime,+ register_telemetry,+ unregister_telemetry,+)+from openviking.telemetry.backends.memory import MemoryOperationTelemetry+from openviking.telemetry.context import bind_telemetry+from openviking.telemetry.snapshot import TelemetrySnapshot+from openviking_cli.session.user_id import UserIdentifier+++def test_telemetry_module_exports_snapshot_and_runtime():+ snapshot = TelemetrySnapshot(+ telemetry_id="tm_demo",+ summary={"duration_ms": 1.2},+ )+ usage = snapshot.to_usage_dict()++ assert usage == {"duration_ms": 1.2, "token_total": 0}+ assert get_telemetry_runtime().meter() is not None+++def test_telemetry_snapshot_to_dict_supports_summary_only():+ snapshot = TelemetrySnapshot(+ telemetry_id="tm_demo",+ summary={"duration_ms": 1.2, "tokens": {"total": 3}},+ )++ payload = snapshot.to_dict(include_summary=True)++ assert payload == {+ "id": "tm_demo",+ "summary": {"duration_ms": 1.2, "tokens": {"total": 3}},+ }+++def test_telemetry_summary_breaks_down_llm_and_embedding_token_usage():+ telemetry = MemoryOperationTelemetry(operation="resources.add_resource", enabled=True)+ telemetry.record_token_usage("llm", 11, 7)+ telemetry.record_token_usage("embedding", 13, 0)++ summary = telemetry.finish().summary+ assert telemetry.telemetry_id+ assert telemetry.telemetry_id.startswith("tm_")+ assert summary["tokens"]["total"] == 31+ assert summary["duration_ms"] >= 0+ assert summary["tokens"]["llm"] == {+ "input": 11,+ "output": 7,+ "total": 18,+ }+ assert summary["tokens"]["embedding"] == {"total": 13}+ assert "queue" not in summary+ assert "vector" not in summary+ assert "semantic_nodes" not in summary+ assert "memory" not in summary+ assert "errors" not in summary+++def test_telemetry_summary_uses_simplified_internal_metric_keys():+ summary = MemoryOperationTelemetry(+ operation="search.find",+ enabled=True,+ )+ summary.count("vector.searches", 2)+ summary.count("vector.scored", 5)+ summary.count("vector.passed", 3)+ summary.set("vector.returned", 2)+ summary.count("vector.scanned", 5)+ summary.set("vector.scan_reason", "")+ summary.set("semantic_nodes.total", 4)+ summary.set("semantic_nodes.done", 3)+ summary.set("semantic_nodes.pending", 1)+ summary.set("semantic_nodes.running", 0)+ summary.set("memory.extracted", 6)++ result = summary.finish().summary++ assert result["vector"] == {+ "searches": 2,+ "scored": 5,+ "passed": 3,+ "returned": 2,+ "scanned": 5,+ "scan_reason": "",+ }+ assert result["semantic_nodes"] == {+ "total": 4,+ "done": 3,+ "pending": 1,+ "running": 0,+ }+ assert result["memory"] == {"extracted": 6}+++def test_telemetry_summary_detects_groups_by_prefix_without_static_key_lists():+ telemetry = MemoryOperationTelemetry(operation="search.find", enabled=True)+ telemetry.set("vector.debug_probe", 1)+ telemetry.set("queue.semantic.processed", 2)+ telemetry.set("memory.extracted", 1)++ result = telemetry.finish().summary++ assert "vector" in result+ assert "queue" in result+ assert "memory" in result+++async def test_semantic_processor_binds_registered_operation_telemetry(monkeypatch):+ telemetry = MemoryOperationTelemetry(operation="resources.add_resource", enabled=True)+ register_telemetry(telemetry)++ processor = SemanticProcessor()++ class FakeVikingFS:+ async def ls(self, uri, ctx=None):+ return []++ async def fake_process_single_directory(**kwargs):+ assert get_current_telemetry() is telemetry+ get_current_telemetry().record_token_usage("llm", 11, 7)++ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_processor.get_viking_fs",+ lambda: FakeVikingFS(),+ )+ monkeypatch.setattr(processor, "_process_single_directory", fake_process_single_directory)++ try:+ await processor.on_dequeue(+ SemanticMsg(+ uri="viking://resources/demo",+ context_type="resource",+ recursive=False,+ telemetry_id=telemetry.telemetry_id,+ ).to_dict()+ )+ finally:+ unregister_telemetry(telemetry.telemetry_id)++ result = telemetry.finish()+ summary = result.summary+ assert summary["tokens"]["total"] == 18+ assert summary["tokens"]["llm"]["total"] == 18+ assert summary["tokens"]["embedding"]["total"] == 0+++async def test_embedding_handler_binds_registered_operation_telemetry(monkeypatch):+ telemetry = MemoryOperationTelemetry(operation="resources.add_resource", enabled=True)+ register_telemetry(telemetry)++ class _TelemetryAwareEmbedder:+ def embed(self, text: str) -> EmbedResult:+ assert text == "hello"+ get_current_telemetry().record_token_usage("embedding", 9, 0)+ return EmbedResult(dense_vector=[0.1, 0.2])++ class _DummyConfig:+ def __init__(self):+ self.storage = SimpleNamespace(vectordb=SimpleNamespace(name="context"))+ self.embedding = SimpleNamespace(+ dimension=2,+ get_embedder=lambda: _TelemetryAwareEmbedder(),+ )++ class _DummyVikingDB:+ is_closing = False++ async def upsert(self, _data):+ return "rec-1"++ monkeypatch.setattr(+ "openviking_cli.utils.config.get_openviking_config",+ lambda: _DummyConfig(),+ )++ handler = TextEmbeddingHandler(_DummyVikingDB())+ payload = {+ "data": json.dumps(+ {+ "id": "msg-1",+ "message": "hello",+ "telemetry_id": telemetry.telemetry_id,+ "context_data": {+ "id": "id-1",+ "uri": "viking://resources/sample",+ "account_id": "default",+ "abstract": "sample",+ },+ }+ )+ }++ try:+ await handler.on_dequeue(payload)+ finally:+ unregister_telemetry(telemetry.telemetry_id)++ result = telemetry.finish()+ summary = result.summary+ assert summary["tokens"]["embedding"] == {"total": 9}+++async def test_resource_service_add_resource_reports_queue_summary(monkeypatch):+ telemetry = MemoryOperationTelemetry(operation="resources.add_resource", enabled=True)++ class _DummyProcessor:+ async def process_resource(self, **kwargs):+ return {+ "status": "success",+ "root_uri": "viking://resources/demo",+ }++ class _DummyQueueManager:+ async def wait_complete(self, timeout=None):+ return {+ "Semantic": SimpleNamespace(processed=2, error_count=1, errors=[]),+ "Embedding": SimpleNamespace(processed=5, error_count=0, errors=[]),+ }++ monkeypatch.setattr(+ "openviking.service.resource_service.get_queue_manager",+ lambda: _DummyQueueManager(),+ )++ class _DagStats:+ total_nodes = 3+ done_nodes = 2+ pending_nodes = 1+ in_progress_nodes = 0++ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_processor.SemanticProcessor.consume_dag_stats",+ classmethod(lambda cls, telemetry_id="", uri=None: _DagStats()),+ )++ service = ResourceService(+ vikingdb=object(),+ viking_fs=object(),+ resource_processor=_DummyProcessor(),+ skill_processor=object(),+ )+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)++ with bind_telemetry(telemetry):+ result = await service.add_resource(path="/tmp/demo.md", ctx=ctx, wait=True)++ assert result["root_uri"] == "viking://resources/demo"+ telemetry_result = telemetry.finish()+ summary = telemetry_result.summary+ assert summary["queue"] == {+ "semantic": {"processed": 2, "error_count": 1},+ "embedding": {"processed": 5, "error_count": 0},+ }+ assert summary["semantic_nodes"] == {+ "total": 3,+ "done": 2,+ "pending": 1,+ "running": 0,+ }+ assert "memory" not in summary+ assert "errors" not in summary+++def test_telemetry_summary_includes_only_memory_group_when_memory_metrics_exist():+ telemetry = MemoryOperationTelemetry(operation="session.commit", enabled=True)+ telemetry.record_token_usage("llm", 5, 3)+ telemetry.set("memory.extracted", 4)++ summary = telemetry.finish().summary++ assert summary["memory"] == {"extracted": 4}+ assert "queue" not in summary+ assert "vector" not in summary+ assert "semantic_nodes" not in summary+ assert "errors" not in summarydiff --git a/tests/unit/test_skill_processor_none.py b/tests/unit/test_skill_processor_none.pyindex a87435335..1681ec28b 100644--- a/tests/unit/test_skill_processor_none.py+++ b/tests/unit/test_skill_processor_none.py@@ -43,3 +43,21 @@ class TestParseSkillNoneData:processor = SkillProcessor(vikingdb=None)with pytest.raises(ValueError, match="Unsupported data type"):processor._parse_skill(12345)++ def test_parse_skill_long_raw_content_raises_oserror(self):+ """Long raw SKILL.md content should still surface path probing errors."""+ processor = SkillProcessor(vikingdb=None)+ long_description = "telemetry " * 80+ raw_skill = (+ "---\n"+ "name: telemetry-demo-skill\n"+ f"description: {long_description}\n"+ "tags:\n"+ " - telemetry\n"+ "---\n\n"+ "# Telemetry Demo Skill\n\n"+ "Use this skill to validate telemetry ingestion.\n"+ )++ with pytest.raises(OSError, match="File name too long"):+ processor._parse_skill(raw_skill)