Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(metrics): add vikingbot feedback observability (#2037)
yepper9ebfd59342c0
diff --git a/bot/docs/vikingbot-feedback-observability-design.md b/bot/docs/vikingbot-feedback-observability-design.mdindex 2b80eb0e6..edf682161 100644--- a/bot/docs/vikingbot-feedback-observability-design.md+++ b/bot/docs/vikingbot-feedback-observability-design.md@@ -22,7 +22,7 @@- Phase 1 已经补齐 `response_id`,并打通到 `OutboundMessage`、session JSONL、OpenAPI 返回体与 Langfuse metadata- Phase 2 当前工作树已经补齐显式反馈入口,OpenAPI 提供 `POST /bot/v1/feedback`- `response_completed` 与 `feedback_submitted` 已经进入实现范围;`response_outcome_evaluated` 仍然属于 Phase 3,但当前工作树已经落地最小规则版实现-- session JSONL 目前不是面向分析的响应事实表,`time_cost`、`iteration`、`tools_used_names` 也没有以统一结构持久化到响应记录里+- session JSONL 的 assistant message 仍然不是完整的响应事实表,但当前已经把标准化 `response_completed` 以 `session.metadata["response_facts"][response_id]` 的形式持久化到 metadata 首行- OpenAPI 当前只透出 `response`、`reasoning`、`tool_call`、`tool_result`,并未把 `ITERATION` 等过程事件完整暴露出来- Langfuse 当前写入的 generation metadata 已覆盖 `response_id`;`query_category`、`prompt_version`、`bot_version` 仍未稳定覆盖;Phase 3 outcome 当前通过 trace event 与 observation score 记录,而不是事后回写已结束 generation metadata- 代码中还没有稳定的 `query_category`、`prompt_version`、`bot_version` 字段来源,因此原方案里大量切片分析暂时没有数据基础@@ -112,7 +112,7 @@为了避免方案设计继续偏离当前实现,整体落地顺序调整为以下三层依赖:1. `response identity`:先给每次最终回答分配稳定的 `response_id`,并把它同时写入 `OutboundMessage`、session message、OpenAPI 返回体和 Langfuse metadata。-2. `response facts`:再沉淀一条结构化 `response_completed` 事件,把当前已经能拿到的字段先稳定保存下来,例如 `session_id`、`user_id`、`time_cost`、`token_usage`、`iteration_count`、`tool_count`、`tools_used_names`、`finish_reason`。+2. `response facts`:再沉淀一条结构化 `response_completed` 事件,把当前已经能拿到的字段先稳定保存下来,并以 `session.metadata["response_facts"][response_id]` 的形式落盘,例如 `session_id`、`user_id`、`time_cost_ms`、`prompt_tokens`、`completion_tokens`、`total_tokens`、`iteration_count`、`tool_count`、`tools_used_names`、`response_length`、`created_at`。3. `feedback & outcome`:最后在 `response_id` 基础上补反馈事件与隐式结果判断,否则后续指标会缺少关联主键。这意味着:@@ -141,6 +141,13 @@| `csat_score` | 用户评分均值 | 适用于 5 分制或 10 分制满意度 || `dissatisfaction_reason_distribution` | 各类差评原因占比 | 用于定位主要失败模式 |+当前实现口径补充:++- `responses_total` 以 session JSONL 中所有 `role == "assistant"` 且带 `response_id` 的最终回答为准。+- `feedback_coverage` 的分母是 `responses_total`,分子是出现过显式 feedback 的去重 `response_id` 数量。+- `thumbs_up_rate` / `thumbs_down_rate` 当前仍以 `feedback_total` 为分母,用于衡量显式反馈内部的正负分布。+- `positive_feedback_rate` / `negative_feedback_rate` / `reask_rate` / `one_turn_resolution_rate` 当前统一以 `responses_total` 为分母,用于衡量全部最终回答上的结果占比。+差评原因建议最少支持以下标签:- `irrelevant`: 答非所问@@ -381,6 +388,13 @@ good_answer_rate =该事件在最终回答产生时记录,是整套分析的主事实表。+当前实现状态补充:++1. `response_completed` 当前已经在 `AgentLoop` 中标准化构建。+2. 同一份 payload 会写入 `session.metadata["response_facts"][response_id]`,并随 session save 持久化到 JSONL metadata 首行。+3. 同一份 payload 也会写入 Langfuse generation metadata。+4. 该事件仍然是 analytics-only,不会暴露到 OpenAPI 对外返回或用户可见 channel。+建议字段:| 字段名 | 说明 |@@ -462,6 +476,8 @@ good_answer_rate =4. 当前规则版优先使用显式 `thumb_up` / `thumb_down`,否则结合 10 分钟内 follow-up、后续 user turn 数和是否缺少反馈来推导 outcome。5. 当前实现是 Phase 3 的最小可用版本,不等同于完整离线 judge 或评审模型。+补充说明:`response_outcomes` 当前只覆盖“已经被显式反馈或被后处理规则评估过”的回答,不能等价替代总回答事实表。因此 summary / channel 聚合中的 `responses_total` 不能从 `response_outcomes` 推导,而应从 assistant `response_id` 记录统计。+该事件建议放在第三阶段持续增强,因为它依赖:- session 历史中能稳定关联 user / assistant 消息@@ -693,7 +709,7 @@ MVP 指标集建议为:7. `avg_iteration_count`8. `avg_tool_count`-说明:第一阶段不再把 `feedback_coverage`、`thumbs_up_rate`、`thumbs_down_rate`、`good_answer_rate`、`one_turn_resolution_rate`、`reask_rate` 作为必须指标,因为它们当前没有数据入口或没有稳定计算依据。+说明:原始 Phase 1 规划里,这些结果指标并不属于必须项;但截至当前工作树,`feedback_coverage`、`thumbs_up_rate`、`thumbs_down_rate`、`one_turn_resolution_rate`、`reask_rate` 已经具备最小可用的数据入口与离线聚合能力。当前需要强调的不是“是否存在”,而是“口径是否统一”,尤其是 `responses_total` 必须基于所有 assistant `response_id`,而不是 `response_outcomes`。## 10.2 Phase 2: 增强归因能力@@ -750,7 +766,7 @@ MVP 指标集建议为:2. 在 `AgentLoop._process_message` 生成最终 `OutboundMessage` 前创建 `response_id`。3. 在 `session.add_message("assistant", ...)` 时把 `response_id` 一并写入 JSONL 消息。4. 在 OpenAPI `ChatResponse` 与流式最终事件中返回 `response_id`,让客户端具备回传反馈的主键。-5. 新增 `response_completed` 事件写入点,优先在 agent loop 最终回答落盘后触发。+5. 在 agent loop 中构建标准化 `response_completed`,并随 session save 将其写入 `session.metadata["response_facts"]`,同时继续以 analytics-only 事件形式发布。6. 在 Langfuse generation / tool span metadata 中写入 `response_id`、`iteration_count`、`tool_count`、`tool_names`。第二阶段再补:@@ -816,3 +832,23 @@ vikingbot 的问答效果反馈观测,不能只停留在 token、trace 和工- 哪些回答正在变差。- 为什么变差。- 改完以后是否真的变好。++---++## 14. Verification++如需对当前最小实现做 targeted 验证,推荐使用 `uv` 执行以下回归集:++```bash+uv run --extra test --extra bot -m pytest bot/tests/test_feedback_stats.py bot/tests/test_openapi_auth.py bot/tests/test_outcome_evaluator.py bot/tests/test_agent_loop_outcome.py tests/metrics/collectors/test_feedback_collector.py tests/server/test_prometheus_metrics.py+```++预期结果:++1. feedback stats、OpenAPI feedback、outcome evaluator、agent loop `response_facts` 持久化相关用例全部通过+2. feedback collector 与 `/metrics` 暴露相关用例全部通过++## 15. Related Docs++- [OpenViking Metrics 概念文档](../../docs/zh/concepts/12-metrics.md) - feedback 指标族、PromQL 示例与 `/metrics` 暴露说明+- [OpenViking Metrics API 文档](../../docs/zh/api/09-metrics.md) - `/metrics` 端点行为与抓取方式diff --git a/bot/docs/vikingbot-phase1-validation-with-openviking-server.md b/bot/docs/vikingbot-phase1-validation-with-openviking-server.mdindex b0081b268..3b26c746c 100644--- a/bot/docs/vikingbot-phase1-validation-with-openviking-server.md+++ b/bot/docs/vikingbot-phase1-validation-with-openviking-server.md@@ -10,12 +10,13 @@本文用于验证 Vikingbot feedback observability Phase 1 当前已经落地的能力,验证方式基于 `openviking-server --with-bot` 启动模式。-本次 Phase 1 需要确认的核心点有四个:+本次 Phase 1 需要确认的核心点有五个:1. OpenAPI 返回中存在稳定的 `response_id`2. session JSONL 中 assistant message 持久化了 `response_id`3. Langfuse 的 generation / tool span metadata 中带有 `response_id`-4. `response_completed` 没有暴露到用户侧 channel 或 OpenAPI 对外返回中+4. session JSONL metadata 首行中存在按 `response_id` 组织的 `response_facts`+5. `response_completed` 没有暴露到用户侧 channel 或 OpenAPI 对外返回中---@@ -256,7 +257,48 @@ ls ~/.openviking/data/bot/sessions1. generation metadata 中存在 `response_id`2. Langfuse 中看到的 `response_id` 与 HTTP 返回一致-### 5.6 验证 Langfuse tool span metadata 包含 `response_id`+### 5.6 验证 session metadata 中已写入 `response_facts`++除了 assistant message 上的 `response_id`,当前 Phase 1 还会把标准化后的 `response_completed` facts 持久化到 session JSONL 首行 metadata 中。++找到第 5.4 步对应的 session 文件后,检查首行 metadata,预期结构类似:++```json+{+ "session_key": {+ "type": "bot_api",+ "channel_id": "default",+ "chat_id": "phase1-verify-session"+ },+ "metadata": {+ "response_facts": {+ "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx": {+ "response_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",+ "session_id": "bot_api__default__phase1-verify-session",+ "user_id": "phase1-verify-user",+ "channel": "bot_api__default",+ "time_cost_ms": 123,+ "prompt_tokens": 123,+ "completion_tokens": 45,+ "total_tokens": 168,+ "iteration_count": 1,+ "tool_count": 0,+ "tools_used_names": [],+ "response_length": 12,+ "created_at": "2026-04-30T00:00:00"+ }+ }+ }+}+```++重点检查:++1. 首行 metadata 下存在 `response_facts`+2. `response_facts` 里存在以该次 HTTP 返回 `response_id` 为 key 的记录+3. 该记录里的 `response_id`、`session_id`、`user_id` 与本次请求一致++### 5.7 验证 Langfuse tool span metadata 包含 `response_id`再发一条更容易触发工具的问题,例如:@@ -283,7 +325,7 @@ curl -sS -X POST "http://127.0.0.1:1933/bot/v1/chat" \1. tool span metadata 中存在 `response_id`2. generation 与 tool span 使用的是同一个 `response_id`-### 5.7 验证 `response_completed` 没有对外暴露+### 5.8 验证 `response_completed` 没有对外暴露`response_completed` 当前是内部分析事件,不应直接暴露给 OpenAPI 客户端,也不应出现在用户可见 channel 中。@@ -319,9 +361,10 @@ curl -sS -X POST "http://127.0.0.1:1933/bot/v1/chat" \1. `/bot/v1/chat` 返回顶层 `response_id`2. 最终 `response` 事件的 `data.response_id` 存在3. session JSONL 的 assistant message 中写入了 `response_id`-4. Langfuse generation metadata 中存在 `response_id`-5. 如果触发工具,Langfuse tool span metadata 中也存在 `response_id`-6. OpenAPI 或用户侧 channel 没有暴露 `response_completed`+4. session JSONL metadata 首行中存在对应 `response_id` 的 `response_facts`+5. Langfuse generation metadata 中存在 `response_id`+6. 如果触发工具,Langfuse tool span metadata 中也存在 `response_id`+7. OpenAPI 或用户侧 channel 没有暴露 `response_completed`---diff --git a/bot/docs/vikingbot-phase2-feedback-validation-with-openviking-server.md b/bot/docs/vikingbot-phase2-feedback-validation-with-openviking-server.mdindex 63eed660e..17b05416c 100644--- a/bot/docs/vikingbot-phase2-feedback-validation-with-openviking-server.md+++ b/bot/docs/vikingbot-phase2-feedback-validation-with-openviking-server.md@@ -40,6 +40,13 @@ openviking-server --with-bot本文不把 `http://127.0.0.1:18790` 视为主验证入口。+在执行本文前,建议先完成 Phase 1 验证,至少确认以下前置条件已经成立:++1. `/bot/v1/chat` 返回体中稳定包含 `response_id`+2. session JSONL metadata 首行中已存在按 `response_id` 组织的 `response_facts`++Phase 2 的 `feedback_events` 关联验证,默认建立在这条 Phase 1 response identity + response facts 基础链路已经打通之上。+---## 3. 当前已验证结论diff --git a/bot/docs/vikingbot-phase3-outcome-validation-with-openviking-server.md b/bot/docs/vikingbot-phase3-outcome-validation-with-openviking-server.mdindex f5a509f05..1ddfa2ddb 100644--- a/bot/docs/vikingbot-phase3-outcome-validation-with-openviking-server.md+++ b/bot/docs/vikingbot-phase3-outcome-validation-with-openviking-server.md@@ -41,6 +41,14 @@ OPENVIKING_CLI_CONFIG_FILE=ov_conf/ovcli.conf openviking-server --with-bot --con本文优先验证真实代理路径,而不是直接访问 `http://127.0.0.1:18790`。+在执行本文前,建议先完成 Phase 1 与 Phase 2 的最小验证,至少确认以下前置条件已经成立:++1. `/bot/v1/chat` 返回体中稳定包含 `response_id`+2. session JSONL metadata 首行中已存在按 `response_id` 组织的 `response_facts`+3. 显式 feedback 已能写入 `metadata.feedback_events`++Phase 3 的 `response_outcomes` 评估与回写,默认建立在这几条 response identity / response facts / feedback persistence 基础链路已经打通之上。+---## 3. 当前已验证结论diff --git a/bot/docs/vikingbot-real-user-metrics-cases.md b/bot/docs/vikingbot-real-user-metrics-cases.mdnew file mode 100644index 000000000..b3a47d06d--- /dev/null+++ b/bot/docs/vikingbot-real-user-metrics-cases.md@@ -0,0 +1,500 @@+# Vikingbot 真实用户问答指标验证案例++这份文档给出一组可以直接拿来执行的真实用户问答案例,用来逐项验证本次补充的 feedback / outcome 指标是否正常。++本文只覆盖当前代码里已经落地的指标口径:++- `openviking_feedback_responses_total`+- `openviking_feedback_responses_with_feedback_total`+- `openviking_feedback_events_total`+- `openviking_feedback_thumb_up_total`+- `openviking_feedback_thumb_down_total`+- `openviking_feedback_positive_outcomes_total`+- `openviking_feedback_negative_outcomes_total`+- `openviking_feedback_reasked_outcomes_total`+- `openviking_feedback_resolved_outcomes_total`+- `openviking_feedback_follow_up_without_feedback_outcomes_total`+- `openviking_feedback_coverage`+- `openviking_feedback_thumbs_up_rate`+- `openviking_feedback_thumbs_down_rate`+- `openviking_feedback_positive_feedback_rate`+- `openviking_feedback_negative_feedback_rate`+- `openviking_feedback_reask_rate`+- `openviking_feedback_one_turn_resolution_rate`+- `openviking_feedback_channel_*`++## 1. 验证前说明++当前指标不是在线自增 counter 语义,而是 scrape `/metrics` 时基于 `bot/sessions/*.jsonl` 聚合出来的快照值。++因此验证时建议遵循两个原则:++1. 优先看 total 是否按预期跳变。+2. rate 类指标只看方向是否合理,不要求每次都出现明显大幅变化。++建议所有查询都先带上 `valid="1"`。++当前分母口径说明:++1. `responses_total` 统计的是 session JSONL 中所有带 `response_id` 的 assistant 最终回答,不要求这些回答已经进入 `response_outcomes`。+2. `feedback_coverage`、`positive_feedback_rate`、`negative_feedback_rate`、`reask_rate`、`one_turn_resolution_rate` 都是基于 `responses_total` 重新聚合出来的快照比例。+3. `thumbs_up_rate`、`thumbs_down_rate` 仍然只在显式 feedback 样本内部计算,因此它们的分母是 `feedback_total`,不是 `responses_total`。++## 2. 环境前提++需要先确认:++1. `http://127.0.0.1:30300/bot/v1/health` 返回 `200`。+2. `http://127.0.0.1:30300/metrics` 能正常返回 Prometheus 文本。+3. 使用的是当前这一版已包含 feedback/outcome 聚合逻辑的服务。++建议先记一份基线值:++```promql+openviking_feedback_responses_total{valid="1"}+openviking_feedback_responses_with_feedback_total{valid="1"}+openviking_feedback_events_total{valid="1"}+openviking_feedback_thumb_up_total{valid="1"}+openviking_feedback_thumb_down_total{valid="1"}+openviking_feedback_positive_outcomes_total{valid="1"}+openviking_feedback_negative_outcomes_total{valid="1"}+openviking_feedback_reasked_outcomes_total{valid="1"}+openviking_feedback_resolved_outcomes_total{valid="1"}+openviking_feedback_follow_up_without_feedback_outcomes_total{valid="1"}+```++## 3. 案例 1:正常问答落盘++目的:验证一次普通问答至少会进入 `responses_total`。++真实用户问题:++```text+请用一句话介绍 OpenViking,控制在 20 个字以内。+```++预期 Vikingbot 回答特征:++- 正常返回文本+- 返回体中带 `response_id`+- 语义上是简短介绍++请求示例:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "real-metrics-case-01",+ "user_id": "metrics-user",+ "message": "请用一句话介绍 OpenViking,控制在 20 个字以内。"+ }'+```++重点核对指标:++```promql+openviking_feedback_responses_total{valid="1"}+```++预期结果:++1. 会变:`openviking_feedback_responses_total{valid="1"}` 通常增加 `1`;如果这是一个全新的 session 文件,`openviking_feedback_sessions_scanned_total{valid="1"}` 也可能增加 `1`。+2. 不该变:`events_total`、`thumb_up_total`、`thumb_down_total`、`responses_with_feedback_total` 以及各类 outcome total 在这一步通常不应直接增加,因为这里只发生了一次普通 `/bot/v1/chat`,没有显式 feedback,也没有 follow-up。+3. 可能不明显:如果历史样本很多,rate 类指标通常不会因为这一轮普通问答出现明显波动;这个场景优先看 total 是否跳变,不要把 rate 没变化当成异常。++## 4. 案例 2:用户点赞,验证正向反馈链路++目的:验证 `thumb_up`、`positive_feedback`、`one_turn_resolution_rate` 相关口径。++真实用户问题:++```text+帮我总结一下 OpenViking 的主要作用。+```++预期 Vikingbot 回答特征:++- 是一段简短总结+- 返回体中带 `response_id`++真实用户后续行为:++```text+这次回答有帮助,我给一个赞。+```++执行步骤:++1. 先发 `/bot/v1/chat`。+2. 记下返回里的 `response_id`。+3. 对同一条响应提交 `thumb_up`。++聊天请求:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "real-metrics-case-02",+ "user_id": "metrics-user",+ "message": "帮我总结一下 OpenViking 的主要作用。"+ }'+```++反馈请求:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/feedback" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "real-metrics-case-02",+ "response_id": "<response_id>",+ "feedback_type": "thumb_up",+ "feedback_text": "helpful"+ }'+```++重点核对指标:++```promql+openviking_feedback_events_total{valid="1"}+openviking_feedback_thumb_up_total{valid="1"}+openviking_feedback_responses_with_feedback_total{valid="1"}+openviking_feedback_positive_outcomes_total{valid="1"}+openviking_feedback_coverage{valid="1"}+openviking_feedback_thumbs_up_rate{valid="1"}+openviking_feedback_one_turn_resolution_rate{valid="1"}+```++预期结果:++1. 会变:`events_total`、`thumb_up_total`、`responses_with_feedback_total`、`positive_outcomes_total` 通常各增加 `1`;如果你先看过基线,再在一次 scrape 之后复查,这几个 total 最容易确认。+2. 不该变:`thumb_down_total`、`negative_outcomes_total`、`reasked_outcomes_total`、`follow_up_without_feedback_outcomes_total` 在这个场景通常不应直接增加,因为当前动作是对同一条 response 提交显式正向反馈,而不是差评或追问。+3. 可能不明显:`coverage`、`thumbs_up_rate`、`one_turn_resolution_rate` 通常会上升或保持不变,但如果历史 response 和历史 feedback 已经很多,显示上也可能接近不变;其中 `thumbs_up_rate` 的分母是 `feedback_total`,`one_turn_resolution_rate` 的分母是 `responses_total`,并且当前实现会把 `resolved + positive_feedback` 都算入 one-turn resolution。++## 5. 案例 3:用户点踩,验证负向反馈链路++目的:验证 `thumb_down`、`negative_feedback` 相关口径。++真实用户问题:++```text+请告诉我 OpenViking 的部署步骤,越短越好。+```++预期 Vikingbot 回答特征:++- 返回了一个部署说明+- 返回体中带 `response_id`++真实用户后续行为:++```text+这次回答不够有帮助,我给一个踩。+```++聊天请求:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "real-metrics-case-03",+ "user_id": "metrics-user",+ "message": "请告诉我 OpenViking 的部署步骤,越短越好。"+ }'+```++反馈请求:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/feedback" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "real-metrics-case-03",+ "response_id": "<response_id>",+ "feedback_type": "thumb_down",+ "feedback_text": "not helpful"+ }'+```++重点核对指标:++```promql+openviking_feedback_events_total{valid="1"}+openviking_feedback_thumb_down_total{valid="1"}+openviking_feedback_negative_outcomes_total{valid="1"}+openviking_feedback_thumbs_down_rate{valid="1"}+openviking_feedback_negative_feedback_rate{valid="1"}+```++预期结果:++1. 会变:`events_total`、`thumb_down_total`、`negative_outcomes_total` 通常各增加 `1`;如果基线样本不多,这几个 total 的跳变会比较直观。+2. 不该变:`thumb_up_total`、`positive_outcomes_total`、`reasked_outcomes_total`、`resolved_outcomes_total`、`follow_up_without_feedback_outcomes_total` 在这个场景通常不应直接增加,因为这里只提交了显式负向反馈,没有发生追问或静默结束。+3. 可能不明显:`thumbs_down_rate`、`negative_feedback_rate` 通常会上升或保持不变,但如果历史样本很多,比例变化可能很小;其中 `thumbs_down_rate` 的分母是 `feedback_total`,`negative_feedback_rate` 的分母是 `responses_total`,因此仍然优先看 total 是否按预期跳变。++## 6. 案例 4:用户马上追问,验证 `reasked`++目的:验证“无显式反馈,但 10 分钟内追问”会被判定为 `reasked`。++真实用户第一问:++```text+请解释 OpenViking 的 metrics 是做什么的。+```++预期 Vikingbot 第一轮回答特征:++- 给出解释+- 用户主观感受是“还不够具体”++真实用户第二问:++```text+我还是没明白,请再具体一点。+```++执行要求:++1. 两次 `/bot/v1/chat` 必须使用同一个 `session_id`。+2. 第二次提问必须发生在第一条 assistant 回复后的 10 分钟内。+3. 不要提交 `/bot/v1/feedback`。++第一轮请求:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "real-metrics-case-04",+ "user_id": "metrics-user",+ "message": "请解释 OpenViking 的 metrics 是做什么的。"+ }'+```++第二轮请求:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "real-metrics-case-04",+ "user_id": "metrics-user",+ "message": "我还是没明白,请再具体一点。"+ }'+```++重点核对指标:++```promql+openviking_feedback_reasked_outcomes_total{valid="1"}+openviking_feedback_reask_rate{valid="1"}+```++预期结果:++1. `reasked_outcomes_total` 增加 `1`。+2. `reask_rate` 不下降,通常会上升或保持不变;它的分母是全量历史 `responses_total`,不是已经写入 outcome 的回答子集。+3. `responses_total` 通常还会因为第二次 `/bot/v1/chat` 再增加 `1`,但真正被打上 `reasked` 的是第一轮 assistant response,而不是第二轮新生成的 response。+4. `events_total`、`thumb_up_total`、`thumb_down_total`、`responses_with_feedback_total` 通常不变,因为这个场景没有显式调用 `/bot/v1/feedback`。+5. `positive_outcomes_total`、`negative_outcomes_total`、`resolved_outcomes_total`、`follow_up_without_feedback_outcomes_total` 通常不应因为这一步直接增加;如果你看到增长,通常说明混入了其他验证样本,或第二次提问已经越过 10 分钟窗口而被判成了别的 outcome。++补充说明:++1. `reask_rate` 之所以写成“通常上升或保持不变”,而不是“必然上升”,是因为这是一个基于全量历史 response 重新聚合的比例;如果历史样本很多,分母同步变化后也可能在显示上保持不变。+2. 这个场景最关键的验证点不是第二条回复内容,而是第二次 user turn 到来后,第一条 assistant response 的 outcome 被回填成了 `reasked`。+3. `reasked` 和 `follow_up_without_feedback` 的边界是 follow-up 是否发生在上一条 assistant response 之后 10 分钟内:10 分钟内优先归类为 `reasked`,超过该窗口且又没有显式反馈时,才更可能落入 `follow_up_without_feedback`。++## 7. 案例 5:用户过一段时间再追问,验证 `follow_up_without_feedback`++目的:验证“没有点赞/点踩,也不是 10 分钟内追问”的 follow-up 会被归为 `follow_up_without_feedback`。++真实用户第一问:++```text+OpenViking 和普通向量数据库有什么关系?+```++真实用户第二问:++```text+那你能再举一个更具体的例子吗?+```++执行要求:++1. 使用同一个 `session_id`。+2. 不提交 `/bot/v1/feedback`。+3. 第二次提问必须晚于第一条 assistant 回复 10 分钟之后。++第一轮请求:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "real-metrics-case-05",+ "user_id": "metrics-user",+ "message": "OpenViking 和普通向量数据库有什么关系?"+ }'+```++第二轮请求:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "real-metrics-case-05",+ "user_id": "metrics-user",+ "message": "那你能再举一个更具体的例子吗?"+ }'+```++重点核对指标:++```promql+openviking_feedback_follow_up_without_feedback_outcomes_total{valid="1"}+```++预期结果:++1. 会变:`follow_up_without_feedback_outcomes_total` 通常增加 `1`;同时第二次 `/bot/v1/chat` 也会让 `responses_total` 再增加 `1`,但真正被补判为 `follow_up_without_feedback` 的仍然是第一轮 assistant response。+2. 不该变:`events_total`、`thumb_up_total`、`thumb_down_total`、`responses_with_feedback_total` 通常不变,因为整个场景没有显式调用 `/bot/v1/feedback`;`positive_outcomes_total`、`negative_outcomes_total`、`reasked_outcomes_total` 也通常不应在这一步直接增长。+3. 可能不明显:如果你看到增长落在 `reasked_outcomes_total`,优先检查第二次提问是不是仍在 10 分钟窗口内;rate 类指标同样可能因为历史样本较多而变化不明显,所以这个场景优先看 total 跳变。++## 8. 案例 6:用户问完就结束,验证 `resolved`++目的:补充验证“没有追问、没有负向反馈”的响应会被统计为 `resolved`。++真实用户问题:++```text+请用一句话解释什么是 OpenViking 的 readiness 指标。+```++预期 Vikingbot 回答特征:++- 给出简短说明+- 用户不继续追问+- 用户不提交 feedback++请求示例:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "real-metrics-case-06",+ "user_id": "metrics-user",+ "message": "请用一句话解释什么是 OpenViking 的 readiness 指标。"+ }'+```++重点核对指标:++```promql+openviking_feedback_resolved_outcomes_total{valid="1"}+openviking_feedback_one_turn_resolution_rate{valid="1"}+```++预期结果:++1. 会变:`resolved_outcomes_total` 有机会增加 `1`,`one_turn_resolution_rate` 也可能上升或保持不变;当前该 rate 的分母是全量 `responses_total`。+2. 不该变:`events_total`、`thumb_up_total`、`thumb_down_total`、`responses_with_feedback_total` 在这个场景通常不变,因为既没有显式 feedback,也没有后续追问;`negative_outcomes_total`、`reasked_outcomes_total`、`follow_up_without_feedback_outcomes_total` 也通常不应直接增加。+3. 可能不明显:`resolved` 相比 `thumb_up`、`thumb_down`、`reasked` 更适合作为补充验证项,因为它依赖后续 outcome 评估时机,出现时间和幅度都不如显式反馈稳定;因此看到短时间内没有立刻跳变,不一定表示链路异常。++说明:`resolved` 相比 `thumb_up`、`thumb_down`、`reasked` 更适合作为补充验证项,不建议当成唯一验收依据。++## 9. 案例 7:多 channel 验证 `openviking_feedback_channel_*`++目的:验证 channel 维度聚合是否正常。++前提:环境里已经配置了 `bot_api` 的某个 channel,例如 `demo`,对应聚合维度应为 `bot_api__demo`。++真实用户问题:++```text+请简单介绍一下 OpenViking。+```++真实用户后续行为:++```text+这次回答没帮助,我给一个踩。+```++聊天请求:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat/channel" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "real-metrics-case-07",+ "user_id": "metrics-user",+ "channel_id": "demo",+ "message": "请简单介绍一下 OpenViking。"+ }'+```++反馈请求:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/feedback" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "real-metrics-case-07",+ "channel_id": "demo",+ "response_id": "<response_id>",+ "feedback_type": "thumb_down",+ "feedback_text": "not helpful"+ }'+```++重点核对指标:++```promql+openviking_feedback_channel_events_total{channel="bot_api__demo",valid="1"}+openviking_feedback_channel_thumbs_down_rate{channel="bot_api__demo",valid="1"}+openviking_feedback_channel_negative_outcomes_total{channel="bot_api__demo",valid="1"}+openviking_feedback_channel_one_turn_resolution_rate{channel="bot_api__demo",valid="1"}+```++预期结果:++1. 会变:`channel_events_total{channel="bot_api__demo"}`、`channel_negative_outcomes_total{channel="bot_api__demo"}` 通常增加;如果你同时看 channel 维度和全局维度,两边都应体现这次负向反馈。+2. 不该变:其他无关 channel 的样本通常不应因为这次请求跳变;如果你误用的是 `/bot/v1/chat` 而不是 `/bot/v1/chat/channel`,通常也不会得到目标 `channel="bot_api__demo"` 的聚合样本。+3. 可能不明显:`channel_thumbs_down_rate{channel="bot_api__demo"}`、`channel_one_turn_resolution_rate{channel="bot_api__demo"}` 这类比例在历史 channel 样本较多时可能变化不明显,所以 channel 验收优先看该 channel 的 total 是否跳变,再看 rate 方向是否合理。++## 10. 最小验收顺序++如果只想用最少案例覆盖这次新增指标,建议按下面顺序执行:++1. 案例 1:确认问答落盘,`responses_total` 正常。+2. 案例 2:确认 `thumb_up`、`positive_outcomes_total`、`coverage`、`one_turn_resolution_rate` 正常。+3. 案例 3:确认 `thumb_down`、`negative_outcomes_total` 正常。+4. 案例 4:确认 `reasked_outcomes_total`、`reask_rate` 正常。+5. 案例 5:确认 `follow_up_without_feedback_outcomes_total` 正常。+6. 案例 6:把 `resolved` 作为补充验证。+7. 案例 7:如果你的环境启用了多 channel,再验证 channel 维度。++## 11. 通过标准++满足下面条件,可以认为本次补充的主要指标链路正常:++1. 普通问答后,`openviking_feedback_responses_total{valid="1"}` 增加。+2. 点赞后,`openviking_feedback_events_total`、`openviking_feedback_thumb_up_total`、`openviking_feedback_positive_outcomes_total` 增加。+3. 点踩后,`openviking_feedback_thumb_down_total`、`openviking_feedback_negative_outcomes_total` 增加。+4. 10 分钟内追问后,`openviking_feedback_reasked_outcomes_total` 增加。+5. 10 分钟后 follow-up 且无 feedback 时,`openviking_feedback_follow_up_without_feedback_outcomes_total` 增加。+6. 如环境启用了多 channel,`openviking_feedback_channel_*` 会按目标 channel 出现并跳变。++补充判定原则:如果 rate 没有明显跳变,但对应 total 已按预期变化,通常仍可视为链路正常,因为这些 rate 都是对全量历史回答重新聚合的快照值。++## 12. 相关文档++- `bot/docs/vikingbot-feedback-observability-design.md`+- `bot/docs/vikingbot-phase3-outcome-validation-with-openviking-server.md`+- `docs/zh/guides/12-vikingbot-metrics-validation.md`diff --git a/bot/tests/test_agent_loop_outcome.py b/bot/tests/test_agent_loop_outcome.pyindex c446eb4ed..473220206 100644--- a/bot/tests/test_agent_loop_outcome.py+++ b/bot/tests/test_agent_loop_outcome.py@@ -1,3 +1,4 @@+import jsonfrom datetime import datetimefrom pathlib import Path@@ -23,6 +24,15 @@ class _FakeSubagentManager:self.kwargs = kwargs+class _FakeLangfuseClient:+ def __init__(self):+ self.calls = []++ def update_generation_metadata(self, response_id, metadata):+ self.calls.append((response_id, metadata))+ return metadata++@pytest.mark.asyncioasync def test_agent_loop_evaluates_previous_response_outcome_before_new_user_turn(temp_dir: Path, monkeypatch@@ -120,3 +130,75 @@ async def test_agent_loop_ignores_heartbeat_when_evaluating_previous_response_oupersisted_session = loop.sessions.get_or_create(session_key, skip_heartbeat=False)assert "response_outcomes" not in persisted_session.metadata+++async def test_agent_loop_emits_normalized_response_completed_payload(temp_dir: Path, monkeypatch):+ monkeypatch.setattr(AgentLoop, "_register_builtin_hooks", lambda self: None)+ monkeypatch.setattr(AgentLoop, "_register_default_tools", lambda self: None)+ monkeypatch.setattr("vikingbot.agent.loop.SubagentManager", _FakeSubagentManager)++ fake_langfuse = _FakeLangfuseClient()+ monkeypatch.setattr(+ "vikingbot.agent.loop.LangfuseClient.get_instance",+ staticmethod(lambda: fake_langfuse),+ )++ async def fake_run_agent_loop(self, **kwargs):+ return (+ "final answer",+ None,+ [{"tool_name": "search_docs"}, {"tool_name": "fetch_page"}],+ {"prompt_tokens": 12, "completion_tokens": 8},+ 3,+ )++ monkeypatch.setattr(AgentLoop, "_run_agent_loop", fake_run_agent_loop)++ bus = MessageBus()+ config = Config(storage_workspace=str(temp_dir))+ loop = AgentLoop(+ bus=bus,+ provider=_FakeProvider(),+ workspace=temp_dir / "workspace",+ config=config,+ )++ session_key = SessionKey(type="cli", channel_id="default", chat_id="session-1")+ response = await loop._process_message(+ InboundMessage(+ session_key=session_key,+ sender_id="user-1",+ content="please help",+ timestamp=datetime.fromisoformat("2026-04-30T00:05:00"),+ )+ )++ assert response is not None+ assert response.content == "final answer"+ assert response.response_id is not None+ assert bus.outbound_size == 1++ completed_event = await bus.consume_outbound()+ assert completed_event.event_type == OutboundEventType.RESPONSE_COMPLETED+ payload = completed_event.metadata["response_completed"]+ assert payload["response_id"] == response.response_id+ assert payload["session_id"] == "cli__default__session-1"+ assert payload["channel"] == "cli__default"+ assert payload["session_type"] == "cli"+ assert payload["user_id"] == "user-1"+ assert payload["prompt_tokens"] == 12+ assert payload["completion_tokens"] == 8+ assert payload["total_tokens"] == 20+ assert payload["iteration_count"] == 3+ assert payload["tool_count"] == 2+ assert payload["tools_used_names"] == ["search_docs", "fetch_page"]+ assert payload["response_length"] == len("final answer")+ assert payload["has_reasoning"] is False+ assert payload["time_cost_ms"] >= 0+ assert payload["created_at"]+ assert fake_langfuse.calls == [(response.response_id, payload)]++ session_path = temp_dir / "bot" / "sessions" / "cli__default__session-1.jsonl"+ metadata = json.loads(session_path.read_text().splitlines()[0])+ assert metadata["metadata"]["response_facts"][response.response_id] == payloaddiff --git a/bot/tests/test_feedback_stats.py b/bot/tests/test_feedback_stats.pynew file mode 100644index 000000000..5a31ace7d--- /dev/null+++ b/bot/tests/test_feedback_stats.py@@ -0,0 +1,768 @@+import json++import pytest++from vikingbot.observability.feedback_stats import (+ FEEDBACK_STATS_SORT_FIELDS,+ build_feedback_stats_display,+ compute_feedback_stats,+ format_feedback_stats_markdown,+ format_feedback_stats_table,+ select_feedback_stats,+ validate_feedback_stats_sort_by,+)+++def test_compute_feedback_stats_aggregates_minimal_metrics(temp_dir):+ sessions_dir = temp_dir / "bot" / "sessions"+ sessions_dir.mkdir(parents=True)++ (sessions_dir / "cli__default__session-1.jsonl").write_text(+ "\n".join(+ [+ json.dumps(+ {+ "_type": "metadata",+ "session_key": "cli__default__session-1",+ "metadata": {+ "feedback_events": [+ {"response_id": "resp-1", "feedback_type": "thumb_up"}+ ],+ "response_outcomes": {+ "resp-1": {"outcome_label": "positive_feedback"},+ "resp-2": {"outcome_label": "reasked"},+ },+ },+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 1",+ "response_id": "resp-1",+ "timestamp": "2026-05-01T10:00:00",+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 2",+ "response_id": "resp-2",+ "timestamp": "2026-05-01T10:01:00",+ }+ ),+ ]+ ),+ encoding="utf-8",+ )++ (sessions_dir / "bot_api__demo__session-2.jsonl").write_text(+ "\n".join(+ [+ json.dumps(+ {+ "_type": "metadata",+ "session_key": "bot_api__demo__session-2",+ "metadata": {+ "feedback_events": [+ {"response_id": "resp-3", "feedback_type": "thumb_down"},+ {"response_id": "resp-3", "feedback_type": "thumb_down"},+ ],+ "response_outcomes": {+ "resp-3": {"outcome_label": "negative_feedback"},+ "resp-4": {"outcome_label": "resolved"},+ "resp-5": {"outcome_label": "follow_up_without_feedback"},+ },+ },+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 3",+ "response_id": "resp-3",+ "timestamp": "2026-05-03T10:00:00",+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 4",+ "response_id": "resp-4",+ "timestamp": "2026-05-03T10:01:00",+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 5",+ "response_id": "resp-5",+ "timestamp": "2026-05-03T10:02:00",+ }+ ),+ ]+ ),+ encoding="utf-8",+ )++ stats = compute_feedback_stats(temp_dir / "bot")++ assert stats["summary"]["sessions_scanned"] == 2+ assert stats["summary"]["responses_total"] == 5+ assert stats["summary"]["tracked_responses_total"] == 5+ assert stats["summary"]["responses_with_feedback"] == 2+ assert stats["summary"]["feedback_total"] == 3+ assert stats["summary"]["thumb_up_total"] == 1+ assert stats["summary"]["thumb_down_total"] == 2+ assert stats["summary"]["positive_feedback_total"] == 1+ assert stats["summary"]["negative_feedback_total"] == 1+ assert stats["summary"]["reasked_total"] == 1+ assert stats["summary"]["resolved_total"] == 1+ assert stats["summary"]["follow_up_without_feedback_total"] == 1+ assert stats["summary"]["feedback_coverage"] == 0.4+ assert stats["summary"]["thumbs_up_rate"] == 0.3333+ assert stats["summary"]["thumbs_down_rate"] == 0.6667+ assert stats["summary"]["positive_feedback_rate"] == 0.2+ assert stats["summary"]["negative_feedback_rate"] == 0.2+ assert stats["summary"]["reask_rate"] == 0.2+ assert stats["summary"]["one_turn_resolution_rate"] == 0.4++ assert stats["channels"]["cli__default"]["responses_total"] == 2+ assert stats["channels"]["cli__default"]["thumbs_up_rate"] == 1.0+ assert stats["channels"]["bot_api__demo"]["responses_total"] == 3+ assert stats["channels"]["bot_api__demo"]["thumbs_down_rate"] == 1.0+++def test_compute_feedback_stats_ignores_missing_sessions_dir(temp_dir):+ stats = compute_feedback_stats(temp_dir / "bot")++ assert stats["summary"]["sessions_scanned"] == 0+ assert stats["summary"]["responses_total"] == 0+ assert stats["summary"]["feedback_coverage"] == 0.0+ assert stats["channels"] == {}+ assert stats["sessions"] == []+++def test_compute_feedback_stats_supports_filters(temp_dir):+ sessions_dir = temp_dir / "bot" / "sessions"+ sessions_dir.mkdir(parents=True)++ (sessions_dir / "cli__default__session-1.jsonl").write_text(+ "\n".join(+ [+ json.dumps(+ {+ "_type": "metadata",+ "session_key": "cli__default__session-1",+ "created_at": "2026-05-01T10:00:00",+ "updated_at": "2026-05-01T10:00:00",+ "metadata": {+ "feedback_events": [+ {"response_id": "resp-1", "feedback_type": "thumb_up"}+ ],+ "response_outcomes": {+ "resp-1": {"outcome_label": "positive_feedback"}+ },+ },+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 1",+ "response_id": "resp-1",+ "timestamp": "2026-05-01T10:00:00",+ }+ ),+ ]+ )+ + "\n",+ encoding="utf-8",+ )+ (sessions_dir / "bot_api__demo__session-2.jsonl").write_text(+ "\n".join(+ [+ json.dumps(+ {+ "_type": "metadata",+ "session_key": "bot_api__demo__session-2",+ "created_at": "2026-05-03T10:00:00",+ "updated_at": "2026-05-03T10:00:00",+ "metadata": {+ "feedback_events": [+ {"response_id": "resp-2", "feedback_type": "thumb_down"}+ ],+ "response_outcomes": {+ "resp-2": {"outcome_label": "negative_feedback"}+ },+ },+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 2",+ "response_id": "resp-2",+ "timestamp": "2026-05-03T10:00:00",+ }+ ),+ ]+ )+ + "\n",+ encoding="utf-8",+ )++ channel_stats = compute_feedback_stats(temp_dir / "bot", channel="cli__default")+ assert channel_stats["summary"]["sessions_scanned"] == 1+ assert channel_stats["summary"]["thumb_up_total"] == 1+ assert channel_stats["channels"] == {"cli__default": channel_stats["channels"]["cli__default"]}++ session_stats = compute_feedback_stats(temp_dir / "bot", session_key="bot_api__demo__session-2")+ assert session_stats["summary"]["sessions_scanned"] == 1+ assert session_stats["summary"]["thumb_down_total"] == 1+ assert list(session_stats["channels"].keys()) == ["bot_api__demo"]++ time_filtered_stats = compute_feedback_stats(+ temp_dir / "bot",+ updated_since="2026-05-02T00:00:00",+ updated_until="2026-05-04T00:00:00",+ )+ assert time_filtered_stats["summary"]["sessions_scanned"] == 1+ assert time_filtered_stats["summary"]["responses_total"] == 1+ assert list(time_filtered_stats["channels"].keys()) == ["bot_api__demo"]+++def test_compute_feedback_stats_includes_session_breakdown_when_requested(temp_dir):+ sessions_dir = temp_dir / "bot" / "sessions"+ sessions_dir.mkdir(parents=True)++ (sessions_dir / "cli__default__session-1.jsonl").write_text(+ "\n".join(+ [+ json.dumps(+ {+ "_type": "metadata",+ "session_key": "cli__default__session-1",+ "updated_at": "2026-05-01T10:00:00",+ "metadata": {+ "feedback_events": [+ {"response_id": "resp-1", "feedback_type": "thumb_up"}+ ],+ "response_outcomes": {+ "resp-1": {"outcome_label": "positive_feedback"},+ "resp-2": {"outcome_label": "resolved"},+ },+ },+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 1",+ "response_id": "resp-1",+ "timestamp": "2026-05-01T10:00:00",+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 2",+ "response_id": "resp-2",+ "timestamp": "2026-05-01T10:01:00",+ }+ ),+ ]+ )+ + "\n",+ encoding="utf-8",+ )++ stats = compute_feedback_stats(temp_dir / "bot", include_sessions=True)++ assert len(stats["sessions"]) == 1+ assert stats["sessions"][0]["session_key"] == "cli__default__session-1"+ assert stats["sessions"][0]["feedback_total"] == 1+ assert stats["sessions"][0]["one_turn_resolution_rate"] == 1.0+++def test_select_feedback_stats_sorts_and_limits_channels_and_sessions():+ stats = {+ "summary": {"sessions_scanned": 3},+ "channels": {+ "cli__default": {"responses_total": 2, "negative_feedback_total": 0},+ "bot_api__demo": {"responses_total": 5, "negative_feedback_total": 2},+ "email__alerts": {"responses_total": 3, "negative_feedback_total": 1},+ },+ "sessions": [+ {"session_key": "s1", "updated_at": "2026-05-01T10:00:00"},+ {"session_key": "s2", "updated_at": "2026-05-03T10:00:00"},+ {"session_key": "s3", "updated_at": "2026-05-02T10:00:00"},+ ],+ }++ selected = select_feedback_stats(+ stats,+ sort_by="negative_feedback_total",+ top_n=2,+ session_limit=2,+ )++ assert list(selected["channels"].keys()) == ["bot_api__demo", "email__alerts"]+ assert [session["session_key"] for session in selected["sessions"]] == ["s2", "s3"]+++def test_select_feedback_stats_rejects_unknown_sort_field():+ with pytest.raises(ValueError, match="sort_by must be one of"):+ select_feedback_stats({"summary": {}, "channels": {}, "sessions": []}, sort_by="bad_metric")+++def test_validate_feedback_stats_sort_by_accepts_known_field():+ assert validate_feedback_stats_sort_by("responses_total") == "responses_total"+ assert "negative_feedback_rate" in FEEDBACK_STATS_SORT_FIELDS+++def test_format_feedback_stats_table_returns_summary_and_channel_rows():+ stats = {+ "summary": {+ "sessions_scanned": 1,+ "responses_total": 2,+ "responses_with_feedback": 1,+ "feedback_total": 1,+ "thumb_up_total": 1,+ "thumb_down_total": 0,+ "outcomes_total": 2,+ "positive_feedback_total": 1,+ "negative_feedback_total": 0,+ "reasked_total": 0,+ "resolved_total": 1,+ "follow_up_without_feedback_total": 0,+ "feedback_coverage": 0.5,+ "thumbs_up_rate": 1.0,+ "thumbs_down_rate": 0.0,+ "positive_feedback_rate": 0.5,+ "negative_feedback_rate": 0.0,+ "reask_rate": 0.0,+ "one_turn_resolution_rate": 1.0,+ },+ "channels": {+ "cli__default": {+ "responses_total": 2,+ "responses_with_feedback": 1,+ "feedback_total": 1,+ "thumb_up_total": 1,+ "thumb_down_total": 0,+ "outcomes_total": 2,+ "positive_feedback_total": 1,+ "negative_feedback_total": 0,+ "reasked_total": 0,+ "resolved_total": 1,+ "follow_up_without_feedback_total": 0,+ "feedback_coverage": 0.5,+ "thumbs_up_rate": 1.0,+ "thumbs_down_rate": 0.0,+ "positive_feedback_rate": 0.5,+ "negative_feedback_rate": 0.0,+ "reask_rate": 0.0,+ "one_turn_resolution_rate": 1.0,+ }+ },+ "sessions": [+ {+ "session_key": "cli__default__session-1",+ "channel": "cli__default",+ "updated_at": "2026-05-01T10:00:00",+ "responses_total": 2,+ "feedback_total": 1,+ "negative_feedback_total": 0,+ "reasked_total": 0,+ "one_turn_resolution_rate": 1.0,+ }+ ],+ }++ table_data = format_feedback_stats_table(stats)++ assert ("Sessions Scanned", "1") in table_data["summary_rows"]+ assert ("One-turn Resolution Rate", "100.0%") in table_data["summary_rows"]+ assert table_data["channel_rows"] == [+ ("cli__default", "2", "1", "50.0%", "100.0%", "0.0%", "100.0%")+ ]+ assert table_data["session_rows"] == [+ (+ "cli__default__session-1",+ "cli__default",+ "2026-05-01T10:00:00",+ "2",+ "1",+ "0",+ "0",+ "100.0%",+ )+ ]+++def test_format_feedback_stats_markdown_returns_summary_and_tables():+ stats = {+ "summary": {+ "sessions_scanned": 1,+ "responses_total": 2,+ "responses_with_feedback": 1,+ "feedback_total": 1,+ "thumb_up_total": 1,+ "thumb_down_total": 0,+ "outcomes_total": 2,+ "positive_feedback_total": 1,+ "negative_feedback_total": 0,+ "reasked_total": 0,+ "resolved_total": 1,+ "follow_up_without_feedback_total": 0,+ "feedback_coverage": 0.5,+ "thumbs_up_rate": 1.0,+ "thumbs_down_rate": 0.0,+ "positive_feedback_rate": 0.5,+ "negative_feedback_rate": 0.0,+ "reask_rate": 0.0,+ "one_turn_resolution_rate": 1.0,+ },+ "channels": {+ "cli__default": {+ "responses_total": 2,+ "responses_with_feedback": 1,+ "feedback_total": 1,+ "thumb_up_total": 1,+ "thumb_down_total": 0,+ "outcomes_total": 2,+ "positive_feedback_total": 1,+ "negative_feedback_total": 0,+ "reasked_total": 0,+ "resolved_total": 1,+ "follow_up_without_feedback_total": 0,+ "feedback_coverage": 0.5,+ "thumbs_up_rate": 1.0,+ "thumbs_down_rate": 0.0,+ "positive_feedback_rate": 0.5,+ "negative_feedback_rate": 0.0,+ "reask_rate": 0.0,+ "one_turn_resolution_rate": 1.0,+ }+ },+ "sessions": [+ {+ "session_key": "cli__default__session-1",+ "channel": "cli__default",+ "updated_at": "2026-05-01T10:00:00",+ "responses_total": 2,+ "feedback_total": 1,+ "negative_feedback_total": 0,+ "reasked_total": 0,+ "one_turn_resolution_rate": 1.0,+ }+ ],+ }++ markdown_data = format_feedback_stats_markdown(stats)++ assert "**Sessions Scanned:** 1" in markdown_data["summary_markdown"]+ assert "**One-turn Resolution Rate:** 100.0%" in markdown_data["summary_markdown"]+ assert (+ "| Channel | Responses | Feedback | Coverage | Thumbs Up | Thumbs Down | Resolution |"+ in markdown_data["channels_markdown"]+ )+ assert (+ "| cli__default | 2 | 1 | 50.0% | 100.0% | 0.0% | 100.0% |"+ in markdown_data["channels_markdown"]+ )+ assert (+ "| Session | Channel | Updated At | Responses | Feedback | Negative | Reasked | Resolution |"+ in markdown_data["sessions_markdown"]+ )+ assert (+ "| cli__default__session-1 | cli__default | 2026-05-01T10:00:00 | 2 | 1 | 0 | 0 | 100.0% |"+ in markdown_data["sessions_markdown"]+ )+++def test_build_feedback_stats_display_applies_filters_and_sorting(temp_dir):+ sessions_dir = temp_dir / "bot" / "sessions"+ sessions_dir.mkdir(parents=True)++ (sessions_dir / "cli__default__session-1.jsonl").write_text(+ "\n".join(+ [+ json.dumps(+ {+ "_type": "metadata",+ "session_key": "cli__default__session-1",+ "updated_at": "2026-05-01T10:00:00",+ "metadata": {+ "feedback_events": [+ {"response_id": "resp-1", "feedback_type": "thumb_up"}+ ],+ "response_outcomes": {+ "resp-1": {"outcome_label": "positive_feedback"},+ "resp-2": {"outcome_label": "resolved"},+ },+ },+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 1",+ "response_id": "resp-1",+ "timestamp": "2026-05-01T10:00:00",+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 2",+ "response_id": "resp-2",+ "timestamp": "2026-05-01T10:01:00",+ }+ ),+ ]+ )+ + "\n",+ encoding="utf-8",+ )+ (sessions_dir / "bot_api__demo__session-2.jsonl").write_text(+ "\n".join(+ [+ json.dumps(+ {+ "_type": "metadata",+ "session_key": "bot_api__demo__session-2",+ "updated_at": "2026-05-03T10:00:00",+ "metadata": {+ "feedback_events": [+ {"response_id": "resp-2", "feedback_type": "thumb_down"},+ {"response_id": "resp-2", "feedback_type": "thumb_down"},+ ],+ "response_outcomes": {+ "resp-2": {"outcome_label": "negative_feedback"},+ },+ },+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 3",+ "response_id": "resp-2",+ "timestamp": "2026-05-03T10:00:00",+ }+ ),+ ]+ )+ + "\n",+ encoding="utf-8",+ )++ display = build_feedback_stats_display(+ temp_dir / "bot",+ updated_since="2026-05-02T00:00:00",+ sort_by="feedback_total",+ top_n=1,+ include_sessions=True,+ session_limit=1,+ )++ assert "**Sessions Scanned:** 1" in display["summary_markdown"]+ assert "bot_api__demo" in display["channels_markdown"]+ assert "cli__default" not in display["channels_markdown"]+ assert "bot_api__demo__session-2" in display["sessions_markdown"]+ assert "cli__default__session-1" not in display["sessions_markdown"]+++def test_compute_feedback_stats_preserves_all_assistant_responses_total_but_uses_tracked_denominators(temp_dir):+ sessions_dir = temp_dir / "bot" / "sessions"+ sessions_dir.mkdir(parents=True)++ (sessions_dir / "cli__default__session-1.jsonl").write_text(+ "\n".join(+ [+ json.dumps(+ {+ "_type": "metadata",+ "session_key": "cli__default__session-1",+ "updated_at": "2026-05-01T10:00:00",+ "metadata": {+ "feedback_events": [+ {"response_id": "resp-1", "feedback_type": "thumb_up"}+ ],+ "response_outcomes": {+ "resp-1": {"outcome_label": "positive_feedback"}+ },+ },+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 1",+ "response_id": "resp-1",+ "timestamp": "2026-05-01T10:00:00",+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 2",+ "response_id": "resp-2",+ "timestamp": "2026-05-01T10:01:00",+ }+ ),+ ]+ )+ + "\n",+ encoding="utf-8",+ )++ stats = compute_feedback_stats(temp_dir / "bot", include_sessions=True)++ assert stats["summary"]["responses_total"] == 2+ assert stats["summary"]["tracked_responses_total"] == 1+ assert stats["summary"]["outcomes_total"] == 1+ assert stats["summary"]["responses_with_feedback"] == 1+ assert stats["summary"]["feedback_coverage"] == 1.0+ assert stats["summary"]["positive_feedback_rate"] == 1.0+ assert stats["summary"]["one_turn_resolution_rate"] == 1.0+ assert stats["channels"]["cli__default"]["responses_total"] == 2+ assert stats["channels"]["cli__default"]["tracked_responses_total"] == 1+ assert stats["channels"]["cli__default"]["positive_feedback_rate"] == 1.0+ assert stats["sessions"][0]["responses_total"] == 2+ assert stats["sessions"][0]["tracked_responses_total"] == 1+ assert stats["sessions"][0]["positive_feedback_rate"] == 1.0+++def test_compute_feedback_stats_counts_rating_feedback_via_outcomes(temp_dir):+ sessions_dir = temp_dir / "bot" / "sessions"+ sessions_dir.mkdir(parents=True)++ (sessions_dir / "cli__default__session-1.jsonl").write_text(+ "\n".join(+ [+ json.dumps(+ {+ "_type": "metadata",+ "session_key": "cli__default__session-1",+ "updated_at": "2026-05-01T10:00:00",+ "metadata": {+ "feedback_events": [+ {+ "response_id": "resp-1",+ "feedback_type": "rating",+ "feedback_score": 1,+ }+ ],+ "response_outcomes": {+ "resp-1": {"outcome_label": "positive_feedback"},+ "resp-2": {"outcome_label": "resolved"},+ },+ },+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 1",+ "response_id": "resp-1",+ "timestamp": "2026-05-01T10:00:00",+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 2",+ "response_id": "resp-2",+ "timestamp": "2026-05-01T10:01:00",+ }+ ),+ ]+ )+ + "\n",+ encoding="utf-8",+ )++ stats = compute_feedback_stats(temp_dir / "bot", include_sessions=True)++ assert stats["summary"]["responses_total"] == 2+ assert stats["summary"]["tracked_responses_total"] == 2+ assert stats["summary"]["responses_with_feedback"] == 1+ assert stats["summary"]["feedback_total"] == 1+ assert stats["summary"]["thumb_up_total"] == 0+ assert stats["summary"]["thumb_down_total"] == 0+ assert stats["summary"]["feedback_coverage"] == 0.5+ assert stats["summary"]["positive_feedback_total"] == 1+ assert stats["summary"]["positive_feedback_rate"] == 0.5+ assert stats["summary"]["one_turn_resolution_rate"] == 1.0+ assert stats["channels"]["cli__default"]["feedback_coverage"] == 0.5+ assert stats["sessions"][0]["positive_feedback_rate"] == 0.5+++def test_compute_feedback_stats_uses_tracked_responses_as_rate_denominator(temp_dir):+ sessions_dir = temp_dir / "bot" / "sessions"+ sessions_dir.mkdir(parents=True)++ (sessions_dir / "cli__default__session-1.jsonl").write_text(+ "\n".join(+ [+ json.dumps(+ {+ "_type": "metadata",+ "session_key": "cli__default__session-1",+ "updated_at": "2026-05-01T10:00:00",+ "metadata": {+ "feedback_events": [+ {"response_id": "resp-1", "feedback_type": "thumb_up"}+ ],+ "response_outcomes": {+ "resp-1": {"outcome_label": "positive_feedback"},+ "resp-2": {"outcome_label": "resolved"},+ },+ },+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 1",+ "response_id": "resp-1",+ "timestamp": "2026-05-01T10:00:00",+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "legacy answer 2",+ "response_id": "resp-legacy-2",+ "timestamp": "2026-05-01T10:01:00",+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "legacy answer 3",+ "response_id": "resp-legacy-3",+ "timestamp": "2026-05-01T10:02:00",+ }+ ),+ ]+ )+ + "\n",+ encoding="utf-8",+ )++ stats = compute_feedback_stats(temp_dir / "bot", include_sessions=True)++ assert stats["summary"]["responses_total"] == 3+ assert stats["summary"]["tracked_responses_total"] == 2+ assert stats["summary"]["responses_with_feedback"] == 1+ assert stats["summary"]["feedback_coverage"] == 0.5+ assert stats["summary"]["positive_feedback_rate"] == 0.5+ assert stats["summary"]["one_turn_resolution_rate"] == 1.0+ assert stats["channels"]["cli__default"]["responses_total"] == 3+ assert stats["channels"]["cli__default"]["tracked_responses_total"] == 2+ assert stats["sessions"][0]["responses_total"] == 3+ assert stats["sessions"][0]["tracked_responses_total"] == 2diff --git a/bot/tests/test_langfuse_outcome_metadata.py b/bot/tests/test_langfuse_outcome_metadata.pyindex 287bf3768..5d2ad690c 100644--- a/bot/tests/test_langfuse_outcome_metadata.py+++ b/bot/tests/test_langfuse_outcome_metadata.py@@ -103,3 +103,30 @@ def test_langfuse_client_records_outcome_without_generation_metadata_attribute()assert client._client.scores[0]["observation_id"] == "obs-456"assert client._client.scores[0]["metadata"]["reask_within_10m"] is Trueassert client._client.flush_calls == 1+++def test_langfuse_client_merges_generation_metadata_into_tracked_generation():+ client = LangfuseClient.__new__(LangfuseClient)+ client.enabled = True+ client._client = _FakeClient()+ client._observations_by_response_id = {}+ client._metadata_by_response_id = {}+ client._trace_context_by_response_id = {}++ generation = _FakeGeneration(metadata={"response_id": "resp-789"})+ client.register_generation(+ "resp-789",+ generation,+ metadata={"response_id": "resp-789", "has_tools": False},+ )++ merged = client.update_generation_metadata(+ "resp-789",+ {"channel": "cli__default", "tool_count": 0},+ )++ assert merged["response_id"] == "resp-789"+ assert merged["has_tools"] is False+ assert merged["channel"] == "cli__default"+ assert merged["tool_count"] == 0+ assert generation.metadata == mergeddiff --git a/bot/tests/test_openapi_auth.py b/bot/tests/test_openapi_auth.pyindex 4273b3592..ccd668070 100644--- a/bot/tests/test_openapi_auth.py+++ b/bot/tests/test_openapi_auth.py@@ -286,6 +286,86 @@ class TestOpenAPIAuth:assert response.status_code == 404assert response.json()["detail"] == "Response not found"+ def test_rating_feedback_requires_feedback_score(self, message_bus, temp_workspace):+ channel = OpenAPIChannel(+ OpenAPIChannelConfig(),+ message_bus,+ workspace_path=temp_workspace,+ )+ client = _make_client(channel)++ response = client.post(+ "/bot/v1/feedback",+ json={+ "session_id": "session-1",+ "response_id": "resp-123",+ "feedback_type": "rating",+ },+ )++ assert response.status_code == 422+ assert "feedback_score is required when feedback_type is rating" in response.text++ def test_rating_feedback_persists_score_and_emits_outcome(self, message_bus, temp_workspace):+ channel = OpenAPIChannel(+ OpenAPIChannelConfig(),+ message_bus,+ workspace_path=temp_workspace,+ )+ session_key = SessionKey(type="cli", channel_id="default", chat_id="session-1")+ session = channel._session_manager.get_or_create(session_key)+ session.add_message(+ "assistant",+ "hello",+ sender_id="user-1",+ response_id="resp-123",+ timestamp="2026-04-30T00:00:00",+ )+ asyncio.run(channel._session_manager.save(session))++ client = _make_client(channel)+ response = client.post(+ "/bot/v1/feedback",+ json={+ "session_id": "session-1",+ "response_id": "resp-123",+ "feedback_type": "rating",+ "feedback_score": -1,+ "feedback_text": "bad answer",+ },+ )++ assert response.status_code == 200+ body = response.json()+ assert body["accepted"] is True+ assert body["feedback_type"] == "rating"+ assert message_bus.outbound_size == 2++ first_outbound = asyncio.run(message_bus.consume_outbound())+ second_outbound = asyncio.run(message_bus.consume_outbound())+ assert first_outbound.event_type == OutboundEventType.RESPONSE_OUTCOME_EVALUATED+ assert (+ first_outbound.metadata["response_outcome_evaluated"]["outcome_label"]+ == "negative_feedback"+ )+ assert (+ first_outbound.metadata["response_outcome_evaluated"]["evidence"]["feedback_score"]+ == -1.0+ )+ assert second_outbound.event_type == OutboundEventType.FEEDBACK_SUBMITTED+ assert second_outbound.metadata["feedback_submitted"]["feedback_type"] == "rating"+ assert second_outbound.metadata["feedback_submitted"]["feedback_score"] == -1++ session_path = temp_workspace / "sessions" / "cli__default__session-1.jsonl"+ lines = session_path.read_text(encoding="utf-8").splitlines()+ metadata = json.loads(lines[0])+ assert metadata["metadata"]["feedback_events"][0]["feedback_type"] == "rating"+ assert metadata["metadata"]["feedback_events"][0]["feedback_score"] == -1+ assert (+ metadata["metadata"]["response_outcomes"]["resp-123"]["outcome_label"]+ == "negative_feedback"+ )+def test_feedback_reloads_session_after_stale_cached_miss(self, message_bus, temp_workspace):channel = OpenAPIChannel(OpenAPIChannelConfig(),diff --git a/bot/tests/test_outcome_evaluator.py b/bot/tests/test_outcome_evaluator.pyindex 2cff446d0..a6332514d 100644--- a/bot/tests/test_outcome_evaluator.py+++ b/bot/tests/test_outcome_evaluator.py@@ -84,6 +84,96 @@ def test_evaluate_response_outcome_marks_negative_feedback():assert evaluation.follow_up_without_feedback is False+def test_evaluate_response_outcome_maps_positive_rating_to_positive_feedback():+ evaluation = evaluate_response_outcome(+ [+ {+ "role": "assistant",+ "content": "hello",+ "response_id": "resp-1",+ "timestamp": "2026-04-30T00:00:00",+ },+ {+ "role": "user",+ "content": "thanks",+ "timestamp": "2026-04-30T00:01:00",+ },+ ],+ "resp-1",+ feedback_events=[+ {+ "response_id": "resp-1",+ "feedback_type": "rating",+ "feedback_score": 1,+ }+ ],+ now=datetime.fromisoformat("2026-04-30T00:02:00"),+ )++ assert evaluation is not None+ assert evaluation.outcome_label == "positive_feedback"+ assert evaluation.resolved_in_one_turn is True+ assert evaluation.reask_within_10m is False+ assert evaluation.clarification_turns == 0+ assert evaluation.follow_up_without_feedback is False+ assert evaluation.evidence["feedback_score"] == 1.0+++def test_evaluate_response_outcome_maps_negative_rating_to_negative_feedback():+ evaluation = evaluate_response_outcome(+ [+ {+ "role": "assistant",+ "content": "hello",+ "response_id": "resp-1",+ "timestamp": "2026-04-30T00:00:00",+ }+ ],+ "resp-1",+ feedback_events=[+ {+ "response_id": "resp-1",+ "feedback_type": "rating",+ "feedback_score": -1,+ }+ ],+ now=datetime.fromisoformat("2026-04-30T00:02:00"),+ )++ assert evaluation is not None+ assert evaluation.outcome_label == "negative_feedback"+ assert evaluation.resolved_in_one_turn is False+ assert evaluation.follow_up_without_feedback is False+ assert evaluation.evidence["feedback_score"] == -1.0+++def test_evaluate_response_outcome_keeps_heuristic_outcome_for_neutral_rating():+ evaluation = evaluate_response_outcome(+ [+ {+ "role": "assistant",+ "content": "hello",+ "response_id": "resp-1",+ "timestamp": "2026-04-30T00:00:00",+ }+ ],+ "resp-1",+ feedback_events=[+ {+ "response_id": "resp-1",+ "feedback_type": "rating",+ "feedback_score": 0,+ }+ ],+ now=datetime.fromisoformat("2026-04-30T00:02:00"),+ )++ assert evaluation is not None+ assert evaluation.outcome_label == "resolved"+ assert evaluation.resolved_in_one_turn is True+ assert evaluation.evidence["feedback_score"] == 0.0++def test_evaluate_response_outcome_marks_reasked_within_window():evaluation = evaluate_response_outcome([diff --git a/bot/vikingbot/agent/context.py b/bot/vikingbot/agent/context.pyindex fb1df0486..db81ad92f 100644--- a/bot/vikingbot/agent/context.py+++ b/bot/vikingbot/agent/context.py@@ -70,6 +70,11 @@ class ContextBuilder:ensure_workspace_templates(self.workspace)self._templates_ensured = True+ def _get_workspace_id(self, session_key: SessionKey) -> str:+ if self.sandbox_manager:+ return self.sandbox_manager.to_workspace_id(session_key)+ return session_key.safe_name()+async def build_system_prompt(self,session_key: SessionKey,@@ -89,7 +94,7 @@ class ContextBuilder:"""# Ensure workspace templates exist only when first neededself._ensure_templates_once()- workspace_id = self.sandbox_manager.to_workspace_id(session_key)+ workspace_id = self._get_workspace_id(session_key)parts = []@@ -189,7 +194,7 @@ Skills with available="false" need dependencies installed first - you can try in)parts.append(session_context)- workspace_id = self.sandbox_manager.to_workspace_id(session_key)+ workspace_id = self._get_workspace_id(session_key)# Viking agent memory (only if ov tools are enabled)if ov_tools_enable:diff --git a/bot/vikingbot/agent/loop.py b/bot/vikingbot/agent/loop.pyindex 644f1ecb4..d771eb349 100644--- a/bot/vikingbot/agent/loop.py+++ b/bot/vikingbot/agent/loop.py@@ -8,8 +8,9 @@ import reimport timeimport uuidfrom contextlib import AsyncExitStack+from datetime import datetimefrom pathlib import Path-from typing import TYPE_CHECKING+from typing import TYPE_CHECKING, Anyfrom loguru import logger@@ -688,6 +689,17 @@ class AgentLoop:preview = final_content[:300] + "..." if len(final_content) > 300 else final_contentlogger.info(f"Response to {msg.session_key}: {preview}")+ response_completed = self._build_response_completed_payload(+ msg=msg,+ response_id=response_id,+ final_content=final_content,+ final_reasoning_content=final_reasoning_content,+ token_usage=token_usage,+ time_cost_seconds=time.time() - start_time,+ iteration=iteration,+ tools_used=tools_used,+ )+is_heartbeat = bool(msg.metadata.get(HEARTBEAT_METADATA_KEY))if not (is_heartbeat and is_heartbeat_noop_response(final_content)):session.add_message("user", msg.content, sender_id=msg.sender_id)@@ -700,13 +712,11 @@ class AgentLoop:sender_id=msg.sender_id,reasoning_content=final_reasoning_content,)+ session.metadata.setdefault("response_facts", {})[response_id] = response_completedawait self.sessions.save(session)-- time_cost = round(time.time() - start_time, 2)- if tools_used is not None:- tools_used_names = [tool["tool_name"] for tool in tools_used]- else:- tools_used_names = []+ LangfuseClient.get_instance().update_generation_metadata(+ response_id, response_completed+ )response_metadata = dict(msg.metadata or {})if relevant_memories is not None:response_metadata["relevant_memories"] = relevant_memories@@ -716,18 +726,7 @@ class AgentLoop:content="",event_type=OutboundEventType.RESPONSE_COMPLETED,response_id=response_id,- metadata={- "response_completed": {- "response_id": response_id,- "session_id": session_key.safe_name(),- "sender_id": msg.sender_id,- "content": final_content,- "token_usage": token_usage,- "time_cost": time_cost,- "iteration": iteration,- "tools_used_names": tools_used_names,- }- },+ metadata={"response_completed": response_completed},))return OutboundMessage(@@ -736,9 +735,9 @@ class AgentLoop:metadata=response_metadata,response_id=response_id,token_usage=token_usage,- time_cost=time_cost,- iteration=iteration,- tools_used_names=tools_used_names,+ time_cost=response_completed["time_cost_ms"] / 1000,+ iteration=response_completed["iteration_count"],+ tools_used_names=response_completed["tools_used_names"],)finally:long_running_notified = True@@ -748,6 +747,44 @@ class AgentLoop:except asyncio.CancelledError:pass+ @staticmethod+ def _build_response_completed_payload(+ msg: InboundMessage,+ response_id: str,+ final_content: str,+ final_reasoning_content: str | None,+ token_usage: dict[str, Any],+ time_cost_seconds: float,+ iteration: int,+ tools_used: list[dict[str, Any]] | None,+ ) -> dict[str, Any]:+ """Build a stable response fact shared by analytics sinks."""+ prompt_tokens = int(token_usage.get("prompt_tokens", 0) or 0)+ completion_tokens = int(token_usage.get("completion_tokens", 0) or 0)+ total_tokens = int(token_usage.get("total_tokens", prompt_tokens + completion_tokens) or 0)+ tools_used_names = [+ str(tool_name)+ for tool in (tools_used or [])+ if (tool_name := tool.get("tool_name")) is not None+ ]+ return {+ "response_id": response_id,+ "session_id": msg.session_key.safe_name(),+ "user_id": msg.sender_id,+ "channel": msg.session_key.channel_key(),+ "session_type": msg.session_key.type,+ "time_cost_ms": round(time_cost_seconds * 1000),+ "prompt_tokens": prompt_tokens,+ "completion_tokens": completion_tokens,+ "total_tokens": total_tokens,+ "iteration_count": iteration,+ "tool_count": len(tools_used_names),+ "tools_used_names": tools_used_names,+ "response_length": len(final_content),+ "created_at": datetime.now().isoformat(),+ "has_reasoning": bool(final_reasoning_content),+ }+async def _evaluate_previous_response_outcome(self, session: Session, msg: InboundMessage) -> None:diff --git a/bot/vikingbot/agent/memory.py b/bot/vikingbot/agent/memory.pyindex ef695a732..6c1074581 100644--- a/bot/vikingbot/agent/memory.py+++ b/bot/vikingbot/agent/memory.py@@ -3,7 +3,7 @@import asyncioimport timefrom pathlib import Path-from typing import Any, Optional+from typing import Anyfrom loguru import logger@@ -131,14 +131,6 @@ class MemoryStore:long_term = self.read_long_term()return f"## Long-term Memory\n{long_term}" if long_term else ""- async def _create_client(self, workspace_id: str) -> Optional[VikingClient]:- """Helper method to create a VikingClient with proper error handling."""- try:- return await VikingClient.create(agent_id=workspace_id)- except Exception as e:- logger.error(f"Failed to create VikingClient for workspace {workspace_id}: {e}")- return None-async def get_viking_memory_context(self,current_message: str,@@ -156,9 +148,7 @@ class MemoryStore:logger.info(f"user_ids={search_user_ids}")logger.info(f"admin_user_id={admin_user_id}")- client = await self._create_client(workspace_id)- if not client:- return ""+ client = await VikingClient.create(agent_id=workspace_id)result = await client.search_memory(query=current_message,user_ids=search_user_ids,@@ -204,9 +194,7 @@ class MemoryStore:"""用当前任务 query 检索 experience 记忆,注入到 system prompt。"""client = Nonetry:- client = await self._create_client(workspace_id)- if not client:- return ""+ client = await VikingClient.create(agent_id=workspace_id)experiences = await client.search_experiences(query, limit=5)logger.info(f"[READ_EXPERIENCE_MEMORY]: found {len(experiences)} experiences, query={query[:50]}"@@ -233,13 +221,11 @@ class MemoryStore:async def get_viking_user_profile(self, workspace_id: str, user_id: str) -> str:client = Nonetry:- client = await self._create_client(workspace_id)- if not client:- return ""+ client = await VikingClient.create(agent_id=workspace_id)result = await client.read_user_profile(user_id)return result or ""except Exception as e:- logger.warning(f"[READ_USER_PROFILE]: user_id={user_id}, error. {e}")+ logger.error(f"[READ_USER_PROFILE]: user_id={user_id}, error. {e}")return ""finally:if client:@@ -263,11 +249,10 @@ class MemoryStore:client = Nonetry:- client = await self._create_client(workspace_id)- if not client:- return ""+ client = await VikingClient.create(agent_id=workspace_id)async def fetch_profile(user_id: str) -> tuple[str, str]:+ """Fetch a single user profile."""try:start_time = time.time()profile = await client.read_user_profile(user_id)@@ -281,9 +266,11 @@ class MemoryStore:logger.error(f"[READ_USER_PROFILE]: user_id={user_id}, error. {e}")return (user_id, "")+ # Fetch all profiles concurrentlytasks = [fetch_profile(user_id) for user_id in user_ids]results = await asyncio.gather(*tasks, return_exceptions=True)+ # Build the result stringparts = []for result in results:if isinstance(result, Exception):diff --git a/bot/vikingbot/channels/openapi.py b/bot/vikingbot/channels/openapi.pyindex 3b0af6a51..b6923db15 100644--- a/bot/vikingbot/channels/openapi.py+++ b/bot/vikingbot/channels/openapi.py@@ -150,8 +150,11 @@ class OpenAPIChannel(BaseChannel):logger.warning("No global config provided, cannot load BotChannels")return- # Get all channel configs- channels_config = self._global_config.channels_config+ # Some tests and lightweight callers only provide gateway settings.+ channels_config = getattr(self._global_config, "channels_config", None)+ if channels_config is None:+ return+all_channel_configs = channels_config.get_all_channels()for ch_config in all_channel_configs:diff --git a/bot/vikingbot/channels/openapi_models.py b/bot/vikingbot/channels/openapi_models.pyindex 9fdbc17dd..102081e79 100644--- a/bot/vikingbot/channels/openapi_models.py+++ b/bot/vikingbot/channels/openapi_models.py@@ -4,7 +4,7 @@ from datetime import datetimefrom enum import Enumfrom typing import Any, Dict, List, Optional-from pydantic import BaseModel, Field+from pydantic import BaseModel, Field, model_validatorclass MessageRole(str, Enum):@@ -101,6 +101,13 @@ class FeedbackRequest(BaseModel):description="Bot channel ID for multi-channel routing (optional)",)+ @model_validator(mode="after")+ def validate_rating_feedback(self) -> "FeedbackRequest":+ """Require a numeric score when the client submits rating feedback."""+ if self.feedback_type == FeedbackType.RATING and self.feedback_score is None:+ raise ValueError("feedback_score is required when feedback_type is rating")+ return self+class FeedbackResponse(BaseModel):"""Response from feedback endpoint."""diff --git a/bot/vikingbot/cli/commands.py b/bot/vikingbot/cli/commands.pyindex f773253ca..c2cbab44f 100644--- a/bot/vikingbot/cli/commands.py+++ b/bot/vikingbot/cli/commands.py@@ -32,6 +32,12 @@ from vikingbot.cron.service import CronServicefrom vikingbot.cron.types import CronJobfrom vikingbot.heartbeat.service import HeartbeatServicefrom vikingbot.integrations.langfuse import LangfuseClient+from vikingbot.observability.feedback_stats import (+ compute_feedback_stats,+ format_feedback_stats_table,+ select_feedback_stats,+ validate_feedback_stats_sort_by,+)# Create sandbox managerfrom vikingbot.sandbox.manager import SandboxManager@@ -1056,6 +1062,118 @@ def status():)[email protected]("feedback-stats")+def feedback_stats(+ config_path: str = typer.Option(+ None, "--config", "-c", help="Path to ov.conf, default ~/.openviking/ov.conf"+ ),+ channel: str = typer.Option(None, "--channel", help="Only include one channel key"),+ session_key: str = typer.Option(None, "--session", help="Only include one session key"),+ updated_since: str = typer.Option(+ None, "--updated-since", help="Only include sessions updated at or after this ISO timestamp"+ ),+ updated_until: str = typer.Option(+ None,+ "--updated-until",+ help="Only include sessions updated at or before this ISO timestamp",+ ),+ sort_by: str = typer.Option(+ "responses_total", "--sort-by", help="Sort channel rows by a metric field"+ ),+ top_n: int = typer.Option(None, "--top-n", min=1, help="Limit the number of channel rows"),+ include_sessions: bool = typer.Option(+ False, "--sessions", help="Include per-session breakdown in JSON and table output"+ ),+ session_limit: int = typer.Option(+ None,+ "--session-limit",+ min=1,+ help="Limit the number of session rows when --sessions is used",+ ),+ output: str = typer.Option("json", "--output", help="Output format: json or table"),+):+ """Aggregate minimal feedback observability metrics from persisted sessions."""+ try:+ validate_feedback_stats_sort_by(sort_by)+ except ValueError as exc:+ raise typer.BadParameter(str(exc), param_hint="--sort-by") from exc++ path = Path(config_path).expanduser() if config_path is not None else None+ config = ensure_config(path)+ _init_bot_data(config)+ stats = compute_feedback_stats(+ config.bot_data_path,+ channel=channel,+ session_key=session_key,+ updated_since=updated_since,+ updated_until=updated_until,+ include_sessions=include_sessions,+ )+ stats = select_feedback_stats(+ stats,+ sort_by=sort_by,+ top_n=top_n,+ session_limit=session_limit if include_sessions else None,+ )++ if output == "json":+ console.print_json(json.dumps(stats, ensure_ascii=False))+ return+ if output == "table":+ _print_feedback_stats_table(stats)+ return++ raise typer.BadParameter("output must be one of: json, table")+++def _print_feedback_stats_table(stats: dict) -> None:+ table_data = format_feedback_stats_table(stats)++ summary_table = Table(title="Feedback Summary")+ summary_table.add_column("Metric", style="cyan")+ summary_table.add_column("Value", justify="right")++ for key, value in table_data["summary_rows"]:+ summary_table.add_row(key, value)++ console.print(summary_table)++ if not table_data["channel_rows"]:+ return++ channels_table = Table(title="Feedback By Channel")+ channels_table.add_column("Channel", style="magenta")+ channels_table.add_column("Responses", justify="right")+ channels_table.add_column("Feedback", justify="right")+ channels_table.add_column("Coverage", justify="right")+ channels_table.add_column("Thumbs Up", justify="right")+ channels_table.add_column("Thumbs Down", justify="right")+ channels_table.add_column("Resolution", justify="right")++ for row in table_data["channel_rows"]:+ channels_table.add_row(*row)++ console.print(channels_table)++ if not table_data["session_rows"]:+ return++ sessions_table = Table(title="Feedback By Session")+ sessions_table.add_column("Session", style="green")+ sessions_table.add_column("Channel", style="magenta")+ sessions_table.add_column("Updated At")+ sessions_table.add_column("Responses", justify="right")+ sessions_table.add_column("Feedback", justify="right")+ sessions_table.add_column("Negative", justify="right")+ sessions_table.add_column("Reasked", justify="right")+ sessions_table.add_column("Resolution", justify="right")++ for row in table_data["session_rows"]:+ sessions_table.add_row(*row)++ console.print(sessions_table)++# ============================================================================# Test Commands# ============================================================================diff --git a/bot/vikingbot/console/web_console.py b/bot/vikingbot/console/web_console.pyindex f4afc0044..01f777214 100644--- a/bot/vikingbot/console/web_console.py+++ b/bot/vikingbot/console/web_console.py@@ -7,6 +7,10 @@ import gradio as grfrom vikingbot.config.loader import get_config_path, load_config, save_configfrom vikingbot.config.schema import Config+from vikingbot.observability.feedback_stats import (+ FEEDBACK_STATS_SORT_FIELDS,+ build_feedback_stats_display,+)def resolve_schema_ref(@@ -42,6 +46,97 @@ def create_dashboard_tab():""")+def create_feedback_tab():+ with gr.Tab("Feedback"):+ gr.Markdown("## Feedback Observability")++ with gr.Row():+ channel_input = gr.Textbox(label="Channel", placeholder="cli__default")+ session_input = gr.Textbox(label="Session", placeholder="cli__default__session-1")++ with gr.Row():+ updated_since_input = gr.Textbox(+ label="Updated Since", placeholder="2026-05-01T00:00:00"+ )+ updated_until_input = gr.Textbox(+ label="Updated Until", placeholder="2026-05-31T23:59:59"+ )++ with gr.Row():+ sort_by_input = gr.Dropdown(+ choices=list(FEEDBACK_STATS_SORT_FIELDS),+ value="responses_total",+ label="Sort Channels By",+ )+ top_n_input = gr.Number(value=5, precision=0, label="Top Channels")+ session_limit_input = gr.Number(value=10, precision=0, label="Recent Sessions")++ with gr.Row():+ refresh_btn = gr.Button("Refresh Feedback Stats", variant="primary")+ status_msg = gr.Markdown("")++ summary_md = gr.Markdown("Click refresh to load feedback stats.")+ channels_md = gr.Markdown("")+ sessions_md = gr.Markdown("")++ def refresh_feedback_stats(+ channel,+ session_key,+ updated_since,+ updated_until,+ sort_by,+ top_n,+ session_limit,+ ):+ try:+ config = load_config()+ stats_markdown = build_feedback_stats_display(+ config.bot_data_path,+ channel=channel or None,+ session_key=session_key or None,+ updated_since=updated_since or None,+ updated_until=updated_until or None,+ sort_by=sort_by,+ top_n=_normalize_optional_int(top_n),+ include_sessions=True,+ session_limit=_normalize_optional_int(session_limit),+ )+ return (+ stats_markdown["summary_markdown"],+ stats_markdown["channels_markdown"],+ stats_markdown["sessions_markdown"],+ "✓ Feedback stats loaded.",+ )+ except Exception as exc:+ return (+ "Failed to load feedback stats.",+ "",+ "",+ f"✗ Error: {exc}",+ )++ refresh_btn.click(+ fn=refresh_feedback_stats,+ inputs=[+ channel_input,+ session_input,+ updated_since_input,+ updated_until_input,+ sort_by_input,+ top_n_input,+ session_limit_input,+ ],+ outputs=[summary_md, channels_md, sessions_md, status_msg],+ )+++def _normalize_optional_int(value: Any) -> int | None:+ if value in (None, ""):+ return None+ normalized = int(value)+ return normalized if normalized > 0 else None++def create_field_group(field_name: str,field_info: Dict[str, Any],@@ -407,6 +502,7 @@ def create_workspace_tab():with gr.Blocks(title="Vikingbot Console") as demo:with gr.Tabs():create_dashboard_tab()+ create_feedback_tab()with gr.Tab("Config"):create_config_tabs()create_sessions_tab()diff --git a/bot/vikingbot/integrations/langfuse.py b/bot/vikingbot/integrations/langfuse.pyindex 3361a831a..344689167 100644--- a/bot/vikingbot/integrations/langfuse.py+++ b/bot/vikingbot/integrations/langfuse.py@@ -104,12 +104,21 @@ class LangfuseClient:self._metadata_by_response_id.pop(oldest_response_id, None)self._trace_context_by_response_id.pop(oldest_response_id, None)- def update_generation_metadata(self, response_id: str, metadata: dict[str, Any]) -> None:- """Merge provider-side generation metadata for later outcome updates."""+ def update_generation_metadata(+ self, response_id: str, metadata: dict[str, Any]+ ) -> dict[str, Any]:+ """Merge generation metadata and best-effort sync it to the tracked observation."""if not response_id or not metadata:- return+ return self._metadata_by_response_id.get(response_id, {})current = self._metadata_by_response_id.setdefault(response_id, {})current.update(metadata)+ generation = self._observations_by_response_id.get(response_id)+ if generation is not None and hasattr(generation, "update"):+ try:+ generation.update(metadata=current)+ except Exception as e:+ logger.debug(f"Langfuse update generation metadata error: {e}")+ return current.copy()def update_response_outcome(self,diff --git a/bot/vikingbot/observability/feedback_stats.py b/bot/vikingbot/observability/feedback_stats.pynew file mode 100644index 000000000..ecd9e648e--- /dev/null+++ b/bot/vikingbot/observability/feedback_stats.py@@ -0,0 +1,577 @@+"""Offline feedback observability aggregation over persisted sessions."""++from __future__ import annotations++import json+from datetime import datetime+from pathlib import Path+from typing import Any+++FEEDBACK_STATS_SORT_FIELDS = (+ "responses_total",+ "tracked_responses_total",+ "responses_with_feedback",+ "feedback_total",+ "thumb_up_total",+ "thumb_down_total",+ "outcomes_total",+ "positive_feedback_total",+ "negative_feedback_total",+ "reasked_total",+ "resolved_total",+ "follow_up_without_feedback_total",+ "feedback_coverage",+ "thumbs_up_rate",+ "thumbs_down_rate",+ "positive_feedback_rate",+ "negative_feedback_rate",+ "reask_rate",+ "one_turn_resolution_rate",+)++_SUMMARY_METRICS = (+ ("Sessions Scanned", "sessions_scanned", "count"),+ ("Responses", "responses_total", "count"),+ ("Tracked Responses", "tracked_responses_total", "count"),+ ("Responses With Feedback", "responses_with_feedback", "count"),+ ("Feedback Events", "feedback_total", "count"),+ ("Thumbs Up", "thumb_up_total", "count"),+ ("Thumbs Down", "thumb_down_total", "count"),+ ("Outcome Records", "outcomes_total", "count"),+ ("Positive Feedback Outcomes", "positive_feedback_total", "count"),+ ("Negative Feedback Outcomes", "negative_feedback_total", "count"),+ ("Reasked Outcomes", "reasked_total", "count"),+ ("Resolved Outcomes", "resolved_total", "count"),+ ("Follow-up Without Feedback", "follow_up_without_feedback_total", "count"),+ ("Feedback Coverage", "feedback_coverage", "percent"),+ ("Thumbs Up Rate", "thumbs_up_rate", "percent"),+ ("Thumbs Down Rate", "thumbs_down_rate", "percent"),+ ("Positive Feedback Rate", "positive_feedback_rate", "percent"),+ ("Negative Feedback Rate", "negative_feedback_rate", "percent"),+ ("Reask Rate", "reask_rate", "percent"),+ ("One-turn Resolution Rate", "one_turn_resolution_rate", "percent"),+)+++def compute_feedback_stats(+ bot_data_path: Path,+ *,+ channel: str | None = None,+ session_key: str | None = None,+ updated_since: str | None = None,+ updated_until: str | None = None,+ include_sessions: bool = False,+) -> dict[str, Any]:+ """Aggregate minimal feedback observability metrics from session JSONL files."""+ sessions_dir = bot_data_path / "sessions"+ filters = {+ "channel": channel,+ "session_key": session_key,+ "updated_since": _parse_datetime(updated_since),+ "updated_until": _parse_datetime(updated_until),+ }+ totals: dict[str, Any] = {+ "sessions_scanned": 0,+ "responses_total": 0,+ "tracked_responses_total": 0,+ "responses_with_feedback": 0,+ "feedback_total": 0,+ "thumb_up_total": 0,+ "thumb_down_total": 0,+ "outcomes_total": 0,+ "positive_feedback_total": 0,+ "negative_feedback_total": 0,+ "reasked_total": 0,+ "resolved_total": 0,+ "follow_up_without_feedback_total": 0,+ "channels": {},+ "sessions": [] if include_sessions else None,+ }++ if not sessions_dir.exists():+ return _finalize_feedback_stats(totals)++ for session_path in sorted(sessions_dir.glob("*.jsonl")):+ session_metrics = _read_session_metrics(session_path, filters)+ if session_metrics is None:+ continue+ totals["sessions_scanned"] += 1+ _merge_feedback_stats(totals, session_metrics)++ return _finalize_feedback_stats(totals)+++def format_feedback_stats_table(stats: dict[str, Any]) -> dict[str, Any]:+ """Return table-friendly rows for CLI rendering."""+ summary = stats.get("summary", {})+ summary_rows = [+ (label, _format_metric_value(summary.get(key, 0), value_type))+ for label, key, value_type in _SUMMARY_METRICS+ ]++ channel_rows = []+ for channel_key, channel_stats in stats.get("channels", {}).items():+ channel_rows.append(+ (+ channel_key,+ _format_metric_value(channel_stats.get("responses_total", 0), "count"),+ _format_metric_value(channel_stats.get("feedback_total", 0), "count"),+ _format_metric_value(channel_stats.get("feedback_coverage", 0.0), "percent"),+ _format_metric_value(channel_stats.get("thumbs_up_rate", 0.0), "percent"),+ _format_metric_value(channel_stats.get("thumbs_down_rate", 0.0), "percent"),+ _format_metric_value(channel_stats.get("one_turn_resolution_rate", 0.0), "percent"),+ )+ )++ session_rows = []+ for session_stats in stats.get("sessions", []):+ session_rows.append(+ (+ session_stats.get("session_key", ""),+ session_stats.get("channel", "unknown"),+ str(session_stats.get("updated_at") or ""),+ _format_metric_value(session_stats.get("responses_total", 0), "count"),+ _format_metric_value(session_stats.get("feedback_total", 0), "count"),+ _format_metric_value(session_stats.get("negative_feedback_total", 0), "count"),+ _format_metric_value(session_stats.get("reasked_total", 0), "count"),+ _format_metric_value(session_stats.get("one_turn_resolution_rate", 0.0), "percent"),+ )+ )++ return {+ "summary_rows": summary_rows,+ "channel_rows": channel_rows,+ "session_rows": session_rows,+ }+++def format_feedback_stats_markdown(stats: dict[str, Any]) -> dict[str, str]:+ """Return markdown-friendly sections for console rendering."""+ table_data = format_feedback_stats_table(stats)++ summary_lines = [f"- **{label}:** {value}" for label, value in table_data["summary_rows"]]+ summary_markdown = "\n".join(summary_lines) if summary_lines else "No feedback stats available."++ channels_markdown = _format_markdown_table(+ ["Channel", "Responses", "Feedback", "Coverage", "Thumbs Up", "Thumbs Down", "Resolution"],+ table_data["channel_rows"],+ empty_message="No channel stats available.",+ )+ sessions_markdown = _format_markdown_table(+ [+ "Session",+ "Channel",+ "Updated At",+ "Responses",+ "Feedback",+ "Negative",+ "Reasked",+ "Resolution",+ ],+ table_data["session_rows"],+ empty_message="No session stats available.",+ )++ return {+ "summary_markdown": summary_markdown,+ "channels_markdown": channels_markdown,+ "sessions_markdown": sessions_markdown,+ }+++def build_feedback_stats_display(+ bot_data_path: Path,+ *,+ channel: str | None = None,+ session_key: str | None = None,+ updated_since: str | None = None,+ updated_until: str | None = None,+ sort_by: str = "responses_total",+ top_n: int | None = None,+ include_sessions: bool = True,+ session_limit: int | None = None,+) -> dict[str, str]:+ """Compute, select, and format feedback stats for display surfaces."""+ stats = compute_feedback_stats(+ bot_data_path,+ channel=channel,+ session_key=session_key,+ updated_since=updated_since,+ updated_until=updated_until,+ include_sessions=include_sessions,+ )+ stats = select_feedback_stats(+ stats,+ sort_by=sort_by,+ top_n=top_n,+ session_limit=session_limit if include_sessions else None,+ )+ return format_feedback_stats_markdown(stats)+++def select_feedback_stats(+ stats: dict[str, Any],+ *,+ sort_by: str = "responses_total",+ top_n: int | None = None,+ session_limit: int | None = None,+) -> dict[str, Any]:+ """Sort and optionally trim channel and session breakdowns for display."""+ validate_feedback_stats_sort_by(sort_by)+ channels = stats.get("channels", {})+ sessions = stats.get("sessions", [])++ sorted_channels = sorted(+ channels.items(),+ key=lambda item: _sort_value(item[1], sort_by),+ reverse=True,+ )+ if top_n is not None:+ sorted_channels = sorted_channels[:top_n]++ sorted_sessions = sorted(+ sessions,+ key=lambda item: _sort_value(item, "updated_at"),+ reverse=True,+ )+ if session_limit is not None:+ sorted_sessions = sorted_sessions[:session_limit]++ return {+ "summary": stats.get("summary", {}),+ "channels": dict(sorted_channels),+ "sessions": sorted_sessions,+ }+++def validate_feedback_stats_sort_by(sort_by: str) -> str:+ """Validate the requested sort field for channel-level stats."""+ if sort_by not in FEEDBACK_STATS_SORT_FIELDS:+ allowed = ", ".join(FEEDBACK_STATS_SORT_FIELDS)+ raise ValueError(f"sort_by must be one of: {allowed}")+ return sort_by+++def _read_session_metrics(session_path: Path, filters: dict[str, Any]) -> dict[str, Any] | None:+ try:+ with open(session_path, encoding="utf-8") as f:+ first_line = f.readline().strip()+ remaining_lines = f.readlines()+ except OSError:+ return None++ if not first_line:+ return None++ try:+ first_record = json.loads(first_line)+ except json.JSONDecodeError:+ return None++ if first_record.get("_type") != "metadata":+ return None++ session_key = first_record.get("session_key", "")+ metadata = first_record.get("metadata", {})+ channel = _channel_from_session_key(session_key)+ updated_at = _parse_datetime(first_record.get("updated_at"))++ if filters["channel"] and channel != filters["channel"]:+ return None+ if filters["session_key"] and session_key != filters["session_key"]:+ return None+ if filters["updated_since"] and (updated_at is None or updated_at < filters["updated_since"]):+ return None+ if filters["updated_until"] and (updated_at is None or updated_at > filters["updated_until"]):+ return None++ feedback_events = metadata.get("feedback_events", [])+ response_outcomes = metadata.get("response_outcomes", {})+ response_ids = _collect_response_ids(remaining_lines)++ feedback_by_response = {+ event.get("response_id")+ for event in feedback_events+ if isinstance(event, dict) and event.get("response_id")+ }+ tracked_response_ids = feedback_by_response | {+ response_id+ for response_id, payload in response_outcomes.items()+ if isinstance(response_id, str) and response_id and isinstance(payload, dict)+ }++ metrics: dict[str, Any] = {+ "session_key": session_key,+ "responses_total": len(response_ids),+ "tracked_responses_total": len(tracked_response_ids),+ "responses_with_feedback": len(feedback_by_response),+ "feedback_total": 0,+ "thumb_up_total": 0,+ "thumb_down_total": 0,+ "outcomes_total": 0,+ "positive_feedback_total": 0,+ "negative_feedback_total": 0,+ "reasked_total": 0,+ "resolved_total": 0,+ "follow_up_without_feedback_total": 0,+ "channel": channel,+ "updated_at": first_record.get("updated_at"),+ }++ for event in feedback_events:+ if not isinstance(event, dict):+ continue+ metrics["feedback_total"] += 1+ feedback_type = event.get("feedback_type")+ if feedback_type == "thumb_up":+ metrics["thumb_up_total"] += 1+ elif feedback_type == "thumb_down":+ metrics["thumb_down_total"] += 1++ for payload in response_outcomes.values():+ if not isinstance(payload, dict):+ continue+ metrics["outcomes_total"] += 1+ outcome_label = payload.get("outcome_label")+ if outcome_label == "positive_feedback":+ metrics["positive_feedback_total"] += 1+ elif outcome_label == "negative_feedback":+ metrics["negative_feedback_total"] += 1+ elif outcome_label == "reasked":+ metrics["reasked_total"] += 1+ elif outcome_label == "resolved":+ metrics["resolved_total"] += 1+ elif outcome_label == "follow_up_without_feedback":+ metrics["follow_up_without_feedback_total"] += 1++ return _finalize_session_metrics(metrics)+++def _finalize_session_metrics(metrics: dict[str, Any]) -> dict[str, Any]:+ tracked_responses_total = metrics["tracked_responses_total"]+ metrics["feedback_coverage"] = _safe_ratio(+ metrics["responses_with_feedback"], tracked_responses_total+ )+ metrics["thumbs_up_rate"] = _safe_ratio(metrics["thumb_up_total"], metrics["feedback_total"])+ metrics["thumbs_down_rate"] = _safe_ratio(+ metrics["thumb_down_total"], metrics["feedback_total"]+ )+ metrics["positive_feedback_rate"] = _safe_ratio(+ metrics["positive_feedback_total"], tracked_responses_total+ )+ metrics["negative_feedback_rate"] = _safe_ratio(+ metrics["negative_feedback_total"], tracked_responses_total+ )+ metrics["reask_rate"] = _safe_ratio(metrics["reasked_total"], tracked_responses_total)+ metrics["one_turn_resolution_rate"] = _safe_ratio(+ metrics["resolved_total"] + metrics["positive_feedback_total"],+ tracked_responses_total,+ )++ return metrics+++def _collect_response_ids(lines: list[str]) -> set[str]:+ response_ids: set[str] = set()+ for line in lines:+ line = line.strip()+ if not line:+ continue+ try:+ record = json.loads(line)+ except json.JSONDecodeError:+ continue+ if record.get("role") != "assistant":+ continue+ response_id = record.get("response_id")+ if isinstance(response_id, str) and response_id:+ response_ids.add(response_id)+ return response_ids+++def _merge_feedback_stats(totals: dict[str, Any], session_metrics: dict[str, Any]) -> None:+ channel = session_metrics["channel"]+ channel_totals = totals["channels"].setdefault(+ channel,+ {+ "responses_total": 0,+ "tracked_responses_total": 0,+ "responses_with_feedback": 0,+ "feedback_total": 0,+ "thumb_up_total": 0,+ "thumb_down_total": 0,+ "outcomes_total": 0,+ "positive_feedback_total": 0,+ "negative_feedback_total": 0,+ "reasked_total": 0,+ "resolved_total": 0,+ "follow_up_without_feedback_total": 0,+ },+ )++ for key, value in session_metrics.items():+ if key in {+ "channel",+ "session_key",+ "updated_at",+ "feedback_coverage",+ "thumbs_up_rate",+ "thumbs_down_rate",+ "positive_feedback_rate",+ "negative_feedback_rate",+ "reask_rate",+ "one_turn_resolution_rate",+ }:+ continue+ totals[key] += value+ channel_totals[key] += value++ if totals["sessions"] is None:+ return++ totals["sessions"].append(+ {+ "session_key": session_metrics["session_key"],+ "channel": channel,+ "updated_at": session_metrics["updated_at"],+ "responses_total": session_metrics["responses_total"],+ "tracked_responses_total": session_metrics["tracked_responses_total"],+ "responses_with_feedback": session_metrics["responses_with_feedback"],+ "feedback_total": session_metrics["feedback_total"],+ "thumb_up_total": session_metrics["thumb_up_total"],+ "thumb_down_total": session_metrics["thumb_down_total"],+ "outcomes_total": session_metrics["outcomes_total"],+ "positive_feedback_total": session_metrics["positive_feedback_total"],+ "negative_feedback_total": session_metrics["negative_feedback_total"],+ "reasked_total": session_metrics["reasked_total"],+ "resolved_total": session_metrics["resolved_total"],+ "follow_up_without_feedback_total": session_metrics["follow_up_without_feedback_total"],+ "feedback_coverage": session_metrics["feedback_coverage"],+ "thumbs_up_rate": session_metrics["thumbs_up_rate"],+ "thumbs_down_rate": session_metrics["thumbs_down_rate"],+ "positive_feedback_rate": session_metrics["positive_feedback_rate"],+ "negative_feedback_rate": session_metrics["negative_feedback_rate"],+ "reask_rate": session_metrics["reask_rate"],+ "one_turn_resolution_rate": session_metrics["one_turn_resolution_rate"],+ }+ )+++def _finalize_feedback_stats(totals: dict[str, Any]) -> dict[str, Any]:+ summary = {+ "sessions_scanned": totals["sessions_scanned"],+ "responses_total": totals["responses_total"],+ "tracked_responses_total": totals["tracked_responses_total"],+ "responses_with_feedback": totals["responses_with_feedback"],+ "feedback_total": totals["feedback_total"],+ "thumb_up_total": totals["thumb_up_total"],+ "thumb_down_total": totals["thumb_down_total"],+ "outcomes_total": totals["outcomes_total"],+ "positive_feedback_total": totals["positive_feedback_total"],+ "negative_feedback_total": totals["negative_feedback_total"],+ "reasked_total": totals["reasked_total"],+ "resolved_total": totals["resolved_total"],+ "follow_up_without_feedback_total": totals["follow_up_without_feedback_total"],+ "feedback_coverage": _safe_ratio(+ totals["responses_with_feedback"], totals["tracked_responses_total"]+ ),+ "thumbs_up_rate": _safe_ratio(totals["thumb_up_total"], totals["feedback_total"]),+ "thumbs_down_rate": _safe_ratio(totals["thumb_down_total"], totals["feedback_total"]),+ "positive_feedback_rate": _safe_ratio(+ totals["positive_feedback_total"], totals["tracked_responses_total"]+ ),+ "negative_feedback_rate": _safe_ratio(+ totals["negative_feedback_total"], totals["tracked_responses_total"]+ ),+ "reask_rate": _safe_ratio(totals["reasked_total"], totals["tracked_responses_total"]),+ "one_turn_resolution_rate": _safe_ratio(+ totals["resolved_total"] + totals["positive_feedback_total"],+ totals["tracked_responses_total"],+ ),+ }++ channels = {+ channel: {+ **channel_totals,+ "feedback_coverage": _safe_ratio(+ channel_totals["responses_with_feedback"], channel_totals["tracked_responses_total"]+ ),+ "thumbs_up_rate": _safe_ratio(+ channel_totals["thumb_up_total"], channel_totals["feedback_total"]+ ),+ "thumbs_down_rate": _safe_ratio(+ channel_totals["thumb_down_total"], channel_totals["feedback_total"]+ ),+ "positive_feedback_rate": _safe_ratio(+ channel_totals["positive_feedback_total"], channel_totals["tracked_responses_total"]+ ),+ "negative_feedback_rate": _safe_ratio(+ channel_totals["negative_feedback_total"], channel_totals["tracked_responses_total"]+ ),+ "reask_rate": _safe_ratio(+ channel_totals["reasked_total"], channel_totals["tracked_responses_total"]+ ),+ "one_turn_resolution_rate": _safe_ratio(+ channel_totals["resolved_total"] + channel_totals["positive_feedback_total"],+ channel_totals["tracked_responses_total"],+ ),+ }+ for channel, channel_totals in sorted(totals["channels"].items())+ }++ return {+ "summary": summary,+ "channels": channels,+ "sessions": sorted(totals["sessions"] or [], key=lambda item: item["session_key"]),+ }+++def _channel_from_session_key(session_key: str) -> str:+ if not session_key or "__" not in session_key:+ return "unknown"+ parts = session_key.split("__")+ if len(parts) < 2:+ return "unknown"+ return f"{parts[0]}__{parts[1]}"+++def _safe_ratio(numerator: int, denominator: int) -> float:+ if denominator <= 0:+ return 0.0+ return round(numerator / denominator, 4)+++def _parse_datetime(value: str | None) -> datetime | None:+ if not value:+ return None+ normalized = value.replace("Z", "+00:00")+ return datetime.fromisoformat(normalized)+++def _sort_value(payload: dict[str, Any], key: str) -> Any:+ value = payload.get(key)+ if key == "updated_at":+ parsed = _parse_datetime(value)+ return parsed or datetime.min+ if isinstance(value, (int, float)):+ return value+ return value or 0+++def _format_metric_value(value: Any, value_type: str) -> str:+ if value_type == "percent":+ return f"{float(value) * 100:.1f}%"+ return str(value)+++def _format_markdown_table(+ headers: list[str], rows: list[tuple[Any, ...]], *, empty_message: str+) -> str:+ if not rows:+ return empty_message++ header_row = "| " + " | ".join(headers) + " |"+ separator_row = "| " + " | ".join("---" for _ in headers) + " |"+ data_rows = ["| " + " | ".join(str(cell) for cell in row) + " |" for row in rows]+ return "\n".join([header_row, separator_row, *data_rows])diff --git a/bot/vikingbot/observability/outcome.py b/bot/vikingbot/observability/outcome.pyindex 7d14927d4..0f59223e4 100644--- a/bot/vikingbot/observability/outcome.py+++ b/bot/vikingbot/observability/outcome.py@@ -61,6 +61,7 @@ def evaluate_response_outcome(]latest_feedback = relevant_feedback[-1] if relevant_feedback else Nonefeedback_type = latest_feedback.get("feedback_type") if latest_feedback else None+ feedback_score = _parse_feedback_score(latest_feedback) if latest_feedback else Nonereask_within_10m = Falsefirst_user_after_response = user_messages[0] if user_messages else None@@ -73,10 +74,14 @@ def evaluate_response_outcome(resolved_in_one_turn = not user_messagesfollow_up_without_feedback = bool(user_messages) and not relevant_feedback- if feedback_type == "thumb_down":+ if feedback_type == "thumb_down" or (+ feedback_type == "rating" and feedback_score is not None and feedback_score < 0+ ):outcome_label = "negative_feedback"resolved_in_one_turn = False- elif feedback_type == "thumb_up":+ elif feedback_type == "thumb_up" or (+ feedback_type == "rating" and feedback_score is not None and feedback_score > 0+ ):outcome_label = "positive_feedback"resolved_in_one_turn = Truereask_within_10m = False@@ -105,6 +110,7 @@ def evaluate_response_outcome(evaluated_at=evaluated_at,evidence={"feedback_type": feedback_type,+ "feedback_score": feedback_score,"user_follow_up_count": len(user_messages),"assistant_index": assistant_index,},@@ -142,3 +148,12 @@ def _parse_timestamp(message: dict[str, Any]) -> Optional[datetime]:return datetime.fromisoformat(timestamp)except ValueError:return None+++def _parse_feedback_score(feedback_event: dict[str, Any]) -> Optional[float]:+ score = feedback_event.get("feedback_score")+ if isinstance(score, bool) or score is None:+ return None+ if isinstance(score, (int, float)):+ return float(score)+ return Nonediff --git a/bot/vikingbot/providers/litellm_provider.py b/bot/vikingbot/providers/litellm_provider.pyindex e3bd704e2..3865af214 100644--- a/bot/vikingbot/providers/litellm_provider.py+++ b/bot/vikingbot/providers/litellm_provider.py@@ -284,7 +284,7 @@ class LiteLLMProvider(LLMProvider):response_id = get_current_response_id()if response_id:- self.langfuse.update_generation_metadata(+ update_kwargs["metadata"] = self.langfuse.update_generation_metadata(response_id,update_kwargs.get("metadata", {}),)diff --git a/bot/vikingbot/providers/openai_compatible_provider.py b/bot/vikingbot/providers/openai_compatible_provider.pyindex ab89b14b4..e7969d3ca 100644--- a/bot/vikingbot/providers/openai_compatible_provider.py+++ b/bot/vikingbot/providers/openai_compatible_provider.py@@ -211,7 +211,7 @@ class OpenAICompatibleProvider(LLMProvider):response_id = get_current_response_id()if response_id:- self.langfuse.update_generation_metadata(+ update_kwargs["metadata"] = self.langfuse.update_generation_metadata(response_id,update_kwargs.get("metadata", {}),)diff --git a/bot/vikingbot/utils/tracing.py b/bot/vikingbot/utils/tracing.pyindex b9e239172..4096ef526 100644--- a/bot/vikingbot/utils/tracing.py+++ b/bot/vikingbot/utils/tracing.py@@ -179,8 +179,9 @@ def trace(langfuse = LangfuseClient.get_instance()has_propagate = hasattr(langfuse, "propagate_attributes")+ is_enabled = getattr(langfuse, "enabled", False)# logger.info(f"[LANGFUSE] Client status: enabled={langfuse.enabled}, has_propagate_attributes={has_propagate}")- if langfuse.enabled and has_propagate:+ if is_enabled and has_propagate:# logger.info(f"[LANGFUSE] Starting trace with attributes: session_id={session_id}, user_id={user_id}")with langfuse.propagate_attributes(session_id=session_id, user_id=user_id):return await wrapped_func(*args, **kwargs)diff --git a/docs/en/api/09-metrics.md b/docs/en/api/09-metrics.mdindex 82e882020..84d67d3b7 100644--- a/docs/en/api/09-metrics.md+++ b/docs/en/api/09-metrics.md@@ -80,8 +80,11 @@ If your deployment protects `/metrics` at the gateway layer, configure the scrap- `/metrics` returns Prometheus text, not the standard OpenViking `{status, result, time}` JSON response format.- Analytics-oriented statistics such as memory health, staleness, or session extraction should remain under `/api/v1/stats/*`, not `/metrics`.- For human-readable component snapshots, prefer `/api/v1/observer/*`.+- `/metrics` also includes VikingBot feedback observability metrics derived from scrape-time aggregation of persisted session data; see the Metrics concept documentation for the feedback metric families and examples.## Related Documentation+- [Metrics](../concepts/12-metrics.md) - Metric families, labels, feedback metrics, and PromQL examples+- [VikingBot Feedback Observability Design](../../../bot/docs/vikingbot-feedback-observability-design.md) - feedback observability design background and rollout plan (Chinese)- [System and Monitoring](07-system.md) - Health checks, system status, and Observer APIs- [API Overview](01-overview.md) - Shared conventions for all API endpointsdiff --git a/docs/en/concepts/12-metrics.md b/docs/en/concepts/12-metrics.mdindex 3f87cc008..280c376be 100644--- a/docs/en/concepts/12-metrics.md+++ b/docs/en/concepts/12-metrics.md@@ -261,6 +261,77 @@ These help answer:| `openviking_session_contexts_used_total` | Counter | `account_id, action` | session contexts used total || `openviking_session_archive_total` | Counter | `account_id, status` | session archive count |+### Feedback++These metrics summarize persisted VikingBot feedback and outcome data at scrape time. They are exported as gauges because the collector recomputes the current aggregate snapshot from bot session files instead of incrementing counters online.++| Metric Family | Type | Common Labels | Meaning |+|---------------|------|---------------|---------|+| `openviking_feedback_sessions_scanned_total` | Gauge | `valid` | number of bot sessions scanned for the current snapshot |+| `openviking_feedback_responses_total` | Gauge | `valid` | total persisted assistant responses included in the snapshot, including legacy responses outside the new observability contract |+| `openviking_feedback_tracked_responses_total` | Gauge | `valid` | responses covered by the current feedback observability contract (`metadata.feedback_events` or `metadata.response_outcomes`) |+| `openviking_feedback_responses_with_feedback_total` | Gauge | `valid` | responses that have at least one explicit feedback event |+| `openviking_feedback_events_total` | Gauge | `valid` | explicit feedback event count |+| `openviking_feedback_thumb_up_total` | Gauge | `valid` | thumb-up event count |+| `openviking_feedback_thumb_down_total` | Gauge | `valid` | thumb-down event count |+| `openviking_feedback_positive_outcomes_total` | Gauge | `valid` | responses classified as positive feedback outcomes |+| `openviking_feedback_negative_outcomes_total` | Gauge | `valid` | responses classified as negative feedback outcomes |+| `openviking_feedback_reasked_outcomes_total` | Gauge | `valid` | responses classified as reasked outcomes |+| `openviking_feedback_resolved_outcomes_total` | Gauge | `valid` | responses classified as resolved outcomes |+| `openviking_feedback_follow_up_without_feedback_outcomes_total` | Gauge | `valid` | responses followed up without explicit feedback |+| `openviking_feedback_coverage` | Gauge | `valid` | fraction of tracked responses with explicit feedback |+| `openviking_feedback_thumbs_up_rate` | Gauge | `valid` | fraction of feedback events that are thumbs up |+| `openviking_feedback_thumbs_down_rate` | Gauge | `valid` | fraction of feedback events that are thumbs down |+| `openviking_feedback_positive_feedback_rate` | Gauge | `valid` | fraction of tracked responses with positive feedback outcomes |+| `openviking_feedback_negative_feedback_rate` | Gauge | `valid` | fraction of tracked responses with negative feedback outcomes |+| `openviking_feedback_reask_rate` | Gauge | `valid` | fraction of tracked responses that led to reasking |+| `openviking_feedback_one_turn_resolution_rate` | Gauge | `valid` | fraction of tracked responses resolved in one turn |+| `openviking_feedback_channel_*` | Gauge | `channel, valid` | per-channel variants of response volume, feedback volume, negative outcomes, reasks, coverage, thumb rates, and one-turn resolution |++For mixed historical data, use `openviking_feedback_tracked_responses_total` as the denominator reference for rate panels. `openviking_feedback_responses_total` is kept to show overall persisted assistant-response volume, including legacy responses that predate the current feedback metadata contract.++Typical usage:++- build Grafana panels for feedback coverage, thumbs-down rate, and one-turn resolution rate over time+- compare feedback quality across channels such as `cli__default` and `bot_api__demo`+- alert when `valid="0"` appears persistently, which indicates the collector is serving the last successful snapshot after a refresh failure++PromQL / Grafana examples:++- overall feedback coverage:++```promql+openviking_feedback_coverage{valid="1"}+```++- overall thumbs-down rate:++```promql+openviking_feedback_thumbs_down_rate{valid="1"}+```++- one-turn resolution rate:++```promql+openviking_feedback_one_turn_resolution_rate{valid="1"}+```++- per-channel coverage comparison:++```promql+openviking_feedback_channel_coverage{valid="1"}+```++- detect fallback snapshots:++```promql+max by (job) (openviking_feedback_events_total{valid="0"})+```++Because these are scrape-time snapshot gauges, they work well in Grafana time-series panels and side-by-side channel comparison panels.++For `/metrics` endpoint behavior and scrape usage, see [Metrics API](../api/09-metrics.md).+### Probes and Health State| Metric Family | Type | Common Labels | Meaning |@@ -427,4 +498,5 @@ Example:- [Multi-Tenant](./11-multi-tenant.md) - `account/user/agent` isolation model- [Data Encryption](./10-encryption.md) - storage-layer encryption and isolation- [Metrics API](../api/09-metrics.md) - `/metrics` endpoint usage+- [VikingBot Feedback Observability Design](../../../bot/docs/vikingbot-feedback-observability-design.md) - feedback observability design background and rollout plan (Chinese)- [Metrics Design](../../design/metric-design.md) - metrics system design detailsdiff --git a/docs/zh/api/09-metrics.md b/docs/zh/api/09-metrics.mdindex ff3a38582..35d3e6dad 100644--- a/docs/zh/api/09-metrics.md+++ b/docs/zh/api/09-metrics.md@@ -80,8 +80,11 @@ scrape_configs:- `/metrics` 返回的是 Prometheus 文本,不是标准 OpenViking API 的 `{status, result, time}` JSON 结构。- 业务分析类统计(如 memory health、staleness、session extraction)应继续使用 `/api/v1/stats/*`,而不是迁移到 `/metrics`。- 人工查看组件瞬时状态更适合使用 `/api/v1/observer/*`。+- `/metrics` 现在也包含 VikingBot feedback observability 指标,这些指标来自对持久化 session 数据的 scrape-time 聚合;具体指标族与示例可参见 Metrics 概念文档中的 feedback 章节。## 相关文档+- [指标与 Metrics](../concepts/12-metrics.md) - 指标族、标签、feedback 指标与 PromQL 示例+- [VikingBot 问答效果反馈观测方案设计](../../../bot/docs/vikingbot-feedback-observability-design.md) - feedback 指标与阶段性落地背景- [系统与监控](07-system.md) - 健康检查、系统状态与 Observer API- [API 概览](01-overview.md) - 所有 API 端点共享约定diff --git a/docs/zh/concepts/12-metrics.md b/docs/zh/concepts/12-metrics.mdindex fe76434d3..b002d4bae 100644--- a/docs/zh/concepts/12-metrics.md+++ b/docs/zh/concepts/12-metrics.md@@ -264,6 +264,81 @@ scrape_configs:| `openviking_session_contexts_used_total` | Counter | `account_id, action` | session contexts used 累计量 || `openviking_session_archive_total` | Counter | `account_id, status` | session archive 次数 |+### Feedback++这组 feedback 指标会在 scrape 时对持久化的 VikingBot session 文件进行聚合,汇总反馈事件与 outcome 数据。它们以 gauge 形式导出,因为 collector 每次都会重新计算当前聚合快照,而不是在线持续累加 counter。++| 指标族 | 类型 | 常见标签 | 含义 |+|--------|------|----------|------|+| `openviking_feedback_sessions_scanned_total` | Gauge | `valid` | 当前快照扫描到的 bot session 数量 |+| `openviking_feedback_responses_total` | Gauge | `valid` | 当前快照纳入统计的 assistant response 总数,包含尚未接入新观测契约的历史 response |+| `openviking_feedback_tracked_responses_total` | Gauge | `valid` | 已被当前 feedback 观测契约覆盖的 response 总数(来自 `metadata.feedback_events` 或 `metadata.response_outcomes`) |+| `openviking_feedback_responses_with_feedback_total` | Gauge | `valid` | 至少带有一个显式反馈事件的 response 数量 |+| `openviking_feedback_events_total` | Gauge | `valid` | 显式反馈事件总数 |+| `openviking_feedback_thumb_up_total` | Gauge | `valid` | thumb-up 事件数 |+| `openviking_feedback_thumb_down_total` | Gauge | `valid` | thumb-down 事件数 |+| `openviking_feedback_positive_outcomes_total` | Gauge | `valid` | 被归类为 positive outcome 的 response 数量 |+| `openviking_feedback_negative_outcomes_total` | Gauge | `valid` | 被归类为 negative outcome 的 response 数量 |+| `openviking_feedback_reasked_outcomes_total` | Gauge | `valid` | 被归类为 reask outcome 的 response 数量 |+| `openviking_feedback_resolved_outcomes_total` | Gauge | `valid` | 被归类为 resolved outcome 的 response 数量 |+| `openviking_feedback_follow_up_without_feedback_outcomes_total` | Gauge | `valid` | 有 follow-up 但没有显式反馈的 outcome 数量 |+| `openviking_feedback_coverage` | Gauge | `valid` | 已跟踪 response 中带显式反馈的占比 |+| `openviking_feedback_thumbs_up_rate` | Gauge | `valid` | feedback event 中 thumb-up 的占比 |+| `openviking_feedback_thumbs_down_rate` | Gauge | `valid` | feedback event 中 thumb-down 的占比 |+| `openviking_feedback_positive_feedback_rate` | Gauge | `valid` | 已跟踪 response 中 positive feedback outcome 的占比 |+| `openviking_feedback_negative_feedback_rate` | Gauge | `valid` | 已跟踪 response 中 negative feedback outcome 的占比 |+| `openviking_feedback_reask_rate` | Gauge | `valid` | 已跟踪 response 中导致 reask 的占比 |+| `openviking_feedback_one_turn_resolution_rate` | Gauge | `valid` | 已跟踪 response 中一轮解决的占比 |+| `openviking_feedback_channel_*` | Gauge | `channel, valid` | 按 channel 细分的 response 数量、feedback 数量、negative outcome、reask、coverage、thumb rate 与 one-turn resolution |++对于新旧历史数据混合的场景,rate 类图表应优先结合 `openviking_feedback_tracked_responses_total` 理解分母。`openviking_feedback_responses_total` 仍然保留,用于观察包含历史遗留 response 在内的整体 assistant 响应体量。++适用场景:++- 在 Grafana 中绘制 feedback coverage、thumbs-down rate、one-turn resolution rate 的时间趋势+- 对比不同 channel(如 `cli__default`、`bot_api__demo`)之间的反馈质量差异+- 当 `valid="0"` 持续出现时告警,表示 collector 在刷新失败后回退到了上一次成功快照++PromQL / Grafana 示例:++- 总体 feedback coverage:++```promql+openviking_feedback_coverage{valid="1"}+```++- 总体 thumbs-down rate:++```promql+openviking_feedback_thumbs_down_rate{valid="1"}+```++- 总体 one-turn resolution rate:++```promql+openviking_feedback_one_turn_resolution_rate{valid="1"}+```++- 按 channel 对比 coverage 与 resolution:++```promql+openviking_feedback_channel_coverage{valid="1"}+```++```promql+openviking_feedback_channel_one_turn_resolution_rate{valid="1"}+```++- 检查 stale / fallback snapshot:++```promql+max by (job) (openviking_feedback_events_total{valid="0"})+```++因为这些指标本质上是 scrape-time snapshot gauge,所以很适合直接做 Grafana 时间序列面板,以及按 channel 并排对比的可视化。++关于 `/metrics` 端点行为与抓取方式,可参见 [Metrics API](../api/09-metrics.md)。+### 探针与健康状态| 指标族 | 类型 | 常见标签 | 含义 |@@ -431,4 +506,5 @@ scrape_configs:- [多租户](./11-multi-tenant.md) - `account/user/agent` 隔离模型- [数据加密](./10-encryption.md) - 存储层加密与隔离- [Metrics API](../api/09-metrics.md) - `/metrics` 端点用法+- [VikingBot 问答效果反馈观测方案设计](../../../bot/docs/vikingbot-feedback-observability-design.md) - feedback 指标与阶段性落地背景- [指标体系设计](../../design/metric-design.md) - 指标体系设计细节diff --git a/docs/zh/guides/05-observability.md b/docs/zh/guides/05-observability.mdindex 2ce8969c4..e4af50275 100644--- a/docs/zh/guides/05-observability.md+++ b/docs/zh/guides/05-observability.md@@ -430,6 +430,8 @@ OpenViking 仓库里已经提供了可直接导入的 dashboard JSON:## 相关文档+- [使用 Prometheus 和 Grafana 查看 OpenViking 指标](11-grafana-prometheus.md) - 从 `/metrics` 到 Prometheus、Grafana dashboard 的完整操作流程+- [使用真实问答验证 Vikingbot 指标](12-vikingbot-metrics-validation.md) - 用 `/bot/v1/chat`、`/bot/v1/feedback` 和真实 follow-up 场景校验反馈与 outcome 指标- [部署](03-deployment.md) - 服务器设置- [认证](04-authentication.md) - API Key 设置- [操作级 Telemetry 参考](07-operation-telemetry.md) - 请求级结构化追踪diff --git a/docs/zh/guides/11-grafana-prometheus.md b/docs/zh/guides/11-grafana-prometheus.mdnew file mode 100644index 000000000..241e4f3da--- /dev/null+++ b/docs/zh/guides/11-grafana-prometheus.md@@ -0,0 +1,466 @@+# 使用 Prometheus 和 Grafana 查看 OpenViking 指标++这份文档给出一条从零开始的完整链路:++1. 启动 OpenViking 并确认 `/metrics` 可访问+2. 启动 Prometheus 抓取 OpenViking 指标+3. 启动 Grafana 并连接 Prometheus 数据源+4. 导入 OpenViking 自带 dashboard 或在 Explore 中直接查询++如果你已经能访问 `http://<host>:<port>/metrics`,可以直接从本文的“启动 Prometheus”开始。++## 架构关系++OpenViking 不直接提供 Grafana 页面。标准链路是:++```text+OpenViking -> /metrics -> Prometheus -> Grafana+```++其中:++- OpenViking 负责暴露 Prometheus exposition 文本+- Prometheus 负责定时抓取 `/metrics`+- Grafana 负责读取 Prometheus 并展示 dashboard++## 前置条件++开始前请确认:++- OpenViking Server 已安装并可正常启动+- Docker 已安装,可用于快速启动 Prometheus 和 Grafana+- 你知道 OpenViking 当前监听的 HTTP 地址,例如 `http://localhost:30300`++## 第 1 步:确认 OpenViking 已暴露 `/metrics`++OpenViking 需要先启用 metrics。最小配置参考:++```json+{+ "server": {+ "observability": {+ "metrics": {+ "enabled": true+ }+ }+ }+}+```++配置写入 `~/.openviking/ov.conf` 后,重启 OpenViking Server。++如果你还没有启动服务,可参考:++```bash+openviking-server doctor+openviking-server --port 30300+```++然后验证:++```bash+curl http://localhost:30300/metrics+```++如果返回包含 `openviking_` 前缀的文本,说明 metrics 已经启用。例如:++```text+# HELP openviking_http_requests_total Total number of HTTP requests+# TYPE openviking_http_requests_total counter+openviking_http_requests_total{method="GET",route="/api/v1/system/status",status="200"} 12+```++如果返回 `Prometheus metrics are disabled.`,说明配置未生效或服务未重启。++## 第 2 步:使用仓库自带 compose 文件部署++仓库里已经提供了一套可直接启动的观测示例,文件位于:++- `examples/grafana/docker-compose.yml`+- `examples/grafana/prometheus.yml`+- `examples/grafana/grafana/provisioning/datasources/prometheus.yml`+- `examples/grafana/grafana/provisioning/dashboards/openviking.yml`++另外,针对 Linux 上 OpenViking 继续监听 `127.0.0.1` / `localhost` 的场景,仓库还提供了一套 localhost 专用示例:++- `examples/grafana/docker-compose.localhost.yml`+- `examples/grafana/prometheus.localhost.yml`+- `examples/grafana/grafana/provisioning-localhost/datasources/prometheus.yml`+- `examples/grafana/grafana/provisioning-localhost/dashboards/openviking.yml`++两套方案的区别是:++- `docker-compose.yml`:通用方案,Prometheus 从容器网络访问宿主机,适合 OpenViking 监听 `0.0.0.0`+- `docker-compose.localhost.yml`:Linux localhost 方案,Prometheus 和 Grafana 直接使用宿主机网络,适合 OpenViking 继续监听 `127.0.0.1`++如果你当前不想把 OpenViking 暴露到 `0.0.0.0`,推荐优先使用 `docker-compose.localhost.yml`。++这套配置默认会做几件事:++- 启动 Prometheus,并把宿主机端口映射到 `30909`+- 启动 Grafana,并把宿主机端口映射到 `13000`+- 自动把 Grafana 数据源配置为 `http://127.0.0.1:30909`+- 自动加载仓库里的 OpenViking demo dashboard+- 自动加载 `OpenViking - Feedback Baseline`,方便直接查看 `openviking_feedback_*` 与 `openviking_feedback_channel_*` 的基线指标++### 方案 A:通用方案++直接执行:++```bash+docker compose -f examples/grafana/docker-compose.yml up -d+```++启动完成后可访问:++```text+Prometheus: http://localhost:30909+Grafana: http://localhost:13000+```++Grafana 默认账号密码在这个示例里固定为:++- 用户名:`admin`+- 密码:`admin`++### 方案 B:Linux localhost 方案++如果你的 OpenViking 继续监听在 `127.0.0.1:30300`,并且你不想为了 Prometheus 抓取而把 OpenViking 改成 `0.0.0.0`,请使用下面这套 compose:++```bash+docker compose -f examples/grafana/docker-compose.localhost.yml up -d+```++这套方案的特点是:++- Prometheus 使用宿主机网络,直接抓取 `127.0.0.1:30300/metrics`+- Grafana 也使用宿主机网络,并直接连接 `http://127.0.0.1:30909`+- 不需要把 OpenViking 改成 `0.0.0.0`+- 不会触发“非 localhost 监听必须配置 `root_api_key`”这条安全限制++访问地址仍然是:++```text+Prometheus: http://localhost:30909+Grafana: http://localhost:13000+```++如果宿主机上的 `30909` 或 `13000` 已经被占用:++- Prometheus 端口改 `examples/grafana/docker-compose.localhost.yml` 里的 `--web.listen-address=0.0.0.0:30909`+- Grafana 端口改 `examples/grafana/docker-compose.localhost.yml` 里的 `GF_SERVER_HTTP_PORT=13000`+- 同时把 `examples/grafana/grafana/provisioning-localhost/datasources/prometheus.yml` 中的 `http://127.0.0.1:30909` 改成新端口++如果你只想快速部署,做到这里就可以先跳到“如何判断链路已经完全打通”。++## 第 3 步:理解 Prometheus 抓取配置++compose 示例里使用的 `examples/grafana/prometheus.yml` 内容如下:++```yaml+global:+ scrape_interval: 15s++scrape_configs:+ - job_name: openviking+ metrics_path: /metrics+ static_configs:+ - targets: ["host.docker.internal:30300"]+```++说明:++- 如果 Prometheus 运行在 Docker 容器里,而 OpenViking 运行在宿主机,`targets` 推荐写成 `host.docker.internal:30300`+- 如果 Prometheus 也运行在宿主机,改成 `localhost:30300`+- 如果 `host.docker.internal` 在你的 Linux Docker 环境中不可用,就改成宿主机实际 IP,例如 `192.168.1.10:30300`++如果你的 OpenViking 不是监听在 `30300`,就把这个文件里的目标地址改成你的实际端口,然后重新执行:++```bash+docker compose -f examples/grafana/docker-compose.yml up -d+```++如果你使用的是 Linux localhost 方案,对应修改的是:++- `examples/grafana/prometheus.localhost.yml`++例如 OpenViking 实际监听 `127.0.0.1:1933`,就改成:++```yaml+targets: ["127.0.0.1:1933"]+```++然后重新执行:++```bash+docker compose -f examples/grafana/docker-compose.localhost.yml up -d+```++## 第 4 步:可选,手动部署时创建 Docker 网络++如果你使用的是上面的 compose 文件,这一步不需要手动执行,因为 Compose 会自动创建默认网络。++只有在你坚持使用 `docker run` 分开启动 Prometheus 和 Grafana 时,才需要先创建一个独立网络:++```bash+docker network create openviking-observability+```++如果提示网络已存在,可以忽略。++## 第 5 步:可选,手动启动 Prometheus++如果你已经用了 `docker compose -f examples/grafana/docker-compose.yml up -d`,这一节可以跳过。++很多机器上 `9090` 已经被别的服务占用。为了减少冲突,这里建议把宿主机端口映射到 `30909`:++```bash+docker run -d \+ --name prometheus \+ --network openviking-observability \+ -p 30909:9090 \+ -v "$PWD/prometheus.yml:/etc/prometheus/prometheus.yml:ro" \+ prom/prometheus+```++启动后,在浏览器打开:++```text+http://localhost:30909+```++进入 Prometheus UI 后,在查询框中输入:++```promql+openviking_http_requests_total+```++或者:++```promql+openviking_service_readiness+```++如果能查到时间序列,说明 Prometheus 已经成功抓到 OpenViking 指标。++### 如果 Prometheus 容器启动失败++常见原因:宿主机端口被占用,例如:++```text+Bind for 0.0.0.0:9090 failed: port is already allocated+```++处理方式:++- 改宿主机端口,例如继续使用 `30909:9090`+- 不要改容器内端口 `9090`+- 访问时用新的宿主机端口,例如 `http://localhost:30909`++## 第 6 步:可选,手动启动 Grafana++如果你已经用了 `docker compose -f examples/grafana/docker-compose.yml up -d`,这一节可以跳过。++同样地,很多机器上的 `3000` 也常被占用。建议把 Grafana 映射到宿主机的 `13000`:++```bash+docker run -d \+ --name grafana \+ --network openviking-observability \+ -p 13000:3000 \+ grafana/grafana+```++启动后打开:++```text+http://localhost:13000+```++Grafana 默认初始账号通常是:++- 用户名:`admin`+- 密码:`admin`++如果你的环境已修改默认凭据,以实际值为准。++## 第 7 步:可选,手动在 Grafana 中添加 Prometheus 数据源++如果你使用的是仓库自带 compose 文件,这一步通常也可以跳过,因为数据源会自动 provision。++在 Grafana 页面中操作:++1. 打开左侧 `Connections` 或 `Data sources`+2. 点击 `Add data source`+3. 选择 `Prometheus`+4. 在 `URL` 中填写:`http://prometheus:9090`+5. 点击 `Save & test`++这里填写 `http://prometheus:9090` 的原因是:++- Grafana 和 Prometheus 运行在同一个 Docker 网络 `openviking-observability` 中+- 两个容器可以直接通过容器名通信++如果 `Save & test` 失败,请先执行:++```bash+docker ps+```++确认 `prometheus` 和 `grafana` 两个容器都在运行。++## 第 8 步:先在 Grafana Explore 中直接查询++添加完数据源后,先不要急着导入 dashboard,建议先在 `Explore` 中验证基础查询。++推荐先试这些查询:++请求量:++```promql+rate(openviking_http_requests_total[5m])+```++按路由查看请求量与状态码:++```promql+sum by (route, status) (rate(openviking_http_requests_total[5m]))+```++P95 延迟:++```promql+histogram_quantile(0.95, sum by (le, route) (rate(openviking_http_request_duration_seconds_bucket[5m])))+```++队列积压:++```promql+openviking_queue_pending+```++模型调用量:++```promql+rate(openviking_model_calls_total[5m])+```++Token 用量:++```promql+rate(openviking_operation_tokens_total[5m])+```++如果你还不确定有哪些指标名,可以先查:++```promql+{__name__=~"openviking_.*"}+```++## 第 9 步:导入 OpenViking 自带 Dashboard++如果你使用的是仓库自带 compose 文件,这两个 dashboard 会在 Grafana 启动后自动加载到 `OpenViking` 文件夹下。++如果你想手动导入,继续按下面步骤操作即可。++仓库中已经提供了可直接导入的 Grafana dashboard:++- `examples/grafana/openviking_demo_dashboard.json`+- `examples/grafana/openviking_token_demo_dashboard.json`++导入步骤:++1. 进入 Grafana 左侧 `Dashboards`+2. 点击右上角 `New` 或 `Import`+3. 上传 `examples/grafana/openviking_demo_dashboard.json`+4. 在导入页面选择刚刚创建的 Prometheus 数据源+5. 点击 `Import`++说明:++- `openviking_demo_dashboard.json` 适合作为基础总览 dashboard+- `openviking_token_demo_dashboard.json` 依赖 `tim012432-calendarheatmap-panel` 插件,未安装前部分面板可能无法正常显示++## 第 10 步:如何判断链路已经完全打通++你可以按下面的顺序验证:++1. `curl http://localhost:30300/metrics` 能返回指标文本+2. 打开 `http://localhost:30909`,在 Prometheus 中能查到 `openviking_http_requests_total`+3. 打开 `http://localhost:13000`,能看到 `Prometheus` 数据源已经存在,或手动 `Save & test` 成功+4. Grafana Explore 中运行 `rate(openviking_http_requests_total[5m])` 能出图+5. 导入 demo dashboard 后面板开始显示数据++只要这五步都通过,说明整条链路已经打通。++## 常见问题++### 1. `/metrics` 能访问,但 Prometheus 查不到数据++优先检查:++- `prometheus.yml` 的 `targets` 是否写对+- Prometheus 是否真的重新加载了新的配置+- Docker 容器内是否能访问宿主机上的 `30300`++如果你使用的是仓库自带 compose 文件,优先检查:++```bash+docker compose -f examples/grafana/docker-compose.yml logs prometheus+```++如果怀疑容器访问宿主机有问题,可以把 `host.docker.internal` 改成宿主机实际 IP。++### 2. Prometheus 宿主机端口被占用++报错示例:++```text+Bind for 0.0.0.0:9090 failed: port is already allocated+```++处理方式:改成别的宿主机端口,例如:++```bash+ -p 30909:9090+```++### 3. Grafana 宿主机端口被占用++处理方式:改成别的宿主机端口,例如:++```bash+-p 13000:3000+```++### 4. Grafana 里没有任何 OpenViking 指标++优先检查:++- Grafana 数据源是否真的连到 Prometheus+- Prometheus 中是否已经有 `openviking_*` 指标+- 时间范围是否过短,导致近期没有样本++如果你使用的是 compose 自动导入方案,还可以先确认 dashboard 是否已经被加载:++- 左侧进入 `Dashboards`+- 查看 `OpenViking` 文件夹是否存在++### 5. Dashboard 导入成功但面板为空++这通常不是 dashboard 文件损坏,而是:++- Prometheus 里还没有对应指标样本+- 过滤条件和当前环境不匹配+- 选择了错误的数据源++建议先回到 Explore 手动执行 PromQL,确认基础查询确实有数据。++## 相关文档++- [可观测性与排障](05-observability.md)+- [使用真实问答验证 Vikingbot 指标](12-vikingbot-metrics-validation.md)+- [指标与 Metrics](../concepts/12-metrics.md)+- [Metrics API](../api/09-metrics.md)+- [服务端部署](03-deployment.md)+- [快速开始:服务端模式](../getting-started/03-quickstart-server.md)diff --git a/docs/zh/guides/12-vikingbot-metrics-validation.md b/docs/zh/guides/12-vikingbot-metrics-validation.mdnew file mode 100644index 000000000..3697d7a25--- /dev/null+++ b/docs/zh/guides/12-vikingbot-metrics-validation.md@@ -0,0 +1,1231 @@+# 使用真实问答验证 Vikingbot 指标++这份文档用于验证一条完整链路是否已经打通:++```text+用户提问 -> /bot/v1/chat -> Vikingbot 会话持久化 -> /bot/v1/feedback 或 follow-up -> /metrics -> Prometheus -> Grafana+```++它和 [使用 Prometheus 和 Grafana 查看 OpenViking 指标](11-grafana-prometheus.md) 的区别是:++- `11-grafana-prometheus.md` 解决的是“怎么把链路搭起来”+- 本文解决的是“链路搭起来以后,怎么用真实问答场景确认指标真的在变”++本文只使用当前仓库里已经确认存在、而且比较容易观察到的指标,尤其是:++- `openviking_service_readiness`+- `openviking_component_health`+- `openviking_queue_pending`+- `openviking_vikingdb_collection_vectors`+- `openviking_model_usage_available`+- `openviking_feedback_*`+- `openviking_feedback_channel_*`++## 前提条件++开始前请先确认:++- OpenViking Server 已启动,并且启用了 `server.observability.metrics.enabled=true`+- Vikingbot 已通过 `openviking-server --with-bot` 启动+- `curl http://127.0.0.1:30300/bot/v1/health` 返回 `200`+- `curl http://127.0.0.1:30300/metrics` 能返回 Prometheus exposition 文本+- Prometheus 已经能抓到 OpenViking+- Grafana 已经连上 Prometheus++如果你使用的是本文前面已经补好的 Linux localhost 方案,默认地址是:++- Prometheus: `http://127.0.0.1:30909`+- Grafana: `http://127.0.0.1:13000`+- OpenViking: `http://127.0.0.1:30300`++## 先理解这批反馈指标的口径++这一步很重要,否则很容易把现象看错。++当前仓库里,Vikingbot 反馈相关指标不是在线单调累加 counter,而是 `scrape-time snapshot gauge`:Prometheus 每次抓取 `/metrics` 时,`FeedbackCollector` 会去扫描 `bot/sessions/*.jsonl`,从持久化 session 的 `metadata.feedback_events` 和 `metadata.response_outcomes` 里重新聚合一份最新快照。++这意味着:++- 更适合看“当前总量/占比/按 channel 分布”+- 更适合直接看当前值或短时间内的阶梯变化+- 不建议把它当成普通 counter 去优先看 `rate()`+- 查询时建议优先过滤 `valid="1"`++可以先在 Prometheus 或 Grafana Explore 里执行:++```promql+{__name__=~"openviking_feedback.*"}+```++如果你看到的是 `valid="0"`,说明当前样本是 fallback/stale 快照,不建议把它当成主验证结果。++## 按真实问答场景执行的验收用例++如果你不想按“指标类别”理解文档,而是更希望像真实用户一样逐轮发问、观察 Vikingbot 回复、再去 Prometheus / Grafana 里核对指标变化,可以直接按下面这 7 个场景顺序执行。++建议使用一组全新的 `session_id`,这样更容易看出单次操作带来的阶梯变化。下面示例统一使用:++- `realcase-01-chat`+- `realcase-02-thumb-up`+- `realcase-03-thumb-down`+- `realcase-04-reask`+- `realcase-05-followup-no-feedback`+- `realcase-06-resolved`+- `realcase-07-channel-demo`++### 场景 1:真实用户先发一条正常问题,验证会话落盘与响应计数++用户提问:++```text+请用一句话介绍 OpenViking,控制在 20 个字以内。+```++预期 Vikingbot 回复特征:++- 返回 200+- 返回体包含 `session_id`+- 返回体包含 `response_id`+- `message` 非空++执行命令:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "realcase-01-chat",+ "user_id": "metrics-validation-user",+ "message": "请用一句话介绍 OpenViking,控制在 20 个字以内。"+ }'+```++重点观察:++- `openviking_feedback_responses_total{valid="1"}`+- `openviking_feedback_sessions_scanned_total{valid="1"}`++PromQL:++```promql+openviking_feedback_responses_total{valid="1"}+```++```promql+openviking_feedback_sessions_scanned_total{valid="1"}+```++预期变化:++- 会变:`responses_total` 通常增加 `1`;如果这是全新的 session 文件,`sessions_scanned_total` 也可能增加 `1`。+- 不该变:`events_total`、`thumb_up_total`、`thumb_down_total`、`responses_with_feedback_total` 以及各类 outcome total 在这一步通常不应直接增加,因为这里只发生了一次普通问答,没有显式 feedback,也没有 follow-up。+- 可能不明显:rate 类指标通常不会因为这一轮普通问答出现明显波动;这个场景优先看 total 是否跳变,不要把 rate 没变化当成异常。++### 场景 2:用户认为回答有帮助,点一个赞,验证正向反馈链路++用户提问:++```text+帮我总结一下 OpenViking 的主要作用。+```++预期 Vikingbot 回复特征:++- 会给出一段简短介绍+- 返回体里能拿到 `response_id`++随后用户行为:++```text+这次回答有帮助,我给一个赞。+```++执行步骤:++1. 先调用 `/bot/v1/chat` 发送问题。+2. 记录返回里的 `response_id`。+3. 再调用 `/bot/v1/feedback` 提交 `thumb_up`。++聊天命令:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "realcase-02-thumb-up",+ "user_id": "metrics-validation-user",+ "message": "帮我总结一下 OpenViking 的主要作用。"+ }'+```++反馈命令:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/feedback" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "realcase-02-thumb-up",+ "response_id": "<response_id>",+ "feedback_type": "thumb_up",+ "feedback_text": "helpful"+ }'+```++重点观察:++- `openviking_feedback_events_total{valid="1"}`+- `openviking_feedback_thumb_up_total{valid="1"}`+- `openviking_feedback_responses_with_feedback_total{valid="1"}`+- `openviking_feedback_positive_outcomes_total{valid="1"}`+- `openviking_feedback_coverage{valid="1"}`+- `openviking_feedback_thumbs_up_rate{valid="1"}`+- `openviking_feedback_one_turn_resolution_rate{valid="1"}`++PromQL:++```promql+openviking_feedback_events_total{valid="1"}+```++```promql+openviking_feedback_thumb_up_total{valid="1"}+```++```promql+openviking_feedback_positive_outcomes_total{valid="1"}+```++```promql+openviking_feedback_one_turn_resolution_rate{valid="1"}+```++预期变化:++- 会变:`feedback_events_total`、`thumb_up_total`、`responses_with_feedback_total`、`positive_outcomes_total` 通常各增加 `1`。+- 不该变:`thumb_down_total`、`negative_outcomes_total`、`reasked_outcomes_total`、`follow_up_without_feedback_outcomes_total` 在这个场景通常不应直接增加,因为当前动作是对同一条 response 提交显式正向反馈,而不是差评或追问。+- 可能不明显:`coverage`、`thumbs_up_rate`、`one_turn_resolution_rate` 通常会上升或保持不变,但如果历史 response 和历史 feedback 已经很多,显示上也可能接近不变;其中 `one_turn_resolution_rate` 会受当前实现把 `positive_feedback` 也计入 one-turn resolution 的影响。++### 场景 3:用户认为回答没帮助,点一个踩,验证负向反馈链路++用户提问:++```text+请告诉我 OpenViking 的部署步骤,越短越好。+```++预期 Vikingbot 回复特征:++- 会给出一个简短部署说明+- 返回体里能拿到 `response_id`++随后用户行为:++```text+这次回答不够有帮助,我给一个踩。+```++执行步骤:++1. 先调用 `/bot/v1/chat`。+2. 记录 `response_id`。+3. 调用 `/bot/v1/feedback`,提交 `thumb_down`。++聊天命令:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "realcase-03-thumb-down",+ "user_id": "metrics-validation-user",+ "message": "请告诉我 OpenViking 的部署步骤,越短越好。"+ }'+```++反馈命令:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/feedback" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "realcase-03-thumb-down",+ "response_id": "<response_id>",+ "feedback_type": "thumb_down",+ "feedback_text": "not helpful"+ }'+```++重点观察:++- `openviking_feedback_events_total{valid="1"}`+- `openviking_feedback_thumb_down_total{valid="1"}`+- `openviking_feedback_negative_outcomes_total{valid="1"}`+- `openviking_feedback_thumbs_down_rate{valid="1"}`+- `openviking_feedback_negative_feedback_rate{valid="1"}`++PromQL:++```promql+openviking_feedback_thumb_down_total{valid="1"}+```++```promql+openviking_feedback_negative_outcomes_total{valid="1"}+```++```promql+openviking_feedback_negative_feedback_rate{valid="1"}+```++预期变化:++- 会变:`thumb_down_total`、`negative_outcomes_total` 通常各增加 `1`,`events_total` 也应随这次显式 feedback 增加 `1`。+- 不该变:`thumb_up_total`、`positive_outcomes_total`、`reasked_outcomes_total`、`resolved_outcomes_total`、`follow_up_without_feedback_outcomes_total` 在这个场景通常不应直接增加,因为这里只提交了显式负向反馈,没有发生追问或静默结束。+- 可能不明显:`thumbs_down_rate`、`negative_feedback_rate` 通常会上升或保持不变,但如果历史样本很多,比例变化可能很小,因此仍然优先看 total 是否按预期跳变。++### 场景 4:用户没有显式反馈,但马上追问,验证 `reasked`++用户第一问:++```text+请解释 OpenViking 的 metrics 是做什么的。+```++Vikingbot 第一轮回复特征:++- 会给出解释,但用户主观上仍觉得不够具体++用户第二问:++```text+我还是没明白,请再具体一点。+```++执行步骤:++1. 用同一个 `session_id` 先发第一问。+2. 不要调用 `/bot/v1/feedback`。+3. 在 10 分钟内用同一个 `session_id` 发第二问。++第一轮命令:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "realcase-04-reask",+ "user_id": "metrics-validation-user",+ "message": "请解释 OpenViking 的 metrics 是做什么的。"+ }'+```++第二轮命令:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "realcase-04-reask",+ "user_id": "metrics-validation-user",+ "message": "我还是没明白,请再具体一点。"+ }'+```++重点观察:++- `openviking_feedback_reasked_outcomes_total{valid="1"}`+- `openviking_feedback_reask_rate{valid="1"}`++PromQL:++```promql+openviking_feedback_reasked_outcomes_total{valid="1"}+```++```promql+openviking_feedback_reask_rate{valid="1"}+```++预期变化:++- `reasked_outcomes_total` 通常增加 `1`+- `reask_rate` 通常会上升,或在历史样本很多时保持不变+- 核心不是第二条回复内容本身,而是第一条 assistant response 的 outcome 被评估成了 `reasked`+- `responses_total` 通常还会因为第二次 `/bot/v1/chat` 再增加 `1`,但这不影响“上一条 response 被改判为 `reasked`”这个核心判断+- `events_total`、`thumb_up_total`、`thumb_down_total`、`responses_with_feedback_total` 通常不变,因为这个场景没有显式 feedback+- `positive_outcomes_total`、`negative_outcomes_total`、`resolved_outcomes_total` 通常不应因为这一步直接增加++补充说明:++1. `reask_rate` 不是严格单调变化的在线 counter 比率,而是基于当前持久化 session 快照重新聚合出来的占比,所以更适合看“是否不下降、是否出现阶梯变化”。+2. 如果第二次 follow-up 已经超过 10 分钟窗口,且仍然没有显式 feedback,那么它更可能落入 `follow_up_without_feedback`,而不是 `reasked`。+3. 如果你在这个场景里看到了 `follow_up_without_feedback_outcomes_total` 增加,优先检查是不是第二次提问时间已经超过 10 分钟,或者混入了别的 session 样本。++### 场景 5:用户不点赞也不点踩,隔一段时间再追问,验证 `follow_up_without_feedback`++用户第一问:++```text+OpenViking 和普通向量数据库有什么关系?+```++Vikingbot 第一轮回复特征:++- 会给出概念性回答++用户稍后继续问:++```text+那你能再举一个更具体的例子吗?+```++执行步骤:++1. 用新的 `session_id` 发第一问。+2. 不调用 `/bot/v1/feedback`。+3. 至少等待 10 分钟以后,再用同一个 `session_id` 继续发 follow-up。++说明:当前实现里,10 分钟内的 follow-up 会优先被归类为 `reasked`;只有超过这个窗口、且没有显式反馈时,才会落入 `follow_up_without_feedback`。++第一轮命令:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "realcase-05-followup-no-feedback",+ "user_id": "metrics-validation-user",+ "message": "OpenViking 和普通向量数据库有什么关系?"+ }'+```++第二轮命令:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "realcase-05-followup-no-feedback",+ "user_id": "metrics-validation-user",+ "message": "那你能再举一个更具体的例子吗?"+ }'+```++重点观察:++- `openviking_feedback_follow_up_without_feedback_outcomes_total{valid="1"}`++PromQL:++```promql+openviking_feedback_follow_up_without_feedback_outcomes_total{valid="1"}+```++预期变化:++- 会变:`follow_up_without_feedback_outcomes_total` 通常增加 `1`;同时第二次 `/bot/v1/chat` 也会让 `responses_total` 再增加 `1`,但真正被补判为 `follow_up_without_feedback` 的仍然是第一轮 assistant response。+- 不该变:`events_total`、`thumb_up_total`、`thumb_down_total`、`responses_with_feedback_total` 通常不变,因为整个场景没有显式调用 `/bot/v1/feedback`;`positive_outcomes_total`、`negative_outcomes_total`、`reasked_outcomes_total` 也通常不应在这一步直接增长。+- 可能不明显:如果你看到增长落在 `reasked_outcomes_total`,优先检查第二次提问是不是仍在 10 分钟窗口内;rate 类指标同样可能因为历史样本较多而变化不明显,所以这个场景优先看 total 跳变。++### 场景 6:用户问完就结束,不再追问,验证 `resolved`++用户提问:++```text+请用一句话解释什么是 OpenViking 的 readiness 指标。+```++预期 Vikingbot 回复特征:++- 会给出短回答+- 用户不继续追问,也不提交反馈++执行步骤:++1. 新开一个 `session_id` 发一轮问答。+2. 不调用 `/bot/v1/feedback`。+3. 不继续发 follow-up。+4. 等待系统完成持久化与后续 outcome 评估,再观察指标。++命令:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "realcase-06-resolved",+ "user_id": "metrics-validation-user",+ "message": "请用一句话解释什么是 OpenViking 的 readiness 指标。"+ }'+```++重点观察:++- `openviking_feedback_resolved_outcomes_total{valid="1"}`++PromQL:++```promql+openviking_feedback_resolved_outcomes_total{valid="1"}+```++预期变化:++- 会变:`resolved_outcomes_total` 有机会增加 `1`,`one_turn_resolution_rate` 也可能上升或保持不变。+- 不该变:`events_total`、`thumb_up_total`、`thumb_down_total`、`responses_with_feedback_total` 在这个场景通常不变,因为既没有显式 feedback,也没有后续追问;`negative_outcomes_total`、`reasked_outcomes_total`、`follow_up_without_feedback_outcomes_total` 也通常不应直接增加。+- 可能不明显:`resolved` 相比 `thumb_up`、`thumb_down`、`reasked` 更适合作为补充验证项,因为它依赖后续 outcome 评估时机,出现时间和幅度都不如显式反馈稳定;因此看到短时间内没有立刻跳变,不一定表示链路异常。++### 场景 7:真实多 channel 用户,从指定 channel 访问,验证 `openviking_feedback_channel_*`++前提:++- 你的 bot 配置中已经启用了 `channel_id="demo"` 对应的 `bot_api` channel++用户提问:++```text+请简单介绍一下 OpenViking。+```++预期 Vikingbot 回复特征:++- 请求需要走 `POST /bot/v1/chat/channel`+- 返回样本最终应落到 `channel="bot_api__demo"`++执行步骤:++1. 用 `POST /bot/v1/chat/channel` 发起聊天。+2. 记录返回中的 `response_id`。+3. 调用 `/bot/v1/feedback`,带上同一个 `channel_id`。+4. 对照 channel 维度指标。++聊天命令:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat/channel" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "realcase-07-channel-demo",+ "user_id": "metrics-validation-user",+ "channel_id": "demo",+ "message": "请简单介绍一下 OpenViking。"+ }'+```++反馈命令:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/feedback" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "realcase-07-channel-demo",+ "response_id": "<response_id>",+ "channel_id": "demo",+ "feedback_type": "thumb_down",+ "feedback_text": "not helpful"+ }'+```++重点观察:++- `openviking_feedback_channel_events_total{channel="bot_api__demo",valid="1"}`+- `openviking_feedback_channel_thumbs_down_rate{channel="bot_api__demo",valid="1"}`+- `openviking_feedback_channel_negative_outcomes_total{channel="bot_api__demo",valid="1"}`++PromQL:++```promql+openviking_feedback_channel_events_total{channel="bot_api__demo",valid="1"}+```++```promql+openviking_feedback_channel_thumbs_down_rate{channel="bot_api__demo",valid="1"}+```++```promql+openviking_feedback_channel_negative_outcomes_total{channel="bot_api__demo",valid="1"}+```++```promql+openviking_feedback_channel_events_total{valid="1"}+```++预期变化:++- 会变:能看到 `channel="bot_api__demo"` 这一组样本,并且该 channel 下的 `events_total`、`negative_outcomes_total` 等 total 会随这次问答和负向反馈跳变。+- 不该变:其他无关 channel 的样本通常不应因为这次请求跳变;如果你误用的是 `POST /bot/v1/chat` 而不是 `POST /bot/v1/chat/channel`,通常也不会得到目标 `bot_api__demo` 的聚合样本。+- 可能不明显:`channel_thumbs_down_rate`、`channel_one_turn_resolution_rate` 这类比例在历史 channel 样本较多时可能变化不明显,所以 channel 验收优先看该 channel 的 total 是否跳变,再看 rate 方向是否合理。++如果你不确定当前环境里有哪些可用 `channel_id`,可以先检查 bot 配置里启用的 `bot_api` channel;若配置不存在,对应请求会返回 `404 Channel '<channel_id>' not found`。++### 一次性人工验收的推荐顺序++如果你只想跑一遍最实用的人工验收,建议按下面顺序执行:++1. 跑“场景 1”,确认 `/bot/v1/chat`、会话持久化和 `responses_total` 正常。+2. 跑“场景 2”,确认 `thumb_up`、`positive_outcomes_total`、`coverage` 正常。+3. 跑“场景 3”,确认 `thumb_down`、`negative_outcomes_total` 正常。+4. 跑“场景 4”,确认 `reasked_outcomes_total` 正常。+5. 跑“场景 5”,确认 `follow_up_without_feedback_outcomes_total` 正常。+6. 跑“场景 6”,把 `resolved` 作为补充验证。+7. 如果你的环境启用了 `bot_api` channel,再跑“场景 7”验证 channel 维度指标。++## 第 0 步:先看基线指标++在真正发问答之前,先确认基础链路指标是活的。++推荐依次执行下面这些查询。++服务 readiness:++```promql+openviking_service_readiness{valid="1"}+```++预期:返回 `1`。++组件健康:++```promql+openviking_component_health{valid="1"}+```++预期:++- 至少能看到 `queue`、`vikingdb` 等 component+- 正常情况下大多数值为 `1`++队列积压:++```promql+openviking_queue_pending+```++预期:++- 通常为 `0` 或较小值+- 如果此时已经长期大于 `0`,后面做问答验证时要注意区分“新流量触发”还是“原本就有积压”++VikingDB collection 当前向量数:++```promql+openviking_vikingdb_collection_vectors{valid="1"}+```++预期:++- 至少存在一部分 collection 样本+- 数值不一定在本次验证中变化,但它能证明这类状态指标已经被 Prometheus 抓到++模型使用统计可用性:++```promql+openviking_model_usage_available{valid="1"}+```++预期:++- 可能看到 `model_type="vlm"`、`embedding`、`rerank`+- 值为 `1` 表示对应统计当前可用++反馈快照总览:++```promql+openviking_feedback_sessions_scanned_total{valid="1"}+```++```promql+openviking_feedback_responses_total{valid="1"}+```++```promql+openviking_feedback_events_total{valid="1"}+```++预期:++- 如果你的环境已有历史 bot session,数值通常大于 `0`+- 如果是全新环境,也可能是 `0`++建议把下面这些值先记下来,后面每个场景都拿它们做对比:++- `openviking_feedback_responses_total{valid="1"}`+- `openviking_feedback_responses_with_feedback_total{valid="1"}`+- `openviking_feedback_events_total{valid="1"}`+- `openviking_feedback_thumb_up_total{valid="1"}`+- `openviking_feedback_thumb_down_total{valid="1"}`+- `openviking_feedback_positive_outcomes_total{valid="1"}`+- `openviking_feedback_negative_outcomes_total{valid="1"}`+- `openviking_feedback_reasked_outcomes_total{valid="1"}`+- `openviking_feedback_resolved_outcomes_total{valid="1"}`+- `openviking_feedback_follow_up_without_feedback_outcomes_total{valid="1"}`++## 第 1 步:先做一轮最小聊天,确认 bot 会话能落盘++这个场景不是为了验证反馈指标,而是为了确认 `/bot/v1/chat` 这条链路本身正常。++用户输入示例:++```text+请用一句话介绍 OpenViking,控制在 20 个字以内。+```++期望的 Vikingbot 回复特征:++- 返回 200+- 返回体里包含 `session_id`+- 返回体里包含 `response_id`+- `message` 非空++执行命令:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "metrics-validation-session-a",+ "user_id": "metrics-validation-user",+ "message": "请用一句话介绍 OpenViking,控制在 20 个字以内。"+ }'+```++返回体示例形态:++```json+{+ "session_id": "metrics-validation-session-a",+ "response_id": "<response_id>",+ "message": "..."+}+```++这一轮之后,最值得观察的不是点赞/踩指标,而是:++```promql+openviking_feedback_responses_total{valid="1"}+```++预期:++- 相比第 0 步基线,通常增加 `1`+- 如果 scrape 还没刷新,等 15-30 秒再看一次++可选观察:++```promql+openviking_queue_pending+```++预期:++- 可能短暂波动,但不保证一定变化+- 它更适合用来确认系统没有因为 bot 请求出现异常积压++## 第 2 步:显式 thumb up,验证正向反馈指标++这是最推荐先做的主路径,因为现象最稳定、最好理解。++用户问答场景:++1. 用户提问:++```text+帮我总结一下 OpenViking 的作用。+```++2. Vikingbot 给出一段简短回答。++3. 用户主观判断“这次回答有帮助”,提交 `thumb_up`。++先发起聊天:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "metrics-validation-positive",+ "user_id": "metrics-validation-user",+ "message": "帮我总结一下 OpenViking 的作用。"+ }'+```++记下返回里的 `response_id`,然后提交反馈:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/feedback" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "metrics-validation-positive",+ "response_id": "<response_id>",+ "feedback_type": "thumb_up",+ "feedback_text": "helpful"+ }'+```++期望反馈返回:++```json+{+ "accepted": true,+ "response_id": "<response_id>",+ "session_id": "metrics-validation-positive",+ "feedback_type": "thumb_up",+ "feedback_delay_sec": 1.234,+ "timestamp": "..."+}+```++重点看这些查询。++反馈事件总量:++```promql+openviking_feedback_events_total{valid="1"}+```++点赞总量:++```promql+openviking_feedback_thumb_up_total{valid="1"}+```++带反馈响应数:++```promql+openviking_feedback_responses_with_feedback_total{valid="1"}+```++正向 outcome 总量:++```promql+openviking_feedback_positive_outcomes_total{valid="1"}+```++覆盖率:++```promql+openviking_feedback_coverage{valid="1"}+```++点赞率:++```promql+openviking_feedback_thumbs_up_rate{valid="1"}+```++一轮解决率:++```promql+openviking_feedback_one_turn_resolution_rate{valid="1"}+```++预期变化:++- `openviking_feedback_events_total{valid="1"}` 增加 `1`+- `openviking_feedback_thumb_up_total{valid="1"}` 增加 `1`+- `openviking_feedback_responses_with_feedback_total{valid="1"}` 通常增加 `1`+- `openviking_feedback_positive_outcomes_total{valid="1"}` 通常增加 `1`+- `openviking_feedback_coverage{valid="1"}` 可能上升,也可能不变,取决于历史总样本+- `openviking_feedback_thumbs_up_rate{valid="1"}` 可能上升,也可能不变,取决于历史反馈结构+- `openviking_feedback_one_turn_resolution_rate{valid="1"}` 可能上升,因为当前实现把 `resolved + positive_feedback` 都计入 one-turn resolution++如果你想看“这次操作是否已经刷新进 Prometheus”,最稳妥的做法是把查询改成表格视图,连续刷新 1-2 次,而不是一开始就看时间序列折线。++## 第 3 步:显式 thumb down,验证负向反馈指标++第二个主路径是 `thumb_down`,它同样稳定,而且和正向反馈形成对照。++用户问答场景:++1. 用户提问:++```text+请告诉我 OpenViking 的部署步骤,越短越好。+```++2. Vikingbot 返回一个回答。++3. 假设用户认为这次回答没有帮助,提交 `thumb_down`。++先发聊天:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "metrics-validation-negative",+ "user_id": "metrics-validation-user",+ "message": "请告诉我 OpenViking 的部署步骤,越短越好。"+ }'+```++再提交踩:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/feedback" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "metrics-validation-negative",+ "response_id": "<response_id>",+ "feedback_type": "thumb_down",+ "feedback_text": "not helpful"+ }'+```++重点查询。++```promql+openviking_feedback_events_total{valid="1"}+```++```promql+openviking_feedback_thumb_down_total{valid="1"}+```++```promql+openviking_feedback_negative_outcomes_total{valid="1"}+```++```promql+openviking_feedback_thumbs_down_rate{valid="1"}+```++```promql+openviking_feedback_negative_feedback_rate{valid="1"}+```++预期变化:++- `openviking_feedback_events_total{valid="1"}` 再增加 `1`+- `openviking_feedback_thumb_down_total{valid="1"}` 增加 `1`+- `openviking_feedback_negative_outcomes_total{valid="1"}` 通常增加 `1`+- `openviking_feedback_thumbs_down_rate{valid="1"}` 可能上升+- `openviking_feedback_negative_feedback_rate{valid="1"}` 可能上升++这一步完成后,你已经验证了:++- `/bot/v1/feedback` 能关联到历史 `response_id`+- feedback event 已被落入 session metadata+- scrape-time feedback 聚合已经反映到 `/metrics`++## 第 4 步:follow-up 提问,验证 `reasked`++这个场景用来验证“没有显式反馈,但用户很快追问,上一条回答被判定为 `reasked`”。++用户问答场景:++1. 用户第一问:++```text+请解释 OpenViking 的 metrics 是做什么的。+```++2. Vikingbot 给出回答。++3. 用户在同一个 `session_id` 里继续追问:++```text+我还是没明白,请再具体一点。+```++第一轮聊天:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "metrics-validation-reask",+ "user_id": "metrics-validation-user",+ "message": "请解释 OpenViking 的 metrics 是做什么的。"+ }'+```++第二轮 follow-up,注意要在 10 分钟内发送:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "metrics-validation-reask",+ "user_id": "metrics-validation-user",+ "message": "我还是没明白,请再具体一点。"+ }'+```++重点查询:++```promql+openviking_feedback_reasked_outcomes_total{valid="1"}+```++```promql+openviking_feedback_reask_rate{valid="1"}+```++预期变化:++- `openviking_feedback_reasked_outcomes_total{valid="1"}` 通常增加 `1`+- `openviking_feedback_reask_rate{valid="1"}` 可能上升++这里的关键点不是第二次 `/chat` 的回复内容,而是第一条 assistant response 的 outcome 已经被落成 `reasked`。++## 第 5 步:不提交显式反馈但继续追问,验证 `follow_up_without_feedback`++这个场景和上一步类似,但要验证的是另一个 outcome 维度。++用户问答场景:++1. 用户第一问:++```text+OpenViking 和普通向量数据库有什么关系?+```++2. Vikingbot 给出回答。++3. 用户没有点赞也没有点踩,而是在 10 分钟之后继续问:++```text+那你能再举一个更具体的例子吗?+```++执行方式和第 4 步类似,但这里不要在 10 分钟内追问。应使用新的 `session_id`,不调用 `/bot/v1/feedback`,并在 assistant 回复后至少等待 10 分钟,再发送下一条 follow-up。++推荐查询:++```promql+openviking_feedback_follow_up_without_feedback_outcomes_total{valid="1"}+```++预期变化:++- 该值通常增加 `1`++说明:++- 当前实现里,`reasked` 和 `follow_up_without_feedback` 都依赖后续 user turn+- 但它们的分析口径不同,前者强调“回答后 10 分钟内被追问/重问”,后者强调“用户有 follow-up、没有显式反馈,而且 follow-up 已超出 10 分钟窗口”+- 如果你的环境里历史数据很多,单次验证造成的比例变化可能不明显,所以优先看 total 指标是否跳变++## 第 6 步:`resolved` 作为补充场景验证++`resolved` 不建议作为第一优先验证项,因为它不像 thumb up / thumb down 那样“做完一个动作就容易立刻观察”。++根据当前实现规则:++- 无显式反馈+- 没有新的 user follow-up+- 该 response 被 outcome evaluator 评估为已解决++推荐验证方法:++1. 新开一个 session 发一轮简短问答+2. 不做 `/bot/v1/feedback`+3. 不继续追问+4. 等待系统完成相关持久化与后续评估+5. 再看:++```promql+openviking_feedback_resolved_outcomes_total{valid="1"}+```++预期:++- 该值有机会增加 `1`++如果这一步没有明显变化,不应优先判定系统有问题,建议先以第 2、3、4 步的结果作为主验收依据。++## 第 7 步:按 channel 看反馈指标++当前实现支持通过 `POST /bot/v1/chat/channel` 携带 `channel_id`,把请求路由到 `bot_api__<channel_id>` 这类 channel 维度里,因此可以验证 `openviking_feedback_channel_*`。++这一步是可选项,建议在主路径都通过后再做。++聊天请求示例:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/chat/channel" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "metrics-validation-channel-demo",+ "user_id": "metrics-validation-user",+ "channel_id": "demo",+ "message": "请简单介绍一下 OpenViking。"+ }'+```++注意:普通 `POST /bot/v1/chat` 不会根据 `channel_id` 自动切到 `bot_api` 路由;按 channel 验证时应使用 `POST /bot/v1/chat/channel`。++反馈请求示例:++```bash+curl -sS -X POST "http://127.0.0.1:30300/bot/v1/feedback" \+ -H "Content-Type: application/json" \+ -d '{+ "session_id": "metrics-validation-channel-demo",+ "response_id": "<response_id>",+ "channel_id": "demo",+ "feedback_type": "thumb_down",+ "feedback_text": "not helpful"+ }'+```++重点查询:++```promql+openviking_feedback_channel_events_total{channel="bot_api__demo",valid="1"}+```++```promql+openviking_feedback_channel_thumbs_down_rate{channel="bot_api__demo",valid="1"}+```++```promql+openviking_feedback_channel_negative_outcomes_total{channel="bot_api__demo",valid="1"}+```++如果你想同时对比默认 channel 和 `bot_api__demo`,可以查:++```promql+openviking_feedback_channel_events_total{valid="1"}+```++预期:++- 能看到 `channel="bot_api__demo"`+- 对应数值会随该 channel 下的聊天/反馈变化++注意:如果你当前 bot 配置没有启用对应的 `bot_api` channel,`POST /bot/v1/chat/channel` 会直接返回 `404 Channel 'demo' not found`,需要先确认 bot 配置中存在相应 channel。最直接的方法是查看 bot 配置文件里已启用的 `bot_api` channel 定义。++## 推荐的 Grafana Explore 查询清单++如果你只想快速做一遍人工验收,可以按这个顺序执行。++基础链路:++```promql+openviking_service_readiness{valid="1"}+```++```promql+openviking_component_health{valid="1"}+```++```promql+openviking_queue_pending+```++反馈总览:++```promql+openviking_feedback_responses_total{valid="1"}+```++```promql+openviking_feedback_events_total{valid="1"}+```++```promql+openviking_feedback_coverage{valid="1"}+```++正向反馈:++```promql+openviking_feedback_thumb_up_total{valid="1"}+```++```promql+openviking_feedback_positive_outcomes_total{valid="1"}+```++负向反馈:++```promql+openviking_feedback_thumb_down_total{valid="1"}+```++```promql+openviking_feedback_negative_outcomes_total{valid="1"}+```++追问与解决:++```promql+openviking_feedback_reasked_outcomes_total{valid="1"}+```++```promql+openviking_feedback_follow_up_without_feedback_outcomes_total{valid="1"}+```++```promql+openviking_feedback_resolved_outcomes_total{valid="1"}+```++按 channel:++```promql+openviking_feedback_channel_events_total{valid="1"}+```++```promql+openviking_feedback_channel_thumbs_up_rate{valid="1"}+```++```promql+openviking_feedback_channel_thumbs_down_rate{valid="1"}+```++## 常见误判++### 1. 为什么我刚发完 `/chat`,某些反馈指标没有变化++因为反馈类指标主要来自:++- `metadata.feedback_events`+- `metadata.response_outcomes`++只有普通聊天、还没有显式 feedback 或 follow-up 时,不是所有反馈指标都会立刻动。++### 2. 为什么 total 变了,但 rate 看起来不明显++因为它们本质上是 snapshot gauge,不是面向 `rate()` 设计的 counter。优先看当前值、表格视图或短时间范围内的阶梯变化。++### 3. 为什么我只看到 `valid="0"`++说明 collector 本次刷新失败,当前暴露的是上一次成功快照。主验证时优先使用 `valid="1"`。++### 4. 为什么 channel 维度没有出现 `bot_api__demo`++可能原因通常有三个:++- 你调用的是 `POST /bot/v1/chat`,而不是 `POST /bot/v1/chat/channel`+- 请求里没带 `channel_id`+- bot 配置里并没有启用对应 `bot_api` channel++## 验收建议++如果你只做一次最小验收,建议以这四项作为“通过”标准:++1. `openviking_service_readiness{valid="1"}` 为 `1`+2. 一次 `thumb_up` 后,`openviking_feedback_events_total{valid="1"}` 和 `openviking_feedback_thumb_up_total{valid="1"}` 增加+3. 一次 `thumb_down` 后,`openviking_feedback_thumb_down_total{valid="1"}` 和 `openviking_feedback_negative_outcomes_total{valid="1"}` 增加+4. 一次 follow-up 后,`openviking_feedback_reasked_outcomes_total{valid="1"}` 增加++只要这四项通过,基本就能说明:++- bot 会话已持久化+- feedback/outcome 已写入 metadata+- `/metrics` 已成功聚合反馈快照+- Prometheus / Grafana 已能正确看到这批指标++## 相关文档++- [可观测性与排障](05-observability.md)+- [使用 Prometheus 和 Grafana 查看 OpenViking 指标](11-grafana-prometheus.md)+- [指标与 Metrics](../concepts/12-metrics.md)+- [Metrics API](../api/09-metrics.md)+- [Vikingbot 真实用户问答指标验证案例](../../../bot/docs/vikingbot-real-user-metrics-cases.md)+- [Vikingbot Phase 2 反馈链路验证指南](../../../bot/docs/vikingbot-phase2-feedback-validation-with-openviking-server.md)+- [Vikingbot Phase 3 outcome 验证指南](../../../bot/docs/vikingbot-phase3-outcome-validation-with-openviking-server.md)diff --git a/examples/grafana/docker-compose.localhost.yml b/examples/grafana/docker-compose.localhost.ymlnew file mode 100644index 000000000..ebf6de9c2--- /dev/null+++ b/examples/grafana/docker-compose.localhost.yml@@ -0,0 +1,35 @@+services:+ prometheus:+ image: prom/prometheus+ container_name: openviking-prometheus-localhost+ network_mode: host+ command:+ - --config.file=/etc/prometheus/prometheus.yml+ - --storage.tsdb.path=/prometheus+ - --web.listen-address=0.0.0.0:30909+ volumes:+ - ./prometheus.localhost.yml:/etc/prometheus/prometheus.yml:ro+ - prometheus_localhost_data:/prometheus+ restart: unless-stopped++ grafana:+ image: grafana/grafana+ container_name: openviking-grafana-localhost+ network_mode: host+ depends_on:+ - prometheus+ environment:+ GF_SECURITY_ADMIN_USER: admin+ GF_SECURITY_ADMIN_PASSWORD: admin+ GF_SERVER_HTTP_PORT: 13000+ volumes:+ - grafana_localhost_data:/var/lib/grafana+ - ./grafana/provisioning-localhost:/etc/grafana/provisioning:ro+ - ./openviking_demo_dashboard.json:/var/lib/grafana/dashboards/openviking_demo_dashboard.json:ro+ - ./openviking_feedback_baseline_dashboard.json:/var/lib/grafana/dashboards/openviking_feedback_baseline_dashboard.json:ro+ - ./openviking_token_demo_dashboard.json:/var/lib/grafana/dashboards/openviking_token_demo_dashboard.json:ro+ restart: unless-stopped++volumes:+ prometheus_localhost_data:+ grafana_localhost_data:diff --git a/examples/grafana/docker-compose.yml b/examples/grafana/docker-compose.ymlnew file mode 100644index 000000000..46cb9c130--- /dev/null+++ b/examples/grafana/docker-compose.yml@@ -0,0 +1,34 @@+services:+ prometheus:+ image: prom/prometheus+ container_name: openviking-prometheus+ ports:+ - "30909:9090"+ volumes:+ - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro+ - prometheus_data:/prometheus+ extra_hosts:+ - "host.docker.internal:host-gateway"+ restart: unless-stopped++ grafana:+ image: grafana/grafana+ container_name: openviking-grafana+ depends_on:+ - prometheus+ ports:+ - "13000:3000"+ environment:+ GF_SECURITY_ADMIN_USER: admin+ GF_SECURITY_ADMIN_PASSWORD: admin+ volumes:+ - grafana_data:/var/lib/grafana+ - ./grafana/provisioning:/etc/grafana/provisioning:ro+ - ./openviking_demo_dashboard.json:/var/lib/grafana/dashboards/openviking_demo_dashboard.json:ro+ - ./openviking_feedback_baseline_dashboard.json:/var/lib/grafana/dashboards/openviking_feedback_baseline_dashboard.json:ro+ - ./openviking_token_demo_dashboard.json:/var/lib/grafana/dashboards/openviking_token_demo_dashboard.json:ro+ restart: unless-stopped++volumes:+ prometheus_data:+ grafana_data:diff --git a/examples/grafana/grafana/provisioning-localhost/dashboards/openviking.yml b/examples/grafana/grafana/provisioning-localhost/dashboards/openviking.ymlnew file mode 100644index 000000000..bc3752690--- /dev/null+++ b/examples/grafana/grafana/provisioning-localhost/dashboards/openviking.yml@@ -0,0 +1,12 @@+apiVersion: 1++providers:+ - name: OpenViking+ orgId: 1+ folder: OpenViking+ type: file+ disableDeletion: false+ updateIntervalSeconds: 30+ allowUiUpdates: true+ options:+ path: /var/lib/grafana/dashboardsdiff --git a/examples/grafana/grafana/provisioning-localhost/datasources/prometheus.yml b/examples/grafana/grafana/provisioning-localhost/datasources/prometheus.ymlnew file mode 100644index 000000000..cdaab944c--- /dev/null+++ b/examples/grafana/grafana/provisioning-localhost/datasources/prometheus.yml@@ -0,0 +1,9 @@+apiVersion: 1++datasources:+ - name: Prometheus+ type: prometheus+ access: proxy+ url: http://127.0.0.1:30909+ isDefault: true+ editable: truediff --git a/examples/grafana/grafana/provisioning/dashboards/openviking.yml b/examples/grafana/grafana/provisioning/dashboards/openviking.ymlnew file mode 100644index 000000000..bc3752690--- /dev/null+++ b/examples/grafana/grafana/provisioning/dashboards/openviking.yml@@ -0,0 +1,12 @@+apiVersion: 1++providers:+ - name: OpenViking+ orgId: 1+ folder: OpenViking+ type: file+ disableDeletion: false+ updateIntervalSeconds: 30+ allowUiUpdates: true+ options:+ path: /var/lib/grafana/dashboardsdiff --git a/examples/grafana/grafana/provisioning/datasources/prometheus.yml b/examples/grafana/grafana/provisioning/datasources/prometheus.ymlnew file mode 100644index 000000000..1a57b69c8--- /dev/null+++ b/examples/grafana/grafana/provisioning/datasources/prometheus.yml@@ -0,0 +1,9 @@+apiVersion: 1++datasources:+ - name: Prometheus+ type: prometheus+ access: proxy+ url: http://prometheus:9090+ isDefault: true+ editable: truediff --git a/examples/grafana/openviking_feedback_baseline_dashboard.json b/examples/grafana/openviking_feedback_baseline_dashboard.jsonnew file mode 100644index 000000000..ede817618--- /dev/null+++ b/examples/grafana/openviking_feedback_baseline_dashboard.json@@ -0,0 +1,1196 @@+{+ "annotations": {+ "list": [+ {+ "builtIn": 1,+ "datasource": {+ "type": "grafana",+ "uid": "-- Grafana --"+ },+ "enable": true,+ "hide": true,+ "iconColor": "#00FFE5",+ "name": "Annotations & Alerts",+ "type": "dashboard"+ }+ ]+ },+ "editable": true,+ "fiscalYearStartMonth": 0,+ "graphTooltip": 2,+ "links": [],+ "panels": [+ {+ "collapsed": false,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 0+ },+ "id": 1,+ "panels": [],+ "title": "Feedback Baseline",+ "type": "row"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#9FB0E2",+ "value": null+ },+ {+ "color": "#00FFE5",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 5,+ "w": 4,+ "x": 0,+ "y": 1+ },+ "id": 2,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 12,+ "valueSize": 28+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "openviking_feedback_sessions_scanned_total{valid=\"1\"}",+ "legendFormat": "Sessions scanned",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Sessions",+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#9FB0E2",+ "value": null+ },+ {+ "color": "#00FFE5",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 5,+ "w": 4,+ "x": 4,+ "y": 1+ },+ "id": 3,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 12,+ "valueSize": 28+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "openviking_feedback_responses_total{valid=\"1\"}",+ "legendFormat": "Responses",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Responses",+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#9FB0E2",+ "value": null+ },+ {+ "color": "#6EE7FF",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 5,+ "w": 4,+ "x": 8,+ "y": 1+ },+ "id": 4,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 11,+ "valueSize": 24+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "openviking_feedback_responses_with_feedback_total{valid=\"1\"}",+ "legendFormat": "Responses with feedback",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "With Feedback",+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#9FB0E2",+ "value": null+ },+ {+ "color": "#FFD60A",+ "value": 1+ }+ ]+ },+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 5,+ "w": 4,+ "x": 12,+ "y": 1+ },+ "id": 5,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 12,+ "valueSize": 28+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "openviking_feedback_events_total{valid=\"1\"}",+ "legendFormat": "Feedback events",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Events",+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "decimals": 2,+ "mappings": [],+ "min": 0,+ "max": 1,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#FF3B30",+ "value": 0+ },+ {+ "color": "#FFD60A",+ "value": 0.5+ },+ {+ "color": "#00FFE5",+ "value": 0.8+ }+ ]+ },+ "unit": "percentunit"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 5,+ "w": 4,+ "x": 16,+ "y": 1+ },+ "id": 6,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 12,+ "valueSize": 28+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "openviking_feedback_coverage{valid=\"1\"}",+ "legendFormat": "Coverage",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Coverage",+ "type": "stat"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "thresholds"+ },+ "decimals": 2,+ "mappings": [],+ "min": 0,+ "max": 1,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#FF3B30",+ "value": 0+ },+ {+ "color": "#FFD60A",+ "value": 0.5+ },+ {+ "color": "#00FFE5",+ "value": 0.75+ }+ ]+ },+ "unit": "percentunit"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 5,+ "w": 4,+ "x": 20,+ "y": 1+ },+ "id": 7,+ "options": {+ "colorMode": "value",+ "graphMode": "area",+ "justifyMode": "center",+ "orientation": "auto",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showPercentChange": false,+ "text": {+ "titleSize": 11,+ "valueSize": 24+ },+ "textMode": "value_and_name",+ "wideLayout": false+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "openviking_feedback_one_turn_resolution_rate{valid=\"1\"}",+ "legendFormat": "One-turn resolution",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "1-Turn Resolution",+ "type": "stat"+ },+ {+ "collapsed": false,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 6+ },+ "id": 8,+ "panels": [],+ "title": "Totals And Rate Trends",+ "type": "row"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "drawStyle": "line",+ "fillOpacity": 12,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 2,+ "pointSize": 4,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "spanNulls": true,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "min": 0,+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 7+ },+ "id": 9,+ "options": {+ "legend": {+ "calcs": [+ "lastNotNull"+ ],+ "displayMode": "table",+ "placement": "bottom",+ "showLegend": true+ },+ "tooltip": {+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "openviking_feedback_responses_total{valid=\"1\"}",+ "legendFormat": "Responses",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_responses_with_feedback_total{valid=\"1\"}",+ "legendFormat": "Responses with feedback",+ "range": true,+ "refId": "B"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_events_total{valid=\"1\"}",+ "legendFormat": "Feedback events",+ "range": true,+ "refId": "C"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_thumb_up_total{valid=\"1\"}",+ "legendFormat": "Thumb up",+ "range": true,+ "refId": "D"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_thumb_down_total{valid=\"1\"}",+ "legendFormat": "Thumb down",+ "range": true,+ "refId": "E"+ }+ ],+ "title": "Baseline Totals Snapshot Trend",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "drawStyle": "line",+ "fillOpacity": 10,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 2,+ "pointSize": 4,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "spanNulls": true,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "decimals": 2,+ "mappings": [],+ "min": 0,+ "max": 1,+ "unit": "percentunit"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 7+ },+ "id": 10,+ "options": {+ "legend": {+ "calcs": [+ "lastNotNull"+ ],+ "displayMode": "table",+ "placement": "bottom",+ "showLegend": true+ },+ "tooltip": {+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "openviking_feedback_coverage{valid=\"1\"}",+ "legendFormat": "Coverage",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_thumbs_up_rate{valid=\"1\"}",+ "legendFormat": "Thumbs up rate",+ "range": true,+ "refId": "B"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_thumbs_down_rate{valid=\"1\"}",+ "legendFormat": "Thumbs down rate",+ "range": true,+ "refId": "C"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_positive_feedback_rate{valid=\"1\"}",+ "legendFormat": "Positive feedback rate",+ "range": true,+ "refId": "D"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_negative_feedback_rate{valid=\"1\"}",+ "legendFormat": "Negative feedback rate",+ "range": true,+ "refId": "E"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_reask_rate{valid=\"1\"}",+ "legendFormat": "Reask rate",+ "range": true,+ "refId": "F"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_one_turn_resolution_rate{valid=\"1\"}",+ "legendFormat": "One-turn resolution",+ "range": true,+ "refId": "G"+ }+ ],+ "title": "Rate Snapshot Trend",+ "type": "timeseries"+ },+ {+ "collapsed": false,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 15+ },+ "id": 11,+ "panels": [],+ "title": "Outcome Mix",+ "type": "row"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "mappings": [],+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 16+ },+ "id": 12,+ "options": {+ "displayLabels": [+ "name",+ "percent"+ ],+ "legend": {+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "values": [+ "value",+ "percent"+ ]+ },+ "pieType": "donut",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "tooltip": {+ "mode": "single",+ "sort": "none"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "openviking_feedback_positive_outcomes_total{valid=\"1\"}",+ "legendFormat": "Positive",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_negative_outcomes_total{valid=\"1\"}",+ "legendFormat": "Negative",+ "range": true,+ "refId": "B"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_reasked_outcomes_total{valid=\"1\"}",+ "legendFormat": "Reasked",+ "range": true,+ "refId": "C"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_resolved_outcomes_total{valid=\"1\"}",+ "legendFormat": "Resolved",+ "range": true,+ "refId": "D"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_follow_up_without_feedback_outcomes_total{valid=\"1\"}",+ "legendFormat": "Follow-up without feedback",+ "range": true,+ "refId": "E"+ }+ ],+ "title": "Outcome Totals",+ "type": "piechart"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "drawStyle": "line",+ "fillOpacity": 12,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 2,+ "pointSize": 4,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "spanNulls": true,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "min": 0,+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 16+ },+ "id": 13,+ "options": {+ "legend": {+ "calcs": [+ "lastNotNull"+ ],+ "displayMode": "table",+ "placement": "bottom",+ "showLegend": true+ },+ "tooltip": {+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "openviking_feedback_positive_outcomes_total{valid=\"1\"}",+ "legendFormat": "Positive",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_negative_outcomes_total{valid=\"1\"}",+ "legendFormat": "Negative",+ "range": true,+ "refId": "B"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_reasked_outcomes_total{valid=\"1\"}",+ "legendFormat": "Reasked",+ "range": true,+ "refId": "C"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_resolved_outcomes_total{valid=\"1\"}",+ "legendFormat": "Resolved",+ "range": true,+ "refId": "D"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_follow_up_without_feedback_outcomes_total{valid=\"1\"}",+ "legendFormat": "Follow-up without feedback",+ "range": true,+ "refId": "E"+ }+ ],+ "title": "Outcome Snapshot Trend",+ "type": "timeseries"+ },+ {+ "collapsed": false,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 24+ },+ "id": 14,+ "panels": [],+ "title": "Channel Breakdown",+ "type": "row"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "drawStyle": "line",+ "fillOpacity": 12,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 2,+ "pointSize": 4,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "spanNulls": true,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "min": 0,+ "unit": "short"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 0,+ "y": 25+ },+ "id": 15,+ "options": {+ "legend": {+ "calcs": [+ "lastNotNull"+ ],+ "displayMode": "table",+ "placement": "bottom",+ "showLegend": true+ },+ "tooltip": {+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "openviking_feedback_channel_events_total{channel=~\"$channel\",valid=\"1\"}",+ "legendFormat": "{{channel}}",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_channel_negative_outcomes_total{channel=~\"$channel\",valid=\"1\"}",+ "legendFormat": "{{channel}} negative",+ "range": true,+ "refId": "B"+ }+ ],+ "title": "Channel Event And Negative Outcome Trend",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "axisBorderShow": false,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "drawStyle": "line",+ "fillOpacity": 10,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineWidth": 2,+ "pointSize": 4,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "never",+ "spanNulls": true,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "decimals": 2,+ "mappings": [],+ "min": 0,+ "max": 1,+ "unit": "percentunit"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 12,+ "x": 12,+ "y": 25+ },+ "id": 16,+ "options": {+ "legend": {+ "calcs": [+ "lastNotNull"+ ],+ "displayMode": "table",+ "placement": "bottom",+ "showLegend": true+ },+ "tooltip": {+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "openviking_feedback_channel_coverage{channel=~\"$channel\",valid=\"1\"}",+ "legendFormat": "{{channel}} coverage",+ "range": true,+ "refId": "A"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_channel_thumbs_up_rate{channel=~\"$channel\",valid=\"1\"}",+ "legendFormat": "{{channel}} thumbs up",+ "range": true,+ "refId": "B"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_channel_thumbs_down_rate{channel=~\"$channel\",valid=\"1\"}",+ "legendFormat": "{{channel}} thumbs down",+ "range": true,+ "refId": "C"+ },+ {+ "editorMode": "code",+ "expr": "openviking_feedback_channel_one_turn_resolution_rate{channel=~\"$channel\",valid=\"1\"}",+ "legendFormat": "{{channel}} one-turn resolution",+ "range": true,+ "refId": "D"+ }+ ],+ "title": "Channel Rate Trend",+ "type": "timeseries"+ }+ ],+ "refresh": "15s",+ "schemaVersion": 42,+ "tags": [+ "openviking",+ "metrics",+ "prometheus",+ "feedback",+ "baseline"+ ],+ "templating": {+ "list": [+ {+ "current": {+ "text": "Prometheus",+ "value": "prometheus"+ },+ "includeAll": false,+ "label": "datasource",+ "name": "DS_PROMETHEUS",+ "options": [],+ "query": "prometheus",+ "refresh": 1,+ "regex": "",+ "type": "datasource"+ },+ {+ "current": {+ "text": "All",+ "value": [+ "$__all"+ ]+ },+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "definition": "label_values(openviking_feedback_channel_events_total{valid=\"1\"}, channel)",+ "includeAll": true,+ "label": "channel",+ "multi": true,+ "name": "channel",+ "options": [],+ "query": {+ "query": "label_values(openviking_feedback_channel_events_total{valid=\"1\"}, channel)",+ "refId": "channel-var"+ },+ "refresh": 2,+ "regex": "",+ "regexApplyTo": "value",+ "sort": 1,+ "type": "query"+ }+ ]+ },+ "time": {+ "from": "now-7d",+ "to": "now"+ },+ "timepicker": {},+ "timezone": "browser",+ "title": "OpenViking - Feedback Baseline",+ "uid": "openviking-feedback-baseline",+ "version": 1,+ "weekStart": ""+}diff --git a/examples/grafana/prometheus.localhost.yml b/examples/grafana/prometheus.localhost.ymlnew file mode 100644index 000000000..733aba991--- /dev/null+++ b/examples/grafana/prometheus.localhost.yml@@ -0,0 +1,8 @@+global:+ scrape_interval: 15s++scrape_configs:+ - job_name: openviking+ metrics_path: /metrics+ static_configs:+ - targets: ["127.0.0.1:30300"]diff --git a/examples/grafana/prometheus.yml b/examples/grafana/prometheus.ymlnew file mode 100644index 000000000..35fd8b4cf--- /dev/null+++ b/examples/grafana/prometheus.yml@@ -0,0 +1,8 @@+global:+ scrape_interval: 15s++scrape_configs:+ - job_name: openviking+ metrics_path: /metrics+ static_configs:+ - targets: ["host.docker.internal:30300"]diff --git a/openviking/metrics/bootstrap.py b/openviking/metrics/bootstrap.pyindex 7bf8eca18..9a67ef433 100644--- a/openviking/metrics/bootstrap.py+++ b/openviking/metrics/bootstrap.py@@ -26,10 +26,13 @@ Notes:from __future__ import annotations+from pathlib import Path+from openviking.metrics.collectors import (AsyncSystemProbeCollector,CollectorManager,EncryptionProbeCollector,+ FeedbackCollector,LockCollector,ModelProviderProbeCollector,ModelUsageCollector,@@ -61,7 +64,7 @@ from openviking.metrics.datasources.task import TaskStateDataSourcefrom openviking_cli.utils.config.open_viking_config import get_openviking_config-def create_default_collector_manager(*, app=None, service=None) -> CollectorManager:+def create_default_collector_manager(*, app=None, service=None, config=None) -> CollectorManager:"""Build the default `CollectorManager` used by the server's Prometheus endpoint.@@ -80,6 +83,9 @@ def create_default_collector_manager(*, app=None, service=None) -> CollectorManamanager = CollectorManager()manager.register(QueueCollector(data_source=QueuePipelineStateDataSource()))manager.register(TaskTrackerCollector(data_source=TaskStateDataSource()))+ feedback_bot_data_path = _resolve_feedback_bot_data_path(config)+ if feedback_bot_data_path is not None:+ manager.register(FeedbackCollector(bot_data_path=feedback_bot_data_path))manager.register(ObserverHealthCollector(data_source=ObserverStateDataSource(service=service)))manager.register(ObserverStateCollector(data_source=ObserverStateDataSource(service=service)))manager.register(LockCollector(data_source=LockStateDataSource()))@@ -108,3 +114,13 @@ def create_default_collector_manager(*, app=None, service=None) -> CollectorMana))return manager+++def _resolve_feedback_bot_data_path(config) -> Path | None:+ if config is None:+ return None+ metrics_config = getattr(getattr(config, "observability", None), "metrics", None)+ bot_data_path = getattr(metrics_config, "bot_data_path", None)+ if not bot_data_path:+ return None+ return Path(bot_data_path)diff --git a/openviking/metrics/collectors/__init__.py b/openviking/metrics/collectors/__init__.pyindex 7f63d51ca..985a64648 100644--- a/openviking/metrics/collectors/__init__.py+++ b/openviking/metrics/collectors/__init__.py@@ -16,6 +16,7 @@ from .cache import CacheCollectorfrom .embedding import EmbeddingCollectorfrom .encryption import EncryptionCollectorfrom .encryption_probe import EncryptionProbeCollector+from .feedback import FeedbackCollectorfrom .lock import LockCollectorfrom .manager import CollectorManager, RefreshResultfrom .model_provider_probe import ModelProviderProbeCollector@@ -44,6 +45,7 @@ __all__ = ["VLMCollector","CacheCollector","EncryptionCollector",+ "FeedbackCollector","QueueCollector","RerankCollector","LockCollector",diff --git a/openviking/metrics/collectors/feedback.py b/openviking/metrics/collectors/feedback.pynew file mode 100644index 000000000..285c0cc22--- /dev/null+++ b/openviking/metrics/collectors/feedback.py@@ -0,0 +1,191 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++import importlib+from dataclasses import dataclass, field+from pathlib import Path+from typing import Any, ClassVar++from openviking.metrics.core.base import MetricCollector++from .base import CollectorConfig, DomainStatsMetricCollector+++def load_config():+ module = importlib.import_module("vikingbot.config.loader")+ return module.load_config()+++def compute_feedback_stats(*args, **kwargs):+ module = importlib.import_module("vikingbot.observability.feedback_stats")+ return module.compute_feedback_stats(*args, **kwargs)+++@dataclass+class FeedbackCollector(DomainStatsMetricCollector):+ """Export offline VikingBot feedback aggregates for Prometheus and Grafana."""++ DOMAIN: ClassVar[str] = "feedback"++ SESSIONS_SCANNED_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "sessions_scanned", unit="total"+ )+ RESPONSES_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "responses", unit="total")+ TRACKED_RESPONSES_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "tracked_responses", unit="total"+ )+ RESPONSES_WITH_FEEDBACK_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "responses_with_feedback", unit="total"+ )+ EVENTS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "events", unit="total")+ THUMB_UP_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "thumb_up", unit="total")+ THUMB_DOWN_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "thumb_down", unit="total"+ )+ POSITIVE_OUTCOMES_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "positive_outcomes", unit="total"+ )+ NEGATIVE_OUTCOMES_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "negative_outcomes", unit="total"+ )+ REASKED_OUTCOMES_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "reasked_outcomes", unit="total"+ )+ RESOLVED_OUTCOMES_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "resolved_outcomes", unit="total"+ )+ FOLLOW_UP_WITHOUT_FEEDBACK_OUTCOMES_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "follow_up_without_feedback_outcomes", unit="total"+ )+ COVERAGE: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "coverage")+ THUMBS_UP_RATE: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "thumbs_up_rate")+ THUMBS_DOWN_RATE: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "thumbs_down_rate")+ POSITIVE_FEEDBACK_RATE: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "positive_feedback_rate"+ )+ NEGATIVE_FEEDBACK_RATE: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "negative_feedback_rate"+ )+ REASK_RATE: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "reask_rate")+ ONE_TURN_RESOLUTION_RATE: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "one_turn_resolution_rate"+ )++ CHANNEL_RESPONSES_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "channel_responses", unit="total"+ )+ CHANNEL_TRACKED_RESPONSES_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "channel_tracked_responses", unit="total"+ )+ CHANNEL_EVENTS_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "channel_events", unit="total"+ )+ CHANNEL_NEGATIVE_OUTCOMES_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "channel_negative_outcomes", unit="total"+ )+ CHANNEL_REASKED_OUTCOMES_TOTAL: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "channel_reasked_outcomes", unit="total"+ )+ CHANNEL_COVERAGE: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "channel_coverage")+ CHANNEL_THUMBS_UP_RATE: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "channel_thumbs_up_rate"+ )+ CHANNEL_THUMBS_DOWN_RATE: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "channel_thumbs_down_rate"+ )+ CHANNEL_ONE_TURN_RESOLUTION_RATE: ClassVar[str] = MetricCollector.metric_name(+ DOMAIN, "channel_one_turn_resolution_rate"+ )++ _SUMMARY_METRICS: ClassVar[tuple[tuple[str, str], ...]] = (+ (SESSIONS_SCANNED_TOTAL, "sessions_scanned"),+ (RESPONSES_TOTAL, "responses_total"),+ (TRACKED_RESPONSES_TOTAL, "tracked_responses_total"),+ (RESPONSES_WITH_FEEDBACK_TOTAL, "responses_with_feedback"),+ (EVENTS_TOTAL, "feedback_total"),+ (THUMB_UP_TOTAL, "thumb_up_total"),+ (THUMB_DOWN_TOTAL, "thumb_down_total"),+ (POSITIVE_OUTCOMES_TOTAL, "positive_feedback_total"),+ (NEGATIVE_OUTCOMES_TOTAL, "negative_feedback_total"),+ (REASKED_OUTCOMES_TOTAL, "reasked_total"),+ (RESOLVED_OUTCOMES_TOTAL, "resolved_total"),+ (FOLLOW_UP_WITHOUT_FEEDBACK_OUTCOMES_TOTAL, "follow_up_without_feedback_total"),+ (COVERAGE, "feedback_coverage"),+ (THUMBS_UP_RATE, "thumbs_up_rate"),+ (THUMBS_DOWN_RATE, "thumbs_down_rate"),+ (POSITIVE_FEEDBACK_RATE, "positive_feedback_rate"),+ (NEGATIVE_FEEDBACK_RATE, "negative_feedback_rate"),+ (REASK_RATE, "reask_rate"),+ (ONE_TURN_RESOLUTION_RATE, "one_turn_resolution_rate"),+ )++ _CHANNEL_METRICS: ClassVar[tuple[tuple[str, str], ...]] = (+ (CHANNEL_RESPONSES_TOTAL, "responses_total"),+ (CHANNEL_TRACKED_RESPONSES_TOTAL, "tracked_responses_total"),+ (CHANNEL_EVENTS_TOTAL, "feedback_total"),+ (CHANNEL_NEGATIVE_OUTCOMES_TOTAL, "negative_feedback_total"),+ (CHANNEL_REASKED_OUTCOMES_TOTAL, "reasked_total"),+ (CHANNEL_COVERAGE, "feedback_coverage"),+ (CHANNEL_THUMBS_UP_RATE, "thumbs_up_rate"),+ (CHANNEL_THUMBS_DOWN_RATE, "thumbs_down_rate"),+ (CHANNEL_ONE_TURN_RESOLUTION_RATE, "one_turn_resolution_rate"),+ )++ bot_data_path: Path | None = None+ config: CollectorConfig = CollectorConfig(ttl_seconds=30.0, timeout_seconds=1.0)+ _last_summary: dict[str, Any] = field(default_factory=dict, init=False, repr=False)+ _last_channels: dict[str, dict[str, Any]] = field(default_factory=dict, init=False, repr=False)++ def read_metric_input(self) -> dict[str, Any]:+ """Read the latest persisted feedback aggregates from the bot session store."""+ return compute_feedback_stats(self._resolve_bot_data_path(), include_sessions=False)++ def collect_hook(self, registry, metric_input) -> None:+ """Export the latest summary and per-channel feedback gauges with `valid=1`."""+ self._last_summary = dict(metric_input.get("summary") or {})+ self._last_channels = {+ str(channel): dict(stats or {})+ for channel, stats in (metric_input.get("channels") or {}).items()+ }+ self._write(registry, valid="1", summary=self._last_summary, channels=self._last_channels)++ def collect_error_hook(self, registry, error: Exception) -> None:+ """Re-publish the last successful feedback snapshot with `valid=0` on failure."""+ self._write(registry, valid="0", summary=self._last_summary, channels=self._last_channels)++ def _resolve_bot_data_path(self) -> Path:+ if self.bot_data_path is not None:+ return Path(self.bot_data_path)+ return Path(load_config().bot_data_path)++ def _write(+ self,+ registry,+ *,+ valid: str,+ summary: dict[str, Any],+ channels: dict[str, dict[str, Any]],+ ) -> None:+ for metric_name, _ in self._SUMMARY_METRICS:+ registry.gauge_delete_matching(metric_name, match_labels={})+ for metric_name, key in self._SUMMARY_METRICS:+ registry.set_gauge(+ metric_name,+ float(summary.get(key, 0.0) or 0.0),+ labels={"valid": str(valid)},+ label_names=("valid",),+ )++ for metric_name, _ in self._CHANNEL_METRICS:+ registry.gauge_delete_matching(metric_name, match_labels={})+ for channel, channel_stats in channels.items():+ labels = {"channel": str(channel), "valid": str(valid)}+ for metric_name, key in self._CHANNEL_METRICS:+ registry.set_gauge(+ metric_name,+ float(channel_stats.get(key, 0.0) or 0.0),+ labels=labels,+ label_names=("channel", "valid"),+ )diff --git a/openviking/metrics/global_api.py b/openviking/metrics/global_api.pyindex b11aa07ca..c5d7f3891 100644--- a/openviking/metrics/global_api.py+++ b/openviking/metrics/global_api.py@@ -161,7 +161,7 @@ def init_metrics_from_server_config(max_active_accounts=int(account_dimension.max_active_accounts),)_registry = registry or MetricRegistry()- collector_manager = create_default_collector_manager(app=app, service=service)+ collector_manager = create_default_collector_manager(app=app, service=service, config=config)_event_router = _build_event_router(_registry)register_event_subscriber(_METRICS_EVENT_SUBSCRIBER,diff --git a/openviking/server/config.py b/openviking/server/config.pyindex 0a4ac8bb5..0062742ac 100644--- a/openviking/server/config.py+++ b/openviking/server/config.py@@ -77,6 +77,7 @@ class MetricsConfig(BaseModel):"""Metrics subsystem configuration."""enabled: bool = False+ bot_data_path: Optional[str] = Noneaccount_dimension: MetricsAccountDimensionConfig = Field(default_factory=MetricsAccountDimensionConfig)diff --git a/tests/metrics/collectors/test_feedback_collector.py b/tests/metrics/collectors/test_feedback_collector.pynew file mode 100644index 000000000..9ab27e5c7--- /dev/null+++ b/tests/metrics/collectors/test_feedback_collector.py@@ -0,0 +1,170 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from __future__ import annotations++import json++from openviking.metrics.collectors.feedback import FeedbackCollector+++def test_feedback_collector_exports_summary_and_channel_gauges(+ registry, render_prometheus, tmp_path+):+ sessions_dir = tmp_path / "bot" / "sessions"+ sessions_dir.mkdir(parents=True)++ (sessions_dir / "cli__default__session-1.jsonl").write_text(+ "\n".join(+ [+ json.dumps(+ {+ "_type": "metadata",+ "session_key": "cli__default__session-1",+ "updated_at": "2026-05-01T10:00:00",+ "metadata": {+ "feedback_events": [+ {"response_id": "resp-1", "feedback_type": "thumb_up"}+ ],+ "response_outcomes": {+ "resp-1": {"outcome_label": "positive_feedback"},+ "resp-2": {"outcome_label": "resolved"},+ },+ },+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 1",+ "response_id": "resp-1",+ "timestamp": "2026-05-01T10:00:00",+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 2",+ "response_id": "resp-2",+ "timestamp": "2026-05-01T10:01:00",+ }+ ),+ ]+ )+ + "\n",+ encoding="utf-8",+ )+ (sessions_dir / "bot_api__demo__session-2.jsonl").write_text(+ "\n".join(+ [+ json.dumps(+ {+ "_type": "metadata",+ "session_key": "bot_api__demo__session-2",+ "updated_at": "2026-05-03T10:00:00",+ "metadata": {+ "feedback_events": [+ {"response_id": "resp-3", "feedback_type": "thumb_down"},+ {"response_id": "resp-3", "feedback_type": "thumb_down"},+ ],+ "response_outcomes": {+ "resp-3": {"outcome_label": "negative_feedback"},+ "resp-4": {"outcome_label": "reasked"},+ },+ },+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 3",+ "response_id": "resp-3",+ "timestamp": "2026-05-03T10:00:00",+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 4",+ "response_id": "resp-4",+ "timestamp": "2026-05-03T10:01:00",+ }+ ),+ ]+ )+ + "\n",+ encoding="utf-8",+ )++ FeedbackCollector(bot_data_path=tmp_path / "bot").collect(registry)+ text = render_prometheus(registry)++ assert 'openviking_feedback_sessions_scanned_total{valid="1"} 2.0' in text+ assert 'openviking_feedback_responses_total{valid="1"} 4.0' in text+ assert 'openviking_feedback_tracked_responses_total{valid="1"} 4.0' in text+ assert 'openviking_feedback_events_total{valid="1"} 3.0' in text+ assert 'openviking_feedback_coverage{valid="1"} 0.5' in text+ assert 'openviking_feedback_one_turn_resolution_rate{valid="1"} 0.5' in text+ assert (+ 'openviking_feedback_channel_tracked_responses_total{channel="cli__default",valid="1"} 2.0'+ in text+ )+ assert 'openviking_feedback_channel_events_total{channel="cli__default",valid="1"} 1.0' in text+ assert 'openviking_feedback_channel_events_total{channel="bot_api__demo",valid="1"} 2.0' in text+ assert (+ 'openviking_feedback_channel_thumbs_down_rate{channel="bot_api__demo",valid="1"} 1.0'+ in text+ )+++def test_feedback_collector_reuses_last_values_with_valid_zero_on_failure(+ registry, render_prometheus, tmp_path, monkeypatch+):+ sessions_dir = tmp_path / "bot" / "sessions"+ sessions_dir.mkdir(parents=True)++ (sessions_dir / "cli__default__session-1.jsonl").write_text(+ "\n".join(+ [+ json.dumps(+ {+ "_type": "metadata",+ "session_key": "cli__default__session-1",+ "updated_at": "2026-05-01T10:00:00",+ "metadata": {+ "feedback_events": [+ {"response_id": "resp-1", "feedback_type": "thumb_up"}+ ],+ "response_outcomes": {+ "resp-1": {"outcome_label": "positive_feedback"}+ },+ },+ }+ ),+ json.dumps(+ {+ "role": "assistant",+ "content": "answer 1",+ "response_id": "resp-1",+ "timestamp": "2026-05-01T10:00:00",+ }+ ),+ ]+ )+ + "\n",+ encoding="utf-8",+ )++ collector = FeedbackCollector(bot_data_path=tmp_path / "bot")+ collector.collect(registry)++ def _boom(*args, **kwargs):+ raise RuntimeError("boom")++ monkeypatch.setattr("openviking.metrics.collectors.feedback.compute_feedback_stats", _boom)+ collector.collect(registry)+ text = render_prometheus(registry)++ assert 'openviking_feedback_events_total{valid="0"} 1.0' in text+ assert 'openviking_feedback_coverage{valid="0"} 1.0' in text+ assert 'openviking_feedback_channel_events_total{channel="cli__default",valid="0"} 1.0' in textdiff --git a/tests/metrics/integration/test_bootstrap.py b/tests/metrics/integration/test_bootstrap.pyindex 1226db7ee..b26e961be 100644--- a/tests/metrics/integration/test_bootstrap.py+++ b/tests/metrics/integration/test_bootstrap.py@@ -40,6 +40,53 @@ def test_create_default_collector_manager_registers_expected_collectors_in_order]+def test_create_default_collector_manager_registers_feedback_collector_when_bot_path_configured():+ class _MetricsConfig:+ bot_data_path = "/tmp/bot"++ class _ObservabilityConfig:+ metrics = _MetricsConfig()++ class _Config:+ observability = _ObservabilityConfig()++ manager = bootstrap.create_default_collector_manager(app=None, service=None, config=_Config())++ names = [type(c).__name__ for c in manager._collectors]+ assert names == [+ "QueueCollector",+ "TaskTrackerCollector",+ "FeedbackCollector",+ "ObserverHealthCollector",+ "ObserverStateCollector",+ "LockCollector",+ "VikingDBCollector",+ "ModelUsageCollector",+ "ServiceProbeCollector",+ "StorageProbeCollector",+ "RetrievalBackendProbeCollector",+ "ModelProviderProbeCollector",+ "AsyncSystemProbeCollector",+ "EncryptionProbeCollector",+ ]+++def test_create_default_collector_manager_skips_feedback_collector_without_bot_path():+ class _MetricsConfig:+ bot_data_path = None++ class _ObservabilityConfig:+ metrics = _MetricsConfig()++ class _Config:+ observability = _ObservabilityConfig()++ manager = bootstrap.create_default_collector_manager(app=None, service=None, config=_Config())++ names = [type(c).__name__ for c in manager._collectors]+ assert "FeedbackCollector" not in names++def test_create_default_collector_manager_propagates_construction_failures(monkeypatch):def _boom():raise RuntimeError("cannot init datasource")diff --git a/tests/server/test_prometheus_metrics.py b/tests/server/test_prometheus_metrics.pyindex 345edec23..977750b0f 100644--- a/tests/server/test_prometheus_metrics.py+++ b/tests/server/test_prometheus_metrics.py@@ -3,6 +3,8 @@"""Tests for the Prometheus metrics endpoint and exposition output."""+import json+import httpximport pytest@@ -191,6 +193,69 @@ class TestMetricsEndpoint:finally:shutdown_metrics(app=app)+ async def test_metrics_endpoint_exports_feedback_metrics(self, monkeypatch, tmp_path):+ sessions_dir = tmp_path / "bot" / "sessions"+ sessions_dir.mkdir(parents=True)+ (sessions_dir / "cli__default__session-1.jsonl").write_text(+ json.dumps(+ {+ "_type": "metadata",+ "session_key": "cli__default__session-1",+ "updated_at": "2026-05-01T10:00:00",+ "metadata": {+ "feedback_events": [{"response_id": "resp-1", "feedback_type": "thumb_up"}],+ "response_outcomes": {+ "resp-1": {"outcome_label": "positive_feedback"},+ "resp-2": {"outcome_label": "resolved"},+ },+ },+ }+ )+ + "\n",+ encoding="utf-8",+ )++ config = ServerConfig(+ observability=ObservabilityConfig(+ metrics=MetricsConfig(+ enabled=True,+ bot_data_path=str(tmp_path / "bot"),+ exporters=MetricsExportersConfig(+ prometheus=PrometheusExporterConfig(enabled=True)+ ),+ )+ )+ )+ app = create_app(config=config, service=None)+ init_metrics_from_server_config(config, app=app)+ transport = httpx.ASGITransport(app=app)+ try:+ async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:+ resp = await client.get("/metrics")+ assert resp.status_code == 200+ assert 'openviking_feedback_events_total{valid="1"} 1.0' in resp.text+ assert (+ 'openviking_feedback_channel_events_total{channel="cli__default",valid="1"} 1.0'+ in resp.text+ )+ finally:+ shutdown_metrics(app=app)++ def test_default_collector_manager_import_does_not_require_vikingbot(self, monkeypatch):+ import importlib+ import sys++ monkeypatch.setitem(sys.modules, "vikingbot", None)+ monkeypatch.setitem(sys.modules, "vikingbot.config", None)+ monkeypatch.setitem(sys.modules, "vikingbot.config.loader", None)+ monkeypatch.setitem(sys.modules, "vikingbot.observability", None)+ monkeypatch.setitem(sys.modules, "vikingbot.observability.feedback_stats", None)++ bootstrap = importlib.reload(importlib.import_module("openviking.metrics.bootstrap"))+ manager = bootstrap.create_default_collector_manager(app=None, service=None)++ assert "FeedbackCollector" not in [type(c).__name__ for c in manager._collectors]+def test_reinitializing_metrics_shuts_down_existing_exporters(monkeypatch):class FakeOTelExporter: