Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
fix(gateway): read cache usage from every upstream form and warn about long conversations missing the cache (#6267)
t0saki27db441230f4
diff --git a/docs/en/guides/22-gateway-operations.md b/docs/en/guides/22-gateway-operations.mdindex 465646967..c02299303 100644--- a/docs/en/guides/22-gateway-operations.md+++ b/docs/en/guides/22-gateway-operations.md@@ -388,6 +388,7 @@ The **Overview** tab summarizes the latest 10,000 requests within the log retent- **First-call cache hit rate**: the share of input tokens served from the provider's prompt cache on the first model call of each new user message. This is the most important sign of a healthy gateway. It should stay close to what the provider reaches without the gateway. A clear drop usually means the history the gateway sends no longer matches the previous request; see [The first-call cache hit rate dropped](#the-first-call-cache-hit-rate-dropped).- **Within-turn cache hit rate**: the same for later calls in a turn, such as tool steps and OpenViking tool rounds. It is usually high.- **Memory entries recalled**: entries added to new messages, and how long an OpenViking search took on average (counting only messages that searched).+- **Long conversations missing the cache**: a warning when a conversation's last 3 model calls each sent at least 32,000 input tokens, less than 5 minutes apart, and the provider reported 0 cached tokens every time. It lists the model, upstream and streak; when they are all Anthropic conversations that never wrote the cache either, the suggestion is to check that the client sets `cache_control`. Compaction, new context windows, a switched model or upstream, and requests where gateway tools come and go do not count. When an upstream has not reported any cache hit lately, a note says long conversations through it keep showing a 0% cache hit rate instead. See [The first-call cache hit rate dropped](#the-first-call-cache-hit-rate-dropped).- **OpenViking**: Connected, Degraded or Starting, with the version, and the reason when degraded. While OpenViking is unreachable, requests still reach the model without memory. The card's **Saving conversations** part counts conversations that are retrying or have paused saving.- **Degraded requests**: how often each issue occurred, with an explanation; see [Issues on degraded requests](#issues-on-degraded-requests).- **Recent requests**, with a link to the Requests tab.@@ -874,6 +875,7 @@ Compare it with what the provider reaches without the gateway, then look at the- **Upstream switched** (`upstream_changed`): the conversation's upstream was turned off, is not bound to the key the client now uses, or no longer serves the model, so the conversation moved to another upstream.- **Cache parameters changed** (`ark_cache_parameters_changed`): on Ark or ModelArk, the client changed the model, thinking mode, sampling parameters, system prompt or tools within a conversation.- **No issue flagged**: the client itself may change earlier messages or its system prompt every turn, for example by inserting the current time, which the gateway cannot fix. Each compaction, and each new context window the model starts, also costs one miss; that is expected.+- **Always 0%**: first compare with the cache records of your provider or relay. Hits there mean the upstream isn't passing cache usage back to the gateway, for example a relay that drops the cache fields when it converts protocols; the gateway reads the common fields, such as `cache_read_input_tokens`, `prompt_tokens_details.cached_tokens`, `input_tokens_details.cached_tokens` and `prompt_cache_hit_tokens`. No hits there: check that the provider supports prompt caching, that an Anthropic client sets `cache_control`, and that the relay isn't rotating between accounts or channels.### Long conversations hit the context limitdiff --git a/docs/zh/guides/22-gateway-operations.md b/docs/zh/guides/22-gateway-operations.mdindex c1a27e969..c659af57f 100644--- a/docs/zh/guides/22-gateway-operations.md+++ b/docs/zh/guides/22-gateway-operations.md@@ -388,6 +388,7 @@ ov.example.com {- **首次调用缓存命中率**:每条新用户消息的第一次模型调用中,命中服务商提示缓存的输入 token 所占比例。这是判断网关是否健康最重要的指标,应该接近不经过网关时服务商能达到的水平;明显下降通常说明网关发送的历史和上一次对不上了,见[首次调用缓存命中率下降](#首次调用缓存命中率下降)。- **轮内缓存命中率**:同一轮里后续调用的命中率,例如工具步骤和 OpenViking 工具的往返,通常很高。- **召回的记忆条数**:补充到新消息中的条数,以及每次检索 OpenViking 的平均耗时(只统计实际检索了的消息)。+- **长对话没有命中缓存**:某段对话最近连续 3 次模型调用的输入都不少于 32,000 token、相邻两次相隔不到 5 分钟,服务商返回的缓存命中却都是 0 时出现警告,列出模型、上游和连续次数;如果都是 Anthropic 对话、上游也从没写入缓存,建议改为检查客户端是否设置了 `cache_control`。压缩、开启新窗口、切换模型或上游,以及网关工具时有时无的请求不计入。某个上游最近从没返回过缓存命中时,改为一条提示,说明经过它的长对话缓存命中率一直是 0%。见[首次调用缓存命中率下降](#首次调用缓存命中率下降)。- **OpenViking**:已连接、异常或启动中,附带版本;异常时给出原因。OpenViking 连不上期间,请求照常发给模型,只是不带记忆。卡片中的**保存对话**一栏统计正在重试或已暂停保存的对话数。- **降级请求**:每种异常出现的次数和说明,见[降级请求的异常](#降级请求的异常)。- **最近请求**,以及前往“请求日志”标签页的链接。@@ -874,6 +875,7 @@ URL 必须是 HTTP(S),不能含凭证、查询参数或 fragment。未知字- **上游已切换**(`upstream_changed`):对话原来的上游被停用、没有绑定到客户端当前用的密钥,或者不再提供这个模型,于是对话转到了另一个上游。- **缓存参数有变化**(`ark_cache_parameters_changed`):在火山方舟或 BytePlus 方舟上,客户端在同一段对话里改变了模型、思考模式、采样参数、系统提示词或工具。- **没有标出异常**:客户端本身可能每轮都在改动之前的消息或系统提示词,例如插入当前时间,网关无法修正。每次压缩、模型每次开启新的上下文窗口,也都会造成一次未命中,这是预期内的。+- **一直是 0%**:先对照服务商或中转服务后台的缓存记录。那边有命中,说明上游没有把缓存用量传回网关,例如中转服务转换协议时丢掉了缓存字段;网关读取 `cache_read_input_tokens`、`prompt_tokens_details.cached_tokens`、`input_tokens_details.cached_tokens`、`prompt_cache_hit_tokens` 等常见字段。那边也没有,检查服务商是否支持提示缓存、Anthropic 客户端是否设置了 `cache_control`,以及中转服务是否在多个账号或渠道之间轮换。### 长对话超出上下文长度diff --git a/openviking_gateway/management.py b/openviking_gateway/management.pyindex d8844b898..0ad8650f1 100644--- a/openviking_gateway/management.py+++ b/openviking_gateway/management.py@@ -20,6 +20,7 @@ from .capture import reset_capturefrom .client import VikingErrorfrom .config import MOUNT_PATHfrom .models import CaptureReset, KeyRequest, Policy, Upstream, UpstreamTestRequest+from .overview import overview_summaryfrom .protocols import text_contentfrom .proxy import fill_key, upstream_headers, upstream_key, upstream_urlfrom .records import RecordKind as K@@ -248,7 +249,7 @@ def register_management(app):who = await caller(request)logs = await management.logs(who.account_id, 10000, user_id=who.log_user)return {- **overview_summary(logs),+ **overview_summary(logs, time.time()),"openviking": app.state.health,"log_retention_days": app.state.config.log_retention_days,}@@ -623,55 +624,3 @@ def keep_upstream_secrets(payload, previous):if value or name in stored}return payload---def overview_summary(logs):- """Aggregate newest-first request-log records into the management overview."""- requests = [log for log in logs if log.get("kind") != "capture"]- groups = {}- for label, kinds in (("first_call", {"user"}), ("continuation", {"continuation"})):- subset = [log for log in requests if log.get("kind") in kinds]- total = sum(- log.get("first_upstream_input_tokens", log.get("input_tokens", 0)) for log in subset- )- cached = sum(- log.get("first_upstream_cached_tokens", log.get("cached_tokens", 0)) for log in subset- )- hidden_calls = 0- if label == "continuation":- total += sum(log.get("hidden_upstream_input_tokens", 0) for log in requests)- cached += sum(log.get("hidden_upstream_cached_tokens", 0) for log in requests)- hidden_calls = sum(log.get("hidden_upstream_calls", 0) for log in requests)- groups[label] = {- "requests": len(subset) + hidden_calls,- "input_tokens": total,- "cached_tokens": cached,- "cache_hit_ratio": cached / total if total else 0,- }- degradations = {}- for log in logs:- if reason := log.get("degradation"):- degradations[reason] = degradations.get(reason, 0) + 1- # "disabled" means recall was skipped (off, budget spent or query too short).- recalls = [log for log in requests if log.get("recall_reason", "disabled") != "disabled"]- capture = {}- for log in logs:- if log.get("session") and log.get("capture_status"):- capture.setdefault((log["session"], log.get("protocol")), log["capture_status"])- return {- "requests": len(requests),- "last_request_at": requests[0].get("time") if requests else None,- "output_tokens": sum(log.get("output_tokens", 0) for log in requests),- "cache": groups,- "degradations": degradations,- "recall_count": sum(log.get("recall_count", 0) for log in requests),- "recall_requests": len(recalls),- "recall_ms": sum(log.get("recall_ms", 0) for log in recalls) / len(recalls)- if recalls- else 0,- "capture_issues": {- status: sum(1 for value in capture.values() if value == status)- for status in ("retrying", "paused")- },- "sample_limit": 10000,- }diff --git a/openviking_gateway/overview.py b/openviking_gateway/overview.pynew file mode 100644index 000000000..b97916335--- /dev/null+++ b/openviking_gateway/overview.py@@ -0,0 +1,174 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Management overview: request-log aggregates and prompt-cache alerts."""++# A conversation this long (input tokens) that misses the cache this many calls in a row+# raises an alert; the page lists the newest few.+LONG_CONTEXT = 32_000+MISS_STREAK = 3+LISTED_MISSES = 3+# Seconds; Anthropic's default cache lifetime, the shortest among providers.+CACHE_TTL = 300+# A call this many times the size of the next one was cut down by the client+# (compacted or cleared), so the next could not hit.+CUT_RATIO = 1.3+# Seconds; older calls neither raise alerts nor show an upstream hits.+STALE_AFTER = 24 * 3600+# Degradations that change the prompt prefix, so the next call cannot hit; persistent+# ones show on the degraded-requests card instead.+PREFIX_RESETS = {+ "upstream_changed",+ "missing_injection_record",+ "hidden_tool_history_unavailable",+ "ark_cache_parameters_changed",+}+++def first_upstream(log, name, default=None):+ """The first upstream call's usage field, falling back to the request's own."""+ return log.get(f"first_upstream_{name}", log.get(name, default))+++def started(log):+ """When the request reached the gateway; logs are written once it finishes."""+ return log["time"] - log.get("duration_ms", 0) / 1000+++def miss_streak(calls):+ """The newest calls of one conversation that each sent a long prompt and missed a cache+ the call before should have written."""+ streak = []+ for call in sorted(calls, key=started, reverse=True):+ size = first_upstream(call, "input_tokens")+ if streak:+ newer = streak[-1]+ gap = started(newer) - call["time"]+ if gap < 0:+ continue # Concurrent with the newer call, so neither could warm the other.+ if (+ gap > CACHE_TTL+ or size > first_upstream(newer, "input_tokens") * CUT_RATIO+ or any(+ call.get(name) != newer.get(name)+ for name in ("model", "upstream_id", "tool_skip_reason")+ )+ or bool(call.get("tools_tokens")) != bool(newer.get("tools_tokens"))+ ):+ break+ if (+ size < LONG_CONTEXT+ or first_upstream(call, "cached_tokens")+ or call.get("window_reset")+ or call.get("compaction_tokens")+ or call.get("degradation") in PREFIX_RESETS+ ):+ break+ streak.append(call)+ return streak+++def cache_alerts(requests, now):+ """Long conversations that keep missing the prompt cache, and upstreams that have not+ hit it lately."""+ recent = [log for log in requests if now - log["time"] <= STALE_AFTER]+ hit_fields = ("first_upstream_cached_tokens", "hidden_upstream_cached_tokens", "cached_tokens")+ hits = {log["upstream_id"] for log in recent if any(log.get(n) for n in hit_fields)}+ conversations = {}+ for log in recent:+ if (+ log.get("kind") in {"user", "continuation"}+ and log.get("session")+ # Only answered upstream calls carry usage, along with their status and upstream.+ and first_upstream(log, "input_tokens") is not None+ and 200 <= log["status"] < 300+ ):+ key = log["credential_id"], log["protocol"], log["session"]+ conversations.setdefault(key, []).append(log)+ misses, unreported = [], set()+ for (_, protocol, _), calls in conversations.items():+ streak = miss_streak(calls)+ if len(streak) < MISS_STREAK:+ continue+ newest = streak[0]+ if newest["upstream_id"] not in hits:+ unreported.add(newest["upstream_id"])+ continue+ misses.append(+ {+ "upstream_id": newest["upstream_id"],+ "model": newest.get("model"),+ "streak": len(streak),+ "last_at": newest["time"],+ "input_tokens": min(first_upstream(call, "input_tokens") for call in streak),+ # Anthropic writes the cache whenever a request asks for it, so no write at all+ # means the client never asked (no cache_control) or the relay dropped the count.+ "no_writes": protocol == "anthropic"+ and not any(first_upstream(call, "cache_write_tokens") for call in streak),+ }+ )+ if not misses and not unreported:+ return None+ misses.sort(key=lambda miss: miss["last_at"], reverse=True)+ listed = ("upstream_id", "model", "streak", "last_at")+ return {+ # The page lists a few conversations but speaks for all of them.+ "misses": {+ "conversations": len(misses),+ "streak": min(miss["streak"] for miss in misses),+ "input_tokens": min(miss["input_tokens"] for miss in misses),+ "no_writes": all(miss["no_writes"] for miss in misses),+ "recent": [{k: miss[k] for k in listed} for miss in misses[:LISTED_MISSES]],+ }+ if misses+ else None,+ "unreported": sorted(unreported),+ }+++def overview_summary(logs, now):+ """Aggregate newest-first request-log records into the management overview."""+ requests = [log for log in logs if log.get("kind") != "capture"]+ groups = {}+ for label, kinds in (("first_call", {"user"}), ("continuation", {"continuation"})):+ subset = [log for log in requests if log.get("kind") in kinds]+ total = sum(first_upstream(log, "input_tokens", 0) for log in subset)+ cached = sum(first_upstream(log, "cached_tokens", 0) for log in subset)+ hidden_calls = 0+ if label == "continuation":+ total += sum(log.get("hidden_upstream_input_tokens", 0) for log in requests)+ cached += sum(log.get("hidden_upstream_cached_tokens", 0) for log in requests)+ hidden_calls = sum(log.get("hidden_upstream_calls", 0) for log in requests)+ groups[label] = {+ "requests": len(subset) + hidden_calls,+ "input_tokens": total,+ "cached_tokens": cached,+ "cache_hit_ratio": cached / total if total else 0,+ }+ degradations = {}+ for log in logs:+ if reason := log.get("degradation"):+ degradations[reason] = degradations.get(reason, 0) + 1+ # "disabled" means recall was skipped (off, budget spent or query too short).+ recalls = [log for log in requests if log.get("recall_reason", "disabled") != "disabled"]+ capture = {}+ for log in logs:+ if log.get("session") and log.get("capture_status"):+ capture.setdefault((log["session"], log.get("protocol")), log["capture_status"])+ return {+ "requests": len(requests),+ "last_request_at": requests[0].get("time") if requests else None,+ "output_tokens": sum(log.get("output_tokens", 0) for log in requests),+ "cache": groups,+ "degradations": degradations,+ "recall_count": sum(log.get("recall_count", 0) for log in requests),+ "recall_requests": len(recalls),+ "recall_ms": sum(log.get("recall_ms", 0) for log in recalls) / len(recalls)+ if recalls+ else 0,+ "capture_issues": {+ status: sum(1 for value in capture.values() if value == status)+ for status in ("retrying", "paused")+ },+ "cache_alerts": cache_alerts(requests, now),+ "sample_limit": 10000,+ }diff --git a/openviking_gateway/protocols.py b/openviking_gateway/protocols.pyindex e044b820d..42eb17219 100644--- a/openviking_gateway/protocols.py+++ b/openviking_gateway/protocols.py@@ -383,18 +383,39 @@ def replays_reasoning(upstream: dict) -> bool:return upstream.get("vendor") in REASONING_VENDORS if value is None else value-def usage_of(value: dict) -> dict:- """Token counts from any protocol's usage, including compatible endpoints that mix forms."""+# Fields relays put cache counts in, at the top of usage or in its token details.+CACHE_READ = ("cache_read_input_tokens", "cached_tokens", "prompt_cache_hit_tokens")+CACHE_WRITE = ("cache_creation_input_tokens", "cache_write_tokens", "cached_creation_tokens")+++def usage_of(value: dict, excludes_cache: bool = False) -> dict:+ """Token counts from any protocol's usage, including compatible endpoints that mix forms.++ ``input_tokens`` always includes cached and written tokens: Anthropic counts them apart+ (``excludes_cache``), Chat and Responses totals already hold them.+ """usage = value.get("usage") or {}- details = usage.get("prompt_tokens_details") or usage.get("input_tokens_details") or {}- cached = usage.get("cache_read_input_tokens", details.get("cached_tokens", 0)) or 0- written = usage.get("cache_creation_input_tokens", 0) or 0- input_tokens = usage.get("prompt_tokens", usage.get("input_tokens", 0)) or 0- if "cache_read_input_tokens" in usage:+ scopes = [+ usage,+ usage.get("prompt_tokens_details") or {},+ usage.get("input_tokens_details") or {},+ ]+ # Relays copy one count into several fields and zero-fill the rest.+ cached, written = (+ max((s[k] for s in scopes for k in keys if s.get(k) is not None), default=0)+ for keys in (CACHE_READ, CACHE_WRITE)+ )+ # Each protocol's own total first; compatible endpoints often fill both, one with zero.+ own, other = (+ ("input_tokens", "prompt_tokens") if excludes_cache else ("prompt_tokens", "input_tokens")+ )+ input_tokens = usage.get(own) or usage.get(other) or 0+ # A total below its own cache counted the cache apart, as older relays do too.+ if excludes_cache or input_tokens < cached + written:input_tokens += cached + writtenreturn {"input_tokens": input_tokens,- "output_tokens": usage.get("completion_tokens", usage.get("output_tokens", 0)) or 0,+ "output_tokens": usage.get("completion_tokens") or usage.get("output_tokens") or 0,"cached_tokens": cached,"cache_write_tokens": written,}diff --git a/openviking_gateway/tool_loop.py b/openviking_gateway/tool_loop.pyindex 67f90fc2a..e3a7edb47 100644--- a/openviking_gateway/tool_loop.py+++ b/openviking_gateway/tool_loop.py@@ -42,7 +42,7 @@ class HiddenToolLoop:seconds = self.policy.get("tool_total_seconds")self.deadline = time.monotonic() + seconds if prepared.tools_active and seconds else Noneself.adapter.lead_with(prepared.reply_lead)- self.transcript, self.usage = [], {}+ self.transcript, self.usage, self.counted = [], {}, {}self.final, self.hidden, self.window = None, False, Noneself.rounds, self.token_cost, self.refused = 0, 0, Falseself.round = ToolRound([], [], {}, False)@@ -95,7 +95,7 @@ class HiddenToolLoop:def observe(self):self.transcript.extend(self.round.output)add_usage(self.usage, self.round.usage)- normalized = usage_of({"usage": self.round.usage})+ normalized = usage_of({"usage": self.round.usage}, self.adapter.usage_excludes_cache)if normalized["input_tokens"]:# The latest round measures the window best; native tools read it here.self.capture.context_usage = normalized@@ -106,6 +106,9 @@ class HiddenToolLoop:prefix = "hidden_upstream_" if self.rounds else "first_upstream_"for key, value in normalized.items():self.prepared.metrics[prefix + key] = self.prepared.metrics.get(prefix + key, 0) + value+ if self.round.usage:+ # Rounds may report usage in different forms, so the log sums them read one by one.+ self.counted[key] = self.counted.get(key, 0) + valuedef stream(self, events):# Adapters update visible history either way; only streams send events.@@ -229,6 +232,7 @@ class HiddenToolLoop:# The combined reply reads like any relayed one: calls left for the client# hand the turn off, and the continuation that ends it is captured.self.capture.nonstream(self.final)+ self.capture.usage = self.counted or self.capture.usageself.prepared.metrics["hidden_rounds"] = self.roundsdef error(self, message: str) -> bytes:diff --git a/openviking_gateway/tool_protocols/anthropic.py b/openviking_gateway/tool_protocols/anthropic.pyindex dd7877a03..128e896df 100644--- a/openviking_gateway/tool_protocols/anthropic.py+++ b/openviking_gateway/tool_protocols/anthropic.py@@ -15,6 +15,7 @@ from .common import (ToolRound,append,call,+ merge_usage,strip_notices,)@@ -26,6 +27,7 @@ class AnthropicProtocol(ToolProtocol):summary_drops = ("stop_sequences", "output_config.format")reply_ends = frozenset({"end_turn", "stop_sequence", "tool_use"})signed_thinking = True+ usage_excludes_cache = True@staticmethoddef thinking_as_text(message):@@ -150,7 +152,7 @@ class AnthropicProtocol(ToolProtocol):kind = value.get("type")if kind == "message_start":self.envelope = copy.deepcopy(value["message"])- self.usage.update(self.envelope.get("usage") or {})+ merge_usage(self.usage, self.envelope.get("usage") or {})self.opened = Trueelif kind == "content_block_start":self.blocks[value["index"]] = copy.deepcopy(value["content_block"])@@ -174,7 +176,7 @@ class AnthropicProtocol(ToolProtocol):elif kind == "message_delta":self.finish = value.get("delta", {}).get("stop_reason") or self.finishself.envelope.update(value.get("delta", {}))- self.usage.update(value.get("usage") or {})+ merge_usage(self.usage, value.get("usage") or {})elif kind == "message_stop":self.stopped = Truediff --git a/openviking_gateway/tool_protocols/chat.py b/openviking_gateway/tool_protocols/chat.pyindex b13065792..1b4bb628d 100644--- a/openviking_gateway/tool_protocols/chat.py+++ b/openviking_gateway/tool_protocols/chat.py@@ -13,6 +13,7 @@ from .common import (ToolRound,append_text,merge_delta,+ merge_usage,sse,)@@ -108,7 +109,7 @@ class ChatProtocol(ToolProtocol):def accumulate(self, value):self.envelope.update({k: v for k, v in value.items() if k not in {"choices", "usage"}})if value.get("usage"):- self.usage.update(value["usage"])+ merge_usage(self.usage, value["usage"])# Only the first completion is the reply; the tool loop refuses any other.choice = next((c for c in value.get("choices") or [] if c.get("index", 0) == 0), None)if choice is None:diff --git a/openviking_gateway/tool_protocols/common.py b/openviking_gateway/tool_protocols/common.pyindex b18ad71d0..e49377aea 100644--- a/openviking_gateway/tool_protocols/common.py+++ b/openviking_gateway/tool_protocols/common.py@@ -73,15 +73,24 @@ def merge_delta(target, delta):def add_usage(total, usage):+ """Sum per-round usage; a null in one round never erases another's counts."""for key, value in usage.items():+ if value is None:+ continueif isinstance(value, dict):add_usage(total.setdefault(key, {}), value)- elif isinstance(value, (int, float)):+ # Flags such as OpenRouter's is_byok are kept, not summed.+ elif isinstance(value, (int, float)) and not isinstance(value, bool):total[key] = total.get(key, 0) + valueelse:total[key] = value+def merge_usage(total, usage):+ """Overlay a later usage event; nulls, and zeros over counts, never erase earlier ones."""+ total.update({k: v for k, v in usage.items() if v is not None and (v or not total.get(k))})++def without(value, path):"""A copy of ``value`` without the dotted ``path``; everything else is shared."""key, _, rest = path.partition(".")@@ -202,6 +211,8 @@ class ToolProtocol(ABC):# Thinking signatures cover the history before them, so history the gateway# cannot resend exactly must go upstream without its thinking.signed_thinking = False+ # Usage counts cached and written prompt tokens apart from the input total.+ usage_excludes_cache = Falsedef __init__(self, body: dict):self.body = body@@ -384,7 +395,9 @@ class ToolProtocol(ABC):return {"message": message,"output_items": items,- "usage": usage_of({"usage": self.usage}) if self.usage else None,+ "usage": (+ usage_of({"usage": self.usage}, self.usage_excludes_cache) if self.usage else None+ ),"response_id": self.envelope.get("id") or "","complete": ended and not handoff,"handoff": handoff,diff --git a/tests/gateway/test_native_tools.py b/tests/gateway/test_native_tools.pyindex 0f4b48a96..5334fb671 100644--- a/tests/gateway/test_native_tools.py+++ b/tests/gateway/test_native_tools.py@@ -1,6 +1,7 @@"""Native wire contracts: hidden calls, exact replay and one visible stream."""import copy+from types import SimpleNamespaceimport orjsonimport pytest@@ -8,7 +9,9 @@ from aiohttp import webfrom test_app import enable_toolsfrom openviking_gateway.protocols import SSEDecoder-from openviking_gateway.tool_protocols.common import sse+from openviking_gateway.tool_loop import HiddenToolLoop+from openviking_gateway.tool_protocols import ResponseCapture+from openviking_gateway.tool_protocols.common import ToolRound, ssePATHS = {"responses": "/v1/responses", "anthropic": "/v1/messages"}@@ -1392,3 +1395,31 @@ async def test_admin_disabling_anthropic_tools_preserves_history_and_logs(else:assert not logs[0].get("degradation")assert requests[-1]["messages"] == [requests[0]["messages"][0], *body["messages"][1:]]+++async def test_tool_loop_logs_rounds_read_one_by_one(setup_kernel, credential, policy):+ """Rounds in different usage forms still add up, and nulls never break the sum."""+ kernel, store, _, _ = setup_kernel+ policy.update(gateway_tools=True, recall=False)+ prepared = await kernel.prepare(+ request_body("chat", False), "chat", {}, credential, {"id": "u"}, policy+ )+ loop = HiddenToolLoop(+ prepared, SimpleNamespace(allowed={"openviking_search"}), store, ResponseCapture("chat")+ )+ usages = [+ {"prompt_tokens": 1000, "prompt_cache_hit_tokens": 800, "prompt_tokens_details": None},+ {"prompt_tokens": 1100, "prompt_tokens_details": {"cached_tokens": 900}},+ ]+ for number, usage in enumerate(usages):+ loop.round = ToolRound([], [], usage, False)+ loop.observe()+ loop.rounds = number + 1+ loop.adapter.begin()+ loop.adapter.load(native_response("chat", 1))+ loop.round = loop.adapter.end()+ await loop.persist()+ assert loop.capture.usage["input_tokens"] == 2100+ assert loop.capture.usage["cached_tokens"] == 1700+ assert prepared.metrics["first_upstream_cached_tokens"] == 800+ assert prepared.metrics["hidden_upstream_cached_tokens"] == 900diff --git a/tests/gateway/test_overview.py b/tests/gateway/test_overview.pynew file mode 100644index 000000000..f577b4de4--- /dev/null+++ b/tests/gateway/test_overview.py@@ -0,0 +1,138 @@+"""Overview aggregates and prompt-cache alerts read from request logs."""++import pytest++from openviking_gateway.overview import overview_summary++NOW = 1_000_000.0+++def call(minutes_ago, **fields):+ """A long, missed first call that finished `minutes_ago` before NOW."""+ return {+ "kind": "user",+ "session": "s1",+ "protocol": "anthropic",+ "credential_id": "k1",+ "upstream_id": "chat",+ "model": "m",+ "status": 200,+ "time": NOW - minutes_ago * 60,+ "duration_ms": 2000,+ "input_tokens": 40_000,+ "cached_tokens": 0,+ "cache_write_tokens": 40_000,+ **fields,+ }+++def alerts(*logs, hit=True):+ """Overview cache alerts; `hit` adds another conversation that hit the upstream."""+ logs = sorted(logs, key=lambda log: -log["time"])+ if hit:+ logs.append(call(600, session="warm", cached_tokens=30_000))+ return overview_summary(logs, NOW)["cache_alerts"]+++def streak(*overrides):+ """Calls a minute apart, newest first, each updated with its override."""+ return [call(i, **override) for i, override in enumerate(overrides)]+++def test_cache_alert_after_consecutive_long_misses():+ result = alerts(*streak({}, {"input_tokens": 39_000}, {"input_tokens": 38_000}))+ assert result == {+ "misses": {+ "conversations": 1,+ "streak": 3,+ "input_tokens": 38_000,+ "no_writes": False,+ "recent": [{"upstream_id": "chat", "model": "m", "streak": 3, "last_at": NOW}],+ },+ "unreported": [],+ }+ assert alerts(*streak({}, {})) is None+ assert overview_summary([], NOW)["cache_alerts"] is None+++def test_cache_alert_summary_speaks_for_every_conversation():+ logs = [call(i * 0.1, session=f"s{i}", input_tokens=40_000 + i) for i in range(25)]+ logs = [{**log, "time": log["time"] - j * 60} for log in logs for j in range(3)]+ logs[-1]["input_tokens"] = 33_000+ misses = alerts(*logs)["misses"]+ assert [miss["last_at"] for miss in misses["recent"]] == [NOW, NOW - 6, NOW - 12]+ assert (misses["conversations"], misses["streak"], misses["input_tokens"]) == (25, 3, 33_000)+++def test_cache_alert_suggests_cache_control_only_without_any_write():+ silent = {"cache_write_tokens": 0}+ assert alerts(*streak(silent, silent, silent))["misses"]["no_writes"] is True+ assert alerts(*streak(silent, silent, {}))["misses"]["no_writes"] is False+ chat = {**silent, "protocol": "chat"}+ assert alerts(*streak(chat, chat, chat))["misses"]["no_writes"] is False+++ "older",+ [+ {"cached_tokens": 30_000},+ {"input_tokens": 20_000},+ {"window_reset": True},+ {"compaction_tokens": 9000},+ {"degradation": "upstream_changed"},+ {"degradation": "ark_cache_parameters_changed"},+ {"model": "other"},+ {"upstream_id": "other"},+ {"tools_tokens": 500},+ {"tool_skip_reason": "tool_name_collision"},+ {"input_tokens": 60_000}, # Larger than the newer call: a /clear or a branch since.+ ],+)+def test_cache_alert_breaks(older):+ assert alerts(*streak({}, {}, {}, older))["misses"]["streak"] == 3+ assert alerts(*streak({}, older, {}, {})) is None+++def test_cache_alert_newest_call_decides():+ assert alerts(*streak({"cached_tokens": 30_000}, {}, {}, {})) is None+ assert alerts(*streak({"input_tokens": 20_000}, {}, {}, {})) is None+++def test_cache_alert_expired_cache_breaks():+ assert alerts(call(0), call(1), call(2), call(8))["misses"]["streak"] == 3+ assert alerts(call(0), call(6), call(7), call(8)) is None+++def test_cache_alert_ignores_other_requests_and_concurrent_calls():+ logs = streak({}, {}, {})+ logs += [+ call(0.5, kind="subagent", model="small", input_tokens=1000),+ call(1.5, kind="auxiliary", input_tokens=500, cached_tokens=400),+ call(1.2, status=500),+ call(0.2, session=None),+ call(1.1, duration_ms=60_000, cached_tokens=30_000), # Overlaps the newer call.+ ]+ assert alerts(*logs)["misses"]["streak"] == 3+++def test_cache_alert_ignores_stale_conversations():+ assert alerts(*[call(1500 + i) for i in range(3)]) is None+++def test_cache_alert_prefers_first_upstream_usage():+ first = {"first_upstream_input_tokens": 40_000, "first_upstream_cached_tokens": 0}+ first |= {"first_upstream_cache_write_tokens": 0}+ hidden = {"input_tokens": 90_000, "cached_tokens": 80_000}+ misses = alerts(*streak(*[{**hidden, **first}] * 3))["misses"]+ assert misses["input_tokens"] == 40_000 and misses["no_writes"] is True+ hit = {**first, "first_upstream_cached_tokens": 30_000}+ assert alerts(*streak(hit, *[{**hidden, **first}] * 2)) is None+++def test_cache_alert_upstreams_without_hits_are_unreported():+ other = {"session": "s2", "upstream_id": "silent"}+ result = alerts(*streak({}, {}, {}), *streak(other, other, other), hit=False)+ assert result == {"misses": None, "unreported": ["chat", "silent"]}+ # Only recent hits count: a relay that started dropping cache fields shows up today.+ old_hit = call(3 * 24 * 60, session="old", cached_tokens=30_000)+ assert alerts(*streak({}, {}, {}), old_hit, hit=False)["unreported"] == ["chat"]diff --git a/tests/gateway/test_usage.py b/tests/gateway/test_usage.pynew file mode 100644index 000000000..ef3cb026c--- /dev/null+++ b/tests/gateway/test_usage.py@@ -0,0 +1,199 @@+import pytest++from openviking_gateway.protocols import usage_of+from openviking_gateway.tool_protocols import ResponseCapture+from openviking_gateway.tool_protocols.common import add_usage+++ ("usage", "anthropic", "expected"),+ [+ # Each protocol's own form.+ (+ {+ "input_tokens": 100,+ "cache_read_input_tokens": 800,+ "cache_creation_input_tokens": 50,+ },+ True,+ (950, 800, 50),+ ),+ (+ {"prompt_tokens": 950, "prompt_tokens_details": {"cached_tokens": 800}},+ False,+ (950, 800, 0),+ ),+ (+ {"input_tokens": 950, "input_tokens_details": {"cached_tokens": 800}},+ False,+ (950, 800, 0),+ ),+ # A zero-filled Anthropic field no longer hides the count a relay put in details.+ (+ {+ "prompt_tokens": 950,+ "prompt_tokens_details": {"cached_tokens": 800},+ "cache_read_input_tokens": 0,+ "cache_creation_input_tokens": 0,+ },+ False,+ (950, 800, 0),+ ),+ # LiteLLM: Chat totals already hold both cache fields it adds at the top.+ (+ {+ "prompt_tokens": 1000,+ "prompt_tokens_details": {"cached_tokens": 800},+ "cache_read_input_tokens": 800,+ "cache_creation_input_tokens": 50,+ },+ False,+ (1000, 800, 50),+ ),+ ({"input_tokens": 950, "cache_read_input_tokens": 800}, False, (950, 800, 0)),+ # Anthropic-compatible endpoints that also send a Chat total.+ (+ {"prompt_tokens": 0, "input_tokens": 100, "cache_read_input_tokens": 800},+ True,+ (900, 800, 0),+ ),+ (+ {"input_tokens": 100, "cache_read_input_tokens": None, "cached_tokens": 800},+ True,+ (900, 800, 0),+ ),+ # DeepSeek, Kimi, DashScope and OpenRouter names.+ (+ {"prompt_tokens": 950, "prompt_cache_hit_tokens": 800, "prompt_cache_miss_tokens": 150},+ False,+ (950, 800, 0),+ ),+ (+ {+ "prompt_tokens": 19,+ "cached_tokens": 10,+ "prompt_tokens_details": {"cached_tokens": 10},+ },+ False,+ (19, 10, 0),+ ),+ (+ {+ "prompt_tokens": 1000,+ "prompt_tokens_details": {"cached_tokens": 600, "cache_creation_input_tokens": 300},+ },+ False,+ (1000, 600, 300),+ ),+ (+ {+ "prompt_tokens": 1000,+ "prompt_tokens_details": {"cached_tokens": 600, "cache_write_tokens": 300},+ },+ False,+ (1000, 600, 300),+ ),+ # new-api fills both totals, one with zero.+ (+ {+ "prompt_tokens": 0,+ "input_tokens": 950,+ "input_tokens_details": {"cached_tokens": 800},+ },+ False,+ (950, 800, 0),+ ),+ (+ {+ "prompt_tokens": 950,+ "input_tokens": 0,+ "prompt_tokens_details": {"cached_tokens": 800},+ "input_tokens_details": None,+ },+ False,+ (950, 800, 0),+ ),+ # Older new-api: a Chat total without the cache, which is added back.+ (+ {"prompt_tokens": 10, "prompt_tokens_details": {"cached_tokens": 800}},+ False,+ (810, 800, 0),+ ),+ # No cache field at all.+ ({"input_tokens": 100, "cache_read_input_tokens": None}, True, (100, 0, 0)),+ ({"prompt_tokens": 950, "prompt_tokens_details": {}}, False, (950, 0, 0)),+ ],+)+def test_usage_of_reads_every_upstream_form(usage, anthropic, expected):+ result = usage_of({"usage": usage}, anthropic)+ assert (+ result["input_tokens"],+ result["cached_tokens"],+ result["cache_write_tokens"],+ ) == expected+++def test_add_usage_skips_nulls_and_keeps_flags():+ total = {}+ for usage in (+ {"cache_read_input_tokens": None, "prompt_tokens_details": None, "is_byok": False},+ {"cache_read_input_tokens": 5, "prompt_tokens_details": {"cached_tokens": 80}},+ {"cache_read_input_tokens": None, "prompt_tokens_details": None, "is_byok": False},+ ):+ add_usage(total, usage)+ assert total == {+ "cache_read_input_tokens": 5,+ "prompt_tokens_details": {"cached_tokens": 80},+ "is_byok": False,+ }++[email protected]("late", [None, 0])+def test_anthropic_delta_keeps_start_cache(late):+ capture = ResponseCapture("anthropic")+ start = {"input_tokens": 12, "cache_read_input_tokens": 88}+ capture.event({"type": "message_start", "message": {"id": "x", "usage": start}})+ capture.event(+ {+ "type": "message_delta",+ "delta": {"stop_reason": "end_turn"},+ "usage": {"output_tokens": 20, "input_tokens": late, "cache_read_input_tokens": late},+ }+ )+ assert capture.usage["input_tokens"] == 100+ assert capture.usage["cached_tokens"] == 88+++def test_anthropic_delta_carries_relayed_cache():+ """Relays converting other protocols send zeros at the start and counts at the end."""+ capture = ResponseCapture("anthropic")+ start = {"input_tokens": 12, "cache_read_input_tokens": 0, "cache_creation_input_tokens": 0}+ capture.event({"type": "message_start", "message": {"id": "x", "usage": start}})+ capture.event(+ {+ "type": "message_delta",+ "delta": {"stop_reason": "end_turn"},+ "usage": {"output_tokens": 20, "input_tokens": 12, "cache_read_input_tokens": 88},+ }+ )+ assert capture.usage["input_tokens"] == 100+ assert capture.usage["cached_tokens"] == 88+++def test_chat_stream_null_details_keep_cache():+ capture = ResponseCapture("chat")+ capture.event(+ {+ "id": "x",+ "choices": [],+ "usage": {"prompt_tokens": 950, "prompt_tokens_details": {"cached_tokens": 800}},+ }+ )+ capture.event(+ {+ "id": "x",+ "choices": [],+ "usage": {"prompt_tokens": 950, "completion_tokens": 20, "prompt_tokens_details": None},+ }+ )+ assert capture.usage["cached_tokens"] == 800diff --git a/web-studio/src/i18n/locales/en/gateway/overview.ts b/web-studio/src/i18n/locales/en/gateway/overview.tsindex a7c1ca3a7..bf7fd8215 100644--- a/web-studio/src/i18n/locales/en/gateway/overview.ts+++ b/web-studio/src/i18n/locales/en/gateway/overview.ts@@ -114,6 +114,30 @@ const overview = {none: 'No recalls yet',},},+ cacheDocs: 'Troubleshooting',+ cacheMiss: {+ title_one: '{{count}} long conversation keeps missing the prompt cache',+ title_other: '{{count}} long conversations keep missing the prompt cache',+ description_one:+ 'Its last {{streak}} model calls each sent at least {{tokens}} input tokens and got a 0% cache hit rate. This usually means the start of the request (system prompt or tool definitions) changes every time, or the upstream spreads requests across different accounts or channels.',+ description_other:+ 'Their last {{streak}} model calls each sent at least {{tokens}} input tokens and got a 0% cache hit rate. This usually means the start of the request (system prompt or tool definitions) changes every time, or the upstream spreads requests across different accounts or channels.',+ action:+ "Compare with your provider's cache records. If they show hits, the upstream isn't passing cache usage back to the gateway. If not, check the system prompt for timestamps or other content that changes on every request, and make sure your relay isn't rotating between channels.",+ noWrites:+ "The upstream didn't write the cache either, so the client most likely doesn't set cache_control. Compare with your provider's cache records: hits there mean the upstream isn't passing cache usage back to the gateway.",+ streak_one: '{{count}} miss in a row',+ streak_other: '{{count}} misses in a row',+ },+ cacheUnreported: {+ title_one: "{{upstreams}} hasn't reported any cache hits lately",+ title_other: "{{upstreams}} haven't reported any cache hits lately",+ description_one:+ 'Long conversations through it keep showing a 0% cache hit rate, and they count as misses in the stats. The provider may not support prompt caching, the client may not enable it, or a relay may not pass cache usage back to the gateway.',+ description_other:+ 'Long conversations through them keep showing a 0% cache hit rate, and they count as misses in the stats. The providers may not support prompt caching, the client may not enable it, or a relay may not pass cache usage back to the gateway.',+ separator: ', ',+ },openviking: {title: 'OpenViking',description:diff --git a/web-studio/src/i18n/locales/zh-CN/gateway/overview.ts b/web-studio/src/i18n/locales/zh-CN/gateway/overview.tsindex ed40dd84b..5ca8c7516 100644--- a/web-studio/src/i18n/locales/zh-CN/gateway/overview.ts+++ b/web-studio/src/i18n/locales/zh-CN/gateway/overview.ts@@ -108,6 +108,30 @@ const overview = {none: '还没有召回',},},+ cacheDocs: '排查说明',+ cacheMiss: {+ title_one: '有 {{count}} 段长对话连续没有命中提示词缓存',+ title_other: '有 {{count}} 段长对话连续没有命中提示词缓存',+ description_one:+ '最近连续 {{streak}} 次模型调用的输入都不少于 {{tokens}} Token,缓存命中率都是 0%。常见原因是请求开头的内容(系统提示词、工具定义)每次都在变,或者上游把请求分给了不同的账号或渠道。',+ description_other:+ '最近连续 {{streak}} 次模型调用的输入都不少于 {{tokens}} Token,缓存命中率都是 0%。常见原因是请求开头的内容(系统提示词、工具定义)每次都在变,或者上游把请求分给了不同的账号或渠道。',+ action:+ '对照服务商后台的缓存记录:那边有命中,说明上游没有把缓存用量传回网关;那边也没有,检查系统提示词里有没有时间这类每次都变的内容,以及中转服务是否在多个渠道之间轮换。',+ noWrites:+ '上游也没有写入缓存,客户端很可能没有在请求里设置 cache_control。对照服务商后台的缓存记录:那边有命中,说明上游没有把缓存用量传回网关。',+ streak_one: '连续 {{count}} 次未命中',+ streak_other: '连续 {{count}} 次未命中',+ },+ cacheUnreported: {+ title_one: '上游 {{upstreams}} 最近没有返回过缓存命中',+ title_other: '上游 {{upstreams}} 最近没有返回过缓存命中',+ description_one:+ '相关长对话的缓存命中率一直是 0%,统计时都按未命中计算。可能是服务商不支持提示词缓存、客户端没有开启缓存,或者中转服务没有把缓存用量传回网关。',+ description_other:+ '相关长对话的缓存命中率一直是 0%,统计时都按未命中计算。可能是服务商不支持提示词缓存、客户端没有开启缓存,或者中转服务没有把缓存用量传回网关。',+ separator: '、',+ },openviking: {title: 'OpenViking',description: '为新消息检索记忆,并保存对话。',diff --git a/web-studio/src/routes/gateway/-components/overview-cache.tsx b/web-studio/src/routes/gateway/-components/overview-cache.tsxnew file mode 100644index 000000000..52f100713--- /dev/null+++ b/web-studio/src/routes/gateway/-components/overview-cache.tsx@@ -0,0 +1,146 @@+import { ExternalLinkIcon, LightbulbIcon } from 'lucide-react'+import { useTranslation } from 'react-i18next'++import { Button } from '#/components/ui/button'++import type { CacheAlerts, CacheMiss, Upstream } from '../-lib/api'+import {+ formatCompact,+ formatDateTime,+ formatRelativeTime,+ shortenId,+} from '../-lib/format'+import { gatewayDocsUrl } from '../-lib/client-guides'+import { useUpstreams } from '../-lib/use-gateway'+import { Notice } from './notice'++/** Upstream name, or its shortened id once deleted or not yet loaded. */+function upstreamName(id: string, upstreams?: Upstream[]): string {+ return (+ upstreams?.find((upstream) => upstream.id === id)?.name ?? shortenId(id)+ )+}++/** One conversation: model · upstream · streak · last call. */+function CacheMissRow({+ miss,+ upstreams,+}: {+ miss: CacheMiss+ upstreams?: Upstream[]+}) {+ const { t, i18n } = useTranslation('gateway')+ const locale = i18n.resolvedLanguage+ return (+ <li className="flex min-w-0 flex-wrap gap-x-1.5 text-xs leading-5 text-foreground/80">+ {miss.model ? (+ <>+ <span className="truncate font-mono">{miss.model}</span>+ <span aria-hidden>·</span>+ </>+ ) : null}+ <span title={miss.upstream_id}>+ {upstreamName(miss.upstream_id, upstreams)}+ </span>+ <span aria-hidden>·</span>+ <span className="tabular-nums">+ {t('overview.cacheMiss.streak', { count: miss.streak })}+ </span>+ <span aria-hidden>·</span>+ <span title={formatDateTime(miss.last_at, locale, true)}>+ {formatRelativeTime(miss.last_at, locale)}+ </span>+ </li>+ )+}++/**+ * Long conversations that keep missing the prompt cache, and upstreams that+ * never report a hit; upstream names load only while there is one to show.+ */+export function CacheNotices({ alerts }: { alerts: CacheAlerts }) {+ const { t, i18n } = useTranslation('gateway')+ const locale = i18n.resolvedLanguage+ const upstreams = useUpstreams().data+ const { misses, unreported } = alerts+ const docs = (+ <Button+ size="sm"+ variant="outline"+ nativeButton={false}+ render={+ <a+ href={gatewayDocsUrl('operations', locale, 'cacheHitRate')}+ target="_blank"+ rel="noreferrer"+ />+ }+ >+ <ExternalLinkIcon />+ {t('overview.cacheDocs')}+ </Button>+ )+ return (+ <div className="grid gap-3">+ {misses ? (+ <Notice+ tone="warning"+ title={t('overview.cacheMiss.title', { count: misses.conversations })}+ action={docs}+ >+ <p className="text-foreground/80">+ {t('overview.cacheMiss.description', {+ count: misses.conversations,+ streak: misses.streak,+ // Rounded down, so "at least" holds for every conversation.+ tokens: formatCompact(+ Math.floor(misses.input_tokens / 1000) * 1000,+ locale,+ ),+ })}+ </p>+ <p className="flex gap-1.5 text-xs leading-5 text-foreground/80">+ <LightbulbIcon className="mt-0.5 size-3.5 shrink-0" />+ <span>+ <span className="sr-only">+ {t('overview.degraded.suggestion')}+ </span>+ {t(+ misses.no_writes+ ? 'overview.cacheMiss.noWrites'+ : 'overview.cacheMiss.action',+ )}+ </span>+ </p>+ <ul className="mt-1 grid gap-0.5">+ {misses.recent.map((miss, index) => (+ <CacheMissRow+ key={`${miss.upstream_id}-${miss.last_at}-${index}`}+ miss={miss}+ upstreams={upstreams}+ />+ ))}+ </ul>+ </Notice>+ ) : null}+ {unreported.length ? (+ <Notice+ tone="info"+ title={t('overview.cacheUnreported.title', {+ count: unreported.length,+ upstreams: unreported+ .map((id) => upstreamName(id, upstreams))+ .join(t('overview.cacheUnreported.separator')),+ })}+ action={docs}+ >+ <p className="text-foreground/80">+ {t('overview.cacheUnreported.description', {+ count: unreported.length,+ })}+ </p>+ </Notice>+ ) : null}+ </div>+ )+}diff --git a/web-studio/src/routes/gateway/-components/overview-page.test.tsx b/web-studio/src/routes/gateway/-components/overview-page.test.tsxindex 03adefcc1..fcb7cc14d 100644--- a/web-studio/src/routes/gateway/-components/overview-page.test.tsx+++ b/web-studio/src/routes/gateway/-components/overview-page.test.tsx@@ -32,8 +32,11 @@ import {import i18n from '#/i18n'import { GatewayError } from '../-lib/api'+import { gatewayDocsUrl } from '../-lib/client-guides'import type * as Api from '../-lib/api'import type {+ CacheAlerts,+ CacheMiss,GatewayAccess,IssuedKey,LogRecord,@@ -112,6 +115,7 @@ const EMPTY_OVERVIEW: Overview = {capture_issues: { retrying: 0, paused: 0 },sample_limit: 10000,log_retention_days: 30,+ cache_alerts: null,}const BUSY_OVERVIEW: Overview = {@@ -140,6 +144,7 @@ const BUSY_OVERVIEW: Overview = {capture_issues: { retrying: 1, paused: 2 },sample_limit: 10000,log_retention_days: 7,+ cache_alerts: null,}const LOGS: LogRecord[] = [@@ -171,6 +176,34 @@ const LOGS: LogRecord[] = [},]+const UPSTREAMS = [+ { id: 'u1', name: 'OpenAI production' },+ { id: 'u2', name: 'Kimi relay' },+] as Upstream[]++function cacheMiss(fields: Partial<CacheMiss>): CacheMiss {+ return {+ upstream_id: 'u1',+ model: 'gpt-5',+ streak: 5,+ last_at: NOW - 60,+ ...fields,+ }+}++function cacheMisses(+ fields: Partial<NonNullable<CacheAlerts['misses']>>,+): CacheAlerts['misses'] {+ return {+ conversations: 1,+ streak: 5,+ input_tokens: 50000,+ no_writes: false,+ recent: [cacheMiss({})],+ ...fields,+ }+}+/** Pages the overview links to; links only get an href for known routes. */const LINK_TARGETS = ['/gateway/upstreams',@@ -372,6 +405,8 @@ describe('OverviewPage', () => {expect(screen.queryByText('Memory sync')).toBeNull()expect(screen.queryByText('Paused')).toBeNull()expect(screen.getByText('502')).toBeTruthy()+ // No cache alerts, so no notice and no upstream list to load.+ expect(screen.queryByText(/prompt cache|cache hits lately/)).toBeNull()expect(hrefOf('View all')).toBe('/gateway/requests')expect(screen.getByText(@@ -380,6 +415,103 @@ describe('OverviewPage', () => {).toBeTruthy()})+ it('warns about long conversations that keep missing the prompt cache', async () => {+ api.getOverview.mockResolvedValue({+ ...BUSY_OVERVIEW,+ cache_alerts: {+ misses: cacheMisses({+ conversations: 25,+ streak: 3,+ input_tokens: 33950,+ recent: [+ cacheMiss({ model: 'claude-sonnet-4-5', streak: 7 }),+ cacheMiss({ upstream_id: 'deadbeefcafe0001', streak: 5 }),+ cacheMiss({ model: 'kimi-k2', streak: 9, last_at: NOW - 7200 }),+ ],+ }),+ unreported: [],+ },+ })+ api.listUpstreams.mockResolvedValue(UPSTREAMS)+ renderOverview()++ expect(+ await screen.findByText(+ '25 long conversations keep missing the prompt cache',+ ),+ ).toBeTruthy()+ // The summary speaks for every conversation, listed or not, rounded down.+ expect(+ screen.getByText(+ /^Their last 3 model calls each sent at least 33K input tokens and got a 0% cache hit rate\./,+ ),+ ).toBeTruthy()+ expect(+ screen.getByText(/^Compare with your provider's cache records\./),+ ).toBeTruthy()+ // The newest conversations, upstreams by name.+ expect(screen.getByText('7 misses in a row')).toBeTruthy()+ expect(screen.getByText('5 misses in a row')).toBeTruthy()+ expect(screen.getByText('9 misses in a row')).toBeTruthy()+ expect(screen.getAllByText('claude-sonnet-4-5').length).toBeGreaterThan(0)+ expect(screen.getByText('2 hours ago')).toBeTruthy()+ expect(await screen.findAllByText('OpenAI production')).toHaveLength(2)+ expect(screen.getByText('deadbeef…').getAttribute('title')).toBe(+ 'deadbeefcafe0001',+ )+ expect(screen.queryByText(/cache hits lately/)).toBeNull()+ expect(+ screen.getByText('Troubleshooting').closest('a')?.getAttribute('href'),+ ).toBe(gatewayDocsUrl('operations', 'en', 'cacheHitRate'))+ })++ it('points Anthropic conversations that never write the cache at cache_control', async () => {+ api.getOverview.mockResolvedValue({+ ...BUSY_OVERVIEW,+ cache_alerts: {+ misses: cacheMisses({ streak: 3, no_writes: true }),+ unreported: [],+ },+ })+ api.listUpstreams.mockResolvedValue(UPSTREAMS)+ renderOverview()++ expect(+ await screen.findByText(+ '1 long conversation keeps missing the prompt cache',+ ),+ ).toBeTruthy()+ expect(+ screen.getByText(/^Its last 3 model calls each sent at least 50K/),+ ).toBeTruthy()+ expect(+ screen.getByText(/the client most likely doesn't set cache_control/),+ ).toBeTruthy()+ expect(screen.queryByText(/^Compare with your provider's/)).toBeNull()+ })++ it('lists every upstream that reports no cache hits in one note', async () => {+ api.getOverview.mockResolvedValue({+ ...BUSY_OVERVIEW,+ cache_alerts: {+ misses: null,+ unreported: ['deadbeefcafe0001', 'u1', 'u2'],+ },+ })+ api.listUpstreams.mockResolvedValue(UPSTREAMS)+ renderOverview()++ expect(+ await screen.findByText(+ "deadbeef…, OpenAI production, Kimi relay haven't reported any cache hits lately",+ ),+ ).toBeTruthy()+ expect(+ screen.getAllByText(/^Long conversations through them keep showing/),+ ).toHaveLength(1)+ expect(screen.queryByText(/prompt cache$/)).toBeNull()+ })+it('shows the error with a retry when the overview fails', async () => {api.getOverview.mockRejectedValue(new GatewayError('Network Error'))renderOverview()@@ -411,11 +543,28 @@ describe('OverviewPage', () => {expect(api.saveProfile.mock.calls[0][2].name).toBe('默认')cleanup()- api.getOverview.mockResolvedValue(BUSY_OVERVIEW)+ api.getOverview.mockResolvedValue({+ ...BUSY_OVERVIEW,+ cache_alerts: {+ misses: cacheMisses({+ streak: 6,+ recent: [cacheMiss({ model: 'kimi-k2', streak: 6 })],+ }),+ unreported: ['u1'],+ },+ })api.listLogs.mockResolvedValue(LOGS)+ api.listUpstreams.mockResolvedValue(UPSTREAMS)const busy = renderOverview()expect(await screen.findByText('claude-sonnet-4-5')).toBeTruthy()expect(screen.getByText('1,234 条新消息')).toBeTruthy()+ expect(screen.getByText('有 1 段长对话连续没有命中提示词缓存')).toBeTruthy()+ expect(screen.getByText('连续 6 次未命中')).toBeTruthy()+ expect(+ await screen.findByText('上游 OpenAI production 最近没有返回过缓存命中'),+ ).toBeTruthy()+ // Chinese has no singular form, so the copy reads right for any count.+ expect(screen.getByText(/^相关长对话的缓存命中率一直是 0%/)).toBeTruthy()expect(busy.container.textContent).not.toMatch(/overview\.|enums\.|states\.|actions\./,)diff --git a/web-studio/src/routes/gateway/-components/overview-page.tsx b/web-studio/src/routes/gateway/-components/overview-page.tsxindex 3384c007f..0d2c5b685 100644--- a/web-studio/src/routes/gateway/-components/overview-page.tsx+++ b/web-studio/src/routes/gateway/-components/overview-page.tsx@@ -30,6 +30,7 @@ import {} from '../-lib/use-gateway'import { ErrorState, LoadingState } from './empty-state'import { MetricCard } from './metric-card'+import { CacheNotices } from './overview-cache'import { RecentRequests } from './overview-recent'import { OverviewSetup } from './overview-setup'import { DegradedRequestsCard, OpenVikingCard } from './overview-status'@@ -175,6 +176,9 @@ export function OverviewPage() {<>{data.last_request_at == null ? <OverviewSetup /> : null}<MetricRow data={data} />+ {data.cache_alerts ? (+ <CacheNotices alerts={data.cache_alerts} />+ ) : null}<div className="grid gap-3 lg:grid-cols-2"><OpenVikingCardhealth={data.openviking}diff --git a/web-studio/src/routes/gateway/-lib/api.ts b/web-studio/src/routes/gateway/-lib/api.tsindex 5fda09912..7852b479d 100644--- a/web-studio/src/routes/gateway/-lib/api.ts+++ b/web-studio/src/routes/gateway/-lib/api.ts@@ -207,6 +207,32 @@ export type OpenVikingHealth = {reason?: string}+/** A conversation whose latest long model calls all missed the prompt cache. */+export type CacheMiss = {+ upstream_id: string+ model: string+ /** Long calls in a row without a cache hit. */+ streak: number+ /** Epoch seconds of the newest call. */+ last_at: number+}++export type CacheAlerts = {+ /** Conversations missing the cache; null without any. */+ misses: {+ conversations: number+ /** Shortest streak and smallest first-call input among them. */+ streak: number+ input_tokens: number+ /** All are Anthropic conversations that never wrote the cache either. */+ no_writes: boolean+ /** The newest three conversations. */+ recent: CacheMiss[]+ } | null+ /** Ids of upstreams whose long calls have not hit the cache lately, sorted. */+ unreported: string[]+}+export type Overview = {/** Model requests in the sample (memory-sync events excluded). */requests: number@@ -226,6 +252,8 @@ export type Overview = {sample_limit: number/** Days the request log keeps records. */log_retention_days: number+ /** Conversations and upstreams missing the prompt cache; null without any. */+ cache_alerts: CacheAlerts | null}/** One request-log record (metadata only); every field but `time` is optional. */diff --git a/web-studio/src/routes/gateway/-lib/client-guides.test.ts b/web-studio/src/routes/gateway/-lib/client-guides.test.tsindex ef407fdab..6c102d53d 100644--- a/web-studio/src/routes/gateway/-lib/client-guides.test.ts+++ b/web-studio/src/routes/gateway/-lib/client-guides.test.ts@@ -182,6 +182,9 @@ it('links the docs in the UI language', () => {expect(gatewayDocsUrl('operations', 'zh-CN')).toBe('https://docs.openviking.ai/zh/guides/22-gateway-operations',)+ expect(gatewayDocsUrl('operations', 'zh-CN', 'cacheHitRate')).toBe(+ 'https://docs.openviking.ai/zh/guides/22-gateway-operations#首次调用缓存命中率下降',+ )})describe('protocols', () => {diff --git a/web-studio/src/routes/gateway/-lib/client-guides.ts b/web-studio/src/routes/gateway/-lib/client-guides.tsindex a8d725f0c..6edb04336 100644--- a/web-studio/src/routes/gateway/-lib/client-guides.ts+++ b/web-studio/src/routes/gateway/-lib/client-guides.ts@@ -357,12 +357,22 @@ export const SESSION_HEADERS = ['session-id',]+/** Headings Studio links to, by each docs language's anchor. */+const DOCS_SECTIONS = {+ cacheHitRate: {+ en: 'the-first-call-cache-hit-rate-dropped',+ zh: '首次调用缓存命中率下降',+ },+}+/** OpenViking docs page for the gateway, in the UI language. */export function gatewayDocsUrl(page: 'guide' | 'operations',language?: string,+ section?: keyof typeof DOCS_SECTIONS,): string {const locale = language?.startsWith('zh') ? 'zh' : 'en'const slug = page === 'guide' ? '15-gateway' : '22-gateway-operations'- return `https://docs.openviking.ai/${locale}/guides/${slug}`+ const anchor = section ? `#${DOCS_SECTIONS[section][locale]}` : ''+ return `https://docs.openviking.ai/${locale}/guides/${slug}${anchor}`}