Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

fix(privacy): pause automatic session secret discovery (#6680)

MaojiaSheng3f85145b71a8
diff --git a/docs/en/concepts/13-privacy.md b/docs/en/concepts/13-privacy.md
index 967879161..15b495183 100644
--- a/docs/en/concepts/13-privacy.md
+++ b/docs/en/concepts/13-privacy.md
@@ -60,7 +60,7 @@ viking://user/{user_id}/privacy/{category}/{target_key}/
## Skill Privacy Extraction and Restore
-### Automatic Extraction (Write Path)
+### Registered-Value Protection (Write Path)
When adding a Skill via `add_skill`, sensitive values are extracted and replaced with placeholders.
@@ -135,26 +135,25 @@ Placeholders that reference the current user's Privacy fields are also restored.
### Automatic Extraction (Write Path)
-When writing messages to a Session, common credential rules extract sensitive values and replace them with placeholders before tool output externalization and message persistence.
+When writing messages to a Session, values matching registered Privacy entries are replaced with placeholders before tool output externalization and message persistence. Automatic discovery of new sensitive values is temporarily disabled in code because false positives create excessive invalid entries. It will be revisited later; there is currently no runtime configuration switch.
```
Session.add_messages_async
-> extract_and_store_session_privacy
-> PrivacySanitizer (protect registered personal credentials)
- -> PrivacyExtractor (extract new values using rules and replace them with placeholders)
- -> privacy.insert_values(category="session", target_key=session_id, updates=...)
+ -> skip new-value discovery; do not add Session Privacy fields
-> externalize tool output
-> write placeholderized messages
```
#### Key Points
-1. **Extraction source**: fixed rules identify sensitive values without model calls; the results are used as privacy key-value pairs.
-2. **Coverage**: TextPart text, ContextPart abstracts, and ToolPart input and output. Values under sensitive tool input keys are stored and masked as a whole.
+1. **Matching source**: only registered Privacy values are matched; fixed rules and sensitive field names no longer trigger new-value extraction.
+2. **Coverage**: TextPart text, ContextPart abstracts, and ToolPart input and output. Tool inputs are traversed recursively to match registered values; field names no longer cause entire values to be stored and masked.
3. **Content replacement**: sensitive spans in the original content are replaced with placeholders:
- Sensitive values belonging to the current Session use the same placeholder format, with the Session ID or its hash value: <code>&#123;&#123;ov_privacy:session:&#123;session_id&#125;:&#123;field_name&#125;&#125;&#125;</code>.
- If a sensitive span matches registered personal credentials (`credential`), prefer the uniquely matching field's placeholder, for example <code>&#123;&#123;ov_privacy:alice:viking://~/privacy/credential/github#/values/pat&#125;&#125;</code>.
-4. **Persisted content**: matched spans are saved as placeholders; unmatched content remains unchanged. Newly extracted privacy values are saved in the current Session's config.
+4. **Persisted content**: matched spans are saved as placeholders; unmatched content remains unchanged. No new Session Privacy data is created. Existing configs and restoration of historical placeholders remain supported.
---
diff --git a/docs/zh/concepts/13-privacy.md b/docs/zh/concepts/13-privacy.md
index b5dca8e7d..523c41ab3 100644
--- a/docs/zh/concepts/13-privacy.md
+++ b/docs/zh/concepts/13-privacy.md
@@ -60,7 +60,7 @@ viking://user/{user_id}/privacy/{category}/{target_key}/
## 技能的隐私提取与读取还原
-### 自动提取(写入链路)
+### 已登记值保护(写入链路)
当通过 `add_skill` 写入 Skill 时,会走隐私提取与占位符化流程。
@@ -135,26 +135,25 @@ fs.read(uri)
### 自动提取(写入链路)
-向 Session 写入消息时,会在工具输出外置和消息落盘前,通过常见的规则,提取敏感值并替换为占位符。
+向 Session 写入消息时,会在工具输出外置和消息落盘前,将匹配已登记 Privacy 值的内容替换为占位符。新敏感值的自动探测因误报产生大量无效数据,已在代码中暂时禁用,后续再处理;目前没有运行时配置开关。
```
Session.add_messages_async
-> extract_and_store_session_privacy
-> PrivacySanitizer(保护已登记的个人密钥)
- -> PrivacyExtractor(规则提取新值并占位符化)
- -> privacy.insert_values(category="session", target_key=session_id, updates=...)
+ -> 跳过新值探测,不新增 Session Privacy 字段
-> 工具输出外置
-> 写入占位符化后的消息
```
#### 关键点
-1. **抽取结果来源**:通过固定的常见规则(不调用模型)识别敏感值,将结果作为隐私键值。
-2. **覆盖范围**:处理 TextPart 文本、ContextPart 摘要和 ToolPart 输入、输出。工具输入中,敏感字段对应的值整体保存和遮罩。
+1. **匹配来源**:仅匹配已登记的 Privacy 值;固定规则和敏感字段名不再触发新值提取。
+2. **覆盖范围**:处理 TextPart 文本、ContextPart 摘要和 ToolPart 输入、输出。工具输入递归匹配已登记值,不再根据字段名整体保存和遮罩。
3. **内容替换**:原文中的敏感片段会替换为占位符:
- 当前 Session 的敏感值统一使用以下占位符,以 Session ID 或其哈希映射值标识会话:<code>&#123;&#123;ov_privacy:session:&#123;session_id&#125;:&#123;field_name&#125;&#125;&#125;</code>。
- 若敏感片段与已登记的个人密钥(credential)匹配,则优先使用唯一匹配字段的占位符,例如 <code>&#123;&#123;ov_privacy:alice:viking://~/privacy/credential/github#/values/pat&#125;&#125;</code>。
-4. **写盘结果**:匹配成功的片段以占位符保存,未匹配内容保持原样。新提取的隐私值保存在当前 Session 的配置中。
+4. **写盘结果**:匹配成功的片段以占位符保存,未匹配内容保持原样;不新增 Session Privacy 数据。历史配置和已有占位符的读取还原继续保留。
---
diff --git a/openviking/privacy/session_extractor.py b/openviking/privacy/session_extractor.py
index c68bcdaba..6b1684403 100644
--- a/openviking/privacy/session_extractor.py
+++ b/openviking/privacy/session_extractor.py
@@ -1,6 +1,6 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
-"""Extract Session credentials before externalization and message persistence."""
+"""Protect registered Session credentials before externalization and persistence."""
import asyncio
import hashlib
@@ -32,6 +32,10 @@ MAX_BATCH_CHARS = 16 * 1024 * 1024
MAX_DEPTH = 32
MAX_NODES = 100_000
MAX_FIELD_ID_ATTEMPTS = 8
+# TODO: Revisit new-value discovery after fixing false positives that pollute
+# Session privacy storage. Keep registered-value matching and restoration active.
+# This is intentionally disabled in code, not exposed as a runtime setting.
+_SESSION_PRIVACY_DISCOVERY_ENABLED = False
_Matcher = Callable[[str], list[PrivacyReplacement]]
_Text = TypeVar("_Text", str, None)
_HASHED_TARGET = re.compile(r"_[0-9a-f]{24}")
@@ -44,7 +48,7 @@ async def extract_and_store_session_privacy(
privacy: UserPrivacyConfigService,
sanitizer: PrivacySanitizer | None,
) -> list[dict]:
- """Extract on copies, persist discoveries, then protect remaining registered values."""
+ """Protect copies with registered values; new-value discovery is paused."""
target_key = session_privacy_target(session_id)
labels = None if target_key == session_id else {"session_id": session_id}
matcher = await _prepare_session_matcher(sanitizer, ctx, target_key)
@@ -246,7 +250,7 @@ class _SessionExtraction(_PayloadTransformer):
return super().transform_part(replace(part))
def transform_field(self, key: str, value: Any, depth: int) -> Any:
- if not is_sensitive_key(key):
+ if not _SESSION_PRIVACY_DISCOVERY_ENABLED or not is_sensitive_key(key):
return self.transform_value(value, depth)
copied = self.literal.transform_value(value, depth)
# Empty strings, references and environment variables retain their meaning.
@@ -276,7 +280,8 @@ class _SessionExtraction(_PayloadTransformer):
spans: list[tuple[int, int, str | None]] = [
(match.start, match.end, match.reference) for match in self.value_matcher(text)
]
- spans.extend((match.start, match.end, None) for match in find_privacy_matches(text))
+ if _SESSION_PRIVACY_DISCOVERY_ENABLED:
+ spans.extend((match.start, match.end, None) for match in find_privacy_matches(text))
replacements = []
for start, end, reference in _merge_spans(spans):
if reference is None:
diff --git a/tests/server/test_session_privacy.py b/tests/server/test_session_privacy.py
index a1169e95a..a652bc3e6 100644
--- a/tests/server/test_session_privacy.py
+++ b/tests/server/test_session_privacy.py
@@ -11,6 +11,7 @@ from unittest.mock import AsyncMock
import pytest
from openviking.message import ContextPart, ImagePart, TextPart, ToolPart
+from openviking.privacy import session_extractor
from openviking.privacy.reference import build_privacy_field_reference
from openviking.privacy.session_extractor import (
build_session_placeholder,
@@ -29,6 +30,61 @@ def ctx():
return RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)
+def enable_session_privacy_discovery(monkeypatch):
+ """Keep coverage of the retained discovery code while production disables it."""
+ monkeypatch.setattr(session_extractor, "_SESSION_PRIVACY_DISCOVERY_ENABLED", True)
+
+
[email protected]("batch", [False, True], ids=["single", "batch"])
+async def test_http_skips_discovery_but_protects_registered_values(
+ client, service, ctx, monkeypatch, batch
+):
+ sid = "discovery-disabled"
+ session = await service.sessions.create(ctx, sid)
+ service.sessions.set_tool_output_externalization_config(
+ ToolOutputExternalizationConfig(enabled=True, threshold_chars=1024, preview_chars=200)
+ )
+ await service.privacy_configs.upsert(ctx, "credential", "github", {"pat": "registered-secret"})
+ reference = build_privacy_field_reference(ctx.user.user_id, "credential", "github", "pat")
+ insert = AsyncMock(wraps=service.privacy_configs.insert_values)
+ monkeypatch.setattr(service.privacy_configs, "insert_values", insert)
+ output = "padding\n" * 1500 + "password=example-password registered-secret"
+ message = {
+ "role": "assistant",
+ "parts": [
+ {"type": "text", "text": "password=example-password registered-secret"},
+ {
+ "type": "tool",
+ "tool_id": "read",
+ "tool_input": {
+ "password": "example-password",
+ "credentials": {"api_key": "registered-secret", "token": 1234},
+ },
+ "tool_output": output,
+ },
+ ],
+ }
+ endpoint = f"/api/v1/sessions/{sid}/messages" + ("/batch" if batch else "")
+ response = await client.post(endpoint, json={"messages": [message]} if batch else message)
+ assert response.status_code == 200, response.text
+ insert.assert_not_awaited()
+ assert await service.privacy_configs.get_current(ctx, "session", sid) is None
+ disk = await service.viking_fs.read_file(session.uri + "/messages.jsonl", ctx=ctx)
+ assert "registered-secret" not in disk
+ parts = json.loads(disk)["parts"]
+ assert parts[0]["text"] == f"password=example-password {reference}"
+ assert parts[1]["tool_input"] == {
+ "password": "example-password",
+ "credentials": {"api_key": reference, "token": 1234},
+ }
+ output_uri = parts[1]["tool_output_ref"] + "/output.txt"
+ stored_output = await service.viking_fs.read_file(output_uri, ctx=ctx)
+ assert stored_output == output.replace("registered-secret", reference)
+ assert await service.fs.read(output_uri, ctx=ctx) == output
+
+
[email protected]("enable_session_privacy_discovery")
async def test_http_preserves_protected_content_and_restores_only_raw_reads(client, service, ctx):
sid = "http-privacy"
response = await client.post("/api/v1/sessions", json={"session_id": sid})
@@ -79,6 +135,7 @@ async def test_http_preserves_protected_content_and_restores_only_raw_reads(clie
assert (await client.get(f"/api/v1/privacy-configs/session/{sid}")).status_code == 200
[email protected]("enable_session_privacy_discovery")
async def test_default_read_keeps_escaped_short_and_container_values_protected(
client, service, ctx
):
@@ -101,6 +158,7 @@ async def test_default_read_keeps_escaped_short_and_container_values_protected(
assert json.loads(raw.json()["result"])["parts"][0]["tool_input"] == original
[email protected]("enable_session_privacy_discovery")
async def test_new_equal_fields_keep_independent_references_when_one_rotates(service, ctx):
session = await service.sessions.create(ctx, "equal-new-fields")
original = {"password": "same-new-value", "api_key": "same-new-value"}
@@ -119,6 +177,7 @@ async def test_new_equal_fields_keep_independent_references_when_one_rotates(ser
}
[email protected]("enable_session_privacy_discovery")
async def test_batch_repeated_text_values_allow_later_messages(service, ctx):
session = await service.sessions.create(ctx, "batch-repeat")
texts = [
@@ -156,6 +215,7 @@ async def test_basic_prose_does_not_register_words_for_later_messages(service, c
@pytest.mark.parametrize("sid", ["markdown-restatement", "中文会话"])
[email protected]("enable_session_privacy_discovery")
async def test_markdown_restatement_reuses_references_without_new_versions(service, ctx, sid):
session = await service.sessions.create(ctx, sid)
user_text = "API_KEY=demo-api-secret\nAuthorization: Bearer demo-bearer-secret\nsk-abcdefghijklmnopqrstuv"
@@ -222,6 +282,7 @@ async def test_session_and_skill_require_explicit_reference_for_ambiguous_value(
assert restored["parts"][0]["text"] == "same-registered-value same-registered-value"
[email protected]("enable_session_privacy_discovery")
async def test_batch_discovery_protects_all_payloads_without_mutating_inputs(service, ctx):
await service.privacy_configs.upsert(ctx, "credential", "github", {"pat": "registered-value"})
fresh = "alpha-987654"
@@ -280,6 +341,7 @@ async def test_batch_discovery_protects_all_payloads_without_mutating_inputs(ser
assert json.loads(restored) == json.loads(expected)
[email protected]("enable_session_privacy_discovery")
async def test_batch_discovery_rejects_key_collision_before_message_write(
service, ctx, monkeypatch
):
@@ -302,6 +364,7 @@ async def test_batch_discovery_rejects_key_collision_before_message_write(
assert await service.viking_fs.read_file(session.uri + "/messages.jsonl", ctx=ctx) == ""
[email protected]("enable_session_privacy_discovery")
async def test_externalized_output_restores_without_corrupting_metadata(service, ctx):
session = await service.sessions.create(ctx, "externalized")
session._tool_output_externalization_config = ToolOutputExternalizationConfig(
@@ -326,6 +389,7 @@ async def test_externalized_output_restores_without_corrupting_metadata(service,
)
[email protected]("enable_session_privacy_discovery")
async def test_auto_created_session_extracts_before_persistence(service, ctx):
session = await service.sessions.get("auto-created", ctx, auto_create=True)
await session.add_message_async("user", [TextPart("password=default-secret")])
@@ -341,6 +405,7 @@ async def test_auto_created_session_extracts_before_persistence(service, ctx):
"sid",
["中文会话", "team session@@group", " leading and trailing ", "x" * 300, "group:100%~"],
)
[email protected]("enable_session_privacy_discovery")
async def test_custom_session_ids_support_privacy_and_cross_session_reuse(service, ctx, sid):
session = await service.sessions.create(ctx, sid)
await session.add_message_async("user", [TextPart("password=custom-session-secret")])
@@ -363,6 +428,7 @@ async def test_custom_session_ids_support_privacy_and_cross_session_reuse(servic
assert json.loads(restored)["parts"][0]["text"] == "password=custom-session-secret"
[email protected]("enable_session_privacy_discovery")
async def test_concurrent_writers_preserve_messages_and_independent_fields(
service, ctx, monkeypatch
):
@@ -400,6 +466,7 @@ async def test_concurrent_writers_preserve_messages_and_independent_fields(
@pytest.mark.parametrize(
"error", [ResourceExhaustedError, OSError], ids=["id-exhaustion", "storage"]
)
[email protected]("enable_session_privacy_discovery")
async def test_privacy_write_failure_rejects_batch_before_externalization(
service, ctx, monkeypatch, error
):
@@ -428,6 +495,7 @@ async def test_privacy_write_failure_rejects_batch_before_externalization(
assert await service.privacy_configs.list_versions(ctx, "session", session.session_id) == [1]
[email protected]("enable_session_privacy_discovery")
async def test_values_equal_to_json_keys_do_not_change_message_identity(service, ctx):
session = await service.sessions.create(ctx, "identity")
original = {"password": "tool_input", "token": "assistant", "secret": "default"}
@@ -496,6 +564,7 @@ async def test_restore_leaves_foreign_missing_and_nested_references_unchanged(
read.assert_any_await(ctx, "credential", "absent")
[email protected]("enable_session_privacy_discovery")
async def test_later_credential_registration_preserves_session_reference_identity(
client, service, ctx
):
@@ -533,6 +602,7 @@ async def test_later_credential_registration_preserves_session_reference_identit
assert await service.viking_fs.read_file(first_uri, ctx=ctx) == stored
[email protected]("enable_session_privacy_discovery")
async def test_cross_session_reuse_requires_full_reference_but_short_reference_stays_local(
service, ctx
):
diff --git a/tests/unit/session/test_privacy.py b/tests/unit/session/test_privacy.py
index 34f6b1a85..d0e56e972 100644
--- a/tests/unit/session/test_privacy.py
+++ b/tests/unit/session/test_privacy.py
@@ -35,6 +35,53 @@ def privacy():
return SimpleNamespace(get_current=AsyncMock())
+def enable_session_privacy_discovery(monkeypatch):
+ """Keep coverage of the retained discovery code while production disables it."""
+ monkeypatch.setattr(extraction, "_SESSION_PRIVACY_DISCOVERY_ENABLED", True)
+
+
+async def test_new_privacy_discovery_is_disabled(monkeypatch, request_context):
+ def unexpected_discovery(*args, **kwargs):
+ pytest.fail("Session writes must not run new-value discovery")
+
+ monkeypatch.setattr(extraction, "find_privacy_matches", unexpected_discovery)
+ monkeypatch.setattr(extraction, "is_sensitive_key", unexpected_discovery)
+ privacy = SimpleNamespace(insert_values=AsyncMock())
+ specs = [
+ {
+ "role": "assistant",
+ "parts": [
+ TextPart("password=example-password sk-abcdefghijklmnopqrstuv"),
+ ContextPart(uri="viking://resources/doc", abstract="api_key=example-key"),
+ ToolPart(
+ tool_input={
+ "password": "example-password",
+ "token": 1234,
+ "secret": [],
+ "credentials": {"api_key": "example-key", "items": [False, None]},
+ },
+ tool_output="Authorization: Bearer example-token",
+ tool_output_externalization_error="password=example-error",
+ ),
+ ToolPart(tool_output=None),
+ ImagePart(url="https://example.test/image"),
+ ],
+ }
+ ]
+ before = deepcopy(specs)
+ protected = await extraction.extract_and_store_session_privacy(
+ "discovery-disabled", specs, request_context, privacy, None
+ )
+ assert protected == before
+ assert specs == before
+ privacy.insert_values.assert_not_awaited()
+ protected[0]["parts"][2].tool_input["credentials"]["items"].append("changed")
+ protected[0]["parts"][0].text = "changed"
+ assert specs == before
+
+
[email protected]("enable_session_privacy_discovery")
async def test_extract_and_restore_complete_messages(privacy, request_context):
specs = [
{
@@ -94,6 +141,7 @@ async def test_extract_and_restore_complete_messages(privacy, request_context):
assert "changed" not in json.dumps(list(values.values()))
[email protected]("enable_session_privacy_discovery")
async def test_batch_field_id_collision_preserves_both_values(
monkeypatch, privacy, request_context
):
@@ -108,6 +156,7 @@ async def test_batch_field_id_collision_preserves_both_values(
) == message_jsonl(specs)
[email protected]("enable_session_privacy_discovery")
def test_exhausted_field_ids_reject_without_mutating_input(monkeypatch):
monkeypatch.setattr(extraction, "token_urlsafe", lambda _: "same0001")
specs = [{"parts": [ToolPart(tool_input={"password": "first", "api_key": "second"})]}]
@@ -117,6 +166,7 @@ def test_exhausted_field_ids_reject_without_mutating_input(monkeypatch):
assert specs == before
[email protected]("enable_session_privacy_discovery")
async def test_sensitive_field_names_do_not_leak_into_references(privacy, request_context):
key = "sk-proj-12345678901234567890_api_key"
specs = [{"parts": [ToolPart(tool_input={key: "secret-value"})]}]
@@ -128,6 +178,7 @@ async def test_sensitive_field_names_do_not_leak_into_references(privacy, reques
) == message_jsonl(specs)
[email protected]("enable_session_privacy_discovery")
@pytest.mark.parametrize("sensitive", [False, True], ids=["nested-input", "sensitive-container"])
@pytest.mark.parametrize(
"limit,bound",