Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

fix(retrieve): skip preset scope directories and backfill unused context quota slots (#5679)

t0saki0d29cb4770cc
diff --git a/docs/en/api/06-retrieval.md b/docs/en/api/06-retrieval.md
index c3841f867..79827345e 100644
--- a/docs/en/api/06-retrieval.md
+++ b/docs/en/api/06-retrieval.md
@@ -660,7 +660,7 @@ Injecting context every turn used to mean searching per type, reading each hit b
#### 2. Parameters
-**L0 retrieval domain**: `query`, `image_url`, `context_type`, `limit`, `score_threshold`, `filter`, `tags`, `since`/`until`, and the optional `events_time_decay_protection` behave as in list mode. `limit` applies only to quota-free retrieval. Once `purpose` or explicit `quotas` enables bucketed retrieval, the per-category quotas are the only candidate ceilings. `target_uri` is not supported in context mode yet (returns 400); `level` is ignored because `detail` governs tiers.
+**L0 retrieval domain**: `query`, `image_url`, `context_type`, `limit`, `score_threshold`, `filter`, `tags`, `since`/`until`, and the optional `events_time_decay_protection` behave as in list mode. `limit` applies only to quota-free retrieval. Once `purpose` or explicit `quotas` enables bucketed retrieval, the sum of the per-category quotas is the candidate ceiling. `target_uri` is not supported in context mode yet (returns 400); `level` is ignored because `detail` governs tiers.
**L1 query understanding**
@@ -675,8 +675,8 @@ Injecting context every turn used to mean searching per type, reading each hit b
|-----------|------|---------|-------------|
| `limit` | int | 10 | Candidate ceiling for quota-free retrieval only; ignored when `purpose` or `quotas` enables bucketed retrieval |
| `max_tokens` | int | 1600 | The single budget parameter, estimated with a CJK-aware heuristic (codepoint ≥ 0x3000 counts 1.5 tok/char, otherwise chars/4) |
-| `quotas` | object | None | Absolute per-bucket limits; keys are `events`/`entities`/`preferences`/`experiences`/`resources`/`skills`. Explicit quotas ignore `limit` |
-| `purpose` | `chat` \| `coding` | None | Enables six-domain bucket sampling with the absolute preset quotas below. Applies only when `quotas` is not given |
+| `quotas` | object | None | Per-bucket first-pass limits whose sum caps the candidates; slots a bucket leaves unused go to the best remaining hits of the other buckets, and `0` turns a bucket off. Keys are `events`/`entities`/`preferences`/`experiences`/`resources`/`skills`. Explicit quotas ignore `limit` |
+| `purpose` | `chat` \| `coding` | None | Enables six-domain bucket sampling with the preset quotas below. Applies only when `quotas` is not given |
| `detail` | `abstract` \| `overview` \| `full` \| object | None | Requests one starting/maximum tier for every entry. Entries whose requested tier is unavailable or does not fit step down instead of being truncated. Omitted, each category takes its default tier (below). Also accepts a per-category object such as `{"events":"overview","preferences":"abstract"}`; categories left out keep their default. `"auto"` is a deprecated spelling and behaves as if omitted |
| `dedup_turns` | int | 0 | Cooldown window in turns; needs `session_id`. Ledger lives at `{session_uri}/.recall_log.json` |
| `exclude_uris` | string[] | [] | Stateless dedup fallback, up to 200 entries, unioned with `dedup_turns` |
@@ -692,8 +692,8 @@ Injecting context every turn used to mean searching per type, reading each hit b
**Tier rules**
-- **Purpose presets**: `chat` uses `events:3, entities:3, preferences:1, experiences:1, resources:1, skills:1`; `coding` uses `events:1, entities:2, preferences:1, experiences:1, resources:3, skills:2`. These are absolute per-category ceilings, not weights. Results are deduplicated and globally sorted after gathering, but are not truncated by a second global `limit`
-- **Default tier per category**: with `detail` omitted, each category lands on the tier below. Among file entries, only `events` needs a body read at its default tier. Directory hits read `.overview.md`
+- **Purpose presets**: `chat` uses `events:3, entities:3, preferences:1, experiences:1, resources:1, skills:1`; `coding` uses `events:1, entities:2, preferences:1, experiences:1, resources:3, skills:2`. Each bucket first takes up to its own quota; when a bucket finds fewer matches, its unused slots go to the best remaining hits of the other buckets, so the total (10 for either preset) is never exceeded. Results are deduplicated and globally sorted after gathering, but are not truncated by a second global `limit`
+- **Default tier per category**: with `detail` omitted, each category lands on the tier below. Among file entries, only `events` needs a body read at its default tier. Directory hits read `.overview.md`. Preset scope directories (`viking://resources`, `viking://agent` and its preset children, the user root and its first-level directories) serve only for navigation and are never returned as entries, even after their `.overview.md` has been regenerated from their contents
| Category | Default tier | Leftover budget may reach | Why |
|----------|--------------|---------------------------|-----|
diff --git a/docs/zh/api/06-retrieval.md b/docs/zh/api/06-retrieval.md
index 302c27614..6a4b57909 100644
--- a/docs/zh/api/06-retrieval.md
+++ b/docs/zh/api/06-retrieval.md
@@ -660,7 +660,7 @@ Agent 插件每轮注入上下文时,过去需要按类型逐个检索、再
#### 2. 接口和参数说明
-**L0 检索域**:`query`、`image_url`、`context_type`、`limit`、`score_threshold`、`filter`、`tags`、`since`/`until` 以及可选的 `events_time_decay_protection` 与 list 模式一致。`limit` 只约束 quota-free 检索;一旦 `purpose` 或显式 `quotas` 启用分桶检索,各分类配额就是唯一候选上限。`target_uri` 在 context 模式下暂不支持(返回 400);`level` 被忽略,档位由 `detail` 决定。
+**L0 检索域**:`query`、`image_url`、`context_type`、`limit`、`score_threshold`、`filter`、`tags`、`since`/`until` 以及可选的 `events_time_decay_protection` 与 list 模式一致。`limit` 只约束 quota-free 检索;一旦 `purpose` 或显式 `quotas` 启用分桶检索,各分类配额之和就是候选上限。`target_uri` 在 context 模式下暂不支持(返回 400);`level` 被忽略,档位由 `detail` 决定。
**L1 查询理解**
@@ -675,8 +675,8 @@ Agent 插件每轮注入上下文时,过去需要按类型逐个检索、再
|------|------|--------|------|
| `limit` | int | 10 | 仅作为 quota-free 检索的候选条目上限;`purpose` 或 `quotas` 启用分桶后被忽略 |
| `max_tokens` | int | 1600 | 唯一的预算参数,采用感知 CJK 的启发式估算(codepoint ≥ 0x3000 记 1.5 token/字,其余按 chars/4) |
-| `quotas` | object | None | 各桶绝对条数上限;键取 `events`/`entities`/`preferences`/`experiences`/`resources`/`skills`。显式传入后忽略 `limit` |
-| `purpose` | `chat` \| `coding` | None | 按下表的绝对分类配额启用六域分桶采样;仅在未显式传 `quotas` 时生效 |
+| `quotas` | object | None | 各桶首轮条数上限,总和即候选上限;某个桶用不满的名额会让给其他桶剩余的最佳命中,`0` 表示关闭该桶。键取 `events`/`entities`/`preferences`/`experiences`/`resources`/`skills`。显式传入后忽略 `limit` |
+| `purpose` | `chat` \| `coding` | None | 按下表的分类配额启用六域分桶采样;仅在未显式传 `quotas` 时生效 |
| `detail` | `abstract` \| `overview` \| `full` \| object | None | 为每条结果请求同一个起始档和最高档;请求档不可用或装不进预算时仍逐档退档而不截断。省略时按类别取默认档(见下)。也可传按类别的对象,如 `{"events":"overview","preferences":"abstract"}`,未列出的类别仍取默认档。`"auto"` 是已废弃的写法,等价于省略 |
| `dedup_turns` | int | 0 | 跨轮冷却轮数,需要 `session_id`;账本存在 `{session_uri}/.recall_log.json` |
| `exclude_uris` | string[] | [] | 无状态去重兜底,最多 200 条,与 `dedup_turns` 取并集 |
@@ -692,8 +692,8 @@ Agent 插件每轮注入上下文时,过去需要按类型逐个检索、再
**档位规则**
-- **Purpose 预设**:`chat` 使用 `events:3, entities:3, preferences:1, experiences:1, resources:1, skills:1`;`coding` 使用 `events:1, entities:2, preferences:1, experiences:1, resources:3, skills:2`。这些值是每个分类的绝对上限,不是权重。各桶结果汇总后仍会去重并全局排序,但不会再被第二个全局 `limit` 截断
-- **按类别的默认档**:省略 `detail` 时,各类别落在下表的档位;文件条目中只有 `events` 的默认档需要回读正文;目录命中会读取 `.overview.md`
+- **Purpose 预设**:`chat` 使用 `events:3, entities:3, preferences:1, experiences:1, resources:1, skills:1`;`coding` 使用 `events:1, entities:2, preferences:1, experiences:1, resources:3, skills:2`。每个桶先按自己的配额取条目;某个桶命中不足时,空出的名额让给其他桶剩余的最佳命中,总数(两个预设都是 10)不会超出。各桶结果汇总后仍会去重并全局排序,但不会再被第二个全局 `limit` 截断
+- **按类别的默认档**:省略 `detail` 时,各类别落在下表的档位;文件条目中只有 `events` 的默认档需要回读正文;目录命中会读取 `.overview.md`。预置的作用域目录(`viking://resources`、`viking://agent` 及其预置子目录、用户根目录及其一级目录)只用于导航,不会作为条目返回,即使它们的 `.overview.md` 已按内容重新生成也一样
| 类别 | 默认档 | 剩余预算可加深到 | 原因 |
|------|--------|------------------|------|
diff --git a/openviking/core/directories.py b/openviking/core/directories.py
index f887ffa75..6c2a1cd7f 100644
--- a/openviking/core/directories.py
+++ b/openviking/core/directories.py
@@ -427,3 +427,11 @@ class DirectoryInitializer:
if scope in {"user", "session"}:
return user_space_fragment(ctx)
return ""
+
+
+def preset_directory_uris(ctx: RequestContext) -> frozenset[str]:
+ """URIs of the preset directories whose boilerplate L0/L1 is vectorized for ``ctx``."""
+ account_roots, account_children = DirectoryInitializer._account_directory_targets(ctx)
+ user_root, user_children = DirectoryInitializer._user_directory_targets(ctx)
+ targets = (*account_roots, *account_children, user_root, *user_children)
+ return frozenset(target.uri for target in targets)
diff --git a/openviking/retrieve/context_assembler/gather.py b/openviking/retrieve/context_assembler/gather.py
index 4b4f5c2c0..3fca3fe05 100644
--- a/openviking/retrieve/context_assembler/gather.py
+++ b/openviking/retrieve/context_assembler/gather.py
@@ -12,6 +12,7 @@ import asyncio
from dataclasses import dataclass, replace
from typing import Any, Dict, List, Mapping, Optional, Sequence, Set, Tuple
+from openviking.core.directories import preset_directory_uris
from openviking.core.namespace import AGENT_SKILLS_ROOT, canonical_user_root
from openviking.core.retrieval_targets import default_target_directories
from openviking.core.retrieval_types import SearchType
@@ -224,6 +225,7 @@ async def gather_candidates(
penalties = penalties or {}
excluded = excluded or set()
planned = [q for q in queries if q] or [""]
+ preset_dirs = preset_directory_uris(ctx)
searched: Dict[str, int] = {}
retrieval_errors: List[str] = []
@@ -237,7 +239,7 @@ async def gather_candidates(
if not uri:
continue
base_uri, is_directory = strip_level_suffix(uri)
- if base_uri.endswith("/profile.md"):
+ if base_uri.endswith("/profile.md") or base_uri in preset_dirs:
continue
category = category_for(item, bucket)
abstract = _abstract(item)
@@ -308,7 +310,7 @@ async def gather_candidates(
events_time_decay_protection=events_time_decay_protection,
)
- async def gather_bucket(bucket: str, quota: int) -> List[Candidate]:
+ async def gather_bucket(bucket: str, quota: int, width: int) -> List[Candidate]:
targets = category_targets(bucket, ctx)
context_type = {
"resources": ContextType.RESOURCE,
@@ -333,7 +335,7 @@ async def gather_candidates(
query=query,
ctx=ctx,
target_uri=targets,
- limit=_overfetch(quota),
+ limit=_overfetch(width),
score_threshold=score_threshold,
filter=bucket_filter,
search_type=search_type,
@@ -345,7 +347,7 @@ async def gather_candidates(
query=query,
find_ctx=ctx,
target_uri=target,
- find_limit=_overfetch(quota),
+ find_limit=_overfetch(width),
find_filter=bucket_filter,
)
for target in targets
@@ -356,7 +358,7 @@ async def gather_candidates(
query=query,
find_ctx=ctx,
target_uri=target,
- find_limit=_overfetch(quota),
+ find_limit=_overfetch(width),
find_filter=bucket_filter,
)
for query in planned
@@ -369,7 +371,7 @@ async def gather_candidates(
query=query,
find_ctx=open_ctx,
target_uri=f"{user_root}/peers",
- find_limit=_overfetch(max(quota * OTHER_PEER_OVERFETCH, quota)),
+ find_limit=_overfetch(max(quota * OTHER_PEER_OVERFETCH, width)),
find_filter=bucket_filter,
)
for query in planned
@@ -392,7 +394,7 @@ async def gather_candidates(
if bucket == "skills":
candidates = _dedupe_candidates(candidates)
candidates.sort(key=_rank_key, reverse=True)
- return candidates[: max(0, quota)]
+ return candidates
async def gather_flat() -> List[Candidate]:
searches = [
@@ -448,9 +450,23 @@ async def gather_candidates(
candidates = await gather_flat()
else:
active = [(bucket, quota) for bucket, quota in quotas.items() if quota > 0]
- buckets = await asyncio.gather(*(gather_bucket(b, q) for b, q in active))
- candidates = [candidate for bucket in buckets for candidate in bucket]
- candidates = _dedupe_candidates(candidates)
+ total = sum(quota for _, quota in active)
+ buckets = await asyncio.gather(*(gather_bucket(b, q, total) for b, q in active))
+ # Each quota is a first-pass ceiling; slots a bucket leaves unused go to
+ # the best remaining hits of the other buckets.
+ first: List[Candidate] = []
+ spare: List[Candidate] = []
+ for (_, quota), found in zip(active, buckets, strict=True):
+ first.extend(found[:quota])
+ spare.extend(found[quota:])
+ candidates = _dedupe_candidates(first)
+ taken = {c.base_uri for c in candidates}
+ for candidate in sorted(spare, key=_rank_key, reverse=True):
+ if len(candidates) >= total:
+ break
+ if candidate.base_uri not in taken:
+ taken.add(candidate.base_uri)
+ candidates.append(candidate)
candidates.sort(key=_rank_key, reverse=True)
candidates = await _fill_skill_abstracts(service, candidates)
diff --git a/openviking/retrieve/context_assembler/params.py b/openviking/retrieve/context_assembler/params.py
index 1372adc1d..656496ce3 100644
--- a/openviking/retrieve/context_assembler/params.py
+++ b/openviking/retrieve/context_assembler/params.py
@@ -78,8 +78,9 @@ FULL_BODY_ABSTRACT_CATEGORIES: frozenset[str] = frozenset(
# reading resource bodies.
DEPTH_CEILING_BY_CATEGORY: Dict[str, Tier] = {"events": "full"}
-# Purpose presets are absolute bucket ceilings. Their total is the candidate
-# width for bucketed retrieval; ``limit`` belongs only to quota-free flat mode.
+# Purpose presets are first-pass bucket ceilings. Their total is the candidate
+# width for bucketed retrieval, and slots a bucket leaves unused go to the best
+# remaining hits of the other buckets; ``limit`` belongs only to flat mode.
PURPOSE_PRESETS: Dict[str, Dict[str, int]] = {
"coding": {
"events": 1,
diff --git a/tests/retrieve/test_context_assembler_pipeline.py b/tests/retrieve/test_context_assembler_pipeline.py
index fa977b5e9..b0798b9f9 100644
--- a/tests/retrieve/test_context_assembler_pipeline.py
+++ b/tests/retrieve/test_context_assembler_pipeline.py
@@ -2,6 +2,7 @@
# SPDX-License-Identifier: AGPL-3.0
import re
+from collections import Counter
from types import SimpleNamespace
from openviking.retrieve.context_assembler import pipeline as pipeline_module
@@ -11,7 +12,7 @@ from openviking.retrieve.context_assembler.budget import (
per_entry_cap,
plan_entries,
)
-from openviking.retrieve.context_assembler.gather import Candidate, category_for
+from openviking.retrieve.context_assembler.gather import Candidate, category_for, category_targets
from openviking.retrieve.context_assembler.models import AssembledEntry
from openviking.retrieve.context_assembler.params import (
OTHER_MEMORY_CATEGORY,
@@ -129,7 +130,7 @@ def test_render_neutralizes_forged_memory_tags():
assert rendered.endswith("</memory>")
-def test_coding_purpose_uses_absolute_cross_domain_quotas():
+def test_coding_purpose_uses_cross_domain_quotas():
assert normalize_quotas(None, "coding") == {
"events": 1,
"entities": 2,
@@ -959,3 +960,159 @@ async def test_flat_retrieval_collapses_skills_and_leaves_other_hits_alone():
assert result.stats["candidates"] == 3
assert [entry.uri for entry in result.entries] == [f"{DEPLOY}/SKILL.md", events_dir]
assert result.stats["deduped"] == 1
+
+
+def _hit(uri, score, abstract=None):
+ return {"uri": uri, "score": score, "abstract": abstract or uri, "level": 2}
+
+
+def _bucket_service(hits_by_target, finds=None):
+ """Answers each target from ``hits_by_target`` and honours the requested limit."""
+
+ async def fake_find(**kwargs):
+ if finds is not None:
+ finds.append(kwargs)
+ hits = hits_by_target.get(kwargs["target_uri"], [])[: kwargs["limit"]]
+ return _FakeFindResult(memories=list(hits), resources=list(hits))
+
+ async def fake_find_skills(**kwargs):
+ del kwargs
+ return _FakeFindResult()
+
+ async def fake_read(uri, **kwargs):
+ del kwargs
+ return f"# Summary\n{uri}"
+
+ return SimpleNamespace(
+ search=SimpleNamespace(find=fake_find, find_skills=fake_find_skills),
+ fs=SimpleNamespace(read=fake_read),
+ sessions=SimpleNamespace(),
+ viking_fs=None,
+ )
+
+
+async def test_preset_scope_directories_are_not_served():
+ project = "viking://resources/proj"
+ doc = f"{USER_ROOT}/resources/doc.md"
+ hits = [
+ _hit("viking://resources/.overview.md", 0.6, "Resource storage root"),
+ _hit(f"{USER_ROOT}/resources/.abstract.md", 0.6, "User resources"),
+ _hit(f"{USER_ROOT}/.overview.md", 0.6, "User root"),
+ _hit(f"{USER_ROOT}/privacy/.abstract.md", 0.6, "Privacy"),
+ _hit(f"{project}/.overview.md", 0.5, "project overview"),
+ _hit(doc, 0.4, "doc"),
+ ]
+
+ async def fake_find(**kwargs):
+ del kwargs
+ return _FakeFindResult(resources=list(hits))
+
+ async def fake_read(uri, **kwargs):
+ del kwargs
+ return f"overview of {uri}"
+
+ service = SimpleNamespace(
+ search=SimpleNamespace(find=fake_find),
+ fs=SimpleNamespace(read=fake_read),
+ sessions=SimpleNamespace(),
+ viking_fs=None,
+ )
+ result = await assemble_context(
+ service=service,
+ ctx=_ctx(),
+ params=AssembleParams(query="q", quotas={"resources": 3}, peer_scope="actor"),
+ )
+
+ assert [entry.uri for entry in result.entries] == [project, doc]
+
+
+async def test_flat_retrieval_skips_preset_directories():
+ memory = f"{USER_ROOT}/memories/preferences/lang.md"
+ hits = [
+ _hit(f"{USER_ROOT}/memories/.overview.md", 0.7, "Memory root"),
+ _hit("viking://agent/tools/.abstract.md", 0.7, "Tools"),
+ _hit(memory, 0.5, "prefers Python"),
+ ]
+ service = _service(hits=hits, bodies={})
+
+ result = await assemble_context(
+ service=service,
+ ctx=_ctx(),
+ params=AssembleParams(query="q", limit=5, peer_scope="actor"),
+ )
+
+ assert [entry.uri for entry in result.entries] == [memory]
+
+
+async def test_unused_bucket_slots_backfill_with_the_best_remaining_hits():
+ preferences = f"{USER_ROOT}/memories/preferences"
+ service = _bucket_service(
+ {
+ preferences: [
+ _hit(f"{preferences}/nickname.md", 0.75, "call me Z"),
+ _hit(f"{preferences}/style.md", 0.67, "terse answers"),
+ ]
+ }
+ )
+
+ result = await assemble_context(
+ service=service,
+ ctx=_ctx(),
+ params=AssembleParams(query="my preferences", purpose="coding", peer_scope="actor"),
+ )
+
+ assert [entry.uri for entry in result.entries] == [
+ f"{preferences}/nickname.md",
+ f"{preferences}/style.md",
+ ]
+
+
+async def test_backfill_never_exceeds_the_quota_total_or_reaches_zero_quota_buckets():
+ finds = []
+ preferences = f"{USER_ROOT}/memories/preferences"
+ entities = f"{USER_ROOT}/memories/entities"
+ service = _bucket_service(
+ {
+ preferences: [_hit(f"{preferences}/{i}.md", 0.9 - i * 0.1) for i in range(3)],
+ entities: [_hit(f"{entities}/e.md", 0.95)],
+ },
+ finds=finds,
+ )
+
+ result = await assemble_context(
+ service=service,
+ ctx=_ctx(),
+ params=AssembleParams(
+ query="q",
+ quotas={"preferences": 1, "events": 1, "entities": 0},
+ peer_scope="actor",
+ ),
+ )
+
+ assert [entry.uri for entry in result.entries] == [
+ f"{preferences}/0.md",
+ f"{preferences}/1.md",
+ ]
+ assert all("/entities" not in call["target_uri"] for call in finds)
+
+
+async def test_full_buckets_keep_their_quotas_when_one_bucket_scores_highest():
+ quotas = {"events": 1, "entities": 2, "preferences": 1, "experiences": 1, "resources": 3}
+ hits_by_target = {}
+ for bucket, base in (
+ ("events", 0.9),
+ ("entities", 0.5),
+ ("preferences", 0.5),
+ ("experiences", 0.5),
+ ("resources", 0.5),
+ ):
+ target = category_targets(bucket, _ctx())[0]
+ hits_by_target[target] = [_hit(f"{target}/{i}.md", base - i * 0.01) for i in range(8)]
+
+ result = await assemble_context(
+ service=_bucket_service(hits_by_target),
+ ctx=_ctx(),
+ params=AssembleParams(query="q", quotas=quotas, max_tokens=8000, peer_scope="actor"),
+ )
+
+ assert Counter(entry.category for entry in result.entries) == quotas
diff --git a/tests/unit/test_directory_initializer.py b/tests/unit/test_directory_initializer.py
index 9c2a5fcd6..f4e937838 100644
--- a/tests/unit/test_directory_initializer.py
+++ b/tests/unit/test_directory_initializer.py
@@ -132,3 +132,25 @@ async def test_initialize_user_directories_ignores_actor_peer_view_for_preset_st
}
assert count == len(expected_uris)
assert set(viking_fs.contexts) == expected_uris
+
+
+def test_preset_directory_uris_cover_seeded_directories_only():
+ from openviking.core.directories import preset_directory_uris
+
+ ctx = RequestContext(user=UserIdentifier("acme", "alice"), role=Role.USER)
+ user_root = canonical_user_root(ctx)
+
+ uris = preset_directory_uris(ctx)
+
+ assert {
+ "viking://resources",
+ "viking://agent",
+ "viking://agent/skills",
+ user_root,
+ f"{user_root}/memories",
+ f"{user_root}/resources",
+ f"{user_root}/privacy",
+ f"{user_root}/peers",
+ } <= uris
+ assert f"{user_root}/memories/preferences" not in uris
+ assert "viking://resources/proj" not in uris