Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat(skills): index full skill packages and return the best hit per skill (#5045)

DuTaoef7f258a7eff
diff --git a/crates/ov_cli/src/commands/search.rs b/crates/ov_cli/src/commands/search.rs
index b20b0add1..99c670a92 100644
--- a/crates/ov_cli/src/commands/search.rs
+++ b/crates/ov_cli/src/commands/search.rs
@@ -279,6 +279,7 @@ fn render_search_results_for_table_with_context(
text_width,
hide_level_and_score,
split_name_description,
+ context.is_some_and(|context| context.mode == SearchRenderMode::SkillsFind),
&mut lines,
);
}
@@ -409,6 +410,7 @@ fn render_search_result_card(
text_width: usize,
hide_level_and_score: bool,
split_name_and_description: bool,
+ prefer_root_uri: bool,
lines: &mut Vec<String>,
) {
let object = item.as_object();
@@ -434,7 +436,13 @@ fn render_search_result_card(
metadata.join(" · ")
));
- if let Some(uri) = search_result_uri(object) {
+ let root_uri = object
+ .filter(|_| prefer_root_uri)
+ .and_then(|object| object.get("root_uri"))
+ .and_then(Value::as_str)
+ .map(str::trim)
+ .filter(|uri| !uri.is_empty());
+ if let Some(uri) = root_uri.or_else(|| search_result_uri(object)) {
for line in wrap_display_text(uri, text_width, SEARCH_MAX_URI_LINES) {
lines.push(format!("{SEARCH_INDENT}{}", theme::sky_value(line).bold()));
}
diff --git a/docs/en/api/04-skills.md b/docs/en/api/04-skills.md
index 7e81415b1..ef3ff5034 100644
--- a/docs/en/api/04-skills.md
+++ b/docs/en/api/04-skills.md
@@ -33,6 +33,12 @@ viking://user/{user_id}/skills/
+-- ...
```
+### Package Processing and Indexing
+
+New Skills process the entire package by default, without an additional flag. Root L0 still comes from Skill metadata, and root L1 still uses only `SKILL.md`. Subdirectories use resource directory summaries. Directory L0/L1 and supported L2 files are indexed; nested `SKILL.md` files remain ordinary attachments.
+
+Auxiliary files now enter the existing summary and embedding pipeline, subject to the current hidden-file filters, input limits, and media settings. Images use supported image or text inputs; audio and video use text summaries with the existing filename fallback. `wait=true` waits for the whole package. Upgrades do not automatically rebuild existing Skills: use `semantic_and_vectors` to populate missing summaries; rebuilding vectors alone does not create them.
+
### SKILL.md Format
Skills can be defined using SKILL.md files with YAML frontmatter:
@@ -607,13 +613,31 @@ Search and `get_skill` return content and manifests without executing scripts or
| Parameter | Default | Meaning |
|-----------|---------|---------|
| `query` | Required | Search text |
-| `limit` | `10` | Requested candidate limit per search root |
+| `limit` | `10` | Maximum distinct Skills returned after merging private and shared scopes |
| `score_threshold` | `null` | Minimum score; omission uses underlying retrieval defaults |
-| `level` | `null` | Level filter list such as `[0]`; unlike the single integer on get |
+| `level` | `null` | Levels allowed to match, such as `[0]`; omission allows all levels, unlike the single integer on get |
| `target_uri` | `null` | Search scope; omitted searches private and shared roots separately |
| `telemetry` | `false` | Telemetry configuration |
-Merged results are sorted by score, without name deduplication or a final `limit` truncation, so the total can exceed `limit`. `total` is the returned array length, not a count of all possible matches. Results are summaries, not complete packages.
+Package hits are grouped by their full Skill root URI, ranked by their highest final score, then limited to `limit` Skills. Same-named Skills in different scopes remain distinct. `total` is the returned array length, not a count of all possible matches.
+
+Each Skill is represented by its highest-scoring package hit. The existing `uri`, `level`, `score`, and `abstract` fields come directly from that hit, without additional response fields. Package grouping and pagination apply only to dedicated `skills/find`; general `find/search` continues to return individual hits.
+
+| Response field | Meaning |
+| --- | --- |
+| `uri` / `level` | Actual hit URI and level: L0 points to `.abstract.md`, L1 to `.overview.md`, and L2 to the matching file |
+| `score` | The hit's final score, also used to rank its Skill |
+| `abstract` | The hit record's existing summary, following the original search rules |
+| `name` / `description` / `tags` / `allowed_tools` | Metadata read separately from the Skill root by the dedicated `skills/find` endpoint |
+| `root_uri` / `skill_md_uri` | The Skill root and main `SKILL.md` addresses returned by the dedicated `skills/find` endpoint |
+
+The dedicated `skills/find` endpoint now returns the actual hit URI instead of replacing it with the package root. List and get-by-name responses are unchanged. `level=[2]` restricts matching to files and returns `level=2` with the URI of the highest-scoring file in each package.
+
+General search with `read_content=true` continues to read the returned URI. The dedicated `skills/find` endpoint does not accept this option.
+
+The URI rules above apply to semantic search. Filter-only general `find` retains the stored record URI and returns `score=0`, without adding summary-file suffixes for L0 or L1.
+
+Scope, level, and permission filters apply before grouping; the Skill root must also be accessible.
**Python SDK**
@@ -670,6 +694,8 @@ Missing names/descriptions and invalid YAML produce errors. Directory/name misma
Supply new content/an upload, or set `from_source=true`. The new name must match `skill_name` in the URL; update is neither rename nor a partial patch. The server parses and checks the new package before backing up and replacing the old one, and attempts restoration on synchronous failure. Include all auxiliary files that should remain in the replacement package.
+Background processing starts after the update's files, privacy configuration, and task setup are ready. When an update needs restoration, including a `wait=true` timeout, it cancels that update's summary and indexing work, waits for started writes to exit, and restores the original files, index, and privacy configuration. Restoration failures are reported. The timeout response may arrive after `timeout` while cancellation settles; it does not wait for the entire package to finish processing. Background failures after a successful `wait=false` response do not trigger restoration. A timeout when adding a new Skill still only ends the wait.
+
**Python SDK**
```python
diff --git a/docs/zh/api/04-skills.md b/docs/zh/api/04-skills.md
index 46f85a232..196779d65 100644
--- a/docs/zh/api/04-skills.md
+++ b/docs/zh/api/04-skills.md
@@ -33,6 +33,12 @@ viking://user/{user_id}/skills/
+-- ...
```
+### 整包处理与索引
+
+新增 Skill 默认处理整个包,不需要额外开关。主目录 L0 仍来自技能元数据,L1 仍只根据 `SKILL.md` 生成;子目录沿用 resource 的方式生成 L0、L1。目录 L0、L1 和支持的文件 L2 都建立索引,包内嵌套的 `SKILL.md` 作为普通附件处理。
+
+辅助文件会进入已有的摘要和 embedding 处理流程,受现有隐藏文件过滤、长度上限和媒体配置约束。图片使用模型支持的图片或文字输入;音视频使用文字摘要,无法理解时沿用文件名回退。`wait=true` 等待整个包处理完成。升级不会自动重建旧 Skill;首次补齐应使用 `semantic_and_vectors` 重建,仅重建向量不会生成缺失的目录摘要。
+
### SKILL.md 格式
技能可以使用带有 YAML frontmatter 的 SKILL.md 文件来定义:
@@ -608,13 +614,31 @@ HTTP 对应查询为 `GET /api/v1/skills/search-web?include_content=true&include
| 参数 | 默认值 | 含义 |
|------|--------|------|
| `query` | 必填 | 检索文本 |
-| `limit` | `10` | 每个检索根请求的候选上限 |
+| `limit` | `10` | 最多返回的不同 Skill 数量,包含私有与共享空间的合并结果 |
| `score_threshold` | `null` | 最低分数,未指定时使用底层检索默认行为 |
-| `level` | `null` | 层级过滤列表,例如 `[0]`;与读取接口的单个整数不同 |
+| `level` | `null` | 参与匹配的层级列表,例如 `[0]`;未指定时不限制层级,与读取接口的单个整数不同 |
| `target_uri` | `null` | 限定检索范围;省略时分别检索私有与共享根 |
| `telemetry` | `false` | 遥测配置 |
-双根检索合并后按分数排序,不按名称去重,也不再按 `limit` 截断,因此总结果可能超过 `limit`。`total` 是本次返回数组长度,不是所有匹配项的总数。结果是摘要,不是完整 Skill 包。
+包内命中按完整 Skill 根 URI 合并,使用最高最终得分排序,再截取 `limit` 个 Skill。不同空间的同名 Skill 分别保留。`total` 是本次返回数组长度,不是所有匹配项的总数。
+
+每个 Skill 返回包内最终得分最高的一条命中。`uri`、`level`、`score`、`abstract` 直接使用该命中的原有字段,不增加额外返回字段。按 Skill 合并和补页仅用于专用 `skills/find`;通用 `find/search` 保持按命中内容返回。
+
+| 返回字段 | 含义 |
+| --- | --- |
+| `uri` / `level` | 实际命中地址及层级:L0 指向 `.abstract.md`,L1 指向 `.overview.md`,L2 指向具体文件 |
+| `score` | 该命中的最终得分,也是所属 Skill 的排序得分 |
+| `abstract` | 该命中记录已有的摘要,沿用原搜索规则 |
+| `name` / `description` / `tags` / `allowed_tools` | 专用 `skills/find` 从 Skill 主目录单独读取的元数据 |
+| `root_uri` / `skill_md_uri` | 专用 `skills/find` 返回的 Skill 根目录和主 `SKILL.md` 地址 |
+
+专用 `skills/find` 的 `uri` 从原先的包根地址调整为实际命中地址,列表和按名称读取接口保持原样。`level=[2]` 只让文件参与匹配,返回的 `level` 为 `2`、`uri` 指向包内得分最高的文件。
+
+通用检索中 `read_content=true` 继续读取实际返回的 `uri`。专用 `skills/find` 不支持该参数。
+
+上表的 URI 规则适用于语义检索。通用 `find` 仅按 `filter` 筛选时,仍保留索引记录的 URI、返回 `score=0`,不为 L0、L1 补摘要文件后缀。
+
+搜索范围、层级和权限限制先作用于包内命中,再合并 Skill;根目录也必须可访问。
**Python SDK**:
@@ -672,6 +696,8 @@ curl -X POST http://localhost:1933/api/v1/skills/find \
必须提供新内容/上传包,或设 `from_source=true`。新内容的名称必须与 URL 中的 `skill_name` 一致;更新不是重命名或局部 patch。先解析并检查新包,再备份替换;同步失败会尝试恢复旧包。需保留的辅助文件应随新包一起提交。
+更新的文件、隐私配置和任务准备完成后才启动后台。需要恢复旧包时,包括 `wait=true` 超时,先取消本次摘要和索引任务,确认已开始的写入退出,再恢复原文件、索引及隐私配置;恢复失败会明确报告。超时响应可能晚于 `timeout`,但只等待取消收尾,不继续处理完整包。`wait=false` 已成功返回后的后台失败不自动恢复;新增 Skill 的等待超时仍只结束等待。
+
**Python SDK**:
```python
diff --git a/openviking/privacy/service.py b/openviking/privacy/service.py
index b2bc08c7b..1bbef6c20 100644
--- a/openviking/privacy/service.py
+++ b/openviking/privacy/service.py
@@ -40,8 +40,10 @@ class UserPrivacyConfigService:
async def exists(self, ctx: RequestContext, category: str, target_key: str) -> bool:
try:
+ # A Skill update may retain an empty directory solely for its
+ # lock. Only persisted configuration makes the target readable.
await self._viking_fs.stat(
- self.get_config_root(ctx, category, target_key),
+ current_uri(self._user_space(ctx), category, target_key),
ctx=ctx,
skip_count=True,
)
@@ -49,14 +51,44 @@ class UserPrivacyConfigService:
except (NotFoundError, FileNotFoundError):
return False
- async def delete(self, ctx: RequestContext, category: str, target_key: str) -> bool:
+ async def delete(
+ self,
+ ctx: RequestContext,
+ category: str,
+ target_key: str,
+ *,
+ owner_lease_ref: Optional[dict[str, Any]] = None,
+ ) -> bool:
root_uri = self.get_config_root(ctx, category, target_key)
- async with self._config_lock(ctx, category, target_key) as lease:
- if not await self.exists(ctx, category, target_key):
+ async with self._config_lock(
+ ctx, category, target_key, owner_lease_ref=owner_lease_ref
+ ) as lease:
+ if not await self._viking_fs.exists(root_uri, ctx=ctx):
return False
- await self._viking_fs.rm(root_uri, recursive=True, ctx=ctx, lease_ref=lease)
+ if owner_lease_ref is not None:
+ # An enclosing Skill update still needs this root's lock for
+ # rollback. Its final cleanup removes the root if left empty.
+ await self._clear_contents(ctx, category, target_key, lease)
+ else:
+ await self._viking_fs.rm(root_uri, recursive=True, ctx=ctx, lease_ref=lease)
return True
+ async def _clear_contents(
+ self,
+ ctx: RequestContext,
+ category: str,
+ target_key: str,
+ lease_ref: Optional[dict[str, Any]],
+ ) -> None:
+ """Clear config data under an existing lock, preserving the locked root."""
+ root = self.get_config_root(ctx, category, target_key)
+ for name in (".meta.json", "current.json", "history"):
+ uri = f"{root}/{name}"
+ if await self._viking_fs.exists(uri, ctx=ctx):
+ await self._viking_fs.rm(
+ uri, recursive=name == "history", ctx=ctx, lease_ref=lease_ref
+ )
+
async def _ensure_root(
self,
ctx: RequestContext,
@@ -75,7 +107,12 @@ class UserPrivacyConfigService:
@asynccontextmanager
async def _config_lock(
- self, ctx: RequestContext, category: str, target_key: str
+ self,
+ ctx: RequestContext,
+ category: str,
+ target_key: str,
+ *,
+ owner_lease_ref: Optional[dict[str, Any]] = None,
) -> AsyncIterator[Optional[dict[str, Any]]]:
uri_to_path = getattr(self._viking_fs, "_uri_to_path", None)
agfs = getattr(self._viking_fs, "_async_agfs", None)
@@ -87,6 +124,7 @@ class UserPrivacyConfigService:
lease = await acquire(
uri_to_path(self.get_config_root(ctx, category, target_key), ctx=ctx),
timeout_secs=30.0,
+ **({"owner_lease_ref": owner_lease_ref} if owner_lease_ref is not None else {}),
)
try:
yield lease
@@ -181,8 +219,12 @@ class UserPrivacyConfigService:
updated_by: str = "",
change_reason: str = "",
labels: Optional[dict[str, Any]] = None,
+ *,
+ owner_lease_ref: Optional[dict[str, Any]] = None,
) -> UserPrivacyConfigVersion:
- async with self._config_lock(ctx, category, target_key) as lease:
+ async with self._config_lock(
+ ctx, category, target_key, owner_lease_ref=owner_lease_ref
+ ) as lease:
await self._ensure_root(ctx, category, target_key, lease)
now = get_current_timestamp()
meta = await self.get_meta(ctx, category, target_key)
@@ -250,8 +292,12 @@ class UserPrivacyConfigService:
target_key: str,
version: int,
updated_by: str = "",
+ *,
+ owner_lease_ref: Optional[dict[str, Any]] = None,
) -> UserPrivacyConfigVersion:
- async with self._config_lock(ctx, category, target_key) as lease:
+ async with self._config_lock(
+ ctx, category, target_key, owner_lease_ref=owner_lease_ref
+ ) as lease:
meta = await self.get_meta(ctx, category, target_key)
snapshot = await self.get_version(ctx, category, target_key, version)
if meta is None or snapshot is None:
diff --git a/openviking/retrieve/skill_package_retriever.py b/openviking/retrieve/skill_package_retriever.py
new file mode 100644
index 000000000..257fba6af
--- /dev/null
+++ b/openviking/retrieve/skill_package_retriever.py
@@ -0,0 +1,107 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Package search used exclusively by the Skills API."""
+
+import math
+import time
+from typing import Optional
+
+from openviking.models.embedder.base import embed_compat
+from openviking.retrieve.hierarchical_retriever import HierarchicalRetriever
+from openviking.retrieve.retrieval_stats import get_stats_collector
+from openviking.retrieve.skill_results import SkillResultResolver, candidate_key, pagination_key
+from openviking.server.identity import RequestContext
+from openviking.storage.vikingdb_manager import VikingDBManagerProxy
+from openviking.telemetry import get_current_telemetry
+from openviking_cli.retrieve.types import QueryResult, TypedQuery
+
+
+class SkillPackageRetriever(HierarchicalRetriever):
+ """Reuse hit conversion and thresholds; keep package pagination out of general search."""
+
+ async def retrieve_skills(
+ self,
+ query: TypedQuery,
+ ctx: RequestContext,
+ *,
+ skill_resolver: SkillResultResolver,
+ limit: int = 10,
+ score_threshold: Optional[float] = None,
+ score_gte: bool = False,
+ level: Optional[list[int]] = None,
+ scope_dsl=None,
+ ) -> QueryResult:
+ started = time.monotonic()
+ telemetry = get_current_telemetry()
+ proxy = VikingDBManagerProxy(self.vector_store, ctx)
+ target_dirs = query.target_directories or []
+ if limit <= 0 or not await proxy.collection_exists_bound():
+ return QueryResult(query=query, matched_contexts=[], searched_directories=target_dirs)
+
+ threshold = self._resolve_threshold(score_threshold)
+ query_vector = sparse_vector = None
+ if self.embedder:
+ with telemetry.measure("search.embed_query"):
+ embedded = await embed_compat(self.embedder, query.query, is_query=True)
+ query_vector, sparse_vector = embedded.dense_vector, embedded.sparse_vector
+
+ page_size = max(limit, self.GLOBAL_SEARCH_TOPK)
+ offset = 0
+ seen = set()
+ candidates = {}
+ matches = []
+ while True:
+ with telemetry.measure("search.vector_retrieval"):
+ page = await proxy.search_in_tenant(
+ query_vector=query_vector,
+ sparse_query_vector=sparse_vector,
+ context_type="skill",
+ target_directories=target_dirs,
+ extra_filter=scope_dsl,
+ level=level,
+ limit=page_size,
+ offset=offset,
+ )
+ telemetry.count("vector.searches", 1)
+ telemetry.count("vector.scored", len(page))
+ telemetry.count("vector.scanned", len(page))
+ keys = {pagination_key(item) for item in page}
+ if offset and keys and keys <= seen:
+ raise RuntimeError(
+ "Skill search pagination did not advance; results are incomplete"
+ )
+ seen.update(keys)
+ for item in page:
+ if item.get("context_type") != "skill" or not item.get("uri"):
+ continue
+ score = self._finite_score(item.get("_score", 0.0))
+ if not self._passes_threshold(score, threshold, score_gte):
+ continue
+ key = candidate_key(item)
+ previous = candidates.get(key)
+ if previous is None or score > previous["_final_score"]:
+ candidates[key] = {**item, "_score": score, "_final_score": score}
+ converted = await self._convert_to_matched_contexts(
+ list(candidates.values()), ctx=ctx, apply_hotness=False
+ )
+ matches = await skill_resolver.resolve(converted)
+ if len(matches) >= limit or len(page) < page_size:
+ break
+ # Every page uses the same query vectors and vector-score ordering.
+ boundary = self._finite_score(page[-1].get("_score"), default=math.nan)
+ if math.isfinite(boundary) and not self._passes_threshold(
+ boundary, threshold, score_gte
+ ):
+ break
+ offset += len(page)
+
+ final = matches[:limit]
+ telemetry.set("vector.returned", len(final))
+ get_stats_collector().record_query(
+ context_type="skill",
+ result_count=len(final),
+ scores=[item.score for item in final],
+ latency_ms=(time.monotonic() - started) * 1000,
+ rerank_used=False,
+ )
+ return QueryResult(query=query, matched_contexts=final, searched_directories=target_dirs)
diff --git a/openviking/retrieve/skill_results.py b/openviking/retrieve/skill_results.py
new file mode 100644
index 000000000..f8ed5e9e7
--- /dev/null
+++ b/openviking/retrieve/skill_results.py
@@ -0,0 +1,113 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Resolve package-level Skill results without changing other context types."""
+
+from typing import Any, Dict, List, Sequence
+
+from openviking.core.namespace import classify_uri
+from openviking.server.error_mapping import is_not_found_error
+from openviking.server.identity import RequestContext
+from openviking_cli.exceptions import PermissionDeniedError
+from openviking_cli.retrieve.types import ContextType, MatchedContext
+
+LEVEL_SUFFIXES = ("/.abstract.md", "/.overview.md")
+
+
+def skill_root_uri(uri: str) -> str:
+ """Return the enclosing Skill root, including its full owning namespace."""
+ if not isinstance(uri, str):
+ return ""
+ uri = uri.rstrip("/")
+ for suffix in LEVEL_SUFFIXES:
+ if uri.endswith(suffix):
+ uri = uri[: -len(suffix)]
+ break
+ try:
+ shape = classify_uri(uri)
+ except (TypeError, ValueError):
+ return ""
+ index = shape.content_index
+ if not shape.is_skill or index is None or len(shape.parts) <= index + 1:
+ return ""
+ if shape.parts[index + 1].startswith("."):
+ # Internal update backups retain their original vectors, but are not
+ # installed Skills. Hidden attachments inside a normal package remain valid.
+ return ""
+ return "viking://" + "/".join(shape.parts[: index + 2])
+
+
+def candidate_key(candidate: Dict[str, Any]) -> Any:
+ """Keep Skill directory levels distinct until their final scores are known."""
+ uri = candidate.get("uri", "")
+ if candidate.get("context_type") == ContextType.SKILL.value:
+ return uri, candidate.get("level", 2)
+ return uri
+
+
+def pagination_key(record: Dict[str, Any]) -> tuple:
+ """Identify a stored layer even when result merging keeps only its URI."""
+ return record.get("context_type", ""), record.get("uri", ""), record.get("level", 2)
+
+
+def merge_skill_results(matches: Sequence[MatchedContext]) -> List[MatchedContext]:
+ """Merge Skill identities only; preserve other contexts and their relative order."""
+ if not any(matched.context_type == ContextType.SKILL for matched in matches):
+ return list(matches)
+ skills: Dict[str, MatchedContext] = {}
+ others = []
+ for matched in matches:
+ if matched.context_type != ContextType.SKILL:
+ others.append(matched)
+ continue
+ root = skill_root_uri(matched.uri)
+ if not root:
+ continue
+ previous = skills.get(root)
+ if previous is None or _skill_rank(matched) < _skill_rank(previous):
+ skills[root] = matched
+ # Non-skill equal-score hits retain their original order.
+ return sorted(
+ [*others, *skills.values()],
+ key=lambda item: (
+ -item.score,
+ item.uri if item.context_type == ContextType.SKILL else "",
+ ),
+ )
+
+
+def _skill_rank(matched: MatchedContext) -> tuple:
+ return -matched.score, matched.uri, matched.level
+
+
+class SkillResultResolver:
+ """Keep each readable package's highest-scored hit, without rewriting it."""
+
+ def __init__(self, fs: Any, ctx: RequestContext):
+ self.fs = fs
+ self.ctx = ctx
+ self._root_access: Dict[str, bool] = {}
+
+ async def _root_readable(self, root: str) -> bool:
+ if root not in self._root_access:
+ try:
+ # stat() checks read permission and existence; skip_count avoids
+ # an unnecessary vector-store count query.
+ info = await self.fs.stat(root, ctx=self.ctx, skip_count=True)
+ self._root_access[root] = bool(info and info.get("isDir", False))
+ except Exception as exc:
+ if isinstance(exc, PermissionDeniedError) or is_not_found_error(exc):
+ self._root_access[root] = False
+ else:
+ raise
+ return self._root_access[root]
+
+ async def resolve(self, matches: Sequence[MatchedContext]) -> List[MatchedContext]:
+ resolved = []
+ for matched in merge_skill_results(matches):
+ if matched.context_type != ContextType.SKILL:
+ resolved.append(matched)
+ continue
+ root = skill_root_uri(matched.uri)
+ if await self._root_readable(root):
+ resolved.append(matched)
+ return resolved
diff --git a/openviking/server/routers/resources.py b/openviking/server/routers/resources.py
index 880484a4f..23aa74188 100644
--- a/openviking/server/routers/resources.py
+++ b/openviking/server/routers/resources.py
@@ -18,7 +18,6 @@ from openviking.server.identity import RequestContext
from openviking.server.local_input_guard import require_remote_resource_source
from openviking.server.resource_ingest import ingest_temp_upload
from openviking.server.responses import response_from_result
-from openviking.server.skill_source_metadata import persist_skill_source_metadata
from openviking.server.telemetry import run_operation
from openviking.server.temp_upload_store import TempUploadStore
from openviking.service.skill_sources import describe_skill_sources, resolve_skill_source
@@ -402,8 +401,8 @@ async def add_skill(
allow_local_path_resolution=isinstance(skill_data, Path),
source_path_hint=source_path_hint,
target_uri=target_uri,
+ source_metadata=skill_source,
)
- await persist_skill_source_metadata(service, _ctx, result, skill_source)
return result
# Each skill owns its own queue wait tracker and task ID.
diff --git a/openviking/server/routers/skills.py b/openviking/server/routers/skills.py
index 9b3d593cf..ce2a8d94e 100644
--- a/openviking/server/routers/skills.py
+++ b/openviking/server/routers/skills.py
@@ -21,24 +21,37 @@ from openviking.core.path_variables import resolve_path_variables
from openviking.core.skill_loader import validate_skill_format
from openviking.core.uri_validation import validate_request_viking_uri
from openviking.models.embedder.base import query_embed_cache_scope
+from openviking.privacy.helpers import version_filename
from openviking.privacy.service import UserPrivacyConfigVersion
+from openviking.retrieve.skill_results import skill_root_uri
from openviking.server.auth import get_request_context
from openviking.server.dependencies import get_service
from openviking.server.identity import RequestContext
from openviking.server.models import Response
+from openviking.server.skill_package_update import (
+ clear_skill_package_contents,
+ transfer_skill_package,
+)
from openviking.server.skill_source_metadata import (
SOURCE_METADATA_FILENAME,
- persist_skill_source_metadata,
read_skill_source_metadata,
)
from openviking.server.telemetry import run_operation
from openviking.server.temp_upload_store import TempUploadStore
from openviking.service.skill_sources import resolve_skill_source
+from openviking.service.task_tracker_concurrency import run_to_completion
from openviking.telemetry import TelemetryRequest
from openviking.utils.skill_processor import validate_skill_name
-from openviking_cli.exceptions import InvalidArgumentError, NotFoundError, ResourceExhaustedError
+from openviking_cli.exceptions import (
+ InternalError,
+ InvalidArgumentError,
+ NotFoundError,
+ ResourceExhaustedError,
+)
+from openviking_cli.utils import get_logger
router = APIRouter(prefix="/api/v1/skills", tags=["skills"])
+logger = get_logger(__name__)
_SKILL_INTEGRITY_MAX_ENTRIES = 512
_SKILL_INTEGRITY_MAX_FILE_BYTES = 16 * 1024 * 1024
@@ -233,13 +246,15 @@ def _skill_summary_from_entry(entry: Dict[str, Any]) -> Dict[str, Any]:
return _skill_summary_from_meta(name, root_uri, _parse_abstract_meta(entry.get("abstract", "")))
-def _skill_summary_from_hit(hit: Dict[str, Any]) -> Dict[str, Any]:
+async def _skill_summary_from_hit(
+ hit: Dict[str, Any], service, ctx: RequestContext
+) -> Dict[str, Any]:
hit_uri = hit.get("uri", "")
root_uri = _skill_root_from_hit_uri(hit_uri)
- name = _skill_name_from_uri(root_uri) if root_uri else _skill_name_from_uri(hit_uri)
- summary = _skill_summary_from_meta(
- name, root_uri or hit_uri, _parse_abstract_meta(hit.get("abstract", ""))
- )
+ meta = _parse_abstract_meta(await service.fs.abstract(root_uri, ctx=ctx))
+ name = meta.get("name") or _skill_name_from_uri(root_uri)
+ summary = _skill_summary_from_meta(name, root_uri, meta)
+ summary["uri"] = hit_uri
summary["score"] = hit.get("score", 0.0)
summary["match_reason"] = hit.get("match_reason", "")
summary["level"] = hit.get("level", 0)
@@ -248,22 +263,8 @@ def _skill_summary_from_hit(hit: Dict[str, Any]) -> Dict[str, Any]:
def _skill_root_from_hit_uri(hit_uri: str) -> str:
- """Strip a trailing chunk filename (e.g. ``.abstract.md``) from a hit URI.
-
- Search results point at the indexed chunk file (typically ``.abstract.md``
- sitting alongside ``SKILL.md`` inside the skill directory). The summary
- consumed by the CLI expects the URI to identify the skill directory itself,
- so we trim a single trailing filename when one is present.
- """
- if not hit_uri:
- return ""
- trimmed = hit_uri.rstrip("/")
- last_segment = trimmed.rsplit("/", 1)[-1]
- if "." in last_segment:
- parent = trimmed.rsplit("/", 1)[0]
- if parent:
- return parent
- return trimmed
+ """Resolve the package root even when a hit is several directories deep."""
+ return skill_root_uri(hit_uri) or hit_uri.rstrip("/")
async def _require_skill(
@@ -517,12 +518,30 @@ async def _restore_skill_privacy(
ctx: RequestContext,
skill_name: str,
previous_privacy: Optional[UserPrivacyConfigVersion],
+ deleted_snapshot: Optional[Dict[str, bytes]] = None,
+ owner_lease_ref: Optional[Dict[str, Any]] = None,
) -> None:
privacy = service.privacy_configs
if privacy is None:
return
+ if deleted_snapshot is not None:
+ # Empty configuration deletes the whole history. Restore the saved
+ # files as well as current values if the package update then fails.
+ viking_fs = service.fs._ensure_initialized() # noqa: SLF001
+ root = privacy.get_config_root(ctx, "skill", skill_name)
+ async with privacy._config_lock( # noqa: SLF001
+ ctx, "skill", skill_name, owner_lease_ref=owner_lease_ref
+ ) as lease:
+ # Keep the config root: removing it would also remove the tree
+ # lock and allow a successful concurrent save to be overwritten.
+ await privacy._clear_contents(ctx, "skill", skill_name, lease) # noqa: SLF001
+ for path, content in deleted_snapshot.items():
+ await viking_fs.write_file_bytes(
+ f"{root}/{path}", content, ctx=ctx, lease_ref=lease
+ )
+ return
if previous_privacy is None:
- await privacy.delete(ctx, "skill", skill_name)
+ await privacy.delete(ctx, "skill", skill_name, owner_lease_ref=owner_lease_ref)
return
await privacy.activate_version(
ctx,
@@ -530,6 +549,7 @@ async def _restore_skill_privacy(
skill_name,
previous_privacy.version,
updated_by=ctx.user.user_id,
+ owner_lease_ref=owner_lease_ref,
)
@@ -584,7 +604,7 @@ async def find_skills(
execution = await run_operation(
operation="skills.find",
telemetry=request.telemetry,
- fn=lambda: service.search.find(
+ fn=lambda: service.search.find_skills(
query=request.query,
ctx=_ctx,
target_uri=resolved_uri,
@@ -595,7 +615,10 @@ async def find_skills(
)
result = execution.result
result_dict = result.to_dict() if hasattr(result, "to_dict") else dict(result or {})
- hits = [_skill_summary_from_hit(hit) for hit in result_dict.get("skills", [])]
+ hits = [
+ await _skill_summary_from_hit(hit, service, _ctx)
+ for hit in result_dict.get("skills", [])
+ ]
return Response(
status="ok",
result={"root_uri": resolved_uri, "skills": hits, "total": len(hits)},
@@ -612,7 +635,7 @@ async def find_skills(
run_operation(
operation="skills.find",
telemetry=request.telemetry,
- fn=lambda: service.search.find(
+ fn=lambda: service.search.find_skills(
query=request.query,
ctx=_ctx,
target_uri=user_root,
@@ -624,7 +647,7 @@ async def find_skills(
run_operation(
operation="skills.find",
telemetry=request.telemetry,
- fn=lambda: service.search.find(
+ fn=lambda: service.search.find_skills(
query=request.query,
ctx=_ctx,
target_uri=agent_root,
@@ -639,18 +662,23 @@ async def find_skills(
user_result_dict = (
user_result.to_dict() if hasattr(user_result, "to_dict") else dict(user_result or {})
)
- user_hits = [_skill_summary_from_hit(hit) for hit in user_result_dict.get("skills", [])]
+ user_hits = [
+ await _skill_summary_from_hit(hit, service, _ctx)
+ for hit in user_result_dict.get("skills", [])
+ ]
agent_result = agent_execution.result
agent_result_dict = (
agent_result.to_dict() if hasattr(agent_result, "to_dict") else dict(agent_result or {})
)
- agent_hits = [_skill_summary_from_hit(hit) for hit in agent_result_dict.get("skills", [])]
+ agent_hits = [
+ await _skill_summary_from_hit(hit, service, _ctx)
+ for hit in agent_result_dict.get("skills", [])
+ ]
merged_hits = [*user_hits, *agent_hits]
- # Sort merged hits by score descending if available
- if merged_hits and "score" in merged_hits[0]:
- merged_hits.sort(key=lambda x: x.get("score", 0), reverse=True)
+ merged_hits.sort(key=lambda hit: (-hit.get("score", 0), hit["root_uri"]))
+ merged_hits = merged_hits[: request.limit]
return Response(
status="ok",
@@ -771,9 +799,125 @@ async def update_skill(
previous_privacy = None
preparation = None
privacy = service.privacy_configs
- try:
+ viking_fs = service.fs._ensure_initialized() # noqa: SLF001
+ update_lease = None
+ privacy_lease = None
+ task_id = str(uuid.uuid4())
+ deleted_privacy_snapshot = None
+ cleanup_warnings = []
+
+ def report_cleanup_failure(location: str, exc: Exception) -> None:
+ warning = f"Skill update cleanup failed at {location}: {exc}"
+ cleanup_warnings.append(warning)
+ logger.warning(warning, exc_info=True)
+
+ async def discard_backup() -> None:
+ # Called only after commit or successful restoration. Failure to
+ # discard this now-unused copy must not change that outcome.
+ if backup_created:
+ try:
+ await viking_fs.rm(backup_uri, ctx=_ctx, recursive=True, lease_ref=update_lease)
+ except Exception as exc:
+ report_cleanup_failure(backup_uri, exc)
+
+ async def back_up_package() -> None:
+ nonlocal update_lease, privacy_lease, backup_created, previous_privacy
+ update_lease = await viking_fs._async_agfs.pathlock_acquire_tree_batch(
+ [viking_fs._uri_to_path(uri, ctx=_ctx) for uri in (root_uri, backup_uri)]
+ )
+ # Tree acquisition may recreate a missing directory for its token.
+ # Recheck the installed definition and snapshot privacy under the lock.
+ if not await viking_fs.exists(f"{root_uri}/SKILL.md", ctx=_ctx):
+ raise NotFoundError(root_uri, "skill")
if privacy is not None:
+ # Always acquire package then config. Keep config ownership
+ # through commit/rollback so a separate successful save cannot
+ # be silently replaced by restoration of this request's state.
+ privacy_lease = await viking_fs._async_agfs.pathlock_acquire_tree(
+ viking_fs._uri_to_path(
+ privacy.get_config_root(_ctx, "skill", skill_name), ctx=_ctx
+ ),
+ timeout_secs=30.0,
+ )
previous_privacy = await privacy.get_current(_ctx, "skill", skill_name)
+ try:
+ await transfer_skill_package(
+ viking_fs, root_uri, backup_uri, ctx=_ctx, lease_ref=update_lease
+ )
+ except Exception as exc:
+ raise InternalError(
+ f"Skill backup failed; backup location: {backup_uri}", cause=exc
+ ) from exc
+ backup_created = True
+ await clear_skill_package_contents(
+ viking_fs, root_uri, ctx=_ctx, lease_ref=update_lease
+ )
+
+ async def restore_package() -> None:
+ # Do not restore while either queued or active work can write back.
+ await service.resources.cancel_skill_processing(task_id, _ctx)
+ failures = []
+ if backup_created:
+ try:
+ await clear_skill_package_contents(
+ viking_fs, root_uri, ctx=_ctx, lease_ref=update_lease
+ )
+ await transfer_skill_package(
+ viking_fs, backup_uri, root_uri, ctx=_ctx, lease_ref=update_lease
+ )
+ except Exception as exc:
+ failures.append(f"package: {exc}")
+ if privacy_update_attempted:
+ try:
+ await _restore_skill_privacy(
+ service,
+ _ctx,
+ skill_name,
+ previous_privacy,
+ deleted_privacy_snapshot,
+ owner_lease_ref=privacy_lease,
+ )
+ except Exception as exc:
+ failures.append(f"privacy: {exc}")
+ if failures:
+ raise InternalError(
+ f"Skill update rollback failed; backup location: {backup_uri}; "
+ + "; ".join(failures)
+ )
+ await discard_backup()
+
+ async def release_update_locks() -> None:
+ try:
+ if privacy_lease is not None:
+ try:
+ # Deletion inside the update preserved the locked
+ # directory. Remove an empty config only now, after
+ # all possible config writes and rollback have ended.
+ if (
+ await privacy.get_meta(_ctx, "skill", skill_name) is None
+ and await privacy.get_current(_ctx, "skill", skill_name) is None
+ and not await privacy.list_versions(_ctx, "skill", skill_name)
+ ):
+ await viking_fs.rm(
+ privacy.get_config_root(_ctx, "skill", skill_name),
+ recursive=True,
+ ctx=_ctx,
+ lease_ref=privacy_lease,
+ )
+ except Exception as exc:
+ # Config data has already been committed/restored.
+ # Retain the result (or original error), but still
+ # release both locks and report the leftover directory.
+ report_cleanup_failure(
+ privacy.get_config_root(_ctx, "skill", skill_name), exc
+ )
+ finally:
+ await viking_fs._async_agfs.pathlock_release(privacy_lease)
+ finally:
+ if update_lease is not None:
+ await viking_fs._async_agfs.pathlock_release(update_lease)
+
+ try:
async with resolve_skill_source(
data,
allow_local_path_resolution=allow_local_path_resolution,
@@ -802,8 +946,35 @@ async def update_skill(
"actual": preparation.skill_dict.get("name"),
},
)
- await service.fs.mv(root_uri, backup_uri, ctx=_ctx)
- backup_created = True
+ await run_to_completion(back_up_package)
+ # All synchronous configuration changes must finish before the
+ # package can produce background summary or embedding work.
+ if (
+ privacy is not None
+ and previous_privacy is not None
+ and not preparation.privacy_values
+ ):
+ privacy_root = privacy.get_config_root(_ctx, "skill", skill_name)
+ privacy_paths = [".meta.json", "current.json"]
+ privacy_paths.extend(
+ f"history/{version_filename(version)}"
+ for version in await privacy.list_versions(_ctx, "skill", skill_name)
+ )
+ deleted_privacy_snapshot = {
+ path: await viking_fs.read_file_bytes(f"{privacy_root}/{path}", ctx=_ctx)
+ for path in privacy_paths
+ }
+ privacy_update_attempted = True
+ await run_to_completion(
+ lambda: service.resources._skill_processor.apply_skill_privacy( # noqa: SLF001
+ preparation.skill_dict,
+ preparation.privacy_values,
+ _ctx,
+ change_reason="auto-extracted from update_skill",
+ delete_if_empty=True,
+ owner_lease_ref=privacy_lease,
+ )
+ )
result = await service.resources.add_skill(
data=preparation,
ctx=_ctx,
@@ -814,42 +985,35 @@ async def update_skill(
apply_privacy=False,
privacy_change_reason="auto-extracted from update_skill",
target_uri=skill_root_parent,
+ source_metadata=skill_source,
+ task_id=task_id,
+ owner_lease_ref=update_lease,
)
- await persist_skill_source_metadata(service, _ctx, result, skill_source)
- privacy_update_attempted = True
- await service.resources._skill_processor.apply_skill_privacy( # noqa: SLF001
- preparation.skill_dict,
- preparation.privacy_values,
- _ctx,
- change_reason="auto-extracted from update_skill",
- delete_if_empty=True,
- )
- except Exception:
- if backup_created:
- try:
- await service.fs.rm(root_uri, ctx=_ctx, recursive=True)
- except Exception:
- pass
- try:
- await service.fs.mv(backup_uri, root_uri, ctx=_ctx)
- except Exception:
- pass
- if privacy_update_attempted:
- try:
- await _restore_skill_privacy(service, _ctx, skill_name, previous_privacy)
- except Exception:
- pass
+ except BaseException as update_error:
+ try:
+ await run_to_completion(restore_package)
+ except Exception as rollback_error:
+ raise InternalError(
+ "Skill update failed and rollback could not complete", cause=rollback_error
+ ) from update_error
raise
else:
- if backup_created:
- await service.fs.rm(backup_uri, ctx=_ctx, recursive=True)
+ await run_to_completion(discard_backup)
result["action"] = "update"
- return result
finally:
- if preparation and preparation.cleanup_path:
- shutil.rmtree(preparation.cleanup_path, ignore_errors=True)
- if resolved:
- await resolved.cleanup()
+ try:
+ await run_to_completion(release_update_locks)
+ finally:
+ if preparation and preparation.cleanup_path:
+ shutil.rmtree(preparation.cleanup_path, ignore_errors=True)
+ if resolved:
+ try:
+ await run_to_completion(resolved.cleanup)
+ except Exception as exc:
+ report_cleanup_failure(resolved.local_path, exc)
+ if cleanup_warnings:
+ result.setdefault("warnings", []).extend(cleanup_warnings)
+ return result
execution = await run_operation(
operation="skills.update",
diff --git a/openviking/server/skill_package_update.py b/openviking/server/skill_package_update.py
new file mode 100644
index 000000000..1dc9129e2
--- /dev/null
+++ b/openviking/server/skill_package_update.py
@@ -0,0 +1,76 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Skill update transfers that never remove the directory holding the package lock."""
+
+from typing import Any, Dict
+
+from openviking.server.identity import RequestContext
+from openviking.storage.internal_names import (
+ MULTIWRITE_EXACT_LOCK_FILE_PREFIX,
+ MULTIWRITE_INTERNAL_FILE_NAMES,
+)
+from openviking.storage.viking_fs import VikingFS
+
+
+async def transfer_skill_package(
+ viking_fs: VikingFS,
+ source_uri: str,
+ target_uri: str,
+ *,
+ ctx: RequestContext,
+ lease_ref: Dict[str, Any],
+) -> None:
+ """Copy files and move existing vectors, leaving both locked roots intact.
+
+ The caller owns both tree locks and decides when the source can be cleared.
+ Do not use cp/mv here: their failure cleanup can delete the target root,
+ which during rollback is the directory holding the live package lock.
+ """
+ source_uris = await viking_fs._prepare_transfer_entries(
+ source_uri, target_uri, is_dir=True, move=True, ctx=ctx
+ )
+ await viking_fs._copy_directory_under_tree_locks(
+ viking_fs._uri_to_path(source_uri, ctx=ctx),
+ viking_fs._uri_to_path(target_uri, ctx=ctx),
+ old_uri=source_uri,
+ new_uri=target_uri,
+ ctx=ctx,
+ lease_ref=lease_ref,
+ )
+ # Move preserves the existing vector payloads and compensates on failure.
+ # Keep the source files available even if index transfer/restoration fails.
+ await viking_fs._update_vector_store_uris(
+ source_uri, target_uri, recursive=True, ctx=ctx, source_uris=source_uris
+ )
+
+
+async def clear_skill_package_contents(
+ viking_fs: VikingFS,
+ root_uri: str,
+ *,
+ ctx: RequestContext,
+ lease_ref: Dict[str, Any],
+) -> None:
+ """Remove package entries and root vectors, keeping the root and its lock."""
+ root_path = viking_fs._uri_to_path(root_uri, ctx=ctx)
+ # Storage listing includes user-hidden files that the public Skill listing
+ # omits. Runtime control entries must remain owned by the storage layer.
+ entries = await viking_fs._async_agfs.ls(
+ root_path, fs_ctx=viking_fs._pathlock_fs_ctx(ctx, lease_ref)
+ )
+ for entry in entries:
+ name = entry.get("name", "")
+ if (
+ not name
+ or name in {".", ".."}
+ or name in MULTIWRITE_INTERNAL_FILE_NAMES
+ or name.startswith(MULTIWRITE_EXACT_LOCK_FILE_PREFIX)
+ ):
+ continue
+ await viking_fs.rm(
+ f"{root_uri}/{name}",
+ recursive=bool(entry.get("isDir", False)),
+ ctx=ctx,
+ lease_ref=lease_ref,
+ )
+ await viking_fs._delete_from_vector_store([root_uri], ctx=ctx)
diff --git a/openviking/server/skill_source_metadata.py b/openviking/server/skill_source_metadata.py
index 62afc6b11..76ece1a3d 100644
--- a/openviking/server/skill_source_metadata.py
+++ b/openviking/server/skill_source_metadata.py
@@ -14,6 +14,37 @@ def skill_source_metadata_uri(root_uri: str) -> str:
return f"{root_uri.rstrip('/')}/{SOURCE_METADATA_FILENAME}"
+def _source_record(
+ result: Dict[str, Any], source: Optional[Dict[str, Any]]
+) -> Optional[tuple[str, str]]:
+ root_uri = result.get("root_uri") or result.get("uri")
+ if not source or not root_uri:
+ return None
+ record = dict(source)
+ skill_name = result.get("name") or record.get("skill_name")
+ if skill_name:
+ record["skill_name"] = skill_name
+ return (
+ skill_source_metadata_uri(root_uri),
+ json.dumps(record, ensure_ascii=False, indent=2, sort_keys=True),
+ )
+
+
+async def write_skill_source_metadata(
+ viking_fs,
+ ctx: RequestContext,
+ result: Dict[str, Any],
+ source: Optional[Dict[str, Any]],
+ *,
+ lease_ref: Optional[Dict[str, Any]] = None,
+) -> None:
+ """Save the source before handing the package lock to background processing."""
+ record = _source_record(result, source)
+ if record is not None:
+ uri, content = record
+ await viking_fs.write_file(uri, content, ctx=ctx, lease_ref=lease_ref)
+
+
async def read_skill_source_metadata(
service,
ctx: RequestContext,
@@ -44,34 +75,3 @@ async def read_skill_source_metadata(
metadata["tracked"] = True
metadata["metadata_uri"] = uri
return metadata
-
-
-async def persist_skill_source_metadata(
- service,
- ctx: RequestContext,
- result: Dict[str, Any],
- source: Optional[Dict[str, Any]],
-) -> None:
- if not source:
- return
-
- root_uri = result.get("root_uri") or result.get("uri")
- if not root_uri:
- return
-
- record = dict(source)
- skill_name = result.get("name") or record.get("skill_name")
- if skill_name:
- record["skill_name"] = skill_name
-
- uri = skill_source_metadata_uri(root_uri)
- content = json.dumps(record, ensure_ascii=False, indent=2, sort_keys=True)
- viking_fs = getattr(service, "viking_fs", None)
- if viking_fs is not None:
- await viking_fs.write(uri, content, ctx=ctx)
- return
-
- try:
- await service.fs.write(uri=uri, content=content, ctx=ctx, mode="replace", wait=True)
- except Exception:
- await service.fs.write(uri=uri, content=content, ctx=ctx, mode="create", wait=True)
diff --git a/openviking/service/fs_service.py b/openviking/service/fs_service.py
index 3a507b2aa..c81d47a85 100644
--- a/openviking/service/fs_service.py
+++ b/openviking/service/fs_service.py
@@ -478,9 +478,19 @@ class FSService:
@staticmethod
def _semantic_refresh_parent_uri(uri: str, context_type: str) -> Optional[str]:
- if context_type != "resource":
+ if context_type not in {"resource", "skill"}:
return None
parent = VikingURI(uri).parent
+ if context_type == "skill":
+ if parent is None:
+ return None
+ classification = classify_uri(parent.uri)
+ if (
+ not classification.is_skill
+ or classification.is_skill_root
+ or classification.is_skill_namespace
+ ):
+ return None
return parent.uri if parent and parent.scope else None
@staticmethod
diff --git a/openviking/service/reindex_executor.py b/openviking/service/reindex_executor.py
index 9bbebf993..b7717d801 100644
--- a/openviking/service/reindex_executor.py
+++ b/openviking/service/reindex_executor.py
@@ -34,6 +34,7 @@ from openviking.storage.abstract_overview import body_for_preview, embedding_tex
from openviking.storage.errors import ResourceBusyError
from openviking.storage.expr import And, Eq, Or, PathScope
from openviking.storage.queuefs.embedding_msg_converter import EmbeddingMsgConverter
+from openviking.storage.queuefs.semantic_dag import SemanticDagExecutor
from openviking.storage.queuefs.semantic_msg import SemanticMsg
from openviking.storage.queuefs.semantic_processor import SemanticProcessor
from openviking.storage.viking_fs import get_viking_fs
@@ -44,9 +45,10 @@ from openviking.utils.embedding_utils import (
_apply_ingest_options,
_truncate_abstract_bytes,
get_resource_content_type,
+ vectorize_directory_meta,
+ vectorize_file,
)
from openviking.utils.ingest_options import IngestOptions
-from openviking.utils.skill_processor import SkillProcessor
from openviking_cli.exceptions import InvalidArgumentError, NotFoundError, OpenVikingError
from openviking_cli.session.user_id import UserIdentifier
from openviking_cli.utils import VikingURI, get_logger
@@ -984,13 +986,46 @@ class ReindexExecutor:
counters = run.counters
ctx = run.ctx
if mode == "semantic_and_vectors":
- await self._regenerate_skill_semantics(uri=uri, ctx=ctx)
- vector_kwargs = {"uri": uri, "counters": counters, "ctx": ctx}
+ processor = SemanticProcessor(
+ max_concurrent_llm=get_openviking_config().vlm.max_concurrent
+ )
if not recursive:
- vector_kwargs["recursive"] = False
- await self._reindex_skill_vectors(
- **self._with_ingest_options(vector_kwargs, run.ingest_options)
+ await self._regenerate_skill_semantics(uri=uri, ctx=ctx, lock=run.lock)
+ await self._reindex_skill_vectors(
+ uri=uri,
+ counters=counters,
+ ctx=ctx,
+ recursive=False,
+ ingest_options=run.ingest_options,
+ )
+ return
+ executor = SemanticDagExecutor(
+ processor=processor,
+ context_type="skill",
+ max_concurrent_llm=processor.max_concurrent_llm,
+ ctx=self._content_owner_ctx(uri, ctx),
+ lock=run.lock,
+ generation_trigger="reindex",
+ ingest_options=run.ingest_options,
+ source={
+ "path": (await self._skill_meta(uri=uri, abstract="", ctx=ctx)).get(
+ "source_path", ""
+ )
+ },
)
+ try:
+ await executor.run(uri)
+ finally:
+ # Even a semantic error must settle already submitted Skill
+ # vectors before the caller releases its package lease.
+ await get_request_wait_tracker().wait_for_embeddings(
+ get_current_telemetry().telemetry_id
+ )
+ stats = executor.get_stats()
+ counters.scanned_records += stats.total_nodes
+ counters.rebuilt_records += stats.indexed_records
+ counters.failed_records += len(stats.failures)
+ counters.warnings.extend(stats.failures)
return
await self._reindex_skill_vectors(
**self._with_ingest_options(
@@ -1486,91 +1521,92 @@ class ReindexExecutor:
ingest_options: IngestOptions | None = None,
) -> None:
viking_fs = get_viking_fs()
- counters.scanned_records += 1
-
- abstract = await self._read_directory_abstract(uri, ctx=ctx)
- overview = await self._read_directory_overview(uri, ctx=ctx)
- if not abstract:
- record = await self._fetch_existing_record(
- uri=uri,
- level=0,
- ctx=self._content_owner_ctx(uri, ctx),
- )
- abstract = self._record_abstract(record)
- if not overview:
- record = await self._fetch_existing_record(
- uri=uri,
- level=1,
- ctx=self._content_owner_ctx(uri, ctx),
- )
- overview = self._record_abstract(record) or abstract
-
- if not abstract and not overview:
- counters.unsupported_records += 1
- counters.warnings.append(f"No semantic source found for {uri}")
- return
-
- parent_uri = VikingURI(uri).parent.uri
- if abstract:
- try:
- await self._upsert_context(
- uri=uri,
- parent_uri=parent_uri,
- abstract=abstract,
- vector_text=embedding_text_for_body(ContextLevel.ABSTRACT, uri, abstract),
- is_leaf=False,
- context_type=ContextType.SKILL.value,
- level=ContextLevel.ABSTRACT,
- meta=await self._skill_meta(uri=uri, abstract=abstract, ctx=ctx),
- ctx=ctx,
- ingest_options=ingest_options,
- )
- counters.rebuilt_records += 1
- except Exception as exc:
- counters.failed_records += 1
- counters.warnings.append(f"Failed to reindex {uri} L0 vector: {exc}")
- if overview:
- try:
- await self._upsert_context(
- uri=uri,
- parent_uri=parent_uri,
- # L1 abstract scalar carries the overview for Rerank.
- abstract=_truncate_abstract_bytes(overview),
- vector_text=embedding_text_for_body(ContextLevel.OVERVIEW, uri, overview),
- is_leaf=False,
- context_type=ContextType.SKILL.value,
- level=ContextLevel.OVERVIEW,
- meta=await self._skill_meta(uri=uri, abstract=abstract, ctx=ctx),
- ctx=ctx,
- ingest_options=ingest_options,
+ owner_ctx = self._content_owner_ctx(uri, ctx)
+ pending = [uri]
+ while pending:
+ directory_uri = pending.pop()
+ counters.scanned_records += 1
+ abstract = await self._read_directory_abstract(directory_uri, ctx=ctx)
+ overview = await self._read_directory_overview(directory_uri, ctx=ctx)
+ for level in (0, 1):
+ if abstract if level == 0 else overview:
+ continue
+ record = await self._fetch_existing_record(
+ uri=directory_uri, level=level, ctx=owner_ctx
)
- counters.rebuilt_records += 1
- except Exception as exc:
- counters.failed_records += 1
- counters.warnings.append(f"Failed to reindex {uri} L1 vector: {exc}")
+ if level == 0:
+ abstract = self._record_abstract(record)
+ else:
+ overview = self._record_abstract(record)
+ if abstract or overview:
+ try:
+ await vectorize_directory_meta(
+ directory_uri,
+ abstract,
+ overview,
+ context_type="skill",
+ ctx=owner_ctx,
+ include_abstract=bool(abstract),
+ include_overview=bool(overview),
+ ingest_options=ingest_options,
+ content_is_body=True,
+ **(
+ {
+ "meta": await self._skill_meta(
+ uri=directory_uri, abstract=abstract, ctx=ctx
+ )
+ }
+ if classify_uri(directory_uri).is_skill_root
+ else {}
+ ),
+ )
+ counters.rebuilt_records += int(bool(abstract)) + int(bool(overview))
+ except Exception as exc:
+ counters.failed_records += 1
+ counters.warnings.append(f"Failed to reindex {directory_uri}: {exc}")
+ else:
+ counters.unsupported_records += 1
+ counters.warnings.append(f"No semantic source found for {directory_uri}")
- skill_file_uri = f"{uri}/SKILL.md"
- if recursive and await viking_fs.exists(skill_file_uri, ctx=ctx):
- counters.scanned_records += 1
- skill_content = await self._safe_read_text(skill_file_uri, ctx=ctx)
- if skill_content:
- detail_abstract = self._prefer_non_empty(abstract, skill_content)
+ if not recursive:
+ continue
+ from openviking.storage.queuefs.semantic_dag import _SKIP_FILENAMES
+ from openviking.storage.viking_fs import LS_ALL_NODES
+
+ entries = await viking_fs.ls(directory_uri, node_limit=LS_ALL_NODES, ctx=ctx)
+ for entry in entries:
+ name = entry.get("name", "")
+ if not name or name.startswith(".") or name in _SKIP_FILENAMES:
+ continue
+ child_uri = VikingURI(directory_uri).join(name).uri
+ if entry.get("isDir", False):
+ pending.append(child_uri)
+ continue
+ counters.scanned_records += 1
try:
- await self._upsert_context(
- uri=skill_file_uri,
- parent_uri=uri,
- abstract=detail_abstract,
- vector_text=skill_content,
- is_leaf=True,
- context_type=ContextType.SKILL.value,
- level=ContextLevel.DETAIL,
- ctx=ctx,
+ record = await self._fetch_existing_record(
+ uri=child_uri, level=2, ctx=owner_ctx
+ )
+ summary = self._record_abstract(record)
+ if not summary:
+ summary = await self._best_file_summary(child_uri, ctx=ctx)
+ enqueued = await vectorize_file(
+ file_path=child_uri,
+ summary_dict={"name": name, "summary": summary},
+ parent_uri=directory_uri,
+ context_type="skill",
+ ctx=owner_ctx,
+ preserve_existing_created_at=True,
ingest_options=ingest_options,
)
- counters.rebuilt_records += 1
+ if enqueued:
+ counters.rebuilt_records += 1
+ else:
+ counters.unsupported_records += 1
+ counters.warnings.append(f"No vector source found for {child_uri}")
except Exception as exc:
counters.failed_records += 1
- counters.warnings.append(f"Failed to reindex {skill_file_uri} vector: {exc}")
+ counters.warnings.append(f"Failed to reindex {child_uri}: {exc}")
async def _reindex_memory_vectors(
self,
@@ -1749,38 +1785,10 @@ class ReindexExecutor:
counters.failed_records += 1
counters.warnings.append(f"Failed to reindex {directory_uri} L1 vector: {exc}")
- async def _regenerate_skill_semantics(self, *, uri: str, ctx: RequestContext) -> None:
- service = get_service()
- if service.viking_fs is None or service.vikingdb_manager is None:
- raise RuntimeError("OpenVikingService not initialized")
-
- viking_fs = service.viking_fs
- skill_file_uri = f"{uri}/SKILL.md"
- skill_content = await self._safe_read_text(skill_file_uri, ctx=ctx)
- if not skill_content:
- raise OpenVikingError(
- f"SKILL.md not found for {uri}",
- code="NOT_FOUND",
- details={"uri": uri},
- )
-
- skill_dict, _, _, _ = SkillProcessor(service.vikingdb_manager)._parse_skill(
- skill_content,
- allow_local_path_resolution=False,
- )
- overview = await SkillProcessor(service.vikingdb_manager)._generate_overview(
- skill_dict,
- get_openviking_config(),
- )
- await viking_fs.write_context(
- uri=uri,
- content=skill_content,
- abstract=skill_dict.get("description", ""),
- overview=overview,
- content_filename="SKILL.md",
- is_leaf=False,
- ctx=ctx,
- )
+ async def _regenerate_skill_semantics(
+ self, *, uri: str, ctx: RequestContext, lock: dict | None = None
+ ) -> None:
+ await SemanticProcessor()._skill_root_semantics(uri, ctx=ctx, regenerate=True, lock=lock)
async def _read_directory_abstract(self, uri: str, *, ctx: RequestContext) -> str:
try:
@@ -1918,8 +1926,38 @@ class ReindexExecutor:
abstract: str,
ctx: RequestContext,
) -> dict[str, Any]:
- name = uri.rstrip("/").split("/")[-1]
- return {"name": name, "description": abstract}
+ import yaml
+
+ body = body_for_preview(abstract)
+ try:
+ parsed = yaml.safe_load(body)
+ except yaml.YAMLError:
+ parsed = None
+ record = await self._fetch_existing_record(
+ uri=uri, level=0, ctx=self._content_owner_ctx(uri, ctx)
+ )
+ previous_meta = (record or {}).get("meta") or {}
+ source_path = previous_meta.get("source_path", "")
+ if (
+ isinstance(parsed, dict)
+ and isinstance(parsed.get("name"), str)
+ and isinstance(parsed.get("description"), str)
+ ):
+ return {
+ "source_path": source_path,
+ **{
+ key: parsed.get(key, [] if key in {"tags", "allowed_tools"} else "")
+ for key in ("name", "description", "tags", "allowed_tools")
+ },
+ }
+ return {
+ **{
+ key: previous_meta[key] for key in ("tags", "allowed_tools") if key in previous_meta
+ },
+ "name": previous_meta.get("name") or uri.rstrip("/").split("/")[-1],
+ "description": body,
+ "source_path": source_path,
+ }
def _record_abstract(self, record: Optional[dict[str, Any]]) -> str:
if not record:
diff --git a/openviking/service/resource_service.py b/openviking/service/resource_service.py
index df5e80a90..c479d4f6c 100644
--- a/openviking/service/resource_service.py
+++ b/openviking/service/resource_service.py
@@ -84,6 +84,7 @@ from openviking_cli.exceptions import (
InternalError,
InvalidArgumentError,
NotInitializedError,
+ OpenVikingError,
)
from openviking_cli.utils import get_logger
@@ -2313,6 +2314,9 @@ class ResourceService:
apply_privacy: bool = True,
privacy_change_reason: str = "auto-extracted from add_skill",
target_uri: Optional[str] = None,
+ source_metadata: Optional[Dict[str, Any]] = None,
+ task_id: Optional[str] = None,
+ owner_lease_ref: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
"""Add skill to OpenViking.
@@ -2337,29 +2341,70 @@ class ResourceService:
telemetry_id = get_current_telemetry().telemetry_id
request_wait_tracker = get_request_wait_tracker()
monitor_started = False
+ processing_started = False
+ from openviking.service.task_tracker import get_task_tracker
+ from openviking.service.task_tracker_concurrency import run_to_completion
+ from openviking.service.task_work_index import bind_task_context
+
+ task_tracker = get_task_tracker()
+ task = None
if telemetry_id:
request_wait_tracker.register_request(telemetry_id)
try:
- if isinstance(data, SkillProcessingPreparation):
- result = await self._skill_processor.process_prepared_skill(
- data,
- viking_fs=self._viking_fs,
- ctx=ctx,
- apply_privacy=apply_privacy,
- privacy_change_reason=privacy_change_reason,
- target_uri=target_uri,
+ # Establish ownership before any queue work can start. Descendant
+ # semantic and embedding messages inherit this same cancellable task.
+ async def create_skill_task() -> None:
+ nonlocal task
+ task = await task_tracker.create(
+ "add_skill",
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ task_id=task_id,
)
- else:
- result = await self._skill_processor.process_skill(
- data=data,
- viking_fs=self._viking_fs,
- ctx=ctx,
- allow_local_path_resolution=allow_local_path_resolution,
- source_path_hint=source_path_hint,
- apply_privacy=apply_privacy,
- privacy_change_reason=privacy_change_reason,
- target_uri=target_uri,
+
+ # Keep the returned record before propagating cancellation, so a
+ # committed task can still be settled by the exception handler.
+ await run_to_completion(create_skill_task)
+ await task_tracker.start(
+ task.task_id, account_id=ctx.account_id, user_id=ctx.user.user_id
+ )
+ task_tracker.register_running_task(task.task_id)
+ try:
+ with bind_task_context(task.task_id, ctx.account_id, ctx.user.user_id):
+ if isinstance(data, SkillProcessingPreparation):
+ result = await self._skill_processor.process_prepared_skill(
+ data,
+ viking_fs=self._viking_fs,
+ ctx=ctx,
+ apply_privacy=apply_privacy,
+ privacy_change_reason=privacy_change_reason,
+ target_uri=target_uri,
+ source_metadata=source_metadata,
+ owner_lease_ref=owner_lease_ref,
+ )
+ else:
+ result = await self._skill_processor.process_skill(
+ data=data,
+ viking_fs=self._viking_fs,
+ ctx=ctx,
+ allow_local_path_resolution=allow_local_path_resolution,
+ source_path_hint=source_path_hint,
+ apply_privacy=apply_privacy,
+ privacy_change_reason=privacy_change_reason,
+ target_uri=target_uri,
+ source_metadata=source_metadata,
+ owner_lease_ref=owner_lease_ref,
+ )
+ finally:
+ await task_tracker.unregister_running_task(task.task_id)
+ processing_started = True
+ if not wait:
+ monitor_started = True
+ asyncio.create_task(
+ self._monitor_queue_processing(
+ task.task_id, telemetry_id, ctx.account_id, ctx.user.user_id
+ )
)
if isinstance(result, dict) and "root_uri" not in result and result.get("uri"):
result["root_uri"] = result["uri"]
@@ -2386,43 +2431,67 @@ class ResourceService:
)
result["queue_status"] = status
self._raise_queue_status_errors(status)
- else:
- from openviking.service.task_tracker import get_task_tracker
-
- task_tracker = get_task_tracker()
- task = await task_tracker.create(
- "add_skill",
+ await task_tracker.complete(
+ task.task_id,
+ {"queue_status": status},
account_id=ctx.account_id,
user_id=ctx.user.user_id,
)
+ # Cancellation settles queue entries without counting errors.
+ # complete() preserves cancellation, so check it before the
+ # update caller commits the package and discards its backup.
+ if task_tracker.is_cancellation_requested(task.task_id):
+ raise OpenVikingError(
+ "Skill processing was cancelled",
+ code="PROCESSING_ERROR",
+ details={"task_id": task.task_id, "status": "cancelled"},
+ )
+ else:
result["task_id"] = task.task_id
- if telemetry_id:
+
+ return result
+ except BaseException as exc:
+ if task is not None and not monitor_started:
+ if processing_started and not request_wait_tracker.is_complete(telemetry_id):
+ # A standalone add retains its original timeout behavior.
+ # An update's caller separately cancels before restoring.
monitor_started = True
asyncio.create_task(
self._monitor_queue_processing(
- task.task_id,
- telemetry_id,
- ctx.account_id,
- ctx.user.user_id,
+ task.task_id, telemetry_id, ctx.account_id, ctx.user.user_id
)
)
else:
- await task_tracker.start(
- task.task_id, account_id=ctx.account_id, user_id=ctx.user.user_id
- )
- await task_tracker.complete(
- task.task_id,
- {},
- account_id=ctx.account_id,
- user_id=ctx.user.user_id,
+ failure_message = str(exc) or "Skill processing cancelled"
+ await run_to_completion(
+ lambda: task_tracker.fail(
+ task.task_id,
+ failure_message,
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ )
)
-
- return result
+ raise
finally:
- if wait or not telemetry_id or not monitor_started:
+ if not monitor_started:
request_wait_tracker.cleanup(telemetry_id)
unregister_telemetry(telemetry_id)
+ async def cancel_skill_processing(self, task_id: str, ctx: RequestContext) -> None:
+ """Stop one failed update's work before its caller restores the old package."""
+ from openviking.service.task_tracker import get_task_tracker
+
+ tracker = get_task_tracker()
+ task = await tracker.cancel_skill_update_for_rollback(
+ task_id, account_id=ctx.account_id, user_id=ctx.user.user_id
+ )
+ if task is None:
+ return # Synchronous preparation failed before the task was created.
+ # This includes queued work that must be discarded and active handlers
+ # that must finish their cancellation/write cleanup, not just task status.
+ while tracker.has_work(task_id):
+ await asyncio.sleep(0.05)
+
async def build_index(
self, resource_uris: List[str], ctx: RequestContext, **kwargs
) -> Dict[str, Any]:
diff --git a/openviking/service/search_service.py b/openviking/service/search_service.py
index 944899bef..89b09c415 100644
--- a/openviking/service/search_service.py
+++ b/openviking/service/search_service.py
@@ -174,3 +174,41 @@ class SearchService:
image_url=resolved_image_url,
)
return result
+
+ async def find_skills(
+ self,
+ query: str,
+ ctx: RequestContext,
+ target_uri: str,
+ limit: int = 10,
+ score_threshold: Optional[float] = None,
+ level: Optional[List[int]] = None,
+ ) -> Any:
+ """Find distinct packages for /skills/find; general find/search stay item-based."""
+ from openviking.core.retrieval_targets import resolve_retrieval_targets
+ from openviking.retrieve.skill_package_retriever import SkillPackageRetriever
+ from openviking.retrieve.skill_results import SkillResultResolver
+ from openviking_cli.retrieve import ContextType, FindResult, TypedQuery
+
+ _ensure_non_empty_query(query)
+ fs = self._ensure_initialized()
+ targets = resolve_retrieval_targets(target_uri, ctx).target_directories
+ for target in targets:
+ await fs._ensure_retrieval_scope(target, ctx)
+ storage, embedder = fs._get_vector_store(), fs._get_embedder()
+ if not storage:
+ raise RuntimeError("Vector store not initialized. Call OpenViking.initialize() first.")
+ if not embedder:
+ raise RuntimeError("Embedder not configured.")
+ retriever = SkillPackageRetriever(
+ storage=storage, embedder=embedder, retrieval_config=fs.retrieval_config
+ )
+ result = await retriever.retrieve_skills(
+ TypedQuery(query, ContextType.SKILL, "", target_directories=targets),
+ ctx,
+ skill_resolver=SkillResultResolver(fs, ctx),
+ limit=limit,
+ score_threshold=score_threshold,
+ level=level,
+ )
+ return FindResult(memories=[], resources=[], skills=result.matched_contexts)
diff --git a/openviking/service/task_tracker.py b/openviking/service/task_tracker.py
index 01e8fd443..901f1e14c 100644
--- a/openviking/service/task_tracker.py
+++ b/openviking/service/task_tracker.py
@@ -62,6 +62,7 @@ _ACTIVE_STATUSES = (TaskStatus.PENDING, TaskStatus.RUNNING, TaskStatus.CANCELLIN
_CANCELLABLE_TASK_TYPES = {
"add_resource",
+ "add_skill",
"compile",
"session_commit",
"admin_reindex",
@@ -654,18 +655,30 @@ class TaskTracker:
user_id: Optional[str] = None,
) -> Optional[TaskRecord]:
"""Request cooperative cancellation and return the current task snapshot."""
+ return await self._cancel_task(task_id, account_id, user_id)
+
+ async def _cancel_task(
+ self,
+ task_id: str,
+ account_id: Optional[str],
+ user_id: Optional[str],
+ *,
+ rollback_skill_update: bool = False,
+ ) -> Optional[TaskRecord]:
cancellation: asyncio.CancelledError | None = None
cancellation_persisted = False
async with self._task_locks.acquire(task_id):
task = await self._load_for_update(task_id, account_id, user_id)
if task is None:
return None
+ if rollback_skill_update and task.task_type != "add_skill":
+ raise ValueError("Only Skill processing can be cancelled for package rollback")
if task.status == TaskStatus.CANCELLED:
return self._copy(task)
if task.status != TaskStatus.CANCELLING:
if task.task_type not in _CANCELLABLE_TASK_TYPES:
raise ValueError(f"Task type '{task.task_type}' does not support cancellation")
- if task.status in _TERMINAL_STATUSES:
+ if task.status in _TERMINAL_STATUSES and not rollback_skill_update:
raise ValueError(f"Task is already {task.status.value}")
updated = deepcopy(task)
@@ -701,6 +714,17 @@ class TaskTracker:
raise cancellation
return self._copy(task)
+ async def cancel_skill_update_for_rollback(
+ self, task_id: str, *, account_id: str, user_id: str
+ ) -> Optional[TaskRecord]:
+ """Prevent even a late queue replay from writing into a restored Skill.
+
+ A last ACK can fail after the task appears finished. Persist cancellation
+ for this rolled-back update even if it has reached a terminal state.
+ Normal public cancellation keeps its existing terminal-state rules.
+ """
+ return await self._cancel_task(task_id, account_id, user_id, rollback_skill_update=True)
+
async def _finalize_task(
self,
task_id: str,
diff --git a/openviking/storage/collection_schemas.py b/openviking/storage/collection_schemas.py
index 80bd3f572..d67bc562e 100644
--- a/openviking/storage/collection_schemas.py
+++ b/openviking/storage/collection_schemas.py
@@ -19,6 +19,7 @@ from typing import Any, Dict, List, Optional
from openviking.core.context import ContextType, ResourceContentType
from openviking.models.embedder.base import embed_compat
from openviking.server.identity import RequestContext, Role
+from openviking.service.task_tracker_concurrency import run_to_completion
from openviking.storage.acl import ACL_GRANT_FIELDS, ACL_MODE_FIELD, AclMode
from openviking.storage.errors import (
CollectionNotFoundError,
@@ -648,7 +649,7 @@ class TextEmbeddingHandler(DequeueHandlerBase):
wait = self._circuit_breaker.retry_after
if wait > 0:
await asyncio.sleep(wait)
- await self._vikingdb.enqueue_embedding_msg(embedding_msg)
+ await self._reenqueue_embedding_msg(embedding_msg)
self._merge_request_stats(
embedding_msg.telemetry_id,
requeue_count=1,
@@ -739,7 +740,7 @@ class TextEmbeddingHandler(DequeueHandlerBase):
self._circuit_breaker.record_failure(embed_err)
if self._vikingdb.has_queue_manager:
try:
- await self._vikingdb.enqueue_embedding_msg(embedding_msg)
+ await self._reenqueue_embedding_msg(embedding_msg)
self._merge_request_stats(
embedding_msg.telemetry_id,
requeue_count=1,
@@ -819,11 +820,20 @@ class TextEmbeddingHandler(DequeueHandlerBase):
embedding_msg,
ctx,
)
- result = await self._vikingdb.upsert(
- inserted_data,
- ctx=ctx,
- options=upsert_options,
- )
+ if inserted_data.get("context_type") == ContextType.SKILL.value:
+ # Cancelling the waiter cannot stop a threaded DB write.
+ # Keep this task active until that write has settled.
+ result = await run_to_completion(
+ lambda: self._vikingdb.upsert(
+ inserted_data, ctx=ctx, options=upsert_options
+ )
+ )
+ else:
+ result = await self._vikingdb.upsert(
+ inserted_data,
+ ctx=ctx,
+ options=upsert_options,
+ )
record_id = result
if record_id:
logger.debug("Successfully wrote embedding: uri=%s", uri)
@@ -868,6 +878,15 @@ class TextEmbeddingHandler(DequeueHandlerBase):
self._circuit_breaker.record_success()
return ProcessResult.success(inserted_data)
+ except asyncio.CancelledError:
+ if (
+ embedding_msg is not None
+ and embedding_msg.context_data.get("context_type") == ContextType.SKILL.value
+ ):
+ # Active cancellation does not call on_cancelled in NamedQueue.
+ # Settle only after any already-started vector write has exited.
+ self._record_request_success(embedding_msg)
+ raise
except Exception as e:
error_msg = self._embedding_error_msg(
embedding_msg,
@@ -901,6 +920,12 @@ class TextEmbeddingHandler(DequeueHandlerBase):
self._record_request_success(embedding_msg)
return ProcessResult.cancelled()
+ async def _reenqueue_embedding_msg(self, msg: EmbeddingMsg) -> None:
+ if msg.context_data.get("context_type") == ContextType.SKILL.value:
+ await run_to_completion(lambda: self._vikingdb.enqueue_embedding_msg(msg))
+ else:
+ await self._vikingdb.enqueue_embedding_msg(msg)
+
@staticmethod
def _record_request_success(
embedding_msg: EmbeddingMsg,
diff --git a/openviking/storage/queuefs/queue_manager.py b/openviking/storage/queuefs/queue_manager.py
index b248fae50..cd55d3fee 100644
--- a/openviking/storage/queuefs/queue_manager.py
+++ b/openviking/storage/queuefs/queue_manager.py
@@ -125,6 +125,7 @@ class QueueManager:
self._started = False
self._queue_threads: Dict[str, threading.Thread] = {}
self._queue_stop_events: Dict[str, threading.Event] = {}
+ self._embedding_worker_stopped = threading.Event()
self._poll_interval = 0.2
self._task_work_index = TaskWorkIndex()
# Import at composition time to avoid a service <-> queue package cycle.
@@ -185,7 +186,10 @@ class QueueManager:
logger.info("Embedding queue initialized with TextEmbeddingHandler")
# Semantic Queue
- semantic_processor = SemanticProcessor(max_concurrent_llm=self._max_concurrent_semantic)
+ semantic_processor = SemanticProcessor(
+ max_concurrent_llm=self._max_concurrent_semantic,
+ embedding_worker_stopped=self._embedding_worker_stopped.is_set,
+ )
self.get_queue(
self.SEMANTIC,
dequeue_handler=semantic_processor,
@@ -206,6 +210,8 @@ class QueueManager:
max_concurrent = self._max_concurrent_for_queue(queue.name)
stop_event = threading.Event()
self._queue_stop_events[queue.name] = stop_event
+ if queue.name == self.EMBEDDING:
+ self._embedding_worker_stopped.clear()
thread = threading.Thread(
target=self._queue_worker_loop,
args=(queue, stop_event, max_concurrent),
@@ -262,6 +268,10 @@ class QueueManager:
stop_event.wait(poll_interval)
else:
stop_event.wait(poll_interval)
+ except asyncio.CancelledError:
+ if not stop_event.is_set():
+ raise
+ break
except Exception as e:
logger.error(f"[QueueManager] Worker error for {queue.name}: {e}")
traceback.print_exc()
@@ -277,6 +287,10 @@ class QueueManager:
loop.run_until_complete(loop.shutdown_asyncgens())
loop.run_until_complete(loop.shutdown_default_executor())
loop.close()
+ if queue.name == self.EMBEDDING:
+ # No more deliveries can start and active handlers have exited,
+ # including protected writes. Pending messages remain durable.
+ self._embedding_worker_stopped.set()
async def _worker_async_concurrent(
self, queue: NamedQueue, stop_event: threading.Event, max_concurrent: int
@@ -349,6 +363,8 @@ class QueueManager:
# Stop queue workers
for stop_event in self._queue_stop_events.values():
stop_event.set()
+ if self.EMBEDDING not in self._queue_threads:
+ self._embedding_worker_stopped.set()
for name, thread in self._queue_threads.items():
thread.join(timeout=10.0)
if thread.is_alive():
diff --git a/openviking/storage/queuefs/semantic_dag.py b/openviking/storage/queuefs/semantic_dag.py
index 95ca59a1b..9f9e74e28 100644
--- a/openviking/storage/queuefs/semantic_dag.py
+++ b/openviking/storage/queuefs/semantic_dag.py
@@ -5,14 +5,19 @@
import asyncio
import re
import threading
-from contextlib import nullcontext
from dataclasses import dataclass, field
from typing import Any, ClassVar, Dict, List, Optional, Set
from weakref import WeakKeyDictionary
+from openviking.core.namespace import classify_uri
from openviking.parse.parsers.media import get_media_type
from openviking.server.identity import RequestContext
-from openviking.service.task_work_index import bind_task_context, get_task_context
+from openviking.service.task_tracker_concurrency import run_to_completion
+from openviking.service.task_work_index import (
+ bind_task_context,
+ detach_task_context,
+ get_task_context,
+)
from openviking.storage.abstract_overview import (
AbstractOverviewFormatError,
AbstractOverviewWriteResult,
@@ -68,6 +73,8 @@ class DagStats:
pending_nodes: int = 0
in_progress_nodes: int = 0
done_nodes: int = 0
+ failures: List[str] = field(default_factory=list)
+ indexed_records: int = 0
@dataclass(frozen=True)
@@ -215,6 +222,7 @@ class SemanticDagExecutor:
self._root_done: Optional[asyncio.Event] = None
self._scheduler: Optional[SemanticNodeScheduler] = None
self._active_scheduled_work = 0
+ self._skill_node_tasks: Set[asyncio.Task] = set()
self._active_work_idle = asyncio.Event()
self._active_work_idle.set()
self._closed = False
@@ -236,6 +244,10 @@ class SemanticDagExecutor:
return CreatorAclGrant.INHERITED
return CreatorAclGrant.DIRECT if normalized in self._added_paths else None
+ def _record_skill_failure(self, uri: str, error: Exception) -> None:
+ if self._context_type == "skill":
+ self._stats.failures.append(f"{uri}: {error}")
+
async def run(self, root_uri: str) -> None:
"""Run DAG execution starting from root_uri."""
self._root_uri = root_uri
@@ -250,12 +262,23 @@ class SemanticDagExecutor:
raise self._failure
except BaseException:
self._closed = True
+ if self._context_type == "skill":
+ for task in self._skill_node_tasks:
+ task.cancel()
+ await run_to_completion(
+ lambda: asyncio.gather(*self._skill_node_tasks, return_exceptions=True)
+ )
await self._active_work_idle.wait()
raise
finally:
self._closed = True
- await self._active_work_idle.wait()
- self._unregister_active()
+ try:
+ if self._context_type == "skill":
+ await run_to_completion(self._active_work_idle.wait)
+ else:
+ await self._active_work_idle.wait()
+ finally:
+ self._unregister_active()
def _schedule_work(self, work: DagWork) -> None:
if self._closed:
@@ -336,6 +359,22 @@ class SemanticDagExecutor:
return stats
async def _run_work(self, work: DagWork) -> None:
+ if self._context_type == "skill":
+ # Shared scheduler workers must not be cancelled with one package.
+ # Each Skill node has its own cancellable task and ownership context.
+ task = asyncio.create_task(self._run_work_with_context(work))
+ self._skill_node_tasks.add(task)
+ try:
+ await task
+ except asyncio.CancelledError:
+ if not self._closed:
+ raise
+ finally:
+ self._skill_node_tasks.discard(task)
+ else:
+ await self._run_work_with_context(work)
+
+ async def _run_work_with_context(self, work: DagWork) -> None:
task_context = (
bind_task_context(
self._task_context.task_id,
@@ -343,11 +382,16 @@ class SemanticDagExecutor:
self._task_context.user_id,
)
if self._task_context is not None
- else nullcontext()
+ else detach_task_context()
)
with bind_telemetry(self._telemetry), task_context:
await self._run_work_bound(work)
+ async def _await_write(self, operation):
+ if self._context_type == "skill":
+ return await run_to_completion(lambda: operation)
+ return await operation
+
async def _run_work_bound(self, work: DagWork) -> None:
self._mark_node_started()
@@ -474,6 +518,7 @@ class SemanticDagExecutor:
return False
except Exception as e:
logger.error(f"Failed to dispatch directory {dir_uri}: {e}", exc_info=True)
+ self._record_skill_failure(dir_uri, e)
if self._generation_trigger == "content_copy":
raise
if parent_uri:
@@ -801,6 +846,7 @@ class SemanticDagExecutor:
)
except Exception as e:
logger.warning(f"Failed to generate summary for {file_path}: {e}")
+ self._record_skill_failure(file_path, e)
summary_dict = {"name": file_name, "summary": ""}
finally:
self._stats.done_nodes += 1
@@ -811,16 +857,20 @@ class SemanticDagExecutor:
if need_vectorize and not self._skip_vectorization:
use_summary = self._is_code_repo and bool(summary_dict.get("summary"))
try:
- await self._processor._vectorize_single_file(
- parent_uri=parent_uri,
- context_type=self._context_type,
- file_path=file_path,
- summary_dict=summary_dict,
- ctx=self._ctx,
- use_summary=use_summary,
- ingest_options=self._ingest_options_for_file(file_path),
- creator_acl_grant=self._creator_acl_grant(file_path),
+ enqueued = await self._await_write(
+ self._processor._vectorize_single_file(
+ parent_uri=parent_uri,
+ context_type=self._context_type,
+ file_path=file_path,
+ summary_dict=summary_dict,
+ ctx=self._ctx,
+ use_summary=use_summary,
+ ingest_options=self._ingest_options_for_file(file_path),
+ creator_acl_grant=self._creator_acl_grant(file_path),
+ )
)
+ if enqueued:
+ self._stats.indexed_records += 1
except Exception as e:
logger.error(
"Failed to schedule vectorization for %s: %s",
@@ -828,7 +878,9 @@ class SemanticDagExecutor:
e,
exc_info=True,
)
- raise
+ if self._context_type != "skill":
+ raise
+ self._record_skill_failure(file_path, e)
await self._on_file_done(
parent_uri,
file_path,
@@ -975,17 +1027,19 @@ class SemanticDagExecutor:
}
if dir_uri == self._root_uri and self._source:
metadata["source"] = self._source
- wrote = await write_abstract_overview(
- viking_fs=self._viking_fs,
- dir_uri=dir_uri,
- overview=overview,
- abstract=abstract,
- ctx=self._ctx,
- is_stale=self._is_stale,
- metadata=metadata,
- consume_pending=consume_pending,
- lock=self._lock,
- log_prefix="[SemanticDag]",
+ wrote = await self._await_write(
+ write_abstract_overview(
+ viking_fs=self._viking_fs,
+ dir_uri=dir_uri,
+ overview=overview,
+ abstract=abstract,
+ ctx=self._ctx,
+ is_stale=self._is_stale,
+ metadata=metadata,
+ consume_pending=consume_pending,
+ lock=self._lock,
+ log_prefix="[SemanticDag]",
+ )
)
if not wrote.wrote:
self._stale = True
@@ -995,7 +1049,12 @@ class SemanticDagExecutor:
node = self._nodes.get(dir_uri)
if not node:
return
- if not self._aggregate_directory:
+ skill_definition_changed = (
+ self._context_type == "skill"
+ and classify_uri(dir_uri).is_skill_root
+ and f"{dir_uri}/SKILL.md" in self._changed_paths
+ )
+ if not self._aggregate_directory and not skill_definition_changed:
# Deferred aggregation still ran changed-file work. Finish without
# touching directory sidecars or their vectors.
self._stats.done_nodes += 1
@@ -1015,8 +1074,22 @@ class SemanticDagExecutor:
else len(node.file_paths) + len(node.children_dirs)
)
sampled_inputs: List[tuple[str, Dict[str, str]]] = []
+ sampled_entries = 0
try:
- if self._generation_trigger == "content_copy" and not node.transfer_inputs_ready:
+ if self._context_type == "skill" and classify_uri(dir_uri).is_skill_root:
+ # The package root describes the skill definition, independently
+ # of the summaries produced for its attachments.
+ definition_changed = f"{dir_uri}/SKILL.md" in self._changed_paths
+ overview, abstract = await self._processor._skill_root_semantics(
+ dir_uri,
+ ctx=self._ctx,
+ regenerate=self._generation_trigger == "reindex" or definition_changed,
+ lock=self._lock,
+ )
+ should_write = False
+ need_vectorize = not self._incremental_update or definition_changed
+ children_changed = definition_changed
+ elif self._generation_trigger == "content_copy" and not node.transfer_inputs_ready:
need_vectorize = False
should_write = False
elif self._incremental_update and self._generation_trigger != "content_copy":
@@ -1081,27 +1154,38 @@ class SemanticDagExecutor:
need_vectorize = False
except AbstractOverviewFormatError:
raise
- except Exception:
+ except Exception as exc:
need_vectorize = False
logger.info(f"[SemanticDag] {dir_uri} write failed, skipping")
+ self._record_skill_failure(dir_uri, exc)
except AbstractOverviewFormatError:
raise
except Exception as e:
logger.error(f"Failed to generate overview for {dir_uri}: {e}", exc_info=True)
+ self._record_skill_failure(dir_uri, e)
else:
if need_vectorize and not self._skip_vectorization:
assert overview is not None and abstract is not None
try:
- await self._processor._vectorize_directory(
- dir_uri,
- context_type=self._context_type,
- abstract=abstract,
- overview=overview,
- ctx=self._ctx,
- ingest_options=self._ingest_options_for_directory(),
- creator_acl_grant=self._creator_acl_grant(dir_uri),
+ await self._await_write(
+ self._processor._vectorize_directory(
+ dir_uri,
+ context_type=self._context_type,
+ abstract=abstract,
+ overview=overview,
+ ctx=self._ctx,
+ ingest_options=self._ingest_options_for_directory(),
+ creator_acl_grant=self._creator_acl_grant(dir_uri),
+ **(
+ {"skill_source_path": (self._source or {}).get("path", "")}
+ if self._context_type == "skill"
+ and classify_uri(dir_uri).is_skill_root
+ else {}
+ ),
+ )
)
+ self._stats.indexed_records += int(bool(abstract)) + int(bool(overview))
except Exception as e:
logger.error(
"Failed to schedule vectorization for %s: %s",
@@ -1109,7 +1193,9 @@ class SemanticDagExecutor:
e,
exc_info=True,
)
- raise
+ if self._context_type != "skill":
+ raise
+ self._record_skill_failure(dir_uri, e)
finally:
self._stats.done_nodes += 1
self._stats.in_progress_nodes = max(0, self._stats.in_progress_nodes - 1)
@@ -1132,6 +1218,8 @@ class SemanticDagExecutor:
pending_nodes=self._stats.pending_nodes,
in_progress_nodes=self._stats.in_progress_nodes,
done_nodes=self._stats.done_nodes,
+ failures=list(self._stats.failures),
+ indexed_records=self._stats.indexed_records,
)
@property
diff --git a/openviking/storage/queuefs/semantic_processor.py b/openviking/storage/queuefs/semantic_processor.py
index b0099acd7..c43fb9036 100644
--- a/openviking/storage/queuefs/semantic_processor.py
+++ b/openviking/storage/queuefs/semantic_processor.py
@@ -6,9 +6,10 @@ import asyncio
import re
import threading
from contextlib import nullcontext
-from typing import Any, Dict, List, Optional, Set, Tuple
+from typing import Any, Awaitable, Callable, Dict, List, Optional, Set, Tuple
from urllib.parse import quote, unquote, urlsplit
+from openviking.core.namespace import classify_uri
from openviking.observability.context import (
bind_root_observability_context,
reset_root_observability_context,
@@ -33,6 +34,7 @@ from openviking.parse.parsers.media.utils import (
)
from openviking.prompts import render_prompt
from openviking.server.identity import RequestContext, Role
+from openviking.service.task_tracker_concurrency import run_to_completion
from openviking.service.task_work_index import detach_task_context
from openviking.storage.abstract_overview import (
AbstractOverviewWriteResult,
@@ -51,8 +53,12 @@ from openviking.storage.queuefs.semantic_lock import SemanticLockScope
from openviking.storage.queuefs.semantic_msg import SemanticMsg, build_semantic_coalesce_key
from openviking.storage.queuefs.semantic_ops.freshness_policy import FreshnessAction
from openviking.storage.queuefs.semantic_queue import is_semantic_msg_stale
+from openviking.storage.queuefs.semantic_work import SemanticMessageWork, SkillSemanticMessageWork
from openviking.storage.viking_fs import LS_ALL_NODES, SyncDiff, get_viking_fs
-from openviking.telemetry import bind_telemetry, bind_telemetry_stage, resolve_telemetry
+from openviking.telemetry import (
+ bind_telemetry,
+ bind_telemetry_stage,
+)
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
from openviking.telemetry.span_models import create_root_span_attributes
from openviking.utils.circuit_breaker import (
@@ -95,7 +101,12 @@ class SemanticProcessor(DequeueHandlerBase):
_request_stats_order: List[str] = []
_max_cached_stats = 256
- def __init__(self, max_concurrent_llm: int = 32):
+ def __init__(
+ self,
+ max_concurrent_llm: int = 32,
+ *,
+ embedding_worker_stopped: Optional[Callable[[], bool]] = None,
+ ):
"""
Initialize SemanticProcessor.
@@ -105,6 +116,7 @@ class SemanticProcessor(DequeueHandlerBase):
self.max_concurrent_llm = max_concurrent_llm
self._default_ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)
self._circuit_breaker = CircuitBreaker()
+ self._embedding_worker_stopped = embedding_worker_stopped
@classmethod
def _cache_dag_stats(cls, telemetry_id: str, uri: str, stats: DagStats) -> None:
@@ -203,7 +215,12 @@ class SemanticProcessor(DequeueHandlerBase):
# Default to other
return FILE_TYPE_OTHER
- async def _reenqueue_semantic_msg(self, msg: SemanticMsg) -> None:
+ async def _reenqueue_semantic_msg(
+ self,
+ msg: SemanticMsg,
+ *,
+ enqueue: Optional[Callable[[Any, SemanticMsg], Awaitable[None]]] = None,
+ ) -> None:
"""Re-enqueue a semantic message for later processing.
Throttles with a sleep when the circuit breaker is open to prevent
@@ -221,18 +238,44 @@ class SemanticProcessor(DequeueHandlerBase):
queue_manager = get_queue_manager()
if queue_manager is not None:
semantic_queue = queue_manager.get_queue(queue_manager.SEMANTIC)
- await semantic_queue.enqueue(msg)
+ if enqueue is None:
+ await semantic_queue.enqueue(msg)
+ else:
+ await enqueue(semantic_queue, msg)
logger.info(f"Re-enqueued semantic message: {msg.uri}")
else:
logger.warning(f"No queue manager available, cannot re-enqueue: {msg.uri}")
+ async def _enqueue_skill_retry(self, queue, msg: SemanticMsg, scope: SemanticLockScope) -> None:
+ """Transfer the live package lease to a retry before releasing this worker.
+
+ Reusing the consumed handoff would require acquiring an unrelated lock,
+ which conflicts with an update request still waiting under its outer lease.
+ """
+ agfs = get_viking_fs()._async_agfs
+ handoff = await agfs.pathlock_to_handoff(scope.lock)
+ handed_off = False
+ try:
+ await agfs.pathlock_handoff(scope.lock)
+ handed_off = True
+ scope._owned = False
+ msg.lock_handoff = handoff
+ await queue.enqueue(msg)
+ except BaseException:
+ if handed_off:
+ scope.lock = await agfs.pathlock_adopt(handoff)
+ scope._owned = True
+ raise
+
async def _requeue_semantic_msg_after_error(
self,
msg: SemanticMsg,
error: Exception,
+ *,
+ work: SemanticMessageWork,
) -> ProcessResult:
try:
- await self._reenqueue_semantic_msg(msg)
+ await work.reenqueue_after_error()
self._merge_request_stats(msg.telemetry_id, requeue_count=1)
get_request_wait_tracker().record_semantic_requeue(msg.telemetry_id)
except Exception as requeue_err:
@@ -255,6 +298,14 @@ class SemanticProcessor(DequeueHandlerBase):
if parent is None:
return
parent_uri = parent.uri.rstrip("/")
+ if msg.context_type == "skill":
+ classification = classify_uri(parent_uri)
+ if (
+ not classification.is_skill
+ or classification.is_skill_namespace
+ or classification.is_skill_root
+ ):
+ return
if (
not parent_uri
or parent_uri in {"viking://", "viking:", "viking://user", "viking://agent"}
@@ -323,6 +374,12 @@ class SemanticProcessor(DequeueHandlerBase):
await semantic_queue.enqueue(parent_msg)
logger.info("Enqueued parent semantic refresh: %s", parent_uri)
+ def _message_work(
+ self, msg: SemanticMsg, lock: Optional[Dict[str, Any]] = None
+ ) -> SemanticMessageWork:
+ work_type = SkillSemanticMessageWork if msg.context_type == "skill" else SemanticMessageWork
+ return work_type(self, msg, lock)
+
async def on_dequeue(
self,
data: Optional[Dict[str, Any]],
@@ -331,6 +388,7 @@ class SemanticProcessor(DequeueHandlerBase):
"""Process dequeued SemanticMsg, recursively process all subdirectories."""
msg: Optional[SemanticMsg] = None
collector = None
+ work: Optional[SemanticMessageWork] = None
try:
import json
@@ -342,6 +400,8 @@ class SemanticProcessor(DequeueHandlerBase):
assert data is not None
msg = SemanticMsg.from_dict(data)
+ work = self._message_work(msg, lock)
+ work.start()
if VikingURI(msg.uri).parent is None:
logger.warning("Skipping semantic generation for root URI: %s", msg.uri)
if msg.telemetry_id and msg.id:
@@ -372,6 +432,7 @@ class SemanticProcessor(DequeueHandlerBase):
msg.uri,
msg.coalesce_version,
)
+ await work.skip()
if msg.telemetry_id and msg.id:
get_request_wait_tracker().mark_semantic_done(msg.telemetry_id, msg.id)
return ProcessResult.success()
@@ -382,11 +443,11 @@ class SemanticProcessor(DequeueHandlerBase):
logger.warning(
f"Circuit breaker is open, re-enqueueing semantic message: {msg.uri}"
)
- await self._reenqueue_semantic_msg(msg)
+ await work.reenqueue()
self._merge_request_stats(msg.telemetry_id, requeue_count=1)
get_request_wait_tracker().record_semantic_requeue(msg.telemetry_id)
return ProcessResult.requeued()
- collector = resolve_telemetry(msg.telemetry_id)
+ collector = work.resolve_telemetry()
telemetry_ctx = bind_telemetry(collector) if collector is not None else nullcontext()
with telemetry_ctx:
root_attrs = create_root_span_attributes(
@@ -406,22 +467,22 @@ class SemanticProcessor(DequeueHandlerBase):
logger.info(f"Processing semantic generation for: {msg})")
- # Resolving the lock scope for a root that was deleted
- # after enqueue would recreate it (lock metadata needs a
- # parent directory). There is nothing to summarize; ack.
+ # Resolving a deleted root can recreate it for lock metadata.
+ # Settle queued ownership before acknowledging skipped work.
if not await get_viking_fs().exists(msg.uri, ctx=current_ctx):
logger.info("Skipping semantic message for missing root: uri=%s", msg.uri)
+ await work.skip()
if msg.telemetry_id and msg.id:
get_request_wait_tracker().mark_semantic_done(msg.telemetry_id, msg.id)
return ProcessResult.success()
- semantic_lock = await SemanticLockScope.resolve(
- msg.lock_handoff,
- caller_lock=lock,
- fallback_path_factory=lambda: get_viking_fs()._uri_to_path(
- msg.uri, ctx=current_ctx
- ),
- )
+ if not await work.acquire_lock(current_ctx):
+ get_request_wait_tracker().mark_semantic_done(msg.telemetry_id, msg.id)
+ return ProcessResult.success()
+ semantic_lock = work.scope
+ assert semantic_lock is not None
+ dag_stats = None
+ processing_succeeded = False
try:
# Regular memory writes keep their specialized update path.
# Callers must explicitly opt into directory aggregation; the
@@ -447,11 +508,13 @@ class SemanticProcessor(DequeueHandlerBase):
"Syncing semantic source into target before processing: "
f"{msg.uri} -> {msg.target_uri}"
)
- diff = await self._sync_topdown_recursive(
- msg.uri,
- msg.target_uri,
- ctx=current_ctx,
- lock=semantic_lock.lock,
+ diff = await work.run_write(
+ lambda: self._sync_topdown_recursive(
+ msg.uri,
+ msg.target_uri,
+ ctx=current_ctx,
+ lock=semantic_lock.lock,
+ )
)
logger.info(
"[SyncDiff] Diff computed: "
@@ -499,10 +562,11 @@ class SemanticProcessor(DequeueHandlerBase):
copy_source_uri=msg.copy_source_uri,
)
await executor.run(run_uri)
+ dag_stats = executor.get_stats()
self._cache_dag_stats(
msg.telemetry_id,
run_uri,
- executor.get_stats(),
+ dag_stats,
)
if not executor.stale and msg.aggregate_directory:
write_result = getattr(
@@ -517,8 +581,12 @@ class SemanticProcessor(DequeueHandlerBase):
target_uri or msg.uri,
l0_body_changed=write_result.abstract_body_changed,
)
+ processing_succeeded = True
finally:
- await semantic_lock.close()
+ await work.finish_processing(processing_succeeded)
+ failure = work.failure_result(dag_stats)
+ if failure is not None:
+ return failure
get_request_wait_tracker().mark_semantic_done(msg.telemetry_id, msg.id)
self._merge_request_stats(msg.telemetry_id, processed=1)
logger.info(f"Completed semantic generation for: {msg.uri}")
@@ -527,6 +595,10 @@ class SemanticProcessor(DequeueHandlerBase):
finally:
reset_root_observability_context(root_context_token)
+ except asyncio.CancelledError:
+ if work is not None:
+ await work.cancel()
+ raise
except Exception as e:
if isinstance(e, LockAcquisitionError):
logger.warning(
@@ -535,8 +607,12 @@ class SemanticProcessor(DequeueHandlerBase):
e,
exc_info=True,
)
- if msg is not None:
- return await self._requeue_semantic_msg_after_error(msg, e)
+ if msg is not None and work is not None:
+ return await self._requeue_semantic_msg_after_error(
+ msg,
+ e,
+ work=work,
+ )
return ProcessResult.failed(str(e))
error_class = classify_api_error(e)
@@ -570,10 +646,18 @@ class SemanticProcessor(DequeueHandlerBase):
exc_info=True,
)
self._circuit_breaker.record_failure(e)
- if msg is not None:
- return await self._requeue_semantic_msg_after_error(msg, e)
+ if msg is not None and work is not None:
+ return await self._requeue_semantic_msg_after_error(
+ msg,
+ e,
+ work=work,
+ )
return ProcessResult.failed(str(e))
+ finally:
+ if work is not None:
+ await work.close()
+
async def on_cancelled(self, data: Optional[Dict[str, Any]]) -> ProcessResult:
"""Release a queued semantic lock before cancelled work is ACKed."""
try:
@@ -586,8 +670,10 @@ class SemanticProcessor(DequeueHandlerBase):
except (TypeError, ValueError) as exc:
return ProcessResult.failed(str(exc))
- if msg.telemetry_id and msg.id:
- get_request_wait_tracker().mark_semantic_done(msg.telemetry_id, msg.id)
+ await self._message_work(msg).cancel_queued()
+ return ProcessResult.cancelled()
+
+ async def _release_cancelled_semantic_lock(self, msg: SemanticMsg) -> None:
if msg.lock_handoff is not None:
try:
viking_fs = get_viking_fs()
@@ -595,7 +681,37 @@ class SemanticProcessor(DequeueHandlerBase):
await viking_fs._async_agfs.pathlock_release(lock)
except Exception as exc:
logger.warning("Failed to release cancelled semantic lock: %s", exc)
- return ProcessResult.cancelled()
+
+ async def _resolve_skill_semantic_lock(
+ self,
+ msg: SemanticMsg,
+ ctx: RequestContext,
+ caller_lock: Optional[Dict[str, Any]],
+ ) -> SemanticLockScope:
+ scope = None
+
+ async def acquire():
+ nonlocal scope
+ viking_fs = get_viking_fs()
+ scope = await SemanticLockScope.resolve(
+ msg.lock_handoff,
+ caller_lock=caller_lock,
+ fallback_path_factory=lambda: viking_fs._uri_to_path(msg.uri, ctx=ctx),
+ )
+ if scope.lock is None and await viking_fs.exists(msg.uri, ctx=ctx):
+ lease = await viking_fs._async_agfs.pathlock_acquire_tree(
+ viking_fs._uri_to_path(msg.uri, ctx=ctx)
+ )
+ scope = SemanticLockScope(lease, _owned=True)
+ return scope
+
+ try:
+ return await run_to_completion(acquire)
+ except asyncio.CancelledError:
+ # Acquiring/adopting a lease can itself outlive a cancelled await.
+ if scope is not None:
+ await run_to_completion(scope.close)
+ raise
def get_dag_stats(self) -> Optional["DagStats"]:
return SemanticDagExecutor.get_active_stats()
@@ -1544,6 +1660,52 @@ class SemanticProcessor(DequeueHandlerBase):
)
return partial_overviews[0]
+ async def _skill_root_semantics(
+ self,
+ uri: str,
+ *,
+ ctx: RequestContext,
+ regenerate: bool = False,
+ lock: Optional[Dict[str, Any]] = None,
+ ) -> Tuple[str, str]:
+ """Keep the package root tied only to its SKILL.md definition."""
+ viking_fs = get_viking_fs()
+ if (
+ not regenerate
+ and await viking_fs.exists(f"{uri}/.abstract.md", ctx=ctx)
+ and await viking_fs.exists(f"{uri}/.overview.md", ctx=ctx)
+ ):
+ abstract = body_for_preview(await viking_fs.read_file(f"{uri}/.abstract.md", ctx=ctx))
+ overview = body_for_preview(await viking_fs.read_file(f"{uri}/.overview.md", ctx=ctx))
+ if abstract and overview:
+ return overview, abstract
+
+ from openviking.core.skill_loader import SkillLoader
+ from openviking.utils.skill_processor import SkillProcessor
+
+ content = await viking_fs.read_file(f"{uri}/SKILL.md", ctx=ctx)
+ if isinstance(content, bytes):
+ content = content.decode("utf-8")
+ definition = SkillLoader.parse(content)
+ processor = SkillProcessor(vikingdb=None)
+ abstract = processor._build_skill_abstract(definition)
+ overview = await processor._generate_overview(definition, get_openviking_config())
+ await run_to_completion(
+ lambda: write_abstract_overview(
+ viking_fs=viking_fs,
+ dir_uri=uri,
+ abstract=abstract,
+ overview=overview,
+ ctx=ctx,
+ lock=lock,
+ is_stale=lambda: False,
+ metadata={
+ "generated_by": {"component": "SkillProcessor", "trigger": "skill_refresh"}
+ },
+ )
+ )
+ return overview, abstract
+
async def _vectorize_directory(
self,
uri: str,
@@ -1553,12 +1715,24 @@ class SemanticProcessor(DequeueHandlerBase):
ctx: Optional[RequestContext] = None,
ingest_options: IngestOptions | None = None,
creator_acl_grant: CreatorAclGrant | None = None,
+ skill_source_path: str = "",
) -> None:
"""Create directory Context and enqueue to EmbeddingQueue."""
from openviking.utils.embedding_utils import vectorize_directory_meta
active_ctx = ctx or self._default_ctx
+ skill_meta = None
+ if context_type == "skill" and classify_uri(uri).is_skill_root:
+ import yaml
+
+ parsed = yaml.safe_load(body_for_preview(abstract))
+ if isinstance(parsed, dict):
+ skill_meta = {
+ key: parsed.get(key, [] if key in {"tags", "allowed_tools"} else "")
+ for key in ("name", "description", "tags", "allowed_tools")
+ }
+ skill_meta["source_path"] = skill_source_path
await vectorize_directory_meta(
uri=uri,
abstract=abstract,
@@ -1567,6 +1741,8 @@ class SemanticProcessor(DequeueHandlerBase):
ctx=active_ctx,
ingest_options=ingest_options,
creator_acl_grant=creator_acl_grant,
+ content_is_body=context_type == "skill",
+ **({"meta": skill_meta} if skill_meta is not None else {}),
)
async def _load_transfer_file_summaries(
@@ -1595,12 +1771,12 @@ class SemanticProcessor(DequeueHandlerBase):
preserve_existing_created_at: bool = False,
ingest_options: IngestOptions | None = None,
creator_acl_grant: CreatorAclGrant | None = None,
- ) -> None:
+ ) -> bool:
"""Vectorize a single file using its content or summary."""
from openviking.utils.embedding_utils import vectorize_file
active_ctx = ctx or self._default_ctx
- await vectorize_file(
+ return await vectorize_file(
file_path=file_path,
summary_dict=summary_dict,
parent_uri=parent_uri,
diff --git a/openviking/storage/queuefs/semantic_work.py b/openviking/storage/queuefs/semantic_work.py
new file mode 100644
index 000000000..db1970c23
--- /dev/null
+++ b/openviking/storage/queuefs/semantic_work.py
@@ -0,0 +1,212 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Per-message semantic lifecycle, including Skill update cancellation and locks."""
+
+from __future__ import annotations
+
+import asyncio
+from typing import TYPE_CHECKING, Any, Awaitable, Callable, TypeVar
+
+from openviking.server.identity import RequestContext
+from openviking.service.task_tracker_concurrency import run_to_completion
+from openviking.storage.queuefs.process_result import ProcessResult
+from openviking.storage.queuefs.semantic_dag import DagStats
+from openviking.storage.queuefs.semantic_lock import SemanticLockScope
+from openviking.storage.queuefs.semantic_msg import SemanticMsg
+from openviking.storage.viking_fs import get_viking_fs
+from openviking.telemetry import (
+ OperationTelemetry,
+ register_telemetry,
+ resolve_telemetry,
+ unregister_telemetry,
+)
+from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
+
+if TYPE_CHECKING:
+ from openviking.storage.queuefs.semantic_processor import SemanticProcessor
+
+_Result = TypeVar("_Result")
+
+
+class SemanticMessageWork:
+ """Original Resource/Memory lifecycle; each dequeued message owns its scope."""
+
+ def __init__(
+ self,
+ processor: SemanticProcessor,
+ msg: SemanticMsg,
+ caller_lock: dict[str, Any] | None,
+ ) -> None:
+ self.processor = processor
+ self.msg = msg
+ self.caller_lock = caller_lock
+ self.scope: SemanticLockScope | None = None
+
+ def start(self) -> None:
+ pass
+
+ def resolve_telemetry(self) -> OperationTelemetry | None:
+ return resolve_telemetry(self.msg.telemetry_id)
+
+ async def acquire_lock(self, ctx: RequestContext) -> bool:
+ self.scope = await SemanticLockScope.resolve(
+ self.msg.lock_handoff,
+ caller_lock=self.caller_lock,
+ fallback_path_factory=lambda: get_viking_fs()._uri_to_path(self.msg.uri, ctx=ctx),
+ )
+ return True
+
+ async def run_write(self, factory: Callable[[], Awaitable[_Result]]) -> _Result:
+ return await factory()
+
+ async def finish_processing(self, succeeded: bool) -> None:
+ assert self.scope is not None
+ await self.scope.close()
+
+ def failure_result(self, stats: DagStats | None) -> ProcessResult | None:
+ return None
+
+ async def reenqueue(self) -> None:
+ await self.processor._reenqueue_semantic_msg(self.msg)
+
+ async def reenqueue_after_error(self) -> None:
+ await self.reenqueue()
+
+ async def skip(self) -> None:
+ pass
+
+ async def cancel(self) -> None:
+ pass
+
+ async def cancel_queued(self) -> None:
+ if self.msg.telemetry_id and self.msg.id:
+ get_request_wait_tracker().mark_semantic_done(self.msg.telemetry_id, self.msg.id)
+ await self.processor._release_cancelled_semantic_lock(self.msg)
+
+ async def close(self) -> None:
+ pass
+
+
+class SkillSemanticMessageWork(SemanticMessageWork):
+ """Keep Skill tracking and package ownership alive until writes settle.
+
+ A failed run retains its lease for retry; cancellation drains started work
+ before update rollback can proceed. None of this state is shared by messages.
+ """
+
+ def __init__(
+ self,
+ processor: SemanticProcessor,
+ msg: SemanticMsg,
+ caller_lock: dict[str, Any] | None,
+ ) -> None:
+ super().__init__(processor, msg, caller_lock)
+ self._internal_tracking = False
+ self._lock_started = False
+ self._lock_closed = False
+
+ def start(self) -> None:
+ msg = self.msg
+ if not msg.telemetry_id:
+ msg.telemetry_id = OperationTelemetry(
+ operation="skill_index", enabled=False
+ ).telemetry_id
+ tracker = get_request_wait_tracker()
+ self._internal_tracking = not tracker.has_request(msg.telemetry_id)
+ tracker.retain_request(msg.telemetry_id, msg.id)
+ tracker.register_semantic_root(msg.telemetry_id, msg.id)
+ if self._internal_tracking:
+ tracker.cleanup(msg.telemetry_id)
+
+ def resolve_telemetry(self) -> OperationTelemetry:
+ collector = super().resolve_telemetry()
+ if collector is None:
+ collector = OperationTelemetry(operation="skill_index", enabled=False)
+ collector.telemetry_id = self.msg.telemetry_id
+ register_telemetry(collector)
+ return collector
+
+ async def acquire_lock(self, ctx: RequestContext) -> bool:
+ self._lock_started = True
+ self.scope = await self.processor._resolve_skill_semantic_lock(
+ self.msg, ctx, self.caller_lock
+ )
+ return self.scope.lock is not None
+
+ async def run_write(self, factory: Callable[[], Awaitable[_Result]]) -> _Result:
+ return await run_to_completion(factory)
+
+ async def finish_processing(self, succeeded: bool) -> None:
+ async def finish():
+ if not self.msg.skip_vectorization:
+ await get_request_wait_tracker().wait_for_embeddings(
+ self.msg.telemetry_id,
+ stop_waiting=self.processor._embedding_worker_stopped,
+ )
+ if succeeded:
+ assert self.scope is not None
+ await self.scope.close()
+ self._lock_closed = True
+
+ # On failure retain the lease until reenqueue hands it to the retry.
+ await run_to_completion(finish)
+
+ def failure_result(self, stats: DagStats | None) -> ProcessResult | None:
+ if stats is None or not stats.failures:
+ return None
+ for failure in stats.failures:
+ get_request_wait_tracker().mark_semantic_failed(
+ self.msg.telemetry_id, self.msg.id, failure
+ )
+ # Report every file error, but settle this queue item only once.
+ self.processor._merge_request_stats(self.msg.telemetry_id, error_count=len(stats.failures))
+ return ProcessResult.failed("\n".join(stats.failures))
+
+ async def reenqueue(self) -> None:
+ async def enqueue(queue, msg):
+ if self.scope is not None and self.scope.lock is not None and self.scope._owned:
+ await run_to_completion(
+ lambda: self.processor._enqueue_skill_retry(queue, msg, self.scope)
+ )
+ else:
+ await run_to_completion(lambda: queue.enqueue(msg))
+
+ # The shared circuit-breaker delay remains interruptible. Only the
+ # actual lock handoff and enqueue operation must finish once started.
+ await self.processor._reenqueue_semantic_msg(self.msg, enqueue=enqueue)
+
+ async def reenqueue_after_error(self) -> None:
+ try:
+ await self.reenqueue()
+ except asyncio.CancelledError:
+ await run_to_completion(
+ lambda: self.processor._release_cancelled_semantic_lock(self.msg)
+ )
+ get_request_wait_tracker().mark_semantic_done(self.msg.telemetry_id, self.msg.id)
+ raise
+
+ async def skip(self) -> None:
+ if self.msg.lock_handoff is not None:
+ await run_to_completion(
+ lambda: self.processor._release_cancelled_semantic_lock(self.msg)
+ )
+
+ async def cancel(self) -> None:
+ if not self._lock_started:
+ # Cancellation during the circuit-breaker delay precedes adoption.
+ await self.skip()
+ get_request_wait_tracker().mark_semantic_done(self.msg.telemetry_id, self.msg.id)
+
+ async def close(self) -> None:
+ if self.scope is not None and not self._lock_closed:
+ await run_to_completion(self.scope.close)
+ if self._internal_tracking and get_request_wait_tracker().is_complete(
+ self.msg.telemetry_id
+ ):
+ unregister_telemetry(self.msg.telemetry_id)
+
+ async def cancel_queued(self) -> None:
+ # Skill rollback must observe release before this item is settled.
+ await self.processor._release_cancelled_semantic_lock(self.msg)
+ if self.msg.telemetry_id and self.msg.id:
+ get_request_wait_tracker().mark_semantic_done(self.msg.telemetry_id, self.msg.id)
diff --git a/openviking/telemetry/request_wait_tracker.py b/openviking/telemetry/request_wait_tracker.py
index 76dc9d52c..fb1cd5ecc 100644
--- a/openviking/telemetry/request_wait_tracker.py
+++ b/openviking/telemetry/request_wait_tracker.py
@@ -8,7 +8,7 @@ import asyncio
import threading
import time
from dataclasses import dataclass, field
-from typing import Dict, List, Optional, Set
+from typing import Callable, Dict, List, Optional, Set
@dataclass
@@ -25,6 +25,8 @@ class _RequestWaitState:
embedding_error_count: int = 0
embedding_errors: List[str] = field(default_factory=list)
created_at: float = field(default_factory=time.time)
+ retained_roots: Set[str] = field(default_factory=set)
+ cleanup_requested: bool = False
class RequestWaitTracker:
@@ -56,6 +58,25 @@ class RequestWaitTracker:
def register_request(self, telemetry_id: str) -> None:
self._create_state(telemetry_id)
+ def has_request(self, telemetry_id: str) -> bool:
+ with self._lock:
+ return telemetry_id in self._states
+
+ def retain_request(self, telemetry_id: str, root_id: str) -> None:
+ """Keep a Skill worker's accounting alive if its HTTP waiter times out."""
+ if not telemetry_id or not root_id:
+ return
+ with self._lock:
+ state = self._states.setdefault(telemetry_id, _RequestWaitState())
+ state.retained_roots.add(root_id)
+
+ def _release_retained_root(
+ self, telemetry_id: str, root_id: str, state: _RequestWaitState
+ ) -> None:
+ state.retained_roots.discard(root_id)
+ if state.cleanup_requested and not state.retained_roots:
+ self._states.pop(telemetry_id, None)
+
def register_semantic_root(self, telemetry_id: str, root_id: str) -> None:
if not telemetry_id or not root_id:
return
@@ -105,6 +126,7 @@ class RequestWaitTracker:
return
state.pending_semantic_roots.discard(root_id)
state.semantic_processed += max(processed_delta, 0)
+ self._release_retained_root(telemetry_id, root_id, state)
def record_semantic_requeue(self, telemetry_id: str, delta: int = 1) -> None:
if not telemetry_id:
@@ -126,6 +148,7 @@ class RequestWaitTracker:
state.semantic_error_count += 1
if message:
state.semantic_errors.append(message)
+ self._release_retained_root(telemetry_id, root_id, state)
def mark_embedding_done(
self,
@@ -184,6 +207,28 @@ class RequestWaitTracker:
raise TimeoutError(f"Request processing not complete after {timeout}s")
await asyncio.sleep(poll_interval)
+ async def wait_for_embeddings(
+ self,
+ telemetry_id: str,
+ poll_interval: float = 0.05,
+ *,
+ stop_waiting: Optional[Callable[[], bool]] = None,
+ ) -> None:
+ """Drain embeddings while the producing semantic root remains pending."""
+ if not telemetry_id:
+ return
+ while True:
+ with self._lock:
+ state = self._states.get(telemetry_id)
+ if state is None or not state.pending_embedding_roots:
+ return
+ if stop_waiting is not None and stop_waiting():
+ # Shutdown has drained the embedding consumer's active writes.
+ # Leave this semantic delivery unacked for recovery rather than
+ # waiting forever for embeddings still in the persistent queue.
+ raise asyncio.CancelledError("Embedding worker stopped with queued work")
+ await asyncio.sleep(poll_interval)
+
def build_queue_status(self, telemetry_id: str) -> Dict[str, Dict[str, object]]:
with self._lock:
state = self._states.get(telemetry_id) or _RequestWaitState()
@@ -206,7 +251,11 @@ class RequestWaitTracker:
if not telemetry_id:
return
with self._lock:
- self._states.pop(telemetry_id, None)
+ state = self._states.get(telemetry_id)
+ if state is not None and state.retained_roots:
+ state.cleanup_requested = True
+ else:
+ self._states.pop(telemetry_id, None)
def get_request_wait_tracker() -> RequestWaitTracker:
diff --git a/openviking/utils/embedding_utils.py b/openviking/utils/embedding_utils.py
index 9488c6bb8..085f07a5b 100644
--- a/openviking/utils/embedding_utils.py
+++ b/openviking/utils/embedding_utils.py
@@ -365,6 +365,9 @@ async def vectorize_directory_meta(
ingest_options: IngestOptions | None = None,
creator_acl_grant: CreatorAclGrant | None = None,
include_abstract: bool = True,
+ meta: Optional[Dict[str, Any]] = None,
+ *,
+ content_is_body: bool = False,
) -> None:
"""
Vectorize directory metadata (.abstract.md and .overview.md).
@@ -374,8 +377,11 @@ async def vectorize_directory_meta(
# Callers may provide either freshly generated bodies or raw sidecar bytes
# read during reindex/import. Normalize at this shared boundary so protected
# operational metadata never leaks into vector text or rerank scalars.
- abstract = body_for_preview(abstract)
- overview = body_for_preview(overview)
+ # Skill producers have already extracted the bodies. Their Markdown may
+ # itself start with YAML frontmatter, which must not be parsed as OKF again.
+ if not content_is_body:
+ abstract = body_for_preview(abstract)
+ overview = body_for_preview(overview)
first_enqueue_error: Optional[Exception] = None
try:
if not ctx:
@@ -409,13 +415,12 @@ async def vectorize_directory_meta(
user=ctx.user,
account_id=ctx.account_id,
owner_space=owner_space,
+ meta=meta,
)
context_abstract.set_vectorize(
Vectorize(text=embedding_text_for_body(ContextLevel.ABSTRACT, uri, abstract))
)
- msg_abstract = EmbeddingMsgConverter.from_context(
- context_abstract, creator_acl_grant
- )
+ msg_abstract = EmbeddingMsgConverter.from_context(context_abstract, creator_acl_grant)
_apply_scalar_overrides(
msg_abstract,
(scalar_overrides or {}).get(int(ContextLevel.ABSTRACT.value)),
@@ -456,13 +461,12 @@ async def vectorize_directory_meta(
user=ctx.user,
account_id=ctx.account_id,
owner_space=owner_space,
+ meta=meta,
)
context_overview.set_vectorize(
Vectorize(text=embedding_text_for_body(ContextLevel.OVERVIEW, uri, overview))
)
- msg_overview = EmbeddingMsgConverter.from_context(
- context_overview, creator_acl_grant
- )
+ msg_overview = EmbeddingMsgConverter.from_context(context_overview, creator_acl_grant)
_apply_scalar_overrides(
msg_overview,
(scalar_overrides or {}).get(int(ContextLevel.OVERVIEW.value)),
diff --git a/openviking/utils/skill_processor.py b/openviking/utils/skill_processor.py
index 4460c5005..4ef064c30 100644
--- a/openviking/utils/skill_processor.py
+++ b/openviking/utils/skill_processor.py
@@ -17,9 +17,8 @@ from typing import Any, Dict, List, Optional
import yaml
-from openviking.core.context import Context, ContextType, Vectorize
from openviking.core.mcp_converter import is_mcp_format, mcp_to_skill
-from openviking.core.namespace import canonical_user_root, user_space_fragment
+from openviking.core.namespace import canonical_user_root
from openviking.core.skill_loader import SkillLoader
from openviking.privacy import (
UserPrivacyConfigService,
@@ -27,10 +26,10 @@ from openviking.privacy import (
)
from openviking.server.identity import RequestContext
from openviking.server.local_input_guard import deny_direct_local_skill_input
-from openviking.storage.vikingdb_manager import VikingDBManager
-from openviking.storage.queuefs.embedding_msg_converter import EmbeddingMsgConverter
+from openviking.service.task_tracker_concurrency import run_to_completion
from openviking.storage.viking_fs import VikingFS
-from openviking.telemetry import get_current_telemetry
+from openviking.storage.vikingdb_manager import VikingDBManager
+from openviking.telemetry import get_current_telemetry, register_telemetry
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
from openviking.utils.path_safety import safe_join_viking_uri
from openviking.utils.zip_safe import safe_extract_zip
@@ -118,6 +117,8 @@ class SkillProcessor:
apply_privacy: bool = True,
privacy_change_reason: str = "auto-extracted from add_skill",
target_uri: Optional[str] = None,
+ source_metadata: Optional[Dict[str, Any]] = None,
+ owner_lease_ref: Optional[Dict[str, Any]] = None,
lease_ref: Any = None,
) -> Dict[str, Any]:
"""
@@ -155,6 +156,8 @@ class SkillProcessor:
apply_privacy=apply_privacy,
privacy_change_reason=privacy_change_reason,
target_uri=target_uri,
+ source_metadata=source_metadata,
+ owner_lease_ref=owner_lease_ref,
lease_ref=lease_ref,
)
@@ -167,6 +170,8 @@ class SkillProcessor:
apply_privacy: bool = True,
privacy_change_reason: str = "auto-extracted from add_skill",
target_uri: Optional[str] = None,
+ source_metadata: Optional[Dict[str, Any]] = None,
+ owner_lease_ref: Optional[Dict[str, Any]] = None,
lease_ref: Any = None,
) -> Dict[str, Any]:
cleanup_path = preparation.cleanup_path
@@ -174,78 +179,102 @@ class SkillProcessor:
auxiliary_files = preparation.auxiliary_files
base_path = preparation.base_path
telemetry = get_current_telemetry()
+ lease = None
+ # Training supplies its enclosing tree lease. Acquire a separate package
+ # reference so background indexing never takes ownership of that lease.
+ if owner_lease_ref is None:
+ owner_lease_ref = lease_ref
try:
- if apply_privacy:
- skill_dict = await self.apply_skill_privacy(
- skill_dict,
- preparation.privacy_values,
- ctx,
- change_reason=privacy_change_reason,
- delete_if_empty=False,
+ effective_root_uri = self._resolve_skill_root_uri(ctx, target_uri)
+ skill_dir_uri = f"{effective_root_uri}/{skill_dict['name']}"
+
+ async def acquire_package_lock() -> None:
+ nonlocal lease
+ lease = await viking_fs._async_agfs.pathlock_acquire_tree(
+ viking_fs._uri_to_path(skill_dir_uri, ctx=ctx),
+ **({"owner_lease_ref": owner_lease_ref} if owner_lease_ref is not None else {}),
)
- skill_abstract = self._build_skill_abstract(skill_dict)
- effective_root_uri = self._resolve_skill_root_uri(ctx, target_uri)
- context = Context(
- uri=f"{effective_root_uri}/{skill_dict['name']}",
- parent_uri=effective_root_uri,
- is_leaf=False,
- abstract=skill_abstract,
- context_type=ContextType.SKILL.value,
- user=ctx.user,
- account_id=ctx.account_id,
- owner_space=user_space_fragment(ctx),
- meta={
- "name": skill_dict["name"],
- "description": skill_dict.get("description", ""),
- "allowed_tools": skill_dict.get("allowed_tools", []),
- "tags": skill_dict.get("tags", []),
- "source_path": skill_dict.get("source_path", ""),
- },
- )
- context.set_vectorize(Vectorize(text=context.abstract))
+ await run_to_completion(acquire_package_lock)
- skill_dir_uri = context.uri
+ # Preparation above is read-only. A rejected package writer must
+ # not change its privacy config, and cancellation must let an
+ # in-flight config write finish before releasing the package lock.
+ if apply_privacy:
+ skill_dict = await run_to_completion(
+ lambda: self.apply_skill_privacy(
+ skill_dict,
+ preparation.privacy_values,
+ ctx,
+ change_reason=privacy_change_reason,
+ delete_if_empty=False,
+ )
+ )
+ skill_abstract = self._build_skill_abstract(skill_dict)
write_start = time.perf_counter()
- await self._write_skill_content(
- viking_fs=viking_fs,
- skill_dict=skill_dict,
- skill_dir_uri=skill_dir_uri,
- abstract=skill_abstract,
- overview=skill_dict.get("content", ""),
- ctx=ctx,
- lease_ref=lease_ref,
+ await run_to_completion(
+ lambda: self._write_skill_content(
+ viking_fs=viking_fs,
+ skill_dict=skill_dict,
+ skill_dir_uri=skill_dir_uri,
+ abstract=skill_abstract,
+ overview=skill_dict.get("content", ""),
+ ctx=ctx,
+ lease_ref=lease,
+ )
)
- await self._write_auxiliary_files(
- viking_fs=viking_fs,
- auxiliary_files=auxiliary_files,
- base_path=base_path,
- skill_dir_uri=skill_dir_uri,
- ctx=ctx,
- lease_ref=lease_ref,
+ await run_to_completion(
+ lambda: self._write_auxiliary_files(
+ viking_fs=viking_fs,
+ auxiliary_files=auxiliary_files,
+ base_path=base_path,
+ skill_dir_uri=skill_dir_uri,
+ ctx=ctx,
+ lease_ref=lease,
+ )
)
telemetry.set(
"skill.write.duration_ms", round((time.perf_counter() - write_start) * 1000, 3)
)
- index_start = time.perf_counter()
- await self._index_skill(
- context=context,
- skill_dir_uri=skill_dir_uri,
- )
- telemetry.set(
- "skill.index.duration_ms", round((time.perf_counter() - index_start) * 1000, 3)
- )
- return {
+ result = {
"status": "success",
"root_uri": skill_dir_uri,
"uri": skill_dir_uri,
"name": skill_dict["name"],
"auxiliary_files": len(auxiliary_files),
}
+ if source_metadata:
+ from openviking.server.skill_source_metadata import write_skill_source_metadata
+
+ await run_to_completion(
+ lambda: write_skill_source_metadata(
+ viking_fs, ctx, result, source_metadata, lease_ref=lease
+ )
+ )
+ index_start = time.perf_counter()
+
+ async def enqueue_package() -> None:
+ nonlocal lease
+ await self._enqueue_skill_package(
+ skill_dir_uri,
+ viking_fs,
+ ctx,
+ lease,
+ source_path=skill_dict.get("source_path", ""),
+ )
+ lease = None # The semantic worker owns the handed-off lease.
+
+ await run_to_completion(enqueue_package)
+ telemetry.set(
+ "skill.index.duration_ms", round((time.perf_counter() - index_start) * 1000, 3)
+ )
+ return result
finally:
+ if lease is not None:
+ await run_to_completion(lambda: viking_fs._async_agfs.pathlock_release(lease))
if cleanup_path:
shutil.rmtree(cleanup_path, ignore_errors=True)
@@ -393,9 +422,7 @@ class SkillProcessor:
return normalized
@staticmethod
- def _resolve_skill_root_uri(
- ctx: RequestContext, target_uri: Optional[str]
- ) -> str:
+ def _resolve_skill_root_uri(ctx: RequestContext, target_uri: Optional[str]) -> str:
"""Resolve the skill storage root URI.
Defaults to the per-user private skills root. Callers may pass
@@ -472,6 +499,7 @@ class SkillProcessor:
*,
change_reason: str,
delete_if_empty: bool,
+ owner_lease_ref: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
if not self._privacy_config_service:
return skill_dict
@@ -484,11 +512,17 @@ class SkillProcessor:
values=privacy_values,
updated_by=ctx.user.user_id,
change_reason=change_reason,
+ **({"owner_lease_ref": owner_lease_ref} if owner_lease_ref is not None else {}),
)
return skill_dict
if delete_if_empty:
- await self._privacy_config_service.delete(ctx, "skill", skill_dict["name"])
+ await self._privacy_config_service.delete(
+ ctx,
+ "skill",
+ skill_dict["name"],
+ **({"owner_lease_ref": owner_lease_ref} if owner_lease_ref is not None else {}),
+ )
return skill_dict
async def sanitize_skill_privacy(
@@ -535,18 +569,26 @@ class SkillProcessor:
abstract: str,
overview: str,
ctx: RequestContext,
- lease_ref: Any = None,
+ lease_ref: Dict[str, Any],
):
"""Write main skill content to VikingFS."""
- await viking_fs.write_context(
- uri=skill_dir_uri,
- content=SkillLoader.to_skill_md(skill_dict),
+ from openviking.storage.abstract_overview import write_abstract_overview
+
+ await viking_fs.write_file(
+ f"{skill_dir_uri}/SKILL.md",
+ SkillLoader.to_skill_md(skill_dict),
+ ctx=ctx,
+ lease_ref=lease_ref,
+ )
+ await write_abstract_overview(
+ viking_fs=viking_fs,
+ dir_uri=skill_dir_uri,
abstract=abstract,
overview=overview,
- content_filename="SKILL.md",
- is_leaf=False,
ctx=ctx,
- lease_ref=lease_ref,
+ lock=lease_ref,
+ is_stale=lambda: False,
+ metadata={"generated_by": {"component": "SkillProcessor", "trigger": "skill_ingest"}},
)
async def _write_auxiliary_files(
@@ -556,7 +598,7 @@ class SkillProcessor:
base_path: Optional[Path],
skill_dir_uri: str,
ctx: RequestContext,
- lease_ref: Any = None,
+ lease_ref: Optional[Dict[str, Any]] = None,
):
"""Write auxiliary files to VikingFS."""
for aux_file in auxiliary_files:
@@ -576,28 +618,65 @@ class SkillProcessor:
if is_text:
await viking_fs.write_file(
- aux_uri, file_bytes.decode("utf-8"), ctx=ctx, lease_ref=lease_ref
+ aux_uri,
+ file_bytes.decode("utf-8"),
+ ctx=ctx,
+ lease_ref=lease_ref,
)
else:
- await viking_fs.write_file_bytes(aux_uri, file_bytes, ctx=ctx, lease_ref=lease_ref)
-
- async def _index_skill(self, context: Context, skill_dir_uri: str):
- """Write skill directory vector via async queue as L0."""
- context.uri = skill_dir_uri
- context.is_leaf = False
- context.level = 0
-
- context.set_vectorize(Vectorize(text=context.abstract))
- embedding_msg = EmbeddingMsgConverter.from_context(context)
- if embedding_msg:
- if embedding_msg.telemetry_id:
- get_request_wait_tracker().register_embedding_root(
- embedding_msg.telemetry_id, embedding_msg.id
- )
- enqueued = await self.vikingdb.enqueue_embedding_msg(embedding_msg)
- if not enqueued and embedding_msg.telemetry_id:
- get_request_wait_tracker().mark_embedding_failed(
- embedding_msg.telemetry_id,
- embedding_msg.id,
- "embedding enqueue returned false",
+ await viking_fs.write_file_bytes(
+ aux_uri,
+ file_bytes,
+ ctx=ctx,
+ lease_ref=lease_ref,
)
+
+ async def _enqueue_skill_package(
+ self,
+ uri: str,
+ viking_fs: VikingFS,
+ ctx: RequestContext,
+ lease: Dict[str, Any],
+ *,
+ source_path: str = "",
+ ) -> None:
+ from openviking.storage.queuefs import get_queue_manager
+ from openviking.storage.queuefs.semantic_msg import SemanticMsg
+
+ telemetry = get_current_telemetry()
+ register_telemetry(telemetry)
+ tracker = get_request_wait_tracker()
+ tracker.register_request(telemetry.telemetry_id)
+ msg = SemanticMsg(
+ uri=uri,
+ context_type="skill",
+ recursive=True,
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ group_ids=ctx.group_ids,
+ role=str(ctx.role),
+ telemetry_id=telemetry.telemetry_id,
+ lock_handoff=await viking_fs._async_agfs.pathlock_to_handoff(lease),
+ generation_trigger="skill_ingest",
+ propagate_to_parent=False,
+ source={"path": source_path},
+ )
+ tracker.register_semantic_root(msg.telemetry_id, msg.id)
+ tracker.retain_request(msg.telemetry_id, msg.id)
+ handed_off = False
+ try:
+ queue_manager = get_queue_manager()
+ # Consumers, including cancelled-message cleanup, must never see
+ # a handoff that the producer has not made ready yet.
+ await viking_fs._async_agfs.pathlock_handoff(lease)
+ handed_off = True
+ await queue_manager.get_queue(queue_manager.SEMANTIC, allow_create=True).enqueue(msg)
+ except BaseException as exc:
+ tracker.mark_semantic_failed(msg.telemetry_id, msg.id, str(exc))
+ if handed_off:
+ # Enqueue was rejected/failed. Return ownership to the caller's
+ # finally block; adopting creates a new owned lease reference.
+ reclaimed = await viking_fs._async_agfs.pathlock_adopt(msg.lock_handoff)
+ lease.clear()
+ lease.update(reclaimed)
+ raise
diff --git a/tests/retrieve/test_skill_mixed_pagination.py b/tests/retrieve/test_skill_mixed_pagination.py
new file mode 100644
index 000000000..16c49abae
--- /dev/null
+++ b/tests/retrieve/test_skill_mixed_pagination.py
@@ -0,0 +1,38 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""General search keeps item results; package grouping belongs to the Skills API."""
+
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import pytest
+
+from openviking.retrieve.hierarchical_retriever import HierarchicalRetriever, RetrieverMode
+from openviking.storage.viking_fs._semantic import _SemanticMixin
+from openviking_cli.retrieve.types import TypedQuery
+from tests.retrieve.test_skill_package_results import SKILLS, PagedStore, ctx, row
+
+
[email protected]("route", ["quick", "thinking", "filter"])
+async def test_general_search_preserves_multiple_hits_from_one_skill(route):
+ records = [row(f"{SKILLS}/a/{i}.md", 0.99 - i / 1000) for i in range(12)]
+ records.append(row("viking://resources/doc.md", 0.8, kind="resource"))
+ store = PagedStore(records)
+ if route == "filter":
+ fs = SimpleNamespace(_get_vector_store=lambda: store, _ensure_retrieval_scope=AsyncMock())
+ result = await _SemanticMixin._find_by_filter(
+ fs, {"op": "must", "field": "search_tags", "conds": ["team=x"]}, ctx(), [], 2
+ )
+ matches = result.skills
+ else:
+ # Thinking obtains files through the ACL-aware global leaf query.
+ store._acl_enabled = lambda ctx: True
+ result = await HierarchicalRetriever(store, None).retrieve(
+ TypedQuery("hello", None, ""),
+ ctx(),
+ limit=2,
+ mode=RetrieverMode.THINKING if route == "thinking" else RetrieverMode.QUICK,
+ )
+ matches = result.matched_contexts
+ assert [item.uri for item in matches] == [f"{SKILLS}/a/0.md", f"{SKILLS}/a/1.md"]
+ assert all(call.get("offset", 0) == 0 for call in store.calls)
diff --git a/tests/retrieve/test_skill_package_results.py b/tests/retrieve/test_skill_package_results.py
new file mode 100644
index 000000000..0b2305fff
--- /dev/null
+++ b/tests/retrieve/test_skill_package_results.py
@@ -0,0 +1,186 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Skill package grouping, original hit preservation and retrieval completeness."""
+
+import pytest
+
+from openviking.retrieve.skill_package_retriever import SkillPackageRetriever
+from openviking.retrieve.skill_results import (
+ SkillResultResolver,
+ skill_root_uri,
+)
+from openviking.server.identity import RequestContext, Role
+from openviking_cli.exceptions import NotFoundError, PermissionDeniedError
+from openviking_cli.retrieve.types import (
+ ContextType,
+ TypedQuery,
+)
+from openviking_cli.session.user_id import UserIdentifier
+
+SKILLS = "viking://agent/skills"
+
+
+def ctx():
+ return RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
+
+
+def row(uri, score, level=2, abstract="file summary", kind="skill"):
+ return {"uri": uri, "_score": score, "level": level, "abstract": abstract, "context_type": kind}
+
+
+class Files:
+ def __init__(self, denied=(), missing=()):
+ self.denied = set(denied)
+ self.missing = set(missing)
+ self.stat_calls = []
+
+ async def stat(self, uri, ctx=None, skip_count=False):
+ assert skip_count
+ self.stat_calls.append(uri)
+ if uri in self.denied:
+ raise PermissionDeniedError("root denied", resource=uri)
+ if uri in self.missing:
+ raise NotFoundError(uri)
+ return {"isDir": True}
+
+ async def abstract(self, *args, **kwargs):
+ pytest.fail("Skill grouping must not fetch or substitute root metadata")
+
+ async def read(self, *args, **kwargs):
+ pytest.fail("Skill grouping must not build an additional content preview")
+
+
+class PagedStore:
+ collection_name = "context"
+
+ def __init__(self, records=(), children=None):
+ self.records = list(records)
+ self.children = children or {}
+ self.calls = []
+ self.child_calls = []
+
+ def _acl_enabled(self, ctx):
+ return False
+
+ async def collection_exists_bound(self):
+ return True
+
+ def _page(self, records, kwargs):
+ levels = kwargs.get("level")
+ targets = kwargs.get("target_directories")
+ filtered = [
+ dict(record)
+ for record in records
+ if (not kwargs.get("context_type") or record["context_type"] == kwargs["context_type"])
+ and (levels is None or record["level"] in levels)
+ and (
+ not targets
+ or any(
+ record["uri"] == root or record["uri"].startswith(root + "/")
+ for root in targets
+ )
+ )
+ ]
+ filtered.sort(key=lambda item: item["_score"], reverse=True)
+ offset = kwargs.get("offset", 0)
+ return filtered[offset : offset + kwargs.get("limit", 10)]
+
+ async def search_in_tenant(self, ctx, **kwargs):
+ self.calls.append(kwargs)
+ return self._page(self.records, kwargs)
+
+ async def filter_in_tenant(self, ctx, **kwargs):
+ self.calls.append(kwargs)
+ return self._page(self.records, kwargs)
+
+ async def search_children_in_tenant(self, ctx, parent_uri, **kwargs):
+ self.child_calls.append({"parent_uri": parent_uri, **kwargs})
+ return self._page(self.children.get(parent_uri, []), kwargs)
+
+
+def query(target=SKILLS):
+ return TypedQuery("backup recovery", ContextType.SKILL, "", target_directories=[target])
+
+
+async def test_quick_fills_distinct_skills_beyond_fixed_overfetch():
+ records = [row(SKILLS, 1, 0), row(f"{SKILLS}/.a.update-backup-123/SKILL.md", 1)]
+ records.extend(row(f"{SKILLS}/a/ref/{i}.md", 0.99 - i / 1000) for i in range(120))
+ records.append(row(f"{SKILLS}/b/ref/backup.md", 0.5))
+ store = PagedStore(records)
+ files = Files()
+ result = await SkillPackageRetriever(store, None).retrieve_skills(
+ query(),
+ ctx(),
+ limit=2,
+ skill_resolver=SkillResultResolver(files, ctx()),
+ )
+ assert [skill_root_uri(item.uri) for item in result.matched_contexts] == [
+ f"{SKILLS}/a",
+ f"{SKILLS}/b",
+ ]
+ assert result.matched_contexts[0].uri == f"{SKILLS}/a/ref/0.md"
+ assert result.matched_contexts[0].level == 2
+ assert result.matched_contexts[0].abstract == "file summary"
+ assert result.matched_contexts[1].score == 0.5
+ assert [call["offset"] for call in store.calls] == list(range(0, len(records), 10))
+ assert files.stat_calls == [f"{SKILLS}/a", f"{SKILLS}/b"]
+
+
+async def test_quick_fills_after_root_denial_and_missing_root():
+ records = [row(f"{SKILLS}/denied/ref/{i}.md", 0.99) for i in range(10)]
+ records.extend(row(f"{SKILLS}/missing/ref/{i}.md", 0.9, abstract="") for i in range(10))
+ records.extend([row(f"{SKILLS}/b/SKILL.md", 0.8), row(f"{SKILLS}/c/SKILL.md", 0.7)])
+ files = Files(denied=[f"{SKILLS}/denied"], missing=[f"{SKILLS}/missing"])
+ store = PagedStore(records)
+ result = await SkillPackageRetriever(store, None).retrieve_skills(
+ query(),
+ ctx(),
+ limit=2,
+ skill_resolver=SkillResultResolver(files, ctx()),
+ )
+ assert [item.uri for item in result.matched_contexts] == [
+ f"{SKILLS}/b/SKILL.md",
+ f"{SKILLS}/c/SKILL.md",
+ ]
+ assert [call["offset"] for call in store.calls] == [0, 10, 20]
+
+
+ ("level", "relative_uri", "score", "abstract"),
+ [
+ (0, "reference/.abstract.md", 0.92, "directory abstract"),
+ (1, "reference/.overview.md", 0.95, "directory overview"),
+ (2, "reference/SKILL.md", 0.8, "nested definition"),
+ ],
+)
+async def test_scoped_hits_preserve_requested_level_and_do_not_use_outside_scores(
+ level, relative_uri, score, abstract
+):
+ root = f"{SKILLS}/demo"
+ store = PagedStore(
+ [
+ row(root, 1, 0, "name: wrong"),
+ row(f"{root}/other.md", 0.99),
+ row(f"{root}/reference", 0.92, 0, "directory abstract"),
+ row(f"{root}/reference", 0.95, 1, "directory overview"),
+ row(f"{root}/reference/SKILL.md", 0.8, abstract="nested definition"),
+ ]
+ )
+ filters = {"op": "must", "field": "search_tags", "conds": ["team=infra"]}
+ result = await SkillPackageRetriever(store, None).retrieve_skills(
+ query(f"{root}/reference"),
+ ctx(),
+ limit=2,
+ level=[level],
+ scope_dsl=filters,
+ skill_resolver=SkillResultResolver(Files(), ctx()),
+ )
+ [matched] = result.matched_contexts
+ assert matched.uri == f"{root}/{relative_uri}"
+ assert matched.level == level
+ assert matched.score == score
+ assert matched.abstract == abstract
+ assert store.calls[0]["extra_filter"] is filters
diff --git a/tests/server/test_admin_rebuild_api.py b/tests/server/test_admin_rebuild_api.py
index 037d50d15..4762ba1c3 100644
--- a/tests/server/test_admin_rebuild_api.py
+++ b/tests/server/test_admin_rebuild_api.py
@@ -3088,58 +3088,6 @@ async def test_reindex_global_namespace_partitions_user_and_resources(monkeypatc
assert seen["resource_files"] == ["viking://resources/demo.txt"]
-async def test_reindex_skill_l2_falls_back_to_skill_content_when_abstract_missing(monkeypatch):
- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters
-
- class FakeVikingFS:
- async def exists(self, uri, ctx=None):
- return True
-
- seen = {}
-
- async def fake_read_directory_abstract(self, uri, *, ctx):
- return "skill abstract"
-
- async def fake_read_directory_overview(self, uri, *, ctx):
- return ""
-
- async def fake_fetch_existing_record(self, *, uri, level, ctx):
- return None
-
- async def fake_safe_read_text(self, uri, *, ctx):
- return "# Skill Title\n\nDo things well."
-
- async def fake_upsert_context(self, **kwargs):
- seen[kwargs["uri"]] = kwargs
-
- async def fake_skill_meta(self, *, uri, abstract, ctx):
- return {"name": "my_skill", "description": abstract}
-
- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())
- monkeypatch.setattr(ReindexExecutor, "_read_directory_abstract", fake_read_directory_abstract)
- monkeypatch.setattr(ReindexExecutor, "_read_directory_overview", fake_read_directory_overview)
- monkeypatch.setattr(ReindexExecutor, "_fetch_existing_record", fake_fetch_existing_record)
- monkeypatch.setattr(ReindexExecutor, "_safe_read_text", fake_safe_read_text)
- monkeypatch.setattr(ReindexExecutor, "_skill_meta", fake_skill_meta)
- monkeypatch.setattr(ReindexExecutor, "_upsert_context", fake_upsert_context)
-
- service = ReindexExecutor()
- counters = _ReindexCounters()
- ctx = RequestContext(
- user=UserIdentifier(account_id="test", user_id="alice"),
- role=Role.ROOT,
- )
-
- await service._reindex_skill_vectors(
- uri="viking://user/skills/my_skill",
- counters=counters,
- ctx=ctx,
- )
-
- assert seen["viking://user/skills/my_skill/SKILL.md"]["abstract"] == "skill abstract"
-
-
@pytest.mark.asyncio
async def test_reindex_skill_vectors_non_recursive_skips_skill_detail(monkeypatch):
from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters
@@ -3162,15 +3110,17 @@ async def test_reindex_skill_vectors_non_recursive_skips_skill_detail(monkeypatc
upserts = []
- async def fake_upsert_context(self, **kwargs):
- upserts.append((kwargs["uri"], kwargs["level"]))
+ async def fake_vectorize_directory(uri, abstract, overview, **kwargs):
+ upserts.extend([(uri, ContextLevel.ABSTRACT), (uri, ContextLevel.OVERVIEW)])
monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())
monkeypatch.setattr(ReindexExecutor, "_read_directory_abstract", fake_read_directory_abstract)
monkeypatch.setattr(ReindexExecutor, "_read_directory_overview", fake_read_directory_overview)
monkeypatch.setattr(ReindexExecutor, "_safe_read_text", fake_safe_read_text)
monkeypatch.setattr(ReindexExecutor, "_skill_meta", fake_skill_meta)
- monkeypatch.setattr(ReindexExecutor, "_upsert_context", fake_upsert_context)
+ monkeypatch.setattr(
+ "openviking.service.reindex_executor.vectorize_directory_meta", fake_vectorize_directory
+ )
counters = _ReindexCounters()
ctx = RequestContext(
diff --git a/tests/server/test_api_skills.py b/tests/server/test_api_skills.py
index 14df14bdc..7662cfcee 100644
--- a/tests/server/test_api_skills.py
+++ b/tests/server/test_api_skills.py
@@ -4,6 +4,7 @@
import sys
import types
import zipfile
+from unittest.mock import AsyncMock
import pytest
from starlette.responses import PlainTextResponse
@@ -261,22 +262,23 @@ async def test_skills_api_update_rolls_back_when_replace_fails(client, monkeypat
raise RuntimeError("source metadata write failed")
monkeypatch.setattr(
- "openviking.server.routers.skills.persist_skill_source_metadata",
+ "openviking.server.skill_source_metadata.write_skill_source_metadata",
_fail_persist,
)
- with pytest.raises(RuntimeError, match="source metadata write failed"):
- await client.put(
- "/api/v1/skills/rollback-skill",
- json={
- "data": _skill_md(
- "rollback-skill",
- "Updated description",
- "This update should be rolled back.",
- ),
- "wait": True,
- },
- )
+ response = await client.put(
+ "/api/v1/skills/rollback-skill",
+ json={
+ "data": _skill_md(
+ "rollback-skill",
+ "Updated description",
+ "This update should be rolled back.",
+ ),
+ "wait": True,
+ },
+ )
+ assert response.status_code == 500, response.text
+ assert response.json()["error"]["code"] == "INTERNAL"
show_response = await client.get(
"/api/v1/skills/rollback-skill",
@@ -337,32 +339,38 @@ async def test_skills_api_update_restores_previous_privacy_on_failure(client, mo
raise RuntimeError("source metadata write failed")
monkeypatch.setattr(
- "openviking.server.routers.skills.persist_skill_source_metadata",
+ "openviking.server.skill_source_metadata.write_skill_source_metadata",
_fail_persist,
)
- with pytest.raises(RuntimeError, match="source metadata write failed"):
- await client.put(
- "/api/v1/skills/rollback-privacy-skill",
- json={
- "data": _skill_md(
- "rollback-privacy-skill",
- "Updated description",
- 'api_key: "secret-new"\n',
- ),
- "wait": True,
- },
- )
+ response = await client.put(
+ "/api/v1/skills/rollback-privacy-skill",
+ json={
+ "data": _skill_md(
+ "rollback-privacy-skill",
+ "Updated description",
+ 'api_key: "secret-new"\n',
+ ),
+ "wait": True,
+ },
+ )
+ assert response.status_code == 500, response.text
+ assert response.json()["error"]["code"] == "INTERNAL"
privacy_response = await client.get("/api/v1/privacy-configs/skill/rollback-privacy-skill")
assert privacy_response.status_code == 200, privacy_response.text
- assert privacy_response.json()["result"]["values"]["api_key"] == "secret-old"
+ assert privacy_response.json()["result"]["current"]["values"]["api_key"] == "secret-old"
async def test_skills_api_update_restores_previous_privacy_after_privacy_write(client, monkeypatch):
+ from openviking.storage.queuefs import get_queue_manager
from openviking.utils.skill_processor import SkillProcessor
await _add_skill(client, "rollback-privacy-after-write-skill", "Original description")
+ queue_manager = get_queue_manager()
+ queue = queue_manager.get_queue(queue_manager.SEMANTIC)
+ enqueue = AsyncMock(wraps=queue.enqueue)
+ monkeypatch.setattr(queue, "enqueue", enqueue)
seeded_privacy = await client.post(
"/api/v1/privacy-configs/skill/rollback-privacy-after-write-skill",
@@ -370,6 +378,9 @@ async def test_skills_api_update_restores_previous_privacy_after_privacy_write(c
)
assert seeded_privacy.status_code == 200, seeded_privacy.text
+ before = (await client.get("/api/v1/skills/rollback-privacy-after-write-skill")).json()[
+ "result"
+ ]
original_prepare = SkillProcessor.prepare_skill_privacy
original_apply = SkillProcessor.apply_skill_privacy
@@ -386,6 +397,7 @@ async def test_skills_api_update_restores_previous_privacy_after_privacy_write(c
*,
change_reason,
delete_if_empty,
+ owner_lease_ref=None,
):
result = await original_apply(
self,
@@ -394,6 +406,7 @@ async def test_skills_api_update_restores_previous_privacy_after_privacy_write(c
ctx,
change_reason=change_reason,
delete_if_empty=delete_if_empty,
+ owner_lease_ref=owner_lease_ref,
)
if skill_dict.get("name") == "rollback-privacy-after-write-skill":
raise RuntimeError("privacy post-write failure")
@@ -402,24 +415,26 @@ async def test_skills_api_update_restores_previous_privacy_after_privacy_write(c
monkeypatch.setattr(SkillProcessor, "prepare_skill_privacy", _prepare_new_privacy)
monkeypatch.setattr(SkillProcessor, "apply_skill_privacy", _apply_then_fail)
- with pytest.raises(RuntimeError, match="privacy post-write failure"):
- await client.put(
- "/api/v1/skills/rollback-privacy-after-write-skill",
- json={
- "data": _skill_md(
- "rollback-privacy-after-write-skill",
- "Updated description",
- 'api_key: "secret-new"\n',
- ),
- "wait": True,
- },
- )
+ response = await client.put(
+ "/api/v1/skills/rollback-privacy-after-write-skill",
+ json={
+ "data": _skill_md(
+ "rollback-privacy-after-write-skill",
+ "Updated description",
+ 'api_key: "secret-new"\n',
+ ),
+ "wait": True,
+ },
+ )
+ assert response.status_code == 500, response.text
+ assert response.json()["error"]["code"] == "INTERNAL"
+ enqueue.assert_not_awaited()
privacy_response = await client.get(
"/api/v1/privacy-configs/skill/rollback-privacy-after-write-skill"
)
assert privacy_response.status_code == 200, privacy_response.text
- assert privacy_response.json()["result"]["values"]["api_key"] == "secret-old"
+ assert privacy_response.json()["result"]["current"]["values"]["api_key"] == "secret-old"
show_response = await client.get(
"/api/v1/skills/rollback-privacy-after-write-skill",
@@ -427,7 +442,8 @@ async def test_skills_api_update_restores_previous_privacy_after_privacy_write(c
)
assert show_response.status_code == 200, show_response.text
shown = show_response.json()["result"]
- assert shown["description"] == "Original description"
+ for field in ("description", "abstract", "overview", "content"):
+ assert shown[field] == before[field]
assert "secret-new" not in shown["content"]
@@ -681,3 +697,125 @@ async def test_skills_api_validate_rfc_strict_and_loose_rules(client):
long_body_result = long_body_response.json()["result"]
assert long_body_result["valid"] is True
assert any(issue["rule"] == "body_max_lines" for issue in long_body_result["warnings"])
+
+
+async def test_skill_package_indexes_nested_content_and_returns_actual_hit(
+ client, service, tmp_path
+):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage.expr import Eq
+ from openviking_cli.session.user_id import UserIdentifier
+
+ archive = tmp_path / "package-search.zip"
+ with zipfile.ZipFile(archive, "w") as package:
+ skill = _skill_md("package-search", "Root skill description")
+ package.writestr(
+ "SKILL.md", skill.replace("tags:\n", "allowed-tools: [Read, Bash]\ntags:\n")
+ )
+ package.writestr("reference/nested/recovery.md", "Restore a backup in another region.")
+ package.writestr("reference/nested/recovery-copy.md", "Restore a backup in another region.")
+ package.writestr("reference/SKILL.md", _skill_md("attachment", "An ordinary attachment"))
+ with archive.open("rb") as handle:
+ uploaded = await client.post(
+ "/api/v1/resources/temp_upload",
+ files={"file": (archive.name, handle, "application/zip")},
+ )
+ assert uploaded.status_code == 200, uploaded.text
+ added = await client.post(
+ "/api/v1/skills",
+ json={"temp_file_id": uploaded.json()["result"]["temp_file_id"], "wait": True},
+ )
+ assert added.status_code == 200, added.text
+ root = added.json()["result"]["root_uri"]
+ # Both root summary levels retain the package identity in the stored index.
+ records = await service.vikingdb_manager.filter(
+ filter=Eq("uri", root),
+ output_fields=["level", "name", "description"],
+ ctx=RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT),
+ )
+ assert {record["level"] for record in records} == {0, 1}
+ for record in records:
+ assert record["name"] == "package-search"
+ assert record["description"] == "Root skill description"
+ for directory in (root, f"{root}/reference", f"{root}/reference/nested"):
+ for endpoint in ("abstract", "overview"):
+ content = await client.get(f"/api/v1/content/{endpoint}", params={"uri": directory})
+ assert content.status_code == 200, content.text
+ assert content.json()["result"]
+
+ for endpoint in ("/api/v1/skills/find", "/api/v1/search/find"):
+ found = await client.post(
+ endpoint,
+ json={
+ "query": "backup recovery",
+ "target_uri": f"{root}/reference/nested",
+ "level": [2],
+ "limit": 10,
+ },
+ )
+ assert found.status_code == 200, found.text
+ hits = found.json()["result"]["skills"]
+ assert len(hits) == (1 if endpoint == "/api/v1/skills/find" else 2)
+ expected_uris = {
+ f"{root}/reference/nested/recovery.md",
+ f"{root}/reference/nested/recovery-copy.md",
+ }
+ assert all(hit["uri"] in expected_uris and hit["level"] == 2 for hit in hits)
+ if endpoint == "/api/v1/skills/find":
+ assert hits[0]["root_uri"] == root
+ assert hits[0]["skill_md_uri"] == f"{root}/SKILL.md"
+ assert hits[0]["description"] == "Root skill description"
+ assert hits[0]["tags"] == ["test"]
+ assert hits[0]["allowed_tools"] == ["Read", "Bash"]
+
+ listed = await client.get("/api/v1/skills")
+ assert [skill["name"] for skill in listed.json()["result"]["skills"]] == ["package-search"]
+ deleted = await client.delete("/api/v1/skills/package-search")
+ assert deleted.status_code == 200, deleted.text
+ found = await client.post("/api/v1/skills/find", json={"query": "backup recovery"})
+ assert found.json()["result"]["skills"] == []
+
+
+async def test_mcp_skill_import_and_vector_rebuild_preserve_l1_frontmatter(client, service):
+ from openviking.core.mcp_converter import mcp_to_skill
+ from openviking.server.identity import RequestContext, Role
+ from openviking_cli.session.user_id import UserIdentifier
+
+ tool = {
+ "name": "review_calculator",
+ "description": "Perform mathematical calculations",
+ "inputSchema": {
+ "type": "object",
+ "properties": {"expression": {"type": "string", "description": "Expression"}},
+ "required": ["expression"],
+ },
+ }
+ added = await client.post("/api/v1/skills", json={"data": tool, "wait": True, "timeout": 10})
+ assert added.status_code == 200, added.text
+ root = added.json()["result"]["root_uri"]
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)
+ expected_body = mcp_to_skill(tool)["content"].strip()
+
+ async def assert_indexed_body():
+ for level in (0, 1):
+ records = await service.vikingdb_manager.filter_in_tenant(
+ ctx=ctx, target_directories=[root], level=[level]
+ )
+ assert len(records) == 1
+ if level == 1:
+ assert records[0]["abstract"].strip() == expected_body
+ records = await service.vikingdb_manager.get_context_by_uri(
+ f"{root}/SKILL.md", level=2, ctx=ctx
+ )
+ assert len(records) == 1
+
+ await assert_indexed_body()
+ rebuilt = await client.post(
+ "/api/v1/content/reindex",
+ json={"uri": root, "mode": "vectors_only", "wait": True},
+ )
+ assert rebuilt.status_code == 200, rebuilt.text
+ assert rebuilt.json()["result"]["status"] == "completed"
+ assert rebuilt.json()["result"]["failed_records"] == 0
+ assert rebuilt.json()["result"]["rebuilt_records"] == 3
+ await assert_indexed_body()
diff --git a/tests/server/test_skill_failure_accounting.py b/tests/server/test_skill_failure_accounting.py
new file mode 100644
index 000000000..07f3a1b64
--- /dev/null
+++ b/tests/server/test_skill_failure_accounting.py
@@ -0,0 +1,118 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+"""A failed Skill package settles one queue item, regardless of file count."""
+
+from openviking.storage.queuefs import get_queue_manager
+from openviking.storage.queuefs.semantic_processor import SemanticProcessor
+from tests.server.test_api_skills import _add_skill, _skill_md
+from tests.server.test_api_skills import _stub_mcp_endpoint as _stub_mcp_endpoint
+from tests.server.test_skill_update_cancellation import (
+ _download,
+ _indexed_record,
+ _upload_package,
+)
+
+
+def _fail_package_summaries(monkeypatch, name):
+ original = SemanticProcessor._generate_single_file_summary
+ failed_paths = ["SKILL.md", "references/a.md", "references/b.md"]
+
+ async def fail_selected(self, file_path, *args, **kwargs):
+ if any(file_path.endswith(f"/skills/{name}/{path}") for path in failed_paths):
+ raise ValueError("injected file summary failure")
+ return await original(self, file_path, *args, **kwargs)
+
+ monkeypatch.setattr(SemanticProcessor, "_generate_single_file_summary", fail_selected)
+ return failed_paths, original
+
+
+async def _assert_failed_package_settled(client, failed_paths):
+ # Resource follow-up work may include a separate parent refresh. Wait for
+ # the whole queue, including that work, before checking its final counts.
+ waited = await client.post("/api/v1/system/wait", json={"timeout": 5})
+ assert waited.status_code == 200, waited.text
+ assert waited.json()["result"]["Semantic"]["error_count"] == 1
+ queue_manager = get_queue_manager()
+ status = await queue_manager.get_queue(queue_manager.SEMANTIC).get_status()
+ assert status.pending == 0
+ assert status.in_progress == 0
+ assert status.error_count == 1
+ assert len(status.errors) == 1
+ for path in failed_paths:
+ assert path in status.errors[0].message
+
+ return status
+
+
+async def test_failed_skill_files_do_not_break_global_wait_or_followup_work(
+ client, monkeypatch, tmp_path, upload_temp_dir
+):
+ name = "multiple-file-failures"
+ failed_paths, _ = _fail_package_summaries(monkeypatch, name)
+ upload = await _upload_package(
+ client,
+ tmp_path,
+ name,
+ "Failing package",
+ {"references/a.md": "First attachment", "references/b.md": "Second attachment"},
+ )
+ response = await client.post(
+ "/api/v1/skills", json={"temp_file_id": upload, "wait": True, "timeout": 5}
+ )
+ assert response.status_code == 500, response.text
+ # The request still reports every failed file, not merely the first failure.
+ for path in failed_paths:
+ assert path in response.json()["error"]["message"]
+ before = await _assert_failed_package_settled(client, failed_paths)
+
+ uploaded = await client.post(
+ "/api/v1/resources/temp_upload",
+ files={"file": ("healthy.md", b"# A healthy resource\nContent.\n", "text/markdown")},
+ )
+ assert uploaded.status_code == 200, uploaded.text
+ added = await client.post(
+ "/api/v1/resources",
+ json={"temp_file_id": uploaded.json()["result"]["temp_file_id"], "wait": True},
+ )
+ assert added.status_code == 200, added.text
+
+ after = await _assert_failed_package_settled(client, failed_paths)
+ assert after.processed > before.processed
+
+
+async def test_update_with_multiple_file_failures_restores_package_and_settles_queue(
+ client, monkeypatch, tmp_path, upload_temp_dir
+):
+ name = "update-multiple-file-failures"
+ root = (await _add_skill(client, name, "Original"))["root_uri"]
+ old_content = await _download(client, f"{root}/SKILL.md")
+ old_index = await _indexed_record(client, root, 0)
+ failed_paths, original = _fail_package_summaries(monkeypatch, name)
+ upload = await _upload_package(
+ client,
+ tmp_path,
+ name,
+ "Failed replacement",
+ {"references/a.md": "First attachment", "references/b.md": "Second attachment"},
+ )
+
+ response = await client.put(
+ f"/api/v1/skills/{name}",
+ json={"temp_file_id": upload, "wait": True, "timeout": 5},
+ )
+ assert response.status_code == 500, response.text
+ for path in failed_paths:
+ assert path in response.json()["error"]["message"]
+ before = await _assert_failed_package_settled(client, failed_paths)
+ assert await _download(client, f"{root}/SKILL.md") == old_content
+ assert await _indexed_record(client, root, 0) == old_index
+
+ monkeypatch.setattr(SemanticProcessor, "_generate_single_file_summary", original)
+ retry = await client.put(
+ f"/api/v1/skills/{name}",
+ json={"data": _skill_md(name, "Successful replacement"), "wait": True},
+ )
+ assert retry.status_code == 200, retry.text
+ after = await _assert_failed_package_settled(client, failed_paths)
+ assert after.processed > before.processed
diff --git a/tests/server/test_skill_privacy_lock.py b/tests/server/test_skill_privacy_lock.py
new file mode 100644
index 000000000..7211d7632
--- /dev/null
+++ b/tests/server/test_skill_privacy_lock.py
@@ -0,0 +1,187 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+"""Check Skill configuration isolation with real storage locks and HTTP requests."""
+
+import asyncio
+import threading
+
+import pytest
+
+from openviking.server.identity import RequestContext, Role
+from openviking.storage.queuefs import get_queue_manager
+from openviking.storage.queuefs.semantic_processor import SemanticProcessor
+from openviking.utils.skill_processor import SkillProcessor
+from openviking_cli.session.user_id import UserIdentifier
+from tests.server.test_api_skills import _add_skill, _skill_md
+from tests.server.test_api_skills import _stub_mcp_endpoint as _stub_mcp_endpoint
+from tests.server.test_skill_update_cancellation import _wait_until
+from tests.server.test_skill_update_lock import _assert_locked
+
+
+async def _privacy_snapshot(client, endpoint):
+ current = await client.get(endpoint)
+ assert current.status_code in (200, 404), current.text
+ versions = await client.get(f"{endpoint}/versions")
+ if current.status_code == 404:
+ assert versions.status_code == 404, versions.text
+ return 404, None, {}
+ assert versions.status_code == 200, versions.text
+ history = {}
+ for version in versions.json()["result"]:
+ response = await client.get(f"{endpoint}/versions/{version}")
+ assert response.status_code == 200, response.text
+ history[version] = response.json()["result"]
+ return current.status_code, current.json().get("result"), history
+
+
+def _observe_save_start(privacy, monkeypatch):
+ started = asyncio.Event()
+ original_upsert = privacy.upsert
+
+ async def observe_save(*args, **kwargs):
+ if kwargs.get("values") == {"api_key": "concurrent-value"}:
+ started.set()
+ return await original_upsert(*args, **kwargs)
+
+ monkeypatch.setattr(privacy, "upsert", observe_save)
+ return started
+
+
+async def _assert_unlocked(fs, root, ctx):
+ lease = await fs._async_agfs.pathlock_acquire_tree(
+ fs._uri_to_path(root, ctx=ctx), timeout_secs=0.01
+ )
+ await fs._async_agfs.pathlock_release(lease)
+
+
[email protected]("had_privacy", [False, True])
+async def test_update_rollback_preserves_waiting_privacy_save(
+ client, service, monkeypatch, had_privacy
+):
+ name = "privacy-update-rollback"
+ await _add_skill(client, name, "Original")
+ endpoint = f"/api/v1/privacy-configs/skill/{name}"
+ if had_privacy:
+ seeded = await client.post(endpoint, json={"values": {"api_key": "old-value"}})
+ assert seeded.status_code == 200, seeded.text
+ _, _, old_history = await _privacy_snapshot(client, endpoint)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)
+ privacy = service.privacy_configs
+ root = privacy.get_config_root(ctx, "skill", name)
+ paused = asyncio.Event()
+ resume = asyncio.Event()
+ save_started = _observe_save_start(privacy, monkeypatch)
+
+ async def prepare_privacy(self, skill_dict, ctx):
+ return skill_dict, {"api_key": "replacement-value"}
+
+ async def fail_metadata(*args, **kwargs):
+ paused.set()
+ await resume.wait()
+ raise RuntimeError("injected metadata failure after configuration change")
+
+ monkeypatch.setattr(SkillProcessor, "prepare_skill_privacy", prepare_privacy)
+ monkeypatch.setattr(
+ "openviking.server.skill_source_metadata.write_skill_source_metadata", fail_metadata
+ )
+ updating = asyncio.create_task(
+ client.put(
+ f"/api/v1/skills/{name}",
+ json={"data": _skill_md(name, "Failed replacement"), "wait": True},
+ )
+ )
+ saving = None
+ try:
+ await asyncio.wait_for(paused.wait(), 5)
+ await _assert_locked(service.viking_fs, root)
+ saving = asyncio.create_task(
+ client.post(endpoint, json={"values": {"api_key": "concurrent-value"}})
+ )
+ await asyncio.wait_for(save_started.wait(), 5)
+ assert not saving.done(), "Configuration writes must wait through update and rollback"
+ resume.set()
+ failed = await asyncio.wait_for(updating, 10)
+ assert failed.status_code == 500, failed.text
+ saved = await asyncio.wait_for(saving, 10)
+ assert saved.status_code == 200, saved.text
+ status, current, history = await _privacy_snapshot(client, endpoint)
+ assert status == 200
+ assert current["current"]["values"] == {"api_key": "concurrent-value"}
+ for version, snapshot in old_history.items():
+ assert history[version] == snapshot
+ shown = await client.get(f"/api/v1/skills/{name}")
+ assert shown.status_code == 200, shown.text
+ assert shown.json()["result"]["description"] == "Original"
+ finally:
+ resume.set()
+ pending = [task for task in (updating, saving) if task is not None and not task.done()]
+ if pending:
+ await asyncio.wait_for(asyncio.gather(*pending), 10)
+
+
+async def test_update_timeout_restores_privacy_and_releases_both_locks(
+ client, service, monkeypatch
+):
+ name = "privacy-update-timeout"
+ root = (await _add_skill(client, name, "Original"))["root_uri"]
+ endpoint = f"/api/v1/privacy-configs/skill/{name}"
+ seeded = await client.post(endpoint, json={"values": {"api_key": "old-value"}})
+ assert seeded.status_code == 200, seeded.text
+ _, _, old_history = await _privacy_snapshot(client, endpoint)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)
+ fs = service.viking_fs
+ privacy_root = service.privacy_configs.get_config_root(ctx, "skill", name)
+ started = threading.Event()
+ cancelled = threading.Event()
+ resume = threading.Event()
+ original_summary = SemanticProcessor._generate_single_file_summary
+
+ async def replacement_privacy(self, skill_dict, ctx):
+ return skill_dict, {"api_key": "replacement-value"}
+
+ async def hold_summary(self, *args, **kwargs):
+ started.set()
+ try:
+ while not resume.is_set():
+ await asyncio.sleep(0.01)
+ except asyncio.CancelledError:
+ cancelled.set()
+ raise
+ return await original_summary(self, *args, **kwargs)
+
+ monkeypatch.setattr(SkillProcessor, "prepare_skill_privacy", replacement_privacy)
+ monkeypatch.setattr(SemanticProcessor, "_generate_single_file_summary", hold_summary)
+ updating = asyncio.create_task(
+ client.put(
+ f"/api/v1/skills/{name}",
+ json={"data": _skill_md(name, "Timed out replacement"), "wait": True, "timeout": 2},
+ )
+ )
+ try:
+ await _wait_until(started.is_set)
+ await _assert_locked(fs, root)
+ await _assert_locked(fs, privacy_root)
+ failed = await asyncio.wait_for(asyncio.shield(updating), 5)
+ assert failed.status_code == 504, failed.text
+ assert cancelled.is_set(), "The timeout must cancel the actual background summary"
+ await _assert_unlocked(fs, root, ctx)
+ await _assert_unlocked(fs, privacy_root, ctx)
+ status, current, history = await _privacy_snapshot(client, endpoint)
+ assert status == 200
+ assert current["current"]["values"] == {"api_key": "old-value"}
+ for version, snapshot in old_history.items():
+ assert history[version] == snapshot
+ saved = await asyncio.wait_for(
+ client.post(endpoint, json={"values": {"api_key": "concurrent-value"}}), 5
+ )
+ assert saved.status_code == 200, saved.text
+ shown = await client.get(f"/api/v1/skills/{name}")
+ assert shown.status_code == 200, shown.text
+ assert shown.json()["result"]["description"] == "Original"
+ finally:
+ resume.set()
+ await get_queue_manager().wait_complete(timeout=5)
+ if not updating.done():
+ updating.cancel()
+ await asyncio.gather(updating, return_exceptions=True)
diff --git a/tests/server/test_skill_update_cancellation.py b/tests/server/test_skill_update_cancellation.py
new file mode 100644
index 000000000..889c6e4db
--- /dev/null
+++ b/tests/server/test_skill_update_cancellation.py
@@ -0,0 +1,218 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+"""Exercise Skill update failures through the HTTP API with local fake models."""
+
+import asyncio
+import threading
+import zipfile
+
+import pytest
+
+from openviking.service.task_tracker import get_task_tracker
+from openviking.service.task_work_index import get_task_context
+from openviking.storage.queuefs import get_queue_manager
+from openviking.storage.queuefs.semantic_processor import SemanticProcessor
+from tests.server.test_api_skills import _add_skill, _skill_md
+from tests.server.test_api_skills import _stub_mcp_endpoint as _stub_mcp_endpoint
+
+
+async def _upload_package(client, tmp_path, name, description, files):
+ archive = tmp_path / f"{name}.zip"
+ with zipfile.ZipFile(archive, "w") as package:
+ package.writestr("SKILL.md", _skill_md(name, description))
+ for path, content in files.items():
+ package.writestr(path, content)
+ with archive.open("rb") as handle:
+ response = await client.post(
+ "/api/v1/resources/temp_upload",
+ files={"file": (archive.name, handle, "application/zip")},
+ )
+ assert response.status_code == 200, response.text
+ return response.json()["result"]["temp_file_id"]
+
+
+async def _wait_until(predicate, timeout=5):
+ async with asyncio.timeout(timeout):
+ while not predicate():
+ await asyncio.sleep(0.01)
+
+
+async def _download(client, uri):
+ response = await client.get("/api/v1/content/download", params={"uri": uri})
+ assert response.status_code == 200, response.text
+ return response.content
+
+
+async def _indexed_record(client, uri, level):
+ response = await client.post(
+ "/api/v1/search/find",
+ json={
+ "query": "",
+ "target_uri": uri,
+ "level": [level],
+ "filter": {"op": "must", "field": "uri", "conds": [uri], "para": "-d=0"},
+ },
+ )
+ assert response.status_code == 200, response.text
+ return [
+ (hit["uri"], hit["level"], hit["abstract"]) for hit in response.json()["result"]["skills"]
+ ]
+
+
[email protected]("cancel_mode", ["timeout", "task"])
+async def test_update_cancellation_restores_package_and_index(
+ client, tmp_path, monkeypatch, cancel_mode
+):
+ from openviking.storage.queuefs import semantic_dag
+
+ name = "timeout-package-rollback"
+ old_upload = await _upload_package(
+ client,
+ tmp_path,
+ name,
+ "Original description",
+ {"references/old.md": "Original recovery instructions."},
+ )
+ added = await client.post("/api/v1/skills", json={"temp_file_id": old_upload, "wait": True})
+ assert added.status_code == 200, added.text
+ root = added.json()["result"]["root_uri"]
+ paths = (
+ "SKILL.md",
+ ".abstract.md",
+ ".overview.md",
+ "references/old.md",
+ "references/.abstract.md",
+ "references/.overview.md",
+ )
+ old_contents = {path: await _download(client, f"{root}/{path}") for path in paths}
+ record_locations = (
+ (root, 0),
+ (root, 1),
+ (f"{root}/SKILL.md", 2),
+ (f"{root}/references", 0),
+ (f"{root}/references", 1),
+ (f"{root}/references/old.md", 2),
+ )
+ old_records = {
+ location: await _indexed_record(client, *location) for location in record_locations
+ }
+ assert all(len(records) == 1 for records in old_records.values())
+
+ files = {"references/000-fast.md": "A replacement file that reaches the index."}
+ files.update({f"references/slow-{index}.md": "Unfinished replacement." for index in range(8)})
+ new_upload = await _upload_package(client, tmp_path, name, "Replacement description", files)
+ release = threading.Event()
+ fast_finished = threading.Event()
+ started = set()
+ cancelled = set()
+ finished = set()
+ task_owners = {}
+ original_summary = SemanticProcessor._generate_single_file_summary
+ original_scheduler = semantic_dag.get_semantic_node_scheduler
+
+ async def hold_summary(self, file_path, *args, **kwargs):
+ task_context = get_task_context()
+ task_owners[task_context.task_id] = task_context
+ if file_path.endswith("/000-fast.md"):
+ result = await original_summary(self, file_path, *args, **kwargs)
+ fast_finished.set()
+ return result
+ started.add(file_path)
+ try:
+ while not release.is_set():
+ await asyncio.sleep(0.01)
+ except asyncio.CancelledError:
+ cancelled.add(file_path)
+ raise
+ finished.add(file_path)
+ return await original_summary(self, file_path, *args, **kwargs)
+
+ monkeypatch.setattr(SemanticProcessor, "_generate_single_file_summary", hold_summary)
+ monkeypatch.setattr(
+ semantic_dag, "get_semantic_node_scheduler", lambda _workers: original_scheduler(2)
+ )
+ updating = asyncio.create_task(
+ client.put(
+ f"/api/v1/skills/{name}",
+ json={
+ "temp_file_id": new_upload,
+ "wait": True,
+ "timeout": 2 if cancel_mode == "timeout" else 10,
+ },
+ )
+ )
+ try:
+ await _wait_until(lambda: fast_finished.is_set() and len(started) == 2)
+ async with asyncio.timeout(1):
+ while not await _indexed_record(client, f"{root}/references/000-fast.md", 2):
+ await asyncio.sleep(0.01)
+ if cancel_mode == "task":
+ assert len(task_owners) == 1
+ owner = next(iter(task_owners.values()))
+ await get_task_tracker().cancel(
+ owner.task_id, account_id=owner.account_id, user_id=owner.user_id
+ )
+ response = await asyncio.wait_for(asyncio.shield(updating), timeout=5)
+ expected_error = "DEADLINE_EXCEEDED" if cancel_mode == "timeout" else "PROCESSING_ERROR"
+ assert response.status_code >= 400, response.text
+ assert response.json()["error"]["code"] == expected_error
+ assert cancelled == started
+ assert not finished, "Rollback must not wait for blocked summaries to finish normally"
+ assert len(started) < len(files), "Pending summaries should be skipped on cancellation"
+
+ # Releasing the fault after the response must not let stale work modify the restored pack.
+ release.set()
+ await get_queue_manager().wait_complete(timeout=5)
+ for path, content in old_contents.items():
+ assert await _download(client, f"{root}/{path}") == content
+ for location, records in old_records.items():
+ assert await _indexed_record(client, *location) == records
+ for path in files:
+ missing = await client.get("/api/v1/content/download", params={"uri": f"{root}/{path}"})
+ assert missing.status_code == 404, missing.text
+ assert await _indexed_record(client, f"{root}/{path}", 2) == []
+ finally:
+ release.set()
+ await get_queue_manager().wait_complete(timeout=5)
+ if not updating.done():
+ updating.cancel()
+ await asyncio.gather(updating, return_exceptions=True)
+
+
+async def test_async_update_background_failure_keeps_successfully_returned_replacement(
+ client, monkeypatch
+):
+ name = "async-failure-keeps-replacement"
+ await _add_skill(client, name, "Original description")
+ release_failure = threading.Event()
+
+ async def fail_after_response(self, file_path, *args, **kwargs):
+ while not release_failure.is_set():
+ await asyncio.sleep(0.01)
+ raise RuntimeError("injected background summary failure")
+
+ monkeypatch.setattr(SemanticProcessor, "_generate_single_file_summary", fail_after_response)
+ try:
+ response = await client.put(
+ f"/api/v1/skills/{name}",
+ json={"data": _skill_md(name, "Replacement description"), "wait": False},
+ )
+ assert response.status_code == 200, response.text
+ task_id = response.json()["result"]["task_id"]
+ release_failure.set()
+ async with asyncio.timeout(5):
+ while True:
+ task_response = await client.get(f"/api/v1/tasks/{task_id}")
+ assert task_response.status_code == 200, task_response.text
+ status = task_response.json()["result"]["status"]
+ if status in {"failed", "cancelled", "completed"}:
+ break
+ await asyncio.sleep(0.01)
+ assert status == "failed", task_response.text
+ shown = await client.get(f"/api/v1/skills/{name}")
+ assert shown.status_code == 200, shown.text
+ assert shown.json()["result"]["description"] == "Replacement description"
+ finally:
+ release_failure.set()
+ await get_queue_manager().wait_complete(timeout=5)
diff --git a/tests/server/test_skill_update_lock.py b/tests/server/test_skill_update_lock.py
new file mode 100644
index 000000000..a5d9d82ea
--- /dev/null
+++ b/tests/server/test_skill_update_lock.py
@@ -0,0 +1,135 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+"""Check update lock continuity using the real filesystem and HTTP endpoints."""
+
+import asyncio
+
+import pytest
+
+from openviking.server.identity import RequestContext, Role
+from openviking.storage.errors import LockAcquisitionError
+from openviking.utils.skill_processor import SkillProcessor
+from openviking_cli.session.user_id import UserIdentifier
+from tests.server.test_api_skills import _add_skill, _skill_md
+from tests.server.test_api_skills import _stub_mcp_endpoint as _stub_mcp_endpoint
+from tests.server.test_skill_update_cancellation import _download, _indexed_record
+
+
+def _ctx():
+ return RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)
+
+
+async def _assert_locked(fs, root):
+ other_lease = None
+ try:
+ with pytest.raises(LockAcquisitionError):
+ other_lease = await fs._async_agfs.pathlock_acquire_tree(
+ fs._uri_to_path(root, ctx=_ctx()), timeout_secs=0.01
+ )
+ finally:
+ if other_lease is not None:
+ await fs._async_agfs.pathlock_release(other_lease)
+
+
[email protected]("competing_operation", ["add", "update", "delete"])
+async def test_update_lock_blocks_same_package_through_backup_and_rollback(
+ client, service, monkeypatch, competing_operation
+):
+ name = f"update-lock-{competing_operation}"
+ added = await _add_skill(client, name, "Original")
+ root = added["root_uri"]
+ fs = service.viking_fs
+ hidden_uri = f"{root}/.custom.json"
+ await fs.write_file(hidden_uri, '{"original": true}', ctx=_ctx())
+ old_content = await _download(client, f"{root}/SKILL.md")
+ old_index = await _indexed_record(client, root, 0)
+ paused = asyncio.Event()
+ resume = asyncio.Event()
+ original_apply = SkillProcessor.apply_skill_privacy
+
+ async def fail_after_backup(self, *args, **kwargs):
+ if kwargs.get("change_reason") == "auto-extracted from update_skill":
+ paused.set()
+ await resume.wait()
+ raise RuntimeError("injected preparation failure after backup")
+ return await original_apply(self, *args, **kwargs)
+
+ monkeypatch.setattr(SkillProcessor, "apply_skill_privacy", fail_after_backup)
+ updating = asyncio.create_task(
+ client.put(
+ f"/api/v1/skills/{name}",
+ json={"data": _skill_md(name, "Failed replacement"), "wait": True},
+ )
+ )
+ try:
+ await asyncio.wait_for(paused.wait(), 5)
+ assert await fs.exists(root, ctx=_ctx())
+ assert not await fs.exists(f"{root}/SKILL.md", ctx=_ctx())
+ assert not await fs.exists(hidden_uri, ctx=_ctx())
+ await _assert_locked(fs, root)
+ if competing_operation == "delete":
+ competing = await client.delete(f"/api/v1/skills/{name}")
+ else:
+ method = client.post if competing_operation == "add" else client.put
+ endpoint = (
+ "/api/v1/skills" if competing_operation == "add" else f"/api/v1/skills/{name}"
+ )
+ competing = await asyncio.wait_for(
+ method(endpoint, json={"data": _skill_md(name, "Concurrent"), "wait": True}),
+ 5,
+ )
+ assert competing.status_code != 200, competing.text
+ # Holding this package must not lock the entire skills namespace.
+ await _add_skill(client, f"unrelated-{competing_operation}", "Independent")
+ resume.set()
+ failed = await asyncio.wait_for(updating, 10)
+ assert failed.status_code == 500, failed.text
+ assert await _download(client, f"{root}/SKILL.md") == old_content
+ assert await _download(client, hidden_uri) == b'{"original": true}'
+ assert await _indexed_record(client, root, 0) == old_index
+ # Both the request and background references must have been released.
+ lease = await fs._async_agfs.pathlock_acquire_tree(fs._uri_to_path(root, ctx=_ctx()))
+ await fs._async_agfs.pathlock_release(lease)
+ finally:
+ resume.set()
+ if not updating.done():
+ await asyncio.wait_for(updating, 10)
+
+
+async def test_restore_index_failure_preserves_live_lock_and_backup(client, service, monkeypatch):
+ name = "restore-index-lock"
+ root = (await _add_skill(client, name, "Original"))["root_uri"]
+ fs = service.viking_fs
+ old_content = await _download(client, f"{root}/SKILL.md")
+ original_transfer = fs._update_vector_store_uris
+ backup = None
+
+ async def fail_new_source(*args, **kwargs):
+ raise RuntimeError("injected new source write failure")
+
+ async def fail_restore_index(source, target, **kwargs):
+ nonlocal backup
+ if ".update-backup-" in source and target == root:
+ backup = source
+ await _assert_locked(fs, root)
+ assert await _download(client, f"{root}/SKILL.md") == old_content
+ raise RuntimeError("injected index restoration failure")
+ return await original_transfer(source, target, **kwargs)
+
+ monkeypatch.setattr(
+ "openviking.server.skill_source_metadata.write_skill_source_metadata", fail_new_source
+ )
+ monkeypatch.setattr(fs, "_update_vector_store_uris", fail_restore_index)
+ response = await client.put(
+ f"/api/v1/skills/{name}",
+ json={"data": _skill_md(name, "Replacement"), "wait": True},
+ )
+ assert response.status_code == 500, response.text
+ assert "injected index restoration failure" in response.text
+ assert backup is not None and backup in response.text
+ # Generic cp/mv cleanup would remove the live root when vector restore fails.
+ assert await _download(client, f"{root}/SKILL.md") == old_content
+ assert await _download(client, f"{backup}/SKILL.md") == old_content
+ # Internal backups are hidden from public search; inspect their stored vectors.
+ assert await fs.vector_store.get_context_by_uri(backup, level=0, ctx=_ctx())
diff --git a/tests/server/test_skill_update_retry.py b/tests/server/test_skill_update_retry.py
new file mode 100644
index 000000000..d10604c26
--- /dev/null
+++ b/tests/server/test_skill_update_retry.py
@@ -0,0 +1,114 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+"""Skill retries preserve the lease shared with the waiting update request."""
+
+import asyncio
+import threading
+
+import pytest
+
+from openviking.storage.queuefs import get_queue_manager
+from openviking.storage.queuefs.semantic_dag import SemanticDagExecutor
+from tests.server.test_api_skills import _add_skill, _skill_md
+from tests.server.test_api_skills import _stub_mcp_endpoint as _stub_mcp_endpoint
+from tests.server.test_skill_update_cancellation import _download, _wait_until
+from tests.server.test_skill_update_lock import _assert_locked, _ctx
+
+
+def _fail_first_run(monkeypatch, root, *, block_retries=False):
+ original = SemanticDagExecutor.run
+ attempts = []
+
+ async def fail_once(self, root_uri):
+ if root_uri == root:
+ attempts.append(root_uri)
+ if len(attempts) == 1:
+ raise RuntimeError("one transient failure after adopting the package lock")
+ if block_retries:
+ await asyncio.Event().wait()
+ return await original(self, root_uri)
+
+ monkeypatch.setattr(SemanticDagExecutor, "run", fail_once)
+ return attempts
+
+
+async def _assert_unlocked(fs, root):
+ lease = await fs._async_agfs.pathlock_acquire_tree(
+ fs._uri_to_path(root, ctx=_ctx()), timeout_secs=0.01
+ )
+ await fs._async_agfs.pathlock_release(lease)
+
+
+async def test_update_wait_true_retries_transient_failure_without_conflicting_with_its_lock(
+ client, service, monkeypatch
+):
+ name = "update-transient-retry"
+ root = (await _add_skill(client, name, "Original"))["root_uri"]
+ attempts = _fail_first_run(monkeypatch, root)
+
+ response = await client.put(
+ f"/api/v1/skills/{name}",
+ json={"data": _skill_md(name, "Replacement"), "wait": True, "timeout": 5},
+ )
+
+ assert response.status_code == 200, response.text
+ assert len(attempts) == 2
+ assert b"Replacement" in await _download(client, f"{root}/SKILL.md")
+ await _assert_unlocked(service.viking_fs, root)
+
+
[email protected]("enqueue_committed", [False, True])
+async def test_update_timeout_waits_for_retry_handoff_to_settle_before_restoring(
+ client, service, monkeypatch, enqueue_committed
+):
+ name = f"update-cancel-retry-enqueue-{enqueue_committed}"
+ root = (await _add_skill(client, name, "Original"))["root_uri"]
+ old_content = await _download(client, f"{root}/SKILL.md")
+ # A published retry may already be running while its enqueue is returning.
+ # Keep that work pending so this case actually exercises a request timeout.
+ _fail_first_run(monkeypatch, root, block_retries=enqueue_committed)
+ queue_manager = get_queue_manager()
+ queue = queue_manager.get_queue(queue_manager.SEMANTIC)
+ original_enqueue = queue.enqueue
+ retry_started = threading.Event()
+ resume_enqueue = threading.Event()
+ enqueues = 0
+
+ def pause():
+ retry_started.set()
+ assert resume_enqueue.wait(5)
+
+ async def slow_retry_enqueue(msg):
+ nonlocal enqueues
+ if msg.uri == root:
+ enqueues += 1
+ if enqueues == 2:
+ if enqueue_committed:
+ result = await original_enqueue(msg)
+ await asyncio.to_thread(pause)
+ return result
+ await asyncio.to_thread(pause)
+ return await original_enqueue(msg)
+
+ monkeypatch.setattr(queue, "enqueue", slow_retry_enqueue)
+ updating = asyncio.create_task(
+ client.put(
+ f"/api/v1/skills/{name}",
+ json={"data": _skill_md(name, "Replacement"), "wait": True, "timeout": 2},
+ )
+ )
+ try:
+ await _wait_until(retry_started.is_set)
+ await asyncio.sleep(2.1)
+ assert not updating.done(), "Rollback must wait for the started retry enqueue"
+ await _assert_locked(service.viking_fs, root)
+ resume_enqueue.set()
+ response = await asyncio.wait_for(updating, 10)
+ assert response.status_code == 504, response.text
+ assert await _download(client, f"{root}/SKILL.md") == old_content
+ await _assert_unlocked(service.viking_fs, root)
+ finally:
+ resume_enqueue.set()
+ if not updating.done():
+ await asyncio.wait_for(updating, 10)
diff --git a/tests/service/test_skill_package_reindex.py b/tests/service/test_skill_package_reindex.py
new file mode 100644
index 000000000..5d983dc68
--- /dev/null
+++ b/tests/service/test_skill_package_reindex.py
@@ -0,0 +1,97 @@
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import pytest
+
+from openviking.server.identity import RequestContext, Role
+from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters
+from openviking_cli.session.user_id import UserIdentifier
+
+
+async def test_vectors_only_reindexes_entire_skill_with_common_media_inputs(monkeypatch):
+ root = "viking://agent/skills/demo"
+ tree = {
+ root: [
+ {"name": "SKILL.md"},
+ {"name": "reference", "isDir": True},
+ {"name": ".source.json"},
+ ],
+ root + "/reference": [
+ {"name": "image.png"},
+ {"name": "video.mp4"},
+ {"name": "nested", "isDir": True},
+ ],
+ root + "/reference/nested": [
+ {"name": "api.md"},
+ {"name": ".abstract.md"},
+ {"name": "messages.jsonl"},
+ ],
+ }
+ fs = SimpleNamespace(ls=AsyncMock(side_effect=lambda uri, **kw: tree[uri]))
+ monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: fs)
+ executor = ReindexExecutor()
+ executor._read_directory_abstract = AsyncMock(return_value="name: demo\ndescription: Demo")
+ executor._read_directory_overview = AsyncMock(return_value="overview")
+ executor._fetch_existing_record = AsyncMock(
+ side_effect=lambda **kw: {"abstract": "media summary", "meta": {"source_path": "/tmp/demo"}}
+ )
+ directories = AsyncMock()
+ files = AsyncMock(return_value=True)
+ monkeypatch.setattr("openviking.service.reindex_executor.vectorize_directory_meta", directories)
+ monkeypatch.setattr("openviking.service.reindex_executor.vectorize_file", files)
+ monkeypatch.setattr(
+ "openviking.service.reindex_executor.SemanticProcessor",
+ lambda **kw: pytest.fail("vectors_only must not generate summaries"),
+ )
+ counters = _ReindexCounters()
+ ctx = RequestContext(user=UserIdentifier("acc", "alice"), role=Role.USER)
+ await executor._reindex_skill_vectors(uri=root, ctx=ctx, counters=counters)
+ assert {call.args[0] for call in directories.await_args_list} == set(tree)
+ expected = {
+ root + "/SKILL.md",
+ root + "/reference/image.png",
+ root + "/reference/video.mp4",
+ root + "/reference/nested/api.md",
+ }
+ assert {call.kwargs["file_path"] for call in files.await_args_list} == expected
+ assert all(call.kwargs["context_type"] == "skill" for call in files.await_args_list)
+ assert all(
+ call.kwargs["summary_dict"]["summary"] == "media summary" for call in files.await_args_list
+ )
+ assert directories.await_args_list[0].kwargs["meta"]["source_path"] == "/tmp/demo"
+ assert counters.scanned_records == 7 and counters.rebuilt_records == 10
+
+
+ "abstract",
+ [
+ "Manage backups.\nUse when: a backup needs recovery.",
+ "Use when: recovery is needed",
+ "A plain description",
+ ],
+)
+async def test_vectors_only_preserves_legacy_skill_descriptions(monkeypatch, abstract):
+ executor = ReindexExecutor()
+ executor._read_directory_abstract = AsyncMock(return_value=abstract)
+ executor._read_directory_overview = AsyncMock(return_value="Overview")
+ previous_meta = {
+ "name": "demo",
+ "tags": ["backup"],
+ "allowed_tools": ["Read"],
+ "source_path": "/demo",
+ }
+ executor._fetch_existing_record = AsyncMock(return_value={"meta": previous_meta})
+ monkeypatch.setattr(
+ "openviking.service.reindex_executor.get_viking_fs", lambda: SimpleNamespace()
+ )
+ vectorize = AsyncMock()
+ monkeypatch.setattr("openviking.service.reindex_executor.vectorize_directory_meta", vectorize)
+ counters = _ReindexCounters()
+ ctx = RequestContext(user=UserIdentifier("acc", "alice"), role=Role.USER)
+ await executor._reindex_skill_vectors(
+ uri="viking://agent/skills/demo", counters=counters, ctx=ctx, recursive=False
+ )
+ assert counters.failed_records == 0 and counters.rebuilt_records == 2
+ assert vectorize.await_args.kwargs["meta"] == {**previous_meta, "description": abstract}
diff --git a/tests/service/test_skill_processing_cancellation.py b/tests/service/test_skill_processing_cancellation.py
new file mode 100644
index 000000000..abcb3d430
--- /dev/null
+++ b/tests/service/test_skill_processing_cancellation.py
@@ -0,0 +1,102 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+import asyncio
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import pytest
+
+from openviking.server.identity import RequestContext, Role
+from openviking.service.resource_service import ResourceService
+from openviking.service.task_store import PersistentTaskStore
+from openviking.service.task_tracker import TaskStatus, TaskTracker
+from openviking.service.task_work_index import QueueTaskMetadata, TaskWorkIndex
+from openviking.telemetry import OperationTelemetry, bind_telemetry
+from openviking_cli.session.user_id import UserIdentifier
+from tests.test_task_tracker import _FakeAgfs
+
+
[email protected]("outcome", ["complete", "fail"])
+async def test_skill_rollback_persists_cancellation_for_finished_work(monkeypatch, outcome):
+ store = PersistentTaskStore(_FakeAgfs())
+ tracker = TaskTracker(store=store)
+ monkeypatch.setattr("openviking.service.task_tracker.get_task_tracker", lambda: tracker)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)
+ owner = {"account_id": ctx.account_id, "user_id": ctx.user.user_id}
+ task = await tracker.create("add_skill", **owner)
+ await getattr(tracker, outcome)(
+ task.task_id, {} if outcome == "complete" else "failed", **owner
+ )
+ with pytest.raises(ValueError, match="already"):
+ await tracker.cancel(task.task_id, **owner)
+ await ResourceService.__new__(ResourceService).cancel_skill_processing(task.task_id, ctx)
+ restarted = TaskTracker(store=store)
+ restored = await restarted.get(task.task_id, **owner)
+ assert restored.status == TaskStatus.CANCELLED
+ assert restarted.is_cancellation_requested(task.task_id)
+ if outcome == "fail":
+ assert restored.error == "failed"
+
+
+async def test_skill_rollback_prevents_late_ack_failure_from_replaying_new_index(monkeypatch):
+ tracker = TaskTracker(store=PersistentTaskStore(_FakeAgfs()))
+ work = TaskWorkIndex()
+ tracker.attach_work_index(work)
+ monkeypatch.setattr("openviking.service.task_tracker.get_task_tracker", lambda: tracker)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)
+ owner = {"account_id": ctx.account_id, "user_id": ctx.user.user_id}
+ task = await tracker.create("add_skill", **owner)
+ child = QueueTaskMetadata(task.task_id, "last-vector", ctx.account_id, ctx.user.user_id)
+ work.register("Embedding", child)
+ await tracker.complete(task.task_id, {}, **owner)
+ await work.prepare_ack("Embedding", child)
+ assert (await tracker.get(task.task_id, **owner)).status == TaskStatus.COMPLETED
+ await ResourceService.__new__(ResourceService).cancel_skill_processing(task.task_id, ctx)
+ work.rollback_ack("Embedding", child)
+ assert work.cancellation_requested(task.task_id)
+ assert not work.register("Embedding", child)
+
+
+async def test_cancel_during_skill_task_creation_settles_persisted_record(monkeypatch):
+ store = PersistentTaskStore(_FakeAgfs())
+ started, resume = asyncio.Event(), asyncio.Event()
+ original_create = store.create
+
+ async def slow_create(task):
+ started.set()
+ await resume.wait()
+ await original_create(task)
+
+ monkeypatch.setattr(store, "create", slow_create)
+ tracker = TaskTracker(store=store)
+ monkeypatch.setattr("openviking.service.task_tracker.get_task_tracker", lambda: tracker)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)
+ owner = {"account_id": ctx.account_id, "user_id": ctx.user.user_id}
+ service = ResourceService.__new__(ResourceService)
+ service._resource_processor = object()
+ service._viking_fs = object()
+ service._skill_processor = SimpleNamespace(process_skill=AsyncMock())
+ with bind_telemetry(OperationTelemetry("cancel-skill-create")):
+ adding = asyncio.create_task(
+ service.add_skill(
+ "body", ctx, wait=True, target_uri="viking://agent/skills", task_id="create-test"
+ )
+ )
+ try:
+ await asyncio.wait_for(started.wait(), 1)
+ adding.cancel()
+ resume.set()
+ with pytest.raises(asyncio.CancelledError):
+ await asyncio.wait_for(adding, 1)
+ record = await tracker.get("create-test", **owner)
+ assert record.status == TaskStatus.FAILED
+ assert (await store.get("create-test", **owner))["status"] == "failed"
+ assert not tracker.has_work(record.task_id)
+ service._skill_processor.process_skill.assert_not_awaited()
+ finally:
+ resume.set()
+ await asyncio.gather(adding, return_exceptions=True)
diff --git a/tests/session/train/test_train_components.py b/tests/session/train/test_train_components.py
index 9999c632f..3c2c35cc6 100644
--- a/tests/session/train/test_train_components.py
+++ b/tests/session/train/test_train_components.py
@@ -1250,10 +1250,9 @@ async def test_skill_policy_creation_preserves_lock_ownership(tmp_path, monkeypa
root = "viking://user/default/skills"
uri = f"{root}/code-review/SKILL.md"
processor = SkillProcessor(vikingdb=AsyncMock())
- config = SimpleNamespace(
- vlm=SimpleNamespace(get_completion_async=AsyncMock(return_value="Overview"))
- )
- monkeypatch.setattr("openviking.utils.skill_processor.get_openviking_config", lambda: config)
+ queue = SimpleNamespace(enqueue=AsyncMock())
+ manager = SimpleNamespace(SEMANTIC="Semantic", get_queue=lambda *args, **kwargs: queue)
+ monkeypatch.setattr("openviking.storage.queuefs.get_queue_manager", lambda: manager)
policy_set = ExperienceSet(root_uri=root, policies=[], viking_fs=fs, request_context=ctx)
updater = SkillPolicyUpdater(skill_processor=processor, viking_fs=fs)
plan = PolicyUpdatePlan(
@@ -1281,8 +1280,22 @@ async def test_skill_policy_creation_preserves_lock_ownership(tmp_path, monkeypa
# The callee must not release the caller's tree lock.
with pytest.raises(LockAcquisitionError):
await fs._async_agfs.pathlock_acquire_exact(fs._uri_to_path(uri, ctx=ctx))
- # The outer context releases its lease after the nested writes.
+ # Queued indexing keeps its own package lease after the caller exits.
+ with pytest.raises(LockAcquisitionError):
+ await fs._async_agfs.pathlock_acquire_exact(fs._uri_to_path(uri, ctx=ctx))
+ msg = queue.enqueue.await_args.args[0]
+ worker_lease = await fs._async_agfs.pathlock_adopt(msg.lock_handoff)
+ await fs._async_agfs.pathlock_release(worker_lease)
lease = await fs._async_agfs.pathlock_acquire_tree(fs._uri_to_path(root, ctx=ctx))
await fs._async_agfs.pathlock_release(lease)
finally:
+ from openviking.telemetry import unregister_telemetry
+ from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
+
+ if queue.enqueue.await_args:
+ msg = queue.enqueue.await_args.args[0]
+ tracker = get_request_wait_tracker()
+ tracker.mark_semantic_done(msg.telemetry_id, msg.id)
+ tracker.cleanup(msg.telemetry_id)
+ unregister_telemetry(msg.telemetry_id)
agfs.close()
diff --git a/tests/storage/test_memory_semantic_stall.py b/tests/storage/test_memory_semantic_stall.py
index 7c91668ad..158de7062 100644
--- a/tests/storage/test_memory_semantic_stall.py
+++ b/tests/storage/test_memory_semantic_stall.py
@@ -7,14 +7,18 @@ Ensures that _process_memory_directory() error paths propagate exceptions
so that on_dequeue() returns an explicit processing outcome.
"""
+import asyncio
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock, patch
+from uuid import uuid4
import pytest
from openviking.storage.queuefs.process_result import ProcessOutcome
from openviking.storage.queuefs.semantic_msg import SemanticMsg
from openviking.storage.queuefs.semantic_processor import SemanticProcessor
+from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
+from openviking.utils.circuit_breaker import CircuitBreakerOpen
def _make_msg(uri="viking://user/usr1/memories", context_type="memory", **kwargs):
@@ -42,6 +46,46 @@ def _build_data(msg: SemanticMsg) -> dict:
return msg.to_dict()
[email protected]("context_type", ["resource", "memory", "skill"])
+async def test_retry_cancellation_keeps_skill_wait_isolated(monkeypatch, context_type):
+ processor = SemanticProcessor()
+ processor._circuit_breaker = SimpleNamespace(
+ check=MagicMock(side_effect=CircuitBreakerOpen), retry_after=0
+ )
+ entered, release = asyncio.Event(), asyncio.Event()
+ written = []
+
+ async def enqueue(msg):
+ entered.set()
+ await release.wait()
+ written.append(msg.id)
+
+ queue = SimpleNamespace(enqueue=enqueue)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.get_queue_manager",
+ lambda: SimpleNamespace(SEMANTIC="Semantic", get_queue=lambda _: queue),
+ )
+ msg = _make_msg(context_type=context_type, telemetry_id=str(uuid4()))
+ worker = asyncio.create_task(processor.on_dequeue(msg.to_dict()))
+ try:
+ await asyncio.wait_for(entered.wait(), 1)
+ worker.cancel()
+ if context_type == "skill":
+ await asyncio.sleep(0)
+ assert not worker.done()
+ assert not written
+ release.set()
+ with pytest.raises(asyncio.CancelledError):
+ await asyncio.wait_for(worker, 1)
+ assert written == ([msg.id] if context_type == "skill" else [])
+ finally:
+ release.set()
+ if not worker.done():
+ worker.cancel()
+ await asyncio.gather(worker, return_exceptions=True)
+ get_request_wait_tracker().cleanup(msg.telemetry_id)
+
+
@pytest.mark.asyncio
async def test_root_semantic_message_is_acknowledged_without_processing():
processor = SemanticProcessor()
@@ -73,7 +117,7 @@ async def test_memory_empty_dir_still_returns_success():
return_value=fake_fs,
),
patch(
- "openviking.storage.queuefs.semantic_processor.resolve_telemetry",
+ "openviking.storage.queuefs.semantic_work.resolve_telemetry",
return_value=None,
),
):
@@ -106,7 +150,7 @@ async def test_memory_ls_error_returns_failed():
return_value=fake_fs,
),
patch(
- "openviking.storage.queuefs.semantic_processor.resolve_telemetry",
+ "openviking.storage.queuefs.semantic_work.resolve_telemetry",
return_value=None,
),
):
@@ -143,7 +187,7 @@ async def test_memory_ls_transient_error_requeues():
return_value=fake_fs,
),
patch(
- "openviking.storage.queuefs.semantic_processor.resolve_telemetry",
+ "openviking.storage.queuefs.semantic_work.resolve_telemetry",
return_value=None,
),
patch.object(processor, "_reenqueue_semantic_msg", new=reenqueue_mock),
@@ -185,7 +229,7 @@ async def test_memory_write_error_returns_failed():
return_value=fake_fs,
),
patch(
- "openviking.storage.queuefs.semantic_processor.resolve_telemetry",
+ "openviking.storage.queuefs.semantic_work.resolve_telemetry",
return_value=None,
),
patch(
diff --git a/tests/storage/test_queue_manager.py b/tests/storage/test_queue_manager.py
index 83eaccead..a5cf69a08 100644
--- a/tests/storage/test_queue_manager.py
+++ b/tests/storage/test_queue_manager.py
@@ -2,11 +2,13 @@
# SPDX-License-Identifier: AGPL-3.0
"""Focused tests for QueueManager concurrency selection."""
+import asyncio
import json
import os
import subprocess
import sys
import threading
+from types import SimpleNamespace
from unittest.mock import AsyncMock
import pytest
@@ -46,6 +48,134 @@ def test_queue_concurrency_uses_separate_configured_values() -> None:
assert manager._max_concurrent_for_queue(manager.SESSION_COMMIT) == 5
[email protected]("concurrency", [1, 2])
+async def test_skill_shutdown_releases_lock_after_embedding_worker_exits(
+ transport, monkeypatch, concurrency
+):
+ from openviking.service.task_tracker_concurrency import run_to_completion
+ from openviking.storage.queuefs.semantic_dag import DagStats
+ from openviking.storage.queuefs.semantic_msg import SemanticMsg
+ from openviking.storage.queuefs.semantic_processor import SemanticProcessor
+ from openviking.telemetry import OperationTelemetry
+ from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
+
+ tracker = get_request_wait_tracker()
+ telemetry_id = OperationTelemetry("skill-shutdown").telemetry_id
+ tracker.register_request(telemetry_id)
+ msg = SemanticMsg(
+ uri="viking://agent/skills/demo", context_type="skill", telemetry_id=telemetry_id
+ )
+ started, queued, release = threading.Event(), threading.Event(), threading.Event()
+ events = []
+
+ class Lease:
+ lock = {"lease_ref": "demo"}
+
+ async def close(self):
+ events.append("lock-released")
+
+ class Dag:
+ stale = False
+
+ def __init__(self, **kwargs):
+ pass
+
+ async def run(self, uri):
+ tracker.register_embedding_root(telemetry_id, "active")
+ tracker.register_embedding_root(telemetry_id, "queued")
+ queued.set()
+
+ def get_stats(self):
+ return DagStats()
+
+ async def write(data):
+ async def finish():
+ started.set()
+ while not release.is_set():
+ await asyncio.sleep(0.01)
+ events.append("write-finished")
+ tracker.mark_embedding_done(telemetry_id, "active")
+
+ await run_to_completion(finish)
+ return ProcessResult.success()
+
+ class Queue:
+ def __init__(self, name, handler):
+ self.name, self.handler, self.dispatched = name, handler, False
+
+ def has_dequeue_handler(self):
+ return True
+
+ async def size(self):
+ return int(not self.dispatched)
+
+ async def dequeue_raw(self):
+ if self.dispatched:
+ return None
+ self.dispatched = True
+ return {"id": self.name, "data": msg.to_json()}
+
+ async def process_dequeued(self, data):
+ return await self.handler(data)
+
+ async def ack(self, *args):
+ events.append(f"ack-{self.name}")
+
+ async def dequeue(self):
+ data = await self.dequeue_raw()
+ await self.process_dequeued(data)
+ await self.ack(data)
+
+ monkeypatch.setattr("openviking.storage.queuefs.semantic_processor.SemanticDagExecutor", Dag)
+ monkeypatch.setattr(
+ SemanticProcessor, "_resolve_skill_semantic_lock", AsyncMock(return_value=Lease())
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_processor.get_viking_fs",
+ lambda: SimpleNamespace(exists=AsyncMock(return_value=True)),
+ )
+ monkeypatch.setattr(
+ "openviking.storage.collection_schemas.TextEmbeddingHandler", lambda _: SimpleNamespace()
+ )
+ manager = QueueManager(
+ object(), max_concurrent_semantic=concurrency, max_concurrent_embedding=concurrency
+ )
+ manager._poll_interval = 0.001
+ manager.setup_standard_queues(object(), start=False)
+ semantic = manager._queues[manager.SEMANTIC]._dequeue_handler
+ manager._queues = {
+ manager.EMBEDDING: Queue(manager.EMBEDDING, write),
+ manager.SEMANTIC: Queue(manager.SEMANTIC, semantic.on_dequeue),
+ }
+ manager.start()
+ threads = list(manager._queue_threads.values())
+ stopping = None
+ try:
+ async with asyncio.timeout(2):
+ while not (started.is_set() and queued.is_set()):
+ await asyncio.sleep(0.01)
+ stopping = asyncio.create_task(asyncio.to_thread(manager.stop))
+ # Cross the concurrent worker's cancellation deadline while a physical
+ # write is still active: the Skill lease must remain held.
+ await asyncio.sleep(5.2 if concurrency == 2 else 0.1)
+ assert "lock-released" not in events
+ assert not manager._embedding_worker_stopped.is_set()
+ release.set()
+ await asyncio.wait_for(asyncio.shield(stopping), 2)
+ assert all(not thread.is_alive() for thread in threads)
+ assert events.index("write-finished") < events.index("lock-released")
+ assert f"ack-{manager.SEMANTIC}" not in events
+ assert not tracker.is_complete(telemetry_id) # Queued vectors await restart.
+ finally:
+ release.set()
+ if stopping is not None:
+ await stopping
+ else:
+ await asyncio.to_thread(manager.stop)
+ tracker.cleanup(telemetry_id)
+
+
async def test_status_waits_for_processing_messages_from_other_workers(monkeypatch) -> None:
client = AsyncMock()
diff --git a/tests/storage/test_semantic_parent_bubbling.py b/tests/storage/test_semantic_parent_bubbling.py
index 3fa52653b..275291e2f 100644
--- a/tests/storage/test_semantic_parent_bubbling.py
+++ b/tests/storage/test_semantic_parent_bubbling.py
@@ -35,9 +35,7 @@ async def test_unchanged_l0_does_not_mark_or_enqueue_parent(monkeypatch):
lambda: SimpleNamespace(),
)
get_queue_manager = AsyncMock(side_effect=AssertionError("parent must not be enqueued"))
- monkeypatch.setattr(
- "openviking.storage.queuefs.get_queue_manager", get_queue_manager
- )
+ monkeypatch.setattr("openviking.storage.queuefs.get_queue_manager", get_queue_manager)
msg = SemanticMsg(
uri="viking://resources/root/child",
@@ -45,9 +43,7 @@ async def test_unchanged_l0_does_not_mark_or_enqueue_parent(monkeypatch):
role=str(Role.USER),
generation_trigger="resource_ingest",
)
- await SemanticProcessor()._enqueue_parent_refresh(
- msg, msg.uri, l0_body_changed=False
- )
+ await SemanticProcessor()._enqueue_parent_refresh(msg, msg.uri, l0_body_changed=False)
assert plan.await_args.kwargs["l0_body_changed"] is False
assert plan.await_args.kwargs["force_refresh"] is False
@@ -70,23 +66,23 @@ async def test_non_recursive_reindex_does_not_bubble_to_parent(monkeypatch):
generation_trigger="reindex",
propagate_to_parent=False,
)
- await SemanticProcessor()._enqueue_parent_refresh(
- msg, msg.uri, l0_body_changed=True
- )
+ await SemanticProcessor()._enqueue_parent_refresh(msg, msg.uri, l0_body_changed=True)
plan.assert_not_awaited()
+
@pytest.mark.parametrize(
("uri", "context_type"),
[
("viking://user/alice", "resource"),
("viking://agent/skills", "skill"),
+ ("viking://agent/skills/demo", "skill"),
+ ("viking://agent/skills/demo/reference", "skill"),
+ ("viking://user/alice/skills/demo/reference", "skill"),
],
)
@pytest.mark.asyncio
-async def test_parent_refresh_stops_at_nonsemantic_namespace_root(
- monkeypatch, uri, context_type
-):
+async def test_parent_refresh_stops_at_nonsemantic_namespace_root(monkeypatch, uri, context_type):
plan = AsyncMock(
side_effect=AssertionError("non-semantic namespace root must not be refreshed")
)
@@ -123,7 +119,11 @@ async def test_parent_refresh_stops_at_nonsemantic_namespace_root(
[
("viking://resources/project", "resource", "viking://resources"),
("viking://user/alice/resources", "resource", "viking://user/alice"),
- ("viking://agent/skills/demo", "skill", "viking://agent/skills"),
+ (
+ "viking://agent/skills/demo/reference/nested",
+ "skill",
+ "viking://agent/skills/demo/reference",
+ ),
],
)
@pytest.mark.asyncio
@@ -151,9 +151,7 @@ async def test_parent_refresh_preserves_semantic_roots(
get_queue_manager = AsyncMock(
side_effect=AssertionError("best-effort lock miss must not enqueue parent work")
)
- monkeypatch.setattr(
- "openviking.storage.queuefs.get_queue_manager", get_queue_manager
- )
+ monkeypatch.setattr("openviking.storage.queuefs.get_queue_manager", get_queue_manager)
msg = SemanticMsg(uri=uri, context_type=context_type)
await SemanticProcessor()._enqueue_parent_refresh(
diff --git a/tests/storage/test_semantic_processor_lock_ownership.py b/tests/storage/test_semantic_processor_lock_ownership.py
index 6c438744f..34a3b0cea 100644
--- a/tests/storage/test_semantic_processor_lock_ownership.py
+++ b/tests/storage/test_semantic_processor_lock_ownership.py
@@ -5,6 +5,7 @@ from __future__ import annotations
import pytest
+from openviking.storage.queuefs.process_result import ProcessOutcome
from openviking.storage.queuefs.semantic_msg import SemanticMsg
from openviking.storage.queuefs.semantic_processor import SemanticProcessor
@@ -60,13 +61,20 @@ async def test_memory_semantic_directory_does_not_release_borrowed_lock(monkeypa
@pytest.mark.asyncio
-async def test_missing_root_is_acked_before_lock_scope_is_resolved(monkeypatch):
[email protected]("context_type", ["resource", "memory", "skill"])
+async def test_missing_root_is_acked_before_lock_scope_is_resolved(monkeypatch, context_type):
"""Resolving the lock for a deleted root would recreate it to hold lock metadata."""
processor = SemanticProcessor()
+ fs = _FakeVikingFS()
+
+ async def adopt(handoff):
+ return handoff
+
+ fs._async_agfs.pathlock_adopt = adopt
monkeypatch.setattr(
"openviking.storage.queuefs.semantic_processor.get_viking_fs",
- lambda: _FakeVikingFS(),
+ lambda: fs,
)
async def fail_resolve(*args, **kwargs):
@@ -77,14 +85,23 @@ async def test_missing_root_is_acked_before_lock_scope_is_resolved(monkeypatch):
fail_resolve,
)
+ roots = {
+ "resource": "viking://resources/deleted-parent",
+ "memory": "viking://user/alice/memories/deleted-parent",
+ "skill": "viking://user/alice/skills/deleted-parent",
+ }
msg = SemanticMsg(
- uri="viking://resources/deleted-parent",
- context_type="resource",
+ uri=roots[context_type],
+ context_type=context_type,
recursive=False,
account_id="default",
user_id="alice",
- changes={"deleted": ["viking://resources/deleted-parent/a.md"]},
+ changes={"deleted": [f"{roots[context_type]}/a.md"]},
generation_trigger="content_delete",
+ lock_handoff={"id": "queued-skill-lock"} if context_type == "skill" else None,
)
result = await processor.on_dequeue(msg.to_dict())
- assert result.success
+ assert result.outcome is ProcessOutcome.SUCCESS
+ assert fs._async_agfs.release_calls == (
+ ["queued-skill-lock"] if context_type == "skill" else []
+ )
diff --git a/tests/storage/test_skill_package_cancellation.py b/tests/storage/test_skill_package_cancellation.py
new file mode 100644
index 000000000..623ebe301
--- /dev/null
+++ b/tests/storage/test_skill_package_cancellation.py
@@ -0,0 +1,287 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+import asyncio
+import json
+import threading
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+from uuid import uuid4
+
+import pytest
+
+from openviking.models.embedder.base import EmbedResult
+from openviking.server.identity import RequestContext, Role
+from openviking.service.task_queue_middleware import TaskWorkQueueMiddleware
+from openviking.service.task_work_index import (
+ TaskWorkIndex,
+ bind_task_context,
+ get_task_context,
+ prepare_task_payload,
+)
+from openviking.storage.collection_schemas import TextEmbeddingHandler
+from openviking.storage.queuefs.embedding_msg import EmbeddingMsg
+from openviking.storage.queuefs.named_queue import NamedQueue
+from openviking.storage.queuefs.process_result import ProcessOutcome
+from openviking.storage.queuefs.semantic_dag import DagStats, SemanticDagExecutor
+from openviking.storage.queuefs.semantic_msg import SemanticMsg
+from openviking.storage.queuefs.semantic_processor import SemanticProcessor
+from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
+from openviking_cli.session.user_id import UserIdentifier
+from tests.storage.test_collection_schemas import _DummyConfig, _DummyEmbedder
+from tests.storage.test_semantic_dag_skip_files import _FakeVikingFS
+
+
+def _ctx():
+ return RequestContext(user=UserIdentifier("acc", "alice"), role=Role.USER)
+
+
+def _processor():
+ processor = SemanticProcessor()
+ processor._generate_overview = AsyncMock(return_value="overview")
+ processor._skill_root_semantics = AsyncMock(return_value=("overview", "abstract"))
+ processor._vectorize_single_file = AsyncMock(return_value=True)
+ processor._vectorize_directory = AsyncMock()
+ return processor
+
+
+async def test_cancel_skill_stops_model_work_without_cancelling_other_packages(monkeypatch):
+ root_a = "viking://agent/skills/a"
+ root_b = "viking://agent/skills/b"
+ resource = "viking://resources/document"
+ fs = _FakeVikingFS(
+ {
+ root_a: [{"name": "active.md"}, {"name": "pending.md"}],
+ root_b: [{"name": "b.md"}],
+ resource: [{"name": "resource.md"}],
+ }
+ )
+ monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fs)
+ processor = _processor()
+ started = asyncio.Event()
+ cancelled = asyncio.Event()
+ summaries = []
+ owners = {}
+
+ async def summarize(uri, **kwargs):
+ summaries.append(uri)
+ if uri == f"{root_a}/active.md":
+ started.set()
+ try:
+ await asyncio.Event().wait()
+ finally:
+ cancelled.set()
+ return {"name": uri.rsplit("/", 1)[-1], "summary": "summary"}
+
+ async def vectorize(**kwargs):
+ owner = get_task_context()
+ owners[kwargs["file_path"]] = owner.task_id if owner else None
+ return True
+
+ processor._generate_single_file_summary = summarize
+ processor._vectorize_single_file = vectorize
+ with bind_task_context("skill-a", "acc", "alice"):
+ executor_a = SemanticDagExecutor(processor, "skill", 1, _ctx())
+ task_a = asyncio.create_task(executor_a.run(root_a))
+ await asyncio.wait_for(started.wait(), 1)
+ with bind_task_context("skill-b", "acc", "alice"):
+ executor_b = SemanticDagExecutor(processor, "skill", 1, _ctx())
+ task_b = asyncio.create_task(executor_b.run(root_b))
+ executor_resource = SemanticDagExecutor(processor, "resource", 1, _ctx())
+ task_resource = asyncio.create_task(executor_resource.run(resource))
+ task_a.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await asyncio.wait_for(task_a, 1)
+ await asyncio.wait_for(asyncio.gather(task_b, task_resource), 1)
+ assert cancelled.is_set()
+ assert f"{root_a}/pending.md" not in summaries
+ assert owners == {f"{root_b}/b.md": "skill-b", f"{resource}/resource.md": None}
+ assert not any(uri.startswith(root_a) for uri, _content in fs.writes)
+
+
+async def test_cancel_skill_waits_for_started_threaded_sidecar_write(monkeypatch):
+ root = "viking://agent/skills/demo/reference"
+ fs = _FakeVikingFS({root: []})
+ monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fs)
+ started = asyncio.Event()
+ release = threading.Event()
+ loop = asyncio.get_running_loop()
+
+ def write_in_thread(path, content):
+ loop.call_soon_threadsafe(started.set)
+ assert release.wait(3)
+ fs.writes.append((path, content))
+
+ async def write(path, content, **kwargs):
+ await asyncio.to_thread(write_in_thread, path, content)
+
+ fs.write_file = write
+ processor = _processor()
+ executor = SemanticDagExecutor(processor, "skill", 1, _ctx())
+ worker = asyncio.create_task(executor.run(root))
+ try:
+ await asyncio.wait_for(started.wait(), 1)
+ worker.cancel()
+ await asyncio.sleep(0)
+ worker.cancel()
+ await asyncio.sleep(0.03)
+ assert not worker.done()
+ assert not fs.writes
+ finally:
+ release.set()
+ with pytest.raises(asyncio.CancelledError):
+ await asyncio.wait_for(worker, 1)
+ assert fs.writes
+ writes_at_exit = list(fs.writes)
+ await asyncio.sleep(0.03)
+ assert fs.writes == writes_at_exit
+ processor._vectorize_directory.assert_not_awaited()
+
+
[email protected]("during_write", [False, True])
+async def test_active_skill_embedding_cancel_settles_only_after_write_exit(
+ monkeypatch, during_write
+):
+ started = asyncio.Event()
+ release = threading.Event()
+ loop = asyncio.get_running_loop()
+ writes = []
+ model_cancelled = asyncio.Event()
+
+ class Embedder(_DummyEmbedder):
+ async def embed_async(self, text, is_query=False):
+ if not during_write:
+ started.set()
+ try:
+ await asyncio.Event().wait()
+ finally:
+ model_cancelled.set()
+ return EmbedResult(dense_vector=[0.1, 0.2])
+
+ def write_in_thread(data):
+ loop.call_soon_threadsafe(started.set)
+ assert release.wait(3)
+ writes.append(data["uri"])
+ return "vector-id"
+
+ async def upsert(data, **kwargs):
+ return await asyncio.to_thread(write_in_thread, data)
+
+ monkeypatch.setattr(
+ "openviking_cli.utils.config.get_openviking_config", lambda: _DummyConfig(Embedder())
+ )
+ handler = TextEmbeddingHandler(
+ SimpleNamespace(is_closing=False, uses_content_field=False, upsert=upsert)
+ )
+ telemetry_id = str(uuid4())
+ tracker = get_request_wait_tracker()
+ tracker.register_request(telemetry_id)
+ msg = EmbeddingMsg(
+ "content",
+ {"uri": "viking://agent/skills/demo/file.md", "context_type": "skill"},
+ telemetry_id=telemetry_id,
+ )
+ tracker.register_embedding_root(telemetry_id, msg.id)
+ index = TaskWorkIndex()
+ cancelling = set()
+ index.set_callbacks(
+ finalize_before_ack=AsyncMock(),
+ is_cancellation_requested=lambda task_id: task_id in cancelling,
+ )
+ with bind_task_context("skill-task", "acc", "alice"):
+ payload, metadata = prepare_task_payload(msg.to_dict())
+ queue = NamedQueue(
+ object(),
+ "/queue",
+ "Embedding",
+ dequeue_handler=handler,
+ middlewares=[TaskWorkQueueMiddleware(index)],
+ )
+ envelope = {"data": json.dumps(payload)}
+ assert index.register(queue.name, metadata)
+ worker = asyncio.create_task(queue.process_dequeued(envelope))
+ try:
+ await asyncio.wait_for(started.wait(), 1)
+ cancelling.add("skill-task")
+ index.cancel_active("skill-task")
+ await asyncio.sleep(0.03)
+ if during_write:
+ assert not worker.done()
+ assert not tracker.is_complete(telemetry_id)
+ assert index.has_work("skill-task")
+ assert not writes
+ else:
+ await asyncio.wait_for(worker, 1)
+ assert model_cancelled.is_set()
+ finally:
+ release.set()
+ result = await asyncio.wait_for(worker, 1)
+ assert result.outcome is ProcessOutcome.CANCELLED
+ await index.prepare_ack(queue.name, envelope)
+ assert not index.has_work("skill-task")
+ assert tracker.is_complete(telemetry_id)
+ tracker.cleanup(telemetry_id)
+ assert len(writes) == int(during_write)
+
+
+async def test_semantic_cancel_drains_embeddings_before_releasing_package(monkeypatch):
+ tracker = get_request_wait_tracker()
+ telemetry_id = str(uuid4())
+ tracker.register_request(telemetry_id)
+ msg = SemanticMsg(
+ uri="viking://agent/skills/demo",
+ context_type="skill",
+ telemetry_id=telemetry_id,
+ propagate_to_parent=False,
+ )
+ tracker.register_semantic_root(telemetry_id, msg.id)
+ started = asyncio.Event()
+ events = []
+
+ class Executor:
+ def __init__(self, **kwargs):
+ self.stale = False
+
+ async def run(self, uri):
+ tracker.register_embedding_root(telemetry_id, "embedding")
+ started.set()
+
+ def get_stats(self):
+ return DagStats()
+
+ class Lease:
+ lock = {"lease_ref": "skill"}
+
+ async def close(self):
+ events.append("released")
+
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor", Executor
+ )
+ monkeypatch.setattr(
+ SemanticProcessor, "_resolve_skill_semantic_lock", AsyncMock(return_value=Lease())
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_processor.get_viking_fs",
+ lambda: SimpleNamespace(exists=AsyncMock(return_value=True)),
+ )
+ worker = asyncio.create_task(SemanticProcessor().on_dequeue({"data": msg.to_json()}))
+ await asyncio.wait_for(started.wait(), 1)
+ worker.cancel()
+ await asyncio.sleep(0)
+ worker.cancel()
+ await asyncio.sleep(0.03)
+ assert not worker.done()
+ assert events == []
+ assert not tracker.is_complete(telemetry_id)
+ events.append("embedding-settled")
+ tracker.mark_embedding_done(telemetry_id, "embedding")
+ with pytest.raises(asyncio.CancelledError):
+ await asyncio.wait_for(worker, 1)
+ assert events == ["embedding-settled", "released"]
+ assert tracker.is_complete(telemetry_id)
+ tracker.cleanup(telemetry_id)
diff --git a/tests/storage/test_skill_package_semantics.py b/tests/storage/test_skill_package_semantics.py
new file mode 100644
index 000000000..fd15f873b
--- /dev/null
+++ b/tests/storage/test_skill_package_semantics.py
@@ -0,0 +1,76 @@
+import asyncio
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import pytest
+
+from openviking.storage.queuefs.semantic_dag import DagStats
+from openviking.storage.queuefs.semantic_msg import SemanticMsg
+from openviking.storage.queuefs.semantic_processor import SemanticProcessor
+from openviking.telemetry import get_current_telemetry, register_telemetry
+from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
+
+
+async def test_skill_worker_keeps_package_locked_until_embeddings_finish(monkeypatch):
+ root = "viking://agent/skills/demo"
+ emitted = asyncio.Event()
+ locked = True
+ events = []
+ tracker = get_request_wait_tracker()
+ telemetry = get_current_telemetry()
+ register_telemetry(telemetry)
+ tracker.register_request(telemetry.telemetry_id)
+ msg = SemanticMsg(
+ uri=root,
+ context_type="skill",
+ telemetry_id=telemetry.telemetry_id,
+ generation_trigger="skill_ingest",
+ propagate_to_parent=False,
+ )
+ tracker.register_semantic_root(msg.telemetry_id, msg.id)
+
+ class Lease:
+ lock = {"lease_ref": "package"}
+
+ async def close(self):
+ nonlocal locked
+ locked = False
+ events.append("released")
+
+ class Executor:
+ def __init__(self, **kwargs):
+ self.stale = False
+
+ async def run(self, uri):
+ tracker.register_embedding_root(msg.telemetry_id, "file-embedding")
+ emitted.set()
+
+ def get_stats(self):
+ return DagStats()
+
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor", Executor
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_processor.SemanticLockScope.resolve",
+ AsyncMock(return_value=Lease()),
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_processor.get_viking_fs",
+ lambda: SimpleNamespace(exists=AsyncMock(return_value=True)),
+ )
+ worker = asyncio.create_task(SemanticProcessor().on_dequeue({"data": msg.to_json()}))
+ await asyncio.wait_for(emitted.wait(), 1)
+ assert locked and not worker.done() and not tracker.is_complete(msg.telemetry_id)
+ # The HTTP timeout cleanup must not release the package while embedding runs.
+ tracker.cleanup(msg.telemetry_id)
+ await asyncio.sleep(0.06)
+ assert locked and not worker.done() and not tracker.is_complete(msg.telemetry_id)
+ # Deletion cannot pass the package lease while a file vector can still write.
+ events.append("embedding-written")
+ tracker.mark_embedding_done(msg.telemetry_id, "file-embedding", vector_written=True)
+ await asyncio.wait_for(worker, 1)
+ assert not locked and tracker.is_complete(msg.telemetry_id)
+ assert events == ["embedding-written", "released"]
+ tracker.cleanup(msg.telemetry_id)
diff --git a/tests/unit/test_skill_privacy_cancellation.py b/tests/unit/test_skill_privacy_cancellation.py
new file mode 100644
index 000000000..b742929ca
--- /dev/null
+++ b/tests/unit/test_skill_privacy_cancellation.py
@@ -0,0 +1,74 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+"""Skill config writes must settle before their package lock is released."""
+
+import asyncio
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import pytest
+
+from openviking.server.identity import RequestContext, Role
+from openviking.utils.skill_processor import SkillProcessingPreparation, SkillProcessor
+from openviking_cli.session.user_id import UserIdentifier
+
+
+def processing():
+ processor = SkillProcessor(vikingdb=None)
+ processor.apply_skill_privacy = AsyncMock()
+ processor._write_skill_content = AsyncMock()
+ processor._write_auxiliary_files = AsyncMock()
+ processor._enqueue_skill_package = AsyncMock()
+ agfs = SimpleNamespace(
+ pathlock_acquire_tree=AsyncMock(return_value={"lease_ref": "package"}),
+ pathlock_release=AsyncMock(),
+ )
+ fs = SimpleNamespace(_async_agfs=agfs, _uri_to_path=lambda uri, **kw: uri)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)
+ prepared = SkillProcessingPreparation(
+ skill_dict={"name": "config-cancellation", "description": "Example", "content": "body"},
+ auxiliary_files=[],
+ base_path=None,
+ cleanup_path=None,
+ privacy_values={"api_key": "test-value"},
+ )
+ return processor, fs, ctx, prepared
+
+
+async def test_cancelling_config_write_keeps_package_locked_until_write_exits(processing):
+ processor, fs, ctx, prepared = processing
+ entered = asyncio.Event()
+ finish_write = asyncio.Event()
+ write_exited = asyncio.Event()
+
+ async def write_config(*args, **kwargs):
+ fs._async_agfs.pathlock_acquire_tree.assert_awaited_once()
+ entered.set()
+ await finish_write.wait()
+ fs._async_agfs.pathlock_release.assert_not_awaited()
+ write_exited.set()
+ return prepared.skill_dict
+
+ processor.apply_skill_privacy.side_effect = write_config
+ task = asyncio.create_task(processor.process_prepared_skill(prepared, fs, ctx))
+ try:
+ await asyncio.wait_for(entered.wait(), timeout=1)
+ task.cancel()
+ # Observe cancellation while the config operation remains in flight.
+ await asyncio.sleep(0)
+ await asyncio.sleep(0)
+ assert not task.done()
+ fs._async_agfs.pathlock_release.assert_not_awaited()
+ finish_write.set()
+ with pytest.raises(asyncio.CancelledError):
+ await asyncio.wait_for(task, timeout=1)
+ assert write_exited.is_set()
+ fs._async_agfs.pathlock_release.assert_awaited_once_with({"lease_ref": "package"})
+ processor._write_skill_content.assert_not_awaited()
+ processor._write_auxiliary_files.assert_not_awaited()
+ processor._enqueue_skill_package.assert_not_awaited()
+ finally:
+ finish_write.set()
+ await asyncio.gather(task, return_exceptions=True)
diff --git a/tests/unit/test_skill_processor_none.py b/tests/unit/test_skill_processor_none.py
index 6d9f7df6f..4a6f426db 100644
--- a/tests/unit/test_skill_processor_none.py
+++ b/tests/unit/test_skill_processor_none.py
@@ -157,8 +157,12 @@ async def test_process_skill_preserves_hyphenated_allowed_tools_in_meta():
vikingdb.enqueue_embedding_msg = AsyncMock(return_value=False)
viking_fs = MagicMock()
viking_fs.write_context = AsyncMock()
+ viking_fs._async_agfs.pathlock_acquire_tree = AsyncMock(return_value={"lease_ref": "test"})
+ viking_fs._async_agfs.pathlock_release = AsyncMock()
processor = SkillProcessor(vikingdb=vikingdb)
+ processor._write_skill_content = AsyncMock()
+ processor._enqueue_skill_package = AsyncMock()
result = await processor.process_skill(
data={
"name": "dict-skill",
@@ -173,9 +177,14 @@ async def test_process_skill_preserves_hyphenated_allowed_tools_in_meta():
)
assert result["name"] == "dict-skill"
- written_content = viking_fs.write_context.await_args.kwargs["content"]
+ from openviking.core.skill_loader import SkillLoader
+
+ written = processor._write_skill_content.await_args.kwargs
+ assert written["overview"] == "# Dict Skill"
+ written_content = SkillLoader.to_skill_md(written["skill_dict"])
assert "allowed-tools: Read" in written_content
assert "tags:" in written_content
- embedding_msg = vikingdb.enqueue_embedding_msg.await_args.args[0]
- assert embedding_msg.context_data["meta"]["allowed_tools"] == ["Read"]
- assert embedding_msg.context_data["meta"]["tags"] == ["dict"]
+ assert written["skill_dict"]["allowed_tools"] == ["Read"]
+ assert written["skill_dict"]["tags"] == ["dict"]
+ processor._enqueue_skill_package.assert_awaited_once()
+ viking_fs._async_agfs.pathlock_release.assert_not_awaited()
diff --git a/tests/unit/test_skill_search_contract.py b/tests/unit/test_skill_search_contract.py
new file mode 100644
index 000000000..a72eae71a
--- /dev/null
+++ b/tests/unit/test_skill_search_contract.py
@@ -0,0 +1,78 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+"""Skill router contracts using grouped search results and no running server."""
+
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import pytest
+
+from openviking.server.routers import skills as skills_router
+
+
+def _grouped_hit(root_uri, score=0.91):
+ return {
+ "uri": root_uri + "/reference/nested/backup.md",
+ "level": 2,
+ "context_type": "skill",
+ "score": score,
+ "abstract": "name: file-name\ndescription: File-only description\ntags: [file-only]",
+ }
+
+
+def fake_search(monkeypatch):
+ async def root_abstract(uri, *, ctx):
+ name = uri.rsplit("/", 1)[-1]
+ return (
+ f"name: {name}\n"
+ "description: Root skill description\n"
+ "tags: [backup, production]\n"
+ "allowed_tools: [Read, Bash]\n"
+ )
+
+ find = AsyncMock()
+ abstract = AsyncMock(side_effect=root_abstract)
+ service = SimpleNamespace(
+ search=SimpleNamespace(find_skills=find),
+ fs=SimpleNamespace(abstract=abstract),
+ )
+
+ async def run_operation(*, operation, telemetry, fn):
+ return SimpleNamespace(result=await fn(), telemetry=None)
+
+ monkeypatch.setattr(skills_router, "get_service", lambda: service)
+ monkeypatch.setattr(skills_router, "run_operation", run_operation)
+ return SimpleNamespace(find=find, abstract=abstract)
+
+
+async def test_find_skills_applies_one_limit_across_same_named_skills_in_both_spaces(
+ fake_search, request_context
+):
+ user_root = f"viking://user/{request_context.user.user_id}/skills"
+ agent_root = "viking://agent/skills"
+ user_hit = _grouped_hit(f"{user_root}/data.service", 0.89)
+ agent_hit = _grouped_hit(f"{agent_root}/data.service", 0.94)
+
+ async def find(**kwargs):
+ if kwargs["target_uri"] == user_root:
+ return {"skills": [user_hit, _grouped_hit(f"{user_root}/other", 0.71)]}
+ assert kwargs["target_uri"] == agent_root
+ return {"skills": [agent_hit, _grouped_hit(f"{agent_root}/another", 0.74)]}
+
+ fake_search.find.side_effect = find
+
+ response = await skills_router.find_skills(
+ skills_router.FindSkillsRequest(query="backup", limit=2),
+ _ctx=request_context,
+ )
+
+ result = response["result"]
+ assert result["total"] == 2
+ assert [hit["name"] for hit in result["skills"]] == ["data.service", "data.service"]
+ assert [hit["root_uri"] for hit in result["skills"]] == [
+ f"{agent_root}/data.service",
+ f"{user_root}/data.service",
+ ]
+ assert [hit["uri"] for hit in result["skills"]] == [agent_hit["uri"], user_hit["uri"]]
diff --git a/tests/unit/test_vectorize_file_strategy.py b/tests/unit/test_vectorize_file_strategy.py
index 3d9853d31..c3003323a 100644
--- a/tests/unit/test_vectorize_file_strategy.py
+++ b/tests/unit/test_vectorize_file_strategy.py
@@ -594,7 +594,9 @@ async def test_vectorize_directory_meta_l1_abstract_is_overview(monkeypatch):
}
await embedding_utils.vectorize_directory_meta(
uri=uri,
- abstract=render_abstract_overview(ContextLevel.ABSTRACT, uri, "Visible abstract.", metadata),
+ abstract=render_abstract_overview(
+ ContextLevel.ABSTRACT, uri, "Visible abstract.", metadata
+ ),
overview=render_abstract_overview(ContextLevel.OVERVIEW, uri, overview, metadata),
ctx=DummyReq(),
)
@@ -1073,3 +1075,35 @@ async def test_vectorize_directory_meta_truncates_oversized_abstract(monkeypatch
abstract = item.context_data["abstract"]
assert len(abstract.encode("utf-8")) <= embedding_utils._ABSTRACT_MAX_BYTES
assert abstract.encode("utf-8").decode("utf-8") == abstract
+
+
+async def test_skill_directory_body_frontmatter_is_not_parsed_twice(monkeypatch):
+ from openviking.storage.abstract_overview import (
+ AbstractOverviewFormatError,
+ render_abstract_overview,
+ )
+
+ queue = DummyQueue()
+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))
+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: DummyFS("ignored"))
+ uri = "viking://agent/skills/demo"
+ body = "---\nname: demo\ndescription: Tool instructions\n---\n\n# Parameters\nKeep this body."
+ await embedding_utils.vectorize_directory_meta(
+ uri, "name: demo", body, context_type="skill", ctx=DummyReq(), content_is_body=True
+ )
+ overview_msg = next(item for item in queue.items if item.context_data["level"] == 1)
+ assert overview_msg.context_data["abstract"] == body
+ assert body in overview_msg.message
+
+ # Existing callers still validate serialized sidecars, and strip only the
+ # outer system metadata from a valid document containing frontmatter.
+ with pytest.raises(AbstractOverviewFormatError, match="must contain directory"):
+ await embedding_utils.vectorize_directory_meta(uri, "abstract", body, ctx=DummyReq())
+ queue.items.clear()
+ await embedding_utils.vectorize_directory_meta(
+ uri, "abstract", render_abstract_overview(1, uri, body), ctx=DummyReq()
+ )
+ overview_msg = next(item for item in queue.items if item.context_data["level"] == 1)
+ assert overview_msg.context_data["abstract"] == body
+ assert body in overview_msg.message