Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(reindex) user orphan convergence (#6663)
Jiahui Zhou6afd9ba30084
diff --git a/docs/en/api/12-content.md b/docs/en/api/12-content.mdindex 33f91b8fa..f5a223393 100644--- a/docs/en/api/12-content.md+++ b/docs/en/api/12-content.md@@ -647,7 +647,7 @@ ov set-tags viking://resources/project/ \### reindex()-Validate and repair semantic and/or vector artifacts for existing content already stored in OpenViking. Resource and skill targets use incremental RFV (Request / Formal / Vector) convergence by default: unchanged file fingerprints, directory L0/L1 visible-body fingerprints, and request scalars are skipped. Use `force=true` for an unconditional rebuild.+Validate and repair semantic and/or vector artifacts for existing content already stored in OpenViking. For resources and skills, reindex compares persisted content with existing vector records: missing or stale records are rebuilt, vector records whose content no longer exists are deleted, and matching records are skipped. Deletion is allowed only after both sides are read completely. Use `force=true` to rebuild existing content unconditionally; completeness checks still apply.This API operates on existing `viking://...` content. It does not import new files. For normal ingestion, use [Resources](02-resources.md).@@ -685,20 +685,28 @@ Session namespaces are not supported by `reindex()`. Requests for`viking://session/...` or `viking://user/<user_id>/sessions/...` are rejected;when reindexing a broader user namespace, session subtrees are skipped.-**Modes**+For a concrete user root, namespace reindex processes `resources`, skill packages, memory shards, peer resources/memories, and other indexed files or custom top-level directories. Broad operations use shallow paginated discovery and skip session, privacy, internal settings, and storage-control subtrees. An actor-peer view must use an explicit visible content target because a broader namespace cannot provide a complete persisted-content view.++Namespace reindex discovers targets from both persisted content and existing vector records. It reads and reuses one complete lightweight vector-record list per supported URI prefix, so it can clean missing content roots. Retryable failures or record-count changes retry the whole read at most three times; an incomplete read is never used for deletion. Directly requesting an already-missing target still returns `NOT_FOUND`; use an existing parent scope or namespace to remove its leftover vectors. A concrete content request never expands to sibling scopes.++Memory uses the original file's L2 and no longer generates virtual `#chunk_NNNN` records. Reindex removes records whose content is gone and all legacy chunks, including a requested file's own old chunks. If the original memory file cannot be processed, its existing L2 is preserved, and real files whose names resemble chunk URIs are not deleted.-- `vectors_only`: compares current source MD5 values with vector-record MD5 values through RFV and rebuilds only missing or stale L0/L1/L2 records; it does not rewrite `.abstract.md` or `.overview.md`-- `semantic_and_vectors`: uses the same RFV state and `ContextUpdatePlan` to drive semantic repair and L0/L1/L2 updates, without a second manual vector scan+Memory directory L0/L1 records require a nonempty, valid `.abstract.md`/`.overview.md` source for the respective level. A confirmed missing or empty source removes old records at that level; IO or format errors preserve them, increase `failed_records`, and add a warning. The other level and healthy files continue processing.++For `semantic_and_vectors`, a memory directory removes records whose content is gone first, then reuses the same processing flow as `add_resources` and waits for the related semantic and vector work. An exact memory file only rebuilds its own L2 and removes legacy chunks; it does not regenerate directory semantics.++**Modes**-For `resource` and `skill`, `semantic_and_vectors` refreshes directory/file semantic artifacts, including `.abstract.md` and `.overview.md`. For `memory`, it rebuilds the current persisted memory subtree semantics and vectors, but it does not replay historical extraction order.+- `vectors_only`: compares current content fingerprints with vector-record fingerprints and rebuilds only missing or stale L0/L1/L2 records; it does not rewrite `.abstract.md` or `.overview.md`+- `semantic_and_vectors`: repairs semantic artifacts first, then updates the corresponding L0/L1/L2 vectors-Resource/skill `vectors_only` and `semantic_and_vectors` share one F traversal and one narrow-field V inventory. Each source is read once during state resolution and that result is reused while executing the plan. `force=true` bypasses MD5 equality but still performs completeness and scope checks.+For `resource` and `skill`, `semantic_and_vectors` refreshes directory/file semantic artifacts, including `.abstract.md` and `.overview.md`. For `memory`, it rebuilds the current persisted memory subtree semantics and vectors.-For a resource/skill directory, `recursive=false` limits both modes to the target directory's own L0/L1 records. Direct children are read only as existing REUSE inputs for the target aggregation; descendants are neither scanned nor rebuilt. Namespace containers have no index records of their own, so namespace targets reject `recursive=false` instead of silently becoming a no-op. Memory remains on the existing reindex path.+Both resource/skill modes share one persisted-content scan and one lightweight vector-record read. Each source is read once and reused during later processing. `force=true` bypasses fingerprint equality but still performs completeness and scope checks.-When the F snapshot is complete, the RFV diff identifies records present in V but absent from F as orphans and removes them during the same reindex. Deletion fails closed when the F snapshot is incomplete.+For a resource/skill directory, `recursive=false` limits both modes to the target directory's own L0/L1 records. Direct children are used only to aggregate the target directory; they are not scanned or rebuilt. Namespace containers have no index records of their own, so namespace targets reject `recursive=false` instead of silently becoming a no-op. Memory remains on the existing reindex path.-When non-empty `tags` are provided, tags are included in the same upsert as each vector record produced by reindex; reindex does not call `set_tags` afterwards. Directory and namespace reindex operations apply tags to successfully rebuilt directory L0/L1 and leaf L2 records. `replace` overwrites existing tags, while `append` merges by key. `replace` with an empty tag list is a no-op. `clear` removes existing tags and does not require `tags`; if values are supplied with `clear`, they are ignored.+When non-empty `tags` are provided, tags are saved in the same write as each vector record produced by reindex; reindex does not call `set_tags` afterwards. Directory and namespace reindex operations apply tags to successfully rebuilt directory L0/L1 and leaf L2 records. `replace` overwrites existing tags, while `append` merges by key. `replace` with an empty tag list is a no-op. `clear` removes existing tags and does not require `tags`; if values are supplied with `clear`, they are ignored.Subtree reindex is not transactional. Records skipped because no semantic source is available, or records whose embedding fails, do not receive the new tags.@@ -834,7 +842,7 @@ Task records are persisted under `/local/{account_id}/_system/tasks/{user_id}/{t| mode | Effective reindex mode || scanned_records | Number of records or semantic sources considered || rebuilt_records | Number of vector records successfully rebuilt |-| deleted_records | Number of orphan vector records confirmed and deleted by the RFV diff |+| deleted_records | Number of vector records confirmed to have no corresponding content and deleted || unsupported_records | Number of records skipped because no usable vector source was available || failed_records | Number of records that failed while rebuilding || duration_ms | Synchronous run duration in milliseconds |@@ -843,11 +851,7 @@ Task records are persisted under `/local/{account_id}/_system/tasks/{user_id}/{t**Behavior notes**-- `vectors_only` and `semantic_and_vectors` are non-destructive. They use rebuild/upsert behavior and do not require dropping the vector collection first.-- `viking://` reindex fans out to supported top-level namespaces and excludes `session`.-- Namespace reindex operations such as `viking://user` propagate to supported child content types.-- `vectors_only` is the right mode when only the embedding model or vector index needs to be refreshed.-- `semantic_and_vectors` is the right mode when semantic artifacts themselves must be regenerated before re-vectorization.+- Reindex does not delete the whole vector collection. It overwrites records for live content and deletes only records confirmed to have no corresponding content.- Only one reindex task can run for the same URI and owner at a time. A concurrent request for the same target returns a conflict.- For resource files, text files can use file content when no summary is available. Non-text files require a generated summary or existing vector record fallback; otherwise they are counted as unsupported.diff --git a/docs/zh/api/12-content.md b/docs/zh/api/12-content.mdindex 5062a107f..3616d22df 100644--- a/docs/zh/api/12-content.md+++ b/docs/zh/api/12-content.md@@ -647,7 +647,7 @@ ov set-tags viking://resources/project/ \### reindex()-对已经存储在 OpenViking 中的现有内容,校验并修复语义产物和/或向量索引。resource 和 skill 默认使用 RFV(Request / Formal / Vector)增量收敛:文件内容指纹、目录 L0/L1 可见正文指纹和请求标量都未变化时直接跳过。需要无条件重建时使用 `force=true`。+对已经存储在 OpenViking 中的现有内容,校验并修复语义产物和/或向量索引。对于 resource 和 skill,reindex 会比较正式内容与已有向量记录:缺失或过期的记录会被重建,内容已不存在的孤儿向量会被删除,状态一致的记录会被跳过。只有完整读取两边状态后才允许删除;需要无条件重建现有内容时使用 `force=true`,但完整性检查仍会执行。这个接口面向已有的 `viking://...` 内容,不负责导入新文件。常规导入请使用 [Resources](02-resources.md)。@@ -685,20 +685,28 @@ HTTP 请求体会忽略未知字段,便于客户端与服务端在字段演进`viking://user/<user_id>/sessions/...` 会被拒绝;重建更大的 user 命名空间时,session 子树会被跳过。-**模式说明**+具体 user 根的 namespace reindex 会处理 `resources`、skill package、memory shard、peer 的 resources/memories,以及其他已索引的直属文件和自定义一级目录。宽范围操作使用分页浅层发现,跳过 session、privacy、内部 settings 与存储控制子树。带 actor-peer 视图的请求应指定明确可见的内容目标,因为更大的 namespace 无法提供完整的正式内容视图。++namespace 会结合正式内容和已有向量记录发现目标,并按受支持的 URI 前缀一次读取、复用完整的轻量向量记录清单,因此可以清理已经缺失的内容根。可重试错误或记录数量变化最多触发三次整份重读;读取不完整时不执行删除。直接请求已经缺失的目标仍返回 `NOT_FOUND`,应从仍存在的父范围或 namespace 清理其残留向量;具体内容请求不会扩大到兄弟范围。++memory 使用原文件的 L2,不再生成虚拟 `#chunk_NNNN` 记录。reindex 会清理全部孤儿记录和历史 chunk;显式文件请求也会清理自身旧 chunk。原 memory 文件处理失败时保留其已有 L2,名称类似 chunk URI 的真实文件也不会被误删。-- `vectors_only`:通过 RFV 对比当前 source MD5 与向量记录 MD5,只重建缺失或过期的 L0/L1/L2;不会重写 `.abstract.md` 和 `.overview.md`-- `semantic_and_vectors`:通过同一 RFV 状态和 `ContextUpdatePlan` 驱动语义修复及 L0/L1/L2 向量更新,不再执行第二次手工向量扫描+memory 目录 L0/L1 分别要求有效且非空的 `.abstract.md`/`.overview.md` 来源。确认来源缺失或为空时删除该 level 的旧记录;IO 或格式错误时保留旧记录,增加 `failed_records` 并报告 warning。另一 level 与健康文件继续处理。++`semantic_and_vectors` 模式中,memory 目录目标先清理孤儿记录,再复用与 `add_resources` 相同的处理流程,并等待相关语义和向量任务完成。精确 memory 文件目标只重建自身 L2 和清理历史 chunk,不重新生成目录语义。++**模式说明**-对于 `resource` 和 `skill`,`semantic_and_vectors` 会刷新目录/文件语义产物,包括 `.abstract.md` 和 `.overview.md`。对于 `memory`,它会重建当前已持久化 memory 子树的语义和向量,但不会回放历史记忆抽取顺序。+- `vectors_only`:比较当前内容指纹与向量记录指纹,只重建缺失或过期的 L0/L1/L2;不会重写 `.abstract.md` 和 `.overview.md`+- `semantic_and_vectors`:先修复语义产物,再更新对应的 L0/L1/L2 向量-resource/skill 的 `vectors_only` 和 `semantic_and_vectors` 共用一次 F 遍历和一次窄字段 V inventory。每个 source 在状态解析时读取一次,实际处理复用本次读取结果,不会为了执行 Plan 再读取一次 F。`force=true` 会跳过 MD5 相等判断并重新处理作用域内全部可用 level;完整性检查仍然执行。+对于 `resource` 和 `skill`,`semantic_and_vectors` 会刷新目录/文件语义产物,包括 `.abstract.md` 和 `.overview.md`。对于 `memory`,它会重建当前已持久化 memory 子树的语义和向量。-对 resource/skill 目录设置 `recursive=false` 时,两种模式都只处理目标目录自身的 L0/L1。直接 children 只作为目标聚合的既有 REUSE 输入读取,不扫描或重建下级内容。namespace 容器自身没有索引记录,因此 namespace 目标不接受 `recursive=false`,避免请求静默成为 no-op。memory 仍使用原有 reindex 链路。+resource/skill 的两种模式共用一次正式内容扫描和一次轻量向量记录读取。每个来源只读取一次,并在后续处理中复用。`force=true` 会跳过指纹相等判断并重新处理范围内全部可用 level;完整性检查仍然执行。-F snapshot 完整时,RFV 差分会把 V 中存在但 F 中不存在的记录识别为 orphan,并在同一次 reindex 中清理;F snapshot 不完整时 fail-closed,不执行删除。+对 resource/skill 目录设置 `recursive=false` 时,两种模式都只处理目标目录自身的 L0/L1。直接子项只用于聚合目标目录,不会被扫描或重建。namespace 容器自身没有索引记录,因此 namespace 目标不接受 `recursive=false`,避免请求静默成为 no-op。memory 仍使用原有 reindex 链路。-传入非空 `tags` 时,标签会随 reindex 生成的向量记录在同一次 upsert 中写入,不会在完成后额外调用 `set_tags`。目录或 namespace reindex 会把标签应用到本次成功重建的目录 L0/L1 和叶子 L2 记录。`replace` 覆盖已有标签,`append` 按 key 合并;`replace` 配合空数组时不修改已有标签。`clear` 不要求传 `tags`,会清空已有标签;即使同时传入标签值也会忽略。+传入非空 `tags` 时,标签会随 reindex 生成的向量记录在同一次写入中保存,不会在完成后额外调用 `set_tags`。目录或 namespace reindex 会把标签应用到本次成功重建的目录 L0/L1 和叶子 L2 记录。`replace` 覆盖已有标签,`append` 按 key 合并;`replace` 配合空数组时不修改已有标签。`clear` 不要求传 `tags`,会清空已有标签;即使同时传入标签值也会忽略。子树 reindex 不是事务性操作。如果部分记录因缺少语义来源或 embedding 失败而未重建,只有成功写入的记录会更新标签。@@ -834,7 +842,7 @@ GET /api/v1/tasks?task_type=admin_reindex&resource_id=viking://resources| mode | 实际执行的 reindex 模式 || scanned_records | 被检查的记录或语义源数量 || rebuilt_records | 成功重建的向量记录数量 |-| deleted_records | RFV 差分确认并删除的 orphan 向量记录数量 |+| deleted_records | 已确认没有对应内容并删除的向量记录数量 || unsupported_records | 因没有可用向量来源而跳过的记录数量 || failed_records | 重建失败的记录数量 || duration_ms | 同步执行耗时,单位毫秒 |@@ -843,11 +851,7 @@ GET /api/v1/tasks?task_type=admin_reindex&resource_id=viking://resources**行为说明**-- `vectors_only` 和 `semantic_and_vectors` 是非破坏式的,采用重建/覆盖写入,不需要先 drop 向量集合。-- 对 `viking://` 发起 reindex 时,会向下分发到支持的顶层命名空间,并显式排除 `session`。-- 命名空间级 reindex,例如 `viking://user`,会继续传播到其支持的子内容类型。-- 如果只是 embedding 模型或向量索引需要刷新,应使用 `vectors_only`。-- 如果语义产物本身也需要重建,再做重向量化,应使用 `semantic_and_vectors`。+- Reindex 不会删除整个向量集合;它覆盖有效内容的向量记录,并只删除已确认没有对应内容的记录。- 同一个 URI 和 owner 同时只能运行一个 reindex 任务。对同一目标的并发请求会返回 conflict。- 对 resource 文件,文本文件在没有 summary 时可以使用文件正文;非文本文件需要已生成的 summary 或已有向量记录 fallback,否则会计为 unsupported。diff --git a/openviking/service/core.py b/openviking/service/core.pyindex c39797e01..993c805bc 100644--- a/openviking/service/core.py+++ b/openviking/service/core.py@@ -948,10 +948,7 @@ class OpenVikingService:effective_ctx = ctx or RequestContext(user=self.user, role=Role.ROOT)from openviking.service.reindex_executor import ReindexExecutor- executor = ReindexExecutor(- vlm_resolver=self._vlm_resolver,- vector_config_resolver=self._vector_config_resolver,- )+ executor = ReindexExecutor()execute_kwargs = {"uri": uri,"mode": mode,diff --git a/openviking/service/reindex_executor.py b/openviking/service/reindex_executor.pyindex 6883d8c99..91805f403 100644--- a/openviking/service/reindex_executor.py+++ b/openviking/service/reindex_executor.py@@ -5,58 +5,75 @@from __future__ import annotationsimport asyncio+import reimport timefrom copy import deepcopyfrom dataclasses import dataclass, field, replace-from typing import Any, Iterable, Optional+from typing import Any, Iterable, Mapping, Optional-from openviking.config.vlm import VLMResolverfrom openviking.core.context import (Context,ContextLevel,ContextType,- ResourceContentType,Vectorize,)from openviking.core.namespace import (classify_uri,content_owner_context_for_uri,context_type_for_uri,+ is_hidden_by_actor_peer_view,is_session_uri,+ may_include_hidden_actor_peers,owner_space_for_uri,+ uri_parts,)from openviking.core.namespace_policy import namespace_policy_for_urifrom openviking.server.dependencies import get_servicefrom openviking.server.identity import RequestContextfrom openviking.service.task_domain import TaskPatchfrom openviking.service.task_tracker import TaskStatus, get_task_tracker+from openviking.service.task_tracker_concurrency import run_to_completionfrom openviking.service.task_work_index import bind_task_context, get_task_contextfrom openviking.session.memory.utils.memory_file_utils import MemoryFileUtils-from openviking.storage.abstract_overview import body_for_preview, embedding_text_for_body+from openviking.storage.abstract_overview import (+ ABSTRACT_OVERVIEW_FILENAMES,+ body_for_preview,+ embedding_text_for_body,+)+from openviking.storage.errors import LockError+from openviking.storage.internal_names import is_storage_internal_namefrom openviking.storage.pathlock_transfer import PathLockGuardfrom openviking.storage.queuefs.embedding_msg_converter import EmbeddingMsgConverterfrom openviking.storage.queuefs.semantic_msg import SemanticMsgfrom openviking.storage.queuefs.semantic_processor import SemanticProcessorfrom openviking.storage.viking_fs import get_viking_fsfrom openviking.telemetry import get_current_telemetry-from openviking.utils.embedding_input import truncate_embedding_inputfrom openviking.utils.embedding_utils import (_apply_ingest_options,- _decode_text_bytes,_truncate_abstract_bytes,- get_resource_content_type,)from openviking.utils.ingest_options import IngestOptions-from openviking_cli.exceptions import InvalidArgumentError, NotFoundError, OpenVikingError+from openviking_cli.exceptions import (+ InvalidArgumentError,+ NotFoundError,+ OpenVikingError,+ PermissionDeniedError,+)from openviking_cli.utils import VikingURI, get_loggerfrom openviking_cli.utils.config import get_openviking_config-from openviking_cli.utils.config.embedding_config import SUMMARY_TEXT_SOURCESlogger = get_logger(__name__)+_MEMORY_CHUNK_PATTERN = re.compile(r"^(?P<base>.+)#chunk_[0-9]{4,}$")REINDEX_TASK_TYPE = "admin_reindex"REINDEX_ENQUEUE_ERROR = "Failed to enqueue reindex processing"_MAX_FILE_VECTORIZATION_CONCURRENCY = 64+_USER_REINDEX_FIXED_ROOTS = ("resources", "skills", "memories", "peers")+_USER_REINDEX_EXCLUDED_ROOTS = frozenset({"sessions", "privacy", "settings"})+_MEMORY_NON_SOURCE_FILENAMES = ABSTRACT_OVERVIEW_FILENAMES | {+ ".relations.json",+ ".ovgitignore",+}# Trailing markers VikingFS appends when a directory has no generated .abstract.md/.overview.md@@ -83,16 +100,6 @@ def _is_not_ready_sentinel(text: str, suffix: str) -> bool:return head.startswith("#") and "\n" not in head-_reindex_executor: "ReindexExecutor | None" = None---def get_reindex_executor() -> "ReindexExecutor":- global _reindex_executor- if _reindex_executor is None:- _reindex_executor = ReindexExecutor()- return _reindex_executor--@dataclassclass _ReindexCounters:scanned_records: int = 0@@ -132,7 +139,9 @@ class _SourceRead:class ReindexExecutor:- """Non-destructive reindex orchestration for admin maintenance flows."""+ """Reindex convergence orchestration for admin maintenance flows."""++ _NAMESPACE_LS_PAGE_SIZE = 1000SUPPORTED_MODES_BY_TYPE = {"global_namespace": {"vectors_only", "semantic_and_vectors"},@@ -143,23 +152,6 @@ class ReindexExecutor:"memory": {"vectors_only", "semantic_and_vectors"},}- def __init__(- self,- vlm_resolver: VLMResolver | None = None,- vector_config_resolver=None,- ) -> None:- self.vlm_resolver = vlm_resolver- self.vector_config_resolver = vector_config_resolver-- async def _semantic_processor_for(self, ctx: RequestContext) -> SemanticProcessor:- if self.vlm_resolver is None:- raise RuntimeError("ReindexExecutor requires a VLM resolver for semantic reindexing")- vlm = await self.vlm_resolver.get_vlm(ctx.account_id)- return SemanticProcessor(- max_concurrent_llm=vlm.max_concurrent,- vlm_resolver=self.vlm_resolver,- )-@staticmethoddef _effective_file_vectorization_concurrency() -> int:config = get_openviking_config().reindex@@ -181,7 +173,21 @@ class ReindexExecutor:) -> None:for start in range(0, len(items), concurrency):batch = items[start : start + concurrency]- for item_counters in await asyncio.gather(*(processor(item) for item in batch)):+ tasks = [asyncio.create_task(processor(item)) for item in batch]+ batch_result = asyncio.gather(*tasks)+ try:+ results = await asyncio.shield(batch_result)+ except BaseException:+ for task in tasks:+ if not task.done():+ task.cancel()+ await run_to_completion(+ lambda batch_result=batch_result, tasks=tasks: asyncio.gather(+ batch_result, *tasks, return_exceptions=True+ )+ )+ raise+ for item_counters in results:counters.merge_from(item_counters)@staticmethod@@ -213,6 +219,7 @@ class ReindexExecutor:"""if not namespace_policy_for_uri(uri).supports_reindex:raise InvalidArgumentError(f"reindex is not supported for namespace: {uri}")+ self._ensure_reconciliation_scope_visible(uri, ctx)if task_id is not None and wait:raise InvalidArgumentError("task_id requires asynchronous reindex")object_type = self._infer_target_type(uri)@@ -449,6 +456,23 @@ class ReindexExecutor:details={"uri": uri},)+ @staticmethod+ def _ensure_reconciliation_scope_visible(uri: str, ctx: RequestContext) -> None:+ if not ctx.actor_peer_id:+ return+ # Reconciliation also probes F under the content owner's identity.+ scope_ctx = content_owner_context_for_uri(uri, ctx)+ if (+ uri.rstrip("/") in {"viking:", "viking://user"}+ or may_include_hidden_actor_peers(uri, scope_ctx)+ or is_hidden_by_actor_peer_view(uri, scope_ctx)+ ):+ raise PermissionDeniedError(+ "Reindex cannot reconcile a scope containing hidden actor peers; "+ "request an explicit visible content scope",+ resource=uri,+ )+async def _tree_all(self,viking_fs: Any,@@ -457,7 +481,7 @@ class ReindexExecutor:show_all_hidden: bool,ctx: RequestContext,) -> list[dict[str, Any]]:- return await viking_fs.tree(+ entries = await viking_fs.tree(uri,output="original",show_all_hidden=show_all_hidden,@@ -465,62 +489,350 @@ class ReindexExecutor:level_limit=None,ctx=ctx,)+ root_parts = uri_parts(uri)+ if any(+ entry.get("access") == "denied"+ or uri_parts(str(entry.get("uri") or ""))[: len(root_parts)] != root_parts+ for entry in entries+ ):+ raise ValueError("cannot reindex from an incomplete formal tree")+ return entries- async def _refresh_namespace_resource_semantics(+ async def _list_scope_children(self,+ uri: str,+ *,+ ctx: RequestContext,+ show_all_hidden: bool = False,+ ) -> list[dict[str, Any]]:+ entries, seen = [], set()+ offset = 0+ root_parts = uri_parts(uri)+ while True:+ page = await get_viking_fs().ls(+ uri,+ output="original",+ include_abstract=False,+ include_overview=False,+ show_all_hidden=show_all_hidden,+ node_limit=self._NAMESPACE_LS_PAGE_SIZE,+ offset=offset,+ ctx=ctx,+ )+ for entry in page:+ entry_uri = str(entry.get("uri") or "")+ parts = uri_parts(entry_uri)+ if entry.get("access") == "denied" or entry_uri in seen or parts[:-1] != root_parts:+ raise ValueError("cannot discover reindex roots from an incomplete listing")+ seen.add(entry_uri)+ entries.append(entry)+ if len(page) < self._NAMESPACE_LS_PAGE_SIZE:+ return entries+ offset += len(page)++ async def _namespace_facts(+ self,+ uri: str,*,- target_root: str,- directories: list[str],- files: list[str],+ ctx: RequestContext,+ list_children: bool = False,+ show_all_hidden: bool = False,+ ) -> tuple[bool, bool, list[dict[str, Any]]]:+ try:+ stat = await get_viking_fs().stat(uri, ctx=ctx, skip_count=True)+ except NotFoundError:+ return False, True, []+ if stat.get("access") == "denied":+ raise ValueError("cannot reindex an unreadable formal root")+ is_dir = bool(stat.get("isDir", stat.get("is_dir")))+ entries = []+ if is_dir and list_children:+ try:+ entries = await self._list_scope_children(+ uri,+ ctx=ctx,+ show_all_hidden=show_all_hidden,+ )+ except NotFoundError:+ exists, _, _ = await self._namespace_facts(uri, ctx=ctx)+ if exists:+ raise+ return False, True, []+ return True, is_dir, entries++ @staticmethod+ def _child_inventories(+ root: str,+ inventory: Mapping[str, Mapping[str, Any]],+ ) -> dict[str, dict[str, Mapping[str, Any]]]:+ root_parts = uri_parts(root)+ groups: dict[str, dict[str, Mapping[str, Any]]] = {}+ for record_id, record in inventory.items():+ parts = uri_parts(str(record.get("uri") or ""))+ if parts[: len(root_parts)] != root_parts:+ raise ValueError("reindex inventory is outside its namespace")+ if len(parts) > len(root_parts):+ child = root + "/" + parts[len(root_parts)]+ groups.setdefault(child, {})[record_id] = record+ return groups++ def _custom_user_scopes(+ self,+ user_uri: str,+ children: list[dict[str, Any]],+ direct_inventory: Mapping[str, Mapping[str, Any]],+ ) -> list[tuple[str, bool, bool, dict[str, Mapping[str, Any]]]]:+ children_by_uri = {str(entry["uri"]): entry for entry in children}+ groups = self._child_inventories(user_uri, direct_inventory)+ scopes = []+ for root in sorted(set(children_by_uri) | set(groups)):+ name = uri_parts(root)[-1]+ if (+ name in _USER_REINDEX_FIXED_ROOTS+ or name in _USER_REINDEX_EXCLUDED_ROOTS+ or is_storage_internal_name(name)+ or self._is_hidden_meta_file(root)+ ):+ continue+ entry = children_by_uri.get(root)+ records = groups.get(root, {})+ if entry is not None:+ is_dir = bool(entry.get("isDir", entry.get("is_dir")))+ else:+ levels = {int(record.get("level", -1)) for record in records.values()}+ if 2 in levels and levels.intersection({0, 1}):+ raise ValueError(f"custom user scope has conflicting vector levels: {root}")+ is_dir = bool(levels.intersection({0, 1}))+ scopes.append((root, entry is not None, is_dir, records))+ return scopes++ async def _resource_scope(+ self,+ *,+ uri: str,+ mode: str,run: _ReindexRunContext,- ) -> tuple[list[str], list[str]]:- counters = run.counters- ctx = run.ctx- prefix = self._child_prefix(target_root)- semantic_roots = sorted(- {- directory_uri- for directory_uri in directories- if directory_uri.startswith(prefix)- and "/" not in directory_uri[len(prefix) :]- and directory_uri[len(prefix) :]- }- )- filtered_directories = [- directory_uri- for directory_uri in directories- if any(- directory_uri == root or directory_uri.startswith(root + "/")- for root in semantic_roots+ facts: tuple[bool, bool, list[dict[str, Any]]],+ inventory: Mapping[str, Mapping[str, Any]],+ ) -> None:+ exists, is_dir, _ = facts+ if exists or inventory:+ await self._reindex_rfv(+ uri=uri,+ mode=mode,+ context_type="resource",+ recursive=True,+ run=run,+ root_is_dir=is_dir,+ root_exists=exists,+ formal_entries=None if exists else [],+ vector_inventory=inventory,)- ]- filtered_files = [- file_uri- for file_uri in files- if any(file_uri.startswith(root + "/") for root in semantic_roots)- ]- filtered_file_set = set(filtered_files)- for file_uri in files:- if file_uri in filtered_file_set:++ async def _skill_scopes(+ self,+ *,+ uri: str,+ mode: str,+ run: _ReindexRunContext,+ facts: tuple[bool, bool, list[dict[str, Any]]],+ inventory: Mapping[str, Mapping[str, Any]],+ ) -> None:+ exists, is_dir, children = facts+ if exists and not is_dir:+ raise InvalidArgumentError(f"skill namespace must be a directory: {uri}")+ children_by_uri = {str(entry["uri"]): entry for entry in children}+ groups = self._child_inventories(uri, inventory)+ roots = {+ key for key, entry in children_by_uri.items() if entry.get("isDir", entry.get("is_dir"))+ }+ roots.update(groups)+ for root in sorted(roots):+ if uri_parts(root)[-1].startswith("."):continue- counters.unsupported_records += 1- counters.warnings.append(- f"Skipped {file_uri}: namespace semantic_and_vectors only refreshes resource directories"+ child = children_by_uri.get(root)+ if child is not None and not child.get("isDir", child.get("is_dir")):+ raise InvalidArgumentError(f"skill package must be a directory: {root}")+ await self._reindex_rfv(+ uri=root,+ mode=mode,+ context_type="skill",+ recursive=True,+ run=run,+ root_is_dir=True,+ root_exists=child is not None,+ formal_entries=None if child is not None else [],+ vector_inventory=groups.get(root, {}),)- for semantic_root in semantic_roots:- await self._run_semantic_processor(- uri=semantic_root,- context_type="resource",- ctx=ctx,- lock=run.lock,++ async def _reindex_peer_scopes(+ self,+ *,+ uri: str,+ mode: str,+ run: _ReindexRunContext,+ inventory: Mapping[str, Mapping[str, Any]],+ facts: tuple[bool, bool, list[dict[str, Any]]] | None = None,+ ) -> None:+ self._ensure_reconciliation_scope_visible(uri, run.ctx)+ if facts is None:+ facts = await self._namespace_facts(uri, ctx=run.ctx, list_children=True)+ exists, is_dir, children = facts+ if exists and not is_dir:+ raise InvalidArgumentError(f"peer namespace must be a directory: {uri}")+ peers = {str(entry["uri"]): entry for entry in children}+ groups = self._child_inventories(uri, inventory)+ roots = {key for key, entry in peers.items() if entry.get("isDir", entry.get("is_dir"))}+ roots.update(groups)+ for peer in sorted(roots):+ if uri_parts(peer)[-1].startswith("."):+ continue+ child = peers.get(peer)+ if child is not None and not child.get("isDir", child.get("is_dir")):+ raise InvalidArgumentError(f"peer root must be a directory: {peer}")+ peer_facts = (+ await self._namespace_facts(peer, ctx=run.ctx, list_children=True)+ if child is not None+ else (False, True, []))- return filtered_directories, filtered_files+ content = {str(entry["uri"]): entry for entry in peer_facts[2]}+ content_groups = self._child_inventories(peer, groups.get(peer, {}))+ resources, memories = peer + "/resources", peer + "/memories"+ resource_entry = content.get(resources)+ resource_facts = (+ resource_entry is not None,+ bool((resource_entry or {}).get("isDir", True)),+ [],+ )+ await self._resource_scope(+ uri=resources,+ mode=mode,+ run=run,+ facts=resource_facts,+ inventory=content_groups.get(resources, {}),+ )+ memory_entry = content.get(memories)+ memory_facts = (+ await self._namespace_facts(+ memories,+ ctx=run.ctx,+ list_children=True,+ show_all_hidden=True,+ )+ if memory_entry is not None+ else (False, True, [])+ )+ if memory_entry is not None or content_groups.get(memories):+ await self._reindex_memory(+ uri=memories,+ mode=mode,+ run=run,+ root_exists=memory_facts[0],+ root_is_dir=memory_facts[1],+ formal_entries=memory_facts[2],+ vector_inventory=content_groups.get(memories, {}),+ )- @staticmethod- def _child_prefix(root: str) -> str:- if root.rstrip("/") == "viking:":- return "viking://"- return root.rstrip("/") + "/"+ async def _memory_scopes(+ self,+ *,+ uri: str,+ run: _ReindexRunContext,+ children: list[dict[str, Any]],+ inventory: Mapping[str, Mapping[str, Any]],+ reconcile_only: bool = False,+ ) -> None:+ by_uri = {+ str(entry["uri"]): entry+ for entry in children+ if entry.get("isDir", entry.get("is_dir"))+ or self._is_memory_source_uri(str(entry["uri"]))+ }+ root_parts = uri_parts(uri)+ directory_roots = {+ key for key, entry in by_uri.items() if entry.get("isDir", entry.get("is_dir"))+ }+ real_files = {+ key for key, entry in by_uri.items() if not entry.get("isDir", entry.get("is_dir"))+ }++ def scope_root(record):+ record_uri = str(record.get("uri") or "")+ if int(record.get("level", -1)) == 2 and record_uri not in real_files:+ match = _MEMORY_CHUNK_PATTERN.match(record_uri)+ if match:+ record_uri = match.group("base")+ parts = uri_parts(record_uri)+ if parts[: len(root_parts)] != root_parts:+ raise ValueError("memory inventory is outside its namespace")+ suffix = parts[len(root_parts) :]+ return (uri + "/" + suffix[0] if suffix else None), len(suffix)++ for record in inventory.values():+ root, depth = scope_root(record)+ if (+ root+ and (depth > 1 or int(record.get("level", -1)) in {0, 1})+ ):+ directory_roots.add(root)+ groups: dict[str, dict[str, Mapping[str, Any]]] = {}+ root_inventory = {}+ for record_id, record in inventory.items():+ root, _ = scope_root(record)+ if root in directory_roots:+ groups.setdefault(root, {})[record_id] = record+ else:+ root_inventory[record_id] = record+ root_files = [+ entry+ for key, entry in by_uri.items()+ if key not in directory_roots and not entry.get("isDir", entry.get("is_dir"))+ ]+ await self._reindex_memory_vectors(+ **self._with_ingest_options(+ {+ "uri": uri,+ "ctx": run.ctx,+ "counters": run.counters,+ "formal_entries": root_files,+ "root_is_dir": True,+ "vector_inventory": root_inventory,+ "lease_ref": run.lease,+ "reconcile_only": reconcile_only,+ },+ run.ingest_options,+ )+ )+ for root in sorted(directory_roots):+ entry = by_uri.get(root)+ root_exists = entry is not None+ is_dir = bool((entry or {}).get("isDir", True))+ entries = []+ if root_exists and is_dir:+ try:+ entries = await self._tree_all(+ get_viking_fs(), root, show_all_hidden=True, ctx=run.ctx+ )+ except NotFoundError:+ root_exists, _, _ = await self._namespace_facts(root, ctx=run.ctx)+ if root_exists:+ raise+ await self._reindex_memory_vectors(+ **self._with_ingest_options(+ {+ "uri": root,+ "ctx": run.ctx,+ "counters": run.counters,+ "formal_entries": entries,+ "root_is_dir": is_dir,+ "root_exists": root_exists,+ "vector_inventory": groups.get(root, {}),+ "lease_ref": run.lease,+ "reconcile_only": reconcile_only,+ },+ run.ingest_options,+ )+ )@staticmethoddef _apply_embedding_wait_status(@@ -547,60 +859,32 @@ class ReindexExecutor:counters.failed_records += len(failures)counters.warnings.extend(failures)- def _is_resource_entry_for_namespace(self, uri: str, target_root: str) -> bool:- if not uri.startswith(self._child_prefix(target_root)):- return False- classification = classify_uri(uri)- if classification.is_memory or classification.is_skill:- return False- return True-- def _is_global_resource_entry(self, uri: str) -> bool:- return uri == "viking://resources" or uri.startswith("viking://resources/")-async def _reindex_skill_namespace(self,*,uri: str,mode: str,run: _ReindexRunContext,- recursive: bool = True,+ vector_inventory: Mapping[str, Mapping[str, Any]] | None = None,+ facts: tuple[bool, bool, list[dict[str, Any]]] | None = None,) -> None:- counters = run.counters- ctx = run.ctx- if not recursive:- return- viking_fs = get_viking_fs()- try:- entries = await self._tree_all(viking_fs, uri, show_all_hidden=True, ctx=ctx)- except Exception as exc:- raise NotFoundError(uri, "resource") from exc-- skill_roots = []- for entry in entries:- entry_uri = entry.get("uri")- if entry_uri and entry.get("isDir") and classify_uri(entry_uri).is_skill_root:- skill_roots.append(entry_uri)-- for skill_root in sorted(set(skill_roots)):- subtree = [- entry- for entry in entries- if str(entry.get("uri") or "").startswith(skill_root.rstrip("/") + "/")- ]- await self._reindex_rfv(- uri=skill_root,- mode=mode,- context_type="skill",+ self._ensure_reconciliation_scope_visible(uri, run.ctx)+ if vector_inventory is None:+ vector_inventory = await self._read_reindex_inventory(+ get_service().vikingdb_manager,+ uri=uri,+ ctx=self._content_owner_ctx(uri, run.ctx),recursive=True,- run=run,- root_is_dir=True,- formal_entries=subtree,)-- if not skill_roots:- counters.unsupported_records += 1- counters.warnings.append(f"No skill roots found under {uri}")+ if facts is None:+ facts = await self._namespace_facts(uri, ctx=run.ctx, list_children=True)+ await self._skill_scopes(+ uri=uri,+ mode=mode,+ run=run,+ facts=facts,+ inventory=vector_inventory,+ )def _validate_mode(self, object_type: str, mode: str) -> None:supported_modes = self.SUPPORTED_MODES_BY_TYPE[object_type]@@ -625,31 +909,34 @@ class ReindexExecutor:recursive: bool = True,ingest_options: IngestOptions | None = None,ctx: RequestContext,- existing_lease: dict | None = None,+ lease_guard: PathLockGuard | None = None,) -> dict[str, Any]:- service = get_service()- if service.viking_fs is None or service.vikingdb_manager is None:- raise RuntimeError("OpenVikingService not initialized")- if not service.vikingdb_manager.has_queue_manager:- raise OpenVikingError(- "Reindex requires embedding queue",- code="FAILED_PRECONDITION",- details={"uri": uri},- )-- path = service.viking_fs._uri_to_path(uri, ctx=ctx)- started_at = time.perf_counter()- counters = _ReindexCounters()- telemetry_id = get_current_telemetry().telemetry_id- acquire_lock = service.viking_fs._async_agfs.pathlock_acquire_tree- stat = await service.viking_fs.stat(uri, ctx=ctx, skip_count=True)- root_is_dir = bool(stat.get("isDir", stat.get("is_dir")))- if not root_is_dir:- acquire_lock = service.viking_fs._async_agfs.pathlock_acquire_exact- lease = existing_lease or await acquire_lock(path)- lease_guard = PathLockGuard(service.viking_fs._async_agfs, lease)- run: _ReindexRunContext | None = Nonetry:+ self._ensure_reconciliation_scope_visible(uri, ctx)+ service = get_service()+ if service.viking_fs is None or service.vikingdb_manager is None:+ raise RuntimeError("OpenVikingService not initialized")+ if not service.vikingdb_manager.has_queue_manager:+ raise OpenVikingError(+ "Reindex requires embedding queue",+ code="FAILED_PRECONDITION",+ details={"uri": uri},+ )+ if lease_guard is None:+ lease_guard = PathLockGuard(service.viking_fs._async_agfs)++ path = service.viking_fs._uri_to_path(uri, ctx=ctx)+ started_at = time.perf_counter()+ counters = _ReindexCounters()+ telemetry_id = get_current_telemetry().telemetry_id+ acquire_lock = service.viking_fs._async_agfs.pathlock_acquire_tree+ stat = await service.viking_fs.stat(uri, ctx=ctx, skip_count=True)+ root_is_dir = bool(stat.get("isDir", stat.get("is_dir")))+ if not root_is_dir:+ acquire_lock = service.viking_fs._async_agfs.pathlock_acquire_exact+ lease = lease_guard.lease+ if lease is None:+ lease = await lease_guard.acquire(lambda: acquire_lock(path))borrowed = (await service.viking_fs._async_agfs.pathlock_as_borrowed(lease)if lease is not None@@ -717,7 +1004,8 @@ class ReindexExecutor:await get_task_tracker().build_queue_status(task_context.task_id),)finally:- await lease_guard.close()+ if lease_guard is not None:+ await lease_guard.close()return {"status": "completed",@@ -759,6 +1047,8 @@ class ReindexExecutor:run: _ReindexRunContext,root_is_dir: bool | None = None,formal_entries: list[dict[str, Any]] | None = None,+ root_exists: bool = True,+ vector_inventory: Mapping[str, Mapping[str, Any]] | None = None,) -> None:from openviking.storage.context_update_plan import build_rfv_context_update_planfrom openviking.storage.resource_rfv import build_rfv_snapshot@@ -778,6 +1068,18 @@ class ReindexExecutor:force=run.force or (context_type == "skill" and mode == "vectors_only"),scalar_target_levels=frozenset({0, 1, 2}),)+ resolved_root_is_dir = run.root_is_dir if root_is_dir is None else root_is_dir+ if not recursive and resolved_root_is_dir is None:+ stat = await service.viking_fs.stat(uri, ctx=run.ctx, skip_count=True)+ resolved_root_is_dir = bool(stat.get("isDir", stat.get("is_dir")))+ if vector_inventory is None:+ vector_inventory = await self._read_reindex_inventory(+ service.vikingdb_manager,+ uri=uri,+ ctx=self._content_owner_ctx(uri, run.ctx),+ recursive=recursive,+ include_direct_children=bool(resolved_root_is_dir) and not recursive,+ )snapshot = await build_rfv_snapshot(viking_fs=service.viking_fs,vikingdb=service.vikingdb_manager,@@ -786,12 +1088,14 @@ class ReindexExecutor:request_intent=request,recursive=recursive,source_read_concurrency=self._effective_file_vectorization_concurrency(),- root_is_dir=run.root_is_dir if root_is_dir is None else root_is_dir,+ root_is_dir=resolved_root_is_dir,formal_inventory=((True if root_is_dir is None else root_is_dir, formal_entries, True)if formal_entries is not Noneelse None),+ root_exists=root_exists,+ vector_inventory=vector_inventory,)diff, plan = build_rfv_context_update_plan(snapshot=snapshot,@@ -799,9 +1103,11 @@ class ReindexExecutor:account_id=run.ctx.account_id,source_metadata=snapshot.source_metadata,)+ root_entry = snapshot.formal.entries.get("")if (mode == "vectors_only"- and not snapshot.formal.entries[""].is_dir+ and root_entry is not None+ and not root_entry.is_dirand any(action.level == int(ContextLevel.DETAIL)and action.action.value in {"merge", "upsert"}@@ -898,8 +1204,6 @@ class ReindexExecutor:run: _ReindexRunContext,) -> None:"""Persist one fully planned reindex DAG without persisting its source bytes."""- from openviking.storage.queuefs import get_queue_manager-owner_ctx = self._content_owner_ctx(semantic_plan.root_uri, run.ctx)telemetry_id = get_current_telemetry().telemetry_idmsg = SemanticMsg(@@ -916,30 +1220,44 @@ class ReindexExecutor:propagate_to_parent=recursive,plan=semantic_plan,)+ root_entry = next(entry for entry in semantic_plan.tree.entries if not entry.relative_path)+ await self._enqueue_reindex_semantic_message(+ msg=msg,+ root_is_dir=root_entry.kind == "directory",+ run=run,+ )++ async def _enqueue_reindex_semantic_message(+ self,+ *,+ msg: SemanticMsg,+ root_is_dir: bool,+ run: _ReindexRunContext,+ ) -> None:+ """Queue shared semantic/vector work with the current reindex lock."""+ from openviking.storage.queuefs import get_queue_manager+queue_manager = get_queue_manager()semantic_queue = queue_manager.get_queue(queue_manager.SEMANTIC, allow_create=True)++ async def enqueue():+ if msg.context_type == "memory":+ return await semantic_queue.enqueue(msg, deduplicate_memory=False)+ return await semantic_queue.enqueue(msg)+if run.lease is None:- await semantic_queue.enqueue(msg)+ await enqueue()returnviking_fs = get_service().viking_fsagfs = viking_fs._async_agfs- use_outer_lease = run.lock_root_uri in {None, semantic_plan.root_uri.rstrip("/")}+ use_outer_lease = run.lock_root_uri in {None, msg.uri.rstrip("/")}plan_lease = run.leaseacquired_plan_lease = Falseif not use_outer_lease:- root_entry = next(- entry for entry in semantic_plan.tree.entries if not entry.relative_path- )- acquire = (- agfs.pathlock_acquire_tree- if root_entry.kind == "directory"- else agfs.pathlock_acquire_exact- )+ acquire = agfs.pathlock_acquire_tree if root_is_dir else agfs.pathlock_acquire_exactkwargs = {"owner_lease_ref": run.lease} if run.lease is not None else {}- plan_lease = await acquire(- viking_fs._uri_to_path(semantic_plan.root_uri, ctx=owner_ctx), **kwargs- )+ plan_lease = await acquire(viking_fs._uri_to_path(msg.uri, ctx=run.ctx), **kwargs)acquired_plan_lease = Trueassert plan_lease is not Noneplan_guard = (@@ -950,17 +1268,7 @@ class ReindexExecutor:handoff = await plan_guard.to_handoff()msg.lock_handoff = handofftry:- enqueue_id = await plan_guard.handoff_and_dispatch(- handoff,- lambda: semantic_queue.enqueue(msg),- accepted=lambda result: result != "deduplicated",- )- if enqueue_id == "deduplicated":- await plan_guard.close()- if not acquired_plan_lease:- run.lease = None- run.lock = None- return+ await plan_guard.handoff_and_dispatch(handoff, enqueue)if not acquired_plan_lease:run.lease = Nonerun.lock = None@@ -1000,68 +1308,79 @@ class ReindexExecutor:mode: str,run: _ReindexRunContext,recursive: bool = True,+ vector_inventory: Mapping[str, Mapping[str, Any]] | None = None,+ root_exists: bool = True,+ root_is_dir: bool | None = None,+ formal_entries: list[dict[str, Any]] | None = None,) -> None:- counters = run.counters- ctx = run.ctx- if mode == "semantic_and_vectors":- stat = await get_viking_fs().stat(uri, ctx=ctx)- if stat.get("isDir", stat.get("is_dir")):- semantic_kwargs = {- "uri": uri,- "context_type": "memory",- "ctx": ctx,- "lock": run.lock,- }- if not recursive:- semantic_kwargs["recursive"] = False- await self._run_semantic_processor(**semantic_kwargs)- vector_kwargs = {"uri": uri, "counters": counters, "ctx": ctx}- if not recursive:- vector_kwargs["recursive"] = False+ self._ensure_reconciliation_scope_visible(uri, run.ctx)+ if root_exists and root_is_dir is None:+ stat = await get_viking_fs().stat(uri, ctx=run.ctx, skip_count=True)+ root_is_dir = bool(stat.get("isDir", stat.get("is_dir")))+ if vector_inventory is None:+ vector_inventory = await self._read_reindex_inventory(+ get_service().vikingdb_manager,+ uri=uri,+ ctx=self._content_owner_ctx(uri, run.ctx),+ recursive=(bool(root_is_dir) and recursive) or not root_exists,+ include_virtual_chunks=root_exists and not root_is_dir,+ )+ shard_root = root_exists and root_is_dir and recursive and classify_uri(uri).is_memory_root+ children = formal_entries+ if shard_root and children is None:+ children = await self._list_scope_children(+ uri,+ ctx=run.ctx,+ show_all_hidden=True,+ )+ queue_semantics = mode == "semantic_and_vectors" and root_exists and root_is_dir+ if shard_root:+ await self._memory_scopes(+ uri=uri,+ run=run,+ children=children,+ inventory=vector_inventory,+ reconcile_only=queue_semantics,+ )+ else:await self._reindex_memory_vectors(**self._with_ingest_options(- vector_kwargs,+ {+ "uri": uri,+ "counters": run.counters,+ "ctx": run.ctx,+ "recursive": recursive,+ "vector_inventory": vector_inventory,+ "root_exists": root_exists,+ "root_is_dir": root_is_dir,+ "formal_entries": formal_entries,+ "lease_ref": run.lease,+ "reconcile_only": queue_semantics,+ },run.ingest_options,))- return- await self._reindex_memory_vectors(- **self._with_ingest_options(- {- "uri": uri,- "counters": counters,- "ctx": ctx,- "recursive": recursive,- },- run.ingest_options,+ if queue_semantics:+ owner_ctx = self._content_owner_ctx(uri, run.ctx)+ await self._enqueue_reindex_semantic_message(+ msg=SemanticMsg(+ uri=uri,+ context_type="memory",+ recursive=recursive,+ account_id=owner_ctx.account_id,+ user_id=owner_ctx.user.user_id,+ group_ids=list(owner_ctx.group_ids),+ peer_id=owner_ctx.user.user_id,+ role=str(owner_ctx.role),+ telemetry_id=get_current_telemetry().telemetry_id,+ ingest_options=run.ingest_options,+ generation_trigger="reindex",+ use_hierarchical_aggregation=True,+ propagate_to_parent=recursive,+ ),+ root_is_dir=True,+ run=run,)- )-- async def _run_semantic_processor(- self,- *,- uri: str,- context_type: str,- ctx: RequestContext,- lock: dict | None = None,- recursive: bool = True,- ) -> None:- owner_ctx = self._content_owner_ctx(uri, ctx)- processor = await self._semantic_processor_for(owner_ctx)- msg = SemanticMsg(- uri=uri,- context_type=context_type,- recursive=recursive,- account_id=owner_ctx.account_id,- user_id=owner_ctx.user.user_id,- peer_id=owner_ctx.user.user_id,- role=str(ctx.role),- skip_vectorization=True,- generation_trigger="reindex",- use_hierarchical_aggregation=context_type == "memory",- propagate_to_parent=recursive,- )- await processor.on_dequeue({"data": msg.to_json()}, lock=lock)async def reindex_directory_marker(self, *, dir_uri: str, level: ContextLevel, ctx: RequestContext@@ -1138,124 +1457,104 @@ class ReindexExecutor:run: _ReindexRunContext,recursive: bool = True,) -> None:- ctx = run.ctx- normalized_uri = uri.rstrip("/")- target_root = normalized_uri if normalized_uri else uri+ self._ensure_reconciliation_scope_visible(uri, run.ctx)if not recursive:return- viking_fs = get_viking_fs()- try:- entries = await self._tree_all(viking_fs, target_root, show_all_hidden=True, ctx=ctx)- except Exception as exc:- raise NotFoundError(uri, "resource") from exc-- if target_root == "viking://user":- user_roots = [- entry.get("uri")- for entry in entries- if entry.get("isDir") and classify_uri(entry.get("uri", "")).is_user_namespace_root- ]- if user_roots:- for user_root in sorted(set(user_roots)):- await self._reindex_user_namespace(- uri=user_root,- mode=mode,- run=run,- recursive=True,- )- return-- memory_roots: list[str] = []- skill_roots: list[str] = []- resource_directories: list[str] = []- resource_files: list[str] = []-- for entry in entries:- entry_uri = entry.get("uri")- if not entry_uri:- continue- if is_session_uri(entry_uri):- continue- classification = classify_uri(entry_uri)- if classification.is_memory:- if entry.get("isDir") and classification.is_memory_root:- memory_roots.append(entry_uri)- continue- if classification.is_skill:- if entry.get("isDir") and classification.is_skill_root:- skill_roots.append(entry_uri)- continue- if not self._is_resource_entry_for_namespace(entry_uri, target_root):- continue- if entry.get("isDir"):- resource_directories.append(entry_uri)- elif not self._is_hidden_meta_file(entry_uri):- resource_files.append(entry_uri)-- for memory_root in sorted(set(memory_roots)):- memory_mode = (- "semantic_and_vectors" if mode == "semantic_and_vectors" else "vectors_only"- )- await self._reindex_memory(- uri=memory_root,- mode=memory_mode,- run=run,- )-- for skill_root in sorted(set(skill_roots)):- skill_mode = (- "semantic_and_vectors" if mode == "semantic_and_vectors" else "vectors_only"- )- await self._reindex_skill(- uri=skill_root,- mode=skill_mode,- run=run,+ uri = uri.rstrip("/")+ if uri == "viking://user":+ children = await self._list_scope_children(uri, ctx=run.ctx)+ for entry in children:+ child = str(entry["uri"])+ if (+ entry.get("isDir", entry.get("is_dir"))+ and classify_uri(child).is_user_namespace_root+ ):+ await self._reindex_user_namespace(uri=child, mode=mode, run=run)+ return+ children = await self._list_scope_children(uri, ctx=run.ctx, show_all_hidden=True)+ roots = {name: uri + "/" + name for name in _USER_REINDEX_FIXED_ROOTS}+ vikingdb = get_service().vikingdb_manager+ inventories = {}+ for name, root in roots.items():+ inventories[name] = await self._read_reindex_inventory(+ vikingdb,+ uri=root,+ ctx=self._content_owner_ctx(root, run.ctx),recursive=True,- formal_entries=[- entry- for entry in entries- if str(entry.get("uri") or "").startswith(skill_root.rstrip("/") + "/")- ],)-- resource_roots = [- directory_uri- for directory_uri in resource_directories- if directory_uri.rstrip("/").endswith("/resources")- ]- covered_prefixes = tuple(root.rstrip("/") + "/" for root in resource_roots)- for resource_root in resource_roots:- await self._reindex_rfv(- uri=resource_root,+ direct_inventory = await self._read_reindex_inventory(+ vikingdb,+ uri=uri,+ ctx=self._content_owner_ctx(uri, run.ctx),+ recursive=False,+ include_direct_children=True,+ )+ custom_scopes = []+ for root, root_exists, root_is_dir, inventory in self._custom_user_scopes(+ uri, children, direct_inventory+ ):+ if root_is_dir:+ inventory = await self._read_reindex_inventory(+ vikingdb,+ uri=root,+ ctx=self._content_owner_ctx(root, run.ctx),+ recursive=True,+ )+ custom_scopes.append((root, root_exists, root_is_dir, inventory))+ facts = {}+ for name, root in roots.items():+ facts[name] = await self._namespace_facts(+ root,+ ctx=run.ctx,+ list_children=name != "resources",+ show_all_hidden=name == "memories",+ )+ for name in ("skills", "peers"):+ if facts[name][0] and not facts[name][1]:+ raise InvalidArgumentError(f"{name} namespace must be a directory: {roots[name]}")+ await self._resource_scope(+ uri=roots["resources"],+ mode=mode,+ run=run,+ facts=facts["resources"],+ inventory=inventories["resources"],+ )+ await self._reindex_skill_namespace(+ uri=roots["skills"],+ mode=mode,+ run=run,+ facts=facts["skills"],+ vector_inventory=inventories["skills"],+ )+ memory_facts = facts["memories"]+ if memory_facts[0] or inventories["memories"]:+ await self._reindex_memory(+ uri=roots["memories"],mode=mode,- context_type="resource",- recursive=True,run=run,- root_is_dir=True,- formal_entries=[- entry- for entry in entries- if str(entry.get("uri") or "").startswith(resource_root.rstrip("/") + "/")- ],+ root_exists=memory_facts[0],+ root_is_dir=memory_facts[1],+ formal_entries=memory_facts[2],+ vector_inventory=inventories["memories"],)- for file_uri in resource_files:- if any(file_uri.startswith(prefix) for prefix in covered_prefixes):- continue- if mode == "semantic_and_vectors":- run.counters.unsupported_records += 1- run.counters.warnings.append(- f"Skipped {file_uri}: namespace semantic_and_vectors only refreshes "- "resource directories"- )- continue+ await self._reindex_peer_scopes(+ uri=roots["peers"],+ mode=mode,+ run=run,+ facts=facts["peers"],+ inventory=inventories["peers"],+ )+ for root, root_exists, root_is_dir, inventory in custom_scopes:await self._reindex_rfv(- uri=file_uri,+ uri=root,mode=mode,context_type="resource",- recursive=False,+ recursive=root_is_dir,run=run,- root_is_dir=False,- formal_entries=[],+ root_is_dir=root_is_dir,+ root_exists=root_exists,+ formal_entries=None if root_exists else [],+ vector_inventory=inventory,)async def _reindex_global_namespace(@@ -1266,80 +1565,46 @@ class ReindexExecutor:run: _ReindexRunContext,recursive: bool = True,) -> None:- ctx = run.ctx- target_root = "viking://"+ self._ensure_reconciliation_scope_visible(uri, run.ctx)if not recursive:return- viking_fs = get_viking_fs()- try:- entries = await self._tree_all(viking_fs, target_root, show_all_hidden=True, ctx=ctx)- except Exception as exc:- raise NotFoundError(uri, "resource") from exc-- user_roots: list[str] = []- resource_directories: list[str] = []- resource_files: list[str] = []-- for entry in entries:- entry_uri = entry.get("uri")- if not entry_uri:- continue- if entry_uri == "viking://user":- continue- if entry_uri.startswith("viking://user/"):- remainder = entry_uri[len("viking://user/") :]- if entry.get("isDir") and remainder and "/" not in remainder:- user_roots.append(entry_uri)- continue- if is_session_uri(entry_uri):- continue- if not self._is_global_resource_entry(entry_uri):- continue- if entry.get("isDir"):- resource_directories.append(entry_uri)- elif not self._is_hidden_meta_file(entry_uri):- resource_files.append(entry_uri)-- for user_root in sorted(set(user_roots)):- await self._reindex_user_namespace(- uri=user_root,- mode=mode,- run=run,- )-- if "viking://resources" in resource_directories:- await self._reindex_rfv(- uri="viking://resources",- mode=mode,- context_type="resource",+ resources, skills = "viking://resources", "viking://agent/skills"+ inventories = {}+ for root in (resources, skills):+ inventories[root] = await self._read_reindex_inventory(+ get_service().vikingdb_manager,+ uri=root,+ ctx=run.ctx,recursive=True,- run=run,- root_is_dir=True,- formal_entries=[- entry- for entry in entries- if str(entry.get("uri") or "").startswith("viking://resources/")- ],- )- for file_uri in resource_files:- if file_uri.startswith("viking://resources/"):- continue- if mode == "semantic_and_vectors":- run.counters.unsupported_records += 1- run.counters.warnings.append(- f"Skipped {file_uri}: namespace semantic_and_vectors only refreshes "- "resource directories"- )- continue- await self._reindex_rfv(- uri=file_uri,- mode=mode,- context_type="resource",- recursive=False,- run=run,- root_is_dir=False,- formal_entries=[],)+ resource_facts = await self._namespace_facts(resources, ctx=run.ctx)+ skill_facts = await self._namespace_facts(skills, ctx=run.ctx, list_children=True)+ _, _, users = await self._namespace_facts(+ "viking://user",+ ctx=run.ctx,+ list_children=True,+ )+ await self._resource_scope(+ uri=resources,+ mode=mode,+ run=run,+ facts=resource_facts,+ inventory=inventories[resources],+ )+ await self._reindex_skill_namespace(+ uri=skills,+ mode=mode,+ run=run,+ facts=skill_facts,+ vector_inventory=inventories[skills],+ )+ for entry in users:+ child = str(entry["uri"])+ if (+ entry.get("isDir", entry.get("is_dir"))+ and classify_uri(child).is_user_namespace_root+ ):+ await self._reindex_user_namespace(uri=child, mode=mode, run=run)async def _reindex_memory_vectors(self,@@ -1349,89 +1614,174 @@ class ReindexExecutor:ctx: RequestContext,recursive: bool = True,ingest_options: IngestOptions | None = None,+ vector_inventory: Optional[Mapping[str, Mapping[str, Any]]] = None,+ formal_entries: list[dict[str, Any]] | None = None,+ root_is_dir: bool | None = None,+ root_exists: bool = True,+ lease_ref: dict[str, Any] | None = None,+ reconcile_only: bool = False,) -> None:viking_fs = get_viking_fs()- if await viking_fs.exists(uri, ctx=ctx):- stat = await viking_fs.stat(uri, ctx=ctx, skip_count=True)- if stat.get("isDir", stat.get("is_dir")):- entries = (- await self._tree_all(viking_fs, uri, show_all_hidden=False, ctx=ctx)- if recursive- else []+ if not root_exists:+ service = get_service()+ inventory = vector_inventory+ if inventory is None:+ inventory = await self._read_reindex_inventory(+ service.vikingdb_manager,+ uri=uri,+ ctx=self._content_owner_ctx(uri, ctx),+ recursive=True,)- directory_uris = {uri}- for entry in entries:- entry_uri = entry.get("uri")- if entry_uri and entry.get("isDir"):- directory_uris.add(entry_uri)- await self._reindex_memory_directory_chain(- **self._with_ingest_options(- {- "directory_uris": sorted(directory_uris),- "counters": counters,- "ctx": ctx,- },- ingest_options,- )+ root_parts = uri_parts(uri)+ ids = []+ for record_id, record in inventory.items():+ record_uri = str(record.get("uri") or "")+ if uri_parts(record_uri)[: len(root_parts)] != root_parts:+ raise ValueError("memory inventory is outside the missing root")+ if classify_uri(record_uri).is_memory:+ ids.append(str(record_id))+ if ids:+ counters.deleted_records += await service.vikingdb_manager.strict_delete(+ ids,+ ctx=self._content_owner_ctx(uri, ctx),)- file_uris = [entry["uri"] for entry in entries if not entry.get("isDir")]- else:- file_uris = [uri]+ return+ directory_uris: set[str] = set()+ if formal_entries is not None:+ root_parts = uri_parts(uri)+ if any(+ entry.get("access") == "denied"+ or uri_parts(str(entry.get("uri") or ""))[: len(root_parts)] != root_parts+ for entry in formal_entries+ ):+ raise ValueError("cannot reindex from an incomplete formal tree")+ root_is_dir = True if root_is_dir is None else root_is_dir+ entries = formal_entries+ elif await viking_fs.exists(uri, ctx=ctx):+ stat = await viking_fs.stat(uri, ctx=ctx, skip_count=True)+ root_is_dir = bool(stat.get("isDir", stat.get("is_dir")))+ entries = (+ await self._tree_all(viking_fs, uri, show_all_hidden=True, ctx=ctx)+ if root_is_dir and recursive+ else []+ )else:raise NotFoundError(uri, "memory")+ if root_is_dir:+ directory_uris = {uri}+ for entry in entries:+ entry_uri = entry.get("uri")+ if entry_uri and entry.get("isDir", entry.get("is_dir")):+ directory_uris.add(entry_uri)+ file_uris = [+ entry["uri"]+ for entry in entries+ if not entry.get("isDir", entry.get("is_dir"))+ and self._is_memory_source_uri(str(entry["uri"]))+ ]+ else:+ file_uris = [uri]- async def process_file(file_uri: str) -> _ReindexCounters:- file_counters = _ReindexCounters(scanned_records=1)- body_source = await self._read_memory_body(file_uri, ctx=ctx)- if body_source.error:- file_counters.failed_records += 1- file_counters.warnings.append(- f"Skipped {file_uri}: failed to read memory body: {body_source.error}"- )- return file_counters- body = body_source.text if body_source.exists else ""- memory_content = MemoryFileUtils.read(body).content if body else ""- existing = await self._fetch_existing_record(- uri=file_uri,- level=2,- ctx=self._content_owner_ctx(file_uri, ctx),- )- abstract = self._best_non_empty(- self._record_abstract(existing),- await self._best_file_summary(file_uri, ctx=ctx),+ # Read the complete V inventory before mutating any directory or file so a+ # rate-limited or failed vector read never leaves partial directory writes.+ if vector_inventory is None:+ service = get_service()+ assert service.vikingdb_manager is not None+ inventory = await self._read_reindex_inventory(+ service.vikingdb_manager,+ uri=uri,+ ctx=self._content_owner_ctx(uri, ctx),+ recursive=root_is_dir and recursive,+ include_virtual_chunks=not root_is_dir,)- if not body and existing is None:- file_counters.unsupported_records += 1- file_counters.warnings.append(f"No memory source found for {file_uri}")- return file_counters-- parent_uri = VikingURI(file_uri.split("#", 1)[0]).parent.uri- if body:- detail_abstract = self._prefer_non_empty(abstract, memory_content, body)+ else:+ inventory = dict(vector_inventory)++ if not root_is_dir:+ for chunk_uri in sorted(+ {str(record.get("uri") or "") for record in inventory.values()}+ ):+ if not re.fullmatch(re.escape(uri) + r"#chunk_[0-9]{4,}", chunk_uri):+ continuetry:- await self._upsert_context(- uri=file_uri,- parent_uri=parent_uri,- abstract=detail_abstract,- vector_text=body,- is_leaf=True,- context_type=ContextType.MEMORY.value,- level=ContextLevel.DETAIL,- ctx=ctx,- ingest_options=ingest_options,+ chunk_stat = await viking_fs.stat(chunk_uri, ctx=ctx, skip_count=True)+ except NotFoundError:+ continue+ if chunk_stat.get("access") == "denied":+ raise PermissionDeniedError(f"Cannot confirm memory chunk source: {chunk_uri}")+ if not chunk_stat.get("isDir", chunk_stat.get("is_dir")):+ file_uris.append(chunk_uri)++ directory_sources: dict[tuple[str, int], _SourceRead] = {}+ if root_is_dir and directory_uris:+ if not reconcile_only:+ directory_sources = (+ await self._reindex_memory_directory_chain(+ **self._with_ingest_options(+ {+ "directory_uris": sorted(directory_uris),+ "counters": counters,+ "ctx": ctx,+ },+ ingest_options,+ ))- file_counters.rebuilt_records += 1- except Exception as exc:- file_counters.failed_records += 1- file_counters.warnings.append(f"Failed to reindex {file_uri} vector: {exc}")- return file_counters+ or {}+ )+ else:+ for directory_uri in directory_uris:+ for level, filename in ((0, ".abstract.md"), (1, ".overview.md")):+ directory_sources[+ (directory_uri, level)+ ] = await self._read_memory_directory_source(+ directory_uri + "/" + filename, ctx=ctx+ )+ existing_by_uri_level: dict[tuple[str, int], dict[str, Any]] = {}+ for record_id, record in sorted(inventory.items()):+ record_uri = str(record.get("uri") or "")+ try:+ level = int(record.get("level", -1))+ except (TypeError, ValueError):+ continue+ if record_uri and level in {0, 1, 2}:+ existing_by_uri_level.setdefault((record_uri, level), dict(record, id=record_id))++ overview_cache: dict[str, dict[str, str]] = {+ directory: SemanticProcessor._parse_overview_md(source.text) if source.text else {}+ for (directory, level), source in directory_sources.items()+ if level == 1+ }+ async def process_file(file_uri: str) -> _ReindexCounters:+ file_counters = _ReindexCounters(scanned_records=1)+ failure_message = f"Skipped {file_uri}: failed to read memory body"try:+ body_source = await self._read_memory_body(file_uri, ctx=ctx)+ if body_source.error:+ raise body_source.error+ failure_message = f"Failed to reindex {file_uri} vector"+ body = body_source.text if body_source.exists else ""+ memory_content = MemoryFileUtils.read(body).content if body else ""+ existing = existing_by_uri_level.get((file_uri, int(ContextLevel.DETAIL)))+ abstract = self._record_abstract(existing)+ if not abstract:+ abstract = await self._best_memory_file_summary(+ file_uri,+ ctx=ctx,+ overview_cache=overview_cache,+ )+ if not body and existing is None:+ file_counters.unsupported_records += 1+ file_counters.warnings.append(f"No memory source found for {file_uri}")+ return file_counters++ parent_uri = VikingURI(file_uri).parent.uri+ detail_abstract = abstract or memory_content or bodyawait self._upsert_context(uri=file_uri,parent_uri=parent_uri,- abstract=abstract,- vector_text=abstract,+ abstract=detail_abstract,+ vector_text=body or abstract,is_leaf=True,context_type=ContextType.MEMORY.value,level=ContextLevel.DETAIL,@@ -1439,12 +1789,15 @@ class ReindexExecutor:ingest_options=ingest_options,)file_counters.rebuilt_records += 1- file_counters.warnings.append(- f"Reindexed {file_uri} from abstract fallback because original memory body is unavailable"- )+ if not body:+ file_counters.warnings.append(+ f"Reindexed {file_uri} from abstract fallback because original memory body is unavailable"+ )+ except (asyncio.CancelledError, LockError):+ raiseexcept Exception as exc:file_counters.failed_records += 1- file_counters.warnings.append(f"Failed to reindex {file_uri} vector: {exc}")+ file_counters.warnings.append(f"{failure_message}: {exc}")return file_countersconcurrency = self._effective_file_vectorization_concurrency()@@ -1455,13 +1808,197 @@ class ReindexExecutor:len(file_uris),concurrency,)- await self._run_ordered_counter_batches(- file_uris,- concurrency=concurrency,- processor=process_file,+ if reconcile_only:+ counters.scanned_records += len(file_uris) + len(directory_uris)+ else:+ await self._run_ordered_counter_batches(+ [file_uri for file_uri in file_uris if root_is_dir or file_uri == uri],+ concurrency=concurrency,+ processor=process_file,+ counters=counters,+ )++ await self._delete_memory_orphans(+ root=uri,+ vector_inventory=inventory,+ live_file_uris=set(file_uris),+ live_directory_uris=directory_uris,+ live_directory_sources={+ key for key, source in directory_sources.items() if source.text or source.error+ },counters=counters,+ ctx=ctx,+ file_scope=not root_is_dir,+ lease_ref=lease_ref,)+ async def _delete_memory_orphans(+ self,+ *,+ root: str,+ vector_inventory: Mapping[str, Mapping[str, Any]],+ live_file_uris: set[str],+ live_directory_uris: set[str],+ live_directory_sources: set[tuple[str, int]],+ counters: _ReindexCounters,+ ctx: RequestContext,+ file_scope: bool = False,+ lease_ref: dict[str, Any] | None = None,+ ) -> None:+ """Delete memory V records whose formal source no longer exists.++ Covers plain L2 files, directory L0/L1 markers, and virtual+ legacy ``#chunk_NNNN`` records, which current memory indexing no longer produces.+ Session URIs and any record outside the memory namespace are never+ deleted here.+ """+ orphan_ids: list[str] = []+ for record_id, record in vector_inventory.items():+ record_uri = str(record.get("uri") or "")+ try:+ level = int(record.get("level", -1))+ except (TypeError, ValueError):+ continue+ if level not in {0, 1, 2}:+ continue+ if is_session_uri(record_uri) or classify_uri(record_uri).context_type != "memory":+ continue+ record_id = str(record_id)+ if not record_id:+ continue+ if level in {0, 1}:+ if (+ record_uri not in live_directory_uris+ or (record_uri, level) not in live_directory_sources+ ):+ orphan_ids.append(record_id)+ continue+ # level == 2+ # An actual filename wins over the virtual-chunk suffix syntax.+ if record_uri in live_file_uris:+ continue+ orphan_ids.append(record_id)++ if not orphan_ids:+ return+ service = get_service()+ if service.vikingdb_manager is None:+ return+ owner_ctx = self._content_owner_ctx(root, ctx)+ chunk_uris = {+ str(vector_inventory[record_id].get("uri") or "")+ for record_id in orphan_ids+ if file_scope+ and re.fullmatch(+ re.escape(root) + r"#chunk_[0-9]{4,}",+ str(vector_inventory[record_id].get("uri") or ""),+ )+ }+ guard = None+ try:+ if chunk_uris:+ fs = get_viking_fs()+ guard = PathLockGuard(fs._async_agfs)+ paths = [fs._uri_to_path(uri, ctx=owner_ctx) for uri in sorted(chunk_uris)]+ kwargs = {"owner_lease_ref": lease_ref} if lease_ref is not None else {}+ await guard.acquire(+ lambda: fs._async_agfs.pathlock_acquire_exact_batch(paths, **kwargs)+ )+ real_files = set()+ for uri in sorted(chunk_uris):+ try:+ stat = await fs.stat(uri, ctx=owner_ctx, skip_count=True)+ except NotFoundError:+ continue+ if stat.get("access") == "denied":+ raise PermissionDeniedError(f"Cannot confirm memory chunk source: {uri}")+ if not stat.get("isDir", stat.get("is_dir")):+ real_files.add(uri)+ orphan_ids = [+ record_id+ for record_id in orphan_ids+ if str(vector_inventory[record_id].get("uri") or "") not in real_files+ ]+ if orphan_ids:+ counters.deleted_records += await service.vikingdb_manager.strict_delete(+ orphan_ids, ctx=owner_ctx+ )+ finally:+ if guard is not None:+ await guard.close()++ async def _read_reindex_inventory(+ self,+ vikingdb: Any,+ *,+ uri: str,+ ctx: RequestContext,+ recursive: bool,+ include_direct_children: bool = False,+ include_virtual_chunks: bool = False,+ ) -> dict[str, dict[str, Any]]:+ retry_delays = (0.0, 0.25, 1.0)+ for attempt, delay in enumerate(retry_delays, start=1):+ if delay:+ await asyncio.sleep(delay)+ try:+ kwargs = {"include_direct_children": True} if include_direct_children else {}+ if include_virtual_chunks:+ kwargs["include_virtual_chunks"] = True+ return await vikingdb.get_reindex_inventory_under_uri(+ uri,+ ctx=ctx,+ output_fields=[+ "abstract",+ "context_type",+ "owner_user_id",+ "name",+ "description",+ "tags",+ "search_tags",+ ],+ recursive=recursive,+ **kwargs,+ )+ except asyncio.CancelledError:+ raise+ except Exception as exc:+ incomplete = isinstance(exc, RuntimeError) and any(+ marker in str(exc)+ for marker in (+ "cursor ended after",+ "ended after",+ "records but count was",+ )+ )+ if attempt == len(retry_delays) or not (+ bool(getattr(exc, "retryable", False)) or incomplete+ ):+ raise+ logger.warning(+ "Reindex vector inventory was rate limited; retrying uri=%s attempt=%d/%d "+ "error=%s",+ uri,+ attempt,+ len(retry_delays),+ exc,+ )+ raise AssertionError("unreachable")++ async def _best_memory_file_summary(+ self,+ uri: str,+ *,+ ctx: RequestContext,+ overview_cache: dict[str, dict[str, str]],+ ) -> str:+ parent_uri = VikingURI(uri).parent.uri+ if parent_uri not in overview_cache:+ raw = await self._safe_read_text(f"{parent_uri}/.overview.md", ctx=ctx)+ overview_cache[parent_uri] = SemanticProcessor._parse_overview_md(raw) if raw else {}+ file_name = uri.rsplit("/", 1)[-1]+ return overview_cache[parent_uri].get(file_name, "")+async def _reindex_memory_directory_chain(self,*,@@ -1469,11 +2006,22 @@ class ReindexExecutor:counters: _ReindexCounters,ctx: RequestContext,ingest_options: IngestOptions | None = None,- ) -> None:+ ) -> dict[tuple[str, int], _SourceRead]:+ sources: dict[tuple[str, int], _SourceRead] = {}for directory_uri in directory_uris:counters.scanned_records += 1- abstract = await self._read_directory_abstract(directory_uri, ctx=ctx)- overview = await self._read_directory_overview(directory_uri, ctx=ctx)+ for level, filename in ((0, ".abstract.md"), (1, ".overview.md")):+ source = await self._read_memory_directory_source(+ directory_uri + "/" + filename, ctx=ctx+ )+ sources[(directory_uri, level)] = source+ if source.error:+ counters.failed_records += 1+ counters.warnings.append(+ f"Failed to read {directory_uri}/{filename}: {source.error}"+ )+ abstract = sources[(directory_uri, 0)].text+ overview = sources[(directory_uri, 1)].textif not abstract and not overview:continue@@ -1494,6 +2042,8 @@ class ReindexExecutor:ingest_options=ingest_options,)counters.rebuilt_records += 1+ except (asyncio.CancelledError, LockError):+ raiseexcept Exception as exc:counters.failed_records += 1counters.warnings.append(f"Failed to reindex {directory_uri} L0 vector: {exc}")@@ -1514,20 +2064,31 @@ class ReindexExecutor:ingest_options=ingest_options,)counters.rebuilt_records += 1+ except (asyncio.CancelledError, LockError):+ raiseexcept Exception as exc:counters.failed_records += 1counters.warnings.append(f"Failed to reindex {directory_uri} L1 vector: {exc}")+ return sources- async def _regenerate_skill_semantics(- self, *, uri: str, ctx: RequestContext, lock: dict | None = None- ) -> None:- processor = await self._semantic_processor_for(ctx)- await processor._skill_root_semantics(- uri,- ctx=ctx,- regenerate=True,- lock=lock,- )+ async def _read_memory_directory_source(self, uri: str, *, ctx: RequestContext) -> _SourceRead:+ try:+ raw = await get_viking_fs().read_file(uri, ctx=ctx)+ except (asyncio.CancelledError, LockError):+ raise+ except NotFoundError:+ return _SourceRead(exists=False)+ except Exception as exc:+ return _SourceRead(exists=True, error=exc)+ try:+ text = body_for_preview(raw).strip()+ if _is_not_ready_sentinel(text, _ABSTRACT_NOT_READY_SUFFIX) or _is_not_ready_sentinel(+ text, _OVERVIEW_NOT_READY_SUFFIX+ ):+ text = ""+ return _SourceRead(exists=True, text=text)+ except Exception as exc:+ return _SourceRead(exists=True, error=exc)async def _read_directory_abstract(self, uri: str, *, ctx: RequestContext) -> str:try:@@ -1543,56 +2104,6 @@ class ReindexExecutor:return ""return "" if _is_not_ready_sentinel(value, _OVERVIEW_NOT_READY_SUFFIX) else value- async def _best_file_summary(self, uri: str, *, ctx: RequestContext) -> str:- parent_uri = VikingURI(uri).parent.uri- file_name = uri.rsplit("/", 1)[-1]- overviews = await self._safe_read_text(f"{parent_uri}/.overview.md", ctx=ctx)- if overviews:- parsed = SemanticProcessor._parse_overview_md(overviews)- if file_name in parsed:- return parsed[file_name]- existing = await self._fetch_existing_record(- uri=uri,- level=2,- ctx=self._content_owner_ctx(uri, ctx),- )- return self._record_abstract(existing)-- async def _best_resource_file_vector_text(- self,- uri: str,- summary: str,- ctx: RequestContext,- file_content: bytes | None = None,- ) -> str:- existing = await self._fetch_existing_record(- uri=uri,- level=2,- ctx=self._content_owner_ctx(uri, ctx),- )- fallback = self._record_abstract(existing)- content_type = get_resource_content_type(uri.rsplit("/", 1)[-1])-- if content_type == ResourceContentType.TEXT:- if self.vector_config_resolver is None:- raise RuntimeError("ReindexExecutor requires a vector config resolver")- embedding_config = (await self.vector_config_resolver.resolve(ctx.account_id)).embedding- text_source = embedding_config.text_source- if text_source in SUMMARY_TEXT_SOURCES and summary:- return summary- content = (- _decode_text_bytes(file_content)- if file_content is not None- else await self._safe_read_text(uri, ctx=ctx)- )- if content:- return truncate_embedding_input(content, embedding_config.max_input_tokens)- return summary or fallback-- if summary:- return summary- return fallback-async def _upsert_context(self,*,@@ -1644,68 +2155,16 @@ class ReindexExecutor:details={"uri": uri, "level": int(level)},)- async def _fetch_existing_record(- self,- *,- uri: str,- level: int,- ctx: RequestContext,- ) -> Optional[dict[str, Any]]:- service = get_service()- assert service.vikingdb_manager is not None- records = await service.vikingdb_manager.get_context_by_uri(- uri=uri,- level=level,- limit=1,- ctx=ctx,- )- return records[0] if records else None-- async def _skill_meta(- self,- *,- uri: str,- abstract: str,- ctx: RequestContext,- ) -> dict[str, Any]:- import yaml-- body = body_for_preview(abstract)- try:- parsed = yaml.safe_load(body)- except yaml.YAMLError:- parsed = None- record = await self._fetch_existing_record(- uri=uri, level=0, ctx=self._content_owner_ctx(uri, ctx)- )- previous_meta = (record or {}).get("meta") or {}- source_path = previous_meta.get("source_path", "")- if (- isinstance(parsed, dict)- and isinstance(parsed.get("name"), str)- and isinstance(parsed.get("description"), str)- ):- return {- "source_path": source_path,- **{- key: parsed.get(key, [] if key in {"tags", "allowed_tools"} else "")- for key in ("name", "description", "tags", "allowed_tools")- },- }- return {- **{- key: previous_meta[key] for key in ("tags", "allowed_tools") if key in previous_meta- },- "name": previous_meta.get("name") or uri.rstrip("/").split("/")[-1],- "description": body,- "source_path": source_path,- }-def _record_abstract(self, record: Optional[dict[str, Any]]) -> str:if not record:return ""return str(record.get("abstract") or "")+ @staticmethod+ def _is_memory_source_uri(uri: str) -> bool:+ name = uri_parts(uri)[-1]+ return name not in _MEMORY_NON_SOURCE_FILENAMES and not is_storage_internal_name(name)+def _is_hidden_meta_file(self, uri: str) -> bool:return uri.endswith("/.abstract.md") or uri.endswith("/.overview.md")@@ -1738,40 +2197,3 @@ class ReindexExecutor:else:text = str(content or "")return _SourceRead(exists=True, text=text)-- def _chunk_memory_body(self, uri: str, body: str) -> Iterable[tuple[str, str]]:- semantic = get_openviking_config().semantic- chunk_chars = semantic.memory_chunk_chars- overlap = semantic.memory_chunk_overlap- if len(body) <= chunk_chars:- return []-- chunks: list[str] = []- start = 0- while start < len(body):- previous_start = start- end = start + chunk_chars- if end < len(body):- boundary = body.rfind("\n\n", start, end)- if boundary > start + chunk_chars // 2:- end = boundary + 2- chunks.append(body[start:end].strip())- start = end - overlap- if start <= previous_start:- start = previous_start + 1- if start >= len(body):- break-- return [(f"{uri}#chunk_{idx:04d}", chunk) for idx, chunk in enumerate(chunks) if chunk]-- def _best_non_empty(self, *values: str) -> str:- for value in values:- if value:- return value- return ""-- def _prefer_non_empty(self, *values: str) -> str:- for value in values:- if value:- return value- return ""diff --git a/openviking/storage/queuefs/reindex_processor.py b/openviking/storage/queuefs/reindex_processor.pyindex 8ea397928..eac1e38be 100644--- a/openviking/storage/queuefs/reindex_processor.py+++ b/openviking/storage/queuefs/reindex_processor.py@@ -131,19 +131,12 @@ class ReindexProcessor(DequeueHandlerBase):await tracker.start(msg.task_id, account_id=ctx.account_id, user_id=ctx.user.user_id, stage="queued")- from openviking.server.dependencies import get_servicefrom openviking.service.reindex_executor import ReindexExecutor- service = get_service()- executor = ReindexExecutor(- vlm_resolver=service._vlm_resolver,- vector_config_resolver=service._vector_config_resolver,- )+ executor = ReindexExecutor()with bind_telemetry(telemetry):- # _run owns an existing lease and releases or hands it off on- # every exit path. Clear this consumer's reference before the- # call so an exception cannot release the same lease twice.- executor_lease = lock_guard.detach()+ # Share ownership state so even argument/initialization failures+ # release the lease once, while handoff clears both references.result = await executor._run(uri=msg.uri,object_type=msg.object_type,@@ -154,7 +147,7 @@ class ReindexProcessor(DequeueHandlerBase):mode=msg.mode, tags=msg.tags, tag_mode=msg.tag_mode),ctx=ctx,- existing_lease=executor_lease,+ lease_guard=lock_guard,)metadata = extract_task_metadata(data)await tracker.wait_for_task_work(diff --git a/openviking/storage/resource_rfv.py b/openviking/storage/resource_rfv.pyindex 8fcd60b08..b423db1ee 100644--- a/openviking/storage/resource_rfv.py+++ b/openviking/storage/resource_rfv.py@@ -91,8 +91,16 @@ async def build_rfv_snapshot(source_read_concurrency: int = 8,root_is_dir: bool | None = None,formal_inventory: tuple[bool, list[Mapping[str, Any]], bool] | None = None,+ vector_inventory: Mapping[str, Mapping[str, Any]] | None = None,+ root_exists: bool = True,) -> RFVSnapshot:- """Read one complete F/V inventory and each selected embedding source once."""+ """Read one complete F/V inventory and each selected embedding source once.++ ``root_exists=False`` means the target root is already absent from the formal+ tree (for example, an entire ``resources`` or skill package removed outside the+ normal APIs). No formal node is synthesized for it, so every vector record under+ the root resolves to an orphan and is deleted during the same reindex.+ """root = target_uri.rstrip("/")projection = request_intent.required_vector_fields() | {"abstract"}if target_uri.rstrip("/").startswith(("viking://user/", "viking://agent/skills/")):@@ -134,16 +142,19 @@ async def build_rfv_snapshot(complete = not any(entry.get("access") == "denied" for entry in raw_entries)return is_dir, list(raw_entries), complete- formal_task = asyncio.create_task(read_formal_inventory())- vector_task = asyncio.create_task(- vikingdb.get_incremental_inventory_under_uri(+ async def read_vector_inventory() -> Mapping[str, Mapping[str, Any]]:+ if vector_inventory is not None:+ return vector_inventory+ return await vikingdb.get_incremental_inventory_under_uri(root,ctx=ctx,output_fields=sorted(projection),recursive=recursive,include_direct_children=include_direct_children,)- )++ formal_task = asyncio.create_task(read_formal_inventory())+ vector_task = asyncio.create_task(read_vector_inventory())try:(root_is_dir, raw_entries, complete), inventory = await asyncio.gather(formal_task, vector_task@@ -158,7 +169,9 @@ async def build_rfv_snapshot(if not complete:raise ValueError("cannot build RFV snapshot from an incomplete formal tree")- nodes: dict[str, tuple[str, bool]] = {"": (root, root_is_dir)}+ nodes: dict[str, tuple[str, bool]] = (+ {"": (root, root_is_dir)} if root_exists else {}+ )sidecars: dict[tuple[str, int], str] = {}source_metadata_uri: str | None = Nonefor raw in raw_entries:@@ -185,7 +198,7 @@ async def build_rfv_snapshot(continuenodes[relative_path] = (uri, bool(raw.get("isDir")))- if root_is_dir and not recursive:+ if root_exists and root_is_dir and not recursive:for level, name in ((0, ".abstract.md"), (1, ".overview.md")):sidecars[("", level)] = f"{root}/{name}"@@ -280,7 +293,11 @@ async def build_rfv_snapshot(for record_id, record in inventory.items():uri = str(record.get("uri") or "")relative_path = _relative_uri(root, uri)- if relative_path is None:+ if relative_path is None or (+ not recursive+ and relative_path+ and (not include_direct_children or "/" in relative_path)+ ):raise ValueError(f"RFV vector record is outside target: {uri}")level = int(record.get("level", -1))fields = {diff --git a/openviking/storage/viking_fs/_snapshot.py b/openviking/storage/viking_fs/_snapshot.pyindex bb6ef18f0..9fc415700 100644--- a/openviking/storage/viking_fs/_snapshot.py+++ b/openviking/storage/viking_fs/_snapshot.py@@ -883,10 +883,7 @@ class _SnapshotMixin:if not tasks:return- executor = ReindexExecutor(- vlm_resolver=self._vlm_resolver,- vector_config_resolver=self._vector_config_resolver,- )+ executor = ReindexExecutor()with detach_task_context():for op, uri, level in tasks:loop.create_task(@@ -931,10 +928,7 @@ class _SnapshotMixin:user_id=ctx.user.user_id,stage="reindexing",)- executor = ReindexExecutor(- vlm_resolver=self._vlm_resolver,- vector_config_resolver=self._vector_config_resolver,- )+ executor = ReindexExecutor()reindex_ctx = self._restore_reindex_context(ctx)with bind_task_context(task_id,diff --git a/openviking/storage/viking_vector_index_backend.py b/openviking/storage/viking_vector_index_backend.pyindex 067d61d1a..e8197a6cb 100644--- a/openviking/storage/viking_vector_index_backend.py+++ b/openviking/storage/viking_vector_index_backend.py@@ -5,6 +5,7 @@from __future__ import annotationsimport asyncio+import reimport threadingimport uuidfrom contextlib import asynccontextmanager@@ -2362,6 +2363,92 @@ class VikingVectorIndexBackend:assert last_error is not Noneraise last_error+ async def get_reindex_inventory_under_uri(+ self,+ uri: str,+ *,+ ctx: RequestContext,+ output_fields: Optional[List[str]] = None,+ recursive: bool = True,+ include_direct_children: bool = False,+ include_virtual_chunks: bool = False,+ ) -> Dict[str, Dict[str, Any]]:+ """Load one complete lightweight reindex inventory without page-level planning."""+ projection = list(+ dict.fromkeys(+ [+ *INCREMENTAL_INVENTORY_OUTPUT_FIELDS,+ *(output_fields or []),+ ]+ )+ )+ canonical_uri = resolve_uri(uri).uri.rstrip("/")+ depth = -1 if recursive else 1 if include_direct_children else 0+ if include_virtual_chunks and depth != 0:+ raise ValueError("virtual chunk inventory requires an exact file scope")+ # Path-index queries are portable; some VikingDB services reject string prefixes.+ inventory_uri = canonical_uri.rsplit("/", 1)[0] if include_virtual_chunks else canonical_uri+ inventory_depth = 1 if include_virtual_chunks else depth+ root_parts = uri_parts(inventory_uri)+ chunk_prefix = canonical_uri + "#chunk_"+ scope = And(+ [+ Eq("account_id", ctx.account_id),+ PathScope("uri", inventory_uri, depth=inventory_depth),+ In("level", [0, 1, 2]),+ ]+ )+ records: Dict[str, Dict[str, Any]] = {}+ seen_ids: set[str] = set()+ async for record in self._strict_scan(+ ctx,+ scope,+ output_fields=projection,+ batch_size=5000,+ what=f"Reindex inventory under {canonical_uri}",+ ):+ record_id = str(record.get("id") or "")+ record_uri = str(record.get("uri") or "")+ try:+ record_parts = uri_parts(record_uri)+ except ValueError:+ record_parts = None+ try:+ level = int(record.get("level"))+ except (TypeError, ValueError):+ level = -1+ is_chunk = (+ include_virtual_chunks+ and level == 2+ and re.fullmatch(re.escape(chunk_prefix) + r"[0-9]{4,}", record_uri) is not None+ )+ if (+ not record_id+ or level not in {0, 1, 2}+ or record_parts is None+ or record_parts[: len(root_parts)] != root_parts+ or (inventory_depth >= 0 and len(record_parts) - len(root_parts) > inventory_depth)+ ):+ raise RuntimeError(+ f"Reindex inventory under {canonical_uri} returned an invalid record: "+ f"id={record_id or '<missing>'} uri={record_uri or '<missing>'} level={level}"+ )+ if record_id in seen_ids:+ raise RuntimeError(f"Reindex inventory returned duplicate record id: {record_id}")+ seen_ids.add(record_id)+ if include_virtual_chunks and record_uri != canonical_uri and not is_chunk:+ continue+ records[record_id] = {field: record[field] for field in projection if field in record}+ records[record_id].update(+ {+ "id": record_id,+ "uri": record_uri,+ "level": level,+ "md5": str(record.get("md5") or ""),+ }+ )+ return records+async def get_incremental_inventory_by_uris(self,uris: List[str],diff --git a/openviking_cli/utils/config/parser_config.py b/openviking_cli/utils/config/parser_config.pyindex c5c04f986..9c8caa365 100644--- a/openviking_cli/utils/config/parser_config.py+++ b/openviking_cli/utils/config/parser_config.py@@ -719,11 +719,10 @@ class SemanticConfig:"""Maximum characters for generated overviews."""memory_chunk_chars: int = 2000- """Maximum characters per chunk when splitting long memories for vectorization.- Memories shorter than this are vectorized as a single record."""+ """Legacy setting retained for configuration compatibility; memory vectors are not chunked."""memory_chunk_overlap: int = 200- """Character overlap between adjacent memory chunks for context continuity."""+ """Legacy setting retained for configuration compatibility; unused by memory vectorization."""def __post_init__(self):if self.overview_sample_limit <= 0:diff --git a/tests/server/test_admin_rebuild_api.py b/tests/server/test_admin_rebuild_api.pyindex 8cb93d032..e66b292cc 100644--- a/tests/server/test_admin_rebuild_api.py+++ b/tests/server/test_admin_rebuild_api.py@@ -7,10 +7,9 @@ from unittest.mock import AsyncMockimport httpximport pytest-from openviking.core.context import ContextLevelfrom openviking.server.identity import RequestContext, Rolefrom openviking.storage.queuefs.process_result import ProcessResult-from openviking_cli.exceptions import OpenVikingError, PermissionDeniedError+from openviking_cli.exceptions import PermissionDeniedErrorfrom openviking_cli.session.user_id import UserIdentifierfrom tests.server.test_admin_api import ROOT_KEYfrom tests.server.test_admin_api import admin_app as _admin_app_fixture@@ -27,16 +26,6 @@ ROOT_ACCOUNT_HEADERS = {}-def semantic_config():- class Resolver:- async def get_vlm(self, account_id):- del account_id- return SimpleNamespace(max_concurrent=2)-- return Resolver()--def _make_reindex_run(ctx, counters):from openviking.service.reindex_executor import _ReindexRunContext@@ -52,16 +41,13 @@ async def test_reindex_requires_admin_role(admin_client: httpx.AsyncClient):async def test_reindex_user_can_only_target_own_user_scope(monkeypatch):- from inspect import signature-from openviking.server.routers.content import ReindexRequest, reindex+ from openviking.service.reindex_executor import ReindexExecutorctx = RequestContext(user=UserIdentifier(account_id="reindex_user_scope", user_id="bob"),role=Role.USER,)- role_dependency = signature(reindex).parameters["ctx"].default.dependency- assert await role_dependency(ctx=ctx) == ctxseen = {}class FakeService:@@ -107,16 +93,20 @@ async def test_reindex_user_can_only_target_own_user_scope(monkeypatch):assert peer_scope.status == "ok"assert seen["uri"] == "viking://user/bob/peers/peer-a/resources"- for hidden_uri in (- "viking://user/bob",- "viking://user/bob/peers",- "viking://user/bob/peers/peer-b/resources",- ):- with pytest.raises(PermissionDeniedError):- await reindex(- body=ReindexRequest(uri=hidden_uri, mode="vectors_only"),- ctx=peer_ctx,- )+ with pytest.raises(PermissionDeniedError):+ await reindex(+ body=ReindexRequest(uri="viking://user/bob/peers/peer-b/resources"),+ ctx=peer_ctx,+ )++ # An administrator's actor-peer view is also incomplete for reconciliation.+ monkeypatch.setattr(+ "openviking.server.routers.content.get_service",+ lambda: SimpleNamespace(reindex=ReindexExecutor().execute),+ )+ admin_peer_ctx = RequestContext(user=ctx.user, role=Role.ADMIN, actor_peer_id="peer-a")+ with pytest.raises(PermissionDeniedError):+ await reindex(body=ReindexRequest(uri="viking://user/bob"), ctx=admin_peer_ctx)async def test_reindex_rejects_unsupported_uri(admin_client: httpx.AsyncClient):@@ -148,22 +138,6 @@ async def test_reindex_rejects_session_uri(admin_client: httpx.AsyncClient, uri:assert body["status"] == "error"-def test_reindex_ignores_unknown_request_fields():- from openviking.server.routers.content import ReindexRequest-- request = ReindexRequest.model_validate(- {- "uri": "viking://resources/demo",- "mode": "vectors_only",- "reason": "unused",- }- )-- assert request.uri == "viking://resources/demo"- assert request.mode == "vectors_only"- assert "reason" not in request.model_dump()--async def test_reindex_root_requires_explicit_account(admin_client: httpx.AsyncClient):resp = await admin_client.post("/api/v1/content/reindex",@@ -214,8 +188,6 @@ async def test_reindex_resource_vectors_only_wait_true(tmp_path, monkeypatch):service = SimpleNamespace(viking_fs=fs,vikingdb_manager=SimpleNamespace(has_queue_manager=True),- _vlm_resolver=None,- _vector_config_resolver=None,reindex=ReindexExecutor().execute,)monkeypatch.setattr("openviking.server.routers.content.get_service", lambda: service)@@ -287,2483 +259,520 @@ async def test_reindex_resource_vectors_only_wait_true(tmp_path, monkeypatch):@pytest.mark.asyncio-async def test_reindex_clear_without_tags_is_forwarded(monkeypatch):- from openviking.server.routers.content import ReindexRequest, reindex-- seen = {}-- class FakeService:- async def reindex(self, **kwargs):- seen.update(kwargs)- return {"status": "completed"}-- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- monkeypatch.setattr("openviking.server.routers.content.get_service", lambda: FakeService())-- await reindex(- body=ReindexRequest(- uri="viking://resources/demo",- tag_mode="clear",- ),- ctx=ctx,- )-- assert seen["tags"] is None- assert seen["tag_mode"] == "clear"---def test_reindex_executor_resolves_clear_without_tags():- from openviking.service.reindex_executor import ReindexExecutor- from openviking.utils.ingest_options import IngestOptions+async def test_reindex_rfv_skips_complete_file_without_enqueue_or_second_read(monkeypatch):+ from unittest.mock import AsyncMock- assert ReindexExecutor._resolve_ingest_options(- mode="vectors_only",- tags=None,- tag_mode="clear",- ) == IngestOptions(search_tags=[], search_tag_mode="clear")+ from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters+ from openviking.utils.content_hash import content_md5+ uri = "viking://resources/demo.md"+ body = b"same body"-async def test_reindex_resource_vectors_only_wait_false(monkeypatch):- from openviking.server.routers.content import ReindexRequest, reindex+ class FakeFS:+ stat = AsyncMock(return_value={"isDir": False})+ read_file_bytes = AsyncMock(return_value=body)- class FakeService:- async def reindex(self, *, uri, mode, wait, ctx):+ class FakeDB:+ async def get_reindex_inventory_under_uri(+ self, target, *, ctx, output_fields, recursive=True, include_direct_children=False+ ):+ assert target == uri+ assert recursive is False+ assert include_direct_children is Falsereturn {- "task_id": "rbld_123",- "status": "accepted",- "uri": uri,- "object_type": "resource",- "mode": mode,+ "file-l2": {+ "id": "file-l2",+ "uri": uri,+ "level": 2,+ "md5": content_md5(body),+ }}+ resource_processor = SimpleNamespace(_enqueue_index_actions=AsyncMock())+ service = SimpleNamespace(+ viking_fs=FakeFS(),+ vikingdb_manager=FakeDB(),+ _resource_processor=resource_processor,+ )+ monkeypatch.setattr("openviking.service.reindex_executor.get_service", lambda: service)ctx = RequestContext(user=UserIdentifier(account_id="test", user_id="alice"),role=Role.ROOT,)- request = ReindexRequest(uri="viking://resources/demo", mode="vectors_only", wait=False)+ counters = _ReindexCounters()- monkeypatch.setattr("openviking.server.routers.content.get_service", lambda: FakeService())- response = await reindex(body=request, ctx=ctx)+ await ReindexExecutor()._reindex_rfv(+ uri=uri,+ mode="vectors_only",+ context_type="resource",+ recursive=False,+ run=_make_reindex_run(ctx, counters),+ )- assert response.status == "ok"- assert response.result["status"] == "accepted"- assert response.result["task_id"] == "rbld_123"- assert response.result["object_type"] == "resource"- assert "reason" not in response.result+ resource_processor._enqueue_index_actions.assert_not_awaited()+ service.viking_fs.read_file_bytes.assert_awaited_once_with(uri, ctx=ctx)+ assert counters.scanned_records == 1+ assert counters.rebuilt_records == 0@pytest.mark.asyncio-async def test_reindex_passes_non_recursive_to_service(monkeypatch):- from openviking.server.routers.content import ReindexRequest, reindex-- seen = {}-- class FakeService:- async def reindex(self, **kwargs):- seen.update(kwargs)- return {"status": "completed"}+async def test_reindex_rfv_deletes_missing_root_with_owner_context(monkeypatch):+ from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- monkeypatch.setattr("openviking.server.routers.content.get_service", lambda: FakeService())+ root = "viking://user/bob/resources/gone"+ inventory = {+ "root-l0": {"id": "root-l0", "uri": root, "level": 0},+ "root-l1": {"id": "root-l1", "uri": root, "level": 1},+ "file-l2": {"id": "file-l2", "uri": root + "/old.md", "level": 2},+ }+ db = SimpleNamespace(strict_delete=AsyncMock(return_value=3))+ service = SimpleNamespace(viking_fs=SimpleNamespace(), vikingdb_manager=db)+ monkeypatch.setattr("openviking.service.reindex_executor.get_service", lambda: service)+ ctx = RequestContext(user=UserIdentifier("test", "admin"), role=Role.ADMIN)+ counters = _ReindexCounters()- await reindex(- body=ReindexRequest(- uri="viking://resources/demo",- mode="semantic_and_vectors",- recursive=False,- ),- ctx=ctx,+ await ReindexExecutor()._reindex_rfv(+ uri=root,+ mode="vectors_only",+ context_type="resource",+ recursive=True,+ run=_make_reindex_run(ctx, counters),+ root_is_dir=True,+ root_exists=False,+ formal_entries=[],+ vector_inventory=inventory,)- assert seen["recursive"] is False+ (deleted,) = db.strict_delete.await_args.args+ assert set(deleted) == set(inventory)+ assert db.strict_delete.await_args.kwargs["ctx"].user == UserIdentifier("test", "bob")+ assert counters.deleted_records == 3@pytest.mark.asyncio-async def test_reindex_passes_force_to_service(monkeypatch):- from openviking.server.routers.content import ReindexRequest, reindex-- seen = {}-- class FakeService:- async def reindex(self, **kwargs):- seen.update(kwargs)- return {"status": "completed"}+async def test_reindex_memory_semantic_request_survives_write_deduplication(monkeypatch):+ from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters+ from openviking.storage.queuefs.semantic_msg import SemanticMsg+ from openviking.storage.queuefs.semantic_queue import SemanticQueue+ from openviking.utils.ingest_options import IngestOptions+ from openviking_cli.exceptions import NotFoundError- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,+ root = "viking://user/bob/memories/events"+ ctx = RequestContext(user=UserIdentifier("test", "bob"), role=Role.USER)+ queue_fs = SimpleNamespace(mkdir=AsyncMock(), write=AsyncMock(return_value="queued"))+ monkeypatch.setattr(+ "openviking.storage.queuefs.named_queue.AsyncAGFSClient", lambda _: queue_fs)- monkeypatch.setattr("openviking.server.routers.content.get_service", lambda: FakeService())-- await reindex(- body=ReindexRequest(- uri="viking://resources/demo",- mode="vectors_only",- force=True,+ queue = SemanticQueue(object(), "/queue", "Semantic")+ manager = SimpleNamespace(SEMANTIC="Semantic", get_queue=lambda *args, **kw: queue)+ monkeypatch.setattr("openviking.storage.queuefs.get_queue_manager", lambda: manager)+ await queue.enqueue(+ SemanticMsg(+ uri=root,+ context_type="memory",+ account_id="test",+ user_id="bob",+ peer_id="bob",+ )+ )+ fs = SimpleNamespace(+ stat=AsyncMock(return_value={"isDir": True}),+ exists=AsyncMock(return_value=True),+ tree=AsyncMock(return_value=[{"uri": root + "/live.md", "isDir": False}]),+ read_file=AsyncMock(side_effect=NotFoundError(root)),+ )+ db = SimpleNamespace(+ get_reindex_inventory_under_uri=AsyncMock(+ return_value={+ "gone": {"id": "gone", "uri": root + "/gone.md", "level": 2},+ }),- ctx=ctx,+ strict_delete=AsyncMock(return_value=1),)+ monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ "openviking.service.reindex_executor.get_service",+ lambda: SimpleNamespace(vikingdb_manager=db),+ )+ counters = _ReindexCounters()+ run = _make_reindex_run(ctx, counters)+ run.ingest_options = IngestOptions.from_search_tags(["team=search"], mode="append")- assert seen["force"] is True+ await ReindexExecutor()._reindex_memory(uri=root, mode="semantic_and_vectors", run=run)++ # Explicit reindex survives ordinary write deduplication.+ assert queue_fs.write.await_count == 2+ queued = SemanticMsg.from_dict(json.loads(queue_fs.write.await_args.args[1]))+ assert queued.uri == root and queued.recursive+ assert queued.use_hierarchical_aggregation and not queued.skip_vectorization+ assert queued.ingest_options == run.ingest_options+ assert counters.deleted_records == 1+ db.strict_delete.assert_awaited_once_with(["gone"], ctx=ctx)@pytest.mark.asyncio-async def test_reindex_rejects_non_recursive_namespace_before_work_starts():+async def test_reindex_rejects_non_recursive_namespace_before_starting_work(monkeypatch):from openviking.service.reindex_executor import ReindexExecutor+ from openviking_cli.exceptions import InvalidArgumentError++ def unexpected_tracker():+ pytest.fail("unsupported non-recursive reindex must not start task tracking")+ monkeypatch.setattr("openviking.service.reindex_executor.get_task_tracker", unexpected_tracker)ctx = RequestContext(user=UserIdentifier(account_id="test", user_id="alice"),role=Role.ROOT,)-- with pytest.raises(OpenVikingError, match="recursive=false"):+ with pytest.raises(InvalidArgumentError, match="recursive=false.*namespace"):await ReindexExecutor().execute(uri="viking://user/alice",- mode="vectors_only",- wait=True,+ mode="semantic_and_vectors",recursive=False,+ wait=True,ctx=ctx,)@pytest.mark.asyncio-async def test_reindex_executor_discards_invalid_tags_before_creating_background_task():- from openviking.service.reindex_executor import ReindexExecutor+async def test_reindex_memory_cancellation_waits_for_inflight_reads(monkeypatch):+ import asyncio- ingest_options = ReindexExecutor._resolve_ingest_options(- mode="vectors_only",- tags=["invalid", "team=search"],- tag_mode="replace",- )+ from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters- assert ingest_options is not None- assert ingest_options.search_tags == ["team=search"]+ root = "viking://user/bob/memories/events"+ files = [root + "/a.md", root + "/b.md"]+ started, cleaning = asyncio.Queue(), asyncio.Queue()+ finish = asyncio.Event()+ settled = set()++ async def read(uri, **kwargs):+ started.put_nowait(uri)+ try:+ await asyncio.Future()+ finally:+ cleaning.put_nowait(uri)+ await finish.wait()+ settled.add(uri)++ fs = SimpleNamespace()+ db = SimpleNamespace(strict_delete=AsyncMock())+ monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ "openviking.service.reindex_executor.get_service",+ lambda: SimpleNamespace(vikingdb_manager=db),+ )+ executor = ReindexExecutor()+ executor._read_memory_body = read+ executor._reindex_memory_directory_chain = AsyncMock(return_value={})+ executor._upsert_context = AsyncMock()+ monkeypatch.setattr(executor, "_effective_file_vectorization_concurrency", lambda: 2)+ ctx = RequestContext(user=UserIdentifier("test", "bob"), role=Role.USER)+ operation = asyncio.create_task(+ executor._reindex_memory_vectors(+ uri=root,+ ctx=ctx,+ counters=_ReindexCounters(),+ root_is_dir=True,+ formal_entries=[{"uri": uri, "isDir": False} for uri in files],+ vector_inventory={"gone": {"uri": root + "/gone.md", "level": 2}},+ )+ )+ try:+ await asyncio.wait_for(started.get(), 1)+ await asyncio.wait_for(started.get(), 1)+ operation.cancel()+ await asyncio.wait_for(cleaning.get(), 1)+ await asyncio.wait_for(cleaning.get(), 1)+ operation.cancel()+ await asyncio.sleep(0)+ assert not operation.done()+ finish.set()+ with pytest.raises(asyncio.CancelledError):+ await asyncio.wait_for(operation, 1)+ assert settled == set(files)+ db.strict_delete.assert_not_awaited()+ executor._upsert_context.assert_not_awaited()+ finally:+ finish.set()+ if not operation.done():+ operation.cancel()+ await asyncio.gather(operation, return_exceptions=True)@pytest.mark.asyncio-async def test_reindex_file_target_uses_exact_lock(monkeypatch):- from types import SimpleNamespace-- from openviking.service.reindex_executor import ReindexExecutor-- calls = []-- class FakeAGFS:- async def pathlock_acquire_exact(self, path):- calls.append(("exact", path))- return {"id": "lease-1"}-- async def pathlock_acquire_tree(self, path):- calls.append(("tree", path))- raise AssertionError("file target must not acquire a tree lock")-- async def pathlock_as_borrowed(self, lease):- return lease+async def test_reindex_memory_l2_strips_memory_fields_from_abstract(monkeypatch):+ from openviking.service.reindex_executor import (+ ReindexExecutor,+ _ReindexCounters,+ _SourceRead,+ )- async def pathlock_release(self, lease):- calls.append(("release", lease["id"]))+ class FakeVikingFS:+ async def exists(self, uri, ctx=None):+ return True- class FakeFS:- _async_agfs = FakeAGFS()+ async def stat(self, uri, ctx=None, skip_count=True):+ return {"isDir": False}- def _uri_to_path(self, uri, ctx):- return "/local/default/resources/demo.md"+ seen = {}+ raw_body = (+ "User has a preference for watermelon, as mentioned in the conversation: "+ '\'我爱吃西瓜\'. <!-- MEMORY_FIELDS { "user": "user", "topic": "food_preference" } -->'+ )- async def stat(self, uri, ctx, skip_count=True):- return {"isDir": False}+ async def fake_read_memory_body(self, uri, *, ctx):+ return _SourceRead(exists=True, text=raw_body)- executor = ReindexExecutor()+ async def fake_upsert_context(self, **kwargs):+ seen[kwargs["uri"]] = kwargs- async def fake_reindex_resource(*, uri, mode, run):- return None+ monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())+ monkeypatch.setattr(ReindexExecutor, "_read_memory_body", fake_read_memory_body)+ monkeypatch.setattr(ReindexExecutor, "_upsert_context", fake_upsert_context)- monkeypatch.setattr(executor, "_reindex_resource", fake_reindex_resource)- monkeypatch.setattr(- "openviking.service.reindex_executor.get_service",- lambda: SimpleNamespace(- viking_fs=FakeFS(),- vikingdb_manager=SimpleNamespace(has_queue_manager=True),- ),- )+ service = ReindexExecutor()+ counters = _ReindexCounters()ctx = RequestContext(user=UserIdentifier(account_id="test", user_id="alice"),role=Role.ROOT,)- result = await executor._run(- uri="viking://resources/demo.md",- object_type="resource",- mode="vectors_only",+ await service._reindex_memory_vectors(+ uri="viking://user/default/memories/preferences/food_preference.md",+ counters=counters,ctx=ctx,+ vector_inventory={},)- assert result["status"] == "completed"- assert calls == [- ("exact", "/local/default/resources/demo.md"),- ("release", "lease-1"),- ]+ assert (+ seen["viking://user/default/memories/preferences/food_preference.md"]["abstract"]+ == "User has a preference for watermelon, as mentioned in the conversation: '我爱吃西瓜'."+ )+ assert (+ seen["viking://user/default/memories/preferences/food_preference.md"]["vector_text"]+ == raw_body+ )@pytest.mark.asyncio-async def test_reindex_executor_passes_tags_to_background_run(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor- from openviking.storage.queuefs.reindex_msg import ReindexMsg-- seen = {"enqueued": []}-- class FakeTask:- task_id = "task-1"+async def test_reindex_memory_vectors_preserves_live_sources_and_cleans_orphans(monkeypatch):+ from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters+ from openviking_cli.exceptions import NotFoundError++ root = "viking://user/bob/memories/preferences"+ nested = root + "/user"+ hidden, broken, literal = nested + "/.note.md", nested + "/broken.md", root + "/real#chunk_0000"+ sources = {+ root + "/.abstract.md": "root abstract",+ root + "/.overview.md": OSError("overview unavailable"),+ nested + "/.overview.md": "nested overview",+ hidden: "likes spicy food",+ broken: OSError("body unavailable"),+ literal: "real memory",+ }+ records = {+ "hidden": (hidden, 2),+ "broken": (broken, 2),+ "literal": (literal, 2),+ "root-l0": (root, 0),+ "root-l1": (root, 1),+ "missing-sidecar": (nested, 0),+ "gone-a": (root + "/gone.md", 2),+ "gone-b": (root + "/gone.md", 2),+ "legacy-chunk": (hidden + "#chunk_0000", 2),+ }+ inventory = {+ key: {"id": key, "uri": uri, "level": level} for key, (uri, level) in records.items()+ }+ stored = dict(inventory)++ async def read(uri, **kwargs):+ if uri not in sources:+ raise NotFoundError(uri)+ if isinstance(sources[uri], Exception):+ raise sources[uri]+ return sources[uri]++ async def tree(uri, *, show_all_hidden, node_limit, level_limit, **kwargs):+ assert uri == root and node_limit is None and level_limit is None+ return [{"uri": nested, "isDir": True}] + [+ {"uri": path, "isDir": False}+ for path in sources+ if show_all_hidden or not path.rsplit("/", 1)[-1].startswith(".")+ ]++ async def delete(ids, **kwargs):+ for key in ids:+ del stored[key]+ return len(ids)- class FakeTracker:- async def create_if_no_running(self, *args, **kwargs):- return FakeTask()+ fs = SimpleNamespace(+ exists=AsyncMock(side_effect=lambda uri, **kw: uri == root or uri in sources),+ stat=AsyncMock(return_value={"isDir": True}),+ tree=tree,+ read_file=read,+ )+ db = SimpleNamespace(+ get_reindex_inventory_under_uri=AsyncMock(return_value=inventory),+ strict_delete=AsyncMock(side_effect=delete),+ enqueue_embedding_msg=AsyncMock(return_value=True),+ )+ monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ "openviking.service.reindex_executor.get_service",+ lambda: SimpleNamespace(vikingdb_manager=db),+ )+ executor = ReindexExecutor()+ monkeypatch.setattr(executor, "_effective_file_vectorization_concurrency", lambda: 2)+ counters = _ReindexCounters()+ ctx = RequestContext(user=UserIdentifier("test", "admin"), role=Role.ADMIN)- patch_task = AsyncMock()- fail = AsyncMock()+ await executor._reindex_memory(+ uri=root, mode="vectors_only", run=_make_reindex_run(ctx, counters)+ )- class FakeAGFS:- async def pathlock_acquire_tree(self, path):- seen["path"] = path- return {"lease": "root"}+ queued = [call.args[0].context_data for call in db.enqueue_embedding_msg.await_args_list]+ assert {(record["uri"], record["level"]) for record in queued} == {+ (root, 0),+ (nested, 1),+ (hidden, 2),+ (literal, 2),+ }+ assert all(record["owner_user_id"] == "bob" for record in queued)+ assert set(stored) == {"hidden", "broken", "literal", "root-l0", "root-l1"}+ assert counters.deleted_records == 4+ assert counters.rebuilt_records == 4+ assert counters.failed_records == 2+ assert any("overview unavailable" in warning for warning in counters.warnings)+ assert any("body unavailable" in warning for warning in counters.warnings)- async def pathlock_to_handoff(self, lease):- assert lease == {"lease": "root"}- return {"handoff": "root"}- async def pathlock_handoff(self, lease):- assert lease == {"lease": "root"}+async def test_reindex_user_namespace_partitions_memory_skill_and_resource(monkeypatch):+ from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters+ from openviking_cli.exceptions import NotFoundError++ root = "viking://user/bob"+ directories = {+ root: [+ "resources",+ "skills",+ "memories",+ "peers",+ "notes",+ "sessions",+ "privacy",+ "settings",+ ],+ root + "/skills": ["live"],+ root + "/memories": [],+ root + "/peers": [],+ }+ children = {+ uri: [{"uri": uri + "/" + name, "isDir": True} for name in names]+ for uri, names in directories.items()+ }+ children[root].append({"uri": root + "/persona.md", "isDir": False})+ hidden = root + "/memories/.note.md"+ children[root + "/memories"].append({"uri": hidden, "isDir": False})+ records = {+ "skill": {"uri": root + "/skills/gone/SKILL.md", "level": 2},+ "peer": {"uri": root + "/peers/gone/memories/events/a.md", "level": 2},+ }- async def pathlock_release(self, lease):- raise AssertionError(f"unexpected release: {lease}")+ async def stat(uri, **kwargs):+ if uri not in directories and uri != root + "/resources":+ raise NotFoundError(uri)+ return {"isDir": True}++ async def listing(uri, *, offset, node_limit, show_all_hidden, **kwargs):+ entries = [+ entry+ for entry in children[uri]+ if entry["isDir"]+ or show_all_hidden+ or not entry["uri"].rsplit("/", 1)[-1].startswith(".")+ ]+ return entries[offset : offset + node_limit]++ async def inventory(uri, *, recursive, **kwargs):+ return {+ key: dict(record, id=key)+ for key, record in records.items()+ if recursive and record["uri"].startswith(uri + "/")+ }- class FakeVikingFS:- _async_agfs = FakeAGFS()+ fs = SimpleNamespace(stat=stat, ls=AsyncMock(side_effect=listing))+ db = SimpleNamespace(get_reindex_inventory_under_uri=AsyncMock(side_effect=inventory))+ monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ "openviking.service.reindex_executor.get_service",+ lambda: SimpleNamespace(vikingdb_manager=db),+ )+ executor = ReindexExecutor()+ executor._reindex_rfv = AsyncMock()+ executor._reindex_memory = AsyncMock()+ ctx = RequestContext(user=UserIdentifier("test", "bob"), role=Role.USER)- async def stat(self, uri, *, ctx, skip_count):- assert uri == "viking://resources/demo"- assert skip_count is True- return {"isDir": True}+ await executor._reindex_user_namespace(+ uri=root, mode="vectors_only", run=_make_reindex_run(ctx, _ReindexCounters())+ )- def _uri_to_path(self, uri, *, ctx):- return "/resources/demo"+ scopes = {call.kwargs["uri"]: call.kwargs for call in executor._reindex_rfv.await_args_list}+ assert set(scopes) == {+ root + suffix+ for suffix in ("/resources", "/skills/live", "/skills/gone", "/notes", "/persona.md")+ }+ assert scopes[root + "/skills/gone"]["root_exists"] is False+ assert set(scopes[root + "/skills/gone"]["vector_inventory"]) == {"skill"}+ memories = {+ call.kwargs["uri"]: call.kwargs for call in executor._reindex_memory.await_args_list+ }+ assert set(memories) == {root + "/memories", root + "/peers/gone/memories"}+ assert memories[root + "/memories"]["formal_entries"] == [{"uri": hidden, "isDir": False}]+ assert memories[root + "/peers/gone/memories"]["root_exists"] is False+ assert set(memories[root + "/peers/gone/memories"]["vector_inventory"]) == {"peer"}+ assert not any(+ part in call.args[0]+ for call in fs.ls.await_args_list+ for part in ("/sessions", "/privacy", "/settings")+ )- class FakeQueueManager:- REINDEX = "Reindex"- async def enqueue(self, queue_name, data):- seen["enqueued"].append((queue_name, data))- return "message-1"+async def test_reindex_global_namespace_partitions_user_and_resources(monkeypatch):+ from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters- executor = ReindexExecutor()- monkeypatch.setattr(- "openviking.service.reindex_executor.get_task_tracker",- lambda: FakeTracker(),+ children = {+ "viking://agent/skills": [{"uri": "viking://agent/skills/demo", "isDir": True}],+ "viking://user": [{"uri": "viking://user/bob", "isDir": True}],+ }+ fs = SimpleNamespace(+ stat=AsyncMock(return_value={"isDir": True}),+ ls=AsyncMock(side_effect=lambda uri, **kw: children[uri]),)+ db = SimpleNamespace(get_reindex_inventory_under_uri=AsyncMock(return_value={}))+ monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: fs)monkeypatch.setattr("openviking.service.reindex_executor.get_service",- lambda: SimpleNamespace(viking_fs=FakeVikingFS()),- )- monkeypatch.setattr("openviking.storage.queuefs.get_queue_manager", lambda: FakeQueueManager())- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- result = await executor.execute(- uri="viking://resources/demo",- mode="vectors_only",- wait=False,- tags=["Team=Search"],- tag_mode="append",- ctx=ctx,- )-- assert result["status"] == "accepted"- assert seen["path"] == "/resources/demo"- assert len(seen["enqueued"]) == 1- queue_name, payload = seen["enqueued"][0]- assert queue_name == "Reindex"- message = ReindexMsg.from_dict(payload)- assert message.task_id == "task-1"- assert message.tags == ["Team=Search"]- assert message.tag_mode == "append"- assert message.lock_handoff == {"handoff": "root"}- assert message.actor_peer_id == ctx.actor_peer_id- assert message.bypass_acl is ctx.bypass_acl--[email protected]("failure", ["stat", "lock"])[email protected]("has_work", [False, True])-async def test_reindex_submit_failure_only_fails_tasks_without_accepted_work(- monkeypatch, failure, has_work-):- from openviking.service.reindex_executor import ReindexExecutor-- class FakeTask:- task_id = "task-1"-- tracker = SimpleNamespace(- create_if_no_running=AsyncMock(return_value=FakeTask()),- has_work=AsyncMock(return_value=has_work),- fail=AsyncMock(),- )-- class FakeAGFS:- async def pathlock_acquire_tree(self, _path):- if failure == "lock":- raise RuntimeError("root lock is busy")- raise AssertionError("lock must not be acquired after stat failure")-- class FakeVikingFS:- _async_agfs = FakeAGFS()-- async def stat(self, _uri, *, ctx, skip_count):- del ctx, skip_count- if failure == "stat":- raise RuntimeError("root stat is unavailable")- return {"isDir": True}-- def _uri_to_path(self, _uri, *, ctx):- del ctx- return "/resources/demo"-- monkeypatch.setattr("openviking.service.reindex_executor.get_task_tracker", lambda: tracker)- monkeypatch.setattr(- "openviking.service.reindex_executor.get_service",- lambda: SimpleNamespace(viking_fs=FakeVikingFS()),- )- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- with pytest.raises(RuntimeError, match=f"root {failure}"):- await ReindexExecutor().execute(- uri="viking://resources/demo",- mode="vectors_only",- wait=False,- ctx=ctx,- )-- if has_work:- tracker.fail.assert_not_awaited()- else:- tracker.fail.assert_awaited_once_with(- "task-1",- "Failed to enqueue reindex processing",- account_id="test",- user_id="alice",- )---async def test_reindex_resource_passes_run_tags_to_vector_rebuild(monkeypatch):- from openviking.service.reindex_executor import (- ReindexExecutor,- _ReindexCounters,- _ReindexRunContext,- )- from openviking.utils.ingest_options import IngestOptions-- seen = {}-- async def fake_reindex_rfv(self, **kwargs):- seen.update(kwargs)-- monkeypatch.setattr(- ReindexExecutor,- "_reindex_rfv",- fake_reindex_rfv,- )- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- ingest_options = IngestOptions.from_search_tags(["team=search"], mode="replace")-- await ReindexExecutor()._reindex_resource(- uri="viking://resources/demo",- mode="vectors_only",- run=_ReindexRunContext(- ctx=ctx,- counters=_ReindexCounters(),- ingest_options=ingest_options,- ),- )-- assert seen["run"].ingest_options is ingest_options--[email protected]("mode", ["vectors_only", "semantic_and_vectors"])-async def test_reindex_resource_routes_both_modes_through_rfv_plan(monkeypatch, mode):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- seen = {}-- async def fake_reindex_rfv(self, **kwargs):- seen.update(kwargs)-- monkeypatch.setattr(ReindexExecutor, "_reindex_rfv", fake_reindex_rfv)- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- run = _make_reindex_run(ctx, _ReindexCounters())- run.force = True-- await ReindexExecutor()._reindex_resource(- uri="viking://resources/demo",- mode=mode,- recursive=False,- run=run,- )-- assert seen == {- "uri": "viking://resources/demo",- "mode": mode,- "context_type": "resource",- "recursive": False,- "run": run,- }--[email protected]("mode", ["vectors_only", "semantic_and_vectors"])-async def test_reindex_skill_routes_both_modes_through_rfv_plan(monkeypatch, mode):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- seen = {}-- async def fake_reindex_rfv(self, **kwargs):- seen.update(kwargs)-- monkeypatch.setattr(ReindexExecutor, "_reindex_rfv", fake_reindex_rfv)- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- run = _make_reindex_run(ctx, _ReindexCounters())-- await ReindexExecutor()._reindex_skill(- uri="viking://user/alice/skills/demo",- mode=mode,- recursive=False,- run=run,- )-- assert seen == {- "uri": "viking://user/alice/skills/demo",- "mode": mode,- "context_type": "skill",- "recursive": False,- "run": run,- }---async def test_reindex_rfv_skips_complete_file_without_enqueue_or_second_read(monkeypatch):- from unittest.mock import AsyncMock-- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters- from openviking.utils.content_hash import content_md5-- uri = "viking://resources/demo.md"- body = b"same body"-- class FakeFS:- stat = AsyncMock(return_value={"isDir": False})- read_file_bytes = AsyncMock(return_value=body)-- class FakeDB:- async def get_incremental_inventory_under_uri(- self, target, *, ctx, output_fields, recursive=True, include_direct_children=False- ):- assert target == uri- assert recursive is False- assert include_direct_children is False- return {- "file-l2": {- "id": "file-l2",- "uri": uri,- "level": 2,- "md5": content_md5(body),- }- }-- resource_processor = SimpleNamespace(_enqueue_index_actions=AsyncMock())- service = SimpleNamespace(- viking_fs=FakeFS(),- vikingdb_manager=FakeDB(),- _resource_processor=resource_processor,- )- monkeypatch.setattr("openviking.service.reindex_executor.get_service", lambda: service)- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- counters = _ReindexCounters()-- await ReindexExecutor()._reindex_rfv(- uri=uri,- mode="vectors_only",- context_type="resource",- recursive=False,- run=_make_reindex_run(ctx, counters),- )-- resource_processor._enqueue_index_actions.assert_not_awaited()- service.viking_fs.read_file_bytes.assert_awaited_once_with(uri, ctx=ctx)- assert counters.scanned_records == 1- assert counters.rebuilt_records == 0---async def test_reindex_rfv_deletes_orphans_synchronously_with_owner_context(monkeypatch):- from unittest.mock import AsyncMock-- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- root = "viking://user/bob/resources/demo"-- class FakeFS:- stat = AsyncMock(return_value={"isDir": True})- tree = AsyncMock(return_value=[])-- class FakeDB:- strict_delete = AsyncMock(return_value=1)-- async def get_incremental_inventory_under_uri(- self, target, *, ctx, output_fields, recursive=True, include_direct_children=False- ):- assert include_direct_children is False- return {- "orphan-l2": {- "id": "orphan-l2",- "uri": f"{root}/old.md",- "level": 2,- "md5": "old",- }- }-- db = FakeDB()- service = SimpleNamespace(viking_fs=FakeFS(), vikingdb_manager=db)- monkeypatch.setattr("openviking.service.reindex_executor.get_service", lambda: service)- admin_ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="admin"),- role=Role.ROOT,- )- counters = _ReindexCounters()-- await ReindexExecutor()._reindex_rfv(- uri=root,- mode="vectors_only",- context_type="resource",- recursive=True,- run=_make_reindex_run(admin_ctx, counters),- )-- db.strict_delete.assert_awaited_once()- assert db.strict_delete.await_args.kwargs["ctx"].user.user_id == "bob"- assert counters.deleted_records == 1---async def test_reindex_rfv_enqueues_semantic_plan_without_snapshot_source_bytes(monkeypatch):- import json-- import openviking.service.reindex_executor as reindex_mod- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters- from openviking.storage.context_update_plan import (- ContextUpdatePlan,- SemanticPlan,- SemanticTreeEntry,- SemanticTreeSnapshot,- )-- root = "viking://resources/demo"- lease = {"id": "reindex-lease"}- plan = SemanticPlan(- root,- "resource",- SemanticTreeSnapshot((SemanticTreeEntry("", "directory", "unchanged", "aggregate"),)),- )- snapshot = SimpleNamespace(- formal=SimpleNamespace(entries={"": object()}),- source_contents={(root, 2): b"reindex source must stay in memory"},- source_raw_contents={(root, 2): b"raw source must stay in memory"},- source_metadata={"uri": "s3://source"},- )- handoff = {"owner_id": "semantic-worker", "covered_paths": []}- calls = []-- class PathLock:- async def pathlock_to_handoff(self, current):- assert current == lease- calls.append("to_handoff")- return handoff-- async def pathlock_handoff(self, current):- assert current == lease- calls.append("handoff")-- class Queue:- async def enqueue(self, msg):- calls.append("enqueue")- self.message = msg- return "queued"-- queue = Queue()- queue_manager = SimpleNamespace(SEMANTIC="semantic", get_queue=lambda *args, **kwargs: queue)- service = SimpleNamespace(- viking_fs=SimpleNamespace(_async_agfs=PathLock()),- vikingdb_manager=SimpleNamespace(uses_content_field=False),- )- monkeypatch.setattr(reindex_mod, "get_service", lambda: service)- monkeypatch.setattr(- "openviking.storage.resource_rfv.build_rfv_snapshot",- AsyncMock(return_value=snapshot),- )- monkeypatch.setattr(- "openviking.storage.context_update_plan.build_rfv_context_update_plan",- lambda **kwargs: (None, ContextUpdatePlan(root, "resource", semantic_plan=plan)),- )- monkeypatch.setattr("openviking.storage.queuefs.get_queue_manager", lambda: queue_manager)-- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- run = _make_reindex_run(ctx, _ReindexCounters())- run.lease = lease-- await ReindexExecutor()._reindex_rfv(- uri=root,- mode="semantic_and_vectors",- context_type="resource",- recursive=False,- run=run,- )-- assert calls == ["to_handoff", "handoff", "enqueue"]- assert run.lease is None- assert run.lock is None- assert queue.message.plan == plan- assert queue.message.propagate_to_parent is False- assert queue.message.generation_trigger == "reindex"- assert queue.message.lock_handoff == handoff- assert b"reindex source must stay in memory" not in json.dumps(queue.message.to_dict()).encode()---async def test_single_file_vectors_only_reindex_reads_parent_overview_only_for_rebuild(monkeypatch):- import openviking.service.reindex_executor as reindex_mod- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters- from openviking.storage.resource_rfv import RFVEntry, RFVFormalSnapshot, RFVSnapshot- from openviking.storage.resource_rnfv import RequestIntent, VectorIndexSnapshot-- file_uri = "viking://resources/demo/a.md"- parent_uri = "viking://resources/demo"- snapshot = RFVSnapshot(- request=RequestIntent(file_uri, "vectors_only", force=True),- formal=RFVFormalSnapshot({"": RFVEntry(file_uri, "", False, {2: "a-md5"})}),- vectors=VectorIndexSnapshot({}, frozenset({"id", "uri", "level", "md5", "abstract"})),- source_contents={(file_uri, 2): b"current body"},- )- fs = SimpleNamespace(- read_file_bytes=AsyncMock(return_value=b"### a.md\nOverview summary."),- )- service = SimpleNamespace(- viking_fs=fs,- vikingdb_manager=SimpleNamespace(uses_content_field=False),- _resource_processor=None,- )- enqueued = AsyncMock(return_value=1)- monkeypatch.setattr(reindex_mod, "get_service", lambda: service)- monkeypatch.setattr(- "openviking.storage.resource_rfv.build_rfv_snapshot",- AsyncMock(return_value=snapshot),- )- monkeypatch.setattr(- "openviking.utils.resource_processor.ResourceProcessor._enqueue_index_actions",- enqueued,- )- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- run = _make_reindex_run(ctx, _ReindexCounters())- run.root_is_dir = False-- await ReindexExecutor()._reindex_rfv(- uri=file_uri,- mode="vectors_only",- context_type="resource",- recursive=False,- run=run,- )-- fs.read_file_bytes.assert_awaited_once_with(f"{parent_uri}/.overview.md", ctx=ctx)- actions = enqueued.await_args.args[0]- assert len(actions) == 1- assert actions[0].summary == "Overview summary."---async def test_single_file_vectors_only_noop_does_not_read_parent_overview(monkeypatch):- import openviking.service.reindex_executor as reindex_mod- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters- from openviking.storage.resource_rfv import RFVEntry, RFVFormalSnapshot, RFVSnapshot- from openviking.storage.resource_rnfv import (- RequestIntent,- VectorIndexSnapshot,- VectorRecordSnapshot,- )-- file_uri = "viking://resources/demo/a.md"- snapshot = RFVSnapshot(- request=RequestIntent(file_uri, "vectors_only"),- formal=RFVFormalSnapshot({"": RFVEntry(file_uri, "", False, {2: "a-md5"})}),- vectors=VectorIndexSnapshot(- {- "a-l2": VectorRecordSnapshot(- "a-l2", file_uri, "", 2, {"md5": "a-md5", "abstract": "summary"}- )- },- frozenset({"id", "uri", "level", "md5", "abstract"}),- ),- )- fs = SimpleNamespace(read_file_bytes=AsyncMock())- service = SimpleNamespace(- viking_fs=fs,- vikingdb_manager=SimpleNamespace(uses_content_field=False),- _resource_processor=None,- )- monkeypatch.setattr(reindex_mod, "get_service", lambda: service)- monkeypatch.setattr(- "openviking.storage.resource_rfv.build_rfv_snapshot",- AsyncMock(return_value=snapshot),- )- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- run = _make_reindex_run(ctx, _ReindexCounters())- run.root_is_dir = False-- await ReindexExecutor()._reindex_rfv(- uri=file_uri,- mode="vectors_only",- context_type="resource",- recursive=False,- run=run,- )-- fs.read_file_bytes.assert_not_awaited()---def test_reindex_merges_persisted_semantic_plan_stats_into_response_counters():- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters- from openviking.storage.queuefs.semantic_executor import SemanticTreeStats-- counters = _ReindexCounters(rebuilt_records=2)- ReindexExecutor._apply_semantic_tree_stats(- counters,- SemanticTreeStats(indexed_records=3, failures=["summary failed"]),- )-- assert counters.rebuilt_records == 5- assert counters.failed_records == 1- assert counters.warnings == ["summary failed"]---async def test_reindex_upsert_uses_uri_owner_for_user_scoped_records(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor-- captured = {}-- class FakeVikingDB:- async def enqueue_embedding_msg(self, msg):- captured["msg"] = msg- return True-- fake_service = type("Svc", (), {"vikingdb_manager": FakeVikingDB()})()- monkeypatch.setattr("openviking.service.reindex_executor.get_service", lambda: fake_service)-- service = ReindexExecutor()- ctx = RequestContext(- user=UserIdentifier(account_id="acct", user_id="admin"),- role=Role.ROOT,- )-- await service._upsert_context(- uri="viking://user/bob/memories/preferences/theme.md",- parent_uri="viking://user/bob/memories/preferences",- abstract="theme",- vector_text="theme",- is_leaf=True,- context_type="memory",- level=ContextLevel.DETAIL,- ctx=ctx,- )-- data = captured["msg"].context_data- assert data["user"]["user_id"] == "bob"- assert data["owner_user_id"] == "bob"- assert data["owner_space"] == "bob"---async def test_reindex_semantic_processor_uses_uri_owner_for_user_scoped_records(- monkeypatch, semantic_config-):- from openviking.service.reindex_executor import ReindexExecutor-- captured = {}-- class FakeSemanticProcessor:- def __init__(self, **_kwargs):- pass-- async def on_dequeue(self, payload, lock=None):- captured["payload"] = payload- captured["lock"] = lock- return ProcessResult.success()-- monkeypatch.setattr(- "openviking.service.reindex_executor.SemanticProcessor",- FakeSemanticProcessor,- )-- service = ReindexExecutor(vlm_resolver=semantic_config)- ctx = RequestContext(- user=UserIdentifier(account_id="acct", user_id="admin"),- role=Role.ROOT,- )-- await service._run_semantic_processor(- uri="viking://user/bob/memories/preferences",- context_type="memory",- ctx=ctx,- )-- data = captured["payload"]["data"]- assert '"user_id": "bob"' in data- assert '"peer_id": "bob"' in data- assert '"account_id": "acct"' in data---async def test_reindex_memory_supports_semantic_and_vectors(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor-- service = ReindexExecutor()- service._validate_mode("memory", "semantic_and_vectors")---async def test_reindex_semantic_processor_uses_configured_vlm_concurrency(- monkeypatch, semantic_config-):- import openviking.service.reindex_executor as reindex_mod-- seen = {}-- class FakeSemanticProcessor:- def __init__(self, max_concurrent_llm=64, **kwargs):- del kwargs- seen["max_concurrent_llm"] = max_concurrent_llm-- async def on_dequeue(self, data, lock=None):- seen["data"] = data- return ProcessResult.success()-- monkeypatch.setattr(reindex_mod, "SemanticProcessor", FakeSemanticProcessor)-- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- await reindex_mod.ReindexExecutor(vlm_resolver=semantic_config)._run_semantic_processor(- uri="viking://resources/demo",- context_type="resource",- ctx=ctx,- )-- assert seen["max_concurrent_llm"] == 2---async def test_reindex_memory_semantic_and_vectors_rebuilds_full_subtree(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- seen = {"semantic": [], "vectors": []}-- async def fake_run_semantic_processor(self, *, uri, context_type, ctx, lock=None):- seen["semantic"].append((uri, context_type))-- async def fake_reindex_memory_vectors(self, *, uri, counters, ctx):- seen["vectors"].append(uri)-- class FakeVikingFS:- async def stat(self, uri, ctx=None, skip_count=True):- return {"isDir": True}-- monkeypatch.setattr(ReindexExecutor, "_run_semantic_processor", fake_run_semantic_processor)- monkeypatch.setattr(ReindexExecutor, "_reindex_memory_vectors", fake_reindex_memory_vectors)- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())-- service = ReindexExecutor()- counters = _ReindexCounters()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await service._reindex_memory(- uri="viking://user/default/memories",- mode="semantic_and_vectors",- run=_make_reindex_run(ctx, counters),- )-- assert seen["semantic"] == [("viking://user/default/memories", "memory")]- assert seen["vectors"] == ["viking://user/default/memories"]--[email protected]("wait", [True, False])- "uri",- [- "viking://",- "viking://user",- "viking://user/alice",- "viking://user/alice/skills",- "viking://agent/skills",- ],-)-async def test_reindex_rejects_non_recursive_namespace_before_starting_work(monkeypatch, uri, wait):- from openviking.service.reindex_executor import ReindexExecutor- from openviking_cli.exceptions import InvalidArgumentError-- def unexpected_tracker():- pytest.fail("unsupported non-recursive reindex must not start task tracking")-- monkeypatch.setattr("openviking.service.reindex_executor.get_task_tracker", unexpected_tracker)- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role(Role.ROOT),- )- with pytest.raises(InvalidArgumentError, match="recursive=false.*namespace"):- await ReindexExecutor().execute(- uri=uri,- mode="semantic_and_vectors",- recursive=False,- wait=wait,- ctx=ctx,- )---async def test_reindex_resource_non_recursive_limits_semantics_and_vectors(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- seen = {}-- async def fake_reindex_rfv(self, **kwargs):- seen.update(kwargs)-- monkeypatch.setattr(ReindexExecutor, "_reindex_rfv", fake_reindex_rfv)-- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- await ReindexExecutor()._reindex_resource(- uri="viking://resources/demo",- mode="semantic_and_vectors",- recursive=False,- run=_make_reindex_run(ctx, _ReindexCounters()),- )-- assert seen["recursive"] is False- assert seen["mode"] == "semantic_and_vectors"---async def test_reindex_memory_non_recursive_limits_semantics_and_vectors(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- seen = {}-- async def fake_run_semantic_processor(self, **kwargs):- seen["semantic"] = kwargs-- async def fake_reindex_memory_vectors(self, **kwargs):- seen["vectors"] = kwargs-- class FakeVikingFS:- async def stat(self, uri, ctx=None, skip_count=True):- return {"isDir": True}-- monkeypatch.setattr(ReindexExecutor, "_run_semantic_processor", fake_run_semantic_processor)- monkeypatch.setattr(ReindexExecutor, "_reindex_memory_vectors", fake_reindex_memory_vectors)- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())-- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- await ReindexExecutor()._reindex_memory(- uri="viking://user/alice/memories/preferences",- mode="semantic_and_vectors",- recursive=False,- run=_make_reindex_run(ctx, _ReindexCounters()),- )-- assert seen["semantic"]["recursive"] is False- assert seen["vectors"]["recursive"] is False---async def test_reindex_skill_non_recursive_limits_vectors_to_directory(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- seen = {}-- async def fake_reindex_rfv(self, **kwargs):- seen.update(kwargs)-- monkeypatch.setattr(ReindexExecutor, "_reindex_rfv", fake_reindex_rfv)-- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- await ReindexExecutor()._reindex_skill(- uri="viking://user/alice/skills/demo",- mode="semantic_and_vectors",- recursive=False,- run=_make_reindex_run(ctx, _ReindexCounters()),- )-- assert seen["uri"] == "viking://user/alice/skills/demo"- assert seen["recursive"] is False---async def test_reindex_skill_vectors_only_honors_non_recursive_flag(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- seen = {}-- async def fake_reindex_rfv(self, **kwargs):- seen.update(kwargs)-- monkeypatch.setattr(ReindexExecutor, "_reindex_rfv", fake_reindex_rfv)-- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- await ReindexExecutor()._reindex_skill(- uri="viking://user/alice/skills/demo",- mode="vectors_only",- recursive=False,- run=_make_reindex_run(ctx, _ReindexCounters()),- )-- assert seen["recursive"] is False---async def test_reindex_resource_vectors_only_honors_non_recursive(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- seen = {}-- async def fake_reindex_rfv(self, **kwargs):- seen.update(kwargs)-- monkeypatch.setattr(- ReindexExecutor,- "_reindex_rfv",- fake_reindex_rfv,- )- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await ReindexExecutor()._reindex_resource(- uri="viking://resources/demo",- mode="vectors_only",- recursive=False,- run=_make_reindex_run(ctx, _ReindexCounters()),- )-- assert seen["recursive"] is False---async def test_reindex_executor_infers_skill_supports_semantic_and_vectors():- from openviking.service.reindex_executor import ReindexExecutor-- service = ReindexExecutor()- service._validate_mode("skill", "semantic_and_vectors")- service._validate_mode("skill_namespace", "semantic_and_vectors")---async def test_reindex_executor_infers_resource_and_skill_container_scopes():- from openviking.service.reindex_executor import ReindexExecutor-- service = ReindexExecutor()-- assert service._infer_target_type("viking://resources") == "resource"- assert service._infer_target_type("viking://resources/demo.md") == "resource"- assert service._infer_target_type("viking://user/default/skills") == "skill_namespace"- assert service._infer_target_type("viking://user/default/skills/demo") == "skill"- with pytest.raises(OpenVikingError, match="Unsupported reindex URI"):- service._infer_target_type("viking://user/default/skills/demo/SKILL.md")---async def test_reindex_executor_infers_user_namespace_root():- from openviking.service.reindex_executor import ReindexExecutor-- service = ReindexExecutor()-- assert service._infer_target_type("viking://user/") == "user_namespace"- assert service._infer_target_type("viking://user/default") == "user_namespace"---async def test_reindex_executor_infers_shared_agent_content():- from openviking.service.reindex_executor import ReindexExecutor-- service = ReindexExecutor()-- assert service._infer_target_type("viking://agent/skills") == "skill_namespace"- assert service._infer_target_type("viking://agent/skills/demo") == "skill"- assert service._infer_target_type("viking://agent/workflows/daily.md") == "resource"- with pytest.raises(OpenVikingError, match="Unsupported reindex URI"):- service._infer_target_type("viking://agent/")---async def test_reindex_executor_infers_global_namespace_root():- from openviking.service.reindex_executor import ReindexExecutor-- service = ReindexExecutor()-- assert service._infer_target_type("viking://") == "global_namespace"---async def test_reindex_executor_does_not_treat_resource_named_memories_as_memory():- from openviking.core.namespace import classify_uri- from openviking.service.reindex_executor import ReindexExecutor-- service = ReindexExecutor()-- assert (- service._infer_target_type("viking://user/default/resources/memories/report.md")- == "resource"- )- assert not classify_uri("viking://user/default/resources/memories-report.md").is_memory- assert not classify_uri("viking://user/default/resources/memories-report.md").is_memory---async def test_reindex_executor_does_not_treat_skill_subdirectories_as_skill_roots():- from openviking.core.namespace import classify_uri-- assert classify_uri("viking://user/default/skills/my_skill").is_skill_root- assert not classify_uri("viking://user/default/skills/my_skill/assets").is_skill_root- assert not classify_uri("viking://user/default/resources/skills-report.md").is_skill---async def test_reindex_user_namespace_semantic_and_vectors_promotes_memory_mode(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- class FakeVikingFS:- async def tree(- self,- uri,- output="original",- show_all_hidden=True,- node_limit=1000,- level_limit=None,- ctx=None,- ):- return [- {"uri": "viking://user/default/memories", "isDir": True},- {"uri": "viking://user/default/resources", "isDir": True},- ]-- seen = {"memory_modes": [], "rfv_calls": []}-- async def fake_reindex_memory(self, *, uri, mode, run):- seen["memory_modes"].append((uri, mode))-- async def fake_reindex_rfv(self, **kwargs):- seen["rfv_calls"].append(kwargs)-- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())- monkeypatch.setattr(ReindexExecutor, "_reindex_memory", fake_reindex_memory)- monkeypatch.setattr(ReindexExecutor, "_reindex_rfv", fake_reindex_rfv)-- service = ReindexExecutor()- counters = _ReindexCounters()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await service._reindex_user_namespace(- uri="viking://user/default",- mode="semantic_and_vectors",- run=_make_reindex_run(ctx, counters),- )-- assert seen["memory_modes"] == [("viking://user/default/memories", "semantic_and_vectors")]- assert [call["uri"] for call in seen["rfv_calls"]] == ["viking://user/default/resources"]---async def test_reindex_user_namespace_semantic_and_vectors_does_not_reprocess_memory_as_resource(- monkeypatch,-):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- class FakeVikingFS:- async def tree(- self,- uri,- output="original",- show_all_hidden=True,- node_limit=1000,- level_limit=None,- ctx=None,- ):- return [- {"uri": "viking://user/default/memories", "isDir": True},- {"uri": "viking://user/default/memories/preferences", "isDir": True},- {"uri": "viking://user/default/resources", "isDir": True},- ]-- seen = {"memory_modes": [], "rfv_calls": []}-- async def fake_reindex_memory(self, *, uri, mode, run):- seen["memory_modes"].append((uri, mode))-- async def fake_reindex_rfv(self, **kwargs):- seen["rfv_calls"].append(kwargs)-- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())- monkeypatch.setattr(ReindexExecutor, "_reindex_memory", fake_reindex_memory)- monkeypatch.setattr(ReindexExecutor, "_reindex_rfv", fake_reindex_rfv)-- service = ReindexExecutor()- counters = _ReindexCounters()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await service._reindex_user_namespace(- uri="viking://user/default",- mode="semantic_and_vectors",- run=_make_reindex_run(ctx, counters),- )-- assert seen["memory_modes"] == [("viking://user/default/memories", "semantic_and_vectors")]- assert [call["uri"] for call in seen["rfv_calls"]] == ["viking://user/default/resources"]---async def test_reindex_user_namespace_semantic_and_vectors_skips_uncovered_root_files(- monkeypatch,-):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- class FakeVikingFS:- async def tree(- self,- uri,- output="original",- show_all_hidden=True,- node_limit=1000,- level_limit=None,- ctx=None,- ):- return [- {"uri": "viking://user/default/resources", "isDir": True},- {"uri": "viking://user/default/resources/doc.md", "isDir": False},- {"uri": "viking://user/default/profile.md", "isDir": False},- ]-- seen = {"rfv_calls": []}-- async def fake_reindex_rfv(self, **kwargs):- seen["rfv_calls"].append(kwargs)-- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())- monkeypatch.setattr(ReindexExecutor, "_reindex_rfv", fake_reindex_rfv)-- service = ReindexExecutor()- counters = _ReindexCounters()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await service._reindex_user_namespace(- uri="viking://user/default",- mode="semantic_and_vectors",- run=_make_reindex_run(ctx, counters),- )-- assert [call["uri"] for call in seen["rfv_calls"]] == ["viking://user/default/resources"]- assert counters.unsupported_records == 1- assert "viking://user/default/profile.md" in counters.warnings[0]---async def test_reindex_skill_namespace_reindexes_only_skill_roots(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- class FakeVikingFS:- async def tree(- self,- uri,- output="original",- show_all_hidden=True,- node_limit=1000,- level_limit=3,- ctx=None,- ):- assert node_limit is None- assert level_limit is None- return [- {"uri": "viking://user/default/skills/my_skill", "isDir": True},- {"uri": "viking://user/default/skills/my_skill/assets", "isDir": True},- {"uri": "viking://user/default/skills/my_skill/SKILL.md", "isDir": False},- ]-- seen = []-- async def fake_reindex_rfv(self, **kwargs):- seen.append(kwargs)-- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())- monkeypatch.setattr(ReindexExecutor, "_reindex_rfv", fake_reindex_rfv)-- service = ReindexExecutor()- counters = _ReindexCounters()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await service._reindex_skill_namespace(- uri="viking://user/default/skills",- mode="semantic_and_vectors",- run=_make_reindex_run(ctx, counters),- )-- assert [(call["uri"], call["context_type"]) for call in seen] == [- ("viking://user/default/skills/my_skill", "skill")- ]- assert {entry["uri"] for entry in seen[0]["formal_entries"]} == {- "viking://user/default/skills/my_skill/assets",- "viking://user/default/skills/my_skill/SKILL.md",- }---async def test_reindex_global_namespace_semantic_and_vectors_propagates_to_child_namespaces(- monkeypatch,-):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- class FakeVikingFS:- async def tree(- self,- uri,- output="original",- show_all_hidden=True,- node_limit=1000,- level_limit=None,- ctx=None,- ):- return [- {"uri": "viking://user/default", "isDir": True},- {"uri": "viking://user/default", "isDir": True},- {"uri": "viking://resources", "isDir": True},- ]-- seen = {"user_modes": [], "rfv_calls": []}-- async def fake_reindex_user_namespace(self, *, uri, mode, run):- seen["user_modes"].append((uri, mode))-- async def fake_reindex_rfv(self, **kwargs):- seen["rfv_calls"].append(kwargs)-- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())- monkeypatch.setattr(ReindexExecutor, "_reindex_user_namespace", fake_reindex_user_namespace)- monkeypatch.setattr(ReindexExecutor, "_reindex_rfv", fake_reindex_rfv)-- service = ReindexExecutor()- counters = _ReindexCounters()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await service._reindex_global_namespace(- uri="viking://",- mode="semantic_and_vectors",- run=_make_reindex_run(ctx, counters),- )-- assert seen["user_modes"] == [("viking://user/default", "semantic_and_vectors")]- assert [call["uri"] for call in seen["rfv_calls"]] == ["viking://resources"]---async def test_reindex_fetch_existing_record_uses_get_context_by_uri(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor-- class FakeVikingDB:- def __init__(self):- self.fetch_calls = []- self.lookup_calls = []-- async def fetch_by_uri(self, uri, *, ctx):- self.fetch_calls.append((uri, ctx))- return {"uri": uri, "level": 2, "abstract": "from-fetch"}-- async def get_context_by_uri(self, uri, owner_space=None, level=None, limit=1, *, ctx=None):- self.lookup_calls.append((uri, owner_space, level, limit, ctx))- return [{"uri": uri, "level": level, "abstract": "from-lookup"}]-- fake_service = type("Svc", (), {"vikingdb_manager": FakeVikingDB()})()- monkeypatch.setattr("openviking.service.reindex_executor.get_service", lambda: fake_service)-- service = ReindexExecutor()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- record = await service._fetch_existing_record(- uri="viking://resources/demo.txt",- level=2,- ctx=ctx,- )-- assert record["abstract"] == "from-lookup"- assert not fake_service.vikingdb_manager.fetch_calls- assert fake_service.vikingdb_manager.lookup_calls---async def test_reindex_upsert_context_omits_search_tags_without_ingest_options(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor-- captured = {}-- class FakeVikingDB:- async def enqueue_embedding_msg(self, msg):- captured["msg"] = msg- return True-- fake_service = type("Svc", (), {"vikingdb_manager": FakeVikingDB()})()- monkeypatch.setattr("openviking.service.reindex_executor.get_service", lambda: fake_service)-- class _FakeMsg:- def __init__(self):- self.telemetry_id = ""- self.id = "msg-1"- self.context_data = {}-- def fake_from_context(context):- captured["meta"] = dict(context.meta or {})- msg = _FakeMsg()- msg.context_data = context.to_dict()- return msg-- monkeypatch.setattr(- "openviking.service.reindex_executor.EmbeddingMsgConverter.from_context",- fake_from_context,- )-- service = ReindexExecutor()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await service._upsert_context(- uri="viking://resources/demo.txt",- parent_uri="viking://resources",- abstract="new abstract",- vector_text="new vector text",- is_leaf=True,- context_type="resource",- level=ContextLevel.DETAIL,- ctx=ctx,- md5="final-md5",- )-- assert "search_tags" not in captured["meta"]- assert "search_tags" not in captured["msg"].context_data- assert captured["msg"].context_data["md5"] == "final-md5"---async def test_reindex_semantic_processor_runs_with_skip_vectorization(- monkeypatch, semantic_config-):- from openviking.service.reindex_executor import ReindexExecutor-- seen = {}-- class FakeSemanticProcessor:- def __init__(self, **_kwargs):- pass-- async def on_dequeue(self, payload, lock=None):- seen["payload"] = payload- return ProcessResult.success()-- monkeypatch.setattr(- "openviking.service.reindex_executor.SemanticProcessor",- FakeSemanticProcessor,- )-- service = ReindexExecutor(vlm_resolver=semantic_config)- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await service._run_semantic_processor(- uri="viking://resources/demo",- context_type="resource",- ctx=ctx,- )-- import json-- msg = json.loads(seen["payload"]["data"])- assert msg["skip_vectorization"] is True- assert msg["recursive"] is True- assert msg["use_hierarchical_aggregation"] is False- assert msg["propagate_to_parent"] is True---async def test_reindex_semantic_processor_passes_non_recursive_message(- monkeypatch, semantic_config-):- from openviking.service.reindex_executor import ReindexExecutor-- seen = {}-- class FakeSemanticProcessor:- def __init__(self, **_kwargs):- pass-- async def on_dequeue(self, payload, lock=None):- seen["payload"] = payload- return ProcessResult.success()-- monkeypatch.setattr(- "openviking.service.reindex_executor.SemanticProcessor",- FakeSemanticProcessor,- )- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- await ReindexExecutor(vlm_resolver=semantic_config)._run_semantic_processor(- uri="viking://resources/demo",- context_type="resource",- ctx=ctx,- recursive=False,- )-- import json-- msg = json.loads(seen["payload"]["data"])- assert msg["recursive"] is False- assert msg["use_hierarchical_aggregation"] is False- assert msg["propagate_to_parent"] is False---async def test_reindex_semantic_processor_sets_memory_aggregation_policy(- monkeypatch, semantic_config-):- from openviking.service.reindex_executor import ReindexExecutor-- seen = {}-- class FakeSemanticProcessor:- def __init__(self, **_kwargs):- pass-- async def on_dequeue(self, payload, lock=None):- seen["payload"] = payload- return ProcessResult.success()-- monkeypatch.setattr(- "openviking.service.reindex_executor.SemanticProcessor",- FakeSemanticProcessor,- )- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await ReindexExecutor(vlm_resolver=semantic_config)._run_semantic_processor(- uri="viking://user/alice/memories/preferences",- context_type="memory",- ctx=ctx,- recursive=False,- )-- import json-- msg = json.loads(seen["payload"]["data"])- assert msg["generation_trigger"] == "reindex"- assert msg["use_hierarchical_aggregation"] is True- assert msg["propagate_to_parent"] is False---async def test_reindex_resource_vector_text_uses_existing_record_for_non_text(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor-- async def fake_safe_read_text(self, uri, *, ctx):- return "decoded binary payload"-- async def fake_fetch_existing_record(self, *, uri, level, ctx):- return {"abstract": "existing image summary"}-- monkeypatch.setattr(ReindexExecutor, "_safe_read_text", fake_safe_read_text)- monkeypatch.setattr(ReindexExecutor, "_fetch_existing_record", fake_fetch_existing_record)-- service = ReindexExecutor()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- vector_text = await service._best_resource_file_vector_text(- "viking://resources/demo/image.png",- "",- ctx=ctx,- )-- assert vector_text == "existing image summary"---async def test_reindex_resource_vector_text_summary_first_skips_content_read(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor-- async def fail_if_content_read(self, uri, *, ctx):- raise AssertionError("summary_first should not read text content when summary exists")-- async def fake_fetch_existing_record(self, *, uri, level, ctx):- return {"abstract": "existing fallback"}-- monkeypatch.setattr(ReindexExecutor, "_safe_read_text", fail_if_content_read)- monkeypatch.setattr(ReindexExecutor, "_fetch_existing_record", fake_fetch_existing_record)- resolver = SimpleNamespace(- resolve=AsyncMock(- return_value=SimpleNamespace(- embedding=SimpleNamespace(text_source="summary_first"),- )- )- )- service = ReindexExecutor(vector_config_resolver=resolver)- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- vector_text = await service._best_resource_file_vector_text(- "viking://resources/demo/large.txt",- "summary for embedding",- ctx=ctx,- )-- assert vector_text == "summary for embedding"- resolver.resolve.assert_awaited_once_with("test")---async def test_reindex_file_summary_reads_existing_record_as_uri_owner(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor- from openviking.storage.abstract_overview import render_abstract_overview-- captured = {}-- async def fake_safe_read_text(self, uri, *, ctx):- return ""-- async def fake_fetch_existing_record(self, *, uri, level, ctx):- captured["ctx"] = ctx- return {"abstract": "owner summary"}-- monkeypatch.setattr(ReindexExecutor, "_safe_read_text", fake_safe_read_text)- monkeypatch.setattr(ReindexExecutor, "_fetch_existing_record", fake_fetch_existing_record)-- service = ReindexExecutor()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="admin"),- role=Role.ROOT,- )-- summary = await service._best_file_summary(- "viking://user/bob/resources/demo/image.png",- ctx=ctx,- )-- assert summary == "owner summary"- assert captured["ctx"].user.user_id == "bob"-- raw = render_abstract_overview(- ContextLevel.OVERVIEW,- "viking://resources/demo",- (- "# Demo\n\n"- "## Detailed Description\n\n"- "### [Image](viking://resources/demo/image.png)\n"- "Visible file summary."- ),- {- "source": {- "kind": "http",- "uri": "https://example.com/private.pdf",- }- },- )-- async def fake_safe_read_text(self, uri, *, ctx):- del self, uri, ctx- return raw-- async def fail_if_existing_record_read(self, *, uri, level, ctx):- raise AssertionError("body-only overview summary should be used before fallback")-- monkeypatch.setattr(ReindexExecutor, "_safe_read_text", fake_safe_read_text)- monkeypatch.setattr(ReindexExecutor, "_fetch_existing_record", fail_if_existing_record_read)-- summary = await service._best_file_summary(- "viking://resources/demo/image.png",- ctx=ctx,- )-- assert summary == "Visible file summary."- assert "source:" not in summary---async def test_reindex_memory_fallback_reads_existing_record_as_uri_owner(monkeypatch):- from openviking.service.reindex_executor import (- ReindexExecutor,- _ReindexCounters,- _SourceRead,- )-- captured = {}- upserts = []-- class FakeVikingFS:- async def exists(self, uri, ctx=None):- return True-- async def stat(self, uri, ctx=None, skip_count=True):- return {"isDir": False}-- async def fake_read_memory_body(self, uri, *, ctx):- return _SourceRead(exists=True, text="")-- async def fake_fetch_existing_record(self, *, uri, level, ctx):- captured["ctx"] = ctx- return {"abstract": "owner memory summary"}-- async def fake_best_file_summary(self, uri, *, ctx):- return ""-- async def fake_upsert_context(self, **kwargs):- upserts.append(kwargs)-- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())- monkeypatch.setattr(ReindexExecutor, "_read_memory_body", fake_read_memory_body)- monkeypatch.setattr(ReindexExecutor, "_fetch_existing_record", fake_fetch_existing_record)- monkeypatch.setattr(ReindexExecutor, "_best_file_summary", fake_best_file_summary)- monkeypatch.setattr(ReindexExecutor, "_upsert_context", fake_upsert_context)-- service = ReindexExecutor()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="admin"),- role=Role.ROOT,- )-- await service._reindex_memory_vectors(- uri="viking://user/bob/memories/preferences/theme.md",- counters=_ReindexCounters(),- ctx=ctx,- )-- assert captured["ctx"].user.user_id == "bob"- assert upserts[0]["abstract"] == "owner memory summary"---async def test_reindex_memory_skips_fallback_when_body_read_fails(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- upserts = []-- class FakeVikingFS:- async def exists(self, uri, ctx=None):- return True-- async def stat(self, uri, ctx=None, skip_count=True):- return {"isDir": False}-- async def read_file(self, uri, ctx=None):- raise OSError("backend read failed")-- async def fake_fetch_existing_record(self, *, uri, level, ctx):- return {"abstract": "existing summary"}-- async def fake_best_file_summary(self, uri, *, ctx):- return ""-- async def fake_upsert_context(self, **kwargs):- upserts.append(kwargs)-- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())- monkeypatch.setattr(ReindexExecutor, "_fetch_existing_record", fake_fetch_existing_record)- monkeypatch.setattr(ReindexExecutor, "_best_file_summary", fake_best_file_summary)- monkeypatch.setattr(ReindexExecutor, "_upsert_context", fake_upsert_context)-- service = ReindexExecutor()- counters = _ReindexCounters()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await service._reindex_memory_vectors(- uri="viking://user/default/memories/events/item.md",- counters=counters,- ctx=ctx,- )-- assert upserts == []- assert counters.failed_records == 1- assert any("failed to read memory body" in warning for warning in counters.warnings)---async def test_reindex_resource_vector_text_skips_non_text_body_without_summary(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor-- async def fail_if_content_read(self, uri, *, ctx):- raise AssertionError("non-text resource content should not be read for vector text")-- async def fake_fetch_existing_record(self, *, uri, level, ctx):- return None-- monkeypatch.setattr(ReindexExecutor, "_safe_read_text", fail_if_content_read)- monkeypatch.setattr(ReindexExecutor, "_fetch_existing_record", fake_fetch_existing_record)-- service = ReindexExecutor()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- vector_text = await service._best_resource_file_vector_text(- "viking://resources/demo/image.png",- "",- ctx=ctx,- )-- assert vector_text == ""---async def test_reindex_memory_l2_falls_back_to_body_when_abstract_missing(monkeypatch):- from openviking.service.reindex_executor import (- ReindexExecutor,- _ReindexCounters,- _SourceRead,- )-- class FakeVikingFS:- async def exists(self, uri, ctx=None):- return True-- async def stat(self, uri, ctx=None, skip_count=True):- return {"isDir": False}-- seen = {}-- async def fake_read_memory_body(self, uri, *, ctx):- return _SourceRead(exists=True, text="memory body text")-- async def fake_fetch_existing_record(self, *, uri, level, ctx):- return None-- async def fake_best_file_summary(self, uri, *, ctx):- return ""-- async def fake_upsert_context(self, **kwargs):- seen[kwargs["uri"]] = kwargs-- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())- monkeypatch.setattr(ReindexExecutor, "_read_memory_body", fake_read_memory_body)- monkeypatch.setattr(ReindexExecutor, "_fetch_existing_record", fake_fetch_existing_record)- monkeypatch.setattr(ReindexExecutor, "_best_file_summary", fake_best_file_summary)- monkeypatch.setattr(ReindexExecutor, "_upsert_context", fake_upsert_context)-- service = ReindexExecutor()- counters = _ReindexCounters()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await service._reindex_memory_vectors(- uri="viking://user/default/memories/events/item.md",- counters=counters,- ctx=ctx,- )-- assert seen["viking://user/default/memories/events/item.md"]["abstract"] == "memory body text"---async def test_reindex_memory_l2_strips_memory_fields_from_abstract(monkeypatch):- from openviking.service.reindex_executor import (- ReindexExecutor,- _ReindexCounters,- _SourceRead,- )-- class FakeVikingFS:- async def exists(self, uri, ctx=None):- return True-- async def stat(self, uri, ctx=None, skip_count=True):- return {"isDir": False}-- seen = {}- raw_body = (- "User has a preference for watermelon, as mentioned in the conversation: "- '\'我爱吃西瓜\'. <!-- MEMORY_FIELDS { "user": "user", "topic": "food_preference" } -->'- )-- async def fake_read_memory_body(self, uri, *, ctx):- return _SourceRead(exists=True, text=raw_body)-- async def fake_fetch_existing_record(self, *, uri, level, ctx):- return None-- async def fake_best_file_summary(self, uri, *, ctx):- return ""-- async def fake_upsert_context(self, **kwargs):- seen[kwargs["uri"]] = kwargs-- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())- monkeypatch.setattr(ReindexExecutor, "_read_memory_body", fake_read_memory_body)- monkeypatch.setattr(ReindexExecutor, "_fetch_existing_record", fake_fetch_existing_record)- monkeypatch.setattr(ReindexExecutor, "_best_file_summary", fake_best_file_summary)- monkeypatch.setattr(ReindexExecutor, "_upsert_context", fake_upsert_context)-- service = ReindexExecutor()- counters = _ReindexCounters()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await service._reindex_memory_vectors(- uri="viking://user/default/memories/preferences/food_preference.md",- counters=counters,- ctx=ctx,- )-- assert (- seen["viking://user/default/memories/preferences/food_preference.md"]["abstract"]- == "User has a preference for watermelon, as mentioned in the conversation: '我爱吃西瓜'."- )- assert (- seen["viking://user/default/memories/preferences/food_preference.md"]["vector_text"]- == raw_body- )---async def test_reindex_memory_vectors_walks_deep_subtree(monkeypatch):- from openviking.service.reindex_executor import (- ReindexExecutor,- _ReindexCounters,- _SourceRead,- )-- class FakeVikingFS:- async def exists(self, uri, ctx=None):- return True-- async def stat(self, uri, ctx=None, skip_count=True):- return {"isDir": True}-- async def tree(- self,- uri,- output="original",- show_all_hidden=False,- node_limit=1000,- level_limit=3,- ctx=None,- ):- if level_limit is not None:- return [- {"uri": "viking://user/default/memories/preferences/user", "isDir": True},- ]- return [- {"uri": "viking://user/default/memories/preferences/user", "isDir": True},- {- "uri": "viking://user/default/memories/preferences/user/food_preference.md",- "isDir": False,- },- ]-- seen = {}-- async def fake_read_memory_body(self, uri, *, ctx):- return _SourceRead(exists=True, text="likes spicy food")-- async def fake_fetch_existing_record(self, *, uri, level, ctx):- return None-- async def fake_best_file_summary(self, uri, *, ctx):- return ""-- async def fake_upsert_context(self, **kwargs):- seen[kwargs["uri"]] = kwargs-- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())- monkeypatch.setattr(ReindexExecutor, "_read_memory_body", fake_read_memory_body)- monkeypatch.setattr(ReindexExecutor, "_fetch_existing_record", fake_fetch_existing_record)- monkeypatch.setattr(ReindexExecutor, "_best_file_summary", fake_best_file_summary)- monkeypatch.setattr(ReindexExecutor, "_upsert_context", fake_upsert_context)-- service = ReindexExecutor()- counters = _ReindexCounters()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await service._reindex_memory_vectors(- uri="viking://user/default/memories/preferences",- counters=counters,- ctx=ctx,- )-- assert "viking://user/default/memories/preferences/user/food_preference.md" in seen---async def test_reindex_memory_vectors_rebuilds_directory_levels_without_regenerating_semantics(- monkeypatch,-):- from openviking.core.context import ContextLevel- from openviking.service.reindex_executor import (- ReindexExecutor,- _ReindexCounters,- _SourceRead,- )-- class FakeVikingFS:- async def exists(self, uri, ctx=None):- return True-- async def stat(self, uri, ctx=None, skip_count=True):- return {"isDir": True}-- async def tree(- self,- uri,- output="original",- show_all_hidden=False,- node_limit=1000,- level_limit=3,- ctx=None,- ):- return [- {"uri": "viking://user/default/memories/preferences/user", "isDir": True},- {- "uri": "viking://user/default/memories/preferences/user/food_preference.md",- "isDir": False,- },- ]-- seen = []-- async def fake_read_directory_abstract(self, uri, *, ctx):- if uri == "viking://user/default/memories/preferences/user":- return "user preferences abstract"- return ""-- async def fake_read_directory_overview(self, uri, *, ctx):- if uri == "viking://user/default/memories/preferences":- return "preferences overview"- if uri == "viking://user/default/memories/preferences/user":- return "user preferences overview"- return ""-- async def fake_read_memory_body(self, uri, *, ctx):- return _SourceRead(exists=True, text="likes spicy food")-- async def fake_fetch_existing_record(self, *, uri, level, ctx):- return None-- async def fake_best_file_summary(self, uri, *, ctx):- return ""-- async def fake_upsert_context(self, **kwargs):- seen.append(- {- "uri": kwargs["uri"],- "level": int(kwargs["level"]),- "vector_text": kwargs["vector_text"],- }- )-- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())- monkeypatch.setattr(ReindexExecutor, "_read_directory_abstract", fake_read_directory_abstract)- monkeypatch.setattr(ReindexExecutor, "_read_directory_overview", fake_read_directory_overview)- monkeypatch.setattr(ReindexExecutor, "_read_memory_body", fake_read_memory_body)- monkeypatch.setattr(ReindexExecutor, "_fetch_existing_record", fake_fetch_existing_record)- monkeypatch.setattr(ReindexExecutor, "_best_file_summary", fake_best_file_summary)- monkeypatch.setattr(ReindexExecutor, "_upsert_context", fake_upsert_context)-- service = ReindexExecutor()- counters = _ReindexCounters()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,+ lambda: SimpleNamespace(vikingdb_manager=db),)+ executor = ReindexExecutor()+ executor._reindex_user_namespace = AsyncMock()+ executor._reindex_rfv = AsyncMock()+ ctx = RequestContext(user=UserIdentifier("test", "admin"), role=Role.ADMIN)- await service._reindex_memory_vectors(- uri="viking://user/default/memories/preferences",- counters=counters,- ctx=ctx,+ await executor._reindex_global_namespace(+ uri="viking://", mode="vectors_only", run=_make_reindex_run(ctx, _ReindexCounters()))assert {- ("viking://user/default/memories/preferences", int(ContextLevel.OVERVIEW)),- ("viking://user/default/memories/preferences/user", int(ContextLevel.ABSTRACT)),- ("viking://user/default/memories/preferences/user", int(ContextLevel.OVERVIEW)),- (- "viking://user/default/memories/preferences/user/food_preference.md",- int(ContextLevel.DETAIL),- ),- } <= {(item["uri"], item["level"]) for item in seen}---async def test_reindex_user_namespace_partitions_memory_skill_and_resource(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- class FakeVikingFS:- async def tree(- self,- uri,- output="original",- show_all_hidden=True,- node_limit=1000,- level_limit=None,- ctx=None,- ):- return [- {"uri": "viking://user/default/memories", "isDir": True},- {"uri": "viking://user/default/memories/preferences", "isDir": True},- {"uri": "viking://user/default/skills", "isDir": True},- {"uri": "viking://user/default/skills/my_skill", "isDir": True},- {"uri": "viking://user/default/sessions", "isDir": True},- {"uri": "viking://user/default/sessions/s1", "isDir": True},- {"uri": "viking://user/default/resources", "isDir": True},- {"uri": "viking://user/default/resources/doc.md", "isDir": False},- {"uri": "viking://user/default/sessions/s1/messages.jsonl", "isDir": False},- {"uri": "viking://user/default/profile.md", "isDir": False},- {"uri": "viking://user/default/memories/preferences/theme.md", "isDir": False},- {"uri": "viking://user/default/skills/my_skill/SKILL.md", "isDir": False},- ]-- seen = {"memory": [], "rfv": []}-- async def fake_reindex_memory(self, *, uri, mode, run):- seen["memory"].append((uri, mode))-- async def fake_reindex_rfv(self, **kwargs):- seen["rfv"].append(kwargs)-- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())- monkeypatch.setattr(ReindexExecutor, "_reindex_memory", fake_reindex_memory)- monkeypatch.setattr(ReindexExecutor, "_reindex_rfv", fake_reindex_rfv)-- service = ReindexExecutor()- counters = _ReindexCounters()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await service._reindex_user_namespace(- uri="viking://user/",- mode="vectors_only",- run=_make_reindex_run(ctx, counters),- )-- assert seen["memory"] == [("viking://user/default/memories", "vectors_only")]- assert [(call["uri"], call["context_type"]) for call in seen["rfv"]] == [- ("viking://user/default/skills/my_skill", "skill"),- ("viking://user/default/resources", "resource"),- ("viking://user/default/profile.md", "resource"),- ]---async def test_reindex_global_namespace_partitions_user_and_resources(monkeypatch):- from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters-- class FakeVikingFS:- async def tree(- self,- uri,- output="original",- show_all_hidden=True,- node_limit=1000,- level_limit=None,- ctx=None,- ):- return [- {"uri": "viking://user", "isDir": True},- {"uri": "viking://user/default", "isDir": True},- {"uri": "viking://user/default/memories", "isDir": True},- {"uri": "viking://user/default", "isDir": True},- {"uri": "viking://user/default/skills", "isDir": True},- {"uri": "viking://session", "isDir": True},- {"uri": "viking://session/default", "isDir": True},- {"uri": "viking://resources", "isDir": True},- {"uri": "viking://session/default/archive.txt", "isDir": False},- {"uri": "viking://resources/demo.txt", "isDir": False},- {"uri": "viking://README.md", "isDir": False},- ]-- seen = {"user": [], "rfv": []}-- async def fake_reindex_user_namespace(self, *, uri, mode, run):- seen["user"].append((uri, mode))-- async def fake_reindex_rfv(self, **kwargs):- seen["rfv"].append(kwargs)-- monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())- monkeypatch.setattr(ReindexExecutor, "_reindex_user_namespace", fake_reindex_user_namespace)- monkeypatch.setattr(ReindexExecutor, "_reindex_rfv", fake_reindex_rfv)-- service = ReindexExecutor()- counters = _ReindexCounters()- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )-- await service._reindex_global_namespace(- uri="viking://",- mode="vectors_only",- run=_make_reindex_run(ctx, counters),- )-- assert seen["user"] == [("viking://user/default", "vectors_only")]- assert [(call["uri"], call["context_type"]) for call in seen["rfv"]] == [- ("viking://resources", "resource")+ (call.kwargs["uri"], call.kwargs["context_type"])+ for call in executor._reindex_rfv.await_args_list+ } == {+ ("viking://resources", "resource"),+ ("viking://agent/skills/demo", "skill"),+ }+ assert [call.kwargs["uri"] for call in executor._reindex_user_namespace.await_args_list] == [+ "viking://user/bob"]--[email protected]("tag_mode", ["replace", "clear"])-async def test_reindex_upsert_empty_tags_preserves_replace_and_clear_semantics(- monkeypatch, tag_mode-):- from types import SimpleNamespace-- from openviking.service.reindex_executor import ReindexExecutor- from openviking.utils.ingest_options import IngestOptions-- queued = []-- class FakeVikingDB:- async def enqueue_embedding_msg(self, msg):- queued.append(msg)- return True-- monkeypatch.setattr(- "openviking.service.reindex_executor.get_service",- lambda: SimpleNamespace(vikingdb_manager=FakeVikingDB()),- )-- ctx = RequestContext(- user=UserIdentifier(account_id="test", user_id="alice"),- role=Role.ROOT,- )- await ReindexExecutor()._upsert_context(- uri="viking://resources/demo.md",- parent_uri="viking://resources",- abstract="demo",- vector_text="demo",- is_leaf=True,- context_type="resource",- level=ContextLevel.DETAIL,- ctx=ctx,- ingest_options=IngestOptions.from_search_tags(- [] if tag_mode == "replace" else None, mode=tag_mode- ),- )-- if tag_mode == "replace":- assert "search_tags" not in queued[0].context_data- assert "_upsert_options" not in queued[0].context_data- else:- assert queued[0].context_data["search_tags"] == []- assert queued[0].context_data["_upsert_options"] == {"search_tag_mode": "replace"}---async def test_openviking_service_reindex_uses_default_root_context(monkeypatch):- from openviking.service.core import OpenVikingService-- seen = {}-- class FakeExecutor:- def __init__(self, vlm_resolver=None, vector_config_resolver=None):- seen["vlm_resolver"] = vlm_resolver- seen["vector_config_resolver"] = vector_config_resolver-- async def execute(self, *, uri, mode, wait, ctx):- seen["uri"] = uri- seen["mode"] = mode- seen["wait"] = wait- seen["ctx"] = ctx- return {"status": "completed", "uri": uri}-- import openviking.service.reindex_executor as reindex_executor-- monkeypatch.setattr(- reindex_executor,- "ReindexExecutor",- FakeExecutor,- )-- service = OpenVikingService.__new__(OpenVikingService)- service._initialized = True- service._user = UserIdentifier(account_id="acct", user_id="alice")- service._vlm_resolver = object()- service._vector_config_resolver = object()-- result = await OpenVikingService.reindex(- service,- uri="viking://resources/demo",- mode="vectors_only",- wait=True,- )-- assert result == {"status": "completed", "uri": "viking://resources/demo"}- assert seen["ctx"].role == Role.ROOT- assert seen["ctx"].user.account_id == "acct"- assert seen["ctx"].user.user_id == "alice"- assert seen["vlm_resolver"] is service._vlm_resolver- assert seen["vector_config_resolver"] is service._vector_config_resolver---async def test_openviking_service_reindex_keeps_canonical_user_id(monkeypatch):- from openviking.service.core import OpenVikingService-- seen = {}-- class FakeExecutor:- def __init__(self, vlm_resolver=None, vector_config_resolver=None):- seen["vlm_resolver"] = vlm_resolver- seen["vector_config_resolver"] = vector_config_resolver-- async def execute(self, *, uri, mode, wait, ctx):- seen["uri"] = uri- seen["ctx"] = ctx- return {"status": "completed", "uri": uri}-- import openviking.service.reindex_executor as reindex_executor-- monkeypatch.setattr(- reindex_executor,- "ReindexExecutor",- FakeExecutor,- )-- service = OpenVikingService.__new__(OpenVikingService)- service._initialized = True- service._user = UserIdentifier(account_id="acct", user_id="alice")- service._vlm_resolver = object()- service._vector_config_resolver = object()- ctx = RequestContext(- user=UserIdentifier(account_id="acct", user_id="alice"),- role=Role.ADMIN,- )-- result = await OpenVikingService.reindex(- service,- uri="viking://user/resources/memories",- mode="vectors_only",- wait=True,- ctx=ctx,- )-- assert result == {"status": "completed", "uri": "viking://user/resources/memories"}- assert seen["uri"] == "viking://user/resources/memories"- assert seen["ctx"] is ctx- assert seen["vlm_resolver"] is service._vlm_resolver- assert seen["vector_config_resolver"] is service._vector_config_resolverdiff --git a/tests/server/test_api_snapshot.py b/tests/server/test_api_snapshot.pyindex 9e7df7cfb..4230970d0 100644--- a/tests/server/test_api_snapshot.py+++ b/tests/server/test_api_snapshot.py@@ -286,9 +286,7 @@ async def test_restore_acl_and_reindex_identity(client_with_resource, service, mreindex_contexts.append(ctx)return 0- def factory(*, vlm_resolver, vector_config_resolver):- assert vlm_resolver is service.viking_fs._vlm_resolver- assert vector_config_resolver is service.vector_config_resolver+ def factory():return _SpyExecutor()monkeypatch.setattr(reindex_mod, "ReindexExecutor", factory)@@ -679,9 +677,7 @@ async def test_restore_apply_triggers_reindex_hook(calls.append(uri)return {"ok": True}- def factory(*, vlm_resolver, vector_config_resolver):- assert vlm_resolver is service.viking_fs._vlm_resolver- assert vector_config_resolver is service.vector_config_resolver+ def factory():return _SpyExecutor()monkeypatch.setattr(reindex_mod, "ReindexExecutor", factory)@@ -739,9 +735,7 @@ async def test_restore_delete_removes_orphaned_vectors(deleted_calls.append((uri, int(level)))return 0- def factory(*, vlm_resolver, vector_config_resolver):- assert vlm_resolver is service.viking_fs._vlm_resolver- assert vector_config_resolver is service.vector_config_resolver+ def factory():return _SpyExecutor()monkeypatch.setattr(reindex_mod, "ReindexExecutor", factory)diff --git a/tests/service/test_reindex_file_lock.py b/tests/service/test_reindex_file_lock.pyindex 1e8f54dce..5df7896bc 100644--- a/tests/service/test_reindex_file_lock.py+++ b/tests/service/test_reindex_file_lock.py@@ -1,6 +1,6 @@# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.# SPDX-License-Identifier: AGPL-3.0-"""Regression tests for reindexing single-file URIs under tree locks."""+"""Exact-file reindex must protect real files during legacy chunk cleanup."""from types import SimpleNamespacefrom unittest.mock import AsyncMock@@ -8,67 +8,87 @@ from unittest.mock import AsyncMockfrom openviking.server.identity import RequestContext, Rolefrom openviking.service import reindex_executor as reindex_modulefrom openviking.service.reindex_executor import ReindexExecutor+from openviking_cli.exceptions import NotFoundErrorfrom openviking_cli.session.user_id import UserIdentifier-class _FakeVikingFS:- def __init__(self, uri: str, path: str):- self._uri = uri- self._path = path+async def test_reindex_single_file_keeps_concurrent_chunk_named_file(monkeypatch):+ uri = "viking://user/bob/memories/events/a.md"+ stale, created = uri + "#chunk_0000", uri + "#chunk_0001"+ sources = {uri: "memory body"}+ stored = {+ "base": {"id": "base", "uri": uri, "level": 2, "abstract": "summary"},+ "stale": {"id": "stale", "uri": stale, "level": 2},+ "created": {"id": "created", "uri": created, "level": 2},+ }+ owned = set()- def _uri_to_path(self, uri: str, ctx=None):- assert uri == self._uri- return self._path+ async def acquire_file(path):+ assert path == "/local/test/" + uri.removeprefix("viking://")+ owned.add("file")+ return {"lease_id": "file"}- async def exists(self, uri: str, ctx=None):- return uri == self._uri+ async def acquire_chunks(paths, *, owner_lease_ref):+ assert owner_lease_ref == {"lease_id": "file"}+ assert set(paths) == {+ "/local/test/" + item.removeprefix("viking://") for item in (stale, created)+ }+ owned.add("chunks")+ return {"lease_id": "chunks"}- async def stat(self, uri: str, ctx=None, skip_count=False):- assert uri == self._uri- assert skip_count is True- return {"isDir": False}+ async def release(lease):+ owned.remove(lease["lease_id"])- async def read_file(self, uri: str, ctx=None):- assert uri == self._uri- return "# Profile\nSingle file reindex source.\n"+ async def stat(target, **kwargs):+ if target not in sources:+ raise NotFoundError(target)+ return {"isDir": False}+ async def enqueue(msg):+ assert msg.context_data["uri"] == uri+ # A sibling writer finishes after the first existence check.+ sources[created] = "new real file"+ stored["created"]["abstract"] = "new vector"+ return True-async def test_reindex_single_file_uri_acquires_tree_lock_without_not_a_directory(- agfs_client, test_dir, monkeypatch, caplog-):- uri = "viking://resources/profile.md"- path = f"{test_dir}/profile.md"- agfs_client.write(path, b"# Profile\nSingle file reindex source.\n")+ async def delete(ids, **kwargs):+ assert owned == {"file", "chunks"}+ for record_id in ids:+ del stored[record_id]+ return len(ids)- viking_fs = _FakeVikingFS(uri, path)- service = SimpleNamespace(- viking_fs=viking_fs,- vikingdb_manager=SimpleNamespace(has_queue_manager=True),+ agfs = SimpleNamespace(+ pathlock_acquire_exact=acquire_file,+ pathlock_acquire_tree=AsyncMock(side_effect=AssertionError("file target used tree lock")),+ pathlock_acquire_exact_batch=acquire_chunks,+ pathlock_as_borrowed=AsyncMock(return_value={"lease_ref": "file"}),+ pathlock_release=release,+ )+ fs = SimpleNamespace(+ _async_agfs=agfs,+ _uri_to_path=lambda value, **kw: "/local/test/" + value.removeprefix("viking://"),+ stat=stat,+ exists=AsyncMock(side_effect=lambda value, **kw: value in sources),+ read_file=AsyncMock(side_effect=lambda value, **kw: sources[value]),)+ db = SimpleNamespace(+ has_queue_manager=True,+ get_reindex_inventory_under_uri=AsyncMock(return_value=dict(stored)),+ enqueue_embedding_msg=enqueue,+ strict_delete=delete,+ )+ service = SimpleNamespace(viking_fs=fs, vikingdb_manager=db)monkeypatch.setattr(reindex_module, "get_service", lambda: service)- monkeypatch.setattr(reindex_module, "get_viking_fs", lambda: viking_fs)-+ monkeypatch.setattr(reindex_module, "get_viking_fs", lambda: fs)executor = ReindexExecutor()- executor._fetch_existing_record = AsyncMock(return_value=None)- executor._upsert_context = AsyncMock()- ctx = RequestContext(- user=UserIdentifier("acc1", "test_user"),- role=Role.ROOT,- )+ monkeypatch.setattr(executor, "_effective_file_vectorization_concurrency", lambda: 1)+ ctx = RequestContext(user=UserIdentifier("test", "bob"), role=Role.USER)- try:- result = await executor._run(- uri=uri,- object_type="resource",- mode="vectors_only",- ctx=ctx,- )- finally:- pass+ result = await executor._run(uri=uri, object_type="memory", mode="vectors_only", ctx=ctx)assert result["status"] == "completed"- assert result["uri"] == uri- assert result["scanned_records"] == 1- assert result["rebuilt_records"] == 1- assert "Not a directory" not in caplog.text- assert "Failed to create lock file" not in caplog.text+ assert result["rebuilt_records"] == result["deleted_records"] == 1+ assert set(stored) == {"base", "created"}+ assert stored["created"]["abstract"] == "new vector"+ assert sources[created] == "new real file"+ assert not owneddiff --git a/tests/storage/test_reindex_msg.py b/tests/storage/test_reindex_msg.pyindex b529efae2..4aa7ffbc3 100644--- a/tests/storage/test_reindex_msg.py+++ b/tests/storage/test_reindex_msg.py@@ -346,7 +346,7 @@ async def test_reindex_processor_cancellation_releases_root_lock_handoff():@pytest.mark.asyncio-async def test_reindex_processor_does_not_release_lease_owned_by_failed_executor(monkeypatch):+async def test_reindex_processor_releases_adopted_lease_when_executor_startup_fails(monkeypatch):from openviking.storage.queuefs.reindex_processor import ReindexProcessortracker = SimpleNamespace(@@ -354,32 +354,28 @@ async def test_reindex_processor_does_not_release_lease_owned_by_failed_executorstart=AsyncMock(),complete=AsyncMock(),fail=AsyncMock(),- wait_for_task_work=AsyncMock(),)monkeypatch.setattr(- "openviking.storage.queuefs.reindex_processor.get_task_tracker", Mock(return_value=tracker)+ "openviking.storage.queuefs.reindex_processor.get_task_tracker", lambda: tracker)- monkeypatch.setattr(- "openviking.server.dependencies.get_service",- lambda: SimpleNamespace(_vlm_resolver=None, _vector_config_resolver=None),+ agfs = SimpleNamespace(pathlock_release=AsyncMock(), pathlock_acquire_tree=AsyncMock())+ fs = SimpleNamespace(+ _async_agfs=agfs,+ _uri_to_path=lambda uri, **kwargs: "/local/account-1/resources/demo",+ stat=AsyncMock(side_effect=OSError("storage unavailable")),)- agfs = SimpleNamespace(pathlock_release=AsyncMock())-- async def fail_after_releasing(_self, *, existing_lease, **_kwargs):- await agfs.pathlock_release(existing_lease)- raise RuntimeError("semantic plan enqueue failed")-- monkeypatch.setattr(- "openviking.service.reindex_executor.ReindexExecutor._run",- fail_after_releasing,+ service = SimpleNamespace(+ viking_fs=fs, vikingdb_manager=SimpleNamespace(has_queue_manager=True))- processor = ReindexProcessor(SimpleNamespace(_async_agfs=agfs))- processor._adopt_or_reacquire = AsyncMock(return_value={"lease": "root"})+ monkeypatch.setattr("openviking.service.reindex_executor.get_service", lambda: service)+ processor = ReindexProcessor(fs)+ lease = {"lease": "root"}+ processor._adopt_or_reacquire = AsyncMock(return_value=lease)message = ReindexMsg(task_id="task-1",uri="viking://resources/demo",object_type="resource",- mode="semantic_and_vectors",+ mode="vectors_only",account_id="account-1",user_id="user-1",role="admin",@@ -388,5 +384,6 @@ async def test_reindex_processor_does_not_release_lease_owned_by_failed_executoroutcome = await processor.on_dequeue({"id": "queue-1", "data": message.to_dict()})assert outcome.outcome.value == "failed"- agfs.pathlock_release.assert_awaited_once_with({"lease": "root"})- tracker.fail.assert_awaited_once()+ agfs.pathlock_release.assert_awaited_once_with(lease)+ tracker.complete.assert_not_awaited()+ assert tracker.fail.await_args.args[:2] == ("task-1", "storage unavailable")diff --git a/tests/storage/test_resource_rfv.py b/tests/storage/test_resource_rfv.pyindex b27bfd328..f31471f69 100644--- a/tests/storage/test_resource_rfv.py+++ b/tests/storage/test_resource_rfv.py@@ -517,7 +517,7 @@ class _SnapshotDB:@pytest.mark.asyncio-async def test_build_rfv_snapshot_reads_one_f_inventory_one_v_inventory_and_each_source_once():+async def test_build_rfv_snapshot_reuses_inventory_and_reads_each_source_once():from openviking.storage.resource_rfv import build_rfv_snapshotfrom openviking.utils.content_hash import content_md5@@ -525,6 +525,7 @@ async def test_build_rfv_snapshot_reads_one_f_inventory_one_v_inventory_and_eachfs = _SnapshotFS(root)db = _SnapshotDB()+ inventory = {"old": {"id": "old", "uri": f"{root}/gone.md", "level": 2, "md5": "old"}}snapshot = await build_rfv_snapshot(viking_fs=fs,vikingdb=db,@@ -533,11 +534,13 @@ async def test_build_rfv_snapshot_reads_one_f_inventory_one_v_inventory_and_eachrequest_intent=RequestIntent(root, "vectors_only"),recursive=True,root_is_dir=True,+ vector_inventory=inventory,)fs.tree.assert_awaited_once()fs.stat.assert_not_awaited()- db.get_incremental_inventory_under_uri.assert_awaited_once()+ db.get_incremental_inventory_under_uri.assert_not_awaited()+ assert snapshot.vectors.records_by_id["old"].relative_path == "gone.md"assert fs.read_file_bytes.await_count == 3assert snapshot.formal.entries["docs"].level_md5s == {0: content_md5(b"docs abstract"),diff --git a/tests/storage/test_semantic_processor_permanent_storage_error.py b/tests/storage/test_semantic_processor_permanent_storage_error.pyindex 0d55a5577..f696975cf 100644--- a/tests/storage/test_semantic_processor_permanent_storage_error.py+++ b/tests/storage/test_semantic_processor_permanent_storage_error.py@@ -228,7 +228,7 @@ async def test_lock_error_requeues_without_consuming_failure_budget(semantic_envresult = await asyncio.wait_for(env.processor.on_dequeue(msg.to_dict()), timeout=1)assert result.outcome is ProcessOutcome.REQUEUED- env.retry_queue.enqueue.assert_awaited_once()+ env.retry_queue.enqueue.assert_awaited_once_with(msg, deduplicate_memory=False)requeued = env.retry_queue.enqueue.call_args.args[0]assert requeued.id == msg.idassert requeued.failure_requeue_count == 0diff --git a/tests/storage/test_vector_transfer.py b/tests/storage/test_vector_transfer.pyindex 15fbfde4e..fd44d7e48 100644--- a/tests/storage/test_vector_transfer.py+++ b/tests/storage/test_vector_transfer.py@@ -683,61 +683,79 @@ async def test_l2_diff_scan_reads_every_page_under_target_directory():@pytest.mark.asyncioasync def test_incremental_inventory_reads_all_semantic_levels_under_target():root = "viking://resources/docs"+ files = [_record(f"file-{i}", f"{root}/{i}.md", md5=f"md5-{i}") for i in range(5998)]backend = _RealAclMemoryTransferBackend([- _record("root-l0", root, level=0, abstract="root abstract"),- _record("root-l1", root, level=1, abstract="root overview"),- _record("file-l2", f"{root}/a.py", level=2, md5="ma"),- _record("outside", "viking://resources/other.py", level=2),- _record(- "other-account",- f"{root}/private.py",- level=2,- account_id="other",- ),+ _record("root-l0", root, level=0),+ _record("root-l1", root, level=1),+ *files,+ _record("outside", "viking://resources/other.py"),+ _record("other-account", f"{root}/private.py", account_id="other"),])+ original_page = backend._strict_transfer_page+ original_filter = backend.filter- records = await backend.get_incremental_inventory_under_uri(root, ctx=_ctx(), batch_size=2)+ async def bounded_filter(*args, limit=10, **kwargs):+ assert limit <= 5000, "VikingDB rejects oversized queries"+ return await original_filter(*args, limit=limit, **kwargs)- assert records == {- "root-l0": {"id": "root-l0", "uri": root, "level": 0, "md5": ""},- "root-l1": {"id": "root-l1", "uri": root, "level": 1, "md5": ""},- "file-l2": {- "id": "file-l2",- "uri": f"{root}/a.py",- "level": 2,- "md5": "ma",- },- }- assert len(backend.scroll_filters) == 2+ async def bounded_page(ctx, filter, *, limit, cursor, output_fields):+ assert limit <= 5000, "VikingDB rejects oversized pages"+ return await original_page(+ ctx, filter, limit=limit, cursor=cursor, output_fields=output_fields+ )++ backend.filter = bounded_filter+ backend._strict_transfer_page = bounded_page+ incremental = await backend.get_incremental_inventory_under_uri(root, ctx=_ctx())+ reindex = await backend.get_reindex_inventory_under_uri(root, ctx=_ctx())++ assert reindex == incremental+ assert len(reindex) == 6000+ assert reindex["root-l0"]["level"] == 0+ assert reindex["root-l1"]["level"] == 1+ assert reindex["file-5997"]["md5"] == "md5-5997"+ assert "outside" not in reindex and "other-account" not in reindex+ backend._strict_transfer_count = AsyncMock(return_value=6001)+ with pytest.raises(RuntimeError, match="ended after 6000 of 6001"):+ await backend.get_reindex_inventory_under_uri(root, ctx=_ctx())@pytest.mark.asyncioasync def test_incremental_inventory_projects_request_scalar_fields():- root = "viking://resources/docs"+ uri = "viking://user/alice/memories/events/a.md"backend = _RealAclMemoryTransferBackend([- _record(- "file-l2",- f"{root}/a.py",- level=2,- md5="ma",- search_tags=["env=test"],- )+ _record("base", uri, md5="ma", search_tags=["env=test"]),+ _record("chunk", uri + "#chunk_0000", search_tags=["env=test"]),+ _record("sibling", uri + "-other.md"),+ _record("literal", uri + "#chunk_notes"),])-- records = await backend.get_incremental_inventory_under_uri(- root,+ incremental = await backend.get_incremental_inventory_under_uri(+ uri,ctx=_ctx(),- output_fields=["id", "uri", "level", "md5", "search_tags"],+ recursive=False,+ output_fields=["search_tags"],+ )+ reindex = await backend.get_reindex_inventory_under_uri(+ uri,+ ctx=_ctx(),+ recursive=False,+ include_virtual_chunks=True,+ output_fields=["search_tags"],)- assert records["file-l2"]["search_tags"] == ["env=test"]+ assert set(incremental) == {"base"}+ assert set(reindex) == {"base", "chunk"}+ assert reindex["base"] == incremental["base"]+ assert reindex["chunk"]["search_tags"] == ["env=test"]assert all("search_tags" in fields for fields in backend.scroll_output_fields)- assert all("vector" not in fields for fields in backend.scroll_output_fields)- assert all("sparse_vector" not in fields for fields in backend.scroll_output_fields)+ assert all(+ not {"vector", "sparse_vector", "content"}.intersection(fields)+ for fields in backend.scroll_output_fields+ )@pytest.mark.asyncio