Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
perf(resources): optimize incremental resource ingestion (#5175)
Jiahui Zhou336f2173b451
diff --git a/docs/design/freshness-aware-parent-bubbling-design.md b/docs/design/freshness-aware-parent-bubbling-design.mdindex 2f223e5c2..0a4b9db78 100644--- a/docs/design/freshness-aware-parent-bubbling-design.md+++ b/docs/design/freshness-aware-parent-bubbling-design.md@@ -52,7 +52,7 @@### 3. 采样发生在子项摘要工作之后-当前目录 DAG 会先生成或读取所有直接子项的摘要,最后才使用 `overview_sample_limit` 选出进入父目录 overview prompt 的部分输入。+当前目录语义树会先生成或读取所有直接子项的摘要,最后才使用 `overview_sample_limit` 选出进入父目录 overview prompt 的部分输入。这能够限制 prompt 大小,却没有避免未参与本次聚合的子项摘要工作。对于宽目录,如果我们的目标是降低重新摘要成本,就应当在调度本轮父目录聚合所需的子项摘要之前完成有界采样。@@ -324,7 +324,7 @@ metadata 从 `pending_child_changes > 0` 重置或减少,也可能造成 raw s| --- | --- || `openviking/storage/queuefs/semantic_ops/freshness_policy.py` | 纯粹的 `NOOP` / `MARK_PENDING` / `REFRESH_NOW` 判断 || `openviking/storage/semantic_sidecar.py` | 在 lease 下原子累加、读取和消费现有 freshness 计数 |-| `SemanticDagExecutor` | 在宽目录聚合前完成本轮采样,并返回 L0/L1 正文变化结果 |+| `SemanticTreeExecutor` | 在宽目录聚合前完成本轮采样,并返回 L0/L1 正文变化结果 || `SemanticProcessor` | 根据 L0 是否变化决定是否继续向父目录冒泡 || content write、delete、resource 路径 | 将文件增删改统一记录为所在目录的变化计数 |diff --git a/docs/design/l0-l1-okf-sidecars-rfc.md b/docs/design/l0-l1-okf-sidecars-rfc.mdindex d947ddd71..0bbf69584 100644--- a/docs/design/l0-l1-okf-sidecars-rfc.md+++ b/docs/design/l0-l1-okf-sidecars-rfc.md@@ -23,7 +23,7 @@ OpenViking uses a three-layer context model:| L1 | `.overview.md` | directory overview for navigation and rerank || L2 | source files and subdirectories | full content loaded on demand |-The current implementation treats `.abstract.md` and `.overview.md` as generated hidden sidecars. `SemanticProcessor` / `SemanticDagExecutor` generate them bottom-up, and the vector pipeline maps those two filenames to `ContextLevel.ABSTRACT` and `ContextLevel.OVERVIEW`.+The current implementation treats `.abstract.md` and `.overview.md` as generated hidden sidecars. `SemanticProcessor` / `SemanticTreeExecutor` generate them bottom-up, and the vector pipeline maps those two filenames to `ContextLevel.ABSTRACT` and `ContextLevel.OVERVIEW`.## Goalsdiff --git a/docs/design/metric-design.md b/docs/design/metric-design.mdindex 605381b58..d715d49da 100644--- a/docs/design/metric-design.md+++ b/docs/design/metric-design.md@@ -303,7 +303,7 @@ graph LR| `ResourceIngestionEventDataSource` | `EventMetricDataSource` | ResourceProcessor、ResourceService、watch / wait 流程 | 资源处理事件与阶段摘要 | parse / finalize / summarize / wait / watch duration || `SessionLifecycleDataSource` | `EventMetricDataSource` | session create / used / commit / archive | 会话生命周期状态与事件 | commit 生命周期、contexts_used、archive 状态 || `EncryptionEventDataSource` | `EventMetricDataSource` | Encryptor、API Key 验证路径、KDF / Key Loader | 加密操作事件与密钥处理事件 | encrypt / decrypt / verify count、duration、bytes、kdf / key_load 耗时、auth_failed |-| `QueuePipelineStateDataSource` | `StateMetricDataSource` | `QueueManager`、Semantic DAG、request queue stats | 队列与流水线状态 | pending、in_progress、processed、error_count、semantic_nodes |+| `QueuePipelineStateDataSource` | `StateMetricDataSource` | `QueueManager`、Semantic tree、request queue stats | 队列与流水线状态 | pending、in_progress、processed、error_count、semantic_nodes || `TaskStateDataSource` | `StateMetricDataSource` | `TaskTracker` | 当前任务状态 | pending、running、completed、failed 数量 || `RetrievalStatsDataSource` | `DomainStatsMetricDataSource` | `RetrievalStatsCollector` | 检索累计统计 | query count、zero result、latency、rerank 情况 || `ModelUsageDataSource` | `DomainStatsMetricDataSource` | VLM / Embedding / Rerank token tracker | 模型使用统计 | 调用次数、耗时、token 消耗 |@@ -415,7 +415,7 @@ Collector 与 DataSource 的主映射关系如下:| `VLMCollector` | `EventMetricCollector` | token / duration 事件 | 调用数、耗时、token 统计 || `CacheCollector` | `EventMetricCollector` | cache 命中或未命中事件 | hit / miss || `EncryptionCollector` | `EventMetricCollector` | encrypt / decrypt / verify / kdf / key_load 事件 | 加密次数、认证失败、耗时、字节量、密钥处理统计 |-| `QueueCollector` | `StateMetricCollector` | QueueManager / DAG 当前状态 | pending、in_progress、processed、errors |+| `QueueCollector` | `StateMetricCollector` | QueueManager / tree 当前状态 | pending、in_progress、processed、errors || `RagfsMetricCollector` | `DomainStatsMetricCollector` | `RagfsMetricDataSource`,一次 RAGFS `metrics()` 调用 | 文件操作、Cache、multi-backend、Lock || `VikingDBCollector` | `StateMetricCollector` | VikingDB collection 当前状态 | health、vectors、collections || `ObserverHealthCollector` | `StateMetricCollector` | ObserverService 结果 | component health、component errors |diff --git a/docs/en/about/02-changelog.md b/docs/en/about/02-changelog.mdindex b8429e1a2..6205dc0e8 100644--- a/docs/en/about/02-changelog.md+++ b/docs/en/about/02-changelog.md@@ -227,7 +227,7 @@ This changelog is automatically generated from [GitHub Releases](https://github.- **Request-scoped HTTP profiling**: Servers can enable `server.profile_enabled`; requests with `profile=1` then run `cProfile` for only that HTTP request and append `profile` lines to JSON responses. The `ov` CLI can enable and display this with `--profile`.- **Batch Session message ingestion**: Added `POST /api/v1/sessions/{session_id}/messages/batch` plus Python HTTP client and Session wrapper support via `batch_add_messages` (up to 100 messages per request), reducing HTTP round trips for LangChain/LangGraph-style integrations.- **Memory embedding templates**: Memory schemas now support a top-level `embedding_template`, replacing field-level `searchable` flags. The built-in `entities`, `events`, and `preferences` templates include key fields plus final content in embedding input.-- **Semantic indexing reliability**: Resource processing now syncs temp source trees into the target before running the semantic DAG (diff results use target URIs); stale semantic lock handoffs can be recovered by reacquiring tree locks, and lock acquisition failures requeue work instead of tripping the API circuit breaker.+- **Semantic indexing reliability**: Resource processing now syncs temp source trees into the target before running the semantic tree (diff results use target URIs); stale semantic lock handoffs can be recovered by reacquiring tree locks, and lock acquisition failures requeue work instead of tripping the API circuit breaker.- **Embedding input guardrails**: Embedding queue input is truncated according to `embedding.max_input_tokens`, and oversized-input errors are classified as `input_too_large` to avoid repeated retries for unrecoverable payloads.### Upgrade Notesdiff --git a/docs/en/concepts/09-transaction.md b/docs/en/concepts/09-transaction.mdindex 3835a184e..1178548d2 100644--- a/docs/en/concepts/09-transaction.md+++ b/docs/en/concepts/09-transaction.md@@ -134,51 +134,38 @@ Operation flow:| File moved from temp to final directory, then crash -> file exists but never searchable | Two separate paths for first-time add vs incremental update || Resource already on disk but rm deletes it while semantic processing / vectorization is still running -> wasted work | Lifecycle TreeLock held from finalization through processing completion |-**First-time add** (target does not exist) — handled in `ResourceProcessor.process_resource` Phase 3.5:+**First-time add and incremental update** use the same planned commit path:```-1. Acquire TreeLock on final_uri- - If final_uri does not exist, check ancestor/descendant/same-path conflicts first- - If there is no conflict, create final_uri and write final_uri/.path.ovlock as a T lock-2. Keep temp as the source directory and enqueue SemanticMsg(uri=temp, target_uri=final_uri, lifecycle_lock_handle_id=...)-3. DAG runs on temp and syncs temp content into final_uri after completion- - Do not use raw agfs.mv(temp -> final_uri), because final_uri already exists for the lock file-4. Clean up temp directory-5. DAG starts lock refresh loop (refreshes the lock token and updates handle activity every lock_expire/2 seconds)-6. DAG complete + all embeddings done -> release TreeLock+1. Acquire the resource lock on final_uri.+2. Build the R/N/F/V snapshot while holding the lock:+ - R: normalized request intent+ - N: parsed artifact inventory+ - F: current formal resource tree+ - V: current vector records, unless build_index=false+3. Compile a ContextUpdatePlan.+4. Apply the plan's content actions to final_uri synchronously.+5. Clean up the parser artifact.+6. Enqueue direct index actions and, when needed, a SemanticMsg containing the remaining SemanticPlan.+7. Hand off the resource lock to semantic processing, or release it when there is no semantic work.```-If summarization and indexing are both disabled, no downstream DAG takes over.-In that case `ResourceProcessor` copies temp directory content into `final_uri`-under the same TreeLock, deletes temp, then releases the lock. It does not call-`VikingFS.mv(temp, final_uri, lock_handle=handle)`, because move cleanup can-remove the directory lock file.+The formal content tree is therefore updated before semantic or embedding work+runs. A successful content commit may temporarily be ahead of its derived+summaries and vectors. Queue-backed work repairs that derived state; it no+longer copies the parser temp tree into the formal tree.During this period, `rm` attempting to acquire a TreeLock on the same path will fail with `ResourceBusyError`.-**Incremental update** (target already exists) — temp stays in place:--```-1. Acquire TreeLock on target_uri (protect existing resource)-2. Enqueue SemanticMsg(uri=temp, target_uri=final, lifecycle_lock_handle_id=...)-3. DAG runs on temp, lock refresh loop active-4. DAG completion triggers sync_diff_callback or move_temp_to_target_callback-5. Callback completes -> release TreeLock-```--Note: DAG callbacks do NOT wrap operations in an outer lock. Each `VikingFS.rm` and `VikingFS.mv` has its own lock internally. An outer lock would conflict with these inner locks causing deadlock.--Both first-time add and incremental update hold only `TreeLock(resource_dir)`.-There is no `ExactPathLock(resource_dir) -> TreeLock(resource_dir)` handoff, so-the two modes cannot accidentally release the same `.path.ovlock` file in the-wrong scope.-Automatic naming is handled by the resource layer, not the lock service:`ResourceProcessor` checks `exists(candidate_uri)` first; occupied candidatestry `_1`, `_2`, and so on. Only a non-existing candidate attempts `TreeLock`,without waiting. If that candidate is busy, the next suffix is tried.-**Server restart recovery**: SemanticMsg is persisted in QueueFS. On restart, `SemanticProcessor` detects that the `lifecycle_lock_handle_id` handle is missing from the in-memory LockManager and re-acquires a TreeLock.+**Server restart recovery**: `SemanticMsg` and its `SemanticPlan` are persisted+in QueueFS. On restart, `SemanticProcessor` detects that the+`lifecycle_lock_handle_id` handle is missing from the in-memory LockManager and+re-acquires a TreeLock before continuing derived work.### Derived Semantic Files (.abstract.md / .overview.md)diff --git a/docs/en/configuration/01-server.md b/docs/en/configuration/01-server.mdindex f5dca04a1..ecd510fad 100644--- a/docs/en/configuration/01-server.md+++ b/docs/en/configuration/01-server.md@@ -185,6 +185,9 @@ Search and Find requests default to `limit: 10`; override the limit on each API},"vectordb": {"backend": "local"+ },+ "parse_output": {+ "mode": "agfs"}}}@@ -198,10 +201,17 @@ Search and Find requests default to `limit: 10`; override the limit on each API| `agfs.backend` | `local`, `memory`, `s3` | `local` | File and metadata backend || `vectordb.backend` | `local`, `cuvs`, `http`, `volcengine`, `vikingdb` | `local` | Vector database backend || `vectordb.dimension` | integer | follows Embedding | Vector collection dimension |+| `parse_output.mode` | `agfs`, `local` | `agfs` | Backend for intermediate parser artifacts |+| `parse_output.local_root` | path or `null` | system temp directory | Root directory used by local parser artifacts || `skip_process_lock` | boolean | `false` | Skip the workspace process lock; use only when accepting concurrent-write risk |Remote backends also require endpoint, bucket/collection, credentials, and timeout fields. See [Configuration](../guides/01-configuration.md#storage) for complete examples.+`parse_output.mode=local` avoids writing parser intermediates to shared AGFS.+The same worker must commit the required bytes to the formal resource tree before+enqueueing downstream work. Artifacts are temporary and are removed after the+content commit; provision `local_root` with enough space for concurrent imports.+## Queue Worker Settings### `queue_workers.external_parse`@@ -217,9 +227,10 @@ This setting controls queue-job concurrency. It is separate from `vlm.media.max_| Field | Type | Default | Description ||---|---|---:|---|| `max_concurrent` | integer | `4` | Number of complete AddResource jobs consumed concurrently; must be greater than `0`; requires a server restart after changes |+| `file_operation_concurrency` | integer | `16` | Maximum concurrent file-level commit and fallback comparison operations within one AddResource job; must be greater than `0`; requires a server restart after changes || `file_vectorization_concurrency` | integer | `8` | Number of files concurrently read, prepared, and enqueued within one directory AddResource job when `processing_mode="vectors_only"`; must be greater than `0`; values above the internal safety limit of `64` are capped; requires a server restart after changes |-`max_concurrent` controls independent AddResource jobs, while `file_vectorization_concurrency` controls files within one vectors-only directory job. It does not affect single-file resources or `semantic_and_vectors` processing.+`max_concurrent` controls independent AddResource jobs. `file_operation_concurrency` controls file commit and fallback comparison work within one AddResource job, while `file_vectorization_concurrency` controls files within one vectors-only directory job.### `queue_workers.session_commit`diff --git a/docs/en/guides/07-operation-telemetry.md b/docs/en/guides/07-operation-telemetry.mdindex cb93cee70..482074347 100644--- a/docs/en/guides/07-operation-telemetry.md+++ b/docs/en/guides/07-operation-telemetry.md@@ -227,14 +227,21 @@ This group appears on resource ingestion operations such as `resources.add_resou| Field | Meaning || --- | --- |-| `summary.resource.request.duration_ms` | Total request-side duration for the add-resource flow |-| `summary.resource.process.duration_ms` | Duration of the main resource-processing flow |-| `summary.resource.process.parse.duration_ms` | Time spent parsing the resource |-| `summary.resource.process.parse.warnings_count` | Number of parse warnings |-| `summary.resource.process.finalize.duration_ms` | Time spent finalizing the resource tree |-| `summary.resource.process.summarize.duration_ms` | Time spent on summarize/vectorize processing |-| `summary.resource.wait.duration_ms` | Time spent waiting for downstream processing when `wait=true` |-| `summary.resource.watch.duration_ms` | Time spent creating, updating, or removing watch tasks |+| `summary.resource.total.duration_ms` | Total duration of the add-resource operation |+| `summary.resource.source_execute.duration_ms` | Time spent executing the source operation |+| `summary.resource.source_prepare.duration_ms` | Time spent preparing a durable source |+| `summary.resource.parse_artifact.duration_ms` | Time spent producing the parser artifact |+| `summary.resource.target_resolve.duration_ms` | Time spent resolving the final resource target |+| `summary.resource.update_plan.duration_ms` | Total time spent building the incremental update plan |+| `summary.resource.update_plan.artifact_inventory.duration_ms` | Time spent scanning and normalizing the parser artifact |+| `summary.resource.update_plan.rnfv_snapshot.duration_ms` | Time spent reading the R/N/F/V snapshot |+| `summary.resource.update_plan.diff_and_compile.duration_ms` | Time spent resolving the diff and compiling actions |+| `summary.resource.content_commit.duration_ms` | Time spent applying content actions to the formal tree |+| `summary.resource.derived_enqueue.duration_ms` | Time spent enqueueing semantic and index work |+| `summary.resource.semantic.queue_wait.duration_ms` | Time semantic work spent waiting in QueueFS |+| `summary.resource.semantic.execute.duration_ms` | Time spent executing semantic work |+| `summary.resource.embedding.queue_wait.duration_ms` | Time embedding work spent waiting in QueueFS |+| `summary.resource.embedding.execute.duration_ms` | Time spent executing embedding work || `summary.resource.flags.wait` | Whether the request used `wait=true` || `summary.resource.flags.build_index` | Whether the request enabled `build_index` || `summary.resource.flags.summarize` | Whether the request explicitly enabled `summarize` |@@ -247,15 +254,17 @@ This group appears when OpenViking waits for queue-backed work to complete.| Field | Meaning || --- | --- || `summary.queue.semantic.processed` | Number of semantic-queue messages processed |+| `summary.queue.semantic.requeue_count` | Number of semantic messages re-enqueued for retry || `summary.queue.semantic.error_count` | Number of semantic-queue errors || `summary.queue.embedding.processed` | Number of embedding-queue messages processed |+| `summary.queue.embedding.requeue_count` | Number of embedding messages re-enqueued for retry || `summary.queue.embedding.error_count` | Number of embedding-queue errors |### `summary.semantic_nodes`| Field | Meaning || --- | --- |-| `summary.semantic_nodes.total` | Total DAG or semantic-node count |+| `summary.semantic_nodes.total` | Total semantic-tree node count || `summary.semantic_nodes.done` | Number of completed nodes || `summary.semantic_nodes.pending` | Number of pending nodes || `summary.semantic_nodes.running` | Number of nodes still running |diff --git a/docs/zh/concepts/09-transaction.md b/docs/zh/concepts/09-transaction.mdindex 96fecfca5..ff3d17665 100644--- a/docs/zh/concepts/09-transaction.md+++ b/docs/zh/concepts/09-transaction.md@@ -132,47 +132,35 @@ Memory 提取是幂等的,从同一个 archive 重新提取会得到相同结| 文件从临时目录移到正式目录后崩溃 -> 文件存在但永远搜不到 | 首次添加与增量更新分离为两条独立路径 || 资源已落盘但语义处理/向量化还在跑时被 rm 删除 -> 处理白跑 | 生命周期 TreeLock,从落盘持续到处理完成 |-**首次添加**(target 不存在)— 在 `ResourceProcessor.process_resource` Phase 3.5 中处理:+**首次添加和增量更新**使用同一条计划提交路径:```-1. 获取 TreeLock,锁 final_uri- - 如果 final_uri 目录不存在,先检查祖先/后代/同路径锁冲突- - 无冲突则创建 final_uri 目录,并在 final_uri/.path.ovlock 写 T 锁-2. 保留 temp 作为源目录,入队 SemanticMsg(uri=temp, target_uri=final_uri, lifecycle_lock_handle_id=...)-3. DAG 在 temp 上跑,完成后把 temp 内容同步到 final_uri- - final_uri 已经用于放锁文件,所以不做裸 agfs.mv(temp -> final_uri)-4. 清理临时目录-5. DAG 启动锁刷新循环(每 lock_expire/2 秒刷新锁 token 并更新 handle 活跃时间)-6. DAG 完成 + 所有 embedding 完成 -> 释放 TreeLock+1. 获取 final_uri 的资源锁。+2. 持锁构建 R/N/F/V 快照:+ - R:归一化后的请求意图+ - N:解析产物清单+ - F:当前正式资源树+ - V:当前向量记录;build_index=false 时跳过+3. 编译 ContextUpdatePlan。+4. 同步把计划中的内容动作提交到 final_uri。+5. 清理 parser artifact。+6. 入队直接索引动作;需要语义处理时,再入队携带剩余 SemanticPlan 的 SemanticMsg。+7. 将资源锁交接给语义处理;没有语义任务时直接释放。```-如果本次调用关闭了摘要和索引(没有下游 DAG 接管),则在同一把 TreeLock-里把 temp 目录内容复制到 `final_uri`,清理 temp,然后释放锁。这里不调用-`VikingFS.mv(temp, final_uri, lock_handle=handle)`,避免移动逻辑清理目录锁文件。+因此正式内容树会先于 semantic 和 embedding 工作更新。内容提交成功后,+派生摘要和向量可能短暂落后,并由队列任务补齐;队列不再负责把 parser+临时树复制到正式树。此期间 `rm` 尝试获取同路径 TreeLock 会失败,抛出 `ResourceBusyError`。-**增量更新**(target 已存在)— temp 保持不动:--```-1. 获取 TreeLock,锁 target_uri(保护已有资源)-2. 入队 SemanticMsg(uri=temp, target_uri=final, lifecycle_lock_handle_id=...)-3. DAG 在 temp 上跑,启动锁刷新循环-4. DAG 完成后触发 sync_diff_callback 或 move_temp_to_target_callback-5. callback 执行完毕 -> 释放 TreeLock-```--注意:DAG callback 不在外层加锁。每个 `VikingFS.rm` 和 `VikingFS.mv` 内部各自有独立锁保护。外层锁会与内部锁冲突导致死锁。--首次添加和增量更新都只持有 `TreeLock(resource_dir)`。这里不再做-`ExactPathLock(resource_dir) -> TreeLock(resource_dir)` 的锁转交,避免两种锁复用-同一个 `.path.ovlock` 时出现释放顺序错误。-自动命名由资源层处理,不属于锁服务:`ResourceProcessor` 先用 `exists(candidate_uri)`判断候选目录是否已占用;已存在则尝试 `_1`、`_2` 后缀。候选目录不存在时才尝试获取该目录的 `TreeLock`,且不等待;如果同名正在被并发请求处理,就直接尝试下一个后缀。-**服务重启恢复**:SemanticMsg 持久化在 QueueFS 中。重启后 `SemanticProcessor` 发现 `lifecycle_lock_handle_id` 对应的 handle 不在内存中,会重新获取 TreeLock。+**服务重启恢复**:`SemanticMsg` 及其中的 `SemanticPlan` 持久化在 QueueFS+中。重启后 `SemanticProcessor` 发现 `lifecycle_lock_handle_id` 对应的 handle+不在内存中,会重新获取 TreeLock 后继续派生处理。### 派生语义文件(.abstract.md / .overview.md)diff --git a/docs/zh/configuration/01-server.md b/docs/zh/configuration/01-server.mdindex c6194f9a3..7d81bec08 100644--- a/docs/zh/configuration/01-server.md+++ b/docs/zh/configuration/01-server.md@@ -185,6 +185,9 @@ Search 和 Find 请求的默认 `limit` 为 `10`,可以在每次 API 或 SDK},"vectordb": {"backend": "local"+ },+ "parse_output": {+ "mode": "agfs"}}}@@ -198,10 +201,16 @@ Search 和 Find 请求的默认 `limit` 为 `10`,可以在每次 API 或 SDK| `agfs.backend` | `local`、`memory`、`s3` | `local` | 文件与元数据存储后端 || `vectordb.backend` | `local`、`cuvs`、`http`、`volcengine`、`vikingdb` | `local` | 向量数据库后端 || `vectordb.dimension` | integer | 跟随 Embedding | 向量集合维度 |+| `parse_output.mode` | `agfs`、`local` | `agfs` | parser 中间产物的存储后端 |+| `parse_output.local_root` | 路径或 `null` | 系统临时目录 | local parser artifact 的根目录 || `skip_process_lock` | boolean | `false` | 是否跳过 workspace 进程锁;仅在明确接受并发写风险时启用 |远程存储后端还需要配置 endpoint、bucket/collection、鉴权和超时等字段。完整后端示例见[配置指南](../guides/01-configuration.md#storage)。+`parse_output.mode=local` 可避免把 parser 中间产物写入共享 AGFS。当前 worker+必须在下游任务入队前把所需字节提交到正式资源树。产物会在内容提交后清理;+请为 `local_root` 预留足够空间以容纳并发导入。+## 队列 Worker 配置### `queue_workers.external_parse`@@ -217,9 +226,10 @@ Search 和 Find 请求的默认 `limit` 为 `10`,可以在每次 API 或 SDK| 字段 | 类型 | 默认值 | 说明 ||---|---|---:|---|| `max_concurrent` | integer | `4` | 同时消费的完整 AddResource 作业数,必须大于 `0`;修改后需重启服务 |+| `file_operation_concurrency` | integer | `16` | 单个 AddResource 作业内文件级提交和 fallback 比较操作的最大并发数,必须大于 `0`;修改后需重启服务 || `file_vectorization_concurrency` | integer | `8` | 当目录 AddResource 使用 `processing_mode="vectors_only"` 时,单个作业内并发读取、准备并入队的文件数,必须大于 `0`;超过内部安全上限 `64` 的值会被截断;修改后需重启服务 |-`max_concurrent` 控制相互独立的 AddResource 作业并发,`file_vectorization_concurrency` 控制单个 vectors-only 目录作业内的文件并发。该配置不影响单文件资源或 `semantic_and_vectors` 处理。+`max_concurrent` 控制相互独立的 AddResource 作业并发,`file_operation_concurrency` 控制单个 AddResource 作业内文件提交和 fallback 比较操作的并发,`file_vectorization_concurrency` 控制单个 vectors-only 目录作业内的文件并发。### `queue_workers.session_commit`diff --git a/docs/zh/guides/07-operation-telemetry.md b/docs/zh/guides/07-operation-telemetry.mdindex 09616afed..a6fb04ce7 100644--- a/docs/zh/guides/07-operation-telemetry.md+++ b/docs/zh/guides/07-operation-telemetry.md@@ -221,14 +221,21 @@ admission,所以其 `gpu_gate_queue` 为零;worker 侧等待仍计入 `batch| 字段 | 含义 || --- | --- |-| `summary.resource.request.duration_ms` | add-resource 请求主流程总耗时 |-| `summary.resource.process.duration_ms` | 资源处理主流程耗时 |-| `summary.resource.process.parse.duration_ms` | 资源解析阶段耗时 |-| `summary.resource.process.parse.warnings_count` | 解析阶段 warning 数量 |-| `summary.resource.process.finalize.duration_ms` | 资源树 finalize 阶段耗时 |-| `summary.resource.process.summarize.duration_ms` | summarize 或 vectorize 阶段耗时 |-| `summary.resource.wait.duration_ms` | `wait=true` 时等待下游处理完成的耗时 |-| `summary.resource.watch.duration_ms` | 创建、更新或移除 watch 任务的耗时 |+| `summary.resource.total.duration_ms` | add-resource 操作总耗时 |+| `summary.resource.source_execute.duration_ms` | 执行源操作的耗时 |+| `summary.resource.source_prepare.duration_ms` | 准备持久化 source 的耗时 |+| `summary.resource.parse_artifact.duration_ms` | 生成 parser artifact 的耗时 |+| `summary.resource.target_resolve.duration_ms` | 解析最终资源目标的耗时 |+| `summary.resource.update_plan.duration_ms` | 构建增量更新计划的总耗时 |+| `summary.resource.update_plan.artifact_inventory.duration_ms` | 扫描并归一化 parser artifact 的耗时 |+| `summary.resource.update_plan.rnfv_snapshot.duration_ms` | 读取 R/N/F/V 快照的耗时 |+| `summary.resource.update_plan.diff_and_compile.duration_ms` | 计算 diff 并编译动作的耗时 |+| `summary.resource.content_commit.duration_ms` | 把内容动作应用到正式树的耗时 |+| `summary.resource.derived_enqueue.duration_ms` | 将 semantic 和 index 工作入队的耗时 |+| `summary.resource.semantic.queue_wait.duration_ms` | semantic 工作在 QueueFS 中等待的耗时 |+| `summary.resource.semantic.execute.duration_ms` | semantic 工作执行耗时 |+| `summary.resource.embedding.queue_wait.duration_ms` | embedding 工作在 QueueFS 中等待的耗时 |+| `summary.resource.embedding.execute.duration_ms` | embedding 工作执行耗时 || `summary.resource.flags.wait` | 本次请求是否使用了 `wait=true` || `summary.resource.flags.build_index` | 本次请求是否启用了 `build_index` || `summary.resource.flags.summarize` | 本次请求是否显式启用了 `summarize` |@@ -241,15 +248,17 @@ admission,所以其 `gpu_gate_queue` 为零;worker 侧等待仍计入 `batch| 字段 | 含义 || --- | --- || `summary.queue.semantic.processed` | 已处理的 semantic queue 消息数 |+| `summary.queue.semantic.requeue_count` | 为重试而重新入队的 semantic 消息数 || `summary.queue.semantic.error_count` | semantic queue 错误数 || `summary.queue.embedding.processed` | 已处理的 embedding queue 消息数 |+| `summary.queue.embedding.requeue_count` | 为重试而重新入队的 embedding 消息数 || `summary.queue.embedding.error_count` | embedding queue 错误数 |### `summary.semantic_nodes`| 字段 | 含义 || --- | --- |-| `summary.semantic_nodes.total` | DAG 或语义节点总数 |+| `summary.semantic_nodes.total` | 语义树节点总数 || `summary.semantic_nodes.done` | 已完成节点数 || `summary.semantic_nodes.pending` | 待处理节点数 || `summary.semantic_nodes.running` | 正在处理中的节点数 |diff --git a/examples/grafana/openviking_demo_dashboard.json b/examples/grafana/openviking_demo_dashboard.jsonindex 024762161..da863a8c7 100644--- a/examples/grafana/openviking_demo_dashboard.json+++ b/examples/grafana/openviking_demo_dashboard.json@@ -5170,6 +5170,89 @@],"title": "Memory Operations / Type + Action","type": "timeseries"+ },+ {+ "collapsed": false,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 205+ },+ "id": 62,+ "panels": [],+ "title": "Incremental Resource Diagnostics",+ "type": "row"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${DS_PROMETHEUS}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "#73BF69",+ "value": 0+ },+ {+ "color": "#FF9830",+ "value": 1+ },+ {+ "color": "#F2495C",+ "value": 5+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 8,+ "w": 24,+ "x": 0,+ "y": 206+ },+ "id": 63,+ "options": {+ "legend": {+ "calcs": [+ "lastNotNull",+ "max"+ ],+ "displayMode": "table",+ "placement": "right",+ "showLegend": true,+ "sortBy": "max",+ "sortDesc": true+ },+ "tooltip": {+ "hideZeros": false,+ "mode": "multi",+ "sort": "desc"+ }+ },+ "pluginVersion": "12.4.1",+ "targets": [+ {+ "editorMode": "code",+ "expr": "histogram_quantile(0.95, sum by (le, stage) (rate(openviking_resource_stage_duration_seconds_bucket{stage=~\"semantic_queue_wait|semantic_execute|embedding_queue_wait|embedding_execute\"}[$__rate_interval])))",+ "legendFormat": "{{stage}}",+ "range": true,+ "refId": "A"+ }+ ],+ "title": "Semantic / Embedding Queue Timing / P95",+ "type": "timeseries"}],"preload": false,diff --git a/openviking/concurrency.py b/openviking/concurrency.pyindex e86f74dc8..408f5ff0c 100644--- a/openviking/concurrency.py+++ b/openviking/concurrency.py@@ -5,7 +5,59 @@import asyncioimport threadingfrom collections import deque+from collections.abc import Awaitable, Callable, Iterablefrom concurrent.futures import Future+from typing import TypeVar++T = TypeVar("T")+R = TypeVar("R")+++async def bounded_map(+ items: Iterable[T],+ worker: Callable[[T], Awaitable[R]],+ *,+ concurrency: int,+) -> list[R]:+ """Apply ``worker`` with a fixed number of tasks and preserve input order.++ A failed worker prevents new items from being claimed, then all active+ workers are joined before the original failure is re-raised.+ """+ if concurrency <= 0:+ raise ValueError("concurrency must be positive")++ iterator = iter(enumerate(items))+ cursor_lock = asyncio.Lock()+ stop = asyncio.Event()+ results: dict[int, R] = {}++ async def next_item() -> tuple[int, T] | None:+ async with cursor_lock:+ if stop.is_set():+ return None+ try:+ return next(iterator)+ except StopIteration:+ return None++ async def run_worker() -> None:+ while item := await next_item():+ index, value = item+ try:+ results[index] = await worker(value)+ except BaseException:+ stop.set()+ raise++ workers = [asyncio.create_task(run_worker()) for _ in range(concurrency)]+ try:+ await asyncio.gather(*workers)+ except BaseException:+ stop.set()+ await asyncio.gather(*workers, return_exceptions=True)+ raise+ return [results[index] for index in range(len(results))]class AsyncSemaphore:diff --git a/openviking/core/context.py b/openviking/core/context.pyindex 845df6346..b20d8f575 100644--- a/openviking/core/context.py+++ b/openviking/core/context.py@@ -80,6 +80,7 @@ class Context:account_id: Optional[str] = None,owner_user_id: Optional[str] = None,owner_space: Optional[str] = None,+ md5: Optional[str] = None,id: Optional[str] = None,):"""@@ -110,6 +111,10 @@ class Context:owner_user_id if owner_user_id is not None else owner_fields["owner_user_id"])self.owner_space = owner_space or owner_fields["owner_user_id"] or ""+ # md5 of the final stored bytes for this URI; None/"" means unknown (old+ # records or non-file records), and incremental diff falls back to reading+ # bytes rather than assuming equality.+ self.md5 = md5 or ""self.vector: Optional[List[float]] = Noneself.vectorize = Vectorize(abstract)@@ -171,6 +176,10 @@ class Context:}if self.level is not None:data["level"] = int(self.level)+ # Only emit md5 when known so we never overwrite an existing fingerprint+ # with an empty string via partial update.+ if self.md5:+ data["md5"] = self.md5if self.user:data["user"] = self.user.to_dict()@@ -229,6 +238,7 @@ class Context:account_id=data.get("account_id"),owner_user_id=data.get("owner_user_id"),owner_space=data.get("owner_space"),+ md5=data.get("md5"),)obj.id = data.get("id", obj.id)obj.vector = data.get("vector")diff --git a/openviking/parse/base.py b/openviking/parse/base.pyindex 54f065686..5e934f8c4 100644--- a/openviking/parse/base.py+++ b/openviking/parse/base.py@@ -14,7 +14,7 @@ from pathlib import Pathfrom typing import TYPE_CHECKING, Any, Dict, List, Optionalif TYPE_CHECKING:- pass+ from openviking.parse.output import ParseArtifactRef# ============================================================================# Common utility functions@@ -291,6 +291,10 @@ class ParseResult:# Temporary directory path (for v4.0 architecture)temp_dir_path: Optional[str] = None # e.g., "/tmp/openviking_parse_a1b2c3d4"+ # Serializable handle to the artifact backing ``temp_dir_path``. New parsers+ # set it explicitly; legacy AGFS-only results may derive it from their temp URI.+ artifact_ref: Optional["ParseArtifactRef"] = None+# Core metadata fieldssource_format: Optional[str] = None # File format (e.g., "pdf", "markdown")parser_name: Optional[str] = None # Parser name (e.g., "PDFParser")@@ -306,6 +310,27 @@ class ParseResult:"""Check if parsing was successful."""return len(self.warnings) == 0+ def ensure_artifact_ref(self) -> Optional["ParseArtifactRef"]:+ """Return the artifact ref, deriving an AGFS one from ``temp_dir_path``.++ Legacy parsers may still record an AGFS temp URI without a ref. A local+ filesystem path is ambiguous and must always carry an explicit ref.+ """+ if self.artifact_ref is not None:+ return self.artifact_ref+ if not self.temp_dir_path:+ return None+ if not self.temp_dir_path.startswith("viking://temp/"):+ raise ValueError("parse result with a non-AGFS temp path must provide artifact_ref")+ from openviking.parse.output import ParseArtifactRef++ self.artifact_ref = ParseArtifactRef(+ backend="agfs",+ root=self.temp_dir_path,+ root_type="dir",+ )+ return self.artifact_ref+def get_all_nodes(self) -> List[ResourceNode]:"""Get all nodes in the tree (flattened)."""nodes = []diff --git a/openviking/parse/output.py b/openviking/parse/output.pynew file mode 100644index 000000000..4f9d13da8--- /dev/null+++ b/openviking/parse/output.py@@ -0,0 +1,669 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Parse output store abstraction.++Parsers write their intermediate artifacts (the temp tree that TreeBuilder later+persists into the final resource location) through a small store interface rather+than reaching for the global VikingFS singleton directly. This decouples "which+backend holds the artifact" from "how a parser lays out its output".++The AGFS implementation preserves the original shared-temp behavior, while the+local implementation keeps parser artifacts on a configured shared local path.++Design notes:+- Business rules (filtering, encoding, flatten, name conflicts, sidecar+ retention) live in the parsers / shared helpers, NOT in the store. A store+ only moves bytes and directory entries for a single artifact.+- :class:`ParseArtifactRef` is the serializable handle that crosses the queue in+ place of a bare ``viking://temp/...`` string; runtime store objects never do.+- Relative paths inside an artifact are validated with+ :func:`sanitize_relative_viking_path` so a parser cannot escape its root.+"""++from __future__ import annotations++import asyncio+import json+import shutil+import uuid+from abc import ABC, abstractmethod+from dataclasses import dataclass+from pathlib import Path+from typing import Any, Dict, List, Literal, Optional++from openviking.utils.content_hash import content_md5+from openviking.utils.path_safety import safe_join_viking_uri, sanitize_relative_viking_path++_BACKENDS = frozenset({"agfs", "local"})+_ROOT_TYPES = frozenset({"dir", "file"})++# Sidecar at the artifact root mapping each business file's artifact-relative+# path to the md5 of its final (encoding-normalized) bytes. The incremental diff+# reads this so it can compare fingerprints without re-reading file contents.+ARTIFACT_MANIFEST_NAME = ".artifact_manifest.json"+++@dataclass(frozen=True)+class ParseArtifactRef:+ """Serializable handle to one parse artifact.++ ``root`` is backend-scoped (an AGFS ``viking://temp/<uuid>`` URI, or a local+ absolute directory). ``resource_rel`` is the artifact-relative path of the+ resource root inside ``root`` (empty when the root is the resource itself).+ """++ backend: str+ root: str+ resource_rel: str = ""+ root_type: str = "dir"++ def to_dict(self) -> dict[str, Any]:+ return {+ "backend": self.backend,+ "root": self.root,+ "resource_rel": self.resource_rel,+ "root_type": self.root_type,+ }++ @classmethod+ def from_dict(cls, data: dict[str, Any]) -> "ParseArtifactRef":+ if not isinstance(data, dict):+ raise ValueError("parse artifact ref must be an object")+ backend = data.get("backend")+ root = data.get("root")+ resource_rel = data.get("resource_rel") or ""+ root_type = data.get("root_type") or "dir"+ if backend not in _BACKENDS:+ raise ValueError(f"parse artifact ref backend must be one of {sorted(_BACKENDS)}")+ if not isinstance(root, str) or not root:+ raise ValueError("parse artifact ref root must be a non-empty string")+ if not isinstance(resource_rel, str):+ raise ValueError("parse artifact ref resource_rel must be a string")+ if root_type not in _ROOT_TYPES:+ raise ValueError(f"parse artifact ref root_type must be one of {sorted(_ROOT_TYPES)}")+ return cls(+ backend=backend,+ root=root,+ resource_rel=resource_rel,+ root_type=root_type,+ )+++@dataclass(frozen=True)+class ArtifactEntry:+ """One directory entry inside an artifact, with an artifact-relative path."""++ name: str+ rel_path: str+ is_dir: bool+++class ParseOutputStore(ABC):+ """Backend-agnostic storage for a parser's intermediate artifacts.++ Methods take a :class:`ParseArtifactRef` plus an artifact-relative path; the+ store resolves the absolute location. Only byte and directory operations live+ here — parsers keep owning layout and filtering decisions.+ """++ backend: str++ @abstractmethod+ async def create_artifact(self, *, root_type: str = "dir") -> ParseArtifactRef:+ """Allocate a fresh artifact root and return its reference."""++ @abstractmethod+ async def mkdir(self, ref: ParseArtifactRef, rel_path: str = "") -> None: ...++ @abstractmethod+ async def write_bytes(self, ref: ParseArtifactRef, rel_path: str, content: bytes) -> None: ...++ @abstractmethod+ async def read_bytes(self, ref: ParseArtifactRef, rel_path: str) -> bytes: ...++ @abstractmethod+ async def list(self, ref: ParseArtifactRef, rel_path: str = "") -> List[ArtifactEntry]: ...++ @abstractmethod+ async def cleanup(self, ref: ParseArtifactRef) -> None:+ """Delete the artifact. Safe to call more than once."""++ @abstractmethod+ async def move_file(+ self,+ source_ref: ParseArtifactRef,+ source_rel: str,+ target_ref: ParseArtifactRef,+ target_rel: str,+ ) -> None:+ """Move one file between artifacts owned by this store."""++ # -- text convenience -------------------------------------------------+ async def write_text(+ self, ref: ParseArtifactRef, rel_path: str, content: str, *, encoding: str = "utf-8"+ ) -> None:+ await self.write_bytes(ref, rel_path, content.encode(encoding))++ async def read_text(+ self, ref: ParseArtifactRef, rel_path: str, *, encoding: str = "utf-8"+ ) -> str:+ return (await self.read_bytes(ref, rel_path)).decode(encoding)+++def _join_artifact_rel(parent: str, child: str) -> str:+ parent = (parent or "").strip("/")+ child = (child or "").strip("/")+ joined = f"{parent}/{child}" if parent and child else parent or child+ return sanitize_relative_viking_path(joined) if joined else ""+++class ParseArtifactWriter:+ """Request-scoped writer that owns one parse artifact and its manifest."""++ def __init__(self, store: ParseOutputStore, ref: ParseArtifactRef) -> None:+ if ref.backend != store.backend:+ raise ValueError(+ f"artifact backend {ref.backend!r} does not match store {store.backend!r}"+ )+ self.store = store+ self.ref = ref+ self._md5_by_rel: Dict[str, str] = {}+ self._finalized = False++ @classmethod+ async def create(+ cls, store: ParseOutputStore, *, root_type: str = "dir"+ ) -> "ParseArtifactWriter":+ return cls(store, await store.create_artifact(root_type=root_type))++ async def mkdir(self, rel_path: str = "") -> None:+ await self.store.mkdir(self.ref, self.relative_path(rel_path))++ def relative_path(self, path: str) -> str:+ """Normalize an artifact-relative path or a path rooted at this artifact."""+ value = str(path or "")+ root = self.ref.root.rstrip("/")+ if value == root:+ return ""+ if value.startswith(root + "/"):+ value = value[len(root) + 1 :]+ return _join_artifact_rel("", value)++ async def write_bytes(self, rel_path: str, content: bytes, *, md5: str | None = None) -> None:+ rel_path = self.relative_path(rel_path)+ if not rel_path or rel_path == ARTIFACT_MANIFEST_NAME:+ raise ValueError("artifact business file path must not be the manifest path")+ await self.store.write_bytes(self.ref, rel_path, content)+ self._md5_by_rel[rel_path] = md5 or content_md5(content)+ self._finalized = False++ async def write_text(self, rel_path: str, content: str, *, encoding: str = "utf-8") -> None:+ await self.write_bytes(rel_path, content.encode(encoding))++ async def finalize(self, *, resource_rel: str = "") -> ParseArtifactRef:+ if not self._finalized:+ await write_artifact_manifest(self.store, self.ref, self._md5_by_rel)+ self._finalized = True+ return ParseArtifactRef(+ backend=self.ref.backend,+ root=self.ref.root,+ resource_rel=_join_artifact_rel("", resource_rel),+ root_type=self.ref.root_type,+ )++ async def cleanup(self) -> None:+ await self.store.cleanup(self.ref)++ def record_md5(self, rel_path: str, md5: str) -> None:+ if md5:+ self._md5_by_rel[self.relative_path(rel_path)] = md5+ self._finalized = False+++async def create_parse_artifact_writer(+ store: ParseOutputStore | None = None,+ *,+ viking_fs: Any = None,+ ctx: Any = None,+ root_type: str = "dir",+) -> ParseArtifactWriter:+ """Create one request-scoped writer, defaulting to the AGFS backend."""+ resolved_store = store or AgfsParseOutputStore(viking_fs=viking_fs, ctx=ctx)+ return await ParseArtifactWriter.create(resolved_store, root_type=root_type)+++async def copy_artifact_tree(+ *,+ source_store: ParseOutputStore,+ source_ref: ParseArtifactRef,+ source_rel: str,+ target: ParseArtifactWriter,+ target_rel: str,+ allowed_hidden: frozenset[str] = frozenset(),+ flatten_single_output: bool = False,+) -> int:+ """Transfer one artifact subtree into another and return the file count."""+ source_rel = _join_artifact_rel("", source_rel)+ target_rel = _join_artifact_rel("", target_rel)+ source_manifest = await read_artifact_manifest(source_store, source_ref)+ copied = 0+ source_entries = [+ entry+ for entry in await source_store.list(source_ref, source_rel)+ if entry.rel_path != ARTIFACT_MANIFEST_NAME+ and (not entry.name.startswith(".") or entry.name in allowed_hidden)+ ]+ if flatten_single_output and len(source_entries) == 1 and source_entries[0].is_dir:+ wrapper = source_entries[0]+ wrapper_entries = [+ entry+ for entry in await source_store.list(source_ref, wrapper.rel_path)+ if not entry.name.startswith(".") or entry.name in allowed_hidden+ ]+ if len(wrapper_entries) == 1 and not wrapper_entries[0].is_dir:+ source_entries = [wrapper_entries[0]]++ async def _copy(src_dir: str, dst_dir: str, entries: List[ArtifactEntry] | None = None) -> None:+ nonlocal copied+ if dst_dir:+ await target.mkdir(dst_dir)+ for entry in (+ entries if entries is not None else await source_store.list(source_ref, src_dir)+ ):+ if entry.rel_path == ARTIFACT_MANIFEST_NAME:+ continue+ if entry.name.startswith(".") and entry.name not in allowed_hidden:+ continue+ child_dst = _join_artifact_rel(dst_dir, entry.name)+ if entry.is_dir:+ await _copy(entry.rel_path, child_dst)+ else:+ md5 = source_manifest.get(entry.rel_path, "")+ if source_store is target.store:+ if not md5:+ md5 = content_md5(await source_store.read_bytes(source_ref, entry.rel_path))+ await target.store.move_file(source_ref, entry.rel_path, target.ref, child_dst)+ target.record_md5(child_dst, md5)+ else:+ await target.write_bytes(+ child_dst,+ await source_store.read_bytes(source_ref, entry.rel_path),+ md5=md5,+ )+ copied += 1++ await _copy(source_rel, target_rel, source_entries)+ return copied+++class AgfsParseOutputStore(ParseOutputStore):+ """Artifact store backed by the global VikingFS ``viking://temp`` space.++ Every operation forwards to the VikingFS singleton exactly as parsers did+ before this abstraction existed, so enabling it changes no behaviour.+ """++ backend = "agfs"++ def __init__(self, viking_fs: Any = None, ctx: Any = None) -> None:+ self._viking_fs = viking_fs+ self._ctx = ctx+ self._cleaned: set[str] = set()++ def _fs(self) -> Any:+ if self._viking_fs is not None:+ return self._viking_fs+ from openviking.storage.viking_fs import get_viking_fs++ return get_viking_fs()++ @staticmethod+ def _resolve(ref: ParseArtifactRef, rel_path: str) -> str:+ rel = (rel_path or "").strip("/")+ if not rel:+ return ref.root+ return safe_join_viking_uri(ref.root, rel)++ async def create_artifact(self, *, root_type: str = "dir") -> ParseArtifactRef:+ if root_type not in _ROOT_TYPES:+ raise ValueError(f"root_type must be one of {sorted(_ROOT_TYPES)}")+ fs = self._fs()+ root = fs.create_temp_uri(ctx=self._ctx) if self._ctx is not None else fs.create_temp_uri()+ return ParseArtifactRef(backend=self.backend, root=root, root_type=root_type)++ async def mkdir(self, ref: ParseArtifactRef, rel_path: str = "") -> None:+ kwargs = {"ctx": self._ctx} if self._ctx is not None else {}+ await self._fs().mkdir(self._resolve(ref, rel_path), exist_ok=True, **kwargs)++ async def write_bytes(self, ref: ParseArtifactRef, rel_path: str, content: bytes) -> None:+ fs = self._fs()+ uri = self._resolve(ref, rel_path)+ kwargs = {"ctx": self._ctx} if self._ctx is not None else {}+ if hasattr(fs, "write_file_bytes"):+ await fs.write_file_bytes(uri, content, **kwargs)+ else:+ await fs.write_file(uri, content.decode("utf-8"), **kwargs)++ async def read_bytes(self, ref: ParseArtifactRef, rel_path: str) -> bytes:+ fs = self._fs()+ uri = self._resolve(ref, rel_path)+ kwargs = {"ctx": self._ctx} if self._ctx is not None else {}+ if hasattr(fs, "read_file_bytes"):+ return await fs.read_file_bytes(uri, **kwargs)+ content = await fs.read(uri, **kwargs)+ return content.encode("utf-8") if isinstance(content, str) else content++ async def list(self, ref: ParseArtifactRef, rel_path: str = "") -> List[ArtifactEntry]:+ from openviking.storage.viking_fs import LS_ALL_NODES++ base = self._resolve(ref, rel_path)+ kwargs = {"ctx": self._ctx} if self._ctx is not None else {}+ entries = await self._fs().ls(base, show_all_hidden=True, node_limit=LS_ALL_NODES, **kwargs)+ rel_prefix = (rel_path or "").strip("/")+ result: List[ArtifactEntry] = []+ for entry in entries:+ name = entry.get("name")+ if not name or name in {".", ".."}:+ continue+ child_rel = f"{rel_prefix}/{name}" if rel_prefix else str(name)+ result.append(+ ArtifactEntry(+ name=str(name),+ rel_path=child_rel,+ is_dir=bool(entry.get("isDir")),+ )+ )+ return result++ async def cleanup(self, ref: ParseArtifactRef) -> None:+ if ref.root in self._cleaned:+ return+ self._cleaned.add(ref.root)+ kwargs = {"ctx": self._ctx} if self._ctx is not None else {}+ await self._fs().delete_temp(ref.root, **kwargs)++ async def move_file(+ self,+ source_ref: ParseArtifactRef,+ source_rel: str,+ target_ref: ParseArtifactRef,+ target_rel: str,+ ) -> None:+ kwargs = {"ctx": self._ctx} if self._ctx is not None else {}+ await self._fs().move_file(+ self._resolve(source_ref, source_rel),+ self._resolve(target_ref, target_rel),+ **kwargs,+ )+++class LocalParseOutputStore(ParseOutputStore):+ """Artifact store backed by a local directory tree.++ Artifacts stay on the parsing worker until add-resources commits the required+ bytes to the formal AGFS tree. They are not handed to asynchronous semantic+ workers on the SemanticPlan path.++ Case-only name collisions are detected explicitly rather than relying on the+ host filesystem's case sensitivity, so behaviour matches across platforms.+ """++ backend = "local"++ def __init__(self, local_root: str) -> None:+ if not local_root:+ raise ValueError("LocalParseOutputStore requires a local_root")+ self._root = Path(local_root).expanduser().resolve()+ self._root.mkdir(parents=True, exist_ok=True)++ def _resolve(self, ref: ParseArtifactRef, rel_path: str) -> Path:+ artifact_root = Path(ref.root).resolve()+ # The artifact root must stay inside the configured store root.+ if artifact_root != self._root and self._root not in artifact_root.parents:+ raise ValueError("parse artifact root escapes the configured local_root")+ rel = (rel_path or "").strip("/")+ if not rel:+ return artifact_root+ safe_rel = sanitize_relative_viking_path(rel)+ target = artifact_root.joinpath(*Path(safe_rel).parts).resolve()+ if artifact_root != target and artifact_root not in target.parents:+ raise ValueError(f"resolved path escapes the artifact root: {rel_path}")+ return target++ @staticmethod+ def _check_case_conflict(target: Path) -> None:+ parent = target.parent+ if not parent.is_dir():+ return+ lower = target.name.casefold()+ for existing in parent.iterdir():+ if existing.name != target.name and existing.name.casefold() == lower:+ raise ValueError(+ f"case-only name conflict: {target.name} vs existing {existing.name}"+ )++ async def create_artifact(self, *, root_type: str = "dir") -> ParseArtifactRef:+ if root_type not in _ROOT_TYPES:+ raise ValueError(f"root_type must be one of {sorted(_ROOT_TYPES)}")+ artifact_root = self._root / f"artifact-{uuid.uuid4().hex}"+ await asyncio.to_thread(artifact_root.mkdir, parents=True, exist_ok=False)+ return ParseArtifactRef(+ backend=self.backend,+ root=str(artifact_root),+ root_type=root_type,+ )++ async def mkdir(self, ref: ParseArtifactRef, rel_path: str = "") -> None:+ target = self._resolve(ref, rel_path)+ await asyncio.to_thread(target.mkdir, parents=True, exist_ok=True)++ async def write_bytes(self, ref: ParseArtifactRef, rel_path: str, content: bytes) -> None:+ target = self._resolve(ref, rel_path)++ def _write() -> None:+ target.parent.mkdir(parents=True, exist_ok=True)+ self._check_case_conflict(target)+ target.write_bytes(content)++ await asyncio.to_thread(_write)++ async def read_bytes(self, ref: ParseArtifactRef, rel_path: str) -> bytes:+ target = self._resolve(ref, rel_path)+ return await asyncio.to_thread(target.read_bytes)++ async def list(self, ref: ParseArtifactRef, rel_path: str = "") -> List[ArtifactEntry]:+ base = self._resolve(ref, rel_path)++ def _scan() -> List[ArtifactEntry]:+ if not base.is_dir():+ return []+ rel_prefix = (rel_path or "").strip("/")+ entries: List[ArtifactEntry] = []+ for child in sorted(base.iterdir(), key=lambda p: p.name):+ child_rel = f"{rel_prefix}/{child.name}" if rel_prefix else child.name+ entries.append(+ ArtifactEntry(+ name=child.name,+ rel_path=child_rel,+ is_dir=child.is_dir(),+ )+ )+ return entries++ return await asyncio.to_thread(_scan)++ async def cleanup(self, ref: ParseArtifactRef) -> None:+ artifact_root = self._resolve(ref, "")+ await asyncio.to_thread(shutil.rmtree, artifact_root, ignore_errors=True)++ async def move_file(+ self,+ source_ref: ParseArtifactRef,+ source_rel: str,+ target_ref: ParseArtifactRef,+ target_rel: str,+ ) -> None:+ source = self._resolve(source_ref, source_rel)+ target = self._resolve(target_ref, target_rel)++ def _move() -> None:+ target.parent.mkdir(parents=True, exist_ok=True)+ self._check_case_conflict(target)+ shutil.move(str(source), str(target))++ await asyncio.to_thread(_move)+++@dataclass(frozen=True)+class ResolvedDocRoot:+ """The single document root located inside a parse artifact.++ ``doc_name`` is the original (un-sanitized) entry name; ``doc_rel`` is its+ artifact-relative path (the file itself when flattened to a single file).+ """++ doc_name: str+ doc_rel: str+ root_is_file: bool+++async def resolve_artifact_doc_root(+ store: ParseOutputStore,+ ref: ParseArtifactRef,+ *,+ flatten_single_file: bool = False,+) -> ResolvedDocRoot:+ """Locate the single document root inside an artifact, backend-agnostically.++ Parsers lay out both AGFS and local artifacts the same way — exactly one+ document directory under the root — so this structural walk is shared. It+ only reads the artifact via ``store``; resolving the final target URI against+ the live resource tree is a separate, VikingFS-only concern handled by the+ caller (TreeBuilder.resolve_target_uri).+ """+ top = [+ entry+ for entry in await store.list(ref, "")+ if entry.name not in {".", "..", ARTIFACT_MANIFEST_NAME}+ ]+ doc_dirs = [e for e in top if e.is_dir]+ if flatten_single_file and not doc_dirs:+ files = [entry for entry in top if not entry.is_dir]+ if len(files) == 1:+ return ResolvedDocRoot(+ doc_name=files[0].name,+ doc_rel=files[0].rel_path,+ root_is_file=True,+ )+ if len(doc_dirs) != 1:+ raise ValueError(+ f"expected exactly 1 document directory in artifact {ref.root}, found {len(doc_dirs)}"+ )++ doc_entry = doc_dirs[0]+ doc_name = doc_entry.name+ doc_rel = doc_entry.rel_path+ root_is_file = False++ if flatten_single_file:+ children = [+ entry+ for entry in await store.list(ref, doc_rel)+ if entry.name not in {".", "..", ARTIFACT_MANIFEST_NAME}+ ]+ if len(children) == 1 and not children[0].is_dir:+ doc_name = children[0].name+ doc_rel = children[0].rel_path+ root_is_file = True++ return ResolvedDocRoot(doc_name=doc_name, doc_rel=doc_rel, root_is_file=root_is_file)+++def build_parse_output_store(+ *,+ viking_fs: Any = None,+ backend: Optional[Literal["agfs", "local"]] = None,+ local_root: Optional[str] = None,+) -> ParseOutputStore:+ """Return the configured artifact store.++ ``backend`` defaults to AGFS. The local backend requires ``local_root``.+ """+ if backend in (None, "agfs"):+ return AgfsParseOutputStore(viking_fs=viking_fs)+ if backend == "local":+ if not local_root:+ raise ValueError("local parse output backend requires storage.parse_output.local_root")+ return LocalParseOutputStore(local_root=local_root)+ raise ValueError(f"unsupported parse output backend: {backend}")+++def store_for_artifact_ref(+ ref: ParseArtifactRef,+ *,+ viking_fs: Any = None,+ ctx: Any = None,+) -> ParseOutputStore:+ """Return the store that owns ``ref``, resolving the local root from config.++ Callers that receive a serialized ref across the queue (semantic worker,+ add-resource cleanup, post-process) must reconstruct the matching backend;+ this centralizes that mapping so the local-root lookup lives in one place.+ """+ if ref.backend == "local":+ from openviking_cli.utils.config import get_openviking_config++ parse_output = get_openviking_config().storage.parse_output+ return build_parse_output_store(+ backend="local", local_root=parse_output.resolved_local_root()+ )+ return AgfsParseOutputStore(viking_fs=viking_fs, ctx=ctx)+++async def read_artifact_manifest(store: ParseOutputStore, ref: ParseArtifactRef) -> Dict[str, str]:+ """Return the md5 manifest (artifact-relative path -> md5), or empty if absent.++ A missing/unreadable/malformed manifest yields ``{}`` so the incremental diff+ falls back to comparing file bytes instead of assuming equality.+ """+ try:+ raw = await store.read_bytes(ref, ARTIFACT_MANIFEST_NAME)+ loaded = json.loads(raw.decode("utf-8"))+ except Exception:+ return {}+ if not isinstance(loaded, dict):+ return {}+ return {str(key): str(value) for key, value in loaded.items()}+++async def write_artifact_manifest(+ store: ParseOutputStore, ref: ParseArtifactRef, md5_by_rel: Dict[str, str]+) -> None:+ """Persist the md5 manifest at the artifact root using the shared encoding."""+ await store.write_bytes(+ ref,+ ARTIFACT_MANIFEST_NAME,+ json.dumps(md5_by_rel, ensure_ascii=False, sort_keys=True).encode("utf-8"),+ )+++# Re-exported so callers importing the sanitizer alongside the store stay in one+# module; keeps parser edits from sprinkling path_safety imports everywhere.+__all__ = [+ "ARTIFACT_MANIFEST_NAME",+ "ArtifactEntry",+ "AgfsParseOutputStore",+ "LocalParseOutputStore",+ "ParseArtifactRef",+ "ParseArtifactWriter",+ "ParseOutputStore",+ "ResolvedDocRoot",+ "build_parse_output_store",+ "copy_artifact_tree",+ "create_parse_artifact_writer",+ "read_artifact_manifest",+ "resolve_artifact_doc_root",+ "sanitize_relative_viking_path",+ "store_for_artifact_ref",+ "write_artifact_manifest",+]diff --git a/openviking/parse/parser_router.py b/openviking/parse/parser_router.pyindex 69512653d..308d8e81d 100644--- a/openviking/parse/parser_router.py+++ b/openviking/parse/parser_router.py@@ -136,9 +136,11 @@ class ParserRouter:if use_understanding:if isinstance(source, LocalResource):- kwargs["source_name"] = source.meta["resolved_name"]+ kwargs["source_name"] = source.meta.get("resolved_name") or Path(source_path).namekwargs["resolved_extension"] = (- kwargs.get("resolved_extension") or source.meta["resolved_extension"]+ kwargs.get("resolved_extension")+ or source.meta.get("resolved_extension")+ or Path(source_path).suffix)display = source_pathif isinstance(source_path, str) and source_path.startswith(("http://", "https://")):diff --git a/openviking/parse/parsers/base_parser.py b/openviking/parse/parsers/base_parser.pyindex 85911c2b1..a8d67ca00 100644--- a/openviking/parse/parsers/base_parser.py+++ b/openviking/parse/parsers/base_parser.py@@ -109,15 +109,3 @@ class BaseParser(ABC):from openviking.storage.viking_fs import get_viking_fsreturn get_viking_fs()-- def _create_temp_uri(self) -> str:- """- Create a temporary URI for storing intermediate files during parsing.-- This is a common utility method for all parsers that follow the- three-phase parsing architecture.-- Returns:- Temporary URI string (e.g., "viking://temp/abc12345")- """- return self._get_viking_fs().create_temp_uri()diff --git a/openviking/parse/parsers/code/code.py b/openviking/parse/parsers/code/code.pyindex 4890cc285..be4c7f307 100644--- a/openviking/parse/parsers/code/code.py+++ b/openviking/parse/parsers/code/code.py@@ -125,6 +125,15 @@ class CodeRepositoryParser(BaseParser):"""start_time = time.time()source_path = Path(source)+ # Always work through a ParseOutputStore. Local mode threads one in via+ # kwargs; otherwise default to the AGFS-backed store so the parser has a+ # single, backend-agnostic write path (no viking_fs/store branching).+ output_store = kwargs.get("parse_output_store")+ if output_store is None:+ from openviking.parse.output import AgfsParseOutputStore++ output_store = AgfsParseOutputStore(viking_fs=self._get_viking_fs())+ artifact_ref = None# Check if source is already a local directory (should always be true)if not source_path.is_dir():@@ -160,24 +169,31 @@ class CodeRepositoryParser(BaseParser):local_dir = source_path- # 3. Create VikingFS temp URI- viking_fs = self._get_viking_fs()- temp_viking_uri = self._create_temp_uri()- # The structure in temp should be: viking://temp/{uuid}/repository/...- # Use simple name 'repository' for temp, TreeBuilder will rename it to org/repo later+ # 3. Allocate the artifact root through the store. Local mode writes+ # to a shared local dir; AGFS mode writes to viking://temp. Either way+ # the layout is <root>/repository/... and the parser code is identical.+ artifact_ref = await output_store.create_artifact(root_type="dir")+ temp_viking_uri = artifact_ref.roottarget_root_uri = f"{temp_viking_uri}/repository"- logger.info(f"Uploading to VikingFS: {target_root_uri}")+ logger.debug(f"Uploading code repository artifacts to: {target_root_uri}")- # 4. Upload to VikingFS (filtering on the fly)- file_count = await self._upload_directory(+ # 4. Upload to the artifact store (filtering on the fly)+ file_count, upload_failures = await self._upload_directory(local_dir,- target_root_uri,- viking_fs,+ output_store=output_store,+ artifact_ref=artifact_ref,ignore_dirs=kwargs.get("ignore_dirs"),include=kwargs.get("include"),exclude=kwargs.get("exclude"),)+ if upload_failures:+ detail = "; ".join(upload_failures[:5])+ if len(upload_failures) > 5:+ detail += f"; ... {len(upload_failures) - 5} more"+ raise RuntimeError(+ f"Failed to upload {len(upload_failures)} repository file(s): {detail}"+ )logger.info(f"Uploaded {file_count} files to {target_root_uri}")@@ -209,6 +225,7 @@ class CodeRepositoryParser(BaseParser):parse_time=time.time() - start_time,)result.temp_dir_path = temp_viking_uri # Points to parent of repo_name+ result.artifact_ref = artifact_refresult.meta["file_count"] = file_countresult.meta["repo_name"] = repo_nameif branch:@@ -220,6 +237,15 @@ class CodeRepositoryParser(BaseParser):except Exception as e:logger.error(f"Failed to parse repository {source}: {e}", exc_info=True)+ if artifact_ref is not None:+ try:+ await output_store.cleanup(artifact_ref)+ except Exception as cleanup_error:+ logger.warning(+ "Failed to clean repository parse artifact %s: %s",+ artifact_ref.root,+ cleanup_error,+ )# Use original URL for error case as well - still important for TreeBuilder# Even on failure, we want TreeBuilder to potentially get org/repo from the URLoriginal_source = kwargs.get("original_source") or source_meta.get("original_source")@@ -582,20 +608,20 @@ class CodeRepositoryParser(BaseParser):async def _upload_directory(self,local_dir: Path,- viking_uri_base: str,- viking_fs: Any,*,+ output_store: Any,+ artifact_ref: Any,ignore_dirs: Optional[Union[Set[str], List[str], str]] = None,include: Optional[str] = None,exclude: Optional[str] = None,- ) -> int:- """Recursively upload directory to VikingFS using shared upload utilities."""- count, _ = await upload_directory(+ ) -> Tuple[int, List[str]]:+ """Recursively upload the repository into the artifact store under repository/."""+ return await upload_directory(local_dir,- viking_uri_base,- viking_fs,+ "repository",+ store=output_store,+ artifact_ref=artifact_ref,ignore_dirs=ignore_dirs,include=include,exclude=exclude,)- return countdiff --git a/openviking/parse/parsers/directory.py b/openviking/parse/parsers/directory.pyindex d8abdfe28..886b8108a 100644--- a/openviking/parse/parsers/directory.py+++ b/openviking/parse/parsers/directory.py@@ -32,9 +32,15 @@ from openviking.parse.base import ()from openviking.parse.gitignore import GitignoreMatcherfrom openviking.parse.image_rewrite import IMAGE_MAPPINGS_FILENAME+from openviking.parse.output import (+ ParseArtifactWriter,+ copy_artifact_tree,+ create_parse_artifact_writer,+ store_for_artifact_ref,+)from openviking.parse.parsers.base_parser import BaseParserfrom openviking.parse.parsers.media.constants import MEDIA_EXTENSIONS-from openviking.parse.parsers.upload_utils import is_text_file+from openviking.parse.parsers.upload_utils import detect_and_convert_encoding, is_text_filefrom openviking.storage.viking_fs import LS_ALL_NODESfrom openviking_cli.exceptions import InvalidArgumentErrorfrom openviking_cli.utils.logger import get_logger@@ -146,7 +152,9 @@ class DirectoryParser(BaseParser):warnings: List[str] = []temp_uri: Optional[str] = Nonekeep_temp = False- pending_temp_uris: set[str] = set()+ pending_results: list[ParseResult] = []+ output_store = kwargs.get("parse_output_store")+ writer: Optional[ParseArtifactWriter] = Nonetry:# ── Phase 1: scan directory ───────────────────────────────@@ -309,11 +317,13 @@ class DirectoryParser(BaseParser):if use_understanding:understanding_jobs.append(job)- viking_fs = self._get_viking_fs()- temp_uri = self._create_temp_uri()+ viking_fs = self._get_viking_fs() if output_store is None else None+ writer = await create_parse_artifact_writer(output_store, viking_fs=viking_fs)+ output_store = writer.store+ temp_uri = writer.ref.roottarget_uri = f"{temp_uri}/{dir_name}"- await viking_fs.mkdir(temp_uri, exist_ok=True)- await viking_fs.mkdir(target_uri, exist_ok=True)+ await writer.mkdir()+ await writer.mkdir(dir_name)if not processable_files:root = ResourceNode(@@ -330,6 +340,7 @@ class DirectoryParser(BaseParser):warnings=warnings,)result.temp_dir_path = temp_uri+ result.artifact_ref = await writer.finalize(resource_rel=dir_name)result.meta["file_count"] = 0result.meta["dir_name"] = dir_nameresult.meta["total_processable"] = 0@@ -363,9 +374,10 @@ class DirectoryParser(BaseParser):split_content=split_content,max_concurrent=directory_config.max_concurrent,job_timeout=job_timeout,+ output_store=output_store,)- pending_temp_uris.update(- parsed["result"].temp_dir_path+ pending_results.extend(+ parsed["result"]for parsed in understanding_results.values()if parsed.get("result") and parsed["result"].temp_dir_path)@@ -391,12 +403,14 @@ class DirectoryParser(BaseParser):else:try:# The merge helper owns this artifact once merging starts.- pending_temp_uris.discard(sub_result.temp_dir_path)+ if sub_result in pending_results:+ pending_results.remove(sub_result)await self._merge_parser_result(cf,sub_result,target_uri,viking_fs,+ target_writer=writer,preserve_structure=preserve_structure,split_content=split_content,)@@ -425,6 +439,7 @@ class DirectoryParser(BaseParser):target_uri,viking_fs,warnings,+ target_writer=writer,preserve_structure=preserve_structure,)parser_name = "direct_upload"@@ -435,6 +450,8 @@ class DirectoryParser(BaseParser):target_uri,viking_fs,warnings,+ target_writer=writer,+ output_store=output_store,preserve_structure=preserve_structure,import_root=str(source_path),split_content=split_content,@@ -488,6 +505,7 @@ class DirectoryParser(BaseParser):warnings=warnings,)result.temp_dir_path = temp_uri+ result.artifact_ref = await writer.finalize(resource_rel=dir_name)result.meta["file_count"] = file_countresult.meta["dir_name"] = dir_nameresult.meta["total_processable"] = len(processable_files)@@ -515,15 +533,28 @@ class DirectoryParser(BaseParser):warnings=[f"Failed to parse directory: {exc}"],)finally:- if temp_uri and not keep_temp:- pending_temp_uris.add(temp_uri)- for pending_uri in pending_temp_uris:+ if writer is not None and not keep_temp:+ await writer.cleanup()+ for pending_result in pending_results:try:- await viking_fs.delete_temp(pending_uri)+ pending_ref = getattr(pending_result, "artifact_ref", None)+ if pending_ref is not None:+ pending_store = (+ output_store+ if output_store is not None+ and output_store.backend == pending_ref.backend+ else store_for_artifact_ref(+ pending_ref,+ viking_fs=self._get_viking_fs(),+ )+ )+ await pending_store.cleanup(pending_ref)+ elif pending_result.temp_dir_path:+ await self._get_viking_fs().delete_temp(pending_result.temp_dir_path)except Exception as exc:logger.warning("[DirectoryParser] Failed to clean temporary artifact %s: %s",- pending_uri,+ pending_result.temp_dir_path,exc,)@@ -745,6 +776,7 @@ class DirectoryParser(BaseParser):split_content: bool,max_concurrent: int,job_timeout: float,+ output_store: Any = None,) -> Dict[int, Dict[str, Any]]:"""Parse jobs with a fixed local pool and a shared service-loop limit."""limiter = _get_understanding_limiter(max_concurrent)@@ -770,6 +802,7 @@ class DirectoryParser(BaseParser):import_root=import_root,split_content=split_content,parse_options=job.get("parse_options"),+ output_store=output_store,)sub_result = await asyncio.wait_for(parse_coro, timeout=job_timeout)results[job["index"]] = {"result": sub_result, "error": None}@@ -806,9 +839,12 @@ class DirectoryParser(BaseParser):import_root: Optional[str],split_content: bool,parse_options: Optional[Dict[str, Any]] = None,+ output_store: Any = None,) -> ParseResult:"""Run one parser without mutating the directory destination tree."""options = dict(parse_options or {})+ if output_store is not None:+ options["parse_output_store"] = output_storesource = options.pop("_source", str(classified_file.path))options.update(enable_link_rewrite=preserve_structure,@@ -826,6 +862,7 @@ class DirectoryParser(BaseParser):target_uri: str,viking_fs: Any,*,+ target_writer: Optional[ParseArtifactWriter] = None,preserve_structure: bool,split_content: bool,) -> None:@@ -841,6 +878,30 @@ class DirectoryParser(BaseParser):dest = f"{target_uri}/{parent}" if parent != "." else target_urielse:dest = target_uri+ source_ref = getattr(sub_result, "artifact_ref", None)+ if target_writer is not None and source_ref is not None:+ source_store = (+ target_writer.store+ if source_ref.backend == target_writer.store.backend+ else store_for_artifact_ref(source_ref, viking_fs=viking_fs)+ )+ target_rel = target_writer.relative_path(dest)+ try:+ merged = await copy_artifact_tree(+ source_store=source_store,+ source_ref=source_ref,+ source_rel="",+ target=target_writer,+ target_rel=target_rel,+ allowed_hidden=_MERGE_SIDECAR_ALLOWLIST,+ flatten_single_output=bool(not split_content and preserve_structure),+ )+ finally:+ await source_store.cleanup(source_ref)+ if not merged:+ raise ValueError(no_content_error)+ return+try:merged = await DirectoryParser._merge_temp(viking_fs,@@ -868,6 +929,8 @@ class DirectoryParser(BaseParser):target_uri: str,viking_fs: Any,warnings: List[str],+ target_writer: Optional[ParseArtifactWriter] = None,+ output_store: Any = None,preserve_structure: bool = True,import_root: Optional[str] = None,split_content: bool = True,@@ -898,12 +961,14 @@ class DirectoryParser(BaseParser):preserve_structure=preserve_structure,import_root=import_root,split_content=split_content,+ output_store=output_store,)await DirectoryParser._merge_parser_result(classified_file,sub_result,target_uri,viking_fs,+ target_writer=target_writer,preserve_structure=preserve_structure,split_content=split_content,)@@ -922,12 +987,15 @@ class DirectoryParser(BaseParser):}else:try:- content = src_file.read_bytes()+ content = detect_and_convert_encoding(src_file.read_bytes(), src_file)if preserve_structure:dst_uri = f"{target_uri}/{rel_path}"else:dst_uri = f"{target_uri}/{PurePosixPath(rel_path).name}"- await viking_fs.write_file(dst_uri, content)+ if target_writer is not None:+ await target_writer.write_bytes(dst_uri, content)+ else:+ await viking_fs.write_file(dst_uri, content)return {"ok": True, "meta": {}, "error": None}except Exception as exc:warnings.append(f"Failed to upload {rel_path}: {exc}")@@ -939,6 +1007,7 @@ class DirectoryParser(BaseParser):target_uri: str,viking_fs: Any,warnings: List[str],+ target_writer: Optional[ParseArtifactWriter] = None,preserve_structure: bool = True,) -> Dict[str, Any]:"""Directly upload a file without using its parser.@@ -957,12 +1026,15 @@ class DirectoryParser(BaseParser):src_file = classified_file.pathtry:- content = src_file.read_bytes()+ content = detect_and_convert_encoding(src_file.read_bytes(), src_file)if preserve_structure:dst_uri = f"{target_uri}/{rel_path}"else:dst_uri = f"{target_uri}/{PurePosixPath(rel_path).name}"- await viking_fs.write_file(dst_uri, content)+ if target_writer is not None:+ await target_writer.write_bytes(dst_uri, content)+ else:+ await viking_fs.write_file(dst_uri, content)return {"ok": True, "meta": {}, "error": None}except Exception as exc:warnings.append(f"Failed to upload {rel_path}: {exc}")diff --git a/openviking/parse/parsers/markdown.py b/openviking/parse/parsers/markdown.pyindex 8f269e57b..e78b17e14 100644--- a/openviking/parse/parsers/markdown.py+++ b/openviking/parse/parsers/markdown.py@@ -29,6 +29,12 @@ from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Unionfrom openviking.parse.accessors.mime_types import IANA_MEDIA_TYPE_TO_EXTENSIONfrom openviking.parse.base import NodeType, ParseResult, ResourceNode, create_parse_resultfrom openviking.parse.image_validation import is_valid_image+from openviking.parse.output import (+ AgfsParseOutputStore,+ ParseArtifactRef,+ ParseArtifactWriter,+ create_parse_artifact_writer,+)from openviking.parse.parsers.base_parser import BaseParserfrom openviking.parse.parsers.code.ast.providers import supports_code_skeletonfrom openviking.parse.parsers.constants import (@@ -289,6 +295,10 @@ class MarkdownParser(BaseParser):ParseResult with temp_dir_path (Viking URI)"""start_time = time.time()+ output_store = kwargs.pop("parse_output_store", None)+ writer = await create_parse_artifact_writer(+ output_store, viking_fs=self._get_viking_fs() if output_store is None else None+ )try:split_content = bool(kwargs.get("split_content", True))@@ -305,7 +315,7 @@ class MarkdownParser(BaseParser):# Phase 1 — parse only: turn the markdown into an ordered VikingFS write# plan, touching nothing. The temp URI is allocated here (the one# FS-scoped step) and threaded in so layout planning stays side-effect free.- temp_uri = self._create_temp_uri()+ temp_uri = writer.ref.rootlayout_kwargs = dict(kwargs)layout_kwargs["flatten_single_output"] = flatten_single_outputlayout = await self._compute_layout(@@ -335,6 +345,7 @@ class MarkdownParser(BaseParser):# rewriting links and ingesting local images.await self._apply_layout(layout,+ writer=writer,rewrite_ctx=rewrite_ctx,base_dir=base_dir,allowed_media_dirs=allowed_media_dirs,@@ -360,12 +371,15 @@ class MarkdownParser(BaseParser):warnings=layout.warnings,)- result.temp_dir_path = layout.temp_uri+ root_rel = writer.relative_path(layout.root_dir)+ result.artifact_ref = await writer.finalize(resource_rel=root_rel)+ result.temp_dir_path = result.artifact_ref.rootreturn resultexcept Exception as e:logger.error(f"[MarkdownParser] Parse failed: {e}", exc_info=True)+ await writer.cleanup()raiseasync def _compute_layout(@@ -454,6 +468,7 @@ class MarkdownParser(BaseParser):self,layout: _Layout,*,+ writer: Optional[ParseArtifactWriter] = None,rewrite_ctx: Optional[_RewriteContext] = None,base_dir: Optional[Path] = None,allowed_media_dirs: Optional[List[Path]] = None,@@ -462,7 +477,11 @@ class MarkdownParser(BaseParser):create dirs, write each section (rewriting relative links when enabled), theningest the local images those sections reference."""started = time.perf_counter()- viking_fs = self._get_viking_fs()+ if writer is None:+ writer = ParseArtifactWriter(+ AgfsParseOutputStore(viking_fs=self._get_viking_fs()),+ ParseArtifactRef(backend="agfs", root=layout.temp_uri),+ )mkdir_ops = [op for op in layout.ops if op.kind == "mkdir"]write_ops = [op for op in layout.ops if op.kind == "write"]write_chars = sum(len(op.content or "") for op in write_ops)@@ -473,13 +492,17 @@ class MarkdownParser(BaseParser):mkdir_count = 0for op in layout.ops:if op.kind == "mkdir":- await viking_fs.mkdir(op.uri, exist_ok=op.exist_ok)+ await writer.mkdir(op.uri)mkdir_count += 1mkdir_s = time.perf_counter() - mkdir_startedwrite_started = time.perf_counter()+ written_sections: list[tuple[str, str]] = []for op in write_ops:- await self._write_section(op.uri, op.content, rewrite_ctx=rewrite_ctx)+ written = await self._write_section(+ op.uri, op.content, writer=writer, rewrite_ctx=rewrite_ctx+ )+ written_sections.append((op.uri, written))write_s = time.perf_counter() - write_started# Ingest local image files, placing each image next to the markdown file@@ -488,7 +511,13 @@ class MarkdownParser(BaseParser):images_started = time.perf_counter()has_image_refs = self._layout_has_local_image_refs(write_ops)if has_image_refs:- await self._ingest_local_images(layout.root_dir, base_dir, allowed_media_dirs)+ await self._ingest_local_images(+ layout.root_dir,+ base_dir,+ allowed_media_dirs,+ writer=writer,+ markdown_files=written_sections,+ )images_s = time.perf_counter() - images_startedtotal_s = time.perf_counter() - started@@ -804,6 +833,9 @@ class MarkdownParser(BaseParser):root_dir: str,base_dir: Optional[Path] = None,allowed_media_dirs: Optional[List[Path]] = None,+ *,+ writer: Optional[ParseArtifactWriter] = None,+ markdown_files: Optional[List[Tuple[str, str]]] = None,) -> None:"""Scan every processed markdown file under ``root_dir`` and copy the local@@ -821,26 +853,20 @@ class MarkdownParser(BaseParser):allowed_media_dirs: Additional directories from which derived mediamay be read (passed through to ``_resolve_image_path``)"""- viking_fs = self._get_viking_fs()-- # Find all processed markdown files under the root directory- glob_result = await viking_fs.glob("**/*.md", uri=root_dir)- md_uris = glob_result.get("matches", [])- if not md_uris:- return+ if writer is None:+ writer = ParseArtifactWriter(+ AgfsParseOutputStore(viking_fs=self._get_viking_fs()),+ ParseArtifactRef(backend="agfs", root=root_dir),+ )+ if markdown_files is None:+ raise ValueError("markdown_files are required for artifact image ingestion")root_prefix = root_dir.rstrip("/")# mapping: rel_md_path -> {original_path_str -> unique_filename}mappings: Dict[str, Dict[str, str]] = {}- for md_uri in md_uris:- try:- content = await viking_fs.read_file(md_uri)- except Exception:- logger.warning(f"[MarkdownParser] Failed to read markdown file: {md_uri}")- continue-+ for md_uri, content in markdown_files:# Collect all image references in this markdown file: markdown# embeds (![...]) and HTML <img src="..."> tags alike.from openviking.parse.image_rewrite import HTML_IMG_PATTERN@@ -903,7 +929,7 @@ class MarkdownParser(BaseParser):# Write next to the markdown fileviking_path = f"{md_dir}/{unique_filename}"- await viking_fs.write_file_bytes(viking_path, image_bytes)+ await writer.write_bytes(viking_path, image_bytes)# Record mapping for post-commit rewritefile_mappings[origin_link] = unique_filename@@ -923,7 +949,7 @@ class MarkdownParser(BaseParser):from openviking.parse.image_rewrite import IMAGE_MAPPINGS_FILENAME- await viking_fs.write_file(+ await writer.write_text(f"{root_prefix}/{IMAGE_MAPPINGS_FILENAME}",json.dumps(mappings, ensure_ascii=False),)@@ -1341,8 +1367,9 @@ class MarkdownParser(BaseParser):uri: str,content: str,*,+ writer: Optional[ParseArtifactWriter] = None,rewrite_ctx: Optional[_RewriteContext] = None,- ) -> None:+ ) -> str:"""Write a markdown section file, rewriting relative links when enabled."""if rewrite_ctx is not None:content = await self._rewrite_relative_links(@@ -1350,7 +1377,13 @@ class MarkdownParser(BaseParser):section_subpath=self._section_subpath(uri, rewrite_ctx.root_dir),rewrite_ctx=rewrite_ctx,)- await self._get_viking_fs().write_file(uri, content)+ if writer is None:+ writer = ParseArtifactWriter(+ AgfsParseOutputStore(viking_fs=self._get_viking_fs()),+ ParseArtifactRef(backend="agfs", root=uri.rsplit("/", 1)[0]),+ )+ await writer.write_text(uri, content)+ return content# ========== New Parsing Logic (v5.0) ==========diff --git a/openviking/parse/parsers/media/audio.py b/openviking/parse/parsers/media/audio.pyindex de0b75a82..3b20dd1fb 100644--- a/openviking/parse/parsers/media/audio.py+++ b/openviking/parse/parsers/media/audio.py@@ -31,6 +31,7 @@ from pathlib import Pathfrom typing import List, Optional, Unionfrom openviking.parse.base import NodeType, ParseResult, ResourceNode+from openviking.parse.output import create_parse_artifact_writerfrom openviking.parse.parsers.base_parser import BaseParserfrom openviking.parse.parsers.media.constants import AUDIO_EXTENSIONSfrom openviking.parse.parsers.media.naming import resolve_media_names@@ -75,16 +76,11 @@ class AudioParser(BaseParser):FileNotFoundError: If source file does not existIOError: If audio processing fails"""- from openviking.storage.viking_fs import get_viking_fs-# Convert to Path objectfile_path = Path(source) if isinstance(source, str) else sourceif not file_path.exists():raise FileNotFoundError(f"Audio file not found: {source}")- viking_fs = get_viking_fs()- temp_uri = viking_fs.create_temp_uri()-# Phase 1: Generate temporary filesaudio_bytes = file_path.read_bytes()ext = file_path.suffix@@ -98,12 +94,6 @@ class AudioParser(BaseParser):# Root directory name: filename stem + _ + extension (without dot)ext_no_dot = ext[1:] if ext else ""root_dir_name = VikingURI.sanitize_segment(f"{stem}_{ext_no_dot}")- root_dir_uri = f"{temp_uri}/{root_dir_name}"- await viking_fs.mkdir(root_dir_uri, exist_ok=True)-- # 1.1 Save original audio with original filename (sanitized)- await viking_fs.write_file_bytes(f"{root_dir_uri}/{original_filename}", audio_bytes)-# 1.2 Validate audio file using magic bytes# Define magic bytes for supported audio formatsaudio_magic_bytes = {@@ -131,6 +121,18 @@ class AudioParser(BaseParser):f"Invalid audio file: {file_path}. File signature does not match expected format {ext_lower}")+ writer = await create_parse_artifact_writer(+ kwargs.get("parse_output_store"),+ viking_fs=self._get_viking_fs() if kwargs.get("parse_output_store") is None else None,+ )+ try:+ await writer.mkdir(root_dir_name)+ await writer.write_bytes(f"{root_dir_name}/{original_filename}", audio_bytes)+ artifact_ref = await writer.finalize(resource_rel=root_dir_name)+ except BaseException:+ await writer.cleanup()+ raise+# Extract audio metadata (placeholder)duration = 0sample_rate = 0@@ -161,7 +163,8 @@ class AudioParser(BaseParser):return ParseResult(root=root_node,source_path=str(file_path),- temp_dir_path=temp_uri,+ temp_dir_path=artifact_ref.root,+ artifact_ref=artifact_ref,source_format="audio",parser_name="AudioParser",meta={"content_type": "audio", "format": format_str.lower()},diff --git a/openviking/parse/parsers/media/image.py b/openviking/parse/parsers/media/image.pyindex 32a267f01..f64c14cd1 100644--- a/openviking/parse/parsers/media/image.py+++ b/openviking/parse/parsers/media/image.py@@ -17,16 +17,15 @@ from typing import List, Optional, Unionfrom PIL import Imagefrom openviking.parse.base import NodeType, ParseResult, ResourceNode+from openviking.parse.output import create_parse_artifact_writerfrom openviking.parse.parsers.base_parser import BaseParserfrom openviking.parse.parsers.media.constants import IMAGE_EXTENSIONSfrom openviking.parse.parsers.media.large_image_processor import (- LargeImageResult,process_large_image,save_image_to_bytes,)from openviking.parse.parsers.media.naming import resolve_media_namesfrom openviking.parse.parsers.media.utils import _convert_svg_to_png-from openviking.prompts import render_promptfrom openviking.storage.viking_fs import get_viking_fsfrom openviking_cli.utils.config.parser_config import ImageConfigfrom openviking_cli.utils.logger import get_logger@@ -104,20 +103,19 @@ class ImageParser(BaseParser):if not file_path.exists():raise FileNotFoundError(f"Image file not found: {source}")- viking_fs = get_viking_fs()- temp_uri = viking_fs.create_temp_uri()-# Load image (SVG is converted to PNG first since PIL doesn't support it)# For other formats not natively supported by VLM/embedding, convert to PNG# to ensure end-to-end compatibility with image understanding and vectorizationtry:file_bytes = file_path.read_bytes()- if file_path.suffix.lower() == ".svg" or file_bytes[:4] == b"<svg" or (file_bytes[:5] == b"<?xml" and b"<svg" in file_bytes[:100]):+ if (+ file_path.suffix.lower() == ".svg"+ or file_bytes[:4] == b"<svg"+ or (file_bytes[:5] == b"<?xml" and b"<svg" in file_bytes[:100])+ ):png_bytes = _convert_svg_to_png(file_bytes)if png_bytes is None:- raise ValueError(- f"SVG files require cairosvg. Install it: pip install cairosvg"- )+ raise ValueError("SVG files require cairosvg. Install it: pip install cairosvg")img = Image.open(io.BytesIO(png_bytes))img.verify()img.close()@@ -128,7 +126,14 @@ class ImageParser(BaseParser):else:# Check if extension is in VLM-natively supported listsuffix_lower = file_path.suffix.lower()- supported_by_vlm = suffix_lower in {".png", ".jpg", ".jpeg", ".gif", ".bmp", ".webp"}+ supported_by_vlm = suffix_lower in {+ ".png",+ ".jpg",+ ".jpeg",+ ".gif",+ ".bmp",+ ".webp",+ }if supported_by_vlm:img = Image.open(file_path)img.verify() # Verify that it's a valid image@@ -148,7 +153,9 @@ class ImageParser(BaseParser):raise ValueError(f"Invalid image file: {file_path}. Error: {e}") from e# Resolve names- display_stem, stem, original_filename = resolve_media_names(file_path, file_path.suffix, **kwargs)+ display_stem, stem, original_filename = resolve_media_names(+ file_path, file_path.suffix, **kwargs+ )ext_no_dot = file_path.suffix[1:] if file_path.suffix else "jpg"if needs_png_conversion:# Convert to PNG and change extension for VLM/embedding compatibility@@ -157,98 +164,109 @@ class ImageParser(BaseParser):format_str = "png"root_dir_name = VikingURI.sanitize_segment(f"{stem}_{ext_no_dot}")- root_dir_uri = f"{temp_uri}/{root_dir_name}"- await viking_fs.mkdir(root_dir_uri, exist_ok=True)-- # Process the image (check if large)- large_image_result = process_large_image(file_path, img, filename_prefix=stem, config=self.config)- img.close()-- # Create root node with metadata- root_node = ResourceNode(- type=NodeType.ROOT,- title=display_stem,- level=0,- detail_file=None,- content_path=None,- children=[],- meta={- "width": width,- "height": height,- "format": format_str.lower(),- "content_type": "image",- "source_title": display_stem,- "semantic_name": display_stem,- "original_filename": original_filename,- "is_large_image": large_image_result.needs_processing,- },+ output_store = kwargs.get("parse_output_store")+ writer = await create_parse_artifact_writer(+ output_store, viking_fs=get_viking_fs() if output_store is None else None)+ try:+ await writer.mkdir(root_dir_name)- if large_image_result.needs_processing:- # Large image processing mode: save preview, grid, tiles- # NOTE: The original file is intentionally NOT saved. Large images can be- # hundreds of MB; storing the full-resolution original would exceed storage- # budgets. The preview + tiles are the canonical representation. The- # `original_filename` metadata field records the original name for reference- # but the corresponding file is not persisted.- logger.info(f"Processing large image {original_filename}: {width}x{height}")-- # Save low-res preview (original is too large to store)- await viking_fs.write_file_bytes(- f"{root_dir_uri}/{large_image_result.preview_filename}",- large_image_result.preview_bytes+ # Process the image (check if large)+ large_image_result = process_large_image(+ file_path, img, filename_prefix=stem, config=self.config+ )+ img.close()++ # Create root node with metadata+ root_node = ResourceNode(+ type=NodeType.ROOT,+ title=display_stem,+ level=0,+ detail_file=None,+ content_path=None,+ children=[],+ meta={+ "width": width,+ "height": height,+ "format": format_str.lower(),+ "content_type": "image",+ "source_title": display_stem,+ "semantic_name": display_stem,+ "original_filename": original_filename,+ "is_large_image": large_image_result.needs_processing,+ },)- root_node.meta["preview_filename"] = large_image_result.preview_filename-- # Save grid overlay- if large_image_result.grid_overlay_bytes:- grid_filename = large_image_result.grid_overlay_filename or f"{stem}_grid.jpg"- await viking_fs.write_file_bytes(- f"{root_dir_uri}/{grid_filename}",- large_image_result.grid_overlay_bytes++ if large_image_result.needs_processing:+ # Large image processing mode: save preview, grid, tiles+ # NOTE: The original file is intentionally NOT saved. Large images can be+ # hundreds of MB; storing the full-resolution original would exceed storage+ # budgets. The preview + tiles are the canonical representation. The+ # `original_filename` metadata field records the original name for reference+ # but the corresponding file is not persisted.+ logger.debug(f"Processing large image {original_filename}: {width}x{height}")++ # Save low-res preview (original is too large to store)+ await writer.write_bytes(+ f"{root_dir_name}/{large_image_result.preview_filename}",+ large_image_result.preview_bytes,)- root_node.meta["grid_overlay"] = grid_filename-- # Create tiles directory- if large_image_result.tiles:- tiles_dir_name = "tiles"- tiles_dir_uri = f"{root_dir_uri}/{tiles_dir_name}"- await viking_fs.mkdir(tiles_dir_uri, exist_ok=True)-- # Save all tiles- for tile in large_image_result.tiles:- if tile.bytes_data:- await viking_fs.write_file_bytes(- f"{tiles_dir_uri}/{tile.filename}",- tile.bytes_data- )-- # Update metadata- root_node.meta["tiles_dir"] = tiles_dir_name- root_node.meta["num_tiles"] = len(large_image_result.tiles)- root_node.meta["grid_rows"] = large_image_result.total_rows- root_node.meta["grid_cols"] = large_image_result.total_cols- root_node.meta["original_width"] = width- root_node.meta["original_height"] = height-- else:- # Small image: save as PNG if format is not VLM-supported, else as-is- if needs_png_conversion:- # SVG was already converted to PNG bytes during loading- if file_path.suffix.lower() == ".svg":- image_bytes = converted_png_bytes- else:- with Image.open(file_path) as converted_img:- image_bytes = save_image_to_bytes(converted_img, format="PNG")+ root_node.meta["preview_filename"] = large_image_result.preview_filename++ # Save grid overlay+ if large_image_result.grid_overlay_bytes:+ grid_filename = large_image_result.grid_overlay_filename or f"{stem}_grid.jpg"+ await writer.write_bytes(+ f"{root_dir_name}/{grid_filename}",+ large_image_result.grid_overlay_bytes,+ )+ root_node.meta["grid_overlay"] = grid_filename++ # Create tiles directory+ if large_image_result.tiles:+ tiles_dir_name = "tiles"+ tiles_dir_rel = f"{root_dir_name}/{tiles_dir_name}"+ await writer.mkdir(tiles_dir_rel)++ # Save all tiles+ for tile in large_image_result.tiles:+ if tile.bytes_data:+ await writer.write_bytes(+ f"{tiles_dir_rel}/{tile.filename}", tile.bytes_data+ )++ # Update metadata+ root_node.meta["tiles_dir"] = tiles_dir_name+ root_node.meta["num_tiles"] = len(large_image_result.tiles)+ root_node.meta["grid_rows"] = large_image_result.total_rows+ root_node.meta["grid_cols"] = large_image_result.total_cols+ root_node.meta["original_width"] = width+ root_node.meta["original_height"] = height+else:- image_bytes = file_path.read_bytes()- await viking_fs.write_file_bytes(f"{root_dir_uri}/{original_filename}", image_bytes)+ # Small image: save as PNG if format is not VLM-supported, else as-is+ if needs_png_conversion:+ # SVG was already converted to PNG bytes during loading+ if file_path.suffix.lower() == ".svg":+ image_bytes = converted_png_bytes+ else:+ with Image.open(file_path) as converted_img:+ image_bytes = save_image_to_bytes(converted_img, format="PNG")+ else:+ image_bytes = file_path.read_bytes()+ await writer.write_bytes(f"{root_dir_name}/{original_filename}", image_bytes)++ artifact_ref = await writer.finalize(resource_rel=root_dir_name)+ except BaseException:+ await writer.cleanup()+ raise# Phase 3: Build directory structure (handled by TreeBuilder)return ParseResult(root=root_node,source_path=str(file_path),- temp_dir_path=temp_uri,+ temp_dir_path=artifact_ref.root,+ artifact_ref=artifact_ref,source_format="image",parser_name="ImageParser",meta={diff --git a/openviking/parse/parsers/media/video.py b/openviking/parse/parsers/media/video.pyindex 26105155e..e6caf128f 100644--- a/openviking/parse/parsers/media/video.py+++ b/openviking/parse/parsers/media/video.py@@ -28,6 +28,7 @@ from pathlib import Pathfrom typing import List, Optional, Unionfrom openviking.parse.base import NodeType, ParseResult, ResourceNode+from openviking.parse.output import create_parse_artifact_writerfrom openviking.parse.parsers.base_parser import BaseParserfrom openviking.parse.parsers.media.constants import VIDEO_EXTENSIONSfrom openviking.parse.parsers.media.naming import resolve_media_names@@ -69,16 +70,11 @@ class VideoParser(BaseParser):FileNotFoundError: If source file does not existIOError: If video processing fails"""- from openviking.storage.viking_fs import get_viking_fs-# Convert to Path objectfile_path = Path(source) if isinstance(source, str) else sourceif not file_path.exists():raise FileNotFoundError(f"Video file not found: {source}")- viking_fs = get_viking_fs()- temp_uri = viking_fs.create_temp_uri()-# Phase 1: Generate temporary filesvideo_bytes = file_path.read_bytes()ext = file_path.suffix@@ -92,12 +88,6 @@ class VideoParser(BaseParser):# Root directory name: filename stem + _ + extension (without dot)ext_no_dot = ext[1:] if ext else ""root_dir_name = VikingURI.sanitize_segment(f"{stem}_{ext_no_dot}")- root_dir_uri = f"{temp_uri}/{root_dir_name}"- await viking_fs.mkdir(root_dir_uri, exist_ok=True)-- # 1.1 Save original video with original filename (sanitized)- await viking_fs.write_file_bytes(f"{root_dir_uri}/{original_filename}", video_bytes)-# 1.2 Validate video file using magic bytes# Define magic bytes for supported video formatsvideo_magic_bytes = {@@ -124,6 +114,18 @@ class VideoParser(BaseParser):f"Invalid video file: {file_path}. File signature does not match expected format {ext_lower}")+ writer = await create_parse_artifact_writer(+ kwargs.get("parse_output_store"),+ viking_fs=self._get_viking_fs() if kwargs.get("parse_output_store") is None else None,+ )+ try:+ await writer.mkdir(root_dir_name)+ await writer.write_bytes(f"{root_dir_name}/{original_filename}", video_bytes)+ artifact_ref = await writer.finalize(resource_rel=root_dir_name)+ except BaseException:+ await writer.cleanup()+ raise+# Extract video metadata (placeholder)duration = 0width = 0@@ -156,7 +158,8 @@ class VideoParser(BaseParser):return ParseResult(root=root_node,source_path=str(file_path),- temp_dir_path=temp_uri,+ temp_dir_path=artifact_ref.root,+ artifact_ref=artifact_ref,source_format="video",parser_name="VideoParser",meta={"content_type": "video", "format": format_str.lower()},diff --git a/openviking/parse/parsers/pdf.py b/openviking/parse/parsers/pdf.pyindex f299a0680..f46a5cec8 100644--- a/openviking/parse/parsers/pdf.py+++ b/openviking/parse/parsers/pdf.py@@ -142,6 +142,7 @@ class PDFParser(BaseParser):base_dir=pdf_path.parent,allowed_media_dirs=[storage.media_dir],split_content=kwargs.get("split_content", True),+ parse_output_store=kwargs.get("parse_output_store"),)# Step 3: Update metadata for PDF origindiff --git a/openviking/parse/parsers/upload_utils.py b/openviking/parse/parsers/upload_utils.pyindex fc9071664..c2ff438b4 100644--- a/openviking/parse/parsers/upload_utils.py+++ b/openviking/parse/parsers/upload_utils.py@@ -8,6 +8,12 @@ from pathlib import Pathfrom typing import Any, List, Optional, Set, Tuple, Unionfrom openviking.parse.gitignore import GitignoreMatcher+from openviking.parse.output import (+ ARTIFACT_MANIFEST_NAME as ARTIFACT_MANIFEST_NAME,+)+from openviking.parse.output import (+ write_artifact_manifest,+)from openviking.parse.parsers.constants import (ADDITIONAL_TEXT_EXTENSIONS,CODE_EXTENSIONS,@@ -15,7 +21,7 @@ from openviking.parse.parsers.constants import (IGNORE_EXTENSIONS,)from openviking.parse.parsers.text_encoding import normalize_text_bytes-from openviking.utils.path_safety import safe_join_viking_uri+from openviking.utils.content_hash import content_md5from openviking_cli.utils.logger import get_loggerlogger = get_logger(__name__)@@ -116,18 +122,24 @@ _UPLOAD_CONCURRENCY = 8async def upload_directory(local_dir: Path,- viking_uri_base: str,- viking_fs: Any,+ base_rel: str,+ *,+ store: Any,+ artifact_ref: Any,ignore_dirs: Optional[Union[Set[str], List[str], str]] = None,ignore_extensions: Optional[Set[str]] = None,max_file_size: int = 10 * 1024 * 1024,include: Optional[str] = None,exclude: Optional[str] = None,) -> Tuple[int, List[str]]:- """Upload an entire directory recursively and return uploaded count with warnings.-- Optimized: collects all files in one pass, pre-creates directories upfront,- then uploads all files concurrently (up to _UPLOAD_CONCURRENCY at a time).+ """Upload a directory into a parse output store, returning (count, warnings).++ Artifacts are written through the backend-agnostic :class:`ParseOutputStore`,+ so AGFS and local backends share one path with no branching. Files land under+ ``base_rel`` (e.g. ``repository``) as artifact-relative paths the store+ resolves to a local dir or an AGFS temp URI. An md5 manifest of the final+ (encoding-normalized) bytes is written at the artifact root so the incremental+ diff can compare fingerprints without re-reading file contents."""effective_ignore_extensions = (ignore_extensions if ignore_extensions is not None else IGNORE_EXTENSIONS@@ -153,9 +165,10 @@ async def upload_directory(warnings: List[str] = []- # --- Phase 1: Collect files and unique parent directory URIs in one pass ---- files_to_upload: List[Tuple[Path, str]] = [] # (local_path, target_uri)- parent_uris: Set[str] = {viking_uri_base}+ # --- Phase 1: Collect files and their artifact-relative parent dirs ---+ base = base_rel.strip("/")+ files_to_upload: List[Tuple[Path, str]] = []+ parent_dirs: Set[str] = set()for root, dirs, files in os.walk(local_dir):dir_path = Path(root)@@ -202,39 +215,29 @@ async def upload_directory(exclude_patterns,):continue- try:- target_uri = safe_join_viking_uri(viking_uri_base, rel_path_str)- except ValueError as exc:- warning = f"Skipping {file_path}: {exc}"- warnings.append(warning)- logger.warning(warning)- continue- files_to_upload.append((file_path, target_uri))- parent_uris.add(target_uri.rsplit("/", 1)[0])-- # --- Phase 2: Pre-create all directories ---- # Memoized mkdir: each unique VikingFS path is created at most once.- # This is equivalent to _ensure_parent_dirs but avoids redundant HTTP calls- # by tracking already-processed paths across all directories.- _created: Set[str] = set()-- for dir_uri in sorted(parent_uris):- if dir_uri in _created:- continue+ # Artifact-relative path under the resource root; the store resolves it+ # to a local/agfs location.+ target = f"{base}/{rel_path_str}" if base else rel_path_str+ files_to_upload.append((file_path, target))+ if "/" in target:+ parent_dirs.add(target.rsplit("/", 1)[0])++ # --- Phase 2: Pre-create unique parent dirs (shallowest first) ---+ # store.write_bytes also creates parents implicitly; this batches the unique+ # directories once to avoid redundant per-file creation on remote backends.+ for rel_dir in sorted(parent_dirs, key=lambda value: (value.count("/"), value)):try:- await viking_fs.mkdir(dir_uri, exist_ok=True)- _created.add(dir_uri)+ await store.mkdir(artifact_ref, rel_dir)except Exception as e:- if "already" in str(e).lower():- _created.add(dir_uri)- else:- logger.warning(f"Failed to create directory {dir_uri}: {e}")+ logger.warning(f"Failed to create artifact directory {rel_dir}: {e}")# --- Phase 3: Upload files concurrently ---sem = asyncio.Semaphore(_UPLOAD_CONCURRENCY)errors: List[Optional[str]] = [None] * len(files_to_upload)+ # rel_path -> md5 of final bytes, collected for the manifest.+ md5_by_target: dict[str, str] = {}- async def _upload_one(idx: int, file_path: Path, target_uri: str) -> None:+ async def _upload_one(idx: int, file_path: Path, target: str) -> None:async with sem:def _read_and_encode() -> bytes:@@ -243,16 +246,25 @@ async def upload_directory(try:encoded = await asyncio.to_thread(_read_and_encode)- await viking_fs.write_file_bytes(target_uri, encoded)+ await store.write_bytes(artifact_ref, target, encoded)+ # Fingerprint the final (normalized) bytes at the write point; this+ # is a local read, no extra remote IO.+ md5_by_target[target] = content_md5(encoded)except Exception as exc:errors[idx] = f"Failed to upload {file_path}: {exc}"- await asyncio.gather(*[_upload_one(i, fp, uri) for i, (fp, uri) in enumerate(files_to_upload)])+ await asyncio.gather(*[_upload_one(i, fp, tgt) for i, (fp, tgt) in enumerate(files_to_upload)])for err in errors:if err:warnings.append(err)logger.warning(err)+ # Persist the md5 manifest at the artifact root so the incremental diff can+ # compare fingerprints without re-reading files. Only write it when every file+ # uploaded cleanly, so a partial manifest never masquerades as complete.+ if not any(errors):+ await write_artifact_manifest(store, artifact_ref, md5_by_target)+uploaded_count = sum(1 for e in errors if e is None)return uploaded_count, warningsdiff --git a/openviking/parse/tree_builder.py b/openviking/parse/tree_builder.pyindex deae20b76..765ce3300 100644--- a/openviking/parse/tree_builder.py+++ b/openviking/parse/tree_builder.py@@ -21,7 +21,7 @@ IMPORTANT (v5.0 Architecture):- Content splitting is handled by Parser, not TreeBuilder"""-from typing import Optional+from typing import Any, Optionalfrom openviking.core.building_tree import BuildingTreefrom openviking.core.context import Context@@ -145,6 +145,8 @@ class TreeBuilder:source_format: Optional[str] = None,create_parent: bool = False,flatten_single_file: bool = False,+ artifact_ref: Optional[Any] = None,+ output_store: Optional[Any] = None,) -> "BuildingTree":"""Finalize URI metadata for a temp parse result.@@ -153,38 +155,57 @@ class TreeBuilder:to_uri: Exact target URI, or resources root to import underparent_uri: Target parent URI (must exist unless create_parent is True)create_parent: Whether to automatically create parent directory if it doesn't exist+ artifact_ref / output_store: When provided, the artifact structure is+ read through the backend-agnostic store (supporting local+ artifacts); otherwise the legacy AGFS ``ls(temp_uri)`` path is+ used. Either way the resulting ``temp_doc_uri`` keeps the AGFS URI+ shape and target-URI resolution stays on VikingFS."""viking_fs = get_viking_fs()temp_uri = temp_dir_path- # 1. Find document root directory- entries = await viking_fs.ls(temp_uri, ctx=ctx)- doc_dirs = [e for e in entries if e.get("isDir") and e["name"] not in [".", ".."]]+ # 1. Find the single document root. The structural walk is identical for+ # AGFS and local artifacts, so it is delegated to resolve_artifact_doc_root+ # when a store/ref is available. ``temp_doc_uri`` remains an AGFS-shaped+ # URI relative to temp_uri for downstream persistence.+ if output_store is not None and artifact_ref is not None:+ from openviking.parse.output import resolve_artifact_doc_root- if len(doc_dirs) != 1:- logger.error(- f"[TreeBuilder] Expected 1 document directory in {temp_uri}, found {len(doc_dirs)}"+ resolved = await resolve_artifact_doc_root(+ output_store, artifact_ref, flatten_single_file=flatten_single_file)- raise ValueError(- f"[TreeBuilder] Expected 1 document directory in {temp_uri}, found {len(doc_dirs)}"- )-- original_name = doc_dirs[0]["name"]- doc_name = VikingURI.sanitize_segment(original_name)- temp_doc_uri = f"{temp_uri}/{original_name}" # use original name to find temp dir- root_is_file = False- if flatten_single_file:- doc_entries = [- entry- for entry in await viking_fs.ls(temp_doc_uri, ctx=ctx)- if entry.get("name") not in [".", ".."]- ]- if len(doc_entries) == 1 and not doc_entries[0].get("isDir"):- original_name = doc_entries[0]["name"]- doc_name = VikingURI.sanitize_segment(original_name)- temp_doc_uri = f"{temp_doc_uri}/{original_name}"- root_is_file = True+ original_name = resolved.doc_name+ doc_name = VikingURI.sanitize_segment(original_name)+ temp_doc_uri = f"{temp_uri.rstrip('/')}/{resolved.doc_rel}"+ root_is_file = resolved.root_is_file+ else:+ entries = await viking_fs.ls(temp_uri, ctx=ctx)+ doc_dirs = [e for e in entries if e.get("isDir") and e["name"] not in [".", ".."]]++ if len(doc_dirs) != 1:+ logger.error(+ f"[TreeBuilder] Expected 1 document directory in {temp_uri}, found {len(doc_dirs)}"+ )+ raise ValueError(+ f"[TreeBuilder] Expected 1 document directory in {temp_uri}, found {len(doc_dirs)}"+ )++ original_name = doc_dirs[0]["name"]+ doc_name = VikingURI.sanitize_segment(original_name)+ temp_doc_uri = f"{temp_uri}/{original_name}" # use original name to find temp dir+ root_is_file = False+ if flatten_single_file:+ doc_entries = [+ entry+ for entry in await viking_fs.ls(temp_doc_uri, ctx=ctx)+ if entry.get("name") not in [".", ".."]+ ]+ if len(doc_entries) == 1 and not doc_entries[0].get("isDir"):+ original_name = doc_entries[0]["name"]+ doc_name = VikingURI.sanitize_segment(original_name)+ temp_doc_uri = f"{temp_doc_uri}/{original_name}"+ root_is_file = Trueif original_name != doc_name:logger.debug(f"[TreeBuilder] Sanitized doc name: {original_name!r} -> {doc_name!r}")diff --git a/openviking/parse/understanding_api.py b/openviking/parse/understanding_api.pyindex fdf57616d..65f482930 100644--- a/openviking/parse/understanding_api.py+++ b/openviking/parse/understanding_api.py@@ -8,7 +8,7 @@ Workflow:2. Submit a parse request to Responses API (response_id)3. Poll Responses API until completed/failed4. Download result zip (zip_url)-5. Materialize the result into VikingFS temp directory+5. Materialize the result through the configured ParseOutputStore6. Return ParseResult for downstream TreeBuilder/SemanticQueue processing"""@@ -28,6 +28,10 @@ from openviking.parse.image_rewrite import (IMAGE_MAPPINGS_FILENAME,build_artifact_image_mappings,)+from openviking.parse.output import (+ ParseArtifactRef,+ create_parse_artifact_writer,+)from openviking.parse.parsers.base_parser import BaseParserfrom openviking.parse.parsers.constants import MPEG_TS_EXTENSION_ALIAS, TYPESCRIPT_MPEG_TS_EXTENSIONfrom openviking.parse.parsers.media.constants import (@@ -229,10 +233,15 @@ class UnderstandingAPI(BaseParser):if archive_root:doc_name = archive_roottask_meta["doc_name"] = doc_name- temp_dir_path = await self._unpack_zip_to_temp_dir(- zip_path=zip_path,- resource_name=doc_name,- )+ unpack_kwargs = {+ "zip_path": zip_path,+ "resource_name": doc_name,+ }+ if kwargs.get("parse_output_store") is not None:+ unpack_kwargs["parse_output_store"] = kwargs["parse_output_store"]+ unpacked = await self._unpack_zip_to_temp_dir(**unpack_kwargs)+ artifact_ref = unpacked if isinstance(unpacked, ParseArtifactRef) else None+ temp_dir_path = artifact_ref.root if artifact_ref is not None else unpackedfinally:try:zip_path.unlink()@@ -276,6 +285,7 @@ class UnderstandingAPI(BaseParser):content_type if content_type in {"image", "audio", "video"} else doc_type),temp_dir_path=temp_dir_path,+ artifact_ref=artifact_ref,parser_name="UnderstandingAPI",meta={key: task_meta[key] for key in ("file_id", "response_id") if task_meta.get(key)},)@@ -762,14 +772,20 @@ class UnderstandingAPI(BaseParser):f.write(rsp.content)return Path(f.name)- async def _unpack_zip_to_temp_dir(self, zip_path: Path, resource_name: str) -> str:- viking_fs = get_viking_fs()- temp_uri = viking_fs.create_temp_uri()+ async def _unpack_zip_to_temp_dir(+ self,+ zip_path: Path,+ resource_name: str,+ *,+ parse_output_store: Any = None,+ ) -> ParseArtifactRef:+ writer = await create_parse_artifact_writer(+ parse_output_store,+ viking_fs=get_viking_fs() if parse_output_store is None else None,+ )+ resource_rel = str(resource_name).strip("/")try:- await viking_fs.mkdir(temp_uri)-- temp_doc_uri = f"{temp_uri}/{resource_name}"- await viking_fs.mkdir(temp_doc_uri)+ await writer.mkdir(resource_rel)with tempfile.TemporaryDirectory() as extract_dir:with zipfile.ZipFile(zip_path, "r") as zf:@@ -783,51 +799,32 @@ class UnderstandingAPI(BaseParser):image_mappings = await asyncio.to_thread(build_artifact_image_mappings, root_dir)- for child in root_dir.iterdir():+ for child in root_dir.rglob("*"):+ rel = child.relative_to(root_dir).as_posix()if child.name in {".", "..", IMAGE_MAPPINGS_FILENAME}:continueif child.is_dir():- sub_uri = f"{temp_doc_uri}/{child.name}"- await viking_fs.mkdir(sub_uri)- await self._copy_dir_to_fs(child, sub_uri)+ await writer.mkdir(f"{resource_rel}/{rel}")else:- await viking_fs.write_file_bytes(- f"{temp_doc_uri}/{child.name}", child.read_bytes()+ await writer.write_bytes(+ f"{resource_rel}/{rel}",+ await asyncio.to_thread(child.read_bytes),)if image_mappings:- await viking_fs.write_file(- f"{temp_doc_uri}/{IMAGE_MAPPINGS_FILENAME}",+ await writer.write_text(+ f"{resource_rel}/{IMAGE_MAPPINGS_FILENAME}",json.dumps(image_mappings, ensure_ascii=False),)except BaseException:try:- await viking_fs.delete_temp(temp_uri)+ await writer.cleanup()except Exception as cleanup_exc:logger.warning("[UnderstandingAPI] Failed to clean temporary artifact %s: %s",- temp_uri,+ writer.ref.root,cleanup_exc,)raise- return temp_uri-- async def _copy_dir_to_fs(self, local_dir: Path, fs_uri: str):- """- Recursively copy a local directory to VikingFS.- """- viking_fs = get_viking_fs()-- for item in local_dir.iterdir():- if item.name in [".", "..", IMAGE_MAPPINGS_FILENAME]:- continue-- if item.is_dir():- sub_uri = f"{fs_uri}/{item.name}"- await viking_fs.mkdir(sub_uri)- await self._copy_dir_to_fs(item, sub_uri)- else:- file_content = item.read_bytes()- file_uri = f"{fs_uri}/{item.name}"- await viking_fs.write_file_bytes(file_uri, file_content)+ return await writer.finalize(resource_rel=resource_rel)diff --git a/openviking/resource/shared_source.py b/openviking/resource/shared_source.pynew file mode 100644index 000000000..01a5731d6--- /dev/null+++ b/openviking/resource/shared_source.py@@ -0,0 +1,112 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Shared-upload SOURCE references for durable resource ingestion.++Unlike :mod:`openviking.resource.staged_source`, a shared upload already lives in+durable VikingFS storage (``viking://upload/...``) with its own TTL cleanup, so+the SOURCE phase does not copy it into a task-owned temp bundle. The API only+validates and records a reference; the worker downloads the content once into a+worker-local file for parsing. Ownership of the shared object stays with the+upload store's TTL policy — the task never deletes it as its own temp.+"""++import asyncio+import shutil+import tempfile+from dataclasses import asdict, dataclass+from pathlib import Path+from typing import Any, Dict++from openviking.parse.accessors.base import LocalResource, SourceType+from openviking.server.error_mapping import is_not_found_error+from openviking.server.identity import RequestContext, Role++_SHARED_UPLOAD_ROOT = "viking://upload"+++@dataclass(frozen=True)+class SharedSource:+ """A validated reference to a shared upload consumed as a SOURCE input."""++ temp_file_id: str+ content_uri: str+ account_id: str+ original_filename: str+ file_ext: str+ meta: Dict[str, Any]++ def to_dict(self) -> Dict[str, Any]:+ return asdict(self)++ @classmethod+ def from_dict(cls, data: Dict[str, Any]) -> "SharedSource":+ if not isinstance(data, dict):+ raise ValueError("shared_source must be an object")+ temp_file_id = data.get("temp_file_id")+ content_uri = data.get("content_uri")+ account_id = data.get("account_id")+ original_filename = data.get("original_filename") or ""+ file_ext = data.get("file_ext") or ""+ meta = data.get("meta")+ if not isinstance(temp_file_id, str) or not temp_file_id.startswith("shared_"):+ raise ValueError("shared_source.temp_file_id must be a shared upload id")+ if not isinstance(content_uri, str) or not content_uri.startswith(+ f"{_SHARED_UPLOAD_ROOT}/"+ ):+ raise ValueError("shared_source.content_uri must live under the upload root")+ if not isinstance(account_id, str) or not account_id:+ raise ValueError("shared_source.account_id must be a non-empty string")+ if not isinstance(original_filename, str):+ raise ValueError("shared_source.original_filename must be a string")+ if not isinstance(file_ext, str):+ raise ValueError("shared_source.file_ext must be a string")+ if meta is not None and not isinstance(meta, dict):+ raise ValueError("shared_source.meta must be an object")+ return cls(+ temp_file_id=temp_file_id,+ content_uri=content_uri.rstrip("/"),+ account_id=account_id,+ original_filename=original_filename,+ file_ext=file_ext,+ meta=dict(meta or {}),+ )+++async def materialize_shared_source(+ shared: "SharedSource",+ *,+ viking_fs: Any,+ ctx: RequestContext,+) -> LocalResource:+ """Download a shared upload into a worker-local file for parsing.++ The shared object is owned by the upload store's TTL policy, so the returned+ resource only cleans up the worker-local copy. A missing or expired object+ fails loudly instead of yielding an empty input.+ """+ suffix = shared.file_ext or Path(shared.original_filename).suffix or ".tmp"+ local_root = Path(tempfile.mkdtemp(prefix="ov_shared_source_"))+ filename = shared.original_filename or f"resource{suffix}"+ local_path = local_root / Path(filename).name+ try:+ internal_ctx = RequestContext(user=ctx.user, role=Role.ROOT)+ content = await viking_fs.read_file_bytes(shared.content_uri, ctx=internal_ctx)+ await asyncio.to_thread(local_path.write_bytes, content)+ except asyncio.CancelledError:+ shutil.rmtree(local_root, ignore_errors=True)+ raise+ except Exception as exc:+ shutil.rmtree(local_root, ignore_errors=True)+ if is_not_found_error(exc):+ raise ValueError("Shared upload content is missing or expired") from exc+ raise++ meta = dict(shared.meta)+ meta["_cleanup_path"] = str(local_root)+ return LocalResource(+ path=local_path,+ source_type=SourceType.LOCAL,+ original_source=shared.original_filename or shared.temp_file_id,+ meta=meta,+ is_temporary=True,+ )diff --git a/openviking/server/routers/resources.py b/openviking/server/routers/resources.pyindex 23aa74188..a300b87f0 100644--- a/openviking/server/routers/resources.py+++ b/openviking/server/routers/resources.py@@ -256,6 +256,7 @@ async def add_resource(allow_local_path_resolution = Falseoriginal_filename = Noneresolved = None+ shared_source_ref = Noneif request.temp_file_id:if request.watch_interval > 0:raise InvalidArgumentError(@@ -265,12 +266,20 @@ async def add_resource("sitemap / RSS source instead, or re-add the resource when the ""source changes.")- resolved = await TempUploadStore.build(http_request.app.state.config).resolve_for_consume(- request.temp_file_id, _ctx- )- path = resolved.local_path- original_filename = resolved.original_filename- allow_local_path_resolution = True+ store = TempUploadStore.build(http_request.app.state.config)+ # A shared upload already lives in durable storage: the API only validates+ # a reference and the SOURCE worker downloads it once, avoiding a second+ # API-side download + task re-stage. Local uploads keep the copy path.+ shared_source_ref = await store.resolve_shared_reference(request.temp_file_id, _ctx)+ if shared_source_ref is not None:+ path = shared_source_ref.original_filename or request.temp_file_id+ original_filename = shared_source_ref.original_filename or None+ allow_local_path_resolution = True+ else:+ resolved = await store.resolve_for_consume(request.temp_file_id, _ctx)+ path = resolved.local_path+ original_filename = resolved.original_filename+ allow_local_path_resolution = Trueelif path is not None:path = require_remote_resource_source(path, declared_connector_add_type=request.add_type)if path is None:@@ -320,6 +329,7 @@ async def add_resource(internal_task=request.internal_task,is_active=request.is_active,args=request.args,+ shared_source=shared_source_ref,**kwargs,)except Exception:diff --git a/openviking/server/temp_upload_store.py b/openviking/server/temp_upload_store.pyindex 00241a6cd..7244a9843 100644--- a/openviking/server/temp_upload_store.py+++ b/openviking/server/temp_upload_store.py@@ -17,7 +17,7 @@ from dataclasses import dataclassfrom pathlib import Pathfrom queue import Full, Queuefrom threading import Lock, Thread-from typing import Any, Optional+from typing import TYPE_CHECKING, Any, Optionalfrom openviking.server.config import ServerConfig, TempUploadConfigfrom openviking.server.identity import RequestContext, Role@@ -26,6 +26,9 @@ from openviking.storage.viking_fs import LS_ALL_NODES, get_viking_fsfrom openviking_cli.exceptions import InvalidArgumentError, PermissionDeniedErrorfrom openviking_cli.utils.config.open_viking_config import get_openviking_config+if TYPE_CHECKING:+ from openviking.resource.shared_source import SharedSource+_CHUNK_SIZE = 1024 * 1024_SHARED_UPLOAD_ROOT = "viking://upload"_SHARED_CLEANUP_QUEUE_MAX_SIZE = 100@@ -186,9 +189,7 @@ def _shared_upload_created_at(upload_id: str) -> Optional[float]:async def _stream_upload_to_local_temp(upload_file: Any, max_size_bytes: int) -> tuple[str, int]:suffix = Path(upload_file.filename or "upload.tmp").suffix or ".tmp"- temp_path = await asyncio.to_thread(- _create_temp_file, prefix="ov_http_upload_", suffix=suffix- )+ temp_path = await asyncio.to_thread(_create_temp_file, prefix="ov_http_upload_", suffix=suffix)total = 0f = Nonetry:@@ -199,9 +200,7 @@ async def _stream_upload_to_local_temp(upload_file: Any, max_size_bytes: int) ->breaktotal += len(chunk)if total > max_size_bytes:- raise InvalidArgumentError(- f"Upload exceeds size limit ({max_size_bytes} bytes)."- )+ raise InvalidArgumentError(f"Upload exceeds size limit ({max_size_bytes} bytes).")# UploadFile reads already yield to the event loop. Local disk writes do# not, so run each bounded write in the default executor rather than# stalling the Core worker event loop on slow local storage.@@ -254,6 +253,40 @@ class TempUploadStore:return await asyncio.to_thread(self._resolve_local, temp_file_id)return await self._resolve_shared(temp_file_id, shared_id, ctx)+ async def resolve_shared_reference(+ self,+ temp_file_id: str,+ ctx: RequestContext,+ ) -> Optional["SharedSource"]:+ """Validate a shared upload and return a download-free SOURCE reference.++ The API only checks ownership and existence; the worker downloads the+ content once (see :func:`materialize_shared_source`). Returns ``None`` for+ non-shared ids so callers can fall back to the local-copy path.+ """+ from openviking.resource.shared_source import SharedSource++ shared_id = _parse_shared_temp_file_id(temp_file_id)+ if shared_id is None:+ return None+ meta = await self._read_shared_meta(shared_id, ctx)+ self._validate_shared_meta(meta, temp_file_id, ctx)+ content_uri = meta["storage_uri"]+ vfs = get_viking_fs()+ if not await vfs.exists(content_uri, ctx=self._internal_ctx(ctx)):+ raise PermissionDeniedError("Temporary upload is invalid: content missing.")+ original_filename = meta.get("original_filename") or ""+ return SharedSource.from_dict(+ {+ "temp_file_id": temp_file_id,+ "content_uri": content_uri,+ "account_id": ctx.account_id,+ "original_filename": original_filename,+ "file_ext": meta.get("file_ext") or Path(original_filename).suffix,+ "meta": {},+ }+ )+async def _save_local(self, upload_file: Any) -> str:config = get_openviking_config()temp_dir = config.storage.get_upload_temp_dir()@@ -316,9 +349,7 @@ class TempUploadStore:try:content = await asyncio.to_thread(Path(temp_path).read_bytes)- await vfs.write_file_bytes(- content_uri, content, ctx=internal_ctx, auto_pathlock=False- )+ await vfs.write_file_bytes(content_uri, content, ctx=internal_ctx, auto_pathlock=False)await vfs.write_file(meta_uri,json.dumps(meta, ensure_ascii=False),@@ -357,9 +388,7 @@ class TempUploadStore:now = time.time()with _SHARED_CLEANUP_STATE_LOCK:if account_id in _SHARED_CLEANUP_PENDING:- logger.debug(- "[TempUpload] Shared cleanup already pending account=%s", account_id- )+ logger.debug("[TempUpload] Shared cleanup already pending account=%s", account_id)returndue_at = _SHARED_CLEANUP_DUE_AT.get(account_id)if due_at is not None and now < due_at:@@ -528,9 +557,7 @@ class TempUploadStore:try:data = json.loads(await vfs.read_file(meta_uri, ctx=internal_ctx))except Exception as exc:- raise PermissionDeniedError(- "Temporary upload metadata is invalid or missing."- ) from exc+ raise PermissionDeniedError("Temporary upload metadata is invalid or missing.") from excif not isinstance(data, dict):raise PermissionDeniedError("Temporary upload metadata is invalid or missing.")return data@@ -615,9 +642,9 @@ class TempUploadStore:# Classify the single listing into independent groups (order preserved,# so each group stays name-ascending / oldest-first).- buckets: list[tuple[str, float]] = [] # (uri, bucket_expiry)- legacy: list[tuple[str, float]] = [] # (uri, upload_expiry)- invalid: list[str] = [] # uri+ buckets: list[tuple[str, float]] = [] # (uri, bucket_expiry)+ legacy: list[tuple[str, float]] = [] # (uri, upload_expiry)+ invalid: list[str] = [] # urifor entry in entries:if not entry.get("isDir"):continue@@ -636,12 +663,18 @@ class TempUploadStore:invalid.append(uri)# Process the bucket and legacy groups independently, oldest-first.- bucket_scanned, bucket_removed, bucket_due, bucket_failed = (- await self._cleanup_expiry_group(vfs, internal_ctx, buckets, now, kind="bucket")- )- legacy_scanned, legacy_removed, legacy_due, legacy_failed = (- await self._cleanup_expiry_group(vfs, internal_ctx, legacy, now, kind="flat")- )+ (+ bucket_scanned,+ bucket_removed,+ bucket_due,+ bucket_failed,+ ) = await self._cleanup_expiry_group(vfs, internal_ctx, buckets, now, kind="bucket")+ (+ legacy_scanned,+ legacy_removed,+ legacy_due,+ legacy_failed,+ ) = await self._cleanup_expiry_group(vfs, internal_ctx, legacy, now, kind="flat")# Invalid/foreign dirs: only when explicitly enabled; never blocks the# expiry groups and failures are logged inside the helper.@@ -729,9 +762,7 @@ class TempUploadStore:"""started_at = time.monotonic()try:- await vfs.remove_files(- uri, recursive=True, ctx=internal_ctx, auto_pathlock=False- )+ await vfs.remove_files(uri, recursive=True, ctx=internal_ctx, auto_pathlock=False)except Exception:logger.warning("[TempUpload] cleanup remove failed kind=%s uri=%s elapsed_ms=%.1f",diff --git a/openviking/service/reindex_executor.py b/openviking/service/reindex_executor.pyindex b7717d801..7b6c0334e 100644--- a/openviking/service/reindex_executor.py+++ b/openviking/service/reindex_executor.py@@ -34,15 +34,17 @@ from openviking.storage.abstract_overview import body_for_preview, embedding_texfrom openviking.storage.errors import ResourceBusyErrorfrom openviking.storage.expr import And, Eq, Or, PathScopefrom openviking.storage.queuefs.embedding_msg_converter import EmbeddingMsgConverter-from openviking.storage.queuefs.semantic_dag import SemanticDagExecutor+from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutorfrom openviking.storage.queuefs.semantic_msg import SemanticMsgfrom openviking.storage.queuefs.semantic_processor import SemanticProcessorfrom openviking.storage.viking_fs import get_viking_fsfrom openviking.telemetry import get_current_telemetryfrom openviking.telemetry.request_wait_tracker import get_request_wait_tracker+from openviking.utils.content_hash import content_md5from openviking.utils.embedding_input import truncate_embedding_inputfrom openviking.utils.embedding_utils import (_apply_ingest_options,+ _decode_text_bytes,_truncate_abstract_bytes,get_resource_content_type,vectorize_directory_meta,@@ -999,7 +1001,7 @@ class ReindexExecutor:ingest_options=run.ingest_options,)return- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="skill",max_concurrent_llm=processor.max_concurrent_llm,@@ -1230,8 +1232,16 @@ class ReindexExecutor:async def process_file(file_uri: str) -> _ReindexCounters:file_counters = _ReindexCounters(scanned_records=1)parent_uri = VikingURI(file_uri).parent.uri+ try:+ file_bytes = await get_viking_fs().read_file_bytes(file_uri, ctx=ctx)+ except Exception as exc:+ file_counters.failed_records += 1+ file_counters.warnings.append(f"Failed to read {file_uri} for reindex: {exc}")+ return file_counterssummary = await self._best_file_summary(file_uri, ctx=ctx)- vector_text = await self._best_resource_file_vector_text(file_uri, summary, ctx=ctx)+ vector_text = await self._best_resource_file_vector_text(+ file_uri, summary, ctx=ctx, file_content=file_bytes+ )if not vector_text:file_counters.unsupported_records += 1file_counters.warnings.append(f"No vector source found for {file_uri}")@@ -1248,6 +1258,7 @@ class ReindexExecutor:level=ContextLevel.DETAIL,ctx=ctx,ingest_options=ingest_options,+ md5=content_md5(file_bytes),)file_counters.rebuilt_records += 1except Exception as exc:@@ -1570,7 +1581,7 @@ class ReindexExecutor:if not recursive:continue- from openviking.storage.queuefs.semantic_dag import _SKIP_FILENAMES+ from openviking.storage.queuefs.semantic_executor import _SKIP_FILENAMESfrom openviking.storage.viking_fs import LS_ALL_NODESentries = await viking_fs.ls(directory_uri, node_limit=LS_ALL_NODES, ctx=ctx)@@ -1824,6 +1835,7 @@ class ReindexExecutor:uri: str,summary: str,ctx: RequestContext,+ file_content: bytes | None = None,) -> str:existing = await self._fetch_existing_record(uri=uri,@@ -1838,7 +1850,11 @@ class ReindexExecutor:text_source = embedding_config.text_sourceif text_source in SUMMARY_TEXT_SOURCES and summary:return summary- content = await self._safe_read_text(uri, ctx=ctx)+ content = (+ _decode_text_bytes(file_content)+ if file_content is not None+ else await self._safe_read_text(uri, ctx=ctx)+ )if content:return truncate_embedding_input(content, embedding_config.max_input_tokens)return summary or fallback@@ -1860,6 +1876,7 @@ class ReindexExecutor:ctx: RequestContext,meta: Optional[dict[str, Any]] = None,ingest_options: IngestOptions | None = None,+ md5: str | None = None,) -> None:service = get_service()assert service.vikingdb_manager is not None@@ -1877,6 +1894,7 @@ class ReindexExecutor:account_id=owner_ctx.account_id,owner_space=owner_space_for_uri(uri),meta=merged_meta,+ md5=md5,)context.set_vectorize(Vectorize(text=vector_text))msg = EmbeddingMsgConverter.from_context(context)diff --git a/openviking/service/resource_service.py b/openviking/service/resource_service.pyindex 886bd0f9b..d2c20d64f 100644--- a/openviking/service/resource_service.py+++ b/openviking/service/resource_service.py@@ -93,6 +93,7 @@ from openviking_cli.utils import get_loggerif TYPE_CHECKING:from openviking.connector.delegate import ConnectorDelegatefrom openviking.parse.accessors.base import LocalResource+ from openviking.resource.shared_source import SharedSourcefrom openviking.resource.staged_source import StagedSourcefrom openviking.resource.watch_manager import WatchManager, WatchTaskfrom openviking.resource.watch_scheduler import WatchScheduler@@ -187,6 +188,7 @@ class _SourcePlan:processor_args: Dict[str, Any]task_auth: Dict[str, Any] = field(repr=False)staged_source: Optional["StagedSource"] = None+ shared_source: Optional["SharedSource"] = Noneunderstanding_response_id: Optional[str] = Noneunderstanding_file_id: Optional[str] = Nonedefer_unnamed_target: bool = False@@ -768,11 +770,24 @@ class ResourceService:internal_kwargs["_feishu_checkpoint"] = (saved, save_response)prepared_resource = Noneif msg.staged_source is not None:- prepared_resource = await materialize_source(- StagedSource.from_dict(msg.staged_source),- viking_fs=self._viking_fs,- ctx=ctx,+ with get_current_telemetry().measure("resource.source_prepare"):+ prepared_resource = await materialize_source(+ StagedSource.from_dict(msg.staged_source),+ viking_fs=self._viking_fs,+ ctx=ctx,+ )+ elif msg.shared_source is not None:+ from openviking.resource.shared_source import (+ SharedSource,+ materialize_shared_source,)++ with get_current_telemetry().measure("resource.source_prepare"):+ prepared_resource = await materialize_shared_source(+ SharedSource.from_dict(msg.shared_source),+ viking_fs=self._viking_fs,+ ctx=ctx,+ )if msg.defer_target_resolution:from openviking_cli.utils.uri import VikingURI@@ -834,7 +849,7 @@ class ResourceService:ctx=ctx,resource_lock=resource_lock,)- if msg.staged_source is not None:+ if msg.staged_source is not None or msg.shared_source is not None:result["source_path"] = msg.source_pathstage_result = stage_callback("processing_queue")if inspect.isawaitable(stage_result):@@ -914,12 +929,39 @@ class ResourceService:allow_local_path_resolution: bool,processor_kwargs: Dict[str, Any],watch_auth_state: Optional[Dict[str, Any]],+ shared_source: Optional["SharedSource"] = None,) -> Optional[_SourcePlan]:"""Freeze one durable standard-pipeline source before it crosses QueueFS."""from openviking.parse.accessors.feishu_accessor import FeishuAccessorfrom openviking.resource.staged_source import stage_sourcesource_name = processor_kwargs.get("source_name")+ # A shared upload is already durable; reference it directly instead of+ # downloading + re-staging a second copy. Its identity comes from the+ # validated upload meta, so no accessor preflight is needed.+ if shared_source is not None:+ queued_args = {+ key: value+ for key, value in processor_kwargs.items()+ if key not in _ADD_RESOURCE_ARGS_RESERVED_FIELDS | _ADD_RESOURCE_TRANSIENT_ARGS+ }+ queued_args = self._sanitize_watch_processor_kwargs(queued_args)+ resolved_name = source_name or shared_source.original_filename or None+ source_format = (+ Path(shared_source.original_filename).suffix.lower().lstrip(".") or "file"+ )+ return _SourcePlan(+ path=path,+ source_identity=_ResourceSourceInfo(+ source_name=resolved_name,+ source_path=shared_source.original_filename or path,+ source_format=source_format,+ ),+ processor_args=queued_args,+ task_auth={},+ shared_source=shared_source,+ )+git_source = is_git_repo_url(path)feishu_source = FeishuAccessor._is_feishu_url(path)remote_source = is_remote_resource_source(path)@@ -953,11 +995,7 @@ class ResourceService:if request_git_auth is not None:task_auth = create_git_http_auth_state(request_git_auth, path)preflight_git_auth = request_git_auth- if (- preflight_git_auth is None- and is_git_https_url(path)- and is_github_url(path)- ):+ if preflight_git_auth is None and is_git_https_url(path) and is_github_url(path):github_token = await self._resource_processor.github_token_for(path, ctx)if github_token:preflight_git_auth = GitHttpAuthConfig(@@ -1196,6 +1234,9 @@ class ResourceService:staged_source=(plan.staged_source.to_dict() if plan.staged_source is not None else None),+ shared_source=(+ plan.shared_source.to_dict() if plan.shared_source is not None else None+ ),telemetry_id=get_current_telemetry().telemetry_id or None,account_id=ctx.account_id,user_id=ctx.user.user_id,@@ -1217,10 +1258,14 @@ class ResourceService:tags=tags,tag_mode=tag_mode,allow_local_path_resolution=(- True if plan.staged_source is not None else allow_local_path_resolution+ True+ if (plan.staged_source is not None or plan.shared_source is not None)+ else allow_local_path_resolution),enforce_public_remote_targets=(- enforce_public_remote_targets and plan.staged_source is None+ enforce_public_remote_targets+ and plan.staged_source is None+ and plan.shared_source is None),strict=bool(processor_kwargs.get("strict", False)),ignore_dirs=processor_kwargs.get("ignore_dirs"),@@ -1433,6 +1478,7 @@ class ResourceService:add_type: Optional[str] = None,internal_task: bool = False,args: Optional[Dict[str, Any]] = None,+ shared_source: Optional["SharedSource"] = None,**kwargs,) -> Dict[str, Any]:"""Accept and route a new resource-add request."""@@ -1466,6 +1512,7 @@ class ResourceService:enforce_public_remote_targets=enforce_public_remote_targets,internal_task=internal_task,args=args,+ shared_source=shared_source,**kwargs,)@@ -1540,6 +1587,7 @@ class ResourceService:internal_task: bool = False,args: Optional[Dict[str, Any]] = None,connector_states: Optional[Dict[str, Any]] = None,+ shared_source: Optional["SharedSource"] = None,**kwargs,) -> Dict[str, Any]:"""Validate and route one resource ingestion request.@@ -1836,6 +1884,7 @@ class ResourceService:allow_local_path_resolution=allow_local_path_resolution,processor_kwargs=kwargs,watch_auth_state=normalized_args.watch_auth_state,+ shared_source=shared_source,)if source_plan is not None:result = await self._enqueue_source_plan(@@ -1959,7 +2008,6 @@ class ResourceService:mode = normalize_parse_mode(parse_mode)if mode is ParseMode.NO_SPLIT:kwargs["parse_mode"] = mode.value- request_start = time.perf_counter()telemetry = get_current_telemetry()telemetry_id = telemetry.telemetry_idregister_telemetry(telemetry)@@ -2132,10 +2180,6 @@ class ResourceService:finally:if prepared_resource is not None:prepared_resource.cleanup()- telemetry.set(- "resource.request.duration_ms",- round((time.perf_counter() - request_start) * 1000, 3),- )if not telemetry_id or (defer_post_processing and not job_enqueued):unregister_telemetry(telemetry_id)if deferred_lock is not None:diff --git a/openviking/storage/collection_schemas.py b/openviking/storage/collection_schemas.pyindex d67bc562e..62a1fefe9 100644--- a/openviking/storage/collection_schemas.py+++ b/openviking/storage/collection_schemas.py@@ -10,6 +10,7 @@ similar to how init_viking_fs encapsulates VikingFS initialization.import asyncioimport hashlibimport json+import loggingimport threadingimport timefrom contextlib import nullcontext@@ -26,7 +27,12 @@ from openviking.storage.errors import (EmbeddingConfigurationError,EmbeddingRebuildRequiredError,)-from openviking.storage.queuefs.embedding_msg import EmbeddingMsg+from openviking.storage.index_action import FieldPatch, IndexAction+from openviking.storage.queuefs.embedding_msg import (+ EmbeddingMsg,+ IncompleteInitialRecordError,+ missing_initial_record_fields,+)from openviking.storage.queuefs.named_queue import DequeueHandlerBasefrom openviking.storage.queuefs.process_result import ProcessResultfrom openviking.storage.vector_ids import vector_record_id@@ -42,11 +48,13 @@ from openviking.utils.circuit_breaker import (CircuitBreakerOpen,classify_api_error,)+from openviking.utils.log_correlation import log_correlationfrom openviking.utils.model_retry import (ERROR_CLASS_AUTH,ERROR_CLASS_INPUT_TOO_LARGE,ERROR_CLASS_PERMANENT,)+from openviking.utils.time_utils import get_current_timestampfrom openviking_cli.session.user_id import UserIdentifierfrom openviking_cli.utils import get_loggerfrom openviking_cli.utils.config.open_viking_config import OpenVikingConfig@@ -120,6 +128,11 @@ class CollectionSchemas:{"FieldName": "search_tags", "FieldType": "list<string>"},{"FieldName": "abstract", "FieldType": "string"},{"FieldName": "content", "FieldType": "text"},+ # md5 of the final stored file bytes for this record's URI. Used by+ # incremental diff to skip re-processing unchanged files. Older+ # records may lack it; callers must treat missing/empty as "unknown"+ # and fall back to reading file bytes.+ {"FieldName": "md5", "FieldType": "string", "DefaultValue": ""},{"FieldName": "account_id", "FieldType": "string"},{"FieldName": "owner_user_id", "FieldType": "string"},{@@ -483,7 +496,6 @@ class TextEmbeddingHandler(DequeueHandlerBase):config = get_openviking_config()self._collection_name = config.storage.vectordb.nameself._vector_dim = config.embedding.dimension- self._initialize_embedder(config)breaker_cfg = config.embedding.circuit_breakerself._circuit_breaker = CircuitBreaker(failure_threshold=breaker_cfg.failure_threshold,@@ -546,13 +558,23 @@ class TextEmbeddingHandler(DequeueHandlerBase):@staticmethoddef _embedding_msg_log_context(embedding_msg: Optional[EmbeddingMsg]) -> str:- """Return the URI allowed in embedding logs."""+ """Return the URI-safe context retained in public error messages."""if embedding_msg is None:return "uri=<unknown>"context_data = embedding_msg.context_data or {}return f"uri={context_data.get('uri') or '<unknown>'}"+ @classmethod+ def _embedding_delivery_log_context(cls, embedding_msg: Optional[EmbeddingMsg]) -> str:+ """Return request/message IDs plus the safe embedding URI for logs."""+ if embedding_msg is None:+ return f"{log_correlation()} {cls._embedding_msg_log_context(None)}"+ return (+ f"{log_correlation(telemetry_id=embedding_msg.telemetry_id, message_id=embedding_msg.id)} "+ f"{cls._embedding_msg_log_context(embedding_msg)}"+ )+@classmethoddef _embedding_error_msg(cls,@@ -561,6 +583,13 @@ class TextEmbeddingHandler(DequeueHandlerBase):) -> str:return f"{message} ({cls._embedding_msg_log_context(embedding_msg)})"+ @classmethod+ def _log_embedding_error(+ cls, level: int, message: str, embedding_msg: Optional[EmbeddingMsg]+ ) -> None:+ """Log an internal correlation suffix without changing public errors."""+ logger.log(level, "%s [%s]", message, cls._embedding_delivery_log_context(embedding_msg))+@staticmethodasync def _materialize_content(embedding_msg: EmbeddingMsg,@@ -601,6 +630,53 @@ class TextEmbeddingHandler(DequeueHandlerBase):else inserted_data["abstract"][:VIKINGDB_CONTENT_MAX_SIZE])+ async def _apply_non_embedding_operation(+ self,+ embedding_msg: EmbeddingMsg,+ ctx: RequestContext,+ ) -> Dict[str, Any]:+ if embedding_msg.action is IndexAction.DELETE:+ deleted_count = await self._vikingdb.strict_delete(+ embedding_msg.record_ids,+ ctx=ctx,+ )+ return {"deleted_count": deleted_count}++ record_id = embedding_msg.record_ids[0]+ field_patch = embedding_msg.field_patch+ assert field_patch is not None+ # UPDATE_FIELDS is a read-modify-write patch. The exact read happens at+ # execution time so append semantics use the latest stored tags rather+ # than the inventory snapshot that produced the durable plan.+ existing_records = await self._vikingdb.get_strict([record_id], ctx=ctx)+ if existing_records:+ existing = dict(existing_records[0])+ resolved = field_patch.resolve(existing)+ changed_fields = {+ field: resolved.get(field)+ for field in field_patch.values+ if resolved.get(field) != existing.get(field)+ }+ if not changed_fields:+ return {"id": record_id, "status": "skipped"}+ changed_fields["updated_at"] = get_current_timestamp()+ updated_record = {"id": record_id, **changed_fields}+ result = await self._vikingdb.update(updated_record, ctx=ctx)+ if not result.ok:+ raise RuntimeError(+ result.error_message or f"failed to update vector record: {record_id}"+ )+ return updated_record++ initial_record = field_patch.apply({**field_patch.seed_fields, "id": record_id})+ missing_fields = missing_initial_record_fields(initial_record)+ if missing_fields:+ raise IncompleteInitialRecordError(record_id, missing_fields)+ created_id = await self._vikingdb.upsert(initial_record, ctx=ctx)+ if not created_id:+ raise RuntimeError(f"failed to create missing vector record: {record_id}")+ return {"id": record_id, "status": "created"}+async def on_dequeue(self, data: Optional[Dict[str, Any]]) -> ProcessResult:"""Process dequeued message and add embedding vector(s)."""if not data:@@ -608,16 +684,23 @@ class TextEmbeddingHandler(DequeueHandlerBase):embedding_msg: Optional[EmbeddingMsg] = Nonerequest_failed_message: Optional[str] = None+ execute_started_at: float | None = None+ queue_wait_ms = 0.0+ execute_status = "ok"try:embedding_msg = EmbeddingMsg.from_json(data["data"])+ execute_started_at = time.perf_counter()+ if embedding_msg.queue_enqueued_at > 0:+ queue_wait_ms = max((time.time() - embedding_msg.queue_enqueued_at) * 1000.0, 0.0)inserted_data = embedding_msg.context_data+ logger.debug(+ "Processing embedding message: %s action=%s",+ self._embedding_delivery_log_context(embedding_msg),+ embedding_msg.action.value,+ )account_id = inserted_data.get("account_id", "default")context_user = inserted_data.get("user") or {}- user_id = (- context_user.get("user_id")- or inserted_data.get("owner_user_id")- or "default"- )+ user_id = context_user.get("user_id") or inserted_data.get("owner_user_id") or "default"user = UserIdentifier(account_id=account_id, user_id=user_id)ctx = RequestContext(user=user, role=Role.USER, bypass_acl=True)collector = resolve_telemetry(embedding_msg.telemetry_id)@@ -630,6 +713,18 @@ class TextEmbeddingHandler(DequeueHandlerBase):self._record_request_success(embedding_msg)return ProcessResult.success()+ if embedding_msg.action is IndexAction.NONE:+ self._merge_request_stats(embedding_msg.telemetry_id, processed=1)+ self._record_request_success(embedding_msg)+ logger.debug("Skipping no-op embedding message: %s", embedding_msg.id)+ return ProcessResult.success()++ if embedding_msg.action not in {IndexAction.UPSERT, IndexAction.MERGE}:+ result = await self._apply_non_embedding_operation(embedding_msg, ctx)+ self._merge_request_stats(embedding_msg.telemetry_id, processed=1)+ self._record_request_success(embedding_msg)+ return ProcessResult.success(result)+if not isinstance(embedding_msg.message, (str, list)):logger.debug(f"Skipping unsupported message type: {type(embedding_msg.message)}"@@ -646,6 +741,7 @@ class TextEmbeddingHandler(DequeueHandlerBase):except CircuitBreakerOpen:self._log_breaker_open_reenqueue_summary()if self._vikingdb.has_queue_manager:+ execute_status = "requeued"wait = self._circuit_breaker.retry_afterif wait > 0:await asyncio.sleep(wait)@@ -659,6 +755,7 @@ class TextEmbeddingHandler(DequeueHandlerBase):)return ProcessResult.requeued()# No queue manager — cannot re-enqueue, drop with error+ execute_status = "error"error_msg = self._embedding_error_msg(embedding_msg,"Circuit breaker open and no queue manager",@@ -711,32 +808,36 @@ class TextEmbeddingHandler(DequeueHandlerBase):passif error_class == ERROR_CLASS_INPUT_TOO_LARGE:- logger.error(error_msg)+ execute_status = "error"+ self._log_embedding_error(logging.ERROR, error_msg, embedding_msg)self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)request_failed_message = error_msgreturn ProcessResult.failed(error_msg)if error_class == ERROR_CLASS_PERMANENT:- logger.critical(error_msg)+ execute_status = "error"+ self._log_embedding_error(logging.CRITICAL, error_msg, embedding_msg)self._circuit_breaker.record_failure(embed_err)self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)request_failed_message = error_msgreturn ProcessResult.failed(error_msg)if error_class == ERROR_CLASS_AUTH:+ execute_status = "error"# Bad/expired credential: retrying cannot succeed. Fail# terminally instead of re-enqueueing, which would cycle# forever and hold this resource's tree lock and its# add-resource --wait open. Don't trip the breaker: an open# breaker re-enqueues later messages and reintroduces the# same leak. See #2916.- logger.error(error_msg)+ self._log_embedding_error(logging.ERROR, error_msg, embedding_msg)self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)request_failed_message = error_msgreturn ProcessResult.failed(error_msg)# Transient or unknown — re-enqueue for retry- logger.warning(error_msg)+ self._log_embedding_error(logging.WARNING, error_msg, embedding_msg)+ execute_status = "requeued"self._circuit_breaker.record_failure(embed_err)if self._vikingdb.has_queue_manager:try:@@ -749,8 +850,8 @@ class TextEmbeddingHandler(DequeueHandlerBase):embedding_msg.telemetry_id)logger.info(- "Re-enqueued embedding message after transient error "- f"({self._embedding_msg_log_context(embedding_msg)})"+ "Re-enqueued embedding message after transient error: %s",+ self._embedding_delivery_log_context(embedding_msg),)return ProcessResult.requeued()except Exception as requeue_err:@@ -762,6 +863,7 @@ class TextEmbeddingHandler(DequeueHandlerBase):)self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)+ execute_status = "error"request_failed_message = error_msgreturn ProcessResult.failed(error_msg)@@ -770,12 +872,13 @@ class TextEmbeddingHandler(DequeueHandlerBase):inserted_data["vector"] = result.dense_vector# Validate vector dimensionif len(result.dense_vector) != self._vector_dim:+ execute_status = "error"error_msg = self._embedding_error_msg(embedding_msg,"Dense vector dimension mismatch: "f"expected {self._vector_dim}, got {len(result.dense_vector)}",)- logger.error(error_msg)+ self._log_embedding_error(logging.ERROR, error_msg, embedding_msg)self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)request_failed_message = error_msgreturn ProcessResult.failed(error_msg)@@ -791,7 +894,7 @@ class TextEmbeddingHandler(DequeueHandlerBase):embedding_msg,"Embedder not initialized, skipping vector generation",)- logger.warning(error_msg)+ self._log_embedding_error(logging.WARNING, error_msg, embedding_msg)try:from openviking.metrics.datasources import EmbeddingEventDataSource@@ -799,18 +902,21 @@ class TextEmbeddingHandler(DequeueHandlerBase):except Exception:passself._merge_request_stats(embedding_msg.telemetry_id, error_count=1)+ execute_status = "error"request_failed_message = error_msgreturn ProcessResult.failed(error_msg)# Write to vector databasetry:raw_upsert_options = inserted_data.pop("_upsert_options", {})- upsert_options = normalize_upsert_options(- {**raw_upsert_options, "partial_update": True}- )- # Ensure vector DB has deterministic IDs per semantic layer.+ # Reuse the actual vector-store ID when a semantic plan+ # rebuilds an existing same-level record. Only genuinely new+ # records derive an ID locally from (account, uri, level).+ existing_record_id = inserted_data.pop("_upsert_record_id", None)uri = inserted_data.get("uri")- if uri:+ if existing_record_id:+ inserted_data["id"] = str(existing_record_id)+ elif uri:inserted_data["id"] = vector_record_id(account_id, uri, inserted_data.get("level", 2))@@ -820,6 +926,45 @@ class TextEmbeddingHandler(DequeueHandlerBase):embedding_msg,ctx,)+ if embedding_msg.action is IndexAction.MERGE:+ field_patch = embedding_msg.field_patch+ merge_fields = dict(field_patch.values) if field_patch is not None else {}+ merge_modes = dict(field_patch.modes) if field_patch is not None else {}+ # Legacy MERGE producers encoded tag intent in context_data+ # plus _upsert_options. Normalize them to the explicit patch+ # protocol before the exact read.+ if "search_tags" in inserted_data and "search_tags" not in merge_fields:+ merge_fields["search_tags"] = inserted_data.pop("search_tags")+ merge_modes["search_tags"] = str(+ raw_upsert_options.get("search_tag_mode", "replace")+ )+ existing_records = await self._vikingdb.get_strict(+ [inserted_data["id"]], ctx=ctx+ )+ base = (+ dict(existing_records[0])+ if existing_records+ else dict(field_patch.seed_fields)+ if field_patch is not None+ else {}+ )+ # Fresh content/model outputs win over the stored record;+ # scalar patches are then interpreted against that latest+ # exact-get result.+ inserted_data = FieldPatch(merge_fields, merge_modes).apply(+ {**base, **inserted_data}+ )+ if not existing_records:+ missing_fields = missing_initial_record_fields(inserted_data)+ if missing_fields:+ raise RuntimeError(+ "merge could not create missing vector record: "+ f"record_id={inserted_data.get('id')} "+ f"missing_fields={missing_fields}"+ )+ upsert_options = normalize_upsert_options(+ {**raw_upsert_options, "partial_update": False}+ )if inserted_data.get("context_type") == ContextType.SKILL.value:# Cancelling the waiter cannot stop a threaded DB write.# Keep this task active until that write has settled.@@ -836,7 +981,10 @@ class TextEmbeddingHandler(DequeueHandlerBase):)record_id = resultif record_id:- logger.debug("Successfully wrote embedding: uri=%s", uri)+ logger.debug(+ "Successfully wrote embedding: %s",+ self._embedding_delivery_log_context(embedding_msg),+ )except CollectionNotFoundError as db_err:# During shutdown, queue workers may finish one dequeued item.if self._vikingdb.is_closing:@@ -848,11 +996,12 @@ class TextEmbeddingHandler(DequeueHandlerBase):embedding_msg,f"Failed to write to vector database: {db_err}",)- logger.error(error_msg)+ self._log_embedding_error(logging.ERROR, error_msg, embedding_msg)import tracebacktraceback.print_exc()self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)+ execute_status = "error"request_failed_message = error_msgreturn ProcessResult.failed(error_msg)except Exception as db_err:@@ -865,8 +1014,9 @@ class TextEmbeddingHandler(DequeueHandlerBase):embedding_msg,f"Failed to write to vector database: {db_err}",)- logger.error(error_msg)+ self._log_embedding_error(logging.ERROR, error_msg, embedding_msg)self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)+ execute_status = "error"request_failed_message = error_msgreturn ProcessResult.failed(error_msg)@@ -898,9 +1048,33 @@ class TextEmbeddingHandler(DequeueHandlerBase):traceback.print_exc()if embedding_msg is not None:self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)+ execute_status = "error"request_failed_message = error_msgreturn ProcessResult.failed(error_msg)finally:+ if embedding_msg is not None and execute_started_at is not None:+ tracker = get_request_wait_tracker()+ record_timing = getattr(tracker, "record_embedding_timing", None)+ if callable(record_timing):+ record_timing(+ embedding_msg.telemetry_id,+ queue_wait_ms=queue_wait_ms,+ execute_ms=(time.perf_counter() - execute_started_at) * 1000.0,+ )+ from openviking.metrics.datasources.resource import ResourceIngestionEventDataSource++ ResourceIngestionEventDataSource.record_stage(+ stage="embedding_queue_wait",+ status=execute_status,+ duration_seconds=queue_wait_ms / 1000.0,+ account_id=embedding_msg.context_data.get("account_id"),+ )+ ResourceIngestionEventDataSource.record_stage(+ stage="embedding_execute",+ status=execute_status,+ duration_seconds=(time.perf_counter() - execute_started_at),+ account_id=embedding_msg.context_data.get("account_id"),+ )if embedding_msg is not None and request_failed_message is not None:self._record_request_failure(embedding_msg, request_failed_message)diff --git a/openviking/storage/content_write.py b/openviking/storage/content_write.pyindex a236d20ff..56ae89ce6 100644--- a/openviking/storage/content_write.py+++ b/openviking/storage/content_write.py@@ -46,6 +46,7 @@ from openviking.storage.viking_fs import VikingFSfrom openviking.telemetry import get_current_telemetryfrom openviking.telemetry.request_wait_tracker import get_request_wait_trackerfrom openviking.telemetry.resource_summary import build_queue_status_payload+from openviking.utils.content_hash import content_md5from openviking.utils.embedding_utils import vectorize_directory_meta, vectorize_filefrom openviking.utils.ingest_options import IngestOptionsfrom openviking.utils.path_safety import validate_safe_viking_uri_path@@ -242,6 +243,8 @@ class ContentWriteCoordinator:updated: list[str] = []unchanged: list[str] = []refresh_kinds: dict[str, str] = {}+ file_md5s: dict[str, str] = {}+ file_abstracts: dict[str, str] = {}sidecar_directories: set[str] = set()pending: list[tuple[dict[str, Any], bool, str]] = []write_error: Exception | None = None@@ -270,16 +273,32 @@ class ContentWriteCoordinator:raise NotFoundError(uri, "file")pending.append((operation, exists, write_mode))+ file_abstracts = await self._load_file_abstracts(+ [+ operation["uri"]+ for operation, existed, _ in pending+ if existed+ and context_type_for_uri(operation["uri"]) in {"resource", "skill"}+ and not is_abstract_overview_uri(operation["uri"])+ ],+ ctx=ctx,+ )+for operation, existed, write_mode in pending:uri = operation["uri"]try:- await self._write_in_place(+ final_content = await self._write_in_place(uri,operation["content"],mode=write_mode,ctx=ctx,lease_ref=lease,)+ if context_type_for_uri(uri) in {+ "resource",+ "skill",+ } and not is_abstract_overview_uri(uri):+ file_md5s[uri] = content_md5(final_content)except Exception as exc:write_error = excbreak@@ -319,6 +338,8 @@ class ContentWriteCoordinator:wait=wait,timeout=timeout,telemetry_id=telemetry_id,+ file_md5s=file_md5s,+ file_abstracts=file_abstracts,)if refresh_kindselse (@@ -506,6 +527,8 @@ class ContentWriteCoordinator:wait: bool,timeout: Optional[float],telemetry_id: str,+ file_md5s: dict[str, str] | None = None,+ file_abstracts: dict[str, str] | None = None,) -> _BatchRefreshOutcome:resource_groups: dict[tuple[str, str], dict[str, list[str]]] = defaultdict(lambda: {"added": [], "modified": []}@@ -528,6 +551,18 @@ class ContentWriteCoordinator:changes=changes,ctx=ctx,force_refresh=wait,+ file_md5s={+ uri: file_md5s[uri]+ for values in changes.values()+ for uri in values+ if file_md5s and uri in file_md5s+ },+ file_abstracts={+ uri: file_abstracts[uri]+ for values in changes.values()+ for uri in values+ if file_abstracts and uri in file_abstracts+ },)semantic_actions.append(action)@@ -580,6 +615,8 @@ class ContentWriteCoordinator:recursive: bool = False,force_refresh: bool = False,ingest_options: IngestOptions | None = None,+ file_md5s: dict[str, str] | None = None,+ file_abstracts: dict[str, str] | None = None,) -> FreshnessAction:changed_entries = len({uri for values in changes.values() for uri in values})semantic_config = get_openviking_config().semantic@@ -634,6 +671,8 @@ class ContentWriteCoordinator:},generation_trigger="content_write",aggregate_directory=aggregate_directory,+ file_md5s=file_md5s,+ file_abstracts=file_abstracts,)if msg.telemetry_id:get_request_wait_tracker().register_semantic_root(msg.telemetry_id, msg.id)@@ -798,6 +837,8 @@ class ContentWriteCoordinator:) from excprevious_content: Optional[str] = None+ final_content = content.encode("utf-8")+ file_abstract = ""content_written = Falsepost_process_started = Falselock_released = False@@ -810,9 +851,15 @@ class ContentWriteCoordinator:raise InvalidArgumentError(f"cannot create generated abstract overview directly: {uri}")+ if (+ mode != "create"+ and not is_abstract_overview_uri(uri)+ and processing_mode != VECTORS_ONLY+ ):+ file_abstract = (await self._load_file_abstracts([uri], ctx=ctx)).get(uri, "")if wait and telemetry_id:get_request_wait_tracker().register_request(telemetry_id)- await self._write_in_place(+ written_content = await self._write_in_place(uri,content,mode=mode,@@ -820,6 +867,8 @@ class ContentWriteCoordinator:lease_ref=lease,existing_raw=previous_content,)+ if written_content is not None:+ final_content = written_contentcontent_written = Trueif is_abstract_overview_uri(uri):vector_enqueued = await self._vectorize_abstract_overview(@@ -833,6 +882,7 @@ class ContentWriteCoordinator:ctx=ctx,creator_acl_grant=(CreatorAclGrant.DIRECT if mode == "create" else None),ingest_options=ingest_options,+ file_md5=content_md5(final_content),)post_process_started = Trueelse:@@ -844,6 +894,8 @@ class ContentWriteCoordinator:change_type="added" if mode == "create" else "modified",force_refresh=wait,ingest_options=ingest_options,+ file_md5=content_md5(final_content),+ file_abstract=file_abstract,)post_process_started = Trueawait self._viking_fs._async_agfs.pathlock_release(lease)@@ -890,7 +942,7 @@ class ContentWriteCoordinator:uri=uri,root_uri=root_uri,context_type=context_type,- mode=mode,+ mode=response_mode or mode,written_bytes=written_bytes,wait=wait,queue_status=queue_status,@@ -943,6 +995,7 @@ class ContentWriteCoordinator:ctx: RequestContext,creator_acl_grant: CreatorAclGrant | None = None,ingest_options: IngestOptions | None = None,+ file_md5: str | None = None,) -> bool:parent = VikingURI(uri).parentif parent is None:@@ -956,6 +1009,7 @@ class ContentWriteCoordinator:ctx=ctx,creator_acl_grant=creator_acl_grant,ingest_options=ingest_options,+ file_md5=file_md5,)async def _vectorize_abstract_overview(@@ -1121,13 +1175,13 @@ class ContentWriteCoordinator:async def _write_in_place(self,uri: str,- content: str,+ content: str | bytes,*,mode: str,ctx: RequestContext,lease_ref: Optional[Dict[str, Any]] = None,- existing_raw: Optional[str] = None,- ) -> None:+ existing_raw: str | bytes | None = None,+ ) -> bytes:if is_abstract_overview_uri(uri):current_raw = (existing_raw@@ -1141,7 +1195,7 @@ class ContentWriteCoordinator:mode=mode,)await self._viking_fs.write_file(uri, rendered, ctx=ctx, lease_ref=lease_ref)- return+ return rendered.encode("utf-8")if context_type_for_uri(uri) == "memory":if mode == "replace":@@ -1155,25 +1209,50 @@ class ContentWriteCoordinator:else:mf = MemoryFileUtils.read(content, uri=uri)sync_memory_resource_refs(mf, source=RESOURCE_REF_SOURCE_CONTENT_WRITE)+ rendered = MemoryFileUtils.write(mf)await self._viking_fs.write_file(uri,- MemoryFileUtils.write(mf),+ rendered,ctx=ctx,lease_ref=lease_ref,)- return+ return rendered.encode("utf-8")if mode == "append":# Plain concatenation for resource/skill files: MEMORY_FIELDS is a# reserved trailer of memory namespaces only (see content_visibility),# so non-memory appends must not round-trip through MemoryFileUtils# (which strips trailing newlines and injects a metadata trailer).- existing_raw = await self._viking_fs.read_file(uri, ctx=ctx)- await self._viking_fs.write_file(- uri, existing_raw + content, ctx=ctx, lease_ref=lease_ref+ existing_raw = (+ existing_raw+ if existing_raw is not None+ else await self._viking_fs.read_file(uri, ctx=ctx))- return+ if not isinstance(existing_raw, str) or not isinstance(content, str):+ raise InvalidArgumentError(f"append only supports text content: {uri}")+ final_content = existing_raw + content+ await self._viking_fs.write_file(uri, final_content, ctx=ctx, lease_ref=lease_ref)+ return final_content.encode("utf-8")await self._viking_fs.write_file(uri, content, ctx=ctx, lease_ref=lease_ref)+ return content if isinstance(content, bytes) else content.encode("utf-8")++ async def _load_file_abstracts(self, uris: list[str], *, ctx: RequestContext) -> dict[str, str]:+ vector_store = self._vikingdb+ if vector_store is None:+ get_vector_store = getattr(self._viking_fs, "_get_vector_store", None)+ vector_store = get_vector_store() if callable(get_vector_store) else None+ if vector_store is None or not uris:+ return {}+ try:+ records = await vector_store.get_l2_diff_records_by_uris(sorted(set(uris)), ctx=ctx)+ except Exception as exc:+ logger.warning("Failed to load existing file abstracts: %s", exc)+ return {}+ return {+ uri: str(record.get("abstract") or "")+ for uri, record in records.items()+ if uri in uris and record.get("abstract") is not None+ }@staticmethoddef _prepare_abstract_overview_content(@@ -1196,6 +1275,8 @@ class ContentWriteCoordinator:recursive: bool = False,force_refresh: bool = False,ingest_options: IngestOptions | None = None,+ file_md5: str | None = None,+ file_abstract: str = "",) -> FreshnessAction:return await self._enqueue_semantic_refresh_changes(root_uri=root_uri,@@ -1206,6 +1287,8 @@ class ContentWriteCoordinator:recursive=recursive,force_refresh=force_refresh,ingest_options=ingest_options,+ file_md5s={changed_uri: file_md5} if file_md5 else None,+ file_abstracts={changed_uri: file_abstract} if file_abstract else None,)async def _wait_for_queues(self, *, timeout: Optional[float]) -> Dict[str, Any]:@@ -1523,7 +1606,7 @@ class ContentWriteCoordinator:if anchor_to_parent:# Content writes anchor the semantic refresh at the written file's# direct parent directory, so the changed file is a direct child of- # the DAG run root: its own L2 vector and the parent's L0/L1 are+ # the semantic-tree root: its own L2 vector and the parent's L0/L1 are# (re)generated from a single-directory run, while ancestor summaries# refresh via the existing parent bubble. Collapsing to the project# root instead would force the run to traverse the whole projectdiff --git a/openviking/storage/context_update_plan.py b/openviking/storage/context_update_plan.pynew file mode 100644index 000000000..1c3de1253--- /dev/null+++ b/openviking/storage/context_update_plan.py@@ -0,0 +1,1208 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Build and execute explicit content, semantic and index actions."""++from __future__ import annotations++import logging+from dataclasses import asdict, dataclass, field+from enum import Enum+from pathlib import PurePosixPath+from typing import Any, Mapping++from openviking.concurrency import bounded_map+from openviking.storage.index_action import FieldPatch, IndexAction+from openviking.storage.resource_diff import ContentState, IndexState+from openviking.storage.resource_rnfv import (+ NON_PORTABLE_VECTOR_RECORD_FIELDS,+ FormalEntry,+ FormalTreeSnapshot,+ NewArtifactSnapshot,+ NewEntry,+ RequestIntent,+ RNFVSnapshot,+ VectorRecordSnapshot,+ canonical_vector_records_by_level,+)+from openviking.storage.vector_ids import vector_record_id+from openviking.utils.ingest_options import IngestOptions+from openviking.utils.log_correlation import log_correlation+from openviking_cli.utils import VikingURI++logger = logging.getLogger(__name__)+++class SemanticAction(str, Enum):+ """Per-node semantic work executed by the minimal semantic tree."""++ # Consume a hydrated existing abstract; do not call a semantic model.+ REUSE = "reuse"+ # Generate a file abstract from the current formal file contents.+ GENERATE = "generate"+ # Generate directory L0/L1 bodies from direct child abstracts.+ AGGREGATE = "aggregate"+++class FileVectorSource(str, Enum):+ """Input selected for a file's embedding request."""++ # Embed the current file body using the configured text-source policy.+ CONTENT = "content"+ # Prefer the generated file summary when one is available (code repositories).+ SUMMARY_WHEN_AVAILABLE = "summary_when_available"+++@dataclass(frozen=True)+class ParentPropagation:+ enabled: bool = True+++@dataclass(frozen=True)+class FileRefreshIntent:+ """Refresh a flat file and its parent after synchronous content commit."""++ file_uri: str+ md5: str | None = None++ def __post_init__(self) -> None:+ if not self.file_uri.startswith("viking://"):+ raise ValueError("file refresh requires a Viking URI")+++def _validate_relative_path(value: str) -> str:+ if not isinstance(value, str):+ raise ValueError("relative_path must be a string")+ if not value:+ return value+ path = PurePosixPath(value)+ if path.is_absolute() or value != str(path) or any(part == ".." for part in path.parts):+ raise ValueError(f"invalid relative path: {value}")+ return value+++def _validate_index_fields(fields: Mapping[str, Any]) -> None:+ if set(fields) & {"id", "uri", "level", "vector", "sparse_vector"}:+ raise ValueError("index fields contain identity or vector payload")+++@dataclass(frozen=True)+class IndexSlot:+ """Planned operation for one semantic node and one vector level.++ Existing record identity and portable scalar fields travel with the slot so a+ later embedding worker can update the real stored record rather than derive a+ replacement ID locally. ``DELETE`` and pure ``UPDATE_FIELDS`` actions never+ belong here because they do not depend on semantic output.+ """++ level: int+ record_id: str+ existing_fields: Mapping[str, Any] | None = None+ action: IndexAction = IndexAction.NONE+ upsert_fields: Mapping[str, Any] = field(default_factory=dict)+ field_patch: FieldPatch | None = None+ fallback_to_patch: bool = False++ def __post_init__(self) -> None:+ object.__setattr__(self, "action", IndexAction(self.action))+ if self.level not in {0, 1, 2} or not self.record_id:+ raise ValueError("index slot requires a level and record ID")+ if self.action is IndexAction.DELETE:+ raise ValueError("delete belongs to direct index actions")+ if self.action is IndexAction.UPDATE_FIELDS:+ raise ValueError("semantic index slots only supports none, upsert, or merge")+ _validate_index_fields(self.existing_fields or {})+ _validate_index_fields(self.upsert_fields)+ if self.action is IndexAction.NONE and (self.upsert_fields or self.field_patch):+ raise ValueError("none index slot cannot carry index mutations")+ if self.fallback_to_patch and self.field_patch is None:+ raise ValueError("fallback_to_patch requires a field patch")+ if self.action is IndexAction.MERGE and self.upsert_fields:+ raise ValueError("merge index slot uses field_patch, not upsert_fields")++ @property+ def abstract(self) -> str:+ return str((self.existing_fields or {}).get("abstract") or "")++ def scalar_override(self) -> dict[str, Any]:+ existing = {+ key: value+ for key, value in (self.existing_fields or {}).items()+ if key not in NON_PORTABLE_VECTOR_RECORD_FIELDS and not key.startswith("_")+ }+ return {+ **({} if self.action is IndexAction.MERGE else existing),+ **dict(self.upsert_fields),+ "_record_id": self.record_id,+ }++ @classmethod+ def from_dict(cls, data: Mapping[str, Any]) -> "IndexSlot":+ values = dict(data)+ if any(+ name in values+ for name in ("fields", "update_fields", "field_modes", "fallback_update_fields")+ ):+ legacy_fields = dict(values.pop("fields", {}) or {})+ legacy_update = dict(values.pop("update_fields", {}) or {})+ legacy_modes = dict(values.pop("field_modes", {}) or {})+ action = IndexAction(values.get("action", IndexAction.NONE))+ values.setdefault("upsert_fields", {} if action is IndexAction.MERGE else legacy_fields)+ patch_values = legacy_update or (legacy_fields if action is IndexAction.MERGE else {})+ values.setdefault(+ "field_patch",+ FieldPatch(patch_values, legacy_modes) if patch_values else None,+ )+ legacy_fallback = bool(values.pop("fallback_update_fields", False))+ # Older planners marked every modified semantic node for fallback,+ # even when there was no scalar patch to apply. Normalize that+ # inert state while decoding already queued plans.+ values["fallback_to_patch"] = legacy_fallback and bool(patch_values)+ values.setdefault("upsert_fields", {})+ if isinstance(values.get("field_patch"), Mapping):+ values["field_patch"] = FieldPatch.from_dict(values["field_patch"])+ return cls(**values)+++@dataclass(frozen=True)+class SemanticTreeEntry:+ """A retained node in the serializable minimal semantic-tree closure.++ ``REUSE`` entries provide existing child abstracts needed by an ancestor but+ do not run work themselves. Active file entries ``GENERATE`` and active+ directory entries ``AGGREGATE``.+ """++ relative_path: str+ kind: str+ content_state: ContentState+ semantic_action: SemanticAction+ md5: str | None = None+ index_slots: tuple[IndexSlot, ...] = ()+ membership_changed: bool = False+ repair: bool = False++ def __post_init__(self) -> None:+ _validate_relative_path(self.relative_path)+ object.__setattr__(self, "content_state", ContentState(self.content_state))+ object.__setattr__(self, "semantic_action", SemanticAction(self.semantic_action))+ if self.kind not in {"file", "directory"}:+ raise ValueError("invalid semantic entry kind")+ if self.kind == "file" and self.semantic_action is SemanticAction.AGGREGATE:+ raise ValueError("file semantic action must be generate or reuse")+ if self.kind == "directory" and self.semantic_action is SemanticAction.GENERATE:+ raise ValueError("directory semantic action must be aggregate or reuse")+ if self.kind == "directory" and self.md5:+ raise ValueError("directory entry cannot carry md5")+ levels = [slot.level for slot in self.index_slots]+ if len(levels) != len(set(levels)):+ raise ValueError(f"duplicate index slot level for {self.relative_path}")+ if self.semantic_action is SemanticAction.REUSE:+ if any(slot.action is not IndexAction.NONE for slot in self.index_slots):+ raise ValueError("reuse semantic entry cannot contain index mutations")+ slot = self.slot(2 if self.kind == "file" else 0)+ if slot is None or not slot.abstract.strip():+ raise ValueError("reuse requires an existing abstract")++ def slot(self, level: int) -> IndexSlot | None:+ return next((slot for slot in self.index_slots if slot.level == level), None)++ @classmethod+ def from_dict(cls, data: Mapping[str, Any]) -> "SemanticTreeEntry":+ values = dict(data)+ values["index_slots"] = tuple(+ IndexSlot.from_dict(item) for item in values.get("index_slots", ())+ )+ return cls(**values)+++@dataclass(frozen=True)+class SemanticTreeSnapshot:+ """Compact tree retained after diff closure and vector hydration."""++ entries: tuple[SemanticTreeEntry, ...]++ def __post_init__(self) -> None:+ paths = [entry.relative_path for entry in self.entries]+ if len(paths) != len(set(paths)):+ raise ValueError("duplicate semantic tree entry path")++ @classmethod+ def from_dict(cls, data: Mapping[str, Any]) -> "SemanticTreeSnapshot":+ return cls(+ entries=tuple(SemanticTreeEntry.from_dict(item) for item in data.get("entries", ()))+ )+++@dataclass(frozen=True)+class SemanticPlan:+ """Asynchronous semantic/derived-index portion of one context update.++ The plan contains only final resource URIs and hydrated non-vector facts. It+ must never require a parser artifact, local file path, or another RNFV scan+ after it has crossed the durable semantic queue boundary.+ """++ root_uri: str+ context_type: str+ tree: SemanticTreeSnapshot+ vectorize: bool = True+ propagation: ParentPropagation = field(default_factory=ParentPropagation)+ file_vector_source: FileVectorSource = FileVectorSource.CONTENT+ ingest_options: IngestOptions = field(default_factory=IngestOptions)+ source_metadata: Mapping[str, str] | None = None++ def __post_init__(self) -> None:+ object.__setattr__(self, "root_uri", VikingURI(self.root_uri).uri.rstrip("/"))+ object.__setattr__(self, "file_vector_source", FileVectorSource(self.file_vector_source))+ object.__setattr__(self, "ingest_options", IngestOptions.from_value(self.ingest_options))+ if self.context_type not in {"resource", "skill", "memory"}:+ raise ValueError(f"invalid context_type: {self.context_type}")+ entries_by_path = {entry.relative_path: entry for entry in self.tree.entries}+ if entries_by_path and "" not in entries_by_path:+ raise ValueError("semantic plan must contain the resource root")+ for entry in entries_by_path.values():+ if not entry.relative_path:+ continue+ parent = _parent(entry.relative_path)+ if parent not in entries_by_path:+ raise ValueError(+ f"semantic plan lacks parent {parent!r} for {entry.relative_path!r}"+ )+ current = parent+ while True:+ if current not in entries_by_path:+ raise ValueError(+ f"semantic plan lacks ancestor {current!r} for {entry.relative_path!r}"+ )+ ancestor_entry = entries_by_path[current]+ if (+ entry.semantic_action is not SemanticAction.REUSE+ and ancestor_entry.semantic_action is not SemanticAction.AGGREGATE+ ):+ raise ValueError(+ f"semantic plan ancestor {current!r} must aggregate active descendant "+ f"{entry.relative_path!r}"+ )+ if not current:+ break+ current = _parent(current)++ def to_dict(self) -> dict[str, Any]:+ data = asdict(self)+ data["file_vector_source"] = self.file_vector_source.value+ data["ingest_options"] = self.ingest_options.to_dict()+ return data++ @classmethod+ def from_dict(cls, data: Mapping[str, Any]) -> "SemanticPlan":+ return cls(+ root_uri=str(data["root_uri"]),+ context_type=str(data["context_type"]),+ tree=SemanticTreeSnapshot.from_dict(data.get("tree", {})),+ vectorize=bool(data.get("vectorize", True)),+ propagation=ParentPropagation(**dict(data.get("propagation", {}))),+ file_vector_source=FileVectorSource(+ data.get("file_vector_source", FileVectorSource.CONTENT.value)+ ),+ ingest_options=IngestOptions.from_value(data.get("ingest_options")),+ source_metadata=data.get("source_metadata"),+ )++ def execution_root_uris(self) -> tuple[str, ...]:+ directories = {+ entry.relative_path+ for entry in self.tree.entries+ if entry.kind == "directory" and entry.semantic_action is SemanticAction.AGGREGATE+ }+ roots = sorted(+ path+ for path in directories+ if not any(+ parent != path+ and (not parent or path.startswith(parent + "/"))+ and parent in directories+ for parent in directories+ )+ )+ if not roots and any(+ entry.semantic_action is not SemanticAction.REUSE for entry in self.tree.entries+ ):+ roots = [""]+ return tuple(self.root_uri if not path else f"{self.root_uri}/{path}" for path in roots)+++class ContentTreeOperation(str, Enum):+ """Synchronous formal-tree mutation performed before queue handoff."""++ # Write a new file or ensure a planned directory exists.+ UPSERT = "upsert"+ # Remove an existing file or subtree.+ DELETE = "delete"+ # Delete the old kind before creating the different new kind.+ REPLACE_KIND = "replace_kind"+++@dataclass(frozen=True)+class ContentTreeAction:+ """One synchronous mutation from a parse artifact into the formal tree.++ File writes carry the final artifact-relative source and MD5. Directory+ actions only establish topology; they do not invent a directory fingerprint.+ """++ operation: ContentTreeOperation+ relative_path: str+ old_kind: str | None = None+ new_kind: str | None = None+ artifact_path: str | None = None+ md5: str | None = None++ def __post_init__(self) -> None:+ object.__setattr__(self, "operation", ContentTreeOperation(self.operation))+ _validate_relative_path(self.relative_path)+ if self.artifact_path is not None:+ _validate_relative_path(self.artifact_path)+ if self.operation is not ContentTreeOperation.DELETE and self.new_kind == "file":+ if self.artifact_path is None or not self.md5:+ raise ValueError("file write requires artifact_path and md5")+ if self.operation is ContentTreeOperation.REPLACE_KIND and (+ not self.old_kind or not self.new_kind or self.old_kind == self.new_kind+ ):+ raise ValueError("replacement requires different old and new kinds")+++@dataclass(frozen=True)+class DirectIndexAction:+ """Direct vector operation independent of semantic model output.++ Deletes remove exact inventory IDs. ``UPDATE_FIELDS`` mutates only approved+ scalars. Direct upserts are restricted to file L2 vectors in vectors-only+ processing, where no semantic node produces the embedding request.+ """++ action: IndexAction+ uri: str+ level: int+ record_id: str+ upsert_fields: Mapping[str, Any] = field(default_factory=dict)+ field_patch: FieldPatch | None = None+ md5: str | None = None++ def __post_init__(self) -> None:+ object.__setattr__(self, "action", IndexAction(self.action))+ if self.action is IndexAction.NONE or self.level not in {0, 1, 2}:+ raise ValueError("invalid direct index action")+ if not self.record_id:+ raise ValueError("direct index action requires a record ID")+ _validate_index_fields(self.upsert_fields)+ if self.action is IndexAction.DELETE and (self.upsert_fields or self.field_patch):+ raise ValueError("delete cannot carry index fields")+ if self.action is IndexAction.UPDATE_FIELDS and (+ self.field_patch is None or not self.field_patch.values or self.upsert_fields+ ):+ raise ValueError("field update requires only a non-empty field patch")+ if self.action is IndexAction.UPSERT and self.field_patch is not None:+ raise ValueError("upsert uses resolved upsert_fields, not a field patch")+ if self.action is IndexAction.MERGE and self.upsert_fields:+ raise ValueError("merge uses field_patch, not upsert_fields")+ if self.action in {IndexAction.DELETE, IndexAction.UPDATE_FIELDS} and self.md5 is not None:+ raise ValueError(f"{self.action.value} cannot carry a vectorization md5")++ @classmethod+ def from_dict(cls, data: Mapping[str, Any]) -> "DirectIndexAction":+ values = dict(data)+ if any(+ name in values+ for name in ("fields", "field_modes", "initial_fields", "search_tag_mode")+ ):+ legacy_fields = dict(values.pop("fields", {}) or {})+ legacy_modes = dict(values.pop("field_modes", {}) or {})+ legacy_seed = dict(values.pop("initial_fields", {}) or {})+ values.pop("search_tag_mode", None)+ action = IndexAction(values.get("action"))+ values.setdefault(+ "upsert_fields", legacy_fields if action is IndexAction.UPSERT else {}+ )+ values.setdefault(+ "field_patch",+ (+ FieldPatch(legacy_fields, legacy_modes, legacy_seed)+ if action in {IndexAction.MERGE, IndexAction.UPDATE_FIELDS} and legacy_fields+ else None+ ),+ )+ values.setdefault("upsert_fields", {})+ if isinstance(values.get("field_patch"), Mapping):+ values["field_patch"] = FieldPatch.from_dict(values["field_patch"])+ return cls(**values)+++@dataclass(frozen=True)+class ContextUpdatePlan:+ """Complete execution contract for one resource-tree update.++ Content actions run synchronously under the resource lock. The remaining+ semantic plan and direct index actions are durable queue work. A no-op plan+ therefore means no formal-tree mutation, semantic generation, or vector+ mutation is required.+ """++ root_uri: str+ context_type: str+ content_tree_actions: tuple[ContentTreeAction, ...] = ()+ semantic_plan: SemanticPlan | None = None+ direct_index_actions: tuple[DirectIndexAction, ...] = ()+ file_refresh: FileRefreshIntent | None = None++ def __post_init__(self) -> None:+ root = self.root_uri.rstrip("/")+ paths = [action.relative_path for action in self.content_tree_actions]+ if len(paths) != len(set(paths)):+ raise ValueError("conflicting content actions")+ record_ids: set[str] = set()+ for action in self.direct_index_actions:+ if action.uri != root and not action.uri.startswith(root + "/"):+ raise ValueError("index action outside root")+ if action.record_id in record_ids:+ raise ValueError("conflicting index actions")+ record_ids.add(action.record_id)+ if self.semantic_plan is not None:+ if (+ self.semantic_plan.root_uri != root+ or self.semantic_plan.context_type != self.context_type+ ):+ raise ValueError("semantic plan identity mismatch")+ for entry in self.semantic_plan.tree.entries:+ for slot in entry.index_slots:+ if slot.action is IndexAction.NONE:+ continue+ if slot.record_id in record_ids:+ raise ValueError("conflicting index actions")+ record_ids.add(slot.record_id)+ if self.file_refresh is not None and self.file_refresh.file_uri != root:+ raise ValueError("file refresh must target the plan root")++ def is_noop(self) -> bool:+ return (+ not self.content_tree_actions+ and self.semantic_plan is None+ and not self.direct_index_actions+ and self.file_refresh is None+ )++ def after_content_commit(self) -> "ContextUpdatePlan":+ """Drop synchronous content actions before the async handoff."""+ if not self.content_tree_actions:+ return self+ return ContextUpdatePlan(+ root_uri=self.root_uri,+ context_type=self.context_type,+ semantic_plan=self.semantic_plan,+ direct_index_actions=self.direct_index_actions,+ file_refresh=self.file_refresh,+ )++ def to_dict(self) -> dict[str, Any]:+ data = asdict(self)+ if self.semantic_plan is not None:+ data["semantic_plan"] = self.semantic_plan.to_dict()+ return data++ @classmethod+ def from_dict(cls, data: Mapping[str, Any]) -> "ContextUpdatePlan":+ return cls(+ root_uri=str(data["root_uri"]),+ context_type=str(data["context_type"]),+ content_tree_actions=tuple(+ ContentTreeAction(**item) for item in data.get("content_tree_actions", ())+ ),+ semantic_plan=(+ SemanticPlan.from_dict(data["semantic_plan"]) if data.get("semantic_plan") else None+ ),+ direct_index_actions=tuple(+ DirectIndexAction.from_dict(item) for item in data.get("direct_index_actions", ())+ ),+ file_refresh=(+ FileRefreshIntent(**data["file_refresh"]) if data.get("file_refresh") else None+ ),+ )+++def _uri(root_uri: str, relative_path: str) -> str:+ return root_uri if not relative_path else f"{root_uri}/{relative_path}"+++def _parent(relative_path: str) -> str:+ value = str(PurePosixPath(relative_path).parent)+ return "" if value == "." else value+++def _records_by_path(+ records: Mapping[str, VectorRecordSnapshot],+) -> tuple[+ dict[str, dict[int, VectorRecordSnapshot]],+ tuple[VectorRecordSnapshot, ...],+]:+ result: dict[str, dict[int, VectorRecordSnapshot]] = {}+ duplicates: list[VectorRecordSnapshot] = []+ records_by_relative_path: dict[str, list[VectorRecordSnapshot]] = {}+ for record in records.values():+ records_by_relative_path.setdefault(record.relative_path, []).append(record)+ for relative_path, path_records in records_by_relative_path.items():+ levels, path_duplicates = canonical_vector_records_by_level(path_records)+ result[relative_path] = levels+ duplicates.extend(path_duplicates)+ return result, tuple(duplicates)+++def _portable_existing_fields(record: VectorRecordSnapshot | None) -> dict[str, Any] | None:+ if record is None:+ return None+ return {+ key: value+ for key, value in record.fields.items()+ if key not in {"id", "uri", "level", "vector", "sparse_vector", "content"}+ and value is not None+ }+++def _field_patch(request: RequestIntent, record: VectorRecordSnapshot | None) -> FieldPatch | None:+ existing = dict(record.fields) if record is not None else {}+ values: dict[str, Any] = {}+ modes: dict[str, str] = {}+ for intent in request.scalar_intents:+ if record is not None and record.level not in intent.target_levels:+ continue+ if intent.field != "search_tags":+ continue+ candidate = FieldPatch({intent.field: intent.value}, {intent.field: intent.mode})+ desired = candidate.resolve(existing).get(intent.field)+ if record is None or existing.get(intent.field) != desired:+ values[intent.field] = intent.value+ modes[intent.field] = intent.mode+ return FieldPatch(values, modes) if values else None+++def _semantic_closure(+ diff: Any,+ new_kinds: Mapping[str, str],+ *,+ repair_indexes: bool = True,+) -> tuple[set[str], set[str], set[str]]:+ """Return active nodes, membership-changing parents, and retained closure.++ Active content/index repair nodes and all their ancestors must aggregate.+ Direct children of every active directory are retained as reusable inputs, so+ aggregation sees the same sibling set without traversing unchanged subtrees.+ """+ active: set[str] = set()+ membership_changed: set[str] = set()++ def add_ancestors(path: str) -> None:+ parent = _parent(path)+ while True:+ active.add(parent)+ if not parent:+ return+ parent = _parent(parent)++ for path, entry in diff.entries.items():+ content_state = ContentState(entry.content_state)+ index_state = IndexState(entry.index_state)+ content_requires_semantics = content_state in {+ ContentState.ADDED,+ ContentState.MODIFIED,+ ContentState.REPLACE_KIND,+ } or (content_state is ContentState.RESTORE and index_state is not IndexState.COMPLETE)+ if (+ content_requires_semantics+ or (+ repair_indexes+ and index_state+ in {IndexState.MISSING, IndexState.PARTIAL, IndexState.LEVEL_CONFLICT}+ )+ ) and entry.new_kind in {"file", "directory"}:+ active.add(path)+ add_ancestors(path)+ if content_state in {+ ContentState.ADDED,+ ContentState.DELETED,+ ContentState.RESTORE,+ ContentState.REPLACE_KIND,+ }:+ membership_changed.add(_parent(path))+ add_ancestors(path)++ retained = set(active)+ for path in new_kinds:+ if path and _parent(path) in active:+ retained.add(path)+ return active, membership_changed, retained+++async def hydrate_context_plan_records(+ *,+ diff: Any,+ new_kinds: Mapping[str, str],+ inventory: Mapping[str, VectorRecordSnapshot],+ vikingdb: Any,+ ctx: Any,+ request: RequestIntent | None = None,+ repair_indexes: bool = True,+) -> tuple[Mapping[str, VectorRecordSnapshot], tuple[set[str], set[str], set[str]]]:+ request = request or RequestIntent("", "semantic_and_vectors")+ if not request.vectorize:+ inventory = {}+ active, membership_changed, retained = _semantic_closure(+ diff, new_kinds, repair_indexes=repair_indexes+ )+ result = dict(inventory)+ summary_hydrated_ids: set[str] = set()++ def required_summary_ids() -> dict[str, Mapping[str, Any]]:+ required: dict[str, Mapping[str, Any]] = {}+ for record_id, record in inventory.items():+ if record.relative_path not in retained:+ continue+ kind = new_kinds.get(record.relative_path)+ wanted = {2} if kind == "file" else {0, 1} if record.relative_path in active else {0}+ if (+ record.level in wanted+ and record_id not in summary_hydrated_ids+ and not str(record.fields.get("abstract") or "").strip()+ ):+ required[record_id] = {"uri": record.uri, "level": record.level}+ return required++ def merge_hydrated(payloads: Mapping[str, Mapping[str, Any]]) -> None:+ for record_id, payload in payloads.items():+ current = result[record_id]+ fields = {+ key: value+ for key, value in payload.items()+ if key not in {"id", "uri", "level", "vector", "sparse_vector", "content"}+ and value is not None+ }+ result[record_id] = VectorRecordSnapshot(+ current.record_id,+ current.uri,+ current.relative_path,+ current.level,+ {**current.fields, **fields},+ )++ while True:+ required = required_summary_ids()+ hydrated = (+ await vikingdb.hydrate_incremental_records(+ required, ctx=ctx, output_fields={"abstract"}+ )+ if required+ else {}+ )+ # Inventory and hydration are separate reads. A missing hydration result+ # has no reusable abstract, so the dependency is promoted below. Keep+ # the inventory identity: an existing record must never be replaced by a+ # locally recomputed ID merely because its second read raced or failed.+ merge_hydrated(hydrated)+ summary_hydrated_ids.update(required)++ by_path, _ = _records_by_path(result)+ promoted = {+ path+ for path in retained - active+ if (+ (record := by_path.get(path, {}).get(2 if new_kinds.get(path) == "file" else 0))+ is None+ or not str(record.fields.get("abstract") or "").strip()+ )+ }+ if not promoted:+ break+ active.update(promoted)+ for path in new_kinds:+ if path and _parent(path) in active:+ retained.add(path)++ if request.vectorize:+ required_scalars: dict[str, Mapping[str, Any]] = {}+ for record_id, record in result.items():+ kind = new_kinds.get(record.relative_path)+ if record.relative_path not in active:+ continue+ if request.processing_mode == "vectors_only":+ if kind != "file" or record.level != 2:+ continue+ elif record.level not in ({2} if kind == "file" else {0, 1}):+ continue+ required_scalars[record_id] = {"uri": record.uri, "level": record.level}+ if required_scalars:+ merge_hydrated(await vikingdb.hydrate_incremental_records(required_scalars, ctx=ctx))++ return result, (active, membership_changed, retained)+++def build_context_update_plan(+ *,+ root_uri: str,+ context_type: str,+ request: RequestIntent,+ diff: Any,+ new_kinds: Mapping[str, str],+ artifact_paths: Mapping[str, str],+ records: Mapping[str, VectorRecordSnapshot],+ is_code_repo: bool,+ account_id: str,+ ingest_options: IngestOptions | Mapping[str, Any] | None = None,+ source_metadata: Mapping[str, str] | None = None,+ closure: tuple[set[str], set[str], set[str]] | None = None,+) -> ContextUpdatePlan:+ """Compile resolved RNFV facts into synchronous and asynchronous actions.++ Content mutations happen before queue handoff. Semantic-dependent index work+ becomes ``IndexSlot`` data; pure deletes and scalar-only changes become direct+ embedding-queue actions. Existing vector record IDs are preserved when an+ existing same-level record is overwritten. Portable existing scalars may be+ carried for normal updates; derived content, vectors, and ACL fields always+ come from the current execution.+ """+ root_uri = root_uri.rstrip("/")+ records_by_path, duplicate_records = (+ _records_by_path(records) if request.vectorize else ({}, ())+ )+ active, membership_changed, retained = closure or _semantic_closure(+ diff,+ new_kinds,+ repair_indexes=request.vectorize and request.processing_mode != "vectors_only",+ )+ semantic_enabled = request.processing_mode != "vectors_only"+ if not semantic_enabled:+ active, membership_changed, retained = set(), set(), set()++ content_actions: list[ContentTreeAction] = []+ direct_actions: list[DirectIndexAction] = [+ DirectIndexAction(IndexAction.DELETE, record.uri, record.level, record.record_id)+ for record in duplicate_records+ ]+ for path, entry in sorted(diff.entries.items()):+ state = ContentState(entry.content_state)+ kind = entry.new_kind+ if state in {ContentState.ADDED, ContentState.RESTORE, ContentState.MODIFIED}:+ content_actions.append(+ ContentTreeAction(+ ContentTreeOperation.UPSERT,+ path,+ entry.old_kind,+ kind,+ artifact_paths.get(path),+ entry.md5,+ )+ )+ elif state is ContentState.DELETED:+ content_actions.append(+ ContentTreeAction(ContentTreeOperation.DELETE, path, old_kind=entry.old_kind)+ )+ elif state is ContentState.REPLACE_KIND:+ content_actions.append(+ ContentTreeAction(+ ContentTreeOperation.REPLACE_KIND,+ path,+ entry.old_kind,+ kind,+ artifact_paths.get(path),+ entry.md5,+ )+ )++ existing = records_by_path.get(path, {})+ valid_levels = {2} if kind == "file" else {0, 1} if kind == "directory" else set()+ for level, record in existing.items():+ if state in {ContentState.DELETED, ContentState.ABSENT} or level not in valid_levels:+ direct_actions.append(+ DirectIndexAction(IndexAction.DELETE, record.uri, level, record.record_id)+ )+ if (+ not semantic_enabled+ and request.vectorize+ and kind == "file"+ and (+ state+ in {+ ContentState.ADDED,+ ContentState.MODIFIED,+ ContentState.REPLACE_KIND,+ }+ or existing.get(2) is None+ or IndexState(entry.index_state) is IndexState.STALE+ )+ ):+ record = existing.get(2)+ field_patch = _field_patch(request, record) if request.vectorize else None+ index_action = (+ IndexAction.MERGE+ if (+ record is None+ and entry.old_kind == kind+ and IndexState(entry.index_state) in {IndexState.MISSING, IndexState.PARTIAL}+ )+ else IndexAction.UPSERT+ )+ direct_actions.append(+ DirectIndexAction(+ index_action,+ _uri(root_uri, path),+ 2,+ record.record_id+ if record is not None+ else vector_record_id(account_id, _uri(root_uri, path), 2),+ upsert_fields=(+ {+ **(_portable_existing_fields(record) or {}),+ **(+ field_patch.resolve(record.fields if record else {})+ if field_patch+ else {}+ ),+ }+ if index_action is IndexAction.UPSERT+ else {}+ ),+ field_patch=field_patch if index_action is IndexAction.MERGE else None,+ md5=entry.md5,+ )+ )++ scheduled_direct_ids = {action.record_id for action in direct_actions}+ for path, levels in records_by_path.items():+ for record in levels.values():+ field_patch = _field_patch(request, record) if request.vectorize else None+ patch_values = dict(field_patch.values) if field_patch is not None else {}+ diff_entry = diff.entries.get(path)+ if (+ record.level == 2+ and diff_entry is not None+ and ContentState(diff_entry.content_state) is ContentState.UNCHANGED+ and diff_entry.md5+ and not str(record.fields.get("md5") or "")+ ):+ patch_values["md5"] = diff_entry.md5+ field_patch = FieldPatch(+ patch_values,+ field_patch.modes if field_patch is not None else {},+ )+ if (+ field_patch is not None+ and record.record_id not in scheduled_direct_ids+ and (path not in active or not request.vectorize)+ ):+ seed = {+ "uri": record.uri,+ "account_id": account_id,+ "level": record.level,+ **(_portable_existing_fields(record) or {}),+ **field_patch.resolve(record.fields),+ }+ direct_actions.append(+ DirectIndexAction(+ IndexAction.UPDATE_FIELDS,+ record.uri,+ record.level,+ record.record_id,+ field_patch=field_patch.with_seed(seed),+ )+ )+ scheduled_direct_ids.add(record.record_id)++ semantic_entries: list[SemanticTreeEntry] = []+ for path in sorted(retained, key=lambda value: (value.count("/"), value)):+ kind = new_kinds.get(path)+ if kind not in {"file", "directory"}:+ continue+ diff_entry = diff.entries.get(path)+ state = ContentState(diff_entry.content_state) if diff_entry else ContentState.UNCHANGED+ action = (+ SemanticAction.GENERATE+ if kind == "file" and path in active+ else SemanticAction.AGGREGATE+ if kind == "directory" and path in active+ else SemanticAction.REUSE+ )+ slots: list[IndexSlot] = []+ for level in (2,) if kind == "file" else (0, 1):+ record = records_by_path.get(path, {}).get(level)+ if action is SemanticAction.REUSE:+ if level != (2 if kind == "file" else 0):+ continue+ if record is None or not str(record.fields.get("abstract") or "").strip():+ raise ValueError(f"semantic closure lacks reusable abstract for {path}")+ slots.append(IndexSlot(level, record.record_id, _portable_existing_fields(record)))+ continue+ record_id = (+ record.record_id+ if record is not None+ else vector_record_id(account_id, _uri(root_uri, path), level)+ )+ field_patch = _field_patch(request, record) if request.vectorize else None+ index_action = (+ IndexAction.MERGE+ if (+ record is None+ and diff_entry+ and diff_entry.old_kind == kind+ and IndexState(diff_entry.index_state)+ in {IndexState.MISSING, IndexState.PARTIAL}+ )+ else IndexAction.UPSERT+ )+ slots.append(+ IndexSlot(+ level,+ record_id,+ _portable_existing_fields(record),+ action=(index_action if request.vectorize else IndexAction.NONE),+ upsert_fields=(+ field_patch.resolve(record.fields if record else {})+ if index_action is IndexAction.UPSERT and field_patch is not None+ else {}+ ),+ field_patch=field_patch,+ # Keep the original patch mode even for a resolved UPSERT.+ # If semantic output is unchanged, the executor falls back+ # to UPDATE_FIELDS and must still interpret append against+ # the execution-time exact-get record. The normal UPSERT+ # path deliberately does not forward these modes.+ fallback_to_patch=field_patch is not None,+ )+ )+ semantic_entries.append(+ SemanticTreeEntry(+ path,+ kind,+ state,+ action,+ md5=diff_entry.md5 if diff_entry else None,+ index_slots=tuple(slots),+ membership_changed=path in membership_changed,+ repair=bool(+ request.vectorize+ and diff_entry+ and IndexState(diff_entry.index_state)+ in {IndexState.MISSING, IndexState.PARTIAL, IndexState.LEVEL_CONFLICT}+ ),+ )+ )++ semantic_plan = (+ SemanticPlan(+ root_uri,+ context_type,+ SemanticTreeSnapshot(tuple(semantic_entries)),+ vectorize=request.vectorize,+ file_vector_source=(+ FileVectorSource.SUMMARY_WHEN_AVAILABLE+ if is_code_repo+ else FileVectorSource.CONTENT+ ),+ ingest_options=(+ IngestOptions.from_value(ingest_options) if request.vectorize else IngestOptions()+ ),+ source_metadata=source_metadata,+ )+ if semantic_entries+ else None+ )+ return ContextUpdatePlan(+ root_uri,+ context_type,+ tuple(content_actions),+ semantic_plan,+ tuple(direct_actions),+ )+++def _with_directory_root(snapshot: RNFVSnapshot, *, root_preexisting: bool) -> RNFVSnapshot:+ """Add the logical directory root omitted by artifact/tree walks."""+ new_entries = {"": NewEntry(is_dir=True), **snapshot.new.entries}+ formal_entries = (+ {"": FormalEntry(is_dir=True), **snapshot.formal.entries}+ if root_preexisting+ else dict(snapshot.formal.entries)+ )+ return RNFVSnapshot(+ request=snapshot.request,+ new=NewArtifactSnapshot(new_entries, complete=snapshot.new.complete),+ formal=FormalTreeSnapshot(formal_entries, complete=snapshot.formal.complete),+ vectors=snapshot.vectors,+ )+++async def build_context_update_plan_from_snapshot(+ *,+ snapshot: RNFVSnapshot,+ store: Any,+ artifact_ref: Any,+ target: Any,+ vikingdb: Any,+ context_type: str,+ is_code_repo: bool,+ account_id: str,+ ctx: Any,+ root_preexisting: bool,+ artifact_paths: Mapping[str, str] | None = None,+ ingest_options: Any = None,+ source_metadata: Mapping[str, str] | None = None,+ root_is_file: bool = False,+) -> tuple[Any, ContextUpdatePlan]:+ """Resolve RNFV facts, hydrate the minimal closure, and build one plan."""+ from openviking.storage.resource_diff import resolve_resource_diff++ if not root_is_file:+ snapshot = _with_directory_root(snapshot, root_preexisting=root_preexisting)+ paths = dict(artifact_paths or {})+ diff = await resolve_resource_diff(+ snapshot,+ store=store,+ artifact_ref=artifact_ref,+ target=target,+ artifact_paths=paths,+ )+ new_kinds = {+ path: "directory" if entry.is_dir else "file"+ for path, entry in snapshot.new.entries.items()+ }+ records, closure = await hydrate_context_plan_records(+ diff=diff,+ new_kinds=new_kinds,+ inventory=snapshot.vectors.records_by_id,+ vikingdb=vikingdb,+ ctx=ctx,+ request=snapshot.request,+ repair_indexes=(+ snapshot.request.vectorize and snapshot.request.processing_mode != "vectors_only"+ ),+ )+ plan = build_context_update_plan(+ root_uri=snapshot.request.target_uri,+ context_type=context_type,+ request=snapshot.request,+ diff=diff,+ new_kinds=new_kinds,+ artifact_paths=paths,+ records=records,+ is_code_repo=is_code_repo,+ account_id=account_id,+ ingest_options=ingest_options,+ source_metadata=source_metadata,+ closure=closure,+ )+ if not root_is_file:+ return diff, plan++ root_entry = diff.entries[""]+ needs_refresh = ContentState(root_entry.content_state) in {+ ContentState.ADDED,+ ContentState.RESTORE,+ ContentState.MODIFIED,+ ContentState.REPLACE_KIND,+ } or (+ snapshot.request.vectorize+ and IndexState(root_entry.index_state)+ in {IndexState.MISSING, IndexState.PARTIAL, IndexState.STALE, IndexState.LEVEL_CONFLICT}+ )+ return diff, ContextUpdatePlan(+ root_uri=plan.root_uri,+ context_type=plan.context_type,+ content_tree_actions=plan.content_tree_actions,+ direct_index_actions=plan.direct_index_actions,+ file_refresh=(+ FileRefreshIntent(snapshot.request.target_uri, root_entry.md5)+ if needs_refresh and snapshot.request.processing_mode != "vectors_only"+ else None+ ),+ )+++async def execute_content_tree_actions(+ actions: tuple[ContentTreeAction, ...],+ *,+ store: Any,+ artifact_ref: Any,+ target: Any,+ concurrency: int | None = None,+) -> None:+ """Commit planned content mutations before any asynchronous work."""++ async def run_action(action: ContentTreeAction, operation: Any) -> None:+ try:+ await operation+ except Exception:+ logger.exception(+ "[ContentTreeActionFailed] %s operation=%s relative_path=%s "+ "old_kind=%s new_kind=%s",+ log_correlation(),+ action.operation.value,+ action.relative_path,+ action.old_kind or "-",+ action.new_kind or "-",+ )+ raise++ destructive = [+ action+ for action in actions+ if action.operation in {ContentTreeOperation.DELETE, ContentTreeOperation.REPLACE_KIND}+ ]+ for action in sorted(+ destructive, key=lambda item: (-item.relative_path.count("/"), item.relative_path)+ ):+ await run_action(+ action,+ target.delete_path(action.relative_path, is_dir=action.old_kind == "directory"),+ )+ for action in sorted(+ (action for action in actions if action.new_kind == "directory"),+ key=lambda item: (item.relative_path.count("/"), item.relative_path),+ ):+ await run_action(action, target.mkdir(action.relative_path))+ file_actions = [action for action in actions if action.new_kind == "file"]+ if not file_actions:+ return+ if concurrency is None:+ from openviking_cli.utils.config import get_openviking_config++ concurrency = int(+ get_openviking_config().queue_workers.add_resource.file_operation_concurrency+ )++ async def write(action: ContentTreeAction) -> None:+ async def read_and_write() -> None:+ data = await store.read_bytes(artifact_ref, action.artifact_path)+ await target.write_file(action.relative_path, data)++ await run_action(action, read_and_write())++ await bounded_map(file_actions, write, concurrency=max(1, concurrency))+++__all__ = [+ "ContentState",+ "ContentTreeAction",+ "ContentTreeOperation",+ "ContextUpdatePlan",+ "FileVectorSource",+ "FileRefreshIntent",+ "DirectIndexAction",+ "IndexAction",+ "IndexSlot",+ "IndexState",+ "ParentPropagation",+ "SemanticAction",+ "SemanticPlan",+ "SemanticTreeEntry",+ "SemanticTreeSnapshot",+ "build_context_update_plan",+ "build_context_update_plan_from_snapshot",+ "execute_content_tree_actions",+ "hydrate_context_plan_records",+]diff --git a/openviking/storage/index_action.py b/openviking/storage/index_action.pynew file mode 100644index 000000000..e47d59251--- /dev/null+++ b/openviking/storage/index_action.py@@ -0,0 +1,102 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Shared index-operation protocol used by plans and queue messages."""++from __future__ import annotations++from dataclasses import dataclass, field+from enum import Enum+from typing import Any, Mapping++from openviking.utils.tags import merge_search_tags, normalize_search_tags++PATCHABLE_INDEX_FIELDS = frozenset(+ {"md5", "content", "abstract", "updated_at", "active_count", "tags", "search_tags"}+)+_FIELD_MODES = frozenset({"replace", "append"})+++class IndexAction(str, Enum):+ """Action applied to one or more vector-index records."""++ NONE = "none"+ UPSERT = "upsert"+ MERGE = "merge"+ UPDATE_FIELDS = "update_fields"+ DELETE = "delete"+++@dataclass(frozen=True)+class FieldPatch:+ """One execution-time scalar mutation and its missing-record seed."""++ values: Mapping[str, Any]+ modes: Mapping[str, str] = field(default_factory=dict)+ seed_fields: Mapping[str, Any] = field(default_factory=dict)++ def __post_init__(self) -> None:+ values = dict(self.values)+ if "search_tags" in values:+ values["search_tags"] = normalize_search_tags(+ values["search_tags"], discard_invalid=True+ )+ object.__setattr__(self, "values", values)+ object.__setattr__(+ self, "modes", {str(key): str(value) for key, value in self.modes.items()}+ )+ object.__setattr__(self, "seed_fields", dict(self.seed_fields))+ unknown = set(self.values) - PATCHABLE_INDEX_FIELDS+ if unknown:+ raise ValueError(f"field patch contains forbidden fields: {sorted(unknown)}")+ unknown_modes = set(self.modes) - set(self.values)+ if unknown_modes:+ raise ValueError(f"field modes without values: {sorted(unknown_modes)}")+ invalid_modes = {+ name: mode for name, mode in self.modes.items() if mode not in _FIELD_MODES+ }+ if invalid_modes:+ raise ValueError(f"invalid field modes: {invalid_modes}")+ unsupported_append = {+ name for name, mode in self.modes.items() if mode == "append" and name != "search_tags"+ }+ if unsupported_append:+ raise ValueError(f"append is unsupported for fields: {sorted(unsupported_append)}")++ def resolve(self, existing: Mapping[str, Any]) -> dict[str, Any]:+ """Resolve only patched fields against the latest stored values."""+ resolved: dict[str, Any] = {}+ for name, value in self.values.items():+ if name == "search_tags":+ incoming = normalize_search_tags(value, discard_invalid=True)+ resolved[name] = (+ merge_search_tags(existing.get(name), incoming)+ if self.modes.get(name, "replace") == "append"+ else incoming+ )+ else:+ resolved[name] = value+ return resolved++ def apply(self, existing: Mapping[str, Any]) -> dict[str, Any]:+ return {**dict(existing), **self.resolve(existing)}++ def with_seed(self, seed_fields: Mapping[str, Any]) -> "FieldPatch":+ return FieldPatch(self.values, self.modes, seed_fields)++ def to_dict(self) -> dict[str, Any]:+ return {+ "values": dict(self.values),+ "modes": dict(self.modes),+ "seed_fields": dict(self.seed_fields),+ }++ @classmethod+ def from_dict(cls, data: Mapping[str, Any]) -> "FieldPatch":+ return cls(+ values=dict(data.get("values") or {}),+ modes=dict(data.get("modes") or {}),+ seed_fields=dict(data.get("seed_fields") or {}),+ )+++__all__ = ["FieldPatch", "IndexAction", "PATCHABLE_INDEX_FIELDS"]diff --git a/openviking/storage/observers/queue_observer.py b/openviking/storage/observers/queue_observer.pyindex 8707a7773..946e6696f 100644--- a/openviking/storage/observers/queue_observer.py+++ b/openviking/storage/observers/queue_observer.py@@ -29,8 +29,8 @@ class QueueObserver(BaseObserver):async def get_status_table_async(self) -> str:statuses = await self._queue_manager.check_status()- dag_stats = self._get_semantic_dag_stats()- return self._format_status_as_table(statuses, dag_stats)+ tree_stats = self._get_semantic_tree_stats()+ return self._format_status_as_table(statuses, tree_stats)def get_status_table(self) -> str:return run_async(self.get_status_table_async())@@ -39,7 +39,7 @@ class QueueObserver(BaseObserver):return self.get_status_table()def _format_status_as_table(- self, statuses: Dict[str, QueueStatus], dag_stats: Optional[object]+ self, statuses: Dict[str, QueueStatus], tree_stats: Optional[object]) -> str:"""Format queue statuses as a table using tabulate.@@ -84,12 +84,12 @@ class QueueObserver(BaseObserver):data.append({"Queue": "Semantic-Nodes",- "Pending": getattr(dag_stats, "pending_nodes", 0) if dag_stats else 0,- "In Progress": getattr(dag_stats, "in_progress_nodes", 0) if dag_stats else 0,- "Processed": getattr(dag_stats, "done_nodes", 0) if dag_stats else 0,+ "Pending": getattr(tree_stats, "pending_nodes", 0) if tree_stats else 0,+ "In Progress": getattr(tree_stats, "in_progress_nodes", 0) if tree_stats else 0,+ "Processed": getattr(tree_stats, "done_nodes", 0) if tree_stats else 0,"Requeued": 0,"Errors": 0,- "Total": getattr(dag_stats, "total_nodes", 0) if dag_stats else 0,+ "Total": getattr(tree_stats, "total_nodes", 0) if tree_stats else 0,})@@ -109,13 +109,13 @@ class QueueObserver(BaseObserver):return tabulate(data, headers="keys", tablefmt="pretty")- def _get_semantic_dag_stats(self) -> Optional[object]:+ def _get_semantic_tree_stats(self) -> Optional[object]:semantic_queue = self._queue_manager._queues.get(self._queue_manager.SEMANTIC)if not semantic_queue:return Nonehandler = getattr(semantic_queue, "_dequeue_handler", None)- if handler and hasattr(handler, "get_dag_stats"):- return handler.get_dag_stats()+ if handler and hasattr(handler, "get_tree_stats"):+ return handler.get_tree_stats()return Nonedef is_healthy(self) -> bool:@@ -131,6 +131,5 @@ class QueueObserver(BaseObserver):@staticmethoddef _has_active_errors(statuses: Dict[str, QueueStatus]) -> bool:return any(- status.error_count > 0 and not status.is_complete- for status in statuses.values()+ status.error_count > 0 and not status.is_complete for status in statuses.values())diff --git a/openviking/storage/ovpack/index.py b/openviking/storage/ovpack/index.pyindex b7ed2cf16..499636d77 100644--- a/openviking/storage/ovpack/index.py+++ b/openviking/storage/ovpack/index.py@@ -28,6 +28,7 @@ PORTABLE_VECTOR_SCALAR_FIELDS = ["description","tags","abstract",+ "md5",]EXPORT_VECTOR_FIELDS = [*PORTABLE_VECTOR_SCALAR_FIELDS, "vector"]diff --git a/openviking/storage/queuefs/__init__.py b/openviking/storage/queuefs/__init__.pyindex 241a9b3a3..4a34054c1 100644--- a/openviking/storage/queuefs/__init__.py+++ b/openviking/storage/queuefs/__init__.py@@ -6,7 +6,7 @@ from .named_queue import NamedQueue, QueueError, QueueStatusfrom .process_result import ProcessOutcome, ProcessResultfrom .queue_manager import QueueManager, get_queue_manager, init_queue_managerfrom .queue_middleware import QueueMiddleware-from .semantic_dag import SemanticDagExecutor+from .semantic_executor import SemanticTreeExecutorfrom .semantic_msg import SemanticMsgfrom .semantic_processor import SemanticProcessorfrom .semantic_queue import SemanticQueue@@ -25,7 +25,7 @@ __all__ = ["EmbeddingQueue","EmbeddingMsg","SemanticQueue",- "SemanticDagExecutor",+ "SemanticTreeExecutor","SemanticMsg","SemanticProcessor","SessionCommitMsg",diff --git a/openviking/storage/queuefs/add_resource_msg.py b/openviking/storage/queuefs/add_resource_msg.pyindex 27fa3057e..4d46b462c 100644--- a/openviking/storage/queuefs/add_resource_msg.py+++ b/openviking/storage/queuefs/add_resource_msg.py@@ -27,6 +27,7 @@ class AddResourceMsg:telemetry_id: Optional[str] = Noneprepared: Optional[Dict[str, Any]] = Nonestaged_source: Optional[Dict[str, Any]] = None+ shared_source: Optional[Dict[str, Any]] = Nonejob_phase: AddResourcePhase | str | None = Nonelock_handoff: Optional[Dict[str, Any]] = Noneactor_peer_id: Optional[str] = None@@ -77,6 +78,7 @@ class AddResourceMsg:raise ValueError("source jobs cannot contain prepared post-process data")if self.prepared is not None and (self.staged_source is not None+ or self.shared_source is not Noneor self.understanding_response_id is not Noneor self.understanding_file_id is not None):@@ -85,6 +87,7 @@ class AddResourceMsg:payload is not Nonefor payload in (self.staged_source,+ self.shared_source,self.understanding_response_id,self.understanding_file_id,)@@ -112,8 +115,15 @@ class AddResourceMsg:from openviking.resource.staged_source import StagedSourcestaged_source = StagedSource.from_dict(data["staged_source"]).to_dict()- if prepared is not None and staged_source is not None:- raise ValueError("prepared and staged_source are mutually exclusive")+ shared_source = None+ if data.get("shared_source") is not None:+ from openviking.resource.shared_source import SharedSource++ shared_source = SharedSource.from_dict(data["shared_source"]).to_dict()+ if prepared is not None and (staged_source is not None or shared_source is not None):+ raise ValueError("prepared and source payloads are mutually exclusive")+ if staged_source is not None and shared_source is not None:+ raise ValueError("staged_source and shared_source are mutually exclusive")job_phase = data.get("job_phase") or (AddResourcePhase.POST_PROCESS.valueif prepared is not None@@ -127,12 +137,13 @@ class AddResourceMsg:lock_handoff_retry = 0if prepared is not None:args.clear()- if not task_id or (not path and not prepared and not staged_source) or not root_uri:+ has_source_payload = bool(prepared or staged_source or shared_source)+ if not task_id or (not path and not has_source_payload) or not root_uri:missing = []if not task_id:missing.append("task_id")- if not path and not prepared and not staged_source:- missing.append("path, prepared, or staged_source")+ if not path and not has_source_payload:+ missing.append("path, prepared, staged_source, or shared_source")if not root_uri:missing.append("root_uri")raise ValueError(f"Missing required fields: {missing}")@@ -184,6 +195,7 @@ class AddResourceMsg:),prepared=prepared,staged_source=staged_source,+ shared_source=shared_source,job_phase=job_phase,watch_interval=float(data.get("watch_interval", 0) or 0),is_active=(data.get("is_active") if isinstance(data.get("is_active"), bool) else None),diff --git a/openviking/storage/queuefs/add_resource_processor.py b/openviking/storage/queuefs/add_resource_processor.pyindex ba9ce7390..a9abebe8b 100644--- a/openviking/storage/queuefs/add_resource_processor.py+++ b/openviking/storage/queuefs/add_resource_processor.py@@ -24,6 +24,7 @@ from openviking.telemetry import ()from openviking.telemetry.request_wait_tracker import get_request_wait_trackerfrom openviking.telemetry.resource_summary import record_resource_queue_metrics+from openviking.utils.log_correlation import log_correlationfrom openviking_cli.exceptions import OpenVikingErrorfrom openviking_cli.session.user_id import UserIdentifierfrom openviking_cli.utils.logger import get_logger@@ -67,6 +68,15 @@ class AddResourceProcessor(DequeueHandlerBase):staged = StagedSource.from_dict(msg.staged_source)await self._viking_fs.delete_temp(staged.temp_uri, ctx=ctx)+ async def _cleanup_prepared_artifact(self, msg: AddResourceMsg, ctx: RequestContext) -> None:+ if not msg.prepared or not isinstance(msg.prepared.get("artifact_ref"), dict):+ return+ from openviking.parse.output import ParseArtifactRef, store_for_artifact_ref++ artifact_ref = ParseArtifactRef.from_dict(msg.prepared["artifact_ref"])+ store = store_for_artifact_ref(artifact_ref, viking_fs=self._viking_fs, ctx=ctx)+ await store.cleanup(artifact_ref)+async def _release_cancelled_resources(self,msg: AddResourceMsg,@@ -86,6 +96,8 @@ class AddResourceProcessor(DequeueHandlerBase):logger.warning("[AddResource] Failed to release cancelled lock handoff: %s", exc)with suppress(Exception):await self._cleanup_staged_source(msg, ctx)+ with suppress(Exception):+ await self._cleanup_prepared_artifact(msg, ctx)async def _record_watch_execution(self,@@ -154,6 +166,8 @@ class AddResourceProcessor(DequeueHandlerBase):else:with suppress(Exception):await self._cleanup_staged_source(msg, ctx)+ with suppress(Exception):+ await self._cleanup_prepared_artifact(msg, ctx)status = ("cancelled"if task.status in (TaskStatus.CANCELLING, TaskStatus.CANCELLED)@@ -186,6 +200,8 @@ class AddResourceProcessor(DequeueHandlerBase):unregister_telemetry(telemetry_id)with suppress(Exception):await self._cleanup_staged_source(msg, ctx)+ with suppress(Exception):+ await self._cleanup_prepared_artifact(msg, ctx)return ProcessResult.failed(f"Invalid lock_handoff: {exc}")telemetry = resolve_telemetry(telemetry_id) if telemetry_id else None@@ -198,8 +214,11 @@ class AddResourceProcessor(DequeueHandlerBase):register_telemetry(telemetry)request_wait_tracker = get_request_wait_tracker()request_wait_tracker.register_request(telemetry_id)+ current_stage = "queued"async def _set_stage(stage: str) -> None:+ nonlocal current_stage+ current_stage = stageawait tracker.update_stage(msg.task_id,stage,@@ -214,6 +233,17 @@ class AddResourceProcessor(DequeueHandlerBase):):terminal = Falsetry:+ queue_message_id = str(data.get("id") or "")+ logger.info(+ "[AddResourceStarted] %s root=%s phase=%s",+ log_correlation(+ task_id=msg.task_id,+ telemetry_id=telemetry_id,+ message_id=queue_message_id,+ ),+ msg.root_uri,+ msg.job_phase.value,+ )if replay_result is None:await tracker.start(msg.task_id,@@ -235,6 +265,17 @@ class AddResourceProcessor(DequeueHandlerBase):if result.get("status") == "error":errors = result.get("errors") or ["resource processing failed"]error = "; ".join(str(error) for error in errors)+ logger.error(+ "[AddResourceFailed] %s root=%s task_stage=%s error=%s",+ log_correlation(+ task_id=msg.task_id,+ telemetry_id=telemetry_id,+ message_id=queue_message_id,+ ),+ msg.root_uri,+ current_stage,+ error,+ )code = result.get("code")failure_result = {"code": code} if isinstance(code, str) and code else Noneawait tracker.fail(@@ -270,14 +311,30 @@ class AddResourceProcessor(DequeueHandlerBase):telemetry_id=telemetry_id,root_uri=result.get("root_uri"),)+ telemetry.set("resource.total.duration_ms", telemetry.elapsed_ms())# Extract token usage summary from telemetry and inject into result_snapshot = telemetry.finish()if _snapshot is not None:+ result["telemetry"] = _snapshot.to_dict(include_summary=True)_tokens = _snapshot.summary.get("tokens", {})if _tokens:result.setdefault("usage", {})result["usage"]["tokens"] = _tokens+ resource_summary = _snapshot.summary.get("resource", {})+ queue_summary = _snapshot.summary.get("queue", {})+ logger.info(+ "[AddResourceCompleted] %s root=%s total_ms=%s semantic=%s embedding=%s",+ log_correlation(+ task_id=msg.task_id,+ telemetry_id=telemetry_id,+ message_id=queue_message_id,+ ),+ result.get("root_uri"),+ (resource_summary.get("total") or {}).get("duration_ms"),+ queue_summary.get("semantic", {}),+ queue_summary.get("embedding", {}),+ )await self._resource_service._link_resource_reason_memory(result=result,@@ -297,10 +354,30 @@ class AddResourceProcessor(DequeueHandlerBase):terminal = Truereturn ProcessResult.success()except asyncio.CancelledError:+ logger.warning(+ "[AddResourceCancelled] %s root=%s",+ log_correlation(+ task_id=msg.task_id,+ telemetry_id=telemetry_id,+ message_id=queue_message_id,+ ),+ msg.root_uri,+ )await self._record_watch_execution(msg, "cancelled")terminal = Trueraiseexcept Exception as exc:+ logger.exception(+ "[AddResourceFailed] %s root=%s task_stage=%s error=%s",+ log_correlation(+ task_id=msg.task_id,+ telemetry_id=telemetry_id,+ message_id=queue_message_id,+ ),+ msg.root_uri,+ current_stage,+ exc,+ )await self._record_watch_execution(msg,"failed",diff --git a/openviking/storage/queuefs/embedding_msg.py b/openviking/storage/queuefs/embedding_msg.pyindex 4e529318e..af8aba7a7 100644--- a/openviking/storage/queuefs/embedding_msg.py+++ b/openviking/storage/queuefs/embedding_msg.py@@ -1,53 +1,416 @@# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.# SPDX-License-Identifier: AGPL-3.0+"""Durable, action-specific messages for the embedding queue."""++from __future__ import annotations+import json-from dataclasses import asdict, dataclass, field-from typing import Any, Dict, List, Union+from dataclasses import dataclass, field+from typing import Any, Dict, List, Mapping, Optional, Unionfrom uuid import uuid4+from openviking.storage.index_action import FieldPatch, IndexAction++EmbeddingInput = Union[str, List[Dict[str, Any]]]+++class IncompleteInitialRecordError(RuntimeError):+ """A missing record cannot be recreated from the queued seed."""++ def __init__(self, record_id: str, missing_fields: List[str]):+ self.record_id = record_id+ self.missing_fields = list(missing_fields)+ super().__init__(+ "update_fields could not recreate missing vector record: "+ f"record_id={record_id} missing_fields={self.missing_fields}"+ )+++def missing_initial_record_fields(record: Mapping[str, Any]) -> List[str]:+ """Return fields required to create a usable vector record."""+ missing = [name for name in ("id", "uri", "account_id", "level") if record.get(name) is None]+ if not record.get("vector") and not record.get("sparse_vector"):+ missing.append("vector")+ return missing+@dataclass-class EmbeddingMsg:- message: Union[str, List[Dict[str, Any]]]+class EmbedPayload:+ message: EmbeddingInput+ context_data: Dict[str, Any]+ field_patch: FieldPatch | None = None++ def to_dict(self) -> dict[str, Any]:+ data = {"type": "embed", "message": self.message, "context_data": self.context_data}+ if self.field_patch is not None:+ data["field_patch"] = self.field_patch.to_dict()+ return data+++@dataclass+class UpdateFieldsPayload:+ record_id: str+ context_data: Dict[str, Any]+ field_patch: FieldPatch++ def to_dict(self) -> dict[str, Any]:+ return {+ "type": "update_fields",+ "record_id": self.record_id,+ "context_data": self.context_data,+ "field_patch": self.field_patch.to_dict(),+ }+++@dataclass+class DeletePayload:+ record_ids: List[str]+ context_data: Dict[str, Any]++ def to_dict(self) -> dict[str, Any]:+ return {+ "type": "delete",+ "record_ids": list(self.record_ids),+ "context_data": self.context_data,+ }+++@dataclass+class NoopPayload:context_data: Dict[str, Any]++ def to_dict(self) -> dict[str, Any]:+ return {"type": "noop", "context_data": self.context_data}+++EmbeddingPayload = EmbedPayload | UpdateFieldsPayload | DeletePayload | NoopPayload+_PAYLOAD_FIELDS = {+ "embed": frozenset({"type", "message", "context_data", "field_patch"}),+ "update_fields": frozenset({"type", "record_id", "context_data", "field_patch"}),+ "delete": frozenset({"type", "record_ids", "context_data"}),+ "noop": frozenset({"type", "context_data"}),+}+++@dataclass(init=False)+class EmbeddingMsg:+ """Stable queue envelope whose payload is determined by ``action``."""++ action: IndexAction+ payload: EmbeddingPayloadid: str = field(default_factory=lambda: str(uuid4()))telemetry_id: str = ""+ queue_enqueued_at: float = 0.0def __init__(self,- message: Union[str, List[Dict[str, Any]]],- context_data: Dict[str, Any],+ message: Optional[EmbeddingInput] = None,+ context_data: Optional[Dict[str, Any]] = None,telemetry_id: str = "",- ):+ action: IndexAction | str = IndexAction.UPSERT,+ record_ids: Optional[List[str]] = None,+ update_fields: Optional[Dict[str, Any]] = None,+ field_modes: Optional[Dict[str, str]] = None,+ initial_fields: Optional[Dict[str, Any]] = None,+ queue_enqueued_at: float = 0.0,+ *,+ payload: EmbeddingPayload | None = None,+ ) -> None:self.id = str(uuid4())- self.message = message- self.context_data = context_dataself.telemetry_id = telemetry_id+ self.queue_enqueued_at = max(float(queue_enqueued_at or 0.0), 0.0)+ self.action = IndexAction(action)+ if payload is not None:+ if any(+ value is not None+ for value in (+ message,+ context_data,+ record_ids,+ update_fields,+ field_modes,+ initial_fields,+ )+ ):+ raise ValueError("payload cannot be combined with legacy embedding fields")+ self.payload = payload+ else:+ self.payload = self._legacy_payload(+ message=message,+ context_data=dict(context_data or {}),+ record_ids=list(record_ids or []),+ update_fields=dict(update_fields or {}),+ field_modes=dict(field_modes or {}),+ initial_fields=dict(initial_fields or {}),+ )+ self._validate_payload()++ def _legacy_payload(+ self,+ *,+ message: Optional[EmbeddingInput],+ context_data: Dict[str, Any],+ record_ids: List[str],+ update_fields: Dict[str, Any],+ field_modes: Dict[str, str],+ initial_fields: Dict[str, Any],+ ) -> EmbeddingPayload:+ patch = (+ FieldPatch(update_fields, field_modes, initial_fields)+ if update_fields or field_modes or initial_fields+ else None+ )+ if self.action is IndexAction.NONE:+ if message is not None or record_ids or patch is not None:+ raise ValueError("none action cannot carry work")+ return NoopPayload(context_data)+ if self.action is IndexAction.DELETE:+ if message is not None or patch is not None:+ raise ValueError("delete does not accept embedding or field-patch data")+ return DeletePayload(record_ids, context_data)+ if self.action is IndexAction.UPDATE_FIELDS:+ if message is not None or len(record_ids) != 1 or patch is None:+ raise ValueError("update_fields requires one record id and a field patch")+ return UpdateFieldsPayload(record_ids[0], context_data, patch)+ if not isinstance(message, (str, list)):+ raise ValueError(f"{self.action.value} requires an embedding message")+ if self.action is IndexAction.UPSERT:+ if field_modes:+ raise ValueError("upsert carries resolved fields and cannot use field modes")+ # Historical flat UPSERT messages allowed these fields but never+ # consumed them. Preserve queue-read compatibility while emitting+ # only the strict action-specific payload for new messages.+ return EmbedPayload(message, context_data)+ return EmbedPayload(message, context_data, patch)++ def _validate_payload(self) -> None:+ expected = {+ IndexAction.NONE: NoopPayload,+ IndexAction.UPSERT: EmbedPayload,+ IndexAction.MERGE: EmbedPayload,+ IndexAction.UPDATE_FIELDS: UpdateFieldsPayload,+ IndexAction.DELETE: DeletePayload,+ }[self.action]+ if not isinstance(self.payload, expected):+ raise ValueError(+ f"{self.action.value} requires {expected.__name__}, got {type(self.payload).__name__}"+ )+ if isinstance(self.payload, EmbedPayload) and not isinstance(+ self.payload.message, (str, list)+ ):+ raise ValueError(f"{self.action.value} requires an embedding message")+ if isinstance(self.payload, DeletePayload) and not self.payload.record_ids:+ raise ValueError("delete requires record_ids")+ if isinstance(self.payload, UpdateFieldsPayload):+ if not self.payload.record_id or not self.payload.field_patch.values:+ raise ValueError("update_fields requires one record id and a non-empty field patch")+ if (+ self.action is IndexAction.UPSERT+ and isinstance(self.payload, EmbedPayload)+ and self.payload.field_patch is not None+ ):+ raise ValueError("upsert does not accept a field patch")++ @classmethod+ def for_embed(+ cls,+ *,+ message: EmbeddingInput,+ context_data: Dict[str, Any],+ action: IndexAction | str = IndexAction.UPSERT,+ field_patch: FieldPatch | None = None,+ telemetry_id: str = "",+ ) -> "EmbeddingMsg":+ action = IndexAction(action)+ return cls(+ action=action,+ payload=EmbedPayload(message, context_data, field_patch),+ telemetry_id=telemetry_id,+ )++ @classmethod+ def for_update_fields(+ cls,+ *,+ record_id: str,+ context_data: Dict[str, Any],+ field_patch: FieldPatch | None = None,+ fields: Optional[Dict[str, Any]] = None,+ field_modes: Optional[Dict[str, str]] = None,+ initial_fields: Optional[Dict[str, Any]] = None,+ telemetry_id: str = "",+ ) -> "EmbeddingMsg":+ if field_patch is not None and any(+ value is not None for value in (fields, field_modes, initial_fields)+ ):+ raise ValueError(+ "field_patch cannot be combined with legacy fields, field_modes, or initial_fields"+ )+ resolved_patch = field_patch or FieldPatch(+ fields or {}, field_modes or {}, initial_fields or {}+ )+ return cls(+ action=IndexAction.UPDATE_FIELDS,+ payload=UpdateFieldsPayload(record_id, context_data, resolved_patch),+ telemetry_id=telemetry_id,+ )++ @classmethod+ def for_delete(+ cls,+ *,+ record_ids: List[str],+ context_data: Dict[str, Any],+ telemetry_id: str = "",+ ) -> "EmbeddingMsg":+ return cls(+ action=IndexAction.DELETE,+ payload=DeletePayload(record_ids, context_data),+ telemetry_id=telemetry_id,+ )++ @classmethod+ def noop(cls, *, context_data: Dict[str, Any], telemetry_id: str = "") -> "EmbeddingMsg":+ return cls(+ action=IndexAction.NONE,+ payload=NoopPayload(context_data),+ telemetry_id=telemetry_id,+ )++ @property+ def context_data(self) -> Dict[str, Any]:+ return self.payload.context_data++ @property+ def message(self) -> Optional[EmbeddingInput]:+ return self.payload.message if isinstance(self.payload, EmbedPayload) else None++ @property+ def record_ids(self) -> List[str]:+ if isinstance(self.payload, DeletePayload):+ return self.payload.record_ids+ if isinstance(self.payload, UpdateFieldsPayload):+ return [self.payload.record_id]+ return []++ @property+ def field_patch(self) -> FieldPatch | None:+ if isinstance(self.payload, (EmbedPayload, UpdateFieldsPayload)):+ return self.payload.field_patch+ return None++ @property+ def update_fields(self) -> Dict[str, Any]:+ return dict(self.field_patch.values) if self.field_patch is not None else {}++ @property+ def field_modes(self) -> Dict[str, str]:+ return dict(self.field_patch.modes) if self.field_patch is not None else {}++ @property+ def initial_fields(self) -> Dict[str, Any]:+ return dict(self.field_patch.seed_fields) if self.field_patch is not None else {}def to_dict(self) -> Dict[str, Any]:- """Convert embedding message to dictionary format."""- return asdict(self)+ return {+ "id": self.id,+ "telemetry_id": self.telemetry_id,+ "queue_enqueued_at": self.queue_enqueued_at,+ "action": self.action.value,+ "payload": self.payload.to_dict(),+ }def to_json(self) -> str:- """Convert embedding message to JSON string."""return json.dumps(self.to_dict())@classmethoddef from_dict(cls, data: Dict[str, Any]) -> "EmbeddingMsg":- """Create an embedding message object from dictionary."""- obj = EmbeddingMsg(- message=data["message"],- context_data=data["context_data"],- telemetry_id=data.get("telemetry_id", ""),- )- obj.id = data.get("id", obj.id)+ if not isinstance(data, dict):+ raise ValueError("embedding message must be an object")+ action = IndexAction(data.get("action", IndexAction.UPSERT.value))+ payload_data = data.get("payload")+ if isinstance(payload_data, dict):+ ambiguous = {+ name+ for name in (+ "message",+ "context_data",+ "record_ids",+ "update_fields",+ "field_modes",+ "initial_fields",+ )+ if name in data+ }+ if ambiguous:+ raise ValueError(+ f"payload cannot be combined with legacy embedding fields: {sorted(ambiguous)}"+ )+ payload_type = payload_data.get("type")+ allowed_fields = _PAYLOAD_FIELDS.get(payload_type)+ if allowed_fields is None:+ raise ValueError(f"unsupported embedding payload type: {payload_type}")+ unknown_fields = set(payload_data) - allowed_fields+ if unknown_fields:+ raise ValueError(+ f"{payload_type} payload contains unsupported fields: {sorted(unknown_fields)}"+ )+ context_data = dict(payload_data.get("context_data") or {})+ if payload_type == "embed":+ raw_patch = payload_data.get("field_patch")+ payload: EmbeddingPayload = EmbedPayload(+ payload_data.get("message"),+ context_data,+ FieldPatch.from_dict(raw_patch) if isinstance(raw_patch, dict) else None,+ )+ elif payload_type == "update_fields":+ payload = UpdateFieldsPayload(+ str(payload_data.get("record_id") or ""),+ context_data,+ FieldPatch.from_dict(dict(payload_data.get("field_patch") or {})),+ )+ elif payload_type == "delete":+ payload = DeletePayload(+ [str(value) for value in payload_data.get("record_ids") or []],+ context_data,+ )+ elif payload_type == "noop":+ payload = NoopPayload(context_data)+ obj = cls(+ action=action,+ payload=payload,+ telemetry_id=str(data.get("telemetry_id") or ""),+ queue_enqueued_at=data.get("queue_enqueued_at", 0.0),+ )+ else:+ obj = cls(+ message=data.get("message"),+ context_data=dict(data.get("context_data") or {}),+ telemetry_id=str(data.get("telemetry_id") or ""),+ action=action,+ record_ids=data.get("record_ids"),+ update_fields=data.get("update_fields"),+ field_modes=data.get("field_modes"),+ initial_fields=data.get("initial_fields"),+ queue_enqueued_at=data.get("queue_enqueued_at", 0.0),+ )+ obj.id = str(data.get("id") or obj.id)return obj@classmethoddef from_json(cls, json_str: str) -> "EmbeddingMsg":- """Safely create object from JSON string."""try:- data = json.loads(json_str)- return cls.from_dict(data)- except json.JSONDecodeError as e:- raise ValueError(f"Invalid JSON string: {e}")+ return cls.from_dict(json.loads(json_str))+ except json.JSONDecodeError as exc:+ raise ValueError(f"Invalid JSON string: {exc}") from exc+++__all__ = [+ "DeletePayload",+ "EmbedPayload",+ "EmbeddingMsg",+ "IncompleteInitialRecordError",+ "NoopPayload",+ "UpdateFieldsPayload",+ "missing_initial_record_fields",+]diff --git a/openviking/storage/queuefs/embedding_msg_converter.py b/openviking/storage/queuefs/embedding_msg_converter.pyindex 495085622..08378d7f1 100644--- a/openviking/storage/queuefs/embedding_msg_converter.py+++ b/openviking/storage/queuefs/embedding_msg_converter.py@@ -10,6 +10,7 @@ to EmbeddingMsg objects for asynchronous vector processing.from openviking.core.context import Context, ContextLevelfrom openviking.core.namespace import owner_fields_for_urifrom openviking.storage.acl import ACL_CREATOR_GRANT_FIELD, CreatorAclGrant+from openviking.storage.index_action import IndexActionfrom openviking.storage.queuefs.embedding_msg import EmbeddingMsgfrom openviking.telemetry import get_current_telemetryfrom openviking_cli.utils import get_logger@@ -22,10 +23,18 @@ class EmbeddingMsgConverter:@staticmethoddef from_context(- context: Context, creator_acl_grant: CreatorAclGrant | None = None+ context: Context,+ creator_acl_grant: CreatorAclGrant | None = None,+ action: IndexAction = IndexAction.MERGE,) -> EmbeddingMsg | None:"""Convert a Context object to EmbeddingMsg.++ Context-based producers normally carry only the fields they just+ generated. They therefore default to ``MERGE`` so an execution-time+ exact read retains stored scalar state such as tags and ACLs. Producers+ with a fully resolved record (notably an RNFV SemanticPlan) must pass+ ``UPSERT`` explicitly."""vectorization_text = context.get_vectorization_text()vectorization_images = context.get_vectorization_images()@@ -82,9 +91,10 @@ class EmbeddingMsgConverter:if creator_acl_grant is not None:context_data[ACL_CREATOR_GRANT_FIELD] = creator_acl_grant- embedding_msg = EmbeddingMsg(+ embedding_msg = EmbeddingMsg.for_embed(message=message,context_data=context_data,telemetry_id=get_current_telemetry().telemetry_id,+ action=action,)return embedding_msgdiff --git a/openviking/storage/queuefs/embedding_queue.py b/openviking/storage/queuefs/embedding_queue.pyindex 57d58bf54..9899b8d97 100644--- a/openviking/storage/queuefs/embedding_queue.py+++ b/openviking/storage/queuefs/embedding_queue.py@@ -1,5 +1,6 @@# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.# SPDX-License-Identifier: AGPL-3.0+import timefrom typing import Optionalfrom openviking_cli.utils.logger import get_logger@@ -21,6 +22,7 @@ class EmbeddingQueue(NamedQueue):if msg is None:logger.warning("Embedding message is None, skipping enqueuing")return ""+ msg.queue_enqueued_at = time.time()logger.debug("Enqueued embedding message: uri=%s", msg.context_data.get("uri", ""))return await super().enqueue(msg.to_dict())diff --git a/openviking/storage/queuefs/semantic_dag.py b/openviking/storage/queuefs/semantic_executor.pysimilarity index 67%rename from openviking/storage/queuefs/semantic_dag.pyrename to openviking/storage/queuefs/semantic_executor.pyindex a470caa83..9a360c639 100644--- a/openviking/storage/queuefs/semantic_dag.py+++ b/openviking/storage/queuefs/semantic_executor.py@@ -7,7 +7,7 @@ import reimport threadingfrom contextlib import nullcontextfrom dataclasses import dataclass, field-from typing import Any, ClassVar, Dict, List, Optional, Set+from typing import TYPE_CHECKING, Any, ClassVar, Dict, List, Optional, Setfrom weakref import WeakKeyDictionaryfrom openviking.core.namespace import classify_uri@@ -30,9 +30,12 @@ from openviking.storage.abstract_overview import (write_abstract_overview,)from openviking.storage.acl import CreatorAclGrant+from openviking.storage.context_update_plan import FileVectorSource, SemanticActionfrom openviking.storage.errors import LockAcquisitionError+from openviking.storage.index_action import FieldPatchfrom openviking.storage.viking_fs import LS_ALL_NODES, get_viking_fsfrom openviking.telemetry import bind_telemetry, get_current_telemetry+from openviking.utils.content_hash import content_md5from openviking.utils.ingest_options import IngestOptionsfrom openviking_cli.utils import VikingURIfrom openviking_cli.utils.config import get_openviking_config@@ -40,6 +43,9 @@ from openviking_cli.utils.logger import get_loggerlogger = get_logger(__name__)+if TYPE_CHECKING:+ from openviking.storage.context_update_plan import SemanticPlan, SemanticTreeEntry+# Session-internal files that should never be summarized by the semantic pipeline.# These are canonical archives (e.g. session transcripts) whose content provides# no additional retrieval value and would only waste tokens and add latency.@@ -70,7 +76,7 @@ class DirNode:@dataclass-class DagStats:+class SemanticTreeStats:total_nodes: int = 0pending_nodes: int = 0in_progress_nodes: int = 0@@ -80,7 +86,7 @@ class DagStats:@dataclass(frozen=True)-class DagWork:+class SemanticTreeWork:"""A scheduled unit of DAG work."""kind: str@@ -90,27 +96,27 @@ class DagWork:@dataclass(frozen=True)-class ScheduledDagWork:- executor: "SemanticDagExecutor"- work: DagWork+class ScheduledTreeWork:+ executor: "SemanticTreeExecutor"+ work: SemanticTreeWork-class SemanticNodeScheduler:+class SemanticTreeScheduler:"""Shared node executor for semantic DAG work in one event loop."""_idle_timeout = 0.05def __init__(self, max_workers: int):self._max_workers = max(1, max_workers)- self._queue: asyncio.Queue[ScheduledDagWork] = asyncio.Queue()+ self._queue: asyncio.Queue[ScheduledTreeWork] = asyncio.Queue()self._workers: Set[asyncio.Task] = set()def configure(self, max_workers: int) -> None:self._max_workers = max(1, max_workers)self._ensure_workers()- def submit(self, executor: "SemanticDagExecutor", work: DagWork) -> None:- self._queue.put_nowait(ScheduledDagWork(executor=executor, work=work))+ def submit(self, executor: "SemanticTreeExecutor", work: SemanticTreeWork) -> None:+ self._queue.put_nowait(ScheduledTreeWork(executor=executor, work=work))self._ensure_workers()def _ensure_workers(self) -> None:@@ -146,27 +152,27 @@ class SemanticNodeScheduler:self._queue.task_done()-_node_schedulers: "WeakKeyDictionary[asyncio.AbstractEventLoop, SemanticNodeScheduler]" = (+_node_schedulers: "WeakKeyDictionary[asyncio.AbstractEventLoop, SemanticTreeScheduler]" = (WeakKeyDictionary())-def get_semantic_node_scheduler(max_workers: int) -> SemanticNodeScheduler:+def get_semantic_tree_scheduler(max_workers: int) -> SemanticTreeScheduler:loop = asyncio.get_running_loop()scheduler = _node_schedulers.get(loop)if scheduler is None:- scheduler = SemanticNodeScheduler(max_workers=max_workers)+ scheduler = SemanticTreeScheduler(max_workers=max_workers)_node_schedulers[loop] = schedulerelse:scheduler.configure(max_workers)return scheduler-class SemanticDagExecutor:+class SemanticTreeExecutor:"""Execute semantic generation with DAG-style, event-driven lazy dispatch."""_active_lock: ClassVar[threading.Lock] = threading.Lock()- _active_executors: ClassVar[Set["SemanticDagExecutor"]] = set()+ _active_executors: ClassVar[Set["SemanticTreeExecutor"]] = set()def __init__(self,@@ -189,6 +195,10 @@ class SemanticDagExecutor:generation_trigger: str = "semantic_refresh",aggregate_directory: bool = True,copy_source_uri: str = "",+ file_md5s: Optional[Dict[str, str]] = None,+ artifact_files: Optional[List[str]] = None,+ file_abstracts: Optional[Dict[str, str]] = None,+ semantic_plan: Optional["SemanticPlan"] = None,):self._processor = processorself._context_type = context_type@@ -198,10 +208,25 @@ class SemanticDagExecutor:self._target_preexisting = target_preexistingself._recursive = recursiveself._lock = lock- self._is_code_repo = is_code_repo+ self._is_code_repo = bool(+ is_code_repo+ or (+ semantic_plan is not None+ and semantic_plan.file_vector_source is FileVectorSource.SUMMARY_WHEN_AVAILABLE+ )+ )+ self._changes_provided = changes is not Noneself._changes = changes or {}- self._skip_vectorization = skip_vectorization- self._ingest_options = IngestOptions.from_value(ingest_options)+ self._semantic_plan = semantic_plan+ self._semantic_resource_root = (+ semantic_plan.root_uri.rstrip("/") if semantic_plan is not None else None+ )+ self._skip_vectorization = (+ not semantic_plan.vectorize if semantic_plan is not None else skip_vectorization+ )+ self._ingest_options = IngestOptions.from_value(+ semantic_plan.ingest_options if semantic_plan is not None else ingest_options+ )self._coalesce_key = coalesce_keyself._coalesce_version = coalesce_versionself._source = dict(source) if source else None@@ -216,6 +241,19 @@ class SemanticDagExecutor:path for key in ("added", "modified", "deleted") for path in self._changes.get(key, [])}self._added_paths = {path.rstrip("/") for path in self._changes.get("added", [])}+ self._tree_changed_paths = {+ path.rstrip("/") for key in ("added", "deleted") for path in self._changes.get(key, [])+ }+ # Per-file md5 (target-URI keyed) supplied by local incremental apply, so+ # re-vectorization records the fresh fingerprint instead of leaving it stale.+ self._file_md5s = dict(file_md5s or {})+ self._artifact_files = [path.strip("/") for path in (artifact_files or []) if path]+ self._file_abstracts = dict(file_abstracts or {})+ self._plan_entries_by_uri: Dict[str, "SemanticTreeEntry"] = {}+ self._plan_children: Dict[str, tuple[List[str], List[str]]] = {}+ self._plan_active_dirs: Set[str] = set()+ if semantic_plan is not None:+ self._initialize_semantic_plan(semantic_plan)self._node_concurrency = max(1, max_concurrent_llm)self._llm_sem = asyncio.Semaphore(max_concurrent_llm)self._viking_fs = get_viking_fs()@@ -223,24 +261,103 @@ class SemanticDagExecutor:self._parent: Dict[str, Optional[str]] = {}self._root_uri: Optional[str] = Noneself._root_done: Optional[asyncio.Event] = None- self._scheduler: Optional[SemanticNodeScheduler] = None+ self._scheduler: Optional[SemanticTreeScheduler] = Noneself._active_scheduled_work = 0self._skill_node_tasks: Set[asyncio.Task] = set()self._active_work_idle = asyncio.Event()self._active_work_idle.set()self._closed = Falseself._failure: Optional[Exception] = None- self._stats = DagStats()+ self._stats = SemanticTreeStats()self._file_change_status: Dict[str, bool] = {}self._dir_change_status: Dict[str, bool] = {}self._overview_cache: Dict[str, Dict[str, str]] = {}self._overview_cache_lock = asyncio.Lock()self._root_write_result = AbstractOverviewWriteResult(wrote=False)+ self._scheduled_vector_record_ids: set[str] = set()++ @staticmethod+ def _entry_record_abstract(entry: "SemanticTreeEntry", level: int) -> str:+ slot = entry.slot(level)+ return slot.abstract if slot is not None else ""++ def _initialize_semantic_plan(self, plan: "SemanticPlan") -> None:+ from openviking.storage.context_update_plan import SemanticAction++ root = plan.root_uri.rstrip("/")+ root_entry = next(+ (entry for entry in plan.tree.entries if entry.relative_path == ""),+ None,+ )+ self._target_preexisting = not (+ root_entry is not None and root_entry.content_state.value in {"added", "restore"}+ )+ current_entries = list(plan.tree.entries)+ children: Dict[str, tuple[List[str], List[str]]] = {}+ for entry in current_entries:+ uri = root if not entry.relative_path else f"{root}/{entry.relative_path}"+ self._plan_entries_by_uri[uri] = entry+ if not entry.relative_path:+ continue+ parent_rel = entry.relative_path.rsplit("/", 1)[0] if "/" in entry.relative_path else ""+ parent_uri = root if not parent_rel else f"{root}/{parent_rel}"+ dirs, files = children.setdefault(parent_uri, ([], []))+ (dirs if entry.kind == "directory" else files).append(uri)++ changed_uris: List[str] = []+ for entry in plan.tree.entries:+ if entry.semantic_action == SemanticAction.REUSE:+ continue+ uri = root if not entry.relative_path else f"{root}/{entry.relative_path}"+ changed_uris.append(uri)+ if entry.kind == "directory" and entry.semantic_action == SemanticAction.AGGREGATE:+ self._plan_active_dirs.add(uri)+ if entry.relative_path:+ self._plan_active_dirs.add(uri.rsplit("/", 1)[0])+ for active_uri in list(self._plan_active_dirs):+ current = active_uri+ while current != root and current.startswith(root + "/"):+ parent = current.rsplit("/", 1)[0]+ if parent not in self._plan_entries_by_uri:+ break+ self._plan_active_dirs.add(parent)+ current = parent++ self._plan_children = {+ uri: (sorted(dirs), sorted(files)) for uri, (dirs, files) in children.items()+ }+ self._incremental_update = True+ self._target_uri = root+ self._recursive = True+ self._changes_provided = True+ self._changed_paths = set(changed_uris)+ self._added_paths = {+ uri+ for uri, entry in self._plan_entries_by_uri.items()+ if entry.content_state.value in {"added", "restore", "replace_kind"}+ }+ self._tree_changed_paths = {+ root if not entry.relative_path else f"{root}/{entry.relative_path}"+ for entry in plan.tree.entries+ if entry.membership_changed+ }+ self._file_md5s = {+ uri: str(entry.md5)+ for uri, entry in self._plan_entries_by_uri.items()+ if entry.kind == "file" and entry.md5+ }+ self._file_abstracts = {+ uri: abstract+ for uri, entry in self._plan_entries_by_uri.items()+ if entry.kind == "file" and (abstract := self._entry_record_abstract(entry, 2))+ }def _creator_acl_grant(self, uri: str) -> CreatorAclGrant | None:normalized = uri.rstrip("/")- if self._generation_trigger == "resource_ingest" and self._target_preexisting is False:- root = self._root_uri.rstrip("/")+ if (+ self._generation_trigger == "resource_ingest" or self._semantic_plan is not None+ ) and self._target_preexisting is False:+ root = (self._semantic_resource_root or self._root_uri).rstrip("/")if normalized == root:return CreatorAclGrant.DIRECTif normalized.startswith(f"{root}/"):@@ -255,7 +372,21 @@ class SemanticDagExecutor:"""Run DAG execution starting from root_uri."""self._root_uri = root_uriself._root_done = asyncio.Event()- self._scheduler = get_semantic_node_scheduler(self._node_concurrency)+ self._scheduler = get_semantic_tree_scheduler(self._node_concurrency)+ if self._semantic_plan is not None:+ from openviking.storage.context_update_plan import (+ SemanticAction,+ SemanticTreeEntry,+ )++ entry = self._plan_entries_by_uri.get(root_uri.rstrip("/"))+ if (+ isinstance(entry, SemanticTreeEntry)+ and entry.semantic_action == SemanticAction.REUSE+ ):+ self._stats.total_nodes = 1+ self._stats.done_nodes = 1+ returnowner = processing_owner.get()if owner is not None and self._task_context is not None:@@ -288,11 +419,11 @@ class SemanticDagExecutor:finally:self._unregister_active()- def _schedule_work(self, work: DagWork) -> None:+ def _schedule_work(self, work: SemanticTreeWork) -> None:if self._closed:returnif self._scheduler is None:- self._scheduler = get_semantic_node_scheduler(self._node_concurrency)+ self._scheduler = get_semantic_tree_scheduler(self._node_concurrency)self._scheduler.submit(self, work)def _start_scheduled_work(self) -> None:@@ -309,14 +440,14 @@ class SemanticDagExecutor:returnself._stats.total_nodes += 1self._stats.pending_nodes += 1- self._schedule_work(DagWork(kind="dir", dir_uri=dir_uri, parent_uri=parent_uri))+ self._schedule_work(SemanticTreeWork(kind="dir", dir_uri=dir_uri, parent_uri=parent_uri))def _schedule_file(self, parent_uri: str, file_path: str) -> None:if self._closed:returnself._stats.total_nodes += 1self._stats.pending_nodes += 1- self._schedule_work(DagWork(kind="file", dir_uri=parent_uri, file_path=file_path))+ self._schedule_work(SemanticTreeWork(kind="file", dir_uri=parent_uri, file_path=file_path))def _mark_node_started(self) -> None:self._stats.pending_nodes = max(0, self._stats.pending_nodes - 1)@@ -354,8 +485,8 @@ class SemanticDagExecutor:self._active_executors.discard(self)@classmethod- def get_active_stats(cls) -> DagStats:- stats = DagStats()+ def get_active_stats(cls) -> SemanticTreeStats:+ stats = SemanticTreeStats()with cls._active_lock:executors = list(cls._active_executors)for executor in executors:@@ -366,7 +497,7 @@ class SemanticDagExecutor:stats.done_nodes += current.done_nodesreturn stats- async def _run_work(self, work: DagWork) -> None:+ async def _run_work(self, work: SemanticTreeWork) -> None:if self._context_type == "skill":# Shared scheduler workers must not be cancelled with one package.# Each Skill node has its own cancellable task and ownership context.@@ -382,7 +513,7 @@ class SemanticDagExecutor:else:await self._run_work_with_context(work)- async def _run_work_with_context(self, work: DagWork) -> None:+ async def _run_work_with_context(self, work: SemanticTreeWork) -> None:task_context = (bind_task_context(self._task_context.task_id,@@ -411,7 +542,7 @@ class SemanticDagExecutor:return await run_to_completion(lambda: operation)return await operation- async def _run_work_bound(self, work: DagWork) -> None:+ async def _run_work_bound(self, work: SemanticTreeWork) -> None:self._mark_node_started()if work.kind == "dir":@@ -484,6 +615,60 @@ class SemanticDagExecutor:logger.info("Skipping busy parent semantic refresh: %s", dir_uri)file_index = {path: idx for idx, path in enumerate(file_paths)}child_index = {path: idx for idx, path in enumerate(children_dirs)}+ if self._semantic_plan is not None:+ required_file_paths = {+ path+ for path in file_paths+ if self._plan_entries_by_uri[path].semantic_action is not SemanticAction.REUSE+ }+ required_children_dirs = {+ path for path in children_dirs if path in self._plan_active_dirs+ }+ file_summaries: List[Optional[Dict[str, str]]] = [None] * len(file_paths)+ children_abstracts: List[Optional[Dict[str, str]]] = [None] * len(children_dirs)+ for path in file_paths:+ if path in required_file_paths or path not in sampled_file_paths:+ continue+ abstract = self._entry_record_abstract(self._plan_entries_by_uri[path], 2)+ if not abstract:+ raise RuntimeError(f"Semantic plan lacks L2 abstract for {path}")+ file_summaries[file_index[path]] = {+ "name": path.rsplit("/", 1)[-1],+ "summary": abstract,+ }+ for path in children_dirs:+ if path in required_children_dirs or path not in sampled_children_dirs:+ continue+ abstract = self._entry_record_abstract(self._plan_entries_by_uri[path], 0)+ if not abstract:+ raise RuntimeError(f"Semantic plan lacks L0 abstract for {path}")+ children_abstracts[child_index[path]] = {+ "name": path.rsplit("/", 1)[-1],+ "abstract": abstract,+ }+ pending = len(required_file_paths) + len(required_children_dirs)+ node = DirNode(+ uri=dir_uri,+ children_dirs=children_dirs,+ file_paths=file_paths,+ file_index=file_index,+ child_index=child_index,+ file_summaries=file_summaries,+ children_abstracts=children_abstracts,+ pending=pending,+ pending_snapshot=pending_snapshot,+ sampled_children_dirs=sampled_children_dirs,+ sampled_file_paths=sampled_file_paths,+ dispatched=True,+ )+ self._nodes[dir_uri] = node+ for file_path in sorted(required_file_paths):+ self._schedule_file(dir_uri, file_path)+ for child_uri in sorted(required_children_dirs):+ self._schedule_dir(child_uri, dir_uri)+ if pending == 0:+ self._schedule_overview(dir_uri)+ return False# Recursive/initial work still maintains every file. Incremental# parent aggregation prepares only sampled inputs plus files that# changed and therefore need their own vector maintenance.@@ -538,7 +723,7 @@ class SemanticDagExecutor:except Exception as e:logger.error(f"Failed to dispatch directory {dir_uri}: {e}", exc_info=True)self._record_skill_failure(dir_uri, e)- if self._generation_trigger == "content_copy":+ if self._semantic_plan is not None or self._generation_trigger == "content_copy":raiseif parent_uri:await self._on_child_done(parent_uri, dir_uri, "")@@ -636,11 +821,15 @@ class SemanticDagExecutor:async def _list_dir(self, uri: str, from_hint: str) -> tuple[list[str], list[str]]:"""List directory entries and return (child_dirs, file_paths)."""+ if self._semantic_plan is not None:+ return self._plan_children.get(uri.rstrip("/"), ([], []))+ if self._artifact_files and self._root_uri:+ return self._list_artifact_dir(uri)try:entries = await self._viking_fs.ls(uri, node_limit=LS_ALL_NODES, ctx=self._ctx)except Exception as e:logger.warning(- f"[SemanticDagExecutor] Failed to list directory {uri}: {e} from {from_hint}"+ f"[SemanticTreeExecutor] Failed to list directory {uri}: {e} from {from_hint}")raise@@ -660,6 +849,34 @@ class SemanticDagExecutor:return sorted(children_dirs), sorted(file_paths)+ def _list_artifact_dir(self, uri: str) -> tuple[list[str], list[str]]:+ root = self._root_uri.rstrip("/")+ current = uri.rstrip("/")+ if current == root:+ prefix = ""+ elif current.startswith(f"{root}/"):+ prefix = current[len(root) + 1 :] + "/"+ else:+ return [], []++ child_dirs: set[str] = set()+ file_paths: list[str] = []+ for rel_path in self._artifact_files:+ if not rel_path.startswith(prefix):+ continue+ remainder = rel_path[len(prefix) :]+ if not remainder:+ continue+ head, separator, _ = remainder.partition("/")+ if not head or head.startswith(".") or head in _SKIP_FILENAMES:+ continue+ child_uri = VikingURI(current).join(head).uri+ if separator:+ child_dirs.add(child_uri)+ else:+ file_paths.append(child_uri)+ return sorted(child_dirs), sorted(file_paths)+def _get_target_file_path(self, current_uri: str) -> Optional[str]:if not self._incremental_update or not self._target_uri or not self._root_uri:logger.warning(@@ -676,17 +893,19 @@ class SemanticDagExecutor:return current_uridef _is_direct_incremental_update(self) -> bool:- return (+ return self._semantic_plan is not None or (self._incremental_update- and bool(self._changed_paths)+ and self._changes_providedand self._target_uri == self._root_uri)- def _path_has_direct_change(self, uri: str) -> bool:- if uri in self._changed_paths:- return True- prefix = uri.rstrip("/") + "/"- return any(path.startswith(prefix) for path in self._changed_paths)+ def _directory_entries_changed(self, dir_uri: str) -> bool:+ """Return whether this directory subtree gained or lost an entry."""+ normalized = dir_uri.rstrip("/")+ prefix = normalized + "/"+ return any(+ path == normalized or path.startswith(prefix) for path in self._tree_changed_paths+ )def _ingest_options_for_file(self, file_path: str) -> IngestOptions:if self._generation_trigger == "content_write" and file_path not in self._changed_paths:@@ -698,6 +917,18 @@ class SemanticDagExecutor:return IngestOptions()return self._ingest_options+ def _plan_scalar_override(self, uri: str, level: int) -> Optional[Dict[str, Any]]:+ entry = self._plan_entries_by_uri.get(uri.rstrip("/"))+ if entry is None:+ return None+ slot = entry.slot(level)+ return slot.scalar_override() if slot is not None else None++ def _plan_field_patch(self, uri: str, level: int) -> FieldPatch | None:+ entry = self._plan_entries_by_uri.get(uri.rstrip("/"))+ slot = entry.slot(level) if entry is not None else None+ return slot.field_patch if slot is not None else None+async def _check_file_content_changed(self, file_path: str) -> bool:if self._is_direct_incremental_update():return file_path in self._changed_paths@@ -729,6 +960,12 @@ class SemanticDagExecutor:target_path = self._get_target_file_path(file_path)if not target_path:return None+ vector_abstract = self._file_abstracts.get(target_path.rstrip("/"))+ if vector_abstract:+ return {+ "name": file_path.rsplit("/", 1)[-1],+ "summary": vector_abstract,+ }try:parent_uri = "/".join(target_path.rsplit("/", 1)[:-1])@@ -777,13 +1014,28 @@ class SemanticDagExecutor:self, dir_uri: str, current_files: List[str], current_dirs: List[str]) -> bool:if self._is_direct_incremental_update():- if self._path_has_direct_change(dir_uri):+ node = self._nodes.get(dir_uri)+ if node is not None and node.pending_snapshot > 0:+ return True+ if self._directory_entries_changed(dir_uri):return Truefor current_file in current_files:- if self._file_change_status.get(current_file, True):+ default_changed = (+ self._plan_entries_by_uri[current_file].semantic_action+ is not SemanticAction.REUSE+ if self._semantic_plan is not None+ else True+ )+ if self._file_change_status.get(current_file, default_changed):return Truefor current_dir in current_dirs:- if self._dir_change_status.get(current_dir, True):+ default_changed = (+ self._plan_entries_by_uri[current_dir].semantic_action+ is not SemanticAction.REUSE+ if self._semantic_plan is not None+ else True+ )+ if self._dir_change_status.get(current_dir, default_changed):return Truereturn False@@ -815,6 +1067,13 @@ class SemanticDagExecutor:async def _read_existing_overview_abstract(self, dir_uri: str) -> tuple[Optional[str], Optional[str]]:+ if self._semantic_plan is not None:+ entry = self._plan_entries_by_uri.get(dir_uri.rstrip("/"))+ if entry is None:+ return None, None+ overview = self._entry_record_abstract(entry, 1) or None+ abstract = self._entry_record_abstract(entry, 0) or None+ return overview, abstracttarget_path = self._get_target_file_path(dir_uri)if not target_path:return None, None@@ -830,6 +1089,8 @@ class SemanticDagExecutor:file_name = file_path.split("/")[-1]need_vectorize = True+ semantic_succeeded = True+ file_content = Nonetry:summary_dict = Noneif self._incremental_update:@@ -860,36 +1121,80 @@ class SemanticDagExecutor:else:self._file_change_status[file_path] = Trueif summary_dict is None:+ file_content = None+ if hasattr(self._viking_fs, "read_file_bytes"):+ file_content = await self._viking_fs.read_file_bytes(file_path, ctx=self._ctx)+ summary_kwargs: Dict[str, Any] = {+ "llm_sem": self._llm_sem,+ "ctx": self._ctx,+ }+ if file_content is not None:+ summary_kwargs["file_content"] = file_contentsummary_dict = await self._processor._generate_single_file_summary(- file_path, llm_sem=self._llm_sem, ctx=self._ctx+ file_path, **summary_kwargs)+ except (AbstractOverviewFormatError, FileNotFoundError, ValueError):+ # A generated sidecar that opted into OKF must never be treated as+ # an empty file summary; doing so would silently feed metadata or+ # corrupted YAML into a later regeneration.+ raiseexcept Exception as e:logger.warning(f"Failed to generate summary for {file_path}: {e}")self._record_skill_failure(file_path, e)summary_dict = {"name": file_name, "summary": ""}+ semantic_succeeded = Falsefinally:self._stats.done_nodes += 1self._stats.in_progress_nodes = max(0, self._stats.in_progress_nodes - 1)if self._closed:return+ from openviking.storage.context_update_plan import (+ IndexAction,+ SemanticTreeEntry,+ )++ entry = self._plan_entries_by_uri.get(file_path.rstrip("/"))+ slot = entry.slot(2) if isinstance(entry, SemanticTreeEntry) else None+ if isinstance(entry, SemanticTreeEntry):+ need_vectorize = bool(+ semantic_succeeded+ and slot is not None+ and slot.action in {IndexAction.UPSERT, IndexAction.MERGE}+ )if need_vectorize and not self._skip_vectorization:use_summary = self._is_code_repo and bool(summary_dict.get("summary"))try:- enqueued = await self._await_write(- self._processor._vectorize_single_file(- parent_uri=parent_uri,- context_type=self._context_type,- file_path=file_path,- summary_dict=summary_dict,- ctx=self._ctx,- use_summary=use_summary,- ingest_options=self._ingest_options_for_file(file_path),- creator_acl_grant=self._creator_acl_grant(file_path),- )+ vectorize_kwargs: Dict[str, Any] = {}+ if file_content is not None:+ vectorize_kwargs["file_content"] = file_content+ manifest_md5 = self._file_md5s.get(file_path.rstrip("/")) or None+ if file_content is not None and not manifest_md5:+ file_md5 = content_md5(file_content)+ else:+ file_md5 = manifest_md5+ if self._semantic_plan is not None:+ vectorize_kwargs["scalar_override"] = self._plan_scalar_override(file_path, 2)+ vectorize_kwargs["field_patch"] = self._plan_field_patch(file_path, 2)+ vectorize_kwargs["action"] = slot.action.value if slot is not None else "upsert"+ enqueued = await self._processor._vectorize_single_file(+ parent_uri=parent_uri,+ context_type=self._context_type,+ file_path=file_path,+ summary_dict=summary_dict,+ ctx=self._ctx,+ use_summary=use_summary,+ ingest_options=(+ IngestOptions()+ if self._semantic_plan is not None+ else self._ingest_options_for_file(file_path)+ ),+ creator_acl_grant=self._creator_acl_grant(file_path),+ file_md5=file_md5,+ **vectorize_kwargs,)- if enqueued:- self._stats.indexed_records += 1+ if enqueued and slot is not None:+ self._scheduled_vector_record_ids.add(slot.record_id)except Exception as e:logger.error("Failed to schedule vectorization for %s: %s",@@ -947,7 +1252,7 @@ class SemanticDagExecutor:if not node or node.overview_scheduled:returnnode.overview_scheduled = True- self._schedule_work(DagWork(kind="overview", dir_uri=dir_uri))+ self._schedule_work(SemanticTreeWork(kind="overview", dir_uri=dir_uri))def _finalize_file_summaries(self, node: DirNode) -> List[Dict[str, str]]:summaries: List[Dict[str, str]] = []@@ -1044,7 +1349,8 @@ class SemanticDagExecutor:missing_summary_entries=missing_summary_entries,),}- if dir_uri == self._root_uri and self._source:+ source_root = self._semantic_resource_root or self._root_uri+ if dir_uri == source_root and self._source:metadata["source"] = self._sourcewrote = await self._await_write(write_abstract_overview(@@ -1057,7 +1363,7 @@ class SemanticDagExecutor:metadata=metadata,consume_pending=consume_pending,lock=self._lock,- log_prefix="[SemanticDag]",+ log_prefix="[SemanticTree]",))if not wrote.wrote:@@ -1085,6 +1391,7 @@ class SemanticDagExecutor:need_vectorize = Truechildren_changed = Trueshould_write = True+ write_result = AbstractOverviewWriteResult(wrote=False)abstract: Optional[str] = Noneoverview: Optional[str] = Nonetotal_entries = (@@ -1158,7 +1465,7 @@ class SemanticDagExecutor:if should_write:assert overview is not None and abstract is not Nonetry:- wrote = await self._write_directory_semantics(+ write_result = await self._write_directory_semantics(dir_uri,overview,abstract,@@ -1168,15 +1475,17 @@ class SemanticDagExecutor:missing_summary_entries=node.missing_summary_entries,)if dir_uri == self._root_uri:- self._root_write_result = wrote- if not wrote.wrote:+ self._root_write_result = write_result+ if not write_result.wrote:need_vectorize = Falseexcept AbstractOverviewFormatError:raiseexcept Exception as exc:- need_vectorize = False- logger.info(f"[SemanticDag] {dir_uri} write failed, skipping")self._record_skill_failure(dir_uri, exc)+ if self._semantic_plan is not None:+ self.fail(exc)+ return+ raiseexcept AbstractOverviewFormatError:raise@@ -1184,17 +1493,66 @@ class SemanticDagExecutor:logger.error(f"Failed to generate overview for {dir_uri}: {e}", exc_info=True)self._record_skill_failure(dir_uri, e)else:+ slots = {}+ if self._semantic_plan is not None:+ entry = self._plan_entries_by_uri.get(dir_uri.rstrip("/"))+ slots = {slot.level: slot for slot in entry.index_slots} if entry else {}+if need_vectorize and not self._skip_vectorization:assert overview is not None and abstract is not Nonetry:- await self._await_write(- self._processor._vectorize_directory(+ directory_vector_kwargs: Dict[str, Any] = {}+ include_abstract = True+ include_overview = True+ if self._semantic_plan is not None:+ from openviking.storage.context_update_plan import (+ IndexAction,+ )++ def should_emit(level: int) -> bool:+ slot = slots.get(level)+ if slot is None or slot.action not in {+ IndexAction.UPSERT,+ IndexAction.MERGE,+ }:+ return False+ old_body = slot.abstract+ new_body = abstract if level == 0 else overview+ return not old_body or old_body != new_body++ include_abstract = should_emit(0)+ include_overview = should_emit(1)+ directory_vector_kwargs = {+ "scalar_overrides": {+ level: values+ for level in (0, 1)+ if (values := self._plan_scalar_override(dir_uri, level))+ },+ "actions": {+ level: slot.action+ for level, slot in slots.items()+ if slot.action in {IndexAction.UPSERT, IndexAction.MERGE}+ },+ "field_patches": {+ level: slot.field_patch+ for level, slot in slots.items()+ if slot.field_patch is not None+ },+ "include_abstract": include_abstract,+ "include_overview": include_overview,+ }+ if include_abstract or include_overview:+ enqueued_levels = await self._processor._vectorize_directory(dir_uri,context_type=self._context_type,abstract=abstract,overview=overview,ctx=self._ctx,- ingest_options=self._ingest_options_for_directory(),+ ingest_options=(+ IngestOptions()+ if self._semantic_plan is not None+ else self._ingest_options_for_directory()+ ),creator_acl_grant=self._creator_acl_grant(dir_uri),**({"skill_source_path": (self._source or {}).get("path", "")}@@ -1202,9 +1560,10 @@ class SemanticDagExecutor:and classify_uri(dir_uri).is_skill_rootelse {}),+ **directory_vector_kwargs,)- )- self._stats.indexed_records += int(bool(abstract)) + int(bool(overview))+ else:+ enqueued_levels = set()except Exception as e:logger.error("Failed to schedule vectorization for %s: %s",@@ -1215,6 +1574,37 @@ class SemanticDagExecutor:if self._context_type != "skill":raiseself._record_skill_failure(dir_uri, e)+ else:+ enqueued_levels = set()++ for level, slot in sorted(slots.items()):+ if (+ level in enqueued_levels+ or not slot.fallback_to_patch+ or slot.field_patch is None+ ):+ continue+ enqueued = await self._processor._update_vector_fields(+ record_id=slot.record_id,+ uri=dir_uri,+ level=level,+ field_patch=slot.field_patch.with_seed(+ {+ **dict(slot.existing_fields or {}),+ "uri": dir_uri,+ "level": level,+ "account_id": self._ctx.account_id,+ }+ ),+ ctx=self._ctx,+ )+ if enqueued:+ enqueued_levels.add(level)++ if self._semantic_plan is not None:+ self._scheduled_vector_record_ids.update(+ slot.record_id for slot in slots.values() if slot.level in enqueued_levels+ )finally:self._stats.done_nodes += 1self._stats.in_progress_nodes = max(0, self._stats.in_progress_nodes - 1)@@ -1231,8 +1621,8 @@ class SemanticDagExecutor:await self._on_child_done(parent_uri, dir_uri, abstract or "")self._release_dir_node(dir_uri)- def get_stats(self) -> DagStats:- return DagStats(+ def get_stats(self) -> SemanticTreeStats:+ return SemanticTreeStats(total_nodes=self._stats.total_nodes,pending_nodes=self._stats.pending_nodes,in_progress_nodes=self._stats.in_progress_nodes,@@ -1247,6 +1637,10 @@ class SemanticDagExecutor:return self._root_write_result+ @property+ def scheduled_vector_record_ids(self) -> frozenset[str]:+ return frozenset(self._scheduled_vector_record_ids)+if False: # pragma: no cover - for type checkers onlyfrom openviking.storage.queuefs.semantic_processor import SemanticProcessordiff --git a/openviking/storage/queuefs/semantic_msg.py b/openviking/storage/queuefs/semantic_msg.pyindex 5a60c8161..3e6f8dfb0 100644--- a/openviking/storage/queuefs/semantic_msg.py+++ b/openviking/storage/queuefs/semantic_msg.py@@ -8,6 +8,7 @@ from datetime import datetimefrom typing import Any, Dict, List, Optional, Sequencefrom uuid import uuid4+from openviking.storage.context_update_plan import SemanticPlanfrom openviking.utils.ingest_options import IngestOptions@@ -37,7 +38,7 @@ class SemanticMsg:enqueue them for processing (bottom-up order).When False, only the specified directory will be processed.use_hierarchical_aggregation: Route memory directories through the- shared directory DAG instead of the specialized flat memory+ shared semantic tree instead of the specialized flat memoryupdate path.propagate_to_parent: Whether a completed directory refresh may enqueuea freshness refresh for its parent.@@ -48,6 +49,7 @@ class SemanticMsg:context_type: str # resource, memory, skill, sessionstatus: str = "pending" # pending/processing/completedtimestamp: int = field(default_factory=lambda: int(datetime.now().timestamp()))+ queue_enqueued_at: float = 0.0recursive: bool = True # Whether to recursively process subdirectoriesaccount_id: str = "default"user_id: str = "default"@@ -73,6 +75,14 @@ class SemanticMsg:use_hierarchical_aggregation: bool = Falsepropagate_to_parent: bool = Truecopy_source_uri: str = ""+ # Per-file md5 of final stored bytes, keyed by target URI. Supplied by the+ # local incremental apply so the tree executor's re-vectorization writes a fresh+ # fingerprint; empty for all other flows.+ file_md5s: Dict[str, str] = field(default_factory=dict)+ artifact_ref: Optional[Dict[str, Any]] = None+ artifact_files: List[str] = field(default_factory=list)+ file_abstracts: Dict[str, str] = field(default_factory=dict)+ plan: Optional[SemanticPlan] = Nonedef __init__(self,@@ -100,9 +110,16 @@ class SemanticMsg:use_hierarchical_aggregation: bool = False,propagate_to_parent: bool = True,copy_source_uri: str = "",+ file_md5s: Optional[Dict[str, str]] = None,+ artifact_ref: Optional[Dict[str, Any]] = None,+ artifact_files: Optional[List[str]] = None,+ file_abstracts: Optional[Dict[str, str]] = None,+ plan: SemanticPlan | Dict[str, Any] | None = None,+ queue_enqueued_at: float = 0.0,):self.id = str(uuid4())self.timestamp = int(datetime.now().timestamp())+ self.queue_enqueued_at = max(float(queue_enqueued_at or 0.0), 0.0)self.uri = uriself.context_type = context_typeself.recursive = recursive@@ -127,6 +144,17 @@ class SemanticMsg:self.use_hierarchical_aggregation = bool(use_hierarchical_aggregation)self.propagate_to_parent = bool(propagate_to_parent)self.copy_source_uri = copy_source_uri+ self.file_md5s = dict(file_md5s or {})+ self.artifact_ref = dict(artifact_ref) if artifact_ref else None+ self.artifact_files = list(artifact_files or [])+ self.file_abstracts = dict(file_abstracts or {})+ self.plan = (+ plan+ if isinstance(plan, SemanticPlan)+ else SemanticPlan.from_dict(plan)+ if isinstance(plan, dict)+ else None+ )def to_dict(self) -> Dict[str, Any]:"""Convert object to dictionary."""@@ -186,6 +214,18 @@ class SemanticMsg:use_hierarchical_aggregation=data.get("use_hierarchical_aggregation", False),propagate_to_parent=data.get("propagate_to_parent", True),copy_source_uri=data.get("copy_source_uri", ""),+ file_md5s=(data.get("file_md5s") if isinstance(data.get("file_md5s"), dict) else None),+ artifact_ref=(+ data.get("artifact_ref") if isinstance(data.get("artifact_ref"), dict) else None+ ),+ artifact_files=(+ data.get("artifact_files") if isinstance(data.get("artifact_files"), list) else None+ ),+ file_abstracts=(+ data.get("file_abstracts") if isinstance(data.get("file_abstracts"), dict) else None+ ),+ plan=data.get("plan") if isinstance(data.get("plan"), dict) else None,+ queue_enqueued_at=data.get("queue_enqueued_at", 0.0),)if "id" in data and data["id"]:obj.id = data["id"]diff --git a/openviking/storage/queuefs/semantic_processor.py b/openviking/storage/queuefs/semantic_processor.pyindex b8b1d6cf0..1fd57bc66 100644--- a/openviking/storage/queuefs/semantic_processor.py+++ b/openviking/storage/queuefs/semantic_processor.py@@ -5,6 +5,7 @@import asyncioimport reimport threading+import timefrom contextlib import nullcontextfrom typing import Any, Awaitable, Callable, Dict, List, Optional, Set, Tuplefrom urllib.parse import quote, unquote, urlsplit@@ -48,9 +49,10 @@ from openviking.storage.abstract_overview import ()from openviking.storage.acl import CreatorAclGrantfrom openviking.storage.errors import LockAcquisitionError+from openviking.storage.index_action import FieldPatchfrom openviking.storage.queuefs.named_queue import DequeueHandlerBasefrom openviking.storage.queuefs.process_result import ProcessResult-from openviking.storage.queuefs.semantic_dag import DagStats, SemanticDagExecutor+from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor, SemanticTreeStatsfrom openviking.storage.queuefs.semantic_lock import SemanticLockScopefrom openviking.storage.queuefs.semantic_msg import SemanticMsg, build_semantic_coalesce_keyfrom openviking.storage.queuefs.semantic_ops.freshness_policy import FreshnessAction@@ -96,9 +98,25 @@ class SemanticProcessor(DequeueHandlerBase):"""_stats_lock = threading.Lock()- _dag_stats_by_telemetry_id: Dict[str, DagStats] = {}- _dag_stats_by_uri: Dict[str, DagStats] = {}- _dag_stats_order: List[Tuple[str, str]] = []++ @staticmethod+ async def _cleanup_local_artifact(msg: SemanticMsg) -> None:+ try:+ if not msg.artifact_ref:+ return+ from openviking.parse.output import ParseArtifactRef, store_for_artifact_ref++ artifact_ref = ParseArtifactRef.from_dict(msg.artifact_ref)+ if artifact_ref.backend != "local":+ return+ store = store_for_artifact_ref(artifact_ref)+ await store.cleanup(artifact_ref)+ except Exception as exc:+ logger.warning("Failed to clean local parse artifact: %s", exc)++ _tree_stats_by_telemetry_id: Dict[str, SemanticTreeStats] = {}+ _tree_stats_by_uri: Dict[str, SemanticTreeStats] = {}+ _tree_stats_order: List[Tuple[str, str]] = []_request_stats_by_telemetry_id: Dict[str, RequestQueueStats] = {}_request_stats_order: List[str] = []_max_cached_stats = 256@@ -121,32 +139,32 @@ class SemanticProcessor(DequeueHandlerBase):self._embedding_worker_stopped = embedding_worker_stopped@classmethod- def _cache_dag_stats(cls, telemetry_id: str, uri: str, stats: DagStats) -> None:+ def _cache_tree_stats(cls, telemetry_id: str, uri: str, stats: SemanticTreeStats) -> None:with cls._stats_lock:if telemetry_id:- cls._dag_stats_by_telemetry_id[telemetry_id] = stats- cls._dag_stats_by_uri[uri] = stats- cls._dag_stats_order.append((telemetry_id, uri))- if len(cls._dag_stats_order) > cls._max_cached_stats:- old_telemetry_id, old_uri = cls._dag_stats_order.pop(0)+ cls._tree_stats_by_telemetry_id[telemetry_id] = stats+ cls._tree_stats_by_uri[uri] = stats+ cls._tree_stats_order.append((telemetry_id, uri))+ if len(cls._tree_stats_order) > cls._max_cached_stats:+ old_telemetry_id, old_uri = cls._tree_stats_order.pop(0)if old_telemetry_id:- cls._dag_stats_by_telemetry_id.pop(old_telemetry_id, None)- cls._dag_stats_by_uri.pop(old_uri, None)+ cls._tree_stats_by_telemetry_id.pop(old_telemetry_id, None)+ cls._tree_stats_by_uri.pop(old_uri, None)@classmethod- def consume_dag_stats(+ def consume_tree_stats(cls,telemetry_id: str = "",uri: Optional[str] = None,- ) -> Optional[DagStats]:+ ) -> Optional[SemanticTreeStats]:with cls._stats_lock:- if telemetry_id and telemetry_id in cls._dag_stats_by_telemetry_id:- stats = cls._dag_stats_by_telemetry_id.pop(telemetry_id, None)+ if telemetry_id and telemetry_id in cls._tree_stats_by_telemetry_id:+ stats = cls._tree_stats_by_telemetry_id.pop(telemetry_id, None)if uri:- cls._dag_stats_by_uri.pop(uri, None)+ cls._tree_stats_by_uri.pop(uri, None)return stats- if uri and uri in cls._dag_stats_by_uri:- return cls._dag_stats_by_uri.pop(uri, None)+ if uri and uri in cls._tree_stats_by_uri:+ return cls._tree_stats_by_uri.pop(uri, None)return None@classmethod@@ -285,6 +303,7 @@ class SemanticProcessor(DequeueHandlerBase):logger.error(f"Failed to re-enqueue semantic message: {requeue_err}")self._merge_request_stats(msg.telemetry_id, error_count=1)get_request_wait_tracker().mark_semantic_failed(msg.telemetry_id, msg.id, str(error))+ await self._cleanup_local_artifact(msg)return ProcessResult.failed(str(error))return ProcessResult.requeued()@@ -392,6 +411,9 @@ class SemanticProcessor(DequeueHandlerBase):msg: Optional[SemanticMsg] = Nonecollector = Nonework: Optional[SemanticMessageWork] = None+ execute_started_at: float | None = None+ queue_wait_ms = 0.0+ execute_status = "ok"try:import json@@ -405,10 +427,14 @@ class SemanticProcessor(DequeueHandlerBase):msg = SemanticMsg.from_dict(data)work = self._message_work(msg, lock)work.start()+ execute_started_at = time.perf_counter()+ if msg.queue_enqueued_at > 0:+ queue_wait_ms = max((time.time() - msg.queue_enqueued_at) * 1000.0, 0.0)if VikingURI(msg.uri).parent is None:logger.warning("Skipping semantic generation for root URI: %s", msg.uri)if msg.telemetry_id and msg.id:get_request_wait_tracker().mark_semantic_done(msg.telemetry_id, msg.id)+ await self._cleanup_local_artifact(msg)return ProcessResult.success()if is_semantic_msg_stale(msg):live_file_changes = {@@ -438,6 +464,7 @@ class SemanticProcessor(DequeueHandlerBase):await work.skip()if msg.telemetry_id and msg.id:get_request_wait_tracker().mark_semantic_done(msg.telemetry_id, msg.id)+ await self._cleanup_local_artifact(msg)return ProcessResult.success()# Circuit breaker: if API is known-broken, re-enqueue and waittry:@@ -468,7 +495,7 @@ class SemanticProcessor(DequeueHandlerBase):f"Processing semantic generation for: {msg.uri} (recursive={msg.recursive})")- logger.info(f"Processing semantic generation for: {msg})")+ logger.debug("Processing semantic message id=%s uri=%s", msg.id, msg.uri)# Resolving a deleted root can recreate it for lock metadata.# Settle queued ownership before acknowledging skipped work.@@ -487,10 +514,64 @@ class SemanticProcessor(DequeueHandlerBase):dag_stats = Noneprocessing_succeeded = Falsetry:+ if msg.plan is not None:+ if msg.uri.rstrip("/") != msg.plan.root_uri:+ raise ValueError("semantic message URI must match plan root_uri")+ if msg.context_type != msg.plan.context_type:+ raise ValueError(+ "semantic message context_type must match semantic plan"+ )+ for run_uri in msg.plan.execution_root_uris():+ executor = SemanticTreeExecutor(+ processor=self,+ context_type=msg.context_type,+ max_concurrent_llm=self.max_concurrent_llm,+ ctx=current_ctx,+ lock=semantic_lock.lock,+ source=msg.plan.source_metadata,+ semantic_plan=msg.plan,+ )+ await executor.run(run_uri)+ self._cache_tree_stats(+ msg.telemetry_id, run_uri, executor.get_stats()+ )+ if not executor.stale and msg.plan.propagation.enabled:+ write_result = getattr(+ executor,+ "root_write_result",+ AbstractOverviewWriteResult(+ wrote=True, abstract_body_changed=True+ ),+ )+ await self._enqueue_parent_refresh(+ msg,+ run_uri,+ l0_body_changed=write_result.abstract_body_changed,+ )+ from collections import Counter++ entries = msg.plan.tree.entries+ action_counts = Counter(+ entry.semantic_action.value for entry in entries+ )+ vector_slots = sum(+ slot.action.value in {"upsert", "merge"}+ for entry in entries+ for slot in entry.index_slots+ )+ logger.debug(+ "[SemanticPlanExecution] root=%s execution_roots=%d "+ "semantic_entries=%d actions=%s planned_vector_upserts=%d",+ msg.plan.root_uri,+ len(msg.plan.execution_root_uris()),+ len(entries),+ dict(action_counts),+ vector_slots,+ )# Regular memory writes keep their specialized update path.# Callers must explicitly opt into directory aggregation; the# trigger remains descriptive metadata, not an algorithm switch.- if msg.context_type == "memory" and not msg.use_hierarchical_aggregation:+ elif msg.context_type == "memory" and not msg.use_hierarchical_aggregation:await self._process_memory_directory(msg,ctx=current_ctx,@@ -531,19 +612,23 @@ class SemanticProcessor(DequeueHandlerBase):is_incremental = Truetarget_uri = msg.target_urirun_uri = msg.target_uri- elif target_exists and msg.changes and msg.uri == msg.target_uri:+ elif (+ target_exists+ and msg.changes is not None+ and msg.uri == msg.target_uri+ ):is_incremental = Truelogger.info(f"Using direct incremental semantic update for: {msg.uri}")- elif msg.changes:+ elif msg.changes is not None:is_incremental = Truetarget_uri = msg.urilogger.info(f"Using direct incremental semantic update for: {msg.uri}")- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=self,context_type=msg.context_type,max_concurrent_llm=self.max_concurrent_llm,@@ -563,10 +648,13 @@ class SemanticProcessor(DequeueHandlerBase):generation_trigger=msg.generation_trigger,aggregate_directory=msg.aggregate_directory,copy_source_uri=msg.copy_source_uri,+ file_md5s=msg.file_md5s,+ artifact_files=msg.artifact_files,+ file_abstracts=msg.file_abstracts,)await executor.run(run_uri)dag_stats = executor.get_stats()- self._cache_dag_stats(+ self._cache_tree_stats(msg.telemetry_id,run_uri,dag_stats,@@ -594,6 +682,7 @@ class SemanticProcessor(DequeueHandlerBase):self._merge_request_stats(msg.telemetry_id, processed=1)logger.info(f"Completed semantic generation for: {msg.uri}")self._circuit_breaker.record_success()+ await self._cleanup_local_artifact(msg)return ProcessResult.success()finally:reset_root_observability_context(root_context_token)@@ -604,6 +693,7 @@ class SemanticProcessor(DequeueHandlerBase):raiseexcept Exception as e:if isinstance(e, LockAcquisitionError):+ execute_status = "requeued"logger.warning("Lock error processing semantic message, re-enqueueing without ""tripping API circuit breaker: %s",@@ -620,6 +710,7 @@ class SemanticProcessor(DequeueHandlerBase):error_class = classify_api_error(e)if error_class == ERROR_CLASS_INPUT_TOO_LARGE:+ execute_status = "error"logger.error(f"Input too large processing semantic message, dropping: {e}",exc_info=True,@@ -629,8 +720,11 @@ class SemanticProcessor(DequeueHandlerBase):get_request_wait_tracker().mark_semantic_failed(msg.telemetry_id, msg.id, str(e))+ if msg is not None:+ await self._cleanup_local_artifact(msg)return ProcessResult.failed(str(e))elif error_class == ERROR_CLASS_PERMANENT:+ execute_status = "error"logger.critical(f"Permanent error processing semantic message, dropping: {e}",exc_info=True,@@ -643,9 +737,12 @@ class SemanticProcessor(DequeueHandlerBase):get_request_wait_tracker().mark_semantic_failed(msg.telemetry_id, msg.id, str(e))+ if msg is not None:+ await self._cleanup_local_artifact(msg)return ProcessResult.failed(str(e))else:# Transient or unknown — re-enqueue for retry+ execute_status = "requeued"logger.warning(f"Transient API error processing semantic message, re-enqueueing: {e}",exc_info=True,@@ -658,8 +755,30 @@ class SemanticProcessor(DequeueHandlerBase):work=work,)return ProcessResult.failed(str(e))-finally:+ if msg is not None and execute_started_at is not None:+ tracker = get_request_wait_tracker()+ record_timing = getattr(tracker, "record_semantic_timing", None)+ if callable(record_timing):+ record_timing(+ msg.telemetry_id,+ queue_wait_ms=queue_wait_ms,+ execute_ms=(time.perf_counter() - execute_started_at) * 1000.0,+ )+ from openviking.metrics.datasources.resource import ResourceIngestionEventDataSource++ ResourceIngestionEventDataSource.record_stage(+ stage="semantic_queue_wait",+ status=execute_status,+ duration_seconds=queue_wait_ms / 1000.0,+ account_id=msg.account_id,+ )+ ResourceIngestionEventDataSource.record_stage(+ stage="semantic_execute",+ status=execute_status,+ duration_seconds=(time.perf_counter() - execute_started_at),+ account_id=msg.account_id,+ )if work is not None:await work.close()@@ -676,6 +795,7 @@ class SemanticProcessor(DequeueHandlerBase):return ProcessResult.failed(str(exc))await self._message_work(msg).cancel_queued()+ await self._cleanup_local_artifact(msg)return ProcessResult.cancelled()async def _release_cancelled_semantic_lock(self, msg: SemanticMsg) -> None:@@ -718,8 +838,8 @@ class SemanticProcessor(DequeueHandlerBase):await run_to_completion(scope.close)raise- def get_dag_stats(self) -> Optional["DagStats"]:- return SemanticDagExecutor.get_active_stats()+ def get_tree_stats(self) -> Optional["SemanticTreeStats"]:+ return SemanticTreeExecutor.get_active_stats()async def _process_memory_directory(self,@@ -1078,13 +1198,18 @@ class SemanticProcessor(DequeueHandlerBase):file_name: str,llm_sem: asyncio.Semaphore,ctx: Optional[RequestContext] = None,+ file_content: Optional[bytes] = None,) -> Dict[str, Any]:"""Generate summary for a single text file (code, documentation, or other text)."""viking_fs = get_viking_fs()vlm = get_openviking_config().vlmactive_ctx = ctx or self._default_ctx- content = await viking_fs.read_file(file_path, ctx=active_ctx)+ content = (+ file_content+ if file_content is not None+ else await viking_fs.read_file(file_path, ctx=active_ctx)+ )if isinstance(content, bytes):from openviking.utils.embedding_utils import _decode_text_bytes@@ -1125,7 +1250,7 @@ class SemanticProcessor(DequeueHandlerBase):{"file_name": file_name, "content": content, "output_language": output_language},)async with llm_sem:- with bind_telemetry_stage("resource_summarize"):+ with bind_telemetry_stage("semantic_execute"):summary = await vlm.get_completion_async(prompt)return result(summary.strip())@@ -1147,7 +1272,7 @@ class SemanticProcessor(DequeueHandlerBase):)async with llm_sem:- with bind_telemetry_stage("resource_summarize"):+ with bind_telemetry_stage("semantic_execute"):summary = await vlm.get_completion_async(prompt)return result(summary.strip())@@ -1156,6 +1281,7 @@ class SemanticProcessor(DequeueHandlerBase):file_path: str,llm_sem: Optional[asyncio.Semaphore] = None,ctx: Optional[RequestContext] = None,+ file_content: Optional[bytes] = None,) -> Dict[str, Any]:"""Generate summary for a single file.@@ -1170,11 +1296,15 @@ class SemanticProcessor(DequeueHandlerBase):media_type = get_media_type(file_name, None)if file_name.lower().endswith(".ts"):try:- prefix = await get_viking_fs().read(- file_path,- offset=0,- size=MPEG_TS_PROBE_BYTES,- ctx=ctx,+ prefix = (+ file_content[:MPEG_TS_PROBE_BYTES]+ if file_content is not None+ else await get_viking_fs().read(+ file_path,+ offset=0,+ size=MPEG_TS_PROBE_BYTES,+ ctx=ctx,+ ))except Exception:prefix = None@@ -1186,7 +1316,9 @@ class SemanticProcessor(DequeueHandlerBase):elif media_type == "video":return await generate_video_summary(file_path, file_name, llm_sem, ctx=ctx)else:- return await self._generate_text_summary(file_path, file_name, llm_sem, ctx=ctx)+ return await self._generate_text_summary(+ file_path, file_name, llm_sem, ctx=ctx, file_content=file_content+ )def _child_summary_line(self,@@ -1539,7 +1671,7 @@ class SemanticProcessor(DequeueHandlerBase):},)- with bind_telemetry_stage("resource_summarize"):+ with bind_telemetry_stage("semantic_execute"):overview = await vlm.get_completion_async(prompt)overview = self._replace_link_references(overview, link_map)@@ -1618,7 +1750,7 @@ class SemanticProcessor(DequeueHandlerBase):async def _run_batch(batch_idx: int, prompt: str, batch_link_map: Dict[str, str]) -> None:try:async with llm_sem:- with bind_telemetry_stage("resource_summarize"):+ with bind_telemetry_stage("semantic_execute"):partial = await vlm.get_completion_async(prompt)partial = self._replace_link_references(partial, batch_link_map)partial_overviews[batch_idx] = partial.strip()@@ -1654,7 +1786,7 @@ class SemanticProcessor(DequeueHandlerBase):"directory_coverage": directory_coverage,},)- with bind_telemetry_stage("resource_summarize"):+ with bind_telemetry_stage("semantic_execute"):overview = await vlm.get_completion_async(prompt)overview = self._replace_link_references(overview, link_map)return overview.strip()@@ -1721,7 +1853,12 @@ class SemanticProcessor(DequeueHandlerBase):ingest_options: IngestOptions | None = None,creator_acl_grant: CreatorAclGrant | None = None,skill_source_path: str = "",- ) -> None:+ scalar_overrides: Optional[Dict[int, Dict[str, Any]]] = None,+ actions: Optional[Dict[int, str]] = None,+ field_patches: Optional[Dict[int, FieldPatch]] = None,+ include_abstract: bool = True,+ include_overview: bool = True,+ ) -> set[int]:"""Create directory Context and enqueue to EmbeddingQueue."""from openviking.utils.embedding_utils import vectorize_directory_meta@@ -1738,7 +1875,7 @@ class SemanticProcessor(DequeueHandlerBase):for key in ("name", "description", "tags", "allowed_tools")}skill_meta["source_path"] = skill_source_path- await vectorize_directory_meta(+ return await vectorize_directory_meta(uri=uri,abstract=abstract,overview=overview,@@ -1748,6 +1885,11 @@ class SemanticProcessor(DequeueHandlerBase):creator_acl_grant=creator_acl_grant,content_is_body=context_type == "skill",**({"meta": skill_meta} if skill_meta is not None else {}),+ scalar_overrides=scalar_overrides,+ actions=actions,+ field_patches=field_patches,+ include_abstract=include_abstract,+ include_overview=include_overview,)async def _load_transfer_file_summaries(@@ -1765,6 +1907,39 @@ class SemanticProcessor(DequeueHandlerBase):active_ctx = ctx or self._default_ctxreturn await vector_store.get_l2_abstracts_by_uris(file_paths, ctx=active_ctx)+ async def _update_vector_fields(+ self,+ *,+ record_id: str,+ uri: str,+ level: int,+ field_patch: FieldPatch,+ ctx: RequestContext,+ ) -> bool:+ from openviking.storage.queuefs import get_queue_manager+ from openviking.storage.queuefs.embedding_msg import EmbeddingMsg+ from openviking.telemetry import get_current_telemetry+ from openviking.utils.embedding_utils import _enqueue_embedding_message++ embedding_msg = EmbeddingMsg.for_update_fields(+ record_id=record_id,+ field_patch=field_patch,+ context_data={+ "uri": uri,+ "level": level,+ "account_id": ctx.account_id,+ "owner_user_id": ctx.user.user_id,+ },+ telemetry_id=get_current_telemetry().telemetry_id,+ )+ queue_manager = get_queue_manager()+ embedding_queue = queue_manager.get_queue(queue_manager.EMBEDDING, allow_create=True)+ return await _enqueue_embedding_message(+ embedding_queue,+ embedding_msg,+ failure_message=f"Failed to enqueue vector scalar update for {uri}",+ )+async def _vectorize_single_file(self,parent_uri: str,@@ -1776,6 +1951,11 @@ class SemanticProcessor(DequeueHandlerBase):preserve_existing_created_at: bool = False,ingest_options: IngestOptions | None = None,creator_acl_grant: CreatorAclGrant | None = None,+ file_md5: Optional[str] = None,+ file_content: Optional[bytes] = None,+ scalar_override: Optional[Dict[str, Any]] = None,+ field_patch: FieldPatch | None = None,+ action: str = "merge",) -> bool:"""Vectorize a single file using its content or summary."""from openviking.utils.embedding_utils import vectorize_file@@ -1791,4 +1971,9 @@ class SemanticProcessor(DequeueHandlerBase):preserve_existing_created_at=preserve_existing_created_at,ingest_options=ingest_options,creator_acl_grant=creator_acl_grant,+ file_md5=file_md5,+ file_content=file_content,+ scalar_override=scalar_override,+ field_patch=field_patch,+ action=action,)diff --git a/openviking/storage/queuefs/semantic_queue.py b/openviking/storage/queuefs/semantic_queue.pyindex 48e35bbf5..e85db2280 100644--- a/openviking/storage/queuefs/semantic_queue.py+++ b/openviking/storage/queuefs/semantic_queue.py@@ -70,6 +70,8 @@ class SemanticQueue(NamedQueue):_SEMANTIC_COALESCE_VERSION[msg.coalesce_key] = versionmsg.coalesce_version = version+ msg.queue_enqueued_at = time.time()+return await super().enqueue(msg.to_dict())async def dequeue(self) -> Optional[SemanticMsg]:diff --git a/openviking/storage/queuefs/semantic_work.py b/openviking/storage/queuefs/semantic_work.pyindex db1970c23..553221af5 100644--- a/openviking/storage/queuefs/semantic_work.py+++ b/openviking/storage/queuefs/semantic_work.py@@ -10,7 +10,7 @@ from typing import TYPE_CHECKING, Any, Awaitable, Callable, TypeVarfrom openviking.server.identity import RequestContextfrom openviking.service.task_tracker_concurrency import run_to_completionfrom openviking.storage.queuefs.process_result import ProcessResult-from openviking.storage.queuefs.semantic_dag import DagStats+from openviking.storage.queuefs.semantic_executor import SemanticTreeStatsfrom openviking.storage.queuefs.semantic_lock import SemanticLockScopefrom openviking.storage.queuefs.semantic_msg import SemanticMsgfrom openviking.storage.viking_fs import get_viking_fs@@ -63,7 +63,7 @@ class SemanticMessageWork:assert self.scope is not Noneawait self.scope.close()- def failure_result(self, stats: DagStats | None) -> ProcessResult | None:+ def failure_result(self, stats: SemanticTreeStats | None) -> ProcessResult | None:return Noneasync def reenqueue(self) -> None:@@ -151,7 +151,7 @@ class SkillSemanticMessageWork(SemanticMessageWork):# On failure retain the lease until reenqueue hands it to the retry.await run_to_completion(finish)- def failure_result(self, stats: DagStats | None) -> ProcessResult | None:+ def failure_result(self, stats: SemanticTreeStats | None) -> ProcessResult | None:if stats is None or not stats.failures:return Nonefor failure in stats.failures:diff --git a/openviking/storage/resource_diff.py b/openviking/storage/resource_diff.pynew file mode 100644index 000000000..3045c38e9--- /dev/null+++ b/openviking/storage/resource_diff.py@@ -0,0 +1,644 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Read the R / N / F / V snapshots used by resource update planning.++This module gathers the typed R/N/F/V inputs used by the canonical resource+update plan.++- ``R`` normalized request intent — target, processing mode, and scalar fields+ such as tags that this request explicitly wants to mutate.++- ``N`` new-artifact manifest — the files a parser produced, read from the+ parse output store. Canonical add-resource prepares it once before planning:+ image references are rewritten to their final URI and their final-byte md5 is+ retained in the resulting inventory.+- ``F`` target file tree — one unbounded ``tree`` call. Any permission-denied+ subtree (or a truncated scan) marks the snapshot incomplete so the planner+ refuses deletions instead of treating unreadable files as removed.+- ``V`` target vectors — lightweight L0/L1/L2 records under the target URI.++The canonical ``add_resources`` path resolves these snapshots into a+``ResourceDiffResult`` before building a ``ContextUpdatePlan``.+"""++from __future__ import annotations++import asyncio+import json+import logging+from collections import Counter+from dataclasses import dataclass+from enum import Enum+from typing import Any, Dict, Mapping, Tuple++from openviking.concurrency import bounded_map+from openviking.storage.internal_names import STORAGE_INTERNAL_ENTRY_NAMES+from openviking.storage.resource_rnfv import (+ CONTROL_BASENAMES,+ FormalEntry,+ FormalTreeSnapshot,+ NewArtifactSnapshot,+ NewEntry,+ RequestIntent,+ RNFVSnapshot,+ VectorIndexSnapshot,+ VectorRecordSnapshot,+ canonical_vector_records_by_level,+)+from openviking.utils.log_correlation import log_correlation++logger = logging.getLogger(__name__)+++class ContentState(str, Enum):+ """Relationship between the new artifact (N) and formal tree (F).++ F is the source of truth for whether content currently exists. Vector+ records are deliberately not part of this axis; their presence is expressed+ independently by :class:`IndexState`.+ """++ # Neither N nor F contains this path.+ ABSENT = "absent"+ # N and F contain the same file bytes, or the same directory node.+ UNCHANGED = "unchanged"+ # N contains a path that has no formal file and no prior vector records.+ ADDED = "added"+ # N and F both contain a file, but their bytes differ.+ MODIFIED = "modified"+ # F contains a path omitted from N.+ DELETED = "deleted"+ # N contains a path absent from F while V still has records for that path.+ RESTORE = "restore"+ # N and F contain the path with different node kinds (file versus directory).+ REPLACE_KIND = "replace_kind"+++class IndexState(str, Enum):+ """Health of V for the node kind selected by N/F.++ A file is valid only with L2. A directory is valid only with both L0 and+ L1. This axis is independent from content change, so an unchanged file can+ still require index repair.+ """++ # No formal node and no vector record exist for the path.+ ABSENT = "absent"+ # All and only the levels expected for the current node kind are present.+ COMPLETE = "complete"+ # None of the levels expected for the current node kind are present.+ MISSING = "missing"+ # Some expected levels are present, but the complete expected set is not.+ PARTIAL = "partial"+ # Expected levels exist but are derived from content known to be out of date.+ STALE = "stale"+ # V belongs to content removed by the current update.+ OBSOLETE = "obsolete"+ # V has records while neither N nor F has a corresponding content node.+ ORPHAN = "orphan"+ # V contains a level that is invalid for the current file/directory kind.+ LEVEL_CONFLICT = "level_conflict"+++@dataclass(frozen=True)+class ResourceDiffEntry:+ """One path's independent content and index facts after RNFV resolution.++ ``content_state`` determines formal-tree mutation. ``index_state`` determines+ vector repair or cleanup. ``md5`` is the final new-file fingerprint when the+ path is a file; directories intentionally never carry an aggregate MD5.+ """++ relative_path: str+ content_state: ContentState+ index_state: IndexState+ old_kind: str | None = None+ new_kind: str | None = None+ md5: str | None = None+++@dataclass(frozen=True)+class ResourceDiffResult:+ entries: Mapping[str, ResourceDiffEntry]+++@dataclass(frozen=True)+class ArtifactInventory:+ """Prepared N snapshot plus paths relative to its parse artifact.++ ``entries`` are rooted at the final resource URI; ``artifact_paths`` point+ to the same files under the parse artifact. The latter keeps diff body+ fallbacks independent from the parser's optional document wrapper.+ """++ entries: Mapping[str, NewEntry]+ artifact_paths: Mapping[str, str]+ rewritten_paths: frozenset[str] = frozenset()+++def count_tree_entry_kinds(+ entries: Mapping[str, NewEntry | FormalEntry],+) -> tuple[int, int, bool]:+ """Return business file/dir counts and whether a logical root is present.++ Directory plans add ``relative_path=""`` as a traversal-only root. Keeping+ it separate prevents logs from presenting that synthetic node as a file or+ directory stored below the target. A flat resource uses the same empty path+ with ``is_dir=False`` and is therefore counted as one real file.+ """+ logical_root = bool((root := entries.get("")) is not None and root.is_dir)+ files = sum(not entry.is_dir for entry in entries.values())+ directories = sum(entry.is_dir for path, entry in entries.items() if path)+ return files, directories, logical_root+++def _index_state(+ records: tuple[VectorRecordSnapshot, ...],+ *,+ kind: str | None,+ content_state: ContentState,+ md5: str | None,+) -> IndexState:+ """Classify V with strict precedence.++ Content removed by this update makes its V record obsolete. A vector with+ neither new nor formal content is an orphan. For an existing node, invalid+ levels win over missing or stale checks; then missing expected levels, then+ content/fingerprint staleness, and finally complete.+ """+ if content_state is ContentState.DELETED:+ return IndexState.OBSOLETE if records else IndexState.ABSENT+ if kind is None:+ return IndexState.ORPHAN if records else IndexState.ABSENT+ valid_levels = {2} if kind == "file" else {0, 1}+ levels = {record.level for record in records}+ if levels - valid_levels:+ return IndexState.LEVEL_CONFLICT+ if not levels:+ return IndexState.MISSING+ if levels != valid_levels:+ return IndexState.PARTIAL+ if content_state == ContentState.MODIFIED:+ return IndexState.STALE+ if kind == "file" and md5:+ record_md5 = next(+ (str(record.fields.get("md5") or "") for record in records if record.level == 2),+ "",+ )+ if record_md5 and record_md5 != md5:+ return IndexState.STALE+ return IndexState.COMPLETE+++async def resolve_resource_diff(+ snapshot: RNFVSnapshot,+ *,+ store: Any,+ artifact_ref: Any,+ target: Any,+ artifact_paths: Mapping[str, str] | None = None,+ concurrency: int | None = None,+) -> ResourceDiffResult:+ """Resolve R/N/F/V into final states, including bounded body fallbacks."""+ snapshot.validate_for_planning()+ if (+ not snapshot.new.complete+ or not snapshot.formal.complete+ or (snapshot.request.vectorize and not snapshot.vectors.complete)+ ):+ raise ValueError("cannot resolve an incomplete RNFV snapshot")++ new = snapshot.new.entries+ formal = snapshot.formal.entries+ vector_records = snapshot.vectors.records_by_id if snapshot.request.vectorize else {}+ records_by_rel: dict[str, list[VectorRecordSnapshot]] = {}+ for record in vector_records.values():+ records_by_rel.setdefault(record.relative_path, []).append(record)++ states: dict[str, tuple[ContentState, str | None]] = {}+ compare_paths: list[str] = []+ hash_paths: list[str] = []+ md5_fast_path_count = 0+ keys = set(new) | set(formal) | set(records_by_rel)+ for rel_path in sorted(keys):+ new_entry = new.get(rel_path)+ old_entry = formal.get(rel_path)+ if new_entry is not None and old_entry is not None:+ new_kind = "directory" if new_entry.is_dir else "file"+ old_kind = "directory" if old_entry.is_dir else "file"+ if new_kind != old_kind:+ states[rel_path] = (ContentState.REPLACE_KIND, new_entry.md5 or None)+ elif new_entry.is_dir:+ states[rel_path] = (ContentState.UNCHANGED, None)+ else:+ canonical_records, _ = canonical_vector_records_by_level(+ records_by_rel.get(rel_path, ())+ )+ l2_record = canonical_records.get(2)+ l2_md5 = str(l2_record.fields.get("md5") or "") if l2_record else ""+ if new_entry.md5 and l2_md5:+ md5_fast_path_count += 1+ state = (+ ContentState.UNCHANGED if new_entry.md5 == l2_md5 else ContentState.MODIFIED+ )+ states[rel_path] = (state, new_entry.md5)+ else:+ compare_paths.append(rel_path)+ elif new_entry is not None:+ state = ContentState.RESTORE if records_by_rel.get(rel_path) else ContentState.ADDED+ states[rel_path] = (state, new_entry.md5 or None)+ if not new_entry.is_dir and not new_entry.md5:+ hash_paths.append(rel_path)+ elif old_entry is not None:+ states[rel_path] = (ContentState.DELETED, None)+ else:+ states[rel_path] = (ContentState.ABSENT, None)++ async def compare(rel_path: str) -> tuple[str, ContentState, str]:+ artifact_path = (artifact_paths or {}).get(rel_path, rel_path)+ new_bytes, old_bytes = await asyncio.gather(+ store.read_bytes(artifact_ref, artifact_path),+ target.read_file(rel_path),+ )+ from openviking.utils.content_hash import content_md5++ return (+ rel_path,+ ContentState.UNCHANGED if new_bytes == old_bytes else ContentState.MODIFIED,+ content_md5(new_bytes),+ )++ async def hash_new(rel_path: str) -> tuple[str, str]:+ artifact_path = (artifact_paths or {}).get(rel_path, rel_path)+ new_bytes = await store.read_bytes(artifact_ref, artifact_path)+ from openviking.utils.content_hash import content_md5++ return rel_path, content_md5(new_bytes)++ if concurrency is None:+ from openviking_cli.utils.config import get_openviking_config++ concurrency = int(+ get_openviking_config().queue_workers.add_resource.file_operation_concurrency+ )++ for rel_path, state, md5 in await bounded_map(+ compare_paths, compare, concurrency=max(1, concurrency)+ ):+ states[rel_path] = (state, md5)+ for rel_path, md5 in await bounded_map(hash_paths, hash_new, concurrency=max(1, concurrency)):+ state, _ = states[rel_path]+ states[rel_path] = (state, md5)++ result: dict[str, ResourceDiffEntry] = {}+ for rel_path in sorted(keys):+ state, md5 = states[rel_path]+ new_entry = new.get(rel_path)+ old_entry = formal.get(rel_path)+ new_kind = ("directory" if new_entry.is_dir else "file") if new_entry is not None else None+ old_kind = ("directory" if old_entry.is_dir else "file") if old_entry is not None else None+ current_kind = (+ new_kind if state not in {ContentState.DELETED, ContentState.ABSENT} else None+ )+ records = tuple(records_by_rel.get(rel_path, ()))+ canonical_records, _ = canonical_vector_records_by_level(records)+ result[rel_path] = ResourceDiffEntry(+ relative_path=rel_path,+ content_state=state,+ index_state=_index_state(+ tuple(canonical_records.values()),+ kind=current_kind,+ content_state=state,+ md5=md5,+ ),+ old_kind=old_kind,+ new_kind=new_kind,+ md5=md5,+ )+ resolved = ResourceDiffResult(entries=result)+ n_files, n_dirs, n_has_root = count_tree_entry_kinds(new)+ f_files, f_dirs, f_has_root = count_tree_entry_kinds(formal)+ file_entries = [+ entry+ for path, entry in resolved.entries.items()+ if path and (entry.new_kind or entry.old_kind) == "file"+ ]+ directory_entries = [+ entry+ for path, entry in resolved.entries.items()+ if path and (entry.new_kind or entry.old_kind) == "directory"+ ]+ root_entry = resolved.entries.get("")+ file_content_counts = Counter(entry.content_state.value for entry in file_entries)+ directory_content_counts = Counter(entry.content_state.value for entry in directory_entries)+ file_index_counts = Counter(entry.index_state.value for entry in file_entries)+ directory_index_counts = Counter(entry.index_state.value for entry in directory_entries)+ logger.info(+ "[ResourceDiffResult] %s target=%s n_files=%d n_dirs=%d f_files=%d f_dirs=%d "+ "logical_root=%s v_records=%d root_state=%s root_index=%s "+ "md5_fast_path=%d body_compared=%d new_files_hashed=%d "+ "states={file:%s,dir:%s} index={file:%s,dir:%s}",+ log_correlation(),+ snapshot.request.target_uri,+ n_files,+ n_dirs,+ f_files,+ f_dirs,+ str(n_has_root or f_has_root).lower(),+ len(vector_records),+ root_entry.content_state.value if root_entry is not None else "none",+ root_entry.index_state.value if root_entry is not None else "none",+ md5_fast_path_count,+ len(compare_paths),+ len(hash_paths),+ dict(file_content_counts),+ dict(directory_content_counts),+ dict(file_index_counts),+ dict(directory_index_counts),+ )+ return resolved+++def _is_excluded_rel_path(rel_path: str) -> bool:+ """Control sidecars and storage-internal entries are never business files."""+ if not rel_path:+ return True+ for segment in rel_path.split("/"):+ if segment in STORAGE_INTERNAL_ENTRY_NAMES or segment in CONTROL_BASENAMES:+ return True+ return False+++async def read_target_file_snapshot(+ viking_fs: Any,+ target_uri: str,+ *,+ ctx: Any,+ root_is_file: bool = False,+) -> Tuple[Dict[str, FormalEntry], bool]:+ """Return ``(rel_path -> FormalEntry, complete)`` for the formal tree.++ ``complete`` is False when any entry is permission-denied, because a subtree+ we cannot see must not be interpreted as absent (which would drive deletion).+ """+ if root_is_file:+ stat = await viking_fs.stat(target_uri, ctx=ctx, skip_count=True)+ return {"": FormalEntry(is_dir=bool(stat.get("isDir")))}, True++ entries = await viking_fs.tree(+ target_uri,+ output="original",+ show_all_hidden=True,+ node_limit=None,+ level_limit=None,+ ctx=ctx,+ )+ files: Dict[str, FormalEntry] = {}+ complete = True+ for entry in entries:+ if entry.get("access") == "denied":+ complete = False+ continue+ rel_path = str(entry.get("rel_path") or "").strip("/")+ if _is_excluded_rel_path(rel_path):+ continue+ files[rel_path] = FormalEntry(is_dir=bool(entry.get("isDir")))+ return files, complete+++async def prepare_artifact_inventory(+ store: Any,+ ref: Any,+ *,+ doc_rel: str = "",+ target_root_uri: str | None = None,+ root_is_file: bool = False,+) -> ArtifactInventory:+ """Prepare an artifact once and return its final-byte N snapshot.++ The one recursive walk supplies the file inventory consumed by RNFV and,+ when a final target URI is known, rewrites mapped markdown image references+ in place. Rewritten markdown immediately receives an updated manifest MD5,+ so the next diff compares the exact bytes that would be uploaded.+ """+ from openviking.parse.image_rewrite import IMAGE_MAPPINGS_FILENAME, _rewrite_content+ from openviking.parse.output import read_artifact_manifest, write_artifact_manifest+ from openviking.utils.content_hash import content_md5++ base = doc_rel.strip("/")+ prefix = f"{base}/" if base else ""+ md5_by_artifact_rel = await read_artifact_manifest(store, ref)+ entries: Dict[str, NewEntry] = {}+ artifact_paths: Dict[str, str] = {}+ rewritten_paths: set[str] = set()++ def target_relative(artifact_rel: str) -> str:+ if base and artifact_rel.startswith(prefix):+ return artifact_rel[len(prefix) :]+ return artifact_rel++ if root_is_file:+ return ArtifactInventory(+ entries={"": NewEntry(md5=md5_by_artifact_rel.get(base, ""), is_dir=False)},+ artifact_paths={"": base},+ )++ async def walk(+ directory: str,+ mapping_dir: str = "",+ inherited_mappings: Mapping[str, Mapping[str, str]] | None = None,+ ) -> None:+ directory_entries = await store.list(ref, directory)+ current_mappings = inherited_mappings or {}+ current_mapping_dir = mapping_dir+ mapping_entry = next(+ (entry for entry in directory_entries if entry.name == IMAGE_MAPPINGS_FILENAME),+ None,+ )+ if mapping_entry is not None:+ try:+ loaded = json.loads(+ (await store.read_bytes(ref, mapping_entry.rel_path)).decode("utf-8")+ )+ except Exception as exc:+ logger.warning(+ "[ArtifactInventory] Ignoring unreadable image mapping sidecar %s: %s",+ mapping_entry.rel_path,+ exc,+ )+ loaded = None+ if not isinstance(loaded, dict):+ if loaded is not None:+ logger.warning(+ "[ArtifactInventory] Ignoring invalid image mapping sidecar %s",+ mapping_entry.rel_path,+ )+ else:+ current_mappings = loaded+ current_mapping_dir = directory++ available_files = {+ entry.name+ for entry in directory_entries+ if not entry.is_dir and not entry.name.startswith(".")+ }+ for entry in directory_entries:+ if _is_excluded_rel_path(entry.rel_path):+ continue+ relative_path = target_relative(entry.rel_path)+ if entry.is_dir:+ if relative_path:+ entries[relative_path] = NewEntry(is_dir=True)+ await walk(entry.rel_path, current_mapping_dir, current_mappings)+ continue++ if not relative_path:+ continue+ md5 = md5_by_artifact_rel.get(entry.rel_path, "")+ if target_root_uri and entry.name.lower().endswith((".md", ".markdown")):+ mapping_key = (+ entry.rel_path[len(current_mapping_dir) + 1 :]+ if current_mapping_dir and entry.rel_path.startswith(current_mapping_dir + "/")+ else entry.rel_path+ )+ path_mappings = current_mappings.get(mapping_key)+ if isinstance(path_mappings, dict) and path_mappings:+ content = (await store.read_bytes(ref, entry.rel_path)).decode("utf-8")+ target_uri = f"{target_root_uri.rstrip('/')}/{relative_path}"+ rewritten, count = _rewrite_content(+ content,+ target_uri.rsplit("/", 1)[0],+ available_files,+ {str(key): str(value) for key, value in path_mappings.items()},+ )+ if count:+ final_bytes = rewritten.encode("utf-8")+ await store.write_bytes(ref, entry.rel_path, final_bytes)+ md5 = content_md5(final_bytes)+ md5_by_artifact_rel[entry.rel_path] = md5+ rewritten_paths.add(relative_path)+ entries[relative_path] = NewEntry(md5=md5, is_dir=False)+ artifact_paths[relative_path] = entry.rel_path++ await walk(base)+ if rewritten_paths:+ await write_artifact_manifest(store, ref, md5_by_artifact_rel)+ return ArtifactInventory(+ entries=entries,+ artifact_paths=artifact_paths,+ rewritten_paths=frozenset(rewritten_paths),+ )+++async def build_rnfv_snapshot(+ *,+ viking_fs: Any,+ vikingdb: Any,+ store: Any,+ artifact_ref: Any,+ target_uri: str,+ ctx: Any,+ doc_rel: str = "",+ request_intent: RequestIntent | None = None,+ root_is_file: bool = False,+ target_preexisting: bool = True,+ artifact_inventory: ArtifactInventory | None = None,+) -> RNFVSnapshot:+ """Read the complete R/N/F/V inputs without deriving an executable plan."""+ request = request_intent or RequestIntent(+ target_uri=target_uri, processing_mode="semantic_and_vectors"+ )++ async def read_artifact() -> ArtifactInventory:+ return artifact_inventory or await prepare_artifact_inventory(+ store, artifact_ref, doc_rel=doc_rel, root_is_file=root_is_file+ )++ async def read_formal() -> tuple[Dict[str, FormalEntry], bool]:+ if not target_preexisting:+ return {}, True+ return await read_target_file_snapshot(+ viking_fs, target_uri, ctx=ctx, root_is_file=root_is_file+ )++ projection = request.required_vector_fields()+ if request.vectorize and (root_is_file or request.processing_mode == "vectors_only"):+ projection = projection | {"abstract"}++ tasks = [+ asyncio.create_task(read_artifact()),+ asyncio.create_task(read_formal()),+ ]+ if request.vectorize:+ tasks.append(+ asyncio.create_task(+ _read_incremental_vector_inventory(+ vikingdb, target_uri=target_uri, ctx=ctx, projection=projection+ )+ )+ )+ try:+ results = await asyncio.gather(*tasks)+ except BaseException:+ for task in tasks:+ if not task.done():+ task.cancel()+ await asyncio.gather(*tasks, return_exceptions=True)+ raise+ artifact, formal = results[:2]+ inventory = results[2] if request.vectorize else {}+ target_files, files_complete = formal+ base = target_uri.rstrip("/")+ prefix = base + "/"+ vector_records: Dict[str, VectorRecordSnapshot] = {}+ for record_id, record in inventory.items():+ level = int(record.get("level", -1))+ uri = str(record.get("uri") or "")+ rel = "" if uri == base else uri[len(prefix) :] if uri.startswith(prefix) else None+ if rel is None:+ raise RuntimeError(f"Vector inventory returned an out-of-scope URI: {uri}")+ fields = {+ field: record[field] for field in projection - {"id", "uri", "level"} if field in record+ }+ vector_records[record_id] = VectorRecordSnapshot(+ record_id=record_id,+ uri=uri,+ relative_path=rel,+ level=level,+ fields=fields,+ )+ return RNFVSnapshot(+ request=request,+ new=NewArtifactSnapshot(entries=artifact.entries),+ formal=FormalTreeSnapshot(entries=target_files, complete=files_complete),+ vectors=VectorIndexSnapshot(+ records_by_id=vector_records,+ projected_fields=projection,+ ),+ )+++async def _read_incremental_vector_inventory(+ vikingdb: Any,+ *,+ target_uri: str,+ ctx: Any,+ projection: frozenset[str],+) -> Dict[str, Dict[str, Any]]:+ """Read the lightweight V snapshot used to build RNFV."""+ return await vikingdb.get_incremental_inventory_under_uri(+ target_uri, ctx=ctx, output_fields=sorted(projection)+ )+++__all__ = [+ "ArtifactInventory",+ "ContentState",+ "IndexState",+ "ResourceDiffEntry",+ "ResourceDiffResult",+ "build_rnfv_snapshot",+ "count_tree_entry_kinds",+ "prepare_artifact_inventory",+ "read_target_file_snapshot",+]diff --git a/openviking/storage/resource_rnfv.py b/openviking/storage/resource_rnfv.pynew file mode 100644index 000000000..9077eff66--- /dev/null+++ b/openviking/storage/resource_rnfv.py@@ -0,0 +1,240 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Typed R/N/F/V inputs for resource update planning.++R is normalized request intent, N is the parser artifact snapshot, F is the+formal resource tree, and V is the vector-index snapshot. A validated snapshot+contains every fact needed by the update planner; the planner must not read the+request or either storage backend again.+"""++from __future__ import annotations++from dataclasses import dataclass, field+from typing import Any, Iterable, Mapping++from openviking.utils.ingest_options import IngestOptions++BASE_VECTOR_PROJECTION = frozenset({"id", "uri", "level", "md5"})+NON_PORTABLE_VECTOR_RECORD_FIELDS = frozenset(+ {+ # Identity and tenancy are derived from the current URI/request.+ "id",+ "uri",+ "level",+ "account_id",+ "owner_user_id",+ # Model/content outputs belong to the current execution.+ "vector",+ "sparse_vector",+ "content",+ "updated_at",+ "md5",+ "abstract",+ "type",+ "name",+ # ACL is re-materialized from the current resource hierarchy.+ "acl_mode",+ "acl_direct_grants",+ "acl_inherited_grants",+ }+)+_SCALAR_MODES = frozenset({"replace", "append"})+_SCALAR_FIELDS = frozenset({"search_tags"})++# Parser and formal-tree entries use the same compact shape. They are defined+# beside RNFV rather than a legacy planner so every update path shares them.+CONTROL_BASENAMES = frozenset(+ {".abstract.md", ".overview.md", ".image_mappings.json", ".artifact_manifest.json"}+)+++@dataclass(frozen=True)+class NewEntry:+ """One parser-artifact path: final file MD5 or a topology-only directory."""++ md5: str = ""+ is_dir: bool = False+++@dataclass(frozen=True)+class FormalEntry:+ """One formal-tree path; F records topology, not historical file bytes."""++ is_dir: bool = False+++@dataclass(frozen=True)+class ScalarIntent:+ """One normalized request-side scalar mutation."""++ field: str+ mode: str+ value: Any+ target_levels: frozenset[int] = frozenset({0, 1, 2})++ def __post_init__(self) -> None:+ if self.field not in _SCALAR_FIELDS:+ raise ValueError(f"unsupported resource scalar intent: {self.field}")+ if self.mode not in _SCALAR_MODES:+ raise ValueError(f"unsupported scalar update mode: {self.mode}")+ if not self.target_levels or not self.target_levels <= {0, 1, 2}:+ raise ValueError(f"invalid scalar target levels: {sorted(self.target_levels)}")++ def required_existing_fields(self) -> frozenset[str]:+ # Reading the old value is useful for both modes: append needs it to+ # calculate the result, while replace needs it to detect an effective+ # no-op instead of issuing a redundant update.+ return frozenset({self.field})+++@dataclass(frozen=True)+class RequestIntent:+ """Normalized request intent (R), independent of HTTP/queue payload shape."""++ target_uri: str+ processing_mode: str+ vectorize: bool = True+ scalar_intents: tuple[ScalarIntent, ...] = ()++ @classmethod+ def from_ingest_options(+ cls,+ *,+ target_uri: str,+ processing_mode: Any,+ ingest_options: IngestOptions | Mapping[str, Any] | None,+ vectorize: bool = True,+ ) -> "RequestIntent":+ options = IngestOptions.from_value(ingest_options)+ scalar_intents: tuple[ScalarIntent, ...] = ()+ if options.search_tags is not None:+ mode = getattr(processing_mode, "value", processing_mode)+ scalar_intents = (+ ScalarIntent(+ field="search_tags",+ mode=options.search_tag_mode,+ value=tuple(options.search_tags),+ target_levels=(+ frozenset({2}) if str(mode) == "vectors_only" else frozenset({0, 1, 2})+ ),+ ),+ )+ else:+ mode = getattr(processing_mode, "value", processing_mode)+ return cls(+ target_uri=target_uri.rstrip("/"),+ processing_mode=str(mode),+ vectorize=bool(vectorize),+ scalar_intents=scalar_intents,+ )++ def required_vector_fields(self) -> frozenset[str]:+ if not self.vectorize:+ return frozenset()+ fields = set(BASE_VECTOR_PROJECTION)+ for intent in self.scalar_intents:+ fields.update(intent.required_existing_fields())+ return frozenset(fields)+++@dataclass(frozen=True)+class NewArtifactSnapshot:+ entries: Mapping[str, Any]+ complete: bool = True+++@dataclass(frozen=True)+class FormalTreeSnapshot:+ entries: Mapping[str, Any]+ complete: bool = True+++@dataclass(frozen=True)+class VectorRecordSnapshot:+ """One V inventory record with actual backend identity and hydrated scalars."""++ record_id: str+ uri: str+ relative_path: str+ level: int+ fields: Mapping[str, Any] = field(default_factory=dict)++ def __post_init__(self) -> None:+ if not self.record_id:+ raise ValueError("vector snapshot record_id must not be empty")+ if self.level not in {0, 1, 2}:+ raise ValueError(f"invalid vector snapshot level: {self.level}")+++def canonical_vector_records_by_level(+ records: Iterable[VectorRecordSnapshot],+) -> tuple[dict[int, VectorRecordSnapshot], tuple[VectorRecordSnapshot, ...]]:+ """Choose the lowest record ID per level and return the duplicates."""+ canonical: dict[int, VectorRecordSnapshot] = {}+ duplicates: list[VectorRecordSnapshot] = []+ for record in records:+ current = canonical.get(record.level)+ if current is None:+ canonical[record.level] = record+ elif record.record_id < current.record_id:+ duplicates.append(current)+ canonical[record.level] = record+ else:+ duplicates.append(record)+ return canonical, tuple(duplicates)+++@dataclass(frozen=True)+class VectorIndexSnapshot:+ """Complete target-prefix V inventory and the projection used to read it."""++ records_by_id: Mapping[str, VectorRecordSnapshot]+ projected_fields: frozenset[str]+ complete: bool = True+++@dataclass(frozen=True)+class RNFVSnapshot:+ """Immutable R/N/F/V facts consumed by diff resolution and plan compilation.++ R carries only explicit request intent, N/F decide content topology, and V+ supplies indexed levels, record identities, and selectively hydrated scalars.+ No later planner step may silently reread those sources.+ """++ request: RequestIntent+ new: NewArtifactSnapshot+ formal: FormalTreeSnapshot+ vectors: VectorIndexSnapshot++ def validate_for_planning(self) -> None:+ if not self.request.vectorize:+ return+ required = self.request.required_vector_fields()+ missing = required - self.vectors.projected_fields+ if missing:+ raise ValueError(+ "RNFV vector projection misses required fields: " + ", ".join(sorted(missing))+ )+ root = self.request.target_uri.rstrip("/")+ for record in self.vectors.records_by_id.values():+ if record.uri != root and not record.uri.startswith(root + "/"):+ raise ValueError(f"RNFV vector record is outside target: {record.uri}")+++__all__ = [+ "BASE_VECTOR_PROJECTION",+ "CONTROL_BASENAMES",+ "FormalTreeSnapshot",+ "NewArtifactSnapshot",+ "NewEntry",+ "NON_PORTABLE_VECTOR_RECORD_FIELDS",+ "RequestIntent",+ "RNFVSnapshot",+ "ScalarIntent",+ "FormalEntry",+ "VectorIndexSnapshot",+ "VectorRecordSnapshot",+ "canonical_vector_records_by_level",+]diff --git a/openviking/storage/resource_target.py b/openviking/storage/resource_target.pynew file mode 100644index 000000000..ceee8b7fd--- /dev/null+++ b/openviking/storage/resource_target.py@@ -0,0 +1,69 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Concrete AGFS write side for planned content-tree actions.++This is the real AGFS implementation: it joins artifact-relative paths onto the+resource root and writes the already-normalized artifact bytes unchanged so the+stored bytes and their manifest md5 stay identical. Content mutation stays behind+this adapter so the planner remains independent from VikingFS URI and lease details.++An initial import is simply "the plan is all added", so the same target serves+both first-import (full upload) and incremental (diff subset) via one channel.+"""++from __future__ import annotations++from typing import Any, Optional++from openviking.utils.path_safety import safe_join_viking_uri+++class AgfsResourceTarget:+ """Write/delete side of a diff apply against an AGFS resource tree."""++ def __init__(+ self,+ *,+ viking_fs: Any,+ root_uri: str,+ ctx: Any,+ lease_ref: Optional[dict] = None,+ ) -> None:+ self._viking_fs = viking_fs+ self._root_uri = root_uri.rstrip("/")+ self._ctx = ctx+ self._lease_ref = lease_ref++ def _resolve(self, rel_path: str) -> str:+ # safe_join_viking_uri rejects absolute paths, drive prefixes and ..+ # traversal, keeping every write inside the resource root.+ return self._root_uri if not rel_path else safe_join_viking_uri(self._root_uri, rel_path)++ async def write_file(self, rel_path: str, data: bytes) -> bytes:+ """Write the artifact bytes unchanged; normalization happened at creation."""+ uri = self._resolve(rel_path)+ await self._viking_fs.write_file_bytes(uri, data, ctx=self._ctx, lease_ref=self._lease_ref)+ return data++ async def mkdir(self, rel_path: str) -> None:+ await self._viking_fs.mkdir(+ self._resolve(rel_path),+ exist_ok=True,+ ctx=self._ctx,+ lease_ref=self._lease_ref,+ )++ async def read_file(self, rel_path: str) -> bytes:+ return await self._viking_fs.read_file_bytes(self._resolve(rel_path), ctx=self._ctx)++ async def delete_path(self, rel_path: str, *, is_dir: bool) -> None:+ """Delete one planned path under the already-held resource tree lease."""+ await self._viking_fs.remove_files(+ self._resolve(rel_path),+ recursive=is_dir,+ ctx=self._ctx,+ lease_ref=self._lease_ref,+ )+++__all__ = ["AgfsResourceTarget"]diff --git a/openviking/storage/vectordb_adapters/base.py b/openviking/storage/vectordb_adapters/base.pyindex b0a42dd56..0701ef5b3 100644--- a/openviking/storage/vectordb_adapters/base.py+++ b/openviking/storage/vectordb_adapters/base.py@@ -635,6 +635,19 @@ class CollectionAdapter(ABC):raise RuntimeError("Vector backend returned an invalid count result")+ def strict_count(self, filter: Optional[Dict[str, Any] | FilterExpr] = None) -> int:+ """Count records and reject responses without an explicit total."""+ coll = self.get_collection()+ result = coll.aggregate_data(+ index_name=self._index_name,+ op="count",+ filters=self._compile_filter(filter),+ )+ parsed_total = self._extract_count_total(result.agg)+ if parsed_total is None:+ raise RuntimeError("Vector backend returned an invalid count response")+ return parsed_total+def search_by_keywords(self,keywords: Optional[list[str]] = None,diff --git a/openviking/storage/viking_fs/_base.py b/openviking/storage/viking_fs/_base.pyindex 98a1b6b0e..9def2bf1b 100644--- a/openviking/storage/viking_fs/_base.py+++ b/openviking/storage/viking_fs/_base.py@@ -21,7 +21,7 @@ logger = get_logger(__name__)# Sentinel node_limit for internal callers that MUST enumerate an entire# directory. ``ls()`` defaults to ``node_limit=1000`` to protect agent-facing# context from being flooded, but internal system operations (parse merge,-# temp->final sync, summary DAG, vectorization) must see every child or they+# temp->final sync, semantic-tree execution, vectorization) must see every child or they# silently drop entries beyond the cap — e.g. a >1000-doc directory ingest only# materializes its first 1000 subdirectories. Pass this explicitly at those# call sites.diff --git a/openviking/storage/viking_vector_index_backend.py b/openviking/storage/viking_vector_index_backend.pyindex a74b89678..ce1dc4c8d 100644--- a/openviking/storage/viking_vector_index_backend.py+++ b/openviking/storage/viking_vector_index_backend.py@@ -20,6 +20,7 @@ from openviking.server.identity import RequestContext, Rolefrom openviking.service.task_tracker_concurrency import run_to_completionfrom openviking.storage.acl import (ACL_CONTEXT_FIELDS,+ ACL_GRANT_FIELDS,ACL_MODE_FIELD,AclAction,AclManager,@@ -80,6 +81,40 @@ FETCH_BY_URI_OUTPUT_FIELDS = ["owner_user_id",]+# Fields an incremental diff needs from existing target records. md5 is listed+# explicitly because the default lookup projection omits it; a missing value+# means "unknown" and the caller falls back to reading file bytes.+INCREMENTAL_DIFF_OUTPUT_FIELDS = [+ "id",+ "uri",+ "level",+ "abstract",+ "md5",+]++INCREMENTAL_INVENTORY_OUTPUT_FIELDS = ["id", "uri", "level", "md5"]++INCREMENTAL_HYDRATION_OUTPUT_FIELDS = [+ "id",+ "uri",+ "type",+ "context_type",+ "created_at",+ "updated_at",+ "active_count",+ "level",+ "name",+ "description",+ "tags",+ "search_tags",+ "abstract",+ "md5",+ "account_id",+ "owner_user_id",+ ACL_MODE_FIELD,+ *ACL_GRANT_FIELDS,+]+VIKINGDB_CONTENT_MAX_SIZE = 1024 * 1024@@ -292,6 +327,19 @@ class _SingleAccountBackend:"""Prepare a batch in one worker-thread handoff."""return [self._prepare_upsert_payload(data) for data in data_list]+ def _prepare_update_payload(self, data: Dict[str, Any]) -> Dict[str, Any]:+ """Prepare a strict partial update without inventing omitted fields."""+ payload = self._filter_known_fields(+ {key: value for key, value in data.items() if value is not None}+ )+ if self._adapter.USE_CONTENT_FIELD:+ content = payload.get("content")+ if isinstance(content, (str, bytes)):+ payload["content"] = content[:VIKINGDB_CONTENT_MAX_SIZE]+ else:+ payload.pop("content", None)+ return payload+def _bind_upsert_payload(self, data: Dict[str, Any]) -> Dict[str, Any]:"""Copy a record, enforce its bound account, and apply write defaults."""payload = dict(data)@@ -506,7 +554,7 @@ class _SingleAccountBackend:allowed = sorted(VikingVectorIndexBackend.ALLOWED_CONTEXT_TYPES)raise ValueError(f"Invalid context_type: {context_type}. Must be one of {allowed}")- payload = await self._async_adapter.run(self._prepare_upsert_payload, payload)+ payload = await self._async_adapter.run(self._prepare_update_payload, payload)ids = await self._async_adapter.call("update_data", [payload])normalized_ids = [str(item) for item in (ids or []) if item is not None]return UpdateResult(@@ -609,7 +657,7 @@ class _SingleAccountBackend:async def strict_count(self, filter: Optional[Dict[str, Any] | FilterExpr] = None) -> int:"""Count transaction records without converting backend errors to zero."""return int(- await self._async_adapter.call("count", filter=self._with_account_filter(filter)) or 0+ await self._async_adapter.call("strict_count", filter=self._with_account_filter(filter)))async def delete(self, ids: List[str]) -> int:@@ -1188,6 +1236,10 @@ class VikingVectorIndexBackend:backend = self._get_backend_for_context(ctx)return await backend.delete(ids)+ async def strict_delete(self, ids: List[str], *, ctx: RequestContext) -> int:+ """Delete exact record IDs without converting backend failures to success."""+ return await self._get_backend_for_context(ctx).strict_delete(ids)+async def exists(self, id: str, *, ctx: RequestContext) -> bool:backend = self._get_backend_for_context(ctx)return await backend.exists(id)@@ -1438,6 +1490,60 @@ class VikingVectorIndexBackend:backend = self._get_backend_for_context(ctx)return await backend.strict_count(filter=filter)+ async def _strict_scan(+ self,+ ctx: RequestContext,+ scope: FilterExpr,+ *,+ output_fields: List[str],+ batch_size: int,+ what: str,+ ) -> AsyncIterator[Dict[str, Any]]:+ """Yield every record under ``scope`` with strict count/cursor guarantees.++ Consolidates the paginated scan skeleton shared by the incremental+ readers: it pins the expected total up front, then walks cursor pages,+ raising if a page ends early, overshoots the count, or the cursor repeats+ or dies before the total is reached — so a truncated scan can never look+ like "records absent". Per-record parsing/validation stays with the+ caller; this only guarantees the traversal is complete.+ """+ if batch_size <= 0:+ raise ValueError("batch_size must be positive")+ expected_count = await self._strict_transfer_count(ctx, scope)+ cursor: Optional[str] = None+ scanned_count = 0+ seen_cursors: set[str] = set()+ while True:+ page, next_cursor = await self._strict_transfer_page(+ ctx,+ scope,+ limit=batch_size,+ cursor=cursor,+ output_fields=output_fields,+ )+ if not page and scanned_count < expected_count:+ raise RuntimeError(+ f"{what} ended after {scanned_count} of {expected_count} records"+ )+ scanned_count += len(page)+ if scanned_count > expected_count:+ raise RuntimeError(+ f"{what} returned {scanned_count} records but count was {expected_count}"+ )+ for record in page:+ yield record+ if next_cursor is None:+ if scanned_count == expected_count:+ return+ raise RuntimeError(+ f"{what} cursor ended after {scanned_count} of {expected_count} records"+ )+ if next_cursor in seen_cursors:+ raise RuntimeError(f"{what} cursor repeated: {next_cursor}")+ seen_cursors.add(next_cursor)+ cursor = next_cursor+async def _strict_transfer_get(self, ctx: RequestContext, ids: List[str]) -> List[Dict[str, Any]]:@@ -1712,6 +1818,273 @@ class VikingVectorIndexBackend:if uri in requested_by_canonical}+ async def get_l2_diff_records_by_uris(+ self,+ uris: List[str],+ *,+ ctx: RequestContext,+ ) -> Dict[str, Dict[str, Any]]:+ """Load existing L2 diff metadata (md5 + abstract) for a bounded URI set.++ Returns a map keyed by the caller's original URI to+ ``{"md5": str, "abstract": str}``. ``md5`` is empty when the record+ predates the field, in which case incremental diff must fall back to+ comparing file bytes rather than assuming equality. Uses full strict+ pagination so a truncated page never masquerades as "record absent".+ """+ requested_by_canonical: Dict[str, str] = {}+ for uri in uris:+ requested_by_canonical.setdefault(resolve_uri(uri).uri, uri)+ canonical_uris = list(requested_by_canonical)+ if not canonical_uris:+ return {}++ records_by_uri: Dict[str, Dict[str, Any]] = {}+ chunk_size = 100+ for start in range(0, len(canonical_uris), chunk_size):+ chunk = canonical_uris[start : start + chunk_size]+ cursor: Optional[str] = None+ while True:+ records, cursor = await self._strict_transfer_page(+ ctx,+ And([In("uri", chunk), Eq("level", 2)]),+ limit=100,+ cursor=cursor,+ output_fields=["uri", "md5", "abstract"],+ )+ for record in records:+ uri = str(record.get("uri") or "")+ if uri and uri not in records_by_uri:+ records_by_uri[uri] = {+ "md5": str(record.get("md5") or ""),+ "abstract": str(record.get("abstract") or ""),+ }+ if cursor is None:+ break+ return {+ requested_by_canonical[uri]: record+ for uri, record in records_by_uri.items()+ if uri in requested_by_canonical+ }++ async def get_l2_diff_records_under_uri(+ self,+ uri: str,+ *,+ ctx: RequestContext,+ batch_size: int = 100,+ ) -> Dict[str, Dict[str, Any]]:+ """Strictly load every L2 diff record below one target directory."""+ canonical_uri = resolve_uri(uri).uri.rstrip("/")+ scope = And(+ [+ Eq("account_id", ctx.account_id),+ PathScope("uri", canonical_uri, depth=-1),+ Eq("level", 2),+ ]+ )+ records: Dict[str, Dict[str, Any]] = {}+ what = f"Vector scan under {canonical_uri}"+ async for record in self._strict_scan(+ ctx,+ scope,+ output_fields=INCREMENTAL_DIFF_OUTPUT_FIELDS,+ batch_size=batch_size,+ what=what,+ ):+ record_uri = str(record.get("uri") or "")+ if not record_uri or not uri_in_transfer_scope(+ record_uri, canonical_uri, recursive=True+ ):+ raise RuntimeError(f"{what} returned invalid L2 URI: {record_uri or '<missing>'}")+ if record_uri in records:+ raise RuntimeError(f"{what} returned duplicate L2 URI: {record_uri}")+ records[record_uri] = {+ "md5": str(record.get("md5") or ""),+ "abstract": str(record.get("abstract") or ""),+ }+ return records++ async def get_incremental_inventory_under_uri(+ self,+ uri: str,+ *,+ ctx: RequestContext,+ batch_size: int = 100,+ output_fields: Optional[List[str]] = None,+ ) -> Dict[str, Dict[str, Any]]:+ """Strictly load lightweight L0/L1/L2 metadata below a resource root."""+ projection = list(+ dict.fromkeys(+ [+ *INCREMENTAL_INVENTORY_OUTPUT_FIELDS,+ *(output_fields or []),+ ]+ )+ )+ canonical_uri = resolve_uri(uri).uri.rstrip("/")+ scope = And(+ [+ Eq("account_id", ctx.account_id),+ PathScope("uri", canonical_uri, depth=-1),+ In("level", [0, 1, 2]),+ ]+ )+ what = f"Incremental inventory under {canonical_uri}"+ retry_delays = (0.0, 0.25, 1.0)+ last_error: Exception | None = None+ for attempt, delay in enumerate(retry_delays, start=1):+ if delay:+ await asyncio.sleep(delay)+ records: Dict[str, Dict[str, Any]] = {}+ try:+ async for record in self._strict_scan(+ ctx,+ scope,+ output_fields=projection,+ batch_size=batch_size,+ what=what,+ ):+ record_id = str(record.get("id") or "")+ record_uri = str(record.get("uri") or "")+ try:+ level = int(record.get("level"))+ except (TypeError, ValueError):+ level = -1+ if (+ not record_id+ or level not in {0, 1, 2}+ or not uri_in_transfer_scope(record_uri, canonical_uri, recursive=True)+ ):+ raise RuntimeError(+ f"{what} returned an invalid record: id={record_id or '<missing>'} "+ f"uri={record_uri or '<missing>'} level={level}"+ )+ if record_id in records:+ raise RuntimeError(f"{what} returned duplicate record id: {record_id}")+ records[record_id] = {+ field: record[field] for field in projection if field in record+ }+ records[record_id].update(+ {+ "id": record_id,+ "uri": record_uri,+ "level": level,+ "md5": str(record.get("md5") or ""),+ }+ )+ return records+ except RuntimeError as exc:+ last_error = exc+ if attempt == len(retry_delays) or not any(+ marker in str(exc)+ for marker in (+ "cursor ended after",+ "ended after",+ "records but count was",+ )+ ):+ raise+ logger.warning(+ "%s was incomplete; retrying full inventory attempt=%d/%d error=%s",+ what,+ attempt,+ len(retry_delays),+ exc,+ )+ assert last_error is not None+ raise last_error++ async def hydrate_incremental_records(+ self,+ expected: Mapping[str, Mapping[str, Any]],+ *,+ ctx: RequestContext,+ batch_size: int = 100,+ output_fields: Optional[Container[str]] = None,+ ) -> Dict[str, Dict[str, Any]]:+ """Load requested non-vector fields, using strict DSL before ID fallback."""+ if batch_size <= 0:+ raise ValueError("batch_size must be positive")+ requested_ids = list(expected)+ if output_fields is None:+ selected_fields = list(INCREMENTAL_HYDRATION_OUTPUT_FIELDS)+ try:+ collection_meta = await self.get_collection_meta(ctx=ctx)+ schema_fields = [+ str(item.get("FieldName") or "")+ for item in (collection_meta or {}).get("Fields", [])+ ]+ dynamic_fields = [+ field+ for field in schema_fields+ if field and field not in {"vector", "sparse_vector", "content"}+ ]+ if dynamic_fields:+ selected_fields = list(dict.fromkeys(dynamic_fields))+ except Exception:+ # Backends without collection metadata retain the known portable+ # projection; correctness remains fail-closed at identity checks.+ pass+ else:+ selected_fields = list(dict.fromkeys(["id", "uri", "level", *sorted(output_fields)]))+ hydrated: Dict[str, Dict[str, Any]] = {}++ def _accept(record: Mapping[str, Any]) -> None:+ record_id = str(record.get("id") or "")+ wanted = expected.get(record_id)+ if wanted is None:+ raise RuntimeError(+ f"Incremental hydration returned unexpected record id: {record_id or '<missing>'}"+ )+ record_uri = str(record.get("uri") or "")+ try:+ level = int(record.get("level"))+ except (TypeError, ValueError):+ level = -1+ if (+ record_uri != str(wanted.get("uri") or "")+ or level != int(wanted.get("level", -1))+ or (record.get("account_id") not in {None, ctx.account_id})+ ):+ raise RuntimeError(+ f"Incremental hydration identity mismatch for record: {record_id}"+ )+ if record_id in hydrated:+ raise RuntimeError(+ f"Incremental hydration returned duplicate record id: {record_id}"+ )+ hydrated[record_id] = {+ field: record[field] for field in selected_fields if field in record+ }++ for start in range(0, len(requested_ids), batch_size):+ chunk = requested_ids[start : start + batch_size]+ cursor: Optional[str] = None+ seen_cursors: set[str] = set()+ while True:+ page, next_cursor = await self._strict_transfer_page(+ ctx,+ In("id", chunk),+ limit=batch_size,+ cursor=cursor,+ output_fields=selected_fields,+ )+ for record in page:+ _accept(record)+ if next_cursor is None:+ break+ if next_cursor in seen_cursors:+ raise RuntimeError(f"Incremental hydration cursor repeated: {next_cursor}")+ seen_cursors.add(next_cursor)+ cursor = next_cursor++ missing = [record_id for record_id in chunk if record_id not in hydrated]+ if missing:+ for record in await self._strict_transfer_get(ctx, missing):+ _accept(record)+ return hydrated+async def delete_account_data(self, account_id: str, *, ctx: RequestContext) -> int:"""删除指定 account 的所有数据(仅限,root 角色操作)"""self._check_root_role(ctx)@@ -2276,11 +2649,7 @@ class VikingVectorIndexBackend:targets = [target_dir for target_dir in target_directories or [] if target_dir]tenant_filter = self._tenant_filter(ctx, acl_enabled=acl_enabled)- if (- tenant_filter- and not acl_enabled- and self._targets_within_visible_roots(ctx, targets)- ):+ if tenant_filter and not acl_enabled and self._targets_within_visible_roots(ctx, targets):# The target scopes are already narrower than the tenant-visible# roots. Keep account isolation, but avoid recursively evaluating# the broader path scopes as an additional filter.diff --git a/openviking/telemetry/operation.py b/openviking/telemetry/operation.pyindex a8951bba4..6ff3eb626 100644--- a/openviking/telemetry/operation.py+++ b/openviking/telemetry/operation.py@@ -264,11 +264,29 @@ class TelemetrySummaryBuilder:"processed": cls._i(gauges.get("queue.semantic.processed"), 0),"requeue_count": cls._i(gauges.get("queue.semantic.requeue_count"), 0),"error_count": cls._i(gauges.get("queue.semantic.error_count"), 0),+ "queue_wait": {+ "duration_ms": cls._f(+ gauges.get("queue.semantic.queue_wait.duration_ms"), 0.0+ )+ },+ "execute": {+ "duration_ms": cls._f(gauges.get("queue.semantic.execute.duration_ms"), 0.0)+ },},"embedding": {"processed": cls._i(gauges.get("queue.embedding.processed"), 0),"requeue_count": cls._i(gauges.get("queue.embedding.requeue_count"), 0),"error_count": cls._i(gauges.get("queue.embedding.error_count"), 0),+ "queue_wait": {+ "duration_ms": cls._f(+ gauges.get("queue.embedding.queue_wait.duration_ms"), 0.0+ )+ },+ "execute": {+ "duration_ms": cls._f(+ gauges.get("queue.embedding.execute.duration_ms"), 0.0+ )+ },},}@@ -393,27 +411,68 @@ class TelemetrySummaryBuilder:if cls._has_metric_prefix("resource", counters, gauges):summary["resource"] = {- "request": {- "duration_ms": cls._f(gauges.get("resource.request.duration_ms"), 0.0),+ "total": {+ "duration_ms": cls._f(gauges.get("resource.total.duration_ms"), 0.0),+ },+ "source_execute": {+ "duration_ms": cls._f(gauges.get("resource.source_execute.duration_ms"), 0.0),+ },+ "source_prepare": {+ "duration_ms": cls._f(gauges.get("resource.source_prepare.duration_ms"), 0.0),},- "process": {- "duration_ms": cls._f(gauges.get("resource.process.duration_ms"), 0.0),- "parse": {- "duration_ms": cls._f(gauges.get("resource.parse.duration_ms"), 0.0),- "warnings_count": cls._i(gauges.get("resource.parse.warnings_count"), 0),+ "parse_artifact": {+ "duration_ms": cls._f(gauges.get("resource.parse_artifact.duration_ms"), 0.0),+ },+ "target_resolve": {+ "duration_ms": cls._f(gauges.get("resource.target_resolve.duration_ms"), 0.0),+ },+ "update_plan": {+ "duration_ms": cls._f(gauges.get("resource.update_plan.duration_ms"), 0.0),+ "artifact_inventory": {+ "duration_ms": cls._f(+ gauges.get("resource.update_plan.artifact_inventory.duration_ms"), 0.0+ ),},- "finalize": {- "duration_ms": cls._f(gauges.get("resource.finalize.duration_ms"), 0.0),+ "rnfv_snapshot": {+ "duration_ms": cls._f(+ gauges.get("resource.update_plan.rnfv_snapshot.duration_ms"), 0.0+ ),},- "summarize": {- "duration_ms": cls._f(gauges.get("resource.summarize.duration_ms"), 0.0),+ "diff_and_compile": {+ "duration_ms": cls._f(+ gauges.get("resource.update_plan.diff_and_compile.duration_ms"), 0.0+ ),},},- "wait": {- "duration_ms": cls._f(gauges.get("resource.wait.duration_ms"), 0.0),+ "content_commit": {+ "duration_ms": cls._f(gauges.get("resource.content_commit.duration_ms"), 0.0),+ },+ "derived_enqueue": {+ "duration_ms": cls._f(gauges.get("resource.derived_enqueue.duration_ms"), 0.0),},- "watch": {- "duration_ms": cls._f(gauges.get("resource.watch.duration_ms"), 0.0),+ "semantic": {+ "queue_wait": {+ "duration_ms": cls._f(+ gauges.get("queue.semantic.queue_wait.duration_ms"), 0.0+ ),+ },+ "execute": {+ "duration_ms": cls._f(+ gauges.get("queue.semantic.execute.duration_ms"), 0.0+ ),+ },+ },+ "embedding": {+ "queue_wait": {+ "duration_ms": cls._f(+ gauges.get("queue.embedding.queue_wait.duration_ms"), 0.0+ ),+ },+ "execute": {+ "duration_ms": cls._f(+ gauges.get("queue.embedding.execute.duration_ms"), 0.0+ ),+ },},"flags": {public_key: cls._bool(gauges.get(metric_key), False)@@ -498,6 +557,10 @@ class OperationTelemetry:def set_value(self, key: str, value: Any) -> None:self.set(key, value)+ def elapsed_ms(self) -> float:+ """Return the current operation wall-clock duration without finishing it."""+ return max((time.perf_counter() - self._start_time) * 1000.0, 0.0)+@staticmethoddef _metric_bucket(value: Any, allowed: set[str]) -> str:normalized = str(value or "").strip().lower()diff --git a/openviking/telemetry/request_wait_tracker.py b/openviking/telemetry/request_wait_tracker.pyindex 1a27bd35e..85223d798 100644--- a/openviking/telemetry/request_wait_tracker.py+++ b/openviking/telemetry/request_wait_tracker.py@@ -26,6 +26,10 @@ class _RequestWaitState:embedding_requeue_count: int = 0embedding_error_count: int = 0embedding_errors: List[str] = field(default_factory=list)+ semantic_queue_wait_ms: float = 0.0+ semantic_execute_ms: float = 0.0+ embedding_queue_wait_ms: float = 0.0+ embedding_execute_ms: float = 0.0created_at: float = field(default_factory=time.time)retained_roots: Set[str] = field(default_factory=set)cleanup_requested: bool = False@@ -106,6 +110,48 @@ class RequestWaitTracker:returnstate.embedding_requeue_count += max(delta, 0)+ def record_semantic_timing(+ self, telemetry_id: str, *, queue_wait_ms: float = 0.0, execute_ms: float = 0.0+ ) -> None:+ self._record_timing(+ telemetry_id,+ queue_wait_ms=queue_wait_ms,+ execute_ms=execute_ms,+ queue_wait_attr="semantic_queue_wait_ms",+ execute_attr="semantic_execute_ms",+ )++ def record_embedding_timing(+ self, telemetry_id: str, *, queue_wait_ms: float = 0.0, execute_ms: float = 0.0+ ) -> None:+ self._record_timing(+ telemetry_id,+ queue_wait_ms=queue_wait_ms,+ execute_ms=execute_ms,+ queue_wait_attr="embedding_queue_wait_ms",+ execute_attr="embedding_execute_ms",+ )++ def _record_timing(+ self,+ telemetry_id: str,+ *,+ queue_wait_ms: float,+ execute_ms: float,+ queue_wait_attr: str,+ execute_attr: str,+ ) -> None:+ if not telemetry_id:+ return+ with self._lock:+ state = self._states.get(telemetry_id)+ if state is None:+ return+ setattr(+ state, queue_wait_attr, getattr(state, queue_wait_attr) + max(queue_wait_ms, 0.0)+ )+ setattr(state, execute_attr, getattr(state, execute_attr) + max(execute_ms, 0.0))+def get_embedding_context_count(self, telemetry_id: str) -> int:"""Return contexts successfully indexed for one request."""if not telemetry_id:@@ -114,6 +160,30 @@ class RequestWaitTracker:state = self._states.get(telemetry_id)return state.embedding_context_count if state is not None else 0+ def get_queue_timing(self, telemetry_id: str) -> Dict[str, Dict[str, float]]:+ if not telemetry_id:+ return {+ "semantic": {"queue_wait_ms": 0.0, "execute_ms": 0.0},+ "embedding": {"queue_wait_ms": 0.0, "execute_ms": 0.0},+ }+ with self._lock:+ state = self._states.get(telemetry_id)+ if state is None:+ return {+ "semantic": {"queue_wait_ms": 0.0, "execute_ms": 0.0},+ "embedding": {"queue_wait_ms": 0.0, "execute_ms": 0.0},+ }+ return {+ "semantic": {+ "queue_wait_ms": state.semantic_queue_wait_ms,+ "execute_ms": state.semantic_execute_ms,+ },+ "embedding": {+ "queue_wait_ms": state.embedding_queue_wait_ms,+ "execute_ms": state.embedding_execute_ms,+ },+ }+def mark_semantic_done(self,telemetry_id: str,diff --git a/openviking/telemetry/resource_summary.py b/openviking/telemetry/resource_summary.pyindex 90d19834e..3ba2a44af 100644--- a/openviking/telemetry/resource_summary.py+++ b/openviking/telemetry/resource_summary.py@@ -7,6 +7,7 @@ from __future__ import annotationsfrom typing import Any, Dictfrom .operation import OperationTelemetry+from .request_wait_tracker import get_request_wait_trackerdef _consume_semantic_request_stats(telemetry_id: str):@@ -27,11 +28,11 @@ def _consume_embedding_request_stats(telemetry_id: str):return None-def _consume_semantic_dag_stats(telemetry_id: str, root_uri: str | None):+def _consume_semantic_tree_stats(telemetry_id: str, root_uri: str | None):try:from openviking.storage.queuefs.semantic_processor import SemanticProcessor- return SemanticProcessor.consume_dag_stats(telemetry_id=telemetry_id, uri=root_uri)+ return SemanticProcessor.consume_tree_stats(telemetry_id=telemetry_id, uri=root_uri)except Exception:return None@@ -65,26 +66,32 @@ def record_resource_queue_metrics(telemetry_id: str,root_uri: str | None,) -> None:- """Apply queue and DAG metrics to a resource operation collector."""+ """Apply queue and semantic-tree metrics to a resource operation collector."""if not telemetry.enabled:returnsemantic = _queue_metrics(_consume_semantic_request_stats(telemetry_id))embedding = _queue_metrics(_consume_embedding_request_stats(telemetry_id))+ timing = get_request_wait_tracker().get_queue_timing(telemetry_id)telemetry.set("queue.semantic.processed", semantic["processed"])telemetry.set("queue.semantic.requeue_count", semantic["requeue_count"])telemetry.set("queue.semantic.error_count", semantic["error_count"])+ telemetry.set("queue.semantic.queue_wait.duration_ms", timing["semantic"]["queue_wait_ms"])+ telemetry.set("queue.semantic.execute.duration_ms", timing["semantic"]["execute_ms"])telemetry.set("queue.embedding.processed", embedding["processed"])telemetry.set("queue.embedding.requeue_count", embedding["requeue_count"])telemetry.set("queue.embedding.error_count", embedding["error_count"])+ telemetry.set("queue.embedding.queue_wait.duration_ms", timing["embedding"]["queue_wait_ms"])+ telemetry.set("queue.embedding.execute.duration_ms", timing["embedding"]["execute_ms"])++ tree_stats = _consume_semantic_tree_stats(telemetry_id, root_uri)+ if tree_stats is not None:+ telemetry.set("semantic_nodes.total", tree_stats.total_nodes)+ telemetry.set("semantic_nodes.done", tree_stats.done_nodes)+ telemetry.set("semantic_nodes.pending", tree_stats.pending_nodes)+ telemetry.set("semantic_nodes.running", tree_stats.in_progress_nodes)- dag_stats = _consume_semantic_dag_stats(telemetry_id, root_uri)- if dag_stats is not None:- telemetry.set("semantic_nodes.total", dag_stats.total_nodes)- telemetry.set("semantic_nodes.done", dag_stats.done_nodes)- telemetry.set("semantic_nodes.pending", dag_stats.pending_nodes)- telemetry.set("semantic_nodes.running", dag_stats.in_progress_nodes)__all__ = ["build_queue_status_payload",diff --git a/openviking/utils/content_hash.py b/openviking/utils/content_hash.pynew file mode 100644index 000000000..3660908d5--- /dev/null+++ b/openviking/utils/content_hash.py@@ -0,0 +1,17 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Single source of truth for the file-content fingerprint used by incremental diff.++The fingerprint is the md5 of the *final stored* file bytes for a resource URI+(after any encoding normalization or link rewriting). Incremental diff compares+this fingerprint on both sides to skip re-processing unchanged files. Every write+path that maintains the fingerprint MUST use :func:`content_md5` so the scheme+stays consistent; it is a content-equality hint, not a security digest.+"""++import hashlib+++def content_md5(data: bytes) -> str:+ """Return the hex md5 of the final stored file bytes."""+ return hashlib.md5(data, usedforsecurity=False).hexdigest()diff --git a/openviking/utils/embedding_utils.py b/openviking/utils/embedding_utils.pyindex 085f07a5b..318704b4c 100644--- a/openviking/utils/embedding_utils.py+++ b/openviking/utils/embedding_utils.py@@ -28,8 +28,10 @@ from openviking.server.identity import RequestContextfrom openviking.service.task_work_index import TaskWorkRejectedfrom openviking.storage.abstract_overview import body_for_preview, embedding_text_for_bodyfrom openviking.storage.acl import CreatorAclGrant+from openviking.storage.index_action import FieldPatch, IndexActionfrom openviking.storage.queuefs import get_queue_managerfrom openviking.storage.queuefs.embedding_msg_converter import EmbeddingMsgConverter+from openviking.storage.resource_rnfv import NON_PORTABLE_VECTOR_RECORD_FIELDSfrom openviking.storage.viking_fs import LS_ALL_NODES, get_viking_fsfrom openviking.telemetry.request_wait_tracker import get_request_wait_trackerfrom openviking.utils.embedding_input import truncate_embedding_input@@ -62,25 +64,44 @@ def _truncate_abstract_bytes(abstract: str) -> str:return encoded[:_ABSTRACT_MAX_BYTES].decode("utf-8", errors="ignore")-_PORTABLE_SCALAR_FIELDS = frozenset(- {- "type",- "level",- "name",- "description",- "tags",- "abstract",- }-)--def _apply_scalar_overrides(embedding_msg, overrides: Optional[Dict[str, Any]]) -> None:if not embedding_msg or not overrides:return- for field in _PORTABLE_SCALAR_FIELDS:- value = overrides.get(field)- if value is not None:- embedding_msg.context_data[field] = value+ record_id = overrides.get("_record_id")+ if record_id:+ # Internal queue metadata, removed by TextEmbeddingHandler before upsert.+ embedding_msg.context_data["_upsert_record_id"] = str(record_id)+ for field, value in overrides.items():+ if field.startswith("_") or field in NON_PORTABLE_VECTOR_RECORD_FIELDS or value is None:+ continue+ embedding_msg.context_data[field] = value+++def _apply_planned_field_patch(+ embedding_msg,+ field_patch: FieldPatch | None,+) -> None:+ """Attach a MERGE patch without overwriting the exact-get base prematurely."""++ if not embedding_msg or embedding_msg.action is not IndexAction.MERGE:+ return+ patch_values = {+ field: value+ for field, value in (field_patch.values if field_patch is not None else {}).items()+ if not field.startswith("_")+ and field not in NON_PORTABLE_VECTOR_RECORD_FIELDS+ and value is not None+ }+ for field in patch_values:+ embedding_msg.context_data.pop(field, None)+ embedding_msg.payload.field_patch = (+ FieldPatch(+ values=patch_values,+ modes=field_patch.modes if field_patch is not None else {},+ )+ if patch_values+ else None+ )def _apply_ingest_options(@@ -90,10 +111,20 @@ def _apply_ingest_options(ingest_options = IngestOptions.from_value(ingest_options)if not embedding_msg or ingest_options.search_tags is None:return- embedding_msg.context_data["search_tags"] = list(ingest_options.search_tags or [])- embedding_msg.context_data["_upsert_options"] = {- "search_tag_mode": ingest_options.search_tag_mode- }+ incoming_tags = list(ingest_options.search_tags or [])+ if ingest_options.search_tag_mode == "append" and (+ embedding_msg.action is IndexAction.MERGE+ or embedding_msg.context_data.get("_upsert_options", {}).get("partial_update") is False+ ):+ from openviking.utils.tags import merge_search_tags++ incoming_tags = merge_search_tags(+ embedding_msg.context_data.get("search_tags"), incoming_tags+ )+ embedding_msg.context_data["search_tags"] = incoming_tags+ embedding_msg.context_data.setdefault("_upsert_options", {})["search_tag_mode"] = (+ ingest_options.search_tag_mode+ )async def _enqueue_embedding_message(@@ -292,16 +323,21 @@ async def _resolve_resource_content_type(file_name: str,viking_fs: Any,ctx: Optional[RequestContext],+ file_content: Optional[bytes] = None,) -> Optional[ResourceContentType]:content_type = get_resource_content_type(file_name)if Path(file_name).suffix.lower() != ".ts":return content_typetry:- prefix = await viking_fs.read(- file_path,- offset=0,- size=MPEG_TS_PROBE_BYTES,- ctx=ctx,+ prefix = (+ file_content[:MPEG_TS_PROBE_BYTES]+ if file_content is not None+ else await viking_fs.read(+ file_path,+ offset=0,+ size=MPEG_TS_PROBE_BYTES,+ ctx=ctx,+ ))except Exception:return content_type@@ -368,7 +404,9 @@ async def vectorize_directory_meta(meta: Optional[Dict[str, Any]] = None,*,content_is_body: bool = False,-) -> None:+ actions: Optional[Dict[int, IndexAction | str]] = None,+ field_patches: Optional[Dict[int, FieldPatch]] = None,+) -> set[int]:"""Vectorize directory metadata (.abstract.md and .overview.md).@@ -383,10 +421,11 @@ async def vectorize_directory_meta(abstract = body_for_preview(abstract)overview = body_for_preview(overview)first_enqueue_error: Optional[Exception] = None+ enqueued_levels: set[int] = set()try:if not ctx:logger.warning("No context provided for vectorization")- return+ return enqueued_levelsqueue_manager = get_queue_manager()embedding_queue = queue_manager.get_queue(queue_manager.EMBEDDING)@@ -420,10 +459,24 @@ async def vectorize_directory_meta(context_abstract.set_vectorize(Vectorize(text=embedding_text_for_body(ContextLevel.ABSTRACT, uri, abstract)))- msg_abstract = EmbeddingMsgConverter.from_context(context_abstract, creator_acl_grant)+ msg_abstract = EmbeddingMsgConverter.from_context(+ context_abstract,+ creator_acl_grant,+ action=IndexAction(+ (actions or {}).get(+ int(ContextLevel.ABSTRACT.value),+ IndexAction.MERGE,+ )+ ),+ )+ level_overrides = (scalar_overrides or {}).get(int(ContextLevel.ABSTRACT.value))_apply_scalar_overrides(msg_abstract,- (scalar_overrides or {}).get(int(ContextLevel.ABSTRACT.value)),+ level_overrides,+ )+ _apply_planned_field_patch(+ msg_abstract,+ (field_patches or {}).get(int(ContextLevel.ABSTRACT.value)),)_apply_ingest_options(msg_abstract, ingest_options)if msg_abstract:@@ -434,10 +487,11 @@ async def vectorize_directory_meta(failure_message=f"Failed to enqueue directory L0 vector for {uri}",)if enqueued:+ enqueued_levels.add(int(ContextLevel.ABSTRACT.value))logger.debug(f"Enqueued directory L0 (abstract) for vectorization: {uri}")except TaskWorkRejected:logger.debug("Skipped directory vectorization for cancelling task: %s", uri)- return+ return enqueued_levelsexcept Exception as e:logger.error(f"Failed to enqueue directory L0 (abstract) for vectorization: {uri}: {e}",@@ -466,10 +520,24 @@ async def vectorize_directory_meta(context_overview.set_vectorize(Vectorize(text=embedding_text_for_body(ContextLevel.OVERVIEW, uri, overview)))- msg_overview = EmbeddingMsgConverter.from_context(context_overview, creator_acl_grant)+ msg_overview = EmbeddingMsgConverter.from_context(+ context_overview,+ creator_acl_grant,+ action=IndexAction(+ (actions or {}).get(+ int(ContextLevel.OVERVIEW.value),+ IndexAction.MERGE,+ )+ ),+ )+ level_overrides = (scalar_overrides or {}).get(int(ContextLevel.OVERVIEW.value))_apply_scalar_overrides(msg_overview,- (scalar_overrides or {}).get(int(ContextLevel.OVERVIEW.value)),+ level_overrides,+ )+ _apply_planned_field_patch(+ msg_overview,+ (field_patches or {}).get(int(ContextLevel.OVERVIEW.value)),)_apply_ingest_options(msg_overview, ingest_options)if msg_overview:@@ -480,10 +548,11 @@ async def vectorize_directory_meta(failure_message=f"Failed to enqueue directory L1 vector for {uri}",)if enqueued:+ enqueued_levels.add(int(ContextLevel.OVERVIEW.value))logger.debug(f"Enqueued directory L1 (overview) for vectorization: {uri}")except TaskWorkRejected:logger.debug("Skipped directory vectorization for cancelling task: %s", uri)- return+ return enqueued_levelsexcept Exception as e:logger.error(f"Failed to enqueue directory L1 (overview) for vectorization: {uri}: {e}",@@ -499,6 +568,7 @@ async def vectorize_directory_meta(exc_info=True,)raise+ return enqueued_levelsasync def vectorize_file(@@ -510,8 +580,12 @@ async def vectorize_file(use_summary: bool = False,preserve_existing_created_at: bool = False,scalar_override: Optional[Dict[str, Any]] = None,+ field_patch: FieldPatch | None = None,ingest_options: IngestOptions | None = None,creator_acl_grant: CreatorAclGrant | None = None,+ file_md5: Optional[str] = None,+ file_content: Optional[bytes] = None,+ action: str = "merge",) -> bool:"""Vectorize a single file.@@ -553,7 +627,9 @@ async def vectorize_file(owner_space=owner_space_for_uri(file_path),)- content_type = await _resolve_resource_content_type(file_path, file_name, viking_fs, ctx)+ content_type = await _resolve_resource_content_type(+ file_path, file_name, viking_fs, ctx, file_content=file_content+ )embedding_cfg = get_openviking_config().embeddingconfigured_text_source = embedding_cfg.text_sourceeffective_text_source = TEXT_SOURCE_SUMMARY_FIRST if use_summary else configured_text_source@@ -570,7 +646,11 @@ async def vectorize_file()context.set_vectorize(Vectorize(text=summary))elif is_text_file(file_name):- content = _coerce_text_file_content(await viking_fs.read_file(file_path, ctx=ctx))+ content = _coerce_text_file_content(+ file_content+ if file_content is not None+ else await viking_fs.read_file(file_path, ctx=ctx)+ )embedding_text = truncate_embedding_input(content,embedding_cfg.max_input_tokens,@@ -588,7 +668,9 @@ async def vectorize_file(else:try:content = _coerce_text_file_content(- await viking_fs.read_file(file_path, ctx=ctx)+ file_content+ if file_content is not None+ else await viking_fs.read_file(file_path, ctx=ctx))except Exception as e:logger.warning(@@ -627,12 +709,28 @@ async def vectorize_file(logger.debug(f"Skipping file {file_path} (no text content or summary)")return False- embedding_msg = EmbeddingMsgConverter.from_context(context, creator_acl_grant)+ # md5 fingerprints the final stored bytes so incremental diff can skip+ # unchanged files. It is supplied by the upload site that already holds+ # those bytes (parser output store / diff apply); vectorize_file never+ # reads the file back just to hash it. Unknown (None) leaves md5 empty and+ # diff falls back to comparing bytes.+ if file_md5:+ context.md5 = file_md5++ resolved_action = IndexAction(action)+ if resolved_action not in {IndexAction.UPSERT, IndexAction.MERGE}:+ raise ValueError(f"vectorize_file only supports upsert or merge actions: {action}")+ embedding_msg = EmbeddingMsgConverter.from_context(+ context,+ creator_acl_grant,+ action=resolved_action,+ )if not embedding_msg:return False- _apply_scalar_overrides(embedding_msg, scalar_override)_apply_ingest_options(embedding_msg, ingest_options)+ _apply_scalar_overrides(embedding_msg, scalar_override)+ _apply_planned_field_patch(embedding_msg, field_patch)enqueued = await _enqueue_embedding_message(embedding_queue,embedding_msg,diff --git a/openviking/utils/log_correlation.py b/openviking/utils/log_correlation.pynew file mode 100644index 000000000..ad9091789--- /dev/null+++ b/openviking/utils/log_correlation.py@@ -0,0 +1,21 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""Stable correlation fields for logs emitted by durable queue work."""++from openviking.service.task_work_index import get_task_context+from openviking.telemetry import get_current_telemetry+++def log_correlation(*, task_id: str = "", telemetry_id: str = "", message_id: str = "") -> str:+ """Return searchable task, telemetry, and optional queue-message IDs."""+ task_context = get_task_context()+ task_id = task_id or (task_context.task_id if task_context is not None else "-")+ if not telemetry_id:+ telemetry = get_current_telemetry()+ telemetry_id = telemetry.telemetry_id if telemetry.enabled else "-"+ fields = f"task_id={task_id} telemetry_id={telemetry_id or '-'}"+ return f"{fields} message_id={message_id}" if message_id else fields+++__all__ = ["log_correlation"]diff --git a/openviking/utils/media_processor.py b/openviking/utils/media_processor.pyindex 989775309..2912ec136 100644--- a/openviking/utils/media_processor.py+++ b/openviking/utils/media_processor.py@@ -2,10 +2,13 @@# SPDX-License-Identifier: AGPL-3.0"""Unified resource processor with strategy-based routing."""+import time+from collections.abc import Awaitablefrom pathlib import Pathfrom typing import TYPE_CHECKING, Optionalfrom urllib.parse import urlparse+from openviking.metrics.datasources.resource import ResourceIngestionEventDataSourcefrom openviking.parse.accessors.base import LocalResource, SourceTypefrom openviking.parse.accessors.mime_types import IANA_MEDIA_TYPE_TO_EXTENSIONfrom openviking.parse.backend import ParserBackend, normalize_parser_backend@@ -25,6 +28,7 @@ from openviking.server.local_input_guard import (is_remote_resource_source,looks_like_local_path,)+from openviking.telemetry import get_current_telemetryfrom openviking_cli.exceptions import InvalidArgumentError, PermissionDeniedErrorfrom openviking_cli.utils.logger import get_logger@@ -287,6 +291,25 @@ class UnifiedResourceProcessor:- Directories needed for TreeBuilder are preserved via ParseResult.temp_dir_path"""+ metrics_account_id = kwargs.pop("_metrics_account_id", None)++ async def run_stage(stage: str, operation: Awaitable[ParseResult | LocalResource]):+ started_at = time.perf_counter()+ status = "ok"+ try:+ with get_current_telemetry().measure(f"resource.{stage}"):+ return await operation+ except Exception:+ status = "error"+ raise+ finally:+ ResourceIngestionEventDataSource.record_stage(+ stage=stage,+ status=status,+ duration_seconds=time.perf_counter() - started_at,+ account_id=metrics_account_id,+ )+mode = normalize_parse_mode(parse_mode)# First check if source is raw content (not URL/path)@@ -295,10 +318,13 @@ class UnifiedResourceProcessor:)if not is_potential_path and not self._is_url(source):# Treat as raw content- return await parse(- source,- instruction=instruction,- split_content=mode is ParseMode.DEFAULT,+ return await run_stage(+ "parse_artifact",+ parse(+ source,+ instruction=instruction,+ split_content=mode is ParseMode.DEFAULT,+ ),)if (@@ -323,7 +349,9 @@ class UnifiedResourceProcessor:explicit_name = kwargs.get("resource_name") or kwargs.get("source_name")if explicit_name:parse_kwargs["resource_name"] = _smart_stem(explicit_name)- return await self._get_parser_router().parse(source, **parse_kwargs)+ return await run_stage(+ "parse_artifact", self._get_parser_router().parse(source, **parse_kwargs)+ )if (mode is ParseMode.DEFAULT@@ -345,16 +373,24 @@ class UnifiedResourceProcessor:explicit_name = kwargs.get("resource_name") or kwargs.get("source_name")if explicit_name:parse_kwargs["resource_name"] = _smart_stem(explicit_name)- return await self._get_parser_router().parse(source, **parse_kwargs)+ return await run_stage(+ "parse_artifact", self._get_parser_router().parse(source, **parse_kwargs)+ )# Phase 1: Accessor - get local resource. A caller may prepare a remote# resource first so async routing uses the same detected file type.- local_resource = prepared_resource or await self.prepare(- source,- allow_local_path_resolution=allow_local_path_resolution,- **({"parse_mode": mode} if mode is ParseMode.NO_SPLIT else {}),- **kwargs,- )+ if prepared_resource is not None:+ local_resource = prepared_resource+ else:+ local_resource = await run_stage(+ "source_prepare",+ self.prepare(+ source,+ allow_local_path_resolution=allow_local_path_resolution,+ **({"parse_mode": mode} if mode is ParseMode.NO_SPLIT else {}),+ **kwargs,+ ),+ )# Use context manager for automatic cleanup, but preserve directories for TreeBuildertry:@@ -410,7 +446,9 @@ class UnifiedResourceProcessor:parser = DirectoryParser()parse_kwargs["_feishu_import_plan"] = local_resource.feishu_plan- result = await parser.parse(str(local_resource.path), **parse_kwargs)+ result = await run_stage(+ "parse_artifact", parser.parse(str(local_resource.path), **parse_kwargs)+ )# Preserve temporary directory for TreeBuilderif local_resource.is_temporary and not result.temp_dir_path:result.temp_dir_path = str(local_resource.path)@@ -438,7 +476,9 @@ class UnifiedResourceProcessor:await save("file", response_id)parse_kwargs["_response_checkpoint"] = record- return await parser_router.parse(local_resource, **parse_kwargs)+ return await run_stage(+ "parse_artifact", parser_router.parse(local_resource, **parse_kwargs)+ )finally:# Clean up temporary resources unless they need to be preservedlocal_resource.cleanup()diff --git a/openviking/utils/resource_processor.py b/openviking/utils/resource_processor.pyindex 866c330b4..aa811c8eb 100644--- a/openviking/utils/resource_processor.py+++ b/openviking/utils/resource_processor.py@@ -29,8 +29,10 @@ from openviking.server.identity import RequestContextfrom openviking.storage.acl import AclAction, CreatorAclGrantfrom openviking.storage.errors import LockAcquisitionErrorfrom openviking.storage.expr import And, Eq, PathScope+from openviking.storage.index_action import FieldPatchfrom openviking.storage.internal_names import STORAGE_INTERNAL_ENTRY_NAMESfrom openviking.storage.queuefs.semantic_processor import SemanticProcessor+from openviking.storage.resource_rnfv import RequestIntentfrom openviking.storage.viking_fs import LS_ALL_NODES, get_viking_fsfrom openviking.storage.vikingdb_manager import VikingDBManagerfrom openviking.telemetry import get_current_telemetry@@ -38,6 +40,7 @@ from openviking.utils import is_github_urlfrom openviking.utils.embedding_utils import index_resource, vectorize_filefrom openviking.utils.git_auth import is_git_https_urlfrom openviking.utils.ingest_options import IngestOptions+from openviking.utils.log_correlation import log_correlationfrom openviking.utils.summarizer import Summarizerfrom openviking_cli.exceptions import OpenVikingErrorfrom openviking_cli.utils import VikingURI, get_logger@@ -53,6 +56,28 @@ _MAX_FILE_VECTORIZATION_CONCURRENCY = 64VECTORDB_MAX_QUERY_LIMIT = 100_000+class _DocRelStore:+ """Adapt a parse output store so commit reads use target-relative paths.++ Resource plans key files relative to the target root, while the underlying+ artifact may store them below ``<doc_rel>/...``.+ """++ def __init__(self, store: Any, doc_rel: str) -> None:+ self._store = store+ base = (doc_rel or "").strip("/")+ self._base = base++ async def read_bytes(self, ref: Any, rel_path: str) -> bytes:+ normalized = str(rel_path or "").strip("/")+ artifact_rel = (+ normalized+ if not self._base or normalized == self._base or normalized.startswith(self._base + "/")+ else f"{self._base}/{normalized}"+ )+ return await self._store.read_bytes(ref, artifact_rel)++class ResourceProcessor:"""Handles coordinated write operations.@@ -137,6 +162,329 @@ class ResourceProcessor:)return self._media_processor+ def _build_parse_output_store(self):+ """Return the configured parse output store, or None for AGFS mode.++ Local artifacts are consumed and committed to formal AGFS storage in+ this synchronous request phase. They never cross an asynchronous queue+ boundary, so parse and commit only need to run in the same process.+ AGFS (the default) returns None to preserve its temp-tree path.+ """+ try:+ parse_output = get_openviking_config().storage.parse_output+ except Exception:+ return None+ if getattr(parse_output, "mode", "agfs") != "local":+ return None+ from openviking.parse.output import build_parse_output_store++ return build_parse_output_store(+ backend="local", local_root=parse_output.resolved_local_root()+ )++ @staticmethod+ def _artifact_doc_rel(artifact_ref: Any, temp_doc_uri: Optional[str]) -> str:+ """Return the document root's path relative to the artifact root.++ ``temp_doc_uri`` is ``<artifact_root>/<doc_rel>`` (finalize built it), so+ the relative document path is the suffix after the artifact root.+ """+ root = artifact_ref.root.rstrip("/")+ doc = str(temp_doc_uri or "").rstrip("/")+ if doc.startswith(f"{root}/"):+ return doc[len(root) + 1 :]+ return str(getattr(artifact_ref, "resource_rel", "") or "").strip("/")++ @staticmethod+ def _ensure_parse_artifact_ref(parse_result: Any) -> Any:+ """Resolve an artifact ref for every supported parser result."""+ artifact_ref = getattr(parse_result, "artifact_ref", None)+ ensure = getattr(parse_result, "ensure_artifact_ref", None)+ if callable(ensure):+ artifact_ref = ensure()+ if artifact_ref is not None:+ return artifact_ref+ temp_dir_path = str(getattr(parse_result, "temp_dir_path", "") or "")+ if not temp_dir_path.startswith("viking://temp/"):+ raise RuntimeError("add_resources requires a parse artifact")+ from openviking.parse.output import ParseArtifactRef++ return ParseArtifactRef(backend="agfs", root=temp_dir_path, root_type="dir")++ @staticmethod+ async def _cleanup_parse_result_artifact(+ parse_result: Any, *, output_store: Any, viking_fs: Any, ctx: RequestContext+ ) -> None:+ artifact_ref = getattr(parse_result, "artifact_ref", None)+ if artifact_ref is not None:+ from openviking.parse.output import store_for_artifact_ref++ store = (+ output_store+ if output_store is not None and output_store.backend == artifact_ref.backend+ else store_for_artifact_ref(artifact_ref, viking_fs=viking_fs, ctx=ctx)+ )+ await store.cleanup(artifact_ref)+ elif parse_result.temp_dir_path:+ await viking_fs.delete_temp(parse_result.temp_dir_path, ctx=ctx)++ @staticmethod+ def _store_for_parse_artifact(+ artifact_ref: Any, *, output_store: Any, viking_fs: Any, ctx: RequestContext+ ) -> Any:+ if output_store is not None and output_store.backend == artifact_ref.backend:+ return output_store+ from openviking.parse.output import store_for_artifact_ref++ return store_for_artifact_ref(artifact_ref, viking_fs=viking_fs, ctx=ctx)++ async def _commit_directory_artifact_with_plan(+ self,+ *,+ output_store: Any,+ artifact_ref: Any,+ doc_rel: str,+ root_uri: str,+ target_preexisting: bool,+ root_is_file: bool = False,+ ctx: RequestContext,+ lease_ref: Optional[Dict[str, Any]],+ vectorize: bool,+ summarize: bool,+ processing_mode: ProcessingMode,+ is_code_repo: bool,+ ingest_options: IngestOptions,+ source_metadata: Optional[Dict[str, str]],+ ) -> Any:+ """Resolve and commit one artifact through the canonical update plan."""+ from openviking.metrics.datasources.resource import ResourceIngestionEventDataSource+ from openviking.storage.context_update_plan import (+ build_context_update_plan_from_snapshot,+ execute_content_tree_actions,+ )+ from openviking.storage.resource_diff import (+ build_rnfv_snapshot,+ count_tree_entry_kinds,+ prepare_artifact_inventory,+ )+ from openviking.storage.resource_target import AgfsResourceTarget++ target = AgfsResourceTarget(+ viking_fs=get_viking_fs(),+ root_uri=root_uri,+ ctx=ctx,+ lease_ref=lease_ref,+ )+ telemetry = get_current_telemetry()+ artifact_backend = str(getattr(artifact_ref, "backend", "unknown"))+ update_plan_started_at = time.perf_counter()+ try:+ with telemetry.measure("resource.update_plan.artifact_inventory"):+ artifact_inventory = await prepare_artifact_inventory(+ output_store,+ artifact_ref,+ doc_rel=doc_rel,+ target_root_uri=root_uri,+ root_is_file=root_is_file,+ )+ plan_processing_mode = (+ processing_mode+ if processing_mode == VECTORS_ONLY or summarize or vectorize+ else VECTORS_ONLY+ )+ request = RequestIntent.from_ingest_options(+ target_uri=root_uri,+ processing_mode=plan_processing_mode,+ ingest_options=ingest_options,+ vectorize=vectorize,+ )+ with telemetry.measure("resource.update_plan.rnfv_snapshot"):+ rnfv = await build_rnfv_snapshot(+ viking_fs=get_viking_fs(),+ vikingdb=self.vikingdb,+ store=output_store,+ artifact_ref=artifact_ref,+ target_uri=root_uri,+ ctx=ctx,+ doc_rel=doc_rel,+ request_intent=request,+ target_preexisting=target_preexisting,+ artifact_inventory=artifact_inventory,+ root_is_file=root_is_file,+ )+ from collections import Counter++ n_files, n_dirs, _ = count_tree_entry_kinds(rnfv.new.entries)+ f_files, f_dirs, _ = count_tree_entry_kinds(rnfv.formal.entries)+ logger.debug(+ "[RNFVSnapshot] %s target=%s artifact_backend=%s root_is_file=%s "+ "target_preexisting=%s n_files=%d n_dirs=%d f_files=%d f_dirs=%d "+ "v_records=%d v_levels=%s",+ log_correlation(),+ root_uri,+ artifact_backend,+ root_is_file,+ target_preexisting,+ n_files,+ n_dirs,+ f_files,+ f_dirs,+ len(rnfv.vectors.records_by_id),+ dict(Counter(record.level for record in rnfv.vectors.records_by_id.values())),+ )+ with telemetry.measure("resource.update_plan.diff_and_compile"):+ diff, context_plan = await build_context_update_plan_from_snapshot(+ snapshot=rnfv,+ store=_DocRelStore(output_store, doc_rel),+ artifact_ref=artifact_ref,+ target=target,+ vikingdb=self.vikingdb,+ context_type=context_type_for_uri(root_uri),+ is_code_repo=is_code_repo,+ account_id=ctx.account_id,+ ctx=ctx,+ root_preexisting=target_preexisting,+ artifact_paths=artifact_inventory.artifact_paths,+ ingest_options=ingest_options,+ source_metadata=source_metadata,+ root_is_file=root_is_file,+ )+ except Exception:+ telemetry.set(+ "resource.update_plan.duration_ms",+ (time.perf_counter() - update_plan_started_at) * 1000.0,+ )+ ResourceIngestionEventDataSource.record_stage(+ stage="update_plan",+ status="error",+ duration_seconds=time.perf_counter() - update_plan_started_at,+ account_id=ctx.account_id,+ )+ raise+ telemetry.set(+ "resource.update_plan.duration_ms",+ (time.perf_counter() - update_plan_started_at) * 1000.0,+ )+ ResourceIngestionEventDataSource.record_stage(+ stage="update_plan",+ status="ok",+ duration_seconds=time.perf_counter() - update_plan_started_at,+ account_id=ctx.account_id,+ )++ content_commit_started_at = time.perf_counter()+ try:+ with telemetry.measure("resource.content_commit"):+ await execute_content_tree_actions(+ context_plan.content_tree_actions,+ store=_DocRelStore(output_store, doc_rel),+ artifact_ref=artifact_ref,+ target=target,+ )+ except Exception as exc:+ ResourceIngestionEventDataSource.record_stage(+ stage="content_commit",+ status="error",+ duration_seconds=time.perf_counter() - content_commit_started_at,+ account_id=ctx.account_id,+ )+ logger.exception(+ "[ContentTreeCommitFailed] %s target=%s artifact_backend=%s actions=%d error=%s",+ log_correlation(),+ root_uri,+ artifact_backend,+ len(context_plan.content_tree_actions),+ exc,+ )+ raise+ ResourceIngestionEventDataSource.record_stage(+ stage="content_commit",+ status="ok",+ duration_seconds=time.perf_counter() - content_commit_started_at,+ account_id=ctx.account_id,+ )+ self._log_content_tree_commit(+ root_uri=root_uri,+ artifact_backend=artifact_backend,+ actions=context_plan.content_tree_actions,+ duration_ms=(time.perf_counter() - content_commit_started_at) * 1000.0,+ )+ self._log_context_update_plan(context_plan)+ self._log_context_commit_summary(+ diff,+ content_actions=context_plan.content_tree_actions,+ root_uri=root_uri,+ is_initial=not target_preexisting,+ )+ return context_plan.after_content_commit()++ @staticmethod+ def _log_content_tree_commit(+ *, root_uri: str, artifact_backend: str, actions: Any, duration_ms: float+ ) -> None:+ log = logger.info if actions else logger.debug+ log(+ "[ContentTreeCommit] %s target=%s artifact_backend=%s uploaded_files=%d "+ "created_dirs=%d deleted_paths=%d replaced_kinds=%d duration_ms=%.3f",+ log_correlation(),+ root_uri,+ artifact_backend,+ sum(action.new_kind == "file" for action in actions),+ sum(action.new_kind == "directory" for action in actions),+ sum(action.operation.value == "delete" for action in actions),+ sum(action.operation.value == "replace_kind" for action in actions),+ duration_ms,+ )++ @staticmethod+ def _log_context_update_plan(plan: Any) -> None:+ from collections import Counter++ content = Counter(action.operation.value for action in plan.content_tree_actions)+ direct = Counter(action.action.value for action in plan.direct_index_actions)+ entries = plan.semantic_plan.tree.entries if plan.semantic_plan is not None else ()+ semantic = Counter(entry.semantic_action.value for entry in entries)+ slots = Counter(slot.action.value for entry in entries for slot in entry.index_slots)+ logger.debug(+ "[ContextUpdatePlan] %s root=%s content=%s semantic=%s direct_index=%s "+ "index_slots=%s "+ "semantic_entries=%d execution_roots=%d",+ log_correlation(),+ plan.root_uri,+ dict(content),+ dict(semantic),+ dict(direct),+ dict(slots),+ len(entries),+ len(plan.semantic_plan.execution_root_uris()) if plan.semantic_plan is not None else 0,+ )++ @staticmethod+ def _log_context_commit_summary(+ diff: Any, *, content_actions: Any, root_uri: str, is_initial: bool+ ) -> None:+ from collections import Counter++ states = Counter(entry.content_state.value for entry in diff.entries.values())+ uploaded_files = sum(action.new_kind == "file" for action in content_actions)+ created_dirs = sum(action.new_kind == "directory" for action in content_actions)+ if is_initial:+ logger.debug(+ "[add_resource] %s initial import committed root=%s files=%d dirs=%d",+ log_correlation(),+ root_uri,+ uploaded_files,+ created_dirs,+ )+ return+ logger.debug(+ "[add_resource] %s incremental diff committed root=%s states=%s uploaded=%d",+ log_correlation(),+ root_uri,+ dict(states),+ uploaded_files,+ )+@staticmethoddef _empty_directory_error(meta: Dict[str, Any]) -> str:"""Build a bounded error message for a directory with no successful files."""@@ -155,6 +503,38 @@ class ResourceProcessor:else:message = "Directory import produced no content: no processable files were selected"+ details = ResourceProcessor._failed_file_details(failures)+ if details:+ message += "; failed files: " + "; ".join(details)+ if len(failures) > len(details):+ message += f"; ... {len(failures) - len(details)} more"+ return message++ @staticmethod+ def _directory_parse_failures(meta: Dict[str, Any]) -> List[Dict[str, Any]]:+ """Return files selected for parsing whose parser did not produce content."""+ failed_files = meta.get("failed_files")+ if not isinstance(failed_files, list):+ return []+ return [+ item+ for item in failed_files+ if isinstance(item, dict)+ and (item.get("status") == "failed" or (not item.get("status") and "error" in item))+ ]++ @staticmethod+ def _incomplete_directory_error(failures: List[Dict[str, Any]]) -> str:+ message = f"Directory import incomplete: {len(failures)} file(s) failed to parse"+ details = ResourceProcessor._failed_file_details(failures)+ if details:+ message += "; failed files: " + "; ".join(details)+ if len(failures) > len(details):+ message += f"; ... {len(failures) - len(details)} more"+ return message++ @staticmethod+ def _failed_file_details(failures: List[Dict[str, Any]]) -> List[str]:details: List[str] = []for item in failures[:5]:if not isinstance(item, dict):@@ -167,11 +547,7 @@ class ResourceProcessor:if remote_ids:path = f"{path} ({', '.join(remote_ids)})"details.append(f"{path}: {reason[:120]}")- if details:- message += "; failed files: " + "; ".join(details)- if len(failures) > len(details):- message += f"; ... {len(failures) - len(details)} more"- return message+ return detailsasync def prepare_durable_source(self,@@ -270,6 +646,7 @@ class ResourceProcessor:ingest_options = IngestOptions.from_value(kwargs.pop("ingest_options", None))to_is_directory = bool(kwargs.pop("to_is_directory", False))telemetry = get_current_telemetry()+ metrics_account_id = getattr(ctx, "account_id", None)async def _set_stage(stage: str) -> None:if stage_callback is None:@@ -278,21 +655,25 @@ class ResourceProcessor:if inspect.isawaitable(result):await result- with telemetry.measure("resource.process"):+ with telemetry.measure("resource.source_execute"):# ============ Phase 1: Parse source and writes to temp viking fs ============try:from openviking.metrics.datasources.resource import (ResourceIngestionEventDataSource,)- parse_start = time.perf_counter()- stage_start = time.perf_counter()- stage_status = "ok"media_processor = self._get_media_processor()viking_fs = get_viking_fs()# Use reason as instruction fallback so it influences L0/L1# generation and improves search relevance as documented.effective_instruction = instruction or reason+ # Local artifact mode writes parse output to this worker's disk+ # instead of AGFS temp. The same synchronous request commits all+ # required bytes to formal AGFS before any async queue handoff.+ # The store is task-scoped; default AGFS mode leaves kwargs untouched.+ output_store = self._build_parse_output_store()+ if output_store is not None:+ kwargs.setdefault("parse_output_store", output_store)if path.startswith(("http://", "https://", "git@", "ssh://", "git://")):await _set_stage("fetching")else:@@ -303,6 +684,7 @@ class ResourceProcessor:source=path,instruction=effective_instruction,prepared_resource=prepared_resource,+ _metrics_account_id=metrics_account_id,**kwargs,)result["source_path"] = parse_result.source_path or path@@ -316,7 +698,6 @@ class ResourceProcessor:result["errors"].extend(parse_result.warnings or ["Parse failed: no content generated"],)- stage_status = "error"return resultparse_meta = parse_result.meta if isinstance(parse_result.meta, dict) else {}@@ -333,27 +714,37 @@ class ResourceProcessor:result["status"] = "error"result["errors"].append(self._empty_directory_error(parse_meta))try:- await viking_fs.delete_temp(parse_result.temp_dir_path, ctx=ctx)+ await self._cleanup_parse_result_artifact(+ parse_result, output_store=output_store, viking_fs=viking_fs, ctx=ctx+ )except Exception as exc:logger.warning("[ResourceProcessor] Failed to clean empty directory temp %s: %s",parse_result.temp_dir_path,exc,)- stage_status = "error"+ return result++ parse_failures = self._directory_parse_failures(parse_meta)+ if is_directory_aggregate and parse_failures:+ result["status"] = "error"+ result["errors"].append(self._incomplete_directory_error(parse_failures))+ try:+ await self._cleanup_parse_result_artifact(+ parse_result, output_store=output_store, viking_fs=viking_fs, ctx=ctx+ )+ except Exception as exc:+ logger.warning(+ "[ResourceProcessor] Failed to clean incomplete directory temp %s: %s",+ parse_result.temp_dir_path,+ exc,+ )return resultif parse_result.warnings and kwargs.get("strict", False):result.setdefault("warnings", []).extend(parse_result.warnings)- telemetry.set(- "resource.parse.duration_ms",- round((time.perf_counter() - parse_start) * 1000, 3),- )- telemetry.set("resource.parse.warnings_count", len(parse_result.warnings or []))-except OpenVikingError:- stage_status = "error"raiseexcept Exception as e:result["status"] = "error"@@ -363,22 +754,11 @@ class ResourceProcessor:error_message += f" (response_id={error_meta['response_id']})"result["errors"].append(error_message)logger.error(f"[ResourceProcessor] Parse error: {e}")- telemetry.set_error("resource_processor.parse", "PROCESSING_ERROR", str(e))+ telemetry.set_error("resource_processor.parse_artifact", "PROCESSING_ERROR", str(e))import tracebacktraceback.print_exc()- stage_status = "error"return result- finally:- try:- ResourceIngestionEventDataSource.record_stage(- stage="parse",- status=str(stage_status),- duration_seconds=float(time.perf_counter() - stage_start),- account_id=getattr(ctx, "account_id", None),- )- except Exception:- pass# parse_result contains:# - root: ResourceNode tree (with L0/L1 in meta)@@ -387,10 +767,21 @@ class ResourceProcessor:# ============ Phase 3: TreeBuilder finalizes from temp (scan + move to AGFS) ============try:- await _set_stage("finalizing")+ await _set_stage("target_resolve")stage_start = time.perf_counter()stage_status = "ok"finalize_start = time.perf_counter()+ artifact_ref = getattr(parse_result, "artifact_ref", None)+ artifact_output_store = (+ self._store_for_parse_artifact(+ artifact_ref,+ output_store=output_store,+ viking_fs=viking_fs,+ ctx=ctx,+ )+ if artifact_ref is not None+ else None+ )with get_viking_fs().bind_request_context(ctx):context_tree = await self.tree_builder.finalize_from_temp(temp_dir_path=parse_result.temp_dir_path,@@ -407,25 +798,31 @@ class ResourceProcessor:and parse_result.source_format not in {"directory", "repository"}and not to_is_directory),+ artifact_ref=artifact_ref,+ output_store=artifact_output_store,)if context_tree and context_tree.root:result["root_uri"] = context_tree.root.uriresult["temp_uri"] = context_tree.root.temp_uriroot_is_file = bool(getattr(context_tree, "_root_is_file", False))telemetry.set(- "resource.finalize.duration_ms",+ "resource.target_resolve.duration_ms",round((time.perf_counter() - finalize_start) * 1000, 3),)except Exception as e:result["status"] = "error"result["errors"].append(f"Finalize from temp error: {e}")- telemetry.set_error("resource_processor.finalize", "PROCESSING_ERROR", str(e))+ telemetry.set_error("resource_processor.target_resolve", "PROCESSING_ERROR", str(e))stage_status = "error"- # Cleanup temporary directory on error (via VikingFS)+ # Cleanup the parser-owned artifact through its own backend.try:- if parse_result.temp_dir_path:- await get_viking_fs().delete_temp(parse_result.temp_dir_path, ctx=ctx)+ await self._cleanup_parse_result_artifact(+ parse_result,+ output_store=output_store,+ viking_fs=get_viking_fs(),+ ctx=ctx,+ )except Exception:pass@@ -433,7 +830,7 @@ class ResourceProcessor:finally:try:ResourceIngestionEventDataSource.record_stage(- stage="finalize",+ stage="target_resolve",status=str(stage_status),duration_seconds=float(time.perf_counter() - stage_start),account_id=getattr(ctx, "account_id", None),@@ -449,10 +846,11 @@ class ResourceProcessor:resource_lock: Optional[Dict[str, Any]] = preacquired_locktarget_preexisting = Falsesource_committed = False+ local_artifact_doc_rel = ""+ incremental_noop = False+ context_update_plan = Noneif root_uri and temp_uri:- stage_start = time.perf_counter()- stage_status = "ok"viking_fs = get_viking_fs()try:if candidate_uri:@@ -499,56 +897,83 @@ class ResourceProcessor:uri=root_uri,root_is_file=root_is_file,)- if not target_preexisting:- await viking_fs.persist_temp_tree(- temp_uri,- root_uri,- ctx=ctx,- lease_ref=resource_lock,- )- if not root_is_file:- await rewrite_image_uris(- root_uri,- ctx=ctx,- lease_ref=resource_lock,- )- await viking_fs.delete_temp(- parse_result.temp_dir_path,- ctx=ctx,- )- temp_uri = root_uri- source_committed = True+ artifact_ref = self._ensure_parse_artifact_ref(parse_result)+ artifact_store = self._store_for_parse_artifact(+ artifact_ref, output_store=output_store, viking_fs=viking_fs, ctx=ctx+ )+ local_artifact_doc_rel = self._artifact_doc_rel(artifact_ref, temp_uri)+ context_update_plan = await self._commit_directory_artifact_with_plan(+ output_store=artifact_store,+ artifact_ref=artifact_ref,+ doc_rel=local_artifact_doc_rel,+ root_uri=root_uri,+ target_preexisting=target_preexisting,+ root_is_file=root_is_file,+ ctx=ctx,+ lease_ref=resource_lock,+ vectorize=bool(kwargs.get("build_index", True)),+ summarize=summarize,+ processing_mode=normalize_processing_mode(kwargs.get("processing_mode")),+ ingest_options=ingest_options,+ is_code_repo=parse_result.source_format == "repository",+ source_metadata=self._semantic_source_metadata(+ path=path,+ prepared_resource=prepared_resource,+ source_format=parse_result.source_format,+ ),+ )+ incremental_noop = target_preexisting and context_update_plan.is_noop()+ temp_uri = root_uri+ source_committed = Trueexcept Exception:- stage_status = "error"# Mirror the Phase 3 (finalize) on-error cleanup: a lock or# persist failure here would otherwise orphan the# viking://temp tree with no GC (#2478). Skip when the temp# tree was already persisted + deleted on the success path.- if not source_committed and parse_result.temp_dir_path:+ if not source_committed:try:- await get_viking_fs().delete_temp(parse_result.temp_dir_path, ctx=ctx)+ await self._cleanup_parse_result_artifact(+ parse_result,+ output_store=output_store,+ viking_fs=get_viking_fs(),+ ctx=ctx,+ )except Exception:passraise- finally:- try:- ResourceIngestionEventDataSource.record_stage(- stage="persist",- status=str(stage_status),- duration_seconds=float(time.perf_counter() - stage_start),- account_id=getattr(ctx, "account_id", None),- )- except Exception:- pass+ if artifact_ref is not None:+ artifact_store = self._store_for_parse_artifact(+ artifact_ref,+ output_store=output_store,+ viking_fs=get_viking_fs(),+ ctx=ctx,+ )+ try:+ await artifact_store.cleanup(artifact_ref)+ except Exception as exc:+ logger.warning(+ "[ResourceProcessor] Failed to clean committed parse artifact %s: %s",+ artifact_ref.root,+ exc,+ )+ artifact_ref = None+ prepared_artifact_ref = artifact_ref.to_dict() if artifact_ref is not None else None+ if prepared_artifact_ref is not None and artifact_ref.backend == "local":+ prepared_artifact_ref["resource_rel"] = local_artifact_doc_relprepared = {"root_uri": root_uri,"temp_uri": temp_uri or parse_result.temp_dir_path,"temp_dir_path": parse_result.temp_dir_path,+ "artifact_ref": prepared_artifact_ref,"source_committed": source_committed,"target_preexisting": target_preexisting,"is_code_repo": parse_result.source_format == "repository","root_is_file": root_is_file,+ "incremental_noop": incremental_noop,+ "context_update_plan": (+ context_update_plan.to_dict() if context_update_plan is not None else None+ ),"semantic_source": self._semantic_source_metadata(path=path,prepared_resource=prepared_resource,@@ -592,7 +1017,20 @@ class ResourceProcessor:root_uri = str(prepared.get("root_uri") or "")temp_uri = prepared.get("temp_uri")temp_dir_path = prepared.get("temp_dir_path")+ # Canonical plans contain only final resource URIs. An artifact ref is+ # present here only for legacy non-plan paths that still need it.+ artifact_ref_data = prepared.get("artifact_ref")+ artifact_ref = None+ if artifact_ref_data is not None:+ from openviking.parse.output import ParseArtifactRef++ artifact_ref = ParseArtifactRef.from_dict(artifact_ref_data)+ if artifact_ref.backend not in {"agfs", "local"}:+ raise ValueError(+ f"Unsupported parse artifact backend in post-process: {artifact_ref.backend}"+ )source_committed = bool(prepared.get("source_committed"))+ metrics_account_id = getattr(ctx, "account_id", None)target_preexisting = bool(prepared.get("target_preexisting"))build_index = bool(kwargs.get("build_index", True))processing_mode = normalize_processing_mode(processing_mode)@@ -600,51 +1038,124 @@ class ResourceProcessor:root_is_file = bool(prepared.get("root_is_file"))ingest_options = IngestOptions.from_value(kwargs.pop("ingest_options", None))semantic_source = prepared.get("semantic_source")- should_summarize = not root_is_file and not vectors_only and (summarize or build_index)- should_refresh_file_parent = (- root_is_file and not vectors_only and (summarize or build_index)+ context_update_plan_data = prepared.get("context_update_plan")+ context_update_plan = None+ if context_update_plan_data is not None:+ from openviking.storage.context_update_plan import ContextUpdatePlan++ context_update_plan = ContextUpdatePlan.from_dict(context_update_plan_data)+ semantic_plan = (+ context_update_plan.semantic_plan.to_dict()+ if context_update_plan.semantic_plan is not None+ else None+ )+ direct_index_actions = context_update_plan.direct_index_actions+ else:+ semantic_plan = None+ direct_index_actions = ()+ uses_context_update_plan = context_update_plan_data is not None+ should_summarize = (+ not root_is_file+ and not vectors_only+ and (summarize or build_index)+ and (not uses_context_update_plan or semantic_plan is not None))+ if context_update_plan is not None:+ file_refresh = context_update_plan.file_refresh+ elif root_is_file and source_committed and not vectors_only and (summarize or build_index):+ from openviking.storage.context_update_plan import FileRefreshIntent++ file_refresh = FileRefreshIntent(+ root_uri, (prepared.get("file_md5s") or {}).get(root_uri)+ )+ else:+ file_refresh = None+ should_refresh_file_parent = file_refresh is not None and not vectors_onlyresult: Dict[str, Any] = {"status": "success", "root_uri": root_uri}+ artifact_cleaned = False- if should_summarize:- stage_start = time.perf_counter()- stage_status = "ok"- try:- with get_current_telemetry().measure("resource.summarize"):- summary_result = await self._get_summarizer().summarize(- resource_uris=[root_uri],- ctx=ctx,- skip_vectorization=not build_index,- lock=resource_lock,- temp_uris=[temp_uri],- is_code_repo=bool(prepared.get("is_code_repo")),- target_preexisting=target_preexisting,- ingest_options=ingest_options,- semantic_source=semantic_source,- generation_trigger="resource_ingest",- **kwargs,- )- if (- resource_lock is not None- and summary_result.get("status") == "success"- and summary_result.get("enqueued_count", 0) > 0- ):- await get_viking_fs()._async_agfs.pathlock_handoff(resource_lock)+ async def cleanup_artifact_if_owned() -> None:+ nonlocal artifact_cleaned+ if artifact_cleaned or artifact_ref is None:+ return+ from openviking.parse.output import AgfsParseOutputStore++ # Local mode reuses the processor's own store seam (test-injectable);+ # agfs mode builds a temp-backed store on demand.+ output_store = (+ self._build_parse_output_store()+ if artifact_ref.backend == "local"+ else AgfsParseOutputStore(viking_fs=get_viking_fs(), ctx=ctx)+ )+ if output_store is not None:+ await output_store.cleanup(artifact_ref)+ artifact_cleaned = True++ derived_enqueue_started_at = time.perf_counter()+ derived_enqueue_status = "ok"+ try:+ with get_current_telemetry().measure("resource.derived_enqueue"):+ if prepared.get("incremental_noop") and not direct_index_actions:+ await cleanup_artifact_if_owned()+ if resource_lock is not None:+ await get_viking_fs()._async_agfs.pathlock_release(resource_lock)resource_lock = None- except Exception as exc:- logger.error("Summarization failed: %s", exc)- result["warnings"] = [f"Summarization failed: {exc}"]- stage_status = "error"- finally:- try:- ResourceIngestionEventDataSource.record_stage(- stage="summarize",- status=stage_status,- duration_seconds=float(time.perf_counter() - stage_start),- account_id=getattr(ctx, "account_id", None),- )- except Exception:- pass+ return result++ if direct_index_actions:+ await self._enqueue_index_actions(direct_index_actions, ctx=ctx)++ if should_summarize:+ try:+ summary_result = await self._get_summarizer().summarize(+ resource_uris=[root_uri],+ ctx=ctx,+ skip_vectorization=not build_index,+ lock=resource_lock,+ temp_uris=[temp_uri],+ is_code_repo=bool(prepared.get("is_code_repo")),+ target_preexisting=target_preexisting,+ ingest_options=ingest_options,+ semantic_source=semantic_source,+ generation_trigger="resource_ingest",+ semantic_plan=semantic_plan,+ **kwargs,+ )+ if semantic_plan is not None and summary_result.get("status") != "success":+ raise RuntimeError(+ str(summary_result.get("message") or "semantic plan enqueue failed")+ )+ if (+ resource_lock is not None+ and summary_result.get("status") == "success"+ and summary_result.get("enqueued_count", 0) > 0+ ):+ await get_viking_fs()._async_agfs.pathlock_handoff(resource_lock)+ resource_lock = None+ if semantic_plan is not None and (+ summary_result.get("status") == "success"+ and summary_result.get("enqueued_count", 0) > 0+ ):+ await cleanup_artifact_if_owned()+ except Exception as exc:+ logger.error("Semantic enqueue failed: %s", exc)+ if semantic_plan is not None:+ raise+ result["warnings"] = [f"Semantic enqueue failed: {exc}"]+ except Exception:+ derived_enqueue_status = "error"+ await cleanup_artifact_if_owned()+ if resource_lock is not None:+ await get_viking_fs()._async_agfs.pathlock_release(resource_lock)+ resource_lock = None+ raise+ finally:+ ResourceIngestionEventDataSource.record_stage(+ stage="derived_enqueue",+ status=derived_enqueue_status,+ duration_seconds=time.perf_counter() - derived_enqueue_started_at,+ account_id=metrics_account_id,+ )if resource_lock is not None:try:@@ -654,19 +1165,13 @@ class ResourceProcessor:viking_fs = get_viking_fs()if vectors_only and target_preexisting and not root_is_file:diff = await SemanticProcessor()._sync_topdown_recursive(- temp_uri,- root_uri,- ctx=ctx,- lock=resource_lock,+ temp_uri, root_uri, ctx=ctx, lock=resource_lock)sync_deleted_files = list(getattr(diff, "deleted_files", []))sync_deleted_dirs = list(getattr(diff, "deleted_dirs", []))else:await viking_fs.persist_temp_tree(- temp_uri,- root_uri,- ctx=ctx,- lease_ref=resource_lock,+ temp_uri, root_uri, ctx=ctx, lease_ref=resource_lock)if not root_is_file:await rewrite_image_uris(@@ -676,22 +1181,25 @@ class ResourceProcessor:)if temp_dir_path:await viking_fs.delete_temp(temp_dir_path, ctx=ctx)- if vectors_only:- if sync_deleted_files or sync_deleted_dirs:- await self._delete_removed_resource_vectors(- files=sync_deleted_files,- dirs=sync_deleted_dirs,- ctx=ctx,- )+ if (+ context_update_plan is None+ and vectors_only+ and (sync_deleted_files or sync_deleted_dirs)+ ):+ await self._delete_removed_resource_vectors(+ files=sync_deleted_files, dirs=sync_deleted_dirs, ctx=ctx+ )if should_refresh_file_parent:await self._get_summarizer().refresh_file_parent(- file_uri=root_uri,+ file_uri=file_refresh.file_uri,ctx=ctx,skip_vectorization=not build_index,ingest_options=ingest_options,created=not target_preexisting,+ file_md5=file_refresh.md5,+ file_abstract="",)- elif build_index:+ elif build_index and context_update_plan is None:if root_is_file:await self._vectorize_resource_file(root_uri,@@ -700,35 +1208,61 @@ class ResourceProcessor:creator_acl_grant=(CreatorAclGrant.DIRECT if not target_preexisting else None),+ file_md5=(prepared.get("file_md5s") or {}).get(root_uri),)elif vectors_only:await self._vectorize_resource_files(root_uri, ctx=ctx, ingest_options=ingest_options)+ except BaseException:+ await cleanup_artifact_if_owned()+ raisefinally:await get_viking_fs()._async_agfs.pathlock_release(resource_lock)elif should_refresh_file_parent:- await self._get_summarizer().refresh_file_parent(- file_uri=root_uri,- ctx=ctx,- skip_vectorization=not build_index,- ingest_options=ingest_options,- created=not target_preexisting,- )- elif vectors_only or root_is_file:- if not build_index:- return result- if root_is_file:+ try:+ await self._get_summarizer().refresh_file_parent(+ file_uri=file_refresh.file_uri,+ ctx=ctx,+ skip_vectorization=not build_index,+ ingest_options=ingest_options,+ created=not target_preexisting,+ file_md5=file_refresh.md5,+ file_abstract="",+ )+ except BaseException:+ await cleanup_artifact_if_owned()+ raise+ elif build_index and context_update_plan is None and root_is_file:+ try:await self._vectorize_resource_file(root_uri,ctx=ctx,ingest_options=ingest_options,creator_acl_grant=(CreatorAclGrant.DIRECT if not target_preexisting else None),+ file_md5=(prepared.get("file_md5s") or {}).get(root_uri),+ )+ except BaseException:+ await cleanup_artifact_if_owned()+ raise+ elif build_index and context_update_plan is None and vectors_only:+ try:+ artifact_store = (+ self._build_parse_output_store() if artifact_ref is not None else None)- else:await self._vectorize_resource_files(- root_uri, ctx=ctx, ingest_options=ingest_options+ root_uri,+ ctx=ctx,+ ingest_options=ingest_options,+ artifact_store=artifact_store,+ artifact_ref=artifact_ref,+ artifact_files=prepared.get("artifact_files"),+ file_md5s=prepared.get("file_md5s"),)+ except BaseException:+ await cleanup_artifact_if_owned()+ raise+ await cleanup_artifact_if_owned()return result@staticmethod@@ -755,6 +1289,78 @@ class ResourceProcessor:kind = "local"return {"kind": kind, "uri": str(path)}+ async def _enqueue_index_actions(self, actions: Any, *, ctx: RequestContext) -> None:+ from collections import Counter++ from openviking.storage.index_action import IndexAction+ from openviking.storage.queuefs import get_queue_manager+ from openviking.storage.queuefs.embedding_msg import EmbeddingMsg+ from openviking.telemetry import get_current_telemetry+ from openviking.utils.embedding_utils import _enqueue_embedding_message++ queue_manager = get_queue_manager()+ embedding_queue = queue_manager.get_queue(queue_manager.EMBEDDING, allow_create=True)+ telemetry_id = get_current_telemetry().telemetry_id+ action_counts = Counter(action.action.value for action in actions)+ delete_ids = [action.record_id for action in actions if action.action == IndexAction.DELETE]+ if delete_ids:+ message = EmbeddingMsg.for_delete(+ record_ids=delete_ids,+ context_data={+ "uri": actions[0].uri,+ "account_id": ctx.account_id,+ "owner_user_id": ctx.user.user_id,+ },+ telemetry_id=telemetry_id,+ )+ await _enqueue_embedding_message(+ embedding_queue,+ message,+ failure_message="Failed to enqueue planned vector deletes",+ )+ for action in actions:+ if action.action in {IndexAction.UPSERT, IndexAction.MERGE}:+ if action.level != int(ContextLevel.DETAIL):+ raise ValueError("Direct index upsert only supports file detail records")+ await self._vectorize_resource_file(+ action.uri,+ ctx=ctx,+ file_md5=action.md5,+ scalar_override={+ **dict(action.upsert_fields),+ "_record_id": action.record_id,+ },+ action=action.action.value,+ field_patch=action.field_patch,+ )+ continue+ if action.action != IndexAction.UPDATE_FIELDS:+ continue+ assert action.field_patch is not None+ message = EmbeddingMsg.for_update_fields(+ record_id=action.record_id,+ field_patch=action.field_patch,+ context_data={+ "uri": action.uri,+ "level": action.level,+ "account_id": ctx.account_id,+ "owner_user_id": ctx.user.user_id,+ },+ telemetry_id=telemetry_id,+ )+ await _enqueue_embedding_message(+ embedding_queue,+ message,+ failure_message=f"Failed to enqueue scalar update for {action.uri}",+ )+ logger.debug(+ "[DirectIndexActions] %s root=%s action_counts=%s action_count=%d",+ log_correlation(),+ actions[0].uri if actions else "",+ dict(action_counts),+ len(actions),+ )+async def _delete_removed_resource_vectors(self,*,@@ -795,27 +1401,38 @@ class ResourceProcessor:*,ctx: RequestContext,ingest_options: IngestOptions | None = None,+ artifact_store: Any = None,+ artifact_ref: Any = None,+ artifact_files: Optional[List[str]] = None,+ file_md5s: Optional[Dict[str, str]] = None,) -> None:ingest_options = IngestOptions.from_value(ingest_options)viking_fs = get_viking_fs()- entries = await viking_fs.tree(- root_uri,- node_limit=None,- level_limit=None,- ctx=ctx,- )files: list[tuple[str, str, str]] = []- for entry in entries:- entry_uri = entry.get("uri") if isinstance(entry, dict) else None- if not entry_uri or entry.get("isDir"):- continue- name = entry.get("name") or entry_uri.rsplit("/", 1)[-1]- if str(name).startswith("."):- continue- parent = VikingURI(entry_uri).parent- if parent is None:- continue- files.append((entry_uri, str(name), parent.uri))+ if artifact_files is not None:+ for rel_path in artifact_files:+ entry_uri = VikingURI(root_uri).join(rel_path).uri+ parent = VikingURI(entry_uri).parent+ if parent is not None:+ files.append((entry_uri, rel_path.rsplit("/", 1)[-1], parent.uri))+ else:+ entries = await viking_fs.tree(+ root_uri,+ node_limit=None,+ level_limit=None,+ ctx=ctx,+ )+ for entry in entries:+ entry_uri = entry.get("uri") if isinstance(entry, dict) else None+ if not entry_uri or entry.get("isDir"):+ continue+ name = entry.get("name") or entry_uri.rsplit("/", 1)[-1]+ if str(name).startswith("."):+ continue+ parent = VikingURI(entry_uri).parent+ if parent is None:+ continue+ files.append((entry_uri, str(name), parent.uri))config = get_openviking_config().queue_workers.add_resourceconcurrency = max(@@ -827,6 +1444,15 @@ class ResourceProcessor:)async def vectorize(entry_uri: str, name: str, parent_uri: str) -> None:+ file_content = None+ if artifact_store is not None and artifact_ref is not None:+ rel_path = entry_uri[len(root_uri.rstrip("/")) + 1 :]+ artifact_rel = (+ f"{artifact_ref.resource_rel.strip('/')}/{rel_path}"+ if artifact_ref.resource_rel+ else rel_path+ )+ file_content = await artifact_store.read_bytes(artifact_ref, artifact_rel)await vectorize_file(file_path=entry_uri,summary_dict={"name": name, "summary": ""},@@ -834,6 +1460,8 @@ class ResourceProcessor:context_type=context_type_for_uri(entry_uri),ctx=ctx,ingest_options=ingest_options,+ file_md5=(file_md5s or {}).get(entry_uri),+ file_content=file_content,)for start in range(0, len(files), concurrency):@@ -856,6 +1484,10 @@ class ResourceProcessor:ctx: RequestContext,ingest_options: IngestOptions | None = None,creator_acl_grant: CreatorAclGrant | None = None,+ file_md5: str | None = None,+ scalar_override: Optional[Dict[str, Any]] = None,+ field_patch: FieldPatch | None = None,+ action: str = "merge",) -> None:parent = VikingURI(file_uri).parentif parent is None:@@ -869,6 +1501,10 @@ class ResourceProcessor:ctx=ctx,ingest_options=IngestOptions.from_value(ingest_options),creator_acl_grant=creator_acl_grant,+ file_md5=file_md5,+ scalar_override=scalar_override,+ field_patch=field_patch,+ action=action,)async def reserve_unique_candidate(diff --git a/openviking/utils/summarizer.py b/openviking/utils/summarizer.pyindex c39f6d443..a59b109f6 100644--- a/openviking/utils/summarizer.py+++ b/openviking/utils/summarizer.py@@ -40,6 +40,8 @@ class Summarizer:skip_vectorization: bool = False,ingest_options: IngestOptions | None = None,created: bool = False,+ file_md5: str | None = None,+ file_abstract: str = "",) -> Dict[str, Any]:"""Summarize one flat file and refresh its parent directory semantics."""parent = VikingURI(file_uri).parent@@ -69,6 +71,8 @@ class Summarizer:peer_id=ctx.user.user_id,),ingest_options=ingest_options,+ file_md5s={file_uri: file_md5} if file_md5 else None,+ file_abstracts={file_uri: file_abstract} if file_abstract else None,)if telemetry_id:get_request_wait_tracker().register_semantic_root(telemetry_id, msg.id)@@ -111,6 +115,17 @@ class Summarizer:ingest_options = IngestOptions.from_value(kwargs.get("ingest_options"))source = kwargs.get("semantic_source")generation_trigger = str(kwargs.get("generation_trigger") or "manual_refresh")+ # Pre-computed change set (local incremental import): when present, the+ # semantic tree restricts re-summarization/vectorization to these files+ # instead of diffing the whole tree.+ changes = kwargs.get("changes")+ # Per-file md5 (target-URI keyed) for those changed files, so the tree executor's+ # re-vectorization records the fresh fingerprint.+ file_md5s = kwargs.get("file_md5s") or {}+ artifact_ref = kwargs.get("artifact_ref")+ artifact_files = kwargs.get("artifact_files") or []+ file_abstracts = kwargs.get("file_abstracts") or {}+ semantic_plan = kwargs.get("semantic_plan")if not temp_uris:temp_uris = resource_urisif len(temp_uris) != len(resource_uris):@@ -195,6 +210,12 @@ class Summarizer:ingest_options=ingest_options,source=source,generation_trigger=generation_trigger,+ changes=changes,+ file_md5s=file_md5s,+ artifact_ref=artifact_ref,+ artifact_files=artifact_files,+ file_abstracts=file_abstracts,+ plan=semantic_plan,)if msg.telemetry_id:get_request_wait_tracker().register_semantic_root(msg.telemetry_id, msg.id)diff --git a/openviking_cli/utils/config/parse_output_config.py b/openviking_cli/utils/config/parse_output_config.pynew file mode 100644index 000000000..61fca333f--- /dev/null+++ b/openviking_cli/utils/config/parse_output_config.py@@ -0,0 +1,43 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+from pathlib import Path+from typing import Optional++from pydantic import BaseModel, Field, model_validator+++class ParseOutputConfig(BaseModel):+ """Where parsers write intermediate artifacts.++ ``agfs`` (default) writes to the shared AGFS ``viking://temp`` space. ``local``+ keeps parser intermediates on the current worker; add-resources commits the+ required bytes to the formal AGFS tree before enqueueing downstream work.+ """++ mode: str = Field(+ default="agfs",+ description="Parse artifact backend: 'agfs' (shared temp) or 'local' (local dir).",+ )+ local_root: Optional[str] = Field(+ default=None,+ description=(+ "Directory root for the 'local' backend. Defaults to a dedicated subdir "+ "under the system temp dir when unset."+ ),+ )++ model_config = {"extra": "forbid"}++ @model_validator(mode="after")+ def validate_mode(self):+ if self.mode not in {"agfs", "local"}:+ raise ValueError("storage.parse_output.mode must be 'agfs' or 'local'")+ return self++ def resolved_local_root(self) -> str:+ """Return the local root, falling back to a dedicated system-temp subdir."""+ if self.local_root:+ return str(Path(self.local_root).expanduser())+ import tempfile++ return str(Path(tempfile.gettempdir()) / "openviking-parse")diff --git a/openviking_cli/utils/config/queue_worker_config.py b/openviking_cli/utils/config/queue_worker_config.pyindex e26ce36c1..3ebb7f01a 100644--- a/openviking_cli/utils/config/queue_worker_config.py+++ b/openviking_cli/utils/config/queue_worker_config.py@@ -18,6 +18,12 @@ class QueueWorkerConfig(BaseModel):class AddResourceQueueWorkerConfig(QueueWorkerConfig):"""Runtime limits for add-resource queue workers."""+ file_operation_concurrency: int = Field(+ default=16,+ gt=0,+ description="Maximum concurrent file-level commit and comparison operations within one add-resource job",+ )+file_vectorization_concurrency: int = Field(default=8,gt=0,diff --git a/openviking_cli/utils/config/storage_config.py b/openviking_cli/utils/config/storage_config.pyindex ab1953c61..f738c0b75 100644--- a/openviking_cli/utils/config/storage_config.py+++ b/openviking_cli/utils/config/storage_config.py@@ -8,6 +8,7 @@ from pydantic import BaseModel, Field, model_validatorfrom openviking_cli.utils.logger import get_loggerfrom .agfs_config import AGFSConfig+from .parse_output_config import ParseOutputConfigfrom .transaction_config import TransactionConfigfrom .vectordb_config import VectorDBBackendConfig@@ -44,6 +45,11 @@ class StorageConfig(BaseModel):description="VectorDB backend configuration",)+ parse_output: ParseOutputConfig = Field(+ default_factory=ParseOutputConfig,+ description="Where parsers write intermediate artifacts (agfs temp or local dir)",+ )+params: Dict[str, Any] = Field(default_factory=dict, description="Additional storage-specific parameters")diff --git a/tests/metrics/integration/test_bootstrap.py b/tests/metrics/integration/test_bootstrap.pyindex 5c0f93432..186432153 100644--- a/tests/metrics/integration/test_bootstrap.py+++ b/tests/metrics/integration/test_bootstrap.py@@ -99,7 +99,7 @@ def test_create_default_collector_manager_propagates_construction_failures(monkedef test_optional_cache_datasource_instrumentation_is_wired_into_key_call_sites():project_root = Path(__file__).resolve().parents[3]targets = [- project_root / "openviking" / "storage" / "queuefs" / "semantic_dag.py",+ project_root / "openviking" / "storage" / "queuefs" / "semantic_executor.py",project_root / "openviking" / "storage" / "queuefs" / "semantic_processor.py",project_root / "openviking" / "session" / "memory" / "extract_loop.py",]diff --git a/tests/misc/test_ovpack_import_policy.py b/tests/misc/test_ovpack_import_policy.pyindex 7f93a3440..9b2df3b77 100644--- a/tests/misc/test_ovpack_import_policy.py+++ b/tests/misc/test_ovpack_import_policy.py@@ -342,6 +342,7 @@ class FakeVectorStore:"context_type": "resource","level": 2,"abstract": "note summary",+ "md5": "note-md5","tags": ["snapshot"],"vector": [0.1, 0.2, 0.3],}@@ -567,11 +568,13 @@ def test_index_consistency_report_limits_public_and_error_records():async def test_export_ovpack_writes_v3_manifest_with_abstract_overviews(temp_ovpack_path: Path, request_ctx: RequestContext):+ vector_store = FakeVectorStore()await export_ovpack(FakeExportVikingFS(),"viking://resources/demo",str(temp_ovpack_path),ctx=request_ctx,+ vector_store=vector_store,)with zipfile.ZipFile(temp_ovpack_path, "r") as zf:@@ -602,6 +605,8 @@ async def test_export_ovpack_writes_v3_manifest_with_abstract_overviews(assert manifest["index"]["records"]["count"] == len(index_records)assert index_records[0]["path"] == ""assert index_records[0]["text"] == "root abstract"+ note_record = next(record for record in index_records if record["path"] == "notes.txt")+ assert note_record["scalars"]["md5"] == "note-md5"@pytest.mark.asyncio@@ -748,6 +753,7 @@ async def test_restore_ovpack_applies_backup_manifest_scalar_metadata("abstract": "portable summary","description": "portable description","tags": ["portable"],+ "md5": "portable-md5",},}],@@ -771,6 +777,7 @@ async def test_restore_ovpack_applies_backup_manifest_scalar_metadata("summary": "portable summary",}assert vectorized_files[0]["scalar_override"]["tags"] == ["portable"]+ assert vectorized_files[0]["scalar_override"]["md5"] == "portable-md5"@pytest.mark.asynciodiff --git a/tests/misc/test_resource_processor_mv.py b/tests/misc/test_resource_processor_mv.pyindex ea710786d..a3c1320e2 100644--- a/tests/misc/test_resource_processor_mv.py+++ b/tests/misc/test_resource_processor_mv.py@@ -6,6 +6,14 @@ from unittest.mock import AsyncMock, MagicMockimport pytest+from openviking.storage.context_update_plan import (+ ContextUpdatePlan,+ FileRefreshIntent,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+)+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))@@ -14,6 +22,18 @@ class _DummyVikingDB:return None+def _directory_plan(root_uri: str) -> ContextUpdatePlan:+ return ContextUpdatePlan(+ root_uri,+ "resource",+ semantic_plan=SemanticPlan(+ root_uri,+ "resource",+ SemanticTreeSnapshot((SemanticTreeEntry("", "directory", "added", "aggregate"),)),+ ),+ )++class _DummyTelemetry:def set(self, *args, **kwargs):return None@@ -70,6 +90,9 @@ class _FakePathLock:async def pathlock_release(self, lease):pass+ async def pathlock_handoff(self, lease):+ pass+class _FakeVikingFS:def __init__(self, *, exists_result=False, existing_uris=None, pathlock=None):@@ -86,6 +109,9 @@ class _FakeVikingFS:def bind_request_context(self, ctx):return _CtxMgr()+ async def _ensure_access(self, uri, ctx, action):+ return None+async def exists(self, uri, ctx=None):self.exists_calls.append(uri)if self._existing_uris:@@ -180,6 +206,131 @@ async def test_resource_processor_rejects_directory_when_every_file_failed(monkerp.tree_builder.finalize_from_temp.assert_not_awaited()[email protected]("build_index", [True, False])+async def test_resource_processor_rejects_partial_directory_before_semantic_plan_diff(+ monkeypatch, build_index+):+ from openviking.parse.output import ParseArtifactRef+ from openviking.utils.resource_processor import ResourceProcessor++ fake_fs = _FakeVikingFS(exists_result=True)+ monkeypatch.setattr(+ "openviking.utils.resource_processor.get_current_telemetry",+ lambda: _DummyTelemetry(),+ )+ _patch_viking_fs(monkeypatch, fake_fs)++ ref = ParseArtifactRef(backend="agfs", root="viking://temp/partial")+ parse_result = SimpleNamespace(+ temp_dir_path=ref.root,+ source_path="repo",+ source_format="repository",+ meta={+ "file_count": 1,+ "total_processable": 2,+ "processed_files": [{"path": "kept.py"}],+ "failed_files": [+ {+ "path": "missing.py",+ "parser": "native",+ "error": "parser failed",+ }+ ],+ },+ warnings=[],+ artifact_ref=ref,+ ensure_artifact_ref=lambda: ref,+ )+ rp = ResourceProcessor(vikingdb=_DummyVikingDB(), media_storage=None)+ rp._get_media_processor = MagicMock()+ rp._get_media_processor.return_value.process = AsyncMock(return_value=parse_result)+ rp.tree_builder.finalize_from_temp = AsyncMock(+ side_effect=AssertionError("partial artifact must not reach finalize/diff")+ )+ rp._commit_directory_artifact_with_plan = AsyncMock(+ return_value=_directory_plan("viking://resources/root")+ )++ result = await rp.process_resource(path="repo", ctx=object(), build_index=build_index)++ assert result["status"] == "error"+ assert result["errors"] == [+ "Directory import incomplete: 1 file(s) failed to parse; "+ "failed files: missing.py: parser failed"+ ]+ assert fake_fs.delete_temp_calls == [(ref.root, None)]+ rp.tree_builder.finalize_from_temp.assert_not_awaited()+++async def test_partial_directory_rejection_cleans_local_parse_artifact(monkeypatch, tmp_path):+ from openviking.parse.output import LocalParseOutputStore+ from openviking.utils.resource_processor import ResourceProcessor++ fake_fs = _FakeVikingFS(exists_result=True)+ monkeypatch.setattr(+ "openviking.utils.resource_processor.get_current_telemetry",+ lambda: _DummyTelemetry(),+ )+ _patch_viking_fs(monkeypatch, fake_fs)++ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))+ ref = await store.create_artifact()+ await store.write_bytes(ref, "repo/kept.py", b"pass")+ parse_result = SimpleNamespace(+ temp_dir_path=ref.root,+ source_path="repo",+ source_format="repository",+ meta={+ "file_count": 1,+ "total_processable": 2,+ "processed_files": [{"path": "kept.py"}],+ "failed_files": [{"path": "missing.py", "error": "failed"}],+ },+ warnings=[],+ artifact_ref=ref,+ ensure_artifact_ref=lambda: ref,+ )+ rp = ResourceProcessor(vikingdb=_DummyVikingDB(), media_storage=None)+ rp._build_parse_output_store = MagicMock(return_value=store)+ rp._get_media_processor = MagicMock()+ rp._get_media_processor.return_value.process = AsyncMock(return_value=parse_result)+ rp.tree_builder.finalize_from_temp = AsyncMock()++ result = await rp.process_resource(path="repo", ctx=object(), build_index=True)++ assert result["status"] == "error"+ assert not tmp_path.joinpath("artifacts", ref.root.split("/")[-1]).exists()+ assert fake_fs.delete_temp_calls == []+ rp.tree_builder.finalize_from_temp.assert_not_awaited()+++def test_directory_parse_failures_include_failed_source_access_items():+ from openviking.utils.resource_processor import ResourceProcessor++ failures = ResourceProcessor._directory_parse_failures(+ {+ "failed_files": [+ {"path": "broken.pdf", "error": "invalid PDF"},+ {"path": "denied.docx", "status": "failed", "reason": "HTTP 403"},+ {"path": "binary.bin", "status": "unsupported", "reason": "binary"},+ {+ "path": "unsupported.csv",+ "status": "unsupported",+ "error": "unsupported type",+ },+ {"path": "ignored.tmp", "status": "skipped", "error": "ignored"},+ ]+ }+ )++ assert failures == [+ {"path": "broken.pdf", "error": "invalid PDF"},+ {"path": "denied.docx", "status": "failed", "reason": "HTTP 403"},+ ]++@pytest.mark.parametrize("length", [119, 120, 121, 240])def test_empty_directory_error_keeps_120_character_limit(length):from openviking.utils.resource_processor import ResourceProcessor@@ -277,6 +428,9 @@ async def test_resource_processor_first_add_summarizes_from_committed_uri(monkeyroot=SimpleNamespace(uri="viking://resources/root", temp_uri="viking://temp/root_tmp")))+ rp._commit_directory_artifact_with_plan = AsyncMock(+ return_value=_directory_plan("viking://resources/root")+ )rp._summarizer = SimpleNamespace(summarize=AsyncMock(side_effect=lambda *args, **kwargs: (@@ -289,14 +443,80 @@ async def test_resource_processor_first_add_summarizes_from_committed_uri(monkeyassert result["status"] == "success"assert result["root_uri"] == "viking://resources/root"- assert fake_fs.persist_calls == [- ("viking://temp/root_tmp", "viking://resources/root", {"id": "lock-1"})- ]+ assert fake_fs.persist_calls == []assert fake_fs.delete_temp_calls == [("viking://temp/tmpdir", None)]assert summarize_calls[0]["temp_uris"] == ["viking://resources/root"]assert summarize_calls[0]["target_preexisting"] is False+async def test_directory_semantic_ingest_commits_plan_before_summarizer(monkeypatch):+ from openviking.parse.output import ParseArtifactRef+ from openviking.storage.context_update_plan import (+ ContextUpdatePlan,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+ )+ from openviking.utils.resource_processor import ResourceProcessor++ fake_fs = _FakeVikingFS()+ monkeypatch.setattr(+ "openviking.utils.resource_processor.get_current_telemetry",+ lambda: _DummyTelemetry(),+ )+ _patch_viking_fs(monkeypatch, fake_fs)+ ref = ParseArtifactRef(+ backend="agfs",+ root="viking://temp/tmpdir",+ resource_rel="root_tmp",+ )+ parse_result = SimpleNamespace(+ temp_dir_path=ref.root,+ source_path="x",+ source_format="repository",+ meta={},+ warnings=[],+ artifact_ref=ref,+ ensure_artifact_ref=lambda: ref,+ )+ plan = SemanticPlan(+ root_uri="viking://resources/root",+ context_type="resource",+ tree=SemanticTreeSnapshot(+ entries=(+ SemanticTreeEntry("", "directory", "added", "aggregate"),+ SemanticTreeEntry("a.py", "file", "added", "generate", md5="a"),+ )+ ),+ )+ context_plan = ContextUpdatePlan(plan.root_uri, "resource", semantic_plan=plan)+ rp = ResourceProcessor(vikingdb=_DummyVikingDB(), media_storage=None)+ rp._get_media_processor = MagicMock()+ rp._get_media_processor.return_value.process = AsyncMock(return_value=parse_result)+ rp.tree_builder.finalize_from_temp = AsyncMock(+ return_value=SimpleNamespace(+ root=SimpleNamespace(+ uri="viking://resources/root",+ temp_uri="viking://temp/tmpdir/root_tmp",+ ),+ _root_is_file=False,+ )+ )+ rp._commit_directory_artifact_with_plan = AsyncMock(return_value=context_plan)+ summarize = AsyncMock(return_value={"status": "success", "enqueued_count": 1})+ rp._summarizer = SimpleNamespace(summarize=summarize)++ result = await rp.process_resource(path="x", ctx=object(), build_index=True)++ assert result["status"] == "success"+ assert fake_fs.persist_calls == []+ kwargs = summarize.await_args.kwargs+ assert SemanticPlan.from_dict(kwargs["semantic_plan"]) == plan+ assert kwargs["temp_uris"] == ["viking://resources/root"]+ assert kwargs.get("artifact_ref") is None++@pytest.mark.asyncio@pytest.mark.parametrize("auto_candidate", [False, True])async def test_resource_processor_allows_flat_root_only_for_single_no_split_source(@@ -337,6 +557,11 @@ async def test_resource_processor_allows_flat_root_only_for_single_no_split_sour_candidate_uri=root_uri if auto_candidate else None,))+ rp._commit_directory_artifact_with_plan = AsyncMock(+ return_value=ContextUpdatePlan(+ root_uri, "resource", file_refresh=FileRefreshIntent(root_uri, "new-md5")+ )+ )rp._summarizer = SimpleNamespace(summarize=AsyncMock(return_value={"status": "success"}),refresh_file_parent=AsyncMock(return_value={"status": "success"}),@@ -387,6 +612,9 @@ async def test_resource_processor_keeps_wrapper_for_directory_to_no_split(monkey_root_is_file=False,))+ rp._commit_directory_artifact_with_plan = AsyncMock(+ return_value=_directory_plan("viking://resources/0803_shendiao_01")+ )rp._summarizer = SimpleNamespace(summarize=AsyncMock(return_value={"status": "success"}))result = await rp.process_resource(@@ -431,6 +659,9 @@ async def test_resource_processor_second_add_preserves_temp_uri_for_incremental(root=SimpleNamespace(uri="viking://resources/root", temp_uri="viking://temp/root_tmp"))rp.tree_builder.finalize_from_temp = AsyncMock(return_value=context_tree)+ rp._commit_directory_artifact_with_plan = AsyncMock(+ return_value=_directory_plan("viking://resources/root")+ )rp._summarizer = SimpleNamespace(summarize=AsyncMock(side_effect=lambda *args, **kwargs: (@@ -443,7 +674,7 @@ async def test_resource_processor_second_add_preserves_temp_uri_for_incremental(assert result["status"] == "success"assert result["root_uri"] == "viking://resources/root"- assert summarize_calls[0]["temp_uris"] == ["viking://temp/root_tmp"]+ assert summarize_calls[0]["temp_uris"] == ["viking://resources/root"]assert summarize_calls[0]["target_preexisting"] is Trueassert fake_fs.persist_calls == []@@ -479,6 +710,9 @@ async def test_resource_processor_auto_candidate_skips_existing_and_busy(monkeyp_candidate_uri="viking://resources/root",)rp.tree_builder.finalize_from_temp = AsyncMock(return_value=context_tree)+ rp._commit_directory_artifact_with_plan = AsyncMock(+ side_effect=lambda **kwargs: _directory_plan(kwargs["root_uri"])+ )rp._summarizer = SimpleNamespace(summarize=AsyncMock(side_effect=lambda *args, **kwargs: (@@ -503,6 +737,4 @@ async def test_resource_processor_auto_candidate_skips_existing_and_busy(monkeypassert fake_pathlock.acquired_tree_paths == ["/mock/resources/root_2"]assert summarize_calls[0]["temp_uris"] == ["viking://resources/root_2"]assert summarize_calls[0]["target_preexisting"] is False- assert fake_fs.persist_calls == [- ("viking://temp/root_tmp", "viking://resources/root_2", {"id": "lock-1"})- ]+ assert fake_fs.persist_calls == []diff --git a/tests/parse/test_add_directory.py b/tests/parse/test_add_directory.pyindex 542c7db51..cc8153e3e 100644--- a/tests/parse/test_add_directory.py+++ b/tests/parse/test_add_directory.py@@ -24,6 +24,11 @@ from openviking.parse.base import (ResourceNode,create_parse_result,)+from openviking.parse.output import (+ ARTIFACT_MANIFEST_NAME,+ LocalParseOutputStore,+ read_artifact_manifest,+)from openviking.parse.parsers.base_parser import BaseParserfrom openviking.parse.parsers.directory import DirectoryParser@@ -57,7 +62,7 @@ class FakeVikingFS:content = content.encode("utf-8")self.files[uri] = content- async def write_file_bytes(self, uri: str, content: bytes) -> None:+ async def write_file_bytes(self, uri: str, content: bytes, **kw) -> None:self.files[uri] = content# ---- read / list operations ------------------------------------------@@ -109,7 +114,7 @@ class FakeVikingFS:# ---- temp URI --------------------------------------------------------- def create_temp_uri(self) -> str:+ def create_temp_uri(self, ctx: Any = None) -> str:self._temp_counter += 1return f"viking://temp/dir_{self._temp_counter}"@@ -217,12 +222,8 @@ class TestDirectoryParserBasic:(tmp_path / "notes").mkdir()(tmp_path / "08_Attachments").mkdir()(tmp_path / "notes" / "article.md").write_text("keep", encoding="utf-8")- (tmp_path / "notes" / "private.excalidraw.md").write_text(- "exclude", encoding="utf-8"- )- (tmp_path / "08_Attachments" / "diagram.md").write_text(- "ignore", encoding="utf-8"- )+ (tmp_path / "notes" / "private.excalidraw.md").write_text("exclude", encoding="utf-8")+ (tmp_path / "08_Attachments" / "diagram.md").write_text("ignore", encoding="utf-8")(tmp_path / "main.py").write_text("exclude by include", encoding="utf-8")with (@@ -236,7 +237,9 @@ class TestDirectoryParserBasic:exclude="*.excalidraw.md",)- uploaded_paths = {uri.split("/repository/", 1)[-1] for uri in fake_fs.files}+ uploaded_paths = {+ uri.split("/repository/", 1)[-1] for uri in fake_fs.files if "/repository/" in uri+ }assert result.parser_name == "CodeRepositoryParser"assert uploaded_paths == {"notes/article.md"}@@ -330,6 +333,8 @@ class TestDirectWriteFiles:dir_name = tmp_code.namefor uri in fake_fs.files:+ if uri.endswith(ARTIFACT_MANIFEST_NAME):+ continueassert f"/{dir_name}/" in uri@pytest.mark.asyncio@@ -344,6 +349,27 @@ class TestDirectWriteFiles:else:pytest.fail("hello.py not found in uploaded files")+ @pytest.mark.asyncio+ async def test_local_store_merges_parsed_and_direct_files(self, tmp_path: Path) -> None:+ source = tmp_path / "source"+ source.mkdir()+ (source / "guide.md").write_text("# Guide\n\nbody", encoding="utf-8")+ (source / "main.py").write_text("print('ok')", encoding="utf-8")+ store = LocalParseOutputStore(str(tmp_path / "artifacts"))++ result = await DirectoryParser().parse(source, parse_output_store=store)++ assert result.artifact_ref is not None+ assert result.artifact_ref.backend == "local"+ assert result.artifact_ref.resource_rel == "source"+ root = Path(result.artifact_ref.root) / "source"+ assert (root / "main.py").read_text() == "print('ok')"+ assert (root / "guide" / "guide.md").read_text() == "# Guide\n\nbody"+ assert set(await read_artifact_manifest(store, result.artifact_ref)) == {+ "source/main.py",+ "source/guide/guide.md",+ }+# ---------------------------------------------------------------------------# Tests: nested directory structure@@ -373,7 +399,8 @@ class TestNestedDirectory:@pytest.mark.asyncioasync def test_file_count(self, tmp_nested_code: Path, parser, fake_fs) -> None:await parser.parse(str(tmp_nested_code))- assert len(fake_fs.files) == 4+ business_files = [uri for uri in fake_fs.files if not uri.endswith(ARTIFACT_MANIFEST_NAME)]+ assert len(business_files) == 4# ---------------------------------------------------------------------------@@ -428,9 +455,7 @@ class TestParserDelegation:) -> None:nested = tmp_path / "scripts"nested.mkdir()- content = "\n\n".join(- f"paragraph {index} " + "x" * 1000 for index in range(20)- )+ content = "\n\n".join(f"paragraph {index} " + "x" * 1000 for index in range(20))(nested / "screenplay.md").write_text(content, encoding="utf-8")result = await parser.parse(str(tmp_path), split_content=False)@@ -441,9 +466,7 @@ class TestParserDelegation:for uri, value in fake_fs.files.items()if uri.startswith(root) and uri.endswith(".md")}- assert body_files == {- f"{root}/scripts/screenplay.md": content.encode("utf-8")- }+ assert body_files == {f"{root}/scripts/screenplay.md": content.encode("utf-8")}@pytest.mark.asyncioasync def test_txt_file_goes_through_parser(self, tmp_path: Path, parser, fake_fs) -> None:diff --git a/tests/parse/test_code_local_artifact.py b/tests/parse/test_code_local_artifact.pynew file mode 100644index 000000000..c8fa30447--- /dev/null+++ b/tests/parse/test_code_local_artifact.py@@ -0,0 +1,160 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Local-artifact-mode parsing for CodeRepositoryParser.++When a parse_output_store is threaded through parse kwargs (local mode), the+repository parser must write its artifacts into that store and never touch the+VikingFS singleton (AGFS temp). Without a store it keeps the legacy AGFS path.+"""++from pathlib import Path++import pytest++from openviking.parse.output import LocalParseOutputStore+from openviking.parse.parsers.code.code import CodeRepositoryParser+++def _make_repo(tmp_path: Path) -> Path:+ repo = tmp_path / "repo"+ (repo / "src").mkdir(parents=True)+ (repo / "src" / "main.py").write_text("print('hi')", encoding="utf-8")+ (repo / "README.md").write_text("# demo", encoding="utf-8")+ # Minimal git marker so DirectoryParser-style detection is consistent; the+ # parser itself accepts a plain local dir.+ (repo / ".git").mkdir()+ (repo / ".git" / "HEAD").write_text("ref: refs/heads/main", encoding="utf-8")+ return repo+++class _ExplodingVikingFS:+ """Any write here means the local path leaked back to AGFS temp."""++ def create_temp_uri(self, ctx=None):+ raise AssertionError("local mode must not allocate an AGFS temp uri")++ async def write_file_bytes(self, uri, content, **kwargs):+ raise AssertionError("local mode must not write VikingFS")++ async def mkdir(self, uri, exist_ok=False, **kwargs):+ raise AssertionError("local mode must not mkdir on VikingFS")+++class _FailingLocalStore(LocalParseOutputStore):+ async def write_bytes(self, ref, rel_path, content):+ if rel_path.endswith("src/main.py"):+ raise OSError("injected artifact write failure")+ await super().write_bytes(ref, rel_path, content)+++async def test_code_parse_writes_to_local_store(tmp_path, monkeypatch):+ repo = _make_repo(tmp_path)+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))++ parser = CodeRepositoryParser()+ monkeypatch.setattr(parser, "_get_viking_fs", lambda: _ExplodingVikingFS())++ result = await parser.parse(+ str(repo),+ _source_meta={"repo_name": "acme/demo"},+ parse_output_store=store,+ )++ # Artifact ref points at the local backend; temp_dir_path is the local root.+ assert result.artifact_ref is not None+ assert result.artifact_ref.backend == "local"+ assert result.temp_dir_path == result.artifact_ref.root++ # Files landed in the store under repository/... and match source bytes.+ ref = result.artifact_ref+ names = {e.rel_path for e in await store.list(ref, "repository")}+ assert "repository/README.md" in names+ assert await store.read_bytes(ref, "repository/src/main.py") == b"print('hi')"+++async def test_code_parse_rejects_and_cleans_partial_local_artifact(tmp_path, monkeypatch):+ repo = _make_repo(tmp_path)+ artifact_root = tmp_path / "artifacts"+ store = _FailingLocalStore(local_root=str(artifact_root))+ parser = CodeRepositoryParser()+ monkeypatch.setattr(parser, "_get_viking_fs", lambda: _ExplodingVikingFS())++ result = await parser.parse(+ str(repo),+ _source_meta={"repo_name": "acme/demo"},+ parse_output_store=store,+ )++ assert result.temp_dir_path is None+ assert result.artifact_ref is None+ assert result.warnings == [+ "Failed to parse repository: Failed to upload 1 repository file(s): "+ f"Failed to upload {repo / 'src/main.py'}: injected artifact write failure"+ ]+ assert list(artifact_root.iterdir()) == []+++async def test_code_parse_without_store_uses_agfs(tmp_path):+ # No store supplied -> legacy AGFS path. We only assert it does not raise the+ # local-mode branch and yields an AGFS-shaped temp_dir_path via a fake fs.+ repo = _make_repo(tmp_path)++ class _FakeAgfs:+ def __init__(self):+ self.files = {}+ self._n = 0++ def create_temp_uri(self, ctx=None):+ self._n += 1+ return f"viking://temp/code{self._n}"++ async def mkdir(self, uri, exist_ok=False, **kwargs):+ pass++ async def write_file_bytes(self, uri, content, **kwargs):+ self.files[uri] = content++ parser = CodeRepositoryParser()+ fake = _FakeAgfs()+ parser._get_viking_fs = lambda: fake # type: ignore[method-assign]++ result = await parser.parse(str(repo), _source_meta={"repo_name": "acme/demo"})++ assert result.artifact_ref is None or result.artifact_ref.backend == "agfs"+ assert result.temp_dir_path.startswith("viking://temp/")+ assert any("/repository/" in uri for uri in fake.files)+++async def test_code_parse_rejects_and_cleans_partial_agfs_artifact(tmp_path):+ repo = _make_repo(tmp_path)++ class _FailingAgfs:+ def __init__(self):+ self.deleted = []++ def create_temp_uri(self, ctx=None):+ return "viking://temp/code-failed"++ async def mkdir(self, uri, exist_ok=False, **kwargs):+ return None++ async def write_file_bytes(self, uri, content, **kwargs):+ if uri.endswith("src/main.py"):+ raise OSError("injected AGFS write failure")++ async def delete_temp(self, uri, **kwargs):+ self.deleted.append(uri)++ parser = CodeRepositoryParser()+ fake = _FailingAgfs()+ parser._get_viking_fs = lambda: fake # type: ignore[method-assign]++ result = await parser.parse(str(repo), _source_meta={"repo_name": "acme/demo"})++ assert result.temp_dir_path is None+ assert "injected AGFS write failure" in result.warnings[0]+ assert fake.deleted == ["viking://temp/code-failed"]diff --git a/tests/parse/test_directory_understanding_routing.py b/tests/parse/test_directory_understanding_routing.pyindex 17365c91f..c4230754c 100644--- a/tests/parse/test_directory_understanding_routing.py+++ b/tests/parse/test_directory_understanding_routing.py@@ -44,6 +44,10 @@ class _FakeVikingFS:async def write_file_bytes(self, uri: str, content: bytes) -> None:self.files[uri] = content+ async def delete_temp(self, uri: str) -> None:+ prefix = uri.rstrip("/") + "/"+ self.files = {key: value for key, value in self.files.items() if not key.startswith(prefix)}+def _configure_understanding(monkeypatch,@@ -542,7 +546,13 @@ async def test_direct_upload_failure_includes_file_error(video = tmp_path / "video.mp4"video.write_bytes(b"01234567")fake_fs = _FakeVikingFS()- fake_fs.write_file = AsyncMock(side_effect=OSError("storage rejected video.mp4"))++ async def write_file_bytes(uri, content):+ if uri.endswith("video.mp4"):+ raise OSError("storage rejected video.mp4")+ fake_fs.files[uri] = content++ fake_fs.write_file_bytes = AsyncMock(side_effect=write_file_bytes)with patch.object(BaseParser, "_get_viking_fs", return_value=fake_fs):result = await DirectoryParser().parse(str(tmp_path), strict=True)diff --git a/tests/parse/test_document_parser_threading.py b/tests/parse/test_document_parser_threading.pyindex e54d5021f..4db990a3c 100644--- a/tests/parse/test_document_parser_threading.py+++ b/tests/parse/test_document_parser_threading.py@@ -9,7 +9,10 @@ from typing import Any, Callableimport pytestfrom openviking.parse.base import NodeType, ResourceNode, create_parse_result-from openviking.parse.parsers import anydoc, anydoc_converter+from openviking.parse.output import LocalParseOutputStore+from openviking.parse.parsers import anydoc, anydoc_converter, pdf+from openviking.parse.parsers.html import HTMLParser+from openviking.parse.parsers.text import TextParserfrom openviking_cli.utils.config.parser_config import AnydocConfig@@ -84,3 +87,40 @@ async def test_anydoc_parser_offloads_docx_conversion(monkeypatch, tmp_path: Patassert seen["kwargs"]["allowed_media_dirs"] == [storage.media_dir]assert result.source_format == "docx"assert result.parser_name == "AnyDocParser"+++async def test_pdf_parser_forwards_parse_output_store_to_markdown(monkeypatch, tmp_path: Path):+ parser = pdf.PDFParser()+ markdown_parser = parser._get_markdown_parser()+ seen = _stub_markdown_parse(SimpleNamespace(_md_parser=markdown_parser))+ source = tmp_path / "sample.pdf"+ source.write_bytes(b"%PDF-placeholder")+ store = LocalParseOutputStore(str(tmp_path / "artifacts"))++ async def convert(_path, *, resource_name=None):+ return "# converted pdf", {}++ monkeypatch.setattr(parser, "_convert_to_markdown", convert)++ await parser.parse(source, parse_output_store=store)++ assert seen["kwargs"]["parse_output_store"] is store+++ "parser",+ [TextParser(), HTMLParser(), anydoc.AnyDocParser()],+)+async def test_markdown_delegate_parsers_write_local_artifacts(tmp_path: Path, parser) -> None:+ store = LocalParseOutputStore(str(tmp_path / type(parser).__name__))++ result = await parser.parse_content(+ "# title\n\nbody",+ source_path=str(tmp_path / "document.md"),+ parse_output_store=store,+ )++ assert result.artifact_ref is not None+ assert result.artifact_ref.backend == "local"diff --git a/tests/parse/test_feishu_parser_api.py b/tests/parse/test_feishu_parser_api.pyindex eff5ccd83..9b22a6954 100644--- a/tests/parse/test_feishu_parser_api.py+++ b/tests/parse/test_feishu_parser_api.py@@ -1071,28 +1071,30 @@ async def test_add_resource_processor_persists_final_uri_and_cleans_staged_sourc"resource_id": final_uri,}final_complete = task_tracker.complete.await_args_list[-1]- assert final_complete.args == (- "task-1",- {- "status": "success",- "root_uri": final_uri,- "context_count": 9,- "queue_status": {- "Semantic": {- "processed": 0,- "requeue_count": 0,- "error_count": 0,- "errors": [],- },- "Embedding": {- "processed": 9,- "requeue_count": 0,- "error_count": 0,- "errors": [],- },+ final_result = dict(final_complete.args[1])+ telemetry = final_result.pop("telemetry")+ assert telemetry["id"] == telemetry_id+ assert telemetry["summary"]["operation"] == "add_resource_job"+ assert final_complete.args[0] == "task-1"+ assert final_result == {+ "status": "success",+ "root_uri": final_uri,+ "context_count": 9,+ "queue_status": {+ "Semantic": {+ "processed": 0,+ "requeue_count": 0,+ "error_count": 0,+ "errors": [],+ },+ "Embedding": {+ "processed": 9,+ "requeue_count": 0,+ "error_count": 0,+ "errors": [],},},- )+ }assert final_complete.kwargs == {"account_id": "account-1","user_id": "user-1",diff --git a/tests/parse/test_feishu_unified_import.py b/tests/parse/test_feishu_unified_import.pyindex 7884608e8..55cd8f8d3 100644--- a/tests/parse/test_feishu_unified_import.py+++ b/tests/parse/test_feishu_unified_import.py@@ -107,7 +107,6 @@ async def test_directory_executes_url_with_auth_and_preserves_output_parent(fs = FakeVikingFS()parser = DirectoryParser()monkeypatch.setattr(parser, "_get_viking_fs", lambda: fs)- monkeypatch.setattr(parser, "_create_temp_uri", lambda: "viking://temp/import")calls = []async def parse_api(self, source, **options):@@ -143,8 +142,8 @@ async def test_directory_executes_url_with_auth_and_preserves_output_parent(assert calls[0][1].get("understanding_response_id") == ("old" if resume else None)save.assert_awaited_once_with(plan.entries[0].checkpoint_key(plan.root), "response-1")assert result.meta["failed_files"] == []- assert "viking://temp/import/Folder/nested/Cloud/0.md" in fs.files- assert "viking://temp/import/Folder/empty" not in fs.dirs+ assert f"{result.temp_dir_path}/Folder/nested/Cloud/0.md" in fs.files+ assert f"{result.temp_dir_path}/Folder/empty" not in fs.dirsassert "secret" not in str(result.meta)@@ -172,7 +171,6 @@ async def test_directory_omits_source_directories_without_imported_content(fs = FakeVikingFS()parser = DirectoryParser()monkeypatch.setattr(parser, "_get_viking_fs", lambda: fs)- monkeypatch.setattr(parser, "_create_temp_uri", lambda: "viking://temp/empty-filter")parse = AsyncMock(side_effect=ValueError("document parse failed"))monkeypatch.setattr(DirectoryParser, "_parse_file_with_parser", parse)result = await parser.parse(@@ -186,7 +184,7 @@ async def test_directory_omits_source_directories_without_imported_content(assert result.meta["file_count"] == 1assert len(result.meta["failed_files"]) == 1assert result.meta["failed_files"][0]["path"] == "failed/Broken.md"- target = "viking://temp/empty-filter/Folder"+ target = f"{result.temp_dir_path}/Folder"assert fs.files[f"{target}/kept/nested/data.json"] == b'{"body": "keep"}'assert [entry["name"] for entry in await fs.ls(target)] == ["kept"]@@ -199,7 +197,6 @@ async def test_directory_virtual_url_obeys_include_filter(monkeypatch, tmp_path)parser = DirectoryParser()fs = FakeVikingFS()monkeypatch.setattr(parser, "_get_viking_fs", lambda: fs)- monkeypatch.setattr(parser, "_create_temp_uri", lambda: "viking://temp/filter")parse = AsyncMock(side_effect=AssertionError("excluded URL was submitted"))monkeypatch.setattr(UnderstandingAPI, "parse", parse)result = await parser.parse(tmp_path, _feishu_import_plan=plan, include="*.pdf")@@ -374,7 +371,6 @@ async def test_folder_processor_routes_each_content_and_keeps_tree(monkeypatch,processor._accessor_registry = SimpleNamespace(access=access)fs = FakeVikingFS()monkeypatch.setattr(DirectoryParser, "_get_viking_fs", lambda self: fs)- monkeypatch.setattr(DirectoryParser, "_create_temp_uri", lambda self: "viking://temp/tree")calls = []async def parse_api(self, source, **options):@@ -414,9 +410,9 @@ async def test_folder_processor_routes_each_content_and_keeps_tree(monkeypatch,)assert "feishu_access_token" not in binary_optionsassert "lark_file" not in binary_options- assert "viking://temp/tree/Root/Root/0.md" in fs.files- assert "viking://temp/tree/Root/Report/Leaf/0.md" in fs.files- assert "viking://temp/tree/Root/Report/Report/0.md" in fs.files+ assert f"{result.temp_dir_path}/Root/Root/0.md" in fs.files+ assert f"{result.temp_dir_path}/Root/Report/Leaf/0.md" in fs.files+ assert f"{result.temp_dir_path}/Root/Report/Report/0.md" in fs.filesfinally:for resource in resources:resource.is_temporary = True@@ -568,7 +564,6 @@ async def test_directory_failure_cleans_unmerged_artifacts(fs = FakeVikingFS()parser = DirectoryParser()monkeypatch.setattr(parser, "_get_viking_fs", lambda: fs)- monkeypatch.setattr(parser, "_create_temp_uri", lambda: "viking://temp/import")async def parse_api(self, source, **options):name = source.rsplit("/", 1)[-1]@@ -830,7 +825,6 @@ async def test_recursive_wiki_processor_routes_each_content_and_keeps_tree(monkeprocessor._accessor_registry = SimpleNamespace(access=access)fs = FakeVikingFS()monkeypatch.setattr(DirectoryParser, "_get_viking_fs", lambda self: fs)- monkeypatch.setattr(DirectoryParser, "_create_temp_uri", lambda self: "viking://temp/tree")calls = []async def parse_api(self, source, **options):@@ -871,9 +865,9 @@ async def test_recursive_wiki_processor_routes_each_content_and_keeps_tree(monke)assert "feishu_access_token" not in binary_optionsassert "lark_file" not in binary_options- assert "viking://temp/tree/Root/Root/0.md" in fs.files- assert "viking://temp/tree/Root/Report/Leaf/0.md" in fs.files- assert "viking://temp/tree/Root/Report/Report/0.md" in fs.files+ assert f"{result.temp_dir_path}/Root/Root/0.md" in fs.files+ assert f"{result.temp_dir_path}/Root/Report/Leaf/0.md" in fs.files+ assert f"{result.temp_dir_path}/Root/Report/Report/0.md" in fs.filesfinally:for resource in resources:resource.is_temporary = Truediff --git a/tests/parse/test_markdown_apply_layout.py b/tests/parse/test_markdown_apply_layout.pyindex a356a14ca..4ae43c836 100644--- a/tests/parse/test_markdown_apply_layout.py+++ b/tests/parse/test_markdown_apply_layout.py@@ -9,8 +9,10 @@ isolated while replaying their layouts."""import asyncio+from pathlib import Pathfrom unittest.mock import patch+from openviking.parse.output import LocalParseOutputStore, read_artifact_manifestfrom openviking.parse.parsers.base_parser import BaseParserfrom openviking.parse.parsers.markdown import MarkdownParser, _Layout, _LayoutOp@@ -123,3 +125,20 @@ class TestApplyLayout:for name, result in zip(("a", "b"), results, strict=True):uri = f"{result.temp_dir_path}/{name}/{name}.md"assert fake.files[uri] == f"[target](../{name}.txt)"+++async def test_markdown_parser_writes_local_artifact_and_manifest(tmp_path: Path):+ store = LocalParseOutputStore(str(tmp_path / "artifacts"))+ parser = MarkdownParser()++ result = await parser.parse_content(+ "# Hello\n\nworld",+ source_path=str(tmp_path / "hello.md"),+ parse_output_store=store,+ )++ assert result.artifact_ref is not None+ assert result.artifact_ref.backend == "local"+ assert result.artifact_ref.resource_rel == "hello"+ assert Path(result.artifact_ref.root, "hello", "hello.md").read_text() == "# Hello\n\nworld"+ assert set(await read_artifact_manifest(store, result.artifact_ref)) == {"hello/hello.md"}diff --git a/tests/parse/test_markdown_no_split.py b/tests/parse/test_markdown_no_split.pyindex 1f56afed7..bc48b696f 100644--- a/tests/parse/test_markdown_no_split.py+++ b/tests/parse/test_markdown_no_split.py@@ -69,7 +69,7 @@ async def test_flattened_no_split_markdown_creates_temp_root_once(assert result.warnings == []assert fake_fs.directories == {"viking://temp/pdf-no-split"}- assert fake_fs.files == {"viking://temp/pdf-no-split/document.md": content}+ assert fake_fs.files["viking://temp/pdf-no-split/document.md"] == content@pytest.mark.asyncio@@ -142,7 +142,8 @@ async def test_pdf_no_split_converts_to_one_complete_markdown()assert result.parser_name == "PDFParser"- assert fake_fs.files == {- ("viking://temp/pdf-no-split/社交网络中英文剧本/社交网络中英文剧本.md"): content- }+ assert (+ fake_fs.files["viking://temp/pdf-no-split/社交网络中英文剧本/社交网络中英文剧本.md"]+ == content+ )assert not any(uri.endswith(".pdf") for uri in fake_fs.files)diff --git a/tests/parse/test_media_resource_name.py b/tests/parse/test_media_resource_name.pyindex bd189d716..e97f96607 100644--- a/tests/parse/test_media_resource_name.py+++ b/tests/parse/test_media_resource_name.py@@ -17,8 +17,11 @@ from unittest.mock import AsyncMock, MagicMock, patchimport pytestfrom PIL import Image+from openviking.parse.output import LocalParseOutputStore, read_artifact_manifest+from openviking.parse.parsers.media.audio import AudioParserfrom openviking.parse.parsers.media.image import ImageParserfrom openviking.parse.parsers.media.naming import resolve_media_names+from openviking.parse.parsers.media.video import VideoParserfrom openviking_cli.utils.config.parser_config import ImageConfig@@ -143,3 +146,88 @@ async def test_image_parser_falls_back_to_temp_name(tmp_path):result = await parser.parse(str(upload))assert result.root.meta["original_filename"] == "upload_0123456789abcdef.png"+++async def test_image_parser_writes_local_artifact(tmp_path):+ upload = tmp_path / "upload.png"+ content = _png_bytes()+ upload.write_bytes(content)+ store = LocalParseOutputStore(str(tmp_path / "artifacts"))++ result = await ImageParser(config=ImageConfig()).parse(+ upload, resource_name="vacation", parse_output_store=store+ )++ assert result.artifact_ref is not None+ assert result.artifact_ref.backend == "local"+ rel = "vacation_png/vacation.png"+ assert Path(result.artifact_ref.root, rel).read_bytes() == content+ assert set(await read_artifact_manifest(store, result.artifact_ref)) == {rel}+++ ("parser", "filename", "content", "resource_rel"),+ [+ (AudioParser(), "clip.mp3", b"ID3audio", "clip_mp3"),+ (VideoParser(), "clip.mp4", b"\x00\x00\x00\x18ftypvideo", "clip_mp4"),+ ],+)+async def test_audio_and_video_write_local_artifacts(+ tmp_path, parser, filename, content, resource_rel+):+ source = tmp_path / filename+ source.write_bytes(content)+ store = LocalParseOutputStore(str(tmp_path / f"artifacts-{resource_rel}"))++ result = await parser.parse(source, parse_output_store=store)++ assert result.artifact_ref is not None+ assert result.artifact_ref.resource_rel == resource_rel+ rel = f"{resource_rel}/{filename}"+ assert Path(result.artifact_ref.root, rel).read_bytes() == content+ assert set(await read_artifact_manifest(store, result.artifact_ref)) == {rel}+++async def test_image_parser_cleans_local_artifact_when_write_fails(tmp_path):+ class FailingStore(LocalParseOutputStore):+ async def write_bytes(self, ref, rel_path, content):+ raise OSError("disk full")++ upload = tmp_path / "upload.png"+ upload.write_bytes(_png_bytes())+ artifact_root = tmp_path / "artifacts"+ store = FailingStore(str(artifact_root))++ with pytest.raises(OSError, match="disk full"):+ await ImageParser(config=ImageConfig()).parse(upload, parse_output_store=store)++ assert list(artifact_root.iterdir()) == []+++ ("parser", "filename", "content"),+ [+ (AudioParser(), "clip.mp3", b"ID3audio"),+ (VideoParser(), "clip.mp4", b"\x00\x00\x00\x18ftypvideo"),+ ],+)+async def test_audio_and_video_clean_local_artifact_when_write_fails(+ tmp_path, parser, filename, content+):+ class FailingStore(LocalParseOutputStore):+ async def write_bytes(self, ref, rel_path, content):+ raise OSError("disk full")++ source = tmp_path / filename+ source.write_bytes(content)+ artifact_root = tmp_path / f"artifacts-{filename}"+ store = FailingStore(str(artifact_root))++ with pytest.raises(OSError, match="disk full"):+ await parser.parse(source, parse_output_store=store)++ assert list(artifact_root.iterdir()) == []diff --git a/tests/parse/test_parse_output.py b/tests/parse/test_parse_output.pynew file mode 100644index 000000000..8517d0ca4--- /dev/null+++ b/tests/parse/test_parse_output.py@@ -0,0 +1,467 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Contract tests for the parse output store abstraction.++The AGFS backend forwards 1:1 to the VikingFS singleton; the local backend+lays artifacts out under a configured root directory. Both must satisfy the same+byte/directory contract so parsers stay backend-agnostic.+"""++from pathlib import Path++import pytest++from openviking.parse.base import ParseResult, ResourceNode+from openviking.parse.output import (+ ARTIFACT_MANIFEST_NAME,+ AgfsParseOutputStore,+ LocalParseOutputStore,+ ParseArtifactRef,+ ParseArtifactWriter,+ build_parse_output_store,+ copy_artifact_tree,+ read_artifact_manifest,+ resolve_artifact_doc_root,+)+from openviking.utils.content_hash import content_md5+++class _FakeVikingFS:+ """Records the VikingFS calls a parser would make while writing artifacts."""++ def __init__(self) -> None:+ self.files: dict[str, bytes] = {}+ self.dirs: list[str] = []+ self.deleted: list[str] = []+ self._temp_seq = 0+ self.ls_kwargs = []++ def create_temp_uri(self, ctx=None) -> str:+ self._temp_seq += 1+ return f"viking://temp/fake{self._temp_seq}"++ async def mkdir(self, uri: str, exist_ok: bool = False, ctx=None) -> None:+ self.dirs.append(uri)++ async def write_file_bytes(self, uri: str, content: bytes, ctx=None) -> None:+ self.files[uri] = content++ async def read_file_bytes(self, uri: str, ctx=None) -> bytes:+ return self.files[uri]++ async def ls(self, uri: str, ctx=None, **kwargs) -> list[dict]:+ self.ls_kwargs.append(kwargs)+ prefix = f"{uri.rstrip('/')}/"+ entries = {}+ for stored in self.files:+ if stored.startswith(prefix):+ relative = stored[len(prefix) :]+ name, separator, _ = relative.partition("/")+ entries[name] = {+ "name": name,+ "uri": f"{prefix}{name}",+ "isDir": bool(separator),+ }+ return list(entries.values())++ async def delete_temp(self, uri: str, ctx=None) -> None:+ self.deleted.append(uri)+++# ---------------------------------------------------------------------------+# ParseArtifactRef+# ---------------------------------------------------------------------------+++class TestParseArtifactRef:+ def test_serialization_roundtrip(self) -> None:+ ref = ParseArtifactRef(+ backend="agfs",+ root="viking://temp/abc",+ resource_rel="repository",+ root_type="dir",+ )+ assert ParseArtifactRef.from_dict(ref.to_dict()) == ref++ def test_rejects_unknown_backend(self) -> None:+ with pytest.raises(ValueError, match="backend"):+ ParseArtifactRef.from_dict(+ {"backend": "s3", "root": "x", "resource_rel": "", "root_type": "dir"}+ )++ def test_rejects_invalid_root_type(self) -> None:+ with pytest.raises(ValueError, match="root_type"):+ ParseArtifactRef.from_dict(+ {"backend": "agfs", "root": "x", "resource_rel": "", "root_type": "blob"}+ )+++# ---------------------------------------------------------------------------+# Shared contract, parametrized over both backends+# ---------------------------------------------------------------------------+++def _make_store(backend: str, tmp_path):+ if backend == "agfs":+ return AgfsParseOutputStore(viking_fs=_FakeVikingFS())+ return LocalParseOutputStore(local_root=str(tmp_path / "parse-out"))++[email protected]("backend", ["agfs", "local"])+class TestParseOutputStoreContract:+ async def test_create_artifact_reports_backend(self, backend, tmp_path) -> None:+ store = _make_store(backend, tmp_path)+ ref = await store.create_artifact(root_type="dir")+ assert ref.backend == backend+ assert ref.root_type == "dir"+ assert ref.root++ async def test_write_and_read_bytes_roundtrip(self, backend, tmp_path) -> None:+ store = _make_store(backend, tmp_path)+ ref = await store.create_artifact(root_type="dir")++ await store.mkdir(ref, "repo")+ await store.write_bytes(ref, "repo/main.py", b"print(1)")++ assert await store.read_bytes(ref, "repo/main.py") == b"print(1)"++ async def test_write_and_read_text(self, backend, tmp_path) -> None:+ store = _make_store(backend, tmp_path)+ ref = await store.create_artifact(root_type="dir")+ await store.write_text(ref, "note.md", "# hi")+ assert await store.read_text(ref, "note.md") == "# hi"++ async def test_list_returns_relative_entries(self, backend, tmp_path) -> None:+ store = _make_store(backend, tmp_path)+ ref = await store.create_artifact(root_type="dir")+ await store.write_bytes(ref, "a.py", b"a")+ await store.write_bytes(ref, "b.py", b"b")++ names = {entry.name for entry in await store.list(ref, "")}+ assert names == {"a.py", "b.py"}++ async def test_cleanup_is_idempotent(self, backend, tmp_path) -> None:+ store = _make_store(backend, tmp_path)+ ref = await store.create_artifact(root_type="dir")+ await store.write_bytes(ref, "a.py", b"a")++ await store.cleanup(ref)+ await store.cleanup(ref) # must not raise++ async def test_rejects_unsafe_relative_path(self, backend, tmp_path) -> None:+ store = _make_store(backend, tmp_path)+ ref = await store.create_artifact(root_type="dir")+ with pytest.raises(ValueError):+ await store.write_bytes(ref, "../escape.py", b"x")++ async def test_ref_roundtrip_reopens_same_artifact(self, backend, tmp_path) -> None:+ store = _make_store(backend, tmp_path)+ ref = await store.create_artifact(root_type="dir")+ await store.write_bytes(ref, "a.py", b"a")++ # A serialized ref must reopen the same artifact bytes.+ reopened = ParseArtifactRef.from_dict(ref.to_dict())+ assert await store.read_bytes(reopened, "a.py") == b"a"+++# ---------------------------------------------------------------------------+# AGFS-specific: must forward to the VikingFS singleton+# ---------------------------------------------------------------------------+++class TestAgfsParseOutputStore:+ async def test_explicit_context_is_forwarded(self) -> None:+ class _ContextVikingFS(_FakeVikingFS):+ def __init__(self):+ super().__init__()+ self.contexts = []++ async def read_file_bytes(self, uri: str, ctx=None) -> bytes:+ self.contexts.append(ctx)+ return await super().read_file_bytes(uri, ctx=ctx)++ ctx = object()+ vfs = _ContextVikingFS()+ store = AgfsParseOutputStore(viking_fs=vfs, ctx=ctx)+ ref = await store.create_artifact(root_type="dir")+ await store.write_bytes(ref, "a.py", b"a")++ assert await store.read_bytes(ref, "a.py") == b"a"+ assert vfs.contexts == [ctx]++ async def test_write_round_trips_through_vikingfs(self) -> None:+ vfs = _FakeVikingFS()+ store = AgfsParseOutputStore(viking_fs=vfs)+ ref = await store.create_artifact(root_type="dir")++ await store.write_bytes(ref, "repo/main.py", b"print(1)")++ assert vfs.files["viking://temp/fake1/repo/main.py"] == b"print(1)"++ async def test_list_requests_all_children(self) -> None:+ from openviking.storage.viking_fs import LS_ALL_NODES++ vfs = _FakeVikingFS()+ store = AgfsParseOutputStore(viking_fs=vfs)+ ref = await store.create_artifact(root_type="dir")++ await store.list(ref)++ assert vfs.ls_kwargs == [{"show_all_hidden": True, "node_limit": LS_ALL_NODES}]++ async def test_cleanup_deletes_via_vikingfs(self) -> None:+ vfs = _FakeVikingFS()+ store = AgfsParseOutputStore(viking_fs=vfs)+ ref = await store.create_artifact(root_type="dir")++ await store.cleanup(ref)+ await store.cleanup(ref)++ assert vfs.deleted == ["viking://temp/fake1"]++ async def test_move_forwards_explicit_context(self) -> None:+ class _MovingVikingFS(_FakeVikingFS):+ def __init__(self):+ super().__init__()+ self.moves = []++ async def move_file(self, source, target, ctx=None):+ self.moves.append((source, target, ctx))++ ctx = object()+ vfs = _MovingVikingFS()+ store = AgfsParseOutputStore(viking_fs=vfs, ctx=ctx)+ source = await store.create_artifact()+ target = await store.create_artifact()++ await store.move_file(source, "a.txt", target, "nested/a.txt")++ assert vfs.moves == [(f"{source.root}/a.txt", f"{target.root}/nested/a.txt", ctx)]+++# ---------------------------------------------------------------------------+# Local-specific: filesystem layout, isolation, case conflicts+# ---------------------------------------------------------------------------+++class TestLocalParseOutputStore:+ async def test_artifacts_are_isolated(self, tmp_path) -> None:+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))+ a = await store.create_artifact(root_type="dir")+ b = await store.create_artifact(root_type="dir")+ assert a.root != b.root++ await store.write_bytes(a, "f.py", b"a")+ await store.write_bytes(b, "f.py", b"b")+ assert await store.read_bytes(a, "f.py") == b"a"+ assert await store.read_bytes(b, "f.py") == b"b"++ async def test_root_stays_under_configured_local_root(self, tmp_path) -> None:+ root = tmp_path / "out"+ store = LocalParseOutputStore(local_root=str(root))+ ref = await store.create_artifact(root_type="dir")+ assert str(root) in ref.root++ async def test_cleanup_removes_directory(self, tmp_path) -> None:+ from pathlib import Path++ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))+ ref = await store.create_artifact(root_type="dir")+ await store.write_bytes(ref, "a.py", b"a")+ assert Path(ref.root).exists()++ await store.cleanup(ref)+ assert not Path(ref.root).exists()++ async def test_cleanup_rejects_artifact_outside_store_root(self, tmp_path) -> None:+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))+ outside = tmp_path / "outside"+ outside.mkdir()+ ref = ParseArtifactRef(backend="local", root=str(outside))++ with pytest.raises(ValueError, match="escapes"):+ await store.cleanup(ref)++ assert outside.exists()++ async def test_case_only_conflict_is_detected(self, tmp_path) -> None:+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))+ ref = await store.create_artifact(root_type="dir")+ await store.write_bytes(ref, "Readme.md", b"a")+ with pytest.raises(ValueError, match="case"):+ await store.write_bytes(ref, "README.MD", b"b")++ async def test_missing_artifact_read_raises(self, tmp_path) -> None:+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))+ ref = await store.create_artifact(root_type="dir")+ with pytest.raises(FileNotFoundError):+ await store.read_bytes(ref, "missing.py")+++# ---------------------------------------------------------------------------+# Factory+# ---------------------------------------------------------------------------+++class TestBuildParseOutputStore:+ def test_defaults_to_agfs_backend(self) -> None:+ store = build_parse_output_store(viking_fs=_FakeVikingFS())+ assert isinstance(store, AgfsParseOutputStore)++ def test_local_backend_requires_root(self) -> None:+ with pytest.raises(ValueError, match="local_root"):+ build_parse_output_store(backend="local", local_root=None)++ def test_builds_local_backend(self, tmp_path) -> None:+ store = build_parse_output_store(backend="local", local_root=str(tmp_path))+ assert isinstance(store, LocalParseOutputStore)++[email protected]("backend", ["agfs", "local"])+async def test_artifact_writer_records_final_bytes_in_manifest(backend, tmp_path) -> None:+ store = _make_store(backend, tmp_path)+ writer = await ParseArtifactWriter.create(store)++ await writer.write_bytes("doc/a.txt", b"final bytes")+ ref = await writer.finalize(resource_rel="doc")++ assert ref.resource_rel == "doc"+ assert await read_artifact_manifest(store, ref) == {"doc/a.txt": content_md5(b"final bytes")}+++async def test_copy_artifact_tree_merges_files_without_manifest_sidecar(tmp_path) -> None:+ source_store = LocalParseOutputStore(local_root=str(tmp_path / "source"))+ target_store = LocalParseOutputStore(local_root=str(tmp_path / "target"))+ source = await ParseArtifactWriter.create(source_store)+ target = await ParseArtifactWriter.create(target_store)+ await source.write_bytes("wrapper/a.txt", b"a")+ await source.write_bytes("wrapper/nested/b.txt", b"b")+ source_ref = await source.finalize(resource_rel="wrapper")++ copied = await copy_artifact_tree(+ source_store=source_store,+ source_ref=source_ref,+ source_rel="wrapper",+ target=target,+ target_rel="dest",+ )+ target_ref = await target.finalize(resource_rel="dest")++ assert copied == 2+ assert await target_store.read_bytes(target_ref, "dest/a.txt") == b"a"+ assert await target_store.read_bytes(target_ref, "dest/nested/b.txt") == b"b"+ assert await read_artifact_manifest(target_store, target_ref) == {+ "dest/a.txt": content_md5(b"a"),+ "dest/nested/b.txt": content_md5(b"b"),+ }+ assert all(+ entry.name != ARTIFACT_MANIFEST_NAME+ for entry in await target_store.list(target_ref, "dest")+ )+++async def test_copy_artifact_tree_flattens_single_wrapped_file(tmp_path) -> None:+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))+ source = await ParseArtifactWriter.create(store)+ target = await ParseArtifactWriter.create(store)+ await source.write_text("wrapper/file.md", "body")+ source_ref = await source.finalize(resource_rel="wrapper")++ copied = await copy_artifact_tree(+ source_store=store,+ source_ref=source_ref,+ source_rel="",+ target=target,+ target_rel="dest",+ flatten_single_output=True,+ )+ target_ref = await target.finalize(resource_rel="dest")++ assert copied == 1+ assert await store.read_text(target_ref, "dest/file.md") == "body"+ assert not Path(target_ref.root, "dest", "wrapper").exists()+++def test_parse_result_does_not_guess_local_path_is_agfs(tmp_path) -> None:+ result = ParseResult(+ root=ResourceNode(type="root"),+ temp_dir_path=str(tmp_path / "artifact"),+ )++ with pytest.raises(ValueError, match="artifact_ref"):+ result.ensure_artifact_ref()+++async def test_resolve_artifact_doc_root_accepts_flat_single_file(tmp_path) -> None:+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))+ writer = await ParseArtifactWriter.create(store)+ await writer.write_text("document.md", "body")+ ref = await writer.finalize()++ resolved = await resolve_artifact_doc_root(store, ref, flatten_single_file=True)++ assert resolved.doc_name == "document.md"+ assert resolved.doc_rel == "document.md"+ assert resolved.root_is_file is True++[email protected]("backend", ["agfs", "local"])+async def test_resolve_artifact_doc_root_accepts_single_directory(backend, tmp_path) -> None:+ store = _make_store(backend, tmp_path)+ ref = await store.create_artifact(root_type="dir")+ await store.write_bytes(ref, "repo/a.py", b"a")+ await store.write_bytes(ref, "repo/b.py", b"b")++ resolved = await resolve_artifact_doc_root(store, ref, flatten_single_file=False)++ assert resolved.doc_name == "repo"+ assert resolved.doc_rel == "repo"+ assert resolved.root_is_file is False++[email protected]("backend", ["agfs", "local"])+async def test_resolve_artifact_doc_root_flattens_wrapped_single_file(backend, tmp_path) -> None:+ store = _make_store(backend, tmp_path)+ ref = await store.create_artifact(root_type="dir")+ await store.write_bytes(ref, "doc/report.md", b"# hi")++ resolved = await resolve_artifact_doc_root(store, ref, flatten_single_file=True)++ assert resolved.doc_name == "report.md"+ assert resolved.doc_rel == "doc/report.md"+ assert resolved.root_is_file is True++[email protected]("backend", ["agfs", "local"])+async def test_resolve_artifact_doc_root_keeps_multi_file_directory(backend, tmp_path) -> None:+ store = _make_store(backend, tmp_path)+ ref = await store.create_artifact(root_type="dir")+ await store.write_bytes(ref, "doc/a.md", b"a")+ await store.write_bytes(ref, "doc/b.md", b"b")++ resolved = await resolve_artifact_doc_root(store, ref, flatten_single_file=True)++ assert resolved.doc_name == "doc"+ assert resolved.root_is_file is False++[email protected]("backend", ["agfs", "local"])+async def test_resolve_artifact_doc_root_rejects_multiple_roots(backend, tmp_path) -> None:+ store = _make_store(backend, tmp_path)+ ref = await store.create_artifact(root_type="dir")+ await store.write_bytes(ref, "one/a.py", b"a")+ await store.write_bytes(ref, "two/b.py", b"b")++ with pytest.raises(ValueError, match="document"):+ await resolve_artifact_doc_root(store, ref, flatten_single_file=False)diff --git a/tests/parse/test_understanding_api_artifact_images.py b/tests/parse/test_understanding_api_artifact_images.pyindex b9317e799..e70e11555 100644--- a/tests/parse/test_understanding_api_artifact_images.py+++ b/tests/parse/test_understanding_api_artifact_images.py@@ -9,6 +9,7 @@ from openviking.parse.image_rewrite import (IMAGE_MAPPINGS_FILENAME,build_artifact_image_mappings,)+from openviking.parse.output import LocalParseOutputStore, read_artifact_manifestfrom openviking.parse.understanding_api import UnderstandingAPI@@ -76,14 +77,15 @@ async def test_unpack_artifact_writes_image_mapping_sidecar(tmp_path: Path):fake_fs = _FakeVikingFS()api = UnderstandingAPI.__new__(UnderstandingAPI)with patch("openviking.parse.understanding_api.get_viking_fs", return_value=fake_fs):- temp_uri = await api._unpack_zip_to_temp_dir(zip_path, "resource")+ artifact_ref = await api._unpack_zip_to_temp_dir(zip_path, "resource")- assert temp_uri == "viking://temp/artifact"- sidecar_uri = f"{temp_uri}/resource/{IMAGE_MAPPINGS_FILENAME}"+ assert artifact_ref.root == "viking://temp/artifact"+ assert artifact_ref.resource_rel == "resource"+ sidecar_uri = f"{artifact_ref.root}/resource/{IMAGE_MAPPINGS_FILENAME}"assert json.loads(fake_fs.files[sidecar_uri]) == {"Ov测试_1.md": {"Ov测试_1_img1.png": "Ov测试_1_img1.png"}}- assert fake_fs.files[f"{temp_uri}/resource/Ov测试_1_img1.png"] == b"png"+ assert fake_fs.files[f"{artifact_ref.root}/resource/Ov测试_1_img1.png"] == b"png"@pytest.mark.asyncio@@ -100,3 +102,19 @@ async def test_unpack_artifact_cleans_temp_on_failure(tmp_path: Path):await api._unpack_zip_to_temp_dir(invalid_zip, "resource")assert fake_fs.deleted_temps == ["viking://temp/artifact"]+++async def test_unpack_artifact_supports_local_output_store(tmp_path: Path):+ zip_path = tmp_path / "artifact.zip"+ with zipfile.ZipFile(zip_path, "w") as archive:+ archive.writestr("artifact/section.md", "content")++ store = LocalParseOutputStore(str(tmp_path / "artifacts"))+ api = UnderstandingAPI.__new__(UnderstandingAPI)+ ref = await api._unpack_zip_to_temp_dir(zip_path, "resource", parse_output_store=store)++ assert ref.backend == "local"+ assert ref.resource_rel == "resource"+ assert Path(ref.root, "resource", "section.md").read_text() == "content"+ assert set(await read_artifact_manifest(store, ref)) == {"resource/section.md"}diff --git a/tests/resource/test_shared_source.py b/tests/resource/test_shared_source.pynew file mode 100644index 000000000..ffdc99d2f--- /dev/null+++ b/tests/resource/test_shared_source.py@@ -0,0 +1,116 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Tests for the shared-upload SOURCE reference and worker materialization."""++import pytest++from openviking.parse.accessors.base import SourceType+from openviking.resource.shared_source import SharedSource, materialize_shared_source+from openviking.server.identity import RequestContext, Role+from openviking_cli.session.user_id import UserIdentifier+++def _valid_payload() -> dict:+ return {+ "temp_file_id": "shared_2026010112abcdef",+ "content_uri": "viking://upload/2026010112/abcdef/content",+ "account_id": "acct-1",+ "original_filename": "repo.zip",+ "file_ext": ".zip",+ "meta": {"resolved_extension": ".zip"},+ }+++class TestSharedSourceValidation:+ def test_roundtrip(self) -> None:+ src = SharedSource.from_dict(_valid_payload())+ assert src.temp_file_id == "shared_2026010112abcdef"+ assert src.content_uri == "viking://upload/2026010112/abcdef/content"+ assert SharedSource.from_dict(src.to_dict()) == src++ def test_rejects_non_shared_temp_file_id(self) -> None:+ payload = _valid_payload()+ payload["temp_file_id"] = "upload_plainlocal"+ with pytest.raises(ValueError, match="temp_file_id"):+ SharedSource.from_dict(payload)++ def test_rejects_content_uri_outside_upload_root(self) -> None:+ payload = _valid_payload()+ payload["content_uri"] = "viking://resources/secret/content"+ with pytest.raises(ValueError, match="content_uri"):+ SharedSource.from_dict(payload)++ def test_rejects_missing_account(self) -> None:+ payload = _valid_payload()+ payload["account_id"] = ""+ with pytest.raises(ValueError, match="account_id"):+ SharedSource.from_dict(payload)+++class _FakeVikingFS:+ def __init__(+ self,+ content: bytes,+ content_uri: str,+ account_id: str,+ *,+ exists: bool = True,+ ) -> None:+ self._content = content+ self._content_uri = content_uri+ self._account_id = account_id+ self._exists = exists+ self.read_calls: list[str] = []+ self.read_contexts = []++ async def exists(self, uri: str, *, ctx) -> bool:+ return self._exists and uri == self._content_uri++ async def read_file_bytes(self, uri: str, *, ctx) -> bytes:+ self.read_calls.append(uri)+ self.read_contexts.append(ctx)+ if uri != self._content_uri:+ raise FileNotFoundError(uri)+ return self._content+++def _ctx() -> RequestContext:+ return RequestContext(user=UserIdentifier("acct-1", "user-1"), role=Role.USER)+++class TestMaterializeSharedSource:+ async def test_downloads_content_once_to_local_file(self) -> None:+ src = SharedSource.from_dict(_valid_payload())+ vfs = _FakeVikingFS(b"PK\x03\x04zip-bytes", src.content_uri, "acct-1")++ resource = await materialize_shared_source(src, viking_fs=vfs, ctx=_ctx())+ try:+ assert resource.source_type == SourceType.LOCAL+ assert resource.path.exists()+ assert resource.path.suffix == ".zip"+ assert resource.path.read_bytes() == b"PK\x03\x04zip-bytes"+ # Worker downloads the shared object exactly once; API never staged it.+ assert vfs.read_calls == [src.content_uri]+ assert vfs.read_contexts[0].role == "root"+ assert vfs.read_contexts[0].account_id == "acct-1"+ finally:+ resource.cleanup()+ assert not resource.path.exists()++ async def test_missing_content_fails_without_writing_file(self) -> None:+ src = SharedSource.from_dict(_valid_payload())+ vfs = _FakeVikingFS(b"", "viking://upload/other/leaf/content", "acct-1")++ with pytest.raises(ValueError, match="missing"):+ await materialize_shared_source(src, viking_fs=vfs, ctx=_ctx())++ async def test_exact_read_succeeds_even_when_exists_is_stale(self) -> None:+ src = SharedSource.from_dict(_valid_payload())+ vfs = _FakeVikingFS(b"payload", src.content_uri, "acct-1", exists=False)++ resource = await materialize_shared_source(src, viking_fs=vfs, ctx=_ctx())+ try:+ assert resource.path.read_bytes() == b"payload"+ finally:+ resource.cleanup()diff --git a/tests/server/test_admin_rebuild_api.py b/tests/server/test_admin_rebuild_api.pyindex 4762ba1c3..c94dd9db8 100644--- a/tests/server/test_admin_rebuild_api.py+++ b/tests/server/test_admin_rebuild_api.py@@ -1572,7 +1572,7 @@ async def test_reindex_resource_vectors_non_recursive_skips_tree(monkeypatch):async def exists(self, uri, ctx=None):return True- async def stat(self, uri, ctx=None):+ async def stat(self, uri, ctx=None, skip_count=True):return {"isDir": True}async def fail_tree_all(*args, **kwargs):@@ -2029,7 +2029,9 @@ async def test_reindex_upsert_context_omits_search_tags_without_ingest_options(mdef fake_from_context(context):captured["meta"] = dict(context.meta or {})- return _FakeMsg()+ msg = _FakeMsg()+ msg.context_data = context.to_dict()+ return msgmonkeypatch.setattr("openviking.service.reindex_executor.EmbeddingMsgConverter.from_context",@@ -2051,10 +2053,12 @@ async def test_reindex_upsert_context_omits_search_tags_without_ingest_options(mcontext_type="resource",level=ContextLevel.DETAIL,ctx=ctx,+ md5="final-md5",)assert "search_tags" not in captured["meta"]assert "search_tags" not in captured["msg"].context_data+ assert captured["msg"].context_data["md5"] == "final-md5"@pytest.mark.asyncio@@ -2066,9 +2070,12 @@ async def test_reindex_resource_vectors_parallelize_files_and_isolate_failures(masync def exists(self, uri, ctx=None):return True- async def stat(self, uri, ctx=None):+ async def stat(self, uri, ctx=None, skip_count=True):return {"isDir": True}+ async def read_file_bytes(self, uri, ctx=None):+ return uri.encode()+async def tree(self,uri,@@ -2094,7 +2101,7 @@ async def test_reindex_resource_vectors_parallelize_files_and_isolate_failures(masync def fake_best_file_summary(self, uri, *, ctx):return f"summary:{uri.rsplit('/', 1)[-1]}"- async def fake_best_resource_file_vector_text(self, uri, summary, ctx):+ async def fake_best_resource_file_vector_text(self, uri, summary, ctx, file_content=None):return summaryseen = []@@ -2281,9 +2288,12 @@ async def test_reindex_resource_l2_falls_back_to_vector_text_when_summary_missinasync def exists(self, uri, ctx=None):return True- async def stat(self, uri, ctx=None):+ async def stat(self, uri, ctx=None, skip_count=True):return {"isDir": True}+ async def read_file_bytes(self, uri, ctx=None):+ return b"raw file body"+async def tree(self,uri,@@ -2308,7 +2318,7 @@ async def test_reindex_resource_l2_falls_back_to_vector_text_when_summary_missinasync def fake_best_file_summary(self, uri, *, ctx):return ""- async def fake_best_resource_file_vector_text(self, uri, summary, ctx):+ async def fake_best_resource_file_vector_text(self, uri, summary, ctx, file_content=None):return "raw file body"async def fake_upsert_context(self, **kwargs):@@ -2341,6 +2351,52 @@ async def test_reindex_resource_l2_falls_back_to_vector_text_when_summary_missinassert seen["viking://resources/demo/file.txt"]["abstract"] == "raw file body"+async def test_reindex_resource_file_writes_md5_from_same_bytes(monkeypatch):+ from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters+ from openviking.utils.content_hash import content_md5++ file_uri = "viking://resources/demo/file.txt"+ raw = b"current body"++ class FakeVikingFS:+ async def exists(self, uri, ctx=None):+ return True++ async def stat(self, uri, ctx=None, skip_count=True):+ return {"isDir": False}++ async def read_file_bytes(self, uri, ctx=None):+ assert uri == file_uri+ return raw++ seen = {}++ async def fake_best_file_summary(self, uri, *, ctx):+ return "summary"++ async def fake_best_resource_file_vector_text(self, uri, summary, ctx, file_content=None):+ return "current body"++ async def fake_upsert_context(self, **kwargs):+ seen.update(kwargs)++ monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())+ monkeypatch.setattr(ReindexExecutor, "_best_file_summary", fake_best_file_summary)+ monkeypatch.setattr(+ ReindexExecutor,+ "_best_resource_file_vector_text",+ fake_best_resource_file_vector_text,+ )+ monkeypatch.setattr(ReindexExecutor, "_upsert_context", fake_upsert_context)++ counters = _ReindexCounters()+ ctx = RequestContext(user=UserIdentifier(account_id="test", user_id="alice"), role=Role.ROOT)+ await ReindexExecutor()._reindex_resource_vectors(uri=file_uri, counters=counters, ctx=ctx)++ assert seen["md5"] == content_md5(raw)++@pytest.mark.asyncioasync def test_reindex_resource_vector_text_uses_existing_record_for_non_text(monkeypatch):from openviking.service.reindex_executor import ReindexExecutor@@ -2602,9 +2658,12 @@ async def test_reindex_resource_vectors_accepts_single_file_uri(monkeypatch):async def exists(self, uri, ctx=None):return True- async def stat(self, uri, ctx=None):+ async def stat(self, uri, ctx=None, skip_count=True):return {"isDir": False}+ async def read_file_bytes(self, uri, ctx=None):+ return b"file summary"+async def tree(self, *args, **kwargs):raise AssertionError("single-file reindex should not call tree")@@ -2613,7 +2672,7 @@ async def test_reindex_resource_vectors_accepts_single_file_uri(monkeypatch):async def fake_best_file_summary(self, uri, *, ctx):return "file summary"- async def fake_best_resource_file_vector_text(self, uri, summary, ctx):+ async def fake_best_resource_file_vector_text(self, uri, summary, ctx, file_content=None):return summaryasync def fake_upsert_context(self, **kwargs):diff --git a/tests/server/test_content_batch_write.py b/tests/server/test_content_batch_write.pyindex 0d2384927..7b15579b2 100644--- a/tests/server/test_content_batch_write.py+++ b/tests/server/test_content_batch_write.py@@ -8,6 +8,7 @@ from openviking.session.memory.dataclass import MemoryFilefrom openviking.session.memory.utils import MemoryFileUtilsfrom openviking.storage.content_write import ContentWriteCoordinatorfrom openviking.storage.queuefs.semantic_ops.freshness_policy import FreshnessAction+from openviking.utils.content_hash import content_md5from openviking_cli.exceptions import (AlreadyExistsError,InvalidArgumentError,@@ -187,6 +188,45 @@ async def test_batch_releases_file_locks_before_one_aggregated_refresh(monkeypatassert locks.releases == 1+async def test_batch_passes_final_md5s_and_old_abstracts_to_refresh(monkeypatch):+ root = "viking://resources/wiki"+ existing = f"{root}/existing.md"+ created = f"{root}/new.md"+ vfs = _VFS(root, {existing: "old"})++ class _VikingDB:+ async def get_l2_diff_records_by_uris(self, uris, *, ctx):+ assert uris == [existing]+ return {existing: {"abstract": "old abstract", "md5": "old-md5"}}++ coordinator = ContentWriteCoordinator(vfs, vikingdb=_VikingDB())+ captured = {}++ async def refresh(**kwargs):+ captured.update(kwargs)+ return None++ monkeypatch.setattr(coordinator, "_refresh_batch", refresh)+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)++ await coordinator.batch_write(+ root_uri=root,+ operations=[+ {"uri": existing, "content": "new", "mode": "replace"},+ {"uri": created, "content": "created", "mode": "create"},+ ],+ ctx=ctx,+ wait=False,+ )++ assert captured["file_md5s"] == {+ existing: content_md5(b"new"),+ created: content_md5(b"created"),+ }+ assert captured["file_abstracts"] == {existing: "old abstract"}++@pytest.mark.asyncioasync def test_batch_replace_memory_preserves_metadata(monkeypatch):root = "viking://user/default/memories/preferences"@@ -269,7 +309,10 @@ async def test_batch_upserts_binary_content(monkeypatch):vfs = _VFS(root, {image: original})coordinator = ContentWriteCoordinator(vfs)+ refresh_calls = []+async def refresh(**kwargs):+ refresh_calls.append(kwargs)return Nonemonkeypatch.setattr(coordinator, "_refresh_batch", refresh)@@ -284,6 +327,7 @@ async def test_batch_upserts_binary_content(monkeypatch):)assert result["updated"] == [image]assert vfs.files[image] == replacement+ assert refresh_calls[0]["file_md5s"] == {image: content_md5(replacement)}retry = await coordinator.batch_write(root_uri=root, operations=[operation], ctx=ctx, wait=Falsediff --git a/tests/server/test_content_write_service.py b/tests/server/test_content_write_service.pyindex 9af29ab41..2e9285ad6 100644--- a/tests/server/test_content_write_service.py+++ b/tests/server/test_content_write_service.py@@ -869,7 +869,7 @@ async def test_memory_write_wait_skips_semantic_queue_and_releases_write_lock(moasync def _fake_refresh_schema_overview(**kwargs):del kwargs- return None+ return Truemonkeypatch.setattr(coordinator, "_write_in_place", _fake_write_in_place)monkeypatch.setattr(coordinator, "_wait_for_request", _fail_wait_for_request)@@ -1288,7 +1288,7 @@ async def test_resource_write_anchors_nested_file_to_direct_parent():"""A resource content write anchors the semantic refresh at the written file'sdirect parent directory (anchor_to_parent=True), so the changed file is a direct- child of the DAG run root: its own L2 vector and the parent's L0/L1 are generated+ child of the semantic-tree root: its own L2 vector and the parent's L0/L1 are generatedfrom a single-directory run instead of a recursive walk of the whole project subtree.set_tags keeps the project-root collapse (the default), which the derived``.abstract.md`` sidecar mapping relies on."""diff --git a/tests/server/test_skill_update_cancellation.py b/tests/server/test_skill_update_cancellation.pyindex 889c6e4db..5741b34c5 100644--- a/tests/server/test_skill_update_cancellation.py+++ b/tests/server/test_skill_update_cancellation.py@@ -64,7 +64,7 @@ async def _indexed_record(client, uri, level):async def test_update_cancellation_restores_package_and_index(client, tmp_path, monkeypatch, cancel_mode):- from openviking.storage.queuefs import semantic_dag+ from openviking.storage.queuefs import semantic_executorname = "timeout-package-rollback"old_upload = await _upload_package(@@ -109,7 +109,7 @@ async def test_update_cancellation_restores_package_and_index(finished = set()task_owners = {}original_summary = SemanticProcessor._generate_single_file_summary- original_scheduler = semantic_dag.get_semantic_node_scheduler+ original_scheduler = semantic_executor.get_semantic_tree_schedulerasync def hold_summary(self, file_path, *args, **kwargs):task_context = get_task_context()@@ -130,7 +130,7 @@ async def test_update_cancellation_restores_package_and_index(monkeypatch.setattr(SemanticProcessor, "_generate_single_file_summary", hold_summary)monkeypatch.setattr(- semantic_dag, "get_semantic_node_scheduler", lambda _workers: original_scheduler(2)+ semantic_executor, "get_semantic_tree_scheduler", lambda _workers: original_scheduler(2))updating = asyncio.create_task(client.put(diff --git a/tests/server/test_skill_update_retry.py b/tests/server/test_skill_update_retry.pyindex d10604c26..e2a15381e 100644--- a/tests/server/test_skill_update_retry.py+++ b/tests/server/test_skill_update_retry.py@@ -9,7 +9,7 @@ import threadingimport pytestfrom openviking.storage.queuefs import get_queue_manager-from openviking.storage.queuefs.semantic_dag import SemanticDagExecutor+from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutorfrom tests.server.test_api_skills import _add_skill, _skill_mdfrom tests.server.test_api_skills import _stub_mcp_endpoint as _stub_mcp_endpointfrom tests.server.test_skill_update_cancellation import _download, _wait_until@@ -17,7 +17,7 @@ from tests.server.test_skill_update_lock import _assert_locked, _ctxdef _fail_first_run(monkeypatch, root, *, block_retries=False):- original = SemanticDagExecutor.run+ original = SemanticTreeExecutor.runattempts = []async def fail_once(self, root_uri):@@ -29,7 +29,7 @@ def _fail_first_run(monkeypatch, root, *, block_retries=False):await asyncio.Event().wait()return await original(self, root_uri)- monkeypatch.setattr(SemanticDagExecutor, "run", fail_once)+ monkeypatch.setattr(SemanticTreeExecutor, "run", fail_once)return attemptsdiff --git a/tests/service/test_resource_service_reserved_target_cleanup.py b/tests/service/test_resource_service_reserved_target_cleanup.pyindex 19748b17d..121f681ad 100644--- a/tests/service/test_resource_service_reserved_target_cleanup.py+++ b/tests/service/test_resource_service_reserved_target_cleanup.py@@ -5,7 +5,8 @@ import pytestfrom openviking.server.identity import RequestContext, Rolefrom openviking.service.resource_service import ResourceService-from openviking.storage.queuefs.add_resource_msg import AddResourceMsg+from openviking.storage.queuefs.add_resource_msg import AddResourceMsg, AddResourcePhase+from openviking.storage.queuefs.add_resource_processor import AddResourceProcessorfrom openviking_cli.exceptions import FailedPreconditionErrorfrom openviking_cli.session.user_id import UserIdentifier@@ -84,6 +85,38 @@ async def test_cleanup_reserved_target_preserves_nonempty_directory():)+async def test_cancelled_post_process_cleans_agfs_plan_artifact():+ viking_fs = SimpleNamespace(+ delete_temp=AsyncMock(),+ _async_agfs=SimpleNamespace(),+ )+ processor = AddResourceProcessor(SimpleNamespace(), "add_resource", viking_fs)+ msg = AddResourceMsg(+ task_id="task-1",+ job_phase=AddResourcePhase.POST_PROCESS,+ root_uri="viking://resources/repo",+ account_id="acct",+ user_id="alice",+ role="user",+ prepared={+ "root_uri": "viking://resources/repo",+ "artifact_ref": {+ "backend": "agfs",+ "root": "viking://temp/task-1",+ "resource_rel": "repository",+ "root_type": "dir",+ },+ "plan_artifact_committed": True,+ },+ )+ ctx = RequestContext(user=UserIdentifier("acct", "alice"), role=Role.USER)++ await processor._release_cancelled_resources(msg, ctx)++ viking_fs.delete_temp.assert_awaited_once_with("viking://temp/task-1", ctx=ctx)++@pytest.mark.asyncio@pytest.mark.parametrize(("target_preexisting", "cleanup_empty_target_on_failure"),diff --git a/tests/service/test_task_queue_middleware.py b/tests/service/test_task_queue_middleware.pyindex 9811b8e5c..3d79bc89e 100644--- a/tests/service/test_task_queue_middleware.py+++ b/tests/service/test_task_queue_middleware.py@@ -22,6 +22,8 @@ from openviking.storage.queuefs.named_queue import DequeueHandlerBase, NamedQueufrom openviking.storage.queuefs.process_result import ProcessOutcome, ProcessResultfrom openviking.storage.queuefs.queue_middleware import QueueMiddlewarefrom openviking.storage.queuefs.session_commit_processor import SessionCommitProcessor+from openviking.telemetry import OperationTelemetry, bind_telemetry+from openviking.utils.log_correlation import log_correlation@pytest.fixture@@ -59,6 +61,25 @@ async def enqueue_task(queue, transport, task_id="task-1"):return {"id": "message-1", "data": transport.write.await_args.args[1].decode()}+def test_log_correlation_uses_bound_task_and_telemetry() -> None:+ telemetry = OperationTelemetry(operation="add_resource_job", enabled=True)++ with (+ bind_task_context("task-123", "account", "user"),+ bind_telemetry(telemetry),+ ):+ assert log_correlation(message_id="semantic-456") == (+ f"task_id=task-123 telemetry_id={telemetry.telemetry_id} message_id=semantic-456"+ )+++def test_log_correlation_accepts_explicit_queue_ids_without_context() -> None:+ assert (+ log_correlation(task_id="task-1", telemetry_id="tm-1", message_id="embedding-2")+ == "task_id=task-1 telemetry_id=tm-1 message_id=embedding-2"+ )++async def test_enqueue_registers_before_write_and_ack_finalizes_before_delete(tracked_queue):queue, index, transport, finalize, _ = tracked_queue@@ -227,16 +248,19 @@ async def test_process_result_tracks_children_and_errors(tracked_queue):queue, index, transport, _, _ = tracked_queuemessage = await enqueue_task(queue, transport)contexts = []+ correlations = []class Handler(DequeueHandlerBase):async def on_dequeue(self, data):contexts.append(get_task_context())+ correlations.append(log_correlation(telemetry_id="tm-1", message_id="semantic-1"))await queue.enqueue({"child": True})return ProcessResult.failed("failed work")queue.set_dequeue_handler(Handler())assert (await queue.process_dequeued(message)).outcome is ProcessOutcome.FAILEDassert contexts[0].task_id == "task-1"+ assert correlations == ["task_id=task-1 telemetry_id=tm-1 message_id=semantic-1"]assert get_task_context() is Noneassert index.failure("task-1") == "failed work"child_payload = next(diff --git a/tests/storage/test_collection_schemas.py b/tests/storage/test_collection_schemas.pyindex c62aba5b8..aed262f89 100644--- a/tests/storage/test_collection_schemas.py+++ b/tests/storage/test_collection_schemas.py@@ -25,11 +25,12 @@ from openviking.storage.errors import (VikingDBException,)from openviking.storage.expr import Eq+from openviking.storage.index_action import IndexActionfrom openviking.storage.queuefs.embedding_msg import EmbeddingMsgfrom openviking.storage.queuefs.process_result import ProcessOutcomefrom openviking.storage.vector_ids import vector_record_idfrom openviking.storage.vectordb import engine as vectordb_engine-from openviking.storage.vectordb.collection.result import UpsertDataResult+from openviking.storage.vectordb.collection.result import UpdateResult, UpsertDataResultfrom openviking.storage.vectordb.collection.vikingdb_clients import VikingDBClientfrom openviking.storage.vectordb.collection.vikingdb_collection import VikingDBCollectionfrom openviking.storage.vectordb.collection.volcengine_api_key_collection import (@@ -137,6 +138,443 @@ def _build_queue_payload_for_account(account_id: str) -> dict:return {"data": json.dumps(msg.to_dict())}+def _build_operation_payload(msg: EmbeddingMsg) -> dict:+ return {"data": msg.to_json()}+++async def test_embedding_handler_delete_skips_embedder_and_strictly_deletes_ids(monkeypatch):+ class _DeletingVikingDB:+ is_closing = False++ def __init__(self):+ self.deleted = []++ async def strict_delete(self, ids, *, ctx):+ self.deleted.append((list(ids), ctx.account_id))+ return len(ids)++ config = _DummyConfig(_DummyEmbedder())+ monkeypatch.setattr(+ "openviking_cli.utils.config.get_openviking_config",+ lambda: config,+ )+ monkeypatch.setattr(+ config.embedding,+ "get_embedder",+ lambda: (_ for _ in ()).throw(AssertionError("embedder must stay lazy")),+ )+ vikingdb = _DeletingVikingDB()+ handler = TextEmbeddingHandler(vikingdb)+ msg = EmbeddingMsg.for_delete(+ record_ids=["l0-id", "l1-id"],+ context_data={"account_id": "acct", "uri": "viking://resources/repo"},+ )++ result = await handler.on_dequeue(_build_operation_payload(msg))++ assert result.outcome is ProcessOutcome.SUCCESS+ assert result.value == {"deleted_count": 2}+ assert vikingdb.deleted == [(["l0-id", "l1-id"], "acct")]+++async def test_embedding_handler_update_fields_reads_merges_and_uses_strict_partial_update(+ monkeypatch,+):+ existing = {+ "id": "l2-id",+ "uri": "viking://resources/repo/a.py",+ "account_id": "acct",+ "abstract": "old abstract",+ "md5": "old-md5",+ "vector": [0.1, 0.2],+ }++ class _UpdatingVikingDB:+ is_closing = False++ def __init__(self):+ self.reads = []+ self.updates = []++ async def get_strict(self, ids, *, ctx):+ self.reads.append((list(ids), ctx.account_id))+ return [dict(existing)]++ async def update(self, data, *, ctx):+ self.updates.append((dict(data), ctx.account_id))+ return UpdateResult(ok=True, ids=[data["id"]], updated_count=1)++ config = _DummyConfig(_DummyEmbedder())+ monkeypatch.setattr(+ "openviking_cli.utils.config.get_openviking_config",+ lambda: config,+ )+ monkeypatch.setattr(+ config.embedding,+ "get_embedder",+ lambda: (_ for _ in ()).throw(AssertionError("embedder must stay lazy")),+ )+ vikingdb = _UpdatingVikingDB()+ handler = TextEmbeddingHandler(vikingdb)+ msg = EmbeddingMsg.for_update_fields(+ record_id="l2-id",+ fields={"abstract": "new abstract", "md5": "new-md5"},+ context_data={"account_id": "acct", "uri": existing["uri"]},+ )++ result = await handler.on_dequeue(_build_operation_payload(msg))++ assert vikingdb.reads == [(["l2-id"], "acct")]+ assert len(vikingdb.updates) == 1+ updated, account_id = vikingdb.updates[0]+ assert account_id == "acct"+ assert updated == {+ "id": "l2-id",+ "abstract": "new abstract",+ "md5": "new-md5",+ "updated_at": updated["updated_at"],+ }+ assert result.outcome is ProcessOutcome.SUCCESS+ assert result.value == updated+++async def test_embedding_handler_update_fields_append_uses_latest_tags_and_skips_noop(+ monkeypatch,+):+ existing = {+ "id": "l2-id",+ "uri": "viking://resources/repo/a.py",+ "account_id": "acct",+ "level": 2,+ "vector": [0.1, 0.2],+ "search_tags": ["env=old", "scope=new"],+ }++ class _UpdatingVikingDB:+ is_closing = False++ def __init__(self):+ self.updates = []++ async def get_strict(self, ids, *, ctx):+ return [dict(existing)]++ async def update(self, data, *, ctx):+ self.updates.append(dict(data))+ return UpdateResult(ok=True, ids=[data["id"]], updated_count=1)++ config = _DummyConfig(_DummyEmbedder())+ monkeypatch.setattr("openviking_cli.utils.config.get_openviking_config", lambda: config)+ monkeypatch.setattr(+ config.embedding,+ "get_embedder",+ lambda: (_ for _ in ()).throw(AssertionError("embedder must stay lazy")),+ )+ vikingdb = _UpdatingVikingDB()+ handler = TextEmbeddingHandler(vikingdb)+ msg = EmbeddingMsg.for_update_fields(+ record_id="l2-id",+ fields={"search_tags": ["scope=new"]},+ field_modes={"search_tags": "append"},+ context_data={"account_id": "acct", "uri": existing["uri"]},+ )++ result = await handler.on_dequeue(_build_operation_payload(msg))++ assert result.outcome is ProcessOutcome.SUCCESS+ assert result.value == {"id": "l2-id", "status": "skipped"}+ assert vikingdb.updates == []+++async def test_embedding_handler_update_fields_replace_and_clear_do_not_merge_old_tags(+ monkeypatch,+):+ existing = {+ "id": "l2-id",+ "uri": "viking://resources/repo/a.py",+ "account_id": "acct",+ "level": 2,+ "vector": [0.1, 0.2],+ "search_tags": ["env=old"],+ }++ class _UpdatingVikingDB:+ is_closing = False++ def __init__(self):+ self.updates = []++ async def get_strict(self, ids, *, ctx):+ return [dict(existing)]++ async def update(self, data, *, ctx):+ self.updates.append(dict(data))+ return UpdateResult(ok=True, ids=[data["id"]], updated_count=1)++ config = _DummyConfig(_DummyEmbedder())+ monkeypatch.setattr("openviking_cli.utils.config.get_openviking_config", lambda: config)+ monkeypatch.setattr(+ config.embedding,+ "get_embedder",+ lambda: (_ for _ in ()).throw(AssertionError("embedder must stay lazy")),+ )+ vikingdb = _UpdatingVikingDB()+ handler = TextEmbeddingHandler(vikingdb)++ for tags in (["scope=new"], []):+ msg = EmbeddingMsg.for_update_fields(+ record_id="l2-id",+ fields={"search_tags": tags},+ field_modes={"search_tags": "replace"},+ context_data={"account_id": "acct", "uri": existing["uri"]},+ )+ result = await handler.on_dequeue(_build_operation_payload(msg))+ assert result.outcome is ProcessOutcome.SUCCESS++ assert [item["search_tags"] for item in vikingdb.updates] == [["scope=new"], []]+++async def test_embedding_handler_update_fields_read_failure_does_not_create(monkeypatch):+ class _BrokenVikingDB:+ is_closing = False++ async def get_strict(self, ids, *, ctx):+ raise RuntimeError("read failed")++ async def upsert(self, data, *, ctx, options=UpsertOptions()):+ raise AssertionError("read failure must not be treated as not found")++ config = _DummyConfig(_DummyEmbedder())+ monkeypatch.setattr("openviking_cli.utils.config.get_openviking_config", lambda: config)+ monkeypatch.setattr(+ config.embedding,+ "get_embedder",+ lambda: (_ for _ in ()).throw(AssertionError("embedder must stay lazy")),+ )+ handler = TextEmbeddingHandler(_BrokenVikingDB())+ msg = EmbeddingMsg.for_update_fields(+ record_id="l2-id",+ fields={"search_tags": ["scope=new"]},+ field_modes={"search_tags": "append"},+ initial_fields={+ "uri": "viking://resources/repo/a.py",+ "account_id": "acct",+ "level": 2,+ "vector": [0.1, 0.2],+ },+ context_data={"account_id": "acct", "uri": "viking://resources/repo/a.py"},+ )++ result = await handler.on_dequeue(_build_operation_payload(msg))++ assert result.outcome is ProcessOutcome.FAILED+ assert "read failed" in result.error+++async def test_embedding_handler_update_fields_not_found_creates_from_complete_initial_fields(+ monkeypatch,+):+ class _MissingVikingDB:+ is_closing = False++ def __init__(self):+ self.upserts = []++ async def get_strict(self, ids, *, ctx):+ return []++ async def upsert(self, data, *, ctx, options=UpsertOptions()):+ self.upserts.append((dict(data), ctx.account_id, options))+ return data["id"]++ config = _DummyConfig(_DummyEmbedder())+ monkeypatch.setattr("openviking_cli.utils.config.get_openviking_config", lambda: config)+ monkeypatch.setattr(+ config.embedding,+ "get_embedder",+ lambda: (_ for _ in ()).throw(AssertionError("embedder must stay lazy")),+ )+ vikingdb = _MissingVikingDB()+ handler = TextEmbeddingHandler(vikingdb)+ msg = EmbeddingMsg.for_update_fields(+ record_id="l2-id",+ fields={"search_tags": ["scope=new"]},+ field_modes={"search_tags": "append"},+ initial_fields={+ "uri": "viking://resources/repo/a.py",+ "account_id": "acct",+ "level": 2,+ "vector": [0.1, 0.2],+ },+ context_data={"account_id": "acct", "uri": "viking://resources/repo/a.py"},+ )++ result = await handler.on_dequeue(_build_operation_payload(msg))++ assert result.outcome is ProcessOutcome.SUCCESS+ created, account_id, options = vikingdb.upserts[0]+ assert account_id == "acct"+ assert created["search_tags"] == ["scope=new"]+ assert options.partial_update is False+++async def test_embedding_handler_update_fields_not_found_rejects_incomplete_initial_fields(+ monkeypatch,+):+ class _MissingVikingDB:+ is_closing = False++ async def get_strict(self, ids, *, ctx):+ return []++ async def upsert(self, data, *, ctx, options=UpsertOptions()):+ raise AssertionError("incomplete record must not be written")++ config = _DummyConfig(_DummyEmbedder())+ monkeypatch.setattr("openviking_cli.utils.config.get_openviking_config", lambda: config)+ monkeypatch.setattr(+ config.embedding,+ "get_embedder",+ lambda: (_ for _ in ()).throw(AssertionError("embedder must stay lazy")),+ )+ handler = TextEmbeddingHandler(_MissingVikingDB())+ msg = EmbeddingMsg.for_update_fields(+ record_id="l2-id",+ fields={"search_tags": ["scope=new"]},+ field_modes={"search_tags": "append"},+ initial_fields={+ "uri": "viking://resources/repo/a.py",+ "account_id": "acct",+ "level": 2,+ },+ context_data={"account_id": "acct", "uri": "viking://resources/repo/a.py"},+ )++ result = await handler.on_dequeue(_build_operation_payload(msg))++ assert result.outcome is ProcessOutcome.FAILED+ assert "missing_fields=['vector']" in result.error+++async def test_embedding_handler_merge_action_reads_and_merges_before_full_upsert(monkeypatch):+ embedder = _DummyEmbedder()+ config = _DummyConfig(embedder)+ monkeypatch.setattr(+ "openviking_cli.utils.config.get_openviking_config",+ lambda: config,+ )+ captured = {}++ class _MergingVikingDB:+ is_closing = False+ uses_content_field = False++ async def get_strict(self, ids, *, ctx):+ captured["read"] = (list(ids), ctx.account_id)+ return [{"id": ids[0], "search_tags": ["env=old"], "created_at": "old"}]++ async def upsert(self, data, *, ctx, options=UpsertOptions()):+ captured["data"] = dict(data)+ captured["options"] = options+ return data["id"]++ handler = TextEmbeddingHandler(_MergingVikingDB())+ msg = EmbeddingMsg(+ message="body",+ action=IndexAction.MERGE,+ context_data={+ "id": "generated-id",+ "_upsert_record_id": "generated-id",+ "uri": "viking://resources/repo/a.py",+ "account_id": "acct",+ "abstract": "summary",+ },+ update_fields={"search_tags": ["scope=new"]},+ field_modes={"search_tags": "append"},+ )++ result = await handler.on_dequeue(_build_operation_payload(msg))++ assert result.outcome is ProcessOutcome.SUCCESS+ assert captured["read"] == (["generated-id"], "acct")+ assert captured["data"]["search_tags"] == ["env=old", "scope=new"]+ assert captured["data"]["created_at"] == "old"+ assert captured["options"].partial_update is False+++async def test_embedding_handler_merge_not_found_creates_from_generated_vector(monkeypatch):+ config = _DummyConfig(_DummyEmbedder())+ monkeypatch.setattr("openviking_cli.utils.config.get_openviking_config", lambda: config)+ captured = {}++ class _MissingVikingDB:+ is_closing = False+ uses_content_field = False++ async def get_strict(self, ids, *, ctx):+ return []++ async def upsert(self, data, *, ctx, options=UpsertOptions()):+ captured["data"] = dict(data)+ return data["id"]++ handler = TextEmbeddingHandler(_MissingVikingDB())+ msg = EmbeddingMsg(+ message="body",+ action=IndexAction.MERGE,+ context_data={+ "_upsert_record_id": "generated-id",+ "uri": "viking://resources/repo/a.py",+ "account_id": "acct",+ "level": 2,+ "abstract": "summary",+ },+ update_fields={"search_tags": ["scope=new"]},+ field_modes={"search_tags": "append"},+ )++ result = await handler.on_dequeue(_build_operation_payload(msg))++ assert result.outcome is ProcessOutcome.SUCCESS+ assert captured["data"]["id"] == "generated-id"+ assert captured["data"]["vector"] == [0.1, 0.2]+ assert captured["data"]["search_tags"] == ["scope=new"]+++async def test_embedding_handler_skips_noop_action_without_vector_db_or_embedder(monkeypatch):+ from openviking.storage.index_action import IndexAction++ config = _DummyConfig(_DummyEmbedder())+ monkeypatch.setattr("openviking_cli.utils.config.get_openviking_config", lambda: config)++ class _NoopVikingDB:+ is_closing = False++ def __getattr__(self, name):+ raise AssertionError(f"noop must not call {name}")++ handler = TextEmbeddingHandler(_NoopVikingDB())+ msg = EmbeddingMsg(+ message=None,+ context_data={"account_id": "acct", "uri": "viking://resources/repo"},+ action=IndexAction.NONE,+ )++ result = await handler.on_dequeue(_build_operation_payload(msg))++ assert result.outcome is ProcessOutcome.SUCCESS++def test_embedding_handler_builds_circuit_breaker_from_config(monkeypatch):class _DummyVikingDB:is_closing = False@@ -410,7 +848,7 @@ async def test_embedding_handler_treats_shutdown_write_lock_as_success(monkeypatself.calls = 0async def upsert(self, _data, *, ctx, options=UpsertOptions()):- assert options.partial_update is True+ assert options.partial_update is Falseself.calls += 1self.is_closing = Trueraise RuntimeError("IO error: lock /tmp/LOCK: already held by process")@@ -579,7 +1017,7 @@ async def test_embedding_handler_truncates_queue_input_before_embed(monkeypatch)handler = TextEmbeddingHandler(_CapturingVikingDB())payload = _build_queue_payload()queue_data = json.loads(payload["data"])- queue_data["message"] = " ".join(f"token-{idx}" for idx in range(200))+ queue_data["payload"]["message"] = " ".join(f"token-{idx}" for idx in range(200))payload["data"] = json.dumps(queue_data)await handler.on_dequeue(payload)@@ -644,7 +1082,7 @@ async def test_embedding_handler_preserves_parent_uri_for_backend_upsert_logic(muses_content_field = Falseasync def upsert(self, data, *, ctx, options=UpsertOptions()):- assert options.partial_update is True+ assert options.partial_update is Falsecaptured["data"] = dict(data)return "rec-1"@@ -657,7 +1095,7 @@ async def test_embedding_handler_preserves_parent_uri_for_backend_upsert_logic(mhandler = TextEmbeddingHandler(_CapturingVikingDB())payload = _build_queue_payload()queue_data = json.loads(payload["data"])- queue_data["context_data"]["parent_uri"] = "viking://resources"+ queue_data["payload"]["context_data"]["parent_uri"] = "viking://resources"payload["data"] = json.dumps(queue_data)result = await handler.on_dequeue(payload)@@ -665,7 +1103,7 @@ async def test_embedding_handler_preserves_parent_uri_for_backend_upsert_logic(massert result.outcome is ProcessOutcome.SUCCESSassert result.error is Noneassert result.value == {- **queue_data["context_data"],+ **queue_data["payload"]["context_data"],"id": vector_record_id("default", "viking://resources/sample", 2),"vector": [0.1, 0.2],}@@ -673,6 +1111,87 @@ async def test_embedding_handler_preserves_parent_uri_for_backend_upsert_logic(massert captured["data"]["parent_uri"] == "viking://resources"+async def test_embedding_handler_honors_explicit_full_upsert(monkeypatch):+ captured = {}++ class _CapturingVikingDB:+ is_closing = False+ uses_content_field = False++ async def upsert(self, data, *, ctx, options=UpsertOptions()):+ captured["partial_update"] = options.partial_update+ return data["id"]++ embedder = _DummyEmbedder()+ monkeypatch.setattr(+ "openviking_cli.utils.config.get_openviking_config",+ lambda: _DummyConfig(embedder),+ )+ payload = _build_queue_payload()+ queue_data = json.loads(payload["data"])+ queue_data["payload"]["context_data"]["_upsert_options"] = {"partial_update": False}+ payload["data"] = json.dumps(queue_data)++ await TextEmbeddingHandler(_CapturingVikingDB()).on_dequeue(payload)++ assert captured["partial_update"] is False+++async def test_embedding_handler_honors_existing_record_id_override(monkeypatch):+ captured = {}++ class _CapturingVikingDB:+ is_closing = False+ uses_content_field = False++ async def upsert(self, data, *, ctx, options=UpsertOptions()):+ captured["data"] = dict(data)+ return data["id"]++ embedder = _DummyEmbedder()+ monkeypatch.setattr(+ "openviking_cli.utils.config.get_openviking_config",+ lambda: _DummyConfig(embedder),+ )+ payload = _build_queue_payload_for_account("acct-existing-id")+ queue_data = json.loads(payload["data"])+ queue_data["payload"]["context_data"]["_upsert_record_id"] = "id-from-vector-db"+ payload["data"] = json.dumps(queue_data)++ await TextEmbeddingHandler(_CapturingVikingDB()).on_dequeue(payload)++ assert captured["data"]["id"] == "id-from-vector-db"+ assert "_upsert_record_id" not in captured["data"]+++async def test_embedding_handler_generates_id_when_no_existing_override(monkeypatch):+ from openviking.storage.vector_ids import vector_record_id++ captured = {}++ class _CapturingVikingDB:+ is_closing = False+ uses_content_field = False++ async def upsert(self, data, *, ctx, options=UpsertOptions()):+ captured["data"] = dict(data)+ return data["id"]++ embedder = _DummyEmbedder()+ monkeypatch.setattr(+ "openviking_cli.utils.config.get_openviking_config",+ lambda: _DummyConfig(embedder),+ )+ payload = _build_queue_payload_for_account("acct-new-id")++ await TextEmbeddingHandler(_CapturingVikingDB()).on_dequeue(payload)++ assert captured["data"]["id"] == vector_record_id("acct-new-id", "viking://resources/sample", 2)++@pytest.mark.asyncioasync def test_embedding_handler_settles_request_wait_by_message_id(monkeypatch):class _CapturingVikingDB:@@ -681,7 +1200,7 @@ async def test_embedding_handler_settles_request_wait_by_message_id(monkeypatch)uses_content_field = Falseasync def upsert(self, _data, *, ctx, options=UpsertOptions()):- assert options.partial_update is True+ assert options.partial_update is Falsereturn "rec-1"embedder = _DummyEmbedder()@@ -726,6 +1245,11 @@ def test_context_collection_uses_acl_mode_and_excludes_parent_uri():assert "acl_enabled" not in field_namesassert "acl_enabled" not in schema["ScalarIndex"]assert "parent_uri" not in field_names+ # md5 fingerprint field backs incremental diff; it is a scalar value column,+ # not indexed (looked up alongside a URI/level scan, never filtered on).+ md5_field = next(field for field in schema["Fields"] if field["FieldName"] == "md5")+ assert md5_field == {"FieldName": "md5", "FieldType": "string", "DefaultValue": ""}+ assert "md5" not in schema["ScalarIndex"]assert "parent_uri" not in schema["ScalarIndex"]assert "acl_restricted" not in field_namesassert "acl_restricted" not in schema["ScalarIndex"]@@ -960,7 +1484,7 @@ def test_volcengine_aksk_collection_ignores_unknown_fields_on_fetch_and_search()calls = []class _Collection(VolcengineCollection):- def _data_post(self, path, data):+ def _data_post(self, path, data, **kwargs):calls.append((path, data))return {}@@ -1442,7 +1966,7 @@ async def test_single_account_backend_update_runs_adapter_in_threadpool(monkeypaassert result.updated_count == 1assert result.error_code is Noneassert result.error_message is None- assert [call[0] for call in calls] == ["_prepare_upsert_payload", "update_data"]+ assert [call[0] for call in calls] == ["_prepare_update_payload", "update_data"]assert calls[-1][1] == ([{@@ -1455,6 +1979,54 @@ async def test_single_account_backend_update_runs_adapter_in_threadpool(monkeypa)+async def test_single_account_backend_partial_update_does_not_fill_omitted_text_or_vector_fields():+ calls = []++ class _Collection:+ def get_meta_data(self):+ return {+ "Fields": [+ {"FieldName": "id", "FieldType": "string"},+ {"FieldName": "abstract", "FieldType": "string"},+ {"FieldName": "content", "FieldType": "text"},+ {"FieldName": "vector", "FieldType": "vector"},+ {"FieldName": "search_tags", "FieldType": "list<string>"},+ {"FieldName": "account_id", "FieldType": "string"},+ ]+ }++ class _Adapter:+ mode = "local"+ USE_CONTENT_FIELD = True++ def get_collection(self):+ return _Collection()++ def update_data(self, data):+ calls.append(data)+ return [data[0]["id"]]++ backend = _SingleAccountBackend(+ config=VectorDBBackendConfig(backend="local", name="context", dimension=2),+ bound_account_id="acc1",+ shared_adapter=_Adapter(),+ )++ result = await backend.update({"id": "rec-1", "search_tags": ["team=search"]})++ assert result.ok is True+ assert calls == [+ [+ {+ "id": "rec-1",+ "search_tags": ["team=search"],+ "account_id": "acc1",+ }+ ]+ ]++@pytest.mark.asyncioasync def test_local_backend_update_preserves_omitted_fields_end_to_end(tmp_path):if not getattr(vectordb_engine, "PersistStore", None):diff --git a/tests/storage/test_content_write_processing_mode.py b/tests/storage/test_content_write_processing_mode.pyindex e36f03c47..21cae071a 100644--- a/tests/storage/test_content_write_processing_mode.py+++ b/tests/storage/test_content_write_processing_mode.py@@ -15,6 +15,7 @@ from openviking.storage.abstract_overview import ()from openviking.storage.content_write import ContentWriteCoordinatorfrom openviking.storage.queuefs.semantic_ops.freshness_policy import FreshnessAction+from openviking.utils.content_hash import content_md5from openviking.utils.ingest_options import IngestOptionsfrom openviking_cli.session.user_id import UserIdentifier@@ -116,6 +117,7 @@ async def test_direct_write_skips_semantic_refresh_for_vectors_only_and_sidecar_"name": "demo.md","summary": "",}+ assert vectorize_file.await_args.kwargs["file_md5"] == content_md5(b"updated")assert "register_request_wait" not in vectorize_file.await_args.kwargsassert result["semantic_status"] == "skipped"assert result["vector_status"] == "queued"@@ -160,6 +162,134 @@ async def test_direct_write_skips_semantic_refresh_for_vectors_only_and_sidecar_assert sidecar_result["vector_status"] == "queued"+async def test_direct_write_passes_final_md5_and_old_abstract_to_semantic_refresh(ctx):+ file_uri = "viking://resources/demo.py"++ class _VikingDB:+ async def get_l2_diff_records_by_uris(self, uris, *, ctx):+ assert uris == [file_uri]+ return {file_uri: {"md5": "old-md5", "abstract": "old abstract"}}++ fake_fs = _FakeVikingFS()+ coordinator = ContentWriteCoordinator(viking_fs=fake_fs, vikingdb=_VikingDB())+ enqueue = AsyncMock(return_value=FreshnessAction.REFRESH_NOW)+ coordinator._enqueue_semantic_refresh = enqueue++ await coordinator._write_direct_with_refresh(+ uri=file_uri,+ root_uri="viking://resources",+ content="updated",+ mode="replace",+ context_type="resource",+ wait=False,+ timeout=None,+ ctx=ctx,+ written_bytes=7,+ telemetry_id="",+ )++ assert enqueue.await_args.kwargs["file_md5"] == content_md5(b"updated")+ assert enqueue.await_args.kwargs["file_abstract"] == "old abstract"+++async def test_semantic_message_carries_file_md5_and_old_abstract(monkeypatch, ctx):+ file_uri = "viking://resources/demo.py"+ queue = SimpleNamespace(enqueue=AsyncMock(return_value="enqueued"))+ manager = SimpleNamespace(SEMANTIC="Semantic", get_queue=lambda *args, **kwargs: queue)+ monkeypatch.setattr(content_write_module, "get_queue_manager", lambda: manager)+ monkeypatch.setattr(+ content_write_module,+ "plan_abstract_overview_refresh",+ AsyncMock(return_value=SimpleNamespace(action=FreshnessAction.REFRESH_NOW)),+ )+ coordinator = ContentWriteCoordinator(viking_fs=_FakeVikingFS())++ await coordinator._enqueue_semantic_refresh_changes(+ root_uri="viking://resources",+ context_type="resource",+ changes={"modified": [file_uri]},+ ctx=ctx,+ file_md5s={file_uri: "new-md5"},+ file_abstracts={file_uri: "old abstract"},+ )++ msg = queue.enqueue.await_args.args[0]+ assert msg.file_md5s == {file_uri: "new-md5"}+ assert msg.file_abstracts == {file_uri: "old abstract"}+++async def test_direct_append_passes_md5_of_final_content_without_vector_db(ctx):+ fake_fs = _FakeVikingFS()+ fake_fs.read_file.return_value = "previous"+ coordinator = ContentWriteCoordinator(viking_fs=fake_fs)+ enqueue = AsyncMock(return_value=FreshnessAction.REFRESH_NOW)+ coordinator._enqueue_semantic_refresh = enqueue++ await coordinator._write_direct_with_refresh(+ uri="viking://resources/demo.py",+ root_uri="viking://resources",+ content=" updated",+ mode="append",+ context_type="resource",+ wait=False,+ timeout=None,+ ctx=ctx,+ written_bytes=8,+ telemetry_id="",+ )++ assert enqueue.await_args.kwargs["file_md5"] == content_md5(b"previous updated")+ assert enqueue.await_args.kwargs["file_abstract"] == ""+++async def test_direct_write_continues_when_old_abstract_lookup_fails(ctx):+ class _FailingVikingDB:+ async def get_l2_diff_records_by_uris(self, uris, *, ctx):+ raise RuntimeError("vector lookup unavailable")++ fake_fs = _FakeVikingFS()+ coordinator = ContentWriteCoordinator(viking_fs=fake_fs, vikingdb=_FailingVikingDB())+ enqueue = AsyncMock(return_value=FreshnessAction.REFRESH_NOW)+ coordinator._enqueue_semantic_refresh = enqueue++ await coordinator._write_direct_with_refresh(+ uri="viking://resources/demo.py",+ root_uri="viking://resources",+ content="updated",+ mode="replace",+ context_type="resource",+ wait=False,+ timeout=None,+ ctx=ctx,+ written_bytes=7,+ telemetry_id="",+ )++ assert enqueue.await_args.kwargs["file_abstract"] == ""+ assert enqueue.await_args.kwargs["file_md5"] == content_md5(b"updated")+++async def test_old_abstract_lookup_uses_viking_fs_vector_store_when_not_injected(ctx):+ file_uri = "viking://agent/skills/demo/SKILL.md"++ class _VectorStore:+ async def get_l2_diff_records_by_uris(self, uris, *, ctx):+ return {file_uri: {"abstract": "skill summary"}}++ fake_fs = _FakeVikingFS()+ fake_fs._get_vector_store = lambda: _VectorStore()+ coordinator = ContentWriteCoordinator(viking_fs=fake_fs)++ assert await coordinator._load_file_abstracts([file_uri], ctx=ctx) == {+ file_uri: "skill summary"+ }++@pytest.mark.asyncioasync def test_write_builds_ingest_options_before_scheduling_resource_refresh(ctx):coordinator = ContentWriteCoordinator(viking_fs=_FakeVikingFS())diff --git a/tests/storage/test_context_update_plan.py b/tests/storage/test_context_update_plan.pynew file mode 100644index 000000000..e3538aed1--- /dev/null+++ b/tests/storage/test_context_update_plan.py@@ -0,0 +1,3143 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++import asyncio+from contextlib import nullcontext+from types import SimpleNamespace+from unittest.mock import AsyncMock, MagicMock, Mock++import pytest++from openviking.storage import resource_diff+from openviking.storage.index_action import FieldPatch+++def test_context_plan_has_explicit_actions_and_compact_semantic_roundtrip():+ from openviking.storage.context_update_plan import (+ ContextUpdatePlan,+ IndexSlot,+ SemanticAction,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+ )++ entry = SemanticTreeEntry(+ relative_path="a.py",+ kind="file",+ content_state="modified",+ semantic_action=SemanticAction.GENERATE,+ md5="new",+ index_slots=(+ IndexSlot(+ level=2,+ record_id="external-id",+ existing_fields={"abstract": "old"},+ action="upsert",+ upsert_fields={"search_tags": ["new"]},+ ),+ ),+ )+ plan = ContextUpdatePlan(+ root_uri="viking://resources/repo",+ context_type="resource",+ semantic_plan=SemanticPlan(+ root_uri="viking://resources/repo",+ context_type="resource",+ tree=SemanticTreeSnapshot(+ (+ SemanticTreeEntry("", "directory", "unchanged", "aggregate"),+ entry,+ )+ ),+ ),+ )+ payload = plan.to_dict()+ restored = ContextUpdatePlan.from_dict(payload)+ assert restored == plan+ node = next(+ item+ for item in payload["semantic_plan"]["tree"]["entries"]+ if item["relative_path"] == "a.py"+ )+ assert "indexed_records" not in node+ assert "previous_abstracts" not in node+ assert "uri" not in node["index_slots"][0]+ assert node["index_slots"][0]["record_id"] == "external-id"+++def test_index_actions_reject_unconsumed_field_patch_state():+ from openviking.storage.context_update_plan import DirectIndexAction, IndexSlot++ patch = FieldPatch({"search_tags": ["scope=new"]})+ with pytest.raises(ValueError, match="upsert uses resolved upsert_fields"):+ DirectIndexAction(+ "upsert",+ "viking://resources/repo/a.py",+ 2,+ "a-l2",+ field_patch=patch,+ )+ with pytest.raises(ValueError, match="fallback_to_patch requires"):+ IndexSlot(2, "a-l2", action="upsert", fallback_to_patch=True)+++def test_context_plan_reads_legacy_flat_patch_fields():+ from openviking.storage.context_update_plan import ContextUpdatePlan++ restored = ContextUpdatePlan.from_dict(+ {+ "root_uri": "viking://resources/repo",+ "context_type": "resource",+ "direct_index_actions": [+ {+ "action": "update_fields",+ "uri": "viking://resources/repo/a.py",+ "level": 2,+ "record_id": "a-l2",+ "fields": {"search_tags": ["scope=new"]},+ "field_modes": {"search_tags": "append"},+ "initial_fields": {"vector": [0.1, 0.2]},+ "search_tag_mode": "append",+ }+ ],+ }+ )++ action = restored.direct_index_actions[0]+ assert action.upsert_fields == {}+ assert action.field_patch == FieldPatch(+ {"search_tags": ["scope=new"]},+ {"search_tags": "append"},+ {"vector": [0.1, 0.2]},+ )+++def test_legacy_semantic_slot_drops_inert_fallback_without_patch():+ from openviking.storage.context_update_plan import IndexSlot++ slot = IndexSlot.from_dict(+ {+ "level": 2,+ "record_id": "a-l2",+ "action": "upsert",+ "fields": {},+ "update_fields": {},+ "field_modes": {},+ "fallback_update_fields": True,+ }+ )++ assert slot.field_patch is None+ assert slot.fallback_to_patch is False+++def test_after_content_commit_keeps_only_derived_actions():+ from openviking.storage.context_update_plan import (+ ContentTreeAction,+ ContextUpdatePlan,+ DirectIndexAction,+ )++ plan = ContextUpdatePlan(+ "viking://resources/repo",+ "resource",+ content_tree_actions=(+ ContentTreeAction(+ "upsert",+ "a.py",+ new_kind="file",+ artifact_path="repository/a.py",+ md5="new",+ ),+ ),+ direct_index_actions=(+ DirectIndexAction("delete", "viking://resources/repo/old.py", 2, "old-l2"),+ ),+ )++ committed = plan.after_content_commit()++ assert committed.content_tree_actions == ()+ assert committed.semantic_plan is plan.semantic_plan+ assert committed.direct_index_actions is plan.direct_index_actions+++def test_context_plan_rejects_conflicting_record_operations():+ from openviking.storage.context_update_plan import (+ ContextUpdatePlan,+ DirectIndexAction,+ IndexSlot,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+ )++ semantic = SemanticPlan(+ root_uri="viking://resources/repo",+ context_type="resource",+ tree=SemanticTreeSnapshot(+ (+ SemanticTreeEntry("", "directory", "unchanged", "aggregate"),+ SemanticTreeEntry(+ "a.py",+ "file",+ "modified",+ "generate",+ md5="new",+ index_slots=(+ IndexSlot(+ 2,+ "same-id",+ {"abstract": "old"},+ action="upsert",+ ),+ ),+ ),+ )+ ),+ )+ with pytest.raises(ValueError, match="conflict"):+ ContextUpdatePlan(+ root_uri=semantic.root_uri,+ context_type="resource",+ semantic_plan=semantic,+ direct_index_actions=(+ DirectIndexAction("delete", semantic.root_uri + "/a.py", 2, "same-id"),+ ),+ )+++def test_semantic_plan_rejects_disconnected_or_mistyped_actions():+ from openviking.storage.context_update_plan import (+ IndexSlot,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+ )++ with pytest.raises(ValueError, match="root"):+ SemanticPlan(+ "viking://resources/repo",+ "resource",+ SemanticTreeSnapshot(+ (+ SemanticTreeEntry(+ "src/a.py",+ "file",+ "modified",+ "generate",+ index_slots=(IndexSlot(2, "a-l2", action="upsert"),),+ ),+ )+ ),+ )+ with pytest.raises(ValueError, match="directory.*aggregate"):+ SemanticTreeEntry("src", "directory", "modified", "generate")+ with pytest.raises(ValueError, match="file.*generate"):+ SemanticTreeEntry("a.py", "file", "modified", "aggregate", md5="m")+ with pytest.raises(ValueError, match="ancestor.*aggregate"):+ SemanticPlan(+ "viking://resources/repo",+ "resource",+ SemanticTreeSnapshot(+ (+ SemanticTreeEntry(+ "",+ "directory",+ "unchanged",+ "reuse",+ index_slots=(IndexSlot(0, "root-l0", {"abstract": "root"}),),+ ),+ SemanticTreeEntry(+ "a.py",+ "file",+ "modified",+ "generate",+ md5="new",+ index_slots=(IndexSlot(2, "a-l2", action="upsert"),),+ ),+ )+ ),+ )+++def test_semantic_plan_validates_ancestors_without_rescanning_entries():+ from openviking.storage.context_update_plan import (+ IndexSlot,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+ )++ class CountingEntries:+ def __init__(self, entries):+ self._entries = tuple(entries)+ self.iterations = 0++ def __iter__(self):+ self.iterations += 1+ return iter(self._entries)++ directories = [+ "/".join(["src", *[f"level-{index}" for index in range(depth)]]) for depth in range(8)+ ]+ entries = CountingEntries(+ [SemanticTreeEntry("", "directory", "unchanged", "aggregate")]+ + [SemanticTreeEntry(path, "directory", "unchanged", "aggregate") for path in directories]+ + [+ SemanticTreeEntry(+ directories[-1] + "/a.py",+ "file",+ "modified",+ "generate",+ md5="new",+ index_slots=(IndexSlot(2, "a-l2", action="upsert"),),+ )+ ]+ )++ SemanticPlan(+ "viking://resources/repo",+ "resource",+ SemanticTreeSnapshot(entries),+ )++ assert entries.iterations <= 4+++def test_semantic_plan_rejects_missing_higher_ancestor_independent_of_entry_order():+ from openviking.storage.context_update_plan import (+ IndexSlot,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+ )++ with pytest.raises(ValueError, match="lacks ancestor 'a'.*a/b/c.py"):+ SemanticPlan(+ "viking://resources/repo",+ "resource",+ SemanticTreeSnapshot(+ (+ SemanticTreeEntry("", "directory", "unchanged", "aggregate"),+ SemanticTreeEntry(+ "a/b/c.py",+ "file",+ "modified",+ "generate",+ md5="new",+ index_slots=(IndexSlot(2, "c-l2", action="upsert"),),+ ),+ SemanticTreeEntry("a/b", "directory", "unchanged", "aggregate"),+ )+ ),+ )+++def test_index_slot_rejects_non_semantic_operations():+ from openviking.storage.context_update_plan import IndexSlot++ with pytest.raises(ValueError, match="only supports"):+ IndexSlot(2, "a-l2", action="update_fields")+++def test_reuse_entry_rejects_index_mutation():+ from openviking.storage.context_update_plan import IndexSlot, SemanticTreeEntry++ with pytest.raises(ValueError, match="reuse.*index"):+ SemanticTreeEntry(+ "a.py",+ "file",+ "unchanged",+ "reuse",+ index_slots=(+ IndexSlot(+ 2,+ "a-l2",+ {"abstract": "old"},+ action="upsert",+ ),+ ),+ )+++async def test_resolver_compares_missing_fingerprints_with_bounded_reads():+ from openviking.storage.resource_rnfv import (+ FormalEntry,+ FormalTreeSnapshot,+ NewArtifactSnapshot,+ NewEntry,+ RequestIntent,+ RNFVSnapshot,+ VectorIndexSnapshot,+ )++ paths = [f"{i}.py" for i in range(8)]+ snapshot = RNFVSnapshot(+ RequestIntent("viking://resources/repo", "semantic_and_vectors"),+ NewArtifactSnapshot({p: NewEntry() for p in paths}),+ FormalTreeSnapshot({p: FormalEntry() for p in paths}),+ VectorIndexSnapshot({}, frozenset({"id", "uri", "level", "md5"})),+ )+ active = peak = 0+ gate = asyncio.Event()++ async def read(*args):+ nonlocal active, peak+ active += 1+ peak = max(peak, active)+ if active == 4:+ gate.set()+ try:+ await asyncio.wait_for(gate.wait(), 1)+ return b"same"+ finally:+ active -= 1++ resolve = getattr(resource_diff, "resolve_resource_diff", None)+ assert resolve is not None+ result = await resolve(+ snapshot,+ store=SimpleNamespace(read_bytes=read),+ artifact_ref=None,+ target=SimpleNamespace(read_file=read),+ concurrency=2,+ )+ assert peak == 4+ assert active == 0+ assert all(+ e.content_state == "unchanged" and e.index_state == "missing"+ for e in result.entries.values()+ )+ assert all(e.md5 for e in result.entries.values())+ assert not hasattr(result, "needs_body_compare")+++async def test_resolver_rejects_incomplete_snapshot_before_io():+ from openviking.storage.resource_rnfv import (+ FormalTreeSnapshot,+ NewArtifactSnapshot,+ RequestIntent,+ RNFVSnapshot,+ VectorIndexSnapshot,+ )++ snapshot = RNFVSnapshot(+ RequestIntent("viking://resources/repo", "semantic_and_vectors"),+ NewArtifactSnapshot({}, complete=False),+ FormalTreeSnapshot({}),+ VectorIndexSnapshot({}, frozenset({"id", "uri", "level", "md5"})),+ )+ store, target = AsyncMock(), AsyncMock()+ resolve = getattr(resource_diff, "resolve_resource_diff", None)+ assert resolve is not None+ with pytest.raises(ValueError, match="incomplete"):+ await resolve(snapshot, store=store, artifact_ref=None, target=target)+ store.read_bytes.assert_not_called()+ target.read_file.assert_not_called()+++async def test_resolver_hashes_new_file_when_manifest_md5_is_missing():+ from openviking.storage.context_update_plan import ContentState+ from openviking.storage.resource_rnfv import (+ FormalTreeSnapshot,+ NewArtifactSnapshot,+ NewEntry,+ RequestIntent,+ RNFVSnapshot,+ VectorIndexSnapshot,+ )+ from openviking.utils.content_hash import content_md5++ snapshot = RNFVSnapshot(+ RequestIntent("viking://resources/repo", "semantic_and_vectors"),+ NewArtifactSnapshot({"a.py": NewEntry()}),+ FormalTreeSnapshot({}),+ VectorIndexSnapshot({}, frozenset({"id", "uri", "level", "md5"})),+ )+ store = AsyncMock()+ store.read_bytes.return_value = b"new body"+ result = await resource_diff.resolve_resource_diff(+ snapshot,+ store=store,+ artifact_ref=object(),+ target=AsyncMock(),+ artifact_paths={"a.py": "repository/a.py"},+ )++ assert result.entries["a.py"].content_state is ContentState.ADDED+ assert result.entries["a.py"].md5 == content_md5(b"new body")+ store.read_bytes.assert_awaited_once()+++async def test_resolver_marks_removed_content_vectors_as_obsolete():+ from openviking.storage.context_update_plan import ContentState, IndexState+ from openviking.storage.resource_rnfv import (+ FormalEntry,+ FormalTreeSnapshot,+ NewArtifactSnapshot,+ RequestIntent,+ RNFVSnapshot,+ VectorIndexSnapshot,+ VectorRecordSnapshot,+ )++ root = "viking://resources/repo"+ records = {+ "removed-l2": VectorRecordSnapshot("removed-l2", f"{root}/removed.md", "removed.md", 2),+ "orphan-l2": VectorRecordSnapshot("orphan-l2", f"{root}/orphan.md", "orphan.md", 2),+ }+ snapshot = RNFVSnapshot(+ RequestIntent(root, "semantic_and_vectors"),+ NewArtifactSnapshot({}),+ FormalTreeSnapshot({"removed.md": FormalEntry()}),+ VectorIndexSnapshot(records, frozenset({"id", "uri", "level", "md5"})),+ )++ result = await resource_diff.resolve_resource_diff(+ snapshot, store=AsyncMock(), artifact_ref=object(), target=AsyncMock()+ )++ assert result.entries["removed.md"].content_state is ContentState.DELETED+ assert result.entries["removed.md"].index_state is IndexState.OBSOLETE+ assert result.entries["orphan.md"].content_state is ContentState.ABSENT+ assert result.entries["orphan.md"].index_state is IndexState.ORPHAN+++async def test_resolver_uses_same_duplicate_record_winner_as_planner():+ from openviking.storage.context_update_plan import ContentState+ from openviking.storage.resource_rnfv import (+ FormalEntry,+ FormalTreeSnapshot,+ NewArtifactSnapshot,+ NewEntry,+ RequestIntent,+ RNFVSnapshot,+ VectorIndexSnapshot,+ VectorRecordSnapshot,+ )++ root = "viking://resources/repo"+ records = {+ "z-new": VectorRecordSnapshot("z-new", f"{root}/a.py", "a.py", 2, {"md5": "new"}),+ "a-old": VectorRecordSnapshot("a-old", f"{root}/a.py", "a.py", 2, {"md5": "old"}),+ }+ snapshot = RNFVSnapshot(+ RequestIntent(root, "semantic_and_vectors"),+ NewArtifactSnapshot({"a.py": NewEntry(md5="new")}),+ FormalTreeSnapshot({"a.py": FormalEntry()}),+ VectorIndexSnapshot(records, frozenset({"id", "uri", "level", "md5"})),+ )++ result = await resource_diff.resolve_resource_diff(+ snapshot, store=AsyncMock(), artifact_ref=object(), target=AsyncMock()+ )++ assert result.entries["a.py"].content_state is ContentState.MODIFIED+++async def test_resolver_index_state_uses_canonical_duplicate_record():+ from openviking.storage.context_update_plan import ContentState, IndexState+ from openviking.storage.resource_rnfv import (+ FormalEntry,+ FormalTreeSnapshot,+ NewArtifactSnapshot,+ NewEntry,+ RequestIntent,+ RNFVSnapshot,+ VectorIndexSnapshot,+ VectorRecordSnapshot,+ )++ root = "viking://resources/repo"+ records = {+ "z-new": VectorRecordSnapshot("z-new", f"{root}/a.py", "a.py", 2, {"md5": "new"}),+ "a-old": VectorRecordSnapshot("a-old", f"{root}/a.py", "a.py", 2, {"md5": "old"}),+ }+ snapshot = RNFVSnapshot(+ RequestIntent(root, "semantic_and_vectors"),+ NewArtifactSnapshot({"a.py": NewEntry(md5="old")}),+ FormalTreeSnapshot({"a.py": FormalEntry()}),+ VectorIndexSnapshot(records, frozenset({"id", "uri", "level", "md5"})),+ )++ result = await resource_diff.resolve_resource_diff(+ snapshot, store=AsyncMock(), artifact_ref=object(), target=AsyncMock()+ )++ entry = result.entries["a.py"]+ assert entry.content_state is ContentState.UNCHANGED+ assert entry.index_state is IndexState.COMPLETE+++async def test_resolver_log_separates_files_directories_and_logical_root(monkeypatch):+ from openviking.storage import resource_diff+ from openviking.storage.resource_rnfv import (+ FormalEntry,+ FormalTreeSnapshot,+ NewArtifactSnapshot,+ NewEntry,+ RequestIntent,+ RNFVSnapshot,+ VectorIndexSnapshot,+ VectorRecordSnapshot,+ )++ root = "viking://resources/repo"+ records = {+ "root-l0": VectorRecordSnapshot("root-l0", root, "", 0),+ "root-l1": VectorRecordSnapshot("root-l1", root, "", 1),+ "file-l2": VectorRecordSnapshot("file-l2", f"{root}/a.py", "a.py", 2, {"md5": "same"}),+ "sub-l0": VectorRecordSnapshot("sub-l0", f"{root}/sub", "sub", 0),+ "sub-l1": VectorRecordSnapshot("sub-l1", f"{root}/sub", "sub", 1),+ }+ snapshot = RNFVSnapshot(+ RequestIntent(root, "semantic_and_vectors"),+ NewArtifactSnapshot(+ {"": NewEntry(is_dir=True), "a.py": NewEntry(md5="same"), "sub": NewEntry(is_dir=True)}+ ),+ FormalTreeSnapshot(+ {"": FormalEntry(is_dir=True), "a.py": FormalEntry(), "sub": FormalEntry(is_dir=True)}+ ),+ VectorIndexSnapshot(records, frozenset({"id", "uri", "level", "md5"})),+ )++ log_info = Mock()+ monkeypatch.setattr(resource_diff.logger, "info", log_info)+ await resource_diff.resolve_resource_diff(+ snapshot,+ store=AsyncMock(),+ artifact_ref=object(),+ target=AsyncMock(),+ )++ message = log_info.call_args.args[0] % log_info.call_args.args[1:]+ fixed_fields = "n_files=1 n_dirs=1 f_files=1 f_dirs=1 logical_root=true v_records=5 root_state=unchanged root_index=complete md5_fast_path=1 body_compared=0 new_files_hashed=0"+ assert fixed_fields in message+ assert "states={file:{'unchanged': 1},dir:{'unchanged': 1}}" in message+ assert "index={file:{'complete': 1},dir:{'complete': 1}}" in message+ assert message.index("md5_fast_path=1") < message.index("states=")+ assert "n_entries=" not in message+++def test_tree_entry_log_counts_empty_path_as_a_real_flat_file():+ from openviking.storage.resource_diff import count_tree_entry_kinds+ from openviking.storage.resource_rnfv import FormalEntry, NewEntry++ assert count_tree_entry_kinds({"": NewEntry(md5="digest")}) == (1, 0, False)+ assert count_tree_entry_kinds({"": FormalEntry()}) == (1, 0, False)+++def test_planning_details_log_at_debug(monkeypatch):+ from openviking.storage.context_update_plan import ContextUpdatePlan+ from openviking.storage.resource_diff import ResourceDiffResult+ from openviking.utils import resource_processor++ log_info = Mock()+ log_debug = Mock()+ monkeypatch.setattr(resource_processor.logger, "info", log_info)+ monkeypatch.setattr(resource_processor.logger, "debug", log_debug)+ plan = ContextUpdatePlan("viking://resources/repo", "resource")++ resource_processor.ResourceProcessor._log_context_update_plan(plan)+ resource_processor.ResourceProcessor._log_context_commit_summary(+ ResourceDiffResult({}),+ content_actions=(),+ root_uri=plan.root_uri,+ is_initial=False,+ )+ resource_processor.ResourceProcessor._log_context_commit_summary(+ ResourceDiffResult({}),+ content_actions=(),+ root_uri=plan.root_uri,+ is_initial=True,+ )++ assert log_info.call_count == 0+ assert log_debug.call_count == 3+++def test_content_commit_log_uses_info_only_for_mutations(monkeypatch):+ from openviking.storage.context_update_plan import (+ ContentTreeAction,+ ContentTreeOperation,+ )+ from openviking.utils import resource_processor++ log_info = Mock()+ log_debug = Mock()+ monkeypatch.setattr(resource_processor.logger, "info", log_info)+ monkeypatch.setattr(resource_processor.logger, "debug", log_debug)++ resource_processor.ResourceProcessor._log_content_tree_commit(+ root_uri="viking://resources/repo",+ artifact_backend="local",+ actions=(),+ duration_ms=1.0,+ )+ resource_processor.ResourceProcessor._log_content_tree_commit(+ root_uri="viking://resources/repo",+ artifact_backend="local",+ actions=(+ ContentTreeAction(+ ContentTreeOperation.UPSERT,+ "a.py",+ new_kind="file",+ artifact_path="a.py",+ md5="digest",+ ),+ ),+ duration_ms=2.0,+ )++ log_debug.assert_called_once()+ log_info.assert_called_once()+++def test_add_resource_log_level_contract():+ import ast+ from pathlib import Path++ root = Path(__file__).resolve().parents[2]+ expected = {+ "openviking/storage/queuefs/add_resource_processor.py": {+ "[AddResourceStarted]": "info",+ "[AddResourceCompleted]": "info",+ },+ "openviking/storage/resource_diff.py": {+ "[ResourceDiffResult]": "info",+ },+ "openviking/utils/resource_processor.py": {+ "[RNFVSnapshot]": "debug",+ "[ContextUpdatePlan]": "debug",+ "[add_resource]": "debug",+ "[DirectIndexActions]": "debug",+ },+ "openviking/storage/queuefs/semantic_processor.py": {+ "[SemanticPlanExecution]": "debug",+ },+ "openviking/parse/parsers/code/code.py": {+ "Uploading code repository artifacts": "debug",+ },+ "openviking/parse/parsers/media/image.py": {+ "Processing large image": "debug",+ },+ }++ for relative_path, markers in expected.items():+ tree = ast.parse(root.joinpath(relative_path).read_text())+ found = {}+ for node in ast.walk(tree):+ if not isinstance(node, ast.Call) or not isinstance(node.func, ast.Attribute):+ continue+ if not node.args:+ continue+ message = "".join(+ part.value+ for part in ast.walk(node.args[0])+ if isinstance(part, ast.Constant) and isinstance(part.value, str)+ )+ for marker in markers:+ if marker in message:+ found[marker] = node.func.attr+ assert found == markers+++def test_builder_maps_content_semantic_and_direct_index_actions():+ from openviking.storage.context_update_plan import (+ ContentState,+ ContextUpdatePlan,+ IndexAction,+ IndexState,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot++ diff = ResourceDiffResult(+ entries={+ "changed.py": ResourceDiffEntry(+ "changed.py",+ ContentState.MODIFIED,+ IndexState.STALE,+ old_kind="file",+ new_kind="file",+ md5="new-md5",+ ),+ "gone.py": ResourceDiffEntry(+ "gone.py", ContentState.DELETED, IndexState.OBSOLETE, old_kind="file"+ ),+ "ghost.py": ResourceDiffEntry("ghost.py", ContentState.ABSENT, IndexState.ORPHAN),+ }+ )+ inventory = {+ "changed-id": VectorRecordSnapshot(+ "changed-id",+ "viking://resources/repo/changed.py",+ "changed.py",+ 2,+ {"md5": "old-md5", "abstract": "old abstract", "search_tags": ["old"]},+ ),+ "gone-id": VectorRecordSnapshot("gone-id", "viking://resources/repo/gone.py", "gone.py", 2),+ "ghost-id": VectorRecordSnapshot(+ "ghost-id", "viking://resources/repo/ghost.py", "ghost.py", 2+ ),+ }++ from openviking.storage.context_update_plan import build_context_update_plan++ plan = build_context_update_plan(+ root_uri="viking://resources/repo",+ context_type="resource",+ request=RequestIntent("viking://resources/repo", "semantic_and_vectors"),+ diff=diff,+ new_kinds={"": "directory", "changed.py": "file"},+ artifact_paths={"changed.py": "repository/changed.py"},+ records=inventory,+ is_code_repo=True,+ account_id="acc",+ )++ assert isinstance(plan, ContextUpdatePlan)+ assert [(a.operation.value, a.relative_path) for a in plan.content_tree_actions] == [+ ("upsert", "changed.py"),+ ("delete", "gone.py"),+ ]+ assert {(a.action.value, a.record_id) for a in plan.direct_index_actions} == {+ ("delete", "gone-id"),+ ("delete", "ghost-id"),+ }+ changed = next(e for e in plan.semantic_plan.tree.entries if e.relative_path == "changed.py")+ assert changed.semantic_action.value == "generate"+ assert changed.index_slots[0].action == IndexAction.UPSERT+ assert changed.index_slots[0].record_id == "changed-id"+++def test_level_conflict_deletes_invalid_record_and_rebuilds_valid_slots():+ from openviking.storage.context_update_plan import (+ ContentState,+ IndexState,+ build_context_update_plan,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot++ root = "viking://resources/repo"+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(root, "semantic_and_vectors"),+ diff=ResourceDiffResult(+ {+ "module": ResourceDiffEntry(+ "module",+ ContentState.UNCHANGED,+ IndexState.LEVEL_CONFLICT,+ old_kind="directory",+ new_kind="directory",+ )+ }+ ),+ new_kinds={"": "directory", "module": "directory"},+ artifact_paths={},+ records={"stale-l2": VectorRecordSnapshot("stale-l2", f"{root}/module", "module", 2)},+ is_code_repo=False,+ account_id="acc",+ )++ assert [(a.action.value, a.record_id) for a in plan.direct_index_actions] == [+ ("delete", "stale-l2")+ ]+ entry = next(+ entry for entry in plan.semantic_plan.tree.entries if entry.relative_path == "module"+ )+ assert entry.semantic_action.value == "aggregate"+ assert [(slot.level, slot.action.value) for slot in entry.index_slots] == [+ (0, "upsert"),+ (1, "upsert"),+ ]+++def test_builder_deletes_duplicate_same_level_records_without_id_conflict():+ from openviking.storage.context_update_plan import (+ ContentState,+ IndexState,+ build_context_update_plan,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot++ root = "viking://resources/repo"+ records = {+ record_id: VectorRecordSnapshot(+ record_id,+ root + "/a.py",+ "a.py",+ 2,+ {"md5": "old", "abstract": "old"},+ )+ for record_id in ("a-primary", "a-duplicate")+ }+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(root, "semantic_and_vectors"),+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py",+ ContentState.MODIFIED,+ IndexState.STALE,+ old_kind="file",+ new_kind="file",+ md5="new",+ )+ }+ ),+ new_kinds={"": "directory", "a.py": "file"},+ artifact_paths={"a.py": "repository/a.py"},+ records=records,+ is_code_repo=True,+ account_id="acc",+ )++ slot = next(+ entry for entry in plan.semantic_plan.tree.entries if entry.relative_path == "a.py"+ ).slot(2)+ assert slot.record_id in records+ assert [(action.action.value, action.record_id) for action in plan.direct_index_actions] == [+ ("delete", ({"a-primary", "a-duplicate"} - {slot.record_id}).pop())+ ]+++def test_healthy_noop_produces_no_actions_or_semantic_plan():+ from openviking.storage.context_update_plan import (+ ContentState,+ IndexState,+ build_context_update_plan,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot++ root = "viking://resources/repo"+ records = {+ "root-l0": VectorRecordSnapshot("root-l0", root, "", 0, {"abstract": "root abstract"}),+ "root-l1": VectorRecordSnapshot("root-l1", root, "", 1, {"abstract": "root overview"}),+ "a-l2": VectorRecordSnapshot(+ "a-l2",+ root + "/a.py",+ "a.py",+ 2,+ {"md5": "same", "abstract": "a summary"},+ ),+ }+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(root, "semantic_and_vectors"),+ diff=ResourceDiffResult(+ {+ "": ResourceDiffEntry(+ "",+ ContentState.UNCHANGED,+ IndexState.COMPLETE,+ old_kind="directory",+ new_kind="directory",+ ),+ "a.py": ResourceDiffEntry(+ "a.py",+ ContentState.UNCHANGED,+ IndexState.COMPLETE,+ old_kind="file",+ new_kind="file",+ md5="same",+ ),+ }+ ),+ new_kinds={"": "directory", "a.py": "file"},+ artifact_paths={"a.py": "repository/a.py"},+ records=records,+ is_code_repo=True,+ account_id="acc",+ )++ assert plan.is_noop()+ assert plan.content_tree_actions == ()+ assert plan.semantic_plan is None+ assert plan.direct_index_actions == ()+++async def test_vectorize_disabled_plan_ignores_stale_vectors_and_compares_formal_content():+ from openviking.storage.context_update_plan import (+ ContentState,+ build_context_update_plan_from_snapshot,+ )+ from openviking.storage.resource_rnfv import (+ FormalEntry,+ FormalTreeSnapshot,+ NewArtifactSnapshot,+ NewEntry,+ RequestIntent,+ RNFVSnapshot,+ VectorIndexSnapshot,+ VectorRecordSnapshot,+ )++ root = "viking://resources/repo"+ store = AsyncMock()+ store.read_bytes.return_value = b"v1"+ target = AsyncMock()+ target.read_file.return_value = b"v2"+ vikingdb = AsyncMock()+ snapshot = RNFVSnapshot(+ RequestIntent(root, "vectors_only", vectorize=False),+ NewArtifactSnapshot({"a.txt": NewEntry(md5="hash-v1")}),+ FormalTreeSnapshot({"a.txt": FormalEntry()}),+ VectorIndexSnapshot(+ {+ "stale-l2": VectorRecordSnapshot(+ "stale-l2",+ "viking://resources/other/a.txt",+ "a.txt",+ 2,+ {"md5": "hash-v1", "abstract": "stale summary"},+ )+ },+ frozenset({"id", "uri", "level", "md5", "abstract"}),+ complete=False,+ ),+ )++ diff, plan = await build_context_update_plan_from_snapshot(+ snapshot=snapshot,+ store=store,+ artifact_ref=object(),+ target=target,+ vikingdb=vikingdb,+ context_type="resource",+ is_code_repo=False,+ account_id="acc",+ ctx=object(),+ root_preexisting=True,+ artifact_paths={"a.txt": "repository/a.txt"},+ )++ assert diff.entries["a.txt"].content_state is ContentState.MODIFIED+ assert [+ (action.operation.value, action.relative_path) for action in plan.content_tree_actions+ ] == [("upsert", "a.txt")]+ assert plan.semantic_plan is None+ assert plan.direct_index_actions == ()+ store.read_bytes.assert_awaited_once()+ target.read_file.assert_awaited_once_with("a.txt")+ vikingdb.hydrate_incremental_records.assert_not_awaited()+++async def test_vectorize_enabled_keeps_md5_noop_without_formal_content_reads():+ from openviking.storage.context_update_plan import ContentState+ from openviking.storage.resource_diff import resolve_resource_diff+ from openviking.storage.resource_rnfv import (+ FormalEntry,+ FormalTreeSnapshot,+ NewArtifactSnapshot,+ NewEntry,+ RequestIntent,+ RNFVSnapshot,+ VectorIndexSnapshot,+ VectorRecordSnapshot,+ )++ root = "viking://resources/repo"+ snapshot = RNFVSnapshot(+ RequestIntent(root, "semantic_and_vectors", vectorize=True),+ NewArtifactSnapshot({"a.txt": NewEntry(md5="same")}),+ FormalTreeSnapshot({"a.txt": FormalEntry()}),+ VectorIndexSnapshot(+ {"a-l2": VectorRecordSnapshot("a-l2", root + "/a.txt", "a.txt", 2, {"md5": "same"})},+ frozenset({"id", "uri", "level", "md5"}),+ ),+ )+ store = AsyncMock()+ target = AsyncMock()++ diff = await resolve_resource_diff(+ snapshot,+ store=store,+ artifact_ref=object(),+ target=target,+ artifact_paths={"a.txt": "repository/a.txt"},+ )++ assert diff.entries["a.txt"].content_state is ContentState.UNCHANGED+ store.read_bytes.assert_not_awaited()+ target.read_file.assert_not_awaited()+++async def test_vectorize_disabled_semantic_plan_regenerates_without_vector_hydration():+ from openviking.storage.context_update_plan import build_context_update_plan_from_snapshot+ from openviking.storage.resource_rnfv import (+ FormalEntry,+ FormalTreeSnapshot,+ NewArtifactSnapshot,+ NewEntry,+ RequestIntent,+ RNFVSnapshot,+ ScalarIntent,+ VectorIndexSnapshot,+ )++ root = "viking://resources/repo"+ store = AsyncMock()+ store.read_bytes.side_effect = lambda _ref, path: {+ "repository/a.txt": b"new a",+ "repository/b.txt": b"same b",+ }[path]+ target = AsyncMock()+ target.read_file.side_effect = lambda path: {+ "a.txt": b"old a",+ "b.txt": b"same b",+ }[path]+ vikingdb = AsyncMock()+ snapshot = RNFVSnapshot(+ RequestIntent(+ root,+ "semantic_and_vectors",+ vectorize=False,+ scalar_intents=(ScalarIntent("search_tags", "replace", ("team=search",)),),+ ),+ NewArtifactSnapshot(+ {+ "a.txt": NewEntry(md5="new-a"),+ "b.txt": NewEntry(md5="same-b"),+ }+ ),+ FormalTreeSnapshot({"a.txt": FormalEntry(), "b.txt": FormalEntry()}),+ VectorIndexSnapshot({}, frozenset()),+ )++ _, plan = await build_context_update_plan_from_snapshot(+ snapshot=snapshot,+ store=store,+ artifact_ref=object(),+ target=target,+ vikingdb=vikingdb,+ context_type="resource",+ is_code_repo=False,+ account_id="acc",+ ctx=object(),+ root_preexisting=True,+ artifact_paths={+ "a.txt": "repository/a.txt",+ "b.txt": "repository/b.txt",+ },+ )++ assert plan.semantic_plan is not None+ assert {+ entry.relative_path: entry.semantic_action.value+ for entry in plan.semantic_plan.tree.entries+ } == {"": "aggregate", "a.txt": "generate", "b.txt": "generate"}+ assert all(+ slot.action.value == "none"+ for entry in plan.semantic_plan.tree.entries+ for slot in entry.index_slots+ )+ assert all(+ not slot.upsert_fields and slot.field_patch is None+ for entry in plan.semantic_plan.tree.entries+ for slot in entry.index_slots+ )+ assert all(not entry.repair for entry in plan.semantic_plan.tree.entries)+ assert plan.semantic_plan.ingest_options.search_tags is None+ assert plan.direct_index_actions == ()+ vikingdb.hydrate_incremental_records.assert_not_awaited()+++def test_restore_with_matching_index_reuses_file_summary_and_aggregates_parent():+ from openviking.storage.context_update_plan import (+ ContentState,+ IndexState,+ build_context_update_plan,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot++ root = "viking://resources/repo"+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(root, "semantic_and_vectors"),+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py",+ ContentState.RESTORE,+ IndexState.COMPLETE,+ new_kind="file",+ md5="same",+ )+ }+ ),+ new_kinds={"": "directory", "a.py": "file"},+ artifact_paths={"a.py": "repository/a.py"},+ records={+ "a-l2": VectorRecordSnapshot(+ "a-l2",+ root + "/a.py",+ "a.py",+ 2,+ {"md5": "same", "abstract": "old summary"},+ ),+ "root-l0": VectorRecordSnapshot("root-l0", root, "", 0, {"abstract": "old root"}),+ "root-l1": VectorRecordSnapshot("root-l1", root, "", 1, {"abstract": "old overview"}),+ },+ is_code_repo=True,+ account_id="acc",+ )++ entries = {entry.relative_path: entry for entry in plan.semantic_plan.tree.entries}+ assert entries["a.py"].semantic_action.value == "reuse"+ assert entries[""].semantic_action.value == "aggregate"+ assert entries[""].membership_changed is True+++def test_builder_keeps_append_intent_as_fallback_for_semantic_upsert():+ from openviking.storage.context_update_plan import (+ ContentState,+ IndexState,+ build_context_update_plan,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent, ScalarIntent, VectorRecordSnapshot++ root = "viking://resources/repo"+ record = VectorRecordSnapshot(+ "id-a",+ root + "/a.py",+ "a.py",+ 2,+ {"md5": "old", "abstract": "old abstract", "search_tags": ["scope=old"]},+ )+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(+ root,+ "semantic_and_vectors",+ scalar_intents=(ScalarIntent("search_tags", "append", ("owner=new",)),),+ ),+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py",+ ContentState.MODIFIED,+ IndexState.STALE,+ old_kind="file",+ new_kind="file",+ md5="new",+ ),+ }+ ),+ new_kinds={"": "directory", "a.py": "file"},+ artifact_paths={"a.py": "repository/a.py"},+ records={"id-a": record},+ is_code_repo=True,+ account_id="acc",+ )+ slot = next(+ e for e in plan.semantic_plan.tree.entries if e.relative_path == "a.py"+ ).index_slots[0]+ assert slot.upsert_fields == {"search_tags": ["scope=old", "owner=new"]}+ assert slot.field_patch == FieldPatch(+ {"search_tags": ["owner=new"]},+ {"search_tags": "append"},+ )+ assert slot.fallback_to_patch is True+ assert plan.semantic_plan.ingest_options.search_tags is None+++def test_builder_skips_scalar_update_when_vectorization_is_disabled():+ from openviking.storage.context_update_plan import (+ ContentState,+ IndexState,+ build_context_update_plan,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import (+ RequestIntent,+ ScalarIntent,+ VectorRecordSnapshot,+ )++ root = "viking://resources/repo"+ record = VectorRecordSnapshot(+ "a-l2",+ root + "/a.py",+ "a.py",+ 2,+ {"abstract": "old", "search_tags": ["scope=old"]},+ )+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(+ root,+ "semantic_and_vectors",+ vectorize=False,+ scalar_intents=(ScalarIntent("search_tags", "replace", ("scope=new",)),),+ ),+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py",+ ContentState.MODIFIED,+ IndexState.STALE,+ old_kind="file",+ new_kind="file",+ md5="new",+ )+ }+ ),+ new_kinds={"": "directory", "a.py": "file"},+ artifact_paths={"a.py": "repository/a.py"},+ records={record.record_id: record},+ is_code_repo=True,+ account_id="acc",+ )++ assert plan.semantic_plan is not None+ assert plan.direct_index_actions == ()+++def test_vectors_only_upsert_preserves_existing_custom_scalars():+ from openviking.storage.context_update_plan import (+ ContentState,+ IndexState,+ build_context_update_plan,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot++ root = "viking://resources/repo"+ record = VectorRecordSnapshot(+ "external-id",+ root + "/a.py",+ "a.py",+ 2,+ {+ "abstract": "old",+ "md5": "old",+ "business_priority": 7,+ "vector": [1.0],+ },+ )+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(root, "vectors_only"),+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py",+ ContentState.MODIFIED,+ IndexState.STALE,+ old_kind="file",+ new_kind="file",+ md5="new",+ )+ }+ ),+ new_kinds={"": "directory", "a.py": "file"},+ artifact_paths={"a.py": "repository/a.py"},+ records={record.record_id: record},+ is_code_repo=False,+ account_id="acc",+ )++ action = plan.direct_index_actions[0]+ assert action.action.value == "upsert"+ assert action.record_id == "external-id"+ assert action.upsert_fields["business_priority"] == 7+ assert "vector" not in action.upsert_fields+++def test_vectors_only_restore_with_complete_index_does_not_reembed():+ from openviking.storage.context_update_plan import (+ ContentState,+ IndexState,+ build_context_update_plan,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot++ root = "viking://resources/repo"+ record = VectorRecordSnapshot(+ "existing-id",+ root + "/a.py",+ "a.py",+ 2,+ {"md5": "same", "abstract": "old", "search_tags": ["scope=old"]},+ )+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(root, "vectors_only"),+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py",+ ContentState.RESTORE,+ IndexState.COMPLETE,+ new_kind="file",+ md5="same",+ )+ }+ ),+ new_kinds={"": "directory", "a.py": "file"},+ artifact_paths={"a.py": "repository/a.py"},+ records={record.record_id: record},+ is_code_repo=False,+ account_id="acc",+ )++ assert [+ (action.operation.value, action.relative_path) for action in plan.content_tree_actions+ ] == [("upsert", "a.py")]+ assert plan.semantic_plan is None+ assert plan.direct_index_actions == ()+++def test_vectors_only_unchanged_file_with_extra_level_only_deletes_conflict():+ from openviking.storage.context_update_plan import (+ ContentState,+ IndexState,+ build_context_update_plan,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot++ root = "viking://resources/repo"+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(root, "vectors_only"),+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py",+ ContentState.UNCHANGED,+ IndexState.LEVEL_CONFLICT,+ old_kind="file",+ new_kind="file",+ md5="same",+ )+ }+ ),+ new_kinds={"": "directory", "a.py": "file"},+ artifact_paths={"a.py": "repository/a.py"},+ records={+ "valid-l2": VectorRecordSnapshot(+ "valid-l2", root + "/a.py", "a.py", 2, {"md5": "same"}+ ),+ "invalid-l0": VectorRecordSnapshot("invalid-l0", root + "/a.py", "a.py", 0, {}),+ },+ is_code_repo=False,+ account_id="acc",+ )++ assert [(action.action.value, action.record_id) for action in plan.direct_index_actions] == [+ ("delete", "invalid-l0")+ ]+++def test_builder_carries_request_ingest_options_into_semantic_plan():+ from openviking.storage.context_update_plan import (+ ContentState,+ IndexState,+ build_context_update_plan,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent+ from openviking.utils.ingest_options import IngestOptions++ root = "viking://resources/repo"+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(root, "semantic_and_vectors"),+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py",+ ContentState.ADDED,+ IndexState.MISSING,+ new_kind="file",+ md5="new",+ )+ }+ ),+ new_kinds={"": "directory", "a.py": "file"},+ artifact_paths={"a.py": "repository/a.py"},+ records={},+ is_code_repo=False,+ account_id="acc",+ ingest_options=IngestOptions.from_search_tags(["team=search"], mode="append"),+ )++ assert plan.semantic_plan.ingest_options.search_tags == ["team=search"]+ assert plan.semantic_plan.ingest_options.search_tag_mode == "append"+++def test_missing_index_for_existing_content_uses_partial_update_repair():+ from openviking.storage.context_update_plan import (+ ContentState,+ IndexState,+ build_context_update_plan,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent++ root = "viking://resources/repo"+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(root, "semantic_and_vectors"),+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py",+ ContentState.UNCHANGED,+ IndexState.MISSING,+ old_kind="file",+ new_kind="file",+ md5="same",+ )+ }+ ),+ new_kinds={"": "directory", "a.py": "file"},+ artifact_paths={"a.py": "repository/a.py"},+ records={},+ is_code_repo=False,+ account_id="acc",+ )++ file_entry = next(+ entry for entry in plan.semantic_plan.tree.entries if entry.relative_path == "a.py"+ )+ assert file_entry.slot(2).action.value == "merge"++[email protected]("content_state", ["added", "restore"])+def test_new_or_restored_content_does_not_use_partial_update(content_state):+ from openviking.storage.context_update_plan import build_context_update_plan+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent++ root = "viking://resources/repo"+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(root, "semantic_and_vectors"),+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py", content_state, "missing", new_kind="file", md5="new"+ )+ }+ ),+ new_kinds={"": "directory", "a.py": "file"},+ artifact_paths={"a.py": "repository/a.py"},+ records={},+ is_code_repo=False,+ account_id="acc",+ )++ file_entry = next(+ entry for entry in plan.semantic_plan.tree.entries if entry.relative_path == "a.py"+ )+ assert file_entry.slot(2).action.value == "upsert"+++def test_vectors_only_missing_index_for_existing_content_uses_partial_update_repair():+ from openviking.storage.context_update_plan import build_context_update_plan+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent++ root = "viking://resources/repo"+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(root, "vectors_only"),+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py",+ "unchanged",+ "missing",+ old_kind="file",+ new_kind="file",+ md5="same",+ )+ }+ ),+ new_kinds={"a.py": "file"},+ artifact_paths={"a.py": "repository/a.py"},+ records={},+ is_code_repo=False,+ account_id="acc",+ )++ assert len(plan.direct_index_actions) == 1+ assert plan.direct_index_actions[0].action.value == "merge"+++def test_vectors_only_partial_repair_preserves_explicit_replace_empty_tags():+ from openviking.storage.context_update_plan import build_context_update_plan+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent, ScalarIntent++ root = "viking://resources/repo"+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(+ root,+ "vectors_only",+ scalar_intents=(ScalarIntent("search_tags", "replace", ()),),+ ),+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py",+ "unchanged",+ "missing",+ old_kind="file",+ new_kind="file",+ md5="same",+ )+ }+ ),+ new_kinds={"a.py": "file"},+ artifact_paths={"a.py": "repository/a.py"},+ records={},+ is_code_repo=False,+ account_id="acc",+ )++ action = plan.direct_index_actions[0]+ assert action.action.value == "merge"+ assert action.field_patch == FieldPatch(+ {"search_tags": []},+ {"search_tags": "replace"},+ )+++def test_builder_backfills_missing_md5_and_merges_scalar_update():+ from openviking.storage.context_update_plan import (+ ContentState,+ IndexState,+ build_context_update_plan,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import (+ RequestIntent,+ ScalarIntent,+ VectorRecordSnapshot,+ )++ root = "viking://resources/repo"+ record = VectorRecordSnapshot(+ "a-l2",+ root + "/a.py",+ "a.py",+ 2,+ {+ "md5": "",+ "abstract": "old",+ "search_tags": ["scope=old"],+ },+ )+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(+ root,+ "semantic_and_vectors",+ scalar_intents=(ScalarIntent("search_tags", "replace", ("scope=new",)),),+ ),+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py",+ ContentState.UNCHANGED,+ IndexState.STALE,+ old_kind="file",+ new_kind="file",+ md5="resolved-md5",+ )+ }+ ),+ new_kinds={"": "directory", "a.py": "file"},+ artifact_paths={"a.py": "repository/a.py"},+ records={"a-l2": record},+ is_code_repo=True,+ account_id="acc",+ )++ assert plan.semantic_plan is None+ assert len(plan.direct_index_actions) == 1+ action = plan.direct_index_actions[0]+ assert action.record_id == "a-l2"+ assert action.action.value == "update_fields"+ assert action.field_patch.values == {+ "md5": "resolved-md5",+ "search_tags": ["scope=new"],+ }+++def test_builder_preserves_existing_record_identity_and_custom_scalars_on_upsert():+ from openviking.storage.context_update_plan import (+ ContentState,+ IndexState,+ build_context_update_plan,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot++ root = "viking://resources/repo"+ record = VectorRecordSnapshot(+ "external-record-id",+ root + "/a.py",+ "a.py",+ 2,+ {+ "abstract": "old",+ "business_priority": 7,+ "vector": [1.0],+ "content": "large old body",+ },+ )+ plan = build_context_update_plan(+ root_uri=root,+ context_type="resource",+ request=RequestIntent(root, "semantic_and_vectors"),+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py",+ ContentState.MODIFIED,+ IndexState.STALE,+ old_kind="file",+ new_kind="file",+ md5="new",+ )+ }+ ),+ new_kinds={"": "directory", "a.py": "file"},+ artifact_paths={"a.py": "repository/a.py"},+ records={record.record_id: record},+ is_code_repo=True,+ account_id="acc",+ )++ slot = next(+ entry for entry in plan.semantic_plan.tree.entries if entry.relative_path == "a.py"+ ).slot(2)+ assert slot.record_id == "external-record-id"+ assert slot.scalar_override()["business_priority"] == 7+ assert "vector" not in slot.scalar_override()+ assert "content" not in slot.scalar_override()+++async def test_snapshot_builder_returns_one_canonical_context_plan():+ from openviking.storage.context_update_plan import (+ ContentState,+ build_context_update_plan_from_snapshot,+ )+ from openviking.storage.resource_rnfv import (+ FormalTreeSnapshot,+ NewArtifactSnapshot,+ NewEntry,+ RequestIntent,+ RNFVSnapshot,+ VectorIndexSnapshot,+ )++ root = "viking://resources/repo"+ snapshot = RNFVSnapshot(+ request=RequestIntent(root, "semantic_and_vectors"),+ new=NewArtifactSnapshot({"a.py": NewEntry(md5="new")}),+ formal=FormalTreeSnapshot({}),+ vectors=VectorIndexSnapshot({}, frozenset({"id", "uri", "level", "md5"})),+ )+ diff, plan = await build_context_update_plan_from_snapshot(+ snapshot=snapshot,+ store=AsyncMock(),+ artifact_ref=object(),+ target=AsyncMock(),+ vikingdb=AsyncMock(),+ context_type="resource",+ is_code_repo=True,+ account_id="acc",+ ctx=object(),+ root_preexisting=False,+ artifact_paths={"a.py": "repository/a.py"},+ )++ assert diff.entries["a.py"].content_state is ContentState.ADDED+ assert [a.relative_path for a in plan.content_tree_actions] == ["", "a.py"]+ assert plan.content_tree_actions[1].artifact_path == "repository/a.py"+ assert plan.semantic_plan is not None+ entries = {entry.relative_path: entry for entry in plan.semantic_plan.tree.entries}+ assert entries[""].semantic_action.value == "aggregate"+ assert entries["a.py"].semantic_action.value == "generate"+++async def test_snapshot_builder_hydrates_scalars_for_vectors_only_upsert():+ from openviking.storage.context_update_plan import build_context_update_plan_from_snapshot+ from openviking.storage.resource_rnfv import (+ FormalEntry,+ FormalTreeSnapshot,+ NewArtifactSnapshot,+ NewEntry,+ RequestIntent,+ RNFVSnapshot,+ VectorIndexSnapshot,+ VectorRecordSnapshot,+ )++ root = "viking://resources/repo"+ record = VectorRecordSnapshot(+ "existing-id",+ root + "/a.py",+ "a.py",+ 2,+ {"md5": "old", "abstract": "old summary"},+ )+ snapshot = RNFVSnapshot(+ request=RequestIntent(root, "vectors_only"),+ new=NewArtifactSnapshot({"a.py": NewEntry(md5="new")}),+ formal=FormalTreeSnapshot({"a.py": FormalEntry()}),+ vectors=VectorIndexSnapshot(+ {record.record_id: record},+ frozenset({"id", "uri", "level", "md5", "abstract"}),+ ),+ )+ vikingdb = AsyncMock()+ vikingdb.hydrate_incremental_records.return_value = {+ record.record_id: {+ "id": record.record_id,+ "uri": record.uri,+ "level": 2,+ "md5": "old",+ "abstract": "old summary",+ "business_priority": 7,+ }+ }++ _, plan = await build_context_update_plan_from_snapshot(+ snapshot=snapshot,+ store=AsyncMock(),+ artifact_ref=object(),+ target=AsyncMock(),+ vikingdb=vikingdb,+ context_type="resource",+ is_code_repo=False,+ account_id="acc",+ ctx=object(),+ root_preexisting=True,+ artifact_paths={"a.py": "repository/a.py"},+ )++ vikingdb.hydrate_incremental_records.assert_awaited_once()+ assert plan.direct_index_actions[0].record_id == record.record_id+ assert plan.direct_index_actions[0].upsert_fields["business_priority"] == 7+++async def test_file_root_plan_uses_file_refresh_without_directory_semantic_tree():+ from openviking.storage.context_update_plan import build_context_update_plan_from_snapshot+ from openviking.storage.resource_rnfv import (+ FormalEntry,+ FormalTreeSnapshot,+ NewArtifactSnapshot,+ NewEntry,+ RequestIntent,+ RNFVSnapshot,+ VectorIndexSnapshot,+ VectorRecordSnapshot,+ )++ root = "viking://resources/report.md"+ record = VectorRecordSnapshot("report-l2", root, "", 2, {"md5": "old", "abstract": "old"})+ snapshot = RNFVSnapshot(+ RequestIntent(root, "semantic_and_vectors"),+ NewArtifactSnapshot({"": NewEntry(md5="new")}),+ FormalTreeSnapshot({"": FormalEntry()}),+ VectorIndexSnapshot({"report-l2": record}, frozenset({"id", "uri", "level", "md5"})),+ )+ vikingdb = AsyncMock()+ vikingdb.hydrate_incremental_records.return_value = {"report-l2": {"abstract": "old"}}++ _, plan = await build_context_update_plan_from_snapshot(+ snapshot=snapshot,+ store=AsyncMock(read_bytes=AsyncMock(return_value=b"new")),+ artifact_ref=object(),+ target=AsyncMock(read_file=AsyncMock(return_value=b"old")),+ vikingdb=vikingdb,+ context_type="resource",+ is_code_repo=False,+ account_id="acc",+ ctx=object(),+ root_preexisting=True,+ artifact_paths={"": "report.md"},+ root_is_file=True,+ )++ assert [+ (action.operation.value, action.relative_path) for action in plan.content_tree_actions+ ] == [("upsert", "")]+ assert plan.semantic_plan is None+ assert plan.file_refresh is not None+ assert plan.file_refresh.md5 == "new"+++async def test_hydration_promotes_a_dependency_when_its_record_disappears():+ from openviking.storage.context_update_plan import (+ ContentState,+ IndexState,+ hydrate_context_plan_records,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import VectorRecordSnapshot++ root = "viking://resources/repo"+ inventory = {+ "changed": VectorRecordSnapshot("changed", f"{root}/a.py", "a.py", 2, {"md5": "old"}),+ "sibling": VectorRecordSnapshot("sibling", f"{root}/b.py", "b.py", 2, {"md5": "same"}),+ }+ vikingdb = AsyncMock()+ vikingdb.hydrate_incremental_records.return_value = {+ "changed": {+ "id": "changed",+ "uri": f"{root}/a.py",+ "level": 2,+ "abstract": "old a",+ }+ }+ records, (active, _, retained) = await hydrate_context_plan_records(+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py",+ ContentState.MODIFIED,+ IndexState.STALE,+ old_kind="file",+ new_kind="file",+ md5="new",+ ),+ "b.py": ResourceDiffEntry(+ "b.py",+ ContentState.UNCHANGED,+ IndexState.COMPLETE,+ old_kind="file",+ new_kind="file",+ md5="same",+ ),+ }+ ),+ new_kinds={"": "directory", "a.py": "file", "b.py": "file"},+ inventory=inventory,+ vikingdb=vikingdb,+ ctx=object(),+ )++ assert records["sibling"].record_id == "sibling"+ assert "b.py" in active+ assert "b.py" in retained+++async def test_hydration_reads_summaries_before_full_scalars_for_active_records():+ from openviking.storage.context_update_plan import (+ ContentState,+ IndexState,+ hydrate_context_plan_records,+ )+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot++ root = "viking://resources/repo"+ inventory = {+ "changed": VectorRecordSnapshot("changed", f"{root}/a.py", "a.py", 2, {"md5": "old"}),+ "sibling": VectorRecordSnapshot("sibling", f"{root}/b.py", "b.py", 2, {"md5": "same"}),+ }+ vikingdb = AsyncMock()+ vikingdb.hydrate_incremental_records.side_effect = [+ {+ "changed": {"abstract": "old a"},+ "sibling": {"abstract": "old b"},+ },+ {"changed": {"business_priority": 7}},+ ]+ records, (active, _, retained) = await hydrate_context_plan_records(+ diff=ResourceDiffResult(+ {+ "a.py": ResourceDiffEntry(+ "a.py",+ ContentState.MODIFIED,+ IndexState.STALE,+ old_kind="file",+ new_kind="file",+ md5="new",+ ),+ "b.py": ResourceDiffEntry(+ "b.py",+ ContentState.UNCHANGED,+ IndexState.COMPLETE,+ old_kind="file",+ new_kind="file",+ md5="same",+ ),+ }+ ),+ new_kinds={"": "directory", "a.py": "file", "b.py": "file"},+ inventory=inventory,+ vikingdb=vikingdb,+ ctx=object(),+ request=RequestIntent(root, "semantic_and_vectors"),+ )++ assert active == {"", "a.py"}+ assert retained == {"", "a.py", "b.py"}+ assert records["sibling"].fields == {"md5": "same", "abstract": "old b"}+ assert records["changed"].fields == {+ "md5": "old",+ "abstract": "old a",+ "business_priority": 7,+ }+ assert vikingdb.hydrate_incremental_records.await_args_list[0].kwargs["output_fields"] == {+ "abstract"+ }+ assert "output_fields" not in vikingdb.hydrate_incremental_records.await_args_list[1].kwargs+ assert set(vikingdb.hydrate_incremental_records.await_args_list[1].args[0]) == {"changed"}+++async def test_content_executor_uploads_files_with_bounded_concurrency():+ from unittest.mock import patch++ import openviking.concurrency as concurrency_utils+ from openviking.storage.context_update_plan import (+ ContentTreeAction,+ execute_content_tree_actions,+ )++ active = peak = 0+ gate = asyncio.Event()++ class Store:+ async def read_bytes(self, ref, path):+ return path.encode()++ class Target:+ async def write_file(self, path, data):+ nonlocal active, peak+ active += 1+ peak = max(peak, active)+ if active == 3:+ gate.set()+ try:+ await asyncio.wait_for(gate.wait(), 1)+ finally:+ active -= 1++ async def delete_path(self, path, *, is_dir):+ return None++ async def mkdir(self, path):+ return None++ actions = tuple(+ ContentTreeAction(+ "upsert",+ f"{index}.py",+ new_kind="file",+ artifact_path=f"{index}.py",+ md5=str(index),+ )+ for index in range(8)+ )+ create_task = asyncio.create_task+ with patch.object(concurrency_utils.asyncio, "create_task", wraps=create_task) as created:+ await execute_content_tree_actions(+ actions,+ store=Store(),+ artifact_ref=object(),+ target=Target(),+ concurrency=3,+ )++ assert peak == 3+ assert created.call_count == 3+++async def test_bounded_map_preserves_order_and_processes_all_items():+ from unittest.mock import patch++ import openviking.concurrency as concurrency_utils++ started = 0+ release = asyncio.Event()++ async def work(value: int) -> int:+ nonlocal started+ started += 1+ if started == 3:+ release.set()+ await release.wait()+ return value * 2++ create_task = asyncio.create_task+ with patch.object(concurrency_utils.asyncio, "create_task", wraps=create_task) as created:+ result = await concurrency_utils.bounded_map(range(100), work, concurrency=3)++ assert started == 100+ assert created.call_count == 3+ assert result == [index * 2 for index in range(100)]+++async def test_bounded_map_stops_claiming_after_worker_failure():+ from openviking.concurrency import bounded_map++ started: list[int] = []+ release = asyncio.Event()++ async def work(value: int) -> int:+ started.append(value)+ if value == 0:+ raise RuntimeError("boom")+ await release.wait()+ return value++ task = asyncio.create_task(bounded_map(range(100), work, concurrency=3))+ await asyncio.sleep(0)+ release.set()+ with pytest.raises(RuntimeError, match="boom"):+ await task++ assert set(started) <= {0, 1, 2}+++async def test_content_executor_logs_failed_action_with_correlation(monkeypatch):+ import openviking.storage.context_update_plan as plan_module+ from openviking.service.task_work_index import bind_task_context+ from openviking.storage.context_update_plan import (+ ContentTreeAction,+ execute_content_tree_actions,+ )+ from openviking.telemetry import OperationTelemetry, bind_telemetry++ class Target:+ async def delete_path(self, path, *, is_dir):+ raise OSError("delete failed")++ action = ContentTreeAction("delete", "old.md", old_kind="file")+ telemetry = OperationTelemetry("add_resource_job", enabled=True)+ log_exception = MagicMock()+ monkeypatch.setattr(plan_module.logger, "exception", log_exception)++ with (+ bind_task_context("task-1", "acct", "user"),+ bind_telemetry(telemetry),+ pytest.raises(OSError, match="delete failed"),+ ):+ await execute_content_tree_actions(+ (action,),+ store=object(),+ artifact_ref=object(),+ target=Target(),+ )++ message, correlation, operation, relative_path, old_kind, new_kind = (+ log_exception.call_args.args+ )+ assert "[ContentTreeActionFailed]" in message+ assert correlation == f"task_id=task-1 telemetry_id={telemetry.telemetry_id}"+ assert (operation, relative_path, old_kind, new_kind) == (+ "delete",+ "old.md",+ "file",+ "-",+ )+ assert log_exception.call_args.kwargs == {}+++async def test_resource_processor_dispatches_direct_index_actions_without_semantic(monkeypatch):+ from openviking.server.identity import RequestContext, Role+ from openviking.storage.context_update_plan import DirectIndexAction+ from openviking.utils.resource_processor import ResourceProcessor+ from openviking_cli.session.user_id import UserIdentifier++ enqueued = []++ async def enqueue(queue, message, **kwargs):+ del queue, kwargs+ enqueued.append(message)+ return True++ queue_manager = SimpleNamespace(+ EMBEDDING="embedding",+ get_queue=lambda *args, **kwargs: object(),+ )+ monkeypatch.setattr("openviking.storage.queuefs.get_queue_manager", lambda: queue_manager)+ monkeypatch.setattr("openviking.utils.embedding_utils._enqueue_embedding_message", enqueue)+ processor = ResourceProcessor(SimpleNamespace(get_embedder=lambda: None))+ processor._vectorize_resource_file = AsyncMock()+ ctx = RequestContext(UserIdentifier("acc", "user"), Role.USER)+ await processor._enqueue_index_actions(+ (+ DirectIndexAction("delete", "viking://resources/repo/a.py", 2, "id-a"),+ DirectIndexAction(+ "update_fields",+ "viking://resources/repo/b.py",+ 2,+ "id-b",+ field_patch=FieldPatch(+ {"search_tags": ["scope=new"]},+ {"search_tags": "replace"},+ ),+ ),+ DirectIndexAction(+ "merge",+ "viking://resources/repo/c.py",+ 2,+ "id-c",+ field_patch=FieldPatch(+ {"search_tags": ["scope=new"]},+ {"search_tags": "append"},+ ),+ md5="new-md5",+ ),+ ),+ ctx=ctx,+ )+ assert [msg.action.value for msg in enqueued] == ["delete", "update_fields"]+ assert enqueued[0].record_ids == ["id-a"]+ assert enqueued[1].update_fields["search_tags"] == ["scope=new"]+ processor._vectorize_resource_file.assert_awaited_once_with(+ "viking://resources/repo/c.py",+ ctx=ctx,+ file_md5="new-md5",+ scalar_override={"_record_id": "id-c"},+ action="merge",+ field_patch=FieldPatch(+ {"search_tags": ["scope=new"]},+ {"search_tags": "append"},+ ),+ )+++def test_semantic_message_roundtrip_uses_explicit_plan():+ from openviking.storage.context_update_plan import (+ IndexSlot,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+ )+ from openviking.storage.queuefs.semantic_msg import SemanticMsg++ plan = SemanticPlan(+ "viking://resources/repo",+ "resource",+ tree=SemanticTreeSnapshot(+ (+ SemanticTreeEntry("", "directory", "unchanged", "aggregate"),+ SemanticTreeEntry(+ "a.py",+ "file",+ "modified",+ "generate",+ md5="new",+ index_slots=(+ IndexSlot(+ 2,+ "actual-id",+ {"abstract": "old"},+ action="upsert",+ ),+ ),+ ),+ )+ ),+ )+ msg = SemanticMsg(uri=plan.root_uri, context_type="resource", plan=plan)+ restored = SemanticMsg.from_json(msg.to_json())+ assert restored.plan == plan+ assert (+ next(entry for entry in restored.plan.tree.entries if entry.relative_path == "a.py")+ .index_slots[0]+ .record_id+ == "actual-id"+ )+++async def test_v3_tree_reuses_explicit_node_without_listing_its_subtree(monkeypatch):+ from openviking.server.identity import RequestContext, Role+ from openviking.storage.context_update_plan import (+ IndexSlot,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+ )+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor+ from openviking_cli.session.user_id import UserIdentifier++ root = "viking://resources/repo"+ calls = []++ class FS:+ _async_agfs = None++ async def ls(self, uri, **kwargs):+ calls.append(uri)+ raise AssertionError("v3 plan must not list the live tree")++ def _uri_to_path(self, uri, ctx=None):+ return uri++ fs = FS()+ fs._async_agfs = fs+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)+ processor = AsyncMock()+ plan = SemanticPlan(+ root,+ "resource",+ tree=SemanticTreeSnapshot(+ (+ SemanticTreeEntry(+ "",+ "directory",+ "unchanged",+ "reuse",+ index_slots=(IndexSlot(0, "root-l0", {"abstract": "old root"}),),+ ),+ )+ ),+ )+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=1,+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),+ semantic_plan=plan,+ )+ await executor.run(root)+ assert calls == []+ assert executor.root_write_result.wrote is False+++async def test_directory_index_slots_choose_exact_levels(monkeypatch):+ from openviking.server.identity import RequestContext, Role+ from openviking.storage.abstract_overview import AbstractOverviewWriteResult+ from openviking.storage.context_update_plan import (+ IndexSlot,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+ )+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor+ from openviking_cli.session.user_id import UserIdentifier++ root = "viking://resources/repo"+ fs = SimpleNamespace(+ _async_agfs=None,+ _uri_to_path=lambda uri, ctx=None: uri,+ )+ fs._async_agfs = fs+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )++ class Processor:+ _generate_overview = AsyncMock(return_value="overview")+ _vectorize_directory = AsyncMock(return_value={1})++ @staticmethod+ def _normalize_overview_generation(overview):+ return overview, "abstract"++ processor = Processor()+ plan = SemanticPlan(+ root,+ "resource",+ SemanticTreeSnapshot(+ (+ SemanticTreeEntry(+ "",+ "directory",+ "unchanged",+ "aggregate",+ index_slots=(+ IndexSlot(0, "l0", {"abstract": "old"}),+ IndexSlot(1, "l1", None, action="upsert"),+ ),+ ),+ )+ ),+ )+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=1,+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),+ semantic_plan=plan,+ )+ executor._write_directory_semantics = AsyncMock(+ return_value=AbstractOverviewWriteResult(wrote=True, overview_body_changed=True)+ )++ await executor.run(root)++ kwargs = processor._vectorize_directory.await_args.kwargs+ assert kwargs["include_abstract"] is False+ assert kwargs["include_overview"] is True+++async def test_directory_sidecar_write_failure_fails_semantic_plan(monkeypatch):+ from openviking.server.identity import RequestContext, Role+ from openviking.storage.context_update_plan import (+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+ )+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor+ from openviking_cli.session.user_id import UserIdentifier++ root = "viking://resources/repo"+ fs = SimpleNamespace(_async_agfs=None, _uri_to_path=lambda uri, ctx=None: uri)+ fs._async_agfs = fs+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )++ class Processor:+ _generate_overview = AsyncMock(return_value="overview")+ _vectorize_directory = AsyncMock(return_value={0, 1})++ @staticmethod+ def _normalize_overview_generation(overview):+ return overview, "abstract"++ processor = Processor()+ plan = SemanticPlan(+ root,+ "resource",+ SemanticTreeSnapshot((SemanticTreeEntry("", "directory", "unchanged", "aggregate"),)),+ )+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=1,+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),+ semantic_plan=plan,+ )+ executor._write_directory_semantics = AsyncMock(side_effect=OSError("sidecar unavailable"))++ with pytest.raises(OSError, match="sidecar unavailable"):+ await executor.run(root)++ processor._vectorize_directory.assert_not_awaited()+++async def test_directory_repair_passes_partial_update_per_missing_level(monkeypatch):+ from openviking.server.identity import RequestContext, Role+ from openviking.storage.abstract_overview import AbstractOverviewWriteResult+ from openviking.storage.context_update_plan import (+ IndexAction,+ IndexSlot,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+ )+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor+ from openviking_cli.session.user_id import UserIdentifier++ root = "viking://resources/repo"+ fs = SimpleNamespace(_async_agfs=None, _uri_to_path=lambda uri, ctx=None: uri)+ fs._async_agfs = fs+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )++ class Processor:+ _generate_overview = AsyncMock(return_value="overview")+ _vectorize_directory = AsyncMock(return_value={0, 1})++ @staticmethod+ def _normalize_overview_generation(overview):+ return overview, "abstract"++ processor = Processor()+ plan = SemanticPlan(+ root,+ "resource",+ SemanticTreeSnapshot(+ (+ SemanticTreeEntry(+ "",+ "directory",+ "unchanged",+ "aggregate",+ index_slots=(+ IndexSlot(+ 0,+ "root-l0",+ None,+ action="merge",+ ),+ IndexSlot(+ 1,+ "root-l1",+ None,+ action="upsert",+ ),+ ),+ ),+ )+ ),+ )+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=1,+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),+ semantic_plan=plan,+ )+ executor._write_directory_semantics = AsyncMock(+ return_value=AbstractOverviewWriteResult(+ wrote=True, overview_body_changed=True, abstract_body_changed=True+ )+ )++ await executor.run(root)++ assert processor._vectorize_directory.await_args.kwargs["actions"] == {+ 0: IndexAction.MERGE,+ 1: IndexAction.UPSERT,+ }+++async def test_directory_output_unchanged_still_applies_planned_scalar_fields(monkeypatch):+ from openviking.server.identity import RequestContext, Role+ from openviking.storage.abstract_overview import AbstractOverviewWriteResult+ from openviking.storage.context_update_plan import (+ IndexSlot,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+ )+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor+ from openviking_cli.session.user_id import UserIdentifier++ root = "viking://resources/repo"+ fs = SimpleNamespace(_async_agfs=None, _uri_to_path=lambda uri, ctx=None: uri)+ fs._async_agfs = fs+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )++ class Processor:+ _generate_overview = AsyncMock(return_value="overview")+ _vectorize_directory = AsyncMock(return_value=set())+ _update_vector_fields = AsyncMock(return_value=True)++ @staticmethod+ def _normalize_overview_generation(overview):+ return overview, "abstract"++ processor = Processor()+ plan = SemanticPlan(+ root,+ "resource",+ SemanticTreeSnapshot(+ (+ SemanticTreeEntry(+ "",+ "directory",+ "unchanged",+ "aggregate",+ index_slots=(+ IndexSlot(+ 0,+ "root-l0",+ {"abstract": "abstract"},+ action="upsert",+ upsert_fields={"search_tags": ["scope=new"]},+ field_patch=FieldPatch(+ {"search_tags": ["scope=new"]},+ {"search_tags": "append"},+ ),+ fallback_to_patch=True,+ ),+ IndexSlot(+ 1,+ "root-l1",+ None,+ action="upsert",+ ),+ ),+ ),+ )+ ),+ )+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=1,+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),+ semantic_plan=plan,+ )+ executor._write_directory_semantics = AsyncMock(+ return_value=AbstractOverviewWriteResult(wrote=True)+ )+ executor._check_dir_children_changed = AsyncMock(return_value=False)+ executor._read_existing_overview_abstract = AsyncMock(return_value=("overview", "abstract"))++ await executor.run(root)++ processor._vectorize_directory.assert_not_awaited()+ processor._update_vector_fields.assert_awaited_once_with(+ record_id="root-l0",+ uri=root,+ level=0,+ field_patch=FieldPatch(+ {"search_tags": ["scope=new"]},+ {"search_tags": "append"},+ {+ "abstract": "abstract",+ "uri": root,+ "level": 0,+ "account_id": "acc",+ },+ ),+ ctx=executor._ctx,+ )+++async def test_code_summary_unchanged_still_reembeds_after_content_change(monkeypatch):+ from openviking.server.identity import RequestContext, Role+ from openviking.storage.abstract_overview import AbstractOverviewWriteResult+ from openviking.storage.context_update_plan import (+ IndexSlot,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+ )+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor+ from openviking_cli.session.user_id import UserIdentifier++ root = "viking://resources/repo"+ fs = SimpleNamespace(+ _async_agfs=None,+ _uri_to_path=lambda uri, ctx=None: uri,+ read_file_bytes=AsyncMock(return_value=b"changed body"),+ )+ fs._async_agfs = fs+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )++ class Processor:+ _generate_single_file_summary = AsyncMock(return_value={"name": "a.py", "summary": "same"})+ _update_file_vector_fields = AsyncMock(return_value=True)+ _vectorize_single_file = AsyncMock(return_value=True)+ _generate_overview = AsyncMock(return_value="overview")+ _vectorize_directory = AsyncMock(return_value=set())++ @staticmethod+ def _normalize_overview_generation(overview):+ return overview, "abstract"++ processor = Processor()+ plan = SemanticPlan(+ root,+ "resource",+ SemanticTreeSnapshot(+ (+ SemanticTreeEntry(+ "",+ "directory",+ "unchanged",+ "aggregate",+ index_slots=(IndexSlot(0, "root-l0", {"abstract": "old"}),),+ ),+ SemanticTreeEntry(+ "a.py",+ "file",+ "modified",+ "generate",+ md5="new-md5",+ index_slots=(+ IndexSlot(+ 2,+ "a-l2",+ {"abstract": "same"},+ action="upsert",+ ),+ ),+ ),+ )+ ),+ file_vector_source="summary_when_available",+ )+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=1,+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),+ semantic_plan=plan,+ )+ executor._write_directory_semantics = AsyncMock(+ return_value=AbstractOverviewWriteResult(wrote=True)+ )++ await executor.run(root)++ processor._vectorize_single_file.assert_awaited_once()+ processor._update_file_vector_fields.assert_not_awaited()+++async def test_file_repair_passes_partial_update_to_embedding(monkeypatch):+ from openviking.server.identity import RequestContext, Role+ from openviking.storage.abstract_overview import AbstractOverviewWriteResult+ from openviking.storage.context_update_plan import (+ IndexSlot,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+ )+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor+ from openviking_cli.session.user_id import UserIdentifier++ root = "viking://resources/repo"+ fs = SimpleNamespace(+ _async_agfs=None,+ _uri_to_path=lambda uri, ctx=None: uri,+ read_file_bytes=AsyncMock(return_value=b"body"),+ )+ fs._async_agfs = fs+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )++ class Processor:+ _generate_single_file_summary = AsyncMock(+ return_value={"name": "a.py", "summary": "summary"}+ )+ _vectorize_single_file = AsyncMock(return_value=True)+ _generate_overview = AsyncMock(return_value="overview")+ _vectorize_directory = AsyncMock(return_value=set())++ @staticmethod+ def _normalize_overview_generation(overview):+ return overview, "abstract"++ processor = Processor()+ plan = SemanticPlan(+ root,+ "resource",+ SemanticTreeSnapshot(+ (+ SemanticTreeEntry("", "directory", "unchanged", "aggregate"),+ SemanticTreeEntry(+ "a.py",+ "file",+ "unchanged",+ "generate",+ md5="same",+ index_slots=(+ IndexSlot(+ 2,+ "a-l2",+ None,+ action="merge",+ ),+ ),+ ),+ )+ ),+ )+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=1,+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),+ semantic_plan=plan,+ )+ executor._write_directory_semantics = AsyncMock(+ return_value=AbstractOverviewWriteResult(wrote=True)+ )++ await executor.run(root)++ assert processor._vectorize_single_file.await_args.kwargs["action"] == "merge"+++async def test_direct_only_plan_skips_semantic_queue(monkeypatch):+ from openviking.server.identity import RequestContext, Role+ from openviking.storage.context_update_plan import DirectIndexAction+ from openviking.utils.resource_processor import ResourceProcessor+ from openviking_cli.session.user_id import UserIdentifier++ root = "viking://resources/repo"+ lock = {"lease_ref": "plan"}+ viking_fs = SimpleNamespace(+ _async_agfs=SimpleNamespace(pathlock_release=AsyncMock()),+ )+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)+ processor = ResourceProcessor(SimpleNamespace(get_embedder=lambda: None))+ processor._enqueue_index_actions = AsyncMock()+ summarizer = SimpleNamespace(summarize=AsyncMock())+ processor._get_summarizer = lambda: summarizer+ action = DirectIndexAction(+ "update_fields",+ root + "/a.py",+ 2,+ "a-l2",+ field_patch=FieldPatch({"search_tags": ["scope=new"]}),+ )++ await processor.finish_prepared_resource(+ {+ "root_uri": root,+ "temp_uri": root,+ "source_committed": True,+ "target_preexisting": True,+ "root_is_file": False,+ "context_update_plan": {+ "root_uri": root,+ "context_type": "resource",+ "content_tree_actions": [],+ "semantic_plan": None,+ "direct_index_actions": [+ {+ "action": action.action.value,+ "uri": action.uri,+ "level": action.level,+ "record_id": action.record_id,+ "field_patch": action.field_patch.to_dict(),+ "md5": action.md5,+ }+ ],+ },+ },+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),+ resource_lock=lock,+ build_index=True,+ processing_mode="semantic_and_vectors",+ )++ processor._enqueue_index_actions.assert_awaited_once()+ summarizer.summarize.assert_not_awaited()+ viking_fs._async_agfs.pathlock_release.assert_awaited_once_with(lock)+++async def test_vectors_only_context_plan_does_not_run_legacy_vectorization(monkeypatch):+ from openviking.server.identity import RequestContext, Role+ from openviking.storage.context_update_plan import ContextUpdatePlan, DirectIndexAction+ from openviking.utils.resource_processor import ResourceProcessor+ from openviking_cli.session.user_id import UserIdentifier++ root = "viking://resources/repo"+ lock = {"lease_ref": "plan"}+ viking_fs = SimpleNamespace(+ _async_agfs=SimpleNamespace(pathlock_release=AsyncMock()),+ )+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)+ processor = ResourceProcessor(SimpleNamespace(get_embedder=lambda: None))+ processor._enqueue_index_actions = AsyncMock()+ processor._vectorize_prepared_files = AsyncMock(+ side_effect=AssertionError("legacy vectors_only path must not run")+ )+ action = DirectIndexAction("upsert", root + "/a.py", 2, "a-l2", md5="m")+ plan = ContextUpdatePlan(root, "resource", direct_index_actions=(action,))++ await processor.finish_prepared_resource(+ {+ "root_uri": root,+ "temp_uri": root,+ "source_committed": True,+ "target_preexisting": True,+ "root_is_file": False,+ "context_update_plan": plan.to_dict(),+ },+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),+ resource_lock=lock,+ build_index=True,+ processing_mode="vectors_only",+ )++ processor._enqueue_index_actions.assert_awaited_once_with(+ plan.direct_index_actions, ctx=processor._enqueue_index_actions.await_args.kwargs["ctx"]+ )+ processor._vectorize_prepared_files.assert_not_awaited()+++async def test_direct_index_enqueue_failure_releases_lock():+ from openviking.server.identity import RequestContext, Role+ from openviking.storage.context_update_plan import ContextUpdatePlan, DirectIndexAction+ from openviking.utils.resource_processor import ResourceProcessor+ from openviking_cli.session.user_id import UserIdentifier++ root = "viking://resources/repo"+ lock = {"lease_ref": "plan"}+ viking_fs = SimpleNamespace(+ _async_agfs=SimpleNamespace(pathlock_release=AsyncMock()),+ )+ processor = ResourceProcessor(SimpleNamespace(get_embedder=lambda: None))+ processor._enqueue_index_actions = AsyncMock(side_effect=RuntimeError("queue unavailable"))+ plan = ContextUpdatePlan(+ root,+ "resource",+ direct_index_actions=(DirectIndexAction("delete", root + "/a.py", 2, "a-l2"),),+ )++ with pytest.raises(RuntimeError, match="queue unavailable"):+ with pytest.MonkeyPatch.context() as monkeypatch:+ monkeypatch.setattr(+ "openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs+ )+ await processor.finish_prepared_resource(+ {+ "root_uri": root,+ "temp_uri": root,+ "source_committed": True,+ "target_preexisting": True,+ "root_is_file": False,+ "context_update_plan": plan.to_dict(),+ },+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),+ resource_lock=lock,+ build_index=True,+ )++ viking_fs._async_agfs.pathlock_release.assert_awaited_once_with(lock)+++async def test_semantic_processor_runs_only_plan_execution_roots(monkeypatch):+ from openviking.storage.context_update_plan import (+ IndexSlot,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+ )+ from openviking.storage.queuefs.semantic_msg import SemanticMsg+ from openviking.storage.queuefs.semantic_processor import SemanticProcessor++ calls = []++ class Executor:+ stale = True+ root_write_result = None++ def __init__(self, **kwargs):+ calls.append(("init", kwargs["semantic_plan"]))++ async def run(self, uri):+ calls.append(("run", uri))++ def get_stats(self):+ return SimpleNamespace()++ root = "viking://resources/repo"+ plan = SemanticPlan(+ root,+ "resource",+ SemanticTreeSnapshot(+ (+ SemanticTreeEntry(+ "",+ "directory",+ "unchanged",+ "aggregate",+ index_slots=(IndexSlot(0, "root-l0", {"abstract": "root"}),),+ ),+ SemanticTreeEntry(+ "src",+ "directory",+ "unchanged",+ "aggregate",+ index_slots=(IndexSlot(0, "src-l0", {"abstract": "old"}),),+ ),+ SemanticTreeEntry(+ "src/a.py",+ "file",+ "modified",+ "generate",+ md5="new",+ index_slots=(+ IndexSlot(+ 2,+ "a-l2",+ {"abstract": "old a"},+ action="upsert",+ ),+ ),+ ),+ SemanticTreeEntry(+ "docs",+ "directory",+ "unchanged",+ "aggregate",+ index_slots=(IndexSlot(0, "docs-l0", {"abstract": "old"}),),+ ),+ SemanticTreeEntry(+ "docs/b.md",+ "file",+ "modified",+ "generate",+ md5="new",+ index_slots=(+ IndexSlot(+ 2,+ "b-l2",+ {"abstract": "old b"},+ action="upsert",+ ),+ ),+ ),+ )+ ),+ )+ fs = SimpleNamespace(exists=AsyncMock(return_value=True))+ monkeypatch.setattr("openviking.storage.queuefs.semantic_processor.get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor", Executor+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_processor.SemanticLockScope.resolve",+ AsyncMock(return_value=SimpleNamespace(lock=None, close=AsyncMock())),+ )+ processor = SemanticProcessor()+ processor._cleanup_local_artifact = AsyncMock()+ msg = SemanticMsg(+ uri=root,+ context_type="resource",+ account_id="acc",+ user_id="user",+ role="user",+ plan=plan,+ )++ await processor.on_dequeue(msg.to_dict())++ assert [value for kind, value in calls if kind == "run"] == [root]++[email protected]("cleanup_fails", [False, True])+async def test_process_resource_does_not_handoff_local_artifact(+ monkeypatch, tmp_path, cleanup_fails+):+ from openviking.parse.output import LocalParseOutputStore+ from openviking.server.identity import RequestContext, Role+ from openviking.storage.context_update_plan import ContextUpdatePlan+ from openviking.utils.resource_processor import ResourceProcessor+ from openviking_cli.session.user_id import UserIdentifier++ store = LocalParseOutputStore(str(tmp_path / "artifacts"))+ ref = await store.create_artifact()+ await store.write_bytes(ref, "repository/a.py", b"a")+ if cleanup_fails:+ store.cleanup = AsyncMock(side_effect=OSError("cleanup unavailable"))+ processor = ResourceProcessor(SimpleNamespace(get_embedder=lambda: None))+ processor._build_parse_output_store = lambda: store+ processor._get_media_processor = lambda: SimpleNamespace(+ process=AsyncMock(+ return_value=SimpleNamespace(+ temp_dir_path=ref.root,+ source_path="x",+ source_format="repository",+ meta={},+ warnings=[],+ artifact_ref=ref,+ ensure_artifact_ref=lambda: ref,+ )+ )+ )+ processor.tree_builder.finalize_from_temp = AsyncMock(+ return_value=SimpleNamespace(+ root=SimpleNamespace(+ uri="viking://resources/repo",+ temp_uri=ref.root + "/repository",+ ),+ _root_is_file=False,+ )+ )+ processor._commit_directory_artifact_with_plan = AsyncMock(+ return_value=ContextUpdatePlan("viking://resources/repo", "resource")+ )+ viking_fs = SimpleNamespace(+ exists=AsyncMock(return_value=False),+ _uri_to_path=lambda uri, ctx=None: uri,+ bind_request_context=lambda ctx: nullcontext(),+ _async_agfs=SimpleNamespace(pathlock_release=AsyncMock()),+ )+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)+ processor.acquire_resource_lock = AsyncMock(return_value={"lease_ref": "x"})++ result = await processor.process_resource(+ path="x",+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),+ to="viking://resources/repo",+ defer_post_processing=True,+ build_index=True,+ )++ assert result["_post_process"]["artifact_ref"] is None+ assert "file_md5s" not in result["_post_process"]+ assert "file_abstracts" not in result["_post_process"]+ assert "artifact_files" not in result["_post_process"]+ artifact_path = tmp_path.joinpath("artifacts", ref.root.split("/")[-1])+ assert artifact_path.exists() is cleanup_failsdiff --git a/tests/storage/test_embedding_msg.py b/tests/storage/test_embedding_msg.pyindex da2c9176f..a9f45b1c3 100644--- a/tests/storage/test_embedding_msg.py+++ b/tests/storage/test_embedding_msg.py@@ -3,7 +3,16 @@from uuid import uuid4-from openviking.storage.queuefs.embedding_msg import EmbeddingMsg+import pytest++from openviking.storage.index_action import FieldPatch, IndexAction+from openviking.storage.queuefs.embedding_msg import (+ DeletePayload,+ EmbeddingMsg,+ EmbedPayload,+ NoopPayload,+ UpdateFieldsPayload,+)from openviking.telemetry.request_wait_tracker import RequestWaitTracker@@ -27,3 +36,273 @@ def test_embedding_msg_roundtrip_preserves_id_for_request_wait_tracker():assert tracker.is_complete(telemetry_id)finally:tracker.cleanup(telemetry_id)+++def test_embedding_msg_roundtrip_preserves_queue_enqueue_time():+ msg = EmbeddingMsg(+ "hello",+ {"uri": "viking://resources/demo"},+ queue_enqueued_at=123.456,+ )++ assert EmbeddingMsg.from_dict(msg.to_dict()).queue_enqueued_at == 123.456+++def test_legacy_embedding_msg_defaults_to_embed_and_upsert():+ restored = EmbeddingMsg.from_dict(+ {+ "message": "hello",+ "context_data": {"uri": "viking://resources/demo"},+ }+ )++ assert restored.action is IndexAction.UPSERT+++def test_embedding_msg_accepts_noop_index_action_roundtrip():+ msg = EmbeddingMsg(None, {"uri": "viking://resources/demo"}, action=IndexAction.NONE)++ restored = EmbeddingMsg.from_json(msg.to_json())++ assert restored.action is IndexAction.NONE+ assert restored.message is None+++def test_embedding_update_fields_roundtrip():+ msg = EmbeddingMsg.for_update_fields(+ record_id="record-1",+ fields={+ "md5": "new-md5",+ "updated_at": "2026-09-13T00:00:00Z",+ "search_tags": ["scope=new"],+ },+ context_data={+ "uri": "viking://resources/demo/a.py",+ "account_id": "account-a",+ "level": 2,+ },+ field_modes={"search_tags": "append"},+ initial_fields={+ "uri": "viking://resources/demo/a.py",+ "account_id": "account-a",+ "level": 2,+ "vector": [0.1, 0.2],+ },+ )++ restored = EmbeddingMsg.from_json(msg.to_json())++ assert restored.action is IndexAction.UPDATE_FIELDS+ assert restored.record_ids == ["record-1"]+ assert restored.update_fields == {+ "md5": "new-md5",+ "updated_at": "2026-09-13T00:00:00Z",+ "search_tags": ["scope=new"],+ }+ assert restored.message is None+ assert restored.field_modes == {"search_tags": "append"}+ assert restored.initial_fields["vector"] == [0.1, 0.2]+++def test_embedding_delete_roundtrip():+ msg = EmbeddingMsg.for_delete(+ record_ids=["record-1", "record-2"],+ context_data={+ "uri": "viking://resources/demo",+ "account_id": "account-a",+ },+ )++ restored = EmbeddingMsg.from_json(msg.to_json())++ assert restored.action is IndexAction.DELETE+ assert restored.record_ids == ["record-1", "record-2"]+ assert restored.message is None+++def test_embedding_update_fields_rejects_vector_fields():+ with pytest.raises(ValueError, match="vector"):+ EmbeddingMsg.for_update_fields(+ record_id="record-1",+ fields={"vector": [0.1, 0.2]},+ context_data={+ "uri": "viking://resources/demo/a.py",+ "account_id": "account-a",+ "level": 2,+ },+ )+++def test_field_patch_roundtrip_and_resolution():+ patch = FieldPatch(+ values={"search_tags": ["scope=new"], "md5": "new-md5"},+ modes={"search_tags": "append"},+ seed_fields={+ "uri": "viking://resources/demo/a.py",+ "account_id": "account-a",+ "level": 2,+ "vector": [0.1, 0.2],+ },+ )++ restored = FieldPatch.from_dict(patch.to_dict())++ assert restored.resolve({"search_tags": ["env=old"], "md5": "old-md5"}) == {+ "search_tags": ["env=old", "scope=new"],+ "md5": "new-md5",+ }+ assert restored.seed_fields["vector"] == [0.1, 0.2]+++ ("message", "payload_type"),+ [+ (+ EmbeddingMsg.for_embed(+ message="body",+ context_data={"uri": "viking://resources/demo/a.py"},+ ),+ EmbedPayload,+ ),+ (+ EmbeddingMsg.for_update_fields(+ record_id="record-1",+ field_patch=FieldPatch(values={"md5": "new-md5"}),+ context_data={"uri": "viking://resources/demo/a.py"},+ ),+ UpdateFieldsPayload,+ ),+ (+ EmbeddingMsg.for_delete(+ record_ids=["record-1"],+ context_data={"uri": "viking://resources/demo/a.py"},+ ),+ DeletePayload,+ ),+ (EmbeddingMsg.noop(context_data={"uri": "viking://resources/demo"}), NoopPayload),+ ],+)+def test_embedding_msg_roundtrip_uses_action_specific_payload(message, payload_type):+ serialized = message.to_dict()++ assert "payload" in serialized+ assert "message" not in serialized+ restored = EmbeddingMsg.from_dict(serialized)+ assert isinstance(restored.payload, payload_type)+ assert restored.to_dict() == serialized+++def test_embed_payload_rejects_patch_for_upsert():+ with pytest.raises(ValueError, match="upsert.*field patch"):+ EmbeddingMsg.for_embed(+ message="body",+ context_data={"uri": "viking://resources/demo/a.py"},+ action=IndexAction.UPSERT,+ field_patch=FieldPatch(values={"search_tags": ["scope=new"]}),+ )+++def test_action_specific_payload_rejects_mismatched_action():+ with pytest.raises(ValueError, match="delete requires DeletePayload"):+ EmbeddingMsg(+ action=IndexAction.DELETE,+ payload=EmbedPayload(+ "body",+ {"uri": "viking://resources/demo/a.py"},+ ),+ )+++def test_payload_rejects_legacy_fields_in_same_message():+ with pytest.raises(ValueError, match="payload cannot be combined"):+ EmbeddingMsg.from_dict(+ {+ "action": "upsert",+ "payload": {+ "type": "embed",+ "message": "body",+ "context_data": {"uri": "viking://resources/demo/a.py"},+ },+ "record_ids": ["silently-ignored-before"],+ }+ )+++ "payload",+ [+ {+ "type": "delete",+ "record_ids": ["record-1"],+ "context_data": {},+ "message": "must-not-be-ignored",+ },+ {+ "type": "embed",+ "message": "body",+ "context_data": {},+ "record_ids": ["must-not-be-ignored"],+ },+ ],+)+def test_action_specific_payload_rejects_cross_action_fields(payload):+ with pytest.raises(ValueError, match="payload contains unsupported fields"):+ EmbeddingMsg.from_dict(+ {+ "action": "delete" if payload["type"] == "delete" else "merge",+ "payload": payload,+ }+ )+++def test_update_fields_factory_rejects_duplicate_patch_representations():+ with pytest.raises(ValueError, match="field_patch cannot be combined"):+ EmbeddingMsg.for_update_fields(+ record_id="record-1",+ context_data={"uri": "viking://resources/demo/a.py"},+ field_patch=FieldPatch({"md5": "new-md5"}),+ fields={"md5": "ignored-md5"},+ )+++def test_legacy_upsert_discards_historically_ignored_fields():+ restored = EmbeddingMsg.from_dict(+ {+ "action": "upsert",+ "message": "body",+ "context_data": {"uri": "viking://resources/demo/a.py"},+ "record_ids": ["ignored-record"],+ "update_fields": {"md5": "ignored-md5"},+ "initial_fields": {"vector": [0.1, 0.2]},+ }+ )++ assert restored.action is IndexAction.UPSERT+ assert restored.record_ids == []+ assert restored.field_patch is None+ assert set(restored.to_dict()) == {+ "id",+ "telemetry_id",+ "queue_enqueued_at",+ "action",+ "payload",+ }+++def test_legacy_update_fields_message_normalizes_to_field_patch_payload():+ restored = EmbeddingMsg.from_dict(+ {+ "message": None,+ "context_data": {"uri": "viking://resources/demo/a.py"},+ "action": "update_fields",+ "record_ids": ["record-1"],+ "update_fields": {"search_tags": ["scope=new"]},+ "field_modes": {"search_tags": "append"},+ "initial_fields": {"vector": [0.1, 0.2]},+ }+ )++ assert isinstance(restored.payload, UpdateFieldsPayload)+ assert restored.payload.field_patch.values == {"search_tags": ["scope=new"]}+ assert restored.payload.field_patch.modes == {"search_tags": "append"}+ assert restored.payload.field_patch.seed_fields == {"vector": [0.1, 0.2]}diff --git a/tests/storage/test_embedding_msg_converter_tenant.py b/tests/storage/test_embedding_msg_converter_tenant.pyindex a20595406..7a0a8a313 100644--- a/tests/storage/test_embedding_msg_converter_tenant.py+++ b/tests/storage/test_embedding_msg_converter_tenant.py@@ -6,6 +6,7 @@import pytestfrom openviking.core.context import Context, Vectorize+from openviking.storage.index_action import IndexActionfrom openviking.storage.queuefs.embedding_msg_converter import EmbeddingMsgConverterfrom openviking_cli.session.user_id import UserIdentifier@@ -45,6 +46,7 @@ def test_embedding_msg_converter_backfills_account_and_owner_fields(expected_owner_user_id(user) if callable(expected_owner_user_id) else expected_owner_user_id)assert msg.context_data["owner_user_id"] == expected_user+ assert msg.action is IndexAction.MERGEdef test_embedding_msg_converter_keeps_only_embedding_input():@@ -56,3 +58,4 @@ def test_embedding_msg_converter_keeps_only_embedding_input():assert msg is not Noneassert msg.message == "bounded embedding text"assert "content" not in msg.context_data+ assert msg.action is IndexAction.MERGEdiff --git a/tests/storage/test_ingest_ls_node_limit.py b/tests/storage/test_ingest_ls_node_limit.pyindex c688b4ad7..f5ca0d36c 100644--- a/tests/storage/test_ingest_ls_node_limit.py+++ b/tests/storage/test_ingest_ls_node_limit.py@@ -64,17 +64,22 @@ class _TruncatingVikingFS:passasync def sync_tree(- self, root_uri, target_uri, *, ctx=None, file_change_status=None,- lease_ref=None, delete_temp_after=False, is_changed=None,+ self,+ root_uri,+ target_uri,+ *,+ ctx=None,+ file_change_status=None,+ lease_ref=None,+ delete_temp_after=False,+ is_changed=None,):from openviking.storage.viking_fs import SyncDifffrom openviking_cli.utils.uri import VikingURIdiff = SyncDiff()# Mirror _SyncMixin.list_children: must pass node_limit=LS_ALL_NODES.- entries = await self.ls(- root_uri, show_all_hidden=True, node_limit=LS_ALL_NODES, ctx=ctx- )+ entries = await self.ls(root_uri, show_all_hidden=True, node_limit=LS_ALL_NODES, ctx=ctx)for entry in entries:name = entry.get("name", "")if not name or name in (".", "..") or name.startswith("."):@@ -113,9 +118,9 @@ async def test_sync_materializes_all_children_above_default_node_limit(monkeypat"sync was truncated at ls node_limit")assert len(diff.added_dirs) == n_children- assert all(- lim == LS_ALL_NODES for lim in fake.ls_node_limits- ), f"sync_tree must pass LS_ALL_NODES to ls, got node_limits={fake.ls_node_limits}"+ assert all(lim == LS_ALL_NODES for lim in fake.ls_node_limits), (+ f"sync_tree must pass LS_ALL_NODES to ls, got node_limits={fake.ls_node_limits}"+ )class _TruncatingLsFS:@@ -132,22 +137,22 @@ class _TruncatingLsFS:@pytest.mark.asyncio-async def test_semantic_dag_list_dir_enumerates_all_children(monkeypatch):- """The summary DAG dispatches one summary/recursion task per child it lists.+async def test_semantic_executor_list_dir_enumerates_all_children(monkeypatch):+ """The semantic tree dispatches one summary/recursion task per child it lists.If ``_list_dir`` is capped at node_limit, children beyond 1000 never get an``.abstract.md``/``.overview.md`` (and are never recursed into), so a largenamespace silently loses its L0/L1 layers."""- from openviking.storage.queuefs.semantic_dag import SemanticDagExecutor+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutordir_uri = "viking://resources/wixqa"n_children = 1500 # > the default node_limit of 1000fake = _TruncatingLsFS(dir_uri, n_children)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_dag.get_viking_fs",+ "openviking.storage.queuefs.semantic_executor.get_viking_fs",lambda: fake,)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=None,context_type="resource",max_concurrent_llm=1,@@ -156,6 +161,6 @@ async def test_semantic_dag_list_dir_enumerates_all_children(monkeypatch):children_dirs, file_paths = await executor._list_dir(dir_uri, from_hint="test")assert len(children_dirs) == n_children, (- f"only {len(children_dirs)}/{n_children} children listed — summary DAG "+ f"only {len(children_dirs)}/{n_children} children listed — semantic tree ""was truncated at ls node_limit")diff --git a/tests/storage/test_queue_manager.py b/tests/storage/test_queue_manager.pyindex a5cf69a08..b19fa6530 100644--- a/tests/storage/test_queue_manager.py+++ b/tests/storage/test_queue_manager.py@@ -54,7 +54,7 @@ async def test_skill_shutdown_releases_lock_after_embedding_worker_exits(transport, monkeypatch, concurrency):from openviking.service.task_tracker_concurrency import run_to_completion- from openviking.storage.queuefs.semantic_dag import DagStats+ from openviking.storage.queuefs.semantic_executor import SemanticTreeStatsfrom openviking.storage.queuefs.semantic_msg import SemanticMsgfrom openviking.storage.queuefs.semantic_processor import SemanticProcessorfrom openviking.telemetry import OperationTelemetry@@ -87,7 +87,7 @@ async def test_skill_shutdown_releases_lock_after_embedding_worker_exits(queued.set()def get_stats(self):- return DagStats()+ return SemanticTreeStats()async def write(data):async def finish():@@ -127,7 +127,7 @@ async def test_skill_shutdown_releases_lock_after_embedding_worker_exits(await self.process_dequeued(data)await self.ack(data)- monkeypatch.setattr("openviking.storage.queuefs.semantic_processor.SemanticDagExecutor", Dag)+ monkeypatch.setattr("openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor", Dag)monkeypatch.setattr(SemanticProcessor, "_resolve_skill_semantic_lock", AsyncMock(return_value=Lease()))diff --git a/tests/storage/test_resource_diff_snapshot.py b/tests/storage/test_resource_diff_snapshot.pynew file mode 100644index 000000000..7020700ac--- /dev/null+++ b/tests/storage/test_resource_diff_snapshot.py@@ -0,0 +1,404 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Tests for reading N/F/V snapshots and preparing artifact inventory."""++import asyncio+import json++import pytest++from openviking.parse.output import AgfsParseOutputStore, ParseArtifactRef+from openviking.storage.resource_diff import (+ build_rnfv_snapshot,+ prepare_artifact_inventory,+ read_target_file_snapshot,+)+from openviking.storage.resource_rnfv import FormalEntry, NewEntry+++class _Ctx:+ account_id = "acct"+++class _FakeVikingFS:+ """Minimal VikingFS supporting tree() plus the output-store surface."""++ def __init__(self, tree_entries, files=None):+ self._tree_entries = tree_entries+ self.files = files or {}+ self.tree_calls = []++ async def tree(+ self,+ uri,+ *,+ output="original",+ show_all_hidden=False,+ node_limit=1000,+ level_limit=3,+ ctx=None,+ ):+ self.tree_calls.append({"uri": uri, "node_limit": node_limit, "level_limit": level_limit})+ return list(self._tree_entries)++ # Output-store surface for ParseOutputStore-backed artifact snapshots.+ def create_temp_uri(self, ctx=None):+ return "viking://temp/n"++ async def ls(self, uri, ctx=None, **kwargs):+ prefix = f"{uri.rstrip('/')}/"+ seen = {}+ for stored in self.files:+ if stored.startswith(prefix):+ rest = stored[len(prefix) :]+ head = rest.split("/", 1)+ name = head[0]+ seen[name] = {"name": name, "uri": f"{prefix}{name}", "isDir": len(head) > 1}+ return list(seen.values())++ async def read(self, uri, **kwargs):+ return self.files[uri]++ async def write_file(self, uri, content, **kwargs):+ self.files[uri] = content.encode("utf-8") if isinstance(content, str) else content+++class _FakeVikingDB:+ def __init__(self, records):+ self._records = records+ self.requested = None+ self.inventory_output_fields = None++ async def get_l2_diff_records_by_uris(self, uris, *, ctx):+ self.requested = list(uris)+ return {u: self._records[u] for u in uris if u in self._records}++ async def get_l2_diff_records_under_uri(self, target_uri, *, ctx):+ self.requested = target_uri+ prefix = target_uri.rstrip("/") + "/"+ return {+ uri: value+ for uri, value in self._records.items()+ if uri == target_uri or uri.startswith(prefix)+ }++ async def get_incremental_inventory_under_uri(self, target_uri, *, ctx, output_fields=None):+ del ctx+ self.inventory_output_fields = list(output_fields or [])+ prefix = target_uri.rstrip("/") + "/"+ return {+ str(value.get("id") or f"id-{index}"): {+ key: item+ for key, item in {+ **value,+ "id": str(value.get("id") or f"id-{index}"),+ "uri": uri,+ "level": int(value.get("level", 2)),+ "md5": str(value.get("md5") or ""),+ }.items()+ if not output_fields or key in output_fields+ }+ for index, (uri, value) in enumerate(self._records.items())+ if uri == target_uri or uri.startswith(prefix)+ }+++class TestReadFormalTreeSnapshot:+ async def test_lists_business_files_with_unlimited_scan(self) -> None:+ vfs = _FakeVikingFS(+ [+ {"rel_path": "a.py", "isDir": False, "uri": "viking://resources/x/a.py"},+ {"rel_path": "sub", "isDir": True, "uri": "viking://resources/x/sub"},+ {"rel_path": "sub/b.py", "isDir": False, "uri": "viking://resources/x/sub/b.py"},+ ]+ )+ files, complete = await read_target_file_snapshot(vfs, "viking://resources/x", ctx=_Ctx())+ assert complete is True+ assert set(files) == {"a.py", "sub", "sub/b.py"}+ assert files["sub"].is_dir is True+ assert files["a.py"].is_dir is False+ # Must scan the whole tree, not the default bounded window.+ assert vfs.tree_calls[0]["node_limit"] is None+ assert vfs.tree_calls[0]["level_limit"] is None++ async def test_denied_entry_marks_snapshot_incomplete(self) -> None:+ vfs = _FakeVikingFS(+ [+ {"rel_path": "a.py", "isDir": False, "uri": "viking://resources/x/a.py"},+ {+ "rel_path": "secret",+ "isDir": True,+ "access": "denied",+ "uri": "viking://resources/x/secret",+ },+ ]+ )+ files, complete = await read_target_file_snapshot(vfs, "viking://resources/x", ctx=_Ctx())+ # A permission-hidden subtree means we cannot trust the tree for deletion.+ assert complete is False+ assert "secret" not in files++ async def test_control_sidecars_excluded(self) -> None:+ vfs = _FakeVikingFS(+ [+ {"rel_path": "a.py", "isDir": False, "uri": "viking://resources/x/a.py"},+ {+ "rel_path": ".abstract.md",+ "isDir": False,+ "uri": "viking://resources/x/.abstract.md",+ },+ {"rel_path": "_system", "isDir": True, "uri": "viking://resources/x/_system"},+ ]+ )+ files, complete = await read_target_file_snapshot(vfs, "viking://resources/x", ctx=_Ctx())+ assert set(files) == {"a.py"}+ assert complete is True+++async def test_prepare_artifact_inventory_rewrites_images_during_single_artifact_walk(tmp_path):+ from unittest.mock import AsyncMock++ from openviking.parse.output import LocalParseOutputStore+ from openviking.utils.content_hash import content_md5++ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))+ ref = await store.create_artifact(root_type="dir")+ await store.write_bytes(ref, "repository/docs/guide.md", b"\n")+ await store.write_bytes(ref, "repository/docs/diagram.png", b"png")+ await store.write_text(+ ref,+ "repository/.image_mappings.json",+ json.dumps({"docs/guide.md": {"diagram.png": "diagram.png"}}),+ )+ store.list = AsyncMock(wraps=store.list)++ inventory = await prepare_artifact_inventory(+ store,+ ref,+ doc_rel="repository",+ target_root_uri="viking://resources/repo",+ )++ assert set(inventory.entries) == {"docs", "docs/guide.md", "docs/diagram.png"}+ assert inventory.entries["docs/guide.md"].md5 == content_md5(+ b"\n"+ )+ assert inventory.artifact_paths["docs/guide.md"] == "repository/docs/guide.md"+ assert inventory.rewritten_paths == frozenset({"docs/guide.md"})+ assert [call.args[1] for call in store.list.await_args_list] == [+ "repository",+ "repository/docs",+ ]+++async def test_prepare_artifact_inventory_rewrites_images_in_agfs_artifact():+ from openviking.utils.content_hash import content_md5++ vfs = _FakeVikingFS(+ [],+ files={+ "viking://temp/n/repository/docs/guide.md": b"\n",+ "viking://temp/n/repository/docs/diagram.png": b"png",+ "viking://temp/n/repository/.image_mappings.json": json.dumps(+ {"docs/guide.md": {"diagram.png": "diagram.png"}}+ ).encode(),+ },+ )+ store = AgfsParseOutputStore(viking_fs=vfs)+ ref = ParseArtifactRef(backend="agfs", root="viking://temp/n", root_type="dir")++ inventory = await prepare_artifact_inventory(+ store,+ ref,+ doc_rel="repository",+ target_root_uri="viking://resources/repo",+ )++ expected = b"\n"+ assert vfs.files["viking://temp/n/repository/docs/guide.md"] == expected+ assert inventory.entries["docs/guide.md"].md5 == content_md5(expected)+++async def test_build_rnfv_snapshot_reuses_prepared_artifact_inventory(tmp_path):+ from unittest.mock import AsyncMock++ from openviking.parse.output import LocalParseOutputStore++ root = "viking://resources/x"+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))+ ref = await store.create_artifact(root_type="dir")+ await store.write_bytes(ref, "repository/a.py", b"a")+ store.list = AsyncMock(wraps=store.list)+ inventory = await prepare_artifact_inventory(store, ref, doc_rel="repository")+ store.list.reset_mock()++ snapshot = await build_rnfv_snapshot(+ viking_fs=_FakeVikingFS([]),+ vikingdb=_FakeVikingDB({}),+ store=store,+ artifact_ref=ref,+ target_uri=root,+ ctx=_Ctx(),+ doc_rel="repository",+ artifact_inventory=inventory,+ target_preexisting=False,+ )++ assert set(snapshot.new.entries) == {"a.py"}+ store.list.assert_not_awaited()+++async def test_build_rnfv_snapshot_skips_vector_inventory_when_vectorize_disabled(tmp_path):+ from unittest.mock import AsyncMock++ from openviking.parse.output import LocalParseOutputStore+ from openviking.storage.resource_rnfv import RequestIntent+ from openviking.utils.ingest_options import IngestOptions++ root = "viking://resources/x"+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))+ ref = await store.create_artifact(root_type="dir")+ await store.write_bytes(ref, "repository/a.py", b"new")+ inventory = await prepare_artifact_inventory(store, ref, doc_rel="repository")+ vikingdb = _FakeVikingDB({f"{root}/a.py": {"id": "stale-l2", "level": 2, "md5": "new"}})+ vikingdb.get_incremental_inventory_under_uri = AsyncMock(+ side_effect=AssertionError("build_index=false must not read vector inventory")+ )++ snapshot = await build_rnfv_snapshot(+ viking_fs=_FakeVikingFS([{"rel_path": "a.py", "isDir": False, "uri": f"{root}/a.py"}]),+ vikingdb=vikingdb,+ store=store,+ artifact_ref=ref,+ target_uri=root,+ ctx=_Ctx(),+ doc_rel="repository",+ request_intent=RequestIntent.from_ingest_options(+ target_uri=root,+ processing_mode="vectors_only",+ ingest_options=IngestOptions.from_search_tags(["team=search"]),+ vectorize=False,+ ),+ artifact_inventory=inventory,+ target_preexisting=True,+ )++ assert snapshot.vectors.records_by_id == {}+ assert snapshot.vectors.projected_fields == frozenset()+ vikingdb.get_incremental_inventory_under_uri.assert_not_awaited()+++async def test_build_rnfv_snapshot_starts_new_formal_and_vector_reads_concurrently(+ tmp_path, monkeypatch+):+ from openviking.parse.output import LocalParseOutputStore+ from openviking.storage.resource_diff import ArtifactInventory++ root = "viking://resources/x"+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))+ ref = await store.create_artifact(root_type="dir")+ started = set()+ release = asyncio.Event()++ async def read_new(*args, **kwargs):+ started.add("new")+ await release.wait()+ return ArtifactInventory(entries={"a.py": NewEntry(md5="new")}, artifact_paths={})++ async def read_formal(*args, **kwargs):+ started.add("formal")+ await release.wait()+ return {"a.py": FormalEntry()}, True++ async def read_vectors(*args, **kwargs):+ started.add("vectors")+ await release.wait()+ return {"record": {"id": "record", "uri": f"{root}/a.py", "level": 2, "md5": "old"}}++ monkeypatch.setattr("openviking.storage.resource_diff.prepare_artifact_inventory", read_new)+ monkeypatch.setattr("openviking.storage.resource_diff.read_target_file_snapshot", read_formal)+ monkeypatch.setattr(+ "openviking.storage.resource_diff._read_incremental_vector_inventory", read_vectors+ )++ task = asyncio.create_task(+ build_rnfv_snapshot(+ viking_fs=_FakeVikingFS([]),+ vikingdb=_FakeVikingDB({}),+ store=store,+ artifact_ref=ref,+ target_uri=root,+ ctx=_Ctx(),+ )+ )+ for _ in range(10):+ if started == {"new", "formal", "vectors"}:+ break+ await asyncio.sleep(0)+ assert started == {"new", "formal", "vectors"}+ release.set()+ snapshot = await task++ assert snapshot.new.entries["a.py"].md5 == "new"+ assert snapshot.formal.entries["a.py"].is_dir is False+ assert snapshot.vectors.records_by_id["record"].fields["md5"] == "old"+++async def test_build_rnfv_snapshot_cancels_sibling_reads_after_failure(tmp_path, monkeypatch):+ from openviking.parse.output import LocalParseOutputStore++ root = "viking://resources/x"+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))+ ref = await store.create_artifact(root_type="dir")+ started = set()+ cancelled = set()+ fail = asyncio.Event()++ async def read_new(*args, **kwargs):+ started.add("new")+ await fail.wait()+ raise RuntimeError("inventory failed")++ async def wait_for_cancel(name):+ started.add(name)+ try:+ await asyncio.Event().wait()+ finally:+ cancelled.add(name)++ monkeypatch.setattr("openviking.storage.resource_diff.prepare_artifact_inventory", read_new)+ monkeypatch.setattr(+ "openviking.storage.resource_diff.read_target_file_snapshot",+ lambda *args, **kwargs: wait_for_cancel("formal"),+ )+ monkeypatch.setattr(+ "openviking.storage.resource_diff._read_incremental_vector_inventory",+ lambda *args, **kwargs: wait_for_cancel("vectors"),+ )++ task = asyncio.create_task(+ build_rnfv_snapshot(+ viking_fs=_FakeVikingFS([]),+ vikingdb=_FakeVikingDB({}),+ store=store,+ artifact_ref=ref,+ target_uri=root,+ ctx=_Ctx(),+ )+ )+ for _ in range(10):+ if started == {"new", "formal", "vectors"}:+ break+ await asyncio.sleep(0)+ fail.set()++ with pytest.raises(RuntimeError, match="inventory failed"):+ await task+ assert cancelled == {"formal", "vectors"}diff --git a/tests/storage/test_resource_target.py b/tests/storage/test_resource_target.pynew file mode 100644index 000000000..bc856e969--- /dev/null+++ b/tests/storage/test_resource_target.py@@ -0,0 +1,109 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Tests for the AGFS resource target used by content tree actions."""++import pytest++from openviking.storage.resource_target import AgfsResourceTarget+++class _Ctx:+ account_id = "acct"+++class _FakeVikingFS:+ def __init__(self, existing=None):+ self.files = dict(existing or {})+ self.written = []+ self.removed = []++ async def write_file_bytes(self, uri, content, *, ctx=None, lease_ref=None):+ self.files[uri] = content+ self.written.append(uri)++ async def read_file_bytes(self, uri, *, ctx=None):+ return self.files[uri]++ async def rm(self, uri, *, recursive=False, ctx=None, lease_ref=None):+ self.removed.append(uri)+ self.files.pop(uri, None)++ async def remove_files(self, uri, *, recursive=False, ctx=None, lease_ref=None):+ self.removed.append((uri, recursive, lease_ref))+ self.files.pop(uri, None)+++_ROOT = "viking://resources/proj"+++def _target(vfs):+ return AgfsResourceTarget(viking_fs=vfs, root_uri=_ROOT, ctx=_Ctx())+++class TestAgfsResourceTarget:+ async def test_write_joins_rel_path_under_root(self) -> None:+ vfs = _FakeVikingFS()+ target = _target(vfs)++ await target.write_file("sub/a.py", b"print(1)")++ assert vfs.files[f"{_ROOT}/sub/a.py"] == b"print(1)"++ async def test_empty_rel_path_targets_root_file(self) -> None:+ vfs = _FakeVikingFS()+ target = _target(vfs)++ await target.write_file("", b"body")++ assert vfs.files[_ROOT] == b"body"++ async def test_write_preserves_artifact_bytes(self) -> None:+ vfs = _FakeVikingFS()+ target = _target(vfs)++ artifact_bytes = "你好".encode("gbk")+ await target.write_file("a.py", artifact_bytes)++ assert vfs.files[f"{_ROOT}/a.py"] == artifact_bytes++ async def test_write_rejects_path_escape(self) -> None:+ vfs = _FakeVikingFS()+ target = _target(vfs)++ with pytest.raises(ValueError):+ await target.write_file("../evil.py", b"x")++ async def test_delete_path_uses_exact_semantics_for_file(self) -> None:+ lease = {"lease_ref": "root-tree"}+ vfs = _FakeVikingFS(existing={f"{_ROOT}/gone.py": b"x"})+ target = AgfsResourceTarget(+ viking_fs=vfs,+ root_uri=_ROOT,+ ctx=_Ctx(),+ lease_ref=lease,+ )++ await target.delete_path("gone.py", is_dir=False)++ assert vfs.removed == [(f"{_ROOT}/gone.py", False, lease)]++ async def test_delete_path_uses_tree_semantics_for_directory(self) -> None:+ lease = {"lease_ref": "root-tree"}+ vfs = _FakeVikingFS()+ target = AgfsResourceTarget(+ viking_fs=vfs,+ root_uri=_ROOT,+ ctx=_Ctx(),+ lease_ref=lease,+ )++ await target.delete_path("old-dir", is_dir=True)++ assert vfs.removed == [(f"{_ROOT}/old-dir", True, lease)]++ async def test_read_file_returns_existing_bytes(self) -> None:+ vfs = _FakeVikingFS(existing={f"{_ROOT}/a.py": b"body"})+ target = _target(vfs)++ assert await target.read_file("a.py") == b"body"diff --git a/tests/storage/test_semantic_dag_incremental.py b/tests/storage/test_semantic_executor_incremental.pysimilarity index 53%rename from tests/storage/test_semantic_dag_incremental.pyrename to tests/storage/test_semantic_executor_incremental.pyindex 308f6c741..09a9c88a7 100644--- a/tests/storage/test_semantic_dag_incremental.py+++ b/tests/storage/test_semantic_executor_incremental.py@@ -16,7 +16,8 @@ from openviking.storage.abstract_overview import (render_abstract_overview,)from openviking.storage.errors import LockAcquisitionError-from openviking.storage.queuefs.semantic_dag import SemanticDagExecutor+from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor+from openviking.utils.content_hash import content_md5from openviking.utils.ingest_options import IngestOptionsfrom openviking_cli.session.user_id import UserIdentifier@@ -45,6 +46,10 @@ class _FakeVikingFS:async def read_file(self, path, ctx=None):return self._file_contents.get(self._norm(path), "")+ async def read_file_bytes(self, path, ctx=None):+ content = self._file_contents.get(self._norm(path), b"")+ return content if isinstance(content, bytes) else content.encode()+async def abstract(self, uri, ctx=None):return self._file_contents.get(self._norm(f"{uri}/.abstract.md"),@@ -75,9 +80,12 @@ class _FakeProcessor:self.sync_calls = []self.vectorized_files = []self.file_ingest_options = {}+ self.file_md5s = {}+ self.file_contents = {}self.directory_ingest_options = {}self.vectorized_dirs = []self.generated_overviews = []+ self.updated_file_vectors = []def _parse_overview_md(self, overview_content):results = {}@@ -88,8 +96,11 @@ class _FakeProcessor:results[m.group("name").strip()] = m.group("summary").strip()return results- async def _generate_single_file_summary(self, file_path, llm_sem=None, ctx=None):+ async def _generate_single_file_summary(+ self, file_path, llm_sem=None, ctx=None, file_content=None+ ):self.summarized_files.append(file_path)+ self.file_contents[("summary", file_path)] = file_contentreturn {"name": file_path.split("/")[-1], "summary": "summary"}async def _generate_overview(self, dir_uri, file_summaries, children_abstracts, **kwargs):@@ -126,10 +137,19 @@ class _FakeProcessor:use_summary=False,ingest_options=None,creator_acl_grant=None,+ file_md5=None,+ file_content=None,+ scalar_override=None,+ action="merge",):del creator_acl_grantself.vectorized_files.append(file_path)self.file_ingest_options[file_path] = ingest_options+ self.file_md5s[file_path] = file_md5+ self.file_contents[("vector", file_path)] = file_content+ self.file_contents[("action", file_path)] = action+ self.file_contents[("scalar_override", file_path)] = scalar_override+ return Trueasync def _vectorize_directory(self,@@ -140,11 +160,41 @@ class _FakeProcessor:ctx=None,ingest_options=None,creator_acl_grant=None,+ scalar_overrides=None,+ actions=None,+ include_abstract=True,+ include_overview=True,):del creator_acl_grantself.directory_ingest_options[uri] = ingest_optionsself.vectorized_dirs.append(uri)- return None+ self.file_contents[("actions", uri)] = actions+ self.file_contents[("scalar_overrides", uri)] = scalar_overrides+ return {+ level for level, included in ((0, include_abstract), (1, include_overview)) if included+ }++ async def _update_file_vector_fields(+ self,+ *,+ record_id,+ file_path,+ file_md5,+ file_content,+ ctx,+ scalar_fields=None,+ ):+ self.updated_file_vectors.append(+ {+ "record_id": record_id,+ "file_path": file_path,+ "file_md5": file_md5,+ "file_content": file_content,+ "ctx": ctx,+ "scalar_fields": scalar_fields,+ }+ )+ return Trueasync def _sync_topdown_recursive(self, root_uri, target_uri, ctx=None, file_change_status=None, lock=None@@ -198,15 +248,17 @@ async def test_direct_incremental_update_uses_changes_without_temp_sync(monkeypaif sidecar_state != "valid":filename = ".overview.md" if sidecar_state == "malformed_overview" else ".abstract.md"fake_fs._file_contents[f"{root_uri}/{filename}"] = "---\n"- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_dag.get_openviking_config",+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),)processor = _FakeProcessor(fake_fs)ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=2,@@ -214,6 +266,7 @@ async def test_direct_incremental_update_uses_changes_without_temp_sync(monkeypaincremental_update=True,target_uri=root_uri,changes={"modified": [f"{root_uri}/a.txt"]} if sidecar_state == "valid" else {},+ file_md5s={f"{root_uri}/a.txt": "md5-a-new"},)await executor.run(root_uri)@@ -227,6 +280,12 @@ async def test_direct_incremental_update_uses_changes_without_temp_sync(monkeypaassert processor.vectorized_files == expected_filesassert processor.vectorized_dirs == [root_uri]assert processor.sync_calls == []+ # Resource ingestion uses the manifest md5 whenever the file is re-vectorized;+ # a skipped file produces no vector write.+ if f"{root_uri}/a.txt" in expected_files:+ assert processor.file_md5s[f"{root_uri}/a.txt"] == "md5-a-new"+ else:+ assert f"{root_uri}/a.txt" not in processor.file_md5soverview = parse_abstract_overview(fake_fs._file_contents[f"{root_uri}/.overview.md"]).bodyexpected_a = "old-a" if sidecar_state == "malformed_abstract" else "summary"expected_b = "summary" if sidecar_state == "malformed_overview" else "old-b"@@ -235,6 +294,296 @@ async def test_direct_incremental_update_uses_changes_without_temp_sync(monkeypaassert parse_abstract_overview(fake_fs._file_contents[f"{root_uri}/.abstract.md"]).body.strip()+async def test_modified_file_with_same_abstract_still_propagates(monkeypatch):+ root_uri = "viking://resources/root"+ child_uri = f"{root_uri}/nested"+ file_uri = f"{child_uri}/a.txt"+ old_root_overview = render_abstract_overview(+ ContextLevel.OVERVIEW, root_uri, "FILES:\n- nested/: old-root-child"+ )+ old_child_overview = render_abstract_overview(+ ContextLevel.OVERVIEW, child_uri, "FILES:\n- a.txt: summary"+ )+ fake_fs = _FakeVikingFS(+ tree={+ root_uri: [{"name": "nested", "isDir": True}],+ child_uri: [{"name": "a.txt", "isDir": False}],+ },+ file_contents={+ file_uri: "changed body",+ f"{root_uri}/.overview.md": old_root_overview,+ f"{root_uri}/.abstract.md": render_abstract_overview(+ ContextLevel.ABSTRACT, root_uri, "old-root-child"+ ),+ f"{child_uri}/.overview.md": old_child_overview,+ f"{child_uri}/.abstract.md": render_abstract_overview(+ ContextLevel.ABSTRACT, child_uri, "old-child"+ ),+ },+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )++ processor = _FakeProcessor(fake_fs)+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=2,+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),+ incremental_update=True,+ target_uri=root_uri,+ changes={"modified": [file_uri]},+ file_abstracts={file_uri: "summary"},+ file_md5s={file_uri: "new-md5"},+ )++ await executor.run(root_uri)++ assert processor.summarized_files == [file_uri]+ assert processor.vectorized_files == [file_uri]+ assert processor.file_md5s[file_uri] == "new-md5"+ assert processor.generated_overviews == [child_uri, root_uri]+ assert processor.vectorized_dirs == [child_uri, root_uri]+ assert fake_fs._file_contents[f"{child_uri}/.overview.md"] != old_child_overview+ assert fake_fs._file_contents[f"{root_uri}/.overview.md"] != old_root_overview+++async def test_modified_file_with_different_abstract_still_propagates(monkeypatch):+ root_uri = "viking://resources/root"+ file_uri = f"{root_uri}/a.txt"+ fake_fs = _FakeVikingFS(+ tree={root_uri: [{"name": "a.txt", "isDir": False}]},+ file_contents={+ file_uri: "changed body",+ f"{root_uri}/.overview.md": render_abstract_overview(+ ContextLevel.OVERVIEW, root_uri, "FILES:\n- a.txt: old-summary"+ ),+ f"{root_uri}/.abstract.md": render_abstract_overview(+ ContextLevel.ABSTRACT, root_uri, "old-abstract"+ ),+ },+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )++ processor = _FakeProcessor(fake_fs)+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=2,+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),+ incremental_update=True,+ target_uri=root_uri,+ changes={"modified": [file_uri]},+ file_abstracts={file_uri: "old-summary"},+ )++ await executor.run(root_uri)++ assert processor.generated_overviews == [root_uri]+ assert processor.vectorized_dirs == [root_uri]+++async def test_added_file_with_matching_stale_abstract_still_propagates(monkeypatch):+ root_uri = "viking://resources/root"+ child_uri = f"{root_uri}/nested"+ file_uri = f"{child_uri}/a.txt"+ fake_fs = _FakeVikingFS(+ tree={+ root_uri: [{"name": "nested", "isDir": True}],+ child_uri: [{"name": "a.txt", "isDir": False}],+ },+ file_contents={file_uri: "new body"},+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )++ processor = _FakeProcessor(fake_fs)+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=2,+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),+ incremental_update=True,+ target_uri=root_uri,+ changes={"added": [file_uri]},+ file_abstracts={file_uri: "summary"},+ )++ await executor.run(root_uri)++ assert processor.generated_overviews == [child_uri, root_uri]+ assert processor.vectorized_dirs == [child_uri, root_uri]+++async def test_deleted_last_nested_file_still_refreshes_existing_ancestor(monkeypatch):+ root_uri = "viking://resources/root"+ deleted_uri = f"{root_uri}/removed/a.txt"+ fake_fs = _FakeVikingFS(+ tree={root_uri: []},+ file_contents={+ f"{root_uri}/.overview.md": render_abstract_overview(+ ContextLevel.OVERVIEW, root_uri, "FILES:\n- removed/: old-child"+ ),+ f"{root_uri}/.abstract.md": render_abstract_overview(+ ContextLevel.ABSTRACT, root_uri, "old-root"+ ),+ },+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )++ processor = _FakeProcessor(fake_fs)+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=2,+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),+ incremental_update=True,+ target_uri=root_uri,+ changes={"deleted": [deleted_uri]},+ )++ await executor.run(root_uri)++ assert processor.generated_overviews == [root_uri]+ assert processor.vectorized_dirs == [root_uri]+++async def test_artifact_files_drive_tree_structure_when_target_listing_is_empty(+ tmp_path, monkeypatch+):+ root_uri = "viking://resources/root"+ # artifact_files supplies the committed tree structure so the executor does not+ # depend on a target ls; file bytes are always read from the (already+ # applied) target via viking_fs.+ fake_fs = _FakeVikingFS(+ tree={},+ file_contents={+ f"{root_uri}/a.txt": b"alpha",+ f"{root_uri}/src/b.txt": b"beta",+ },+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )++ processor = _FakeProcessor(fake_fs)+ ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=2,+ ctx=ctx,+ artifact_files=["a.txt", "src/b.txt"],+ )++ await executor.run(root_uri)++ assert sorted(processor.summarized_files) == [+ f"{root_uri}/a.txt",+ f"{root_uri}/src/b.txt",+ ]+ assert sorted(processor.vectorized_files) == [+ f"{root_uri}/a.txt",+ f"{root_uri}/src/b.txt",+ ]+ assert processor.file_contents[("summary", f"{root_uri}/a.txt")] == b"alpha"+ assert processor.file_contents[("vector", f"{root_uri}/src/b.txt")] == b"beta"+++async def test_direct_incremental_reuses_vector_abstract_without_overview(monkeypatch):+ root_uri = "viking://resources/root"+ file_uri = f"{root_uri}/a.txt"+ fake_fs = _FakeVikingFS(+ tree={root_uri: [{"name": "a.txt", "isDir": False}]},+ file_contents={},+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )++ processor = _FakeProcessor(fake_fs)+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=2,+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),+ incremental_update=True,+ target_uri=root_uri,+ changes={},+ artifact_files=["a.txt"],+ file_abstracts={file_uri: "summary from vector"},+ )++ await executor.run(root_uri)++ assert processor.summarized_files == []+ assert processor.vectorized_files == []+++async def test_missing_target_file_fails_instead_of_using_empty_summary(tmp_path, monkeypatch):+ root_uri = "viking://resources/root"++ class _MissingFileVikingFS(_FakeVikingFS):+ async def read_file_bytes(self, path, ctx=None):+ raise FileNotFoundError(path)++ fake_fs = _MissingFileVikingFS(tree={}, file_contents={})+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )+ executor = SemanticTreeExecutor(+ processor=_FakeProcessor(fake_fs),+ context_type="resource",+ max_concurrent_llm=2,+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),+ artifact_files=["a.txt"],+ )++ with pytest.raises(FileNotFoundError):+ await executor.run(root_uri)++@pytest.mark.asyncioasync def test_content_write_tags_apply_only_to_changed_file(monkeypatch):root_uri = "viking://resources/root"@@ -258,16 +607,18 @@ async def test_content_write_tags_apply_only_to_changed_file(monkeypatch):f"{root_uri}/.abstract.md": "old-abstract",},)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_dag.get_openviking_config",+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),)processor = _FakeProcessor(fake_fs)ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)tag_options = IngestOptions(search_tags=["team=search"])- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=2,@@ -285,6 +636,126 @@ async def test_content_write_tags_apply_only_to_changed_file(monkeypatch):assert processor.directory_ingest_options[root_uri] == IngestOptions()+async def test_content_write_with_same_abstract_updates_file_and_parent(monkeypatch):+ root_uri = "viking://resources/root"+ changed_uri = f"{root_uri}/a.txt"+ old_overview = render_abstract_overview(+ ContextLevel.OVERVIEW, root_uri, "FILES:\n- a.txt: summary"+ )+ fake_fs = _FakeVikingFS(+ tree={root_uri: [{"name": "a.txt", "isDir": False}]},+ file_contents={+ changed_uri: "new content",+ f"{root_uri}/.overview.md": old_overview,+ f"{root_uri}/.abstract.md": render_abstract_overview(+ ContextLevel.ABSTRACT, root_uri, "old-abstract"+ ),+ },+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )++ processor = _FakeProcessor(fake_fs)+ tag_options = IngestOptions(search_tags=["team=search"])+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=2,+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),+ incremental_update=True,+ target_uri=root_uri,+ changes={"modified": [changed_uri]},+ file_abstracts={changed_uri: "summary"},+ ingest_options=tag_options,+ generation_trigger="content_write",+ )++ await executor.run(root_uri)++ assert processor.file_ingest_options == {changed_uri: tag_options}+ assert processor.vectorized_files == [changed_uri]+ assert processor.generated_overviews == [root_uri]+ assert processor.vectorized_dirs == [root_uri]+ assert fake_fs._file_contents[f"{root_uri}/.overview.md"] != old_overview+++async def test_content_write_keeps_matching_upstream_md5(monkeypatch):+ root_uri = "viking://resources/root"+ changed_uri = f"{root_uri}/a.txt"+ latest_content = "newer content"+ fake_fs = _FakeVikingFS(+ tree={root_uri: [{"name": "a.txt", "isDir": False}]},+ file_contents={changed_uri: latest_content},+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )++ processor = _FakeProcessor(fake_fs)+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=2,+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),+ incremental_update=True,+ target_uri=root_uri,+ changes={"modified": [changed_uri]},+ file_md5s={changed_uri: content_md5(latest_content.encode())},+ generation_trigger="content_write",+ )++ await executor.run(root_uri)++ assert processor.file_contents[("vector", changed_uri)] == latest_content.encode()+ assert processor.file_md5s[changed_uri] == content_md5(latest_content.encode())+++async def test_semantic_generate_hashes_read_content_when_manifest_md5_is_missing(monkeypatch):+ root_uri = "viking://resources/root"+ changed_uri = f"{root_uri}/a.txt"+ content = b"content read by semantic worker"+ fake_fs = _FakeVikingFS(+ tree={root_uri: [{"name": "a.txt", "isDir": False}]},+ file_contents={changed_uri: content},+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),+ )++ processor = _FakeProcessor(fake_fs)+ executor = SemanticTreeExecutor(+ processor=processor,+ context_type="resource",+ max_concurrent_llm=2,+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),+ incremental_update=True,+ target_uri=root_uri,+ changes={"modified": [changed_uri]},+ generation_trigger="resource_ingest",+ )++ await executor.run(root_uri)++ assert processor.file_contents[("vector", changed_uri)] == content+ assert processor.file_md5s[changed_uri] == content_md5(content)++@pytest.mark.asyncioasync def test_pending_refresh_rebuilds_every_sampled_file_summary(monkeypatch):root_uri = "viking://resources/wide"@@ -308,15 +779,17 @@ async def test_pending_refresh_rebuilds_every_sampled_file_summary(monkeypatch):),},)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_dag.get_openviking_config",+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=4)),)processor = _FakeProcessor(fake_fs)ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=2,@@ -325,6 +798,7 @@ async def test_pending_refresh_rebuilds_every_sampled_file_summary(monkeypatch):target_uri=root_uri,recursive=False,changes={"modified": [changed_path]},+ file_abstracts={changed_path: "summary"},)await executor.run(root_uri)@@ -343,9 +817,11 @@ async def test_directory_vectorization_retries_after_matching_sidecar_write(monktree={root_uri: [{"name": "a.txt", "isDir": False}]},file_contents={file_path: "new content"},)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_dag.get_openviking_config",+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),)@@ -355,7 +831,7 @@ async def test_directory_vectorization_retries_after_matching_sidecar_write(monkctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)def make_executor():- return SemanticDagExecutor(+ return SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=2,@@ -409,9 +885,11 @@ async def test_content_copy_rebuilds_target_overview_from_target_l2_summaries(mo),},)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_dag.get_openviking_config",+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),)processor = _FakeProcessor(@@ -421,7 +899,7 @@ async def test_content_copy_rebuilds_target_overview_from_target_l2_summaries(mof"{root_uri}/keep.txt": "kept target L2 summary",},)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=2,@@ -456,16 +934,18 @@ async def test_content_copy_samples_before_loading_summaries(monkeypatch):tree={root_uri: [{"name": path.rsplit("/", 1)[-1], "isDir": False} for path in file_paths]},file_contents={path: path for path in file_paths},)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_dag.get_openviking_config",+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=2)),)processor = _FakeProcessor(fake_fs,transfer_summaries={path: f"summary-{path.rsplit('/', 1)[-1]}" for path in file_paths},)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=2,@@ -495,16 +975,18 @@ async def test_content_copy_does_not_backfill_missing_sample_summary(monkeypatchtree={root_uri: [{"name": path.rsplit("/", 1)[-1], "isDir": False} for path in file_paths]},file_contents={path: path for path in file_paths},)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_dag.get_openviking_config",+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=2)),)processor = _FakeProcessor(fake_fs,transfer_summaries={path: f"summary-{path.rsplit('/', 1)[-1]}" for path in file_paths[1:]},)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=2,@@ -540,13 +1022,15 @@ async def test_content_copy_with_no_ready_summaries_preserves_existing_sidecars(f"{root_uri}/.abstract.md": old_abstract,},)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_dag.get_openviking_config",+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),)processor = _FakeProcessor(fake_fs)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=2,@@ -578,13 +1062,15 @@ async def test_content_copy_rebuilds_semantics_when_move_leaves_source_directoryf"{root_uri}/.abstract.md": old_abstract,},)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_dag.get_openviking_config",+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),)processor = _FakeProcessor(fake_fs)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=2,@@ -614,9 +1100,11 @@ async def test_content_copy_propagates_vector_summary_read_failure(monkeypatch):tree={root_uri: [{"name": "copied.txt", "isDir": False}]},file_contents={file_path: "copied"},)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_dag.get_openviking_config",+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),)processor = _FakeProcessor(fake_fs)@@ -625,7 +1113,7 @@ async def test_content_copy_propagates_vector_summary_read_failure(monkeypatch):raise RuntimeError("vector backend unavailable")processor._load_transfer_file_summaries = fail_summary_read- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=2,diff --git a/tests/storage/test_semantic_dag_media_overview.py b/tests/storage/test_semantic_executor_media_overview.pysimilarity index 88%rename from tests/storage/test_semantic_dag_media_overview.pyrename to tests/storage/test_semantic_executor_media_overview.pyindex 0f6521660..88fa7b0ee 100644--- a/tests/storage/test_semantic_dag_media_overview.py+++ b/tests/storage/test_semantic_executor_media_overview.py@@ -5,9 +5,9 @@ from types import SimpleNamespacefrom unittest.mock import patchfrom openviking.server.identity import RequestContext, Role-from openviking.storage.queuefs.semantic_dag import (+from openviking.storage.queuefs.semantic_executor import (DirNode,- SemanticDagExecutor,+ SemanticTreeExecutor,)from openviking_cli.session.user_id import UserIdentifier@@ -18,10 +18,10 @@ def _executor():role=Role.USER,)with patch(- "openviking.storage.queuefs.semantic_dag.get_viking_fs",+ "openviking.storage.queuefs.semantic_executor.get_viking_fs",return_value=SimpleNamespace(),):- return SemanticDagExecutor(SimpleNamespace(), "resource", 2, ctx)+ return SemanticTreeExecutor(SimpleNamespace(), "resource", 2, ctx)def _node(diff --git a/tests/storage/test_semantic_dag_skip_files.py b/tests/storage/test_semantic_executor_skip_files.pysimilarity index 88%rename from tests/storage/test_semantic_dag_skip_files.pyrename to tests/storage/test_semantic_executor_skip_files.pyindex 9b6092063..8632d4d37 100644--- a/tests/storage/test_semantic_dag_skip_files.py+++ b/tests/storage/test_semantic_executor_skip_files.py@@ -6,7 +6,7 @@ from types import SimpleNamespaceimport pytestfrom openviking.server.identity import RequestContext, Role-from openviking.storage.queuefs.semantic_dag import SemanticDagExecutor+from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutorfrom openviking_cli.session.user_id import UserIdentifier@@ -89,6 +89,7 @@ class _FakeProcessor:use_summary=False,ingest_options=None,creator_acl_grant=None,+ file_md5=None,):self.vectorized_files.append(file_path)@@ -105,11 +106,13 @@ async def test_messages_jsonl_excluded_from_summary(monkeypatch):],}fake_fs = _FakeVikingFS(tree)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )processor = _FakeProcessor()ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="session",max_concurrent_llm=2,@@ -137,11 +140,13 @@ async def test_messages_jsonl_excluded_in_subdirectory(monkeypatch):],}fake_fs = _FakeVikingFS(tree)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )processor = _FakeProcessor()ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="session",max_concurrent_llm=2,@@ -165,15 +170,17 @@ async def test_unlistable_semantic_root_is_not_materialized_as_directory(fail_sidecar_write=fail_sidecar_write,list_error=list_error,)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_dag.get_openviking_config",+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),)processor = _FakeProcessor()ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="memory",max_concurrent_llm=2,@@ -192,15 +199,17 @@ async def test_unlistable_semantic_root_is_not_materialized_as_directory(async def test_empty_semantic_directory_still_receives_sidecars(monkeypatch):root_uri = "viking://user/user1/memories/empty"fake_fs = _FakeVikingFS({root_uri: []})- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_dag.get_openviking_config",+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),)processor = _FakeProcessor()ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="memory",max_concurrent_llm=2,diff --git a/tests/storage/test_semantic_dag_stats.py b/tests/storage/test_semantic_executor_stats.pysimilarity index 89%rename from tests/storage/test_semantic_dag_stats.pyrename to tests/storage/test_semantic_executor_stats.pyindex 4be498fb0..9f0cee542 100644--- a/tests/storage/test_semantic_dag_stats.py+++ b/tests/storage/test_semantic_executor_stats.py@@ -23,11 +23,11 @@ from openviking.storage.abstract_overview import ()from openviking.storage.errors import LockAcquisitionErrorfrom openviking.storage.queuefs.named_queue import NamedQueue-from openviking.storage.queuefs.semantic_dag import (- DagStats,- DagWork,- SemanticDagExecutor,- SemanticNodeScheduler,+from openviking.storage.queuefs.semantic_executor import (+ SemanticTreeExecutor,+ SemanticTreeScheduler,+ SemanticTreeStats,+ SemanticTreeWork,)from openviking.telemetry import (OperationTelemetry,@@ -114,6 +114,7 @@ class _FakeProcessor:use_summary=False,ingest_options=None,creator_acl_grant=None,+ file_md5=None,):if self.verify_streaming:assert summary_dict["content"]@@ -165,7 +166,7 @@ class _ScheduledExecutor:def _patch_semantic_config(monkeypatch, *, overview_sample_limit=32):monkeypatch.setattr(- "openviking.storage.queuefs.semantic_dag.get_openviking_config",+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=overview_sample_limit)),@@ -173,7 +174,7 @@ def _patch_semantic_config(monkeypatch, *, overview_sample_limit=32):@pytest.mark.asyncio-async def test_semantic_dag_stats_collects_nodes(monkeypatch):+async def test_semantic_executor_stats_collects_nodes(monkeypatch):root_uri = "viking://resources/root"tree = {root_uri: [@@ -186,12 +187,14 @@ async def test_semantic_dag_stats_collects_nodes(monkeypatch):],}fake_fs = _FakeVikingFS(tree)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )_patch_semantic_config(monkeypatch)processor = _FakeProcessor(verify_streaming=True)ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=2,@@ -201,7 +204,7 @@ async def test_semantic_dag_stats_collects_nodes(monkeypatch):await asyncio.sleep(0)stats = executor.get_stats()- assert isinstance(stats, DagStats)+ assert isinstance(stats, SemanticTreeStats)assert stats.total_nodes == 5 # 2 dirs + 3 filesassert stats.pending_nodes == 0assert stats.done_nodes == 5@@ -217,18 +220,20 @@ async def test_semantic_dag_stats_collects_nodes(monkeypatch):@pytest.mark.asyncio-async def test_semantic_dag_bounds_active_node_work(monkeypatch):+async def test_semantic_executor_bounds_active_node_work(monkeypatch):root_uri = "viking://resources/root"tree = {root_uri: [{"name": f"file-{idx}.txt", "isDir": False} for idx in range(40)],}fake_fs = _FakeVikingFS(tree)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )_patch_semantic_config(monkeypatch)processor = _TrackingProcessor()ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=3,@@ -259,12 +264,14 @@ async def test_incremental_wide_directory_samples_before_summary_work(monkeypatcroot_uri: [{"name": f"file-{idx:03}.txt", "isDir": False} for idx in range(40)],}fake_fs = _FakeVikingFS(tree)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )_patch_semantic_config(monkeypatch, overview_sample_limit=4)processor = _FakeProcessor()ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=2,@@ -302,12 +309,14 @@ async def test_non_recursive_memory_samples_files_and_reads_child_abstracts(monkchild_d: [{"name": "nested.md", "isDir": False}],}fake_fs = _FakeVikingFS(tree, abstracts={child_a: "child a abstract"})- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )_patch_semantic_config(monkeypatch, overview_sample_limit=3)processor = _FakeProcessor()ctx = RequestContext(user=UserIdentifier("acc1", "alice"), role=Role.USER)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="memory",max_concurrent_llm=2,@@ -350,7 +359,9 @@ async def test_busy_parent_snapshot_preserves_changed_file_work(monkeypatch):raise FileNotFoundError(uri)monkeypatch.setattr(fake_fs, "read_file", read_file, raising=False)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )_patch_semantic_config(monkeypatch, overview_sample_limit=4)monkeypatch.setattr(fake_fs, "pathlock_acquire_exact_batch", AsyncMock(side_effect=LockAcquisitionError("busy"))@@ -358,7 +369,7 @@ async def test_busy_parent_snapshot_preserves_changed_file_work(monkeypatch):processor = _FakeProcessor()ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=2,@@ -378,7 +389,7 @@ async def test_busy_parent_snapshot_preserves_changed_file_work(monkeypatch):@pytest.mark.asyncio-async def test_semantic_dag_shares_node_scheduler_across_roots(monkeypatch):+async def test_semantic_executor_shares_node_scheduler_across_roots(monkeypatch):root_a = "viking://resources/root-a"root_b = "viking://resources/root-b"tree = {@@ -386,7 +397,9 @@ async def test_semantic_dag_shares_node_scheduler_across_roots(monkeypatch):root_b: [{"name": f"b-{idx}.txt", "isDir": False} for idx in range(20)],}fake_fs = _FakeVikingFS(tree)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )_patch_semantic_config(monkeypatch)processor = _TrackingProcessor()@@ -397,7 +410,7 @@ async def test_semantic_dag_shares_node_scheduler_across_roots(monkeypatch):bind_task_context("task-a", "acc1", "user1"),bind_telemetry(telemetry_a),):- executor_a = SemanticDagExecutor(+ executor_a = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=1,@@ -407,7 +420,7 @@ async def test_semantic_dag_shares_node_scheduler_across_roots(monkeypatch):bind_task_context("task-b", "acc1", "user1"),bind_telemetry(telemetry_b),):- executor_b = SemanticDagExecutor(+ executor_b = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=1,@@ -461,9 +474,9 @@ async def test_task_work_rejection_does_not_stop_shared_semantic_worker():rejected = _ScheduledExecutor(rejected_work)unrelated = _ScheduledExecutor(unrelated_work)- scheduler = SemanticNodeScheduler(max_workers=1)- scheduler.submit(rejected, DagWork(kind="vectorize", dir_uri="a"))- scheduler.submit(unrelated, DagWork(kind="vectorize", dir_uri="b"))+ scheduler = SemanticTreeScheduler(max_workers=1)+ scheduler.submit(rejected, SemanticTreeWork(kind="vectorize", dir_uri="a"))+ scheduler.submit(unrelated, SemanticTreeWork(kind="vectorize", dir_uri="b"))await asyncio.wait_for(unrelated_ran.wait(), timeout=0.5)await asyncio.wait_for(scheduler._queue.join(), timeout=0.5)@@ -476,7 +489,7 @@ async def test_task_work_rejection_does_not_stop_shared_semantic_worker():@pytest.mark.asyncio-async def test_semantic_dag_skip_vectorization_does_not_schedule_tasks(monkeypatch):+async def test_semantic_executor_skip_vectorization_does_not_schedule_tasks(monkeypatch):root_uri = "viking://resources/root"tree = {root_uri: [@@ -488,12 +501,14 @@ async def test_semantic_dag_skip_vectorization_does_not_schedule_tasks(monkeypat],}fake_fs = _FakeVikingFS(tree)- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)+ monkeypatch.setattr(+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs+ )_patch_semantic_config(monkeypatch)processor = _FakeProcessor()ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)- executor = SemanticDagExecutor(+ executor = SemanticTreeExecutor(processor=processor,context_type="resource",max_concurrent_llm=2,diff --git a/tests/storage/test_semantic_msg_ingest_options.py b/tests/storage/test_semantic_msg_ingest_options.pyindex 53d1922dd..a9a31f89a 100644--- a/tests/storage/test_semantic_msg_ingest_options.py+++ b/tests/storage/test_semantic_msg_ingest_options.py@@ -74,3 +74,52 @@ def test_semantic_msg_round_trips_hierarchical_aggregation_policy():assert restored.use_hierarchical_aggregation is Trueassert restored.propagate_to_parent is False+++def test_semantic_msg_roundtrip_preserves_file_md5s():+ msg = SemanticMsg(+ uri="viking://resources/x",+ context_type="resource",+ file_md5s={"viking://resources/x/a.py": "md5a"},+ )++ assert SemanticMsg.from_dict(msg.to_dict()).file_md5s == {"viking://resources/x/a.py": "md5a"}+++def test_semantic_msg_defaults_file_md5s_to_empty():+ msg = SemanticMsg(uri="viking://resources/x", context_type="resource")++ assert msg.file_md5s == {}+ assert SemanticMsg.from_dict(msg.to_dict()).file_md5s == {}+++def test_semantic_msg_roundtrip_preserves_queue_enqueue_time():+ msg = SemanticMsg(+ uri="viking://resources/x",+ context_type="resource",+ queue_enqueued_at=123.456,+ )++ assert SemanticMsg.from_dict(msg.to_dict()).queue_enqueued_at == 123.456+++def test_semantic_msg_roundtrip_preserves_local_artifact_snapshot():+ msg = SemanticMsg(+ uri="viking://resources/x",+ context_type="resource",+ artifact_ref={+ "backend": "local",+ "root": "/tmp/artifact-1",+ "resource_rel": "repository",+ "root_type": "dir",+ },+ artifact_files=["a.py", "src/b.py"],+ file_abstracts={"viking://resources/x/a.py": "summary a"},+ )++ restored = SemanticMsg.from_json(msg.to_json())++ assert restored.artifact_ref == msg.artifact_ref+ assert restored.artifact_ref["resource_rel"] == "repository"+ assert restored.artifact_files == ["a.py", "src/b.py"]+ assert restored.file_abstracts == {"viking://resources/x/a.py": "summary a"}diff --git a/tests/storage/test_semantic_processor_target_preexisting.py b/tests/storage/test_semantic_processor_target_preexisting.pyindex 8df69a6bc..caf822e0c 100644--- a/tests/storage/test_semantic_processor_target_preexisting.py+++ b/tests/storage/test_semantic_processor_target_preexisting.py@@ -61,42 +61,59 @@ class _SyncWrapperVikingFS:return {"matches": []}-class _FakeDagExecutor:+class _FakeTreeExecutor:calls = []runs = []def __init__(self, **kwargs):self.kwargs = kwargsself.stale = False- _FakeDagExecutor.calls.append(kwargs)+ _FakeTreeExecutor.calls.append(kwargs)async def run(self, root_uri):self.root_uri = root_uri- _FakeDagExecutor.runs.append(root_uri)+ _FakeTreeExecutor.runs.append(root_uri)def get_stats(self):- from openviking.storage.queuefs.semantic_dag import DagStats+ from openviking.storage.queuefs.semantic_executor import SemanticTreeStats- return DagStats()+ return SemanticTreeStats()@pytest.mark.asyncio-async def test_target_source_syncs_before_semantic_dag(monkeypatch):+async def test_vectorize_directory_returns_enqueued_levels(monkeypatch):+ vectorize = AsyncMock(return_value={0, 1})+ monkeypatch.setattr("openviking.utils.embedding_utils.vectorize_directory_meta", vectorize)+ processor = object.__new__(SemanticProcessor)+ processor._default_ctx = object()++ result = await processor._vectorize_directory(+ "viking://resources/root",+ "resource",+ "abstract",+ "overview",+ )++ assert result == {0, 1}+++async def test_target_source_syncs_before_semantic_executor(monkeypatch):monkeypatch.setattr("openviking.storage.queuefs.semantic_processor.get_viking_fs",lambda: _FakeVikingFS(),)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor",- _FakeDagExecutor,+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor",+ _FakeTreeExecutor,)monkeypatch.setattr("openviking.storage.queuefs.semantic_processor.SemanticLockScope.resolve",AsyncMock(return_value=SimpleNamespace(lock=None, close=AsyncMock())),)- _FakeDagExecutor.calls = []- _FakeDagExecutor.runs = []+ _FakeTreeExecutor.calls = []+ _FakeTreeExecutor.runs = []processor = SemanticProcessor()processor._enqueue_parent_refresh = AsyncMock()processor._sync_topdown_recursive = AsyncMock(@@ -113,14 +130,14 @@ async def test_target_source_syncs_before_semantic_dag(monkeypatch):await processor.on_dequeue(msg.to_dict())- assert _FakeDagExecutor.calls[0]["incremental_update"] is True- assert _FakeDagExecutor.calls[0]["target_uri"] == "viking://resources/org/repo"- assert _FakeDagExecutor.calls[0]["changes"] == {+ assert _FakeTreeExecutor.calls[0]["incremental_update"] is True+ assert _FakeTreeExecutor.calls[0]["target_uri"] == "viking://resources/org/repo"+ assert _FakeTreeExecutor.calls[0]["changes"] == {"added": [],"modified": ["viking://resources/org/repo/a.md"],"deleted": [],}- assert _FakeDagExecutor.runs == ["viking://resources/org/repo"]+ assert _FakeTreeExecutor.runs == ["viking://resources/org/repo"]@pytest.mark.asyncio@@ -130,8 +147,8 @@ async def test_stale_content_write_keeps_file_work_without_directory_aggregationlambda: _FakeVikingFS(),)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor",- _FakeDagExecutor,+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor",+ _FakeTreeExecutor,)monkeypatch.setattr("openviking.storage.queuefs.semantic_processor.SemanticLockScope.resolve",@@ -142,8 +159,8 @@ async def test_stale_content_write_keeps_file_work_without_directory_aggregationlambda msg: bool(msg.coalesce_key),)- _FakeDagExecutor.calls = []- _FakeDagExecutor.runs = []+ _FakeTreeExecutor.calls = []+ _FakeTreeExecutor.runs = []processor = SemanticProcessor()processor._enqueue_parent_refresh = AsyncMock()changed = "viking://resources/wiki/changed.md"@@ -159,31 +176,31 @@ async def test_stale_content_write_keeps_file_work_without_directory_aggregationawait processor.on_dequeue(msg.to_dict())- assert _FakeDagExecutor.calls[0]["aggregate_directory"] is False- assert _FakeDagExecutor.calls[0]["changes"] == {"modified": [changed]}- assert _FakeDagExecutor.calls[0]["coalesce_key"] == ""- assert _FakeDagExecutor.runs == ["viking://resources/wiki"]+ assert _FakeTreeExecutor.calls[0]["aggregate_directory"] is False+ assert _FakeTreeExecutor.calls[0]["changes"] == {"modified": [changed]}+ assert _FakeTreeExecutor.calls[0]["coalesce_key"] == ""+ assert _FakeTreeExecutor.runs == ["viking://resources/wiki"]processor._enqueue_parent_refresh.assert_not_awaited()@pytest.mark.asyncio@pytest.mark.parametrize("recursive", [False, True])-async def test_memory_reindex_uses_semantic_dag(monkeypatch, recursive):+async def test_memory_reindex_uses_semantic_executor(monkeypatch, recursive):monkeypatch.setattr("openviking.storage.queuefs.semantic_processor.get_viking_fs",lambda: _FakeVikingFS(),)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor",- _FakeDagExecutor,+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor",+ _FakeTreeExecutor,)monkeypatch.setattr("openviking.storage.queuefs.semantic_processor.SemanticLockScope.resolve",AsyncMock(return_value=SimpleNamespace(lock=None, close=AsyncMock())),)- _FakeDagExecutor.calls = []- _FakeDagExecutor.runs = []+ _FakeTreeExecutor.calls = []+ _FakeTreeExecutor.runs = []processor = SemanticProcessor()processor._process_memory_directory = AsyncMock()msg = SemanticMsg(@@ -197,9 +214,9 @@ async def test_memory_reindex_uses_semantic_dag(monkeypatch, recursive):await processor.on_dequeue(msg.to_dict())processor._process_memory_directory.assert_not_awaited()- assert _FakeDagExecutor.calls[0]["context_type"] == "memory"- assert _FakeDagExecutor.calls[0]["recursive"] is recursive- assert _FakeDagExecutor.runs == [msg.uri]+ assert _FakeTreeExecutor.calls[0]["context_type"] == "memory"+ assert _FakeTreeExecutor.calls[0]["recursive"] is recursive+ assert _FakeTreeExecutor.runs == [msg.uri]@pytest.mark.asynciodiff --git a/tests/storage/test_skill_package_cancellation.py b/tests/storage/test_skill_package_cancellation.pyindex 623ebe301..e19b7f74a 100644--- a/tests/storage/test_skill_package_cancellation.py+++ b/tests/storage/test_skill_package_cancellation.py@@ -23,13 +23,13 @@ from openviking.storage.collection_schemas import TextEmbeddingHandlerfrom openviking.storage.queuefs.embedding_msg import EmbeddingMsgfrom openviking.storage.queuefs.named_queue import NamedQueuefrom openviking.storage.queuefs.process_result import ProcessOutcome-from openviking.storage.queuefs.semantic_dag import DagStats, SemanticDagExecutor+from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor, SemanticTreeStatsfrom openviking.storage.queuefs.semantic_msg import SemanticMsgfrom openviking.storage.queuefs.semantic_processor import SemanticProcessorfrom openviking.telemetry.request_wait_tracker import get_request_wait_trackerfrom openviking_cli.session.user_id import UserIdentifierfrom tests.storage.test_collection_schemas import _DummyConfig, _DummyEmbedder-from tests.storage.test_semantic_dag_skip_files import _FakeVikingFS+from tests.storage.test_semantic_executor_skip_files import _FakeVikingFSdef _ctx():@@ -57,7 +57,7 @@ async def test_cancel_skill_stops_model_work_without_cancelling_other_packages(mresource: [{"name": "resource.md"}],})- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fs)+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)processor = _processor()started = asyncio.Event()cancelled = asyncio.Event()@@ -82,13 +82,13 @@ async def test_cancel_skill_stops_model_work_without_cancelling_other_packages(mprocessor._generate_single_file_summary = summarizeprocessor._vectorize_single_file = vectorizewith bind_task_context("skill-a", "acc", "alice"):- executor_a = SemanticDagExecutor(processor, "skill", 1, _ctx())+ executor_a = SemanticTreeExecutor(processor, "skill", 1, _ctx())task_a = asyncio.create_task(executor_a.run(root_a))await asyncio.wait_for(started.wait(), 1)with bind_task_context("skill-b", "acc", "alice"):- executor_b = SemanticDagExecutor(processor, "skill", 1, _ctx())+ executor_b = SemanticTreeExecutor(processor, "skill", 1, _ctx())task_b = asyncio.create_task(executor_b.run(root_b))- executor_resource = SemanticDagExecutor(processor, "resource", 1, _ctx())+ executor_resource = SemanticTreeExecutor(processor, "resource", 1, _ctx())task_resource = asyncio.create_task(executor_resource.run(resource))task_a.cancel()with pytest.raises(asyncio.CancelledError):@@ -104,7 +104,7 @@ async def test_cancel_skill_stops_model_work_without_cancelling_other_packages(masync def test_cancel_skill_waits_for_started_threaded_sidecar_write(monkeypatch):root = "viking://agent/skills/demo/reference"fs = _FakeVikingFS({root: []})- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fs)+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)started = asyncio.Event()release = threading.Event()loop = asyncio.get_running_loop()@@ -119,7 +119,7 @@ async def test_cancel_skill_waits_for_started_threaded_sidecar_write(monkeypatchfs.write_file = writeprocessor = _processor()- executor = SemanticDagExecutor(processor, "skill", 1, _ctx())+ executor = SemanticTreeExecutor(processor, "skill", 1, _ctx())worker = asyncio.create_task(executor.run(root))try:await asyncio.wait_for(started.wait(), 1)@@ -251,7 +251,7 @@ async def test_semantic_cancel_drains_embeddings_before_releasing_package(monkeystarted.set()def get_stats(self):- return DagStats()+ return SemanticTreeStats()class Lease:lock = {"lease_ref": "skill"}@@ -260,7 +260,7 @@ async def test_semantic_cancel_drains_embeddings_before_releasing_package(monkeyevents.append("released")monkeypatch.setattr(- "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor", Executor+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor", Executor)monkeypatch.setattr(SemanticProcessor, "_resolve_skill_semantic_lock", AsyncMock(return_value=Lease())diff --git a/tests/storage/test_skill_package_semantics.py b/tests/storage/test_skill_package_semantics.pyindex fd15f873b..4c1060179 100644--- a/tests/storage/test_skill_package_semantics.py+++ b/tests/storage/test_skill_package_semantics.py@@ -4,7 +4,7 @@ from unittest.mock import AsyncMockimport pytest-from openviking.storage.queuefs.semantic_dag import DagStats+from openviking.storage.queuefs.semantic_executor import SemanticTreeStatsfrom openviking.storage.queuefs.semantic_msg import SemanticMsgfrom openviking.storage.queuefs.semantic_processor import SemanticProcessorfrom openviking.telemetry import get_current_telemetry, register_telemetry@@ -47,10 +47,10 @@ async def test_skill_worker_keeps_package_locked_until_embeddings_finish(monkeypemitted.set()def get_stats(self):- return DagStats()+ return SemanticTreeStats()monkeypatch.setattr(- "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor", Executor+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor", Executor)monkeypatch.setattr("openviking.storage.queuefs.semantic_processor.SemanticLockScope.resolve",diff --git a/tests/storage/test_vector_transfer.py b/tests/storage/test_vector_transfer.pyindex 9d221d076..d3cbd35a7 100644--- a/tests/storage/test_vector_transfer.py+++ b/tests/storage/test_vector_transfer.py@@ -230,6 +230,7 @@ class _RealAclMemoryTransferBackend(_MemoryTransferBackend):def __init__(self, records: list[dict[str, Any]]) -> None:super().__init__(records)self.acl_manager = AclManager(self, _EnabledAclConfig())+ self.scroll_output_fields = []async def scroll(self,@@ -240,7 +241,8 @@ class _RealAclMemoryTransferBackend(_MemoryTransferBackend):*,ctx: RequestContext,) -> tuple[list[dict[str, Any]], str | None]:- del output_fields, ctx+ del ctx+ self.scroll_output_fields.append(list(output_fields or []))self.scroll_filters.append(filter)offset = int(cursor or 0)ordered = [@@ -628,6 +630,472 @@ async def test_legacy_transfer_reads_use_private_adapter_query_and_fetch(monkeypassert paths == ["/api/vikingdb/data/search/vector", "/api/vikingdb/data/fetch_in_collection"]+async def test_l2_diff_scan_reads_every_page_under_target_directory():+ root = "viking://resources/docs"+ backend = _RealAclMemoryTransferBackend(+ [+ _record("a", f"{root}/a.py", md5="ma", abstract="A"),+ _record("b", f"{root}/sub/b.py", md5="mb", abstract="B"),+ _record("ghost", f"{root}/ghost.py", md5="mg", abstract="G"),+ _record("directory", root, level=1),+ _record("outside", "viking://resources/other.py"),+ _record("other-account", f"{root}/private.py", account_id="other"),+ ]+ )++ records = await backend.get_l2_diff_records_under_uri(root, ctx=_ctx(), batch_size=2)++ assert records == {+ f"{root}/a.py": {"md5": "ma", "abstract": "A"},+ f"{root}/ghost.py": {"md5": "mg", "abstract": "G"},+ f"{root}/sub/b.py": {"md5": "mb", "abstract": "B"},+ }+ assert len(backend.scroll_filters) == 2+++async def test_incremental_inventory_reads_all_semantic_levels_under_target():+ root = "viking://resources/docs"+ backend = _RealAclMemoryTransferBackend(+ [+ _record("root-l0", root, level=0, abstract="root abstract"),+ _record("root-l1", root, level=1, abstract="root overview"),+ _record("file-l2", f"{root}/a.py", level=2, md5="ma"),+ _record("outside", "viking://resources/other.py", level=2),+ _record(+ "other-account",+ f"{root}/private.py",+ level=2,+ account_id="other",+ ),+ ]+ )++ records = await backend.get_incremental_inventory_under_uri(root, ctx=_ctx(), batch_size=2)++ assert records == {+ "root-l0": {"id": "root-l0", "uri": root, "level": 0, "md5": ""},+ "root-l1": {"id": "root-l1", "uri": root, "level": 1, "md5": ""},+ "file-l2": {+ "id": "file-l2",+ "uri": f"{root}/a.py",+ "level": 2,+ "md5": "ma",+ },+ }+ assert len(backend.scroll_filters) == 2+++async def test_incremental_inventory_projects_request_scalar_fields():+ root = "viking://resources/docs"+ backend = _RealAclMemoryTransferBackend(+ [+ _record(+ "file-l2",+ f"{root}/a.py",+ level=2,+ md5="ma",+ search_tags=["env=test"],+ )+ ]+ )++ records = await backend.get_incremental_inventory_under_uri(+ root,+ ctx=_ctx(),+ output_fields=["id", "uri", "level", "md5", "search_tags"],+ )++ assert records["file-l2"]["search_tags"] == ["env=test"]+ assert all("search_tags" in fields for fields in backend.scroll_output_fields)+ assert all("vector" not in fields for fields in backend.scroll_output_fields)+ assert all("sparse_vector" not in fields for fields in backend.scroll_output_fields)+++async def test_incremental_hydration_uses_dsl_then_fetches_only_missing_ids():+ root = "viking://resources/docs"+ first = _record("root-l0", root, level=0, abstract="root abstract")+ second = _record("file-l2", f"{root}/a.py", level=2, md5="ma")+ backend = _MemoryTransferBackend([first, second])+ backend._strict_transfer_page = AsyncMock(return_value=([dict(first)], None))+ backend._strict_transfer_get = AsyncMock(return_value=[dict(second)])++ records = await backend.hydrate_incremental_records(+ {+ "root-l0": {"uri": root, "level": 0},+ "file-l2": {"uri": f"{root}/a.py", "level": 2},+ },+ ctx=_ctx(),+ )++ assert set(records) == {"root-l0", "file-l2"}+ assert records["root-l0"]["abstract"] == "root abstract"+ assert records["file-l2"]["md5"] == "ma"+ backend._strict_transfer_get.assert_awaited_once_with(_ctx(), ["file-l2"])+ query = backend._strict_transfer_page.await_args+ assert query.args[1] == In("id", ["root-l0", "file-l2"])+ assert "vector" not in query.kwargs["output_fields"]+ assert "sparse_vector" not in query.kwargs["output_fields"]+++async def test_incremental_hydration_propagates_dsl_failure_without_fallback():+ backend = _MemoryTransferBackend([])+ backend._strict_transfer_page = AsyncMock(side_effect=RuntimeError("query failed"))+ backend._strict_transfer_get = AsyncMock()++ with pytest.raises(RuntimeError, match="query failed"):+ await backend.hydrate_incremental_records(+ {"file-l2": {"uri": "viking://resources/docs/a.py", "level": 2}},+ ctx=_ctx(),+ )++ backend._strict_transfer_get.assert_not_awaited()+++async def test_incremental_hydration_rejects_mismatched_record_identity():+ root = "viking://resources/docs"+ backend = _MemoryTransferBackend([])+ backend._strict_transfer_page = AsyncMock(+ return_value=(+ [_record("file-l2", f"{root}/wrong.py", level=2)],+ None,+ )+ )++ with pytest.raises(RuntimeError, match="identity mismatch"):+ await backend.hydrate_incremental_records(+ {"file-l2": {"uri": f"{root}/a.py", "level": 2}},+ ctx=_ctx(),+ )+++async def test_incremental_hydration_returns_only_records_still_present():+ backend = _MemoryTransferBackend([])+ backend._strict_transfer_page = AsyncMock(return_value=([], None))+ backend._strict_transfer_get = AsyncMock(return_value=[])++ records = await backend.hydrate_incremental_records(+ {"gone": {"uri": "viking://resources/docs/a.py", "level": 2}},+ ctx=_ctx(),+ )++ assert records == {}+++async def test_incremental_hydration_projects_dynamic_non_vector_schema_fields():+ root = "viking://resources/docs"+ record = _record(+ "file-l2",+ f"{root}/a.py",+ level=2,+ business_priority=7,+ )+ backend = _MemoryTransferBackend([record])+ backend.get_collection_meta = AsyncMock(+ return_value={+ "Fields": [+ {"FieldName": "id"},+ {"FieldName": "uri"},+ {"FieldName": "level"},+ {"FieldName": "abstract"},+ {"FieldName": "business_priority"},+ {"FieldName": "vector"},+ {"FieldName": "sparse_vector"},+ {"FieldName": "content"},+ ]+ }+ )+ backend._strict_transfer_page = AsyncMock(return_value=([record], None))++ hydrated = await backend.hydrate_incremental_records(+ {"file-l2": {"uri": f"{root}/a.py", "level": 2}},+ ctx=_ctx(),+ )++ assert hydrated["file-l2"]["business_priority"] == 7+ fields = backend._strict_transfer_page.await_args.kwargs["output_fields"]+ assert "business_priority" in fields+ assert "vector" not in fields+ assert "sparse_vector" not in fields+ assert "content" not in fields+++async def test_incremental_hydration_honors_explicit_summary_projection():+ root = "viking://resources/docs"+ record = _record(+ "file-l2",+ f"{root}/a.py",+ level=2,+ abstract="summary",+ business_priority=7,+ )+ backend = _MemoryTransferBackend([record])+ backend.get_collection_meta = AsyncMock(+ return_value={+ "Fields": [+ {"FieldName": "id"},+ {"FieldName": "uri"},+ {"FieldName": "level"},+ {"FieldName": "abstract"},+ {"FieldName": "business_priority"},+ ]+ }+ )+ backend._strict_transfer_page = AsyncMock(return_value=([record], None))++ hydrated = await backend.hydrate_incremental_records(+ {"file-l2": {"uri": f"{root}/a.py", "level": 2}},+ ctx=_ctx(),+ output_fields={"abstract"},+ )++ assert hydrated["file-l2"] == {+ "id": "file-l2",+ "uri": f"{root}/a.py",+ "level": 2,+ "abstract": "summary",+ }+ fields = backend._strict_transfer_page.await_args.kwargs["output_fields"]+ assert fields == ["id", "uri", "level", "abstract"]+ backend.get_collection_meta.assert_not_awaited()+++async def test_l2_diff_scan_reads_real_local_backend(tmp_path):+ if not getattr(vectordb_engine, "PersistStore", None):+ pytest.skip("local persistent vectordb engine is not available in this environment")++ root = "viking://resources/docs"+ backend = VikingVectorIndexBackend(+ config=VectorDBBackendConfig(+ backend="local", name="context", dimension=4, path=str(tmp_path)+ )+ )+ try:+ assert await backend.create_collection(+ "context", CollectionSchemas.context_collection("context", 4)+ )+ records = [+ _record(+ "a",+ f"{root}/a.py",+ md5="ma",+ abstract="A",+ vector=[0.1, 0.2, 0.3, 0.4],+ created_at="2026-08-20T00:00:00Z",+ updated_at="2026-08-20T00:00:00Z",+ ),+ _record(+ "ghost",+ f"{root}/ghost.py",+ md5="mg",+ abstract="G",+ vector=[0.1, 0.2, 0.3, 0.4],+ created_at="2026-08-20T00:00:01Z",+ updated_at="2026-08-20T00:00:01Z",+ ),+ _record(+ "outside",+ "viking://resources/other.py",+ vector=[0.1, 0.2, 0.3, 0.4],+ created_at="2026-08-20T00:00:02Z",+ updated_at="2026-08-20T00:00:02Z",+ ),+ ]+ await backend.upsert_many(records, ctx=_ctx())++ result = await backend.get_l2_diff_records_under_uri(root, ctx=_ctx(), batch_size=1)++ assert result == {+ f"{root}/a.py": {"md5": "ma", "abstract": "A"},+ f"{root}/ghost.py": {"md5": "mg", "abstract": "G"},+ }+ finally:+ await backend.close()+++async def test_incremental_inventory_and_hydration_read_real_local_backend(tmp_path):+ if not getattr(vectordb_engine, "PersistStore", None):+ pytest.skip("local persistent vectordb engine is not available in this environment")++ root = "viking://resources/docs"+ backend = VikingVectorIndexBackend(+ config=VectorDBBackendConfig(+ backend="local", name="context", dimension=4, path=str(tmp_path)+ )+ )+ try:+ assert await backend.create_collection(+ "context", CollectionSchemas.context_collection("context", 4)+ )+ await backend.upsert_many(+ [+ _record(+ "root-l0",+ root,+ level=0,+ abstract="root abstract",+ vector=[0.1, 0.2, 0.3, 0.4],+ created_at="2026-08-20T00:00:00Z",+ updated_at="2026-08-20T00:00:00Z",+ ),+ _record(+ "root-l1",+ root,+ level=1,+ abstract="root overview",+ vector=[0.1, 0.2, 0.3, 0.4],+ created_at="2026-08-20T00:00:01Z",+ updated_at="2026-08-20T00:00:01Z",+ ),+ _record(+ "a-l2",+ f"{root}/a.py",+ md5="ma",+ abstract="A",+ vector=[0.1, 0.2, 0.3, 0.4],+ created_at="2026-08-20T00:00:02Z",+ updated_at="2026-08-20T00:00:02Z",+ ),+ ],+ ctx=_ctx(),+ )++ inventory = await backend.get_incremental_inventory_under_uri(+ root, ctx=_ctx(), batch_size=1+ )+ hydrated = await backend.hydrate_incremental_records(inventory, ctx=_ctx())++ assert set(inventory) == {"root-l0", "root-l1", "a-l2"}+ assert inventory["a-l2"]["md5"] == "ma"+ assert hydrated["root-l0"]["abstract"] == "root abstract"+ assert hydrated["a-l2"]["abstract"] == "A"+ assert "vector" not in hydrated["a-l2"]+ assert "sparse_vector" not in hydrated["a-l2"]+ finally:+ await backend.close()+++async def test_l2_diff_scan_fails_closed_on_repeated_cursor(monkeypatch):+ backend = _MemoryTransferBackend([])+ monkeypatch.setattr(backend, "_strict_transfer_count", AsyncMock(return_value=3))+ page = [{"uri": "viking://resources/docs/a.py", "md5": "m"}]+ monkeypatch.setattr(+ backend,+ "_strict_transfer_page",+ AsyncMock(side_effect=[(page, "same"), (page, "same")]),+ )++ with pytest.raises(RuntimeError, match="duplicate L2 URI|cursor repeated"):+ await backend.get_l2_diff_records_under_uri(+ "viking://resources/docs", ctx=_ctx(), batch_size=1+ )+++async def test_l2_diff_scan_does_not_ignore_trailing_page_after_count(monkeypatch):+ backend = _MemoryTransferBackend([])+ monkeypatch.setattr(backend, "_strict_transfer_count", AsyncMock(return_value=1))+ monkeypatch.setattr(+ backend,+ "_strict_transfer_page",+ AsyncMock(+ side_effect=[+ ([{"uri": "viking://resources/docs/a.py"}], "1"),+ ([{"uri": "viking://resources/docs/new.py"}], None),+ ]+ ),+ )++ with pytest.raises(RuntimeError, match="count was 1"):+ await backend.get_l2_diff_records_under_uri(+ "viking://resources/docs", ctx=_ctx(), batch_size=1+ )+++async def test_incremental_inventory_retries_a_transiently_short_cursor(monkeypatch):+ backend = _MemoryTransferBackend([])+ root = "viking://resources/docs"+ monkeypatch.setattr(backend, "_strict_transfer_count", AsyncMock(return_value=2))+ first = {"id": "root-l0", "uri": root, "level": 0, "md5": ""}+ second = {"id": "root-l1", "uri": root, "level": 1, "md5": ""}+ monkeypatch.setattr(+ backend,+ "_strict_transfer_page",+ AsyncMock(+ side_effect=[+ ([first], None),+ ([first, second], None),+ ]+ ),+ )++ records = await backend.get_incremental_inventory_under_uri(root, ctx=_ctx(), batch_size=2)++ assert set(records) == {"root-l0", "root-l1"}+ assert backend._strict_transfer_count.await_count == 2+++async def test_incremental_inventory_retries_a_transient_count_undershoot(monkeypatch):+ backend = _MemoryTransferBackend([])+ root = "viking://resources/docs"+ monkeypatch.setattr(+ backend,+ "_strict_transfer_count",+ AsyncMock(side_effect=[1, 2]),+ )+ first = {"id": "root-l0", "uri": root, "level": 0, "md5": ""}+ second = {"id": "root-l1", "uri": root, "level": 1, "md5": ""}+ monkeypatch.setattr(+ backend,+ "_strict_transfer_page",+ AsyncMock(+ side_effect=[+ ([first, second], None),+ ([first, second], None),+ ]+ ),+ )++ records = await backend.get_incremental_inventory_under_uri(root, ctx=_ctx(), batch_size=2)++ assert set(records) == {"root-l0", "root-l1"}+ assert backend._strict_transfer_count.await_count == 2+++ "bad_record",+ [+ {"md5": "missing-uri"},+ {"uri": "viking://resources/other.py", "md5": "outside"},+ ],+)+async def test_l2_diff_scan_fails_closed_on_invalid_scoped_record(monkeypatch, bad_record):+ backend = _MemoryTransferBackend([])+ monkeypatch.setattr(backend, "_strict_transfer_count", AsyncMock(return_value=1))+ monkeypatch.setattr(+ backend,+ "_strict_transfer_page",+ AsyncMock(return_value=([bad_record], None)),+ )++ with pytest.raises(RuntimeError, match="invalid L2 URI"):+ await backend.get_l2_diff_records_under_uri("viking://resources/docs", ctx=_ctx())++@pytest.mark.asyncio@pytest.mark.parametrize("transfer_method", ["copy_uri_mapping", "update_uri_mapping"])async def test_uri_mapping_overwrites_exact_target_and_leaves_independent_chunks(diff --git a/tests/storage/test_volcengine_clients.py b/tests/storage/test_volcengine_clients.pyindex d715784f1..ed84f357f 100644--- a/tests/storage/test_volcengine_clients.py+++ b/tests/storage/test_volcengine_clients.py@@ -297,6 +297,7 @@ def test_volcengine_collection_update_data_posts_to_update_endpoint(monkeypatch)"project": "default","collection_name": "context","data": [{"id": "doc-1", "name": "updated"}],+ "ignore_unknown_fields": True,}@@ -334,6 +335,7 @@ def test_volcengine_collection_update_data_sanitizes_uri_fields(monkeypatch):"project": "default","collection_name": "context","data": [{"id": "doc-1", "uri": "/resources/demo", "parent_uri": "/resources"}],+ "ignore_unknown_fields": True,}@@ -484,6 +486,7 @@ def test_volcengine_api_key_collection_update_data_posts_to_update_endpoint(monk"project": "default","collection_name": "context","data": [{"id": "doc-1", "name": "updated"}],+ "ignore_unknown_fields": True,}@@ -523,6 +526,7 @@ def test_volcengine_api_key_collection_update_data_sanitizes_uri_fields(monkeypa"project": "default","collection_name": "context","data": [{"id": "doc-1", "uri": "/resources/demo", "parent_uri": "/resources"}],+ "ignore_unknown_fields": True,}@@ -1079,3 +1083,39 @@ def test_http_collection_update_data_posts_to_update_endpoint(monkeypatch):"collection_name": "context","fields": '[{"id": "doc-1", "name": "updated"}]',}+++def test_http_adapter_strict_count_propagates_http_failure(monkeypatch):+ from openviking.storage.vectordb.collection.collection import Collection+ from openviking.storage.vectordb.collection.http_collection import HttpCollection+ from openviking.storage.vectordb_adapters.http_adapter import HttpCollectionAdapter++ class _Response:+ status_code = 503+ text = "unavailable"++ @staticmethod+ def raise_for_status():+ raise requests.HTTPError("503 unavailable")++ monkeypatch.setattr(+ "openviking.storage.vectordb.collection.http_collection.requests.post",+ lambda *args, **kwargs: _Response(),+ )+ adapter = HttpCollectionAdapter(+ host="127.0.0.1",+ port=1933,+ project_name="default",+ collection_name="context",+ index_name="default",+ )+ adapter._collection = Collection(+ HttpCollection(+ ip="127.0.0.1",+ port=1933,+ meta_data={"ProjectName": "default", "CollectionName": "context"},+ )+ )++ with pytest.raises(requests.HTTPError, match="503 unavailable"):+ adapter.strict_count()diff --git a/tests/telemetry/test_execution.py b/tests/telemetry/test_execution.pyindex 6b4a11070..79ea9cdc2 100644--- a/tests/telemetry/test_execution.py+++ b/tests/telemetry/test_execution.py@@ -86,14 +86,21 @@ def test_operation_telemetry_summary_includes_resource_breakdown():from openviking.telemetry.operation import OperationTelemetrytelemetry = OperationTelemetry(operation="resources.add_resource", enabled=True)- telemetry.set("resource.request.duration_ms", 152.3)- telemetry.set("resource.process.duration_ms", 101.7)- telemetry.set("resource.parse.duration_ms", 38.1)- telemetry.set("resource.parse.warnings_count", 1)- telemetry.set("resource.finalize.duration_ms", 22.4)- telemetry.set("resource.summarize.duration_ms", 31.8)- telemetry.set("resource.wait.duration_ms", 46.9)- telemetry.set("resource.watch.duration_ms", 0.8)+ telemetry.set("resource.total.duration_ms", 152.3)+ telemetry.set("resource.source_execute.duration_ms", 101.7)+ telemetry.set("resource.source_prepare.duration_ms", 9.1)+ telemetry.set("resource.parse_artifact.duration_ms", 38.1)+ telemetry.set("resource.target_resolve.duration_ms", 22.4)+ telemetry.set("resource.update_plan.duration_ms", 31.8)+ telemetry.set("resource.update_plan.artifact_inventory.duration_ms", 4.0)+ telemetry.set("resource.update_plan.rnfv_snapshot.duration_ms", 11.0)+ telemetry.set("resource.update_plan.diff_and_compile.duration_ms", 16.8)+ telemetry.set("resource.content_commit.duration_ms", 12.4)+ telemetry.set("resource.derived_enqueue.duration_ms", 4.6)+ telemetry.set("queue.semantic.queue_wait.duration_ms", 46.9)+ telemetry.set("queue.semantic.execute.duration_ms", 70.0)+ telemetry.set("queue.embedding.queue_wait.duration_ms", 0.8)+ telemetry.set("queue.embedding.execute.duration_ms", 10.0)telemetry.set("resource.flags.wait", True)telemetry.set("resource.flags.build_index", True)telemetry.set("resource.flags.summarize", False)@@ -102,15 +109,27 @@ def test_operation_telemetry_summary_includes_resource_breakdown():summary = telemetry.finish().summaryassert summary["resource"] == {- "request": {"duration_ms": 152.3},- "process": {- "duration_ms": 101.7,- "parse": {"duration_ms": 38.1, "warnings_count": 1},- "finalize": {"duration_ms": 22.4},- "summarize": {"duration_ms": 31.8},+ "total": {"duration_ms": 152.3},+ "source_execute": {"duration_ms": 101.7},+ "source_prepare": {"duration_ms": 9.1},+ "parse_artifact": {"duration_ms": 38.1},+ "target_resolve": {"duration_ms": 22.4},+ "update_plan": {+ "duration_ms": 31.8,+ "artifact_inventory": {"duration_ms": 4.0},+ "rnfv_snapshot": {"duration_ms": 11.0},+ "diff_and_compile": {"duration_ms": 16.8},+ },+ "content_commit": {"duration_ms": 12.4},+ "derived_enqueue": {"duration_ms": 4.6},+ "semantic": {+ "queue_wait": {"duration_ms": 46.9},+ "execute": {"duration_ms": 70.0},+ },+ "embedding": {+ "queue_wait": {"duration_ms": 0.8},+ "execute": {"duration_ms": 10.0},},- "wait": {"duration_ms": 46.9},- "watch": {"duration_ms": 0.8},"flags": {"wait": True,"build_index": True,@@ -153,9 +172,7 @@ def test_operation_telemetry_summary_omits_zero_valued_fields():telemetry.set("semantic_nodes.done", 8)telemetry.set("semantic_nodes.pending", 1)telemetry.set("semantic_nodes.running", 0)- telemetry.set("resource.process.duration_ms", 12.3)- telemetry.set("resource.parse.duration_ms", 0.0)- telemetry.set("resource.parse.warnings_count", 0)+ telemetry.set("resource.source_execute.duration_ms", 12.3)telemetry.set("resource.flags.wait", False)telemetry.set("resource.flags.build_index", True)@@ -166,7 +183,7 @@ def test_operation_telemetry_summary_omits_zero_valued_fields():assert summary["queue"]["embedding"] == {"processed": 4}assert "running" not in summary["semantic_nodes"]assert summary["resource"] == {- "process": {"duration_ms": 12.3},+ "source_execute": {"duration_ms": 12.3},"flags": {"wait": False, "build_index": True, "summarize": False, "watch_enabled": False},}diff --git a/tests/telemetry/test_request_wait_tracker.py b/tests/telemetry/test_request_wait_tracker.pyindex 122788fd2..01efd2052 100644--- a/tests/telemetry/test_request_wait_tracker.py+++ b/tests/telemetry/test_request_wait_tracker.py@@ -71,6 +71,21 @@ def test_request_wait_tracker_counts_contexts_indexed_by_request():assert tracker.get_embedding_context_count(telemetry_id) == 0+def test_request_wait_tracker_accumulates_queue_wait_and_execution_durations():+ tracker = RequestWaitTracker()+ telemetry_id = "tm_queue_timing"++ tracker.register_request(telemetry_id)+ tracker.record_semantic_timing(telemetry_id, queue_wait_ms=12.5, execute_ms=30.0)+ tracker.record_semantic_timing(telemetry_id, queue_wait_ms=7.5, execute_ms=10.0)+ tracker.record_embedding_timing(telemetry_id, queue_wait_ms=5.0, execute_ms=8.0)++ assert tracker.get_queue_timing(telemetry_id) == {+ "semantic": {"queue_wait_ms": 20.0, "execute_ms": 40.0},+ "embedding": {"queue_wait_ms": 5.0, "execute_ms": 8.0},+ }++async def test_wait_for_request_timeout_keeps_existing_error():tracker = RequestWaitTracker()telemetry_id = "tm_wait_timeout"diff --git a/tests/telemetry/test_resource_summary.py b/tests/telemetry/test_resource_summary.pyindex e12127983..076c644fb 100644--- a/tests/telemetry/test_resource_summary.py+++ b/tests/telemetry/test_resource_summary.py@@ -1,12 +1,12 @@-from openviking.telemetry.operation import OperationTelemetry-def test_record_resource_queue_metrics_collects_queue_and_dag_stats(monkeypatch):+def test_record_resource_queue_metrics_collects_queue_and_tree_stats(monkeypatch):from openviking.telemetry.resource_summary import record_resource_queue_metricstelemetry = OperationTelemetry(operation="resources.add_resource", enabled=True)telemetry_id = telemetry.telemetry_id+class _SemanticStats:processed = 7requeue_count = 0@@ -17,7 +17,15 @@ def test_record_resource_queue_metrics_collects_queue_and_dag_stats(monkeypatch)requeue_count = 0error_count = 1- class _DagStats:+ class _Timing:+ @staticmethod+ def get_queue_timing(_tid):+ return {+ "semantic": {"queue_wait_ms": 12.5, "execute_ms": 30.0},+ "embedding": {"queue_wait_ms": 8.0, "execute_ms": 20.0},+ }++ class _SemanticTreeStats:total_nodes = 9done_nodes = 8pending_nodes = 1@@ -32,8 +40,12 @@ def test_record_resource_queue_metrics_collects_queue_and_dag_stats(monkeypatch)lambda _tid: _EmbeddingStats(),)monkeypatch.setattr(- "openviking.telemetry.resource_summary._consume_semantic_dag_stats",- lambda _tid, _uri: _DagStats(),+ "openviking.telemetry.resource_summary._consume_semantic_tree_stats",+ lambda _tid, _uri: _SemanticTreeStats(),+ )+ monkeypatch.setattr(+ "openviking.telemetry.resource_summary.get_request_wait_tracker",+ lambda: _Timing(),)record_resource_queue_metrics(@@ -47,6 +59,10 @@ def test_record_resource_queue_metrics_collects_queue_and_dag_stats(monkeypatch)assert summary["queue"]["semantic"]["error_count"] == 2assert summary["queue"]["embedding"]["processed"] == 11assert summary["queue"]["embedding"]["error_count"] == 1+ assert summary["queue"]["semantic"]["queue_wait"]["duration_ms"] == 12.5+ assert summary["queue"]["semantic"]["execute"]["duration_ms"] == 30.0+ assert summary["queue"]["embedding"]["queue_wait"]["duration_ms"] == 8.0+ assert summary["queue"]["embedding"]["execute"]["duration_ms"] == 20.0assert summary["semantic_nodes"]["total"] == 9assert summary["semantic_nodes"]["done"] == 8assert summary["semantic_nodes"]["pending"] == 1diff --git a/tests/test_config_loader.py b/tests/test_config_loader.pyindex 7f0717d68..132ad93c3 100644--- a/tests/test_config_loader.py+++ b/tests/test_config_loader.py@@ -130,6 +130,7 @@ def test_runtime_concurrency_uses_scope_specific_defaults():assert config.queue_workers.external_parse.max_concurrent == 4assert config.queue_workers.add_resource.max_concurrent == 4+ assert config.queue_workers.add_resource.file_operation_concurrency == 16assert config.queue_workers.add_resource.file_vectorization_concurrency == 8assert config.queue_workers.session_commit.max_concurrent == 8assert config.queue_workers.external_task.max_concurrent == 10@@ -150,6 +151,7 @@ def test_runtime_concurrency_accepts_separate_values():"external_parse": {"max_concurrent": 9},"add_resource": {"max_concurrent": 7,+ "file_operation_concurrency": 20,"file_vectorization_concurrency": 12,},"session_commit": {"max_concurrent": 50},@@ -161,6 +163,7 @@ def test_runtime_concurrency_accepts_separate_values():assert config.queue_workers.external_parse.max_concurrent == 9assert config.queue_workers.add_resource.max_concurrent == 7+ assert config.queue_workers.add_resource.file_operation_concurrency == 20assert config.queue_workers.add_resource.file_vectorization_concurrency == 12assert config.queue_workers.session_commit.max_concurrent == 50assert config.queue_workers.external_task.max_concurrent == 11diff --git a/tests/test_telemetry_runtime.py b/tests/test_telemetry_runtime.pyindex b159c2c5d..eb798f11b 100644--- a/tests/test_telemetry_runtime.py+++ b/tests/test_telemetry_runtime.py@@ -21,7 +21,7 @@ from openviking.observability.context import (reset_root_observability_context,)from openviking.storage.collection_schemas import TextEmbeddingHandler-from openviking.storage.queuefs.semantic_dag import DagStats+from openviking.storage.queuefs.semantic_executor import SemanticTreeStatsfrom openviking.storage.queuefs.semantic_msg import SemanticMsgfrom openviking.storage.queuefs.semantic_processor import SemanticProcessorfrom openviking.telemetry import (@@ -731,10 +731,13 @@ async def test_semantic_processor_binds_registered_operation_telemetry(monkeypatprocessor = SemanticProcessor()class FakeVikingFS:+ async def exists(self, uri, ctx=None):+ return True+async def ls(self, uri, ctx=None):return []- class _FakeDagExecutor:+ class _FakeTreeExecutor:def __init__(self, **kwargs):pass@@ -743,15 +746,15 @@ async def test_semantic_processor_binds_registered_operation_telemetry(monkeypatget_current_telemetry().record_token_usage("llm", 11, 7)def get_stats(self):- return DagStats()+ return SemanticTreeStats()monkeypatch.setattr("openviking.storage.queuefs.semantic_processor.get_viking_fs",lambda: FakeVikingFS(),)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor",- lambda **kwargs: _FakeDagExecutor(**kwargs),+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor",+ lambda **kwargs: _FakeTreeExecutor(**kwargs),)try:@@ -779,10 +782,13 @@ async def test_semantic_processor_binds_metric_account_context(monkeypatch):ran = {"value": False}class FakeVikingFS:+ async def exists(self, uri, ctx=None):+ return True+async def ls(self, uri, ctx=None):return []- class _FakeDagExecutor:+ class _FakeTreeExecutor:def __init__(self, **kwargs):pass@@ -793,15 +799,15 @@ async def test_semantic_processor_binds_metric_account_context(monkeypatch):assert root_context.account_id == "acct-semantic"def get_stats(self):- return DagStats()+ return SemanticTreeStats()monkeypatch.setattr("openviking.storage.queuefs.semantic_processor.get_viking_fs",lambda: FakeVikingFS(),)monkeypatch.setattr(- "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor",- lambda **kwargs: _FakeDagExecutor(**kwargs),+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor",+ lambda **kwargs: _FakeTreeExecutor(**kwargs),)await processor.on_dequeue(diff --git a/tests/test_upload_utils.py b/tests/test_upload_utils.pyindex 25b088fb5..373ac8403 100644--- a/tests/test_upload_utils.py+++ b/tests/test_upload_utils.py@@ -20,46 +20,6 @@ from openviking.utils.path_safety import sanitize_relative_viking_path# ----------------------------------------------------------------------------class FakeAGFS:- """Minimal AGFS mock that stores files and directories by path key."""-- def __init__(self, storage: Dict[str, bytes]) -> None:- self._storage = storage- self.dirs: List[str] = []-- def mkdir(self, path: str) -> None:- self.dirs.append(path)-- def write(self, path: str, content: bytes) -> None:- self._storage[path] = content---class FakeVikingFS:- """Minimal VikingFS mock for testing upload functions."""-- def __init__(self) -> None:- self.files: Dict[str, bytes] = {}- self.dirs: List[str] = []- self.write_file_bytes_calls: List[str] = []- self.agfs = FakeAGFS(self.files)-- def _uri_to_path(self, uri: str) -> str:- # Simplified: use the URI itself as the storage key so test assertions work.- return uri-- async def write_file_bytes(self, uri: str, content: bytes) -> None:- self.write_file_bytes_calls.append(uri)- self.files[uri] = content-- async def mkdir(self, uri: str, exist_ok: bool = False) -> None:- self.dirs.append(uri)---def viking_fs() -> FakeVikingFS:- return FakeVikingFS()--@pytest.fixturedef tmp_dir(tmp_path: Path) -> Path:"""Create a temporary directory with sample files for testing."""@@ -257,96 +217,106 @@ class TestShouldSkipFile:# ----------------------------------------------------------------------------class TestUploadDirectory:- @pytest.mark.asyncio- async def test_basic_upload(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:- count, warnings = await upload_directory(tmp_dir, "viking://temp/test", viking_fs)+class _RecordingStore:+ """Minimal parse output store recording writes/mkdirs by relative path."""- # Should upload: hello.py, readme.md, config.yaml, src/main.go- # Should skip: .hidden, image.png, empty.txt, __pycache__/mod.pyc- assert count == 4- assert "viking://temp/test/hello.py" in viking_fs.files- assert "viking://temp/test/readme.md" in viking_fs.files- assert "viking://temp/test/config.yaml" in viking_fs.files- assert "viking://temp/test/src/main.go" in viking_fs.files- assert "viking://temp/test/hello.py" in viking_fs.write_file_bytes_calls+ def __init__(self) -> None:+ self.writes: Dict[str, bytes] = {}+ self.dirs: List[str] = []- @pytest.mark.asyncio- async def test_uses_vikingfs_write_api_for_file_content(self, tmp_path: Path) -> None:- class GuardedAGFS(FakeAGFS):- def write(self, path: str, content: bytes) -> None:- raise AssertionError("upload_directory must not bypass VikingFS writes")+ async def mkdir(self, ref, rel_path: str = "") -> None:+ self.dirs.append(rel_path)- class GuardedVikingFS(FakeVikingFS):- def __init__(self) -> None:- super().__init__()- self.agfs = GuardedAGFS(self.files)+ async def write_bytes(self, ref, rel_path: str, content: bytes) -> None:+ self.writes[rel_path] = content- (tmp_path / "hello.py").write_text("print('hello')", encoding="utf-8")- viking_fs = GuardedVikingFS()+ async def read_bytes(self, ref, rel_path: str) -> bytes:+ return self.writes[rel_path]- count, warnings = await upload_directory(tmp_path, "viking://temp/guarded", viking_fs)- assert count == 1- assert warnings == []- assert viking_fs.files["viking://temp/guarded/hello.py"] == b"print('hello')"+class TestUploadDirectory:+ @pytest.mark.asyncio+ async def test_basic_upload(self, tmp_dir: Path) -> None:+ store = _RecordingStore()+ count, warnings = await upload_directory(+ tmp_dir, "repo", store=store, artifact_ref=object()+ )++ # Should upload: hello.py, readme.md, config.yaml, src/main.go+ # Should skip: .hidden, image.png, empty.txt, __pycache__/mod.pyc+ assert count == 4+ assert "repo/hello.py" in store.writes+ assert "repo/readme.md" in store.writes+ assert "repo/config.yaml" in store.writes+ assert "repo/src/main.go" in store.writes@pytest.mark.asyncio- async def test_skips_hidden_files(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:- await upload_directory(tmp_dir, "viking://temp/test", viking_fs)- assert all(".hidden" not in uri for uri in viking_fs.files)+ async def test_skips_hidden_files(self, tmp_dir: Path) -> None:+ store = _RecordingStore()+ await upload_directory(tmp_dir, "repo", store=store, artifact_ref=object())+ assert all(".hidden" not in rel for rel in store.writes)@pytest.mark.asyncio- async def test_skips_ignored_dirs(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:- await upload_directory(tmp_dir, "viking://temp/test", viking_fs)- assert all("__pycache__" not in uri for uri in viking_fs.files)+ async def test_skips_ignored_dirs(self, tmp_dir: Path) -> None:+ store = _RecordingStore()+ await upload_directory(tmp_dir, "repo", store=store, artifact_ref=object())+ assert all("__pycache__" not in rel for rel in store.writes)@pytest.mark.asyncio- async def test_skips_ignored_extensions(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:- await upload_directory(tmp_dir, "viking://temp/test", viking_fs)- assert all(".png" not in uri for uri in viking_fs.files)+ async def test_skips_ignored_extensions(self, tmp_dir: Path) -> None:+ store = _RecordingStore()+ await upload_directory(tmp_dir, "repo", store=store, artifact_ref=object())+ assert all(".png" not in rel for rel in store.writes)@pytest.mark.asyncio- async def test_skips_empty_files(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:+ async def test_skips_empty_files(self, tmp_dir: Path) -> None:(tmp_dir / "application.properties").write_bytes(b"\n")(tmp_dir / "blank.txt").write_bytes(b" \t\r\n")- await upload_directory(tmp_dir, "viking://temp/test", viking_fs)- assert all("empty.txt" not in uri for uri in viking_fs.files)- assert "viking://temp/test/application.properties" not in viking_fs.files- assert "viking://temp/test/blank.txt" not in viking_fs.files+ store = _RecordingStore()+ await upload_directory(tmp_dir, "repo", store=store, artifact_ref=object())+ assert all("empty.txt" not in rel for rel in store.writes)+ assert "repo/application.properties" not in store.writes+ assert "repo/blank.txt" not in store.writes@pytest.mark.asyncio- async def test_creates_root_dir(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:- await upload_directory(tmp_dir, "viking://temp/root", viking_fs)- assert "viking://temp/root" in viking_fs.dirs+ async def test_precreates_subdirectories(self, tmp_dir: Path) -> None:+ store = _RecordingStore()+ await upload_directory(tmp_dir, "repo", store=store, artifact_ref=object())+ # The nested src/ dir is pre-created before its files are written.+ assert "repo/src" in store.dirs@pytest.mark.asyncio- async def test_custom_ignore_dirs(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:+ async def test_custom_ignore_dirs(self, tmp_dir: Path) -> None:+ store = _RecordingStore()count, _ = await upload_directory(- tmp_dir, "viking://temp/test", viking_fs, ignore_dirs={"src"}+ tmp_dir, "repo", store=store, artifact_ref=object(), ignore_dirs={"src"})- assert all("src/" not in uri for uri in viking_fs.files)+ assert all("src/" not in rel for rel in store.writes)# Positive assertion: non-ignored files should still be uploadedassert count > 0- assert "viking://temp/test/hello.py" in viking_fs.files+ assert "repo/hello.py" in store.writes@pytest.mark.asyncio- async def test_custom_max_file_size(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:- count, _ = await upload_directory(tmp_dir, "viking://temp/test", viking_fs, max_file_size=5)+ async def test_custom_max_file_size(self, tmp_dir: Path) -> None:+ store = _RecordingStore()+ count, _ = await upload_directory(+ tmp_dir, "repo", store=store, artifact_ref=object(), max_file_size=5+ )# Most files are > 5 bytes, so fewer uploadsassert count < 4@pytest.mark.asyncio- async def test_respects_gitignore(self, tmp_path: Path, viking_fs: FakeVikingFS) -> None:+ async def test_respects_gitignore(self, tmp_path: Path) -> None:(tmp_path / ".gitignore").write_text("*.tmp\n", encoding="utf-8")(tmp_path / "keep.txt").write_text("ok", encoding="utf-8")(tmp_path / "skip.tmp").write_text("no", encoding="utf-8")- count, _ = await upload_directory(tmp_path, "viking://temp/gi", viking_fs)+ store = _RecordingStore()+ count, _ = await upload_directory(tmp_path, "gi", store=store, artifact_ref=object())assert count == 1- assert "viking://temp/gi/keep.txt" in viking_fs.files- assert "viking://temp/gi/skip.tmp" not in viking_fs.files+ assert "gi/keep.txt" in store.writes+ assert "gi/skip.tmp" not in store.writes# ---------------------------------------------------------------------------@@ -429,36 +399,55 @@ class TestSanitizeRelPath:# ----------------------------------------------------------------------------# upload_directory (additional edge cases)+# upload_directory (failure handling + md5 manifest)# ---------------------------------------------------------------------------class TestUploadDirectoryEdgeCases:@pytest.mark.asyncio- async def test_write_failure_produces_warning(self, tmp_path: Path) -> None:- class FailingAGFS:- def mkdir(self, path: str) -> None:- pass+ async def test_writes_content_md5_manifest(self, tmp_dir: Path) -> None:+ import hashlib+ import json- class FailingWriteFS:- agfs = FailingAGFS()+ from openviking.parse.parsers.upload_utils import ARTIFACT_MANIFEST_NAME- def _uri_to_path(self, uri: str) -> str:- return uri+ store = _RecordingStore()- async def mkdir(self, uri: str, exist_ok: bool = False) -> None:+ await upload_directory(tmp_dir, "repo", store=store, artifact_ref=object())++ # A manifest of final-byte md5 keyed by artifact-relative path is written+ # so the incremental diff can compare fingerprints without re-reading.+ assert ARTIFACT_MANIFEST_NAME in store.writes+ manifest = json.loads(store.writes[ARTIFACT_MANIFEST_NAME].decode("utf-8"))+ assert manifest["repo/hello.py"] == hashlib.md5(b"print('hello')").hexdigest()+ # Every uploaded business file has an md5; the manifest itself is excluded.+ assert "repo/src/main.go" in manifest+ assert ARTIFACT_MANIFEST_NAME not in manifest++ @pytest.mark.asyncio+ async def test_store_write_failure_produces_warning_and_skips_manifest(+ self, tmp_path: Path+ ) -> None:+ from openviking.parse.parsers.upload_utils import ARTIFACT_MANIFEST_NAME++ class FailingStore:+ async def mkdir(self, ref, rel_path: str = "") -> None:pass- async def write_file_bytes(self, uri: str, content: bytes) -> None:- raise IOError("write error")+ async def write_bytes(self, ref, rel_path: str, content: bytes) -> None:+ if rel_path == ARTIFACT_MANIFEST_NAME:+ raise AssertionError("manifest must not be written after a failed upload")+ raise IOError("store write error")(tmp_path / "ok.py").write_text("print(1)", encoding="utf-8")- count, warnings = await upload_directory(tmp_path, "viking://temp/fail", FailingWriteFS())+ count, warnings = await upload_directory(+ tmp_path, "repo", store=FailingStore(), artifact_ref=object()+ )assert count == 0assert len(warnings) == 1- assert "write error" in warnings[0]+ assert "store write error" in warnings[0]# ---------------------------------------------------------------------------diff --git a/tests/unit/test_context.py b/tests/unit/test_context.pyindex 732ad1a53..1e950dea1 100644--- a/tests/unit/test_context.py+++ b/tests/unit/test_context.py@@ -12,6 +12,7 @@ from openviking.core.context import (ResourceContentType,Vectorize,)+from openviking.utils.content_hash import content_md5from openviking_cli.session.user_id import UserIdentifier@@ -273,6 +274,7 @@ class TestContextMethods:assert ctx.get_vectorization_text() == "Test abstract"+class TestContextToDict:"""Test Context.to_dict."""@@ -543,3 +545,22 @@ class TestContextEdgeCases:ctx = Context(uri="viking://test/path%20with%20spaces/")assert ctx.uri == "viking://test/path%20with%20spaces/"++ def test_md5_roundtrip_preserves_known_fingerprint(self):+ md5 = content_md5(b"print(1)")+ ctx = Context(uri="viking://resources/x/a.py", is_leaf=True, md5=md5)++ assert ctx.to_dict()["md5"] == md5+ assert Context.from_dict(ctx.to_dict()).md5 == md5++ def test_missing_md5_is_omitted_from_dict(self):+ ctx = Context(uri="viking://resources/x/b.py", is_leaf=True)++ assert "md5" not in ctx.to_dict()+ assert Context.from_dict({"uri": "viking://resources/x/b.py"}).md5 == ""+++def test_content_md5_is_stable_hex_digest():+ assert content_md5(b"print(1)") == content_md5(b"print(1)")+ assert content_md5(b"a") != content_md5(b"b")+ assert len(content_md5(b"x")) == 32diff --git a/tests/unit/test_vectorize_file_strategy.py b/tests/unit/test_vectorize_file_strategy.pyindex c3003323a..5d4f598b9 100644--- a/tests/unit/test_vectorize_file_strategy.py+++ b/tests/unit/test_vectorize_file_strategy.py@@ -10,6 +10,7 @@ from openviking.parse.parsers.media.utils import (MPEG_TS_PACKET_SIZE,MPEG_TS_PROBE_BYTES,)+from openviking.storage.index_action import FieldPatchfrom openviking.utils import embedding_utilsfrom openviking.utils.ingest_options import IngestOptionsfrom openviking_cli.utils.config.parser_config import ImageConfig@@ -200,6 +201,89 @@ async def test_vectorize_file_uses_summary_first(monkeypatch, text_source, summaassert "content" not in queue.items[0].context_data+async def test_vectorize_file_threads_supplied_md5_without_reading_bytes(monkeypatch):+ # The upload site already holds the final bytes; vectorize_file must accept+ # the fingerprint via file_md5 and never read the file back to compute it.+ queue = DummyQueue()+ fs = DummyFS("deployment guide")+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ embedding_utils,+ "get_openviking_config",+ lambda: types.SimpleNamespace(+ embedding=types.SimpleNamespace(text_source="summary_first", max_input_tokens=1000)+ ),+ )++ await embedding_utils.vectorize_file(+ file_path="viking://user/default/resources/demo.md",+ summary_dict={"name": "demo.md", "summary": "deployment summary"},+ parent_uri="viking://user/default/resources",+ ctx=DummyReq(),+ file_md5="a" * 32,+ )++ assert len(queue.items) == 1+ assert queue.items[0].context_data["md5"] == "a" * 32+ # No file read was issued just to fingerprint the content.+ assert fs.read_file_bytes_calls == 0+++async def test_vectorize_file_uses_supplied_content_without_reading_file(monkeypatch):+ queue = DummyQueue()+ fs = DummyFS("remote content")+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ embedding_utils,+ "get_openviking_config",+ lambda: types.SimpleNamespace(+ embedding=types.SimpleNamespace(text_source="content_only", max_input_tokens=1000)+ ),+ )++ await embedding_utils.vectorize_file(+ file_path="viking://user/default/resources/a.py",+ summary_dict={"name": "a.py", "summary": ""},+ parent_uri="viking://user/default/resources",+ ctx=DummyReq(),+ file_content=b"print('local')",+ )++ assert queue.items[0].message == "print('local')"+ assert fs.read_file_calls == 0+++async def test_vectorize_file_omits_md5_when_not_supplied(monkeypatch):+ queue = DummyQueue()+ fs = DummyFS("deployment guide")+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ embedding_utils,+ "get_openviking_config",+ lambda: types.SimpleNamespace(+ embedding=types.SimpleNamespace(text_source="summary_first", max_input_tokens=1000)+ ),+ )++ await embedding_utils.vectorize_file(+ file_path="viking://user/default/resources/demo.md",+ summary_dict={"name": "demo.md", "summary": "deployment summary"},+ parent_uri="viking://user/default/resources",+ ctx=DummyReq(),+ )++ assert len(queue.items) == 1+ # Unknown fingerprint must not be emitted, so a partial update never clears it.+ assert "md5" not in queue.items[0].context_data+ assert fs.read_file_bytes_calls == 0++@pytest.mark.asyncioasync def test_vectorize_image_downsamples_large_embedding_input(monkeypatch):queue = DummyQueue()@@ -569,6 +653,26 @@ async def test_vectorize_directory_meta_appends_search_tags_by_level(monkeypatchassert queue.items[1].context_data["_upsert_options"] == {"search_tag_mode": "append"}+async def test_vectorize_directory_meta_applies_merge_per_level(monkeypatch):+ queue = DummyQueue()+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: DummyFS("ignored"))++ await embedding_utils.vectorize_directory_meta(+ uri="viking://user/default/resources/demo",+ abstract="demo abstract",+ overview="demo overview",+ ctx=DummyReq(),+ actions={0: "merge", 1: "upsert"},+ )++ assert len(queue.items) == 2+ assert "_upsert_options" not in queue.items[0].context_data+ assert queue.items[0].action.value == "merge"+ assert queue.items[1].action.value == "upsert"++@pytest.mark.asyncioasync def test_vectorize_directory_meta_l1_abstract_is_overview(monkeypatch):"""L1 records must carry the overview in the abstract scalar so Rerank@@ -1107,3 +1211,138 @@ async def test_skill_directory_body_frontmatter_is_not_parsed_twice(monkeypatch)overview_msg = next(item for item in queue.items if item.context_data["level"] == 1)assert overview_msg.context_data["abstract"] == bodyassert body in overview_msg.message+++async def test_full_upsert_uses_resolved_search_tags_from_plan(monkeypatch):+ queue = DummyQueue()+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: DummyFS("body"))+ monkeypatch.setattr(+ embedding_utils,+ "get_openviking_config",+ lambda: types.SimpleNamespace(+ embedding=types.SimpleNamespace(text_source="summary_only", max_input_tokens=1000)+ ),+ )++ await embedding_utils.vectorize_file(+ file_path="viking://resources/repo/a.py",+ summary_dict={"name": "a.py", "summary": "summary"},+ parent_uri="viking://resources/repo",+ ctx=DummyReq(),+ scalar_override={"search_tags": ["team=old", "lang=python", "team=new", "owner=alice"]},+ ingest_options=IngestOptions.from_search_tags(["team=new", "owner=alice"], mode="append"),+ action="upsert",+ )++ msg = queue.items[0]+ assert msg.context_data["search_tags"] == [+ "team=old",+ "lang=python",+ "team=new",+ "owner=alice",+ ]+ assert msg.action.value == "upsert"+ assert msg.context_data["_upsert_options"] == {"search_tag_mode": "append"}+++async def test_context_vectorization_defaults_to_merge_for_legacy_producers(monkeypatch):+ queue = DummyQueue()+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: DummyFS("body"))+ monkeypatch.setattr(+ embedding_utils,+ "get_openviking_config",+ lambda: types.SimpleNamespace(+ embedding=types.SimpleNamespace(text_source="summary_only", max_input_tokens=1000)+ ),+ )++ await embedding_utils.vectorize_file(+ file_path="viking://resources/repo/a.py",+ summary_dict={"name": "a.py", "summary": "summary"},+ parent_uri="viking://resources/repo",+ ctx=DummyReq(),+ )+ await embedding_utils.vectorize_directory_meta(+ uri="viking://resources/repo",+ abstract="abstract",+ overview="overview",+ ctx=DummyReq(),+ )++ assert [message.action.value for message in queue.items] == ["merge", "merge", "merge"]+ assert all(message.field_patch is None for message in queue.items)+++async def test_planned_merge_does_not_copy_context_into_patch_seed(monkeypatch):+ queue = DummyQueue()+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: DummyFS("body"))+ monkeypatch.setattr(+ embedding_utils,+ "get_openviking_config",+ lambda: types.SimpleNamespace(+ embedding=types.SimpleNamespace(text_source="summary_only", max_input_tokens=1000)+ ),+ )++ await embedding_utils.vectorize_file(+ file_path="viking://resources/repo/a.py",+ summary_dict={"name": "a.py", "summary": "large summary"},+ parent_uri="viking://resources/repo",+ ctx=DummyReq(),+ action="merge",+ field_patch=FieldPatch(+ {"search_tags": ["scope=new"]},+ {"search_tags": "append"},+ ),+ )++ msg = queue.items[0]+ assert msg.context_data["abstract"] == "large summary"+ assert msg.field_patch == FieldPatch(+ {"search_tags": ["scope=new"]},+ {"search_tags": "append"},+ )+++def test_semantic_processor_uses_merge_for_unplanned_vectorization():+ import inspect++ from openviking.storage.queuefs.semantic_processor import SemanticProcessor++ assert (+ inspect.signature(SemanticProcessor._vectorize_single_file).parameters["action"].default+ == "merge"+ )+++async def test_full_upsert_carries_existing_record_id_as_internal_override(monkeypatch):+ queue = DummyQueue()+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: DummyFS("body"))+ monkeypatch.setattr(+ embedding_utils,+ "get_openviking_config",+ lambda: types.SimpleNamespace(+ embedding=types.SimpleNamespace(text_source="summary_only", max_input_tokens=1000)+ ),+ )++ await embedding_utils.vectorize_file(+ file_path="viking://resources/repo/a.py",+ summary_dict={"name": "a.py", "summary": "summary"},+ parent_uri="viking://resources/repo",+ ctx=DummyReq(),+ scalar_override={"_record_id": "id-from-vector-db"},+ action="upsert",+ )++ msg = queue.items[0]+ assert msg.context_data["_upsert_record_id"] == "id-from-vector-db"+ assert "_record_id" not in msg.context_datadiff --git a/tests/utils/test_local_artifact_persist.py b/tests/utils/test_local_artifact_persist.pynew file mode 100644index 000000000..a3f946496--- /dev/null+++ b/tests/utils/test_local_artifact_persist.py@@ -0,0 +1,220 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""End-to-end-ish test for the local-artifact persist branch.++Rather than faking the whole process_resource pipeline (tree/stat/pathlock), this+exercises the actual production method resource_processor uses to land a local+parse artifact into the AGFS resource tree, pinning the key invariant: a local+artifact is uploaded file-by-file to the final resource location, matching source+bytes, without going through persist_temp_tree or any AGFS temp write.+"""++from types import SimpleNamespace+from unittest.mock import AsyncMock++import pytest++from openviking.parse.output import LocalParseOutputStore+from openviking.storage.context_update_plan import ContentTreeAction, ContextUpdatePlan+from openviking.utils.content_hash import content_md5+from openviking.utils.resource_processor import ResourceProcessor+++class _DummyVikingDB:+ def get_embedder(self):+ return None+++class _RecordingAgfs:+ """Fake AGFS resource tree recording writes; forbids temp-tree persistence."""++ def __init__(self) -> None:+ self.files: dict[str, bytes] = {}+ self.persist_temp_tree_calls = 0+ self.created_temp_uris = 0++ def create_temp_uri(self, ctx=None):+ self.created_temp_uris += 1+ raise AssertionError("local persist must not allocate an AGFS temp uri")++ async def persist_temp_tree(self, *args, **kwargs):+ self.persist_temp_tree_calls += 1+ raise AssertionError("local persist must not call persist_temp_tree")++ async def write_file_bytes(self, uri, content, *, ctx=None, lease_ref=None):+ self.files[uri] = content++ async def mkdir(self, uri, exist_ok=False, ctx=None, lease_ref=None):+ del exist_ok, ctx, lease_ref++ async def read_file_bytes(self, uri, *, ctx=None):+ return self.files[uri]++ async def remove_files(self, uri, *, recursive=False, ctx=None, lease_ref=None):+ self.files.pop(uri, None)+++async def test_commit_local_artifact_uploads_to_resource_tree(tmp_path, monkeypatch):+ import json++ from openviking.parse.parsers.upload_utils import ARTIFACT_MANIFEST_NAME+ from openviking.utils.content_hash import content_md5++ # A local artifact laid out as <root>/repository/<files>, as code.parse writes.+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))+ ref = await store.create_artifact(root_type="dir")+ await store.write_bytes(ref, "repository/a.py", b"print('a')")+ await store.write_bytes(ref, "repository/src/b.py", b"print('b')")+ # The manifest sidecar is the md5 source of truth (written by upload_directory).+ await store.write_bytes(+ ref,+ ARTIFACT_MANIFEST_NAME,+ json.dumps(+ {+ "repository/a.py": content_md5(b"print('a')"),+ "repository/src/b.py": content_md5(b"print('b')"),+ }+ ).encode("utf-8"),+ )++ agfs = _RecordingAgfs()+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: agfs)++ rp = ResourceProcessor(vikingdb=_DummyVikingDB(), media_storage=None)++ # temp_doc_uri is <artifact_root>/repository (finalize builds it this way).+ doc_rel = rp._artifact_doc_rel(ref, f"{ref.root}/repository")+ assert doc_rel == "repository"++ plan = ContextUpdatePlan(+ root_uri="viking://resources/acme/demo",+ context_type="resource",+ content_tree_actions=(+ ContentTreeAction(+ "upsert",+ "a.py",+ new_kind="file",+ artifact_path="a.py",+ md5=content_md5(b"print('a')"),+ ),+ ContentTreeAction(+ "upsert",+ "src/b.py",+ new_kind="file",+ artifact_path="src/b.py",+ md5=content_md5(b"print('b')"),+ ),+ ),+ )+ snapshot = SimpleNamespace(+ new=SimpleNamespace(entries={}),+ formal=SimpleNamespace(entries={}),+ vectors=SimpleNamespace(records_by_id={}),+ )+ monkeypatch.setattr(+ "openviking.storage.resource_diff.build_rnfv_snapshot", AsyncMock(return_value=snapshot)+ )+ monkeypatch.setattr(+ "openviking.storage.context_update_plan.build_context_update_plan_from_snapshot",+ AsyncMock(return_value=(SimpleNamespace(entries={}), plan)),+ )++ committed = await rp._commit_directory_artifact_with_plan(+ output_store=store,+ artifact_ref=ref,+ doc_rel=doc_rel,+ root_uri="viking://resources/acme/demo",+ target_preexisting=False,+ ctx=SimpleNamespace(account_id="test-account"),+ lease_ref=None,+ vectorize=True,+ summarize=False,+ processing_mode="semantic_and_vectors",+ is_code_repo=True,+ ingest_options=None,+ source_metadata=None,+ )++ # Files landed under the resource root with the repository prefix stripped,+ # bytes intact, and no temp-tree persistence / AGFS temp allocation occurred.+ assert agfs.files["viking://resources/acme/demo/a.py"] == b"print('a')"+ assert agfs.files["viking://resources/acme/demo/src/b.py"] == b"print('b')"+ assert agfs.persist_temp_tree_calls == 0+ assert agfs.created_temp_uris == 0+ assert committed.content_tree_actions == ()+++async def test_commit_local_flat_file_uploads_exact_target(tmp_path, monkeypatch):+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))+ ref = await store.create_artifact(root_type="dir")+ await store.write_bytes(ref, "document/report.md", b"report")+ agfs = _RecordingAgfs()+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: agfs)++ plan = ContextUpdatePlan(+ root_uri="viking://resources/report.md",+ context_type="resource",+ content_tree_actions=(+ ContentTreeAction(+ "upsert",+ "",+ new_kind="file",+ artifact_path="",+ md5=content_md5(b"report"),+ ),+ ),+ )+ snapshot = SimpleNamespace(+ new=SimpleNamespace(entries={}),+ formal=SimpleNamespace(entries={}),+ vectors=SimpleNamespace(records_by_id={}),+ )+ monkeypatch.setattr(+ "openviking.storage.resource_diff.build_rnfv_snapshot", AsyncMock(return_value=snapshot)+ )+ monkeypatch.setattr(+ "openviking.storage.context_update_plan.build_context_update_plan_from_snapshot",+ AsyncMock(return_value=(SimpleNamespace(entries={}), plan)),+ )++ result = await ResourceProcessor(+ vikingdb=_DummyVikingDB(), media_storage=None+ )._commit_directory_artifact_with_plan(+ output_store=store,+ artifact_ref=ref,+ doc_rel="document/report.md",+ root_uri="viking://resources/report.md",+ target_preexisting=False,+ root_is_file=True,+ ctx=SimpleNamespace(account_id="test-account"),+ lease_ref=None,+ vectorize=True,+ summarize=False,+ processing_mode="semantic_and_vectors",+ is_code_repo=False,+ ingest_options=None,+ source_metadata=None,+ )++ assert agfs.files == {"viking://resources/report.md": b"report"}+ assert result.content_tree_actions == ()+++def test_build_parse_output_store_defaults_to_none(monkeypatch):+ # Default (agfs) config yields no store, so callers keep legacy behaviour.+ rp = ResourceProcessor(vikingdb=_DummyVikingDB(), media_storage=None)+ assert rp._build_parse_output_store() is None+++def test_artifact_doc_rel_falls_back_to_ref_metadata(tmp_path):+ from openviking.parse.output import ParseArtifactRef++ ref = ParseArtifactRef(+ backend="local",+ root=str(tmp_path / "artifact"),+ resource_rel="document",+ )++ assert ResourceProcessor._artifact_doc_rel(ref, "not-a-path-below-the-root") == "document"diff --git a/tests/utils/test_resource_processor_processing_mode.py b/tests/utils/test_resource_processor_processing_mode.pyindex dec253fe3..3f9cd90fd 100644--- a/tests/utils/test_resource_processor_processing_mode.py+++ b/tests/utils/test_resource_processor_processing_mode.py@@ -8,6 +8,13 @@ from unittest.mock import AsyncMock, Mockimport pytestfrom openviking.server.identity import RequestContext, Role+from openviking.storage.context_update_plan import (+ ContextUpdatePlan,+ SemanticPlan,+ SemanticTreeEntry,+ SemanticTreeSnapshot,+)+from openviking.storage.index_action import FieldPatchfrom openviking.utils.ingest_options import IngestOptionsfrom openviking.utils.resource_processor import ResourceProcessorfrom openviking_cli.session.user_id import UserIdentifier@@ -149,8 +156,37 @@ async def test_flat_file_refreshes_parent_semantics_and_vectorizes_via_summary(skip_vectorization=False,ingest_options=IngestOptions(),created=True,+ file_md5=None,+ file_abstract="",+ )+++async def test_local_flat_file_refresh_passes_final_md5(monkeypatch, ctx):+ root_uri = "viking://resources/report.md"+ viking_fs = SimpleNamespace(+ _async_agfs=SimpleNamespace(pathlock_release=AsyncMock()),+ )+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)+ processor = ResourceProcessor(_FakeVikingDB())+ summarizer = SimpleNamespace(refresh_file_parent=AsyncMock(return_value={"status": "success"}))+ processor._get_summarizer = Mock(return_value=summarizer)++ await processor.finish_prepared_resource(+ {+ "root_uri": root_uri,+ "temp_uri": root_uri,+ "source_committed": True,+ "target_preexisting": True,+ "root_is_file": True,+ "file_md5s": {root_uri: "final-md5"},+ },+ ctx=ctx,+ build_index=True,)+ assert summarizer.refresh_file_parent.await_args.kwargs["file_md5"] == "final-md5"+@pytest.mark.asyncioasync def test_flat_file_skips_all_post_processing_when_build_index_false(@@ -178,6 +214,384 @@ async def test_flat_file_skips_all_post_processing_when_build_index_false(vectorize_file.assert_not_awaited()+async def test_local_artifact_without_semantic_work_is_cleaned(monkeypatch, ctx, tmp_path):+ from openviking.parse.output import LocalParseOutputStore++ store = LocalParseOutputStore(local_root=str(tmp_path))+ ref = await store.create_artifact()+ await store.write_bytes(ref, "repository/a.py", b"a")+ processor = ResourceProcessor(_FakeVikingDB())+ processor._build_parse_output_store = Mock(return_value=store)++ await processor.finish_prepared_resource(+ {+ "root_uri": "viking://resources/demo",+ "temp_uri": "viking://resources/demo",+ "source_committed": True,+ "artifact_ref": ref.to_dict(),+ "artifact_files": ["a.py"],+ },+ ctx=ctx,+ build_index=False,+ )++ assert not tmp_path.joinpath(ref.root).exists()+++async def test_local_incremental_noop_skips_semantic_queue_and_releases_resources(+ monkeypatch, ctx, tmp_path+):+ from openviking.parse.output import LocalParseOutputStore++ store = LocalParseOutputStore(local_root=str(tmp_path))+ ref = await store.create_artifact()+ await store.write_bytes(ref, "repository/a.py", b"a")+ lock = {"lease_ref": "noop"}+ viking_fs = SimpleNamespace(+ _async_agfs=SimpleNamespace(pathlock_release=AsyncMock()),+ )+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)+ processor = ResourceProcessor(_FakeVikingDB())+ processor._build_parse_output_store = Mock(return_value=store)+ summarizer = SimpleNamespace(summarize=AsyncMock(return_value={"status": "success"}))+ processor._get_summarizer = Mock(return_value=summarizer)++ result = await processor.finish_prepared_resource(+ {+ "root_uri": "viking://resources/demo",+ "temp_uri": "viking://resources/demo",+ "source_committed": True,+ "target_preexisting": True,+ "artifact_ref": ref.to_dict(),+ "artifact_files": ["a.py"],+ "changes": {},+ "file_md5s": {},+ "file_abstracts": {},+ "incremental_noop": True,+ },+ ctx=ctx,+ resource_lock=lock,+ build_index=True,+ )++ assert result == {"status": "success", "root_uri": "viking://resources/demo"}+ summarizer.summarize.assert_not_awaited()+ viking_fs._async_agfs.pathlock_release.assert_awaited_once_with(lock)+ assert not tmp_path.joinpath(ref.root).exists()+++async def test_vectors_only_scalar_update_is_enqueued_before_lock_release(monkeypatch, ctx):+ from openviking.storage.context_update_plan import DirectIndexAction++ queue = SimpleNamespace(enqueue=AsyncMock(return_value="queued"))+ manager = SimpleNamespace(+ EMBEDDING="embedding",+ get_queue=lambda *_args, **_kwargs: queue,+ )+ monkeypatch.setattr("openviking.storage.queuefs.get_queue_manager", lambda: manager)+ viking_fs = SimpleNamespace(+ _async_agfs=SimpleNamespace(pathlock_release=AsyncMock()),+ )+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)+ processor = ResourceProcessor(_FakeVikingDB())+ action = DirectIndexAction(+ action="update_fields",+ record_id="a-l2",+ uri="viking://resources/demo/a.py",+ level=2,+ field_patch=FieldPatch({"search_tags": ["team=search"]}),+ )+ plan = ContextUpdatePlan(+ root_uri="viking://resources/demo",+ context_type="resource",+ direct_index_actions=(action,),+ )+ lock = {"lease_ref": "tags"}++ result = await processor.finish_prepared_resource(+ {+ "root_uri": "viking://resources/demo",+ "temp_uri": "viking://resources/demo",+ "source_committed": True,+ "target_preexisting": True,+ "incremental_noop": True,+ "context_update_plan": plan.to_dict(),+ },+ ctx=ctx,+ resource_lock=lock,+ build_index=False,+ processing_mode="vectors_only",+ )++ assert result == {"status": "success", "root_uri": "viking://resources/demo"}+ message = queue.enqueue.await_args.args[0]+ assert message.action.value == "update_fields"+ assert message.record_ids == ["a-l2"]+ assert message.update_fields["search_tags"] == ["team=search"]+ viking_fs._async_agfs.pathlock_release.assert_awaited_once_with(lock)+++async def test_directory_semantic_plan_is_enqueued_without_artifact_or_legacy_diff(+ monkeypatch, ctx+):+ root_uri = "viking://resources/demo"+ plan = SemanticPlan(+ root_uri=root_uri,+ context_type="resource",+ tree=SemanticTreeSnapshot(+ entries=(+ SemanticTreeEntry("", "directory", "unchanged", "aggregate"),+ SemanticTreeEntry("a.py", "file", "modified", "generate", md5="new"),+ )+ ),+ )+ context_plan = ContextUpdatePlan(root_uri, "resource", semantic_plan=plan)+ viking_fs = SimpleNamespace(+ _async_agfs=SimpleNamespace(+ pathlock_to_handoff=AsyncMock(return_value={"lease": "x"}),+ pathlock_handoff=AsyncMock(),+ pathlock_release=AsyncMock(),+ )+ )+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)+ processor = ResourceProcessor(_FakeVikingDB())+ summarizer = SimpleNamespace(+ summarize=AsyncMock(return_value={"status": "success", "enqueued_count": 1})+ )+ processor._get_summarizer = Mock(return_value=summarizer)++ await processor.finish_prepared_resource(+ {+ "root_uri": root_uri,+ "temp_uri": root_uri,+ "source_committed": True,+ "target_preexisting": True,+ "root_is_file": False,+ "context_update_plan": context_plan.to_dict(),+ },+ ctx=ctx,+ resource_lock={"lease_ref": "plan"},+ build_index=True,+ processing_mode="semantic_and_vectors",+ )++ kwargs = summarizer.summarize.await_args.kwargs+ assert SemanticPlan.from_dict(kwargs["semantic_plan"]) == plan+ assert kwargs["temp_uris"] == [root_uri]+ assert kwargs.get("changes") is None+ assert kwargs.get("artifact_ref") is None+++async def test_plan_artifact_is_cleaned_only_after_semantic_enqueue(monkeypatch, ctx, tmp_path):+ from openviking.parse.output import LocalParseOutputStore++ root_uri = "viking://resources/demo"+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))+ ref = await store.create_artifact()+ await store.write_bytes(ref, "repository/a.py", b"a")+ plan = SemanticPlan(+ root_uri=root_uri,+ context_type="resource",+ tree=SemanticTreeSnapshot(+ entries=(+ SemanticTreeEntry("", "directory", "added", "aggregate"),+ SemanticTreeEntry("a.py", "file", "added", "generate", md5="a"),+ )+ ),+ )+ context_plan = ContextUpdatePlan(root_uri, "resource", semantic_plan=plan)+ lock = {"lease_ref": "plan"}+ viking_fs = SimpleNamespace(+ _async_agfs=SimpleNamespace(+ pathlock_handoff=AsyncMock(),+ pathlock_release=AsyncMock(),+ )+ )+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)+ processor = ResourceProcessor(_FakeVikingDB())+ processor._build_parse_output_store = Mock(return_value=store)++ async def summarize(**kwargs):+ assert kwargs.get("artifact_ref") is None+ assert tmp_path.joinpath("artifacts", ref.root.split("/")[-1]).exists()+ return {"status": "success", "enqueued_count": 1}++ processor._get_summarizer = Mock(+ return_value=SimpleNamespace(summarize=AsyncMock(side_effect=summarize))+ )++ await processor.finish_prepared_resource(+ {+ "root_uri": root_uri,+ "temp_uri": root_uri,+ "source_committed": True,+ "target_preexisting": True,+ "root_is_file": False,+ "artifact_ref": ref.to_dict(),+ "context_update_plan": context_plan.to_dict(),+ "plan_artifact_committed": True,+ },+ ctx=ctx,+ resource_lock=lock,+ build_index=True,+ )++ assert not tmp_path.joinpath("artifacts", ref.root.split("/")[-1]).exists()+ viking_fs._async_agfs.pathlock_handoff.assert_awaited_once_with(lock)+++async def test_plan_enqueue_failure_cleans_artifact_and_releases_lock(monkeypatch, ctx, tmp_path):+ from openviking.parse.output import LocalParseOutputStore++ root_uri = "viking://resources/demo"+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))+ ref = await store.create_artifact()+ await store.write_bytes(ref, "repository/a.py", b"a")+ plan = SemanticPlan(+ root_uri=root_uri,+ context_type="resource",+ tree=SemanticTreeSnapshot(+ entries=(+ SemanticTreeEntry("", "directory", "added", "aggregate"),+ SemanticTreeEntry("a.py", "file", "added", "generate", md5="a"),+ )+ ),+ )+ context_plan = ContextUpdatePlan(root_uri, "resource", semantic_plan=plan)+ lock = {"lease_ref": "plan"}+ viking_fs = SimpleNamespace(+ _async_agfs=SimpleNamespace(+ pathlock_handoff=AsyncMock(),+ pathlock_release=AsyncMock(),+ )+ )+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)+ processor = ResourceProcessor(_FakeVikingDB())+ processor._build_parse_output_store = Mock(return_value=store)+ processor._get_summarizer = Mock(+ return_value=SimpleNamespace(+ summarize=AsyncMock(side_effect=RuntimeError("queue unavailable"))+ )+ )++ with pytest.raises(RuntimeError, match="queue unavailable"):+ await processor.finish_prepared_resource(+ {+ "root_uri": root_uri,+ "temp_uri": root_uri,+ "source_committed": True,+ "target_preexisting": True,+ "root_is_file": False,+ "artifact_ref": ref.to_dict(),+ "context_update_plan": context_plan.to_dict(),+ "plan_artifact_committed": True,+ },+ ctx=ctx,+ resource_lock=lock,+ build_index=True,+ )++ assert not tmp_path.joinpath("artifacts", ref.root.split("/")[-1]).exists()+ viking_fs._async_agfs.pathlock_handoff.assert_not_awaited()+ viking_fs._async_agfs.pathlock_release.assert_awaited_once_with(lock)+++async def test_plan_enqueue_error_result_cleans_artifact_and_fails(monkeypatch, ctx, tmp_path):+ from openviking.parse.output import LocalParseOutputStore++ root_uri = "viking://resources/demo"+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))+ ref = await store.create_artifact()+ await store.write_bytes(ref, "repository/a.py", b"a")+ plan = SemanticPlan(+ root_uri=root_uri,+ context_type="resource",+ tree=SemanticTreeSnapshot(+ entries=(+ SemanticTreeEntry("", "directory", "added", "aggregate"),+ SemanticTreeEntry("a.py", "file", "added", "generate", md5="a"),+ )+ ),+ )+ context_plan = ContextUpdatePlan(root_uri, "resource", semantic_plan=plan)+ lock = {"lease_ref": "plan"}+ viking_fs = SimpleNamespace(+ _async_agfs=SimpleNamespace(+ pathlock_handoff=AsyncMock(),+ pathlock_release=AsyncMock(),+ )+ )+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)+ processor = ResourceProcessor(_FakeVikingDB())+ processor._build_parse_output_store = Mock(return_value=store)+ processor._get_summarizer = Mock(+ return_value=SimpleNamespace(+ summarize=AsyncMock(return_value={"status": "error", "message": "queue rejected plan"})+ )+ )++ with pytest.raises(RuntimeError, match="queue rejected plan"):+ await processor.finish_prepared_resource(+ {+ "root_uri": root_uri,+ "temp_uri": root_uri,+ "source_committed": True,+ "target_preexisting": True,+ "root_is_file": False,+ "artifact_ref": ref.to_dict(),+ "context_update_plan": context_plan.to_dict(),+ "plan_artifact_committed": True,+ },+ ctx=ctx,+ resource_lock=lock,+ build_index=True,+ )++ assert not tmp_path.joinpath("artifacts", ref.root.split("/")[-1]).exists()+ viking_fs._async_agfs.pathlock_handoff.assert_not_awaited()+ viking_fs._async_agfs.pathlock_release.assert_awaited_once_with(lock)+++async def test_local_flat_refresh_failure_cleans_artifact(monkeypatch, ctx, tmp_path):+ from openviking.parse.output import LocalParseOutputStore++ store = LocalParseOutputStore(local_root=str(tmp_path))+ ref = await store.create_artifact()+ await store.write_bytes(ref, "document/report.md", b"body")+ processor = ResourceProcessor(_FakeVikingDB())+ processor._build_parse_output_store = Mock(return_value=store)+ processor._get_summarizer = Mock(+ return_value=SimpleNamespace(+ refresh_file_parent=AsyncMock(side_effect=RuntimeError("queue unavailable"))+ )+ )++ with pytest.raises(RuntimeError, match="queue unavailable"):+ await processor.finish_prepared_resource(+ {+ "root_uri": "viking://resources/report.md",+ "temp_uri": "viking://resources/report.md",+ "source_committed": True,+ "target_preexisting": True,+ "root_is_file": True,+ "artifact_ref": ref.to_dict(),+ "artifact_files": [""],+ },+ ctx=ctx,+ build_index=True,+ )++ assert not tmp_path.joinpath(ref.root).exists()++@pytest.mark.asyncioasync def test_vectors_only_replaces_preexisting_flat_file_without_directory_sync(monkeypatch,@@ -352,9 +766,56 @@ async def test_vectors_only_persists_tree_and_vectorizes_files_only(monkeypatch,search_tags=["team=search"],search_tag_mode="append",)- processor._delete_resource_semantic_markers.assert_not_awaited()- processor._delete_resource_semantic_vectors.assert_not_awaited()- viking_fs._async_agfs.pathlock_release.assert_awaited_once_with(lock)+++async def test_local_vectors_only_uses_artifact_snapshot_when_target_tree_is_empty(+ monkeypatch, ctx, tmp_path+):+ from openviking.parse.output import LocalParseOutputStore, ParseArtifactRef+ from openviking.utils.content_hash import content_md5++ store = LocalParseOutputStore(local_root=str(tmp_path))+ raw_ref = await store.create_artifact()+ ref = ParseArtifactRef(+ backend="local",+ root=raw_ref.root,+ resource_rel="repository",+ )+ await store.write_bytes(ref, "repository/a.py", b"print('a')")+ viking_fs = SimpleNamespace(tree=AsyncMock(return_value=[]))+ vectorize_file = AsyncMock(return_value=True)+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)+ monkeypatch.setattr("openviking.utils.resource_processor.vectorize_file", vectorize_file)+ monkeypatch.setattr(+ "openviking.utils.resource_processor.get_openviking_config",+ lambda: SimpleNamespace(+ queue_workers=SimpleNamespace(+ add_resource=SimpleNamespace(file_vectorization_concurrency=8)+ )+ ),+ )+ processor = ResourceProcessor(_FakeVikingDB())+ processor._build_parse_output_store = Mock(return_value=store)++ await processor.finish_prepared_resource(+ {+ "root_uri": "viking://resources/demo",+ "temp_uri": "viking://resources/demo",+ "source_committed": True,+ "artifact_ref": ref.to_dict(),+ "artifact_files": ["a.py"],+ "file_md5s": {"viking://resources/demo/a.py": content_md5(b"print('a')")},+ },+ ctx=ctx,+ build_index=True,+ processing_mode="vectors_only",+ )++ viking_fs.tree.assert_not_awaited()+ vectorize_file.assert_awaited_once()+ assert vectorize_file.await_args.kwargs["file_content"] == b"print('a')"+ assert vectorize_file.await_args.kwargs["file_md5"] == content_md5(b"print('a')")@pytest.mark.asyncio