Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

perf(resources): optimize incremental resource ingestion (#5175)

Jiahui Zhou336f2173b451
diff --git a/docs/design/freshness-aware-parent-bubbling-design.md b/docs/design/freshness-aware-parent-bubbling-design.md
index 2f223e5c2..0a4b9db78 100644
--- a/docs/design/freshness-aware-parent-bubbling-design.md
+++ b/docs/design/freshness-aware-parent-bubbling-design.md
@@ -52,7 +52,7 @@
### 3. 采样发生在子项摘要工作之后
-当前目录 DAG 会先生成或读取所有直接子项的摘要,最后才使用 `overview_sample_limit` 选出进入父目录 overview prompt 的部分输入。
+当前目录语义树会先生成或读取所有直接子项的摘要,最后才使用 `overview_sample_limit` 选出进入父目录 overview prompt 的部分输入。
这能够限制 prompt 大小,却没有避免未参与本次聚合的子项摘要工作。对于宽目录,如果我们的目标是降低重新摘要成本,就应当在调度本轮父目录聚合所需的子项摘要之前完成有界采样。
@@ -324,7 +324,7 @@ metadata 从 `pending_child_changes > 0` 重置或减少,也可能造成 raw s
| --- | --- |
| `openviking/storage/queuefs/semantic_ops/freshness_policy.py` | 纯粹的 `NOOP` / `MARK_PENDING` / `REFRESH_NOW` 判断 |
| `openviking/storage/semantic_sidecar.py` | 在 lease 下原子累加、读取和消费现有 freshness 计数 |
-| `SemanticDagExecutor` | 在宽目录聚合前完成本轮采样,并返回 L0/L1 正文变化结果 |
+| `SemanticTreeExecutor` | 在宽目录聚合前完成本轮采样,并返回 L0/L1 正文变化结果 |
| `SemanticProcessor` | 根据 L0 是否变化决定是否继续向父目录冒泡 |
| content write、delete、resource 路径 | 将文件增删改统一记录为所在目录的变化计数 |
diff --git a/docs/design/l0-l1-okf-sidecars-rfc.md b/docs/design/l0-l1-okf-sidecars-rfc.md
index d947ddd71..0bbf69584 100644
--- a/docs/design/l0-l1-okf-sidecars-rfc.md
+++ b/docs/design/l0-l1-okf-sidecars-rfc.md
@@ -23,7 +23,7 @@ OpenViking uses a three-layer context model:
| L1 | `.overview.md` | directory overview for navigation and rerank |
| L2 | source files and subdirectories | full content loaded on demand |
-The current implementation treats `.abstract.md` and `.overview.md` as generated hidden sidecars. `SemanticProcessor` / `SemanticDagExecutor` generate them bottom-up, and the vector pipeline maps those two filenames to `ContextLevel.ABSTRACT` and `ContextLevel.OVERVIEW`.
+The current implementation treats `.abstract.md` and `.overview.md` as generated hidden sidecars. `SemanticProcessor` / `SemanticTreeExecutor` generate them bottom-up, and the vector pipeline maps those two filenames to `ContextLevel.ABSTRACT` and `ContextLevel.OVERVIEW`.
## Goals
diff --git a/docs/design/metric-design.md b/docs/design/metric-design.md
index 605381b58..d715d49da 100644
--- a/docs/design/metric-design.md
+++ b/docs/design/metric-design.md
@@ -303,7 +303,7 @@ graph LR
| `ResourceIngestionEventDataSource` | `EventMetricDataSource` | ResourceProcessor、ResourceService、watch / wait 流程 | 资源处理事件与阶段摘要 | parse / finalize / summarize / wait / watch duration |
| `SessionLifecycleDataSource` | `EventMetricDataSource` | session create / used / commit / archive | 会话生命周期状态与事件 | commit 生命周期、contexts_used、archive 状态 |
| `EncryptionEventDataSource` | `EventMetricDataSource` | Encryptor、API Key 验证路径、KDF / Key Loader | 加密操作事件与密钥处理事件 | encrypt / decrypt / verify count、duration、bytes、kdf / key_load 耗时、auth_failed |
-| `QueuePipelineStateDataSource` | `StateMetricDataSource` | `QueueManager`、Semantic DAG、request queue stats | 队列与流水线状态 | pending、in_progress、processed、error_count、semantic_nodes |
+| `QueuePipelineStateDataSource` | `StateMetricDataSource` | `QueueManager`、Semantic tree、request queue stats | 队列与流水线状态 | pending、in_progress、processed、error_count、semantic_nodes |
| `TaskStateDataSource` | `StateMetricDataSource` | `TaskTracker` | 当前任务状态 | pending、running、completed、failed 数量 |
| `RetrievalStatsDataSource` | `DomainStatsMetricDataSource` | `RetrievalStatsCollector` | 检索累计统计 | query count、zero result、latency、rerank 情况 |
| `ModelUsageDataSource` | `DomainStatsMetricDataSource` | VLM / Embedding / Rerank token tracker | 模型使用统计 | 调用次数、耗时、token 消耗 |
@@ -415,7 +415,7 @@ Collector 与 DataSource 的主映射关系如下:
| `VLMCollector` | `EventMetricCollector` | token / duration 事件 | 调用数、耗时、token 统计 |
| `CacheCollector` | `EventMetricCollector` | cache 命中或未命中事件 | hit / miss |
| `EncryptionCollector` | `EventMetricCollector` | encrypt / decrypt / verify / kdf / key_load 事件 | 加密次数、认证失败、耗时、字节量、密钥处理统计 |
-| `QueueCollector` | `StateMetricCollector` | QueueManager / DAG 当前状态 | pending、in_progress、processed、errors |
+| `QueueCollector` | `StateMetricCollector` | QueueManager / tree 当前状态 | pending、in_progress、processed、errors |
| `RagfsMetricCollector` | `DomainStatsMetricCollector` | `RagfsMetricDataSource`,一次 RAGFS `metrics()` 调用 | 文件操作、Cache、multi-backend、Lock |
| `VikingDBCollector` | `StateMetricCollector` | VikingDB collection 当前状态 | health、vectors、collections |
| `ObserverHealthCollector` | `StateMetricCollector` | ObserverService 结果 | component health、component errors |
diff --git a/docs/en/about/02-changelog.md b/docs/en/about/02-changelog.md
index b8429e1a2..6205dc0e8 100644
--- a/docs/en/about/02-changelog.md
+++ b/docs/en/about/02-changelog.md
@@ -227,7 +227,7 @@ This changelog is automatically generated from [GitHub Releases](https://github.
- **Request-scoped HTTP profiling**: Servers can enable `server.profile_enabled`; requests with `profile=1` then run `cProfile` for only that HTTP request and append `profile` lines to JSON responses. The `ov` CLI can enable and display this with `--profile`.
- **Batch Session message ingestion**: Added `POST /api/v1/sessions/{session_id}/messages/batch` plus Python HTTP client and Session wrapper support via `batch_add_messages` (up to 100 messages per request), reducing HTTP round trips for LangChain/LangGraph-style integrations.
- **Memory embedding templates**: Memory schemas now support a top-level `embedding_template`, replacing field-level `searchable` flags. The built-in `entities`, `events`, and `preferences` templates include key fields plus final content in embedding input.
-- **Semantic indexing reliability**: Resource processing now syncs temp source trees into the target before running the semantic DAG (diff results use target URIs); stale semantic lock handoffs can be recovered by reacquiring tree locks, and lock acquisition failures requeue work instead of tripping the API circuit breaker.
+- **Semantic indexing reliability**: Resource processing now syncs temp source trees into the target before running the semantic tree (diff results use target URIs); stale semantic lock handoffs can be recovered by reacquiring tree locks, and lock acquisition failures requeue work instead of tripping the API circuit breaker.
- **Embedding input guardrails**: Embedding queue input is truncated according to `embedding.max_input_tokens`, and oversized-input errors are classified as `input_too_large` to avoid repeated retries for unrecoverable payloads.
### Upgrade Notes
diff --git a/docs/en/concepts/09-transaction.md b/docs/en/concepts/09-transaction.md
index 3835a184e..1178548d2 100644
--- a/docs/en/concepts/09-transaction.md
+++ b/docs/en/concepts/09-transaction.md
@@ -134,51 +134,38 @@ Operation flow:
| File moved from temp to final directory, then crash -> file exists but never searchable | Two separate paths for first-time add vs incremental update |
| Resource already on disk but rm deletes it while semantic processing / vectorization is still running -> wasted work | Lifecycle TreeLock held from finalization through processing completion |
-**First-time add** (target does not exist) — handled in `ResourceProcessor.process_resource` Phase 3.5:
+**First-time add and incremental update** use the same planned commit path:
```
-1. Acquire TreeLock on final_uri
- - If final_uri does not exist, check ancestor/descendant/same-path conflicts first
- - If there is no conflict, create final_uri and write final_uri/.path.ovlock as a T lock
-2. Keep temp as the source directory and enqueue SemanticMsg(uri=temp, target_uri=final_uri, lifecycle_lock_handle_id=...)
-3. DAG runs on temp and syncs temp content into final_uri after completion
- - Do not use raw agfs.mv(temp -> final_uri), because final_uri already exists for the lock file
-4. Clean up temp directory
-5. DAG starts lock refresh loop (refreshes the lock token and updates handle activity every lock_expire/2 seconds)
-6. DAG complete + all embeddings done -> release TreeLock
+1. Acquire the resource lock on final_uri.
+2. Build the R/N/F/V snapshot while holding the lock:
+ - R: normalized request intent
+ - N: parsed artifact inventory
+ - F: current formal resource tree
+ - V: current vector records, unless build_index=false
+3. Compile a ContextUpdatePlan.
+4. Apply the plan's content actions to final_uri synchronously.
+5. Clean up the parser artifact.
+6. Enqueue direct index actions and, when needed, a SemanticMsg containing the remaining SemanticPlan.
+7. Hand off the resource lock to semantic processing, or release it when there is no semantic work.
```
-If summarization and indexing are both disabled, no downstream DAG takes over.
-In that case `ResourceProcessor` copies temp directory content into `final_uri`
-under the same TreeLock, deletes temp, then releases the lock. It does not call
-`VikingFS.mv(temp, final_uri, lock_handle=handle)`, because move cleanup can
-remove the directory lock file.
+The formal content tree is therefore updated before semantic or embedding work
+runs. A successful content commit may temporarily be ahead of its derived
+summaries and vectors. Queue-backed work repairs that derived state; it no
+longer copies the parser temp tree into the formal tree.
During this period, `rm` attempting to acquire a TreeLock on the same path will fail with `ResourceBusyError`.
-**Incremental update** (target already exists) — temp stays in place:
-
-```
-1. Acquire TreeLock on target_uri (protect existing resource)
-2. Enqueue SemanticMsg(uri=temp, target_uri=final, lifecycle_lock_handle_id=...)
-3. DAG runs on temp, lock refresh loop active
-4. DAG completion triggers sync_diff_callback or move_temp_to_target_callback
-5. Callback completes -> release TreeLock
-```
-
-Note: DAG callbacks do NOT wrap operations in an outer lock. Each `VikingFS.rm` and `VikingFS.mv` has its own lock internally. An outer lock would conflict with these inner locks causing deadlock.
-
-Both first-time add and incremental update hold only `TreeLock(resource_dir)`.
-There is no `ExactPathLock(resource_dir) -> TreeLock(resource_dir)` handoff, so
-the two modes cannot accidentally release the same `.path.ovlock` file in the
-wrong scope.
-
Automatic naming is handled by the resource layer, not the lock service:
`ResourceProcessor` checks `exists(candidate_uri)` first; occupied candidates
try `_1`, `_2`, and so on. Only a non-existing candidate attempts `TreeLock`,
without waiting. If that candidate is busy, the next suffix is tried.
-**Server restart recovery**: SemanticMsg is persisted in QueueFS. On restart, `SemanticProcessor` detects that the `lifecycle_lock_handle_id` handle is missing from the in-memory LockManager and re-acquires a TreeLock.
+**Server restart recovery**: `SemanticMsg` and its `SemanticPlan` are persisted
+in QueueFS. On restart, `SemanticProcessor` detects that the
+`lifecycle_lock_handle_id` handle is missing from the in-memory LockManager and
+re-acquires a TreeLock before continuing derived work.
### Derived Semantic Files (.abstract.md / .overview.md)
diff --git a/docs/en/configuration/01-server.md b/docs/en/configuration/01-server.md
index f5dca04a1..ecd510fad 100644
--- a/docs/en/configuration/01-server.md
+++ b/docs/en/configuration/01-server.md
@@ -185,6 +185,9 @@ Search and Find requests default to `limit: 10`; override the limit on each API
},
"vectordb": {
"backend": "local"
+ },
+ "parse_output": {
+ "mode": "agfs"
}
}
}
@@ -198,10 +201,17 @@ Search and Find requests default to `limit: 10`; override the limit on each API
| `agfs.backend` | `local`, `memory`, `s3` | `local` | File and metadata backend |
| `vectordb.backend` | `local`, `cuvs`, `http`, `volcengine`, `vikingdb` | `local` | Vector database backend |
| `vectordb.dimension` | integer | follows Embedding | Vector collection dimension |
+| `parse_output.mode` | `agfs`, `local` | `agfs` | Backend for intermediate parser artifacts |
+| `parse_output.local_root` | path or `null` | system temp directory | Root directory used by local parser artifacts |
| `skip_process_lock` | boolean | `false` | Skip the workspace process lock; use only when accepting concurrent-write risk |
Remote backends also require endpoint, bucket/collection, credentials, and timeout fields. See [Configuration](../guides/01-configuration.md#storage) for complete examples.
+`parse_output.mode=local` avoids writing parser intermediates to shared AGFS.
+The same worker must commit the required bytes to the formal resource tree before
+enqueueing downstream work. Artifacts are temporary and are removed after the
+content commit; provision `local_root` with enough space for concurrent imports.
+
## Queue Worker Settings
### `queue_workers.external_parse`
@@ -217,9 +227,10 @@ This setting controls queue-job concurrency. It is separate from `vlm.media.max_
| Field | Type | Default | Description |
|---|---|---:|---|
| `max_concurrent` | integer | `4` | Number of complete AddResource jobs consumed concurrently; must be greater than `0`; requires a server restart after changes |
+| `file_operation_concurrency` | integer | `16` | Maximum concurrent file-level commit and fallback comparison operations within one AddResource job; must be greater than `0`; requires a server restart after changes |
| `file_vectorization_concurrency` | integer | `8` | Number of files concurrently read, prepared, and enqueued within one directory AddResource job when `processing_mode="vectors_only"`; must be greater than `0`; values above the internal safety limit of `64` are capped; requires a server restart after changes |
-`max_concurrent` controls independent AddResource jobs, while `file_vectorization_concurrency` controls files within one vectors-only directory job. It does not affect single-file resources or `semantic_and_vectors` processing.
+`max_concurrent` controls independent AddResource jobs. `file_operation_concurrency` controls file commit and fallback comparison work within one AddResource job, while `file_vectorization_concurrency` controls files within one vectors-only directory job.
### `queue_workers.session_commit`
diff --git a/docs/en/guides/07-operation-telemetry.md b/docs/en/guides/07-operation-telemetry.md
index cb93cee70..482074347 100644
--- a/docs/en/guides/07-operation-telemetry.md
+++ b/docs/en/guides/07-operation-telemetry.md
@@ -227,14 +227,21 @@ This group appears on resource ingestion operations such as `resources.add_resou
| Field | Meaning |
| --- | --- |
-| `summary.resource.request.duration_ms` | Total request-side duration for the add-resource flow |
-| `summary.resource.process.duration_ms` | Duration of the main resource-processing flow |
-| `summary.resource.process.parse.duration_ms` | Time spent parsing the resource |
-| `summary.resource.process.parse.warnings_count` | Number of parse warnings |
-| `summary.resource.process.finalize.duration_ms` | Time spent finalizing the resource tree |
-| `summary.resource.process.summarize.duration_ms` | Time spent on summarize/vectorize processing |
-| `summary.resource.wait.duration_ms` | Time spent waiting for downstream processing when `wait=true` |
-| `summary.resource.watch.duration_ms` | Time spent creating, updating, or removing watch tasks |
+| `summary.resource.total.duration_ms` | Total duration of the add-resource operation |
+| `summary.resource.source_execute.duration_ms` | Time spent executing the source operation |
+| `summary.resource.source_prepare.duration_ms` | Time spent preparing a durable source |
+| `summary.resource.parse_artifact.duration_ms` | Time spent producing the parser artifact |
+| `summary.resource.target_resolve.duration_ms` | Time spent resolving the final resource target |
+| `summary.resource.update_plan.duration_ms` | Total time spent building the incremental update plan |
+| `summary.resource.update_plan.artifact_inventory.duration_ms` | Time spent scanning and normalizing the parser artifact |
+| `summary.resource.update_plan.rnfv_snapshot.duration_ms` | Time spent reading the R/N/F/V snapshot |
+| `summary.resource.update_plan.diff_and_compile.duration_ms` | Time spent resolving the diff and compiling actions |
+| `summary.resource.content_commit.duration_ms` | Time spent applying content actions to the formal tree |
+| `summary.resource.derived_enqueue.duration_ms` | Time spent enqueueing semantic and index work |
+| `summary.resource.semantic.queue_wait.duration_ms` | Time semantic work spent waiting in QueueFS |
+| `summary.resource.semantic.execute.duration_ms` | Time spent executing semantic work |
+| `summary.resource.embedding.queue_wait.duration_ms` | Time embedding work spent waiting in QueueFS |
+| `summary.resource.embedding.execute.duration_ms` | Time spent executing embedding work |
| `summary.resource.flags.wait` | Whether the request used `wait=true` |
| `summary.resource.flags.build_index` | Whether the request enabled `build_index` |
| `summary.resource.flags.summarize` | Whether the request explicitly enabled `summarize` |
@@ -247,15 +254,17 @@ This group appears when OpenViking waits for queue-backed work to complete.
| Field | Meaning |
| --- | --- |
| `summary.queue.semantic.processed` | Number of semantic-queue messages processed |
+| `summary.queue.semantic.requeue_count` | Number of semantic messages re-enqueued for retry |
| `summary.queue.semantic.error_count` | Number of semantic-queue errors |
| `summary.queue.embedding.processed` | Number of embedding-queue messages processed |
+| `summary.queue.embedding.requeue_count` | Number of embedding messages re-enqueued for retry |
| `summary.queue.embedding.error_count` | Number of embedding-queue errors |
### `summary.semantic_nodes`
| Field | Meaning |
| --- | --- |
-| `summary.semantic_nodes.total` | Total DAG or semantic-node count |
+| `summary.semantic_nodes.total` | Total semantic-tree node count |
| `summary.semantic_nodes.done` | Number of completed nodes |
| `summary.semantic_nodes.pending` | Number of pending nodes |
| `summary.semantic_nodes.running` | Number of nodes still running |
diff --git a/docs/zh/concepts/09-transaction.md b/docs/zh/concepts/09-transaction.md
index 96fecfca5..ff3d17665 100644
--- a/docs/zh/concepts/09-transaction.md
+++ b/docs/zh/concepts/09-transaction.md
@@ -132,47 +132,35 @@ Memory 提取是幂等的,从同一个 archive 重新提取会得到相同结
| 文件从临时目录移到正式目录后崩溃 -> 文件存在但永远搜不到 | 首次添加与增量更新分离为两条独立路径 |
| 资源已落盘但语义处理/向量化还在跑时被 rm 删除 -> 处理白跑 | 生命周期 TreeLock,从落盘持续到处理完成 |
-**首次添加**(target 不存在)— 在 `ResourceProcessor.process_resource` Phase 3.5 中处理:
+**首次添加和增量更新**使用同一条计划提交路径:
```
-1. 获取 TreeLock,锁 final_uri
- - 如果 final_uri 目录不存在,先检查祖先/后代/同路径锁冲突
- - 无冲突则创建 final_uri 目录,并在 final_uri/.path.ovlock 写 T 锁
-2. 保留 temp 作为源目录,入队 SemanticMsg(uri=temp, target_uri=final_uri, lifecycle_lock_handle_id=...)
-3. DAG 在 temp 上跑,完成后把 temp 内容同步到 final_uri
- - final_uri 已经用于放锁文件,所以不做裸 agfs.mv(temp -> final_uri)
-4. 清理临时目录
-5. DAG 启动锁刷新循环(每 lock_expire/2 秒刷新锁 token 并更新 handle 活跃时间)
-6. DAG 完成 + 所有 embedding 完成 -> 释放 TreeLock
+1. 获取 final_uri 的资源锁。
+2. 持锁构建 R/N/F/V 快照:
+ - R:归一化后的请求意图
+ - N:解析产物清单
+ - F:当前正式资源树
+ - V:当前向量记录;build_index=false 时跳过
+3. 编译 ContextUpdatePlan。
+4. 同步把计划中的内容动作提交到 final_uri。
+5. 清理 parser artifact。
+6. 入队直接索引动作;需要语义处理时,再入队携带剩余 SemanticPlan 的 SemanticMsg。
+7. 将资源锁交接给语义处理;没有语义任务时直接释放。
```
-如果本次调用关闭了摘要和索引(没有下游 DAG 接管),则在同一把 TreeLock
-里把 temp 目录内容复制到 `final_uri`,清理 temp,然后释放锁。这里不调用
-`VikingFS.mv(temp, final_uri, lock_handle=handle)`,避免移动逻辑清理目录锁文件。
+因此正式内容树会先于 semantic 和 embedding 工作更新。内容提交成功后,
+派生摘要和向量可能短暂落后,并由队列任务补齐;队列不再负责把 parser
+临时树复制到正式树。
此期间 `rm` 尝试获取同路径 TreeLock 会失败,抛出 `ResourceBusyError`。
-**增量更新**(target 已存在)— temp 保持不动:
-
-```
-1. 获取 TreeLock,锁 target_uri(保护已有资源)
-2. 入队 SemanticMsg(uri=temp, target_uri=final, lifecycle_lock_handle_id=...)
-3. DAG 在 temp 上跑,启动锁刷新循环
-4. DAG 完成后触发 sync_diff_callback 或 move_temp_to_target_callback
-5. callback 执行完毕 -> 释放 TreeLock
-```
-
-注意:DAG callback 不在外层加锁。每个 `VikingFS.rm` 和 `VikingFS.mv` 内部各自有独立锁保护。外层锁会与内部锁冲突导致死锁。
-
-首次添加和增量更新都只持有 `TreeLock(resource_dir)`。这里不再做
-`ExactPathLock(resource_dir) -> TreeLock(resource_dir)` 的锁转交,避免两种锁复用
-同一个 `.path.ovlock` 时出现释放顺序错误。
-
自动命名由资源层处理,不属于锁服务:`ResourceProcessor` 先用 `exists(candidate_uri)`
判断候选目录是否已占用;已存在则尝试 `_1`、`_2` 后缀。候选目录不存在时才尝试
获取该目录的 `TreeLock`,且不等待;如果同名正在被并发请求处理,就直接尝试下一个后缀。
-**服务重启恢复**:SemanticMsg 持久化在 QueueFS 中。重启后 `SemanticProcessor` 发现 `lifecycle_lock_handle_id` 对应的 handle 不在内存中,会重新获取 TreeLock。
+**服务重启恢复**:`SemanticMsg` 及其中的 `SemanticPlan` 持久化在 QueueFS
+中。重启后 `SemanticProcessor` 发现 `lifecycle_lock_handle_id` 对应的 handle
+不在内存中,会重新获取 TreeLock 后继续派生处理。
### 派生语义文件(.abstract.md / .overview.md)
diff --git a/docs/zh/configuration/01-server.md b/docs/zh/configuration/01-server.md
index c6194f9a3..7d81bec08 100644
--- a/docs/zh/configuration/01-server.md
+++ b/docs/zh/configuration/01-server.md
@@ -185,6 +185,9 @@ Search 和 Find 请求的默认 `limit` 为 `10`,可以在每次 API 或 SDK
},
"vectordb": {
"backend": "local"
+ },
+ "parse_output": {
+ "mode": "agfs"
}
}
}
@@ -198,10 +201,16 @@ Search 和 Find 请求的默认 `limit` 为 `10`,可以在每次 API 或 SDK
| `agfs.backend` | `local`、`memory`、`s3` | `local` | 文件与元数据存储后端 |
| `vectordb.backend` | `local`、`cuvs`、`http`、`volcengine`、`vikingdb` | `local` | 向量数据库后端 |
| `vectordb.dimension` | integer | 跟随 Embedding | 向量集合维度 |
+| `parse_output.mode` | `agfs`、`local` | `agfs` | parser 中间产物的存储后端 |
+| `parse_output.local_root` | 路径或 `null` | 系统临时目录 | local parser artifact 的根目录 |
| `skip_process_lock` | boolean | `false` | 是否跳过 workspace 进程锁;仅在明确接受并发写风险时启用 |
远程存储后端还需要配置 endpoint、bucket/collection、鉴权和超时等字段。完整后端示例见[配置指南](../guides/01-configuration.md#storage)。
+`parse_output.mode=local` 可避免把 parser 中间产物写入共享 AGFS。当前 worker
+必须在下游任务入队前把所需字节提交到正式资源树。产物会在内容提交后清理;
+请为 `local_root` 预留足够空间以容纳并发导入。
+
## 队列 Worker 配置
### `queue_workers.external_parse`
@@ -217,9 +226,10 @@ Search 和 Find 请求的默认 `limit` 为 `10`,可以在每次 API 或 SDK
| 字段 | 类型 | 默认值 | 说明 |
|---|---|---:|---|
| `max_concurrent` | integer | `4` | 同时消费的完整 AddResource 作业数,必须大于 `0`;修改后需重启服务 |
+| `file_operation_concurrency` | integer | `16` | 单个 AddResource 作业内文件级提交和 fallback 比较操作的最大并发数,必须大于 `0`;修改后需重启服务 |
| `file_vectorization_concurrency` | integer | `8` | 当目录 AddResource 使用 `processing_mode="vectors_only"` 时,单个作业内并发读取、准备并入队的文件数,必须大于 `0`;超过内部安全上限 `64` 的值会被截断;修改后需重启服务 |
-`max_concurrent` 控制相互独立的 AddResource 作业并发,`file_vectorization_concurrency` 控制单个 vectors-only 目录作业内的文件并发。该配置不影响单文件资源或 `semantic_and_vectors` 处理。
+`max_concurrent` 控制相互独立的 AddResource 作业并发,`file_operation_concurrency` 控制单个 AddResource 作业内文件提交和 fallback 比较操作的并发,`file_vectorization_concurrency` 控制单个 vectors-only 目录作业内的文件并发。
### `queue_workers.session_commit`
diff --git a/docs/zh/guides/07-operation-telemetry.md b/docs/zh/guides/07-operation-telemetry.md
index 09616afed..a6fb04ce7 100644
--- a/docs/zh/guides/07-operation-telemetry.md
+++ b/docs/zh/guides/07-operation-telemetry.md
@@ -221,14 +221,21 @@ admission,所以其 `gpu_gate_queue` 为零;worker 侧等待仍计入 `batch
| 字段 | 含义 |
| --- | --- |
-| `summary.resource.request.duration_ms` | add-resource 请求主流程总耗时 |
-| `summary.resource.process.duration_ms` | 资源处理主流程耗时 |
-| `summary.resource.process.parse.duration_ms` | 资源解析阶段耗时 |
-| `summary.resource.process.parse.warnings_count` | 解析阶段 warning 数量 |
-| `summary.resource.process.finalize.duration_ms` | 资源树 finalize 阶段耗时 |
-| `summary.resource.process.summarize.duration_ms` | summarize 或 vectorize 阶段耗时 |
-| `summary.resource.wait.duration_ms` | `wait=true` 时等待下游处理完成的耗时 |
-| `summary.resource.watch.duration_ms` | 创建、更新或移除 watch 任务的耗时 |
+| `summary.resource.total.duration_ms` | add-resource 操作总耗时 |
+| `summary.resource.source_execute.duration_ms` | 执行源操作的耗时 |
+| `summary.resource.source_prepare.duration_ms` | 准备持久化 source 的耗时 |
+| `summary.resource.parse_artifact.duration_ms` | 生成 parser artifact 的耗时 |
+| `summary.resource.target_resolve.duration_ms` | 解析最终资源目标的耗时 |
+| `summary.resource.update_plan.duration_ms` | 构建增量更新计划的总耗时 |
+| `summary.resource.update_plan.artifact_inventory.duration_ms` | 扫描并归一化 parser artifact 的耗时 |
+| `summary.resource.update_plan.rnfv_snapshot.duration_ms` | 读取 R/N/F/V 快照的耗时 |
+| `summary.resource.update_plan.diff_and_compile.duration_ms` | 计算 diff 并编译动作的耗时 |
+| `summary.resource.content_commit.duration_ms` | 把内容动作应用到正式树的耗时 |
+| `summary.resource.derived_enqueue.duration_ms` | 将 semantic 和 index 工作入队的耗时 |
+| `summary.resource.semantic.queue_wait.duration_ms` | semantic 工作在 QueueFS 中等待的耗时 |
+| `summary.resource.semantic.execute.duration_ms` | semantic 工作执行耗时 |
+| `summary.resource.embedding.queue_wait.duration_ms` | embedding 工作在 QueueFS 中等待的耗时 |
+| `summary.resource.embedding.execute.duration_ms` | embedding 工作执行耗时 |
| `summary.resource.flags.wait` | 本次请求是否使用了 `wait=true` |
| `summary.resource.flags.build_index` | 本次请求是否启用了 `build_index` |
| `summary.resource.flags.summarize` | 本次请求是否显式启用了 `summarize` |
@@ -241,15 +248,17 @@ admission,所以其 `gpu_gate_queue` 为零;worker 侧等待仍计入 `batch
| 字段 | 含义 |
| --- | --- |
| `summary.queue.semantic.processed` | 已处理的 semantic queue 消息数 |
+| `summary.queue.semantic.requeue_count` | 为重试而重新入队的 semantic 消息数 |
| `summary.queue.semantic.error_count` | semantic queue 错误数 |
| `summary.queue.embedding.processed` | 已处理的 embedding queue 消息数 |
+| `summary.queue.embedding.requeue_count` | 为重试而重新入队的 embedding 消息数 |
| `summary.queue.embedding.error_count` | embedding queue 错误数 |
### `summary.semantic_nodes`
| 字段 | 含义 |
| --- | --- |
-| `summary.semantic_nodes.total` | DAG 或语义节点总数 |
+| `summary.semantic_nodes.total` | 语义树节点总数 |
| `summary.semantic_nodes.done` | 已完成节点数 |
| `summary.semantic_nodes.pending` | 待处理节点数 |
| `summary.semantic_nodes.running` | 正在处理中的节点数 |
diff --git a/examples/grafana/openviking_demo_dashboard.json b/examples/grafana/openviking_demo_dashboard.json
index 024762161..da863a8c7 100644
--- a/examples/grafana/openviking_demo_dashboard.json
+++ b/examples/grafana/openviking_demo_dashboard.json
@@ -5170,6 +5170,89 @@
],
"title": "Memory Operations / Type + Action",
"type": "timeseries"
+ },
+ {
+ "collapsed": false,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 205
+ },
+ "id": 62,
+ "panels": [],
+ "title": "Incremental Resource Diagnostics",
+ "type": "row"
+ },
+ {
+ "datasource": {
+ "type": "prometheus",
+ "uid": "${DS_PROMETHEUS}"
+ },
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "#73BF69",
+ "value": 0
+ },
+ {
+ "color": "#FF9830",
+ "value": 1
+ },
+ {
+ "color": "#F2495C",
+ "value": 5
+ }
+ ]
+ },
+ "unit": "s"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 24,
+ "x": 0,
+ "y": 206
+ },
+ "id": 63,
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "right",
+ "showLegend": true,
+ "sortBy": "max",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "hideZeros": false,
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.1",
+ "targets": [
+ {
+ "editorMode": "code",
+ "expr": "histogram_quantile(0.95, sum by (le, stage) (rate(openviking_resource_stage_duration_seconds_bucket{stage=~\"semantic_queue_wait|semantic_execute|embedding_queue_wait|embedding_execute\"}[$__rate_interval])))",
+ "legendFormat": "{{stage}}",
+ "range": true,
+ "refId": "A"
+ }
+ ],
+ "title": "Semantic / Embedding Queue Timing / P95",
+ "type": "timeseries"
}
],
"preload": false,
diff --git a/openviking/concurrency.py b/openviking/concurrency.py
index e86f74dc8..408f5ff0c 100644
--- a/openviking/concurrency.py
+++ b/openviking/concurrency.py
@@ -5,7 +5,59 @@
import asyncio
import threading
from collections import deque
+from collections.abc import Awaitable, Callable, Iterable
from concurrent.futures import Future
+from typing import TypeVar
+
+T = TypeVar("T")
+R = TypeVar("R")
+
+
+async def bounded_map(
+ items: Iterable[T],
+ worker: Callable[[T], Awaitable[R]],
+ *,
+ concurrency: int,
+) -> list[R]:
+ """Apply ``worker`` with a fixed number of tasks and preserve input order.
+
+ A failed worker prevents new items from being claimed, then all active
+ workers are joined before the original failure is re-raised.
+ """
+ if concurrency <= 0:
+ raise ValueError("concurrency must be positive")
+
+ iterator = iter(enumerate(items))
+ cursor_lock = asyncio.Lock()
+ stop = asyncio.Event()
+ results: dict[int, R] = {}
+
+ async def next_item() -> tuple[int, T] | None:
+ async with cursor_lock:
+ if stop.is_set():
+ return None
+ try:
+ return next(iterator)
+ except StopIteration:
+ return None
+
+ async def run_worker() -> None:
+ while item := await next_item():
+ index, value = item
+ try:
+ results[index] = await worker(value)
+ except BaseException:
+ stop.set()
+ raise
+
+ workers = [asyncio.create_task(run_worker()) for _ in range(concurrency)]
+ try:
+ await asyncio.gather(*workers)
+ except BaseException:
+ stop.set()
+ await asyncio.gather(*workers, return_exceptions=True)
+ raise
+ return [results[index] for index in range(len(results))]
class AsyncSemaphore:
diff --git a/openviking/core/context.py b/openviking/core/context.py
index 845df6346..b20d8f575 100644
--- a/openviking/core/context.py
+++ b/openviking/core/context.py
@@ -80,6 +80,7 @@ class Context:
account_id: Optional[str] = None,
owner_user_id: Optional[str] = None,
owner_space: Optional[str] = None,
+ md5: Optional[str] = None,
id: Optional[str] = None,
):
"""
@@ -110,6 +111,10 @@ class Context:
owner_user_id if owner_user_id is not None else owner_fields["owner_user_id"]
)
self.owner_space = owner_space or owner_fields["owner_user_id"] or ""
+ # md5 of the final stored bytes for this URI; None/"" means unknown (old
+ # records or non-file records), and incremental diff falls back to reading
+ # bytes rather than assuming equality.
+ self.md5 = md5 or ""
self.vector: Optional[List[float]] = None
self.vectorize = Vectorize(abstract)
@@ -171,6 +176,10 @@ class Context:
}
if self.level is not None:
data["level"] = int(self.level)
+ # Only emit md5 when known so we never overwrite an existing fingerprint
+ # with an empty string via partial update.
+ if self.md5:
+ data["md5"] = self.md5
if self.user:
data["user"] = self.user.to_dict()
@@ -229,6 +238,7 @@ class Context:
account_id=data.get("account_id"),
owner_user_id=data.get("owner_user_id"),
owner_space=data.get("owner_space"),
+ md5=data.get("md5"),
)
obj.id = data.get("id", obj.id)
obj.vector = data.get("vector")
diff --git a/openviking/parse/base.py b/openviking/parse/base.py
index 54f065686..5e934f8c4 100644
--- a/openviking/parse/base.py
+++ b/openviking/parse/base.py
@@ -14,7 +14,7 @@ from pathlib import Path
from typing import TYPE_CHECKING, Any, Dict, List, Optional
if TYPE_CHECKING:
- pass
+ from openviking.parse.output import ParseArtifactRef
# ============================================================================
# Common utility functions
@@ -291,6 +291,10 @@ class ParseResult:
# Temporary directory path (for v4.0 architecture)
temp_dir_path: Optional[str] = None # e.g., "/tmp/openviking_parse_a1b2c3d4"
+ # Serializable handle to the artifact backing ``temp_dir_path``. New parsers
+ # set it explicitly; legacy AGFS-only results may derive it from their temp URI.
+ artifact_ref: Optional["ParseArtifactRef"] = None
+
# Core metadata fields
source_format: Optional[str] = None # File format (e.g., "pdf", "markdown")
parser_name: Optional[str] = None # Parser name (e.g., "PDFParser")
@@ -306,6 +310,27 @@ class ParseResult:
"""Check if parsing was successful."""
return len(self.warnings) == 0
+ def ensure_artifact_ref(self) -> Optional["ParseArtifactRef"]:
+ """Return the artifact ref, deriving an AGFS one from ``temp_dir_path``.
+
+ Legacy parsers may still record an AGFS temp URI without a ref. A local
+ filesystem path is ambiguous and must always carry an explicit ref.
+ """
+ if self.artifact_ref is not None:
+ return self.artifact_ref
+ if not self.temp_dir_path:
+ return None
+ if not self.temp_dir_path.startswith("viking://temp/"):
+ raise ValueError("parse result with a non-AGFS temp path must provide artifact_ref")
+ from openviking.parse.output import ParseArtifactRef
+
+ self.artifact_ref = ParseArtifactRef(
+ backend="agfs",
+ root=self.temp_dir_path,
+ root_type="dir",
+ )
+ return self.artifact_ref
+
def get_all_nodes(self) -> List[ResourceNode]:
"""Get all nodes in the tree (flattened)."""
nodes = []
diff --git a/openviking/parse/output.py b/openviking/parse/output.py
new file mode 100644
index 000000000..4f9d13da8
--- /dev/null
+++ b/openviking/parse/output.py
@@ -0,0 +1,669 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Parse output store abstraction.
+
+Parsers write their intermediate artifacts (the temp tree that TreeBuilder later
+persists into the final resource location) through a small store interface rather
+than reaching for the global VikingFS singleton directly. This decouples "which
+backend holds the artifact" from "how a parser lays out its output".
+
+The AGFS implementation preserves the original shared-temp behavior, while the
+local implementation keeps parser artifacts on a configured shared local path.
+
+Design notes:
+- Business rules (filtering, encoding, flatten, name conflicts, sidecar
+ retention) live in the parsers / shared helpers, NOT in the store. A store
+ only moves bytes and directory entries for a single artifact.
+- :class:`ParseArtifactRef` is the serializable handle that crosses the queue in
+ place of a bare ``viking://temp/...`` string; runtime store objects never do.
+- Relative paths inside an artifact are validated with
+ :func:`sanitize_relative_viking_path` so a parser cannot escape its root.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import shutil
+import uuid
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Dict, List, Literal, Optional
+
+from openviking.utils.content_hash import content_md5
+from openviking.utils.path_safety import safe_join_viking_uri, sanitize_relative_viking_path
+
+_BACKENDS = frozenset({"agfs", "local"})
+_ROOT_TYPES = frozenset({"dir", "file"})
+
+# Sidecar at the artifact root mapping each business file's artifact-relative
+# path to the md5 of its final (encoding-normalized) bytes. The incremental diff
+# reads this so it can compare fingerprints without re-reading file contents.
+ARTIFACT_MANIFEST_NAME = ".artifact_manifest.json"
+
+
+@dataclass(frozen=True)
+class ParseArtifactRef:
+ """Serializable handle to one parse artifact.
+
+ ``root`` is backend-scoped (an AGFS ``viking://temp/<uuid>`` URI, or a local
+ absolute directory). ``resource_rel`` is the artifact-relative path of the
+ resource root inside ``root`` (empty when the root is the resource itself).
+ """
+
+ backend: str
+ root: str
+ resource_rel: str = ""
+ root_type: str = "dir"
+
+ def to_dict(self) -> dict[str, Any]:
+ return {
+ "backend": self.backend,
+ "root": self.root,
+ "resource_rel": self.resource_rel,
+ "root_type": self.root_type,
+ }
+
+ @classmethod
+ def from_dict(cls, data: dict[str, Any]) -> "ParseArtifactRef":
+ if not isinstance(data, dict):
+ raise ValueError("parse artifact ref must be an object")
+ backend = data.get("backend")
+ root = data.get("root")
+ resource_rel = data.get("resource_rel") or ""
+ root_type = data.get("root_type") or "dir"
+ if backend not in _BACKENDS:
+ raise ValueError(f"parse artifact ref backend must be one of {sorted(_BACKENDS)}")
+ if not isinstance(root, str) or not root:
+ raise ValueError("parse artifact ref root must be a non-empty string")
+ if not isinstance(resource_rel, str):
+ raise ValueError("parse artifact ref resource_rel must be a string")
+ if root_type not in _ROOT_TYPES:
+ raise ValueError(f"parse artifact ref root_type must be one of {sorted(_ROOT_TYPES)}")
+ return cls(
+ backend=backend,
+ root=root,
+ resource_rel=resource_rel,
+ root_type=root_type,
+ )
+
+
+@dataclass(frozen=True)
+class ArtifactEntry:
+ """One directory entry inside an artifact, with an artifact-relative path."""
+
+ name: str
+ rel_path: str
+ is_dir: bool
+
+
+class ParseOutputStore(ABC):
+ """Backend-agnostic storage for a parser's intermediate artifacts.
+
+ Methods take a :class:`ParseArtifactRef` plus an artifact-relative path; the
+ store resolves the absolute location. Only byte and directory operations live
+ here — parsers keep owning layout and filtering decisions.
+ """
+
+ backend: str
+
+ @abstractmethod
+ async def create_artifact(self, *, root_type: str = "dir") -> ParseArtifactRef:
+ """Allocate a fresh artifact root and return its reference."""
+
+ @abstractmethod
+ async def mkdir(self, ref: ParseArtifactRef, rel_path: str = "") -> None: ...
+
+ @abstractmethod
+ async def write_bytes(self, ref: ParseArtifactRef, rel_path: str, content: bytes) -> None: ...
+
+ @abstractmethod
+ async def read_bytes(self, ref: ParseArtifactRef, rel_path: str) -> bytes: ...
+
+ @abstractmethod
+ async def list(self, ref: ParseArtifactRef, rel_path: str = "") -> List[ArtifactEntry]: ...
+
+ @abstractmethod
+ async def cleanup(self, ref: ParseArtifactRef) -> None:
+ """Delete the artifact. Safe to call more than once."""
+
+ @abstractmethod
+ async def move_file(
+ self,
+ source_ref: ParseArtifactRef,
+ source_rel: str,
+ target_ref: ParseArtifactRef,
+ target_rel: str,
+ ) -> None:
+ """Move one file between artifacts owned by this store."""
+
+ # -- text convenience -------------------------------------------------
+ async def write_text(
+ self, ref: ParseArtifactRef, rel_path: str, content: str, *, encoding: str = "utf-8"
+ ) -> None:
+ await self.write_bytes(ref, rel_path, content.encode(encoding))
+
+ async def read_text(
+ self, ref: ParseArtifactRef, rel_path: str, *, encoding: str = "utf-8"
+ ) -> str:
+ return (await self.read_bytes(ref, rel_path)).decode(encoding)
+
+
+def _join_artifact_rel(parent: str, child: str) -> str:
+ parent = (parent or "").strip("/")
+ child = (child or "").strip("/")
+ joined = f"{parent}/{child}" if parent and child else parent or child
+ return sanitize_relative_viking_path(joined) if joined else ""
+
+
+class ParseArtifactWriter:
+ """Request-scoped writer that owns one parse artifact and its manifest."""
+
+ def __init__(self, store: ParseOutputStore, ref: ParseArtifactRef) -> None:
+ if ref.backend != store.backend:
+ raise ValueError(
+ f"artifact backend {ref.backend!r} does not match store {store.backend!r}"
+ )
+ self.store = store
+ self.ref = ref
+ self._md5_by_rel: Dict[str, str] = {}
+ self._finalized = False
+
+ @classmethod
+ async def create(
+ cls, store: ParseOutputStore, *, root_type: str = "dir"
+ ) -> "ParseArtifactWriter":
+ return cls(store, await store.create_artifact(root_type=root_type))
+
+ async def mkdir(self, rel_path: str = "") -> None:
+ await self.store.mkdir(self.ref, self.relative_path(rel_path))
+
+ def relative_path(self, path: str) -> str:
+ """Normalize an artifact-relative path or a path rooted at this artifact."""
+ value = str(path or "")
+ root = self.ref.root.rstrip("/")
+ if value == root:
+ return ""
+ if value.startswith(root + "/"):
+ value = value[len(root) + 1 :]
+ return _join_artifact_rel("", value)
+
+ async def write_bytes(self, rel_path: str, content: bytes, *, md5: str | None = None) -> None:
+ rel_path = self.relative_path(rel_path)
+ if not rel_path or rel_path == ARTIFACT_MANIFEST_NAME:
+ raise ValueError("artifact business file path must not be the manifest path")
+ await self.store.write_bytes(self.ref, rel_path, content)
+ self._md5_by_rel[rel_path] = md5 or content_md5(content)
+ self._finalized = False
+
+ async def write_text(self, rel_path: str, content: str, *, encoding: str = "utf-8") -> None:
+ await self.write_bytes(rel_path, content.encode(encoding))
+
+ async def finalize(self, *, resource_rel: str = "") -> ParseArtifactRef:
+ if not self._finalized:
+ await write_artifact_manifest(self.store, self.ref, self._md5_by_rel)
+ self._finalized = True
+ return ParseArtifactRef(
+ backend=self.ref.backend,
+ root=self.ref.root,
+ resource_rel=_join_artifact_rel("", resource_rel),
+ root_type=self.ref.root_type,
+ )
+
+ async def cleanup(self) -> None:
+ await self.store.cleanup(self.ref)
+
+ def record_md5(self, rel_path: str, md5: str) -> None:
+ if md5:
+ self._md5_by_rel[self.relative_path(rel_path)] = md5
+ self._finalized = False
+
+
+async def create_parse_artifact_writer(
+ store: ParseOutputStore | None = None,
+ *,
+ viking_fs: Any = None,
+ ctx: Any = None,
+ root_type: str = "dir",
+) -> ParseArtifactWriter:
+ """Create one request-scoped writer, defaulting to the AGFS backend."""
+ resolved_store = store or AgfsParseOutputStore(viking_fs=viking_fs, ctx=ctx)
+ return await ParseArtifactWriter.create(resolved_store, root_type=root_type)
+
+
+async def copy_artifact_tree(
+ *,
+ source_store: ParseOutputStore,
+ source_ref: ParseArtifactRef,
+ source_rel: str,
+ target: ParseArtifactWriter,
+ target_rel: str,
+ allowed_hidden: frozenset[str] = frozenset(),
+ flatten_single_output: bool = False,
+) -> int:
+ """Transfer one artifact subtree into another and return the file count."""
+ source_rel = _join_artifact_rel("", source_rel)
+ target_rel = _join_artifact_rel("", target_rel)
+ source_manifest = await read_artifact_manifest(source_store, source_ref)
+ copied = 0
+ source_entries = [
+ entry
+ for entry in await source_store.list(source_ref, source_rel)
+ if entry.rel_path != ARTIFACT_MANIFEST_NAME
+ and (not entry.name.startswith(".") or entry.name in allowed_hidden)
+ ]
+ if flatten_single_output and len(source_entries) == 1 and source_entries[0].is_dir:
+ wrapper = source_entries[0]
+ wrapper_entries = [
+ entry
+ for entry in await source_store.list(source_ref, wrapper.rel_path)
+ if not entry.name.startswith(".") or entry.name in allowed_hidden
+ ]
+ if len(wrapper_entries) == 1 and not wrapper_entries[0].is_dir:
+ source_entries = [wrapper_entries[0]]
+
+ async def _copy(src_dir: str, dst_dir: str, entries: List[ArtifactEntry] | None = None) -> None:
+ nonlocal copied
+ if dst_dir:
+ await target.mkdir(dst_dir)
+ for entry in (
+ entries if entries is not None else await source_store.list(source_ref, src_dir)
+ ):
+ if entry.rel_path == ARTIFACT_MANIFEST_NAME:
+ continue
+ if entry.name.startswith(".") and entry.name not in allowed_hidden:
+ continue
+ child_dst = _join_artifact_rel(dst_dir, entry.name)
+ if entry.is_dir:
+ await _copy(entry.rel_path, child_dst)
+ else:
+ md5 = source_manifest.get(entry.rel_path, "")
+ if source_store is target.store:
+ if not md5:
+ md5 = content_md5(await source_store.read_bytes(source_ref, entry.rel_path))
+ await target.store.move_file(source_ref, entry.rel_path, target.ref, child_dst)
+ target.record_md5(child_dst, md5)
+ else:
+ await target.write_bytes(
+ child_dst,
+ await source_store.read_bytes(source_ref, entry.rel_path),
+ md5=md5,
+ )
+ copied += 1
+
+ await _copy(source_rel, target_rel, source_entries)
+ return copied
+
+
+class AgfsParseOutputStore(ParseOutputStore):
+ """Artifact store backed by the global VikingFS ``viking://temp`` space.
+
+ Every operation forwards to the VikingFS singleton exactly as parsers did
+ before this abstraction existed, so enabling it changes no behaviour.
+ """
+
+ backend = "agfs"
+
+ def __init__(self, viking_fs: Any = None, ctx: Any = None) -> None:
+ self._viking_fs = viking_fs
+ self._ctx = ctx
+ self._cleaned: set[str] = set()
+
+ def _fs(self) -> Any:
+ if self._viking_fs is not None:
+ return self._viking_fs
+ from openviking.storage.viking_fs import get_viking_fs
+
+ return get_viking_fs()
+
+ @staticmethod
+ def _resolve(ref: ParseArtifactRef, rel_path: str) -> str:
+ rel = (rel_path or "").strip("/")
+ if not rel:
+ return ref.root
+ return safe_join_viking_uri(ref.root, rel)
+
+ async def create_artifact(self, *, root_type: str = "dir") -> ParseArtifactRef:
+ if root_type not in _ROOT_TYPES:
+ raise ValueError(f"root_type must be one of {sorted(_ROOT_TYPES)}")
+ fs = self._fs()
+ root = fs.create_temp_uri(ctx=self._ctx) if self._ctx is not None else fs.create_temp_uri()
+ return ParseArtifactRef(backend=self.backend, root=root, root_type=root_type)
+
+ async def mkdir(self, ref: ParseArtifactRef, rel_path: str = "") -> None:
+ kwargs = {"ctx": self._ctx} if self._ctx is not None else {}
+ await self._fs().mkdir(self._resolve(ref, rel_path), exist_ok=True, **kwargs)
+
+ async def write_bytes(self, ref: ParseArtifactRef, rel_path: str, content: bytes) -> None:
+ fs = self._fs()
+ uri = self._resolve(ref, rel_path)
+ kwargs = {"ctx": self._ctx} if self._ctx is not None else {}
+ if hasattr(fs, "write_file_bytes"):
+ await fs.write_file_bytes(uri, content, **kwargs)
+ else:
+ await fs.write_file(uri, content.decode("utf-8"), **kwargs)
+
+ async def read_bytes(self, ref: ParseArtifactRef, rel_path: str) -> bytes:
+ fs = self._fs()
+ uri = self._resolve(ref, rel_path)
+ kwargs = {"ctx": self._ctx} if self._ctx is not None else {}
+ if hasattr(fs, "read_file_bytes"):
+ return await fs.read_file_bytes(uri, **kwargs)
+ content = await fs.read(uri, **kwargs)
+ return content.encode("utf-8") if isinstance(content, str) else content
+
+ async def list(self, ref: ParseArtifactRef, rel_path: str = "") -> List[ArtifactEntry]:
+ from openviking.storage.viking_fs import LS_ALL_NODES
+
+ base = self._resolve(ref, rel_path)
+ kwargs = {"ctx": self._ctx} if self._ctx is not None else {}
+ entries = await self._fs().ls(base, show_all_hidden=True, node_limit=LS_ALL_NODES, **kwargs)
+ rel_prefix = (rel_path or "").strip("/")
+ result: List[ArtifactEntry] = []
+ for entry in entries:
+ name = entry.get("name")
+ if not name or name in {".", ".."}:
+ continue
+ child_rel = f"{rel_prefix}/{name}" if rel_prefix else str(name)
+ result.append(
+ ArtifactEntry(
+ name=str(name),
+ rel_path=child_rel,
+ is_dir=bool(entry.get("isDir")),
+ )
+ )
+ return result
+
+ async def cleanup(self, ref: ParseArtifactRef) -> None:
+ if ref.root in self._cleaned:
+ return
+ self._cleaned.add(ref.root)
+ kwargs = {"ctx": self._ctx} if self._ctx is not None else {}
+ await self._fs().delete_temp(ref.root, **kwargs)
+
+ async def move_file(
+ self,
+ source_ref: ParseArtifactRef,
+ source_rel: str,
+ target_ref: ParseArtifactRef,
+ target_rel: str,
+ ) -> None:
+ kwargs = {"ctx": self._ctx} if self._ctx is not None else {}
+ await self._fs().move_file(
+ self._resolve(source_ref, source_rel),
+ self._resolve(target_ref, target_rel),
+ **kwargs,
+ )
+
+
+class LocalParseOutputStore(ParseOutputStore):
+ """Artifact store backed by a local directory tree.
+
+ Artifacts stay on the parsing worker until add-resources commits the required
+ bytes to the formal AGFS tree. They are not handed to asynchronous semantic
+ workers on the SemanticPlan path.
+
+ Case-only name collisions are detected explicitly rather than relying on the
+ host filesystem's case sensitivity, so behaviour matches across platforms.
+ """
+
+ backend = "local"
+
+ def __init__(self, local_root: str) -> None:
+ if not local_root:
+ raise ValueError("LocalParseOutputStore requires a local_root")
+ self._root = Path(local_root).expanduser().resolve()
+ self._root.mkdir(parents=True, exist_ok=True)
+
+ def _resolve(self, ref: ParseArtifactRef, rel_path: str) -> Path:
+ artifact_root = Path(ref.root).resolve()
+ # The artifact root must stay inside the configured store root.
+ if artifact_root != self._root and self._root not in artifact_root.parents:
+ raise ValueError("parse artifact root escapes the configured local_root")
+ rel = (rel_path or "").strip("/")
+ if not rel:
+ return artifact_root
+ safe_rel = sanitize_relative_viking_path(rel)
+ target = artifact_root.joinpath(*Path(safe_rel).parts).resolve()
+ if artifact_root != target and artifact_root not in target.parents:
+ raise ValueError(f"resolved path escapes the artifact root: {rel_path}")
+ return target
+
+ @staticmethod
+ def _check_case_conflict(target: Path) -> None:
+ parent = target.parent
+ if not parent.is_dir():
+ return
+ lower = target.name.casefold()
+ for existing in parent.iterdir():
+ if existing.name != target.name and existing.name.casefold() == lower:
+ raise ValueError(
+ f"case-only name conflict: {target.name} vs existing {existing.name}"
+ )
+
+ async def create_artifact(self, *, root_type: str = "dir") -> ParseArtifactRef:
+ if root_type not in _ROOT_TYPES:
+ raise ValueError(f"root_type must be one of {sorted(_ROOT_TYPES)}")
+ artifact_root = self._root / f"artifact-{uuid.uuid4().hex}"
+ await asyncio.to_thread(artifact_root.mkdir, parents=True, exist_ok=False)
+ return ParseArtifactRef(
+ backend=self.backend,
+ root=str(artifact_root),
+ root_type=root_type,
+ )
+
+ async def mkdir(self, ref: ParseArtifactRef, rel_path: str = "") -> None:
+ target = self._resolve(ref, rel_path)
+ await asyncio.to_thread(target.mkdir, parents=True, exist_ok=True)
+
+ async def write_bytes(self, ref: ParseArtifactRef, rel_path: str, content: bytes) -> None:
+ target = self._resolve(ref, rel_path)
+
+ def _write() -> None:
+ target.parent.mkdir(parents=True, exist_ok=True)
+ self._check_case_conflict(target)
+ target.write_bytes(content)
+
+ await asyncio.to_thread(_write)
+
+ async def read_bytes(self, ref: ParseArtifactRef, rel_path: str) -> bytes:
+ target = self._resolve(ref, rel_path)
+ return await asyncio.to_thread(target.read_bytes)
+
+ async def list(self, ref: ParseArtifactRef, rel_path: str = "") -> List[ArtifactEntry]:
+ base = self._resolve(ref, rel_path)
+
+ def _scan() -> List[ArtifactEntry]:
+ if not base.is_dir():
+ return []
+ rel_prefix = (rel_path or "").strip("/")
+ entries: List[ArtifactEntry] = []
+ for child in sorted(base.iterdir(), key=lambda p: p.name):
+ child_rel = f"{rel_prefix}/{child.name}" if rel_prefix else child.name
+ entries.append(
+ ArtifactEntry(
+ name=child.name,
+ rel_path=child_rel,
+ is_dir=child.is_dir(),
+ )
+ )
+ return entries
+
+ return await asyncio.to_thread(_scan)
+
+ async def cleanup(self, ref: ParseArtifactRef) -> None:
+ artifact_root = self._resolve(ref, "")
+ await asyncio.to_thread(shutil.rmtree, artifact_root, ignore_errors=True)
+
+ async def move_file(
+ self,
+ source_ref: ParseArtifactRef,
+ source_rel: str,
+ target_ref: ParseArtifactRef,
+ target_rel: str,
+ ) -> None:
+ source = self._resolve(source_ref, source_rel)
+ target = self._resolve(target_ref, target_rel)
+
+ def _move() -> None:
+ target.parent.mkdir(parents=True, exist_ok=True)
+ self._check_case_conflict(target)
+ shutil.move(str(source), str(target))
+
+ await asyncio.to_thread(_move)
+
+
+@dataclass(frozen=True)
+class ResolvedDocRoot:
+ """The single document root located inside a parse artifact.
+
+ ``doc_name`` is the original (un-sanitized) entry name; ``doc_rel`` is its
+ artifact-relative path (the file itself when flattened to a single file).
+ """
+
+ doc_name: str
+ doc_rel: str
+ root_is_file: bool
+
+
+async def resolve_artifact_doc_root(
+ store: ParseOutputStore,
+ ref: ParseArtifactRef,
+ *,
+ flatten_single_file: bool = False,
+) -> ResolvedDocRoot:
+ """Locate the single document root inside an artifact, backend-agnostically.
+
+ Parsers lay out both AGFS and local artifacts the same way — exactly one
+ document directory under the root — so this structural walk is shared. It
+ only reads the artifact via ``store``; resolving the final target URI against
+ the live resource tree is a separate, VikingFS-only concern handled by the
+ caller (TreeBuilder.resolve_target_uri).
+ """
+ top = [
+ entry
+ for entry in await store.list(ref, "")
+ if entry.name not in {".", "..", ARTIFACT_MANIFEST_NAME}
+ ]
+ doc_dirs = [e for e in top if e.is_dir]
+ if flatten_single_file and not doc_dirs:
+ files = [entry for entry in top if not entry.is_dir]
+ if len(files) == 1:
+ return ResolvedDocRoot(
+ doc_name=files[0].name,
+ doc_rel=files[0].rel_path,
+ root_is_file=True,
+ )
+ if len(doc_dirs) != 1:
+ raise ValueError(
+ f"expected exactly 1 document directory in artifact {ref.root}, found {len(doc_dirs)}"
+ )
+
+ doc_entry = doc_dirs[0]
+ doc_name = doc_entry.name
+ doc_rel = doc_entry.rel_path
+ root_is_file = False
+
+ if flatten_single_file:
+ children = [
+ entry
+ for entry in await store.list(ref, doc_rel)
+ if entry.name not in {".", "..", ARTIFACT_MANIFEST_NAME}
+ ]
+ if len(children) == 1 and not children[0].is_dir:
+ doc_name = children[0].name
+ doc_rel = children[0].rel_path
+ root_is_file = True
+
+ return ResolvedDocRoot(doc_name=doc_name, doc_rel=doc_rel, root_is_file=root_is_file)
+
+
+def build_parse_output_store(
+ *,
+ viking_fs: Any = None,
+ backend: Optional[Literal["agfs", "local"]] = None,
+ local_root: Optional[str] = None,
+) -> ParseOutputStore:
+ """Return the configured artifact store.
+
+ ``backend`` defaults to AGFS. The local backend requires ``local_root``.
+ """
+ if backend in (None, "agfs"):
+ return AgfsParseOutputStore(viking_fs=viking_fs)
+ if backend == "local":
+ if not local_root:
+ raise ValueError("local parse output backend requires storage.parse_output.local_root")
+ return LocalParseOutputStore(local_root=local_root)
+ raise ValueError(f"unsupported parse output backend: {backend}")
+
+
+def store_for_artifact_ref(
+ ref: ParseArtifactRef,
+ *,
+ viking_fs: Any = None,
+ ctx: Any = None,
+) -> ParseOutputStore:
+ """Return the store that owns ``ref``, resolving the local root from config.
+
+ Callers that receive a serialized ref across the queue (semantic worker,
+ add-resource cleanup, post-process) must reconstruct the matching backend;
+ this centralizes that mapping so the local-root lookup lives in one place.
+ """
+ if ref.backend == "local":
+ from openviking_cli.utils.config import get_openviking_config
+
+ parse_output = get_openviking_config().storage.parse_output
+ return build_parse_output_store(
+ backend="local", local_root=parse_output.resolved_local_root()
+ )
+ return AgfsParseOutputStore(viking_fs=viking_fs, ctx=ctx)
+
+
+async def read_artifact_manifest(store: ParseOutputStore, ref: ParseArtifactRef) -> Dict[str, str]:
+ """Return the md5 manifest (artifact-relative path -> md5), or empty if absent.
+
+ A missing/unreadable/malformed manifest yields ``{}`` so the incremental diff
+ falls back to comparing file bytes instead of assuming equality.
+ """
+ try:
+ raw = await store.read_bytes(ref, ARTIFACT_MANIFEST_NAME)
+ loaded = json.loads(raw.decode("utf-8"))
+ except Exception:
+ return {}
+ if not isinstance(loaded, dict):
+ return {}
+ return {str(key): str(value) for key, value in loaded.items()}
+
+
+async def write_artifact_manifest(
+ store: ParseOutputStore, ref: ParseArtifactRef, md5_by_rel: Dict[str, str]
+) -> None:
+ """Persist the md5 manifest at the artifact root using the shared encoding."""
+ await store.write_bytes(
+ ref,
+ ARTIFACT_MANIFEST_NAME,
+ json.dumps(md5_by_rel, ensure_ascii=False, sort_keys=True).encode("utf-8"),
+ )
+
+
+# Re-exported so callers importing the sanitizer alongside the store stay in one
+# module; keeps parser edits from sprinkling path_safety imports everywhere.
+__all__ = [
+ "ARTIFACT_MANIFEST_NAME",
+ "ArtifactEntry",
+ "AgfsParseOutputStore",
+ "LocalParseOutputStore",
+ "ParseArtifactRef",
+ "ParseArtifactWriter",
+ "ParseOutputStore",
+ "ResolvedDocRoot",
+ "build_parse_output_store",
+ "copy_artifact_tree",
+ "create_parse_artifact_writer",
+ "read_artifact_manifest",
+ "resolve_artifact_doc_root",
+ "sanitize_relative_viking_path",
+ "store_for_artifact_ref",
+ "write_artifact_manifest",
+]
diff --git a/openviking/parse/parser_router.py b/openviking/parse/parser_router.py
index 69512653d..308d8e81d 100644
--- a/openviking/parse/parser_router.py
+++ b/openviking/parse/parser_router.py
@@ -136,9 +136,11 @@ class ParserRouter:
if use_understanding:
if isinstance(source, LocalResource):
- kwargs["source_name"] = source.meta["resolved_name"]
+ kwargs["source_name"] = source.meta.get("resolved_name") or Path(source_path).name
kwargs["resolved_extension"] = (
- kwargs.get("resolved_extension") or source.meta["resolved_extension"]
+ kwargs.get("resolved_extension")
+ or source.meta.get("resolved_extension")
+ or Path(source_path).suffix
)
display = source_path
if isinstance(source_path, str) and source_path.startswith(("http://", "https://")):
diff --git a/openviking/parse/parsers/base_parser.py b/openviking/parse/parsers/base_parser.py
index 85911c2b1..a8d67ca00 100644
--- a/openviking/parse/parsers/base_parser.py
+++ b/openviking/parse/parsers/base_parser.py
@@ -109,15 +109,3 @@ class BaseParser(ABC):
from openviking.storage.viking_fs import get_viking_fs
return get_viking_fs()
-
- def _create_temp_uri(self) -> str:
- """
- Create a temporary URI for storing intermediate files during parsing.
-
- This is a common utility method for all parsers that follow the
- three-phase parsing architecture.
-
- Returns:
- Temporary URI string (e.g., "viking://temp/abc12345")
- """
- return self._get_viking_fs().create_temp_uri()
diff --git a/openviking/parse/parsers/code/code.py b/openviking/parse/parsers/code/code.py
index 4890cc285..be4c7f307 100644
--- a/openviking/parse/parsers/code/code.py
+++ b/openviking/parse/parsers/code/code.py
@@ -125,6 +125,15 @@ class CodeRepositoryParser(BaseParser):
"""
start_time = time.time()
source_path = Path(source)
+ # Always work through a ParseOutputStore. Local mode threads one in via
+ # kwargs; otherwise default to the AGFS-backed store so the parser has a
+ # single, backend-agnostic write path (no viking_fs/store branching).
+ output_store = kwargs.get("parse_output_store")
+ if output_store is None:
+ from openviking.parse.output import AgfsParseOutputStore
+
+ output_store = AgfsParseOutputStore(viking_fs=self._get_viking_fs())
+ artifact_ref = None
# Check if source is already a local directory (should always be true)
if not source_path.is_dir():
@@ -160,24 +169,31 @@ class CodeRepositoryParser(BaseParser):
local_dir = source_path
- # 3. Create VikingFS temp URI
- viking_fs = self._get_viking_fs()
- temp_viking_uri = self._create_temp_uri()
- # The structure in temp should be: viking://temp/{uuid}/repository/...
- # Use simple name 'repository' for temp, TreeBuilder will rename it to org/repo later
+ # 3. Allocate the artifact root through the store. Local mode writes
+ # to a shared local dir; AGFS mode writes to viking://temp. Either way
+ # the layout is <root>/repository/... and the parser code is identical.
+ artifact_ref = await output_store.create_artifact(root_type="dir")
+ temp_viking_uri = artifact_ref.root
target_root_uri = f"{temp_viking_uri}/repository"
- logger.info(f"Uploading to VikingFS: {target_root_uri}")
+ logger.debug(f"Uploading code repository artifacts to: {target_root_uri}")
- # 4. Upload to VikingFS (filtering on the fly)
- file_count = await self._upload_directory(
+ # 4. Upload to the artifact store (filtering on the fly)
+ file_count, upload_failures = await self._upload_directory(
local_dir,
- target_root_uri,
- viking_fs,
+ output_store=output_store,
+ artifact_ref=artifact_ref,
ignore_dirs=kwargs.get("ignore_dirs"),
include=kwargs.get("include"),
exclude=kwargs.get("exclude"),
)
+ if upload_failures:
+ detail = "; ".join(upload_failures[:5])
+ if len(upload_failures) > 5:
+ detail += f"; ... {len(upload_failures) - 5} more"
+ raise RuntimeError(
+ f"Failed to upload {len(upload_failures)} repository file(s): {detail}"
+ )
logger.info(f"Uploaded {file_count} files to {target_root_uri}")
@@ -209,6 +225,7 @@ class CodeRepositoryParser(BaseParser):
parse_time=time.time() - start_time,
)
result.temp_dir_path = temp_viking_uri # Points to parent of repo_name
+ result.artifact_ref = artifact_ref
result.meta["file_count"] = file_count
result.meta["repo_name"] = repo_name
if branch:
@@ -220,6 +237,15 @@ class CodeRepositoryParser(BaseParser):
except Exception as e:
logger.error(f"Failed to parse repository {source}: {e}", exc_info=True)
+ if artifact_ref is not None:
+ try:
+ await output_store.cleanup(artifact_ref)
+ except Exception as cleanup_error:
+ logger.warning(
+ "Failed to clean repository parse artifact %s: %s",
+ artifact_ref.root,
+ cleanup_error,
+ )
# Use original URL for error case as well - still important for TreeBuilder
# Even on failure, we want TreeBuilder to potentially get org/repo from the URL
original_source = kwargs.get("original_source") or source_meta.get("original_source")
@@ -582,20 +608,20 @@ class CodeRepositoryParser(BaseParser):
async def _upload_directory(
self,
local_dir: Path,
- viking_uri_base: str,
- viking_fs: Any,
*,
+ output_store: Any,
+ artifact_ref: Any,
ignore_dirs: Optional[Union[Set[str], List[str], str]] = None,
include: Optional[str] = None,
exclude: Optional[str] = None,
- ) -> int:
- """Recursively upload directory to VikingFS using shared upload utilities."""
- count, _ = await upload_directory(
+ ) -> Tuple[int, List[str]]:
+ """Recursively upload the repository into the artifact store under repository/."""
+ return await upload_directory(
local_dir,
- viking_uri_base,
- viking_fs,
+ "repository",
+ store=output_store,
+ artifact_ref=artifact_ref,
ignore_dirs=ignore_dirs,
include=include,
exclude=exclude,
)
- return count
diff --git a/openviking/parse/parsers/directory.py b/openviking/parse/parsers/directory.py
index d8abdfe28..886b8108a 100644
--- a/openviking/parse/parsers/directory.py
+++ b/openviking/parse/parsers/directory.py
@@ -32,9 +32,15 @@ from openviking.parse.base import (
)
from openviking.parse.gitignore import GitignoreMatcher
from openviking.parse.image_rewrite import IMAGE_MAPPINGS_FILENAME
+from openviking.parse.output import (
+ ParseArtifactWriter,
+ copy_artifact_tree,
+ create_parse_artifact_writer,
+ store_for_artifact_ref,
+)
from openviking.parse.parsers.base_parser import BaseParser
from openviking.parse.parsers.media.constants import MEDIA_EXTENSIONS
-from openviking.parse.parsers.upload_utils import is_text_file
+from openviking.parse.parsers.upload_utils import detect_and_convert_encoding, is_text_file
from openviking.storage.viking_fs import LS_ALL_NODES
from openviking_cli.exceptions import InvalidArgumentError
from openviking_cli.utils.logger import get_logger
@@ -146,7 +152,9 @@ class DirectoryParser(BaseParser):
warnings: List[str] = []
temp_uri: Optional[str] = None
keep_temp = False
- pending_temp_uris: set[str] = set()
+ pending_results: list[ParseResult] = []
+ output_store = kwargs.get("parse_output_store")
+ writer: Optional[ParseArtifactWriter] = None
try:
# ── Phase 1: scan directory ───────────────────────────────
@@ -309,11 +317,13 @@ class DirectoryParser(BaseParser):
if use_understanding:
understanding_jobs.append(job)
- viking_fs = self._get_viking_fs()
- temp_uri = self._create_temp_uri()
+ viking_fs = self._get_viking_fs() if output_store is None else None
+ writer = await create_parse_artifact_writer(output_store, viking_fs=viking_fs)
+ output_store = writer.store
+ temp_uri = writer.ref.root
target_uri = f"{temp_uri}/{dir_name}"
- await viking_fs.mkdir(temp_uri, exist_ok=True)
- await viking_fs.mkdir(target_uri, exist_ok=True)
+ await writer.mkdir()
+ await writer.mkdir(dir_name)
if not processable_files:
root = ResourceNode(
@@ -330,6 +340,7 @@ class DirectoryParser(BaseParser):
warnings=warnings,
)
result.temp_dir_path = temp_uri
+ result.artifact_ref = await writer.finalize(resource_rel=dir_name)
result.meta["file_count"] = 0
result.meta["dir_name"] = dir_name
result.meta["total_processable"] = 0
@@ -363,9 +374,10 @@ class DirectoryParser(BaseParser):
split_content=split_content,
max_concurrent=directory_config.max_concurrent,
job_timeout=job_timeout,
+ output_store=output_store,
)
- pending_temp_uris.update(
- parsed["result"].temp_dir_path
+ pending_results.extend(
+ parsed["result"]
for parsed in understanding_results.values()
if parsed.get("result") and parsed["result"].temp_dir_path
)
@@ -391,12 +403,14 @@ class DirectoryParser(BaseParser):
else:
try:
# The merge helper owns this artifact once merging starts.
- pending_temp_uris.discard(sub_result.temp_dir_path)
+ if sub_result in pending_results:
+ pending_results.remove(sub_result)
await self._merge_parser_result(
cf,
sub_result,
target_uri,
viking_fs,
+ target_writer=writer,
preserve_structure=preserve_structure,
split_content=split_content,
)
@@ -425,6 +439,7 @@ class DirectoryParser(BaseParser):
target_uri,
viking_fs,
warnings,
+ target_writer=writer,
preserve_structure=preserve_structure,
)
parser_name = "direct_upload"
@@ -435,6 +450,8 @@ class DirectoryParser(BaseParser):
target_uri,
viking_fs,
warnings,
+ target_writer=writer,
+ output_store=output_store,
preserve_structure=preserve_structure,
import_root=str(source_path),
split_content=split_content,
@@ -488,6 +505,7 @@ class DirectoryParser(BaseParser):
warnings=warnings,
)
result.temp_dir_path = temp_uri
+ result.artifact_ref = await writer.finalize(resource_rel=dir_name)
result.meta["file_count"] = file_count
result.meta["dir_name"] = dir_name
result.meta["total_processable"] = len(processable_files)
@@ -515,15 +533,28 @@ class DirectoryParser(BaseParser):
warnings=[f"Failed to parse directory: {exc}"],
)
finally:
- if temp_uri and not keep_temp:
- pending_temp_uris.add(temp_uri)
- for pending_uri in pending_temp_uris:
+ if writer is not None and not keep_temp:
+ await writer.cleanup()
+ for pending_result in pending_results:
try:
- await viking_fs.delete_temp(pending_uri)
+ pending_ref = getattr(pending_result, "artifact_ref", None)
+ if pending_ref is not None:
+ pending_store = (
+ output_store
+ if output_store is not None
+ and output_store.backend == pending_ref.backend
+ else store_for_artifact_ref(
+ pending_ref,
+ viking_fs=self._get_viking_fs(),
+ )
+ )
+ await pending_store.cleanup(pending_ref)
+ elif pending_result.temp_dir_path:
+ await self._get_viking_fs().delete_temp(pending_result.temp_dir_path)
except Exception as exc:
logger.warning(
"[DirectoryParser] Failed to clean temporary artifact %s: %s",
- pending_uri,
+ pending_result.temp_dir_path,
exc,
)
@@ -745,6 +776,7 @@ class DirectoryParser(BaseParser):
split_content: bool,
max_concurrent: int,
job_timeout: float,
+ output_store: Any = None,
) -> Dict[int, Dict[str, Any]]:
"""Parse jobs with a fixed local pool and a shared service-loop limit."""
limiter = _get_understanding_limiter(max_concurrent)
@@ -770,6 +802,7 @@ class DirectoryParser(BaseParser):
import_root=import_root,
split_content=split_content,
parse_options=job.get("parse_options"),
+ output_store=output_store,
)
sub_result = await asyncio.wait_for(parse_coro, timeout=job_timeout)
results[job["index"]] = {"result": sub_result, "error": None}
@@ -806,9 +839,12 @@ class DirectoryParser(BaseParser):
import_root: Optional[str],
split_content: bool,
parse_options: Optional[Dict[str, Any]] = None,
+ output_store: Any = None,
) -> ParseResult:
"""Run one parser without mutating the directory destination tree."""
options = dict(parse_options or {})
+ if output_store is not None:
+ options["parse_output_store"] = output_store
source = options.pop("_source", str(classified_file.path))
options.update(
enable_link_rewrite=preserve_structure,
@@ -826,6 +862,7 @@ class DirectoryParser(BaseParser):
target_uri: str,
viking_fs: Any,
*,
+ target_writer: Optional[ParseArtifactWriter] = None,
preserve_structure: bool,
split_content: bool,
) -> None:
@@ -841,6 +878,30 @@ class DirectoryParser(BaseParser):
dest = f"{target_uri}/{parent}" if parent != "." else target_uri
else:
dest = target_uri
+ source_ref = getattr(sub_result, "artifact_ref", None)
+ if target_writer is not None and source_ref is not None:
+ source_store = (
+ target_writer.store
+ if source_ref.backend == target_writer.store.backend
+ else store_for_artifact_ref(source_ref, viking_fs=viking_fs)
+ )
+ target_rel = target_writer.relative_path(dest)
+ try:
+ merged = await copy_artifact_tree(
+ source_store=source_store,
+ source_ref=source_ref,
+ source_rel="",
+ target=target_writer,
+ target_rel=target_rel,
+ allowed_hidden=_MERGE_SIDECAR_ALLOWLIST,
+ flatten_single_output=bool(not split_content and preserve_structure),
+ )
+ finally:
+ await source_store.cleanup(source_ref)
+ if not merged:
+ raise ValueError(no_content_error)
+ return
+
try:
merged = await DirectoryParser._merge_temp(
viking_fs,
@@ -868,6 +929,8 @@ class DirectoryParser(BaseParser):
target_uri: str,
viking_fs: Any,
warnings: List[str],
+ target_writer: Optional[ParseArtifactWriter] = None,
+ output_store: Any = None,
preserve_structure: bool = True,
import_root: Optional[str] = None,
split_content: bool = True,
@@ -898,12 +961,14 @@ class DirectoryParser(BaseParser):
preserve_structure=preserve_structure,
import_root=import_root,
split_content=split_content,
+ output_store=output_store,
)
await DirectoryParser._merge_parser_result(
classified_file,
sub_result,
target_uri,
viking_fs,
+ target_writer=target_writer,
preserve_structure=preserve_structure,
split_content=split_content,
)
@@ -922,12 +987,15 @@ class DirectoryParser(BaseParser):
}
else:
try:
- content = src_file.read_bytes()
+ content = detect_and_convert_encoding(src_file.read_bytes(), src_file)
if preserve_structure:
dst_uri = f"{target_uri}/{rel_path}"
else:
dst_uri = f"{target_uri}/{PurePosixPath(rel_path).name}"
- await viking_fs.write_file(dst_uri, content)
+ if target_writer is not None:
+ await target_writer.write_bytes(dst_uri, content)
+ else:
+ await viking_fs.write_file(dst_uri, content)
return {"ok": True, "meta": {}, "error": None}
except Exception as exc:
warnings.append(f"Failed to upload {rel_path}: {exc}")
@@ -939,6 +1007,7 @@ class DirectoryParser(BaseParser):
target_uri: str,
viking_fs: Any,
warnings: List[str],
+ target_writer: Optional[ParseArtifactWriter] = None,
preserve_structure: bool = True,
) -> Dict[str, Any]:
"""Directly upload a file without using its parser.
@@ -957,12 +1026,15 @@ class DirectoryParser(BaseParser):
src_file = classified_file.path
try:
- content = src_file.read_bytes()
+ content = detect_and_convert_encoding(src_file.read_bytes(), src_file)
if preserve_structure:
dst_uri = f"{target_uri}/{rel_path}"
else:
dst_uri = f"{target_uri}/{PurePosixPath(rel_path).name}"
- await viking_fs.write_file(dst_uri, content)
+ if target_writer is not None:
+ await target_writer.write_bytes(dst_uri, content)
+ else:
+ await viking_fs.write_file(dst_uri, content)
return {"ok": True, "meta": {}, "error": None}
except Exception as exc:
warnings.append(f"Failed to upload {rel_path}: {exc}")
diff --git a/openviking/parse/parsers/markdown.py b/openviking/parse/parsers/markdown.py
index 8f269e57b..e78b17e14 100644
--- a/openviking/parse/parsers/markdown.py
+++ b/openviking/parse/parsers/markdown.py
@@ -29,6 +29,12 @@ from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
from openviking.parse.accessors.mime_types import IANA_MEDIA_TYPE_TO_EXTENSION
from openviking.parse.base import NodeType, ParseResult, ResourceNode, create_parse_result
from openviking.parse.image_validation import is_valid_image
+from openviking.parse.output import (
+ AgfsParseOutputStore,
+ ParseArtifactRef,
+ ParseArtifactWriter,
+ create_parse_artifact_writer,
+)
from openviking.parse.parsers.base_parser import BaseParser
from openviking.parse.parsers.code.ast.providers import supports_code_skeleton
from openviking.parse.parsers.constants import (
@@ -289,6 +295,10 @@ class MarkdownParser(BaseParser):
ParseResult with temp_dir_path (Viking URI)
"""
start_time = time.time()
+ output_store = kwargs.pop("parse_output_store", None)
+ writer = await create_parse_artifact_writer(
+ output_store, viking_fs=self._get_viking_fs() if output_store is None else None
+ )
try:
split_content = bool(kwargs.get("split_content", True))
@@ -305,7 +315,7 @@ class MarkdownParser(BaseParser):
# Phase 1 — parse only: turn the markdown into an ordered VikingFS write
# plan, touching nothing. The temp URI is allocated here (the one
# FS-scoped step) and threaded in so layout planning stays side-effect free.
- temp_uri = self._create_temp_uri()
+ temp_uri = writer.ref.root
layout_kwargs = dict(kwargs)
layout_kwargs["flatten_single_output"] = flatten_single_output
layout = await self._compute_layout(
@@ -335,6 +345,7 @@ class MarkdownParser(BaseParser):
# rewriting links and ingesting local images.
await self._apply_layout(
layout,
+ writer=writer,
rewrite_ctx=rewrite_ctx,
base_dir=base_dir,
allowed_media_dirs=allowed_media_dirs,
@@ -360,12 +371,15 @@ class MarkdownParser(BaseParser):
warnings=layout.warnings,
)
- result.temp_dir_path = layout.temp_uri
+ root_rel = writer.relative_path(layout.root_dir)
+ result.artifact_ref = await writer.finalize(resource_rel=root_rel)
+ result.temp_dir_path = result.artifact_ref.root
return result
except Exception as e:
logger.error(f"[MarkdownParser] Parse failed: {e}", exc_info=True)
+ await writer.cleanup()
raise
async def _compute_layout(
@@ -454,6 +468,7 @@ class MarkdownParser(BaseParser):
self,
layout: _Layout,
*,
+ writer: Optional[ParseArtifactWriter] = None,
rewrite_ctx: Optional[_RewriteContext] = None,
base_dir: Optional[Path] = None,
allowed_media_dirs: Optional[List[Path]] = None,
@@ -462,7 +477,11 @@ class MarkdownParser(BaseParser):
create dirs, write each section (rewriting relative links when enabled), then
ingest the local images those sections reference."""
started = time.perf_counter()
- viking_fs = self._get_viking_fs()
+ if writer is None:
+ writer = ParseArtifactWriter(
+ AgfsParseOutputStore(viking_fs=self._get_viking_fs()),
+ ParseArtifactRef(backend="agfs", root=layout.temp_uri),
+ )
mkdir_ops = [op for op in layout.ops if op.kind == "mkdir"]
write_ops = [op for op in layout.ops if op.kind == "write"]
write_chars = sum(len(op.content or "") for op in write_ops)
@@ -473,13 +492,17 @@ class MarkdownParser(BaseParser):
mkdir_count = 0
for op in layout.ops:
if op.kind == "mkdir":
- await viking_fs.mkdir(op.uri, exist_ok=op.exist_ok)
+ await writer.mkdir(op.uri)
mkdir_count += 1
mkdir_s = time.perf_counter() - mkdir_started
write_started = time.perf_counter()
+ written_sections: list[tuple[str, str]] = []
for op in write_ops:
- await self._write_section(op.uri, op.content, rewrite_ctx=rewrite_ctx)
+ written = await self._write_section(
+ op.uri, op.content, writer=writer, rewrite_ctx=rewrite_ctx
+ )
+ written_sections.append((op.uri, written))
write_s = time.perf_counter() - write_started
# Ingest local image files, placing each image next to the markdown file
@@ -488,7 +511,13 @@ class MarkdownParser(BaseParser):
images_started = time.perf_counter()
has_image_refs = self._layout_has_local_image_refs(write_ops)
if has_image_refs:
- await self._ingest_local_images(layout.root_dir, base_dir, allowed_media_dirs)
+ await self._ingest_local_images(
+ layout.root_dir,
+ base_dir,
+ allowed_media_dirs,
+ writer=writer,
+ markdown_files=written_sections,
+ )
images_s = time.perf_counter() - images_started
total_s = time.perf_counter() - started
@@ -804,6 +833,9 @@ class MarkdownParser(BaseParser):
root_dir: str,
base_dir: Optional[Path] = None,
allowed_media_dirs: Optional[List[Path]] = None,
+ *,
+ writer: Optional[ParseArtifactWriter] = None,
+ markdown_files: Optional[List[Tuple[str, str]]] = None,
) -> None:
"""
Scan every processed markdown file under ``root_dir`` and copy the local
@@ -821,26 +853,20 @@ class MarkdownParser(BaseParser):
allowed_media_dirs: Additional directories from which derived media
may be read (passed through to ``_resolve_image_path``)
"""
- viking_fs = self._get_viking_fs()
-
- # Find all processed markdown files under the root directory
- glob_result = await viking_fs.glob("**/*.md", uri=root_dir)
- md_uris = glob_result.get("matches", [])
- if not md_uris:
- return
+ if writer is None:
+ writer = ParseArtifactWriter(
+ AgfsParseOutputStore(viking_fs=self._get_viking_fs()),
+ ParseArtifactRef(backend="agfs", root=root_dir),
+ )
+ if markdown_files is None:
+ raise ValueError("markdown_files are required for artifact image ingestion")
root_prefix = root_dir.rstrip("/")
# mapping: rel_md_path -> {original_path_str -> unique_filename}
mappings: Dict[str, Dict[str, str]] = {}
- for md_uri in md_uris:
- try:
- content = await viking_fs.read_file(md_uri)
- except Exception:
- logger.warning(f"[MarkdownParser] Failed to read markdown file: {md_uri}")
- continue
-
+ for md_uri, content in markdown_files:
# Collect all image references in this markdown file: markdown
# embeds (![...]) and HTML <img src="..."> tags alike.
from openviking.parse.image_rewrite import HTML_IMG_PATTERN
@@ -903,7 +929,7 @@ class MarkdownParser(BaseParser):
# Write next to the markdown file
viking_path = f"{md_dir}/{unique_filename}"
- await viking_fs.write_file_bytes(viking_path, image_bytes)
+ await writer.write_bytes(viking_path, image_bytes)
# Record mapping for post-commit rewrite
file_mappings[origin_link] = unique_filename
@@ -923,7 +949,7 @@ class MarkdownParser(BaseParser):
from openviking.parse.image_rewrite import IMAGE_MAPPINGS_FILENAME
- await viking_fs.write_file(
+ await writer.write_text(
f"{root_prefix}/{IMAGE_MAPPINGS_FILENAME}",
json.dumps(mappings, ensure_ascii=False),
)
@@ -1341,8 +1367,9 @@ class MarkdownParser(BaseParser):
uri: str,
content: str,
*,
+ writer: Optional[ParseArtifactWriter] = None,
rewrite_ctx: Optional[_RewriteContext] = None,
- ) -> None:
+ ) -> str:
"""Write a markdown section file, rewriting relative links when enabled."""
if rewrite_ctx is not None:
content = await self._rewrite_relative_links(
@@ -1350,7 +1377,13 @@ class MarkdownParser(BaseParser):
section_subpath=self._section_subpath(uri, rewrite_ctx.root_dir),
rewrite_ctx=rewrite_ctx,
)
- await self._get_viking_fs().write_file(uri, content)
+ if writer is None:
+ writer = ParseArtifactWriter(
+ AgfsParseOutputStore(viking_fs=self._get_viking_fs()),
+ ParseArtifactRef(backend="agfs", root=uri.rsplit("/", 1)[0]),
+ )
+ await writer.write_text(uri, content)
+ return content
# ========== New Parsing Logic (v5.0) ==========
diff --git a/openviking/parse/parsers/media/audio.py b/openviking/parse/parsers/media/audio.py
index de0b75a82..3b20dd1fb 100644
--- a/openviking/parse/parsers/media/audio.py
+++ b/openviking/parse/parsers/media/audio.py
@@ -31,6 +31,7 @@ from pathlib import Path
from typing import List, Optional, Union
from openviking.parse.base import NodeType, ParseResult, ResourceNode
+from openviking.parse.output import create_parse_artifact_writer
from openviking.parse.parsers.base_parser import BaseParser
from openviking.parse.parsers.media.constants import AUDIO_EXTENSIONS
from openviking.parse.parsers.media.naming import resolve_media_names
@@ -75,16 +76,11 @@ class AudioParser(BaseParser):
FileNotFoundError: If source file does not exist
IOError: If audio processing fails
"""
- from openviking.storage.viking_fs import get_viking_fs
-
# Convert to Path object
file_path = Path(source) if isinstance(source, str) else source
if not file_path.exists():
raise FileNotFoundError(f"Audio file not found: {source}")
- viking_fs = get_viking_fs()
- temp_uri = viking_fs.create_temp_uri()
-
# Phase 1: Generate temporary files
audio_bytes = file_path.read_bytes()
ext = file_path.suffix
@@ -98,12 +94,6 @@ class AudioParser(BaseParser):
# Root directory name: filename stem + _ + extension (without dot)
ext_no_dot = ext[1:] if ext else ""
root_dir_name = VikingURI.sanitize_segment(f"{stem}_{ext_no_dot}")
- root_dir_uri = f"{temp_uri}/{root_dir_name}"
- await viking_fs.mkdir(root_dir_uri, exist_ok=True)
-
- # 1.1 Save original audio with original filename (sanitized)
- await viking_fs.write_file_bytes(f"{root_dir_uri}/{original_filename}", audio_bytes)
-
# 1.2 Validate audio file using magic bytes
# Define magic bytes for supported audio formats
audio_magic_bytes = {
@@ -131,6 +121,18 @@ class AudioParser(BaseParser):
f"Invalid audio file: {file_path}. File signature does not match expected format {ext_lower}"
)
+ writer = await create_parse_artifact_writer(
+ kwargs.get("parse_output_store"),
+ viking_fs=self._get_viking_fs() if kwargs.get("parse_output_store") is None else None,
+ )
+ try:
+ await writer.mkdir(root_dir_name)
+ await writer.write_bytes(f"{root_dir_name}/{original_filename}", audio_bytes)
+ artifact_ref = await writer.finalize(resource_rel=root_dir_name)
+ except BaseException:
+ await writer.cleanup()
+ raise
+
# Extract audio metadata (placeholder)
duration = 0
sample_rate = 0
@@ -161,7 +163,8 @@ class AudioParser(BaseParser):
return ParseResult(
root=root_node,
source_path=str(file_path),
- temp_dir_path=temp_uri,
+ temp_dir_path=artifact_ref.root,
+ artifact_ref=artifact_ref,
source_format="audio",
parser_name="AudioParser",
meta={"content_type": "audio", "format": format_str.lower()},
diff --git a/openviking/parse/parsers/media/image.py b/openviking/parse/parsers/media/image.py
index 32a267f01..f64c14cd1 100644
--- a/openviking/parse/parsers/media/image.py
+++ b/openviking/parse/parsers/media/image.py
@@ -17,16 +17,15 @@ from typing import List, Optional, Union
from PIL import Image
from openviking.parse.base import NodeType, ParseResult, ResourceNode
+from openviking.parse.output import create_parse_artifact_writer
from openviking.parse.parsers.base_parser import BaseParser
from openviking.parse.parsers.media.constants import IMAGE_EXTENSIONS
from openviking.parse.parsers.media.large_image_processor import (
- LargeImageResult,
process_large_image,
save_image_to_bytes,
)
from openviking.parse.parsers.media.naming import resolve_media_names
from openviking.parse.parsers.media.utils import _convert_svg_to_png
-from openviking.prompts import render_prompt
from openviking.storage.viking_fs import get_viking_fs
from openviking_cli.utils.config.parser_config import ImageConfig
from openviking_cli.utils.logger import get_logger
@@ -104,20 +103,19 @@ class ImageParser(BaseParser):
if not file_path.exists():
raise FileNotFoundError(f"Image file not found: {source}")
- viking_fs = get_viking_fs()
- temp_uri = viking_fs.create_temp_uri()
-
# Load image (SVG is converted to PNG first since PIL doesn't support it)
# For other formats not natively supported by VLM/embedding, convert to PNG
# to ensure end-to-end compatibility with image understanding and vectorization
try:
file_bytes = file_path.read_bytes()
- if file_path.suffix.lower() == ".svg" or file_bytes[:4] == b"<svg" or (file_bytes[:5] == b"<?xml" and b"<svg" in file_bytes[:100]):
+ if (
+ file_path.suffix.lower() == ".svg"
+ or file_bytes[:4] == b"<svg"
+ or (file_bytes[:5] == b"<?xml" and b"<svg" in file_bytes[:100])
+ ):
png_bytes = _convert_svg_to_png(file_bytes)
if png_bytes is None:
- raise ValueError(
- f"SVG files require cairosvg. Install it: pip install cairosvg"
- )
+ raise ValueError("SVG files require cairosvg. Install it: pip install cairosvg")
img = Image.open(io.BytesIO(png_bytes))
img.verify()
img.close()
@@ -128,7 +126,14 @@ class ImageParser(BaseParser):
else:
# Check if extension is in VLM-natively supported list
suffix_lower = file_path.suffix.lower()
- supported_by_vlm = suffix_lower in {".png", ".jpg", ".jpeg", ".gif", ".bmp", ".webp"}
+ supported_by_vlm = suffix_lower in {
+ ".png",
+ ".jpg",
+ ".jpeg",
+ ".gif",
+ ".bmp",
+ ".webp",
+ }
if supported_by_vlm:
img = Image.open(file_path)
img.verify() # Verify that it's a valid image
@@ -148,7 +153,9 @@ class ImageParser(BaseParser):
raise ValueError(f"Invalid image file: {file_path}. Error: {e}") from e
# Resolve names
- display_stem, stem, original_filename = resolve_media_names(file_path, file_path.suffix, **kwargs)
+ display_stem, stem, original_filename = resolve_media_names(
+ file_path, file_path.suffix, **kwargs
+ )
ext_no_dot = file_path.suffix[1:] if file_path.suffix else "jpg"
if needs_png_conversion:
# Convert to PNG and change extension for VLM/embedding compatibility
@@ -157,98 +164,109 @@ class ImageParser(BaseParser):
format_str = "png"
root_dir_name = VikingURI.sanitize_segment(f"{stem}_{ext_no_dot}")
- root_dir_uri = f"{temp_uri}/{root_dir_name}"
- await viking_fs.mkdir(root_dir_uri, exist_ok=True)
-
- # Process the image (check if large)
- large_image_result = process_large_image(file_path, img, filename_prefix=stem, config=self.config)
- img.close()
-
- # Create root node with metadata
- root_node = ResourceNode(
- type=NodeType.ROOT,
- title=display_stem,
- level=0,
- detail_file=None,
- content_path=None,
- children=[],
- meta={
- "width": width,
- "height": height,
- "format": format_str.lower(),
- "content_type": "image",
- "source_title": display_stem,
- "semantic_name": display_stem,
- "original_filename": original_filename,
- "is_large_image": large_image_result.needs_processing,
- },
+ output_store = kwargs.get("parse_output_store")
+ writer = await create_parse_artifact_writer(
+ output_store, viking_fs=get_viking_fs() if output_store is None else None
)
+ try:
+ await writer.mkdir(root_dir_name)
- if large_image_result.needs_processing:
- # Large image processing mode: save preview, grid, tiles
- # NOTE: The original file is intentionally NOT saved. Large images can be
- # hundreds of MB; storing the full-resolution original would exceed storage
- # budgets. The preview + tiles are the canonical representation. The
- # `original_filename` metadata field records the original name for reference
- # but the corresponding file is not persisted.
- logger.info(f"Processing large image {original_filename}: {width}x{height}")
-
- # Save low-res preview (original is too large to store)
- await viking_fs.write_file_bytes(
- f"{root_dir_uri}/{large_image_result.preview_filename}",
- large_image_result.preview_bytes
+ # Process the image (check if large)
+ large_image_result = process_large_image(
+ file_path, img, filename_prefix=stem, config=self.config
+ )
+ img.close()
+
+ # Create root node with metadata
+ root_node = ResourceNode(
+ type=NodeType.ROOT,
+ title=display_stem,
+ level=0,
+ detail_file=None,
+ content_path=None,
+ children=[],
+ meta={
+ "width": width,
+ "height": height,
+ "format": format_str.lower(),
+ "content_type": "image",
+ "source_title": display_stem,
+ "semantic_name": display_stem,
+ "original_filename": original_filename,
+ "is_large_image": large_image_result.needs_processing,
+ },
)
- root_node.meta["preview_filename"] = large_image_result.preview_filename
-
- # Save grid overlay
- if large_image_result.grid_overlay_bytes:
- grid_filename = large_image_result.grid_overlay_filename or f"{stem}_grid.jpg"
- await viking_fs.write_file_bytes(
- f"{root_dir_uri}/{grid_filename}",
- large_image_result.grid_overlay_bytes
+
+ if large_image_result.needs_processing:
+ # Large image processing mode: save preview, grid, tiles
+ # NOTE: The original file is intentionally NOT saved. Large images can be
+ # hundreds of MB; storing the full-resolution original would exceed storage
+ # budgets. The preview + tiles are the canonical representation. The
+ # `original_filename` metadata field records the original name for reference
+ # but the corresponding file is not persisted.
+ logger.debug(f"Processing large image {original_filename}: {width}x{height}")
+
+ # Save low-res preview (original is too large to store)
+ await writer.write_bytes(
+ f"{root_dir_name}/{large_image_result.preview_filename}",
+ large_image_result.preview_bytes,
)
- root_node.meta["grid_overlay"] = grid_filename
-
- # Create tiles directory
- if large_image_result.tiles:
- tiles_dir_name = "tiles"
- tiles_dir_uri = f"{root_dir_uri}/{tiles_dir_name}"
- await viking_fs.mkdir(tiles_dir_uri, exist_ok=True)
-
- # Save all tiles
- for tile in large_image_result.tiles:
- if tile.bytes_data:
- await viking_fs.write_file_bytes(
- f"{tiles_dir_uri}/{tile.filename}",
- tile.bytes_data
- )
-
- # Update metadata
- root_node.meta["tiles_dir"] = tiles_dir_name
- root_node.meta["num_tiles"] = len(large_image_result.tiles)
- root_node.meta["grid_rows"] = large_image_result.total_rows
- root_node.meta["grid_cols"] = large_image_result.total_cols
- root_node.meta["original_width"] = width
- root_node.meta["original_height"] = height
-
- else:
- # Small image: save as PNG if format is not VLM-supported, else as-is
- if needs_png_conversion:
- # SVG was already converted to PNG bytes during loading
- if file_path.suffix.lower() == ".svg":
- image_bytes = converted_png_bytes
- else:
- with Image.open(file_path) as converted_img:
- image_bytes = save_image_to_bytes(converted_img, format="PNG")
+ root_node.meta["preview_filename"] = large_image_result.preview_filename
+
+ # Save grid overlay
+ if large_image_result.grid_overlay_bytes:
+ grid_filename = large_image_result.grid_overlay_filename or f"{stem}_grid.jpg"
+ await writer.write_bytes(
+ f"{root_dir_name}/{grid_filename}",
+ large_image_result.grid_overlay_bytes,
+ )
+ root_node.meta["grid_overlay"] = grid_filename
+
+ # Create tiles directory
+ if large_image_result.tiles:
+ tiles_dir_name = "tiles"
+ tiles_dir_rel = f"{root_dir_name}/{tiles_dir_name}"
+ await writer.mkdir(tiles_dir_rel)
+
+ # Save all tiles
+ for tile in large_image_result.tiles:
+ if tile.bytes_data:
+ await writer.write_bytes(
+ f"{tiles_dir_rel}/{tile.filename}", tile.bytes_data
+ )
+
+ # Update metadata
+ root_node.meta["tiles_dir"] = tiles_dir_name
+ root_node.meta["num_tiles"] = len(large_image_result.tiles)
+ root_node.meta["grid_rows"] = large_image_result.total_rows
+ root_node.meta["grid_cols"] = large_image_result.total_cols
+ root_node.meta["original_width"] = width
+ root_node.meta["original_height"] = height
+
else:
- image_bytes = file_path.read_bytes()
- await viking_fs.write_file_bytes(f"{root_dir_uri}/{original_filename}", image_bytes)
+ # Small image: save as PNG if format is not VLM-supported, else as-is
+ if needs_png_conversion:
+ # SVG was already converted to PNG bytes during loading
+ if file_path.suffix.lower() == ".svg":
+ image_bytes = converted_png_bytes
+ else:
+ with Image.open(file_path) as converted_img:
+ image_bytes = save_image_to_bytes(converted_img, format="PNG")
+ else:
+ image_bytes = file_path.read_bytes()
+ await writer.write_bytes(f"{root_dir_name}/{original_filename}", image_bytes)
+
+ artifact_ref = await writer.finalize(resource_rel=root_dir_name)
+ except BaseException:
+ await writer.cleanup()
+ raise
# Phase 3: Build directory structure (handled by TreeBuilder)
return ParseResult(
root=root_node,
source_path=str(file_path),
- temp_dir_path=temp_uri,
+ temp_dir_path=artifact_ref.root,
+ artifact_ref=artifact_ref,
source_format="image",
parser_name="ImageParser",
meta={
diff --git a/openviking/parse/parsers/media/video.py b/openviking/parse/parsers/media/video.py
index 26105155e..e6caf128f 100644
--- a/openviking/parse/parsers/media/video.py
+++ b/openviking/parse/parsers/media/video.py
@@ -28,6 +28,7 @@ from pathlib import Path
from typing import List, Optional, Union
from openviking.parse.base import NodeType, ParseResult, ResourceNode
+from openviking.parse.output import create_parse_artifact_writer
from openviking.parse.parsers.base_parser import BaseParser
from openviking.parse.parsers.media.constants import VIDEO_EXTENSIONS
from openviking.parse.parsers.media.naming import resolve_media_names
@@ -69,16 +70,11 @@ class VideoParser(BaseParser):
FileNotFoundError: If source file does not exist
IOError: If video processing fails
"""
- from openviking.storage.viking_fs import get_viking_fs
-
# Convert to Path object
file_path = Path(source) if isinstance(source, str) else source
if not file_path.exists():
raise FileNotFoundError(f"Video file not found: {source}")
- viking_fs = get_viking_fs()
- temp_uri = viking_fs.create_temp_uri()
-
# Phase 1: Generate temporary files
video_bytes = file_path.read_bytes()
ext = file_path.suffix
@@ -92,12 +88,6 @@ class VideoParser(BaseParser):
# Root directory name: filename stem + _ + extension (without dot)
ext_no_dot = ext[1:] if ext else ""
root_dir_name = VikingURI.sanitize_segment(f"{stem}_{ext_no_dot}")
- root_dir_uri = f"{temp_uri}/{root_dir_name}"
- await viking_fs.mkdir(root_dir_uri, exist_ok=True)
-
- # 1.1 Save original video with original filename (sanitized)
- await viking_fs.write_file_bytes(f"{root_dir_uri}/{original_filename}", video_bytes)
-
# 1.2 Validate video file using magic bytes
# Define magic bytes for supported video formats
video_magic_bytes = {
@@ -124,6 +114,18 @@ class VideoParser(BaseParser):
f"Invalid video file: {file_path}. File signature does not match expected format {ext_lower}"
)
+ writer = await create_parse_artifact_writer(
+ kwargs.get("parse_output_store"),
+ viking_fs=self._get_viking_fs() if kwargs.get("parse_output_store") is None else None,
+ )
+ try:
+ await writer.mkdir(root_dir_name)
+ await writer.write_bytes(f"{root_dir_name}/{original_filename}", video_bytes)
+ artifact_ref = await writer.finalize(resource_rel=root_dir_name)
+ except BaseException:
+ await writer.cleanup()
+ raise
+
# Extract video metadata (placeholder)
duration = 0
width = 0
@@ -156,7 +158,8 @@ class VideoParser(BaseParser):
return ParseResult(
root=root_node,
source_path=str(file_path),
- temp_dir_path=temp_uri,
+ temp_dir_path=artifact_ref.root,
+ artifact_ref=artifact_ref,
source_format="video",
parser_name="VideoParser",
meta={"content_type": "video", "format": format_str.lower()},
diff --git a/openviking/parse/parsers/pdf.py b/openviking/parse/parsers/pdf.py
index f299a0680..f46a5cec8 100644
--- a/openviking/parse/parsers/pdf.py
+++ b/openviking/parse/parsers/pdf.py
@@ -142,6 +142,7 @@ class PDFParser(BaseParser):
base_dir=pdf_path.parent,
allowed_media_dirs=[storage.media_dir],
split_content=kwargs.get("split_content", True),
+ parse_output_store=kwargs.get("parse_output_store"),
)
# Step 3: Update metadata for PDF origin
diff --git a/openviking/parse/parsers/upload_utils.py b/openviking/parse/parsers/upload_utils.py
index fc9071664..c2ff438b4 100644
--- a/openviking/parse/parsers/upload_utils.py
+++ b/openviking/parse/parsers/upload_utils.py
@@ -8,6 +8,12 @@ from pathlib import Path
from typing import Any, List, Optional, Set, Tuple, Union
from openviking.parse.gitignore import GitignoreMatcher
+from openviking.parse.output import (
+ ARTIFACT_MANIFEST_NAME as ARTIFACT_MANIFEST_NAME,
+)
+from openviking.parse.output import (
+ write_artifact_manifest,
+)
from openviking.parse.parsers.constants import (
ADDITIONAL_TEXT_EXTENSIONS,
CODE_EXTENSIONS,
@@ -15,7 +21,7 @@ from openviking.parse.parsers.constants import (
IGNORE_EXTENSIONS,
)
from openviking.parse.parsers.text_encoding import normalize_text_bytes
-from openviking.utils.path_safety import safe_join_viking_uri
+from openviking.utils.content_hash import content_md5
from openviking_cli.utils.logger import get_logger
logger = get_logger(__name__)
@@ -116,18 +122,24 @@ _UPLOAD_CONCURRENCY = 8
async def upload_directory(
local_dir: Path,
- viking_uri_base: str,
- viking_fs: Any,
+ base_rel: str,
+ *,
+ store: Any,
+ artifact_ref: Any,
ignore_dirs: Optional[Union[Set[str], List[str], str]] = None,
ignore_extensions: Optional[Set[str]] = None,
max_file_size: int = 10 * 1024 * 1024,
include: Optional[str] = None,
exclude: Optional[str] = None,
) -> Tuple[int, List[str]]:
- """Upload an entire directory recursively and return uploaded count with warnings.
-
- Optimized: collects all files in one pass, pre-creates directories upfront,
- then uploads all files concurrently (up to _UPLOAD_CONCURRENCY at a time).
+ """Upload a directory into a parse output store, returning (count, warnings).
+
+ Artifacts are written through the backend-agnostic :class:`ParseOutputStore`,
+ so AGFS and local backends share one path with no branching. Files land under
+ ``base_rel`` (e.g. ``repository``) as artifact-relative paths the store
+ resolves to a local dir or an AGFS temp URI. An md5 manifest of the final
+ (encoding-normalized) bytes is written at the artifact root so the incremental
+ diff can compare fingerprints without re-reading file contents.
"""
effective_ignore_extensions = (
ignore_extensions if ignore_extensions is not None else IGNORE_EXTENSIONS
@@ -153,9 +165,10 @@ async def upload_directory(
warnings: List[str] = []
- # --- Phase 1: Collect files and unique parent directory URIs in one pass ---
- files_to_upload: List[Tuple[Path, str]] = [] # (local_path, target_uri)
- parent_uris: Set[str] = {viking_uri_base}
+ # --- Phase 1: Collect files and their artifact-relative parent dirs ---
+ base = base_rel.strip("/")
+ files_to_upload: List[Tuple[Path, str]] = []
+ parent_dirs: Set[str] = set()
for root, dirs, files in os.walk(local_dir):
dir_path = Path(root)
@@ -202,39 +215,29 @@ async def upload_directory(
exclude_patterns,
):
continue
- try:
- target_uri = safe_join_viking_uri(viking_uri_base, rel_path_str)
- except ValueError as exc:
- warning = f"Skipping {file_path}: {exc}"
- warnings.append(warning)
- logger.warning(warning)
- continue
- files_to_upload.append((file_path, target_uri))
- parent_uris.add(target_uri.rsplit("/", 1)[0])
-
- # --- Phase 2: Pre-create all directories ---
- # Memoized mkdir: each unique VikingFS path is created at most once.
- # This is equivalent to _ensure_parent_dirs but avoids redundant HTTP calls
- # by tracking already-processed paths across all directories.
- _created: Set[str] = set()
-
- for dir_uri in sorted(parent_uris):
- if dir_uri in _created:
- continue
+ # Artifact-relative path under the resource root; the store resolves it
+ # to a local/agfs location.
+ target = f"{base}/{rel_path_str}" if base else rel_path_str
+ files_to_upload.append((file_path, target))
+ if "/" in target:
+ parent_dirs.add(target.rsplit("/", 1)[0])
+
+ # --- Phase 2: Pre-create unique parent dirs (shallowest first) ---
+ # store.write_bytes also creates parents implicitly; this batches the unique
+ # directories once to avoid redundant per-file creation on remote backends.
+ for rel_dir in sorted(parent_dirs, key=lambda value: (value.count("/"), value)):
try:
- await viking_fs.mkdir(dir_uri, exist_ok=True)
- _created.add(dir_uri)
+ await store.mkdir(artifact_ref, rel_dir)
except Exception as e:
- if "already" in str(e).lower():
- _created.add(dir_uri)
- else:
- logger.warning(f"Failed to create directory {dir_uri}: {e}")
+ logger.warning(f"Failed to create artifact directory {rel_dir}: {e}")
# --- Phase 3: Upload files concurrently ---
sem = asyncio.Semaphore(_UPLOAD_CONCURRENCY)
errors: List[Optional[str]] = [None] * len(files_to_upload)
+ # rel_path -> md5 of final bytes, collected for the manifest.
+ md5_by_target: dict[str, str] = {}
- async def _upload_one(idx: int, file_path: Path, target_uri: str) -> None:
+ async def _upload_one(idx: int, file_path: Path, target: str) -> None:
async with sem:
def _read_and_encode() -> bytes:
@@ -243,16 +246,25 @@ async def upload_directory(
try:
encoded = await asyncio.to_thread(_read_and_encode)
- await viking_fs.write_file_bytes(target_uri, encoded)
+ await store.write_bytes(artifact_ref, target, encoded)
+ # Fingerprint the final (normalized) bytes at the write point; this
+ # is a local read, no extra remote IO.
+ md5_by_target[target] = content_md5(encoded)
except Exception as exc:
errors[idx] = f"Failed to upload {file_path}: {exc}"
- await asyncio.gather(*[_upload_one(i, fp, uri) for i, (fp, uri) in enumerate(files_to_upload)])
+ await asyncio.gather(*[_upload_one(i, fp, tgt) for i, (fp, tgt) in enumerate(files_to_upload)])
for err in errors:
if err:
warnings.append(err)
logger.warning(err)
+ # Persist the md5 manifest at the artifact root so the incremental diff can
+ # compare fingerprints without re-reading files. Only write it when every file
+ # uploaded cleanly, so a partial manifest never masquerades as complete.
+ if not any(errors):
+ await write_artifact_manifest(store, artifact_ref, md5_by_target)
+
uploaded_count = sum(1 for e in errors if e is None)
return uploaded_count, warnings
diff --git a/openviking/parse/tree_builder.py b/openviking/parse/tree_builder.py
index deae20b76..765ce3300 100644
--- a/openviking/parse/tree_builder.py
+++ b/openviking/parse/tree_builder.py
@@ -21,7 +21,7 @@ IMPORTANT (v5.0 Architecture):
- Content splitting is handled by Parser, not TreeBuilder
"""
-from typing import Optional
+from typing import Any, Optional
from openviking.core.building_tree import BuildingTree
from openviking.core.context import Context
@@ -145,6 +145,8 @@ class TreeBuilder:
source_format: Optional[str] = None,
create_parent: bool = False,
flatten_single_file: bool = False,
+ artifact_ref: Optional[Any] = None,
+ output_store: Optional[Any] = None,
) -> "BuildingTree":
"""
Finalize URI metadata for a temp parse result.
@@ -153,38 +155,57 @@ class TreeBuilder:
to_uri: Exact target URI, or resources root to import under
parent_uri: Target parent URI (must exist unless create_parent is True)
create_parent: Whether to automatically create parent directory if it doesn't exist
+ artifact_ref / output_store: When provided, the artifact structure is
+ read through the backend-agnostic store (supporting local
+ artifacts); otherwise the legacy AGFS ``ls(temp_uri)`` path is
+ used. Either way the resulting ``temp_doc_uri`` keeps the AGFS URI
+ shape and target-URI resolution stays on VikingFS.
"""
viking_fs = get_viking_fs()
temp_uri = temp_dir_path
- # 1. Find document root directory
- entries = await viking_fs.ls(temp_uri, ctx=ctx)
- doc_dirs = [e for e in entries if e.get("isDir") and e["name"] not in [".", ".."]]
+ # 1. Find the single document root. The structural walk is identical for
+ # AGFS and local artifacts, so it is delegated to resolve_artifact_doc_root
+ # when a store/ref is available. ``temp_doc_uri`` remains an AGFS-shaped
+ # URI relative to temp_uri for downstream persistence.
+ if output_store is not None and artifact_ref is not None:
+ from openviking.parse.output import resolve_artifact_doc_root
- if len(doc_dirs) != 1:
- logger.error(
- f"[TreeBuilder] Expected 1 document directory in {temp_uri}, found {len(doc_dirs)}"
+ resolved = await resolve_artifact_doc_root(
+ output_store, artifact_ref, flatten_single_file=flatten_single_file
)
- raise ValueError(
- f"[TreeBuilder] Expected 1 document directory in {temp_uri}, found {len(doc_dirs)}"
- )
-
- original_name = doc_dirs[0]["name"]
- doc_name = VikingURI.sanitize_segment(original_name)
- temp_doc_uri = f"{temp_uri}/{original_name}" # use original name to find temp dir
- root_is_file = False
- if flatten_single_file:
- doc_entries = [
- entry
- for entry in await viking_fs.ls(temp_doc_uri, ctx=ctx)
- if entry.get("name") not in [".", ".."]
- ]
- if len(doc_entries) == 1 and not doc_entries[0].get("isDir"):
- original_name = doc_entries[0]["name"]
- doc_name = VikingURI.sanitize_segment(original_name)
- temp_doc_uri = f"{temp_doc_uri}/{original_name}"
- root_is_file = True
+ original_name = resolved.doc_name
+ doc_name = VikingURI.sanitize_segment(original_name)
+ temp_doc_uri = f"{temp_uri.rstrip('/')}/{resolved.doc_rel}"
+ root_is_file = resolved.root_is_file
+ else:
+ entries = await viking_fs.ls(temp_uri, ctx=ctx)
+ doc_dirs = [e for e in entries if e.get("isDir") and e["name"] not in [".", ".."]]
+
+ if len(doc_dirs) != 1:
+ logger.error(
+ f"[TreeBuilder] Expected 1 document directory in {temp_uri}, found {len(doc_dirs)}"
+ )
+ raise ValueError(
+ f"[TreeBuilder] Expected 1 document directory in {temp_uri}, found {len(doc_dirs)}"
+ )
+
+ original_name = doc_dirs[0]["name"]
+ doc_name = VikingURI.sanitize_segment(original_name)
+ temp_doc_uri = f"{temp_uri}/{original_name}" # use original name to find temp dir
+ root_is_file = False
+ if flatten_single_file:
+ doc_entries = [
+ entry
+ for entry in await viking_fs.ls(temp_doc_uri, ctx=ctx)
+ if entry.get("name") not in [".", ".."]
+ ]
+ if len(doc_entries) == 1 and not doc_entries[0].get("isDir"):
+ original_name = doc_entries[0]["name"]
+ doc_name = VikingURI.sanitize_segment(original_name)
+ temp_doc_uri = f"{temp_doc_uri}/{original_name}"
+ root_is_file = True
if original_name != doc_name:
logger.debug(f"[TreeBuilder] Sanitized doc name: {original_name!r} -> {doc_name!r}")
diff --git a/openviking/parse/understanding_api.py b/openviking/parse/understanding_api.py
index fdf57616d..65f482930 100644
--- a/openviking/parse/understanding_api.py
+++ b/openviking/parse/understanding_api.py
@@ -8,7 +8,7 @@ Workflow:
2. Submit a parse request to Responses API (response_id)
3. Poll Responses API until completed/failed
4. Download result zip (zip_url)
-5. Materialize the result into VikingFS temp directory
+5. Materialize the result through the configured ParseOutputStore
6. Return ParseResult for downstream TreeBuilder/SemanticQueue processing
"""
@@ -28,6 +28,10 @@ from openviking.parse.image_rewrite import (
IMAGE_MAPPINGS_FILENAME,
build_artifact_image_mappings,
)
+from openviking.parse.output import (
+ ParseArtifactRef,
+ create_parse_artifact_writer,
+)
from openviking.parse.parsers.base_parser import BaseParser
from openviking.parse.parsers.constants import MPEG_TS_EXTENSION_ALIAS, TYPESCRIPT_MPEG_TS_EXTENSION
from openviking.parse.parsers.media.constants import (
@@ -229,10 +233,15 @@ class UnderstandingAPI(BaseParser):
if archive_root:
doc_name = archive_root
task_meta["doc_name"] = doc_name
- temp_dir_path = await self._unpack_zip_to_temp_dir(
- zip_path=zip_path,
- resource_name=doc_name,
- )
+ unpack_kwargs = {
+ "zip_path": zip_path,
+ "resource_name": doc_name,
+ }
+ if kwargs.get("parse_output_store") is not None:
+ unpack_kwargs["parse_output_store"] = kwargs["parse_output_store"]
+ unpacked = await self._unpack_zip_to_temp_dir(**unpack_kwargs)
+ artifact_ref = unpacked if isinstance(unpacked, ParseArtifactRef) else None
+ temp_dir_path = artifact_ref.root if artifact_ref is not None else unpacked
finally:
try:
zip_path.unlink()
@@ -276,6 +285,7 @@ class UnderstandingAPI(BaseParser):
content_type if content_type in {"image", "audio", "video"} else doc_type
),
temp_dir_path=temp_dir_path,
+ artifact_ref=artifact_ref,
parser_name="UnderstandingAPI",
meta={key: task_meta[key] for key in ("file_id", "response_id") if task_meta.get(key)},
)
@@ -762,14 +772,20 @@ class UnderstandingAPI(BaseParser):
f.write(rsp.content)
return Path(f.name)
- async def _unpack_zip_to_temp_dir(self, zip_path: Path, resource_name: str) -> str:
- viking_fs = get_viking_fs()
- temp_uri = viking_fs.create_temp_uri()
+ async def _unpack_zip_to_temp_dir(
+ self,
+ zip_path: Path,
+ resource_name: str,
+ *,
+ parse_output_store: Any = None,
+ ) -> ParseArtifactRef:
+ writer = await create_parse_artifact_writer(
+ parse_output_store,
+ viking_fs=get_viking_fs() if parse_output_store is None else None,
+ )
+ resource_rel = str(resource_name).strip("/")
try:
- await viking_fs.mkdir(temp_uri)
-
- temp_doc_uri = f"{temp_uri}/{resource_name}"
- await viking_fs.mkdir(temp_doc_uri)
+ await writer.mkdir(resource_rel)
with tempfile.TemporaryDirectory() as extract_dir:
with zipfile.ZipFile(zip_path, "r") as zf:
@@ -783,51 +799,32 @@ class UnderstandingAPI(BaseParser):
image_mappings = await asyncio.to_thread(build_artifact_image_mappings, root_dir)
- for child in root_dir.iterdir():
+ for child in root_dir.rglob("*"):
+ rel = child.relative_to(root_dir).as_posix()
if child.name in {".", "..", IMAGE_MAPPINGS_FILENAME}:
continue
if child.is_dir():
- sub_uri = f"{temp_doc_uri}/{child.name}"
- await viking_fs.mkdir(sub_uri)
- await self._copy_dir_to_fs(child, sub_uri)
+ await writer.mkdir(f"{resource_rel}/{rel}")
else:
- await viking_fs.write_file_bytes(
- f"{temp_doc_uri}/{child.name}", child.read_bytes()
+ await writer.write_bytes(
+ f"{resource_rel}/{rel}",
+ await asyncio.to_thread(child.read_bytes),
)
if image_mappings:
- await viking_fs.write_file(
- f"{temp_doc_uri}/{IMAGE_MAPPINGS_FILENAME}",
+ await writer.write_text(
+ f"{resource_rel}/{IMAGE_MAPPINGS_FILENAME}",
json.dumps(image_mappings, ensure_ascii=False),
)
except BaseException:
try:
- await viking_fs.delete_temp(temp_uri)
+ await writer.cleanup()
except Exception as cleanup_exc:
logger.warning(
"[UnderstandingAPI] Failed to clean temporary artifact %s: %s",
- temp_uri,
+ writer.ref.root,
cleanup_exc,
)
raise
- return temp_uri
-
- async def _copy_dir_to_fs(self, local_dir: Path, fs_uri: str):
- """
- Recursively copy a local directory to VikingFS.
- """
- viking_fs = get_viking_fs()
-
- for item in local_dir.iterdir():
- if item.name in [".", "..", IMAGE_MAPPINGS_FILENAME]:
- continue
-
- if item.is_dir():
- sub_uri = f"{fs_uri}/{item.name}"
- await viking_fs.mkdir(sub_uri)
- await self._copy_dir_to_fs(item, sub_uri)
- else:
- file_content = item.read_bytes()
- file_uri = f"{fs_uri}/{item.name}"
- await viking_fs.write_file_bytes(file_uri, file_content)
+ return await writer.finalize(resource_rel=resource_rel)
diff --git a/openviking/resource/shared_source.py b/openviking/resource/shared_source.py
new file mode 100644
index 000000000..01a5731d6
--- /dev/null
+++ b/openviking/resource/shared_source.py
@@ -0,0 +1,112 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Shared-upload SOURCE references for durable resource ingestion.
+
+Unlike :mod:`openviking.resource.staged_source`, a shared upload already lives in
+durable VikingFS storage (``viking://upload/...``) with its own TTL cleanup, so
+the SOURCE phase does not copy it into a task-owned temp bundle. The API only
+validates and records a reference; the worker downloads the content once into a
+worker-local file for parsing. Ownership of the shared object stays with the
+upload store's TTL policy — the task never deletes it as its own temp.
+"""
+
+import asyncio
+import shutil
+import tempfile
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any, Dict
+
+from openviking.parse.accessors.base import LocalResource, SourceType
+from openviking.server.error_mapping import is_not_found_error
+from openviking.server.identity import RequestContext, Role
+
+_SHARED_UPLOAD_ROOT = "viking://upload"
+
+
+@dataclass(frozen=True)
+class SharedSource:
+ """A validated reference to a shared upload consumed as a SOURCE input."""
+
+ temp_file_id: str
+ content_uri: str
+ account_id: str
+ original_filename: str
+ file_ext: str
+ meta: Dict[str, Any]
+
+ def to_dict(self) -> Dict[str, Any]:
+ return asdict(self)
+
+ @classmethod
+ def from_dict(cls, data: Dict[str, Any]) -> "SharedSource":
+ if not isinstance(data, dict):
+ raise ValueError("shared_source must be an object")
+ temp_file_id = data.get("temp_file_id")
+ content_uri = data.get("content_uri")
+ account_id = data.get("account_id")
+ original_filename = data.get("original_filename") or ""
+ file_ext = data.get("file_ext") or ""
+ meta = data.get("meta")
+ if not isinstance(temp_file_id, str) or not temp_file_id.startswith("shared_"):
+ raise ValueError("shared_source.temp_file_id must be a shared upload id")
+ if not isinstance(content_uri, str) or not content_uri.startswith(
+ f"{_SHARED_UPLOAD_ROOT}/"
+ ):
+ raise ValueError("shared_source.content_uri must live under the upload root")
+ if not isinstance(account_id, str) or not account_id:
+ raise ValueError("shared_source.account_id must be a non-empty string")
+ if not isinstance(original_filename, str):
+ raise ValueError("shared_source.original_filename must be a string")
+ if not isinstance(file_ext, str):
+ raise ValueError("shared_source.file_ext must be a string")
+ if meta is not None and not isinstance(meta, dict):
+ raise ValueError("shared_source.meta must be an object")
+ return cls(
+ temp_file_id=temp_file_id,
+ content_uri=content_uri.rstrip("/"),
+ account_id=account_id,
+ original_filename=original_filename,
+ file_ext=file_ext,
+ meta=dict(meta or {}),
+ )
+
+
+async def materialize_shared_source(
+ shared: "SharedSource",
+ *,
+ viking_fs: Any,
+ ctx: RequestContext,
+) -> LocalResource:
+ """Download a shared upload into a worker-local file for parsing.
+
+ The shared object is owned by the upload store's TTL policy, so the returned
+ resource only cleans up the worker-local copy. A missing or expired object
+ fails loudly instead of yielding an empty input.
+ """
+ suffix = shared.file_ext or Path(shared.original_filename).suffix or ".tmp"
+ local_root = Path(tempfile.mkdtemp(prefix="ov_shared_source_"))
+ filename = shared.original_filename or f"resource{suffix}"
+ local_path = local_root / Path(filename).name
+ try:
+ internal_ctx = RequestContext(user=ctx.user, role=Role.ROOT)
+ content = await viking_fs.read_file_bytes(shared.content_uri, ctx=internal_ctx)
+ await asyncio.to_thread(local_path.write_bytes, content)
+ except asyncio.CancelledError:
+ shutil.rmtree(local_root, ignore_errors=True)
+ raise
+ except Exception as exc:
+ shutil.rmtree(local_root, ignore_errors=True)
+ if is_not_found_error(exc):
+ raise ValueError("Shared upload content is missing or expired") from exc
+ raise
+
+ meta = dict(shared.meta)
+ meta["_cleanup_path"] = str(local_root)
+ return LocalResource(
+ path=local_path,
+ source_type=SourceType.LOCAL,
+ original_source=shared.original_filename or shared.temp_file_id,
+ meta=meta,
+ is_temporary=True,
+ )
diff --git a/openviking/server/routers/resources.py b/openviking/server/routers/resources.py
index 23aa74188..a300b87f0 100644
--- a/openviking/server/routers/resources.py
+++ b/openviking/server/routers/resources.py
@@ -256,6 +256,7 @@ async def add_resource(
allow_local_path_resolution = False
original_filename = None
resolved = None
+ shared_source_ref = None
if request.temp_file_id:
if request.watch_interval > 0:
raise InvalidArgumentError(
@@ -265,12 +266,20 @@ async def add_resource(
"sitemap / RSS source instead, or re-add the resource when the "
"source changes."
)
- resolved = await TempUploadStore.build(http_request.app.state.config).resolve_for_consume(
- request.temp_file_id, _ctx
- )
- path = resolved.local_path
- original_filename = resolved.original_filename
- allow_local_path_resolution = True
+ store = TempUploadStore.build(http_request.app.state.config)
+ # A shared upload already lives in durable storage: the API only validates
+ # a reference and the SOURCE worker downloads it once, avoiding a second
+ # API-side download + task re-stage. Local uploads keep the copy path.
+ shared_source_ref = await store.resolve_shared_reference(request.temp_file_id, _ctx)
+ if shared_source_ref is not None:
+ path = shared_source_ref.original_filename or request.temp_file_id
+ original_filename = shared_source_ref.original_filename or None
+ allow_local_path_resolution = True
+ else:
+ resolved = await store.resolve_for_consume(request.temp_file_id, _ctx)
+ path = resolved.local_path
+ original_filename = resolved.original_filename
+ allow_local_path_resolution = True
elif path is not None:
path = require_remote_resource_source(path, declared_connector_add_type=request.add_type)
if path is None:
@@ -320,6 +329,7 @@ async def add_resource(
internal_task=request.internal_task,
is_active=request.is_active,
args=request.args,
+ shared_source=shared_source_ref,
**kwargs,
)
except Exception:
diff --git a/openviking/server/temp_upload_store.py b/openviking/server/temp_upload_store.py
index 00241a6cd..7244a9843 100644
--- a/openviking/server/temp_upload_store.py
+++ b/openviking/server/temp_upload_store.py
@@ -17,7 +17,7 @@ from dataclasses import dataclass
from pathlib import Path
from queue import Full, Queue
from threading import Lock, Thread
-from typing import Any, Optional
+from typing import TYPE_CHECKING, Any, Optional
from openviking.server.config import ServerConfig, TempUploadConfig
from openviking.server.identity import RequestContext, Role
@@ -26,6 +26,9 @@ from openviking.storage.viking_fs import LS_ALL_NODES, get_viking_fs
from openviking_cli.exceptions import InvalidArgumentError, PermissionDeniedError
from openviking_cli.utils.config.open_viking_config import get_openviking_config
+if TYPE_CHECKING:
+ from openviking.resource.shared_source import SharedSource
+
_CHUNK_SIZE = 1024 * 1024
_SHARED_UPLOAD_ROOT = "viking://upload"
_SHARED_CLEANUP_QUEUE_MAX_SIZE = 100
@@ -186,9 +189,7 @@ def _shared_upload_created_at(upload_id: str) -> Optional[float]:
async def _stream_upload_to_local_temp(upload_file: Any, max_size_bytes: int) -> tuple[str, int]:
suffix = Path(upload_file.filename or "upload.tmp").suffix or ".tmp"
- temp_path = await asyncio.to_thread(
- _create_temp_file, prefix="ov_http_upload_", suffix=suffix
- )
+ temp_path = await asyncio.to_thread(_create_temp_file, prefix="ov_http_upload_", suffix=suffix)
total = 0
f = None
try:
@@ -199,9 +200,7 @@ async def _stream_upload_to_local_temp(upload_file: Any, max_size_bytes: int) ->
break
total += len(chunk)
if total > max_size_bytes:
- raise InvalidArgumentError(
- f"Upload exceeds size limit ({max_size_bytes} bytes)."
- )
+ raise InvalidArgumentError(f"Upload exceeds size limit ({max_size_bytes} bytes).")
# UploadFile reads already yield to the event loop. Local disk writes do
# not, so run each bounded write in the default executor rather than
# stalling the Core worker event loop on slow local storage.
@@ -254,6 +253,40 @@ class TempUploadStore:
return await asyncio.to_thread(self._resolve_local, temp_file_id)
return await self._resolve_shared(temp_file_id, shared_id, ctx)
+ async def resolve_shared_reference(
+ self,
+ temp_file_id: str,
+ ctx: RequestContext,
+ ) -> Optional["SharedSource"]:
+ """Validate a shared upload and return a download-free SOURCE reference.
+
+ The API only checks ownership and existence; the worker downloads the
+ content once (see :func:`materialize_shared_source`). Returns ``None`` for
+ non-shared ids so callers can fall back to the local-copy path.
+ """
+ from openviking.resource.shared_source import SharedSource
+
+ shared_id = _parse_shared_temp_file_id(temp_file_id)
+ if shared_id is None:
+ return None
+ meta = await self._read_shared_meta(shared_id, ctx)
+ self._validate_shared_meta(meta, temp_file_id, ctx)
+ content_uri = meta["storage_uri"]
+ vfs = get_viking_fs()
+ if not await vfs.exists(content_uri, ctx=self._internal_ctx(ctx)):
+ raise PermissionDeniedError("Temporary upload is invalid: content missing.")
+ original_filename = meta.get("original_filename") or ""
+ return SharedSource.from_dict(
+ {
+ "temp_file_id": temp_file_id,
+ "content_uri": content_uri,
+ "account_id": ctx.account_id,
+ "original_filename": original_filename,
+ "file_ext": meta.get("file_ext") or Path(original_filename).suffix,
+ "meta": {},
+ }
+ )
+
async def _save_local(self, upload_file: Any) -> str:
config = get_openviking_config()
temp_dir = config.storage.get_upload_temp_dir()
@@ -316,9 +349,7 @@ class TempUploadStore:
try:
content = await asyncio.to_thread(Path(temp_path).read_bytes)
- await vfs.write_file_bytes(
- content_uri, content, ctx=internal_ctx, auto_pathlock=False
- )
+ await vfs.write_file_bytes(content_uri, content, ctx=internal_ctx, auto_pathlock=False)
await vfs.write_file(
meta_uri,
json.dumps(meta, ensure_ascii=False),
@@ -357,9 +388,7 @@ class TempUploadStore:
now = time.time()
with _SHARED_CLEANUP_STATE_LOCK:
if account_id in _SHARED_CLEANUP_PENDING:
- logger.debug(
- "[TempUpload] Shared cleanup already pending account=%s", account_id
- )
+ logger.debug("[TempUpload] Shared cleanup already pending account=%s", account_id)
return
due_at = _SHARED_CLEANUP_DUE_AT.get(account_id)
if due_at is not None and now < due_at:
@@ -528,9 +557,7 @@ class TempUploadStore:
try:
data = json.loads(await vfs.read_file(meta_uri, ctx=internal_ctx))
except Exception as exc:
- raise PermissionDeniedError(
- "Temporary upload metadata is invalid or missing."
- ) from exc
+ raise PermissionDeniedError("Temporary upload metadata is invalid or missing.") from exc
if not isinstance(data, dict):
raise PermissionDeniedError("Temporary upload metadata is invalid or missing.")
return data
@@ -615,9 +642,9 @@ class TempUploadStore:
# Classify the single listing into independent groups (order preserved,
# so each group stays name-ascending / oldest-first).
- buckets: list[tuple[str, float]] = [] # (uri, bucket_expiry)
- legacy: list[tuple[str, float]] = [] # (uri, upload_expiry)
- invalid: list[str] = [] # uri
+ buckets: list[tuple[str, float]] = [] # (uri, bucket_expiry)
+ legacy: list[tuple[str, float]] = [] # (uri, upload_expiry)
+ invalid: list[str] = [] # uri
for entry in entries:
if not entry.get("isDir"):
continue
@@ -636,12 +663,18 @@ class TempUploadStore:
invalid.append(uri)
# Process the bucket and legacy groups independently, oldest-first.
- bucket_scanned, bucket_removed, bucket_due, bucket_failed = (
- await self._cleanup_expiry_group(vfs, internal_ctx, buckets, now, kind="bucket")
- )
- legacy_scanned, legacy_removed, legacy_due, legacy_failed = (
- await self._cleanup_expiry_group(vfs, internal_ctx, legacy, now, kind="flat")
- )
+ (
+ bucket_scanned,
+ bucket_removed,
+ bucket_due,
+ bucket_failed,
+ ) = await self._cleanup_expiry_group(vfs, internal_ctx, buckets, now, kind="bucket")
+ (
+ legacy_scanned,
+ legacy_removed,
+ legacy_due,
+ legacy_failed,
+ ) = await self._cleanup_expiry_group(vfs, internal_ctx, legacy, now, kind="flat")
# Invalid/foreign dirs: only when explicitly enabled; never blocks the
# expiry groups and failures are logged inside the helper.
@@ -729,9 +762,7 @@ class TempUploadStore:
"""
started_at = time.monotonic()
try:
- await vfs.remove_files(
- uri, recursive=True, ctx=internal_ctx, auto_pathlock=False
- )
+ await vfs.remove_files(uri, recursive=True, ctx=internal_ctx, auto_pathlock=False)
except Exception:
logger.warning(
"[TempUpload] cleanup remove failed kind=%s uri=%s elapsed_ms=%.1f",
diff --git a/openviking/service/reindex_executor.py b/openviking/service/reindex_executor.py
index b7717d801..7b6c0334e 100644
--- a/openviking/service/reindex_executor.py
+++ b/openviking/service/reindex_executor.py
@@ -34,15 +34,17 @@ from openviking.storage.abstract_overview import body_for_preview, embedding_tex
from openviking.storage.errors import ResourceBusyError
from openviking.storage.expr import And, Eq, Or, PathScope
from openviking.storage.queuefs.embedding_msg_converter import EmbeddingMsgConverter
-from openviking.storage.queuefs.semantic_dag import SemanticDagExecutor
+from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor
from openviking.storage.queuefs.semantic_msg import SemanticMsg
from openviking.storage.queuefs.semantic_processor import SemanticProcessor
from openviking.storage.viking_fs import get_viking_fs
from openviking.telemetry import get_current_telemetry
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
+from openviking.utils.content_hash import content_md5
from openviking.utils.embedding_input import truncate_embedding_input
from openviking.utils.embedding_utils import (
_apply_ingest_options,
+ _decode_text_bytes,
_truncate_abstract_bytes,
get_resource_content_type,
vectorize_directory_meta,
@@ -999,7 +1001,7 @@ class ReindexExecutor:
ingest_options=run.ingest_options,
)
return
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="skill",
max_concurrent_llm=processor.max_concurrent_llm,
@@ -1230,8 +1232,16 @@ class ReindexExecutor:
async def process_file(file_uri: str) -> _ReindexCounters:
file_counters = _ReindexCounters(scanned_records=1)
parent_uri = VikingURI(file_uri).parent.uri
+ try:
+ file_bytes = await get_viking_fs().read_file_bytes(file_uri, ctx=ctx)
+ except Exception as exc:
+ file_counters.failed_records += 1
+ file_counters.warnings.append(f"Failed to read {file_uri} for reindex: {exc}")
+ return file_counters
summary = await self._best_file_summary(file_uri, ctx=ctx)
- vector_text = await self._best_resource_file_vector_text(file_uri, summary, ctx=ctx)
+ vector_text = await self._best_resource_file_vector_text(
+ file_uri, summary, ctx=ctx, file_content=file_bytes
+ )
if not vector_text:
file_counters.unsupported_records += 1
file_counters.warnings.append(f"No vector source found for {file_uri}")
@@ -1248,6 +1258,7 @@ class ReindexExecutor:
level=ContextLevel.DETAIL,
ctx=ctx,
ingest_options=ingest_options,
+ md5=content_md5(file_bytes),
)
file_counters.rebuilt_records += 1
except Exception as exc:
@@ -1570,7 +1581,7 @@ class ReindexExecutor:
if not recursive:
continue
- from openviking.storage.queuefs.semantic_dag import _SKIP_FILENAMES
+ from openviking.storage.queuefs.semantic_executor import _SKIP_FILENAMES
from openviking.storage.viking_fs import LS_ALL_NODES
entries = await viking_fs.ls(directory_uri, node_limit=LS_ALL_NODES, ctx=ctx)
@@ -1824,6 +1835,7 @@ class ReindexExecutor:
uri: str,
summary: str,
ctx: RequestContext,
+ file_content: bytes | None = None,
) -> str:
existing = await self._fetch_existing_record(
uri=uri,
@@ -1838,7 +1850,11 @@ class ReindexExecutor:
text_source = embedding_config.text_source
if text_source in SUMMARY_TEXT_SOURCES and summary:
return summary
- content = await self._safe_read_text(uri, ctx=ctx)
+ content = (
+ _decode_text_bytes(file_content)
+ if file_content is not None
+ else await self._safe_read_text(uri, ctx=ctx)
+ )
if content:
return truncate_embedding_input(content, embedding_config.max_input_tokens)
return summary or fallback
@@ -1860,6 +1876,7 @@ class ReindexExecutor:
ctx: RequestContext,
meta: Optional[dict[str, Any]] = None,
ingest_options: IngestOptions | None = None,
+ md5: str | None = None,
) -> None:
service = get_service()
assert service.vikingdb_manager is not None
@@ -1877,6 +1894,7 @@ class ReindexExecutor:
account_id=owner_ctx.account_id,
owner_space=owner_space_for_uri(uri),
meta=merged_meta,
+ md5=md5,
)
context.set_vectorize(Vectorize(text=vector_text))
msg = EmbeddingMsgConverter.from_context(context)
diff --git a/openviking/service/resource_service.py b/openviking/service/resource_service.py
index 886bd0f9b..d2c20d64f 100644
--- a/openviking/service/resource_service.py
+++ b/openviking/service/resource_service.py
@@ -93,6 +93,7 @@ from openviking_cli.utils import get_logger
if TYPE_CHECKING:
from openviking.connector.delegate import ConnectorDelegate
from openviking.parse.accessors.base import LocalResource
+ from openviking.resource.shared_source import SharedSource
from openviking.resource.staged_source import StagedSource
from openviking.resource.watch_manager import WatchManager, WatchTask
from openviking.resource.watch_scheduler import WatchScheduler
@@ -187,6 +188,7 @@ class _SourcePlan:
processor_args: Dict[str, Any]
task_auth: Dict[str, Any] = field(repr=False)
staged_source: Optional["StagedSource"] = None
+ shared_source: Optional["SharedSource"] = None
understanding_response_id: Optional[str] = None
understanding_file_id: Optional[str] = None
defer_unnamed_target: bool = False
@@ -768,11 +770,24 @@ class ResourceService:
internal_kwargs["_feishu_checkpoint"] = (saved, save_response)
prepared_resource = None
if msg.staged_source is not None:
- prepared_resource = await materialize_source(
- StagedSource.from_dict(msg.staged_source),
- viking_fs=self._viking_fs,
- ctx=ctx,
+ with get_current_telemetry().measure("resource.source_prepare"):
+ prepared_resource = await materialize_source(
+ StagedSource.from_dict(msg.staged_source),
+ viking_fs=self._viking_fs,
+ ctx=ctx,
+ )
+ elif msg.shared_source is not None:
+ from openviking.resource.shared_source import (
+ SharedSource,
+ materialize_shared_source,
)
+
+ with get_current_telemetry().measure("resource.source_prepare"):
+ prepared_resource = await materialize_shared_source(
+ SharedSource.from_dict(msg.shared_source),
+ viking_fs=self._viking_fs,
+ ctx=ctx,
+ )
if msg.defer_target_resolution:
from openviking_cli.utils.uri import VikingURI
@@ -834,7 +849,7 @@ class ResourceService:
ctx=ctx,
resource_lock=resource_lock,
)
- if msg.staged_source is not None:
+ if msg.staged_source is not None or msg.shared_source is not None:
result["source_path"] = msg.source_path
stage_result = stage_callback("processing_queue")
if inspect.isawaitable(stage_result):
@@ -914,12 +929,39 @@ class ResourceService:
allow_local_path_resolution: bool,
processor_kwargs: Dict[str, Any],
watch_auth_state: Optional[Dict[str, Any]],
+ shared_source: Optional["SharedSource"] = None,
) -> Optional[_SourcePlan]:
"""Freeze one durable standard-pipeline source before it crosses QueueFS."""
from openviking.parse.accessors.feishu_accessor import FeishuAccessor
from openviking.resource.staged_source import stage_source
source_name = processor_kwargs.get("source_name")
+ # A shared upload is already durable; reference it directly instead of
+ # downloading + re-staging a second copy. Its identity comes from the
+ # validated upload meta, so no accessor preflight is needed.
+ if shared_source is not None:
+ queued_args = {
+ key: value
+ for key, value in processor_kwargs.items()
+ if key not in _ADD_RESOURCE_ARGS_RESERVED_FIELDS | _ADD_RESOURCE_TRANSIENT_ARGS
+ }
+ queued_args = self._sanitize_watch_processor_kwargs(queued_args)
+ resolved_name = source_name or shared_source.original_filename or None
+ source_format = (
+ Path(shared_source.original_filename).suffix.lower().lstrip(".") or "file"
+ )
+ return _SourcePlan(
+ path=path,
+ source_identity=_ResourceSourceInfo(
+ source_name=resolved_name,
+ source_path=shared_source.original_filename or path,
+ source_format=source_format,
+ ),
+ processor_args=queued_args,
+ task_auth={},
+ shared_source=shared_source,
+ )
+
git_source = is_git_repo_url(path)
feishu_source = FeishuAccessor._is_feishu_url(path)
remote_source = is_remote_resource_source(path)
@@ -953,11 +995,7 @@ class ResourceService:
if request_git_auth is not None:
task_auth = create_git_http_auth_state(request_git_auth, path)
preflight_git_auth = request_git_auth
- if (
- preflight_git_auth is None
- and is_git_https_url(path)
- and is_github_url(path)
- ):
+ if preflight_git_auth is None and is_git_https_url(path) and is_github_url(path):
github_token = await self._resource_processor.github_token_for(path, ctx)
if github_token:
preflight_git_auth = GitHttpAuthConfig(
@@ -1196,6 +1234,9 @@ class ResourceService:
staged_source=(
plan.staged_source.to_dict() if plan.staged_source is not None else None
),
+ shared_source=(
+ plan.shared_source.to_dict() if plan.shared_source is not None else None
+ ),
telemetry_id=get_current_telemetry().telemetry_id or None,
account_id=ctx.account_id,
user_id=ctx.user.user_id,
@@ -1217,10 +1258,14 @@ class ResourceService:
tags=tags,
tag_mode=tag_mode,
allow_local_path_resolution=(
- True if plan.staged_source is not None else allow_local_path_resolution
+ True
+ if (plan.staged_source is not None or plan.shared_source is not None)
+ else allow_local_path_resolution
),
enforce_public_remote_targets=(
- enforce_public_remote_targets and plan.staged_source is None
+ enforce_public_remote_targets
+ and plan.staged_source is None
+ and plan.shared_source is None
),
strict=bool(processor_kwargs.get("strict", False)),
ignore_dirs=processor_kwargs.get("ignore_dirs"),
@@ -1433,6 +1478,7 @@ class ResourceService:
add_type: Optional[str] = None,
internal_task: bool = False,
args: Optional[Dict[str, Any]] = None,
+ shared_source: Optional["SharedSource"] = None,
**kwargs,
) -> Dict[str, Any]:
"""Accept and route a new resource-add request."""
@@ -1466,6 +1512,7 @@ class ResourceService:
enforce_public_remote_targets=enforce_public_remote_targets,
internal_task=internal_task,
args=args,
+ shared_source=shared_source,
**kwargs,
)
@@ -1540,6 +1587,7 @@ class ResourceService:
internal_task: bool = False,
args: Optional[Dict[str, Any]] = None,
connector_states: Optional[Dict[str, Any]] = None,
+ shared_source: Optional["SharedSource"] = None,
**kwargs,
) -> Dict[str, Any]:
"""Validate and route one resource ingestion request.
@@ -1836,6 +1884,7 @@ class ResourceService:
allow_local_path_resolution=allow_local_path_resolution,
processor_kwargs=kwargs,
watch_auth_state=normalized_args.watch_auth_state,
+ shared_source=shared_source,
)
if source_plan is not None:
result = await self._enqueue_source_plan(
@@ -1959,7 +2008,6 @@ class ResourceService:
mode = normalize_parse_mode(parse_mode)
if mode is ParseMode.NO_SPLIT:
kwargs["parse_mode"] = mode.value
- request_start = time.perf_counter()
telemetry = get_current_telemetry()
telemetry_id = telemetry.telemetry_id
register_telemetry(telemetry)
@@ -2132,10 +2180,6 @@ class ResourceService:
finally:
if prepared_resource is not None:
prepared_resource.cleanup()
- telemetry.set(
- "resource.request.duration_ms",
- round((time.perf_counter() - request_start) * 1000, 3),
- )
if not telemetry_id or (defer_post_processing and not job_enqueued):
unregister_telemetry(telemetry_id)
if deferred_lock is not None:
diff --git a/openviking/storage/collection_schemas.py b/openviking/storage/collection_schemas.py
index d67bc562e..62a1fefe9 100644
--- a/openviking/storage/collection_schemas.py
+++ b/openviking/storage/collection_schemas.py
@@ -10,6 +10,7 @@ similar to how init_viking_fs encapsulates VikingFS initialization.
import asyncio
import hashlib
import json
+import logging
import threading
import time
from contextlib import nullcontext
@@ -26,7 +27,12 @@ from openviking.storage.errors import (
EmbeddingConfigurationError,
EmbeddingRebuildRequiredError,
)
-from openviking.storage.queuefs.embedding_msg import EmbeddingMsg
+from openviking.storage.index_action import FieldPatch, IndexAction
+from openviking.storage.queuefs.embedding_msg import (
+ EmbeddingMsg,
+ IncompleteInitialRecordError,
+ missing_initial_record_fields,
+)
from openviking.storage.queuefs.named_queue import DequeueHandlerBase
from openviking.storage.queuefs.process_result import ProcessResult
from openviking.storage.vector_ids import vector_record_id
@@ -42,11 +48,13 @@ from openviking.utils.circuit_breaker import (
CircuitBreakerOpen,
classify_api_error,
)
+from openviking.utils.log_correlation import log_correlation
from openviking.utils.model_retry import (
ERROR_CLASS_AUTH,
ERROR_CLASS_INPUT_TOO_LARGE,
ERROR_CLASS_PERMANENT,
)
+from openviking.utils.time_utils import get_current_timestamp
from openviking_cli.session.user_id import UserIdentifier
from openviking_cli.utils import get_logger
from openviking_cli.utils.config.open_viking_config import OpenVikingConfig
@@ -120,6 +128,11 @@ class CollectionSchemas:
{"FieldName": "search_tags", "FieldType": "list<string>"},
{"FieldName": "abstract", "FieldType": "string"},
{"FieldName": "content", "FieldType": "text"},
+ # md5 of the final stored file bytes for this record's URI. Used by
+ # incremental diff to skip re-processing unchanged files. Older
+ # records may lack it; callers must treat missing/empty as "unknown"
+ # and fall back to reading file bytes.
+ {"FieldName": "md5", "FieldType": "string", "DefaultValue": ""},
{"FieldName": "account_id", "FieldType": "string"},
{"FieldName": "owner_user_id", "FieldType": "string"},
{
@@ -483,7 +496,6 @@ class TextEmbeddingHandler(DequeueHandlerBase):
config = get_openviking_config()
self._collection_name = config.storage.vectordb.name
self._vector_dim = config.embedding.dimension
- self._initialize_embedder(config)
breaker_cfg = config.embedding.circuit_breaker
self._circuit_breaker = CircuitBreaker(
failure_threshold=breaker_cfg.failure_threshold,
@@ -546,13 +558,23 @@ class TextEmbeddingHandler(DequeueHandlerBase):
@staticmethod
def _embedding_msg_log_context(embedding_msg: Optional[EmbeddingMsg]) -> str:
- """Return the URI allowed in embedding logs."""
+ """Return the URI-safe context retained in public error messages."""
if embedding_msg is None:
return "uri=<unknown>"
context_data = embedding_msg.context_data or {}
return f"uri={context_data.get('uri') or '<unknown>'}"
+ @classmethod
+ def _embedding_delivery_log_context(cls, embedding_msg: Optional[EmbeddingMsg]) -> str:
+ """Return request/message IDs plus the safe embedding URI for logs."""
+ if embedding_msg is None:
+ return f"{log_correlation()} {cls._embedding_msg_log_context(None)}"
+ return (
+ f"{log_correlation(telemetry_id=embedding_msg.telemetry_id, message_id=embedding_msg.id)} "
+ f"{cls._embedding_msg_log_context(embedding_msg)}"
+ )
+
@classmethod
def _embedding_error_msg(
cls,
@@ -561,6 +583,13 @@ class TextEmbeddingHandler(DequeueHandlerBase):
) -> str:
return f"{message} ({cls._embedding_msg_log_context(embedding_msg)})"
+ @classmethod
+ def _log_embedding_error(
+ cls, level: int, message: str, embedding_msg: Optional[EmbeddingMsg]
+ ) -> None:
+ """Log an internal correlation suffix without changing public errors."""
+ logger.log(level, "%s [%s]", message, cls._embedding_delivery_log_context(embedding_msg))
+
@staticmethod
async def _materialize_content(
embedding_msg: EmbeddingMsg,
@@ -601,6 +630,53 @@ class TextEmbeddingHandler(DequeueHandlerBase):
else inserted_data["abstract"][:VIKINGDB_CONTENT_MAX_SIZE]
)
+ async def _apply_non_embedding_operation(
+ self,
+ embedding_msg: EmbeddingMsg,
+ ctx: RequestContext,
+ ) -> Dict[str, Any]:
+ if embedding_msg.action is IndexAction.DELETE:
+ deleted_count = await self._vikingdb.strict_delete(
+ embedding_msg.record_ids,
+ ctx=ctx,
+ )
+ return {"deleted_count": deleted_count}
+
+ record_id = embedding_msg.record_ids[0]
+ field_patch = embedding_msg.field_patch
+ assert field_patch is not None
+ # UPDATE_FIELDS is a read-modify-write patch. The exact read happens at
+ # execution time so append semantics use the latest stored tags rather
+ # than the inventory snapshot that produced the durable plan.
+ existing_records = await self._vikingdb.get_strict([record_id], ctx=ctx)
+ if existing_records:
+ existing = dict(existing_records[0])
+ resolved = field_patch.resolve(existing)
+ changed_fields = {
+ field: resolved.get(field)
+ for field in field_patch.values
+ if resolved.get(field) != existing.get(field)
+ }
+ if not changed_fields:
+ return {"id": record_id, "status": "skipped"}
+ changed_fields["updated_at"] = get_current_timestamp()
+ updated_record = {"id": record_id, **changed_fields}
+ result = await self._vikingdb.update(updated_record, ctx=ctx)
+ if not result.ok:
+ raise RuntimeError(
+ result.error_message or f"failed to update vector record: {record_id}"
+ )
+ return updated_record
+
+ initial_record = field_patch.apply({**field_patch.seed_fields, "id": record_id})
+ missing_fields = missing_initial_record_fields(initial_record)
+ if missing_fields:
+ raise IncompleteInitialRecordError(record_id, missing_fields)
+ created_id = await self._vikingdb.upsert(initial_record, ctx=ctx)
+ if not created_id:
+ raise RuntimeError(f"failed to create missing vector record: {record_id}")
+ return {"id": record_id, "status": "created"}
+
async def on_dequeue(self, data: Optional[Dict[str, Any]]) -> ProcessResult:
"""Process dequeued message and add embedding vector(s)."""
if not data:
@@ -608,16 +684,23 @@ class TextEmbeddingHandler(DequeueHandlerBase):
embedding_msg: Optional[EmbeddingMsg] = None
request_failed_message: Optional[str] = None
+ execute_started_at: float | None = None
+ queue_wait_ms = 0.0
+ execute_status = "ok"
try:
embedding_msg = EmbeddingMsg.from_json(data["data"])
+ execute_started_at = time.perf_counter()
+ if embedding_msg.queue_enqueued_at > 0:
+ queue_wait_ms = max((time.time() - embedding_msg.queue_enqueued_at) * 1000.0, 0.0)
inserted_data = embedding_msg.context_data
+ logger.debug(
+ "Processing embedding message: %s action=%s",
+ self._embedding_delivery_log_context(embedding_msg),
+ embedding_msg.action.value,
+ )
account_id = inserted_data.get("account_id", "default")
context_user = inserted_data.get("user") or {}
- user_id = (
- context_user.get("user_id")
- or inserted_data.get("owner_user_id")
- or "default"
- )
+ user_id = context_user.get("user_id") or inserted_data.get("owner_user_id") or "default"
user = UserIdentifier(account_id=account_id, user_id=user_id)
ctx = RequestContext(user=user, role=Role.USER, bypass_acl=True)
collector = resolve_telemetry(embedding_msg.telemetry_id)
@@ -630,6 +713,18 @@ class TextEmbeddingHandler(DequeueHandlerBase):
self._record_request_success(embedding_msg)
return ProcessResult.success()
+ if embedding_msg.action is IndexAction.NONE:
+ self._merge_request_stats(embedding_msg.telemetry_id, processed=1)
+ self._record_request_success(embedding_msg)
+ logger.debug("Skipping no-op embedding message: %s", embedding_msg.id)
+ return ProcessResult.success()
+
+ if embedding_msg.action not in {IndexAction.UPSERT, IndexAction.MERGE}:
+ result = await self._apply_non_embedding_operation(embedding_msg, ctx)
+ self._merge_request_stats(embedding_msg.telemetry_id, processed=1)
+ self._record_request_success(embedding_msg)
+ return ProcessResult.success(result)
+
if not isinstance(embedding_msg.message, (str, list)):
logger.debug(
f"Skipping unsupported message type: {type(embedding_msg.message)}"
@@ -646,6 +741,7 @@ class TextEmbeddingHandler(DequeueHandlerBase):
except CircuitBreakerOpen:
self._log_breaker_open_reenqueue_summary()
if self._vikingdb.has_queue_manager:
+ execute_status = "requeued"
wait = self._circuit_breaker.retry_after
if wait > 0:
await asyncio.sleep(wait)
@@ -659,6 +755,7 @@ class TextEmbeddingHandler(DequeueHandlerBase):
)
return ProcessResult.requeued()
# No queue manager — cannot re-enqueue, drop with error
+ execute_status = "error"
error_msg = self._embedding_error_msg(
embedding_msg,
"Circuit breaker open and no queue manager",
@@ -711,32 +808,36 @@ class TextEmbeddingHandler(DequeueHandlerBase):
pass
if error_class == ERROR_CLASS_INPUT_TOO_LARGE:
- logger.error(error_msg)
+ execute_status = "error"
+ self._log_embedding_error(logging.ERROR, error_msg, embedding_msg)
self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)
request_failed_message = error_msg
return ProcessResult.failed(error_msg)
if error_class == ERROR_CLASS_PERMANENT:
- logger.critical(error_msg)
+ execute_status = "error"
+ self._log_embedding_error(logging.CRITICAL, error_msg, embedding_msg)
self._circuit_breaker.record_failure(embed_err)
self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)
request_failed_message = error_msg
return ProcessResult.failed(error_msg)
if error_class == ERROR_CLASS_AUTH:
+ execute_status = "error"
# Bad/expired credential: retrying cannot succeed. Fail
# terminally instead of re-enqueueing, which would cycle
# forever and hold this resource's tree lock and its
# add-resource --wait open. Don't trip the breaker: an open
# breaker re-enqueues later messages and reintroduces the
# same leak. See #2916.
- logger.error(error_msg)
+ self._log_embedding_error(logging.ERROR, error_msg, embedding_msg)
self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)
request_failed_message = error_msg
return ProcessResult.failed(error_msg)
# Transient or unknown — re-enqueue for retry
- logger.warning(error_msg)
+ self._log_embedding_error(logging.WARNING, error_msg, embedding_msg)
+ execute_status = "requeued"
self._circuit_breaker.record_failure(embed_err)
if self._vikingdb.has_queue_manager:
try:
@@ -749,8 +850,8 @@ class TextEmbeddingHandler(DequeueHandlerBase):
embedding_msg.telemetry_id
)
logger.info(
- "Re-enqueued embedding message after transient error "
- f"({self._embedding_msg_log_context(embedding_msg)})"
+ "Re-enqueued embedding message after transient error: %s",
+ self._embedding_delivery_log_context(embedding_msg),
)
return ProcessResult.requeued()
except Exception as requeue_err:
@@ -762,6 +863,7 @@ class TextEmbeddingHandler(DequeueHandlerBase):
)
self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)
+ execute_status = "error"
request_failed_message = error_msg
return ProcessResult.failed(error_msg)
@@ -770,12 +872,13 @@ class TextEmbeddingHandler(DequeueHandlerBase):
inserted_data["vector"] = result.dense_vector
# Validate vector dimension
if len(result.dense_vector) != self._vector_dim:
+ execute_status = "error"
error_msg = self._embedding_error_msg(
embedding_msg,
"Dense vector dimension mismatch: "
f"expected {self._vector_dim}, got {len(result.dense_vector)}",
)
- logger.error(error_msg)
+ self._log_embedding_error(logging.ERROR, error_msg, embedding_msg)
self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)
request_failed_message = error_msg
return ProcessResult.failed(error_msg)
@@ -791,7 +894,7 @@ class TextEmbeddingHandler(DequeueHandlerBase):
embedding_msg,
"Embedder not initialized, skipping vector generation",
)
- logger.warning(error_msg)
+ self._log_embedding_error(logging.WARNING, error_msg, embedding_msg)
try:
from openviking.metrics.datasources import EmbeddingEventDataSource
@@ -799,18 +902,21 @@ class TextEmbeddingHandler(DequeueHandlerBase):
except Exception:
pass
self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)
+ execute_status = "error"
request_failed_message = error_msg
return ProcessResult.failed(error_msg)
# Write to vector database
try:
raw_upsert_options = inserted_data.pop("_upsert_options", {})
- upsert_options = normalize_upsert_options(
- {**raw_upsert_options, "partial_update": True}
- )
- # Ensure vector DB has deterministic IDs per semantic layer.
+ # Reuse the actual vector-store ID when a semantic plan
+ # rebuilds an existing same-level record. Only genuinely new
+ # records derive an ID locally from (account, uri, level).
+ existing_record_id = inserted_data.pop("_upsert_record_id", None)
uri = inserted_data.get("uri")
- if uri:
+ if existing_record_id:
+ inserted_data["id"] = str(existing_record_id)
+ elif uri:
inserted_data["id"] = vector_record_id(
account_id, uri, inserted_data.get("level", 2)
)
@@ -820,6 +926,45 @@ class TextEmbeddingHandler(DequeueHandlerBase):
embedding_msg,
ctx,
)
+ if embedding_msg.action is IndexAction.MERGE:
+ field_patch = embedding_msg.field_patch
+ merge_fields = dict(field_patch.values) if field_patch is not None else {}
+ merge_modes = dict(field_patch.modes) if field_patch is not None else {}
+ # Legacy MERGE producers encoded tag intent in context_data
+ # plus _upsert_options. Normalize them to the explicit patch
+ # protocol before the exact read.
+ if "search_tags" in inserted_data and "search_tags" not in merge_fields:
+ merge_fields["search_tags"] = inserted_data.pop("search_tags")
+ merge_modes["search_tags"] = str(
+ raw_upsert_options.get("search_tag_mode", "replace")
+ )
+ existing_records = await self._vikingdb.get_strict(
+ [inserted_data["id"]], ctx=ctx
+ )
+ base = (
+ dict(existing_records[0])
+ if existing_records
+ else dict(field_patch.seed_fields)
+ if field_patch is not None
+ else {}
+ )
+ # Fresh content/model outputs win over the stored record;
+ # scalar patches are then interpreted against that latest
+ # exact-get result.
+ inserted_data = FieldPatch(merge_fields, merge_modes).apply(
+ {**base, **inserted_data}
+ )
+ if not existing_records:
+ missing_fields = missing_initial_record_fields(inserted_data)
+ if missing_fields:
+ raise RuntimeError(
+ "merge could not create missing vector record: "
+ f"record_id={inserted_data.get('id')} "
+ f"missing_fields={missing_fields}"
+ )
+ upsert_options = normalize_upsert_options(
+ {**raw_upsert_options, "partial_update": False}
+ )
if inserted_data.get("context_type") == ContextType.SKILL.value:
# Cancelling the waiter cannot stop a threaded DB write.
# Keep this task active until that write has settled.
@@ -836,7 +981,10 @@ class TextEmbeddingHandler(DequeueHandlerBase):
)
record_id = result
if record_id:
- logger.debug("Successfully wrote embedding: uri=%s", uri)
+ logger.debug(
+ "Successfully wrote embedding: %s",
+ self._embedding_delivery_log_context(embedding_msg),
+ )
except CollectionNotFoundError as db_err:
# During shutdown, queue workers may finish one dequeued item.
if self._vikingdb.is_closing:
@@ -848,11 +996,12 @@ class TextEmbeddingHandler(DequeueHandlerBase):
embedding_msg,
f"Failed to write to vector database: {db_err}",
)
- logger.error(error_msg)
+ self._log_embedding_error(logging.ERROR, error_msg, embedding_msg)
import traceback
traceback.print_exc()
self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)
+ execute_status = "error"
request_failed_message = error_msg
return ProcessResult.failed(error_msg)
except Exception as db_err:
@@ -865,8 +1014,9 @@ class TextEmbeddingHandler(DequeueHandlerBase):
embedding_msg,
f"Failed to write to vector database: {db_err}",
)
- logger.error(error_msg)
+ self._log_embedding_error(logging.ERROR, error_msg, embedding_msg)
self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)
+ execute_status = "error"
request_failed_message = error_msg
return ProcessResult.failed(error_msg)
@@ -898,9 +1048,33 @@ class TextEmbeddingHandler(DequeueHandlerBase):
traceback.print_exc()
if embedding_msg is not None:
self._merge_request_stats(embedding_msg.telemetry_id, error_count=1)
+ execute_status = "error"
request_failed_message = error_msg
return ProcessResult.failed(error_msg)
finally:
+ if embedding_msg is not None and execute_started_at is not None:
+ tracker = get_request_wait_tracker()
+ record_timing = getattr(tracker, "record_embedding_timing", None)
+ if callable(record_timing):
+ record_timing(
+ embedding_msg.telemetry_id,
+ queue_wait_ms=queue_wait_ms,
+ execute_ms=(time.perf_counter() - execute_started_at) * 1000.0,
+ )
+ from openviking.metrics.datasources.resource import ResourceIngestionEventDataSource
+
+ ResourceIngestionEventDataSource.record_stage(
+ stage="embedding_queue_wait",
+ status=execute_status,
+ duration_seconds=queue_wait_ms / 1000.0,
+ account_id=embedding_msg.context_data.get("account_id"),
+ )
+ ResourceIngestionEventDataSource.record_stage(
+ stage="embedding_execute",
+ status=execute_status,
+ duration_seconds=(time.perf_counter() - execute_started_at),
+ account_id=embedding_msg.context_data.get("account_id"),
+ )
if embedding_msg is not None and request_failed_message is not None:
self._record_request_failure(embedding_msg, request_failed_message)
diff --git a/openviking/storage/content_write.py b/openviking/storage/content_write.py
index a236d20ff..56ae89ce6 100644
--- a/openviking/storage/content_write.py
+++ b/openviking/storage/content_write.py
@@ -46,6 +46,7 @@ from openviking.storage.viking_fs import VikingFS
from openviking.telemetry import get_current_telemetry
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
from openviking.telemetry.resource_summary import build_queue_status_payload
+from openviking.utils.content_hash import content_md5
from openviking.utils.embedding_utils import vectorize_directory_meta, vectorize_file
from openviking.utils.ingest_options import IngestOptions
from openviking.utils.path_safety import validate_safe_viking_uri_path
@@ -242,6 +243,8 @@ class ContentWriteCoordinator:
updated: list[str] = []
unchanged: list[str] = []
refresh_kinds: dict[str, str] = {}
+ file_md5s: dict[str, str] = {}
+ file_abstracts: dict[str, str] = {}
sidecar_directories: set[str] = set()
pending: list[tuple[dict[str, Any], bool, str]] = []
write_error: Exception | None = None
@@ -270,16 +273,32 @@ class ContentWriteCoordinator:
raise NotFoundError(uri, "file")
pending.append((operation, exists, write_mode))
+ file_abstracts = await self._load_file_abstracts(
+ [
+ operation["uri"]
+ for operation, existed, _ in pending
+ if existed
+ and context_type_for_uri(operation["uri"]) in {"resource", "skill"}
+ and not is_abstract_overview_uri(operation["uri"])
+ ],
+ ctx=ctx,
+ )
+
for operation, existed, write_mode in pending:
uri = operation["uri"]
try:
- await self._write_in_place(
+ final_content = await self._write_in_place(
uri,
operation["content"],
mode=write_mode,
ctx=ctx,
lease_ref=lease,
)
+ if context_type_for_uri(uri) in {
+ "resource",
+ "skill",
+ } and not is_abstract_overview_uri(uri):
+ file_md5s[uri] = content_md5(final_content)
except Exception as exc:
write_error = exc
break
@@ -319,6 +338,8 @@ class ContentWriteCoordinator:
wait=wait,
timeout=timeout,
telemetry_id=telemetry_id,
+ file_md5s=file_md5s,
+ file_abstracts=file_abstracts,
)
if refresh_kinds
else (
@@ -506,6 +527,8 @@ class ContentWriteCoordinator:
wait: bool,
timeout: Optional[float],
telemetry_id: str,
+ file_md5s: dict[str, str] | None = None,
+ file_abstracts: dict[str, str] | None = None,
) -> _BatchRefreshOutcome:
resource_groups: dict[tuple[str, str], dict[str, list[str]]] = defaultdict(
lambda: {"added": [], "modified": []}
@@ -528,6 +551,18 @@ class ContentWriteCoordinator:
changes=changes,
ctx=ctx,
force_refresh=wait,
+ file_md5s={
+ uri: file_md5s[uri]
+ for values in changes.values()
+ for uri in values
+ if file_md5s and uri in file_md5s
+ },
+ file_abstracts={
+ uri: file_abstracts[uri]
+ for values in changes.values()
+ for uri in values
+ if file_abstracts and uri in file_abstracts
+ },
)
semantic_actions.append(action)
@@ -580,6 +615,8 @@ class ContentWriteCoordinator:
recursive: bool = False,
force_refresh: bool = False,
ingest_options: IngestOptions | None = None,
+ file_md5s: dict[str, str] | None = None,
+ file_abstracts: dict[str, str] | None = None,
) -> FreshnessAction:
changed_entries = len({uri for values in changes.values() for uri in values})
semantic_config = get_openviking_config().semantic
@@ -634,6 +671,8 @@ class ContentWriteCoordinator:
},
generation_trigger="content_write",
aggregate_directory=aggregate_directory,
+ file_md5s=file_md5s,
+ file_abstracts=file_abstracts,
)
if msg.telemetry_id:
get_request_wait_tracker().register_semantic_root(msg.telemetry_id, msg.id)
@@ -798,6 +837,8 @@ class ContentWriteCoordinator:
) from exc
previous_content: Optional[str] = None
+ final_content = content.encode("utf-8")
+ file_abstract = ""
content_written = False
post_process_started = False
lock_released = False
@@ -810,9 +851,15 @@ class ContentWriteCoordinator:
raise InvalidArgumentError(
f"cannot create generated abstract overview directly: {uri}"
)
+ if (
+ mode != "create"
+ and not is_abstract_overview_uri(uri)
+ and processing_mode != VECTORS_ONLY
+ ):
+ file_abstract = (await self._load_file_abstracts([uri], ctx=ctx)).get(uri, "")
if wait and telemetry_id:
get_request_wait_tracker().register_request(telemetry_id)
- await self._write_in_place(
+ written_content = await self._write_in_place(
uri,
content,
mode=mode,
@@ -820,6 +867,8 @@ class ContentWriteCoordinator:
lease_ref=lease,
existing_raw=previous_content,
)
+ if written_content is not None:
+ final_content = written_content
content_written = True
if is_abstract_overview_uri(uri):
vector_enqueued = await self._vectorize_abstract_overview(
@@ -833,6 +882,7 @@ class ContentWriteCoordinator:
ctx=ctx,
creator_acl_grant=(CreatorAclGrant.DIRECT if mode == "create" else None),
ingest_options=ingest_options,
+ file_md5=content_md5(final_content),
)
post_process_started = True
else:
@@ -844,6 +894,8 @@ class ContentWriteCoordinator:
change_type="added" if mode == "create" else "modified",
force_refresh=wait,
ingest_options=ingest_options,
+ file_md5=content_md5(final_content),
+ file_abstract=file_abstract,
)
post_process_started = True
await self._viking_fs._async_agfs.pathlock_release(lease)
@@ -890,7 +942,7 @@ class ContentWriteCoordinator:
uri=uri,
root_uri=root_uri,
context_type=context_type,
- mode=mode,
+ mode=response_mode or mode,
written_bytes=written_bytes,
wait=wait,
queue_status=queue_status,
@@ -943,6 +995,7 @@ class ContentWriteCoordinator:
ctx: RequestContext,
creator_acl_grant: CreatorAclGrant | None = None,
ingest_options: IngestOptions | None = None,
+ file_md5: str | None = None,
) -> bool:
parent = VikingURI(uri).parent
if parent is None:
@@ -956,6 +1009,7 @@ class ContentWriteCoordinator:
ctx=ctx,
creator_acl_grant=creator_acl_grant,
ingest_options=ingest_options,
+ file_md5=file_md5,
)
async def _vectorize_abstract_overview(
@@ -1121,13 +1175,13 @@ class ContentWriteCoordinator:
async def _write_in_place(
self,
uri: str,
- content: str,
+ content: str | bytes,
*,
mode: str,
ctx: RequestContext,
lease_ref: Optional[Dict[str, Any]] = None,
- existing_raw: Optional[str] = None,
- ) -> None:
+ existing_raw: str | bytes | None = None,
+ ) -> bytes:
if is_abstract_overview_uri(uri):
current_raw = (
existing_raw
@@ -1141,7 +1195,7 @@ class ContentWriteCoordinator:
mode=mode,
)
await self._viking_fs.write_file(uri, rendered, ctx=ctx, lease_ref=lease_ref)
- return
+ return rendered.encode("utf-8")
if context_type_for_uri(uri) == "memory":
if mode == "replace":
@@ -1155,25 +1209,50 @@ class ContentWriteCoordinator:
else:
mf = MemoryFileUtils.read(content, uri=uri)
sync_memory_resource_refs(mf, source=RESOURCE_REF_SOURCE_CONTENT_WRITE)
+ rendered = MemoryFileUtils.write(mf)
await self._viking_fs.write_file(
uri,
- MemoryFileUtils.write(mf),
+ rendered,
ctx=ctx,
lease_ref=lease_ref,
)
- return
+ return rendered.encode("utf-8")
if mode == "append":
# Plain concatenation for resource/skill files: MEMORY_FIELDS is a
# reserved trailer of memory namespaces only (see content_visibility),
# so non-memory appends must not round-trip through MemoryFileUtils
# (which strips trailing newlines and injects a metadata trailer).
- existing_raw = await self._viking_fs.read_file(uri, ctx=ctx)
- await self._viking_fs.write_file(
- uri, existing_raw + content, ctx=ctx, lease_ref=lease_ref
+ existing_raw = (
+ existing_raw
+ if existing_raw is not None
+ else await self._viking_fs.read_file(uri, ctx=ctx)
)
- return
+ if not isinstance(existing_raw, str) or not isinstance(content, str):
+ raise InvalidArgumentError(f"append only supports text content: {uri}")
+ final_content = existing_raw + content
+ await self._viking_fs.write_file(uri, final_content, ctx=ctx, lease_ref=lease_ref)
+ return final_content.encode("utf-8")
await self._viking_fs.write_file(uri, content, ctx=ctx, lease_ref=lease_ref)
+ return content if isinstance(content, bytes) else content.encode("utf-8")
+
+ async def _load_file_abstracts(self, uris: list[str], *, ctx: RequestContext) -> dict[str, str]:
+ vector_store = self._vikingdb
+ if vector_store is None:
+ get_vector_store = getattr(self._viking_fs, "_get_vector_store", None)
+ vector_store = get_vector_store() if callable(get_vector_store) else None
+ if vector_store is None or not uris:
+ return {}
+ try:
+ records = await vector_store.get_l2_diff_records_by_uris(sorted(set(uris)), ctx=ctx)
+ except Exception as exc:
+ logger.warning("Failed to load existing file abstracts: %s", exc)
+ return {}
+ return {
+ uri: str(record.get("abstract") or "")
+ for uri, record in records.items()
+ if uri in uris and record.get("abstract") is not None
+ }
@staticmethod
def _prepare_abstract_overview_content(
@@ -1196,6 +1275,8 @@ class ContentWriteCoordinator:
recursive: bool = False,
force_refresh: bool = False,
ingest_options: IngestOptions | None = None,
+ file_md5: str | None = None,
+ file_abstract: str = "",
) -> FreshnessAction:
return await self._enqueue_semantic_refresh_changes(
root_uri=root_uri,
@@ -1206,6 +1287,8 @@ class ContentWriteCoordinator:
recursive=recursive,
force_refresh=force_refresh,
ingest_options=ingest_options,
+ file_md5s={changed_uri: file_md5} if file_md5 else None,
+ file_abstracts={changed_uri: file_abstract} if file_abstract else None,
)
async def _wait_for_queues(self, *, timeout: Optional[float]) -> Dict[str, Any]:
@@ -1523,7 +1606,7 @@ class ContentWriteCoordinator:
if anchor_to_parent:
# Content writes anchor the semantic refresh at the written file's
# direct parent directory, so the changed file is a direct child of
- # the DAG run root: its own L2 vector and the parent's L0/L1 are
+ # the semantic-tree root: its own L2 vector and the parent's L0/L1 are
# (re)generated from a single-directory run, while ancestor summaries
# refresh via the existing parent bubble. Collapsing to the project
# root instead would force the run to traverse the whole project
diff --git a/openviking/storage/context_update_plan.py b/openviking/storage/context_update_plan.py
new file mode 100644
index 000000000..1c3de1253
--- /dev/null
+++ b/openviking/storage/context_update_plan.py
@@ -0,0 +1,1208 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Build and execute explicit content, semantic and index actions."""
+
+from __future__ import annotations
+
+import logging
+from dataclasses import asdict, dataclass, field
+from enum import Enum
+from pathlib import PurePosixPath
+from typing import Any, Mapping
+
+from openviking.concurrency import bounded_map
+from openviking.storage.index_action import FieldPatch, IndexAction
+from openviking.storage.resource_diff import ContentState, IndexState
+from openviking.storage.resource_rnfv import (
+ NON_PORTABLE_VECTOR_RECORD_FIELDS,
+ FormalEntry,
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ NewEntry,
+ RequestIntent,
+ RNFVSnapshot,
+ VectorRecordSnapshot,
+ canonical_vector_records_by_level,
+)
+from openviking.storage.vector_ids import vector_record_id
+from openviking.utils.ingest_options import IngestOptions
+from openviking.utils.log_correlation import log_correlation
+from openviking_cli.utils import VikingURI
+
+logger = logging.getLogger(__name__)
+
+
+class SemanticAction(str, Enum):
+ """Per-node semantic work executed by the minimal semantic tree."""
+
+ # Consume a hydrated existing abstract; do not call a semantic model.
+ REUSE = "reuse"
+ # Generate a file abstract from the current formal file contents.
+ GENERATE = "generate"
+ # Generate directory L0/L1 bodies from direct child abstracts.
+ AGGREGATE = "aggregate"
+
+
+class FileVectorSource(str, Enum):
+ """Input selected for a file's embedding request."""
+
+ # Embed the current file body using the configured text-source policy.
+ CONTENT = "content"
+ # Prefer the generated file summary when one is available (code repositories).
+ SUMMARY_WHEN_AVAILABLE = "summary_when_available"
+
+
+@dataclass(frozen=True)
+class ParentPropagation:
+ enabled: bool = True
+
+
+@dataclass(frozen=True)
+class FileRefreshIntent:
+ """Refresh a flat file and its parent after synchronous content commit."""
+
+ file_uri: str
+ md5: str | None = None
+
+ def __post_init__(self) -> None:
+ if not self.file_uri.startswith("viking://"):
+ raise ValueError("file refresh requires a Viking URI")
+
+
+def _validate_relative_path(value: str) -> str:
+ if not isinstance(value, str):
+ raise ValueError("relative_path must be a string")
+ if not value:
+ return value
+ path = PurePosixPath(value)
+ if path.is_absolute() or value != str(path) or any(part == ".." for part in path.parts):
+ raise ValueError(f"invalid relative path: {value}")
+ return value
+
+
+def _validate_index_fields(fields: Mapping[str, Any]) -> None:
+ if set(fields) & {"id", "uri", "level", "vector", "sparse_vector"}:
+ raise ValueError("index fields contain identity or vector payload")
+
+
+@dataclass(frozen=True)
+class IndexSlot:
+ """Planned operation for one semantic node and one vector level.
+
+ Existing record identity and portable scalar fields travel with the slot so a
+ later embedding worker can update the real stored record rather than derive a
+ replacement ID locally. ``DELETE`` and pure ``UPDATE_FIELDS`` actions never
+ belong here because they do not depend on semantic output.
+ """
+
+ level: int
+ record_id: str
+ existing_fields: Mapping[str, Any] | None = None
+ action: IndexAction = IndexAction.NONE
+ upsert_fields: Mapping[str, Any] = field(default_factory=dict)
+ field_patch: FieldPatch | None = None
+ fallback_to_patch: bool = False
+
+ def __post_init__(self) -> None:
+ object.__setattr__(self, "action", IndexAction(self.action))
+ if self.level not in {0, 1, 2} or not self.record_id:
+ raise ValueError("index slot requires a level and record ID")
+ if self.action is IndexAction.DELETE:
+ raise ValueError("delete belongs to direct index actions")
+ if self.action is IndexAction.UPDATE_FIELDS:
+ raise ValueError("semantic index slots only supports none, upsert, or merge")
+ _validate_index_fields(self.existing_fields or {})
+ _validate_index_fields(self.upsert_fields)
+ if self.action is IndexAction.NONE and (self.upsert_fields or self.field_patch):
+ raise ValueError("none index slot cannot carry index mutations")
+ if self.fallback_to_patch and self.field_patch is None:
+ raise ValueError("fallback_to_patch requires a field patch")
+ if self.action is IndexAction.MERGE and self.upsert_fields:
+ raise ValueError("merge index slot uses field_patch, not upsert_fields")
+
+ @property
+ def abstract(self) -> str:
+ return str((self.existing_fields or {}).get("abstract") or "")
+
+ def scalar_override(self) -> dict[str, Any]:
+ existing = {
+ key: value
+ for key, value in (self.existing_fields or {}).items()
+ if key not in NON_PORTABLE_VECTOR_RECORD_FIELDS and not key.startswith("_")
+ }
+ return {
+ **({} if self.action is IndexAction.MERGE else existing),
+ **dict(self.upsert_fields),
+ "_record_id": self.record_id,
+ }
+
+ @classmethod
+ def from_dict(cls, data: Mapping[str, Any]) -> "IndexSlot":
+ values = dict(data)
+ if any(
+ name in values
+ for name in ("fields", "update_fields", "field_modes", "fallback_update_fields")
+ ):
+ legacy_fields = dict(values.pop("fields", {}) or {})
+ legacy_update = dict(values.pop("update_fields", {}) or {})
+ legacy_modes = dict(values.pop("field_modes", {}) or {})
+ action = IndexAction(values.get("action", IndexAction.NONE))
+ values.setdefault("upsert_fields", {} if action is IndexAction.MERGE else legacy_fields)
+ patch_values = legacy_update or (legacy_fields if action is IndexAction.MERGE else {})
+ values.setdefault(
+ "field_patch",
+ FieldPatch(patch_values, legacy_modes) if patch_values else None,
+ )
+ legacy_fallback = bool(values.pop("fallback_update_fields", False))
+ # Older planners marked every modified semantic node for fallback,
+ # even when there was no scalar patch to apply. Normalize that
+ # inert state while decoding already queued plans.
+ values["fallback_to_patch"] = legacy_fallback and bool(patch_values)
+ values.setdefault("upsert_fields", {})
+ if isinstance(values.get("field_patch"), Mapping):
+ values["field_patch"] = FieldPatch.from_dict(values["field_patch"])
+ return cls(**values)
+
+
+@dataclass(frozen=True)
+class SemanticTreeEntry:
+ """A retained node in the serializable minimal semantic-tree closure.
+
+ ``REUSE`` entries provide existing child abstracts needed by an ancestor but
+ do not run work themselves. Active file entries ``GENERATE`` and active
+ directory entries ``AGGREGATE``.
+ """
+
+ relative_path: str
+ kind: str
+ content_state: ContentState
+ semantic_action: SemanticAction
+ md5: str | None = None
+ index_slots: tuple[IndexSlot, ...] = ()
+ membership_changed: bool = False
+ repair: bool = False
+
+ def __post_init__(self) -> None:
+ _validate_relative_path(self.relative_path)
+ object.__setattr__(self, "content_state", ContentState(self.content_state))
+ object.__setattr__(self, "semantic_action", SemanticAction(self.semantic_action))
+ if self.kind not in {"file", "directory"}:
+ raise ValueError("invalid semantic entry kind")
+ if self.kind == "file" and self.semantic_action is SemanticAction.AGGREGATE:
+ raise ValueError("file semantic action must be generate or reuse")
+ if self.kind == "directory" and self.semantic_action is SemanticAction.GENERATE:
+ raise ValueError("directory semantic action must be aggregate or reuse")
+ if self.kind == "directory" and self.md5:
+ raise ValueError("directory entry cannot carry md5")
+ levels = [slot.level for slot in self.index_slots]
+ if len(levels) != len(set(levels)):
+ raise ValueError(f"duplicate index slot level for {self.relative_path}")
+ if self.semantic_action is SemanticAction.REUSE:
+ if any(slot.action is not IndexAction.NONE for slot in self.index_slots):
+ raise ValueError("reuse semantic entry cannot contain index mutations")
+ slot = self.slot(2 if self.kind == "file" else 0)
+ if slot is None or not slot.abstract.strip():
+ raise ValueError("reuse requires an existing abstract")
+
+ def slot(self, level: int) -> IndexSlot | None:
+ return next((slot for slot in self.index_slots if slot.level == level), None)
+
+ @classmethod
+ def from_dict(cls, data: Mapping[str, Any]) -> "SemanticTreeEntry":
+ values = dict(data)
+ values["index_slots"] = tuple(
+ IndexSlot.from_dict(item) for item in values.get("index_slots", ())
+ )
+ return cls(**values)
+
+
+@dataclass(frozen=True)
+class SemanticTreeSnapshot:
+ """Compact tree retained after diff closure and vector hydration."""
+
+ entries: tuple[SemanticTreeEntry, ...]
+
+ def __post_init__(self) -> None:
+ paths = [entry.relative_path for entry in self.entries]
+ if len(paths) != len(set(paths)):
+ raise ValueError("duplicate semantic tree entry path")
+
+ @classmethod
+ def from_dict(cls, data: Mapping[str, Any]) -> "SemanticTreeSnapshot":
+ return cls(
+ entries=tuple(SemanticTreeEntry.from_dict(item) for item in data.get("entries", ()))
+ )
+
+
+@dataclass(frozen=True)
+class SemanticPlan:
+ """Asynchronous semantic/derived-index portion of one context update.
+
+ The plan contains only final resource URIs and hydrated non-vector facts. It
+ must never require a parser artifact, local file path, or another RNFV scan
+ after it has crossed the durable semantic queue boundary.
+ """
+
+ root_uri: str
+ context_type: str
+ tree: SemanticTreeSnapshot
+ vectorize: bool = True
+ propagation: ParentPropagation = field(default_factory=ParentPropagation)
+ file_vector_source: FileVectorSource = FileVectorSource.CONTENT
+ ingest_options: IngestOptions = field(default_factory=IngestOptions)
+ source_metadata: Mapping[str, str] | None = None
+
+ def __post_init__(self) -> None:
+ object.__setattr__(self, "root_uri", VikingURI(self.root_uri).uri.rstrip("/"))
+ object.__setattr__(self, "file_vector_source", FileVectorSource(self.file_vector_source))
+ object.__setattr__(self, "ingest_options", IngestOptions.from_value(self.ingest_options))
+ if self.context_type not in {"resource", "skill", "memory"}:
+ raise ValueError(f"invalid context_type: {self.context_type}")
+ entries_by_path = {entry.relative_path: entry for entry in self.tree.entries}
+ if entries_by_path and "" not in entries_by_path:
+ raise ValueError("semantic plan must contain the resource root")
+ for entry in entries_by_path.values():
+ if not entry.relative_path:
+ continue
+ parent = _parent(entry.relative_path)
+ if parent not in entries_by_path:
+ raise ValueError(
+ f"semantic plan lacks parent {parent!r} for {entry.relative_path!r}"
+ )
+ current = parent
+ while True:
+ if current not in entries_by_path:
+ raise ValueError(
+ f"semantic plan lacks ancestor {current!r} for {entry.relative_path!r}"
+ )
+ ancestor_entry = entries_by_path[current]
+ if (
+ entry.semantic_action is not SemanticAction.REUSE
+ and ancestor_entry.semantic_action is not SemanticAction.AGGREGATE
+ ):
+ raise ValueError(
+ f"semantic plan ancestor {current!r} must aggregate active descendant "
+ f"{entry.relative_path!r}"
+ )
+ if not current:
+ break
+ current = _parent(current)
+
+ def to_dict(self) -> dict[str, Any]:
+ data = asdict(self)
+ data["file_vector_source"] = self.file_vector_source.value
+ data["ingest_options"] = self.ingest_options.to_dict()
+ return data
+
+ @classmethod
+ def from_dict(cls, data: Mapping[str, Any]) -> "SemanticPlan":
+ return cls(
+ root_uri=str(data["root_uri"]),
+ context_type=str(data["context_type"]),
+ tree=SemanticTreeSnapshot.from_dict(data.get("tree", {})),
+ vectorize=bool(data.get("vectorize", True)),
+ propagation=ParentPropagation(**dict(data.get("propagation", {}))),
+ file_vector_source=FileVectorSource(
+ data.get("file_vector_source", FileVectorSource.CONTENT.value)
+ ),
+ ingest_options=IngestOptions.from_value(data.get("ingest_options")),
+ source_metadata=data.get("source_metadata"),
+ )
+
+ def execution_root_uris(self) -> tuple[str, ...]:
+ directories = {
+ entry.relative_path
+ for entry in self.tree.entries
+ if entry.kind == "directory" and entry.semantic_action is SemanticAction.AGGREGATE
+ }
+ roots = sorted(
+ path
+ for path in directories
+ if not any(
+ parent != path
+ and (not parent or path.startswith(parent + "/"))
+ and parent in directories
+ for parent in directories
+ )
+ )
+ if not roots and any(
+ entry.semantic_action is not SemanticAction.REUSE for entry in self.tree.entries
+ ):
+ roots = [""]
+ return tuple(self.root_uri if not path else f"{self.root_uri}/{path}" for path in roots)
+
+
+class ContentTreeOperation(str, Enum):
+ """Synchronous formal-tree mutation performed before queue handoff."""
+
+ # Write a new file or ensure a planned directory exists.
+ UPSERT = "upsert"
+ # Remove an existing file or subtree.
+ DELETE = "delete"
+ # Delete the old kind before creating the different new kind.
+ REPLACE_KIND = "replace_kind"
+
+
+@dataclass(frozen=True)
+class ContentTreeAction:
+ """One synchronous mutation from a parse artifact into the formal tree.
+
+ File writes carry the final artifact-relative source and MD5. Directory
+ actions only establish topology; they do not invent a directory fingerprint.
+ """
+
+ operation: ContentTreeOperation
+ relative_path: str
+ old_kind: str | None = None
+ new_kind: str | None = None
+ artifact_path: str | None = None
+ md5: str | None = None
+
+ def __post_init__(self) -> None:
+ object.__setattr__(self, "operation", ContentTreeOperation(self.operation))
+ _validate_relative_path(self.relative_path)
+ if self.artifact_path is not None:
+ _validate_relative_path(self.artifact_path)
+ if self.operation is not ContentTreeOperation.DELETE and self.new_kind == "file":
+ if self.artifact_path is None or not self.md5:
+ raise ValueError("file write requires artifact_path and md5")
+ if self.operation is ContentTreeOperation.REPLACE_KIND and (
+ not self.old_kind or not self.new_kind or self.old_kind == self.new_kind
+ ):
+ raise ValueError("replacement requires different old and new kinds")
+
+
+@dataclass(frozen=True)
+class DirectIndexAction:
+ """Direct vector operation independent of semantic model output.
+
+ Deletes remove exact inventory IDs. ``UPDATE_FIELDS`` mutates only approved
+ scalars. Direct upserts are restricted to file L2 vectors in vectors-only
+ processing, where no semantic node produces the embedding request.
+ """
+
+ action: IndexAction
+ uri: str
+ level: int
+ record_id: str
+ upsert_fields: Mapping[str, Any] = field(default_factory=dict)
+ field_patch: FieldPatch | None = None
+ md5: str | None = None
+
+ def __post_init__(self) -> None:
+ object.__setattr__(self, "action", IndexAction(self.action))
+ if self.action is IndexAction.NONE or self.level not in {0, 1, 2}:
+ raise ValueError("invalid direct index action")
+ if not self.record_id:
+ raise ValueError("direct index action requires a record ID")
+ _validate_index_fields(self.upsert_fields)
+ if self.action is IndexAction.DELETE and (self.upsert_fields or self.field_patch):
+ raise ValueError("delete cannot carry index fields")
+ if self.action is IndexAction.UPDATE_FIELDS and (
+ self.field_patch is None or not self.field_patch.values or self.upsert_fields
+ ):
+ raise ValueError("field update requires only a non-empty field patch")
+ if self.action is IndexAction.UPSERT and self.field_patch is not None:
+ raise ValueError("upsert uses resolved upsert_fields, not a field patch")
+ if self.action is IndexAction.MERGE and self.upsert_fields:
+ raise ValueError("merge uses field_patch, not upsert_fields")
+ if self.action in {IndexAction.DELETE, IndexAction.UPDATE_FIELDS} and self.md5 is not None:
+ raise ValueError(f"{self.action.value} cannot carry a vectorization md5")
+
+ @classmethod
+ def from_dict(cls, data: Mapping[str, Any]) -> "DirectIndexAction":
+ values = dict(data)
+ if any(
+ name in values
+ for name in ("fields", "field_modes", "initial_fields", "search_tag_mode")
+ ):
+ legacy_fields = dict(values.pop("fields", {}) or {})
+ legacy_modes = dict(values.pop("field_modes", {}) or {})
+ legacy_seed = dict(values.pop("initial_fields", {}) or {})
+ values.pop("search_tag_mode", None)
+ action = IndexAction(values.get("action"))
+ values.setdefault(
+ "upsert_fields", legacy_fields if action is IndexAction.UPSERT else {}
+ )
+ values.setdefault(
+ "field_patch",
+ (
+ FieldPatch(legacy_fields, legacy_modes, legacy_seed)
+ if action in {IndexAction.MERGE, IndexAction.UPDATE_FIELDS} and legacy_fields
+ else None
+ ),
+ )
+ values.setdefault("upsert_fields", {})
+ if isinstance(values.get("field_patch"), Mapping):
+ values["field_patch"] = FieldPatch.from_dict(values["field_patch"])
+ return cls(**values)
+
+
+@dataclass(frozen=True)
+class ContextUpdatePlan:
+ """Complete execution contract for one resource-tree update.
+
+ Content actions run synchronously under the resource lock. The remaining
+ semantic plan and direct index actions are durable queue work. A no-op plan
+ therefore means no formal-tree mutation, semantic generation, or vector
+ mutation is required.
+ """
+
+ root_uri: str
+ context_type: str
+ content_tree_actions: tuple[ContentTreeAction, ...] = ()
+ semantic_plan: SemanticPlan | None = None
+ direct_index_actions: tuple[DirectIndexAction, ...] = ()
+ file_refresh: FileRefreshIntent | None = None
+
+ def __post_init__(self) -> None:
+ root = self.root_uri.rstrip("/")
+ paths = [action.relative_path for action in self.content_tree_actions]
+ if len(paths) != len(set(paths)):
+ raise ValueError("conflicting content actions")
+ record_ids: set[str] = set()
+ for action in self.direct_index_actions:
+ if action.uri != root and not action.uri.startswith(root + "/"):
+ raise ValueError("index action outside root")
+ if action.record_id in record_ids:
+ raise ValueError("conflicting index actions")
+ record_ids.add(action.record_id)
+ if self.semantic_plan is not None:
+ if (
+ self.semantic_plan.root_uri != root
+ or self.semantic_plan.context_type != self.context_type
+ ):
+ raise ValueError("semantic plan identity mismatch")
+ for entry in self.semantic_plan.tree.entries:
+ for slot in entry.index_slots:
+ if slot.action is IndexAction.NONE:
+ continue
+ if slot.record_id in record_ids:
+ raise ValueError("conflicting index actions")
+ record_ids.add(slot.record_id)
+ if self.file_refresh is not None and self.file_refresh.file_uri != root:
+ raise ValueError("file refresh must target the plan root")
+
+ def is_noop(self) -> bool:
+ return (
+ not self.content_tree_actions
+ and self.semantic_plan is None
+ and not self.direct_index_actions
+ and self.file_refresh is None
+ )
+
+ def after_content_commit(self) -> "ContextUpdatePlan":
+ """Drop synchronous content actions before the async handoff."""
+ if not self.content_tree_actions:
+ return self
+ return ContextUpdatePlan(
+ root_uri=self.root_uri,
+ context_type=self.context_type,
+ semantic_plan=self.semantic_plan,
+ direct_index_actions=self.direct_index_actions,
+ file_refresh=self.file_refresh,
+ )
+
+ def to_dict(self) -> dict[str, Any]:
+ data = asdict(self)
+ if self.semantic_plan is not None:
+ data["semantic_plan"] = self.semantic_plan.to_dict()
+ return data
+
+ @classmethod
+ def from_dict(cls, data: Mapping[str, Any]) -> "ContextUpdatePlan":
+ return cls(
+ root_uri=str(data["root_uri"]),
+ context_type=str(data["context_type"]),
+ content_tree_actions=tuple(
+ ContentTreeAction(**item) for item in data.get("content_tree_actions", ())
+ ),
+ semantic_plan=(
+ SemanticPlan.from_dict(data["semantic_plan"]) if data.get("semantic_plan") else None
+ ),
+ direct_index_actions=tuple(
+ DirectIndexAction.from_dict(item) for item in data.get("direct_index_actions", ())
+ ),
+ file_refresh=(
+ FileRefreshIntent(**data["file_refresh"]) if data.get("file_refresh") else None
+ ),
+ )
+
+
+def _uri(root_uri: str, relative_path: str) -> str:
+ return root_uri if not relative_path else f"{root_uri}/{relative_path}"
+
+
+def _parent(relative_path: str) -> str:
+ value = str(PurePosixPath(relative_path).parent)
+ return "" if value == "." else value
+
+
+def _records_by_path(
+ records: Mapping[str, VectorRecordSnapshot],
+) -> tuple[
+ dict[str, dict[int, VectorRecordSnapshot]],
+ tuple[VectorRecordSnapshot, ...],
+]:
+ result: dict[str, dict[int, VectorRecordSnapshot]] = {}
+ duplicates: list[VectorRecordSnapshot] = []
+ records_by_relative_path: dict[str, list[VectorRecordSnapshot]] = {}
+ for record in records.values():
+ records_by_relative_path.setdefault(record.relative_path, []).append(record)
+ for relative_path, path_records in records_by_relative_path.items():
+ levels, path_duplicates = canonical_vector_records_by_level(path_records)
+ result[relative_path] = levels
+ duplicates.extend(path_duplicates)
+ return result, tuple(duplicates)
+
+
+def _portable_existing_fields(record: VectorRecordSnapshot | None) -> dict[str, Any] | None:
+ if record is None:
+ return None
+ return {
+ key: value
+ for key, value in record.fields.items()
+ if key not in {"id", "uri", "level", "vector", "sparse_vector", "content"}
+ and value is not None
+ }
+
+
+def _field_patch(request: RequestIntent, record: VectorRecordSnapshot | None) -> FieldPatch | None:
+ existing = dict(record.fields) if record is not None else {}
+ values: dict[str, Any] = {}
+ modes: dict[str, str] = {}
+ for intent in request.scalar_intents:
+ if record is not None and record.level not in intent.target_levels:
+ continue
+ if intent.field != "search_tags":
+ continue
+ candidate = FieldPatch({intent.field: intent.value}, {intent.field: intent.mode})
+ desired = candidate.resolve(existing).get(intent.field)
+ if record is None or existing.get(intent.field) != desired:
+ values[intent.field] = intent.value
+ modes[intent.field] = intent.mode
+ return FieldPatch(values, modes) if values else None
+
+
+def _semantic_closure(
+ diff: Any,
+ new_kinds: Mapping[str, str],
+ *,
+ repair_indexes: bool = True,
+) -> tuple[set[str], set[str], set[str]]:
+ """Return active nodes, membership-changing parents, and retained closure.
+
+ Active content/index repair nodes and all their ancestors must aggregate.
+ Direct children of every active directory are retained as reusable inputs, so
+ aggregation sees the same sibling set without traversing unchanged subtrees.
+ """
+ active: set[str] = set()
+ membership_changed: set[str] = set()
+
+ def add_ancestors(path: str) -> None:
+ parent = _parent(path)
+ while True:
+ active.add(parent)
+ if not parent:
+ return
+ parent = _parent(parent)
+
+ for path, entry in diff.entries.items():
+ content_state = ContentState(entry.content_state)
+ index_state = IndexState(entry.index_state)
+ content_requires_semantics = content_state in {
+ ContentState.ADDED,
+ ContentState.MODIFIED,
+ ContentState.REPLACE_KIND,
+ } or (content_state is ContentState.RESTORE and index_state is not IndexState.COMPLETE)
+ if (
+ content_requires_semantics
+ or (
+ repair_indexes
+ and index_state
+ in {IndexState.MISSING, IndexState.PARTIAL, IndexState.LEVEL_CONFLICT}
+ )
+ ) and entry.new_kind in {"file", "directory"}:
+ active.add(path)
+ add_ancestors(path)
+ if content_state in {
+ ContentState.ADDED,
+ ContentState.DELETED,
+ ContentState.RESTORE,
+ ContentState.REPLACE_KIND,
+ }:
+ membership_changed.add(_parent(path))
+ add_ancestors(path)
+
+ retained = set(active)
+ for path in new_kinds:
+ if path and _parent(path) in active:
+ retained.add(path)
+ return active, membership_changed, retained
+
+
+async def hydrate_context_plan_records(
+ *,
+ diff: Any,
+ new_kinds: Mapping[str, str],
+ inventory: Mapping[str, VectorRecordSnapshot],
+ vikingdb: Any,
+ ctx: Any,
+ request: RequestIntent | None = None,
+ repair_indexes: bool = True,
+) -> tuple[Mapping[str, VectorRecordSnapshot], tuple[set[str], set[str], set[str]]]:
+ request = request or RequestIntent("", "semantic_and_vectors")
+ if not request.vectorize:
+ inventory = {}
+ active, membership_changed, retained = _semantic_closure(
+ diff, new_kinds, repair_indexes=repair_indexes
+ )
+ result = dict(inventory)
+ summary_hydrated_ids: set[str] = set()
+
+ def required_summary_ids() -> dict[str, Mapping[str, Any]]:
+ required: dict[str, Mapping[str, Any]] = {}
+ for record_id, record in inventory.items():
+ if record.relative_path not in retained:
+ continue
+ kind = new_kinds.get(record.relative_path)
+ wanted = {2} if kind == "file" else {0, 1} if record.relative_path in active else {0}
+ if (
+ record.level in wanted
+ and record_id not in summary_hydrated_ids
+ and not str(record.fields.get("abstract") or "").strip()
+ ):
+ required[record_id] = {"uri": record.uri, "level": record.level}
+ return required
+
+ def merge_hydrated(payloads: Mapping[str, Mapping[str, Any]]) -> None:
+ for record_id, payload in payloads.items():
+ current = result[record_id]
+ fields = {
+ key: value
+ for key, value in payload.items()
+ if key not in {"id", "uri", "level", "vector", "sparse_vector", "content"}
+ and value is not None
+ }
+ result[record_id] = VectorRecordSnapshot(
+ current.record_id,
+ current.uri,
+ current.relative_path,
+ current.level,
+ {**current.fields, **fields},
+ )
+
+ while True:
+ required = required_summary_ids()
+ hydrated = (
+ await vikingdb.hydrate_incremental_records(
+ required, ctx=ctx, output_fields={"abstract"}
+ )
+ if required
+ else {}
+ )
+ # Inventory and hydration are separate reads. A missing hydration result
+ # has no reusable abstract, so the dependency is promoted below. Keep
+ # the inventory identity: an existing record must never be replaced by a
+ # locally recomputed ID merely because its second read raced or failed.
+ merge_hydrated(hydrated)
+ summary_hydrated_ids.update(required)
+
+ by_path, _ = _records_by_path(result)
+ promoted = {
+ path
+ for path in retained - active
+ if (
+ (record := by_path.get(path, {}).get(2 if new_kinds.get(path) == "file" else 0))
+ is None
+ or not str(record.fields.get("abstract") or "").strip()
+ )
+ }
+ if not promoted:
+ break
+ active.update(promoted)
+ for path in new_kinds:
+ if path and _parent(path) in active:
+ retained.add(path)
+
+ if request.vectorize:
+ required_scalars: dict[str, Mapping[str, Any]] = {}
+ for record_id, record in result.items():
+ kind = new_kinds.get(record.relative_path)
+ if record.relative_path not in active:
+ continue
+ if request.processing_mode == "vectors_only":
+ if kind != "file" or record.level != 2:
+ continue
+ elif record.level not in ({2} if kind == "file" else {0, 1}):
+ continue
+ required_scalars[record_id] = {"uri": record.uri, "level": record.level}
+ if required_scalars:
+ merge_hydrated(await vikingdb.hydrate_incremental_records(required_scalars, ctx=ctx))
+
+ return result, (active, membership_changed, retained)
+
+
+def build_context_update_plan(
+ *,
+ root_uri: str,
+ context_type: str,
+ request: RequestIntent,
+ diff: Any,
+ new_kinds: Mapping[str, str],
+ artifact_paths: Mapping[str, str],
+ records: Mapping[str, VectorRecordSnapshot],
+ is_code_repo: bool,
+ account_id: str,
+ ingest_options: IngestOptions | Mapping[str, Any] | None = None,
+ source_metadata: Mapping[str, str] | None = None,
+ closure: tuple[set[str], set[str], set[str]] | None = None,
+) -> ContextUpdatePlan:
+ """Compile resolved RNFV facts into synchronous and asynchronous actions.
+
+ Content mutations happen before queue handoff. Semantic-dependent index work
+ becomes ``IndexSlot`` data; pure deletes and scalar-only changes become direct
+ embedding-queue actions. Existing vector record IDs are preserved when an
+ existing same-level record is overwritten. Portable existing scalars may be
+ carried for normal updates; derived content, vectors, and ACL fields always
+ come from the current execution.
+ """
+ root_uri = root_uri.rstrip("/")
+ records_by_path, duplicate_records = (
+ _records_by_path(records) if request.vectorize else ({}, ())
+ )
+ active, membership_changed, retained = closure or _semantic_closure(
+ diff,
+ new_kinds,
+ repair_indexes=request.vectorize and request.processing_mode != "vectors_only",
+ )
+ semantic_enabled = request.processing_mode != "vectors_only"
+ if not semantic_enabled:
+ active, membership_changed, retained = set(), set(), set()
+
+ content_actions: list[ContentTreeAction] = []
+ direct_actions: list[DirectIndexAction] = [
+ DirectIndexAction(IndexAction.DELETE, record.uri, record.level, record.record_id)
+ for record in duplicate_records
+ ]
+ for path, entry in sorted(diff.entries.items()):
+ state = ContentState(entry.content_state)
+ kind = entry.new_kind
+ if state in {ContentState.ADDED, ContentState.RESTORE, ContentState.MODIFIED}:
+ content_actions.append(
+ ContentTreeAction(
+ ContentTreeOperation.UPSERT,
+ path,
+ entry.old_kind,
+ kind,
+ artifact_paths.get(path),
+ entry.md5,
+ )
+ )
+ elif state is ContentState.DELETED:
+ content_actions.append(
+ ContentTreeAction(ContentTreeOperation.DELETE, path, old_kind=entry.old_kind)
+ )
+ elif state is ContentState.REPLACE_KIND:
+ content_actions.append(
+ ContentTreeAction(
+ ContentTreeOperation.REPLACE_KIND,
+ path,
+ entry.old_kind,
+ kind,
+ artifact_paths.get(path),
+ entry.md5,
+ )
+ )
+
+ existing = records_by_path.get(path, {})
+ valid_levels = {2} if kind == "file" else {0, 1} if kind == "directory" else set()
+ for level, record in existing.items():
+ if state in {ContentState.DELETED, ContentState.ABSENT} or level not in valid_levels:
+ direct_actions.append(
+ DirectIndexAction(IndexAction.DELETE, record.uri, level, record.record_id)
+ )
+ if (
+ not semantic_enabled
+ and request.vectorize
+ and kind == "file"
+ and (
+ state
+ in {
+ ContentState.ADDED,
+ ContentState.MODIFIED,
+ ContentState.REPLACE_KIND,
+ }
+ or existing.get(2) is None
+ or IndexState(entry.index_state) is IndexState.STALE
+ )
+ ):
+ record = existing.get(2)
+ field_patch = _field_patch(request, record) if request.vectorize else None
+ index_action = (
+ IndexAction.MERGE
+ if (
+ record is None
+ and entry.old_kind == kind
+ and IndexState(entry.index_state) in {IndexState.MISSING, IndexState.PARTIAL}
+ )
+ else IndexAction.UPSERT
+ )
+ direct_actions.append(
+ DirectIndexAction(
+ index_action,
+ _uri(root_uri, path),
+ 2,
+ record.record_id
+ if record is not None
+ else vector_record_id(account_id, _uri(root_uri, path), 2),
+ upsert_fields=(
+ {
+ **(_portable_existing_fields(record) or {}),
+ **(
+ field_patch.resolve(record.fields if record else {})
+ if field_patch
+ else {}
+ ),
+ }
+ if index_action is IndexAction.UPSERT
+ else {}
+ ),
+ field_patch=field_patch if index_action is IndexAction.MERGE else None,
+ md5=entry.md5,
+ )
+ )
+
+ scheduled_direct_ids = {action.record_id for action in direct_actions}
+ for path, levels in records_by_path.items():
+ for record in levels.values():
+ field_patch = _field_patch(request, record) if request.vectorize else None
+ patch_values = dict(field_patch.values) if field_patch is not None else {}
+ diff_entry = diff.entries.get(path)
+ if (
+ record.level == 2
+ and diff_entry is not None
+ and ContentState(diff_entry.content_state) is ContentState.UNCHANGED
+ and diff_entry.md5
+ and not str(record.fields.get("md5") or "")
+ ):
+ patch_values["md5"] = diff_entry.md5
+ field_patch = FieldPatch(
+ patch_values,
+ field_patch.modes if field_patch is not None else {},
+ )
+ if (
+ field_patch is not None
+ and record.record_id not in scheduled_direct_ids
+ and (path not in active or not request.vectorize)
+ ):
+ seed = {
+ "uri": record.uri,
+ "account_id": account_id,
+ "level": record.level,
+ **(_portable_existing_fields(record) or {}),
+ **field_patch.resolve(record.fields),
+ }
+ direct_actions.append(
+ DirectIndexAction(
+ IndexAction.UPDATE_FIELDS,
+ record.uri,
+ record.level,
+ record.record_id,
+ field_patch=field_patch.with_seed(seed),
+ )
+ )
+ scheduled_direct_ids.add(record.record_id)
+
+ semantic_entries: list[SemanticTreeEntry] = []
+ for path in sorted(retained, key=lambda value: (value.count("/"), value)):
+ kind = new_kinds.get(path)
+ if kind not in {"file", "directory"}:
+ continue
+ diff_entry = diff.entries.get(path)
+ state = ContentState(diff_entry.content_state) if diff_entry else ContentState.UNCHANGED
+ action = (
+ SemanticAction.GENERATE
+ if kind == "file" and path in active
+ else SemanticAction.AGGREGATE
+ if kind == "directory" and path in active
+ else SemanticAction.REUSE
+ )
+ slots: list[IndexSlot] = []
+ for level in (2,) if kind == "file" else (0, 1):
+ record = records_by_path.get(path, {}).get(level)
+ if action is SemanticAction.REUSE:
+ if level != (2 if kind == "file" else 0):
+ continue
+ if record is None or not str(record.fields.get("abstract") or "").strip():
+ raise ValueError(f"semantic closure lacks reusable abstract for {path}")
+ slots.append(IndexSlot(level, record.record_id, _portable_existing_fields(record)))
+ continue
+ record_id = (
+ record.record_id
+ if record is not None
+ else vector_record_id(account_id, _uri(root_uri, path), level)
+ )
+ field_patch = _field_patch(request, record) if request.vectorize else None
+ index_action = (
+ IndexAction.MERGE
+ if (
+ record is None
+ and diff_entry
+ and diff_entry.old_kind == kind
+ and IndexState(diff_entry.index_state)
+ in {IndexState.MISSING, IndexState.PARTIAL}
+ )
+ else IndexAction.UPSERT
+ )
+ slots.append(
+ IndexSlot(
+ level,
+ record_id,
+ _portable_existing_fields(record),
+ action=(index_action if request.vectorize else IndexAction.NONE),
+ upsert_fields=(
+ field_patch.resolve(record.fields if record else {})
+ if index_action is IndexAction.UPSERT and field_patch is not None
+ else {}
+ ),
+ field_patch=field_patch,
+ # Keep the original patch mode even for a resolved UPSERT.
+ # If semantic output is unchanged, the executor falls back
+ # to UPDATE_FIELDS and must still interpret append against
+ # the execution-time exact-get record. The normal UPSERT
+ # path deliberately does not forward these modes.
+ fallback_to_patch=field_patch is not None,
+ )
+ )
+ semantic_entries.append(
+ SemanticTreeEntry(
+ path,
+ kind,
+ state,
+ action,
+ md5=diff_entry.md5 if diff_entry else None,
+ index_slots=tuple(slots),
+ membership_changed=path in membership_changed,
+ repair=bool(
+ request.vectorize
+ and diff_entry
+ and IndexState(diff_entry.index_state)
+ in {IndexState.MISSING, IndexState.PARTIAL, IndexState.LEVEL_CONFLICT}
+ ),
+ )
+ )
+
+ semantic_plan = (
+ SemanticPlan(
+ root_uri,
+ context_type,
+ SemanticTreeSnapshot(tuple(semantic_entries)),
+ vectorize=request.vectorize,
+ file_vector_source=(
+ FileVectorSource.SUMMARY_WHEN_AVAILABLE
+ if is_code_repo
+ else FileVectorSource.CONTENT
+ ),
+ ingest_options=(
+ IngestOptions.from_value(ingest_options) if request.vectorize else IngestOptions()
+ ),
+ source_metadata=source_metadata,
+ )
+ if semantic_entries
+ else None
+ )
+ return ContextUpdatePlan(
+ root_uri,
+ context_type,
+ tuple(content_actions),
+ semantic_plan,
+ tuple(direct_actions),
+ )
+
+
+def _with_directory_root(snapshot: RNFVSnapshot, *, root_preexisting: bool) -> RNFVSnapshot:
+ """Add the logical directory root omitted by artifact/tree walks."""
+ new_entries = {"": NewEntry(is_dir=True), **snapshot.new.entries}
+ formal_entries = (
+ {"": FormalEntry(is_dir=True), **snapshot.formal.entries}
+ if root_preexisting
+ else dict(snapshot.formal.entries)
+ )
+ return RNFVSnapshot(
+ request=snapshot.request,
+ new=NewArtifactSnapshot(new_entries, complete=snapshot.new.complete),
+ formal=FormalTreeSnapshot(formal_entries, complete=snapshot.formal.complete),
+ vectors=snapshot.vectors,
+ )
+
+
+async def build_context_update_plan_from_snapshot(
+ *,
+ snapshot: RNFVSnapshot,
+ store: Any,
+ artifact_ref: Any,
+ target: Any,
+ vikingdb: Any,
+ context_type: str,
+ is_code_repo: bool,
+ account_id: str,
+ ctx: Any,
+ root_preexisting: bool,
+ artifact_paths: Mapping[str, str] | None = None,
+ ingest_options: Any = None,
+ source_metadata: Mapping[str, str] | None = None,
+ root_is_file: bool = False,
+) -> tuple[Any, ContextUpdatePlan]:
+ """Resolve RNFV facts, hydrate the minimal closure, and build one plan."""
+ from openviking.storage.resource_diff import resolve_resource_diff
+
+ if not root_is_file:
+ snapshot = _with_directory_root(snapshot, root_preexisting=root_preexisting)
+ paths = dict(artifact_paths or {})
+ diff = await resolve_resource_diff(
+ snapshot,
+ store=store,
+ artifact_ref=artifact_ref,
+ target=target,
+ artifact_paths=paths,
+ )
+ new_kinds = {
+ path: "directory" if entry.is_dir else "file"
+ for path, entry in snapshot.new.entries.items()
+ }
+ records, closure = await hydrate_context_plan_records(
+ diff=diff,
+ new_kinds=new_kinds,
+ inventory=snapshot.vectors.records_by_id,
+ vikingdb=vikingdb,
+ ctx=ctx,
+ request=snapshot.request,
+ repair_indexes=(
+ snapshot.request.vectorize and snapshot.request.processing_mode != "vectors_only"
+ ),
+ )
+ plan = build_context_update_plan(
+ root_uri=snapshot.request.target_uri,
+ context_type=context_type,
+ request=snapshot.request,
+ diff=diff,
+ new_kinds=new_kinds,
+ artifact_paths=paths,
+ records=records,
+ is_code_repo=is_code_repo,
+ account_id=account_id,
+ ingest_options=ingest_options,
+ source_metadata=source_metadata,
+ closure=closure,
+ )
+ if not root_is_file:
+ return diff, plan
+
+ root_entry = diff.entries[""]
+ needs_refresh = ContentState(root_entry.content_state) in {
+ ContentState.ADDED,
+ ContentState.RESTORE,
+ ContentState.MODIFIED,
+ ContentState.REPLACE_KIND,
+ } or (
+ snapshot.request.vectorize
+ and IndexState(root_entry.index_state)
+ in {IndexState.MISSING, IndexState.PARTIAL, IndexState.STALE, IndexState.LEVEL_CONFLICT}
+ )
+ return diff, ContextUpdatePlan(
+ root_uri=plan.root_uri,
+ context_type=plan.context_type,
+ content_tree_actions=plan.content_tree_actions,
+ direct_index_actions=plan.direct_index_actions,
+ file_refresh=(
+ FileRefreshIntent(snapshot.request.target_uri, root_entry.md5)
+ if needs_refresh and snapshot.request.processing_mode != "vectors_only"
+ else None
+ ),
+ )
+
+
+async def execute_content_tree_actions(
+ actions: tuple[ContentTreeAction, ...],
+ *,
+ store: Any,
+ artifact_ref: Any,
+ target: Any,
+ concurrency: int | None = None,
+) -> None:
+ """Commit planned content mutations before any asynchronous work."""
+
+ async def run_action(action: ContentTreeAction, operation: Any) -> None:
+ try:
+ await operation
+ except Exception:
+ logger.exception(
+ "[ContentTreeActionFailed] %s operation=%s relative_path=%s "
+ "old_kind=%s new_kind=%s",
+ log_correlation(),
+ action.operation.value,
+ action.relative_path,
+ action.old_kind or "-",
+ action.new_kind or "-",
+ )
+ raise
+
+ destructive = [
+ action
+ for action in actions
+ if action.operation in {ContentTreeOperation.DELETE, ContentTreeOperation.REPLACE_KIND}
+ ]
+ for action in sorted(
+ destructive, key=lambda item: (-item.relative_path.count("/"), item.relative_path)
+ ):
+ await run_action(
+ action,
+ target.delete_path(action.relative_path, is_dir=action.old_kind == "directory"),
+ )
+ for action in sorted(
+ (action for action in actions if action.new_kind == "directory"),
+ key=lambda item: (item.relative_path.count("/"), item.relative_path),
+ ):
+ await run_action(action, target.mkdir(action.relative_path))
+ file_actions = [action for action in actions if action.new_kind == "file"]
+ if not file_actions:
+ return
+ if concurrency is None:
+ from openviking_cli.utils.config import get_openviking_config
+
+ concurrency = int(
+ get_openviking_config().queue_workers.add_resource.file_operation_concurrency
+ )
+
+ async def write(action: ContentTreeAction) -> None:
+ async def read_and_write() -> None:
+ data = await store.read_bytes(artifact_ref, action.artifact_path)
+ await target.write_file(action.relative_path, data)
+
+ await run_action(action, read_and_write())
+
+ await bounded_map(file_actions, write, concurrency=max(1, concurrency))
+
+
+__all__ = [
+ "ContentState",
+ "ContentTreeAction",
+ "ContentTreeOperation",
+ "ContextUpdatePlan",
+ "FileVectorSource",
+ "FileRefreshIntent",
+ "DirectIndexAction",
+ "IndexAction",
+ "IndexSlot",
+ "IndexState",
+ "ParentPropagation",
+ "SemanticAction",
+ "SemanticPlan",
+ "SemanticTreeEntry",
+ "SemanticTreeSnapshot",
+ "build_context_update_plan",
+ "build_context_update_plan_from_snapshot",
+ "execute_content_tree_actions",
+ "hydrate_context_plan_records",
+]
diff --git a/openviking/storage/index_action.py b/openviking/storage/index_action.py
new file mode 100644
index 000000000..e47d59251
--- /dev/null
+++ b/openviking/storage/index_action.py
@@ -0,0 +1,102 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Shared index-operation protocol used by plans and queue messages."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import Any, Mapping
+
+from openviking.utils.tags import merge_search_tags, normalize_search_tags
+
+PATCHABLE_INDEX_FIELDS = frozenset(
+ {"md5", "content", "abstract", "updated_at", "active_count", "tags", "search_tags"}
+)
+_FIELD_MODES = frozenset({"replace", "append"})
+
+
+class IndexAction(str, Enum):
+ """Action applied to one or more vector-index records."""
+
+ NONE = "none"
+ UPSERT = "upsert"
+ MERGE = "merge"
+ UPDATE_FIELDS = "update_fields"
+ DELETE = "delete"
+
+
+@dataclass(frozen=True)
+class FieldPatch:
+ """One execution-time scalar mutation and its missing-record seed."""
+
+ values: Mapping[str, Any]
+ modes: Mapping[str, str] = field(default_factory=dict)
+ seed_fields: Mapping[str, Any] = field(default_factory=dict)
+
+ def __post_init__(self) -> None:
+ values = dict(self.values)
+ if "search_tags" in values:
+ values["search_tags"] = normalize_search_tags(
+ values["search_tags"], discard_invalid=True
+ )
+ object.__setattr__(self, "values", values)
+ object.__setattr__(
+ self, "modes", {str(key): str(value) for key, value in self.modes.items()}
+ )
+ object.__setattr__(self, "seed_fields", dict(self.seed_fields))
+ unknown = set(self.values) - PATCHABLE_INDEX_FIELDS
+ if unknown:
+ raise ValueError(f"field patch contains forbidden fields: {sorted(unknown)}")
+ unknown_modes = set(self.modes) - set(self.values)
+ if unknown_modes:
+ raise ValueError(f"field modes without values: {sorted(unknown_modes)}")
+ invalid_modes = {
+ name: mode for name, mode in self.modes.items() if mode not in _FIELD_MODES
+ }
+ if invalid_modes:
+ raise ValueError(f"invalid field modes: {invalid_modes}")
+ unsupported_append = {
+ name for name, mode in self.modes.items() if mode == "append" and name != "search_tags"
+ }
+ if unsupported_append:
+ raise ValueError(f"append is unsupported for fields: {sorted(unsupported_append)}")
+
+ def resolve(self, existing: Mapping[str, Any]) -> dict[str, Any]:
+ """Resolve only patched fields against the latest stored values."""
+ resolved: dict[str, Any] = {}
+ for name, value in self.values.items():
+ if name == "search_tags":
+ incoming = normalize_search_tags(value, discard_invalid=True)
+ resolved[name] = (
+ merge_search_tags(existing.get(name), incoming)
+ if self.modes.get(name, "replace") == "append"
+ else incoming
+ )
+ else:
+ resolved[name] = value
+ return resolved
+
+ def apply(self, existing: Mapping[str, Any]) -> dict[str, Any]:
+ return {**dict(existing), **self.resolve(existing)}
+
+ def with_seed(self, seed_fields: Mapping[str, Any]) -> "FieldPatch":
+ return FieldPatch(self.values, self.modes, seed_fields)
+
+ def to_dict(self) -> dict[str, Any]:
+ return {
+ "values": dict(self.values),
+ "modes": dict(self.modes),
+ "seed_fields": dict(self.seed_fields),
+ }
+
+ @classmethod
+ def from_dict(cls, data: Mapping[str, Any]) -> "FieldPatch":
+ return cls(
+ values=dict(data.get("values") or {}),
+ modes=dict(data.get("modes") or {}),
+ seed_fields=dict(data.get("seed_fields") or {}),
+ )
+
+
+__all__ = ["FieldPatch", "IndexAction", "PATCHABLE_INDEX_FIELDS"]
diff --git a/openviking/storage/observers/queue_observer.py b/openviking/storage/observers/queue_observer.py
index 8707a7773..946e6696f 100644
--- a/openviking/storage/observers/queue_observer.py
+++ b/openviking/storage/observers/queue_observer.py
@@ -29,8 +29,8 @@ class QueueObserver(BaseObserver):
async def get_status_table_async(self) -> str:
statuses = await self._queue_manager.check_status()
- dag_stats = self._get_semantic_dag_stats()
- return self._format_status_as_table(statuses, dag_stats)
+ tree_stats = self._get_semantic_tree_stats()
+ return self._format_status_as_table(statuses, tree_stats)
def get_status_table(self) -> str:
return run_async(self.get_status_table_async())
@@ -39,7 +39,7 @@ class QueueObserver(BaseObserver):
return self.get_status_table()
def _format_status_as_table(
- self, statuses: Dict[str, QueueStatus], dag_stats: Optional[object]
+ self, statuses: Dict[str, QueueStatus], tree_stats: Optional[object]
) -> str:
"""
Format queue statuses as a table using tabulate.
@@ -84,12 +84,12 @@ class QueueObserver(BaseObserver):
data.append(
{
"Queue": "Semantic-Nodes",
- "Pending": getattr(dag_stats, "pending_nodes", 0) if dag_stats else 0,
- "In Progress": getattr(dag_stats, "in_progress_nodes", 0) if dag_stats else 0,
- "Processed": getattr(dag_stats, "done_nodes", 0) if dag_stats else 0,
+ "Pending": getattr(tree_stats, "pending_nodes", 0) if tree_stats else 0,
+ "In Progress": getattr(tree_stats, "in_progress_nodes", 0) if tree_stats else 0,
+ "Processed": getattr(tree_stats, "done_nodes", 0) if tree_stats else 0,
"Requeued": 0,
"Errors": 0,
- "Total": getattr(dag_stats, "total_nodes", 0) if dag_stats else 0,
+ "Total": getattr(tree_stats, "total_nodes", 0) if tree_stats else 0,
}
)
@@ -109,13 +109,13 @@ class QueueObserver(BaseObserver):
return tabulate(data, headers="keys", tablefmt="pretty")
- def _get_semantic_dag_stats(self) -> Optional[object]:
+ def _get_semantic_tree_stats(self) -> Optional[object]:
semantic_queue = self._queue_manager._queues.get(self._queue_manager.SEMANTIC)
if not semantic_queue:
return None
handler = getattr(semantic_queue, "_dequeue_handler", None)
- if handler and hasattr(handler, "get_dag_stats"):
- return handler.get_dag_stats()
+ if handler and hasattr(handler, "get_tree_stats"):
+ return handler.get_tree_stats()
return None
def is_healthy(self) -> bool:
@@ -131,6 +131,5 @@ class QueueObserver(BaseObserver):
@staticmethod
def _has_active_errors(statuses: Dict[str, QueueStatus]) -> bool:
return any(
- status.error_count > 0 and not status.is_complete
- for status in statuses.values()
+ status.error_count > 0 and not status.is_complete for status in statuses.values()
)
diff --git a/openviking/storage/ovpack/index.py b/openviking/storage/ovpack/index.py
index b7ed2cf16..499636d77 100644
--- a/openviking/storage/ovpack/index.py
+++ b/openviking/storage/ovpack/index.py
@@ -28,6 +28,7 @@ PORTABLE_VECTOR_SCALAR_FIELDS = [
"description",
"tags",
"abstract",
+ "md5",
]
EXPORT_VECTOR_FIELDS = [*PORTABLE_VECTOR_SCALAR_FIELDS, "vector"]
diff --git a/openviking/storage/queuefs/__init__.py b/openviking/storage/queuefs/__init__.py
index 241a9b3a3..4a34054c1 100644
--- a/openviking/storage/queuefs/__init__.py
+++ b/openviking/storage/queuefs/__init__.py
@@ -6,7 +6,7 @@ from .named_queue import NamedQueue, QueueError, QueueStatus
from .process_result import ProcessOutcome, ProcessResult
from .queue_manager import QueueManager, get_queue_manager, init_queue_manager
from .queue_middleware import QueueMiddleware
-from .semantic_dag import SemanticDagExecutor
+from .semantic_executor import SemanticTreeExecutor
from .semantic_msg import SemanticMsg
from .semantic_processor import SemanticProcessor
from .semantic_queue import SemanticQueue
@@ -25,7 +25,7 @@ __all__ = [
"EmbeddingQueue",
"EmbeddingMsg",
"SemanticQueue",
- "SemanticDagExecutor",
+ "SemanticTreeExecutor",
"SemanticMsg",
"SemanticProcessor",
"SessionCommitMsg",
diff --git a/openviking/storage/queuefs/add_resource_msg.py b/openviking/storage/queuefs/add_resource_msg.py
index 27fa3057e..4d46b462c 100644
--- a/openviking/storage/queuefs/add_resource_msg.py
+++ b/openviking/storage/queuefs/add_resource_msg.py
@@ -27,6 +27,7 @@ class AddResourceMsg:
telemetry_id: Optional[str] = None
prepared: Optional[Dict[str, Any]] = None
staged_source: Optional[Dict[str, Any]] = None
+ shared_source: Optional[Dict[str, Any]] = None
job_phase: AddResourcePhase | str | None = None
lock_handoff: Optional[Dict[str, Any]] = None
actor_peer_id: Optional[str] = None
@@ -77,6 +78,7 @@ class AddResourceMsg:
raise ValueError("source jobs cannot contain prepared post-process data")
if self.prepared is not None and (
self.staged_source is not None
+ or self.shared_source is not None
or self.understanding_response_id is not None
or self.understanding_file_id is not None
):
@@ -85,6 +87,7 @@ class AddResourceMsg:
payload is not None
for payload in (
self.staged_source,
+ self.shared_source,
self.understanding_response_id,
self.understanding_file_id,
)
@@ -112,8 +115,15 @@ class AddResourceMsg:
from openviking.resource.staged_source import StagedSource
staged_source = StagedSource.from_dict(data["staged_source"]).to_dict()
- if prepared is not None and staged_source is not None:
- raise ValueError("prepared and staged_source are mutually exclusive")
+ shared_source = None
+ if data.get("shared_source") is not None:
+ from openviking.resource.shared_source import SharedSource
+
+ shared_source = SharedSource.from_dict(data["shared_source"]).to_dict()
+ if prepared is not None and (staged_source is not None or shared_source is not None):
+ raise ValueError("prepared and source payloads are mutually exclusive")
+ if staged_source is not None and shared_source is not None:
+ raise ValueError("staged_source and shared_source are mutually exclusive")
job_phase = data.get("job_phase") or (
AddResourcePhase.POST_PROCESS.value
if prepared is not None
@@ -127,12 +137,13 @@ class AddResourceMsg:
lock_handoff_retry = 0
if prepared is not None:
args.clear()
- if not task_id or (not path and not prepared and not staged_source) or not root_uri:
+ has_source_payload = bool(prepared or staged_source or shared_source)
+ if not task_id or (not path and not has_source_payload) or not root_uri:
missing = []
if not task_id:
missing.append("task_id")
- if not path and not prepared and not staged_source:
- missing.append("path, prepared, or staged_source")
+ if not path and not has_source_payload:
+ missing.append("path, prepared, staged_source, or shared_source")
if not root_uri:
missing.append("root_uri")
raise ValueError(f"Missing required fields: {missing}")
@@ -184,6 +195,7 @@ class AddResourceMsg:
),
prepared=prepared,
staged_source=staged_source,
+ shared_source=shared_source,
job_phase=job_phase,
watch_interval=float(data.get("watch_interval", 0) or 0),
is_active=(data.get("is_active") if isinstance(data.get("is_active"), bool) else None),
diff --git a/openviking/storage/queuefs/add_resource_processor.py b/openviking/storage/queuefs/add_resource_processor.py
index ba9ce7390..a9abebe8b 100644
--- a/openviking/storage/queuefs/add_resource_processor.py
+++ b/openviking/storage/queuefs/add_resource_processor.py
@@ -24,6 +24,7 @@ from openviking.telemetry import (
)
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
from openviking.telemetry.resource_summary import record_resource_queue_metrics
+from openviking.utils.log_correlation import log_correlation
from openviking_cli.exceptions import OpenVikingError
from openviking_cli.session.user_id import UserIdentifier
from openviking_cli.utils.logger import get_logger
@@ -67,6 +68,15 @@ class AddResourceProcessor(DequeueHandlerBase):
staged = StagedSource.from_dict(msg.staged_source)
await self._viking_fs.delete_temp(staged.temp_uri, ctx=ctx)
+ async def _cleanup_prepared_artifact(self, msg: AddResourceMsg, ctx: RequestContext) -> None:
+ if not msg.prepared or not isinstance(msg.prepared.get("artifact_ref"), dict):
+ return
+ from openviking.parse.output import ParseArtifactRef, store_for_artifact_ref
+
+ artifact_ref = ParseArtifactRef.from_dict(msg.prepared["artifact_ref"])
+ store = store_for_artifact_ref(artifact_ref, viking_fs=self._viking_fs, ctx=ctx)
+ await store.cleanup(artifact_ref)
+
async def _release_cancelled_resources(
self,
msg: AddResourceMsg,
@@ -86,6 +96,8 @@ class AddResourceProcessor(DequeueHandlerBase):
logger.warning("[AddResource] Failed to release cancelled lock handoff: %s", exc)
with suppress(Exception):
await self._cleanup_staged_source(msg, ctx)
+ with suppress(Exception):
+ await self._cleanup_prepared_artifact(msg, ctx)
async def _record_watch_execution(
self,
@@ -154,6 +166,8 @@ class AddResourceProcessor(DequeueHandlerBase):
else:
with suppress(Exception):
await self._cleanup_staged_source(msg, ctx)
+ with suppress(Exception):
+ await self._cleanup_prepared_artifact(msg, ctx)
status = (
"cancelled"
if task.status in (TaskStatus.CANCELLING, TaskStatus.CANCELLED)
@@ -186,6 +200,8 @@ class AddResourceProcessor(DequeueHandlerBase):
unregister_telemetry(telemetry_id)
with suppress(Exception):
await self._cleanup_staged_source(msg, ctx)
+ with suppress(Exception):
+ await self._cleanup_prepared_artifact(msg, ctx)
return ProcessResult.failed(f"Invalid lock_handoff: {exc}")
telemetry = resolve_telemetry(telemetry_id) if telemetry_id else None
@@ -198,8 +214,11 @@ class AddResourceProcessor(DequeueHandlerBase):
register_telemetry(telemetry)
request_wait_tracker = get_request_wait_tracker()
request_wait_tracker.register_request(telemetry_id)
+ current_stage = "queued"
async def _set_stage(stage: str) -> None:
+ nonlocal current_stage
+ current_stage = stage
await tracker.update_stage(
msg.task_id,
stage,
@@ -214,6 +233,17 @@ class AddResourceProcessor(DequeueHandlerBase):
):
terminal = False
try:
+ queue_message_id = str(data.get("id") or "")
+ logger.info(
+ "[AddResourceStarted] %s root=%s phase=%s",
+ log_correlation(
+ task_id=msg.task_id,
+ telemetry_id=telemetry_id,
+ message_id=queue_message_id,
+ ),
+ msg.root_uri,
+ msg.job_phase.value,
+ )
if replay_result is None:
await tracker.start(
msg.task_id,
@@ -235,6 +265,17 @@ class AddResourceProcessor(DequeueHandlerBase):
if result.get("status") == "error":
errors = result.get("errors") or ["resource processing failed"]
error = "; ".join(str(error) for error in errors)
+ logger.error(
+ "[AddResourceFailed] %s root=%s task_stage=%s error=%s",
+ log_correlation(
+ task_id=msg.task_id,
+ telemetry_id=telemetry_id,
+ message_id=queue_message_id,
+ ),
+ msg.root_uri,
+ current_stage,
+ error,
+ )
code = result.get("code")
failure_result = {"code": code} if isinstance(code, str) and code else None
await tracker.fail(
@@ -270,14 +311,30 @@ class AddResourceProcessor(DequeueHandlerBase):
telemetry_id=telemetry_id,
root_uri=result.get("root_uri"),
)
+ telemetry.set("resource.total.duration_ms", telemetry.elapsed_ms())
# Extract token usage summary from telemetry and inject into result
_snapshot = telemetry.finish()
if _snapshot is not None:
+ result["telemetry"] = _snapshot.to_dict(include_summary=True)
_tokens = _snapshot.summary.get("tokens", {})
if _tokens:
result.setdefault("usage", {})
result["usage"]["tokens"] = _tokens
+ resource_summary = _snapshot.summary.get("resource", {})
+ queue_summary = _snapshot.summary.get("queue", {})
+ logger.info(
+ "[AddResourceCompleted] %s root=%s total_ms=%s semantic=%s embedding=%s",
+ log_correlation(
+ task_id=msg.task_id,
+ telemetry_id=telemetry_id,
+ message_id=queue_message_id,
+ ),
+ result.get("root_uri"),
+ (resource_summary.get("total") or {}).get("duration_ms"),
+ queue_summary.get("semantic", {}),
+ queue_summary.get("embedding", {}),
+ )
await self._resource_service._link_resource_reason_memory(
result=result,
@@ -297,10 +354,30 @@ class AddResourceProcessor(DequeueHandlerBase):
terminal = True
return ProcessResult.success()
except asyncio.CancelledError:
+ logger.warning(
+ "[AddResourceCancelled] %s root=%s",
+ log_correlation(
+ task_id=msg.task_id,
+ telemetry_id=telemetry_id,
+ message_id=queue_message_id,
+ ),
+ msg.root_uri,
+ )
await self._record_watch_execution(msg, "cancelled")
terminal = True
raise
except Exception as exc:
+ logger.exception(
+ "[AddResourceFailed] %s root=%s task_stage=%s error=%s",
+ log_correlation(
+ task_id=msg.task_id,
+ telemetry_id=telemetry_id,
+ message_id=queue_message_id,
+ ),
+ msg.root_uri,
+ current_stage,
+ exc,
+ )
await self._record_watch_execution(
msg,
"failed",
diff --git a/openviking/storage/queuefs/embedding_msg.py b/openviking/storage/queuefs/embedding_msg.py
index 4e529318e..af8aba7a7 100644
--- a/openviking/storage/queuefs/embedding_msg.py
+++ b/openviking/storage/queuefs/embedding_msg.py
@@ -1,53 +1,416 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
+"""Durable, action-specific messages for the embedding queue."""
+
+from __future__ import annotations
+
import json
-from dataclasses import asdict, dataclass, field
-from typing import Any, Dict, List, Union
+from dataclasses import dataclass, field
+from typing import Any, Dict, List, Mapping, Optional, Union
from uuid import uuid4
+from openviking.storage.index_action import FieldPatch, IndexAction
+
+EmbeddingInput = Union[str, List[Dict[str, Any]]]
+
+
+class IncompleteInitialRecordError(RuntimeError):
+ """A missing record cannot be recreated from the queued seed."""
+
+ def __init__(self, record_id: str, missing_fields: List[str]):
+ self.record_id = record_id
+ self.missing_fields = list(missing_fields)
+ super().__init__(
+ "update_fields could not recreate missing vector record: "
+ f"record_id={record_id} missing_fields={self.missing_fields}"
+ )
+
+
+def missing_initial_record_fields(record: Mapping[str, Any]) -> List[str]:
+ """Return fields required to create a usable vector record."""
+ missing = [name for name in ("id", "uri", "account_id", "level") if record.get(name) is None]
+ if not record.get("vector") and not record.get("sparse_vector"):
+ missing.append("vector")
+ return missing
+
@dataclass
-class EmbeddingMsg:
- message: Union[str, List[Dict[str, Any]]]
+class EmbedPayload:
+ message: EmbeddingInput
+ context_data: Dict[str, Any]
+ field_patch: FieldPatch | None = None
+
+ def to_dict(self) -> dict[str, Any]:
+ data = {"type": "embed", "message": self.message, "context_data": self.context_data}
+ if self.field_patch is not None:
+ data["field_patch"] = self.field_patch.to_dict()
+ return data
+
+
+@dataclass
+class UpdateFieldsPayload:
+ record_id: str
+ context_data: Dict[str, Any]
+ field_patch: FieldPatch
+
+ def to_dict(self) -> dict[str, Any]:
+ return {
+ "type": "update_fields",
+ "record_id": self.record_id,
+ "context_data": self.context_data,
+ "field_patch": self.field_patch.to_dict(),
+ }
+
+
+@dataclass
+class DeletePayload:
+ record_ids: List[str]
+ context_data: Dict[str, Any]
+
+ def to_dict(self) -> dict[str, Any]:
+ return {
+ "type": "delete",
+ "record_ids": list(self.record_ids),
+ "context_data": self.context_data,
+ }
+
+
+@dataclass
+class NoopPayload:
context_data: Dict[str, Any]
+
+ def to_dict(self) -> dict[str, Any]:
+ return {"type": "noop", "context_data": self.context_data}
+
+
+EmbeddingPayload = EmbedPayload | UpdateFieldsPayload | DeletePayload | NoopPayload
+_PAYLOAD_FIELDS = {
+ "embed": frozenset({"type", "message", "context_data", "field_patch"}),
+ "update_fields": frozenset({"type", "record_id", "context_data", "field_patch"}),
+ "delete": frozenset({"type", "record_ids", "context_data"}),
+ "noop": frozenset({"type", "context_data"}),
+}
+
+
+@dataclass(init=False)
+class EmbeddingMsg:
+ """Stable queue envelope whose payload is determined by ``action``."""
+
+ action: IndexAction
+ payload: EmbeddingPayload
id: str = field(default_factory=lambda: str(uuid4()))
telemetry_id: str = ""
+ queue_enqueued_at: float = 0.0
def __init__(
self,
- message: Union[str, List[Dict[str, Any]]],
- context_data: Dict[str, Any],
+ message: Optional[EmbeddingInput] = None,
+ context_data: Optional[Dict[str, Any]] = None,
telemetry_id: str = "",
- ):
+ action: IndexAction | str = IndexAction.UPSERT,
+ record_ids: Optional[List[str]] = None,
+ update_fields: Optional[Dict[str, Any]] = None,
+ field_modes: Optional[Dict[str, str]] = None,
+ initial_fields: Optional[Dict[str, Any]] = None,
+ queue_enqueued_at: float = 0.0,
+ *,
+ payload: EmbeddingPayload | None = None,
+ ) -> None:
self.id = str(uuid4())
- self.message = message
- self.context_data = context_data
self.telemetry_id = telemetry_id
+ self.queue_enqueued_at = max(float(queue_enqueued_at or 0.0), 0.0)
+ self.action = IndexAction(action)
+ if payload is not None:
+ if any(
+ value is not None
+ for value in (
+ message,
+ context_data,
+ record_ids,
+ update_fields,
+ field_modes,
+ initial_fields,
+ )
+ ):
+ raise ValueError("payload cannot be combined with legacy embedding fields")
+ self.payload = payload
+ else:
+ self.payload = self._legacy_payload(
+ message=message,
+ context_data=dict(context_data or {}),
+ record_ids=list(record_ids or []),
+ update_fields=dict(update_fields or {}),
+ field_modes=dict(field_modes or {}),
+ initial_fields=dict(initial_fields or {}),
+ )
+ self._validate_payload()
+
+ def _legacy_payload(
+ self,
+ *,
+ message: Optional[EmbeddingInput],
+ context_data: Dict[str, Any],
+ record_ids: List[str],
+ update_fields: Dict[str, Any],
+ field_modes: Dict[str, str],
+ initial_fields: Dict[str, Any],
+ ) -> EmbeddingPayload:
+ patch = (
+ FieldPatch(update_fields, field_modes, initial_fields)
+ if update_fields or field_modes or initial_fields
+ else None
+ )
+ if self.action is IndexAction.NONE:
+ if message is not None or record_ids or patch is not None:
+ raise ValueError("none action cannot carry work")
+ return NoopPayload(context_data)
+ if self.action is IndexAction.DELETE:
+ if message is not None or patch is not None:
+ raise ValueError("delete does not accept embedding or field-patch data")
+ return DeletePayload(record_ids, context_data)
+ if self.action is IndexAction.UPDATE_FIELDS:
+ if message is not None or len(record_ids) != 1 or patch is None:
+ raise ValueError("update_fields requires one record id and a field patch")
+ return UpdateFieldsPayload(record_ids[0], context_data, patch)
+ if not isinstance(message, (str, list)):
+ raise ValueError(f"{self.action.value} requires an embedding message")
+ if self.action is IndexAction.UPSERT:
+ if field_modes:
+ raise ValueError("upsert carries resolved fields and cannot use field modes")
+ # Historical flat UPSERT messages allowed these fields but never
+ # consumed them. Preserve queue-read compatibility while emitting
+ # only the strict action-specific payload for new messages.
+ return EmbedPayload(message, context_data)
+ return EmbedPayload(message, context_data, patch)
+
+ def _validate_payload(self) -> None:
+ expected = {
+ IndexAction.NONE: NoopPayload,
+ IndexAction.UPSERT: EmbedPayload,
+ IndexAction.MERGE: EmbedPayload,
+ IndexAction.UPDATE_FIELDS: UpdateFieldsPayload,
+ IndexAction.DELETE: DeletePayload,
+ }[self.action]
+ if not isinstance(self.payload, expected):
+ raise ValueError(
+ f"{self.action.value} requires {expected.__name__}, got {type(self.payload).__name__}"
+ )
+ if isinstance(self.payload, EmbedPayload) and not isinstance(
+ self.payload.message, (str, list)
+ ):
+ raise ValueError(f"{self.action.value} requires an embedding message")
+ if isinstance(self.payload, DeletePayload) and not self.payload.record_ids:
+ raise ValueError("delete requires record_ids")
+ if isinstance(self.payload, UpdateFieldsPayload):
+ if not self.payload.record_id or not self.payload.field_patch.values:
+ raise ValueError("update_fields requires one record id and a non-empty field patch")
+ if (
+ self.action is IndexAction.UPSERT
+ and isinstance(self.payload, EmbedPayload)
+ and self.payload.field_patch is not None
+ ):
+ raise ValueError("upsert does not accept a field patch")
+
+ @classmethod
+ def for_embed(
+ cls,
+ *,
+ message: EmbeddingInput,
+ context_data: Dict[str, Any],
+ action: IndexAction | str = IndexAction.UPSERT,
+ field_patch: FieldPatch | None = None,
+ telemetry_id: str = "",
+ ) -> "EmbeddingMsg":
+ action = IndexAction(action)
+ return cls(
+ action=action,
+ payload=EmbedPayload(message, context_data, field_patch),
+ telemetry_id=telemetry_id,
+ )
+
+ @classmethod
+ def for_update_fields(
+ cls,
+ *,
+ record_id: str,
+ context_data: Dict[str, Any],
+ field_patch: FieldPatch | None = None,
+ fields: Optional[Dict[str, Any]] = None,
+ field_modes: Optional[Dict[str, str]] = None,
+ initial_fields: Optional[Dict[str, Any]] = None,
+ telemetry_id: str = "",
+ ) -> "EmbeddingMsg":
+ if field_patch is not None and any(
+ value is not None for value in (fields, field_modes, initial_fields)
+ ):
+ raise ValueError(
+ "field_patch cannot be combined with legacy fields, field_modes, or initial_fields"
+ )
+ resolved_patch = field_patch or FieldPatch(
+ fields or {}, field_modes or {}, initial_fields or {}
+ )
+ return cls(
+ action=IndexAction.UPDATE_FIELDS,
+ payload=UpdateFieldsPayload(record_id, context_data, resolved_patch),
+ telemetry_id=telemetry_id,
+ )
+
+ @classmethod
+ def for_delete(
+ cls,
+ *,
+ record_ids: List[str],
+ context_data: Dict[str, Any],
+ telemetry_id: str = "",
+ ) -> "EmbeddingMsg":
+ return cls(
+ action=IndexAction.DELETE,
+ payload=DeletePayload(record_ids, context_data),
+ telemetry_id=telemetry_id,
+ )
+
+ @classmethod
+ def noop(cls, *, context_data: Dict[str, Any], telemetry_id: str = "") -> "EmbeddingMsg":
+ return cls(
+ action=IndexAction.NONE,
+ payload=NoopPayload(context_data),
+ telemetry_id=telemetry_id,
+ )
+
+ @property
+ def context_data(self) -> Dict[str, Any]:
+ return self.payload.context_data
+
+ @property
+ def message(self) -> Optional[EmbeddingInput]:
+ return self.payload.message if isinstance(self.payload, EmbedPayload) else None
+
+ @property
+ def record_ids(self) -> List[str]:
+ if isinstance(self.payload, DeletePayload):
+ return self.payload.record_ids
+ if isinstance(self.payload, UpdateFieldsPayload):
+ return [self.payload.record_id]
+ return []
+
+ @property
+ def field_patch(self) -> FieldPatch | None:
+ if isinstance(self.payload, (EmbedPayload, UpdateFieldsPayload)):
+ return self.payload.field_patch
+ return None
+
+ @property
+ def update_fields(self) -> Dict[str, Any]:
+ return dict(self.field_patch.values) if self.field_patch is not None else {}
+
+ @property
+ def field_modes(self) -> Dict[str, str]:
+ return dict(self.field_patch.modes) if self.field_patch is not None else {}
+
+ @property
+ def initial_fields(self) -> Dict[str, Any]:
+ return dict(self.field_patch.seed_fields) if self.field_patch is not None else {}
def to_dict(self) -> Dict[str, Any]:
- """Convert embedding message to dictionary format."""
- return asdict(self)
+ return {
+ "id": self.id,
+ "telemetry_id": self.telemetry_id,
+ "queue_enqueued_at": self.queue_enqueued_at,
+ "action": self.action.value,
+ "payload": self.payload.to_dict(),
+ }
def to_json(self) -> str:
- """Convert embedding message to JSON string."""
return json.dumps(self.to_dict())
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "EmbeddingMsg":
- """Create an embedding message object from dictionary."""
- obj = EmbeddingMsg(
- message=data["message"],
- context_data=data["context_data"],
- telemetry_id=data.get("telemetry_id", ""),
- )
- obj.id = data.get("id", obj.id)
+ if not isinstance(data, dict):
+ raise ValueError("embedding message must be an object")
+ action = IndexAction(data.get("action", IndexAction.UPSERT.value))
+ payload_data = data.get("payload")
+ if isinstance(payload_data, dict):
+ ambiguous = {
+ name
+ for name in (
+ "message",
+ "context_data",
+ "record_ids",
+ "update_fields",
+ "field_modes",
+ "initial_fields",
+ )
+ if name in data
+ }
+ if ambiguous:
+ raise ValueError(
+ f"payload cannot be combined with legacy embedding fields: {sorted(ambiguous)}"
+ )
+ payload_type = payload_data.get("type")
+ allowed_fields = _PAYLOAD_FIELDS.get(payload_type)
+ if allowed_fields is None:
+ raise ValueError(f"unsupported embedding payload type: {payload_type}")
+ unknown_fields = set(payload_data) - allowed_fields
+ if unknown_fields:
+ raise ValueError(
+ f"{payload_type} payload contains unsupported fields: {sorted(unknown_fields)}"
+ )
+ context_data = dict(payload_data.get("context_data") or {})
+ if payload_type == "embed":
+ raw_patch = payload_data.get("field_patch")
+ payload: EmbeddingPayload = EmbedPayload(
+ payload_data.get("message"),
+ context_data,
+ FieldPatch.from_dict(raw_patch) if isinstance(raw_patch, dict) else None,
+ )
+ elif payload_type == "update_fields":
+ payload = UpdateFieldsPayload(
+ str(payload_data.get("record_id") or ""),
+ context_data,
+ FieldPatch.from_dict(dict(payload_data.get("field_patch") or {})),
+ )
+ elif payload_type == "delete":
+ payload = DeletePayload(
+ [str(value) for value in payload_data.get("record_ids") or []],
+ context_data,
+ )
+ elif payload_type == "noop":
+ payload = NoopPayload(context_data)
+ obj = cls(
+ action=action,
+ payload=payload,
+ telemetry_id=str(data.get("telemetry_id") or ""),
+ queue_enqueued_at=data.get("queue_enqueued_at", 0.0),
+ )
+ else:
+ obj = cls(
+ message=data.get("message"),
+ context_data=dict(data.get("context_data") or {}),
+ telemetry_id=str(data.get("telemetry_id") or ""),
+ action=action,
+ record_ids=data.get("record_ids"),
+ update_fields=data.get("update_fields"),
+ field_modes=data.get("field_modes"),
+ initial_fields=data.get("initial_fields"),
+ queue_enqueued_at=data.get("queue_enqueued_at", 0.0),
+ )
+ obj.id = str(data.get("id") or obj.id)
return obj
@classmethod
def from_json(cls, json_str: str) -> "EmbeddingMsg":
- """Safely create object from JSON string."""
try:
- data = json.loads(json_str)
- return cls.from_dict(data)
- except json.JSONDecodeError as e:
- raise ValueError(f"Invalid JSON string: {e}")
+ return cls.from_dict(json.loads(json_str))
+ except json.JSONDecodeError as exc:
+ raise ValueError(f"Invalid JSON string: {exc}") from exc
+
+
+__all__ = [
+ "DeletePayload",
+ "EmbedPayload",
+ "EmbeddingMsg",
+ "IncompleteInitialRecordError",
+ "NoopPayload",
+ "UpdateFieldsPayload",
+ "missing_initial_record_fields",
+]
diff --git a/openviking/storage/queuefs/embedding_msg_converter.py b/openviking/storage/queuefs/embedding_msg_converter.py
index 495085622..08378d7f1 100644
--- a/openviking/storage/queuefs/embedding_msg_converter.py
+++ b/openviking/storage/queuefs/embedding_msg_converter.py
@@ -10,6 +10,7 @@ to EmbeddingMsg objects for asynchronous vector processing.
from openviking.core.context import Context, ContextLevel
from openviking.core.namespace import owner_fields_for_uri
from openviking.storage.acl import ACL_CREATOR_GRANT_FIELD, CreatorAclGrant
+from openviking.storage.index_action import IndexAction
from openviking.storage.queuefs.embedding_msg import EmbeddingMsg
from openviking.telemetry import get_current_telemetry
from openviking_cli.utils import get_logger
@@ -22,10 +23,18 @@ class EmbeddingMsgConverter:
@staticmethod
def from_context(
- context: Context, creator_acl_grant: CreatorAclGrant | None = None
+ context: Context,
+ creator_acl_grant: CreatorAclGrant | None = None,
+ action: IndexAction = IndexAction.MERGE,
) -> EmbeddingMsg | None:
"""
Convert a Context object to EmbeddingMsg.
+
+ Context-based producers normally carry only the fields they just
+ generated. They therefore default to ``MERGE`` so an execution-time
+ exact read retains stored scalar state such as tags and ACLs. Producers
+ with a fully resolved record (notably an RNFV SemanticPlan) must pass
+ ``UPSERT`` explicitly.
"""
vectorization_text = context.get_vectorization_text()
vectorization_images = context.get_vectorization_images()
@@ -82,9 +91,10 @@ class EmbeddingMsgConverter:
if creator_acl_grant is not None:
context_data[ACL_CREATOR_GRANT_FIELD] = creator_acl_grant
- embedding_msg = EmbeddingMsg(
+ embedding_msg = EmbeddingMsg.for_embed(
message=message,
context_data=context_data,
telemetry_id=get_current_telemetry().telemetry_id,
+ action=action,
)
return embedding_msg
diff --git a/openviking/storage/queuefs/embedding_queue.py b/openviking/storage/queuefs/embedding_queue.py
index 57d58bf54..9899b8d97 100644
--- a/openviking/storage/queuefs/embedding_queue.py
+++ b/openviking/storage/queuefs/embedding_queue.py
@@ -1,5 +1,6 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
+import time
from typing import Optional
from openviking_cli.utils.logger import get_logger
@@ -21,6 +22,7 @@ class EmbeddingQueue(NamedQueue):
if msg is None:
logger.warning("Embedding message is None, skipping enqueuing")
return ""
+ msg.queue_enqueued_at = time.time()
logger.debug("Enqueued embedding message: uri=%s", msg.context_data.get("uri", ""))
return await super().enqueue(msg.to_dict())
diff --git a/openviking/storage/queuefs/semantic_dag.py b/openviking/storage/queuefs/semantic_executor.py
similarity index 67%
rename from openviking/storage/queuefs/semantic_dag.py
rename to openviking/storage/queuefs/semantic_executor.py
index a470caa83..9a360c639 100644
--- a/openviking/storage/queuefs/semantic_dag.py
+++ b/openviking/storage/queuefs/semantic_executor.py
@@ -7,7 +7,7 @@ import re
import threading
from contextlib import nullcontext
from dataclasses import dataclass, field
-from typing import Any, ClassVar, Dict, List, Optional, Set
+from typing import TYPE_CHECKING, Any, ClassVar, Dict, List, Optional, Set
from weakref import WeakKeyDictionary
from openviking.core.namespace import classify_uri
@@ -30,9 +30,12 @@ from openviking.storage.abstract_overview import (
write_abstract_overview,
)
from openviking.storage.acl import CreatorAclGrant
+from openviking.storage.context_update_plan import FileVectorSource, SemanticAction
from openviking.storage.errors import LockAcquisitionError
+from openviking.storage.index_action import FieldPatch
from openviking.storage.viking_fs import LS_ALL_NODES, get_viking_fs
from openviking.telemetry import bind_telemetry, get_current_telemetry
+from openviking.utils.content_hash import content_md5
from openviking.utils.ingest_options import IngestOptions
from openviking_cli.utils import VikingURI
from openviking_cli.utils.config import get_openviking_config
@@ -40,6 +43,9 @@ from openviking_cli.utils.logger import get_logger
logger = get_logger(__name__)
+if TYPE_CHECKING:
+ from openviking.storage.context_update_plan import SemanticPlan, SemanticTreeEntry
+
# Session-internal files that should never be summarized by the semantic pipeline.
# These are canonical archives (e.g. session transcripts) whose content provides
# no additional retrieval value and would only waste tokens and add latency.
@@ -70,7 +76,7 @@ class DirNode:
@dataclass
-class DagStats:
+class SemanticTreeStats:
total_nodes: int = 0
pending_nodes: int = 0
in_progress_nodes: int = 0
@@ -80,7 +86,7 @@ class DagStats:
@dataclass(frozen=True)
-class DagWork:
+class SemanticTreeWork:
"""A scheduled unit of DAG work."""
kind: str
@@ -90,27 +96,27 @@ class DagWork:
@dataclass(frozen=True)
-class ScheduledDagWork:
- executor: "SemanticDagExecutor"
- work: DagWork
+class ScheduledTreeWork:
+ executor: "SemanticTreeExecutor"
+ work: SemanticTreeWork
-class SemanticNodeScheduler:
+class SemanticTreeScheduler:
"""Shared node executor for semantic DAG work in one event loop."""
_idle_timeout = 0.05
def __init__(self, max_workers: int):
self._max_workers = max(1, max_workers)
- self._queue: asyncio.Queue[ScheduledDagWork] = asyncio.Queue()
+ self._queue: asyncio.Queue[ScheduledTreeWork] = asyncio.Queue()
self._workers: Set[asyncio.Task] = set()
def configure(self, max_workers: int) -> None:
self._max_workers = max(1, max_workers)
self._ensure_workers()
- def submit(self, executor: "SemanticDagExecutor", work: DagWork) -> None:
- self._queue.put_nowait(ScheduledDagWork(executor=executor, work=work))
+ def submit(self, executor: "SemanticTreeExecutor", work: SemanticTreeWork) -> None:
+ self._queue.put_nowait(ScheduledTreeWork(executor=executor, work=work))
self._ensure_workers()
def _ensure_workers(self) -> None:
@@ -146,27 +152,27 @@ class SemanticNodeScheduler:
self._queue.task_done()
-_node_schedulers: "WeakKeyDictionary[asyncio.AbstractEventLoop, SemanticNodeScheduler]" = (
+_node_schedulers: "WeakKeyDictionary[asyncio.AbstractEventLoop, SemanticTreeScheduler]" = (
WeakKeyDictionary()
)
-def get_semantic_node_scheduler(max_workers: int) -> SemanticNodeScheduler:
+def get_semantic_tree_scheduler(max_workers: int) -> SemanticTreeScheduler:
loop = asyncio.get_running_loop()
scheduler = _node_schedulers.get(loop)
if scheduler is None:
- scheduler = SemanticNodeScheduler(max_workers=max_workers)
+ scheduler = SemanticTreeScheduler(max_workers=max_workers)
_node_schedulers[loop] = scheduler
else:
scheduler.configure(max_workers)
return scheduler
-class SemanticDagExecutor:
+class SemanticTreeExecutor:
"""Execute semantic generation with DAG-style, event-driven lazy dispatch."""
_active_lock: ClassVar[threading.Lock] = threading.Lock()
- _active_executors: ClassVar[Set["SemanticDagExecutor"]] = set()
+ _active_executors: ClassVar[Set["SemanticTreeExecutor"]] = set()
def __init__(
self,
@@ -189,6 +195,10 @@ class SemanticDagExecutor:
generation_trigger: str = "semantic_refresh",
aggregate_directory: bool = True,
copy_source_uri: str = "",
+ file_md5s: Optional[Dict[str, str]] = None,
+ artifact_files: Optional[List[str]] = None,
+ file_abstracts: Optional[Dict[str, str]] = None,
+ semantic_plan: Optional["SemanticPlan"] = None,
):
self._processor = processor
self._context_type = context_type
@@ -198,10 +208,25 @@ class SemanticDagExecutor:
self._target_preexisting = target_preexisting
self._recursive = recursive
self._lock = lock
- self._is_code_repo = is_code_repo
+ self._is_code_repo = bool(
+ is_code_repo
+ or (
+ semantic_plan is not None
+ and semantic_plan.file_vector_source is FileVectorSource.SUMMARY_WHEN_AVAILABLE
+ )
+ )
+ self._changes_provided = changes is not None
self._changes = changes or {}
- self._skip_vectorization = skip_vectorization
- self._ingest_options = IngestOptions.from_value(ingest_options)
+ self._semantic_plan = semantic_plan
+ self._semantic_resource_root = (
+ semantic_plan.root_uri.rstrip("/") if semantic_plan is not None else None
+ )
+ self._skip_vectorization = (
+ not semantic_plan.vectorize if semantic_plan is not None else skip_vectorization
+ )
+ self._ingest_options = IngestOptions.from_value(
+ semantic_plan.ingest_options if semantic_plan is not None else ingest_options
+ )
self._coalesce_key = coalesce_key
self._coalesce_version = coalesce_version
self._source = dict(source) if source else None
@@ -216,6 +241,19 @@ class SemanticDagExecutor:
path for key in ("added", "modified", "deleted") for path in self._changes.get(key, [])
}
self._added_paths = {path.rstrip("/") for path in self._changes.get("added", [])}
+ self._tree_changed_paths = {
+ path.rstrip("/") for key in ("added", "deleted") for path in self._changes.get(key, [])
+ }
+ # Per-file md5 (target-URI keyed) supplied by local incremental apply, so
+ # re-vectorization records the fresh fingerprint instead of leaving it stale.
+ self._file_md5s = dict(file_md5s or {})
+ self._artifact_files = [path.strip("/") for path in (artifact_files or []) if path]
+ self._file_abstracts = dict(file_abstracts or {})
+ self._plan_entries_by_uri: Dict[str, "SemanticTreeEntry"] = {}
+ self._plan_children: Dict[str, tuple[List[str], List[str]]] = {}
+ self._plan_active_dirs: Set[str] = set()
+ if semantic_plan is not None:
+ self._initialize_semantic_plan(semantic_plan)
self._node_concurrency = max(1, max_concurrent_llm)
self._llm_sem = asyncio.Semaphore(max_concurrent_llm)
self._viking_fs = get_viking_fs()
@@ -223,24 +261,103 @@ class SemanticDagExecutor:
self._parent: Dict[str, Optional[str]] = {}
self._root_uri: Optional[str] = None
self._root_done: Optional[asyncio.Event] = None
- self._scheduler: Optional[SemanticNodeScheduler] = None
+ self._scheduler: Optional[SemanticTreeScheduler] = None
self._active_scheduled_work = 0
self._skill_node_tasks: Set[asyncio.Task] = set()
self._active_work_idle = asyncio.Event()
self._active_work_idle.set()
self._closed = False
self._failure: Optional[Exception] = None
- self._stats = DagStats()
+ self._stats = SemanticTreeStats()
self._file_change_status: Dict[str, bool] = {}
self._dir_change_status: Dict[str, bool] = {}
self._overview_cache: Dict[str, Dict[str, str]] = {}
self._overview_cache_lock = asyncio.Lock()
self._root_write_result = AbstractOverviewWriteResult(wrote=False)
+ self._scheduled_vector_record_ids: set[str] = set()
+
+ @staticmethod
+ def _entry_record_abstract(entry: "SemanticTreeEntry", level: int) -> str:
+ slot = entry.slot(level)
+ return slot.abstract if slot is not None else ""
+
+ def _initialize_semantic_plan(self, plan: "SemanticPlan") -> None:
+ from openviking.storage.context_update_plan import SemanticAction
+
+ root = plan.root_uri.rstrip("/")
+ root_entry = next(
+ (entry for entry in plan.tree.entries if entry.relative_path == ""),
+ None,
+ )
+ self._target_preexisting = not (
+ root_entry is not None and root_entry.content_state.value in {"added", "restore"}
+ )
+ current_entries = list(plan.tree.entries)
+ children: Dict[str, tuple[List[str], List[str]]] = {}
+ for entry in current_entries:
+ uri = root if not entry.relative_path else f"{root}/{entry.relative_path}"
+ self._plan_entries_by_uri[uri] = entry
+ if not entry.relative_path:
+ continue
+ parent_rel = entry.relative_path.rsplit("/", 1)[0] if "/" in entry.relative_path else ""
+ parent_uri = root if not parent_rel else f"{root}/{parent_rel}"
+ dirs, files = children.setdefault(parent_uri, ([], []))
+ (dirs if entry.kind == "directory" else files).append(uri)
+
+ changed_uris: List[str] = []
+ for entry in plan.tree.entries:
+ if entry.semantic_action == SemanticAction.REUSE:
+ continue
+ uri = root if not entry.relative_path else f"{root}/{entry.relative_path}"
+ changed_uris.append(uri)
+ if entry.kind == "directory" and entry.semantic_action == SemanticAction.AGGREGATE:
+ self._plan_active_dirs.add(uri)
+ if entry.relative_path:
+ self._plan_active_dirs.add(uri.rsplit("/", 1)[0])
+ for active_uri in list(self._plan_active_dirs):
+ current = active_uri
+ while current != root and current.startswith(root + "/"):
+ parent = current.rsplit("/", 1)[0]
+ if parent not in self._plan_entries_by_uri:
+ break
+ self._plan_active_dirs.add(parent)
+ current = parent
+
+ self._plan_children = {
+ uri: (sorted(dirs), sorted(files)) for uri, (dirs, files) in children.items()
+ }
+ self._incremental_update = True
+ self._target_uri = root
+ self._recursive = True
+ self._changes_provided = True
+ self._changed_paths = set(changed_uris)
+ self._added_paths = {
+ uri
+ for uri, entry in self._plan_entries_by_uri.items()
+ if entry.content_state.value in {"added", "restore", "replace_kind"}
+ }
+ self._tree_changed_paths = {
+ root if not entry.relative_path else f"{root}/{entry.relative_path}"
+ for entry in plan.tree.entries
+ if entry.membership_changed
+ }
+ self._file_md5s = {
+ uri: str(entry.md5)
+ for uri, entry in self._plan_entries_by_uri.items()
+ if entry.kind == "file" and entry.md5
+ }
+ self._file_abstracts = {
+ uri: abstract
+ for uri, entry in self._plan_entries_by_uri.items()
+ if entry.kind == "file" and (abstract := self._entry_record_abstract(entry, 2))
+ }
def _creator_acl_grant(self, uri: str) -> CreatorAclGrant | None:
normalized = uri.rstrip("/")
- if self._generation_trigger == "resource_ingest" and self._target_preexisting is False:
- root = self._root_uri.rstrip("/")
+ if (
+ self._generation_trigger == "resource_ingest" or self._semantic_plan is not None
+ ) and self._target_preexisting is False:
+ root = (self._semantic_resource_root or self._root_uri).rstrip("/")
if normalized == root:
return CreatorAclGrant.DIRECT
if normalized.startswith(f"{root}/"):
@@ -255,7 +372,21 @@ class SemanticDagExecutor:
"""Run DAG execution starting from root_uri."""
self._root_uri = root_uri
self._root_done = asyncio.Event()
- self._scheduler = get_semantic_node_scheduler(self._node_concurrency)
+ self._scheduler = get_semantic_tree_scheduler(self._node_concurrency)
+ if self._semantic_plan is not None:
+ from openviking.storage.context_update_plan import (
+ SemanticAction,
+ SemanticTreeEntry,
+ )
+
+ entry = self._plan_entries_by_uri.get(root_uri.rstrip("/"))
+ if (
+ isinstance(entry, SemanticTreeEntry)
+ and entry.semantic_action == SemanticAction.REUSE
+ ):
+ self._stats.total_nodes = 1
+ self._stats.done_nodes = 1
+ return
owner = processing_owner.get()
if owner is not None and self._task_context is not None:
@@ -288,11 +419,11 @@ class SemanticDagExecutor:
finally:
self._unregister_active()
- def _schedule_work(self, work: DagWork) -> None:
+ def _schedule_work(self, work: SemanticTreeWork) -> None:
if self._closed:
return
if self._scheduler is None:
- self._scheduler = get_semantic_node_scheduler(self._node_concurrency)
+ self._scheduler = get_semantic_tree_scheduler(self._node_concurrency)
self._scheduler.submit(self, work)
def _start_scheduled_work(self) -> None:
@@ -309,14 +440,14 @@ class SemanticDagExecutor:
return
self._stats.total_nodes += 1
self._stats.pending_nodes += 1
- self._schedule_work(DagWork(kind="dir", dir_uri=dir_uri, parent_uri=parent_uri))
+ self._schedule_work(SemanticTreeWork(kind="dir", dir_uri=dir_uri, parent_uri=parent_uri))
def _schedule_file(self, parent_uri: str, file_path: str) -> None:
if self._closed:
return
self._stats.total_nodes += 1
self._stats.pending_nodes += 1
- self._schedule_work(DagWork(kind="file", dir_uri=parent_uri, file_path=file_path))
+ self._schedule_work(SemanticTreeWork(kind="file", dir_uri=parent_uri, file_path=file_path))
def _mark_node_started(self) -> None:
self._stats.pending_nodes = max(0, self._stats.pending_nodes - 1)
@@ -354,8 +485,8 @@ class SemanticDagExecutor:
self._active_executors.discard(self)
@classmethod
- def get_active_stats(cls) -> DagStats:
- stats = DagStats()
+ def get_active_stats(cls) -> SemanticTreeStats:
+ stats = SemanticTreeStats()
with cls._active_lock:
executors = list(cls._active_executors)
for executor in executors:
@@ -366,7 +497,7 @@ class SemanticDagExecutor:
stats.done_nodes += current.done_nodes
return stats
- async def _run_work(self, work: DagWork) -> None:
+ async def _run_work(self, work: SemanticTreeWork) -> None:
if self._context_type == "skill":
# Shared scheduler workers must not be cancelled with one package.
# Each Skill node has its own cancellable task and ownership context.
@@ -382,7 +513,7 @@ class SemanticDagExecutor:
else:
await self._run_work_with_context(work)
- async def _run_work_with_context(self, work: DagWork) -> None:
+ async def _run_work_with_context(self, work: SemanticTreeWork) -> None:
task_context = (
bind_task_context(
self._task_context.task_id,
@@ -411,7 +542,7 @@ class SemanticDagExecutor:
return await run_to_completion(lambda: operation)
return await operation
- async def _run_work_bound(self, work: DagWork) -> None:
+ async def _run_work_bound(self, work: SemanticTreeWork) -> None:
self._mark_node_started()
if work.kind == "dir":
@@ -484,6 +615,60 @@ class SemanticDagExecutor:
logger.info("Skipping busy parent semantic refresh: %s", dir_uri)
file_index = {path: idx for idx, path in enumerate(file_paths)}
child_index = {path: idx for idx, path in enumerate(children_dirs)}
+ if self._semantic_plan is not None:
+ required_file_paths = {
+ path
+ for path in file_paths
+ if self._plan_entries_by_uri[path].semantic_action is not SemanticAction.REUSE
+ }
+ required_children_dirs = {
+ path for path in children_dirs if path in self._plan_active_dirs
+ }
+ file_summaries: List[Optional[Dict[str, str]]] = [None] * len(file_paths)
+ children_abstracts: List[Optional[Dict[str, str]]] = [None] * len(children_dirs)
+ for path in file_paths:
+ if path in required_file_paths or path not in sampled_file_paths:
+ continue
+ abstract = self._entry_record_abstract(self._plan_entries_by_uri[path], 2)
+ if not abstract:
+ raise RuntimeError(f"Semantic plan lacks L2 abstract for {path}")
+ file_summaries[file_index[path]] = {
+ "name": path.rsplit("/", 1)[-1],
+ "summary": abstract,
+ }
+ for path in children_dirs:
+ if path in required_children_dirs or path not in sampled_children_dirs:
+ continue
+ abstract = self._entry_record_abstract(self._plan_entries_by_uri[path], 0)
+ if not abstract:
+ raise RuntimeError(f"Semantic plan lacks L0 abstract for {path}")
+ children_abstracts[child_index[path]] = {
+ "name": path.rsplit("/", 1)[-1],
+ "abstract": abstract,
+ }
+ pending = len(required_file_paths) + len(required_children_dirs)
+ node = DirNode(
+ uri=dir_uri,
+ children_dirs=children_dirs,
+ file_paths=file_paths,
+ file_index=file_index,
+ child_index=child_index,
+ file_summaries=file_summaries,
+ children_abstracts=children_abstracts,
+ pending=pending,
+ pending_snapshot=pending_snapshot,
+ sampled_children_dirs=sampled_children_dirs,
+ sampled_file_paths=sampled_file_paths,
+ dispatched=True,
+ )
+ self._nodes[dir_uri] = node
+ for file_path in sorted(required_file_paths):
+ self._schedule_file(dir_uri, file_path)
+ for child_uri in sorted(required_children_dirs):
+ self._schedule_dir(child_uri, dir_uri)
+ if pending == 0:
+ self._schedule_overview(dir_uri)
+ return False
# Recursive/initial work still maintains every file. Incremental
# parent aggregation prepares only sampled inputs plus files that
# changed and therefore need their own vector maintenance.
@@ -538,7 +723,7 @@ class SemanticDagExecutor:
except Exception as e:
logger.error(f"Failed to dispatch directory {dir_uri}: {e}", exc_info=True)
self._record_skill_failure(dir_uri, e)
- if self._generation_trigger == "content_copy":
+ if self._semantic_plan is not None or self._generation_trigger == "content_copy":
raise
if parent_uri:
await self._on_child_done(parent_uri, dir_uri, "")
@@ -636,11 +821,15 @@ class SemanticDagExecutor:
async def _list_dir(self, uri: str, from_hint: str) -> tuple[list[str], list[str]]:
"""List directory entries and return (child_dirs, file_paths)."""
+ if self._semantic_plan is not None:
+ return self._plan_children.get(uri.rstrip("/"), ([], []))
+ if self._artifact_files and self._root_uri:
+ return self._list_artifact_dir(uri)
try:
entries = await self._viking_fs.ls(uri, node_limit=LS_ALL_NODES, ctx=self._ctx)
except Exception as e:
logger.warning(
- f"[SemanticDagExecutor] Failed to list directory {uri}: {e} from {from_hint}"
+ f"[SemanticTreeExecutor] Failed to list directory {uri}: {e} from {from_hint}"
)
raise
@@ -660,6 +849,34 @@ class SemanticDagExecutor:
return sorted(children_dirs), sorted(file_paths)
+ def _list_artifact_dir(self, uri: str) -> tuple[list[str], list[str]]:
+ root = self._root_uri.rstrip("/")
+ current = uri.rstrip("/")
+ if current == root:
+ prefix = ""
+ elif current.startswith(f"{root}/"):
+ prefix = current[len(root) + 1 :] + "/"
+ else:
+ return [], []
+
+ child_dirs: set[str] = set()
+ file_paths: list[str] = []
+ for rel_path in self._artifact_files:
+ if not rel_path.startswith(prefix):
+ continue
+ remainder = rel_path[len(prefix) :]
+ if not remainder:
+ continue
+ head, separator, _ = remainder.partition("/")
+ if not head or head.startswith(".") or head in _SKIP_FILENAMES:
+ continue
+ child_uri = VikingURI(current).join(head).uri
+ if separator:
+ child_dirs.add(child_uri)
+ else:
+ file_paths.append(child_uri)
+ return sorted(child_dirs), sorted(file_paths)
+
def _get_target_file_path(self, current_uri: str) -> Optional[str]:
if not self._incremental_update or not self._target_uri or not self._root_uri:
logger.warning(
@@ -676,17 +893,19 @@ class SemanticDagExecutor:
return current_uri
def _is_direct_incremental_update(self) -> bool:
- return (
+ return self._semantic_plan is not None or (
self._incremental_update
- and bool(self._changed_paths)
+ and self._changes_provided
and self._target_uri == self._root_uri
)
- def _path_has_direct_change(self, uri: str) -> bool:
- if uri in self._changed_paths:
- return True
- prefix = uri.rstrip("/") + "/"
- return any(path.startswith(prefix) for path in self._changed_paths)
+ def _directory_entries_changed(self, dir_uri: str) -> bool:
+ """Return whether this directory subtree gained or lost an entry."""
+ normalized = dir_uri.rstrip("/")
+ prefix = normalized + "/"
+ return any(
+ path == normalized or path.startswith(prefix) for path in self._tree_changed_paths
+ )
def _ingest_options_for_file(self, file_path: str) -> IngestOptions:
if self._generation_trigger == "content_write" and file_path not in self._changed_paths:
@@ -698,6 +917,18 @@ class SemanticDagExecutor:
return IngestOptions()
return self._ingest_options
+ def _plan_scalar_override(self, uri: str, level: int) -> Optional[Dict[str, Any]]:
+ entry = self._plan_entries_by_uri.get(uri.rstrip("/"))
+ if entry is None:
+ return None
+ slot = entry.slot(level)
+ return slot.scalar_override() if slot is not None else None
+
+ def _plan_field_patch(self, uri: str, level: int) -> FieldPatch | None:
+ entry = self._plan_entries_by_uri.get(uri.rstrip("/"))
+ slot = entry.slot(level) if entry is not None else None
+ return slot.field_patch if slot is not None else None
+
async def _check_file_content_changed(self, file_path: str) -> bool:
if self._is_direct_incremental_update():
return file_path in self._changed_paths
@@ -729,6 +960,12 @@ class SemanticDagExecutor:
target_path = self._get_target_file_path(file_path)
if not target_path:
return None
+ vector_abstract = self._file_abstracts.get(target_path.rstrip("/"))
+ if vector_abstract:
+ return {
+ "name": file_path.rsplit("/", 1)[-1],
+ "summary": vector_abstract,
+ }
try:
parent_uri = "/".join(target_path.rsplit("/", 1)[:-1])
@@ -777,13 +1014,28 @@ class SemanticDagExecutor:
self, dir_uri: str, current_files: List[str], current_dirs: List[str]
) -> bool:
if self._is_direct_incremental_update():
- if self._path_has_direct_change(dir_uri):
+ node = self._nodes.get(dir_uri)
+ if node is not None and node.pending_snapshot > 0:
+ return True
+ if self._directory_entries_changed(dir_uri):
return True
for current_file in current_files:
- if self._file_change_status.get(current_file, True):
+ default_changed = (
+ self._plan_entries_by_uri[current_file].semantic_action
+ is not SemanticAction.REUSE
+ if self._semantic_plan is not None
+ else True
+ )
+ if self._file_change_status.get(current_file, default_changed):
return True
for current_dir in current_dirs:
- if self._dir_change_status.get(current_dir, True):
+ default_changed = (
+ self._plan_entries_by_uri[current_dir].semantic_action
+ is not SemanticAction.REUSE
+ if self._semantic_plan is not None
+ else True
+ )
+ if self._dir_change_status.get(current_dir, default_changed):
return True
return False
@@ -815,6 +1067,13 @@ class SemanticDagExecutor:
async def _read_existing_overview_abstract(
self, dir_uri: str
) -> tuple[Optional[str], Optional[str]]:
+ if self._semantic_plan is not None:
+ entry = self._plan_entries_by_uri.get(dir_uri.rstrip("/"))
+ if entry is None:
+ return None, None
+ overview = self._entry_record_abstract(entry, 1) or None
+ abstract = self._entry_record_abstract(entry, 0) or None
+ return overview, abstract
target_path = self._get_target_file_path(dir_uri)
if not target_path:
return None, None
@@ -830,6 +1089,8 @@ class SemanticDagExecutor:
file_name = file_path.split("/")[-1]
need_vectorize = True
+ semantic_succeeded = True
+ file_content = None
try:
summary_dict = None
if self._incremental_update:
@@ -860,36 +1121,80 @@ class SemanticDagExecutor:
else:
self._file_change_status[file_path] = True
if summary_dict is None:
+ file_content = None
+ if hasattr(self._viking_fs, "read_file_bytes"):
+ file_content = await self._viking_fs.read_file_bytes(file_path, ctx=self._ctx)
+ summary_kwargs: Dict[str, Any] = {
+ "llm_sem": self._llm_sem,
+ "ctx": self._ctx,
+ }
+ if file_content is not None:
+ summary_kwargs["file_content"] = file_content
summary_dict = await self._processor._generate_single_file_summary(
- file_path, llm_sem=self._llm_sem, ctx=self._ctx
+ file_path, **summary_kwargs
)
+ except (AbstractOverviewFormatError, FileNotFoundError, ValueError):
+ # A generated sidecar that opted into OKF must never be treated as
+ # an empty file summary; doing so would silently feed metadata or
+ # corrupted YAML into a later regeneration.
+ raise
except Exception as e:
logger.warning(f"Failed to generate summary for {file_path}: {e}")
self._record_skill_failure(file_path, e)
summary_dict = {"name": file_name, "summary": ""}
+ semantic_succeeded = False
finally:
self._stats.done_nodes += 1
self._stats.in_progress_nodes = max(0, self._stats.in_progress_nodes - 1)
if self._closed:
return
+ from openviking.storage.context_update_plan import (
+ IndexAction,
+ SemanticTreeEntry,
+ )
+
+ entry = self._plan_entries_by_uri.get(file_path.rstrip("/"))
+ slot = entry.slot(2) if isinstance(entry, SemanticTreeEntry) else None
+ if isinstance(entry, SemanticTreeEntry):
+ need_vectorize = bool(
+ semantic_succeeded
+ and slot is not None
+ and slot.action in {IndexAction.UPSERT, IndexAction.MERGE}
+ )
if need_vectorize and not self._skip_vectorization:
use_summary = self._is_code_repo and bool(summary_dict.get("summary"))
try:
- enqueued = await self._await_write(
- self._processor._vectorize_single_file(
- parent_uri=parent_uri,
- context_type=self._context_type,
- file_path=file_path,
- summary_dict=summary_dict,
- ctx=self._ctx,
- use_summary=use_summary,
- ingest_options=self._ingest_options_for_file(file_path),
- creator_acl_grant=self._creator_acl_grant(file_path),
- )
+ vectorize_kwargs: Dict[str, Any] = {}
+ if file_content is not None:
+ vectorize_kwargs["file_content"] = file_content
+ manifest_md5 = self._file_md5s.get(file_path.rstrip("/")) or None
+ if file_content is not None and not manifest_md5:
+ file_md5 = content_md5(file_content)
+ else:
+ file_md5 = manifest_md5
+ if self._semantic_plan is not None:
+ vectorize_kwargs["scalar_override"] = self._plan_scalar_override(file_path, 2)
+ vectorize_kwargs["field_patch"] = self._plan_field_patch(file_path, 2)
+ vectorize_kwargs["action"] = slot.action.value if slot is not None else "upsert"
+ enqueued = await self._processor._vectorize_single_file(
+ parent_uri=parent_uri,
+ context_type=self._context_type,
+ file_path=file_path,
+ summary_dict=summary_dict,
+ ctx=self._ctx,
+ use_summary=use_summary,
+ ingest_options=(
+ IngestOptions()
+ if self._semantic_plan is not None
+ else self._ingest_options_for_file(file_path)
+ ),
+ creator_acl_grant=self._creator_acl_grant(file_path),
+ file_md5=file_md5,
+ **vectorize_kwargs,
)
- if enqueued:
- self._stats.indexed_records += 1
+ if enqueued and slot is not None:
+ self._scheduled_vector_record_ids.add(slot.record_id)
except Exception as e:
logger.error(
"Failed to schedule vectorization for %s: %s",
@@ -947,7 +1252,7 @@ class SemanticDagExecutor:
if not node or node.overview_scheduled:
return
node.overview_scheduled = True
- self._schedule_work(DagWork(kind="overview", dir_uri=dir_uri))
+ self._schedule_work(SemanticTreeWork(kind="overview", dir_uri=dir_uri))
def _finalize_file_summaries(self, node: DirNode) -> List[Dict[str, str]]:
summaries: List[Dict[str, str]] = []
@@ -1044,7 +1349,8 @@ class SemanticDagExecutor:
missing_summary_entries=missing_summary_entries,
),
}
- if dir_uri == self._root_uri and self._source:
+ source_root = self._semantic_resource_root or self._root_uri
+ if dir_uri == source_root and self._source:
metadata["source"] = self._source
wrote = await self._await_write(
write_abstract_overview(
@@ -1057,7 +1363,7 @@ class SemanticDagExecutor:
metadata=metadata,
consume_pending=consume_pending,
lock=self._lock,
- log_prefix="[SemanticDag]",
+ log_prefix="[SemanticTree]",
)
)
if not wrote.wrote:
@@ -1085,6 +1391,7 @@ class SemanticDagExecutor:
need_vectorize = True
children_changed = True
should_write = True
+ write_result = AbstractOverviewWriteResult(wrote=False)
abstract: Optional[str] = None
overview: Optional[str] = None
total_entries = (
@@ -1158,7 +1465,7 @@ class SemanticDagExecutor:
if should_write:
assert overview is not None and abstract is not None
try:
- wrote = await self._write_directory_semantics(
+ write_result = await self._write_directory_semantics(
dir_uri,
overview,
abstract,
@@ -1168,15 +1475,17 @@ class SemanticDagExecutor:
missing_summary_entries=node.missing_summary_entries,
)
if dir_uri == self._root_uri:
- self._root_write_result = wrote
- if not wrote.wrote:
+ self._root_write_result = write_result
+ if not write_result.wrote:
need_vectorize = False
except AbstractOverviewFormatError:
raise
except Exception as exc:
- need_vectorize = False
- logger.info(f"[SemanticDag] {dir_uri} write failed, skipping")
self._record_skill_failure(dir_uri, exc)
+ if self._semantic_plan is not None:
+ self.fail(exc)
+ return
+ raise
except AbstractOverviewFormatError:
raise
@@ -1184,17 +1493,66 @@ class SemanticDagExecutor:
logger.error(f"Failed to generate overview for {dir_uri}: {e}", exc_info=True)
self._record_skill_failure(dir_uri, e)
else:
+ slots = {}
+ if self._semantic_plan is not None:
+ entry = self._plan_entries_by_uri.get(dir_uri.rstrip("/"))
+ slots = {slot.level: slot for slot in entry.index_slots} if entry else {}
+
if need_vectorize and not self._skip_vectorization:
assert overview is not None and abstract is not None
try:
- await self._await_write(
- self._processor._vectorize_directory(
+ directory_vector_kwargs: Dict[str, Any] = {}
+ include_abstract = True
+ include_overview = True
+ if self._semantic_plan is not None:
+ from openviking.storage.context_update_plan import (
+ IndexAction,
+ )
+
+ def should_emit(level: int) -> bool:
+ slot = slots.get(level)
+ if slot is None or slot.action not in {
+ IndexAction.UPSERT,
+ IndexAction.MERGE,
+ }:
+ return False
+ old_body = slot.abstract
+ new_body = abstract if level == 0 else overview
+ return not old_body or old_body != new_body
+
+ include_abstract = should_emit(0)
+ include_overview = should_emit(1)
+ directory_vector_kwargs = {
+ "scalar_overrides": {
+ level: values
+ for level in (0, 1)
+ if (values := self._plan_scalar_override(dir_uri, level))
+ },
+ "actions": {
+ level: slot.action
+ for level, slot in slots.items()
+ if slot.action in {IndexAction.UPSERT, IndexAction.MERGE}
+ },
+ "field_patches": {
+ level: slot.field_patch
+ for level, slot in slots.items()
+ if slot.field_patch is not None
+ },
+ "include_abstract": include_abstract,
+ "include_overview": include_overview,
+ }
+ if include_abstract or include_overview:
+ enqueued_levels = await self._processor._vectorize_directory(
dir_uri,
context_type=self._context_type,
abstract=abstract,
overview=overview,
ctx=self._ctx,
- ingest_options=self._ingest_options_for_directory(),
+ ingest_options=(
+ IngestOptions()
+ if self._semantic_plan is not None
+ else self._ingest_options_for_directory()
+ ),
creator_acl_grant=self._creator_acl_grant(dir_uri),
**(
{"skill_source_path": (self._source or {}).get("path", "")}
@@ -1202,9 +1560,10 @@ class SemanticDagExecutor:
and classify_uri(dir_uri).is_skill_root
else {}
),
+ **directory_vector_kwargs,
)
- )
- self._stats.indexed_records += int(bool(abstract)) + int(bool(overview))
+ else:
+ enqueued_levels = set()
except Exception as e:
logger.error(
"Failed to schedule vectorization for %s: %s",
@@ -1215,6 +1574,37 @@ class SemanticDagExecutor:
if self._context_type != "skill":
raise
self._record_skill_failure(dir_uri, e)
+ else:
+ enqueued_levels = set()
+
+ for level, slot in sorted(slots.items()):
+ if (
+ level in enqueued_levels
+ or not slot.fallback_to_patch
+ or slot.field_patch is None
+ ):
+ continue
+ enqueued = await self._processor._update_vector_fields(
+ record_id=slot.record_id,
+ uri=dir_uri,
+ level=level,
+ field_patch=slot.field_patch.with_seed(
+ {
+ **dict(slot.existing_fields or {}),
+ "uri": dir_uri,
+ "level": level,
+ "account_id": self._ctx.account_id,
+ }
+ ),
+ ctx=self._ctx,
+ )
+ if enqueued:
+ enqueued_levels.add(level)
+
+ if self._semantic_plan is not None:
+ self._scheduled_vector_record_ids.update(
+ slot.record_id for slot in slots.values() if slot.level in enqueued_levels
+ )
finally:
self._stats.done_nodes += 1
self._stats.in_progress_nodes = max(0, self._stats.in_progress_nodes - 1)
@@ -1231,8 +1621,8 @@ class SemanticDagExecutor:
await self._on_child_done(parent_uri, dir_uri, abstract or "")
self._release_dir_node(dir_uri)
- def get_stats(self) -> DagStats:
- return DagStats(
+ def get_stats(self) -> SemanticTreeStats:
+ return SemanticTreeStats(
total_nodes=self._stats.total_nodes,
pending_nodes=self._stats.pending_nodes,
in_progress_nodes=self._stats.in_progress_nodes,
@@ -1247,6 +1637,10 @@ class SemanticDagExecutor:
return self._root_write_result
+ @property
+ def scheduled_vector_record_ids(self) -> frozenset[str]:
+ return frozenset(self._scheduled_vector_record_ids)
+
if False: # pragma: no cover - for type checkers only
from openviking.storage.queuefs.semantic_processor import SemanticProcessor
diff --git a/openviking/storage/queuefs/semantic_msg.py b/openviking/storage/queuefs/semantic_msg.py
index 5a60c8161..3e6f8dfb0 100644
--- a/openviking/storage/queuefs/semantic_msg.py
+++ b/openviking/storage/queuefs/semantic_msg.py
@@ -8,6 +8,7 @@ from datetime import datetime
from typing import Any, Dict, List, Optional, Sequence
from uuid import uuid4
+from openviking.storage.context_update_plan import SemanticPlan
from openviking.utils.ingest_options import IngestOptions
@@ -37,7 +38,7 @@ class SemanticMsg:
enqueue them for processing (bottom-up order).
When False, only the specified directory will be processed.
use_hierarchical_aggregation: Route memory directories through the
- shared directory DAG instead of the specialized flat memory
+ shared semantic tree instead of the specialized flat memory
update path.
propagate_to_parent: Whether a completed directory refresh may enqueue
a freshness refresh for its parent.
@@ -48,6 +49,7 @@ class SemanticMsg:
context_type: str # resource, memory, skill, session
status: str = "pending" # pending/processing/completed
timestamp: int = field(default_factory=lambda: int(datetime.now().timestamp()))
+ queue_enqueued_at: float = 0.0
recursive: bool = True # Whether to recursively process subdirectories
account_id: str = "default"
user_id: str = "default"
@@ -73,6 +75,14 @@ class SemanticMsg:
use_hierarchical_aggregation: bool = False
propagate_to_parent: bool = True
copy_source_uri: str = ""
+ # Per-file md5 of final stored bytes, keyed by target URI. Supplied by the
+ # local incremental apply so the tree executor's re-vectorization writes a fresh
+ # fingerprint; empty for all other flows.
+ file_md5s: Dict[str, str] = field(default_factory=dict)
+ artifact_ref: Optional[Dict[str, Any]] = None
+ artifact_files: List[str] = field(default_factory=list)
+ file_abstracts: Dict[str, str] = field(default_factory=dict)
+ plan: Optional[SemanticPlan] = None
def __init__(
self,
@@ -100,9 +110,16 @@ class SemanticMsg:
use_hierarchical_aggregation: bool = False,
propagate_to_parent: bool = True,
copy_source_uri: str = "",
+ file_md5s: Optional[Dict[str, str]] = None,
+ artifact_ref: Optional[Dict[str, Any]] = None,
+ artifact_files: Optional[List[str]] = None,
+ file_abstracts: Optional[Dict[str, str]] = None,
+ plan: SemanticPlan | Dict[str, Any] | None = None,
+ queue_enqueued_at: float = 0.0,
):
self.id = str(uuid4())
self.timestamp = int(datetime.now().timestamp())
+ self.queue_enqueued_at = max(float(queue_enqueued_at or 0.0), 0.0)
self.uri = uri
self.context_type = context_type
self.recursive = recursive
@@ -127,6 +144,17 @@ class SemanticMsg:
self.use_hierarchical_aggregation = bool(use_hierarchical_aggregation)
self.propagate_to_parent = bool(propagate_to_parent)
self.copy_source_uri = copy_source_uri
+ self.file_md5s = dict(file_md5s or {})
+ self.artifact_ref = dict(artifact_ref) if artifact_ref else None
+ self.artifact_files = list(artifact_files or [])
+ self.file_abstracts = dict(file_abstracts or {})
+ self.plan = (
+ plan
+ if isinstance(plan, SemanticPlan)
+ else SemanticPlan.from_dict(plan)
+ if isinstance(plan, dict)
+ else None
+ )
def to_dict(self) -> Dict[str, Any]:
"""Convert object to dictionary."""
@@ -186,6 +214,18 @@ class SemanticMsg:
use_hierarchical_aggregation=data.get("use_hierarchical_aggregation", False),
propagate_to_parent=data.get("propagate_to_parent", True),
copy_source_uri=data.get("copy_source_uri", ""),
+ file_md5s=(data.get("file_md5s") if isinstance(data.get("file_md5s"), dict) else None),
+ artifact_ref=(
+ data.get("artifact_ref") if isinstance(data.get("artifact_ref"), dict) else None
+ ),
+ artifact_files=(
+ data.get("artifact_files") if isinstance(data.get("artifact_files"), list) else None
+ ),
+ file_abstracts=(
+ data.get("file_abstracts") if isinstance(data.get("file_abstracts"), dict) else None
+ ),
+ plan=data.get("plan") if isinstance(data.get("plan"), dict) else None,
+ queue_enqueued_at=data.get("queue_enqueued_at", 0.0),
)
if "id" in data and data["id"]:
obj.id = data["id"]
diff --git a/openviking/storage/queuefs/semantic_processor.py b/openviking/storage/queuefs/semantic_processor.py
index b8b1d6cf0..1fd57bc66 100644
--- a/openviking/storage/queuefs/semantic_processor.py
+++ b/openviking/storage/queuefs/semantic_processor.py
@@ -5,6 +5,7 @@
import asyncio
import re
import threading
+import time
from contextlib import nullcontext
from typing import Any, Awaitable, Callable, Dict, List, Optional, Set, Tuple
from urllib.parse import quote, unquote, urlsplit
@@ -48,9 +49,10 @@ from openviking.storage.abstract_overview import (
)
from openviking.storage.acl import CreatorAclGrant
from openviking.storage.errors import LockAcquisitionError
+from openviking.storage.index_action import FieldPatch
from openviking.storage.queuefs.named_queue import DequeueHandlerBase
from openviking.storage.queuefs.process_result import ProcessResult
-from openviking.storage.queuefs.semantic_dag import DagStats, SemanticDagExecutor
+from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor, SemanticTreeStats
from openviking.storage.queuefs.semantic_lock import SemanticLockScope
from openviking.storage.queuefs.semantic_msg import SemanticMsg, build_semantic_coalesce_key
from openviking.storage.queuefs.semantic_ops.freshness_policy import FreshnessAction
@@ -96,9 +98,25 @@ class SemanticProcessor(DequeueHandlerBase):
"""
_stats_lock = threading.Lock()
- _dag_stats_by_telemetry_id: Dict[str, DagStats] = {}
- _dag_stats_by_uri: Dict[str, DagStats] = {}
- _dag_stats_order: List[Tuple[str, str]] = []
+
+ @staticmethod
+ async def _cleanup_local_artifact(msg: SemanticMsg) -> None:
+ try:
+ if not msg.artifact_ref:
+ return
+ from openviking.parse.output import ParseArtifactRef, store_for_artifact_ref
+
+ artifact_ref = ParseArtifactRef.from_dict(msg.artifact_ref)
+ if artifact_ref.backend != "local":
+ return
+ store = store_for_artifact_ref(artifact_ref)
+ await store.cleanup(artifact_ref)
+ except Exception as exc:
+ logger.warning("Failed to clean local parse artifact: %s", exc)
+
+ _tree_stats_by_telemetry_id: Dict[str, SemanticTreeStats] = {}
+ _tree_stats_by_uri: Dict[str, SemanticTreeStats] = {}
+ _tree_stats_order: List[Tuple[str, str]] = []
_request_stats_by_telemetry_id: Dict[str, RequestQueueStats] = {}
_request_stats_order: List[str] = []
_max_cached_stats = 256
@@ -121,32 +139,32 @@ class SemanticProcessor(DequeueHandlerBase):
self._embedding_worker_stopped = embedding_worker_stopped
@classmethod
- def _cache_dag_stats(cls, telemetry_id: str, uri: str, stats: DagStats) -> None:
+ def _cache_tree_stats(cls, telemetry_id: str, uri: str, stats: SemanticTreeStats) -> None:
with cls._stats_lock:
if telemetry_id:
- cls._dag_stats_by_telemetry_id[telemetry_id] = stats
- cls._dag_stats_by_uri[uri] = stats
- cls._dag_stats_order.append((telemetry_id, uri))
- if len(cls._dag_stats_order) > cls._max_cached_stats:
- old_telemetry_id, old_uri = cls._dag_stats_order.pop(0)
+ cls._tree_stats_by_telemetry_id[telemetry_id] = stats
+ cls._tree_stats_by_uri[uri] = stats
+ cls._tree_stats_order.append((telemetry_id, uri))
+ if len(cls._tree_stats_order) > cls._max_cached_stats:
+ old_telemetry_id, old_uri = cls._tree_stats_order.pop(0)
if old_telemetry_id:
- cls._dag_stats_by_telemetry_id.pop(old_telemetry_id, None)
- cls._dag_stats_by_uri.pop(old_uri, None)
+ cls._tree_stats_by_telemetry_id.pop(old_telemetry_id, None)
+ cls._tree_stats_by_uri.pop(old_uri, None)
@classmethod
- def consume_dag_stats(
+ def consume_tree_stats(
cls,
telemetry_id: str = "",
uri: Optional[str] = None,
- ) -> Optional[DagStats]:
+ ) -> Optional[SemanticTreeStats]:
with cls._stats_lock:
- if telemetry_id and telemetry_id in cls._dag_stats_by_telemetry_id:
- stats = cls._dag_stats_by_telemetry_id.pop(telemetry_id, None)
+ if telemetry_id and telemetry_id in cls._tree_stats_by_telemetry_id:
+ stats = cls._tree_stats_by_telemetry_id.pop(telemetry_id, None)
if uri:
- cls._dag_stats_by_uri.pop(uri, None)
+ cls._tree_stats_by_uri.pop(uri, None)
return stats
- if uri and uri in cls._dag_stats_by_uri:
- return cls._dag_stats_by_uri.pop(uri, None)
+ if uri and uri in cls._tree_stats_by_uri:
+ return cls._tree_stats_by_uri.pop(uri, None)
return None
@classmethod
@@ -285,6 +303,7 @@ class SemanticProcessor(DequeueHandlerBase):
logger.error(f"Failed to re-enqueue semantic message: {requeue_err}")
self._merge_request_stats(msg.telemetry_id, error_count=1)
get_request_wait_tracker().mark_semantic_failed(msg.telemetry_id, msg.id, str(error))
+ await self._cleanup_local_artifact(msg)
return ProcessResult.failed(str(error))
return ProcessResult.requeued()
@@ -392,6 +411,9 @@ class SemanticProcessor(DequeueHandlerBase):
msg: Optional[SemanticMsg] = None
collector = None
work: Optional[SemanticMessageWork] = None
+ execute_started_at: float | None = None
+ queue_wait_ms = 0.0
+ execute_status = "ok"
try:
import json
@@ -405,10 +427,14 @@ class SemanticProcessor(DequeueHandlerBase):
msg = SemanticMsg.from_dict(data)
work = self._message_work(msg, lock)
work.start()
+ execute_started_at = time.perf_counter()
+ if msg.queue_enqueued_at > 0:
+ queue_wait_ms = max((time.time() - msg.queue_enqueued_at) * 1000.0, 0.0)
if VikingURI(msg.uri).parent is None:
logger.warning("Skipping semantic generation for root URI: %s", msg.uri)
if msg.telemetry_id and msg.id:
get_request_wait_tracker().mark_semantic_done(msg.telemetry_id, msg.id)
+ await self._cleanup_local_artifact(msg)
return ProcessResult.success()
if is_semantic_msg_stale(msg):
live_file_changes = {
@@ -438,6 +464,7 @@ class SemanticProcessor(DequeueHandlerBase):
await work.skip()
if msg.telemetry_id and msg.id:
get_request_wait_tracker().mark_semantic_done(msg.telemetry_id, msg.id)
+ await self._cleanup_local_artifact(msg)
return ProcessResult.success()
# Circuit breaker: if API is known-broken, re-enqueue and wait
try:
@@ -468,7 +495,7 @@ class SemanticProcessor(DequeueHandlerBase):
f"Processing semantic generation for: {msg.uri} (recursive={msg.recursive})"
)
- logger.info(f"Processing semantic generation for: {msg})")
+ logger.debug("Processing semantic message id=%s uri=%s", msg.id, msg.uri)
# Resolving a deleted root can recreate it for lock metadata.
# Settle queued ownership before acknowledging skipped work.
@@ -487,10 +514,64 @@ class SemanticProcessor(DequeueHandlerBase):
dag_stats = None
processing_succeeded = False
try:
+ if msg.plan is not None:
+ if msg.uri.rstrip("/") != msg.plan.root_uri:
+ raise ValueError("semantic message URI must match plan root_uri")
+ if msg.context_type != msg.plan.context_type:
+ raise ValueError(
+ "semantic message context_type must match semantic plan"
+ )
+ for run_uri in msg.plan.execution_root_uris():
+ executor = SemanticTreeExecutor(
+ processor=self,
+ context_type=msg.context_type,
+ max_concurrent_llm=self.max_concurrent_llm,
+ ctx=current_ctx,
+ lock=semantic_lock.lock,
+ source=msg.plan.source_metadata,
+ semantic_plan=msg.plan,
+ )
+ await executor.run(run_uri)
+ self._cache_tree_stats(
+ msg.telemetry_id, run_uri, executor.get_stats()
+ )
+ if not executor.stale and msg.plan.propagation.enabled:
+ write_result = getattr(
+ executor,
+ "root_write_result",
+ AbstractOverviewWriteResult(
+ wrote=True, abstract_body_changed=True
+ ),
+ )
+ await self._enqueue_parent_refresh(
+ msg,
+ run_uri,
+ l0_body_changed=write_result.abstract_body_changed,
+ )
+ from collections import Counter
+
+ entries = msg.plan.tree.entries
+ action_counts = Counter(
+ entry.semantic_action.value for entry in entries
+ )
+ vector_slots = sum(
+ slot.action.value in {"upsert", "merge"}
+ for entry in entries
+ for slot in entry.index_slots
+ )
+ logger.debug(
+ "[SemanticPlanExecution] root=%s execution_roots=%d "
+ "semantic_entries=%d actions=%s planned_vector_upserts=%d",
+ msg.plan.root_uri,
+ len(msg.plan.execution_root_uris()),
+ len(entries),
+ dict(action_counts),
+ vector_slots,
+ )
# Regular memory writes keep their specialized update path.
# Callers must explicitly opt into directory aggregation; the
# trigger remains descriptive metadata, not an algorithm switch.
- if msg.context_type == "memory" and not msg.use_hierarchical_aggregation:
+ elif msg.context_type == "memory" and not msg.use_hierarchical_aggregation:
await self._process_memory_directory(
msg,
ctx=current_ctx,
@@ -531,19 +612,23 @@ class SemanticProcessor(DequeueHandlerBase):
is_incremental = True
target_uri = msg.target_uri
run_uri = msg.target_uri
- elif target_exists and msg.changes and msg.uri == msg.target_uri:
+ elif (
+ target_exists
+ and msg.changes is not None
+ and msg.uri == msg.target_uri
+ ):
is_incremental = True
logger.info(
f"Using direct incremental semantic update for: {msg.uri}"
)
- elif msg.changes:
+ elif msg.changes is not None:
is_incremental = True
target_uri = msg.uri
logger.info(
f"Using direct incremental semantic update for: {msg.uri}"
)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=self,
context_type=msg.context_type,
max_concurrent_llm=self.max_concurrent_llm,
@@ -563,10 +648,13 @@ class SemanticProcessor(DequeueHandlerBase):
generation_trigger=msg.generation_trigger,
aggregate_directory=msg.aggregate_directory,
copy_source_uri=msg.copy_source_uri,
+ file_md5s=msg.file_md5s,
+ artifact_files=msg.artifact_files,
+ file_abstracts=msg.file_abstracts,
)
await executor.run(run_uri)
dag_stats = executor.get_stats()
- self._cache_dag_stats(
+ self._cache_tree_stats(
msg.telemetry_id,
run_uri,
dag_stats,
@@ -594,6 +682,7 @@ class SemanticProcessor(DequeueHandlerBase):
self._merge_request_stats(msg.telemetry_id, processed=1)
logger.info(f"Completed semantic generation for: {msg.uri}")
self._circuit_breaker.record_success()
+ await self._cleanup_local_artifact(msg)
return ProcessResult.success()
finally:
reset_root_observability_context(root_context_token)
@@ -604,6 +693,7 @@ class SemanticProcessor(DequeueHandlerBase):
raise
except Exception as e:
if isinstance(e, LockAcquisitionError):
+ execute_status = "requeued"
logger.warning(
"Lock error processing semantic message, re-enqueueing without "
"tripping API circuit breaker: %s",
@@ -620,6 +710,7 @@ class SemanticProcessor(DequeueHandlerBase):
error_class = classify_api_error(e)
if error_class == ERROR_CLASS_INPUT_TOO_LARGE:
+ execute_status = "error"
logger.error(
f"Input too large processing semantic message, dropping: {e}",
exc_info=True,
@@ -629,8 +720,11 @@ class SemanticProcessor(DequeueHandlerBase):
get_request_wait_tracker().mark_semantic_failed(
msg.telemetry_id, msg.id, str(e)
)
+ if msg is not None:
+ await self._cleanup_local_artifact(msg)
return ProcessResult.failed(str(e))
elif error_class == ERROR_CLASS_PERMANENT:
+ execute_status = "error"
logger.critical(
f"Permanent error processing semantic message, dropping: {e}",
exc_info=True,
@@ -643,9 +737,12 @@ class SemanticProcessor(DequeueHandlerBase):
get_request_wait_tracker().mark_semantic_failed(
msg.telemetry_id, msg.id, str(e)
)
+ if msg is not None:
+ await self._cleanup_local_artifact(msg)
return ProcessResult.failed(str(e))
else:
# Transient or unknown — re-enqueue for retry
+ execute_status = "requeued"
logger.warning(
f"Transient API error processing semantic message, re-enqueueing: {e}",
exc_info=True,
@@ -658,8 +755,30 @@ class SemanticProcessor(DequeueHandlerBase):
work=work,
)
return ProcessResult.failed(str(e))
-
finally:
+ if msg is not None and execute_started_at is not None:
+ tracker = get_request_wait_tracker()
+ record_timing = getattr(tracker, "record_semantic_timing", None)
+ if callable(record_timing):
+ record_timing(
+ msg.telemetry_id,
+ queue_wait_ms=queue_wait_ms,
+ execute_ms=(time.perf_counter() - execute_started_at) * 1000.0,
+ )
+ from openviking.metrics.datasources.resource import ResourceIngestionEventDataSource
+
+ ResourceIngestionEventDataSource.record_stage(
+ stage="semantic_queue_wait",
+ status=execute_status,
+ duration_seconds=queue_wait_ms / 1000.0,
+ account_id=msg.account_id,
+ )
+ ResourceIngestionEventDataSource.record_stage(
+ stage="semantic_execute",
+ status=execute_status,
+ duration_seconds=(time.perf_counter() - execute_started_at),
+ account_id=msg.account_id,
+ )
if work is not None:
await work.close()
@@ -676,6 +795,7 @@ class SemanticProcessor(DequeueHandlerBase):
return ProcessResult.failed(str(exc))
await self._message_work(msg).cancel_queued()
+ await self._cleanup_local_artifact(msg)
return ProcessResult.cancelled()
async def _release_cancelled_semantic_lock(self, msg: SemanticMsg) -> None:
@@ -718,8 +838,8 @@ class SemanticProcessor(DequeueHandlerBase):
await run_to_completion(scope.close)
raise
- def get_dag_stats(self) -> Optional["DagStats"]:
- return SemanticDagExecutor.get_active_stats()
+ def get_tree_stats(self) -> Optional["SemanticTreeStats"]:
+ return SemanticTreeExecutor.get_active_stats()
async def _process_memory_directory(
self,
@@ -1078,13 +1198,18 @@ class SemanticProcessor(DequeueHandlerBase):
file_name: str,
llm_sem: asyncio.Semaphore,
ctx: Optional[RequestContext] = None,
+ file_content: Optional[bytes] = None,
) -> Dict[str, Any]:
"""Generate summary for a single text file (code, documentation, or other text)."""
viking_fs = get_viking_fs()
vlm = get_openviking_config().vlm
active_ctx = ctx or self._default_ctx
- content = await viking_fs.read_file(file_path, ctx=active_ctx)
+ content = (
+ file_content
+ if file_content is not None
+ else await viking_fs.read_file(file_path, ctx=active_ctx)
+ )
if isinstance(content, bytes):
from openviking.utils.embedding_utils import _decode_text_bytes
@@ -1125,7 +1250,7 @@ class SemanticProcessor(DequeueHandlerBase):
{"file_name": file_name, "content": content, "output_language": output_language},
)
async with llm_sem:
- with bind_telemetry_stage("resource_summarize"):
+ with bind_telemetry_stage("semantic_execute"):
summary = await vlm.get_completion_async(prompt)
return result(summary.strip())
@@ -1147,7 +1272,7 @@ class SemanticProcessor(DequeueHandlerBase):
)
async with llm_sem:
- with bind_telemetry_stage("resource_summarize"):
+ with bind_telemetry_stage("semantic_execute"):
summary = await vlm.get_completion_async(prompt)
return result(summary.strip())
@@ -1156,6 +1281,7 @@ class SemanticProcessor(DequeueHandlerBase):
file_path: str,
llm_sem: Optional[asyncio.Semaphore] = None,
ctx: Optional[RequestContext] = None,
+ file_content: Optional[bytes] = None,
) -> Dict[str, Any]:
"""Generate summary for a single file.
@@ -1170,11 +1296,15 @@ class SemanticProcessor(DequeueHandlerBase):
media_type = get_media_type(file_name, None)
if file_name.lower().endswith(".ts"):
try:
- prefix = await get_viking_fs().read(
- file_path,
- offset=0,
- size=MPEG_TS_PROBE_BYTES,
- ctx=ctx,
+ prefix = (
+ file_content[:MPEG_TS_PROBE_BYTES]
+ if file_content is not None
+ else await get_viking_fs().read(
+ file_path,
+ offset=0,
+ size=MPEG_TS_PROBE_BYTES,
+ ctx=ctx,
+ )
)
except Exception:
prefix = None
@@ -1186,7 +1316,9 @@ class SemanticProcessor(DequeueHandlerBase):
elif media_type == "video":
return await generate_video_summary(file_path, file_name, llm_sem, ctx=ctx)
else:
- return await self._generate_text_summary(file_path, file_name, llm_sem, ctx=ctx)
+ return await self._generate_text_summary(
+ file_path, file_name, llm_sem, ctx=ctx, file_content=file_content
+ )
def _child_summary_line(
self,
@@ -1539,7 +1671,7 @@ class SemanticProcessor(DequeueHandlerBase):
},
)
- with bind_telemetry_stage("resource_summarize"):
+ with bind_telemetry_stage("semantic_execute"):
overview = await vlm.get_completion_async(prompt)
overview = self._replace_link_references(overview, link_map)
@@ -1618,7 +1750,7 @@ class SemanticProcessor(DequeueHandlerBase):
async def _run_batch(batch_idx: int, prompt: str, batch_link_map: Dict[str, str]) -> None:
try:
async with llm_sem:
- with bind_telemetry_stage("resource_summarize"):
+ with bind_telemetry_stage("semantic_execute"):
partial = await vlm.get_completion_async(prompt)
partial = self._replace_link_references(partial, batch_link_map)
partial_overviews[batch_idx] = partial.strip()
@@ -1654,7 +1786,7 @@ class SemanticProcessor(DequeueHandlerBase):
"directory_coverage": directory_coverage,
},
)
- with bind_telemetry_stage("resource_summarize"):
+ with bind_telemetry_stage("semantic_execute"):
overview = await vlm.get_completion_async(prompt)
overview = self._replace_link_references(overview, link_map)
return overview.strip()
@@ -1721,7 +1853,12 @@ class SemanticProcessor(DequeueHandlerBase):
ingest_options: IngestOptions | None = None,
creator_acl_grant: CreatorAclGrant | None = None,
skill_source_path: str = "",
- ) -> None:
+ scalar_overrides: Optional[Dict[int, Dict[str, Any]]] = None,
+ actions: Optional[Dict[int, str]] = None,
+ field_patches: Optional[Dict[int, FieldPatch]] = None,
+ include_abstract: bool = True,
+ include_overview: bool = True,
+ ) -> set[int]:
"""Create directory Context and enqueue to EmbeddingQueue."""
from openviking.utils.embedding_utils import vectorize_directory_meta
@@ -1738,7 +1875,7 @@ class SemanticProcessor(DequeueHandlerBase):
for key in ("name", "description", "tags", "allowed_tools")
}
skill_meta["source_path"] = skill_source_path
- await vectorize_directory_meta(
+ return await vectorize_directory_meta(
uri=uri,
abstract=abstract,
overview=overview,
@@ -1748,6 +1885,11 @@ class SemanticProcessor(DequeueHandlerBase):
creator_acl_grant=creator_acl_grant,
content_is_body=context_type == "skill",
**({"meta": skill_meta} if skill_meta is not None else {}),
+ scalar_overrides=scalar_overrides,
+ actions=actions,
+ field_patches=field_patches,
+ include_abstract=include_abstract,
+ include_overview=include_overview,
)
async def _load_transfer_file_summaries(
@@ -1765,6 +1907,39 @@ class SemanticProcessor(DequeueHandlerBase):
active_ctx = ctx or self._default_ctx
return await vector_store.get_l2_abstracts_by_uris(file_paths, ctx=active_ctx)
+ async def _update_vector_fields(
+ self,
+ *,
+ record_id: str,
+ uri: str,
+ level: int,
+ field_patch: FieldPatch,
+ ctx: RequestContext,
+ ) -> bool:
+ from openviking.storage.queuefs import get_queue_manager
+ from openviking.storage.queuefs.embedding_msg import EmbeddingMsg
+ from openviking.telemetry import get_current_telemetry
+ from openviking.utils.embedding_utils import _enqueue_embedding_message
+
+ embedding_msg = EmbeddingMsg.for_update_fields(
+ record_id=record_id,
+ field_patch=field_patch,
+ context_data={
+ "uri": uri,
+ "level": level,
+ "account_id": ctx.account_id,
+ "owner_user_id": ctx.user.user_id,
+ },
+ telemetry_id=get_current_telemetry().telemetry_id,
+ )
+ queue_manager = get_queue_manager()
+ embedding_queue = queue_manager.get_queue(queue_manager.EMBEDDING, allow_create=True)
+ return await _enqueue_embedding_message(
+ embedding_queue,
+ embedding_msg,
+ failure_message=f"Failed to enqueue vector scalar update for {uri}",
+ )
+
async def _vectorize_single_file(
self,
parent_uri: str,
@@ -1776,6 +1951,11 @@ class SemanticProcessor(DequeueHandlerBase):
preserve_existing_created_at: bool = False,
ingest_options: IngestOptions | None = None,
creator_acl_grant: CreatorAclGrant | None = None,
+ file_md5: Optional[str] = None,
+ file_content: Optional[bytes] = None,
+ scalar_override: Optional[Dict[str, Any]] = None,
+ field_patch: FieldPatch | None = None,
+ action: str = "merge",
) -> bool:
"""Vectorize a single file using its content or summary."""
from openviking.utils.embedding_utils import vectorize_file
@@ -1791,4 +1971,9 @@ class SemanticProcessor(DequeueHandlerBase):
preserve_existing_created_at=preserve_existing_created_at,
ingest_options=ingest_options,
creator_acl_grant=creator_acl_grant,
+ file_md5=file_md5,
+ file_content=file_content,
+ scalar_override=scalar_override,
+ field_patch=field_patch,
+ action=action,
)
diff --git a/openviking/storage/queuefs/semantic_queue.py b/openviking/storage/queuefs/semantic_queue.py
index 48e35bbf5..e85db2280 100644
--- a/openviking/storage/queuefs/semantic_queue.py
+++ b/openviking/storage/queuefs/semantic_queue.py
@@ -70,6 +70,8 @@ class SemanticQueue(NamedQueue):
_SEMANTIC_COALESCE_VERSION[msg.coalesce_key] = version
msg.coalesce_version = version
+ msg.queue_enqueued_at = time.time()
+
return await super().enqueue(msg.to_dict())
async def dequeue(self) -> Optional[SemanticMsg]:
diff --git a/openviking/storage/queuefs/semantic_work.py b/openviking/storage/queuefs/semantic_work.py
index db1970c23..553221af5 100644
--- a/openviking/storage/queuefs/semantic_work.py
+++ b/openviking/storage/queuefs/semantic_work.py
@@ -10,7 +10,7 @@ from typing import TYPE_CHECKING, Any, Awaitable, Callable, TypeVar
from openviking.server.identity import RequestContext
from openviking.service.task_tracker_concurrency import run_to_completion
from openviking.storage.queuefs.process_result import ProcessResult
-from openviking.storage.queuefs.semantic_dag import DagStats
+from openviking.storage.queuefs.semantic_executor import SemanticTreeStats
from openviking.storage.queuefs.semantic_lock import SemanticLockScope
from openviking.storage.queuefs.semantic_msg import SemanticMsg
from openviking.storage.viking_fs import get_viking_fs
@@ -63,7 +63,7 @@ class SemanticMessageWork:
assert self.scope is not None
await self.scope.close()
- def failure_result(self, stats: DagStats | None) -> ProcessResult | None:
+ def failure_result(self, stats: SemanticTreeStats | None) -> ProcessResult | None:
return None
async def reenqueue(self) -> None:
@@ -151,7 +151,7 @@ class SkillSemanticMessageWork(SemanticMessageWork):
# On failure retain the lease until reenqueue hands it to the retry.
await run_to_completion(finish)
- def failure_result(self, stats: DagStats | None) -> ProcessResult | None:
+ def failure_result(self, stats: SemanticTreeStats | None) -> ProcessResult | None:
if stats is None or not stats.failures:
return None
for failure in stats.failures:
diff --git a/openviking/storage/resource_diff.py b/openviking/storage/resource_diff.py
new file mode 100644
index 000000000..3045c38e9
--- /dev/null
+++ b/openviking/storage/resource_diff.py
@@ -0,0 +1,644 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Read the R / N / F / V snapshots used by resource update planning.
+
+This module gathers the typed R/N/F/V inputs used by the canonical resource
+update plan.
+
+- ``R`` normalized request intent — target, processing mode, and scalar fields
+ such as tags that this request explicitly wants to mutate.
+
+- ``N`` new-artifact manifest — the files a parser produced, read from the
+ parse output store. Canonical add-resource prepares it once before planning:
+ image references are rewritten to their final URI and their final-byte md5 is
+ retained in the resulting inventory.
+- ``F`` target file tree — one unbounded ``tree`` call. Any permission-denied
+ subtree (or a truncated scan) marks the snapshot incomplete so the planner
+ refuses deletions instead of treating unreadable files as removed.
+- ``V`` target vectors — lightweight L0/L1/L2 records under the target URI.
+
+The canonical ``add_resources`` path resolves these snapshots into a
+``ResourceDiffResult`` before building a ``ContextUpdatePlan``.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import logging
+from collections import Counter
+from dataclasses import dataclass
+from enum import Enum
+from typing import Any, Dict, Mapping, Tuple
+
+from openviking.concurrency import bounded_map
+from openviking.storage.internal_names import STORAGE_INTERNAL_ENTRY_NAMES
+from openviking.storage.resource_rnfv import (
+ CONTROL_BASENAMES,
+ FormalEntry,
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ NewEntry,
+ RequestIntent,
+ RNFVSnapshot,
+ VectorIndexSnapshot,
+ VectorRecordSnapshot,
+ canonical_vector_records_by_level,
+)
+from openviking.utils.log_correlation import log_correlation
+
+logger = logging.getLogger(__name__)
+
+
+class ContentState(str, Enum):
+ """Relationship between the new artifact (N) and formal tree (F).
+
+ F is the source of truth for whether content currently exists. Vector
+ records are deliberately not part of this axis; their presence is expressed
+ independently by :class:`IndexState`.
+ """
+
+ # Neither N nor F contains this path.
+ ABSENT = "absent"
+ # N and F contain the same file bytes, or the same directory node.
+ UNCHANGED = "unchanged"
+ # N contains a path that has no formal file and no prior vector records.
+ ADDED = "added"
+ # N and F both contain a file, but their bytes differ.
+ MODIFIED = "modified"
+ # F contains a path omitted from N.
+ DELETED = "deleted"
+ # N contains a path absent from F while V still has records for that path.
+ RESTORE = "restore"
+ # N and F contain the path with different node kinds (file versus directory).
+ REPLACE_KIND = "replace_kind"
+
+
+class IndexState(str, Enum):
+ """Health of V for the node kind selected by N/F.
+
+ A file is valid only with L2. A directory is valid only with both L0 and
+ L1. This axis is independent from content change, so an unchanged file can
+ still require index repair.
+ """
+
+ # No formal node and no vector record exist for the path.
+ ABSENT = "absent"
+ # All and only the levels expected for the current node kind are present.
+ COMPLETE = "complete"
+ # None of the levels expected for the current node kind are present.
+ MISSING = "missing"
+ # Some expected levels are present, but the complete expected set is not.
+ PARTIAL = "partial"
+ # Expected levels exist but are derived from content known to be out of date.
+ STALE = "stale"
+ # V belongs to content removed by the current update.
+ OBSOLETE = "obsolete"
+ # V has records while neither N nor F has a corresponding content node.
+ ORPHAN = "orphan"
+ # V contains a level that is invalid for the current file/directory kind.
+ LEVEL_CONFLICT = "level_conflict"
+
+
+@dataclass(frozen=True)
+class ResourceDiffEntry:
+ """One path's independent content and index facts after RNFV resolution.
+
+ ``content_state`` determines formal-tree mutation. ``index_state`` determines
+ vector repair or cleanup. ``md5`` is the final new-file fingerprint when the
+ path is a file; directories intentionally never carry an aggregate MD5.
+ """
+
+ relative_path: str
+ content_state: ContentState
+ index_state: IndexState
+ old_kind: str | None = None
+ new_kind: str | None = None
+ md5: str | None = None
+
+
+@dataclass(frozen=True)
+class ResourceDiffResult:
+ entries: Mapping[str, ResourceDiffEntry]
+
+
+@dataclass(frozen=True)
+class ArtifactInventory:
+ """Prepared N snapshot plus paths relative to its parse artifact.
+
+ ``entries`` are rooted at the final resource URI; ``artifact_paths`` point
+ to the same files under the parse artifact. The latter keeps diff body
+ fallbacks independent from the parser's optional document wrapper.
+ """
+
+ entries: Mapping[str, NewEntry]
+ artifact_paths: Mapping[str, str]
+ rewritten_paths: frozenset[str] = frozenset()
+
+
+def count_tree_entry_kinds(
+ entries: Mapping[str, NewEntry | FormalEntry],
+) -> tuple[int, int, bool]:
+ """Return business file/dir counts and whether a logical root is present.
+
+ Directory plans add ``relative_path=""`` as a traversal-only root. Keeping
+ it separate prevents logs from presenting that synthetic node as a file or
+ directory stored below the target. A flat resource uses the same empty path
+ with ``is_dir=False`` and is therefore counted as one real file.
+ """
+ logical_root = bool((root := entries.get("")) is not None and root.is_dir)
+ files = sum(not entry.is_dir for entry in entries.values())
+ directories = sum(entry.is_dir for path, entry in entries.items() if path)
+ return files, directories, logical_root
+
+
+def _index_state(
+ records: tuple[VectorRecordSnapshot, ...],
+ *,
+ kind: str | None,
+ content_state: ContentState,
+ md5: str | None,
+) -> IndexState:
+ """Classify V with strict precedence.
+
+ Content removed by this update makes its V record obsolete. A vector with
+ neither new nor formal content is an orphan. For an existing node, invalid
+ levels win over missing or stale checks; then missing expected levels, then
+ content/fingerprint staleness, and finally complete.
+ """
+ if content_state is ContentState.DELETED:
+ return IndexState.OBSOLETE if records else IndexState.ABSENT
+ if kind is None:
+ return IndexState.ORPHAN if records else IndexState.ABSENT
+ valid_levels = {2} if kind == "file" else {0, 1}
+ levels = {record.level for record in records}
+ if levels - valid_levels:
+ return IndexState.LEVEL_CONFLICT
+ if not levels:
+ return IndexState.MISSING
+ if levels != valid_levels:
+ return IndexState.PARTIAL
+ if content_state == ContentState.MODIFIED:
+ return IndexState.STALE
+ if kind == "file" and md5:
+ record_md5 = next(
+ (str(record.fields.get("md5") or "") for record in records if record.level == 2),
+ "",
+ )
+ if record_md5 and record_md5 != md5:
+ return IndexState.STALE
+ return IndexState.COMPLETE
+
+
+async def resolve_resource_diff(
+ snapshot: RNFVSnapshot,
+ *,
+ store: Any,
+ artifact_ref: Any,
+ target: Any,
+ artifact_paths: Mapping[str, str] | None = None,
+ concurrency: int | None = None,
+) -> ResourceDiffResult:
+ """Resolve R/N/F/V into final states, including bounded body fallbacks."""
+ snapshot.validate_for_planning()
+ if (
+ not snapshot.new.complete
+ or not snapshot.formal.complete
+ or (snapshot.request.vectorize and not snapshot.vectors.complete)
+ ):
+ raise ValueError("cannot resolve an incomplete RNFV snapshot")
+
+ new = snapshot.new.entries
+ formal = snapshot.formal.entries
+ vector_records = snapshot.vectors.records_by_id if snapshot.request.vectorize else {}
+ records_by_rel: dict[str, list[VectorRecordSnapshot]] = {}
+ for record in vector_records.values():
+ records_by_rel.setdefault(record.relative_path, []).append(record)
+
+ states: dict[str, tuple[ContentState, str | None]] = {}
+ compare_paths: list[str] = []
+ hash_paths: list[str] = []
+ md5_fast_path_count = 0
+ keys = set(new) | set(formal) | set(records_by_rel)
+ for rel_path in sorted(keys):
+ new_entry = new.get(rel_path)
+ old_entry = formal.get(rel_path)
+ if new_entry is not None and old_entry is not None:
+ new_kind = "directory" if new_entry.is_dir else "file"
+ old_kind = "directory" if old_entry.is_dir else "file"
+ if new_kind != old_kind:
+ states[rel_path] = (ContentState.REPLACE_KIND, new_entry.md5 or None)
+ elif new_entry.is_dir:
+ states[rel_path] = (ContentState.UNCHANGED, None)
+ else:
+ canonical_records, _ = canonical_vector_records_by_level(
+ records_by_rel.get(rel_path, ())
+ )
+ l2_record = canonical_records.get(2)
+ l2_md5 = str(l2_record.fields.get("md5") or "") if l2_record else ""
+ if new_entry.md5 and l2_md5:
+ md5_fast_path_count += 1
+ state = (
+ ContentState.UNCHANGED if new_entry.md5 == l2_md5 else ContentState.MODIFIED
+ )
+ states[rel_path] = (state, new_entry.md5)
+ else:
+ compare_paths.append(rel_path)
+ elif new_entry is not None:
+ state = ContentState.RESTORE if records_by_rel.get(rel_path) else ContentState.ADDED
+ states[rel_path] = (state, new_entry.md5 or None)
+ if not new_entry.is_dir and not new_entry.md5:
+ hash_paths.append(rel_path)
+ elif old_entry is not None:
+ states[rel_path] = (ContentState.DELETED, None)
+ else:
+ states[rel_path] = (ContentState.ABSENT, None)
+
+ async def compare(rel_path: str) -> tuple[str, ContentState, str]:
+ artifact_path = (artifact_paths or {}).get(rel_path, rel_path)
+ new_bytes, old_bytes = await asyncio.gather(
+ store.read_bytes(artifact_ref, artifact_path),
+ target.read_file(rel_path),
+ )
+ from openviking.utils.content_hash import content_md5
+
+ return (
+ rel_path,
+ ContentState.UNCHANGED if new_bytes == old_bytes else ContentState.MODIFIED,
+ content_md5(new_bytes),
+ )
+
+ async def hash_new(rel_path: str) -> tuple[str, str]:
+ artifact_path = (artifact_paths or {}).get(rel_path, rel_path)
+ new_bytes = await store.read_bytes(artifact_ref, artifact_path)
+ from openviking.utils.content_hash import content_md5
+
+ return rel_path, content_md5(new_bytes)
+
+ if concurrency is None:
+ from openviking_cli.utils.config import get_openviking_config
+
+ concurrency = int(
+ get_openviking_config().queue_workers.add_resource.file_operation_concurrency
+ )
+
+ for rel_path, state, md5 in await bounded_map(
+ compare_paths, compare, concurrency=max(1, concurrency)
+ ):
+ states[rel_path] = (state, md5)
+ for rel_path, md5 in await bounded_map(hash_paths, hash_new, concurrency=max(1, concurrency)):
+ state, _ = states[rel_path]
+ states[rel_path] = (state, md5)
+
+ result: dict[str, ResourceDiffEntry] = {}
+ for rel_path in sorted(keys):
+ state, md5 = states[rel_path]
+ new_entry = new.get(rel_path)
+ old_entry = formal.get(rel_path)
+ new_kind = ("directory" if new_entry.is_dir else "file") if new_entry is not None else None
+ old_kind = ("directory" if old_entry.is_dir else "file") if old_entry is not None else None
+ current_kind = (
+ new_kind if state not in {ContentState.DELETED, ContentState.ABSENT} else None
+ )
+ records = tuple(records_by_rel.get(rel_path, ()))
+ canonical_records, _ = canonical_vector_records_by_level(records)
+ result[rel_path] = ResourceDiffEntry(
+ relative_path=rel_path,
+ content_state=state,
+ index_state=_index_state(
+ tuple(canonical_records.values()),
+ kind=current_kind,
+ content_state=state,
+ md5=md5,
+ ),
+ old_kind=old_kind,
+ new_kind=new_kind,
+ md5=md5,
+ )
+ resolved = ResourceDiffResult(entries=result)
+ n_files, n_dirs, n_has_root = count_tree_entry_kinds(new)
+ f_files, f_dirs, f_has_root = count_tree_entry_kinds(formal)
+ file_entries = [
+ entry
+ for path, entry in resolved.entries.items()
+ if path and (entry.new_kind or entry.old_kind) == "file"
+ ]
+ directory_entries = [
+ entry
+ for path, entry in resolved.entries.items()
+ if path and (entry.new_kind or entry.old_kind) == "directory"
+ ]
+ root_entry = resolved.entries.get("")
+ file_content_counts = Counter(entry.content_state.value for entry in file_entries)
+ directory_content_counts = Counter(entry.content_state.value for entry in directory_entries)
+ file_index_counts = Counter(entry.index_state.value for entry in file_entries)
+ directory_index_counts = Counter(entry.index_state.value for entry in directory_entries)
+ logger.info(
+ "[ResourceDiffResult] %s target=%s n_files=%d n_dirs=%d f_files=%d f_dirs=%d "
+ "logical_root=%s v_records=%d root_state=%s root_index=%s "
+ "md5_fast_path=%d body_compared=%d new_files_hashed=%d "
+ "states={file:%s,dir:%s} index={file:%s,dir:%s}",
+ log_correlation(),
+ snapshot.request.target_uri,
+ n_files,
+ n_dirs,
+ f_files,
+ f_dirs,
+ str(n_has_root or f_has_root).lower(),
+ len(vector_records),
+ root_entry.content_state.value if root_entry is not None else "none",
+ root_entry.index_state.value if root_entry is not None else "none",
+ md5_fast_path_count,
+ len(compare_paths),
+ len(hash_paths),
+ dict(file_content_counts),
+ dict(directory_content_counts),
+ dict(file_index_counts),
+ dict(directory_index_counts),
+ )
+ return resolved
+
+
+def _is_excluded_rel_path(rel_path: str) -> bool:
+ """Control sidecars and storage-internal entries are never business files."""
+ if not rel_path:
+ return True
+ for segment in rel_path.split("/"):
+ if segment in STORAGE_INTERNAL_ENTRY_NAMES or segment in CONTROL_BASENAMES:
+ return True
+ return False
+
+
+async def read_target_file_snapshot(
+ viking_fs: Any,
+ target_uri: str,
+ *,
+ ctx: Any,
+ root_is_file: bool = False,
+) -> Tuple[Dict[str, FormalEntry], bool]:
+ """Return ``(rel_path -> FormalEntry, complete)`` for the formal tree.
+
+ ``complete`` is False when any entry is permission-denied, because a subtree
+ we cannot see must not be interpreted as absent (which would drive deletion).
+ """
+ if root_is_file:
+ stat = await viking_fs.stat(target_uri, ctx=ctx, skip_count=True)
+ return {"": FormalEntry(is_dir=bool(stat.get("isDir")))}, True
+
+ entries = await viking_fs.tree(
+ target_uri,
+ output="original",
+ show_all_hidden=True,
+ node_limit=None,
+ level_limit=None,
+ ctx=ctx,
+ )
+ files: Dict[str, FormalEntry] = {}
+ complete = True
+ for entry in entries:
+ if entry.get("access") == "denied":
+ complete = False
+ continue
+ rel_path = str(entry.get("rel_path") or "").strip("/")
+ if _is_excluded_rel_path(rel_path):
+ continue
+ files[rel_path] = FormalEntry(is_dir=bool(entry.get("isDir")))
+ return files, complete
+
+
+async def prepare_artifact_inventory(
+ store: Any,
+ ref: Any,
+ *,
+ doc_rel: str = "",
+ target_root_uri: str | None = None,
+ root_is_file: bool = False,
+) -> ArtifactInventory:
+ """Prepare an artifact once and return its final-byte N snapshot.
+
+ The one recursive walk supplies the file inventory consumed by RNFV and,
+ when a final target URI is known, rewrites mapped markdown image references
+ in place. Rewritten markdown immediately receives an updated manifest MD5,
+ so the next diff compares the exact bytes that would be uploaded.
+ """
+ from openviking.parse.image_rewrite import IMAGE_MAPPINGS_FILENAME, _rewrite_content
+ from openviking.parse.output import read_artifact_manifest, write_artifact_manifest
+ from openviking.utils.content_hash import content_md5
+
+ base = doc_rel.strip("/")
+ prefix = f"{base}/" if base else ""
+ md5_by_artifact_rel = await read_artifact_manifest(store, ref)
+ entries: Dict[str, NewEntry] = {}
+ artifact_paths: Dict[str, str] = {}
+ rewritten_paths: set[str] = set()
+
+ def target_relative(artifact_rel: str) -> str:
+ if base and artifact_rel.startswith(prefix):
+ return artifact_rel[len(prefix) :]
+ return artifact_rel
+
+ if root_is_file:
+ return ArtifactInventory(
+ entries={"": NewEntry(md5=md5_by_artifact_rel.get(base, ""), is_dir=False)},
+ artifact_paths={"": base},
+ )
+
+ async def walk(
+ directory: str,
+ mapping_dir: str = "",
+ inherited_mappings: Mapping[str, Mapping[str, str]] | None = None,
+ ) -> None:
+ directory_entries = await store.list(ref, directory)
+ current_mappings = inherited_mappings or {}
+ current_mapping_dir = mapping_dir
+ mapping_entry = next(
+ (entry for entry in directory_entries if entry.name == IMAGE_MAPPINGS_FILENAME),
+ None,
+ )
+ if mapping_entry is not None:
+ try:
+ loaded = json.loads(
+ (await store.read_bytes(ref, mapping_entry.rel_path)).decode("utf-8")
+ )
+ except Exception as exc:
+ logger.warning(
+ "[ArtifactInventory] Ignoring unreadable image mapping sidecar %s: %s",
+ mapping_entry.rel_path,
+ exc,
+ )
+ loaded = None
+ if not isinstance(loaded, dict):
+ if loaded is not None:
+ logger.warning(
+ "[ArtifactInventory] Ignoring invalid image mapping sidecar %s",
+ mapping_entry.rel_path,
+ )
+ else:
+ current_mappings = loaded
+ current_mapping_dir = directory
+
+ available_files = {
+ entry.name
+ for entry in directory_entries
+ if not entry.is_dir and not entry.name.startswith(".")
+ }
+ for entry in directory_entries:
+ if _is_excluded_rel_path(entry.rel_path):
+ continue
+ relative_path = target_relative(entry.rel_path)
+ if entry.is_dir:
+ if relative_path:
+ entries[relative_path] = NewEntry(is_dir=True)
+ await walk(entry.rel_path, current_mapping_dir, current_mappings)
+ continue
+
+ if not relative_path:
+ continue
+ md5 = md5_by_artifact_rel.get(entry.rel_path, "")
+ if target_root_uri and entry.name.lower().endswith((".md", ".markdown")):
+ mapping_key = (
+ entry.rel_path[len(current_mapping_dir) + 1 :]
+ if current_mapping_dir and entry.rel_path.startswith(current_mapping_dir + "/")
+ else entry.rel_path
+ )
+ path_mappings = current_mappings.get(mapping_key)
+ if isinstance(path_mappings, dict) and path_mappings:
+ content = (await store.read_bytes(ref, entry.rel_path)).decode("utf-8")
+ target_uri = f"{target_root_uri.rstrip('/')}/{relative_path}"
+ rewritten, count = _rewrite_content(
+ content,
+ target_uri.rsplit("/", 1)[0],
+ available_files,
+ {str(key): str(value) for key, value in path_mappings.items()},
+ )
+ if count:
+ final_bytes = rewritten.encode("utf-8")
+ await store.write_bytes(ref, entry.rel_path, final_bytes)
+ md5 = content_md5(final_bytes)
+ md5_by_artifact_rel[entry.rel_path] = md5
+ rewritten_paths.add(relative_path)
+ entries[relative_path] = NewEntry(md5=md5, is_dir=False)
+ artifact_paths[relative_path] = entry.rel_path
+
+ await walk(base)
+ if rewritten_paths:
+ await write_artifact_manifest(store, ref, md5_by_artifact_rel)
+ return ArtifactInventory(
+ entries=entries,
+ artifact_paths=artifact_paths,
+ rewritten_paths=frozenset(rewritten_paths),
+ )
+
+
+async def build_rnfv_snapshot(
+ *,
+ viking_fs: Any,
+ vikingdb: Any,
+ store: Any,
+ artifact_ref: Any,
+ target_uri: str,
+ ctx: Any,
+ doc_rel: str = "",
+ request_intent: RequestIntent | None = None,
+ root_is_file: bool = False,
+ target_preexisting: bool = True,
+ artifact_inventory: ArtifactInventory | None = None,
+) -> RNFVSnapshot:
+ """Read the complete R/N/F/V inputs without deriving an executable plan."""
+ request = request_intent or RequestIntent(
+ target_uri=target_uri, processing_mode="semantic_and_vectors"
+ )
+
+ async def read_artifact() -> ArtifactInventory:
+ return artifact_inventory or await prepare_artifact_inventory(
+ store, artifact_ref, doc_rel=doc_rel, root_is_file=root_is_file
+ )
+
+ async def read_formal() -> tuple[Dict[str, FormalEntry], bool]:
+ if not target_preexisting:
+ return {}, True
+ return await read_target_file_snapshot(
+ viking_fs, target_uri, ctx=ctx, root_is_file=root_is_file
+ )
+
+ projection = request.required_vector_fields()
+ if request.vectorize and (root_is_file or request.processing_mode == "vectors_only"):
+ projection = projection | {"abstract"}
+
+ tasks = [
+ asyncio.create_task(read_artifact()),
+ asyncio.create_task(read_formal()),
+ ]
+ if request.vectorize:
+ tasks.append(
+ asyncio.create_task(
+ _read_incremental_vector_inventory(
+ vikingdb, target_uri=target_uri, ctx=ctx, projection=projection
+ )
+ )
+ )
+ try:
+ results = await asyncio.gather(*tasks)
+ except BaseException:
+ for task in tasks:
+ if not task.done():
+ task.cancel()
+ await asyncio.gather(*tasks, return_exceptions=True)
+ raise
+ artifact, formal = results[:2]
+ inventory = results[2] if request.vectorize else {}
+ target_files, files_complete = formal
+ base = target_uri.rstrip("/")
+ prefix = base + "/"
+ vector_records: Dict[str, VectorRecordSnapshot] = {}
+ for record_id, record in inventory.items():
+ level = int(record.get("level", -1))
+ uri = str(record.get("uri") or "")
+ rel = "" if uri == base else uri[len(prefix) :] if uri.startswith(prefix) else None
+ if rel is None:
+ raise RuntimeError(f"Vector inventory returned an out-of-scope URI: {uri}")
+ fields = {
+ field: record[field] for field in projection - {"id", "uri", "level"} if field in record
+ }
+ vector_records[record_id] = VectorRecordSnapshot(
+ record_id=record_id,
+ uri=uri,
+ relative_path=rel,
+ level=level,
+ fields=fields,
+ )
+ return RNFVSnapshot(
+ request=request,
+ new=NewArtifactSnapshot(entries=artifact.entries),
+ formal=FormalTreeSnapshot(entries=target_files, complete=files_complete),
+ vectors=VectorIndexSnapshot(
+ records_by_id=vector_records,
+ projected_fields=projection,
+ ),
+ )
+
+
+async def _read_incremental_vector_inventory(
+ vikingdb: Any,
+ *,
+ target_uri: str,
+ ctx: Any,
+ projection: frozenset[str],
+) -> Dict[str, Dict[str, Any]]:
+ """Read the lightweight V snapshot used to build RNFV."""
+ return await vikingdb.get_incremental_inventory_under_uri(
+ target_uri, ctx=ctx, output_fields=sorted(projection)
+ )
+
+
+__all__ = [
+ "ArtifactInventory",
+ "ContentState",
+ "IndexState",
+ "ResourceDiffEntry",
+ "ResourceDiffResult",
+ "build_rnfv_snapshot",
+ "count_tree_entry_kinds",
+ "prepare_artifact_inventory",
+ "read_target_file_snapshot",
+]
diff --git a/openviking/storage/resource_rnfv.py b/openviking/storage/resource_rnfv.py
new file mode 100644
index 000000000..9077eff66
--- /dev/null
+++ b/openviking/storage/resource_rnfv.py
@@ -0,0 +1,240 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Typed R/N/F/V inputs for resource update planning.
+
+R is normalized request intent, N is the parser artifact snapshot, F is the
+formal resource tree, and V is the vector-index snapshot. A validated snapshot
+contains every fact needed by the update planner; the planner must not read the
+request or either storage backend again.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Iterable, Mapping
+
+from openviking.utils.ingest_options import IngestOptions
+
+BASE_VECTOR_PROJECTION = frozenset({"id", "uri", "level", "md5"})
+NON_PORTABLE_VECTOR_RECORD_FIELDS = frozenset(
+ {
+ # Identity and tenancy are derived from the current URI/request.
+ "id",
+ "uri",
+ "level",
+ "account_id",
+ "owner_user_id",
+ # Model/content outputs belong to the current execution.
+ "vector",
+ "sparse_vector",
+ "content",
+ "updated_at",
+ "md5",
+ "abstract",
+ "type",
+ "name",
+ # ACL is re-materialized from the current resource hierarchy.
+ "acl_mode",
+ "acl_direct_grants",
+ "acl_inherited_grants",
+ }
+)
+_SCALAR_MODES = frozenset({"replace", "append"})
+_SCALAR_FIELDS = frozenset({"search_tags"})
+
+# Parser and formal-tree entries use the same compact shape. They are defined
+# beside RNFV rather than a legacy planner so every update path shares them.
+CONTROL_BASENAMES = frozenset(
+ {".abstract.md", ".overview.md", ".image_mappings.json", ".artifact_manifest.json"}
+)
+
+
+@dataclass(frozen=True)
+class NewEntry:
+ """One parser-artifact path: final file MD5 or a topology-only directory."""
+
+ md5: str = ""
+ is_dir: bool = False
+
+
+@dataclass(frozen=True)
+class FormalEntry:
+ """One formal-tree path; F records topology, not historical file bytes."""
+
+ is_dir: bool = False
+
+
+@dataclass(frozen=True)
+class ScalarIntent:
+ """One normalized request-side scalar mutation."""
+
+ field: str
+ mode: str
+ value: Any
+ target_levels: frozenset[int] = frozenset({0, 1, 2})
+
+ def __post_init__(self) -> None:
+ if self.field not in _SCALAR_FIELDS:
+ raise ValueError(f"unsupported resource scalar intent: {self.field}")
+ if self.mode not in _SCALAR_MODES:
+ raise ValueError(f"unsupported scalar update mode: {self.mode}")
+ if not self.target_levels or not self.target_levels <= {0, 1, 2}:
+ raise ValueError(f"invalid scalar target levels: {sorted(self.target_levels)}")
+
+ def required_existing_fields(self) -> frozenset[str]:
+ # Reading the old value is useful for both modes: append needs it to
+ # calculate the result, while replace needs it to detect an effective
+ # no-op instead of issuing a redundant update.
+ return frozenset({self.field})
+
+
+@dataclass(frozen=True)
+class RequestIntent:
+ """Normalized request intent (R), independent of HTTP/queue payload shape."""
+
+ target_uri: str
+ processing_mode: str
+ vectorize: bool = True
+ scalar_intents: tuple[ScalarIntent, ...] = ()
+
+ @classmethod
+ def from_ingest_options(
+ cls,
+ *,
+ target_uri: str,
+ processing_mode: Any,
+ ingest_options: IngestOptions | Mapping[str, Any] | None,
+ vectorize: bool = True,
+ ) -> "RequestIntent":
+ options = IngestOptions.from_value(ingest_options)
+ scalar_intents: tuple[ScalarIntent, ...] = ()
+ if options.search_tags is not None:
+ mode = getattr(processing_mode, "value", processing_mode)
+ scalar_intents = (
+ ScalarIntent(
+ field="search_tags",
+ mode=options.search_tag_mode,
+ value=tuple(options.search_tags),
+ target_levels=(
+ frozenset({2}) if str(mode) == "vectors_only" else frozenset({0, 1, 2})
+ ),
+ ),
+ )
+ else:
+ mode = getattr(processing_mode, "value", processing_mode)
+ return cls(
+ target_uri=target_uri.rstrip("/"),
+ processing_mode=str(mode),
+ vectorize=bool(vectorize),
+ scalar_intents=scalar_intents,
+ )
+
+ def required_vector_fields(self) -> frozenset[str]:
+ if not self.vectorize:
+ return frozenset()
+ fields = set(BASE_VECTOR_PROJECTION)
+ for intent in self.scalar_intents:
+ fields.update(intent.required_existing_fields())
+ return frozenset(fields)
+
+
+@dataclass(frozen=True)
+class NewArtifactSnapshot:
+ entries: Mapping[str, Any]
+ complete: bool = True
+
+
+@dataclass(frozen=True)
+class FormalTreeSnapshot:
+ entries: Mapping[str, Any]
+ complete: bool = True
+
+
+@dataclass(frozen=True)
+class VectorRecordSnapshot:
+ """One V inventory record with actual backend identity and hydrated scalars."""
+
+ record_id: str
+ uri: str
+ relative_path: str
+ level: int
+ fields: Mapping[str, Any] = field(default_factory=dict)
+
+ def __post_init__(self) -> None:
+ if not self.record_id:
+ raise ValueError("vector snapshot record_id must not be empty")
+ if self.level not in {0, 1, 2}:
+ raise ValueError(f"invalid vector snapshot level: {self.level}")
+
+
+def canonical_vector_records_by_level(
+ records: Iterable[VectorRecordSnapshot],
+) -> tuple[dict[int, VectorRecordSnapshot], tuple[VectorRecordSnapshot, ...]]:
+ """Choose the lowest record ID per level and return the duplicates."""
+ canonical: dict[int, VectorRecordSnapshot] = {}
+ duplicates: list[VectorRecordSnapshot] = []
+ for record in records:
+ current = canonical.get(record.level)
+ if current is None:
+ canonical[record.level] = record
+ elif record.record_id < current.record_id:
+ duplicates.append(current)
+ canonical[record.level] = record
+ else:
+ duplicates.append(record)
+ return canonical, tuple(duplicates)
+
+
+@dataclass(frozen=True)
+class VectorIndexSnapshot:
+ """Complete target-prefix V inventory and the projection used to read it."""
+
+ records_by_id: Mapping[str, VectorRecordSnapshot]
+ projected_fields: frozenset[str]
+ complete: bool = True
+
+
+@dataclass(frozen=True)
+class RNFVSnapshot:
+ """Immutable R/N/F/V facts consumed by diff resolution and plan compilation.
+
+ R carries only explicit request intent, N/F decide content topology, and V
+ supplies indexed levels, record identities, and selectively hydrated scalars.
+ No later planner step may silently reread those sources.
+ """
+
+ request: RequestIntent
+ new: NewArtifactSnapshot
+ formal: FormalTreeSnapshot
+ vectors: VectorIndexSnapshot
+
+ def validate_for_planning(self) -> None:
+ if not self.request.vectorize:
+ return
+ required = self.request.required_vector_fields()
+ missing = required - self.vectors.projected_fields
+ if missing:
+ raise ValueError(
+ "RNFV vector projection misses required fields: " + ", ".join(sorted(missing))
+ )
+ root = self.request.target_uri.rstrip("/")
+ for record in self.vectors.records_by_id.values():
+ if record.uri != root and not record.uri.startswith(root + "/"):
+ raise ValueError(f"RNFV vector record is outside target: {record.uri}")
+
+
+__all__ = [
+ "BASE_VECTOR_PROJECTION",
+ "CONTROL_BASENAMES",
+ "FormalTreeSnapshot",
+ "NewArtifactSnapshot",
+ "NewEntry",
+ "NON_PORTABLE_VECTOR_RECORD_FIELDS",
+ "RequestIntent",
+ "RNFVSnapshot",
+ "ScalarIntent",
+ "FormalEntry",
+ "VectorIndexSnapshot",
+ "VectorRecordSnapshot",
+ "canonical_vector_records_by_level",
+]
diff --git a/openviking/storage/resource_target.py b/openviking/storage/resource_target.py
new file mode 100644
index 000000000..ceee8b7fd
--- /dev/null
+++ b/openviking/storage/resource_target.py
@@ -0,0 +1,69 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Concrete AGFS write side for planned content-tree actions.
+
+This is the real AGFS implementation: it joins artifact-relative paths onto the
+resource root and writes the already-normalized artifact bytes unchanged so the
+stored bytes and their manifest md5 stay identical. Content mutation stays behind
+this adapter so the planner remains independent from VikingFS URI and lease details.
+
+An initial import is simply "the plan is all added", so the same target serves
+both first-import (full upload) and incremental (diff subset) via one channel.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Optional
+
+from openviking.utils.path_safety import safe_join_viking_uri
+
+
+class AgfsResourceTarget:
+ """Write/delete side of a diff apply against an AGFS resource tree."""
+
+ def __init__(
+ self,
+ *,
+ viking_fs: Any,
+ root_uri: str,
+ ctx: Any,
+ lease_ref: Optional[dict] = None,
+ ) -> None:
+ self._viking_fs = viking_fs
+ self._root_uri = root_uri.rstrip("/")
+ self._ctx = ctx
+ self._lease_ref = lease_ref
+
+ def _resolve(self, rel_path: str) -> str:
+ # safe_join_viking_uri rejects absolute paths, drive prefixes and ..
+ # traversal, keeping every write inside the resource root.
+ return self._root_uri if not rel_path else safe_join_viking_uri(self._root_uri, rel_path)
+
+ async def write_file(self, rel_path: str, data: bytes) -> bytes:
+ """Write the artifact bytes unchanged; normalization happened at creation."""
+ uri = self._resolve(rel_path)
+ await self._viking_fs.write_file_bytes(uri, data, ctx=self._ctx, lease_ref=self._lease_ref)
+ return data
+
+ async def mkdir(self, rel_path: str) -> None:
+ await self._viking_fs.mkdir(
+ self._resolve(rel_path),
+ exist_ok=True,
+ ctx=self._ctx,
+ lease_ref=self._lease_ref,
+ )
+
+ async def read_file(self, rel_path: str) -> bytes:
+ return await self._viking_fs.read_file_bytes(self._resolve(rel_path), ctx=self._ctx)
+
+ async def delete_path(self, rel_path: str, *, is_dir: bool) -> None:
+ """Delete one planned path under the already-held resource tree lease."""
+ await self._viking_fs.remove_files(
+ self._resolve(rel_path),
+ recursive=is_dir,
+ ctx=self._ctx,
+ lease_ref=self._lease_ref,
+ )
+
+
+__all__ = ["AgfsResourceTarget"]
diff --git a/openviking/storage/vectordb_adapters/base.py b/openviking/storage/vectordb_adapters/base.py
index b0a42dd56..0701ef5b3 100644
--- a/openviking/storage/vectordb_adapters/base.py
+++ b/openviking/storage/vectordb_adapters/base.py
@@ -635,6 +635,19 @@ class CollectionAdapter(ABC):
raise RuntimeError("Vector backend returned an invalid count result")
+ def strict_count(self, filter: Optional[Dict[str, Any] | FilterExpr] = None) -> int:
+ """Count records and reject responses without an explicit total."""
+ coll = self.get_collection()
+ result = coll.aggregate_data(
+ index_name=self._index_name,
+ op="count",
+ filters=self._compile_filter(filter),
+ )
+ parsed_total = self._extract_count_total(result.agg)
+ if parsed_total is None:
+ raise RuntimeError("Vector backend returned an invalid count response")
+ return parsed_total
+
def search_by_keywords(
self,
keywords: Optional[list[str]] = None,
diff --git a/openviking/storage/viking_fs/_base.py b/openviking/storage/viking_fs/_base.py
index 98a1b6b0e..9def2bf1b 100644
--- a/openviking/storage/viking_fs/_base.py
+++ b/openviking/storage/viking_fs/_base.py
@@ -21,7 +21,7 @@ logger = get_logger(__name__)
# Sentinel node_limit for internal callers that MUST enumerate an entire
# directory. ``ls()`` defaults to ``node_limit=1000`` to protect agent-facing
# context from being flooded, but internal system operations (parse merge,
-# temp->final sync, summary DAG, vectorization) must see every child or they
+# temp->final sync, semantic-tree execution, vectorization) must see every child or they
# silently drop entries beyond the cap — e.g. a >1000-doc directory ingest only
# materializes its first 1000 subdirectories. Pass this explicitly at those
# call sites.
diff --git a/openviking/storage/viking_vector_index_backend.py b/openviking/storage/viking_vector_index_backend.py
index a74b89678..ce1dc4c8d 100644
--- a/openviking/storage/viking_vector_index_backend.py
+++ b/openviking/storage/viking_vector_index_backend.py
@@ -20,6 +20,7 @@ from openviking.server.identity import RequestContext, Role
from openviking.service.task_tracker_concurrency import run_to_completion
from openviking.storage.acl import (
ACL_CONTEXT_FIELDS,
+ ACL_GRANT_FIELDS,
ACL_MODE_FIELD,
AclAction,
AclManager,
@@ -80,6 +81,40 @@ FETCH_BY_URI_OUTPUT_FIELDS = [
"owner_user_id",
]
+# Fields an incremental diff needs from existing target records. md5 is listed
+# explicitly because the default lookup projection omits it; a missing value
+# means "unknown" and the caller falls back to reading file bytes.
+INCREMENTAL_DIFF_OUTPUT_FIELDS = [
+ "id",
+ "uri",
+ "level",
+ "abstract",
+ "md5",
+]
+
+INCREMENTAL_INVENTORY_OUTPUT_FIELDS = ["id", "uri", "level", "md5"]
+
+INCREMENTAL_HYDRATION_OUTPUT_FIELDS = [
+ "id",
+ "uri",
+ "type",
+ "context_type",
+ "created_at",
+ "updated_at",
+ "active_count",
+ "level",
+ "name",
+ "description",
+ "tags",
+ "search_tags",
+ "abstract",
+ "md5",
+ "account_id",
+ "owner_user_id",
+ ACL_MODE_FIELD,
+ *ACL_GRANT_FIELDS,
+]
+
VIKINGDB_CONTENT_MAX_SIZE = 1024 * 1024
@@ -292,6 +327,19 @@ class _SingleAccountBackend:
"""Prepare a batch in one worker-thread handoff."""
return [self._prepare_upsert_payload(data) for data in data_list]
+ def _prepare_update_payload(self, data: Dict[str, Any]) -> Dict[str, Any]:
+ """Prepare a strict partial update without inventing omitted fields."""
+ payload = self._filter_known_fields(
+ {key: value for key, value in data.items() if value is not None}
+ )
+ if self._adapter.USE_CONTENT_FIELD:
+ content = payload.get("content")
+ if isinstance(content, (str, bytes)):
+ payload["content"] = content[:VIKINGDB_CONTENT_MAX_SIZE]
+ else:
+ payload.pop("content", None)
+ return payload
+
def _bind_upsert_payload(self, data: Dict[str, Any]) -> Dict[str, Any]:
"""Copy a record, enforce its bound account, and apply write defaults."""
payload = dict(data)
@@ -506,7 +554,7 @@ class _SingleAccountBackend:
allowed = sorted(VikingVectorIndexBackend.ALLOWED_CONTEXT_TYPES)
raise ValueError(f"Invalid context_type: {context_type}. Must be one of {allowed}")
- payload = await self._async_adapter.run(self._prepare_upsert_payload, payload)
+ payload = await self._async_adapter.run(self._prepare_update_payload, payload)
ids = await self._async_adapter.call("update_data", [payload])
normalized_ids = [str(item) for item in (ids or []) if item is not None]
return UpdateResult(
@@ -609,7 +657,7 @@ class _SingleAccountBackend:
async def strict_count(self, filter: Optional[Dict[str, Any] | FilterExpr] = None) -> int:
"""Count transaction records without converting backend errors to zero."""
return int(
- await self._async_adapter.call("count", filter=self._with_account_filter(filter)) or 0
+ await self._async_adapter.call("strict_count", filter=self._with_account_filter(filter))
)
async def delete(self, ids: List[str]) -> int:
@@ -1188,6 +1236,10 @@ class VikingVectorIndexBackend:
backend = self._get_backend_for_context(ctx)
return await backend.delete(ids)
+ async def strict_delete(self, ids: List[str], *, ctx: RequestContext) -> int:
+ """Delete exact record IDs without converting backend failures to success."""
+ return await self._get_backend_for_context(ctx).strict_delete(ids)
+
async def exists(self, id: str, *, ctx: RequestContext) -> bool:
backend = self._get_backend_for_context(ctx)
return await backend.exists(id)
@@ -1438,6 +1490,60 @@ class VikingVectorIndexBackend:
backend = self._get_backend_for_context(ctx)
return await backend.strict_count(filter=filter)
+ async def _strict_scan(
+ self,
+ ctx: RequestContext,
+ scope: FilterExpr,
+ *,
+ output_fields: List[str],
+ batch_size: int,
+ what: str,
+ ) -> AsyncIterator[Dict[str, Any]]:
+ """Yield every record under ``scope`` with strict count/cursor guarantees.
+
+ Consolidates the paginated scan skeleton shared by the incremental
+ readers: it pins the expected total up front, then walks cursor pages,
+ raising if a page ends early, overshoots the count, or the cursor repeats
+ or dies before the total is reached — so a truncated scan can never look
+ like "records absent". Per-record parsing/validation stays with the
+ caller; this only guarantees the traversal is complete.
+ """
+ if batch_size <= 0:
+ raise ValueError("batch_size must be positive")
+ expected_count = await self._strict_transfer_count(ctx, scope)
+ cursor: Optional[str] = None
+ scanned_count = 0
+ seen_cursors: set[str] = set()
+ while True:
+ page, next_cursor = await self._strict_transfer_page(
+ ctx,
+ scope,
+ limit=batch_size,
+ cursor=cursor,
+ output_fields=output_fields,
+ )
+ if not page and scanned_count < expected_count:
+ raise RuntimeError(
+ f"{what} ended after {scanned_count} of {expected_count} records"
+ )
+ scanned_count += len(page)
+ if scanned_count > expected_count:
+ raise RuntimeError(
+ f"{what} returned {scanned_count} records but count was {expected_count}"
+ )
+ for record in page:
+ yield record
+ if next_cursor is None:
+ if scanned_count == expected_count:
+ return
+ raise RuntimeError(
+ f"{what} cursor ended after {scanned_count} of {expected_count} records"
+ )
+ if next_cursor in seen_cursors:
+ raise RuntimeError(f"{what} cursor repeated: {next_cursor}")
+ seen_cursors.add(next_cursor)
+ cursor = next_cursor
+
async def _strict_transfer_get(
self, ctx: RequestContext, ids: List[str]
) -> List[Dict[str, Any]]:
@@ -1712,6 +1818,273 @@ class VikingVectorIndexBackend:
if uri in requested_by_canonical
}
+ async def get_l2_diff_records_by_uris(
+ self,
+ uris: List[str],
+ *,
+ ctx: RequestContext,
+ ) -> Dict[str, Dict[str, Any]]:
+ """Load existing L2 diff metadata (md5 + abstract) for a bounded URI set.
+
+ Returns a map keyed by the caller's original URI to
+ ``{"md5": str, "abstract": str}``. ``md5`` is empty when the record
+ predates the field, in which case incremental diff must fall back to
+ comparing file bytes rather than assuming equality. Uses full strict
+ pagination so a truncated page never masquerades as "record absent".
+ """
+ requested_by_canonical: Dict[str, str] = {}
+ for uri in uris:
+ requested_by_canonical.setdefault(resolve_uri(uri).uri, uri)
+ canonical_uris = list(requested_by_canonical)
+ if not canonical_uris:
+ return {}
+
+ records_by_uri: Dict[str, Dict[str, Any]] = {}
+ chunk_size = 100
+ for start in range(0, len(canonical_uris), chunk_size):
+ chunk = canonical_uris[start : start + chunk_size]
+ cursor: Optional[str] = None
+ while True:
+ records, cursor = await self._strict_transfer_page(
+ ctx,
+ And([In("uri", chunk), Eq("level", 2)]),
+ limit=100,
+ cursor=cursor,
+ output_fields=["uri", "md5", "abstract"],
+ )
+ for record in records:
+ uri = str(record.get("uri") or "")
+ if uri and uri not in records_by_uri:
+ records_by_uri[uri] = {
+ "md5": str(record.get("md5") or ""),
+ "abstract": str(record.get("abstract") or ""),
+ }
+ if cursor is None:
+ break
+ return {
+ requested_by_canonical[uri]: record
+ for uri, record in records_by_uri.items()
+ if uri in requested_by_canonical
+ }
+
+ async def get_l2_diff_records_under_uri(
+ self,
+ uri: str,
+ *,
+ ctx: RequestContext,
+ batch_size: int = 100,
+ ) -> Dict[str, Dict[str, Any]]:
+ """Strictly load every L2 diff record below one target directory."""
+ canonical_uri = resolve_uri(uri).uri.rstrip("/")
+ scope = And(
+ [
+ Eq("account_id", ctx.account_id),
+ PathScope("uri", canonical_uri, depth=-1),
+ Eq("level", 2),
+ ]
+ )
+ records: Dict[str, Dict[str, Any]] = {}
+ what = f"Vector scan under {canonical_uri}"
+ async for record in self._strict_scan(
+ ctx,
+ scope,
+ output_fields=INCREMENTAL_DIFF_OUTPUT_FIELDS,
+ batch_size=batch_size,
+ what=what,
+ ):
+ record_uri = str(record.get("uri") or "")
+ if not record_uri or not uri_in_transfer_scope(
+ record_uri, canonical_uri, recursive=True
+ ):
+ raise RuntimeError(f"{what} returned invalid L2 URI: {record_uri or '<missing>'}")
+ if record_uri in records:
+ raise RuntimeError(f"{what} returned duplicate L2 URI: {record_uri}")
+ records[record_uri] = {
+ "md5": str(record.get("md5") or ""),
+ "abstract": str(record.get("abstract") or ""),
+ }
+ return records
+
+ async def get_incremental_inventory_under_uri(
+ self,
+ uri: str,
+ *,
+ ctx: RequestContext,
+ batch_size: int = 100,
+ output_fields: Optional[List[str]] = None,
+ ) -> Dict[str, Dict[str, Any]]:
+ """Strictly load lightweight L0/L1/L2 metadata below a resource root."""
+ projection = list(
+ dict.fromkeys(
+ [
+ *INCREMENTAL_INVENTORY_OUTPUT_FIELDS,
+ *(output_fields or []),
+ ]
+ )
+ )
+ canonical_uri = resolve_uri(uri).uri.rstrip("/")
+ scope = And(
+ [
+ Eq("account_id", ctx.account_id),
+ PathScope("uri", canonical_uri, depth=-1),
+ In("level", [0, 1, 2]),
+ ]
+ )
+ what = f"Incremental inventory under {canonical_uri}"
+ retry_delays = (0.0, 0.25, 1.0)
+ last_error: Exception | None = None
+ for attempt, delay in enumerate(retry_delays, start=1):
+ if delay:
+ await asyncio.sleep(delay)
+ records: Dict[str, Dict[str, Any]] = {}
+ try:
+ async for record in self._strict_scan(
+ ctx,
+ scope,
+ output_fields=projection,
+ batch_size=batch_size,
+ what=what,
+ ):
+ record_id = str(record.get("id") or "")
+ record_uri = str(record.get("uri") or "")
+ try:
+ level = int(record.get("level"))
+ except (TypeError, ValueError):
+ level = -1
+ if (
+ not record_id
+ or level not in {0, 1, 2}
+ or not uri_in_transfer_scope(record_uri, canonical_uri, recursive=True)
+ ):
+ raise RuntimeError(
+ f"{what} returned an invalid record: id={record_id or '<missing>'} "
+ f"uri={record_uri or '<missing>'} level={level}"
+ )
+ if record_id in records:
+ raise RuntimeError(f"{what} returned duplicate record id: {record_id}")
+ records[record_id] = {
+ field: record[field] for field in projection if field in record
+ }
+ records[record_id].update(
+ {
+ "id": record_id,
+ "uri": record_uri,
+ "level": level,
+ "md5": str(record.get("md5") or ""),
+ }
+ )
+ return records
+ except RuntimeError as exc:
+ last_error = exc
+ if attempt == len(retry_delays) or not any(
+ marker in str(exc)
+ for marker in (
+ "cursor ended after",
+ "ended after",
+ "records but count was",
+ )
+ ):
+ raise
+ logger.warning(
+ "%s was incomplete; retrying full inventory attempt=%d/%d error=%s",
+ what,
+ attempt,
+ len(retry_delays),
+ exc,
+ )
+ assert last_error is not None
+ raise last_error
+
+ async def hydrate_incremental_records(
+ self,
+ expected: Mapping[str, Mapping[str, Any]],
+ *,
+ ctx: RequestContext,
+ batch_size: int = 100,
+ output_fields: Optional[Container[str]] = None,
+ ) -> Dict[str, Dict[str, Any]]:
+ """Load requested non-vector fields, using strict DSL before ID fallback."""
+ if batch_size <= 0:
+ raise ValueError("batch_size must be positive")
+ requested_ids = list(expected)
+ if output_fields is None:
+ selected_fields = list(INCREMENTAL_HYDRATION_OUTPUT_FIELDS)
+ try:
+ collection_meta = await self.get_collection_meta(ctx=ctx)
+ schema_fields = [
+ str(item.get("FieldName") or "")
+ for item in (collection_meta or {}).get("Fields", [])
+ ]
+ dynamic_fields = [
+ field
+ for field in schema_fields
+ if field and field not in {"vector", "sparse_vector", "content"}
+ ]
+ if dynamic_fields:
+ selected_fields = list(dict.fromkeys(dynamic_fields))
+ except Exception:
+ # Backends without collection metadata retain the known portable
+ # projection; correctness remains fail-closed at identity checks.
+ pass
+ else:
+ selected_fields = list(dict.fromkeys(["id", "uri", "level", *sorted(output_fields)]))
+ hydrated: Dict[str, Dict[str, Any]] = {}
+
+ def _accept(record: Mapping[str, Any]) -> None:
+ record_id = str(record.get("id") or "")
+ wanted = expected.get(record_id)
+ if wanted is None:
+ raise RuntimeError(
+ f"Incremental hydration returned unexpected record id: {record_id or '<missing>'}"
+ )
+ record_uri = str(record.get("uri") or "")
+ try:
+ level = int(record.get("level"))
+ except (TypeError, ValueError):
+ level = -1
+ if (
+ record_uri != str(wanted.get("uri") or "")
+ or level != int(wanted.get("level", -1))
+ or (record.get("account_id") not in {None, ctx.account_id})
+ ):
+ raise RuntimeError(
+ f"Incremental hydration identity mismatch for record: {record_id}"
+ )
+ if record_id in hydrated:
+ raise RuntimeError(
+ f"Incremental hydration returned duplicate record id: {record_id}"
+ )
+ hydrated[record_id] = {
+ field: record[field] for field in selected_fields if field in record
+ }
+
+ for start in range(0, len(requested_ids), batch_size):
+ chunk = requested_ids[start : start + batch_size]
+ cursor: Optional[str] = None
+ seen_cursors: set[str] = set()
+ while True:
+ page, next_cursor = await self._strict_transfer_page(
+ ctx,
+ In("id", chunk),
+ limit=batch_size,
+ cursor=cursor,
+ output_fields=selected_fields,
+ )
+ for record in page:
+ _accept(record)
+ if next_cursor is None:
+ break
+ if next_cursor in seen_cursors:
+ raise RuntimeError(f"Incremental hydration cursor repeated: {next_cursor}")
+ seen_cursors.add(next_cursor)
+ cursor = next_cursor
+
+ missing = [record_id for record_id in chunk if record_id not in hydrated]
+ if missing:
+ for record in await self._strict_transfer_get(ctx, missing):
+ _accept(record)
+ return hydrated
+
async def delete_account_data(self, account_id: str, *, ctx: RequestContext) -> int:
"""删除指定 account 的所有数据(仅限,root 角色操作)"""
self._check_root_role(ctx)
@@ -2276,11 +2649,7 @@ class VikingVectorIndexBackend:
targets = [target_dir for target_dir in target_directories or [] if target_dir]
tenant_filter = self._tenant_filter(ctx, acl_enabled=acl_enabled)
- if (
- tenant_filter
- and not acl_enabled
- and self._targets_within_visible_roots(ctx, targets)
- ):
+ if tenant_filter and not acl_enabled and self._targets_within_visible_roots(ctx, targets):
# The target scopes are already narrower than the tenant-visible
# roots. Keep account isolation, but avoid recursively evaluating
# the broader path scopes as an additional filter.
diff --git a/openviking/telemetry/operation.py b/openviking/telemetry/operation.py
index a8951bba4..6ff3eb626 100644
--- a/openviking/telemetry/operation.py
+++ b/openviking/telemetry/operation.py
@@ -264,11 +264,29 @@ class TelemetrySummaryBuilder:
"processed": cls._i(gauges.get("queue.semantic.processed"), 0),
"requeue_count": cls._i(gauges.get("queue.semantic.requeue_count"), 0),
"error_count": cls._i(gauges.get("queue.semantic.error_count"), 0),
+ "queue_wait": {
+ "duration_ms": cls._f(
+ gauges.get("queue.semantic.queue_wait.duration_ms"), 0.0
+ )
+ },
+ "execute": {
+ "duration_ms": cls._f(gauges.get("queue.semantic.execute.duration_ms"), 0.0)
+ },
},
"embedding": {
"processed": cls._i(gauges.get("queue.embedding.processed"), 0),
"requeue_count": cls._i(gauges.get("queue.embedding.requeue_count"), 0),
"error_count": cls._i(gauges.get("queue.embedding.error_count"), 0),
+ "queue_wait": {
+ "duration_ms": cls._f(
+ gauges.get("queue.embedding.queue_wait.duration_ms"), 0.0
+ )
+ },
+ "execute": {
+ "duration_ms": cls._f(
+ gauges.get("queue.embedding.execute.duration_ms"), 0.0
+ )
+ },
},
}
@@ -393,27 +411,68 @@ class TelemetrySummaryBuilder:
if cls._has_metric_prefix("resource", counters, gauges):
summary["resource"] = {
- "request": {
- "duration_ms": cls._f(gauges.get("resource.request.duration_ms"), 0.0),
+ "total": {
+ "duration_ms": cls._f(gauges.get("resource.total.duration_ms"), 0.0),
+ },
+ "source_execute": {
+ "duration_ms": cls._f(gauges.get("resource.source_execute.duration_ms"), 0.0),
+ },
+ "source_prepare": {
+ "duration_ms": cls._f(gauges.get("resource.source_prepare.duration_ms"), 0.0),
},
- "process": {
- "duration_ms": cls._f(gauges.get("resource.process.duration_ms"), 0.0),
- "parse": {
- "duration_ms": cls._f(gauges.get("resource.parse.duration_ms"), 0.0),
- "warnings_count": cls._i(gauges.get("resource.parse.warnings_count"), 0),
+ "parse_artifact": {
+ "duration_ms": cls._f(gauges.get("resource.parse_artifact.duration_ms"), 0.0),
+ },
+ "target_resolve": {
+ "duration_ms": cls._f(gauges.get("resource.target_resolve.duration_ms"), 0.0),
+ },
+ "update_plan": {
+ "duration_ms": cls._f(gauges.get("resource.update_plan.duration_ms"), 0.0),
+ "artifact_inventory": {
+ "duration_ms": cls._f(
+ gauges.get("resource.update_plan.artifact_inventory.duration_ms"), 0.0
+ ),
},
- "finalize": {
- "duration_ms": cls._f(gauges.get("resource.finalize.duration_ms"), 0.0),
+ "rnfv_snapshot": {
+ "duration_ms": cls._f(
+ gauges.get("resource.update_plan.rnfv_snapshot.duration_ms"), 0.0
+ ),
},
- "summarize": {
- "duration_ms": cls._f(gauges.get("resource.summarize.duration_ms"), 0.0),
+ "diff_and_compile": {
+ "duration_ms": cls._f(
+ gauges.get("resource.update_plan.diff_and_compile.duration_ms"), 0.0
+ ),
},
},
- "wait": {
- "duration_ms": cls._f(gauges.get("resource.wait.duration_ms"), 0.0),
+ "content_commit": {
+ "duration_ms": cls._f(gauges.get("resource.content_commit.duration_ms"), 0.0),
+ },
+ "derived_enqueue": {
+ "duration_ms": cls._f(gauges.get("resource.derived_enqueue.duration_ms"), 0.0),
},
- "watch": {
- "duration_ms": cls._f(gauges.get("resource.watch.duration_ms"), 0.0),
+ "semantic": {
+ "queue_wait": {
+ "duration_ms": cls._f(
+ gauges.get("queue.semantic.queue_wait.duration_ms"), 0.0
+ ),
+ },
+ "execute": {
+ "duration_ms": cls._f(
+ gauges.get("queue.semantic.execute.duration_ms"), 0.0
+ ),
+ },
+ },
+ "embedding": {
+ "queue_wait": {
+ "duration_ms": cls._f(
+ gauges.get("queue.embedding.queue_wait.duration_ms"), 0.0
+ ),
+ },
+ "execute": {
+ "duration_ms": cls._f(
+ gauges.get("queue.embedding.execute.duration_ms"), 0.0
+ ),
+ },
},
"flags": {
public_key: cls._bool(gauges.get(metric_key), False)
@@ -498,6 +557,10 @@ class OperationTelemetry:
def set_value(self, key: str, value: Any) -> None:
self.set(key, value)
+ def elapsed_ms(self) -> float:
+ """Return the current operation wall-clock duration without finishing it."""
+ return max((time.perf_counter() - self._start_time) * 1000.0, 0.0)
+
@staticmethod
def _metric_bucket(value: Any, allowed: set[str]) -> str:
normalized = str(value or "").strip().lower()
diff --git a/openviking/telemetry/request_wait_tracker.py b/openviking/telemetry/request_wait_tracker.py
index 1a27bd35e..85223d798 100644
--- a/openviking/telemetry/request_wait_tracker.py
+++ b/openviking/telemetry/request_wait_tracker.py
@@ -26,6 +26,10 @@ class _RequestWaitState:
embedding_requeue_count: int = 0
embedding_error_count: int = 0
embedding_errors: List[str] = field(default_factory=list)
+ semantic_queue_wait_ms: float = 0.0
+ semantic_execute_ms: float = 0.0
+ embedding_queue_wait_ms: float = 0.0
+ embedding_execute_ms: float = 0.0
created_at: float = field(default_factory=time.time)
retained_roots: Set[str] = field(default_factory=set)
cleanup_requested: bool = False
@@ -106,6 +110,48 @@ class RequestWaitTracker:
return
state.embedding_requeue_count += max(delta, 0)
+ def record_semantic_timing(
+ self, telemetry_id: str, *, queue_wait_ms: float = 0.0, execute_ms: float = 0.0
+ ) -> None:
+ self._record_timing(
+ telemetry_id,
+ queue_wait_ms=queue_wait_ms,
+ execute_ms=execute_ms,
+ queue_wait_attr="semantic_queue_wait_ms",
+ execute_attr="semantic_execute_ms",
+ )
+
+ def record_embedding_timing(
+ self, telemetry_id: str, *, queue_wait_ms: float = 0.0, execute_ms: float = 0.0
+ ) -> None:
+ self._record_timing(
+ telemetry_id,
+ queue_wait_ms=queue_wait_ms,
+ execute_ms=execute_ms,
+ queue_wait_attr="embedding_queue_wait_ms",
+ execute_attr="embedding_execute_ms",
+ )
+
+ def _record_timing(
+ self,
+ telemetry_id: str,
+ *,
+ queue_wait_ms: float,
+ execute_ms: float,
+ queue_wait_attr: str,
+ execute_attr: str,
+ ) -> None:
+ if not telemetry_id:
+ return
+ with self._lock:
+ state = self._states.get(telemetry_id)
+ if state is None:
+ return
+ setattr(
+ state, queue_wait_attr, getattr(state, queue_wait_attr) + max(queue_wait_ms, 0.0)
+ )
+ setattr(state, execute_attr, getattr(state, execute_attr) + max(execute_ms, 0.0))
+
def get_embedding_context_count(self, telemetry_id: str) -> int:
"""Return contexts successfully indexed for one request."""
if not telemetry_id:
@@ -114,6 +160,30 @@ class RequestWaitTracker:
state = self._states.get(telemetry_id)
return state.embedding_context_count if state is not None else 0
+ def get_queue_timing(self, telemetry_id: str) -> Dict[str, Dict[str, float]]:
+ if not telemetry_id:
+ return {
+ "semantic": {"queue_wait_ms": 0.0, "execute_ms": 0.0},
+ "embedding": {"queue_wait_ms": 0.0, "execute_ms": 0.0},
+ }
+ with self._lock:
+ state = self._states.get(telemetry_id)
+ if state is None:
+ return {
+ "semantic": {"queue_wait_ms": 0.0, "execute_ms": 0.0},
+ "embedding": {"queue_wait_ms": 0.0, "execute_ms": 0.0},
+ }
+ return {
+ "semantic": {
+ "queue_wait_ms": state.semantic_queue_wait_ms,
+ "execute_ms": state.semantic_execute_ms,
+ },
+ "embedding": {
+ "queue_wait_ms": state.embedding_queue_wait_ms,
+ "execute_ms": state.embedding_execute_ms,
+ },
+ }
+
def mark_semantic_done(
self,
telemetry_id: str,
diff --git a/openviking/telemetry/resource_summary.py b/openviking/telemetry/resource_summary.py
index 90d19834e..3ba2a44af 100644
--- a/openviking/telemetry/resource_summary.py
+++ b/openviking/telemetry/resource_summary.py
@@ -7,6 +7,7 @@ from __future__ import annotations
from typing import Any, Dict
from .operation import OperationTelemetry
+from .request_wait_tracker import get_request_wait_tracker
def _consume_semantic_request_stats(telemetry_id: str):
@@ -27,11 +28,11 @@ def _consume_embedding_request_stats(telemetry_id: str):
return None
-def _consume_semantic_dag_stats(telemetry_id: str, root_uri: str | None):
+def _consume_semantic_tree_stats(telemetry_id: str, root_uri: str | None):
try:
from openviking.storage.queuefs.semantic_processor import SemanticProcessor
- return SemanticProcessor.consume_dag_stats(telemetry_id=telemetry_id, uri=root_uri)
+ return SemanticProcessor.consume_tree_stats(telemetry_id=telemetry_id, uri=root_uri)
except Exception:
return None
@@ -65,26 +66,32 @@ def record_resource_queue_metrics(
telemetry_id: str,
root_uri: str | None,
) -> None:
- """Apply queue and DAG metrics to a resource operation collector."""
+ """Apply queue and semantic-tree metrics to a resource operation collector."""
if not telemetry.enabled:
return
semantic = _queue_metrics(_consume_semantic_request_stats(telemetry_id))
embedding = _queue_metrics(_consume_embedding_request_stats(telemetry_id))
+ timing = get_request_wait_tracker().get_queue_timing(telemetry_id)
telemetry.set("queue.semantic.processed", semantic["processed"])
telemetry.set("queue.semantic.requeue_count", semantic["requeue_count"])
telemetry.set("queue.semantic.error_count", semantic["error_count"])
+ telemetry.set("queue.semantic.queue_wait.duration_ms", timing["semantic"]["queue_wait_ms"])
+ telemetry.set("queue.semantic.execute.duration_ms", timing["semantic"]["execute_ms"])
telemetry.set("queue.embedding.processed", embedding["processed"])
telemetry.set("queue.embedding.requeue_count", embedding["requeue_count"])
telemetry.set("queue.embedding.error_count", embedding["error_count"])
+ telemetry.set("queue.embedding.queue_wait.duration_ms", timing["embedding"]["queue_wait_ms"])
+ telemetry.set("queue.embedding.execute.duration_ms", timing["embedding"]["execute_ms"])
+
+ tree_stats = _consume_semantic_tree_stats(telemetry_id, root_uri)
+ if tree_stats is not None:
+ telemetry.set("semantic_nodes.total", tree_stats.total_nodes)
+ telemetry.set("semantic_nodes.done", tree_stats.done_nodes)
+ telemetry.set("semantic_nodes.pending", tree_stats.pending_nodes)
+ telemetry.set("semantic_nodes.running", tree_stats.in_progress_nodes)
- dag_stats = _consume_semantic_dag_stats(telemetry_id, root_uri)
- if dag_stats is not None:
- telemetry.set("semantic_nodes.total", dag_stats.total_nodes)
- telemetry.set("semantic_nodes.done", dag_stats.done_nodes)
- telemetry.set("semantic_nodes.pending", dag_stats.pending_nodes)
- telemetry.set("semantic_nodes.running", dag_stats.in_progress_nodes)
__all__ = [
"build_queue_status_payload",
diff --git a/openviking/utils/content_hash.py b/openviking/utils/content_hash.py
new file mode 100644
index 000000000..3660908d5
--- /dev/null
+++ b/openviking/utils/content_hash.py
@@ -0,0 +1,17 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Single source of truth for the file-content fingerprint used by incremental diff.
+
+The fingerprint is the md5 of the *final stored* file bytes for a resource URI
+(after any encoding normalization or link rewriting). Incremental diff compares
+this fingerprint on both sides to skip re-processing unchanged files. Every write
+path that maintains the fingerprint MUST use :func:`content_md5` so the scheme
+stays consistent; it is a content-equality hint, not a security digest.
+"""
+
+import hashlib
+
+
+def content_md5(data: bytes) -> str:
+ """Return the hex md5 of the final stored file bytes."""
+ return hashlib.md5(data, usedforsecurity=False).hexdigest()
diff --git a/openviking/utils/embedding_utils.py b/openviking/utils/embedding_utils.py
index 085f07a5b..318704b4c 100644
--- a/openviking/utils/embedding_utils.py
+++ b/openviking/utils/embedding_utils.py
@@ -28,8 +28,10 @@ from openviking.server.identity import RequestContext
from openviking.service.task_work_index import TaskWorkRejected
from openviking.storage.abstract_overview import body_for_preview, embedding_text_for_body
from openviking.storage.acl import CreatorAclGrant
+from openviking.storage.index_action import FieldPatch, IndexAction
from openviking.storage.queuefs import get_queue_manager
from openviking.storage.queuefs.embedding_msg_converter import EmbeddingMsgConverter
+from openviking.storage.resource_rnfv import NON_PORTABLE_VECTOR_RECORD_FIELDS
from openviking.storage.viking_fs import LS_ALL_NODES, get_viking_fs
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
from openviking.utils.embedding_input import truncate_embedding_input
@@ -62,25 +64,44 @@ def _truncate_abstract_bytes(abstract: str) -> str:
return encoded[:_ABSTRACT_MAX_BYTES].decode("utf-8", errors="ignore")
-_PORTABLE_SCALAR_FIELDS = frozenset(
- {
- "type",
- "level",
- "name",
- "description",
- "tags",
- "abstract",
- }
-)
-
-
def _apply_scalar_overrides(embedding_msg, overrides: Optional[Dict[str, Any]]) -> None:
if not embedding_msg or not overrides:
return
- for field in _PORTABLE_SCALAR_FIELDS:
- value = overrides.get(field)
- if value is not None:
- embedding_msg.context_data[field] = value
+ record_id = overrides.get("_record_id")
+ if record_id:
+ # Internal queue metadata, removed by TextEmbeddingHandler before upsert.
+ embedding_msg.context_data["_upsert_record_id"] = str(record_id)
+ for field, value in overrides.items():
+ if field.startswith("_") or field in NON_PORTABLE_VECTOR_RECORD_FIELDS or value is None:
+ continue
+ embedding_msg.context_data[field] = value
+
+
+def _apply_planned_field_patch(
+ embedding_msg,
+ field_patch: FieldPatch | None,
+) -> None:
+ """Attach a MERGE patch without overwriting the exact-get base prematurely."""
+
+ if not embedding_msg or embedding_msg.action is not IndexAction.MERGE:
+ return
+ patch_values = {
+ field: value
+ for field, value in (field_patch.values if field_patch is not None else {}).items()
+ if not field.startswith("_")
+ and field not in NON_PORTABLE_VECTOR_RECORD_FIELDS
+ and value is not None
+ }
+ for field in patch_values:
+ embedding_msg.context_data.pop(field, None)
+ embedding_msg.payload.field_patch = (
+ FieldPatch(
+ values=patch_values,
+ modes=field_patch.modes if field_patch is not None else {},
+ )
+ if patch_values
+ else None
+ )
def _apply_ingest_options(
@@ -90,10 +111,20 @@ def _apply_ingest_options(
ingest_options = IngestOptions.from_value(ingest_options)
if not embedding_msg or ingest_options.search_tags is None:
return
- embedding_msg.context_data["search_tags"] = list(ingest_options.search_tags or [])
- embedding_msg.context_data["_upsert_options"] = {
- "search_tag_mode": ingest_options.search_tag_mode
- }
+ incoming_tags = list(ingest_options.search_tags or [])
+ if ingest_options.search_tag_mode == "append" and (
+ embedding_msg.action is IndexAction.MERGE
+ or embedding_msg.context_data.get("_upsert_options", {}).get("partial_update") is False
+ ):
+ from openviking.utils.tags import merge_search_tags
+
+ incoming_tags = merge_search_tags(
+ embedding_msg.context_data.get("search_tags"), incoming_tags
+ )
+ embedding_msg.context_data["search_tags"] = incoming_tags
+ embedding_msg.context_data.setdefault("_upsert_options", {})["search_tag_mode"] = (
+ ingest_options.search_tag_mode
+ )
async def _enqueue_embedding_message(
@@ -292,16 +323,21 @@ async def _resolve_resource_content_type(
file_name: str,
viking_fs: Any,
ctx: Optional[RequestContext],
+ file_content: Optional[bytes] = None,
) -> Optional[ResourceContentType]:
content_type = get_resource_content_type(file_name)
if Path(file_name).suffix.lower() != ".ts":
return content_type
try:
- prefix = await viking_fs.read(
- file_path,
- offset=0,
- size=MPEG_TS_PROBE_BYTES,
- ctx=ctx,
+ prefix = (
+ file_content[:MPEG_TS_PROBE_BYTES]
+ if file_content is not None
+ else await viking_fs.read(
+ file_path,
+ offset=0,
+ size=MPEG_TS_PROBE_BYTES,
+ ctx=ctx,
+ )
)
except Exception:
return content_type
@@ -368,7 +404,9 @@ async def vectorize_directory_meta(
meta: Optional[Dict[str, Any]] = None,
*,
content_is_body: bool = False,
-) -> None:
+ actions: Optional[Dict[int, IndexAction | str]] = None,
+ field_patches: Optional[Dict[int, FieldPatch]] = None,
+) -> set[int]:
"""
Vectorize directory metadata (.abstract.md and .overview.md).
@@ -383,10 +421,11 @@ async def vectorize_directory_meta(
abstract = body_for_preview(abstract)
overview = body_for_preview(overview)
first_enqueue_error: Optional[Exception] = None
+ enqueued_levels: set[int] = set()
try:
if not ctx:
logger.warning("No context provided for vectorization")
- return
+ return enqueued_levels
queue_manager = get_queue_manager()
embedding_queue = queue_manager.get_queue(queue_manager.EMBEDDING)
@@ -420,10 +459,24 @@ async def vectorize_directory_meta(
context_abstract.set_vectorize(
Vectorize(text=embedding_text_for_body(ContextLevel.ABSTRACT, uri, abstract))
)
- msg_abstract = EmbeddingMsgConverter.from_context(context_abstract, creator_acl_grant)
+ msg_abstract = EmbeddingMsgConverter.from_context(
+ context_abstract,
+ creator_acl_grant,
+ action=IndexAction(
+ (actions or {}).get(
+ int(ContextLevel.ABSTRACT.value),
+ IndexAction.MERGE,
+ )
+ ),
+ )
+ level_overrides = (scalar_overrides or {}).get(int(ContextLevel.ABSTRACT.value))
_apply_scalar_overrides(
msg_abstract,
- (scalar_overrides or {}).get(int(ContextLevel.ABSTRACT.value)),
+ level_overrides,
+ )
+ _apply_planned_field_patch(
+ msg_abstract,
+ (field_patches or {}).get(int(ContextLevel.ABSTRACT.value)),
)
_apply_ingest_options(msg_abstract, ingest_options)
if msg_abstract:
@@ -434,10 +487,11 @@ async def vectorize_directory_meta(
failure_message=f"Failed to enqueue directory L0 vector for {uri}",
)
if enqueued:
+ enqueued_levels.add(int(ContextLevel.ABSTRACT.value))
logger.debug(f"Enqueued directory L0 (abstract) for vectorization: {uri}")
except TaskWorkRejected:
logger.debug("Skipped directory vectorization for cancelling task: %s", uri)
- return
+ return enqueued_levels
except Exception as e:
logger.error(
f"Failed to enqueue directory L0 (abstract) for vectorization: {uri}: {e}",
@@ -466,10 +520,24 @@ async def vectorize_directory_meta(
context_overview.set_vectorize(
Vectorize(text=embedding_text_for_body(ContextLevel.OVERVIEW, uri, overview))
)
- msg_overview = EmbeddingMsgConverter.from_context(context_overview, creator_acl_grant)
+ msg_overview = EmbeddingMsgConverter.from_context(
+ context_overview,
+ creator_acl_grant,
+ action=IndexAction(
+ (actions or {}).get(
+ int(ContextLevel.OVERVIEW.value),
+ IndexAction.MERGE,
+ )
+ ),
+ )
+ level_overrides = (scalar_overrides or {}).get(int(ContextLevel.OVERVIEW.value))
_apply_scalar_overrides(
msg_overview,
- (scalar_overrides or {}).get(int(ContextLevel.OVERVIEW.value)),
+ level_overrides,
+ )
+ _apply_planned_field_patch(
+ msg_overview,
+ (field_patches or {}).get(int(ContextLevel.OVERVIEW.value)),
)
_apply_ingest_options(msg_overview, ingest_options)
if msg_overview:
@@ -480,10 +548,11 @@ async def vectorize_directory_meta(
failure_message=f"Failed to enqueue directory L1 vector for {uri}",
)
if enqueued:
+ enqueued_levels.add(int(ContextLevel.OVERVIEW.value))
logger.debug(f"Enqueued directory L1 (overview) for vectorization: {uri}")
except TaskWorkRejected:
logger.debug("Skipped directory vectorization for cancelling task: %s", uri)
- return
+ return enqueued_levels
except Exception as e:
logger.error(
f"Failed to enqueue directory L1 (overview) for vectorization: {uri}: {e}",
@@ -499,6 +568,7 @@ async def vectorize_directory_meta(
exc_info=True,
)
raise
+ return enqueued_levels
async def vectorize_file(
@@ -510,8 +580,12 @@ async def vectorize_file(
use_summary: bool = False,
preserve_existing_created_at: bool = False,
scalar_override: Optional[Dict[str, Any]] = None,
+ field_patch: FieldPatch | None = None,
ingest_options: IngestOptions | None = None,
creator_acl_grant: CreatorAclGrant | None = None,
+ file_md5: Optional[str] = None,
+ file_content: Optional[bytes] = None,
+ action: str = "merge",
) -> bool:
"""
Vectorize a single file.
@@ -553,7 +627,9 @@ async def vectorize_file(
owner_space=owner_space_for_uri(file_path),
)
- content_type = await _resolve_resource_content_type(file_path, file_name, viking_fs, ctx)
+ content_type = await _resolve_resource_content_type(
+ file_path, file_name, viking_fs, ctx, file_content=file_content
+ )
embedding_cfg = get_openviking_config().embedding
configured_text_source = embedding_cfg.text_source
effective_text_source = TEXT_SOURCE_SUMMARY_FIRST if use_summary else configured_text_source
@@ -570,7 +646,11 @@ async def vectorize_file(
)
context.set_vectorize(Vectorize(text=summary))
elif is_text_file(file_name):
- content = _coerce_text_file_content(await viking_fs.read_file(file_path, ctx=ctx))
+ content = _coerce_text_file_content(
+ file_content
+ if file_content is not None
+ else await viking_fs.read_file(file_path, ctx=ctx)
+ )
embedding_text = truncate_embedding_input(
content,
embedding_cfg.max_input_tokens,
@@ -588,7 +668,9 @@ async def vectorize_file(
else:
try:
content = _coerce_text_file_content(
- await viking_fs.read_file(file_path, ctx=ctx)
+ file_content
+ if file_content is not None
+ else await viking_fs.read_file(file_path, ctx=ctx)
)
except Exception as e:
logger.warning(
@@ -627,12 +709,28 @@ async def vectorize_file(
logger.debug(f"Skipping file {file_path} (no text content or summary)")
return False
- embedding_msg = EmbeddingMsgConverter.from_context(context, creator_acl_grant)
+ # md5 fingerprints the final stored bytes so incremental diff can skip
+ # unchanged files. It is supplied by the upload site that already holds
+ # those bytes (parser output store / diff apply); vectorize_file never
+ # reads the file back just to hash it. Unknown (None) leaves md5 empty and
+ # diff falls back to comparing bytes.
+ if file_md5:
+ context.md5 = file_md5
+
+ resolved_action = IndexAction(action)
+ if resolved_action not in {IndexAction.UPSERT, IndexAction.MERGE}:
+ raise ValueError(f"vectorize_file only supports upsert or merge actions: {action}")
+ embedding_msg = EmbeddingMsgConverter.from_context(
+ context,
+ creator_acl_grant,
+ action=resolved_action,
+ )
if not embedding_msg:
return False
- _apply_scalar_overrides(embedding_msg, scalar_override)
_apply_ingest_options(embedding_msg, ingest_options)
+ _apply_scalar_overrides(embedding_msg, scalar_override)
+ _apply_planned_field_patch(embedding_msg, field_patch)
enqueued = await _enqueue_embedding_message(
embedding_queue,
embedding_msg,
diff --git a/openviking/utils/log_correlation.py b/openviking/utils/log_correlation.py
new file mode 100644
index 000000000..ad9091789
--- /dev/null
+++ b/openviking/utils/log_correlation.py
@@ -0,0 +1,21 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+"""Stable correlation fields for logs emitted by durable queue work."""
+
+from openviking.service.task_work_index import get_task_context
+from openviking.telemetry import get_current_telemetry
+
+
+def log_correlation(*, task_id: str = "", telemetry_id: str = "", message_id: str = "") -> str:
+ """Return searchable task, telemetry, and optional queue-message IDs."""
+ task_context = get_task_context()
+ task_id = task_id or (task_context.task_id if task_context is not None else "-")
+ if not telemetry_id:
+ telemetry = get_current_telemetry()
+ telemetry_id = telemetry.telemetry_id if telemetry.enabled else "-"
+ fields = f"task_id={task_id} telemetry_id={telemetry_id or '-'}"
+ return f"{fields} message_id={message_id}" if message_id else fields
+
+
+__all__ = ["log_correlation"]
diff --git a/openviking/utils/media_processor.py b/openviking/utils/media_processor.py
index 989775309..2912ec136 100644
--- a/openviking/utils/media_processor.py
+++ b/openviking/utils/media_processor.py
@@ -2,10 +2,13 @@
# SPDX-License-Identifier: AGPL-3.0
"""Unified resource processor with strategy-based routing."""
+import time
+from collections.abc import Awaitable
from pathlib import Path
from typing import TYPE_CHECKING, Optional
from urllib.parse import urlparse
+from openviking.metrics.datasources.resource import ResourceIngestionEventDataSource
from openviking.parse.accessors.base import LocalResource, SourceType
from openviking.parse.accessors.mime_types import IANA_MEDIA_TYPE_TO_EXTENSION
from openviking.parse.backend import ParserBackend, normalize_parser_backend
@@ -25,6 +28,7 @@ from openviking.server.local_input_guard import (
is_remote_resource_source,
looks_like_local_path,
)
+from openviking.telemetry import get_current_telemetry
from openviking_cli.exceptions import InvalidArgumentError, PermissionDeniedError
from openviking_cli.utils.logger import get_logger
@@ -287,6 +291,25 @@ class UnifiedResourceProcessor:
- Directories needed for TreeBuilder are preserved via ParseResult.temp_dir_path
"""
+ metrics_account_id = kwargs.pop("_metrics_account_id", None)
+
+ async def run_stage(stage: str, operation: Awaitable[ParseResult | LocalResource]):
+ started_at = time.perf_counter()
+ status = "ok"
+ try:
+ with get_current_telemetry().measure(f"resource.{stage}"):
+ return await operation
+ except Exception:
+ status = "error"
+ raise
+ finally:
+ ResourceIngestionEventDataSource.record_stage(
+ stage=stage,
+ status=status,
+ duration_seconds=time.perf_counter() - started_at,
+ account_id=metrics_account_id,
+ )
+
mode = normalize_parse_mode(parse_mode)
# First check if source is raw content (not URL/path)
@@ -295,10 +318,13 @@ class UnifiedResourceProcessor:
)
if not is_potential_path and not self._is_url(source):
# Treat as raw content
- return await parse(
- source,
- instruction=instruction,
- split_content=mode is ParseMode.DEFAULT,
+ return await run_stage(
+ "parse_artifact",
+ parse(
+ source,
+ instruction=instruction,
+ split_content=mode is ParseMode.DEFAULT,
+ ),
)
if (
@@ -323,7 +349,9 @@ class UnifiedResourceProcessor:
explicit_name = kwargs.get("resource_name") or kwargs.get("source_name")
if explicit_name:
parse_kwargs["resource_name"] = _smart_stem(explicit_name)
- return await self._get_parser_router().parse(source, **parse_kwargs)
+ return await run_stage(
+ "parse_artifact", self._get_parser_router().parse(source, **parse_kwargs)
+ )
if (
mode is ParseMode.DEFAULT
@@ -345,16 +373,24 @@ class UnifiedResourceProcessor:
explicit_name = kwargs.get("resource_name") or kwargs.get("source_name")
if explicit_name:
parse_kwargs["resource_name"] = _smart_stem(explicit_name)
- return await self._get_parser_router().parse(source, **parse_kwargs)
+ return await run_stage(
+ "parse_artifact", self._get_parser_router().parse(source, **parse_kwargs)
+ )
# Phase 1: Accessor - get local resource. A caller may prepare a remote
# resource first so async routing uses the same detected file type.
- local_resource = prepared_resource or await self.prepare(
- source,
- allow_local_path_resolution=allow_local_path_resolution,
- **({"parse_mode": mode} if mode is ParseMode.NO_SPLIT else {}),
- **kwargs,
- )
+ if prepared_resource is not None:
+ local_resource = prepared_resource
+ else:
+ local_resource = await run_stage(
+ "source_prepare",
+ self.prepare(
+ source,
+ allow_local_path_resolution=allow_local_path_resolution,
+ **({"parse_mode": mode} if mode is ParseMode.NO_SPLIT else {}),
+ **kwargs,
+ ),
+ )
# Use context manager for automatic cleanup, but preserve directories for TreeBuilder
try:
@@ -410,7 +446,9 @@ class UnifiedResourceProcessor:
parser = DirectoryParser()
parse_kwargs["_feishu_import_plan"] = local_resource.feishu_plan
- result = await parser.parse(str(local_resource.path), **parse_kwargs)
+ result = await run_stage(
+ "parse_artifact", parser.parse(str(local_resource.path), **parse_kwargs)
+ )
# Preserve temporary directory for TreeBuilder
if local_resource.is_temporary and not result.temp_dir_path:
result.temp_dir_path = str(local_resource.path)
@@ -438,7 +476,9 @@ class UnifiedResourceProcessor:
await save("file", response_id)
parse_kwargs["_response_checkpoint"] = record
- return await parser_router.parse(local_resource, **parse_kwargs)
+ return await run_stage(
+ "parse_artifact", parser_router.parse(local_resource, **parse_kwargs)
+ )
finally:
# Clean up temporary resources unless they need to be preserved
local_resource.cleanup()
diff --git a/openviking/utils/resource_processor.py b/openviking/utils/resource_processor.py
index 866c330b4..aa811c8eb 100644
--- a/openviking/utils/resource_processor.py
+++ b/openviking/utils/resource_processor.py
@@ -29,8 +29,10 @@ from openviking.server.identity import RequestContext
from openviking.storage.acl import AclAction, CreatorAclGrant
from openviking.storage.errors import LockAcquisitionError
from openviking.storage.expr import And, Eq, PathScope
+from openviking.storage.index_action import FieldPatch
from openviking.storage.internal_names import STORAGE_INTERNAL_ENTRY_NAMES
from openviking.storage.queuefs.semantic_processor import SemanticProcessor
+from openviking.storage.resource_rnfv import RequestIntent
from openviking.storage.viking_fs import LS_ALL_NODES, get_viking_fs
from openviking.storage.vikingdb_manager import VikingDBManager
from openviking.telemetry import get_current_telemetry
@@ -38,6 +40,7 @@ from openviking.utils import is_github_url
from openviking.utils.embedding_utils import index_resource, vectorize_file
from openviking.utils.git_auth import is_git_https_url
from openviking.utils.ingest_options import IngestOptions
+from openviking.utils.log_correlation import log_correlation
from openviking.utils.summarizer import Summarizer
from openviking_cli.exceptions import OpenVikingError
from openviking_cli.utils import VikingURI, get_logger
@@ -53,6 +56,28 @@ _MAX_FILE_VECTORIZATION_CONCURRENCY = 64
VECTORDB_MAX_QUERY_LIMIT = 100_000
+class _DocRelStore:
+ """Adapt a parse output store so commit reads use target-relative paths.
+
+ Resource plans key files relative to the target root, while the underlying
+ artifact may store them below ``<doc_rel>/...``.
+ """
+
+ def __init__(self, store: Any, doc_rel: str) -> None:
+ self._store = store
+ base = (doc_rel or "").strip("/")
+ self._base = base
+
+ async def read_bytes(self, ref: Any, rel_path: str) -> bytes:
+ normalized = str(rel_path or "").strip("/")
+ artifact_rel = (
+ normalized
+ if not self._base or normalized == self._base or normalized.startswith(self._base + "/")
+ else f"{self._base}/{normalized}"
+ )
+ return await self._store.read_bytes(ref, artifact_rel)
+
+
class ResourceProcessor:
"""
Handles coordinated write operations.
@@ -137,6 +162,329 @@ class ResourceProcessor:
)
return self._media_processor
+ def _build_parse_output_store(self):
+ """Return the configured parse output store, or None for AGFS mode.
+
+ Local artifacts are consumed and committed to formal AGFS storage in
+ this synchronous request phase. They never cross an asynchronous queue
+ boundary, so parse and commit only need to run in the same process.
+ AGFS (the default) returns None to preserve its temp-tree path.
+ """
+ try:
+ parse_output = get_openviking_config().storage.parse_output
+ except Exception:
+ return None
+ if getattr(parse_output, "mode", "agfs") != "local":
+ return None
+ from openviking.parse.output import build_parse_output_store
+
+ return build_parse_output_store(
+ backend="local", local_root=parse_output.resolved_local_root()
+ )
+
+ @staticmethod
+ def _artifact_doc_rel(artifact_ref: Any, temp_doc_uri: Optional[str]) -> str:
+ """Return the document root's path relative to the artifact root.
+
+ ``temp_doc_uri`` is ``<artifact_root>/<doc_rel>`` (finalize built it), so
+ the relative document path is the suffix after the artifact root.
+ """
+ root = artifact_ref.root.rstrip("/")
+ doc = str(temp_doc_uri or "").rstrip("/")
+ if doc.startswith(f"{root}/"):
+ return doc[len(root) + 1 :]
+ return str(getattr(artifact_ref, "resource_rel", "") or "").strip("/")
+
+ @staticmethod
+ def _ensure_parse_artifact_ref(parse_result: Any) -> Any:
+ """Resolve an artifact ref for every supported parser result."""
+ artifact_ref = getattr(parse_result, "artifact_ref", None)
+ ensure = getattr(parse_result, "ensure_artifact_ref", None)
+ if callable(ensure):
+ artifact_ref = ensure()
+ if artifact_ref is not None:
+ return artifact_ref
+ temp_dir_path = str(getattr(parse_result, "temp_dir_path", "") or "")
+ if not temp_dir_path.startswith("viking://temp/"):
+ raise RuntimeError("add_resources requires a parse artifact")
+ from openviking.parse.output import ParseArtifactRef
+
+ return ParseArtifactRef(backend="agfs", root=temp_dir_path, root_type="dir")
+
+ @staticmethod
+ async def _cleanup_parse_result_artifact(
+ parse_result: Any, *, output_store: Any, viking_fs: Any, ctx: RequestContext
+ ) -> None:
+ artifact_ref = getattr(parse_result, "artifact_ref", None)
+ if artifact_ref is not None:
+ from openviking.parse.output import store_for_artifact_ref
+
+ store = (
+ output_store
+ if output_store is not None and output_store.backend == artifact_ref.backend
+ else store_for_artifact_ref(artifact_ref, viking_fs=viking_fs, ctx=ctx)
+ )
+ await store.cleanup(artifact_ref)
+ elif parse_result.temp_dir_path:
+ await viking_fs.delete_temp(parse_result.temp_dir_path, ctx=ctx)
+
+ @staticmethod
+ def _store_for_parse_artifact(
+ artifact_ref: Any, *, output_store: Any, viking_fs: Any, ctx: RequestContext
+ ) -> Any:
+ if output_store is not None and output_store.backend == artifact_ref.backend:
+ return output_store
+ from openviking.parse.output import store_for_artifact_ref
+
+ return store_for_artifact_ref(artifact_ref, viking_fs=viking_fs, ctx=ctx)
+
+ async def _commit_directory_artifact_with_plan(
+ self,
+ *,
+ output_store: Any,
+ artifact_ref: Any,
+ doc_rel: str,
+ root_uri: str,
+ target_preexisting: bool,
+ root_is_file: bool = False,
+ ctx: RequestContext,
+ lease_ref: Optional[Dict[str, Any]],
+ vectorize: bool,
+ summarize: bool,
+ processing_mode: ProcessingMode,
+ is_code_repo: bool,
+ ingest_options: IngestOptions,
+ source_metadata: Optional[Dict[str, str]],
+ ) -> Any:
+ """Resolve and commit one artifact through the canonical update plan."""
+ from openviking.metrics.datasources.resource import ResourceIngestionEventDataSource
+ from openviking.storage.context_update_plan import (
+ build_context_update_plan_from_snapshot,
+ execute_content_tree_actions,
+ )
+ from openviking.storage.resource_diff import (
+ build_rnfv_snapshot,
+ count_tree_entry_kinds,
+ prepare_artifact_inventory,
+ )
+ from openviking.storage.resource_target import AgfsResourceTarget
+
+ target = AgfsResourceTarget(
+ viking_fs=get_viking_fs(),
+ root_uri=root_uri,
+ ctx=ctx,
+ lease_ref=lease_ref,
+ )
+ telemetry = get_current_telemetry()
+ artifact_backend = str(getattr(artifact_ref, "backend", "unknown"))
+ update_plan_started_at = time.perf_counter()
+ try:
+ with telemetry.measure("resource.update_plan.artifact_inventory"):
+ artifact_inventory = await prepare_artifact_inventory(
+ output_store,
+ artifact_ref,
+ doc_rel=doc_rel,
+ target_root_uri=root_uri,
+ root_is_file=root_is_file,
+ )
+ plan_processing_mode = (
+ processing_mode
+ if processing_mode == VECTORS_ONLY or summarize or vectorize
+ else VECTORS_ONLY
+ )
+ request = RequestIntent.from_ingest_options(
+ target_uri=root_uri,
+ processing_mode=plan_processing_mode,
+ ingest_options=ingest_options,
+ vectorize=vectorize,
+ )
+ with telemetry.measure("resource.update_plan.rnfv_snapshot"):
+ rnfv = await build_rnfv_snapshot(
+ viking_fs=get_viking_fs(),
+ vikingdb=self.vikingdb,
+ store=output_store,
+ artifact_ref=artifact_ref,
+ target_uri=root_uri,
+ ctx=ctx,
+ doc_rel=doc_rel,
+ request_intent=request,
+ target_preexisting=target_preexisting,
+ artifact_inventory=artifact_inventory,
+ root_is_file=root_is_file,
+ )
+ from collections import Counter
+
+ n_files, n_dirs, _ = count_tree_entry_kinds(rnfv.new.entries)
+ f_files, f_dirs, _ = count_tree_entry_kinds(rnfv.formal.entries)
+ logger.debug(
+ "[RNFVSnapshot] %s target=%s artifact_backend=%s root_is_file=%s "
+ "target_preexisting=%s n_files=%d n_dirs=%d f_files=%d f_dirs=%d "
+ "v_records=%d v_levels=%s",
+ log_correlation(),
+ root_uri,
+ artifact_backend,
+ root_is_file,
+ target_preexisting,
+ n_files,
+ n_dirs,
+ f_files,
+ f_dirs,
+ len(rnfv.vectors.records_by_id),
+ dict(Counter(record.level for record in rnfv.vectors.records_by_id.values())),
+ )
+ with telemetry.measure("resource.update_plan.diff_and_compile"):
+ diff, context_plan = await build_context_update_plan_from_snapshot(
+ snapshot=rnfv,
+ store=_DocRelStore(output_store, doc_rel),
+ artifact_ref=artifact_ref,
+ target=target,
+ vikingdb=self.vikingdb,
+ context_type=context_type_for_uri(root_uri),
+ is_code_repo=is_code_repo,
+ account_id=ctx.account_id,
+ ctx=ctx,
+ root_preexisting=target_preexisting,
+ artifact_paths=artifact_inventory.artifact_paths,
+ ingest_options=ingest_options,
+ source_metadata=source_metadata,
+ root_is_file=root_is_file,
+ )
+ except Exception:
+ telemetry.set(
+ "resource.update_plan.duration_ms",
+ (time.perf_counter() - update_plan_started_at) * 1000.0,
+ )
+ ResourceIngestionEventDataSource.record_stage(
+ stage="update_plan",
+ status="error",
+ duration_seconds=time.perf_counter() - update_plan_started_at,
+ account_id=ctx.account_id,
+ )
+ raise
+ telemetry.set(
+ "resource.update_plan.duration_ms",
+ (time.perf_counter() - update_plan_started_at) * 1000.0,
+ )
+ ResourceIngestionEventDataSource.record_stage(
+ stage="update_plan",
+ status="ok",
+ duration_seconds=time.perf_counter() - update_plan_started_at,
+ account_id=ctx.account_id,
+ )
+
+ content_commit_started_at = time.perf_counter()
+ try:
+ with telemetry.measure("resource.content_commit"):
+ await execute_content_tree_actions(
+ context_plan.content_tree_actions,
+ store=_DocRelStore(output_store, doc_rel),
+ artifact_ref=artifact_ref,
+ target=target,
+ )
+ except Exception as exc:
+ ResourceIngestionEventDataSource.record_stage(
+ stage="content_commit",
+ status="error",
+ duration_seconds=time.perf_counter() - content_commit_started_at,
+ account_id=ctx.account_id,
+ )
+ logger.exception(
+ "[ContentTreeCommitFailed] %s target=%s artifact_backend=%s actions=%d error=%s",
+ log_correlation(),
+ root_uri,
+ artifact_backend,
+ len(context_plan.content_tree_actions),
+ exc,
+ )
+ raise
+ ResourceIngestionEventDataSource.record_stage(
+ stage="content_commit",
+ status="ok",
+ duration_seconds=time.perf_counter() - content_commit_started_at,
+ account_id=ctx.account_id,
+ )
+ self._log_content_tree_commit(
+ root_uri=root_uri,
+ artifact_backend=artifact_backend,
+ actions=context_plan.content_tree_actions,
+ duration_ms=(time.perf_counter() - content_commit_started_at) * 1000.0,
+ )
+ self._log_context_update_plan(context_plan)
+ self._log_context_commit_summary(
+ diff,
+ content_actions=context_plan.content_tree_actions,
+ root_uri=root_uri,
+ is_initial=not target_preexisting,
+ )
+ return context_plan.after_content_commit()
+
+ @staticmethod
+ def _log_content_tree_commit(
+ *, root_uri: str, artifact_backend: str, actions: Any, duration_ms: float
+ ) -> None:
+ log = logger.info if actions else logger.debug
+ log(
+ "[ContentTreeCommit] %s target=%s artifact_backend=%s uploaded_files=%d "
+ "created_dirs=%d deleted_paths=%d replaced_kinds=%d duration_ms=%.3f",
+ log_correlation(),
+ root_uri,
+ artifact_backend,
+ sum(action.new_kind == "file" for action in actions),
+ sum(action.new_kind == "directory" for action in actions),
+ sum(action.operation.value == "delete" for action in actions),
+ sum(action.operation.value == "replace_kind" for action in actions),
+ duration_ms,
+ )
+
+ @staticmethod
+ def _log_context_update_plan(plan: Any) -> None:
+ from collections import Counter
+
+ content = Counter(action.operation.value for action in plan.content_tree_actions)
+ direct = Counter(action.action.value for action in plan.direct_index_actions)
+ entries = plan.semantic_plan.tree.entries if plan.semantic_plan is not None else ()
+ semantic = Counter(entry.semantic_action.value for entry in entries)
+ slots = Counter(slot.action.value for entry in entries for slot in entry.index_slots)
+ logger.debug(
+ "[ContextUpdatePlan] %s root=%s content=%s semantic=%s direct_index=%s "
+ "index_slots=%s "
+ "semantic_entries=%d execution_roots=%d",
+ log_correlation(),
+ plan.root_uri,
+ dict(content),
+ dict(semantic),
+ dict(direct),
+ dict(slots),
+ len(entries),
+ len(plan.semantic_plan.execution_root_uris()) if plan.semantic_plan is not None else 0,
+ )
+
+ @staticmethod
+ def _log_context_commit_summary(
+ diff: Any, *, content_actions: Any, root_uri: str, is_initial: bool
+ ) -> None:
+ from collections import Counter
+
+ states = Counter(entry.content_state.value for entry in diff.entries.values())
+ uploaded_files = sum(action.new_kind == "file" for action in content_actions)
+ created_dirs = sum(action.new_kind == "directory" for action in content_actions)
+ if is_initial:
+ logger.debug(
+ "[add_resource] %s initial import committed root=%s files=%d dirs=%d",
+ log_correlation(),
+ root_uri,
+ uploaded_files,
+ created_dirs,
+ )
+ return
+ logger.debug(
+ "[add_resource] %s incremental diff committed root=%s states=%s uploaded=%d",
+ log_correlation(),
+ root_uri,
+ dict(states),
+ uploaded_files,
+ )
+
@staticmethod
def _empty_directory_error(meta: Dict[str, Any]) -> str:
"""Build a bounded error message for a directory with no successful files."""
@@ -155,6 +503,38 @@ class ResourceProcessor:
else:
message = "Directory import produced no content: no processable files were selected"
+ details = ResourceProcessor._failed_file_details(failures)
+ if details:
+ message += "; failed files: " + "; ".join(details)
+ if len(failures) > len(details):
+ message += f"; ... {len(failures) - len(details)} more"
+ return message
+
+ @staticmethod
+ def _directory_parse_failures(meta: Dict[str, Any]) -> List[Dict[str, Any]]:
+ """Return files selected for parsing whose parser did not produce content."""
+ failed_files = meta.get("failed_files")
+ if not isinstance(failed_files, list):
+ return []
+ return [
+ item
+ for item in failed_files
+ if isinstance(item, dict)
+ and (item.get("status") == "failed" or (not item.get("status") and "error" in item))
+ ]
+
+ @staticmethod
+ def _incomplete_directory_error(failures: List[Dict[str, Any]]) -> str:
+ message = f"Directory import incomplete: {len(failures)} file(s) failed to parse"
+ details = ResourceProcessor._failed_file_details(failures)
+ if details:
+ message += "; failed files: " + "; ".join(details)
+ if len(failures) > len(details):
+ message += f"; ... {len(failures) - len(details)} more"
+ return message
+
+ @staticmethod
+ def _failed_file_details(failures: List[Dict[str, Any]]) -> List[str]:
details: List[str] = []
for item in failures[:5]:
if not isinstance(item, dict):
@@ -167,11 +547,7 @@ class ResourceProcessor:
if remote_ids:
path = f"{path} ({', '.join(remote_ids)})"
details.append(f"{path}: {reason[:120]}")
- if details:
- message += "; failed files: " + "; ".join(details)
- if len(failures) > len(details):
- message += f"; ... {len(failures) - len(details)} more"
- return message
+ return details
async def prepare_durable_source(
self,
@@ -270,6 +646,7 @@ class ResourceProcessor:
ingest_options = IngestOptions.from_value(kwargs.pop("ingest_options", None))
to_is_directory = bool(kwargs.pop("to_is_directory", False))
telemetry = get_current_telemetry()
+ metrics_account_id = getattr(ctx, "account_id", None)
async def _set_stage(stage: str) -> None:
if stage_callback is None:
@@ -278,21 +655,25 @@ class ResourceProcessor:
if inspect.isawaitable(result):
await result
- with telemetry.measure("resource.process"):
+ with telemetry.measure("resource.source_execute"):
# ============ Phase 1: Parse source and writes to temp viking fs ============
try:
from openviking.metrics.datasources.resource import (
ResourceIngestionEventDataSource,
)
- parse_start = time.perf_counter()
- stage_start = time.perf_counter()
- stage_status = "ok"
media_processor = self._get_media_processor()
viking_fs = get_viking_fs()
# Use reason as instruction fallback so it influences L0/L1
# generation and improves search relevance as documented.
effective_instruction = instruction or reason
+ # Local artifact mode writes parse output to this worker's disk
+ # instead of AGFS temp. The same synchronous request commits all
+ # required bytes to formal AGFS before any async queue handoff.
+ # The store is task-scoped; default AGFS mode leaves kwargs untouched.
+ output_store = self._build_parse_output_store()
+ if output_store is not None:
+ kwargs.setdefault("parse_output_store", output_store)
if path.startswith(("http://", "https://", "git@", "ssh://", "git://")):
await _set_stage("fetching")
else:
@@ -303,6 +684,7 @@ class ResourceProcessor:
source=path,
instruction=effective_instruction,
prepared_resource=prepared_resource,
+ _metrics_account_id=metrics_account_id,
**kwargs,
)
result["source_path"] = parse_result.source_path or path
@@ -316,7 +698,6 @@ class ResourceProcessor:
result["errors"].extend(
parse_result.warnings or ["Parse failed: no content generated"],
)
- stage_status = "error"
return result
parse_meta = parse_result.meta if isinstance(parse_result.meta, dict) else {}
@@ -333,27 +714,37 @@ class ResourceProcessor:
result["status"] = "error"
result["errors"].append(self._empty_directory_error(parse_meta))
try:
- await viking_fs.delete_temp(parse_result.temp_dir_path, ctx=ctx)
+ await self._cleanup_parse_result_artifact(
+ parse_result, output_store=output_store, viking_fs=viking_fs, ctx=ctx
+ )
except Exception as exc:
logger.warning(
"[ResourceProcessor] Failed to clean empty directory temp %s: %s",
parse_result.temp_dir_path,
exc,
)
- stage_status = "error"
+ return result
+
+ parse_failures = self._directory_parse_failures(parse_meta)
+ if is_directory_aggregate and parse_failures:
+ result["status"] = "error"
+ result["errors"].append(self._incomplete_directory_error(parse_failures))
+ try:
+ await self._cleanup_parse_result_artifact(
+ parse_result, output_store=output_store, viking_fs=viking_fs, ctx=ctx
+ )
+ except Exception as exc:
+ logger.warning(
+ "[ResourceProcessor] Failed to clean incomplete directory temp %s: %s",
+ parse_result.temp_dir_path,
+ exc,
+ )
return result
if parse_result.warnings and kwargs.get("strict", False):
result.setdefault("warnings", []).extend(parse_result.warnings)
- telemetry.set(
- "resource.parse.duration_ms",
- round((time.perf_counter() - parse_start) * 1000, 3),
- )
- telemetry.set("resource.parse.warnings_count", len(parse_result.warnings or []))
-
except OpenVikingError:
- stage_status = "error"
raise
except Exception as e:
result["status"] = "error"
@@ -363,22 +754,11 @@ class ResourceProcessor:
error_message += f" (response_id={error_meta['response_id']})"
result["errors"].append(error_message)
logger.error(f"[ResourceProcessor] Parse error: {e}")
- telemetry.set_error("resource_processor.parse", "PROCESSING_ERROR", str(e))
+ telemetry.set_error("resource_processor.parse_artifact", "PROCESSING_ERROR", str(e))
import traceback
traceback.print_exc()
- stage_status = "error"
return result
- finally:
- try:
- ResourceIngestionEventDataSource.record_stage(
- stage="parse",
- status=str(stage_status),
- duration_seconds=float(time.perf_counter() - stage_start),
- account_id=getattr(ctx, "account_id", None),
- )
- except Exception:
- pass
# parse_result contains:
# - root: ResourceNode tree (with L0/L1 in meta)
@@ -387,10 +767,21 @@ class ResourceProcessor:
# ============ Phase 3: TreeBuilder finalizes from temp (scan + move to AGFS) ============
try:
- await _set_stage("finalizing")
+ await _set_stage("target_resolve")
stage_start = time.perf_counter()
stage_status = "ok"
finalize_start = time.perf_counter()
+ artifact_ref = getattr(parse_result, "artifact_ref", None)
+ artifact_output_store = (
+ self._store_for_parse_artifact(
+ artifact_ref,
+ output_store=output_store,
+ viking_fs=viking_fs,
+ ctx=ctx,
+ )
+ if artifact_ref is not None
+ else None
+ )
with get_viking_fs().bind_request_context(ctx):
context_tree = await self.tree_builder.finalize_from_temp(
temp_dir_path=parse_result.temp_dir_path,
@@ -407,25 +798,31 @@ class ResourceProcessor:
and parse_result.source_format not in {"directory", "repository"}
and not to_is_directory
),
+ artifact_ref=artifact_ref,
+ output_store=artifact_output_store,
)
if context_tree and context_tree.root:
result["root_uri"] = context_tree.root.uri
result["temp_uri"] = context_tree.root.temp_uri
root_is_file = bool(getattr(context_tree, "_root_is_file", False))
telemetry.set(
- "resource.finalize.duration_ms",
+ "resource.target_resolve.duration_ms",
round((time.perf_counter() - finalize_start) * 1000, 3),
)
except Exception as e:
result["status"] = "error"
result["errors"].append(f"Finalize from temp error: {e}")
- telemetry.set_error("resource_processor.finalize", "PROCESSING_ERROR", str(e))
+ telemetry.set_error("resource_processor.target_resolve", "PROCESSING_ERROR", str(e))
stage_status = "error"
- # Cleanup temporary directory on error (via VikingFS)
+ # Cleanup the parser-owned artifact through its own backend.
try:
- if parse_result.temp_dir_path:
- await get_viking_fs().delete_temp(parse_result.temp_dir_path, ctx=ctx)
+ await self._cleanup_parse_result_artifact(
+ parse_result,
+ output_store=output_store,
+ viking_fs=get_viking_fs(),
+ ctx=ctx,
+ )
except Exception:
pass
@@ -433,7 +830,7 @@ class ResourceProcessor:
finally:
try:
ResourceIngestionEventDataSource.record_stage(
- stage="finalize",
+ stage="target_resolve",
status=str(stage_status),
duration_seconds=float(time.perf_counter() - stage_start),
account_id=getattr(ctx, "account_id", None),
@@ -449,10 +846,11 @@ class ResourceProcessor:
resource_lock: Optional[Dict[str, Any]] = preacquired_lock
target_preexisting = False
source_committed = False
+ local_artifact_doc_rel = ""
+ incremental_noop = False
+ context_update_plan = None
if root_uri and temp_uri:
- stage_start = time.perf_counter()
- stage_status = "ok"
viking_fs = get_viking_fs()
try:
if candidate_uri:
@@ -499,56 +897,83 @@ class ResourceProcessor:
uri=root_uri,
root_is_file=root_is_file,
)
- if not target_preexisting:
- await viking_fs.persist_temp_tree(
- temp_uri,
- root_uri,
- ctx=ctx,
- lease_ref=resource_lock,
- )
- if not root_is_file:
- await rewrite_image_uris(
- root_uri,
- ctx=ctx,
- lease_ref=resource_lock,
- )
- await viking_fs.delete_temp(
- parse_result.temp_dir_path,
- ctx=ctx,
- )
- temp_uri = root_uri
- source_committed = True
+ artifact_ref = self._ensure_parse_artifact_ref(parse_result)
+ artifact_store = self._store_for_parse_artifact(
+ artifact_ref, output_store=output_store, viking_fs=viking_fs, ctx=ctx
+ )
+ local_artifact_doc_rel = self._artifact_doc_rel(artifact_ref, temp_uri)
+ context_update_plan = await self._commit_directory_artifact_with_plan(
+ output_store=artifact_store,
+ artifact_ref=artifact_ref,
+ doc_rel=local_artifact_doc_rel,
+ root_uri=root_uri,
+ target_preexisting=target_preexisting,
+ root_is_file=root_is_file,
+ ctx=ctx,
+ lease_ref=resource_lock,
+ vectorize=bool(kwargs.get("build_index", True)),
+ summarize=summarize,
+ processing_mode=normalize_processing_mode(kwargs.get("processing_mode")),
+ ingest_options=ingest_options,
+ is_code_repo=parse_result.source_format == "repository",
+ source_metadata=self._semantic_source_metadata(
+ path=path,
+ prepared_resource=prepared_resource,
+ source_format=parse_result.source_format,
+ ),
+ )
+ incremental_noop = target_preexisting and context_update_plan.is_noop()
+ temp_uri = root_uri
+ source_committed = True
except Exception:
- stage_status = "error"
# Mirror the Phase 3 (finalize) on-error cleanup: a lock or
# persist failure here would otherwise orphan the
# viking://temp tree with no GC (#2478). Skip when the temp
# tree was already persisted + deleted on the success path.
- if not source_committed and parse_result.temp_dir_path:
+ if not source_committed:
try:
- await get_viking_fs().delete_temp(parse_result.temp_dir_path, ctx=ctx)
+ await self._cleanup_parse_result_artifact(
+ parse_result,
+ output_store=output_store,
+ viking_fs=get_viking_fs(),
+ ctx=ctx,
+ )
except Exception:
pass
raise
- finally:
- try:
- ResourceIngestionEventDataSource.record_stage(
- stage="persist",
- status=str(stage_status),
- duration_seconds=float(time.perf_counter() - stage_start),
- account_id=getattr(ctx, "account_id", None),
- )
- except Exception:
- pass
+ if artifact_ref is not None:
+ artifact_store = self._store_for_parse_artifact(
+ artifact_ref,
+ output_store=output_store,
+ viking_fs=get_viking_fs(),
+ ctx=ctx,
+ )
+ try:
+ await artifact_store.cleanup(artifact_ref)
+ except Exception as exc:
+ logger.warning(
+ "[ResourceProcessor] Failed to clean committed parse artifact %s: %s",
+ artifact_ref.root,
+ exc,
+ )
+ artifact_ref = None
+ prepared_artifact_ref = artifact_ref.to_dict() if artifact_ref is not None else None
+ if prepared_artifact_ref is not None and artifact_ref.backend == "local":
+ prepared_artifact_ref["resource_rel"] = local_artifact_doc_rel
prepared = {
"root_uri": root_uri,
"temp_uri": temp_uri or parse_result.temp_dir_path,
"temp_dir_path": parse_result.temp_dir_path,
+ "artifact_ref": prepared_artifact_ref,
"source_committed": source_committed,
"target_preexisting": target_preexisting,
"is_code_repo": parse_result.source_format == "repository",
"root_is_file": root_is_file,
+ "incremental_noop": incremental_noop,
+ "context_update_plan": (
+ context_update_plan.to_dict() if context_update_plan is not None else None
+ ),
"semantic_source": self._semantic_source_metadata(
path=path,
prepared_resource=prepared_resource,
@@ -592,7 +1017,20 @@ class ResourceProcessor:
root_uri = str(prepared.get("root_uri") or "")
temp_uri = prepared.get("temp_uri")
temp_dir_path = prepared.get("temp_dir_path")
+ # Canonical plans contain only final resource URIs. An artifact ref is
+ # present here only for legacy non-plan paths that still need it.
+ artifact_ref_data = prepared.get("artifact_ref")
+ artifact_ref = None
+ if artifact_ref_data is not None:
+ from openviking.parse.output import ParseArtifactRef
+
+ artifact_ref = ParseArtifactRef.from_dict(artifact_ref_data)
+ if artifact_ref.backend not in {"agfs", "local"}:
+ raise ValueError(
+ f"Unsupported parse artifact backend in post-process: {artifact_ref.backend}"
+ )
source_committed = bool(prepared.get("source_committed"))
+ metrics_account_id = getattr(ctx, "account_id", None)
target_preexisting = bool(prepared.get("target_preexisting"))
build_index = bool(kwargs.get("build_index", True))
processing_mode = normalize_processing_mode(processing_mode)
@@ -600,51 +1038,124 @@ class ResourceProcessor:
root_is_file = bool(prepared.get("root_is_file"))
ingest_options = IngestOptions.from_value(kwargs.pop("ingest_options", None))
semantic_source = prepared.get("semantic_source")
- should_summarize = not root_is_file and not vectors_only and (summarize or build_index)
- should_refresh_file_parent = (
- root_is_file and not vectors_only and (summarize or build_index)
+ context_update_plan_data = prepared.get("context_update_plan")
+ context_update_plan = None
+ if context_update_plan_data is not None:
+ from openviking.storage.context_update_plan import ContextUpdatePlan
+
+ context_update_plan = ContextUpdatePlan.from_dict(context_update_plan_data)
+ semantic_plan = (
+ context_update_plan.semantic_plan.to_dict()
+ if context_update_plan.semantic_plan is not None
+ else None
+ )
+ direct_index_actions = context_update_plan.direct_index_actions
+ else:
+ semantic_plan = None
+ direct_index_actions = ()
+ uses_context_update_plan = context_update_plan_data is not None
+ should_summarize = (
+ not root_is_file
+ and not vectors_only
+ and (summarize or build_index)
+ and (not uses_context_update_plan or semantic_plan is not None)
)
+ if context_update_plan is not None:
+ file_refresh = context_update_plan.file_refresh
+ elif root_is_file and source_committed and not vectors_only and (summarize or build_index):
+ from openviking.storage.context_update_plan import FileRefreshIntent
+
+ file_refresh = FileRefreshIntent(
+ root_uri, (prepared.get("file_md5s") or {}).get(root_uri)
+ )
+ else:
+ file_refresh = None
+ should_refresh_file_parent = file_refresh is not None and not vectors_only
result: Dict[str, Any] = {"status": "success", "root_uri": root_uri}
+ artifact_cleaned = False
- if should_summarize:
- stage_start = time.perf_counter()
- stage_status = "ok"
- try:
- with get_current_telemetry().measure("resource.summarize"):
- summary_result = await self._get_summarizer().summarize(
- resource_uris=[root_uri],
- ctx=ctx,
- skip_vectorization=not build_index,
- lock=resource_lock,
- temp_uris=[temp_uri],
- is_code_repo=bool(prepared.get("is_code_repo")),
- target_preexisting=target_preexisting,
- ingest_options=ingest_options,
- semantic_source=semantic_source,
- generation_trigger="resource_ingest",
- **kwargs,
- )
- if (
- resource_lock is not None
- and summary_result.get("status") == "success"
- and summary_result.get("enqueued_count", 0) > 0
- ):
- await get_viking_fs()._async_agfs.pathlock_handoff(resource_lock)
+ async def cleanup_artifact_if_owned() -> None:
+ nonlocal artifact_cleaned
+ if artifact_cleaned or artifact_ref is None:
+ return
+ from openviking.parse.output import AgfsParseOutputStore
+
+ # Local mode reuses the processor's own store seam (test-injectable);
+ # agfs mode builds a temp-backed store on demand.
+ output_store = (
+ self._build_parse_output_store()
+ if artifact_ref.backend == "local"
+ else AgfsParseOutputStore(viking_fs=get_viking_fs(), ctx=ctx)
+ )
+ if output_store is not None:
+ await output_store.cleanup(artifact_ref)
+ artifact_cleaned = True
+
+ derived_enqueue_started_at = time.perf_counter()
+ derived_enqueue_status = "ok"
+ try:
+ with get_current_telemetry().measure("resource.derived_enqueue"):
+ if prepared.get("incremental_noop") and not direct_index_actions:
+ await cleanup_artifact_if_owned()
+ if resource_lock is not None:
+ await get_viking_fs()._async_agfs.pathlock_release(resource_lock)
resource_lock = None
- except Exception as exc:
- logger.error("Summarization failed: %s", exc)
- result["warnings"] = [f"Summarization failed: {exc}"]
- stage_status = "error"
- finally:
- try:
- ResourceIngestionEventDataSource.record_stage(
- stage="summarize",
- status=stage_status,
- duration_seconds=float(time.perf_counter() - stage_start),
- account_id=getattr(ctx, "account_id", None),
- )
- except Exception:
- pass
+ return result
+
+ if direct_index_actions:
+ await self._enqueue_index_actions(direct_index_actions, ctx=ctx)
+
+ if should_summarize:
+ try:
+ summary_result = await self._get_summarizer().summarize(
+ resource_uris=[root_uri],
+ ctx=ctx,
+ skip_vectorization=not build_index,
+ lock=resource_lock,
+ temp_uris=[temp_uri],
+ is_code_repo=bool(prepared.get("is_code_repo")),
+ target_preexisting=target_preexisting,
+ ingest_options=ingest_options,
+ semantic_source=semantic_source,
+ generation_trigger="resource_ingest",
+ semantic_plan=semantic_plan,
+ **kwargs,
+ )
+ if semantic_plan is not None and summary_result.get("status") != "success":
+ raise RuntimeError(
+ str(summary_result.get("message") or "semantic plan enqueue failed")
+ )
+ if (
+ resource_lock is not None
+ and summary_result.get("status") == "success"
+ and summary_result.get("enqueued_count", 0) > 0
+ ):
+ await get_viking_fs()._async_agfs.pathlock_handoff(resource_lock)
+ resource_lock = None
+ if semantic_plan is not None and (
+ summary_result.get("status") == "success"
+ and summary_result.get("enqueued_count", 0) > 0
+ ):
+ await cleanup_artifact_if_owned()
+ except Exception as exc:
+ logger.error("Semantic enqueue failed: %s", exc)
+ if semantic_plan is not None:
+ raise
+ result["warnings"] = [f"Semantic enqueue failed: {exc}"]
+ except Exception:
+ derived_enqueue_status = "error"
+ await cleanup_artifact_if_owned()
+ if resource_lock is not None:
+ await get_viking_fs()._async_agfs.pathlock_release(resource_lock)
+ resource_lock = None
+ raise
+ finally:
+ ResourceIngestionEventDataSource.record_stage(
+ stage="derived_enqueue",
+ status=derived_enqueue_status,
+ duration_seconds=time.perf_counter() - derived_enqueue_started_at,
+ account_id=metrics_account_id,
+ )
if resource_lock is not None:
try:
@@ -654,19 +1165,13 @@ class ResourceProcessor:
viking_fs = get_viking_fs()
if vectors_only and target_preexisting and not root_is_file:
diff = await SemanticProcessor()._sync_topdown_recursive(
- temp_uri,
- root_uri,
- ctx=ctx,
- lock=resource_lock,
+ temp_uri, root_uri, ctx=ctx, lock=resource_lock
)
sync_deleted_files = list(getattr(diff, "deleted_files", []))
sync_deleted_dirs = list(getattr(diff, "deleted_dirs", []))
else:
await viking_fs.persist_temp_tree(
- temp_uri,
- root_uri,
- ctx=ctx,
- lease_ref=resource_lock,
+ temp_uri, root_uri, ctx=ctx, lease_ref=resource_lock
)
if not root_is_file:
await rewrite_image_uris(
@@ -676,22 +1181,25 @@ class ResourceProcessor:
)
if temp_dir_path:
await viking_fs.delete_temp(temp_dir_path, ctx=ctx)
- if vectors_only:
- if sync_deleted_files or sync_deleted_dirs:
- await self._delete_removed_resource_vectors(
- files=sync_deleted_files,
- dirs=sync_deleted_dirs,
- ctx=ctx,
- )
+ if (
+ context_update_plan is None
+ and vectors_only
+ and (sync_deleted_files or sync_deleted_dirs)
+ ):
+ await self._delete_removed_resource_vectors(
+ files=sync_deleted_files, dirs=sync_deleted_dirs, ctx=ctx
+ )
if should_refresh_file_parent:
await self._get_summarizer().refresh_file_parent(
- file_uri=root_uri,
+ file_uri=file_refresh.file_uri,
ctx=ctx,
skip_vectorization=not build_index,
ingest_options=ingest_options,
created=not target_preexisting,
+ file_md5=file_refresh.md5,
+ file_abstract="",
)
- elif build_index:
+ elif build_index and context_update_plan is None:
if root_is_file:
await self._vectorize_resource_file(
root_uri,
@@ -700,35 +1208,61 @@ class ResourceProcessor:
creator_acl_grant=(
CreatorAclGrant.DIRECT if not target_preexisting else None
),
+ file_md5=(prepared.get("file_md5s") or {}).get(root_uri),
)
elif vectors_only:
await self._vectorize_resource_files(
root_uri, ctx=ctx, ingest_options=ingest_options
)
+ except BaseException:
+ await cleanup_artifact_if_owned()
+ raise
finally:
await get_viking_fs()._async_agfs.pathlock_release(resource_lock)
elif should_refresh_file_parent:
- await self._get_summarizer().refresh_file_parent(
- file_uri=root_uri,
- ctx=ctx,
- skip_vectorization=not build_index,
- ingest_options=ingest_options,
- created=not target_preexisting,
- )
- elif vectors_only or root_is_file:
- if not build_index:
- return result
- if root_is_file:
+ try:
+ await self._get_summarizer().refresh_file_parent(
+ file_uri=file_refresh.file_uri,
+ ctx=ctx,
+ skip_vectorization=not build_index,
+ ingest_options=ingest_options,
+ created=not target_preexisting,
+ file_md5=file_refresh.md5,
+ file_abstract="",
+ )
+ except BaseException:
+ await cleanup_artifact_if_owned()
+ raise
+ elif build_index and context_update_plan is None and root_is_file:
+ try:
await self._vectorize_resource_file(
root_uri,
ctx=ctx,
ingest_options=ingest_options,
creator_acl_grant=(CreatorAclGrant.DIRECT if not target_preexisting else None),
+ file_md5=(prepared.get("file_md5s") or {}).get(root_uri),
+ )
+ except BaseException:
+ await cleanup_artifact_if_owned()
+ raise
+ elif build_index and context_update_plan is None and vectors_only:
+ try:
+ artifact_store = (
+ self._build_parse_output_store() if artifact_ref is not None else None
)
- else:
await self._vectorize_resource_files(
- root_uri, ctx=ctx, ingest_options=ingest_options
+ root_uri,
+ ctx=ctx,
+ ingest_options=ingest_options,
+ artifact_store=artifact_store,
+ artifact_ref=artifact_ref,
+ artifact_files=prepared.get("artifact_files"),
+ file_md5s=prepared.get("file_md5s"),
)
+ except BaseException:
+ await cleanup_artifact_if_owned()
+ raise
+ await cleanup_artifact_if_owned()
return result
@staticmethod
@@ -755,6 +1289,78 @@ class ResourceProcessor:
kind = "local"
return {"kind": kind, "uri": str(path)}
+ async def _enqueue_index_actions(self, actions: Any, *, ctx: RequestContext) -> None:
+ from collections import Counter
+
+ from openviking.storage.index_action import IndexAction
+ from openviking.storage.queuefs import get_queue_manager
+ from openviking.storage.queuefs.embedding_msg import EmbeddingMsg
+ from openviking.telemetry import get_current_telemetry
+ from openviking.utils.embedding_utils import _enqueue_embedding_message
+
+ queue_manager = get_queue_manager()
+ embedding_queue = queue_manager.get_queue(queue_manager.EMBEDDING, allow_create=True)
+ telemetry_id = get_current_telemetry().telemetry_id
+ action_counts = Counter(action.action.value for action in actions)
+ delete_ids = [action.record_id for action in actions if action.action == IndexAction.DELETE]
+ if delete_ids:
+ message = EmbeddingMsg.for_delete(
+ record_ids=delete_ids,
+ context_data={
+ "uri": actions[0].uri,
+ "account_id": ctx.account_id,
+ "owner_user_id": ctx.user.user_id,
+ },
+ telemetry_id=telemetry_id,
+ )
+ await _enqueue_embedding_message(
+ embedding_queue,
+ message,
+ failure_message="Failed to enqueue planned vector deletes",
+ )
+ for action in actions:
+ if action.action in {IndexAction.UPSERT, IndexAction.MERGE}:
+ if action.level != int(ContextLevel.DETAIL):
+ raise ValueError("Direct index upsert only supports file detail records")
+ await self._vectorize_resource_file(
+ action.uri,
+ ctx=ctx,
+ file_md5=action.md5,
+ scalar_override={
+ **dict(action.upsert_fields),
+ "_record_id": action.record_id,
+ },
+ action=action.action.value,
+ field_patch=action.field_patch,
+ )
+ continue
+ if action.action != IndexAction.UPDATE_FIELDS:
+ continue
+ assert action.field_patch is not None
+ message = EmbeddingMsg.for_update_fields(
+ record_id=action.record_id,
+ field_patch=action.field_patch,
+ context_data={
+ "uri": action.uri,
+ "level": action.level,
+ "account_id": ctx.account_id,
+ "owner_user_id": ctx.user.user_id,
+ },
+ telemetry_id=telemetry_id,
+ )
+ await _enqueue_embedding_message(
+ embedding_queue,
+ message,
+ failure_message=f"Failed to enqueue scalar update for {action.uri}",
+ )
+ logger.debug(
+ "[DirectIndexActions] %s root=%s action_counts=%s action_count=%d",
+ log_correlation(),
+ actions[0].uri if actions else "",
+ dict(action_counts),
+ len(actions),
+ )
+
async def _delete_removed_resource_vectors(
self,
*,
@@ -795,27 +1401,38 @@ class ResourceProcessor:
*,
ctx: RequestContext,
ingest_options: IngestOptions | None = None,
+ artifact_store: Any = None,
+ artifact_ref: Any = None,
+ artifact_files: Optional[List[str]] = None,
+ file_md5s: Optional[Dict[str, str]] = None,
) -> None:
ingest_options = IngestOptions.from_value(ingest_options)
viking_fs = get_viking_fs()
- entries = await viking_fs.tree(
- root_uri,
- node_limit=None,
- level_limit=None,
- ctx=ctx,
- )
files: list[tuple[str, str, str]] = []
- for entry in entries:
- entry_uri = entry.get("uri") if isinstance(entry, dict) else None
- if not entry_uri or entry.get("isDir"):
- continue
- name = entry.get("name") or entry_uri.rsplit("/", 1)[-1]
- if str(name).startswith("."):
- continue
- parent = VikingURI(entry_uri).parent
- if parent is None:
- continue
- files.append((entry_uri, str(name), parent.uri))
+ if artifact_files is not None:
+ for rel_path in artifact_files:
+ entry_uri = VikingURI(root_uri).join(rel_path).uri
+ parent = VikingURI(entry_uri).parent
+ if parent is not None:
+ files.append((entry_uri, rel_path.rsplit("/", 1)[-1], parent.uri))
+ else:
+ entries = await viking_fs.tree(
+ root_uri,
+ node_limit=None,
+ level_limit=None,
+ ctx=ctx,
+ )
+ for entry in entries:
+ entry_uri = entry.get("uri") if isinstance(entry, dict) else None
+ if not entry_uri or entry.get("isDir"):
+ continue
+ name = entry.get("name") or entry_uri.rsplit("/", 1)[-1]
+ if str(name).startswith("."):
+ continue
+ parent = VikingURI(entry_uri).parent
+ if parent is None:
+ continue
+ files.append((entry_uri, str(name), parent.uri))
config = get_openviking_config().queue_workers.add_resource
concurrency = max(
@@ -827,6 +1444,15 @@ class ResourceProcessor:
)
async def vectorize(entry_uri: str, name: str, parent_uri: str) -> None:
+ file_content = None
+ if artifact_store is not None and artifact_ref is not None:
+ rel_path = entry_uri[len(root_uri.rstrip("/")) + 1 :]
+ artifact_rel = (
+ f"{artifact_ref.resource_rel.strip('/')}/{rel_path}"
+ if artifact_ref.resource_rel
+ else rel_path
+ )
+ file_content = await artifact_store.read_bytes(artifact_ref, artifact_rel)
await vectorize_file(
file_path=entry_uri,
summary_dict={"name": name, "summary": ""},
@@ -834,6 +1460,8 @@ class ResourceProcessor:
context_type=context_type_for_uri(entry_uri),
ctx=ctx,
ingest_options=ingest_options,
+ file_md5=(file_md5s or {}).get(entry_uri),
+ file_content=file_content,
)
for start in range(0, len(files), concurrency):
@@ -856,6 +1484,10 @@ class ResourceProcessor:
ctx: RequestContext,
ingest_options: IngestOptions | None = None,
creator_acl_grant: CreatorAclGrant | None = None,
+ file_md5: str | None = None,
+ scalar_override: Optional[Dict[str, Any]] = None,
+ field_patch: FieldPatch | None = None,
+ action: str = "merge",
) -> None:
parent = VikingURI(file_uri).parent
if parent is None:
@@ -869,6 +1501,10 @@ class ResourceProcessor:
ctx=ctx,
ingest_options=IngestOptions.from_value(ingest_options),
creator_acl_grant=creator_acl_grant,
+ file_md5=file_md5,
+ scalar_override=scalar_override,
+ field_patch=field_patch,
+ action=action,
)
async def reserve_unique_candidate(
diff --git a/openviking/utils/summarizer.py b/openviking/utils/summarizer.py
index c39f6d443..a59b109f6 100644
--- a/openviking/utils/summarizer.py
+++ b/openviking/utils/summarizer.py
@@ -40,6 +40,8 @@ class Summarizer:
skip_vectorization: bool = False,
ingest_options: IngestOptions | None = None,
created: bool = False,
+ file_md5: str | None = None,
+ file_abstract: str = "",
) -> Dict[str, Any]:
"""Summarize one flat file and refresh its parent directory semantics."""
parent = VikingURI(file_uri).parent
@@ -69,6 +71,8 @@ class Summarizer:
peer_id=ctx.user.user_id,
),
ingest_options=ingest_options,
+ file_md5s={file_uri: file_md5} if file_md5 else None,
+ file_abstracts={file_uri: file_abstract} if file_abstract else None,
)
if telemetry_id:
get_request_wait_tracker().register_semantic_root(telemetry_id, msg.id)
@@ -111,6 +115,17 @@ class Summarizer:
ingest_options = IngestOptions.from_value(kwargs.get("ingest_options"))
source = kwargs.get("semantic_source")
generation_trigger = str(kwargs.get("generation_trigger") or "manual_refresh")
+ # Pre-computed change set (local incremental import): when present, the
+ # semantic tree restricts re-summarization/vectorization to these files
+ # instead of diffing the whole tree.
+ changes = kwargs.get("changes")
+ # Per-file md5 (target-URI keyed) for those changed files, so the tree executor's
+ # re-vectorization records the fresh fingerprint.
+ file_md5s = kwargs.get("file_md5s") or {}
+ artifact_ref = kwargs.get("artifact_ref")
+ artifact_files = kwargs.get("artifact_files") or []
+ file_abstracts = kwargs.get("file_abstracts") or {}
+ semantic_plan = kwargs.get("semantic_plan")
if not temp_uris:
temp_uris = resource_uris
if len(temp_uris) != len(resource_uris):
@@ -195,6 +210,12 @@ class Summarizer:
ingest_options=ingest_options,
source=source,
generation_trigger=generation_trigger,
+ changes=changes,
+ file_md5s=file_md5s,
+ artifact_ref=artifact_ref,
+ artifact_files=artifact_files,
+ file_abstracts=file_abstracts,
+ plan=semantic_plan,
)
if msg.telemetry_id:
get_request_wait_tracker().register_semantic_root(msg.telemetry_id, msg.id)
diff --git a/openviking_cli/utils/config/parse_output_config.py b/openviking_cli/utils/config/parse_output_config.py
new file mode 100644
index 000000000..61fca333f
--- /dev/null
+++ b/openviking_cli/utils/config/parse_output_config.py
@@ -0,0 +1,43 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+from pathlib import Path
+from typing import Optional
+
+from pydantic import BaseModel, Field, model_validator
+
+
+class ParseOutputConfig(BaseModel):
+ """Where parsers write intermediate artifacts.
+
+ ``agfs`` (default) writes to the shared AGFS ``viking://temp`` space. ``local``
+ keeps parser intermediates on the current worker; add-resources commits the
+ required bytes to the formal AGFS tree before enqueueing downstream work.
+ """
+
+ mode: str = Field(
+ default="agfs",
+ description="Parse artifact backend: 'agfs' (shared temp) or 'local' (local dir).",
+ )
+ local_root: Optional[str] = Field(
+ default=None,
+ description=(
+ "Directory root for the 'local' backend. Defaults to a dedicated subdir "
+ "under the system temp dir when unset."
+ ),
+ )
+
+ model_config = {"extra": "forbid"}
+
+ @model_validator(mode="after")
+ def validate_mode(self):
+ if self.mode not in {"agfs", "local"}:
+ raise ValueError("storage.parse_output.mode must be 'agfs' or 'local'")
+ return self
+
+ def resolved_local_root(self) -> str:
+ """Return the local root, falling back to a dedicated system-temp subdir."""
+ if self.local_root:
+ return str(Path(self.local_root).expanduser())
+ import tempfile
+
+ return str(Path(tempfile.gettempdir()) / "openviking-parse")
diff --git a/openviking_cli/utils/config/queue_worker_config.py b/openviking_cli/utils/config/queue_worker_config.py
index e26ce36c1..3ebb7f01a 100644
--- a/openviking_cli/utils/config/queue_worker_config.py
+++ b/openviking_cli/utils/config/queue_worker_config.py
@@ -18,6 +18,12 @@ class QueueWorkerConfig(BaseModel):
class AddResourceQueueWorkerConfig(QueueWorkerConfig):
"""Runtime limits for add-resource queue workers."""
+ file_operation_concurrency: int = Field(
+ default=16,
+ gt=0,
+ description="Maximum concurrent file-level commit and comparison operations within one add-resource job",
+ )
+
file_vectorization_concurrency: int = Field(
default=8,
gt=0,
diff --git a/openviking_cli/utils/config/storage_config.py b/openviking_cli/utils/config/storage_config.py
index ab1953c61..f738c0b75 100644
--- a/openviking_cli/utils/config/storage_config.py
+++ b/openviking_cli/utils/config/storage_config.py
@@ -8,6 +8,7 @@ from pydantic import BaseModel, Field, model_validator
from openviking_cli.utils.logger import get_logger
from .agfs_config import AGFSConfig
+from .parse_output_config import ParseOutputConfig
from .transaction_config import TransactionConfig
from .vectordb_config import VectorDBBackendConfig
@@ -44,6 +45,11 @@ class StorageConfig(BaseModel):
description="VectorDB backend configuration",
)
+ parse_output: ParseOutputConfig = Field(
+ default_factory=ParseOutputConfig,
+ description="Where parsers write intermediate artifacts (agfs temp or local dir)",
+ )
+
params: Dict[str, Any] = Field(
default_factory=dict, description="Additional storage-specific parameters"
)
diff --git a/tests/metrics/integration/test_bootstrap.py b/tests/metrics/integration/test_bootstrap.py
index 5c0f93432..186432153 100644
--- a/tests/metrics/integration/test_bootstrap.py
+++ b/tests/metrics/integration/test_bootstrap.py
@@ -99,7 +99,7 @@ def test_create_default_collector_manager_propagates_construction_failures(monke
def test_optional_cache_datasource_instrumentation_is_wired_into_key_call_sites():
project_root = Path(__file__).resolve().parents[3]
targets = [
- project_root / "openviking" / "storage" / "queuefs" / "semantic_dag.py",
+ project_root / "openviking" / "storage" / "queuefs" / "semantic_executor.py",
project_root / "openviking" / "storage" / "queuefs" / "semantic_processor.py",
project_root / "openviking" / "session" / "memory" / "extract_loop.py",
]
diff --git a/tests/misc/test_ovpack_import_policy.py b/tests/misc/test_ovpack_import_policy.py
index 7f93a3440..9b2df3b77 100644
--- a/tests/misc/test_ovpack_import_policy.py
+++ b/tests/misc/test_ovpack_import_policy.py
@@ -342,6 +342,7 @@ class FakeVectorStore:
"context_type": "resource",
"level": 2,
"abstract": "note summary",
+ "md5": "note-md5",
"tags": ["snapshot"],
"vector": [0.1, 0.2, 0.3],
}
@@ -567,11 +568,13 @@ def test_index_consistency_report_limits_public_and_error_records():
async def test_export_ovpack_writes_v3_manifest_with_abstract_overviews(
temp_ovpack_path: Path, request_ctx: RequestContext
):
+ vector_store = FakeVectorStore()
await export_ovpack(
FakeExportVikingFS(),
"viking://resources/demo",
str(temp_ovpack_path),
ctx=request_ctx,
+ vector_store=vector_store,
)
with zipfile.ZipFile(temp_ovpack_path, "r") as zf:
@@ -602,6 +605,8 @@ async def test_export_ovpack_writes_v3_manifest_with_abstract_overviews(
assert manifest["index"]["records"]["count"] == len(index_records)
assert index_records[0]["path"] == ""
assert index_records[0]["text"] == "root abstract"
+ note_record = next(record for record in index_records if record["path"] == "notes.txt")
+ assert note_record["scalars"]["md5"] == "note-md5"
@pytest.mark.asyncio
@@ -748,6 +753,7 @@ async def test_restore_ovpack_applies_backup_manifest_scalar_metadata(
"abstract": "portable summary",
"description": "portable description",
"tags": ["portable"],
+ "md5": "portable-md5",
},
}
],
@@ -771,6 +777,7 @@ async def test_restore_ovpack_applies_backup_manifest_scalar_metadata(
"summary": "portable summary",
}
assert vectorized_files[0]["scalar_override"]["tags"] == ["portable"]
+ assert vectorized_files[0]["scalar_override"]["md5"] == "portable-md5"
@pytest.mark.asyncio
diff --git a/tests/misc/test_resource_processor_mv.py b/tests/misc/test_resource_processor_mv.py
index ea710786d..a3c1320e2 100644
--- a/tests/misc/test_resource_processor_mv.py
+++ b/tests/misc/test_resource_processor_mv.py
@@ -6,6 +6,14 @@ from unittest.mock import AsyncMock, MagicMock
import pytest
+from openviking.storage.context_update_plan import (
+ ContextUpdatePlan,
+ FileRefreshIntent,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+)
+
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
@@ -14,6 +22,18 @@ class _DummyVikingDB:
return None
+def _directory_plan(root_uri: str) -> ContextUpdatePlan:
+ return ContextUpdatePlan(
+ root_uri,
+ "resource",
+ semantic_plan=SemanticPlan(
+ root_uri,
+ "resource",
+ SemanticTreeSnapshot((SemanticTreeEntry("", "directory", "added", "aggregate"),)),
+ ),
+ )
+
+
class _DummyTelemetry:
def set(self, *args, **kwargs):
return None
@@ -70,6 +90,9 @@ class _FakePathLock:
async def pathlock_release(self, lease):
pass
+ async def pathlock_handoff(self, lease):
+ pass
+
class _FakeVikingFS:
def __init__(self, *, exists_result=False, existing_uris=None, pathlock=None):
@@ -86,6 +109,9 @@ class _FakeVikingFS:
def bind_request_context(self, ctx):
return _CtxMgr()
+ async def _ensure_access(self, uri, ctx, action):
+ return None
+
async def exists(self, uri, ctx=None):
self.exists_calls.append(uri)
if self._existing_uris:
@@ -180,6 +206,131 @@ async def test_resource_processor_rejects_directory_when_every_file_failed(monke
rp.tree_builder.finalize_from_temp.assert_not_awaited()
[email protected]("build_index", [True, False])
+async def test_resource_processor_rejects_partial_directory_before_semantic_plan_diff(
+ monkeypatch, build_index
+):
+ from openviking.parse.output import ParseArtifactRef
+ from openviking.utils.resource_processor import ResourceProcessor
+
+ fake_fs = _FakeVikingFS(exists_result=True)
+ monkeypatch.setattr(
+ "openviking.utils.resource_processor.get_current_telemetry",
+ lambda: _DummyTelemetry(),
+ )
+ _patch_viking_fs(monkeypatch, fake_fs)
+
+ ref = ParseArtifactRef(backend="agfs", root="viking://temp/partial")
+ parse_result = SimpleNamespace(
+ temp_dir_path=ref.root,
+ source_path="repo",
+ source_format="repository",
+ meta={
+ "file_count": 1,
+ "total_processable": 2,
+ "processed_files": [{"path": "kept.py"}],
+ "failed_files": [
+ {
+ "path": "missing.py",
+ "parser": "native",
+ "error": "parser failed",
+ }
+ ],
+ },
+ warnings=[],
+ artifact_ref=ref,
+ ensure_artifact_ref=lambda: ref,
+ )
+ rp = ResourceProcessor(vikingdb=_DummyVikingDB(), media_storage=None)
+ rp._get_media_processor = MagicMock()
+ rp._get_media_processor.return_value.process = AsyncMock(return_value=parse_result)
+ rp.tree_builder.finalize_from_temp = AsyncMock(
+ side_effect=AssertionError("partial artifact must not reach finalize/diff")
+ )
+ rp._commit_directory_artifact_with_plan = AsyncMock(
+ return_value=_directory_plan("viking://resources/root")
+ )
+
+ result = await rp.process_resource(path="repo", ctx=object(), build_index=build_index)
+
+ assert result["status"] == "error"
+ assert result["errors"] == [
+ "Directory import incomplete: 1 file(s) failed to parse; "
+ "failed files: missing.py: parser failed"
+ ]
+ assert fake_fs.delete_temp_calls == [(ref.root, None)]
+ rp.tree_builder.finalize_from_temp.assert_not_awaited()
+
+
+async def test_partial_directory_rejection_cleans_local_parse_artifact(monkeypatch, tmp_path):
+ from openviking.parse.output import LocalParseOutputStore
+ from openviking.utils.resource_processor import ResourceProcessor
+
+ fake_fs = _FakeVikingFS(exists_result=True)
+ monkeypatch.setattr(
+ "openviking.utils.resource_processor.get_current_telemetry",
+ lambda: _DummyTelemetry(),
+ )
+ _patch_viking_fs(monkeypatch, fake_fs)
+
+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))
+ ref = await store.create_artifact()
+ await store.write_bytes(ref, "repo/kept.py", b"pass")
+ parse_result = SimpleNamespace(
+ temp_dir_path=ref.root,
+ source_path="repo",
+ source_format="repository",
+ meta={
+ "file_count": 1,
+ "total_processable": 2,
+ "processed_files": [{"path": "kept.py"}],
+ "failed_files": [{"path": "missing.py", "error": "failed"}],
+ },
+ warnings=[],
+ artifact_ref=ref,
+ ensure_artifact_ref=lambda: ref,
+ )
+ rp = ResourceProcessor(vikingdb=_DummyVikingDB(), media_storage=None)
+ rp._build_parse_output_store = MagicMock(return_value=store)
+ rp._get_media_processor = MagicMock()
+ rp._get_media_processor.return_value.process = AsyncMock(return_value=parse_result)
+ rp.tree_builder.finalize_from_temp = AsyncMock()
+
+ result = await rp.process_resource(path="repo", ctx=object(), build_index=True)
+
+ assert result["status"] == "error"
+ assert not tmp_path.joinpath("artifacts", ref.root.split("/")[-1]).exists()
+ assert fake_fs.delete_temp_calls == []
+ rp.tree_builder.finalize_from_temp.assert_not_awaited()
+
+
+def test_directory_parse_failures_include_failed_source_access_items():
+ from openviking.utils.resource_processor import ResourceProcessor
+
+ failures = ResourceProcessor._directory_parse_failures(
+ {
+ "failed_files": [
+ {"path": "broken.pdf", "error": "invalid PDF"},
+ {"path": "denied.docx", "status": "failed", "reason": "HTTP 403"},
+ {"path": "binary.bin", "status": "unsupported", "reason": "binary"},
+ {
+ "path": "unsupported.csv",
+ "status": "unsupported",
+ "error": "unsupported type",
+ },
+ {"path": "ignored.tmp", "status": "skipped", "error": "ignored"},
+ ]
+ }
+ )
+
+ assert failures == [
+ {"path": "broken.pdf", "error": "invalid PDF"},
+ {"path": "denied.docx", "status": "failed", "reason": "HTTP 403"},
+ ]
+
+
@pytest.mark.parametrize("length", [119, 120, 121, 240])
def test_empty_directory_error_keeps_120_character_limit(length):
from openviking.utils.resource_processor import ResourceProcessor
@@ -277,6 +428,9 @@ async def test_resource_processor_first_add_summarizes_from_committed_uri(monkey
root=SimpleNamespace(uri="viking://resources/root", temp_uri="viking://temp/root_tmp")
)
)
+ rp._commit_directory_artifact_with_plan = AsyncMock(
+ return_value=_directory_plan("viking://resources/root")
+ )
rp._summarizer = SimpleNamespace(
summarize=AsyncMock(
side_effect=lambda *args, **kwargs: (
@@ -289,14 +443,80 @@ async def test_resource_processor_first_add_summarizes_from_committed_uri(monkey
assert result["status"] == "success"
assert result["root_uri"] == "viking://resources/root"
- assert fake_fs.persist_calls == [
- ("viking://temp/root_tmp", "viking://resources/root", {"id": "lock-1"})
- ]
+ assert fake_fs.persist_calls == []
assert fake_fs.delete_temp_calls == [("viking://temp/tmpdir", None)]
assert summarize_calls[0]["temp_uris"] == ["viking://resources/root"]
assert summarize_calls[0]["target_preexisting"] is False
+async def test_directory_semantic_ingest_commits_plan_before_summarizer(monkeypatch):
+ from openviking.parse.output import ParseArtifactRef
+ from openviking.storage.context_update_plan import (
+ ContextUpdatePlan,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+ )
+ from openviking.utils.resource_processor import ResourceProcessor
+
+ fake_fs = _FakeVikingFS()
+ monkeypatch.setattr(
+ "openviking.utils.resource_processor.get_current_telemetry",
+ lambda: _DummyTelemetry(),
+ )
+ _patch_viking_fs(monkeypatch, fake_fs)
+ ref = ParseArtifactRef(
+ backend="agfs",
+ root="viking://temp/tmpdir",
+ resource_rel="root_tmp",
+ )
+ parse_result = SimpleNamespace(
+ temp_dir_path=ref.root,
+ source_path="x",
+ source_format="repository",
+ meta={},
+ warnings=[],
+ artifact_ref=ref,
+ ensure_artifact_ref=lambda: ref,
+ )
+ plan = SemanticPlan(
+ root_uri="viking://resources/root",
+ context_type="resource",
+ tree=SemanticTreeSnapshot(
+ entries=(
+ SemanticTreeEntry("", "directory", "added", "aggregate"),
+ SemanticTreeEntry("a.py", "file", "added", "generate", md5="a"),
+ )
+ ),
+ )
+ context_plan = ContextUpdatePlan(plan.root_uri, "resource", semantic_plan=plan)
+ rp = ResourceProcessor(vikingdb=_DummyVikingDB(), media_storage=None)
+ rp._get_media_processor = MagicMock()
+ rp._get_media_processor.return_value.process = AsyncMock(return_value=parse_result)
+ rp.tree_builder.finalize_from_temp = AsyncMock(
+ return_value=SimpleNamespace(
+ root=SimpleNamespace(
+ uri="viking://resources/root",
+ temp_uri="viking://temp/tmpdir/root_tmp",
+ ),
+ _root_is_file=False,
+ )
+ )
+ rp._commit_directory_artifact_with_plan = AsyncMock(return_value=context_plan)
+ summarize = AsyncMock(return_value={"status": "success", "enqueued_count": 1})
+ rp._summarizer = SimpleNamespace(summarize=summarize)
+
+ result = await rp.process_resource(path="x", ctx=object(), build_index=True)
+
+ assert result["status"] == "success"
+ assert fake_fs.persist_calls == []
+ kwargs = summarize.await_args.kwargs
+ assert SemanticPlan.from_dict(kwargs["semantic_plan"]) == plan
+ assert kwargs["temp_uris"] == ["viking://resources/root"]
+ assert kwargs.get("artifact_ref") is None
+
+
@pytest.mark.asyncio
@pytest.mark.parametrize("auto_candidate", [False, True])
async def test_resource_processor_allows_flat_root_only_for_single_no_split_source(
@@ -337,6 +557,11 @@ async def test_resource_processor_allows_flat_root_only_for_single_no_split_sour
_candidate_uri=root_uri if auto_candidate else None,
)
)
+ rp._commit_directory_artifact_with_plan = AsyncMock(
+ return_value=ContextUpdatePlan(
+ root_uri, "resource", file_refresh=FileRefreshIntent(root_uri, "new-md5")
+ )
+ )
rp._summarizer = SimpleNamespace(
summarize=AsyncMock(return_value={"status": "success"}),
refresh_file_parent=AsyncMock(return_value={"status": "success"}),
@@ -387,6 +612,9 @@ async def test_resource_processor_keeps_wrapper_for_directory_to_no_split(monkey
_root_is_file=False,
)
)
+ rp._commit_directory_artifact_with_plan = AsyncMock(
+ return_value=_directory_plan("viking://resources/0803_shendiao_01")
+ )
rp._summarizer = SimpleNamespace(summarize=AsyncMock(return_value={"status": "success"}))
result = await rp.process_resource(
@@ -431,6 +659,9 @@ async def test_resource_processor_second_add_preserves_temp_uri_for_incremental(
root=SimpleNamespace(uri="viking://resources/root", temp_uri="viking://temp/root_tmp")
)
rp.tree_builder.finalize_from_temp = AsyncMock(return_value=context_tree)
+ rp._commit_directory_artifact_with_plan = AsyncMock(
+ return_value=_directory_plan("viking://resources/root")
+ )
rp._summarizer = SimpleNamespace(
summarize=AsyncMock(
side_effect=lambda *args, **kwargs: (
@@ -443,7 +674,7 @@ async def test_resource_processor_second_add_preserves_temp_uri_for_incremental(
assert result["status"] == "success"
assert result["root_uri"] == "viking://resources/root"
- assert summarize_calls[0]["temp_uris"] == ["viking://temp/root_tmp"]
+ assert summarize_calls[0]["temp_uris"] == ["viking://resources/root"]
assert summarize_calls[0]["target_preexisting"] is True
assert fake_fs.persist_calls == []
@@ -479,6 +710,9 @@ async def test_resource_processor_auto_candidate_skips_existing_and_busy(monkeyp
_candidate_uri="viking://resources/root",
)
rp.tree_builder.finalize_from_temp = AsyncMock(return_value=context_tree)
+ rp._commit_directory_artifact_with_plan = AsyncMock(
+ side_effect=lambda **kwargs: _directory_plan(kwargs["root_uri"])
+ )
rp._summarizer = SimpleNamespace(
summarize=AsyncMock(
side_effect=lambda *args, **kwargs: (
@@ -503,6 +737,4 @@ async def test_resource_processor_auto_candidate_skips_existing_and_busy(monkeyp
assert fake_pathlock.acquired_tree_paths == ["/mock/resources/root_2"]
assert summarize_calls[0]["temp_uris"] == ["viking://resources/root_2"]
assert summarize_calls[0]["target_preexisting"] is False
- assert fake_fs.persist_calls == [
- ("viking://temp/root_tmp", "viking://resources/root_2", {"id": "lock-1"})
- ]
+ assert fake_fs.persist_calls == []
diff --git a/tests/parse/test_add_directory.py b/tests/parse/test_add_directory.py
index 542c7db51..cc8153e3e 100644
--- a/tests/parse/test_add_directory.py
+++ b/tests/parse/test_add_directory.py
@@ -24,6 +24,11 @@ from openviking.parse.base import (
ResourceNode,
create_parse_result,
)
+from openviking.parse.output import (
+ ARTIFACT_MANIFEST_NAME,
+ LocalParseOutputStore,
+ read_artifact_manifest,
+)
from openviking.parse.parsers.base_parser import BaseParser
from openviking.parse.parsers.directory import DirectoryParser
@@ -57,7 +62,7 @@ class FakeVikingFS:
content = content.encode("utf-8")
self.files[uri] = content
- async def write_file_bytes(self, uri: str, content: bytes) -> None:
+ async def write_file_bytes(self, uri: str, content: bytes, **kw) -> None:
self.files[uri] = content
# ---- read / list operations ------------------------------------------
@@ -109,7 +114,7 @@ class FakeVikingFS:
# ---- temp URI --------------------------------------------------------
- def create_temp_uri(self) -> str:
+ def create_temp_uri(self, ctx: Any = None) -> str:
self._temp_counter += 1
return f"viking://temp/dir_{self._temp_counter}"
@@ -217,12 +222,8 @@ class TestDirectoryParserBasic:
(tmp_path / "notes").mkdir()
(tmp_path / "08_Attachments").mkdir()
(tmp_path / "notes" / "article.md").write_text("keep", encoding="utf-8")
- (tmp_path / "notes" / "private.excalidraw.md").write_text(
- "exclude", encoding="utf-8"
- )
- (tmp_path / "08_Attachments" / "diagram.md").write_text(
- "ignore", encoding="utf-8"
- )
+ (tmp_path / "notes" / "private.excalidraw.md").write_text("exclude", encoding="utf-8")
+ (tmp_path / "08_Attachments" / "diagram.md").write_text("ignore", encoding="utf-8")
(tmp_path / "main.py").write_text("exclude by include", encoding="utf-8")
with (
@@ -236,7 +237,9 @@ class TestDirectoryParserBasic:
exclude="*.excalidraw.md",
)
- uploaded_paths = {uri.split("/repository/", 1)[-1] for uri in fake_fs.files}
+ uploaded_paths = {
+ uri.split("/repository/", 1)[-1] for uri in fake_fs.files if "/repository/" in uri
+ }
assert result.parser_name == "CodeRepositoryParser"
assert uploaded_paths == {"notes/article.md"}
@@ -330,6 +333,8 @@ class TestDirectWriteFiles:
dir_name = tmp_code.name
for uri in fake_fs.files:
+ if uri.endswith(ARTIFACT_MANIFEST_NAME):
+ continue
assert f"/{dir_name}/" in uri
@pytest.mark.asyncio
@@ -344,6 +349,27 @@ class TestDirectWriteFiles:
else:
pytest.fail("hello.py not found in uploaded files")
+ @pytest.mark.asyncio
+ async def test_local_store_merges_parsed_and_direct_files(self, tmp_path: Path) -> None:
+ source = tmp_path / "source"
+ source.mkdir()
+ (source / "guide.md").write_text("# Guide\n\nbody", encoding="utf-8")
+ (source / "main.py").write_text("print('ok')", encoding="utf-8")
+ store = LocalParseOutputStore(str(tmp_path / "artifacts"))
+
+ result = await DirectoryParser().parse(source, parse_output_store=store)
+
+ assert result.artifact_ref is not None
+ assert result.artifact_ref.backend == "local"
+ assert result.artifact_ref.resource_rel == "source"
+ root = Path(result.artifact_ref.root) / "source"
+ assert (root / "main.py").read_text() == "print('ok')"
+ assert (root / "guide" / "guide.md").read_text() == "# Guide\n\nbody"
+ assert set(await read_artifact_manifest(store, result.artifact_ref)) == {
+ "source/main.py",
+ "source/guide/guide.md",
+ }
+
# ---------------------------------------------------------------------------
# Tests: nested directory structure
@@ -373,7 +399,8 @@ class TestNestedDirectory:
@pytest.mark.asyncio
async def test_file_count(self, tmp_nested_code: Path, parser, fake_fs) -> None:
await parser.parse(str(tmp_nested_code))
- assert len(fake_fs.files) == 4
+ business_files = [uri for uri in fake_fs.files if not uri.endswith(ARTIFACT_MANIFEST_NAME)]
+ assert len(business_files) == 4
# ---------------------------------------------------------------------------
@@ -428,9 +455,7 @@ class TestParserDelegation:
) -> None:
nested = tmp_path / "scripts"
nested.mkdir()
- content = "\n\n".join(
- f"paragraph {index} " + "x" * 1000 for index in range(20)
- )
+ content = "\n\n".join(f"paragraph {index} " + "x" * 1000 for index in range(20))
(nested / "screenplay.md").write_text(content, encoding="utf-8")
result = await parser.parse(str(tmp_path), split_content=False)
@@ -441,9 +466,7 @@ class TestParserDelegation:
for uri, value in fake_fs.files.items()
if uri.startswith(root) and uri.endswith(".md")
}
- assert body_files == {
- f"{root}/scripts/screenplay.md": content.encode("utf-8")
- }
+ assert body_files == {f"{root}/scripts/screenplay.md": content.encode("utf-8")}
@pytest.mark.asyncio
async def test_txt_file_goes_through_parser(self, tmp_path: Path, parser, fake_fs) -> None:
diff --git a/tests/parse/test_code_local_artifact.py b/tests/parse/test_code_local_artifact.py
new file mode 100644
index 000000000..c8fa30447
--- /dev/null
+++ b/tests/parse/test_code_local_artifact.py
@@ -0,0 +1,160 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Local-artifact-mode parsing for CodeRepositoryParser.
+
+When a parse_output_store is threaded through parse kwargs (local mode), the
+repository parser must write its artifacts into that store and never touch the
+VikingFS singleton (AGFS temp). Without a store it keeps the legacy AGFS path.
+"""
+
+from pathlib import Path
+
+import pytest
+
+from openviking.parse.output import LocalParseOutputStore
+from openviking.parse.parsers.code.code import CodeRepositoryParser
+
+
+def _make_repo(tmp_path: Path) -> Path:
+ repo = tmp_path / "repo"
+ (repo / "src").mkdir(parents=True)
+ (repo / "src" / "main.py").write_text("print('hi')", encoding="utf-8")
+ (repo / "README.md").write_text("# demo", encoding="utf-8")
+ # Minimal git marker so DirectoryParser-style detection is consistent; the
+ # parser itself accepts a plain local dir.
+ (repo / ".git").mkdir()
+ (repo / ".git" / "HEAD").write_text("ref: refs/heads/main", encoding="utf-8")
+ return repo
+
+
+class _ExplodingVikingFS:
+ """Any write here means the local path leaked back to AGFS temp."""
+
+ def create_temp_uri(self, ctx=None):
+ raise AssertionError("local mode must not allocate an AGFS temp uri")
+
+ async def write_file_bytes(self, uri, content, **kwargs):
+ raise AssertionError("local mode must not write VikingFS")
+
+ async def mkdir(self, uri, exist_ok=False, **kwargs):
+ raise AssertionError("local mode must not mkdir on VikingFS")
+
+
+class _FailingLocalStore(LocalParseOutputStore):
+ async def write_bytes(self, ref, rel_path, content):
+ if rel_path.endswith("src/main.py"):
+ raise OSError("injected artifact write failure")
+ await super().write_bytes(ref, rel_path, content)
+
+
+async def test_code_parse_writes_to_local_store(tmp_path, monkeypatch):
+ repo = _make_repo(tmp_path)
+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))
+
+ parser = CodeRepositoryParser()
+ monkeypatch.setattr(parser, "_get_viking_fs", lambda: _ExplodingVikingFS())
+
+ result = await parser.parse(
+ str(repo),
+ _source_meta={"repo_name": "acme/demo"},
+ parse_output_store=store,
+ )
+
+ # Artifact ref points at the local backend; temp_dir_path is the local root.
+ assert result.artifact_ref is not None
+ assert result.artifact_ref.backend == "local"
+ assert result.temp_dir_path == result.artifact_ref.root
+
+ # Files landed in the store under repository/... and match source bytes.
+ ref = result.artifact_ref
+ names = {e.rel_path for e in await store.list(ref, "repository")}
+ assert "repository/README.md" in names
+ assert await store.read_bytes(ref, "repository/src/main.py") == b"print('hi')"
+
+
+async def test_code_parse_rejects_and_cleans_partial_local_artifact(tmp_path, monkeypatch):
+ repo = _make_repo(tmp_path)
+ artifact_root = tmp_path / "artifacts"
+ store = _FailingLocalStore(local_root=str(artifact_root))
+ parser = CodeRepositoryParser()
+ monkeypatch.setattr(parser, "_get_viking_fs", lambda: _ExplodingVikingFS())
+
+ result = await parser.parse(
+ str(repo),
+ _source_meta={"repo_name": "acme/demo"},
+ parse_output_store=store,
+ )
+
+ assert result.temp_dir_path is None
+ assert result.artifact_ref is None
+ assert result.warnings == [
+ "Failed to parse repository: Failed to upload 1 repository file(s): "
+ f"Failed to upload {repo / 'src/main.py'}: injected artifact write failure"
+ ]
+ assert list(artifact_root.iterdir()) == []
+
+
+async def test_code_parse_without_store_uses_agfs(tmp_path):
+ # No store supplied -> legacy AGFS path. We only assert it does not raise the
+ # local-mode branch and yields an AGFS-shaped temp_dir_path via a fake fs.
+ repo = _make_repo(tmp_path)
+
+ class _FakeAgfs:
+ def __init__(self):
+ self.files = {}
+ self._n = 0
+
+ def create_temp_uri(self, ctx=None):
+ self._n += 1
+ return f"viking://temp/code{self._n}"
+
+ async def mkdir(self, uri, exist_ok=False, **kwargs):
+ pass
+
+ async def write_file_bytes(self, uri, content, **kwargs):
+ self.files[uri] = content
+
+ parser = CodeRepositoryParser()
+ fake = _FakeAgfs()
+ parser._get_viking_fs = lambda: fake # type: ignore[method-assign]
+
+ result = await parser.parse(str(repo), _source_meta={"repo_name": "acme/demo"})
+
+ assert result.artifact_ref is None or result.artifact_ref.backend == "agfs"
+ assert result.temp_dir_path.startswith("viking://temp/")
+ assert any("/repository/" in uri for uri in fake.files)
+
+
+async def test_code_parse_rejects_and_cleans_partial_agfs_artifact(tmp_path):
+ repo = _make_repo(tmp_path)
+
+ class _FailingAgfs:
+ def __init__(self):
+ self.deleted = []
+
+ def create_temp_uri(self, ctx=None):
+ return "viking://temp/code-failed"
+
+ async def mkdir(self, uri, exist_ok=False, **kwargs):
+ return None
+
+ async def write_file_bytes(self, uri, content, **kwargs):
+ if uri.endswith("src/main.py"):
+ raise OSError("injected AGFS write failure")
+
+ async def delete_temp(self, uri, **kwargs):
+ self.deleted.append(uri)
+
+ parser = CodeRepositoryParser()
+ fake = _FailingAgfs()
+ parser._get_viking_fs = lambda: fake # type: ignore[method-assign]
+
+ result = await parser.parse(str(repo), _source_meta={"repo_name": "acme/demo"})
+
+ assert result.temp_dir_path is None
+ assert "injected AGFS write failure" in result.warnings[0]
+ assert fake.deleted == ["viking://temp/code-failed"]
diff --git a/tests/parse/test_directory_understanding_routing.py b/tests/parse/test_directory_understanding_routing.py
index 17365c91f..c4230754c 100644
--- a/tests/parse/test_directory_understanding_routing.py
+++ b/tests/parse/test_directory_understanding_routing.py
@@ -44,6 +44,10 @@ class _FakeVikingFS:
async def write_file_bytes(self, uri: str, content: bytes) -> None:
self.files[uri] = content
+ async def delete_temp(self, uri: str) -> None:
+ prefix = uri.rstrip("/") + "/"
+ self.files = {key: value for key, value in self.files.items() if not key.startswith(prefix)}
+
def _configure_understanding(
monkeypatch,
@@ -542,7 +546,13 @@ async def test_direct_upload_failure_includes_file_error(
video = tmp_path / "video.mp4"
video.write_bytes(b"01234567")
fake_fs = _FakeVikingFS()
- fake_fs.write_file = AsyncMock(side_effect=OSError("storage rejected video.mp4"))
+
+ async def write_file_bytes(uri, content):
+ if uri.endswith("video.mp4"):
+ raise OSError("storage rejected video.mp4")
+ fake_fs.files[uri] = content
+
+ fake_fs.write_file_bytes = AsyncMock(side_effect=write_file_bytes)
with patch.object(BaseParser, "_get_viking_fs", return_value=fake_fs):
result = await DirectoryParser().parse(str(tmp_path), strict=True)
diff --git a/tests/parse/test_document_parser_threading.py b/tests/parse/test_document_parser_threading.py
index e54d5021f..4db990a3c 100644
--- a/tests/parse/test_document_parser_threading.py
+++ b/tests/parse/test_document_parser_threading.py
@@ -9,7 +9,10 @@ from typing import Any, Callable
import pytest
from openviking.parse.base import NodeType, ResourceNode, create_parse_result
-from openviking.parse.parsers import anydoc, anydoc_converter
+from openviking.parse.output import LocalParseOutputStore
+from openviking.parse.parsers import anydoc, anydoc_converter, pdf
+from openviking.parse.parsers.html import HTMLParser
+from openviking.parse.parsers.text import TextParser
from openviking_cli.utils.config.parser_config import AnydocConfig
@@ -84,3 +87,40 @@ async def test_anydoc_parser_offloads_docx_conversion(monkeypatch, tmp_path: Pat
assert seen["kwargs"]["allowed_media_dirs"] == [storage.media_dir]
assert result.source_format == "docx"
assert result.parser_name == "AnyDocParser"
+
+
+async def test_pdf_parser_forwards_parse_output_store_to_markdown(monkeypatch, tmp_path: Path):
+ parser = pdf.PDFParser()
+ markdown_parser = parser._get_markdown_parser()
+ seen = _stub_markdown_parse(SimpleNamespace(_md_parser=markdown_parser))
+ source = tmp_path / "sample.pdf"
+ source.write_bytes(b"%PDF-placeholder")
+ store = LocalParseOutputStore(str(tmp_path / "artifacts"))
+
+ async def convert(_path, *, resource_name=None):
+ return "# converted pdf", {}
+
+ monkeypatch.setattr(parser, "_convert_to_markdown", convert)
+
+ await parser.parse(source, parse_output_store=store)
+
+ assert seen["kwargs"]["parse_output_store"] is store
+
+
+ "parser",
+ [TextParser(), HTMLParser(), anydoc.AnyDocParser()],
+)
+async def test_markdown_delegate_parsers_write_local_artifacts(tmp_path: Path, parser) -> None:
+ store = LocalParseOutputStore(str(tmp_path / type(parser).__name__))
+
+ result = await parser.parse_content(
+ "# title\n\nbody",
+ source_path=str(tmp_path / "document.md"),
+ parse_output_store=store,
+ )
+
+ assert result.artifact_ref is not None
+ assert result.artifact_ref.backend == "local"
diff --git a/tests/parse/test_feishu_parser_api.py b/tests/parse/test_feishu_parser_api.py
index eff5ccd83..9b22a6954 100644
--- a/tests/parse/test_feishu_parser_api.py
+++ b/tests/parse/test_feishu_parser_api.py
@@ -1071,28 +1071,30 @@ async def test_add_resource_processor_persists_final_uri_and_cleans_staged_sourc
"resource_id": final_uri,
}
final_complete = task_tracker.complete.await_args_list[-1]
- assert final_complete.args == (
- "task-1",
- {
- "status": "success",
- "root_uri": final_uri,
- "context_count": 9,
- "queue_status": {
- "Semantic": {
- "processed": 0,
- "requeue_count": 0,
- "error_count": 0,
- "errors": [],
- },
- "Embedding": {
- "processed": 9,
- "requeue_count": 0,
- "error_count": 0,
- "errors": [],
- },
+ final_result = dict(final_complete.args[1])
+ telemetry = final_result.pop("telemetry")
+ assert telemetry["id"] == telemetry_id
+ assert telemetry["summary"]["operation"] == "add_resource_job"
+ assert final_complete.args[0] == "task-1"
+ assert final_result == {
+ "status": "success",
+ "root_uri": final_uri,
+ "context_count": 9,
+ "queue_status": {
+ "Semantic": {
+ "processed": 0,
+ "requeue_count": 0,
+ "error_count": 0,
+ "errors": [],
+ },
+ "Embedding": {
+ "processed": 9,
+ "requeue_count": 0,
+ "error_count": 0,
+ "errors": [],
},
},
- )
+ }
assert final_complete.kwargs == {
"account_id": "account-1",
"user_id": "user-1",
diff --git a/tests/parse/test_feishu_unified_import.py b/tests/parse/test_feishu_unified_import.py
index 7884608e8..55cd8f8d3 100644
--- a/tests/parse/test_feishu_unified_import.py
+++ b/tests/parse/test_feishu_unified_import.py
@@ -107,7 +107,6 @@ async def test_directory_executes_url_with_auth_and_preserves_output_parent(
fs = FakeVikingFS()
parser = DirectoryParser()
monkeypatch.setattr(parser, "_get_viking_fs", lambda: fs)
- monkeypatch.setattr(parser, "_create_temp_uri", lambda: "viking://temp/import")
calls = []
async def parse_api(self, source, **options):
@@ -143,8 +142,8 @@ async def test_directory_executes_url_with_auth_and_preserves_output_parent(
assert calls[0][1].get("understanding_response_id") == ("old" if resume else None)
save.assert_awaited_once_with(plan.entries[0].checkpoint_key(plan.root), "response-1")
assert result.meta["failed_files"] == []
- assert "viking://temp/import/Folder/nested/Cloud/0.md" in fs.files
- assert "viking://temp/import/Folder/empty" not in fs.dirs
+ assert f"{result.temp_dir_path}/Folder/nested/Cloud/0.md" in fs.files
+ assert f"{result.temp_dir_path}/Folder/empty" not in fs.dirs
assert "secret" not in str(result.meta)
@@ -172,7 +171,6 @@ async def test_directory_omits_source_directories_without_imported_content(
fs = FakeVikingFS()
parser = DirectoryParser()
monkeypatch.setattr(parser, "_get_viking_fs", lambda: fs)
- monkeypatch.setattr(parser, "_create_temp_uri", lambda: "viking://temp/empty-filter")
parse = AsyncMock(side_effect=ValueError("document parse failed"))
monkeypatch.setattr(DirectoryParser, "_parse_file_with_parser", parse)
result = await parser.parse(
@@ -186,7 +184,7 @@ async def test_directory_omits_source_directories_without_imported_content(
assert result.meta["file_count"] == 1
assert len(result.meta["failed_files"]) == 1
assert result.meta["failed_files"][0]["path"] == "failed/Broken.md"
- target = "viking://temp/empty-filter/Folder"
+ target = f"{result.temp_dir_path}/Folder"
assert fs.files[f"{target}/kept/nested/data.json"] == b'{"body": "keep"}'
assert [entry["name"] for entry in await fs.ls(target)] == ["kept"]
@@ -199,7 +197,6 @@ async def test_directory_virtual_url_obeys_include_filter(monkeypatch, tmp_path)
parser = DirectoryParser()
fs = FakeVikingFS()
monkeypatch.setattr(parser, "_get_viking_fs", lambda: fs)
- monkeypatch.setattr(parser, "_create_temp_uri", lambda: "viking://temp/filter")
parse = AsyncMock(side_effect=AssertionError("excluded URL was submitted"))
monkeypatch.setattr(UnderstandingAPI, "parse", parse)
result = await parser.parse(tmp_path, _feishu_import_plan=plan, include="*.pdf")
@@ -374,7 +371,6 @@ async def test_folder_processor_routes_each_content_and_keeps_tree(monkeypatch,
processor._accessor_registry = SimpleNamespace(access=access)
fs = FakeVikingFS()
monkeypatch.setattr(DirectoryParser, "_get_viking_fs", lambda self: fs)
- monkeypatch.setattr(DirectoryParser, "_create_temp_uri", lambda self: "viking://temp/tree")
calls = []
async def parse_api(self, source, **options):
@@ -414,9 +410,9 @@ async def test_folder_processor_routes_each_content_and_keeps_tree(monkeypatch,
)
assert "feishu_access_token" not in binary_options
assert "lark_file" not in binary_options
- assert "viking://temp/tree/Root/Root/0.md" in fs.files
- assert "viking://temp/tree/Root/Report/Leaf/0.md" in fs.files
- assert "viking://temp/tree/Root/Report/Report/0.md" in fs.files
+ assert f"{result.temp_dir_path}/Root/Root/0.md" in fs.files
+ assert f"{result.temp_dir_path}/Root/Report/Leaf/0.md" in fs.files
+ assert f"{result.temp_dir_path}/Root/Report/Report/0.md" in fs.files
finally:
for resource in resources:
resource.is_temporary = True
@@ -568,7 +564,6 @@ async def test_directory_failure_cleans_unmerged_artifacts(
fs = FakeVikingFS()
parser = DirectoryParser()
monkeypatch.setattr(parser, "_get_viking_fs", lambda: fs)
- monkeypatch.setattr(parser, "_create_temp_uri", lambda: "viking://temp/import")
async def parse_api(self, source, **options):
name = source.rsplit("/", 1)[-1]
@@ -830,7 +825,6 @@ async def test_recursive_wiki_processor_routes_each_content_and_keeps_tree(monke
processor._accessor_registry = SimpleNamespace(access=access)
fs = FakeVikingFS()
monkeypatch.setattr(DirectoryParser, "_get_viking_fs", lambda self: fs)
- monkeypatch.setattr(DirectoryParser, "_create_temp_uri", lambda self: "viking://temp/tree")
calls = []
async def parse_api(self, source, **options):
@@ -871,9 +865,9 @@ async def test_recursive_wiki_processor_routes_each_content_and_keeps_tree(monke
)
assert "feishu_access_token" not in binary_options
assert "lark_file" not in binary_options
- assert "viking://temp/tree/Root/Root/0.md" in fs.files
- assert "viking://temp/tree/Root/Report/Leaf/0.md" in fs.files
- assert "viking://temp/tree/Root/Report/Report/0.md" in fs.files
+ assert f"{result.temp_dir_path}/Root/Root/0.md" in fs.files
+ assert f"{result.temp_dir_path}/Root/Report/Leaf/0.md" in fs.files
+ assert f"{result.temp_dir_path}/Root/Report/Report/0.md" in fs.files
finally:
for resource in resources:
resource.is_temporary = True
diff --git a/tests/parse/test_markdown_apply_layout.py b/tests/parse/test_markdown_apply_layout.py
index a356a14ca..4ae43c836 100644
--- a/tests/parse/test_markdown_apply_layout.py
+++ b/tests/parse/test_markdown_apply_layout.py
@@ -9,8 +9,10 @@ isolated while replaying their layouts.
"""
import asyncio
+from pathlib import Path
from unittest.mock import patch
+from openviking.parse.output import LocalParseOutputStore, read_artifact_manifest
from openviking.parse.parsers.base_parser import BaseParser
from openviking.parse.parsers.markdown import MarkdownParser, _Layout, _LayoutOp
@@ -123,3 +125,20 @@ class TestApplyLayout:
for name, result in zip(("a", "b"), results, strict=True):
uri = f"{result.temp_dir_path}/{name}/{name}.md"
assert fake.files[uri] == f"[target](../{name}.txt)"
+
+
+async def test_markdown_parser_writes_local_artifact_and_manifest(tmp_path: Path):
+ store = LocalParseOutputStore(str(tmp_path / "artifacts"))
+ parser = MarkdownParser()
+
+ result = await parser.parse_content(
+ "# Hello\n\nworld",
+ source_path=str(tmp_path / "hello.md"),
+ parse_output_store=store,
+ )
+
+ assert result.artifact_ref is not None
+ assert result.artifact_ref.backend == "local"
+ assert result.artifact_ref.resource_rel == "hello"
+ assert Path(result.artifact_ref.root, "hello", "hello.md").read_text() == "# Hello\n\nworld"
+ assert set(await read_artifact_manifest(store, result.artifact_ref)) == {"hello/hello.md"}
diff --git a/tests/parse/test_markdown_no_split.py b/tests/parse/test_markdown_no_split.py
index 1f56afed7..bc48b696f 100644
--- a/tests/parse/test_markdown_no_split.py
+++ b/tests/parse/test_markdown_no_split.py
@@ -69,7 +69,7 @@ async def test_flattened_no_split_markdown_creates_temp_root_once(
assert result.warnings == []
assert fake_fs.directories == {"viking://temp/pdf-no-split"}
- assert fake_fs.files == {"viking://temp/pdf-no-split/document.md": content}
+ assert fake_fs.files["viking://temp/pdf-no-split/document.md"] == content
@pytest.mark.asyncio
@@ -142,7 +142,8 @@ async def test_pdf_no_split_converts_to_one_complete_markdown(
)
assert result.parser_name == "PDFParser"
- assert fake_fs.files == {
- ("viking://temp/pdf-no-split/社交网络中英文剧本/社交网络中英文剧本.md"): content
- }
+ assert (
+ fake_fs.files["viking://temp/pdf-no-split/社交网络中英文剧本/社交网络中英文剧本.md"]
+ == content
+ )
assert not any(uri.endswith(".pdf") for uri in fake_fs.files)
diff --git a/tests/parse/test_media_resource_name.py b/tests/parse/test_media_resource_name.py
index bd189d716..e97f96607 100644
--- a/tests/parse/test_media_resource_name.py
+++ b/tests/parse/test_media_resource_name.py
@@ -17,8 +17,11 @@ from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from PIL import Image
+from openviking.parse.output import LocalParseOutputStore, read_artifact_manifest
+from openviking.parse.parsers.media.audio import AudioParser
from openviking.parse.parsers.media.image import ImageParser
from openviking.parse.parsers.media.naming import resolve_media_names
+from openviking.parse.parsers.media.video import VideoParser
from openviking_cli.utils.config.parser_config import ImageConfig
@@ -143,3 +146,88 @@ async def test_image_parser_falls_back_to_temp_name(tmp_path):
result = await parser.parse(str(upload))
assert result.root.meta["original_filename"] == "upload_0123456789abcdef.png"
+
+
+async def test_image_parser_writes_local_artifact(tmp_path):
+ upload = tmp_path / "upload.png"
+ content = _png_bytes()
+ upload.write_bytes(content)
+ store = LocalParseOutputStore(str(tmp_path / "artifacts"))
+
+ result = await ImageParser(config=ImageConfig()).parse(
+ upload, resource_name="vacation", parse_output_store=store
+ )
+
+ assert result.artifact_ref is not None
+ assert result.artifact_ref.backend == "local"
+ rel = "vacation_png/vacation.png"
+ assert Path(result.artifact_ref.root, rel).read_bytes() == content
+ assert set(await read_artifact_manifest(store, result.artifact_ref)) == {rel}
+
+
+ ("parser", "filename", "content", "resource_rel"),
+ [
+ (AudioParser(), "clip.mp3", b"ID3audio", "clip_mp3"),
+ (VideoParser(), "clip.mp4", b"\x00\x00\x00\x18ftypvideo", "clip_mp4"),
+ ],
+)
+async def test_audio_and_video_write_local_artifacts(
+ tmp_path, parser, filename, content, resource_rel
+):
+ source = tmp_path / filename
+ source.write_bytes(content)
+ store = LocalParseOutputStore(str(tmp_path / f"artifacts-{resource_rel}"))
+
+ result = await parser.parse(source, parse_output_store=store)
+
+ assert result.artifact_ref is not None
+ assert result.artifact_ref.resource_rel == resource_rel
+ rel = f"{resource_rel}/{filename}"
+ assert Path(result.artifact_ref.root, rel).read_bytes() == content
+ assert set(await read_artifact_manifest(store, result.artifact_ref)) == {rel}
+
+
+async def test_image_parser_cleans_local_artifact_when_write_fails(tmp_path):
+ class FailingStore(LocalParseOutputStore):
+ async def write_bytes(self, ref, rel_path, content):
+ raise OSError("disk full")
+
+ upload = tmp_path / "upload.png"
+ upload.write_bytes(_png_bytes())
+ artifact_root = tmp_path / "artifacts"
+ store = FailingStore(str(artifact_root))
+
+ with pytest.raises(OSError, match="disk full"):
+ await ImageParser(config=ImageConfig()).parse(upload, parse_output_store=store)
+
+ assert list(artifact_root.iterdir()) == []
+
+
+ ("parser", "filename", "content"),
+ [
+ (AudioParser(), "clip.mp3", b"ID3audio"),
+ (VideoParser(), "clip.mp4", b"\x00\x00\x00\x18ftypvideo"),
+ ],
+)
+async def test_audio_and_video_clean_local_artifact_when_write_fails(
+ tmp_path, parser, filename, content
+):
+ class FailingStore(LocalParseOutputStore):
+ async def write_bytes(self, ref, rel_path, content):
+ raise OSError("disk full")
+
+ source = tmp_path / filename
+ source.write_bytes(content)
+ artifact_root = tmp_path / f"artifacts-{filename}"
+ store = FailingStore(str(artifact_root))
+
+ with pytest.raises(OSError, match="disk full"):
+ await parser.parse(source, parse_output_store=store)
+
+ assert list(artifact_root.iterdir()) == []
diff --git a/tests/parse/test_parse_output.py b/tests/parse/test_parse_output.py
new file mode 100644
index 000000000..8517d0ca4
--- /dev/null
+++ b/tests/parse/test_parse_output.py
@@ -0,0 +1,467 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Contract tests for the parse output store abstraction.
+
+The AGFS backend forwards 1:1 to the VikingFS singleton; the local backend
+lays artifacts out under a configured root directory. Both must satisfy the same
+byte/directory contract so parsers stay backend-agnostic.
+"""
+
+from pathlib import Path
+
+import pytest
+
+from openviking.parse.base import ParseResult, ResourceNode
+from openviking.parse.output import (
+ ARTIFACT_MANIFEST_NAME,
+ AgfsParseOutputStore,
+ LocalParseOutputStore,
+ ParseArtifactRef,
+ ParseArtifactWriter,
+ build_parse_output_store,
+ copy_artifact_tree,
+ read_artifact_manifest,
+ resolve_artifact_doc_root,
+)
+from openviking.utils.content_hash import content_md5
+
+
+class _FakeVikingFS:
+ """Records the VikingFS calls a parser would make while writing artifacts."""
+
+ def __init__(self) -> None:
+ self.files: dict[str, bytes] = {}
+ self.dirs: list[str] = []
+ self.deleted: list[str] = []
+ self._temp_seq = 0
+ self.ls_kwargs = []
+
+ def create_temp_uri(self, ctx=None) -> str:
+ self._temp_seq += 1
+ return f"viking://temp/fake{self._temp_seq}"
+
+ async def mkdir(self, uri: str, exist_ok: bool = False, ctx=None) -> None:
+ self.dirs.append(uri)
+
+ async def write_file_bytes(self, uri: str, content: bytes, ctx=None) -> None:
+ self.files[uri] = content
+
+ async def read_file_bytes(self, uri: str, ctx=None) -> bytes:
+ return self.files[uri]
+
+ async def ls(self, uri: str, ctx=None, **kwargs) -> list[dict]:
+ self.ls_kwargs.append(kwargs)
+ prefix = f"{uri.rstrip('/')}/"
+ entries = {}
+ for stored in self.files:
+ if stored.startswith(prefix):
+ relative = stored[len(prefix) :]
+ name, separator, _ = relative.partition("/")
+ entries[name] = {
+ "name": name,
+ "uri": f"{prefix}{name}",
+ "isDir": bool(separator),
+ }
+ return list(entries.values())
+
+ async def delete_temp(self, uri: str, ctx=None) -> None:
+ self.deleted.append(uri)
+
+
+# ---------------------------------------------------------------------------
+# ParseArtifactRef
+# ---------------------------------------------------------------------------
+
+
+class TestParseArtifactRef:
+ def test_serialization_roundtrip(self) -> None:
+ ref = ParseArtifactRef(
+ backend="agfs",
+ root="viking://temp/abc",
+ resource_rel="repository",
+ root_type="dir",
+ )
+ assert ParseArtifactRef.from_dict(ref.to_dict()) == ref
+
+ def test_rejects_unknown_backend(self) -> None:
+ with pytest.raises(ValueError, match="backend"):
+ ParseArtifactRef.from_dict(
+ {"backend": "s3", "root": "x", "resource_rel": "", "root_type": "dir"}
+ )
+
+ def test_rejects_invalid_root_type(self) -> None:
+ with pytest.raises(ValueError, match="root_type"):
+ ParseArtifactRef.from_dict(
+ {"backend": "agfs", "root": "x", "resource_rel": "", "root_type": "blob"}
+ )
+
+
+# ---------------------------------------------------------------------------
+# Shared contract, parametrized over both backends
+# ---------------------------------------------------------------------------
+
+
+def _make_store(backend: str, tmp_path):
+ if backend == "agfs":
+ return AgfsParseOutputStore(viking_fs=_FakeVikingFS())
+ return LocalParseOutputStore(local_root=str(tmp_path / "parse-out"))
+
+
[email protected]("backend", ["agfs", "local"])
+class TestParseOutputStoreContract:
+ async def test_create_artifact_reports_backend(self, backend, tmp_path) -> None:
+ store = _make_store(backend, tmp_path)
+ ref = await store.create_artifact(root_type="dir")
+ assert ref.backend == backend
+ assert ref.root_type == "dir"
+ assert ref.root
+
+ async def test_write_and_read_bytes_roundtrip(self, backend, tmp_path) -> None:
+ store = _make_store(backend, tmp_path)
+ ref = await store.create_artifact(root_type="dir")
+
+ await store.mkdir(ref, "repo")
+ await store.write_bytes(ref, "repo/main.py", b"print(1)")
+
+ assert await store.read_bytes(ref, "repo/main.py") == b"print(1)"
+
+ async def test_write_and_read_text(self, backend, tmp_path) -> None:
+ store = _make_store(backend, tmp_path)
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_text(ref, "note.md", "# hi")
+ assert await store.read_text(ref, "note.md") == "# hi"
+
+ async def test_list_returns_relative_entries(self, backend, tmp_path) -> None:
+ store = _make_store(backend, tmp_path)
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_bytes(ref, "a.py", b"a")
+ await store.write_bytes(ref, "b.py", b"b")
+
+ names = {entry.name for entry in await store.list(ref, "")}
+ assert names == {"a.py", "b.py"}
+
+ async def test_cleanup_is_idempotent(self, backend, tmp_path) -> None:
+ store = _make_store(backend, tmp_path)
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_bytes(ref, "a.py", b"a")
+
+ await store.cleanup(ref)
+ await store.cleanup(ref) # must not raise
+
+ async def test_rejects_unsafe_relative_path(self, backend, tmp_path) -> None:
+ store = _make_store(backend, tmp_path)
+ ref = await store.create_artifact(root_type="dir")
+ with pytest.raises(ValueError):
+ await store.write_bytes(ref, "../escape.py", b"x")
+
+ async def test_ref_roundtrip_reopens_same_artifact(self, backend, tmp_path) -> None:
+ store = _make_store(backend, tmp_path)
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_bytes(ref, "a.py", b"a")
+
+ # A serialized ref must reopen the same artifact bytes.
+ reopened = ParseArtifactRef.from_dict(ref.to_dict())
+ assert await store.read_bytes(reopened, "a.py") == b"a"
+
+
+# ---------------------------------------------------------------------------
+# AGFS-specific: must forward to the VikingFS singleton
+# ---------------------------------------------------------------------------
+
+
+class TestAgfsParseOutputStore:
+ async def test_explicit_context_is_forwarded(self) -> None:
+ class _ContextVikingFS(_FakeVikingFS):
+ def __init__(self):
+ super().__init__()
+ self.contexts = []
+
+ async def read_file_bytes(self, uri: str, ctx=None) -> bytes:
+ self.contexts.append(ctx)
+ return await super().read_file_bytes(uri, ctx=ctx)
+
+ ctx = object()
+ vfs = _ContextVikingFS()
+ store = AgfsParseOutputStore(viking_fs=vfs, ctx=ctx)
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_bytes(ref, "a.py", b"a")
+
+ assert await store.read_bytes(ref, "a.py") == b"a"
+ assert vfs.contexts == [ctx]
+
+ async def test_write_round_trips_through_vikingfs(self) -> None:
+ vfs = _FakeVikingFS()
+ store = AgfsParseOutputStore(viking_fs=vfs)
+ ref = await store.create_artifact(root_type="dir")
+
+ await store.write_bytes(ref, "repo/main.py", b"print(1)")
+
+ assert vfs.files["viking://temp/fake1/repo/main.py"] == b"print(1)"
+
+ async def test_list_requests_all_children(self) -> None:
+ from openviking.storage.viking_fs import LS_ALL_NODES
+
+ vfs = _FakeVikingFS()
+ store = AgfsParseOutputStore(viking_fs=vfs)
+ ref = await store.create_artifact(root_type="dir")
+
+ await store.list(ref)
+
+ assert vfs.ls_kwargs == [{"show_all_hidden": True, "node_limit": LS_ALL_NODES}]
+
+ async def test_cleanup_deletes_via_vikingfs(self) -> None:
+ vfs = _FakeVikingFS()
+ store = AgfsParseOutputStore(viking_fs=vfs)
+ ref = await store.create_artifact(root_type="dir")
+
+ await store.cleanup(ref)
+ await store.cleanup(ref)
+
+ assert vfs.deleted == ["viking://temp/fake1"]
+
+ async def test_move_forwards_explicit_context(self) -> None:
+ class _MovingVikingFS(_FakeVikingFS):
+ def __init__(self):
+ super().__init__()
+ self.moves = []
+
+ async def move_file(self, source, target, ctx=None):
+ self.moves.append((source, target, ctx))
+
+ ctx = object()
+ vfs = _MovingVikingFS()
+ store = AgfsParseOutputStore(viking_fs=vfs, ctx=ctx)
+ source = await store.create_artifact()
+ target = await store.create_artifact()
+
+ await store.move_file(source, "a.txt", target, "nested/a.txt")
+
+ assert vfs.moves == [(f"{source.root}/a.txt", f"{target.root}/nested/a.txt", ctx)]
+
+
+# ---------------------------------------------------------------------------
+# Local-specific: filesystem layout, isolation, case conflicts
+# ---------------------------------------------------------------------------
+
+
+class TestLocalParseOutputStore:
+ async def test_artifacts_are_isolated(self, tmp_path) -> None:
+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))
+ a = await store.create_artifact(root_type="dir")
+ b = await store.create_artifact(root_type="dir")
+ assert a.root != b.root
+
+ await store.write_bytes(a, "f.py", b"a")
+ await store.write_bytes(b, "f.py", b"b")
+ assert await store.read_bytes(a, "f.py") == b"a"
+ assert await store.read_bytes(b, "f.py") == b"b"
+
+ async def test_root_stays_under_configured_local_root(self, tmp_path) -> None:
+ root = tmp_path / "out"
+ store = LocalParseOutputStore(local_root=str(root))
+ ref = await store.create_artifact(root_type="dir")
+ assert str(root) in ref.root
+
+ async def test_cleanup_removes_directory(self, tmp_path) -> None:
+ from pathlib import Path
+
+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_bytes(ref, "a.py", b"a")
+ assert Path(ref.root).exists()
+
+ await store.cleanup(ref)
+ assert not Path(ref.root).exists()
+
+ async def test_cleanup_rejects_artifact_outside_store_root(self, tmp_path) -> None:
+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))
+ outside = tmp_path / "outside"
+ outside.mkdir()
+ ref = ParseArtifactRef(backend="local", root=str(outside))
+
+ with pytest.raises(ValueError, match="escapes"):
+ await store.cleanup(ref)
+
+ assert outside.exists()
+
+ async def test_case_only_conflict_is_detected(self, tmp_path) -> None:
+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_bytes(ref, "Readme.md", b"a")
+ with pytest.raises(ValueError, match="case"):
+ await store.write_bytes(ref, "README.MD", b"b")
+
+ async def test_missing_artifact_read_raises(self, tmp_path) -> None:
+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))
+ ref = await store.create_artifact(root_type="dir")
+ with pytest.raises(FileNotFoundError):
+ await store.read_bytes(ref, "missing.py")
+
+
+# ---------------------------------------------------------------------------
+# Factory
+# ---------------------------------------------------------------------------
+
+
+class TestBuildParseOutputStore:
+ def test_defaults_to_agfs_backend(self) -> None:
+ store = build_parse_output_store(viking_fs=_FakeVikingFS())
+ assert isinstance(store, AgfsParseOutputStore)
+
+ def test_local_backend_requires_root(self) -> None:
+ with pytest.raises(ValueError, match="local_root"):
+ build_parse_output_store(backend="local", local_root=None)
+
+ def test_builds_local_backend(self, tmp_path) -> None:
+ store = build_parse_output_store(backend="local", local_root=str(tmp_path))
+ assert isinstance(store, LocalParseOutputStore)
+
+
[email protected]("backend", ["agfs", "local"])
+async def test_artifact_writer_records_final_bytes_in_manifest(backend, tmp_path) -> None:
+ store = _make_store(backend, tmp_path)
+ writer = await ParseArtifactWriter.create(store)
+
+ await writer.write_bytes("doc/a.txt", b"final bytes")
+ ref = await writer.finalize(resource_rel="doc")
+
+ assert ref.resource_rel == "doc"
+ assert await read_artifact_manifest(store, ref) == {"doc/a.txt": content_md5(b"final bytes")}
+
+
+async def test_copy_artifact_tree_merges_files_without_manifest_sidecar(tmp_path) -> None:
+ source_store = LocalParseOutputStore(local_root=str(tmp_path / "source"))
+ target_store = LocalParseOutputStore(local_root=str(tmp_path / "target"))
+ source = await ParseArtifactWriter.create(source_store)
+ target = await ParseArtifactWriter.create(target_store)
+ await source.write_bytes("wrapper/a.txt", b"a")
+ await source.write_bytes("wrapper/nested/b.txt", b"b")
+ source_ref = await source.finalize(resource_rel="wrapper")
+
+ copied = await copy_artifact_tree(
+ source_store=source_store,
+ source_ref=source_ref,
+ source_rel="wrapper",
+ target=target,
+ target_rel="dest",
+ )
+ target_ref = await target.finalize(resource_rel="dest")
+
+ assert copied == 2
+ assert await target_store.read_bytes(target_ref, "dest/a.txt") == b"a"
+ assert await target_store.read_bytes(target_ref, "dest/nested/b.txt") == b"b"
+ assert await read_artifact_manifest(target_store, target_ref) == {
+ "dest/a.txt": content_md5(b"a"),
+ "dest/nested/b.txt": content_md5(b"b"),
+ }
+ assert all(
+ entry.name != ARTIFACT_MANIFEST_NAME
+ for entry in await target_store.list(target_ref, "dest")
+ )
+
+
+async def test_copy_artifact_tree_flattens_single_wrapped_file(tmp_path) -> None:
+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))
+ source = await ParseArtifactWriter.create(store)
+ target = await ParseArtifactWriter.create(store)
+ await source.write_text("wrapper/file.md", "body")
+ source_ref = await source.finalize(resource_rel="wrapper")
+
+ copied = await copy_artifact_tree(
+ source_store=store,
+ source_ref=source_ref,
+ source_rel="",
+ target=target,
+ target_rel="dest",
+ flatten_single_output=True,
+ )
+ target_ref = await target.finalize(resource_rel="dest")
+
+ assert copied == 1
+ assert await store.read_text(target_ref, "dest/file.md") == "body"
+ assert not Path(target_ref.root, "dest", "wrapper").exists()
+
+
+def test_parse_result_does_not_guess_local_path_is_agfs(tmp_path) -> None:
+ result = ParseResult(
+ root=ResourceNode(type="root"),
+ temp_dir_path=str(tmp_path / "artifact"),
+ )
+
+ with pytest.raises(ValueError, match="artifact_ref"):
+ result.ensure_artifact_ref()
+
+
+async def test_resolve_artifact_doc_root_accepts_flat_single_file(tmp_path) -> None:
+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))
+ writer = await ParseArtifactWriter.create(store)
+ await writer.write_text("document.md", "body")
+ ref = await writer.finalize()
+
+ resolved = await resolve_artifact_doc_root(store, ref, flatten_single_file=True)
+
+ assert resolved.doc_name == "document.md"
+ assert resolved.doc_rel == "document.md"
+ assert resolved.root_is_file is True
+
+
[email protected]("backend", ["agfs", "local"])
+async def test_resolve_artifact_doc_root_accepts_single_directory(backend, tmp_path) -> None:
+ store = _make_store(backend, tmp_path)
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_bytes(ref, "repo/a.py", b"a")
+ await store.write_bytes(ref, "repo/b.py", b"b")
+
+ resolved = await resolve_artifact_doc_root(store, ref, flatten_single_file=False)
+
+ assert resolved.doc_name == "repo"
+ assert resolved.doc_rel == "repo"
+ assert resolved.root_is_file is False
+
+
[email protected]("backend", ["agfs", "local"])
+async def test_resolve_artifact_doc_root_flattens_wrapped_single_file(backend, tmp_path) -> None:
+ store = _make_store(backend, tmp_path)
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_bytes(ref, "doc/report.md", b"# hi")
+
+ resolved = await resolve_artifact_doc_root(store, ref, flatten_single_file=True)
+
+ assert resolved.doc_name == "report.md"
+ assert resolved.doc_rel == "doc/report.md"
+ assert resolved.root_is_file is True
+
+
[email protected]("backend", ["agfs", "local"])
+async def test_resolve_artifact_doc_root_keeps_multi_file_directory(backend, tmp_path) -> None:
+ store = _make_store(backend, tmp_path)
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_bytes(ref, "doc/a.md", b"a")
+ await store.write_bytes(ref, "doc/b.md", b"b")
+
+ resolved = await resolve_artifact_doc_root(store, ref, flatten_single_file=True)
+
+ assert resolved.doc_name == "doc"
+ assert resolved.root_is_file is False
+
+
[email protected]("backend", ["agfs", "local"])
+async def test_resolve_artifact_doc_root_rejects_multiple_roots(backend, tmp_path) -> None:
+ store = _make_store(backend, tmp_path)
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_bytes(ref, "one/a.py", b"a")
+ await store.write_bytes(ref, "two/b.py", b"b")
+
+ with pytest.raises(ValueError, match="document"):
+ await resolve_artifact_doc_root(store, ref, flatten_single_file=False)
diff --git a/tests/parse/test_understanding_api_artifact_images.py b/tests/parse/test_understanding_api_artifact_images.py
index b9317e799..e70e11555 100644
--- a/tests/parse/test_understanding_api_artifact_images.py
+++ b/tests/parse/test_understanding_api_artifact_images.py
@@ -9,6 +9,7 @@ from openviking.parse.image_rewrite import (
IMAGE_MAPPINGS_FILENAME,
build_artifact_image_mappings,
)
+from openviking.parse.output import LocalParseOutputStore, read_artifact_manifest
from openviking.parse.understanding_api import UnderstandingAPI
@@ -76,14 +77,15 @@ async def test_unpack_artifact_writes_image_mapping_sidecar(tmp_path: Path):
fake_fs = _FakeVikingFS()
api = UnderstandingAPI.__new__(UnderstandingAPI)
with patch("openviking.parse.understanding_api.get_viking_fs", return_value=fake_fs):
- temp_uri = await api._unpack_zip_to_temp_dir(zip_path, "resource")
+ artifact_ref = await api._unpack_zip_to_temp_dir(zip_path, "resource")
- assert temp_uri == "viking://temp/artifact"
- sidecar_uri = f"{temp_uri}/resource/{IMAGE_MAPPINGS_FILENAME}"
+ assert artifact_ref.root == "viking://temp/artifact"
+ assert artifact_ref.resource_rel == "resource"
+ sidecar_uri = f"{artifact_ref.root}/resource/{IMAGE_MAPPINGS_FILENAME}"
assert json.loads(fake_fs.files[sidecar_uri]) == {
"Ov测试_1.md": {"Ov测试_1_img1.png": "Ov测试_1_img1.png"}
}
- assert fake_fs.files[f"{temp_uri}/resource/Ov测试_1_img1.png"] == b"png"
+ assert fake_fs.files[f"{artifact_ref.root}/resource/Ov测试_1_img1.png"] == b"png"
@pytest.mark.asyncio
@@ -100,3 +102,19 @@ async def test_unpack_artifact_cleans_temp_on_failure(tmp_path: Path):
await api._unpack_zip_to_temp_dir(invalid_zip, "resource")
assert fake_fs.deleted_temps == ["viking://temp/artifact"]
+
+
+async def test_unpack_artifact_supports_local_output_store(tmp_path: Path):
+ zip_path = tmp_path / "artifact.zip"
+ with zipfile.ZipFile(zip_path, "w") as archive:
+ archive.writestr("artifact/section.md", "content")
+
+ store = LocalParseOutputStore(str(tmp_path / "artifacts"))
+ api = UnderstandingAPI.__new__(UnderstandingAPI)
+ ref = await api._unpack_zip_to_temp_dir(zip_path, "resource", parse_output_store=store)
+
+ assert ref.backend == "local"
+ assert ref.resource_rel == "resource"
+ assert Path(ref.root, "resource", "section.md").read_text() == "content"
+ assert set(await read_artifact_manifest(store, ref)) == {"resource/section.md"}
diff --git a/tests/resource/test_shared_source.py b/tests/resource/test_shared_source.py
new file mode 100644
index 000000000..ffdc99d2f
--- /dev/null
+++ b/tests/resource/test_shared_source.py
@@ -0,0 +1,116 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Tests for the shared-upload SOURCE reference and worker materialization."""
+
+import pytest
+
+from openviking.parse.accessors.base import SourceType
+from openviking.resource.shared_source import SharedSource, materialize_shared_source
+from openviking.server.identity import RequestContext, Role
+from openviking_cli.session.user_id import UserIdentifier
+
+
+def _valid_payload() -> dict:
+ return {
+ "temp_file_id": "shared_2026010112abcdef",
+ "content_uri": "viking://upload/2026010112/abcdef/content",
+ "account_id": "acct-1",
+ "original_filename": "repo.zip",
+ "file_ext": ".zip",
+ "meta": {"resolved_extension": ".zip"},
+ }
+
+
+class TestSharedSourceValidation:
+ def test_roundtrip(self) -> None:
+ src = SharedSource.from_dict(_valid_payload())
+ assert src.temp_file_id == "shared_2026010112abcdef"
+ assert src.content_uri == "viking://upload/2026010112/abcdef/content"
+ assert SharedSource.from_dict(src.to_dict()) == src
+
+ def test_rejects_non_shared_temp_file_id(self) -> None:
+ payload = _valid_payload()
+ payload["temp_file_id"] = "upload_plainlocal"
+ with pytest.raises(ValueError, match="temp_file_id"):
+ SharedSource.from_dict(payload)
+
+ def test_rejects_content_uri_outside_upload_root(self) -> None:
+ payload = _valid_payload()
+ payload["content_uri"] = "viking://resources/secret/content"
+ with pytest.raises(ValueError, match="content_uri"):
+ SharedSource.from_dict(payload)
+
+ def test_rejects_missing_account(self) -> None:
+ payload = _valid_payload()
+ payload["account_id"] = ""
+ with pytest.raises(ValueError, match="account_id"):
+ SharedSource.from_dict(payload)
+
+
+class _FakeVikingFS:
+ def __init__(
+ self,
+ content: bytes,
+ content_uri: str,
+ account_id: str,
+ *,
+ exists: bool = True,
+ ) -> None:
+ self._content = content
+ self._content_uri = content_uri
+ self._account_id = account_id
+ self._exists = exists
+ self.read_calls: list[str] = []
+ self.read_contexts = []
+
+ async def exists(self, uri: str, *, ctx) -> bool:
+ return self._exists and uri == self._content_uri
+
+ async def read_file_bytes(self, uri: str, *, ctx) -> bytes:
+ self.read_calls.append(uri)
+ self.read_contexts.append(ctx)
+ if uri != self._content_uri:
+ raise FileNotFoundError(uri)
+ return self._content
+
+
+def _ctx() -> RequestContext:
+ return RequestContext(user=UserIdentifier("acct-1", "user-1"), role=Role.USER)
+
+
+class TestMaterializeSharedSource:
+ async def test_downloads_content_once_to_local_file(self) -> None:
+ src = SharedSource.from_dict(_valid_payload())
+ vfs = _FakeVikingFS(b"PK\x03\x04zip-bytes", src.content_uri, "acct-1")
+
+ resource = await materialize_shared_source(src, viking_fs=vfs, ctx=_ctx())
+ try:
+ assert resource.source_type == SourceType.LOCAL
+ assert resource.path.exists()
+ assert resource.path.suffix == ".zip"
+ assert resource.path.read_bytes() == b"PK\x03\x04zip-bytes"
+ # Worker downloads the shared object exactly once; API never staged it.
+ assert vfs.read_calls == [src.content_uri]
+ assert vfs.read_contexts[0].role == "root"
+ assert vfs.read_contexts[0].account_id == "acct-1"
+ finally:
+ resource.cleanup()
+ assert not resource.path.exists()
+
+ async def test_missing_content_fails_without_writing_file(self) -> None:
+ src = SharedSource.from_dict(_valid_payload())
+ vfs = _FakeVikingFS(b"", "viking://upload/other/leaf/content", "acct-1")
+
+ with pytest.raises(ValueError, match="missing"):
+ await materialize_shared_source(src, viking_fs=vfs, ctx=_ctx())
+
+ async def test_exact_read_succeeds_even_when_exists_is_stale(self) -> None:
+ src = SharedSource.from_dict(_valid_payload())
+ vfs = _FakeVikingFS(b"payload", src.content_uri, "acct-1", exists=False)
+
+ resource = await materialize_shared_source(src, viking_fs=vfs, ctx=_ctx())
+ try:
+ assert resource.path.read_bytes() == b"payload"
+ finally:
+ resource.cleanup()
diff --git a/tests/server/test_admin_rebuild_api.py b/tests/server/test_admin_rebuild_api.py
index 4762ba1c3..c94dd9db8 100644
--- a/tests/server/test_admin_rebuild_api.py
+++ b/tests/server/test_admin_rebuild_api.py
@@ -1572,7 +1572,7 @@ async def test_reindex_resource_vectors_non_recursive_skips_tree(monkeypatch):
async def exists(self, uri, ctx=None):
return True
- async def stat(self, uri, ctx=None):
+ async def stat(self, uri, ctx=None, skip_count=True):
return {"isDir": True}
async def fail_tree_all(*args, **kwargs):
@@ -2029,7 +2029,9 @@ async def test_reindex_upsert_context_omits_search_tags_without_ingest_options(m
def fake_from_context(context):
captured["meta"] = dict(context.meta or {})
- return _FakeMsg()
+ msg = _FakeMsg()
+ msg.context_data = context.to_dict()
+ return msg
monkeypatch.setattr(
"openviking.service.reindex_executor.EmbeddingMsgConverter.from_context",
@@ -2051,10 +2053,12 @@ async def test_reindex_upsert_context_omits_search_tags_without_ingest_options(m
context_type="resource",
level=ContextLevel.DETAIL,
ctx=ctx,
+ md5="final-md5",
)
assert "search_tags" not in captured["meta"]
assert "search_tags" not in captured["msg"].context_data
+ assert captured["msg"].context_data["md5"] == "final-md5"
@pytest.mark.asyncio
@@ -2066,9 +2070,12 @@ async def test_reindex_resource_vectors_parallelize_files_and_isolate_failures(m
async def exists(self, uri, ctx=None):
return True
- async def stat(self, uri, ctx=None):
+ async def stat(self, uri, ctx=None, skip_count=True):
return {"isDir": True}
+ async def read_file_bytes(self, uri, ctx=None):
+ return uri.encode()
+
async def tree(
self,
uri,
@@ -2094,7 +2101,7 @@ async def test_reindex_resource_vectors_parallelize_files_and_isolate_failures(m
async def fake_best_file_summary(self, uri, *, ctx):
return f"summary:{uri.rsplit('/', 1)[-1]}"
- async def fake_best_resource_file_vector_text(self, uri, summary, ctx):
+ async def fake_best_resource_file_vector_text(self, uri, summary, ctx, file_content=None):
return summary
seen = []
@@ -2281,9 +2288,12 @@ async def test_reindex_resource_l2_falls_back_to_vector_text_when_summary_missin
async def exists(self, uri, ctx=None):
return True
- async def stat(self, uri, ctx=None):
+ async def stat(self, uri, ctx=None, skip_count=True):
return {"isDir": True}
+ async def read_file_bytes(self, uri, ctx=None):
+ return b"raw file body"
+
async def tree(
self,
uri,
@@ -2308,7 +2318,7 @@ async def test_reindex_resource_l2_falls_back_to_vector_text_when_summary_missin
async def fake_best_file_summary(self, uri, *, ctx):
return ""
- async def fake_best_resource_file_vector_text(self, uri, summary, ctx):
+ async def fake_best_resource_file_vector_text(self, uri, summary, ctx, file_content=None):
return "raw file body"
async def fake_upsert_context(self, **kwargs):
@@ -2341,6 +2351,52 @@ async def test_reindex_resource_l2_falls_back_to_vector_text_when_summary_missin
assert seen["viking://resources/demo/file.txt"]["abstract"] == "raw file body"
+async def test_reindex_resource_file_writes_md5_from_same_bytes(monkeypatch):
+ from openviking.service.reindex_executor import ReindexExecutor, _ReindexCounters
+ from openviking.utils.content_hash import content_md5
+
+ file_uri = "viking://resources/demo/file.txt"
+ raw = b"current body"
+
+ class FakeVikingFS:
+ async def exists(self, uri, ctx=None):
+ return True
+
+ async def stat(self, uri, ctx=None, skip_count=True):
+ return {"isDir": False}
+
+ async def read_file_bytes(self, uri, ctx=None):
+ assert uri == file_uri
+ return raw
+
+ seen = {}
+
+ async def fake_best_file_summary(self, uri, *, ctx):
+ return "summary"
+
+ async def fake_best_resource_file_vector_text(self, uri, summary, ctx, file_content=None):
+ return "current body"
+
+ async def fake_upsert_context(self, **kwargs):
+ seen.update(kwargs)
+
+ monkeypatch.setattr("openviking.service.reindex_executor.get_viking_fs", lambda: FakeVikingFS())
+ monkeypatch.setattr(ReindexExecutor, "_best_file_summary", fake_best_file_summary)
+ monkeypatch.setattr(
+ ReindexExecutor,
+ "_best_resource_file_vector_text",
+ fake_best_resource_file_vector_text,
+ )
+ monkeypatch.setattr(ReindexExecutor, "_upsert_context", fake_upsert_context)
+
+ counters = _ReindexCounters()
+ ctx = RequestContext(user=UserIdentifier(account_id="test", user_id="alice"), role=Role.ROOT)
+ await ReindexExecutor()._reindex_resource_vectors(uri=file_uri, counters=counters, ctx=ctx)
+
+ assert seen["md5"] == content_md5(raw)
+
+
@pytest.mark.asyncio
async def test_reindex_resource_vector_text_uses_existing_record_for_non_text(monkeypatch):
from openviking.service.reindex_executor import ReindexExecutor
@@ -2602,9 +2658,12 @@ async def test_reindex_resource_vectors_accepts_single_file_uri(monkeypatch):
async def exists(self, uri, ctx=None):
return True
- async def stat(self, uri, ctx=None):
+ async def stat(self, uri, ctx=None, skip_count=True):
return {"isDir": False}
+ async def read_file_bytes(self, uri, ctx=None):
+ return b"file summary"
+
async def tree(self, *args, **kwargs):
raise AssertionError("single-file reindex should not call tree")
@@ -2613,7 +2672,7 @@ async def test_reindex_resource_vectors_accepts_single_file_uri(monkeypatch):
async def fake_best_file_summary(self, uri, *, ctx):
return "file summary"
- async def fake_best_resource_file_vector_text(self, uri, summary, ctx):
+ async def fake_best_resource_file_vector_text(self, uri, summary, ctx, file_content=None):
return summary
async def fake_upsert_context(self, **kwargs):
diff --git a/tests/server/test_content_batch_write.py b/tests/server/test_content_batch_write.py
index 0d2384927..7b15579b2 100644
--- a/tests/server/test_content_batch_write.py
+++ b/tests/server/test_content_batch_write.py
@@ -8,6 +8,7 @@ from openviking.session.memory.dataclass import MemoryFile
from openviking.session.memory.utils import MemoryFileUtils
from openviking.storage.content_write import ContentWriteCoordinator
from openviking.storage.queuefs.semantic_ops.freshness_policy import FreshnessAction
+from openviking.utils.content_hash import content_md5
from openviking_cli.exceptions import (
AlreadyExistsError,
InvalidArgumentError,
@@ -187,6 +188,45 @@ async def test_batch_releases_file_locks_before_one_aggregated_refresh(monkeypat
assert locks.releases == 1
+async def test_batch_passes_final_md5s_and_old_abstracts_to_refresh(monkeypatch):
+ root = "viking://resources/wiki"
+ existing = f"{root}/existing.md"
+ created = f"{root}/new.md"
+ vfs = _VFS(root, {existing: "old"})
+
+ class _VikingDB:
+ async def get_l2_diff_records_by_uris(self, uris, *, ctx):
+ assert uris == [existing]
+ return {existing: {"abstract": "old abstract", "md5": "old-md5"}}
+
+ coordinator = ContentWriteCoordinator(vfs, vikingdb=_VikingDB())
+ captured = {}
+
+ async def refresh(**kwargs):
+ captured.update(kwargs)
+ return None
+
+ monkeypatch.setattr(coordinator, "_refresh_batch", refresh)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+
+ await coordinator.batch_write(
+ root_uri=root,
+ operations=[
+ {"uri": existing, "content": "new", "mode": "replace"},
+ {"uri": created, "content": "created", "mode": "create"},
+ ],
+ ctx=ctx,
+ wait=False,
+ )
+
+ assert captured["file_md5s"] == {
+ existing: content_md5(b"new"),
+ created: content_md5(b"created"),
+ }
+ assert captured["file_abstracts"] == {existing: "old abstract"}
+
+
@pytest.mark.asyncio
async def test_batch_replace_memory_preserves_metadata(monkeypatch):
root = "viking://user/default/memories/preferences"
@@ -269,7 +309,10 @@ async def test_batch_upserts_binary_content(monkeypatch):
vfs = _VFS(root, {image: original})
coordinator = ContentWriteCoordinator(vfs)
+ refresh_calls = []
+
async def refresh(**kwargs):
+ refresh_calls.append(kwargs)
return None
monkeypatch.setattr(coordinator, "_refresh_batch", refresh)
@@ -284,6 +327,7 @@ async def test_batch_upserts_binary_content(monkeypatch):
)
assert result["updated"] == [image]
assert vfs.files[image] == replacement
+ assert refresh_calls[0]["file_md5s"] == {image: content_md5(replacement)}
retry = await coordinator.batch_write(
root_uri=root, operations=[operation], ctx=ctx, wait=False
diff --git a/tests/server/test_content_write_service.py b/tests/server/test_content_write_service.py
index 9af29ab41..2e9285ad6 100644
--- a/tests/server/test_content_write_service.py
+++ b/tests/server/test_content_write_service.py
@@ -869,7 +869,7 @@ async def test_memory_write_wait_skips_semantic_queue_and_releases_write_lock(mo
async def _fake_refresh_schema_overview(**kwargs):
del kwargs
- return None
+ return True
monkeypatch.setattr(coordinator, "_write_in_place", _fake_write_in_place)
monkeypatch.setattr(coordinator, "_wait_for_request", _fail_wait_for_request)
@@ -1288,7 +1288,7 @@ async def test_resource_write_anchors_nested_file_to_direct_parent(
):
"""A resource content write anchors the semantic refresh at the written file's
direct parent directory (anchor_to_parent=True), so the changed file is a direct
- child of the DAG run root: its own L2 vector and the parent's L0/L1 are generated
+ child of the semantic-tree root: its own L2 vector and the parent's L0/L1 are generated
from a single-directory run instead of a recursive walk of the whole project subtree.
set_tags keeps the project-root collapse (the default), which the derived
``.abstract.md`` sidecar mapping relies on."""
diff --git a/tests/server/test_skill_update_cancellation.py b/tests/server/test_skill_update_cancellation.py
index 889c6e4db..5741b34c5 100644
--- a/tests/server/test_skill_update_cancellation.py
+++ b/tests/server/test_skill_update_cancellation.py
@@ -64,7 +64,7 @@ async def _indexed_record(client, uri, level):
async def test_update_cancellation_restores_package_and_index(
client, tmp_path, monkeypatch, cancel_mode
):
- from openviking.storage.queuefs import semantic_dag
+ from openviking.storage.queuefs import semantic_executor
name = "timeout-package-rollback"
old_upload = await _upload_package(
@@ -109,7 +109,7 @@ async def test_update_cancellation_restores_package_and_index(
finished = set()
task_owners = {}
original_summary = SemanticProcessor._generate_single_file_summary
- original_scheduler = semantic_dag.get_semantic_node_scheduler
+ original_scheduler = semantic_executor.get_semantic_tree_scheduler
async def hold_summary(self, file_path, *args, **kwargs):
task_context = get_task_context()
@@ -130,7 +130,7 @@ async def test_update_cancellation_restores_package_and_index(
monkeypatch.setattr(SemanticProcessor, "_generate_single_file_summary", hold_summary)
monkeypatch.setattr(
- semantic_dag, "get_semantic_node_scheduler", lambda _workers: original_scheduler(2)
+ semantic_executor, "get_semantic_tree_scheduler", lambda _workers: original_scheduler(2)
)
updating = asyncio.create_task(
client.put(
diff --git a/tests/server/test_skill_update_retry.py b/tests/server/test_skill_update_retry.py
index d10604c26..e2a15381e 100644
--- a/tests/server/test_skill_update_retry.py
+++ b/tests/server/test_skill_update_retry.py
@@ -9,7 +9,7 @@ import threading
import pytest
from openviking.storage.queuefs import get_queue_manager
-from openviking.storage.queuefs.semantic_dag import SemanticDagExecutor
+from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor
from tests.server.test_api_skills import _add_skill, _skill_md
from tests.server.test_api_skills import _stub_mcp_endpoint as _stub_mcp_endpoint
from tests.server.test_skill_update_cancellation import _download, _wait_until
@@ -17,7 +17,7 @@ from tests.server.test_skill_update_lock import _assert_locked, _ctx
def _fail_first_run(monkeypatch, root, *, block_retries=False):
- original = SemanticDagExecutor.run
+ original = SemanticTreeExecutor.run
attempts = []
async def fail_once(self, root_uri):
@@ -29,7 +29,7 @@ def _fail_first_run(monkeypatch, root, *, block_retries=False):
await asyncio.Event().wait()
return await original(self, root_uri)
- monkeypatch.setattr(SemanticDagExecutor, "run", fail_once)
+ monkeypatch.setattr(SemanticTreeExecutor, "run", fail_once)
return attempts
diff --git a/tests/service/test_resource_service_reserved_target_cleanup.py b/tests/service/test_resource_service_reserved_target_cleanup.py
index 19748b17d..121f681ad 100644
--- a/tests/service/test_resource_service_reserved_target_cleanup.py
+++ b/tests/service/test_resource_service_reserved_target_cleanup.py
@@ -5,7 +5,8 @@ import pytest
from openviking.server.identity import RequestContext, Role
from openviking.service.resource_service import ResourceService
-from openviking.storage.queuefs.add_resource_msg import AddResourceMsg
+from openviking.storage.queuefs.add_resource_msg import AddResourceMsg, AddResourcePhase
+from openviking.storage.queuefs.add_resource_processor import AddResourceProcessor
from openviking_cli.exceptions import FailedPreconditionError
from openviking_cli.session.user_id import UserIdentifier
@@ -84,6 +85,38 @@ async def test_cleanup_reserved_target_preserves_nonempty_directory():
)
+async def test_cancelled_post_process_cleans_agfs_plan_artifact():
+ viking_fs = SimpleNamespace(
+ delete_temp=AsyncMock(),
+ _async_agfs=SimpleNamespace(),
+ )
+ processor = AddResourceProcessor(SimpleNamespace(), "add_resource", viking_fs)
+ msg = AddResourceMsg(
+ task_id="task-1",
+ job_phase=AddResourcePhase.POST_PROCESS,
+ root_uri="viking://resources/repo",
+ account_id="acct",
+ user_id="alice",
+ role="user",
+ prepared={
+ "root_uri": "viking://resources/repo",
+ "artifact_ref": {
+ "backend": "agfs",
+ "root": "viking://temp/task-1",
+ "resource_rel": "repository",
+ "root_type": "dir",
+ },
+ "plan_artifact_committed": True,
+ },
+ )
+ ctx = RequestContext(user=UserIdentifier("acct", "alice"), role=Role.USER)
+
+ await processor._release_cancelled_resources(msg, ctx)
+
+ viking_fs.delete_temp.assert_awaited_once_with("viking://temp/task-1", ctx=ctx)
+
+
@pytest.mark.asyncio
@pytest.mark.parametrize(
("target_preexisting", "cleanup_empty_target_on_failure"),
diff --git a/tests/service/test_task_queue_middleware.py b/tests/service/test_task_queue_middleware.py
index 9811b8e5c..3d79bc89e 100644
--- a/tests/service/test_task_queue_middleware.py
+++ b/tests/service/test_task_queue_middleware.py
@@ -22,6 +22,8 @@ from openviking.storage.queuefs.named_queue import DequeueHandlerBase, NamedQueu
from openviking.storage.queuefs.process_result import ProcessOutcome, ProcessResult
from openviking.storage.queuefs.queue_middleware import QueueMiddleware
from openviking.storage.queuefs.session_commit_processor import SessionCommitProcessor
+from openviking.telemetry import OperationTelemetry, bind_telemetry
+from openviking.utils.log_correlation import log_correlation
@pytest.fixture
@@ -59,6 +61,25 @@ async def enqueue_task(queue, transport, task_id="task-1"):
return {"id": "message-1", "data": transport.write.await_args.args[1].decode()}
+def test_log_correlation_uses_bound_task_and_telemetry() -> None:
+ telemetry = OperationTelemetry(operation="add_resource_job", enabled=True)
+
+ with (
+ bind_task_context("task-123", "account", "user"),
+ bind_telemetry(telemetry),
+ ):
+ assert log_correlation(message_id="semantic-456") == (
+ f"task_id=task-123 telemetry_id={telemetry.telemetry_id} message_id=semantic-456"
+ )
+
+
+def test_log_correlation_accepts_explicit_queue_ids_without_context() -> None:
+ assert (
+ log_correlation(task_id="task-1", telemetry_id="tm-1", message_id="embedding-2")
+ == "task_id=task-1 telemetry_id=tm-1 message_id=embedding-2"
+ )
+
+
async def test_enqueue_registers_before_write_and_ack_finalizes_before_delete(tracked_queue):
queue, index, transport, finalize, _ = tracked_queue
@@ -227,16 +248,19 @@ async def test_process_result_tracks_children_and_errors(tracked_queue):
queue, index, transport, _, _ = tracked_queue
message = await enqueue_task(queue, transport)
contexts = []
+ correlations = []
class Handler(DequeueHandlerBase):
async def on_dequeue(self, data):
contexts.append(get_task_context())
+ correlations.append(log_correlation(telemetry_id="tm-1", message_id="semantic-1"))
await queue.enqueue({"child": True})
return ProcessResult.failed("failed work")
queue.set_dequeue_handler(Handler())
assert (await queue.process_dequeued(message)).outcome is ProcessOutcome.FAILED
assert contexts[0].task_id == "task-1"
+ assert correlations == ["task_id=task-1 telemetry_id=tm-1 message_id=semantic-1"]
assert get_task_context() is None
assert index.failure("task-1") == "failed work"
child_payload = next(
diff --git a/tests/storage/test_collection_schemas.py b/tests/storage/test_collection_schemas.py
index c62aba5b8..aed262f89 100644
--- a/tests/storage/test_collection_schemas.py
+++ b/tests/storage/test_collection_schemas.py
@@ -25,11 +25,12 @@ from openviking.storage.errors import (
VikingDBException,
)
from openviking.storage.expr import Eq
+from openviking.storage.index_action import IndexAction
from openviking.storage.queuefs.embedding_msg import EmbeddingMsg
from openviking.storage.queuefs.process_result import ProcessOutcome
from openviking.storage.vector_ids import vector_record_id
from openviking.storage.vectordb import engine as vectordb_engine
-from openviking.storage.vectordb.collection.result import UpsertDataResult
+from openviking.storage.vectordb.collection.result import UpdateResult, UpsertDataResult
from openviking.storage.vectordb.collection.vikingdb_clients import VikingDBClient
from openviking.storage.vectordb.collection.vikingdb_collection import VikingDBCollection
from openviking.storage.vectordb.collection.volcengine_api_key_collection import (
@@ -137,6 +138,443 @@ def _build_queue_payload_for_account(account_id: str) -> dict:
return {"data": json.dumps(msg.to_dict())}
+def _build_operation_payload(msg: EmbeddingMsg) -> dict:
+ return {"data": msg.to_json()}
+
+
+async def test_embedding_handler_delete_skips_embedder_and_strictly_deletes_ids(monkeypatch):
+ class _DeletingVikingDB:
+ is_closing = False
+
+ def __init__(self):
+ self.deleted = []
+
+ async def strict_delete(self, ids, *, ctx):
+ self.deleted.append((list(ids), ctx.account_id))
+ return len(ids)
+
+ config = _DummyConfig(_DummyEmbedder())
+ monkeypatch.setattr(
+ "openviking_cli.utils.config.get_openviking_config",
+ lambda: config,
+ )
+ monkeypatch.setattr(
+ config.embedding,
+ "get_embedder",
+ lambda: (_ for _ in ()).throw(AssertionError("embedder must stay lazy")),
+ )
+ vikingdb = _DeletingVikingDB()
+ handler = TextEmbeddingHandler(vikingdb)
+ msg = EmbeddingMsg.for_delete(
+ record_ids=["l0-id", "l1-id"],
+ context_data={"account_id": "acct", "uri": "viking://resources/repo"},
+ )
+
+ result = await handler.on_dequeue(_build_operation_payload(msg))
+
+ assert result.outcome is ProcessOutcome.SUCCESS
+ assert result.value == {"deleted_count": 2}
+ assert vikingdb.deleted == [(["l0-id", "l1-id"], "acct")]
+
+
+async def test_embedding_handler_update_fields_reads_merges_and_uses_strict_partial_update(
+ monkeypatch,
+):
+ existing = {
+ "id": "l2-id",
+ "uri": "viking://resources/repo/a.py",
+ "account_id": "acct",
+ "abstract": "old abstract",
+ "md5": "old-md5",
+ "vector": [0.1, 0.2],
+ }
+
+ class _UpdatingVikingDB:
+ is_closing = False
+
+ def __init__(self):
+ self.reads = []
+ self.updates = []
+
+ async def get_strict(self, ids, *, ctx):
+ self.reads.append((list(ids), ctx.account_id))
+ return [dict(existing)]
+
+ async def update(self, data, *, ctx):
+ self.updates.append((dict(data), ctx.account_id))
+ return UpdateResult(ok=True, ids=[data["id"]], updated_count=1)
+
+ config = _DummyConfig(_DummyEmbedder())
+ monkeypatch.setattr(
+ "openviking_cli.utils.config.get_openviking_config",
+ lambda: config,
+ )
+ monkeypatch.setattr(
+ config.embedding,
+ "get_embedder",
+ lambda: (_ for _ in ()).throw(AssertionError("embedder must stay lazy")),
+ )
+ vikingdb = _UpdatingVikingDB()
+ handler = TextEmbeddingHandler(vikingdb)
+ msg = EmbeddingMsg.for_update_fields(
+ record_id="l2-id",
+ fields={"abstract": "new abstract", "md5": "new-md5"},
+ context_data={"account_id": "acct", "uri": existing["uri"]},
+ )
+
+ result = await handler.on_dequeue(_build_operation_payload(msg))
+
+ assert vikingdb.reads == [(["l2-id"], "acct")]
+ assert len(vikingdb.updates) == 1
+ updated, account_id = vikingdb.updates[0]
+ assert account_id == "acct"
+ assert updated == {
+ "id": "l2-id",
+ "abstract": "new abstract",
+ "md5": "new-md5",
+ "updated_at": updated["updated_at"],
+ }
+ assert result.outcome is ProcessOutcome.SUCCESS
+ assert result.value == updated
+
+
+async def test_embedding_handler_update_fields_append_uses_latest_tags_and_skips_noop(
+ monkeypatch,
+):
+ existing = {
+ "id": "l2-id",
+ "uri": "viking://resources/repo/a.py",
+ "account_id": "acct",
+ "level": 2,
+ "vector": [0.1, 0.2],
+ "search_tags": ["env=old", "scope=new"],
+ }
+
+ class _UpdatingVikingDB:
+ is_closing = False
+
+ def __init__(self):
+ self.updates = []
+
+ async def get_strict(self, ids, *, ctx):
+ return [dict(existing)]
+
+ async def update(self, data, *, ctx):
+ self.updates.append(dict(data))
+ return UpdateResult(ok=True, ids=[data["id"]], updated_count=1)
+
+ config = _DummyConfig(_DummyEmbedder())
+ monkeypatch.setattr("openviking_cli.utils.config.get_openviking_config", lambda: config)
+ monkeypatch.setattr(
+ config.embedding,
+ "get_embedder",
+ lambda: (_ for _ in ()).throw(AssertionError("embedder must stay lazy")),
+ )
+ vikingdb = _UpdatingVikingDB()
+ handler = TextEmbeddingHandler(vikingdb)
+ msg = EmbeddingMsg.for_update_fields(
+ record_id="l2-id",
+ fields={"search_tags": ["scope=new"]},
+ field_modes={"search_tags": "append"},
+ context_data={"account_id": "acct", "uri": existing["uri"]},
+ )
+
+ result = await handler.on_dequeue(_build_operation_payload(msg))
+
+ assert result.outcome is ProcessOutcome.SUCCESS
+ assert result.value == {"id": "l2-id", "status": "skipped"}
+ assert vikingdb.updates == []
+
+
+async def test_embedding_handler_update_fields_replace_and_clear_do_not_merge_old_tags(
+ monkeypatch,
+):
+ existing = {
+ "id": "l2-id",
+ "uri": "viking://resources/repo/a.py",
+ "account_id": "acct",
+ "level": 2,
+ "vector": [0.1, 0.2],
+ "search_tags": ["env=old"],
+ }
+
+ class _UpdatingVikingDB:
+ is_closing = False
+
+ def __init__(self):
+ self.updates = []
+
+ async def get_strict(self, ids, *, ctx):
+ return [dict(existing)]
+
+ async def update(self, data, *, ctx):
+ self.updates.append(dict(data))
+ return UpdateResult(ok=True, ids=[data["id"]], updated_count=1)
+
+ config = _DummyConfig(_DummyEmbedder())
+ monkeypatch.setattr("openviking_cli.utils.config.get_openviking_config", lambda: config)
+ monkeypatch.setattr(
+ config.embedding,
+ "get_embedder",
+ lambda: (_ for _ in ()).throw(AssertionError("embedder must stay lazy")),
+ )
+ vikingdb = _UpdatingVikingDB()
+ handler = TextEmbeddingHandler(vikingdb)
+
+ for tags in (["scope=new"], []):
+ msg = EmbeddingMsg.for_update_fields(
+ record_id="l2-id",
+ fields={"search_tags": tags},
+ field_modes={"search_tags": "replace"},
+ context_data={"account_id": "acct", "uri": existing["uri"]},
+ )
+ result = await handler.on_dequeue(_build_operation_payload(msg))
+ assert result.outcome is ProcessOutcome.SUCCESS
+
+ assert [item["search_tags"] for item in vikingdb.updates] == [["scope=new"], []]
+
+
+async def test_embedding_handler_update_fields_read_failure_does_not_create(monkeypatch):
+ class _BrokenVikingDB:
+ is_closing = False
+
+ async def get_strict(self, ids, *, ctx):
+ raise RuntimeError("read failed")
+
+ async def upsert(self, data, *, ctx, options=UpsertOptions()):
+ raise AssertionError("read failure must not be treated as not found")
+
+ config = _DummyConfig(_DummyEmbedder())
+ monkeypatch.setattr("openviking_cli.utils.config.get_openviking_config", lambda: config)
+ monkeypatch.setattr(
+ config.embedding,
+ "get_embedder",
+ lambda: (_ for _ in ()).throw(AssertionError("embedder must stay lazy")),
+ )
+ handler = TextEmbeddingHandler(_BrokenVikingDB())
+ msg = EmbeddingMsg.for_update_fields(
+ record_id="l2-id",
+ fields={"search_tags": ["scope=new"]},
+ field_modes={"search_tags": "append"},
+ initial_fields={
+ "uri": "viking://resources/repo/a.py",
+ "account_id": "acct",
+ "level": 2,
+ "vector": [0.1, 0.2],
+ },
+ context_data={"account_id": "acct", "uri": "viking://resources/repo/a.py"},
+ )
+
+ result = await handler.on_dequeue(_build_operation_payload(msg))
+
+ assert result.outcome is ProcessOutcome.FAILED
+ assert "read failed" in result.error
+
+
+async def test_embedding_handler_update_fields_not_found_creates_from_complete_initial_fields(
+ monkeypatch,
+):
+ class _MissingVikingDB:
+ is_closing = False
+
+ def __init__(self):
+ self.upserts = []
+
+ async def get_strict(self, ids, *, ctx):
+ return []
+
+ async def upsert(self, data, *, ctx, options=UpsertOptions()):
+ self.upserts.append((dict(data), ctx.account_id, options))
+ return data["id"]
+
+ config = _DummyConfig(_DummyEmbedder())
+ monkeypatch.setattr("openviking_cli.utils.config.get_openviking_config", lambda: config)
+ monkeypatch.setattr(
+ config.embedding,
+ "get_embedder",
+ lambda: (_ for _ in ()).throw(AssertionError("embedder must stay lazy")),
+ )
+ vikingdb = _MissingVikingDB()
+ handler = TextEmbeddingHandler(vikingdb)
+ msg = EmbeddingMsg.for_update_fields(
+ record_id="l2-id",
+ fields={"search_tags": ["scope=new"]},
+ field_modes={"search_tags": "append"},
+ initial_fields={
+ "uri": "viking://resources/repo/a.py",
+ "account_id": "acct",
+ "level": 2,
+ "vector": [0.1, 0.2],
+ },
+ context_data={"account_id": "acct", "uri": "viking://resources/repo/a.py"},
+ )
+
+ result = await handler.on_dequeue(_build_operation_payload(msg))
+
+ assert result.outcome is ProcessOutcome.SUCCESS
+ created, account_id, options = vikingdb.upserts[0]
+ assert account_id == "acct"
+ assert created["search_tags"] == ["scope=new"]
+ assert options.partial_update is False
+
+
+async def test_embedding_handler_update_fields_not_found_rejects_incomplete_initial_fields(
+ monkeypatch,
+):
+ class _MissingVikingDB:
+ is_closing = False
+
+ async def get_strict(self, ids, *, ctx):
+ return []
+
+ async def upsert(self, data, *, ctx, options=UpsertOptions()):
+ raise AssertionError("incomplete record must not be written")
+
+ config = _DummyConfig(_DummyEmbedder())
+ monkeypatch.setattr("openviking_cli.utils.config.get_openviking_config", lambda: config)
+ monkeypatch.setattr(
+ config.embedding,
+ "get_embedder",
+ lambda: (_ for _ in ()).throw(AssertionError("embedder must stay lazy")),
+ )
+ handler = TextEmbeddingHandler(_MissingVikingDB())
+ msg = EmbeddingMsg.for_update_fields(
+ record_id="l2-id",
+ fields={"search_tags": ["scope=new"]},
+ field_modes={"search_tags": "append"},
+ initial_fields={
+ "uri": "viking://resources/repo/a.py",
+ "account_id": "acct",
+ "level": 2,
+ },
+ context_data={"account_id": "acct", "uri": "viking://resources/repo/a.py"},
+ )
+
+ result = await handler.on_dequeue(_build_operation_payload(msg))
+
+ assert result.outcome is ProcessOutcome.FAILED
+ assert "missing_fields=['vector']" in result.error
+
+
+async def test_embedding_handler_merge_action_reads_and_merges_before_full_upsert(monkeypatch):
+ embedder = _DummyEmbedder()
+ config = _DummyConfig(embedder)
+ monkeypatch.setattr(
+ "openviking_cli.utils.config.get_openviking_config",
+ lambda: config,
+ )
+ captured = {}
+
+ class _MergingVikingDB:
+ is_closing = False
+ uses_content_field = False
+
+ async def get_strict(self, ids, *, ctx):
+ captured["read"] = (list(ids), ctx.account_id)
+ return [{"id": ids[0], "search_tags": ["env=old"], "created_at": "old"}]
+
+ async def upsert(self, data, *, ctx, options=UpsertOptions()):
+ captured["data"] = dict(data)
+ captured["options"] = options
+ return data["id"]
+
+ handler = TextEmbeddingHandler(_MergingVikingDB())
+ msg = EmbeddingMsg(
+ message="body",
+ action=IndexAction.MERGE,
+ context_data={
+ "id": "generated-id",
+ "_upsert_record_id": "generated-id",
+ "uri": "viking://resources/repo/a.py",
+ "account_id": "acct",
+ "abstract": "summary",
+ },
+ update_fields={"search_tags": ["scope=new"]},
+ field_modes={"search_tags": "append"},
+ )
+
+ result = await handler.on_dequeue(_build_operation_payload(msg))
+
+ assert result.outcome is ProcessOutcome.SUCCESS
+ assert captured["read"] == (["generated-id"], "acct")
+ assert captured["data"]["search_tags"] == ["env=old", "scope=new"]
+ assert captured["data"]["created_at"] == "old"
+ assert captured["options"].partial_update is False
+
+
+async def test_embedding_handler_merge_not_found_creates_from_generated_vector(monkeypatch):
+ config = _DummyConfig(_DummyEmbedder())
+ monkeypatch.setattr("openviking_cli.utils.config.get_openviking_config", lambda: config)
+ captured = {}
+
+ class _MissingVikingDB:
+ is_closing = False
+ uses_content_field = False
+
+ async def get_strict(self, ids, *, ctx):
+ return []
+
+ async def upsert(self, data, *, ctx, options=UpsertOptions()):
+ captured["data"] = dict(data)
+ return data["id"]
+
+ handler = TextEmbeddingHandler(_MissingVikingDB())
+ msg = EmbeddingMsg(
+ message="body",
+ action=IndexAction.MERGE,
+ context_data={
+ "_upsert_record_id": "generated-id",
+ "uri": "viking://resources/repo/a.py",
+ "account_id": "acct",
+ "level": 2,
+ "abstract": "summary",
+ },
+ update_fields={"search_tags": ["scope=new"]},
+ field_modes={"search_tags": "append"},
+ )
+
+ result = await handler.on_dequeue(_build_operation_payload(msg))
+
+ assert result.outcome is ProcessOutcome.SUCCESS
+ assert captured["data"]["id"] == "generated-id"
+ assert captured["data"]["vector"] == [0.1, 0.2]
+ assert captured["data"]["search_tags"] == ["scope=new"]
+
+
+async def test_embedding_handler_skips_noop_action_without_vector_db_or_embedder(monkeypatch):
+ from openviking.storage.index_action import IndexAction
+
+ config = _DummyConfig(_DummyEmbedder())
+ monkeypatch.setattr("openviking_cli.utils.config.get_openviking_config", lambda: config)
+
+ class _NoopVikingDB:
+ is_closing = False
+
+ def __getattr__(self, name):
+ raise AssertionError(f"noop must not call {name}")
+
+ handler = TextEmbeddingHandler(_NoopVikingDB())
+ msg = EmbeddingMsg(
+ message=None,
+ context_data={"account_id": "acct", "uri": "viking://resources/repo"},
+ action=IndexAction.NONE,
+ )
+
+ result = await handler.on_dequeue(_build_operation_payload(msg))
+
+ assert result.outcome is ProcessOutcome.SUCCESS
+
+
def test_embedding_handler_builds_circuit_breaker_from_config(monkeypatch):
class _DummyVikingDB:
is_closing = False
@@ -410,7 +848,7 @@ async def test_embedding_handler_treats_shutdown_write_lock_as_success(monkeypat
self.calls = 0
async def upsert(self, _data, *, ctx, options=UpsertOptions()):
- assert options.partial_update is True
+ assert options.partial_update is False
self.calls += 1
self.is_closing = True
raise RuntimeError("IO error: lock /tmp/LOCK: already held by process")
@@ -579,7 +1017,7 @@ async def test_embedding_handler_truncates_queue_input_before_embed(monkeypatch)
handler = TextEmbeddingHandler(_CapturingVikingDB())
payload = _build_queue_payload()
queue_data = json.loads(payload["data"])
- queue_data["message"] = " ".join(f"token-{idx}" for idx in range(200))
+ queue_data["payload"]["message"] = " ".join(f"token-{idx}" for idx in range(200))
payload["data"] = json.dumps(queue_data)
await handler.on_dequeue(payload)
@@ -644,7 +1082,7 @@ async def test_embedding_handler_preserves_parent_uri_for_backend_upsert_logic(m
uses_content_field = False
async def upsert(self, data, *, ctx, options=UpsertOptions()):
- assert options.partial_update is True
+ assert options.partial_update is False
captured["data"] = dict(data)
return "rec-1"
@@ -657,7 +1095,7 @@ async def test_embedding_handler_preserves_parent_uri_for_backend_upsert_logic(m
handler = TextEmbeddingHandler(_CapturingVikingDB())
payload = _build_queue_payload()
queue_data = json.loads(payload["data"])
- queue_data["context_data"]["parent_uri"] = "viking://resources"
+ queue_data["payload"]["context_data"]["parent_uri"] = "viking://resources"
payload["data"] = json.dumps(queue_data)
result = await handler.on_dequeue(payload)
@@ -665,7 +1103,7 @@ async def test_embedding_handler_preserves_parent_uri_for_backend_upsert_logic(m
assert result.outcome is ProcessOutcome.SUCCESS
assert result.error is None
assert result.value == {
- **queue_data["context_data"],
+ **queue_data["payload"]["context_data"],
"id": vector_record_id("default", "viking://resources/sample", 2),
"vector": [0.1, 0.2],
}
@@ -673,6 +1111,87 @@ async def test_embedding_handler_preserves_parent_uri_for_backend_upsert_logic(m
assert captured["data"]["parent_uri"] == "viking://resources"
+async def test_embedding_handler_honors_explicit_full_upsert(monkeypatch):
+ captured = {}
+
+ class _CapturingVikingDB:
+ is_closing = False
+ uses_content_field = False
+
+ async def upsert(self, data, *, ctx, options=UpsertOptions()):
+ captured["partial_update"] = options.partial_update
+ return data["id"]
+
+ embedder = _DummyEmbedder()
+ monkeypatch.setattr(
+ "openviking_cli.utils.config.get_openviking_config",
+ lambda: _DummyConfig(embedder),
+ )
+ payload = _build_queue_payload()
+ queue_data = json.loads(payload["data"])
+ queue_data["payload"]["context_data"]["_upsert_options"] = {"partial_update": False}
+ payload["data"] = json.dumps(queue_data)
+
+ await TextEmbeddingHandler(_CapturingVikingDB()).on_dequeue(payload)
+
+ assert captured["partial_update"] is False
+
+
+async def test_embedding_handler_honors_existing_record_id_override(monkeypatch):
+ captured = {}
+
+ class _CapturingVikingDB:
+ is_closing = False
+ uses_content_field = False
+
+ async def upsert(self, data, *, ctx, options=UpsertOptions()):
+ captured["data"] = dict(data)
+ return data["id"]
+
+ embedder = _DummyEmbedder()
+ monkeypatch.setattr(
+ "openviking_cli.utils.config.get_openviking_config",
+ lambda: _DummyConfig(embedder),
+ )
+ payload = _build_queue_payload_for_account("acct-existing-id")
+ queue_data = json.loads(payload["data"])
+ queue_data["payload"]["context_data"]["_upsert_record_id"] = "id-from-vector-db"
+ payload["data"] = json.dumps(queue_data)
+
+ await TextEmbeddingHandler(_CapturingVikingDB()).on_dequeue(payload)
+
+ assert captured["data"]["id"] == "id-from-vector-db"
+ assert "_upsert_record_id" not in captured["data"]
+
+
+async def test_embedding_handler_generates_id_when_no_existing_override(monkeypatch):
+ from openviking.storage.vector_ids import vector_record_id
+
+ captured = {}
+
+ class _CapturingVikingDB:
+ is_closing = False
+ uses_content_field = False
+
+ async def upsert(self, data, *, ctx, options=UpsertOptions()):
+ captured["data"] = dict(data)
+ return data["id"]
+
+ embedder = _DummyEmbedder()
+ monkeypatch.setattr(
+ "openviking_cli.utils.config.get_openviking_config",
+ lambda: _DummyConfig(embedder),
+ )
+ payload = _build_queue_payload_for_account("acct-new-id")
+
+ await TextEmbeddingHandler(_CapturingVikingDB()).on_dequeue(payload)
+
+ assert captured["data"]["id"] == vector_record_id("acct-new-id", "viking://resources/sample", 2)
+
+
@pytest.mark.asyncio
async def test_embedding_handler_settles_request_wait_by_message_id(monkeypatch):
class _CapturingVikingDB:
@@ -681,7 +1200,7 @@ async def test_embedding_handler_settles_request_wait_by_message_id(monkeypatch)
uses_content_field = False
async def upsert(self, _data, *, ctx, options=UpsertOptions()):
- assert options.partial_update is True
+ assert options.partial_update is False
return "rec-1"
embedder = _DummyEmbedder()
@@ -726,6 +1245,11 @@ def test_context_collection_uses_acl_mode_and_excludes_parent_uri():
assert "acl_enabled" not in field_names
assert "acl_enabled" not in schema["ScalarIndex"]
assert "parent_uri" not in field_names
+ # md5 fingerprint field backs incremental diff; it is a scalar value column,
+ # not indexed (looked up alongside a URI/level scan, never filtered on).
+ md5_field = next(field for field in schema["Fields"] if field["FieldName"] == "md5")
+ assert md5_field == {"FieldName": "md5", "FieldType": "string", "DefaultValue": ""}
+ assert "md5" not in schema["ScalarIndex"]
assert "parent_uri" not in schema["ScalarIndex"]
assert "acl_restricted" not in field_names
assert "acl_restricted" not in schema["ScalarIndex"]
@@ -960,7 +1484,7 @@ def test_volcengine_aksk_collection_ignores_unknown_fields_on_fetch_and_search()
calls = []
class _Collection(VolcengineCollection):
- def _data_post(self, path, data):
+ def _data_post(self, path, data, **kwargs):
calls.append((path, data))
return {}
@@ -1442,7 +1966,7 @@ async def test_single_account_backend_update_runs_adapter_in_threadpool(monkeypa
assert result.updated_count == 1
assert result.error_code is None
assert result.error_message is None
- assert [call[0] for call in calls] == ["_prepare_upsert_payload", "update_data"]
+ assert [call[0] for call in calls] == ["_prepare_update_payload", "update_data"]
assert calls[-1][1] == (
[
{
@@ -1455,6 +1979,54 @@ async def test_single_account_backend_update_runs_adapter_in_threadpool(monkeypa
)
+async def test_single_account_backend_partial_update_does_not_fill_omitted_text_or_vector_fields():
+ calls = []
+
+ class _Collection:
+ def get_meta_data(self):
+ return {
+ "Fields": [
+ {"FieldName": "id", "FieldType": "string"},
+ {"FieldName": "abstract", "FieldType": "string"},
+ {"FieldName": "content", "FieldType": "text"},
+ {"FieldName": "vector", "FieldType": "vector"},
+ {"FieldName": "search_tags", "FieldType": "list<string>"},
+ {"FieldName": "account_id", "FieldType": "string"},
+ ]
+ }
+
+ class _Adapter:
+ mode = "local"
+ USE_CONTENT_FIELD = True
+
+ def get_collection(self):
+ return _Collection()
+
+ def update_data(self, data):
+ calls.append(data)
+ return [data[0]["id"]]
+
+ backend = _SingleAccountBackend(
+ config=VectorDBBackendConfig(backend="local", name="context", dimension=2),
+ bound_account_id="acc1",
+ shared_adapter=_Adapter(),
+ )
+
+ result = await backend.update({"id": "rec-1", "search_tags": ["team=search"]})
+
+ assert result.ok is True
+ assert calls == [
+ [
+ {
+ "id": "rec-1",
+ "search_tags": ["team=search"],
+ "account_id": "acc1",
+ }
+ ]
+ ]
+
+
@pytest.mark.asyncio
async def test_local_backend_update_preserves_omitted_fields_end_to_end(tmp_path):
if not getattr(vectordb_engine, "PersistStore", None):
diff --git a/tests/storage/test_content_write_processing_mode.py b/tests/storage/test_content_write_processing_mode.py
index e36f03c47..21cae071a 100644
--- a/tests/storage/test_content_write_processing_mode.py
+++ b/tests/storage/test_content_write_processing_mode.py
@@ -15,6 +15,7 @@ from openviking.storage.abstract_overview import (
)
from openviking.storage.content_write import ContentWriteCoordinator
from openviking.storage.queuefs.semantic_ops.freshness_policy import FreshnessAction
+from openviking.utils.content_hash import content_md5
from openviking.utils.ingest_options import IngestOptions
from openviking_cli.session.user_id import UserIdentifier
@@ -116,6 +117,7 @@ async def test_direct_write_skips_semantic_refresh_for_vectors_only_and_sidecar_
"name": "demo.md",
"summary": "",
}
+ assert vectorize_file.await_args.kwargs["file_md5"] == content_md5(b"updated")
assert "register_request_wait" not in vectorize_file.await_args.kwargs
assert result["semantic_status"] == "skipped"
assert result["vector_status"] == "queued"
@@ -160,6 +162,134 @@ async def test_direct_write_skips_semantic_refresh_for_vectors_only_and_sidecar_
assert sidecar_result["vector_status"] == "queued"
+async def test_direct_write_passes_final_md5_and_old_abstract_to_semantic_refresh(ctx):
+ file_uri = "viking://resources/demo.py"
+
+ class _VikingDB:
+ async def get_l2_diff_records_by_uris(self, uris, *, ctx):
+ assert uris == [file_uri]
+ return {file_uri: {"md5": "old-md5", "abstract": "old abstract"}}
+
+ fake_fs = _FakeVikingFS()
+ coordinator = ContentWriteCoordinator(viking_fs=fake_fs, vikingdb=_VikingDB())
+ enqueue = AsyncMock(return_value=FreshnessAction.REFRESH_NOW)
+ coordinator._enqueue_semantic_refresh = enqueue
+
+ await coordinator._write_direct_with_refresh(
+ uri=file_uri,
+ root_uri="viking://resources",
+ content="updated",
+ mode="replace",
+ context_type="resource",
+ wait=False,
+ timeout=None,
+ ctx=ctx,
+ written_bytes=7,
+ telemetry_id="",
+ )
+
+ assert enqueue.await_args.kwargs["file_md5"] == content_md5(b"updated")
+ assert enqueue.await_args.kwargs["file_abstract"] == "old abstract"
+
+
+async def test_semantic_message_carries_file_md5_and_old_abstract(monkeypatch, ctx):
+ file_uri = "viking://resources/demo.py"
+ queue = SimpleNamespace(enqueue=AsyncMock(return_value="enqueued"))
+ manager = SimpleNamespace(SEMANTIC="Semantic", get_queue=lambda *args, **kwargs: queue)
+ monkeypatch.setattr(content_write_module, "get_queue_manager", lambda: manager)
+ monkeypatch.setattr(
+ content_write_module,
+ "plan_abstract_overview_refresh",
+ AsyncMock(return_value=SimpleNamespace(action=FreshnessAction.REFRESH_NOW)),
+ )
+ coordinator = ContentWriteCoordinator(viking_fs=_FakeVikingFS())
+
+ await coordinator._enqueue_semantic_refresh_changes(
+ root_uri="viking://resources",
+ context_type="resource",
+ changes={"modified": [file_uri]},
+ ctx=ctx,
+ file_md5s={file_uri: "new-md5"},
+ file_abstracts={file_uri: "old abstract"},
+ )
+
+ msg = queue.enqueue.await_args.args[0]
+ assert msg.file_md5s == {file_uri: "new-md5"}
+ assert msg.file_abstracts == {file_uri: "old abstract"}
+
+
+async def test_direct_append_passes_md5_of_final_content_without_vector_db(ctx):
+ fake_fs = _FakeVikingFS()
+ fake_fs.read_file.return_value = "previous"
+ coordinator = ContentWriteCoordinator(viking_fs=fake_fs)
+ enqueue = AsyncMock(return_value=FreshnessAction.REFRESH_NOW)
+ coordinator._enqueue_semantic_refresh = enqueue
+
+ await coordinator._write_direct_with_refresh(
+ uri="viking://resources/demo.py",
+ root_uri="viking://resources",
+ content=" updated",
+ mode="append",
+ context_type="resource",
+ wait=False,
+ timeout=None,
+ ctx=ctx,
+ written_bytes=8,
+ telemetry_id="",
+ )
+
+ assert enqueue.await_args.kwargs["file_md5"] == content_md5(b"previous updated")
+ assert enqueue.await_args.kwargs["file_abstract"] == ""
+
+
+async def test_direct_write_continues_when_old_abstract_lookup_fails(ctx):
+ class _FailingVikingDB:
+ async def get_l2_diff_records_by_uris(self, uris, *, ctx):
+ raise RuntimeError("vector lookup unavailable")
+
+ fake_fs = _FakeVikingFS()
+ coordinator = ContentWriteCoordinator(viking_fs=fake_fs, vikingdb=_FailingVikingDB())
+ enqueue = AsyncMock(return_value=FreshnessAction.REFRESH_NOW)
+ coordinator._enqueue_semantic_refresh = enqueue
+
+ await coordinator._write_direct_with_refresh(
+ uri="viking://resources/demo.py",
+ root_uri="viking://resources",
+ content="updated",
+ mode="replace",
+ context_type="resource",
+ wait=False,
+ timeout=None,
+ ctx=ctx,
+ written_bytes=7,
+ telemetry_id="",
+ )
+
+ assert enqueue.await_args.kwargs["file_abstract"] == ""
+ assert enqueue.await_args.kwargs["file_md5"] == content_md5(b"updated")
+
+
+async def test_old_abstract_lookup_uses_viking_fs_vector_store_when_not_injected(ctx):
+ file_uri = "viking://agent/skills/demo/SKILL.md"
+
+ class _VectorStore:
+ async def get_l2_diff_records_by_uris(self, uris, *, ctx):
+ return {file_uri: {"abstract": "skill summary"}}
+
+ fake_fs = _FakeVikingFS()
+ fake_fs._get_vector_store = lambda: _VectorStore()
+ coordinator = ContentWriteCoordinator(viking_fs=fake_fs)
+
+ assert await coordinator._load_file_abstracts([file_uri], ctx=ctx) == {
+ file_uri: "skill summary"
+ }
+
+
@pytest.mark.asyncio
async def test_write_builds_ingest_options_before_scheduling_resource_refresh(ctx):
coordinator = ContentWriteCoordinator(viking_fs=_FakeVikingFS())
diff --git a/tests/storage/test_context_update_plan.py b/tests/storage/test_context_update_plan.py
new file mode 100644
index 000000000..e3538aed1
--- /dev/null
+++ b/tests/storage/test_context_update_plan.py
@@ -0,0 +1,3143 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+import asyncio
+from contextlib import nullcontext
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, MagicMock, Mock
+
+import pytest
+
+from openviking.storage import resource_diff
+from openviking.storage.index_action import FieldPatch
+
+
+def test_context_plan_has_explicit_actions_and_compact_semantic_roundtrip():
+ from openviking.storage.context_update_plan import (
+ ContextUpdatePlan,
+ IndexSlot,
+ SemanticAction,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+ )
+
+ entry = SemanticTreeEntry(
+ relative_path="a.py",
+ kind="file",
+ content_state="modified",
+ semantic_action=SemanticAction.GENERATE,
+ md5="new",
+ index_slots=(
+ IndexSlot(
+ level=2,
+ record_id="external-id",
+ existing_fields={"abstract": "old"},
+ action="upsert",
+ upsert_fields={"search_tags": ["new"]},
+ ),
+ ),
+ )
+ plan = ContextUpdatePlan(
+ root_uri="viking://resources/repo",
+ context_type="resource",
+ semantic_plan=SemanticPlan(
+ root_uri="viking://resources/repo",
+ context_type="resource",
+ tree=SemanticTreeSnapshot(
+ (
+ SemanticTreeEntry("", "directory", "unchanged", "aggregate"),
+ entry,
+ )
+ ),
+ ),
+ )
+ payload = plan.to_dict()
+ restored = ContextUpdatePlan.from_dict(payload)
+ assert restored == plan
+ node = next(
+ item
+ for item in payload["semantic_plan"]["tree"]["entries"]
+ if item["relative_path"] == "a.py"
+ )
+ assert "indexed_records" not in node
+ assert "previous_abstracts" not in node
+ assert "uri" not in node["index_slots"][0]
+ assert node["index_slots"][0]["record_id"] == "external-id"
+
+
+def test_index_actions_reject_unconsumed_field_patch_state():
+ from openviking.storage.context_update_plan import DirectIndexAction, IndexSlot
+
+ patch = FieldPatch({"search_tags": ["scope=new"]})
+ with pytest.raises(ValueError, match="upsert uses resolved upsert_fields"):
+ DirectIndexAction(
+ "upsert",
+ "viking://resources/repo/a.py",
+ 2,
+ "a-l2",
+ field_patch=patch,
+ )
+ with pytest.raises(ValueError, match="fallback_to_patch requires"):
+ IndexSlot(2, "a-l2", action="upsert", fallback_to_patch=True)
+
+
+def test_context_plan_reads_legacy_flat_patch_fields():
+ from openviking.storage.context_update_plan import ContextUpdatePlan
+
+ restored = ContextUpdatePlan.from_dict(
+ {
+ "root_uri": "viking://resources/repo",
+ "context_type": "resource",
+ "direct_index_actions": [
+ {
+ "action": "update_fields",
+ "uri": "viking://resources/repo/a.py",
+ "level": 2,
+ "record_id": "a-l2",
+ "fields": {"search_tags": ["scope=new"]},
+ "field_modes": {"search_tags": "append"},
+ "initial_fields": {"vector": [0.1, 0.2]},
+ "search_tag_mode": "append",
+ }
+ ],
+ }
+ )
+
+ action = restored.direct_index_actions[0]
+ assert action.upsert_fields == {}
+ assert action.field_patch == FieldPatch(
+ {"search_tags": ["scope=new"]},
+ {"search_tags": "append"},
+ {"vector": [0.1, 0.2]},
+ )
+
+
+def test_legacy_semantic_slot_drops_inert_fallback_without_patch():
+ from openviking.storage.context_update_plan import IndexSlot
+
+ slot = IndexSlot.from_dict(
+ {
+ "level": 2,
+ "record_id": "a-l2",
+ "action": "upsert",
+ "fields": {},
+ "update_fields": {},
+ "field_modes": {},
+ "fallback_update_fields": True,
+ }
+ )
+
+ assert slot.field_patch is None
+ assert slot.fallback_to_patch is False
+
+
+def test_after_content_commit_keeps_only_derived_actions():
+ from openviking.storage.context_update_plan import (
+ ContentTreeAction,
+ ContextUpdatePlan,
+ DirectIndexAction,
+ )
+
+ plan = ContextUpdatePlan(
+ "viking://resources/repo",
+ "resource",
+ content_tree_actions=(
+ ContentTreeAction(
+ "upsert",
+ "a.py",
+ new_kind="file",
+ artifact_path="repository/a.py",
+ md5="new",
+ ),
+ ),
+ direct_index_actions=(
+ DirectIndexAction("delete", "viking://resources/repo/old.py", 2, "old-l2"),
+ ),
+ )
+
+ committed = plan.after_content_commit()
+
+ assert committed.content_tree_actions == ()
+ assert committed.semantic_plan is plan.semantic_plan
+ assert committed.direct_index_actions is plan.direct_index_actions
+
+
+def test_context_plan_rejects_conflicting_record_operations():
+ from openviking.storage.context_update_plan import (
+ ContextUpdatePlan,
+ DirectIndexAction,
+ IndexSlot,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+ )
+
+ semantic = SemanticPlan(
+ root_uri="viking://resources/repo",
+ context_type="resource",
+ tree=SemanticTreeSnapshot(
+ (
+ SemanticTreeEntry("", "directory", "unchanged", "aggregate"),
+ SemanticTreeEntry(
+ "a.py",
+ "file",
+ "modified",
+ "generate",
+ md5="new",
+ index_slots=(
+ IndexSlot(
+ 2,
+ "same-id",
+ {"abstract": "old"},
+ action="upsert",
+ ),
+ ),
+ ),
+ )
+ ),
+ )
+ with pytest.raises(ValueError, match="conflict"):
+ ContextUpdatePlan(
+ root_uri=semantic.root_uri,
+ context_type="resource",
+ semantic_plan=semantic,
+ direct_index_actions=(
+ DirectIndexAction("delete", semantic.root_uri + "/a.py", 2, "same-id"),
+ ),
+ )
+
+
+def test_semantic_plan_rejects_disconnected_or_mistyped_actions():
+ from openviking.storage.context_update_plan import (
+ IndexSlot,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+ )
+
+ with pytest.raises(ValueError, match="root"):
+ SemanticPlan(
+ "viking://resources/repo",
+ "resource",
+ SemanticTreeSnapshot(
+ (
+ SemanticTreeEntry(
+ "src/a.py",
+ "file",
+ "modified",
+ "generate",
+ index_slots=(IndexSlot(2, "a-l2", action="upsert"),),
+ ),
+ )
+ ),
+ )
+ with pytest.raises(ValueError, match="directory.*aggregate"):
+ SemanticTreeEntry("src", "directory", "modified", "generate")
+ with pytest.raises(ValueError, match="file.*generate"):
+ SemanticTreeEntry("a.py", "file", "modified", "aggregate", md5="m")
+ with pytest.raises(ValueError, match="ancestor.*aggregate"):
+ SemanticPlan(
+ "viking://resources/repo",
+ "resource",
+ SemanticTreeSnapshot(
+ (
+ SemanticTreeEntry(
+ "",
+ "directory",
+ "unchanged",
+ "reuse",
+ index_slots=(IndexSlot(0, "root-l0", {"abstract": "root"}),),
+ ),
+ SemanticTreeEntry(
+ "a.py",
+ "file",
+ "modified",
+ "generate",
+ md5="new",
+ index_slots=(IndexSlot(2, "a-l2", action="upsert"),),
+ ),
+ )
+ ),
+ )
+
+
+def test_semantic_plan_validates_ancestors_without_rescanning_entries():
+ from openviking.storage.context_update_plan import (
+ IndexSlot,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+ )
+
+ class CountingEntries:
+ def __init__(self, entries):
+ self._entries = tuple(entries)
+ self.iterations = 0
+
+ def __iter__(self):
+ self.iterations += 1
+ return iter(self._entries)
+
+ directories = [
+ "/".join(["src", *[f"level-{index}" for index in range(depth)]]) for depth in range(8)
+ ]
+ entries = CountingEntries(
+ [SemanticTreeEntry("", "directory", "unchanged", "aggregate")]
+ + [SemanticTreeEntry(path, "directory", "unchanged", "aggregate") for path in directories]
+ + [
+ SemanticTreeEntry(
+ directories[-1] + "/a.py",
+ "file",
+ "modified",
+ "generate",
+ md5="new",
+ index_slots=(IndexSlot(2, "a-l2", action="upsert"),),
+ )
+ ]
+ )
+
+ SemanticPlan(
+ "viking://resources/repo",
+ "resource",
+ SemanticTreeSnapshot(entries),
+ )
+
+ assert entries.iterations <= 4
+
+
+def test_semantic_plan_rejects_missing_higher_ancestor_independent_of_entry_order():
+ from openviking.storage.context_update_plan import (
+ IndexSlot,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+ )
+
+ with pytest.raises(ValueError, match="lacks ancestor 'a'.*a/b/c.py"):
+ SemanticPlan(
+ "viking://resources/repo",
+ "resource",
+ SemanticTreeSnapshot(
+ (
+ SemanticTreeEntry("", "directory", "unchanged", "aggregate"),
+ SemanticTreeEntry(
+ "a/b/c.py",
+ "file",
+ "modified",
+ "generate",
+ md5="new",
+ index_slots=(IndexSlot(2, "c-l2", action="upsert"),),
+ ),
+ SemanticTreeEntry("a/b", "directory", "unchanged", "aggregate"),
+ )
+ ),
+ )
+
+
+def test_index_slot_rejects_non_semantic_operations():
+ from openviking.storage.context_update_plan import IndexSlot
+
+ with pytest.raises(ValueError, match="only supports"):
+ IndexSlot(2, "a-l2", action="update_fields")
+
+
+def test_reuse_entry_rejects_index_mutation():
+ from openviking.storage.context_update_plan import IndexSlot, SemanticTreeEntry
+
+ with pytest.raises(ValueError, match="reuse.*index"):
+ SemanticTreeEntry(
+ "a.py",
+ "file",
+ "unchanged",
+ "reuse",
+ index_slots=(
+ IndexSlot(
+ 2,
+ "a-l2",
+ {"abstract": "old"},
+ action="upsert",
+ ),
+ ),
+ )
+
+
+async def test_resolver_compares_missing_fingerprints_with_bounded_reads():
+ from openviking.storage.resource_rnfv import (
+ FormalEntry,
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ NewEntry,
+ RequestIntent,
+ RNFVSnapshot,
+ VectorIndexSnapshot,
+ )
+
+ paths = [f"{i}.py" for i in range(8)]
+ snapshot = RNFVSnapshot(
+ RequestIntent("viking://resources/repo", "semantic_and_vectors"),
+ NewArtifactSnapshot({p: NewEntry() for p in paths}),
+ FormalTreeSnapshot({p: FormalEntry() for p in paths}),
+ VectorIndexSnapshot({}, frozenset({"id", "uri", "level", "md5"})),
+ )
+ active = peak = 0
+ gate = asyncio.Event()
+
+ async def read(*args):
+ nonlocal active, peak
+ active += 1
+ peak = max(peak, active)
+ if active == 4:
+ gate.set()
+ try:
+ await asyncio.wait_for(gate.wait(), 1)
+ return b"same"
+ finally:
+ active -= 1
+
+ resolve = getattr(resource_diff, "resolve_resource_diff", None)
+ assert resolve is not None
+ result = await resolve(
+ snapshot,
+ store=SimpleNamespace(read_bytes=read),
+ artifact_ref=None,
+ target=SimpleNamespace(read_file=read),
+ concurrency=2,
+ )
+ assert peak == 4
+ assert active == 0
+ assert all(
+ e.content_state == "unchanged" and e.index_state == "missing"
+ for e in result.entries.values()
+ )
+ assert all(e.md5 for e in result.entries.values())
+ assert not hasattr(result, "needs_body_compare")
+
+
+async def test_resolver_rejects_incomplete_snapshot_before_io():
+ from openviking.storage.resource_rnfv import (
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ RequestIntent,
+ RNFVSnapshot,
+ VectorIndexSnapshot,
+ )
+
+ snapshot = RNFVSnapshot(
+ RequestIntent("viking://resources/repo", "semantic_and_vectors"),
+ NewArtifactSnapshot({}, complete=False),
+ FormalTreeSnapshot({}),
+ VectorIndexSnapshot({}, frozenset({"id", "uri", "level", "md5"})),
+ )
+ store, target = AsyncMock(), AsyncMock()
+ resolve = getattr(resource_diff, "resolve_resource_diff", None)
+ assert resolve is not None
+ with pytest.raises(ValueError, match="incomplete"):
+ await resolve(snapshot, store=store, artifact_ref=None, target=target)
+ store.read_bytes.assert_not_called()
+ target.read_file.assert_not_called()
+
+
+async def test_resolver_hashes_new_file_when_manifest_md5_is_missing():
+ from openviking.storage.context_update_plan import ContentState
+ from openviking.storage.resource_rnfv import (
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ NewEntry,
+ RequestIntent,
+ RNFVSnapshot,
+ VectorIndexSnapshot,
+ )
+ from openviking.utils.content_hash import content_md5
+
+ snapshot = RNFVSnapshot(
+ RequestIntent("viking://resources/repo", "semantic_and_vectors"),
+ NewArtifactSnapshot({"a.py": NewEntry()}),
+ FormalTreeSnapshot({}),
+ VectorIndexSnapshot({}, frozenset({"id", "uri", "level", "md5"})),
+ )
+ store = AsyncMock()
+ store.read_bytes.return_value = b"new body"
+ result = await resource_diff.resolve_resource_diff(
+ snapshot,
+ store=store,
+ artifact_ref=object(),
+ target=AsyncMock(),
+ artifact_paths={"a.py": "repository/a.py"},
+ )
+
+ assert result.entries["a.py"].content_state is ContentState.ADDED
+ assert result.entries["a.py"].md5 == content_md5(b"new body")
+ store.read_bytes.assert_awaited_once()
+
+
+async def test_resolver_marks_removed_content_vectors_as_obsolete():
+ from openviking.storage.context_update_plan import ContentState, IndexState
+ from openviking.storage.resource_rnfv import (
+ FormalEntry,
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ RequestIntent,
+ RNFVSnapshot,
+ VectorIndexSnapshot,
+ VectorRecordSnapshot,
+ )
+
+ root = "viking://resources/repo"
+ records = {
+ "removed-l2": VectorRecordSnapshot("removed-l2", f"{root}/removed.md", "removed.md", 2),
+ "orphan-l2": VectorRecordSnapshot("orphan-l2", f"{root}/orphan.md", "orphan.md", 2),
+ }
+ snapshot = RNFVSnapshot(
+ RequestIntent(root, "semantic_and_vectors"),
+ NewArtifactSnapshot({}),
+ FormalTreeSnapshot({"removed.md": FormalEntry()}),
+ VectorIndexSnapshot(records, frozenset({"id", "uri", "level", "md5"})),
+ )
+
+ result = await resource_diff.resolve_resource_diff(
+ snapshot, store=AsyncMock(), artifact_ref=object(), target=AsyncMock()
+ )
+
+ assert result.entries["removed.md"].content_state is ContentState.DELETED
+ assert result.entries["removed.md"].index_state is IndexState.OBSOLETE
+ assert result.entries["orphan.md"].content_state is ContentState.ABSENT
+ assert result.entries["orphan.md"].index_state is IndexState.ORPHAN
+
+
+async def test_resolver_uses_same_duplicate_record_winner_as_planner():
+ from openviking.storage.context_update_plan import ContentState
+ from openviking.storage.resource_rnfv import (
+ FormalEntry,
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ NewEntry,
+ RequestIntent,
+ RNFVSnapshot,
+ VectorIndexSnapshot,
+ VectorRecordSnapshot,
+ )
+
+ root = "viking://resources/repo"
+ records = {
+ "z-new": VectorRecordSnapshot("z-new", f"{root}/a.py", "a.py", 2, {"md5": "new"}),
+ "a-old": VectorRecordSnapshot("a-old", f"{root}/a.py", "a.py", 2, {"md5": "old"}),
+ }
+ snapshot = RNFVSnapshot(
+ RequestIntent(root, "semantic_and_vectors"),
+ NewArtifactSnapshot({"a.py": NewEntry(md5="new")}),
+ FormalTreeSnapshot({"a.py": FormalEntry()}),
+ VectorIndexSnapshot(records, frozenset({"id", "uri", "level", "md5"})),
+ )
+
+ result = await resource_diff.resolve_resource_diff(
+ snapshot, store=AsyncMock(), artifact_ref=object(), target=AsyncMock()
+ )
+
+ assert result.entries["a.py"].content_state is ContentState.MODIFIED
+
+
+async def test_resolver_index_state_uses_canonical_duplicate_record():
+ from openviking.storage.context_update_plan import ContentState, IndexState
+ from openviking.storage.resource_rnfv import (
+ FormalEntry,
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ NewEntry,
+ RequestIntent,
+ RNFVSnapshot,
+ VectorIndexSnapshot,
+ VectorRecordSnapshot,
+ )
+
+ root = "viking://resources/repo"
+ records = {
+ "z-new": VectorRecordSnapshot("z-new", f"{root}/a.py", "a.py", 2, {"md5": "new"}),
+ "a-old": VectorRecordSnapshot("a-old", f"{root}/a.py", "a.py", 2, {"md5": "old"}),
+ }
+ snapshot = RNFVSnapshot(
+ RequestIntent(root, "semantic_and_vectors"),
+ NewArtifactSnapshot({"a.py": NewEntry(md5="old")}),
+ FormalTreeSnapshot({"a.py": FormalEntry()}),
+ VectorIndexSnapshot(records, frozenset({"id", "uri", "level", "md5"})),
+ )
+
+ result = await resource_diff.resolve_resource_diff(
+ snapshot, store=AsyncMock(), artifact_ref=object(), target=AsyncMock()
+ )
+
+ entry = result.entries["a.py"]
+ assert entry.content_state is ContentState.UNCHANGED
+ assert entry.index_state is IndexState.COMPLETE
+
+
+async def test_resolver_log_separates_files_directories_and_logical_root(monkeypatch):
+ from openviking.storage import resource_diff
+ from openviking.storage.resource_rnfv import (
+ FormalEntry,
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ NewEntry,
+ RequestIntent,
+ RNFVSnapshot,
+ VectorIndexSnapshot,
+ VectorRecordSnapshot,
+ )
+
+ root = "viking://resources/repo"
+ records = {
+ "root-l0": VectorRecordSnapshot("root-l0", root, "", 0),
+ "root-l1": VectorRecordSnapshot("root-l1", root, "", 1),
+ "file-l2": VectorRecordSnapshot("file-l2", f"{root}/a.py", "a.py", 2, {"md5": "same"}),
+ "sub-l0": VectorRecordSnapshot("sub-l0", f"{root}/sub", "sub", 0),
+ "sub-l1": VectorRecordSnapshot("sub-l1", f"{root}/sub", "sub", 1),
+ }
+ snapshot = RNFVSnapshot(
+ RequestIntent(root, "semantic_and_vectors"),
+ NewArtifactSnapshot(
+ {"": NewEntry(is_dir=True), "a.py": NewEntry(md5="same"), "sub": NewEntry(is_dir=True)}
+ ),
+ FormalTreeSnapshot(
+ {"": FormalEntry(is_dir=True), "a.py": FormalEntry(), "sub": FormalEntry(is_dir=True)}
+ ),
+ VectorIndexSnapshot(records, frozenset({"id", "uri", "level", "md5"})),
+ )
+
+ log_info = Mock()
+ monkeypatch.setattr(resource_diff.logger, "info", log_info)
+ await resource_diff.resolve_resource_diff(
+ snapshot,
+ store=AsyncMock(),
+ artifact_ref=object(),
+ target=AsyncMock(),
+ )
+
+ message = log_info.call_args.args[0] % log_info.call_args.args[1:]
+ fixed_fields = "n_files=1 n_dirs=1 f_files=1 f_dirs=1 logical_root=true v_records=5 root_state=unchanged root_index=complete md5_fast_path=1 body_compared=0 new_files_hashed=0"
+ assert fixed_fields in message
+ assert "states={file:{'unchanged': 1},dir:{'unchanged': 1}}" in message
+ assert "index={file:{'complete': 1},dir:{'complete': 1}}" in message
+ assert message.index("md5_fast_path=1") < message.index("states=")
+ assert "n_entries=" not in message
+
+
+def test_tree_entry_log_counts_empty_path_as_a_real_flat_file():
+ from openviking.storage.resource_diff import count_tree_entry_kinds
+ from openviking.storage.resource_rnfv import FormalEntry, NewEntry
+
+ assert count_tree_entry_kinds({"": NewEntry(md5="digest")}) == (1, 0, False)
+ assert count_tree_entry_kinds({"": FormalEntry()}) == (1, 0, False)
+
+
+def test_planning_details_log_at_debug(monkeypatch):
+ from openviking.storage.context_update_plan import ContextUpdatePlan
+ from openviking.storage.resource_diff import ResourceDiffResult
+ from openviking.utils import resource_processor
+
+ log_info = Mock()
+ log_debug = Mock()
+ monkeypatch.setattr(resource_processor.logger, "info", log_info)
+ monkeypatch.setattr(resource_processor.logger, "debug", log_debug)
+ plan = ContextUpdatePlan("viking://resources/repo", "resource")
+
+ resource_processor.ResourceProcessor._log_context_update_plan(plan)
+ resource_processor.ResourceProcessor._log_context_commit_summary(
+ ResourceDiffResult({}),
+ content_actions=(),
+ root_uri=plan.root_uri,
+ is_initial=False,
+ )
+ resource_processor.ResourceProcessor._log_context_commit_summary(
+ ResourceDiffResult({}),
+ content_actions=(),
+ root_uri=plan.root_uri,
+ is_initial=True,
+ )
+
+ assert log_info.call_count == 0
+ assert log_debug.call_count == 3
+
+
+def test_content_commit_log_uses_info_only_for_mutations(monkeypatch):
+ from openviking.storage.context_update_plan import (
+ ContentTreeAction,
+ ContentTreeOperation,
+ )
+ from openviking.utils import resource_processor
+
+ log_info = Mock()
+ log_debug = Mock()
+ monkeypatch.setattr(resource_processor.logger, "info", log_info)
+ monkeypatch.setattr(resource_processor.logger, "debug", log_debug)
+
+ resource_processor.ResourceProcessor._log_content_tree_commit(
+ root_uri="viking://resources/repo",
+ artifact_backend="local",
+ actions=(),
+ duration_ms=1.0,
+ )
+ resource_processor.ResourceProcessor._log_content_tree_commit(
+ root_uri="viking://resources/repo",
+ artifact_backend="local",
+ actions=(
+ ContentTreeAction(
+ ContentTreeOperation.UPSERT,
+ "a.py",
+ new_kind="file",
+ artifact_path="a.py",
+ md5="digest",
+ ),
+ ),
+ duration_ms=2.0,
+ )
+
+ log_debug.assert_called_once()
+ log_info.assert_called_once()
+
+
+def test_add_resource_log_level_contract():
+ import ast
+ from pathlib import Path
+
+ root = Path(__file__).resolve().parents[2]
+ expected = {
+ "openviking/storage/queuefs/add_resource_processor.py": {
+ "[AddResourceStarted]": "info",
+ "[AddResourceCompleted]": "info",
+ },
+ "openviking/storage/resource_diff.py": {
+ "[ResourceDiffResult]": "info",
+ },
+ "openviking/utils/resource_processor.py": {
+ "[RNFVSnapshot]": "debug",
+ "[ContextUpdatePlan]": "debug",
+ "[add_resource]": "debug",
+ "[DirectIndexActions]": "debug",
+ },
+ "openviking/storage/queuefs/semantic_processor.py": {
+ "[SemanticPlanExecution]": "debug",
+ },
+ "openviking/parse/parsers/code/code.py": {
+ "Uploading code repository artifacts": "debug",
+ },
+ "openviking/parse/parsers/media/image.py": {
+ "Processing large image": "debug",
+ },
+ }
+
+ for relative_path, markers in expected.items():
+ tree = ast.parse(root.joinpath(relative_path).read_text())
+ found = {}
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.Call) or not isinstance(node.func, ast.Attribute):
+ continue
+ if not node.args:
+ continue
+ message = "".join(
+ part.value
+ for part in ast.walk(node.args[0])
+ if isinstance(part, ast.Constant) and isinstance(part.value, str)
+ )
+ for marker in markers:
+ if marker in message:
+ found[marker] = node.func.attr
+ assert found == markers
+
+
+def test_builder_maps_content_semantic_and_direct_index_actions():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ ContextUpdatePlan,
+ IndexAction,
+ IndexState,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot
+
+ diff = ResourceDiffResult(
+ entries={
+ "changed.py": ResourceDiffEntry(
+ "changed.py",
+ ContentState.MODIFIED,
+ IndexState.STALE,
+ old_kind="file",
+ new_kind="file",
+ md5="new-md5",
+ ),
+ "gone.py": ResourceDiffEntry(
+ "gone.py", ContentState.DELETED, IndexState.OBSOLETE, old_kind="file"
+ ),
+ "ghost.py": ResourceDiffEntry("ghost.py", ContentState.ABSENT, IndexState.ORPHAN),
+ }
+ )
+ inventory = {
+ "changed-id": VectorRecordSnapshot(
+ "changed-id",
+ "viking://resources/repo/changed.py",
+ "changed.py",
+ 2,
+ {"md5": "old-md5", "abstract": "old abstract", "search_tags": ["old"]},
+ ),
+ "gone-id": VectorRecordSnapshot("gone-id", "viking://resources/repo/gone.py", "gone.py", 2),
+ "ghost-id": VectorRecordSnapshot(
+ "ghost-id", "viking://resources/repo/ghost.py", "ghost.py", 2
+ ),
+ }
+
+ from openviking.storage.context_update_plan import build_context_update_plan
+
+ plan = build_context_update_plan(
+ root_uri="viking://resources/repo",
+ context_type="resource",
+ request=RequestIntent("viking://resources/repo", "semantic_and_vectors"),
+ diff=diff,
+ new_kinds={"": "directory", "changed.py": "file"},
+ artifact_paths={"changed.py": "repository/changed.py"},
+ records=inventory,
+ is_code_repo=True,
+ account_id="acc",
+ )
+
+ assert isinstance(plan, ContextUpdatePlan)
+ assert [(a.operation.value, a.relative_path) for a in plan.content_tree_actions] == [
+ ("upsert", "changed.py"),
+ ("delete", "gone.py"),
+ ]
+ assert {(a.action.value, a.record_id) for a in plan.direct_index_actions} == {
+ ("delete", "gone-id"),
+ ("delete", "ghost-id"),
+ }
+ changed = next(e for e in plan.semantic_plan.tree.entries if e.relative_path == "changed.py")
+ assert changed.semantic_action.value == "generate"
+ assert changed.index_slots[0].action == IndexAction.UPSERT
+ assert changed.index_slots[0].record_id == "changed-id"
+
+
+def test_level_conflict_deletes_invalid_record_and_rebuilds_valid_slots():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ IndexState,
+ build_context_update_plan,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot
+
+ root = "viking://resources/repo"
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(root, "semantic_and_vectors"),
+ diff=ResourceDiffResult(
+ {
+ "module": ResourceDiffEntry(
+ "module",
+ ContentState.UNCHANGED,
+ IndexState.LEVEL_CONFLICT,
+ old_kind="directory",
+ new_kind="directory",
+ )
+ }
+ ),
+ new_kinds={"": "directory", "module": "directory"},
+ artifact_paths={},
+ records={"stale-l2": VectorRecordSnapshot("stale-l2", f"{root}/module", "module", 2)},
+ is_code_repo=False,
+ account_id="acc",
+ )
+
+ assert [(a.action.value, a.record_id) for a in plan.direct_index_actions] == [
+ ("delete", "stale-l2")
+ ]
+ entry = next(
+ entry for entry in plan.semantic_plan.tree.entries if entry.relative_path == "module"
+ )
+ assert entry.semantic_action.value == "aggregate"
+ assert [(slot.level, slot.action.value) for slot in entry.index_slots] == [
+ (0, "upsert"),
+ (1, "upsert"),
+ ]
+
+
+def test_builder_deletes_duplicate_same_level_records_without_id_conflict():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ IndexState,
+ build_context_update_plan,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot
+
+ root = "viking://resources/repo"
+ records = {
+ record_id: VectorRecordSnapshot(
+ record_id,
+ root + "/a.py",
+ "a.py",
+ 2,
+ {"md5": "old", "abstract": "old"},
+ )
+ for record_id in ("a-primary", "a-duplicate")
+ }
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(root, "semantic_and_vectors"),
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ ContentState.MODIFIED,
+ IndexState.STALE,
+ old_kind="file",
+ new_kind="file",
+ md5="new",
+ )
+ }
+ ),
+ new_kinds={"": "directory", "a.py": "file"},
+ artifact_paths={"a.py": "repository/a.py"},
+ records=records,
+ is_code_repo=True,
+ account_id="acc",
+ )
+
+ slot = next(
+ entry for entry in plan.semantic_plan.tree.entries if entry.relative_path == "a.py"
+ ).slot(2)
+ assert slot.record_id in records
+ assert [(action.action.value, action.record_id) for action in plan.direct_index_actions] == [
+ ("delete", ({"a-primary", "a-duplicate"} - {slot.record_id}).pop())
+ ]
+
+
+def test_healthy_noop_produces_no_actions_or_semantic_plan():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ IndexState,
+ build_context_update_plan,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot
+
+ root = "viking://resources/repo"
+ records = {
+ "root-l0": VectorRecordSnapshot("root-l0", root, "", 0, {"abstract": "root abstract"}),
+ "root-l1": VectorRecordSnapshot("root-l1", root, "", 1, {"abstract": "root overview"}),
+ "a-l2": VectorRecordSnapshot(
+ "a-l2",
+ root + "/a.py",
+ "a.py",
+ 2,
+ {"md5": "same", "abstract": "a summary"},
+ ),
+ }
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(root, "semantic_and_vectors"),
+ diff=ResourceDiffResult(
+ {
+ "": ResourceDiffEntry(
+ "",
+ ContentState.UNCHANGED,
+ IndexState.COMPLETE,
+ old_kind="directory",
+ new_kind="directory",
+ ),
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ ContentState.UNCHANGED,
+ IndexState.COMPLETE,
+ old_kind="file",
+ new_kind="file",
+ md5="same",
+ ),
+ }
+ ),
+ new_kinds={"": "directory", "a.py": "file"},
+ artifact_paths={"a.py": "repository/a.py"},
+ records=records,
+ is_code_repo=True,
+ account_id="acc",
+ )
+
+ assert plan.is_noop()
+ assert plan.content_tree_actions == ()
+ assert plan.semantic_plan is None
+ assert plan.direct_index_actions == ()
+
+
+async def test_vectorize_disabled_plan_ignores_stale_vectors_and_compares_formal_content():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ build_context_update_plan_from_snapshot,
+ )
+ from openviking.storage.resource_rnfv import (
+ FormalEntry,
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ NewEntry,
+ RequestIntent,
+ RNFVSnapshot,
+ VectorIndexSnapshot,
+ VectorRecordSnapshot,
+ )
+
+ root = "viking://resources/repo"
+ store = AsyncMock()
+ store.read_bytes.return_value = b"v1"
+ target = AsyncMock()
+ target.read_file.return_value = b"v2"
+ vikingdb = AsyncMock()
+ snapshot = RNFVSnapshot(
+ RequestIntent(root, "vectors_only", vectorize=False),
+ NewArtifactSnapshot({"a.txt": NewEntry(md5="hash-v1")}),
+ FormalTreeSnapshot({"a.txt": FormalEntry()}),
+ VectorIndexSnapshot(
+ {
+ "stale-l2": VectorRecordSnapshot(
+ "stale-l2",
+ "viking://resources/other/a.txt",
+ "a.txt",
+ 2,
+ {"md5": "hash-v1", "abstract": "stale summary"},
+ )
+ },
+ frozenset({"id", "uri", "level", "md5", "abstract"}),
+ complete=False,
+ ),
+ )
+
+ diff, plan = await build_context_update_plan_from_snapshot(
+ snapshot=snapshot,
+ store=store,
+ artifact_ref=object(),
+ target=target,
+ vikingdb=vikingdb,
+ context_type="resource",
+ is_code_repo=False,
+ account_id="acc",
+ ctx=object(),
+ root_preexisting=True,
+ artifact_paths={"a.txt": "repository/a.txt"},
+ )
+
+ assert diff.entries["a.txt"].content_state is ContentState.MODIFIED
+ assert [
+ (action.operation.value, action.relative_path) for action in plan.content_tree_actions
+ ] == [("upsert", "a.txt")]
+ assert plan.semantic_plan is None
+ assert plan.direct_index_actions == ()
+ store.read_bytes.assert_awaited_once()
+ target.read_file.assert_awaited_once_with("a.txt")
+ vikingdb.hydrate_incremental_records.assert_not_awaited()
+
+
+async def test_vectorize_enabled_keeps_md5_noop_without_formal_content_reads():
+ from openviking.storage.context_update_plan import ContentState
+ from openviking.storage.resource_diff import resolve_resource_diff
+ from openviking.storage.resource_rnfv import (
+ FormalEntry,
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ NewEntry,
+ RequestIntent,
+ RNFVSnapshot,
+ VectorIndexSnapshot,
+ VectorRecordSnapshot,
+ )
+
+ root = "viking://resources/repo"
+ snapshot = RNFVSnapshot(
+ RequestIntent(root, "semantic_and_vectors", vectorize=True),
+ NewArtifactSnapshot({"a.txt": NewEntry(md5="same")}),
+ FormalTreeSnapshot({"a.txt": FormalEntry()}),
+ VectorIndexSnapshot(
+ {"a-l2": VectorRecordSnapshot("a-l2", root + "/a.txt", "a.txt", 2, {"md5": "same"})},
+ frozenset({"id", "uri", "level", "md5"}),
+ ),
+ )
+ store = AsyncMock()
+ target = AsyncMock()
+
+ diff = await resolve_resource_diff(
+ snapshot,
+ store=store,
+ artifact_ref=object(),
+ target=target,
+ artifact_paths={"a.txt": "repository/a.txt"},
+ )
+
+ assert diff.entries["a.txt"].content_state is ContentState.UNCHANGED
+ store.read_bytes.assert_not_awaited()
+ target.read_file.assert_not_awaited()
+
+
+async def test_vectorize_disabled_semantic_plan_regenerates_without_vector_hydration():
+ from openviking.storage.context_update_plan import build_context_update_plan_from_snapshot
+ from openviking.storage.resource_rnfv import (
+ FormalEntry,
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ NewEntry,
+ RequestIntent,
+ RNFVSnapshot,
+ ScalarIntent,
+ VectorIndexSnapshot,
+ )
+
+ root = "viking://resources/repo"
+ store = AsyncMock()
+ store.read_bytes.side_effect = lambda _ref, path: {
+ "repository/a.txt": b"new a",
+ "repository/b.txt": b"same b",
+ }[path]
+ target = AsyncMock()
+ target.read_file.side_effect = lambda path: {
+ "a.txt": b"old a",
+ "b.txt": b"same b",
+ }[path]
+ vikingdb = AsyncMock()
+ snapshot = RNFVSnapshot(
+ RequestIntent(
+ root,
+ "semantic_and_vectors",
+ vectorize=False,
+ scalar_intents=(ScalarIntent("search_tags", "replace", ("team=search",)),),
+ ),
+ NewArtifactSnapshot(
+ {
+ "a.txt": NewEntry(md5="new-a"),
+ "b.txt": NewEntry(md5="same-b"),
+ }
+ ),
+ FormalTreeSnapshot({"a.txt": FormalEntry(), "b.txt": FormalEntry()}),
+ VectorIndexSnapshot({}, frozenset()),
+ )
+
+ _, plan = await build_context_update_plan_from_snapshot(
+ snapshot=snapshot,
+ store=store,
+ artifact_ref=object(),
+ target=target,
+ vikingdb=vikingdb,
+ context_type="resource",
+ is_code_repo=False,
+ account_id="acc",
+ ctx=object(),
+ root_preexisting=True,
+ artifact_paths={
+ "a.txt": "repository/a.txt",
+ "b.txt": "repository/b.txt",
+ },
+ )
+
+ assert plan.semantic_plan is not None
+ assert {
+ entry.relative_path: entry.semantic_action.value
+ for entry in plan.semantic_plan.tree.entries
+ } == {"": "aggregate", "a.txt": "generate", "b.txt": "generate"}
+ assert all(
+ slot.action.value == "none"
+ for entry in plan.semantic_plan.tree.entries
+ for slot in entry.index_slots
+ )
+ assert all(
+ not slot.upsert_fields and slot.field_patch is None
+ for entry in plan.semantic_plan.tree.entries
+ for slot in entry.index_slots
+ )
+ assert all(not entry.repair for entry in plan.semantic_plan.tree.entries)
+ assert plan.semantic_plan.ingest_options.search_tags is None
+ assert plan.direct_index_actions == ()
+ vikingdb.hydrate_incremental_records.assert_not_awaited()
+
+
+def test_restore_with_matching_index_reuses_file_summary_and_aggregates_parent():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ IndexState,
+ build_context_update_plan,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot
+
+ root = "viking://resources/repo"
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(root, "semantic_and_vectors"),
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ ContentState.RESTORE,
+ IndexState.COMPLETE,
+ new_kind="file",
+ md5="same",
+ )
+ }
+ ),
+ new_kinds={"": "directory", "a.py": "file"},
+ artifact_paths={"a.py": "repository/a.py"},
+ records={
+ "a-l2": VectorRecordSnapshot(
+ "a-l2",
+ root + "/a.py",
+ "a.py",
+ 2,
+ {"md5": "same", "abstract": "old summary"},
+ ),
+ "root-l0": VectorRecordSnapshot("root-l0", root, "", 0, {"abstract": "old root"}),
+ "root-l1": VectorRecordSnapshot("root-l1", root, "", 1, {"abstract": "old overview"}),
+ },
+ is_code_repo=True,
+ account_id="acc",
+ )
+
+ entries = {entry.relative_path: entry for entry in plan.semantic_plan.tree.entries}
+ assert entries["a.py"].semantic_action.value == "reuse"
+ assert entries[""].semantic_action.value == "aggregate"
+ assert entries[""].membership_changed is True
+
+
+def test_builder_keeps_append_intent_as_fallback_for_semantic_upsert():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ IndexState,
+ build_context_update_plan,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent, ScalarIntent, VectorRecordSnapshot
+
+ root = "viking://resources/repo"
+ record = VectorRecordSnapshot(
+ "id-a",
+ root + "/a.py",
+ "a.py",
+ 2,
+ {"md5": "old", "abstract": "old abstract", "search_tags": ["scope=old"]},
+ )
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(
+ root,
+ "semantic_and_vectors",
+ scalar_intents=(ScalarIntent("search_tags", "append", ("owner=new",)),),
+ ),
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ ContentState.MODIFIED,
+ IndexState.STALE,
+ old_kind="file",
+ new_kind="file",
+ md5="new",
+ ),
+ }
+ ),
+ new_kinds={"": "directory", "a.py": "file"},
+ artifact_paths={"a.py": "repository/a.py"},
+ records={"id-a": record},
+ is_code_repo=True,
+ account_id="acc",
+ )
+ slot = next(
+ e for e in plan.semantic_plan.tree.entries if e.relative_path == "a.py"
+ ).index_slots[0]
+ assert slot.upsert_fields == {"search_tags": ["scope=old", "owner=new"]}
+ assert slot.field_patch == FieldPatch(
+ {"search_tags": ["owner=new"]},
+ {"search_tags": "append"},
+ )
+ assert slot.fallback_to_patch is True
+ assert plan.semantic_plan.ingest_options.search_tags is None
+
+
+def test_builder_skips_scalar_update_when_vectorization_is_disabled():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ IndexState,
+ build_context_update_plan,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import (
+ RequestIntent,
+ ScalarIntent,
+ VectorRecordSnapshot,
+ )
+
+ root = "viking://resources/repo"
+ record = VectorRecordSnapshot(
+ "a-l2",
+ root + "/a.py",
+ "a.py",
+ 2,
+ {"abstract": "old", "search_tags": ["scope=old"]},
+ )
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(
+ root,
+ "semantic_and_vectors",
+ vectorize=False,
+ scalar_intents=(ScalarIntent("search_tags", "replace", ("scope=new",)),),
+ ),
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ ContentState.MODIFIED,
+ IndexState.STALE,
+ old_kind="file",
+ new_kind="file",
+ md5="new",
+ )
+ }
+ ),
+ new_kinds={"": "directory", "a.py": "file"},
+ artifact_paths={"a.py": "repository/a.py"},
+ records={record.record_id: record},
+ is_code_repo=True,
+ account_id="acc",
+ )
+
+ assert plan.semantic_plan is not None
+ assert plan.direct_index_actions == ()
+
+
+def test_vectors_only_upsert_preserves_existing_custom_scalars():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ IndexState,
+ build_context_update_plan,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot
+
+ root = "viking://resources/repo"
+ record = VectorRecordSnapshot(
+ "external-id",
+ root + "/a.py",
+ "a.py",
+ 2,
+ {
+ "abstract": "old",
+ "md5": "old",
+ "business_priority": 7,
+ "vector": [1.0],
+ },
+ )
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(root, "vectors_only"),
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ ContentState.MODIFIED,
+ IndexState.STALE,
+ old_kind="file",
+ new_kind="file",
+ md5="new",
+ )
+ }
+ ),
+ new_kinds={"": "directory", "a.py": "file"},
+ artifact_paths={"a.py": "repository/a.py"},
+ records={record.record_id: record},
+ is_code_repo=False,
+ account_id="acc",
+ )
+
+ action = plan.direct_index_actions[0]
+ assert action.action.value == "upsert"
+ assert action.record_id == "external-id"
+ assert action.upsert_fields["business_priority"] == 7
+ assert "vector" not in action.upsert_fields
+
+
+def test_vectors_only_restore_with_complete_index_does_not_reembed():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ IndexState,
+ build_context_update_plan,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot
+
+ root = "viking://resources/repo"
+ record = VectorRecordSnapshot(
+ "existing-id",
+ root + "/a.py",
+ "a.py",
+ 2,
+ {"md5": "same", "abstract": "old", "search_tags": ["scope=old"]},
+ )
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(root, "vectors_only"),
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ ContentState.RESTORE,
+ IndexState.COMPLETE,
+ new_kind="file",
+ md5="same",
+ )
+ }
+ ),
+ new_kinds={"": "directory", "a.py": "file"},
+ artifact_paths={"a.py": "repository/a.py"},
+ records={record.record_id: record},
+ is_code_repo=False,
+ account_id="acc",
+ )
+
+ assert [
+ (action.operation.value, action.relative_path) for action in plan.content_tree_actions
+ ] == [("upsert", "a.py")]
+ assert plan.semantic_plan is None
+ assert plan.direct_index_actions == ()
+
+
+def test_vectors_only_unchanged_file_with_extra_level_only_deletes_conflict():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ IndexState,
+ build_context_update_plan,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot
+
+ root = "viking://resources/repo"
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(root, "vectors_only"),
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ ContentState.UNCHANGED,
+ IndexState.LEVEL_CONFLICT,
+ old_kind="file",
+ new_kind="file",
+ md5="same",
+ )
+ }
+ ),
+ new_kinds={"": "directory", "a.py": "file"},
+ artifact_paths={"a.py": "repository/a.py"},
+ records={
+ "valid-l2": VectorRecordSnapshot(
+ "valid-l2", root + "/a.py", "a.py", 2, {"md5": "same"}
+ ),
+ "invalid-l0": VectorRecordSnapshot("invalid-l0", root + "/a.py", "a.py", 0, {}),
+ },
+ is_code_repo=False,
+ account_id="acc",
+ )
+
+ assert [(action.action.value, action.record_id) for action in plan.direct_index_actions] == [
+ ("delete", "invalid-l0")
+ ]
+
+
+def test_builder_carries_request_ingest_options_into_semantic_plan():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ IndexState,
+ build_context_update_plan,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent
+ from openviking.utils.ingest_options import IngestOptions
+
+ root = "viking://resources/repo"
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(root, "semantic_and_vectors"),
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ ContentState.ADDED,
+ IndexState.MISSING,
+ new_kind="file",
+ md5="new",
+ )
+ }
+ ),
+ new_kinds={"": "directory", "a.py": "file"},
+ artifact_paths={"a.py": "repository/a.py"},
+ records={},
+ is_code_repo=False,
+ account_id="acc",
+ ingest_options=IngestOptions.from_search_tags(["team=search"], mode="append"),
+ )
+
+ assert plan.semantic_plan.ingest_options.search_tags == ["team=search"]
+ assert plan.semantic_plan.ingest_options.search_tag_mode == "append"
+
+
+def test_missing_index_for_existing_content_uses_partial_update_repair():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ IndexState,
+ build_context_update_plan,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent
+
+ root = "viking://resources/repo"
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(root, "semantic_and_vectors"),
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ ContentState.UNCHANGED,
+ IndexState.MISSING,
+ old_kind="file",
+ new_kind="file",
+ md5="same",
+ )
+ }
+ ),
+ new_kinds={"": "directory", "a.py": "file"},
+ artifact_paths={"a.py": "repository/a.py"},
+ records={},
+ is_code_repo=False,
+ account_id="acc",
+ )
+
+ file_entry = next(
+ entry for entry in plan.semantic_plan.tree.entries if entry.relative_path == "a.py"
+ )
+ assert file_entry.slot(2).action.value == "merge"
+
+
[email protected]("content_state", ["added", "restore"])
+def test_new_or_restored_content_does_not_use_partial_update(content_state):
+ from openviking.storage.context_update_plan import build_context_update_plan
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent
+
+ root = "viking://resources/repo"
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(root, "semantic_and_vectors"),
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py", content_state, "missing", new_kind="file", md5="new"
+ )
+ }
+ ),
+ new_kinds={"": "directory", "a.py": "file"},
+ artifact_paths={"a.py": "repository/a.py"},
+ records={},
+ is_code_repo=False,
+ account_id="acc",
+ )
+
+ file_entry = next(
+ entry for entry in plan.semantic_plan.tree.entries if entry.relative_path == "a.py"
+ )
+ assert file_entry.slot(2).action.value == "upsert"
+
+
+def test_vectors_only_missing_index_for_existing_content_uses_partial_update_repair():
+ from openviking.storage.context_update_plan import build_context_update_plan
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent
+
+ root = "viking://resources/repo"
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(root, "vectors_only"),
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ "unchanged",
+ "missing",
+ old_kind="file",
+ new_kind="file",
+ md5="same",
+ )
+ }
+ ),
+ new_kinds={"a.py": "file"},
+ artifact_paths={"a.py": "repository/a.py"},
+ records={},
+ is_code_repo=False,
+ account_id="acc",
+ )
+
+ assert len(plan.direct_index_actions) == 1
+ assert plan.direct_index_actions[0].action.value == "merge"
+
+
+def test_vectors_only_partial_repair_preserves_explicit_replace_empty_tags():
+ from openviking.storage.context_update_plan import build_context_update_plan
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent, ScalarIntent
+
+ root = "viking://resources/repo"
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(
+ root,
+ "vectors_only",
+ scalar_intents=(ScalarIntent("search_tags", "replace", ()),),
+ ),
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ "unchanged",
+ "missing",
+ old_kind="file",
+ new_kind="file",
+ md5="same",
+ )
+ }
+ ),
+ new_kinds={"a.py": "file"},
+ artifact_paths={"a.py": "repository/a.py"},
+ records={},
+ is_code_repo=False,
+ account_id="acc",
+ )
+
+ action = plan.direct_index_actions[0]
+ assert action.action.value == "merge"
+ assert action.field_patch == FieldPatch(
+ {"search_tags": []},
+ {"search_tags": "replace"},
+ )
+
+
+def test_builder_backfills_missing_md5_and_merges_scalar_update():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ IndexState,
+ build_context_update_plan,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import (
+ RequestIntent,
+ ScalarIntent,
+ VectorRecordSnapshot,
+ )
+
+ root = "viking://resources/repo"
+ record = VectorRecordSnapshot(
+ "a-l2",
+ root + "/a.py",
+ "a.py",
+ 2,
+ {
+ "md5": "",
+ "abstract": "old",
+ "search_tags": ["scope=old"],
+ },
+ )
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(
+ root,
+ "semantic_and_vectors",
+ scalar_intents=(ScalarIntent("search_tags", "replace", ("scope=new",)),),
+ ),
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ ContentState.UNCHANGED,
+ IndexState.STALE,
+ old_kind="file",
+ new_kind="file",
+ md5="resolved-md5",
+ )
+ }
+ ),
+ new_kinds={"": "directory", "a.py": "file"},
+ artifact_paths={"a.py": "repository/a.py"},
+ records={"a-l2": record},
+ is_code_repo=True,
+ account_id="acc",
+ )
+
+ assert plan.semantic_plan is None
+ assert len(plan.direct_index_actions) == 1
+ action = plan.direct_index_actions[0]
+ assert action.record_id == "a-l2"
+ assert action.action.value == "update_fields"
+ assert action.field_patch.values == {
+ "md5": "resolved-md5",
+ "search_tags": ["scope=new"],
+ }
+
+
+def test_builder_preserves_existing_record_identity_and_custom_scalars_on_upsert():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ IndexState,
+ build_context_update_plan,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot
+
+ root = "viking://resources/repo"
+ record = VectorRecordSnapshot(
+ "external-record-id",
+ root + "/a.py",
+ "a.py",
+ 2,
+ {
+ "abstract": "old",
+ "business_priority": 7,
+ "vector": [1.0],
+ "content": "large old body",
+ },
+ )
+ plan = build_context_update_plan(
+ root_uri=root,
+ context_type="resource",
+ request=RequestIntent(root, "semantic_and_vectors"),
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ ContentState.MODIFIED,
+ IndexState.STALE,
+ old_kind="file",
+ new_kind="file",
+ md5="new",
+ )
+ }
+ ),
+ new_kinds={"": "directory", "a.py": "file"},
+ artifact_paths={"a.py": "repository/a.py"},
+ records={record.record_id: record},
+ is_code_repo=True,
+ account_id="acc",
+ )
+
+ slot = next(
+ entry for entry in plan.semantic_plan.tree.entries if entry.relative_path == "a.py"
+ ).slot(2)
+ assert slot.record_id == "external-record-id"
+ assert slot.scalar_override()["business_priority"] == 7
+ assert "vector" not in slot.scalar_override()
+ assert "content" not in slot.scalar_override()
+
+
+async def test_snapshot_builder_returns_one_canonical_context_plan():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ build_context_update_plan_from_snapshot,
+ )
+ from openviking.storage.resource_rnfv import (
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ NewEntry,
+ RequestIntent,
+ RNFVSnapshot,
+ VectorIndexSnapshot,
+ )
+
+ root = "viking://resources/repo"
+ snapshot = RNFVSnapshot(
+ request=RequestIntent(root, "semantic_and_vectors"),
+ new=NewArtifactSnapshot({"a.py": NewEntry(md5="new")}),
+ formal=FormalTreeSnapshot({}),
+ vectors=VectorIndexSnapshot({}, frozenset({"id", "uri", "level", "md5"})),
+ )
+ diff, plan = await build_context_update_plan_from_snapshot(
+ snapshot=snapshot,
+ store=AsyncMock(),
+ artifact_ref=object(),
+ target=AsyncMock(),
+ vikingdb=AsyncMock(),
+ context_type="resource",
+ is_code_repo=True,
+ account_id="acc",
+ ctx=object(),
+ root_preexisting=False,
+ artifact_paths={"a.py": "repository/a.py"},
+ )
+
+ assert diff.entries["a.py"].content_state is ContentState.ADDED
+ assert [a.relative_path for a in plan.content_tree_actions] == ["", "a.py"]
+ assert plan.content_tree_actions[1].artifact_path == "repository/a.py"
+ assert plan.semantic_plan is not None
+ entries = {entry.relative_path: entry for entry in plan.semantic_plan.tree.entries}
+ assert entries[""].semantic_action.value == "aggregate"
+ assert entries["a.py"].semantic_action.value == "generate"
+
+
+async def test_snapshot_builder_hydrates_scalars_for_vectors_only_upsert():
+ from openviking.storage.context_update_plan import build_context_update_plan_from_snapshot
+ from openviking.storage.resource_rnfv import (
+ FormalEntry,
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ NewEntry,
+ RequestIntent,
+ RNFVSnapshot,
+ VectorIndexSnapshot,
+ VectorRecordSnapshot,
+ )
+
+ root = "viking://resources/repo"
+ record = VectorRecordSnapshot(
+ "existing-id",
+ root + "/a.py",
+ "a.py",
+ 2,
+ {"md5": "old", "abstract": "old summary"},
+ )
+ snapshot = RNFVSnapshot(
+ request=RequestIntent(root, "vectors_only"),
+ new=NewArtifactSnapshot({"a.py": NewEntry(md5="new")}),
+ formal=FormalTreeSnapshot({"a.py": FormalEntry()}),
+ vectors=VectorIndexSnapshot(
+ {record.record_id: record},
+ frozenset({"id", "uri", "level", "md5", "abstract"}),
+ ),
+ )
+ vikingdb = AsyncMock()
+ vikingdb.hydrate_incremental_records.return_value = {
+ record.record_id: {
+ "id": record.record_id,
+ "uri": record.uri,
+ "level": 2,
+ "md5": "old",
+ "abstract": "old summary",
+ "business_priority": 7,
+ }
+ }
+
+ _, plan = await build_context_update_plan_from_snapshot(
+ snapshot=snapshot,
+ store=AsyncMock(),
+ artifact_ref=object(),
+ target=AsyncMock(),
+ vikingdb=vikingdb,
+ context_type="resource",
+ is_code_repo=False,
+ account_id="acc",
+ ctx=object(),
+ root_preexisting=True,
+ artifact_paths={"a.py": "repository/a.py"},
+ )
+
+ vikingdb.hydrate_incremental_records.assert_awaited_once()
+ assert plan.direct_index_actions[0].record_id == record.record_id
+ assert plan.direct_index_actions[0].upsert_fields["business_priority"] == 7
+
+
+async def test_file_root_plan_uses_file_refresh_without_directory_semantic_tree():
+ from openviking.storage.context_update_plan import build_context_update_plan_from_snapshot
+ from openviking.storage.resource_rnfv import (
+ FormalEntry,
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ NewEntry,
+ RequestIntent,
+ RNFVSnapshot,
+ VectorIndexSnapshot,
+ VectorRecordSnapshot,
+ )
+
+ root = "viking://resources/report.md"
+ record = VectorRecordSnapshot("report-l2", root, "", 2, {"md5": "old", "abstract": "old"})
+ snapshot = RNFVSnapshot(
+ RequestIntent(root, "semantic_and_vectors"),
+ NewArtifactSnapshot({"": NewEntry(md5="new")}),
+ FormalTreeSnapshot({"": FormalEntry()}),
+ VectorIndexSnapshot({"report-l2": record}, frozenset({"id", "uri", "level", "md5"})),
+ )
+ vikingdb = AsyncMock()
+ vikingdb.hydrate_incremental_records.return_value = {"report-l2": {"abstract": "old"}}
+
+ _, plan = await build_context_update_plan_from_snapshot(
+ snapshot=snapshot,
+ store=AsyncMock(read_bytes=AsyncMock(return_value=b"new")),
+ artifact_ref=object(),
+ target=AsyncMock(read_file=AsyncMock(return_value=b"old")),
+ vikingdb=vikingdb,
+ context_type="resource",
+ is_code_repo=False,
+ account_id="acc",
+ ctx=object(),
+ root_preexisting=True,
+ artifact_paths={"": "report.md"},
+ root_is_file=True,
+ )
+
+ assert [
+ (action.operation.value, action.relative_path) for action in plan.content_tree_actions
+ ] == [("upsert", "")]
+ assert plan.semantic_plan is None
+ assert plan.file_refresh is not None
+ assert plan.file_refresh.md5 == "new"
+
+
+async def test_hydration_promotes_a_dependency_when_its_record_disappears():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ IndexState,
+ hydrate_context_plan_records,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import VectorRecordSnapshot
+
+ root = "viking://resources/repo"
+ inventory = {
+ "changed": VectorRecordSnapshot("changed", f"{root}/a.py", "a.py", 2, {"md5": "old"}),
+ "sibling": VectorRecordSnapshot("sibling", f"{root}/b.py", "b.py", 2, {"md5": "same"}),
+ }
+ vikingdb = AsyncMock()
+ vikingdb.hydrate_incremental_records.return_value = {
+ "changed": {
+ "id": "changed",
+ "uri": f"{root}/a.py",
+ "level": 2,
+ "abstract": "old a",
+ }
+ }
+ records, (active, _, retained) = await hydrate_context_plan_records(
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ ContentState.MODIFIED,
+ IndexState.STALE,
+ old_kind="file",
+ new_kind="file",
+ md5="new",
+ ),
+ "b.py": ResourceDiffEntry(
+ "b.py",
+ ContentState.UNCHANGED,
+ IndexState.COMPLETE,
+ old_kind="file",
+ new_kind="file",
+ md5="same",
+ ),
+ }
+ ),
+ new_kinds={"": "directory", "a.py": "file", "b.py": "file"},
+ inventory=inventory,
+ vikingdb=vikingdb,
+ ctx=object(),
+ )
+
+ assert records["sibling"].record_id == "sibling"
+ assert "b.py" in active
+ assert "b.py" in retained
+
+
+async def test_hydration_reads_summaries_before_full_scalars_for_active_records():
+ from openviking.storage.context_update_plan import (
+ ContentState,
+ IndexState,
+ hydrate_context_plan_records,
+ )
+ from openviking.storage.resource_diff import ResourceDiffEntry, ResourceDiffResult
+ from openviking.storage.resource_rnfv import RequestIntent, VectorRecordSnapshot
+
+ root = "viking://resources/repo"
+ inventory = {
+ "changed": VectorRecordSnapshot("changed", f"{root}/a.py", "a.py", 2, {"md5": "old"}),
+ "sibling": VectorRecordSnapshot("sibling", f"{root}/b.py", "b.py", 2, {"md5": "same"}),
+ }
+ vikingdb = AsyncMock()
+ vikingdb.hydrate_incremental_records.side_effect = [
+ {
+ "changed": {"abstract": "old a"},
+ "sibling": {"abstract": "old b"},
+ },
+ {"changed": {"business_priority": 7}},
+ ]
+ records, (active, _, retained) = await hydrate_context_plan_records(
+ diff=ResourceDiffResult(
+ {
+ "a.py": ResourceDiffEntry(
+ "a.py",
+ ContentState.MODIFIED,
+ IndexState.STALE,
+ old_kind="file",
+ new_kind="file",
+ md5="new",
+ ),
+ "b.py": ResourceDiffEntry(
+ "b.py",
+ ContentState.UNCHANGED,
+ IndexState.COMPLETE,
+ old_kind="file",
+ new_kind="file",
+ md5="same",
+ ),
+ }
+ ),
+ new_kinds={"": "directory", "a.py": "file", "b.py": "file"},
+ inventory=inventory,
+ vikingdb=vikingdb,
+ ctx=object(),
+ request=RequestIntent(root, "semantic_and_vectors"),
+ )
+
+ assert active == {"", "a.py"}
+ assert retained == {"", "a.py", "b.py"}
+ assert records["sibling"].fields == {"md5": "same", "abstract": "old b"}
+ assert records["changed"].fields == {
+ "md5": "old",
+ "abstract": "old a",
+ "business_priority": 7,
+ }
+ assert vikingdb.hydrate_incremental_records.await_args_list[0].kwargs["output_fields"] == {
+ "abstract"
+ }
+ assert "output_fields" not in vikingdb.hydrate_incremental_records.await_args_list[1].kwargs
+ assert set(vikingdb.hydrate_incremental_records.await_args_list[1].args[0]) == {"changed"}
+
+
+async def test_content_executor_uploads_files_with_bounded_concurrency():
+ from unittest.mock import patch
+
+ import openviking.concurrency as concurrency_utils
+ from openviking.storage.context_update_plan import (
+ ContentTreeAction,
+ execute_content_tree_actions,
+ )
+
+ active = peak = 0
+ gate = asyncio.Event()
+
+ class Store:
+ async def read_bytes(self, ref, path):
+ return path.encode()
+
+ class Target:
+ async def write_file(self, path, data):
+ nonlocal active, peak
+ active += 1
+ peak = max(peak, active)
+ if active == 3:
+ gate.set()
+ try:
+ await asyncio.wait_for(gate.wait(), 1)
+ finally:
+ active -= 1
+
+ async def delete_path(self, path, *, is_dir):
+ return None
+
+ async def mkdir(self, path):
+ return None
+
+ actions = tuple(
+ ContentTreeAction(
+ "upsert",
+ f"{index}.py",
+ new_kind="file",
+ artifact_path=f"{index}.py",
+ md5=str(index),
+ )
+ for index in range(8)
+ )
+ create_task = asyncio.create_task
+ with patch.object(concurrency_utils.asyncio, "create_task", wraps=create_task) as created:
+ await execute_content_tree_actions(
+ actions,
+ store=Store(),
+ artifact_ref=object(),
+ target=Target(),
+ concurrency=3,
+ )
+
+ assert peak == 3
+ assert created.call_count == 3
+
+
+async def test_bounded_map_preserves_order_and_processes_all_items():
+ from unittest.mock import patch
+
+ import openviking.concurrency as concurrency_utils
+
+ started = 0
+ release = asyncio.Event()
+
+ async def work(value: int) -> int:
+ nonlocal started
+ started += 1
+ if started == 3:
+ release.set()
+ await release.wait()
+ return value * 2
+
+ create_task = asyncio.create_task
+ with patch.object(concurrency_utils.asyncio, "create_task", wraps=create_task) as created:
+ result = await concurrency_utils.bounded_map(range(100), work, concurrency=3)
+
+ assert started == 100
+ assert created.call_count == 3
+ assert result == [index * 2 for index in range(100)]
+
+
+async def test_bounded_map_stops_claiming_after_worker_failure():
+ from openviking.concurrency import bounded_map
+
+ started: list[int] = []
+ release = asyncio.Event()
+
+ async def work(value: int) -> int:
+ started.append(value)
+ if value == 0:
+ raise RuntimeError("boom")
+ await release.wait()
+ return value
+
+ task = asyncio.create_task(bounded_map(range(100), work, concurrency=3))
+ await asyncio.sleep(0)
+ release.set()
+ with pytest.raises(RuntimeError, match="boom"):
+ await task
+
+ assert set(started) <= {0, 1, 2}
+
+
+async def test_content_executor_logs_failed_action_with_correlation(monkeypatch):
+ import openviking.storage.context_update_plan as plan_module
+ from openviking.service.task_work_index import bind_task_context
+ from openviking.storage.context_update_plan import (
+ ContentTreeAction,
+ execute_content_tree_actions,
+ )
+ from openviking.telemetry import OperationTelemetry, bind_telemetry
+
+ class Target:
+ async def delete_path(self, path, *, is_dir):
+ raise OSError("delete failed")
+
+ action = ContentTreeAction("delete", "old.md", old_kind="file")
+ telemetry = OperationTelemetry("add_resource_job", enabled=True)
+ log_exception = MagicMock()
+ monkeypatch.setattr(plan_module.logger, "exception", log_exception)
+
+ with (
+ bind_task_context("task-1", "acct", "user"),
+ bind_telemetry(telemetry),
+ pytest.raises(OSError, match="delete failed"),
+ ):
+ await execute_content_tree_actions(
+ (action,),
+ store=object(),
+ artifact_ref=object(),
+ target=Target(),
+ )
+
+ message, correlation, operation, relative_path, old_kind, new_kind = (
+ log_exception.call_args.args
+ )
+ assert "[ContentTreeActionFailed]" in message
+ assert correlation == f"task_id=task-1 telemetry_id={telemetry.telemetry_id}"
+ assert (operation, relative_path, old_kind, new_kind) == (
+ "delete",
+ "old.md",
+ "file",
+ "-",
+ )
+ assert log_exception.call_args.kwargs == {}
+
+
+async def test_resource_processor_dispatches_direct_index_actions_without_semantic(monkeypatch):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage.context_update_plan import DirectIndexAction
+ from openviking.utils.resource_processor import ResourceProcessor
+ from openviking_cli.session.user_id import UserIdentifier
+
+ enqueued = []
+
+ async def enqueue(queue, message, **kwargs):
+ del queue, kwargs
+ enqueued.append(message)
+ return True
+
+ queue_manager = SimpleNamespace(
+ EMBEDDING="embedding",
+ get_queue=lambda *args, **kwargs: object(),
+ )
+ monkeypatch.setattr("openviking.storage.queuefs.get_queue_manager", lambda: queue_manager)
+ monkeypatch.setattr("openviking.utils.embedding_utils._enqueue_embedding_message", enqueue)
+ processor = ResourceProcessor(SimpleNamespace(get_embedder=lambda: None))
+ processor._vectorize_resource_file = AsyncMock()
+ ctx = RequestContext(UserIdentifier("acc", "user"), Role.USER)
+ await processor._enqueue_index_actions(
+ (
+ DirectIndexAction("delete", "viking://resources/repo/a.py", 2, "id-a"),
+ DirectIndexAction(
+ "update_fields",
+ "viking://resources/repo/b.py",
+ 2,
+ "id-b",
+ field_patch=FieldPatch(
+ {"search_tags": ["scope=new"]},
+ {"search_tags": "replace"},
+ ),
+ ),
+ DirectIndexAction(
+ "merge",
+ "viking://resources/repo/c.py",
+ 2,
+ "id-c",
+ field_patch=FieldPatch(
+ {"search_tags": ["scope=new"]},
+ {"search_tags": "append"},
+ ),
+ md5="new-md5",
+ ),
+ ),
+ ctx=ctx,
+ )
+ assert [msg.action.value for msg in enqueued] == ["delete", "update_fields"]
+ assert enqueued[0].record_ids == ["id-a"]
+ assert enqueued[1].update_fields["search_tags"] == ["scope=new"]
+ processor._vectorize_resource_file.assert_awaited_once_with(
+ "viking://resources/repo/c.py",
+ ctx=ctx,
+ file_md5="new-md5",
+ scalar_override={"_record_id": "id-c"},
+ action="merge",
+ field_patch=FieldPatch(
+ {"search_tags": ["scope=new"]},
+ {"search_tags": "append"},
+ ),
+ )
+
+
+def test_semantic_message_roundtrip_uses_explicit_plan():
+ from openviking.storage.context_update_plan import (
+ IndexSlot,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+ )
+ from openviking.storage.queuefs.semantic_msg import SemanticMsg
+
+ plan = SemanticPlan(
+ "viking://resources/repo",
+ "resource",
+ tree=SemanticTreeSnapshot(
+ (
+ SemanticTreeEntry("", "directory", "unchanged", "aggregate"),
+ SemanticTreeEntry(
+ "a.py",
+ "file",
+ "modified",
+ "generate",
+ md5="new",
+ index_slots=(
+ IndexSlot(
+ 2,
+ "actual-id",
+ {"abstract": "old"},
+ action="upsert",
+ ),
+ ),
+ ),
+ )
+ ),
+ )
+ msg = SemanticMsg(uri=plan.root_uri, context_type="resource", plan=plan)
+ restored = SemanticMsg.from_json(msg.to_json())
+ assert restored.plan == plan
+ assert (
+ next(entry for entry in restored.plan.tree.entries if entry.relative_path == "a.py")
+ .index_slots[0]
+ .record_id
+ == "actual-id"
+ )
+
+
+async def test_v3_tree_reuses_explicit_node_without_listing_its_subtree(monkeypatch):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage.context_update_plan import (
+ IndexSlot,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+ )
+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor
+ from openviking_cli.session.user_id import UserIdentifier
+
+ root = "viking://resources/repo"
+ calls = []
+
+ class FS:
+ _async_agfs = None
+
+ async def ls(self, uri, **kwargs):
+ calls.append(uri)
+ raise AssertionError("v3 plan must not list the live tree")
+
+ def _uri_to_path(self, uri, ctx=None):
+ return uri
+
+ fs = FS()
+ fs._async_agfs = fs
+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)
+ processor = AsyncMock()
+ plan = SemanticPlan(
+ root,
+ "resource",
+ tree=SemanticTreeSnapshot(
+ (
+ SemanticTreeEntry(
+ "",
+ "directory",
+ "unchanged",
+ "reuse",
+ index_slots=(IndexSlot(0, "root-l0", {"abstract": "old root"}),),
+ ),
+ )
+ ),
+ )
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=1,
+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),
+ semantic_plan=plan,
+ )
+ await executor.run(root)
+ assert calls == []
+ assert executor.root_write_result.wrote is False
+
+
+async def test_directory_index_slots_choose_exact_levels(monkeypatch):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage.abstract_overview import AbstractOverviewWriteResult
+ from openviking.storage.context_update_plan import (
+ IndexSlot,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+ )
+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor
+ from openviking_cli.session.user_id import UserIdentifier
+
+ root = "viking://resources/repo"
+ fs = SimpleNamespace(
+ _async_agfs=None,
+ _uri_to_path=lambda uri, ctx=None: uri,
+ )
+ fs._async_agfs = fs
+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+
+ class Processor:
+ _generate_overview = AsyncMock(return_value="overview")
+ _vectorize_directory = AsyncMock(return_value={1})
+
+ @staticmethod
+ def _normalize_overview_generation(overview):
+ return overview, "abstract"
+
+ processor = Processor()
+ plan = SemanticPlan(
+ root,
+ "resource",
+ SemanticTreeSnapshot(
+ (
+ SemanticTreeEntry(
+ "",
+ "directory",
+ "unchanged",
+ "aggregate",
+ index_slots=(
+ IndexSlot(0, "l0", {"abstract": "old"}),
+ IndexSlot(1, "l1", None, action="upsert"),
+ ),
+ ),
+ )
+ ),
+ )
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=1,
+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),
+ semantic_plan=plan,
+ )
+ executor._write_directory_semantics = AsyncMock(
+ return_value=AbstractOverviewWriteResult(wrote=True, overview_body_changed=True)
+ )
+
+ await executor.run(root)
+
+ kwargs = processor._vectorize_directory.await_args.kwargs
+ assert kwargs["include_abstract"] is False
+ assert kwargs["include_overview"] is True
+
+
+async def test_directory_sidecar_write_failure_fails_semantic_plan(monkeypatch):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage.context_update_plan import (
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+ )
+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor
+ from openviking_cli.session.user_id import UserIdentifier
+
+ root = "viking://resources/repo"
+ fs = SimpleNamespace(_async_agfs=None, _uri_to_path=lambda uri, ctx=None: uri)
+ fs._async_agfs = fs
+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+
+ class Processor:
+ _generate_overview = AsyncMock(return_value="overview")
+ _vectorize_directory = AsyncMock(return_value={0, 1})
+
+ @staticmethod
+ def _normalize_overview_generation(overview):
+ return overview, "abstract"
+
+ processor = Processor()
+ plan = SemanticPlan(
+ root,
+ "resource",
+ SemanticTreeSnapshot((SemanticTreeEntry("", "directory", "unchanged", "aggregate"),)),
+ )
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=1,
+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),
+ semantic_plan=plan,
+ )
+ executor._write_directory_semantics = AsyncMock(side_effect=OSError("sidecar unavailable"))
+
+ with pytest.raises(OSError, match="sidecar unavailable"):
+ await executor.run(root)
+
+ processor._vectorize_directory.assert_not_awaited()
+
+
+async def test_directory_repair_passes_partial_update_per_missing_level(monkeypatch):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage.abstract_overview import AbstractOverviewWriteResult
+ from openviking.storage.context_update_plan import (
+ IndexAction,
+ IndexSlot,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+ )
+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor
+ from openviking_cli.session.user_id import UserIdentifier
+
+ root = "viking://resources/repo"
+ fs = SimpleNamespace(_async_agfs=None, _uri_to_path=lambda uri, ctx=None: uri)
+ fs._async_agfs = fs
+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+
+ class Processor:
+ _generate_overview = AsyncMock(return_value="overview")
+ _vectorize_directory = AsyncMock(return_value={0, 1})
+
+ @staticmethod
+ def _normalize_overview_generation(overview):
+ return overview, "abstract"
+
+ processor = Processor()
+ plan = SemanticPlan(
+ root,
+ "resource",
+ SemanticTreeSnapshot(
+ (
+ SemanticTreeEntry(
+ "",
+ "directory",
+ "unchanged",
+ "aggregate",
+ index_slots=(
+ IndexSlot(
+ 0,
+ "root-l0",
+ None,
+ action="merge",
+ ),
+ IndexSlot(
+ 1,
+ "root-l1",
+ None,
+ action="upsert",
+ ),
+ ),
+ ),
+ )
+ ),
+ )
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=1,
+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),
+ semantic_plan=plan,
+ )
+ executor._write_directory_semantics = AsyncMock(
+ return_value=AbstractOverviewWriteResult(
+ wrote=True, overview_body_changed=True, abstract_body_changed=True
+ )
+ )
+
+ await executor.run(root)
+
+ assert processor._vectorize_directory.await_args.kwargs["actions"] == {
+ 0: IndexAction.MERGE,
+ 1: IndexAction.UPSERT,
+ }
+
+
+async def test_directory_output_unchanged_still_applies_planned_scalar_fields(monkeypatch):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage.abstract_overview import AbstractOverviewWriteResult
+ from openviking.storage.context_update_plan import (
+ IndexSlot,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+ )
+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor
+ from openviking_cli.session.user_id import UserIdentifier
+
+ root = "viking://resources/repo"
+ fs = SimpleNamespace(_async_agfs=None, _uri_to_path=lambda uri, ctx=None: uri)
+ fs._async_agfs = fs
+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+
+ class Processor:
+ _generate_overview = AsyncMock(return_value="overview")
+ _vectorize_directory = AsyncMock(return_value=set())
+ _update_vector_fields = AsyncMock(return_value=True)
+
+ @staticmethod
+ def _normalize_overview_generation(overview):
+ return overview, "abstract"
+
+ processor = Processor()
+ plan = SemanticPlan(
+ root,
+ "resource",
+ SemanticTreeSnapshot(
+ (
+ SemanticTreeEntry(
+ "",
+ "directory",
+ "unchanged",
+ "aggregate",
+ index_slots=(
+ IndexSlot(
+ 0,
+ "root-l0",
+ {"abstract": "abstract"},
+ action="upsert",
+ upsert_fields={"search_tags": ["scope=new"]},
+ field_patch=FieldPatch(
+ {"search_tags": ["scope=new"]},
+ {"search_tags": "append"},
+ ),
+ fallback_to_patch=True,
+ ),
+ IndexSlot(
+ 1,
+ "root-l1",
+ None,
+ action="upsert",
+ ),
+ ),
+ ),
+ )
+ ),
+ )
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=1,
+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),
+ semantic_plan=plan,
+ )
+ executor._write_directory_semantics = AsyncMock(
+ return_value=AbstractOverviewWriteResult(wrote=True)
+ )
+ executor._check_dir_children_changed = AsyncMock(return_value=False)
+ executor._read_existing_overview_abstract = AsyncMock(return_value=("overview", "abstract"))
+
+ await executor.run(root)
+
+ processor._vectorize_directory.assert_not_awaited()
+ processor._update_vector_fields.assert_awaited_once_with(
+ record_id="root-l0",
+ uri=root,
+ level=0,
+ field_patch=FieldPatch(
+ {"search_tags": ["scope=new"]},
+ {"search_tags": "append"},
+ {
+ "abstract": "abstract",
+ "uri": root,
+ "level": 0,
+ "account_id": "acc",
+ },
+ ),
+ ctx=executor._ctx,
+ )
+
+
+async def test_code_summary_unchanged_still_reembeds_after_content_change(monkeypatch):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage.abstract_overview import AbstractOverviewWriteResult
+ from openviking.storage.context_update_plan import (
+ IndexSlot,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+ )
+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor
+ from openviking_cli.session.user_id import UserIdentifier
+
+ root = "viking://resources/repo"
+ fs = SimpleNamespace(
+ _async_agfs=None,
+ _uri_to_path=lambda uri, ctx=None: uri,
+ read_file_bytes=AsyncMock(return_value=b"changed body"),
+ )
+ fs._async_agfs = fs
+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+
+ class Processor:
+ _generate_single_file_summary = AsyncMock(return_value={"name": "a.py", "summary": "same"})
+ _update_file_vector_fields = AsyncMock(return_value=True)
+ _vectorize_single_file = AsyncMock(return_value=True)
+ _generate_overview = AsyncMock(return_value="overview")
+ _vectorize_directory = AsyncMock(return_value=set())
+
+ @staticmethod
+ def _normalize_overview_generation(overview):
+ return overview, "abstract"
+
+ processor = Processor()
+ plan = SemanticPlan(
+ root,
+ "resource",
+ SemanticTreeSnapshot(
+ (
+ SemanticTreeEntry(
+ "",
+ "directory",
+ "unchanged",
+ "aggregate",
+ index_slots=(IndexSlot(0, "root-l0", {"abstract": "old"}),),
+ ),
+ SemanticTreeEntry(
+ "a.py",
+ "file",
+ "modified",
+ "generate",
+ md5="new-md5",
+ index_slots=(
+ IndexSlot(
+ 2,
+ "a-l2",
+ {"abstract": "same"},
+ action="upsert",
+ ),
+ ),
+ ),
+ )
+ ),
+ file_vector_source="summary_when_available",
+ )
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=1,
+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),
+ semantic_plan=plan,
+ )
+ executor._write_directory_semantics = AsyncMock(
+ return_value=AbstractOverviewWriteResult(wrote=True)
+ )
+
+ await executor.run(root)
+
+ processor._vectorize_single_file.assert_awaited_once()
+ processor._update_file_vector_fields.assert_not_awaited()
+
+
+async def test_file_repair_passes_partial_update_to_embedding(monkeypatch):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage.abstract_overview import AbstractOverviewWriteResult
+ from openviking.storage.context_update_plan import (
+ IndexSlot,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+ )
+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor
+ from openviking_cli.session.user_id import UserIdentifier
+
+ root = "viking://resources/repo"
+ fs = SimpleNamespace(
+ _async_agfs=None,
+ _uri_to_path=lambda uri, ctx=None: uri,
+ read_file_bytes=AsyncMock(return_value=b"body"),
+ )
+ fs._async_agfs = fs
+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+
+ class Processor:
+ _generate_single_file_summary = AsyncMock(
+ return_value={"name": "a.py", "summary": "summary"}
+ )
+ _vectorize_single_file = AsyncMock(return_value=True)
+ _generate_overview = AsyncMock(return_value="overview")
+ _vectorize_directory = AsyncMock(return_value=set())
+
+ @staticmethod
+ def _normalize_overview_generation(overview):
+ return overview, "abstract"
+
+ processor = Processor()
+ plan = SemanticPlan(
+ root,
+ "resource",
+ SemanticTreeSnapshot(
+ (
+ SemanticTreeEntry("", "directory", "unchanged", "aggregate"),
+ SemanticTreeEntry(
+ "a.py",
+ "file",
+ "unchanged",
+ "generate",
+ md5="same",
+ index_slots=(
+ IndexSlot(
+ 2,
+ "a-l2",
+ None,
+ action="merge",
+ ),
+ ),
+ ),
+ )
+ ),
+ )
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=1,
+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),
+ semantic_plan=plan,
+ )
+ executor._write_directory_semantics = AsyncMock(
+ return_value=AbstractOverviewWriteResult(wrote=True)
+ )
+
+ await executor.run(root)
+
+ assert processor._vectorize_single_file.await_args.kwargs["action"] == "merge"
+
+
+async def test_direct_only_plan_skips_semantic_queue(monkeypatch):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage.context_update_plan import DirectIndexAction
+ from openviking.utils.resource_processor import ResourceProcessor
+ from openviking_cli.session.user_id import UserIdentifier
+
+ root = "viking://resources/repo"
+ lock = {"lease_ref": "plan"}
+ viking_fs = SimpleNamespace(
+ _async_agfs=SimpleNamespace(pathlock_release=AsyncMock()),
+ )
+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)
+ processor = ResourceProcessor(SimpleNamespace(get_embedder=lambda: None))
+ processor._enqueue_index_actions = AsyncMock()
+ summarizer = SimpleNamespace(summarize=AsyncMock())
+ processor._get_summarizer = lambda: summarizer
+ action = DirectIndexAction(
+ "update_fields",
+ root + "/a.py",
+ 2,
+ "a-l2",
+ field_patch=FieldPatch({"search_tags": ["scope=new"]}),
+ )
+
+ await processor.finish_prepared_resource(
+ {
+ "root_uri": root,
+ "temp_uri": root,
+ "source_committed": True,
+ "target_preexisting": True,
+ "root_is_file": False,
+ "context_update_plan": {
+ "root_uri": root,
+ "context_type": "resource",
+ "content_tree_actions": [],
+ "semantic_plan": None,
+ "direct_index_actions": [
+ {
+ "action": action.action.value,
+ "uri": action.uri,
+ "level": action.level,
+ "record_id": action.record_id,
+ "field_patch": action.field_patch.to_dict(),
+ "md5": action.md5,
+ }
+ ],
+ },
+ },
+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),
+ resource_lock=lock,
+ build_index=True,
+ processing_mode="semantic_and_vectors",
+ )
+
+ processor._enqueue_index_actions.assert_awaited_once()
+ summarizer.summarize.assert_not_awaited()
+ viking_fs._async_agfs.pathlock_release.assert_awaited_once_with(lock)
+
+
+async def test_vectors_only_context_plan_does_not_run_legacy_vectorization(monkeypatch):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage.context_update_plan import ContextUpdatePlan, DirectIndexAction
+ from openviking.utils.resource_processor import ResourceProcessor
+ from openviking_cli.session.user_id import UserIdentifier
+
+ root = "viking://resources/repo"
+ lock = {"lease_ref": "plan"}
+ viking_fs = SimpleNamespace(
+ _async_agfs=SimpleNamespace(pathlock_release=AsyncMock()),
+ )
+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)
+ processor = ResourceProcessor(SimpleNamespace(get_embedder=lambda: None))
+ processor._enqueue_index_actions = AsyncMock()
+ processor._vectorize_prepared_files = AsyncMock(
+ side_effect=AssertionError("legacy vectors_only path must not run")
+ )
+ action = DirectIndexAction("upsert", root + "/a.py", 2, "a-l2", md5="m")
+ plan = ContextUpdatePlan(root, "resource", direct_index_actions=(action,))
+
+ await processor.finish_prepared_resource(
+ {
+ "root_uri": root,
+ "temp_uri": root,
+ "source_committed": True,
+ "target_preexisting": True,
+ "root_is_file": False,
+ "context_update_plan": plan.to_dict(),
+ },
+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),
+ resource_lock=lock,
+ build_index=True,
+ processing_mode="vectors_only",
+ )
+
+ processor._enqueue_index_actions.assert_awaited_once_with(
+ plan.direct_index_actions, ctx=processor._enqueue_index_actions.await_args.kwargs["ctx"]
+ )
+ processor._vectorize_prepared_files.assert_not_awaited()
+
+
+async def test_direct_index_enqueue_failure_releases_lock():
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage.context_update_plan import ContextUpdatePlan, DirectIndexAction
+ from openviking.utils.resource_processor import ResourceProcessor
+ from openviking_cli.session.user_id import UserIdentifier
+
+ root = "viking://resources/repo"
+ lock = {"lease_ref": "plan"}
+ viking_fs = SimpleNamespace(
+ _async_agfs=SimpleNamespace(pathlock_release=AsyncMock()),
+ )
+ processor = ResourceProcessor(SimpleNamespace(get_embedder=lambda: None))
+ processor._enqueue_index_actions = AsyncMock(side_effect=RuntimeError("queue unavailable"))
+ plan = ContextUpdatePlan(
+ root,
+ "resource",
+ direct_index_actions=(DirectIndexAction("delete", root + "/a.py", 2, "a-l2"),),
+ )
+
+ with pytest.raises(RuntimeError, match="queue unavailable"):
+ with pytest.MonkeyPatch.context() as monkeypatch:
+ monkeypatch.setattr(
+ "openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs
+ )
+ await processor.finish_prepared_resource(
+ {
+ "root_uri": root,
+ "temp_uri": root,
+ "source_committed": True,
+ "target_preexisting": True,
+ "root_is_file": False,
+ "context_update_plan": plan.to_dict(),
+ },
+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),
+ resource_lock=lock,
+ build_index=True,
+ )
+
+ viking_fs._async_agfs.pathlock_release.assert_awaited_once_with(lock)
+
+
+async def test_semantic_processor_runs_only_plan_execution_roots(monkeypatch):
+ from openviking.storage.context_update_plan import (
+ IndexSlot,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+ )
+ from openviking.storage.queuefs.semantic_msg import SemanticMsg
+ from openviking.storage.queuefs.semantic_processor import SemanticProcessor
+
+ calls = []
+
+ class Executor:
+ stale = True
+ root_write_result = None
+
+ def __init__(self, **kwargs):
+ calls.append(("init", kwargs["semantic_plan"]))
+
+ async def run(self, uri):
+ calls.append(("run", uri))
+
+ def get_stats(self):
+ return SimpleNamespace()
+
+ root = "viking://resources/repo"
+ plan = SemanticPlan(
+ root,
+ "resource",
+ SemanticTreeSnapshot(
+ (
+ SemanticTreeEntry(
+ "",
+ "directory",
+ "unchanged",
+ "aggregate",
+ index_slots=(IndexSlot(0, "root-l0", {"abstract": "root"}),),
+ ),
+ SemanticTreeEntry(
+ "src",
+ "directory",
+ "unchanged",
+ "aggregate",
+ index_slots=(IndexSlot(0, "src-l0", {"abstract": "old"}),),
+ ),
+ SemanticTreeEntry(
+ "src/a.py",
+ "file",
+ "modified",
+ "generate",
+ md5="new",
+ index_slots=(
+ IndexSlot(
+ 2,
+ "a-l2",
+ {"abstract": "old a"},
+ action="upsert",
+ ),
+ ),
+ ),
+ SemanticTreeEntry(
+ "docs",
+ "directory",
+ "unchanged",
+ "aggregate",
+ index_slots=(IndexSlot(0, "docs-l0", {"abstract": "old"}),),
+ ),
+ SemanticTreeEntry(
+ "docs/b.md",
+ "file",
+ "modified",
+ "generate",
+ md5="new",
+ index_slots=(
+ IndexSlot(
+ 2,
+ "b-l2",
+ {"abstract": "old b"},
+ action="upsert",
+ ),
+ ),
+ ),
+ )
+ ),
+ )
+ fs = SimpleNamespace(exists=AsyncMock(return_value=True))
+ monkeypatch.setattr("openviking.storage.queuefs.semantic_processor.get_viking_fs", lambda: fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor", Executor
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_processor.SemanticLockScope.resolve",
+ AsyncMock(return_value=SimpleNamespace(lock=None, close=AsyncMock())),
+ )
+ processor = SemanticProcessor()
+ processor._cleanup_local_artifact = AsyncMock()
+ msg = SemanticMsg(
+ uri=root,
+ context_type="resource",
+ account_id="acc",
+ user_id="user",
+ role="user",
+ plan=plan,
+ )
+
+ await processor.on_dequeue(msg.to_dict())
+
+ assert [value for kind, value in calls if kind == "run"] == [root]
+
+
[email protected]("cleanup_fails", [False, True])
+async def test_process_resource_does_not_handoff_local_artifact(
+ monkeypatch, tmp_path, cleanup_fails
+):
+ from openviking.parse.output import LocalParseOutputStore
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage.context_update_plan import ContextUpdatePlan
+ from openviking.utils.resource_processor import ResourceProcessor
+ from openviking_cli.session.user_id import UserIdentifier
+
+ store = LocalParseOutputStore(str(tmp_path / "artifacts"))
+ ref = await store.create_artifact()
+ await store.write_bytes(ref, "repository/a.py", b"a")
+ if cleanup_fails:
+ store.cleanup = AsyncMock(side_effect=OSError("cleanup unavailable"))
+ processor = ResourceProcessor(SimpleNamespace(get_embedder=lambda: None))
+ processor._build_parse_output_store = lambda: store
+ processor._get_media_processor = lambda: SimpleNamespace(
+ process=AsyncMock(
+ return_value=SimpleNamespace(
+ temp_dir_path=ref.root,
+ source_path="x",
+ source_format="repository",
+ meta={},
+ warnings=[],
+ artifact_ref=ref,
+ ensure_artifact_ref=lambda: ref,
+ )
+ )
+ )
+ processor.tree_builder.finalize_from_temp = AsyncMock(
+ return_value=SimpleNamespace(
+ root=SimpleNamespace(
+ uri="viking://resources/repo",
+ temp_uri=ref.root + "/repository",
+ ),
+ _root_is_file=False,
+ )
+ )
+ processor._commit_directory_artifact_with_plan = AsyncMock(
+ return_value=ContextUpdatePlan("viking://resources/repo", "resource")
+ )
+ viking_fs = SimpleNamespace(
+ exists=AsyncMock(return_value=False),
+ _uri_to_path=lambda uri, ctx=None: uri,
+ bind_request_context=lambda ctx: nullcontext(),
+ _async_agfs=SimpleNamespace(pathlock_release=AsyncMock()),
+ )
+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)
+ processor.acquire_resource_lock = AsyncMock(return_value={"lease_ref": "x"})
+
+ result = await processor.process_resource(
+ path="x",
+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),
+ to="viking://resources/repo",
+ defer_post_processing=True,
+ build_index=True,
+ )
+
+ assert result["_post_process"]["artifact_ref"] is None
+ assert "file_md5s" not in result["_post_process"]
+ assert "file_abstracts" not in result["_post_process"]
+ assert "artifact_files" not in result["_post_process"]
+ artifact_path = tmp_path.joinpath("artifacts", ref.root.split("/")[-1])
+ assert artifact_path.exists() is cleanup_fails
diff --git a/tests/storage/test_embedding_msg.py b/tests/storage/test_embedding_msg.py
index da2c9176f..a9f45b1c3 100644
--- a/tests/storage/test_embedding_msg.py
+++ b/tests/storage/test_embedding_msg.py
@@ -3,7 +3,16 @@
from uuid import uuid4
-from openviking.storage.queuefs.embedding_msg import EmbeddingMsg
+import pytest
+
+from openviking.storage.index_action import FieldPatch, IndexAction
+from openviking.storage.queuefs.embedding_msg import (
+ DeletePayload,
+ EmbeddingMsg,
+ EmbedPayload,
+ NoopPayload,
+ UpdateFieldsPayload,
+)
from openviking.telemetry.request_wait_tracker import RequestWaitTracker
@@ -27,3 +36,273 @@ def test_embedding_msg_roundtrip_preserves_id_for_request_wait_tracker():
assert tracker.is_complete(telemetry_id)
finally:
tracker.cleanup(telemetry_id)
+
+
+def test_embedding_msg_roundtrip_preserves_queue_enqueue_time():
+ msg = EmbeddingMsg(
+ "hello",
+ {"uri": "viking://resources/demo"},
+ queue_enqueued_at=123.456,
+ )
+
+ assert EmbeddingMsg.from_dict(msg.to_dict()).queue_enqueued_at == 123.456
+
+
+def test_legacy_embedding_msg_defaults_to_embed_and_upsert():
+ restored = EmbeddingMsg.from_dict(
+ {
+ "message": "hello",
+ "context_data": {"uri": "viking://resources/demo"},
+ }
+ )
+
+ assert restored.action is IndexAction.UPSERT
+
+
+def test_embedding_msg_accepts_noop_index_action_roundtrip():
+ msg = EmbeddingMsg(None, {"uri": "viking://resources/demo"}, action=IndexAction.NONE)
+
+ restored = EmbeddingMsg.from_json(msg.to_json())
+
+ assert restored.action is IndexAction.NONE
+ assert restored.message is None
+
+
+def test_embedding_update_fields_roundtrip():
+ msg = EmbeddingMsg.for_update_fields(
+ record_id="record-1",
+ fields={
+ "md5": "new-md5",
+ "updated_at": "2026-09-13T00:00:00Z",
+ "search_tags": ["scope=new"],
+ },
+ context_data={
+ "uri": "viking://resources/demo/a.py",
+ "account_id": "account-a",
+ "level": 2,
+ },
+ field_modes={"search_tags": "append"},
+ initial_fields={
+ "uri": "viking://resources/demo/a.py",
+ "account_id": "account-a",
+ "level": 2,
+ "vector": [0.1, 0.2],
+ },
+ )
+
+ restored = EmbeddingMsg.from_json(msg.to_json())
+
+ assert restored.action is IndexAction.UPDATE_FIELDS
+ assert restored.record_ids == ["record-1"]
+ assert restored.update_fields == {
+ "md5": "new-md5",
+ "updated_at": "2026-09-13T00:00:00Z",
+ "search_tags": ["scope=new"],
+ }
+ assert restored.message is None
+ assert restored.field_modes == {"search_tags": "append"}
+ assert restored.initial_fields["vector"] == [0.1, 0.2]
+
+
+def test_embedding_delete_roundtrip():
+ msg = EmbeddingMsg.for_delete(
+ record_ids=["record-1", "record-2"],
+ context_data={
+ "uri": "viking://resources/demo",
+ "account_id": "account-a",
+ },
+ )
+
+ restored = EmbeddingMsg.from_json(msg.to_json())
+
+ assert restored.action is IndexAction.DELETE
+ assert restored.record_ids == ["record-1", "record-2"]
+ assert restored.message is None
+
+
+def test_embedding_update_fields_rejects_vector_fields():
+ with pytest.raises(ValueError, match="vector"):
+ EmbeddingMsg.for_update_fields(
+ record_id="record-1",
+ fields={"vector": [0.1, 0.2]},
+ context_data={
+ "uri": "viking://resources/demo/a.py",
+ "account_id": "account-a",
+ "level": 2,
+ },
+ )
+
+
+def test_field_patch_roundtrip_and_resolution():
+ patch = FieldPatch(
+ values={"search_tags": ["scope=new"], "md5": "new-md5"},
+ modes={"search_tags": "append"},
+ seed_fields={
+ "uri": "viking://resources/demo/a.py",
+ "account_id": "account-a",
+ "level": 2,
+ "vector": [0.1, 0.2],
+ },
+ )
+
+ restored = FieldPatch.from_dict(patch.to_dict())
+
+ assert restored.resolve({"search_tags": ["env=old"], "md5": "old-md5"}) == {
+ "search_tags": ["env=old", "scope=new"],
+ "md5": "new-md5",
+ }
+ assert restored.seed_fields["vector"] == [0.1, 0.2]
+
+
+ ("message", "payload_type"),
+ [
+ (
+ EmbeddingMsg.for_embed(
+ message="body",
+ context_data={"uri": "viking://resources/demo/a.py"},
+ ),
+ EmbedPayload,
+ ),
+ (
+ EmbeddingMsg.for_update_fields(
+ record_id="record-1",
+ field_patch=FieldPatch(values={"md5": "new-md5"}),
+ context_data={"uri": "viking://resources/demo/a.py"},
+ ),
+ UpdateFieldsPayload,
+ ),
+ (
+ EmbeddingMsg.for_delete(
+ record_ids=["record-1"],
+ context_data={"uri": "viking://resources/demo/a.py"},
+ ),
+ DeletePayload,
+ ),
+ (EmbeddingMsg.noop(context_data={"uri": "viking://resources/demo"}), NoopPayload),
+ ],
+)
+def test_embedding_msg_roundtrip_uses_action_specific_payload(message, payload_type):
+ serialized = message.to_dict()
+
+ assert "payload" in serialized
+ assert "message" not in serialized
+ restored = EmbeddingMsg.from_dict(serialized)
+ assert isinstance(restored.payload, payload_type)
+ assert restored.to_dict() == serialized
+
+
+def test_embed_payload_rejects_patch_for_upsert():
+ with pytest.raises(ValueError, match="upsert.*field patch"):
+ EmbeddingMsg.for_embed(
+ message="body",
+ context_data={"uri": "viking://resources/demo/a.py"},
+ action=IndexAction.UPSERT,
+ field_patch=FieldPatch(values={"search_tags": ["scope=new"]}),
+ )
+
+
+def test_action_specific_payload_rejects_mismatched_action():
+ with pytest.raises(ValueError, match="delete requires DeletePayload"):
+ EmbeddingMsg(
+ action=IndexAction.DELETE,
+ payload=EmbedPayload(
+ "body",
+ {"uri": "viking://resources/demo/a.py"},
+ ),
+ )
+
+
+def test_payload_rejects_legacy_fields_in_same_message():
+ with pytest.raises(ValueError, match="payload cannot be combined"):
+ EmbeddingMsg.from_dict(
+ {
+ "action": "upsert",
+ "payload": {
+ "type": "embed",
+ "message": "body",
+ "context_data": {"uri": "viking://resources/demo/a.py"},
+ },
+ "record_ids": ["silently-ignored-before"],
+ }
+ )
+
+
+ "payload",
+ [
+ {
+ "type": "delete",
+ "record_ids": ["record-1"],
+ "context_data": {},
+ "message": "must-not-be-ignored",
+ },
+ {
+ "type": "embed",
+ "message": "body",
+ "context_data": {},
+ "record_ids": ["must-not-be-ignored"],
+ },
+ ],
+)
+def test_action_specific_payload_rejects_cross_action_fields(payload):
+ with pytest.raises(ValueError, match="payload contains unsupported fields"):
+ EmbeddingMsg.from_dict(
+ {
+ "action": "delete" if payload["type"] == "delete" else "merge",
+ "payload": payload,
+ }
+ )
+
+
+def test_update_fields_factory_rejects_duplicate_patch_representations():
+ with pytest.raises(ValueError, match="field_patch cannot be combined"):
+ EmbeddingMsg.for_update_fields(
+ record_id="record-1",
+ context_data={"uri": "viking://resources/demo/a.py"},
+ field_patch=FieldPatch({"md5": "new-md5"}),
+ fields={"md5": "ignored-md5"},
+ )
+
+
+def test_legacy_upsert_discards_historically_ignored_fields():
+ restored = EmbeddingMsg.from_dict(
+ {
+ "action": "upsert",
+ "message": "body",
+ "context_data": {"uri": "viking://resources/demo/a.py"},
+ "record_ids": ["ignored-record"],
+ "update_fields": {"md5": "ignored-md5"},
+ "initial_fields": {"vector": [0.1, 0.2]},
+ }
+ )
+
+ assert restored.action is IndexAction.UPSERT
+ assert restored.record_ids == []
+ assert restored.field_patch is None
+ assert set(restored.to_dict()) == {
+ "id",
+ "telemetry_id",
+ "queue_enqueued_at",
+ "action",
+ "payload",
+ }
+
+
+def test_legacy_update_fields_message_normalizes_to_field_patch_payload():
+ restored = EmbeddingMsg.from_dict(
+ {
+ "message": None,
+ "context_data": {"uri": "viking://resources/demo/a.py"},
+ "action": "update_fields",
+ "record_ids": ["record-1"],
+ "update_fields": {"search_tags": ["scope=new"]},
+ "field_modes": {"search_tags": "append"},
+ "initial_fields": {"vector": [0.1, 0.2]},
+ }
+ )
+
+ assert isinstance(restored.payload, UpdateFieldsPayload)
+ assert restored.payload.field_patch.values == {"search_tags": ["scope=new"]}
+ assert restored.payload.field_patch.modes == {"search_tags": "append"}
+ assert restored.payload.field_patch.seed_fields == {"vector": [0.1, 0.2]}
diff --git a/tests/storage/test_embedding_msg_converter_tenant.py b/tests/storage/test_embedding_msg_converter_tenant.py
index a20595406..7a0a8a313 100644
--- a/tests/storage/test_embedding_msg_converter_tenant.py
+++ b/tests/storage/test_embedding_msg_converter_tenant.py
@@ -6,6 +6,7 @@
import pytest
from openviking.core.context import Context, Vectorize
+from openviking.storage.index_action import IndexAction
from openviking.storage.queuefs.embedding_msg_converter import EmbeddingMsgConverter
from openviking_cli.session.user_id import UserIdentifier
@@ -45,6 +46,7 @@ def test_embedding_msg_converter_backfills_account_and_owner_fields(
expected_owner_user_id(user) if callable(expected_owner_user_id) else expected_owner_user_id
)
assert msg.context_data["owner_user_id"] == expected_user
+ assert msg.action is IndexAction.MERGE
def test_embedding_msg_converter_keeps_only_embedding_input():
@@ -56,3 +58,4 @@ def test_embedding_msg_converter_keeps_only_embedding_input():
assert msg is not None
assert msg.message == "bounded embedding text"
assert "content" not in msg.context_data
+ assert msg.action is IndexAction.MERGE
diff --git a/tests/storage/test_ingest_ls_node_limit.py b/tests/storage/test_ingest_ls_node_limit.py
index c688b4ad7..f5ca0d36c 100644
--- a/tests/storage/test_ingest_ls_node_limit.py
+++ b/tests/storage/test_ingest_ls_node_limit.py
@@ -64,17 +64,22 @@ class _TruncatingVikingFS:
pass
async def sync_tree(
- self, root_uri, target_uri, *, ctx=None, file_change_status=None,
- lease_ref=None, delete_temp_after=False, is_changed=None,
+ self,
+ root_uri,
+ target_uri,
+ *,
+ ctx=None,
+ file_change_status=None,
+ lease_ref=None,
+ delete_temp_after=False,
+ is_changed=None,
):
from openviking.storage.viking_fs import SyncDiff
from openviking_cli.utils.uri import VikingURI
diff = SyncDiff()
# Mirror _SyncMixin.list_children: must pass node_limit=LS_ALL_NODES.
- entries = await self.ls(
- root_uri, show_all_hidden=True, node_limit=LS_ALL_NODES, ctx=ctx
- )
+ entries = await self.ls(root_uri, show_all_hidden=True, node_limit=LS_ALL_NODES, ctx=ctx)
for entry in entries:
name = entry.get("name", "")
if not name or name in (".", "..") or name.startswith("."):
@@ -113,9 +118,9 @@ async def test_sync_materializes_all_children_above_default_node_limit(monkeypat
"sync was truncated at ls node_limit"
)
assert len(diff.added_dirs) == n_children
- assert all(
- lim == LS_ALL_NODES for lim in fake.ls_node_limits
- ), f"sync_tree must pass LS_ALL_NODES to ls, got node_limits={fake.ls_node_limits}"
+ assert all(lim == LS_ALL_NODES for lim in fake.ls_node_limits), (
+ f"sync_tree must pass LS_ALL_NODES to ls, got node_limits={fake.ls_node_limits}"
+ )
class _TruncatingLsFS:
@@ -132,22 +137,22 @@ class _TruncatingLsFS:
@pytest.mark.asyncio
-async def test_semantic_dag_list_dir_enumerates_all_children(monkeypatch):
- """The summary DAG dispatches one summary/recursion task per child it lists.
+async def test_semantic_executor_list_dir_enumerates_all_children(monkeypatch):
+ """The semantic tree dispatches one summary/recursion task per child it lists.
If ``_list_dir`` is capped at node_limit, children beyond 1000 never get an
``.abstract.md``/``.overview.md`` (and are never recursed into), so a large
namespace silently loses its L0/L1 layers."""
- from openviking.storage.queuefs.semantic_dag import SemanticDagExecutor
+ from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor
dir_uri = "viking://resources/wixqa"
n_children = 1500 # > the default node_limit of 1000
fake = _TruncatingLsFS(dir_uri, n_children)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_dag.get_viking_fs",
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs",
lambda: fake,
)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=None,
context_type="resource",
max_concurrent_llm=1,
@@ -156,6 +161,6 @@ async def test_semantic_dag_list_dir_enumerates_all_children(monkeypatch):
children_dirs, file_paths = await executor._list_dir(dir_uri, from_hint="test")
assert len(children_dirs) == n_children, (
- f"only {len(children_dirs)}/{n_children} children listed — summary DAG "
+ f"only {len(children_dirs)}/{n_children} children listed — semantic tree "
"was truncated at ls node_limit"
)
diff --git a/tests/storage/test_queue_manager.py b/tests/storage/test_queue_manager.py
index a5cf69a08..b19fa6530 100644
--- a/tests/storage/test_queue_manager.py
+++ b/tests/storage/test_queue_manager.py
@@ -54,7 +54,7 @@ async def test_skill_shutdown_releases_lock_after_embedding_worker_exits(
transport, monkeypatch, concurrency
):
from openviking.service.task_tracker_concurrency import run_to_completion
- from openviking.storage.queuefs.semantic_dag import DagStats
+ from openviking.storage.queuefs.semantic_executor import SemanticTreeStats
from openviking.storage.queuefs.semantic_msg import SemanticMsg
from openviking.storage.queuefs.semantic_processor import SemanticProcessor
from openviking.telemetry import OperationTelemetry
@@ -87,7 +87,7 @@ async def test_skill_shutdown_releases_lock_after_embedding_worker_exits(
queued.set()
def get_stats(self):
- return DagStats()
+ return SemanticTreeStats()
async def write(data):
async def finish():
@@ -127,7 +127,7 @@ async def test_skill_shutdown_releases_lock_after_embedding_worker_exits(
await self.process_dequeued(data)
await self.ack(data)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_processor.SemanticDagExecutor", Dag)
+ monkeypatch.setattr("openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor", Dag)
monkeypatch.setattr(
SemanticProcessor, "_resolve_skill_semantic_lock", AsyncMock(return_value=Lease())
)
diff --git a/tests/storage/test_resource_diff_snapshot.py b/tests/storage/test_resource_diff_snapshot.py
new file mode 100644
index 000000000..7020700ac
--- /dev/null
+++ b/tests/storage/test_resource_diff_snapshot.py
@@ -0,0 +1,404 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Tests for reading N/F/V snapshots and preparing artifact inventory."""
+
+import asyncio
+import json
+
+import pytest
+
+from openviking.parse.output import AgfsParseOutputStore, ParseArtifactRef
+from openviking.storage.resource_diff import (
+ build_rnfv_snapshot,
+ prepare_artifact_inventory,
+ read_target_file_snapshot,
+)
+from openviking.storage.resource_rnfv import FormalEntry, NewEntry
+
+
+class _Ctx:
+ account_id = "acct"
+
+
+class _FakeVikingFS:
+ """Minimal VikingFS supporting tree() plus the output-store surface."""
+
+ def __init__(self, tree_entries, files=None):
+ self._tree_entries = tree_entries
+ self.files = files or {}
+ self.tree_calls = []
+
+ async def tree(
+ self,
+ uri,
+ *,
+ output="original",
+ show_all_hidden=False,
+ node_limit=1000,
+ level_limit=3,
+ ctx=None,
+ ):
+ self.tree_calls.append({"uri": uri, "node_limit": node_limit, "level_limit": level_limit})
+ return list(self._tree_entries)
+
+ # Output-store surface for ParseOutputStore-backed artifact snapshots.
+ def create_temp_uri(self, ctx=None):
+ return "viking://temp/n"
+
+ async def ls(self, uri, ctx=None, **kwargs):
+ prefix = f"{uri.rstrip('/')}/"
+ seen = {}
+ for stored in self.files:
+ if stored.startswith(prefix):
+ rest = stored[len(prefix) :]
+ head = rest.split("/", 1)
+ name = head[0]
+ seen[name] = {"name": name, "uri": f"{prefix}{name}", "isDir": len(head) > 1}
+ return list(seen.values())
+
+ async def read(self, uri, **kwargs):
+ return self.files[uri]
+
+ async def write_file(self, uri, content, **kwargs):
+ self.files[uri] = content.encode("utf-8") if isinstance(content, str) else content
+
+
+class _FakeVikingDB:
+ def __init__(self, records):
+ self._records = records
+ self.requested = None
+ self.inventory_output_fields = None
+
+ async def get_l2_diff_records_by_uris(self, uris, *, ctx):
+ self.requested = list(uris)
+ return {u: self._records[u] for u in uris if u in self._records}
+
+ async def get_l2_diff_records_under_uri(self, target_uri, *, ctx):
+ self.requested = target_uri
+ prefix = target_uri.rstrip("/") + "/"
+ return {
+ uri: value
+ for uri, value in self._records.items()
+ if uri == target_uri or uri.startswith(prefix)
+ }
+
+ async def get_incremental_inventory_under_uri(self, target_uri, *, ctx, output_fields=None):
+ del ctx
+ self.inventory_output_fields = list(output_fields or [])
+ prefix = target_uri.rstrip("/") + "/"
+ return {
+ str(value.get("id") or f"id-{index}"): {
+ key: item
+ for key, item in {
+ **value,
+ "id": str(value.get("id") or f"id-{index}"),
+ "uri": uri,
+ "level": int(value.get("level", 2)),
+ "md5": str(value.get("md5") or ""),
+ }.items()
+ if not output_fields or key in output_fields
+ }
+ for index, (uri, value) in enumerate(self._records.items())
+ if uri == target_uri or uri.startswith(prefix)
+ }
+
+
+class TestReadFormalTreeSnapshot:
+ async def test_lists_business_files_with_unlimited_scan(self) -> None:
+ vfs = _FakeVikingFS(
+ [
+ {"rel_path": "a.py", "isDir": False, "uri": "viking://resources/x/a.py"},
+ {"rel_path": "sub", "isDir": True, "uri": "viking://resources/x/sub"},
+ {"rel_path": "sub/b.py", "isDir": False, "uri": "viking://resources/x/sub/b.py"},
+ ]
+ )
+ files, complete = await read_target_file_snapshot(vfs, "viking://resources/x", ctx=_Ctx())
+ assert complete is True
+ assert set(files) == {"a.py", "sub", "sub/b.py"}
+ assert files["sub"].is_dir is True
+ assert files["a.py"].is_dir is False
+ # Must scan the whole tree, not the default bounded window.
+ assert vfs.tree_calls[0]["node_limit"] is None
+ assert vfs.tree_calls[0]["level_limit"] is None
+
+ async def test_denied_entry_marks_snapshot_incomplete(self) -> None:
+ vfs = _FakeVikingFS(
+ [
+ {"rel_path": "a.py", "isDir": False, "uri": "viking://resources/x/a.py"},
+ {
+ "rel_path": "secret",
+ "isDir": True,
+ "access": "denied",
+ "uri": "viking://resources/x/secret",
+ },
+ ]
+ )
+ files, complete = await read_target_file_snapshot(vfs, "viking://resources/x", ctx=_Ctx())
+ # A permission-hidden subtree means we cannot trust the tree for deletion.
+ assert complete is False
+ assert "secret" not in files
+
+ async def test_control_sidecars_excluded(self) -> None:
+ vfs = _FakeVikingFS(
+ [
+ {"rel_path": "a.py", "isDir": False, "uri": "viking://resources/x/a.py"},
+ {
+ "rel_path": ".abstract.md",
+ "isDir": False,
+ "uri": "viking://resources/x/.abstract.md",
+ },
+ {"rel_path": "_system", "isDir": True, "uri": "viking://resources/x/_system"},
+ ]
+ )
+ files, complete = await read_target_file_snapshot(vfs, "viking://resources/x", ctx=_Ctx())
+ assert set(files) == {"a.py"}
+ assert complete is True
+
+
+async def test_prepare_artifact_inventory_rewrites_images_during_single_artifact_walk(tmp_path):
+ from unittest.mock import AsyncMock
+
+ from openviking.parse.output import LocalParseOutputStore
+ from openviking.utils.content_hash import content_md5
+
+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_bytes(ref, "repository/docs/guide.md", b"![diagram](diagram.png)\n")
+ await store.write_bytes(ref, "repository/docs/diagram.png", b"png")
+ await store.write_text(
+ ref,
+ "repository/.image_mappings.json",
+ json.dumps({"docs/guide.md": {"diagram.png": "diagram.png"}}),
+ )
+ store.list = AsyncMock(wraps=store.list)
+
+ inventory = await prepare_artifact_inventory(
+ store,
+ ref,
+ doc_rel="repository",
+ target_root_uri="viking://resources/repo",
+ )
+
+ assert set(inventory.entries) == {"docs", "docs/guide.md", "docs/diagram.png"}
+ assert inventory.entries["docs/guide.md"].md5 == content_md5(
+ b"![diagram](viking://resources/repo/docs/diagram.png)\n"
+ )
+ assert inventory.artifact_paths["docs/guide.md"] == "repository/docs/guide.md"
+ assert inventory.rewritten_paths == frozenset({"docs/guide.md"})
+ assert [call.args[1] for call in store.list.await_args_list] == [
+ "repository",
+ "repository/docs",
+ ]
+
+
+async def test_prepare_artifact_inventory_rewrites_images_in_agfs_artifact():
+ from openviking.utils.content_hash import content_md5
+
+ vfs = _FakeVikingFS(
+ [],
+ files={
+ "viking://temp/n/repository/docs/guide.md": b"![diagram](diagram.png)\n",
+ "viking://temp/n/repository/docs/diagram.png": b"png",
+ "viking://temp/n/repository/.image_mappings.json": json.dumps(
+ {"docs/guide.md": {"diagram.png": "diagram.png"}}
+ ).encode(),
+ },
+ )
+ store = AgfsParseOutputStore(viking_fs=vfs)
+ ref = ParseArtifactRef(backend="agfs", root="viking://temp/n", root_type="dir")
+
+ inventory = await prepare_artifact_inventory(
+ store,
+ ref,
+ doc_rel="repository",
+ target_root_uri="viking://resources/repo",
+ )
+
+ expected = b"![diagram](viking://resources/repo/docs/diagram.png)\n"
+ assert vfs.files["viking://temp/n/repository/docs/guide.md"] == expected
+ assert inventory.entries["docs/guide.md"].md5 == content_md5(expected)
+
+
+async def test_build_rnfv_snapshot_reuses_prepared_artifact_inventory(tmp_path):
+ from unittest.mock import AsyncMock
+
+ from openviking.parse.output import LocalParseOutputStore
+
+ root = "viking://resources/x"
+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_bytes(ref, "repository/a.py", b"a")
+ store.list = AsyncMock(wraps=store.list)
+ inventory = await prepare_artifact_inventory(store, ref, doc_rel="repository")
+ store.list.reset_mock()
+
+ snapshot = await build_rnfv_snapshot(
+ viking_fs=_FakeVikingFS([]),
+ vikingdb=_FakeVikingDB({}),
+ store=store,
+ artifact_ref=ref,
+ target_uri=root,
+ ctx=_Ctx(),
+ doc_rel="repository",
+ artifact_inventory=inventory,
+ target_preexisting=False,
+ )
+
+ assert set(snapshot.new.entries) == {"a.py"}
+ store.list.assert_not_awaited()
+
+
+async def test_build_rnfv_snapshot_skips_vector_inventory_when_vectorize_disabled(tmp_path):
+ from unittest.mock import AsyncMock
+
+ from openviking.parse.output import LocalParseOutputStore
+ from openviking.storage.resource_rnfv import RequestIntent
+ from openviking.utils.ingest_options import IngestOptions
+
+ root = "viking://resources/x"
+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_bytes(ref, "repository/a.py", b"new")
+ inventory = await prepare_artifact_inventory(store, ref, doc_rel="repository")
+ vikingdb = _FakeVikingDB({f"{root}/a.py": {"id": "stale-l2", "level": 2, "md5": "new"}})
+ vikingdb.get_incremental_inventory_under_uri = AsyncMock(
+ side_effect=AssertionError("build_index=false must not read vector inventory")
+ )
+
+ snapshot = await build_rnfv_snapshot(
+ viking_fs=_FakeVikingFS([{"rel_path": "a.py", "isDir": False, "uri": f"{root}/a.py"}]),
+ vikingdb=vikingdb,
+ store=store,
+ artifact_ref=ref,
+ target_uri=root,
+ ctx=_Ctx(),
+ doc_rel="repository",
+ request_intent=RequestIntent.from_ingest_options(
+ target_uri=root,
+ processing_mode="vectors_only",
+ ingest_options=IngestOptions.from_search_tags(["team=search"]),
+ vectorize=False,
+ ),
+ artifact_inventory=inventory,
+ target_preexisting=True,
+ )
+
+ assert snapshot.vectors.records_by_id == {}
+ assert snapshot.vectors.projected_fields == frozenset()
+ vikingdb.get_incremental_inventory_under_uri.assert_not_awaited()
+
+
+async def test_build_rnfv_snapshot_starts_new_formal_and_vector_reads_concurrently(
+ tmp_path, monkeypatch
+):
+ from openviking.parse.output import LocalParseOutputStore
+ from openviking.storage.resource_diff import ArtifactInventory
+
+ root = "viking://resources/x"
+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))
+ ref = await store.create_artifact(root_type="dir")
+ started = set()
+ release = asyncio.Event()
+
+ async def read_new(*args, **kwargs):
+ started.add("new")
+ await release.wait()
+ return ArtifactInventory(entries={"a.py": NewEntry(md5="new")}, artifact_paths={})
+
+ async def read_formal(*args, **kwargs):
+ started.add("formal")
+ await release.wait()
+ return {"a.py": FormalEntry()}, True
+
+ async def read_vectors(*args, **kwargs):
+ started.add("vectors")
+ await release.wait()
+ return {"record": {"id": "record", "uri": f"{root}/a.py", "level": 2, "md5": "old"}}
+
+ monkeypatch.setattr("openviking.storage.resource_diff.prepare_artifact_inventory", read_new)
+ monkeypatch.setattr("openviking.storage.resource_diff.read_target_file_snapshot", read_formal)
+ monkeypatch.setattr(
+ "openviking.storage.resource_diff._read_incremental_vector_inventory", read_vectors
+ )
+
+ task = asyncio.create_task(
+ build_rnfv_snapshot(
+ viking_fs=_FakeVikingFS([]),
+ vikingdb=_FakeVikingDB({}),
+ store=store,
+ artifact_ref=ref,
+ target_uri=root,
+ ctx=_Ctx(),
+ )
+ )
+ for _ in range(10):
+ if started == {"new", "formal", "vectors"}:
+ break
+ await asyncio.sleep(0)
+ assert started == {"new", "formal", "vectors"}
+ release.set()
+ snapshot = await task
+
+ assert snapshot.new.entries["a.py"].md5 == "new"
+ assert snapshot.formal.entries["a.py"].is_dir is False
+ assert snapshot.vectors.records_by_id["record"].fields["md5"] == "old"
+
+
+async def test_build_rnfv_snapshot_cancels_sibling_reads_after_failure(tmp_path, monkeypatch):
+ from openviking.parse.output import LocalParseOutputStore
+
+ root = "viking://resources/x"
+ store = LocalParseOutputStore(local_root=str(tmp_path / "out"))
+ ref = await store.create_artifact(root_type="dir")
+ started = set()
+ cancelled = set()
+ fail = asyncio.Event()
+
+ async def read_new(*args, **kwargs):
+ started.add("new")
+ await fail.wait()
+ raise RuntimeError("inventory failed")
+
+ async def wait_for_cancel(name):
+ started.add(name)
+ try:
+ await asyncio.Event().wait()
+ finally:
+ cancelled.add(name)
+
+ monkeypatch.setattr("openviking.storage.resource_diff.prepare_artifact_inventory", read_new)
+ monkeypatch.setattr(
+ "openviking.storage.resource_diff.read_target_file_snapshot",
+ lambda *args, **kwargs: wait_for_cancel("formal"),
+ )
+ monkeypatch.setattr(
+ "openviking.storage.resource_diff._read_incremental_vector_inventory",
+ lambda *args, **kwargs: wait_for_cancel("vectors"),
+ )
+
+ task = asyncio.create_task(
+ build_rnfv_snapshot(
+ viking_fs=_FakeVikingFS([]),
+ vikingdb=_FakeVikingDB({}),
+ store=store,
+ artifact_ref=ref,
+ target_uri=root,
+ ctx=_Ctx(),
+ )
+ )
+ for _ in range(10):
+ if started == {"new", "formal", "vectors"}:
+ break
+ await asyncio.sleep(0)
+ fail.set()
+
+ with pytest.raises(RuntimeError, match="inventory failed"):
+ await task
+ assert cancelled == {"formal", "vectors"}
diff --git a/tests/storage/test_resource_target.py b/tests/storage/test_resource_target.py
new file mode 100644
index 000000000..bc856e969
--- /dev/null
+++ b/tests/storage/test_resource_target.py
@@ -0,0 +1,109 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Tests for the AGFS resource target used by content tree actions."""
+
+import pytest
+
+from openviking.storage.resource_target import AgfsResourceTarget
+
+
+class _Ctx:
+ account_id = "acct"
+
+
+class _FakeVikingFS:
+ def __init__(self, existing=None):
+ self.files = dict(existing or {})
+ self.written = []
+ self.removed = []
+
+ async def write_file_bytes(self, uri, content, *, ctx=None, lease_ref=None):
+ self.files[uri] = content
+ self.written.append(uri)
+
+ async def read_file_bytes(self, uri, *, ctx=None):
+ return self.files[uri]
+
+ async def rm(self, uri, *, recursive=False, ctx=None, lease_ref=None):
+ self.removed.append(uri)
+ self.files.pop(uri, None)
+
+ async def remove_files(self, uri, *, recursive=False, ctx=None, lease_ref=None):
+ self.removed.append((uri, recursive, lease_ref))
+ self.files.pop(uri, None)
+
+
+_ROOT = "viking://resources/proj"
+
+
+def _target(vfs):
+ return AgfsResourceTarget(viking_fs=vfs, root_uri=_ROOT, ctx=_Ctx())
+
+
+class TestAgfsResourceTarget:
+ async def test_write_joins_rel_path_under_root(self) -> None:
+ vfs = _FakeVikingFS()
+ target = _target(vfs)
+
+ await target.write_file("sub/a.py", b"print(1)")
+
+ assert vfs.files[f"{_ROOT}/sub/a.py"] == b"print(1)"
+
+ async def test_empty_rel_path_targets_root_file(self) -> None:
+ vfs = _FakeVikingFS()
+ target = _target(vfs)
+
+ await target.write_file("", b"body")
+
+ assert vfs.files[_ROOT] == b"body"
+
+ async def test_write_preserves_artifact_bytes(self) -> None:
+ vfs = _FakeVikingFS()
+ target = _target(vfs)
+
+ artifact_bytes = "你好".encode("gbk")
+ await target.write_file("a.py", artifact_bytes)
+
+ assert vfs.files[f"{_ROOT}/a.py"] == artifact_bytes
+
+ async def test_write_rejects_path_escape(self) -> None:
+ vfs = _FakeVikingFS()
+ target = _target(vfs)
+
+ with pytest.raises(ValueError):
+ await target.write_file("../evil.py", b"x")
+
+ async def test_delete_path_uses_exact_semantics_for_file(self) -> None:
+ lease = {"lease_ref": "root-tree"}
+ vfs = _FakeVikingFS(existing={f"{_ROOT}/gone.py": b"x"})
+ target = AgfsResourceTarget(
+ viking_fs=vfs,
+ root_uri=_ROOT,
+ ctx=_Ctx(),
+ lease_ref=lease,
+ )
+
+ await target.delete_path("gone.py", is_dir=False)
+
+ assert vfs.removed == [(f"{_ROOT}/gone.py", False, lease)]
+
+ async def test_delete_path_uses_tree_semantics_for_directory(self) -> None:
+ lease = {"lease_ref": "root-tree"}
+ vfs = _FakeVikingFS()
+ target = AgfsResourceTarget(
+ viking_fs=vfs,
+ root_uri=_ROOT,
+ ctx=_Ctx(),
+ lease_ref=lease,
+ )
+
+ await target.delete_path("old-dir", is_dir=True)
+
+ assert vfs.removed == [(f"{_ROOT}/old-dir", True, lease)]
+
+ async def test_read_file_returns_existing_bytes(self) -> None:
+ vfs = _FakeVikingFS(existing={f"{_ROOT}/a.py": b"body"})
+ target = _target(vfs)
+
+ assert await target.read_file("a.py") == b"body"
diff --git a/tests/storage/test_semantic_dag_incremental.py b/tests/storage/test_semantic_executor_incremental.py
similarity index 53%
rename from tests/storage/test_semantic_dag_incremental.py
rename to tests/storage/test_semantic_executor_incremental.py
index 308f6c741..09a9c88a7 100644
--- a/tests/storage/test_semantic_dag_incremental.py
+++ b/tests/storage/test_semantic_executor_incremental.py
@@ -16,7 +16,8 @@ from openviking.storage.abstract_overview import (
render_abstract_overview,
)
from openviking.storage.errors import LockAcquisitionError
-from openviking.storage.queuefs.semantic_dag import SemanticDagExecutor
+from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor
+from openviking.utils.content_hash import content_md5
from openviking.utils.ingest_options import IngestOptions
from openviking_cli.session.user_id import UserIdentifier
@@ -45,6 +46,10 @@ class _FakeVikingFS:
async def read_file(self, path, ctx=None):
return self._file_contents.get(self._norm(path), "")
+ async def read_file_bytes(self, path, ctx=None):
+ content = self._file_contents.get(self._norm(path), b"")
+ return content if isinstance(content, bytes) else content.encode()
+
async def abstract(self, uri, ctx=None):
return self._file_contents.get(
self._norm(f"{uri}/.abstract.md"),
@@ -75,9 +80,12 @@ class _FakeProcessor:
self.sync_calls = []
self.vectorized_files = []
self.file_ingest_options = {}
+ self.file_md5s = {}
+ self.file_contents = {}
self.directory_ingest_options = {}
self.vectorized_dirs = []
self.generated_overviews = []
+ self.updated_file_vectors = []
def _parse_overview_md(self, overview_content):
results = {}
@@ -88,8 +96,11 @@ class _FakeProcessor:
results[m.group("name").strip()] = m.group("summary").strip()
return results
- async def _generate_single_file_summary(self, file_path, llm_sem=None, ctx=None):
+ async def _generate_single_file_summary(
+ self, file_path, llm_sem=None, ctx=None, file_content=None
+ ):
self.summarized_files.append(file_path)
+ self.file_contents[("summary", file_path)] = file_content
return {"name": file_path.split("/")[-1], "summary": "summary"}
async def _generate_overview(self, dir_uri, file_summaries, children_abstracts, **kwargs):
@@ -126,10 +137,19 @@ class _FakeProcessor:
use_summary=False,
ingest_options=None,
creator_acl_grant=None,
+ file_md5=None,
+ file_content=None,
+ scalar_override=None,
+ action="merge",
):
del creator_acl_grant
self.vectorized_files.append(file_path)
self.file_ingest_options[file_path] = ingest_options
+ self.file_md5s[file_path] = file_md5
+ self.file_contents[("vector", file_path)] = file_content
+ self.file_contents[("action", file_path)] = action
+ self.file_contents[("scalar_override", file_path)] = scalar_override
+ return True
async def _vectorize_directory(
self,
@@ -140,11 +160,41 @@ class _FakeProcessor:
ctx=None,
ingest_options=None,
creator_acl_grant=None,
+ scalar_overrides=None,
+ actions=None,
+ include_abstract=True,
+ include_overview=True,
):
del creator_acl_grant
self.directory_ingest_options[uri] = ingest_options
self.vectorized_dirs.append(uri)
- return None
+ self.file_contents[("actions", uri)] = actions
+ self.file_contents[("scalar_overrides", uri)] = scalar_overrides
+ return {
+ level for level, included in ((0, include_abstract), (1, include_overview)) if included
+ }
+
+ async def _update_file_vector_fields(
+ self,
+ *,
+ record_id,
+ file_path,
+ file_md5,
+ file_content,
+ ctx,
+ scalar_fields=None,
+ ):
+ self.updated_file_vectors.append(
+ {
+ "record_id": record_id,
+ "file_path": file_path,
+ "file_md5": file_md5,
+ "file_content": file_content,
+ "ctx": ctx,
+ "scalar_fields": scalar_fields,
+ }
+ )
+ return True
async def _sync_topdown_recursive(
self, root_uri, target_uri, ctx=None, file_change_status=None, lock=None
@@ -198,15 +248,17 @@ async def test_direct_incremental_update_uses_changes_without_temp_sync(monkeypa
if sidecar_state != "valid":
filename = ".overview.md" if sidecar_state == "malformed_overview" else ".abstract.md"
fake_fs._file_contents[f"{root_uri}/{filename}"] = "---\n"
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_dag.get_openviking_config",
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
)
processor = _FakeProcessor(fake_fs)
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=2,
@@ -214,6 +266,7 @@ async def test_direct_incremental_update_uses_changes_without_temp_sync(monkeypa
incremental_update=True,
target_uri=root_uri,
changes={"modified": [f"{root_uri}/a.txt"]} if sidecar_state == "valid" else {},
+ file_md5s={f"{root_uri}/a.txt": "md5-a-new"},
)
await executor.run(root_uri)
@@ -227,6 +280,12 @@ async def test_direct_incremental_update_uses_changes_without_temp_sync(monkeypa
assert processor.vectorized_files == expected_files
assert processor.vectorized_dirs == [root_uri]
assert processor.sync_calls == []
+ # Resource ingestion uses the manifest md5 whenever the file is re-vectorized;
+ # a skipped file produces no vector write.
+ if f"{root_uri}/a.txt" in expected_files:
+ assert processor.file_md5s[f"{root_uri}/a.txt"] == "md5-a-new"
+ else:
+ assert f"{root_uri}/a.txt" not in processor.file_md5s
overview = parse_abstract_overview(fake_fs._file_contents[f"{root_uri}/.overview.md"]).body
expected_a = "old-a" if sidecar_state == "malformed_abstract" else "summary"
expected_b = "summary" if sidecar_state == "malformed_overview" else "old-b"
@@ -235,6 +294,296 @@ async def test_direct_incremental_update_uses_changes_without_temp_sync(monkeypa
assert parse_abstract_overview(fake_fs._file_contents[f"{root_uri}/.abstract.md"]).body.strip()
+async def test_modified_file_with_same_abstract_still_propagates(monkeypatch):
+ root_uri = "viking://resources/root"
+ child_uri = f"{root_uri}/nested"
+ file_uri = f"{child_uri}/a.txt"
+ old_root_overview = render_abstract_overview(
+ ContextLevel.OVERVIEW, root_uri, "FILES:\n- nested/: old-root-child"
+ )
+ old_child_overview = render_abstract_overview(
+ ContextLevel.OVERVIEW, child_uri, "FILES:\n- a.txt: summary"
+ )
+ fake_fs = _FakeVikingFS(
+ tree={
+ root_uri: [{"name": "nested", "isDir": True}],
+ child_uri: [{"name": "a.txt", "isDir": False}],
+ },
+ file_contents={
+ file_uri: "changed body",
+ f"{root_uri}/.overview.md": old_root_overview,
+ f"{root_uri}/.abstract.md": render_abstract_overview(
+ ContextLevel.ABSTRACT, root_uri, "old-root-child"
+ ),
+ f"{child_uri}/.overview.md": old_child_overview,
+ f"{child_uri}/.abstract.md": render_abstract_overview(
+ ContextLevel.ABSTRACT, child_uri, "old-child"
+ ),
+ },
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+
+ processor = _FakeProcessor(fake_fs)
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=2,
+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),
+ incremental_update=True,
+ target_uri=root_uri,
+ changes={"modified": [file_uri]},
+ file_abstracts={file_uri: "summary"},
+ file_md5s={file_uri: "new-md5"},
+ )
+
+ await executor.run(root_uri)
+
+ assert processor.summarized_files == [file_uri]
+ assert processor.vectorized_files == [file_uri]
+ assert processor.file_md5s[file_uri] == "new-md5"
+ assert processor.generated_overviews == [child_uri, root_uri]
+ assert processor.vectorized_dirs == [child_uri, root_uri]
+ assert fake_fs._file_contents[f"{child_uri}/.overview.md"] != old_child_overview
+ assert fake_fs._file_contents[f"{root_uri}/.overview.md"] != old_root_overview
+
+
+async def test_modified_file_with_different_abstract_still_propagates(monkeypatch):
+ root_uri = "viking://resources/root"
+ file_uri = f"{root_uri}/a.txt"
+ fake_fs = _FakeVikingFS(
+ tree={root_uri: [{"name": "a.txt", "isDir": False}]},
+ file_contents={
+ file_uri: "changed body",
+ f"{root_uri}/.overview.md": render_abstract_overview(
+ ContextLevel.OVERVIEW, root_uri, "FILES:\n- a.txt: old-summary"
+ ),
+ f"{root_uri}/.abstract.md": render_abstract_overview(
+ ContextLevel.ABSTRACT, root_uri, "old-abstract"
+ ),
+ },
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+
+ processor = _FakeProcessor(fake_fs)
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=2,
+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),
+ incremental_update=True,
+ target_uri=root_uri,
+ changes={"modified": [file_uri]},
+ file_abstracts={file_uri: "old-summary"},
+ )
+
+ await executor.run(root_uri)
+
+ assert processor.generated_overviews == [root_uri]
+ assert processor.vectorized_dirs == [root_uri]
+
+
+async def test_added_file_with_matching_stale_abstract_still_propagates(monkeypatch):
+ root_uri = "viking://resources/root"
+ child_uri = f"{root_uri}/nested"
+ file_uri = f"{child_uri}/a.txt"
+ fake_fs = _FakeVikingFS(
+ tree={
+ root_uri: [{"name": "nested", "isDir": True}],
+ child_uri: [{"name": "a.txt", "isDir": False}],
+ },
+ file_contents={file_uri: "new body"},
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+
+ processor = _FakeProcessor(fake_fs)
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=2,
+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),
+ incremental_update=True,
+ target_uri=root_uri,
+ changes={"added": [file_uri]},
+ file_abstracts={file_uri: "summary"},
+ )
+
+ await executor.run(root_uri)
+
+ assert processor.generated_overviews == [child_uri, root_uri]
+ assert processor.vectorized_dirs == [child_uri, root_uri]
+
+
+async def test_deleted_last_nested_file_still_refreshes_existing_ancestor(monkeypatch):
+ root_uri = "viking://resources/root"
+ deleted_uri = f"{root_uri}/removed/a.txt"
+ fake_fs = _FakeVikingFS(
+ tree={root_uri: []},
+ file_contents={
+ f"{root_uri}/.overview.md": render_abstract_overview(
+ ContextLevel.OVERVIEW, root_uri, "FILES:\n- removed/: old-child"
+ ),
+ f"{root_uri}/.abstract.md": render_abstract_overview(
+ ContextLevel.ABSTRACT, root_uri, "old-root"
+ ),
+ },
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+
+ processor = _FakeProcessor(fake_fs)
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=2,
+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),
+ incremental_update=True,
+ target_uri=root_uri,
+ changes={"deleted": [deleted_uri]},
+ )
+
+ await executor.run(root_uri)
+
+ assert processor.generated_overviews == [root_uri]
+ assert processor.vectorized_dirs == [root_uri]
+
+
+async def test_artifact_files_drive_tree_structure_when_target_listing_is_empty(
+ tmp_path, monkeypatch
+):
+ root_uri = "viking://resources/root"
+ # artifact_files supplies the committed tree structure so the executor does not
+ # depend on a target ls; file bytes are always read from the (already
+ # applied) target via viking_fs.
+ fake_fs = _FakeVikingFS(
+ tree={},
+ file_contents={
+ f"{root_uri}/a.txt": b"alpha",
+ f"{root_uri}/src/b.txt": b"beta",
+ },
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+
+ processor = _FakeProcessor(fake_fs)
+ ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=2,
+ ctx=ctx,
+ artifact_files=["a.txt", "src/b.txt"],
+ )
+
+ await executor.run(root_uri)
+
+ assert sorted(processor.summarized_files) == [
+ f"{root_uri}/a.txt",
+ f"{root_uri}/src/b.txt",
+ ]
+ assert sorted(processor.vectorized_files) == [
+ f"{root_uri}/a.txt",
+ f"{root_uri}/src/b.txt",
+ ]
+ assert processor.file_contents[("summary", f"{root_uri}/a.txt")] == b"alpha"
+ assert processor.file_contents[("vector", f"{root_uri}/src/b.txt")] == b"beta"
+
+
+async def test_direct_incremental_reuses_vector_abstract_without_overview(monkeypatch):
+ root_uri = "viking://resources/root"
+ file_uri = f"{root_uri}/a.txt"
+ fake_fs = _FakeVikingFS(
+ tree={root_uri: [{"name": "a.txt", "isDir": False}]},
+ file_contents={},
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+
+ processor = _FakeProcessor(fake_fs)
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=2,
+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),
+ incremental_update=True,
+ target_uri=root_uri,
+ changes={},
+ artifact_files=["a.txt"],
+ file_abstracts={file_uri: "summary from vector"},
+ )
+
+ await executor.run(root_uri)
+
+ assert processor.summarized_files == []
+ assert processor.vectorized_files == []
+
+
+async def test_missing_target_file_fails_instead_of_using_empty_summary(tmp_path, monkeypatch):
+ root_uri = "viking://resources/root"
+
+ class _MissingFileVikingFS(_FakeVikingFS):
+ async def read_file_bytes(self, path, ctx=None):
+ raise FileNotFoundError(path)
+
+ fake_fs = _MissingFileVikingFS(tree={}, file_contents={})
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+ executor = SemanticTreeExecutor(
+ processor=_FakeProcessor(fake_fs),
+ context_type="resource",
+ max_concurrent_llm=2,
+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),
+ artifact_files=["a.txt"],
+ )
+
+ with pytest.raises(FileNotFoundError):
+ await executor.run(root_uri)
+
+
@pytest.mark.asyncio
async def test_content_write_tags_apply_only_to_changed_file(monkeypatch):
root_uri = "viking://resources/root"
@@ -258,16 +607,18 @@ async def test_content_write_tags_apply_only_to_changed_file(monkeypatch):
f"{root_uri}/.abstract.md": "old-abstract",
},
)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_dag.get_openviking_config",
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
)
processor = _FakeProcessor(fake_fs)
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
tag_options = IngestOptions(search_tags=["team=search"])
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=2,
@@ -285,6 +636,126 @@ async def test_content_write_tags_apply_only_to_changed_file(monkeypatch):
assert processor.directory_ingest_options[root_uri] == IngestOptions()
+async def test_content_write_with_same_abstract_updates_file_and_parent(monkeypatch):
+ root_uri = "viking://resources/root"
+ changed_uri = f"{root_uri}/a.txt"
+ old_overview = render_abstract_overview(
+ ContextLevel.OVERVIEW, root_uri, "FILES:\n- a.txt: summary"
+ )
+ fake_fs = _FakeVikingFS(
+ tree={root_uri: [{"name": "a.txt", "isDir": False}]},
+ file_contents={
+ changed_uri: "new content",
+ f"{root_uri}/.overview.md": old_overview,
+ f"{root_uri}/.abstract.md": render_abstract_overview(
+ ContextLevel.ABSTRACT, root_uri, "old-abstract"
+ ),
+ },
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+
+ processor = _FakeProcessor(fake_fs)
+ tag_options = IngestOptions(search_tags=["team=search"])
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=2,
+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),
+ incremental_update=True,
+ target_uri=root_uri,
+ changes={"modified": [changed_uri]},
+ file_abstracts={changed_uri: "summary"},
+ ingest_options=tag_options,
+ generation_trigger="content_write",
+ )
+
+ await executor.run(root_uri)
+
+ assert processor.file_ingest_options == {changed_uri: tag_options}
+ assert processor.vectorized_files == [changed_uri]
+ assert processor.generated_overviews == [root_uri]
+ assert processor.vectorized_dirs == [root_uri]
+ assert fake_fs._file_contents[f"{root_uri}/.overview.md"] != old_overview
+
+
+async def test_content_write_keeps_matching_upstream_md5(monkeypatch):
+ root_uri = "viking://resources/root"
+ changed_uri = f"{root_uri}/a.txt"
+ latest_content = "newer content"
+ fake_fs = _FakeVikingFS(
+ tree={root_uri: [{"name": "a.txt", "isDir": False}]},
+ file_contents={changed_uri: latest_content},
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+
+ processor = _FakeProcessor(fake_fs)
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=2,
+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),
+ incremental_update=True,
+ target_uri=root_uri,
+ changes={"modified": [changed_uri]},
+ file_md5s={changed_uri: content_md5(latest_content.encode())},
+ generation_trigger="content_write",
+ )
+
+ await executor.run(root_uri)
+
+ assert processor.file_contents[("vector", changed_uri)] == latest_content.encode()
+ assert processor.file_md5s[changed_uri] == content_md5(latest_content.encode())
+
+
+async def test_semantic_generate_hashes_read_content_when_manifest_md5_is_missing(monkeypatch):
+ root_uri = "viking://resources/root"
+ changed_uri = f"{root_uri}/a.txt"
+ content = b"content read by semantic worker"
+ fake_fs = _FakeVikingFS(
+ tree={root_uri: [{"name": "a.txt", "isDir": False}]},
+ file_contents={changed_uri: content},
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+
+ processor = _FakeProcessor(fake_fs)
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=2,
+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),
+ incremental_update=True,
+ target_uri=root_uri,
+ changes={"modified": [changed_uri]},
+ generation_trigger="resource_ingest",
+ )
+
+ await executor.run(root_uri)
+
+ assert processor.file_contents[("vector", changed_uri)] == content
+ assert processor.file_md5s[changed_uri] == content_md5(content)
+
+
@pytest.mark.asyncio
async def test_pending_refresh_rebuilds_every_sampled_file_summary(monkeypatch):
root_uri = "viking://resources/wide"
@@ -308,15 +779,17 @@ async def test_pending_refresh_rebuilds_every_sampled_file_summary(monkeypatch):
),
},
)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_dag.get_openviking_config",
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=4)),
)
processor = _FakeProcessor(fake_fs)
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=2,
@@ -325,6 +798,7 @@ async def test_pending_refresh_rebuilds_every_sampled_file_summary(monkeypatch):
target_uri=root_uri,
recursive=False,
changes={"modified": [changed_path]},
+ file_abstracts={changed_path: "summary"},
)
await executor.run(root_uri)
@@ -343,9 +817,11 @@ async def test_directory_vectorization_retries_after_matching_sidecar_write(monk
tree={root_uri: [{"name": "a.txt", "isDir": False}]},
file_contents={file_path: "new content"},
)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_dag.get_openviking_config",
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
)
@@ -355,7 +831,7 @@ async def test_directory_vectorization_retries_after_matching_sidecar_write(monk
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
def make_executor():
- return SemanticDagExecutor(
+ return SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=2,
@@ -409,9 +885,11 @@ async def test_content_copy_rebuilds_target_overview_from_target_l2_summaries(mo
),
},
)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_dag.get_openviking_config",
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
)
processor = _FakeProcessor(
@@ -421,7 +899,7 @@ async def test_content_copy_rebuilds_target_overview_from_target_l2_summaries(mo
f"{root_uri}/keep.txt": "kept target L2 summary",
},
)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=2,
@@ -456,16 +934,18 @@ async def test_content_copy_samples_before_loading_summaries(monkeypatch):
tree={root_uri: [{"name": path.rsplit("/", 1)[-1], "isDir": False} for path in file_paths]},
file_contents={path: path for path in file_paths},
)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_dag.get_openviking_config",
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=2)),
)
processor = _FakeProcessor(
fake_fs,
transfer_summaries={path: f"summary-{path.rsplit('/', 1)[-1]}" for path in file_paths},
)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=2,
@@ -495,16 +975,18 @@ async def test_content_copy_does_not_backfill_missing_sample_summary(monkeypatch
tree={root_uri: [{"name": path.rsplit("/", 1)[-1], "isDir": False} for path in file_paths]},
file_contents={path: path for path in file_paths},
)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_dag.get_openviking_config",
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=2)),
)
processor = _FakeProcessor(
fake_fs,
transfer_summaries={path: f"summary-{path.rsplit('/', 1)[-1]}" for path in file_paths[1:]},
)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=2,
@@ -540,13 +1022,15 @@ async def test_content_copy_with_no_ready_summaries_preserves_existing_sidecars(
f"{root_uri}/.abstract.md": old_abstract,
},
)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_dag.get_openviking_config",
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
)
processor = _FakeProcessor(fake_fs)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=2,
@@ -578,13 +1062,15 @@ async def test_content_copy_rebuilds_semantics_when_move_leaves_source_directory
f"{root_uri}/.abstract.md": old_abstract,
},
)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_dag.get_openviking_config",
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
)
processor = _FakeProcessor(fake_fs)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=2,
@@ -614,9 +1100,11 @@ async def test_content_copy_propagates_vector_summary_read_failure(monkeypatch):
tree={root_uri: [{"name": "copied.txt", "isDir": False}]},
file_contents={file_path: "copied"},
)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_dag.get_openviking_config",
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
)
processor = _FakeProcessor(fake_fs)
@@ -625,7 +1113,7 @@ async def test_content_copy_propagates_vector_summary_read_failure(monkeypatch):
raise RuntimeError("vector backend unavailable")
processor._load_transfer_file_summaries = fail_summary_read
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=2,
diff --git a/tests/storage/test_semantic_dag_media_overview.py b/tests/storage/test_semantic_executor_media_overview.py
similarity index 88%
rename from tests/storage/test_semantic_dag_media_overview.py
rename to tests/storage/test_semantic_executor_media_overview.py
index 0f6521660..88fa7b0ee 100644
--- a/tests/storage/test_semantic_dag_media_overview.py
+++ b/tests/storage/test_semantic_executor_media_overview.py
@@ -5,9 +5,9 @@ from types import SimpleNamespace
from unittest.mock import patch
from openviking.server.identity import RequestContext, Role
-from openviking.storage.queuefs.semantic_dag import (
+from openviking.storage.queuefs.semantic_executor import (
DirNode,
- SemanticDagExecutor,
+ SemanticTreeExecutor,
)
from openviking_cli.session.user_id import UserIdentifier
@@ -18,10 +18,10 @@ def _executor():
role=Role.USER,
)
with patch(
- "openviking.storage.queuefs.semantic_dag.get_viking_fs",
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs",
return_value=SimpleNamespace(),
):
- return SemanticDagExecutor(SimpleNamespace(), "resource", 2, ctx)
+ return SemanticTreeExecutor(SimpleNamespace(), "resource", 2, ctx)
def _node(
diff --git a/tests/storage/test_semantic_dag_skip_files.py b/tests/storage/test_semantic_executor_skip_files.py
similarity index 88%
rename from tests/storage/test_semantic_dag_skip_files.py
rename to tests/storage/test_semantic_executor_skip_files.py
index 9b6092063..8632d4d37 100644
--- a/tests/storage/test_semantic_dag_skip_files.py
+++ b/tests/storage/test_semantic_executor_skip_files.py
@@ -6,7 +6,7 @@ from types import SimpleNamespace
import pytest
from openviking.server.identity import RequestContext, Role
-from openviking.storage.queuefs.semantic_dag import SemanticDagExecutor
+from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor
from openviking_cli.session.user_id import UserIdentifier
@@ -89,6 +89,7 @@ class _FakeProcessor:
use_summary=False,
ingest_options=None,
creator_acl_grant=None,
+ file_md5=None,
):
self.vectorized_files.append(file_path)
@@ -105,11 +106,13 @@ async def test_messages_jsonl_excluded_from_summary(monkeypatch):
],
}
fake_fs = _FakeVikingFS(tree)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
processor = _FakeProcessor()
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="session",
max_concurrent_llm=2,
@@ -137,11 +140,13 @@ async def test_messages_jsonl_excluded_in_subdirectory(monkeypatch):
],
}
fake_fs = _FakeVikingFS(tree)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
processor = _FakeProcessor()
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="session",
max_concurrent_llm=2,
@@ -165,15 +170,17 @@ async def test_unlistable_semantic_root_is_not_materialized_as_directory(
fail_sidecar_write=fail_sidecar_write,
list_error=list_error,
)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_dag.get_openviking_config",
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
)
processor = _FakeProcessor()
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="memory",
max_concurrent_llm=2,
@@ -192,15 +199,17 @@ async def test_unlistable_semantic_root_is_not_materialized_as_directory(
async def test_empty_semantic_directory_still_receives_sidecars(monkeypatch):
root_uri = "viking://user/user1/memories/empty"
fake_fs = _FakeVikingFS({root_uri: []})
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_dag.get_openviking_config",
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
)
processor = _FakeProcessor()
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="memory",
max_concurrent_llm=2,
diff --git a/tests/storage/test_semantic_dag_stats.py b/tests/storage/test_semantic_executor_stats.py
similarity index 89%
rename from tests/storage/test_semantic_dag_stats.py
rename to tests/storage/test_semantic_executor_stats.py
index 4be498fb0..9f0cee542 100644
--- a/tests/storage/test_semantic_dag_stats.py
+++ b/tests/storage/test_semantic_executor_stats.py
@@ -23,11 +23,11 @@ from openviking.storage.abstract_overview import (
)
from openviking.storage.errors import LockAcquisitionError
from openviking.storage.queuefs.named_queue import NamedQueue
-from openviking.storage.queuefs.semantic_dag import (
- DagStats,
- DagWork,
- SemanticDagExecutor,
- SemanticNodeScheduler,
+from openviking.storage.queuefs.semantic_executor import (
+ SemanticTreeExecutor,
+ SemanticTreeScheduler,
+ SemanticTreeStats,
+ SemanticTreeWork,
)
from openviking.telemetry import (
OperationTelemetry,
@@ -114,6 +114,7 @@ class _FakeProcessor:
use_summary=False,
ingest_options=None,
creator_acl_grant=None,
+ file_md5=None,
):
if self.verify_streaming:
assert summary_dict["content"]
@@ -165,7 +166,7 @@ class _ScheduledExecutor:
def _patch_semantic_config(monkeypatch, *, overview_sample_limit=32):
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_dag.get_openviking_config",
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
lambda: SimpleNamespace(
semantic=SimpleNamespace(overview_sample_limit=overview_sample_limit)
),
@@ -173,7 +174,7 @@ def _patch_semantic_config(monkeypatch, *, overview_sample_limit=32):
@pytest.mark.asyncio
-async def test_semantic_dag_stats_collects_nodes(monkeypatch):
+async def test_semantic_executor_stats_collects_nodes(monkeypatch):
root_uri = "viking://resources/root"
tree = {
root_uri: [
@@ -186,12 +187,14 @@ async def test_semantic_dag_stats_collects_nodes(monkeypatch):
],
}
fake_fs = _FakeVikingFS(tree)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
_patch_semantic_config(monkeypatch)
processor = _FakeProcessor(verify_streaming=True)
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=2,
@@ -201,7 +204,7 @@ async def test_semantic_dag_stats_collects_nodes(monkeypatch):
await asyncio.sleep(0)
stats = executor.get_stats()
- assert isinstance(stats, DagStats)
+ assert isinstance(stats, SemanticTreeStats)
assert stats.total_nodes == 5 # 2 dirs + 3 files
assert stats.pending_nodes == 0
assert stats.done_nodes == 5
@@ -217,18 +220,20 @@ async def test_semantic_dag_stats_collects_nodes(monkeypatch):
@pytest.mark.asyncio
-async def test_semantic_dag_bounds_active_node_work(monkeypatch):
+async def test_semantic_executor_bounds_active_node_work(monkeypatch):
root_uri = "viking://resources/root"
tree = {
root_uri: [{"name": f"file-{idx}.txt", "isDir": False} for idx in range(40)],
}
fake_fs = _FakeVikingFS(tree)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
_patch_semantic_config(monkeypatch)
processor = _TrackingProcessor()
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=3,
@@ -259,12 +264,14 @@ async def test_incremental_wide_directory_samples_before_summary_work(monkeypatc
root_uri: [{"name": f"file-{idx:03}.txt", "isDir": False} for idx in range(40)],
}
fake_fs = _FakeVikingFS(tree)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
_patch_semantic_config(monkeypatch, overview_sample_limit=4)
processor = _FakeProcessor()
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=2,
@@ -302,12 +309,14 @@ async def test_non_recursive_memory_samples_files_and_reads_child_abstracts(monk
child_d: [{"name": "nested.md", "isDir": False}],
}
fake_fs = _FakeVikingFS(tree, abstracts={child_a: "child a abstract"})
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
_patch_semantic_config(monkeypatch, overview_sample_limit=3)
processor = _FakeProcessor()
ctx = RequestContext(user=UserIdentifier("acc1", "alice"), role=Role.USER)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="memory",
max_concurrent_llm=2,
@@ -350,7 +359,9 @@ async def test_busy_parent_snapshot_preserves_changed_file_work(monkeypatch):
raise FileNotFoundError(uri)
monkeypatch.setattr(fake_fs, "read_file", read_file, raising=False)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
_patch_semantic_config(monkeypatch, overview_sample_limit=4)
monkeypatch.setattr(
fake_fs, "pathlock_acquire_exact_batch", AsyncMock(side_effect=LockAcquisitionError("busy"))
@@ -358,7 +369,7 @@ async def test_busy_parent_snapshot_preserves_changed_file_work(monkeypatch):
processor = _FakeProcessor()
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=2,
@@ -378,7 +389,7 @@ async def test_busy_parent_snapshot_preserves_changed_file_work(monkeypatch):
@pytest.mark.asyncio
-async def test_semantic_dag_shares_node_scheduler_across_roots(monkeypatch):
+async def test_semantic_executor_shares_node_scheduler_across_roots(monkeypatch):
root_a = "viking://resources/root-a"
root_b = "viking://resources/root-b"
tree = {
@@ -386,7 +397,9 @@ async def test_semantic_dag_shares_node_scheduler_across_roots(monkeypatch):
root_b: [{"name": f"b-{idx}.txt", "isDir": False} for idx in range(20)],
}
fake_fs = _FakeVikingFS(tree)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
_patch_semantic_config(monkeypatch)
processor = _TrackingProcessor()
@@ -397,7 +410,7 @@ async def test_semantic_dag_shares_node_scheduler_across_roots(monkeypatch):
bind_task_context("task-a", "acc1", "user1"),
bind_telemetry(telemetry_a),
):
- executor_a = SemanticDagExecutor(
+ executor_a = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=1,
@@ -407,7 +420,7 @@ async def test_semantic_dag_shares_node_scheduler_across_roots(monkeypatch):
bind_task_context("task-b", "acc1", "user1"),
bind_telemetry(telemetry_b),
):
- executor_b = SemanticDagExecutor(
+ executor_b = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=1,
@@ -461,9 +474,9 @@ async def test_task_work_rejection_does_not_stop_shared_semantic_worker():
rejected = _ScheduledExecutor(rejected_work)
unrelated = _ScheduledExecutor(unrelated_work)
- scheduler = SemanticNodeScheduler(max_workers=1)
- scheduler.submit(rejected, DagWork(kind="vectorize", dir_uri="a"))
- scheduler.submit(unrelated, DagWork(kind="vectorize", dir_uri="b"))
+ scheduler = SemanticTreeScheduler(max_workers=1)
+ scheduler.submit(rejected, SemanticTreeWork(kind="vectorize", dir_uri="a"))
+ scheduler.submit(unrelated, SemanticTreeWork(kind="vectorize", dir_uri="b"))
await asyncio.wait_for(unrelated_ran.wait(), timeout=0.5)
await asyncio.wait_for(scheduler._queue.join(), timeout=0.5)
@@ -476,7 +489,7 @@ async def test_task_work_rejection_does_not_stop_shared_semantic_worker():
@pytest.mark.asyncio
-async def test_semantic_dag_skip_vectorization_does_not_schedule_tasks(monkeypatch):
+async def test_semantic_executor_skip_vectorization_does_not_schedule_tasks(monkeypatch):
root_uri = "viking://resources/root"
tree = {
root_uri: [
@@ -488,12 +501,14 @@ async def test_semantic_dag_skip_vectorization_does_not_schedule_tasks(monkeypat
],
}
fake_fs = _FakeVikingFS(tree)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fake_fs)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
_patch_semantic_config(monkeypatch)
processor = _FakeProcessor()
ctx = RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER)
- executor = SemanticDagExecutor(
+ executor = SemanticTreeExecutor(
processor=processor,
context_type="resource",
max_concurrent_llm=2,
diff --git a/tests/storage/test_semantic_msg_ingest_options.py b/tests/storage/test_semantic_msg_ingest_options.py
index 53d1922dd..a9a31f89a 100644
--- a/tests/storage/test_semantic_msg_ingest_options.py
+++ b/tests/storage/test_semantic_msg_ingest_options.py
@@ -74,3 +74,52 @@ def test_semantic_msg_round_trips_hierarchical_aggregation_policy():
assert restored.use_hierarchical_aggregation is True
assert restored.propagate_to_parent is False
+
+
+def test_semantic_msg_roundtrip_preserves_file_md5s():
+ msg = SemanticMsg(
+ uri="viking://resources/x",
+ context_type="resource",
+ file_md5s={"viking://resources/x/a.py": "md5a"},
+ )
+
+ assert SemanticMsg.from_dict(msg.to_dict()).file_md5s == {"viking://resources/x/a.py": "md5a"}
+
+
+def test_semantic_msg_defaults_file_md5s_to_empty():
+ msg = SemanticMsg(uri="viking://resources/x", context_type="resource")
+
+ assert msg.file_md5s == {}
+ assert SemanticMsg.from_dict(msg.to_dict()).file_md5s == {}
+
+
+def test_semantic_msg_roundtrip_preserves_queue_enqueue_time():
+ msg = SemanticMsg(
+ uri="viking://resources/x",
+ context_type="resource",
+ queue_enqueued_at=123.456,
+ )
+
+ assert SemanticMsg.from_dict(msg.to_dict()).queue_enqueued_at == 123.456
+
+
+def test_semantic_msg_roundtrip_preserves_local_artifact_snapshot():
+ msg = SemanticMsg(
+ uri="viking://resources/x",
+ context_type="resource",
+ artifact_ref={
+ "backend": "local",
+ "root": "/tmp/artifact-1",
+ "resource_rel": "repository",
+ "root_type": "dir",
+ },
+ artifact_files=["a.py", "src/b.py"],
+ file_abstracts={"viking://resources/x/a.py": "summary a"},
+ )
+
+ restored = SemanticMsg.from_json(msg.to_json())
+
+ assert restored.artifact_ref == msg.artifact_ref
+ assert restored.artifact_ref["resource_rel"] == "repository"
+ assert restored.artifact_files == ["a.py", "src/b.py"]
+ assert restored.file_abstracts == {"viking://resources/x/a.py": "summary a"}
diff --git a/tests/storage/test_semantic_processor_target_preexisting.py b/tests/storage/test_semantic_processor_target_preexisting.py
index 8df69a6bc..caf822e0c 100644
--- a/tests/storage/test_semantic_processor_target_preexisting.py
+++ b/tests/storage/test_semantic_processor_target_preexisting.py
@@ -61,42 +61,59 @@ class _SyncWrapperVikingFS:
return {"matches": []}
-class _FakeDagExecutor:
+class _FakeTreeExecutor:
calls = []
runs = []
def __init__(self, **kwargs):
self.kwargs = kwargs
self.stale = False
- _FakeDagExecutor.calls.append(kwargs)
+ _FakeTreeExecutor.calls.append(kwargs)
async def run(self, root_uri):
self.root_uri = root_uri
- _FakeDagExecutor.runs.append(root_uri)
+ _FakeTreeExecutor.runs.append(root_uri)
def get_stats(self):
- from openviking.storage.queuefs.semantic_dag import DagStats
+ from openviking.storage.queuefs.semantic_executor import SemanticTreeStats
- return DagStats()
+ return SemanticTreeStats()
@pytest.mark.asyncio
-async def test_target_source_syncs_before_semantic_dag(monkeypatch):
+async def test_vectorize_directory_returns_enqueued_levels(monkeypatch):
+ vectorize = AsyncMock(return_value={0, 1})
+ monkeypatch.setattr("openviking.utils.embedding_utils.vectorize_directory_meta", vectorize)
+ processor = object.__new__(SemanticProcessor)
+ processor._default_ctx = object()
+
+ result = await processor._vectorize_directory(
+ "viking://resources/root",
+ "resource",
+ "abstract",
+ "overview",
+ )
+
+ assert result == {0, 1}
+
+
+async def test_target_source_syncs_before_semantic_executor(monkeypatch):
monkeypatch.setattr(
"openviking.storage.queuefs.semantic_processor.get_viking_fs",
lambda: _FakeVikingFS(),
)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor",
- _FakeDagExecutor,
+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor",
+ _FakeTreeExecutor,
)
monkeypatch.setattr(
"openviking.storage.queuefs.semantic_processor.SemanticLockScope.resolve",
AsyncMock(return_value=SimpleNamespace(lock=None, close=AsyncMock())),
)
- _FakeDagExecutor.calls = []
- _FakeDagExecutor.runs = []
+ _FakeTreeExecutor.calls = []
+ _FakeTreeExecutor.runs = []
processor = SemanticProcessor()
processor._enqueue_parent_refresh = AsyncMock()
processor._sync_topdown_recursive = AsyncMock(
@@ -113,14 +130,14 @@ async def test_target_source_syncs_before_semantic_dag(monkeypatch):
await processor.on_dequeue(msg.to_dict())
- assert _FakeDagExecutor.calls[0]["incremental_update"] is True
- assert _FakeDagExecutor.calls[0]["target_uri"] == "viking://resources/org/repo"
- assert _FakeDagExecutor.calls[0]["changes"] == {
+ assert _FakeTreeExecutor.calls[0]["incremental_update"] is True
+ assert _FakeTreeExecutor.calls[0]["target_uri"] == "viking://resources/org/repo"
+ assert _FakeTreeExecutor.calls[0]["changes"] == {
"added": [],
"modified": ["viking://resources/org/repo/a.md"],
"deleted": [],
}
- assert _FakeDagExecutor.runs == ["viking://resources/org/repo"]
+ assert _FakeTreeExecutor.runs == ["viking://resources/org/repo"]
@pytest.mark.asyncio
@@ -130,8 +147,8 @@ async def test_stale_content_write_keeps_file_work_without_directory_aggregation
lambda: _FakeVikingFS(),
)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor",
- _FakeDagExecutor,
+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor",
+ _FakeTreeExecutor,
)
monkeypatch.setattr(
"openviking.storage.queuefs.semantic_processor.SemanticLockScope.resolve",
@@ -142,8 +159,8 @@ async def test_stale_content_write_keeps_file_work_without_directory_aggregation
lambda msg: bool(msg.coalesce_key),
)
- _FakeDagExecutor.calls = []
- _FakeDagExecutor.runs = []
+ _FakeTreeExecutor.calls = []
+ _FakeTreeExecutor.runs = []
processor = SemanticProcessor()
processor._enqueue_parent_refresh = AsyncMock()
changed = "viking://resources/wiki/changed.md"
@@ -159,31 +176,31 @@ async def test_stale_content_write_keeps_file_work_without_directory_aggregation
await processor.on_dequeue(msg.to_dict())
- assert _FakeDagExecutor.calls[0]["aggregate_directory"] is False
- assert _FakeDagExecutor.calls[0]["changes"] == {"modified": [changed]}
- assert _FakeDagExecutor.calls[0]["coalesce_key"] == ""
- assert _FakeDagExecutor.runs == ["viking://resources/wiki"]
+ assert _FakeTreeExecutor.calls[0]["aggregate_directory"] is False
+ assert _FakeTreeExecutor.calls[0]["changes"] == {"modified": [changed]}
+ assert _FakeTreeExecutor.calls[0]["coalesce_key"] == ""
+ assert _FakeTreeExecutor.runs == ["viking://resources/wiki"]
processor._enqueue_parent_refresh.assert_not_awaited()
@pytest.mark.asyncio
@pytest.mark.parametrize("recursive", [False, True])
-async def test_memory_reindex_uses_semantic_dag(monkeypatch, recursive):
+async def test_memory_reindex_uses_semantic_executor(monkeypatch, recursive):
monkeypatch.setattr(
"openviking.storage.queuefs.semantic_processor.get_viking_fs",
lambda: _FakeVikingFS(),
)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor",
- _FakeDagExecutor,
+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor",
+ _FakeTreeExecutor,
)
monkeypatch.setattr(
"openviking.storage.queuefs.semantic_processor.SemanticLockScope.resolve",
AsyncMock(return_value=SimpleNamespace(lock=None, close=AsyncMock())),
)
- _FakeDagExecutor.calls = []
- _FakeDagExecutor.runs = []
+ _FakeTreeExecutor.calls = []
+ _FakeTreeExecutor.runs = []
processor = SemanticProcessor()
processor._process_memory_directory = AsyncMock()
msg = SemanticMsg(
@@ -197,9 +214,9 @@ async def test_memory_reindex_uses_semantic_dag(monkeypatch, recursive):
await processor.on_dequeue(msg.to_dict())
processor._process_memory_directory.assert_not_awaited()
- assert _FakeDagExecutor.calls[0]["context_type"] == "memory"
- assert _FakeDagExecutor.calls[0]["recursive"] is recursive
- assert _FakeDagExecutor.runs == [msg.uri]
+ assert _FakeTreeExecutor.calls[0]["context_type"] == "memory"
+ assert _FakeTreeExecutor.calls[0]["recursive"] is recursive
+ assert _FakeTreeExecutor.runs == [msg.uri]
@pytest.mark.asyncio
diff --git a/tests/storage/test_skill_package_cancellation.py b/tests/storage/test_skill_package_cancellation.py
index 623ebe301..e19b7f74a 100644
--- a/tests/storage/test_skill_package_cancellation.py
+++ b/tests/storage/test_skill_package_cancellation.py
@@ -23,13 +23,13 @@ from openviking.storage.collection_schemas import TextEmbeddingHandler
from openviking.storage.queuefs.embedding_msg import EmbeddingMsg
from openviking.storage.queuefs.named_queue import NamedQueue
from openviking.storage.queuefs.process_result import ProcessOutcome
-from openviking.storage.queuefs.semantic_dag import DagStats, SemanticDagExecutor
+from openviking.storage.queuefs.semantic_executor import SemanticTreeExecutor, SemanticTreeStats
from openviking.storage.queuefs.semantic_msg import SemanticMsg
from openviking.storage.queuefs.semantic_processor import SemanticProcessor
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
from openviking_cli.session.user_id import UserIdentifier
from tests.storage.test_collection_schemas import _DummyConfig, _DummyEmbedder
-from tests.storage.test_semantic_dag_skip_files import _FakeVikingFS
+from tests.storage.test_semantic_executor_skip_files import _FakeVikingFS
def _ctx():
@@ -57,7 +57,7 @@ async def test_cancel_skill_stops_model_work_without_cancelling_other_packages(m
resource: [{"name": "resource.md"}],
}
)
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fs)
+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)
processor = _processor()
started = asyncio.Event()
cancelled = asyncio.Event()
@@ -82,13 +82,13 @@ async def test_cancel_skill_stops_model_work_without_cancelling_other_packages(m
processor._generate_single_file_summary = summarize
processor._vectorize_single_file = vectorize
with bind_task_context("skill-a", "acc", "alice"):
- executor_a = SemanticDagExecutor(processor, "skill", 1, _ctx())
+ executor_a = SemanticTreeExecutor(processor, "skill", 1, _ctx())
task_a = asyncio.create_task(executor_a.run(root_a))
await asyncio.wait_for(started.wait(), 1)
with bind_task_context("skill-b", "acc", "alice"):
- executor_b = SemanticDagExecutor(processor, "skill", 1, _ctx())
+ executor_b = SemanticTreeExecutor(processor, "skill", 1, _ctx())
task_b = asyncio.create_task(executor_b.run(root_b))
- executor_resource = SemanticDagExecutor(processor, "resource", 1, _ctx())
+ executor_resource = SemanticTreeExecutor(processor, "resource", 1, _ctx())
task_resource = asyncio.create_task(executor_resource.run(resource))
task_a.cancel()
with pytest.raises(asyncio.CancelledError):
@@ -104,7 +104,7 @@ async def test_cancel_skill_stops_model_work_without_cancelling_other_packages(m
async def test_cancel_skill_waits_for_started_threaded_sidecar_write(monkeypatch):
root = "viking://agent/skills/demo/reference"
fs = _FakeVikingFS({root: []})
- monkeypatch.setattr("openviking.storage.queuefs.semantic_dag.get_viking_fs", lambda: fs)
+ monkeypatch.setattr("openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fs)
started = asyncio.Event()
release = threading.Event()
loop = asyncio.get_running_loop()
@@ -119,7 +119,7 @@ async def test_cancel_skill_waits_for_started_threaded_sidecar_write(monkeypatch
fs.write_file = write
processor = _processor()
- executor = SemanticDagExecutor(processor, "skill", 1, _ctx())
+ executor = SemanticTreeExecutor(processor, "skill", 1, _ctx())
worker = asyncio.create_task(executor.run(root))
try:
await asyncio.wait_for(started.wait(), 1)
@@ -251,7 +251,7 @@ async def test_semantic_cancel_drains_embeddings_before_releasing_package(monkey
started.set()
def get_stats(self):
- return DagStats()
+ return SemanticTreeStats()
class Lease:
lock = {"lease_ref": "skill"}
@@ -260,7 +260,7 @@ async def test_semantic_cancel_drains_embeddings_before_releasing_package(monkey
events.append("released")
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor", Executor
+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor", Executor
)
monkeypatch.setattr(
SemanticProcessor, "_resolve_skill_semantic_lock", AsyncMock(return_value=Lease())
diff --git a/tests/storage/test_skill_package_semantics.py b/tests/storage/test_skill_package_semantics.py
index fd15f873b..4c1060179 100644
--- a/tests/storage/test_skill_package_semantics.py
+++ b/tests/storage/test_skill_package_semantics.py
@@ -4,7 +4,7 @@ from unittest.mock import AsyncMock
import pytest
-from openviking.storage.queuefs.semantic_dag import DagStats
+from openviking.storage.queuefs.semantic_executor import SemanticTreeStats
from openviking.storage.queuefs.semantic_msg import SemanticMsg
from openviking.storage.queuefs.semantic_processor import SemanticProcessor
from openviking.telemetry import get_current_telemetry, register_telemetry
@@ -47,10 +47,10 @@ async def test_skill_worker_keeps_package_locked_until_embeddings_finish(monkeyp
emitted.set()
def get_stats(self):
- return DagStats()
+ return SemanticTreeStats()
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor", Executor
+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor", Executor
)
monkeypatch.setattr(
"openviking.storage.queuefs.semantic_processor.SemanticLockScope.resolve",
diff --git a/tests/storage/test_vector_transfer.py b/tests/storage/test_vector_transfer.py
index 9d221d076..d3cbd35a7 100644
--- a/tests/storage/test_vector_transfer.py
+++ b/tests/storage/test_vector_transfer.py
@@ -230,6 +230,7 @@ class _RealAclMemoryTransferBackend(_MemoryTransferBackend):
def __init__(self, records: list[dict[str, Any]]) -> None:
super().__init__(records)
self.acl_manager = AclManager(self, _EnabledAclConfig())
+ self.scroll_output_fields = []
async def scroll(
self,
@@ -240,7 +241,8 @@ class _RealAclMemoryTransferBackend(_MemoryTransferBackend):
*,
ctx: RequestContext,
) -> tuple[list[dict[str, Any]], str | None]:
- del output_fields, ctx
+ del ctx
+ self.scroll_output_fields.append(list(output_fields or []))
self.scroll_filters.append(filter)
offset = int(cursor or 0)
ordered = [
@@ -628,6 +630,472 @@ async def test_legacy_transfer_reads_use_private_adapter_query_and_fetch(monkeyp
assert paths == ["/api/vikingdb/data/search/vector", "/api/vikingdb/data/fetch_in_collection"]
+async def test_l2_diff_scan_reads_every_page_under_target_directory():
+ root = "viking://resources/docs"
+ backend = _RealAclMemoryTransferBackend(
+ [
+ _record("a", f"{root}/a.py", md5="ma", abstract="A"),
+ _record("b", f"{root}/sub/b.py", md5="mb", abstract="B"),
+ _record("ghost", f"{root}/ghost.py", md5="mg", abstract="G"),
+ _record("directory", root, level=1),
+ _record("outside", "viking://resources/other.py"),
+ _record("other-account", f"{root}/private.py", account_id="other"),
+ ]
+ )
+
+ records = await backend.get_l2_diff_records_under_uri(root, ctx=_ctx(), batch_size=2)
+
+ assert records == {
+ f"{root}/a.py": {"md5": "ma", "abstract": "A"},
+ f"{root}/ghost.py": {"md5": "mg", "abstract": "G"},
+ f"{root}/sub/b.py": {"md5": "mb", "abstract": "B"},
+ }
+ assert len(backend.scroll_filters) == 2
+
+
+async def test_incremental_inventory_reads_all_semantic_levels_under_target():
+ root = "viking://resources/docs"
+ backend = _RealAclMemoryTransferBackend(
+ [
+ _record("root-l0", root, level=0, abstract="root abstract"),
+ _record("root-l1", root, level=1, abstract="root overview"),
+ _record("file-l2", f"{root}/a.py", level=2, md5="ma"),
+ _record("outside", "viking://resources/other.py", level=2),
+ _record(
+ "other-account",
+ f"{root}/private.py",
+ level=2,
+ account_id="other",
+ ),
+ ]
+ )
+
+ records = await backend.get_incremental_inventory_under_uri(root, ctx=_ctx(), batch_size=2)
+
+ assert records == {
+ "root-l0": {"id": "root-l0", "uri": root, "level": 0, "md5": ""},
+ "root-l1": {"id": "root-l1", "uri": root, "level": 1, "md5": ""},
+ "file-l2": {
+ "id": "file-l2",
+ "uri": f"{root}/a.py",
+ "level": 2,
+ "md5": "ma",
+ },
+ }
+ assert len(backend.scroll_filters) == 2
+
+
+async def test_incremental_inventory_projects_request_scalar_fields():
+ root = "viking://resources/docs"
+ backend = _RealAclMemoryTransferBackend(
+ [
+ _record(
+ "file-l2",
+ f"{root}/a.py",
+ level=2,
+ md5="ma",
+ search_tags=["env=test"],
+ )
+ ]
+ )
+
+ records = await backend.get_incremental_inventory_under_uri(
+ root,
+ ctx=_ctx(),
+ output_fields=["id", "uri", "level", "md5", "search_tags"],
+ )
+
+ assert records["file-l2"]["search_tags"] == ["env=test"]
+ assert all("search_tags" in fields for fields in backend.scroll_output_fields)
+ assert all("vector" not in fields for fields in backend.scroll_output_fields)
+ assert all("sparse_vector" not in fields for fields in backend.scroll_output_fields)
+
+
+async def test_incremental_hydration_uses_dsl_then_fetches_only_missing_ids():
+ root = "viking://resources/docs"
+ first = _record("root-l0", root, level=0, abstract="root abstract")
+ second = _record("file-l2", f"{root}/a.py", level=2, md5="ma")
+ backend = _MemoryTransferBackend([first, second])
+ backend._strict_transfer_page = AsyncMock(return_value=([dict(first)], None))
+ backend._strict_transfer_get = AsyncMock(return_value=[dict(second)])
+
+ records = await backend.hydrate_incremental_records(
+ {
+ "root-l0": {"uri": root, "level": 0},
+ "file-l2": {"uri": f"{root}/a.py", "level": 2},
+ },
+ ctx=_ctx(),
+ )
+
+ assert set(records) == {"root-l0", "file-l2"}
+ assert records["root-l0"]["abstract"] == "root abstract"
+ assert records["file-l2"]["md5"] == "ma"
+ backend._strict_transfer_get.assert_awaited_once_with(_ctx(), ["file-l2"])
+ query = backend._strict_transfer_page.await_args
+ assert query.args[1] == In("id", ["root-l0", "file-l2"])
+ assert "vector" not in query.kwargs["output_fields"]
+ assert "sparse_vector" not in query.kwargs["output_fields"]
+
+
+async def test_incremental_hydration_propagates_dsl_failure_without_fallback():
+ backend = _MemoryTransferBackend([])
+ backend._strict_transfer_page = AsyncMock(side_effect=RuntimeError("query failed"))
+ backend._strict_transfer_get = AsyncMock()
+
+ with pytest.raises(RuntimeError, match="query failed"):
+ await backend.hydrate_incremental_records(
+ {"file-l2": {"uri": "viking://resources/docs/a.py", "level": 2}},
+ ctx=_ctx(),
+ )
+
+ backend._strict_transfer_get.assert_not_awaited()
+
+
+async def test_incremental_hydration_rejects_mismatched_record_identity():
+ root = "viking://resources/docs"
+ backend = _MemoryTransferBackend([])
+ backend._strict_transfer_page = AsyncMock(
+ return_value=(
+ [_record("file-l2", f"{root}/wrong.py", level=2)],
+ None,
+ )
+ )
+
+ with pytest.raises(RuntimeError, match="identity mismatch"):
+ await backend.hydrate_incremental_records(
+ {"file-l2": {"uri": f"{root}/a.py", "level": 2}},
+ ctx=_ctx(),
+ )
+
+
+async def test_incremental_hydration_returns_only_records_still_present():
+ backend = _MemoryTransferBackend([])
+ backend._strict_transfer_page = AsyncMock(return_value=([], None))
+ backend._strict_transfer_get = AsyncMock(return_value=[])
+
+ records = await backend.hydrate_incremental_records(
+ {"gone": {"uri": "viking://resources/docs/a.py", "level": 2}},
+ ctx=_ctx(),
+ )
+
+ assert records == {}
+
+
+async def test_incremental_hydration_projects_dynamic_non_vector_schema_fields():
+ root = "viking://resources/docs"
+ record = _record(
+ "file-l2",
+ f"{root}/a.py",
+ level=2,
+ business_priority=7,
+ )
+ backend = _MemoryTransferBackend([record])
+ backend.get_collection_meta = AsyncMock(
+ return_value={
+ "Fields": [
+ {"FieldName": "id"},
+ {"FieldName": "uri"},
+ {"FieldName": "level"},
+ {"FieldName": "abstract"},
+ {"FieldName": "business_priority"},
+ {"FieldName": "vector"},
+ {"FieldName": "sparse_vector"},
+ {"FieldName": "content"},
+ ]
+ }
+ )
+ backend._strict_transfer_page = AsyncMock(return_value=([record], None))
+
+ hydrated = await backend.hydrate_incremental_records(
+ {"file-l2": {"uri": f"{root}/a.py", "level": 2}},
+ ctx=_ctx(),
+ )
+
+ assert hydrated["file-l2"]["business_priority"] == 7
+ fields = backend._strict_transfer_page.await_args.kwargs["output_fields"]
+ assert "business_priority" in fields
+ assert "vector" not in fields
+ assert "sparse_vector" not in fields
+ assert "content" not in fields
+
+
+async def test_incremental_hydration_honors_explicit_summary_projection():
+ root = "viking://resources/docs"
+ record = _record(
+ "file-l2",
+ f"{root}/a.py",
+ level=2,
+ abstract="summary",
+ business_priority=7,
+ )
+ backend = _MemoryTransferBackend([record])
+ backend.get_collection_meta = AsyncMock(
+ return_value={
+ "Fields": [
+ {"FieldName": "id"},
+ {"FieldName": "uri"},
+ {"FieldName": "level"},
+ {"FieldName": "abstract"},
+ {"FieldName": "business_priority"},
+ ]
+ }
+ )
+ backend._strict_transfer_page = AsyncMock(return_value=([record], None))
+
+ hydrated = await backend.hydrate_incremental_records(
+ {"file-l2": {"uri": f"{root}/a.py", "level": 2}},
+ ctx=_ctx(),
+ output_fields={"abstract"},
+ )
+
+ assert hydrated["file-l2"] == {
+ "id": "file-l2",
+ "uri": f"{root}/a.py",
+ "level": 2,
+ "abstract": "summary",
+ }
+ fields = backend._strict_transfer_page.await_args.kwargs["output_fields"]
+ assert fields == ["id", "uri", "level", "abstract"]
+ backend.get_collection_meta.assert_not_awaited()
+
+
+async def test_l2_diff_scan_reads_real_local_backend(tmp_path):
+ if not getattr(vectordb_engine, "PersistStore", None):
+ pytest.skip("local persistent vectordb engine is not available in this environment")
+
+ root = "viking://resources/docs"
+ backend = VikingVectorIndexBackend(
+ config=VectorDBBackendConfig(
+ backend="local", name="context", dimension=4, path=str(tmp_path)
+ )
+ )
+ try:
+ assert await backend.create_collection(
+ "context", CollectionSchemas.context_collection("context", 4)
+ )
+ records = [
+ _record(
+ "a",
+ f"{root}/a.py",
+ md5="ma",
+ abstract="A",
+ vector=[0.1, 0.2, 0.3, 0.4],
+ created_at="2026-08-20T00:00:00Z",
+ updated_at="2026-08-20T00:00:00Z",
+ ),
+ _record(
+ "ghost",
+ f"{root}/ghost.py",
+ md5="mg",
+ abstract="G",
+ vector=[0.1, 0.2, 0.3, 0.4],
+ created_at="2026-08-20T00:00:01Z",
+ updated_at="2026-08-20T00:00:01Z",
+ ),
+ _record(
+ "outside",
+ "viking://resources/other.py",
+ vector=[0.1, 0.2, 0.3, 0.4],
+ created_at="2026-08-20T00:00:02Z",
+ updated_at="2026-08-20T00:00:02Z",
+ ),
+ ]
+ await backend.upsert_many(records, ctx=_ctx())
+
+ result = await backend.get_l2_diff_records_under_uri(root, ctx=_ctx(), batch_size=1)
+
+ assert result == {
+ f"{root}/a.py": {"md5": "ma", "abstract": "A"},
+ f"{root}/ghost.py": {"md5": "mg", "abstract": "G"},
+ }
+ finally:
+ await backend.close()
+
+
+async def test_incremental_inventory_and_hydration_read_real_local_backend(tmp_path):
+ if not getattr(vectordb_engine, "PersistStore", None):
+ pytest.skip("local persistent vectordb engine is not available in this environment")
+
+ root = "viking://resources/docs"
+ backend = VikingVectorIndexBackend(
+ config=VectorDBBackendConfig(
+ backend="local", name="context", dimension=4, path=str(tmp_path)
+ )
+ )
+ try:
+ assert await backend.create_collection(
+ "context", CollectionSchemas.context_collection("context", 4)
+ )
+ await backend.upsert_many(
+ [
+ _record(
+ "root-l0",
+ root,
+ level=0,
+ abstract="root abstract",
+ vector=[0.1, 0.2, 0.3, 0.4],
+ created_at="2026-08-20T00:00:00Z",
+ updated_at="2026-08-20T00:00:00Z",
+ ),
+ _record(
+ "root-l1",
+ root,
+ level=1,
+ abstract="root overview",
+ vector=[0.1, 0.2, 0.3, 0.4],
+ created_at="2026-08-20T00:00:01Z",
+ updated_at="2026-08-20T00:00:01Z",
+ ),
+ _record(
+ "a-l2",
+ f"{root}/a.py",
+ md5="ma",
+ abstract="A",
+ vector=[0.1, 0.2, 0.3, 0.4],
+ created_at="2026-08-20T00:00:02Z",
+ updated_at="2026-08-20T00:00:02Z",
+ ),
+ ],
+ ctx=_ctx(),
+ )
+
+ inventory = await backend.get_incremental_inventory_under_uri(
+ root, ctx=_ctx(), batch_size=1
+ )
+ hydrated = await backend.hydrate_incremental_records(inventory, ctx=_ctx())
+
+ assert set(inventory) == {"root-l0", "root-l1", "a-l2"}
+ assert inventory["a-l2"]["md5"] == "ma"
+ assert hydrated["root-l0"]["abstract"] == "root abstract"
+ assert hydrated["a-l2"]["abstract"] == "A"
+ assert "vector" not in hydrated["a-l2"]
+ assert "sparse_vector" not in hydrated["a-l2"]
+ finally:
+ await backend.close()
+
+
+async def test_l2_diff_scan_fails_closed_on_repeated_cursor(monkeypatch):
+ backend = _MemoryTransferBackend([])
+ monkeypatch.setattr(backend, "_strict_transfer_count", AsyncMock(return_value=3))
+ page = [{"uri": "viking://resources/docs/a.py", "md5": "m"}]
+ monkeypatch.setattr(
+ backend,
+ "_strict_transfer_page",
+ AsyncMock(side_effect=[(page, "same"), (page, "same")]),
+ )
+
+ with pytest.raises(RuntimeError, match="duplicate L2 URI|cursor repeated"):
+ await backend.get_l2_diff_records_under_uri(
+ "viking://resources/docs", ctx=_ctx(), batch_size=1
+ )
+
+
+async def test_l2_diff_scan_does_not_ignore_trailing_page_after_count(monkeypatch):
+ backend = _MemoryTransferBackend([])
+ monkeypatch.setattr(backend, "_strict_transfer_count", AsyncMock(return_value=1))
+ monkeypatch.setattr(
+ backend,
+ "_strict_transfer_page",
+ AsyncMock(
+ side_effect=[
+ ([{"uri": "viking://resources/docs/a.py"}], "1"),
+ ([{"uri": "viking://resources/docs/new.py"}], None),
+ ]
+ ),
+ )
+
+ with pytest.raises(RuntimeError, match="count was 1"):
+ await backend.get_l2_diff_records_under_uri(
+ "viking://resources/docs", ctx=_ctx(), batch_size=1
+ )
+
+
+async def test_incremental_inventory_retries_a_transiently_short_cursor(monkeypatch):
+ backend = _MemoryTransferBackend([])
+ root = "viking://resources/docs"
+ monkeypatch.setattr(backend, "_strict_transfer_count", AsyncMock(return_value=2))
+ first = {"id": "root-l0", "uri": root, "level": 0, "md5": ""}
+ second = {"id": "root-l1", "uri": root, "level": 1, "md5": ""}
+ monkeypatch.setattr(
+ backend,
+ "_strict_transfer_page",
+ AsyncMock(
+ side_effect=[
+ ([first], None),
+ ([first, second], None),
+ ]
+ ),
+ )
+
+ records = await backend.get_incremental_inventory_under_uri(root, ctx=_ctx(), batch_size=2)
+
+ assert set(records) == {"root-l0", "root-l1"}
+ assert backend._strict_transfer_count.await_count == 2
+
+
+async def test_incremental_inventory_retries_a_transient_count_undershoot(monkeypatch):
+ backend = _MemoryTransferBackend([])
+ root = "viking://resources/docs"
+ monkeypatch.setattr(
+ backend,
+ "_strict_transfer_count",
+ AsyncMock(side_effect=[1, 2]),
+ )
+ first = {"id": "root-l0", "uri": root, "level": 0, "md5": ""}
+ second = {"id": "root-l1", "uri": root, "level": 1, "md5": ""}
+ monkeypatch.setattr(
+ backend,
+ "_strict_transfer_page",
+ AsyncMock(
+ side_effect=[
+ ([first, second], None),
+ ([first, second], None),
+ ]
+ ),
+ )
+
+ records = await backend.get_incremental_inventory_under_uri(root, ctx=_ctx(), batch_size=2)
+
+ assert set(records) == {"root-l0", "root-l1"}
+ assert backend._strict_transfer_count.await_count == 2
+
+
+ "bad_record",
+ [
+ {"md5": "missing-uri"},
+ {"uri": "viking://resources/other.py", "md5": "outside"},
+ ],
+)
+async def test_l2_diff_scan_fails_closed_on_invalid_scoped_record(monkeypatch, bad_record):
+ backend = _MemoryTransferBackend([])
+ monkeypatch.setattr(backend, "_strict_transfer_count", AsyncMock(return_value=1))
+ monkeypatch.setattr(
+ backend,
+ "_strict_transfer_page",
+ AsyncMock(return_value=([bad_record], None)),
+ )
+
+ with pytest.raises(RuntimeError, match="invalid L2 URI"):
+ await backend.get_l2_diff_records_under_uri("viking://resources/docs", ctx=_ctx())
+
+
@pytest.mark.asyncio
@pytest.mark.parametrize("transfer_method", ["copy_uri_mapping", "update_uri_mapping"])
async def test_uri_mapping_overwrites_exact_target_and_leaves_independent_chunks(
diff --git a/tests/storage/test_volcengine_clients.py b/tests/storage/test_volcengine_clients.py
index d715784f1..ed84f357f 100644
--- a/tests/storage/test_volcengine_clients.py
+++ b/tests/storage/test_volcengine_clients.py
@@ -297,6 +297,7 @@ def test_volcengine_collection_update_data_posts_to_update_endpoint(monkeypatch)
"project": "default",
"collection_name": "context",
"data": [{"id": "doc-1", "name": "updated"}],
+ "ignore_unknown_fields": True,
}
@@ -334,6 +335,7 @@ def test_volcengine_collection_update_data_sanitizes_uri_fields(monkeypatch):
"project": "default",
"collection_name": "context",
"data": [{"id": "doc-1", "uri": "/resources/demo", "parent_uri": "/resources"}],
+ "ignore_unknown_fields": True,
}
@@ -484,6 +486,7 @@ def test_volcengine_api_key_collection_update_data_posts_to_update_endpoint(monk
"project": "default",
"collection_name": "context",
"data": [{"id": "doc-1", "name": "updated"}],
+ "ignore_unknown_fields": True,
}
@@ -523,6 +526,7 @@ def test_volcengine_api_key_collection_update_data_sanitizes_uri_fields(monkeypa
"project": "default",
"collection_name": "context",
"data": [{"id": "doc-1", "uri": "/resources/demo", "parent_uri": "/resources"}],
+ "ignore_unknown_fields": True,
}
@@ -1079,3 +1083,39 @@ def test_http_collection_update_data_posts_to_update_endpoint(monkeypatch):
"collection_name": "context",
"fields": '[{"id": "doc-1", "name": "updated"}]',
}
+
+
+def test_http_adapter_strict_count_propagates_http_failure(monkeypatch):
+ from openviking.storage.vectordb.collection.collection import Collection
+ from openviking.storage.vectordb.collection.http_collection import HttpCollection
+ from openviking.storage.vectordb_adapters.http_adapter import HttpCollectionAdapter
+
+ class _Response:
+ status_code = 503
+ text = "unavailable"
+
+ @staticmethod
+ def raise_for_status():
+ raise requests.HTTPError("503 unavailable")
+
+ monkeypatch.setattr(
+ "openviking.storage.vectordb.collection.http_collection.requests.post",
+ lambda *args, **kwargs: _Response(),
+ )
+ adapter = HttpCollectionAdapter(
+ host="127.0.0.1",
+ port=1933,
+ project_name="default",
+ collection_name="context",
+ index_name="default",
+ )
+ adapter._collection = Collection(
+ HttpCollection(
+ ip="127.0.0.1",
+ port=1933,
+ meta_data={"ProjectName": "default", "CollectionName": "context"},
+ )
+ )
+
+ with pytest.raises(requests.HTTPError, match="503 unavailable"):
+ adapter.strict_count()
diff --git a/tests/telemetry/test_execution.py b/tests/telemetry/test_execution.py
index 6b4a11070..79ea9cdc2 100644
--- a/tests/telemetry/test_execution.py
+++ b/tests/telemetry/test_execution.py
@@ -86,14 +86,21 @@ def test_operation_telemetry_summary_includes_resource_breakdown():
from openviking.telemetry.operation import OperationTelemetry
telemetry = OperationTelemetry(operation="resources.add_resource", enabled=True)
- telemetry.set("resource.request.duration_ms", 152.3)
- telemetry.set("resource.process.duration_ms", 101.7)
- telemetry.set("resource.parse.duration_ms", 38.1)
- telemetry.set("resource.parse.warnings_count", 1)
- telemetry.set("resource.finalize.duration_ms", 22.4)
- telemetry.set("resource.summarize.duration_ms", 31.8)
- telemetry.set("resource.wait.duration_ms", 46.9)
- telemetry.set("resource.watch.duration_ms", 0.8)
+ telemetry.set("resource.total.duration_ms", 152.3)
+ telemetry.set("resource.source_execute.duration_ms", 101.7)
+ telemetry.set("resource.source_prepare.duration_ms", 9.1)
+ telemetry.set("resource.parse_artifact.duration_ms", 38.1)
+ telemetry.set("resource.target_resolve.duration_ms", 22.4)
+ telemetry.set("resource.update_plan.duration_ms", 31.8)
+ telemetry.set("resource.update_plan.artifact_inventory.duration_ms", 4.0)
+ telemetry.set("resource.update_plan.rnfv_snapshot.duration_ms", 11.0)
+ telemetry.set("resource.update_plan.diff_and_compile.duration_ms", 16.8)
+ telemetry.set("resource.content_commit.duration_ms", 12.4)
+ telemetry.set("resource.derived_enqueue.duration_ms", 4.6)
+ telemetry.set("queue.semantic.queue_wait.duration_ms", 46.9)
+ telemetry.set("queue.semantic.execute.duration_ms", 70.0)
+ telemetry.set("queue.embedding.queue_wait.duration_ms", 0.8)
+ telemetry.set("queue.embedding.execute.duration_ms", 10.0)
telemetry.set("resource.flags.wait", True)
telemetry.set("resource.flags.build_index", True)
telemetry.set("resource.flags.summarize", False)
@@ -102,15 +109,27 @@ def test_operation_telemetry_summary_includes_resource_breakdown():
summary = telemetry.finish().summary
assert summary["resource"] == {
- "request": {"duration_ms": 152.3},
- "process": {
- "duration_ms": 101.7,
- "parse": {"duration_ms": 38.1, "warnings_count": 1},
- "finalize": {"duration_ms": 22.4},
- "summarize": {"duration_ms": 31.8},
+ "total": {"duration_ms": 152.3},
+ "source_execute": {"duration_ms": 101.7},
+ "source_prepare": {"duration_ms": 9.1},
+ "parse_artifact": {"duration_ms": 38.1},
+ "target_resolve": {"duration_ms": 22.4},
+ "update_plan": {
+ "duration_ms": 31.8,
+ "artifact_inventory": {"duration_ms": 4.0},
+ "rnfv_snapshot": {"duration_ms": 11.0},
+ "diff_and_compile": {"duration_ms": 16.8},
+ },
+ "content_commit": {"duration_ms": 12.4},
+ "derived_enqueue": {"duration_ms": 4.6},
+ "semantic": {
+ "queue_wait": {"duration_ms": 46.9},
+ "execute": {"duration_ms": 70.0},
+ },
+ "embedding": {
+ "queue_wait": {"duration_ms": 0.8},
+ "execute": {"duration_ms": 10.0},
},
- "wait": {"duration_ms": 46.9},
- "watch": {"duration_ms": 0.8},
"flags": {
"wait": True,
"build_index": True,
@@ -153,9 +172,7 @@ def test_operation_telemetry_summary_omits_zero_valued_fields():
telemetry.set("semantic_nodes.done", 8)
telemetry.set("semantic_nodes.pending", 1)
telemetry.set("semantic_nodes.running", 0)
- telemetry.set("resource.process.duration_ms", 12.3)
- telemetry.set("resource.parse.duration_ms", 0.0)
- telemetry.set("resource.parse.warnings_count", 0)
+ telemetry.set("resource.source_execute.duration_ms", 12.3)
telemetry.set("resource.flags.wait", False)
telemetry.set("resource.flags.build_index", True)
@@ -166,7 +183,7 @@ def test_operation_telemetry_summary_omits_zero_valued_fields():
assert summary["queue"]["embedding"] == {"processed": 4}
assert "running" not in summary["semantic_nodes"]
assert summary["resource"] == {
- "process": {"duration_ms": 12.3},
+ "source_execute": {"duration_ms": 12.3},
"flags": {"wait": False, "build_index": True, "summarize": False, "watch_enabled": False},
}
diff --git a/tests/telemetry/test_request_wait_tracker.py b/tests/telemetry/test_request_wait_tracker.py
index 122788fd2..01efd2052 100644
--- a/tests/telemetry/test_request_wait_tracker.py
+++ b/tests/telemetry/test_request_wait_tracker.py
@@ -71,6 +71,21 @@ def test_request_wait_tracker_counts_contexts_indexed_by_request():
assert tracker.get_embedding_context_count(telemetry_id) == 0
+def test_request_wait_tracker_accumulates_queue_wait_and_execution_durations():
+ tracker = RequestWaitTracker()
+ telemetry_id = "tm_queue_timing"
+
+ tracker.register_request(telemetry_id)
+ tracker.record_semantic_timing(telemetry_id, queue_wait_ms=12.5, execute_ms=30.0)
+ tracker.record_semantic_timing(telemetry_id, queue_wait_ms=7.5, execute_ms=10.0)
+ tracker.record_embedding_timing(telemetry_id, queue_wait_ms=5.0, execute_ms=8.0)
+
+ assert tracker.get_queue_timing(telemetry_id) == {
+ "semantic": {"queue_wait_ms": 20.0, "execute_ms": 40.0},
+ "embedding": {"queue_wait_ms": 5.0, "execute_ms": 8.0},
+ }
+
+
async def test_wait_for_request_timeout_keeps_existing_error():
tracker = RequestWaitTracker()
telemetry_id = "tm_wait_timeout"
diff --git a/tests/telemetry/test_resource_summary.py b/tests/telemetry/test_resource_summary.py
index e12127983..076c644fb 100644
--- a/tests/telemetry/test_resource_summary.py
+++ b/tests/telemetry/test_resource_summary.py
@@ -1,12 +1,12 @@
-
from openviking.telemetry.operation import OperationTelemetry
-def test_record_resource_queue_metrics_collects_queue_and_dag_stats(monkeypatch):
+def test_record_resource_queue_metrics_collects_queue_and_tree_stats(monkeypatch):
from openviking.telemetry.resource_summary import record_resource_queue_metrics
telemetry = OperationTelemetry(operation="resources.add_resource", enabled=True)
telemetry_id = telemetry.telemetry_id
+
class _SemanticStats:
processed = 7
requeue_count = 0
@@ -17,7 +17,15 @@ def test_record_resource_queue_metrics_collects_queue_and_dag_stats(monkeypatch)
requeue_count = 0
error_count = 1
- class _DagStats:
+ class _Timing:
+ @staticmethod
+ def get_queue_timing(_tid):
+ return {
+ "semantic": {"queue_wait_ms": 12.5, "execute_ms": 30.0},
+ "embedding": {"queue_wait_ms": 8.0, "execute_ms": 20.0},
+ }
+
+ class _SemanticTreeStats:
total_nodes = 9
done_nodes = 8
pending_nodes = 1
@@ -32,8 +40,12 @@ def test_record_resource_queue_metrics_collects_queue_and_dag_stats(monkeypatch)
lambda _tid: _EmbeddingStats(),
)
monkeypatch.setattr(
- "openviking.telemetry.resource_summary._consume_semantic_dag_stats",
- lambda _tid, _uri: _DagStats(),
+ "openviking.telemetry.resource_summary._consume_semantic_tree_stats",
+ lambda _tid, _uri: _SemanticTreeStats(),
+ )
+ monkeypatch.setattr(
+ "openviking.telemetry.resource_summary.get_request_wait_tracker",
+ lambda: _Timing(),
)
record_resource_queue_metrics(
@@ -47,6 +59,10 @@ def test_record_resource_queue_metrics_collects_queue_and_dag_stats(monkeypatch)
assert summary["queue"]["semantic"]["error_count"] == 2
assert summary["queue"]["embedding"]["processed"] == 11
assert summary["queue"]["embedding"]["error_count"] == 1
+ assert summary["queue"]["semantic"]["queue_wait"]["duration_ms"] == 12.5
+ assert summary["queue"]["semantic"]["execute"]["duration_ms"] == 30.0
+ assert summary["queue"]["embedding"]["queue_wait"]["duration_ms"] == 8.0
+ assert summary["queue"]["embedding"]["execute"]["duration_ms"] == 20.0
assert summary["semantic_nodes"]["total"] == 9
assert summary["semantic_nodes"]["done"] == 8
assert summary["semantic_nodes"]["pending"] == 1
diff --git a/tests/test_config_loader.py b/tests/test_config_loader.py
index 7f0717d68..132ad93c3 100644
--- a/tests/test_config_loader.py
+++ b/tests/test_config_loader.py
@@ -130,6 +130,7 @@ def test_runtime_concurrency_uses_scope_specific_defaults():
assert config.queue_workers.external_parse.max_concurrent == 4
assert config.queue_workers.add_resource.max_concurrent == 4
+ assert config.queue_workers.add_resource.file_operation_concurrency == 16
assert config.queue_workers.add_resource.file_vectorization_concurrency == 8
assert config.queue_workers.session_commit.max_concurrent == 8
assert config.queue_workers.external_task.max_concurrent == 10
@@ -150,6 +151,7 @@ def test_runtime_concurrency_accepts_separate_values():
"external_parse": {"max_concurrent": 9},
"add_resource": {
"max_concurrent": 7,
+ "file_operation_concurrency": 20,
"file_vectorization_concurrency": 12,
},
"session_commit": {"max_concurrent": 50},
@@ -161,6 +163,7 @@ def test_runtime_concurrency_accepts_separate_values():
assert config.queue_workers.external_parse.max_concurrent == 9
assert config.queue_workers.add_resource.max_concurrent == 7
+ assert config.queue_workers.add_resource.file_operation_concurrency == 20
assert config.queue_workers.add_resource.file_vectorization_concurrency == 12
assert config.queue_workers.session_commit.max_concurrent == 50
assert config.queue_workers.external_task.max_concurrent == 11
diff --git a/tests/test_telemetry_runtime.py b/tests/test_telemetry_runtime.py
index b159c2c5d..eb798f11b 100644
--- a/tests/test_telemetry_runtime.py
+++ b/tests/test_telemetry_runtime.py
@@ -21,7 +21,7 @@ from openviking.observability.context import (
reset_root_observability_context,
)
from openviking.storage.collection_schemas import TextEmbeddingHandler
-from openviking.storage.queuefs.semantic_dag import DagStats
+from openviking.storage.queuefs.semantic_executor import SemanticTreeStats
from openviking.storage.queuefs.semantic_msg import SemanticMsg
from openviking.storage.queuefs.semantic_processor import SemanticProcessor
from openviking.telemetry import (
@@ -731,10 +731,13 @@ async def test_semantic_processor_binds_registered_operation_telemetry(monkeypat
processor = SemanticProcessor()
class FakeVikingFS:
+ async def exists(self, uri, ctx=None):
+ return True
+
async def ls(self, uri, ctx=None):
return []
- class _FakeDagExecutor:
+ class _FakeTreeExecutor:
def __init__(self, **kwargs):
pass
@@ -743,15 +746,15 @@ async def test_semantic_processor_binds_registered_operation_telemetry(monkeypat
get_current_telemetry().record_token_usage("llm", 11, 7)
def get_stats(self):
- return DagStats()
+ return SemanticTreeStats()
monkeypatch.setattr(
"openviking.storage.queuefs.semantic_processor.get_viking_fs",
lambda: FakeVikingFS(),
)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor",
- lambda **kwargs: _FakeDagExecutor(**kwargs),
+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor",
+ lambda **kwargs: _FakeTreeExecutor(**kwargs),
)
try:
@@ -779,10 +782,13 @@ async def test_semantic_processor_binds_metric_account_context(monkeypatch):
ran = {"value": False}
class FakeVikingFS:
+ async def exists(self, uri, ctx=None):
+ return True
+
async def ls(self, uri, ctx=None):
return []
- class _FakeDagExecutor:
+ class _FakeTreeExecutor:
def __init__(self, **kwargs):
pass
@@ -793,15 +799,15 @@ async def test_semantic_processor_binds_metric_account_context(monkeypatch):
assert root_context.account_id == "acct-semantic"
def get_stats(self):
- return DagStats()
+ return SemanticTreeStats()
monkeypatch.setattr(
"openviking.storage.queuefs.semantic_processor.get_viking_fs",
lambda: FakeVikingFS(),
)
monkeypatch.setattr(
- "openviking.storage.queuefs.semantic_processor.SemanticDagExecutor",
- lambda **kwargs: _FakeDagExecutor(**kwargs),
+ "openviking.storage.queuefs.semantic_processor.SemanticTreeExecutor",
+ lambda **kwargs: _FakeTreeExecutor(**kwargs),
)
await processor.on_dequeue(
diff --git a/tests/test_upload_utils.py b/tests/test_upload_utils.py
index 25b088fb5..373ac8403 100644
--- a/tests/test_upload_utils.py
+++ b/tests/test_upload_utils.py
@@ -20,46 +20,6 @@ from openviking.utils.path_safety import sanitize_relative_viking_path
# ---------------------------------------------------------------------------
-class FakeAGFS:
- """Minimal AGFS mock that stores files and directories by path key."""
-
- def __init__(self, storage: Dict[str, bytes]) -> None:
- self._storage = storage
- self.dirs: List[str] = []
-
- def mkdir(self, path: str) -> None:
- self.dirs.append(path)
-
- def write(self, path: str, content: bytes) -> None:
- self._storage[path] = content
-
-
-class FakeVikingFS:
- """Minimal VikingFS mock for testing upload functions."""
-
- def __init__(self) -> None:
- self.files: Dict[str, bytes] = {}
- self.dirs: List[str] = []
- self.write_file_bytes_calls: List[str] = []
- self.agfs = FakeAGFS(self.files)
-
- def _uri_to_path(self, uri: str) -> str:
- # Simplified: use the URI itself as the storage key so test assertions work.
- return uri
-
- async def write_file_bytes(self, uri: str, content: bytes) -> None:
- self.write_file_bytes_calls.append(uri)
- self.files[uri] = content
-
- async def mkdir(self, uri: str, exist_ok: bool = False) -> None:
- self.dirs.append(uri)
-
-
-def viking_fs() -> FakeVikingFS:
- return FakeVikingFS()
-
-
@pytest.fixture
def tmp_dir(tmp_path: Path) -> Path:
"""Create a temporary directory with sample files for testing."""
@@ -257,96 +217,106 @@ class TestShouldSkipFile:
# ---------------------------------------------------------------------------
-class TestUploadDirectory:
- @pytest.mark.asyncio
- async def test_basic_upload(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:
- count, warnings = await upload_directory(tmp_dir, "viking://temp/test", viking_fs)
+class _RecordingStore:
+ """Minimal parse output store recording writes/mkdirs by relative path."""
- # Should upload: hello.py, readme.md, config.yaml, src/main.go
- # Should skip: .hidden, image.png, empty.txt, __pycache__/mod.pyc
- assert count == 4
- assert "viking://temp/test/hello.py" in viking_fs.files
- assert "viking://temp/test/readme.md" in viking_fs.files
- assert "viking://temp/test/config.yaml" in viking_fs.files
- assert "viking://temp/test/src/main.go" in viking_fs.files
- assert "viking://temp/test/hello.py" in viking_fs.write_file_bytes_calls
+ def __init__(self) -> None:
+ self.writes: Dict[str, bytes] = {}
+ self.dirs: List[str] = []
- @pytest.mark.asyncio
- async def test_uses_vikingfs_write_api_for_file_content(self, tmp_path: Path) -> None:
- class GuardedAGFS(FakeAGFS):
- def write(self, path: str, content: bytes) -> None:
- raise AssertionError("upload_directory must not bypass VikingFS writes")
+ async def mkdir(self, ref, rel_path: str = "") -> None:
+ self.dirs.append(rel_path)
- class GuardedVikingFS(FakeVikingFS):
- def __init__(self) -> None:
- super().__init__()
- self.agfs = GuardedAGFS(self.files)
+ async def write_bytes(self, ref, rel_path: str, content: bytes) -> None:
+ self.writes[rel_path] = content
- (tmp_path / "hello.py").write_text("print('hello')", encoding="utf-8")
- viking_fs = GuardedVikingFS()
+ async def read_bytes(self, ref, rel_path: str) -> bytes:
+ return self.writes[rel_path]
- count, warnings = await upload_directory(tmp_path, "viking://temp/guarded", viking_fs)
- assert count == 1
- assert warnings == []
- assert viking_fs.files["viking://temp/guarded/hello.py"] == b"print('hello')"
+class TestUploadDirectory:
+ @pytest.mark.asyncio
+ async def test_basic_upload(self, tmp_dir: Path) -> None:
+ store = _RecordingStore()
+ count, warnings = await upload_directory(
+ tmp_dir, "repo", store=store, artifact_ref=object()
+ )
+
+ # Should upload: hello.py, readme.md, config.yaml, src/main.go
+ # Should skip: .hidden, image.png, empty.txt, __pycache__/mod.pyc
+ assert count == 4
+ assert "repo/hello.py" in store.writes
+ assert "repo/readme.md" in store.writes
+ assert "repo/config.yaml" in store.writes
+ assert "repo/src/main.go" in store.writes
@pytest.mark.asyncio
- async def test_skips_hidden_files(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:
- await upload_directory(tmp_dir, "viking://temp/test", viking_fs)
- assert all(".hidden" not in uri for uri in viking_fs.files)
+ async def test_skips_hidden_files(self, tmp_dir: Path) -> None:
+ store = _RecordingStore()
+ await upload_directory(tmp_dir, "repo", store=store, artifact_ref=object())
+ assert all(".hidden" not in rel for rel in store.writes)
@pytest.mark.asyncio
- async def test_skips_ignored_dirs(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:
- await upload_directory(tmp_dir, "viking://temp/test", viking_fs)
- assert all("__pycache__" not in uri for uri in viking_fs.files)
+ async def test_skips_ignored_dirs(self, tmp_dir: Path) -> None:
+ store = _RecordingStore()
+ await upload_directory(tmp_dir, "repo", store=store, artifact_ref=object())
+ assert all("__pycache__" not in rel for rel in store.writes)
@pytest.mark.asyncio
- async def test_skips_ignored_extensions(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:
- await upload_directory(tmp_dir, "viking://temp/test", viking_fs)
- assert all(".png" not in uri for uri in viking_fs.files)
+ async def test_skips_ignored_extensions(self, tmp_dir: Path) -> None:
+ store = _RecordingStore()
+ await upload_directory(tmp_dir, "repo", store=store, artifact_ref=object())
+ assert all(".png" not in rel for rel in store.writes)
@pytest.mark.asyncio
- async def test_skips_empty_files(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:
+ async def test_skips_empty_files(self, tmp_dir: Path) -> None:
(tmp_dir / "application.properties").write_bytes(b"\n")
(tmp_dir / "blank.txt").write_bytes(b" \t\r\n")
- await upload_directory(tmp_dir, "viking://temp/test", viking_fs)
- assert all("empty.txt" not in uri for uri in viking_fs.files)
- assert "viking://temp/test/application.properties" not in viking_fs.files
- assert "viking://temp/test/blank.txt" not in viking_fs.files
+ store = _RecordingStore()
+ await upload_directory(tmp_dir, "repo", store=store, artifact_ref=object())
+ assert all("empty.txt" not in rel for rel in store.writes)
+ assert "repo/application.properties" not in store.writes
+ assert "repo/blank.txt" not in store.writes
@pytest.mark.asyncio
- async def test_creates_root_dir(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:
- await upload_directory(tmp_dir, "viking://temp/root", viking_fs)
- assert "viking://temp/root" in viking_fs.dirs
+ async def test_precreates_subdirectories(self, tmp_dir: Path) -> None:
+ store = _RecordingStore()
+ await upload_directory(tmp_dir, "repo", store=store, artifact_ref=object())
+ # The nested src/ dir is pre-created before its files are written.
+ assert "repo/src" in store.dirs
@pytest.mark.asyncio
- async def test_custom_ignore_dirs(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:
+ async def test_custom_ignore_dirs(self, tmp_dir: Path) -> None:
+ store = _RecordingStore()
count, _ = await upload_directory(
- tmp_dir, "viking://temp/test", viking_fs, ignore_dirs={"src"}
+ tmp_dir, "repo", store=store, artifact_ref=object(), ignore_dirs={"src"}
)
- assert all("src/" not in uri for uri in viking_fs.files)
+ assert all("src/" not in rel for rel in store.writes)
# Positive assertion: non-ignored files should still be uploaded
assert count > 0
- assert "viking://temp/test/hello.py" in viking_fs.files
+ assert "repo/hello.py" in store.writes
@pytest.mark.asyncio
- async def test_custom_max_file_size(self, tmp_dir: Path, viking_fs: FakeVikingFS) -> None:
- count, _ = await upload_directory(tmp_dir, "viking://temp/test", viking_fs, max_file_size=5)
+ async def test_custom_max_file_size(self, tmp_dir: Path) -> None:
+ store = _RecordingStore()
+ count, _ = await upload_directory(
+ tmp_dir, "repo", store=store, artifact_ref=object(), max_file_size=5
+ )
# Most files are > 5 bytes, so fewer uploads
assert count < 4
@pytest.mark.asyncio
- async def test_respects_gitignore(self, tmp_path: Path, viking_fs: FakeVikingFS) -> None:
+ async def test_respects_gitignore(self, tmp_path: Path) -> None:
(tmp_path / ".gitignore").write_text("*.tmp\n", encoding="utf-8")
(tmp_path / "keep.txt").write_text("ok", encoding="utf-8")
(tmp_path / "skip.tmp").write_text("no", encoding="utf-8")
- count, _ = await upload_directory(tmp_path, "viking://temp/gi", viking_fs)
+ store = _RecordingStore()
+ count, _ = await upload_directory(tmp_path, "gi", store=store, artifact_ref=object())
assert count == 1
- assert "viking://temp/gi/keep.txt" in viking_fs.files
- assert "viking://temp/gi/skip.tmp" not in viking_fs.files
+ assert "gi/keep.txt" in store.writes
+ assert "gi/skip.tmp" not in store.writes
# ---------------------------------------------------------------------------
@@ -429,36 +399,55 @@ class TestSanitizeRelPath:
# ---------------------------------------------------------------------------
-# upload_directory (additional edge cases)
+# upload_directory (failure handling + md5 manifest)
# ---------------------------------------------------------------------------
class TestUploadDirectoryEdgeCases:
@pytest.mark.asyncio
- async def test_write_failure_produces_warning(self, tmp_path: Path) -> None:
- class FailingAGFS:
- def mkdir(self, path: str) -> None:
- pass
+ async def test_writes_content_md5_manifest(self, tmp_dir: Path) -> None:
+ import hashlib
+ import json
- class FailingWriteFS:
- agfs = FailingAGFS()
+ from openviking.parse.parsers.upload_utils import ARTIFACT_MANIFEST_NAME
- def _uri_to_path(self, uri: str) -> str:
- return uri
+ store = _RecordingStore()
- async def mkdir(self, uri: str, exist_ok: bool = False) -> None:
+ await upload_directory(tmp_dir, "repo", store=store, artifact_ref=object())
+
+ # A manifest of final-byte md5 keyed by artifact-relative path is written
+ # so the incremental diff can compare fingerprints without re-reading.
+ assert ARTIFACT_MANIFEST_NAME in store.writes
+ manifest = json.loads(store.writes[ARTIFACT_MANIFEST_NAME].decode("utf-8"))
+ assert manifest["repo/hello.py"] == hashlib.md5(b"print('hello')").hexdigest()
+ # Every uploaded business file has an md5; the manifest itself is excluded.
+ assert "repo/src/main.go" in manifest
+ assert ARTIFACT_MANIFEST_NAME not in manifest
+
+ @pytest.mark.asyncio
+ async def test_store_write_failure_produces_warning_and_skips_manifest(
+ self, tmp_path: Path
+ ) -> None:
+ from openviking.parse.parsers.upload_utils import ARTIFACT_MANIFEST_NAME
+
+ class FailingStore:
+ async def mkdir(self, ref, rel_path: str = "") -> None:
pass
- async def write_file_bytes(self, uri: str, content: bytes) -> None:
- raise IOError("write error")
+ async def write_bytes(self, ref, rel_path: str, content: bytes) -> None:
+ if rel_path == ARTIFACT_MANIFEST_NAME:
+ raise AssertionError("manifest must not be written after a failed upload")
+ raise IOError("store write error")
(tmp_path / "ok.py").write_text("print(1)", encoding="utf-8")
- count, warnings = await upload_directory(tmp_path, "viking://temp/fail", FailingWriteFS())
+ count, warnings = await upload_directory(
+ tmp_path, "repo", store=FailingStore(), artifact_ref=object()
+ )
assert count == 0
assert len(warnings) == 1
- assert "write error" in warnings[0]
+ assert "store write error" in warnings[0]
# ---------------------------------------------------------------------------
diff --git a/tests/unit/test_context.py b/tests/unit/test_context.py
index 732ad1a53..1e950dea1 100644
--- a/tests/unit/test_context.py
+++ b/tests/unit/test_context.py
@@ -12,6 +12,7 @@ from openviking.core.context import (
ResourceContentType,
Vectorize,
)
+from openviking.utils.content_hash import content_md5
from openviking_cli.session.user_id import UserIdentifier
@@ -273,6 +274,7 @@ class TestContextMethods:
assert ctx.get_vectorization_text() == "Test abstract"
+
class TestContextToDict:
"""Test Context.to_dict."""
@@ -543,3 +545,22 @@ class TestContextEdgeCases:
ctx = Context(uri="viking://test/path%20with%20spaces/")
assert ctx.uri == "viking://test/path%20with%20spaces/"
+
+ def test_md5_roundtrip_preserves_known_fingerprint(self):
+ md5 = content_md5(b"print(1)")
+ ctx = Context(uri="viking://resources/x/a.py", is_leaf=True, md5=md5)
+
+ assert ctx.to_dict()["md5"] == md5
+ assert Context.from_dict(ctx.to_dict()).md5 == md5
+
+ def test_missing_md5_is_omitted_from_dict(self):
+ ctx = Context(uri="viking://resources/x/b.py", is_leaf=True)
+
+ assert "md5" not in ctx.to_dict()
+ assert Context.from_dict({"uri": "viking://resources/x/b.py"}).md5 == ""
+
+
+def test_content_md5_is_stable_hex_digest():
+ assert content_md5(b"print(1)") == content_md5(b"print(1)")
+ assert content_md5(b"a") != content_md5(b"b")
+ assert len(content_md5(b"x")) == 32
diff --git a/tests/unit/test_vectorize_file_strategy.py b/tests/unit/test_vectorize_file_strategy.py
index c3003323a..5d4f598b9 100644
--- a/tests/unit/test_vectorize_file_strategy.py
+++ b/tests/unit/test_vectorize_file_strategy.py
@@ -10,6 +10,7 @@ from openviking.parse.parsers.media.utils import (
MPEG_TS_PACKET_SIZE,
MPEG_TS_PROBE_BYTES,
)
+from openviking.storage.index_action import FieldPatch
from openviking.utils import embedding_utils
from openviking.utils.ingest_options import IngestOptions
from openviking_cli.utils.config.parser_config import ImageConfig
@@ -200,6 +201,89 @@ async def test_vectorize_file_uses_summary_first(monkeypatch, text_source, summa
assert "content" not in queue.items[0].context_data
+async def test_vectorize_file_threads_supplied_md5_without_reading_bytes(monkeypatch):
+ # The upload site already holds the final bytes; vectorize_file must accept
+ # the fingerprint via file_md5 and never read the file back to compute it.
+ queue = DummyQueue()
+ fs = DummyFS("deployment guide")
+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))
+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: fs)
+ monkeypatch.setattr(
+ embedding_utils,
+ "get_openviking_config",
+ lambda: types.SimpleNamespace(
+ embedding=types.SimpleNamespace(text_source="summary_first", max_input_tokens=1000)
+ ),
+ )
+
+ await embedding_utils.vectorize_file(
+ file_path="viking://user/default/resources/demo.md",
+ summary_dict={"name": "demo.md", "summary": "deployment summary"},
+ parent_uri="viking://user/default/resources",
+ ctx=DummyReq(),
+ file_md5="a" * 32,
+ )
+
+ assert len(queue.items) == 1
+ assert queue.items[0].context_data["md5"] == "a" * 32
+ # No file read was issued just to fingerprint the content.
+ assert fs.read_file_bytes_calls == 0
+
+
+async def test_vectorize_file_uses_supplied_content_without_reading_file(monkeypatch):
+ queue = DummyQueue()
+ fs = DummyFS("remote content")
+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))
+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: fs)
+ monkeypatch.setattr(
+ embedding_utils,
+ "get_openviking_config",
+ lambda: types.SimpleNamespace(
+ embedding=types.SimpleNamespace(text_source="content_only", max_input_tokens=1000)
+ ),
+ )
+
+ await embedding_utils.vectorize_file(
+ file_path="viking://user/default/resources/a.py",
+ summary_dict={"name": "a.py", "summary": ""},
+ parent_uri="viking://user/default/resources",
+ ctx=DummyReq(),
+ file_content=b"print('local')",
+ )
+
+ assert queue.items[0].message == "print('local')"
+ assert fs.read_file_calls == 0
+
+
+async def test_vectorize_file_omits_md5_when_not_supplied(monkeypatch):
+ queue = DummyQueue()
+ fs = DummyFS("deployment guide")
+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))
+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: fs)
+ monkeypatch.setattr(
+ embedding_utils,
+ "get_openviking_config",
+ lambda: types.SimpleNamespace(
+ embedding=types.SimpleNamespace(text_source="summary_first", max_input_tokens=1000)
+ ),
+ )
+
+ await embedding_utils.vectorize_file(
+ file_path="viking://user/default/resources/demo.md",
+ summary_dict={"name": "demo.md", "summary": "deployment summary"},
+ parent_uri="viking://user/default/resources",
+ ctx=DummyReq(),
+ )
+
+ assert len(queue.items) == 1
+ # Unknown fingerprint must not be emitted, so a partial update never clears it.
+ assert "md5" not in queue.items[0].context_data
+ assert fs.read_file_bytes_calls == 0
+
+
@pytest.mark.asyncio
async def test_vectorize_image_downsamples_large_embedding_input(monkeypatch):
queue = DummyQueue()
@@ -569,6 +653,26 @@ async def test_vectorize_directory_meta_appends_search_tags_by_level(monkeypatch
assert queue.items[1].context_data["_upsert_options"] == {"search_tag_mode": "append"}
+async def test_vectorize_directory_meta_applies_merge_per_level(monkeypatch):
+ queue = DummyQueue()
+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))
+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: DummyFS("ignored"))
+
+ await embedding_utils.vectorize_directory_meta(
+ uri="viking://user/default/resources/demo",
+ abstract="demo abstract",
+ overview="demo overview",
+ ctx=DummyReq(),
+ actions={0: "merge", 1: "upsert"},
+ )
+
+ assert len(queue.items) == 2
+ assert "_upsert_options" not in queue.items[0].context_data
+ assert queue.items[0].action.value == "merge"
+ assert queue.items[1].action.value == "upsert"
+
+
@pytest.mark.asyncio
async def test_vectorize_directory_meta_l1_abstract_is_overview(monkeypatch):
"""L1 records must carry the overview in the abstract scalar so Rerank
@@ -1107,3 +1211,138 @@ async def test_skill_directory_body_frontmatter_is_not_parsed_twice(monkeypatch)
overview_msg = next(item for item in queue.items if item.context_data["level"] == 1)
assert overview_msg.context_data["abstract"] == body
assert body in overview_msg.message
+
+
+async def test_full_upsert_uses_resolved_search_tags_from_plan(monkeypatch):
+ queue = DummyQueue()
+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))
+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: DummyFS("body"))
+ monkeypatch.setattr(
+ embedding_utils,
+ "get_openviking_config",
+ lambda: types.SimpleNamespace(
+ embedding=types.SimpleNamespace(text_source="summary_only", max_input_tokens=1000)
+ ),
+ )
+
+ await embedding_utils.vectorize_file(
+ file_path="viking://resources/repo/a.py",
+ summary_dict={"name": "a.py", "summary": "summary"},
+ parent_uri="viking://resources/repo",
+ ctx=DummyReq(),
+ scalar_override={"search_tags": ["team=old", "lang=python", "team=new", "owner=alice"]},
+ ingest_options=IngestOptions.from_search_tags(["team=new", "owner=alice"], mode="append"),
+ action="upsert",
+ )
+
+ msg = queue.items[0]
+ assert msg.context_data["search_tags"] == [
+ "team=old",
+ "lang=python",
+ "team=new",
+ "owner=alice",
+ ]
+ assert msg.action.value == "upsert"
+ assert msg.context_data["_upsert_options"] == {"search_tag_mode": "append"}
+
+
+async def test_context_vectorization_defaults_to_merge_for_legacy_producers(monkeypatch):
+ queue = DummyQueue()
+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))
+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: DummyFS("body"))
+ monkeypatch.setattr(
+ embedding_utils,
+ "get_openviking_config",
+ lambda: types.SimpleNamespace(
+ embedding=types.SimpleNamespace(text_source="summary_only", max_input_tokens=1000)
+ ),
+ )
+
+ await embedding_utils.vectorize_file(
+ file_path="viking://resources/repo/a.py",
+ summary_dict={"name": "a.py", "summary": "summary"},
+ parent_uri="viking://resources/repo",
+ ctx=DummyReq(),
+ )
+ await embedding_utils.vectorize_directory_meta(
+ uri="viking://resources/repo",
+ abstract="abstract",
+ overview="overview",
+ ctx=DummyReq(),
+ )
+
+ assert [message.action.value for message in queue.items] == ["merge", "merge", "merge"]
+ assert all(message.field_patch is None for message in queue.items)
+
+
+async def test_planned_merge_does_not_copy_context_into_patch_seed(monkeypatch):
+ queue = DummyQueue()
+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))
+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: DummyFS("body"))
+ monkeypatch.setattr(
+ embedding_utils,
+ "get_openviking_config",
+ lambda: types.SimpleNamespace(
+ embedding=types.SimpleNamespace(text_source="summary_only", max_input_tokens=1000)
+ ),
+ )
+
+ await embedding_utils.vectorize_file(
+ file_path="viking://resources/repo/a.py",
+ summary_dict={"name": "a.py", "summary": "large summary"},
+ parent_uri="viking://resources/repo",
+ ctx=DummyReq(),
+ action="merge",
+ field_patch=FieldPatch(
+ {"search_tags": ["scope=new"]},
+ {"search_tags": "append"},
+ ),
+ )
+
+ msg = queue.items[0]
+ assert msg.context_data["abstract"] == "large summary"
+ assert msg.field_patch == FieldPatch(
+ {"search_tags": ["scope=new"]},
+ {"search_tags": "append"},
+ )
+
+
+def test_semantic_processor_uses_merge_for_unplanned_vectorization():
+ import inspect
+
+ from openviking.storage.queuefs.semantic_processor import SemanticProcessor
+
+ assert (
+ inspect.signature(SemanticProcessor._vectorize_single_file).parameters["action"].default
+ == "merge"
+ )
+
+
+async def test_full_upsert_carries_existing_record_id_as_internal_override(monkeypatch):
+ queue = DummyQueue()
+ monkeypatch.setattr(embedding_utils, "get_queue_manager", lambda: DummyQueueManager(queue))
+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: DummyFS("body"))
+ monkeypatch.setattr(
+ embedding_utils,
+ "get_openviking_config",
+ lambda: types.SimpleNamespace(
+ embedding=types.SimpleNamespace(text_source="summary_only", max_input_tokens=1000)
+ ),
+ )
+
+ await embedding_utils.vectorize_file(
+ file_path="viking://resources/repo/a.py",
+ summary_dict={"name": "a.py", "summary": "summary"},
+ parent_uri="viking://resources/repo",
+ ctx=DummyReq(),
+ scalar_override={"_record_id": "id-from-vector-db"},
+ action="upsert",
+ )
+
+ msg = queue.items[0]
+ assert msg.context_data["_upsert_record_id"] == "id-from-vector-db"
+ assert "_record_id" not in msg.context_data
diff --git a/tests/utils/test_local_artifact_persist.py b/tests/utils/test_local_artifact_persist.py
new file mode 100644
index 000000000..a3f946496
--- /dev/null
+++ b/tests/utils/test_local_artifact_persist.py
@@ -0,0 +1,220 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""End-to-end-ish test for the local-artifact persist branch.
+
+Rather than faking the whole process_resource pipeline (tree/stat/pathlock), this
+exercises the actual production method resource_processor uses to land a local
+parse artifact into the AGFS resource tree, pinning the key invariant: a local
+artifact is uploaded file-by-file to the final resource location, matching source
+bytes, without going through persist_temp_tree or any AGFS temp write.
+"""
+
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import pytest
+
+from openviking.parse.output import LocalParseOutputStore
+from openviking.storage.context_update_plan import ContentTreeAction, ContextUpdatePlan
+from openviking.utils.content_hash import content_md5
+from openviking.utils.resource_processor import ResourceProcessor
+
+
+class _DummyVikingDB:
+ def get_embedder(self):
+ return None
+
+
+class _RecordingAgfs:
+ """Fake AGFS resource tree recording writes; forbids temp-tree persistence."""
+
+ def __init__(self) -> None:
+ self.files: dict[str, bytes] = {}
+ self.persist_temp_tree_calls = 0
+ self.created_temp_uris = 0
+
+ def create_temp_uri(self, ctx=None):
+ self.created_temp_uris += 1
+ raise AssertionError("local persist must not allocate an AGFS temp uri")
+
+ async def persist_temp_tree(self, *args, **kwargs):
+ self.persist_temp_tree_calls += 1
+ raise AssertionError("local persist must not call persist_temp_tree")
+
+ async def write_file_bytes(self, uri, content, *, ctx=None, lease_ref=None):
+ self.files[uri] = content
+
+ async def mkdir(self, uri, exist_ok=False, ctx=None, lease_ref=None):
+ del exist_ok, ctx, lease_ref
+
+ async def read_file_bytes(self, uri, *, ctx=None):
+ return self.files[uri]
+
+ async def remove_files(self, uri, *, recursive=False, ctx=None, lease_ref=None):
+ self.files.pop(uri, None)
+
+
+async def test_commit_local_artifact_uploads_to_resource_tree(tmp_path, monkeypatch):
+ import json
+
+ from openviking.parse.parsers.upload_utils import ARTIFACT_MANIFEST_NAME
+ from openviking.utils.content_hash import content_md5
+
+ # A local artifact laid out as <root>/repository/<files>, as code.parse writes.
+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_bytes(ref, "repository/a.py", b"print('a')")
+ await store.write_bytes(ref, "repository/src/b.py", b"print('b')")
+ # The manifest sidecar is the md5 source of truth (written by upload_directory).
+ await store.write_bytes(
+ ref,
+ ARTIFACT_MANIFEST_NAME,
+ json.dumps(
+ {
+ "repository/a.py": content_md5(b"print('a')"),
+ "repository/src/b.py": content_md5(b"print('b')"),
+ }
+ ).encode("utf-8"),
+ )
+
+ agfs = _RecordingAgfs()
+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: agfs)
+
+ rp = ResourceProcessor(vikingdb=_DummyVikingDB(), media_storage=None)
+
+ # temp_doc_uri is <artifact_root>/repository (finalize builds it this way).
+ doc_rel = rp._artifact_doc_rel(ref, f"{ref.root}/repository")
+ assert doc_rel == "repository"
+
+ plan = ContextUpdatePlan(
+ root_uri="viking://resources/acme/demo",
+ context_type="resource",
+ content_tree_actions=(
+ ContentTreeAction(
+ "upsert",
+ "a.py",
+ new_kind="file",
+ artifact_path="a.py",
+ md5=content_md5(b"print('a')"),
+ ),
+ ContentTreeAction(
+ "upsert",
+ "src/b.py",
+ new_kind="file",
+ artifact_path="src/b.py",
+ md5=content_md5(b"print('b')"),
+ ),
+ ),
+ )
+ snapshot = SimpleNamespace(
+ new=SimpleNamespace(entries={}),
+ formal=SimpleNamespace(entries={}),
+ vectors=SimpleNamespace(records_by_id={}),
+ )
+ monkeypatch.setattr(
+ "openviking.storage.resource_diff.build_rnfv_snapshot", AsyncMock(return_value=snapshot)
+ )
+ monkeypatch.setattr(
+ "openviking.storage.context_update_plan.build_context_update_plan_from_snapshot",
+ AsyncMock(return_value=(SimpleNamespace(entries={}), plan)),
+ )
+
+ committed = await rp._commit_directory_artifact_with_plan(
+ output_store=store,
+ artifact_ref=ref,
+ doc_rel=doc_rel,
+ root_uri="viking://resources/acme/demo",
+ target_preexisting=False,
+ ctx=SimpleNamespace(account_id="test-account"),
+ lease_ref=None,
+ vectorize=True,
+ summarize=False,
+ processing_mode="semantic_and_vectors",
+ is_code_repo=True,
+ ingest_options=None,
+ source_metadata=None,
+ )
+
+ # Files landed under the resource root with the repository prefix stripped,
+ # bytes intact, and no temp-tree persistence / AGFS temp allocation occurred.
+ assert agfs.files["viking://resources/acme/demo/a.py"] == b"print('a')"
+ assert agfs.files["viking://resources/acme/demo/src/b.py"] == b"print('b')"
+ assert agfs.persist_temp_tree_calls == 0
+ assert agfs.created_temp_uris == 0
+ assert committed.content_tree_actions == ()
+
+
+async def test_commit_local_flat_file_uploads_exact_target(tmp_path, monkeypatch):
+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))
+ ref = await store.create_artifact(root_type="dir")
+ await store.write_bytes(ref, "document/report.md", b"report")
+ agfs = _RecordingAgfs()
+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: agfs)
+
+ plan = ContextUpdatePlan(
+ root_uri="viking://resources/report.md",
+ context_type="resource",
+ content_tree_actions=(
+ ContentTreeAction(
+ "upsert",
+ "",
+ new_kind="file",
+ artifact_path="",
+ md5=content_md5(b"report"),
+ ),
+ ),
+ )
+ snapshot = SimpleNamespace(
+ new=SimpleNamespace(entries={}),
+ formal=SimpleNamespace(entries={}),
+ vectors=SimpleNamespace(records_by_id={}),
+ )
+ monkeypatch.setattr(
+ "openviking.storage.resource_diff.build_rnfv_snapshot", AsyncMock(return_value=snapshot)
+ )
+ monkeypatch.setattr(
+ "openviking.storage.context_update_plan.build_context_update_plan_from_snapshot",
+ AsyncMock(return_value=(SimpleNamespace(entries={}), plan)),
+ )
+
+ result = await ResourceProcessor(
+ vikingdb=_DummyVikingDB(), media_storage=None
+ )._commit_directory_artifact_with_plan(
+ output_store=store,
+ artifact_ref=ref,
+ doc_rel="document/report.md",
+ root_uri="viking://resources/report.md",
+ target_preexisting=False,
+ root_is_file=True,
+ ctx=SimpleNamespace(account_id="test-account"),
+ lease_ref=None,
+ vectorize=True,
+ summarize=False,
+ processing_mode="semantic_and_vectors",
+ is_code_repo=False,
+ ingest_options=None,
+ source_metadata=None,
+ )
+
+ assert agfs.files == {"viking://resources/report.md": b"report"}
+ assert result.content_tree_actions == ()
+
+
+def test_build_parse_output_store_defaults_to_none(monkeypatch):
+ # Default (agfs) config yields no store, so callers keep legacy behaviour.
+ rp = ResourceProcessor(vikingdb=_DummyVikingDB(), media_storage=None)
+ assert rp._build_parse_output_store() is None
+
+
+def test_artifact_doc_rel_falls_back_to_ref_metadata(tmp_path):
+ from openviking.parse.output import ParseArtifactRef
+
+ ref = ParseArtifactRef(
+ backend="local",
+ root=str(tmp_path / "artifact"),
+ resource_rel="document",
+ )
+
+ assert ResourceProcessor._artifact_doc_rel(ref, "not-a-path-below-the-root") == "document"
diff --git a/tests/utils/test_resource_processor_processing_mode.py b/tests/utils/test_resource_processor_processing_mode.py
index dec253fe3..3f9cd90fd 100644
--- a/tests/utils/test_resource_processor_processing_mode.py
+++ b/tests/utils/test_resource_processor_processing_mode.py
@@ -8,6 +8,13 @@ from unittest.mock import AsyncMock, Mock
import pytest
from openviking.server.identity import RequestContext, Role
+from openviking.storage.context_update_plan import (
+ ContextUpdatePlan,
+ SemanticPlan,
+ SemanticTreeEntry,
+ SemanticTreeSnapshot,
+)
+from openviking.storage.index_action import FieldPatch
from openviking.utils.ingest_options import IngestOptions
from openviking.utils.resource_processor import ResourceProcessor
from openviking_cli.session.user_id import UserIdentifier
@@ -149,8 +156,37 @@ async def test_flat_file_refreshes_parent_semantics_and_vectorizes_via_summary(
skip_vectorization=False,
ingest_options=IngestOptions(),
created=True,
+ file_md5=None,
+ file_abstract="",
+ )
+
+
+async def test_local_flat_file_refresh_passes_final_md5(monkeypatch, ctx):
+ root_uri = "viking://resources/report.md"
+ viking_fs = SimpleNamespace(
+ _async_agfs=SimpleNamespace(pathlock_release=AsyncMock()),
+ )
+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)
+ processor = ResourceProcessor(_FakeVikingDB())
+ summarizer = SimpleNamespace(refresh_file_parent=AsyncMock(return_value={"status": "success"}))
+ processor._get_summarizer = Mock(return_value=summarizer)
+
+ await processor.finish_prepared_resource(
+ {
+ "root_uri": root_uri,
+ "temp_uri": root_uri,
+ "source_committed": True,
+ "target_preexisting": True,
+ "root_is_file": True,
+ "file_md5s": {root_uri: "final-md5"},
+ },
+ ctx=ctx,
+ build_index=True,
)
+ assert summarizer.refresh_file_parent.await_args.kwargs["file_md5"] == "final-md5"
+
@pytest.mark.asyncio
async def test_flat_file_skips_all_post_processing_when_build_index_false(
@@ -178,6 +214,384 @@ async def test_flat_file_skips_all_post_processing_when_build_index_false(
vectorize_file.assert_not_awaited()
+async def test_local_artifact_without_semantic_work_is_cleaned(monkeypatch, ctx, tmp_path):
+ from openviking.parse.output import LocalParseOutputStore
+
+ store = LocalParseOutputStore(local_root=str(tmp_path))
+ ref = await store.create_artifact()
+ await store.write_bytes(ref, "repository/a.py", b"a")
+ processor = ResourceProcessor(_FakeVikingDB())
+ processor._build_parse_output_store = Mock(return_value=store)
+
+ await processor.finish_prepared_resource(
+ {
+ "root_uri": "viking://resources/demo",
+ "temp_uri": "viking://resources/demo",
+ "source_committed": True,
+ "artifact_ref": ref.to_dict(),
+ "artifact_files": ["a.py"],
+ },
+ ctx=ctx,
+ build_index=False,
+ )
+
+ assert not tmp_path.joinpath(ref.root).exists()
+
+
+async def test_local_incremental_noop_skips_semantic_queue_and_releases_resources(
+ monkeypatch, ctx, tmp_path
+):
+ from openviking.parse.output import LocalParseOutputStore
+
+ store = LocalParseOutputStore(local_root=str(tmp_path))
+ ref = await store.create_artifact()
+ await store.write_bytes(ref, "repository/a.py", b"a")
+ lock = {"lease_ref": "noop"}
+ viking_fs = SimpleNamespace(
+ _async_agfs=SimpleNamespace(pathlock_release=AsyncMock()),
+ )
+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)
+ processor = ResourceProcessor(_FakeVikingDB())
+ processor._build_parse_output_store = Mock(return_value=store)
+ summarizer = SimpleNamespace(summarize=AsyncMock(return_value={"status": "success"}))
+ processor._get_summarizer = Mock(return_value=summarizer)
+
+ result = await processor.finish_prepared_resource(
+ {
+ "root_uri": "viking://resources/demo",
+ "temp_uri": "viking://resources/demo",
+ "source_committed": True,
+ "target_preexisting": True,
+ "artifact_ref": ref.to_dict(),
+ "artifact_files": ["a.py"],
+ "changes": {},
+ "file_md5s": {},
+ "file_abstracts": {},
+ "incremental_noop": True,
+ },
+ ctx=ctx,
+ resource_lock=lock,
+ build_index=True,
+ )
+
+ assert result == {"status": "success", "root_uri": "viking://resources/demo"}
+ summarizer.summarize.assert_not_awaited()
+ viking_fs._async_agfs.pathlock_release.assert_awaited_once_with(lock)
+ assert not tmp_path.joinpath(ref.root).exists()
+
+
+async def test_vectors_only_scalar_update_is_enqueued_before_lock_release(monkeypatch, ctx):
+ from openviking.storage.context_update_plan import DirectIndexAction
+
+ queue = SimpleNamespace(enqueue=AsyncMock(return_value="queued"))
+ manager = SimpleNamespace(
+ EMBEDDING="embedding",
+ get_queue=lambda *_args, **_kwargs: queue,
+ )
+ monkeypatch.setattr("openviking.storage.queuefs.get_queue_manager", lambda: manager)
+ viking_fs = SimpleNamespace(
+ _async_agfs=SimpleNamespace(pathlock_release=AsyncMock()),
+ )
+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)
+ processor = ResourceProcessor(_FakeVikingDB())
+ action = DirectIndexAction(
+ action="update_fields",
+ record_id="a-l2",
+ uri="viking://resources/demo/a.py",
+ level=2,
+ field_patch=FieldPatch({"search_tags": ["team=search"]}),
+ )
+ plan = ContextUpdatePlan(
+ root_uri="viking://resources/demo",
+ context_type="resource",
+ direct_index_actions=(action,),
+ )
+ lock = {"lease_ref": "tags"}
+
+ result = await processor.finish_prepared_resource(
+ {
+ "root_uri": "viking://resources/demo",
+ "temp_uri": "viking://resources/demo",
+ "source_committed": True,
+ "target_preexisting": True,
+ "incremental_noop": True,
+ "context_update_plan": plan.to_dict(),
+ },
+ ctx=ctx,
+ resource_lock=lock,
+ build_index=False,
+ processing_mode="vectors_only",
+ )
+
+ assert result == {"status": "success", "root_uri": "viking://resources/demo"}
+ message = queue.enqueue.await_args.args[0]
+ assert message.action.value == "update_fields"
+ assert message.record_ids == ["a-l2"]
+ assert message.update_fields["search_tags"] == ["team=search"]
+ viking_fs._async_agfs.pathlock_release.assert_awaited_once_with(lock)
+
+
+async def test_directory_semantic_plan_is_enqueued_without_artifact_or_legacy_diff(
+ monkeypatch, ctx
+):
+ root_uri = "viking://resources/demo"
+ plan = SemanticPlan(
+ root_uri=root_uri,
+ context_type="resource",
+ tree=SemanticTreeSnapshot(
+ entries=(
+ SemanticTreeEntry("", "directory", "unchanged", "aggregate"),
+ SemanticTreeEntry("a.py", "file", "modified", "generate", md5="new"),
+ )
+ ),
+ )
+ context_plan = ContextUpdatePlan(root_uri, "resource", semantic_plan=plan)
+ viking_fs = SimpleNamespace(
+ _async_agfs=SimpleNamespace(
+ pathlock_to_handoff=AsyncMock(return_value={"lease": "x"}),
+ pathlock_handoff=AsyncMock(),
+ pathlock_release=AsyncMock(),
+ )
+ )
+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)
+ processor = ResourceProcessor(_FakeVikingDB())
+ summarizer = SimpleNamespace(
+ summarize=AsyncMock(return_value={"status": "success", "enqueued_count": 1})
+ )
+ processor._get_summarizer = Mock(return_value=summarizer)
+
+ await processor.finish_prepared_resource(
+ {
+ "root_uri": root_uri,
+ "temp_uri": root_uri,
+ "source_committed": True,
+ "target_preexisting": True,
+ "root_is_file": False,
+ "context_update_plan": context_plan.to_dict(),
+ },
+ ctx=ctx,
+ resource_lock={"lease_ref": "plan"},
+ build_index=True,
+ processing_mode="semantic_and_vectors",
+ )
+
+ kwargs = summarizer.summarize.await_args.kwargs
+ assert SemanticPlan.from_dict(kwargs["semantic_plan"]) == plan
+ assert kwargs["temp_uris"] == [root_uri]
+ assert kwargs.get("changes") is None
+ assert kwargs.get("artifact_ref") is None
+
+
+async def test_plan_artifact_is_cleaned_only_after_semantic_enqueue(monkeypatch, ctx, tmp_path):
+ from openviking.parse.output import LocalParseOutputStore
+
+ root_uri = "viking://resources/demo"
+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))
+ ref = await store.create_artifact()
+ await store.write_bytes(ref, "repository/a.py", b"a")
+ plan = SemanticPlan(
+ root_uri=root_uri,
+ context_type="resource",
+ tree=SemanticTreeSnapshot(
+ entries=(
+ SemanticTreeEntry("", "directory", "added", "aggregate"),
+ SemanticTreeEntry("a.py", "file", "added", "generate", md5="a"),
+ )
+ ),
+ )
+ context_plan = ContextUpdatePlan(root_uri, "resource", semantic_plan=plan)
+ lock = {"lease_ref": "plan"}
+ viking_fs = SimpleNamespace(
+ _async_agfs=SimpleNamespace(
+ pathlock_handoff=AsyncMock(),
+ pathlock_release=AsyncMock(),
+ )
+ )
+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)
+ processor = ResourceProcessor(_FakeVikingDB())
+ processor._build_parse_output_store = Mock(return_value=store)
+
+ async def summarize(**kwargs):
+ assert kwargs.get("artifact_ref") is None
+ assert tmp_path.joinpath("artifacts", ref.root.split("/")[-1]).exists()
+ return {"status": "success", "enqueued_count": 1}
+
+ processor._get_summarizer = Mock(
+ return_value=SimpleNamespace(summarize=AsyncMock(side_effect=summarize))
+ )
+
+ await processor.finish_prepared_resource(
+ {
+ "root_uri": root_uri,
+ "temp_uri": root_uri,
+ "source_committed": True,
+ "target_preexisting": True,
+ "root_is_file": False,
+ "artifact_ref": ref.to_dict(),
+ "context_update_plan": context_plan.to_dict(),
+ "plan_artifact_committed": True,
+ },
+ ctx=ctx,
+ resource_lock=lock,
+ build_index=True,
+ )
+
+ assert not tmp_path.joinpath("artifacts", ref.root.split("/")[-1]).exists()
+ viking_fs._async_agfs.pathlock_handoff.assert_awaited_once_with(lock)
+
+
+async def test_plan_enqueue_failure_cleans_artifact_and_releases_lock(monkeypatch, ctx, tmp_path):
+ from openviking.parse.output import LocalParseOutputStore
+
+ root_uri = "viking://resources/demo"
+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))
+ ref = await store.create_artifact()
+ await store.write_bytes(ref, "repository/a.py", b"a")
+ plan = SemanticPlan(
+ root_uri=root_uri,
+ context_type="resource",
+ tree=SemanticTreeSnapshot(
+ entries=(
+ SemanticTreeEntry("", "directory", "added", "aggregate"),
+ SemanticTreeEntry("a.py", "file", "added", "generate", md5="a"),
+ )
+ ),
+ )
+ context_plan = ContextUpdatePlan(root_uri, "resource", semantic_plan=plan)
+ lock = {"lease_ref": "plan"}
+ viking_fs = SimpleNamespace(
+ _async_agfs=SimpleNamespace(
+ pathlock_handoff=AsyncMock(),
+ pathlock_release=AsyncMock(),
+ )
+ )
+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)
+ processor = ResourceProcessor(_FakeVikingDB())
+ processor._build_parse_output_store = Mock(return_value=store)
+ processor._get_summarizer = Mock(
+ return_value=SimpleNamespace(
+ summarize=AsyncMock(side_effect=RuntimeError("queue unavailable"))
+ )
+ )
+
+ with pytest.raises(RuntimeError, match="queue unavailable"):
+ await processor.finish_prepared_resource(
+ {
+ "root_uri": root_uri,
+ "temp_uri": root_uri,
+ "source_committed": True,
+ "target_preexisting": True,
+ "root_is_file": False,
+ "artifact_ref": ref.to_dict(),
+ "context_update_plan": context_plan.to_dict(),
+ "plan_artifact_committed": True,
+ },
+ ctx=ctx,
+ resource_lock=lock,
+ build_index=True,
+ )
+
+ assert not tmp_path.joinpath("artifacts", ref.root.split("/")[-1]).exists()
+ viking_fs._async_agfs.pathlock_handoff.assert_not_awaited()
+ viking_fs._async_agfs.pathlock_release.assert_awaited_once_with(lock)
+
+
+async def test_plan_enqueue_error_result_cleans_artifact_and_fails(monkeypatch, ctx, tmp_path):
+ from openviking.parse.output import LocalParseOutputStore
+
+ root_uri = "viking://resources/demo"
+ store = LocalParseOutputStore(local_root=str(tmp_path / "artifacts"))
+ ref = await store.create_artifact()
+ await store.write_bytes(ref, "repository/a.py", b"a")
+ plan = SemanticPlan(
+ root_uri=root_uri,
+ context_type="resource",
+ tree=SemanticTreeSnapshot(
+ entries=(
+ SemanticTreeEntry("", "directory", "added", "aggregate"),
+ SemanticTreeEntry("a.py", "file", "added", "generate", md5="a"),
+ )
+ ),
+ )
+ context_plan = ContextUpdatePlan(root_uri, "resource", semantic_plan=plan)
+ lock = {"lease_ref": "plan"}
+ viking_fs = SimpleNamespace(
+ _async_agfs=SimpleNamespace(
+ pathlock_handoff=AsyncMock(),
+ pathlock_release=AsyncMock(),
+ )
+ )
+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)
+ processor = ResourceProcessor(_FakeVikingDB())
+ processor._build_parse_output_store = Mock(return_value=store)
+ processor._get_summarizer = Mock(
+ return_value=SimpleNamespace(
+ summarize=AsyncMock(return_value={"status": "error", "message": "queue rejected plan"})
+ )
+ )
+
+ with pytest.raises(RuntimeError, match="queue rejected plan"):
+ await processor.finish_prepared_resource(
+ {
+ "root_uri": root_uri,
+ "temp_uri": root_uri,
+ "source_committed": True,
+ "target_preexisting": True,
+ "root_is_file": False,
+ "artifact_ref": ref.to_dict(),
+ "context_update_plan": context_plan.to_dict(),
+ "plan_artifact_committed": True,
+ },
+ ctx=ctx,
+ resource_lock=lock,
+ build_index=True,
+ )
+
+ assert not tmp_path.joinpath("artifacts", ref.root.split("/")[-1]).exists()
+ viking_fs._async_agfs.pathlock_handoff.assert_not_awaited()
+ viking_fs._async_agfs.pathlock_release.assert_awaited_once_with(lock)
+
+
+async def test_local_flat_refresh_failure_cleans_artifact(monkeypatch, ctx, tmp_path):
+ from openviking.parse.output import LocalParseOutputStore
+
+ store = LocalParseOutputStore(local_root=str(tmp_path))
+ ref = await store.create_artifact()
+ await store.write_bytes(ref, "document/report.md", b"body")
+ processor = ResourceProcessor(_FakeVikingDB())
+ processor._build_parse_output_store = Mock(return_value=store)
+ processor._get_summarizer = Mock(
+ return_value=SimpleNamespace(
+ refresh_file_parent=AsyncMock(side_effect=RuntimeError("queue unavailable"))
+ )
+ )
+
+ with pytest.raises(RuntimeError, match="queue unavailable"):
+ await processor.finish_prepared_resource(
+ {
+ "root_uri": "viking://resources/report.md",
+ "temp_uri": "viking://resources/report.md",
+ "source_committed": True,
+ "target_preexisting": True,
+ "root_is_file": True,
+ "artifact_ref": ref.to_dict(),
+ "artifact_files": [""],
+ },
+ ctx=ctx,
+ build_index=True,
+ )
+
+ assert not tmp_path.joinpath(ref.root).exists()
+
+
@pytest.mark.asyncio
async def test_vectors_only_replaces_preexisting_flat_file_without_directory_sync(
monkeypatch,
@@ -352,9 +766,56 @@ async def test_vectors_only_persists_tree_and_vectorizes_files_only(monkeypatch,
search_tags=["team=search"],
search_tag_mode="append",
)
- processor._delete_resource_semantic_markers.assert_not_awaited()
- processor._delete_resource_semantic_vectors.assert_not_awaited()
- viking_fs._async_agfs.pathlock_release.assert_awaited_once_with(lock)
+
+
+async def test_local_vectors_only_uses_artifact_snapshot_when_target_tree_is_empty(
+ monkeypatch, ctx, tmp_path
+):
+ from openviking.parse.output import LocalParseOutputStore, ParseArtifactRef
+ from openviking.utils.content_hash import content_md5
+
+ store = LocalParseOutputStore(local_root=str(tmp_path))
+ raw_ref = await store.create_artifact()
+ ref = ParseArtifactRef(
+ backend="local",
+ root=raw_ref.root,
+ resource_rel="repository",
+ )
+ await store.write_bytes(ref, "repository/a.py", b"print('a')")
+ viking_fs = SimpleNamespace(tree=AsyncMock(return_value=[]))
+ vectorize_file = AsyncMock(return_value=True)
+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)
+ monkeypatch.setattr("openviking.utils.resource_processor.vectorize_file", vectorize_file)
+ monkeypatch.setattr(
+ "openviking.utils.resource_processor.get_openviking_config",
+ lambda: SimpleNamespace(
+ queue_workers=SimpleNamespace(
+ add_resource=SimpleNamespace(file_vectorization_concurrency=8)
+ )
+ ),
+ )
+ processor = ResourceProcessor(_FakeVikingDB())
+ processor._build_parse_output_store = Mock(return_value=store)
+
+ await processor.finish_prepared_resource(
+ {
+ "root_uri": "viking://resources/demo",
+ "temp_uri": "viking://resources/demo",
+ "source_committed": True,
+ "artifact_ref": ref.to_dict(),
+ "artifact_files": ["a.py"],
+ "file_md5s": {"viking://resources/demo/a.py": content_md5(b"print('a')")},
+ },
+ ctx=ctx,
+ build_index=True,
+ processing_mode="vectors_only",
+ )
+
+ viking_fs.tree.assert_not_awaited()
+ vectorize_file.assert_awaited_once()
+ assert vectorize_file.await_args.kwargs["file_content"] == b"print('a')"
+ assert vectorize_file.await_args.kwargs["file_md5"] == content_md5(b"print('a')")
@pytest.mark.asyncio