Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

Feat/batch write rnfv (#5897)

Jiahui Zhouc90dae83df99
diff --git a/docs/en/api/12-content.md b/docs/en/api/12-content.md
index fdf13ee39..470eda6c3 100644
--- a/docs/en/api/12-content.md
+++ b/docs/en/api/12-content.md
@@ -216,7 +216,7 @@ Write a file and automatically refresh related semantics and vectors.
|-----------|------|----------|---------|-------------|
| uri | str | Yes | - | File URI to write |
| content | str | Yes | - | New content to write |
-| mode | str | No | `replace` | `replace` overwrites an existing file or creates a missing file; `append` appends to an existing file or creates a missing file; `create` creates only a missing file and returns `409 Conflict` if it already exists |
+| mode | str | No | `replace` | `replace` overwrites an existing file or creates a missing file; `append` appends to an existing file or creates a missing file; `create` is a compatibility alias for `replace` |
| wait | bool | No | `false` | Wait for background semantic/vector refresh |
| timeout | float | No | `null` | Timeout in seconds when `wait=true` |
| tags | string[] | No | Unset | Explicit retrieval tags for the written file, for example `["team=search", "env=prod"]` |
@@ -224,10 +224,11 @@ Write a file and automatically refresh related semantics and vectors.
**Notes**
-- `replace` and `append` create a missing target file. `append` uses the supplied content as the initial file content in that case. `create` targets only a missing file and returns `409 Conflict` when the path already exists. Directories are always rejected.
-- Explicit `create` only accepts text-writable extensions: `.md`, `.txt`, `.json`, `.yaml`, `.yml`, `.toml`, `.py`, `.js`, `.ts`. Parent directories are created automatically for every write mode.
+- `replace`, `append`, and the `create` compatibility alias create a missing target file. `append` uses the supplied content as the initial file content in that case; `create` otherwise behaves exactly like `replace`. Directories are always rejected.
+- Creating a missing file with any mode only accepts text-writable extensions: `.md`, `.txt`, `.json`, `.yaml`, `.yml`, `.toml`, `.py`, `.js`, `.ts`. Parent directories are created automatically for every write mode.
- Existing `.abstract.md` and `.overview.md` bodies may be updated, but public APIs cannot create them. A body-only request preserves stored OKF metadata; a full-OKF request must match the stored metadata. Unknown metadata fields are silently dropped. A sidecar body write rebuilds only the directory's existing L0/L1 vectors and does not regenerate semantics.
- File content is updated before the API returns. `wait` only controls whether the call waits for semantic/vector refresh to finish.
+- Once file content is committed, a later semantic/vector enqueue failure does not roll it back. The request returns an error; retry the same write to resubmit derived maintenance. Explicit writes always recommit their final bytes, so retries remain safe while the vector index lags the file.
- The public API no longer accepts `regenerate_semantics` or `revectorize`; write automatically schedules related semantic and vector processing.
- Parent L0/L1 refreshes for resource writes are best-effort: a parent lock conflict skips that directory refresh while preserving the file write and its own summary/vector work. Skipping L0/L1 persistence also skips directory vector updates; a later refresh is not guaranteed. Locks on the written file itself still raise conflicts. Contention detected before enqueueing returns `semantic_status: "skipped"`; skips during background execution are logged, and `wait=true` does not guarantee updated parent summaries.
- When non-empty `tags` are supplied, tags are included in the file's first vector upsert rather than updated after processing. Omitting `tags`, or using `tags: []` with `tag_mode: "replace"`, preserves existing tags. Use `tag_mode: "clear"` to remove all existing tags; `clear` ignores any supplied tag values.
@@ -347,6 +348,7 @@ Write multiple files below one Resource or Memory directory, then refresh the af
| `wait` | boolean | No | `true` | Wait for semantic/vector refresh |
| `timeout` | number | No | `null` | Refresh timeout in seconds when `wait=true` |
| `telemetry` | boolean/object | No | `false` | Include operation telemetry |
+| `processing_mode` | string | No | `semantic_and_vectors` | One processing mode for the whole batch: `semantic_and_vectors` or `vectors_only` |
Each operation contains:
@@ -356,18 +358,24 @@ Each operation contains:
| `content` | string | Conditional | UTF-8 text; exactly one of `content` and `content_base64` is required |
| `content_base64` | string | Conditional | Base64-encoded bytes; not supported for Memory targets |
| `mode` | string | No | `replace` (default), `append`, `create`, or `upsert` |
+| `tags` | string[] | No | - | Explicit retrieval tags for this file |
+| `tag_mode` | string | No | `replace` | Tag write mode for this file: `replace`, `append`, or `clear` |
+| `acl` | object | No | - | Direct ACL for this file; requires manage permission on the file |
**Notes**
- Operation count, individual file size, and total content size have no application-level quotas.
- All targets must be files below `root_uri`, use the same context type, and have unique canonical URIs.
- Resource targets may use any safe file extension; Memory targets retain the text extension allowlist and do not accept binary content.
-- `replace`, `append`, and `create` match `write()` semantics. `upsert` replaces an existing file or creates a missing file.
+- `create`, `replace`, and `upsert` are compatible spellings for setting the requested final body: they overwrite an existing file and create a missing one. `append` appends when the file exists and creates it with the supplied body when it does not.
+- `processing_mode` applies to the whole batch; operations cannot mix semantic modes. `vectors_only` maintains changed-file vectors without refreshing parent semantics.
+- `tags`, `tag_mode`, and `acl` apply per operation. Tags travel with the file's first vector upsert; `clear` does not require `tags` and ignores supplied tag values.
- The batch acquires exact locks for all target files before validating file state and writing. Writes to disjoint files in the same directory can proceed concurrently; overlapping writes and parent-directory deletion or moves still conflict. Semantic processing starts after all writes finish and the locks are released, refreshing the affected `.overview.md` and `.abstract.md` files together.
- Resource parent refreshes use the same best-effort behavior as `write()`: L0/L1 lock conflicts skip the directory refresh and its vector updates while preserving file writes and file processing. A later refresh is not guaranteed.
- An underlying I/O failure can still leave writes completed earlier in the batch visible.
- Existing `.abstract.md` and `.overview.md` bodies may be replaced or appended. OpenViking preserves and validates protected OKF metadata and rebuilds only the directory's existing L0/L1 vectors for these operations.
-- In the response body, `semantic_status` (`queued`, `complete`, `deferred`, or `skipped`) reports the directory aggregation status; it is `skipped` if any directory encounters contention before enqueueing. Meanwhile, `vector_status` reports vector maintenance for changed files.
+- In the response body, `semantic_status` (`queued`, `complete`, `deferred`, or `skipped`) reports grouped parent-directory aggregation. It is `skipped` only when no parent group was enqueued; one busy parent does not hide work enqueued for another. `vector_status` reports vector maintenance for changed files.
+- If formal content commits but asynchronous refresh enqueue fails, the body is kept. Re-run with the final body using `replace`, `create`, or `upsert`; do not blindly replay an already-successful `append`, because it would append twice.
**Python SDK**
diff --git a/docs/en/guides/06-mcp-integration.md b/docs/en/guides/06-mcp-integration.md
index ff627796e..175f7b06f 100644
--- a/docs/en/guides/06-mcp-integration.md
+++ b/docs/en/guides/06-mcp-integration.md
@@ -132,7 +132,7 @@ The built-in MCP tools are listed below. Use the connected server's `tools/list`
| `list` | List entries under a `viking://` directory | `uri`, `recursive`, `offset`, `limit`, `sort_by`, `sort_order` (optional) |
| `tree` | Show the recursive directory tree under a `viking://` URI, indented by depth — use when you need a full picture of the file tree (prefer `list` for a single level, `glob` for filename patterns) | `uri` (optional), `level_limit` (default 3), `node_limit` (default 1000), `offset`, `limit`, `include_abstract` (optional — also show each directory's summary; for a skill directory that is its name and description) |
| `remember` | Submit messages for long-term memory extraction; returns the background extraction `task_id` right away | `messages` (list of `{role, content}`) |
-| `write` | Write text to a `viking://` file (create/overwrite/append). Parent directories are created automatically; use `read` first to see current content before overwriting, and prefer `edit` for changing part of an existing file. Skill packages are not maintained this way: the caller's own `skills/` subtree is refused, and a write under `viking://agent/skills` produces a plain file that skips skill installation — use `add_skill` | `uri`, `content`, `mode` (optional: `replace` default — overwrites or creates if missing; `append` — appends or creates if missing; `create` — fails if it exists), `wait` (optional, block until re-indexed), `timeout` (optional), `acl` (optional) |
+| `write` | Write text to a `viking://` file (create/overwrite/append). Parent directories are created automatically; use `read` first to see current content before overwriting, and prefer `edit` for changing part of an existing file. Skill packages are not maintained this way: the caller's own `skills/` subtree is refused, and a write under `viking://agent/skills` produces a plain file that skips skill installation — use `add_skill` | `uri`, `content`, `mode` (optional: `replace` default — overwrites or creates if missing; `append` — appends or creates if missing; `create` — compatibility alias for `replace`), `wait` (optional, block until re-indexed), `timeout` (optional), `acl` (optional) |
| `edit` | Replace an exact string with new text in an existing `viking://` file — for targeted changes instead of a full rewrite. The file is left unchanged if `old_string` is not found, or matches multiple times while `replace_all` is false. Editing a file inside a skill package does not re-run skill installation — use `add_skill` | `uri`, `old_string`, `new_string`, `replace_all` (optional), `wait` (optional, block until re-indexed), `timeout` (optional) |
| `add_resource` | Add a local file or URL as a resource (local files trigger a progressive upload flow) | `path`, `temp_file_id` (optional), `description` (optional), `watch_interval` (optional, minutes — auto-refresh cadence for remote URLs), `processing_mode` (optional: `semantic_and_vectors` default, or `vectors_only` to skip VLM semantic understanding and only vectorize current files), `to` (optional, target `viking://resources/...` URI; if omitted when `watch_interval > 0`, the watch auto-binds to the resource's created URI), `args` (optional parser-specific options, including `{"parse_mode":"no_split"}` to parse each source document into one Markdown body, `{"feishu_access_token":"u-..."}` for one-time Feishu user-token imports, or access/refresh tokens plus an optional `feishu_app_id` / `feishu_app_secret` pair for Feishu user-token watches), `acl` (optional) |
| `add_skill` | Create, install, or replace an agent skill. New skills pass the full SKILL.md text; Git and GitHub tree URLs install every skill in the source unless `skills` names some; a local SKILL.md, directory, or zip returns a signed upload URL like `add_resource` | `data` (SKILL.md text) or `path` (Git URL or local path), `skills` (optional), `target_uri` (optional; `viking://agent/skills` shares with the account), `list_only` (optional) |
diff --git a/docs/zh/api/12-content.md b/docs/zh/api/12-content.md
index cc8a3aec1..4991f5db8 100644
--- a/docs/zh/api/12-content.md
+++ b/docs/zh/api/12-content.md
@@ -216,7 +216,7 @@ ov read viking://resources/docs/api.md
|------|------|------|--------|------|
| uri | str | 是 | - | 要写入的文件 URI |
| content | str | 是 | - | 要写入的新内容 |
-| mode | str | 否 | `replace` | `replace` 覆盖已有文件、缺失时创建;`append` 追加已有文件、缺失时创建;`create` 仅创建缺失文件,目标已存在时返回 `409 Conflict` |
+| mode | str | 否 | `replace` | `replace` 覆盖已有文件、缺失时创建;`append` 追加已有文件、缺失时创建;`create` 是 `replace` 的兼容别名 |
| wait | bool | 否 | `false` | 是否等待后台语义/向量刷新完成 |
| timeout | float | 否 | `null` | 当 `wait=true` 时的超时时间(秒) |
| tags | string[] | 否 | 未设置 | 写入文件的显式检索标签,例如 `["team=search", "env=prod"]` |
@@ -224,10 +224,11 @@ ov read viking://resources/docs/api.md
**说明**
-- `replace` 和 `append` 在目标文件缺失时都会创建文件;其中 `append` 会以传入内容作为新文件的初始内容。`create` 仅用于创建缺失文件,目标路径已存在时返回 `409 Conflict`。目录始终会被拒绝。
-- 显式 `create` 只允许以下文本类扩展名:`.md`、`.txt`、`.json`、`.yaml`、`.yml`、`.toml`、`.py`、`.js`、`.ts`。所有写入模式都会自动创建父目录。
+- `replace`、`append` 和作为兼容别名的 `create` 都会在目标缺失时创建文件;`append` 会以传入内容作为新文件的初始内容,其他情况下 `create` 与 `replace` 完全相同。目录始终会被拒绝。
+- 使用任意模式创建缺失文件时,都只允许以下文本类扩展名:`.md`、`.txt`、`.json`、`.yaml`、`.yml`、`.toml`、`.py`、`.js`、`.ts`。所有写入模式都会自动创建父目录。
- 已存在的 `.abstract.md` / `.overview.md` 可以修改正文,但不能通过公共 API 创建;只提交正文时会保留现有 OKF metadata,提交完整 OKF 时 metadata 必须与存量值一致。未知 metadata 字段会静默丢弃。sidecar 正文写入只重建该目录实际存在的 L0/L1 向量,不触发语义重新生成。
- 文件内容会在 API 返回前完成更新;`wait` 只控制是否等待语义/向量刷新完成。
+- 正文提交后,如果后续语义/向量任务入队失败,不会回滚已写正文;接口会返回错误,调用方应重试同一次 write 以重新派发派生维护。显式 write 始终重新提交最终字节,因此在向量索引暂时落后于文件时也能安全重试。
- 公共 API 已不再接受 `regenerate_semantics` 或 `revectorize`;写入后会自动调度相关语义与向量处理。
- 资源写入附带的父目录 L0/L1 刷新采用尽力更新:父目录锁冲突时跳过本次目录刷新,保留原文写入及文件自身的摘要、向量处理。跳过 L0/L1 写回时也跳过目录向量更新,不保证自动补刷;原文文件本身的锁冲突仍报错。提交任务前发现冲突时返回 `semantic_status: "skipped"`;后台运行期间的跳过记录在日志中,`wait=true` 也不保证父目录摘要更新。
- 提供非空 `tags` 时,标签会在该文件首次向量 upsert 时写入,而非在处理完成后再单独更新。省略 `tags`,或显式传 `tags: []` 配合 `tag_mode: "replace"`,都不会修改已有标签。使用 `tag_mode: "clear"` 可清空全部已有标签,且 `clear` 会忽略同时传入的标签值。
@@ -347,6 +348,7 @@ ov write viking://resources/docs/api.md \
| `wait` | boolean | 否 | `true` | 是否等待语义/向量刷新 |
| `timeout` | number | 否 | `null` | `wait=true` 时的刷新超时时间(秒) |
| `telemetry` | boolean/object | 否 | `false` | 是否返回操作遥测信息 |
+| `processing_mode` | string | 否 | `semantic_and_vectors` | 整批统一的处理模式:`semantic_and_vectors` 或 `vectors_only` |
每个 operation 包含:
@@ -356,18 +358,24 @@ ov write viking://resources/docs/api.md \
| `content` | string | 条件必填 | UTF-8 文本;与 `content_base64` 必须且只能提供一个 |
| `content_base64` | string | 条件必填 | Base64 编码的字节;Memory 目标不支持 |
| `mode` | string | 否 | `replace`(默认)、`append`、`create` 或 `upsert` |
+| `tags` | string[] | 否 | - | 该文件的显式检索标签 |
+| `tag_mode` | string | 否 | `replace` | 该文件标签写入方式:`replace`、`append` 或 `clear` |
+| `acl` | object | 否 | - | 该文件的直接 ACL;需要该文件的 manage 权限 |
**说明**
- 不设置 operation 数量、单文件大小或总内容大小的应用层上限。
- 所有目标必须是 `root_uri` 下的文件、属于同一 context type,且 canonical URI 不能重复。
- Resource 目标允许任意安全文件扩展名;Memory 目标仍使用文本扩展名白名单,且不接受二进制内容。
-- `replace`、`append`、`create` 与 `write()` 语义一致;`upsert` 会覆盖已有文件或创建缺失文件。
+- `create`、`replace`、`upsert` 是“写成请求最终正文”的兼容写法:文件存在时覆盖,缺失时创建。`append` 在文件存在时追加,文件缺失时以请求正文创建文件。
+- `processing_mode` 对整个 batch 生效,不能在不同 operation 中混用语义模式;`vectors_only` 不刷新父目录语义,只维护变化文件的向量。
+- `tags`、`tag_mode` 与 `acl` 按 operation 生效;tags 会随该文件首次向量 upsert 写入,`clear` 不要求传 `tags` 并会忽略同时传入的标签值。
- 整批先获取所有目标文件的精确锁,再校验文件状态并写入;同一目录下不涉及相同文件的写入可以并行,重叠文件的写入或父目录删除、移动仍会冲突。所有文件写完并释放锁后才启动语义处理,统一刷新受影响的 `.overview.md` / `.abstract.md`。
- 资源父目录刷新与 `write()` 一样采用尽力更新:L0/L1 锁冲突时跳过目录刷新及对应目录向量更新,保留原文和文件自身的处理,不保证自动补刷。
- 底层 I/O 中途失败时,本批次较早完成的写入仍可能已经可见。
- 已存在的 `.abstract.md` / `.overview.md` 可以 replace 或 append;系统会保留并校验受保护的 OKF metadata,并只重建对应目录实际存在的 L0/L1 向量。
-- 响应体中,通过 `semantic_status`(`queued`、`complete`、`deferred` 或 `skipped`)表达目录聚合状态;提交任务前任一目录因锁冲突跳过时为 `skipped`,通过 `vector_status` 表达变化文件的向量维护状态。
+- 响应体中,通过 `semantic_status`(`queued`、`complete`、`deferred` 或 `skipped`)表达按直接父目录聚合的状态;只有所有父目录都未入队时才为 `skipped`,某一个目录因锁冲突跳过不会掩盖其他目录已入队的工作。`vector_status` 表达变化文件的向量维护状态。
+- 若正文已提交但异步刷新入队失败,正文不会回滚。可使用相同最终正文通过 `replace`、`create` 或 `upsert` 重写;不要直接重放已经成功过的 `append`,否则会重复追加。
**Python SDK**
diff --git a/docs/zh/guides/06-mcp-integration.md b/docs/zh/guides/06-mcp-integration.md
index ef51464b8..ff1d6fd18 100644
--- a/docs/zh/guides/06-mcp-integration.md
+++ b/docs/zh/guides/06-mcp-integration.md
@@ -128,7 +128,7 @@ claude mcp add --transport http openviking \
| `list` | 列出 `viking://` 目录下的条目 | `uri`, `recursive`(可选) |
| `tree` | 以缩进形式展示 `viking://` URI 下的递归目录树——当需要全面了解文件树结构时使用(单层列表用 `list`,按文件名查找用 `glob`) | `uri`(可选), `level_limit`(默认 3), `node_limit`(默认 1000), `include_abstract`(可选——同时展示每个目录的摘要;skill 目录的摘要就是它的名字和描述) |
| `remember` | 提交消息做长期记忆提取;立即返回后台任务的 `task_id`,提取结果可能什么都不保留 | `messages`(`{role, content}` 列表) |
-| `write` | 向 `viking://` 文件写入文本(创建/覆盖/追加)。自动创建缺失的父目录;覆盖前请先用 `read` 查看当前内容;只改文件局部时优先用 `edit`。skill 包不要用它维护:调用方自己的 `skills/` 子树会被拒绝,写 `viking://agent/skills` 则生成绕过安装流程的普通文件,请改用 `add_skill` | `uri`, `content`, `mode`(可选:默认 `replace` — 覆盖或在缺失时创建,`append` — 追加或在缺失时创建,`create` — 已存在则失败), `wait`(可选,阻塞直到重建索引完成), `timeout`(可选), `acl`(可选) |
+| `write` | 向 `viking://` 文件写入文本(创建/覆盖/追加)。自动创建缺失的父目录;覆盖前请先用 `read` 查看当前内容;只改文件局部时优先用 `edit`。skill 包不要用它维护:调用方自己的 `skills/` 子树会被拒绝,写 `viking://agent/skills` 则生成绕过安装流程的普通文件,请改用 `add_skill` | `uri`, `content`, `mode`(可选:默认 `replace` — 覆盖或在缺失时创建,`append` — 追加或在缺失时创建,`create` — `replace` 的兼容别名), `wait`(可选,阻塞直到重建索引完成), `timeout`(可选), `acl`(可选) |
| `edit` | 在已有 `viking://` 文件中把精确字符串替换为新文本——用于局部修改,避免整文件重写。若 `old_string` 找不到、或匹配多处且 `replace_all` 为 false,则编辑失败且文件保持不变。编辑 skill 包内的文件不会重新触发 skill 安装流程,请改用 `add_skill` | `uri`, `old_string`, `new_string`, `replace_all`(可选), `wait`(可选,阻塞直到重建索引完成), `timeout`(可选) |
| `add_resource` | 添加本地文件或 URL 作为资源(本地文件触发渐进式上传流) | `path`, `temp_file_id`(可选), `description`(可选), `watch_interval`(可选,分钟数 — 远程 URL 的自动刷新周期), `processing_mode`(可选:默认 `semantic_and_vectors`;传 `vectors_only` 时跳过 VLM 语义理解,只向量化当前文件), `to`(可选,目标 `viking://resources/...` URI;`watch_interval > 0` 时若省略 `to`,watch 将自动绑定到本次 add 创建的资源 URI), `args`(可选,特定 parser 参数,包括 `{"parse_mode":"no_split"}` 用于正常解析但每个源文档只生成一个 Markdown 正文、飞书一次性用户 token 导入使用 `{"feishu_access_token":"u-..."}`,或飞书用户 token watch 使用 access/refresh token,并可选传入 `feishu_app_id` / `feishu_app_secret`), `acl`(可选) |
| `add_skill` | 新建、安装或替换 agent skill。新 skill 直接传完整 SKILL.md 文本;Git 与 GitHub tree URL 默认安装源里的全部 skill,可用 `skills` 挑选;本地 SKILL.md、目录或 zip 会和 `add_resource` 一样返回签名上传 URL | `data`(SKILL.md 文本)或 `path`(Git URL 或本地路径), `skills`(可选), `target_uri`(可选;`viking://agent/skills` 表示账户共享), `list_only`(可选) |
diff --git a/examples/langchain/src/langchain_openviking/store.py b/examples/langchain/src/langchain_openviking/store.py
index 34aa37d28..377e88339 100644
--- a/examples/langchain/src/langchain_openviking/store.py
+++ b/examples/langchain/src/langchain_openviking/store.py
@@ -309,21 +309,17 @@ class OpenVikingStore(BaseStore):
value: dict[str, Any],
now: datetime,
) -> dict[str, Any]:
- record = _store_record(namespace, key, value, created_at=now, updated_at=now)
- content = json.dumps(record, ensure_ascii=False, indent=2)
try:
- self._write_create(uri, content)
+ existing = self._read_record(uri)
+ except Exception:
+ record = _store_record(namespace, key, value, created_at=now, updated_at=now)
+ self._write_create(uri, json.dumps(record, ensure_ascii=False, indent=2))
return record
- except Exception as create_exc:
- try:
- existing = self._read_record(uri)
- except Exception:
- raise create_exc
- created_at = _parse_dt(existing["created_at"])
- record = _store_record(namespace, key, value, created_at=created_at, updated_at=now)
- self._write_replace(uri, json.dumps(record, ensure_ascii=False, indent=2))
- return record
+ created_at = _parse_dt(existing["created_at"])
+ record = _store_record(namespace, key, value, created_at=created_at, updated_at=now)
+ self._write_replace(uri, json.dumps(record, ensure_ascii=False, indent=2))
+ return record
def _write(self, uri: str, content: str) -> None:
try:
diff --git a/examples/langchain/tests/test_store.py b/examples/langchain/tests/test_store.py
new file mode 100644
index 000000000..47a0cea13
--- /dev/null
+++ b/examples/langchain/tests/test_store.py
@@ -0,0 +1,28 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+from datetime import datetime, timezone
+from unittest.mock import Mock
+
+from langchain_openviking.store import OpenVikingStore
+
+
+def test_write_record_preserves_created_at_when_create_is_an_upsert_alias():
+ store = object.__new__(OpenVikingStore)
+ created_at = "2026-01-01T00:00:00+00:00"
+ store._read_record = Mock(return_value={"created_at": created_at})
+ store._write_create = Mock()
+ store._write_replace = Mock()
+
+ record = store._write_record(
+ "viking://user/default/langgraph/data/ns/key.json",
+ ("ns",),
+ "key",
+ {"version": 2},
+ datetime(2026, 1, 2, tzinfo=timezone.utc),
+ )
+
+ assert record["created_at"] == created_at
+ assert record["updated_at"] == "2026-01-02T00:00:00+00:00"
+ store._write_create.assert_not_called()
+ store._write_replace.assert_called_once()
diff --git a/examples/skills/ov-resources/docs/filesystem.md b/examples/skills/ov-resources/docs/filesystem.md
index 72e3161a6..ebe3b7a59 100644
--- a/examples/skills/ov-resources/docs/filesystem.md
+++ b/examples/skills/ov-resources/docs/filesystem.md
@@ -75,11 +75,11 @@ ov write viking://resources/docs/api.md \
```
Modes:
-- `replace` (default): overwrite existing file
-- `append`: append to existing file
-- `create`: create new file (fails if exists, accepts `.md`, `.txt`, `.json`, `.yaml`, `.yml`, `.toml`, `.py`, `.js`, `.ts`)
+- `replace` (default): overwrite an existing file or create it when missing
+- `append`: append to an existing file or use the supplied content as a missing file's initial body
+- `create`: compatibility alias for `replace`
-`--wait` blocks until semantic/vector refresh completes. Parent directories are auto-created for `create`.
+New files accept `.md`, `.txt`, `.json`, `.yaml`, `.yml`, `.toml`, `.py`, `.js`, or `.ts` in every mode. `--wait` blocks until semantic/vector refresh completes. Parent directories are auto-created.
Derived semantic files cannot be written directly: `.abstract.md`, `.overview.md`.
diff --git a/openviking/server/mcp_endpoint.py b/openviking/server/mcp_endpoint.py
index 13f22ee69..aa21803d9 100644
--- a/openviking/server/mcp_endpoint.py
+++ b/openviking/server/mcp_endpoint.py
@@ -1206,9 +1206,9 @@ async def write(
"""Write text to a viking:// file. Use this for files you author yourself (notes, profiles, state), the same way you would use a working directory. To change part of an existing file, prefer the edit tool over a full rewrite. To store a file, document, URL, or repo the user gives you, use add_resource; for a skill, use add_skill. Do not copy its text into a file here instead.
- mode="replace" (default): overwrite the file; creates it and any missing parent directories if needed.
- - mode="create": fail if the file already exists.
- - Any new file (whether created by "replace" or "create") must end in one of: .md .txt .json .yaml .yml .toml .py .js .ts
- - mode="append": append to the end of an existing file; fails if the file does not exist.
+ - mode="create": compatibility alias for replace; overwrite an existing file or create it if missing.
+ - Any new file (whether created by "replace", "append", or "create") must end in one of: .md .txt .json .yaml .yml .toml .py .js .ts
+ - mode="append": append to the end of an existing file; use the supplied content as the initial body if it is missing.
- acl: optional shared-resource ACL with acl_mode (inherit/restricted) and entries
(principal=user:<id>/group:<id>/user:*, level=read/write/manage). Explicit ACL
requires manage on the target, or its parent for a new file. Omit to preserve
@@ -1220,19 +1220,9 @@ async def write(
ctx = _get_ctx()
uri = _resolve_mcp_workspace_uri(uri, ctx)
- try:
- result = await service.fs.write(
- uri=uri, content=content, ctx=ctx, mode=mode, wait=wait, timeout=timeout, acl=acl
- )
- except NotFoundError:
- if mode != "replace":
- raise
- # Replace doubles as create-or-overwrite so agents can save a new file
- # without first checking whether it exists; strict creation stays
- # available via mode="create".
- result = await service.fs.write(
- uri=uri, content=content, ctx=ctx, mode="create", wait=wait, timeout=timeout, acl=acl
- )
+ result = await service.fs.write(
+ uri=uri, content=content, ctx=ctx, mode=mode, wait=wait, timeout=timeout, acl=acl
+ )
written = result.get("written_bytes", 0)
message = (
f"Wrote {written} bytes to {result.get('uri', uri)} (mode={result.get('mode', mode)})."
diff --git a/openviking/server/routers/content.py b/openviking/server/routers/content.py
index 69c8d92ed..cacf55da0 100644
--- a/openviking/server/routers/content.py
+++ b/openviking/server/routers/content.py
@@ -62,6 +62,9 @@ class BatchWriteOperation(BaseModel):
mode: Literal["replace", "append", "create", "upsert"] = "replace"
# Optional byte revision, checked under the existing file locks before any writes.
expected_sha256: str | None = Field(default=None, pattern=r"^[a-f0-9]{64}$")
+ tags: list[str] | None = None
+ tag_mode: Literal["replace", "append", "clear"] = "replace"
+ acl: AclSpec | None = None
@model_validator(mode="after")
def validate_content_shape(self) -> "BatchWriteOperation":
@@ -80,6 +83,7 @@ class BatchWriteRequest(BaseModel):
telemetry: TelemetryRequest = False
# Preserve conflicting target files and return their URI/reason while writing other files.
skip_conflicts: bool = False
+ processing_mode: ProcessingMode = DEFAULT_PROCESSING_MODE
class SetTagsRequest(BaseModel):
@@ -286,6 +290,7 @@ async def batch_write(
wait=request.wait,
timeout=request.timeout,
skip_conflicts=request.skip_conflicts,
+ processing_mode=request.processing_mode,
),
)
return Response(
diff --git a/openviking/service/fs_service.py b/openviking/service/fs_service.py
index ee32386a7..72888bd8b 100644
--- a/openviking/service/fs_service.py
+++ b/openviking/service/fs_service.py
@@ -1225,7 +1225,7 @@ class FSService:
tag_mode: str = "replace",
acl: AclSpec | Dict[str, Any] | None = None,
) -> Dict[str, Any]:
- """Write to an existing file and refresh semantics/vectors."""
+ """Create or update a file and refresh semantics/vectors."""
viking_fs = self._ensure_initialized()
coordinator = ContentWriteCoordinator(viking_fs=viking_fs, vikingdb=self._vikingdb)
return await coordinator.write(
@@ -1250,6 +1250,7 @@ class FSService:
wait: bool = True,
timeout: Optional[float] = None,
skip_conflicts: bool = False,
+ processing_mode: str = "semantic_and_vectors",
) -> Dict[str, Any]:
"""Apply multiple file writes and aggregate downstream refresh."""
viking_fs = self._ensure_initialized()
@@ -1261,6 +1262,7 @@ class FSService:
wait=wait,
timeout=timeout,
skip_conflicts=skip_conflicts,
+ processing_mode=processing_mode,
)
async def set_tags(
diff --git a/openviking/storage/content_write.py b/openviking/storage/content_write.py
index 7f97cc7d7..362c73e58 100644
--- a/openviking/storage/content_write.py
+++ b/openviking/storage/content_write.py
@@ -9,9 +9,10 @@ import binascii
import hashlib
import os
from collections import defaultdict
-from dataclasses import dataclass, replace
+from dataclasses import dataclass, field, replace
from typing import Any, Dict, Optional
+from openviking.core.context import ContextLevel
from openviking.core.namespace import (
classify_uri,
context_type_for_uri,
@@ -20,7 +21,6 @@ from openviking.core.namespace import (
)
from openviking.resource.processing_mode import (
DEFAULT_PROCESSING_MODE,
- VECTORS_ONLY,
ProcessingMode,
normalize_processing_mode,
)
@@ -39,25 +39,51 @@ from openviking.storage.abstract_overview import (
prepare_abstract_overview_write,
)
from openviking.storage.acl import AclAction, AclSpec
+from openviking.storage.context_update_execution import (
+ commit_and_enqueue_plan,
+ enqueue_direct_index_actions,
+)
+from openviking.storage.context_update_plan import (
+ ContentTreeAction,
+ ContentTreeOperation,
+ ContextUpdatePlan,
+ DirectIndexAction,
+ FileRefreshIntent,
+ build_context_update_plan_from_snapshot,
+ execute_content_tree_actions,
+)
from openviking.storage.errors import LockAcquisitionError, ResourceBusyError
+from openviking.storage.index_action import FieldPatch, IndexAction
from openviking.storage.internal_names import is_storage_internal_name
from openviking.storage.queuefs import SemanticMsg, get_queue_manager
from openviking.storage.queuefs.semantic_msg import build_semantic_coalesce_key
from openviking.storage.queuefs.semantic_ops.freshness_policy import FreshnessAction
+from openviking.storage.resource_diff import (
+ ContentState,
+ IndexState,
+ InlineBytesStore,
+ build_rnfv_snapshot,
+ make_inline_file_inventory,
+ resolve_resource_diff,
+)
+from openviking.storage.resource_rnfv import (
+ FormalEntry,
+ RequestIntent,
+ canonical_vector_records_by_level,
+)
+from openviking.storage.resource_target import AgfsResourceTarget
+from openviking.storage.vector_ids import vector_record_id
from openviking.storage.viking_fs import VikingFS
from openviking.telemetry import get_current_telemetry
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
from openviking.telemetry.resource_summary import build_queue_status_payload
from openviking.utils.content_hash import content_md5
-from openviking.utils.embedding_utils import vectorize_directory_meta, vectorize_file
+from openviking.utils.embedding_utils import vectorize_directory_meta
from openviking.utils.ingest_options import IngestOptions
-from openviking.utils.path_safety import (
- normalize_storage_target_uri,
- validate_safe_viking_uri_path,
-)
+from openviking.utils.path_safety import normalize_storage_target_uri, validate_safe_viking_uri_path
+from openviking.utils.summarizer import FileRefreshRequest, Summarizer
from openviking.utils.tags import normalize_search_tags
from openviking_cli.exceptions import (
- AlreadyExistsError,
ConflictError,
DeadlineExceededError,
InvalidArgumentError,
@@ -90,6 +116,58 @@ _CREATE_ALLOWED_EXTENSIONS = frozenset(
_USER_MANAGED_SUBTREES = frozenset({"skills", "peers", "privacy", "sessions"})
+@dataclass(frozen=True)
+class _InlineArtifactRef:
+ """Opaque artifact handle for the inline single-file RNFV path.
+
+ The RNFV plan pipeline only needs an object to pass back to
+ ``InlineBytesStore.read_bytes``; the write path holds the bytes directly, so
+ no real parse-output backend is involved.
+ """
+
+ uri: str
+ backend: str = "inline"
+
+
+@dataclass(frozen=True)
+class _LockedWriteTarget:
+ """One locked target classification shared by all single-file write paths."""
+
+ preexisting: bool
+ formal_snapshot: tuple[dict[str, FormalEntry], bool]
+
+ @property
+ def is_new_file(self) -> bool:
+ return not self.preexisting
+
+
+@dataclass(frozen=True)
+class _PreparedInlineResource:
+ """One file-scoped RNFV plan compiled from final bytes and locked facts."""
+
+ uri: str
+ existed: bool
+ final_bytes: bytes
+ plan: ContextUpdatePlan
+ store: InlineBytesStore
+ artifact_ref: _InlineArtifactRef
+ target: AgfsResourceTarget
+ previous_abstract: str = ""
+ ingest_options: IngestOptions = field(default_factory=IngestOptions)
+
+
+def _queue_has_errors(queue_status: Optional[Dict[str, Any]], name: str) -> bool:
+ if not isinstance(queue_status, dict):
+ return False
+ status = queue_status.get(name, {})
+ if not isinstance(status, dict):
+ return False
+ try:
+ return int(status.get("error_count", 0) or 0) > 0
+ except (TypeError, ValueError):
+ return bool(status.get("errors"))
+
+
@dataclass(frozen=True)
class _BatchRefreshOutcome:
queue_status: Optional[Dict[str, Any]]
@@ -98,14 +176,13 @@ class _BatchRefreshOutcome:
embedding_requested: bool = False
def statuses(self, *, wait: bool) -> tuple[str, str]:
- if FreshnessAction.NOOP in self.semantic_actions:
- semantic_status = "skipped"
- elif self.semantic_actions:
- semantic_status = (
- "deferred"
- if all(action is FreshnessAction.MARK_PENDING for action in self.semantic_actions)
- else ("complete" if wait else "queued")
- )
+ if self.semantic_actions:
+ if any(action is FreshnessAction.REFRESH_NOW for action in self.semantic_actions):
+ semantic_status = "complete" if wait else "queued"
+ elif any(action is FreshnessAction.MARK_PENDING for action in self.semantic_actions):
+ semantic_status = "deferred"
+ else:
+ semantic_status = "skipped"
else:
semantic_status = "complete" if self.memory_refreshed else "skipped"
@@ -149,40 +226,22 @@ class ContentWriteCoordinator:
acl_update=await self._viking_fs.prepare_acl_update(normalized_uri, acl, ctx),
)
+ # ``create`` is an upsert alias for ``replace``: it overwrites an existing
+ # file or materializes a missing one, never conflicting. Normalize it away
+ # so no caller-visible create mode enters the write pipeline;
+ # ``response_mode`` still echoes the request for API stability.
+ response_mode = mode
if mode == "create":
- return await self._create_and_write(
- uri=normalized_uri,
- content=content,
- ctx=ctx,
- wait=wait,
- timeout=timeout,
- processing_mode=processing_mode,
- ingest_options=ingest_options,
- )
-
- stat = await self._safe_stat(normalized_uri, ctx=ctx, allow_not_found=True)
- if stat.get("not_found"):
- # replace and append are idempotent writes: a missing target starts
- # from an empty file while retaining the caller's requested mode.
- return await self._create_and_write(
- uri=normalized_uri,
- content=content,
- ctx=ctx,
- wait=wait,
- timeout=timeout,
- processing_mode=processing_mode,
- result_mode=mode,
- validate_extension=False,
- ingest_options=ingest_options,
- )
- if stat.get("isDir"):
- raise InvalidArgumentError(
- f"write only supports existing files, got directory: {normalized_uri}"
- )
+ mode = "replace"
context_type = context_type_for_uri(normalized_uri)
- root_uri = await self._resolve_root_uri(normalized_uri, ctx=ctx, anchor_to_parent=True)
- written_bytes = len(content.encode("utf-8"))
+ root_uri = await self._resolve_root_uri(
+ normalized_uri,
+ ctx=ctx,
+ _allow_not_found=True,
+ anchor_to_parent=True,
+ validate_storage=False,
+ )
telemetry_id = get_current_telemetry().telemetry_id
if context_type == "memory" and not is_abstract_overview_uri(normalized_uri):
@@ -191,12 +250,11 @@ class ContentWriteCoordinator:
root_uri=root_uri,
content=content,
mode=mode,
+ response_mode=response_mode,
wait=wait,
timeout=timeout,
ctx=ctx,
- written_bytes=written_bytes,
telemetry_id=telemetry_id,
- processing_mode=processing_mode,
ingest_options=ingest_options,
)
@@ -205,11 +263,11 @@ class ContentWriteCoordinator:
root_uri=root_uri,
content=content,
mode=mode,
+ response_mode=response_mode,
context_type=context_type,
wait=wait,
timeout=timeout,
ctx=ctx,
- written_bytes=written_bytes,
telemetry_id=telemetry_id,
processing_mode=processing_mode,
ingest_options=ingest_options,
@@ -224,6 +282,7 @@ class ContentWriteCoordinator:
wait: bool = True,
timeout: Optional[float] = None,
skip_conflicts: bool = False,
+ processing_mode: ProcessingMode = DEFAULT_PROCESSING_MODE,
) -> Dict[str, Any]:
"""Write a bundle under one directory, then refresh it as a batch.
@@ -238,6 +297,7 @@ class ContentWriteCoordinator:
Permission, transport and write failures remain errors, not skipped conflicts.
"""
normalized_root = self._validate_uri_path(root_uri, field_name="root_uri")
+ processing_mode = normalize_processing_mode(processing_mode)
await self._validate_batch_root(normalized_root, ctx=ctx)
normalized_operations = await self._normalize_batch_operations(
normalized_root, operations, ctx=ctx
@@ -261,9 +321,11 @@ class ContentWriteCoordinator:
refresh_kinds: dict[str, str] = {}
file_md5s: dict[str, str] = {}
file_abstracts: dict[str, str] = {}
- sidecar_directories: set[str] = set()
+ sidecar_directories: dict[str, IngestOptions] = {}
pending: list[tuple[dict[str, Any], bool, str]] = []
conflicts: list[dict[str, str]] = []
+ prepared_resources: dict[str, _PreparedInlineResource] = {}
+ committed_resource_plans: list[_PreparedInlineResource] = []
write_error: Exception | None = None
lock_released = False
try:
@@ -275,9 +337,11 @@ class ContentWriteCoordinator:
raise InvalidArgumentError(f"batch-write target must be a file: {uri}")
requested_mode = operation["mode"]
- write_mode = requested_mode
- if write_mode == "upsert":
- write_mode = "replace" if exists else "create"
+ # Batch-write follows the same materialization contract as a
+ # single explicit write: create/replace/upsert all establish
+ # the requested final body, while append only needs the old
+ # body when it already exists.
+ write_mode = "append" if requested_mode == "append" and exists else "replace"
if is_abstract_overview_uri(uri):
if not exists:
@@ -287,10 +351,6 @@ class ContentWriteCoordinator:
try:
if exists and stat.get("isDir"):
raise ConflictError(f"Target is a directory: {uri}")
- if write_mode == "create" and exists:
- raise AlreadyExistsError(uri, "file")
- if write_mode in {"replace", "append"} and not exists:
- raise NotFoundError(uri, "file")
expected = operation.get("expected_sha256")
if expected is not None:
if not exists:
@@ -301,16 +361,27 @@ class ContentWriteCoordinator:
desired = operation["content"]
if isinstance(desired, str):
desired = desired.encode("utf-8")
- if skip_conflicts and write_mode == "replace" and current == desired:
+ if (
+ skip_conflicts
+ and write_mode == "replace"
+ and current == desired
+ and operation["ingest_options"] == IngestOptions()
+ ):
unchanged.append(uri)
continue
- except (AlreadyExistsError, NotFoundError, ConflictError) as exc:
+ except ConflictError as exc:
if not skip_conflicts:
raise
conflicts.append({"uri": uri, "code": exc.code, "message": str(exc)})
continue
pending.append((operation, exists, write_mode))
+ prepared_resources = await self._prepare_batch_resource_plans(
+ pending=pending,
+ ctx=ctx,
+ lease=lease,
+ processing_mode=processing_mode,
+ )
file_abstracts = await self._load_file_abstracts(
[
operation["uri"]
@@ -318,6 +389,7 @@ class ContentWriteCoordinator:
if existed
and context_type_for_uri(operation["uri"]) in {"resource", "skill"}
and not is_abstract_overview_uri(operation["uri"])
+ and operation["uri"] not in prepared_resources
],
ctx=ctx,
)
@@ -325,18 +397,35 @@ class ContentWriteCoordinator:
for operation, existed, write_mode in pending:
uri = operation["uri"]
try:
- final_content = await self._write_in_place(
- uri,
- operation["content"],
- mode=write_mode,
- ctx=ctx,
- lease_ref=lease,
- )
+ prepared = prepared_resources.get(uri)
+ if prepared is not None:
+ await execute_content_tree_actions(
+ prepared.plan.content_tree_actions,
+ store=prepared.store,
+ artifact_ref=prepared.artifact_ref,
+ target=prepared.target,
+ )
+ final_content = prepared.final_bytes
+ committed_resource_plans.append(prepared)
+ else:
+ final_content = await self._write_in_place(
+ uri,
+ operation["content"],
+ mode=write_mode,
+ ctx=ctx,
+ lease_ref=lease,
+ is_new_file=not existed,
+ )
if context_type_for_uri(uri) in {
"resource",
"skill",
} and not is_abstract_overview_uri(uri):
file_md5s[uri] = content_md5(final_content)
+ options = operation["ingest_options"]
+ if options.acl_update:
+ await self._viking_fs.acl_manager.apply_indexed_update(
+ options.acl_update, ctx
+ )
except Exception as exc:
write_error = exc
break
@@ -345,12 +434,17 @@ class ContentWriteCoordinator:
if is_abstract_overview_uri(uri):
parent = VikingURI(uri).parent
if parent is not None:
- sidecar_directories.add(parent.uri)
- else:
+ # Sidecars represent directory L0/L1 records. If
+ # multiple sidecar writes share a directory, the
+ # deterministic operation order matches sequential
+ # write semantics: the last operation's options win.
+ sidecar_directories[parent.uri] = options
+ elif prepared is None:
refresh_kinds[uri] = "modified"
else:
created.append(uri)
- refresh_kinds[uri] = "added"
+ if prepared is None:
+ refresh_kinds[uri] = "added"
finally:
await self._viking_fs._async_agfs.pathlock_release(lease)
lock_released = True
@@ -360,39 +454,96 @@ class ContentWriteCoordinator:
request_registered = False
refresh_outcome: Optional[_BatchRefreshOutcome] = None
try:
- if refresh_kinds or sidecar_directories:
+ if refresh_kinds or sidecar_directories or committed_resource_plans:
if wait and telemetry_id:
get_request_wait_tracker().register_request(telemetry_id)
request_registered = True
try:
- for directory_uri in sorted(sidecar_directories):
+ for directory_uri, options in sorted(sidecar_directories.items()):
await self._vectorize_semantic_directory(
- directory_uri=directory_uri, ctx=ctx
+ directory_uri=directory_uri,
+ ctx=ctx,
+ ingest_options=options,
+ )
+ direct_index_requested = False
+ for prepared in committed_resource_plans:
+ direct_index_requested = (
+ await enqueue_direct_index_actions(
+ prepared.plan.direct_index_actions,
+ ctx=ctx,
+ acl_update=prepared.ingest_options.acl_update,
+ )
+ or direct_index_requested
+ )
+ planned_refreshes = [
+ FileRefreshRequest(
+ file_uri=prepared.plan.file_refresh.file_uri,
+ created=not prepared.existed,
+ md5=prepared.plan.file_refresh.md5,
+ abstract=prepared.previous_abstract,
+ vector_action="upsert" if not prepared.existed else "",
+ ingest_options=prepared.ingest_options,
+ )
+ for prepared in committed_resource_plans
+ if prepared.plan.file_refresh is not None
+ ]
+ parent_result = (
+ await Summarizer(vlm_processor=None).refresh_file_parents(
+ changes=planned_refreshes,
+ ctx=ctx,
+ generation_trigger="content_write",
+ force_refresh=wait,
)
- refresh_result = (
+ if planned_refreshes
+ else {"enqueued_count": 0, "parent_actions": {}}
+ )
+ legacy_refresh_result = (
await self._refresh_batch(
refresh_kinds=refresh_kinds,
ctx=ctx,
- wait=wait,
+ # The batch coordinator owns the one request-wide
+ # wait after both legacy special paths and RNFV
+ # plans have enqueued their work.
+ wait=False,
timeout=timeout,
telemetry_id=telemetry_id,
file_md5s=file_md5s,
file_abstracts=file_abstracts,
+ file_ingest_options={
+ operation["uri"]: operation["ingest_options"]
+ for operation, _, _ in pending
+ },
)
if refresh_kinds
- else (
- await self._wait_for_request(telemetry_id=telemetry_id, timeout=timeout)
- if wait
- else None
- )
+ else None
+ )
+ # The production helper always returns _BatchRefreshOutcome;
+ # keep the empty outcome fallback for lightweight callers
+ # and test doubles that deliberately perform no refresh.
+ legacy_outcome = (
+ legacy_refresh_result
+ if isinstance(legacy_refresh_result, _BatchRefreshOutcome)
+ else _BatchRefreshOutcome(queue_status=None)
+ )
+ queue_status = (
+ await self._wait_for_request(telemetry_id=telemetry_id, timeout=timeout)
+ if wait and request_registered
+ else legacy_outcome.queue_status
+ )
+ if wait and queue_status is not None:
+ self._raise_refresh_errors(queue_status)
+ planned_actions = tuple(
+ FreshnessAction(action)
+ for action in parent_result["parent_actions"].values()
+ )
+ refresh_outcome = _BatchRefreshOutcome(
+ queue_status=queue_status,
+ semantic_actions=planned_actions + legacy_outcome.semantic_actions,
+ memory_refreshed=legacy_outcome.memory_refreshed,
+ embedding_requested=(
+ direct_index_requested or legacy_outcome.embedding_requested
+ ),
)
- if isinstance(refresh_result, _BatchRefreshOutcome):
- refresh_outcome = refresh_result
- queue_status = refresh_result.queue_status
- else:
- # Preserve compatibility for tests/extensions replacing
- # the private refresh hook with its historical result.
- queue_status = refresh_result
except Exception as exc:
if write_error is not None:
logger.error(
@@ -406,8 +557,9 @@ class ContentWriteCoordinator:
cause = str(exc).strip() or type(exc).__name__
raise OpenVikingError(
"Content is already at the requested state, but semantic/index "
- f"refresh failed: {cause}. Re-run the same batch-write or ov compile "
- "command to rewrite the final state and retry the refresh.",
+ f"refresh failed: {cause}. Re-run with the final body using "
+ "replace/create/upsert or use ov compile to retry the refresh; "
+ "do not blindly retry an already-successful append.",
code="REFRESH_FAILED",
details={
"root_uri": normalized_root,
@@ -546,12 +698,26 @@ class ContentWriteCoordinator:
self._validate_create_extension(uri)
if context_type == "memory" and mode == "append":
raise InvalidArgumentError("batch-write append is not supported for memories")
+ tags = raw.get("tags")
+ if tags is not None and (
+ not isinstance(tags, list) or any(not isinstance(tag, str) for tag in tags)
+ ):
+ raise InvalidArgumentError(f"batch-write tags must be a string list: {uri}")
+ tag_mode = raw.get("tag_mode", "replace")
+ self._validate_tag_mode(tag_mode)
+ ingest_options = IngestOptions.from_search_tags(tags, mode=tag_mode)
+ if (acl := raw.get("acl")) is not None:
+ ingest_options = replace(
+ ingest_options,
+ acl_update=await self._viking_fs.prepare_acl_update(uri, acl, ctx),
+ )
normalized.append(
{
"uri": uri,
"content": content,
"mode": mode,
"expected_sha256": raw.get("expected_sha256"),
+ "ingest_options": ingest_options,
}
)
return sorted(normalized, key=lambda operation: operation["uri"])
@@ -566,6 +732,7 @@ class ContentWriteCoordinator:
telemetry_id: str,
file_md5s: dict[str, str] | None = None,
file_abstracts: dict[str, str] | None = None,
+ file_ingest_options: dict[str, IngestOptions] | None = None,
) -> _BatchRefreshOutcome:
resource_groups: dict[tuple[str, str], dict[str, list[str]]] = defaultdict(
lambda: {"added": [], "modified": []}
@@ -600,6 +767,12 @@ class ContentWriteCoordinator:
for uri in values
if file_abstracts and uri in file_abstracts
},
+ file_ingest_options={
+ uri: options
+ for values in changes.values()
+ for uri in values
+ if file_ingest_options and (options := file_ingest_options.get(uri)) is not None
+ },
)
semantic_actions.append(action)
@@ -619,6 +792,7 @@ class ContentWriteCoordinator:
memory_type=MemoryUpdater.memory_type_from_uri(uri),
ctx=ctx,
strict=True,
+ ingest_options=(file_ingest_options or {}).get(uri),
)
embedding_requested = embedding_requested or requested
@@ -654,6 +828,7 @@ class ContentWriteCoordinator:
ingest_options: IngestOptions | None = None,
file_md5s: dict[str, str] | None = None,
file_abstracts: dict[str, str] | None = None,
+ file_ingest_options: dict[str, IngestOptions] | None = None,
) -> FreshnessAction:
changed_entries = len({uri for values in changes.values() for uri in values})
semantic_config = get_openviking_config().semantic
@@ -710,6 +885,7 @@ class ContentWriteCoordinator:
aggregate_directory=aggregate_directory,
file_md5s=file_md5s,
file_abstracts=file_abstracts,
+ file_ingest_options=file_ingest_options,
)
if msg.telemetry_id:
get_request_wait_tracker().register_semantic_root(msg.telemetry_id, msg.id)
@@ -793,17 +969,11 @@ class ContentWriteCoordinator:
context_type: str,
mode: str,
written_bytes: int,
- wait: bool,
queue_status: Optional[Dict[str, Any]],
- semantic_status: Optional[str] = None,
- vector_status: Optional[str] = None,
+ semantic_status: str,
+ vector_status: str,
overview_status: Optional[str] = None,
) -> Dict[str, Any]:
- if semantic_status is None or vector_status is None:
- semantic_status, vector_status = self._refresh_statuses(
- wait=wait,
- queue_status=queue_status,
- )
result = {
"uri": uri,
"root_uri": root_uri,
@@ -844,29 +1014,347 @@ class ContentWriteCoordinator:
"tags_updated": len(updated_uris) > 0,
}
- def _refresh_statuses(
- self,
+ @staticmethod
+ def _queue_work_status(
*,
+ requested: bool,
wait: bool,
queue_status: Optional[Dict[str, Any]],
- ) -> tuple[str, str]:
+ queue_name: str,
+ ) -> str:
+ if not requested:
+ return "skipped"
if not wait:
- return "queued", "queued"
- if not queue_status:
- return "complete", "complete"
-
- def _has_errors(name: str) -> bool:
- status = queue_status.get(name, {})
- if not isinstance(status, dict):
- return False
- try:
- return int(status.get("error_count", 0) or 0) > 0
- except (TypeError, ValueError):
- return bool(status.get("errors"))
+ return "queued"
+ return "failed" if _queue_has_errors(queue_status, queue_name) else "complete"
- semantic_status = "failed" if _has_errors("Semantic") else "complete"
- vector_status = "failed" if _has_errors("Embedding") else "complete"
- return semantic_status, vector_status
+ async def _prepare_batch_resource_plans(
+ self,
+ *,
+ pending: list[tuple[dict[str, Any], bool, str]],
+ ctx: RequestContext,
+ lease: Dict[str, Any],
+ processing_mode: ProcessingMode,
+ ) -> dict[str, _PreparedInlineResource]:
+ """Compile file-scoped RNFV plans before the first batch F mutation.
+
+ The legacy fallback is intentionally limited to an absent vector store,
+ which is useful for lightweight callers and old test doubles. Real
+ service batch writes always use the exact-URI hydration path below.
+ """
+ resource_pending = [
+ item
+ for item in pending
+ if context_type_for_uri(item[0]["uri"]) in {"resource", "skill"}
+ and not is_abstract_overview_uri(item[0]["uri"])
+ ]
+ if not resource_pending:
+ return {}
+
+ vector_store = self._vikingdb
+ if vector_store is None:
+ get_vector_store = getattr(self._viking_fs, "_get_vector_store", None)
+ vector_store = get_vector_store() if callable(get_vector_store) else None
+ if vector_store is None or not hasattr(vector_store, "get_incremental_inventory_by_uris"):
+ return {}
+
+ existing_uris = [operation["uri"] for operation, existed, _ in resource_pending if existed]
+ requests = {
+ operation["uri"]: RequestIntent.from_ingest_options(
+ target_uri=operation["uri"],
+ processing_mode=processing_mode,
+ ingest_options=operation["ingest_options"],
+ )
+ for operation, _, _ in resource_pending
+ }
+ projection = set()
+ for request in requests.values():
+ projection.update(request.required_vector_fields())
+ # File-scoped and vectors-only write planning may reuse a hydrated
+ # L2 abstract as an embedding source.
+ projection.add("abstract")
+ inventories = (
+ await vector_store.get_incremental_inventory_by_uris(
+ existing_uris,
+ ctx=ctx,
+ output_fields=sorted(projection),
+ )
+ if existing_uris
+ else {}
+ )
+
+ prepared: dict[str, _PreparedInlineResource] = {}
+ for operation, existed, write_mode in resource_pending:
+ uri = operation["uri"]
+ existing_raw: str | bytes | None = None
+ if write_mode == "append" and existed:
+ existing_raw = await self._viking_fs.read_file(uri, ctx=ctx)
+ final_bytes = self._render_final_bytes(
+ uri,
+ operation["content"],
+ mode=write_mode,
+ existing_raw=existing_raw,
+ is_new_file=not existed,
+ )
+ prepared[uri] = await self._prepare_inline_resource_plan(
+ uri=uri,
+ final_bytes=final_bytes,
+ target_preexisting=existed,
+ formal_snapshot=(
+ ({"": FormalEntry(is_dir=False)}, True) if existed else ({}, True)
+ ),
+ vector_inventory=inventories.get(uri, {}) if existed else {},
+ vector_store=vector_store,
+ request_intent=requests[uri],
+ ctx=ctx,
+ lease=lease,
+ ingest_options=operation["ingest_options"],
+ include_previous_abstract=True,
+ defer_vector_hydration=True,
+ )
+ return prepared
+
+ async def _prepare_inline_resource_plan(
+ self,
+ *,
+ uri: str,
+ final_bytes: bytes,
+ target_preexisting: bool,
+ formal_snapshot: tuple[dict[str, FormalEntry], bool],
+ vector_inventory: dict[str, dict[str, Any]] | None,
+ vector_store: Any,
+ request_intent: RequestIntent,
+ ctx: RequestContext,
+ lease: Dict[str, Any],
+ ingest_options: IngestOptions | None,
+ include_previous_abstract: bool = False,
+ defer_vector_hydration: bool = False,
+ ) -> _PreparedInlineResource:
+ """Build one file-scoped RNFV plan from already-locked, final bytes.
+
+ Both public write adapters own their own locking and final-byte render.
+ This helper only turns those immutable facts into the shared inline N/F/V
+ plan; it never reads target content or acquires/releases a lock.
+ """
+ store = InlineBytesStore(final_bytes)
+ artifact_ref = _InlineArtifactRef(uri)
+ inventory = make_inline_file_inventory(final_bytes)
+ target = AgfsResourceTarget(
+ viking_fs=self._viking_fs,
+ root_uri=uri,
+ ctx=ctx,
+ lease_ref=lease,
+ )
+ rnfv = await build_rnfv_snapshot(
+ viking_fs=self._viking_fs,
+ vikingdb=vector_store,
+ store=store,
+ artifact_ref=artifact_ref,
+ target_uri=uri,
+ ctx=ctx,
+ request_intent=request_intent,
+ root_is_file=True,
+ target_preexisting=target_preexisting,
+ formal_snapshot=formal_snapshot,
+ artifact_inventory=inventory,
+ vector_scope="self",
+ # Explicit write treats missing F as authoritative new state; an
+ # orphan canonical V record is overwritten by the later UPSERT.
+ vector_inventory=vector_inventory,
+ )
+ if defer_vector_hydration:
+ plan = await self._build_deferred_batch_file_plan(
+ rnfv=rnfv,
+ store=store,
+ artifact_ref=artifact_ref,
+ target=target,
+ inventory=inventory,
+ request_intent=request_intent,
+ ctx=ctx,
+ target_preexisting=target_preexisting,
+ )
+ else:
+ _, plan = await build_context_update_plan_from_snapshot(
+ snapshot=rnfv,
+ store=store,
+ artifact_ref=artifact_ref,
+ target=target,
+ vikingdb=vector_store,
+ context_type=context_type_for_uri(uri),
+ is_code_repo=False,
+ account_id=ctx.account_id,
+ ctx=ctx,
+ root_preexisting=target_preexisting,
+ artifact_paths=inventory.artifact_paths,
+ ingest_options=ingest_options,
+ root_is_file=True,
+ )
+ plan = self._ensure_explicit_write_content_action(plan, md5=inventory.entries[""].md5)
+ previous_abstract = (
+ next(
+ (
+ str(record.fields.get("abstract") or "")
+ for record in rnfv.vectors.records_by_id.values()
+ if record.uri == uri and record.level == 2
+ ),
+ "",
+ )
+ if include_previous_abstract
+ else ""
+ )
+ return _PreparedInlineResource(
+ uri=uri,
+ existed=target_preexisting,
+ final_bytes=final_bytes,
+ plan=plan,
+ store=store,
+ artifact_ref=artifact_ref,
+ target=target,
+ previous_abstract=previous_abstract,
+ ingest_options=IngestOptions.from_value(ingest_options),
+ )
+
+ @staticmethod
+ def _file_scalar_patch(request: RequestIntent) -> FieldPatch | None:
+ values: dict[str, Any] = {}
+ modes: dict[str, str] = {}
+ for intent in request.scalar_intents:
+ if int(ContextLevel.DETAIL) not in intent.target_levels:
+ continue
+ values[intent.field] = (
+ list(intent.value) if isinstance(intent.value, tuple) else intent.value
+ )
+ modes[intent.field] = intent.mode
+ return FieldPatch(values, modes) if values else None
+
+ async def _build_deferred_batch_file_plan(
+ self,
+ *,
+ rnfv: Any,
+ store: InlineBytesStore,
+ artifact_ref: _InlineArtifactRef,
+ target: AgfsResourceTarget,
+ inventory: Any,
+ request_intent: RequestIntent,
+ ctx: RequestContext,
+ target_preexisting: bool,
+ ) -> ContextUpdatePlan:
+ """Build a batch file plan without synchronously hydrating V scalars.
+
+ Batch keeps exact file locks only for F state and content mutation. An
+ existing L2 uses the durable embedding queue's MERGE action, whose
+ worker exact-gets the current record before resolving FieldPatch. This
+ avoids N per-file remote hydrations while preserving latest-state scalar
+ merge semantics.
+ """
+ diff = await resolve_resource_diff(
+ rnfv,
+ store=store,
+ artifact_ref=artifact_ref,
+ target=target,
+ artifact_paths=inventory.artifact_paths,
+ )
+ entry = diff.entries[""]
+ records = tuple(rnfv.vectors.records_by_id.values())
+ canonical, duplicates = canonical_vector_records_by_level(records)
+ detail = canonical.get(int(ContextLevel.DETAIL))
+ scalar_patch = self._file_scalar_patch(request_intent)
+ direct_actions: list[DirectIndexAction] = [
+ DirectIndexAction(IndexAction.DELETE, record.uri, record.level, record.record_id)
+ for record in duplicates
+ ]
+ for level, record in canonical.items():
+ if level != int(ContextLevel.DETAIL):
+ direct_actions.append(
+ DirectIndexAction(IndexAction.DELETE, record.uri, level, record.record_id)
+ )
+
+ content_changed = ContentState(entry.content_state) in {
+ ContentState.ADDED,
+ ContentState.RESTORE,
+ ContentState.MODIFIED,
+ ContentState.REPLACE_KIND,
+ }
+ index_needs_vector = IndexState(entry.index_state) in {
+ IndexState.MISSING,
+ IndexState.PARTIAL,
+ IndexState.STALE,
+ IndexState.LEVEL_CONFLICT,
+ }
+ needs_vector = request_intent.vectorize and (
+ content_changed or detail is None or index_needs_vector
+ )
+ has_detail_abstract = detail is not None and bool(
+ str(detail.fields.get("abstract") or "").strip()
+ )
+ needs_file_refresh = (
+ request_intent.processing_mode != "vectors_only"
+ and request_intent.vectorize
+ and (needs_vector or not has_detail_abstract)
+ )
+ if request_intent.processing_mode == "vectors_only" and needs_vector:
+ action = IndexAction.MERGE if detail is not None else IndexAction.UPSERT
+ direct_actions.append(
+ DirectIndexAction(
+ action=action,
+ uri=request_intent.target_uri,
+ level=int(ContextLevel.DETAIL),
+ record_id=(
+ detail.record_id
+ if detail is not None
+ else vector_record_id(
+ ctx.account_id,
+ request_intent.target_uri,
+ int(ContextLevel.DETAIL),
+ )
+ ),
+ upsert_fields=(
+ scalar_patch.resolve({})
+ if action is IndexAction.UPSERT and scalar_patch
+ else {}
+ ),
+ field_patch=scalar_patch if action is IndexAction.MERGE else None,
+ md5=entry.md5,
+ summary=str(detail.fields.get("abstract") or "") if detail else "",
+ )
+ )
+ elif scalar_patch is not None and detail is not None and not needs_vector:
+ direct_actions.append(
+ DirectIndexAction(
+ IndexAction.UPDATE_FIELDS,
+ detail.uri,
+ int(ContextLevel.DETAIL),
+ detail.record_id,
+ field_patch=scalar_patch.with_seed(
+ {
+ "uri": detail.uri,
+ "account_id": ctx.account_id,
+ "level": int(ContextLevel.DETAIL),
+ }
+ ),
+ )
+ )
+
+ return ContextUpdatePlan(
+ root_uri=request_intent.target_uri,
+ context_type=context_type_for_uri(request_intent.target_uri),
+ content_tree_actions=(
+ ContentTreeAction(
+ ContentTreeOperation.UPSERT,
+ "",
+ old_kind="file" if target_preexisting else None,
+ new_kind="file",
+ artifact_path="",
+ md5=inventory.entries[""].md5,
+ ),
+ ),
+ direct_index_actions=tuple(direct_actions),
+ file_refresh=(
+ FileRefreshIntent(request_intent.target_uri, entry.md5)
+ if needs_file_refresh
+ else None
+ ),
+ )
async def _write_direct_with_refresh(
self,
@@ -875,16 +1363,33 @@ class ContentWriteCoordinator:
root_uri: str,
content: str,
mode: str,
- response_mode: Optional[str] = None,
+ response_mode: str,
context_type: str,
wait: bool,
timeout: Optional[float],
ctx: RequestContext,
- written_bytes: int,
telemetry_id: str,
processing_mode: ProcessingMode = DEFAULT_PROCESSING_MODE,
ingest_options: IngestOptions | None = None,
) -> Dict[str, Any]:
+ # ``.abstract.md`` / ``.overview.md`` are generated sidecars whose body is
+ # re-embedded in place (no LLM regeneration and no RNFV plan). Keep that
+ # specialized path; everything else goes through the shared RNFV pipeline.
+ if is_abstract_overview_uri(uri):
+ return await self._write_abstract_overview_with_refresh(
+ uri=uri,
+ root_uri=root_uri,
+ content=content,
+ mode=mode,
+ response_mode=response_mode,
+ context_type=context_type,
+ wait=wait,
+ timeout=timeout,
+ ctx=ctx,
+ telemetry_id=telemetry_id,
+ ingest_options=ingest_options,
+ )
+
lock_path = self._viking_fs._uri_to_path(uri, ctx=ctx)
try:
lease = await self._viking_fs._async_agfs.pathlock_acquire_exact(lock_path)
@@ -895,181 +1400,231 @@ class ContentWriteCoordinator:
) from exc
previous_content: Optional[str] = None
- final_content = content.encode("utf-8")
- file_abstract = ""
- content_written = False
- post_process_started = False
lock_released = False
- vector_enqueued = False
- refresh_action: Optional[FreshnessAction] = None
+ request_registered = False
try:
- if mode != "create":
+ target_state = await self._classify_locked_write_target(uri, ctx=ctx)
+ target_preexisting = target_state.preexisting
+ is_new_file = target_state.is_new_file
+ if mode == "append" and target_preexisting:
previous_content = await self._viking_fs.read_file(uri, ctx=ctx)
- elif is_abstract_overview_uri(uri):
- raise InvalidArgumentError(
- f"cannot create generated abstract overview directly: {uri}"
- )
- if (
- mode != "create"
- and not is_abstract_overview_uri(uri)
- and processing_mode != VECTORS_ONLY
- ):
- file_abstract = (await self._load_file_abstracts([uri], ctx=ctx)).get(uri, "")
- if wait and telemetry_id:
- get_request_wait_tracker().register_request(telemetry_id)
- written_content = await self._write_in_place(
+ final_bytes = self._render_final_bytes(
uri,
content,
mode=mode,
- ctx=ctx,
- lease_ref=lease,
existing_raw=previous_content,
+ is_new_file=is_new_file,
)
- if written_content is not None:
- final_content = written_content
- content_written = True
- if is_abstract_overview_uri(uri):
- vector_enqueued = await self._vectorize_abstract_overview(
- uri=uri, ctx=ctx, ingest_options=ingest_options
- )
- post_process_started = True
- elif processing_mode == VECTORS_ONLY:
- vector_enqueued = await self._vectorize_written_file(
- uri=uri,
- context_type=context_type,
- ctx=ctx,
- ingest_options=ingest_options,
- file_md5=content_md5(final_content),
- )
- post_process_started = True
- else:
- refresh_action = await self._enqueue_semantic_refresh(
- root_uri=root_uri,
- changed_uri=uri,
- context_type=context_type,
- ctx=ctx,
- change_type="added" if mode == "create" else "modified",
- force_refresh=wait,
- ingest_options=ingest_options,
- file_md5=content_md5(final_content),
- file_abstract=file_abstract,
- )
- post_process_started = True
- if ingest_options and ingest_options.acl_update:
+ request = RequestIntent.from_ingest_options(
+ target_uri=uri,
+ processing_mode=processing_mode,
+ ingest_options=ingest_options,
+ )
+ prepared = await self._prepare_inline_resource_plan(
+ uri=uri,
+ final_bytes=final_bytes,
+ target_preexisting=target_preexisting,
+ formal_snapshot=target_state.formal_snapshot,
+ # This write owns the complete new-file state. Do not read or
+ # inherit an orphan V record when locked F is absent.
+ vector_inventory={} if not target_preexisting else None,
+ vector_store=self._vikingdb,
+ request_intent=request,
+ ctx=ctx,
+ lease=lease,
+ ingest_options=ingest_options,
+ )
+
+ if wait and telemetry_id:
+ get_request_wait_tracker().register_request(telemetry_id)
+ request_registered = True
+
+ work = await commit_and_enqueue_plan(
+ prepared.plan,
+ ctx=ctx,
+ inline_store=prepared.store,
+ inline_ref=prepared.artifact_ref,
+ target=prepared.target,
+ ingest_options=ingest_options,
+ file_created=not target_preexisting,
+ # Synchronous writes force parent aggregation; asynchronous writes
+ # defer it through the freshness gate (parity with the previous
+ # ``force_refresh=wait`` behavior).
+ force_refresh=wait,
+ generation_trigger="content_write",
+ )
+ if ingest_options.acl_update:
await self._viking_fs.acl_manager.apply_indexed_update(
ingest_options.acl_update, ctx
)
+
await self._viking_fs._async_agfs.pathlock_release(lease)
lock_released = True
+
queue_status = (
await self._wait_for_request(telemetry_id=telemetry_id, timeout=timeout)
if wait
else None
)
- result_kwargs = {}
- if is_abstract_overview_uri(uri):
- _, vector_status = (
- self._refresh_statuses(wait=wait, queue_status=queue_status)
- if vector_enqueued
- else ("skipped", "skipped")
- )
- result_kwargs = {
- "semantic_status": "skipped",
- "vector_status": vector_status,
- }
- elif processing_mode == VECTORS_ONLY:
- if vector_enqueued:
- _, vector_status = self._refresh_statuses(
- wait=wait,
- queue_status=queue_status,
- )
- else:
- vector_status = "skipped"
- result_kwargs = {
- "semantic_status": "skipped",
- "vector_status": vector_status,
- }
- elif refresh_action in {FreshnessAction.MARK_PENDING, FreshnessAction.NOOP}:
- # Changed-file semantic/vector work may still be queued, while
- # directory aggregation is deferred or skipped on contention.
- _, vector_status = self._refresh_statuses(wait=wait, queue_status=queue_status)
- result_kwargs = {
- "semantic_status": (
- "skipped" if refresh_action is FreshnessAction.NOOP else "deferred"
- ),
- "vector_status": vector_status,
- }
+ written_bytes = len(final_bytes)
+ semantic_status, vector_status = self._plan_statuses(
+ work, wait=wait, queue_status=queue_status
+ )
return self._build_write_result(
uri=uri,
root_uri=root_uri,
context_type=context_type,
- mode=response_mode or mode,
+ mode=response_mode,
written_bytes=written_bytes,
- wait=wait,
queue_status=queue_status,
- **result_kwargs,
+ semantic_status=semantic_status,
+ vector_status=vector_status,
)
except Exception:
- if not post_process_started and content_written:
- await self._rollback_direct_write(
- uri=uri,
- previous_content=previous_content,
- mode=mode,
- ctx=ctx,
- lease_ref=lease,
- )
+ # Once formal content is committed it is not rolled back for a later
+ # enqueue failure. The request propagates that failure so the caller
+ # can retry; explicit writes always recommit their final bytes, even
+ # when N already matches V, which makes that retry safe while V lags F.
if not lock_released:
await self._viking_fs._async_agfs.pathlock_release(lease)
raise
finally:
- if wait and telemetry_id:
+ if request_registered:
get_request_wait_tracker().cleanup(telemetry_id)
- async def _rollback_direct_write(
- self,
+ @staticmethod
+ def _ensure_explicit_write_content_action(
+ plan: ContextUpdatePlan,
*,
- uri: str,
- previous_content: Optional[str],
- mode: str,
- ctx: RequestContext,
- lease_ref: Optional[Dict[str, Any]] = None,
- ) -> None:
- try:
- if mode == "create":
- await self._viking_fs.rm(uri, ctx=ctx, lease_ref=lease_ref)
- return
- if previous_content is not None:
- await self._viking_fs.write_file(
- uri,
- previous_content,
- ctx=ctx,
- lease_ref=lease_ref,
- )
- except Exception:
- logger.error("Failed to rollback direct content write for %s", uri, exc_info=True)
+ md5: str,
+ ) -> ContextUpdatePlan:
+ """Make an explicit single-file write authoritative for formal content.
+
+ RNFV may classify N as unchanged when its MD5 matches V. That is enough
+ to skip derived work, but V can temporarily lag F while an earlier
+ asynchronous write is still being indexed. Keep the shared planner's
+ convergence behavior unchanged and add the formal UPSERT only at the
+ content-write adapter boundary.
+ """
+ if plan.content_tree_actions:
+ return plan
+ return replace(
+ plan,
+ content_tree_actions=(
+ ContentTreeAction(
+ ContentTreeOperation.UPSERT,
+ "",
+ old_kind="file",
+ new_kind="file",
+ artifact_path="",
+ md5=md5,
+ ),
+ ),
+ )
+
+ @staticmethod
+ def _plan_statuses(
+ work: Any,
+ *,
+ wait: bool,
+ queue_status: Optional[Dict[str, Any]],
+ ) -> tuple[str, str]:
+ """Map executed plan work + queue status to semantic/vector statuses."""
+ semantic_action = getattr(work, "semantic_action", None)
+ if semantic_action == FreshnessAction.MARK_PENDING.value:
+ semantic_status = "deferred"
+ elif semantic_action == FreshnessAction.NOOP.value:
+ semantic_status = "skipped"
+ elif not getattr(work, "semantic_requested", False):
+ semantic_status = "skipped"
+ elif not wait:
+ semantic_status = "queued"
+ elif _queue_has_errors(queue_status, "Semantic"):
+ semantic_status = "failed"
+ else:
+ semantic_status = "complete"
+
+ if not getattr(work, "vector_requested", False):
+ vector_status = "skipped"
+ elif not wait:
+ vector_status = "queued"
+ elif _queue_has_errors(queue_status, "Embedding"):
+ vector_status = "failed"
+ else:
+ vector_status = "complete"
+ return semantic_status, vector_status
- async def _vectorize_written_file(
+ async def _write_abstract_overview_with_refresh(
self,
*,
uri: str,
+ root_uri: str,
+ content: str,
+ mode: str,
+ response_mode: str,
context_type: str,
+ wait: bool,
+ timeout: Optional[float],
ctx: RequestContext,
+ telemetry_id: str,
ingest_options: IngestOptions | None = None,
- file_md5: str | None = None,
- ) -> bool:
- parent = VikingURI(uri).parent
- if parent is None:
- return False
- name = uri.rstrip("/").rsplit("/", 1)[-1]
- return await vectorize_file(
- file_path=uri,
- summary_dict={"name": name, "summary": ""},
- parent_uri=parent.uri,
- context_type=context_type,
- ctx=ctx,
- ingest_options=ingest_options,
- file_md5=file_md5,
- )
+ ) -> Dict[str, Any]:
+ """Re-embed a manually edited L0/L1 sidecar body without LLM regeneration."""
+ lock_path = self._viking_fs._uri_to_path(uri, ctx=ctx)
+ try:
+ lease = await self._viking_fs._async_agfs.pathlock_acquire_exact(lock_path)
+ except LockAcquisitionError as exc:
+ raise ResourceBusyError(
+ f"resource is busy and cannot be written now: {uri}",
+ uri=uri,
+ ) from exc
+
+ lock_released = False
+ request_registered = False
+ try:
+ # Preserve the public invariant that generated sidecars cannot be
+ # materialized, while keeping the state check inside the exact lock.
+ await self._classify_locked_write_target(uri, ctx=ctx)
+ previous_content = await self._viking_fs.read_file(uri, ctx=ctx)
+ await self._write_in_place(
+ uri, content, mode=mode, ctx=ctx, lease_ref=lease, existing_raw=previous_content
+ )
+ if wait and telemetry_id:
+ get_request_wait_tracker().register_request(telemetry_id)
+ request_registered = True
+ vector_enqueued = await self._vectorize_abstract_overview(
+ uri=uri, ctx=ctx, ingest_options=ingest_options
+ )
+ await self._viking_fs._async_agfs.pathlock_release(lease)
+ lock_released = True
+ queue_status = (
+ await self._wait_for_request(telemetry_id=telemetry_id, timeout=timeout)
+ if wait
+ else None
+ )
+ vector_status = self._queue_work_status(
+ requested=vector_enqueued,
+ wait=wait,
+ queue_status=queue_status,
+ queue_name="Embedding",
+ )
+ return self._build_write_result(
+ uri=uri,
+ root_uri=root_uri,
+ context_type=context_type,
+ mode=response_mode,
+ written_bytes=len(content.encode("utf-8")),
+ queue_status=queue_status,
+ semantic_status="skipped",
+ vector_status=vector_status,
+ )
+ except Exception:
+ if not lock_released:
+ await self._viking_fs._async_agfs.pathlock_release(lease)
+ raise
+ finally:
+ if request_registered:
+ get_request_wait_tracker().cleanup(telemetry_id)
async def _vectorize_abstract_overview(
self,
@@ -1137,8 +1692,6 @@ class ContentWriteCoordinator:
if name in _DERIVED_FILENAMES:
raise InvalidArgumentError(f"cannot write derived semantic file directly: {uri}")
if any(is_storage_internal_name(part) for part in uri_parts(uri)):
- # Ancestors must also be checked: creating parents could otherwise
- # create hidden directories using storage metadata names.
raise InvalidArgumentError(f"cannot write storage internal file directly: {uri}")
if is_watch_task_control_uri(uri):
raise InvalidArgumentError(f"cannot write watch task control file directly: {uri}")
@@ -1169,136 +1722,131 @@ class ContentWriteCoordinator:
raise NotFoundError(uri, "file") from exc
raise NotFoundError(uri, "file") from exc
- def _validate_create_extension(self, uri: str) -> None:
- _, ext = os.path.splitext(uri)
- if ext.lower() not in _CREATE_ALLOWED_EXTENSIONS:
- raise InvalidArgumentError(f"create mode does not allow extension '{ext}': {uri}")
-
- async def _create_and_write(
+ async def _classify_locked_write_target(
self,
- *,
uri: str,
- content: str,
+ *,
ctx: RequestContext,
- wait: bool,
- timeout: Optional[float],
- processing_mode: ProcessingMode,
- ingest_options: IngestOptions | None = None,
- result_mode: str = "create",
- validate_extension: bool = True,
- ) -> Dict[str, Any]:
+ ) -> _LockedWriteTarget:
+ """Read target state exactly once after its exact write lock is held."""
+ stat = await self._safe_stat(uri, ctx=ctx, allow_not_found=True)
+ preexisting = not stat.get("not_found")
+ if preexisting:
+ if stat.get("isDir"):
+ raise InvalidArgumentError(
+ f"write only supports existing files, got directory: {uri}"
+ )
+ return _LockedWriteTarget(True, ({"": FormalEntry(is_dir=False)}, True))
if is_abstract_overview_uri(uri):
raise InvalidArgumentError(f"cannot create generated abstract overview directly: {uri}")
- if validate_extension:
- self._validate_create_extension(uri)
-
- stat = await self._safe_stat(uri, ctx=ctx, allow_not_found=True)
- if not stat.get("not_found"):
- raise AlreadyExistsError(uri, "file")
-
- context_type = context_type_for_uri(uri)
- root_uri = await self._resolve_root_uri(
- uri, ctx=ctx, _allow_not_found=True, anchor_to_parent=True
- )
- written_bytes = len(content.encode("utf-8"))
- telemetry_id = get_current_telemetry().telemetry_id
+ self._validate_create_extension(uri)
+ return _LockedWriteTarget(False, ({}, True))
- if context_type == "memory":
- return await self._write_memory_with_refresh(
- uri=uri,
- root_uri=root_uri,
- content=content,
- mode="create",
- response_mode=result_mode,
- wait=wait,
- timeout=timeout,
- ctx=ctx,
- written_bytes=written_bytes,
- telemetry_id=telemetry_id,
- processing_mode=processing_mode,
- ingest_options=ingest_options,
+ def _validate_create_extension(self, uri: str) -> None:
+ _, ext = os.path.splitext(uri)
+ if ext.lower() not in _CREATE_ALLOWED_EXTENSIONS:
+ raise InvalidArgumentError(
+ f"creating a new file does not allow extension '{ext}': {uri}"
)
- return await self._write_direct_with_refresh(
- uri=uri,
- root_uri=root_uri,
- content=content,
- mode="create",
- response_mode=result_mode,
- context_type=context_type,
- wait=wait,
- timeout=timeout,
- ctx=ctx,
- written_bytes=written_bytes,
- telemetry_id=telemetry_id,
- processing_mode=processing_mode,
- ingest_options=ingest_options,
- )
-
- async def _write_in_place(
+ def _render_final_bytes(
self,
uri: str,
content: str | bytes,
*,
mode: str,
- ctx: RequestContext,
- lease_ref: Optional[Dict[str, Any]] = None,
- existing_raw: str | bytes | None = None,
+ existing_raw: str | bytes | None,
+ is_new_file: bool = False,
) -> bytes:
+ """Render the final on-disk bytes for a write without touching storage.
+
+ Splitting render from write lets the RNFV single-file path compute the
+ content md5 (its N snapshot) up front and lets ``execute_content_tree_actions``
+ commit the exact same bytes. Abstract-overview and memory renders keep
+ their specialized handling; ``existing_raw`` must be supplied for the
+ modes that need the prior content (append / memory replace-append /
+ abstract-overview merge).
+ """
if is_abstract_overview_uri(uri):
- current_raw = (
- existing_raw
- if existing_raw is not None
- else await self._viking_fs.read_file(uri, ctx=ctx)
- )
+ if existing_raw is None:
+ raise ValueError("abstract-overview render requires existing content")
rendered = self._prepare_abstract_overview_content(
uri=uri,
- current_raw=current_raw,
+ current_raw=existing_raw,
requested_raw=content,
mode=mode,
)
- await self._viking_fs.write_file(uri, rendered, ctx=ctx, lease_ref=lease_ref)
return rendered.encode("utf-8")
if context_type_for_uri(uri) == "memory":
- if mode == "replace":
- existing_raw = await self._viking_fs.read_file(uri, ctx=ctx)
+ if is_new_file:
+ mf = MemoryFileUtils.read(content, uri=uri)
+ elif mode == "replace":
mf = MemoryFileUtils.read(existing_raw, uri=uri)
mf.content = content
elif mode == "append":
- existing_raw = await self._viking_fs.read_file(uri, ctx=ctx)
mf = MemoryFileUtils.read(existing_raw, uri=uri)
mf.content = mf.content + content
else:
- mf = MemoryFileUtils.read(content, uri=uri)
+ raise InvalidArgumentError(f"unsupported memory write mode: {mode}")
sync_memory_resource_refs(mf, source=RESOURCE_REF_SOURCE_CONTENT_WRITE)
- rendered = MemoryFileUtils.write(mf)
- await self._viking_fs.write_file(
- uri,
- rendered,
- ctx=ctx,
- lease_ref=lease_ref,
- )
- return rendered.encode("utf-8")
+ return MemoryFileUtils.write(mf).encode("utf-8")
if mode == "append":
# Plain concatenation for resource/skill files: MEMORY_FIELDS is a
# reserved trailer of memory namespaces only (see content_visibility),
# so non-memory appends must not round-trip through MemoryFileUtils
# (which strips trailing newlines and injects a metadata trailer).
- existing_raw = (
- existing_raw
- if existing_raw is not None
- else await self._viking_fs.read_file(uri, ctx=ctx)
- )
+ if is_new_file:
+ if not isinstance(content, str):
+ raise InvalidArgumentError(f"append only supports text content: {uri}")
+ return content.encode("utf-8")
if not isinstance(existing_raw, str) or not isinstance(content, str):
raise InvalidArgumentError(f"append only supports text content: {uri}")
- final_content = existing_raw + content
- await self._viking_fs.write_file(uri, final_content, ctx=ctx, lease_ref=lease_ref)
- return final_content.encode("utf-8")
- await self._viking_fs.write_file(uri, content, ctx=ctx, lease_ref=lease_ref)
+ return (existing_raw + content).encode("utf-8")
+
return content if isinstance(content, bytes) else content.encode("utf-8")
+ async def _write_in_place(
+ self,
+ uri: str,
+ content: str | bytes,
+ *,
+ mode: str,
+ ctx: RequestContext,
+ lease_ref: Optional[Dict[str, Any]] = None,
+ existing_raw: str | bytes | None = None,
+ is_new_file: bool | None = None,
+ ) -> bytes:
+ if is_new_file is None:
+ # batch-write still uses its historical create mode internally; map it
+ # to the same explicit materialization state as single-file write.
+ is_new_file = mode == "create"
+ if is_abstract_overview_uri(uri) and existing_raw is None:
+ existing_raw = await self._viking_fs.read_file(uri, ctx=ctx)
+ elif (
+ context_type_for_uri(uri) == "memory"
+ and not is_new_file
+ and mode in {"replace", "append"}
+ ):
+ existing_raw = await self._viking_fs.read_file(uri, ctx=ctx)
+ elif (
+ context_type_for_uri(uri) != "memory"
+ and not is_abstract_overview_uri(uri)
+ and mode == "append"
+ and existing_raw is None
+ ):
+ existing_raw = await self._viking_fs.read_file(uri, ctx=ctx)
+ final_bytes = self._render_final_bytes(
+ uri,
+ content,
+ mode=mode,
+ existing_raw=existing_raw,
+ is_new_file=is_new_file,
+ )
+ await self._viking_fs.write_file_bytes(uri, final_bytes, ctx=ctx, lease_ref=lease_ref)
+ return final_bytes
+
async def _load_file_abstracts(self, uris: list[str], *, ctx: RequestContext) -> dict[str, str]:
vector_store = self._vikingdb
if vector_store is None:
@@ -1326,34 +1874,6 @@ class ContentWriteCoordinator:
except AbstractOverviewFormatError as exc:
raise InvalidArgumentError(str(exc)) from exc
- async def _enqueue_semantic_refresh(
- self,
- *,
- root_uri: str,
- changed_uri: str,
- context_type: str,
- ctx: RequestContext,
- change_type: str = "modified",
- target_uri: str = "",
- recursive: bool = False,
- force_refresh: bool = False,
- ingest_options: IngestOptions | None = None,
- file_md5: str | None = None,
- file_abstract: str = "",
- ) -> FreshnessAction:
- return await self._enqueue_semantic_refresh_changes(
- root_uri=root_uri,
- context_type=context_type,
- ctx=ctx,
- changes={change_type: [changed_uri]},
- target_uri=target_uri,
- recursive=recursive,
- force_refresh=force_refresh,
- ingest_options=ingest_options,
- file_md5s={changed_uri: file_md5} if file_md5 else None,
- file_abstracts={changed_uri: file_abstract} if file_abstract else None,
- )
-
async def _wait_for_queues(self, *, timeout: Optional[float]) -> Dict[str, Any]:
queue_manager = get_queue_manager()
try:
@@ -1384,17 +1904,13 @@ class ContentWriteCoordinator:
root_uri: str,
content: str,
mode: str,
- response_mode: Optional[str] = None,
+ response_mode: str,
wait: bool,
timeout: Optional[float],
ctx: RequestContext,
- written_bytes: int,
telemetry_id: str,
- processing_mode: ProcessingMode = DEFAULT_PROCESSING_MODE,
ingest_options: IngestOptions | None = None,
) -> Dict[str, Any]:
- del processing_mode
-
lock_path = self._viking_fs._uri_to_path(uri, ctx=ctx)
try:
lease = await self._viking_fs._async_agfs.pathlock_acquire_exact(lock_path)
@@ -1407,7 +1923,21 @@ class ContentWriteCoordinator:
released = False
request_registered = False
try:
- await self._write_in_place(uri, content, mode=mode, ctx=ctx, lease_ref=lease)
+ target_state = await self._classify_locked_write_target(uri, ctx=ctx)
+ is_new_file = target_state.is_new_file
+ write_kwargs = {
+ "mode": mode,
+ "ctx": ctx,
+ "lease_ref": lease,
+ }
+ if is_new_file:
+ write_kwargs["is_new_file"] = True
+ final_bytes = await self._write_in_place(
+ uri,
+ content,
+ **write_kwargs,
+ )
+ written_bytes = len(final_bytes)
await self._viking_fs._async_agfs.pathlock_release(lease)
released = True
if wait and telemetry_id and self._vikingdb_has_queue():
@@ -1433,18 +1963,18 @@ class ContentWriteCoordinator:
if telemetry_id
else await self._wait_for_queues(timeout=timeout)
)
- vector_status = self._memory_vector_status(
- embedding_requested=embedding_requested,
+ vector_status = self._queue_work_status(
+ requested=embedding_requested,
wait=wait,
queue_status=queue_status,
+ queue_name="Embedding",
)
return self._build_write_result(
uri=uri,
root_uri=root_uri,
context_type="memory",
- mode=response_mode or mode,
+ mode=response_mode,
written_bytes=written_bytes,
- wait=wait,
queue_status=queue_status,
semantic_status="skipped",
vector_status=vector_status,
@@ -1463,20 +1993,6 @@ class ContentWriteCoordinator:
return False
return bool(getattr(self._vikingdb, "has_queue_manager", False))
- def _memory_vector_status(
- self,
- *,
- embedding_requested: bool,
- wait: bool,
- queue_status: Optional[Dict[str, Any]],
- ) -> str:
- if not embedding_requested:
- return "skipped"
- if not wait:
- return "queued"
- _, vector_status = self._refresh_statuses(wait=True, queue_status=queue_status)
- return vector_status
-
async def _set_single_uri_tags(
self,
*,
@@ -1657,6 +2173,7 @@ class ContentWriteCoordinator:
ctx: RequestContext,
_allow_not_found: bool = False,
anchor_to_parent: bool = False,
+ validate_storage: bool = True,
) -> str:
parsed = VikingURI(uri)
parts = [part for part in parsed.full_path.split("/") if part]
@@ -1717,10 +2234,11 @@ class ContentWriteCoordinator:
if parent is not None:
root_uri = parent.uri
- stat = await self._safe_stat(root_uri, ctx=ctx, allow_not_found=_allow_not_found)
- if stat.get("not_found") or not stat.get("isDir"):
- parent = VikingURI(uri).parent
- if parent is None:
- raise InvalidArgumentError(f"could not resolve write root for {uri}")
- root_uri = parent.uri
+ if validate_storage:
+ stat = await self._safe_stat(root_uri, ctx=ctx, allow_not_found=_allow_not_found)
+ if stat.get("not_found") or not stat.get("isDir"):
+ parent = VikingURI(uri).parent
+ if parent is None:
+ raise InvalidArgumentError(f"could not resolve write root for {uri}")
+ root_uri = parent.uri
return root_uri
diff --git a/openviking/storage/context_update_execution.py b/openviking/storage/context_update_execution.py
new file mode 100644
index 000000000..81bc7771e
--- /dev/null
+++ b/openviking/storage/context_update_execution.py
@@ -0,0 +1,297 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Execute a single-file :class:`ContextUpdatePlan` outside the parser pipeline.
+
+``add_resources`` already builds and executes a :class:`ContextUpdatePlan`
+inside :class:`ResourceProcessor`. Content write does not have parsed artifact
+files or the directory-level vector sources that executor consumes, so this
+module adapts the same plan contract to an inline file body. Keeping this
+adapter separate avoids making the resource-ingest executor depend on
+single-file write state.
+
+The helper deliberately does not own the resource lock or the request wait
+tracker: those lifecycles differ per caller (write releases the lock between
+enqueue and wait; ingest hands the lock off to the queue). Callers keep that
+control and only ask this module to perform the plan's own work.
+"""
+
+from __future__ import annotations
+
+from collections import Counter
+from dataclasses import dataclass
+from typing import Any, Optional
+
+from openviking.core.context import ContextLevel
+from openviking.core.namespace import context_type_for_uri
+from openviking.server.identity import RequestContext
+from openviking.storage.acl import AclUpdate
+from openviking.storage.context_update_plan import ContextUpdatePlan, execute_content_tree_actions
+from openviking.storage.index_action import FieldPatch, IndexAction
+from openviking.utils.ingest_options import IngestOptions
+from openviking.utils.log_correlation import log_correlation
+from openviking_cli.utils import VikingURI, get_logger
+
+logger = get_logger(__name__)
+
+
+@dataclass(frozen=True)
+class PlanWork:
+ """Which downstream work a plan actually requested.
+
+ The caller finalizes ``semantic_status`` / ``vector_status`` after it waits,
+ because only the caller knows the queue status once its lock is released.
+ ``semantic_action`` carries the parent-refresh freshness decision so a
+ single-file caller can surface a ``deferred``/``skipped`` status.
+ """
+
+ semantic_requested: bool = False
+ vector_requested: bool = False
+ semantic_action: Optional[str] = None
+
+
+async def enqueue_direct_index_actions(
+ actions: Any,
+ *,
+ ctx: RequestContext,
+ acl_update: AclUpdate | None = None,
+) -> bool:
+ """Enqueue plan-level direct index actions (delete / upsert / field update).
+
+ This is the inline-file counterpart of ``ResourceProcessor``'s richer
+ artifact executor. Deletes are coalesced into one message, file L2 upserts
+ reuse :func:`vectorize_file`, and scalar-only mutations enqueue an
+ ``update_fields`` message. Returns whether any embedding work was enqueued.
+ """
+ if not actions:
+ return False
+
+ from openviking.storage.queuefs import get_queue_manager
+ from openviking.storage.queuefs.embedding_msg import EmbeddingMsg
+ from openviking.telemetry import get_current_telemetry
+ from openviking.utils.embedding_utils import _enqueue_embedding_message
+
+ queue_manager = get_queue_manager()
+ embedding_queue = queue_manager.get_queue(queue_manager.EMBEDDING, allow_create=True)
+ telemetry_id = get_current_telemetry().telemetry_id
+ action_counts = Counter(action.action.value for action in actions)
+ enqueued_any = False
+
+ delete_ids = [action.record_id for action in actions if action.action == IndexAction.DELETE]
+ if delete_ids:
+ message = EmbeddingMsg.for_delete(
+ record_ids=delete_ids,
+ context_data={
+ "uri": actions[0].uri,
+ "account_id": ctx.account_id,
+ "owner_user_id": ctx.user.user_id,
+ },
+ telemetry_id=telemetry_id,
+ )
+ enqueued_any = (
+ await _enqueue_embedding_message(
+ embedding_queue,
+ message,
+ failure_message="Failed to enqueue planned vector deletes",
+ )
+ or enqueued_any
+ )
+ for action in actions:
+ if action.action in {IndexAction.UPSERT, IndexAction.MERGE}:
+ if action.level != int(ContextLevel.DETAIL):
+ raise ValueError("Direct index upsert only supports file detail records")
+ vector_enqueued = await vectorize_resource_file(
+ action.uri,
+ ctx=ctx,
+ acl_update=acl_update,
+ file_md5=action.md5,
+ scalar_override={
+ **dict(action.upsert_fields),
+ "_record_id": action.record_id,
+ },
+ summary=action.summary,
+ action=action.action.value,
+ field_patch=action.field_patch,
+ )
+ if vector_enqueued:
+ enqueued_any = True
+ else:
+ # An empty/unsupported replacement cannot produce a new
+ # embedding. Delete the planned record ID so an older vector
+ # cannot keep representing content that is no longer present.
+ delete_message = EmbeddingMsg.for_delete(
+ record_ids=[action.record_id],
+ context_data={
+ "uri": action.uri,
+ "account_id": ctx.account_id,
+ "owner_user_id": ctx.user.user_id,
+ },
+ telemetry_id=telemetry_id,
+ )
+ enqueued_any = (
+ await _enqueue_embedding_message(
+ embedding_queue,
+ delete_message,
+ failure_message=f"Failed to enqueue stale vector delete for {action.uri}",
+ )
+ or enqueued_any
+ )
+ continue
+ if action.action != IndexAction.UPDATE_FIELDS:
+ continue
+ assert action.field_patch is not None
+ message = EmbeddingMsg.for_update_fields(
+ record_id=action.record_id,
+ field_patch=action.field_patch,
+ context_data={
+ "uri": action.uri,
+ "level": action.level,
+ "account_id": ctx.account_id,
+ "owner_user_id": ctx.user.user_id,
+ },
+ telemetry_id=telemetry_id,
+ )
+ enqueued_any = (
+ await _enqueue_embedding_message(
+ embedding_queue,
+ message,
+ failure_message=f"Failed to enqueue scalar update for {action.uri}",
+ )
+ or enqueued_any
+ )
+ logger.debug(
+ "[DirectIndexActions] %s root=%s action_counts=%s action_count=%d",
+ log_correlation(),
+ actions[0].uri if actions else "",
+ dict(action_counts),
+ len(actions),
+ )
+ return enqueued_any
+
+
+async def vectorize_resource_file(
+ file_uri: str,
+ *,
+ ctx: RequestContext,
+ acl_update: AclUpdate | None = None,
+ file_md5: str | None = None,
+ scalar_override: Optional[dict[str, Any]] = None,
+ field_patch: FieldPatch | None = None,
+ summary: str = "",
+ action: str = "merge",
+) -> bool:
+ """Enqueue a single file's L2 vector using the standard file text policy.
+
+ In ``vectors_only`` mode no semantic node regenerates a summary, but the RNFV
+ V snapshot already carries the existing L2 ``abstract`` (it is projected into
+ the inventory and travels through the direct index action's scalar fields).
+ Seed ``summary_dict`` with that abstract so ``vectorize_file`` can honor the
+ configured ``text_source``: ``summary_first`` embeds the existing abstract
+ when present, ``content_only`` still embeds the file body. When no abstract is
+ available (new file / empty record) the summary stays empty and the body is
+ used regardless of ``text_source``.
+ """
+ from openviking.utils.embedding_utils import vectorize_file
+
+ parent = VikingURI(file_uri).parent
+ if parent is None:
+ return False
+ name = file_uri.rsplit("/", 1)[-1]
+ existing_abstract = str(summary or (scalar_override or {}).get("abstract") or "")
+ return await vectorize_file(
+ file_path=file_uri,
+ summary_dict={"name": name, "summary": existing_abstract},
+ parent_uri=parent.uri,
+ context_type=context_type_for_uri(file_uri),
+ ctx=ctx,
+ # Direct actions carry planner-resolved tag fields in
+ # ``scalar_override`` / ``field_patch``. Forward only ACL intent here so
+ # the first asynchronous upsert can materialize it without applying tag
+ # modes a second time.
+ ingest_options=IngestOptions(acl_update=acl_update),
+ file_md5=file_md5,
+ scalar_override=scalar_override,
+ field_patch=field_patch,
+ action=action,
+ )
+
+
+async def commit_and_enqueue_plan(
+ plan: ContextUpdatePlan,
+ *,
+ ctx: RequestContext,
+ inline_store: Any = None,
+ inline_ref: Any = None,
+ target: Any = None,
+ ingest_options: IngestOptions | None = None,
+ file_created: bool = False,
+ force_refresh: bool | None = None,
+ generation_trigger: str = "content_write",
+ summarizer: Any = None,
+) -> PlanWork:
+ """Commit content synchronously, then enqueue index + file-refresh work.
+
+ Content-tree actions run under the caller's lease before any queue handoff.
+ Direct index actions and the flat-file ``file_refresh`` are enqueued next.
+ The caller owns the lock lifecycle and the request wait tracker (register
+ before calling, wait/finalize after); this helper only performs the plan's
+ own work and reports which downstream work it requested.
+
+ ``force_refresh`` selects the parent-aggregation policy for ``file_refresh``:
+ ``None`` always aggregates (resource ingest), while a boolean opts into the
+ freshness gate so an asynchronous single-file write can defer parent L0/L1
+ aggregation without dropping the changed file's own vector maintenance.
+ """
+ if plan.content_tree_actions:
+ if target is None:
+ raise ValueError("content-tree actions require a resource target")
+ await execute_content_tree_actions(
+ plan.content_tree_actions,
+ store=inline_store,
+ artifact_ref=inline_ref,
+ target=target,
+ )
+
+ vector_requested = False
+ if plan.direct_index_actions:
+ options = IngestOptions.from_value(ingest_options)
+ vector_requested = await enqueue_direct_index_actions(
+ plan.direct_index_actions,
+ ctx=ctx,
+ acl_update=options.acl_update,
+ )
+
+ semantic_requested = False
+ semantic_action: Optional[str] = None
+ if plan.file_refresh is not None:
+ if summarizer is None:
+ from openviking.utils.summarizer import Summarizer
+
+ summarizer = Summarizer(vlm_processor=None)
+ refresh_result = await summarizer.refresh_file_parent(
+ file_uri=plan.file_refresh.file_uri,
+ ctx=ctx,
+ ingest_options=IngestOptions.from_value(ingest_options),
+ created=file_created,
+ file_md5=plan.file_refresh.md5,
+ file_abstract="",
+ file_vector_action="upsert" if file_created else "",
+ generation_trigger=generation_trigger,
+ force_refresh=force_refresh,
+ )
+ semantic_requested = refresh_result.get("enqueued_count", 0) > 0
+ semantic_action = refresh_result.get("semantic_action")
+ vector_requested = True
+
+ return PlanWork(
+ semantic_requested=semantic_requested,
+ vector_requested=vector_requested,
+ semantic_action=semantic_action,
+ )
+
+
+__all__ = [
+ "PlanWork",
+ "commit_and_enqueue_plan",
+ "enqueue_direct_index_actions",
+ "vectorize_resource_file",
+]
diff --git a/openviking/storage/context_update_plan.py b/openviking/storage/context_update_plan.py
index 80a7a9126..776ffff59 100644
--- a/openviking/storage/context_update_plan.py
+++ b/openviking/storage/context_update_plan.py
@@ -1318,15 +1318,30 @@ async def build_context_update_plan_from_snapshot(
return diff, plan
root_entry = diff.entries[""]
- needs_refresh = ContentState(root_entry.content_state) in {
- ContentState.ADDED,
- ContentState.RESTORE,
- ContentState.MODIFIED,
- ContentState.REPLACE_KIND,
- } or (
- snapshot.request.vectorize
- and IndexState(root_entry.index_state)
- in {IndexState.MISSING, IndexState.PARTIAL, IndexState.STALE, IndexState.LEVEL_CONFLICT}
+ has_file_abstract = any(
+ record.relative_path == ""
+ and record.level == 2
+ and str(record.fields.get("abstract") or "").strip()
+ for record in records.values()
+ )
+ needs_refresh = (
+ ContentState(root_entry.content_state)
+ in {
+ ContentState.ADDED,
+ ContentState.RESTORE,
+ ContentState.MODIFIED,
+ ContentState.REPLACE_KIND,
+ }
+ or (
+ snapshot.request.vectorize
+ and IndexState(root_entry.index_state)
+ in {IndexState.MISSING, IndexState.PARTIAL, IndexState.STALE, IndexState.LEVEL_CONFLICT}
+ )
+ or (
+ snapshot.request.processing_mode != "vectors_only"
+ and snapshot.request.vectorize
+ and not has_file_abstract
+ )
)
return diff, ContextUpdatePlan(
root_uri=plan.root_uri,
diff --git a/openviking/storage/queuefs/semantic_executor.py b/openviking/storage/queuefs/semantic_executor.py
index 61580f403..2c3532143 100644
--- a/openviking/storage/queuefs/semantic_executor.py
+++ b/openviking/storage/queuefs/semantic_executor.py
@@ -194,6 +194,8 @@ class SemanticTreeExecutor:
file_md5s: Optional[Dict[str, str]] = None,
artifact_files: Optional[List[str]] = None,
file_abstracts: Optional[Dict[str, str]] = None,
+ file_vector_actions: Optional[Dict[str, str]] = None,
+ file_ingest_options: Optional[Dict[str, IngestOptions | Dict[str, Any]]] = None,
semantic_plan: Optional["SemanticPlan"] = None,
telemetry_id: str | None = None,
source_contents: Optional[Mapping[tuple[str, int], str | bytes]] = None,
@@ -252,6 +254,11 @@ class SemanticTreeExecutor:
self._file_md5s = dict(file_md5s or {})
self._artifact_files = [path.strip("/") for path in (artifact_files or []) if path]
self._file_abstracts = dict(file_abstracts or {})
+ self._file_vector_actions = dict(file_vector_actions or {})
+ self._file_ingest_options = {
+ uri.rstrip("/"): IngestOptions.from_value(options)
+ for uri, options in (file_ingest_options or {}).items()
+ }
self._plan_entries_by_uri: Dict[str, "SemanticTreeEntry"] = {}
self._plan_children: Dict[str, tuple[List[str], List[str]]] = {}
self._plan_active_dirs: Set[str] = set()
@@ -902,6 +909,10 @@ class SemanticTreeExecutor:
)
def _ingest_options_for_file(self, file_path: str) -> IngestOptions:
+ if self._generation_trigger == "content_write":
+ options = self._file_ingest_options.get(file_path.rstrip("/"))
+ if options is not None:
+ return options
if self._generation_trigger == "content_write" and file_path not in self._changed_paths:
return IngestOptions(acl_update=self._ingest_options.acl_update)
return self._ingest_options
@@ -1184,6 +1195,8 @@ class SemanticTreeExecutor:
vectorize_kwargs["scalar_override"] = self._plan_scalar_override(file_path, 2)
vectorize_kwargs["field_patch"] = self._plan_field_patch(file_path, 2)
vectorize_kwargs["action"] = slot.action.value if slot is not None else "upsert"
+ elif action := self._file_vector_actions.get(file_path.rstrip("/")):
+ vectorize_kwargs["action"] = action
enqueued = await self._processor._vectorize_single_file(
parent_uri=parent_uri,
context_type=self._context_type,
diff --git a/openviking/storage/queuefs/semantic_msg.py b/openviking/storage/queuefs/semantic_msg.py
index f30876070..21bd8bc09 100644
--- a/openviking/storage/queuefs/semantic_msg.py
+++ b/openviking/storage/queuefs/semantic_msg.py
@@ -82,6 +82,8 @@ class SemanticMsg:
artifact_ref: Optional[Dict[str, Any]] = None
artifact_files: List[str] = field(default_factory=list)
file_abstracts: Dict[str, str] = field(default_factory=dict)
+ file_vector_actions: Dict[str, str] = field(default_factory=dict)
+ file_ingest_options: Dict[str, IngestOptions] = field(default_factory=dict)
plan: Optional[SemanticPlan] = None
def __init__(
@@ -113,6 +115,8 @@ class SemanticMsg:
artifact_ref: Optional[Dict[str, Any]] = None,
artifact_files: Optional[List[str]] = None,
file_abstracts: Optional[Dict[str, str]] = None,
+ file_vector_actions: Optional[Dict[str, str]] = None,
+ file_ingest_options: Optional[Dict[str, IngestOptions | Dict[str, Any]]] = None,
plan: SemanticPlan | Dict[str, Any] | None = None,
queue_enqueued_at: float = 0.0,
failure_requeue_count: int = 0,
@@ -148,6 +152,16 @@ class SemanticMsg:
self.artifact_ref = dict(artifact_ref) if artifact_ref else None
self.artifact_files = list(artifact_files or [])
self.file_abstracts = dict(file_abstracts or {})
+ self.file_vector_actions = {}
+ for uri, action in (file_vector_actions or {}).items():
+ normalized_action = str(action)
+ if normalized_action not in {"upsert", "merge"}:
+ raise ValueError(f"unsupported file vector action: {normalized_action}")
+ self.file_vector_actions[str(uri)] = normalized_action
+ self.file_ingest_options = {
+ str(uri): IngestOptions.from_value(options)
+ for uri, options in (file_ingest_options or {}).items()
+ }
self.plan = (
plan
if isinstance(plan, SemanticPlan)
@@ -160,6 +174,9 @@ class SemanticMsg:
"""Convert object to dictionary."""
data = asdict(self)
data["ingest_options"] = self.ingest_options.to_dict()
+ data["file_ingest_options"] = {
+ uri: options.to_dict() for uri, options in self.file_ingest_options.items()
+ }
if self.plan is not None:
data["plan"] = self.plan.to_dict()
return data
@@ -228,6 +245,16 @@ class SemanticMsg:
file_abstracts=(
data.get("file_abstracts") if isinstance(data.get("file_abstracts"), dict) else None
),
+ file_vector_actions=(
+ data.get("file_vector_actions")
+ if isinstance(data.get("file_vector_actions"), dict)
+ else None
+ ),
+ file_ingest_options=(
+ data.get("file_ingest_options")
+ if isinstance(data.get("file_ingest_options"), dict)
+ else None
+ ),
plan=data.get("plan") if isinstance(data.get("plan"), dict) else None,
queue_enqueued_at=data.get("queue_enqueued_at", 0.0),
failure_requeue_count=data.get("failure_requeue_count", 0),
diff --git a/openviking/storage/queuefs/semantic_processor.py b/openviking/storage/queuefs/semantic_processor.py
index c248ddced..f033bbcbe 100644
--- a/openviking/storage/queuefs/semantic_processor.py
+++ b/openviking/storage/queuefs/semantic_processor.py
@@ -704,6 +704,8 @@ class SemanticProcessor(DequeueHandlerBase):
file_md5s=msg.file_md5s,
artifact_files=msg.artifact_files,
file_abstracts=msg.file_abstracts,
+ file_vector_actions=msg.file_vector_actions,
+ file_ingest_options=msg.file_ingest_options,
telemetry_id=msg.telemetry_id,
)
await executor.run(run_uri)
diff --git a/openviking/storage/resource_diff.py b/openviking/storage/resource_diff.py
index c6a1c9149..9803f48ae 100644
--- a/openviking/storage/resource_diff.py
+++ b/openviking/storage/resource_diff.py
@@ -29,7 +29,7 @@ import logging
from collections import Counter
from dataclasses import dataclass, field
from enum import Enum
-from typing import Any, Dict, Mapping, Tuple
+from typing import Any, Dict, Literal, Mapping, Tuple
from openviking.concurrency import bounded_map
from openviking.storage.internal_names import is_storage_internal_name
@@ -45,6 +45,7 @@ from openviking.storage.resource_rnfv import (
VectorRecordSnapshot,
canonical_vector_records_by_level,
)
+from openviking.utils.content_hash import content_md5
from openviking.utils.log_correlation import log_correlation
logger = logging.getLogger(__name__)
@@ -138,6 +139,34 @@ class ArtifactInventory:
rewritten_paths: frozenset[str] = frozenset()
+class InlineBytesStore:
+ """Minimal parse-output store backed by a single file's final bytes.
+
+ The single-file write path has no parse artifact; it already holds the
+ fully rendered bytes. This store lets the shared RNFV plan pipeline read the
+ same bytes for body fallbacks and content commit without inventing a real
+ artifact backend. The ref is an opaque placeholder; only ``read_bytes`` is
+ exercised.
+ """
+
+ backend = "inline"
+
+ def __init__(self, data: bytes) -> None:
+ self._data = data
+
+ async def read_bytes(self, ref: Any, rel_path: str) -> bytes:
+ del ref, rel_path
+ return self._data
+
+
+def make_inline_file_inventory(final_bytes: bytes) -> ArtifactInventory:
+ """Build a single-file N snapshot from already-rendered bytes."""
+ return ArtifactInventory(
+ entries={"": NewEntry(md5=content_md5(final_bytes), is_dir=False)},
+ artifact_paths={"": ""},
+ )
+
+
def count_tree_entry_kinds(
entries: Mapping[str, NewEntry | FormalEntry],
) -> tuple[int, int, bool]:
@@ -544,9 +573,18 @@ async def build_rnfv_snapshot(
request_intent: RequestIntent | None = None,
root_is_file: bool = False,
target_preexisting: bool = True,
+ formal_snapshot: tuple[Dict[str, FormalEntry], bool] | None = None,
artifact_inventory: ArtifactInventory | None = None,
+ vector_scope: Literal["subtree", "self"] = "subtree",
+ vector_inventory: Mapping[str, Mapping[str, Any]] | None = None,
) -> RNFVSnapshot:
- """Read the complete R/N/F/V inputs without deriving an executable plan."""
+ """Read the complete R/N/F/V inputs without deriving an executable plan.
+
+ ``vector_scope="self"`` restricts the subtree-strategy V read to the target URI itself
+ (all levels), which the single-file write path uses so an incremental
+ update never reads sibling records under a shared parent directory.
+
+ """
request = request_intent or RequestIntent(
target_uri=target_uri, processing_mode="semantic_and_vectors"
)
@@ -557,6 +595,8 @@ async def build_rnfv_snapshot(
)
async def read_formal() -> tuple[Dict[str, FormalEntry], bool]:
+ if formal_snapshot is not None:
+ return formal_snapshot
if not target_preexisting:
return {}, True
return await read_target_file_snapshot(
@@ -571,11 +611,15 @@ async def build_rnfv_snapshot(
asyncio.create_task(read_artifact()),
asyncio.create_task(read_formal()),
]
- if request.vectorize:
+ if request.vectorize and vector_inventory is None:
tasks.append(
asyncio.create_task(
_read_incremental_vector_inventory(
- vikingdb, target_uri=target_uri, ctx=ctx, projection=projection
+ vikingdb,
+ target_uri=target_uri,
+ ctx=ctx,
+ projection=projection,
+ vector_scope=vector_scope,
)
)
)
@@ -588,7 +632,12 @@ async def build_rnfv_snapshot(
await asyncio.gather(*tasks, return_exceptions=True)
raise
artifact, formal = results[:2]
- inventory = results[2] if request.vectorize else {}
+ if not request.vectorize:
+ inventory: Dict[str, Dict[str, Any]] = {}
+ elif vector_inventory is not None:
+ inventory = dict(vector_inventory)
+ else:
+ inventory = results[2]
target_files, files_complete = formal
base = target_uri.rstrip("/")
prefix = base + "/"
@@ -626,10 +675,19 @@ async def _read_incremental_vector_inventory(
target_uri: str,
ctx: Any,
projection: frozenset[str],
+ vector_scope: Literal["subtree", "self"] = "subtree",
) -> Dict[str, Dict[str, Any]]:
"""Read the lightweight V snapshot used to build RNFV."""
return await vikingdb.get_incremental_inventory_under_uri(
- target_uri, ctx=ctx, output_fields=sorted(projection)
+ target_uri,
+ ctx=ctx,
+ output_fields=sorted(projection),
+ # The shared inventory API expresses an exact URI as a non-recursive
+ # scope without direct children. Keep its richer recursive/direct-child
+ # contract instead of adding a parallel depth parameter solely for the
+ # single-file write path.
+ recursive=vector_scope == "subtree",
+ include_direct_children=False,
)
@@ -637,10 +695,12 @@ __all__ = [
"ArtifactInventory",
"ContentState",
"IndexState",
+ "InlineBytesStore",
"ResourceDiffEntry",
"ResourceDiffResult",
"build_rnfv_snapshot",
"count_tree_entry_kinds",
+ "make_inline_file_inventory",
"prepare_artifact_inventory",
"read_target_file_snapshot",
]
diff --git a/openviking/storage/viking_vector_index_backend.py b/openviking/storage/viking_vector_index_backend.py
index 014200d68..33452dba0 100644
--- a/openviking/storage/viking_vector_index_backend.py
+++ b/openviking/storage/viking_vector_index_backend.py
@@ -2349,6 +2349,93 @@ class VikingVectorIndexBackend:
assert last_error is not None
raise last_error
+ async def get_incremental_inventory_by_uris(
+ self,
+ uris: List[str],
+ *,
+ ctx: RequestContext,
+ batch_size: int = 100,
+ output_fields: Optional[List[str]] = None,
+ ) -> Dict[str, Dict[str, Dict[str, Any]]]:
+ """Read lightweight inventory with one exact scalar query per URI chunk.
+
+ Unlike :meth:`get_incremental_inventory_under_uri`, this never uses a
+ path scope: every returned record must belong to one of ``uris``. The
+ result remains keyed by the caller's URI spelling and then by record ID
+ so file-scoped RNFV snapshots can consume a pre-hydrated V subset
+ without one vector query per file. Canonical V has at most one record
+ per L0/L1/L2 level at each URI; request one extra result so an abnormal
+ duplicate set fails explicitly rather than being silently truncated.
+ """
+ if batch_size <= 0:
+ raise ValueError("batch_size must be positive")
+
+ requested_by_canonical: Dict[str, str] = {}
+ for uri in uris:
+ requested_by_canonical.setdefault(resolve_uri(uri).uri, uri)
+ canonical_uris = list(requested_by_canonical)
+ if not canonical_uris:
+ return {}
+
+ projection = list(
+ dict.fromkeys(
+ [
+ *INCREMENTAL_INVENTORY_OUTPUT_FIELDS,
+ *(output_fields or []),
+ ]
+ )
+ )
+ records_by_uri: Dict[str, Dict[str, Dict[str, Any]]] = {uri: {} for uri in canonical_uris}
+ for start in range(0, len(canonical_uris), batch_size):
+ chunk = canonical_uris[start : start + batch_size]
+ scope = And(
+ [
+ Eq("account_id", ctx.account_id),
+ In("uri", chunk),
+ In("level", [0, 1, 2]),
+ ]
+ )
+ what = f"Incremental inventory for {len(chunk)} exact URI(s)"
+ result_limit = len(chunk) * 3 + 1
+ page = await self.filter(
+ filter=scope,
+ limit=result_limit,
+ output_fields=projection,
+ ctx=ctx,
+ )
+ if len(page) >= result_limit:
+ raise RuntimeError(
+ f"{what} exceeded its canonical L0/L1/L2 record limit; "
+ "run reindex or prune the duplicate records before retrying"
+ )
+ for record in page:
+ record_id = str(record.get("id") or "")
+ record_uri = str(record.get("uri") or "")
+ try:
+ level = int(record.get("level"))
+ except (TypeError, ValueError):
+ level = -1
+ if not record_id or record_uri not in records_by_uri or level not in {0, 1, 2}:
+ raise RuntimeError(
+ f"{what} returned an invalid record: id={record_id or '<missing>'} "
+ f"uri={record_uri or '<missing>'} level={level}"
+ )
+ if record_id in records_by_uri[record_uri]:
+ raise RuntimeError(f"{what} returned duplicate record id: {record_id}")
+ records_by_uri[record_uri][record_id] = {
+ field: record[field] for field in projection if field in record
+ }
+ records_by_uri[record_uri][record_id].update(
+ {
+ "id": record_id,
+ "uri": record_uri,
+ "level": level,
+ "md5": str(record.get("md5") or ""),
+ }
+ )
+
+ return {requested_by_canonical[uri]: records for uri, records in records_by_uri.items()}
+
async def hydrate_incremental_records(
self,
expected: Mapping[str, Mapping[str, Any]],
diff --git a/openviking/utils/summarizer.py b/openviking/utils/summarizer.py
index b8076e7f8..809b35f23 100644
--- a/openviking/utils/summarizer.py
+++ b/openviking/utils/summarizer.py
@@ -5,7 +5,9 @@
Handles summarization and key information extraction.
"""
-from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple
+from collections import defaultdict
+from dataclasses import dataclass, field
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Sequence, Tuple
from openviking.core.namespace import context_type_for_uri
from openviking.storage.queuefs import SemanticMsg, get_queue_manager
@@ -24,6 +26,28 @@ if TYPE_CHECKING:
logger = get_logger(__name__)
+@dataclass(frozen=True)
+class FileRefreshRequest:
+ """One committed flat-file change awaiting parent semantic maintenance.
+
+ This is an execution-time input for the shared summarizer, not a durable
+ RNFV or queue schema. A batch coordinator collects one request per
+ successfully committed file and lets :meth:`refresh_file_parents` coalesce
+ them by direct parent.
+ """
+
+ file_uri: str
+ created: bool = False
+ md5: str | None = None
+ abstract: str = ""
+ vector_action: str = ""
+ ingest_options: IngestOptions = field(default_factory=IngestOptions)
+
+ def __post_init__(self) -> None:
+ if self.vector_action not in {"", "upsert", "merge"}:
+ raise ValueError(f"unsupported file vector action: {self.vector_action}")
+
+
class Summarizer:
"""
Handles summarization of resources.
@@ -42,59 +66,183 @@ class Summarizer:
created: bool = False,
file_md5: str | None = None,
file_abstract: str = "",
+ file_vector_action: str = "",
+ generation_trigger: str = "semantic_refresh",
+ force_refresh: bool | None = None,
) -> Dict[str, Any]:
- """Summarize one flat file and refresh its parent directory semantics."""
+ """Summarize one flat file and refresh its parent directory semantics.
+
+ ``force_refresh`` controls parent-directory aggregation. ``None`` (the
+ default used by resource ingest) always aggregates immediately. A boolean
+ opts into the freshness gate: the changed file is still summarized and
+ vectorized, but parent L0/L1 aggregation is deferred by the freshness
+ policy unless ``force_refresh`` is ``True``. The returned
+ ``semantic_action`` reports the gate decision so callers can surface a
+ ``deferred``/``skipped`` status.
+ """
parent = VikingURI(file_uri).parent
if parent is None:
return {"status": "error", "message": f"file has no parent URI: {file_uri}"}
+ result = await self.refresh_file_parents(
+ changes=(
+ FileRefreshRequest(
+ file_uri=file_uri,
+ created=created,
+ md5=file_md5,
+ abstract=file_abstract,
+ vector_action=file_vector_action,
+ ingest_options=IngestOptions.from_value(ingest_options),
+ ),
+ ),
+ ctx=ctx,
+ skip_vectorization=skip_vectorization,
+ ingest_options=ingest_options,
+ generation_trigger=generation_trigger,
+ force_refresh=force_refresh,
+ )
+ return {
+ "status": "success",
+ "enqueued_count": result["enqueued_count"],
+ "semantic_action": result["parent_actions"][parent.uri.rstrip("/")],
+ }
+
+ async def refresh_file_parents(
+ self,
+ *,
+ changes: Sequence[FileRefreshRequest],
+ ctx: "RequestContext",
+ skip_vectorization: bool = False,
+ ingest_options: IngestOptions | None = None,
+ generation_trigger: str = "semantic_refresh",
+ force_refresh: bool | None = None,
+ ) -> Dict[str, Any]:
+ """Queue flat-file semantic maintenance grouped by each direct parent.
+
+ A file remains the RNFV planning unit, while this execution helper owns
+ the batch-only concern: one freshness decision and at most one
+ ``SemanticMsg`` for every direct parent directory.
+ """
+ from openviking.storage.abstract_overview import plan_abstract_overview_refresh
+ from openviking.storage.errors import LockAcquisitionError
+ from openviking.storage.queuefs.semantic_ops.freshness_policy import FreshnessAction
+ from openviking_cli.utils.config import get_openviking_config
+
+ grouped: dict[str, list[FileRefreshRequest]] = defaultdict(list)
+ for change in changes:
+ parent = VikingURI(change.file_uri).parent
+ if parent is None:
+ raise ValueError(f"file has no parent URI: {change.file_uri}")
+ grouped[parent.uri.rstrip("/")].append(change)
+
+ if not grouped:
+ return {"status": "success", "enqueued_count": 0, "parent_actions": {}}
queue_manager = get_queue_manager()
semantic_queue = queue_manager.get_queue(queue_manager.SEMANTIC, allow_create=True)
telemetry_id = get_current_telemetry().telemetry_id
- parent_uri = parent.uri.rstrip("/")
- msg = SemanticMsg(
- uri=parent_uri,
- context_type=context_type_for_uri(file_uri),
- recursive=False,
- account_id=ctx.account_id,
- user_id=ctx.user.user_id,
- peer_id=ctx.user.user_id,
- role=str(ctx.role),
- skip_vectorization=skip_vectorization,
- telemetry_id=telemetry_id,
- changes={"added" if created else "modified": [file_uri]},
- coalesce_key=build_semantic_coalesce_key(
- context_type=context_type_for_uri(file_uri),
+ parent_actions: dict[str, str] = {}
+ enqueued_count = 0
+ for parent_uri, parent_changes in sorted(grouped.items()):
+ context_types = {context_type_for_uri(change.file_uri) for change in parent_changes}
+ if len(context_types) != 1:
+ raise ValueError(f"mixed context types under parent refresh: {parent_uri}")
+ context_type = context_types.pop()
+ added = sorted(change.file_uri for change in parent_changes if change.created)
+ modified = sorted(change.file_uri for change in parent_changes if not change.created)
+
+ semantic_action = FreshnessAction.REFRESH_NOW
+ if force_refresh is not None:
+ semantic_config = get_openviking_config().semantic
+ try:
+ decision = await plan_abstract_overview_refresh(
+ viking_fs=get_viking_fs(),
+ dir_uri=parent_uri,
+ changed_entries=len(parent_changes),
+ ctx=ctx,
+ overview_sample_limit=getattr(semantic_config, "overview_sample_limit", 32),
+ refresh_ratio=getattr(semantic_config, "freshness_refresh_ratio", 0.10),
+ force_refresh=force_refresh,
+ )
+ semantic_action = decision.action
+ except LockAcquisitionError:
+ # Parent aggregation is best-effort; changed-file work still runs.
+ logger.info("Skipping busy parent semantic refresh: %s", parent_uri)
+ semantic_action = FreshnessAction.NOOP
+ parent_actions[parent_uri] = semantic_action.value
+ aggregate_directory = semantic_action is FreshnessAction.REFRESH_NOW
+
+ msg = SemanticMsg(
uri=parent_uri,
+ context_type=context_type,
+ recursive=False,
account_id=ctx.account_id,
user_id=ctx.user.user_id,
peer_id=ctx.user.user_id,
- ),
- ingest_options=ingest_options,
- file_md5s={file_uri: file_md5} if file_md5 else None,
- file_abstracts={file_uri: file_abstract} if file_abstract else None,
- )
- if telemetry_id:
- get_request_wait_tracker().register_semantic_root(telemetry_id, msg.id)
- try:
- enqueue_id = await semantic_queue.enqueue(msg)
- except Exception as exc:
- if telemetry_id:
- get_request_wait_tracker().mark_semantic_failed(
- telemetry_id,
- msg.id,
- str(exc),
- )
- raise
- if enqueue_id == "deduplicated":
+ role=str(ctx.role),
+ skip_vectorization=skip_vectorization,
+ telemetry_id=telemetry_id,
+ changes={
+ change_type: uris
+ for change_type, uris in (("added", added), ("modified", modified))
+ if uris
+ },
+ coalesce_key=(
+ build_semantic_coalesce_key(
+ context_type=context_type,
+ uri=parent_uri,
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ peer_id=ctx.user.user_id,
+ )
+ if aggregate_directory
+ else ""
+ ),
+ generation_trigger=generation_trigger,
+ aggregate_directory=aggregate_directory,
+ ingest_options=ingest_options,
+ file_md5s={change.file_uri: change.md5 for change in parent_changes if change.md5}
+ or None,
+ file_abstracts={
+ change.file_uri: change.abstract for change in parent_changes if change.abstract
+ }
+ or None,
+ file_vector_actions={
+ change.file_uri: change.vector_action
+ for change in parent_changes
+ if change.vector_action
+ },
+ file_ingest_options={
+ change.file_uri: change.ingest_options
+ for change in parent_changes
+ if change.ingest_options != IngestOptions()
+ },
+ )
if telemetry_id:
- get_request_wait_tracker().mark_semantic_done(
- telemetry_id,
- msg.id,
- processed_delta=0,
- )
- return {"status": "success", "enqueued_count": 0}
- return {"status": "success", "enqueued_count": 1}
+ get_request_wait_tracker().register_semantic_root(telemetry_id, msg.id)
+ try:
+ enqueue_id = await semantic_queue.enqueue(msg)
+ except Exception as exc:
+ if telemetry_id:
+ get_request_wait_tracker().mark_semantic_failed(
+ telemetry_id,
+ msg.id,
+ str(exc),
+ )
+ raise
+ if enqueue_id == "deduplicated":
+ if telemetry_id:
+ get_request_wait_tracker().mark_semantic_done(
+ telemetry_id,
+ msg.id,
+ processed_delta=0,
+ )
+ else:
+ enqueued_count += 1
+ return {
+ "status": "success",
+ "enqueued_count": enqueued_count,
+ "parent_actions": parent_actions,
+ }
async def summarize(
self,
diff --git a/sdk/go/client_test.go b/sdk/go/client_test.go
index 39b6ce82a..b22f3429d 100644
--- a/sdk/go/client_test.go
+++ b/sdk/go/client_test.go
@@ -690,14 +690,14 @@ func TestSearchSendsSessionAndSearchFilters(t *testing.T) {
defer closeServer()
if _, err := client.Search(context.Background(), "auth", &SearchOptions{
- TargetURI: "resources/docs",
- SessionID: "session-1",
+ TargetURI: "resources/docs",
+ SessionID: "session-1",
SearchType: "keywords",
- Since: "1d",
- Until: "2026-06-18",
- TimeField: "updated_at",
- Level: []int{2},
- Tags: []string{"topic=docs"},
+ Since: "1d",
+ Until: "2026-06-18",
+ TimeField: "updated_at",
+ Level: []int{2},
+ Tags: []string{"topic=docs"},
}); err != nil {
t.Fatal(err)
}
@@ -872,7 +872,7 @@ func TestBatchWriteAndDownloadBytes(t *testing.T) {
switch r.Method + " " + r.URL.Path {
case "POST /api/v1/content/batch-write":
body := readJSONBody(t, r)
- if body["root_uri"] != "viking://resources/project" || body["future_flag"] != float64(0) {
+ if body["root_uri"] != "viking://resources/project" || body["future_flag"] != float64(0) || body["processing_mode"] != "vectors_only" {
t.Fatalf("body = %#v", body)
}
operations, ok := body["operations"].([]any)
@@ -881,9 +881,15 @@ func TestBatchWriteAndDownloadBytes(t *testing.T) {
}
operation, ok := operations[0].(map[string]any)
if !ok || !reflect.DeepEqual(operation, map[string]any{
- "uri": "viking://resources/project/a.txt",
- "content": "hello",
- "mode": "upsert",
+ "uri": "viking://resources/project/a.txt",
+ "content": "hello",
+ "mode": "upsert",
+ "tags": []any{"team=search"},
+ "tag_mode": "append",
+ "acl": map[string]any{
+ "acl_mode": "restricted",
+ "entries": []any{map[string]any{"principal": "group:engineering", "level": "write"}},
+ },
}) {
t.Fatalf("operation = %#v", operations[0])
}
@@ -904,8 +910,11 @@ func TestBatchWriteAndDownloadBytes(t *testing.T) {
URI: "resources/project/a.txt",
Content: String("hello"),
Mode: "upsert",
+ Tags: []string{"team=search"},
+ TagMode: "append",
+ ACL: &ACLSpec{ACLMode: "restricted", Entries: []ACLEntry{{Principal: "group:engineering", Level: "write"}}},
},
- }, &BatchWriteOptions{Extra: map[string]any{"future_flag": 0}}); err != nil {
+ }, &BatchWriteOptions{ProcessingMode: "vectors_only", Extra: map[string]any{"future_flag": 0}}); err != nil {
t.Fatal(err)
}
data, err := client.DownloadBytes(context.Background(), "resources/project/a.txt")
diff --git a/sdk/go/filesystem.go b/sdk/go/filesystem.go
index 7d8e06855..bb2a8776d 100644
--- a/sdk/go/filesystem.go
+++ b/sdk/go/filesystem.go
@@ -326,6 +326,7 @@ func (c *Client) BatchWrite(
setAny(payload, "wait", opts.Wait)
setFloatPtr(payload, "timeout", opts.Timeout)
setAny(payload, "telemetry", opts.Telemetry)
+ setString(payload, "processing_mode", opts.ProcessingMode)
if err := mergeExtra(payload, opts.Extra); err != nil {
return nil, err
}
diff --git a/sdk/go/types.go b/sdk/go/types.go
index 2425d6f3d..a40c540d0 100644
--- a/sdk/go/types.go
+++ b/sdk/go/types.go
@@ -257,18 +257,22 @@ type WriteOptions struct {
// BatchWriteOperation is one file write in a batch.
type BatchWriteOperation struct {
- URI string `json:"uri"`
- Content *string `json:"content,omitempty"`
- ContentBase64 *string `json:"content_base64,omitempty"`
- Mode string `json:"mode,omitempty"`
+ URI string `json:"uri"`
+ Content *string `json:"content,omitempty"`
+ ContentBase64 *string `json:"content_base64,omitempty"`
+ Mode string `json:"mode,omitempty"`
+ Tags []string `json:"tags,omitempty"`
+ TagMode string `json:"tag_mode,omitempty"`
+ ACL *ACLSpec `json:"acl,omitempty"`
}
// BatchWriteOptions controls BatchWrite.
type BatchWriteOptions struct {
- Wait *bool
- Timeout *float64
- Telemetry any
- Extra map[string]any
+ Wait *bool
+ Timeout *float64
+ Telemetry any
+ ProcessingMode string
+ Extra map[string]any
}
// SetTagsOptions controls SetTags.
diff --git a/sdk/python/openviking_sdk/options.py b/sdk/python/openviking_sdk/options.py
index 63904076c..b017f0971 100644
--- a/sdk/python/openviking_sdk/options.py
+++ b/sdk/python/openviking_sdk/options.py
@@ -103,6 +103,7 @@ class WriteOptions(_ExtraOptions, total=False):
class BatchWriteOptions(_ExtraOptions, total=False):
telemetry: Any
+ processing_mode: ProcessingMode
# Return target-state conflicts without overwriting them or blocking compatible writes.
skip_conflicts: bool
diff --git a/sdk/python/tests/test_async_client_behaviors.py b/sdk/python/tests/test_async_client_behaviors.py
index 7aa90e2bc..521de8eb9 100644
--- a/sdk/python/tests/test_async_client_behaviors.py
+++ b/sdk/python/tests/test_async_client_behaviors.py
@@ -1636,6 +1636,33 @@ async def test_batch_write_http_timeout_outlives_server_wait_timeout():
assert request_timeout.connect == 180.0
+async def test_batch_write_serializes_processing_mode_and_operation_metadata():
+ client = AsyncHTTPClient(url="http://localhost:1933")
+ client._request = AsyncMock(return_value=object())
+ client._handle_response_data = lambda _response: {"result": {}}
+
+ await client.batch_write(
+ "viking://resources/wiki",
+ [
+ {
+ "uri": "viking://resources/wiki/a.md",
+ "content": "new",
+ "tags": ["team=search"],
+ "tag_mode": "append",
+ "acl": {"acl_mode": "restricted", "entries": []},
+ }
+ ],
+ options={"processing_mode": "vectors_only"},
+ )
+
+ payload = client._request.await_args.kwargs["json"]
+ assert payload["processing_mode"] == "vectors_only"
+ assert payload["operations"][0]["tags"] == ["team=search"]
+ assert payload["operations"][0]["tag_mode"] == "append"
+ assert payload["operations"][0]["acl"] == {"acl_mode": "restricted", "entries": []}
+
+
@pytest.mark.asyncio
async def test_watch_routes_support_uri_lookup_and_normalization():
client = AsyncHTTPClient(url="http://localhost:1933")
diff --git a/sdk/typescript/src/client.ts b/sdk/typescript/src/client.ts
index f0f511912..2ac0d6474 100644
--- a/sdk/typescript/src/client.ts
+++ b/sdk/typescript/src/client.ts
@@ -705,11 +705,15 @@ export class OpenVikingClient {
content: operation.content,
content_base64: operation.contentBase64,
mode: operation.mode,
+ tags: operation.tags,
+ tag_mode: operation.tagMode,
+ acl: operation.acl,
}),
),
wait: options.wait,
timeout: options.timeout,
telemetry: options.telemetry,
+ processing_mode: options.processingMode,
});
return this.request("POST", "/api/v1/content/batch-write", {
body: mergeExtra(body, options.extra),
diff --git a/sdk/typescript/src/types.ts b/sdk/typescript/src/types.ts
index d22620ed4..722b29d17 100644
--- a/sdk/typescript/src/types.ts
+++ b/sdk/typescript/src/types.ts
@@ -118,9 +118,13 @@ export interface BatchWriteOperation {
content?: string;
contentBase64?: string;
mode?: "replace" | "append" | "create" | "upsert";
+ tags?: string[];
+ tagMode?: "replace" | "append" | "clear";
+ acl?: AclSpec;
}
/** Batch-write request options. */
export interface BatchWriteOptions extends WaitOptions {
+ processingMode?: ProcessingMode;
extra?: JsonObject;
}
/** Compile request options. */
diff --git a/sdk/typescript/tests/client.test.ts b/sdk/typescript/tests/client.test.ts
index b49d4d2d9..6e57cab77 100644
--- a/sdk/typescript/tests/client.test.ts
+++ b/sdk/typescript/tests/client.test.ts
@@ -553,9 +553,15 @@ describe("OpenVikingClient", () => {
uri: "resources/project/a.txt",
content: "hello",
mode: "upsert",
+ tags: ["team=search"],
+ tagMode: "append",
+ acl: {
+ acl_mode: "restricted",
+ entries: [{ principal: "group:engineering", level: "write" }],
+ },
},
],
- { extra: { future_flag: 0 } },
+ { processingMode: "vectors_only", extra: { future_flag: 0 } },
);
await expect(
client.downloadBytes("resources/project/a.txt"),
@@ -575,8 +581,15 @@ describe("OpenVikingClient", () => {
uri: "viking://resources/project/a.txt",
content: "hello",
mode: "upsert",
+ tags: ["team=search"],
+ tag_mode: "append",
+ acl: {
+ acl_mode: "restricted",
+ entries: [{ principal: "group:engineering", level: "write" }],
+ },
},
]);
+ expect(batchWriteBody.processing_mode).toBe("vectors_only");
expect(
new URL(String(fetcher.mock.calls[1]![0])).searchParams.get("uri"),
).toBe("viking://resources/project/a.txt");
diff --git a/tests/server/test_api_content.py b/tests/server/test_api_content.py
index b6369207e..2bf003551 100644
--- a/tests/server/test_api_content.py
+++ b/tests/server/test_api_content.py
@@ -10,7 +10,14 @@ import pytest
from openviking.server.identity import RequestContext, Role
from openviking.server.routers import content as content_router
-from openviking.server.routers.content import ReindexRequest, WriteContentRequest, reindex
+from openviking.server.routers.content import (
+ BatchWriteOperation,
+ BatchWriteRequest,
+ ReindexRequest,
+ WriteContentRequest,
+ batch_write,
+ reindex,
+)
from openviking_cli.exceptions import PermissionDeniedError
from openviking_cli.session.user_id import UserIdentifier
@@ -54,6 +61,69 @@ def test_write_content_request_accepts_clear_without_tags():
assert request.tag_mode == "clear"
+def test_batch_write_request_accepts_write_parity_options():
+ request = BatchWriteRequest(
+ root_uri="viking://resources/demo",
+ processing_mode="vectors_only",
+ operations=[
+ BatchWriteOperation(
+ uri="viking://resources/demo/a.md",
+ content="updated",
+ tags=["team=search"],
+ tag_mode="append",
+ acl={
+ "acl_mode": "restricted",
+ "entries": [{"principal": "group:engineering", "level": "write"}],
+ },
+ )
+ ],
+ )
+
+ assert request.processing_mode == "vectors_only"
+ assert request.operations[0].tags == ["team=search"]
+ assert request.operations[0].tag_mode == "append"
+ assert request.operations[0].acl.acl_mode == "restricted"
+
+
+async def test_batch_write_forwards_processing_mode_and_operation_metadata(monkeypatch):
+ seen = {}
+
+ async def fake_batch_write(**kwargs):
+ seen.update(kwargs)
+ return {"updated": [kwargs["operations"][0]["uri"]]}
+
+ monkeypatch.setattr(
+ content_router,
+ "get_service",
+ lambda: SimpleNamespace(fs=SimpleNamespace(batch_write=fake_batch_write)),
+ )
+ ctx = RequestContext(user=UserIdentifier("account-1", "user-1"), role=Role.USER)
+
+ await batch_write(
+ BatchWriteRequest(
+ root_uri="viking://resources/demo",
+ processing_mode="vectors_only",
+ operations=[
+ BatchWriteOperation(
+ uri="viking://resources/demo/a.md",
+ content="updated",
+ tags=["team=search"],
+ tag_mode="append",
+ acl={"acl_mode": "restricted", "entries": []},
+ )
+ ],
+ ),
+ ctx,
+ )
+
+ assert seen["processing_mode"] == "vectors_only"
+ operation = seen["operations"][0]
+ assert operation["tags"] == ["team=search"]
+ assert operation["tag_mode"] == "append"
+ assert operation["acl"]["acl_mode"] == "restricted"
+ assert list(operation["acl"]["entries"]) == []
+
+
async def test_write_forwards_processing_mode_to_service(monkeypatch):
seen = {}
diff --git a/tests/server/test_api_content_write.py b/tests/server/test_api_content_write.py
index 9fba532cc..604d3680c 100644
--- a/tests/server/test_api_content_write.py
+++ b/tests/server/test_api_content_write.py
@@ -155,7 +155,6 @@ async def test_write_without_wait_is_immediately_readable(client_with_resource):
)
assert write_resp.status_code == 200
body = write_resp.json()
- assert body["result"]["content_updated"] is True
assert body["result"]["semantic_status"] == "queued"
assert body["result"]["vector_status"] == "queued"
@@ -225,8 +224,8 @@ async def test_api_create_mode_write_then_read(client):
assert read_resp.json()["result"] == "# Hello\n\nWrite-then-read verification."
-async def test_api_create_mode_existing_file_409(client_with_resource):
- """Test create mode on an existing file should return 409."""
+async def test_api_create_mode_existing_file_overwrites(client_with_resource):
+ """create is an upsert: writing an existing file overwrites it, no 409."""
client, uri = client_with_resource
file_uri = await _first_file_uri(client, uri)
@@ -234,15 +233,19 @@ async def test_api_create_mode_existing_file_409(client_with_resource):
"/api/v1/content/write",
json={
"uri": file_uri,
- "content": "new content",
+ "content": "# Overwritten\n\nvia create.",
"mode": "create",
"wait": True,
},
)
- assert resp.status_code == 409
+ assert resp.status_code == 200
body = resp.json()
- assert body["status"] == "error"
- assert body["error"]["code"] == "ALREADY_EXISTS"
+ assert body["status"] == "ok"
+ assert body["result"]["mode"] == "create"
+
+ read_resp = await client.get("/api/v1/content/read", params={"uri": file_uri})
+ assert read_resp.status_code == 200
+ assert read_resp.json()["result"] == "# Overwritten\n\nvia create."
async def test_api_create_mode_invalid_extension_400(client):
diff --git a/tests/server/test_content_batch_write.py b/tests/server/test_content_batch_write.py
new file mode 100644
index 000000000..0478c656e
--- /dev/null
+++ b/tests/server/test_content_batch_write.py
@@ -0,0 +1,1125 @@
+import base64
+import hashlib
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import pytest
+
+import openviking.storage.content_write as content_write_module
+from openviking.server.identity import RequestContext, Role
+from openviking.session.memory.dataclass import MemoryFile
+from openviking.session.memory.utils import MemoryFileUtils
+from openviking.storage.content_write import ContentWriteCoordinator
+from openviking.storage.context_update_plan import (
+ ContentTreeAction,
+ ContextUpdatePlan,
+ FileRefreshIntent,
+)
+from openviking.storage.queuefs.semantic_ops.freshness_policy import FreshnessAction
+from openviking.utils.content_hash import content_md5
+from openviking_cli.exceptions import (
+ InvalidArgumentError,
+ NotFoundError,
+ OpenVikingError,
+)
+from openviking_cli.session.user_id import UserIdentifier
+
+
+class _PathLockClient:
+ def __init__(self):
+ self.held = False
+ self.releases = 0
+
+ async def pathlock_acquire_exact_batch(self, paths):
+ del paths
+ self.held = True
+ return {"lease_ref": "lock-1"}
+
+ async def pathlock_release(self, lease):
+ assert lease == {"lease_ref": "lock-1"}
+ self.held = False
+ self.releases += 1
+
+
+class _VFS:
+ def __init__(self, root, files=None, fail_uri=None):
+ self.root = root
+ self.files = dict(files or {})
+ self.fail_uri = fail_uri
+ self.writes = []
+ self._async_agfs = _PathLockClient()
+
+ async def _ensure_access(self, uri, ctx, *, action):
+ del uri, ctx, action
+
+ def _uri_to_path(self, uri, ctx=None):
+ del ctx
+ return "/virtual/" + uri.removeprefix("viking://")
+
+ async def stat(self, uri, ctx=None, skip_count=False):
+ del ctx
+ assert skip_count is True
+ if uri == self.root:
+ return {"uri": uri, "isDir": True}
+ if uri in self.files:
+ return {"uri": uri, "isDir": False}
+ raise NotFoundError(uri, "file")
+
+ async def read_file(self, uri, ctx=None):
+ del ctx
+ if uri not in self.files:
+ raise NotFoundError(uri, "file")
+ return self.files[uri]
+
+ async def read_file_bytes(self, uri, ctx=None):
+ value = await self.read_file(uri, ctx=ctx)
+ return value.encode() if isinstance(value, str) else value
+
+ async def write_file(self, uri, content, ctx=None, lease_ref=None):
+ del ctx
+ assert lease_ref is not None
+ if uri == self.fail_uri:
+ raise OSError("injected write failure")
+ self.files[uri] = content
+ self.writes.append(uri)
+
+ async def write_file_bytes(self, uri, content, ctx=None, lease_ref=None):
+ del ctx
+ assert lease_ref is not None
+ if uri == self.fail_uri:
+ raise OSError("injected write failure")
+ try:
+ self.files[uri] = content.decode("utf-8")
+ except UnicodeDecodeError:
+ self.files[uri] = content
+ self.writes.append(uri)
+
+
+async def test_batch_accepts_more_than_256_operations(monkeypatch):
+ root = "viking://resources/wiki"
+ vfs = _VFS(root)
+ coordinator = ContentWriteCoordinator(vfs)
+
+ async def refresh(**kwargs):
+ del kwargs
+ return None
+
+ monkeypatch.setattr(coordinator, "_refresh_batch", refresh)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+
+ def operations(prefix, count):
+ return [
+ {
+ "uri": f"{root}/{prefix}-{index}.txt",
+ "content": "x",
+ "mode": "upsert",
+ }
+ for index in range(count)
+ ]
+
+ result = await coordinator.batch_write(
+ root_uri=root,
+ operations=operations("allowed", 256),
+ ctx=ctx,
+ wait=False,
+ )
+ assert len(result["created"]) == 256
+
+ overflow = await coordinator.batch_write(
+ root_uri=root,
+ operations=operations("overflow", 257),
+ ctx=ctx,
+ wait=False,
+ )
+ assert len(overflow["created"]) == 257
+
+
+async def test_batch_create_replace_and_upsert_overwrite_or_create(monkeypatch):
+ root = "viking://resources/wiki"
+ existing = f"{root}/existing.md"
+ created_by_replace = f"{root}/replace.md"
+ created_by_upsert = f"{root}/upsert.md"
+ vfs = _VFS(root, {existing: "newer"})
+ coordinator = ContentWriteCoordinator(vfs)
+
+ async def refresh(**kwargs):
+ del kwargs
+
+ monkeypatch.setattr(coordinator, "_refresh_batch", refresh)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+ result = await coordinator.batch_write(
+ root_uri=root,
+ operations=[
+ {"uri": existing, "content": "replacement", "mode": "create"},
+ {"uri": created_by_replace, "content": "created", "mode": "replace"},
+ {"uri": created_by_upsert, "content": "upsert", "mode": "upsert"},
+ ],
+ ctx=ctx,
+ wait=False,
+ )
+
+ assert result["updated"] == [existing]
+ assert result["created"] == [created_by_replace, created_by_upsert]
+ assert vfs.files == {
+ existing: "replacement",
+ created_by_replace: "created",
+ created_by_upsert: "upsert",
+ }
+
+
+async def test_batch_missing_append_creates_its_initial_body(monkeypatch):
+ root = "viking://resources/wiki"
+ page = f"{root}/page.md"
+ vfs = _VFS(root)
+ coordinator = ContentWriteCoordinator(vfs)
+
+ async def refresh(**kwargs):
+ del kwargs
+
+ monkeypatch.setattr(coordinator, "_refresh_batch", refresh)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+
+ result = await coordinator.batch_write(
+ root_uri=root,
+ operations=[{"uri": page, "content": "initial body", "mode": "append"}],
+ ctx=ctx,
+ wait=False,
+ )
+
+ assert result["created"] == [page]
+ assert result["updated"] == []
+ assert vfs.files[page] == "initial body"
+
+
+async def test_batch_resource_files_compile_independent_rnfv_plans(monkeypatch):
+ root = "viking://resources/wiki"
+ first = f"{root}/docs/a.md"
+ second = f"{root}/docs/b.md"
+ vfs = _VFS(root, {first: "old A"})
+
+ class _VikingDB:
+ def __init__(self):
+ self.calls = []
+
+ async def get_incremental_inventory_by_uris(self, uris, **kwargs):
+ self.calls.append((list(uris), kwargs))
+ return {
+ uri: {
+ "a-l2": {
+ "id": "a-l2",
+ "uri": uri,
+ "level": 2,
+ "md5": "old-md5",
+ "abstract": "old abstract",
+ }
+ }
+ for uri in uris
+ }
+
+ vikingdb = _VikingDB()
+ coordinator = ContentWriteCoordinator(vfs, vikingdb=vikingdb)
+ snapshots = []
+ commits = []
+ direct_actions = []
+ refresh_calls = []
+
+ async def build_snapshot(**kwargs):
+ snapshots.append(kwargs)
+ return SimpleNamespace(
+ request=kwargs["request_intent"],
+ vectors=SimpleNamespace(records_by_id={}),
+ )
+
+ async def build_plan(**kwargs):
+ uri = kwargs["request_intent"].target_uri
+ return ContextUpdatePlan(
+ root_uri=uri,
+ context_type="resource",
+ content_tree_actions=(
+ ContentTreeAction(
+ "upsert",
+ "",
+ old_kind="file",
+ new_kind="file",
+ artifact_path="",
+ md5="planned-md5",
+ ),
+ ),
+ file_refresh=FileRefreshIntent(file_uri=uri, md5="planned-md5"),
+ )
+
+ async def commit(actions, **kwargs):
+ assert vfs._async_agfs.held is True
+ commits.append((actions, kwargs))
+
+ async def enqueue(actions, **kwargs):
+ direct_actions.extend(actions)
+ return False
+
+ async def grouped_refresh(_self, **kwargs):
+ assert vfs._async_agfs.held is False
+ refresh_calls.append(kwargs["changes"])
+ return {
+ "status": "success",
+ "enqueued_count": 1,
+ "parent_actions": {f"{root}/docs": "refresh_now"},
+ }
+
+ monkeypatch.setattr(content_write_module, "build_rnfv_snapshot", build_snapshot)
+ monkeypatch.setattr(coordinator, "_build_deferred_batch_file_plan", build_plan)
+ monkeypatch.setattr(content_write_module, "execute_content_tree_actions", commit)
+ monkeypatch.setattr(content_write_module, "enqueue_direct_index_actions", enqueue)
+ monkeypatch.setattr(content_write_module.Summarizer, "refresh_file_parents", grouped_refresh)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+
+ result = await coordinator.batch_write(
+ root_uri=root,
+ operations=[
+ {"uri": first, "content": "A", "mode": "replace"},
+ {"uri": second, "content": "B", "mode": "upsert"},
+ ],
+ ctx=ctx,
+ wait=False,
+ )
+
+ assert result["updated"] == [first]
+ assert result["created"] == [second]
+ assert vikingdb.calls[0][0] == [first]
+ assert len(snapshots) == 2
+ assert all(snapshot["root_is_file"] is True for snapshot in snapshots)
+ assert all(snapshot["vector_scope"] == "self" for snapshot in snapshots)
+ snapshots_by_uri = {snapshot["target_uri"]: snapshot for snapshot in snapshots}
+ assert snapshots_by_uri[first]["vector_inventory"]["a-l2"]["uri"] == first
+ assert snapshots_by_uri[second]["vector_inventory"] == {}
+ assert len(commits) == 2
+ assert direct_actions == []
+ assert len(refresh_calls) == 1
+ assert {change.file_uri for change in refresh_calls[0]} == {first, second}
+ assert {change.file_uri: change.vector_action for change in refresh_calls[0]} == {
+ first: "",
+ second: "upsert",
+ }
+
+
+async def test_batch_existing_files_defer_full_vector_reads_to_embedding_worker(monkeypatch):
+ root = "viking://resources/wiki"
+ operations = [
+ {"uri": f"{root}/docs/{index}.md", "content": f"new {index}", "mode": "replace"}
+ for index in range(64)
+ ]
+ vfs = _VFS(
+ root, {operation["uri"]: f"old {index}" for index, operation in enumerate(operations)}
+ )
+
+ class _VikingDB:
+ def __init__(self):
+ self.inventory_calls = []
+ self.hydrate_incremental_records = AsyncMock(
+ side_effect=AssertionError("batch preparation must not hydrate full vector records")
+ )
+
+ async def get_incremental_inventory_by_uris(self, uris, **kwargs):
+ self.inventory_calls.append((list(uris), kwargs))
+ return {
+ uri: {
+ f"{index}-l2": {
+ "id": f"{index}-l2",
+ "uri": uri,
+ "level": 2,
+ "md5": content_md5(f"old {index}".encode()),
+ "abstract": f"old abstract {index}",
+ }
+ }
+ for index, uri in enumerate(uris)
+ }
+
+ vikingdb = _VikingDB()
+ coordinator = ContentWriteCoordinator(vfs, vikingdb=vikingdb)
+ monkeypatch.setattr(
+ content_write_module.Summarizer,
+ "refresh_file_parents",
+ AsyncMock(
+ return_value={"enqueued_count": 1, "parent_actions": {f"{root}/docs": "refresh_now"}}
+ ),
+ )
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+
+ result = await coordinator.batch_write(
+ root_uri=root,
+ operations=operations,
+ ctx=ctx,
+ wait=False,
+ processing_mode="semantic_and_vectors",
+ )
+
+ assert result["updated"] == sorted(operation["uri"] for operation in operations)
+ assert len(vikingdb.inventory_calls) == 1
+ assert vikingdb.inventory_calls[0][0] == sorted(operation["uri"] for operation in operations)
+ vikingdb.hydrate_incremental_records.assert_not_awaited()
+
+
+async def test_batch_same_content_with_missing_abstract_enqueues_file_refresh(monkeypatch):
+ root = "viking://resources/wiki"
+ uri = f"{root}/docs/a.md"
+ vfs = _VFS(root, {uri: "unchanged"})
+
+ class _VikingDB:
+ async def get_incremental_inventory_by_uris(self, uris, **kwargs):
+ del kwargs
+ return {
+ item: {
+ "a-l2": {
+ "id": "a-l2",
+ "uri": item,
+ "level": 2,
+ "md5": content_md5(b"unchanged"),
+ "abstract": "",
+ }
+ }
+ for item in uris
+ }
+
+ refresh_file_parents = AsyncMock(
+ return_value={"enqueued_count": 1, "parent_actions": {f"{root}/docs": "refresh_now"}}
+ )
+ monkeypatch.setattr(
+ content_write_module.Summarizer,
+ "refresh_file_parents",
+ refresh_file_parents,
+ )
+ coordinator = ContentWriteCoordinator(vfs, vikingdb=_VikingDB())
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+
+ await coordinator.batch_write(
+ root_uri=root,
+ operations=[{"uri": uri, "content": "unchanged", "mode": "replace"}],
+ ctx=ctx,
+ wait=False,
+ processing_mode="semantic_and_vectors",
+ )
+
+ changes = refresh_file_parents.await_args.kwargs["changes"]
+ assert [(change.file_uri, change.md5) for change in changes] == [
+ (uri, content_md5(b"unchanged"))
+ ]
+
+
+async def test_batch_vectors_only_existing_file_enqueues_merge_patch_without_hydration(monkeypatch):
+ root = "viking://resources/wiki"
+ uri = f"{root}/a.md"
+ vfs = _VFS(root, {uri: "old"})
+
+ class _VikingDB:
+ hydrate_incremental_records = AsyncMock(
+ side_effect=AssertionError("batch preparation must not hydrate full vector records")
+ )
+
+ async def get_incremental_inventory_by_uris(self, uris, **kwargs):
+ del kwargs
+ return {
+ uri: {
+ "existing-l2": {
+ "id": "existing-l2",
+ "uri": uri,
+ "level": 2,
+ "md5": content_md5(b"old"),
+ "abstract": "old abstract",
+ }
+ }
+ for uri in uris
+ }
+
+ actions = []
+
+ async def enqueue(direct_actions, **kwargs):
+ del kwargs
+ actions.extend(direct_actions)
+ return True
+
+ monkeypatch.setattr(content_write_module, "enqueue_direct_index_actions", enqueue)
+ coordinator = ContentWriteCoordinator(vfs, vikingdb=_VikingDB())
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+
+ await coordinator.batch_write(
+ root_uri=root,
+ operations=[
+ {
+ "uri": uri,
+ "content": "new",
+ "tags": ["team=search"],
+ "tag_mode": "append",
+ }
+ ],
+ ctx=ctx,
+ wait=False,
+ processing_mode="vectors_only",
+ )
+
+ assert len(actions) == 1
+ action = actions[0]
+ assert action.action.value == "merge"
+ assert action.record_id == "existing-l2"
+ assert action.field_patch is not None
+ assert action.field_patch.values == {"search_tags": ["team=search"]}
+ assert action.field_patch.modes == {"search_tags": "append"}
+
+
+async def test_batch_resource_plan_uses_operation_processing_mode_and_tags(monkeypatch):
+ root = "viking://resources/wiki"
+ uri = f"{root}/a.md"
+ vfs = _VFS(root, {uri: "old"})
+
+ class _VikingDB:
+ async def get_incremental_inventory_by_uris(self, uris, **kwargs):
+ del kwargs
+ return {item: {} for item in uris}
+
+ coordinator = ContentWriteCoordinator(vfs, vikingdb=_VikingDB())
+ seen = {}
+
+ async def build_snapshot(**kwargs):
+ seen["request"] = kwargs["request_intent"]
+ return SimpleNamespace(
+ request=kwargs["request_intent"], vectors=SimpleNamespace(records_by_id={})
+ )
+
+ async def build_plan(**kwargs):
+ del kwargs
+ return ContextUpdatePlan(uri, "resource")
+
+ monkeypatch.setattr(content_write_module, "build_rnfv_snapshot", build_snapshot)
+ monkeypatch.setattr(coordinator, "_build_deferred_batch_file_plan", build_plan)
+ monkeypatch.setattr(coordinator, "_refresh_batch", AsyncMock(return_value=None))
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+
+ await coordinator.batch_write(
+ root_uri=root,
+ operations=[
+ {
+ "uri": uri,
+ "content": "new",
+ "mode": "replace",
+ "tags": ["team=search"],
+ "tag_mode": "append",
+ }
+ ],
+ ctx=ctx,
+ wait=False,
+ processing_mode="vectors_only",
+ )
+
+ request = seen["request"]
+ assert request.processing_mode == "vectors_only"
+ assert request.scalar_intents[0].value == ("team=search",)
+ assert request.scalar_intents[0].mode == "append"
+
+
+async def test_batch_resource_plan_allows_clear_tags_without_tags(monkeypatch):
+ root = "viking://resources/wiki"
+ uri = f"{root}/a.md"
+ vfs = _VFS(root, {uri: "old"})
+
+ class _VikingDB:
+ async def get_incremental_inventory_by_uris(self, uris, **kwargs):
+ del kwargs
+ return {item: {} for item in uris}
+
+ coordinator = ContentWriteCoordinator(vfs, vikingdb=_VikingDB())
+ seen = {}
+
+ async def build_snapshot(**kwargs):
+ seen["request"] = kwargs["request_intent"]
+ return SimpleNamespace(
+ request=kwargs["request_intent"], vectors=SimpleNamespace(records_by_id={})
+ )
+
+ async def build_plan(**kwargs):
+ del kwargs
+ return ContextUpdatePlan(uri, "resource")
+
+ monkeypatch.setattr(content_write_module, "build_rnfv_snapshot", build_snapshot)
+ monkeypatch.setattr(coordinator, "_build_deferred_batch_file_plan", build_plan)
+ monkeypatch.setattr(coordinator, "_refresh_batch", AsyncMock(return_value=None))
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+
+ await coordinator.batch_write(
+ root_uri=root,
+ operations=[{"uri": uri, "content": "new", "tag_mode": "clear"}],
+ ctx=ctx,
+ wait=False,
+ )
+
+ assert seen["request"].scalar_intents[0].value == ()
+ # The public ``clear`` spelling reaches RNFV as its backend-compatible
+ # equivalent: replace the scalar field with an empty tag list.
+ assert seen["request"].scalar_intents[0].mode == "replace"
+
+
+async def test_batch_applies_operation_acl_and_forwards_file_options(monkeypatch):
+ root = "viking://resources/wiki"
+ uri = f"{root}/a.md"
+ vfs = _VFS(root)
+ prepared_update = object()
+ vfs.prepare_acl_update = AsyncMock(return_value=prepared_update)
+ vfs.acl_manager = SimpleNamespace(apply_indexed_update=AsyncMock())
+ coordinator = ContentWriteCoordinator(vfs)
+ refresh = AsyncMock(return_value=None)
+ monkeypatch.setattr(coordinator, "_refresh_batch", refresh)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+
+ await coordinator.batch_write(
+ root_uri=root,
+ operations=[
+ {
+ "uri": uri,
+ "content": "new",
+ "tags": ["team=search"],
+ "tag_mode": "append",
+ "acl": {"acl_mode": "restricted", "entries": []},
+ }
+ ],
+ ctx=ctx,
+ wait=False,
+ )
+
+ vfs.prepare_acl_update.assert_awaited_once()
+ vfs.acl_manager.apply_indexed_update.assert_awaited_once_with(prepared_update, ctx)
+ file_options = refresh.await_args.kwargs["file_ingest_options"]
+ assert file_options[uri].search_tags == ["team=search"]
+ assert file_options[uri].search_tag_mode == "append"
+ assert file_options[uri].acl_update is prepared_update
+
+
+async def test_batch_same_content_with_tags_is_not_skipped_as_a_conflict_noop(monkeypatch):
+ root = "viking://resources/wiki"
+ uri = f"{root}/a.md"
+ vfs = _VFS(root, {uri: "same"})
+ coordinator = ContentWriteCoordinator(vfs)
+ refresh = AsyncMock(return_value=None)
+ monkeypatch.setattr(coordinator, "_refresh_batch", refresh)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+
+ await coordinator.batch_write(
+ root_uri=root,
+ operations=[
+ {
+ "uri": uri,
+ "content": "same",
+ "expected_sha256": hashlib.sha256(b"same").hexdigest(),
+ "tags": ["team=search"],
+ }
+ ],
+ ctx=ctx,
+ wait=False,
+ skip_conflicts=True,
+ )
+
+ assert vfs.writes == [uri]
+ assert refresh.await_args.kwargs["file_ingest_options"][uri].search_tags == ["team=search"]
+
+
+async def test_batch_releases_file_locks_before_one_aggregated_refresh(monkeypatch):
+ root = "viking://resources/wiki"
+ a = f"{root}/a.md"
+ b = f"{root}/b.md"
+ vfs = _VFS(root)
+ locks = vfs._async_agfs
+ coordinator = ContentWriteCoordinator(vfs)
+ calls = []
+
+ async def refresh(**kwargs):
+ assert locks.held is False
+ calls.append(kwargs["refresh_kinds"])
+ return {"Semantic": {"processed": 1, "error_count": 0}}
+
+ monkeypatch.setattr(coordinator, "_refresh_batch", refresh)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+ result = await coordinator.batch_write(
+ root_uri=root,
+ operations=[
+ {"uri": b, "content": "B", "mode": "upsert"},
+ {"uri": a, "content": "A", "mode": "upsert"},
+ ],
+ ctx=ctx,
+ wait=False,
+ )
+ assert vfs.writes == [a, b]
+ assert result["created"] == [a, b]
+ assert calls == [{a: "added", b: "added"}]
+ assert locks.releases == 1
+
+
+async def test_batch_passes_final_md5s_and_old_abstracts_to_refresh(monkeypatch):
+ root = "viking://resources/wiki"
+ existing = f"{root}/existing.md"
+ created = f"{root}/new.md"
+ vfs = _VFS(root, {existing: "old"})
+
+ class _VikingDB:
+ async def get_l2_diff_records_by_uris(self, uris, *, ctx):
+ assert uris == [existing]
+ return {existing: {"abstract": "old abstract", "md5": "old-md5"}}
+
+ coordinator = ContentWriteCoordinator(vfs, vikingdb=_VikingDB())
+ captured = {}
+
+ async def refresh(**kwargs):
+ captured.update(kwargs)
+ return None
+
+ monkeypatch.setattr(coordinator, "_refresh_batch", refresh)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+
+ await coordinator.batch_write(
+ root_uri=root,
+ operations=[
+ {"uri": existing, "content": "new", "mode": "replace"},
+ {"uri": created, "content": "created", "mode": "create"},
+ ],
+ ctx=ctx,
+ wait=False,
+ )
+
+ assert captured["file_md5s"] == {
+ existing: content_md5(b"new"),
+ created: content_md5(b"created"),
+ }
+ assert captured["file_abstracts"] == {existing: "old abstract"}
+
+
+async def test_batch_replace_memory_preserves_metadata(monkeypatch):
+ root = "viking://user/default/memories/preferences"
+ memory_uri = f"{root}/theme.md"
+ metadata = {
+ "tags": ["ui", "preference"],
+ "fields": {"topic": "theme"},
+ "version": 7,
+ }
+ original = MemoryFileUtils.write(
+ MemoryFile(content="Original preference", extra_fields=metadata)
+ )
+ expected = MemoryFileUtils.read(original, uri=memory_uri)
+ vfs = _VFS(root, {memory_uri: original})
+ coordinator = ContentWriteCoordinator(vfs)
+
+ async def refresh(**kwargs):
+ del kwargs
+ return None
+
+ monkeypatch.setattr(coordinator, "_refresh_batch", refresh)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+
+ result = await coordinator.batch_write(
+ root_uri=root,
+ operations=[
+ {
+ "uri": memory_uri,
+ "content": "Updated preference",
+ "mode": "replace",
+ }
+ ],
+ ctx=ctx,
+ wait=False,
+ )
+
+ stored = MemoryFileUtils.read(vfs.files[memory_uri], uri=memory_uri)
+ assert result["updated"] == [memory_uri]
+ assert stored.content == "Updated preference"
+ assert stored.extra_fields == expected.extra_fields
+
+
+async def test_batch_reports_skipped_directory_and_queued_file_vectors(monkeypatch):
+ root = "viking://resources/wide"
+ page = f"{root}/page.md"
+ coordinator = ContentWriteCoordinator(_VFS(root))
+
+ async def enqueue(**kwargs):
+ del kwargs
+ return FreshnessAction.NOOP
+
+ monkeypatch.setattr(coordinator, "_enqueue_semantic_refresh_changes", enqueue)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+
+ result = await coordinator.batch_write(
+ root_uri=root,
+ operations=[
+ {
+ "uri": page,
+ "content": "content",
+ "mode": "create",
+ }
+ ],
+ ctx=ctx,
+ wait=False,
+ )
+
+ assert result["semantic_status"] == "skipped"
+ assert result["vector_status"] == "queued"
+ assert result["queue_status"] is None
+
+
+async def test_batch_upserts_binary_content(monkeypatch):
+ root = "viking://resources/wiki"
+ image = f"{root}/figure.png"
+ original = b"\x89PNG\r\n\x1a\nold"
+ replacement = b"\x89PNG\r\n\x1a\nnew"
+ vfs = _VFS(root, {image: original})
+ coordinator = ContentWriteCoordinator(vfs)
+
+ refresh_calls = []
+
+ async def refresh(**kwargs):
+ refresh_calls.append(kwargs)
+ return None
+
+ monkeypatch.setattr(coordinator, "_refresh_batch", refresh)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+ operation = {
+ "uri": image,
+ "content_base64": base64.b64encode(replacement).decode(),
+ "mode": "upsert",
+ }
+ result = await coordinator.batch_write(
+ root_uri=root, operations=[operation], ctx=ctx, wait=False
+ )
+ assert result["updated"] == [image]
+ assert vfs.files[image] == replacement
+ assert refresh_calls[0]["file_md5s"] == {image: content_md5(replacement)}
+
+ retry = await coordinator.batch_write(
+ root_uri=root, operations=[operation], ctx=ctx, wait=False
+ )
+ assert retry["updated"] == [image]
+
+
+async def test_batch_rejects_invalid_binary_payload_and_memory_binary():
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+ resource_root = "viking://resources/wiki"
+ coordinator = ContentWriteCoordinator(_VFS(resource_root))
+ with pytest.raises(InvalidArgumentError, match="content_base64 is invalid"):
+ await coordinator.batch_write(
+ root_uri=resource_root,
+ operations=[
+ {
+ "uri": f"{resource_root}/figure.png",
+ "content_base64": "not base64!",
+ "mode": "upsert",
+ }
+ ],
+ ctx=ctx,
+ wait=False,
+ )
+
+ memory_root = "viking://user/default/memories/preferences/wiki"
+ coordinator = ContentWriteCoordinator(_VFS(memory_root))
+ with pytest.raises(InvalidArgumentError, match="not supported for memories"):
+ await coordinator.batch_write(
+ root_uri=memory_root,
+ operations=[
+ {
+ "uri": f"{memory_root}/figure.png",
+ "content_base64": base64.b64encode(b"png").decode(),
+ "mode": "upsert",
+ }
+ ],
+ ctx=ctx,
+ wait=False,
+ )
+
+
+async def test_batch_partial_failure_refreshes_successful_files_and_retry_is_safe(monkeypatch):
+ root = "viking://resources/wiki"
+ a = f"{root}/a.md"
+ b = f"{root}/b.md"
+ vfs = _VFS(root, fail_uri=b)
+ locks = vfs._async_agfs
+ coordinator = ContentWriteCoordinator(vfs)
+ calls = []
+
+ async def refresh(**kwargs):
+ assert locks.held is False
+ calls.append(dict(kwargs["refresh_kinds"]))
+
+ monkeypatch.setattr(coordinator, "_refresh_batch", refresh)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+ operations = [
+ {"uri": a, "content": "A", "mode": "upsert"},
+ {"uri": b, "content": "B", "mode": "upsert"},
+ ]
+ with pytest.raises(OSError, match="injected"):
+ await coordinator.batch_write(root_uri=root, operations=operations, ctx=ctx, wait=False)
+ assert calls == [{a: "added"}]
+
+ vfs.fail_uri = None
+ result = await coordinator.batch_write(
+ root_uri=root, operations=operations, ctx=ctx, wait=False
+ )
+ assert result["updated"] == [a]
+ assert result["created"] == [b]
+ assert calls[-1] == {a: "modified", b: "added"}
+
+
+async def test_batch_refresh_failure_retry_rewrites_and_refreshes(monkeypatch):
+ root = "viking://resources/wiki"
+ page = f"{root}/page.md"
+ vfs = _VFS(root)
+ coordinator = ContentWriteCoordinator(vfs)
+ calls = []
+
+ async def refresh(**kwargs):
+ calls.append(dict(kwargs["refresh_kinds"]))
+ if len(calls) == 1:
+ raise RuntimeError("injected refresh failure")
+
+ monkeypatch.setattr(coordinator, "_refresh_batch", refresh)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+ operations = [
+ {
+ "uri": page,
+ "content": "landed",
+ "mode": "upsert",
+ }
+ ]
+
+ with pytest.raises(OpenVikingError) as error:
+ await coordinator.batch_write(root_uri=root, operations=operations, ctx=ctx, wait=False)
+ assert error.value.code == "REFRESH_FAILED"
+ assert "injected refresh failure" in str(error.value)
+ assert "do not blindly retry an already-successful append" in str(error.value)
+ assert error.value.details["created"] == [page]
+ assert vfs.files[page] == "landed"
+ assert vfs.writes == [page]
+
+ result = await coordinator.batch_write(
+ root_uri=root, operations=operations, ctx=ctx, wait=False
+ )
+ assert result["updated"] == [page]
+ assert vfs.writes == [page, page]
+ assert calls == [{page: "added"}, {page: "modified"}]
+
+
+async def test_batch_refresh_groups_resource_and_memory_work(monkeypatch):
+ coordinator = ContentWriteCoordinator(_VFS("viking://resources/wiki"), vikingdb=object())
+ semantic_calls = []
+ overview_calls = []
+ embedding_calls = []
+
+ async def resolve_root(uri, **kwargs):
+ del uri, kwargs
+ return "viking://resources/wiki"
+
+ async def enqueue(**kwargs):
+ semantic_calls.append(kwargs)
+ return FreshnessAction.MARK_PENDING
+
+ async def overview(**kwargs):
+ overview_calls.append(kwargs)
+
+ async def embedding(**kwargs):
+ embedding_calls.append(kwargs)
+ return False
+
+ monkeypatch.setattr(coordinator, "_resolve_root_uri", resolve_root)
+ monkeypatch.setattr(coordinator, "_enqueue_semantic_refresh_changes", enqueue)
+ monkeypatch.setattr(content_write_module.MemoryUpdater, "refresh_schema_overview", overview)
+ monkeypatch.setattr(content_write_module.MemoryUpdater, "refresh_file_embedding", embedding)
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+ outcome = await coordinator._refresh_batch(
+ refresh_kinds={
+ "viking://resources/wiki/a.md": "added",
+ "viking://resources/wiki/b.md": "modified",
+ "viking://user/default/memories/preferences/wiki/a.md": "added",
+ "viking://user/default/memories/preferences/wiki/b.md": "modified",
+ },
+ ctx=ctx,
+ wait=False,
+ timeout=None,
+ telemetry_id="",
+ )
+ assert len(semantic_calls) == 1
+ assert semantic_calls[0]["changes"] == {
+ "added": ["viking://resources/wiki/a.md"],
+ "modified": ["viking://resources/wiki/b.md"],
+ }
+ assert len(overview_calls) == 1
+ assert overview_calls[0]["strict"] is True
+ assert len(embedding_calls) == 2
+ assert outcome.statuses(wait=False) == ("deferred", "queued")
+ assert all(call["strict"] is True for call in embedding_calls)
+
+
+async def test_batch_write_api_upserts_files(client_with_resource):
+ client, root = client_with_resource
+ listing = await client.get(
+ "/api/v1/fs/ls",
+ params={"uri": root, "simple": True, "recursive": True},
+ )
+ existing = listing.json()["result"][0]
+ created = f"{root}/compile-batch-created.md"
+ operations = [
+ {
+ "uri": existing,
+ "content": "# Batch updated",
+ "mode": "upsert",
+ },
+ {
+ "uri": created,
+ "content": "# Batch created",
+ "mode": "upsert",
+ },
+ ]
+ first = await client.post(
+ "/api/v1/content/batch-write",
+ json={"root_uri": root, "operations": operations, "wait": False},
+ )
+ assert first.status_code == 200
+ assert first.json()["result"]["updated"] == [existing]
+ assert first.json()["result"]["created"] == [created]
+
+ retry = await client.post(
+ "/api/v1/content/batch-write",
+ json={"root_uri": root, "operations": operations, "wait": False},
+ )
+ assert retry.status_code == 200
+ assert retry.json()["result"]["updated"] == sorted([existing, created])
+
+
+ ("filename", "content"),
+ [
+ ("compile-figure.png", b"\x89PNG\r\n\x1a\ncompile"),
+ ("compile-report.pdf", b"%PDF-1.7\ncompile"),
+ ],
+)
+async def test_batch_write_api_creates_binary_file(client_with_resource, filename, content):
+ client, root = client_with_resource
+ artifact = f"{root}/{filename}"
+ response = await client.post(
+ "/api/v1/content/batch-write",
+ json={
+ "root_uri": root,
+ "wait": False,
+ "operations": [
+ {
+ "uri": artifact,
+ "content_base64": base64.b64encode(content).decode(),
+ "mode": "upsert",
+ }
+ ],
+ },
+ )
+ assert response.status_code == 200
+ assert response.json()["result"]["created"] == [artifact]
+ downloaded = await client.get("/api/v1/content/download", params={"uri": artifact})
+ assert downloaded.status_code == 200
+ assert downloaded.content == content
+
+
+async def test_batch_write_api_create_overwrites_and_batch_continues(client_with_resource):
+ client, root = client_with_resource
+ listing = await client.get(
+ "/api/v1/fs/ls",
+ params={"uri": root, "simple": True, "recursive": True},
+ )
+ existing = listing.json()["result"][0]
+ created = f"{root}/compile-created-by-upsert.md"
+ response = await client.post(
+ "/api/v1/content/batch-write",
+ json={
+ "root_uri": root,
+ "wait": False,
+ "operations": [
+ {
+ "uri": existing,
+ "content": "create aliases replace",
+ "mode": "create",
+ },
+ {
+ "uri": created,
+ "content": "created by upsert",
+ "mode": "upsert",
+ },
+ ],
+ },
+ )
+ assert response.status_code == 200
+ result = response.json()["result"]
+ assert result["updated"] == [existing]
+ assert result["created"] == [created]
+ existing_body = await client.get("/api/v1/content/read", params={"uri": existing, "raw": True})
+ assert existing_body.status_code == 200
+ assert existing_body.json()["result"] == "create aliases replace"
+ created_body = await client.get("/api/v1/content/read", params={"uri": created, "raw": True})
+ assert created_body.status_code == 200
+ assert created_body.json()["result"] == "created by upsert"
+
+
+async def test_batch_write_api_missing_append_creates_file(client_with_resource):
+ client, root = client_with_resource
+ target = f"{root}/compile-created-by-append.md"
+
+ response = await client.post(
+ "/api/v1/content/batch-write",
+ json={
+ "root_uri": root,
+ "wait": False,
+ "operations": [
+ {
+ "uri": target,
+ "content": "initial append body",
+ "mode": "append",
+ }
+ ],
+ },
+ )
+
+ assert response.status_code == 200
+ assert response.json()["result"]["created"] == [target]
+ body = await client.get("/api/v1/content/read", params={"uri": target, "raw": True})
+ assert body.status_code == 200
+ assert body.json()["result"] == "initial append body"
+
+
+async def test_batch_write_rejects_path_traversal(client_with_resource):
+ client, root = client_with_resource
+ response = await client.post(
+ "/api/v1/content/batch-write",
+ json={
+ "root_uri": root,
+ "wait": False,
+ "operations": [
+ {
+ "uri": f"{root}/../escaped.md",
+ "content": "escape",
+ "mode": "upsert",
+ }
+ ],
+ },
+ )
+ assert response.status_code == 400
+ assert response.json()["error"]["code"] == "INVALID_ARGUMENT"
diff --git a/tests/server/test_content_write_service.py b/tests/server/test_content_write_service.py
index 1fd4b3c37..ef73f2971 100644
--- a/tests/server/test_content_write_service.py
+++ b/tests/server/test_content_write_service.py
@@ -4,7 +4,6 @@
"""Service-level tests for content write coordination."""
from types import SimpleNamespace
-from unittest.mock import AsyncMock
import pytest
@@ -16,8 +15,6 @@ from openviking.storage.acl import AclMode
from openviking.storage.content_write import ContentWriteCoordinator
from openviking.storage.errors import LockAcquisitionError, ResourceBusyError
from openviking_cli.exceptions import (
- AlreadyExistsError,
- DeadlineExceededError,
InvalidArgumentError,
NotFoundError,
PermissionDeniedError,
@@ -280,9 +277,7 @@ async def test_shared_resource_creation_inherits_acl_and_preserves_plain_append(
assert (await service.fs.get_acl(explicit_dir, ctx=admin))[
"direct_entries"
] == inherited_entries
- await service.fs.write(
- explicit_file, "first", ctx=admin, wait=True, acl={"entries": []}
- )
+ await service.fs.write(explicit_file, "first", ctx=admin, wait=True, acl={"entries": []})
assert (await service.fs.get_acl(explicit_file, ctx=admin))["effective_entries"] == []
explicit_import = "viking://resources/explicit_import"
@@ -293,9 +288,7 @@ async def test_shared_resource_creation_inherits_acl_and_preserves_plain_append(
assert (await service.fs.get_acl(explicit_import, ctx=admin))[
"direct_entries"
] == inherited_entries
- imported_children = (
- await service.fs.ls(explicit_import, ctx=admin, simple=True)
- ).entries
+ imported_children = (await service.fs.ls(explicit_import, ctx=admin, simple=True)).entries
for child in imported_children:
report = await service.fs.get_acl(child, ctx=admin)
assert report["direct_entries"] == []
@@ -633,67 +626,6 @@ class _FakeQueueManager:
return self.queue
-async def test_resource_write_semantic_refresh_uses_coalesce_key(monkeypatch):
- file_uri = "viking://resources/demo/doc.md"
- root_uri = "viking://resources/demo"
- ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
- queue = _FakeSemanticQueue()
- coordinator = ContentWriteCoordinator(
- viking_fs=_FakeVikingFS(file_uri=file_uri, root_uri=root_uri)
- )
-
- monkeypatch.setattr(
- "openviking.storage.content_write.get_queue_manager",
- lambda: _FakeQueueManager(queue),
- )
-
- await coordinator._enqueue_semantic_refresh(
- root_uri=root_uri,
- changed_uri=file_uri,
- context_type="resource",
- ctx=ctx,
- )
-
- assert len(queue.messages) == 1
- assert queue.messages[0].coalesce_key == (
- "resource|default|default|default|viking://resources/demo"
- )
- assert queue.messages[0].lock_handoff is None
-
-
-async def test_write_timeout_after_enqueue_releases_resource_lock(monkeypatch):
- file_uri = "viking://resources/demo/doc.md"
- root_uri = "viking://resources/demo"
- ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
- viking_fs = _FakeVikingFS(file_uri=file_uri, root_uri=root_uri)
- coordinator = ContentWriteCoordinator(viking_fs=viking_fs)
-
- async def _fake_enqueue_semantic_refresh(**kwargs):
- del kwargs
- return None
-
- async def _fake_wait_for_request(*, telemetry_id, timeout):
- del telemetry_id
- raise DeadlineExceededError("queue processing", timeout)
-
- monkeypatch.setattr(coordinator, "_enqueue_semantic_refresh", _fake_enqueue_semantic_refresh)
- monkeypatch.setattr(coordinator, "_wait_for_request", _fake_wait_for_request)
-
- with pytest.raises(DeadlineExceededError):
- await coordinator.write(
- uri=file_uri,
- content="updated",
- ctx=ctx,
- wait=True,
- )
-
- assert viking_fs._async_agfs.release_calls == ["lock-1"]
- assert viking_fs.delete_temp_calls == []
- assert viking_fs.content[file_uri] == "updated"
-
-
@pytest.mark.asyncio
async def test_resource_write_lock_conflict_raises_resource_busy(monkeypatch):
file_uri = "viking://resources/demo/doc.md"
@@ -760,129 +692,6 @@ async def test_write_lock_storage_error_is_not_mapped_to_resource_busy(file_uri)
)
-async def test_write_direct_reuses_outer_lease_for_viking_fs(monkeypatch):
- file_uri = "viking://resources/demo/doc.md"
- root_uri = "viking://resources/demo"
- ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
- viking_fs = _FakeVikingFS(file_uri=file_uri, root_uri=root_uri)
- coordinator = ContentWriteCoordinator(viking_fs=viking_fs)
-
- async def _fake_enqueue_semantic_refresh(**kwargs):
- del kwargs
- return None
-
- monkeypatch.setattr(coordinator, "_enqueue_semantic_refresh", _fake_enqueue_semantic_refresh)
-
- result = await coordinator._write_direct_with_refresh(
- uri=file_uri,
- root_uri=root_uri,
- content="updated",
- mode="replace",
- context_type="resource",
- wait=False,
- timeout=None,
- ctx=ctx,
- written_bytes=len("updated".encode("utf-8")),
- telemetry_id="",
- )
-
- assert result["uri"] == file_uri
- assert viking_fs.write_file_calls[0][0:2] == (file_uri, "updated")
- assert viking_fs.write_file_calls[0][2].id == "lock-1"
-
-
[email protected]("wait", [False, True])
-async def test_resource_write_skips_busy_parent_and_keeps_file_work(monkeypatch, wait):
- file_uri = "viking://resources/demo/doc.md"
- root_uri = file_uri.rsplit("/", 1)[0]
- ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
- viking_fs = _FakeVikingFS(file_uri=file_uri, root_uri=root_uri)
- coordinator = ContentWriteCoordinator(viking_fs=viking_fs)
- queue = _FakeSemanticQueue()
- monkeypatch.setattr(
- "openviking.storage.content_write.get_queue_manager", lambda: _FakeQueueManager(queue)
- )
- plan = AsyncMock(side_effect=LockAcquisitionError("parent sidecars are busy"))
- monkeypatch.setattr("openviking.storage.content_write.plan_abstract_overview_refresh", plan)
- monkeypatch.setattr(
- "openviking.storage.content_write.get_openviking_config",
- lambda: SimpleNamespace(semantic=SimpleNamespace()),
- )
- monkeypatch.setattr(coordinator, "_wait_for_request", AsyncMock(return_value=None))
-
- result = await coordinator.write(uri=file_uri, content="updated", ctx=ctx, wait=wait)
-
- assert plan.await_args.kwargs["force_refresh"] is wait
- assert viking_fs.content[file_uri] == "updated"
- assert result["content_updated"] is True
- assert result["semantic_status"] == "skipped"
- assert result["vector_status"] == ("complete" if wait else "queued")
- assert len(queue.messages) == 1
- assert queue.messages[0].changes == {"modified": [file_uri]}
- assert queue.messages[0].aggregate_directory is False
- assert viking_fs._async_agfs.release_calls == ["lock-1"]
-
-
-async def test_resource_write_rolls_back_replace_when_enqueue_fails(monkeypatch):
- file_uri = "viking://resources/demo/doc.md"
- root_uri = "viking://resources/demo"
- ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
- viking_fs = _FakeVikingFS(file_uri=file_uri, root_uri=root_uri)
- coordinator = ContentWriteCoordinator(viking_fs=viking_fs)
-
- async def _fail_enqueue(**kwargs):
- del kwargs
- raise RuntimeError("queue unavailable")
-
- monkeypatch.setattr(coordinator, "_enqueue_semantic_refresh", _fail_enqueue)
-
- with pytest.raises(RuntimeError, match="queue unavailable"):
- await coordinator.write(
- uri=file_uri,
- content="updated",
- ctx=ctx,
- mode="replace",
- )
-
- assert viking_fs.content[file_uri] == "original"
- assert [call[0:2] for call in viking_fs.write_file_calls] == [
- (file_uri, "updated"),
- (file_uri, "original"),
- ]
- assert [call[2].id for call in viking_fs.write_file_calls] == ["lock-1", "lock-1"]
- assert viking_fs._async_agfs.release_calls == ["lock-1"]
-
-
-async def test_resource_write_rolls_back_create_when_enqueue_fails(monkeypatch):
- file_uri = "viking://resources/demo/new.md"
- root_uri = "viking://resources/demo"
- ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
- viking_fs = _FakeVikingFSForCreate(file_uri=file_uri, root_uri=root_uri, file_exists=False)
- coordinator = ContentWriteCoordinator(viking_fs=viking_fs)
-
- async def _fail_enqueue(**kwargs):
- del kwargs
- raise RuntimeError("queue unavailable")
-
- monkeypatch.setattr(coordinator, "_enqueue_semantic_refresh", _fail_enqueue)
-
- with pytest.raises(RuntimeError, match="queue unavailable"):
- await coordinator.write(
- uri=file_uri,
- content="new content",
- ctx=ctx,
- mode="create",
- )
-
- assert file_uri not in viking_fs.content
- assert viking_fs.rm_calls == [file_uri]
- assert viking_fs._async_agfs.release_calls == ["lock-1"]
-
-
@pytest.mark.asyncio
async def test_memory_write_wait_skips_semantic_queue_and_releases_write_lock(monkeypatch):
file_uri = "viking://user/default/memories/preferences/theme.md"
@@ -893,7 +702,7 @@ async def test_memory_write_wait_skips_semantic_queue_and_releases_write_lock(mo
async def _fake_write_in_place(uri, content, *, mode, ctx, lock_handle=None, lease_ref=None):
del uri, content, mode, ctx, lock_handle, lease_ref
- return None
+ return b"updated"
async def _fail_wait_for_request(*, telemetry_id, timeout):
del telemetry_id, timeout
@@ -1000,40 +809,6 @@ class _FakeVikingFSForCreate:
# Create-mode tests
[email protected]("mode", ["replace", "append"])
-async def test_replace_and_append_create_missing_file(monkeypatch, mode):
- file_uri = "viking://resources/demo/missing.csv"
- root_uri = "viking://resources/demo"
- ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
- viking_fs = _FakeVikingFSForCreate(file_uri=file_uri, root_uri=root_uri, file_exists=False)
- coordinator = ContentWriteCoordinator(viking_fs=viking_fs)
-
- refresh_calls = []
- write_calls = []
-
- async def _fake_write_in_place(uri, content, *, mode, ctx, lease_ref=None, existing_raw=None):
- del ctx, lease_ref, existing_raw
- write_calls.append((uri, content, mode))
- viking_fs.content[uri] = content
-
- async def _fake_enqueue_semantic_refresh(**kwargs):
- refresh_calls.append(kwargs)
- return None
-
- monkeypatch.setattr(coordinator, "_write_in_place", _fake_write_in_place)
- monkeypatch.setattr(coordinator, "_enqueue_semantic_refresh", _fake_enqueue_semantic_refresh)
-
- result = await coordinator.write(
- uri=file_uri, content="new content", mode=mode, ctx=ctx, wait=False
- )
-
- assert result["mode"] == mode
- assert viking_fs.content[file_uri] == "new content"
- assert write_calls == [(file_uri, "new content", "create")]
- assert refresh_calls[0]["change_type"] == "added"
-
-
@pytest.mark.asyncio
async def test_create_mode_new_file_success(monkeypatch):
file_uri = "viking://user/default/memories/new_file.md"
@@ -1044,9 +819,11 @@ async def test_create_mode_new_file_success(monkeypatch):
write_calls = []
- async def _fake_write_in_place(uri, content, *, mode, ctx, lock_handle=None, lease_ref=None):
- del mode, ctx, lock_handle, lease_ref
- write_calls.append((uri, content))
+ async def _fake_write_in_place(
+ uri, content, *, mode, ctx, lock_handle=None, lease_ref=None, is_new_file=None
+ ):
+ del ctx, lock_handle, lease_ref
+ write_calls.append((uri, content, mode, is_new_file))
return content
async def _fake_wait_for_queues(*, timeout):
@@ -1061,7 +838,7 @@ async def test_create_mode_new_file_success(monkeypatch):
)
assert result["mode"] == "create"
- assert write_calls == [(file_uri, "new content")]
+ assert write_calls == [(file_uri, "new content", "replace", True)]
@pytest.mark.asyncio
@@ -1079,9 +856,11 @@ async def test_create_mode_refreshes_canonical_user_memory_uri(monkeypatch):
write_calls = []
refresh_calls = []
- async def _fake_write_in_place(uri, content, *, mode, ctx, lock_handle=None, lease_ref=None):
- del mode, ctx, lock_handle, lease_ref
- write_calls.append((uri, content))
+ async def _fake_write_in_place(
+ uri, content, *, mode, ctx, lock_handle=None, lease_ref=None, is_new_file=None
+ ):
+ del ctx, lock_handle, lease_ref
+ write_calls.append((uri, content, mode, is_new_file))
return content
async def _fake_refresh_schema_overview(**kwargs):
@@ -1106,31 +885,47 @@ async def test_create_mode_refreshes_canonical_user_memory_uri(monkeypatch):
assert result["uri"] == canonical_uri
assert result["root_uri"] == root_uri
assert result["context_type"] == "memory"
- assert write_calls == [(canonical_uri, "new content")]
+ assert write_calls == [(canonical_uri, "new content", "replace", True)]
assert refresh_calls[0]["directory_uri"] == root_uri
@pytest.mark.asyncio
-async def test_create_mode_existing_file_raises_409(monkeypatch):
+async def test_create_mode_overwrites_existing_file(monkeypatch):
+ # create is an upsert alias for replace: an existing target is overwritten,
+ # never rejected with AlreadyExistsError.
file_uri = "viking://user/default/memories/existing.md"
root_uri = "viking://user/default/memories"
ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
viking_fs = _FakeVikingFSForCreate(file_uri=file_uri, root_uri=root_uri, file_exists=True)
coordinator = ContentWriteCoordinator(viking_fs=viking_fs)
- async def _fake_write_in_place(uri, content, *, mode, ctx, lock_handle=None, lease_ref=None):
- del uri, content, mode, ctx, lock_handle, lease_ref
- return None
+ write_calls = []
- async def _fake_wait_for_queues(*, timeout):
- del timeout
- return None
+ async def _fake_write_in_place(
+ uri, content, *, mode, ctx, lock_handle=None, lease_ref=None, is_new_file=None
+ ):
+ del ctx, lock_handle, lease_ref
+ write_calls.append((uri, content, mode, is_new_file))
+ return content.encode("utf-8")
+
+ async def _fake_refresh_schema_overview(**kwargs):
+ del kwargs
+ return True
monkeypatch.setattr(coordinator, "_write_in_place", _fake_write_in_place)
- monkeypatch.setattr(coordinator, "_wait_for_queues", _fake_wait_for_queues)
+ monkeypatch.setattr(
+ "openviking.storage.content_write.MemoryUpdater.refresh_schema_overview",
+ _fake_refresh_schema_overview,
+ )
- with pytest.raises(AlreadyExistsError):
- await coordinator.write(uri=file_uri, content="content", mode="create", ctx=ctx, wait=True)
+ result = await coordinator.write(
+ uri=file_uri, content="content", mode="create", ctx=ctx, wait=False
+ )
+
+ # response echoes the requested mode, but an existing target uses replace
+ # rendering (memory keeps its metadata trailer).
+ assert result["mode"] == "create"
+ assert write_calls == [(file_uri, "content", "replace", None)]
@pytest.mark.asyncio
@@ -1166,9 +961,11 @@ async def test_create_mode_parent_dirs_auto_created(monkeypatch):
write_calls = []
- async def _fake_write_in_place(uri, content, *, mode, ctx, lock_handle=None, lease_ref=None):
- del mode, ctx, lock_handle, lease_ref
- write_calls.append((uri, content))
+ async def _fake_write_in_place(
+ uri, content, *, mode, ctx, lock_handle=None, lease_ref=None, is_new_file=None
+ ):
+ del ctx, lock_handle, lease_ref
+ write_calls.append((uri, content, mode, is_new_file))
return content
async def _fake_wait_for_queues(*, timeout):
@@ -1183,7 +980,7 @@ async def test_create_mode_parent_dirs_auto_created(monkeypatch):
)
assert result["mode"] == "create"
- assert write_calls == [(file_uri, "nested content")]
+ assert write_calls == [(file_uri, "nested content", "replace", True)]
@pytest.mark.asyncio
@@ -1200,9 +997,9 @@ async def test_create_mode_valid_extensions_pass(monkeypatch):
coordinator = ContentWriteCoordinator(viking_fs=viking_fs)
async def _fake_write_in_place(
- uri, content, *, mode, ctx, lock_handle=None, lease_ref=None
+ uri, content, *, mode, ctx, lock_handle=None, lease_ref=None, is_new_file=None
):
- del uri, mode, ctx, lock_handle, lease_ref
+ del uri, mode, ctx, lock_handle, lease_ref, is_new_file
return content
async def _fake_wait_for_queues(*, timeout):
@@ -1226,8 +1023,10 @@ async def test_create_mode_memory_scope(monkeypatch):
viking_fs = _FakeVikingFSForCreate(file_uri=file_uri, root_uri=root_uri, file_exists=False)
coordinator = ContentWriteCoordinator(viking_fs=viking_fs)
- async def _fake_write_in_place(uri, content, *, mode, ctx, lock_handle=None, lease_ref=None):
- del uri, mode, ctx, lock_handle, lease_ref
+ async def _fake_write_in_place(
+ uri, content, *, mode, ctx, lock_handle=None, lease_ref=None, is_new_file=None
+ ):
+ del uri, mode, ctx, lock_handle, lease_ref, is_new_file
return content
refresh_calls = []
@@ -1254,42 +1053,6 @@ async def test_create_mode_memory_scope(monkeypatch):
assert refresh_calls[0]["directory_uri"] == root_uri
-async def test_create_mode_resource_scope(monkeypatch):
- file_uri = "viking://resources/team_notes/final_draft.md"
- root_uri = "viking://resources/team_notes"
- ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
- viking_fs = _FakeVikingFSForCreate(file_uri=file_uri, root_uri=root_uri, file_exists=False)
- coordinator = ContentWriteCoordinator(viking_fs=viking_fs)
-
- async def _fake_enqueue_semantic_refresh(**kwargs):
- # Verify resource-scope URIs take the resource write path
- assert kwargs["root_uri"] == root_uri
- assert kwargs["changed_uri"] == file_uri
- assert kwargs["context_type"] == "resource"
- assert kwargs["change_type"] == "added"
- del kwargs
- return None
-
- async def _fake_wait_for_queues(*, timeout):
- del timeout
- return None
-
- monkeypatch.setattr(coordinator, "_enqueue_semantic_refresh", _fake_enqueue_semantic_refresh)
- monkeypatch.setattr(coordinator, "_wait_for_queues", _fake_wait_for_queues)
-
- result = await coordinator.write(
- uri="viking://resources/team notes/final draft.md",
- content="content",
- mode="create",
- ctx=ctx,
- wait=True,
- )
- assert result["context_type"] == "resource"
- assert result["uri"] == file_uri
- assert viking_fs.content == {file_uri: "content"}
-
-
class _AnyDirVikingFS:
"""Stats the given file uri as a file and every other uri as a directory."""
@@ -1351,7 +1114,7 @@ async def test_create_mode_regression_replace_unchanged(monkeypatch):
# Verify mode="replace" still works
assert mode == "replace"
del uri, content, ctx, lock_handle, lease_ref
- return None
+ return b"updated"
async def _fake_wait_for_queues(*, timeout):
del timeout
@@ -1379,7 +1142,7 @@ async def test_create_mode_regression_append_unchanged(monkeypatch):
# Verify mode="append" still works
assert mode == "append"
del uri, content, ctx, lock_handle, lease_ref
- return None
+ return b"previousappended"
async def _fake_wait_for_queues(*, timeout):
del timeout
diff --git a/tests/server/test_mcp_endpoint.py b/tests/server/test_mcp_endpoint.py
index 5dd7f9d62..7ac855529 100644
--- a/tests/server/test_mcp_endpoint.py
+++ b/tests/server/test_mcp_endpoint.py
@@ -46,7 +46,6 @@ from openviking.server.mcp_endpoint import (
from openviking.server.mcp_endpoint import ls as list_tool
from openviking.service.fs_service import ListingPage
from openviking_cli.exceptions import (
- AlreadyExistsError,
FailedPreconditionError,
InvalidArgumentError,
InvalidURIError,
@@ -2014,11 +2013,12 @@ async def test_write_replace_overwrites_existing(service):
assert body == "v2-content"
-async def test_write_create_fails_when_file_exists(service):
+async def test_write_create_overwrites_existing_file(service):
uri = "viking://resources/test_write_create_exists.md"
await write(uri=uri, content="v1")
- with pytest.raises(AlreadyExistsError):
- await write(uri=uri, content="v2", mode="create")
+ result = await write(uri=uri, content="v2", mode="create")
+ assert "mode=create" in result
+ assert await service.fs.read(uri, ctx=DEFAULT_CTX) == "v2"
async def test_write_append_appends_to_existing(service):
@@ -2029,11 +2029,11 @@ async def test_write_append_appends_to_existing(service):
assert body == "line1\nline2\n"
-async def test_write_append_missing_file_fails(service):
- with pytest.raises(NotFoundError):
- await write(
- uri="viking://resources/test_write_append_missing.md", content="x", mode="append"
- )
+async def test_write_append_creates_missing_file(service):
+ uri = "viking://resources/test_write_append_missing.md"
+ result = await write(uri=uri, content="x", mode="append")
+ assert "mode=append" in result
+ assert await service.fs.read(uri, ctx=DEFAULT_CTX) == "x"
async def test_write_create_rejects_disallowed_extension(service):
diff --git a/tests/storage/test_content_write_processing_mode.py b/tests/storage/test_content_write_processing_mode.py
index 21cae071a..079e82ec6 100644
--- a/tests/storage/test_content_write_processing_mode.py
+++ b/tests/storage/test_content_write_processing_mode.py
@@ -8,15 +8,16 @@ import pytest
from openviking.core.context import ContextLevel
from openviking.server.identity import RequestContext, Role
+from openviking.session.memory.utils.memory_file_utils import MemoryFileUtils
from openviking.storage import content_write as content_write_module
from openviking.storage.abstract_overview import (
- parse_abstract_overview,
render_abstract_overview,
)
from openviking.storage.content_write import ContentWriteCoordinator
+from openviking.storage.context_update_plan import ContextUpdatePlan
from openviking.storage.queuefs.semantic_ops.freshness_policy import FreshnessAction
-from openviking.utils.content_hash import content_md5
from openviking.utils.ingest_options import IngestOptions
+from openviking_cli.exceptions import InvalidArgumentError, NotFoundError
from openviking_cli.session.user_id import UserIdentifier
@@ -84,115 +85,6 @@ def ctx():
return RequestContext(user=UserIdentifier("account-1", "user-1"), role=Role.USER)
-async def test_direct_write_skips_semantic_refresh_for_vectors_only_and_sidecar_body_edits(
- monkeypatch, ctx
-):
- fake_fs = _FakeVikingFS()
- vectorize_file = AsyncMock(return_value=True)
- semantic_refresh = AsyncMock(side_effect=AssertionError("semantic refresh should not run"))
- monkeypatch.setattr(content_write_module, "vectorize_file", vectorize_file, raising=False)
- coordinator = ContentWriteCoordinator(viking_fs=fake_fs)
- coordinator._enqueue_semantic_refresh = semantic_refresh
-
- result = await coordinator._write_direct_with_refresh(
- uri="viking://resources/demo.md",
- root_uri="viking://resources",
- content="updated",
- mode="replace",
- context_type="resource",
- wait=False,
- timeout=None,
- ctx=ctx,
- written_bytes=7,
- telemetry_id="",
- processing_mode="vectors_only",
- )
-
- semantic_refresh.assert_not_awaited()
- vectorize_file.assert_awaited_once()
- assert vectorize_file.await_args.kwargs["file_path"] == "viking://resources/demo.md"
- assert vectorize_file.await_args.kwargs["parent_uri"] == "viking://resources"
- assert vectorize_file.await_args.kwargs["summary_dict"] == {
- "name": "demo.md",
- "summary": "",
- }
- assert vectorize_file.await_args.kwargs["file_md5"] == content_md5(b"updated")
- assert "register_request_wait" not in vectorize_file.await_args.kwargs
- assert result["semantic_status"] == "skipped"
- assert result["vector_status"] == "queued"
-
- current = _sidecar()
- sidecar_fs = _FakeVikingFS()
- sidecar_fs.read_file.side_effect = [
- current,
- current,
- _sidecar(ContextLevel.OVERVIEW, "Overview."),
- ]
- vectorize_directory = AsyncMock()
- monkeypatch.setattr(content_write_module, "vectorize_directory_meta", vectorize_directory)
- sidecar_coordinator = ContentWriteCoordinator(viking_fs=sidecar_fs)
- sidecar_coordinator._enqueue_semantic_refresh = AsyncMock(
- side_effect=AssertionError("sidecar body writes must not regenerate semantics")
- )
-
- sidecar_result = await sidecar_coordinator._write_direct_with_refresh(
- uri="viking://resources/demo/.abstract.md",
- root_uri="viking://resources/demo",
- content="Updated body only.",
- mode="replace",
- context_type="resource",
- wait=False,
- timeout=None,
- ctx=ctx,
- written_bytes=len("Updated body only.".encode()),
- telemetry_id="",
- ingest_options=IngestOptions.from_search_tags(["team=search"], mode="append"),
- )
-
- written = sidecar_fs.write_file.await_args.args[1]
- assert parse_abstract_overview(written).body == "Updated body only.\n"
- assert parse_abstract_overview(written).metadata == parse_abstract_overview(current).metadata
- sidecar_coordinator._enqueue_semantic_refresh.assert_not_awaited()
- vectorize_directory.assert_awaited_once()
- assert vectorize_directory.await_args.kwargs["ingest_options"] == IngestOptions(
- search_tags=["team=search"], search_tag_mode="append"
- )
- assert sidecar_result["semantic_status"] == "skipped"
- assert sidecar_result["vector_status"] == "queued"
-
-
-async def test_direct_write_passes_final_md5_and_old_abstract_to_semantic_refresh(ctx):
- file_uri = "viking://resources/demo.py"
-
- class _VikingDB:
- async def get_l2_diff_records_by_uris(self, uris, *, ctx):
- assert uris == [file_uri]
- return {file_uri: {"md5": "old-md5", "abstract": "old abstract"}}
-
- fake_fs = _FakeVikingFS()
- coordinator = ContentWriteCoordinator(viking_fs=fake_fs, vikingdb=_VikingDB())
- enqueue = AsyncMock(return_value=FreshnessAction.REFRESH_NOW)
- coordinator._enqueue_semantic_refresh = enqueue
-
- await coordinator._write_direct_with_refresh(
- uri=file_uri,
- root_uri="viking://resources",
- content="updated",
- mode="replace",
- context_type="resource",
- wait=False,
- timeout=None,
- ctx=ctx,
- written_bytes=7,
- telemetry_id="",
- )
-
- assert enqueue.await_args.kwargs["file_md5"] == content_md5(b"updated")
- assert enqueue.await_args.kwargs["file_abstract"] == "old abstract"
-
-
@pytest.mark.asyncio
async def test_semantic_message_carries_file_md5_and_old_abstract(monkeypatch, ctx):
file_uri = "viking://resources/demo.py"
@@ -220,59 +112,6 @@ async def test_semantic_message_carries_file_md5_and_old_abstract(monkeypatch, c
assert msg.file_abstracts == {file_uri: "old abstract"}
-async def test_direct_append_passes_md5_of_final_content_without_vector_db(ctx):
- fake_fs = _FakeVikingFS()
- fake_fs.read_file.return_value = "previous"
- coordinator = ContentWriteCoordinator(viking_fs=fake_fs)
- enqueue = AsyncMock(return_value=FreshnessAction.REFRESH_NOW)
- coordinator._enqueue_semantic_refresh = enqueue
-
- await coordinator._write_direct_with_refresh(
- uri="viking://resources/demo.py",
- root_uri="viking://resources",
- content=" updated",
- mode="append",
- context_type="resource",
- wait=False,
- timeout=None,
- ctx=ctx,
- written_bytes=8,
- telemetry_id="",
- )
-
- assert enqueue.await_args.kwargs["file_md5"] == content_md5(b"previous updated")
- assert enqueue.await_args.kwargs["file_abstract"] == ""
-
-
-async def test_direct_write_continues_when_old_abstract_lookup_fails(ctx):
- class _FailingVikingDB:
- async def get_l2_diff_records_by_uris(self, uris, *, ctx):
- raise RuntimeError("vector lookup unavailable")
-
- fake_fs = _FakeVikingFS()
- coordinator = ContentWriteCoordinator(viking_fs=fake_fs, vikingdb=_FailingVikingDB())
- enqueue = AsyncMock(return_value=FreshnessAction.REFRESH_NOW)
- coordinator._enqueue_semantic_refresh = enqueue
-
- await coordinator._write_direct_with_refresh(
- uri="viking://resources/demo.py",
- root_uri="viking://resources",
- content="updated",
- mode="replace",
- context_type="resource",
- wait=False,
- timeout=None,
- ctx=ctx,
- written_bytes=7,
- telemetry_id="",
- )
-
- assert enqueue.await_args.kwargs["file_abstract"] == ""
- assert enqueue.await_args.kwargs["file_md5"] == content_md5(b"updated")
-
-
@pytest.mark.asyncio
async def test_old_abstract_lookup_uses_viking_fs_vector_store_when_not_injected(ctx):
file_uri = "viking://agent/skills/demo/SKILL.md"
@@ -313,129 +152,330 @@ async def test_write_builds_ingest_options_before_scheduling_resource_refresh(ct
@pytest.mark.asyncio
-async def test_vectors_only_write_wait_reports_embedding_status(monkeypatch, ctx):
- fake_fs = _FakeVikingFS()
- queue_status = {
- "Embedding": {"processed": 1, "error_count": 0, "errors": []},
- "Semantic": {"processed": 0, "error_count": 0, "errors": []},
- }
- monkeypatch.setattr(
- content_write_module, "vectorize_file", AsyncMock(return_value=True), raising=False
+async def test_create_routes_as_replace_without_prelock_target_stat(ctx):
+ coordinator = ContentWriteCoordinator(viking_fs=_FakeVikingFS())
+ coordinator._safe_stat = AsyncMock(return_value={"not_found": True})
+ coordinator._resolve_root_uri = AsyncMock(return_value="viking://resources")
+ coordinator._write_direct_with_refresh = AsyncMock(
+ return_value={"uri": "viking://resources/new.md"}
+ )
+
+ await coordinator.write(
+ uri="viking://resources/new.md",
+ content="new content",
+ mode="create",
+ ctx=ctx,
)
+
+ kwargs = coordinator._write_direct_with_refresh.await_args.kwargs
+ assert kwargs["mode"] == "replace"
+ coordinator._safe_stat.assert_not_awaited()
+ coordinator._resolve_root_uri.assert_awaited_once_with(
+ "viking://resources/new.md",
+ ctx=ctx,
+ _allow_not_found=True,
+ anchor_to_parent=True,
+ validate_storage=False,
+ )
+
+
+async def test_direct_write_reads_target_once_after_lock_and_reuses_formal_state(monkeypatch, ctx):
+ fake_fs = _FakeVikingFS()
+ events = []
+ original_acquire = fake_fs._async_agfs.pathlock_acquire_exact
+
+ async def _acquire(path):
+ events.append("lock")
+ return await original_acquire(path)
+
+ async def _stat(*args, **kwargs):
+ del args, kwargs
+ events.append("stat")
+ return {"isDir": False}
+
+ fake_fs._async_agfs.pathlock_acquire_exact = _acquire
+ fake_fs.stat = _stat
coordinator = ContentWriteCoordinator(viking_fs=fake_fs)
- coordinator._wait_for_request = AsyncMock(return_value=queue_status)
+ captured = {}
- result = await coordinator._write_direct_with_refresh(
+ async def _snapshot(**kwargs):
+ captured["formal_snapshot"] = kwargs["formal_snapshot"]
+ captured["target_preexisting"] = kwargs["target_preexisting"]
+ captured["vector_inventory"] = kwargs.get("vector_inventory")
+ return SimpleNamespace()
+
+ async def _plan(**kwargs):
+ del kwargs
+ return None, ContextUpdatePlan("viking://resources/demo.md", "resource")
+
+ async def _commit(*args, **kwargs):
+ del args, kwargs
+ return SimpleNamespace(
+ semantic_requested=False, vector_requested=False, semantic_action=None
+ )
+
+ monkeypatch.setattr(content_write_module, "build_rnfv_snapshot", _snapshot)
+ monkeypatch.setattr(content_write_module, "build_context_update_plan_from_snapshot", _plan)
+ monkeypatch.setattr(content_write_module, "commit_and_enqueue_plan", _commit)
+
+ await coordinator._write_direct_with_refresh(
uri="viking://resources/demo.md",
root_uri="viking://resources",
content="updated",
mode="replace",
+ response_mode="replace",
context_type="resource",
- wait=True,
- timeout=3.0,
+ wait=False,
+ timeout=None,
ctx=ctx,
- written_bytes=7,
- telemetry_id="tm-test",
- processing_mode="vectors_only",
+ telemetry_id="",
+ ingest_options=IngestOptions(),
)
- assert result["queue_status"] == queue_status
- assert result["semantic_status"] == "skipped"
- assert result["vector_status"] == "complete"
+ assert events == ["lock", "stat"]
+ assert captured["target_preexisting"] is True
+ assert captured["formal_snapshot"] == (
+ {"": content_write_module.FormalEntry(is_dir=False)},
+ True,
+ )
+ assert captured["vector_inventory"] is None
@pytest.mark.asyncio
-async def test_vectors_only_write_wait_reports_skipped_when_nothing_enqueued(monkeypatch, ctx):
+async def test_direct_write_commits_final_bytes_when_rnfv_content_is_unchanged(monkeypatch, ctx):
+ """An explicit write owns F even when N already matches the indexed V md5."""
+ uri = "viking://resources/demo.md"
fake_fs = _FakeVikingFS()
- monkeypatch.setattr(
- content_write_module, "vectorize_file", AsyncMock(return_value=False), raising=False
- )
+ fake_fs.stat = AsyncMock(return_value={"isDir": False})
+ fake_fs.write_file_bytes = AsyncMock()
coordinator = ContentWriteCoordinator(viking_fs=fake_fs)
- coordinator._wait_for_request = AsyncMock(return_value=None)
- result = await coordinator._write_direct_with_refresh(
- uri="viking://resources/obsolete.md",
+ async def _snapshot(**kwargs):
+ del kwargs
+ return SimpleNamespace()
+
+ async def _plan(**kwargs):
+ del kwargs
+ # This is the plan RNFV produces when N.md5 == V.md5. The explicit
+ # write must still commit its bytes to F because V may lag F.
+ return None, ContextUpdatePlan(uri, "resource")
+
+ monkeypatch.setattr(content_write_module, "build_rnfv_snapshot", _snapshot)
+ monkeypatch.setattr(content_write_module, "build_context_update_plan_from_snapshot", _plan)
+
+ await coordinator._write_direct_with_refresh(
+ uri=uri,
root_uri="viking://resources",
- content="",
+ content="indexed content",
mode="replace",
+ response_mode="replace",
context_type="resource",
- wait=True,
- timeout=3.0,
+ wait=False,
+ timeout=None,
+ ctx=ctx,
+ telemetry_id="",
+ ingest_options=IngestOptions(),
+ )
+
+ fake_fs.write_file_bytes.assert_awaited_once()
+ assert fake_fs.write_file_bytes.await_args.args == (uri, b"indexed content")
+
+
+async def test_prepare_inline_resource_plan_reuses_locked_facts_and_preloaded_inventory(
+ monkeypatch, ctx
+):
+ """Single and batch adapters share one file-scoped RNFV plan builder."""
+ uri = "viking://resources/demo.md"
+ fake_fs = _FakeVikingFS()
+ coordinator = ContentWriteCoordinator(viking_fs=fake_fs)
+ vector_store = object()
+ captured = {}
+ existing_record = SimpleNamespace(
+ uri=uri,
+ level=2,
+ fields={"abstract": "existing L2 abstract"},
+ )
+
+ async def _snapshot(**kwargs):
+ captured.update(kwargs)
+ return SimpleNamespace(vectors=SimpleNamespace(records_by_id={"a-l2": existing_record}))
+
+ async def _plan(**kwargs):
+ captured["plan_kwargs"] = kwargs
+ return None, ContextUpdatePlan(uri, "resource")
+
+ monkeypatch.setattr(content_write_module, "build_rnfv_snapshot", _snapshot)
+ monkeypatch.setattr(content_write_module, "build_context_update_plan_from_snapshot", _plan)
+
+ prepared = await coordinator._prepare_inline_resource_plan(
+ uri=uri,
+ final_bytes=b"new bytes",
+ target_preexisting=True,
+ formal_snapshot=({"": content_write_module.FormalEntry(is_dir=False)}, True),
+ vector_inventory={"a-l2": {"id": "a-l2"}},
+ vector_store=vector_store,
+ request_intent=content_write_module.RequestIntent(uri, "semantic_and_vectors"),
ctx=ctx,
- written_bytes=0,
- telemetry_id="tm-test",
- processing_mode="vectors_only",
+ lease=SimpleNamespace(id="lock-1"),
+ ingest_options=IngestOptions(),
+ include_previous_abstract=True,
)
- assert result["semantic_status"] == "skipped"
- assert result["vector_status"] == "skipped"
+ assert prepared.final_bytes == b"new bytes"
+ assert prepared.previous_abstract == "existing L2 abstract"
+ assert captured["root_is_file"] is True
+ assert captured["vector_scope"] == "self"
+ assert captured["formal_snapshot"] == ({"": content_write_module.FormalEntry(False)}, True)
+ assert captured["vector_inventory"] == {"a-l2": {"id": "a-l2"}}
@pytest.mark.asyncio
-async def test_automatic_wide_directory_delay_reports_deferred(monkeypatch, ctx):
+async def test_missing_append_starts_from_empty_content(monkeypatch, ctx):
fake_fs = _FakeVikingFS()
+ fake_fs.stat = AsyncMock(side_effect=NotFoundError("viking://resources/new.md", "file"))
coordinator = ContentWriteCoordinator(viking_fs=fake_fs)
- coordinator._enqueue_semantic_refresh = AsyncMock(return_value=FreshnessAction.MARK_PENDING)
+ captured = {}
+
+ async def _snapshot(**kwargs):
+ captured["bytes"] = kwargs["store"]._data
+ captured["vector_inventory"] = kwargs.get("vector_inventory")
+ return SimpleNamespace()
+
+ async def _plan(**kwargs):
+ del kwargs
+ return None, ContextUpdatePlan("viking://resources/new.md", "resource")
+
+ async def _commit(*args, **kwargs):
+ del args, kwargs
+ return SimpleNamespace(
+ semantic_requested=False, vector_requested=False, semantic_action=None
+ )
+
+ monkeypatch.setattr(content_write_module, "build_rnfv_snapshot", _snapshot)
+ monkeypatch.setattr(content_write_module, "build_context_update_plan_from_snapshot", _plan)
+ monkeypatch.setattr(content_write_module, "commit_and_enqueue_plan", _commit)
result = await coordinator._write_direct_with_refresh(
- uri="viking://resources/wide/demo.md",
- root_uri="viking://resources/wide",
- content="updated",
- mode="replace",
+ uri="viking://resources/new.md",
+ root_uri="viking://resources",
+ content="first line",
+ mode="append",
+ response_mode="append",
context_type="resource",
wait=False,
timeout=None,
ctx=ctx,
- written_bytes=7,
telemetry_id="",
+ ingest_options=IngestOptions(),
+ )
+
+ fake_fs.read_file.assert_not_awaited()
+ assert captured["bytes"] == b"first line"
+ assert captured["vector_inventory"] == {}
+ assert result["mode"] == "append"
+
+
+def test_write_result_keeps_content_updated_compatibility_field():
+ coordinator = ContentWriteCoordinator(viking_fs=_FakeVikingFS())
+
+ result = coordinator._build_write_result(
+ uri="viking://resources/demo.md",
+ root_uri="viking://resources",
+ context_type="resource",
+ mode="replace",
+ written_bytes=7,
+ queue_status=None,
+ semantic_status="skipped",
+ vector_status="skipped",
)
- assert result["semantic_status"] == "deferred"
- assert result["vector_status"] == "queued"
+ assert result["content_updated"] is True
@pytest.mark.asyncio
- ("overview_refreshed", "expected_overview_status"),
- [(True, "complete"), (False, "skipped")],
-)
-async def test_memory_write_accepts_processing_mode_without_switching_refresh(
- monkeypatch, ctx, overview_refreshed, expected_overview_status
-):
+async def test_missing_generated_sidecar_is_rejected_after_exact_lock(monkeypatch, ctx):
fake_fs = _FakeVikingFS()
- monkeypatch.setattr(
- content_write_module.MemoryUpdater,
- "refresh_schema_overview",
- AsyncMock(return_value=overview_refreshed),
+ events = []
+ original_acquire = fake_fs._async_agfs.pathlock_acquire_exact
+
+ async def _acquire(path):
+ events.append("lock")
+ return await original_acquire(path)
+
+ fake_fs._async_agfs.pathlock_acquire_exact = _acquire
+ coordinator = ContentWriteCoordinator(viking_fs=fake_fs)
+ coordinator._classify_locked_write_target = AsyncMock(
+ side_effect=InvalidArgumentError("cannot create generated abstract overview directly")
)
- monkeypatch.setattr(
- content_write_module.MemoryUpdater,
- "refresh_file_embedding",
- AsyncMock(return_value=False),
+
+ with pytest.raises(InvalidArgumentError, match="cannot create generated"):
+ await coordinator.write(
+ uri="viking://resources/demo/.abstract.md",
+ content="manual body",
+ mode="replace",
+ ctx=ctx,
+ )
+
+ assert events == ["lock"]
+ coordinator._classify_locked_write_target.assert_awaited_once()
+
+
+def test_new_memory_file_renders_from_replace_mode_and_explicit_new_file_state():
+ uri = "viking://user/account-1/memories/new.md"
+ coordinator = ContentWriteCoordinator(viking_fs=_FakeVikingFS())
+
+ rendered = coordinator._render_final_bytes(
+ uri,
+ "new memory content",
+ mode="replace",
+ existing_raw=None,
+ is_new_file=True,
)
- monkeypatch.setattr(
- content_write_module.MemoryUpdater,
- "memory_type_from_uri",
- lambda uri: "user",
+
+ assert MemoryFileUtils.read(rendered.decode("utf-8"), uri=uri).content == "new memory content"
+
+
+def test_plan_statuses_report_deferred_parent_and_queued_vector_work():
+ coordinator = ContentWriteCoordinator(viking_fs=_FakeVikingFS())
+ work = SimpleNamespace(
+ semantic_requested=False,
+ vector_requested=True,
+ semantic_action=FreshnessAction.MARK_PENDING.value,
)
- coordinator = ContentWriteCoordinator(viking_fs=fake_fs)
- coordinator._write_in_place = AsyncMock()
- result = await coordinator._write_memory_with_refresh(
- uri="viking://user/user-1/memories/demo.md",
- root_uri="viking://user/user-1/memories",
- content="updated",
- mode="replace",
- wait=True,
- timeout=3.0,
- ctx=ctx,
- written_bytes=7,
- telemetry_id="tm-test",
- processing_mode="vectors_only",
+ assert coordinator._plan_statuses(work, wait=False, queue_status=None) == (
+ "deferred",
+ "queued",
+ )
+
+
+ ("semantic_action", "wait", "queue_status", "expected"),
+ [
+ (
+ FreshnessAction.MARK_PENDING.value,
+ False,
+ None,
+ ("deferred", "queued"),
+ ),
+ (
+ FreshnessAction.NOOP.value,
+ True,
+ {},
+ ("skipped", "complete"),
+ ),
+ ],
+)
+def test_plan_statuses_prioritize_parent_freshness_over_file_semantic_work(
+ semantic_action, wait, queue_status, expected
+):
+ coordinator = ContentWriteCoordinator(viking_fs=_FakeVikingFS())
+ work = SimpleNamespace(
+ semantic_requested=True,
+ vector_requested=True,
+ semantic_action=semantic_action,
)
- content_write_module.MemoryUpdater.refresh_schema_overview.assert_awaited_once()
- content_write_module.MemoryUpdater.refresh_file_embedding.assert_awaited_once()
- assert result["context_type"] == "memory"
- assert result["semantic_status"] == "skipped"
- assert result["overview_status"] == expected_overview_status
+ assert coordinator._plan_statuses(work, wait=wait, queue_status=queue_status) == expected
diff --git a/tests/storage/test_context_update_plan.py b/tests/storage/test_context_update_plan.py
index 3e4cdcacd..5b195762c 100644
--- a/tests/storage/test_context_update_plan.py
+++ b/tests/storage/test_context_update_plan.py
@@ -2342,6 +2342,54 @@ async def test_file_root_plan_uses_file_refresh_without_directory_semantic_tree(
assert plan.file_refresh.md5 == "new"
+async def test_file_root_plan_refreshes_missing_abstract_when_content_is_indexed():
+ """A prior vectors-only write must not suppress a later semantic summary."""
+ from openviking.storage.context_update_plan import build_context_update_plan_from_snapshot
+ from openviking.storage.resource_rnfv import (
+ FormalEntry,
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ NewEntry,
+ RequestIntent,
+ RNFVSnapshot,
+ VectorIndexSnapshot,
+ VectorRecordSnapshot,
+ )
+
+ root = "viking://resources/report.md"
+ snapshot = RNFVSnapshot(
+ RequestIntent(root, "semantic_and_vectors"),
+ NewArtifactSnapshot({"": NewEntry(md5="same")}),
+ FormalTreeSnapshot({"": FormalEntry()}),
+ VectorIndexSnapshot(
+ {"report-l2": VectorRecordSnapshot("report-l2", root, "", 2, {"md5": "same"})},
+ frozenset({"id", "uri", "level", "md5", "abstract"}),
+ ),
+ )
+
+ _, plan = await build_context_update_plan_from_snapshot(
+ snapshot=snapshot,
+ store=AsyncMock(),
+ artifact_ref=object(),
+ target=AsyncMock(),
+ vikingdb=AsyncMock(),
+ context_type="resource",
+ is_code_repo=False,
+ account_id="acc",
+ ctx=object(),
+ root_preexisting=True,
+ artifact_paths={"": "report.md"},
+ root_is_file=True,
+ )
+
+ assert plan.content_tree_actions == ()
+ assert plan.direct_index_actions == ()
+ assert plan.semantic_plan is None
+ assert plan.file_refresh is not None
+ assert plan.file_refresh.md5 == "same"
+
+
@pytest.mark.asyncio
async def test_hydration_promotes_a_dependency_when_its_record_disappears():
from openviking.storage.context_update_plan import (
@@ -2816,6 +2864,241 @@ async def test_resource_processor_preserves_skill_metadata_in_direct_directory_m
}
+ ("scalar_override", "expected_summary"),
+ [
+ # An existing L2 abstract carried by the V snapshot is seeded into
+ # summary_dict so vectorize_file can honor text_source downstream.
+ ({"abstract": "existing summary"}, "existing summary"),
+ # No abstract (new file / empty record) leaves the summary empty, so
+ # vectorize_file falls back to the file body regardless of text_source.
+ ({"abstract": ""}, ""),
+ ({}, ""),
+ ],
+)
+async def test_vectorize_resource_file_seeds_summary_from_existing_abstract(
+ monkeypatch, scalar_override, expected_summary
+):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage import context_update_execution
+ from openviking_cli.session.user_id import UserIdentifier
+
+ captured = {}
+
+ async def _fake_vectorize_file(
+ *,
+ file_path,
+ summary_dict,
+ parent_uri,
+ context_type,
+ ctx,
+ ingest_options,
+ file_md5,
+ scalar_override,
+ field_patch,
+ action,
+ ):
+ captured["summary_dict"] = summary_dict
+ captured["scalar_override"] = scalar_override
+ captured["ingest_options"] = ingest_options
+ return True
+
+ monkeypatch.setattr("openviking.utils.embedding_utils.vectorize_file", _fake_vectorize_file)
+ ctx = RequestContext(UserIdentifier("acc", "user"), Role.USER)
+
+ await context_update_execution.vectorize_resource_file(
+ "viking://resources/repo/a.py",
+ ctx=ctx,
+ scalar_override={"_record_id": "id-a", **scalar_override},
+ action="upsert",
+ )
+
+ # The existing abstract (if any) is seeded into summary_dict; text_source
+ # policy (summary vs body) is applied inside vectorize_file, not here. The
+ # abstract still travels unchanged as a stored scalar.
+ assert captured["summary_dict"]["summary"] == expected_summary
+ assert captured["scalar_override"]["_record_id"] == "id-a"
+ assert captured["ingest_options"].search_tags is None
+ if scalar_override.get("abstract"):
+ assert captured["scalar_override"]["abstract"] == scalar_override["abstract"]
+
+
+async def test_vectorize_resource_file_carries_acl_without_reapplying_tags(monkeypatch):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage import context_update_execution
+ from openviking.storage.acl import AclSpec, AclUpdate
+ from openviking_cli.session.user_id import UserIdentifier
+
+ captured = {}
+
+ async def _fake_vectorize_file(**kwargs):
+ captured.update(kwargs)
+ return True
+
+ monkeypatch.setattr("openviking.utils.embedding_utils.vectorize_file", _fake_vectorize_file)
+ ctx = RequestContext(UserIdentifier("acc", "owner"), Role.USER)
+ acl_update = AclUpdate(
+ uri="viking://resources/private.md",
+ acl=AclSpec(
+ acl_mode="restricted",
+ entries=[{"principal": "user:reader", "level": "read"}],
+ ),
+ )
+
+ await context_update_execution.vectorize_resource_file(
+ "viking://resources/private.md",
+ ctx=ctx,
+ acl_update=acl_update,
+ scalar_override={"_record_id": "private-l2", "search_tags": ["team=search"]},
+ action="upsert",
+ )
+
+ options = captured["ingest_options"]
+ assert options.acl_update == acl_update
+ assert options.search_tags is None
+
+
+async def test_commit_and_enqueue_plan_forwards_acl_to_direct_actions(monkeypatch):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage import context_update_execution
+ from openviking.storage.acl import AclSpec, AclUpdate
+ from openviking.storage.context_update_plan import ContextUpdatePlan, DirectIndexAction
+ from openviking.utils.ingest_options import IngestOptions
+ from openviking_cli.session.user_id import UserIdentifier
+
+ uri = "viking://resources/private.md"
+ acl_update = AclUpdate(
+ uri=uri,
+ acl=AclSpec(acl_mode="restricted", entries=[]),
+ )
+ enqueue = AsyncMock(return_value=True)
+ monkeypatch.setattr(context_update_execution, "enqueue_direct_index_actions", enqueue)
+
+ await context_update_execution.commit_and_enqueue_plan(
+ ContextUpdatePlan(
+ uri,
+ "resource",
+ direct_index_actions=(DirectIndexAction("upsert", uri, 2, "private-l2", md5="new"),),
+ ),
+ ctx=RequestContext(UserIdentifier("acc", "owner"), Role.USER),
+ ingest_options=IngestOptions(
+ search_tags=["team=search"],
+ search_tag_mode="append",
+ acl_update=acl_update,
+ ),
+ )
+
+ enqueue.assert_awaited_once()
+ assert enqueue.await_args.kwargs["acl_update"] == acl_update
+
+
[email protected](("created", "expected_action"), [(True, "upsert"), (False, "")])
+async def test_commit_and_enqueue_plan_limits_created_file_upsert_to_write_adapter(
+ created, expected_action
+):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage import context_update_execution
+ from openviking.storage.context_update_plan import ContextUpdatePlan, FileRefreshIntent
+ from openviking_cli.session.user_id import UserIdentifier
+
+ uri = "viking://resources/new.md"
+ refresh = AsyncMock(
+ return_value={
+ "status": "success",
+ "enqueued_count": 1,
+ "semantic_action": "refresh_now",
+ }
+ )
+ summarizer = SimpleNamespace(refresh_file_parent=refresh)
+
+ await context_update_execution.commit_and_enqueue_plan(
+ ContextUpdatePlan(
+ uri,
+ "resource",
+ file_refresh=FileRefreshIntent(uri, "new-md5"),
+ ),
+ ctx=RequestContext(UserIdentifier("acc", "owner"), Role.USER),
+ file_created=created,
+ summarizer=summarizer,
+ )
+
+ assert refresh.await_args.kwargs["file_vector_action"] == expected_action
+
+
+async def test_direct_index_actions_delete_stale_record_when_embed_is_skipped(monkeypatch):
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage import context_update_execution
+ from openviking.storage.context_update_plan import DirectIndexAction
+ from openviking_cli.session.user_id import UserIdentifier
+
+ monkeypatch.setattr(
+ context_update_execution,
+ "vectorize_resource_file",
+ AsyncMock(return_value=False),
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.get_queue_manager",
+ lambda: SimpleNamespace(EMBEDDING="Embedding", get_queue=lambda *args, **kwargs: object()),
+ )
+ enqueue = AsyncMock(return_value=True)
+ monkeypatch.setattr("openviking.utils.embedding_utils._enqueue_embedding_message", enqueue)
+ ctx = RequestContext(UserIdentifier("acc", "user"), Role.USER)
+
+ enqueued = await context_update_execution.enqueue_direct_index_actions(
+ (
+ DirectIndexAction(
+ "upsert",
+ "viking://resources/empty.md",
+ 2,
+ "empty-l2",
+ md5="d41d8cd98f00b204e9800998ecf8427e",
+ ),
+ ),
+ ctx=ctx,
+ )
+
+ assert enqueued is True
+ delete_message = enqueue.await_args.args[1]
+ assert delete_message.record_ids == ["empty-l2"]
+
+
+async def test_direct_index_action_forwards_existing_summary_to_write_vectorizer(monkeypatch):
+ """vectors_only write must preserve the planner's summary_first input."""
+ from openviking.server.identity import RequestContext, Role
+ from openviking.storage import context_update_execution
+ from openviking.storage.context_update_plan import DirectIndexAction
+ from openviking_cli.session.user_id import UserIdentifier
+
+ vectorize = AsyncMock(return_value=True)
+ monkeypatch.setattr(context_update_execution, "vectorize_resource_file", vectorize)
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.get_queue_manager",
+ lambda: SimpleNamespace(EMBEDDING="Embedding", get_queue=lambda *args, **kwargs: object()),
+ )
+ action = DirectIndexAction(
+ "upsert",
+ "viking://resources/repo/a.md",
+ 2,
+ "a-l2",
+ md5="new-md5",
+ summary="existing L2 abstract",
+ )
+
+ await context_update_execution.enqueue_direct_index_actions(
+ (action,),
+ ctx=RequestContext(UserIdentifier("acc", "user"), Role.USER),
+ )
+
+ assert vectorize.await_args.kwargs["summary"] == "existing L2 abstract"
+ assert vectorize.await_args.kwargs["scalar_override"] == {"_record_id": "a-l2"}
+
+
def test_semantic_message_roundtrip_uses_explicit_plan():
from openviking.storage.context_update_plan import (
IndexSlot,
diff --git a/tests/storage/test_resource_diff_snapshot.py b/tests/storage/test_resource_diff_snapshot.py
index 1b8b2c46f..4474de112 100644
--- a/tests/storage/test_resource_diff_snapshot.py
+++ b/tests/storage/test_resource_diff_snapshot.py
@@ -9,7 +9,9 @@ import pytest
from openviking.parse.output import AgfsParseOutputStore, ParseArtifactRef
from openviking.storage.resource_diff import (
+ InlineBytesStore,
build_rnfv_snapshot,
+ make_inline_file_inventory,
prepare_artifact_inventory,
read_target_file_snapshot,
)
@@ -82,10 +84,26 @@ class _FakeVikingDB:
if uri == target_uri or uri.startswith(prefix)
}
- async def get_incremental_inventory_under_uri(self, target_uri, *, ctx, output_fields=None):
- del ctx
+ async def get_incremental_inventory_under_uri(
+ self,
+ target_uri,
+ *,
+ ctx,
+ output_fields=None,
+ recursive=True,
+ include_direct_children=False,
+ ):
+ del ctx, include_direct_children
self.inventory_output_fields = list(output_fields or [])
- prefix = target_uri.rstrip("/") + "/"
+ self.inventory_depth = -1 if recursive else 0
+ base = target_uri.rstrip("/")
+ prefix = base + "/"
+
+ def in_scope(uri: str) -> bool:
+ if uri == base or uri.startswith(base + "#"):
+ return True
+ return recursive and uri.startswith(prefix)
+
return {
str(value.get("id") or f"id-{index}"): {
key: item
@@ -99,7 +117,7 @@ class _FakeVikingDB:
if not output_fields or key in output_fields
}
for index, (uri, value) in enumerate(self._records.items())
- if uri == target_uri or uri.startswith(prefix)
+ if in_scope(uri)
}
@@ -350,9 +368,7 @@ async def test_build_rnfv_snapshot_projects_tags_only_for_effective_request_inte
)
snapshot = await build_rnfv_snapshot(
- viking_fs=_FakeVikingFS(
- [{"rel_path": "a.py", "isDir": False, "uri": f"{root}/a.py"}]
- ),
+ viking_fs=_FakeVikingFS([{"rel_path": "a.py", "isDir": False, "uri": f"{root}/a.py"}]),
vikingdb=vikingdb,
store=store,
artifact_ref=ref,
@@ -479,3 +495,148 @@ async def test_build_rnfv_snapshot_cancels_sibling_reads_after_failure(tmp_path,
with pytest.raises(RuntimeError, match="inventory failed"):
await task
assert cancelled == {"formal", "vectors"}
+
+
+def test_make_inline_file_inventory_uses_final_byte_md5():
+ from openviking.utils.content_hash import content_md5
+
+ inventory = make_inline_file_inventory(b"hello world")
+ assert set(inventory.entries) == {""}
+ entry = inventory.entries[""]
+ assert entry.is_dir is False
+ assert entry.md5 == content_md5(b"hello world")
+ assert inventory.artifact_paths == {"": ""}
+
+
+async def test_inline_bytes_store_returns_preset_bytes():
+ store = InlineBytesStore(b"payload")
+ assert await store.read_bytes(object(), "ignored") == b"payload"
+ assert await store.read_bytes(None, "") == b"payload"
+
+
+async def test_build_rnfv_snapshot_uses_locked_formal_snapshot_without_second_stat(monkeypatch):
+ root = "viking://resources/x/a.py"
+
+ async def _unexpected_formal_read(*args, **kwargs):
+ raise AssertionError("locked formal state must be reused")
+
+ async def _empty_inventory(*args, **kwargs):
+ return {}
+
+ monkeypatch.setattr(
+ "openviking.storage.resource_diff.read_target_file_snapshot", _unexpected_formal_read
+ )
+ monkeypatch.setattr(
+ "openviking.storage.resource_diff._read_incremental_vector_inventory", _empty_inventory
+ )
+
+ snapshot = await build_rnfv_snapshot(
+ viking_fs=_FakeVikingFS([]),
+ vikingdb=_FakeVikingDB({}),
+ store=InlineBytesStore(b"new"),
+ artifact_ref=object(),
+ target_uri=root,
+ ctx=_Ctx(),
+ artifact_inventory=make_inline_file_inventory(b"new"),
+ root_is_file=True,
+ target_preexisting=True,
+ formal_snapshot=({"": FormalEntry(is_dir=False)}, True),
+ vector_scope="self",
+ )
+
+ assert snapshot.formal.entries == {"": FormalEntry(is_dir=False)}
+
+
+async def test_build_rnfv_snapshot_self_scope_reads_only_target_uri():
+ root = "viking://resources/x/a.py"
+ vikingdb = _FakeVikingDB(
+ {
+ root: {"id": "self-l2", "level": 2, "md5": "old"},
+ "viking://resources/x/b.py": {"id": "sibling-l2", "level": 2, "md5": "sib"},
+ }
+ )
+
+ snapshot = await build_rnfv_snapshot(
+ viking_fs=_FakeVikingFS([]),
+ vikingdb=vikingdb,
+ store=InlineBytesStore(b"new"),
+ artifact_ref=object(),
+ target_uri=root,
+ ctx=_Ctx(),
+ artifact_inventory=make_inline_file_inventory(b"new"),
+ root_is_file=True,
+ target_preexisting=False,
+ vector_scope="self",
+ )
+
+ assert vikingdb.inventory_depth == 0
+ # Only the target file's own record is visible; the sibling is never read.
+ assert {record.uri for record in snapshot.vectors.records_by_id.values()} == {root}
+
+
+async def test_build_rnfv_snapshot_uses_preloaded_file_vector_inventory():
+ from unittest.mock import AsyncMock
+
+ root = "viking://resources/x/a.py"
+ vikingdb = _FakeVikingDB({})
+ vikingdb.get_incremental_inventory_under_uri = AsyncMock(
+ side_effect=AssertionError("preloaded V must avoid a second file inventory read")
+ )
+ preloaded_inventory = {
+ "self-l2": {
+ "id": "self-l2",
+ "uri": root,
+ "level": 2,
+ "md5": "old",
+ "abstract": "existing abstract",
+ }
+ }
+
+ snapshot = await build_rnfv_snapshot(
+ viking_fs=_FakeVikingFS([]),
+ vikingdb=vikingdb,
+ store=InlineBytesStore(b"new"),
+ artifact_ref=object(),
+ target_uri=root,
+ ctx=_Ctx(),
+ artifact_inventory=make_inline_file_inventory(b"new"),
+ root_is_file=True,
+ target_preexisting=False,
+ vector_scope="self",
+ vector_inventory=preloaded_inventory,
+ )
+
+ assert snapshot.vectors.records_by_id["self-l2"].fields["abstract"] == "existing abstract"
+ vikingdb.get_incremental_inventory_under_uri.assert_not_awaited()
+
+
+async def test_build_rnfv_snapshot_default_scope_reads_subtree():
+ root = "viking://resources/x"
+ vikingdb = _FakeVikingDB(
+ {
+ f"{root}/a.py": {"id": "a-l2", "level": 2, "md5": "a"},
+ f"{root}/sub/b.py": {"id": "b-l2", "level": 2, "md5": "b"},
+ }
+ )
+
+ snapshot = await build_rnfv_snapshot(
+ viking_fs=_FakeVikingFS([]),
+ vikingdb=vikingdb,
+ store=InlineBytesStore(b""),
+ artifact_ref=object(),
+ target_uri=root,
+ ctx=_Ctx(),
+ artifact_inventory=make_inline_file_inventory(b""),
+ target_preexisting=False,
+ )
+
+ assert vikingdb.inventory_depth == -1
+ assert {record.uri for record in snapshot.vectors.records_by_id.values()} == {
+ f"{root}/a.py",
+ f"{root}/sub/b.py",
+ }
diff --git a/tests/storage/test_semantic_executor_incremental.py b/tests/storage/test_semantic_executor_incremental.py
index 74213eb4b..14e9d2334 100644
--- a/tests/storage/test_semantic_executor_incremental.py
+++ b/tests/storage/test_semantic_executor_incremental.py
@@ -455,6 +455,7 @@ async def test_direct_incremental_update_uses_changes_without_temp_sync(monkeypa
# a skipped file produces no vector write.
if f"{root_uri}/a.txt" in expected_files:
assert processor.file_md5s[f"{root_uri}/a.txt"] == "md5-a-new"
+ assert processor.file_contents[("action", f"{root_uri}/a.txt")] == "merge"
else:
assert f"{root_uri}/a.txt" not in processor.file_md5s
overview = parse_abstract_overview(fake_fs._file_contents[f"{root_uri}/.overview.md"]).body
@@ -465,6 +466,39 @@ async def test_direct_incremental_update_uses_changes_without_temp_sync(monkeypa
assert parse_abstract_overview(fake_fs._file_contents[f"{root_uri}/.abstract.md"]).body.strip()
+async def test_direct_incremental_created_file_forces_vector_upsert(monkeypatch):
+ root_uri = "viking://resources/root"
+ file_uri = f"{root_uri}/new.txt"
+ fake_fs = _FakeVikingFS(
+ tree={root_uri: [{"name": "new.txt", "isDir": False}]},
+ file_contents={file_uri: "new body"},
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+ processor = _FakeProcessor(fake_fs)
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=1,
+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),
+ incremental_update=True,
+ target_uri=root_uri,
+ changes={"added": [file_uri]},
+ file_md5s={file_uri: "new-md5"},
+ file_vector_actions={file_uri: "upsert"},
+ )
+
+ await executor.run(root_uri)
+
+ assert processor.file_contents[("action", file_uri)] == "upsert"
+
+
@pytest.mark.asyncio
async def test_modified_file_with_same_abstract_still_propagates(monkeypatch):
root_uri = "viking://resources/root"
@@ -807,6 +841,45 @@ async def test_content_write_tags_apply_only_to_changed_file(monkeypatch):
assert processor.directory_ingest_options[root_uri] == IngestOptions()
+async def test_content_write_uses_per_file_ingest_options(monkeypatch):
+ root_uri = "viking://resources/root"
+ first_uri = f"{root_uri}/a.txt"
+ second_uri = f"{root_uri}/b.txt"
+ fake_fs = _FakeVikingFS(
+ tree={root_uri: [{"name": "a.txt", "isDir": False}, {"name": "b.txt", "isDir": False}]},
+ file_contents={first_uri: "new a", second_uri: "new b"},
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_viking_fs", lambda: fake_fs
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_executor.get_openviking_config",
+ lambda: SimpleNamespace(semantic=SimpleNamespace(overview_sample_limit=32)),
+ )
+ processor = _FakeProcessor(fake_fs)
+ first_options = IngestOptions(search_tags=["team=search"])
+ second_options = IngestOptions(search_tags=[], search_tag_mode="clear")
+ executor = SemanticTreeExecutor(
+ processor=processor,
+ context_type="resource",
+ max_concurrent_llm=2,
+ ctx=RequestContext(user=UserIdentifier("acc1", "user1"), role=Role.USER),
+ incremental_update=True,
+ target_uri=root_uri,
+ changes={"modified": [first_uri, second_uri]},
+ generation_trigger="content_write",
+ file_ingest_options={first_uri: first_options, second_uri: second_options},
+ )
+
+ await executor.run(root_uri)
+
+ assert processor.file_ingest_options == {
+ first_uri: first_options,
+ second_uri: second_options,
+ }
+
+
@pytest.mark.asyncio
async def test_content_write_with_same_abstract_updates_file_and_parent(monkeypatch):
root_uri = "viking://resources/root"
diff --git a/tests/storage/test_semantic_msg_ingest_options.py b/tests/storage/test_semantic_msg_ingest_options.py
index c53742e70..9e6644525 100644
--- a/tests/storage/test_semantic_msg_ingest_options.py
+++ b/tests/storage/test_semantic_msg_ingest_options.py
@@ -1,6 +1,7 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
+import pytest
from openviking.storage.acl import AclSpec, AclUpdate
from openviking.storage.queuefs.semantic_msg import SemanticMsg
@@ -103,6 +104,35 @@ def test_semantic_msg_defaults_file_md5s_to_empty():
assert SemanticMsg.from_dict(msg.to_dict()).file_md5s == {}
+def test_semantic_msg_roundtrip_preserves_file_vector_actions():
+ uri = "viking://resources/x/new.md"
+ msg = SemanticMsg(
+ uri="viking://resources/x",
+ context_type="resource",
+ file_vector_actions={uri: "upsert"},
+ )
+
+ restored = SemanticMsg.from_json(msg.to_json())
+
+ assert restored.file_vector_actions == {uri: "upsert"}
+
+
+def test_semantic_msg_defaults_file_vector_actions_to_empty():
+ msg = SemanticMsg(uri="viking://resources/x", context_type="resource")
+
+ assert msg.file_vector_actions == {}
+ assert SemanticMsg.from_dict(msg.to_dict()).file_vector_actions == {}
+
+
+def test_semantic_msg_rejects_invalid_file_vector_action():
+ with pytest.raises(ValueError, match="unsupported file vector action"):
+ SemanticMsg(
+ uri="viking://resources/x",
+ context_type="resource",
+ file_vector_actions={"viking://resources/x/a.md": "delete"},
+ )
+
+
def test_semantic_msg_roundtrip_preserves_queue_enqueue_time():
msg = SemanticMsg(
uri="viking://resources/x",
@@ -167,3 +197,35 @@ def test_clear_ingest_options_survive_semantic_message_round_trip():
search_tag_mode="clear",
acl_update=update,
)
+
+
+def test_semantic_msg_round_trips_per_file_ingest_options():
+ uri = "viking://resources/demo/a.md"
+ update = AclUpdate(
+ uri=uri,
+ acl=AclSpec(
+ acl_mode="restricted",
+ entries=[{"principal": "group:engineering", "level": "write"}],
+ ),
+ )
+ msg = SemanticMsg(
+ uri="viking://resources/demo",
+ context_type="resource",
+ file_ingest_options={
+ uri: IngestOptions(
+ search_tags=["team=search"],
+ search_tag_mode="append",
+ acl_update=update,
+ )
+ },
+ )
+
+ restored = SemanticMsg.from_json(msg.to_json())
+
+ assert restored.file_ingest_options == {
+ uri: IngestOptions(
+ search_tags=["team=search"],
+ search_tag_mode="append",
+ acl_update=update,
+ )
+ }
diff --git a/tests/storage/test_vector_transfer.py b/tests/storage/test_vector_transfer.py
index ffc92fd2c..15fbfde4e 100644
--- a/tests/storage/test_vector_transfer.py
+++ b/tests/storage/test_vector_transfer.py
@@ -70,6 +70,7 @@ class _MemoryTransferBackend(VikingVectorIndexBackend):
self.backend_mode = "local"
self.vector_dim = 2
self.scroll_filters = []
+ self.filter_calls = []
self.acl_manager = None
@property
@@ -104,6 +105,26 @@ class _MemoryTransferBackend(VikingVectorIndexBackend):
next_cursor = str(offset + limit) if offset + limit < len(ordered) else None
return page, next_cursor
+ async def filter(
+ self,
+ filter,
+ limit: int = 10,
+ offset: int = 0,
+ output_fields=None,
+ order_by=None,
+ order_desc: bool = False,
+ *,
+ ctx: RequestContext,
+ ) -> list[dict[str, Any]]:
+ del output_fields, order_by, order_desc, ctx
+ self.filter_calls.append((filter, limit, offset))
+ ordered = [
+ dict(self.records[key])
+ for key in sorted(self.records)
+ if _matches_filter(filter, self.records[key])
+ ]
+ return ordered[offset : offset + limit]
+
async def get(self, ids: list[str], *, ctx: RequestContext) -> list[dict[str, Any]]:
del ctx
return [dict(self.records[record_id]) for record_id in ids if record_id in self.records]
@@ -719,6 +740,85 @@ async def test_incremental_inventory_projects_request_scalar_fields():
assert all("sparse_vector" not in fields for fields in backend.scroll_output_fields)
+async def test_incremental_inventory_by_uris_reads_exact_uri_groups_only():
+ first = "viking://resources/docs/a.py"
+ second = "viking://resources/docs/b.py"
+ backend = _RealAclMemoryTransferBackend(
+ [
+ _record("a-l2", first, level=2, md5="ma", abstract="A"),
+ _record("b-l2", second, level=2, md5="mb", abstract="B"),
+ _record("parent-l1", "viking://resources/docs", level=1),
+ _record("sibling-l2", "viking://resources/docs/c.py", level=2),
+ ]
+ )
+
+ records = await backend.get_incremental_inventory_by_uris(
+ [first, second],
+ ctx=_ctx(),
+ batch_size=1,
+ output_fields=["id", "uri", "level", "md5", "abstract"],
+ )
+
+ assert records == {
+ first: {
+ "a-l2": {
+ "id": "a-l2",
+ "uri": first,
+ "level": 2,
+ "md5": "ma",
+ "abstract": "A",
+ }
+ },
+ second: {
+ "b-l2": {
+ "id": "b-l2",
+ "uri": second,
+ "level": 2,
+ "md5": "mb",
+ "abstract": "B",
+ }
+ },
+ }
+ assert all(isinstance(filter, And) for filter in backend.scroll_filters)
+ assert all(
+ any(isinstance(condition, In) and condition.field == "uri" for condition in filter.conds)
+ for filter in backend.scroll_filters
+ )
+
+
+async def test_incremental_inventory_by_uris_uses_one_exact_filter_per_uri_chunk():
+ first = "viking://resources/docs/a.py"
+ second = "viking://resources/docs/b.py"
+ backend = _RealAclMemoryTransferBackend(
+ [
+ _record("a-l2", first, level=2, md5="ma"),
+ _record("b-l2", second, level=2, md5="mb"),
+ ]
+ )
+
+ records = await backend.get_incremental_inventory_by_uris(
+ [first, second],
+ ctx=_ctx(),
+ batch_size=2,
+ )
+
+ assert set(records) == {first, second}
+ assert len(backend.filter_calls) == 1
+ assert backend.scroll_filters == []
+ scope, limit, offset = backend.filter_calls[0]
+ assert offset == 0
+ assert limit == 7 # 3 canonical levels per URI plus the truncation sentinel.
+ assert isinstance(scope, And)
+ assert any(
+ isinstance(condition, In)
+ and condition.field == "uri"
+ and set(condition.values) == {first, second}
+ for condition in scope.conds
+ )
+
+
@pytest.mark.asyncio
async def test_incremental_hydration_fetches_records_by_primary_key():
root = "viking://resources/docs"
diff --git a/tests/unit/test_summarizer_resources_root_split.py b/tests/unit/test_summarizer_resources_root_split.py
index c105239a4..cab5eed99 100644
--- a/tests/unit/test_summarizer_resources_root_split.py
+++ b/tests/unit/test_summarizer_resources_root_split.py
@@ -7,7 +7,8 @@ from unittest.mock import patch
import pytest
from openviking.server.identity import RequestContext, Role
-from openviking.utils.summarizer import Summarizer
+from openviking.utils.ingest_options import IngestOptions
+from openviking.utils.summarizer import FileRefreshRequest, Summarizer
from openviking_cli.session.user_id import UserIdentifier
@@ -220,7 +221,11 @@ async def test_flat_file_refresh_enqueues_incremental_parent_summary():
skip_vectorization=False,
)
- assert result == {"status": "success", "enqueued_count": 1}
+ assert result == {
+ "status": "success",
+ "enqueued_count": 1,
+ "semantic_action": "refresh_now",
+ }
assert len(queue.msgs) == 1
msg = queue.msgs[0]
assert msg.uri == "viking://resources"
@@ -232,6 +237,66 @@ async def test_flat_file_refresh_enqueues_incremental_parent_summary():
assert wait_tracker.registered == [("tid", msg.id)]
+async def test_grouped_file_refresh_coalesces_only_direct_parents():
+ queue = _DummyQueue()
+ qm = _DummyQueueManager(queue)
+ wait_tracker = _DummyWaitTracker()
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)
+
+ with (
+ patch("openviking.utils.summarizer.get_queue_manager", return_value=qm),
+ patch(
+ "openviking.utils.summarizer.get_current_telemetry",
+ return_value=SimpleNamespace(telemetry_id="tid"),
+ ),
+ patch(
+ "openviking.utils.summarizer.get_request_wait_tracker",
+ return_value=wait_tracker,
+ ),
+ ):
+ result = await Summarizer(vlm_processor=None).refresh_file_parents(
+ changes=[
+ FileRefreshRequest(
+ "viking://resources/docs/a.md",
+ created=True,
+ md5="a",
+ vector_action="upsert",
+ ingest_options=IngestOptions(search_tags=["team=search"]),
+ ),
+ FileRefreshRequest("viking://resources/docs/b.md", md5="b"),
+ FileRefreshRequest("viking://resources/src/main.py", md5="c"),
+ ],
+ ctx=ctx,
+ )
+
+ assert result["enqueued_count"] == 2
+ assert result["parent_actions"] == {
+ "viking://resources/docs": "refresh_now",
+ "viking://resources/src": "refresh_now",
+ }
+ assert len(queue.msgs) == 2
+ messages = {msg.uri: msg for msg in queue.msgs}
+ assert messages["viking://resources/docs"].changes == {
+ "added": ["viking://resources/docs/a.md"],
+ "modified": ["viking://resources/docs/b.md"],
+ }
+ assert messages["viking://resources/docs"].file_md5s == {
+ "viking://resources/docs/a.md": "a",
+ "viking://resources/docs/b.md": "b",
+ }
+ assert messages["viking://resources/docs"].file_vector_actions == {
+ "viking://resources/docs/a.md": "upsert"
+ }
+ assert messages["viking://resources/docs"].file_ingest_options == {
+ "viking://resources/docs/a.md": IngestOptions(search_tags=["team=search"])
+ }
+ assert messages["viking://resources/src"].changes == {
+ "modified": ["viking://resources/src/main.py"]
+ }
+ assert messages["viking://resources/src"].file_vector_actions == {}
+
+
@pytest.mark.asyncio
async def test_flat_file_refresh_marks_wait_failed_when_enqueue_fails():
qm = _DummyQueueManager(_FailingQueue())
diff --git a/tests/utils/test_resource_processor_processing_mode.py b/tests/utils/test_resource_processor_processing_mode.py
index 0cc515353..b8e0649d2 100644
--- a/tests/utils/test_resource_processor_processing_mode.py
+++ b/tests/utils/test_resource_processor_processing_mode.py
@@ -319,6 +319,82 @@ async def test_local_incremental_noop_skips_semantic_queue_and_releases_resource
assert not tmp_path.joinpath(ref.root).exists()
+async def test_add_resources_rnfv_uses_subtree_inventory_to_delete_orphans(monkeypatch, ctx):
+ from openviking.storage.resource_diff import ArtifactInventory
+ from openviking.storage.resource_rnfv import (
+ FormalTreeSnapshot,
+ NewArtifactSnapshot,
+ NewEntry,
+ RNFVSnapshot,
+ VectorIndexSnapshot,
+ )
+
+ root_uri = "viking://resources/demo"
+ captured = {}
+ viking_fs = SimpleNamespace()
+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: viking_fs)
+
+ artifact_inventory = ArtifactInventory(
+ entries={"a.py": NewEntry(md5="new")}, artifact_paths={"a.py": "a.py"}
+ )
+ monkeypatch.setattr(
+ "openviking.storage.resource_diff.prepare_artifact_inventory",
+ AsyncMock(return_value=artifact_inventory),
+ )
+
+ async def build_snapshot(**kwargs):
+ captured.update(kwargs)
+ return RNFVSnapshot(
+ request=kwargs["request_intent"],
+ new=NewArtifactSnapshot(entries=artifact_inventory.entries),
+ formal=FormalTreeSnapshot(entries={}),
+ vectors=VectorIndexSnapshot(records_by_id={}, projected_fields=frozenset()),
+ )
+
+ monkeypatch.setattr("openviking.storage.resource_diff.build_rnfv_snapshot", build_snapshot)
+ context_plan = SimpleNamespace(
+ root_uri=root_uri,
+ context_type="resource",
+ content_tree_actions=(),
+ direct_index_actions=(),
+ semantic_plan=None,
+ after_content_commit=lambda: {"status": "success"},
+ )
+ monkeypatch.setattr(
+ "openviking.storage.context_update_plan.build_context_update_plan_from_snapshot",
+ AsyncMock(return_value=(SimpleNamespace(entries={}), context_plan)),
+ )
+ monkeypatch.setattr(
+ "openviking.storage.context_update_plan.execute_content_tree_actions",
+ AsyncMock(),
+ )
+
+ processor = ResourceProcessor(_FakeVikingDB())
+ result = await processor._commit_directory_artifact_with_plan(
+ output_store=object(),
+ artifact_ref=SimpleNamespace(backend="local", root="artifact"),
+ doc_rel="repository",
+ root_uri=root_uri,
+ target_preexisting=True,
+ ctx=ctx,
+ lease_ref=None,
+ vectorize=True,
+ summarize=True,
+ processing_mode="semantic_and_vectors",
+ is_code_repo=False,
+ ingest_options=IngestOptions(),
+ source_metadata=None,
+ )
+
+ assert result == {"status": "success"}
+ # V must be read for the whole target subtree. Otherwise a V-only URI is
+ # invisible to RNFV and cannot reach the existing ABSENT -> DELETE branch.
+ assert captured.get("vector_inventory_strategy", "subtree") == "subtree"
+ assert "inventory_limit" not in captured
+ assert "inventory_query_batch_size" not in captured
+
+
@pytest.mark.asyncio
async def test_vectors_only_scalar_update_is_enqueued_before_lock_release(monkeypatch, ctx):
from openviking.storage.context_update_plan import DirectIndexAction
@@ -458,9 +534,7 @@ async def test_plan_artifact_is_cleaned_only_after_semantic_enqueue(monkeypatch,
assert tmp_path.joinpath("artifacts", ref.root.split("/")[-1]).exists()
return {"status": "success", "enqueued_count": 1}
- processor._summarizer = SimpleNamespace(
- summarize=AsyncMock(side_effect=summarize)
- )
+ processor._summarizer = SimpleNamespace(summarize=AsyncMock(side_effect=summarize))
await processor.finish_prepared_resource(
{