Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

fix(storage): 优化路径锁与语义刷新并发 (#2029)

Qin Haojie77b604a6412b
diff --git a/docs/en/api/03-filesystem.md b/docs/en/api/03-filesystem.md
index 1c6e8f171..a425b6777 100644
--- a/docs/en/api/03-filesystem.md
+++ b/docs/en/api/03-filesystem.md
@@ -497,7 +497,7 @@ openviking stat viking://resources/my-project/docs
}
```
-The `isLocked` field reports whether the path is currently held by a path lock — either the path itself has a valid `.path.ovlock`, or any ancestor directory holds a SUBTREE lock. Returns `false` when the LockManager is unavailable or the lookup fails, so callers can avoid attempting a write only to observe `ResourceBusyError`.
+The `isLocked` field reports whether the path is currently held by a path lock: the path itself has a valid lock (including an exact-path lock for the target), or any ancestor directory holds a TreeLock. Returns `false` when the LockManager is unavailable or the lookup fails, so callers can avoid attempting a write only to observe `ResourceBusyError`.
The `count` field (directories only) contains the estimated number of items (files and subdirectories) under this directory (from vector index).
diff --git a/docs/en/concepts/09-transaction.md b/docs/en/concepts/09-transaction.md
index 0cc0115c0..b9e4bce1a 100644
--- a/docs/en/concepts/09-transaction.md
+++ b/docs/en/concepts/09-transaction.md
@@ -39,9 +39,9 @@ Storage Layer (VikingFS, VectorDB, QueueManager)
## Two Core Components
-### Component 1: PathLock + LockManager + LockContext (Path Lock System)
+### Component 1: PathLockEngine + LockManager + LockContext (Path Lock System)
-**PathLock** implements file-based distributed locks with two lock types — POINT and SUBTREE — using fencing tokens to prevent TOCTOU races and automatic stale lock detection and cleanup.
+**PathLockEngine** implements file-based distributed locks with two lock types — EXACT and TREE — using fencing tokens to prevent TOCTOU races and automatic stale lock detection and cleanup.
**LockHandle** is a lightweight lock holder token:
@@ -64,7 +64,7 @@ class LockHandle:
```python
from openviking.storage.transaction import LockContext, get_lock_manager
-async with LockContext(get_lock_manager(), [path], lock_mode="point") as handle:
+async with LockContext(get_lock_manager(), [path], lock_mode="exact") as handle:
# Perform operations under lock protection
...
# Lock automatically released on exit (including exceptions)
@@ -89,8 +89,8 @@ Memory extraction is idempotent — re-extracting from the same archive produces
| Delete file first, then index -> file gone but index remains -> search returns non-existent file | **Reverse order**: delete index first, then file. Index deletion failure -> both file and index intact |
**Locking strategy** (depends on target type):
-- Deleting a **directory**: `lock_mode="subtree"`, locks the directory itself
-- Deleting a **file**: `lock_mode="point"`, locks the file's parent directory
+- Deleting a **directory**: `lock_mode="tree"`, locks the directory and its subtree
+- Deleting a **file**: `lock_mode="exact"`, locks the file path itself
Operation flow:
@@ -111,14 +111,14 @@ VectorDB deletion fails -> exception thrown, lock auto-released, file and index
| File moved to new path but index points to old path -> search returns old path (doesn't exist) | Copy first then update index; clean up copy on failure |
**Locking strategy** (handled automatically via `lock_mode="mv"`):
-- Moving a **directory**: SUBTREE lock on both source path and destination parent
-- Moving a **file**: POINT lock on both source's parent and destination parent
+- Moving a **directory**: TreeLock on the source path and ExactPathLock on the destination path
+- Moving a **file**: EXACT lock on both source path and destination path
Operation flow:
```
1. Check whether source is a directory or file, set src_is_dir
-2. Acquire mv lock (internally chooses SUBTREE or POINT based on src_is_dir)
+2. Acquire mv lock (internally chooses TreeLock or ExactPathLock based on src_is_dir)
3. Copy to new location (source still intact, safe)
4. If directory, remove the lock file carried over by cp into the copy
5. Update VectorDB URIs
@@ -132,36 +132,73 @@ Operation flow:
| Problem | Solution |
|---------|----------|
| File moved from temp to final directory, then crash -> file exists but never searchable | Two separate paths for first-time add vs incremental update |
-| Resource already on disk but rm deletes it while semantic processing / vectorization is still running -> wasted work | Lifecycle SUBTREE lock held from finalization through processing completion |
+| Resource already on disk but rm deletes it while semantic processing / vectorization is still running -> wasted work | Lifecycle TreeLock held from finalization through processing completion |
**First-time add** (target does not exist) — handled in `ResourceProcessor.process_resource` Phase 3.5:
```
-1. Acquire POINT lock on parent of final_uri
-2. agfs.mv temp directory -> final location
-3. Acquire SUBTREE lock on final_uri (inside POINT lock, eliminating race window)
-4. Release POINT lock
-5. Clean up temp directory
-6. Enqueue SemanticMsg(lifecycle_lock_handle_id=...) -> DAG runs on final
-7. DAG starts lock refresh loop (refreshes the lock token and updates handle activity every lock_expire/2 seconds)
-8. DAG complete + all embeddings done -> release SUBTREE lock
+1. Acquire TreeLock on final_uri
+ - If final_uri does not exist, check ancestor/descendant/same-path conflicts first
+ - If there is no conflict, create final_uri and write final_uri/.path.ovlock as a T lock
+2. Keep temp as the source directory and enqueue SemanticMsg(uri=temp, target_uri=final_uri, lifecycle_lock_handle_id=...)
+3. DAG runs on temp and syncs temp content into final_uri after completion
+ - Do not use raw agfs.mv(temp -> final_uri), because final_uri already exists for the lock file
+4. Clean up temp directory
+5. DAG starts lock refresh loop (refreshes the lock token and updates handle activity every lock_expire/2 seconds)
+6. DAG complete + all embeddings done -> release TreeLock
```
-During this period, `rm` attempting to acquire a SUBTREE lock on the same path will fail with `ResourceBusyError`.
+If summarization and indexing are both disabled, no downstream DAG takes over.
+In that case `ResourceProcessor` copies temp directory content into `final_uri`
+under the same TreeLock, deletes temp, then releases the lock. It does not call
+`VikingFS.mv(temp, final_uri, lock_handle=handle)`, because move cleanup can
+remove the directory lock file.
+
+During this period, `rm` attempting to acquire a TreeLock on the same path will fail with `ResourceBusyError`.
**Incremental update** (target already exists) — temp stays in place:
```
-1. Acquire SUBTREE lock on target_uri (protect existing resource)
+1. Acquire TreeLock on target_uri (protect existing resource)
2. Enqueue SemanticMsg(uri=temp, target_uri=final, lifecycle_lock_handle_id=...)
3. DAG runs on temp, lock refresh loop active
4. DAG completion triggers sync_diff_callback or move_temp_to_target_callback
-5. Callback completes -> release SUBTREE lock
+5. Callback completes -> release TreeLock
```
Note: DAG callbacks do NOT wrap operations in an outer lock. Each `VikingFS.rm` and `VikingFS.mv` has its own lock internally. An outer lock would conflict with these inner locks causing deadlock.
-**Server restart recovery**: SemanticMsg is persisted in QueueFS. On restart, `SemanticProcessor` detects that the `lifecycle_lock_handle_id` handle is missing from the in-memory LockManager and re-acquires a SUBTREE lock.
+Both first-time add and incremental update hold only `TreeLock(resource_dir)`.
+There is no `ExactPathLock(resource_dir) -> TreeLock(resource_dir)` handoff, so
+the two modes cannot accidentally release the same `.path.ovlock` file in the
+wrong scope.
+
+Automatic naming is handled by the resource layer, not the lock service:
+`ResourceProcessor` checks `exists(candidate_uri)` first; occupied candidates
+try `_1`, `_2`, and so on. Only a non-existing candidate attempts `TreeLock`,
+without waiting. If that candidate is busy, the next suffix is tried.
+
+**Server restart recovery**: SemanticMsg is persisted in QueueFS. On restart, `SemanticProcessor` detects that the `lifecycle_lock_handle_id` handle is missing from the in-memory LockManager and re-acquires a TreeLock.
+
+### Derived Semantic Files (.abstract.md / .overview.md)
+
+`.abstract.md` and `.overview.md` are generated sidecar files, not regular user source files. Their concurrency protection has two layers:
+
+| Problem | Solution |
+|---------|----------|
+| Multiple background tasks refresh the same directory summary and an old result overwrites a newer one | Messages for the same dirty key use `coalesce_version`; only the latest version may write back |
+| Two latest-stage writes interleave on the sidecar files | Acquire ExactPathLock on `.abstract.md` and `.overview.md` before writing |
+
+Example: concurrent writes to `docs/a.md`, `docs/b.md`, and `docs/c.md` hold separate ExactPathLocks and do not block each other. Background refresh may start multiple `docs/` summary tasks, but only the latest version writes `docs/.overview.md` and `docs/.abstract.md`; stale tasks drop their results before writeback.
+
+Memory directory summaries use the same rule. Concurrent writes to:
+
+```text
+viking://user/default/memories/preferences/theme.md
+viking://user/default/memories/preferences/editor.md
+```
+
+hold separate ExactPathLocks for the two source files. Refreshing `preferences/.overview.md` and `preferences/.abstract.md` no longer needs a long TreeLock; stale background tasks are filtered by `coalesce_version`, and final sidecar writes briefly acquire ExactPathLock.
### session.commit()
@@ -189,7 +226,7 @@ Phase 2 — Memory extraction + write (RedoLog):
**Crash recovery analysis**:
-| Crash point | State | Recovery action |
+| Failure moment | State | Recovery action |
|------------|-------|----------------|
| During Phase 1 archive write | No marker | Incomplete archive; next commit scans history/ for index, unaffected |
| Phase 1 archive complete but messages not cleared | No marker | Archive complete + messages still present = redundant but safe |
@@ -205,18 +242,18 @@ from openviking.storage.transaction import LockContext, get_lock_manager
lock_manager = get_lock_manager()
-# Point lock (write operations, semantic processing)
-async with LockContext(lock_manager, [path], lock_mode="point"):
+# Exact lock (write operations, semantic processing)
+async with LockContext(lock_manager, [path], lock_mode="exact"):
# Perform operations...
pass
-# Subtree lock (delete operations)
-async with LockContext(lock_manager, [path], lock_mode="subtree"):
+# Tree lock (directory delete and lifecycle protection)
+async with LockContext(lock_manager, [path], lock_mode="tree"):
# Perform operations...
pass
# MV lock (move operations)
-async with LockContext(lock_manager, [src], lock_mode="mv", mv_dst_parent_path=dst):
+async with LockContext(lock_manager, [src], lock_mode="mv", mv_dst_path=dst):
# Perform operations...
pass
```
@@ -225,50 +262,56 @@ async with LockContext(lock_manager, [src], lock_mode="mv", mv_dst_parent_path=d
| lock_mode | Use case | Behavior |
|-----------|----------|----------|
-| `point` | Write operations, semantic processing | Lock the specified path; conflicts with any lock on the same path and any SUBTREE lock on ancestors |
-| `subtree` | Delete operations | Lock the subtree root; conflicts with any lock on the same path, any lock on descendants, and any SUBTREE lock on ancestors |
-| `mv` | Move operations | Directory move: SUBTREE lock on both source and destination; File move: POINT lock on source parent and destination (controlled by `src_is_dir`) |
+| `exact` | File writes, single-file delete, sidecar writeback | Lock the specified path; conflicts with same-path locks and ancestor TreeLocks |
+| `tree` | Directory delete, resource lifecycle, directory-level protection | Lock the subtree root; conflicts with same-path locks, descendant locks, and ancestor TreeLocks |
+| `mv` | Move operations | Directory move: source TreeLock + destination ExactPathLock; File move: ExactPathLock on both source and destination (controlled by `src_is_dir`) |
**Exception handling**: `__aexit__` always releases locks and does not swallow exceptions. Lock acquisition failure raises `LockAcquisitionError`.
-## Lock Types (POINT vs SUBTREE)
+## Lock Types (EXACT vs TREE)
The lock mechanism uses two lock types to handle different conflict patterns:
-| | POINT on same path | SUBTREE on same path | POINT on descendant | SUBTREE on ancestor |
+| | EXACT on same path | TREE on same path | EXACT on descendant | TREE on ancestor |
|---|---|---|---|---|
-| **POINT** | Conflict | Conflict | — | Conflict |
-| **SUBTREE** | Conflict | Conflict | Conflict | Conflict |
+| **EXACT** | Conflict | Conflict | — | Conflict |
+| **TREE** | Conflict | Conflict | Conflict | Conflict |
-- **POINT (P)**: Used for write and semantic-processing operations. Only locks a single directory. Blocks if any ancestor holds a SUBTREE lock.
-- **SUBTREE (S)**: Used for rm and mv operations. Logically covers the entire subtree but only writes **one lock file** at the root. Before acquiring, scans all descendants and ancestor directories for conflicting locks.
+- **EXACT (E)**: Locks one concrete path. It can protect files, directory names, and not-yet-created target paths. Blocks if any ancestor holds a TreeLock.
+- **TREE (T)**: Used for directory delete, directory move, resource lifecycle protection, and similar subtree-level operations. Logically covers the entire subtree but only writes **one lock file** at the root. Before acquiring, scans all descendants and ancestor directories for conflicting locks. If the target directory is missing, conflicts are checked first; only then is the directory created and locked. If a later double-check finds a new conflict, the acquire fails or retries without rolling back the empty directory.
## Lock Mechanism
### Lock Protocol
-Lock file path: `{path}/.path.ovlock`
+Lock file paths:
+
+```text
+TreeLock(path) -> {path}/.path.ovlock
+ExactPathLock(existing dir path) -> {path}/.path.ovlock
+ExactPathLock(file or missing path) -> {parent}/.exact.ovlock.<name>.<hash>
+```
Lock file content (Fencing Token):
```
{handle_id}:{time_ns}:{lock_type}
```
-Where `lock_type` is `P` (POINT) or `S` (SUBTREE).
+Where `lock_type` is `E` (EXACT) or `T` (TREE).
-### Lock Acquisition (POINT mode)
+### Lock Acquisition (EXACT mode)
```
loop until timeout (poll interval: 200ms):
- 1. Check target directory exists
- 2. Check if target directory is locked by another operation
+ 1. Check if target path is locked by another operation
- Stale lock? -> remove and retry
- Active lock? -> wait
- 3. Check all ancestor directories for SUBTREE locks
+ 2. Check all ancestor directories for TREE locks
- Stale lock? -> remove and retry
- Active lock? -> wait
- 4. Write POINT (P) lock file
- 5. TOCTOU double-check: re-scan ancestors for SUBTREE locks
+ 3. Ensure the lock file's parent directory exists; create it if missing
+ 4. Write EXACT (E) lock file
+ 5. TOCTOU double-check: re-scan target path and ancestors for TREE locks
- Conflict found: compare (timestamp, handle_id)
- Later one (larger timestamp/handle_id) backs off (removes own lock) to prevent livelock
- Wait and retry
@@ -278,21 +321,22 @@ loop until timeout (poll interval: 200ms):
Timeout (default 0 = no-wait) raises LockAcquisitionError
```
-### Lock Acquisition (SUBTREE mode)
+### Lock Acquisition (TREE mode)
```
loop until timeout (poll interval: 200ms):
- 1. Check target directory exists
- 2. Check if target directory is locked by another operation
+ 1. Check if target directory is locked by another operation
- Stale lock? -> remove and retry
- Active lock? -> wait
- 3. Check all ancestor directories for SUBTREE locks
+ 2. Check all ancestor directories for TREE locks
- Stale lock? -> remove and retry
- Active lock? -> wait
- 4. Scan all descendant directories for any locks by other operations
+ 3. Scan all descendant directories for any locks by other operations
+ - Missing target directory? -> treat as no descendant locks
- Stale lock? -> remove and retry
- Active lock? -> wait
- 5. Write SUBTREE (S) lock file (only one file, at the root path)
+ 4. Ensure the target directory exists; create it if missing
+ 5. Write TREE (T) lock file (only one file, at the root path)
6. TOCTOU double-check: re-scan descendants and ancestors
- Conflict found: compare (timestamp, handle_id)
- Later one (larger timestamp/handle_id) backs off (removes own lock) to prevent livelock
@@ -303,9 +347,21 @@ loop until timeout (poll interval: 200ms):
Timeout (default 0 = no-wait) raises LockAcquisitionError
```
+### Missing Directory Creation
+
+The lock system may create directories so it can place lock files, but it checks
+for conflicts first:
+
+```
+1. Ancestor TreeLock / same-path lock / descendant lock conflict -> do not create the directory
+2. No current conflict -> create the directory and write the lock
+3. A post-write double-check finds a new conflict -> remove our own lock and fail or retry
+4. Step 3 does not roll back the empty directory
+```
+
### Lock Expiry Cleanup
-**Stale lock detection**: PathLock checks the fencing token timestamp. Locks older than `lock_expire` (default 300s) are considered stale and are removed automatically during acquisition.
+**Stale lock detection**: PathLockEngine checks the fencing token timestamp. Locks older than `lock_expire` (default 300s) are considered stale and are removed automatically during acquisition.
**In-process cleanup**: LockManager checks active LockHandles every 60 seconds. Handles that still own lock files but have been inactive for longer than `lock_expire` are force-released.
diff --git a/docs/zh/api/03-filesystem.md b/docs/zh/api/03-filesystem.md
index d53b21916..c17c90a1a 100644
--- a/docs/zh/api/03-filesystem.md
+++ b/docs/zh/api/03-filesystem.md
@@ -497,7 +497,7 @@ openviking stat viking://resources/my-project/docs
}
```
-`isLocked` 字段反映路径当前是否被 path lock 持有:路径自身存在有效的 `.path.ovlock`,或者任一祖先目录持有 SUBTREE 锁。当 LockManager 不可用或查询失败时返回 `false`,调用方可据此避免先写入再观察到 `ResourceBusyError`。
+`isLocked` 字段反映路径当前是否被路径锁持有:路径自身存在有效锁(包括目标路径对应的 exact-path lock),或者任一祖先目录持有 TreeLock。当 LockManager 不可用或查询失败时返回 `false`,调用方可据此避免先写入再观察到 `ResourceBusyError`。
`count` 字段(仅目录)包含该目录下的项目(文件和子目录)估计数量(来自向量索引)。
diff --git a/docs/zh/concepts/09-transaction.md b/docs/zh/concepts/09-transaction.md
index 9b29fc506..d00ca3c86 100644
--- a/docs/zh/concepts/09-transaction.md
+++ b/docs/zh/concepts/09-transaction.md
@@ -38,9 +38,9 @@ Storage Layer (VikingFS, VectorDB, QueueManager)
## 两个核心组件
-### 组件 1:PathLock + LockManager + LockContext(路径锁系统)
+### 组件 1:PathLockEngine + LockManager + LockContext(路径锁系统)
-**PathLock** 实现基于文件的分布式锁,支持 POINT 和 SUBTREE 两种锁类型,使用 fencing token 防止 TOCTOU 竞争,自动检测并清理过期锁。
+**PathLockEngine** 实现基于文件的分布式锁,支持 EXACT 和 TREE 两种锁类型,使用 fencing token 防止 TOCTOU 竞争,自动检测并清理过期锁。
**LockHandle** 是轻量的锁持有者令牌:
@@ -63,7 +63,7 @@ class LockHandle:
```python
from openviking.storage.transaction import LockContext, get_lock_manager
-async with LockContext(get_lock_manager(), [path], lock_mode="point") as handle:
+async with LockContext(get_lock_manager(), [path], lock_mode="exact") as handle:
# 在锁保护下执行操作
...
# 退出时自动释放锁(包括异常情况)
@@ -88,8 +88,8 @@ Memory 提取是幂等的 — 从同一个 archive 重新提取会得到相同
| 先删文件再删索引 -> 文件已删但索引残留 -> 搜索返回不存在的文件 | **调换顺序**:先删索引再删文件。索引删除失败 -> 文件和索引都在,搜索正常 |
**加锁策略**(根据目标类型区分):
-- 删除**目录**:`lock_mode="subtree"`,锁目录自身
-- 删除**文件**:`lock_mode="point"`,锁文件的父目录
+- 删除**目录**:`lock_mode="tree"`,锁目录自身及其整棵子树
+- 删除**文件**:`lock_mode="exact"`,锁文件路径本身
操作流程:
@@ -110,14 +110,14 @@ VectorDB 删除失败 -> 直接抛异常,锁自动释放,文件和索引都
| 文件移到新路径但索引指向旧路径 -> 搜索返回旧路径(不存在) | 先 copy 再更新索引,失败时清理副本 |
**加锁策略**(通过 `lock_mode="mv"` 自动处理):
-- 移动**目录**:源路径和目标父目录各加 SUBTREE 锁
-- 移动**文件**:源的父目录和目标父目录各加 POINT 锁
+- 移动**目录**:源路径加 TreeLock,目标路径加 ExactPathLock
+- 移动**文件**:源路径和目标路径各加 EXACT 锁
操作流程:
```
1. 检查源是目录还是文件,确定 src_is_dir
-2. 获取 mv 锁(内部根据 src_is_dir 选择 SUBTREE 或 POINT)
+2. 获取 mv 锁(内部根据 src_is_dir 选择 TreeLock 或 ExactPathLock)
3. Copy 到新位置(源还在,安全)
4. 如果是目录,删除副本中被 cp 带过去的锁文件
5. 更新 VectorDB 中的 URI
@@ -131,36 +131,69 @@ VectorDB 删除失败 -> 直接抛异常,锁自动释放,文件和索引都
| 问题 | 方案 |
|------|------|
| 文件从临时目录移到正式目录后崩溃 -> 文件存在但永远搜不到 | 首次添加与增量更新分离为两条独立路径 |
-| 资源已落盘但语义处理/向量化还在跑时被 rm 删除 -> 处理白跑 | 生命周期 SUBTREE 锁,从落盘持续到处理完成 |
+| 资源已落盘但语义处理/向量化还在跑时被 rm 删除 -> 处理白跑 | 生命周期 TreeLock,从落盘持续到处理完成 |
**首次添加**(target 不存在)— 在 `ResourceProcessor.process_resource` Phase 3.5 中处理:
```
-1. 获取 POINT 锁,锁 final_uri 的父目录
-2. agfs.mv 临时目录 -> 正式位置
-3. 获取 SUBTREE 锁,锁 final_uri(在 POINT 锁内,消除竞态窗口)
-4. 释放 POINT 锁
-5. 清理临时目录
-6. 入队 SemanticMsg(lifecycle_lock_handle_id=...) -> DAG 在 final 上跑
-7. DAG 启动锁刷新循环(每 lock_expire/2 秒刷新锁 token 并更新 handle 活跃时间)
-8. DAG 完成 + 所有 embedding 完成 -> 释放 SUBTREE 锁
+1. 获取 TreeLock,锁 final_uri
+ - 如果 final_uri 目录不存在,先检查祖先/后代/同路径锁冲突
+ - 无冲突则创建 final_uri 目录,并在 final_uri/.path.ovlock 写 T 锁
+2. 保留 temp 作为源目录,入队 SemanticMsg(uri=temp, target_uri=final_uri, lifecycle_lock_handle_id=...)
+3. DAG 在 temp 上跑,完成后把 temp 内容同步到 final_uri
+ - final_uri 已经用于放锁文件,所以不做裸 agfs.mv(temp -> final_uri)
+4. 清理临时目录
+5. DAG 启动锁刷新循环(每 lock_expire/2 秒刷新锁 token 并更新 handle 活跃时间)
+6. DAG 完成 + 所有 embedding 完成 -> 释放 TreeLock
```
-此期间 `rm` 尝试获取同路径 SUBTREE 锁会失败,抛出 `ResourceBusyError`。
+如果本次调用关闭了摘要和索引(没有下游 DAG 接管),则在同一把 TreeLock
+里把 temp 目录内容复制到 `final_uri`,清理 temp,然后释放锁。这里不调用
+`VikingFS.mv(temp, final_uri, lock_handle=handle)`,避免移动逻辑清理目录锁文件。
+
+此期间 `rm` 尝试获取同路径 TreeLock 会失败,抛出 `ResourceBusyError`。
**增量更新**(target 已存在)— temp 保持不动:
```
-1. 获取 SUBTREE 锁,锁 target_uri(保护已有资源)
+1. 获取 TreeLock,锁 target_uri(保护已有资源)
2. 入队 SemanticMsg(uri=temp, target_uri=final, lifecycle_lock_handle_id=...)
3. DAG 在 temp 上跑,启动锁刷新循环
4. DAG 完成后触发 sync_diff_callback 或 move_temp_to_target_callback
-5. callback 执行完毕 -> 释放 SUBTREE 锁
+5. callback 执行完毕 -> 释放 TreeLock
```
注意:DAG callback 不在外层加锁。每个 `VikingFS.rm` 和 `VikingFS.mv` 内部各自有独立锁保护。外层锁会与内部锁冲突导致死锁。
-**服务重启恢复**:SemanticMsg 持久化在 QueueFS 中。重启后 `SemanticProcessor` 发现 `lifecycle_lock_handle_id` 对应的 handle 不在内存中,会重新获取 SUBTREE 锁。
+首次添加和增量更新都只持有 `TreeLock(resource_dir)`。这里不再做
+`ExactPathLock(resource_dir) -> TreeLock(resource_dir)` 的锁转交,避免两种锁复用
+同一个 `.path.ovlock` 时出现释放顺序错误。
+
+自动命名由资源层处理,不属于锁服务:`ResourceProcessor` 先用 `exists(candidate_uri)`
+判断候选目录是否已占用;已存在则尝试 `_1`、`_2` 后缀。候选目录不存在时才尝试
+获取该目录的 `TreeLock`,且不等待;如果同名正在被并发请求处理,就直接尝试下一个后缀。
+
+**服务重启恢复**:SemanticMsg 持久化在 QueueFS 中。重启后 `SemanticProcessor` 发现 `lifecycle_lock_handle_id` 对应的 handle 不在内存中,会重新获取 TreeLock。
+
+### 派生语义文件(.abstract.md / .overview.md)
+
+`.abstract.md` 和 `.overview.md` 是后台生成的派生文件,不作为普通用户源文件写入。它们的并发保护分两层:
+
+| 问题 | 方案 |
+|------|------|
+| 多个后台任务同时刷新同一个目录摘要,旧结果覆盖新结果 | 相同 dirty key 使用 `coalesce_version`,只有最新版本允许写回 |
+| 最新任务写回派生文件时与另一个写回交错 | 写 `.abstract.md`、`.overview.md` 前获取各自的 ExactPathLock |
+
+例子:同一目录下并发写入 `a.md`、`b.md`、`c.md` 时,前台写入分别持有 `ExactPathLock(a.md)`、`ExactPathLock(b.md)`、`ExactPathLock(c.md)`,互不阻塞。后台可能产生多个 `docs/` 摘要刷新任务,但只有最新 version 能写回 `docs/.overview.md` 和 `docs/.abstract.md`;旧任务在写回前发现自己过期后直接丢弃结果。
+
+memory 目录摘要使用同一规则。比如并发更新:
+
+```text
+viking://user/default/memories/preferences/theme.md
+viking://user/default/memories/preferences/editor.md
+```
+
+两个文件写入各自持有 ExactPathLock;`preferences/.overview.md` 和 `preferences/.abstract.md` 的后台刷新不再持有长时间 TreeLock,而是通过 `coalesce_version` 淘汰旧任务,并在最终写派生文件时短暂获取 ExactPathLock。
### session.commit()
@@ -204,18 +237,18 @@ from openviking.storage.transaction import LockContext, get_lock_manager
lock_manager = get_lock_manager()
-# Point 锁(写操作、语义处理)
-async with LockContext(lock_manager, [path], lock_mode="point"):
+# Exact 锁(写操作、语义处理)
+async with LockContext(lock_manager, [path], lock_mode="exact"):
# 执行操作...
pass
-# Subtree 锁(删除操作)
-async with LockContext(lock_manager, [path], lock_mode="subtree"):
+# Tree 锁(删除目录、目录生命周期保护)
+async with LockContext(lock_manager, [path], lock_mode="tree"):
# 执行操作...
pass
# MV 锁(移动操作)
-async with LockContext(lock_manager, [src], lock_mode="mv", mv_dst_parent_path=dst):
+async with LockContext(lock_manager, [src], lock_mode="mv", mv_dst_path=dst):
# 执行操作...
pass
```
@@ -224,50 +257,56 @@ async with LockContext(lock_manager, [src], lock_mode="mv", mv_dst_parent_path=d
| lock_mode | 用途 | 行为 |
|-----------|------|------|
-| `point` | 写操作、语义处理 | 锁定指定路径;与同路径的任何锁和祖先目录的 SUBTREE 锁冲突 |
-| `subtree` | 删除操作 | 锁定子树根节点;与同路径的任何锁、后代目录的任何锁和祖先目录的 SUBTREE 锁冲突 |
-| `mv` | 移动操作 | 目录移动:源和目标均加 SUBTREE 锁;文件移动:源父目录和目标均加 POINT 锁(通过 `src_is_dir` 控制) |
+| `exact` | 文件写入、单文件删除、派生文件写回 | 锁定指定路径;与同路径锁和祖先目录 TreeLock 冲突 |
+| `tree` | 删除目录、资源生命周期、目录级保护 | 锁定子树根节点;与同路径锁、后代锁和祖先 TreeLock 冲突 |
+| `mv` | 移动操作 | 目录移动:源路径 TreeLock + 目标路径 ExactPathLock;文件移动:源路径和目标路径均 ExactPathLock(通过 `src_is_dir` 控制) |
**异常处理**:`__aexit__` 总是释放锁,不吞异常。获取锁失败时抛出 `LockAcquisitionError`。
-## 锁类型(POINT vs SUBTREE)
+## 锁类型(EXACT vs TREE)
锁机制使用两种锁类型来处理不同的冲突场景:
-| | 同路径 POINT | 同路径 SUBTREE | 后代 POINT | 祖先 SUBTREE |
+| | 同路径 EXACT | 同路径 TREE | 后代 EXACT | 祖先 TREE |
|---|---|---|---|---|
-| **POINT** | 冲突 | 冲突 | — | 冲突 |
-| **SUBTREE** | 冲突 | 冲突 | 冲突 | 冲突 |
+| **EXACT** | 冲突 | 冲突 | — | 冲突 |
+| **TREE** | 冲突 | 冲突 | 冲突 | 冲突 |
-- **POINT (P)**:用于写操作和语义处理。只锁单个目录。若祖先目录持有 SUBTREE 锁则阻塞。
-- **SUBTREE (S)**:用于删除和移动操作。逻辑上覆盖整个子树,但只在根目录写**一个锁文件**。获取前扫描所有后代和祖先目录确认无冲突锁。
+- **EXACT (E)**:锁定一个具体路径本身。文件、目录名、尚未创建的目标路径都可以使用;若祖先目录持有 TreeLock 则阻塞。
+- **TREE (T)**:用于删除目录、移动目录、资源生命周期保护等。逻辑上覆盖整棵子树,但只在根目录写**一个锁文件**。获取前扫描所有后代和祖先目录确认无冲突锁。目标目录不存在时,先做冲突检查;无冲突才创建目录并写锁。若创建后又发现并发冲突,本次加锁失败,但不回滚刚创建出来的空目录。
## 锁机制
### 锁协议
-锁文件路径:`{path}/.path.ovlock`
+锁文件路径:
+
+```text
+TreeLock(path) -> {path}/.path.ovlock
+ExactPathLock(已存在目录 path) -> {path}/.path.ovlock
+ExactPathLock(文件或未创建路径) -> {parent}/.exact.ovlock.<name>.<hash>
+```
锁文件内容(Fencing Token):
```
{handle_id}:{time_ns}:{lock_type}
```
-其中 `lock_type` 为 `P`(POINT)或 `S`(SUBTREE)。
+其中 `lock_type` 为 `E`(EXACT)或 `T`(TREE)。
-### 获取锁流程(POINT 模式)
+### 获取锁流程(EXACT 模式)
```
循环直到超时(轮询间隔:200ms):
- 1. 检查目标目录存在
- 2. 检查目标路径是否被其他操作锁定
+ 1. 检查目标路径是否被其他操作锁定
- 陈旧锁? -> 移除后重试
- 活跃锁? -> 等待
- 3. 检查所有祖先目录是否有 SUBTREE 锁
+ 2. 检查所有祖先目录是否有 TREE 锁
- 陈旧锁? -> 移除后重试
- 活跃锁? -> 等待
- 4. 写入 POINT (P) 锁文件
- 5. TOCTOU 双重检查:重新扫描祖先目录的 SUBTREE 锁
+ 3. 确保锁文件所在父目录存在;如果不存在则创建目录
+ 4. 写入 EXACT (E) 锁文件
+ 5. TOCTOU 双重检查:重新扫描目标路径和祖先目录的 TREE 锁
- 发现冲突:比较 (timestamp, handle_id)
- 后到者(更大的 timestamp/handle_id)主动让步(删除自己的锁),防止活锁
- 等待后重试
@@ -277,21 +316,22 @@ async with LockContext(lock_manager, [src], lock_mode="mv", mv_dst_parent_path=d
超时(默认 0 = 不等待)抛出 LockAcquisitionError
```
-### 获取锁流程(SUBTREE 模式)
+### 获取锁流程(TREE 模式)
```
循环直到超时(轮询间隔:200ms):
- 1. 检查目标目录存在
- 2. 检查目标路径是否被其他操作锁定
+ 1. 检查目标路径是否被其他操作锁定
- 陈旧锁? -> 移除后重试
- 活跃锁? -> 等待
- 3. 检查所有祖先目录是否有 SUBTREE 锁
+ 2. 检查所有祖先目录是否有 TREE 锁
- 陈旧锁? -> 移除后重试
- 活跃锁? -> 等待
- 4. 扫描所有后代目录,检查是否有其他操作持有的锁
+ 3. 扫描所有后代目录,检查是否有其他操作持有的锁
+ - 目标目录不存在? -> 视为无后代锁
- 陈旧锁? -> 移除后重试
- 活跃锁? -> 等待
- 5. 写入 SUBTREE (S) 锁文件(只写一个文件,在根路径)
+ 4. 确保目标目录存在;如果不存在则创建目录
+ 5. 写入 TREE (T) 锁文件(只写一个文件,在根路径)
6. TOCTOU 双重检查:重新扫描后代目录和祖先目录
- 发现冲突:比较 (timestamp, handle_id)
- 后到者(更大的 timestamp/handle_id)主动让步(删除自己的锁),防止活锁
@@ -302,9 +342,35 @@ async with LockContext(lock_manager, [src], lock_mode="mv", mv_dst_parent_path=d
超时(默认 0 = 不等待)抛出 LockAcquisitionError
```
+### 缺失目录创建规则
+
+锁系统允许为了放置锁文件而创建目录,但创建前必须先检查冲突:
+
+```
+1. 发现祖先 TreeLock / 同路径锁 / 后代锁冲突 -> 不创建目录,直接失败或等待
+2. 当前无冲突 -> 可以创建目录并写锁
+3. 写锁后再次检查时发现新冲突 -> 删除自己的锁并失败或重试
+4. 第 3 步不会回滚刚创建的空目录
+```
+
+例子:
+
+```text
+请求 A 正在删除 viking://resources/books
+=> A 持有 TreeLock(/resources/books)
+
+请求 B 想添加 viking://resources/books/java-guide
+=> B 在创建 java-guide 目录前发现祖先 TreeLock
+=> B 不创建目录,返回 busy
+```
+
+如果两个请求同时创建 `java-guide`,两边都可能先看到“当前无冲突”,但最终只有
+fencing token 校验通过的一方成功持有 `TreeLock(java-guide)`;失败方会删除自己的锁,
+已创建出来的空目录可以保留。
+
### 锁过期清理
-**陈旧锁检测**:PathLock 检查 fencing token 中的时间戳。超过 `lock_expire`(默认 300s)的锁被视为陈旧锁,在加锁过程中自动移除。
+**陈旧锁检测**:PathLockEngine 检查 fencing token 中的时间戳。超过 `lock_expire`(默认 300s)的锁被视为陈旧锁,在加锁过程中自动移除。
**进程内清理**:LockManager 每 60 秒检查活跃的 LockHandle。仍持有锁文件且失活时间超过 `lock_expire` 的 handle 会被强制释放。
diff --git a/examples/opencode-plugin/lib/repo-context.mjs b/examples/opencode-plugin/lib/repo-context.mjs
index 0e509d869..91e122bea 100644
--- a/examples/opencode-plugin/lib/repo-context.mjs
+++ b/examples/opencode-plugin/lib/repo-context.mjs
@@ -65,4 +65,3 @@ function formatRepoLine(item) {
const abstract = item.abstract || item.overview
return abstract ? `- **${name}** (${item.uri})\n ${abstract}` : `- **${name}** (${item.uri})`
}
-
diff --git a/openviking/async_client.py b/openviking/async_client.py
index d97a25495..92cd1e17e 100644
--- a/openviking/async_client.py
+++ b/openviking/async_client.py
@@ -251,7 +251,7 @@ class AsyncOpenViking:
reason: Context/reason for adding this resource.
instruction: Specific instruction for processing.
wait: If True, wait for processing to complete.
- to: Exact target URI (must not exist yet).
+ to: Exact target URI. Existing targets keep the add_resource incremental-update behavior.
parent: Target parent URI (must already exist).
build_index: Whether to build vector index immediately (default: True).
summarize: Whether to generate summary (default: False).
diff --git a/openviking/parse/tree_builder.py b/openviking/parse/tree_builder.py
index 444ae1f03..1b2b59573 100644
--- a/openviking/parse/tree_builder.py
+++ b/openviking/parse/tree_builder.py
@@ -8,12 +8,13 @@ L0/L1/L2 content and URI structure.
v5.0 Architecture:
1. Parser: parse + create directory structure in temp VikingFS
-2. TreeBuilder: move to AGFS + enqueue to SemanticQueue + create Resources
-3. SemanticProcessor: async generate L0/L1 + vectorize
+2. TreeBuilder: build final URI metadata and keep temp references
+3. ResourceProcessor: source commit from temp to final VikingFS path
+4. SemanticProcessor: async generate L0/L1 + vectorize
IMPORTANT (v5.0 Architecture):
- Parser creates directory structure directly, no LLM calls
-- TreeBuilder moves files and enqueues to SemanticQueue
+- TreeBuilder does not move files; source commit is handled after URI metadata is built
- SemanticProcessor handles all semantic generation asynchronously
- Temporary directory approach eliminates memory pressure and enables concurrency
- Resource objects are lightweight (no content fields)
@@ -40,18 +41,20 @@ class TreeBuilder:
New v5.0 Architecture:
- Parser creates directory structure in temp VikingFS (no LLM calls)
- - TreeBuilder moves to AGFS + enqueues to SemanticQueue + creates Resources
+ - TreeBuilder builds final URI metadata while preserving temp URIs
+ - ResourceProcessor commits temp content to the final source path
- SemanticProcessor handles semantic generation asynchronously
Process flow:
1. Parser creates directory structure with files in temp VikingFS
- 2. TreeBuilder.finalize_from_temp() moves to AGFS, enqueues to SemanticQueue, creates Resources
- 3. SemanticProcessor generates .abstract.md and .overview.md asynchronously
- 4. SemanticProcessor directly vectorizes and inserts to collection
+ 2. TreeBuilder.finalize_from_temp() returns final URI and temp URI metadata
+ 3. ResourceProcessor performs source commit with short path locks
+ 4. SemanticProcessor generates .abstract.md and .overview.md asynchronously
+ 5. SemanticProcessor directly vectorizes and inserts to collection
Key changes from v4.0:
- Semantic generation moved from Parser to SemanticQueue
- - TreeBuilder enqueues directories for async processing
+ - ResourceProcessor enqueues directories for async processing
- Direct vectorization in SemanticProcessor (no EmbeddingQueue)
"""
@@ -75,31 +78,6 @@ class TreeBuilder:
# Agent scope
return "viking://agent"
- async def _resolve_unique_uri(self, uri: str, max_attempts: int = 100) -> str:
- """Return a URI that does not collide with an existing resource.
-
- If *uri* is free, return it unchanged. Otherwise append ``_1``,
- ``_2``, ... until a free name is found.
- """
- viking_fs = get_viking_fs()
-
- async def _exists(u: str) -> bool:
- try:
- await viking_fs.stat(u)
- return True
- except Exception:
- return False
-
- if not await _exists(uri):
- return uri
-
- for i in range(1, max_attempts + 1):
- candidate = f"{uri}_{i}"
- if not await _exists(candidate):
- return candidate
-
- raise FileExistsError(f"Cannot resolve unique name for {uri} after {max_attempts} attempts")
-
# ============================================================================
# v5.0 Methods (temporary directory + SemanticQueue architecture)
# ============================================================================
@@ -116,10 +94,10 @@ class TreeBuilder:
create_parent: bool = False,
) -> "BuildingTree":
"""
- Finalize processing by moving from temp to AGFS.
+ Finalize URI metadata for a temp parse result.
Args:
- to_uri: Exact target URI (must not exist)
+ to_uri: Exact target URI, or resources root to import under
parent_uri: Target parent URI (must exist unless create_parent is True)
create_parent: Whether to automatically create parent directory if it doesn't exist
"""
@@ -171,6 +149,8 @@ class TreeBuilder:
candidate_uri = to_uri
else:
effective_parent_uri = parent_uri or to_uri if use_to_as_parent else parent_uri
+ if effective_parent_uri:
+ effective_parent_uri = effective_parent_uri.rstrip("/")
if effective_parent_uri:
# Parent URI must exist and be a directory, or create it if requested
try:
@@ -202,26 +182,16 @@ class TreeBuilder:
base_uri = effective_parent_uri
candidate_uri = VikingURI(base_uri).join(final_doc_name).uri
- if to_uri and not use_to_as_parent:
- final_uri = candidate_uri
- elif use_to_as_parent:
- # Treat an explicit resources root target as "import under this
- # directory" while preserving the child URI so downstream logic can
- # incrementally update viking://resources/<child> when it exists.
- final_uri = candidate_uri
- else:
- final_uri = await self._resolve_unique_uri(candidate_uri)
-
tree = BuildingTree(
source_path=source_path,
source_format=source_format,
)
- tree._root_uri = final_uri
+ tree._root_uri = candidate_uri
if not to_uri or use_to_as_parent:
tree._candidate_uri = candidate_uri
# Create a minimal Context object for the root so that tree.root is not None
- root_context = Context(uri=final_uri, temp_uri=temp_doc_uri)
+ root_context = Context(uri=candidate_uri, temp_uri=temp_doc_uri)
tree.add_context(root_context)
return tree
diff --git a/openviking/server/error_mapping.py b/openviking/server/error_mapping.py
index e9b8079b0..68be4dd22 100644
--- a/openviking/server/error_mapping.py
+++ b/openviking/server/error_mapping.py
@@ -11,7 +11,7 @@ from openviking.pyagfs.exceptions import (
AGFSHTTPError,
AGFSTimeoutError,
)
-from openviking.storage.errors import ResourceBusyError
+from openviking.storage.errors import LockAcquisitionError, ResourceBusyError
from openviking_cli.exceptions import (
ConflictError,
FailedPreconditionError,
@@ -404,7 +404,21 @@ def map_exception(
if isinstance(exc, OpenVikingError):
return exc
if isinstance(exc, ResourceBusyError):
- return ConflictError(str(exc), resource=resource)
+ details: dict[str, Any] = {
+ "resource": exc.uri or resource,
+ "uri": exc.uri or resource,
+ "conflict_type": exc.conflict_type,
+ "retryable": exc.retryable,
+ }
+ return OpenVikingError(str(exc), code="CONFLICT", details=details)
+ if isinstance(exc, LockAcquisitionError):
+ details = {
+ "resource": resource,
+ "uri": resource,
+ "conflict_type": "path_busy",
+ "retryable": True,
+ }
+ return OpenVikingError(str(exc), code="CONFLICT", details=details)
if isinstance(exc, PermissionError):
return PermissionDeniedError(str(exc), resource=resource)
if isinstance(exc, FileNotFoundError):
diff --git a/openviking/server/routers/filesystem.py b/openviking/server/routers/filesystem.py
index 69f2a8bd4..d68750603 100644
--- a/openviking/server/routers/filesystem.py
+++ b/openviking/server/routers/filesystem.py
@@ -207,4 +207,9 @@ async def mv(
if "not found" in err_msg or "no such file or directory" in err_msg:
raise NotFoundError(from_uri, "file")
raise
+ except Exception as exc:
+ mapped = map_exception(exc, resource=from_uri)
+ if mapped is not None:
+ raise mapped from exc
+ raise
return Response(status="ok", result={"from": from_uri, "to": to_uri})
diff --git a/openviking/server/temp_upload_store.py b/openviking/server/temp_upload_store.py
index b3ebea277..40bb4f1e3 100644
--- a/openviking/server/temp_upload_store.py
+++ b/openviking/server/temp_upload_store.py
@@ -318,7 +318,7 @@ class TempUploadStore:
ctx=internal_ctx,
)
handle = get_lock_manager().create_handle()
- acquired = await get_lock_manager().acquire_subtree(handle, lock_path, timeout=0.0)
+ acquired = await get_lock_manager().acquire_tree(handle, lock_path, timeout=0.0)
if not acquired:
raise PermissionDeniedError("Temporary upload is being consumed.")
diff --git a/openviking/service/reindex_executor.py b/openviking/service/reindex_executor.py
index 334414f7b..6de627e76 100644
--- a/openviking/service/reindex_executor.py
+++ b/openviking/service/reindex_executor.py
@@ -311,7 +311,7 @@ class ReindexExecutor:
started_at = time.perf_counter()
counters = _ReindexCounters()
- async with LockContext(get_lock_manager(), [path], lock_mode="subtree"):
+ async with LockContext(get_lock_manager(), [path], lock_mode="tree"):
if object_type == "global_namespace":
await self._reindex_global_namespace(
uri=uri,
diff --git a/openviking/session/compressor_v2.py b/openviking/session/compressor_v2.py
index d34df31ae..ab473f88c 100644
--- a/openviking/session/compressor_v2.py
+++ b/openviking/session/compressor_v2.py
@@ -10,7 +10,7 @@ Maintains the same interface as compressor.py for backward compatibility.
import asyncio
import json
from datetime import datetime
-from typing import Any, Dict, List, Optional
+from typing import Any, Awaitable, Callable, Dict, List, Optional
from openviking.core.context import Context
from openviking.core.namespace import (
@@ -37,6 +37,59 @@ logger = get_logger(__name__)
MAX_SOURCE_TRAJECTORIES = 5 # keep only the most recent N trajectory URIs per experience
+ExtractPostApply = Callable[[MemoryUpdateResult, Dict[str, List[str]], Any], Awaitable[None]]
+
+
+def _filename_has_variables(schema: Any) -> bool:
+ checker = getattr(schema, "filename_has_variables", None)
+ if callable(checker):
+ return bool(checker())
+ filename_template = getattr(schema, "filename_template", "") or ""
+ return "{{" in filename_template and "}}" in filename_template
+
+
+def _append_unique(paths: list[str], path: str) -> None:
+ if path and path not in paths:
+ paths.append(path)
+
+
+def _render_memory_schema_locks(
+ *,
+ schemas: list[Any],
+ ctx: RequestContext,
+ viking_fs: VikingFS,
+ user_ids: list[str],
+ agent_ids: list[str],
+) -> tuple[list[str], list[str]]:
+ exact_paths: list[str] = []
+ tree_paths: list[str] = []
+ policy = ctx.namespace_policy
+ user_ids = user_ids or ["default"]
+ agent_ids = agent_ids or ["default"]
+
+ for schema in schemas:
+ directory_template = getattr(schema, "directory", "") or ""
+ if not directory_template:
+ continue
+
+ filename_template = getattr(schema, "filename_template", "") or ""
+ for user_id in user_ids:
+ for agent_id in agent_ids:
+ template_vars = {
+ "user_space": to_user_space(policy, user_id, agent_id),
+ "agent_space": to_agent_space(policy, user_id, agent_id),
+ }
+ directory_uri = render_template(directory_template, template_vars)
+ if _filename_has_variables(schema) or not filename_template:
+ _append_unique(tree_paths, viking_fs._uri_to_path(directory_uri, ctx))
+ continue
+
+ filename = render_template(filename_template, template_vars)
+ file_uri = f"{directory_uri.rstrip('/')}/{filename.lstrip('/')}"
+ _append_unique(exact_paths, viking_fs._uri_to_path(file_uri, ctx))
+
+ return exact_paths, tree_paths
+
class SessionCompressorV2:
"""Session memory extractor with v2 templating system."""
@@ -47,7 +100,6 @@ class SessionCompressorV2:
):
"""Initialize session compressor."""
self.vikingdb = vikingdb
- pass
def _get_or_create_react(
self,
@@ -183,37 +235,16 @@ class SessionCompressorV2:
)
read_scope = isolation_handler.get_read_scope()
if lock_manager:
- # 基于 provider 的 schemas 生成目录列表
- # 固定文件名 schema(如 soul.md、identity.md)只需 POINT 锁,
- # 避免因 SUBTREE 锁阻塞子目录(trajectories/experiences)的并发加锁。
schemas = orchestrator.context_provider.get_memory_schemas(ctx)
- point_lock_dirs: list = [] # 固定文件名 schema → POINT 锁
- subtree_lock_dirs: list = [] # 变量文件名 schema → SUBTREE 锁
- for schema in schemas:
- if not schema.directory:
- continue
- for user_id in read_scope.user_ids:
- for agent_id in read_scope.agent_ids:
- dir_path = render_template(
- schema.directory,
- {
- "user_space": to_user_space(
- ctx.namespace_policy, user_id, agent_id
- ),
- "agent_space": to_agent_space(
- ctx.namespace_policy, user_id, agent_id
- ),
- },
- )
- dir_path = viking_fs._uri_to_path(dir_path, ctx)
- if schema.filename_has_variables():
- if dir_path not in subtree_lock_dirs:
- subtree_lock_dirs.append(dir_path)
- else:
- if dir_path not in point_lock_dirs:
- point_lock_dirs.append(dir_path)
+ exact_lock_paths, tree_lock_dirs = _render_memory_schema_locks(
+ schemas=schemas,
+ ctx=ctx,
+ viking_fs=viking_fs,
+ user_ids=read_scope.user_ids,
+ agent_ids=read_scope.agent_ids,
+ )
logger.debug(
- f"Memory schema lock dirs: point={point_lock_dirs}, subtree={subtree_lock_dirs}"
+ f"Memory schema locks: exact={exact_lock_paths}, tree={tree_lock_dirs}"
)
retry_interval = config.memory.v2_lock_retry_interval_seconds
@@ -222,10 +253,10 @@ class SessionCompressorV2:
# 循环重试获取锁(机制确保不会死锁)
while True:
- lock_acquired = await lock_manager.acquire_mixed_batch(
+ lock_acquired = await lock_manager.acquire_exact_tree_batch(
transaction_handle,
- point_paths=point_lock_dirs,
- subtree_paths=subtree_lock_dirs,
+ exact_paths=exact_lock_paths,
+ tree_paths=tree_lock_dirs,
timeout=None,
)
if lock_acquired:
@@ -416,39 +447,49 @@ class SessionCompressorV2:
trajectory_summary=traj_content,
trajectory_uri=traj_uri,
)
+ exp_dir = exp_provider._render_experience_dir(ctx)
+
+ async def _append_sources_before_unlock(
+ result: MemoryUpdateResult,
+ inheritance_map: Dict[str, List[str]],
+ lock_handle: Any,
+ exp_provider=exp_provider,
+ exp_dir=exp_dir,
+ traj_uri=traj_uri,
+ ) -> None:
+ all_exp_uris = await self._resolve_source_target_experience_uris(
+ result=result,
+ provider=exp_provider,
+ exp_dir=exp_dir,
+ ctx=ctx,
+ viking_fs=viking_fs,
+ )
+ for exp_uri in all_exp_uris:
+ inherited = inheritance_map.get(exp_uri, [])
+ source_uris = list(dict.fromkeys([traj_uri] + inherited))
+ await self._append_trajectories_to_experiences(
+ [exp_uri],
+ source_uris,
+ ctx,
+ viking_fs,
+ lock_handle=lock_handle,
+ )
+
exp_result = await self._run_extract_phase(
provider=exp_provider,
messages=messages,
ctx=ctx,
strict_extract_errors=strict_extract_errors,
phase_label=f"experience({traj_uri})",
+ post_apply=_append_sources_before_unlock,
)
- exp_dir = exp_provider._render_experience_dir(ctx)
-
- async def _single_existing_experience_uri(exp_dir: str | None = exp_dir) -> List[str]:
- if not exp_dir:
- return []
- try:
- entries = await viking_fs.ls(exp_dir, output="original", ctx=ctx)
- except Exception:
- return []
- uris = []
- for e in entries or []:
- uri = str(e.get("uri", "")) if isinstance(e, dict) else ""
- name = str(e.get("name", "")) if isinstance(e, dict) else ""
- if not uri.endswith(".md"):
- continue
- if name in {".overview.md", ".abstract.md"}:
- continue
- if uri.endswith("/.overview.md") or uri.endswith("/.abstract.md"):
- continue
- uris.append(uri)
- uris = list(dict.fromkeys(uris))
- return uris if len(uris) == 1 else []
-
if exp_result is None:
- fallback_uris = await _single_existing_experience_uri()
+ fallback_uris = await self._single_existing_experience_uris(
+ exp_dir=exp_dir,
+ ctx=ctx,
+ viking_fs=viking_fs,
+ )
if fallback_uris:
tracer.info(
f"[source_traj] phase2 failed; fallback append to sole experience: {fallback_uris[0]}"
@@ -458,41 +499,75 @@ class SessionCompressorV2:
)
continue
- exp_written_uris, exp_edited_uris, exp_contexts, inheritance_map = exp_result
+ _, _, exp_contexts, _ = exp_result
contexts.extend(exp_contexts)
- all_exp_uris = exp_written_uris + exp_edited_uris
- if not all_exp_uris:
- candidate_uris = list(
- dict.fromkeys(getattr(exp_provider, "prefetched_uris", []) or [])
- )
- candidate_exp_uris = [
- uri
- for uri in candidate_uris
- if uri.endswith(".md")
- and not uri.endswith("/.overview.md")
- and not uri.endswith("/.abstract.md")
- and "/memories/experiences/" in uri
- ]
- if len(candidate_exp_uris) == 1:
- all_exp_uris = candidate_exp_uris
- tracer.info(
- f"[source_traj] fallback append to sole candidate experience: {candidate_exp_uris[0]}"
- )
- else:
- all_exp_uris = await _single_existing_experience_uri()
- if all_exp_uris:
- tracer.info(
- f"[source_traj] fallback append by directory scan: {all_exp_uris[0]}"
- )
+ return contexts
+
+ async def _resolve_source_target_experience_uris(
+ self,
+ *,
+ result: MemoryUpdateResult,
+ provider: Any,
+ exp_dir: str,
+ ctx: RequestContext,
+ viking_fs,
+ ) -> List[str]:
+ all_exp_uris = list(result.written_uris) + list(result.edited_uris)
+ if all_exp_uris:
+ return all_exp_uris
+
+ candidate_uris = list(dict.fromkeys(getattr(provider, "prefetched_uris", []) or []))
+ candidate_exp_uris = [
+ uri
+ for uri in candidate_uris
+ if uri.endswith(".md")
+ and not uri.endswith("/.overview.md")
+ and not uri.endswith("/.abstract.md")
+ and "/memories/experiences/" in uri
+ ]
+ if len(candidate_exp_uris) == 1:
+ tracer.info(
+ f"[source_traj] fallback append to sole candidate experience: {candidate_exp_uris[0]}"
+ )
+ return candidate_exp_uris
- for exp_uri in all_exp_uris:
- # Only the superseding experience inherits source_trajectories from the old file.
- uri_inherited = inheritance_map.get(exp_uri, [])
- traj_uris = list(dict.fromkeys([traj_uri] + uri_inherited))
- await self._append_trajectories_to_experiences([exp_uri], traj_uris, ctx, viking_fs)
+ existing = await self._single_existing_experience_uris(
+ exp_dir=exp_dir,
+ ctx=ctx,
+ viking_fs=viking_fs,
+ )
+ if existing:
+ tracer.info(f"[source_traj] fallback append by directory scan: {existing[0]}")
+ return existing
- return contexts
+ async def _single_existing_experience_uris(
+ self,
+ *,
+ exp_dir: str,
+ ctx: RequestContext,
+ viking_fs,
+ ) -> List[str]:
+ if not exp_dir:
+ return []
+ try:
+ entries = await viking_fs.ls(exp_dir, output="original", ctx=ctx)
+ except Exception:
+ return []
+
+ uris: List[str] = []
+ for entry in entries or []:
+ uri = str(entry.get("uri", "")) if isinstance(entry, dict) else ""
+ name = str(entry.get("name", "")) if isinstance(entry, dict) else ""
+ if not uri.endswith(".md"):
+ continue
+ if name in {".overview.md", ".abstract.md"}:
+ continue
+ if uri.endswith("/.overview.md") or uri.endswith("/.abstract.md"):
+ continue
+ uris.append(uri)
+ uris = list(dict.fromkeys(uris))
+ return uris if len(uris) == 1 else []
async def _run_extract_phase(
self,
@@ -501,6 +576,7 @@ class SessionCompressorV2:
ctx: RequestContext,
strict_extract_errors: bool,
phase_label: str,
+ post_apply: Optional[ExtractPostApply] = None,
):
"""Run one ExtractLoop phase with its own lock scope, then apply operations.
@@ -546,37 +622,24 @@ class SessionCompressorV2:
try:
if lock_manager:
schemas = provider.get_memory_schemas(ctx)
- memory_schema_dirs = []
- for schema in schemas:
- if not schema.directory:
- continue
-
- if ctx and ctx.user:
- user_space = to_user_space(
- ctx.namespace_policy, ctx.user.user_id, ctx.user.agent_id
- )
- agent_space = to_agent_space(
- ctx.namespace_policy, ctx.user.user_id, ctx.user.agent_id
- )
- else:
- user_space = "default"
- agent_space = "default"
- import jinja2
-
- env = jinja2.Environment(autoescape=False)
- template = env.from_string(schema.directory)
- dir_path = template.render(user_space=user_space, agent_space=agent_space)
- dir_path = viking_fs._uri_to_path(dir_path, ctx)
- if dir_path not in memory_schema_dirs:
- memory_schema_dirs.append(dir_path)
+ user_ids = [ctx.user.user_id] if ctx and ctx.user else ["default"]
+ agent_ids = [ctx.user.agent_id] if ctx and ctx.user else ["default"]
+ exact_lock_paths, tree_lock_dirs = _render_memory_schema_locks(
+ schemas=schemas,
+ ctx=ctx,
+ viking_fs=viking_fs,
+ user_ids=user_ids,
+ agent_ids=agent_ids,
+ )
retry_interval = config.memory.v2_lock_retry_interval_seconds
max_retries = config.memory.v2_lock_max_retries
retry_count = 0
while True:
- lock_acquired = await lock_manager.acquire_subtree_batch(
+ lock_acquired = await lock_manager.acquire_exact_tree_batch(
transaction_handle,
- memory_schema_dirs,
+ exact_paths=exact_lock_paths,
+ tree_paths=tree_lock_dirs,
timeout=None,
)
if lock_acquired:
@@ -628,6 +691,9 @@ class SessionCompressorV2:
f"errors={len(result.errors)}"
)
+ if post_apply:
+ await post_apply(result, inheritance_map, transaction_handle)
+
contexts: List[Context] = []
for uri in result.written_uris:
contexts.append(Context(uri=uri, category="memory_write", context_type="memory"))
@@ -733,60 +799,92 @@ class SessionCompressorV2:
traj_uris: List[str],
ctx,
viking_fs,
+ lock_handle: Optional[Any] = None,
) -> None:
"""Append traj_uris to the source_trajectories list of each experience file.
This is the system-side management of source_trajectories — the LLM never
outputs this field; the pipeline appends the batch after a write or edit.
"""
- from openviking.session.memory.utils.content import (
- deserialize_full,
- serialize_with_metadata,
- )
-
normalized_traj_uris = [uri for uri in traj_uris if uri]
if not normalized_traj_uris:
return
for exp_uri in exp_uris:
try:
- raw = await viking_fs.read_file(exp_uri, ctx=ctx) or ""
- file_content = deserialize_full(raw)
- plain_content = file_content.plain_content
- metadata = file_content.memory_fields or {}
-
- existing = metadata.get("source_trajectories", [])
- if isinstance(existing, list):
- uris = list(existing)
- elif isinstance(existing, str) and existing.strip():
- uris = [line.strip() for line in existing.splitlines() if line.strip()]
- else:
- uris = []
-
- changed = False
- for traj_uri in normalized_traj_uris:
- if traj_uri not in uris:
- uris.append(traj_uri)
- changed = True
-
- # Trim to the most recent N entries so the list doesn't grow unboundedly.
- if len(uris) > MAX_SOURCE_TRAJECTORIES:
- uris = uris[-MAX_SOURCE_TRAJECTORIES:]
- changed = True
-
- if changed:
- metadata["source_trajectories"] = uris
- metadata["content"] = plain_content
- new_raw = serialize_with_metadata(metadata)
- await viking_fs.write_file(exp_uri, new_raw, ctx=ctx)
- tracer.info(
- f"[source_traj] appended {len(normalized_traj_uris)} trajectories -> {exp_uri}"
+ try:
+ from openviking.storage.transaction import LockContext, get_lock_manager
+
+ lock_manager = get_lock_manager()
+ except Exception:
+ await self._append_trajectory_metadata(
+ exp_uri,
+ normalized_traj_uris,
+ ctx,
+ viking_fs,
+ )
+ continue
+
+ lock_path = viking_fs._uri_to_path(exp_uri, ctx=ctx)
+ async with LockContext(
+ lock_manager,
+ [lock_path],
+ lock_mode="exact",
+ handle=lock_handle,
+ ):
+ await self._append_trajectory_metadata(
+ exp_uri,
+ normalized_traj_uris,
+ ctx,
+ viking_fs,
)
- else:
- tracer.info(f"[source_traj] already present, skip: {exp_uri}")
except Exception as e:
logger.warning(f"Failed to append source trajectories to {exp_uri}: {e}")
+ async def _append_trajectory_metadata(
+ self,
+ exp_uri: str,
+ traj_uris: List[str],
+ ctx,
+ viking_fs,
+ ) -> None:
+ from openviking.session.memory.utils.content import (
+ deserialize_full,
+ serialize_with_metadata,
+ )
+
+ raw = await viking_fs.read_file(exp_uri, ctx=ctx) or ""
+ file_content = deserialize_full(raw)
+ plain_content = file_content.plain_content
+ metadata = file_content.memory_fields or {}
+
+ existing = metadata.get("source_trajectories", [])
+ if isinstance(existing, list):
+ uris = list(existing)
+ elif isinstance(existing, str) and existing.strip():
+ uris = [line.strip() for line in existing.splitlines() if line.strip()]
+ else:
+ uris = []
+
+ changed = False
+ for traj_uri in traj_uris:
+ if traj_uri not in uris:
+ uris.append(traj_uri)
+ changed = True
+
+ if len(uris) > MAX_SOURCE_TRAJECTORIES:
+ uris = uris[-MAX_SOURCE_TRAJECTORIES:]
+ changed = True
+
+ if changed:
+ metadata["source_trajectories"] = uris
+ metadata["content"] = plain_content
+ new_raw = serialize_with_metadata(metadata)
+ await viking_fs.write_file(exp_uri, new_raw, ctx=ctx)
+ tracer.info(f"[source_traj] appended {len(traj_uris)} trajectories -> {exp_uri}")
+ else:
+ tracer.info(f"[source_traj] already present, skip: {exp_uri}")
+
async def _build_memory_diff(
self,
result: MemoryUpdateResult,
diff --git a/openviking/session/memory/memory_updater.py b/openviking/session/memory/memory_updater.py
index 6f2004189..aff12bb50 100644
--- a/openviking/session/memory/memory_updater.py
+++ b/openviking/session/memory/memory_updater.py
@@ -462,7 +462,7 @@ class MemoryUpdater:
# Delete from VikingFS
# VikingFS automatically handles vector index cleanup
- # Pass transaction_handle so rm() reuses the compressor's subtree lock
+ # Pass transaction_handle so rm() reuses the compressor's tree lock
# instead of trying to acquire a new lock (which would conflict).
try:
await viking_fs.rm(uri, recursive=False, ctx=ctx, lock_handle=self._transaction_handle)
diff --git a/openviking/session/session.py b/openviking/session/session.py
index bbe69f694..3acf5a740 100644
--- a/openviking/session/session.py
+++ b/openviking/session/session.py
@@ -558,9 +558,9 @@ class Session:
async def commit_async(self, keep_recent_count: int = 0) -> Dict[str, Any]:
"""Async commit session: archive immediately, extract memories in background.
- Phase 1 (Archive prep, PathLock-protected): Split messages into
+ Phase 1 (Archive prep, path-lock protected): Split messages into
archive/retain parts, write the retained tail back to messages.jsonl,
- then persist the archive. Uses a distributed filesystem lock (PathLock)
+ then persist the archive. Uses a distributed filesystem lock
so this works across workers and processes.
Phase 2 (Memory extraction): Always runs in background via
asyncio.create_task().
@@ -585,7 +585,7 @@ class Session:
f"keep_recent_count={keep_recent_count}"
)
- # ===== Phase 1: Snapshot + clear (PathLock-protected) =====
+ # ===== Phase 1: Snapshot + clear (path-lock protected) =====
# Fast pre-check: skip lock entirely if no messages (common case avoids
# unnecessary filesystem lock acquisition).
if not self._messages:
@@ -612,7 +612,7 @@ class Session:
# Use filesystem-based distributed lock so this works across workers/processes.
session_path = self._viking_fs._uri_to_path(self._session_uri, ctx=self.ctx)
- async with LockContext(get_lock_manager(), [session_path], lock_mode="point"):
+ async with LockContext(get_lock_manager(), [session_path], lock_mode="exact"):
# Authoritative check under lock: handles the race where two concurrent
# callers both passed the pre-check but only the first should archive.
if not self._messages:
diff --git a/openviking/storage/content_write.py b/openviking/storage/content_write.py
index ac9c878c5..c4f93057b 100644
--- a/openviking/storage/content_write.py
+++ b/openviking/storage/content_write.py
@@ -12,6 +12,7 @@ from openviking.resource.watch_storage import is_watch_task_control_uri
from openviking.server.identity import RequestContext
from openviking.session.memory.utils.content import deserialize_full, serialize_with_metadata
from openviking.storage.queuefs import SemanticMsg, get_queue_manager
+from openviking.storage.queuefs.semantic_msg import build_semantic_coalesce_key
from openviking.storage.queuefs.semantic_processor import SemanticProcessor
from openviking.storage.transaction import get_lock_manager
from openviking.storage.viking_fs import VikingFS
@@ -42,6 +43,14 @@ class ContentWriteCoordinator:
def __init__(self, viking_fs: VikingFS):
self._viking_fs = viking_fs
+ @staticmethod
+ def _memory_content_parts(raw: str) -> tuple[str, Dict[str, Any]]:
+ try:
+ parsed = deserialize_full(raw)
+ except Exception:
+ return raw, {}
+ return parsed.plain_content, parsed.memory_fields or {}
+
async def write(
self,
*,
@@ -171,7 +180,7 @@ class ContentWriteCoordinator:
lock_manager = get_lock_manager()
handle = lock_manager.create_handle()
lock_path = self._viking_fs._uri_to_path(root_uri, ctx=ctx)
- acquired = await lock_manager.acquire_subtree(handle, lock_path)
+ acquired = await lock_manager.acquire_tree(handle, lock_path)
if not acquired:
await lock_manager.release(handle)
raise InvalidArgumentError(f"resource is busy and cannot be written now: {uri}")
@@ -345,9 +354,9 @@ class ContentWriteCoordinator:
) -> None:
if mode == "replace" and context_type_for_uri(uri) == "memory":
existing_raw = await self._viking_fs.read_file(uri, ctx=ctx)
- existing = deserialize_full(existing_raw)
- if existing.memory_fields:
- metadata_with_content = existing.memory_fields.copy()
+ _, metadata = self._memory_content_parts(existing_raw)
+ if metadata:
+ metadata_with_content = metadata.copy()
metadata_with_content["content"] = content
content = serialize_with_metadata(metadata_with_content)
await self._viking_fs.write_file(uri, content, ctx=ctx)
@@ -355,9 +364,7 @@ class ContentWriteCoordinator:
if mode == "append":
existing_raw = await self._viking_fs.read_file(uri, ctx=ctx)
- existing = deserialize_full(existing_raw)
- existing_content = existing.plain_content
- metadata = existing.memory_fields
+ existing_content, metadata = self._memory_content_parts(existing_raw)
updated_content = existing_content + content
if metadata:
metadata_with_content = metadata.copy()
@@ -394,6 +401,17 @@ class ContentWriteCoordinator:
skip_vectorization=False,
telemetry_id=telemetry.telemetry_id,
lifecycle_lock_handle_id=lifecycle_lock_handle_id,
+ coalesce_key=(
+ build_semantic_coalesce_key(
+ context_type=context_type,
+ uri=root_uri,
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ agent_id=ctx.user.agent_id,
+ )
+ if context_type in {"resource", "skill"}
+ else ""
+ ),
changes={change_type: [changed_uri]},
)
if msg.telemetry_id:
@@ -426,6 +444,13 @@ class ContentWriteCoordinator:
skip_vectorization=False,
telemetry_id=telemetry.telemetry_id,
lifecycle_lock_handle_id=lifecycle_lock_handle_id,
+ coalesce_key=build_semantic_coalesce_key(
+ context_type="memory",
+ uri=root_uri,
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ agent_id=ctx.user.agent_id,
+ ),
changes={"modified": [modified_uri]},
)
if msg.telemetry_id:
@@ -519,13 +544,13 @@ class ContentWriteCoordinator:
) -> Dict[str, Any]:
lock_manager = get_lock_manager()
handle = lock_manager.create_handle()
- lock_path = self._viking_fs._uri_to_path(root_uri, ctx=ctx)
- acquired = await lock_manager.acquire_subtree(handle, lock_path)
+ lock_path = self._viking_fs._uri_to_path(uri, ctx=ctx)
+ acquired = await lock_manager.acquire_exact_path(handle, lock_path)
if not acquired:
await lock_manager.release(handle)
raise InvalidArgumentError(f"resource is busy and cannot be written now: {uri}")
- lock_transferred = False
+ released = False
try:
if wait and telemetry_id:
get_request_wait_tracker().register_request(telemetry_id)
@@ -535,9 +560,10 @@ class ContentWriteCoordinator:
root_uri=root_uri,
modified_uri=uri,
ctx=ctx,
- lifecycle_lock_handle_id=handle.id,
+ lifecycle_lock_handle_id="",
)
- lock_transferred = True
+ await lock_manager.release(handle)
+ released = True
queue_status = (
await self._wait_for_request(telemetry_id=telemetry_id, timeout=timeout)
if wait
@@ -553,7 +579,7 @@ class ContentWriteCoordinator:
queue_status=queue_status,
)
except Exception:
- if not lock_transferred:
+ if not released:
await lock_manager.release(handle)
raise
finally:
diff --git a/openviking/storage/errors.py b/openviking/storage/errors.py
index 04c7e3e09..4a16aa76d 100644
--- a/openviking/storage/errors.py
+++ b/openviking/storage/errors.py
@@ -48,4 +48,17 @@ class LockAcquisitionError(LockError):
class ResourceBusyError(LockError):
- """Raised when a resource is locked by an ongoing operation (e.g. semantic processing)."""
+ """Raised when a resource is locked by an ongoing operation."""
+
+ def __init__(
+ self,
+ message: str,
+ *,
+ uri: str | None = None,
+ conflict_type: str = "path_busy",
+ retryable: bool = True,
+ ):
+ super().__init__(message)
+ self.uri = uri
+ self.conflict_type = conflict_type
+ self.retryable = retryable
diff --git a/openviking/storage/queuefs/semantic_dag.py b/openviking/storage/queuefs/semantic_dag.py
index ffba43850..a6d117c29 100644
--- a/openviking/storage/queuefs/semantic_dag.py
+++ b/openviking/storage/queuefs/semantic_dag.py
@@ -7,6 +7,7 @@ from dataclasses import dataclass, field
from typing import Awaitable, Callable, Dict, List, Optional
from openviking.server.identity import RequestContext
+from openviking.storage.queuefs.semantic_sidecar import write_semantic_sidecars
from openviking.storage.viking_fs import get_viking_fs
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
from openviking_cli.utils import VikingURI
@@ -82,6 +83,8 @@ class SemanticDagExecutor:
is_code_repo: bool = False,
changes: Optional[Dict[str, List[str]]] = None,
skip_vectorization: bool = False,
+ coalesce_key: str = "",
+ coalesce_version: int = 0,
):
self._processor = processor
self._context_type = context_type
@@ -96,6 +99,9 @@ class SemanticDagExecutor:
self._is_code_repo = is_code_repo
self._changes = changes or {}
self._skip_vectorization = skip_vectorization
+ self._coalesce_key = coalesce_key
+ self._coalesce_version = coalesce_version
+ self._stale = False
self._changed_paths = {
path for key in ("added", "modified", "deleted") for path in self._changes.get(key, [])
}
@@ -569,16 +575,49 @@ class SemanticDagExecutor:
summaries.append(item)
return summaries
- def _finalize_children_abstracts(self, node: DirNode) -> List[Dict[str, str]]:
+ @property
+ def stale(self) -> bool:
+ return self._stale
+
+ async def _finalize_children_abstracts(self, node: DirNode) -> List[Dict[str, str]]:
results: List[Dict[str, str]] = []
for idx, child_uri in enumerate(node.children_dirs):
item = node.children_abstracts[idx]
if item is None:
- results.append({"name": child_uri.split("/")[-1], "abstract": ""})
+ try:
+ abstract = await self._viking_fs.abstract(child_uri, ctx=self._ctx)
+ except Exception:
+ abstract = ""
+ results.append({"name": child_uri.split("/")[-1], "abstract": abstract})
else:
results.append(item)
return results
+ def _is_stale(self) -> bool:
+ from openviking.storage.queuefs.semantic_queue import is_semantic_coalesce_stale
+
+ return is_semantic_coalesce_stale(self._coalesce_key, self._coalesce_version)
+
+ async def _write_directory_semantics(
+ self,
+ dir_uri: str,
+ overview: str,
+ abstract: str,
+ ) -> bool:
+ wrote = await write_semantic_sidecars(
+ viking_fs=self._viking_fs,
+ dir_uri=dir_uri,
+ overview=overview,
+ abstract=abstract,
+ ctx=self._ctx,
+ is_stale=self._is_stale,
+ lifecycle_lock_handle_id=self._lifecycle_lock_handle_id,
+ log_prefix="[SemanticDag]",
+ )
+ if not wrote:
+ self._stale = True
+ return wrote
+
async def _overview_task(self, dir_uri: str) -> None:
node = self._nodes.get(dir_uri)
if not node:
@@ -600,7 +639,7 @@ class SemanticDagExecutor:
if overview is None or abstract is None:
async with node.lock:
file_summaries = self._finalize_file_summaries(node)
- children_abstracts = self._finalize_children_abstracts(node)
+ children_abstracts = await self._finalize_children_abstracts(node)
async with self._llm_sem:
overview = await self._processor._generate_overview(
dir_uri, file_summaries, children_abstracts
@@ -608,10 +647,11 @@ class SemanticDagExecutor:
abstract = self._processor._extract_abstract_from_overview(overview)
overview, abstract = self._processor._enforce_size_limits(overview, abstract)
- # Write directly — protected by the outer lifecycle SUBTREE lock
+ # Write directly — protected by the outer lifecycle tree lock
try:
- await self._viking_fs.write_file(f"{dir_uri}/.overview.md", overview, ctx=self._ctx)
- await self._viking_fs.write_file(f"{dir_uri}/.abstract.md", abstract, ctx=self._ctx)
+ wrote = await self._write_directory_semantics(dir_uri, overview, abstract)
+ if not wrote:
+ need_vectorize = False
except Exception:
logger.info(f"[SemanticDag] {dir_uri} write failed, skipping")
diff --git a/openviking/storage/queuefs/semantic_msg.py b/openviking/storage/queuefs/semantic_msg.py
index ba27d841a..bc46dc7d3 100644
--- a/openviking/storage/queuefs/semantic_msg.py
+++ b/openviking/storage/queuefs/semantic_msg.py
@@ -9,6 +9,17 @@ from typing import Any, Dict, List, Optional
from uuid import uuid4
+def build_semantic_coalesce_key(
+ *,
+ context_type: str,
+ uri: str,
+ account_id: str = "default",
+ user_id: str = "default",
+ agent_id: str = "default",
+) -> str:
+ return "|".join([context_type, account_id, user_id, agent_id, uri.rstrip("/")])
+
+
@dataclass
class SemanticMsg:
"""Semantic extraction queue message.
@@ -41,6 +52,8 @@ class SemanticMsg:
target_uri: str = ""
lifecycle_lock_handle_id: str = ""
is_code_repo: bool = False
+ coalesce_key: str = ""
+ coalesce_version: int = 0
changes: Optional[Dict[str, List[str]]] = (
None # {"added": [...], "modified": [...], "deleted": [...]}
)
@@ -59,6 +72,8 @@ class SemanticMsg:
target_uri: str = "",
lifecycle_lock_handle_id: str = "",
is_code_repo: bool = False,
+ coalesce_key: str = "",
+ coalesce_version: int = 0,
changes: Optional[Dict[str, List[str]]] = None,
):
self.id = str(uuid4())
@@ -74,6 +89,8 @@ class SemanticMsg:
self.target_uri = target_uri
self.lifecycle_lock_handle_id = lifecycle_lock_handle_id
self.is_code_repo = is_code_repo
+ self.coalesce_key = coalesce_key
+ self.coalesce_version = coalesce_version
self.changes = changes
def to_dict(self) -> Dict[str, Any]:
@@ -114,6 +131,8 @@ class SemanticMsg:
target_uri=data.get("target_uri", ""),
lifecycle_lock_handle_id=data.get("lifecycle_lock_handle_id", ""),
is_code_repo=data.get("is_code_repo", False),
+ coalesce_key=data.get("coalesce_key", ""),
+ coalesce_version=data.get("coalesce_version", 0),
changes=data.get("changes"),
)
if "id" in data and data["id"]:
diff --git a/openviking/storage/queuefs/semantic_processor.py b/openviking/storage/queuefs/semantic_processor.py
index c50afe6f5..7b8f70c43 100644
--- a/openviking/storage/queuefs/semantic_processor.py
+++ b/openviking/storage/queuefs/semantic_processor.py
@@ -29,7 +29,9 @@ from openviking.prompts import render_prompt
from openviking.server.identity import RequestContext, Role
from openviking.storage.queuefs.named_queue import DequeueHandlerBase
from openviking.storage.queuefs.semantic_dag import DagStats, SemanticDagExecutor
-from openviking.storage.queuefs.semantic_msg import SemanticMsg
+from openviking.storage.queuefs.semantic_msg import SemanticMsg, build_semantic_coalesce_key
+from openviking.storage.queuefs.semantic_queue import is_semantic_msg_stale
+from openviking.storage.queuefs.semantic_sidecar import write_semantic_sidecars
from openviking.storage.viking_fs import get_viking_fs
from openviking.telemetry import bind_telemetry, bind_telemetry_stage, resolve_telemetry
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
@@ -227,6 +229,47 @@ class SemanticProcessor(DequeueHandlerBase):
else:
logger.warning(f"No queue manager available, cannot re-enqueue: {msg.uri}")
+ async def _enqueue_parent_refresh(self, msg: SemanticMsg, uri: str) -> None:
+ if msg.context_type not in {"resource", "skill"}:
+ return
+ parent = VikingURI(uri).parent
+ if parent is None:
+ return
+ parent_uri = parent.uri.rstrip("/")
+ if (
+ not parent_uri
+ or parent_uri in {"viking://", "viking:"}
+ or parent_uri == uri.rstrip("/")
+ ):
+ return
+
+ from openviking.storage.queuefs import get_queue_manager
+
+ queue_manager = get_queue_manager()
+ if queue_manager is None:
+ return
+ semantic_queue = queue_manager.get_queue(queue_manager.SEMANTIC, allow_create=True)
+ parent_msg = SemanticMsg(
+ uri=parent_uri,
+ context_type=msg.context_type,
+ recursive=False,
+ account_id=msg.account_id,
+ user_id=msg.user_id,
+ agent_id=msg.agent_id,
+ role=msg.role,
+ skip_vectorization=msg.skip_vectorization,
+ changes={"modified": [uri]},
+ coalesce_key=build_semantic_coalesce_key(
+ context_type=msg.context_type,
+ uri=parent_uri,
+ account_id=msg.account_id,
+ user_id=msg.user_id,
+ agent_id=msg.agent_id,
+ ),
+ )
+ await semantic_queue.enqueue(parent_msg)
+ logger.info("Enqueued parent semantic refresh: %s", parent_uri)
+
async def on_dequeue(self, data: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any]]:
"""Process dequeued SemanticMsg, recursively process all subdirectories."""
msg: Optional[SemanticMsg] = None
@@ -243,6 +286,16 @@ class SemanticProcessor(DequeueHandlerBase):
assert data is not None
msg = SemanticMsg.from_dict(data)
+ if is_semantic_msg_stale(msg):
+ logger.info(
+ "Skipping stale semantic message: uri=%s version=%s",
+ msg.uri,
+ msg.coalesce_version,
+ )
+ if msg.telemetry_id and msg.id:
+ get_request_wait_tracker().mark_semantic_done(msg.telemetry_id, msg.id)
+ self.report_success()
+ return None
# Circuit breaker: if API is known-broken, re-enqueue and wait
try:
self._circuit_breaker.check()
@@ -326,6 +379,8 @@ class SemanticProcessor(DequeueHandlerBase):
is_code_repo=msg.is_code_repo,
changes=msg.changes,
skip_vectorization=msg.skip_vectorization,
+ coalesce_key=msg.coalesce_key,
+ coalesce_version=msg.coalesce_version,
)
self._dag_executor = executor
if msg.lifecycle_lock_handle_id:
@@ -337,6 +392,8 @@ class SemanticProcessor(DequeueHandlerBase):
msg.uri,
executor.get_stats(),
)
+ if not executor.stale:
+ await self._enqueue_parent_refresh(msg, target_uri or msg.uri)
self._merge_request_stats(msg.telemetry_id, processed=1)
logger.info(f"Completed semantic generation for: {msg.uri}")
self.report_success()
@@ -411,7 +468,7 @@ class SemanticProcessor(DequeueHandlerBase):
@staticmethod
async def _ensure_lifecycle_lock(handle_id: str, lock_path: str) -> str:
- """If the handle is missing (server restart), re-acquire a SUBTREE lock.
+ """If the handle is missing (server restart), re-acquire a TreeLock.
Returns the (possibly new) handle ID, or "" on failure.
"""
@@ -421,7 +478,7 @@ class SemanticProcessor(DequeueHandlerBase):
if lm.get_handle(handle_id):
return handle_id
new_handle = lm.create_handle()
- if await lm.acquire_subtree(new_handle, lock_path):
+ if await lm.acquire_tree(new_handle, lock_path):
logger.info(f"Re-acquired lifecycle lock on {lock_path} (handle {new_handle.id})")
return new_handle.id
logger.warning(f"Failed to re-acquire lifecycle lock on {lock_path}")
@@ -564,13 +621,24 @@ class SemanticProcessor(DequeueHandlerBase):
overview, abstract = self._enforce_size_limits(overview, abstract)
try:
- await viking_fs.write_file(f"{dir_uri}/.overview.md", overview, ctx=ctx)
- await viking_fs.write_file(f"{dir_uri}/.abstract.md", abstract, ctx=ctx)
- logger.info(f"Generated abstract.md and overview.md for {dir_uri}")
+ wrote_semantics = await self._write_memory_directory_semantics(
+ msg=msg,
+ viking_fs=viking_fs,
+ dir_uri=dir_uri,
+ overview=overview,
+ abstract=abstract,
+ ctx=ctx,
+ )
except Exception as e:
if msg.lifecycle_lock_handle_id:
await self._release_memory_lifecycle_lock(msg.lifecycle_lock_handle_id)
raise RuntimeError(f"Failed to write abstract/overview for {dir_uri}: {e}") from e
+ if not wrote_semantics:
+ _mark_done()
+ if msg.lifecycle_lock_handle_id:
+ await self._release_memory_lifecycle_lock(msg.lifecycle_lock_handle_id)
+ return
+ logger.info(f"Generated abstract.md and overview.md for {dir_uri}")
try:
if msg.skip_vectorization:
@@ -603,6 +671,27 @@ class SemanticProcessor(DequeueHandlerBase):
if msg.lifecycle_lock_handle_id:
await self._release_memory_lifecycle_lock(msg.lifecycle_lock_handle_id)
+ async def _write_memory_directory_semantics(
+ self,
+ *,
+ msg: SemanticMsg,
+ viking_fs: Any,
+ dir_uri: str,
+ overview: str,
+ abstract: str,
+ ctx: Optional[RequestContext],
+ ) -> bool:
+ return await write_semantic_sidecars(
+ viking_fs=viking_fs,
+ dir_uri=dir_uri,
+ overview=overview,
+ abstract=abstract,
+ ctx=ctx,
+ is_stale=lambda: is_semantic_msg_stale(msg),
+ lifecycle_lock_handle_id=msg.lifecycle_lock_handle_id,
+ log_prefix="[MemorySemantic]",
+ )
+
async def _release_memory_lifecycle_lock(self, handle_id: str) -> None:
"""Release a lifecycle lock held by in-place memory refresh."""
try:
diff --git a/openviking/storage/queuefs/semantic_queue.py b/openviking/storage/queuefs/semantic_queue.py
index a9df7eb12..9ab668eef 100644
--- a/openviking/storage/queuefs/semantic_queue.py
+++ b/openviking/storage/queuefs/semantic_queue.py
@@ -15,6 +15,19 @@ logger = get_logger(__name__)
# Coalesce rapid re-enqueues for the same memory parent directory (github #769).
_MEMORY_PARENT_SEMANTIC_DEDUPE_SEC = 45.0
+_SEMANTIC_COALESCE_LOCK = threading.Lock()
+_SEMANTIC_COALESCE_VERSION: dict[str, int] = {}
+
+
+def is_semantic_coalesce_stale(coalesce_key: str, coalesce_version: int) -> bool:
+ if not coalesce_key or coalesce_version <= 0:
+ return False
+ with _SEMANTIC_COALESCE_LOCK:
+ return coalesce_version < _SEMANTIC_COALESCE_VERSION.get(coalesce_key, 0)
+
+
+def is_semantic_msg_stale(msg: SemanticMsg) -> bool:
+ return is_semantic_coalesce_stale(msg.coalesce_key, msg.coalesce_version)
class SemanticQueue(NamedQueue):
@@ -31,7 +44,7 @@ class SemanticQueue(NamedQueue):
async def enqueue(self, msg: SemanticMsg) -> str:
"""Serialize SemanticMsg object and store in queue."""
- if msg.context_type == "memory":
+ if msg.context_type == "memory" and not msg.coalesce_key:
key = self._memory_parent_semantic_key(msg)
now = time.monotonic()
with self._memory_parent_semantic_lock:
@@ -51,6 +64,12 @@ class SemanticQueue(NamedQueue):
for k in stale[:800]:
self._memory_parent_semantic_last.pop(k, None)
+ if msg.coalesce_key:
+ with _SEMANTIC_COALESCE_LOCK:
+ version = _SEMANTIC_COALESCE_VERSION.get(msg.coalesce_key, 0) + 1
+ _SEMANTIC_COALESCE_VERSION[msg.coalesce_key] = version
+ msg.coalesce_version = version
+
return await super().enqueue(msg.to_dict())
async def dequeue(self) -> Optional[SemanticMsg]:
diff --git a/openviking/storage/queuefs/semantic_sidecar.py b/openviking/storage/queuefs/semantic_sidecar.py
new file mode 100644
index 000000000..58db8bb58
--- /dev/null
+++ b/openviking/storage/queuefs/semantic_sidecar.py
@@ -0,0 +1,68 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Shared writeback for semantic sidecar files."""
+
+from typing import Any, Callable, Optional
+
+from openviking.server.identity import RequestContext
+from openviking_cli.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+async def write_semantic_sidecars(
+ *,
+ viking_fs: Any,
+ dir_uri: str,
+ overview: str,
+ abstract: str,
+ ctx: Optional[RequestContext],
+ is_stale: Callable[[], bool],
+ lifecycle_lock_handle_id: str = "",
+ log_prefix: str = "[Semantic]",
+) -> bool:
+ if is_stale():
+ logger.info("%s Skipping stale semantic write for %s", log_prefix, dir_uri)
+ return False
+
+ try:
+ from openviking.storage.transaction import LockContext, get_lock_manager
+
+ lock_manager = get_lock_manager()
+ except Exception:
+ await _write_sidecars(viking_fs, dir_uri, overview, abstract, ctx)
+ return True
+
+ handle = None
+ owns_handle = False
+ if lifecycle_lock_handle_id:
+ handle = lock_manager.get_handle(lifecycle_lock_handle_id)
+ if handle is None:
+ handle = lock_manager.create_handle()
+ owns_handle = True
+
+ lock_paths = [
+ viking_fs._uri_to_path(f"{dir_uri}/.overview.md", ctx=ctx),
+ viking_fs._uri_to_path(f"{dir_uri}/.abstract.md", ctx=ctx),
+ ]
+ try:
+ async with LockContext(lock_manager, lock_paths, lock_mode="exact", handle=handle):
+ if is_stale():
+ logger.info("%s Skipping stale semantic write for %s", log_prefix, dir_uri)
+ return False
+ await _write_sidecars(viking_fs, dir_uri, overview, abstract, ctx)
+ return True
+ finally:
+ if owns_handle:
+ await lock_manager.release(handle)
+
+
+async def _write_sidecars(
+ viking_fs: Any,
+ dir_uri: str,
+ overview: str,
+ abstract: str,
+ ctx: Optional[RequestContext],
+) -> None:
+ await viking_fs.write_file(f"{dir_uri}/.overview.md", overview, ctx=ctx)
+ await viking_fs.write_file(f"{dir_uri}/.abstract.md", abstract, ctx=ctx)
diff --git a/openviking/storage/transaction/__init__.py b/openviking/storage/transaction/__init__.py
index 9b8739468..6d16e6e37 100644
--- a/openviking/storage/transaction/__init__.py
+++ b/openviking/storage/transaction/__init__.py
@@ -15,7 +15,7 @@ from openviking.storage.transaction.lock_manager import (
release_all_locks,
reset_lock_manager,
)
-from openviking.storage.transaction.path_lock import PathLock
+from openviking.storage.transaction.path_lock import PathLockEngine
from openviking.storage.transaction.redo_log import RedoLog
__all__ = [
@@ -23,7 +23,7 @@ __all__ = [
"LockHandle",
"LockManager",
"LockOwner",
- "PathLock",
+ "PathLockEngine",
"RedoLog",
"get_lock_manager",
"init_lock_manager",
diff --git a/openviking/storage/transaction/lock_context.py b/openviking/storage/transaction/lock_context.py
index 01eee15b0..ec4a561fa 100644
--- a/openviking/storage/transaction/lock_context.py
+++ b/openviking/storage/transaction/lock_context.py
@@ -20,15 +20,15 @@ class LockContext:
self,
lock_manager: LockManager,
paths: list[str],
- lock_mode: str = "point",
- mv_dst_parent_path: Optional[str] = None,
+ lock_mode: str = "exact",
+ mv_dst_path: Optional[str] = None,
src_is_dir: bool = True,
handle: Optional[LockHandle] = None,
):
self._manager = lock_manager
self._paths = paths
self._lock_mode = lock_mode
- self._mv_dst_parent_path = mv_dst_parent_path
+ self._mv_dst_path = mv_dst_path
self._src_is_dir = src_is_dir
self._handle: Optional[LockHandle] = handle
self._owns_handle = handle is None
@@ -41,25 +41,24 @@ class LockContext:
self._locks_before = list(self._handle.locks)
success = False
- if self._lock_mode == "subtree":
+ if self._lock_mode == "tree":
for path in self._paths:
- success = await self._manager.acquire_subtree(self._handle, path)
+ success = await self._manager.acquire_tree(self._handle, path)
if not success:
break
+ elif self._lock_mode == "exact":
+ success = await self._manager.acquire_exact_path_batch(self._handle, self._paths)
elif self._lock_mode == "mv":
- if self._mv_dst_parent_path is None:
- raise LockAcquisitionError("mv lock mode requires mv_dst_parent_path")
+ if self._mv_dst_path is None:
+ raise LockAcquisitionError("mv lock mode requires mv_dst_path")
success = await self._manager.acquire_mv(
self._handle,
self._paths[0],
- self._mv_dst_parent_path,
+ self._mv_dst_path,
src_is_dir=self._src_is_dir,
)
- else: # "point"
- for path in self._paths:
- success = await self._manager.acquire_point(self._handle, path)
- if not success:
- break
+ else:
+ raise LockAcquisitionError(f"Unsupported lock mode: {self._lock_mode}")
self._acquired_lock_paths = [
lock_path for lock_path in self._handle.locks if lock_path not in self._locks_before
diff --git a/openviking/storage/transaction/lock_handle.py b/openviking/storage/transaction/lock_handle.py
index 2fb405fff..0e2c8f084 100644
--- a/openviking/storage/transaction/lock_handle.py
+++ b/openviking/storage/transaction/lock_handle.py
@@ -1,6 +1,6 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
-"""Lock handle and LockOwner protocol for PathLock integration."""
+"""Lock handle and LockOwner protocol for path lock integration."""
import time
import uuid
@@ -14,7 +14,7 @@ def _new_lock_id() -> str:
@runtime_checkable
class LockOwner(Protocol):
- """Minimal interface that PathLock requires from its caller."""
+ """Minimal interface that path lock code requires from its caller."""
id: str
locks: list[str]
@@ -28,7 +28,7 @@ class LockOwner(Protocol):
@dataclass
class LockHandle:
- """Identifies a lock holder. PathLock uses ``id`` to generate fencing tokens
+ """Identifies a lock holder. Path lock code uses ``id`` to generate fencing tokens
and ``locks`` to track acquired lock files."""
id: str = field(default_factory=_new_lock_id)
diff --git a/openviking/storage/transaction/lock_manager.py b/openviking/storage/transaction/lock_manager.py
index 74538804e..f8338bab6 100644
--- a/openviking/storage/transaction/lock_manager.py
+++ b/openviking/storage/transaction/lock_manager.py
@@ -9,7 +9,7 @@ from typing import Any, Dict, List, Optional
from openviking.pyagfs import AGFSClient
from openviking.storage.transaction.lock_handle import LockHandle
-from openviking.storage.transaction.path_lock import PathLock
+from openviking.storage.transaction.path_lock import PathLockEngine
from openviking.storage.transaction.redo_log import RedoLog
from openviking_cli.utils.logger import get_logger
@@ -29,7 +29,7 @@ class LockManager:
redo_recovery_enabled: bool = True,
):
self._agfs = agfs
- self._path_lock = PathLock(agfs, lock_expire=lock_expire)
+ self._path_lock = PathLockEngine(agfs, lock_expire=lock_expire)
self._lock_timeout = lock_timeout
self._redo_recovery_enabled = redo_recovery_enabled
self._redo_log = RedoLog(agfs)
@@ -93,34 +93,34 @@ class LockManager:
self._handles[handle.id] = handle
return handle
- async def acquire_point(
+ async def acquire_exact_path(
self, handle: LockHandle, path: str, timeout: Optional[float] = None
) -> bool:
- acquired = await self._path_lock.acquire_point(
+ acquired = await self._path_lock.acquire_exact_path(
path, handle, timeout=timeout if timeout is not None else self._lock_timeout
)
if acquired:
self._mark_handle_active(handle)
return acquired
- async def acquire_subtree(
+ async def acquire_tree(
self, handle: LockHandle, path: str, timeout: Optional[float] = None
) -> bool:
- acquired = await self._path_lock.acquire_subtree(
+ acquired = await self._path_lock.acquire_tree(
path, handle, timeout=timeout if timeout is not None else self._lock_timeout
)
if acquired:
self._mark_handle_active(handle)
return acquired
- async def acquire_subtree_batch(
+ async def acquire_tree_batch(
self,
handle: LockHandle,
paths: List[str],
timeout: Optional[float] = None,
) -> bool:
"""
- 一次性对多个路径进行子树加锁,使用有序加锁法防止死锁
+ 一次性对多个路径进行树锁加锁,使用有序加锁法防止死锁
核心思想:
1. 对路径按照固定的顺序进行排序,确保所有进程获取锁的顺序一致
@@ -148,10 +148,10 @@ class LockManager:
try:
for path in sorted_paths:
locks_before = set(handle.locks)
- success = await self._path_lock.acquire_subtree(
+ success = await self._path_lock.acquire_tree(
path,
handle,
- timeout=timeout,
+ timeout=timeout if timeout is not None else self._lock_timeout,
)
if not success:
await self._path_lock.release_selected(handle, acquired_lock_paths)
@@ -163,21 +163,56 @@ class LockManager:
return True
except Exception as e:
- logger.error(f"Failed to acquire subtree batch lock: {e}")
+ logger.error(f"Failed to acquire tree batch lock: {e}")
await self._path_lock.release_selected(handle, acquired_lock_paths)
return False
- async def acquire_mixed_batch(
+ async def acquire_exact_path_batch(
self,
handle: LockHandle,
- point_paths: List[str],
- subtree_paths: List[str],
+ paths: List[str],
+ timeout: Optional[float] = None,
+ ) -> bool:
+ if not paths:
+ self._mark_handle_active(handle)
+ return True
+
+ sorted_paths = sorted(dict.fromkeys(paths), key=lambda x: (len(x), x))
+ acquired_lock_paths: List[str] = []
+
+ try:
+ for path in sorted_paths:
+ locks_before = set(handle.locks)
+ success = await self._path_lock.acquire_exact_path(
+ path,
+ handle,
+ timeout=timeout if timeout is not None else self._lock_timeout,
+ )
+ if not success:
+ await self._path_lock.release_selected(handle, acquired_lock_paths)
+ return False
+ newly = [lp for lp in handle.locks if lp not in locks_before]
+ acquired_lock_paths.extend(newly)
+
+ self._mark_handle_active(handle)
+ return True
+
+ except Exception as e:
+ logger.error(f"Failed to acquire exact-path batch lock: {e}")
+ await self._path_lock.release_selected(handle, acquired_lock_paths)
+ return False
+
+ async def acquire_exact_tree_batch(
+ self,
+ handle: LockHandle,
+ exact_paths: List[str],
+ tree_paths: List[str],
timeout: Optional[float] = None,
) -> bool:
- """"""
- subtree_set = set(subtree_paths)
- point_only = [p for p in point_paths if p not in subtree_set]
- all_pairs = [(p, False) for p in point_only] + [(p, True) for p in subtree_set]
+ exact_set = set(exact_paths)
+ tree_set = set(tree_paths)
+ exact_only = [p for p in exact_set if p not in tree_set]
+ all_pairs = [(p, False) for p in exact_only] + [(p, True) for p in tree_set]
if not all_pairs:
self._mark_handle_active(handle)
@@ -187,19 +222,19 @@ class LockManager:
acquired_lock_paths: List[str] = []
try:
- for path, is_subtree in sorted_pairs:
+ for path, is_tree in sorted_pairs:
locks_before = set(handle.locks)
- if is_subtree:
- success = await self._path_lock.acquire_subtree(
+ if is_tree:
+ success = await self._path_lock.acquire_tree(
path,
handle,
- timeout=timeout,
+ timeout=timeout if timeout is not None else self._lock_timeout,
)
else:
- success = await self._path_lock.acquire_point(
+ success = await self._path_lock.acquire_exact_path(
path,
handle,
- timeout=timeout,
+ timeout=timeout if timeout is not None else self._lock_timeout,
)
if not success:
await self._path_lock.release_selected(handle, acquired_lock_paths)
@@ -211,7 +246,7 @@ class LockManager:
return True
except Exception as e:
- logger.error(f"Failed to acquire mixed batch lock: {e}")
+ logger.error(f"Failed to acquire exact/tree batch lock: {e}")
await self._path_lock.release_selected(handle, acquired_lock_paths)
return False
@@ -219,13 +254,13 @@ class LockManager:
self,
handle: LockHandle,
src: str,
- dst_parent: str,
+ dst: str,
src_is_dir: bool = True,
timeout: Optional[float] = None,
) -> bool:
acquired = await self._path_lock.acquire_mv(
src,
- dst_parent,
+ dst,
handle,
timeout=timeout if timeout is not None else self._lock_timeout,
src_is_dir=src_is_dir,
diff --git a/openviking/storage/transaction/path_lock.py b/openviking/storage/transaction/path_lock.py
index 6058e06f7..e72277f3d 100644
--- a/openviking/storage/transaction/path_lock.py
+++ b/openviking/storage/transaction/path_lock.py
@@ -1,4 +1,6 @@
import asyncio
+import hashlib
+import re
import time
from dataclasses import dataclass, field
from typing import Optional, Tuple
@@ -11,10 +13,13 @@ logger = get_logger(__name__)
# Lock file name
LOCK_FILE_NAME = ".path.ovlock"
+EXACT_LOCK_FILE_PREFIX = ".exact.ovlock."
# Lock type constants
-LOCK_TYPE_POINT = "P"
-LOCK_TYPE_SUBTREE = "S"
+LOCK_TYPE_EXACT = "E"
+LOCK_TYPE_TREE = "T"
+# Upgrade compatibility: old POINT/SUBTREE tokens are still treated as tree locks.
+_READ_ONLY_TREE_LOCK_TYPES = {"P", "S"}
# Default poll interval when waiting for a lock (seconds)
_POLL_INTERVAL = 0.2
@@ -27,13 +32,14 @@ class LockRefreshResult:
failed_paths: list[str] = field(default_factory=list)
-def _make_fencing_token(owner_id: str, lock_type: str = LOCK_TYPE_POINT) -> str:
+def _make_fencing_token(owner_id: str, lock_type: str = LOCK_TYPE_EXACT) -> str:
return f"{owner_id}:{time.time_ns()}:{lock_type}"
def _parse_fencing_token(token: str) -> Tuple[str, int, str]:
- if token.endswith(f":{LOCK_TYPE_POINT}") or token.endswith(f":{LOCK_TYPE_SUBTREE}"):
- lock_type = token[-1]
+ known_types = {LOCK_TYPE_EXACT, LOCK_TYPE_TREE, *_READ_ONLY_TREE_LOCK_TYPES}
+ if len(token) >= 2 and token[-2] == ":" and token[-1] in known_types:
+ lock_type = LOCK_TYPE_TREE if token[-1] in _READ_ONLY_TREE_LOCK_TYPES else token[-1]
rest = token[:-2]
idx = rest.rfind(":")
if idx >= 0:
@@ -45,27 +51,54 @@ def _parse_fencing_token(token: str) -> Tuple[str, int, str]:
pass
return rest, 0, lock_type
- if ":" in token:
- idx = token.rfind(":")
- owner_id_part = token[:idx]
- ts_part = token[idx + 1 :]
- try:
- return owner_id_part, int(ts_part), LOCK_TYPE_POINT
- except ValueError:
- pass
-
- return token, 0, LOCK_TYPE_POINT
+ return token, 0, LOCK_TYPE_EXACT
-class PathLock:
+class PathLockEngine:
def __init__(self, agfs_client: AGFSClient, lock_expire: float = 300.0):
self._agfs = agfs_client
self._lock_expire = lock_expire
def _get_lock_path(self, path: str) -> str:
- path = path.rstrip("/")
+ path = path.rstrip("/") or "/"
+ if path == "/":
+ return f"/{LOCK_FILE_NAME}"
return f"{path}/{LOCK_FILE_NAME}"
+ def _is_existing_directory(self, path: str) -> bool:
+ try:
+ stat = self._agfs.stat(path.rstrip("/") or "/")
+ except Exception:
+ return False
+ if isinstance(stat, dict):
+ return stat.get("isDir") is True
+ return getattr(stat, "isDir", None) is True
+
+ def _get_prefixed_exact_lock_path(self, path: str) -> str:
+ path = path.rstrip("/") or "/"
+ parent = self._get_parent_path(path)
+ if not parent:
+ return self._get_lock_path(path)
+ name = path.rsplit("/", 1)[-1]
+ safe_name = re.sub(r"[^A-Za-z0-9._-]+", "_", name).strip("._") or "path"
+ safe_name = safe_name[:80]
+ digest = hashlib.sha1(path.encode("utf-8")).hexdigest()[:16]
+ return f"{parent}/{EXACT_LOCK_FILE_PREFIX}{safe_name}.{digest}"
+
+ def _get_exact_lock_path(self, path: str) -> str:
+ """Return the primary lock-file path for an exact path lock."""
+ if self._is_existing_directory(path):
+ return self._get_lock_path(path)
+ return self._get_prefixed_exact_lock_path(path)
+
+ def _get_exact_lock_paths(self, path: str) -> list[str]:
+ """Return all lock-file paths that can protect this exact path."""
+ primary = self._get_exact_lock_path(path)
+ prefixed = self._get_prefixed_exact_lock_path(path)
+ if primary == prefixed:
+ return [primary]
+ return [primary, prefixed]
+
def _ensure_directory_exists(self, path: str):
"""确保目录存在,不存在则创建"""
try:
@@ -120,7 +153,8 @@ class PathLock:
Detection rules (aligned with conflict checks in the acquire flow):
- The path itself has a valid .path.ovlock; or
- - Any ancestor directory holds a SUBTREE lock.
+ - The path has a valid exact-path lock in the parent directory; or
+ - Any ancestor directory holds a TREE lock.
Args:
path: Path to check (already converted to AGFS internal path).
@@ -136,14 +170,23 @@ class PathLock:
if not (ignore_stale and self.is_lock_stale(own_lock_path, self._lock_expire)):
return True
- # 2. Ancestor SUBTREE locks
+ # 2. ExactPathLock for file, directory name, or not-yet-created paths
+ for exact_lock_path in self._get_exact_lock_paths(path):
+ if exact_lock_path == own_lock_path:
+ continue
+ exact_token = self._read_token(exact_lock_path)
+ if exact_token is not None:
+ if not (ignore_stale and self.is_lock_stale(exact_lock_path, self._lock_expire)):
+ return True
+
+ # 3. Ancestor TREE locks
parent = self._get_parent_path(path)
while parent:
ancestor_lock = self._get_lock_path(parent)
ancestor_token = self._read_token(ancestor_lock)
if ancestor_token is not None:
_, _, lock_type = _parse_fencing_token(ancestor_token)
- if lock_type == LOCK_TYPE_SUBTREE and not (
+ if lock_type == LOCK_TYPE_TREE and not (
ignore_stale and self.is_lock_stale(ancestor_lock, self._lock_expire)
):
return True
@@ -166,7 +209,7 @@ class PathLock:
return lock_owner != owner_id
async def _create_lock_file(
- self, lock_path: str, owner_id: str, lock_type: str = LOCK_TYPE_POINT
+ self, lock_path: str, owner_id: str, lock_type: str = LOCK_TYPE_EXACT
) -> None:
token = _make_fencing_token(owner_id, lock_type)
self._agfs.write(lock_path, token.encode("utf-8"))
@@ -183,10 +226,10 @@ class PathLock:
return None
return lock_type
- async def _has_owned_ancestor_subtree(self, path: str, owner: LockOwner) -> bool:
+ async def _has_owned_ancestor_tree(self, path: str, owner: LockOwner) -> bool:
current = path.rstrip("/")
while current:
- if await self._owned_lock_type(current, owner) == LOCK_TYPE_SUBTREE:
+ if await self._owned_lock_type(current, owner) == LOCK_TYPE_TREE:
return True
current = self._get_parent_path(current) or ""
return False
@@ -210,20 +253,44 @@ class PathLock:
age = (time.time_ns() - ts) / 1e9
return age > expire_seconds
- async def _check_ancestors_for_subtree(self, path: str, exclude_owner_id: str) -> Optional[str]:
+ async def _check_ancestors_for_tree(self, path: str, exclude_owner_id: str) -> Optional[str]:
parent = self._get_parent_path(path)
while parent:
lock_path = self._get_lock_path(parent)
token = self._read_token(lock_path)
if token is not None:
owner_id, _, lock_type = _parse_fencing_token(token)
- if owner_id != exclude_owner_id and lock_type == LOCK_TYPE_SUBTREE:
+ if owner_id != exclude_owner_id and lock_type == LOCK_TYPE_TREE:
return lock_path
parent = self._get_parent_path(parent)
return None
+ async def _check_path_lock(self, path: str, exclude_owner_id: str) -> Optional[str]:
+ lock_path = self._get_lock_path(path)
+ token = self._read_token(lock_path)
+ if token is None:
+ return None
+ owner_id, _, _ = _parse_fencing_token(token)
+ if owner_id != exclude_owner_id:
+ return lock_path
+ return None
+
+ async def _check_exact_path_lock(self, path: str, exclude_owner_id: str) -> Optional[str]:
+ for lock_path in self._get_exact_lock_paths(path):
+ token = self._read_token(lock_path)
+ if token is None:
+ continue
+ owner_id, _, _ = _parse_fencing_token(token)
+ if owner_id != exclude_owner_id:
+ return lock_path
+ return None
+
async def _scan_descendants_for_locks(self, path: str, exclude_owner_id: str) -> Optional[str]:
try:
+ try:
+ self._agfs.stat(path)
+ except Exception:
+ return None
entries = self._agfs.ls(path)
if not isinstance(entries, list):
return None
@@ -233,9 +300,17 @@ class PathLock:
name = entry.get("name", "")
if not name or name in (".", ".."):
continue
+ entry_path = f"{path.rstrip('/')}/{name}"
+ if name.startswith(EXACT_LOCK_FILE_PREFIX):
+ token = self._read_token(entry_path)
+ if token is not None:
+ owner_id, _, _ = _parse_fencing_token(token)
+ if owner_id != exclude_owner_id:
+ return entry_path
+ continue
if not entry.get("isDir", False):
continue
- subdir = f"{path.rstrip('/')}/{name}"
+ subdir = entry_path
subdir_lock = self._get_lock_path(subdir)
token = self._read_token(subdir_lock)
if token is not None:
@@ -249,109 +324,146 @@ class PathLock:
logger.warning(f"Failed to scan descendants of {path}: {e}")
return None
- async def acquire_point(
+ async def acquire_exact_path(
self, path: str, owner: LockOwner, timeout: Optional[float] = 0.0
) -> bool:
+ """Acquire a short lock for one exact path.
+
+ It conflicts with:
+ - another ExactPathLock on the same path;
+ - any ancestor TREE lock;
+ - a TREE lock on the exact same path.
+
+ It does not conflict with sibling exact paths.
+ """
owner_id = owner.id
- lock_path = self._get_lock_path(path)
- owned_lock_type = await self._owned_lock_type(path, owner)
- if owned_lock_type in {LOCK_TYPE_POINT, LOCK_TYPE_SUBTREE}:
+ lock_path = self._get_exact_lock_path(path)
+ if lock_path in owner.locks and self.is_lock_owned_by(lock_path, owner_id):
owner.add_lock(lock_path)
- logger.debug(f"[POINT] Reusing owned lock on: {path}")
+ logger.debug(f"[EXACT] Reusing owned exact lock on: {path}")
return True
- if await self._has_owned_ancestor_subtree(path, owner):
- logger.debug(f"[POINT] Reusing owned ancestor SUBTREE lock on: {path}")
+ if await self._has_owned_ancestor_tree(path, owner):
+ logger.debug(f"[EXACT] Reusing owned ancestor TREE lock on: {path}")
return True
if timeout is None:
- # 无限等待
deadline = float("inf")
else:
- # 有限超时
deadline = asyncio.get_running_loop().time() + timeout
- # 确保目录存在
- if not self._ensure_directory_exists(path):
- logger.warning(f"[POINT] Failed to ensure directory exists: {path}")
- return False
-
while True:
- if await self._is_locked_by_other(lock_path, owner_id):
- if self.is_lock_stale(lock_path, self._lock_expire):
- logger.warning(f"[POINT] Removing stale lock: {lock_path}")
- await self._remove_lock_file(lock_path)
+ existing_exact_lock = await self._check_exact_path_lock(path, owner_id)
+ if existing_exact_lock:
+ if self.is_lock_stale(existing_exact_lock, self._lock_expire):
+ logger.warning(f"[EXACT] Removing stale exact lock: {existing_exact_lock}")
+ await self._remove_lock_file(existing_exact_lock)
continue
if asyncio.get_running_loop().time() >= deadline:
- logger.warning(f"[POINT] Timeout waiting for lock on: {path}")
+ logger.warning(f"[EXACT] Timeout waiting for exact lock on: {path}")
return False
await asyncio.sleep(_POLL_INTERVAL)
continue
- ancestor_conflict = await self._check_ancestors_for_subtree(path, owner_id)
+ same_path_lock = self._get_lock_path(path)
+ if same_path_lock != lock_path:
+ token = self._read_token(same_path_lock)
+ if token is not None:
+ lock_owner, _, _ = _parse_fencing_token(token)
+ if lock_owner != owner_id:
+ if self.is_lock_stale(same_path_lock, self._lock_expire):
+ logger.warning(f"[EXACT] Removing stale lock: {same_path_lock}")
+ await self._remove_lock_file(same_path_lock)
+ continue
+ if asyncio.get_running_loop().time() >= deadline:
+ logger.warning(f"[EXACT] Timeout waiting for lock: {path}")
+ return False
+ await asyncio.sleep(_POLL_INTERVAL)
+ continue
+
+ ancestor_conflict = await self._check_ancestors_for_tree(path, owner_id)
if ancestor_conflict:
if self.is_lock_stale(ancestor_conflict, self._lock_expire):
logger.warning(
- f"[POINT] Removing stale ancestor SUBTREE lock: {ancestor_conflict}"
+ f"[EXACT] Removing stale ancestor TREE lock: {ancestor_conflict}"
)
await self._remove_lock_file(ancestor_conflict)
continue
if asyncio.get_running_loop().time() >= deadline:
logger.warning(
- f"[POINT] Timeout waiting for ancestor SUBTREE lock: {ancestor_conflict}"
+ f"[EXACT] Timeout waiting for ancestor TREE lock: {ancestor_conflict}"
)
return False
await asyncio.sleep(_POLL_INTERVAL)
continue
+ parent = self._get_parent_path(path)
+ if (
+ lock_path != self._get_lock_path(path)
+ and parent
+ and not self._ensure_directory_exists(parent)
+ ):
+ logger.warning(f"[EXACT] Failed to ensure parent directory exists: {parent}")
+ return False
+
try:
- await self._create_lock_file(lock_path, owner_id, LOCK_TYPE_POINT)
+ await self._create_lock_file(lock_path, owner_id, LOCK_TYPE_EXACT)
except Exception as e:
- logger.error(f"[POINT] Failed to create lock file: {e}")
+ logger.error(f"[EXACT] Failed to create lock file: {e}")
return False
- backed_off = False
- conflict_after = await self._check_ancestors_for_subtree(path, owner_id)
+ if not self.is_lock_owned_by(lock_path, owner_id):
+ logger.debug(f"[EXACT] Lost lock write race on: {path}")
+ if asyncio.get_running_loop().time() >= deadline:
+ return False
+ await asyncio.sleep(_POLL_INTERVAL)
+ continue
+
+ conflict_after = await self._check_path_lock(path, owner_id)
+ if conflict_after == lock_path:
+ conflict_after = None
+ if not conflict_after:
+ conflict_after = await self._check_ancestors_for_tree(path, owner_id)
if conflict_after:
their_token = self._read_token(conflict_after)
if their_token:
their_owner_id, their_ts, _ = _parse_fencing_token(their_token)
my_token = self._read_token(lock_path)
_, my_ts, _ = (
- _parse_fencing_token(my_token) if my_token else ("", 0, LOCK_TYPE_POINT)
+ _parse_fencing_token(my_token) if my_token else ("", 0, LOCK_TYPE_EXACT)
)
if (my_ts, owner_id) > (their_ts, their_owner_id):
- logger.debug(f"[POINT] Backing off (livelock guard) on {path}")
- await self._remove_lock_file(lock_path)
- backed_off = True
+ logger.debug(f"[EXACT] Backing off (livelock guard) on {path}")
+ if self.is_lock_owned_by(lock_path, owner_id):
+ await self._remove_lock_file(lock_path)
if asyncio.get_running_loop().time() >= deadline:
- if not backed_off:
+ if self.is_lock_owned_by(lock_path, owner_id):
await self._remove_lock_file(lock_path)
return False
await asyncio.sleep(_POLL_INTERVAL)
continue
if not self.is_lock_owned_by(lock_path, owner_id):
- logger.debug(f"[POINT] Lock ownership verification failed: {path}")
+ logger.debug(f"[EXACT] Lock ownership verification failed: {path}")
if asyncio.get_running_loop().time() >= deadline:
return False
await asyncio.sleep(_POLL_INTERVAL)
continue
owner.add_lock(lock_path)
- logger.debug(f"[POINT] Lock acquired: {lock_path}")
+ logger.debug(f"[EXACT] Lock acquired: {lock_path}")
return True
- async def acquire_subtree(
+ async def acquire_tree(
self, path: str, owner: LockOwner, timeout: Optional[float] = 0.0
) -> bool:
owner_id = owner.id
lock_path = self._get_lock_path(path)
owned_lock_type = await self._owned_lock_type(path, owner)
- if owned_lock_type == LOCK_TYPE_SUBTREE:
+ if owned_lock_type == LOCK_TYPE_TREE:
owner.add_lock(lock_path)
- logger.debug(f"[SUBTREE] Reusing owned SUBTREE lock on: {path}")
+ logger.debug(f"[TREE] Reusing owned TREE lock on: {path}")
return True
- if await self._has_owned_ancestor_subtree(path, owner):
- logger.debug(f"[SUBTREE] Reusing owned ancestor SUBTREE lock on: {path}")
+ if await self._has_owned_ancestor_tree(path, owner):
+ logger.debug(f"[TREE] Reusing owned ancestor TREE lock on: {path}")
return True
if timeout is None:
# 无限等待
@@ -360,74 +472,83 @@ class PathLock:
# 有限超时
deadline = asyncio.get_running_loop().time() + timeout
- # 确保目录存在
- if not self._ensure_directory_exists(path):
- logger.warning(f"[SUBTREE] Failed to ensure directory exists: {path}")
- return False
-
while True:
if await self._is_locked_by_other(lock_path, owner_id):
if self.is_lock_stale(lock_path, self._lock_expire):
- logger.warning(f"[SUBTREE] Removing stale lock: {lock_path}")
+ logger.warning(f"[TREE] Removing stale lock: {lock_path}")
await self._remove_lock_file(lock_path)
continue
if asyncio.get_running_loop().time() >= deadline:
- logger.warning(f"[SUBTREE] Timeout waiting for lock on: {path}")
+ logger.warning(f"[TREE] Timeout waiting for lock on: {path}")
return False
await asyncio.sleep(_POLL_INTERVAL)
continue
- # Check ancestor paths for SUBTREE locks held by other owners
- ancestor_conflict = await self._check_ancestors_for_subtree(path, owner_id)
+ # Check ancestor paths for TREE locks held by other owners
+ ancestor_conflict = await self._check_ancestors_for_tree(path, owner_id)
if ancestor_conflict:
if self.is_lock_stale(ancestor_conflict, self._lock_expire):
- logger.warning(
- f"[SUBTREE] Removing stale ancestor SUBTREE lock: {ancestor_conflict}"
- )
+ logger.warning(f"[TREE] Removing stale ancestor TREE lock: {ancestor_conflict}")
await self._remove_lock_file(ancestor_conflict)
continue
if asyncio.get_running_loop().time() >= deadline:
logger.warning(
- f"[SUBTREE] Timeout waiting for ancestor SUBTREE lock: {ancestor_conflict}"
+ f"[TREE] Timeout waiting for ancestor TREE lock: {ancestor_conflict}"
)
return False
await asyncio.sleep(_POLL_INTERVAL)
continue
+ exact_conflict = await self._check_exact_path_lock(path, owner_id)
+ if exact_conflict:
+ if self.is_lock_stale(exact_conflict, self._lock_expire):
+ logger.warning(f"[TREE] Removing stale exact lock: {exact_conflict}")
+ await self._remove_lock_file(exact_conflict)
+ continue
+ if asyncio.get_running_loop().time() >= deadline:
+ logger.warning(f"[TREE] Timeout waiting for exact lock: {exact_conflict}")
+ return False
+ await asyncio.sleep(_POLL_INTERVAL)
+ continue
+
desc_conflict = await self._scan_descendants_for_locks(path, owner_id)
if desc_conflict:
if self.is_lock_stale(desc_conflict, self._lock_expire):
- logger.warning(f"[SUBTREE] Removing stale descendant lock: {desc_conflict}")
+ logger.warning(f"[TREE] Removing stale descendant lock: {desc_conflict}")
await self._remove_lock_file(desc_conflict)
continue
if asyncio.get_running_loop().time() >= deadline:
- logger.warning(
- f"[SUBTREE] Timeout waiting for descendant lock: {desc_conflict}"
- )
+ logger.warning(f"[TREE] Timeout waiting for descendant lock: {desc_conflict}")
return False
await asyncio.sleep(_POLL_INTERVAL)
continue
+ if not self._ensure_directory_exists(path):
+ logger.warning(f"[TREE] Failed to ensure directory exists: {path}")
+ return False
+
try:
- await self._create_lock_file(lock_path, owner_id, LOCK_TYPE_SUBTREE)
+ await self._create_lock_file(lock_path, owner_id, LOCK_TYPE_TREE)
except Exception as e:
- logger.error(f"[SUBTREE] Failed to create lock file: {e}")
+ logger.error(f"[TREE] Failed to create lock file: {e}")
return False
backed_off = False
conflict_after = await self._scan_descendants_for_locks(path, owner_id)
if not conflict_after:
- conflict_after = await self._check_ancestors_for_subtree(path, owner_id)
+ conflict_after = await self._check_exact_path_lock(path, owner_id)
+ if not conflict_after:
+ conflict_after = await self._check_ancestors_for_tree(path, owner_id)
if conflict_after:
their_token = self._read_token(conflict_after)
if their_token:
their_owner_id, their_ts, _ = _parse_fencing_token(their_token)
my_token = self._read_token(lock_path)
_, my_ts, _ = (
- _parse_fencing_token(my_token) if my_token else ("", 0, LOCK_TYPE_SUBTREE)
+ _parse_fencing_token(my_token) if my_token else ("", 0, LOCK_TYPE_TREE)
)
if (my_ts, owner_id) > (their_ts, their_owner_id):
- logger.debug(f"[SUBTREE] Backing off (livelock guard) on {path}")
+ logger.debug(f"[TREE] Backing off (livelock guard) on {path}")
await self._remove_lock_file(lock_path)
backed_off = True
if asyncio.get_running_loop().time() >= deadline:
@@ -438,20 +559,20 @@ class PathLock:
continue
if not self.is_lock_owned_by(lock_path, owner_id):
- logger.debug(f"[SUBTREE] Lock ownership verification failed: {path}")
+ logger.debug(f"[TREE] Lock ownership verification failed: {path}")
if asyncio.get_running_loop().time() >= deadline:
return False
await asyncio.sleep(_POLL_INTERVAL)
continue
owner.add_lock(lock_path)
- logger.debug(f"[SUBTREE] Lock acquired: {lock_path}")
+ logger.debug(f"[TREE] Lock acquired: {lock_path}")
return True
async def acquire_mv(
self,
src_path: str,
- dst_parent_path: str,
+ dst_path: str,
owner: LockOwner,
timeout: Optional[float] = 0.0,
src_is_dir: bool = True,
@@ -460,38 +581,30 @@ class PathLock:
Args:
src_path: Source path to lock.
- dst_parent_path: Parent directory of the destination to lock.
- Callers typically pass the destination's parent so that the
- lock covers sibling-level conflicts without requiring the
- target to exist yet.
+ dst_path: Destination path to lock.
owner: Lock owner handle.
timeout: Maximum seconds to wait for each lock.
- src_is_dir: Whether the source is a directory (SUBTREE lock)
- or a file (POINT lock on parent).
+ src_is_dir: Whether the source is a directory (TREE lock)
+ or a file (ExactPathLock on source and destination path).
"""
if src_is_dir:
- if not await self.acquire_subtree(src_path, owner, timeout=timeout):
- logger.warning(f"[MV] Failed to acquire SUBTREE lock on source: {src_path}")
+ if not await self.acquire_tree(src_path, owner, timeout=timeout):
+ logger.warning(f"[MV] Failed to acquire TREE lock on source: {src_path}")
return False
- if not await self.acquire_subtree(dst_parent_path, owner, timeout=timeout):
- logger.warning(
- f"[MV] Failed to acquire SUBTREE lock on destination parent: {dst_parent_path}"
- )
+ if not await self.acquire_exact_path(dst_path, owner, timeout=timeout):
+ logger.warning(f"[MV] Failed to acquire exact lock on destination: {dst_path}")
await self.release(owner)
return False
else:
- src_parent = src_path.rsplit("/", 1)[0] if "/" in src_path else src_path
- if not await self.acquire_point(src_parent, owner, timeout=timeout):
- logger.warning(f"[MV] Failed to acquire POINT lock on source parent: {src_parent}")
+ if not await self.acquire_exact_path(src_path, owner, timeout=timeout):
+ logger.warning(f"[MV] Failed to acquire exact lock on source: {src_path}")
return False
- if not await self.acquire_point(dst_parent_path, owner, timeout=timeout):
- logger.warning(
- f"[MV] Failed to acquire POINT lock on destination parent: {dst_parent_path}"
- )
+ if not await self.acquire_exact_path(dst_path, owner, timeout=timeout):
+ logger.warning(f"[MV] Failed to acquire exact lock on destination: {dst_path}")
await self.release(owner)
return False
- logger.debug(f"[MV] Locks acquired: {src_path} -> {dst_parent_path}")
+ logger.debug(f"[MV] Locks acquired: {src_path} -> {dst_path}")
return True
async def refresh(self, owner: LockOwner) -> LockRefreshResult:
diff --git a/openviking/storage/viking_fs.py b/openviking/storage/viking_fs.py
index 36f31acbe..71eabfaa4 100644
--- a/openviking/storage/viking_fs.py
+++ b/openviking/storage/viking_fs.py
@@ -491,11 +491,10 @@ class VikingFS:
details={"resource": uri, "expected_flag": "recursive"},
)
lock_paths = [path]
- lock_mode = "subtree"
+ lock_mode = "tree"
else:
- parent = path.rsplit("/", 1)[0] if "/" in path else path
- lock_paths = [parent]
- lock_mode = "point"
+ lock_paths = [path]
+ lock_mode = "exact"
try:
async with LockContext(
@@ -510,9 +509,7 @@ class VikingFS:
estimated_count = await _estimate_deleted_count(path, real_ctx)
await self._delete_from_vector_store(uris_to_delete, ctx=ctx)
try:
- result = await self._run_in_threadpool(
- self.agfs.rm, path, recursive=recursive
- )
+ result = await self._run_in_threadpool(self.agfs.rm, path, recursive=recursive)
except AGFSDirectoryNotEmptyError:
raise FailedPreconditionError(
f"Directory not empty: {uri}. Use recursive=True to delete non-empty directories."
@@ -531,7 +528,7 @@ class VikingFS:
result = {"estimated_deleted_count": estimated_count}
return result
except LockAcquisitionError:
- raise ResourceBusyError(f"Resource is being processed: {uri}")
+ raise ResourceBusyError(f"Resource is being processed: {uri}", uri=uri)
async def mv(
self,
@@ -561,16 +558,25 @@ class VikingFS:
except Exception:
raise FileNotFoundError(f"mv source not found: {old_uri}")
- dst_parent = new_path.rsplit("/", 1)[0] if "/" in new_path else new_path
+ lock_context = (
+ LockContext(
+ get_lock_manager(),
+ [old_path],
+ lock_mode="mv",
+ mv_dst_path=new_path,
+ src_is_dir=True,
+ handle=lock_handle,
+ )
+ if is_dir
+ else LockContext(
+ get_lock_manager(),
+ [old_path, new_path],
+ lock_mode="exact",
+ handle=lock_handle,
+ )
+ )
- async with LockContext(
- get_lock_manager(),
- [old_path],
- lock_mode="mv",
- mv_dst_parent_path=dst_parent,
- src_is_dir=is_dir,
- handle=lock_handle,
- ):
+ async with lock_context:
uris_to_move = await self._collect_uris(old_path, recursive=True, ctx=ctx)
uris_to_move.append(target_uri)
@@ -919,8 +925,7 @@ class VikingFS:
for start in range(0, len(file_uris), _DEFAULT_GREP_FILE_CONCURRENCY):
batch_uris = file_uris[start : start + _DEFAULT_GREP_FILE_CONCURRENCY]
batch_jobs = [
- self._grep_single_file(entry_uri, compiled_pattern, ctx)
- for entry_uri in batch_uris
+ self._grep_single_file(entry_uri, compiled_pattern, ctx) for entry_uri in batch_uris
]
batch_results = await asyncio.gather(*batch_jobs)
for matches, scanned_count in batch_results:
@@ -1971,14 +1976,13 @@ class VikingFS:
old_path = self._uri_to_path(old_dir, ctx=real_ctx)
new_path = self._uri_to_path(new_dir, ctx=real_ctx)
- dst_parent = new_path.rsplit("/", 1)[0] if "/" in new_path else new_path
try:
async with LockContext(
get_lock_manager(),
[old_path],
lock_mode="mv",
- mv_dst_parent_path=dst_parent,
+ mv_dst_path=new_path,
src_is_dir=True,
handle=lock_handle,
):
@@ -1991,7 +1995,7 @@ class VikingFS:
)
except LockAcquisitionError:
- raise ResourceBusyError(f"Resource is being processed: {old_dir}")
+ raise ResourceBusyError(f"Resource is being processed: {old_dir}", uri=old_dir)
def _get_vector_store(self) -> Optional["VikingVectorIndexBackend"]:
"""Get vector store instance."""
diff --git a/openviking/sync_client.py b/openviking/sync_client.py
index 2d57b8817..d6bd4ed2c 100644
--- a/openviking/sync_client.py
+++ b/openviking/sync_client.py
@@ -148,6 +148,8 @@ class SyncOpenViking:
"""Add resource to OpenViking (resources scope only)
Args:
+ to: Exact target URI. Existing targets keep the add_resource incremental-update behavior.
+ parent: Target parent URI for automatic child naming.
build_index: Whether to build vector index immediately (default: True).
summarize: Whether to generate summary (default: False).
**kwargs: Extra options forwarded to the parser chain, e.g.
diff --git a/openviking/utils/resource_processor.py b/openviking/utils/resource_processor.py
index a3d6d6ac2..d92431826 100644
--- a/openviking/utils/resource_processor.py
+++ b/openviking/utils/resource_processor.py
@@ -114,9 +114,10 @@ class ResourceProcessor:
Workflow:
1. Parse source (writes to temp directory)
- 2. TreeBuilder moves to AGFS
- 3. (Optional) Build vector index
- 4. (Optional) Summarize
+ 2. TreeBuilder builds final URI metadata
+ 3. Source commit moves temp content to the final path
+ 4. (Optional) Build vector index
+ 5. (Optional) Summarize
"""
result = {
"status": "success",
@@ -246,7 +247,7 @@ class ResourceProcessor:
except Exception:
pass
- # ============ Phase 3.5: 首次添加立即落盘 + 生命周期锁 ============
+ # ============ Phase 3.5: Source commit + lifecycle lock ============
root_uri = result.get("root_uri")
temp_uri = result.get("temp_uri") # temp_doc_uri
original_temp_uri = temp_uri # 保存原始 temp_uri 用于最终输出
@@ -254,50 +255,30 @@ class ResourceProcessor:
lifecycle_lock_handle_id = ""
if root_uri and temp_uri:
- from openviking.storage.transaction import LockContext, get_lock_manager
+ from openviking.storage.transaction import get_lock_manager
stage_start = time.perf_counter()
stage_status = "ok"
viking_fs = get_viking_fs()
lock_manager = get_lock_manager()
try:
- target_exists = await viking_fs.exists(root_uri, ctx=ctx)
-
- if not target_exists:
+ if candidate_uri:
+ root_uri, lifecycle_lock_handle_id = await self._commit_unique_candidate(
+ candidate_uri=candidate_uri,
+ ctx=ctx,
+ )
+ result["root_uri"] = root_uri
+ else:
dst_path = viking_fs._uri_to_path(root_uri, ctx=ctx)
- parent_path = dst_path.rsplit("/", 1)[0] if "/" in dst_path else dst_path
-
- parent_uri = "/".join(root_uri.rstrip("/").rsplit("/", 1)[:-1])
- if parent_uri:
- await viking_fs.mkdir(parent_uri, exist_ok=True, ctx=ctx)
-
- async with LockContext(lock_manager, [parent_path], lock_mode="point"):
- if candidate_uri:
- with viking_fs.bind_request_context(ctx):
- root_uri = await self.tree_builder._resolve_unique_uri(
- candidate_uri
- )
- result["root_uri"] = root_uri
- dst_path = viking_fs._uri_to_path(root_uri, ctx=ctx)
-
- src_path = viking_fs._uri_to_path(temp_uri, ctx=ctx)
- await asyncio.to_thread(viking_fs.agfs.mv, src_path, dst_path)
-
- lifecycle_lock_handle_id = await self._try_acquire_lifecycle_lock(
- lock_manager, dst_path
- )
-
+ handle = lock_manager.create_handle()
try:
- await viking_fs.delete_temp(parse_result.temp_dir_path, ctx=ctx)
+ lifecycle_lock_handle_id = await self._acquire_lifecycle_lock(
+ lock_manager, dst_path, uri=root_uri, handle=handle
+ )
except Exception:
- pass
-
- result["temp_uri"] = root_uri
- else:
- resource_path = viking_fs._uri_to_path(root_uri, ctx=ctx)
- lifecycle_lock_handle_id = await self._try_acquire_lifecycle_lock(
- lock_manager, resource_path
- )
+ await lock_manager.release(handle)
+ lifecycle_lock_handle_id = ""
+ raise
except Exception:
stage_status = "error"
raise
@@ -312,12 +293,6 @@ class ResourceProcessor:
except Exception:
pass
- # Only switch to root_uri after a first-time persist. For repeated
- # writes to an existing target we must keep the fresh temp tree so
- # semantic processing can diff temp source against target_uri.
- if not target_exists:
- temp_uri = root_uri
-
# ============ Phase 4: Optional Steps ============
build_index = kwargs.get("build_index", True)
temp_uri_for_summarize = temp_uri or parse_result.temp_dir_path
@@ -353,12 +328,27 @@ class ResourceProcessor:
except Exception:
pass
elif lifecycle_lock_handle_id:
- # 无下游处理接管锁,主动释放
+ # No downstream worker will sync temp content or release the lifecycle lock.
from openviking.storage.transaction import get_lock_manager
- handle = get_lock_manager().get_handle(lifecycle_lock_handle_id)
+ lock_manager = get_lock_manager()
+ handle = lock_manager.get_handle(lifecycle_lock_handle_id)
if handle:
- await get_lock_manager().release(handle)
+ if temp_uri:
+ from openviking.pyagfs.helpers import cp as agfs_cp
+
+ viking_fs = get_viking_fs()
+ src_path = viking_fs._uri_to_path(temp_uri, ctx=ctx)
+ dst_path = viking_fs._uri_to_path(root_uri, ctx=ctx)
+ await asyncio.to_thread(
+ agfs_cp,
+ viking_fs.agfs,
+ src_path,
+ dst_path,
+ recursive=True,
+ )
+ await viking_fs.delete_temp(parse_result.temp_dir_path, ctx=ctx)
+ await lock_manager.release(handle)
# 恢复原始 temp_uri 用于输出
if original_temp_uri is not None:
@@ -366,12 +356,66 @@ class ResourceProcessor:
return result
+ async def _commit_unique_candidate(
+ self,
+ *,
+ candidate_uri: str,
+ ctx: RequestContext,
+ max_attempts: int = 100,
+ ) -> tuple[str, str]:
+ """Pick the first free candidate URI and reserve it with a lifecycle TreeLock."""
+ from openviking.storage.errors import ResourceBusyError
+ from openviking.storage.transaction import get_lock_manager
+
+ viking_fs = get_viking_fs()
+ lock_manager = get_lock_manager()
+
+ for attempt in range(max_attempts + 1):
+ root_uri = candidate_uri if attempt == 0 else f"{candidate_uri}_{attempt}"
+ if await viking_fs.exists(root_uri, ctx=ctx):
+ continue
+
+ dst_path = viking_fs._uri_to_path(root_uri, ctx=ctx)
+ handle = lock_manager.create_handle()
+ try:
+ lifecycle_lock_handle_id = await self._acquire_lifecycle_lock(
+ lock_manager, dst_path, uri=root_uri, handle=handle, timeout=0.0
+ )
+ return root_uri, lifecycle_lock_handle_id
+ except ResourceBusyError:
+ await lock_manager.release(handle)
+ continue
+ except Exception:
+ await lock_manager.release(handle)
+ raise
+
+ raise FileExistsError(
+ f"Cannot resolve unique name for {candidate_uri} after {max_attempts} attempts"
+ )
+
@staticmethod
- async def _try_acquire_lifecycle_lock(lock_manager, path: str) -> str:
- """尝试获取 SUBTREE 生命周期锁,失败时优雅降级返回空字符串。"""
- handle = lock_manager.create_handle()
- if await lock_manager.acquire_subtree(handle, path):
+ async def _acquire_lifecycle_lock(
+ lock_manager,
+ path: str,
+ *,
+ uri: str = "",
+ handle=None,
+ timeout: Optional[float] = None,
+ ) -> str:
+ """Acquire per-resource TreeLock lifecycle lock or raise a structured conflict."""
+ from openviking.storage.errors import ResourceBusyError
+
+ owns_handle = handle is None
+ if handle is None:
+ handle = lock_manager.create_handle()
+ if await lock_manager.acquire_tree(handle, path, timeout=timeout):
return handle.id
+ if owns_handle:
+ await lock_manager.release(handle)
logger.warning(f"[ResourceProcessor] Failed to acquire lifecycle lock on {path}")
- await lock_manager.release(handle)
- return ""
+ raise ResourceBusyError(
+ f"Resource is busy: {uri or path}",
+ uri=uri or path,
+ conflict_type="path_busy",
+ retryable=True,
+ )
diff --git a/openviking/utils/summarizer.py b/openviking/utils/summarizer.py
index d3fb40a59..72b47787e 100644
--- a/openviking/utils/summarizer.py
+++ b/openviking/utils/summarizer.py
@@ -110,13 +110,22 @@ class Summarizer:
if msg.telemetry_id:
get_request_wait_tracker().register_semantic_root(msg.telemetry_id, msg.id)
try:
- await semantic_queue.enqueue(msg)
+ enqueue_id = await semantic_queue.enqueue(msg)
except Exception as e:
if msg.telemetry_id:
get_request_wait_tracker().mark_semantic_failed(
msg.telemetry_id, msg.id, str(e)
)
raise
+ if enqueue_id == "deduplicated":
+ if msg.telemetry_id:
+ get_request_wait_tracker().mark_semantic_done(
+ msg.telemetry_id,
+ msg.id,
+ processed_delta=0,
+ )
+ logger.info("Semantic generation already queued for: %s", target_uri)
+ continue
enqueued_count += 1
logger.info(
f"Enqueued semantic generation for: {target_uri} (skip_vectorization={skip_vectorization})"
diff --git a/tests/misc/test_resource_processor_mv.py b/tests/misc/test_resource_processor_mv.py
index 285daf926..5e7aa1af3 100644
--- a/tests/misc/test_resource_processor_mv.py
+++ b/tests/misc/test_resource_processor_mv.py
@@ -39,15 +39,76 @@ class _CtxMgr:
return False
+class _FakeLockManager:
+ def __init__(self, *, busy_tree_paths=None):
+ from openviking.storage.transaction.lock_handle import LockHandle
+
+ self._lock_handle_cls = LockHandle
+ self._handles = {}
+ self.acquired_exact_paths = []
+ self.acquired_tree_paths = []
+ self.tree_attempts = []
+ self.busy_tree_paths = set(busy_tree_paths or [])
+
+ def create_handle(self):
+ handle = self._lock_handle_cls()
+ self._handles[handle.id] = handle
+ return handle
+
+ async def acquire_exact_path_batch(self, handle, paths, timeout=None):
+ for path in paths:
+ lock_path = f"exact:{path}"
+ handle.add_lock(lock_path)
+ self.acquired_exact_paths.append(path)
+ return True
+
+ async def acquire_tree(self, handle, path, timeout=None):
+ self.tree_attempts.append((path, timeout))
+ if path in self.busy_tree_paths:
+ return False
+ lock_path = f"tree:{path}"
+ handle.add_lock(lock_path)
+ self.acquired_tree_paths.append(path)
+ return True
+
+ async def release_selected(self, handle, lock_paths):
+ for path in lock_paths:
+ handle.remove_lock(path)
+
+ async def release(self, handle):
+ for path in list(handle.locks):
+ handle.remove_lock(path)
+ self._handles.pop(handle.id, None)
+
+ def get_handle(self, handle_id):
+ handle = self._handles.get(handle_id)
+ if handle and handle.locks:
+ return handle
+ return None
+
+
class _FakeVikingFS:
- def __init__(self, *, exists_result=False):
- self.agfs = SimpleNamespace(mv=MagicMock(return_value={"status": "ok"}))
+ def __init__(self, *, exists_result=False, existing_uris=None):
+ self.agfs = SimpleNamespace(
+ cat=MagicMock(return_value=b"content"),
+ ls=MagicMock(return_value=[{"name": "content.md", "isDir": False}]),
+ mkdir=MagicMock(return_value={"status": "ok"}),
+ mv=MagicMock(return_value={"status": "ok"}),
+ stat=MagicMock(return_value={"isDir": True}),
+ write=MagicMock(return_value={"status": "ok"}),
+ )
+ self.mv = AsyncMock(return_value={})
self._exists_result = exists_result
+ self._existing_uris = set(existing_uris or [])
+ self.exists_calls = []
def bind_request_context(self, ctx):
return _CtxMgr()
async def exists(self, uri, ctx=None):
+ self.exists_calls.append(uri)
+ if self._existing_uris:
+ return uri in self._existing_uris
return self._exists_result
async def mkdir(self, uri, exist_ok=False, ctx=None):
@@ -65,12 +126,17 @@ async def test_resource_processor_first_add_persist_does_not_await_agfs_mv(monke
from openviking.utils.resource_processor import ResourceProcessor
fake_fs = _FakeVikingFS()
+ fake_lock_manager = _FakeLockManager()
monkeypatch.setattr(
"openviking.utils.resource_processor.get_current_telemetry",
lambda: _DummyTelemetry(),
)
monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: fake_fs)
+ monkeypatch.setattr(
+ "openviking.storage.transaction.get_lock_manager",
+ lambda: fake_lock_manager,
+ )
rp = ResourceProcessor(vikingdb=_DummyVikingDB(), media_storage=None)
rp._get_media_processor = MagicMock()
@@ -89,11 +155,16 @@ async def test_resource_processor_first_add_persist_does_not_await_agfs_mv(monke
)
rp.tree_builder.finalize_from_temp = AsyncMock(return_value=context_tree)
- result = await rp.process_resource(path="x", ctx=object(), build_index=False, summarize=False)
+ ctx = object()
+ result = await rp.process_resource(path="x", ctx=ctx, build_index=False, summarize=False)
assert result["status"] == "success"
assert result["root_uri"] == "viking://resources/root"
- fake_fs.agfs.mv.assert_called_once()
+ fake_fs.agfs.mv.assert_not_called()
+ fake_fs.agfs.write.assert_called_once()
+ fake_fs.mv.assert_not_awaited()
+ assert fake_lock_manager.acquired_exact_paths == []
+ assert fake_lock_manager.acquired_tree_paths == ["/mock/resources/root"]
@pytest.mark.asyncio
@@ -101,6 +172,7 @@ async def test_resource_processor_second_add_preserves_temp_uri_for_incremental(
from openviking.utils.resource_processor import ResourceProcessor
fake_fs = _FakeVikingFS(exists_result=True)
+ fake_lock_manager = _FakeLockManager()
summarize_calls = []
monkeypatch.setattr(
@@ -108,6 +180,10 @@ async def test_resource_processor_second_add_preserves_temp_uri_for_incremental(
lambda: _DummyTelemetry(),
)
monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: fake_fs)
+ monkeypatch.setattr(
+ "openviking.storage.transaction.get_lock_manager",
+ lambda: fake_lock_manager,
+ )
rp = ResourceProcessor(vikingdb=_DummyVikingDB(), media_storage=None)
rp._get_media_processor = MagicMock()
@@ -127,8 +203,9 @@ async def test_resource_processor_second_add_preserves_temp_uri_for_incremental(
rp.tree_builder.finalize_from_temp = AsyncMock(return_value=context_tree)
rp._summarizer = SimpleNamespace(
summarize=AsyncMock(
- side_effect=lambda *args, **kwargs: summarize_calls.append(kwargs)
- or {"status": "success"}
+ side_effect=lambda *args, **kwargs: (
+ summarize_calls.append(kwargs) or {"status": "success"}
+ )
)
)
@@ -138,3 +215,68 @@ async def test_resource_processor_second_add_preserves_temp_uri_for_incremental(
assert result["root_uri"] == "viking://resources/root"
assert summarize_calls[0]["temp_uris"] == ["viking://temp/root_tmp"]
fake_fs.agfs.mv.assert_not_called()
+ fake_fs.agfs.write.assert_not_called()
+ fake_fs.mv.assert_not_awaited()
+ assert fake_lock_manager.acquired_exact_paths == []
+ assert fake_lock_manager.acquired_tree_paths == ["/mock/resources/root"]
+
+
+async def test_resource_processor_auto_candidate_skips_existing_and_busy(monkeypatch):
+ from openviking.utils.resource_processor import ResourceProcessor
+
+ fake_fs = _FakeVikingFS(existing_uris={"viking://resources/root"})
+ fake_lock_manager = _FakeLockManager(busy_tree_paths={"/mock/resources/root_1"})
+ summarize_calls = []
+
+ monkeypatch.setattr(
+ "openviking.utils.resource_processor.get_current_telemetry",
+ lambda: _DummyTelemetry(),
+ )
+ monkeypatch.setattr("openviking.utils.resource_processor.get_viking_fs", lambda: fake_fs)
+ monkeypatch.setattr(
+ "openviking.storage.transaction.get_lock_manager",
+ lambda: fake_lock_manager,
+ )
+
+ rp = ResourceProcessor(vikingdb=_DummyVikingDB(), media_storage=None)
+ rp._get_media_processor = MagicMock()
+ rp._get_media_processor.return_value.process = AsyncMock(
+ return_value=SimpleNamespace(
+ temp_dir_path="viking://temp/tmpdir",
+ source_path="x",
+ source_format="text",
+ meta={},
+ warnings=[],
+ )
+ )
+
+ context_tree = SimpleNamespace(
+ root=SimpleNamespace(uri="viking://resources/root", temp_uri="viking://temp/root_tmp"),
+ _candidate_uri="viking://resources/root",
+ )
+ rp.tree_builder.finalize_from_temp = AsyncMock(return_value=context_tree)
+ rp._summarizer = SimpleNamespace(
+ summarize=AsyncMock(
+ side_effect=lambda *args, **kwargs: (
+ summarize_calls.append(kwargs) or {"status": "success"}
+ )
+ )
+ )
+
+ result = await rp.process_resource(path="x", ctx=object(), build_index=True)
+
+ assert result["status"] == "success"
+ assert result["root_uri"] == "viking://resources/root_2"
+ assert fake_fs.exists_calls == [
+ "viking://resources/root",
+ "viking://resources/root_1",
+ "viking://resources/root_2",
+ ]
+ assert fake_lock_manager.tree_attempts == [
+ ("/mock/resources/root_1", 0.0),
+ ("/mock/resources/root_2", 0.0),
+ ]
+ assert fake_lock_manager.acquired_exact_paths == []
+ assert fake_lock_manager.acquired_tree_paths == ["/mock/resources/root_2"]
+ assert summarize_calls[0]["temp_uris"] == ["viking://temp/root_tmp"]
diff --git a/tests/misc/test_tree_builder_dedup.py b/tests/misc/test_tree_builder_dedup.py
index 19eb79a07..645095bdd 100644
--- a/tests/misc/test_tree_builder_dedup.py
+++ b/tests/misc/test_tree_builder_dedup.py
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
-"""Tests for TreeBuilder._resolve_unique_uri — duplicate filename auto-rename."""
+"""Tests for TreeBuilder final URI metadata."""
import os
import sys
@@ -12,98 +12,6 @@ import pytest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
-def _make_viking_fs_mock(existing_uris: set[str]):
- """Create a mock VikingFS whose stat() raises for non-existing URIs."""
- fs = MagicMock()
-
- async def _stat(uri, **kwargs):
- if uri in existing_uris:
- return {"name": uri.split("/")[-1], "isDir": True}
- raise FileNotFoundError(f"Not found: {uri}")
-
- fs.stat = AsyncMock(side_effect=_stat)
- return fs
-
-
-class TestResolveUniqueUri:
- @pytest.mark.asyncio
- async def test_no_conflict(self):
- """When the URI is free, return it unchanged."""
- from openviking.parse.tree_builder import TreeBuilder
-
- fs = _make_viking_fs_mock(set())
- builder = TreeBuilder()
-
- with patch("openviking.parse.tree_builder.get_viking_fs", return_value=fs):
- result = await builder._resolve_unique_uri("viking://resources/report")
-
- assert result == "viking://resources/report"
-
- @pytest.mark.asyncio
- async def test_single_conflict(self):
- """When base name exists, should return name_1."""
- from openviking.parse.tree_builder import TreeBuilder
-
- existing = {"viking://resources/report"}
- fs = _make_viking_fs_mock(existing)
- builder = TreeBuilder()
-
- with patch("openviking.parse.tree_builder.get_viking_fs", return_value=fs):
- result = await builder._resolve_unique_uri("viking://resources/report")
-
- assert result == "viking://resources/report_1"
-
- @pytest.mark.asyncio
- async def test_multiple_conflicts(self):
- """When _1 and _2 also exist, should return _3."""
- from openviking.parse.tree_builder import TreeBuilder
-
- existing = {
- "viking://resources/report",
- "viking://resources/report_1",
- "viking://resources/report_2",
- }
- fs = _make_viking_fs_mock(existing)
- builder = TreeBuilder()
-
- with patch("openviking.parse.tree_builder.get_viking_fs", return_value=fs):
- result = await builder._resolve_unique_uri("viking://resources/report")
-
- assert result == "viking://resources/report_3"
-
- @pytest.mark.asyncio
- async def test_max_attempts_exceeded(self):
- """When all candidate names are taken, raise FileExistsError."""
- from openviking.parse.tree_builder import TreeBuilder
-
- existing = {"viking://resources/report"} | {
- f"viking://resources/report_{i}" for i in range(1, 6)
- }
- fs = _make_viking_fs_mock(existing)
- builder = TreeBuilder()
-
- with patch("openviking.parse.tree_builder.get_viking_fs", return_value=fs):
- with pytest.raises(FileExistsError, match="Cannot resolve unique name"):
- await builder._resolve_unique_uri("viking://resources/report", max_attempts=5)
-
- @pytest.mark.asyncio
- async def test_gap_in_sequence(self):
- """If _1 exists but _2 does not, should return _2 (not skip to _3)."""
- from openviking.parse.tree_builder import TreeBuilder
-
- existing = {
- "viking://resources/report",
- "viking://resources/report_1",
- }
- fs = _make_viking_fs_mock(existing)
- builder = TreeBuilder()
-
- with patch("openviking.parse.tree_builder.get_viking_fs", return_value=fs):
- result = await builder._resolve_unique_uri("viking://resources/report")
-
- assert result == "viking://resources/report_2"
-
-
class TestFinalizeFromTemp:
@staticmethod
def _make_fs(entries, existing_uris: set[str]):
@@ -117,8 +25,12 @@ class TestFinalizeFromTemp:
return {"name": uri.split("/")[-1], "isDir": True}
raise FileNotFoundError(f"Not found: {uri}")
+ async def _exists(uri, **kwargs):
+ return uri in existing_uris
+
fs.ls = AsyncMock(side_effect=_ls)
fs.stat = AsyncMock(side_effect=_stat)
+ fs.exists = AsyncMock(side_effect=_exists)
return fs
@pytest.mark.asyncio
diff --git a/tests/server/test_content_write_service.py b/tests/server/test_content_write_service.py
index f6c46840e..b37bb5dca 100644
--- a/tests/server/test_content_write_service.py
+++ b/tests/server/test_content_write_service.py
@@ -181,7 +181,11 @@ class _FakeLockManager:
def create_handle(self):
return self.handle
- async def acquire_subtree(self, handle, path):
+ async def acquire_tree(self, handle, path):
+ del handle, path
+ return True
+
+ async def acquire_exact_path(self, handle, path):
del handle, path
return True
@@ -229,6 +233,57 @@ class _FakeVikingFS:
self.content.pop(uri, None)
+class _FakeSemanticQueue:
+ def __init__(self):
+ self.messages = []
+
+ async def enqueue(self, msg):
+ self.messages.append(msg)
+ return "queued-id"
+
+
+class _FakeQueueManager:
+ SEMANTIC = "semantic"
+
+ def __init__(self, queue):
+ self.queue = queue
+
+ def get_queue(self, name, allow_create=False):
+ del allow_create
+ assert name == self.SEMANTIC
+ return self.queue
+
+
+async def test_resource_write_semantic_refresh_uses_coalesce_key(monkeypatch):
+ file_uri = "viking://resources/demo/doc.md"
+ root_uri = "viking://resources/demo"
+ ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
+ queue = _FakeSemanticQueue()
+ coordinator = ContentWriteCoordinator(
+ viking_fs=_FakeVikingFS(file_uri=file_uri, root_uri=root_uri)
+ )
+
+ monkeypatch.setattr(
+ "openviking.storage.content_write.get_queue_manager",
+ lambda: _FakeQueueManager(queue),
+ )
+
+ await coordinator._enqueue_semantic_refresh(
+ root_uri=root_uri,
+ changed_uri=file_uri,
+ context_type="resource",
+ ctx=ctx,
+ lifecycle_lock_handle_id="lock-1",
+ )
+
+ assert len(queue.messages) == 1
+ assert queue.messages[0].coalesce_key == (
+ "resource|default|default|default|viking://resources/demo"
+ )
+ assert queue.messages[0].lifecycle_lock_handle_id == "lock-1"
+
+
@pytest.mark.asyncio
async def test_write_timeout_after_enqueue_does_not_release_resource_lock(monkeypatch):
file_uri = "viking://resources/demo/doc.md"
@@ -372,7 +427,7 @@ async def test_resource_write_rolls_back_create_when_enqueue_fails(monkeypatch):
@pytest.mark.asyncio
-async def test_memory_write_timeout_after_enqueue_does_not_release_lock(monkeypatch):
+async def test_memory_write_timeout_after_enqueue_releases_write_lock(monkeypatch):
file_uri = "viking://user/default/memories/preferences/theme.md"
root_uri = "viking://user/default/memories/preferences"
ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.USER)
@@ -414,7 +469,7 @@ async def test_memory_write_timeout_after_enqueue_does_not_release_lock(monkeypa
wait=True,
)
- assert lock_manager.release_calls == []
+ assert lock_manager.release_calls == ["lock-1"]
# Create-mode test helpers
diff --git a/tests/server/test_error_mapping.py b/tests/server/test_error_mapping.py
index d6419d34b..81f6b3c19 100644
--- a/tests/server/test_error_mapping.py
+++ b/tests/server/test_error_mapping.py
@@ -5,6 +5,7 @@
from openviking.pyagfs.exceptions import AGFSClientError, AGFSHTTPError
from openviking.server.error_mapping import map_exception
+from openviking.storage.errors import LockAcquisitionError, ResourceBusyError
from openviking_cli.exceptions import FailedPreconditionError, InvalidURIError, NotFoundError
@@ -139,3 +140,40 @@ def test_bare_model_api_key_required_maps_to_failed_precondition():
assert isinstance(mapped, FailedPreconditionError)
assert mapped.code == "FAILED_PRECONDITION"
+
+
+def test_resource_busy_maps_to_structured_conflict():
+ mapped = map_exception(
+ ResourceBusyError(
+ "Reexact is busy: viking://resources/docs/a.md",
+ uri="viking://resources/docs/a.md",
+ conflict_type="path_busy",
+ retryable=True,
+ ),
+ resource="viking://resources/docs",
+ )
+
+ assert mapped is not None
+ assert mapped.code == "CONFLICT"
+ assert mapped.details == {
+ "resource": "viking://resources/docs/a.md",
+ "uri": "viking://resources/docs/a.md",
+ "conflict_type": "path_busy",
+ "retryable": True,
+ }
+
+
+def test_lock_acquisition_maps_to_structured_conflict():
+ mapped = map_exception(
+ LockAcquisitionError("Failed to acquire exact lock"),
+ resource="viking://resources/docs/a.md",
+ )
+
+ assert mapped is not None
+ assert mapped.code == "CONFLICT"
+ assert mapped.details == {
+ "resource": "viking://resources/docs/a.md",
+ "uri": "viking://resources/docs/a.md",
+ "conflict_type": "path_busy",
+ "retryable": True,
+ }
diff --git a/tests/server/test_request_wait_tracking.py b/tests/server/test_request_wait_tracking.py
index 482b22e4a..1ec6f1fef 100644
--- a/tests/server/test_request_wait_tracking.py
+++ b/tests/server/test_request_wait_tracking.py
@@ -264,7 +264,7 @@ async def test_content_write_wait_uses_request_tracker(monkeypatch):
)
lock_manager = SimpleNamespace(
create_handle=lambda: SimpleNamespace(id="lock-1"),
- acquire_subtree=lambda handle, path: _return_true(handle, path),
+ acquire_tree=lambda handle, path: _return_true(handle, path),
release=lambda handle: _return_none(handle),
)
@@ -324,7 +324,7 @@ async def test_content_write_wait_uses_request_tracker_when_telemetry_disabled(m
)
lock_manager = SimpleNamespace(
create_handle=lambda: SimpleNamespace(id="lock-1"),
- acquire_subtree=lambda handle, path: _return_true(handle, path),
+ acquire_tree=lambda handle, path: _return_true(handle, path),
release=lambda handle: _return_none(handle),
)
diff --git a/tests/session/memory/test_compressor_v2.py b/tests/session/memory/test_compressor_v2.py
index 74db3a400..6e86dc5a0 100644
--- a/tests/session/memory/test_compressor_v2.py
+++ b/tests/session/memory/test_compressor_v2.py
@@ -16,6 +16,8 @@ import pytest
from openviking.message import Message, TextPart
from openviking.server.identity import RequestContext, Role
from openviking.session.compressor_v2 import SessionCompressorV2
+from openviking.session.memory.memory_updater import MemoryUpdateResult
+from openviking.session.memory.utils.content import deserialize_metadata, serialize_with_metadata
from openviking_cli.session.user_id import UserIdentifier
from openviking_cli.utils.config import get_openviking_config, initialize_openviking_config
@@ -427,12 +429,23 @@ class TestCompressorV2:
ctx = RequestContext(user=user, role=Role.ROOT)
messages = [Message.create_user("test")]
- class DummySchema:
+ class FixedSchema:
+ directory = "viking://user/{{ user_space }}/memories"
+ filename_template = "profile.md"
+
+ def filename_has_variables(self):
+ return False
+
+ class VariableSchema:
directory = "viking://user/{{ user_space }}/memories/events"
+ filename_template = "{{ event_name }}.md"
+
+ def filename_has_variables(self):
+ return True
class DummyProvider:
def get_memory_schemas(self, _ctx):
- return [DummySchema()]
+ return [FixedSchema(), VariableSchema()]
def _get_registry(self):
return object()
@@ -452,7 +465,7 @@ class TestCompressorV2:
lock_manager = SimpleNamespace(
create_handle=lambda: object(),
- acquire_subtree_batch=AsyncMock(return_value=False),
+ acquire_exact_tree_batch=AsyncMock(return_value=False),
release=AsyncMock(),
)
@@ -478,4 +491,158 @@ class TestCompressorV2:
)
assert result == []
- assert lock_manager.acquire_subtree_batch.await_count == 2
+ assert lock_manager.acquire_exact_tree_batch.await_count == 2
+ _, kwargs = lock_manager.acquire_exact_tree_batch.await_args
+ assert kwargs["exact_paths"] == ["/local/default/user/default/memories/profile.md"]
+ assert kwargs["tree_paths"] == ["/local/default/user/default/memories/events"]
+
+ @pytest.mark.asyncio
+ async def test_extract_phase_runs_post_apply_before_lock_release(self):
+ """Agent experience source metadata should be updated inside the schema lock."""
+ compressor = SessionCompressorV2(vikingdb=None)
+ user = UserIdentifier.the_default_user()
+ ctx = RequestContext(user=user, role=Role.ROOT)
+ messages = [Message.create_user("test")]
+ events: List[str] = []
+
+ class FakeVikingFS:
+ agfs = object()
+
+ def _uri_to_path(self, uri: str, ctx=None) -> str:
+ return uri
+
+ class DummyProvider:
+ def get_memory_schemas(self, _ctx):
+ return []
+
+ def _get_registry(self):
+ return object()
+
+ class DummyExtractLoop:
+ def __init__(self, **kwargs):
+ pass
+
+ async def run(self):
+ return SimpleNamespace(upsert_operations=[], delete_file_contents=[]), []
+
+ class DummyUpdater:
+ async def apply_operations(self, operations, ctx, **kwargs):
+ events.append("apply")
+ result = MemoryUpdateResult()
+ result.written_uris = ["viking://agent/default/memories/experiences/debug.md"]
+ return result
+
+ config = SimpleNamespace(
+ vlm=SimpleNamespace(get_vlm_instance=lambda: object()),
+ memory=SimpleNamespace(
+ enable_role_id_memory_isolate=False,
+ v2_lock_max_retries=1,
+ v2_lock_retry_interval_seconds=0.0,
+ ),
+ )
+ handle = SimpleNamespace(id="handle-1", locks=[])
+
+ async def acquire_exact_tree_batch(*args, **kwargs):
+ events.append("acquire")
+ return True
+
+ async def release(_handle):
+ events.append("release")
+
+ lock_manager = SimpleNamespace(
+ create_handle=lambda: handle,
+ acquire_exact_tree_batch=AsyncMock(side_effect=acquire_exact_tree_batch),
+ release=AsyncMock(side_effect=release),
+ )
+
+ async def post_apply(result, inheritance_map, lock_handle):
+ assert result.written_uris == ["viking://agent/default/memories/experiences/debug.md"]
+ assert inheritance_map == {}
+ assert lock_handle is handle
+ events.append("post_apply")
+
+ with (
+ patch("openviking.session.compressor_v2.get_viking_fs", return_value=FakeVikingFS()),
+ patch("openviking.session.compressor_v2.get_openviking_config", return_value=config),
+ patch(
+ "openviking.session.memory.memory_isolation_handler.get_openviking_config",
+ return_value=config,
+ ),
+ patch("openviking.session.compressor_v2.ExtractLoop", DummyExtractLoop),
+ patch("openviking.storage.transaction.init_lock_manager"),
+ patch("openviking.storage.transaction.get_lock_manager", return_value=lock_manager),
+ patch.object(compressor, "_get_or_create_updater", return_value=DummyUpdater()),
+ ):
+ result = await compressor._run_extract_phase(
+ provider=DummyProvider(),
+ messages=messages,
+ ctx=ctx,
+ strict_extract_errors=True,
+ phase_label="experience(test)",
+ post_apply=post_apply,
+ )
+
+ assert result[0] == ["viking://agent/default/memories/experiences/debug.md"]
+ assert events == ["acquire", "apply", "post_apply", "release"]
+
+ @pytest.mark.asyncio
+ async def test_append_trajectories_uses_exact_lock(self):
+ """Fallback source metadata append should protect the read-modify-write."""
+ compressor = SessionCompressorV2(vikingdb=None)
+ user = UserIdentifier.the_default_user()
+ ctx = RequestContext(user=user, role=Role.ROOT)
+ exp_uri = "viking://agent/default/memories/experiences/debug.md"
+ events: List[str] = []
+
+ class FakeVikingFS:
+ def __init__(self):
+ self.content = serialize_with_metadata(
+ {
+ "content": "debug login issue",
+ "source_trajectories": ["traj-0"],
+ }
+ )
+
+ def _uri_to_path(self, uri: str, ctx=None) -> str:
+ return f"/local/default/agent/default/memories/experiences/{uri.rsplit('/', 1)[-1]}"
+
+ async def read_file(self, uri: str, ctx=None):
+ events.append("read")
+ return self.content
+
+ async def write_file(self, uri: str, content: str, ctx=None):
+ events.append("write")
+ self.content = content
+
+ handle = SimpleNamespace(id="handle-1", locks=[])
+
+ async def acquire_exact_path_batch(_handle, paths):
+ events.append(f"exact:{paths[0]}")
+ return True
+
+ async def release(_handle):
+ events.append("release")
+
+ lock_manager = SimpleNamespace(
+ create_handle=lambda: handle,
+ acquire_exact_path_batch=AsyncMock(side_effect=acquire_exact_path_batch),
+ release=AsyncMock(side_effect=release),
+ )
+ viking_fs = FakeVikingFS()
+
+ with patch("openviking.storage.transaction.get_lock_manager", return_value=lock_manager):
+ await compressor._append_trajectories_to_experiences(
+ [exp_uri],
+ ["traj-1"],
+ ctx,
+ viking_fs,
+ )
+
+ metadata = deserialize_metadata(viking_fs.content)
+ assert metadata["source_trajectories"] == ["traj-0", "traj-1"]
+ assert events == [
+ "exact:/local/default/agent/default/memories/experiences/debug.md",
+ "read",
+ "write",
+ "release",
+ ]
diff --git a/tests/storage/test_semantic_processor_lifecycle_lock.py b/tests/storage/test_semantic_processor_lifecycle_lock.py
index 9fd43e9ae..b1cc09f79 100644
--- a/tests/storage/test_semantic_processor_lifecycle_lock.py
+++ b/tests/storage/test_semantic_processor_lifecycle_lock.py
@@ -32,7 +32,7 @@ class _FakeLockManager:
self._handles[handle.id] = handle
return handle
- async def acquire_subtree(self, handle, lock_path):
+ async def acquire_tree(self, handle, lock_path):
del handle, lock_path
return True
diff --git a/tests/storage/test_semantic_queue_memory_dedupe.py b/tests/storage/test_semantic_queue_memory_dedupe.py
index 5e66e914b..c7fe8bf7d 100644
--- a/tests/storage/test_semantic_queue_memory_dedupe.py
+++ b/tests/storage/test_semantic_queue_memory_dedupe.py
@@ -3,12 +3,14 @@
"""Tests for memory-context semantic enqueue deduplication (#769)."""
from unittest.mock import AsyncMock, MagicMock, patch
+from uuid import uuid4
import pytest
from openviking.storage.queuefs.named_queue import NamedQueue
from openviking.storage.queuefs.semantic_msg import SemanticMsg
-from openviking.storage.queuefs.semantic_queue import SemanticQueue
+from openviking.storage.queuefs.semantic_processor import SemanticProcessor
+from openviking.storage.queuefs.semantic_queue import SemanticQueue, is_semantic_msg_stale
@pytest.mark.asyncio
@@ -70,3 +72,181 @@ async def test_non_memory_context_not_deduped():
await q.enqueue(SemanticMsg(uri=uri, context_type="resource"))
await q.enqueue(SemanticMsg(uri=uri, context_type="resource"))
assert named_enqueue.call_count == 2
+
+
+async def test_coalesced_semantic_messages_mark_old_version_stale():
+ mock_agfs = MagicMock()
+ with patch.object(NamedQueue, "enqueue", new_callable=AsyncMock) as named_enqueue:
+ named_enqueue.return_value = "queued-id"
+ q = SemanticQueue(mock_agfs, "/queue", "semantic")
+ coalesce_key = f"resource|acc|u|a|viking://resources/docs/{uuid4().hex}"
+ first = SemanticMsg(
+ uri="viking://resources/docs",
+ context_type="resource",
+ coalesce_key=coalesce_key,
+ )
+ second = SemanticMsg(
+ uri="viking://resources/docs",
+ context_type="resource",
+ coalesce_key=first.coalesce_key,
+ )
+
+ await q.enqueue(first)
+ await q.enqueue(second)
+
+ assert first.coalesce_version == 1
+ assert second.coalesce_version == 2
+ assert is_semantic_msg_stale(first)
+ assert not is_semantic_msg_stale(second)
+
+
+class _FakeHandle:
+ def __init__(self):
+ self.id = "lock-1"
+ self.locks = []
+
+
+class _FakeLockManager:
+ def __init__(self):
+ self.acquired_batches = []
+ self.release_calls = []
+
+ def create_handle(self):
+ return _FakeHandle()
+
+ def get_handle(self, handle_id):
+ del handle_id
+ return None
+
+ async def acquire_exact_path_batch(self, handle, paths):
+ self.acquired_batches.append(paths)
+ handle.locks.extend(paths)
+ return True
+
+ async def release(self, handle):
+ self.release_calls.append(handle.id)
+
+ async def release_selected(self, handle, lock_paths):
+ del handle, lock_paths
+
+
+class _FakeVikingFS:
+ def __init__(self):
+ self.writes = []
+
+ def _uri_to_path(self, uri, ctx=None):
+ del ctx
+ return f"/fake/{uri.replace('://', '/').strip('/')}"
+
+ async def write_file(self, uri, content, ctx=None):
+ del ctx
+ self.writes.append((uri, content))
+
+
+class _FakeMemoryDirFS:
+ async def ls(self, uri, ctx=None):
+ del uri, ctx
+ return [
+ {"name": "first.md", "isDir": False},
+ {"name": "second.md", "isDir": False},
+ ]
+
+
+async def test_stale_memory_semantic_write_is_skipped(monkeypatch):
+ lock_manager = _FakeLockManager()
+ viking_fs = _FakeVikingFS()
+ processor = SemanticProcessor()
+ coalesce_key = f"memory|acc|u|a|viking://user/default/memories/preferences/{uuid4().hex}"
+
+ with patch.object(NamedQueue, "enqueue", new_callable=AsyncMock):
+ q = SemanticQueue(MagicMock(), "/queue", "semantic")
+ first = SemanticMsg(
+ uri="viking://user/default/memories/preferences",
+ context_type="memory",
+ coalesce_key=coalesce_key,
+ )
+ latest = SemanticMsg(
+ uri="viking://user/default/memories/preferences",
+ context_type="memory",
+ coalesce_key=coalesce_key,
+ )
+ await q.enqueue(first)
+ await q.enqueue(latest)
+
+ monkeypatch.setattr("openviking.storage.transaction.get_lock_manager", lambda: lock_manager)
+
+ wrote_first = await processor._write_memory_directory_semantics(
+ msg=first,
+ viking_fs=viking_fs,
+ dir_uri=first.uri,
+ overview="old overview",
+ abstract="old abstract",
+ ctx=None,
+ )
+ wrote_latest = await processor._write_memory_directory_semantics(
+ msg=latest,
+ viking_fs=viking_fs,
+ dir_uri=latest.uri,
+ overview="latest overview",
+ abstract="latest abstract",
+ ctx=None,
+ )
+
+ assert not wrote_first
+ assert wrote_latest
+ assert lock_manager.acquired_batches == [
+ [
+ "/fake/viking/user/default/memories/preferences/.overview.md",
+ "/fake/viking/user/default/memories/preferences/.abstract.md",
+ ]
+ ]
+ assert viking_fs.writes == [
+ ("viking://user/default/memories/preferences/.overview.md", "latest overview"),
+ ("viking://user/default/memories/preferences/.abstract.md", "latest abstract"),
+ ]
+
+
+async def test_memory_directory_summarizes_all_uncached_files(monkeypatch):
+ processor = SemanticProcessor(max_concurrent_llm=4)
+ summaries = []
+
+ async def generate_file_summary(file_path, llm_sem=None, ctx=None):
+ del llm_sem, ctx
+ name = file_path.rsplit("/", 1)[-1]
+ return {"name": name, "summary": f"summary:{name}"}
+
+ async def generate_overview(dir_uri, file_summaries, children_abstracts, llm_sem=None):
+ del dir_uri, children_abstracts, llm_sem
+ summaries.extend(file_summaries)
+ return "overview"
+
+ async def write_semantics(**kwargs):
+ del kwargs
+ return True
+
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.semantic_processor.get_viking_fs",
+ lambda: _FakeMemoryDirFS(),
+ )
+ monkeypatch.setattr(processor, "_generate_single_file_summary", generate_file_summary)
+ monkeypatch.setattr(processor, "_generate_overview", generate_overview)
+ monkeypatch.setattr(processor, "_extract_abstract_from_overview", lambda overview: "abstract")
+ monkeypatch.setattr(
+ processor,
+ "_enforce_size_limits",
+ lambda overview, abstract: (overview, abstract),
+ )
+ monkeypatch.setattr(processor, "_write_memory_directory_semantics", write_semantics)
+
+ await processor._process_memory_directory(
+ SemanticMsg(
+ uri="viking://user/default/memories/preferences",
+ context_type="memory",
+ skip_vectorization=True,
+ )
+ )
+
+ assert [item["name"] for item in summaries] == ["first.md", "second.md"]
diff --git a/tests/transaction/conftest.py b/tests/transaction/conftest.py
new file mode 100644
index 000000000..29bfde87c
--- /dev/null
+++ b/tests/transaction/conftest.py
@@ -0,0 +1,99 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Fixtures for transaction lock tests."""
+
+import uuid
+
+import pytest
+
+
+class MemoryAgfs:
+ def __init__(self):
+ self.dirs = {"/"}
+ self.files: dict[str, bytes] = {}
+
+ def _parent(self, path: str) -> str:
+ path = path.rstrip("/") or "/"
+ parent = path.rsplit("/", 1)[0]
+ return parent or "/"
+
+ def stat(self, path: str):
+ path = path.rstrip("/") or "/"
+ if path in self.dirs:
+ return {"name": path.rsplit("/", 1)[-1], "isDir": True}
+ if path in self.files:
+ return {"name": path.rsplit("/", 1)[-1], "isDir": False}
+ raise FileNotFoundError(path)
+
+ def mkdir(self, path: str):
+ path = path.rstrip("/") or "/"
+ parent = self._parent(path)
+ if parent not in self.dirs:
+ raise FileNotFoundError(parent)
+ self.dirs.add(path)
+ return {"message": "created"}
+
+ def read(self, path: str):
+ if path not in self.files:
+ raise FileNotFoundError(path)
+ return self.files[path]
+
+ def cat(self, path: str):
+ return self.read(path)
+
+ def write(self, path: str, data: bytes):
+ parent = self._parent(path)
+ if parent not in self.dirs:
+ raise FileNotFoundError(parent)
+ self.files[path] = data
+ return path
+
+ def rm(self, path: str, recursive: bool = False):
+ path = path.rstrip("/") or "/"
+ self.files.pop(path, None)
+ if path in self.dirs:
+ children = [
+ item
+ for item in [*self.dirs, *self.files]
+ if item != path and item.startswith(path.rstrip("/") + "/")
+ ]
+ if children and not recursive:
+ raise RuntimeError("directory not empty")
+ for child in children:
+ self.files.pop(child, None)
+ self.dirs.discard(child)
+ self.dirs.discard(path)
+ return {"message": "deleted"}
+
+ def ls(self, path: str):
+ path = path.rstrip("/") or "/"
+ prefix = path.rstrip("/") + "/"
+ names: dict[str, bool] = {}
+ for item in self.dirs:
+ if item == path or not item.startswith(prefix):
+ continue
+ rest = item[len(prefix) :]
+ if "/" not in rest:
+ names[rest] = True
+ for item in self.files:
+ if not item.startswith(prefix):
+ continue
+ rest = item[len(prefix) :]
+ if "/" not in rest:
+ names[rest] = False
+ return [{"name": name, "isDir": is_dir} for name, is_dir in names.items()]
+
+
+def agfs_client():
+ agfs = MemoryAgfs()
+ agfs.mkdir("/local")
+ agfs.mkdir("/local/default")
+ return agfs
+
+
+def test_dir(agfs_client):
+ path = f"/local/default/transaction-{uuid.uuid4().hex}"
+ agfs_client.mkdir(path)
+ return path
diff --git a/tests/transaction/test_concurrent_lock.py b/tests/transaction/test_concurrent_lock.py
index 76506c625..a853cb253 100644
--- a/tests/transaction/test_concurrent_lock.py
+++ b/tests/transaction/test_concurrent_lock.py
@@ -6,19 +6,19 @@ import asyncio
import uuid
from openviking.storage.transaction.lock_handle import LockHandle
-from openviking.storage.transaction.path_lock import PathLock
+from openviking.storage.transaction.path_lock import PathLockEngine
class TestConcurrentLock:
- async def test_point_mutual_exclusion_same_path(self, agfs_client, test_dir):
- """两个任务竞争同一路径的 POINT 锁,均最终成功(串行执行)。"""
- lock = PathLock(agfs_client)
+ async def test_exact_mutual_exclusion_same_path(self, agfs_client, test_dir):
+ """两个任务竞争同一路径的 EXACT 锁,均最终成功(串行执行)。"""
+ lock = PathLockEngine(agfs_client)
results = {}
async def holder(tx_id):
tx = LockHandle(id=tx_id)
- ok = await lock.acquire_point(test_dir, tx, timeout=5.0)
+ ok = await lock.acquire_exact_path(test_dir, tx, timeout=5.0)
if ok:
await asyncio.sleep(0.3)
await lock.release(tx)
@@ -33,20 +33,20 @@ class TestConcurrentLock:
assert results["tx-conc-1"] is True
assert results["tx-conc-2"] is True
- async def test_subtree_blocks_concurrent_point_child(self, agfs_client, test_dir):
- """SUBTREE on parent 持锁期间,子目录的 POINT 被阻塞,释放后成功。"""
+ async def test_tree_blocks_concurrent_exact_child(self, agfs_client, test_dir):
+ """TreeLock on parent 持锁期间,子目录的 ExactPathLock 被阻塞,释放后成功。"""
child = f"{test_dir}/child-{uuid.uuid4().hex}"
agfs_client.mkdir(child)
- lock = PathLock(agfs_client)
+ lock = PathLockEngine(agfs_client)
parent_acquired = asyncio.Event()
parent_released = asyncio.Event()
child_result = {}
async def parent_holder():
- tx = LockHandle(id="tx-sub-parent")
- ok = await lock.acquire_subtree(test_dir, tx, timeout=5.0)
+ tx = LockHandle(id="tx-tree-parent")
+ ok = await lock.acquire_tree(test_dir, tx, timeout=5.0)
assert ok is True
parent_acquired.set()
await asyncio.sleep(0.5)
@@ -55,8 +55,8 @@ class TestConcurrentLock:
async def child_worker():
await parent_acquired.wait()
- tx = LockHandle(id="tx-sub-child")
- ok = await lock.acquire_point(child, tx, timeout=5.0)
+ tx = LockHandle(id="tx-tree-child")
+ ok = await lock.acquire_exact_path(child, tx, timeout=5.0)
child_result["ok"] = ok
child_result["after_release"] = parent_released.is_set()
if ok:
@@ -68,12 +68,12 @@ class TestConcurrentLock:
# Child should succeed only after parent released
assert child_result["after_release"] is True
- async def test_point_child_blocks_concurrent_subtree_parent(self, agfs_client, test_dir):
- """POINT on child 持锁期间,父目录的 SUBTREE 被阻塞,释放后成功。"""
+ async def test_exact_child_blocks_concurrent_tree_parent(self, agfs_client, test_dir):
+ """ExactPathLock on child 持锁期间,父目录的 TreeLock 被阻塞,释放后成功。"""
child = f"{test_dir}/child-{uuid.uuid4().hex}"
agfs_client.mkdir(child)
- lock = PathLock(agfs_client)
+ lock = PathLockEngine(agfs_client)
child_acquired = asyncio.Event()
child_released = asyncio.Event()
@@ -81,7 +81,7 @@ class TestConcurrentLock:
async def child_holder():
tx = LockHandle(id="tx-rev-child")
- ok = await lock.acquire_point(child, tx, timeout=5.0)
+ ok = await lock.acquire_exact_path(child, tx, timeout=5.0)
assert ok is True
child_acquired.set()
await asyncio.sleep(0.5)
@@ -91,7 +91,7 @@ class TestConcurrentLock:
async def parent_worker():
await child_acquired.wait()
tx = LockHandle(id="tx-rev-parent")
- ok = await lock.acquire_subtree(test_dir, tx, timeout=5.0)
+ ok = await lock.acquire_tree(test_dir, tx, timeout=5.0)
parent_result["ok"] = ok
parent_result["after_release"] = child_released.is_set()
if ok:
diff --git a/tests/transaction/test_e2e.py b/tests/transaction/test_e2e.py
index 6cdf92cb9..0e6dd4453 100644
--- a/tests/transaction/test_e2e.py
+++ b/tests/transaction/test_e2e.py
@@ -2,7 +2,7 @@
# SPDX-License-Identifier: AGPL-3.0
"""End-to-end lock tests using real AGFS backend.
-These tests exercise LockContext -> LockManager -> PathLock -> AGFS,
+These tests exercise LockContext -> LockManager -> PathLockEngine -> AGFS,
verifying the acquire -> operate -> release lifecycle.
"""
@@ -33,9 +33,8 @@ def lock_manager(agfs_client):
class TestLockContextCommit:
async def test_lock_acquired_and_released(self, agfs_client, lock_manager, test_dir):
"""Lock is held inside the context and released after exit."""
- lock_path = f"{test_dir}/{LOCK_FILE_NAME}"
-
- async with LockContext(lock_manager, [test_dir], lock_mode="point"):
+ async with LockContext(lock_manager, [test_dir], lock_mode="exact") as handle:
+ lock_path = handle.locks[0]
token = agfs_client.cat(lock_path)
assert token is not None
@@ -45,7 +44,7 @@ class TestLockContextCommit:
"""Files written inside a lock context persist."""
file_path = f"{test_dir}/committed-file.txt"
- async with LockContext(lock_manager, [test_dir], lock_mode="point"):
+ async with LockContext(lock_manager, [test_dir], lock_mode="exact"):
agfs_client.write(file_path, b"committed data")
content = agfs_client.cat(file_path)
@@ -55,53 +54,55 @@ class TestLockContextCommit:
class TestLockContextException:
async def test_lock_released_on_exception(self, agfs_client, lock_manager, test_dir):
"""Lock is released even when an exception occurs inside the context."""
- lock_path = f"{test_dir}/{LOCK_FILE_NAME}"
+ lock_path = ""
with pytest.raises(RuntimeError):
- async with LockContext(lock_manager, [test_dir], lock_mode="point"):
+ async with LockContext(lock_manager, [test_dir], lock_mode="exact") as handle:
+ lock_path = handle.locks[0]
token = agfs_client.cat(lock_path)
assert token is not None
raise RuntimeError("simulated failure")
+ assert lock_path
assert _lock_file_gone(agfs_client, lock_path)
async def test_exception_not_swallowed(self, agfs_client, lock_manager, test_dir):
"""Exceptions propagate through the context manager."""
with pytest.raises(ValueError, match="test error"):
- async with LockContext(lock_manager, [test_dir], lock_mode="point"):
+ async with LockContext(lock_manager, [test_dir], lock_mode="exact"):
raise ValueError("test error")
class TestLockContextMv:
async def test_mv_lock_acquires_both_paths(self, agfs_client, lock_manager, test_dir):
- """mv lock mode acquires SUBTREE on both source and destination."""
+ """mv lock mode acquires tree source and exact destination locks."""
src = f"{test_dir}/mv-src-{uuid.uuid4().hex}"
dst = f"{test_dir}/mv-dst-{uuid.uuid4().hex}"
agfs_client.mkdir(src)
agfs_client.mkdir(dst)
- async with LockContext(lock_manager, [src], lock_mode="mv", mv_dst_parent_path=dst):
+ async with LockContext(lock_manager, [src], lock_mode="mv", mv_dst_path=dst):
src_token = agfs_client.cat(f"{src}/{LOCK_FILE_NAME}")
dst_token = agfs_client.cat(f"{dst}/{LOCK_FILE_NAME}")
src_token_str = src_token.decode("utf-8") if isinstance(src_token, bytes) else src_token
dst_token_str = dst_token.decode("utf-8") if isinstance(dst_token, bytes) else dst_token
- assert ":S" in src_token_str
- assert ":S" in dst_token_str
+ assert ":T" in src_token_str
+ assert ":E" in dst_token_str
for path in [f"{src}/{LOCK_FILE_NAME}", f"{dst}/{LOCK_FILE_NAME}"]:
assert _lock_file_gone(agfs_client, path)
-class TestLockContextSubtree:
- async def test_subtree_lock_and_release(self, agfs_client, lock_manager, test_dir):
- """Subtree lock is acquired and released."""
- target = f"{test_dir}/sub-{uuid.uuid4().hex}"
+class TestLockContextTree:
+ async def test_tree_lock_and_release(self, agfs_client, lock_manager, test_dir):
+ """Tree lock is acquired and released."""
+ target = f"{test_dir}/tree-{uuid.uuid4().hex}"
agfs_client.mkdir(target)
- async with LockContext(lock_manager, [target], lock_mode="subtree"):
+ async with LockContext(lock_manager, [target], lock_mode="tree"):
token = agfs_client.cat(f"{target}/{LOCK_FILE_NAME}")
token_str = token.decode("utf-8") if isinstance(token, bytes) else token
- assert ":S" in token_str
+ assert ":T" in token_str
assert _lock_file_gone(agfs_client, f"{target}/{LOCK_FILE_NAME}")
@@ -110,7 +111,7 @@ class TestSequentialLocks:
async def test_sequential_locks_on_same_path(self, agfs_client, lock_manager, test_dir):
"""Multiple sequential lock contexts on the same path succeed."""
for i in range(3):
- async with LockContext(lock_manager, [test_dir], lock_mode="point"):
+ async with LockContext(lock_manager, [test_dir], lock_mode="exact"):
agfs_client.write(f"{test_dir}/f{i}.txt", f"data-{i}".encode())
for i in range(3):
@@ -118,8 +119,8 @@ class TestSequentialLocks:
assert content == f"data-{i}".encode()
async def test_lock_acquisition_failure(self, agfs_client, lock_manager, test_dir):
- """LockContext raises LockAcquisitionError for nonexistent path."""
+ """LockContext raises LockAcquisitionError for unsupported lock mode."""
nonexistent = f"{test_dir}/nonexistent-{uuid.uuid4().hex}"
with pytest.raises(LockAcquisitionError):
- async with LockContext(lock_manager, [nonexistent], lock_mode="point"):
+ async with LockContext(lock_manager, [nonexistent], lock_mode="unknown"):
pass
diff --git a/tests/transaction/test_exact_path_lock.py b/tests/transaction/test_exact_path_lock.py
new file mode 100644
index 000000000..7cf8f6f6a
--- /dev/null
+++ b/tests/transaction/test_exact_path_lock.py
@@ -0,0 +1,174 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Tests for ExactPathLock semantics."""
+
+import pytest
+
+from openviking.storage.transaction.lock_handle import LockHandle
+from openviking.storage.transaction.lock_manager import LockManager
+from openviking.storage.transaction.path_lock import EXACT_LOCK_FILE_PREFIX, PathLockEngine
+
+
+class _MemoryAgfs:
+ def __init__(self):
+ self.dirs = {"/"}
+ self.files: dict[str, bytes] = {}
+
+ def _parent(self, path: str) -> str:
+ path = path.rstrip("/")
+ if "/" not in path:
+ return "/"
+ parent = path.rsplit("/", 1)[0]
+ return parent or "/"
+
+ def stat(self, path: str):
+ path = path.rstrip("/") or "/"
+ if path in self.dirs:
+ return {"name": path.rsplit("/", 1)[-1], "isDir": True}
+ if path in self.files:
+ return {"name": path.rsplit("/", 1)[-1], "isDir": False}
+ raise FileNotFoundError(path)
+
+ def mkdir(self, path: str):
+ path = path.rstrip("/") or "/"
+ parent = self._parent(path)
+ if parent not in self.dirs:
+ raise FileNotFoundError(parent)
+ self.dirs.add(path)
+ return {"message": "created"}
+
+ def read(self, path: str):
+ if path not in self.files:
+ raise FileNotFoundError(path)
+ return self.files[path]
+
+ def write(self, path: str, data: bytes):
+ parent = self._parent(path)
+ if parent not in self.dirs:
+ raise FileNotFoundError(parent)
+ self.files[path] = data
+ return path
+
+ def rm(self, path: str, recursive: bool = False):
+ path = path.rstrip("/") or "/"
+ self.files.pop(path, None)
+ if path in self.dirs:
+ children = [
+ item
+ for item in [*self.dirs, *self.files]
+ if item != path and item.startswith(path.rstrip("/") + "/")
+ ]
+ if children and not recursive:
+ raise RuntimeError("directory not empty")
+ for child in children:
+ self.files.pop(child, None)
+ self.dirs.discard(child)
+ self.dirs.discard(path)
+ return {"message": "deleted"}
+
+ def ls(self, path: str):
+ path = path.rstrip("/") or "/"
+ prefix = path.rstrip("/") + "/"
+ names: dict[str, bool] = {}
+ for item in self.dirs:
+ if item == path or not item.startswith(prefix):
+ continue
+ rest = item[len(prefix) :]
+ if "/" not in rest:
+ names[rest] = True
+ for item in self.files:
+ if not item.startswith(prefix):
+ continue
+ rest = item[len(prefix) :]
+ if "/" not in rest:
+ names[rest] = False
+ return [{"name": name, "isDir": is_dir} for name, is_dir in names.items()]
+
+
+def _agfs_with_docs_dir() -> _MemoryAgfs:
+ agfs = _MemoryAgfs()
+ agfs.mkdir("/local")
+ agfs.mkdir("/local/default")
+ agfs.mkdir("/local/default/resources")
+ agfs.mkdir("/local/default/resources/docs")
+ return agfs
+
+
+async def test_exact_path_lock_allows_sibling_paths():
+ agfs = _agfs_with_docs_dir()
+ lock = PathLockEngine(agfs)
+ first = LockHandle(id="exact-a")
+ second = LockHandle(id="exact-b")
+
+ assert await lock.acquire_exact_path("/local/default/resources/docs/a.md", first)
+ assert await lock.acquire_exact_path("/local/default/resources/docs/b.md", second)
+
+ await lock.release(first)
+ await lock.release(second)
+
+
+async def test_exact_path_lock_blocks_same_path_without_creating_target():
+ agfs = _agfs_with_docs_dir()
+ lock = PathLockEngine(agfs)
+ first = LockHandle(id="exact-a")
+ second = LockHandle(id="exact-a-blocked")
+ target = "/local/default/resources/docs/a.md"
+
+ assert await lock.acquire_exact_path(target, first)
+ assert not await lock.acquire_exact_path(target, second, timeout=0.0)
+
+ assert lock.is_locked(target)
+ with pytest.raises(FileNotFoundError):
+ agfs.stat(target)
+ assert any(path.rsplit("/", 1)[-1].startswith(EXACT_LOCK_FILE_PREFIX) for path in agfs.files)
+
+ await lock.release(first)
+ assert not lock.is_locked(target)
+
+
+async def test_exact_path_lock_conflicts_with_tree_lock():
+ agfs = _agfs_with_docs_dir()
+ lock = PathLockEngine(agfs)
+ exact = LockHandle(id="exact-a")
+ tree = LockHandle(id="tree-docs")
+ target = "/local/default/resources/docs/a.md"
+ docs = "/local/default/resources/docs"
+
+ assert await lock.acquire_exact_path(target, exact)
+ assert not await lock.acquire_tree(docs, tree, timeout=0.0)
+
+ await lock.release(exact)
+ assert await lock.acquire_tree(docs, tree, timeout=0.0)
+
+ blocked = LockHandle(id="exact-blocked-by-tree")
+ assert not await lock.acquire_exact_path(target, blocked, timeout=0.0)
+
+ await lock.release(tree)
+
+
+async def test_exact_tree_batch_acquires_exact_and_tree_locks():
+ agfs = _agfs_with_docs_dir()
+ agfs.mkdir("/local/default/resources/docs/events")
+ manager = LockManager(agfs=agfs, lock_timeout=0.0, lock_expire=300.0)
+ handle = manager.create_handle()
+
+ assert await manager.acquire_exact_tree_batch(
+ handle,
+ exact_paths=["/local/default/resources/docs/profile.md"],
+ tree_paths=["/local/default/resources/docs/events"],
+ )
+ assert len(handle.locks) == 2
+
+ blocked = manager.create_handle()
+ assert not await manager.acquire_exact_tree_batch(
+ blocked,
+ exact_paths=["/local/default/resources/docs/profile.md"],
+ tree_paths=[],
+ )
+
+ await manager.release(handle)
+ await manager.release(blocked)
diff --git a/tests/transaction/test_lock_context.py b/tests/transaction/test_lock_context.py
index 65bc1f06c..263819222 100644
--- a/tests/transaction/test_lock_context.py
+++ b/tests/transaction/test_lock_context.py
@@ -25,39 +25,40 @@ def lm(agfs_client):
return LockManager(agfs=agfs_client, lock_timeout=1.0, lock_expire=1.0)
-class TestLockContextPoint:
- async def test_point_lock_lifecycle(self, agfs_client, lm, test_dir):
- lock_path = f"{test_dir}/{LOCK_FILE_NAME}"
-
- async with LockContext(lm, [test_dir], lock_mode="point") as handle:
+class TestLockContextExact:
+ async def test_exact_lock_lifecycle(self, agfs_client, lm, test_dir):
+ async with LockContext(lm, [test_dir], lock_mode="exact") as handle:
assert handle is not None
+ lock_path = handle.locks[0]
token = agfs_client.cat(lock_path)
assert token is not None
assert _lock_file_gone(agfs_client, lock_path)
async def test_lock_released_on_exception(self, agfs_client, lm, test_dir):
- lock_path = f"{test_dir}/{LOCK_FILE_NAME}"
+ lock_path = ""
with pytest.raises(RuntimeError):
- async with LockContext(lm, [test_dir], lock_mode="point"):
+ async with LockContext(lm, [test_dir], lock_mode="exact") as handle:
+ lock_path = handle.locks[0]
assert agfs_client.cat(lock_path) is not None
raise RuntimeError("fail")
+ assert lock_path
assert _lock_file_gone(agfs_client, lock_path)
async def test_exception_propagates(self, lm, test_dir):
with pytest.raises(ValueError, match="test"):
- async with LockContext(lm, [test_dir], lock_mode="point"):
+ async with LockContext(lm, [test_dir], lock_mode="exact"):
raise ValueError("test")
-class TestLockContextSubtree:
- async def test_subtree_lock(self, agfs_client, lm, test_dir):
- async with LockContext(lm, [test_dir], lock_mode="subtree"):
+class TestLockContextTree:
+ async def test_tree_lock(self, agfs_client, lm, test_dir):
+ async with LockContext(lm, [test_dir], lock_mode="tree"):
token = agfs_client.cat(f"{test_dir}/{LOCK_FILE_NAME}")
token_str = token.decode("utf-8") if isinstance(token, bytes) else token
- assert ":S" in token_str
+ assert ":T" in token_str
class TestLockContextMv:
@@ -67,38 +68,38 @@ class TestLockContextMv:
agfs_client.mkdir(src)
agfs_client.mkdir(dst)
- async with LockContext(lm, [src], lock_mode="mv", mv_dst_parent_path=dst) as handle:
+ async with LockContext(lm, [src], lock_mode="mv", mv_dst_path=dst) as handle:
assert len(handle.locks) == 2
class TestLockContextFailure:
- async def test_nonexistent_path_raises(self, lm):
+ async def test_unsupported_lock_mode_raises(self, lm):
with pytest.raises(LockAcquisitionError):
- async with LockContext(lm, ["/local/nonexistent-xyz"], lock_mode="point"):
+ async with LockContext(lm, ["/local/nonexistent-xyz"], lock_mode="unknown"):
pass
async def test_handle_cleaned_up_on_failure(self, lm):
with pytest.raises(LockAcquisitionError):
- async with LockContext(lm, ["/local/nonexistent-xyz"], lock_mode="point"):
+ async with LockContext(lm, ["/local/nonexistent-xyz"], lock_mode="unknown"):
pass
assert len(lm.get_active_handles()) == 0
class TestLockContextExternalHandle:
- async def test_external_handle_reuses_existing_subtree_lock(self, agfs_client, lm, test_dir):
+ async def test_external_handle_reuses_existing_tree_lock(self, agfs_client, lm, test_dir):
lock_path = f"{test_dir}/{LOCK_FILE_NAME}"
- async with LockContext(lm, [test_dir], lock_mode="subtree") as handle:
+ async with LockContext(lm, [test_dir], lock_mode="tree") as handle:
before = agfs_client.cat(lock_path)
before_token = before.decode("utf-8") if isinstance(before, bytes) else before
- assert ":S" in before_token
+ assert ":T" in before_token
- async with LockContext(lm, [test_dir], lock_mode="point", handle=handle):
+ async with LockContext(lm, [test_dir], lock_mode="exact", handle=handle):
current = agfs_client.cat(lock_path)
current_token = current.decode("utf-8") if isinstance(current, bytes) else current
assert current_token == before_token
- assert ":S" in current_token
+ assert ":T" in current_token
still_owned = agfs_client.cat(lock_path)
still_owned_token = (
diff --git a/tests/transaction/test_lock_manager.py b/tests/transaction/test_lock_manager.py
index c6147fc89..06e38fcda 100644
--- a/tests/transaction/test_lock_manager.py
+++ b/tests/transaction/test_lock_manager.py
@@ -26,26 +26,26 @@ def lm(agfs_client):
class TestLockManagerBasic:
- async def test_create_handle_and_acquire_point(self, agfs_client, lm, test_dir):
+ async def test_create_handle_and_acquire_exact_path(self, agfs_client, lm, test_dir):
handle = lm.create_handle()
- ok = await lm.acquire_point(handle, test_dir)
+ ok = await lm.acquire_exact_path(handle, test_dir)
assert ok is True
- lock_path = f"{test_dir}/{LOCK_FILE_NAME}"
+ lock_path = handle.locks[0]
content = agfs_client.cat(lock_path)
assert content is not None
await lm.release(handle)
assert _lock_file_gone(agfs_client, lock_path)
- async def test_acquire_subtree(self, agfs_client, lm, test_dir):
+ async def test_acquire_tree(self, agfs_client, lm, test_dir):
handle = lm.create_handle()
- ok = await lm.acquire_subtree(handle, test_dir)
+ ok = await lm.acquire_tree(handle, test_dir)
assert ok is True
token = agfs_client.cat(f"{test_dir}/{LOCK_FILE_NAME}")
token_str = token.decode("utf-8") if isinstance(token, bytes) else token
- assert ":S" in token_str
+ assert ":T" in token_str
await lm.release(handle)
@@ -65,9 +65,10 @@ class TestLockManagerBasic:
async def test_release_removes_from_active(self, lm, test_dir):
handle = lm.create_handle()
+
+ await lm.acquire_exact_path(handle, test_dir)
assert handle.id in lm.get_active_handles()
- await lm.acquire_point(handle, test_dir)
await lm.release(handle)
assert handle.id not in lm.get_active_handles()
@@ -75,19 +76,21 @@ class TestLockManagerBasic:
async def test_stop_releases_all(self, agfs_client, lm, test_dir):
h1 = lm.create_handle()
h2 = lm.create_handle()
- await lm.acquire_point(h1, test_dir)
+ await lm.acquire_exact_path(h1, test_dir)
sub = f"{test_dir}/sub-{uuid.uuid4().hex}"
agfs_client.mkdir(sub)
- await lm.acquire_point(h2, sub)
+ await lm.acquire_exact_path(h2, sub)
await lm.stop()
assert len(lm.get_active_handles()) == 0
- async def test_nonexistent_path_fails(self, lm):
+ async def test_exact_path_allows_missing_target(self, lm):
handle = lm.create_handle()
- ok = await lm.acquire_point(handle, "/local/nonexistent-xyz")
- assert ok is False
+ ok = await lm.acquire_exact_path(handle, "/local/nonexistent-xyz")
+ assert ok is True
+
+ await lm.release(handle)
async def test_recover_pending_redo_preserves_cancelled_error(self, lm):
lm._redo_log = MagicMock()
diff --git a/tests/transaction/test_lock_reentrancy_unit.py b/tests/transaction/test_lock_reentrancy_unit.py
index 44687e7bc..39986e732 100644
--- a/tests/transaction/test_lock_reentrancy_unit.py
+++ b/tests/transaction/test_lock_reentrancy_unit.py
@@ -9,7 +9,7 @@ import pytest
from openviking.storage.transaction.lock_context import LockContext
from openviking.storage.transaction.lock_manager import LockManager
-from openviking.storage.transaction.path_lock import LOCK_FILE_NAME, PathLock
+from openviking.storage.transaction.path_lock import LOCK_FILE_NAME, PathLockEngine
class _FakeAGFS:
@@ -87,50 +87,50 @@ class _FakeAGFS:
@pytest.mark.asyncio
-async def test_path_lock_reuses_same_owner_subtree_without_overwriting_token():
+async def test_path_lock_reuses_same_owner_tree_without_overwriting_token():
agfs = _FakeAGFS()
agfs.mkdir("/root")
- lock = PathLock(agfs)
+ lock = PathLockEngine(agfs)
handle = LockManager(agfs).create_handle()
- assert await lock.acquire_subtree("/root", handle, timeout=0.1) is True
+ assert await lock.acquire_tree("/root", handle, timeout=0.1) is True
lock_path = f"/root/{LOCK_FILE_NAME}"
before = agfs.read(lock_path).decode("utf-8")
- assert before.endswith(":S")
+ assert before.endswith(":T")
- assert await lock.acquire_point("/root", handle, timeout=0.1) is True
+ assert await lock.acquire_exact_path("/root", handle, timeout=0.1) is True
after = agfs.read(lock_path).decode("utf-8")
assert after == before
- assert after.endswith(":S")
+ assert after.endswith(":T")
@pytest.mark.asyncio
-async def test_path_lock_reuses_ancestor_subtree_without_creating_child_lock():
+async def test_path_lock_reuses_ancestor_tree_without_creating_child_lock():
agfs = _FakeAGFS()
agfs.mkdir("/root")
agfs.mkdir("/root/child")
- lock = PathLock(agfs)
+ lock = PathLockEngine(agfs)
handle = LockManager(agfs).create_handle()
- assert await lock.acquire_subtree("/root", handle, timeout=0.1) is True
- assert await lock.acquire_point("/root/child", handle, timeout=0.1) is True
+ assert await lock.acquire_tree("/root", handle, timeout=0.1) is True
+ assert await lock.acquire_exact_path("/root/child", handle, timeout=0.1) is True
with pytest.raises(FileNotFoundError):
agfs.read(f"/root/child/{LOCK_FILE_NAME}")
@pytest.mark.asyncio
-async def test_lock_context_with_external_handle_keeps_outer_subtree_lock():
+async def test_lock_context_with_external_handle_keeps_outer_tree_lock():
agfs = _FakeAGFS()
agfs.mkdir("/root")
lm = LockManager(agfs=agfs, lock_timeout=0.1, lock_expire=60.0)
lock_path = f"/root/{LOCK_FILE_NAME}"
- async with LockContext(lm, ["/root"], lock_mode="subtree") as handle:
+ async with LockContext(lm, ["/root"], lock_mode="tree") as handle:
before = agfs.read(lock_path).decode("utf-8")
- assert before.endswith(":S")
+ assert before.endswith(":T")
- async with LockContext(lm, ["/root"], lock_mode="point", handle=handle):
+ async with LockContext(lm, ["/root"], lock_mode="exact", handle=handle):
current = agfs.read(lock_path).decode("utf-8")
assert current == before
diff --git a/tests/transaction/test_path_lock.py b/tests/transaction/test_path_lock.py
index 575b5376f..6a6897318 100644
--- a/tests/transaction/test_path_lock.py
+++ b/tests/transaction/test_path_lock.py
@@ -7,43 +7,56 @@ from unittest.mock import AsyncMock, MagicMock
from openviking.storage.transaction.lock_handle import LockHandle
from openviking.storage.transaction.path_lock import (
+ EXACT_LOCK_FILE_PREFIX,
LOCK_FILE_NAME,
- LOCK_TYPE_POINT,
- LOCK_TYPE_SUBTREE,
- PathLock,
+ LOCK_TYPE_EXACT,
+ LOCK_TYPE_TREE,
+ PathLockEngine,
_make_fencing_token,
_parse_fencing_token,
)
+def _single_lock_path(handle: LockHandle) -> str:
+ assert len(handle.locks) == 1
+ return handle.locks[0]
+
+
class TestFencingToken:
def test_make_parse_roundtrip(self):
token = _make_fencing_token("tx-123")
tx_id, ts, lock_type = _parse_fencing_token(token)
assert tx_id == "tx-123"
assert ts > 0
- assert lock_type == LOCK_TYPE_POINT
+ assert lock_type == LOCK_TYPE_EXACT
- def test_make_parse_subtree_roundtrip(self):
- token = _make_fencing_token("tx-456", LOCK_TYPE_SUBTREE)
+ def test_make_parse_tree_roundtrip(self):
+ token = _make_fencing_token("tx-456", LOCK_TYPE_TREE)
tx_id, ts, lock_type = _parse_fencing_token(token)
assert tx_id == "tx-456"
assert ts > 0
- assert lock_type == LOCK_TYPE_SUBTREE
+ assert lock_type == LOCK_TYPE_TREE
+
+ def test_parse_previous_tree_tokens(self):
+ for previous_type in ("P", "S"):
+ tx_id, ts, lock_type = _parse_fencing_token(f"tx-old:123:{previous_type}")
+ assert tx_id == "tx-old"
+ assert ts == 123
+ assert lock_type == LOCK_TYPE_TREE
- def test_parse_legacy_format_two_part(self):
- """Legacy two-part token "{tx_id}:{ts}" defaults to POINT."""
+ def test_parse_untyped_token_defaults_to_stale_exact(self):
+ """Untyped tokens are treated as stale EXACT locks."""
tx_id, ts, lock_type = _parse_fencing_token("tx-old:1234567890")
- assert tx_id == "tx-old"
- assert ts == 1234567890
- assert lock_type == LOCK_TYPE_POINT
+ assert tx_id == "tx-old:1234567890"
+ assert ts == 0
+ assert lock_type == LOCK_TYPE_EXACT
- def test_parse_legacy_format_plain(self):
- """Plain tx_id (no colon) defaults to ts=0, lock_type=POINT."""
+ def test_parse_plain_token_defaults_to_exact(self):
+ """Plain tx_id (no colon) defaults to ts=0, lock_type=EXACT."""
tx_id, ts, lock_type = _parse_fencing_token("tx-bare")
assert tx_id == "tx-bare"
assert ts == 0
- assert lock_type == LOCK_TYPE_POINT
+ assert lock_type == LOCK_TYPE_EXACT
def test_tokens_are_unique(self):
t1 = _make_fencing_token("tx-1")
@@ -56,20 +69,20 @@ class TestPathLockStale:
def test_is_lock_stale_no_file(self):
agfs = MagicMock()
agfs.read.side_effect = Exception("not found")
- lock = PathLock(agfs)
+ lock = PathLockEngine(agfs)
assert lock.is_lock_stale("/test/.path.ovlock") is True
- def test_is_lock_stale_legacy_token(self):
+ def test_is_lock_stale_for_plain_token(self):
agfs = MagicMock()
agfs.read.return_value = b"tx-old-format"
- lock = PathLock(agfs)
+ lock = PathLockEngine(agfs)
assert lock.is_lock_stale("/test/.path.ovlock") is True
def test_is_lock_stale_recent_token(self):
agfs = MagicMock()
token = _make_fencing_token("tx-1")
agfs.read.return_value = token.encode("utf-8")
- lock = PathLock(agfs)
+ lock = PathLockEngine(agfs)
assert lock.is_lock_stale("/test/.path.ovlock", expire_seconds=300.0) is False
@@ -88,40 +101,40 @@ class TestPathLockIsLocked:
def test_is_locked_no_lock(self):
agfs = self._agfs_with_locks({})
- lock = PathLock(agfs)
+ lock = PathLockEngine(agfs)
assert lock.is_locked("/local/u/foo") is False
- def test_is_locked_self_point_lock(self):
- token = _make_fencing_token("tx-1", LOCK_TYPE_POINT)
+ def test_is_locked_self_exact_lock(self):
+ token = _make_fencing_token("tx-1", LOCK_TYPE_EXACT)
agfs = self._agfs_with_locks({"/local/u/foo/.path.ovlock": token.encode("utf-8")})
- lock = PathLock(agfs)
+ lock = PathLockEngine(agfs)
assert lock.is_locked("/local/u/foo") is True
- def test_is_locked_ancestor_subtree_lock(self):
- token = _make_fencing_token("tx-1", LOCK_TYPE_SUBTREE)
+ def test_is_locked_ancestor_tree_lock(self):
+ token = _make_fencing_token("tx-1", LOCK_TYPE_TREE)
agfs = self._agfs_with_locks({"/local/u/.path.ovlock": token.encode("utf-8")})
- lock = PathLock(agfs)
+ lock = PathLockEngine(agfs)
assert lock.is_locked("/local/u/foo/bar") is True
- def test_is_locked_ancestor_point_lock_does_not_propagate(self):
- """An ancestor POINT lock must not affect descendants -- POINT locks only the path itself."""
- token = _make_fencing_token("tx-1", LOCK_TYPE_POINT)
+ def test_is_locked_ancestor_exact_lock_does_not_propagate(self):
+ """An ancestor EXACT lock must not affect descendants -- EXACT locks only the path itself."""
+ token = _make_fencing_token("tx-1", LOCK_TYPE_EXACT)
agfs = self._agfs_with_locks({"/local/u/.path.ovlock": token.encode("utf-8")})
- lock = PathLock(agfs)
+ lock = PathLockEngine(agfs)
assert lock.is_locked("/local/u/foo/bar") is False
def test_is_locked_ignores_stale_by_default(self):
old_ts = time.time_ns() - int(600 * 1e9) # 600s ago
- stale = f"tx-dead:{old_ts}:{LOCK_TYPE_POINT}".encode("utf-8")
+ stale = f"tx-dead:{old_ts}:{LOCK_TYPE_EXACT}".encode("utf-8")
agfs = self._agfs_with_locks({"/local/u/foo/.path.ovlock": stale})
- lock = PathLock(agfs, lock_expire=300.0)
+ lock = PathLockEngine(agfs, lock_expire=300.0)
assert lock.is_locked("/local/u/foo") is False
def test_is_locked_can_include_stale(self):
old_ts = time.time_ns() - int(600 * 1e9)
- stale = f"tx-dead:{old_ts}:{LOCK_TYPE_POINT}".encode("utf-8")
+ stale = f"tx-dead:{old_ts}:{LOCK_TYPE_EXACT}".encode("utf-8")
agfs = self._agfs_with_locks({"/local/u/foo/.path.ovlock": stale})
- lock = PathLock(agfs, lock_expire=300.0)
+ lock = PathLockEngine(agfs, lock_expire=300.0)
assert lock.is_locked("/local/u/foo", ignore_stale=False) is True
@@ -133,9 +146,9 @@ class TestPathLockOwnership:
failed_path = "/locks/failed/.path.ovlock"
tokens = {
- owned_path: _make_fencing_token("tx-1", LOCK_TYPE_POINT),
- lost_path: _make_fencing_token("tx-2", LOCK_TYPE_SUBTREE),
- failed_path: _make_fencing_token("tx-1", LOCK_TYPE_SUBTREE),
+ owned_path: _make_fencing_token("tx-1", LOCK_TYPE_EXACT),
+ lost_path: _make_fencing_token("tx-2", LOCK_TYPE_TREE),
+ failed_path: _make_fencing_token("tx-1", LOCK_TYPE_TREE),
}
agfs = MagicMock()
@@ -152,7 +165,7 @@ class TestPathLockOwnership:
agfs.read.side_effect = read_side_effect
agfs.write.side_effect = write_side_effect
- lock = PathLock(agfs)
+ lock = PathLockEngine(agfs)
tx = LockHandle(id="tx-1")
for lock_path in [owned_path, lost_path, missing_path, failed_path]:
tx.add_lock(lock_path)
@@ -168,13 +181,13 @@ class TestPathLockOwnership:
replaced_path = "/locks/replaced/.path.ovlock"
tokens = {
- owned_path: _make_fencing_token("tx-1", LOCK_TYPE_POINT),
- replaced_path: _make_fencing_token("tx-2", LOCK_TYPE_POINT),
+ owned_path: _make_fencing_token("tx-1", LOCK_TYPE_EXACT),
+ replaced_path: _make_fencing_token("tx-2", LOCK_TYPE_EXACT),
}
agfs = MagicMock()
agfs.read.side_effect = lambda lock_path: tokens[lock_path].encode("utf-8")
- lock = PathLock(agfs)
+ lock = PathLockEngine(agfs)
lock._remove_lock_file = AsyncMock(return_value=True)
tx = LockHandle(id="tx-1")
tx.add_lock(owned_path)
@@ -189,50 +202,111 @@ class TestPathLockOwnership:
class TestPathLockBehavior:
"""Behavioral tests using real AGFS backend."""
- async def test_acquire_point_creates_lock_file(self, agfs_client, test_dir):
- lock = PathLock(agfs_client)
- tx = LockHandle(id="tx-point-1")
+ async def test_acquire_exact_path_creates_lock_file(self, agfs_client, test_dir):
+ lock = PathLockEngine(agfs_client)
+ tx = LockHandle(id="tx-exact-1")
- ok = await lock.acquire_point(test_dir, tx, timeout=3.0)
+ ok = await lock.acquire_exact_path(test_dir, tx, timeout=3.0)
assert ok is True
- lock_path = f"{test_dir}/{LOCK_FILE_NAME}"
+ lock_path = _single_lock_path(tx)
+ assert lock_path == f"{test_dir}/{LOCK_FILE_NAME}"
content = agfs_client.cat(lock_path)
token = content.decode("utf-8") if isinstance(content, bytes) else content
- assert ":P" in token
- assert "tx-point-1" in token
+ assert ":E" in token
+ assert "tx-exact-1" in token
await lock.release(tx)
- async def test_acquire_subtree_creates_lock_file(self, agfs_client, test_dir):
- lock = PathLock(agfs_client)
- tx = LockHandle(id="tx-subtree-1")
+ async def test_acquire_tree_creates_lock_file(self, agfs_client, test_dir):
+ lock = PathLockEngine(agfs_client)
+ tx = LockHandle(id="tx-tree-1")
- ok = await lock.acquire_subtree(test_dir, tx, timeout=3.0)
+ ok = await lock.acquire_tree(test_dir, tx, timeout=3.0)
assert ok is True
lock_path = f"{test_dir}/{LOCK_FILE_NAME}"
content = agfs_client.cat(lock_path)
token = content.decode("utf-8") if isinstance(content, bytes) else content
- assert ":S" in token
- assert "tx-subtree-1" in token
+ assert ":T" in token
+ assert "tx-tree-1" in token
await lock.release(tx)
- async def test_acquire_point_dir_not_found(self, agfs_client):
- lock = PathLock(agfs_client)
+ async def test_acquire_tree_creates_missing_directory_after_conflict_check(
+ self, agfs_client, test_dir
+ ):
+ lock = PathLockEngine(agfs_client)
+ tx = LockHandle(id="tx-tree-missing")
+ target = f"{test_dir}/new-resource"
+
+ ok = await lock.acquire_tree(target, tx, timeout=3.0)
+ assert ok is True
+ assert _single_lock_path(tx) == f"{target}/{LOCK_FILE_NAME}"
+ assert agfs_client.stat(target).get("isDir") is True
+
+ await lock.release(tx)
+
+ async def test_tree_blocked_by_ancestor_tree_does_not_create_missing_directory(
+ self, agfs_client, test_dir
+ ):
+ lock = PathLockEngine(agfs_client)
+ parent = LockHandle(id="tx-parent-tree")
+ child = LockHandle(id="tx-child-tree")
+ target = f"{test_dir}/blocked-resource"
+
+ assert await lock.acquire_tree(test_dir, parent, timeout=3.0) is True
+ assert await lock.acquire_tree(target, child, timeout=0.0) is False
+
+ try:
+ agfs_client.stat(target)
+ raise AssertionError("missing TreeLock target should not be created on conflict")
+ except AssertionError:
+ raise
+ except Exception:
+ pass
+
+ await lock.release(parent)
+
+ async def test_acquire_exact_path_allows_missing_target(self, agfs_client):
+ lock = PathLockEngine(agfs_client)
tx = LockHandle(id="tx-no-dir")
- ok = await lock.acquire_point("/local/nonexistent-path-xyz", tx, timeout=0.5)
- assert ok is False
- assert len(tx.locks) == 0
+ ok = await lock.acquire_exact_path("/local/nonexistent-path-xyz", tx, timeout=0.5)
+ assert ok is True
+ assert len(tx.locks) == 1
+ assert tx.locks[0].rsplit("/", 1)[-1].startswith(EXACT_LOCK_FILE_PREFIX)
+
+ await lock.release(tx)
+
+ async def test_exact_blocked_by_ancestor_tree_does_not_create_missing_parent(
+ self, agfs_client, test_dir
+ ):
+ lock = PathLockEngine(agfs_client)
+ parent = LockHandle(id="tx-parent-tree")
+ child = LockHandle(id="tx-child-exact")
+ missing_parent = f"{test_dir}/blocked-dir"
+ target = f"{missing_parent}/file.md"
+
+ assert await lock.acquire_tree(test_dir, parent, timeout=3.0) is True
+ assert await lock.acquire_exact_path(target, child, timeout=0.0) is False
+
+ try:
+ agfs_client.stat(missing_parent)
+ raise AssertionError("missing exact-lock parent should not be created on conflict")
+ except AssertionError:
+ raise
+ except Exception:
+ pass
+
+ await lock.release(parent)
async def test_release_removes_lock_file(self, agfs_client, test_dir):
- lock = PathLock(agfs_client)
+ lock = PathLockEngine(agfs_client)
tx = LockHandle(id="tx-release-1")
- await lock.acquire_point(test_dir, tx, timeout=3.0)
- lock_path = f"{test_dir}/{LOCK_FILE_NAME}"
+ await lock.acquire_exact_path(test_dir, tx, timeout=3.0)
+ lock_path = _single_lock_path(tx)
await lock.release(tx)
@@ -246,58 +320,58 @@ class TestPathLockBehavior:
pass # Expected: file not found
async def test_sequential_acquire_works(self, agfs_client, test_dir):
- lock = PathLock(agfs_client)
+ lock = PathLockEngine(agfs_client)
tx1 = LockHandle(id="tx-seq-1")
- ok1 = await lock.acquire_point(test_dir, tx1, timeout=3.0)
+ ok1 = await lock.acquire_exact_path(test_dir, tx1, timeout=3.0)
assert ok1 is True
await lock.release(tx1)
tx2 = LockHandle(id="tx-seq-2")
- ok2 = await lock.acquire_point(test_dir, tx2, timeout=3.0)
+ ok2 = await lock.acquire_exact_path(test_dir, tx2, timeout=3.0)
assert ok2 is True
await lock.release(tx2)
- async def test_point_blocked_by_ancestor_subtree(self, agfs_client, test_dir):
- """POINT on child blocked while ancestor holds SUBTREE lock."""
+ async def test_exact_blocked_by_ancestor_tree(self, agfs_client, test_dir):
+ """EXACT on child blocked while ancestor holds TreeLock."""
import uuid as _uuid
child = f"{test_dir}/child-{_uuid.uuid4().hex}"
agfs_client.mkdir(child)
- lock = PathLock(agfs_client)
- tx_parent = LockHandle(id="tx-parent-subtree")
- ok = await lock.acquire_subtree(test_dir, tx_parent, timeout=3.0)
+ lock = PathLockEngine(agfs_client)
+ tx_parent = LockHandle(id="tx-parent-tree")
+ ok = await lock.acquire_tree(test_dir, tx_parent, timeout=3.0)
assert ok is True
- tx_child = LockHandle(id="tx-child-point")
- blocked = await lock.acquire_point(child, tx_child, timeout=0.5)
+ tx_child = LockHandle(id="tx-child-exact")
+ blocked = await lock.acquire_exact_path(child, tx_child, timeout=0.5)
assert blocked is False
await lock.release(tx_parent)
- async def test_subtree_blocked_by_descendant_point(self, agfs_client, test_dir):
- """SUBTREE on parent blocked while descendant holds POINT lock."""
+ async def test_tree_blocked_by_descendant_exact(self, agfs_client, test_dir):
+ """TreeLock on parent blocked while descendant holds ExactPathLock."""
import uuid as _uuid
child = f"{test_dir}/child-{_uuid.uuid4().hex}"
agfs_client.mkdir(child)
- lock = PathLock(agfs_client)
- tx_child = LockHandle(id="tx-desc-point")
- ok = await lock.acquire_point(child, tx_child, timeout=3.0)
+ lock = PathLockEngine(agfs_client)
+ tx_child = LockHandle(id="tx-desc-exact")
+ ok = await lock.acquire_exact_path(child, tx_child, timeout=3.0)
assert ok is True
- tx_parent = LockHandle(id="tx-parent-sub")
- blocked = await lock.acquire_subtree(test_dir, tx_parent, timeout=0.5)
+ tx_parent = LockHandle(id="tx-parent-tree")
+ blocked = await lock.acquire_tree(test_dir, tx_parent, timeout=0.5)
assert blocked is False
await lock.release(tx_child)
- async def test_acquire_mv_creates_subtree_locks(self, agfs_client, test_dir):
- """acquire_mv puts SUBTREE on both src and dst."""
+ async def test_acquire_mv_creates_tree_and_exact_locks(self, agfs_client, test_dir):
+ """Directory move locks the source tree and exact destination path."""
import uuid as _uuid
src = f"{test_dir}/src-{_uuid.uuid4().hex}"
@@ -305,7 +379,7 @@ class TestPathLockBehavior:
agfs_client.mkdir(src)
agfs_client.mkdir(dst)
- lock = PathLock(agfs_client)
+ lock = PathLockEngine(agfs_client)
tx = LockHandle(id="tx-mv-1")
ok = await lock.acquire_mv(src, dst, tx, timeout=3.0)
assert ok is True
@@ -316,7 +390,7 @@ class TestPathLockBehavior:
if isinstance(src_token_bytes, bytes)
else src_token_bytes
)
- assert ":S" in src_token
+ assert ":T" in src_token
dst_token_bytes = agfs_client.cat(f"{dst}/{LOCK_FILE_NAME}")
dst_token = (
@@ -324,12 +398,12 @@ class TestPathLockBehavior:
if isinstance(dst_token_bytes, bytes)
else dst_token_bytes
)
- assert ":S" in dst_token
+ assert ":E" in dst_token
await lock.release(tx)
- async def test_point_does_not_block_sibling_point(self, agfs_client, test_dir):
- """POINT locks on different directories do not conflict."""
+ async def test_exact_does_not_block_sibling_exact(self, agfs_client, test_dir):
+ """EXACT locks on different directories do not conflict."""
import uuid as _uuid
dir_a = f"{test_dir}/sibling-a-{_uuid.uuid4().hex}"
@@ -337,12 +411,12 @@ class TestPathLockBehavior:
agfs_client.mkdir(dir_a)
agfs_client.mkdir(dir_b)
- lock = PathLock(agfs_client)
+ lock = PathLockEngine(agfs_client)
tx_a = LockHandle(id="tx-sib-a")
tx_b = LockHandle(id="tx-sib-b")
- ok_a = await lock.acquire_point(dir_a, tx_a, timeout=3.0)
- ok_b = await lock.acquire_point(dir_b, tx_b, timeout=3.0)
+ ok_a = await lock.acquire_exact_path(dir_a, tx_a, timeout=3.0)
+ ok_b = await lock.acquire_exact_path(dir_b, tx_b, timeout=3.0)
assert ok_a is True
assert ok_b is True
@@ -357,17 +431,17 @@ class TestPathLockBehavior:
target = f"{test_dir}/stale-{_uuid.uuid4().hex}"
agfs_client.mkdir(target)
- lock_path = f"{target}/{LOCK_FILE_NAME}"
+ lock = PathLockEngine(agfs_client, lock_expire=300.0)
+ lock_path = lock._get_exact_lock_path(target)
# Write a lock file with a very old timestamp (simulate crashed process)
old_ts = time.time_ns() - int(600 * 1e9) # 600 seconds ago
- stale_token = f"tx-dead:{old_ts}:{LOCK_TYPE_POINT}"
+ stale_token = f"tx-dead:{old_ts}:{LOCK_TYPE_EXACT}"
agfs_client.write(lock_path, stale_token.encode("utf-8"))
# New transaction should succeed by auto-removing the stale lock
- lock = PathLock(agfs_client, lock_expire=300.0)
tx = LockHandle(id="tx-new-owner")
- ok = await lock.acquire_point(target, tx, timeout=2.0)
+ ok = await lock.acquire_exact_path(target, tx, timeout=2.0)
assert ok is True
# Verify new lock is owned by our transaction
@@ -377,22 +451,22 @@ class TestPathLockBehavior:
await lock.release(tx)
- async def test_stale_subtree_ancestor_auto_removed(self, agfs_client, test_dir):
- """A stale SUBTREE lock on ancestor is auto-removed when child acquires POINT."""
+ async def test_stale_tree_ancestor_auto_removed(self, agfs_client, test_dir):
+ """A stale TreeLock on ancestor is auto-removed when child acquires ExactPathLock."""
import uuid as _uuid
child = f"{test_dir}/child-stale-{_uuid.uuid4().hex}"
agfs_client.mkdir(child)
- # Write stale SUBTREE lock on parent
+ # Write stale TreeLock on parent
parent_lock = f"{test_dir}/{LOCK_FILE_NAME}"
old_ts = time.time_ns() - int(600 * 1e9)
- stale_token = f"tx-dead-parent:{old_ts}:{LOCK_TYPE_SUBTREE}"
+ stale_token = f"tx-dead-parent:{old_ts}:{LOCK_TYPE_TREE}"
agfs_client.write(parent_lock, stale_token.encode("utf-8"))
- lock = PathLock(agfs_client, lock_expire=300.0)
+ lock = PathLockEngine(agfs_client, lock_expire=300.0)
tx = LockHandle(id="tx-child-new")
- ok = await lock.acquire_point(child, tx, timeout=2.0)
+ ok = await lock.acquire_exact_path(child, tx, timeout=2.0)
assert ok is True
await lock.release(tx)
@@ -402,22 +476,22 @@ class TestPathLockBehavior:
except Exception:
pass
- async def test_point_same_path_no_wait_fails_immediately(self, agfs_client, test_dir):
+ async def test_exact_same_path_no_wait_fails_immediately(self, agfs_client, test_dir):
"""With timeout=0, a conflicting lock fails immediately."""
import uuid as _uuid
target = f"{test_dir}/nowait-{_uuid.uuid4().hex}"
agfs_client.mkdir(target)
- lock = PathLock(agfs_client)
+ lock = PathLockEngine(agfs_client)
tx1 = LockHandle(id="tx-hold")
- ok1 = await lock.acquire_point(target, tx1, timeout=3.0)
+ ok1 = await lock.acquire_exact_path(target, tx1, timeout=3.0)
assert ok1 is True
# Second acquire with timeout=0 should fail immediately
tx2 = LockHandle(id="tx-blocked")
t0 = time.monotonic()
- ok2 = await lock.acquire_point(target, tx2, timeout=0.0)
+ ok2 = await lock.acquire_exact_path(target, tx2, timeout=0.0)
elapsed = time.monotonic() - t0
assert ok2 is False
@@ -425,52 +499,52 @@ class TestPathLockBehavior:
await lock.release(tx1)
- async def test_subtree_same_path_mutual_exclusion(self, agfs_client, test_dir):
- """Two SUBTREE locks on the same path: second one blocked until first releases."""
+ async def test_tree_same_path_mutual_exclusion(self, agfs_client, test_dir):
+ """Two TreeLocks on the same path: second one blocked until first releases."""
import uuid as _uuid
- target = f"{test_dir}/sub-excl-{_uuid.uuid4().hex}"
+ target = f"{test_dir}/tree-excl-{_uuid.uuid4().hex}"
agfs_client.mkdir(target)
- lock = PathLock(agfs_client)
- tx1 = LockHandle(id="tx-sub1")
- ok1 = await lock.acquire_subtree(target, tx1, timeout=3.0)
+ lock = PathLockEngine(agfs_client)
+ tx1 = LockHandle(id="tx-tree1")
+ ok1 = await lock.acquire_tree(target, tx1, timeout=3.0)
assert ok1 is True
- tx2 = LockHandle(id="tx-sub2")
- ok2 = await lock.acquire_subtree(target, tx2, timeout=0.5)
+ tx2 = LockHandle(id="tx-tree2")
+ ok2 = await lock.acquire_tree(target, tx2, timeout=0.5)
assert ok2 is False
await lock.release(tx1)
# Now tx2 should succeed
- ok2_retry = await lock.acquire_subtree(target, tx2, timeout=3.0)
+ ok2_retry = await lock.acquire_tree(target, tx2, timeout=3.0)
assert ok2_retry is True
await lock.release(tx2)
- async def test_point_reuses_same_owner_subtree_lock_on_same_path(self, agfs_client, test_dir):
- lock = PathLock(agfs_client)
+ async def test_exact_reuses_same_owner_tree_lock_on_same_path(self, agfs_client, test_dir):
+ lock = PathLockEngine(agfs_client)
tx = LockHandle(id="tx-reentrant-same-path")
- ok = await lock.acquire_subtree(test_dir, tx, timeout=3.0)
+ ok = await lock.acquire_tree(test_dir, tx, timeout=3.0)
assert ok is True
lock_path = f"{test_dir}/{LOCK_FILE_NAME}"
before = agfs_client.cat(lock_path)
before_token = before.decode("utf-8") if isinstance(before, bytes) else before
- assert ":S" in before_token
+ assert ":T" in before_token
- ok_reuse = await lock.acquire_point(test_dir, tx, timeout=0.5)
+ ok_reuse = await lock.acquire_exact_path(test_dir, tx, timeout=0.5)
assert ok_reuse is True
after = agfs_client.cat(lock_path)
after_token = after.decode("utf-8") if isinstance(after, bytes) else after
assert after_token == before_token
- assert ":S" in after_token
+ assert ":T" in after_token
await lock.release(tx)
- async def test_point_under_same_owner_subtree_does_not_create_child_lock(
+ async def test_exact_under_same_owner_tree_does_not_create_child_lock(
self, agfs_client, test_dir
):
import uuid as _uuid
@@ -478,19 +552,19 @@ class TestPathLockBehavior:
child = f"{test_dir}/child-reentrant-{_uuid.uuid4().hex}"
agfs_client.mkdir(child)
- lock = PathLock(agfs_client)
+ lock = PathLockEngine(agfs_client)
tx = LockHandle(id="tx-reentrant-child")
- ok = await lock.acquire_subtree(test_dir, tx, timeout=3.0)
+ ok = await lock.acquire_tree(test_dir, tx, timeout=3.0)
assert ok is True
- ok_child = await lock.acquire_point(child, tx, timeout=0.5)
+ ok_child = await lock.acquire_exact_path(child, tx, timeout=0.5)
assert ok_child is True
- child_lock_path = f"{child}/{LOCK_FILE_NAME}"
+ child_lock_path = lock._get_exact_lock_path(child)
try:
agfs_client.stat(child_lock_path)
- raise AssertionError("child lock should not be created when ancestor subtree is owned")
+ raise AssertionError("child lock should not be created when ancestor tree is owned")
except AssertionError:
raise
except Exception: