Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat(resources): persist async add-resource tasks (#2433)

Qin Haojie72ec6f9287b2
diff --git a/docs/en/about/02-changelog.md b/docs/en/about/02-changelog.md
index 65cadbea8..b7a62130b 100644
--- a/docs/en/about/02-changelog.md
+++ b/docs/en/about/02-changelog.md
@@ -128,11 +128,11 @@ This changelog is automatically generated from [GitHub Releases](https://github.
- **Native CLI distribution**: New `@openviking/cli` npm package installs the platform `ov` binary via `npm i -g @openviking/cli`; Rust CLI release pipeline adds Linux musl artifacts, npm trusted publishing, and a broader integration test suite.
- **Retrieval and filesystem**: `find` / `search` accept a `level` filter for L0 abstracts, L1 overviews, and L2 file hits. Resource files gained a Phase 1 WebDAV adapter, and `observer.filesystem` exposes filesystem-level observability.
- **Console and Usage/Audit**: New Usage/Audit module and `/api/v1/console/*` BFF aggregate token usage, retrieval counts, context commit heatmaps, request audit logs, and context inventory from the existing observability event bus.
-- **Storage and concurrency reliability**: Strengthened exact-path and lifecycle locks fix content-write races; blocking backend calls moved off the event loop; QueueFS SQLite persistence expanded; `storage.task_tracker.backend` adds a `persistent` backend for cross-instance task lookup.
+- **Storage and concurrency reliability**: Strengthened exact-path and lifecycle locks fix content-write races; blocking backend calls moved off the event loop; QueueFS SQLite persistence expanded; task records are now persisted for cross-instance lookup; `add_resource(wait=false)` returns a reserved `root_uri` plus persistent task progress while ingestion completes.
### Upgrade Notes
-- `storage.task_tracker.backend` is new. Single-instance deployments can keep the default `memory`; multi-instance or restart-survivable deployments should switch to `persistent`.
+- `storage.task_tracker` is deprecated and ignored. Task records are always persisted under each account's `_system/tasks` directory.
- `vlm.backup` is a single-tier failover and only triggers on retryable errors (rate limit, `5xx`, connection failure, timeout). Auth, authorization, and billing failures do not trigger failover.
- `vlm.extra_request_body` is merged into the OpenAI SDK / LiteLLM `extra_body`, useful for Ollama, OpenAI-compatible gateways, and providers requiring extra JSON fields.
- New Codex memory plugin deployments should prefer `OPENVIKING_*` environment variables for tuning; the legacy `codex.*` block in `ov.conf` remains supported for backward compatibility but is no longer recommended.
diff --git a/docs/en/api/02-resources.md b/docs/en/api/02-resources.md
index 561a6cefc..eecc2e910 100644
--- a/docs/en/api/02-resources.md
+++ b/docs/en/api/02-resources.md
@@ -82,6 +82,12 @@ Source Input -> Parse -> Resource Tree Build -> Persistence -> Semantic Processi
- **Vector Index**: Vectorizes content for semantic search
- Processed asynchronously via `SemanticQueue`, can wait for completion with `wait=True`
+#### Async `add_resource` Tasks
+- With `wait=false`, OpenViking validates the source, resolves the target URI, and reserves the final `root_uri` before returning.
+- The immediate response contains `status`, `root_uri`, and `task_id`; fetching/parsing/finalizing and queue waiting continue in a persistent background task.
+- Poll `GET /api/v1/tasks/{task_id}` to inspect task state. Resource import tasks use stages such as `queued`, `fetching`, `parsing`, `finalizing`, and `processing_queue`.
+- Task status responses expose the current `stage`; live semantic and embedding queue counts are available through observer queue APIs, and completed imports include `queue_status` in the task `result`.
+
### Incremental Updates for Resources
Resource incremental updates are implemented via the **Watch Task** mechanism:
@@ -115,11 +121,12 @@ Add a resource to the knowledge base. The SDK supports local files/directories,
This endpoint is the core entry point for resource management, supporting adding resources from various sources with optional waiting for semantic processing completion.
**Processing Flow**:
-1. Identify resource source (URL or uploaded temporary file)
-2. Call corresponding Parser to parse content
-3. Build directory tree and write to AGFS
-4. Set up scheduled update task if `watch_interval` is specified
-5. Wait for semantic processing completion if `wait=true`
+1. Identify and validate the resource source (URL or uploaded temporary file)
+2. Resolve the target URI; when `wait=false`, reserve the final `root_uri` before returning
+3. Call the corresponding Parser to parse content
+4. Build the directory tree and write to AGFS
+5. Wait for semantic processing completion when `wait=true`, or track background task state with the returned `task_id`
+6. Set up scheduled update task if `watch_interval` is specified
**Code Entry Points**:
- `openviking/client/local.py:LocalClient.add_resource` - SDK entry (embedded)
@@ -157,6 +164,8 @@ This endpoint is the core entry point for resource management, supporting adding
- `path` and `temp_file_id` cannot be specified together
- Raw HTTP calls for local files require first uploading via [temp_upload](#temp_upload) to obtain `temp_file_id`
- When `to` is specified and the target already exists, triggers incremental update
+- When `wait=false`, the request still performs source preflight and target planning before returning. Unreachable remote URLs/repositories or invalid parent targets fail immediately instead of creating a task.
+- When OpenViking auto-selects a target under `viking://resources` or `parent`, URI conflicts are resolved and reserved before the `wait=false` response is returned, so the returned `root_uri` is stable.
- When `watch_interval > 0`, the watch task binds to `to` if provided; otherwise it binds to the `root_uri` returned by this import. If no stable `root_uri` is available, the request fails and asks for an explicit `to`.
- For local directory inputs, scanning respects `.gitignore` files (root and nested) with standard Git semantics; `ignore_dirs`, `include`, and `exclude` further refine what is ingested.
- To create or update plain text directly, use [content/write](03-filesystem.md#write) instead of `add_resource`. Semantic processing and embeddings are refreshed automatically after resource ingestion and content writes.
@@ -272,7 +281,7 @@ ov add-resource ./documents/guide.md -p viking://resources/docs/{calendar:today}
**Response Example**
-**HTTP API Response (JSON)**
+**HTTP API Response (JSON, `wait=true`)**
```json
{
@@ -296,17 +305,29 @@ ov add-resource ./documents/guide.md -p viking://resources/docs/{calendar:today}
}
```
+**HTTP API Response (JSON, `wait=false`)**
+
+```json
+{
+ "status": "ok",
+ "result": {
+ "status": "success",
+ "root_uri": "viking://resources/guide",
+ "task_id": "uuid-xxx"
+ }
+}
+```
+
+Use the returned `task_id` to poll `/api/v1/tasks/{task_id}`. The completed task result contains the full import result, including `queue_status`.
+
**CLI Response (Default Table Format)**
```
Note: Resource is being processed in the background.
Use 'ov wait' to wait for completion, or 'ov observer queue' to check status.
status success
-errors []
-source_path /Users/bytedance/workspace/github.com/OpenViking/docs/en/api/01-overview.md
-meta {}
root_uri viking://resources/01-overview
-temp_uri viking://temp/shengmaojia/04291108_b62dc7/01-overview
+task_id uuid-xxx
```
**CLI Response (JSON Format, using -o json)**
@@ -315,10 +336,7 @@ temp_uri viking://temp/shengmaojia/04291108_b62dc7/01-overview
{
"status": "success",
"root_uri": "viking://resources/01-overview",
- "temp_uri": "viking://temp/shengmaojia/04291108_b62dc7/01-overview",
- "source_path": "/Users/bytedance/workspace/github.com/OpenViking/docs/en/api/01-overview.md",
- "meta": {},
- "errors": []
+ "task_id": "uuid-xxx"
}
```
@@ -328,6 +346,7 @@ temp_uri viking://temp/shengmaojia/04291108_b62dc7/01-overview
|-------|------|-------------|
| `status` | string | Processing status: "success" or "error" |
| `root_uri` | string | Final URI of the resource in OpenViking |
+| `task_id` | string | (Optional, only when `wait=false`) Background task ID for polling `/api/v1/tasks/{task_id}` |
| `temp_uri` | string | Temporary URI during processing (only valid during background processing) |
| `source_path` | string | Original source file path or URL |
| `meta` | object | Metadata from resource parsing (file type, size, etc.) |
@@ -335,6 +354,8 @@ temp_uri viking://temp/shengmaojia/04291108_b62dc7/01-overview
| `warnings` | array | (Optional) List of warnings (only when `strict=False`) |
| `queue_status` | object | (Optional, only when `wait=true`) Queue processing status with `pending`, `processing`, `completed` counts |
+For `wait=false`, the background task has `task_type="add_resource"` and `resource_id` equal to the returned `root_uri`. Running task records may include `stage`; completed task results include `queue_status` with the final semantic and embedding queue summary.
+
---
### Watch Management
diff --git a/docs/en/api/05-sessions.md b/docs/en/api/05-sessions.md
index 6d498b095..ad73604e7 100644
--- a/docs/en/api/05-sessions.md
+++ b/docs/en/api/05-sessions.md
@@ -886,7 +886,7 @@ await client.session_used(
#### 1. API Implementation Introduction
-Commit a session. Message archiving (Phase 1) completes immediately. Summary generation and memory extraction (Phase 2) run asynchronously in the background. Returns a `task_id` for polling progress.
+Commit a session. Message archiving (Phase 1) completes immediately. Summary generation and memory extraction (Phase 2) run asynchronously in the background. Returns a `task_id` for polling status.
**Two-Phase Commit Flow:**
- **Phase 1 (Synchronous)**: Snapshot current messages, clear live session, create archive directory, write original messages
@@ -943,7 +943,7 @@ result = await client.commit_session("a1b2c3d4")
print(f"Status: {result['status']}")
print(f"Task ID: {result['task_id']}")
-# Poll background task progress
+# Poll background task status
task = await client.get_task(result["task_id"])
if task["status"] == "completed":
memories = task["result"]["memories_extracted"]
@@ -1015,7 +1015,7 @@ The endpoint returns the extracted memory write results as a JSON list. The exac
#### 1. API Implementation Introduction
-Query background task status (e.g., commit summary generation and memory extraction progress).
+Query background task status for APIs that return `task_id`, such as session commit, `add_resource`, and admin reindex.
**Task Statuses:**
- `pending`: Task waiting to execute
@@ -1026,13 +1026,15 @@ Query background task status (e.g., commit summary generation and memory extract
**Code Entries:**
- `openviking/server/routers/tasks.py:get_task()` - HTTP route
+Task records are persisted in AGFS and can be queried after server restart, subject to task retention cleanup.
+
#### 2. Interface and Parameter Description
**Parameters**
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
-| task_id | str | Yes | - | Task ID (returned by commit) |
+| task_id | str | Yes | - | Task ID returned by a background API |
#### 3. Usage Examples
@@ -1058,19 +1060,23 @@ task = await client.get_task(task_id="uuid-xxx")
print(f"Status: {task['status']}")
```
-**Response Example (in progress)**
+**Response Example (resource import in progress)**
```json
{
"status": "ok",
"result": {
"task_id": "uuid-xxx",
- "task_type": "session_commit",
- "status": "running"
+ "task_type": "add_resource",
+ "status": "running",
+ "resource_id": "viking://resources/guide",
+ "stage": "processing_queue"
}
}
```
+`stage` is nullable. Resource import tasks may report `queued`, `fetching`, `parsing`, `finalizing`, or `processing_queue`; other task types may leave it as `null`. Live queue counters are intentionally not part of task status; use observer queue APIs for live counts, or read `result.queue_status` after completion.
+
**Response Example (completed)**
```json
@@ -1159,7 +1165,8 @@ curl -X GET "http://localhost:1933/api/v1/tasks?task_type=session_commit&status=
"created_at": 1770000000.0,
"updated_at": 1770000005.0,
"result": null,
- "error": null
+ "error": null,
+ "stage": null
}
]
}
@@ -1371,7 +1378,7 @@ curl -X POST http://localhost:1933/api/v1/sessions/a1b2c3d4/commit \
-H "X-API-Key: your-key"
# Returns: {"status": "ok", "result": {"status": "accepted", "task_id": "uuid-xxx", ...}}
-# Step 7: Poll background task progress (optional)
+# Step 7: Poll background task status (optional)
curl -X GET http://localhost:1933/api/v1/tasks/uuid-xxx \
-H "X-API-Key: your-key"
```
diff --git a/docs/en/api/07-system.md b/docs/en/api/07-system.md
index 10d1ddd67..19a75ec0a 100644
--- a/docs/en/api/07-system.md
+++ b/docs/en/api/07-system.md
@@ -518,7 +518,7 @@ Reindex background tasks use `task_type="admin_reindex"` and `resource_id` equal
GET /api/v1/tasks?task_type=admin_reindex&resource_id=viking://resources
```
-Task records are kept in memory and can expire or be lost on server restart.
+Task records are persisted under `/local/{account_id}/_system/tasks/{user_id}/{task_id}.json` and can be queried after restart.
**Result fields**
diff --git a/docs/en/guides/01-configuration.md b/docs/en/guides/01-configuration.md
index 97cc89a1a..e2004769f 100644
--- a/docs/en/guides/01-configuration.md
+++ b/docs/en/guides/01-configuration.md
@@ -1289,25 +1289,17 @@ Path locks are enabled by default and usually require no configuration. **The de
For details on the lock mechanism, see [Path Locks and Crash Recovery](../concepts/09-transaction.md).
-## storage.task_tracker Section
+## Task Tracker Persistence
-The task tracker records async task state for endpoints that return a `task_id` (task types include `session_commit`, `add_resource`, `add_skill`, and `admin_reindex`). The `persistent` backend stores task state on the workspace volume so that **a `task_id` returned by one instance can be looked up from another instance**, and task history survives a restart. The default `memory` backend keeps task state per process — sufficient for single-instance deployments.
+The task tracker records async task state for endpoints that return a `task_id` (task types include `session_commit`, `add_resource`, `add_skill`, and `admin_reindex`). Task records are always persisted in AGFS, so a `task_id` returned by one instance can be looked up from another instance and task history survives a restart.
-```json
-{
- "storage": {
- "task_tracker": {
- "backend": "memory"
- }
- }
-}
-```
+No `storage.task_tracker` configuration is required. If an older configuration still includes `storage.task_tracker`, OpenViking logs a warning and ignores it.
-| Parameter | Type | Description | Default |
-|-----------|------|-------------|---------|
-| `backend` | str | Task tracker backend. `"memory"` keeps task state in process memory (single-instance). `"persistent"` enables cross-instance task lookup and survives restarts. | `"memory"` |
+Task record files are stored under the owning account's system directory:
-Set `backend` to `"persistent"` for multi-instance deployments where callers may poll `GET /api/v1/tasks/{task_id}` from any instance, or when task history needs to outlive the process.
+```text
+/local/{account_id}/_system/tasks/{user_id}/{task_id}.json
+```
## encryption Section
diff --git a/docs/zh/about/02-changelog.md b/docs/zh/about/02-changelog.md
index a1a438b7b..cf962b327 100644
--- a/docs/zh/about/02-changelog.md
+++ b/docs/zh/about/02-changelog.md
@@ -128,11 +128,11 @@ OpenViking 的所有重要变更都将记录在此文件中。
- **原生 CLI 分发**:新增 `@openviking/cli` npm 包,可通过 `npm i -g @openviking/cli` 使用 `ov`;Rust CLI 发布流水线扩展 Linux musl 构建、npm trusted publishing 和 CLI 集成测试。
- **检索与文件系统能力**:`find` / `search` 新增 `level` 过滤,可限定 L0 abstract、L1 overview 或 L2 文件命中;资源文件增加 Phase 1 WebDAV 适配;`observer.filesystem` 暴露文件系统观测入口。
- **Console 与 Usage/Audit**:新增 Usage/Audit 模块和 `/api/v1/console/*` BFF,基于现有 observability event bus 统计 token、检索次数、上下文提交热力图、请求审计和上下文库存。
-- **存储与并发可靠性**:增强精确路径锁和生命周期锁修复内容写入并发覆盖;阻塞后端调用移出 event loop;QueueFS SQLite 持久化扩展;`storage.task_tracker.backend` 新增 `persistent` 后端支持多实例 task 查询。
+- **存储与并发可靠性**:增强精确路径锁和生命周期锁修复内容写入并发覆盖;阻塞后端调用移出 event loop;QueueFS SQLite 持久化扩展;task 记录现在会持久化以支持多实例查询;`add_resource(wait=false)` 会返回已预占的 `root_uri`,并在导入完成前提供持久化 task 进度。
### 升级说明
-- `storage.task_tracker.backend` 是新增配置。单实例部署可继续使用默认 `memory`;多实例或需要重启后查询 `task_id` 的部署可切换到 `persistent`。
+- `storage.task_tracker` 已废弃并会被忽略。Task 记录始终持久化到各账号的 `_system/tasks` 目录。
- `vlm.backup` 只支持一层 backup,且只在 rate limit、`5xx`、连接失败和 timeout 等可重试错误上触发;认证、权限和计费类错误不会自动切换。
- `vlm.extra_request_body` 会合并到 OpenAI SDK / LiteLLM 的 `extra_body`,适合接入 Ollama、OpenAI-compatible gateway 或其他需要额外 JSON 字段的 provider。
- Codex 插件新部署建议使用 `OPENVIKING_*` 环境变量调优;旧的 `ov.conf` 中 `codex.*` 配置仍保留兼容,但不再推荐作为首选。
diff --git a/docs/zh/api/02-resources.md b/docs/zh/api/02-resources.md
index 488ae573b..7a4fa8197 100644
--- a/docs/zh/api/02-resources.md
+++ b/docs/zh/api/02-resources.md
@@ -77,6 +77,12 @@ URL/文件 Parser TreeBuilder AGFS Summarizer/Vector
- **向量索引**:将内容向量化用于语义搜索
- 通过 `SemanticQueue` 异步处理,可通过 `wait=True` 等待完成
+#### 异步 `add_resource` 任务
+- `wait=false` 时,OpenViking 仍会先校验资源来源、解析目标 URI,并预占最终 `root_uri`,然后再返回。
+- 立即响应包含 `status`、`root_uri` 和 `task_id`;抓取/解析/finalize 以及队列等待会在持久化后台任务中继续执行。
+- 可通过 `GET /api/v1/tasks/{task_id}` 查询任务状态。资源导入任务的阶段包括 `queued`、`fetching`、`parsing`、`finalizing`、`processing_queue`。
+- 任务状态响应只暴露当前 `stage`;实时 semantic 和 embedding 队列计数可通过 observer queue 查询,导入完成后的任务 `result` 会包含 `queue_status`。
+
### 资源的增量更新
资源增量更新通过**监控任务 (Watch Task)** 机制实现:
@@ -110,11 +116,12 @@ URL/文件 Parser TreeBuilder AGFS Summarizer/Vector
此接口是资源管理的核心入口,支持多种来源的资源添加,并可选择等待语义处理完成。SDK 可直接处理本地文件/目录、URL 等来源;直接 HTTP 调用只通过 `path` 接受远程 URL,或通过 `temp_file_id` 引用先上传的本地文件。
**处理流程**:
-1. 识别资源来源(URL 或上传的临时文件)
-2. 调用对应 Parser 解析内容
-3. 构建目录树并写入 AGFS
-4. 如指定 `--watch-interval`,设置定时更新任务
-5. 如指定 `--wait=true`,等待语义处理完成
+1. 识别并校验资源来源(URL 或上传的临时文件)
+2. 解析目标 URI;`wait=false` 时会先预占最终 `root_uri` 再返回
+3. 调用对应 Parser 解析内容
+4. 构建目录树并写入 AGFS
+5. `wait=true` 时等待语义处理完成;否则通过返回的 `task_id` 查询后台任务状态
+6. 如指定 `--watch-interval`,设置定时更新任务
**代码入口**:
- `openviking/client/local.py:LocalClient.add_resource` - SDK 入口(嵌入式)
@@ -150,6 +157,8 @@ URL/文件 Parser TreeBuilder AGFS Summarizer/Vector
**补充说明**:
- `to` 和 `parent` 不能同时使用;如果使用 `parent` 且希望父目录不存在时自动创建,请传 `create_parent=true`。指定 `to` 且目标已存在时,触发增量更新。
- `path` 和 `temp_file_id` 不能同时指定,上传本地文件需要先通过 [temp_upload](#temp_upload) 上传获取 `temp_file_id`,在 SDK 和 CLI 中已经封装好。
+- `wait=false` 时,请求仍会先完成资源 preflight 和目标规划。远程 URL/仓库不可达或父级目标无效会立即失败,不会创建后台任务。
+- 当 OpenViking 在 `viking://resources` 或 `parent` 下自动选择目标时,会在 `wait=false` 响应前完成冲突重命名和 URI 预占,因此返回的 `root_uri` 是稳定的。
- `watch_interval > 0` 时,如果指定了 `to`,监控任务绑定该目标;如果未指定 `to`,监控任务绑定本次导入返回的 `root_uri`。如果无法得到稳定 `root_uri`,请求会报错并要求显式传 `to`。
- 本地目录输入会遵循 `.gitignore`(根目录和子目录,标准 Git 语义);`ignore_dirs`、`include`、`exclude` 会在此基础上进一步过滤。
- 如果要直接创建或更新纯文本内容,请使用 [content/write](03-filesystem.md#write),不要使用 `add_resource`。资源导入和内容写入后都会自动刷新语义与 embedding。
@@ -265,7 +274,7 @@ ov add-resource ./documents/guide.md -p viking://resources/docs/{calendar:today}
#### 4. 响应示例
-**HTTP API 响应 (JSON)**
+**HTTP API 响应 (JSON, `wait=true`)**
```json
{
@@ -289,17 +298,29 @@ ov add-resource ./documents/guide.md -p viking://resources/docs/{calendar:today}
}
```
+**HTTP API 响应 (JSON, `wait=false`)**
+
+```json
+{
+ "status": "ok",
+ "result": {
+ "status": "success",
+ "root_uri": "viking://resources/guide",
+ "task_id": "uuid-xxx"
+ }
+}
+```
+
+使用返回的 `task_id` 轮询 `/api/v1/tasks/{task_id}`。任务完成后的 `result` 会包含完整导入结果,包括 `queue_status`。
+
**CLI 响应 (默认表格格式)**
```
Note: Resource is being processed in the background.
Use 'ov wait' to wait for completion, or 'ov observer queue' to check status.
status success
-errors []
-source_path /Users/bytedance/workspace/github.com/OpenViking/docs/en/api/01-overview.md
-meta {}
root_uri viking://resources/01-overview
-temp_uri viking://temp/shengmaojia/04291108_b62dc7/01-overview
+task_id uuid-xxx
```
**CLI 响应 (JSON 格式,使用 -o json)**
@@ -308,10 +329,7 @@ temp_uri viking://temp/shengmaojia/04291108_b62dc7/01-overview
{
"status": "success",
"root_uri": "viking://resources/01-overview",
- "temp_uri": "viking://temp/shengmaojia/04291108_b62dc7/01-overview",
- "source_path": "/Users/bytedance/workspace/github.com/OpenViking/docs/en/api/01-overview.md",
- "meta": {},
- "errors": []
+ "task_id": "uuid-xxx"
}
```
@@ -321,6 +339,7 @@ temp_uri viking://temp/shengmaojia/04291108_b62dc7/01-overview
|------|------|------|
| `status` | string | 处理状态:"success" 成功,"error" 失败 |
| `root_uri` | string | 资源在 OpenViking 中的最终 URI |
+| `task_id` | string | (可选,仅当 `wait=false` 时)后台任务 ID,可轮询 `/api/v1/tasks/{task_id}` |
| `temp_uri` | string | 处理过程中的临时 URI(仅在后台处理阶段有效) |
| `source_path` | string | 原始源文件路径或 URL |
| `meta` | object | 资源解析过程中的元数据(如文件类型、大小等) |
@@ -328,6 +347,8 @@ temp_uri viking://temp/shengmaojia/04291108_b62dc7/01-overview
| `warnings` | array | (可选)处理过程中的警告列表(仅在 `strict=False` 时可能出现) |
| `queue_status` | object | (可选,仅当 `wait=true` 时)队列处理状态,包含 `pending`、`processing`、`completed` 计数 |
+对于 `wait=false`,后台任务的 `task_type="add_resource"`,`resource_id` 等于返回的 `root_uri`。运行中的任务记录可能包含 `stage`;完成后的任务 `result` 会包含带有 semantic 和 embedding 汇总的 `queue_status`。
+
---
### Watch Management(监控任务管理)
diff --git a/docs/zh/api/05-sessions.md b/docs/zh/api/05-sessions.md
index 7955b1557..06c64bd6f 100644
--- a/docs/zh/api/05-sessions.md
+++ b/docs/zh/api/05-sessions.md
@@ -886,7 +886,7 @@ await client.session_used(
#### 1. API 实现介绍
-提交会话。归档消息(Phase 1)立即完成,摘要生成和记忆提取(Phase 2)在后台异步执行。返回 `task_id` 用于查询后台任务进度。
+提交会话。归档消息(Phase 1)立即完成,摘要生成和记忆提取(Phase 2)在后台异步执行。返回 `task_id` 用于查询后台任务状态。
**两阶段提交流程**:
- **Phase 1(同步)**: 快照当前消息,清空 live session,创建归档目录,写入原始消息
@@ -943,7 +943,7 @@ result = await client.commit_session("a1b2c3d4")
print(f"Status: {result['status']}")
print(f"Task ID: {result['task_id']}")
-# 查询后台任务进度
+# 查询后台任务状态
task = await client.get_task(result["task_id"])
if task["status"] == "completed":
memories = task["result"]["memories_extracted"]
@@ -1015,7 +1015,7 @@ curl -X POST http://localhost:1933/api/v1/sessions/a1b2c3d4/extract \
#### 1. API 实现介绍
-查询后台任务状态(如 commit 的摘要生成和记忆提取进度)。
+查询返回 `task_id` 的后台任务状态,例如 session commit、`add_resource` 和 admin reindex。
**任务状态**:
- `pending`: 任务等待执行
@@ -1026,13 +1026,15 @@ curl -X POST http://localhost:1933/api/v1/sessions/a1b2c3d4/extract \
**代码入口**:
- `openviking/server/routers/tasks.py:get_task()` - HTTP 路由
+任务记录会持久化到 AGFS,服务重启后仍可查询,但仍受任务保留清理策略影响。
+
#### 2. 接口和参数说明
**参数**
| 参数 | 类型 | 必填 | 默认值 | 说明 |
|------|------|------|--------|------|
-| task_id | str | 是 | - | 任务 ID(由 commit 返回) |
+| task_id | str | 是 | - | 后台 API 返回的任务 ID |
#### 3. 使用示例
@@ -1058,19 +1060,23 @@ task = await client.get_task(task_id="uuid-xxx")
print(f"Status: {task['status']}")
```
-**响应示例(进行中)**
+**响应示例(资源导入进行中)**
```json
{
"status": "ok",
"result": {
"task_id": "uuid-xxx",
- "task_type": "session_commit",
- "status": "running"
+ "task_type": "add_resource",
+ "status": "running",
+ "resource_id": "viking://resources/guide",
+ "stage": "processing_queue"
}
}
```
+`stage` 可以为 `null`。资源导入任务可能报告 `queued`、`fetching`、`parsing`、`finalizing`、`processing_queue`;其他任务类型可能将其留空。实时队列计数不会出现在任务状态中;需要实时数量时使用 observer queue,任务完成后可读取 `result.queue_status`。
+
**响应示例(完成)**
```json
@@ -1157,7 +1163,8 @@ curl -X GET "http://localhost:1933/api/v1/tasks?task_type=session_commit&status=
"created_at": 1770000000.0,
"updated_at": 1770000005.0,
"result": null,
- "error": null
+ "error": null,
+ "stage": null
}
]
}
@@ -1369,7 +1376,7 @@ curl -X POST http://localhost:1933/api/v1/sessions/a1b2c3d4/commit \
-H "X-API-Key: your-key"
# 返回:{"status": "ok", "result": {"status": "accepted", "task_id": "uuid-xxx", ...}}
-# 步骤 7:查询后台任务进度(可选)
+# 步骤 7:查询后台任务状态(可选)
curl -X GET http://localhost:1933/api/v1/tasks/uuid-xxx \
-H "X-API-Key: your-key"
```
diff --git a/docs/zh/api/07-system.md b/docs/zh/api/07-system.md
index fd50e39dd..61ad3137a 100644
--- a/docs/zh/api/07-system.md
+++ b/docs/zh/api/07-system.md
@@ -512,7 +512,7 @@ Reindex 后台任务的 `task_type` 为 `admin_reindex`,`resource_id` 等于
GET /api/v1/tasks?task_type=admin_reindex&resource_id=viking://resources
```
-任务记录保存在内存中,可能过期,也会在服务重启后丢失。
+任务记录持久化在 `/local/{account_id}/_system/tasks/{user_id}/{task_id}.json`,服务重启后仍可查询。
**结果字段**
diff --git a/docs/zh/guides/01-configuration.md b/docs/zh/guides/01-configuration.md
index d4571ffcc..237079bb3 100644
--- a/docs/zh/guides/01-configuration.md
+++ b/docs/zh/guides/01-configuration.md
@@ -1354,25 +1354,17 @@ openviking add-resource ./docs --exclude "*.tmp"
路径锁机制的详细说明见 [路径锁与崩溃恢复](../concepts/09-transaction.md)。
-## storage.task_tracker 段
+## Task Tracker 持久化
-任务跟踪器记录异步任务状态,适用于返回 `task_id` 的接口(任务类型包括 `session_commit`、`add_resource`、`add_skill`、`admin_reindex`)。`persistent` 后端把任务状态写到 workspace 卷上,因此**一个实例返回的 `task_id` 可以在另一个实例上查询**,并且任务历史在重启后仍可访问。默认 `memory` 后端把任务状态保留在进程内存中——适合单实例部署。
+任务跟踪器记录异步任务状态,适用于返回 `task_id` 的接口(任务类型包括 `session_commit`、`add_resource`、`add_skill`、`admin_reindex`)。Task 记录始终持久化到 AGFS,因此一个实例返回的 `task_id` 可以在另一个实例上查询,任务历史也能在重启后继续访问。
-```json
-{
- "storage": {
- "task_tracker": {
- "backend": "memory"
- }
- }
-}
-```
+无需配置 `storage.task_tracker`。如果旧配置里仍包含 `storage.task_tracker`,OpenViking 会记录 warning 并忽略它。
-| 参数 | 类型 | 说明 | 默认值 |
-|------|------|------|--------|
-| `backend` | str | 任务跟踪器后端。`"memory"` 把任务状态保留在进程内存(单实例);`"persistent"` 启用跨实例任务查询并在重启后仍可访问 | `"memory"` |
+Task 记录文件位于所属账号的系统目录:
-多实例部署中如果调用方可能从任一实例 poll `GET /api/v1/tasks/{task_id}`,或者任务历史需要超过进程生命周期,请把 `backend` 设为 `"persistent"`。
+```text
+/local/{account_id}/_system/tasks/{user_id}/{task_id}.json
+```
## 完整 Schema
diff --git a/openviking/parse/tree_builder.py b/openviking/parse/tree_builder.py
index 1b2b59573..88850a9c4 100644
--- a/openviking/parse/tree_builder.py
+++ b/openviking/parse/tree_builder.py
@@ -82,6 +82,67 @@ class TreeBuilder:
# v5.0 Methods (temporary directory + SemanticQueue architecture)
# ============================================================================
+ async def resolve_target_uri(
+ self,
+ *,
+ ctx: RequestContext,
+ doc_name: str,
+ scope: str = "resources",
+ to_uri: Optional[str] = None,
+ parent_uri: Optional[str] = None,
+ source_path: Optional[str] = None,
+ source_format: Optional[str] = None,
+ create_parent: bool = False,
+ ) -> tuple[str, Optional[str]]:
+ """Resolve the final target URI and optional unique-name candidate."""
+
+ def is_resources_root(uri: Optional[str]) -> bool:
+ return (uri or "").rstrip("/") == "viking://resources"
+
+ final_doc_name = VikingURI.sanitize_segment(doc_name)
+ if source_path and source_format == "repository":
+ parsed_org_repo = parse_code_hosting_url(source_path)
+ if parsed_org_repo:
+ final_doc_name = parsed_org_repo
+
+ auto_base_uri = self._get_base_uri(scope, source_path, source_format)
+ base_uri = parent_uri or auto_base_uri
+ use_to_as_parent = is_resources_root(to_uri)
+ if to_uri and not use_to_as_parent:
+ return to_uri, None
+
+ effective_parent_uri = (parent_uri or to_uri) if use_to_as_parent else parent_uri
+ if effective_parent_uri:
+ effective_parent_uri = effective_parent_uri.rstrip("/")
+ if effective_parent_uri:
+ viking_fs = get_viking_fs()
+ try:
+ parent_exists = await viking_fs.exists(effective_parent_uri, ctx=ctx)
+ if not parent_exists:
+ if create_parent:
+ logger.info(
+ f"[TreeBuilder] Parent URI does not exist, creating: {effective_parent_uri}"
+ )
+ await viking_fs.mkdir(effective_parent_uri, exist_ok=True, ctx=ctx)
+ else:
+ raise FileNotFoundError(
+ f"Parent URI does not exist: {effective_parent_uri}. "
+ f"Use --parent-auto-create/-p to automatically create it."
+ )
+ stat_result = await viking_fs.stat(effective_parent_uri, ctx=ctx)
+ except FileNotFoundError:
+ raise
+ except Exception as e:
+ raise FileNotFoundError(
+ f"Parent URI does not exist: {effective_parent_uri}"
+ ) from e
+ if not stat_result.get("isDir"):
+ raise ValueError(f"Parent URI is not a directory: {effective_parent_uri}")
+ base_uri = effective_parent_uri
+
+ planned_uri = VikingURI(base_uri).join(final_doc_name).uri
+ return planned_uri, planned_uri
+
async def finalize_from_temp(
self,
temp_dir_path: str,
@@ -105,9 +166,6 @@ class TreeBuilder:
viking_fs = get_viking_fs()
temp_uri = temp_dir_path
- def is_resources_root(uri: Optional[str]) -> bool:
- return (uri or "").rstrip("/") == "viking://resources"
-
# 1. Find document root directory
entries = await viking_fs.ls(temp_uri, ctx=ctx)
doc_dirs = [e for e in entries if e.get("isDir") and e["name"] not in [".", ".."]]
@@ -126,72 +184,27 @@ class TreeBuilder:
if original_name != doc_name:
logger.debug(f"[TreeBuilder] Sanitized doc name: {original_name!r} -> {doc_name!r}")
- # Check if source_path is a GitHub/GitLab URL and extract org/repo
- # This is critical for getting the full "org/repo" path instead of just repo name!
- # For example:
- # - source_path = "https://github.com/volcengine/OpenViking"
- # - parsed_org_repo = "volcengine/OpenViking"
- # - final root_uri = "viking://resources/volcengine/OpenViking"
- #
- # Without this, we'd just get "viking://resources/OpenViking" without the org prefix
- final_doc_name = doc_name
- if source_path and source_format == "repository":
- parsed_org_repo = parse_code_hosting_url(source_path)
- if parsed_org_repo:
- final_doc_name = parsed_org_repo
-
- # 2. Determine base_uri and final document name with org/repo for GitHub/GitLab
- auto_base_uri = self._get_base_uri(scope, source_path, source_format)
- base_uri = parent_uri or auto_base_uri
- use_to_as_parent = is_resources_root(to_uri)
- # 3. Determine candidate_uri
- if to_uri and not use_to_as_parent:
- candidate_uri = to_uri
- else:
- effective_parent_uri = parent_uri or to_uri if use_to_as_parent else parent_uri
- if effective_parent_uri:
- effective_parent_uri = effective_parent_uri.rstrip("/")
- if effective_parent_uri:
- # Parent URI must exist and be a directory, or create it if requested
- try:
- # First check if parent exists
- parent_exists = await viking_fs.exists(effective_parent_uri, ctx=ctx)
- if not parent_exists:
- if create_parent:
- # Automatically create parent directory
- logger.info(
- f"[TreeBuilder] Parent URI does not exist, creating: {effective_parent_uri}"
- )
- await viking_fs.mkdir(effective_parent_uri, exist_ok=True, ctx=ctx)
- else:
- raise FileNotFoundError(
- f"Parent URI does not exist: {effective_parent_uri}. "
- f"Use --parent-auto-create/-p to automatically create it."
- )
- # Now get stat result
- stat_result = await viking_fs.stat(effective_parent_uri, ctx=ctx)
- except FileNotFoundError:
- # Re-raise without wrapping
- raise
- except Exception as e:
- raise FileNotFoundError(
- f"Parent URI does not exist: {effective_parent_uri}"
- ) from e
- if not stat_result.get("isDir"):
- raise ValueError(f"Parent URI is not a directory: {effective_parent_uri}")
- base_uri = effective_parent_uri
- candidate_uri = VikingURI(base_uri).join(final_doc_name).uri
+ planned_uri, unique_candidate_uri = await self.resolve_target_uri(
+ ctx=ctx,
+ doc_name=original_name,
+ scope=scope,
+ to_uri=to_uri,
+ parent_uri=parent_uri,
+ source_path=source_path,
+ source_format=source_format,
+ create_parent=create_parent,
+ )
tree = BuildingTree(
source_path=source_path,
source_format=source_format,
)
- tree._root_uri = candidate_uri
- if not to_uri or use_to_as_parent:
- tree._candidate_uri = candidate_uri
+ tree._root_uri = planned_uri
+ if unique_candidate_uri:
+ tree._candidate_uri = unique_candidate_uri
# Create a minimal Context object for the root so that tree.root is not None
- root_context = Context(uri=candidate_uri, temp_uri=temp_doc_uri)
+ root_context = Context(uri=planned_uri, temp_uri=temp_doc_uri)
tree.add_context(root_context)
return tree
diff --git a/openviking/server/routers/resources.py b/openviking/server/routers/resources.py
index 5b95ba021..d4983ce7d 100644
--- a/openviking/server/routers/resources.py
+++ b/openviking/server/routers/resources.py
@@ -199,6 +199,12 @@ async def add_resource(
allow_local_path_resolution = False
original_filename = None
resolved = None
+ store = None
+
+ # Resolve path variables before passing to service.
+ to = resolve_path_variables(request.to) if request.to else None
+ parent = resolve_path_variables(request.parent) if request.parent else None
+
if request.temp_file_id:
store = TempUploadStore.build(http_request.app.state.config)
resolved = await store.resolve_for_consume(request.temp_file_id, _ctx)
@@ -228,11 +234,16 @@ async def add_resource(
if request.preserve_structure is not None:
kwargs["preserve_structure"] = request.preserve_structure
- # Resolve path variables before passing to service
- to = resolve_path_variables(request.to) if request.to else None
- parent = resolve_path_variables(request.parent) if request.parent else None
-
- store = TempUploadStore.build(http_request.app.state.config) if resolved else None
+ async def _cleanup_resolved(success: bool) -> None:
+ if not resolved or not store:
+ return
+ try:
+ if success:
+ await store.mark_consumed(resolved, _ctx)
+ else:
+ await store.mark_failed(resolved, _ctx)
+ finally:
+ await resolved.cleanup()
async def _add() -> dict[str, Any]:
try:
@@ -247,19 +258,17 @@ async def add_resource(
timeout=request.timeout,
allow_local_path_resolution=allow_local_path_resolution,
enforce_public_remote_targets=True,
+ source_cleanup=_cleanup_resolved if resolved and not request.wait else None,
**kwargs,
)
except Exception:
- if resolved and store:
- await store.mark_failed(resolved, _ctx)
+ if request.wait:
+ await _cleanup_resolved(False)
raise
else:
- if resolved and store:
- await store.mark_consumed(resolved, _ctx)
+ if request.wait:
+ await _cleanup_resolved(True)
return result
- finally:
- if resolved:
- await resolved.cleanup()
execution = await run_operation(
operation="resources.add_resource",
diff --git a/openviking/service/core.py b/openviking/service/core.py
index f2428cb05..8f73d583a 100644
--- a/openviking/service/core.py
+++ b/openviking/service/core.py
@@ -375,6 +375,8 @@ class OpenVikingService:
async def close(self) -> None:
"""Close OpenViking and release resources."""
+ await self._resource_service.close_background_tasks()
+
if self._watch_scheduler:
await self._watch_scheduler.stop()
self._watch_scheduler = None
diff --git a/openviking/service/resource_service.py b/openviking/service/resource_service.py
index 63041a1c6..1f5c56c1c 100644
--- a/openviking/service/resource_service.py
+++ b/openviking/service/resource_service.py
@@ -7,8 +7,14 @@ Provides resource management operations: add_resource, add_skill, wait_processed
"""
import asyncio
+import contextlib
+import inspect
import json
+import os
import time
+from collections.abc import Awaitable, Callable
+from dataclasses import dataclass
+from pathlib import Path
from typing import TYPE_CHECKING, Any, Dict, List, Optional
from openviking.core.path_variables import resolve_path_variables
@@ -20,6 +26,7 @@ from openviking.server.local_input_guard import (
)
from openviking.storage import VikingDBManager
from openviking.storage.queuefs import get_queue_manager
+from openviking.storage.transaction import NO_LOCK, LockLease
from openviking.storage.viking_fs import VikingFS
from openviking.telemetry import get_current_telemetry
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
@@ -27,8 +34,11 @@ from openviking.telemetry.resource_summary import (
build_queue_status_payload,
record_resource_wait_metrics,
register_wait_telemetry,
+ summarize_queue_errors,
unregister_wait_telemetry,
)
+from openviking.utils import is_git_repo_url, parse_code_hosting_url
+from openviking.utils.media_processor import _smart_stem
from openviking.utils.network_guard import ensure_public_remote_target
from openviking.utils.resource_processor import ResourceProcessor
from openviking.utils.skill_processor import SkillProcessor
@@ -37,6 +47,7 @@ from openviking_cli.exceptions import (
DeadlineExceededError,
InvalidArgumentError,
NotInitializedError,
+ OpenVikingError,
)
from openviking_cli.utils import get_logger
@@ -47,6 +58,13 @@ if TYPE_CHECKING:
logger = get_logger(__name__)
+@dataclass
+class _ResourceSourceInfo:
+ source_name: Optional[str] = None
+ source_path: Optional[str] = None
+ source_format: Optional[str] = None
+
+
class ResourceService:
"""Resource management service."""
@@ -63,6 +81,7 @@ class ResourceService:
self._resource_processor = resource_processor
self._skill_processor = skill_processor
self._watch_scheduler = watch_scheduler
+ self._background_tasks: set[asyncio.Task[Any]] = set()
def set_dependencies(
self,
@@ -103,6 +122,378 @@ class ResourceService:
if not self._viking_fs:
raise NotInitializedError("VikingFS")
+ async def close_background_tasks(self) -> None:
+ """Cancel in-flight background resource ingestion tasks during service shutdown."""
+ if not self._background_tasks:
+ return
+ tasks = list(self._background_tasks)
+ for task in tasks:
+ task.cancel()
+ await asyncio.gather(*tasks, return_exceptions=True)
+ self._background_tasks.clear()
+
+ async def enqueue_add_resource(
+ self,
+ path: str,
+ ctx: RequestContext,
+ to: Optional[str] = None,
+ parent: Optional[str] = None,
+ reason: str = "",
+ instruction: str = "",
+ timeout: Optional[float] = None,
+ build_index: bool = True,
+ summarize: bool = False,
+ watch_interval: float = 0,
+ skip_watch_management: bool = False,
+ allow_local_path_resolution: bool = True,
+ enforce_public_remote_targets: bool = False,
+ source_cleanup: Optional[Callable[[bool], Awaitable[None]]] = None,
+ **kwargs,
+ ) -> Dict[str, Any]:
+ """Start background resource ingestion while preserving the normal result shape."""
+ self._ensure_initialized()
+
+ if to:
+ to = resolve_path_variables(to)
+ if parent:
+ parent = resolve_path_variables(parent)
+ to = validate_optional_viking_uri(
+ to,
+ field_name="to",
+ allowed_scopes={"resources"},
+ )
+ parent = validate_optional_viking_uri(
+ parent,
+ field_name="parent",
+ allowed_scopes={"resources"},
+ )
+
+ from openviking.service.task_tracker import get_task_tracker
+
+ resource_lock: LockLease = NO_LOCK
+ try:
+ if enforce_public_remote_targets and is_remote_resource_source(path):
+ path = require_remote_resource_source(path)
+ kwargs.setdefault("request_validator", ensure_public_remote_target)
+
+ source_info = await self._preflight_resource_source(
+ path,
+ allow_local_path_resolution=allow_local_path_resolution,
+ request_validator=kwargs.get("request_validator"),
+ )
+ source_name = kwargs.get("source_name") or source_info.source_name
+ if source_name:
+ kwargs["source_name"] = source_name
+ root_uri, resource_lock = await self._plan_resource_target(
+ path=path,
+ ctx=ctx,
+ to=to,
+ parent=parent,
+ source_name=source_name,
+ source_info=source_info,
+ create_parent=bool(kwargs.get("create_parent", False)),
+ )
+
+ task_tracker = get_task_tracker()
+ task = await task_tracker.create(
+ "add_resource",
+ resource_id=root_uri,
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ )
+ await task_tracker.update_stage(
+ task.task_id,
+ "queued",
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ )
+
+ add_kwargs = dict(
+ kwargs,
+ path=path,
+ ctx=ctx,
+ to=root_uri,
+ parent=None,
+ reason=reason,
+ instruction=instruction,
+ timeout=timeout,
+ build_index=build_index,
+ summarize=summarize,
+ watch_interval=watch_interval,
+ skip_watch_management=skip_watch_management,
+ allow_local_path_resolution=allow_local_path_resolution,
+ enforce_public_remote_targets=enforce_public_remote_targets,
+ )
+ background = asyncio.create_task(
+ self._run_add_resource_task(
+ task.task_id,
+ ctx=ctx,
+ add_kwargs=add_kwargs,
+ resource_lock=resource_lock,
+ source_cleanup=source_cleanup,
+ )
+ )
+ resource_lock = NO_LOCK
+ self._background_tasks.add(background)
+ background.add_done_callback(self._background_tasks.discard)
+ return {
+ "status": "success",
+ "root_uri": root_uri,
+ "task_id": task.task_id,
+ }
+ except Exception:
+ await resource_lock.close()
+ if source_cleanup is not None:
+ with contextlib.suppress(Exception):
+ await source_cleanup(False)
+ raise
+
+ async def _run_add_resource_task(
+ self,
+ task_id: str,
+ *,
+ ctx: RequestContext,
+ add_kwargs: Dict[str, Any],
+ resource_lock: LockLease,
+ source_cleanup: Optional[Callable[[bool], Awaitable[None]]] = None,
+ ) -> None:
+ from openviking.service.task_tracker import get_task_tracker
+
+ task_tracker = get_task_tracker()
+
+ async def _set_stage(stage: str) -> None:
+ await task_tracker.update_stage(
+ task_id,
+ stage,
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ )
+
+ success = False
+ try:
+ await task_tracker.start(
+ task_id,
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ stage="queued",
+ )
+ result = await self.add_resource(
+ wait=True,
+ resource_lock=resource_lock,
+ stage_callback=_set_stage,
+ **add_kwargs,
+ )
+ if result.get("status") == "error":
+ errors = result.get("errors") or ["resource processing failed"]
+ await task_tracker.fail(
+ task_id,
+ "; ".join(str(error) for error in errors),
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ )
+ return
+ queue_errors = summarize_queue_errors(result.get("queue_status"))
+ if queue_errors:
+ await task_tracker.fail(
+ task_id,
+ "queue processing failed: " + "; ".join(queue_errors),
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ )
+ return
+ success = True
+ await task_tracker.complete(
+ task_id,
+ result,
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ )
+ except asyncio.CancelledError:
+ await task_tracker.fail(
+ task_id,
+ "background resource ingestion cancelled",
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ )
+ raise
+ except Exception as exc:
+ await task_tracker.fail(
+ task_id,
+ str(exc),
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ )
+ finally:
+ await resource_lock.close()
+ if source_cleanup is not None:
+ try:
+ await source_cleanup(success)
+ except Exception as exc:
+ logger.warning(
+ "[ResourceService] Failed to cleanup async resource source for task %s: %s",
+ task_id,
+ exc,
+ )
+
+ async def _plan_resource_target(
+ self,
+ *,
+ path: str,
+ ctx: RequestContext,
+ to: Optional[str],
+ parent: Optional[str],
+ source_name: Optional[str],
+ source_info: _ResourceSourceInfo,
+ create_parent: bool,
+ ) -> tuple[str, LockLease]:
+ if not self._resource_processor or not self._viking_fs:
+ raise NotInitializedError("ResourceProcessor")
+
+ doc_name = self._target_doc_name(path, source_name, source_info)
+ source_path = source_info.source_path or source_name or path
+ root_uri, candidate_uri = await self._resource_processor.tree_builder.resolve_target_uri(
+ ctx=ctx,
+ doc_name=doc_name,
+ scope="resources",
+ to_uri=to,
+ parent_uri=parent,
+ source_path=source_path,
+ source_format=source_info.source_format,
+ create_parent=create_parent,
+ )
+ if candidate_uri:
+ return await self._resource_processor.reserve_unique_candidate(
+ candidate_uri=candidate_uri,
+ ctx=ctx,
+ )
+
+ from openviking.storage.transaction import get_lock_manager
+
+ dst_path = self._viking_fs._uri_to_path(root_uri, ctx=ctx)
+ resource_lock = await self._resource_processor.acquire_resource_lock(
+ get_lock_manager(),
+ dst_path,
+ uri=root_uri,
+ timeout=0.0,
+ )
+ return root_uri, resource_lock
+
+ @staticmethod
+ def _target_doc_name(
+ path: str,
+ source_name: Optional[str],
+ source_info: _ResourceSourceInfo,
+ ) -> str:
+ if source_name:
+ return _smart_stem(source_name)
+ if source_info.source_name:
+ return _smart_stem(source_info.source_name)
+ if source_info.source_format == "repository":
+ parsed = parse_code_hosting_url(path)
+ if parsed:
+ return parsed.rsplit("/", 1)[-1]
+ return _smart_stem(Path(path).name or "resource")
+
+ async def _preflight_resource_source(
+ self,
+ path: str,
+ *,
+ allow_local_path_resolution: bool,
+ request_validator: Optional[Callable[[str], None]] = None,
+ ) -> _ResourceSourceInfo:
+ if not is_remote_resource_source(path):
+ if allow_local_path_resolution and not Path(path).expanduser().exists():
+ raise InvalidArgumentError(f"Resource source does not exist: {path}")
+ return _ResourceSourceInfo(source_name=Path(path).name)
+ if is_git_repo_url(path):
+ return await self._preflight_git_source(path)
+ if path.startswith(("http://", "https://")):
+ return await self._preflight_http_source(path, request_validator=request_validator)
+ return await self._preflight_git_source(path)
+
+ async def _preflight_http_source(
+ self,
+ url: str,
+ *,
+ request_validator: Optional[Callable[[str], None]] = None,
+ ) -> _ResourceSourceInfo:
+ from openviking.parse.accessors.http_accessor import HTTPAccessor
+ from openviking.parse.base import lazy_import
+ from openviking.utils.network_guard import build_httpx_request_validation_hooks
+
+ httpx = lazy_import("httpx")
+ event_hooks = build_httpx_request_validation_hooks(request_validator)
+ client_kwargs: Dict[str, Any] = {
+ "timeout": 5.0,
+ "follow_redirects": True,
+ }
+ if event_hooks:
+ client_kwargs["event_hooks"] = event_hooks
+ client_kwargs["trust_env"] = False
+ headers = {"User-Agent": "OpenViking"}
+ if "github.com" in url and os.environ.get("GITHUB_TOKEN"):
+ headers["Authorization"] = f"token {os.environ['GITHUB_TOKEN']}"
+
+ try:
+ async with httpx.AsyncClient(**client_kwargs) as client:
+ response = await client.head(url, headers=headers)
+ if response.status_code < 400:
+ response.raise_for_status()
+ else:
+ await self._probe_http_get(client, url, headers)
+ except OpenVikingError:
+ raise
+ except Exception as exc:
+ raise InvalidArgumentError(f"Cannot access remote URL: {url}. {exc}") from exc
+
+ source_name = HTTPAccessor._extract_filename_from_url(url)
+ return _ResourceSourceInfo(
+ source_name=source_name,
+ source_path=source_name or url,
+ )
+
+ @staticmethod
+ async def _probe_http_get(client: Any, url: str, headers: Dict[str, str]) -> None:
+ probe_headers = dict(headers)
+ probe_headers["Range"] = "bytes=0-0"
+ async with client.stream("GET", url, headers=probe_headers) as response:
+ response.raise_for_status()
+ async for _ in response.aiter_bytes():
+ break
+
+ async def _preflight_git_source(self, source: str) -> _ResourceSourceInfo:
+ try:
+ proc = await asyncio.create_subprocess_exec(
+ "git",
+ "ls-remote",
+ "--heads",
+ source,
+ stdout=asyncio.subprocess.PIPE,
+ stderr=asyncio.subprocess.PIPE,
+ )
+ stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=10.0)
+ except TimeoutError as exc:
+ with contextlib.suppress(Exception):
+ proc.kill() # type: ignore[possibly-undefined]
+ await proc.communicate() # type: ignore[possibly-undefined]
+ raise InvalidArgumentError(
+ f"Cannot access Git repository: {source}. The check timed out after 10s."
+ ) from exc
+ except Exception as exc:
+ raise InvalidArgumentError(f"Cannot access Git repository: {source}. {exc}") from exc
+
+ if proc.returncode != 0:
+ detail = (stderr or stdout).decode("utf-8", errors="replace").strip()
+ raise InvalidArgumentError(
+ f"Cannot access Git repository: {source}. {detail or 'git ls-remote failed'}"
+ )
+ repo_name = parse_code_hosting_url(source)
+ return _ResourceSourceInfo(
+ source_name=repo_name.rsplit("/", 1)[-1] if repo_name else None,
+ source_path=source,
+ source_format="repository",
+ )
+
async def add_resource(
self,
path: str,
@@ -119,6 +510,9 @@ class ResourceService:
skip_watch_management: bool = False,
allow_local_path_resolution: bool = True,
enforce_public_remote_targets: bool = False,
+ source_cleanup: Optional[Callable[[bool], Awaitable[None]]] = None,
+ resource_lock: Optional[LockLease] = None,
+ stage_callback: Optional[Callable[[str], Any]] = None,
**kwargs,
) -> Dict[str, Any]:
"""Add resource to OpenViking (only supports resources scope).
@@ -158,11 +552,29 @@ class ResourceService:
InvalidArgumentError: If the URI scope is not 'resources'
"""
self._ensure_initialized()
+ if not wait:
+ return await self.enqueue_add_resource(
+ path=path,
+ ctx=ctx,
+ to=to,
+ parent=parent,
+ reason=reason,
+ instruction=instruction,
+ timeout=timeout,
+ build_index=build_index,
+ summarize=summarize,
+ watch_interval=watch_interval,
+ skip_watch_management=skip_watch_management,
+ allow_local_path_resolution=allow_local_path_resolution,
+ enforce_public_remote_targets=enforce_public_remote_targets,
+ source_cleanup=source_cleanup,
+ **kwargs,
+ )
+
request_start = time.perf_counter()
telemetry = get_current_telemetry()
telemetry_id = register_wait_telemetry(wait)
request_wait_tracker = get_request_wait_tracker()
- monitor_started = False
if telemetry_id:
request_wait_tracker.register_request(telemetry_id)
watch_manager = self._get_watch_manager()
@@ -193,6 +605,8 @@ class ResourceService:
if enforce_public_remote_targets and is_remote_resource_source(path):
path = require_remote_resource_source(path)
kwargs.setdefault("request_validator", ensure_public_remote_target)
+ if resource_lock is not None:
+ kwargs["resource_lock"] = resource_lock
result = await self._resource_processor.process_resource(
path=path,
@@ -204,53 +618,57 @@ class ResourceService:
parent=parent,
build_index=build_index,
summarize=summarize,
+ stage_callback=stage_callback,
allow_local_path_resolution=allow_local_path_resolution,
**kwargs,
)
if result.get("status") == "error":
return result
- elif wait:
- wait_start = time.perf_counter()
- try:
- with telemetry.measure("resource.wait"):
- if telemetry_id:
- await request_wait_tracker.wait_for_request(
- telemetry_id, timeout=timeout
- )
- status = request_wait_tracker.build_queue_status(telemetry_id)
- else:
- qm = get_queue_manager()
- status = build_queue_status_payload(
- await qm.wait_complete(timeout=timeout)
- )
- except TimeoutError as exc:
- telemetry.set_error(
- "resource_service.wait_complete",
- "DEADLINE_EXCEEDED",
- str(exc),
- )
- raise DeadlineExceededError("queue processing", timeout) from exc
- queue_wait_duration_ms = round((time.perf_counter() - wait_start) * 1000, 3)
- try:
- from openviking.metrics.datasources.resource import (
- ResourceIngestionEventDataSource,
- )
+ if stage_callback is not None:
+ stage_result = stage_callback("processing_queue")
+ if inspect.isawaitable(stage_result):
+ await stage_result
+ wait_start = time.perf_counter()
+ try:
+ with telemetry.measure("resource.wait"):
+ if telemetry_id:
+ await request_wait_tracker.wait_for_request(
+ telemetry_id,
+ timeout=timeout,
+ poll_interval=0.05,
+ )
+ status = request_wait_tracker.build_queue_status(telemetry_id)
+ else:
+ qm = get_queue_manager()
+ status = build_queue_status_payload(await qm.wait_complete(timeout=timeout))
+ except TimeoutError as exc:
+ telemetry.set_error(
+ "resource_service.wait_complete",
+ "DEADLINE_EXCEEDED",
+ str(exc),
+ )
+ raise DeadlineExceededError("queue processing", timeout) from exc
+ queue_wait_duration_ms = round((time.perf_counter() - wait_start) * 1000, 3)
+ try:
+ from openviking.metrics.datasources.resource import (
+ ResourceIngestionEventDataSource,
+ )
- ResourceIngestionEventDataSource.record_wait(
- operation="queue_processing",
- duration_seconds=float(queue_wait_duration_ms) / 1000.0,
- account_id=getattr(ctx, "account_id", None),
- )
- except Exception:
- pass
- result["queue_status"] = status
- record_resource_wait_metrics(
- telemetry_id=telemetry_id,
- queue_status=status,
- root_uri=result.get("root_uri"),
+ ResourceIngestionEventDataSource.record_wait(
+ operation="queue_processing",
+ duration_seconds=float(queue_wait_duration_ms) / 1000.0,
+ account_id=getattr(ctx, "account_id", None),
)
- telemetry.set("queue.wait.duration_ms", queue_wait_duration_ms)
+ except Exception:
+ pass
+ result["queue_status"] = status
+ record_resource_wait_metrics(
+ telemetry_id=telemetry_id,
+ queue_status=status,
+ root_uri=result.get("root_uri"),
+ )
+ telemetry.set("queue.wait.duration_ms", queue_wait_duration_ms)
if watch_manager and not skip_watch_management:
with telemetry.measure("resource.watch"):
if watch_interval > 0:
@@ -295,38 +713,6 @@ class ResourceService:
logger.warning(
f"[ResourceService] Failed to cancel watch task for {to}: {e}"
)
- if not wait:
- from openviking.service.task_tracker import get_task_tracker
-
- task_tracker = get_task_tracker()
- root_uri = result.get("root_uri", "")
- task = await task_tracker.create(
- "add_resource",
- resource_id=root_uri,
- account_id=ctx.account_id,
- user_id=ctx.user.user_id,
- )
- result["task_id"] = task.task_id
- if telemetry_id:
- monitor_started = True
- asyncio.create_task(
- self._monitor_queue_processing(
- task.task_id,
- telemetry_id,
- ctx.account_id,
- ctx.user.user_id,
- )
- )
- else:
- await task_tracker.start(
- task.task_id, account_id=ctx.account_id, user_id=ctx.user.user_id
- )
- await task_tracker.complete(
- task.task_id,
- {"root_uri": root_uri},
- account_id=ctx.account_id,
- user_id=ctx.user.user_id,
- )
return result
except Exception as exc:
telemetry.set_error(
@@ -340,9 +726,8 @@ class ResourceService:
"resource.request.duration_ms",
round((time.perf_counter() - request_start) * 1000, 3),
)
- if wait or not telemetry_id or not monitor_started:
- get_request_wait_tracker().cleanup(telemetry_id)
- unregister_wait_telemetry(telemetry_id)
+ get_request_wait_tracker().cleanup(telemetry_id)
+ unregister_wait_telemetry(telemetry_id)
async def _monitor_queue_processing(
self,
diff --git a/openviking/service/task_store.py b/openviking/service/task_store.py
index 770e9c8d5..384708259 100644
--- a/openviking/service/task_store.py
+++ b/openviking/service/task_store.py
@@ -34,57 +34,12 @@ class TaskStore(Protocol):
) -> None: ...
-class InMemoryTaskStore:
- """Simple in-process task store."""
-
- def __init__(self) -> None:
- self._tasks: Dict[str, Dict[str, Any]] = {}
-
- async def create(self, task: Any) -> None:
- self._tasks[task.task_id] = _task_to_payload(task)
-
- async def update(self, task: Any) -> None:
- self._tasks[task.task_id] = _task_to_payload(task)
-
- async def get(
- self,
- task_id: str,
- *,
- account_id: Optional[str] = None,
- user_id: Optional[str] = None,
- ) -> Optional[Dict[str, Any]]:
- payload = self._tasks.get(task_id)
- if payload is None:
- return None
- if account_id is not None and payload.get("account_id") != account_id:
- return None
- if user_id is not None and payload.get("user_id") != user_id:
- return None
- return deepcopy(payload)
-
- async def list(self, account_id: str, *, user_id: Optional[str] = None) -> List[Dict[str, Any]]:
- return [
- deepcopy(payload)
- for payload in self._tasks.values()
- if payload.get("account_id") == account_id
- and (user_id is None or payload.get("user_id") == user_id)
- ]
-
- async def delete(self, task_id: str, *, account_id: str, user_id: Optional[str] = None) -> None:
- payload = self._tasks.get(task_id)
- if (
- payload
- and payload.get("account_id") == account_id
- and (user_id is None or payload.get("user_id") == user_id)
- ):
- del self._tasks[task_id]
-
-
class PersistentTaskStore:
- """Persist task records into AGFS under account-scoped task directories."""
+ """Persist task records into AGFS under account-scoped system task directories."""
ROOT_PREFIX = "/local"
- RESERVED_DIRNAME = "tasks"
+ SYSTEM_DIRNAME = "_system"
+ TASKS_DIRNAME = "tasks"
def __init__(self, agfs: Any) -> None:
self._agfs = agfs if isinstance(agfs, AsyncAGFSClient) else AsyncAGFSClient(agfs)
@@ -149,6 +104,7 @@ class PersistentTaskStore:
async def _ensure_task_dir(self, account_id: str, user_id: str) -> None:
await self._mkdir_if_missing(self._account_dir(account_id))
+ await self._mkdir_if_missing(self._system_dir(account_id))
await self._mkdir_if_missing(self._task_root_dir(account_id))
await self._mkdir_if_missing(self._task_dir(account_id, user_id))
@@ -165,8 +121,11 @@ class PersistentTaskStore:
def _account_dir(self, account_id: str) -> str:
return f"{self.ROOT_PREFIX}/{account_id}"
+ def _system_dir(self, account_id: str) -> str:
+ return f"{self._account_dir(account_id)}/{self.SYSTEM_DIRNAME}"
+
def _task_root_dir(self, account_id: str) -> str:
- return f"{self._account_dir(account_id)}/{self.RESERVED_DIRNAME}"
+ return f"{self._system_dir(account_id)}/{self.TASKS_DIRNAME}"
def _task_dir(self, account_id: str, user_id: str) -> str:
return f"{self._task_root_dir(account_id)}/{user_id}"
@@ -186,6 +145,7 @@ def _task_to_payload(task: Any) -> Dict[str, Any]:
"resource_id": task.resource_id,
"account_id": task.account_id,
"user_id": task.user_id,
+ "stage": task.stage,
"result": deepcopy(task.result),
"error": task.error,
}
diff --git a/openviking/service/task_tracker.py b/openviking/service/task_tracker.py
index b4cae3667..f1c243f6b 100644
--- a/openviking/service/task_tracker.py
+++ b/openviking/service/task_tracker.py
@@ -9,7 +9,7 @@ polled via the /tasks API to check completion status, results, or errors.
Design decisions:
- Thread-safe (QueueManager workers run in separate threads).
- - TTL-based cleanup still applies to in-memory cache.
+ - TTL-based cleanup applies to the process-local cache.
- Error messages are sanitized to avoid leaking sensitive data.
"""
@@ -24,7 +24,7 @@ from enum import Enum
from typing import Any, Dict, List, Optional
from uuid import uuid4
-from openviking.service.task_store import InMemoryTaskStore, TaskStore
+from openviking.service.task_store import TaskStore
from openviking_cli.utils.logger import get_logger
logger = get_logger(__name__)
@@ -51,6 +51,7 @@ class TaskRecord:
resource_id: Optional[str] = None # e.g. session_id
account_id: Optional[str] = None
user_id: Optional[str] = None
+ stage: Optional[str] = None
result: Optional[Dict[str, Any]] = None
error: Optional[str] = None
@@ -72,12 +73,12 @@ _init_lock = threading.Lock()
def get_task_tracker() -> "TaskTracker":
- """Get or create the global TaskTracker singleton."""
+ """Get or create the global persistent TaskTracker singleton."""
global _instance
if _instance is None:
with _init_lock:
if _instance is None:
- _instance = TaskTracker()
+ _instance = _build_default_task_tracker()
return _instance
@@ -116,7 +117,7 @@ def _sanitize_error(error: str) -> str:
class TaskTracker:
- """Async task tracker with pluggable storage and in-memory compatibility cache.
+ """Async task tracker with persistent storage and a process-local cache.
Async lifecycle operations are serialized by ``_async_lock``. The thread
lock only protects sync snapshot reads of the local cache.
@@ -127,8 +128,8 @@ class TaskTracker:
TTL_FAILED = 604_800 # 7 days
CLEANUP_INTERVAL = 300 # 5 minutes
- def __init__(self, store: Optional[TaskStore] = None) -> None:
- self._store = store or InMemoryTaskStore()
+ def __init__(self, store: TaskStore) -> None:
+ self._store = store
self._tasks: Dict[str, TaskRecord] = {}
self._lock = threading.Lock()
self._async_lock = asyncio.Lock()
@@ -172,7 +173,6 @@ class TaskTracker:
"""Remove expired tasks and enforce MAX_TASKS."""
now = time.time()
async with self._async_lock:
- deleted_tasks: List[TaskRecord] = []
with self._lock:
expired_ids = []
for tid, t in self._tasks.items():
@@ -185,25 +185,13 @@ class TaskTracker:
expired_ids.append(tid)
for tid in expired_ids:
- task = self._tasks.pop(tid, None)
- if task is not None:
- deleted_tasks.append(task)
+ self._tasks.pop(tid, None)
if len(self._tasks) > self.MAX_TASKS:
sorted_tasks = sorted(self._tasks.items(), key=lambda x: x[1].created_at)
excess = len(self._tasks) - self.MAX_TASKS
for tid, _ in sorted_tasks[:excess]:
- task = self._tasks.pop(tid, None)
- if task is not None:
- deleted_tasks.append(task)
-
- for task in deleted_tasks:
- if isinstance(self._store, InMemoryTaskStore) and task.account_id:
- await self._store.delete(
- task.task_id,
- account_id=task.account_id,
- user_id=task.user_id,
- )
+ self._tasks.pop(tid, None)
if expired_ids:
logger.debug("[TaskTracker] Evicted %d expired tasks", len(expired_ids))
@@ -311,12 +299,32 @@ class TaskTracker:
task_id: str,
account_id: Optional[str] = None,
user_id: Optional[str] = None,
+ stage: Optional[str] = None,
) -> None:
"""Transition task to RUNNING."""
async with self._async_lock:
task = await self._load_for_update(task_id, account_id, user_id)
if task:
task.status = TaskStatus.RUNNING
+ if stage is not None:
+ task.stage = stage
+ task.updated_at = time.time()
+ await self._store.update(task)
+ with self._lock:
+ self._tasks[task.task_id] = task
+
+ async def update_stage(
+ self,
+ task_id: str,
+ stage: str,
+ account_id: Optional[str] = None,
+ user_id: Optional[str] = None,
+ ) -> None:
+ """Update task stage without changing its lifecycle status."""
+ async with self._async_lock:
+ task = await self._load_for_update(task_id, account_id, user_id)
+ if task:
+ task.stage = stage
task.updated_at = time.time()
await self._store.update(task)
with self._lock:
@@ -492,3 +500,13 @@ class TaskTracker:
for t in tasks:
grouped[t.task_type][t.status.value] += 1
return {k: dict(v) for k, v in grouped.items()}
+
+
+def _build_default_task_tracker() -> TaskTracker:
+ """Build the default persistent tracker from the active storage config."""
+ from openviking.utils.agfs_utils import create_agfs_client
+ from openviking_cli.utils.config import get_openviking_config
+
+ config = get_openviking_config()
+ agfs = create_agfs_client(config.storage.agfs)
+ return config.storage.build_task_tracker(agfs)
diff --git a/openviking/telemetry/resource_summary.py b/openviking/telemetry/resource_summary.py
index bef74506b..b88ac1715 100644
--- a/openviking/telemetry/resource_summary.py
+++ b/openviking/telemetry/resource_summary.py
@@ -4,7 +4,7 @@
from __future__ import annotations
-from typing import Any, Dict
+from typing import Any, Dict, List
from .context import get_current_telemetry
from .operation import OperationTelemetry
@@ -66,6 +66,44 @@ def build_queue_status_payload(status: Dict[str, Any]) -> Dict[str, Dict[str, An
}
+def summarize_queue_errors(queue_status: Dict[str, Any] | None) -> List[str]:
+ """Return human-readable summaries for queue groups with recorded errors."""
+ if not queue_status:
+ return []
+
+ summaries: List[str] = []
+ for name, status in queue_status.items():
+ if isinstance(status, dict):
+ raw_error_count = status.get("error_count", 0)
+ raw_errors = status.get("errors") or []
+ else:
+ raw_error_count = getattr(status, "error_count", 0)
+ raw_errors = getattr(status, "errors", []) or []
+
+ try:
+ error_count = int(raw_error_count or 0)
+ except (TypeError, ValueError):
+ error_count = 0
+ if error_count <= 0:
+ continue
+
+ messages: List[str] = []
+ for error in raw_errors[:3]:
+ if isinstance(error, dict):
+ message = error.get("message")
+ else:
+ message = getattr(error, "message", None) or str(error)
+ if message:
+ messages.append(str(message))
+
+ summary = f"{name} error_count={error_count}"
+ if messages:
+ summary = f"{summary}: {', '.join(messages)}"
+ summaries.append(summary)
+
+ return summaries
+
+
def _resolve_queue_group(
*,
explicit_stats: Any,
@@ -140,5 +178,6 @@ __all__ = [
"build_queue_status_payload",
"record_resource_wait_metrics",
"register_wait_telemetry",
+ "summarize_queue_errors",
"unregister_wait_telemetry",
]
diff --git a/openviking/utils/resource_processor.py b/openviking/utils/resource_processor.py
index fdfa3edee..16c5c9078 100644
--- a/openviking/utils/resource_processor.py
+++ b/openviking/utils/resource_processor.py
@@ -7,7 +7,9 @@ Handles coordinated writes and self-iteration processes
as described in the OpenViking design document.
"""
+import inspect
import time
+from collections.abc import Callable
from typing import TYPE_CHECKING, Any, Dict, List, Optional
from openviking.parse.tree_builder import TreeBuilder
@@ -113,6 +115,7 @@ class ResourceProcessor:
to: Optional[str] = None,
parent: Optional[str] = None,
summarize: bool = False,
+ stage_callback: Optional[Callable[[str], Any]] = None,
**kwargs,
) -> Dict[str, Any]:
"""
@@ -130,8 +133,16 @@ class ResourceProcessor:
"errors": [],
"source_path": None,
}
+ preacquired_lock = kwargs.pop("resource_lock", NO_LOCK) or NO_LOCK
telemetry = get_current_telemetry()
+ async def _set_stage(stage: str) -> None:
+ if stage_callback is None:
+ return
+ result = stage_callback(stage)
+ if inspect.isawaitable(result):
+ await result
+
with telemetry.measure("resource.process"):
# ============ Phase 1: Parse source and writes to temp viking fs ============
try:
@@ -147,6 +158,10 @@ class ResourceProcessor:
# Use reason as instruction fallback so it influences L0/L1
# generation and improves search relevance as documented.
effective_instruction = instruction or reason
+ if path.startswith(("http://", "https://", "git@", "ssh://", "git://")):
+ await _set_stage("fetching")
+ else:
+ await _set_stage("parsing")
with viking_fs.bind_request_context(ctx):
parse_result = await media_processor.process(
source=path,
@@ -207,6 +222,7 @@ class ResourceProcessor:
# ============ Phase 3: TreeBuilder finalizes from temp (scan + move to AGFS) ============
try:
+ await _set_stage("finalizing")
stage_start = time.perf_counter()
stage_status = "ok"
finalize_start = time.perf_counter()
@@ -258,7 +274,7 @@ class ResourceProcessor:
temp_uri = result.get("temp_uri") # temp_doc_uri
original_temp_uri = temp_uri # 保存原始 temp_uri 用于最终输出
candidate_uri = getattr(context_tree, "_candidate_uri", None) if context_tree else None
- resource_lock: LockLease = NO_LOCK
+ resource_lock: LockLease = preacquired_lock
target_preexisting = False
source_committed = False
@@ -271,17 +287,21 @@ class ResourceProcessor:
lock_manager = get_lock_manager()
try:
if candidate_uri:
- root_uri, resource_lock = await self._commit_unique_candidate(
- candidate_uri=candidate_uri,
- ctx=ctx,
- )
- result["root_uri"] = root_uri
+ if resource_lock.active:
+ root_uri = candidate_uri
+ else:
+ root_uri, resource_lock = await self.reserve_unique_candidate(
+ candidate_uri=candidate_uri,
+ ctx=ctx,
+ )
+ result["root_uri"] = root_uri
else:
target_preexisting = await viking_fs.exists(root_uri, ctx=ctx)
- dst_path = viking_fs._uri_to_path(root_uri, ctx=ctx)
- resource_lock = await self._acquire_resource_lock(
- lock_manager, dst_path, uri=root_uri
- )
+ if not resource_lock.active:
+ dst_path = viking_fs._uri_to_path(root_uri, ctx=ctx)
+ resource_lock = await self.acquire_resource_lock(
+ lock_manager, dst_path, uri=root_uri
+ )
if not target_preexisting:
await viking_fs.persist_temp_tree(temp_uri, root_uri, ctx=ctx)
await viking_fs.delete_temp(parse_result.temp_dir_path, ctx=ctx)
@@ -357,7 +377,7 @@ class ResourceProcessor:
return result
- async def _commit_unique_candidate(
+ async def reserve_unique_candidate(
self,
*,
candidate_uri: str,
@@ -378,7 +398,7 @@ class ResourceProcessor:
dst_path = viking_fs._uri_to_path(root_uri, ctx=ctx)
try:
- resource_lock = await self._acquire_resource_lock(
+ resource_lock = await self.acquire_resource_lock(
lock_manager, dst_path, uri=root_uri, timeout=0.0
)
return root_uri, resource_lock
@@ -390,7 +410,7 @@ class ResourceProcessor:
)
@staticmethod
- async def _acquire_resource_lock(
+ async def acquire_resource_lock(
lock_manager,
path: str,
*,
diff --git a/openviking_cli/utils/config/storage_config.py b/openviking_cli/utils/config/storage_config.py
index 1ea24a61a..2fa585400 100644
--- a/openviking_cli/utils/config/storage_config.py
+++ b/openviking_cli/utils/config/storage_config.py
@@ -1,7 +1,7 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
from pathlib import Path
-from typing import Any, Dict, Literal
+from typing import Any, Dict
from pydantic import BaseModel, Field, model_validator
@@ -14,15 +14,6 @@ from .vectordb_config import VectorDBBackendConfig
logger = get_logger(__name__)
-class TaskTrackerConfig(BaseModel):
- """Configuration for async task tracking backend."""
-
- backend: Literal["memory", "persistent"] = Field(
- default="memory",
- description="Task tracker backend. 'persistent' enables cross-instance task lookup.",
- )
-
-
class StorageConfig(BaseModel):
"""Configuration for storage backend.
@@ -45,17 +36,24 @@ class StorageConfig(BaseModel):
description="VectorDB backend configuration",
)
- task_tracker: TaskTrackerConfig = Field(
- default_factory=TaskTrackerConfig,
- description="Task tracker backend configuration",
- )
-
params: Dict[str, Any] = Field(
default_factory=dict, description="Additional storage-specific parameters"
)
model_config = {"extra": "forbid"}
+ @model_validator(mode="before")
+ @classmethod
+ def ignore_deprecated_task_tracker(cls, data: Any) -> Any:
+ if isinstance(data, dict) and "task_tracker" in data:
+ data = dict(data)
+ data.pop("task_tracker", None)
+ logger.warning(
+ "StorageConfig: 'task_tracker' is deprecated and ignored. "
+ "Task records are always persisted."
+ )
+ return data
+
@model_validator(mode="after")
def resolve_paths(self):
if self.agfs.path is not None:
@@ -91,10 +89,8 @@ class StorageConfig(BaseModel):
return upload_temp_dir
def build_task_tracker(self, agfs: Any):
- """Build a TaskTracker from storage config."""
+ """Build the persistent TaskTracker from storage config."""
from openviking.service.task_store import PersistentTaskStore
from openviking.service.task_tracker import TaskTracker
- if self.task_tracker.backend == "memory":
- return TaskTracker()
return TaskTracker(store=PersistentTaskStore(agfs))
diff --git a/tests/misc/test_vikingfs_uri_guard.py b/tests/misc/test_vikingfs_uri_guard.py
index 36382a902..e6a53f6fe 100644
--- a/tests/misc/test_vikingfs_uri_guard.py
+++ b/tests/misc/test_vikingfs_uri_guard.py
@@ -15,7 +15,8 @@ from openviking_cli.exceptions import PermissionDeniedError
def _make_viking_fs() -> VikingFS:
"""Create a VikingFS instance with a mocked AGFS backend."""
fs = VikingFS.__new__(VikingFS)
- fs.agfs = MagicMock()
+ fs.agfs = AsyncMock()
+ fs._async_agfs = fs.agfs
fs.query_embedder = None
fs.rerank_config = None
fs.vector_store = None
@@ -119,8 +120,8 @@ class TestVikingFSURITraversalGuard:
@pytest.mark.asyncio
async def test_read_file_keeps_valid_uri_behavior(self) -> None:
fs = _make_viking_fs()
- fs.agfs.stat = MagicMock(return_value=MagicMock())
- fs.agfs.read = MagicMock(return_value=b"hello")
+ fs.agfs.stat = AsyncMock(return_value=MagicMock())
+ fs.agfs.read = AsyncMock(return_value=b"hello")
content = await fs.read_file("viking://resources/docs/guide.md")
assert content == "hello"
@@ -149,7 +150,7 @@ class TestVikingFSURITraversalGuard:
fs = _make_viking_fs()
fs._encryptor = None
fs._ensure_access = MagicMock()
- fs.agfs.grep = MagicMock(
+ fs.agfs.grep = AsyncMock(
return_value={
"matches": [
{
@@ -175,7 +176,7 @@ class TestVikingFSURITraversalGuard:
fs = _make_viking_fs()
fs._encryptor = None
fs._ensure_access = MagicMock()
- fs.agfs.grep = MagicMock(
+ fs.agfs.grep = AsyncMock(
return_value={
"matches": [
{
@@ -195,7 +196,8 @@ class TestVikingFSURITraversalGuard:
assert result["matches"][0]["line"] == 1
assert result["matches"][0]["content"] == "act"
- def test_ls_entries_hides_reserved_tasks_dir_under_account_root(self) -> None:
+ @pytest.mark.asyncio
+ async def test_ls_entries_filters_account_root_to_listable_scopes(self) -> None:
fs = _make_viking_fs()
fs.agfs.ls.return_value = [
{"name": "resources", "isDir": True},
@@ -203,6 +205,6 @@ class TestVikingFSURITraversalGuard:
{"name": "_system", "isDir": True},
]
- entries = fs._ls_entries("/local/default")
+ entries = await fs._ls_entries("/local/default")
assert [entry["name"] for entry in entries] == ["resources"]
diff --git a/tests/server/test_api_resources.py b/tests/server/test_api_resources.py
index c94378a41..a3a0ea6e1 100644
--- a/tests/server/test_api_resources.py
+++ b/tests/server/test_api_resources.py
@@ -7,11 +7,25 @@ import asyncio
import zipfile
import httpx
+import pytest
from openviking.storage.viking_fs import get_viking_fs
from openviking.telemetry import get_current_telemetry
+async def _wait_task_terminal(client: httpx.AsyncClient, task_id: str, timeout: float = 10.0):
+ deadline = asyncio.get_running_loop().time() + timeout
+ last_result = None
+ while asyncio.get_running_loop().time() < deadline:
+ task_resp = await client.get(f"/api/v1/tasks/{task_id}")
+ assert task_resp.status_code == 200
+ last_result = task_resp.json()["result"]
+ if last_result["status"] in {"completed", "failed"}:
+ return last_result
+ await asyncio.sleep(0.05)
+ raise AssertionError(f"Task {task_id} did not finish; last_result={last_result}")
+
+
async def test_add_resource_success(
client: httpx.AsyncClient,
sample_markdown_file,
@@ -31,8 +45,12 @@ async def test_add_resource_success(
assert "time" not in body
assert "usage" not in body
assert "telemetry" not in body
- assert "root_uri" in body["result"]
+ assert body["result"]["status"] == "success"
assert body["result"]["root_uri"].startswith("viking://")
+ assert body["result"]["task_id"]
+ task = await _wait_task_terminal(client, body["result"]["task_id"])
+ assert task["status"] == "completed"
+ assert task["result"]["root_uri"] == body["result"]["root_uri"]
async def test_add_resource_with_wait(
@@ -155,6 +173,7 @@ async def test_add_resource_business_error_uses_error_envelope(
json={
"path": "https://example.com/bad.md",
"reason": "test resource",
+ "wait": True,
},
)
@@ -200,7 +219,6 @@ async def test_add_skill_missing_name_returns_invalid_argument(client: httpx.Asy
"description": "Skill without name",
"content": "# No Name Skill\nTest content.",
},
- "wait": True,
},
)
@@ -292,6 +310,7 @@ async def test_add_resource_with_to(
"temp_file_id": sample_markdown_file.name,
"to": "viking://resources/custom/sample",
"reason": "test resource",
+ "wait": True,
},
)
assert resp.status_code == 200
@@ -314,6 +333,7 @@ async def test_add_resource_with_resources_root_to_uses_child_uri(
"temp_file_id": archive_path.name,
"to": "viking://resources",
"reason": "test resource root import",
+ "wait": True,
},
)
assert resp.status_code == 200
@@ -336,6 +356,7 @@ async def test_add_resource_with_resources_root_to_trailing_slash_uses_child_uri
"temp_file_id": archive_path.name,
"to": "viking://resources/",
"reason": "test resource root import trailing slash",
+ "wait": True,
},
)
assert resp.status_code == 200
@@ -358,6 +379,7 @@ async def test_add_resource_with_resources_root_to_keeps_single_file_directory(
"source_name": "aa.txt",
"to": "viking://resources",
"reason": "test resource root file import",
+ "wait": True,
},
)
assert resp.status_code == 200
@@ -380,6 +402,7 @@ async def test_add_resource_with_resources_root_to_trailing_slash_keeps_single_f
"source_name": "aa.txt",
"to": "viking://resources/",
"reason": "test resource root file import trailing slash",
+ "wait": True,
},
)
assert resp.status_code == 200
@@ -388,6 +411,47 @@ async def test_add_resource_with_resources_root_to_trailing_slash_keeps_single_f
assert body["result"]["root_uri"] == "viking://resources/aa"
+async def test_add_resource_async_auto_name_reserves_unique_root_uri(
+ client: httpx.AsyncClient,
+ upload_temp_dir,
+):
+ first = upload_temp_dir / "first.md"
+ second = upload_temp_dir / "second.md"
+ first.write_text("# first\n")
+ second.write_text("# second\n")
+
+ first_resp = await client.post(
+ "/api/v1/resources",
+ json={
+ "temp_file_id": first.name,
+ "source_name": "same.md",
+ "reason": "first async resource",
+ "wait": False,
+ },
+ )
+ second_resp = await client.post(
+ "/api/v1/resources",
+ json={
+ "temp_file_id": second.name,
+ "source_name": "same.md",
+ "reason": "second async resource",
+ "wait": False,
+ },
+ )
+
+ assert first_resp.status_code == 200
+ assert second_resp.status_code == 200
+ first_result = first_resp.json()["result"]
+ second_result = second_resp.json()["result"]
+ assert first_result["root_uri"] == "viking://resources/same"
+ assert second_result["root_uri"] == "viking://resources/same_1"
+
+ first_task = await _wait_task_terminal(client, first_result["task_id"])
+ second_task = await _wait_task_terminal(client, second_result["task_id"])
+ assert first_task["result"]["root_uri"] == first_result["root_uri"]
+ assert second_task["result"]["root_uri"] == second_result["root_uri"]
+
+
async def test_wait_processed_empty_queue(client: httpx.AsyncClient):
resp = await client.post(
"/api/v1/system/wait",
@@ -405,7 +469,7 @@ async def test_wait_processed_after_add(
):
await client.post(
"/api/v1/resources",
- json={"temp_file_id": sample_markdown_file.name, "reason": "test"},
+ json={"temp_file_id": sample_markdown_file.name, "reason": "test", "wait": True},
)
resp = await client.post(
"/api/v1/system/wait",
@@ -427,6 +491,7 @@ async def test_add_resource_with_watch_interval_auto_binds_root_uri(
"temp_file_id": sample_markdown_file.name,
"reason": "test resource with watch interval",
"watch_interval": 5.0,
+ "wait": True,
},
)
assert resp.status_code == 200
@@ -455,6 +520,7 @@ async def test_add_resource_with_default_watch_interval(
json={
"temp_file_id": sample_markdown_file.name,
"reason": "test resource with default watch interval",
+ "wait": True,
},
)
assert resp.status_code == 200
@@ -516,7 +582,7 @@ async def test_add_resource_accepts_temp_uploaded_file(
resp = await client.post(
"/api/v1/resources",
- json={"temp_file_id": temp_file_id, "reason": "uploaded resource"},
+ json={"temp_file_id": temp_file_id, "reason": "uploaded resource", "wait": True},
)
assert resp.status_code == 200
body = resp.json()
@@ -545,7 +611,7 @@ async def test_shared_temp_upload_and_add_resource_deletes_upload_dir(
resp = await client.post(
"/api/v1/resources",
- json={"temp_file_id": temp_file_id, "reason": "shared upload"},
+ json={"temp_file_id": temp_file_id, "reason": "shared upload", "wait": True},
)
assert resp.status_code == 200
body = resp.json()
@@ -570,11 +636,11 @@ async def test_shared_temp_upload_failed_consume_is_retryable(
raise RuntimeError("boom")
monkeypatch.setattr(service.resources, "add_resource", fake_add_resource)
- resp = await client.post(
- "/api/v1/resources",
- json={"temp_file_id": temp_file_id, "reason": "shared upload"},
- )
- assert resp.status_code == 500
+ with pytest.raises(RuntimeError, match="boom"):
+ await client.post(
+ "/api/v1/resources",
+ json={"temp_file_id": temp_file_id, "reason": "shared upload", "wait": True},
+ )
upload_id = temp_file_id[len("shared_") :]
meta_uri = f"viking://upload/{upload_id}/meta.json"
@@ -582,7 +648,7 @@ async def test_shared_temp_upload_failed_consume_is_retryable(
assert '"state": "uploaded"' in meta_raw
-async def test_shared_upload_fs_read_is_denied_for_non_root(
+async def test_shared_upload_content_read_rejects_internal_scope(
client: httpx.AsyncClient,
):
upload_resp = await client.post(
@@ -595,13 +661,13 @@ async def test_shared_upload_fs_read_is_denied_for_non_root(
upload_id = temp_file_id[len("shared_") :]
resp = await client.get(
- "/api/v1/fs/read",
+ "/api/v1/content/read",
params={"uri": f"viking://upload/{upload_id}/meta.json"},
)
- assert resp.status_code == 403
+ assert resp.status_code == 400
body = resp.json()
assert body["status"] == "error"
- assert body["error"]["code"] == "PERMISSION_DENIED"
+ assert body["error"]["code"] == "INVALID_URI"
async def test_add_resource_rejects_temp_file_id_directory(
@@ -657,8 +723,13 @@ async def test_add_resource_async_returns_task_id(
assert resp.status_code == 200
body = resp.json()
assert body["status"] == "ok"
+ assert body["result"]["status"] == "success"
assert "task_id" in body["result"]
assert body["result"]["task_id"]
+ assert body["result"]["root_uri"].startswith("viking://")
+ task = await _wait_task_terminal(client, body["result"]["task_id"])
+ assert task["status"] == "completed"
+ assert task["result"]["root_uri"] == body["result"]["root_uri"]
async def test_add_resource_sync_no_task_id(
@@ -698,145 +769,11 @@ async def test_add_resource_async_task_queryable(
},
)
task_id = resp.json()["result"]["task_id"]
+ root_uri = resp.json()["result"]["root_uri"]
- await asyncio.sleep(2.0)
-
- task_resp = await client.get(f"/api/v1/tasks/{task_id}")
- assert task_resp.status_code == 200
- result = task_resp.json()["result"]
+ result = await _wait_task_terminal(client, task_id)
assert result["task_id"] == task_id
assert result["task_type"] == "add_resource"
- assert result["status"] in {"running", "completed", "failed"}
-
-
-async def test_add_resource_async_failure_cleans_up_tracker(
- client: httpx.AsyncClient,
- service,
- monkeypatch,
- upload_temp_dir,
-):
- """Regression: when wait=False and telemetry_id is registered but processor
- raises before task/monitor creation, RequestWaitTracker and telemetry
- registry must not leak state."""
-
- from openviking.server.identity import RequestContext, Role
- from openviking.telemetry.registry import _REGISTERED_TELEMETRY
- from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
- from openviking_cli.session.user_id import UserIdentifier
-
- async def _failing_process_resource(**kwargs):
- raise RuntimeError("processor exploded")
-
- monkeypatch.setattr(
- service.resources._resource_processor, "process_resource", _failing_process_resource
- )
-
- ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)
-
- rwt_before = set(get_request_wait_tracker()._states.keys())
- tel_before = set(_REGISTERED_TELEMETRY.keys())
-
- try:
- await service.resources.add_resource(
- path="/tmp/fail_test.md",
- ctx=ctx,
- reason="failure cleanup test",
- wait=False,
- )
- except RuntimeError:
- pass
-
- rwt_after = set(get_request_wait_tracker()._states.keys())
- tel_after = set(_REGISTERED_TELEMETRY.keys())
-
- leaked_rwt = rwt_after - rwt_before
- assert not leaked_rwt, f"RequestWaitTracker leaked: {leaked_rwt}"
-
- leaked_telemetry = tel_after - tel_before
- assert not leaked_telemetry, f"Telemetry registry leaked: {leaked_telemetry}"
-
-
-async def test_add_resource_business_error_no_task(
- client: httpx.AsyncClient,
- service,
- monkeypatch,
- upload_temp_dir,
-):
- """When process_resource returns status=error, no task should be created."""
-
- from openviking.service.task_tracker import get_task_tracker
-
- async def _error_process_resource(**kwargs):
- return {"status": "error", "message": "unsupported format"}
-
- monkeypatch.setattr(
- service.resources._resource_processor, "process_resource", _error_process_resource
- )
-
- task_count_before = get_task_tracker().count()
-
- await client.post(
- "/api/v1/resources",
- json={
- "temp_file_id": "nonexistent",
- "reason": "test business error",
- "wait": False,
- },
- )
-
- task_count_after = get_task_tracker().count()
- assert task_count_after == task_count_before, "Business error should not create a task"
-
-
-async def test_monitor_marks_failed_on_queue_error(
- service,
- monkeypatch,
-):
- """When queue processing has errors, _monitor_queue_processing should mark task as failed."""
-
- from openviking.server.identity import RequestContext, Role
- from openviking.service.task_tracker import get_task_tracker, reset_task_tracker
- from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
- from openviking_cli.session.user_id import UserIdentifier
-
- reset_task_tracker()
-
- async def _fake_process_resource(**kwargs):
- return {"status": "success", "root_uri": "viking://resources/queue-err-test"}
-
- monkeypatch.setattr(
- service.resources._resource_processor, "process_resource", _fake_process_resource
- )
-
- original_wait_for_request = get_request_wait_tracker().wait_for_request
-
- async def _mock_wait_then_error(telemetry_id, timeout=None, poll_interval=0.05):
- rwt = get_request_wait_tracker()
- with rwt._lock:
- state = rwt._states.get(telemetry_id)
- if state:
- state.semantic_error_count = 1
- state.semantic_errors.append("semantic processing failed")
- state.pending_semantic_roots.clear()
- state.pending_embedding_roots.clear()
- await original_wait_for_request(telemetry_id, timeout=timeout, poll_interval=0.01)
-
- monkeypatch.setattr(get_request_wait_tracker(), "wait_for_request", _mock_wait_then_error)
-
- ctx = RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)
- result = await service.resources.add_resource(
- path="/tmp/queue_err_test.md",
- ctx=ctx,
- reason="queue error test",
- wait=False,
- )
-
- task_id = result.get("task_id")
- assert task_id, "Expected task_id in result"
-
- await asyncio.sleep(1.0)
-
- task = await get_task_tracker().get(task_id)
- assert task is not None
- assert task.status.value == "failed", f"Expected failed, got {task.status.value}"
- assert "queue processing failed" in task.error
+ assert result["status"] == "completed"
+ assert result["stage"] in {"queued", "fetching", "parsing", "finalizing", "processing_queue"}
+ assert result["result"]["root_uri"] == root_uri
diff --git a/tests/server/test_api_search.py b/tests/server/test_api_search.py
index dc1f6b87b..3a028a1f2 100644
--- a/tests/server/test_api_search.py
+++ b/tests/server/test_api_search.py
@@ -697,6 +697,7 @@ async def test_grep_level_limit_filters_by_relative_match_path(
"temp_file_id": root_file.name,
"to": "viking://resources/level-limit/root_level.md",
"reason": "test",
+ "wait": True,
},
)
await client.post(
@@ -705,6 +706,7 @@ async def test_grep_level_limit_filters_by_relative_match_path(
"temp_file_id": deep_file.name,
"to": "viking://resources/level-limit/nested/deeper/deep_level.md",
"reason": "test",
+ "wait": True,
},
)
@@ -736,11 +738,11 @@ async def test_grep_exclude_uri_excludes_specific_uri_range(
await client.post(
"/api/v1/resources",
- json={"temp_file_id": include_file.name, "reason": "include"},
+ json={"temp_file_id": include_file.name, "reason": "include", "wait": True},
)
await client.post(
"/api/v1/resources",
- json={"temp_file_id": exclude_file.name, "reason": "exclude"},
+ json={"temp_file_id": exclude_file.name, "reason": "exclude", "wait": True},
)
root_uri = "viking://resources"
@@ -777,6 +779,7 @@ async def test_grep_exclude_uri_does_not_exclude_same_named_sibling_dirs(
"temp_file_id": group_a_file.name,
"to": "viking://resources/group_a/cache/a.md",
"reason": "test",
+ "wait": True,
},
)
await client.post(
@@ -785,6 +788,7 @@ async def test_grep_exclude_uri_does_not_exclude_same_named_sibling_dirs(
"temp_file_id": group_b_file.name,
"to": "viking://resources/group_b/cache/b.md",
"reason": "test",
+ "wait": True,
},
)
diff --git a/tests/server/test_auth.py b/tests/server/test_auth.py
index 9a848aa4b..f11e72a4f 100644
--- a/tests/server/test_auth.py
+++ b/tests/server/test_auth.py
@@ -23,7 +23,8 @@ from openviking.server.dependencies import set_service
from openviking.server.identity import ResolvedIdentity, Role
from openviking.server.models import ERROR_CODE_TO_HTTP_STATUS, ErrorInfo, Response
from openviking.service.core import OpenVikingService
-from openviking.service.task_tracker import get_task_tracker, reset_task_tracker
+from openviking.service.task_store import PersistentTaskStore
+from openviking.service.task_tracker import TaskTracker, get_task_tracker, reset_task_tracker, set_task_tracker
from openviking_cli.exceptions import InvalidArgumentError, OpenVikingError, PermissionDeniedError
from openviking_cli.session.user_id import UserIdentifier
@@ -32,6 +33,48 @@ def _uid() -> str:
return f"acct_{uuid.uuid4().hex[:8]}"
+class _FakeAgfs:
+ def __init__(self):
+ self.files = {}
+ self.dirs = {"/", "/local"}
+
+ def mkdir(self, path: str, mode: str = "755"):
+ self.dirs.add(path.rstrip("/") or "/")
+ return {"message": "created", "mode": mode}
+
+ def write(self, path: str, data):
+ if isinstance(data, str):
+ data = data.encode("utf-8")
+ self.files[path] = data
+ self.dirs.add(path.rsplit("/", 1)[0] or "/")
+ return "OK"
+
+ def read(self, path: str, offset: int = 0, size: int = -1, stream: bool = False):
+ if path not in self.files:
+ raise FileNotFoundError(path)
+ data = self.files[path]
+ return data[offset : offset + size] if size >= 0 else data[offset:]
+
+ def ls(self, path: str = "/"):
+ prefix = path.rstrip("/") or "/"
+ if prefix not in self.dirs:
+ return []
+ return [
+ {"name": file_path[len(prefix) + 1 :], "path": file_path, "is_dir": False}
+ for file_path in self.files
+ if file_path.startswith(prefix + "/")
+ and "/" not in file_path[len(prefix) + 1 :]
+ ]
+
+ def rm(self, path: str, recursive: bool = False, force: bool = True):
+ self.files.pop(path, None)
+ return {"message": "deleted"}
+
+
+def _set_fake_task_tracker():
+ set_task_tracker(TaskTracker(store=PersistentTaskStore(_FakeAgfs())))
+
+
ROOT_KEY = "root-secret-key-for-testing-only-1234567890abcdef"
@@ -318,6 +361,7 @@ async def test_task_endpoints_require_auth():
async def test_task_endpoints_are_user_scoped():
"""Authenticated callers must not see another user's background tasks."""
reset_task_tracker()
+ _set_fake_task_tracker()
account_id = _uid()
tracker = get_task_tracker()
alice_task = await tracker.create(
diff --git a/tests/service/test_resource_service_tasks.py b/tests/service/test_resource_service_tasks.py
new file mode 100644
index 000000000..1153c2865
--- /dev/null
+++ b/tests/service/test_resource_service_tasks.py
@@ -0,0 +1,105 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+import pytest
+
+from openviking.server.identity import RequestContext, Role
+from openviking.service.resource_service import ResourceService
+from openviking.service.task_tracker import (
+ TaskStatus,
+ TaskTracker,
+ reset_task_tracker,
+ set_task_tracker,
+)
+from openviking.storage.transaction import NO_LOCK
+from openviking_cli.session.user_id import UserIdentifier
+
+pytestmark = pytest.mark.asyncio
+
+
+class _NoopTaskStore:
+ async def create(self, task):
+ return None
+
+ async def update(self, task):
+ return None
+
+ async def get(self, task_id, *, account_id=None, user_id=None):
+ return None
+
+ async def list(self, account_id, *, user_id=None):
+ return []
+
+ async def delete(self, task_id, *, account_id, user_id=None):
+ return None
+
+
[email protected](autouse=True)
+def clean_task_tracker():
+ reset_task_tracker()
+ yield
+ reset_task_tracker()
+
+
+async def test_run_add_resource_task_fails_when_queue_status_has_errors(monkeypatch):
+ tracker = TaskTracker(store=_NoopTaskStore())
+ set_task_tracker(tracker)
+ service = ResourceService()
+ ctx = RequestContext(
+ user=UserIdentifier("acme", "alice", "agent-1"),
+ role=Role.ADMIN,
+ )
+ task = await tracker.create(
+ "add_resource",
+ resource_id="viking://resources/demo",
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ )
+ cleanup_calls = []
+
+ async def fake_add_resource(**kwargs):
+ assert kwargs["wait"] is True
+ return {
+ "status": "success",
+ "root_uri": "viking://resources/demo",
+ "queue_status": {
+ "Semantic": {
+ "processed": 0,
+ "requeue_count": 0,
+ "error_count": 1,
+ "errors": [{"message": "semantic processing failed"}],
+ },
+ "Embedding": {
+ "processed": 0,
+ "requeue_count": 0,
+ "error_count": 0,
+ "errors": [],
+ },
+ },
+ }
+
+ async def cleanup(success: bool):
+ cleanup_calls.append(success)
+
+ monkeypatch.setattr(service, "add_resource", fake_add_resource)
+
+ await service._run_add_resource_task(
+ task.task_id,
+ ctx=ctx,
+ add_kwargs={"path": "/tmp/demo.md", "ctx": ctx},
+ resource_lock=NO_LOCK,
+ source_cleanup=cleanup,
+ )
+
+ stored = await tracker.get(
+ task.task_id,
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ )
+ assert stored is not None
+ assert stored.status == TaskStatus.FAILED
+ assert stored.result is None
+ assert "queue processing failed" in stored.error
+ assert "Semantic error_count=1" in stored.error
+ assert "semantic processing failed" in stored.error
+ assert cleanup_calls == [False]
diff --git a/tests/telemetry/test_request_wait_tracker.py b/tests/telemetry/test_request_wait_tracker.py
index 61640e6d6..fffb5550f 100644
--- a/tests/telemetry/test_request_wait_tracker.py
+++ b/tests/telemetry/test_request_wait_tracker.py
@@ -1,6 +1,8 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
+import pytest
+
from openviking.telemetry.request_wait_tracker import RequestWaitTracker
@@ -50,3 +52,14 @@ def test_request_wait_tracker_records_requeues():
"Semantic": {"processed": 0, "requeue_count": 1, "error_count": 0, "errors": []},
"Embedding": {"processed": 0, "requeue_count": 2, "error_count": 0, "errors": []},
}
+
+
+async def test_wait_for_request_timeout_keeps_existing_error():
+ tracker = RequestWaitTracker()
+ telemetry_id = "tm_wait_timeout"
+
+ tracker.register_request(telemetry_id)
+ tracker.register_embedding_root(telemetry_id, "embedding-1")
+
+ with pytest.raises(TimeoutError, match="Request processing not complete after 0.01s"):
+ await tracker.wait_for_request(telemetry_id, timeout=0.01, poll_interval=0.001)
diff --git a/tests/test_task_backend_config.py b/tests/test_task_backend_config.py
index 10d41b0fe..a1b9c024e 100644
--- a/tests/test_task_backend_config.py
+++ b/tests/test_task_backend_config.py
@@ -1,8 +1,9 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
-from openviking.service.task_store import InMemoryTaskStore, PersistentTaskStore
+from openviking.service.task_store import PersistentTaskStore
from openviking.service.task_tracker import TaskTracker
+from openviking_cli.utils.config import storage_config as storage_config_module
from openviking_cli.utils.config.storage_config import StorageConfig
@@ -23,23 +24,23 @@ class _FakeAgfs:
return {"message": "deleted"}
-def test_storage_config_defaults_task_backend_to_memory():
+def test_storage_config_has_no_task_tracker_config():
config = StorageConfig()
- assert config.task_tracker.backend == "memory"
+ assert not hasattr(config, "task_tracker")
-def test_storage_config_accepts_memory_task_backend():
- config = StorageConfig(task_tracker={"backend": "memory"})
- assert config.task_tracker.backend == "memory"
+def test_storage_config_ignores_deprecated_task_tracker_config(monkeypatch):
+ warnings = []
+ monkeypatch.setattr(storage_config_module.logger, "warning", warnings.append)
+ config = StorageConfig(task_tracker={"backend": "memory"})
-def test_storage_config_builds_memory_task_tracker():
- tracker = StorageConfig(task_tracker={"backend": "memory"}).build_task_tracker(_FakeAgfs())
- assert isinstance(tracker, TaskTracker)
- assert isinstance(tracker._store, InMemoryTaskStore)
+ assert not hasattr(config, "task_tracker")
+ assert any("task_tracker" in message for message in warnings)
+ assert any("deprecated and ignored" in message for message in warnings)
def test_storage_config_builds_persistent_task_tracker():
- tracker = StorageConfig(task_tracker={"backend": "persistent"}).build_task_tracker(_FakeAgfs())
+ tracker = StorageConfig().build_task_tracker(_FakeAgfs())
assert isinstance(tracker, TaskTracker)
assert isinstance(tracker._store, PersistentTaskStore)
diff --git a/tests/test_task_tracker.py b/tests/test_task_tracker.py
index da1ca2501..37b4971fb 100644
--- a/tests/test_task_tracker.py
+++ b/tests/test_task_tracker.py
@@ -11,13 +11,14 @@ import pytest
from openviking.pyagfs.exceptions import AGFSAlreadyExistsError
from openviking.server.identity import RequestContext, Role
from openviking.service.session_service import SessionService
-from openviking.service.task_store import InMemoryTaskStore, PersistentTaskStore
+from openviking.service.task_store import PersistentTaskStore
from openviking.service.task_tracker import (
TaskStatus,
TaskTracker,
_sanitize_error,
get_task_tracker,
reset_task_tracker,
+ set_task_tracker,
)
from openviking_cli.session.user_id import UserIdentifier
@@ -34,7 +35,7 @@ def clean_singleton():
@pytest.fixture
def tracker() -> TaskTracker:
- return TaskTracker()
+ return TaskTracker(store=PersistentTaskStore(_FakeAgfs()))
def _owner_kwargs(account_id: str = "acme", user_id: str = "alice"):
@@ -51,6 +52,12 @@ def _make_ctx(account_id: str = "acme", user_id: str = "alice") -> RequestContex
)
+def _set_fake_global_tracker() -> TaskTracker:
+ tracker = TaskTracker(store=PersistentTaskStore(_FakeAgfs()))
+ set_task_tracker(tracker)
+ return tracker
+
+
class _FakeAgfs:
def __init__(self):
self.files = {}
@@ -124,6 +131,16 @@ async def test_start_task(tracker: TaskTracker):
assert retrieved.status == TaskStatus.RUNNING
+async def test_update_stage(tracker: TaskTracker):
+ task = await tracker.create("add_resource", **_owner_kwargs())
+ await tracker.start(task.task_id, stage="queued")
+ await tracker.update_stage(task.task_id, "parsing")
+ retrieved = await tracker.get(task.task_id)
+ assert retrieved is not None
+ assert retrieved.status == TaskStatus.RUNNING
+ assert retrieved.stage == "parsing"
+
+
async def test_complete_task(tracker: TaskTracker):
task = await tracker.create("session_commit", resource_id="s1", **_owner_kwargs())
await tracker.start(task.task_id)
@@ -300,6 +317,7 @@ async def test_to_dict(tracker: TaskTracker):
assert d["status"] == "pending"
assert d["task_type"] == "session_commit"
assert d["resource_id"] == "s1"
+ assert d["stage"] is None
assert isinstance(d["created_at"], float)
assert isinstance(d["updated_at"], float)
assert isinstance(d["created_at_iso"], str)
@@ -375,15 +393,15 @@ async def test_evict_fifo_when_over_limit(tracker: TaskTracker):
async def test_singleton():
- t1 = get_task_tracker()
+ t1 = _set_fake_global_tracker()
t2 = get_task_tracker()
assert t1 is t2
async def test_singleton_reset():
- t1 = get_task_tracker()
+ t1 = _set_fake_global_tracker()
reset_task_tracker()
- t2 = get_task_tracker()
+ t2 = _set_fake_global_tracker()
assert t1 is not t2
@@ -415,18 +433,19 @@ async def test_persistent_store_writes_task_record_json():
**_owner_kwargs(),
)
- raw = agfs.files[f"/local/acme/tasks/alice/{task.task_id}.json"]
+ raw = agfs.files[f"/local/acme/_system/tasks/alice/{task.task_id}.json"]
payload = json.loads(raw.decode("utf-8"))
assert payload["task_id"] == task.task_id
assert payload["task_type"] == "add_resource"
assert payload["account_id"] == "acme"
assert payload["user_id"] == "alice"
+ assert payload["stage"] is None
assert "schema_version" not in payload
-async def test_inmemory_store_keeps_tasktracker_tasks_dict():
- tracker = TaskTracker(store=InMemoryTaskStore())
+async def test_persistent_store_keeps_tasktracker_tasks_dict():
+ tracker = TaskTracker(store=PersistentTaskStore(_FakeAgfs()))
task = await tracker.create("session_commit", **_owner_kwargs())
assert task.task_id in tracker._tasks
@@ -452,8 +471,8 @@ async def test_persistent_store_ignores_existing_task_dirs():
second = await tracker.create("session_commit", resource_id="sess-2", **_owner_kwargs())
assert first.task_id != second.task_id
- assert agfs.files[f"/local/acme/tasks/alice/{first.task_id}.json"]
- assert agfs.files[f"/local/acme/tasks/alice/{second.task_id}.json"]
+ assert agfs.files[f"/local/acme/_system/tasks/alice/{first.task_id}.json"]
+ assert agfs.files[f"/local/acme/_system/tasks/alice/{second.task_id}.json"]
async def test_create_requires_owner(tracker: TaskTracker):
@@ -477,7 +496,7 @@ async def test_create_rejects_blank_owner_values(tracker: TaskTracker):
async def test_session_service_get_commit_task_is_owner_scoped():
- tracker = get_task_tracker()
+ tracker = _set_fake_global_tracker()
task = await tracker.create("session_commit", resource_id="sess-123", **_owner_kwargs())
service = SessionService()
@@ -491,7 +510,7 @@ async def test_session_service_get_commit_task_is_owner_scoped():
async def test_session_service_get_commit_task_also_filters_account():
- tracker = get_task_tracker()
+ tracker = _set_fake_global_tracker()
task = await tracker.create("session_commit", resource_id="sess-123", **_owner_kwargs())
service = SessionService()