Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

fix(resources): avoid remote preflight for non-git imports (#2459)

Qin Haojie34682c8aaebb
diff --git a/docs/en/about/02-changelog.md b/docs/en/about/02-changelog.md
index 5461635bd..eb94008db 100644
--- a/docs/en/about/02-changelog.md
+++ b/docs/en/about/02-changelog.md
@@ -128,7 +128,7 @@ This changelog is automatically generated from [GitHub Releases](https://github.
- **Native CLI distribution**: New `@openviking/cli` npm package installs the platform `ov` binary via `npm i -g @openviking/cli`; Rust CLI release pipeline adds Linux musl artifacts, npm trusted publishing, and a broader integration test suite.
- **Retrieval and filesystem**: `find` / `search` accept a `level` filter for L0 abstracts, L1 overviews, and L2 file hits. Resource files gained a Phase 1 WebDAV adapter, and `observer.filesystem` exposes filesystem-level observability.
- **Console and Usage/Audit**: New Usage/Audit module and `/api/v1/console/*` BFF aggregate token usage, retrieval counts, context commit heatmaps, request audit logs, and context inventory from the existing observability event bus.
-- **Storage and concurrency reliability**: Strengthened exact-path and lifecycle locks fix content-write races; blocking backend calls moved off the event loop; QueueFS SQLite persistence expanded; task records are now persisted for cross-instance lookup; `add_resource(wait=false)` returns a reserved `root_uri` plus persistent task progress while ingestion completes.
+- **Storage and concurrency reliability**: Strengthened exact-path and lifecycle locks fix content-write races; blocking backend calls moved off the event loop; QueueFS SQLite persistence expanded; task records are now persisted for cross-instance lookup; Git repository `add_resource(wait=false)` returns a reserved `root_uri` plus persistent task progress while ingestion completes.
### Upgrade Notes
diff --git a/docs/en/api/02-resources.md b/docs/en/api/02-resources.md
index fa5bd40ca..0c7457f40 100644
--- a/docs/en/api/02-resources.md
+++ b/docs/en/api/02-resources.md
@@ -82,11 +82,11 @@ Source Input -> Parse -> Resource Tree Build -> Persistence -> Semantic Processi
- **Vector Index**: Vectorizes content for semantic search
- Processed asynchronously via `SemanticQueue`, can wait for completion with `wait=True`
-#### Async `add_resource` Tasks
-- With `wait=false`, OpenViking validates the source, resolves the target URI, and reserves the final `root_uri` before returning.
-- The immediate response contains `status`, `root_uri`, and `task_id`; fetching/parsing/finalizing and queue waiting continue in a persistent background task.
-- Poll `GET /api/v1/tasks/{task_id}` to inspect task state. Resource import tasks use stages such as `queued`, `fetching`, `parsing`, `finalizing`, and `processing_queue`.
-- Task status responses expose the current `stage`; live semantic and embedding queue counts are available through observer queue APIs, and completed imports include `queue_status` in the task `result`.
+#### Non-Wait Git Repository Imports
+- For Git repository sources with `wait=false`, OpenViking validates the repository, resolves the target URI, reserves the final `root_uri`, and returns before clone/parse/finalize completes.
+- The immediate response contains `status`, `root_uri`, and `task_id`; fetching, parsing, finalizing, and queue waiting continue in a persistent background task.
+- Poll `GET /api/v1/tasks/{task_id}` to inspect task state. Git resource import tasks use stages such as `queued`, `fetching`, `parsing`, `finalizing`, and `processing_queue`.
+- Other resource sources with `wait=false` finish fetching/parsing/finalizing before the response; their returned `task_id` tracks semantic and embedding queue completion only.
### Incremental Updates for Resources
@@ -122,10 +122,10 @@ This endpoint is the core entry point for resource management, supporting adding
**Processing Flow**:
1. Identify and validate the resource source (URL or uploaded temporary file)
-2. Resolve the target URI; when `wait=false`, reserve the final `root_uri` before returning
+2. Resolve the target URI
3. Call the corresponding Parser to parse content
4. Build the directory tree and write to AGFS
-5. Wait for semantic processing completion when `wait=true`, or track background task state with the returned `task_id`
+5. Wait for semantic processing completion when `wait=true`; with `wait=false`, return a `task_id` for queue tracking
6. Set up scheduled update task if `watch_interval` is specified
**Code Entry Points**:
@@ -164,8 +164,8 @@ This endpoint is the core entry point for resource management, supporting adding
- `path` and `temp_file_id` cannot be specified together
- Raw HTTP calls for local files require first uploading via [temp_upload](#temp_upload) to obtain `temp_file_id`
- When `to` is specified and the target already exists, triggers incremental update
-- When `wait=false`, the request still performs source preflight and target planning before returning. Unreachable remote URLs/repositories or invalid parent targets fail immediately instead of creating a task.
-- When OpenViking auto-selects a target under `viking://resources` or `parent`, URI conflicts are resolved and reserved before the `wait=false` response is returned, so the returned `root_uri` is stable.
+- Only Git repository sources use full background import when `wait=false`; OpenViking performs repository preflight and target planning before returning the `task_id`.
+- Other sources with `wait=false` finish source parsing, target resolution, and AGFS writes before returning. Only semantic and embedding queues continue asynchronously.
- When `watch_interval > 0`, the watch task binds to `to` if provided; otherwise it binds to the `root_uri` returned by this import. If no stable `root_uri` is available, the request fails and asks for an explicit `to`.
- For local directory inputs, scanning respects `.gitignore` files (root and nested) with standard Git semantics; `ignore_dirs`, `include`, and `exclude` further refine what is ingested.
- To create or update plain text directly, use [content/write](03-filesystem.md#write) instead of `add_resource`. Semantic processing and embeddings are refreshed automatically after resource ingestion and content writes.
@@ -305,7 +305,7 @@ ov add-resource ./documents/guide.md -p viking://resources/docs/{calendar:today}
}
```
-**HTTP API Response (JSON, `wait=false`)**
+**HTTP API Response (JSON, non-Git `wait=false`)**
```json
{
@@ -313,12 +313,16 @@ ov add-resource ./documents/guide.md -p viking://resources/docs/{calendar:today}
"result": {
"status": "success",
"root_uri": "viking://resources/guide",
+ "temp_uri": "viking://temp/username/04291108_b62dc7/guide",
+ "source_path": "./documents/guide.md",
+ "meta": {},
+ "errors": [],
"task_id": "uuid-xxx"
}
}
```
-Use the returned `task_id` to poll `/api/v1/tasks/{task_id}`. The completed task result contains the full import result, including `queue_status`.
+Use the returned `task_id` to poll `/api/v1/tasks/{task_id}` for queue completion. For Git repository sources with `wait=false`, the same endpoint tracks the full background import and the completed task result contains the full import result, including `queue_status`.
**CLI Response (Default Table Format)**
@@ -346,15 +350,15 @@ task_id uuid-xxx
|-------|------|-------------|
| `status` | string | Processing status: "success" or "error" |
| `root_uri` | string | Final URI of the resource in OpenViking |
-| `task_id` | string | (Optional, only when `wait=false`) Background task ID for polling `/api/v1/tasks/{task_id}` |
-| `temp_uri` | string | Temporary URI during processing (only valid during background processing) |
+| `task_id` | string | (Optional, only when `wait=false`) Task ID for polling `/api/v1/tasks/{task_id}`. Non-Git imports use it for queue tracking; Git repository imports use it for full background import tracking. |
+| `temp_uri` | string | Temporary URI produced during import |
| `source_path` | string | Original source file path or URL |
| `meta` | object | Metadata from resource parsing (file type, size, etc.) |
| `errors` | array | List of errors encountered during processing |
| `warnings` | array | (Optional) List of warnings (only when `strict=False`) |
| `queue_status` | object | (Optional, only when `wait=true`) Queue processing status with `pending`, `processing`, `completed` counts |
-For `wait=false`, the background task has `task_type="add_resource"` and `resource_id` equal to the returned `root_uri`. Running task records may include `stage`; completed task results include `queue_status` with the final semantic and embedding queue summary.
+For Git repository sources with `wait=false`, the background task has `task_type="add_resource"` and `resource_id` equal to the returned `root_uri`. Running task records may include `stage`; completed task results include `queue_status` with the final semantic and embedding queue summary.
---
diff --git a/docs/en/api/05-sessions.md b/docs/en/api/05-sessions.md
index 6a1ad90cb..f53d84d1b 100644
--- a/docs/en/api/05-sessions.md
+++ b/docs/en/api/05-sessions.md
@@ -1073,7 +1073,7 @@ print(f"Status: {task['status']}")
}
```
-`stage` is nullable. Resource import tasks may report `queued`, `fetching`, `parsing`, `finalizing`, or `processing_queue`; other task types may leave it as `null`. Live queue counters are intentionally not part of task status; use observer queue APIs for live counts, or read `result.queue_status` after completion.
+`stage` is nullable. Git repository resource import tasks may report `queued`, `fetching`, `parsing`, `finalizing`, or `processing_queue`; other task types may leave it as `null`. Live queue counters are intentionally not part of task status; use observer queue APIs for live counts, or read `result.queue_status` after completion.
**Response Example (completed)**
diff --git a/docs/zh/about/02-changelog.md b/docs/zh/about/02-changelog.md
index ddb16d452..4dc602e3b 100644
--- a/docs/zh/about/02-changelog.md
+++ b/docs/zh/about/02-changelog.md
@@ -128,7 +128,7 @@ OpenViking 的所有重要变更都将记录在此文件中。
- **原生 CLI 分发**:新增 `@openviking/cli` npm 包,可通过 `npm i -g @openviking/cli` 使用 `ov`;Rust CLI 发布流水线扩展 Linux musl 构建、npm trusted publishing 和 CLI 集成测试。
- **检索与文件系统能力**:`find` / `search` 新增 `level` 过滤,可限定 L0 abstract、L1 overview 或 L2 文件命中;资源文件增加 Phase 1 WebDAV 适配;`observer.filesystem` 暴露文件系统观测入口。
- **Console 与 Usage/Audit**:新增 Usage/Audit 模块和 `/api/v1/console/*` BFF,基于现有 observability event bus 统计 token、检索次数、上下文提交热力图、请求审计和上下文库存。
-- **存储与并发可靠性**:增强精确路径锁和生命周期锁修复内容写入并发覆盖;阻塞后端调用移出 event loop;QueueFS SQLite 持久化扩展;task 记录现在会持久化以支持多实例查询;`add_resource(wait=false)` 会返回已预占的 `root_uri`,并在导入完成前提供持久化 task 进度。
+- **存储与并发可靠性**:增强精确路径锁和生命周期锁修复内容写入并发覆盖;阻塞后端调用移出 event loop;QueueFS SQLite 持久化扩展;task 记录现在会持久化以支持多实例查询;Git 仓库 `add_resource(wait=false)` 会返回已预占的 `root_uri`,并在导入完成前提供持久化 task 进度。
### 升级说明
diff --git a/docs/zh/api/02-resources.md b/docs/zh/api/02-resources.md
index 77c8a6224..87ded1b51 100644
--- a/docs/zh/api/02-resources.md
+++ b/docs/zh/api/02-resources.md
@@ -77,11 +77,11 @@ URL/文件 Parser TreeBuilder AGFS Summarizer/Vector
- **向量索引**:将内容向量化用于语义搜索
- 通过 `SemanticQueue` 异步处理,可通过 `wait=True` 等待完成
-#### 异步 `add_resource` 任务
-- `wait=false` 时,OpenViking 仍会先校验资源来源、解析目标 URI,并预占最终 `root_uri`,然后再返回。
-- 立即响应包含 `status`、`root_uri` 和 `task_id`;抓取/解析/finalize 以及队列等待会在持久化后台任务中继续执行。
-- 可通过 `GET /api/v1/tasks/{task_id}` 查询任务状态。资源导入任务的阶段包括 `queued`、`fetching`、`parsing`、`finalizing`、`processing_queue`。
-- 任务状态响应只暴露当前 `stage`;实时 semantic 和 embedding 队列计数可通过 observer queue 查询,导入完成后的任务 `result` 会包含 `queue_status`。
+#### 非等待 Git 仓库导入
+- 对 Git 仓库来源使用 `wait=false` 时,OpenViking 会先校验仓库、解析目标 URI、预占最终 `root_uri`,然后在 clone/parse/finalize 完成前返回。
+- 立即响应包含 `status`、`root_uri` 和 `task_id`;抓取、解析、finalize 以及队列等待会在持久化后台任务中继续执行。
+- 可通过 `GET /api/v1/tasks/{task_id}` 查询任务状态。Git 资源导入任务的阶段包括 `queued`、`fetching`、`parsing`、`finalizing`、`processing_queue`。
+- 其他资源来源使用 `wait=false` 时,会在响应前完成抓取/解析/finalize;返回的 `task_id` 只用于跟踪 semantic 和 embedding 队列完成情况。
### 资源的增量更新
@@ -117,10 +117,10 @@ URL/文件 Parser TreeBuilder AGFS Summarizer/Vector
**处理流程**:
1. 识别并校验资源来源(URL 或上传的临时文件)
-2. 解析目标 URI;`wait=false` 时会先预占最终 `root_uri` 再返回
+2. 解析目标 URI
3. 调用对应 Parser 解析内容
4. 构建目录树并写入 AGFS
-5. `wait=true` 时等待语义处理完成;否则通过返回的 `task_id` 查询后台任务状态
+5. `wait=true` 时等待语义处理完成;`wait=false` 时返回 `task_id` 用于队列跟踪
6. 如指定 `--watch-interval`,设置定时更新任务
**代码入口**:
@@ -157,8 +157,8 @@ URL/文件 Parser TreeBuilder AGFS Summarizer/Vector
**补充说明**:
- `to` 和 `parent` 不能同时使用;如果使用 `parent` 且希望父目录不存在时自动创建,请传 `create_parent=true`。指定 `to` 且目标已存在时,触发增量更新。
- `path` 和 `temp_file_id` 不能同时指定,上传本地文件需要先通过 [temp_upload](#temp_upload) 上传获取 `temp_file_id`,在 SDK 和 CLI 中已经封装好。
-- `wait=false` 时,请求仍会先完成资源 preflight 和目标规划。远程 URL/仓库不可达或父级目标无效会立即失败,不会创建后台任务。
-- 当 OpenViking 在 `viking://resources` 或 `parent` 下自动选择目标时,会在 `wait=false` 响应前完成冲突重命名和 URI 预占,因此返回的 `root_uri` 是稳定的。
+- 只有 Git 仓库来源在 `wait=false` 时使用完整后台导入;OpenViking 会先完成仓库 preflight 和目标规划,再返回 `task_id`。
+- 其他来源在 `wait=false` 时会在响应前完成来源解析、目标解析和 AGFS 写入,仅 semantic 与 embedding 队列继续异步处理。
- `watch_interval > 0` 时,如果指定了 `to`,监控任务绑定该目标;如果未指定 `to`,监控任务绑定本次导入返回的 `root_uri`。如果无法得到稳定 `root_uri`,请求会报错并要求显式传 `to`。
- 本地目录输入会遵循 `.gitignore`(根目录和子目录,标准 Git 语义);`ignore_dirs`、`include`、`exclude` 会在此基础上进一步过滤。
- 如果要直接创建或更新纯文本内容,请使用 [content/write](03-filesystem.md#write),不要使用 `add_resource`。资源导入和内容写入后都会自动刷新语义与 embedding。
@@ -298,7 +298,7 @@ ov add-resource ./documents/guide.md -p viking://resources/docs/{calendar:today}
}
```
-**HTTP API 响应 (JSON, `wait=false`)**
+**HTTP API 响应 (JSON, 非 Git `wait=false`)**
```json
{
@@ -306,12 +306,16 @@ ov add-resource ./documents/guide.md -p viking://resources/docs/{calendar:today}
"result": {
"status": "success",
"root_uri": "viking://resources/guide",
+ "temp_uri": "viking://temp/username/04291108_b62dc7/guide",
+ "source_path": "./documents/guide.md",
+ "meta": {},
+ "errors": [],
"task_id": "uuid-xxx"
}
}
```
-使用返回的 `task_id` 轮询 `/api/v1/tasks/{task_id}`。任务完成后的 `result` 会包含完整导入结果,包括 `queue_status`。
+使用返回的 `task_id` 轮询 `/api/v1/tasks/{task_id}` 可查看队列完成情况。对于 `wait=false` 的 Git 仓库来源,同一个端点会跟踪完整后台导入,任务完成后的 `result` 会包含完整导入结果,包括 `queue_status`。
**CLI 响应 (默认表格格式)**
@@ -339,15 +343,15 @@ task_id uuid-xxx
|------|------|------|
| `status` | string | 处理状态:"success" 成功,"error" 失败 |
| `root_uri` | string | 资源在 OpenViking 中的最终 URI |
-| `task_id` | string | (可选,仅当 `wait=false` 时)后台任务 ID,可轮询 `/api/v1/tasks/{task_id}` |
-| `temp_uri` | string | 处理过程中的临时 URI(仅在后台处理阶段有效) |
+| `task_id` | string | (可选,仅当 `wait=false` 时)可轮询 `/api/v1/tasks/{task_id}` 的任务 ID。非 Git 导入用于队列跟踪;Git 仓库导入用于完整后台导入跟踪。 |
+| `temp_uri` | string | 导入过程中生成的临时 URI |
| `source_path` | string | 原始源文件路径或 URL |
| `meta` | object | 资源解析过程中的元数据(如文件类型、大小等) |
| `errors` | array | 处理过程中的错误列表 |
| `warnings` | array | (可选)处理过程中的警告列表(仅在 `strict=False` 时可能出现) |
| `queue_status` | object | (可选,仅当 `wait=true` 时)队列处理状态,包含 `pending`、`processing`、`completed` 计数 |
-对于 `wait=false`,后台任务的 `task_type="add_resource"`,`resource_id` 等于返回的 `root_uri`。运行中的任务记录可能包含 `stage`;完成后的任务 `result` 会包含带有 semantic 和 embedding 汇总的 `queue_status`。
+对于 `wait=false` 的 Git 仓库来源,后台任务的 `task_type="add_resource"`,`resource_id` 等于返回的 `root_uri`。运行中的任务记录可能包含 `stage`;完成后的任务 `result` 会包含带有 semantic 和 embedding 汇总的 `queue_status`。
---
diff --git a/docs/zh/api/05-sessions.md b/docs/zh/api/05-sessions.md
index 17043cc65..1a3d779e2 100644
--- a/docs/zh/api/05-sessions.md
+++ b/docs/zh/api/05-sessions.md
@@ -1073,7 +1073,7 @@ print(f"Status: {task['status']}")
}
```
-`stage` 可以为 `null`。资源导入任务可能报告 `queued`、`fetching`、`parsing`、`finalizing`、`processing_queue`;其他任务类型可能将其留空。实时队列计数不会出现在任务状态中;需要实时数量时使用 observer queue,任务完成后可读取 `result.queue_status`。
+`stage` 可以为 `null`。Git 仓库资源导入任务可能报告 `queued`、`fetching`、`parsing`、`finalizing`、`processing_queue`;其他任务类型可能将其留空。实时队列计数不会出现在任务状态中;需要实时数量时使用 observer queue,任务完成后可读取 `result.queue_status`。
**响应示例(完成)**
diff --git a/openviking/server/routers/resources.py b/openviking/server/routers/resources.py
index b1612859b..77474a565 100644
--- a/openviking/server/routers/resources.py
+++ b/openviking/server/routers/resources.py
@@ -199,11 +199,6 @@ async def add_resource(
original_filename = None
resolved = None
store = None
-
- # Resolve path variables before passing to service.
- to = resolve_path_variables(request.to) if request.to else None
- parent = resolve_path_variables(request.parent) if request.parent else None
-
if request.temp_file_id:
store = TempUploadStore.build(http_request.app.state.config)
resolved = await store.resolve_for_consume(request.temp_file_id, _ctx)
@@ -233,16 +228,9 @@ async def add_resource(
if request.preserve_structure is not None:
kwargs["preserve_structure"] = request.preserve_structure
- async def _cleanup_resolved(success: bool) -> None:
- if not resolved or not store:
- return
- try:
- if success:
- await store.mark_consumed(resolved, _ctx)
- else:
- await store.mark_failed(resolved, _ctx)
- finally:
- await resolved.cleanup()
+ # Resolve path variables before passing to service.
+ to = resolve_path_variables(request.to) if request.to else None
+ parent = resolve_path_variables(request.parent) if request.parent else None
async def _add() -> dict[str, Any]:
try:
@@ -257,17 +245,19 @@ async def add_resource(
timeout=request.timeout,
allow_local_path_resolution=allow_local_path_resolution,
enforce_public_remote_targets=True,
- source_cleanup=_cleanup_resolved if resolved and not request.wait else None,
**kwargs,
)
except Exception:
- if request.wait:
- await _cleanup_resolved(False)
+ if resolved and store:
+ await store.mark_failed(resolved, _ctx)
raise
else:
- if request.wait:
- await _cleanup_resolved(True)
+ if resolved and store:
+ await store.mark_consumed(resolved, _ctx)
return result
+ finally:
+ if resolved:
+ await resolved.cleanup()
execution = await run_operation(
operation="resources.add_resource",
diff --git a/openviking/service/resource_service.py b/openviking/service/resource_service.py
index a0a72f98c..7e26dd30e 100644
--- a/openviking/service/resource_service.py
+++ b/openviking/service/resource_service.py
@@ -10,9 +10,8 @@ import asyncio
import contextlib
import inspect
import json
-import os
import time
-from collections.abc import Awaitable, Callable
+from collections.abc import Callable
from dataclasses import dataclass
from pathlib import Path
from typing import TYPE_CHECKING, Any, Dict, List, Optional
@@ -47,7 +46,6 @@ from openviking_cli.exceptions import (
DeadlineExceededError,
InvalidArgumentError,
NotInitializedError,
- OpenVikingError,
)
from openviking_cli.utils import get_logger
@@ -132,7 +130,7 @@ class ResourceService:
await asyncio.gather(*tasks, return_exceptions=True)
self._background_tasks.clear()
- async def enqueue_add_resource(
+ async def enqueue_git_add_resource(
self,
path: str,
ctx: RequestContext,
@@ -147,10 +145,9 @@ class ResourceService:
skip_watch_management: bool = False,
allow_local_path_resolution: bool = True,
enforce_public_remote_targets: bool = False,
- source_cleanup: Optional[Callable[[bool], Awaitable[None]]] = None,
**kwargs,
) -> Dict[str, Any]:
- """Start background resource ingestion while preserving the normal result shape."""
+ """Start background ingestion for Git repositories while reserving the target URI."""
self._ensure_initialized()
if to:
@@ -176,11 +173,7 @@ class ResourceService:
path = require_remote_resource_source(path)
kwargs.setdefault("request_validator", ensure_public_remote_target)
- source_info = await self._preflight_resource_source(
- path,
- allow_local_path_resolution=allow_local_path_resolution,
- request_validator=kwargs.get("request_validator"),
- )
+ source_info = await self._preflight_git_source(path)
source_name = kwargs.get("source_name") or source_info.source_name
if source_name:
kwargs["source_name"] = source_name
@@ -230,7 +223,6 @@ class ResourceService:
ctx=ctx,
add_kwargs=add_kwargs,
resource_lock=resource_lock,
- source_cleanup=source_cleanup,
)
)
resource_lock = NO_LOCK
@@ -243,9 +235,6 @@ class ResourceService:
}
except Exception:
await resource_lock.close()
- if source_cleanup is not None:
- with contextlib.suppress(Exception):
- await source_cleanup(False)
raise
async def _run_add_resource_task(
@@ -255,7 +244,6 @@ class ResourceService:
ctx: RequestContext,
add_kwargs: Dict[str, Any],
resource_lock: LockLease,
- source_cleanup: Optional[Callable[[bool], Awaitable[None]]] = None,
) -> None:
from openviking.service.task_tracker import get_task_tracker
@@ -269,7 +257,6 @@ class ResourceService:
user_id=ctx.user.user_id,
)
- success = False
try:
await task_tracker.start(
task_id,
@@ -301,7 +288,6 @@ class ResourceService:
user_id=ctx.user.user_id,
)
return
- success = True
await task_tracker.complete(
task_id,
result,
@@ -325,15 +311,6 @@ class ResourceService:
)
finally:
await resource_lock.close()
- if source_cleanup is not None:
- try:
- await source_cleanup(success)
- except Exception as exc:
- logger.warning(
- "[ResourceService] Failed to cleanup async resource source for task %s: %s",
- task_id,
- exc,
- )
async def _plan_resource_target(
self,
@@ -394,73 +371,6 @@ class ResourceService:
return parsed.rsplit("/", 1)[-1]
return _smart_stem(Path(path).name or "resource")
- async def _preflight_resource_source(
- self,
- path: str,
- *,
- allow_local_path_resolution: bool,
- request_validator: Optional[Callable[[str], None]] = None,
- ) -> _ResourceSourceInfo:
- if not is_remote_resource_source(path):
- if allow_local_path_resolution and not Path(path).expanduser().exists():
- raise InvalidArgumentError(f"Resource source does not exist: {path}")
- return _ResourceSourceInfo(source_name=Path(path).name)
- if is_git_repo_url(path):
- return await self._preflight_git_source(path)
- if path.startswith(("http://", "https://")):
- return await self._preflight_http_source(path, request_validator=request_validator)
- return await self._preflight_git_source(path)
-
- async def _preflight_http_source(
- self,
- url: str,
- *,
- request_validator: Optional[Callable[[str], None]] = None,
- ) -> _ResourceSourceInfo:
- from openviking.parse.accessors.http_accessor import HTTPAccessor
- from openviking.parse.base import lazy_import
- from openviking.utils.network_guard import build_httpx_request_validation_hooks
-
- httpx = lazy_import("httpx")
- event_hooks = build_httpx_request_validation_hooks(request_validator)
- client_kwargs: Dict[str, Any] = {
- "timeout": 5.0,
- "follow_redirects": True,
- }
- if event_hooks:
- client_kwargs["event_hooks"] = event_hooks
- client_kwargs["trust_env"] = False
- headers = {"User-Agent": "OpenViking"}
- if "github.com" in url and os.environ.get("GITHUB_TOKEN"):
- headers["Authorization"] = f"token {os.environ['GITHUB_TOKEN']}"
-
- try:
- async with httpx.AsyncClient(**client_kwargs) as client:
- response = await client.head(url, headers=headers)
- if response.status_code < 400:
- response.raise_for_status()
- else:
- await self._probe_http_get(client, url, headers)
- except OpenVikingError:
- raise
- except Exception as exc:
- raise InvalidArgumentError(f"Cannot access remote URL: {url}. {exc}") from exc
-
- source_name = HTTPAccessor._extract_filename_from_url(url)
- return _ResourceSourceInfo(
- source_name=source_name,
- source_path=source_name or url,
- )
-
- @staticmethod
- async def _probe_http_get(client: Any, url: str, headers: Dict[str, str]) -> None:
- probe_headers = dict(headers)
- probe_headers["Range"] = "bytes=0-0"
- async with client.stream("GET", url, headers=probe_headers) as response:
- response.raise_for_status()
- async for _ in response.aiter_bytes():
- break
-
async def _preflight_git_source(self, source: str) -> _ResourceSourceInfo:
try:
proc = await asyncio.create_subprocess_exec(
@@ -510,7 +420,6 @@ class ResourceService:
skip_watch_management: bool = False,
allow_local_path_resolution: bool = True,
enforce_public_remote_targets: bool = False,
- source_cleanup: Optional[Callable[[bool], Awaitable[None]]] = None,
resource_lock: Optional[LockLease] = None,
stage_callback: Optional[Callable[[str], Any]] = None,
**kwargs,
@@ -552,8 +461,8 @@ class ResourceService:
InvalidArgumentError: If the URI scope is not 'resources'
"""
self._ensure_initialized()
- if not wait:
- return await self.enqueue_add_resource(
+ if not wait and is_git_repo_url(path):
+ return await self.enqueue_git_add_resource(
path=path,
ctx=ctx,
to=to,
@@ -567,7 +476,6 @@ class ResourceService:
skip_watch_management=skip_watch_management,
allow_local_path_resolution=allow_local_path_resolution,
enforce_public_remote_targets=enforce_public_remote_targets,
- source_cleanup=source_cleanup,
**kwargs,
)
@@ -575,6 +483,7 @@ class ResourceService:
telemetry = get_current_telemetry()
telemetry_id = register_wait_telemetry(wait)
request_wait_tracker = get_request_wait_tracker()
+ monitor_started = False
if telemetry_id:
request_wait_tracker.register_request(telemetry_id)
watch_manager = self._get_watch_manager()
@@ -625,50 +534,53 @@ class ResourceService:
if result.get("status") == "error":
return result
- if stage_callback is not None:
- stage_result = stage_callback("processing_queue")
- if inspect.isawaitable(stage_result):
- await stage_result
- wait_start = time.perf_counter()
- try:
- with telemetry.measure("resource.wait"):
- if telemetry_id:
- await request_wait_tracker.wait_for_request(
- telemetry_id,
- timeout=timeout,
- poll_interval=0.05,
- )
- status = request_wait_tracker.build_queue_status(telemetry_id)
- else:
- qm = get_queue_manager()
- status = build_queue_status_payload(await qm.wait_complete(timeout=timeout))
- except TimeoutError as exc:
- telemetry.set_error(
- "resource_service.wait_complete",
- "DEADLINE_EXCEEDED",
- str(exc),
- )
- raise DeadlineExceededError("queue processing", timeout) from exc
- queue_wait_duration_ms = round((time.perf_counter() - wait_start) * 1000, 3)
- try:
- from openviking.metrics.datasources.resource import (
- ResourceIngestionEventDataSource,
- )
+ elif wait:
+ if stage_callback is not None:
+ stage_result = stage_callback("processing_queue")
+ if inspect.isawaitable(stage_result):
+ await stage_result
+ wait_start = time.perf_counter()
+ try:
+ with telemetry.measure("resource.wait"):
+ if telemetry_id:
+ await request_wait_tracker.wait_for_request(
+ telemetry_id,
+ timeout=timeout,
+ poll_interval=0.05,
+ )
+ status = request_wait_tracker.build_queue_status(telemetry_id)
+ else:
+ qm = get_queue_manager()
+ status = build_queue_status_payload(
+ await qm.wait_complete(timeout=timeout)
+ )
+ except TimeoutError as exc:
+ telemetry.set_error(
+ "resource_service.wait_complete",
+ "DEADLINE_EXCEEDED",
+ str(exc),
+ )
+ raise DeadlineExceededError("queue processing", timeout) from exc
+ queue_wait_duration_ms = round((time.perf_counter() - wait_start) * 1000, 3)
+ try:
+ from openviking.metrics.datasources.resource import (
+ ResourceIngestionEventDataSource,
+ )
- ResourceIngestionEventDataSource.record_wait(
- operation="queue_processing",
- duration_seconds=float(queue_wait_duration_ms) / 1000.0,
- account_id=getattr(ctx, "account_id", None),
+ ResourceIngestionEventDataSource.record_wait(
+ operation="queue_processing",
+ duration_seconds=float(queue_wait_duration_ms) / 1000.0,
+ account_id=getattr(ctx, "account_id", None),
+ )
+ except Exception:
+ pass
+ result["queue_status"] = status
+ record_resource_wait_metrics(
+ telemetry_id=telemetry_id,
+ queue_status=status,
+ root_uri=result.get("root_uri"),
)
- except Exception:
- pass
- result["queue_status"] = status
- record_resource_wait_metrics(
- telemetry_id=telemetry_id,
- queue_status=status,
- root_uri=result.get("root_uri"),
- )
- telemetry.set("queue.wait.duration_ms", queue_wait_duration_ms)
+ telemetry.set("queue.wait.duration_ms", queue_wait_duration_ms)
if watch_manager and not skip_watch_management:
with telemetry.measure("resource.watch"):
if watch_interval > 0:
@@ -713,6 +625,40 @@ class ResourceService:
logger.warning(
f"[ResourceService] Failed to cancel watch task for {to}: {e}"
)
+ if not wait:
+ from openviking.service.task_tracker import get_task_tracker
+
+ task_tracker = get_task_tracker()
+ root_uri = result.get("root_uri", "")
+ task = await task_tracker.create(
+ "add_resource",
+ resource_id=root_uri,
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ )
+ result["task_id"] = task.task_id
+ if telemetry_id:
+ monitor_started = True
+ background = asyncio.create_task(
+ self._monitor_queue_processing(
+ task.task_id,
+ telemetry_id,
+ ctx.account_id,
+ ctx.user.user_id,
+ )
+ )
+ self._background_tasks.add(background)
+ background.add_done_callback(self._background_tasks.discard)
+ else:
+ await task_tracker.start(
+ task.task_id, account_id=ctx.account_id, user_id=ctx.user.user_id
+ )
+ await task_tracker.complete(
+ task.task_id,
+ {"root_uri": root_uri},
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ )
return result
except Exception as exc:
telemetry.set_error(
@@ -726,8 +672,9 @@ class ResourceService:
"resource.request.duration_ms",
round((time.perf_counter() - request_start) * 1000, 3),
)
- get_request_wait_tracker().cleanup(telemetry_id)
- unregister_wait_telemetry(telemetry_id)
+ if wait or not telemetry_id or not monitor_started:
+ get_request_wait_tracker().cleanup(telemetry_id)
+ unregister_wait_telemetry(telemetry_id)
async def _monitor_queue_processing(
self,
diff --git a/tests/server/test_api_resources.py b/tests/server/test_api_resources.py
index 7074d22c5..c3c2ce516 100644
--- a/tests/server/test_api_resources.py
+++ b/tests/server/test_api_resources.py
@@ -46,10 +46,8 @@ async def test_add_resource_success(
assert "telemetry" not in body
assert body["result"]["status"] == "success"
assert body["result"]["root_uri"].startswith("viking://")
+ assert "source_path" in body["result"]
assert body["result"]["task_id"]
- task = await _wait_task_terminal(client, body["result"]["task_id"])
- assert task["status"] == "completed"
- assert task["result"]["root_uri"] == body["result"]["root_uri"]
async def test_add_resource_with_wait(
@@ -399,7 +397,7 @@ async def test_add_resource_with_resources_root_to_trailing_slash_keeps_single_f
assert body["result"]["root_uri"] == "viking://resources/aa"
-async def test_add_resource_async_auto_name_reserves_unique_root_uri(
+async def test_add_resource_non_wait_auto_name_resolves_unique_root_uri(
client: httpx.AsyncClient,
upload_temp_dir,
):
@@ -434,11 +432,6 @@ async def test_add_resource_async_auto_name_reserves_unique_root_uri(
assert first_result["root_uri"] == "viking://resources/same"
assert second_result["root_uri"] == "viking://resources/same_1"
- first_task = await _wait_task_terminal(client, first_result["task_id"])
- second_task = await _wait_task_terminal(client, second_result["task_id"])
- assert first_task["result"]["root_uri"] == first_result["root_uri"]
- assert second_task["result"]["root_uri"] == second_result["root_uri"]
-
async def test_wait_processed_empty_queue(client: httpx.AsyncClient):
resp = await client.post(
@@ -697,7 +690,7 @@ async def test_add_resource_rejects_temp_file_id_symlink(
assert body["error"]["code"] == "PERMISSION_DENIED"
-async def test_add_resource_async_returns_task_id(
+async def test_add_resource_non_wait_returns_queue_task_id(
client: httpx.AsyncClient,
sample_markdown_file,
upload_temp_dir,
@@ -717,9 +710,6 @@ async def test_add_resource_async_returns_task_id(
assert "task_id" in body["result"]
assert body["result"]["task_id"]
assert body["result"]["root_uri"].startswith("viking://")
- task = await _wait_task_terminal(client, body["result"]["task_id"])
- assert task["status"] == "completed"
- assert task["result"]["root_uri"] == body["result"]["root_uri"]
async def test_add_resource_sync_no_task_id(
@@ -741,7 +731,7 @@ async def test_add_resource_sync_no_task_id(
assert "task_id" not in body["result"]
-async def test_add_resource_async_task_queryable(
+async def test_add_resource_non_wait_queue_task_queryable(
client: httpx.AsyncClient,
sample_markdown_file,
upload_temp_dir,
@@ -759,11 +749,12 @@ async def test_add_resource_async_task_queryable(
},
)
task_id = resp.json()["result"]["task_id"]
- root_uri = resp.json()["result"]["root_uri"]
- result = await _wait_task_terminal(client, task_id)
+ await asyncio.sleep(2.0)
+
+ task_resp = await client.get(f"/api/v1/tasks/{task_id}")
+ assert task_resp.status_code == 200
+ result = task_resp.json()["result"]
assert result["task_id"] == task_id
assert result["task_type"] == "add_resource"
- assert result["status"] == "completed"
- assert result["stage"] in {"queued", "fetching", "parsing", "finalizing", "processing_queue"}
- assert result["result"]["root_uri"] == root_uri
+ assert result["status"] in {"running", "completed", "failed"}
diff --git a/tests/service/test_resource_service_tasks.py b/tests/service/test_resource_service_tasks.py
deleted file mode 100644
index 1153c2865..000000000
--- a/tests/service/test_resource_service_tasks.py
+++ /dev/null
@@ -1,105 +0,0 @@
-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
-# SPDX-License-Identifier: AGPL-3.0
-
-import pytest
-
-from openviking.server.identity import RequestContext, Role
-from openviking.service.resource_service import ResourceService
-from openviking.service.task_tracker import (
- TaskStatus,
- TaskTracker,
- reset_task_tracker,
- set_task_tracker,
-)
-from openviking.storage.transaction import NO_LOCK
-from openviking_cli.session.user_id import UserIdentifier
-
-pytestmark = pytest.mark.asyncio
-
-
-class _NoopTaskStore:
- async def create(self, task):
- return None
-
- async def update(self, task):
- return None
-
- async def get(self, task_id, *, account_id=None, user_id=None):
- return None
-
- async def list(self, account_id, *, user_id=None):
- return []
-
- async def delete(self, task_id, *, account_id, user_id=None):
- return None
-
-
[email protected](autouse=True)
-def clean_task_tracker():
- reset_task_tracker()
- yield
- reset_task_tracker()
-
-
-async def test_run_add_resource_task_fails_when_queue_status_has_errors(monkeypatch):
- tracker = TaskTracker(store=_NoopTaskStore())
- set_task_tracker(tracker)
- service = ResourceService()
- ctx = RequestContext(
- user=UserIdentifier("acme", "alice", "agent-1"),
- role=Role.ADMIN,
- )
- task = await tracker.create(
- "add_resource",
- resource_id="viking://resources/demo",
- account_id=ctx.account_id,
- user_id=ctx.user.user_id,
- )
- cleanup_calls = []
-
- async def fake_add_resource(**kwargs):
- assert kwargs["wait"] is True
- return {
- "status": "success",
- "root_uri": "viking://resources/demo",
- "queue_status": {
- "Semantic": {
- "processed": 0,
- "requeue_count": 0,
- "error_count": 1,
- "errors": [{"message": "semantic processing failed"}],
- },
- "Embedding": {
- "processed": 0,
- "requeue_count": 0,
- "error_count": 0,
- "errors": [],
- },
- },
- }
-
- async def cleanup(success: bool):
- cleanup_calls.append(success)
-
- monkeypatch.setattr(service, "add_resource", fake_add_resource)
-
- await service._run_add_resource_task(
- task.task_id,
- ctx=ctx,
- add_kwargs={"path": "/tmp/demo.md", "ctx": ctx},
- resource_lock=NO_LOCK,
- source_cleanup=cleanup,
- )
-
- stored = await tracker.get(
- task.task_id,
- account_id=ctx.account_id,
- user_id=ctx.user.user_id,
- )
- assert stored is not None
- assert stored.status == TaskStatus.FAILED
- assert stored.result is None
- assert "queue processing failed" in stored.error
- assert "Semantic error_count=1" in stored.error
- assert "semantic processing failed" in stored.error
- assert cleanup_calls == [False]