Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
fix(resources): avoid remote preflight for non-git imports (#2459)
Qin Haojie34682c8aaebb
diff --git a/docs/en/about/02-changelog.md b/docs/en/about/02-changelog.mdindex 5461635bd..eb94008db 100644--- a/docs/en/about/02-changelog.md+++ b/docs/en/about/02-changelog.md@@ -128,7 +128,7 @@ This changelog is automatically generated from [GitHub Releases](https://github.- **Native CLI distribution**: New `@openviking/cli` npm package installs the platform `ov` binary via `npm i -g @openviking/cli`; Rust CLI release pipeline adds Linux musl artifacts, npm trusted publishing, and a broader integration test suite.- **Retrieval and filesystem**: `find` / `search` accept a `level` filter for L0 abstracts, L1 overviews, and L2 file hits. Resource files gained a Phase 1 WebDAV adapter, and `observer.filesystem` exposes filesystem-level observability.- **Console and Usage/Audit**: New Usage/Audit module and `/api/v1/console/*` BFF aggregate token usage, retrieval counts, context commit heatmaps, request audit logs, and context inventory from the existing observability event bus.-- **Storage and concurrency reliability**: Strengthened exact-path and lifecycle locks fix content-write races; blocking backend calls moved off the event loop; QueueFS SQLite persistence expanded; task records are now persisted for cross-instance lookup; `add_resource(wait=false)` returns a reserved `root_uri` plus persistent task progress while ingestion completes.+- **Storage and concurrency reliability**: Strengthened exact-path and lifecycle locks fix content-write races; blocking backend calls moved off the event loop; QueueFS SQLite persistence expanded; task records are now persisted for cross-instance lookup; Git repository `add_resource(wait=false)` returns a reserved `root_uri` plus persistent task progress while ingestion completes.### Upgrade Notesdiff --git a/docs/en/api/02-resources.md b/docs/en/api/02-resources.mdindex fa5bd40ca..0c7457f40 100644--- a/docs/en/api/02-resources.md+++ b/docs/en/api/02-resources.md@@ -82,11 +82,11 @@ Source Input -> Parse -> Resource Tree Build -> Persistence -> Semantic Processi- **Vector Index**: Vectorizes content for semantic search- Processed asynchronously via `SemanticQueue`, can wait for completion with `wait=True`-#### Async `add_resource` Tasks-- With `wait=false`, OpenViking validates the source, resolves the target URI, and reserves the final `root_uri` before returning.-- The immediate response contains `status`, `root_uri`, and `task_id`; fetching/parsing/finalizing and queue waiting continue in a persistent background task.-- Poll `GET /api/v1/tasks/{task_id}` to inspect task state. Resource import tasks use stages such as `queued`, `fetching`, `parsing`, `finalizing`, and `processing_queue`.-- Task status responses expose the current `stage`; live semantic and embedding queue counts are available through observer queue APIs, and completed imports include `queue_status` in the task `result`.+#### Non-Wait Git Repository Imports+- For Git repository sources with `wait=false`, OpenViking validates the repository, resolves the target URI, reserves the final `root_uri`, and returns before clone/parse/finalize completes.+- The immediate response contains `status`, `root_uri`, and `task_id`; fetching, parsing, finalizing, and queue waiting continue in a persistent background task.+- Poll `GET /api/v1/tasks/{task_id}` to inspect task state. Git resource import tasks use stages such as `queued`, `fetching`, `parsing`, `finalizing`, and `processing_queue`.+- Other resource sources with `wait=false` finish fetching/parsing/finalizing before the response; their returned `task_id` tracks semantic and embedding queue completion only.### Incremental Updates for Resources@@ -122,10 +122,10 @@ This endpoint is the core entry point for resource management, supporting adding**Processing Flow**:1. Identify and validate the resource source (URL or uploaded temporary file)-2. Resolve the target URI; when `wait=false`, reserve the final `root_uri` before returning+2. Resolve the target URI3. Call the corresponding Parser to parse content4. Build the directory tree and write to AGFS-5. Wait for semantic processing completion when `wait=true`, or track background task state with the returned `task_id`+5. Wait for semantic processing completion when `wait=true`; with `wait=false`, return a `task_id` for queue tracking6. Set up scheduled update task if `watch_interval` is specified**Code Entry Points**:@@ -164,8 +164,8 @@ This endpoint is the core entry point for resource management, supporting adding- `path` and `temp_file_id` cannot be specified together- Raw HTTP calls for local files require first uploading via [temp_upload](#temp_upload) to obtain `temp_file_id`- When `to` is specified and the target already exists, triggers incremental update-- When `wait=false`, the request still performs source preflight and target planning before returning. Unreachable remote URLs/repositories or invalid parent targets fail immediately instead of creating a task.-- When OpenViking auto-selects a target under `viking://resources` or `parent`, URI conflicts are resolved and reserved before the `wait=false` response is returned, so the returned `root_uri` is stable.+- Only Git repository sources use full background import when `wait=false`; OpenViking performs repository preflight and target planning before returning the `task_id`.+- Other sources with `wait=false` finish source parsing, target resolution, and AGFS writes before returning. Only semantic and embedding queues continue asynchronously.- When `watch_interval > 0`, the watch task binds to `to` if provided; otherwise it binds to the `root_uri` returned by this import. If no stable `root_uri` is available, the request fails and asks for an explicit `to`.- For local directory inputs, scanning respects `.gitignore` files (root and nested) with standard Git semantics; `ignore_dirs`, `include`, and `exclude` further refine what is ingested.- To create or update plain text directly, use [content/write](03-filesystem.md#write) instead of `add_resource`. Semantic processing and embeddings are refreshed automatically after resource ingestion and content writes.@@ -305,7 +305,7 @@ ov add-resource ./documents/guide.md -p viking://resources/docs/{calendar:today}}```-**HTTP API Response (JSON, `wait=false`)**+**HTTP API Response (JSON, non-Git `wait=false`)**```json{@@ -313,12 +313,16 @@ ov add-resource ./documents/guide.md -p viking://resources/docs/{calendar:today}"result": {"status": "success","root_uri": "viking://resources/guide",+ "temp_uri": "viking://temp/username/04291108_b62dc7/guide",+ "source_path": "./documents/guide.md",+ "meta": {},+ "errors": [],"task_id": "uuid-xxx"}}```-Use the returned `task_id` to poll `/api/v1/tasks/{task_id}`. The completed task result contains the full import result, including `queue_status`.+Use the returned `task_id` to poll `/api/v1/tasks/{task_id}` for queue completion. For Git repository sources with `wait=false`, the same endpoint tracks the full background import and the completed task result contains the full import result, including `queue_status`.**CLI Response (Default Table Format)**@@ -346,15 +350,15 @@ task_id uuid-xxx|-------|------|-------------|| `status` | string | Processing status: "success" or "error" || `root_uri` | string | Final URI of the resource in OpenViking |-| `task_id` | string | (Optional, only when `wait=false`) Background task ID for polling `/api/v1/tasks/{task_id}` |-| `temp_uri` | string | Temporary URI during processing (only valid during background processing) |+| `task_id` | string | (Optional, only when `wait=false`) Task ID for polling `/api/v1/tasks/{task_id}`. Non-Git imports use it for queue tracking; Git repository imports use it for full background import tracking. |+| `temp_uri` | string | Temporary URI produced during import || `source_path` | string | Original source file path or URL || `meta` | object | Metadata from resource parsing (file type, size, etc.) || `errors` | array | List of errors encountered during processing || `warnings` | array | (Optional) List of warnings (only when `strict=False`) || `queue_status` | object | (Optional, only when `wait=true`) Queue processing status with `pending`, `processing`, `completed` counts |-For `wait=false`, the background task has `task_type="add_resource"` and `resource_id` equal to the returned `root_uri`. Running task records may include `stage`; completed task results include `queue_status` with the final semantic and embedding queue summary.+For Git repository sources with `wait=false`, the background task has `task_type="add_resource"` and `resource_id` equal to the returned `root_uri`. Running task records may include `stage`; completed task results include `queue_status` with the final semantic and embedding queue summary.---diff --git a/docs/en/api/05-sessions.md b/docs/en/api/05-sessions.mdindex 6a1ad90cb..f53d84d1b 100644--- a/docs/en/api/05-sessions.md+++ b/docs/en/api/05-sessions.md@@ -1073,7 +1073,7 @@ print(f"Status: {task['status']}")}```-`stage` is nullable. Resource import tasks may report `queued`, `fetching`, `parsing`, `finalizing`, or `processing_queue`; other task types may leave it as `null`. Live queue counters are intentionally not part of task status; use observer queue APIs for live counts, or read `result.queue_status` after completion.+`stage` is nullable. Git repository resource import tasks may report `queued`, `fetching`, `parsing`, `finalizing`, or `processing_queue`; other task types may leave it as `null`. Live queue counters are intentionally not part of task status; use observer queue APIs for live counts, or read `result.queue_status` after completion.**Response Example (completed)**diff --git a/docs/zh/about/02-changelog.md b/docs/zh/about/02-changelog.mdindex ddb16d452..4dc602e3b 100644--- a/docs/zh/about/02-changelog.md+++ b/docs/zh/about/02-changelog.md@@ -128,7 +128,7 @@ OpenViking 的所有重要变更都将记录在此文件中。- **原生 CLI 分发**:新增 `@openviking/cli` npm 包,可通过 `npm i -g @openviking/cli` 使用 `ov`;Rust CLI 发布流水线扩展 Linux musl 构建、npm trusted publishing 和 CLI 集成测试。- **检索与文件系统能力**:`find` / `search` 新增 `level` 过滤,可限定 L0 abstract、L1 overview 或 L2 文件命中;资源文件增加 Phase 1 WebDAV 适配;`observer.filesystem` 暴露文件系统观测入口。- **Console 与 Usage/Audit**:新增 Usage/Audit 模块和 `/api/v1/console/*` BFF,基于现有 observability event bus 统计 token、检索次数、上下文提交热力图、请求审计和上下文库存。-- **存储与并发可靠性**:增强精确路径锁和生命周期锁修复内容写入并发覆盖;阻塞后端调用移出 event loop;QueueFS SQLite 持久化扩展;task 记录现在会持久化以支持多实例查询;`add_resource(wait=false)` 会返回已预占的 `root_uri`,并在导入完成前提供持久化 task 进度。+- **存储与并发可靠性**:增强精确路径锁和生命周期锁修复内容写入并发覆盖;阻塞后端调用移出 event loop;QueueFS SQLite 持久化扩展;task 记录现在会持久化以支持多实例查询;Git 仓库 `add_resource(wait=false)` 会返回已预占的 `root_uri`,并在导入完成前提供持久化 task 进度。### 升级说明diff --git a/docs/zh/api/02-resources.md b/docs/zh/api/02-resources.mdindex 77c8a6224..87ded1b51 100644--- a/docs/zh/api/02-resources.md+++ b/docs/zh/api/02-resources.md@@ -77,11 +77,11 @@ URL/文件 Parser TreeBuilder AGFS Summarizer/Vector- **向量索引**:将内容向量化用于语义搜索- 通过 `SemanticQueue` 异步处理,可通过 `wait=True` 等待完成-#### 异步 `add_resource` 任务-- `wait=false` 时,OpenViking 仍会先校验资源来源、解析目标 URI,并预占最终 `root_uri`,然后再返回。-- 立即响应包含 `status`、`root_uri` 和 `task_id`;抓取/解析/finalize 以及队列等待会在持久化后台任务中继续执行。-- 可通过 `GET /api/v1/tasks/{task_id}` 查询任务状态。资源导入任务的阶段包括 `queued`、`fetching`、`parsing`、`finalizing`、`processing_queue`。-- 任务状态响应只暴露当前 `stage`;实时 semantic 和 embedding 队列计数可通过 observer queue 查询,导入完成后的任务 `result` 会包含 `queue_status`。+#### 非等待 Git 仓库导入+- 对 Git 仓库来源使用 `wait=false` 时,OpenViking 会先校验仓库、解析目标 URI、预占最终 `root_uri`,然后在 clone/parse/finalize 完成前返回。+- 立即响应包含 `status`、`root_uri` 和 `task_id`;抓取、解析、finalize 以及队列等待会在持久化后台任务中继续执行。+- 可通过 `GET /api/v1/tasks/{task_id}` 查询任务状态。Git 资源导入任务的阶段包括 `queued`、`fetching`、`parsing`、`finalizing`、`processing_queue`。+- 其他资源来源使用 `wait=false` 时,会在响应前完成抓取/解析/finalize;返回的 `task_id` 只用于跟踪 semantic 和 embedding 队列完成情况。### 资源的增量更新@@ -117,10 +117,10 @@ URL/文件 Parser TreeBuilder AGFS Summarizer/Vector**处理流程**:1. 识别并校验资源来源(URL 或上传的临时文件)-2. 解析目标 URI;`wait=false` 时会先预占最终 `root_uri` 再返回+2. 解析目标 URI3. 调用对应 Parser 解析内容4. 构建目录树并写入 AGFS-5. `wait=true` 时等待语义处理完成;否则通过返回的 `task_id` 查询后台任务状态+5. `wait=true` 时等待语义处理完成;`wait=false` 时返回 `task_id` 用于队列跟踪6. 如指定 `--watch-interval`,设置定时更新任务**代码入口**:@@ -157,8 +157,8 @@ URL/文件 Parser TreeBuilder AGFS Summarizer/Vector**补充说明**:- `to` 和 `parent` 不能同时使用;如果使用 `parent` 且希望父目录不存在时自动创建,请传 `create_parent=true`。指定 `to` 且目标已存在时,触发增量更新。- `path` 和 `temp_file_id` 不能同时指定,上传本地文件需要先通过 [temp_upload](#temp_upload) 上传获取 `temp_file_id`,在 SDK 和 CLI 中已经封装好。-- `wait=false` 时,请求仍会先完成资源 preflight 和目标规划。远程 URL/仓库不可达或父级目标无效会立即失败,不会创建后台任务。-- 当 OpenViking 在 `viking://resources` 或 `parent` 下自动选择目标时,会在 `wait=false` 响应前完成冲突重命名和 URI 预占,因此返回的 `root_uri` 是稳定的。+- 只有 Git 仓库来源在 `wait=false` 时使用完整后台导入;OpenViking 会先完成仓库 preflight 和目标规划,再返回 `task_id`。+- 其他来源在 `wait=false` 时会在响应前完成来源解析、目标解析和 AGFS 写入,仅 semantic 与 embedding 队列继续异步处理。- `watch_interval > 0` 时,如果指定了 `to`,监控任务绑定该目标;如果未指定 `to`,监控任务绑定本次导入返回的 `root_uri`。如果无法得到稳定 `root_uri`,请求会报错并要求显式传 `to`。- 本地目录输入会遵循 `.gitignore`(根目录和子目录,标准 Git 语义);`ignore_dirs`、`include`、`exclude` 会在此基础上进一步过滤。- 如果要直接创建或更新纯文本内容,请使用 [content/write](03-filesystem.md#write),不要使用 `add_resource`。资源导入和内容写入后都会自动刷新语义与 embedding。@@ -298,7 +298,7 @@ ov add-resource ./documents/guide.md -p viking://resources/docs/{calendar:today}}```-**HTTP API 响应 (JSON, `wait=false`)**+**HTTP API 响应 (JSON, 非 Git `wait=false`)**```json{@@ -306,12 +306,16 @@ ov add-resource ./documents/guide.md -p viking://resources/docs/{calendar:today}"result": {"status": "success","root_uri": "viking://resources/guide",+ "temp_uri": "viking://temp/username/04291108_b62dc7/guide",+ "source_path": "./documents/guide.md",+ "meta": {},+ "errors": [],"task_id": "uuid-xxx"}}```-使用返回的 `task_id` 轮询 `/api/v1/tasks/{task_id}`。任务完成后的 `result` 会包含完整导入结果,包括 `queue_status`。+使用返回的 `task_id` 轮询 `/api/v1/tasks/{task_id}` 可查看队列完成情况。对于 `wait=false` 的 Git 仓库来源,同一个端点会跟踪完整后台导入,任务完成后的 `result` 会包含完整导入结果,包括 `queue_status`。**CLI 响应 (默认表格格式)**@@ -339,15 +343,15 @@ task_id uuid-xxx|------|------|------|| `status` | string | 处理状态:"success" 成功,"error" 失败 || `root_uri` | string | 资源在 OpenViking 中的最终 URI |-| `task_id` | string | (可选,仅当 `wait=false` 时)后台任务 ID,可轮询 `/api/v1/tasks/{task_id}` |-| `temp_uri` | string | 处理过程中的临时 URI(仅在后台处理阶段有效) |+| `task_id` | string | (可选,仅当 `wait=false` 时)可轮询 `/api/v1/tasks/{task_id}` 的任务 ID。非 Git 导入用于队列跟踪;Git 仓库导入用于完整后台导入跟踪。 |+| `temp_uri` | string | 导入过程中生成的临时 URI || `source_path` | string | 原始源文件路径或 URL || `meta` | object | 资源解析过程中的元数据(如文件类型、大小等) || `errors` | array | 处理过程中的错误列表 || `warnings` | array | (可选)处理过程中的警告列表(仅在 `strict=False` 时可能出现) || `queue_status` | object | (可选,仅当 `wait=true` 时)队列处理状态,包含 `pending`、`processing`、`completed` 计数 |-对于 `wait=false`,后台任务的 `task_type="add_resource"`,`resource_id` 等于返回的 `root_uri`。运行中的任务记录可能包含 `stage`;完成后的任务 `result` 会包含带有 semantic 和 embedding 汇总的 `queue_status`。+对于 `wait=false` 的 Git 仓库来源,后台任务的 `task_type="add_resource"`,`resource_id` 等于返回的 `root_uri`。运行中的任务记录可能包含 `stage`;完成后的任务 `result` 会包含带有 semantic 和 embedding 汇总的 `queue_status`。---diff --git a/docs/zh/api/05-sessions.md b/docs/zh/api/05-sessions.mdindex 17043cc65..1a3d779e2 100644--- a/docs/zh/api/05-sessions.md+++ b/docs/zh/api/05-sessions.md@@ -1073,7 +1073,7 @@ print(f"Status: {task['status']}")}```-`stage` 可以为 `null`。资源导入任务可能报告 `queued`、`fetching`、`parsing`、`finalizing`、`processing_queue`;其他任务类型可能将其留空。实时队列计数不会出现在任务状态中;需要实时数量时使用 observer queue,任务完成后可读取 `result.queue_status`。+`stage` 可以为 `null`。Git 仓库资源导入任务可能报告 `queued`、`fetching`、`parsing`、`finalizing`、`processing_queue`;其他任务类型可能将其留空。实时队列计数不会出现在任务状态中;需要实时数量时使用 observer queue,任务完成后可读取 `result.queue_status`。**响应示例(完成)**diff --git a/openviking/server/routers/resources.py b/openviking/server/routers/resources.pyindex b1612859b..77474a565 100644--- a/openviking/server/routers/resources.py+++ b/openviking/server/routers/resources.py@@ -199,11 +199,6 @@ async def add_resource(original_filename = Noneresolved = Nonestore = None-- # Resolve path variables before passing to service.- to = resolve_path_variables(request.to) if request.to else None- parent = resolve_path_variables(request.parent) if request.parent else None-if request.temp_file_id:store = TempUploadStore.build(http_request.app.state.config)resolved = await store.resolve_for_consume(request.temp_file_id, _ctx)@@ -233,16 +228,9 @@ async def add_resource(if request.preserve_structure is not None:kwargs["preserve_structure"] = request.preserve_structure- async def _cleanup_resolved(success: bool) -> None:- if not resolved or not store:- return- try:- if success:- await store.mark_consumed(resolved, _ctx)- else:- await store.mark_failed(resolved, _ctx)- finally:- await resolved.cleanup()+ # Resolve path variables before passing to service.+ to = resolve_path_variables(request.to) if request.to else None+ parent = resolve_path_variables(request.parent) if request.parent else Noneasync def _add() -> dict[str, Any]:try:@@ -257,17 +245,19 @@ async def add_resource(timeout=request.timeout,allow_local_path_resolution=allow_local_path_resolution,enforce_public_remote_targets=True,- source_cleanup=_cleanup_resolved if resolved and not request.wait else None,**kwargs,)except Exception:- if request.wait:- await _cleanup_resolved(False)+ if resolved and store:+ await store.mark_failed(resolved, _ctx)raiseelse:- if request.wait:- await _cleanup_resolved(True)+ if resolved and store:+ await store.mark_consumed(resolved, _ctx)return result+ finally:+ if resolved:+ await resolved.cleanup()execution = await run_operation(operation="resources.add_resource",diff --git a/openviking/service/resource_service.py b/openviking/service/resource_service.pyindex a0a72f98c..7e26dd30e 100644--- a/openviking/service/resource_service.py+++ b/openviking/service/resource_service.py@@ -10,9 +10,8 @@ import asyncioimport contextlibimport inspectimport json-import osimport time-from collections.abc import Awaitable, Callable+from collections.abc import Callablefrom dataclasses import dataclassfrom pathlib import Pathfrom typing import TYPE_CHECKING, Any, Dict, List, Optional@@ -47,7 +46,6 @@ from openviking_cli.exceptions import (DeadlineExceededError,InvalidArgumentError,NotInitializedError,- OpenVikingError,)from openviking_cli.utils import get_logger@@ -132,7 +130,7 @@ class ResourceService:await asyncio.gather(*tasks, return_exceptions=True)self._background_tasks.clear()- async def enqueue_add_resource(+ async def enqueue_git_add_resource(self,path: str,ctx: RequestContext,@@ -147,10 +145,9 @@ class ResourceService:skip_watch_management: bool = False,allow_local_path_resolution: bool = True,enforce_public_remote_targets: bool = False,- source_cleanup: Optional[Callable[[bool], Awaitable[None]]] = None,**kwargs,) -> Dict[str, Any]:- """Start background resource ingestion while preserving the normal result shape."""+ """Start background ingestion for Git repositories while reserving the target URI."""self._ensure_initialized()if to:@@ -176,11 +173,7 @@ class ResourceService:path = require_remote_resource_source(path)kwargs.setdefault("request_validator", ensure_public_remote_target)- source_info = await self._preflight_resource_source(- path,- allow_local_path_resolution=allow_local_path_resolution,- request_validator=kwargs.get("request_validator"),- )+ source_info = await self._preflight_git_source(path)source_name = kwargs.get("source_name") or source_info.source_nameif source_name:kwargs["source_name"] = source_name@@ -230,7 +223,6 @@ class ResourceService:ctx=ctx,add_kwargs=add_kwargs,resource_lock=resource_lock,- source_cleanup=source_cleanup,))resource_lock = NO_LOCK@@ -243,9 +235,6 @@ class ResourceService:}except Exception:await resource_lock.close()- if source_cleanup is not None:- with contextlib.suppress(Exception):- await source_cleanup(False)raiseasync def _run_add_resource_task(@@ -255,7 +244,6 @@ class ResourceService:ctx: RequestContext,add_kwargs: Dict[str, Any],resource_lock: LockLease,- source_cleanup: Optional[Callable[[bool], Awaitable[None]]] = None,) -> None:from openviking.service.task_tracker import get_task_tracker@@ -269,7 +257,6 @@ class ResourceService:user_id=ctx.user.user_id,)- success = Falsetry:await task_tracker.start(task_id,@@ -301,7 +288,6 @@ class ResourceService:user_id=ctx.user.user_id,)return- success = Trueawait task_tracker.complete(task_id,result,@@ -325,15 +311,6 @@ class ResourceService:)finally:await resource_lock.close()- if source_cleanup is not None:- try:- await source_cleanup(success)- except Exception as exc:- logger.warning(- "[ResourceService] Failed to cleanup async resource source for task %s: %s",- task_id,- exc,- )async def _plan_resource_target(self,@@ -394,73 +371,6 @@ class ResourceService:return parsed.rsplit("/", 1)[-1]return _smart_stem(Path(path).name or "resource")- async def _preflight_resource_source(- self,- path: str,- *,- allow_local_path_resolution: bool,- request_validator: Optional[Callable[[str], None]] = None,- ) -> _ResourceSourceInfo:- if not is_remote_resource_source(path):- if allow_local_path_resolution and not Path(path).expanduser().exists():- raise InvalidArgumentError(f"Resource source does not exist: {path}")- return _ResourceSourceInfo(source_name=Path(path).name)- if is_git_repo_url(path):- return await self._preflight_git_source(path)- if path.startswith(("http://", "https://")):- return await self._preflight_http_source(path, request_validator=request_validator)- return await self._preflight_git_source(path)-- async def _preflight_http_source(- self,- url: str,- *,- request_validator: Optional[Callable[[str], None]] = None,- ) -> _ResourceSourceInfo:- from openviking.parse.accessors.http_accessor import HTTPAccessor- from openviking.parse.base import lazy_import- from openviking.utils.network_guard import build_httpx_request_validation_hooks-- httpx = lazy_import("httpx")- event_hooks = build_httpx_request_validation_hooks(request_validator)- client_kwargs: Dict[str, Any] = {- "timeout": 5.0,- "follow_redirects": True,- }- if event_hooks:- client_kwargs["event_hooks"] = event_hooks- client_kwargs["trust_env"] = False- headers = {"User-Agent": "OpenViking"}- if "github.com" in url and os.environ.get("GITHUB_TOKEN"):- headers["Authorization"] = f"token {os.environ['GITHUB_TOKEN']}"-- try:- async with httpx.AsyncClient(**client_kwargs) as client:- response = await client.head(url, headers=headers)- if response.status_code < 400:- response.raise_for_status()- else:- await self._probe_http_get(client, url, headers)- except OpenVikingError:- raise- except Exception as exc:- raise InvalidArgumentError(f"Cannot access remote URL: {url}. {exc}") from exc-- source_name = HTTPAccessor._extract_filename_from_url(url)- return _ResourceSourceInfo(- source_name=source_name,- source_path=source_name or url,- )-- @staticmethod- async def _probe_http_get(client: Any, url: str, headers: Dict[str, str]) -> None:- probe_headers = dict(headers)- probe_headers["Range"] = "bytes=0-0"- async with client.stream("GET", url, headers=probe_headers) as response:- response.raise_for_status()- async for _ in response.aiter_bytes():- break-async def _preflight_git_source(self, source: str) -> _ResourceSourceInfo:try:proc = await asyncio.create_subprocess_exec(@@ -510,7 +420,6 @@ class ResourceService:skip_watch_management: bool = False,allow_local_path_resolution: bool = True,enforce_public_remote_targets: bool = False,- source_cleanup: Optional[Callable[[bool], Awaitable[None]]] = None,resource_lock: Optional[LockLease] = None,stage_callback: Optional[Callable[[str], Any]] = None,**kwargs,@@ -552,8 +461,8 @@ class ResourceService:InvalidArgumentError: If the URI scope is not 'resources'"""self._ensure_initialized()- if not wait:- return await self.enqueue_add_resource(+ if not wait and is_git_repo_url(path):+ return await self.enqueue_git_add_resource(path=path,ctx=ctx,to=to,@@ -567,7 +476,6 @@ class ResourceService:skip_watch_management=skip_watch_management,allow_local_path_resolution=allow_local_path_resolution,enforce_public_remote_targets=enforce_public_remote_targets,- source_cleanup=source_cleanup,**kwargs,)@@ -575,6 +483,7 @@ class ResourceService:telemetry = get_current_telemetry()telemetry_id = register_wait_telemetry(wait)request_wait_tracker = get_request_wait_tracker()+ monitor_started = Falseif telemetry_id:request_wait_tracker.register_request(telemetry_id)watch_manager = self._get_watch_manager()@@ -625,50 +534,53 @@ class ResourceService:if result.get("status") == "error":return result- if stage_callback is not None:- stage_result = stage_callback("processing_queue")- if inspect.isawaitable(stage_result):- await stage_result- wait_start = time.perf_counter()- try:- with telemetry.measure("resource.wait"):- if telemetry_id:- await request_wait_tracker.wait_for_request(- telemetry_id,- timeout=timeout,- poll_interval=0.05,- )- status = request_wait_tracker.build_queue_status(telemetry_id)- else:- qm = get_queue_manager()- status = build_queue_status_payload(await qm.wait_complete(timeout=timeout))- except TimeoutError as exc:- telemetry.set_error(- "resource_service.wait_complete",- "DEADLINE_EXCEEDED",- str(exc),- )- raise DeadlineExceededError("queue processing", timeout) from exc- queue_wait_duration_ms = round((time.perf_counter() - wait_start) * 1000, 3)- try:- from openviking.metrics.datasources.resource import (- ResourceIngestionEventDataSource,- )+ elif wait:+ if stage_callback is not None:+ stage_result = stage_callback("processing_queue")+ if inspect.isawaitable(stage_result):+ await stage_result+ wait_start = time.perf_counter()+ try:+ with telemetry.measure("resource.wait"):+ if telemetry_id:+ await request_wait_tracker.wait_for_request(+ telemetry_id,+ timeout=timeout,+ poll_interval=0.05,+ )+ status = request_wait_tracker.build_queue_status(telemetry_id)+ else:+ qm = get_queue_manager()+ status = build_queue_status_payload(+ await qm.wait_complete(timeout=timeout)+ )+ except TimeoutError as exc:+ telemetry.set_error(+ "resource_service.wait_complete",+ "DEADLINE_EXCEEDED",+ str(exc),+ )+ raise DeadlineExceededError("queue processing", timeout) from exc+ queue_wait_duration_ms = round((time.perf_counter() - wait_start) * 1000, 3)+ try:+ from openviking.metrics.datasources.resource import (+ ResourceIngestionEventDataSource,+ )- ResourceIngestionEventDataSource.record_wait(- operation="queue_processing",- duration_seconds=float(queue_wait_duration_ms) / 1000.0,- account_id=getattr(ctx, "account_id", None),+ ResourceIngestionEventDataSource.record_wait(+ operation="queue_processing",+ duration_seconds=float(queue_wait_duration_ms) / 1000.0,+ account_id=getattr(ctx, "account_id", None),+ )+ except Exception:+ pass+ result["queue_status"] = status+ record_resource_wait_metrics(+ telemetry_id=telemetry_id,+ queue_status=status,+ root_uri=result.get("root_uri"),)- except Exception:- pass- result["queue_status"] = status- record_resource_wait_metrics(- telemetry_id=telemetry_id,- queue_status=status,- root_uri=result.get("root_uri"),- )- telemetry.set("queue.wait.duration_ms", queue_wait_duration_ms)+ telemetry.set("queue.wait.duration_ms", queue_wait_duration_ms)if watch_manager and not skip_watch_management:with telemetry.measure("resource.watch"):if watch_interval > 0:@@ -713,6 +625,40 @@ class ResourceService:logger.warning(f"[ResourceService] Failed to cancel watch task for {to}: {e}")+ if not wait:+ from openviking.service.task_tracker import get_task_tracker++ task_tracker = get_task_tracker()+ root_uri = result.get("root_uri", "")+ task = await task_tracker.create(+ "add_resource",+ resource_id=root_uri,+ account_id=ctx.account_id,+ user_id=ctx.user.user_id,+ )+ result["task_id"] = task.task_id+ if telemetry_id:+ monitor_started = True+ background = asyncio.create_task(+ self._monitor_queue_processing(+ task.task_id,+ telemetry_id,+ ctx.account_id,+ ctx.user.user_id,+ )+ )+ self._background_tasks.add(background)+ background.add_done_callback(self._background_tasks.discard)+ else:+ await task_tracker.start(+ task.task_id, account_id=ctx.account_id, user_id=ctx.user.user_id+ )+ await task_tracker.complete(+ task.task_id,+ {"root_uri": root_uri},+ account_id=ctx.account_id,+ user_id=ctx.user.user_id,+ )return resultexcept Exception as exc:telemetry.set_error(@@ -726,8 +672,9 @@ class ResourceService:"resource.request.duration_ms",round((time.perf_counter() - request_start) * 1000, 3),)- get_request_wait_tracker().cleanup(telemetry_id)- unregister_wait_telemetry(telemetry_id)+ if wait or not telemetry_id or not monitor_started:+ get_request_wait_tracker().cleanup(telemetry_id)+ unregister_wait_telemetry(telemetry_id)async def _monitor_queue_processing(self,diff --git a/tests/server/test_api_resources.py b/tests/server/test_api_resources.pyindex 7074d22c5..c3c2ce516 100644--- a/tests/server/test_api_resources.py+++ b/tests/server/test_api_resources.py@@ -46,10 +46,8 @@ async def test_add_resource_success(assert "telemetry" not in bodyassert body["result"]["status"] == "success"assert body["result"]["root_uri"].startswith("viking://")+ assert "source_path" in body["result"]assert body["result"]["task_id"]- task = await _wait_task_terminal(client, body["result"]["task_id"])- assert task["status"] == "completed"- assert task["result"]["root_uri"] == body["result"]["root_uri"]async def test_add_resource_with_wait(@@ -399,7 +397,7 @@ async def test_add_resource_with_resources_root_to_trailing_slash_keeps_single_fassert body["result"]["root_uri"] == "viking://resources/aa"-async def test_add_resource_async_auto_name_reserves_unique_root_uri(+async def test_add_resource_non_wait_auto_name_resolves_unique_root_uri(client: httpx.AsyncClient,upload_temp_dir,):@@ -434,11 +432,6 @@ async def test_add_resource_async_auto_name_reserves_unique_root_uri(assert first_result["root_uri"] == "viking://resources/same"assert second_result["root_uri"] == "viking://resources/same_1"- first_task = await _wait_task_terminal(client, first_result["task_id"])- second_task = await _wait_task_terminal(client, second_result["task_id"])- assert first_task["result"]["root_uri"] == first_result["root_uri"]- assert second_task["result"]["root_uri"] == second_result["root_uri"]-async def test_wait_processed_empty_queue(client: httpx.AsyncClient):resp = await client.post(@@ -697,7 +690,7 @@ async def test_add_resource_rejects_temp_file_id_symlink(assert body["error"]["code"] == "PERMISSION_DENIED"-async def test_add_resource_async_returns_task_id(+async def test_add_resource_non_wait_returns_queue_task_id(client: httpx.AsyncClient,sample_markdown_file,upload_temp_dir,@@ -717,9 +710,6 @@ async def test_add_resource_async_returns_task_id(assert "task_id" in body["result"]assert body["result"]["task_id"]assert body["result"]["root_uri"].startswith("viking://")- task = await _wait_task_terminal(client, body["result"]["task_id"])- assert task["status"] == "completed"- assert task["result"]["root_uri"] == body["result"]["root_uri"]async def test_add_resource_sync_no_task_id(@@ -741,7 +731,7 @@ async def test_add_resource_sync_no_task_id(assert "task_id" not in body["result"]-async def test_add_resource_async_task_queryable(+async def test_add_resource_non_wait_queue_task_queryable(client: httpx.AsyncClient,sample_markdown_file,upload_temp_dir,@@ -759,11 +749,12 @@ async def test_add_resource_async_task_queryable(},)task_id = resp.json()["result"]["task_id"]- root_uri = resp.json()["result"]["root_uri"]- result = await _wait_task_terminal(client, task_id)+ await asyncio.sleep(2.0)++ task_resp = await client.get(f"/api/v1/tasks/{task_id}")+ assert task_resp.status_code == 200+ result = task_resp.json()["result"]assert result["task_id"] == task_idassert result["task_type"] == "add_resource"- assert result["status"] == "completed"- assert result["stage"] in {"queued", "fetching", "parsing", "finalizing", "processing_queue"}- assert result["result"]["root_uri"] == root_uri+ assert result["status"] in {"running", "completed", "failed"}diff --git a/tests/service/test_resource_service_tasks.py b/tests/service/test_resource_service_tasks.pydeleted file mode 100644index 1153c2865..000000000--- a/tests/service/test_resource_service_tasks.py+++ /dev/null@@ -1,105 +0,0 @@-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.-# SPDX-License-Identifier: AGPL-3.0--import pytest--from openviking.server.identity import RequestContext, Role-from openviking.service.resource_service import ResourceService-from openviking.service.task_tracker import (- TaskStatus,- TaskTracker,- reset_task_tracker,- set_task_tracker,-)-from openviking.storage.transaction import NO_LOCK-from openviking_cli.session.user_id import UserIdentifier--pytestmark = pytest.mark.asyncio---class _NoopTaskStore:- async def create(self, task):- return None-- async def update(self, task):- return None-- async def get(self, task_id, *, account_id=None, user_id=None):- return None-- async def list(self, account_id, *, user_id=None):- return []-- async def delete(self, task_id, *, account_id, user_id=None):- return None--[email protected](autouse=True)-def clean_task_tracker():- reset_task_tracker()- yield- reset_task_tracker()---async def test_run_add_resource_task_fails_when_queue_status_has_errors(monkeypatch):- tracker = TaskTracker(store=_NoopTaskStore())- set_task_tracker(tracker)- service = ResourceService()- ctx = RequestContext(- user=UserIdentifier("acme", "alice", "agent-1"),- role=Role.ADMIN,- )- task = await tracker.create(- "add_resource",- resource_id="viking://resources/demo",- account_id=ctx.account_id,- user_id=ctx.user.user_id,- )- cleanup_calls = []-- async def fake_add_resource(**kwargs):- assert kwargs["wait"] is True- return {- "status": "success",- "root_uri": "viking://resources/demo",- "queue_status": {- "Semantic": {- "processed": 0,- "requeue_count": 0,- "error_count": 1,- "errors": [{"message": "semantic processing failed"}],- },- "Embedding": {- "processed": 0,- "requeue_count": 0,- "error_count": 0,- "errors": [],- },- },- }-- async def cleanup(success: bool):- cleanup_calls.append(success)-- monkeypatch.setattr(service, "add_resource", fake_add_resource)-- await service._run_add_resource_task(- task.task_id,- ctx=ctx,- add_kwargs={"path": "/tmp/demo.md", "ctx": ctx},- resource_lock=NO_LOCK,- source_cleanup=cleanup,- )-- stored = await tracker.get(- task.task_id,- account_id=ctx.account_id,- user_id=ctx.user.user_id,- )- assert stored is not None- assert stored.status == TaskStatus.FAILED- assert stored.result is None- assert "queue processing failed" in stored.error- assert "Semantic error_count=1" in stored.error- assert "semantic processing failed" in stored.error- assert cleanup_calls == [False]