Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat: support remote vikingdb glob (#4586)

ByteDanceLiuYang18fa9573ce4e
diff --git a/docs/en/configuration/01-server.md b/docs/en/configuration/01-server.md
index 462e0675f..0e4cbcecb 100644
--- a/docs/en/configuration/01-server.md
+++ b/docs/en/configuration/01-server.md
@@ -50,6 +50,7 @@ Optional sections use their defaults when omitted. Unknown fields are rejected.
| `rerank` | object | disabled | Retrieval result reranking |
| `retrieval` | object | see below | Ranking and intent-analysis behavior |
| `grep` | object | built-in defaults | Text search engine |
+| `glob` | object | built-in defaults | Path glob engine |
| `storage` | object | local | Workspace, file system, and vector database |
| `queue_workers` | object | see below | Runtime concurrency for QueueFS consumer workers |
| `server` | object | local development | HTTP, authentication, uploads, and observability |
diff --git a/docs/en/guides/01-configuration.md b/docs/en/guides/01-configuration.md
index 1df466704..d31ded75d 100644
--- a/docs/en/guides/01-configuration.md
+++ b/docs/en/guides/01-configuration.md
@@ -1006,6 +1006,24 @@ Grep engine configuration for content pattern search. These settings are server-
For VikingDB / Volcengine FullText grep, OpenViking writes a `content` text field for BM25 recall. The source context keeps the full content, while the vector-store write payload truncates this field to **1 MB** at the final adapter boundary to stay within backend payload limits. Only VikingDB-backed backends use `content`; on all other backends (`local`, `cuvs`, `http`) the field is not written.
+### glob
+
+Glob engine configuration for path pattern matching. These settings are server-side only and cannot be overridden per request.
+
+```json
+{
+ "glob": {
+ "engine": "fs",
+ "switch_to_remote_threshold": 100
+ }
+}
+```
+
+| Parameter | Type | Description | Default |
+|-----------|------|-------------|---------|
+| `engine` | str | Path matching engine mode: `"auto"` uses remote `path_glob` processing when a VikingDB / Volcengine vector store is available and the search-scope record count reaches the threshold; otherwise it falls back to local filesystem search. `"fs"` forces local filesystem search only. | `"fs"` |
+| `switch_to_remote_threshold` | int | Record-count threshold at which `auto` mode switches to remote `path_glob`. Set to `0` to always use remote path matching. Must be ≥ 0. | `100` |
+
### storage
Storage configuration for context data, including file storage (RAGFS) and vector database storage (VectorDB).
diff --git a/docs/zh/configuration/01-server.md b/docs/zh/configuration/01-server.md
index 50592f531..bc6c073bc 100644
--- a/docs/zh/configuration/01-server.md
+++ b/docs/zh/configuration/01-server.md
@@ -50,6 +50,7 @@ openviking-server --config /path/to/ov.conf
| `rerank` | object | disabled | 检索结果重排模型 |
| `retrieval` | object | 见下表 | 检索排序和意图分析策略 |
| `grep` | object | 内置默认值 | 文本搜索引擎配置 |
+| `glob` | object | 内置默认值 | 路径模式匹配引擎配置 |
| `storage` | object | 本地存储 | 工作目录、文件系统和向量数据库 |
| `queue_workers` | object | 见下表 | QueueFS 消费 worker 的运行时并发配置 |
| `server` | object | 本地开发模式 | HTTP 服务、鉴权、上传和可观测性 |
diff --git a/docs/zh/guides/01-configuration.md b/docs/zh/guides/01-configuration.md
index 5900e07d9..c711b05e6 100644
--- a/docs/zh/guides/01-configuration.md
+++ b/docs/zh/guides/01-configuration.md
@@ -974,6 +974,24 @@ Grep 引擎配置,用于内容模式搜索。这些设置为服务端配置,
对于 VikingDB / Volcengine FullText grep,OpenViking 会写入 `content` text 字段用于 BM25 召回。源上下文中保留完整内容,仅在最终写入向量库 adapter payload 时将该字段截断到 **1 MB**,以满足后端 payload 限制。只有 VikingDB 系后端使用 `content`;其它后端(`local`、`cuvs`、`http`)不写入该字段。
+### glob
+
+Glob 引擎配置,用于路径模式匹配。这些设置为服务端配置,不支持请求级别覆盖。
+
+```json
+{
+ "glob": {
+ "engine": "fs",
+ "switch_to_remote_threshold": 100
+ }
+}
+```
+
+| 参数 | 类型 | 说明 | 默认值 |
+|------|------|------|--------|
+| `engine` | str | 路径匹配引擎模式:`"auto"` 在 VikingDB / Volcengine 向量库可用且搜索范围记录数达到阈值时,使用远程 `path_glob` 后处理;不可用或失败时回退到本地文件系统搜索。`"fs"` 强制仅使用本地文件系统搜索。 | `"fs"` |
+| `switch_to_remote_threshold` | int | `auto` 模式切换到远程 `path_glob` 的记录数阈值。当搜索范围内记录数达到此阈值时使用远程路径匹配。设为 `0` 表示始终使用远程路径匹配。必须 ≥ 0。 | `100` |
+
### storage
用于存储上下文数据 ,包括文件存储(RAGFS)和向量库存储(VectorDB)。
diff --git a/examples/ov.conf.example b/examples/ov.conf.example
index ac6138692..048a74d3d 100644
--- a/examples/ov.conf.example
+++ b/examples/ov.conf.example
@@ -233,6 +233,7 @@
"recall_rewrite_timeout_s": 30.0,
},
"grep": {"engine": "auto", "switch_to_remote_threshold": 10000},
+ "glob": {"engine": "fs", "switch_to_remote_threshold": 100},
"auto_generate_l0": true,
"auto_generate_l1": true,
"default_search_mode": "thinking",
diff --git a/openviking/service/core.py b/openviking/service/core.py
index 3d80f4861..dc8a916d9 100644
--- a/openviking/service/core.py
+++ b/openviking/service/core.py
@@ -388,6 +388,7 @@ class OpenVikingService:
acl_manager=self._vikingdb_manager.acl_manager,
retrieval_config=config.retrieval,
grep_config=config.grep,
+ glob_config=config.glob,
enable_recorder=enable_recorder,
encryptor=self._encryptor,
)
diff --git a/openviking/service/fs_service.py b/openviking/service/fs_service.py
index d0d712884..ee5b46940 100644
--- a/openviking/service/fs_service.py
+++ b/openviking/service/fs_service.py
@@ -1064,15 +1064,21 @@ class FSService:
viking_fs = self._ensure_initialized()
normalized_tags = normalize_search_tags(tags, discard_invalid=True)
project_tags = bool(normalized_tags) or include_tags
+ tag_filter = None
+ if normalized_tags:
+ from openviking.utils.tags import build_search_tags_filter
+
+ tag_filter = build_search_tags_filter(normalized_tags)
result = dict(
await viking_fs.glob(
pattern,
uri=uri,
- node_limit=None if normalized_tags else node_limit,
+ node_limit=node_limit,
ctx=ctx,
extra_fields=extra_fields
if extra_fields is not None
else ([] if project_tags else None),
+ tag_filter=tag_filter,
)
)
if not project_tags:
diff --git a/openviking/storage/vectordb/collection/collection.py b/openviking/storage/vectordb/collection/collection.py
index 9910c7061..93fc66677 100644
--- a/openviking/storage/vectordb/collection/collection.py
+++ b/openviking/storage/vectordb/collection/collection.py
@@ -116,6 +116,7 @@ class ICollection(ABC):
offset: int = 0,
filters: Optional[Dict[str, Any]] = None,
output_fields: Optional[List[str]] = None,
+ advance: Optional[Dict[str, Any]] = None,
) -> SearchResult:
raise NotImplementedError
@@ -475,6 +476,7 @@ class Collection:
offset: int = 0,
filters: Optional[Dict[str, Any]] = None,
output_fields: Optional[List[str]] = None,
+ advance: Optional[Dict[str, Any]] = None,
):
"""Retrieve random documents from the index.
@@ -494,7 +496,14 @@ class Collection:
"""
if self.__collection is None:
raise RuntimeError("Collection is closed")
- return self.__collection.search_by_random(index_name, limit, offset, filters, output_fields)
+ return self.__collection.search_by_random(
+ index_name=index_name,
+ limit=limit,
+ offset=offset,
+ filters=filters,
+ output_fields=output_fields,
+ advance=advance,
+ )
def search_by_scalar(
self,
diff --git a/openviking/storage/vectordb/collection/http_collection.py b/openviking/storage/vectordb/collection/http_collection.py
index 7386a9c70..5dc9e2d3a 100644
--- a/openviking/storage/vectordb/collection/http_collection.py
+++ b/openviking/storage/vectordb/collection/http_collection.py
@@ -523,20 +523,24 @@ class HttpCollection(ICollection):
offset: int = 0,
filters: Optional[Dict[str, Any]] = None,
output_fields: Optional[List[str]] = None,
+ advance: Optional[Dict[str, Any]] = None,
) -> SearchResult:
url = self.url_prefix + "api/vikingdb/data/search/random"
+ payload = {
+ "project": self.project_name,
+ "collection_name": self.collection_name,
+ "index_name": index_name,
+ "filter": json.dumps(filters) if filters else None,
+ "output_fields": json.dumps(output_fields) if output_fields else None,
+ "limit": limit,
+ "offset": offset,
+ "advance": json.dumps(advance) if advance else None,
+ }
+ payload = {k: v for k, v in payload.items() if v is not None}
response = requests.post(
url,
headers=headers,
- json={
- "project": self.project_name,
- "collection_name": self.collection_name,
- "index_name": index_name,
- "filter": json.dumps(filters) if filters else None,
- "output_fields": json.dumps(output_fields) if output_fields else None,
- "limit": limit,
- "offset": offset,
- },
+ json=payload,
timeout=DEFAULT_TIMEOUT,
)
# logger.info(f"SearchByRandom response: {response.text}")
diff --git a/openviking/storage/vectordb/collection/local_collection.py b/openviking/storage/vectordb/collection/local_collection.py
index 9385504f5..9de47266a 100644
--- a/openviking/storage/vectordb/collection/local_collection.py
+++ b/openviking/storage/vectordb/collection/local_collection.py
@@ -695,6 +695,7 @@ class LocalCollection(ICollection):
offset: int = 0,
filters: Optional[Dict[str, Any]] = None,
output_fields: Optional[List[str]] = None,
+ advance: Optional[Dict[str, Any]] = None,
) -> SearchResult:
dense_vector = [random.uniform(-1, 1) for _ in range(self.meta.vector_dim)]
return self.search_by_vector(
diff --git a/openviking/storage/vectordb/collection/vikingdb_collection.py b/openviking/storage/vectordb/collection/vikingdb_collection.py
index b36bc00e9..098261029 100644
--- a/openviking/storage/vectordb/collection/vikingdb_collection.py
+++ b/openviking/storage/vectordb/collection/vikingdb_collection.py
@@ -351,6 +351,7 @@ class VikingDBCollection(ICollection):
offset: int = 0,
filters: Optional[Dict[str, Any]] = None,
output_fields: Optional[List[str]] = None,
+ advance: Optional[Dict[str, Any]] = None,
) -> SearchResult:
path = "/api/vikingdb/data/search/random"
data = {
@@ -363,6 +364,8 @@ class VikingDBCollection(ICollection):
"offset": offset,
"ignore_unknown_fields": True,
}
+ if advance is not None:
+ data["advance"] = advance
resp_data = self._data_post(path, data)
return self._parse_search_result(resp_data)
diff --git a/openviking/storage/vectordb/collection/volcengine_api_key_collection.py b/openviking/storage/vectordb/collection/volcengine_api_key_collection.py
index 7f5f6fbe5..077369ddc 100644
--- a/openviking/storage/vectordb/collection/volcengine_api_key_collection.py
+++ b/openviking/storage/vectordb/collection/volcengine_api_key_collection.py
@@ -471,6 +471,7 @@ class VolcengineApiKeyCollection(ICollection):
offset: int = 0,
filters: Optional[Dict[str, Any]] = None,
output_fields: Optional[List[str]] = None,
+ advance: Optional[Dict[str, Any]] = None,
) -> SearchResult:
path = "/api/vikingdb/data/search/random"
data = {
@@ -481,6 +482,8 @@ class VolcengineApiKeyCollection(ICollection):
"offset": offset,
"ignore_unknown_fields": True,
}
+ if advance is not None:
+ data["advance"] = advance
resp_data = self._data_post(path, data)
return self._parse_search_result(resp_data)
diff --git a/openviking/storage/vectordb/collection/volcengine_collection.py b/openviking/storage/vectordb/collection/volcengine_collection.py
index 1338f0146..792757f91 100644
--- a/openviking/storage/vectordb/collection/volcengine_collection.py
+++ b/openviking/storage/vectordb/collection/volcengine_collection.py
@@ -274,19 +274,24 @@ class VolcengineCollection(ICollection):
normalized["op"] = "time_range"
return normalized
- def _data_post(self, path: str, data: Dict[str, Any]):
+ def _data_post(self, path: str, data: Dict[str, Any], *, raise_for_status: bool = False):
# Centralized sanitization at the request exit, covering all data API inputs
safe_data = self._sanitize_payload(data)
if isinstance(safe_data, dict) and "filter" in safe_data:
safe_data["filter"] = self._normalize_date_time_filter(safe_data["filter"])
response = self.data_client.do_req("POST", path, req_body=safe_data)
if response.status_code != 200:
- logger.error(f"Request to {path} failed: {response.text}")
+ error = self._build_response_error(response, path)
+ logger.error(str(error))
+ if raise_for_status:
+ raise error
return {}
try:
result = response.json()
return result.get("result", {})
except json.JSONDecodeError:
+ if raise_for_status:
+ raise ConnectionError(f"Request to {path} failed: invalid JSON response")
return {}
def _data_get(self, path: str, params: Dict[str, Any]):
@@ -561,6 +566,7 @@ class VolcengineCollection(ICollection):
offset: int = 0,
filters: Optional[Dict[str, Any]] = None,
output_fields: Optional[List[str]] = None,
+ advance: Optional[Dict[str, Any]] = None,
) -> SearchResult:
path = "/api/vikingdb/data/search/random"
data = {
@@ -573,7 +579,9 @@ class VolcengineCollection(ICollection):
"offset": offset,
"ignore_unknown_fields": True,
}
- resp_data = self._data_post(path, data)
+ if advance is not None:
+ data["advance"] = advance
+ resp_data = self._data_post(path, data, raise_for_status=True)
return self._parse_search_result(resp_data)
def search_by_keywords(
diff --git a/openviking/storage/vectordb/service/api_fastapi.py b/openviking/storage/vectordb/service/api_fastapi.py
index ff2877d49..e7dbee4a7 100644
--- a/openviking/storage/vectordb/service/api_fastapi.py
+++ b/openviking/storage/vectordb/service/api_fastapi.py
@@ -582,6 +582,7 @@ async def search_by_random(request: SearchByRandomRequest, req: Request):
filters = data_utils.convert_dict(request.filter)
output_fields = data_utils.convert_dict(request.output_fields)
+ advance = data_utils.convert_dict(request.advance) if request.advance else None
limit = request.limit or 10
if limit <= 0:
return error_response(
@@ -601,6 +602,7 @@ async def search_by_random(request: SearchByRandomRequest, req: Request):
offset=offset,
filters=filters,
output_fields=output_fields,
+ advance=advance,
)
return success_response("search success", asdict(result), request=req)
except VikingDBException as e:
diff --git a/openviking/storage/vectordb/service/app_models.py b/openviking/storage/vectordb/service/app_models.py
index 2cde62eb6..169eb58a9 100644
--- a/openviking/storage/vectordb/service/app_models.py
+++ b/openviking/storage/vectordb/service/app_models.py
@@ -152,6 +152,7 @@ class SearchByRandomRequest(BaseModel):
project: Optional[str] = Field("default", description="Project name")
filter: Optional[Any] = Field(None, description="Filter conditions")
output_fields: Optional[Any] = Field(None, description="Output fields")
+ advance: Optional[Any] = Field(None, description="Advanced search options")
limit: Optional[int] = Field(10, description="Result limit")
offset: Optional[int] = Field(0, description="Result offset")
diff --git a/openviking/storage/vectordb_adapters/base.py b/openviking/storage/vectordb_adapters/base.py
index 2b1050d26..4c7155f57 100644
--- a/openviking/storage/vectordb_adapters/base.py
+++ b/openviking/storage/vectordb_adapters/base.py
@@ -520,6 +520,34 @@ class CollectionAdapter(ABC):
records.append(record)
return records
+ def search_by_random(
+ self,
+ *,
+ filter: Optional[Dict[str, Any] | FilterExpr] = None,
+ limit: int = 10,
+ offset: int = 0,
+ output_fields: Optional[list[str]] = None,
+ advance: Optional[Dict[str, Any]] = None,
+ ) -> list[Dict[str, Any]]:
+ coll = self.get_collection()
+ result = coll.search_by_random(
+ index_name=self._index_name,
+ limit=limit,
+ offset=offset,
+ filters=self._compile_filter(filter),
+ output_fields=output_fields,
+ advance=advance,
+ )
+
+ records: list[Dict[str, Any]] = []
+ for item in result.data:
+ record = dict(item.fields) if item.fields else {}
+ record["id"] = item.id
+ record["_score"] = _normalize_result_score(item.score)
+ record = self._normalize_record_for_read(record)
+ records.append(record)
+ return records
+
def delete(
self,
*,
diff --git a/openviking/storage/viking_fs/__init__.py b/openviking/storage/viking_fs/__init__.py
index ad958ce8f..3729bf16f 100644
--- a/openviking/storage/viking_fs/__init__.py
+++ b/openviking/storage/viking_fs/__init__.py
@@ -101,7 +101,7 @@ from openviking_cli.utils.uri import VikingURI
if TYPE_CHECKING:
from openviking.storage.acl import AclManager
from openviking.storage.viking_vector_index_backend import VikingVectorIndexBackend
- from openviking_cli.utils.config import GrepConfig, RerankConfig, RetrievalConfig
+ from openviking_cli.utils.config import GlobConfig, GrepConfig, RerankConfig, RetrievalConfig
# ========== VikingFS Main Class ==========
@@ -134,6 +134,7 @@ class VikingFS(
acl_manager: Optional["AclManager"] = None,
retrieval_config: Optional["RetrievalConfig"] = None,
grep_config: Optional["GrepConfig"] = None,
+ glob_config: Optional["GlobConfig"] = None,
timeout: int = 10,
encryptor: Optional[Any] = None,
):
@@ -145,6 +146,7 @@ class VikingFS(
self.acl_manager = acl_manager
self.retrieval_config = retrieval_config
self.grep_config = grep_config
+ self.glob_config = glob_config
self._encryptor = encryptor
self._count_cache: Dict[str, tuple] = {} # cache_key → (count, timestamp)
self._count_cache_max_size = 1024
diff --git a/openviking/storage/viking_fs/_base.py b/openviking/storage/viking_fs/_base.py
index fd987a135..98a1b6b0e 100644
--- a/openviking/storage/viking_fs/_base.py
+++ b/openviking/storage/viking_fs/_base.py
@@ -14,7 +14,7 @@ from openviking_cli.utils.logger import get_logger
if TYPE_CHECKING:
from openviking.storage.acl import AclManager
from openviking.storage.viking_vector_index_backend import VikingVectorIndexBackend
- from openviking_cli.utils.config import GrepConfig, RerankConfig, RetrievalConfig
+ from openviking_cli.utils.config import GlobConfig, GrepConfig, RerankConfig, RetrievalConfig
logger = get_logger(__name__)
@@ -200,6 +200,7 @@ def init_viking_fs(
acl_manager: Optional["AclManager"] = None,
retrieval_config: Optional["RetrievalConfig"] = None,
grep_config: Optional["GrepConfig"] = None,
+ glob_config: Optional["GlobConfig"] = None,
timeout: int = 10,
enable_recorder: bool = False,
encryptor: Optional[Any] = None,
@@ -212,6 +213,7 @@ def init_viking_fs(
rerank_config: Rerank configuration
retrieval_config: Retrieval ranking configuration
grep_config: Grep engine configuration
+ glob_config: Glob engine configuration
vector_store: Vector store instance
enable_recorder: Whether to enable IO recording
encryptor: FileEncryptor instance for encryption/decryption
@@ -228,6 +230,7 @@ def init_viking_fs(
acl_manager=acl_manager,
retrieval_config=retrieval_config,
grep_config=grep_config,
+ glob_config=glob_config,
encryptor=encryptor,
)
diff --git a/openviking/storage/viking_fs/_ops.py b/openviking/storage/viking_fs/_ops.py
index 024cd4b11..571997d6a 100644
--- a/openviking/storage/viking_fs/_ops.py
+++ b/openviking/storage/viking_fs/_ops.py
@@ -3,6 +3,7 @@
"""Core filesystem operations mixin for VikingFS."""
import asyncio
+import math
import uuid
from dataclasses import replace
from datetime import datetime, timezone
@@ -27,7 +28,7 @@ from openviking.storage.abstract_overview import (
rewrite_abstract_overview_for_transfer,
)
from openviking.storage.acl import AclAction, is_acl_uri
-from openviking.storage.expr import PathScope
+from openviking.storage.expr import And, PathScope, RawDSL
from openviking.storage.internal_names import STORAGE_INTERNAL_ENTRY_NAMES
from openviking.storage.vector_ids import is_vector_record_id, vector_record_id
from openviking.storage.viking_fs._base import (
@@ -59,6 +60,78 @@ def _glob_match_uri(entry_uri: str, is_dir: Optional[bool]) -> str:
return f"{entry_uri}/"
+_REMOTE_GLOB_OUTPUT_FIELDS = ["uri", "level", "name"]
+_REMOTE_GLOB_ENTRY_FIELDS = ["size", "mode", "modTime"]
+_REMOTE_GLOB_DEFAULT_LIMIT = 100000
+_REMOTE_GLOB_MAX_LIMIT = 100000
+_REMOTE_GLOB_POST_PROCESS_INPUT_LIMIT = 1000000
+_REMOTE_GLOB_LOCAL_STAT_FIELDS = {
+ "size",
+ "mode",
+ "modTime",
+ "mtime",
+ "is_dir",
+}
+_REMOTE_GLOB_LOCAL_COMPUTED_FIELDS = {
+ "id",
+ "count",
+ "isLocked",
+ "locked",
+}
+_GLOB_SPECIAL_CHARS = set("*?[{")
+
+
+def _normalize_uri_for_glob(uri: str) -> str:
+ return "viking://" if uri == "viking://" else uri.rstrip("/")
+
+
+def _normalize_glob_path(pattern: str) -> str:
+ return "/".join(segment for segment in pattern.split("/") if segment and segment != ".")
+
+
+def _uri_to_remote_path_pattern(uri: str, pattern: str) -> str:
+ base_path = "/" + _normalize_uri_for_glob(uri)[len("viking://") :].strip("/")
+ if base_path == "/":
+ base_path = ""
+ normalized_pattern = _normalize_glob_path(pattern)
+ if not normalized_pattern:
+ return base_path or "/"
+ return f"{base_path}/{normalized_pattern}" if base_path else f"/{normalized_pattern}"
+
+
+def _literal_glob_prefix(pattern: str) -> str:
+ parts = []
+ for segment in _normalize_glob_path(pattern).split("/"):
+ if not segment or any(ch in segment for ch in _GLOB_SPECIAL_CHARS):
+ break
+ parts.append(segment)
+ return "/".join(parts)
+
+
+def _join_uri_path(uri: str, suffix: str) -> str:
+ normalized_uri = _normalize_uri_for_glob(uri)
+ suffix = suffix.strip("/")
+ if not suffix:
+ return normalized_uri
+ if normalized_uri == "viking://":
+ return f"viking://{suffix}"
+ return f"{normalized_uri}/{suffix}"
+
+
+def _rel_path_sort_key(uri: str, root_uri: str) -> List[str]:
+ normalized_root = _normalize_uri_for_glob(root_uri)
+ normalized_uri = uri.rstrip("/")
+ if normalized_uri == normalized_root:
+ rel_path = ""
+ elif normalized_root == "viking://":
+ rel_path = normalized_uri[len("viking://") :]
+ elif normalized_uri.startswith(normalized_root + "/"):
+ rel_path = normalized_uri[len(normalized_root) + 1 :]
+ else:
+ rel_path = normalized_uri
+ return [part for part in rel_path.split("/") if part and part != "."]
+
+
class _OpsMixin:
"""Core filesystem operations (read/write/mkdir/rm/mv/stat/glob/tree/ls/temp)."""
@@ -1116,6 +1189,7 @@ class _OpsMixin:
node_limit: Optional[int] = None,
ctx: Optional[RequestContext] = None,
extra_fields: Optional[List[str]] = None,
+ tag_filter: Optional[Dict[str, Any]] = None,
) -> Dict:
"""File pattern matching, supports **/*.md recursive.
@@ -1142,7 +1216,23 @@ class _OpsMixin:
return {"matches": [], "count": 0}
raise NotFoundError(uri, "directory")
- page_size = self._glob_page_size(node_limit)
+ remote_result = await self._try_glob_vikingdb(
+ pattern=pattern,
+ uri=uri,
+ node_limit=node_limit,
+ return_entries=return_entries,
+ extra_fields=aug_fields,
+ tag_filter=tag_filter,
+ ctx=real_ctx,
+ )
+ if remote_result is not None:
+ return remote_result
+
+ # Tag filtering happens in FSService on the local fallback path. Fetch
+ # all matches so an early filesystem limit cannot discard later tagged
+ # entries. The remote path applies the same filter before its limit.
+ fs_node_limit = None if tag_filter else node_limit
+ page_size = self._glob_page_size(fs_node_limit)
continuation_token: Optional[str] = None
matches = []
while True:
@@ -1197,12 +1287,16 @@ class _OpsMixin:
matches.append(entry_stat)
else:
matches.append(match_uri)
- if node_limit is not None and node_limit > 0 and len(matches) >= node_limit:
+ if (
+ fs_node_limit is not None
+ and fs_node_limit > 0
+ and len(matches) >= fs_node_limit
+ ):
if return_entries:
await self._augment_entries_extra_fields(matches, aug_fields, ctx=ctx)
return {"matches": matches, "count": len(matches)}
- if node_limit is not None and node_limit > 0 and len(matches) >= node_limit:
+ if fs_node_limit is not None and fs_node_limit > 0 and len(matches) >= fs_node_limit:
if return_entries:
await self._augment_entries_extra_fields(matches, aug_fields, ctx=ctx)
return {"matches": matches, "count": len(matches)}
@@ -1213,6 +1307,277 @@ class _OpsMixin:
await self._augment_entries_extra_fields(matches, aug_fields, ctx=ctx)
return {"matches": matches, "count": len(matches)}
+ async def _try_glob_vikingdb(
+ self,
+ *,
+ pattern: str,
+ uri: str,
+ node_limit: Optional[int],
+ return_entries: bool,
+ extra_fields: List[str],
+ tag_filter: Optional[Dict[str, Any]],
+ ctx: RequestContext,
+ ) -> Optional[Dict[str, Any]]:
+ if not await self._should_use_vikingdb_glob(
+ pattern=pattern,
+ uri=uri,
+ node_limit=node_limit,
+ ctx=ctx,
+ ):
+ return None
+
+ vector_store = self._get_vector_store()
+ if vector_store is None or not hasattr(vector_store, "search_by_random"):
+ return None
+
+ remote_limit = self._remote_glob_limit(node_limit)
+ filter_expr = self._remote_glob_filter(uri, pattern)
+ if tag_filter:
+ filter_expr = And([filter_expr, RawDSL(tag_filter)])
+ full_pattern = _uri_to_remote_path_pattern(uri, pattern)
+ advance = {
+ "post_process_input_limit": _REMOTE_GLOB_POST_PROCESS_INPUT_LIMIT,
+ "post_process_ops": [
+ {
+ "op": "path_glob",
+ "field": "uri",
+ "pattern": full_pattern,
+ "stop_after_matches": remote_limit,
+ }
+ ],
+ }
+
+ try:
+ records = await vector_store.search_by_random(
+ filter=filter_expr,
+ limit=remote_limit,
+ offset=0,
+ output_fields=self._remote_glob_output_fields(
+ extra_fields,
+ return_entries=return_entries,
+ ),
+ advance=advance,
+ ctx=ctx,
+ )
+ except Exception as exc:
+ logger.warning("glob vikingdb step failed, falling back to fs: %s", exc)
+ return None
+
+ entries = self._remote_glob_records_to_entries(
+ records,
+ root_uri=uri,
+ ctx=ctx,
+ )
+ entries.sort(key=lambda entry: _rel_path_sort_key(str(entry.get("uri", "")), uri))
+ access = await self._can_access_many(
+ [entry["uri"] for entry in entries],
+ ctx,
+ )
+ entries = [entry for entry in entries if access.get(entry["uri"], False)]
+ if node_limit is not None and node_limit > 0:
+ entries = entries[:node_limit]
+
+ if return_entries:
+ await self._fill_remote_glob_entry_fields(entries, extra_fields, ctx=ctx)
+ return {"matches": entries, "count": len(entries)}
+
+ return {
+ "matches": [_glob_match_uri(entry["uri"], entry.get("isDir")) for entry in entries],
+ "count": len(entries),
+ }
+
+ async def _should_use_vikingdb_glob(
+ self,
+ *,
+ pattern: str,
+ uri: str,
+ node_limit: Optional[int],
+ ctx: RequestContext,
+ ) -> bool:
+ if node_limit is not None and node_limit <= 0:
+ return False
+ if not _normalize_glob_path(pattern):
+ return False
+
+ glob_config = getattr(self, "glob_config", None)
+ engine = getattr(glob_config, "engine", "fs")
+ if engine == "fs":
+ return False
+
+ vector_store = self._get_vector_store()
+ if vector_store is None:
+ return False
+
+ backend_type = getattr(vector_store, "_backend_type", "unknown")
+ if backend_type not in ("volcengine", "vikingdb"):
+ return False
+
+ if not await self._collection_has_glob_uri_field(vector_store, ctx):
+ return False
+
+ threshold = getattr(glob_config, "switch_to_remote_threshold", 100)
+ if threshold == 0:
+ return True
+
+ try:
+ scoped_uri = self._remote_glob_scoped_uri(uri, pattern)
+ count = await self._get_cached_count(scoped_uri, ctx)
+ except Exception:
+ logger.debug(
+ "glob engine=auto: count() check failed, falling back to fs", exc_info=True
+ )
+ return False
+ return count >= threshold
+
+ async def _collection_has_glob_uri_field(self, vector_store: Any, ctx: RequestContext) -> bool:
+ try:
+ if not hasattr(vector_store, "get_collection_meta"):
+ return False
+ meta = await vector_store.get_collection_meta(ctx=ctx)
+ fields = meta.get("Fields", []) if isinstance(meta, dict) else []
+ return any(
+ field.get("FieldName") == "uri" for field in fields if isinstance(field, dict)
+ )
+ except Exception:
+ logger.debug(
+ "Failed to check collection uri field, assuming no path_glob support", exc_info=True
+ )
+ return False
+
+ @staticmethod
+ def _remote_glob_limit(node_limit: Optional[int]) -> int:
+ if node_limit is None:
+ return _REMOTE_GLOB_DEFAULT_LIMIT
+ return min(max(math.ceil(node_limit * 1.2), 1), _REMOTE_GLOB_MAX_LIMIT)
+
+ @staticmethod
+ def _remote_glob_output_fields(
+ extra_fields: List[str],
+ *,
+ return_entries: bool,
+ ) -> List[str]:
+ fields = list(_REMOTE_GLOB_OUTPUT_FIELDS)
+ if return_entries:
+ for field in _REMOTE_GLOB_ENTRY_FIELDS:
+ if field not in fields:
+ fields.append(field)
+ local_only = _REMOTE_GLOB_LOCAL_STAT_FIELDS | _REMOTE_GLOB_LOCAL_COMPUTED_FIELDS
+ for field in extra_fields:
+ if field in local_only or field in fields:
+ continue
+ fields.append(field)
+ return fields
+
+ @staticmethod
+ def _remote_glob_scoped_uri(uri: str, pattern: str) -> str:
+ prefix = _literal_glob_prefix(pattern)
+ return _join_uri_path(uri, prefix) if prefix else _normalize_uri_for_glob(uri)
+
+ def _remote_glob_filter(self, uri: str, pattern: str) -> PathScope:
+ scoped_uri = self._remote_glob_scoped_uri(uri, pattern)
+ return PathScope("uri", scoped_uri, depth=-1)
+
+ def _remote_glob_records_to_entries(
+ self,
+ records: List[Dict[str, Any]],
+ *,
+ root_uri: str,
+ ctx: RequestContext,
+ ) -> List[Dict[str, Any]]:
+ seen: set[str] = set()
+ entries: List[Dict[str, Any]] = []
+ for record in records:
+ if not isinstance(record, dict):
+ continue
+ entry_uri = record.get("uri")
+ if not isinstance(entry_uri, str) or not entry_uri:
+ continue
+ entry_uri = _normalize_uri_for_glob(entry_uri)
+ if entry_uri in seen:
+ continue
+ seen.add(entry_uri)
+
+ level = record.get("level")
+ is_dir = level != 2 if level is not None else bool(record.get("isDir", False))
+ name = record.get("name") or entry_uri.rstrip("/").rsplit("/", 1)[-1]
+ if not self._is_remote_glob_uri_visible(
+ entry_uri,
+ str(name),
+ bool(is_dir),
+ root_uri,
+ ctx,
+ ):
+ continue
+ if not self._is_accessible(entry_uri, ctx):
+ continue
+
+ entry = dict(record)
+ entry["uri"] = entry_uri
+ entry["name"] = entry.get("name") or name
+ entry["isDir"] = bool(is_dir)
+ entries.append(entry)
+ return entries
+
+ def _is_remote_glob_uri_visible(
+ self,
+ entry_uri: str,
+ name: str,
+ is_dir: bool,
+ root_uri: str,
+ ctx: RequestContext,
+ ) -> bool:
+ try:
+ entry_parts = self._safe_uri_parts(entry_uri)
+ root_parts = self._safe_uri_parts(root_uri)
+ except ValueError:
+ return False
+
+ if root_parts and entry_parts[: len(root_parts)] != root_parts:
+ return False
+ if not root_parts and entry_parts:
+ if entry_parts[0] not in VikingURI.LISTABLE_SCOPES:
+ return False
+
+ relative_parts = entry_parts[len(root_parts) :]
+ if not is_dir and name.startswith("."):
+ return False
+ if name in STORAGE_INTERNAL_ENTRY_NAMES:
+ return False
+ return all(part not in STORAGE_INTERNAL_ENTRY_NAMES for part in relative_parts)
+
+ async def _fill_remote_glob_entry_fields(
+ self,
+ entries: List[Dict[str, Any]],
+ extra_fields: List[str],
+ ctx: Optional[RequestContext] = None,
+ ) -> None:
+ requested_stat_fields = {
+ field for field in extra_fields if field in _REMOTE_GLOB_LOCAL_STAT_FIELDS
+ }
+ required_fields = set(_REMOTE_GLOB_ENTRY_FIELDS) | requested_stat_fields
+ for entry in entries:
+ entry.pop("level", None)
+ entry.pop("_score", None)
+ for entry in entries:
+ if all(field in entry for field in required_fields):
+ if entry.get("isDir"):
+ entry.pop("id", None)
+ continue
+ entry_uri = entry.get("uri")
+ if not entry_uri:
+ continue
+ try:
+ stat = await self.stat(entry_uri, ctx=ctx, skip_count=True)
+ except Exception:
+ continue
+ stat.update({k: v for k, v in entry.items() if v is not None})
+ entry.clear()
+ entry.update(stat)
+ if entry.get("isDir"):
+ entry.pop("id", None)
+ if extra_fields:
+ await self._augment_entries_extra_fields(entries, extra_fields, ctx=ctx)
+
async def _batch_fetch_abstracts(
self,
entries: List[Dict[str, Any]],
diff --git a/openviking/storage/viking_vector_index_backend.py b/openviking/storage/viking_vector_index_backend.py
index e75dcbc0b..a5bb4789d 100644
--- a/openviking/storage/viking_vector_index_backend.py
+++ b/openviking/storage/viking_vector_index_backend.py
@@ -199,7 +199,6 @@ class _AsyncVectorAdapter:
await asyncio.to_thread(_update)
-
class _SingleAccountBackend:
"""绑定单个 account 的后端实现(内部类)"""
@@ -717,6 +716,27 @@ class _SingleAccountBackend:
logger.error("Error querying collection: %s", e, exc_info=True)
return []
+ async def search_by_random(
+ self,
+ filter: Optional[Dict[str, Any] | FilterExpr] = None,
+ limit: int = 10,
+ offset: int = 0,
+ output_fields: Optional[List[str]] = None,
+ advance: Optional[Dict[str, Any]] = None,
+ ) -> List[Dict[str, Any]]:
+ try:
+ return await self._async_adapter.call(
+ "search_by_random",
+ filter=self._with_account_filter(filter),
+ limit=limit,
+ offset=offset,
+ output_fields=output_fields,
+ advance=advance,
+ )
+ except Exception as e:
+ logger.error("Error searching collection by random: %s", e, exc_info=True)
+ raise
+
async def search(
self,
query_vector: Optional[List[float]] = None,
@@ -1332,6 +1352,26 @@ class VikingVectorIndexBackend:
order_desc=order_desc,
)
+ async def search_by_random(
+ self,
+ filter: Optional[Dict[str, Any] | FilterExpr] = None,
+ limit: int = 10,
+ offset: int = 0,
+ output_fields: Optional[List[str]] = None,
+ advance: Optional[Dict[str, Any]] = None,
+ *,
+ ctx: RequestContext,
+ ) -> List[Dict[str, Any]]:
+ backend = self._get_backend_for_context(ctx)
+ filter = self._merge_filters(filter, self._tenant_filter(ctx))
+ return await backend.search_by_random(
+ filter=filter,
+ limit=limit,
+ offset=offset,
+ output_fields=output_fields,
+ advance=advance,
+ )
+
async def search(
self,
query_vector: Optional[List[float]] = None,
@@ -2305,16 +2345,19 @@ class VikingVectorIndexBackend:
@staticmethod
def _merge_filters(*filters: Optional[FilterExpr]) -> Optional[FilterExpr]:
- non_empty = [
- f
- for f in filters
- if f
- and not (
- isinstance(f, RawDSL)
- and f.payload.get("op") == "and"
- and not f.payload.get("conds")
- )
- ]
+ non_empty: List[FilterExpr] = []
+ for item in filters:
+ if not item:
+ continue
+ if isinstance(item, dict):
+ item = RawDSL(item)
+ if (
+ isinstance(item, RawDSL)
+ and item.payload.get("op") == "and"
+ and not item.payload.get("conds")
+ ):
+ continue
+ non_empty.append(item)
if not non_empty:
return None
if len(non_empty) == 1:
diff --git a/openviking_cli/utils/config/__init__.py b/openviking_cli/utils/config/__init__.py
index 0a2c1e9e4..07bf7dac1 100644
--- a/openviking_cli/utils/config/__init__.py
+++ b/openviking_cli/utils/config/__init__.py
@@ -53,6 +53,7 @@ from .consts import (
)
from .embedding_config import EmbeddingConfig
from .git_config import GitConfig, GitLocalConfig, GitS3Config
+from .glob_config import GlobConfig, GlobEngine
from .grep_config import GrepConfig, GrepEngine
from .log_config import LogConfig
from .open_viking_config import (
@@ -137,6 +138,8 @@ __all__ = [
"OpenVikingConfig",
"OpenVikingConfigSingleton",
"OVCLIConfig",
+ "GlobConfig",
+ "GlobEngine",
"GrepConfig",
"GrepEngine",
"RerankConfig",
diff --git a/openviking_cli/utils/config/glob_config.py b/openviking_cli/utils/config/glob_config.py
new file mode 100644
index 000000000..81a903b43
--- /dev/null
+++ b/openviking_cli/utils/config/glob_config.py
@@ -0,0 +1,32 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+from typing import Literal
+
+from pydantic import BaseModel, Field
+
+# Glob engine mode type alias — import this instead of repeating Literal["auto", "fs"]
+GlobEngine = Literal["auto", "fs"]
+
+
+class GlobConfig(BaseModel):
+ """Configuration for glob engine behavior."""
+
+ engine: GlobEngine = Field(
+ default="fs",
+ description=(
+ "Glob engine mode: 'auto' uses remote VikingDB path_glob when available, "
+ "'fs' forces local filesystem glob."
+ ),
+ )
+
+ switch_to_remote_threshold: int = Field(
+ default=100,
+ ge=0,
+ description=(
+ "Vector record count threshold to switch to VikingDB path_glob; "
+ "0 means always use VikingDB when available."
+ ),
+ )
+
+ model_config = {"extra": "forbid"}
diff --git a/openviking_cli/utils/config/open_viking_config.py b/openviking_cli/utils/config/open_viking_config.py
index 39ad59f01..ae31e438c 100644
--- a/openviking_cli/utils/config/open_viking_config.py
+++ b/openviking_cli/utils/config/open_viking_config.py
@@ -23,6 +23,7 @@ from .consts import (
from .embedding_config import EmbeddingConfig
from .encryption_config import EncryptionConfig
from .git_config import GitConfig
+from .glob_config import GlobConfig
from .grep_config import GrepConfig
from .ingest_config import IngestConfig
from .log_config import LogConfig
@@ -204,6 +205,11 @@ class OpenVikingConfig(BaseModel):
description="Grep engine configuration",
)
+ glob: GlobConfig = Field(
+ default_factory=GlobConfig,
+ description="Glob engine configuration",
+ )
+
# Encryption configuration
encryption: EncryptionConfig = Field(
default_factory=EncryptionConfig, description="Encryption configuration"
diff --git a/tests/service/test_fs_service.py b/tests/service/test_fs_service.py
index aefcb28dc..afb0db3e3 100644
--- a/tests/service/test_fs_service.py
+++ b/tests/service/test_fs_service.py
@@ -456,7 +456,14 @@ async def test_glob_filters_and_projects_tags_before_applying_node_limit(request
"count": 1,
}
assert viking_fs.glob.await_args.kwargs["extra_fields"] == []
- assert viking_fs.glob.await_args.kwargs["node_limit"] is None
+ assert viking_fs.glob.await_args.kwargs["node_limit"] == 1
+ assert viking_fs.glob.await_args.kwargs["tag_filter"] == {
+ "op": "and",
+ "conds": [
+ {"op": "must", "field": "search_tags", "conds": ["team=search"]},
+ {"op": "must", "field": "search_tags", "conds": ["env=prod"]},
+ ],
+ }
@pytest.mark.asyncio
diff --git a/tests/storage/test_viking_fs_glob.py b/tests/storage/test_viking_fs_glob.py
index 24f876f2c..60fc08d69 100644
--- a/tests/storage/test_viking_fs_glob.py
+++ b/tests/storage/test_viking_fs_glob.py
@@ -1,18 +1,77 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
+from types import SimpleNamespace
+
import pytest
from openviking.server.identity import RequestContext, Role
+from openviking.storage.expr import And, PathScope, RawDSL
from openviking.storage.viking_fs import VikingFS
from openviking_cli.exceptions import InvalidArgumentError, NotFoundError
from openviking_cli.session.user_id import UserIdentifier
+from openviking_cli.utils.config.glob_config import GlobConfig
class _DummyAgfs:
pass
+class _RemoteGlobVectorStore:
+ _backend_type = "vikingdb"
+
+ def __init__(self, records, count=1000, data_count=None):
+ self.records = records
+ self.count_value = count
+ self.data_count = data_count
+ self.count_calls = []
+ self.random_calls = []
+
+ async def count(self, filter=None, ctx=None):
+ self.count_calls.append({"filter": filter, "ctx": ctx})
+ return self.count_value
+
+ async def get_collection_meta(self, ctx=None):
+ meta = {
+ "Fields": [
+ {"FieldName": "uri", "FieldType": "path"},
+ {"FieldName": "level", "FieldType": "int64"},
+ {"FieldName": "name", "FieldType": "string"},
+ ]
+ }
+ if self.data_count is not None:
+ meta["CollectionStats"] = {"DataCount": self.data_count}
+ return meta
+
+ async def search_by_random(
+ self,
+ *,
+ filter=None,
+ limit=10,
+ offset=0,
+ output_fields=None,
+ advance=None,
+ ctx=None,
+ ):
+ self.random_calls.append(
+ {
+ "filter": filter,
+ "limit": limit,
+ "offset": offset,
+ "output_fields": output_fields,
+ "advance": advance,
+ "ctx": ctx,
+ }
+ )
+ return list(self.records)
+
+
+class _FailingRemoteGlobVectorStore(_RemoteGlobVectorStore):
+ async def search_by_random(self, **kwargs):
+ self.random_calls.append(kwargs)
+ raise RuntimeError("remote glob failed")
+
+
def _default_ctx() -> RequestContext:
return RequestContext(user=UserIdentifier.the_default_user(), role=Role.ROOT)
@@ -42,6 +101,549 @@ async def _async_true():
return True
+def test_glob_config_defaults_to_fs_with_threshold_100():
+ config = GlobConfig()
+
+ assert config.engine == "fs"
+ assert config.switch_to_remote_threshold == 100
+
+
+async def test_glob_auto_runtime_fallback_uses_threshold_100(fs):
+ vector_store = _RemoteGlobVectorStore([], count=100)
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto")
+
+ should_use_remote = await fs._should_use_vikingdb_glob(
+ pattern="**/*.md",
+ uri="viking://resources",
+ node_limit=None,
+ ctx=_default_ctx(),
+ )
+
+ assert should_use_remote is True
+
+
+async def test_glob_auto_threshold_zero_skips_count(fs):
+ vector_store = _RemoteGlobVectorStore([], count=0)
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=0)
+
+ should_use_remote = await fs._should_use_vikingdb_glob(
+ pattern="**/*.md",
+ uri="viking://resources",
+ node_limit=None,
+ ctx=_default_ctx(),
+ )
+
+ assert should_use_remote is True
+ assert vector_store.count_calls == []
+
+
+async def test_glob_uses_remote_vikingdb_when_count_reaches_threshold(monkeypatch, fs):
+ vector_store = _RemoteGlobVectorStore(
+ [
+ {"uri": "viking://resources/docs/b.md", "level": 2, "name": "b.md"},
+ {"uri": "viking://resources/docs", "level": 0, "name": "docs"},
+ {"uri": "viking://resources/docs/b.md", "level": 2, "name": "b.md"},
+ ],
+ count=1000,
+ )
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=1000)
+
+ result = await fs.glob(
+ "docs/**/*.md",
+ uri="viking://resources",
+ node_limit=2,
+ ctx=_default_ctx(),
+ )
+
+ assert result == {
+ "matches": [
+ "viking://resources/docs/",
+ "viking://resources/docs/b.md",
+ ],
+ "count": 2,
+ }
+ assert len(vector_store.random_calls) == 1
+ call = vector_store.random_calls[0]
+ assert call["limit"] == 3
+ assert call["offset"] == 0
+ assert call["output_fields"] == ["uri", "level", "name"]
+ assert call["filter"] == PathScope("uri", "viking://resources/docs", depth=-1)
+ assert call["advance"] == {
+ "post_process_input_limit": 1000000,
+ "post_process_ops": [
+ {
+ "op": "path_glob",
+ "field": "uri",
+ "pattern": "/resources/docs/**/*.md",
+ "stop_after_matches": 3,
+ }
+ ],
+ }
+
+
+async def test_glob_remote_uses_limit_100000_when_node_limit_is_none(monkeypatch, fs):
+ vector_store = _RemoteGlobVectorStore([], count=1000)
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=1000)
+
+ await fs.glob("**/*.md", uri="viking://resources", node_limit=None, ctx=_default_ctx())
+
+ call = vector_store.random_calls[0]
+ assert call["limit"] == 100000
+ assert call["advance"]["post_process_ops"][0]["stop_after_matches"] == 100000
+
+
+async def test_glob_auto_uses_scoped_count_instead_of_collection_stats(monkeypatch, fs):
+ vector_store = _RemoteGlobVectorStore(
+ [{"uri": "viking://resources/docs/a.md", "level": 2, "name": "a.md"}],
+ count=999,
+ data_count=1000,
+ )
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=1000)
+
+ async def fake_glob_directory(path, pattern, **kwargs):
+ return {
+ "entries": [
+ {
+ "path": "/local/test_account/resources/docs/local.md",
+ "rel_path": "docs/local.md",
+ "name": "local.md",
+ "is_dir": False,
+ }
+ ],
+ "next_token": None,
+ }
+
+ monkeypatch.setattr(fs._async_agfs, "glob_directory", fake_glob_directory)
+
+ result = await fs.glob("docs/**/*.md", uri="viking://resources", ctx=_default_ctx())
+
+ assert result == {"matches": ["viking://resources/docs/local.md"], "count": 1}
+ assert len(vector_store.count_calls) == 1
+ count_filter = vector_store.count_calls[0]["filter"]
+ assert isinstance(count_filter, PathScope)
+ assert count_filter.path == "viking://resources/docs"
+ assert count_filter.depth == -1
+ assert vector_store.random_calls == []
+
+
+async def test_glob_auto_trusts_empty_remote_result(monkeypatch, fs):
+ vector_store = _RemoteGlobVectorStore([], count=1000)
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=1000)
+
+ async def fake_glob_directory(path, pattern, **kwargs):
+ raise AssertionError("remote empty glob results should not fall back to fs")
+
+ monkeypatch.setattr(fs._async_agfs, "glob_directory", fake_glob_directory)
+
+ result = await fs.glob("**/*.md", uri="viking://resources", ctx=_default_ctx())
+
+ assert result == {"matches": [], "count": 0}
+ assert len(vector_store.random_calls) == 1
+
+
+async def test_glob_auto_falls_back_when_remote_search_raises(monkeypatch, fs):
+ vector_store = _FailingRemoteGlobVectorStore([], count=1000)
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=1000)
+
+ async def fake_glob_directory(path, pattern, **kwargs):
+ return {
+ "entries": [
+ {
+ "path": "/local/test_account/resources/local.md",
+ "rel_path": "local.md",
+ "name": "local.md",
+ "is_dir": False,
+ }
+ ],
+ "next_token": None,
+ }
+
+ monkeypatch.setattr(fs._async_agfs, "glob_directory", fake_glob_directory)
+
+ result = await fs.glob("**/*.md", uri="viking://resources", ctx=_default_ctx())
+
+ assert result == {"matches": ["viking://resources/local.md"], "count": 1}
+ assert len(vector_store.random_calls) == 1
+
+
+async def test_glob_remote_increases_positive_node_limit_by_twenty_percent(monkeypatch, fs):
+ vector_store = _RemoteGlobVectorStore([], count=1000)
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=1000)
+
+ await fs.glob("**/*.md", uri="viking://resources", node_limit=5, ctx=_default_ctx())
+
+ call = vector_store.random_calls[0]
+ assert call["limit"] == 6
+ assert call["advance"]["post_process_ops"][0]["stop_after_matches"] == 6
+
+
+async def test_glob_remote_falls_back_when_node_limit_is_non_positive(monkeypatch, fs):
+ vector_store = _RemoteGlobVectorStore(
+ [{"uri": "viking://resources/remote.md", "level": 2, "name": "remote.md"}],
+ count=1000,
+ )
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=1000)
+
+ async def fake_glob_directory(path, pattern, **kwargs):
+ return {
+ "entries": [
+ {
+ "path": "/local/test_account/resources/local.md",
+ "rel_path": "local.md",
+ "name": "local.md",
+ "is_dir": False,
+ }
+ ],
+ "next_token": None,
+ }
+
+ monkeypatch.setattr(fs._async_agfs, "glob_directory", fake_glob_directory)
+
+ result = await fs.glob("**/*.md", uri="viking://resources", node_limit=0, ctx=_default_ctx())
+
+ assert result == {"matches": ["viking://resources/local.md"], "count": 1}
+ assert vector_store.random_calls == []
+
+
+async def test_glob_remote_uses_remote_fields_for_extra_fields(monkeypatch, fs):
+ vector_store = _RemoteGlobVectorStore(
+ [
+ {
+ "uri": "viking://resources/remote.md",
+ "level": 2,
+ "name": "remote.md",
+ "size": 10,
+ "mode": 0o644,
+ "modTime": "2026-08-30T00:00:00Z",
+ "updated_at": "2026-08-31T00:00:00Z",
+ }
+ ],
+ count=1000,
+ )
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=1000)
+
+ async def fake_glob_directory(path, pattern, **kwargs):
+ return {
+ "entries": [
+ {
+ "path": "/local/test_account/resources/local.md",
+ "rel_path": "local.md",
+ "name": "local.md",
+ "is_dir": False,
+ }
+ ],
+ "next_token": None,
+ }
+
+ monkeypatch.setattr(fs._async_agfs, "glob_directory", fake_glob_directory)
+
+ result = await fs.glob(
+ "**/*.md",
+ uri="viking://resources",
+ node_limit=2,
+ ctx=_default_ctx(),
+ extra_fields=["updated_at"],
+ )
+
+ assert result == {
+ "matches": [
+ {
+ "uri": "viking://resources/remote.md",
+ "name": "remote.md",
+ "isDir": False,
+ "size": 10,
+ "mode": 0o644,
+ "modTime": "2026-08-30T00:00:00Z",
+ "updated_at": "2026-08-31T00:00:00Z",
+ }
+ ],
+ "count": 1,
+ }
+ assert len(vector_store.random_calls) == 1
+ assert vector_store.random_calls[0]["output_fields"] == [
+ "uri",
+ "level",
+ "name",
+ "size",
+ "mode",
+ "modTime",
+ "updated_at",
+ ]
+
+
+async def test_glob_remote_entry_mode_fills_missing_stat_fields(monkeypatch, fs):
+ vector_store = _RemoteGlobVectorStore(
+ [
+ {
+ "uri": "viking://resources/remote.md",
+ "level": 2,
+ "id": "remote-vector-id",
+ "_score": 0.42,
+ "name": "remote.md",
+ }
+ ],
+ count=1000,
+ )
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=1000)
+
+ stat_calls = []
+
+ async def fake_stat(uri, ctx=None, skip_count=False):
+ stat_calls.append({"uri": uri, "skip_count": skip_count})
+ return {
+ "uri": uri,
+ "name": "from-stat.md",
+ "isDir": False,
+ "size": 7,
+ "mode": 0o644,
+ "modTime": "2026-08-29T00:00:00Z",
+ }
+
+ monkeypatch.setattr(fs, "stat", fake_stat)
+
+ result = await fs.glob(
+ "**/*.md",
+ uri="viking://resources",
+ node_limit=2,
+ ctx=_default_ctx(),
+ extra_fields=[],
+ )
+
+ assert result == {
+ "matches": [
+ {
+ "uri": "viking://resources/remote.md",
+ "id": "remote-vector-id",
+ "name": "remote.md",
+ "isDir": False,
+ "size": 7,
+ "mode": 0o644,
+ "modTime": "2026-08-29T00:00:00Z",
+ }
+ ],
+ "count": 1,
+ }
+ assert stat_calls == [{"uri": "viking://resources/remote.md", "skip_count": True}]
+
+
+async def test_glob_remote_filters_hidden_files_but_keeps_hidden_dir_children(monkeypatch, fs):
+ vector_store = _RemoteGlobVectorStore(
+ [
+ {"uri": "viking://resources/.hidden.md", "level": 2, "name": ".hidden.md"},
+ {
+ "uri": "viking://resources/.hidden_dir/nested.md",
+ "level": 2,
+ "name": "nested.md",
+ },
+ ],
+ count=1000,
+ )
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=1000)
+
+ result = await fs.glob("**/*.md", uri="viking://resources", node_limit=10, ctx=_default_ctx())
+
+ assert result == {
+ "matches": ["viking://resources/.hidden_dir/nested.md"],
+ "count": 1,
+ }
+
+
+async def test_glob_remote_leading_slash_stays_relative_to_request_uri(monkeypatch, fs):
+ vector_store = _RemoteGlobVectorStore(
+ [{"uri": "viking://resources/docs/a.md", "level": 2, "name": "a.md"}],
+ count=1000,
+ )
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=1000)
+
+ result = await fs.glob(
+ "/docs/**/*.md",
+ uri="viking://resources",
+ node_limit=2,
+ ctx=_default_ctx(),
+ )
+
+ assert result == {"matches": ["viking://resources/docs/a.md"], "count": 1}
+ call = vector_store.random_calls[0]
+ assert call["filter"] == PathScope("uri", "viking://resources/docs", depth=-1)
+ assert call["advance"]["post_process_ops"][0]["pattern"] == "/resources/docs/**/*.md"
+
+
+ "pattern",
+ ["./docs/**/*.md", "docs//**/*.md"],
+)
+async def test_glob_remote_normalizes_dot_and_empty_path_segments(fs, pattern):
+ vector_store = _RemoteGlobVectorStore(
+ [{"uri": "viking://resources/docs/a.md", "level": 2, "name": "a.md"}],
+ count=1000,
+ )
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=1000)
+
+ result = await fs.glob(
+ pattern,
+ uri="viking://resources",
+ node_limit=2,
+ ctx=_default_ctx(),
+ )
+
+ assert result == {"matches": ["viking://resources/docs/a.md"], "count": 1}
+ call = vector_store.random_calls[0]
+ assert call["filter"] == PathScope("uri", "viking://resources/docs", depth=-1)
+ assert call["advance"]["post_process_ops"][0]["pattern"] == ("/resources/docs/**/*.md")
+
+
+async def test_glob_auto_keeps_dot_only_pattern_validation_in_fs(monkeypatch, fs):
+ vector_store = _RemoteGlobVectorStore([], count=1000)
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=0)
+
+ async def fake_glob_directory(_path, pattern, **_kwargs):
+ raise InvalidArgumentError(f"invalid glob pattern: {pattern}")
+
+ monkeypatch.setattr(fs._async_agfs, "glob_directory", fake_glob_directory)
+
+ with pytest.raises(InvalidArgumentError, match="invalid glob pattern"):
+ await fs.glob("././", uri="viking://resources", ctx=_default_ctx())
+
+ assert vector_store.random_calls == []
+
+
[email protected]("pattern", ["/", "///"])
+async def test_glob_auto_keeps_separator_only_pattern_in_fs(monkeypatch, fs, pattern):
+ vector_store = _RemoteGlobVectorStore([], count=1000)
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=0)
+ fs_calls = []
+
+ async def fake_glob_directory(_path, received_pattern, **_kwargs):
+ fs_calls.append(received_pattern)
+ return {"entries": [], "next_token": None}
+
+ monkeypatch.setattr(fs._async_agfs, "glob_directory", fake_glob_directory)
+
+ assert await fs.glob(pattern, uri="viking://resources", ctx=_default_ctx()) == {
+ "matches": [],
+ "count": 0,
+ }
+ assert fs_calls == [pattern]
+ assert vector_store.random_calls == []
+
+
+async def test_glob_remote_pushes_tag_filter_before_limit(fs):
+ vector_store = _RemoteGlobVectorStore(
+ [{"uri": "viking://resources/b.md", "level": 2, "name": "b.md"}],
+ count=1000,
+ )
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=1000)
+ tag_filter = {
+ "op": "and",
+ "conds": [
+ {"op": "must", "field": "search_tags", "conds": ["team=search"]},
+ {"op": "must", "field": "search_tags", "conds": ["env=prod"]},
+ ],
+ }
+
+ result = await fs.glob(
+ "**/*.md",
+ uri="viking://resources",
+ node_limit=1,
+ ctx=_default_ctx(),
+ extra_fields=[],
+ tag_filter=tag_filter,
+ )
+
+ assert result["count"] == 1
+ call = vector_store.random_calls[0]
+ assert call["limit"] == 2
+ assert call["filter"] == And(
+ [
+ PathScope("uri", "viking://resources", depth=-1),
+ RawDSL(tag_filter),
+ ]
+ )
+
+
+async def test_glob_remote_tag_failure_falls_back_to_unbounded_fs(monkeypatch, fs):
+ vector_store = _FailingRemoteGlobVectorStore([], count=1000)
+ fs.vector_store = vector_store
+ fs.glob_config = SimpleNamespace(engine="auto", switch_to_remote_threshold=1000)
+ calls = []
+
+ async def fake_glob_directory(path, pattern, **kwargs):
+ calls.append({"path": path, "pattern": pattern, **kwargs})
+ return {
+ "entries": [
+ {
+ "path": "/local/test_account/resources/a.md",
+ "name": "a.md",
+ "is_dir": False,
+ },
+ {
+ "path": "/local/test_account/resources/b.md",
+ "name": "b.md",
+ "is_dir": False,
+ },
+ ],
+ "next_token": None,
+ }
+
+ monkeypatch.setattr(fs._async_agfs, "glob_directory", fake_glob_directory)
+
+ async def fake_stat(uri, **_kwargs):
+ return {
+ "uri": uri,
+ "name": uri.rsplit("/", 1)[-1],
+ "isDir": False,
+ }
+
+ monkeypatch.setattr(fs, "stat", fake_stat)
+
+ result = await fs.glob(
+ "**/*.md",
+ uri="viking://resources",
+ node_limit=1,
+ ctx=_default_ctx(),
+ extra_fields=[],
+ tag_filter={"op": "must", "field": "search_tags", "conds": ["team=search"]},
+ )
+
+ assert result["count"] == 2
+ assert calls[0]["page_size"] > 1
+
+
@pytest.mark.asyncio
async def test_glob_delegates_to_agfs_with_paging_and_visibility(monkeypatch, fs):
calls = []
@@ -156,6 +758,24 @@ async def test_glob_trusts_backend_glob_matches(monkeypatch, fs):
assert result == {"matches": ["viking://resources/a.md"], "count": 1}
[email protected]("pattern", ["file].md", "file[]].md"])
+async def test_glob_fs_preserves_globset_compatible_closing_brackets(monkeypatch, fs, pattern):
+ calls = []
+
+ async def fake_glob_directory(path, received_pattern, **kwargs):
+ calls.append({"path": path, "pattern": received_pattern, **kwargs})
+ return {"entries": [], "next_token": None}
+
+ monkeypatch.setattr(fs._async_agfs, "glob_directory", fake_glob_directory)
+
+ assert await fs.glob(pattern, uri="viking://resources", ctx=_default_ctx()) == {
+ "matches": [],
+ "count": 0,
+ }
+ assert calls[0]["pattern"] == pattern
+
+
@pytest.mark.asyncio
async def test_glob_rejects_empty_pattern(fs):
with pytest.raises(InvalidArgumentError):
diff --git a/tests/storage/test_viking_vector_scope_filter.py b/tests/storage/test_viking_vector_scope_filter.py
index a6fe570af..bed68f88f 100644
--- a/tests/storage/test_viking_vector_scope_filter.py
+++ b/tests/storage/test_viking_vector_scope_filter.py
@@ -6,8 +6,11 @@ import pytest
from openviking.server.identity import RequestContext, Role
from openviking.storage.acl import AclManager
from openviking.storage.collection_schemas import CollectionSchemas
-from openviking.storage.expr import And, Eq, In, Or, PathScope
-from openviking.storage.viking_vector_index_backend import VikingVectorIndexBackend
+from openviking.storage.expr import And, Eq, In, Or, PathScope, RawDSL
+from openviking.storage.viking_vector_index_backend import (
+ VikingVectorIndexBackend,
+ _SingleAccountBackend,
+)
from openviking_cli.session.user_id import UserIdentifier
from openviking_cli.utils.config.vectordb_config import VectorDBBackendConfig
@@ -45,6 +48,20 @@ def _tenant_filter(ctx: RequestContext):
return backend._tenant_filter(ctx)
+class _FailingAsyncAdapter:
+ async def call(self, method_name, **kwargs):
+ raise RuntimeError(f"{method_name} failed")
+
+
+class _RecordingAsyncAdapter:
+ def __init__(self):
+ self.calls = []
+
+ async def call(self, method_name, **kwargs):
+ self.calls.append((method_name, kwargs))
+ return []
+
+
def test_descendant_target_elides_only_visible_root_path_filter():
ctx = _ctx()
target = "viking://resources/wiki/physics"
@@ -317,6 +334,22 @@ def test_no_target_keeps_original_tenant_filter():
)
+def test_merge_filters_wraps_raw_dict_filter():
+ backend = object.__new__(VikingVectorIndexBackend)
+
+ result = backend._merge_filters(
+ {"op": "must", "field": "uri", "conds": ["viking://resources"]},
+ Eq("account_id", "acct"),
+ )
+
+ assert result == And(
+ [
+ RawDSL({"op": "must", "field": "uri", "conds": ["viking://resources"]}),
+ Eq("account_id", "acct"),
+ ]
+ )
+
+
def test_root_role_keeps_existing_target_only_behavior():
ctx = _ctx(role=Role.ROOT)
target = "viking://resources/wiki"
@@ -342,3 +375,36 @@ def test_actor_peer_target_retains_account_and_exact_target_scope():
Or([PathScope("uri", target, depth=-1)]),
]
)
+
+
+async def test_search_by_random_propagates_adapter_errors():
+ backend = object.__new__(_SingleAccountBackend)
+ backend._bound_account_id = None
+ backend._async_adapter = _FailingAsyncAdapter()
+
+ with pytest.raises(RuntimeError, match="search_by_random failed"):
+ await backend.search_by_random(filter=Eq("uri", "viking://resources/a.md"))
+
+
+async def test_search_by_random_reuses_account_filter_for_raw_dsl():
+ backend = object.__new__(_SingleAccountBackend)
+ backend._bound_account_id = "acct"
+ backend._async_adapter = _RecordingAsyncAdapter()
+ raw_filter = {"op": "must", "field": "uri", "conds": ["viking://resources"]}
+
+ await backend.search_by_random(filter=raw_filter)
+
+ assert backend._async_adapter.calls == [
+ (
+ "search_by_random",
+ {
+ "filter": And([Eq("account_id", "acct"), RawDSL(raw_filter)]),
+ "limit": 10,
+ "offset": 0,
+ "output_fields": None,
+ "advance": None,
+ },
+ )
+ ]
diff --git a/tests/storage/test_volcengine_collection_random.py b/tests/storage/test_volcengine_collection_random.py
new file mode 100644
index 000000000..05a774768
--- /dev/null
+++ b/tests/storage/test_volcengine_collection_random.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+import pytest
+
+from openviking.storage.errors import ConnectionError
+from openviking.storage.vectordb.collection.volcengine_collection import VolcengineCollection
+
+
+class _FakeResponse:
+ status_code = 429
+ text = '{"ResponseMetadata":{"Error":{"Code":"TooManyRequests","Message":"rate limited"}}}'
+
+ def json(self):
+ return {
+ "ResponseMetadata": {
+ "Error": {
+ "Code": "TooManyRequests",
+ "Message": "rate limited",
+ }
+ }
+ }
+
+
+class _FailingDataClient:
+ def do_req(self, *_args, **_kwargs):
+ return _FakeResponse()
+
+
+def test_volcengine_search_by_random_raises_on_non_200_response():
+ collection = object.__new__(VolcengineCollection)
+ collection.project_name = "default"
+ collection.collection_name = "context"
+ collection.data_client = _FailingDataClient()
+
+ with pytest.raises(ConnectionError, match="TooManyRequests"):
+ collection.search_by_random(index_name="idx")
diff --git a/tests/test_config_loader.py b/tests/test_config_loader.py
index c51e66f1e..9d45b1df6 100644
--- a/tests/test_config_loader.py
+++ b/tests/test_config_loader.py
@@ -136,6 +136,13 @@ def test_runtime_concurrency_uses_scope_specific_defaults():
assert config.reindex.file_vectorization_concurrency == 8
+def test_glob_uses_safe_defaults():
+ config = OpenVikingConfig.from_dict({})
+
+ assert config.glob.engine == "fs"
+ assert config.glob.switch_to_remote_threshold == 100
+
+
def test_runtime_concurrency_accepts_separate_values():
config = OpenVikingConfig.from_dict(
{