Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat: make queuefs backend configurable (#2018)

Jiahui Zhoufa0be9c95810
diff --git a/docs/en/guides/01-configuration.md b/docs/en/guides/01-configuration.md
index 48938e7f8..a4652b4e3 100644
--- a/docs/en/guides/01-configuration.md
+++ b/docs/en/guides/01-configuration.md
@@ -800,13 +800,79 @@ Storage configuration for context data, including file storage (RAGFS) and vecto
|-----------|------|-------------|---------|
| `backend` | str | `"local"`, `"s3"`, or `"memory"` | `"local"` |
| `timeout` | float | Request timeout in seconds | `10.0` |
-| `queue_db_path` | str (optional) | Override path of the queuefs sqlite database file. Defaults to `{storage.workspace}/_system/queue/queue.db` when not set. Useful when the workspace volume does not support sqlite (e.g. some network filesystems) | `null` |
+| `queuefs` | object | QueueFS configuration. Controls the namespace mode, backend, and runtime options for `/queue` | `{ "mode": "shared", "backend": "sqlite", "recover_stale_sec": 0, "busy_timeout_ms": 5000 }` |
+| `queue_db_path` | str (optional) | Legacy compatibility field for QueueFS sqlite DB path. Superseded by `storage.agfs.queuefs.db_path`. Defaults to `{storage.workspace}/_system/queue/queue.db` when not set. Useful when the workspace volume does not support sqlite (e.g. some network filesystems) | `null` |
| `s3` | object | S3 backend configuration (when backend is 's3') | - |
**Configuration Examples**
RAGFS uses Rust binding mode by default, directly accessing the file system through the Rust implementation.
+##### QueueFS Configuration
+
+| Parameter | Type | Description | Default |
+|-----------|------|-------------|---------|
+| `mode` | str | QueueFS namespace mode: `"shared"` uses `/queue`; `"worker"` isolates each worker under `/queue/worker-<index\|pid>` | `"shared"` |
+| `backend` | str | QueueFS backend: `"memory"`, `"sqlite"`, or `"sqlite3"` | `"sqlite"` |
+| `db_path` | str (optional) | SQLite database path for QueueFS when backend is `"sqlite"` or `"sqlite3"` | `null` |
+| `recover_stale_sec` | int | Recover `processing` queue messages older than this many seconds on startup. `0` means recover all stale processing messages | `0` |
+| `busy_timeout_ms` | int | SQLite busy timeout for QueueFS in milliseconds | `5000` |
+
+Notes:
+
+- QueueFS defaults to `sqlite` even if the main AGFS storage backend is `local`, `s3`, or `memory`.
+- `mode=shared` keeps the historical global queue namespace at `/queue`; `mode=worker` isolates each worker under `/queue/worker-<index|pid>`.
+- `db_path` is only used when QueueFS backend is `sqlite` or `sqlite3`.
+- If both `storage.agfs.queuefs.db_path` and legacy `storage.agfs.queue_db_path` are set, `storage.agfs.queuefs.db_path` wins.
+- If QueueFS backend is `memory`, any `db_path` or legacy `queue_db_path` is ignored.
+
+Examples:
+
+```json
+{
+ "storage": {
+ "workspace": "./data",
+ "agfs": {
+ "backend": "local",
+ "queuefs": {
+ "mode": "shared",
+ "backend": "sqlite",
+ "db_path": "./data/_system/queue/custom-queue.db"
+ }
+ }
+ }
+}
+```
+
+```json
+{
+ "storage": {
+ "workspace": "./data",
+ "agfs": {
+ "backend": "local",
+ "queuefs": {
+ "mode": "worker",
+ "backend": "memory"
+ }
+ }
+ }
+}
+```
+
+Legacy compatibility example:
+
+```json
+{
+ "storage": {
+ "workspace": "./data",
+ "agfs": {
+ "backend": "local",
+ "queue_db_path": "./data/_system/queue/queue.db"
+ }
+ }
+}
+```
+
##### S3 Backend Configuration
diff --git a/docs/zh/guides/01-configuration.md b/docs/zh/guides/01-configuration.md
index b77202ae8..28ea2f765 100644
--- a/docs/zh/guides/01-configuration.md
+++ b/docs/zh/guides/01-configuration.md
@@ -772,7 +772,8 @@ AST 提取支持:Python、JavaScript/TypeScript、Rust、Go、Java、C/C++。
|------|------|------|--------|
| `backend` | str | `"local"`、`"s3"` 或 `"memory"` | `"local"` |
| `timeout` | float | 请求超时时间(秒) | `10.0` |
-| `queue_db_path` | str(可选)| 覆盖 queuefs sqlite 数据库文件路径。未设置时默认为 `{storage.workspace}/_system/queue/queue.db`。适用于 workspace 卷不支持 sqlite 的场景(例如某些网络文件系统) | `null` |
+| `queuefs` | object | QueueFS 配置。控制 `/queue` 的命名空间模式、后端和运行时参数 | `{ "mode": "shared", "backend": "sqlite", "recover_stale_sec": 0, "busy_timeout_ms": 5000 }` |
+| `queue_db_path` | str(可选)| 旧版兼容字段,用于覆盖 QueueFS 的 sqlite 数据库文件路径。已被 `storage.agfs.queuefs.db_path` 取代。未设置时默认为 `{storage.workspace}/_system/queue/queue.db`。适用于 workspace 卷不支持 sqlite 的场景(例如某些网络文件系统) | `null` |
| `s3` | object | S3 backend configuration (when backend is 's3') | - |
@@ -780,6 +781,71 @@ AST 提取支持:Python、JavaScript/TypeScript、Rust、Go、Java、C/C++。
RAGFS 默认使用 Rust binding 模式,通过 Rust 实现直接访问文件系统。
+##### QueueFS 配置
+
+| 参数 | 类型 | 说明 | 默认值 |
+|------|------|------|--------|
+| `mode` | str | QueueFS 命名空间模式:`"shared"` 使用 `/queue`;`"worker"` 为每个 worker 隔离到 `/queue/worker-<index\|pid>` | `"shared"` |
+| `backend` | str | QueueFS 后端:`"memory"`、`"sqlite"` 或 `"sqlite3"` | `"sqlite"` |
+| `db_path` | str(可选) | 当 backend 为 `"sqlite"` 或 `"sqlite3"` 时使用的 QueueFS sqlite 数据库路径 | `null` |
+| `recover_stale_sec` | int | 启动时恢复超过该秒数的 `processing` 队列消息;`0` 表示恢复全部 stale processing 消息 | `0` |
+| `busy_timeout_ms` | int | QueueFS sqlite 的 busy timeout,单位毫秒 | `5000` |
+
+说明:
+
+- 即使主 AGFS 存储后端是 `local`、`s3` 或 `memory`,QueueFS 默认仍使用 `sqlite`。
+- `mode=shared` 会继续使用历史上的全局队列命名空间 `/queue`;`mode=worker` 会为每个 worker 隔离到 `/queue/worker-<index|pid>`。
+- `db_path` 仅在 QueueFS backend 为 `sqlite` 或 `sqlite3` 时生效。
+- 如果同时设置了 `storage.agfs.queuefs.db_path` 和旧字段 `storage.agfs.queue_db_path`,以前者为准。
+- 如果 QueueFS backend 为 `memory`,则 `db_path` 和旧字段 `queue_db_path` 都会被忽略。
+
+示例:
+
+```json
+{
+ "storage": {
+ "workspace": "./data",
+ "agfs": {
+ "backend": "local",
+ "queuefs": {
+ "mode": "shared",
+ "backend": "sqlite",
+ "db_path": "./data/_system/queue/custom-queue.db"
+ }
+ }
+ }
+}
+```
+
+```json
+{
+ "storage": {
+ "workspace": "./data",
+ "agfs": {
+ "backend": "local",
+ "queuefs": {
+ "mode": "worker",
+ "backend": "memory"
+ }
+ }
+ }
+}
+```
+
+旧字段兼容示例:
+
+```json
+{
+ "storage": {
+ "workspace": "./data",
+ "agfs": {
+ "backend": "local",
+ "queue_db_path": "./data/_system/queue/queue.db"
+ }
+ }
+}
+```
+
##### S3 后端配置
diff --git a/openviking/service/core.py b/openviking/service/core.py
index 40934e29e..a39f1f42e 100644
--- a/openviking/service/core.py
+++ b/openviking/service/core.py
@@ -29,6 +29,7 @@ from openviking.storage.index_consistency import check_index_consistency
from openviking.storage.queuefs.queue_manager import QueueManager, init_queue_manager
from openviking.storage.transaction import LockManager, init_lock_manager
from openviking.storage.viking_fs import VikingFS, init_viking_fs
+from openviking.utils.agfs_utils import resolve_queuefs_mount_point
from openviking.utils.resource_processor import ResourceProcessor
from openviking.utils.skill_processor import SkillProcessor
from openviking_cli.exceptions import NotInitializedError
@@ -121,9 +122,11 @@ class OpenVikingService:
# Initialize QueueManager with agfs_client
if self._agfs_client:
+ queue_mount_point = resolve_queuefs_mount_point()
self._queue_manager = init_queue_manager(
agfs=self._agfs_client,
timeout=config.agfs.timeout,
+ mount_point=queue_mount_point,
max_concurrent_embedding=max_concurrent_embedding,
max_concurrent_semantic=max_concurrent_semantic,
)
diff --git a/openviking/session/compressor.py b/openviking/session/compressor.py
index 0c04d7825..0938155b8 100644
--- a/openviking/session/compressor.py
+++ b/openviking/session/compressor.py
@@ -137,9 +137,14 @@ class SessionCompressor:
changes=changes_dict,
telemetry_id=telemetry.telemetry_id,
)
- await semantic_queue.enqueue(msg)
if msg.telemetry_id:
request_wait_tracker.register_semantic_root(msg.telemetry_id, msg.id)
+ try:
+ await semantic_queue.enqueue(msg)
+ except Exception as e:
+ if msg.telemetry_id:
+ request_wait_tracker.mark_semantic_failed(msg.telemetry_id, msg.id, str(e))
+ raise
logger.info(
f"Enqueued semantic generation for {parent_uri} with changes: "
f"added={len(changes['added'])}, modified={len(changes['modified'])}, "
@@ -191,19 +196,27 @@ class SessionCompressor:
chunk_memory.set_vectorize(Vectorize(text=chunk))
chunk_msg = EmbeddingMsgConverter.from_context(chunk_memory)
if chunk_msg:
- enqueued = await self.vikingdb.enqueue_embedding_msg(chunk_msg)
- if enqueued and chunk_msg.telemetry_id:
+ if chunk_msg.telemetry_id:
request_wait_tracker.register_embedding_root(
chunk_msg.telemetry_id, chunk_msg.id
)
+ enqueued = await self.vikingdb.enqueue_embedding_msg(chunk_msg)
+ if not enqueued and chunk_msg.telemetry_id:
+ request_wait_tracker.mark_embedding_failed(
+ chunk_msg.telemetry_id, chunk_msg.id, "embedding enqueue returned false"
+ )
# Always enqueue the base record (uses abstract as vector text)
embedding_msg = EmbeddingMsgConverter.from_context(memory)
- enqueued = await self.vikingdb.enqueue_embedding_msg(embedding_msg)
- if enqueued and embedding_msg.telemetry_id:
+ if embedding_msg and embedding_msg.telemetry_id:
request_wait_tracker.register_embedding_root(
embedding_msg.telemetry_id, embedding_msg.id
)
+ enqueued = await self.vikingdb.enqueue_embedding_msg(embedding_msg)
+ if embedding_msg and not enqueued and embedding_msg.telemetry_id:
+ request_wait_tracker.mark_embedding_failed(
+ embedding_msg.telemetry_id, embedding_msg.id, "embedding enqueue returned false"
+ )
logger.info(f"Enqueued memory for vectorization: {memory.uri}")
self._record_semantic_change(memory.uri, change_type, parent_uri=memory.parent_uri)
diff --git a/openviking/session/memory/memory_updater.py b/openviking/session/memory/memory_updater.py
index e9b427292..6f2004189 100644
--- a/openviking/session/memory/memory_updater.py
+++ b/openviking/session/memory/memory_updater.py
@@ -535,11 +535,17 @@ class MemoryUpdater:
# Convert to embedding msg and enqueue
embedding_msg = EmbeddingMsgConverter.from_context(memory_context)
if embedding_msg:
- enqueued = await self._vikingdb.enqueue_embedding_msg(embedding_msg)
- if enqueued and embedding_msg.telemetry_id:
+ if embedding_msg.telemetry_id:
request_wait_tracker.register_embedding_root(
embedding_msg.telemetry_id, embedding_msg.id
)
+ enqueued = await self._vikingdb.enqueue_embedding_msg(embedding_msg)
+ if not enqueued and embedding_msg.telemetry_id:
+ request_wait_tracker.mark_embedding_failed(
+ embedding_msg.telemetry_id,
+ embedding_msg.id,
+ "embedding enqueue returned false",
+ )
logger.debug(f"Enqueued memory for vectorization: {uri}")
except Exception as e:
diff --git a/openviking/session/session.py b/openviking/session/session.py
index 28c728439..bbe69f694 100644
--- a/openviking/session/session.py
+++ b/openviking/session/session.py
@@ -987,6 +987,25 @@ class Session:
user_id=self.ctx.user.user_id,
)
logger.info(f"Session {self.session_id} memory extraction completed")
+ except asyncio.CancelledError as e:
+ if redo_enabled and redo_task_id:
+ redo_log.mark_done(redo_task_id)
+ try:
+ await self._write_failed_marker(
+ archive_uri,
+ stage="memory_extraction",
+ error=f"cancelled: {e}",
+ )
+ except Exception:
+ logger.debug("Failed to write cancelled marker for session %s", self.session_id)
+ tracker.fail(
+ task_id,
+ f"cancelled: {e}",
+ account_id=self.ctx.account_id,
+ user_id=self.ctx.user.user_id,
+ )
+ logger.warning("Memory extraction cancelled for session %s", self.session_id)
+ raise
except Exception as e:
if redo_enabled and redo_task_id:
redo_log.mark_done(redo_task_id)
diff --git a/openviking/storage/content_write.py b/openviking/storage/content_write.py
index 76a612ece..ac9c878c5 100644
--- a/openviking/storage/content_write.py
+++ b/openviking/storage/content_write.py
@@ -396,9 +396,14 @@ class ContentWriteCoordinator:
lifecycle_lock_handle_id=lifecycle_lock_handle_id,
changes={change_type: [changed_uri]},
)
- await semantic_queue.enqueue(msg)
if msg.telemetry_id:
get_request_wait_tracker().register_semantic_root(msg.telemetry_id, msg.id)
+ try:
+ await semantic_queue.enqueue(msg)
+ except Exception as e:
+ if msg.telemetry_id:
+ get_request_wait_tracker().mark_semantic_failed(msg.telemetry_id, msg.id, str(e))
+ raise
async def _enqueue_memory_refresh(
self,
@@ -423,9 +428,14 @@ class ContentWriteCoordinator:
lifecycle_lock_handle_id=lifecycle_lock_handle_id,
changes={"modified": [modified_uri]},
)
- await semantic_queue.enqueue(msg)
if msg.telemetry_id:
get_request_wait_tracker().register_semantic_root(msg.telemetry_id, msg.id)
+ try:
+ await semantic_queue.enqueue(msg)
+ except Exception as e:
+ if msg.telemetry_id:
+ get_request_wait_tracker().mark_semantic_failed(msg.telemetry_id, msg.id, str(e))
+ raise
async def _wait_for_queues(self, *, timeout: Optional[float]) -> Dict[str, Any]:
queue_manager = get_queue_manager()
diff --git a/openviking/storage/queuefs/queue_manager.py b/openviking/storage/queuefs/queue_manager.py
index a9d762804..222ec7c2f 100644
--- a/openviking/storage/queuefs/queue_manager.py
+++ b/openviking/storage/queuefs/queue_manager.py
@@ -104,7 +104,7 @@ class QueueManager:
for queue in list(self._queues.values()):
self._start_queue_worker(queue)
- logger.info("[QueueManager] Started")
+ logger.info(f"[QueueManager] mount_point={self.mount_point} Started")
def setup_standard_queues(self, vector_store: Any, start: bool = True) -> None:
"""
diff --git a/openviking/utils/agfs_utils.py b/openviking/utils/agfs_utils.py
index 5788abf38..dfd7ce87a 100644
--- a/openviking/utils/agfs_utils.py
+++ b/openviking/utils/agfs_utils.py
@@ -4,6 +4,7 @@
RAGFS Client utilities for creating and configuring RAGFS clients.
"""
+import multiprocessing
import os
from pathlib import Path
from typing import Any, Dict
@@ -13,15 +14,70 @@ from openviking_cli.utils.logger import get_logger
logger = get_logger(__name__)
-def _generate_plugin_config(agfs_config: Any, data_path: Path) -> Dict[str, Any]:
- """Dynamically generate RAGFS plugin configuration based on backend type."""
+def resolve_queuefs_mount_point(config: Any = None) -> str:
+ """Resolve QueueFS mount point for the current process.
+
+ `shared` keeps the historical global queue root (`/queue`).
+ `worker` isolates each worker under `/queue/worker-<index|pid>`.
+ """
+ mode = None
+ if config is not None:
+ storage = getattr(config, "storage", None)
+ if storage is None and hasattr(config, "agfs"):
+ storage = config
+ agfs = getattr(storage, "agfs", None) if storage is not None else None
+ queuefs = getattr(agfs, "queuefs", None) if agfs is not None else None
+ mode = getattr(queuefs, "mode", None)
+
+ if not mode:
+ try:
+ from openviking_cli.utils.config import get_openviking_config
+
+ mode = get_openviking_config().storage.agfs.queuefs.mode
+ except Exception:
+ mode = "shared"
+
+ if mode == "worker":
+ identity = getattr(multiprocessing.current_process(), "_identity", ())
+ if identity:
+ worker_id = str(identity[0] - 1)
+ else:
+ worker_id = str(os.getpid())
+ return f"/queue/worker-{worker_id}"
+ return "/queue"
+
+
+def _build_queuefs_plugin_config(agfs_config: Any, data_path: Path) -> Dict[str, Any]:
+ """Build QueueFS plugin configuration from AGFS config with legacy compatibility."""
default_queue_db_path = data_path / "_system" / "queue" / "queue.db"
- configured_queue_db_path = getattr(agfs_config, "queue_db_path", None)
- if configured_queue_db_path:
- queue_db_path = str(Path(configured_queue_db_path).expanduser().resolve())
- else:
- queue_db_path = str(default_queue_db_path)
+ queuefs_config = getattr(agfs_config, "queuefs", None)
+ backend = getattr(queuefs_config, "backend", "sqlite") if queuefs_config else "sqlite"
+ plugin_config: Dict[str, Any] = {
+ "backend": backend,
+ "recover_stale_sec": getattr(queuefs_config, "recover_stale_sec", 0),
+ "busy_timeout_ms": getattr(queuefs_config, "busy_timeout_ms", 5000),
+ }
+
+ if backend in {"sqlite", "sqlite3"}:
+ configured_queue_db_path = None
+ if queuefs_config is not None:
+ configured_queue_db_path = getattr(queuefs_config, "db_path", None)
+ if not configured_queue_db_path:
+ configured_queue_db_path = getattr(agfs_config, "queue_db_path", None)
+
+ if configured_queue_db_path:
+ queue_db_path = str(Path(configured_queue_db_path).expanduser().resolve())
+ else:
+ queue_db_path = str(default_queue_db_path)
+
+ plugin_config["db_path"] = queue_db_path
+
+ return plugin_config
+
+
+def _generate_plugin_config(agfs_config: Any, data_path: Path) -> Dict[str, Any]:
+ """Dynamically generate RAGFS plugin configuration based on backend type."""
config = {
"serverinfofs": {
"enabled": True,
@@ -33,10 +89,7 @@ def _generate_plugin_config(agfs_config: Any, data_path: Path) -> Dict[str, Any]
"queuefs": {
"enabled": True,
"path": "/queue",
- "config": {
- "backend": "sqlite",
- "db_path": queue_db_path,
- },
+ "config": _build_queuefs_plugin_config(agfs_config, data_path),
},
}
@@ -136,7 +189,6 @@ def mount_agfs_backend(agfs: Any, agfs_config: Any) -> None:
vikingfs_path = data_path / "viking"
vikingfs_path.mkdir(parents=True, exist_ok=True)
- (data_path / "_system" / "queue").mkdir(parents=True, exist_ok=True)
# 1. Mount standard plugins
config = _generate_plugin_config(agfs_config, data_path)
diff --git a/openviking/utils/skill_processor.py b/openviking/utils/skill_processor.py
index ad9cb5035..0d1f8d81a 100644
--- a/openviking/utils/skill_processor.py
+++ b/openviking/utils/skill_processor.py
@@ -345,8 +345,14 @@ class SkillProcessor:
context.set_vectorize(Vectorize(text=context.abstract))
embedding_msg = EmbeddingMsgConverter.from_context(context)
if embedding_msg:
- enqueued = await self.vikingdb.enqueue_embedding_msg(embedding_msg)
- if enqueued and embedding_msg.telemetry_id:
+ if embedding_msg.telemetry_id:
get_request_wait_tracker().register_embedding_root(
embedding_msg.telemetry_id, embedding_msg.id
)
+ enqueued = await self.vikingdb.enqueue_embedding_msg(embedding_msg)
+ if not enqueued and embedding_msg.telemetry_id:
+ get_request_wait_tracker().mark_embedding_failed(
+ embedding_msg.telemetry_id,
+ embedding_msg.id,
+ "embedding enqueue returned false",
+ )
diff --git a/openviking/utils/summarizer.py b/openviking/utils/summarizer.py
index 4be0acb71..d3fb40a59 100644
--- a/openviking/utils/summarizer.py
+++ b/openviking/utils/summarizer.py
@@ -107,9 +107,16 @@ class Summarizer:
lifecycle_lock_handle_id=lifecycle_lock_handle_id,
is_code_repo=kwargs.get("is_code_repo", False),
)
- await semantic_queue.enqueue(msg)
if msg.telemetry_id:
get_request_wait_tracker().register_semantic_root(msg.telemetry_id, msg.id)
+ try:
+ await semantic_queue.enqueue(msg)
+ except Exception as e:
+ if msg.telemetry_id:
+ get_request_wait_tracker().mark_semantic_failed(
+ msg.telemetry_id, msg.id, str(e)
+ )
+ raise
enqueued_count += 1
logger.info(
f"Enqueued semantic generation for: {target_uri} (skip_vectorization={skip_vectorization})"
diff --git a/openviking_cli/utils/config/agfs_config.py b/openviking_cli/utils/config/agfs_config.py
index c6e026e5e..39617addf 100644
--- a/openviking_cli/utils/config/agfs_config.py
+++ b/openviking_cli/utils/config/agfs_config.py
@@ -99,6 +99,52 @@ class S3Config(BaseModel):
return self
+class QueueFSConfig(BaseModel):
+ """Configuration for QueueFS backend."""
+
+ mode: str = Field(
+ default="shared",
+ description="QueueFS namespace mode: 'shared' | 'worker'",
+ )
+
+ backend: str = Field(
+ default="sqlite",
+ description="QueueFS backend: 'memory' | 'sqlite' | 'sqlite3'",
+ )
+
+ db_path: Optional[str] = Field(
+ default=None,
+ description="SQLite database path for QueueFS when backend is 'sqlite' or 'sqlite3'.",
+ )
+
+ recover_stale_sec: int = Field(
+ default=0,
+ description="Recover processing messages older than this many seconds on startup (0 = recover all).",
+ )
+
+ busy_timeout_ms: int = Field(
+ default=5000,
+ description="SQLite busy timeout for QueueFS in milliseconds.",
+ )
+
+ model_config = {"extra": "forbid"}
+
+ @model_validator(mode="after")
+ def validate_config(self):
+ valid_modes = {"shared", "worker"}
+ if self.mode not in valid_modes:
+ raise ValueError("queuefs mode must be one of: 'shared', 'worker'")
+
+ valid_backends = {"memory", "sqlite", "sqlite3"}
+ if self.backend not in valid_backends:
+ raise ValueError("queuefs backend must be one of: 'memory', 'sqlite', 'sqlite3'")
+ if self.recover_stale_sec < 0:
+ raise ValueError("queuefs recover_stale_sec must be >= 0")
+ if self.busy_timeout_ms < 0:
+ raise ValueError("queuefs busy_timeout_ms must be >= 0")
+ return self
+
+
class AGFSConfig(BaseModel):
"""Configuration for RAGFS (Rust-based AGFS)."""
@@ -150,6 +196,11 @@ class AGFSConfig(BaseModel):
"Useful when the workspace volume does not support sqlite (e.g. some network filesystems).",
)
+ queuefs: QueueFSConfig = Field(
+ default_factory=QueueFSConfig,
+ description="QueueFS configuration.",
+ )
+
retry_times: Any = Field(
default=None,
exclude=True,
@@ -207,4 +258,17 @@ class AGFSConfig(BaseModel):
# Validate S3 configuration
self.s3.validate_config()
+ if self.queue_db_path is not None and self.queuefs.db_path is None:
+ logger.warning(
+ "AGFSConfig: 'storage.agfs.queue_db_path' is deprecated; "
+ "prefer 'storage.agfs.queuefs.db_path'."
+ )
+
+ if self.queuefs.backend == "memory":
+ if self.queuefs.db_path is not None or self.queue_db_path is not None:
+ logger.warning(
+ "AGFSConfig: QueueFS backend is 'memory'; "
+ "db_path/queue_db_path will be ignored."
+ )
+
return self
diff --git a/tests/misc/test_config_validation.py b/tests/misc/test_config_validation.py
index 1acf2432a..8ea30c318 100644
--- a/tests/misc/test_config_validation.py
+++ b/tests/misc/test_config_validation.py
@@ -6,7 +6,9 @@
import sys
from pathlib import Path
-from openviking.utils.agfs_utils import _generate_plugin_config
+import pytest
+
+from openviking.utils.agfs_utils import _generate_plugin_config, mount_agfs_backend
from openviking_cli.utils.config.agfs_config import AGFSConfig, S3Config
from openviking_cli.utils.config.embedding_config import EmbeddingConfig, EmbeddingModelConfig
from openviking_cli.utils.config.vectordb_config import VectorDBBackendConfig, VolcengineConfig
@@ -95,6 +97,190 @@ def test_agfs_s3_normalize_encoding_chars_is_forwarded_to_ragfs_plugin_config():
print(f" Fail: {e}")
+def test_agfs_queuefs_defaults_to_sqlite_backend():
+ config = AGFSConfig(path="/tmp/ov-test", backend="local")
+
+ assert config.queuefs.mode == "shared"
+ assert config.queuefs.backend == "sqlite"
+ assert config.queuefs.recover_stale_sec == 0
+ assert config.queuefs.busy_timeout_ms == 5000
+
+
+def test_agfs_queuefs_accepts_memory_backend():
+ config = AGFSConfig(
+ path="/tmp/ov-test",
+ backend="local",
+ queuefs={"mode": "worker", "backend": "memory"},
+ )
+
+ assert config.queuefs.mode == "worker"
+ assert config.queuefs.backend == "memory"
+
+
+def test_agfs_queuefs_rejects_legacy_process_mode():
+ with pytest.raises(ValueError, match="queuefs mode"):
+ AGFSConfig(
+ path="/tmp/ov-test",
+ backend="local",
+ queuefs={"mode": "process"},
+ )
+
+
+def test_agfs_queuefs_rejects_invalid_backend():
+ with pytest.raises(ValueError, match="queuefs"):
+ AGFSConfig(
+ path="/tmp/ov-test",
+ backend="local",
+ queuefs={"backend": "bogus"},
+ )
+
+
+def test_agfs_queuefs_rejects_negative_timeouts():
+ with pytest.raises(ValueError, match="busy_timeout_ms"):
+ AGFSConfig(
+ path="/tmp/ov-test",
+ backend="local",
+ queuefs={"busy_timeout_ms": -1},
+ )
+
+ with pytest.raises(ValueError, match="recover_stale_sec"):
+ AGFSConfig(
+ path="/tmp/ov-test",
+ backend="local",
+ queuefs={"recover_stale_sec": -1},
+ )
+
+
+def test_generate_plugin_config_uses_queuefs_memory_backend_without_db_path():
+ config = AGFSConfig(
+ path="/tmp/ov-test",
+ backend="local",
+ queuefs={"backend": "memory"},
+ )
+
+ plugins = _generate_plugin_config(config, Path("/tmp/ov-test"))
+
+ assert plugins["queuefs"]["config"]["backend"] == "memory"
+ assert "db_path" not in plugins["queuefs"]["config"]
+
+
+def test_generate_plugin_config_uses_new_queuefs_db_path_over_legacy_field():
+ config = AGFSConfig(
+ path="/tmp/ov-test",
+ backend="local",
+ queue_db_path="/tmp/legacy-queue.db",
+ queuefs={"backend": "sqlite", "db_path": "/tmp/new-queue.db"},
+ )
+
+ plugins = _generate_plugin_config(config, Path("/tmp/ov-test"))
+
+ assert plugins["queuefs"]["config"]["backend"] == "sqlite"
+ assert plugins["queuefs"]["config"]["db_path"] == str(Path("/tmp/new-queue.db").resolve())
+
+
+def test_generate_plugin_config_supports_legacy_queue_db_path():
+ config = AGFSConfig(
+ path="/tmp/ov-test",
+ backend="local",
+ queue_db_path="/tmp/legacy-queue.db",
+ )
+
+ plugins = _generate_plugin_config(config, Path("/tmp/ov-test"))
+
+ assert plugins["queuefs"]["config"]["backend"] == "sqlite"
+ assert plugins["queuefs"]["config"]["db_path"] == str(Path("/tmp/legacy-queue.db").resolve())
+
+
+def test_generate_plugin_config_uses_workspace_default_queue_db_path():
+ config = AGFSConfig(
+ path="/tmp/ov-test",
+ backend="local",
+ )
+
+ plugins = _generate_plugin_config(config, Path("/tmp/ov-test"))
+
+ assert plugins["queuefs"]["config"]["backend"] == "sqlite"
+ assert plugins["queuefs"]["config"]["db_path"] == "/tmp/ov-test/_system/queue/queue.db"
+
+
+def test_generate_plugin_config_forwards_queuefs_runtime_options():
+ config = AGFSConfig(
+ path="/tmp/ov-test",
+ backend="local",
+ queuefs={
+ "backend": "sqlite3",
+ "recover_stale_sec": 17,
+ "busy_timeout_ms": 1234,
+ },
+ )
+
+ plugins = _generate_plugin_config(config, Path("/tmp/ov-test"))
+
+ assert plugins["queuefs"]["config"]["backend"] == "sqlite3"
+ assert plugins["queuefs"]["config"]["recover_stale_sec"] == 17
+ assert plugins["queuefs"]["config"]["busy_timeout_ms"] == 1234
+
+
+def test_generate_plugin_config_ignores_db_paths_in_memory_mode():
+ config = AGFSConfig(
+ path="/tmp/ov-test",
+ backend="local",
+ queue_db_path="/tmp/legacy-queue.db",
+ queuefs={"backend": "memory", "db_path": "/tmp/new-queue.db"},
+ )
+
+ plugins = _generate_plugin_config(config, Path("/tmp/ov-test"))
+
+ assert plugins["queuefs"]["config"]["backend"] == "memory"
+ assert "db_path" not in plugins["queuefs"]["config"]
+
+
+class _FakeMountClient:
+ def __init__(self):
+ self.mount_calls = []
+
+ def mount(self, plugin_name, mount_path, config):
+ self.mount_calls.append((plugin_name, mount_path, config))
+
+ def unmount(self, _mount_path):
+ return None
+
+
+def test_mount_agfs_backend_skips_queue_sqlite_dirs_for_memory_backend(tmp_path):
+ config = AGFSConfig(
+ path=str(tmp_path),
+ backend="local",
+ queuefs={"backend": "memory"},
+ )
+ client = _FakeMountClient()
+
+ mount_agfs_backend(client, config)
+
+ assert (tmp_path / "viking").exists()
+ assert not (tmp_path / "_system" / "queue").exists()
+ queuefs_mount = next(call for call in client.mount_calls if call[0] == "queuefs")
+ assert queuefs_mount[2]["backend"] == "memory"
+ assert "db_path" not in queuefs_mount[2]
+
+
+def test_mount_agfs_backend_creates_queue_sqlite_dirs_for_sqlite_backend(tmp_path):
+ queue_db_path = tmp_path / "custom-queue" / "queue.db"
+ config = AGFSConfig(
+ path=str(tmp_path),
+ backend="local",
+ queuefs={"backend": "sqlite", "db_path": str(queue_db_path)},
+ )
+ client = _FakeMountClient()
+
+ mount_agfs_backend(client, config)
+
+ assert (tmp_path / "viking").exists()
+ assert queue_db_path.parent.exists()
+ queuefs_mount = next(call for call in client.mount_calls if call[0] == "queuefs")
+ assert queuefs_mount[2]["backend"] == "sqlite"
+ assert queuefs_mount[2]["db_path"] == str(queue_db_path.resolve())
+
+
def test_vectordb_validation():
"""Test VectorDB config validation"""
print("\n" + "=" * 60)
diff --git a/tests/server/test_bootstrap.py b/tests/server/test_bootstrap.py
new file mode 100644
index 000000000..04918c2d9
--- /dev/null
+++ b/tests/server/test_bootstrap.py
@@ -0,0 +1,147 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+from __future__ import annotations
+
+import os
+from types import SimpleNamespace
+
+import openviking.server.bootstrap as bootstrap
+from openviking.server.config import ServerConfig
+from openviking.utils.agfs_utils import resolve_queuefs_mount_point
+from openviking_cli.utils.config.storage_config import StorageConfig
+
+
+def test_main_keeps_config_host_when_cli_host_is_omitted(monkeypatch):
+ config = ServerConfig(host="127.0.0.1", port=1933)
+ captured: dict[str, object] = {}
+
+ monkeypatch.setattr(
+ bootstrap,
+ "load_server_config",
+ lambda config_path: config,
+ )
+ monkeypatch.setattr(
+ bootstrap,
+ "create_app",
+ lambda config: "app",
+ )
+ monkeypatch.setattr(
+ bootstrap,
+ "configure_uvicorn_logging",
+ lambda: None,
+ )
+ monkeypatch.setattr(
+ bootstrap,
+ "OpenVikingConfigSingleton",
+ SimpleNamespace(initialize=lambda config_path: None),
+ raising=False,
+ )
+ monkeypatch.setattr(
+ bootstrap.argparse.ArgumentParser,
+ "parse_args",
+ lambda self: SimpleNamespace(
+ host=None,
+ port=None,
+ config=None,
+ workers=None,
+ bot=False,
+ with_bot=False,
+ bot_url="http://localhost:18790",
+ enable_bot_logging=None,
+ bot_log_dir="/tmp/bot-logs",
+ ),
+ )
+ monkeypatch.setattr(
+ bootstrap.uvicorn,
+ "run",
+ lambda app, host, port, log_config=None, **kwargs: captured.update(
+ {"app": app, "host": host, "port": port, "log_config": log_config, **kwargs}
+ ),
+ )
+
+ bootstrap.main()
+
+ assert captured["host"] == "127.0.0.1"
+ assert captured["port"] == 1933
+
+
+def test_main_coerces_cli_host_all_to_none(monkeypatch):
+ config = ServerConfig(host="127.0.0.1", port=1933)
+ captured: dict[str, object] = {}
+
+ monkeypatch.setattr(
+ bootstrap,
+ "load_server_config",
+ lambda config_path: config,
+ )
+ monkeypatch.setattr(
+ bootstrap,
+ "create_app",
+ lambda config: "app",
+ )
+ monkeypatch.setattr(
+ bootstrap,
+ "configure_uvicorn_logging",
+ lambda: None,
+ )
+ monkeypatch.setattr(
+ bootstrap,
+ "OpenVikingConfigSingleton",
+ SimpleNamespace(initialize=lambda config_path: None),
+ raising=False,
+ )
+ monkeypatch.setattr(
+ bootstrap.argparse.ArgumentParser,
+ "parse_args",
+ lambda self: SimpleNamespace(
+ host="all",
+ port=None,
+ config=None,
+ workers=None,
+ bot=False,
+ with_bot=False,
+ bot_url="http://localhost:18790",
+ enable_bot_logging=None,
+ bot_log_dir="/tmp/bot-logs",
+ ),
+ )
+ monkeypatch.setattr(
+ bootstrap.uvicorn,
+ "run",
+ lambda app, host, port, log_config=None, **kwargs: captured.update(
+ {"app": app, "host": host, "port": port, "log_config": log_config, **kwargs}
+ ),
+ )
+
+ bootstrap.main()
+
+ assert captured["host"] is None
+ assert captured["port"] == 1933
+
+
+def test_resolve_queuefs_mount_point_defaults_to_shared():
+ config = StorageConfig()
+
+ assert resolve_queuefs_mount_point(config) == "/queue"
+
+
+def test_resolve_queuefs_mount_point_worker_mode_uses_process_index(monkeypatch):
+ monkeypatch.setattr(
+ "openviking.utils.agfs_utils.multiprocessing.current_process",
+ lambda: SimpleNamespace(_identity=(3,)),
+ )
+ config = StorageConfig(agfs={"queuefs": {"mode": "worker"}})
+
+ assert resolve_queuefs_mount_point(config) == "/queue/worker-2"
+
+
+def test_resolve_queuefs_mount_point_worker_mode_falls_back_to_pid(monkeypatch):
+ monkeypatch.setattr(
+ "openviking.utils.agfs_utils.multiprocessing.current_process",
+ lambda: SimpleNamespace(_identity=()),
+ )
+ monkeypatch.setattr(os, "getpid", lambda: 43210)
+ config = StorageConfig(agfs={"queuefs": {"mode": "worker"}})
+
+ assert resolve_queuefs_mount_point(config) == "/queue/worker-43210"
diff --git a/tests/test_server_config_loader.py b/tests/test_server_config_loader.py
index abc14e1bd..5ffb145c5 100644
--- a/tests/test_server_config_loader.py
+++ b/tests/test_server_config_loader.py
@@ -60,6 +60,7 @@ def test_load_server_config_preserves_supported_fields(tmp_path):
"bot_api_url": "http://localhost:19999",
"observability": {"metrics": {"exporters": {"prometheus": {"enabled": True}}}},
},
+ "storage": {"agfs": {"queuefs": {"mode": "worker"}}},
"encryption": {"enabled": True},
}
)
@@ -77,6 +78,14 @@ def test_load_server_config_preserves_supported_fields(tmp_path):
assert config.encryption_enabled is True
+def test_load_server_config_rejects_legacy_queuefs_scope(tmp_path):
+ config_path = tmp_path / "ov.conf"
+ config_path.write_text(json.dumps({"server": {"queuefs_scope": "process"}}))
+
+ with pytest.raises(ValueError, match=r"server\.queuefs_scope"):
+ load_server_config(str(config_path))
+
+
def test_load_bot_gateway_token_reads_token_from_bot_gateway_section(tmp_path):
config_path = tmp_path / "ov.conf"
config_path.write_text(json.dumps({"bot": {"gateway": {"token": "gateway-token"}}}))