Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat(resources): ingest whole sites via sitemap / RSS / Atom (#2858)!

t0saki2846bb6e76bd
diff --git a/crates/ov_cli/src/help_ui.rs b/crates/ov_cli/src/help_ui.rs
index 73c8363d0..a3738ee08 100644
--- a/crates/ov_cli/src/help_ui.rs
+++ b/crates/ov_cli/src/help_ui.rs
@@ -125,7 +125,7 @@ const HELP_SECTIONS: &[HelpSection] = &[
const COMMAND_HELP_SPECS: &[CommandHelpSpec] = &[
CommandHelpSpec {
path: &["add-resource"],
- purpose: "Import a local file, folder, URL, or repository into OpenViking.",
+ purpose: "Import a local file, folder, URL, repository, or whole website (sitemap/RSS) into OpenViking.",
examples: &[
HelpItem {
label: "ov add-resource ./docs --parent viking://projects/acme --wait",
@@ -135,6 +135,10 @@ const COMMAND_HELP_SPECS: &[CommandHelpSpec] = &[
label: "ov add-resource https://example.com/spec.md --to viking://specs/api.md",
description: "Import a URL to an exact target URI.",
},
+ HelpItem {
+ label: "ov add-resource https://example.com/sitemap.xml --watch-interval 1440",
+ description: "Import a whole site via sitemap/RSS and refresh it daily.",
+ },
],
next_steps: &[
HelpItem {
diff --git a/docs/en/api/02-resources.md b/docs/en/api/02-resources.md
index 127a42d8b..df5619095 100644
--- a/docs/en/api/02-resources.md
+++ b/docs/en/api/02-resources.md
@@ -49,6 +49,16 @@ OpenViking supports various resource types, categorized by functionality:
|------|-------------|
| Feishu/Lark | URL-based, supports docx, wiki, sheets, bitable. By default uses app credentials from FEISHU_APP_ID and FEISHU_APP_SECRET; user-token imports can pass `args.feishu_access_token`, and user-token watches also pass `args.feishu_refresh_token` |
+**Whole Website (sitemap / RSS / Atom)**
+
+| Type | Resource Name | Description |
+|------|---------------|-------------|
+| Sitemap | `https://host/sitemap.xml`, `https://host/sitemap-index.xml` | Parses the sitemap and ingests every listed page as a single resource tree (one child node per page). Nested `<sitemapindex>` is followed recursively. The whole site becomes one resource under `viking://resources/<host>`. |
+| RSS / Atom feed | `https://host/rss.xml`, `https://host/atom.xml`, `https://host/feed` | Parses RSS 2.0 / Atom and ingests each entry as a tree node; the article body is fetched from its link (or taken inline when the feed carries full content). |
+| Whole-site auto-discovery | `https://host` + `args.site=true` | Forces whole-site ingestion for a bare domain or ordinary page: discovers the site's sitemap/RSS via robots.txt, HTML `<link rel="alternate">` autodiscovery, and conventional paths, then ingests it. |
+
+Crawling is bounded and non-recursive beyond the listed pages, and is governed by the `parsers.webfeed` config (`max_pages`, `max_concurrency`, `politeness_delay`, `same_host_only`, `respect_robots`, `max_depth`); robots.txt is honored. Set `watch_interval` on a sitemap/feed URL to keep the **whole site** refreshed: on each run new pages are added and removed pages drop automatically. When adding a single homepage (without `args.site`), the response may append a one-line hint suggesting whole-site ingestion — it never auto-crawls.
+
### Resource Processing Pipeline
Resources go through the following processing stages when added:
@@ -95,6 +105,7 @@ Resource incremental updates are implemented via the **Watch Task** mechanism:
#### Watch Task Creation
- Set `watch_interval > 0` (in minutes) when calling `add_resource` to create a watch task
- You may specify `to` to define the target URI; if omitted, the task binds to the `root_uri` returned by this import
+- Pointing a watch at a sitemap/RSS/Atom URL keeps the **whole site** in sync: each refresh re-reads the feed and rebuilds the tree, so newly published pages are added and removed pages drop automatically
- `WatchManager` handles task persistence
- Supports multi-tenant permission control (ROOT/ADMIN/USER permission levels)
@@ -157,7 +168,7 @@ This endpoint is the core entry point for resource management, supporting adding
| exclude | string | No | None | File patterns to exclude (glob) |
| directly_upload_media | bool | No | True | Whether to directly upload media files |
| preserve_structure | bool | No | None | Whether to preserve directory structure |
-| args | object | No | `{}` | Parser-specific import options forwarded to the source parser/accessor. Core `add_resource` fields such as `path`, `to`, `watch_interval`, `include`, and `exclude` are not allowed inside `args` |
+| args | object | No | `{}` | Parser-specific import options forwarded to the source parser/accessor. E.g. `args.site=true/false` forces/opts out of whole-site (sitemap/RSS) ingestion, `args.max_pages` etc. override the `webfeed` config, and Feishu user-token imports pass `args.feishu_access_token`. Core `add_resource` fields such as `path`, `to`, `watch_interval`, `include`, and `exclude` are not allowed inside `args` |
| watch_interval | float | No | 0 | Scheduled update interval (minutes). >0 creates task; <=0 cancels task; explicit `to` wins, otherwise binds to the imported `root_uri` |
| telemetry | TelemetryRequest | No | False | Whether to return telemetry data |
diff --git a/docs/zh/api/02-resources.md b/docs/zh/api/02-resources.md
index a4f5ec440..b8aec9c4e 100644
--- a/docs/zh/api/02-resources.md
+++ b/docs/zh/api/02-resources.md
@@ -44,6 +44,15 @@ OpenViking 支持多种资源类型,按照功能分类如下:
|------|------|
| 飞书/Lark | URL 方式,支持 docx, wiki, sheets, bitable。默认使用 FEISHU_APP_ID 和 FEISHU_APP_SECRET 应用凭证;用户 token 导入可传 `args.feishu_access_token`,用户 token watch 还需传 `args.feishu_refresh_token` |
+网站类(sitemap / RSS / Atom 整站导入)
+| 类型 | 资源名 | 说明 |
+|------|--------|------|
+| 站点地图 Sitemap | `https://host/sitemap.xml`、`https://host/sitemap-index.xml` | 解析 sitemap,将站点所有页面抓取为**一棵资源树**(每页一个子节点),支持嵌套 `<sitemapindex>` 递归。整站只生成一个资源,落在 `viking://resources/<host>`。 |
+| RSS / Atom 订阅源 | `https://host/rss.xml`、`https://host/atom.xml`、`https://host/feed` | 解析 RSS 2.0 / Atom,逐条把文章正文抓成树节点(feed 内含全文则直接使用,省一次抓取)。 |
+| 整站自动发现 | `https://host` + `args.site=true` | 对裸域名/普通页面强制整站导入:自动通过 robots.txt、HTML `<link rel="alternate">` autodiscovery、常见路径发现 sitemap/RSS,再整站抓取。 |
+
+抓取**有界、非递归**(不会超出所列页面继续爬),受 `parsers.webfeed` 配置约束(`max_pages`、`max_concurrency`、`politeness_delay`、`same_host_only`、`respect_robots`、`max_depth`),并遵守 robots.txt。对 sitemap/feed URL 设置 `watch_interval` 即可让**整站**周期刷新:每次运行自动纳入新增页面、移除已删除页面。添加单个首页(未带 `args.site`)时,返回信息可能附带一行"整站导入"提示——**只提示,绝不自动爬全站**。
+
### 资源处理流程
资源添加经过以下处理阶段:
@@ -90,6 +99,7 @@ URL/文件 Parser TreeBuilder AGFS Summarizer/Vector
#### 监控任务创建
- 调用 `add_resource` 时设置 `watch_interval > 0` (单位:分钟)创建监控任务
- 可指定 `to` 参数确定目标 URI;未指定时,系统会使用本次导入返回的 `root_uri` 作为监控目标
+- 把监控对象设为 sitemap/RSS/Atom URL,即可让**整站**保持同步:每次刷新重新读取 feed 并重建资源树,新发布的页面自动入库、已删除的页面自动移除
- `WatchManager` 负责任务持久化存储
- 支持多租户权限控制(ROOT/ADMIN/USER 权限分级)
@@ -152,7 +162,7 @@ URL/文件 Parser TreeBuilder AGFS Summarizer/Vector
| exclude | string | 否 | None | 排除的文件模式(glob) |
| directly_upload_media | bool | 否 | True | 是否直接上传媒体文件 |
| preserve_structure | bool | 否 | None | 是否保留目录结构 |
-| args | object | 否 | `{}` | 传给特定 parser/accessor 的导入参数。`path`、`to`、`watch_interval`、`include`、`exclude` 等 `add_resource` 核心字段不能放入 `args` |
+| args | object | 否 | `{}` | 传给特定 parser/accessor 的导入参数。例如 `args.site=true/false` 强制/禁用整站(sitemap/RSS)导入,`args.max_pages` 等可覆盖 `webfeed` 配置,飞书用户 token 导入传 `args.feishu_access_token`。`path`、`to`、`watch_interval`、`include`、`exclude` 等 `add_resource` 核心字段不能放入 `args` |
| watch_interval | float | 否 | 0 | 定时更新间隔(分钟)。>0 创建任务;≤0 取消任务;显式 `to` 优先,否则绑定本次导入的 `root_uri` |
| telemetry | TelemetryRequest | 否 | False | 是否返回遥测数据 |
diff --git a/examples/ov.conf.example b/examples/ov.conf.example
index 3ff148e3f..01294ecd4 100644
--- a/examples/ov.conf.example
+++ b/examples/ov.conf.example
@@ -234,6 +234,16 @@
"max_paragraph_length": 1000,
"preserve_line_breaks": false,
},
+ "webfeed": {
+ "max_pages": 200,
+ "max_concurrency": 5,
+ "request_timeout": 30.0,
+ "politeness_delay": 0.2,
+ "same_host_only": true,
+ "respect_robots": true,
+ "max_depth": 2,
+ "suggest_feed": true,
+ },
},
"encryption": {
"enabled": true,
diff --git a/openviking/async_client.py b/openviking/async_client.py
index 9a81ce5f8..87a04f86a 100644
--- a/openviking/async_client.py
+++ b/openviking/async_client.py
@@ -299,7 +299,9 @@ class AsyncOpenViking:
Add a resource (file/URL) to OpenViking.
Args:
- path: Local file path or URL.
+ path: Local file path or URL. A sitemap / RSS / Atom URL ingests the
+ whole site as one resource tree; pass ``args={"site": True}`` to
+ force whole-site ingestion from a bare domain.
reason: Context/reason for adding this resource.
instruction: Specific instruction for processing.
wait: If True, wait for processing to complete.
@@ -307,6 +309,9 @@ class AsyncOpenViking:
parent: Target parent URI (must already exist).
build_index: Whether to build vector index immediately (default: True).
summarize: Whether to generate summary (default: False).
+ watch_interval: Auto-refresh interval in minutes (>0 enables a watch).
+ On a sitemap/feed URL this keeps the whole site refreshed.
+ args: Parser/accessor-specific options (e.g. ``site``, ``max_pages``).
telemetry: Whether to attach operation telemetry data to the result.
"""
await self._ensure_initialized()
diff --git a/openviking/parse/accessors/__init__.py b/openviking/parse/accessors/__init__.py
index 6ae4fe2c5..6232573f1 100644
--- a/openviking/parse/accessors/__init__.py
+++ b/openviking/parse/accessors/__init__.py
@@ -18,6 +18,7 @@ from .registry import (
access,
get_accessor_registry,
)
+from .web_feed_accessor import WebFeedAccessor, discover_feed_hint
__all__ = [
# Base classes
@@ -32,4 +33,7 @@ __all__ = [
"HTTPAccessor",
"FeishuAccessor",
"LocalAccessor",
+ "WebFeedAccessor",
+ # Helpers
+ "discover_feed_hint",
]
diff --git a/openviking/parse/accessors/base.py b/openviking/parse/accessors/base.py
index 8d5b4c90e..22feee6ec 100644
--- a/openviking/parse/accessors/base.py
+++ b/openviking/parse/accessors/base.py
@@ -103,12 +103,15 @@ class DataAccessor(ABC):
"""
@abstractmethod
- def can_handle(self, source: Union[str, Path]) -> bool:
+ def can_handle(self, source: Union[str, Path], **kwargs) -> bool:
"""
Check if this accessor can handle the given source.
Args:
source: Source string (URL, path, etc.) or Path object
+ **kwargs: Optional accessor-selection hints forwarded from
+ ``access()`` (e.g. an explicit ``site=True`` override). Most
+ accessors ignore these and decide purely from ``source``.
Returns:
True if this accessor can handle the source
diff --git a/openviking/parse/accessors/feishu_accessor.py b/openviking/parse/accessors/feishu_accessor.py
index bf1fc349b..16f561c02 100644
--- a/openviking/parse/accessors/feishu_accessor.py
+++ b/openviking/parse/accessors/feishu_accessor.py
@@ -150,7 +150,7 @@ class FeishuAccessor(DataAccessor):
def priority(self) -> int:
return self.PRIORITY
- def can_handle(self, source: Union[str, Path]) -> bool:
+ def can_handle(self, source: Union[str, Path], **kwargs) -> bool:
"""
Check if this accessor can handle the source.
diff --git a/openviking/parse/accessors/git_accessor.py b/openviking/parse/accessors/git_accessor.py
index a4cc7154f..718f02258 100644
--- a/openviking/parse/accessors/git_accessor.py
+++ b/openviking/parse/accessors/git_accessor.py
@@ -51,7 +51,7 @@ class GitAccessor(DataAccessor):
def priority(self) -> int:
return self.PRIORITY
- def can_handle(self, source: Union[str, Path]) -> bool:
+ def can_handle(self, source: Union[str, Path], **kwargs) -> bool:
"""
Check if this accessor can handle the source.
diff --git a/openviking/parse/accessors/http_accessor.py b/openviking/parse/accessors/http_accessor.py
index cb0a52dfd..3265d9c7e 100644
--- a/openviking/parse/accessors/http_accessor.py
+++ b/openviking/parse/accessors/http_accessor.py
@@ -416,7 +416,7 @@ class HTTPAccessor(DataAccessor):
def priority(self) -> int:
return self.PRIORITY
- def can_handle(self, source: Union[str, Path]) -> bool:
+ def can_handle(self, source: Union[str, Path], **kwargs) -> bool:
"""
Check if this accessor can handle the source.
diff --git a/openviking/parse/accessors/local_accessor.py b/openviking/parse/accessors/local_accessor.py
index a5f71acd1..55c117fd6 100644
--- a/openviking/parse/accessors/local_accessor.py
+++ b/openviking/parse/accessors/local_accessor.py
@@ -28,7 +28,7 @@ class LocalAccessor(DataAccessor):
- Provides clear local source type metadata
"""
- def can_handle(self, source: Union[str, Path]) -> bool:
+ def can_handle(self, source: Union[str, Path], **kwargs) -> bool:
"""
Check if this accessor can handle the source.
diff --git a/openviking/parse/accessors/registry.py b/openviking/parse/accessors/registry.py
index d4aed7526..f37d79e73 100644
--- a/openviking/parse/accessors/registry.py
+++ b/openviking/parse/accessors/registry.py
@@ -6,8 +6,9 @@ Registry for Data Accessors.
Manages DataAccessor registration and provides automatic source routing.
"""
+import inspect
from pathlib import Path
-from typing import List, Optional, Union
+from typing import Callable, List, Optional, Union
from openviking_cli.utils import get_logger
@@ -16,6 +17,20 @@ from .base import DataAccessor, LocalResource
logger = get_logger(__name__)
+def _accepts_var_keyword(func: Callable) -> bool:
+ """Whether ``func`` accepts arbitrary keyword arguments (**kwargs).
+
+ Lets the registry forward accessor-selection hints to modern accessors
+ (e.g. WebFeedAccessor's ``site=True`` override) while staying compatible
+ with accessors whose ``can_handle`` only takes ``source``.
+ """
+ try:
+ sig = inspect.signature(func)
+ except (TypeError, ValueError):
+ return True
+ return any(p.kind == p.VAR_KEYWORD for p in sig.parameters.values())
+
+
class AccessorRegistry:
"""
Registry for data accessors.
@@ -45,6 +60,14 @@ class AccessorRegistry:
except Exception as e:
logger.debug(f"[AccessorRegistry] Failed to register GitAccessor: {e}")
+ # WebFeedAccessor - handles sitemap / RSS / Atom URLs (whole-site ingest)
+ try:
+ from .web_feed_accessor import WebFeedAccessor
+
+ self.register(WebFeedAccessor())
+ except Exception as e:
+ logger.debug(f"[AccessorRegistry] Failed to register WebFeedAccessor: {e}")
+
# HTTPAccessor - handles HTTP/HTTPS URLs
try:
from .http_accessor import HTTPAccessor
@@ -109,18 +132,24 @@ class AccessorRegistry:
return True
return False
- def get_accessor(self, source: Union[str, Path]) -> Optional[DataAccessor]:
+ def get_accessor(self, source: Union[str, Path], **kwargs) -> Optional[DataAccessor]:
"""
Get the highest priority accessor that can handle the source.
Args:
source: Source string or Path to check
+ **kwargs: Optional accessor-selection hints forwarded to
+ ``can_handle`` (e.g. an explicit ``site=True`` override).
Returns:
DataAccessor instance or None if no accessor can handle the source
"""
for accessor in self._accessors:
- if accessor.can_handle(source):
+ if kwargs and _accepts_var_keyword(accessor.can_handle):
+ handled = accessor.can_handle(source, **kwargs)
+ else:
+ handled = accessor.can_handle(source)
+ if handled:
return accessor
return None
@@ -151,8 +180,10 @@ class AccessorRegistry:
"""
source_str = str(source)
- # Find an accessor - LocalAccessor should always be registered as fallback
- accessor = self.get_accessor(source)
+ # Find an accessor - LocalAccessor should always be registered as fallback.
+ # Forward kwargs so accessors can honor explicit selection hints (e.g.
+ # args={"site": True}) that aren't inferable from the URL alone.
+ accessor = self.get_accessor(source, **kwargs)
if accessor:
logger.debug(
f"[AccessorRegistry] Using accessor {accessor.__class__.__name__} for source: {source_str}"
diff --git a/openviking/parse/accessors/web_feed_accessor.py b/openviking/parse/accessors/web_feed_accessor.py
new file mode 100644
index 000000000..bcddf07ae
--- /dev/null
+++ b/openviking/parse/accessors/web_feed_accessor.py
@@ -0,0 +1,832 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""
+Web Feed Accessor.
+
+Ingests an ENTIRE website in one shot from a single "collection" URL:
+- XML sitemaps (``<urlset>``) and sitemap indexes (``<sitemapindex>``, recursed)
+- RSS 2.0 feeds (``<rss><channel><item>``)
+- Atom feeds (``<feed><entry>``)
+
+The shared core capability is "a URL that enumerates many page URLs". Whatever
+the source, this accessor mirrors the listed pages into a single local temp
+directory and returns it as a directory ``LocalResource`` — exactly the contract
+GitAccessor uses. The downstream DirectoryParser then builds ONE resource tree
+(one ``viking://resources/<host>`` URI, one watchable node) with a child per page.
+
+Because a watch re-runs ``add_resource(path=<feed url>)`` on each refresh, watching
+a sitemap/feed URL keeps the whole site up to date: new pages appear and removed
+pages drop automatically on the next rebuild.
+
+This module also exposes ``discover_feed_hint``: a cheap, failure-tolerant probe
+used when a user adds a single webpage, to *suggest* whole-site ingestion without
+ever auto-crawling.
+"""
+
+import asyncio
+import fnmatch
+import hashlib
+import html as html_lib
+import re
+import shutil
+import tempfile
+from dataclasses import dataclass
+from pathlib import Path, PurePosixPath
+from typing import Any, Dict, List, Optional, Tuple, Union
+from urllib.parse import unquote, urljoin, urlparse
+from urllib.robotparser import RobotFileParser
+
+from openviking.parse.base import lazy_import
+from openviking.utils.network_guard import build_httpx_request_validation_hooks
+from openviking_cli.utils.logger import get_logger
+
+from .base import DataAccessor, LocalResource, SourceType
+from .http_accessor import HTTPAccessor
+
+logger = get_logger(__name__)
+
+_USER_AGENT = HTTPAccessor.DEFAULT_USER_AGENT
+
+# kwargs keys that explicitly force / forbid whole-site ingestion (args={"site": ...}).
+_OVERRIDE_KEYS = ("site", "sitemap", "feed", "as_site")
+
+# URL path basenames that confidently indicate a sitemap / feed.
+_FEED_BASENAMES = {
+ "rss.xml",
+ "feed.xml",
+ "atom.xml",
+ "feed.atom",
+ "index.xml",
+ "rss",
+ "feed",
+ "feeds",
+ "atom",
+}
+
+# Web page extensions that should be stripped from a URL slug (we re-save as .html).
+_WEB_PAGE_EXTENSIONS = (".html", ".htm", ".php", ".asp", ".aspx", ".jsp")
+
+# Common conventional locations for a site's sitemap / feed, probed when auto-discovering.
+_COMMON_FEED_PATHS = (
+ "/sitemap.xml",
+ "/sitemap-index.xml",
+ "/sitemap_index.xml",
+ "/rss.xml",
+ "/feed.xml",
+ "/atom.xml",
+ "/index.xml",
+ "/feed",
+ "/rss",
+)
+
+
+@dataclass
+class FeedEntry:
+ """A single page discovered in a sitemap or feed."""
+
+ url: str
+ title: Optional[str] = None
+ lastmod: Optional[str] = None
+ inline_html: Optional[str] = None
+ """Full article HTML carried inline by the feed (RSS content:encoded / Atom
+ content). When present we save it directly instead of fetching the page."""
+
+
+# --------------------------------------------------------------------------- #
+# Module-level helpers (kept free-standing so they're easy to unit test)
+# --------------------------------------------------------------------------- #
+def _localname(tag: str) -> str:
+ """Return the namespace-stripped, lowercased local name of an XML tag."""
+ return tag.rsplit("}", 1)[-1].lower()
+
+
+def looks_like_feed_url(source: Union[str, Path]) -> bool:
+ """Heuristic: does this URL path look like a sitemap / RSS / Atom feed?
+
+ Conservative on purpose — ordinary article URLs must fall through to
+ HTTPAccessor for single-page ingestion.
+ """
+ source_str = str(source)
+ if not source_str.startswith(("http://", "https://")):
+ return False
+ basename = urlparse(source_str).path.lower().rstrip("/").rsplit("/", 1)[-1]
+ if not basename:
+ return False
+ if basename.startswith("sitemap") and basename.endswith(".xml"):
+ return True
+ if basename in _FEED_BASENAMES:
+ return True
+ if basename.endswith((".rss", ".atom")):
+ return True
+ return False
+
+
+def _resolve_override(kwargs: Dict[str, Any]) -> Optional[bool]:
+ """Resolve an explicit site/feed override from kwargs, if any."""
+ for key in _OVERRIDE_KEYS:
+ value = kwargs.get(key)
+ if isinstance(value, bool):
+ return value
+ return None
+
+
+def _sanitize_filename(name: str, max_len: int = 120) -> str:
+ """Make a string safe to use as a single path segment.
+
+ Keeps unicode (e.g. CJK titles) readable; only strips path separators,
+ reserved/control characters, and trims length. The resulting stem becomes
+ the page node's title downstream, so readability matters.
+ """
+ name = (name or "").strip()
+ # Replace path separators, control chars, and filesystem-reserved chars.
+ name = re.sub(r'[\\/\x00-\x1f<>:"|?*]+', "-", name)
+ name = re.sub(r"\s+", " ", name).strip(" .-")
+ if len(name) > max_len:
+ name = name[:max_len].strip(" .-")
+ return name
+
+
+def _slugify_segment(segment: str) -> str:
+ """Turn one URL path segment into a clean filename stem (no extension)."""
+ segment = unquote(segment)
+ lower = segment.lower()
+ for ext in _WEB_PAGE_EXTENSIONS:
+ if lower.endswith(ext):
+ segment = segment[: -len(ext)]
+ break
+ return _sanitize_filename(segment)
+
+
+def url_to_relpath(url: str) -> str:
+ """Map a page URL to a relative mirror path, preserving directory structure.
+
+ Examples:
+ https://h/ -> index.html
+ https://h/a/b -> a/b.html
+ https://h/a/b/ -> a/b.html
+ https://h/post/foo/index.html -> post/foo/index.html
+ """
+ segments = [s for s in urlparse(url).path.split("/") if s]
+ if not segments:
+ return "index.html"
+ *parents, leaf = segments
+ leaf_stem = _slugify_segment(leaf) or "index"
+ parent_parts = [p for p in (_sanitize_filename(unquote(s)) for s in parents) if p]
+ return "/".join(parent_parts + [f"{leaf_stem}.html"])
+
+
+def _entry_to_relpath(entry: FeedEntry) -> str:
+ """Choose the mirror path for an entry.
+
+ Feed entries usually carry a real title -> use it as a flat, readable
+ filename. Sitemap entries have no title -> mirror the URL path structure.
+ """
+ if entry.title:
+ stem = _sanitize_filename(entry.title)
+ if stem:
+ return f"{stem}.html"
+ return url_to_relpath(entry.url)
+
+
+def _dedup_relpath(relpath: str, url: str, used: set) -> str:
+ """Disambiguate a colliding relpath with a short stable hash of the URL."""
+ if relpath not in used:
+ return relpath
+ p = PurePosixPath(relpath)
+ digest = hashlib.sha1(url.encode("utf-8")).hexdigest()[:6]
+ newname = f"{p.stem}-{digest}{p.suffix}"
+ parent = str(p.parent)
+ return newname if parent == "." else f"{parent}/{newname}"
+
+
+def _wrap_inline_html(title: Optional[str], fragment: str) -> str:
+ """Wrap an inline feed HTML fragment into a minimal standalone document."""
+ title_tag = f"<title>{html_lib.escape(title)}</title>" if title else ""
+ return (
+ '<!DOCTYPE html><html><head><meta charset="utf-8">'
+ f"{title_tag}</head><body>{fragment}</body></html>"
+ )
+
+
+def _split_patterns(value: Optional[str]) -> List[str]:
+ if not value:
+ return []
+ return [p.strip() for p in str(value).split(",") if p.strip()]
+
+
+def _match_filters(url: str, include: Optional[str], exclude: Optional[str]) -> bool:
+ """Apply comma-separated include/exclude glob (or substring) filters to a URL."""
+ for pat in _split_patterns(exclude):
+ if fnmatch.fnmatch(url, pat) or pat in url:
+ return False
+ includes = _split_patterns(include)
+ if includes:
+ return any(fnmatch.fnmatch(url, pat) or pat in url for pat in includes)
+ return True
+
+
+def _build_httpx_client_kwargs(request_validator, timeout: float) -> Dict[str, Any]:
+ """Build httpx.AsyncClient kwargs with the shared SSRF network guard.
+
+ Mirrors HTTPAccessor's client construction so sitemaps, feeds, sub-sitemaps,
+ robots.txt and every mirrored page go through the same request validation.
+ """
+ client_kwargs: Dict[str, Any] = {"timeout": timeout, "follow_redirects": True}
+ event_hooks = build_httpx_request_validation_hooks(request_validator)
+ if event_hooks:
+ client_kwargs["event_hooks"] = event_hooks
+ client_kwargs["trust_env"] = False
+ return client_kwargs
+
+
+def _load_webfeed_config():
+ """Return the configured WebFeedConfig, falling back to defaults."""
+ from openviking_cli.utils.config.parser_config import WebFeedConfig
+
+ try:
+ from openviking_cli.utils.config import get_openviking_config
+
+ cfg = getattr(get_openviking_config(), "webfeed", None)
+ if cfg is not None:
+ return cfg
+ except Exception:
+ pass
+ return WebFeedConfig()
+
+
+# --------------------------------------------------------------------------- #
+# Accessor
+# --------------------------------------------------------------------------- #
+class WebFeedAccessor(DataAccessor):
+ """Accessor that ingests a whole website from a sitemap / RSS / Atom URL."""
+
+ PRIORITY = 60 # Feishu 100 > Git 80 > WebFeed 60 > HTTP 50 > Local 1
+
+ @property
+ def priority(self) -> int:
+ return self.PRIORITY
+
+ def can_handle(self, source: Union[str, Path], **kwargs) -> bool:
+ """Handle sitemap/feed URLs (by heuristic, or by explicit override).
+
+ ``args={"site": True}`` forces any http(s) URL through this accessor;
+ ``args={"site": False}`` opts a feed-looking URL back out to HTTPAccessor.
+ """
+ source_str = str(source)
+ if not source_str.startswith(("http://", "https://")):
+ return False
+ override = _resolve_override(kwargs)
+ if override is not None:
+ return override
+ return looks_like_feed_url(source_str)
+
+ async def access(self, source: Union[str, Path], **kwargs) -> LocalResource:
+ """Fetch the sitemap/feed, mirror its pages, return a directory resource."""
+ source_url = str(source)
+ request_validator = kwargs.get("request_validator")
+ cfg = self._resolve_settings(kwargs)
+ include = kwargs.get("include")
+ exclude = kwargs.get("exclude")
+
+ temp_dir = await asyncio.to_thread(tempfile.mkdtemp, prefix="ov_webfeed_")
+ try:
+ client_kwargs = _build_httpx_client_kwargs(request_validator, cfg["request_timeout"])
+ httpx = lazy_import("httpx")
+ async with httpx.AsyncClient(**client_kwargs) as client:
+ # The given URL may already be a sitemap/feed, or it may be a plain
+ # page / bare domain (e.g. args={"site": True} on "https://t0saki.com").
+ # In the latter case, auto-discover the site's sitemap/RSS.
+ feed_url, prefetched = await self._resolve_feed_source(source_url, client)
+ entries, feed_kind = await self._collect(
+ feed_url, client, cfg, prefetched=prefetched
+ )
+ if not entries:
+ raise RuntimeError(f"No pages found in sitemap/feed: {feed_url}")
+
+ entries = self._filter_entries(entries, feed_url, cfg, include, exclude)
+ entries = await self._apply_robots(entries, feed_url, client, cfg)
+
+ if len(entries) > cfg["max_pages"]:
+ logger.warning(
+ "[WebFeedAccessor] %s lists %d pages; truncating to max_pages=%d",
+ feed_url,
+ len(entries),
+ cfg["max_pages"],
+ )
+ entries = entries[: cfg["max_pages"]]
+
+ if not entries:
+ raise RuntimeError(f"No ingestible pages remained after filtering: {feed_url}")
+
+ pages = await self._mirror(entries, client, temp_dir, cfg)
+
+ if not pages:
+ raise RuntimeError(f"Failed to fetch any page from: {feed_url}")
+
+ host = urlparse(feed_url).netloc
+ meta = {
+ # Drives the root URI -> viking://resources/<host>
+ "original_filename": host,
+ "feed_url": feed_url,
+ "feed_kind": feed_kind,
+ "url_type": feed_kind,
+ "page_count": len(pages),
+ "pages": pages,
+ }
+ logger.info(
+ "[WebFeedAccessor] Mirrored %d page(s) from %s (%s)",
+ len(pages),
+ feed_url,
+ feed_kind,
+ )
+ return LocalResource(
+ path=Path(temp_dir),
+ source_type=SourceType.HTTP,
+ original_source=feed_url,
+ meta=meta,
+ is_temporary=True,
+ )
+ except Exception as e:
+ logger.error("[WebFeedAccessor] Failed to access %s: %s", source_url, e, exc_info=True)
+ shutil.rmtree(temp_dir, ignore_errors=True)
+ raise
+
+ # -- settings ----------------------------------------------------------- #
+ def _resolve_settings(self, kwargs: Dict[str, Any]) -> Dict[str, Any]:
+ """Merge WebFeedConfig defaults with per-call kwargs overrides."""
+ cfg = _load_webfeed_config()
+
+ def pick(key, cast, default):
+ value = kwargs.get(key)
+ if value is None:
+ value = getattr(cfg, key, default)
+ try:
+ return cast(value)
+ except (TypeError, ValueError):
+ return default
+
+ return {
+ "max_pages": pick("max_pages", int, 200),
+ "max_concurrency": max(1, pick("max_concurrency", int, 5)),
+ "request_timeout": pick("request_timeout", float, 30.0),
+ "politeness_delay": max(0.0, pick("politeness_delay", float, 0.2)),
+ "same_host_only": bool(
+ kwargs.get("same_host_only", getattr(cfg, "same_host_only", True))
+ ),
+ "respect_robots": bool(
+ kwargs.get("respect_robots", getattr(cfg, "respect_robots", True))
+ ),
+ "max_depth": max(1, pick("max_depth", int, 2)),
+ }
+
+ # -- discovery ---------------------------------------------------------- #
+ async def _fetch_bytes(self, url: str, client) -> bytes:
+ response = await client.get(url, headers={"User-Agent": _USER_AGENT})
+ response.raise_for_status()
+ return response.content
+
+ def _sniff(self, content: bytes) -> Tuple[str, Any]:
+ """Classify a fetched document by its XML root element."""
+ from defusedxml.ElementTree import fromstring
+
+ try:
+ root = fromstring(content)
+ except Exception:
+ # Not well-formed XML we can route — let feedparser try as a feed.
+ return "feed", None
+ name = _localname(root.tag)
+ if name == "urlset":
+ return "urlset", root
+ if name == "sitemapindex":
+ return "sitemapindex", root
+ if name in ("rss", "feed"):
+ return "feed", root
+ return "unknown", root
+
+ def _is_feed_document(self, content: bytes) -> bool:
+ """Whether ``content`` is a usable sitemap / sitemapindex / RSS / Atom doc."""
+ kind, _ = self._sniff(content)
+ if kind in ("urlset", "sitemapindex"):
+ return True
+ if kind == "feed":
+ # "feed" also covers the non-XML fallback; confirm real entries exist.
+ try:
+ import feedparser
+
+ return len(feedparser.parse(content).entries) > 0
+ except Exception:
+ return False
+ return False
+
+ async def _resolve_feed_source(self, url: str, client) -> Tuple[str, bytes]:
+ """Return a ``(feed_url, content)`` that is an actual sitemap/feed.
+
+ If ``url`` already points at a sitemap/feed, it is returned as-is. Otherwise
+ (a plain page or bare domain — e.g. ``args={"site": True}`` on a homepage),
+ auto-discover the site's sitemap/RSS via robots.txt, ``<link rel=alternate>``
+ autodiscovery, and conventional locations, and return the first real one.
+ """
+ content = await self._fetch_bytes(url, client)
+ if self._is_feed_document(content):
+ return url, content
+
+ for candidate in await _gather_feed_candidates(client, url, content):
+ if candidate == url:
+ continue
+ try:
+ cand_content = await self._fetch_bytes(candidate, client)
+ except Exception:
+ continue
+ if self._is_feed_document(cand_content):
+ logger.info("[WebFeedAccessor] auto-discovered feed %s for %s", candidate, url)
+ return candidate, cand_content
+
+ raise RuntimeError(
+ f"{url} is not a sitemap/feed and no sitemap/RSS could be auto-discovered "
+ "for the site. Pass the sitemap/feed URL directly."
+ )
+
+ async def _collect(
+ self,
+ url: str,
+ client,
+ cfg: Dict[str, Any],
+ depth: int = 0,
+ visited: Optional[set] = None,
+ prefetched: Optional[bytes] = None,
+ ) -> Tuple[List[FeedEntry], str]:
+ """Recursively collect entries from a sitemap / sitemapindex / feed."""
+ if visited is None:
+ visited = {url}
+ content = prefetched if prefetched is not None else await self._fetch_bytes(url, client)
+ kind, root = self._sniff(content)
+
+ if kind == "sitemapindex":
+ if depth >= cfg["max_depth"]:
+ logger.warning(
+ "[WebFeedAccessor] sitemapindex depth limit (%d) reached at %s",
+ cfg["max_depth"],
+ url,
+ )
+ return [], "sitemap"
+ all_entries: List[FeedEntry] = []
+ for sub_url in self._parse_sitemapindex(root):
+ if sub_url in visited:
+ continue
+ visited.add(sub_url)
+ try:
+ sub_entries, _ = await self._collect(sub_url, client, cfg, depth + 1, visited)
+ all_entries.extend(sub_entries)
+ except Exception as e:
+ logger.warning(
+ "[WebFeedAccessor] Failed to read sub-sitemap %s: %s", sub_url, e
+ )
+ return all_entries, "sitemap"
+
+ if kind == "urlset":
+ return self._parse_urlset(root), "sitemap"
+
+ if kind == "feed":
+ return self._parse_feed(content, base_url=url)
+
+ raise RuntimeError(f"Unrecognized document (not a sitemap or feed): {url}")
+
+ @staticmethod
+ def _parse_urlset(root) -> List[FeedEntry]:
+ entries: List[FeedEntry] = []
+ for url_el in root:
+ if _localname(url_el.tag) != "url":
+ continue
+ loc = None
+ lastmod = None
+ for child in url_el:
+ ln = _localname(child.tag)
+ if ln == "loc":
+ loc = (child.text or "").strip()
+ elif ln == "lastmod":
+ lastmod = (child.text or "").strip()
+ if loc:
+ entries.append(FeedEntry(url=loc, lastmod=lastmod))
+ return entries
+
+ @staticmethod
+ def _parse_sitemapindex(root) -> List[str]:
+ locs: List[str] = []
+ for sm_el in root:
+ if _localname(sm_el.tag) != "sitemap":
+ continue
+ for child in sm_el:
+ if _localname(child.tag) == "loc":
+ loc = (child.text or "").strip()
+ if loc:
+ locs.append(loc)
+ return locs
+
+ @staticmethod
+ def _parse_feed(content: bytes, base_url: str) -> Tuple[List[FeedEntry], str]:
+ import feedparser
+
+ parsed = feedparser.parse(content)
+ version = str(parsed.get("version") or "")
+ feed_kind = "atom" if version.startswith("atom") else "rss"
+
+ entries: List[FeedEntry] = []
+ for item in parsed.entries:
+ link = item.get("link")
+ if not link:
+ for link_obj in item.get("links", []) or []:
+ if link_obj.get("rel") in (None, "alternate") and link_obj.get("href"):
+ link = link_obj["href"]
+ break
+ if not link:
+ continue
+ link = urljoin(base_url, link)
+
+ inline_html = None
+ contents = item.get("content")
+ if isinstance(contents, list) and contents and contents[0].get("value"):
+ inline_html = contents[0]["value"]
+
+ entries.append(
+ FeedEntry(
+ url=link,
+ title=item.get("title"),
+ lastmod=item.get("published") or item.get("updated"),
+ inline_html=inline_html,
+ )
+ )
+ return entries, feed_kind
+
+ # -- filtering ---------------------------------------------------------- #
+ def _filter_entries(
+ self,
+ entries: List[FeedEntry],
+ feed_url: str,
+ cfg: Dict[str, Any],
+ include: Optional[str],
+ exclude: Optional[str],
+ ) -> List[FeedEntry]:
+ base_host = urlparse(feed_url).netloc
+ seen: set = set()
+ out: List[FeedEntry] = []
+ for entry in entries:
+ url = entry.url
+ if not url.startswith(("http://", "https://")):
+ continue
+ if url in seen:
+ continue
+ if cfg["same_host_only"] and urlparse(url).netloc != base_host:
+ continue
+ if not _match_filters(url, include, exclude):
+ continue
+ seen.add(url)
+ out.append(entry)
+ return out
+
+ async def _apply_robots(
+ self, entries: List[FeedEntry], feed_url: str, client, cfg: Dict[str, Any]
+ ) -> List[FeedEntry]:
+ if not cfg["respect_robots"]:
+ return entries
+ try:
+ parsed = urlparse(feed_url)
+ robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"
+ response = await client.get(robots_url, headers={"User-Agent": _USER_AGENT})
+ if response.status_code >= 400:
+ return entries
+ rp = RobotFileParser()
+ rp.parse(response.text.splitlines())
+ return [e for e in entries if rp.can_fetch(_USER_AGENT, e.url)]
+ except Exception as e:
+ logger.debug("[WebFeedAccessor] robots.txt check skipped for %s: %s", feed_url, e)
+ return entries
+
+ # -- mirroring ---------------------------------------------------------- #
+ async def _mirror(
+ self, entries: List[FeedEntry], client, temp_dir: str, cfg: Dict[str, Any]
+ ) -> Dict[str, Optional[str]]:
+ """Fetch (or write inline) each page into temp_dir; return {url: lastmod}."""
+ used: set = set()
+ planned: List[Tuple[FeedEntry, str]] = []
+ for entry in entries:
+ relpath = _dedup_relpath(_entry_to_relpath(entry), entry.url, used)
+ used.add(relpath)
+ planned.append((entry, relpath))
+
+ sem = asyncio.Semaphore(cfg["max_concurrency"])
+
+ async def mirror_one(entry: FeedEntry, relpath: str) -> bool:
+ async with sem:
+ if cfg["politeness_delay"]:
+ await asyncio.sleep(cfg["politeness_delay"])
+ dest = Path(temp_dir) / relpath
+ await asyncio.to_thread(dest.parent.mkdir, parents=True, exist_ok=True)
+
+ if entry.inline_html:
+ document = _wrap_inline_html(entry.title, entry.inline_html)
+ await asyncio.to_thread(dest.write_text, document, encoding="utf-8")
+ return True
+
+ response = await client.get(entry.url, headers={"User-Agent": _USER_AGENT})
+ response.raise_for_status()
+ content_type = response.headers.get("content-type", "").lower()
+ if content_type and not any(
+ token in content_type for token in ("html", "xml", "text")
+ ):
+ logger.debug(
+ "[WebFeedAccessor] Skipping non-text page %s (%s)",
+ entry.url,
+ content_type,
+ )
+ return False
+ await asyncio.to_thread(dest.write_bytes, response.content)
+ return True
+
+ results = await asyncio.gather(
+ *(mirror_one(entry, relpath) for entry, relpath in planned),
+ return_exceptions=True,
+ )
+
+ pages: Dict[str, Optional[str]] = {}
+ for (entry, _relpath), result in zip(planned, results, strict=True):
+ if isinstance(result, Exception):
+ logger.warning("[WebFeedAccessor] Failed to mirror %s: %s", entry.url, result)
+ continue
+ if result:
+ pages[entry.url] = entry.lastmod
+ return pages
+
+
+# --------------------------------------------------------------------------- #
+# Single-page detect-and-suggest (never auto-crawls)
+# --------------------------------------------------------------------------- #
+def _extract_attr(tag: str, name: str) -> Optional[str]:
+ match = re.search(rf'{name}\s*=\s*"([^"]*)"', tag, re.I) or re.search(
+ rf"{name}\s*=\s*'([^']*)'", tag, re.I
+ )
+ return match.group(1) if match else None
+
+
+def _extract_feed_links(html: str, base_url: str) -> List[str]:
+ """Find <link rel=alternate type=rss/atom> and <link rel=sitemap> hrefs."""
+ out: List[str] = []
+ for match in re.finditer(r"<link\b[^>]*>", html, re.I):
+ tag = match.group(0)
+ href = _extract_attr(tag, "href")
+ if not href:
+ continue
+ rel = (_extract_attr(tag, "rel") or "").lower()
+ typ = (_extract_attr(tag, "type") or "").lower()
+ if "alternate" in rel and typ in ("application/rss+xml", "application/atom+xml"):
+ out.append(urljoin(base_url, href))
+ elif rel == "sitemap":
+ out.append(urljoin(base_url, href))
+ return out
+
+
+def _classify_and_count(content: bytes) -> Optional[Tuple[str, int]]:
+ """Lightly classify a candidate document and count its entries (no recursion)."""
+ from defusedxml.ElementTree import fromstring
+
+ try:
+ root = fromstring(content)
+ except Exception:
+ try:
+ import feedparser
+
+ parsed = feedparser.parse(content)
+ if parsed.entries:
+ version = str(parsed.get("version") or "")
+ kind = "atom" if version.startswith("atom") else "rss"
+ return kind, len(parsed.entries)
+ except Exception:
+ pass
+ return None
+
+ name = _localname(root.tag)
+ if name == "urlset":
+ return "sitemap", sum(1 for el in root if _localname(el.tag) == "url")
+ if name == "sitemapindex":
+ return "sitemapindex", sum(1 for el in root if _localname(el.tag) == "sitemap")
+ if name in ("rss", "feed"):
+ try:
+ import feedparser
+
+ parsed = feedparser.parse(content)
+ kind = "atom" if name == "feed" else "rss"
+ return kind, len(parsed.entries)
+ except Exception:
+ return None
+ return None
+
+
+def _format_hint(feed_url: str, kind: str, count: int) -> str:
+ if kind == "sitemap":
+ what = f"a sitemap with {count} page(s)"
+ elif kind == "sitemapindex":
+ what = f"a sitemap index ({count} sub-sitemap(s))"
+ elif kind == "atom":
+ what = f"an Atom feed with {count} entr{'y' if count == 1 else 'ies'}"
+ else:
+ what = f"an RSS feed with {count} item(s)"
+ return (
+ f"Tip: this site exposes {what}. To ingest the whole site as a single "
+ f"watchable resource, run add_resource('{feed_url}')."
+ )
+
+
+async def _gather_feed_candidates(client, page_url: str, page_content: bytes) -> List[str]:
+ """Collect ordered candidate sitemap/feed URLs for a site.
+
+ Sources, in priority order: robots.txt ``Sitemap:`` directives, ``<link
+ rel=alternate|sitemap>`` autodiscovery in the page HTML, and conventional paths.
+ Shared by the single-page hint probe and the whole-site auto-discovery fallback.
+ """
+ parsed = urlparse(page_url)
+ origin = f"{parsed.scheme}://{parsed.netloc}"
+ candidates: List[str] = []
+ # 1. robots.txt Sitemap: directives
+ try:
+ response = await client.get(origin + "/robots.txt", headers={"User-Agent": _USER_AGENT})
+ if response.status_code < 400:
+ rp = RobotFileParser()
+ rp.parse(response.text.splitlines())
+ candidates.extend(rp.site_maps() or [])
+ except Exception:
+ pass
+ # 2. <link rel=alternate/sitemap> autodiscovery from the page HTML
+ try:
+ if page_content:
+ candidates.extend(
+ _extract_feed_links(page_content.decode("utf-8", "replace"), page_url)
+ )
+ except Exception:
+ pass
+ # 3. conventional locations
+ candidates.extend(origin + path for path in _COMMON_FEED_PATHS)
+
+ seen: set = set()
+ return [c for c in candidates if not (c in seen or seen.add(c))]
+
+
+async def _discover_feed_hint(url: str, request_validator) -> Optional[str]:
+ url = str(url)
+ if not url.startswith(("http://", "https://")):
+ return None
+ try:
+ from openviking.utils.code_hosting_utils import is_git_repo_url
+
+ if is_git_repo_url(url):
+ return None
+ except Exception:
+ pass
+ if looks_like_feed_url(url):
+ return None # already a feed — nothing to suggest
+ ext = Path(urlparse(url).path).suffix.lower()
+ if ext and ext not in _WEB_PAGE_EXTENSIONS:
+ return None # a downloadable file (pdf/img/...), not a webpage
+
+ httpx = lazy_import("httpx")
+ client_kwargs = _build_httpx_client_kwargs(request_validator, 5.0)
+ async with httpx.AsyncClient(**client_kwargs) as client:
+ # Fetch the page once for <link> autodiscovery (best-effort).
+ page_content = b""
+ try:
+ response = await client.get(url, headers={"User-Agent": _USER_AGENT})
+ if (
+ response.status_code < 400
+ and "html" in response.headers.get("content-type", "").lower()
+ ):
+ page_content = response.content
+ except Exception:
+ pass
+
+ for candidate in await _gather_feed_candidates(client, url, page_content):
+ try:
+ response = await client.get(candidate, headers={"User-Agent": _USER_AGENT})
+ if response.status_code >= 400:
+ continue
+ info = _classify_and_count(response.content)
+ if info and info[1] > 0:
+ return _format_hint(candidate, info[0], info[1])
+ except Exception:
+ continue
+ return None
+
+
+async def discover_feed_hint(
+ url: Union[str, Path],
+ *,
+ timeout: float = 2.5,
+ request_validator=None,
+) -> Optional[str]:
+ """Best-effort probe: suggest whole-site ingestion when a single page is added.
+
+ Hard-bounded by ``timeout`` and fully exception-swallowing — it must never
+ block, slow, or fail the single-page add it is attached to. Returns a
+ one-line hint string, or None when nothing relevant is found.
+ """
+ try:
+ return await asyncio.wait_for(_discover_feed_hint(str(url), request_validator), timeout)
+ except Exception:
+ return None
diff --git a/openviking/server/mcp_endpoint.py b/openviking/server/mcp_endpoint.py
index fb8f0e1d3..51da9491d 100644
--- a/openviking/server/mcp_endpoint.py
+++ b/openviking/server/mcp_endpoint.py
@@ -439,6 +439,39 @@ def _resolve_public_base_url() -> tuple[str, str]:
return "http://127.0.0.1:1933", "listen"
+async def _maybe_sitemap_hint(path: str) -> str:
+ """Best-effort sitemap/RSS suggestion for a single-page add (never crawls).
+
+ Policy: only suggest when the added URL is the site root / homepage (path is
+ empty or "/"). Adding a deep article URL implies intent for just that page, and
+ gating to the root also keeps the (bounded, non-recursive) probe from re-running
+ when many pages of the same site are added one-by-one.
+
+ Gated by config (``webfeed.suggest_feed``) and fully exception-safe so it can
+ never block or fail the add it is attached to.
+ """
+ try:
+ from urllib.parse import urlparse
+
+ if urlparse(path).path not in ("", "/"):
+ return ""
+
+ from openviking.parse.accessors import discover_feed_hint
+ from openviking.utils.network_guard import ensure_public_remote_target
+ from openviking_cli.utils.config import get_openviking_config
+
+ webfeed = getattr(get_openviking_config(), "webfeed", None)
+ if webfeed is not None and not getattr(webfeed, "suggest_feed", True):
+ return ""
+ timeout = float(getattr(webfeed, "suggest_timeout", 2.5)) if webfeed else 2.5
+ hint = await discover_feed_hint(
+ path, timeout=timeout, request_validator=ensure_public_remote_target
+ )
+ return hint or ""
+ except Exception:
+ return ""
+
+
@mcp.tool()
async def add_resource(
path: str = "",
@@ -456,6 +489,10 @@ async def add_resource(
Returns a success message immediately. Supports ``watch_interval`` for
auto-refresh subscriptions; pass ``to`` to choose the exact target URI, or
omit it to bind the watch to the URI created by this add.
+ A sitemap / RSS / Atom URL ingests the WHOLE site as one resource tree;
+ pass ``args={"site": true}`` to force whole-site ingestion from a bare
+ domain (auto-discovers the site's sitemap/RSS). Watching a sitemap/feed
+ URL keeps the whole site auto-refreshed.
2. Local file: pass ``path`` set to a local filesystem path (e.g. ``/tmp/foo.pdf``).
The response is NOT a success message — it's a multi-step upload instruction.
@@ -562,11 +599,18 @@ async def add_resource(
watch_suffix = f" (watch enabled, refresh every {watch_interval:g} minute(s))"
else:
watch_suffix = ""
- return (
+ message = (
f"Resource added: {root_uri}{watch_suffix}"
if root_uri
else f"Resource added (processing in background){watch_suffix}."
)
+ # Detect-and-suggest: if this single page belongs to a site that exposes a
+ # sitemap/RSS feed, hint at whole-site ingestion. Never auto-crawls; the
+ # add above is already done, so a slow/failed probe has no functional impact.
+ hint = await _maybe_sitemap_hint(path)
+ if hint:
+ message += "\n" + hint
+ return message
# Branch 4: local path — mint token, return upload instruction
server_config = get_server_config()
diff --git a/openviking/sync_client.py b/openviking/sync_client.py
index 1c2d1b6da..af31adb36 100644
--- a/openviking/sync_client.py
+++ b/openviking/sync_client.py
@@ -210,6 +210,11 @@ class SyncOpenViking:
) -> Dict[str, Any]:
"""Add resource to OpenViking (resources scope only)
+ A sitemap / RSS / Atom URL ingests the whole site as one resource tree;
+ pass ``args={"site": True}`` to force whole-site ingestion from a bare
+ domain. A ``watch_interval`` on a sitemap/feed URL keeps the whole site
+ refreshed.
+
Args:
to: Exact target URI. Existing targets keep the add_resource incremental-update behavior.
parent: Target parent URI for automatic child naming.
diff --git a/openviking_cli/utils/config/open_viking_config.py b/openviking_cli/utils/config/open_viking_config.py
index 85e191c6f..8b6a4b970 100644
--- a/openviking_cli/utils/config/open_viking_config.py
+++ b/openviking_cli/utils/config/open_viking_config.py
@@ -21,8 +21,8 @@ from .consts import (
)
from .embedding_config import EmbeddingConfig
from .encryption_config import EncryptionConfig
-from .grep_config import GrepConfig
from .git_config import GitConfig
+from .grep_config import GrepConfig
from .log_config import LogConfig
from .memory_config import MemoryConfig
from .oauth_config import OAuthConfig
@@ -38,6 +38,7 @@ from .parser_config import (
SemanticConfig,
TextConfig,
VideoConfig,
+ WebFeedConfig,
)
from .prompts_config import PromptsConfig
from .rerank_config import RerankConfig
@@ -185,6 +186,11 @@ class OpenVikingConfig(BaseModel):
description="Feishu/Lark document parsing configuration",
)
+ webfeed: WebFeedConfig = Field(
+ default_factory=WebFeedConfig,
+ description="Whole-site ingestion via sitemap / RSS / Atom feeds",
+ )
+
semantic: SemanticConfig = Field(
default_factory=SemanticConfig,
description="Semantic processing configuration (overview/abstract limits)",
@@ -335,6 +341,7 @@ class OpenVikingConfig(BaseModel):
"text",
"directory",
"feishu",
+ "webfeed",
]
raise_unknown_config_fields(
data=config_copy,
diff --git a/openviking_cli/utils/config/parser_config.py b/openviking_cli/utils/config/parser_config.py
index d48bddafa..2aa84c405 100644
--- a/openviking_cli/utils/config/parser_config.py
+++ b/openviking_cli/utils/config/parser_config.py
@@ -551,6 +551,56 @@ class DirectoryConfig(ParserConfig):
preserve_structure: bool = True
+@dataclass
+class WebFeedConfig(ParserConfig):
+ """
+ Configuration for whole-site ingestion via sitemap / RSS / Atom feeds.
+
+ Used by WebFeedAccessor (and its single-page detect-and-suggest helper).
+ Each setting can be overridden per call via add_resource ``args`` (e.g.
+ ``args={"max_pages": 50}``).
+
+ Attributes:
+ max_pages: Hard cap on the number of pages mirrored per site.
+ max_concurrency: Max concurrent page fetches.
+ request_timeout: Per-request timeout in seconds.
+ politeness_delay: Delay (seconds) before each page fetch, to be polite.
+ same_host_only: Only ingest URLs on the same host as the feed.
+ respect_robots: Honor robots.txt Disallow rules (and discover sitemaps).
+ max_depth: Max recursion depth when following <sitemapindex> entries.
+ suggest_feed: When adding a single webpage, probe for a sitemap/RSS and
+ append a one-line hint suggesting whole-site ingestion (never auto-crawls).
+ suggest_timeout: Hard timeout (seconds) for that single-page probe.
+ """
+
+ max_pages: int = 200
+ max_concurrency: int = 5
+ request_timeout: float = 30.0
+ politeness_delay: float = 0.2
+ same_host_only: bool = True
+ respect_robots: bool = True
+ max_depth: int = 2
+ suggest_feed: bool = True
+ suggest_timeout: float = 2.5
+
+ def validate(self) -> None:
+ """Validate web feed configuration."""
+ super().validate()
+
+ if self.max_pages <= 0:
+ raise ValueError("max_pages must be positive")
+ if self.max_concurrency <= 0:
+ raise ValueError("max_concurrency must be positive")
+ if self.request_timeout <= 0:
+ raise ValueError("request_timeout must be positive")
+ if self.politeness_delay < 0:
+ raise ValueError("politeness_delay must be non-negative")
+ if self.max_depth <= 0:
+ raise ValueError("max_depth must be positive")
+ if self.suggest_timeout <= 0:
+ raise ValueError("suggest_timeout must be positive")
+
+
@dataclass
class SemanticConfig:
"""
@@ -599,6 +649,7 @@ PARSER_CONFIG_REGISTRY = {
"text": TextConfig,
"directory": DirectoryConfig,
"feishu": FeishuConfig,
+ "webfeed": WebFeedConfig,
}
diff --git a/pyproject.toml b/pyproject.toml
index 59922b747..8b2fdca8f 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -38,6 +38,8 @@ dependencies = [
"pdfplumber>=0.10.0",
"readabilipy>=0.2.0",
"markdownify>=0.11.0",
+ "feedparser>=6.0.0",
+ "defusedxml>=0.7.1",
"openai>=1.0.0",
"requests>=2.31.0",
"charset-normalizer>=3.4,<4",
diff --git a/tests/unit/test_accessors_web_feed.py b/tests/unit/test_accessors_web_feed.py
new file mode 100644
index 000000000..861f6817c
--- /dev/null
+++ b/tests/unit/test_accessors_web_feed.py
@@ -0,0 +1,563 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Unit tests for WebFeedAccessor (whole-site ingestion via sitemap / RSS / Atom)."""
+
+import pytest
+
+from openviking.parse.accessors import (
+ AccessorRegistry,
+ GitAccessor,
+ HTTPAccessor,
+ WebFeedAccessor,
+)
+from openviking.parse.accessors.web_feed_accessor import (
+ FeedEntry,
+ _classify_and_count,
+ _entry_to_relpath,
+ _extract_feed_links,
+ discover_feed_hint,
+ looks_like_feed_url,
+ url_to_relpath,
+)
+
+SITEMAP_NS = "http://www.sitemaps.org/schemas/sitemap/0.9"
+
+
+# --------------------------------------------------------------------------- #
+# Fake httpx client (dependency-free; no real network)
+# --------------------------------------------------------------------------- #
+class _FakeStatusError(Exception):
+ pass
+
+
+class _FakeResponse:
+ def __init__(self, status_code=200, content=b"", content_type="text/html"):
+ self.status_code = status_code
+ self.content = content if isinstance(content, bytes) else content.encode("utf-8")
+ self.headers = {"content-type": content_type}
+
+ @property
+ def text(self):
+ return self.content.decode("utf-8", "replace")
+
+ def raise_for_status(self):
+ if self.status_code >= 400:
+ raise _FakeStatusError(f"status {self.status_code}")
+
+
+def _make_fake_client(routes):
+ """routes: dict url -> _FakeResponse (missing urls -> 404)."""
+
+ class _FakeClient:
+ def __init__(self, **kwargs):
+ self.kwargs = kwargs
+
+ async def __aenter__(self):
+ return self
+
+ async def __aexit__(self, *exc):
+ return False
+
+ async def get(self, url, headers=None):
+ resp = routes.get(url)
+ if resp is None:
+ return _FakeResponse(404, b"not found", "text/plain")
+ return resp
+
+ return _FakeClient
+
+
+def patch_httpx(monkeypatch):
+ """Patch httpx.AsyncClient with a fake serving the given routes."""
+
+ def _apply(routes):
+ import httpx
+
+ monkeypatch.setattr(httpx, "AsyncClient", _make_fake_client(routes))
+
+ return _apply
+
+
+def _urlset(urls_with_lastmod):
+ items = "".join(
+ f"<url><loc>{u}</loc>" + (f"<lastmod>{lm}</lastmod>" if lm else "") + "</url>"
+ for u, lm in urls_with_lastmod
+ )
+ return f'<?xml version="1.0"?><urlset xmlns="{SITEMAP_NS}">{items}</urlset>'.encode()
+
+
+def _sitemapindex(sub_urls):
+ items = "".join(f"<sitemap><loc>{u}</loc></sitemap>" for u in sub_urls)
+ return (
+ f'<?xml version="1.0"?><sitemapindex xmlns="{SITEMAP_NS}">{items}</sitemapindex>'.encode()
+ )
+
+
+def _page(title):
+ return f"<html><head><title>{title}</title></head><body><h1>{title}</h1><p>body</p></body></html>".encode()
+
+
+# --------------------------------------------------------------------------- #
+# can_handle
+# --------------------------------------------------------------------------- #
+class TestCanHandle:
+ @pytest.fixture
+ def accessor(self):
+ return WebFeedAccessor()
+
+ def test_priority(self, accessor):
+ assert accessor.priority == 60
+
+ @pytest.mark.parametrize(
+ "url",
+ [
+ "https://blog.openviking.ai/sitemap.xml",
+ "https://t0saki.com/sitemap-index.xml",
+ "https://t0saki.com/sitemap_index.xml",
+ "https://t0saki.com/sitemap-0.xml",
+ "https://t0saki.com/rss.xml",
+ "https://example.com/feed.xml",
+ "https://example.com/atom.xml",
+ "https://example.com/index.xml",
+ "https://example.com/feed",
+ "https://example.com/blog/feed/",
+ ],
+ )
+ def test_handles_feed_like(self, accessor, url):
+ assert accessor.can_handle(url) is True
+
+ @pytest.mark.parametrize(
+ "url",
+ [
+ "https://example.com/post/foo/",
+ "https://example.com/data.xml",
+ "https://example.com/page.html",
+ "/tmp/sitemap.xml",
+ "[email protected]:org/repo.git",
+ ],
+ )
+ def test_rejects_non_feed(self, accessor, url):
+ assert accessor.can_handle(url) is False
+
+ def test_explicit_override(self, accessor):
+ assert accessor.can_handle("https://example.com/post/foo/", site=True) is True
+ assert accessor.can_handle("https://example.com/sitemap.xml", site=False) is False
+ # alias keys
+ assert accessor.can_handle("https://example.com/x", sitemap=True) is True
+ assert accessor.can_handle("https://example.com/x", feed=True) is True
+
+
+# --------------------------------------------------------------------------- #
+# registry routing
+# --------------------------------------------------------------------------- #
+class TestRegistryRouting:
+ @pytest.fixture
+ def registry(self):
+ return AccessorRegistry()
+
+ def test_sitemap_routes_to_web_feed(self, registry):
+ acc = registry.get_accessor("https://blog.openviking.ai/sitemap.xml")
+ assert isinstance(acc, WebFeedAccessor)
+
+ def test_git_url_still_wins(self, registry):
+ acc = registry.get_accessor("https://github.com/volcengine/OpenViking")
+ assert isinstance(acc, GitAccessor)
+
+ def test_plain_page_routes_to_http(self, registry):
+ acc = registry.get_accessor("https://example.com/article")
+ assert isinstance(acc, HTTPAccessor)
+
+ def test_override_routes_plain_page_to_web_feed(self, registry):
+ acc = registry.get_accessor("https://example.com/article", site=True)
+ assert isinstance(acc, WebFeedAccessor)
+
+
+# --------------------------------------------------------------------------- #
+# url -> relpath
+# --------------------------------------------------------------------------- #
+class TestRelpath:
+ @pytest.mark.parametrize(
+ "url,expected",
+ [
+ ("https://h/", "index.html"),
+ ("https://h/a/b", "a/b.html"),
+ ("https://h/post/foo/", "post/foo.html"),
+ ("https://h/post/foo/index.html", "post/foo/index.html"),
+ ("https://h/about.php", "about.html"),
+ ],
+ )
+ def test_url_to_relpath(self, url, expected):
+ assert url_to_relpath(url) == expected
+
+ def test_entry_title_used_when_present(self):
+ assert (
+ _entry_to_relpath(FeedEntry(url="https://h/x", title="Hello World"))
+ == "Hello World.html"
+ )
+
+ def test_entry_title_cjk(self):
+ assert _entry_to_relpath(FeedEntry(url="https://h/x", title="台湾游记")) == "台湾游记.html"
+
+ def test_entry_no_title_falls_back_to_url(self):
+ assert _entry_to_relpath(FeedEntry(url="https://h/post/bar/")) == "post/bar.html"
+
+ def test_title_with_slash_is_sanitized(self):
+ rel = _entry_to_relpath(FeedEntry(url="https://h/x", title="a/b: c"))
+ assert "/" not in rel[:-5] # no path separator in the stem
+
+
+# --------------------------------------------------------------------------- #
+# classify-and-count (used by the single-page hint)
+# --------------------------------------------------------------------------- #
+class TestClassify:
+ def test_urlset(self):
+ content = _urlset([("https://h/a", None), ("https://h/b", "2026-01-01")])
+ assert _classify_and_count(content) == ("sitemap", 2)
+
+ def test_sitemapindex(self):
+ assert _classify_and_count(_sitemapindex(["https://h/s0.xml"])) == ("sitemapindex", 1)
+
+ def test_rss(self):
+ rss = b'<?xml version="1.0"?><rss version="2.0"><channel><item><link>https://h/a</link></item></channel></rss>'
+ assert _classify_and_count(rss) == ("rss", 1)
+
+ def test_garbage_returns_none(self):
+ assert _classify_and_count(b"not xml at all") is None
+
+ def test_extract_feed_links(self):
+ html = (
+ '<link rel="alternate" type="application/rss+xml" href="/rss.xml">'
+ '<link rel="alternate" type="application/atom+xml" href="https://h/atom.xml">'
+ '<link rel="sitemap" href="/sitemap.xml">'
+ '<link rel="stylesheet" href="/x.css">'
+ )
+ links = _extract_feed_links(html, "https://h/page")
+ assert "https://h/rss.xml" in links
+ assert "https://h/atom.xml" in links
+ assert "https://h/sitemap.xml" in links
+ assert all("x.css" not in link for link in links)
+
+
+# --------------------------------------------------------------------------- #
+# access() — sitemap mirroring
+# --------------------------------------------------------------------------- #
+class TestAccessSitemap:
+ async def test_flat_urlset_mirrors_pages(self, patch_httpx):
+ base = "https://blog.example.com"
+ page_urls = [f"{base}/post/p{i}/" for i in range(10)]
+ routes = {
+ f"{base}/sitemap.xml": _FakeResponse(content=_urlset([(u, None) for u in page_urls]))
+ }
+ for i, u in enumerate(page_urls):
+ routes[u] = _FakeResponse(content=_page(f"Post {i}"))
+ patch_httpx(routes)
+
+ acc = WebFeedAccessor()
+ resource = await acc.access(f"{base}/sitemap.xml", respect_robots=False, politeness_delay=0)
+ try:
+ assert resource.is_temporary is True
+ assert resource.path.is_dir()
+ assert resource.meta["original_filename"] == "blog.example.com"
+ assert resource.meta["page_count"] == 10
+ assert resource.meta["url_type"] == "sitemap"
+ html_files = list(resource.path.rglob("*.html"))
+ assert len(html_files) == 10
+ # mirrors URL path structure (post/p0.html ...)
+ assert (resource.path / "post" / "p0.html").exists()
+ finally:
+ resource.cleanup()
+
+ async def test_failed_page_is_skipped(self, patch_httpx):
+ base = "https://blog.example.com"
+ page_urls = [f"{base}/post/p{i}/" for i in range(5)]
+ routes = {
+ f"{base}/sitemap.xml": _FakeResponse(content=_urlset([(u, None) for u in page_urls]))
+ }
+ for i, u in enumerate(page_urls):
+ routes[u] = _FakeResponse(content=_page(f"Post {i}"))
+ routes[page_urls[2]] = _FakeResponse(500, b"err", "text/plain") # one fails
+ patch_httpx(routes)
+
+ acc = WebFeedAccessor()
+ resource = await acc.access(f"{base}/sitemap.xml", respect_robots=False, politeness_delay=0)
+ try:
+ assert resource.meta["page_count"] == 4
+ finally:
+ resource.cleanup()
+
+ async def test_sitemapindex_recursion(self, patch_httpx):
+ base = "https://t0saki.com"
+ page_urls = [f"{base}/posts/p{i}/" for i in range(3)]
+ routes = {
+ f"{base}/sitemap-index.xml": _FakeResponse(
+ content=_sitemapindex([f"{base}/sitemap-0.xml"])
+ ),
+ f"{base}/sitemap-0.xml": _FakeResponse(content=_urlset([(u, None) for u in page_urls])),
+ }
+ for i, u in enumerate(page_urls):
+ routes[u] = _FakeResponse(content=_page(f"P{i}"))
+ patch_httpx(routes)
+
+ acc = WebFeedAccessor()
+ resource = await acc.access(
+ f"{base}/sitemap-index.xml", respect_robots=False, politeness_delay=0
+ )
+ try:
+ assert resource.meta["page_count"] == 3
+ assert resource.meta["url_type"] == "sitemap"
+ finally:
+ resource.cleanup()
+
+ async def test_max_pages_truncates(self, patch_httpx):
+ base = "https://blog.example.com"
+ page_urls = [f"{base}/p{i}/" for i in range(10)]
+ routes = {
+ f"{base}/sitemap.xml": _FakeResponse(content=_urlset([(u, None) for u in page_urls]))
+ }
+ for u in page_urls:
+ routes[u] = _FakeResponse(content=_page("x"))
+ patch_httpx(routes)
+
+ acc = WebFeedAccessor()
+ resource = await acc.access(
+ f"{base}/sitemap.xml", respect_robots=False, politeness_delay=0, max_pages=3
+ )
+ try:
+ assert resource.meta["page_count"] == 3
+ finally:
+ resource.cleanup()
+
+ async def test_same_host_only_filters_external(self, patch_httpx):
+ base = "https://blog.example.com"
+ internal = f"{base}/keep/"
+ external = "https://other.com/drop/"
+ routes = {
+ f"{base}/sitemap.xml": _FakeResponse(
+ content=_urlset([(internal, None), (external, None)])
+ ),
+ internal: _FakeResponse(content=_page("keep")),
+ external: _FakeResponse(content=_page("drop")),
+ }
+ patch_httpx(routes)
+
+ acc = WebFeedAccessor()
+ resource = await acc.access(f"{base}/sitemap.xml", respect_robots=False, politeness_delay=0)
+ try:
+ assert resource.meta["page_count"] == 1
+ assert internal in resource.meta["pages"]
+ assert external not in resource.meta["pages"]
+ finally:
+ resource.cleanup()
+
+
+# --------------------------------------------------------------------------- #
+# access() — auto-discovery from a bare domain / HTML page (args={"site": True})
+# --------------------------------------------------------------------------- #
+class TestAutoDiscovery:
+ async def test_bare_domain_discovers_sitemap_via_robots(self, patch_httpx):
+ base = "https://t0saki.example"
+ page_urls = [f"{base}/posts/p{i}/" for i in range(3)]
+ routes = {
+ base: _FakeResponse(content=b"<html><body>home</body></html>"),
+ f"{base}/robots.txt": _FakeResponse(
+ content=f"User-agent: *\nSitemap: {base}/sitemap.xml\n", content_type="text/plain"
+ ),
+ f"{base}/sitemap.xml": _FakeResponse(content=_urlset([(u, None) for u in page_urls])),
+ }
+ for i, u in enumerate(page_urls):
+ routes[u] = _FakeResponse(content=_page(f"P{i}"))
+ patch_httpx(routes)
+
+ acc = WebFeedAccessor()
+ # access() simulates the args={"site": True} routing (can_handle override).
+ resource = await acc.access(base, respect_robots=False, politeness_delay=0)
+ try:
+ assert resource.meta["page_count"] == 3
+ assert resource.meta["feed_url"] == f"{base}/sitemap.xml"
+ assert resource.meta["original_filename"] == "t0saki.example"
+ finally:
+ resource.cleanup()
+
+ async def test_html_page_discovers_feed_via_link_tag(self, patch_httpx):
+ base = "https://blog.example"
+ page_html = (
+ "<html><head>"
+ f'<link rel="alternate" type="application/rss+xml" href="{base}/rss.xml">'
+ "</head><body>post</body></html>"
+ ).encode()
+ rss = (
+ '<?xml version="1.0"?><rss version="2.0"><channel>'
+ f"<item><title>One</title><link>{base}/a/</link></item>"
+ "</channel></rss>"
+ ).encode()
+ routes = {
+ f"{base}/some-article": _FakeResponse(content=page_html),
+ f"{base}/rss.xml": _FakeResponse(content=rss, content_type="application/rss+xml"),
+ f"{base}/a/": _FakeResponse(content=_page("A")),
+ }
+ patch_httpx(routes)
+
+ acc = WebFeedAccessor()
+ resource = await acc.access(
+ f"{base}/some-article", respect_robots=False, politeness_delay=0
+ )
+ try:
+ assert resource.meta["url_type"] == "rss"
+ assert resource.meta["feed_url"] == f"{base}/rss.xml"
+ finally:
+ resource.cleanup()
+
+ async def test_no_feed_discoverable_raises(self, patch_httpx):
+ base = "https://nofeed.example"
+ routes = {base: _FakeResponse(content=b"<html><body>nothing</body></html>")}
+ patch_httpx(routes)
+ acc = WebFeedAccessor()
+ with pytest.raises(RuntimeError, match="auto-discovered"):
+ await acc.access(base, respect_robots=False, politeness_delay=0)
+
+
+# --------------------------------------------------------------------------- #
+# access() — RSS / Atom
+# --------------------------------------------------------------------------- #
+class TestAccessFeed:
+ async def test_rss_summary_only_fetches_links(self, patch_httpx):
+ base = "https://t0saki.com"
+ items = [(f"{base}/posts/a/", "Post A"), (f"{base}/posts/b/", "Post B")]
+ rss = (
+ '<?xml version="1.0"?><rss version="2.0"><channel>'
+ + "".join(
+ f"<item><title>{t}</title><link>{u}</link><description>summary</description></item>"
+ for u, t in items
+ )
+ + "</channel></rss>"
+ ).encode()
+ routes = {f"{base}/rss.xml": _FakeResponse(content=rss, content_type="application/rss+xml")}
+ for u, _ in items:
+ routes[u] = _FakeResponse(content=_page("full"))
+ patch_httpx(routes)
+
+ acc = WebFeedAccessor()
+ resource = await acc.access(f"{base}/rss.xml", respect_robots=False, politeness_delay=0)
+ try:
+ assert resource.meta["url_type"] == "rss"
+ assert resource.meta["page_count"] == 2
+ # titles used as flat filenames
+ assert (resource.path / "Post A.html").exists()
+ assert (resource.path / "Post B.html").exists()
+ finally:
+ resource.cleanup()
+
+ async def test_rss_inline_content_is_used_without_fetch(self, patch_httpx):
+ base = "https://t0saki.com"
+ rss = (
+ '<?xml version="1.0"?>'
+ '<rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/">'
+ "<channel><item><title>Inline</title><link>https://t0saki.com/posts/inline/</link>"
+ "<content:encoded><![CDATA[<p>FULL INLINE BODY</p>]]></content:encoded>"
+ "</item></channel></rss>"
+ ).encode()
+ # Note: deliberately DO NOT register the article URL — if access tried to
+ # fetch it we'd get a 404 and page_count would drop to 0.
+ routes = {f"{base}/rss.xml": _FakeResponse(content=rss, content_type="application/rss+xml")}
+ patch_httpx(routes)
+
+ acc = WebFeedAccessor()
+ resource = await acc.access(f"{base}/rss.xml", respect_robots=False, politeness_delay=0)
+ try:
+ assert resource.meta["page_count"] == 1
+ saved = (resource.path / "Inline.html").read_text(encoding="utf-8")
+ assert "FULL INLINE BODY" in saved
+ finally:
+ resource.cleanup()
+
+ async def test_atom_feed(self, patch_httpx):
+ base = "https://example.org"
+ atom = (
+ '<?xml version="1.0"?><feed xmlns="http://www.w3.org/2005/Atom">'
+ '<entry><title>Entry One</title><link href="https://example.org/e1/"/></entry>'
+ "</feed>"
+ ).encode()
+ routes = {
+ f"{base}/atom.xml": _FakeResponse(content=atom, content_type="application/atom+xml"),
+ f"{base}/e1/": _FakeResponse(content=_page("E1")),
+ }
+ patch_httpx(routes)
+
+ acc = WebFeedAccessor()
+ resource = await acc.access(f"{base}/atom.xml", respect_robots=False, politeness_delay=0)
+ try:
+ assert resource.meta["url_type"] == "atom"
+ assert resource.meta["page_count"] == 1
+ finally:
+ resource.cleanup()
+
+
+# --------------------------------------------------------------------------- #
+# discover_feed_hint (single-page detect-and-suggest)
+# --------------------------------------------------------------------------- #
+class TestDiscoverHint:
+ async def test_hint_from_robots_sitemap(self, patch_httpx):
+ base = "https://blog.example.com"
+ routes = {
+ f"{base}/robots.txt": _FakeResponse(
+ content=f"User-agent: *\nSitemap: {base}/sitemap.xml\n", content_type="text/plain"
+ ),
+ f"{base}/sitemap.xml": _FakeResponse(
+ content=_urlset([(f"{base}/p{i}/", None) for i in range(3)])
+ ),
+ f"{base}/article": _FakeResponse(content=_page("article")),
+ }
+ patch_httpx(routes)
+
+ hint = await discover_feed_hint(f"{base}/article", timeout=5)
+ assert hint is not None
+ assert "3 page(s)" in hint
+ assert f"{base}/sitemap.xml" in hint
+
+ async def test_hint_from_html_autodiscovery(self, patch_httpx):
+ base = "https://blog.example.com"
+ page_html = (
+ '<html><head><link rel="alternate" type="application/rss+xml" '
+ f'href="{base}/rss.xml"></head><body>hi</body></html>'
+ ).encode()
+ rss = (
+ '<?xml version="1.0"?><rss version="2.0"><channel>'
+ "<item><link>https://blog.example.com/a/</link></item>"
+ "<item><link>https://blog.example.com/b/</link></item>"
+ "</channel></rss>"
+ ).encode()
+ routes = {
+ f"{base}/article": _FakeResponse(content=page_html),
+ f"{base}/rss.xml": _FakeResponse(content=rss, content_type="application/rss+xml"),
+ # robots.txt missing -> 404, fine
+ }
+ patch_httpx(routes)
+
+ hint = await discover_feed_hint(f"{base}/article", timeout=5)
+ assert hint is not None
+ assert "RSS feed with 2 item(s)" in hint
+
+ async def test_no_hint_when_nothing_found(self, patch_httpx):
+ base = "https://nofeed.example.com"
+ routes = {f"{base}/article": _FakeResponse(content=_page("article"))}
+ patch_httpx(routes)
+ assert await discover_feed_hint(f"{base}/article", timeout=5) is None
+
+ async def test_no_hint_for_feed_url_itself(self, patch_httpx):
+ patch_httpx({})
+ assert await discover_feed_hint("https://x.com/sitemap.xml", timeout=5) is None
+
+ async def test_no_hint_for_non_html_extension(self, patch_httpx):
+ patch_httpx({})
+ assert await discover_feed_hint("https://x.com/file.pdf", timeout=5) is None
+
+
+# --------------------------------------------------------------------------- #
+# looks_like_feed_url (standalone)
+# --------------------------------------------------------------------------- #
+def test_looks_like_feed_url():
+ assert looks_like_feed_url("https://h/sitemap.xml")
+ assert looks_like_feed_url("https://h/rss.xml")
+ assert not looks_like_feed_url("https://h/post")
+ assert not looks_like_feed_url("/tmp/sitemap.xml")