Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(resources): ingest whole sites via sitemap / RSS / Atom (#2858)!
t0saki2846bb6e76bd
diff --git a/crates/ov_cli/src/help_ui.rs b/crates/ov_cli/src/help_ui.rsindex 73c8363d0..a3738ee08 100644--- a/crates/ov_cli/src/help_ui.rs+++ b/crates/ov_cli/src/help_ui.rs@@ -125,7 +125,7 @@ const HELP_SECTIONS: &[HelpSection] = &[const COMMAND_HELP_SPECS: &[CommandHelpSpec] = &[CommandHelpSpec {path: &["add-resource"],- purpose: "Import a local file, folder, URL, or repository into OpenViking.",+ purpose: "Import a local file, folder, URL, repository, or whole website (sitemap/RSS) into OpenViking.",examples: &[HelpItem {label: "ov add-resource ./docs --parent viking://projects/acme --wait",@@ -135,6 +135,10 @@ const COMMAND_HELP_SPECS: &[CommandHelpSpec] = &[label: "ov add-resource https://example.com/spec.md --to viking://specs/api.md",description: "Import a URL to an exact target URI.",},+ HelpItem {+ label: "ov add-resource https://example.com/sitemap.xml --watch-interval 1440",+ description: "Import a whole site via sitemap/RSS and refresh it daily.",+ },],next_steps: &[HelpItem {diff --git a/docs/en/api/02-resources.md b/docs/en/api/02-resources.mdindex 127a42d8b..df5619095 100644--- a/docs/en/api/02-resources.md+++ b/docs/en/api/02-resources.md@@ -49,6 +49,16 @@ OpenViking supports various resource types, categorized by functionality:|------|-------------|| Feishu/Lark | URL-based, supports docx, wiki, sheets, bitable. By default uses app credentials from FEISHU_APP_ID and FEISHU_APP_SECRET; user-token imports can pass `args.feishu_access_token`, and user-token watches also pass `args.feishu_refresh_token` |+**Whole Website (sitemap / RSS / Atom)**++| Type | Resource Name | Description |+|------|---------------|-------------|+| Sitemap | `https://host/sitemap.xml`, `https://host/sitemap-index.xml` | Parses the sitemap and ingests every listed page as a single resource tree (one child node per page). Nested `<sitemapindex>` is followed recursively. The whole site becomes one resource under `viking://resources/<host>`. |+| RSS / Atom feed | `https://host/rss.xml`, `https://host/atom.xml`, `https://host/feed` | Parses RSS 2.0 / Atom and ingests each entry as a tree node; the article body is fetched from its link (or taken inline when the feed carries full content). |+| Whole-site auto-discovery | `https://host` + `args.site=true` | Forces whole-site ingestion for a bare domain or ordinary page: discovers the site's sitemap/RSS via robots.txt, HTML `<link rel="alternate">` autodiscovery, and conventional paths, then ingests it. |++Crawling is bounded and non-recursive beyond the listed pages, and is governed by the `parsers.webfeed` config (`max_pages`, `max_concurrency`, `politeness_delay`, `same_host_only`, `respect_robots`, `max_depth`); robots.txt is honored. Set `watch_interval` on a sitemap/feed URL to keep the **whole site** refreshed: on each run new pages are added and removed pages drop automatically. When adding a single homepage (without `args.site`), the response may append a one-line hint suggesting whole-site ingestion — it never auto-crawls.+### Resource Processing PipelineResources go through the following processing stages when added:@@ -95,6 +105,7 @@ Resource incremental updates are implemented via the **Watch Task** mechanism:#### Watch Task Creation- Set `watch_interval > 0` (in minutes) when calling `add_resource` to create a watch task- You may specify `to` to define the target URI; if omitted, the task binds to the `root_uri` returned by this import+- Pointing a watch at a sitemap/RSS/Atom URL keeps the **whole site** in sync: each refresh re-reads the feed and rebuilds the tree, so newly published pages are added and removed pages drop automatically- `WatchManager` handles task persistence- Supports multi-tenant permission control (ROOT/ADMIN/USER permission levels)@@ -157,7 +168,7 @@ This endpoint is the core entry point for resource management, supporting adding| exclude | string | No | None | File patterns to exclude (glob) || directly_upload_media | bool | No | True | Whether to directly upload media files || preserve_structure | bool | No | None | Whether to preserve directory structure |-| args | object | No | `{}` | Parser-specific import options forwarded to the source parser/accessor. Core `add_resource` fields such as `path`, `to`, `watch_interval`, `include`, and `exclude` are not allowed inside `args` |+| args | object | No | `{}` | Parser-specific import options forwarded to the source parser/accessor. E.g. `args.site=true/false` forces/opts out of whole-site (sitemap/RSS) ingestion, `args.max_pages` etc. override the `webfeed` config, and Feishu user-token imports pass `args.feishu_access_token`. Core `add_resource` fields such as `path`, `to`, `watch_interval`, `include`, and `exclude` are not allowed inside `args` || watch_interval | float | No | 0 | Scheduled update interval (minutes). >0 creates task; <=0 cancels task; explicit `to` wins, otherwise binds to the imported `root_uri` || telemetry | TelemetryRequest | No | False | Whether to return telemetry data |diff --git a/docs/zh/api/02-resources.md b/docs/zh/api/02-resources.mdindex a4f5ec440..b8aec9c4e 100644--- a/docs/zh/api/02-resources.md+++ b/docs/zh/api/02-resources.md@@ -44,6 +44,15 @@ OpenViking 支持多种资源类型,按照功能分类如下:|------|------|| 飞书/Lark | URL 方式,支持 docx, wiki, sheets, bitable。默认使用 FEISHU_APP_ID 和 FEISHU_APP_SECRET 应用凭证;用户 token 导入可传 `args.feishu_access_token`,用户 token watch 还需传 `args.feishu_refresh_token` |+网站类(sitemap / RSS / Atom 整站导入)+| 类型 | 资源名 | 说明 |+|------|--------|------|+| 站点地图 Sitemap | `https://host/sitemap.xml`、`https://host/sitemap-index.xml` | 解析 sitemap,将站点所有页面抓取为**一棵资源树**(每页一个子节点),支持嵌套 `<sitemapindex>` 递归。整站只生成一个资源,落在 `viking://resources/<host>`。 |+| RSS / Atom 订阅源 | `https://host/rss.xml`、`https://host/atom.xml`、`https://host/feed` | 解析 RSS 2.0 / Atom,逐条把文章正文抓成树节点(feed 内含全文则直接使用,省一次抓取)。 |+| 整站自动发现 | `https://host` + `args.site=true` | 对裸域名/普通页面强制整站导入:自动通过 robots.txt、HTML `<link rel="alternate">` autodiscovery、常见路径发现 sitemap/RSS,再整站抓取。 |++抓取**有界、非递归**(不会超出所列页面继续爬),受 `parsers.webfeed` 配置约束(`max_pages`、`max_concurrency`、`politeness_delay`、`same_host_only`、`respect_robots`、`max_depth`),并遵守 robots.txt。对 sitemap/feed URL 设置 `watch_interval` 即可让**整站**周期刷新:每次运行自动纳入新增页面、移除已删除页面。添加单个首页(未带 `args.site`)时,返回信息可能附带一行"整站导入"提示——**只提示,绝不自动爬全站**。+### 资源处理流程资源添加经过以下处理阶段:@@ -90,6 +99,7 @@ URL/文件 Parser TreeBuilder AGFS Summarizer/Vector#### 监控任务创建- 调用 `add_resource` 时设置 `watch_interval > 0` (单位:分钟)创建监控任务- 可指定 `to` 参数确定目标 URI;未指定时,系统会使用本次导入返回的 `root_uri` 作为监控目标+- 把监控对象设为 sitemap/RSS/Atom URL,即可让**整站**保持同步:每次刷新重新读取 feed 并重建资源树,新发布的页面自动入库、已删除的页面自动移除- `WatchManager` 负责任务持久化存储- 支持多租户权限控制(ROOT/ADMIN/USER 权限分级)@@ -152,7 +162,7 @@ URL/文件 Parser TreeBuilder AGFS Summarizer/Vector| exclude | string | 否 | None | 排除的文件模式(glob) || directly_upload_media | bool | 否 | True | 是否直接上传媒体文件 || preserve_structure | bool | 否 | None | 是否保留目录结构 |-| args | object | 否 | `{}` | 传给特定 parser/accessor 的导入参数。`path`、`to`、`watch_interval`、`include`、`exclude` 等 `add_resource` 核心字段不能放入 `args` |+| args | object | 否 | `{}` | 传给特定 parser/accessor 的导入参数。例如 `args.site=true/false` 强制/禁用整站(sitemap/RSS)导入,`args.max_pages` 等可覆盖 `webfeed` 配置,飞书用户 token 导入传 `args.feishu_access_token`。`path`、`to`、`watch_interval`、`include`、`exclude` 等 `add_resource` 核心字段不能放入 `args` || watch_interval | float | 否 | 0 | 定时更新间隔(分钟)。>0 创建任务;≤0 取消任务;显式 `to` 优先,否则绑定本次导入的 `root_uri` || telemetry | TelemetryRequest | 否 | False | 是否返回遥测数据 |diff --git a/examples/ov.conf.example b/examples/ov.conf.exampleindex 3ff148e3f..01294ecd4 100644--- a/examples/ov.conf.example+++ b/examples/ov.conf.example@@ -234,6 +234,16 @@"max_paragraph_length": 1000,"preserve_line_breaks": false,},+ "webfeed": {+ "max_pages": 200,+ "max_concurrency": 5,+ "request_timeout": 30.0,+ "politeness_delay": 0.2,+ "same_host_only": true,+ "respect_robots": true,+ "max_depth": 2,+ "suggest_feed": true,+ },},"encryption": {"enabled": true,diff --git a/openviking/async_client.py b/openviking/async_client.pyindex 9a81ce5f8..87a04f86a 100644--- a/openviking/async_client.py+++ b/openviking/async_client.py@@ -299,7 +299,9 @@ class AsyncOpenViking:Add a resource (file/URL) to OpenViking.Args:- path: Local file path or URL.+ path: Local file path or URL. A sitemap / RSS / Atom URL ingests the+ whole site as one resource tree; pass ``args={"site": True}`` to+ force whole-site ingestion from a bare domain.reason: Context/reason for adding this resource.instruction: Specific instruction for processing.wait: If True, wait for processing to complete.@@ -307,6 +309,9 @@ class AsyncOpenViking:parent: Target parent URI (must already exist).build_index: Whether to build vector index immediately (default: True).summarize: Whether to generate summary (default: False).+ watch_interval: Auto-refresh interval in minutes (>0 enables a watch).+ On a sitemap/feed URL this keeps the whole site refreshed.+ args: Parser/accessor-specific options (e.g. ``site``, ``max_pages``).telemetry: Whether to attach operation telemetry data to the result."""await self._ensure_initialized()diff --git a/openviking/parse/accessors/__init__.py b/openviking/parse/accessors/__init__.pyindex 6ae4fe2c5..6232573f1 100644--- a/openviking/parse/accessors/__init__.py+++ b/openviking/parse/accessors/__init__.py@@ -18,6 +18,7 @@ from .registry import (access,get_accessor_registry,)+from .web_feed_accessor import WebFeedAccessor, discover_feed_hint__all__ = [# Base classes@@ -32,4 +33,7 @@ __all__ = ["HTTPAccessor","FeishuAccessor","LocalAccessor",+ "WebFeedAccessor",+ # Helpers+ "discover_feed_hint",]diff --git a/openviking/parse/accessors/base.py b/openviking/parse/accessors/base.pyindex 8d5b4c90e..22feee6ec 100644--- a/openviking/parse/accessors/base.py+++ b/openviking/parse/accessors/base.py@@ -103,12 +103,15 @@ class DataAccessor(ABC):"""@abstractmethod- def can_handle(self, source: Union[str, Path]) -> bool:+ def can_handle(self, source: Union[str, Path], **kwargs) -> bool:"""Check if this accessor can handle the given source.Args:source: Source string (URL, path, etc.) or Path object+ **kwargs: Optional accessor-selection hints forwarded from+ ``access()`` (e.g. an explicit ``site=True`` override). Most+ accessors ignore these and decide purely from ``source``.Returns:True if this accessor can handle the sourcediff --git a/openviking/parse/accessors/feishu_accessor.py b/openviking/parse/accessors/feishu_accessor.pyindex bf1fc349b..16f561c02 100644--- a/openviking/parse/accessors/feishu_accessor.py+++ b/openviking/parse/accessors/feishu_accessor.py@@ -150,7 +150,7 @@ class FeishuAccessor(DataAccessor):def priority(self) -> int:return self.PRIORITY- def can_handle(self, source: Union[str, Path]) -> bool:+ def can_handle(self, source: Union[str, Path], **kwargs) -> bool:"""Check if this accessor can handle the source.diff --git a/openviking/parse/accessors/git_accessor.py b/openviking/parse/accessors/git_accessor.pyindex a4cc7154f..718f02258 100644--- a/openviking/parse/accessors/git_accessor.py+++ b/openviking/parse/accessors/git_accessor.py@@ -51,7 +51,7 @@ class GitAccessor(DataAccessor):def priority(self) -> int:return self.PRIORITY- def can_handle(self, source: Union[str, Path]) -> bool:+ def can_handle(self, source: Union[str, Path], **kwargs) -> bool:"""Check if this accessor can handle the source.diff --git a/openviking/parse/accessors/http_accessor.py b/openviking/parse/accessors/http_accessor.pyindex cb0a52dfd..3265d9c7e 100644--- a/openviking/parse/accessors/http_accessor.py+++ b/openviking/parse/accessors/http_accessor.py@@ -416,7 +416,7 @@ class HTTPAccessor(DataAccessor):def priority(self) -> int:return self.PRIORITY- def can_handle(self, source: Union[str, Path]) -> bool:+ def can_handle(self, source: Union[str, Path], **kwargs) -> bool:"""Check if this accessor can handle the source.diff --git a/openviking/parse/accessors/local_accessor.py b/openviking/parse/accessors/local_accessor.pyindex a5f71acd1..55c117fd6 100644--- a/openviking/parse/accessors/local_accessor.py+++ b/openviking/parse/accessors/local_accessor.py@@ -28,7 +28,7 @@ class LocalAccessor(DataAccessor):- Provides clear local source type metadata"""- def can_handle(self, source: Union[str, Path]) -> bool:+ def can_handle(self, source: Union[str, Path], **kwargs) -> bool:"""Check if this accessor can handle the source.diff --git a/openviking/parse/accessors/registry.py b/openviking/parse/accessors/registry.pyindex d4aed7526..f37d79e73 100644--- a/openviking/parse/accessors/registry.py+++ b/openviking/parse/accessors/registry.py@@ -6,8 +6,9 @@ Registry for Data Accessors.Manages DataAccessor registration and provides automatic source routing."""+import inspectfrom pathlib import Path-from typing import List, Optional, Union+from typing import Callable, List, Optional, Unionfrom openviking_cli.utils import get_logger@@ -16,6 +17,20 @@ from .base import DataAccessor, LocalResourcelogger = get_logger(__name__)+def _accepts_var_keyword(func: Callable) -> bool:+ """Whether ``func`` accepts arbitrary keyword arguments (**kwargs).++ Lets the registry forward accessor-selection hints to modern accessors+ (e.g. WebFeedAccessor's ``site=True`` override) while staying compatible+ with accessors whose ``can_handle`` only takes ``source``.+ """+ try:+ sig = inspect.signature(func)+ except (TypeError, ValueError):+ return True+ return any(p.kind == p.VAR_KEYWORD for p in sig.parameters.values())++class AccessorRegistry:"""Registry for data accessors.@@ -45,6 +60,14 @@ class AccessorRegistry:except Exception as e:logger.debug(f"[AccessorRegistry] Failed to register GitAccessor: {e}")+ # WebFeedAccessor - handles sitemap / RSS / Atom URLs (whole-site ingest)+ try:+ from .web_feed_accessor import WebFeedAccessor++ self.register(WebFeedAccessor())+ except Exception as e:+ logger.debug(f"[AccessorRegistry] Failed to register WebFeedAccessor: {e}")+# HTTPAccessor - handles HTTP/HTTPS URLstry:from .http_accessor import HTTPAccessor@@ -109,18 +132,24 @@ class AccessorRegistry:return Truereturn False- def get_accessor(self, source: Union[str, Path]) -> Optional[DataAccessor]:+ def get_accessor(self, source: Union[str, Path], **kwargs) -> Optional[DataAccessor]:"""Get the highest priority accessor that can handle the source.Args:source: Source string or Path to check+ **kwargs: Optional accessor-selection hints forwarded to+ ``can_handle`` (e.g. an explicit ``site=True`` override).Returns:DataAccessor instance or None if no accessor can handle the source"""for accessor in self._accessors:- if accessor.can_handle(source):+ if kwargs and _accepts_var_keyword(accessor.can_handle):+ handled = accessor.can_handle(source, **kwargs)+ else:+ handled = accessor.can_handle(source)+ if handled:return accessorreturn None@@ -151,8 +180,10 @@ class AccessorRegistry:"""source_str = str(source)- # Find an accessor - LocalAccessor should always be registered as fallback- accessor = self.get_accessor(source)+ # Find an accessor - LocalAccessor should always be registered as fallback.+ # Forward kwargs so accessors can honor explicit selection hints (e.g.+ # args={"site": True}) that aren't inferable from the URL alone.+ accessor = self.get_accessor(source, **kwargs)if accessor:logger.debug(f"[AccessorRegistry] Using accessor {accessor.__class__.__name__} for source: {source_str}"diff --git a/openviking/parse/accessors/web_feed_accessor.py b/openviking/parse/accessors/web_feed_accessor.pynew file mode 100644index 000000000..bcddf07ae--- /dev/null+++ b/openviking/parse/accessors/web_feed_accessor.py@@ -0,0 +1,832 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""+Web Feed Accessor.++Ingests an ENTIRE website in one shot from a single "collection" URL:+- XML sitemaps (``<urlset>``) and sitemap indexes (``<sitemapindex>``, recursed)+- RSS 2.0 feeds (``<rss><channel><item>``)+- Atom feeds (``<feed><entry>``)++The shared core capability is "a URL that enumerates many page URLs". Whatever+the source, this accessor mirrors the listed pages into a single local temp+directory and returns it as a directory ``LocalResource`` — exactly the contract+GitAccessor uses. The downstream DirectoryParser then builds ONE resource tree+(one ``viking://resources/<host>`` URI, one watchable node) with a child per page.++Because a watch re-runs ``add_resource(path=<feed url>)`` on each refresh, watching+a sitemap/feed URL keeps the whole site up to date: new pages appear and removed+pages drop automatically on the next rebuild.++This module also exposes ``discover_feed_hint``: a cheap, failure-tolerant probe+used when a user adds a single webpage, to *suggest* whole-site ingestion without+ever auto-crawling.+"""++import asyncio+import fnmatch+import hashlib+import html as html_lib+import re+import shutil+import tempfile+from dataclasses import dataclass+from pathlib import Path, PurePosixPath+from typing import Any, Dict, List, Optional, Tuple, Union+from urllib.parse import unquote, urljoin, urlparse+from urllib.robotparser import RobotFileParser++from openviking.parse.base import lazy_import+from openviking.utils.network_guard import build_httpx_request_validation_hooks+from openviking_cli.utils.logger import get_logger++from .base import DataAccessor, LocalResource, SourceType+from .http_accessor import HTTPAccessor++logger = get_logger(__name__)++_USER_AGENT = HTTPAccessor.DEFAULT_USER_AGENT++# kwargs keys that explicitly force / forbid whole-site ingestion (args={"site": ...}).+_OVERRIDE_KEYS = ("site", "sitemap", "feed", "as_site")++# URL path basenames that confidently indicate a sitemap / feed.+_FEED_BASENAMES = {+ "rss.xml",+ "feed.xml",+ "atom.xml",+ "feed.atom",+ "index.xml",+ "rss",+ "feed",+ "feeds",+ "atom",+}++# Web page extensions that should be stripped from a URL slug (we re-save as .html).+_WEB_PAGE_EXTENSIONS = (".html", ".htm", ".php", ".asp", ".aspx", ".jsp")++# Common conventional locations for a site's sitemap / feed, probed when auto-discovering.+_COMMON_FEED_PATHS = (+ "/sitemap.xml",+ "/sitemap-index.xml",+ "/sitemap_index.xml",+ "/rss.xml",+ "/feed.xml",+ "/atom.xml",+ "/index.xml",+ "/feed",+ "/rss",+)+++@dataclass+class FeedEntry:+ """A single page discovered in a sitemap or feed."""++ url: str+ title: Optional[str] = None+ lastmod: Optional[str] = None+ inline_html: Optional[str] = None+ """Full article HTML carried inline by the feed (RSS content:encoded / Atom+ content). When present we save it directly instead of fetching the page."""+++# --------------------------------------------------------------------------- #+# Module-level helpers (kept free-standing so they're easy to unit test)+# --------------------------------------------------------------------------- #+def _localname(tag: str) -> str:+ """Return the namespace-stripped, lowercased local name of an XML tag."""+ return tag.rsplit("}", 1)[-1].lower()+++def looks_like_feed_url(source: Union[str, Path]) -> bool:+ """Heuristic: does this URL path look like a sitemap / RSS / Atom feed?++ Conservative on purpose — ordinary article URLs must fall through to+ HTTPAccessor for single-page ingestion.+ """+ source_str = str(source)+ if not source_str.startswith(("http://", "https://")):+ return False+ basename = urlparse(source_str).path.lower().rstrip("/").rsplit("/", 1)[-1]+ if not basename:+ return False+ if basename.startswith("sitemap") and basename.endswith(".xml"):+ return True+ if basename in _FEED_BASENAMES:+ return True+ if basename.endswith((".rss", ".atom")):+ return True+ return False+++def _resolve_override(kwargs: Dict[str, Any]) -> Optional[bool]:+ """Resolve an explicit site/feed override from kwargs, if any."""+ for key in _OVERRIDE_KEYS:+ value = kwargs.get(key)+ if isinstance(value, bool):+ return value+ return None+++def _sanitize_filename(name: str, max_len: int = 120) -> str:+ """Make a string safe to use as a single path segment.++ Keeps unicode (e.g. CJK titles) readable; only strips path separators,+ reserved/control characters, and trims length. The resulting stem becomes+ the page node's title downstream, so readability matters.+ """+ name = (name or "").strip()+ # Replace path separators, control chars, and filesystem-reserved chars.+ name = re.sub(r'[\\/\x00-\x1f<>:"|?*]+', "-", name)+ name = re.sub(r"\s+", " ", name).strip(" .-")+ if len(name) > max_len:+ name = name[:max_len].strip(" .-")+ return name+++def _slugify_segment(segment: str) -> str:+ """Turn one URL path segment into a clean filename stem (no extension)."""+ segment = unquote(segment)+ lower = segment.lower()+ for ext in _WEB_PAGE_EXTENSIONS:+ if lower.endswith(ext):+ segment = segment[: -len(ext)]+ break+ return _sanitize_filename(segment)+++def url_to_relpath(url: str) -> str:+ """Map a page URL to a relative mirror path, preserving directory structure.++ Examples:+ https://h/ -> index.html+ https://h/a/b -> a/b.html+ https://h/a/b/ -> a/b.html+ https://h/post/foo/index.html -> post/foo/index.html+ """+ segments = [s for s in urlparse(url).path.split("/") if s]+ if not segments:+ return "index.html"+ *parents, leaf = segments+ leaf_stem = _slugify_segment(leaf) or "index"+ parent_parts = [p for p in (_sanitize_filename(unquote(s)) for s in parents) if p]+ return "/".join(parent_parts + [f"{leaf_stem}.html"])+++def _entry_to_relpath(entry: FeedEntry) -> str:+ """Choose the mirror path for an entry.++ Feed entries usually carry a real title -> use it as a flat, readable+ filename. Sitemap entries have no title -> mirror the URL path structure.+ """+ if entry.title:+ stem = _sanitize_filename(entry.title)+ if stem:+ return f"{stem}.html"+ return url_to_relpath(entry.url)+++def _dedup_relpath(relpath: str, url: str, used: set) -> str:+ """Disambiguate a colliding relpath with a short stable hash of the URL."""+ if relpath not in used:+ return relpath+ p = PurePosixPath(relpath)+ digest = hashlib.sha1(url.encode("utf-8")).hexdigest()[:6]+ newname = f"{p.stem}-{digest}{p.suffix}"+ parent = str(p.parent)+ return newname if parent == "." else f"{parent}/{newname}"+++def _wrap_inline_html(title: Optional[str], fragment: str) -> str:+ """Wrap an inline feed HTML fragment into a minimal standalone document."""+ title_tag = f"<title>{html_lib.escape(title)}</title>" if title else ""+ return (+ '<!DOCTYPE html><html><head><meta charset="utf-8">'+ f"{title_tag}</head><body>{fragment}</body></html>"+ )+++def _split_patterns(value: Optional[str]) -> List[str]:+ if not value:+ return []+ return [p.strip() for p in str(value).split(",") if p.strip()]+++def _match_filters(url: str, include: Optional[str], exclude: Optional[str]) -> bool:+ """Apply comma-separated include/exclude glob (or substring) filters to a URL."""+ for pat in _split_patterns(exclude):+ if fnmatch.fnmatch(url, pat) or pat in url:+ return False+ includes = _split_patterns(include)+ if includes:+ return any(fnmatch.fnmatch(url, pat) or pat in url for pat in includes)+ return True+++def _build_httpx_client_kwargs(request_validator, timeout: float) -> Dict[str, Any]:+ """Build httpx.AsyncClient kwargs with the shared SSRF network guard.++ Mirrors HTTPAccessor's client construction so sitemaps, feeds, sub-sitemaps,+ robots.txt and every mirrored page go through the same request validation.+ """+ client_kwargs: Dict[str, Any] = {"timeout": timeout, "follow_redirects": True}+ event_hooks = build_httpx_request_validation_hooks(request_validator)+ if event_hooks:+ client_kwargs["event_hooks"] = event_hooks+ client_kwargs["trust_env"] = False+ return client_kwargs+++def _load_webfeed_config():+ """Return the configured WebFeedConfig, falling back to defaults."""+ from openviking_cli.utils.config.parser_config import WebFeedConfig++ try:+ from openviking_cli.utils.config import get_openviking_config++ cfg = getattr(get_openviking_config(), "webfeed", None)+ if cfg is not None:+ return cfg+ except Exception:+ pass+ return WebFeedConfig()+++# --------------------------------------------------------------------------- #+# Accessor+# --------------------------------------------------------------------------- #+class WebFeedAccessor(DataAccessor):+ """Accessor that ingests a whole website from a sitemap / RSS / Atom URL."""++ PRIORITY = 60 # Feishu 100 > Git 80 > WebFeed 60 > HTTP 50 > Local 1++ @property+ def priority(self) -> int:+ return self.PRIORITY++ def can_handle(self, source: Union[str, Path], **kwargs) -> bool:+ """Handle sitemap/feed URLs (by heuristic, or by explicit override).++ ``args={"site": True}`` forces any http(s) URL through this accessor;+ ``args={"site": False}`` opts a feed-looking URL back out to HTTPAccessor.+ """+ source_str = str(source)+ if not source_str.startswith(("http://", "https://")):+ return False+ override = _resolve_override(kwargs)+ if override is not None:+ return override+ return looks_like_feed_url(source_str)++ async def access(self, source: Union[str, Path], **kwargs) -> LocalResource:+ """Fetch the sitemap/feed, mirror its pages, return a directory resource."""+ source_url = str(source)+ request_validator = kwargs.get("request_validator")+ cfg = self._resolve_settings(kwargs)+ include = kwargs.get("include")+ exclude = kwargs.get("exclude")++ temp_dir = await asyncio.to_thread(tempfile.mkdtemp, prefix="ov_webfeed_")+ try:+ client_kwargs = _build_httpx_client_kwargs(request_validator, cfg["request_timeout"])+ httpx = lazy_import("httpx")+ async with httpx.AsyncClient(**client_kwargs) as client:+ # The given URL may already be a sitemap/feed, or it may be a plain+ # page / bare domain (e.g. args={"site": True} on "https://t0saki.com").+ # In the latter case, auto-discover the site's sitemap/RSS.+ feed_url, prefetched = await self._resolve_feed_source(source_url, client)+ entries, feed_kind = await self._collect(+ feed_url, client, cfg, prefetched=prefetched+ )+ if not entries:+ raise RuntimeError(f"No pages found in sitemap/feed: {feed_url}")++ entries = self._filter_entries(entries, feed_url, cfg, include, exclude)+ entries = await self._apply_robots(entries, feed_url, client, cfg)++ if len(entries) > cfg["max_pages"]:+ logger.warning(+ "[WebFeedAccessor] %s lists %d pages; truncating to max_pages=%d",+ feed_url,+ len(entries),+ cfg["max_pages"],+ )+ entries = entries[: cfg["max_pages"]]++ if not entries:+ raise RuntimeError(f"No ingestible pages remained after filtering: {feed_url}")++ pages = await self._mirror(entries, client, temp_dir, cfg)++ if not pages:+ raise RuntimeError(f"Failed to fetch any page from: {feed_url}")++ host = urlparse(feed_url).netloc+ meta = {+ # Drives the root URI -> viking://resources/<host>+ "original_filename": host,+ "feed_url": feed_url,+ "feed_kind": feed_kind,+ "url_type": feed_kind,+ "page_count": len(pages),+ "pages": pages,+ }+ logger.info(+ "[WebFeedAccessor] Mirrored %d page(s) from %s (%s)",+ len(pages),+ feed_url,+ feed_kind,+ )+ return LocalResource(+ path=Path(temp_dir),+ source_type=SourceType.HTTP,+ original_source=feed_url,+ meta=meta,+ is_temporary=True,+ )+ except Exception as e:+ logger.error("[WebFeedAccessor] Failed to access %s: %s", source_url, e, exc_info=True)+ shutil.rmtree(temp_dir, ignore_errors=True)+ raise++ # -- settings ----------------------------------------------------------- #+ def _resolve_settings(self, kwargs: Dict[str, Any]) -> Dict[str, Any]:+ """Merge WebFeedConfig defaults with per-call kwargs overrides."""+ cfg = _load_webfeed_config()++ def pick(key, cast, default):+ value = kwargs.get(key)+ if value is None:+ value = getattr(cfg, key, default)+ try:+ return cast(value)+ except (TypeError, ValueError):+ return default++ return {+ "max_pages": pick("max_pages", int, 200),+ "max_concurrency": max(1, pick("max_concurrency", int, 5)),+ "request_timeout": pick("request_timeout", float, 30.0),+ "politeness_delay": max(0.0, pick("politeness_delay", float, 0.2)),+ "same_host_only": bool(+ kwargs.get("same_host_only", getattr(cfg, "same_host_only", True))+ ),+ "respect_robots": bool(+ kwargs.get("respect_robots", getattr(cfg, "respect_robots", True))+ ),+ "max_depth": max(1, pick("max_depth", int, 2)),+ }++ # -- discovery ---------------------------------------------------------- #+ async def _fetch_bytes(self, url: str, client) -> bytes:+ response = await client.get(url, headers={"User-Agent": _USER_AGENT})+ response.raise_for_status()+ return response.content++ def _sniff(self, content: bytes) -> Tuple[str, Any]:+ """Classify a fetched document by its XML root element."""+ from defusedxml.ElementTree import fromstring++ try:+ root = fromstring(content)+ except Exception:+ # Not well-formed XML we can route — let feedparser try as a feed.+ return "feed", None+ name = _localname(root.tag)+ if name == "urlset":+ return "urlset", root+ if name == "sitemapindex":+ return "sitemapindex", root+ if name in ("rss", "feed"):+ return "feed", root+ return "unknown", root++ def _is_feed_document(self, content: bytes) -> bool:+ """Whether ``content`` is a usable sitemap / sitemapindex / RSS / Atom doc."""+ kind, _ = self._sniff(content)+ if kind in ("urlset", "sitemapindex"):+ return True+ if kind == "feed":+ # "feed" also covers the non-XML fallback; confirm real entries exist.+ try:+ import feedparser++ return len(feedparser.parse(content).entries) > 0+ except Exception:+ return False+ return False++ async def _resolve_feed_source(self, url: str, client) -> Tuple[str, bytes]:+ """Return a ``(feed_url, content)`` that is an actual sitemap/feed.++ If ``url`` already points at a sitemap/feed, it is returned as-is. Otherwise+ (a plain page or bare domain — e.g. ``args={"site": True}`` on a homepage),+ auto-discover the site's sitemap/RSS via robots.txt, ``<link rel=alternate>``+ autodiscovery, and conventional locations, and return the first real one.+ """+ content = await self._fetch_bytes(url, client)+ if self._is_feed_document(content):+ return url, content++ for candidate in await _gather_feed_candidates(client, url, content):+ if candidate == url:+ continue+ try:+ cand_content = await self._fetch_bytes(candidate, client)+ except Exception:+ continue+ if self._is_feed_document(cand_content):+ logger.info("[WebFeedAccessor] auto-discovered feed %s for %s", candidate, url)+ return candidate, cand_content++ raise RuntimeError(+ f"{url} is not a sitemap/feed and no sitemap/RSS could be auto-discovered "+ "for the site. Pass the sitemap/feed URL directly."+ )++ async def _collect(+ self,+ url: str,+ client,+ cfg: Dict[str, Any],+ depth: int = 0,+ visited: Optional[set] = None,+ prefetched: Optional[bytes] = None,+ ) -> Tuple[List[FeedEntry], str]:+ """Recursively collect entries from a sitemap / sitemapindex / feed."""+ if visited is None:+ visited = {url}+ content = prefetched if prefetched is not None else await self._fetch_bytes(url, client)+ kind, root = self._sniff(content)++ if kind == "sitemapindex":+ if depth >= cfg["max_depth"]:+ logger.warning(+ "[WebFeedAccessor] sitemapindex depth limit (%d) reached at %s",+ cfg["max_depth"],+ url,+ )+ return [], "sitemap"+ all_entries: List[FeedEntry] = []+ for sub_url in self._parse_sitemapindex(root):+ if sub_url in visited:+ continue+ visited.add(sub_url)+ try:+ sub_entries, _ = await self._collect(sub_url, client, cfg, depth + 1, visited)+ all_entries.extend(sub_entries)+ except Exception as e:+ logger.warning(+ "[WebFeedAccessor] Failed to read sub-sitemap %s: %s", sub_url, e+ )+ return all_entries, "sitemap"++ if kind == "urlset":+ return self._parse_urlset(root), "sitemap"++ if kind == "feed":+ return self._parse_feed(content, base_url=url)++ raise RuntimeError(f"Unrecognized document (not a sitemap or feed): {url}")++ @staticmethod+ def _parse_urlset(root) -> List[FeedEntry]:+ entries: List[FeedEntry] = []+ for url_el in root:+ if _localname(url_el.tag) != "url":+ continue+ loc = None+ lastmod = None+ for child in url_el:+ ln = _localname(child.tag)+ if ln == "loc":+ loc = (child.text or "").strip()+ elif ln == "lastmod":+ lastmod = (child.text or "").strip()+ if loc:+ entries.append(FeedEntry(url=loc, lastmod=lastmod))+ return entries++ @staticmethod+ def _parse_sitemapindex(root) -> List[str]:+ locs: List[str] = []+ for sm_el in root:+ if _localname(sm_el.tag) != "sitemap":+ continue+ for child in sm_el:+ if _localname(child.tag) == "loc":+ loc = (child.text or "").strip()+ if loc:+ locs.append(loc)+ return locs++ @staticmethod+ def _parse_feed(content: bytes, base_url: str) -> Tuple[List[FeedEntry], str]:+ import feedparser++ parsed = feedparser.parse(content)+ version = str(parsed.get("version") or "")+ feed_kind = "atom" if version.startswith("atom") else "rss"++ entries: List[FeedEntry] = []+ for item in parsed.entries:+ link = item.get("link")+ if not link:+ for link_obj in item.get("links", []) or []:+ if link_obj.get("rel") in (None, "alternate") and link_obj.get("href"):+ link = link_obj["href"]+ break+ if not link:+ continue+ link = urljoin(base_url, link)++ inline_html = None+ contents = item.get("content")+ if isinstance(contents, list) and contents and contents[0].get("value"):+ inline_html = contents[0]["value"]++ entries.append(+ FeedEntry(+ url=link,+ title=item.get("title"),+ lastmod=item.get("published") or item.get("updated"),+ inline_html=inline_html,+ )+ )+ return entries, feed_kind++ # -- filtering ---------------------------------------------------------- #+ def _filter_entries(+ self,+ entries: List[FeedEntry],+ feed_url: str,+ cfg: Dict[str, Any],+ include: Optional[str],+ exclude: Optional[str],+ ) -> List[FeedEntry]:+ base_host = urlparse(feed_url).netloc+ seen: set = set()+ out: List[FeedEntry] = []+ for entry in entries:+ url = entry.url+ if not url.startswith(("http://", "https://")):+ continue+ if url in seen:+ continue+ if cfg["same_host_only"] and urlparse(url).netloc != base_host:+ continue+ if not _match_filters(url, include, exclude):+ continue+ seen.add(url)+ out.append(entry)+ return out++ async def _apply_robots(+ self, entries: List[FeedEntry], feed_url: str, client, cfg: Dict[str, Any]+ ) -> List[FeedEntry]:+ if not cfg["respect_robots"]:+ return entries+ try:+ parsed = urlparse(feed_url)+ robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"+ response = await client.get(robots_url, headers={"User-Agent": _USER_AGENT})+ if response.status_code >= 400:+ return entries+ rp = RobotFileParser()+ rp.parse(response.text.splitlines())+ return [e for e in entries if rp.can_fetch(_USER_AGENT, e.url)]+ except Exception as e:+ logger.debug("[WebFeedAccessor] robots.txt check skipped for %s: %s", feed_url, e)+ return entries++ # -- mirroring ---------------------------------------------------------- #+ async def _mirror(+ self, entries: List[FeedEntry], client, temp_dir: str, cfg: Dict[str, Any]+ ) -> Dict[str, Optional[str]]:+ """Fetch (or write inline) each page into temp_dir; return {url: lastmod}."""+ used: set = set()+ planned: List[Tuple[FeedEntry, str]] = []+ for entry in entries:+ relpath = _dedup_relpath(_entry_to_relpath(entry), entry.url, used)+ used.add(relpath)+ planned.append((entry, relpath))++ sem = asyncio.Semaphore(cfg["max_concurrency"])++ async def mirror_one(entry: FeedEntry, relpath: str) -> bool:+ async with sem:+ if cfg["politeness_delay"]:+ await asyncio.sleep(cfg["politeness_delay"])+ dest = Path(temp_dir) / relpath+ await asyncio.to_thread(dest.parent.mkdir, parents=True, exist_ok=True)++ if entry.inline_html:+ document = _wrap_inline_html(entry.title, entry.inline_html)+ await asyncio.to_thread(dest.write_text, document, encoding="utf-8")+ return True++ response = await client.get(entry.url, headers={"User-Agent": _USER_AGENT})+ response.raise_for_status()+ content_type = response.headers.get("content-type", "").lower()+ if content_type and not any(+ token in content_type for token in ("html", "xml", "text")+ ):+ logger.debug(+ "[WebFeedAccessor] Skipping non-text page %s (%s)",+ entry.url,+ content_type,+ )+ return False+ await asyncio.to_thread(dest.write_bytes, response.content)+ return True++ results = await asyncio.gather(+ *(mirror_one(entry, relpath) for entry, relpath in planned),+ return_exceptions=True,+ )++ pages: Dict[str, Optional[str]] = {}+ for (entry, _relpath), result in zip(planned, results, strict=True):+ if isinstance(result, Exception):+ logger.warning("[WebFeedAccessor] Failed to mirror %s: %s", entry.url, result)+ continue+ if result:+ pages[entry.url] = entry.lastmod+ return pages+++# --------------------------------------------------------------------------- #+# Single-page detect-and-suggest (never auto-crawls)+# --------------------------------------------------------------------------- #+def _extract_attr(tag: str, name: str) -> Optional[str]:+ match = re.search(rf'{name}\s*=\s*"([^"]*)"', tag, re.I) or re.search(+ rf"{name}\s*=\s*'([^']*)'", tag, re.I+ )+ return match.group(1) if match else None+++def _extract_feed_links(html: str, base_url: str) -> List[str]:+ """Find <link rel=alternate type=rss/atom> and <link rel=sitemap> hrefs."""+ out: List[str] = []+ for match in re.finditer(r"<link\b[^>]*>", html, re.I):+ tag = match.group(0)+ href = _extract_attr(tag, "href")+ if not href:+ continue+ rel = (_extract_attr(tag, "rel") or "").lower()+ typ = (_extract_attr(tag, "type") or "").lower()+ if "alternate" in rel and typ in ("application/rss+xml", "application/atom+xml"):+ out.append(urljoin(base_url, href))+ elif rel == "sitemap":+ out.append(urljoin(base_url, href))+ return out+++def _classify_and_count(content: bytes) -> Optional[Tuple[str, int]]:+ """Lightly classify a candidate document and count its entries (no recursion)."""+ from defusedxml.ElementTree import fromstring++ try:+ root = fromstring(content)+ except Exception:+ try:+ import feedparser++ parsed = feedparser.parse(content)+ if parsed.entries:+ version = str(parsed.get("version") or "")+ kind = "atom" if version.startswith("atom") else "rss"+ return kind, len(parsed.entries)+ except Exception:+ pass+ return None++ name = _localname(root.tag)+ if name == "urlset":+ return "sitemap", sum(1 for el in root if _localname(el.tag) == "url")+ if name == "sitemapindex":+ return "sitemapindex", sum(1 for el in root if _localname(el.tag) == "sitemap")+ if name in ("rss", "feed"):+ try:+ import feedparser++ parsed = feedparser.parse(content)+ kind = "atom" if name == "feed" else "rss"+ return kind, len(parsed.entries)+ except Exception:+ return None+ return None+++def _format_hint(feed_url: str, kind: str, count: int) -> str:+ if kind == "sitemap":+ what = f"a sitemap with {count} page(s)"+ elif kind == "sitemapindex":+ what = f"a sitemap index ({count} sub-sitemap(s))"+ elif kind == "atom":+ what = f"an Atom feed with {count} entr{'y' if count == 1 else 'ies'}"+ else:+ what = f"an RSS feed with {count} item(s)"+ return (+ f"Tip: this site exposes {what}. To ingest the whole site as a single "+ f"watchable resource, run add_resource('{feed_url}')."+ )+++async def _gather_feed_candidates(client, page_url: str, page_content: bytes) -> List[str]:+ """Collect ordered candidate sitemap/feed URLs for a site.++ Sources, in priority order: robots.txt ``Sitemap:`` directives, ``<link+ rel=alternate|sitemap>`` autodiscovery in the page HTML, and conventional paths.+ Shared by the single-page hint probe and the whole-site auto-discovery fallback.+ """+ parsed = urlparse(page_url)+ origin = f"{parsed.scheme}://{parsed.netloc}"+ candidates: List[str] = []+ # 1. robots.txt Sitemap: directives+ try:+ response = await client.get(origin + "/robots.txt", headers={"User-Agent": _USER_AGENT})+ if response.status_code < 400:+ rp = RobotFileParser()+ rp.parse(response.text.splitlines())+ candidates.extend(rp.site_maps() or [])+ except Exception:+ pass+ # 2. <link rel=alternate/sitemap> autodiscovery from the page HTML+ try:+ if page_content:+ candidates.extend(+ _extract_feed_links(page_content.decode("utf-8", "replace"), page_url)+ )+ except Exception:+ pass+ # 3. conventional locations+ candidates.extend(origin + path for path in _COMMON_FEED_PATHS)++ seen: set = set()+ return [c for c in candidates if not (c in seen or seen.add(c))]+++async def _discover_feed_hint(url: str, request_validator) -> Optional[str]:+ url = str(url)+ if not url.startswith(("http://", "https://")):+ return None+ try:+ from openviking.utils.code_hosting_utils import is_git_repo_url++ if is_git_repo_url(url):+ return None+ except Exception:+ pass+ if looks_like_feed_url(url):+ return None # already a feed — nothing to suggest+ ext = Path(urlparse(url).path).suffix.lower()+ if ext and ext not in _WEB_PAGE_EXTENSIONS:+ return None # a downloadable file (pdf/img/...), not a webpage++ httpx = lazy_import("httpx")+ client_kwargs = _build_httpx_client_kwargs(request_validator, 5.0)+ async with httpx.AsyncClient(**client_kwargs) as client:+ # Fetch the page once for <link> autodiscovery (best-effort).+ page_content = b""+ try:+ response = await client.get(url, headers={"User-Agent": _USER_AGENT})+ if (+ response.status_code < 400+ and "html" in response.headers.get("content-type", "").lower()+ ):+ page_content = response.content+ except Exception:+ pass++ for candidate in await _gather_feed_candidates(client, url, page_content):+ try:+ response = await client.get(candidate, headers={"User-Agent": _USER_AGENT})+ if response.status_code >= 400:+ continue+ info = _classify_and_count(response.content)+ if info and info[1] > 0:+ return _format_hint(candidate, info[0], info[1])+ except Exception:+ continue+ return None+++async def discover_feed_hint(+ url: Union[str, Path],+ *,+ timeout: float = 2.5,+ request_validator=None,+) -> Optional[str]:+ """Best-effort probe: suggest whole-site ingestion when a single page is added.++ Hard-bounded by ``timeout`` and fully exception-swallowing — it must never+ block, slow, or fail the single-page add it is attached to. Returns a+ one-line hint string, or None when nothing relevant is found.+ """+ try:+ return await asyncio.wait_for(_discover_feed_hint(str(url), request_validator), timeout)+ except Exception:+ return Nonediff --git a/openviking/server/mcp_endpoint.py b/openviking/server/mcp_endpoint.pyindex fb8f0e1d3..51da9491d 100644--- a/openviking/server/mcp_endpoint.py+++ b/openviking/server/mcp_endpoint.py@@ -439,6 +439,39 @@ def _resolve_public_base_url() -> tuple[str, str]:return "http://127.0.0.1:1933", "listen"+async def _maybe_sitemap_hint(path: str) -> str:+ """Best-effort sitemap/RSS suggestion for a single-page add (never crawls).++ Policy: only suggest when the added URL is the site root / homepage (path is+ empty or "/"). Adding a deep article URL implies intent for just that page, and+ gating to the root also keeps the (bounded, non-recursive) probe from re-running+ when many pages of the same site are added one-by-one.++ Gated by config (``webfeed.suggest_feed``) and fully exception-safe so it can+ never block or fail the add it is attached to.+ """+ try:+ from urllib.parse import urlparse++ if urlparse(path).path not in ("", "/"):+ return ""++ from openviking.parse.accessors import discover_feed_hint+ from openviking.utils.network_guard import ensure_public_remote_target+ from openviking_cli.utils.config import get_openviking_config++ webfeed = getattr(get_openviking_config(), "webfeed", None)+ if webfeed is not None and not getattr(webfeed, "suggest_feed", True):+ return ""+ timeout = float(getattr(webfeed, "suggest_timeout", 2.5)) if webfeed else 2.5+ hint = await discover_feed_hint(+ path, timeout=timeout, request_validator=ensure_public_remote_target+ )+ return hint or ""+ except Exception:+ return ""++@mcp.tool()async def add_resource(path: str = "",@@ -456,6 +489,10 @@ async def add_resource(Returns a success message immediately. Supports ``watch_interval`` forauto-refresh subscriptions; pass ``to`` to choose the exact target URI, oromit it to bind the watch to the URI created by this add.+ A sitemap / RSS / Atom URL ingests the WHOLE site as one resource tree;+ pass ``args={"site": true}`` to force whole-site ingestion from a bare+ domain (auto-discovers the site's sitemap/RSS). Watching a sitemap/feed+ URL keeps the whole site auto-refreshed.2. Local file: pass ``path`` set to a local filesystem path (e.g. ``/tmp/foo.pdf``).The response is NOT a success message — it's a multi-step upload instruction.@@ -562,11 +599,18 @@ async def add_resource(watch_suffix = f" (watch enabled, refresh every {watch_interval:g} minute(s))"else:watch_suffix = ""- return (+ message = (f"Resource added: {root_uri}{watch_suffix}"if root_urielse f"Resource added (processing in background){watch_suffix}.")+ # Detect-and-suggest: if this single page belongs to a site that exposes a+ # sitemap/RSS feed, hint at whole-site ingestion. Never auto-crawls; the+ # add above is already done, so a slow/failed probe has no functional impact.+ hint = await _maybe_sitemap_hint(path)+ if hint:+ message += "\n" + hint+ return message# Branch 4: local path — mint token, return upload instructionserver_config = get_server_config()diff --git a/openviking/sync_client.py b/openviking/sync_client.pyindex 1c2d1b6da..af31adb36 100644--- a/openviking/sync_client.py+++ b/openviking/sync_client.py@@ -210,6 +210,11 @@ class SyncOpenViking:) -> Dict[str, Any]:"""Add resource to OpenViking (resources scope only)+ A sitemap / RSS / Atom URL ingests the whole site as one resource tree;+ pass ``args={"site": True}`` to force whole-site ingestion from a bare+ domain. A ``watch_interval`` on a sitemap/feed URL keeps the whole site+ refreshed.+Args:to: Exact target URI. Existing targets keep the add_resource incremental-update behavior.parent: Target parent URI for automatic child naming.diff --git a/openviking_cli/utils/config/open_viking_config.py b/openviking_cli/utils/config/open_viking_config.pyindex 85e191c6f..8b6a4b970 100644--- a/openviking_cli/utils/config/open_viking_config.py+++ b/openviking_cli/utils/config/open_viking_config.py@@ -21,8 +21,8 @@ from .consts import ()from .embedding_config import EmbeddingConfigfrom .encryption_config import EncryptionConfig-from .grep_config import GrepConfigfrom .git_config import GitConfig+from .grep_config import GrepConfigfrom .log_config import LogConfigfrom .memory_config import MemoryConfigfrom .oauth_config import OAuthConfig@@ -38,6 +38,7 @@ from .parser_config import (SemanticConfig,TextConfig,VideoConfig,+ WebFeedConfig,)from .prompts_config import PromptsConfigfrom .rerank_config import RerankConfig@@ -185,6 +186,11 @@ class OpenVikingConfig(BaseModel):description="Feishu/Lark document parsing configuration",)+ webfeed: WebFeedConfig = Field(+ default_factory=WebFeedConfig,+ description="Whole-site ingestion via sitemap / RSS / Atom feeds",+ )+semantic: SemanticConfig = Field(default_factory=SemanticConfig,description="Semantic processing configuration (overview/abstract limits)",@@ -335,6 +341,7 @@ class OpenVikingConfig(BaseModel):"text","directory","feishu",+ "webfeed",]raise_unknown_config_fields(data=config_copy,diff --git a/openviking_cli/utils/config/parser_config.py b/openviking_cli/utils/config/parser_config.pyindex d48bddafa..2aa84c405 100644--- a/openviking_cli/utils/config/parser_config.py+++ b/openviking_cli/utils/config/parser_config.py@@ -551,6 +551,56 @@ class DirectoryConfig(ParserConfig):preserve_structure: bool = True+@dataclass+class WebFeedConfig(ParserConfig):+ """+ Configuration for whole-site ingestion via sitemap / RSS / Atom feeds.++ Used by WebFeedAccessor (and its single-page detect-and-suggest helper).+ Each setting can be overridden per call via add_resource ``args`` (e.g.+ ``args={"max_pages": 50}``).++ Attributes:+ max_pages: Hard cap on the number of pages mirrored per site.+ max_concurrency: Max concurrent page fetches.+ request_timeout: Per-request timeout in seconds.+ politeness_delay: Delay (seconds) before each page fetch, to be polite.+ same_host_only: Only ingest URLs on the same host as the feed.+ respect_robots: Honor robots.txt Disallow rules (and discover sitemaps).+ max_depth: Max recursion depth when following <sitemapindex> entries.+ suggest_feed: When adding a single webpage, probe for a sitemap/RSS and+ append a one-line hint suggesting whole-site ingestion (never auto-crawls).+ suggest_timeout: Hard timeout (seconds) for that single-page probe.+ """++ max_pages: int = 200+ max_concurrency: int = 5+ request_timeout: float = 30.0+ politeness_delay: float = 0.2+ same_host_only: bool = True+ respect_robots: bool = True+ max_depth: int = 2+ suggest_feed: bool = True+ suggest_timeout: float = 2.5++ def validate(self) -> None:+ """Validate web feed configuration."""+ super().validate()++ if self.max_pages <= 0:+ raise ValueError("max_pages must be positive")+ if self.max_concurrency <= 0:+ raise ValueError("max_concurrency must be positive")+ if self.request_timeout <= 0:+ raise ValueError("request_timeout must be positive")+ if self.politeness_delay < 0:+ raise ValueError("politeness_delay must be non-negative")+ if self.max_depth <= 0:+ raise ValueError("max_depth must be positive")+ if self.suggest_timeout <= 0:+ raise ValueError("suggest_timeout must be positive")++@dataclassclass SemanticConfig:"""@@ -599,6 +649,7 @@ PARSER_CONFIG_REGISTRY = {"text": TextConfig,"directory": DirectoryConfig,"feishu": FeishuConfig,+ "webfeed": WebFeedConfig,}diff --git a/pyproject.toml b/pyproject.tomlindex 59922b747..8b2fdca8f 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -38,6 +38,8 @@ dependencies = ["pdfplumber>=0.10.0","readabilipy>=0.2.0","markdownify>=0.11.0",+ "feedparser>=6.0.0",+ "defusedxml>=0.7.1","openai>=1.0.0","requests>=2.31.0","charset-normalizer>=3.4,<4",diff --git a/tests/unit/test_accessors_web_feed.py b/tests/unit/test_accessors_web_feed.pynew file mode 100644index 000000000..861f6817c--- /dev/null+++ b/tests/unit/test_accessors_web_feed.py@@ -0,0 +1,563 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Unit tests for WebFeedAccessor (whole-site ingestion via sitemap / RSS / Atom)."""++import pytest++from openviking.parse.accessors import (+ AccessorRegistry,+ GitAccessor,+ HTTPAccessor,+ WebFeedAccessor,+)+from openviking.parse.accessors.web_feed_accessor import (+ FeedEntry,+ _classify_and_count,+ _entry_to_relpath,+ _extract_feed_links,+ discover_feed_hint,+ looks_like_feed_url,+ url_to_relpath,+)++SITEMAP_NS = "http://www.sitemaps.org/schemas/sitemap/0.9"+++# --------------------------------------------------------------------------- #+# Fake httpx client (dependency-free; no real network)+# --------------------------------------------------------------------------- #+class _FakeStatusError(Exception):+ pass+++class _FakeResponse:+ def __init__(self, status_code=200, content=b"", content_type="text/html"):+ self.status_code = status_code+ self.content = content if isinstance(content, bytes) else content.encode("utf-8")+ self.headers = {"content-type": content_type}++ @property+ def text(self):+ return self.content.decode("utf-8", "replace")++ def raise_for_status(self):+ if self.status_code >= 400:+ raise _FakeStatusError(f"status {self.status_code}")+++def _make_fake_client(routes):+ """routes: dict url -> _FakeResponse (missing urls -> 404)."""++ class _FakeClient:+ def __init__(self, **kwargs):+ self.kwargs = kwargs++ async def __aenter__(self):+ return self++ async def __aexit__(self, *exc):+ return False++ async def get(self, url, headers=None):+ resp = routes.get(url)+ if resp is None:+ return _FakeResponse(404, b"not found", "text/plain")+ return resp++ return _FakeClient+++def patch_httpx(monkeypatch):+ """Patch httpx.AsyncClient with a fake serving the given routes."""++ def _apply(routes):+ import httpx++ monkeypatch.setattr(httpx, "AsyncClient", _make_fake_client(routes))++ return _apply+++def _urlset(urls_with_lastmod):+ items = "".join(+ f"<url><loc>{u}</loc>" + (f"<lastmod>{lm}</lastmod>" if lm else "") + "</url>"+ for u, lm in urls_with_lastmod+ )+ return f'<?xml version="1.0"?><urlset xmlns="{SITEMAP_NS}">{items}</urlset>'.encode()+++def _sitemapindex(sub_urls):+ items = "".join(f"<sitemap><loc>{u}</loc></sitemap>" for u in sub_urls)+ return (+ f'<?xml version="1.0"?><sitemapindex xmlns="{SITEMAP_NS}">{items}</sitemapindex>'.encode()+ )+++def _page(title):+ return f"<html><head><title>{title}</title></head><body><h1>{title}</h1><p>body</p></body></html>".encode()+++# --------------------------------------------------------------------------- #+# can_handle+# --------------------------------------------------------------------------- #+class TestCanHandle:+ @pytest.fixture+ def accessor(self):+ return WebFeedAccessor()++ def test_priority(self, accessor):+ assert accessor.priority == 60++ @pytest.mark.parametrize(+ "url",+ [+ "https://blog.openviking.ai/sitemap.xml",+ "https://t0saki.com/sitemap-index.xml",+ "https://t0saki.com/sitemap_index.xml",+ "https://t0saki.com/sitemap-0.xml",+ "https://t0saki.com/rss.xml",+ "https://example.com/feed.xml",+ "https://example.com/atom.xml",+ "https://example.com/index.xml",+ "https://example.com/feed",+ "https://example.com/blog/feed/",+ ],+ )+ def test_handles_feed_like(self, accessor, url):+ assert accessor.can_handle(url) is True++ @pytest.mark.parametrize(+ "url",+ [+ "https://example.com/post/foo/",+ "https://example.com/data.xml",+ "https://example.com/page.html",+ "/tmp/sitemap.xml",+ "[email protected]:org/repo.git",+ ],+ )+ def test_rejects_non_feed(self, accessor, url):+ assert accessor.can_handle(url) is False++ def test_explicit_override(self, accessor):+ assert accessor.can_handle("https://example.com/post/foo/", site=True) is True+ assert accessor.can_handle("https://example.com/sitemap.xml", site=False) is False+ # alias keys+ assert accessor.can_handle("https://example.com/x", sitemap=True) is True+ assert accessor.can_handle("https://example.com/x", feed=True) is True+++# --------------------------------------------------------------------------- #+# registry routing+# --------------------------------------------------------------------------- #+class TestRegistryRouting:+ @pytest.fixture+ def registry(self):+ return AccessorRegistry()++ def test_sitemap_routes_to_web_feed(self, registry):+ acc = registry.get_accessor("https://blog.openviking.ai/sitemap.xml")+ assert isinstance(acc, WebFeedAccessor)++ def test_git_url_still_wins(self, registry):+ acc = registry.get_accessor("https://github.com/volcengine/OpenViking")+ assert isinstance(acc, GitAccessor)++ def test_plain_page_routes_to_http(self, registry):+ acc = registry.get_accessor("https://example.com/article")+ assert isinstance(acc, HTTPAccessor)++ def test_override_routes_plain_page_to_web_feed(self, registry):+ acc = registry.get_accessor("https://example.com/article", site=True)+ assert isinstance(acc, WebFeedAccessor)+++# --------------------------------------------------------------------------- #+# url -> relpath+# --------------------------------------------------------------------------- #+class TestRelpath:+ @pytest.mark.parametrize(+ "url,expected",+ [+ ("https://h/", "index.html"),+ ("https://h/a/b", "a/b.html"),+ ("https://h/post/foo/", "post/foo.html"),+ ("https://h/post/foo/index.html", "post/foo/index.html"),+ ("https://h/about.php", "about.html"),+ ],+ )+ def test_url_to_relpath(self, url, expected):+ assert url_to_relpath(url) == expected++ def test_entry_title_used_when_present(self):+ assert (+ _entry_to_relpath(FeedEntry(url="https://h/x", title="Hello World"))+ == "Hello World.html"+ )++ def test_entry_title_cjk(self):+ assert _entry_to_relpath(FeedEntry(url="https://h/x", title="台湾游记")) == "台湾游记.html"++ def test_entry_no_title_falls_back_to_url(self):+ assert _entry_to_relpath(FeedEntry(url="https://h/post/bar/")) == "post/bar.html"++ def test_title_with_slash_is_sanitized(self):+ rel = _entry_to_relpath(FeedEntry(url="https://h/x", title="a/b: c"))+ assert "/" not in rel[:-5] # no path separator in the stem+++# --------------------------------------------------------------------------- #+# classify-and-count (used by the single-page hint)+# --------------------------------------------------------------------------- #+class TestClassify:+ def test_urlset(self):+ content = _urlset([("https://h/a", None), ("https://h/b", "2026-01-01")])+ assert _classify_and_count(content) == ("sitemap", 2)++ def test_sitemapindex(self):+ assert _classify_and_count(_sitemapindex(["https://h/s0.xml"])) == ("sitemapindex", 1)++ def test_rss(self):+ rss = b'<?xml version="1.0"?><rss version="2.0"><channel><item><link>https://h/a</link></item></channel></rss>'+ assert _classify_and_count(rss) == ("rss", 1)++ def test_garbage_returns_none(self):+ assert _classify_and_count(b"not xml at all") is None++ def test_extract_feed_links(self):+ html = (+ '<link rel="alternate" type="application/rss+xml" href="/rss.xml">'+ '<link rel="alternate" type="application/atom+xml" href="https://h/atom.xml">'+ '<link rel="sitemap" href="/sitemap.xml">'+ '<link rel="stylesheet" href="/x.css">'+ )+ links = _extract_feed_links(html, "https://h/page")+ assert "https://h/rss.xml" in links+ assert "https://h/atom.xml" in links+ assert "https://h/sitemap.xml" in links+ assert all("x.css" not in link for link in links)+++# --------------------------------------------------------------------------- #+# access() — sitemap mirroring+# --------------------------------------------------------------------------- #+class TestAccessSitemap:+ async def test_flat_urlset_mirrors_pages(self, patch_httpx):+ base = "https://blog.example.com"+ page_urls = [f"{base}/post/p{i}/" for i in range(10)]+ routes = {+ f"{base}/sitemap.xml": _FakeResponse(content=_urlset([(u, None) for u in page_urls]))+ }+ for i, u in enumerate(page_urls):+ routes[u] = _FakeResponse(content=_page(f"Post {i}"))+ patch_httpx(routes)++ acc = WebFeedAccessor()+ resource = await acc.access(f"{base}/sitemap.xml", respect_robots=False, politeness_delay=0)+ try:+ assert resource.is_temporary is True+ assert resource.path.is_dir()+ assert resource.meta["original_filename"] == "blog.example.com"+ assert resource.meta["page_count"] == 10+ assert resource.meta["url_type"] == "sitemap"+ html_files = list(resource.path.rglob("*.html"))+ assert len(html_files) == 10+ # mirrors URL path structure (post/p0.html ...)+ assert (resource.path / "post" / "p0.html").exists()+ finally:+ resource.cleanup()++ async def test_failed_page_is_skipped(self, patch_httpx):+ base = "https://blog.example.com"+ page_urls = [f"{base}/post/p{i}/" for i in range(5)]+ routes = {+ f"{base}/sitemap.xml": _FakeResponse(content=_urlset([(u, None) for u in page_urls]))+ }+ for i, u in enumerate(page_urls):+ routes[u] = _FakeResponse(content=_page(f"Post {i}"))+ routes[page_urls[2]] = _FakeResponse(500, b"err", "text/plain") # one fails+ patch_httpx(routes)++ acc = WebFeedAccessor()+ resource = await acc.access(f"{base}/sitemap.xml", respect_robots=False, politeness_delay=0)+ try:+ assert resource.meta["page_count"] == 4+ finally:+ resource.cleanup()++ async def test_sitemapindex_recursion(self, patch_httpx):+ base = "https://t0saki.com"+ page_urls = [f"{base}/posts/p{i}/" for i in range(3)]+ routes = {+ f"{base}/sitemap-index.xml": _FakeResponse(+ content=_sitemapindex([f"{base}/sitemap-0.xml"])+ ),+ f"{base}/sitemap-0.xml": _FakeResponse(content=_urlset([(u, None) for u in page_urls])),+ }+ for i, u in enumerate(page_urls):+ routes[u] = _FakeResponse(content=_page(f"P{i}"))+ patch_httpx(routes)++ acc = WebFeedAccessor()+ resource = await acc.access(+ f"{base}/sitemap-index.xml", respect_robots=False, politeness_delay=0+ )+ try:+ assert resource.meta["page_count"] == 3+ assert resource.meta["url_type"] == "sitemap"+ finally:+ resource.cleanup()++ async def test_max_pages_truncates(self, patch_httpx):+ base = "https://blog.example.com"+ page_urls = [f"{base}/p{i}/" for i in range(10)]+ routes = {+ f"{base}/sitemap.xml": _FakeResponse(content=_urlset([(u, None) for u in page_urls]))+ }+ for u in page_urls:+ routes[u] = _FakeResponse(content=_page("x"))+ patch_httpx(routes)++ acc = WebFeedAccessor()+ resource = await acc.access(+ f"{base}/sitemap.xml", respect_robots=False, politeness_delay=0, max_pages=3+ )+ try:+ assert resource.meta["page_count"] == 3+ finally:+ resource.cleanup()++ async def test_same_host_only_filters_external(self, patch_httpx):+ base = "https://blog.example.com"+ internal = f"{base}/keep/"+ external = "https://other.com/drop/"+ routes = {+ f"{base}/sitemap.xml": _FakeResponse(+ content=_urlset([(internal, None), (external, None)])+ ),+ internal: _FakeResponse(content=_page("keep")),+ external: _FakeResponse(content=_page("drop")),+ }+ patch_httpx(routes)++ acc = WebFeedAccessor()+ resource = await acc.access(f"{base}/sitemap.xml", respect_robots=False, politeness_delay=0)+ try:+ assert resource.meta["page_count"] == 1+ assert internal in resource.meta["pages"]+ assert external not in resource.meta["pages"]+ finally:+ resource.cleanup()+++# --------------------------------------------------------------------------- #+# access() — auto-discovery from a bare domain / HTML page (args={"site": True})+# --------------------------------------------------------------------------- #+class TestAutoDiscovery:+ async def test_bare_domain_discovers_sitemap_via_robots(self, patch_httpx):+ base = "https://t0saki.example"+ page_urls = [f"{base}/posts/p{i}/" for i in range(3)]+ routes = {+ base: _FakeResponse(content=b"<html><body>home</body></html>"),+ f"{base}/robots.txt": _FakeResponse(+ content=f"User-agent: *\nSitemap: {base}/sitemap.xml\n", content_type="text/plain"+ ),+ f"{base}/sitemap.xml": _FakeResponse(content=_urlset([(u, None) for u in page_urls])),+ }+ for i, u in enumerate(page_urls):+ routes[u] = _FakeResponse(content=_page(f"P{i}"))+ patch_httpx(routes)++ acc = WebFeedAccessor()+ # access() simulates the args={"site": True} routing (can_handle override).+ resource = await acc.access(base, respect_robots=False, politeness_delay=0)+ try:+ assert resource.meta["page_count"] == 3+ assert resource.meta["feed_url"] == f"{base}/sitemap.xml"+ assert resource.meta["original_filename"] == "t0saki.example"+ finally:+ resource.cleanup()++ async def test_html_page_discovers_feed_via_link_tag(self, patch_httpx):+ base = "https://blog.example"+ page_html = (+ "<html><head>"+ f'<link rel="alternate" type="application/rss+xml" href="{base}/rss.xml">'+ "</head><body>post</body></html>"+ ).encode()+ rss = (+ '<?xml version="1.0"?><rss version="2.0"><channel>'+ f"<item><title>One</title><link>{base}/a/</link></item>"+ "</channel></rss>"+ ).encode()+ routes = {+ f"{base}/some-article": _FakeResponse(content=page_html),+ f"{base}/rss.xml": _FakeResponse(content=rss, content_type="application/rss+xml"),+ f"{base}/a/": _FakeResponse(content=_page("A")),+ }+ patch_httpx(routes)++ acc = WebFeedAccessor()+ resource = await acc.access(+ f"{base}/some-article", respect_robots=False, politeness_delay=0+ )+ try:+ assert resource.meta["url_type"] == "rss"+ assert resource.meta["feed_url"] == f"{base}/rss.xml"+ finally:+ resource.cleanup()++ async def test_no_feed_discoverable_raises(self, patch_httpx):+ base = "https://nofeed.example"+ routes = {base: _FakeResponse(content=b"<html><body>nothing</body></html>")}+ patch_httpx(routes)+ acc = WebFeedAccessor()+ with pytest.raises(RuntimeError, match="auto-discovered"):+ await acc.access(base, respect_robots=False, politeness_delay=0)+++# --------------------------------------------------------------------------- #+# access() — RSS / Atom+# --------------------------------------------------------------------------- #+class TestAccessFeed:+ async def test_rss_summary_only_fetches_links(self, patch_httpx):+ base = "https://t0saki.com"+ items = [(f"{base}/posts/a/", "Post A"), (f"{base}/posts/b/", "Post B")]+ rss = (+ '<?xml version="1.0"?><rss version="2.0"><channel>'+ + "".join(+ f"<item><title>{t}</title><link>{u}</link><description>summary</description></item>"+ for u, t in items+ )+ + "</channel></rss>"+ ).encode()+ routes = {f"{base}/rss.xml": _FakeResponse(content=rss, content_type="application/rss+xml")}+ for u, _ in items:+ routes[u] = _FakeResponse(content=_page("full"))+ patch_httpx(routes)++ acc = WebFeedAccessor()+ resource = await acc.access(f"{base}/rss.xml", respect_robots=False, politeness_delay=0)+ try:+ assert resource.meta["url_type"] == "rss"+ assert resource.meta["page_count"] == 2+ # titles used as flat filenames+ assert (resource.path / "Post A.html").exists()+ assert (resource.path / "Post B.html").exists()+ finally:+ resource.cleanup()++ async def test_rss_inline_content_is_used_without_fetch(self, patch_httpx):+ base = "https://t0saki.com"+ rss = (+ '<?xml version="1.0"?>'+ '<rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/">'+ "<channel><item><title>Inline</title><link>https://t0saki.com/posts/inline/</link>"+ "<content:encoded><![CDATA[<p>FULL INLINE BODY</p>]]></content:encoded>"+ "</item></channel></rss>"+ ).encode()+ # Note: deliberately DO NOT register the article URL — if access tried to+ # fetch it we'd get a 404 and page_count would drop to 0.+ routes = {f"{base}/rss.xml": _FakeResponse(content=rss, content_type="application/rss+xml")}+ patch_httpx(routes)++ acc = WebFeedAccessor()+ resource = await acc.access(f"{base}/rss.xml", respect_robots=False, politeness_delay=0)+ try:+ assert resource.meta["page_count"] == 1+ saved = (resource.path / "Inline.html").read_text(encoding="utf-8")+ assert "FULL INLINE BODY" in saved+ finally:+ resource.cleanup()++ async def test_atom_feed(self, patch_httpx):+ base = "https://example.org"+ atom = (+ '<?xml version="1.0"?><feed xmlns="http://www.w3.org/2005/Atom">'+ '<entry><title>Entry One</title><link href="https://example.org/e1/"/></entry>'+ "</feed>"+ ).encode()+ routes = {+ f"{base}/atom.xml": _FakeResponse(content=atom, content_type="application/atom+xml"),+ f"{base}/e1/": _FakeResponse(content=_page("E1")),+ }+ patch_httpx(routes)++ acc = WebFeedAccessor()+ resource = await acc.access(f"{base}/atom.xml", respect_robots=False, politeness_delay=0)+ try:+ assert resource.meta["url_type"] == "atom"+ assert resource.meta["page_count"] == 1+ finally:+ resource.cleanup()+++# --------------------------------------------------------------------------- #+# discover_feed_hint (single-page detect-and-suggest)+# --------------------------------------------------------------------------- #+class TestDiscoverHint:+ async def test_hint_from_robots_sitemap(self, patch_httpx):+ base = "https://blog.example.com"+ routes = {+ f"{base}/robots.txt": _FakeResponse(+ content=f"User-agent: *\nSitemap: {base}/sitemap.xml\n", content_type="text/plain"+ ),+ f"{base}/sitemap.xml": _FakeResponse(+ content=_urlset([(f"{base}/p{i}/", None) for i in range(3)])+ ),+ f"{base}/article": _FakeResponse(content=_page("article")),+ }+ patch_httpx(routes)++ hint = await discover_feed_hint(f"{base}/article", timeout=5)+ assert hint is not None+ assert "3 page(s)" in hint+ assert f"{base}/sitemap.xml" in hint++ async def test_hint_from_html_autodiscovery(self, patch_httpx):+ base = "https://blog.example.com"+ page_html = (+ '<html><head><link rel="alternate" type="application/rss+xml" '+ f'href="{base}/rss.xml"></head><body>hi</body></html>'+ ).encode()+ rss = (+ '<?xml version="1.0"?><rss version="2.0"><channel>'+ "<item><link>https://blog.example.com/a/</link></item>"+ "<item><link>https://blog.example.com/b/</link></item>"+ "</channel></rss>"+ ).encode()+ routes = {+ f"{base}/article": _FakeResponse(content=page_html),+ f"{base}/rss.xml": _FakeResponse(content=rss, content_type="application/rss+xml"),+ # robots.txt missing -> 404, fine+ }+ patch_httpx(routes)++ hint = await discover_feed_hint(f"{base}/article", timeout=5)+ assert hint is not None+ assert "RSS feed with 2 item(s)" in hint++ async def test_no_hint_when_nothing_found(self, patch_httpx):+ base = "https://nofeed.example.com"+ routes = {f"{base}/article": _FakeResponse(content=_page("article"))}+ patch_httpx(routes)+ assert await discover_feed_hint(f"{base}/article", timeout=5) is None++ async def test_no_hint_for_feed_url_itself(self, patch_httpx):+ patch_httpx({})+ assert await discover_feed_hint("https://x.com/sitemap.xml", timeout=5) is None++ async def test_no_hint_for_non_html_extension(self, patch_httpx):+ patch_httpx({})+ assert await discover_feed_hint("https://x.com/file.pdf", timeout=5) is None+++# --------------------------------------------------------------------------- #+# looks_like_feed_url (standalone)+# --------------------------------------------------------------------------- #+def test_looks_like_feed_url():+ assert looks_like_feed_url("https://h/sitemap.xml")+ assert looks_like_feed_url("https://h/rss.xml")+ assert not looks_like_feed_url("https://h/post")+ assert not looks_like_feed_url("/tmp/sitemap.xml")