Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
refactor(parser):Unify Office Document Parsing with AnyDoc and Restore Compatibility Safeguards (#4279)
zgy7ee75611212a
diff --git a/docs/en/api/02-resources.md b/docs/en/api/02-resources.mdindex e52e432b9..0ef353724 100644--- a/docs/en/api/02-resources.md+++ b/docs/en/api/02-resources.md@@ -15,16 +15,16 @@ OpenViking supports various resource types, categorized by functionality:| PDF | `.pdf` | Supports local parsing and MinerU API conversion || Markdown | `.md`, `.markdown`, `.mdown`, `.mkd` | Native support, extracts structure and stores in segments || HTML | `.html`, `.htm` | Cleans navigation/ads and extracts content, converts to Markdown |-| Word | `.docx` | Extracts text, headings, tables and converts to Markdown |+| Word | `.doc`, `.docx`, `.docm`, `.odt`, `.rtf` | Uses anydoc to extract text, headings, tables, and embedded images into Markdown || Plain Text | `.txt`, `.text` | Direct import and processing |-| EPUB | `.epub` | E-book format, supports ebooklib or manual extraction |+| EPUB | `.epub` | Uses anydoc to convert e-book content and embedded images to Markdown |**Spreadsheets & Presentations**| Type | Extensions | Description ||------|------------|-------------|-| Excel | `.xlsx`, `.xls`, `.xlsm` | Supports new and legacy Excel formats, converts to Markdown tables by worksheet |-| PowerPoint | `.pptx` | Extracts content by slide, supports extracting notes |+| Excel | `.xlsx`, `.xls`, `.xlsm`, `.xlsb`, `.ods`, `.csv` | Uses anydoc to convert worksheets to Markdown tables |+| PowerPoint | `.pptx`, `.ppt`, `.pptm`, `.pps`, `.ppsx`, `.ppsm`, `.pot`, `.odp` | Uses anydoc to extract slide content and embedded images into Markdown |**Code**diff --git a/docs/en/configuration/01-server.md b/docs/en/configuration/01-server.mdindex 647984257..88f4fdbe0 100644--- a/docs/en/configuration/01-server.md+++ b/docs/en/configuration/01-server.md@@ -347,7 +347,9 @@ Parsers live under `parsers`:"audio": {},"video": {},"markdown": {},- "excel": {},+ "anydoc": {+ "enabled": true+ },"html": {},"text": {},"directory": {},@@ -369,7 +371,7 @@ Parsers live under `parsers`:| `image` | Image understanding and OCR || `audio`, `video` | Audio/video parsing || `markdown`, `html`, `text` | Text document chunking |-| `excel` | Workbook parsing and chunking |+| `anydoc` | Office and EPUB conversion; `enabled=false` rejects those formats || `directory` | Directory scanning and ignore rules || `feishu` | Feishu/Lark access and parsing || `webfeed` | Sitemap, RSS, and Atom ingestion |diff --git a/docs/zh/api/02-resources.md b/docs/zh/api/02-resources.mdindex 98f4c28dd..8c54824ca 100644--- a/docs/zh/api/02-resources.md+++ b/docs/zh/api/02-resources.md@@ -14,15 +14,15 @@ OpenViking 支持多种资源类型,按照功能分类如下:| PDF | `.pdf` | 支持本地解析和 MinerU API 转换 || Markdown | `.md`, `.markdown`, `.mdown`, `.mkd` | 原生支持,会提取结构并分段存储 || HTML | `.html`, `.htm` | 清理导航/广告后提取内容,转换为 Markdown |-| Word | `.docx` | 提取文本、标题、表格并转换为 Markdown |+| Word | `.doc`, `.docx`, `.docm`, `.odt`, `.rtf` | 基于 anydoc 提取文本、标题、表格和嵌入图片并转换为 Markdown || 纯文本 | `.txt`, `.text` | 直接导入处理 |-| EPUB | `.epub` | 电子书格式,支持 ebooklib 或手动提取 |+| EPUB | `.epub` | 基于 anydoc 将电子书内容和嵌入图片转换为 Markdown |表格类| 类型 | 扩展名 | 说明 ||------|--------|------|-| Excel | `.xlsx`, `.xls`, `.xlsm` | 支持新版和老版 Excel,按工作表转换为 Markdown 表格 |-| PowerPoint | `.pptx` | 按幻灯片提取内容,支持提取备注 |+| Excel | `.xlsx`, `.xls`, `.xlsm`, `.xlsb`, `.ods`, `.csv` | 基于 anydoc 按工作表转换为 Markdown 表格 |+| PowerPoint | `.pptx`, `.ppt`, `.pptm`, `.pps`, `.ppsx`, `.ppsm`, `.pot`, `.odp` | 基于 anydoc 按幻灯片提取内容和嵌入图片并转换为 Markdown |代码类| 类型 | 资源名 | 说明 |diff --git a/docs/zh/configuration/01-server.md b/docs/zh/configuration/01-server.mdindex 2e6d98728..f8325eb2f 100644--- a/docs/zh/configuration/01-server.md+++ b/docs/zh/configuration/01-server.md@@ -347,7 +347,9 @@ Provider 和密钥管理配置见[加密指南](../guides/08-encryption.md)。"audio": {},"video": {},"markdown": {},- "excel": {},+ "anydoc": {+ "enabled": true+ },"html": {},"text": {},"directory": {},@@ -369,7 +371,7 @@ Provider 和密钥管理配置见[加密指南](../guides/08-encryption.md)。| `image` | 图片理解和 OCR || `audio`、`video` | 音视频内容解析 || `markdown`、`html`、`text` | 文本文档分段 |-| `excel` | Excel 工作表解析与分段 |+| `anydoc` | Office 和 EPUB 转换;`enabled=false` 时拒绝这些格式 || `directory` | 目录扫描和忽略规则 || `feishu` | 飞书文档访问与解析 || `webfeed` | Sitemap、RSS 和 Atom 导入 |diff --git a/examples/ov.conf.example b/examples/ov.conf.exampleindex c8676a5e1..b18146705 100644--- a/examples/ov.conf.example+++ b/examples/ov.conf.example@@ -309,6 +309,9 @@"include_metadata": true,"max_heading_depth": 3,},+ "anydoc": {+ "enabled": true,+ },"html": {"extract_text_only": false,"preserve_structure": true,diff --git a/openviking/parse/image_validation.py b/openviking/parse/image_validation.pynew file mode 100644index 000000000..f8ebe4b33--- /dev/null+++ b/openviking/parse/image_validation.py@@ -0,0 +1,58 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Shared acceptance policy for images embedded in parsed documents."""++import io+from pathlib import Path++from openviking_cli.utils.logger import get_logger++logger = get_logger(__name__)++IMAGE_MIN_SIDE = 14+IMAGE_MIN_PIXELS = 196+IMAGE_MAX_PIXELS = 36_000_000+IMAGE_MIN_ASPECT_RATIO = 1 / 150+IMAGE_MAX_ASPECT_RATIO = 150+IMAGE_MAX_FILE_BYTES = 10 * 1024 * 1024+++def is_valid_image(image_bytes: bytes, source_path: Path) -> bool:+ """Return whether an extracted image is safe and useful to ingest."""+ if len(image_bytes) > IMAGE_MAX_FILE_BYTES:+ logger.warning(f"[ImageValidation] Image exceeds 10MB, skipping: {source_path}")+ return False++ try:+ from PIL import Image++ with Image.open(io.BytesIO(image_bytes)) as image:+ width, height = image.size+ except Exception as exc:+ logger.warning(+ f"[ImageValidation] Cannot read image dimensions, skipping {source_path}: {exc}"+ )+ return False++ if width <= IMAGE_MIN_SIDE or height <= IMAGE_MIN_SIDE:+ logger.warning(+ f"[ImageValidation] Image side too small ({width}x{height}), skipping: {source_path}"+ )+ return False++ pixels = width * height+ if pixels < IMAGE_MIN_PIXELS or pixels > IMAGE_MAX_PIXELS:+ logger.warning(+ f"[ImageValidation] Image pixel count out of range ({pixels}), skipping: {source_path}"+ )+ return False++ aspect_ratio = width / height+ if aspect_ratio < IMAGE_MIN_ASPECT_RATIO or aspect_ratio > IMAGE_MAX_ASPECT_RATIO:+ logger.warning(+ f"[ImageValidation] Image aspect ratio out of range ({aspect_ratio:.4f}), "+ f"skipping: {source_path}"+ )+ return False++ return Truediff --git a/openviking/parse/parsers/__init__.py b/openviking/parse/parsers/__init__.pyindex 5f17740d0..bbbe3aede 100644--- a/openviking/parse/parsers/__init__.py+++ b/openviking/parse/parsers/__init__.py@@ -1,26 +1,20 @@# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.# SPDX-License-Identifier: AGPL-3.0+from .anydoc import AnyDocParserfrom .base_parser import BaseParser-from .epub import EPubParser-from .excel import ExcelParserfrom .html import HTMLParserfrom .markdown import MarkdownParserfrom .pdf import PDFParser-from .powerpoint import PowerPointParserfrom .text import TextParser-from .word import WordParserfrom .zip_parser import ZipParser__all__ = ["BaseParser",- "EPubParser",- "ExcelParser",+ "AnyDocParser","HTMLParser","MarkdownParser","PDFParser",- "PowerPointParser","TextParser",- "WordParser","ZipParser",]diff --git a/openviking/parse/parsers/anydoc.py b/openviking/parse/parsers/anydoc.pynew file mode 100644index 000000000..09de5747d--- /dev/null+++ b/openviking/parse/parsers/anydoc.py@@ -0,0 +1,126 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Unified AnyDoc parser for Office documents and EPUB files."""++import asyncio+import time+from pathlib import Path+from typing import List, Optional, Union++from openviking.parse.base import ParseResult+from openviking.parse.parsers.base_parser import BaseParser+from openviking_cli.utils.config.parser_config import AnydocConfig, ParserConfig++_SUPPORTED_EXTENSIONS = [+ ".doc",+ ".docx",+ ".docm",+ ".odt",+ ".rtf",+ ".pptx",+ ".ppt",+ ".pptm",+ ".pps",+ ".ppsx",+ ".ppsm",+ ".pot",+ ".odp",+ ".xlsx",+ ".xls",+ ".xlsm",+ ".xlsb",+ ".ods",+ ".csv",+ ".epub",+]+++class AnyDocParser(BaseParser):+ """Parse Office and EPUB files through AnyDoc."""++ def __init__(+ self,+ config: Optional[ParserConfig] = None,+ anydoc_config: Optional[AnydocConfig] = None,+ ):+ from openviking.parse.parsers.markdown import MarkdownParser++ self.config = config or ParserConfig()+ self.anydoc_config = anydoc_config or AnydocConfig()+ self._md_parser = MarkdownParser(config=self.config)++ @property+ def supported_extensions(self) -> List[str]:+ return list(_SUPPORTED_EXTENSIONS)++ async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:+ started = time.time()+ path = Path(source)+ if not path.is_file():+ raise FileNotFoundError(f"Document file not found: {path}")++ if not self.anydoc_config.enabled:+ raise RuntimeError(+ "AnyDoc parser is disabled; Office and EPUB legacy parsers have been removed"+ )++ from openviking.parse.parsers.anydoc_converter import AnyDocConverter+ from openviking_cli.utils.storage import get_storage++ storage = get_storage()+ resource_name = kwargs.get("resource_name") or kwargs.get("source_name") or path.stem+ conversion = await asyncio.to_thread(+ AnyDocConverter().convert,+ path,+ resource_name=resource_name,+ storage=storage,+ max_table_rows=self.anydoc_config.max_table_rows,+ )++ markdown_kwargs = dict(kwargs)+ allowed_media_dirs = list(markdown_kwargs.get("allowed_media_dirs") or [])+ if storage.media_dir not in allowed_media_dirs:+ allowed_media_dirs.append(storage.media_dir)+ markdown_kwargs.update(+ source_path=str(path),+ base_dir=path.parent,+ allowed_media_dirs=allowed_media_dirs,+ )+ result = await self._md_parser.parse_content(+ conversion.markdown,+ instruction=instruction,+ **markdown_kwargs,+ )+ result.source_format = conversion.source_format or path.suffix.lstrip(".").lower()+ result.parser_name = "AnyDocParser"+ result.parser_version = "1.0"+ result.parse_time = time.time() - started+ result.meta.update(+ {+ "converter": "firecrawl-anydoc",+ "intermediate_markdown_length": len(conversion.markdown),+ "images_extracted": conversion.images_saved,+ "assets_referenced": getattr(conversion, "assets_referenced", 0),+ }+ )+ result.warnings.extend(getattr(conversion, "warnings", ()))+ return result++ async def parse_content(+ self,+ content: str,+ source_path: Optional[str] = None,+ instruction: str = "",+ **kwargs,+ ) -> ParseResult:+ result = await self._md_parser.parse_content(+ content,+ source_path=source_path,+ instruction=instruction,+ **kwargs,+ )+ suffix = Path(source_path).suffix.lstrip(".").lower() if source_path else ""+ result.source_format = suffix or "anydoc"+ result.parser_name = "AnyDocParser"+ result.parser_version = "1.0"+ return resultdiff --git a/openviking/parse/parsers/anydoc_converter.py b/openviking/parse/parsers/anydoc_converter.pynew file mode 100644index 000000000..0cbb8e0a2--- /dev/null+++ b/openviking/parse/parsers/anydoc_converter.py@@ -0,0 +1,87 @@+"""Shared anydoc -> Markdown converter."""++from __future__ import annotations++from dataclasses import dataclass+from pathlib import Path+from typing import Any++from openviking.parse.parsers.anydoc_renderer import _AnyDocMarkdownRenderer, _attr+++@dataclass(frozen=True)+class AnydocConversionResult:+ markdown: str+ images_saved: int+ source_format: str+ assets_referenced: int = 0+ warnings: tuple[str, ...] = ()+++def _format_name(value: Any) -> str | None:+ if value is None:+ return None+ name = _attr(value, "value", "name", default=value)+ return str(name).split(".")[-1].lower()+++def _load_document(path: Path, format_hint: str | None = None):+ """Load an anydoc document while keeping the import optional for unit tests."""+ if path.suffix.lower() == ".pdf" or _format_name(format_hint) == "pdf":+ raise ValueError("AnyDocConverter does not support PDF; use PDFParser")++ import anydoc++ data = path.read_bytes()+ detected = anydoc.format_from_bytes(data)+ detected_name = _format_name(detected)+ source_format = (+ _format_name(format_hint)+ or detected_name+ or _format_name(anydoc.format_from_extension(path.suffix))+ )+ if source_format == "pdf":+ raise ValueError("AnyDocConverter does not support PDF; use PDFParser")++ try:+ document = (+ anydoc.to_document(data, source_format)+ if source_format+ else anydoc.to_document(data)+ )+ except anydoc.ConvertError as exc:+ format_context = f" as {source_format}" if source_format else ""+ raise type(exc)(f"Failed to convert {path}{format_context}: {exc}") from exc+ return source_format or path.suffix.lstrip(".").lower() or "unknown", document+++class AnyDocConverter:+ def convert(+ self,+ path: Path,+ *,+ resource_name: str,+ storage: Any,+ format_hint: str | None = None,+ max_table_rows: int = 1000,+ ) -> AnydocConversionResult:+ path = Path(path)+ if path.suffix.lower() == ".pdf" or _format_name(format_hint) == "pdf":+ raise ValueError("AnyDocConverter does not support PDF; use PDFParser")++ source_format, document = _load_document(path, format_hint=format_hint)+ renderer = _AnyDocMarkdownRenderer(+ document,+ source_format=source_format,+ resource_name=resource_name,+ storage=storage,+ max_table_rows=max_table_rows,+ )+ markdown = renderer.render()+ return AnydocConversionResult(+ markdown=markdown,+ images_saved=renderer.images_saved,+ source_format=str(source_format).lower().removeprefix("format."),+ assets_referenced=len(renderer.assets_referenced),+ warnings=tuple(renderer.warnings),+ )diff --git a/openviking/parse/parsers/anydoc_renderer.py b/openviking/parse/parsers/anydoc_renderer.pynew file mode 100644index 000000000..0e5017461--- /dev/null+++ b/openviking/parse/parsers/anydoc_renderer.py@@ -0,0 +1,792 @@+"""Render firecrawl-anydoc document models into Markdown with OV media rewrites."""++from __future__ import annotations++import html+import mimetypes+import re+from dataclasses import dataclass+from pathlib import Path+from typing import Any, Iterable, Iterator++from openviking.parse.image_validation import is_valid_image+from openviking_cli.utils.logger import get_logger++logger = get_logger(__name__)++_PRESENTATION_FORMATS = {"ppt", "pptx", "pptm", "pps", "ppsx", "ppsm", "pot", "odp"}+++def _attr(obj: Any, *names: str, default: Any = None) -> Any:+ for name in names:+ if obj is None:+ break+ if isinstance(obj, dict) and name in obj:+ return obj[name]+ if hasattr(obj, name):+ return getattr(obj, name)+ return default+++def _asset_id(source: Any) -> int | None:+ raw = _attr(source, "asset_id", "assetId")+ if raw is None:+ return None+ if hasattr(raw, "id"):+ return int(getattr(raw, "id", raw))+ try:+ return int(raw)+ except (TypeError, ValueError):+ return None+++@dataclass(frozen=True)+class _ResolvedAnchor:+ fragment: str+ emit_html: bool+++@dataclass+class _TextRun:+ text: str+ style: Any+++def _kind(value: Any) -> str:+ raw = _attr(value, "kind", default="")+ raw = _attr(raw, "value", "name", default=raw)+ return str(raw).split(".")[-1].lower()+++def _items(value: list[Any] | None) -> list[Any]:+ return value or []+++class _AnyDocMarkdownRenderer:+ """Render AnyDoc's public Document model while materializing image assets."""++ def __init__(+ self,+ document: Any,+ *,+ source_format: str,+ resource_name: str,+ storage: Any,+ max_table_rows: int = 1000,+ ):+ self.document = document+ self.source_format = str(source_format or "").lower().removeprefix("format.")+ self.resource_name = resource_name+ self.storage = storage+ self.max_table_rows = max_table_rows+ self.images_saved = 0+ self.asset_paths: dict[int, str | None] = {}+ self.assets_referenced: set[int] = set()+ self.warnings: list[str] = []+ self.assets = self._collect_assets()+ self.note_numbers = self._number_notes()+ self.anchors = self._resolve_anchors()++ def _collect_assets(self) -> dict[int, Any]:+ assets: dict[int, Any] = {}+ for index, asset in enumerate(_items(self.document.assets)):+ raw_id = _attr(asset.id, "id", default=asset.id)+ try:+ assets[int(raw_id)] = asset+ except (TypeError, ValueError):+ assets[index] = asset+ return assets++ def render(self) -> str:+ parts = [part for block in _items(self.document.blocks) if (part := self._render_block(block))]+ parts.extend(self._render_note_definitions())+ markdown = "\n\n".join(parts)+ return f"{markdown}\n" if markdown else ""++ def _render_blocks(self, blocks: list[Any] | None) -> str:+ return "\n\n".join(+ rendered for block in _items(blocks) if (rendered := self._render_block(block))+ )++ def _render_block(self, block: Any) -> str:+ kind = _kind(block)+ if kind == "heading":+ text = self._render_inlines(block.content, context="heading").strip()+ if not text:+ return ""+ level = min(max(int(block.level or 1), 1), 6)+ return f"{'#' * level} {text}"+ if kind == "paragraph":+ return self._render_inlines(block.content, context="block").strip()+ if kind == "list":+ return self._render_list(block.list)+ if kind == "table":+ if block.table.kind == "layout" and self._table_is_single_cell(block.table):+ cell = block.table.grid[0][0].cell+ return self._render_blocks(cell.blocks if cell else [])+ return self._render_table(block.table)+ if kind == "block_quote":+ inner = self._render_blocks(block.blocks)+ if not inner:+ return ""+ if self.source_format in _PRESENTATION_FORMATS:+ return f"### Speaker Notes\n\n{inner}"+ return "\n".join(">" if not line else f"> {line}" for line in inner.splitlines())+ if kind == "code_block":+ body = (block.text or "").rstrip("\n")+ fence = self._backtick_fence(body, 3)+ return f"{fence}{block.lang or ''}\n{body}\n{fence}"+ if kind == "rule":+ return "---"+ raise RuntimeError(f"Unsupported AnyDoc block kind: {kind}")++ def _render_inlines(+ self,+ inlines: list[Any] | None,+ *,+ context: str,+ in_label: bool = False,+ ) -> str:+ normalized = self._normalize_inlines(_items(inlines))+ parts: list[str] = []+ for index, inline in enumerate(normalized):+ if isinstance(inline, _TextRun):+ next_inline = normalized[index + 1] if index + 1 < len(normalized) else None+ trailing_active = self._next_inline_is_active(next_inline)+ rendered_so_far = "".join(parts)+ parts.append(+ self._render_text(+ inline.text,+ inline.style,+ context=context,+ in_label=in_label,+ trailing_active=trailing_active,+ at_line_start=not rendered_so_far or rendered_so_far.endswith("\n"),+ )+ )+ continue+ parts.append(self._render_inline(inline, context=context, in_label=in_label))+ return "".join(parts)++ def _render_inline(self, inline: Any, *, context: str, in_label: bool) -> str:+ kind = _kind(inline)+ if kind == "link":+ return self._render_link(inline, context=context)+ if kind == "image":+ return self._render_image(inline, context=context, in_label=in_label)+ if kind == "anchor":+ resolved = self.anchors.get(inline.anchor)+ return f'<a id="{html.escape(resolved.fragment, quote=True)}"></a>' if resolved and resolved.emit_html else ""+ if kind == "note_ref":+ number = self.note_numbers.get(inline.note_id or "")+ return f"[^{number}]" if number is not None else ""+ if kind == "line_break":+ return "\\\n" if context == "block" else "\n" if context == "table_cell" else " "+ raise RuntimeError(f"Unsupported AnyDoc inline kind: {kind}")++ def _normalize_inlines(self, inlines: Iterable[Any]) -> list[Any]:+ normalized: list[Any] = []+ plain_style = self._style_key(None)+ for inline in inlines:+ kind = _kind(inline)+ if kind == "anchor":+ resolved = self.anchors.get(inline.anchor)+ if resolved is None or not resolved.emit_html:+ continue+ if kind != "text":+ normalized.append(inline)+ continue+ if not inline.text:+ continue++ style = None if inline.text.isspace() else inline.style+ style_key = self._style_key(style)+ previous = normalized[-1] if normalized else None+ if isinstance(previous, _TextRun) and self._style_key(previous.style) == style_key:+ previous.text += inline.text+ continue+ if (+ style_key != plain_style+ and not style_key[3]+ and len(normalized) >= 2+ and isinstance(normalized[-1], _TextRun)+ and normalized[-1].text.isspace()+ and self._style_key(normalized[-1].style) == plain_style+ and isinstance(normalized[-2], _TextRun)+ and self._style_key(normalized[-2].style) == style_key+ ):+ whitespace = normalized.pop().text+ normalized[-1].text += whitespace + inline.text+ continue+ normalized.append(_TextRun(text=inline.text, style=style))+ return normalized++ def _render_text(+ self,+ text: str,+ style: Any,+ *,+ context: str,+ in_label: bool,+ trailing_active: bool,+ at_line_start: bool,+ ) -> str:+ bold, italic, strike, code, underline = self._style_key(style)+ if not any((bold, italic, strike, code, underline)):+ return self._escape_text(+ text,+ context=context,+ in_label=in_label,+ trailing_active=trailing_active,+ at_line_start=at_line_start,+ )++ leading = text[: len(text) - len(text.lstrip())]+ trailing = text[len(text.rstrip()) :]+ core_end = len(text) - len(trailing) if trailing else len(text)+ core = text[len(leading) : core_end]+ if not core:+ return text+ if code:+ flattened = core.replace("\n", " ")+ fence = self._backtick_fence(flattened, 1)+ pad = " " if flattened.startswith("`") or flattened.endswith("`") else ""+ rendered = f"{fence}{pad}{flattened}{pad}{fence}"+ else:+ opening = ""+ closing = ""+ if strike:+ opening += "~~"+ closing = "~~" + closing+ if bold:+ opening += "**"+ closing = "**" + closing+ if italic:+ opening += "*"+ closing = "*" + closing+ escaped = self._escape_text(core, context=context, styled=True, in_label=in_label)+ rendered = f"{opening}{escaped}{closing}"+ if underline:+ rendered = f"<ins>{rendered}</ins>"+ return f"{leading}{rendered}{trailing}"++ def _render_link(self, inline: Any, *, context: str) -> str:+ target = inline.target+ label = self._render_inlines(inline.content, context=context, in_label=True)+ if not target or not target.value:+ return label+ if target.kind == "anchor":+ resolved = self.anchors.get(target.value)+ if resolved is None:+ return label+ url = f"#{resolved.fragment}"+ elif target.kind in {"external", "relative"}:+ url = target.value+ else:+ raise RuntimeError(f"Unsupported AnyDoc link target kind: {target.kind}")+ if label.strip():+ return f"[{label}]({self._format_url(url)})"+ escaped = self._escape_text(url, context=context, in_label=True, trailing_active=True)+ return f"[{escaped}]({self._format_url(url)})"++ def _render_image(self, image: Any, *, context: str, in_label: bool) -> str:+ alt = (image.alt or "").strip()+ escaped_alt = self._escape_text(alt, context=context, in_label=True)+ source = image.source+ if source is None:+ return self._escape_text(alt, context=context, in_label=in_label)+ if source.kind == "external":+ return f"})" if source.url else escaped_alt+ if source.kind == "unavailable":+ self.warnings.append(f"Embedded image is unavailable: {alt or '<no alt text>'}")+ return self._escape_text(alt, context=context, in_label=in_label)+ if source.kind != "asset":+ return self._escape_text(alt, context=context, in_label=in_label)++ reference = _asset_id(source)+ if reference is None:+ self.warnings.append("AnyDoc image references an asset without an id")+ return self._escape_text(alt, context=context, in_label=in_label)+ self.assets_referenced.add(reference)+ image_ref = self._materialize_asset(reference)+ if image_ref is None:+ return self._escape_text(alt, context=context, in_label=in_label)+ return f"})"++ def _materialize_asset(self, reference: int) -> str | None:+ if reference in self.asset_paths:+ return self.asset_paths[reference]+ asset = self.assets.get(reference)+ if asset is None:+ self.warnings.append(f"AnyDoc image references missing asset {reference}")+ self.asset_paths[reference] = None+ return None+ if not asset.media_type.lower().startswith("image/"):+ self.warnings.append(+ f"AnyDoc asset {reference} is not an image ({asset.media_type}); kept as alt text"+ )+ self.asset_paths[reference] = None+ return None++ image_bytes = bytes(asset.data)+ extension = self._image_extension(asset)+ filename = f"anydoc_asset_{reference}"+ if not is_valid_image(image_bytes, Path(f"{filename}{extension}")):+ self.warnings.append(+ f"AnyDoc asset {reference} is not an ingestable image ({asset.media_type})"+ )+ self.asset_paths[reference] = None+ return None++ try:+ saved_path = self.storage.save_image(+ self.resource_name,+ image_bytes,+ filename=filename,+ extension=extension,+ )+ relative = Path(saved_path).relative_to(Path(self.storage.media_dir)).as_posix()+ except Exception as exc:+ self.warnings.append(f"Failed to save AnyDoc image asset {reference}: {exc}")+ logger.warning(+ "Failed to save anydoc image asset %s from %s",+ reference,+ self.resource_name,+ exc_info=True,+ )+ self.asset_paths[reference] = None+ return None++ self.images_saved += 1+ self.asset_paths[reference] = relative+ return relative++ def _render_list(self, list_model: Any) -> str:+ if list_model is None or not list_model.items:+ return ""+ rendered_items: list[str] = []+ loose = False+ for index, item in enumerate(list_model.items):+ ordinal = int(list_model.start) + index+ marker = self._list_marker(list_model, item, ordinal)+ checkbox = "[x] " if item.checked is True else "[ ] " if item.checked is False else ""+ body = self._render_blocks(item.blocks)+ if len(item.blocks) > 1:+ loose = True+ lines = body.splitlines() or [""]+ indent = " " * len(marker)+ rendered = f"{marker}{checkbox}{lines[0]}"+ for line in lines[1:]:+ if not line:+ loose = True+ rendered += "\n"+ else:+ rendered += f"\n{indent}{line}"+ rendered_items.append(rendered)+ return ("\n\n" if loose else "\n").join(rendered_items)++ def _render_table(self, table: Any) -> str:+ if table is None or not table.grid:+ return ""+ if table.kind == "layout" and self._table_is_single_cell(table):+ cell = table.grid[0][0].cell+ return self._render_blocks(cell.blocks if cell else [])++ grid = _items(table.grid)+ truncated_rows = 0+ if self.max_table_rows > 0 and len(grid) > self.max_table_rows:+ truncated_rows = len(grid) - self.max_table_rows+ grid = grid[: self.max_table_rows]++ rows = self._render_table_rows(grid)+ while len(rows) > 1 and all(not text and not covered for text, covered in rows[-1]):+ rows.pop()+ width = self._table_width(rows)+ if width == 0:+ return ""+ rows = [row[:width] for row in rows]+ header = [text for text, _ in rows.pop(0)] if table.header_rows >= 1 else [""] * width+ lines = [self._format_table_row(header), self._format_table_row(["---"] * width)]+ lines.extend(self._format_table_row([text for text, _ in row]) for row in rows)+ if truncated_rows:+ lines.append(f"\n*... {truncated_rows} more rows truncated ...*")+ return "\n".join(lines)++ def _render_table_rows(self, grid: list[Any]) -> list[list[tuple[str, bool]]]:+ width = max((len(row) for row in grid), default=0)+ rows: list[list[tuple[str, bool]]] = []+ for row in grid:+ rendered = [+ (self._render_cell(slot.cell), False)+ if slot.kind == "origin"+ else ("", True)+ for slot in row+ ]+ rendered.extend(("", False) for _ in range(width - len(rendered)))+ rows.append(rendered)+ return rows++ def _render_cell(self, cell: Any | None) -> str:+ if cell is None:+ return ""+ parts: list[str] = []+ for block in cell.blocks:+ kind = _kind(block)+ if kind == "heading":+ text = self._render_inlines(block.content, context="table_cell").strip()+ if text:+ parts.append(f"**{text}**")+ elif kind == "paragraph":+ text = self._render_inlines(block.content, context="table_cell")+ if text.strip():+ parts.append(text)+ elif kind == "list":+ text = self._render_list(block.list).replace("\n", " ").strip()+ if text:+ parts.append(text)+ elif kind == "table":+ parts.extend(self._flatten_nested_table(block.table))+ elif kind == "block_quote":+ text = self._render_blocks(block.blocks).replace("\n", " ").strip()+ if text:+ parts.append(text)+ elif kind == "code_block":+ text = (block.text or "").strip()+ if text:+ fence = self._backtick_fence(text, 1)+ pad = " " if text.startswith("`") or text.endswith("`") else ""+ parts.append(f"{fence}{pad}{text}{pad}{fence}")+ elif kind != "rule":+ raise RuntimeError(f"Unsupported AnyDoc table-cell block kind: {kind}")+ return "<br>".join(line.strip() for line in "<br>".join(parts).splitlines() if line.strip())++ def _flatten_nested_table(self, table: Any) -> list[str]:+ lines: list[str] = []+ for row in table.grid:+ values = [+ self._render_cell(slot.cell) if slot.kind == "origin" else "" for slot in row+ ]+ if any(values):+ lines.append(" / ".join(values))+ return lines++ def _render_note_definitions(self) -> list[str]:+ rendered: list[str] = []+ notes_by_id = {note.id: note for note in _items(self.document.notes)}+ for note_id, number in sorted(self.note_numbers.items(), key=lambda item: item[1]):+ note = notes_by_id.get(note_id)+ if note is None:+ continue+ body = self._render_blocks(note.blocks)+ if not body:+ continue+ lines = body.splitlines()+ definition = f"[^{number}]: {lines[0]}"+ if len(lines) > 1:+ definition += "\n" + "\n".join(f" {line}" if line else "" for line in lines[1:])+ rendered.append(definition)+ return rendered++ def _number_notes(self) -> dict[str, int]:+ notes = {note.id: note for note in _items(self.document.notes) if note.blocks}+ order: list[str] = []+ seen: set[str] = set()++ def add(note_id: str | None) -> None:+ if note_id and note_id in notes and note_id not in seen:+ seen.add(note_id)+ order.append(note_id)+ for inline in self._walk_inlines(notes[note_id].blocks):+ if _kind(inline) == "note_ref":+ add(inline.note_id)++ for inline in self._walk_inlines(self.document.blocks):+ if _kind(inline) == "note_ref":+ add(inline.note_id)+ for note_id in notes:+ add(note_id)+ return {note_id: index for index, note_id in enumerate(order, 1)}++ def _resolve_anchors(self) -> dict[str, _ResolvedAnchor]:+ linked: set[str] = set()+ headings: list[Any] = []+ anchors: list[str] = []++ for block in self._walk_blocks(self.document.blocks):+ if _kind(block) == "heading":+ headings.append(block)+ for inline in self._block_inlines(block):+ kind = _kind(inline)+ if kind == "link" and inline.target and inline.target.kind == "anchor":+ linked.add(inline.target.value)+ elif kind == "anchor" and inline.anchor:+ anchors.append(inline.anchor)+ for note in _items(self.document.notes):+ for inline in self._walk_inlines(note.blocks):+ if _kind(inline) == "anchor" and inline.anchor:+ anchors.append(inline.anchor)++ resolved: dict[str, _ResolvedAnchor] = {}+ used: set[str] = set()+ for heading in headings:+ slug = self._claim_anchor(self._gfm_slug(self._plain_text(heading.content)), used)+ heading_ids = [heading.anchor] if heading.anchor else []+ heading_ids.extend(self._inline_anchor_ids(heading.content))+ for anchor_id in heading_ids:+ resolved.setdefault(anchor_id, _ResolvedAnchor(slug, False))+ for anchor_id in anchors:+ if anchor_id in linked and anchor_id not in resolved:+ fragment = self._claim_anchor(self._sanitize_anchor(anchor_id), used)+ resolved[anchor_id] = _ResolvedAnchor(fragment, True)+ return resolved++ def _walk_blocks(self, blocks: list[Any] | None) -> Iterator[Any]:+ for block in _items(blocks):+ yield block+ kind = _kind(block)+ if kind == "list" and block.list:+ for item in block.list.items:+ yield from self._walk_blocks(item.blocks)+ elif kind == "table" and block.table:+ for row in block.table.grid:+ for slot in row:+ if slot.kind == "origin" and slot.cell:+ yield from self._walk_blocks(slot.cell.blocks)+ elif kind == "block_quote":+ yield from self._walk_blocks(block.blocks)++ def _walk_inlines(self, blocks: list[Any] | None) -> Iterator[Any]:+ for block in self._walk_blocks(blocks):+ yield from self._block_inlines(block)++ def _block_inlines(self, block: Any) -> Iterator[Any]:+ if _kind(block) not in {"heading", "paragraph"}:+ return+ yield from self._inline_tree(block.content)++ def _inline_tree(self, inlines: list[Any] | None) -> Iterator[Any]:+ for inline in _items(inlines):+ yield inline+ if _kind(inline) == "link":+ yield from self._inline_tree(inline.content)++ def _plain_text(self, inlines: list[Any] | None) -> str:+ parts: list[str] = []+ for inline in _items(inlines):+ kind = _kind(inline)+ if kind == "text":+ parts.append(inline.text or "")+ elif kind == "link":+ parts.append(self._plain_text(inline.content))+ elif kind == "image":+ parts.append(inline.alt or "")+ elif kind == "line_break":+ parts.append(" ")+ return "".join(parts)++ def _inline_anchor_ids(self, inlines: list[Any] | None) -> list[str]:+ ids: list[str] = []+ for inline in _items(inlines):+ kind = _kind(inline)+ if kind == "anchor" and inline.anchor:+ ids.append(inline.anchor)+ elif kind == "link":+ ids.extend(self._inline_anchor_ids(inline.content))+ return ids++ @staticmethod+ def _next_inline_is_active(inline: Any) -> bool:+ if inline is None:+ return False+ return not isinstance(inline, _TextRun) or _AnyDocMarkdownRenderer._style_key(inline.style) != (+ False,+ False,+ False,+ False,+ False,+ )++ @staticmethod+ def _table_is_single_cell(table: Any) -> bool:+ return len(table.grid) == 1 and len(table.grid[0]) == 1 and table.grid[0][0].kind == "origin"++ @staticmethod+ def _table_width(rows: list[list[tuple[str, bool]]]) -> int:+ return max(+ (+ max((index + 1 for index, cell in enumerate(row) if cell[0] or cell[1]), default=0)+ for row in rows+ ),+ default=0,+ )++ @staticmethod+ def _format_table_row(cells: Iterable[str]) -> str:+ return "|" + "".join(f" {cell} |" for cell in cells)++ @staticmethod+ def _style_key(style: Any) -> tuple[bool, bool, bool, bool, bool]:+ if style is None:+ return False, False, False, False, False+ return (+ bool(_attr(style, "bold", default=False)),+ bool(_attr(style, "italic", default=False)),+ bool(_attr(style, "strike", default=False)),+ bool(_attr(style, "code", default=False)),+ bool(_attr(style, "underline", "underlined", default=False)),+ )++ @staticmethod+ def _list_marker(list_model: Any, item: Any, ordinal: int) -> str:+ if item.marker_label:+ return f"- {item.marker_label} "+ if list_model.marker == "bullet":+ return "- "+ if list_model.marker == "decimal":+ return f"{ordinal}. "+ return f"- {_AnyDocMarkdownRenderer._marker_label(list_model.marker, ordinal)} "++ @staticmethod+ def _claim_anchor(base: str, used: set[str]) -> str:+ if base not in used:+ used.add(base)+ return base+ suffix = 1+ while f"{base}-{suffix}" in used:+ suffix += 1+ claimed = f"{base}-{suffix}"+ used.add(claimed)+ return claimed++ @staticmethod+ def _gfm_slug(text: str) -> str:+ slug = "".join(+ "-" if char == " " else char.lower()+ for char in text.strip()+ if char == " " or char == "-" or char == "_" or char.isalnum()+ )+ return slug or "section"++ @staticmethod+ def _sanitize_anchor(anchor: str) -> str:+ sanitized = re.sub(r"[^a-z0-9_-]+", "-", anchor.lower()).strip("-")+ return sanitized or "anchor"++ @staticmethod+ def _marker_label(marker: str, number: int) -> str:+ if marker in {"lower_alpha", "upper_alpha"}:+ alpha = ""+ remaining = max(number, 1)+ while remaining:+ remaining, offset = divmod(remaining - 1, 26)+ alpha = chr(ord("a") + offset) + alpha+ return f"{alpha.upper() if marker == 'upper_alpha' else alpha}."+ if marker in {"lower_roman", "upper_roman"}:+ remaining = max(number, 1)+ result = ""+ for unit, symbol in (+ (1000, "M"),+ (900, "CM"),+ (500, "D"),+ (400, "CD"),+ (100, "C"),+ (90, "XC"),+ (50, "L"),+ (40, "XL"),+ (10, "X"),+ (9, "IX"),+ (5, "V"),+ (4, "IV"),+ (1, "I"),+ ):+ while remaining >= unit:+ result += symbol+ remaining -= unit+ return f"{result.lower() if marker == 'lower_roman' else result}."+ raise RuntimeError(f"Unsupported AnyDoc list marker: {marker}")++ @staticmethod+ def _format_url(url: str) -> str:+ escaped = "".join(+ "%7C" if char == "|" else "%3C" if char == "<" else "%3E" if char == ">" else char+ for char in url+ if ord(char) >= 32 and ord(char) != 127+ )+ return (+ f"<{escaped}>" if any(char.isspace() or char in "()" for char in escaped) else escaped+ )++ @staticmethod+ def _backtick_fence(text: str, minimum: int) -> str:+ longest = max((len(run) for run in re.findall(r"`+", text)), default=0)+ return "`" * max(longest + 1, minimum)++ @staticmethod+ def _escape_text(+ text: str,+ *,+ context: str,+ styled: bool = False,+ trailing_active: bool = False,+ in_label: bool = False,+ at_line_start: bool = False,+ ) -> str:+ characters = list(text)+ output: list[str] = []+ line_has_content = not at_line_start+ for index, char in enumerate(characters):+ if char == "\n":+ output.append(char)+ if context == "block":+ line_has_content = False+ continue+ start_of_line = not line_has_content+ if not char.isspace():+ line_has_content = True+ next_char = characters[index + 1] if index + 1 < len(characters) else None+ next_nonspace = trailing_active if next_char is None else not next_char.isspace()+ later = characters[index + 1 :]+ escape = False+ if char == "\\":+ escape = True+ elif char == "]" and in_label:+ escape = True+ elif char == "`":+ escape = styled or "`" in later+ elif char == "*":+ escape = styled or start_of_line or (next_nonspace and "*" in later)+ elif char == "_":+ previous_alnum = index > 0 and characters[index - 1].isalnum()+ next_alnum = bool(next_char and next_char.isalnum())+ escape = styled or (+ next_nonspace and not (previous_alnum and next_alnum) and "_" in later+ )+ elif char == "~":+ escape = styled or (next_nonspace and "~" in later)+ elif char == "[":+ escape = in_label or "]" in later+ elif char == "<":+ escape = bool(next_char and (next_char.isalpha() or next_char in "/!?"))+ elif char == "!":+ escape = next_char is None and trailing_active+ elif char == "|" and context == "table_cell":+ escape = True+ elif char in "#>" and start_of_line:+ escape = True+ elif char in "-+" and start_of_line:+ escape = next_char is None or next_char.isspace()+ if escape:+ output.append("\\")+ output.append(char)+ return "".join(output)++ @staticmethod+ def _image_extension(asset: Any) -> str:+ extension = mimetypes.guess_extension(asset.media_type.split(";", 1)[0].strip())+ if extension == ".jpe":+ extension = ".jpg"+ if not extension:+ extension = Path(asset.origin_part or "").suffix.lower()+ if not extension or not re.fullmatch(r"\.[a-z0-9]{1,10}", extension):+ extension = ".png"+ return extension if extension.startswith(".") else f".{extension}"diff --git a/openviking/parse/parsers/epub.py b/openviking/parse/parsers/epub.pydeleted file mode 100644index 6ecc76bbc..000000000--- a/openviking/parse/parsers/epub.py+++ /dev/null@@ -1,215 +0,0 @@-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.-# SPDX-License-Identifier: AGPL-3.0-"""-EPub (.epub) parser for OpenViking.--Converts EPub e-books to Markdown then parses using MarkdownParser.-Inspired by microsoft/markitdown approach.-"""--import asyncio-import html-import re-import zipfile-from html.parser import HTMLParser-from pathlib import Path-from typing import List, Optional, Union--from openviking.parse.base import ParseResult-from openviking.parse.parsers.base_parser import BaseParser-from openviking_cli.utils.config.parser_config import ParserConfig-from openviking_cli.utils.logger import get_logger--logger = get_logger(__name__)---class _EPubMarkdownParser(HTMLParser):- """Convert HTML fragments to simple markdown without regex tag stripping."""-- _HEADER_PREFIX = {"h1": "# ", "h2": "## ", "h3": "### ", "h4": "#### "}-- def __init__(self):- super().__init__(convert_charrefs=True)- self._parts: list[str] = []- self._ignored_tag_stack = 0-- def handle_starttag(self, tag: str, attrs):- normalized_tag = tag.lower()- if normalized_tag in {"script", "style"}:- self._ignored_tag_stack += 1- return- if self._ignored_tag_stack:- return- if normalized_tag in self._HEADER_PREFIX:- self._ensure_block_break()- self._parts.append(self._HEADER_PREFIX[normalized_tag])- elif normalized_tag in {"strong", "b"}:- self._parts.append("**")- elif normalized_tag in {"em", "i"}:- self._parts.append("*")- elif normalized_tag == "li":- self._parts.append("\n- ")- elif normalized_tag == "br":- self._parts.append("\n")- elif normalized_tag in {"p", "div", "section", "article"}:- self._ensure_block_break()-- def handle_endtag(self, tag: str):- normalized_tag = tag.lower()- if normalized_tag in {"script", "style"}:- if self._ignored_tag_stack:- self._ignored_tag_stack -= 1- return- if self._ignored_tag_stack:- return- if normalized_tag in self._HEADER_PREFIX or normalized_tag in {- "p",- "div",- "section",- "article",- }:- self._parts.append("\n\n")- elif normalized_tag in {"strong", "b"}:- self._parts.append("**")- elif normalized_tag in {"em", "i"}:- self._parts.append("*")-- def handle_data(self, data: str):- if not self._ignored_tag_stack:- self._parts.append(data)-- def get_markdown(self) -> str:- return "".join(self._parts)-- def _ensure_block_break(self):- if self._parts and not self._parts[-1].endswith("\n"):- self._parts.append("\n\n")---class EPubParser(BaseParser):- """- EPub e-book parser for OpenViking.-- Supports: .epub-- Converts EPub e-books to Markdown using ebooklib (if available)- or falls back to manual extraction, then delegates to MarkdownParser.- """-- def __init__(self, config: Optional[ParserConfig] = None):- """Initialize EPub parser."""- from openviking.parse.parsers.markdown import MarkdownParser-- self._md_parser = MarkdownParser(config=config)- self.config = config or ParserConfig()-- @property- def supported_extensions(self) -> List[str]:- return [".epub"]-- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:- """Parse EPub e-book from file path."""- path = Path(source)-- if path.exists():- markdown_content = await asyncio.to_thread(self._convert_to_markdown, path)- result = await self._md_parser.parse_content(- markdown_content, source_path=str(path), instruction=instruction, **kwargs- )- else:- result = await self._md_parser.parse_content(- str(source), instruction=instruction, **kwargs- )- result.source_format = "epub"- result.parser_name = "EPubParser"- return result-- async def parse_content(- self, content: str, source_path: Optional[str] = None, instruction: str = "", **kwargs- ) -> ParseResult:- """Parse content - delegates to MarkdownParser."""- result = await self._md_parser.parse_content(content, source_path, **kwargs)- result.source_format = "epub"- result.parser_name = "EPubParser"- return result-- def _convert_to_markdown(self, path: Path) -> str:- """Convert EPub e-book to Markdown string."""- # Try using ebooklib first- try:- import ebooklib- from ebooklib import epub-- return self._convert_with_ebooklib(path, ebooklib, epub)- except ImportError:- pass-- # Fall back to manual extraction- return self._convert_manual(path)-- def _convert_with_ebooklib(self, path: Path, ebooklib, epub) -> str:- """Convert EPub using ebooklib."""- book = epub.read_epub(path)- markdown_parts = []-- title = self._get_metadata(book, "title")- author = self._get_metadata(book, "creator")-- if title:- markdown_parts.append(f"# {title}")- if author:- markdown_parts.append(f"**Author:** {author}")-- for item in book.get_items():- if item.get_type() == ebooklib.ITEM_DOCUMENT:- content = item.get_content().decode("utf-8", errors="ignore")- md_content = self._html_to_markdown(content)- if md_content.strip():- markdown_parts.append(md_content)-- return "\n\n".join(markdown_parts)-- def _get_metadata(self, book, key: str) -> str:- """Get metadata from EPub book."""- try:- metadata = book.get_metadata("DC", key)- if metadata:- return metadata[0][0]- except Exception:- pass- return ""-- def _convert_manual(self, path: Path) -> str:- """Convert EPub manually using zipfile and HTML parsing."""- markdown_parts = []-- with zipfile.ZipFile(path, "r") as zf:- html_files = [f for f in zf.namelist() if f.endswith((".html", ".xhtml", ".htm"))]-- for html_file in sorted(html_files):- try:- content = zf.read(html_file).decode("utf-8", errors="ignore")- md_content = self._html_to_markdown(content)- if md_content.strip():- markdown_parts.append(md_content)- except Exception as e:- logger.warning(f"Failed to process {html_file}: {e}")-- return (- "\n\n".join(markdown_parts)- if markdown_parts- else "# EPub Content\n\nUnable to extract content."- )-- def _html_to_markdown(self, html_content: str) -> str:- """Simple HTML to markdown conversion."""- parser = _EPubMarkdownParser()- parser.feed(html_content)- parser.close()- markdown = html.unescape(parser.get_markdown())-- # Normalize whitespace- markdown = re.sub(r"\n\s*\n", "\n\n", markdown)- markdown = re.sub(r"[ \t]+", " ", markdown)-- return markdown.strip()diff --git a/openviking/parse/parsers/excel.py b/openviking/parse/parsers/excel.pydeleted file mode 100644index d8c610c77..000000000--- a/openviking/parse/parsers/excel.py+++ /dev/null@@ -1,440 +0,0 @@-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.-# SPDX-License-Identifier: AGPL-3.0-"""-Excel (.xlsx/.xls/.xlsm) parser for OpenViking.--Converts Excel spreadsheets to Markdown then parses using MarkdownParser.-Inspired by microsoft/markitdown approach.-"""--import asyncio-import concurrent.futures-import time-from dataclasses import asdict, fields-from functools import partial-from multiprocessing import get_context-from pathlib import Path-from typing import Any, Dict, List, Optional, Union--from openviking.parse.base import NodeType, ParseResult, ResourceNode, create_parse_result-from openviking.parse.parsers.base_parser import BaseParser-from openviking_cli.utils.config.parser_config import ExcelConfig, ParserConfig-from openviking_cli.utils.logger import get_logger--logger = get_logger(__name__)--_EXCEL_LAYOUT_EXECUTOR: Optional[concurrent.futures.ProcessPoolExecutor] = None-_EXCEL_LAYOUT_EXECUTOR_WORKERS: Optional[int] = None--# Non-configurable process-pool internals.-_EXCEL_PROCESS_POOL_START_METHOD = "spawn"-_EXCEL_PROCESS_POOL_MIN_BYTES = 200_000-_EXCEL_PROCESS_POOL_TIMEOUT_S = 120.0---def _get_excel_layout_executor(workers: int) -> concurrent.futures.ProcessPoolExecutor:- global _EXCEL_LAYOUT_EXECUTOR, _EXCEL_LAYOUT_EXECUTOR_WORKERS- workers = max(1, int(workers))- if _EXCEL_LAYOUT_EXECUTOR is None:- _EXCEL_LAYOUT_EXECUTOR = concurrent.futures.ProcessPoolExecutor(- max_workers=workers,- mp_context=get_context(_EXCEL_PROCESS_POOL_START_METHOD),- )- _EXCEL_LAYOUT_EXECUTOR_WORKERS = workers- logger.info(- f"[ExcelParserProcessPool] Started process pool "- f"workers={workers} start_method={_EXCEL_PROCESS_POOL_START_METHOD}"- )- elif _EXCEL_LAYOUT_EXECUTOR_WORKERS != workers:- logger.warning(- "[ExcelParserProcessPool] Ignoring process_pool_workers=%s; "- "pool already started with workers=%s",- workers,- _EXCEL_LAYOUT_EXECUTOR_WORKERS,- )- return _EXCEL_LAYOUT_EXECUTOR---def _build_excel_layout_in_process(- *,- path_str: str,- temp_uri: str,- instruction: str,- layout_kwargs: Dict[str, Any],- config_dict: Dict[str, Any],- max_rows_per_sheet: int,-) -> Dict[str, Any]:- """CPU-only child process worker.-- The worker must not touch VikingFS, DB, queues, or RequestContext. It only- converts Excel to markdown and computes MarkdownParser layout ops.- """- import asyncio-- from openviking.parse.parsers.excel import ExcelParser- from openviking_cli.utils.config.parser_config import ParserConfig-- started = time.perf_counter()- path = Path(path_str)- allowed_config_fields = {field.name for field in fields(ParserConfig)}- filtered_config = {- key: value for key, value in config_dict.items() if key in allowed_config_fields- }- parser = ExcelParser(- config=ParserConfig.from_dict(filtered_config),- max_rows_per_sheet=max_rows_per_sheet,- )-- convert_started = time.perf_counter()- if path.suffix.lower() == ".xls":- markdown_content = parser._convert_xls_to_markdown(path)- else:- import openpyxl-- markdown_content = parser._convert_to_markdown(path, openpyxl)- convert_s = time.perf_counter() - convert_started-- layout_started = time.perf_counter()- layout = asyncio.run(- parser._md_parser._compute_layout(- markdown_content,- temp_uri,- source_path=str(path),- instruction=instruction,- **layout_kwargs,- )- )- layout_s = time.perf_counter() - layout_started-- return {- "layout": layout,- "convert_s": convert_s,- "layout_s": layout_s,- "total_s": time.perf_counter() - started,- "markdown_chars": len(markdown_content),- "layout_ops": len(layout.ops),- "layout_write_ops": sum(1 for op in layout.ops if op.kind == "write"),- "layout_mkdir_ops": sum(1 for op in layout.ops if op.kind == "mkdir"),- }---class ExcelParser(BaseParser):- """- Excel spreadsheet parser for OpenViking.-- Supports: .xlsx, .xls, .xlsm-- Converts Excel spreadsheets to Markdown using openpyxl,- then delegates to MarkdownParser for tree structure creation.- """-- def __init__(- self, config: Optional[ParserConfig] = None, max_rows_per_sheet: int = 1000- ):- """- Initialize Excel parser.-- Args:- config: Parser configuration (prefer ExcelConfig for process-pool knobs)- max_rows_per_sheet: Maximum rows to process per sheet (0 = unlimited)- """- from openviking.parse.parsers.markdown import MarkdownParser-- self._md_parser = MarkdownParser(config=config)- self.config = config or ExcelConfig()- self.max_rows_per_sheet = max_rows_per_sheet-- def _process_pool_enabled(self) -> bool:- return bool(getattr(self.config, "enable_process_pool", False))-- def _process_pool_workers(self) -> int:- return max(1, int(getattr(self.config, "process_pool_workers", 2) or 2))-- @property- def supported_extensions(self) -> List[str]:- return [".xlsx", ".xls", ".xlsm"]-- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:- """Parse Excel spreadsheet from file path."""- path = Path(source)-- if path.exists():- result = await self._parse_existing_path(path, instruction=instruction, **kwargs)- else:- result = await self._md_parser.parse_content(- str(source), instruction=instruction, **kwargs- )- result.source_format = path.suffix.lstrip(".") if path.exists() else "xlsx"- result.parser_name = "ExcelParser"- return result-- async def _parse_existing_path(- self, path: Path, instruction: str = "", **kwargs- ) -> ParseResult:- if self._should_use_process_pool(path, kwargs):- try:- return await self._parse_existing_path_process_pool(- path, instruction=instruction, **kwargs- )- except Exception as exc:- logger.warning(- f"[ExcelParserProcessPool] Falling back to in-process parse: {exc}",- exc_info=True,- )-- # Use xlrd for legacy .xls, openpyxl for .xlsx/.xlsm- if path.suffix.lower() == ".xls":- markdown_content = await asyncio.to_thread(self._convert_xls_to_markdown, path)- else:- import openpyxl-- markdown_content = await asyncio.to_thread(self._convert_to_markdown, path, openpyxl)- return await self._md_parser.parse_content(- markdown_content, source_path=str(path), instruction=instruction, **kwargs- )-- def _should_use_process_pool(self, path: Path, kwargs: Dict[str, Any]) -> bool:- if not self._process_pool_enabled():- return False- if path.suffix.lower() == ".xls":- return False- if kwargs.get("enable_link_rewrite") or kwargs.get("base_dir") or kwargs.get("allowed_media_dirs"):- logger.debug("[ExcelParserProcessPool] Disabled for link/media rewrite parse")- return False- try:- return path.stat().st_size >= _EXCEL_PROCESS_POOL_MIN_BYTES- except OSError:- return False-- async def _parse_existing_path_process_pool(- self, path: Path, instruction: str = "", **kwargs- ) -> ParseResult:- parse_started = time.perf_counter()- loop = asyncio.get_running_loop()- temp_uri = self._md_parser._create_temp_uri()- layout_kwargs: Dict[str, Any] = {- key: value- for key, value in kwargs.items()- if key in {"resource_name", "source_name"} and isinstance(value, str)- }- if isinstance(kwargs.get("split_content"), bool):- layout_kwargs["split_content"] = kwargs["split_content"]- future = loop.run_in_executor(- _get_excel_layout_executor(self._process_pool_workers()),- partial(- _build_excel_layout_in_process,- path_str=str(path),- temp_uri=temp_uri,- instruction=instruction,- layout_kwargs=layout_kwargs,- config_dict=asdict(self.config),- max_rows_per_sheet=self.max_rows_per_sheet,- ),- )-- worker_started = time.perf_counter()- worker_result = await asyncio.wait_for(- future, timeout=_EXCEL_PROCESS_POOL_TIMEOUT_S- )- worker_s = time.perf_counter() - worker_started- layout = worker_result["layout"]-- self._md_parser._rewrite_ctx = {- "enabled": False,- "source_path": str(path),- "doc_name": layout.doc_name,- "root_dir": layout.root_dir,- "import_root": None,- "base_dir": None,- "allowed_media_dirs": None,- }- try:- apply_started = time.perf_counter()- await self._md_parser._apply_layout(layout)- apply_s = time.perf_counter() - apply_started- finally:- self._md_parser._rewrite_ctx = None-- parse_time = time.perf_counter() - parse_started- logger.info(- f"[ExcelParserProcessPool] path={path} total={parse_time:.3f}s "- f"worker_wall={worker_s:.3f}s convert={worker_result.get('convert_s', -1.0):.3f}s "- f"layout={worker_result.get('layout_s', -1.0):.3f}s apply={apply_s:.3f}s "- f"chars={worker_result.get('markdown_chars')} ops={worker_result.get('layout_ops')} "- f"writes={worker_result.get('layout_write_ops')} mkdirs={worker_result.get('layout_mkdir_ops')}"- )-- root = ResourceNode(- type=NodeType.ROOT,- title=layout.doc_title,- level=0,- meta=layout.meta.get("frontmatter", {}),- )- result = create_parse_result(- root=root,- source_path=str(path),- source_format="markdown",- parser_name="MarkdownParser",- parse_time=parse_time,- meta=layout.meta,- warnings=layout.warnings,- )- result.temp_dir_path = layout.temp_uri- return result-- async def parse_content(- self, content: str, source_path: Optional[str] = None, instruction: str = "", **kwargs- ) -> ParseResult:- """Parse content - delegates to MarkdownParser."""- result = await self._md_parser.parse_content(content, source_path, **kwargs)- result.source_format = "xlsx"- result.parser_name = "ExcelParser"- return result-- def _convert_xls_to_markdown(self, path: Path) -> str:- """Convert legacy .xls spreadsheet to Markdown using xlrd."""- import xlrd-- # formatting_info=True enables xlrd to detect date cells via XL_CELL_DATE- # instead of reporting them as XL_CELL_NUMBER with raw float serials- wb = xlrd.open_workbook(str(path), formatting_info=True, on_demand=True)- try:- return self._build_xls_markdown(wb, path, xlrd)- finally:- wb.release_resources()-- def _build_xls_markdown(self, wb, path: Path, xlrd) -> str:- """Build markdown from xlrd workbook."""- markdown_parts = []- markdown_parts.append(f"# {path.stem}")- markdown_parts.append(f"**Sheets:** {wb.nsheets}")-- for sheet_idx in range(wb.nsheets):- sheet = wb.sheet_by_index(sheet_idx)- parts = [f"## Sheet: {sheet.name}"]-- if sheet.nrows == 0 or sheet.ncols == 0:- parts.append("*Empty sheet*")- markdown_parts.append("\n\n".join(parts))- continue-- parts.append(f"**Dimensions:** {sheet.nrows} rows × {sheet.ncols} columns")-- rows_to_process = sheet.nrows- if self.max_rows_per_sheet > 0:- rows_to_process = min(sheet.nrows, self.max_rows_per_sheet)-- rows = []- for row_idx in range(rows_to_process):- row_data = []- for col_idx in range(sheet.ncols):- row_data.append(self._format_xls_cell(sheet.cell(row_idx, col_idx), wb, xlrd))- rows.append(row_data)-- if rows:- from openviking.parse.base import format_table_to_markdown-- parts.append(format_table_to_markdown(rows, has_header=True))-- if self.max_rows_per_sheet > 0 and sheet.nrows > self.max_rows_per_sheet:- parts.append(- f"\n*... {sheet.nrows - self.max_rows_per_sheet} more rows truncated ...*"- )-- markdown_parts.append("\n\n".join(parts))-- return "\n\n".join(markdown_parts)-- @staticmethod- def _format_xls_cell(cell, wb, xlrd) -> str:- """Format a single xlrd cell value with proper type handling."""- if cell.ctype == xlrd.XL_CELL_EMPTY or cell.ctype == xlrd.XL_CELL_BLANK:- return ""- if cell.ctype == xlrd.XL_CELL_DATE:- try:- dt = xlrd.xldate_as_tuple(cell.value, wb.datemode)- # Include time component if non-zero- if dt[3] or dt[4] or dt[5]:- return (- f"{dt[0]:04d}-{dt[1]:02d}-{dt[2]:02d} {dt[3]:02d}:{dt[4]:02d}:{dt[5]:02d}"- )- return f"{dt[0]:04d}-{dt[1]:02d}-{dt[2]:02d}"- except Exception:- return str(cell.value)- if cell.ctype == xlrd.XL_CELL_BOOLEAN:- return "TRUE" if cell.value else "FALSE"- if cell.ctype == xlrd.XL_CELL_ERROR:- # xlrd error code map- error_map = {- 0x00: "#NULL!",- 0x07: "#DIV/0!",- 0x0F: "#VALUE!",- 0x17: "#REF!",- 0x1D: "#NAME?",- 0x24: "#NUM!",- 0x2A: "#N/A",- }- return error_map.get(cell.value, f"#ERR({cell.value})")- if cell.ctype == xlrd.XL_CELL_NUMBER:- # Display integers without trailing .0- if cell.value == int(cell.value):- return str(int(cell.value))- return str(cell.value)- # XL_CELL_TEXT or fallback- return str(cell.value) if cell.value is not None else ""-- def _convert_to_markdown(self, path: Path, openpyxl) -> str:- """Convert Excel spreadsheet to Markdown string."""- wb = openpyxl.load_workbook(path, data_only=True)-- markdown_parts = []- markdown_parts.append(f"# {path.stem}")- markdown_parts.append(f"**Sheets:** {len(wb.sheetnames)}")-- for sheet_name in wb.sheetnames:- sheet = wb[sheet_name]- sheet_content = self._convert_sheet(sheet, sheet_name)- markdown_parts.append(sheet_content)-- return "\n\n".join(markdown_parts)-- def _convert_sheet(self, sheet, sheet_name: str) -> str:- """Convert a single sheet to markdown."""- parts = []- parts.append(f"## Sheet: {sheet_name}")-- max_row = sheet.max_row- max_col = sheet.max_column-- if max_row == 0 or max_col == 0:- parts.append("*Empty sheet*")- return "\n\n".join(parts)-- parts.append(f"**Dimensions:** {max_row} rows × {max_col} columns")-- rows_to_process = max_row- if self.max_rows_per_sheet > 0:- rows_to_process = min(max_row, self.max_rows_per_sheet)-- rows = []- for _row_idx, row in enumerate(- sheet.iter_rows(min_row=1, max_row=rows_to_process, values_only=True), 1- ):- row_data = []- for cell in row:- if cell is None:- row_data.append("")- else:- row_data.append(str(cell))- rows.append(row_data)-- if rows:- from openviking.parse.base import format_table_to_markdown-- table_md = format_table_to_markdown(rows, has_header=True)- parts.append(table_md)-- if self.max_rows_per_sheet > 0 and max_row > self.max_rows_per_sheet:- parts.append(f"\n*... {max_row - self.max_rows_per_sheet} more rows truncated ...*")-- return "\n\n".join(parts)diff --git a/openviking/parse/parsers/legacy_doc.py b/openviking/parse/parsers/legacy_doc.pydeleted file mode 100644index 11b48a519..000000000--- a/openviking/parse/parsers/legacy_doc.py+++ /dev/null@@ -1,408 +0,0 @@-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.-# SPDX-License-Identifier: AGPL-3.0-"""-Legacy Word document (.doc) parser for OpenViking.--Extracts text from OLE2 compound binary .doc files using olefile,-then delegates to MarkdownParser for tree structure creation.-"""--import asyncio-import struct-import zipfile-from pathlib import Path-from typing import List, Optional, Union--from openviking.parse.base import ParseResult-from openviking.parse.parsers.base_parser import BaseParser-from openviking_cli.utils.config.parser_config import ParserConfig-from openviking_cli.utils.logger import get_logger--logger = get_logger(__name__)---# Max stream size to read (50MB) — prevents DoS from crafted files-_MAX_STREAM_SIZE = 50 * 1024 * 1024-# Max character count sanity cap for ccpText-_MAX_CCP_TEXT = 10_000_000-_ZIP_SIGNATURES = (b"PK\x03\x04", b"PK\x05\x06", b"PK\x07\x08")---class LegacyDocParser(BaseParser):- """- Legacy .doc (OLE2 binary) parser.-- Extracts text content from Word 97-2003 (.doc) files using olefile- to read the WordDocument and table streams, then delegates to- MarkdownParser for tree structure.- """-- def __init__(self, config: Optional[ParserConfig] = None):- from openviking.parse.parsers.markdown import MarkdownParser-- self._md_parser = MarkdownParser(config=config)- self.config = config or ParserConfig()-- @property- def supported_extensions(self) -> List[str]:- return [".doc"]-- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:- """Parse legacy .doc file."""- path = Path(source)-- if path.exists():- if self._has_zip_signature(path):- package_type = await asyncio.to_thread(self._classify_zip_package, path)- if package_type == "docx":- # python-docx writes OOXML regardless of the filename suffix. This- # also occurs in real uploads whose extension says .doc while the- # payload is a modern Word ZIP package. Never pass those bytes to- # the legacy UTF-16 fallback: it turns ZIP data into plausible but- # meaningless Unicode and silently persists corrupted Markdown.- from openviking.parse.parsers.word import WordParser-- logger.info(- "Detected OOXML Word content in %s; routing to WordParser",- path.name,- )- return await WordParser(config=self.config).parse(- path,- instruction=instruction,- **kwargs,- )-- raise ValueError(f"{path.name} is a ZIP package, not a legacy Word .doc file")-- text = await asyncio.to_thread(self._extract_text, path)- result = await self._md_parser.parse_content(- text, source_path=str(path), instruction=instruction, **kwargs- )- else:- result = await self._md_parser.parse_content(- str(source), instruction=instruction, **kwargs- )- result.source_format = "doc"- result.parser_name = "LegacyDocParser"- return result-- @staticmethod- def _has_zip_signature(path: Path) -> bool:- """Return whether the payload starts with a recognized ZIP signature."""- with path.open("rb") as file_obj:- return file_obj.read(4) in _ZIP_SIGNATURES-- @staticmethod- def _classify_zip_package(path: Path) -> str:- """Identify a ZIP-backed Word package without extracting archive members."""- try:- with zipfile.ZipFile(path) as archive:- try:- archive.getinfo("[Content_Types].xml")- archive.getinfo("word/document.xml")- except KeyError:- return "zip"- return "docx"- except (zipfile.BadZipFile, zipfile.LargeZipFile):- # A payload with ZIP magic is still not a legacy OLE .doc. Classify- # malformed archives as generic ZIP so callers fail closed instead of- # feeding binary bytes into the permissive legacy text fallback.- return "zip"-- async def parse_content(- self, content: str, source_path: Optional[str] = None, instruction: str = "", **kwargs- ) -> ParseResult:- """Parse content string — delegates to MarkdownParser."""- result = await self._md_parser.parse_content(- content, source_path, instruction=instruction, **kwargs- )- result.source_format = "doc"- result.parser_name = "LegacyDocParser"- return result-- def _extract_text(self, path: Path) -> str:- """- Extract text from a legacy .doc OLE2 file.-- Reads the WordDocument stream and uses the FIB (File Information Block)- to locate text in the document body. Falls back to raw byte scanning- if structured extraction fails.- """- import olefile-- try:- ole = olefile.OleFileIO(str(path))- except Exception as e:- logger.warning(f"Failed to open .doc as OLE file: {e}")- return self._fallback_extract(path)-- try:- return self._extract_from_ole(ole)- except Exception as e:- logger.warning(f"Structured OLE extraction failed, using fallback: {e}")- return self._fallback_extract(path)- finally:- ole.close()-- @staticmethod- def _read_ole_stream(ole, stream_name: str) -> bytes:- """Read an OLE stream with size cap to prevent DoS."""- stream = ole.openstream(stream_name)- data = stream.read(_MAX_STREAM_SIZE + 1)- if len(data) > _MAX_STREAM_SIZE:- raise ValueError(f"OLE stream '{stream_name}' exceeds {_MAX_STREAM_SIZE} bytes")- return data-- def _extract_from_ole(self, ole) -> str:- """- Extract text from OLE streams using the Word Binary File Format.-- Reads the FIB to determine if text is stored as UTF-16 or compressed- (CP1252), then extracts the document body text from the appropriate- stream (WordDocument or table stream).- """- if not ole.exists("WordDocument"):- raise ValueError("No WordDocument stream found")-- word_doc = self._read_ole_stream(ole, "WordDocument")-- # Minimum FIB size: need at least 0x01A8 bytes for Word 97+ FIB fields- if len(word_doc) < 0x01A8:- raise ValueError(f"WordDocument stream too small ({len(word_doc)} bytes)")-- # Check FIB version (nFib at offset 0x0002) — require Word 97+ (0x00C1+)- nfib = struct.unpack_from("<H", word_doc, 0x0002)[0]- if nfib < 0x00C1:- raise ValueError(f"Unsupported Word version (nFib=0x{nfib:04X}), need Word 97+")-- # Read FIB flags at offset 0x000A- # Bit 9: table stream selector (0Table vs 1Table)- # Bit 8: fComplex (complex fast-saved format — does not affect encoding)- flags = struct.unpack_from("<H", word_doc, 0x000A)[0]- is_1table = bool(flags & 0x0200)- table_stream_name = "1Table" if is_1table else "0Table"-- # Read ccpText (character count of main document text) at FIB offset 0x004C- ccp_text = struct.unpack_from("<i", word_doc, 0x004C)[0]-- if ccp_text <= 0:- raise ValueError("ccpText is zero or negative")- # Cap ccpText to prevent memory exhaustion from crafted files- ccp_text = min(ccp_text, _MAX_CCP_TEXT)-- # Read the Clx from the table stream to find text positions- if not ole.exists(table_stream_name):- raise ValueError(f"Table stream '{table_stream_name}' not found")- table_data = self._read_ole_stream(ole, table_stream_name)-- # fcClx offset in FIB (Word 97+ standard location)- fc_clx = struct.unpack_from("<i", word_doc, 0x01A2)[0]- lcb_clx = struct.unpack_from("<i", word_doc, 0x01A6)[0]-- if fc_clx <= 0 or lcb_clx <= 0 or fc_clx + lcb_clx > len(table_data):- return self._simple_text_extract(word_doc, ccp_text)-- return self._extract_via_clx(word_doc, table_data, fc_clx, lcb_clx, ccp_text)-- def _simple_text_extract(self, word_doc: bytes, ccp_text: int) -> str:- """- Simple text extraction using FIB text offset.-- The main document text starts at offset 0x0800 in the WordDocument stream- for most Word 97+ files. Tries UTF-16LE first; falls back to CP1252 if- the stream is too small for UTF-16.- """- text_start = 0x0800 # Standard text start offset-- if text_start >= len(word_doc):- raise ValueError("WordDocument stream too small for text extraction")-- # Try UTF-16LE first (2 bytes per char)- if ccp_text * 2 + text_start <= len(word_doc):- end = text_start + ccp_text * 2- raw = word_doc[text_start:end]- text = raw.decode("utf-16-le", errors="replace")- # Sanity: if mostly printable, it's likely correct- if (- sum(1 for c in text[:200] if c.isprintable() or c in "\n\r\t")- > len(text[:200]) * 0.5- ):- return self._clean_word_text(text)-- # Fall back to CP1252 single-byte- end = min(text_start + ccp_text, len(word_doc))- raw = word_doc[text_start:end]- return self._clean_word_text(self._decode_cp1252(raw))-- def _extract_via_clx(- self,- word_doc: bytes,- table_data: bytes,- fc_clx: int,- lcb_clx: int,- ccp_text: int,- ) -> str:- """- Extract text using the Clx (piece table) structure.-- The Clx contains a PiecePLC that maps character positions to file offsets,- allowing reconstruction of the document text even when pieces are scattered.- """- clx = table_data[fc_clx : fc_clx + lcb_clx]- pos = 0- text_parts = []- chars_extracted = 0-- # Skip any Grpprl (type 0x01) entries in the Clx- while pos < len(clx) and clx[pos] == 0x01:- if pos + 3 > len(clx):- break- cb = struct.unpack_from("<H", clx, pos + 1)[0]- advance = 3 + cb- if advance <= 0:- break # Prevent infinite loop on zero-length Grpprl- pos += advance-- # Now we should be at the Pcdt (type 0x02)- if pos >= len(clx) or clx[pos] != 0x02:- return self._simple_text_extract(word_doc, ccp_text)-- pos += 1 # skip type byte- if pos + 4 > len(clx):- return self._simple_text_extract(word_doc, ccp_text)-- lcb_pcd = struct.unpack_from("<I", clx, pos)[0]- pos += 4-- # PLC structure: (n+1) CPs followed by n PCDs (each 8 bytes)- pcd_start = pos- pcd_end = pos + lcb_pcd-- if pcd_end > len(clx):- return self._simple_text_extract(word_doc, ccp_text)-- # Calculate number of pieces: (lcb_pcd - 4) / (4 + 8) per piece,- # but CPs are (n+1)*4 bytes + n*8 bytes = lcb_pcd- # So: 4*(n+1) + 8*n = lcb_pcd → 12n + 4 = lcb_pcd → n = (lcb_pcd - 4) / 12- n_pieces = (lcb_pcd - 4) // 12- if n_pieces <= 0:- return self._simple_text_extract(word_doc, ccp_text)-- # Read character positions (n+1 values)- cps = []- for i in range(n_pieces + 1):- offset = pcd_start + i * 4- if offset + 4 > len(clx):- break- cps.append(struct.unpack_from("<I", clx, offset)[0])-- # Read piece descriptors (start after the CPs)- pcd_array_start = pcd_start + (n_pieces + 1) * 4-- for i in range(min(n_pieces, len(cps) - 1)):- if chars_extracted >= ccp_text:- break-- pcd_offset = pcd_array_start + i * 8- if pcd_offset + 8 > len(clx):- break-- # PCD: 2 bytes flags, 4 bytes fc, 2 bytes prm- fc_value = struct.unpack_from("<I", clx, pcd_offset + 2)[0]-- piece_cp_start = cps[i]- piece_cp_end = cps[i + 1]- piece_char_count = piece_cp_end - piece_cp_start-- # Bit 30 of fc indicates compressed (CP1252) text- is_compressed = bool(fc_value & 0x40000000)- fc_real = fc_value & 0x3FFFFFFF-- if is_compressed:- # CP1252: fc_real / 2 is the byte offset- byte_offset = fc_real // 2- byte_end = byte_offset + piece_char_count- if byte_end <= len(word_doc):- raw = word_doc[byte_offset:byte_end]- text_parts.append(self._decode_cp1252(raw))- else:- logger.warning(- f"Piece {i} extends beyond stream ({byte_end} > {len(word_doc)})"- )- else:- # UTF-16LE- byte_offset = fc_real- byte_end = byte_offset + piece_char_count * 2- if byte_end <= len(word_doc):- raw = word_doc[byte_offset:byte_end]- text_parts.append(raw.decode("utf-16-le", errors="replace"))- else:- logger.warning(- f"Piece {i} extends beyond stream ({byte_end} > {len(word_doc)})"- )-- chars_extracted += piece_char_count-- result = self._clean_word_text("".join(text_parts))- if not result.strip():- return self._simple_text_extract(word_doc, ccp_text)- return result-- @staticmethod- def _decode_cp1252(data: bytes) -> str:- """Decode CP1252 bytes to string."""- return data.decode("cp1252", errors="replace")-- @staticmethod- def _clean_word_text(text: str) -> str:- """Normalize Word control characters to readable equivalents."""- text = text.replace("\r\n", "\n").replace("\r", "\n")- # \x07 = cell/row end, \x0B = soft line break, \x0C = section break- text = text.replace("\x07", "\t").replace("\x0b", "\n").replace("\x0c", "\n\n")- return text-- def _fallback_extract(self, path: Path) -> str:- """- Last-resort text extraction by scanning raw bytes for readable text runs.-- Tries UTF-16LE decoding first (common in .doc), then falls back to CP1252.- """- # Cap read size to prevent DoS from large files- with open(path, "rb") as f:- raw = f.read(_MAX_STREAM_SIZE)-- # Try to find UTF-16LE text (every other byte is often 0x00 for ASCII)- try:- decoded = raw.decode("utf-16-le", errors="ignore")- # Filter to printable text runs- lines = []- current = []- for ch in decoded:- if ch.isprintable() or ch in "\n\t":- current.append(ch)- else:- if len(current) > 3:- lines.append("".join(current))- current = []- if current and len(current) > 3:- lines.append("".join(current))- text = "\n".join(lines)- if len(text) > 50:- return text- except Exception:- pass-- # Fall back to CP1252- text = raw.decode("cp1252", errors="replace")- lines = []- current = []- for ch in text:- if ch.isprintable() or ch in "\n\t":- current.append(ch)- else:- if len(current) > 3:- lines.append("".join(current))- current = []- if current and len(current) > 3:- lines.append("".join(current))- return "\n".join(lines)diff --git a/openviking/parse/parsers/markdown.py b/openviking/parse/parsers/markdown.pyindex 4d267aa54..a352db2d1 100644--- a/openviking/parse/parsers/markdown.py+++ b/openviking/parse/parsers/markdown.py@@ -19,7 +19,6 @@ The parser handles scenarios:import asyncioimport hashlib-import ioimport osimport reimport time@@ -29,6 +28,7 @@ from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Unionfrom openviking.parse.accessors.mime_types import IANA_MEDIA_TYPE_TO_EXTENSIONfrom openviking.parse.base import NodeType, ParseResult, ResourceNode, create_parse_result+from openviking.parse.image_validation import is_valid_imagefrom openviking.parse.parsers.base_parser import BaseParserfrom openviking.parse.parsers.code.ast.providers import supports_code_skeletonfrom openviking.parse.parsers.constants import (@@ -134,6 +134,25 @@ class _Layout:warnings: List[str] = field(default_factory=list)+@dataclass+class _RewriteContext:+ """Link-rewrite inputs and memoized results owned by one parse_content call."""++ source_path: str+ doc_name: str+ root_dir: str+ import_root: str+ base_dir: Optional[Path]+ allowed_media_dirs: Optional[List[Path]]+ split_content: bool+ adaptive_flatten_requested: bool+ flatten_single_output: bool+ split_cache: Dict[Tuple[str, bool, bool], Optional[Dict[str, str]]] = field(+ default_factory=dict+ )+ image_probe_cache: Dict[str, bool] = field(default_factory=dict)++if TYPE_CHECKING:pass@@ -158,6 +177,7 @@ class MarkdownParser(BaseParser):# Worst-case token density used by _estimate_token_count (CJK ~0.7 tok/char).# Used to bound force-split chunk size so it also respects the token budget.MAX_TOKENS_PER_CHAR = 0.7+ TABLE_HEADER_REPEAT_MAX_CHARS = 1024MAX_MERGED_FILENAME_LENGTH = 32 # Maximum length for merged section filenames# Image validation constants@@ -203,9 +223,6 @@ class MarkdownParser(BaseParser):# Cache for VikingFS instanceself._viking_fs = None- # Relative-link rewrite context, set per parse_content() call.- self._rewrite_ctx = None-@propertydef supported_extensions(self) -> List[str]:"""Return list of supported file extensions."""@@ -299,28 +316,28 @@ class MarkdownParser(BaseParser):**layout_kwargs,)- # Set up relative-link rewrite context consumed by _write_section during- # apply. Link rewriting is opt-in via kwargs (DirectoryParser sets these);- # single-file ingestion never passes link_rewrite_root, so it never rewrites.- # base_dir/allowed_media_dirs let the rewrite ask _ingest_will_handle_image- # which image embeds ingestion will take (those are left for #2429).- self._rewrite_ctx = {- "enabled": bool(kwargs.get("enable_link_rewrite", False)),- "source_path": source_path,- "doc_name": layout.doc_name,- "root_dir": layout.root_dir,- "import_root": kwargs.get("link_rewrite_root"),- "base_dir": base_dir,- "allowed_media_dirs": allowed_media_dirs,- "split_content": split_content,- "adaptive_flatten_requested": adaptive_flatten_requested,- "flatten_single_output": flatten_single_output,- }+ import_root = kwargs.get("link_rewrite_root")+ rewrite_ctx = None+ if kwargs.get("enable_link_rewrite", False) and source_path and import_root:+ rewrite_ctx = _RewriteContext(+ source_path=source_path,+ doc_name=layout.doc_name,+ root_dir=layout.root_dir,+ import_root=import_root,+ base_dir=base_dir,+ allowed_media_dirs=allowed_media_dirs,+ split_content=split_content,+ adaptive_flatten_requested=adaptive_flatten_requested,+ flatten_single_output=flatten_single_output,+ )# Phase 2 — write only: replay the plan against the real VikingFS,# rewriting links and ingesting local images.await self._apply_layout(- layout, base_dir=base_dir, allowed_media_dirs=allowed_media_dirs+ layout,+ rewrite_ctx=rewrite_ctx,+ base_dir=base_dir,+ allowed_media_dirs=allowed_media_dirs,)parse_time = time.time() - start_time@@ -350,9 +367,6 @@ class MarkdownParser(BaseParser):except Exception as e:logger.error(f"[MarkdownParser] Parse failed: {e}", exc_info=True)raise- finally:- # Rewrite context lives exactly one parse_content call.- self._rewrite_ctx = Noneasync def _compute_layout(self,@@ -438,6 +452,7 @@ class MarkdownParser(BaseParser):self,layout: _Layout,*,+ rewrite_ctx: Optional[_RewriteContext] = None,base_dir: Optional[Path] = None,allowed_media_dirs: Optional[List[Path]] = None,) -> None:@@ -450,12 +465,7 @@ class MarkdownParser(BaseParser):write_ops = [op for op in layout.ops if op.kind == "write"]write_chars = sum(len(op.content or "") for op in write_ops)- rewrite_enabled = bool(- self._rewrite_ctx- and self._rewrite_ctx.get("enabled")- and self._rewrite_ctx.get("source_path")- and self._rewrite_ctx.get("import_root")- )+ rewrite_enabled = rewrite_ctx is not Nonemkdir_started = time.perf_counter()mkdir_count = 0@@ -467,7 +477,7 @@ class MarkdownParser(BaseParser):write_started = time.perf_counter()for op in write_ops:- await self._write_section(op.uri, op.content)+ await self._write_section(op.uri, op.content, rewrite_ctx=rewrite_ctx)write_s = time.perf_counter() - write_started# Ingest local image files, placing each image next to the markdown file@@ -622,8 +632,10 @@ class MarkdownParser(BaseParser):"""Split oversized content by paragraphs, force split single oversized paragraphs.- Enforces both a token estimate limit (max_size) and a hard character limit- (self.config.max_section_chars) to guard against token estimation errors.+ Targets both the token estimate limit (max_size) and the character limit+ (self.config.max_section_chars). A single oversized Markdown table row is+ kept intact so downstream semantic and embedding stages can apply their own+ input limits without destroying the source table structure.Args:content: Content to split@@ -663,8 +675,16 @@ class MarkdownParser(BaseParser):# to keep every chunk within the token budget.token_safe_chars = max(1, int(max_size / self.MAX_TOKENS_PER_CHAR))step = min(max_chars, token_safe_chars)- for i in range(0, len(para), step):- parts.append(para[i : i + step].strip())+ table_parts = self._split_markdown_table_by_rows(+ para,+ max_size=max_size,+ max_chars=step,+ )+ if table_parts:+ parts.extend(table_parts)+ else:+ for i in range(0, len(para), step):+ parts.append(para[i : i + step].strip())elif (current_tokens + para_tokens > max_size or len(current) + len(para) + 2 > max_chars) and current:@@ -686,6 +706,95 @@ class MarkdownParser(BaseParser):return parts if parts else [content]+ @staticmethod+ def _is_markdown_table_row(line: str) -> bool:+ stripped = line.strip()+ return stripped.startswith("|") and stripped.endswith("|") and "|" in stripped[1:-1]++ @staticmethod+ def _is_markdown_table_separator(line: str) -> bool:+ stripped = line.strip()+ if not stripped.startswith("|") or not stripped.endswith("|"):+ return False+ cells = [cell.strip() for cell in stripped.strip("|").split("|")]+ return bool(cells) and all(re.fullmatch(r":?-{3,}:?", cell or "") for cell in cells)++ def _split_markdown_table_by_rows(+ self,+ content: str,+ *,+ max_size: int,+ max_chars: int,+ ) -> Optional[List[str]]:+ """Split a single oversized Markdown table without cutting rows.++ The first two table rows are treated as the header/separator and repeated+ in subsequent chunks when they are small enough to fit. Any prefix before+ the table, such as a section heading merged into the oversized paragraph,+ is kept on the first chunk only. A single row may exceed the requested+ limits because row integrity takes precedence at this source-storage stage.+ """+ lines = content.splitlines()+ table_start = -1+ for index in range(len(lines) - 1):+ if self._is_markdown_table_row(lines[index]) and self._is_markdown_table_separator(+ lines[index + 1]+ ):+ table_start = index+ break+ if table_start < 0:+ return None++ table_end = table_start + 2+ while table_end < len(lines) and self._is_markdown_table_row(lines[table_end]):+ table_end += 1++ if table_end - table_start < 3:+ return None+ if any(line.strip() for line in lines[table_end:]):+ return None++ prefix = "\n".join(lines[:table_start]).strip()+ header = lines[table_start : table_start + 2]+ rows = lines[table_start + 2 : table_end]+ header_text = "\n".join(header)+ repeat_header = (+ len(header_text) <= self.TABLE_HEADER_REPEAT_MAX_CHARS+ and len(header_text) < max_chars+ and self._estimate_token_count(header_text) < max_size+ )++ parts: List[str] = []+ current_lines: List[str] = []+ current_data_rows = 0++ def flush() -> None:+ nonlocal current_lines, current_data_rows+ if current_lines:+ parts.append("\n".join(current_lines).strip())+ current_lines = []+ current_data_rows = 0++ if prefix:+ current_lines.extend([prefix, ""])+ current_lines.extend(header)++ for row in rows:+ candidate_lines = [*current_lines, row]+ candidate = "\n".join(candidate_lines)+ if current_lines and (+ len(candidate) > max_chars or self._estimate_token_count(candidate) > max_size+ ) and current_data_rows > 0:+ flush()+ current_lines = [*header, row] if repeat_header else [row]+ current_data_rows = 1+ else:+ current_lines.append(row)+ current_data_rows += 1++ flush()+ return parts or None+async def _ingest_local_images(self,root_dir: str,@@ -908,45 +1017,7 @@ class MarkdownParser(BaseParser):Returns:True if the image satisfies all requirements, otherwise False"""- # File size check (local file path limit: 10 MB)- if len(image_bytes) > self.IMAGE_MAX_FILE_BYTES:- logger.warning(f"[MarkdownParser] Image exceeds 10MB, skipping: {source_path}")- return False-- # Pixel size check- try:- from PIL import Image-- with Image.open(io.BytesIO(image_bytes)) as img:- width, height = img.size- except Exception as e:- logger.warning(- f"[MarkdownParser] Cannot read image dimensions, skipping {source_path}: {e}"- )- return False-- if width <= self.IMAGE_MIN_SIDE or height <= self.IMAGE_MIN_SIDE:- logger.warning(- f"[MarkdownParser] Image side too small ({width}x{height}), skipping: {source_path}"- )- return False-- pixels = width * height- if pixels < self.IMAGE_MIN_PIXELS or pixels > self.IMAGE_MAX_PIXELS:- logger.warning(- f"[MarkdownParser] Image pixel count out of range ({pixels}), skipping: {source_path}"- )- return False-- aspect_ratio = width / height- if aspect_ratio < self.IMAGE_MIN_ASPECT_RATIO or aspect_ratio > self.IMAGE_MAX_ASPECT_RATIO:- logger.warning(- f"[MarkdownParser] Image aspect ratio out of range ({aspect_ratio:.4f}), "- f"skipping: {source_path}"- )- return False-- return True+ return is_valid_image(image_bytes, source_path)@staticmethoddef _is_remote_uri(path: str) -> bool:@@ -1021,7 +1092,12 @@ class MarkdownParser(BaseParser):return keys[0], Falseasync def _rewrite_single_link(- self, link: str, src_dir: str, import_root_abs: str, source_new_dir: str+ self,+ link: str,+ src_dir: str,+ import_root_abs: str,+ source_new_dir: str,+ rewrite_ctx: _RewriteContext,) -> str:"""Rewrite one link target; return original `link` if it must not change."""raw = link.strip()@@ -1065,7 +1141,7 @@ class MarkdownParser(BaseParser):# (file vs directory), its name and the section files all come straight from# the parser, so NOTHING about ingest is assumed here. Whatever parse_content# does, running it in-memory tells us the final result.- layout = await self._target_split_files(target_disk)+ layout = await self._target_split_files(target_disk, rewrite_ctx)if not layout:return link # target unparsable → leave the link untouched@@ -1097,19 +1173,17 @@ class MarkdownParser(BaseParser):# D) Directory landing → point at the directory and drop any suffix.return _to_rel(os.path.join(target_parent, landing), keep_suffix=False)- async def _ingest_will_handle_image(self, link: str) -> bool:+ async def _ingest_will_handle_image(self, link: str, rewrite_ctx: _RewriteContext) -> bool:"""Whether ``_ingest_local_images`` will take this image reference: it mustresolve within base_dir/allowed_media_dirs AND pass image validation — theexact conditions ingestion itself applies. Such embeds are left untouched so#2429 can copy them next to the section and ``rewrite_image_uris`` can turnthem into viking:// URIs; everything else falls back to depth adjustment.Results are cached per parse_content call."""- ctx = self._rewrite_ctx or {}- cache = ctx.setdefault("_image_probe_cache", {})- if link not in cache:+ if link not in rewrite_ctx.image_probe_cache:handled = Falseresolved = self._resolve_image_path(- link, ctx.get("base_dir"), ctx.get("allowed_media_dirs")+ link, rewrite_ctx.base_dir, rewrite_ctx.allowed_media_dirs)if resolved is not None:try:@@ -1117,17 +1191,15 @@ class MarkdownParser(BaseParser):handled = await asyncio.to_thread(self._is_valid_image, image_bytes, resolved)except Exception:handled = False- cache[link] = handled- return cache[link]+ rewrite_ctx.image_probe_cache[link] = handled+ return rewrite_ctx.image_probe_cache[link]async def _rewrite_relative_links(self,content: str,*,- source_path: str,- doc_name: str,section_subpath: str,- import_root: str,+ rewrite_ctx: _RewriteContext,) -> str:"""Rewrite relative markdown links in `content` (disk-coordinate).@@ -1140,23 +1212,23 @@ class MarkdownParser(BaseParser):"""from openviking.parse.image_rewrite import HTML_IMG_PATTERN- src_dir = os.path.dirname(os.path.abspath(source_path))- import_root_abs = os.path.abspath(import_root)- if (self._rewrite_ctx or {}).get("flatten_single_output", False):+ src_dir = os.path.dirname(os.path.abspath(rewrite_ctx.source_path))+ import_root_abs = os.path.abspath(rewrite_ctx.import_root)+ if rewrite_ctx.flatten_single_output:source_new_dir = os.path.join(src_dir, section_subpath)else:- source_new_dir = os.path.join(src_dir, doc_name, section_subpath)+ source_new_dir = os.path.join(src_dir, rewrite_ctx.doc_name, section_subpath)out: List[str] = []last = 0for match in _MD_LINK_RE.finditer(content):out.append(content[last : match.start()])bang, text, link = match.group(1), match.group(2), match.group(3)- if bang and await self._ingest_will_handle_image(link):+ if bang and await self._ingest_will_handle_image(link, rewrite_ctx):new_link = linkelse:new_link = await self._rewrite_single_link(- link, src_dir, import_root_abs, source_new_dir+ link, src_dir, import_root_abs, source_new_dir, rewrite_ctx)out.append(f"{bang}[{text}]({new_link})")last = match.end()@@ -1169,32 +1241,33 @@ class MarkdownParser(BaseParser):for match in HTML_IMG_PATTERN.finditer(rewritten):pieces.append(rewritten[last : match.start()])src = match.group(2)- if await self._ingest_will_handle_image(src):+ if await self._ingest_will_handle_image(src, rewrite_ctx):new_src = srcelse:new_src = await self._rewrite_single_link(- src, src_dir, import_root_abs, source_new_dir+ src, src_dir, import_root_abs, source_new_dir, rewrite_ctx)pieces.append(f"{match.group(1)}{new_src}{match.group(3)}")last = match.end()pieces.append(rewritten[last:])return "".join(pieces)- async def _target_split_files(self, target_path: str) -> Optional[Dict[str, str]]:+ async def _target_split_files(+ self, target_path: str, rewrite_ctx: _RewriteContext+ ) -> Optional[Dict[str, str]]:"""The target's real ingest layout {"<doc_dir>/<section...>": text} via aside-effect-free parse, cached per parse_content() call. None on parse failure."""- ctx = self._rewrite_ctx or {}- cache = ctx.setdefault("_split_cache", {})- split_content = bool(ctx.get("split_content", True))- flatten_single_output = bool(ctx.get("adaptive_flatten_requested", False))+ split_content = rewrite_ctx.split_content+ flatten_single_output = rewrite_ctx.adaptive_flatten_requestedkey = (os.path.abspath(target_path), split_content, flatten_single_output)- if key not in cache:- cache[key] = await self._probe_split_layout(+ if key not in rewrite_ctx.split_cache:+ rewrite_ctx.split_cache[key] = await self._probe_split_layout(target_path,split_content=split_content,flatten_single_output=flatten_single_output,+ allowed_media_dirs=rewrite_ctx.allowed_media_dirs,)- return cache[key]+ return rewrite_ctx.split_cache[key]async def _probe_split_layout(self,@@ -1202,6 +1275,7 @@ class MarkdownParser(BaseParser):*,split_content: bool,flatten_single_output: bool = False,+ allowed_media_dirs: Optional[List[Path]] = None,) -> Optional[Dict[str, str]]:"""Plan the target's layout WITHOUT writing anything, returning{"<doc_dir>/<section...>": text} keyed by path relative to the temp root, i.e.@@ -1215,7 +1289,7 @@ class MarkdownParser(BaseParser):flatten_single_output = not await self._has_ingestable_local_image(content,base_dir=Path(target_path).parent,- allowed_media_dirs=(self._rewrite_ctx or {}).get("allowed_media_dirs"),+ allowed_media_dirs=allowed_media_dirs,)layout = await self._compute_layout(content,@@ -1250,16 +1324,19 @@ class MarkdownParser(BaseParser):return section_dir[len(root) + 1 :]return ""- async def _write_section(self, uri: str, content: str) -> None:+ async def _write_section(+ self,+ uri: str,+ content: str,+ *,+ rewrite_ctx: Optional[_RewriteContext] = None,+ ) -> None:"""Write a markdown section file, rewriting relative links when enabled."""- ctx = self._rewrite_ctx- if ctx and ctx.get("enabled") and ctx.get("source_path") and ctx.get("import_root"):+ if rewrite_ctx is not None:content = await self._rewrite_relative_links(content,- source_path=ctx["source_path"],- doc_name=ctx["doc_name"],- section_subpath=self._section_subpath(uri, ctx["root_dir"]),- import_root=ctx["import_root"],+ section_subpath=self._section_subpath(uri, rewrite_ctx.root_dir),+ rewrite_ctx=rewrite_ctx,)await self._get_viking_fs().write_file(uri, content)@@ -1605,9 +1682,9 @@ class MarkdownParser(BaseParser):) -> None:"""Plan merged sections as a single file with smart naming into ``ops``.- If the joined content exceeds max_section_chars it is split further- by _smart_split_content before writing, so no single file ever exceeds- the hard character limit.+ If the joined content exceeds max_section_chars it is split further by+ _smart_split_content before writing. A single oversized Markdown table row+ remains intact to preserve the source table structure."""name = self._generate_merged_filename(sections)content = "\n\n".join(c for _, c, _ in sections)diff --git a/openviking/parse/parsers/powerpoint.py b/openviking/parse/parsers/powerpoint.pydeleted file mode 100644index 9ef183996..000000000--- a/openviking/parse/parsers/powerpoint.py+++ /dev/null@@ -1,150 +0,0 @@-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.-# SPDX-License-Identifier: AGPL-3.0-"""-PowerPoint (.pptx) parser for OpenViking.--Converts PowerPoint presentations to Markdown then parses using MarkdownParser.-Inspired by microsoft/markitdown approach.-"""--import asyncio-from pathlib import Path-from typing import List, Optional, Union--from openviking.parse.base import ParseResult-from openviking.parse.parsers.base_parser import BaseParser-from openviking_cli.utils.config.parser_config import ParserConfig-from openviking_cli.utils.logger import get_logger--logger = get_logger(__name__)---class PowerPointParser(BaseParser):- """- PowerPoint presentation parser for OpenViking.-- Supports: .pptx-- Converts PowerPoint presentations to Markdown using python-pptx,- then delegates to MarkdownParser for tree structure creation.- """-- def __init__(self, config: Optional[ParserConfig] = None, extract_notes: bool = False):- """- Initialize PowerPoint parser.-- Args:- config: Parser configuration- extract_notes: Whether to extract speaker notes- """- from openviking.parse.parsers.markdown import MarkdownParser-- self._md_parser = MarkdownParser(config=config)- self.config = config or ParserConfig()- self.extract_notes = extract_notes-- @property- def supported_extensions(self) -> List[str]:- return [".pptx"]-- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:- """Parse PowerPoint presentation from file path."""- path = Path(source)-- if path.exists():- import pptx-- markdown_content = await asyncio.to_thread(self._convert_to_markdown, path, pptx)- result = await self._md_parser.parse_content(- markdown_content, source_path=str(path), instruction=instruction, **kwargs- )- else:- result = await self._md_parser.parse_content(- str(source), instruction=instruction, **kwargs- )- result.source_format = "pptx"- result.parser_name = "PowerPointParser"- return result-- async def parse_content(- self, content: str, source_path: Optional[str] = None, instruction: str = "", **kwargs- ) -> ParseResult:- """Parse content - delegates to MarkdownParser."""- result = await self._md_parser.parse_content(content, source_path, **kwargs)- result.source_format = "pptx"- result.parser_name = "PowerPointParser"- return result-- def _convert_to_markdown(self, path: Path, pptx) -> str:- """Convert PowerPoint presentation to Markdown string."""- prs = pptx.Presentation(path)- markdown_parts = []- slide_count = len(prs.slides)-- for idx, slide in enumerate(prs.slides, 1):- slide_parts = []- slide_parts.append(f"## Slide {idx}/{slide_count}")-- title = self._extract_slide_title(slide)- if title:- slide_parts.append(f"### {title}")-- content = self._extract_slide_content(slide)- if content:- slide_parts.append(content)-- if self.extract_notes and slide.has_notes_slide:- notes = slide.notes_slide.notes_text_frame.text.strip()- if notes:- slide_parts.append(f"**Notes:** {notes}")-- markdown_parts.append("\n\n".join(slide_parts))-- return "\n\n---\n\n".join(markdown_parts)-- def _extract_slide_title(self, slide) -> str:- """Extract title from a slide."""- from pptx.enum.shapes import PP_PLACEHOLDER-- for shape in slide.shapes:- if shape.is_placeholder:- ph_type = shape.placeholder_format.type- if ph_type in (PP_PLACEHOLDER.TITLE, PP_PLACEHOLDER.CENTER_TITLE):- return shape.text.strip()- return ""-- def _extract_slide_content(self, slide) -> str:- """Extract content from slide shapes."""- from pptx.enum.shapes import PP_PLACEHOLDER-- content_parts = []-- for shape in slide.shapes:- if shape.is_placeholder:- ph_type = shape.placeholder_format.type- if ph_type in (PP_PLACEHOLDER.TITLE, PP_PLACEHOLDER.CENTER_TITLE):- continue-- if hasattr(shape, "text") and shape.text.strip():- if shape.has_table:- content_parts.append(self._convert_table(shape.table))- else:- text = shape.text.strip()- if text:- content_parts.append(text)-- return "\n\n".join(content_parts)-- def _convert_table(self, table) -> str:- """Convert PowerPoint table to markdown format."""- if not table.rows:- return ""-- rows = []- for row in table.rows:- row_data = [cell.text.strip() for cell in row.cells]- rows.append(row_data)-- from openviking.parse.base import format_table_to_markdown-- return format_table_to_markdown(rows, has_header=True)diff --git a/openviking/parse/parsers/word.py b/openviking/parse/parsers/word.pydeleted file mode 100644index 0a76f110a..000000000--- a/openviking/parse/parsers/word.py+++ /dev/null@@ -1,231 +0,0 @@-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.-# SPDX-License-Identifier: AGPL-3.0-"""-Word document (.docx) parser for OpenViking.--Converts Word documents to Markdown then parses using MarkdownParser.-Inspired by microsoft/markitdown approach.-"""--import asyncio-from pathlib import Path-from typing import List, Optional, Union--from openviking.parse.base import ParseResult-from openviking.parse.parsers.base_parser import BaseParser-from openviking_cli.utils.config.parser_config import ParserConfig-from openviking_cli.utils.logger import get_logger--logger = get_logger(__name__)---class WordParser(BaseParser):- """- Word document parser for OpenViking.-- Supports: .docx-- Converts Word documents to Markdown using python-docx,- then delegates to MarkdownParser for tree structure creation.- """-- def __init__(self, config: Optional[ParserConfig] = None):- """Initialize Word parser."""- from openviking.parse.parsers.markdown import MarkdownParser-- self._md_parser = MarkdownParser(config=config)- self.config = config or ParserConfig()-- @property- def supported_extensions(self) -> List[str]:- return [".docx"]-- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:- """Parse Word document from file path."""- path = Path(source)-- if path.exists():- import docx-- from openviking_cli.utils.storage import get_storage-- storage = get_storage()- resource_name = kwargs.get("resource_name") or kwargs.get("source_name") or path.stem-- markdown_content = await asyncio.to_thread(- self._convert_to_markdown, path, docx, resource_name, storage- )- result = await self._md_parser.parse_content(- markdown_content,- source_path=str(path),- # Forward the original upload name explicitly (mirrors pdf.py) so- # the resource is named after it, not the temp upload path. Pass- # the raw kwargs values rather than the path.stem-backed local- # resource_name, leaving MarkdownParser's naming logic unchanged.- resource_name=kwargs.get("resource_name"),- source_name=kwargs.get("source_name"),- instruction=instruction,- base_dir=path.parent,- # docx images are extracted into storage.media_dir, so (like pdf)- # that is the only derived-media root needed.- allowed_media_dirs=[storage.media_dir],- split_content=kwargs.get("split_content", True),- )- else:- result = await self._md_parser.parse_content(- str(source),- instruction=instruction,- resource_name=kwargs.get("resource_name"),- source_name=kwargs.get("source_name"),- split_content=kwargs.get("split_content", True),- )- result.source_format = "docx"- result.parser_name = "WordParser"- return result-- async def parse_content(- self, content: str, source_path: Optional[str] = None, instruction: str = "", **kwargs- ) -> ParseResult:- """Parse content - delegates to MarkdownParser."""- result = await self._md_parser.parse_content(content, source_path, **kwargs)- result.source_format = "docx"- result.parser_name = "WordParser"- return result-- def _convert_to_markdown(self, path: Path, docx, resource_name=None, storage=None) -> str:- """Convert Word document to Markdown string.-- Iterates the document body in order so that tables appear in their- original position rather than being appended at the end. Embedded- images are extracted to local storage and referenced inline so the- MarkdownParser can ingest them into VikingFS.- """- doc = docx.Document(path)- markdown_parts = []-- # Map XML table elements to python-docx Table objects for O(1) lookup- table_by_element = {table._tbl: table for table in doc.tables}-- # Track extracted images to deduplicate and number sequentially- image_counter = [0]-- # Walk the document body in order to preserve table positions- from docx.oxml.ns import qn-- for child in doc.element.body:- if child.tag == qn("w:p"):- # It's a paragraph- from docx.text.paragraph import Paragraph-- paragraph = Paragraph(child, doc)-- # Extract any embedded images in this paragraph first so they- # keep their original document position.- image_md = self._convert_paragraph_images(- paragraph, doc, qn, resource_name, storage, image_counter- )- if image_md:- markdown_parts.append(image_md)-- if not paragraph.text.strip():- continue-- style_name = paragraph.style.name if paragraph.style else "Normal"-- if style_name.startswith("Heading"):- level = self._extract_heading_level(style_name)- markdown_parts.append(f"{'#' * level} {paragraph.text}")- else:- text = self._convert_formatted_text(paragraph)- markdown_parts.append(text)-- elif child.tag == qn("w:tbl"):- # It's a table- if child in table_by_element:- markdown_parts.append(self._convert_table(table_by_element[child]))-- return "\n\n".join(markdown_parts)-- def _convert_paragraph_images(- self, paragraph, doc, qn, resource_name, storage, image_counter- ) -> str:- """Extract embedded images from a paragraph and return markdown references.-- Images in .docx are stored as relationship parts (``word/media/*``)- referenced by ``r:embed`` ids inside ``w:drawing`` elements. We pull the- binary blob for each and persist it via the shared storage helper so the- MarkdownParser's image ingestion can pick it up.- """- if storage is None:- return ""-- parts = []- # Each <a:blip r:embed="rId"> inside the paragraph references an image part- for blip in paragraph._p.findall(".//" + qn("a:blip")):- rid = blip.get(qn("r:embed"))- if not rid:- continue- try:- image_part = doc.part.related_parts[rid]- image_bytes = image_part.blob- except Exception as e:- logger.warning(f"[WordParser] Failed to read embedded image {rid}: {e}")- continue-- try:- extension = Path(image_part.partname).suffix or ".png"- image_counter[0] += 1- filename = f"image{image_counter[0]}"- image_path = storage.save_image(- resource_name, image_bytes, filename=filename, extension=extension- )- # Reference relative to the media root so the MarkdownParser can- # resolve it within the resource's media lifecycle (never cwd).- rel_path = image_path.relative_to(storage.media_dir)- parts.append(f"")- except Exception as e:- logger.warning(f"[WordParser] Failed to save embedded image {rid}: {e}")-- return "\n\n".join(parts)-- def _extract_heading_level(self, style_name: str) -> int:- """Extract heading level from style name."""- try:- if "Heading" in style_name:- parts = style_name.split()- for part in parts:- if part.isdigit():- return min(int(part), 6)- except Exception:- pass- return 1-- def _convert_formatted_text(self, paragraph) -> str:- """Convert paragraph with formatting to markdown."""- text_parts = []- for run in paragraph.runs:- text = run.text- if not text:- continue- if run.bold:- text = f"**{text}**"- if run.italic:- text = f"*{text}*"- if run.underline:- text = f"<ins>{text}</ins>"- text_parts.append(text)- return "".join(text_parts)-- def _convert_table(self, table) -> str:- """Convert Word table to markdown format."""- if not table.rows:- return ""-- rows = []- for row in table.rows:- row_data = [cell.text.strip() for cell in row.cells]- rows.append(row_data)-- from openviking.parse.base import format_table_to_markdown-- return format_table_to_markdown(rows, has_header=True)diff --git a/openviking/parse/registry.py b/openviking/parse/registry.pyindex dc0140562..8cadb5096 100644--- a/openviking/parse/registry.py+++ b/openviking/parse/registry.py@@ -10,24 +10,18 @@ from pathlib import Pathfrom typing import Dict, List, Optional, Unionfrom openviking.parse.base import ParseResult+from openviking.parse.parsers.anydoc import AnyDocParserfrom openviking.parse.parsers.base_parser import BaseParserfrom openviking.parse.parsers.constants import TYPESCRIPT_MPEG_TS_EXTENSIONfrom openviking.parse.parsers.directory import DirectoryParser-from openviking.parse.parsers.epub import EPubParser-from openviking.parse.parsers.excel import ExcelParser# Import will be handled dynamically to avoid dependency issuesfrom openviking.parse.parsers.html import HTMLParser--# Import markitdown-inspired parsers-from openviking.parse.parsers.legacy_doc import LegacyDocParserfrom openviking.parse.parsers.markdown import MarkdownParserfrom openviking.parse.parsers.media import AudioParser, ImageParser, VideoParserfrom openviking.parse.parsers.media.utils import is_mpeg_ts, read_mpeg_ts_probefrom openviking.parse.parsers.pdf import PDFParser-from openviking.parse.parsers.powerpoint import PowerPointParserfrom openviking.parse.parsers.text import TextParser-from openviking.parse.parsers.word import WordParserfrom openviking.parse.parsers.zip_parser import ZipParserfrom openviking_cli.utils.config.parser_config import ParserConfig@@ -59,14 +53,14 @@ class ParserRegistry:self._register("pdf", PDFParser(config=self._parser_configs.get("pdf")))self._register("html", HTMLParser(config=self._parser_configs.get("html")))- # Register markitdown-inspired parsers (built-in)- self._register("word", WordParser(config=self._parser_configs.get("word")))- self._register("legacy_doc", LegacyDocParser(config=self._parser_configs.get("legacy_doc")))+ # Register Office/EPUB through the unified AnyDoc parser.self._register(- "powerpoint", PowerPointParser(config=self._parser_configs.get("powerpoint"))+ "anydoc",+ AnyDocParser(+ config=self._parser_configs.get("markdown"),+ anydoc_config=self._parser_configs.get("anydoc"),+ ),)- self._register("excel", ExcelParser(config=self._parser_configs.get("excel")))- self._register("epub", EPubParser(config=self._parser_configs.get("epub")))self._register("zip", ZipParser())self._register("directory", DirectoryParser())@@ -176,15 +170,7 @@ def get_registry() -> ParserRegistry:"markdown": config.markdown,"pdf": config.pdf,"html": config.html,- "word": config.markdown,- "legacy_doc": config.markdown,- "powerpoint": config.markdown,- # Excel had no dedicated config section and reused- # ``config.markdown``. Keep unset sectioning fields following- # Markdown so existing deployments keep their node structure and- # stable URIs after this upgrade.- "excel": config.excel.with_sectioning_defaults_from(config.markdown),- "epub": config.markdown,+ "anydoc": getattr(config, "anydoc", None),"image": config.image,}except Exception:diff --git a/openviking_cli/utils/config/__init__.py b/openviking_cli/utils/config/__init__.pyindex b84173610..fedbc5cc5 100644--- a/openviking_cli/utils/config/__init__.py+++ b/openviking_cli/utils/config/__init__.py@@ -65,9 +65,9 @@ from .open_viking_config import (from .ovcli_config import OVCLIConfig, load_ovcli_configfrom .parser_config import (PARSER_CONFIG_REGISTRY,+ AnydocConfig,AudioConfig,CodeConfig,- ExcelConfig,HTMLConfig,ImageConfig,MarkdownConfig,@@ -147,7 +147,7 @@ __all__ = ["AudioConfig","VideoConfig","MarkdownConfig",- "ExcelConfig",+ "AnydocConfig","HTMLConfig","TextConfig","get_parser_config",diff --git a/openviking_cli/utils/config/open_viking_config.py b/openviking_cli/utils/config/open_viking_config.pyindex 601c9eb42..1c53330dc 100644--- a/openviking_cli/utils/config/open_viking_config.py+++ b/openviking_cli/utils/config/open_viking_config.py@@ -28,10 +28,10 @@ from .log_config import LogConfigfrom .memory_config import MemoryConfigfrom .oauth_config import OAuthConfigfrom .parser_config import (+ AnydocConfig,AudioConfig,CodeConfig,DirectoryConfig,- ExcelConfig,FeishuConfig,HTMLConfig,ImageConfig,@@ -52,7 +52,7 @@ from .telemetry_config import TelemetryConfigfrom .vlm_config import VLMConfig-def _get_config_warning_logger():+def _get_config_logger():"""Use stdlib logging during config bootstrap to avoid early logger side effects."""return logging.getLogger(__name__)@@ -206,12 +206,9 @@ class OpenVikingConfig(BaseModel):default_factory=MarkdownConfig, description="Markdown parsing configuration")- excel: ExcelConfig = Field(- # from_dict on an empty mapping, not the bare constructor: an absent- # parsers.excel section must record that no key was set, so sectioning- # still follows parsers.markdown for deployments predating this section.- default_factory=lambda: ExcelConfig.from_dict({}),- description="Excel parsing configuration",+ anydoc: AnydocConfig = Field(+ default_factory=AnydocConfig,+ description="Shared anydoc Office conversion configuration",)html: HTMLConfig = Field(default_factory=HTMLConfig, description="HTML parsing configuration")@@ -302,7 +299,7 @@ class OpenVikingConfig(BaseModel):@model_validator(mode="after")def _warn_on_deprecated_language_fallback(self) -> "OpenVikingConfig":if self.language_fallback and self.language_fallback != "en":- _get_config_warning_logger().warning(+ _get_config_logger().warning("Config field 'language_fallback=%s' is deprecated and has no effect; ""remove it, or set 'output_language_override' to pin an explicit language.",self.language_fallback,@@ -407,7 +404,7 @@ class OpenVikingConfig(BaseModel):"audio","video","markdown",- "excel",+ "anydoc","html","text","directory",@@ -432,6 +429,13 @@ class OpenVikingConfig(BaseModel):parser_configs = {}if not isinstance(parser_configs, dict):raise ValueError("Invalid parsers config: 'parsers' section must be an object")+ parser_configs = parser_configs.copy()+ if "excel" in parser_configs:+ parser_configs.pop("excel")+ _get_config_logger().error(+ "Config field 'parsers.excel' was removed and is ignored; "+ "spreadsheet parsing now uses 'parsers.anydoc'."+ )raise_unknown_config_fields(data=parser_configs,valid_fields=set(parser_types),diff --git a/openviking_cli/utils/config/parser_config.py b/openviking_cli/utils/config/parser_config.pyindex abcdce746..31520326f 100644--- a/openviking_cli/utils/config/parser_config.py+++ b/openviking_cli/utils/config/parser_config.py@@ -8,9 +8,9 @@ scattered across different modules. All configurations inherit from ParserConfigand can be loaded from ov.conf files."""-from dataclasses import dataclass, replace+from dataclasses import dataclassfrom pathlib import Path-from typing import Any, Dict, Iterable, Optional, Union+from typing import Any, Dict, Optional, Unionfrom openviking_cli.utils.logger import get_logger@@ -33,7 +33,8 @@ class ParserConfig:encoding: Default file encodingmax_section_size: Maximum tokens per section before splittingsection_size_flexibility: Allow overflow to maintain coherence (0.0-1.0)- max_section_chars: Hard character limit per section (guards against token estimation errors)+ max_section_chars: Target character limit per section. A single oversized+ Markdown table row may remain intact to preserve table semantics."""enabled: bool = True@@ -43,9 +44,7 @@ class ParserConfig:# Smart splitting configurationmax_section_size: int = 2048 # Maximum tokens per section before splittingsection_size_flexibility: float = 0.3 # Allow 30% overflow to maintain coherence- max_section_chars: int = (- 6000 # Hard character limit per section (guards against token estimation errors)- )+ max_section_chars: int = 6000 # Target character limit per section@classmethoddef from_dict(cls, data: Dict[str, Any]) -> "ParserConfig":@@ -466,92 +465,21 @@ class MarkdownConfig(ParserConfig):@dataclass-class ExcelConfig(ParserConfig):- """- Configuration for Excel parsing.-- Attributes:- enable_process_pool: Offload Excel→Markdown conversion and layout- planning to a ProcessPoolExecutor (default off).- process_pool_workers: Max worker processes when the pool is enabled.- """-- enable_process_pool: bool = False- process_pool_workers: int = 2-- # Excel is converted to Markdown and then sectioned by MarkdownParser, so- # these fields decide the resulting node structure and stable URIs.- _SECTIONING_FIELDS = (- "max_content_length",- "encoding",- "max_section_size",- "section_size_flexibility",- "max_section_chars",- )+class AnydocConfig(ParserConfig):+ """Configuration for the shared anydoc Office converter."""- # Names of keys a config source actually provided. Tracked as a plain- # instance attribute rather than a dataclass field so it never appears in- # asdict/model_dump output, cannot be injected from a config file, and does- # not affect equality. Absent means "provenance unknown".- _EXPLICIT_ATTR = "_openviking_explicit_keys"+ max_table_rows: int = 1000@classmethod- def from_dict(cls, data: Dict[str, Any]) -> "ExcelConfig":- """Build the config while remembering which keys were actually present.-- ``with_sectioning_defaults_from`` needs to tell "the user wrote this- value" from "the key was absent". Comparing against class defaults- cannot do that, so record the provided keys here instead.- """- config = super().from_dict(data)- return config.with_explicit_keys(data)-- def with_explicit_keys(self, names: Iterable[str]) -> "ExcelConfig":- """Return this config marked as having ``names`` explicitly configured."""- object.__setattr__(self, self._EXPLICIT_ATTR, frozenset(names))- return self-- @property- def explicit_keys(self) -> Optional[frozenset]:- """Keys a config source provided, or ``None`` when unknown."""- return getattr(self, self._EXPLICIT_ATTR, None)-- def with_sectioning_defaults_from(self, markdown: "ParserConfig") -> "ExcelConfig":- """Inherit sectioning fields that ``parsers.excel`` did not set.-- Excel used to be registered with ``config.markdown`` directly, so a- deployment that tuned ``parsers.markdown`` also tuned Excel imports.- Introducing a dedicated ``parsers.excel`` section must not silently- change that node structure, so a sectioning field absent from- ``parsers.excel`` keeps following Markdown. Explicit ``parsers.excel``- values always win, including one that happens to equal the class- default.-- Configs built without ``from_dict`` carry no key information; those are- treated as fully explicit so a hand-constructed ``ExcelConfig`` is never- silently rewritten.- """- if markdown is None:- return self-- explicit = self.explicit_keys- if explicit is None:- return self-- overrides = {- name: getattr(markdown, name)- for name in self._SECTIONING_FIELDS- if hasattr(markdown, name) and name not in explicit- }- if not overrides:- return self- return replace(self, **overrides).with_explicit_keys(explicit)+ def from_dict(cls, data: Dict[str, Any]) -> "AnydocConfig":+ valid_fields = {f.name for f in cls.__dataclass_fields__.values()}+ raise_unknown_config_fields(data=data, valid_fields=valid_fields, context_name=cls.__name__)+ return cls(**data)def validate(self) -> None:- """Validate Excel-specific configuration."""super().validate()- if self.process_pool_workers < 1:- raise ValueError("process_pool_workers must be at least 1")+ if self.max_table_rows < 0:+ raise ValueError("max_table_rows must be non-negative")@dataclass@@ -790,7 +718,7 @@ PARSER_CONFIG_REGISTRY = {"audio": AudioConfig,"video": VideoConfig,"markdown": MarkdownConfig,- "excel": ExcelConfig,+ "anydoc": AnydocConfig,"html": HTMLConfig,"text": TextConfig,"directory": DirectoryConfig,diff --git a/pyproject.toml b/pyproject.tomlindex 0f1d38aeb..0a4fb028a 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -43,12 +43,7 @@ dependencies = ["openai>=1.0.0","requests>=2.33.0","charset-normalizer>=3.4,<4",- "python-docx>=1.0.0",- "olefile>=0.47",- "xlrd>=2.0.1",- "python-pptx>=1.0.0",- "openpyxl>=3.0.0",- "ebooklib>=0.18.0",+ "firecrawl-anydoc>=0.1.8","json-repair>=0.25.0","apscheduler>=3.11.0","volcengine>=1.0.216",@@ -187,6 +182,7 @@ benchmark = ["tiktoken>=0.5.0","datasets>=2.0.0","pandas>=2.0.0",+ "python-docx>=1.0.0",]local-embed = ["llama-cpp-python>=0.3.0",diff --git a/tests/parse/test_add_directory.py b/tests/parse/test_add_directory.pyindex 3325e5332..542c7db51 100644--- a/tests/parse/test_add_directory.py+++ b/tests/parse/test_add_directory.py@@ -457,7 +457,7 @@ class TestParserDelegation:@pytest.mark.asyncioasync def test_docx_file_goes_through_parser(self, tmp_path: Path, parser, fake_fs) -> None:- """Word (.docx) files should be processed by WordParser.parse()."""+ """Word-compatible (.docx) files should be processed by AnyDocParser.parse()."""(tmp_path / "report.docx").write_bytes(b"PK\x03\x04")mock_temp = fake_fs.create_temp_uri()@@ -470,7 +470,7 @@ class TestParserDelegation:root=ResourceNode(type=NodeType.ROOT),source_path=str(tmp_path / "report.docx"),source_format="docx",- parser_name="WordParser",+ parser_name="AnyDocParser",parse_time=0.1,)fake_result.temp_dir_path = mock_temp@@ -478,14 +478,14 @@ class TestParserDelegation:with patch("openviking.parse.parsers.directory.DirectoryParser._assign_parser",) as mock_assign:- from openviking.parse.parsers.word import WordParser as _Word+ from openviking.parse.parsers.anydoc import AnyDocParser as _AnyDoc- mock_word = AsyncMock(spec=_Word)- mock_word.parse = AsyncMock(return_value=fake_result)+ mock_anydoc = AsyncMock(spec=_AnyDoc)+ mock_anydoc.parse = AsyncMock(return_value=fake_result)def assign_side_effect(cf, registry):if cf.path.suffix == ".docx":- return mock_word+ return mock_anydocreturn registry.get_parser_for_file(cf.path)mock_assign.side_effect = assign_side_effect@@ -497,128 +497,6 @@ class TestParserDelegation:)assert found_md, f"report.md not found. Files: {list(fake_fs.files.keys())}"- @pytest.mark.asyncio- async def test_xlsx_file_goes_through_parser(self, tmp_path: Path, parser, fake_fs) -> None:- """Excel (.xlsx) files should be processed by ExcelParser.parse()."""- (tmp_path / "data.xlsx").write_bytes(b"PK\x03\x04")-- mock_temp = fake_fs.create_temp_uri()- doc_dir = f"{mock_temp}/data"- await fake_fs.mkdir(mock_temp)- await fake_fs.mkdir(doc_dir)- await fake_fs.write_file(f"{doc_dir}/data.md", "# Converted Excel")-- fake_result = create_parse_result(- root=ResourceNode(type=NodeType.ROOT),- source_path=str(tmp_path / "data.xlsx"),- source_format="xlsx",- parser_name="ExcelParser",- parse_time=0.1,- )- fake_result.temp_dir_path = mock_temp-- with patch(- "openviking.parse.parsers.directory.DirectoryParser._assign_parser",- ) as mock_assign:- from openviking.parse.parsers.excel import ExcelParser as _Excel-- mock_excel = AsyncMock(spec=_Excel)- mock_excel.parse = AsyncMock(return_value=fake_result)-- def assign_side_effect(cf, registry):- if cf.path.suffix in {".xlsx", ".xls", ".xlsm"}:- return mock_excel- return registry.get_parser_for_file(cf.path)-- mock_assign.side_effect = assign_side_effect- await parser.parse(str(tmp_path))-- dir_name = tmp_path.name- found_md = any(uri.endswith("data.md") and f"/{dir_name}/" in uri for uri in fake_fs.files)- assert found_md, f"data.md not found. Files: {list(fake_fs.files.keys())}"-- @pytest.mark.asyncio- async def test_epub_file_goes_through_parser(self, tmp_path: Path, parser, fake_fs) -> None:- """EPub (.epub) files should be processed by EPubParser.parse()."""- (tmp_path / "book.epub").write_bytes(b"PK\x03\x04")-- mock_temp = fake_fs.create_temp_uri()- doc_dir = f"{mock_temp}/book"- await fake_fs.mkdir(mock_temp)- await fake_fs.mkdir(doc_dir)- await fake_fs.write_file(f"{doc_dir}/book.md", "# Converted EPub")-- fake_result = create_parse_result(- root=ResourceNode(type=NodeType.ROOT),- source_path=str(tmp_path / "book.epub"),- source_format="epub",- parser_name="EPubParser",- parse_time=0.1,- )- fake_result.temp_dir_path = mock_temp-- with patch(- "openviking.parse.parsers.directory.DirectoryParser._assign_parser",- ) as mock_assign:- from openviking.parse.parsers.epub import EPubParser as _EPub-- mock_epub = AsyncMock(spec=_EPub)- mock_epub.parse = AsyncMock(return_value=fake_result)-- def assign_side_effect(cf, registry):- if cf.path.suffix == ".epub":- return mock_epub- return registry.get_parser_for_file(cf.path)-- mock_assign.side_effect = assign_side_effect- await parser.parse(str(tmp_path))-- dir_name = tmp_path.name- found_md = any(uri.endswith("book.md") and f"/{dir_name}/" in uri for uri in fake_fs.files)- assert found_md, f"book.md not found. Files: {list(fake_fs.files.keys())}"-- @pytest.mark.asyncio- async def test_pptx_file_goes_through_parser(self, tmp_path: Path, parser, fake_fs) -> None:- """PowerPoint (.pptx) files should be processed by PowerPointParser.parse()."""- (tmp_path / "slides.pptx").write_bytes(b"PK\x03\x04")-- mock_temp = fake_fs.create_temp_uri()- doc_dir = f"{mock_temp}/slides"- await fake_fs.mkdir(mock_temp)- await fake_fs.mkdir(doc_dir)- await fake_fs.write_file(f"{doc_dir}/slides.md", "# Converted PowerPoint")-- fake_result = create_parse_result(- root=ResourceNode(type=NodeType.ROOT),- source_path=str(tmp_path / "slides.pptx"),- source_format="pptx",- parser_name="PowerPointParser",- parse_time=0.1,- )- fake_result.temp_dir_path = mock_temp-- with patch(- "openviking.parse.parsers.directory.DirectoryParser._assign_parser",- ) as mock_assign:- from openviking.parse.parsers.powerpoint import PowerPointParser as _PPT-- mock_ppt = AsyncMock(spec=_PPT)- mock_ppt.parse = AsyncMock(return_value=fake_result)-- def assign_side_effect(cf, registry):- if cf.path.suffix == ".pptx":- return mock_ppt- return registry.get_parser_for_file(cf.path)-- mock_assign.side_effect = assign_side_effect- await parser.parse(str(tmp_path))-- dir_name = tmp_path.name- found_md = any(- uri.endswith("slides.md") and f"/{dir_name}/" in uri for uri in fake_fs.files- )- assert found_md, f"slides.md not found. Files: {list(fake_fs.files.keys())}"-@pytest.mark.asyncioasync def test_zip_file_goes_through_parser(self, tmp_path: Path, parser, fake_fs) -> None:"""ZIP (.zip) files should be processed by ZipParser.parse()."""diff --git a/tests/parse/test_directory_parser_routing.py b/tests/parse/test_directory_parser_routing.pyindex ca75798cf..607e0c240 100644--- a/tests/parse/test_directory_parser_routing.py+++ b/tests/parse/test_directory_parser_routing.py@@ -27,14 +27,11 @@ from openviking.parse.directory_scan import (DirectoryScanResult,scan_directory,)-from openviking.parse.parsers.epub import EPubParser-from openviking.parse.parsers.excel import ExcelParser+from openviking.parse.parsers.anydoc import AnyDocParserfrom openviking.parse.parsers.html import HTMLParserfrom openviking.parse.parsers.markdown import MarkdownParserfrom openviking.parse.parsers.pdf import PDFParser-from openviking.parse.parsers.powerpoint import PowerPointParserfrom openviking.parse.parsers.text import TextParser-from openviking.parse.parsers.word import WordParserfrom openviking.parse.parsers.zip_parser import ZipParserfrom openviking.parse.registry import ParserRegistry@@ -72,13 +69,22 @@ def tmp_all_parsers(tmp_path: Path) -> Path:notes.txt -> TextParserlog.text -> TextParseroffice/- report.docx -> WordParser- data.xlsx -> ExcelParser- legacy.xls -> ExcelParser- macro.xlsm -> ExcelParser- slides.pptx -> PowerPointParser+ legacy.doc -> AnyDocParser+ report.docx -> AnyDocParser+ macro.docm -> AnyDocParser+ open.odt -> AnyDocParser+ rich.rtf -> AnyDocParser+ data.xlsx -> AnyDocParser+ legacy.xls -> AnyDocParser+ macro.xlsm -> AnyDocParser+ binary.xlsb -> AnyDocParser+ sheet.ods -> AnyDocParser+ table.csv -> AnyDocParser+ slides.pptx -> AnyDocParser+ legacy.ppt -> AnyDocParser+ show.pps -> AnyDocParserbooks/- book.epub -> EPubParser+ book.epub -> AnyDocParserarchives/bundle.zip -> ZipParsercode/@@ -123,11 +129,20 @@ def tmp_all_parsers(tmp_path: Path) -> Path:(tmp_path / "config" / "rules.toml").write_text("[section]", encoding="utf-8")(tmp_path / "office").mkdir()+ (tmp_path / "office" / "legacy.doc").write_bytes(b"\xd0\xcf\x11\xe0")(tmp_path / "office" / "report.docx").write_bytes(b"PK\x03\x04")+ (tmp_path / "office" / "macro.docm").write_bytes(b"PK\x03\x04")+ (tmp_path / "office" / "open.odt").write_bytes(b"PK\x03\x04")+ (tmp_path / "office" / "rich.rtf").write_text(r"{\rtf1 text}", encoding="utf-8")(tmp_path / "office" / "data.xlsx").write_bytes(b"PK\x03\x04")(tmp_path / "office" / "legacy.xls").write_bytes(b"\xd0\xcf\x11\xe0")(tmp_path / "office" / "macro.xlsm").write_bytes(b"PK\x03\x04")+ (tmp_path / "office" / "binary.xlsb").write_bytes(b"PK\x03\x04")+ (tmp_path / "office" / "sheet.ods").write_bytes(b"PK\x03\x04")+ (tmp_path / "office" / "table.csv").write_text("a,b\n1,2\n", encoding="utf-8")(tmp_path / "office" / "slides.pptx").write_bytes(b"PK\x03\x04")+ (tmp_path / "office" / "legacy.ppt").write_bytes(b"\xd0\xcf\x11\xe0")+ (tmp_path / "office" / "show.pps").write_bytes(b"\xd0\xcf\x11\xe0")(tmp_path / "books").mkdir()(tmp_path / "books" / "book.epub").write_bytes(b"PK\x03\x04")@@ -156,12 +171,21 @@ class TestParserSelection:".pdf": PDFParser,".txt": TextParser,".text": TextParser,- ".docx": WordParser,- ".xlsx": ExcelParser,- ".xls": ExcelParser,- ".xlsm": ExcelParser,- ".epub": EPubParser,- ".pptx": PowerPointParser,+ ".doc": AnyDocParser,+ ".docx": AnyDocParser,+ ".docm": AnyDocParser,+ ".odt": AnyDocParser,+ ".rtf": AnyDocParser,+ ".xlsx": AnyDocParser,+ ".xls": AnyDocParser,+ ".xlsm": AnyDocParser,+ ".xlsb": AnyDocParser,+ ".ods": AnyDocParser,+ ".csv": AnyDocParser,+ ".epub": AnyDocParser,+ ".pptx": AnyDocParser,+ ".ppt": AnyDocParser,+ ".pps": AnyDocParser,".zip": ZipParser,}@@ -230,54 +254,6 @@ class TestParserSelection:)-class TestParserCanParse:- """Parser.can_parse must accept its own supported extensions."""-- @pytest.mark.parametrize(- "parser_cls,filenames",- [- (MarkdownParser, ["doc.md", "spec.markdown", "x.mdown", "y.mkd"]),- (HTMLParser, ["page.html", "site.htm"]),- (PDFParser, ["paper.pdf"]),- (TextParser, ["notes.txt", "log.text"]),- (WordParser, ["report.docx"]),- (ExcelParser, ["data.xlsx", "legacy.xls", "book.xlsm"]),- (EPubParser, ["book.epub"]),- (PowerPointParser, ["slides.pptx"]),- (ZipParser, ["archive.zip"]),- ],- )- def test_can_parse_returns_true(self, parser_cls: type, filenames: List[str]) -> None:- parser = parser_cls()- for name in filenames:- assert parser.can_parse(Path(name)), (- f"{parser_cls.__name__}.can_parse('{name}') should be True"- )-- @pytest.mark.parametrize(- "parser_cls,filenames",- [- (MarkdownParser, ["file.py", "file.html", "file.pdf"]),- (HTMLParser, ["file.md", "file.pdf", "file.txt"]),- (PDFParser, ["file.md", "file.txt", "file.html"]),- (TextParser, ["file.md", "file.html", "file.pdf"]),- (WordParser, ["file.pdf", "file.xlsx", "file.txt"]),- (ExcelParser, ["file.docx", "file.pdf", "file.txt"]),- (EPubParser, ["file.pdf", "file.docx", "file.zip"]),- (PowerPointParser, ["file.pdf", "file.docx", "file.txt"]),- (ZipParser, ["file.rar", "file.pdf", "file.docx"]),- ],- )- def test_can_parse_returns_false_for_wrong_extension(- self, parser_cls: type, filenames: List[str]- ) -> None:- parser = parser_cls()- for name in filenames:- assert not parser.can_parse(Path(name)), (- f"{parser_cls.__name__}.can_parse('{name}') should be False"- )--# ═══════════════════════════════════════════════════════════════════════════# Part 2 – Relative-path → Viking URI mapping# ═══════════════════════════════════════════════════════════════════════════diff --git a/tests/parse/test_directory_scan.py b/tests/parse/test_directory_scan.pyindex 09f7ca8e8..039b6a2eb 100644--- a/tests/parse/test_directory_scan.py+++ b/tests/parse/test_directory_scan.py@@ -135,7 +135,7 @@ class TestScanDirectoryClassification:assert "readme.md" in processable_relassert "doc.html" in processable_relassert "note.txt" in processable_rel- # Word, Excel, EPub, PowerPoint, Zip parsers+ # AnyDoc Office/EPUB and Zip parsersassert "report.docx" in processable_relassert "data.xlsx" in processable_relassert "book.epub" in processable_reldiff --git a/tests/parse/test_document_parser_threading.py b/tests/parse/test_document_parser_threading.pyindex 35e9165a0..e54d5021f 100644--- a/tests/parse/test_document_parser_threading.py+++ b/tests/parse/test_document_parser_threading.py@@ -1,9 +1,7 @@# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.# SPDX-License-Identifier: AGPL-3.0-"""Regression tests for offloading synchronous document conversions."""+"""Regression tests for offloading synchronous AnyDoc conversions."""-import sys-import zipfilefrom pathlib import Pathfrom types import SimpleNamespacefrom typing import Any, Callable@@ -11,7 +9,8 @@ from typing import Any, Callableimport pytestfrom openviking.parse.base import NodeType, ResourceNode, create_parse_result-from openviking.parse.parsers import epub, excel, legacy_doc, powerpoint, word+from openviking.parse.parsers import anydoc, anydoc_converter+from openviking_cli.utils.config.parser_config import AnydocConfigdef _stub_markdown_parse(parser) -> dict[str, Any]:@@ -49,229 +48,39 @@ def _patch_to_thread(monkeypatch, module) -> list[tuple[Callable[..., Any], tuplreturn calls+def _conversion(markdown: str = "# converted docx", source_format: str = "docx"):+ return SimpleNamespace(+ markdown=markdown,+ source_format=source_format,+ images_saved=0,+ assets_referenced=0,+ warnings=(),+ )++@pytest.mark.asyncio-async def test_word_parser_offloads_docx_conversion(monkeypatch, tmp_path: Path):- parser = word.WordParser()+async def test_anydoc_parser_offloads_docx_conversion(monkeypatch, tmp_path: Path):+ parser = anydoc.AnyDocParser(anydoc_config=AnydocConfig(max_table_rows=7))seen = _stub_markdown_parse(parser)- calls = _patch_to_thread(monkeypatch, word)- fake_docx = SimpleNamespace()- monkeypatch.setitem(sys.modules, "docx", fake_docx)+ calls = _patch_to_thread(monkeypatch, anydoc)- def convert(path: Path, docx_module, resource_name=None, storage=None) -> str:- assert docx_module is fake_docx- return "# converted docx"+ def convert(self, path: Path, *, resource_name, storage, max_table_rows=1000):+ return _conversion()- monkeypatch.setattr(parser, "_convert_to_markdown", convert)+ monkeypatch.setattr(anydoc_converter.AnyDocConverter, "convert", convert)source = tmp_path / "sample.docx"source.write_bytes(b"placeholder")result = await parser.parse(source)- # #2429 added resource_name/storage to the offloaded conversion call; assert- # the conversion was offloaded with the doc + docx module, without pinning the- # identity of the storage singleton.assert len(calls) == 1- func, args, _ = calls[0]- assert func is convert- assert args[0] == source- assert args[1] is fake_docx+ func, args, call_kwargs = calls[0]+ assert func.__func__ is convert+ assert args == (source,)+ assert call_kwargs["resource_name"] == "sample"+ assert call_kwargs["max_table_rows"] == 7+ storage = call_kwargs["storage"]assert seen["content"] == "# converted docx"+ assert seen["kwargs"]["allowed_media_dirs"] == [storage.media_dir]assert result.source_format == "docx"- assert result.parser_name == "WordParser"---async def test_word_parser_forwards_original_name_to_markdown(monkeypatch, tmp_path: Path):- # On single-file upload the on-disk path is a temp name (upload_<uuid>.docx)- # and the user's original filename arrives via source_name. WordParser must- # forward that name to MarkdownParser explicitly; otherwise parse_content can- # only fall back to the temp path and the resource ends up named upload_<uuid>.- parser = word.WordParser()- seen = _stub_markdown_parse(parser)- _patch_to_thread(monkeypatch, word)- monkeypatch.setitem(sys.modules, "docx", SimpleNamespace())- monkeypatch.setattr(parser, "_convert_to_markdown", lambda *args, **kwargs: "# converted docx")-- upload = tmp_path / "upload_abc123.docx"- upload.write_bytes(b"placeholder")-- await parser.parse(upload, source_name="季度报告.docx")-- forwarded = seen["kwargs"].get("source_name") or seen["kwargs"].get("resource_name")- assert forwarded == "季度报告.docx", seen["kwargs"]---async def test_word_parser_forwards_no_split_to_markdown(monkeypatch, tmp_path: Path):- parser = word.WordParser()- seen = _stub_markdown_parse(parser)- _patch_to_thread(monkeypatch, word)- monkeypatch.setitem(sys.modules, "docx", SimpleNamespace())- monkeypatch.setattr(parser, "_convert_to_markdown", lambda *args, **kwargs: "# converted docx")-- upload = tmp_path / "screenplay.docx"- upload.write_bytes(b"placeholder")-- await parser.parse(upload, split_content=False)-- assert seen["kwargs"]["split_content"] is False---async def test_excel_parser_offloads_xlsx_conversion(monkeypatch, tmp_path: Path):- parser = excel.ExcelParser()- seen = _stub_markdown_parse(parser)- calls = _patch_to_thread(monkeypatch, excel)- fake_openpyxl = SimpleNamespace()- monkeypatch.setitem(sys.modules, "openpyxl", fake_openpyxl)-- def convert(path: Path, openpyxl_module) -> str:- assert openpyxl_module is fake_openpyxl- return "# converted xlsx"-- monkeypatch.setattr(parser, "_convert_to_markdown", convert)- source = tmp_path / "sample.xlsx"- source.write_bytes(b"placeholder")-- result = await parser.parse(source)-- assert calls == [(convert, (source, fake_openpyxl), {})]- assert seen["content"] == "# converted xlsx"- assert result.source_format == "xlsx"- assert result.parser_name == "ExcelParser"---async def test_excel_parser_offloads_xls_conversion(monkeypatch, tmp_path: Path):- parser = excel.ExcelParser()- seen = _stub_markdown_parse(parser)- calls = _patch_to_thread(monkeypatch, excel)-- def convert(path: Path) -> str:- return "# converted xls"-- monkeypatch.setattr(parser, "_convert_xls_to_markdown", convert)- source = tmp_path / "sample.xls"- source.write_bytes(b"placeholder")-- result = await parser.parse(source)-- assert calls == [(convert, (source,), {})]- assert seen["content"] == "# converted xls"- assert result.source_format == "xls"- assert result.parser_name == "ExcelParser"---async def test_powerpoint_parser_offloads_pptx_conversion(monkeypatch, tmp_path: Path):- parser = powerpoint.PowerPointParser()- seen = _stub_markdown_parse(parser)- calls = _patch_to_thread(monkeypatch, powerpoint)- fake_pptx = SimpleNamespace()- monkeypatch.setitem(sys.modules, "pptx", fake_pptx)-- def convert(path: Path, pptx_module) -> str:- assert pptx_module is fake_pptx- return "# converted pptx"-- monkeypatch.setattr(parser, "_convert_to_markdown", convert)- source = tmp_path / "sample.pptx"- source.write_bytes(b"placeholder")-- result = await parser.parse(source)-- assert calls == [(convert, (source, fake_pptx), {})]- assert seen["content"] == "# converted pptx"- assert result.source_format == "pptx"- assert result.parser_name == "PowerPointParser"---async def test_epub_parser_offloads_epub_conversion(monkeypatch, tmp_path: Path):- parser = epub.EPubParser()- seen = _stub_markdown_parse(parser)- calls = _patch_to_thread(monkeypatch, epub)-- def convert(path: Path) -> str:- return "# converted epub"-- monkeypatch.setattr(parser, "_convert_to_markdown", convert)- source = tmp_path / "sample.epub"- source.write_bytes(b"placeholder")-- result = await parser.parse(source)-- assert calls == [(convert, (source,), {})]- assert seen["content"] == "# converted epub"- assert result.source_format == "epub"- assert result.parser_name == "EPubParser"---async def test_legacy_doc_parser_offloads_doc_extraction(monkeypatch, tmp_path: Path):- parser = legacy_doc.LegacyDocParser()- seen = _stub_markdown_parse(parser)- calls = _patch_to_thread(monkeypatch, legacy_doc)-- def extract(path: Path) -> str:- return "# converted doc"-- monkeypatch.setattr(parser, "_extract_text", extract)- source = tmp_path / "sample.doc"- source.write_bytes(b"placeholder")-- result = await parser.parse(source)-- assert calls == [(extract, (source,), {})]- assert seen["content"] == "# converted doc"- assert result.source_format == "doc"- assert result.parser_name == "LegacyDocParser"---async def test_legacy_doc_parser_routes_ooxml_payload_to_word_parser(monkeypatch, tmp_path: Path):- parser = legacy_doc.LegacyDocParser()- _stub_markdown_parse(parser)- source = tmp_path / "mislabeled.doc"- with zipfile.ZipFile(source, "w") as archive:- archive.writestr("[Content_Types].xml", "<Types />")- archive.writestr("word/document.xml", "<w:document />")-- seen: dict[str, Any] = {}-- async def parse_word(self, source_path, instruction="", **kwargs):- seen["source"] = source_path- seen["instruction"] = instruction- seen["kwargs"] = kwargs- return create_parse_result(- root=ResourceNode(type=NodeType.ROOT),- source_path=str(source_path),- source_format="docx",- parser_name="WordParser",- )-- monkeypatch.setattr(word.WordParser, "parse", parse_word)-- result = await parser.parse(- source,- instruction="preserve tables",- source_name="mislabeled.doc",- )-- assert seen == {- "source": source,- "instruction": "preserve tables",- "kwargs": {"source_name": "mislabeled.doc"},- }- assert result.source_format == "docx"- assert result.parser_name == "WordParser"---async def test_legacy_doc_parser_rejects_non_word_zip_payload(tmp_path: Path):- source = tmp_path / "not-a-word-document.doc"- with zipfile.ZipFile(source, "w") as archive:- archive.writestr("payload.bin", b"binary")-- with pytest.raises(ValueError, match="ZIP package"):- await legacy_doc.LegacyDocParser().parse(source)+ assert result.parser_name == "AnyDocParser"diff --git a/tests/parse/test_excel_process_pool.py b/tests/parse/test_excel_process_pool.pydeleted file mode 100644index 2f376a7d5..000000000--- a/tests/parse/test_excel_process_pool.py+++ /dev/null@@ -1,290 +0,0 @@-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.-# SPDX-License-Identifier: AGPL-3.0-"""Tests for ExcelParser._should_use_process_pool, the config-gated routing decision-that decides whether Excel→Markdown conversion + layout planning run in a-ProcessPoolExecutor child process instead of the main process's event loop.--A real ProcessPoolExecutor is not exercised here: spawning a child process per-test is slow and orthogonal to this decision, and the worker itself-(_build_excel_layout_in_process) is a thin, side-effect-free function that only-touches its own arguments. This covers the routing decision that gates it.-"""--from pathlib import Path--import pytest--from openviking.parse.parsers import excel as excel_module-from openviking.parse.parsers.excel import (- ExcelParser,- _EXCEL_PROCESS_POOL_MIN_BYTES,-)-from openviking_cli.utils.config.parser_config import (- ExcelConfig,- MarkdownConfig,- ParserConfig,-)---class TestShouldUseProcessPool:- def _parser(self, **excel_kwargs) -> ExcelParser:- return ExcelParser(config=ExcelConfig(**excel_kwargs))-- def _make_file(self, tmp_path: Path, suffix: str = ".xlsx", size: int = 10) -> Path:- path = tmp_path / f"sheet{suffix}"- path.write_bytes(b"x" * size)- return path-- def test_disabled_by_default(self, tmp_path: Path):- path = self._make_file(tmp_path, size=_EXCEL_PROCESS_POOL_MIN_BYTES)- assert self._parser()._should_use_process_pool(path, {}) is False-- def test_xls_never_uses_process_pool(self, tmp_path: Path):- # Legacy .xls goes through xlrd, not the openpyxl path the worker assumes.- path = self._make_file(tmp_path, suffix=".xls", size=_EXCEL_PROCESS_POOL_MIN_BYTES)- assert (- self._parser(enable_process_pool=True)._should_use_process_pool(path, {})- is False- )-- @pytest.mark.parametrize(- "kwargs",- [- {"enable_link_rewrite": True},- {"base_dir": Path(".")},- {"allowed_media_dirs": [Path(".")]},- ],- )- def test_link_or_media_rewrite_kwargs_disable_process_pool(self, tmp_path, kwargs):- # The worker never touches VikingFS/base_dir-relative media, so any parse- # that needs link/media rewriting must stay in-process.- path = self._make_file(tmp_path, size=_EXCEL_PROCESS_POOL_MIN_BYTES)- assert (- self._parser(enable_process_pool=True)._should_use_process_pool(path, kwargs)- is False- )-- def test_below_min_bytes_returns_false(self, tmp_path):- path = self._make_file(tmp_path, size=_EXCEL_PROCESS_POOL_MIN_BYTES - 1)- assert (- self._parser(enable_process_pool=True)._should_use_process_pool(path, {})- is False- )-- def test_at_or_above_min_bytes_returns_true(self, tmp_path):- path = self._make_file(tmp_path, size=_EXCEL_PROCESS_POOL_MIN_BYTES)- assert (- self._parser(enable_process_pool=True)._should_use_process_pool(path, {})- is True- )---async def test_process_pool_forwards_no_split_layout_flag(- monkeypatch: pytest.MonkeyPatch,- tmp_path: Path,-):- path = tmp_path / "sheet.xlsx"- path.write_bytes(b"xlsx")- parser = ExcelParser(config=ExcelConfig(enable_process_pool=True))- monkeypatch.setattr(- parser._md_parser,- "_create_temp_uri",- lambda: "viking://temp/test/excel",- )- captured = {}-- class _CapturingLoop:- def run_in_executor(self, _executor, callback):- captured["callback"] = callback- raise RuntimeError("layout callback captured")-- monkeypatch.setattr(excel_module.asyncio, "get_running_loop", lambda: _CapturingLoop())- monkeypatch.setattr(excel_module, "_get_excel_layout_executor", lambda _workers: object())-- with pytest.raises(RuntimeError, match="layout callback captured"):- await parser._parse_existing_path_process_pool(path, split_content=False)-- callback = captured["callback"]- assert callback.keywords["layout_kwargs"]["split_content"] is False---class TestExcelConfig:- def test_defaults(self):- cfg = ExcelConfig()- assert cfg.enable_process_pool is False- assert cfg.process_pool_workers == 2-- def test_validate_rejects_zero_workers(self):- cfg = ExcelConfig(process_pool_workers=0)- with pytest.raises(ValueError, match="process_pool_workers"):- cfg.validate()-- def test_openviking_config_accepts_excel_section(self):- from openviking_cli.utils.config.open_viking_config import OpenVikingConfig-- cfg = OpenVikingConfig.from_dict(- {- "excel": {- "enable_process_pool": True,- "process_pool_workers": 8,- }- }- )- assert cfg.excel.enable_process_pool is True- assert cfg.excel.process_pool_workers == 8---class TestExcelSectioningInheritance:- """Excel used to be registered with ``config.markdown``.-- A dedicated ``parsers.excel`` section must not silently change sectioning,- because section boundaries decide node structure and stable Viking URIs.- """-- def test_unset_sectioning_follows_markdown(self):- markdown = MarkdownConfig(max_section_size=512, max_section_chars=2000)- resolved = ExcelConfig.from_dict({}).with_sectioning_defaults_from(markdown)- assert resolved.max_section_size == 512- assert resolved.max_section_chars == 2000-- def test_explicit_excel_values_win(self):- markdown = MarkdownConfig(max_section_size=512, max_section_chars=2000)- resolved = ExcelConfig.from_dict(- {"max_section_size": 1024}- ).with_sectioning_defaults_from(markdown)- assert resolved.max_section_size == 1024- # Absent from parsers.excel, so this one keeps following Markdown.- assert resolved.max_section_chars == 2000-- def test_explicit_value_equal_to_default_is_not_overwritten(self):- """"Excel keeps 2048" is a legitimate config even though 2048 is the default.-- Comparing against class defaults cannot distinguish it from an absent- key, which would make the documented "explicit values win" contract- false for exactly this case.- """- markdown = MarkdownConfig(max_section_size=512)- resolved = ExcelConfig.from_dict(- {"max_section_size": 2048}- ).with_sectioning_defaults_from(markdown)- assert resolved.max_section_size == 2048-- def test_process_pool_knobs_are_never_inherited(self):- markdown = MarkdownConfig(max_section_size=512)- resolved = ExcelConfig.from_dict(- {"enable_process_pool": True, "process_pool_workers": 8}- ).with_sectioning_defaults_from(markdown)- assert resolved.enable_process_pool is True- assert resolved.process_pool_workers == 8- assert resolved.max_section_size == 512-- def test_default_markdown_leaves_excel_unchanged(self):- resolved = ExcelConfig.from_dict({}).with_sectioning_defaults_from(MarkdownConfig())- assert resolved == ExcelConfig()-- def test_missing_markdown_config_is_tolerated(self):- assert ExcelConfig.from_dict({}).with_sectioning_defaults_from(None) == ExcelConfig()-- def test_hand_built_config_is_treated_as_fully_explicit(self):- """A config built without from_dict carries no key provenance."""- markdown = MarkdownConfig(max_section_size=512)- resolved = ExcelConfig(max_section_size=2048).with_sectioning_defaults_from(markdown)- assert resolved.max_section_size == 2048-- def test_provenance_does_not_affect_equality(self):- assert ExcelConfig.from_dict({}) == ExcelConfig()- assert ExcelConfig.from_dict({"max_section_size": 2048}) == ExcelConfig()-- def test_absent_excel_section_still_inherits_through_full_config(self):- """Deployments predating parsers.excel must keep their Excel sectioning."""- from openviking_cli.utils.config.open_viking_config import OpenVikingConfig-- cfg = OpenVikingConfig.from_dict({"markdown": {"max_section_size": 512}})- resolved = cfg.excel.with_sectioning_defaults_from(cfg.markdown)- assert resolved.max_section_size == 512-- def test_key_provenance_does_not_leak_into_serialization(self):- """Provenance is bookkeeping, not config: it must not reach any output.-- A dataclass field would appear in asdict/model_dump and, being a- frozenset, would break JSON serialization for callers.- """- import dataclasses- import json-- cfg = ExcelConfig.from_dict({"max_section_size": 2048})- dumped = dataclasses.asdict(cfg)- assert "_explicit_fields" not in dumped- assert not any(key.startswith("_") for key in dumped), sorted(dumped)- json.dumps(dumped)-- def test_config_data_cannot_forge_key_provenance(self):- """Provenance must not be settable from a config file."""- with pytest.raises(ValueError):- ExcelConfig.from_dict({"_explicit_fields": ["max_section_size"]})-- def test_replace_and_copy_preserve_key_provenance(self):- import copy- import dataclasses-- markdown = MarkdownConfig(max_section_size=512)- cfg = ExcelConfig.from_dict({"max_section_size": 2048})-- for variant in (- dataclasses.replace(cfg, enable_process_pool=True),- copy.copy(cfg),- copy.deepcopy(cfg),- ):- resolved = variant.with_sectioning_defaults_from(markdown)- assert resolved.max_section_size == 2048-- def test_absent_excel_section_inherits_through_parser_loader(self):- """load_parser_configs_from_dict is a second config entry point."""- from openviking_cli.utils.config.parser_config import (- load_parser_configs_from_dict,- )-- configs = load_parser_configs_from_dict({"markdown": {"max_section_size": 512}})- resolved = configs["excel"].with_sectioning_defaults_from(configs["markdown"])- assert resolved.max_section_size == 512-- def test_explicit_excel_section_wins_through_full_config(self):- from openviking_cli.utils.config.open_viking_config import OpenVikingConfig-- cfg = OpenVikingConfig.from_dict(- {- "markdown": {"max_section_size": 512},- "excel": {"max_section_size": 2048, "enable_process_pool": True},- }- )- resolved = cfg.excel.with_sectioning_defaults_from(cfg.markdown)- assert resolved.max_section_size == 2048- assert resolved.enable_process_pool is True-- def test_registry_resolves_excel_against_markdown(self, monkeypatch):- from types import SimpleNamespace-- from openviking.parse import registry as registry_module-- config = SimpleNamespace(- text=ParserConfig(),- markdown=MarkdownConfig(max_section_size=512, max_section_chars=2000),- pdf=ParserConfig(),- html=ParserConfig(),- excel=ExcelConfig.from_dict({"enable_process_pool": True}),- image=ParserConfig(),- )- monkeypatch.setattr(- "openviking_cli.utils.config.get_openviking_config",- lambda: config,- )- monkeypatch.setattr(registry_module, "_default_registry", None)-- excel_parser = registry_module.get_registry().get_parser_for_file("book.xlsx")-- assert excel_parser.config.enable_process_pool is True- assert excel_parser.config.max_section_size == 512- # The inner MarkdownParser is what actually sections the converted sheet.- assert excel_parser._md_parser.config.max_section_size == 512- assert excel_parser._md_parser.config.max_section_chars == 2000diff --git a/tests/parse/test_markdown_apply_layout.py b/tests/parse/test_markdown_apply_layout.pyindex 9f1d1279b..a356a14ca 100644--- a/tests/parse/test_markdown_apply_layout.py+++ b/tests/parse/test_markdown_apply_layout.py@@ -1,12 +1,14 @@# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.# SPDX-License-Identifier: AGPL-3.0-"""Tests for MarkdownParser._apply_layout.+"""Tests for MarkdownParser layout application.``_apply_layout`` replays ``layout.ops`` verbatim: every mkdir op runs, and eachsection is written through ``_write_section`` so it keeps VikingFS's parent-dir-and encrypted-write handling. These tests pin that contract.+and encrypted-write handling. Concurrent parses must keep link-rewrite inputs+isolated while replaying their layouts."""+import asynciofrom unittest.mock import patchfrom openviking.parse.parsers.base_parser import BaseParser@@ -20,6 +22,7 @@ class FakeVikingFS:self.mkdir_calls = []self.files = {}self.raw_write_calls = []+ self._temp_counter = 0async def mkdir(self, uri, exist_ok=False, **kw):self.mkdir_calls.append(uri)@@ -37,6 +40,27 @@ class FakeVikingFS:# No images in this layout; let _ingest_local_images short-circuit.return {"matches": []}+ def create_temp_uri(self):+ self._temp_counter += 1+ return f"viking://temp/concurrent_{self._temp_counter}"+++class InterleavingFakeVikingFS(FakeVikingFS):+ """Pause the first parse until a second parse enters layout application."""++ def __init__(self):+ super().__init__()+ self._initial_mkdir_count = 0+ self._both_parses_started = asyncio.Event()++ async def mkdir(self, uri, exist_ok=False, **kw):+ await super().mkdir(uri, exist_ok=exist_ok, **kw)+ if self._initial_mkdir_count < 2:+ self._initial_mkdir_count += 1+ if self._initial_mkdir_count == 2:+ self._both_parses_started.set()+ await self._both_parses_started.wait()+class TestApplyLayout:def _layout(self) -> _Layout:@@ -69,14 +93,33 @@ class TestApplyLayout:"viking://temp/root/doc/sec/b.md": "B","viking://temp/root/doc/other/c.md": "C",}+ assert fake.raw_write_calls == []- async def test_sections_go_through_write_section_not_raw_write(self):- """Section writes must keep VikingFS parent-dir/encrypted-write handling."""- fake = FakeVikingFS()+ async def test_concurrent_parses_keep_link_rewrite_inputs_isolated(self, tmp_path):+ fake = InterleavingFakeVikingFS()parser = MarkdownParser()++ roots = {}+ for name in ("a", "b"):+ root = tmp_path / name+ root.mkdir()+ (root / f"{name}.md").write_text("source", encoding="utf-8")+ (root / f"{name}.txt").write_text("target", encoding="utf-8")+ roots[name] = root++ async def parse(name):+ root = roots[name]+ return await parser.parse_content(+ f"[target](./{name}.txt)",+ source_path=str(root / f"{name}.md"),+ base_dir=root,+ enable_link_rewrite=True,+ link_rewrite_root=str(root),+ )+with patch.object(BaseParser, "_get_viking_fs", return_value=fake):- await parser._apply_layout(self._layout())+ results = await asyncio.gather(parse("a"), parse("b"))- assert fake.raw_write_calls == [], (- f"Section writes must not bypass write_file: {fake.raw_write_calls}"- )+ for name, result in zip(("a", "b"), results, strict=True):+ uri = f"{result.temp_dir_path}/{name}/{name}.md"+ assert fake.files[uri] == f"[target](../{name}.txt)"diff --git a/tests/parse/test_markdown_link_rewrite.py b/tests/parse/test_markdown_link_rewrite.pyindex 9c0ad23fb..f878a2f1b 100644--- a/tests/parse/test_markdown_link_rewrite.py+++ b/tests/parse/test_markdown_link_rewrite.py@@ -9,7 +9,7 @@ import pytestfrom openviking.parse.parsers.base_parser import BaseParserfrom openviking.parse.parsers.directory import DirectoryParser-from openviking.parse.parsers.markdown import MarkdownParser+from openviking.parse.parsers.markdown import MarkdownParser, _RewriteContextclass TestRewriteRelativeLinks:@@ -28,12 +28,21 @@ class TestRewriteRelativeLinks:return kbasync def _rewrite(self, parser, kb, content, section_subpath=""):- return await parser._rewrite_relative_links(- content,+ rewrite_ctx = _RewriteContext(source_path=str(kb / "文档.md"),doc_name="文档",- section_subpath=section_subpath,+ root_dir="viking://temp/test/文档",import_root=str(kb),+ base_dir=kb,+ allowed_media_dirs=None,+ split_content=True,+ adaptive_flatten_requested=False,+ flatten_single_output=False,+ )+ return await parser._rewrite_relative_links(+ content,+ section_subpath=section_subpath,+ rewrite_ctx=rewrite_ctx,)async def test_md_target_becomes_directory(self, tmp_path: Path):@@ -172,7 +181,8 @@ class TestRewriteRelativeLinks:kb = self._make_tree(tmp_path)p = self._parser()- async def fake_bare_layout(_path): # 模拟未来:目标入库为单个裸文件,无 <dir>/ 包裹+ async def fake_bare_layout(_path, _context):+ # 模拟未来:目标入库为单个裸文件,无 <dir>/ 包裹return {"文档.md": "# 目标\n\n内容"}p._target_split_files = fake_bare_layout # type: ignore[method-assign]diff --git a/tests/test_config_loader.py b/tests/test_config_loader.pyindex 9750c64f3..7977b9fcf 100644--- a/tests/test_config_loader.py+++ b/tests/test_config_loader.py@@ -206,9 +206,10 @@ def test_generic_code_hosting_domains_load_from_config():assert config.code_hosting_domains == ["git.generic.example.com"]-def test_openviking_config_rejects_unknown_nested_parser_section(monkeypatch):+def test_openviking_config_handles_nested_parser_compatibility(monkeypatch):monkeypatch.setenv(OPENVIKING_CONFIG_ENV, "/tmp/codex-no-config.json")+ from openviking_cli.utils.config import open_viking_config as config_modulefrom openviking_cli.utils.config.open_viking_config import (OpenVikingConfig,OpenVikingConfigSingleton,@@ -228,6 +229,28 @@ def test_openviking_config_rejects_unknown_nested_parser_section(monkeypatch):})+ errors: list[str] = []+ monkeypatch.setattr(+ config_module._get_config_logger(),+ "error",+ lambda message, *args, **kwargs: errors.append(message % args if args else message),+ )+ config = OpenVikingConfig.from_dict(+ {+ "embedding": {+ "dense": {+ "provider": "openai",+ "api_key": "test-key",+ "model": "text-embedding-3-small",+ }+ },+ "parsers": {"excel": {"enable_process_pool": True}},+ }+ )++ assert config.anydoc.enabled is True+ assert any("Config field 'parsers.excel' was removed and is ignored" in item for item in errors)+OpenVikingConfigSingleton.reset_instance()diff --git a/uv.lock b/uv.lockindex 988610c7b..1a1d267d5 100644--- a/uv.lock+++ b/uv.lock@@ -1201,19 +1201,6 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/b0/0d/9feae160378a3553fa9a339b0e9c1a048e147a4127210e286ef18b730f03/durationpy-0.10-py3-none-any.whl", hash = "sha256:3b41e1b601234296b4fb368338fdcd3e13e0b4fb5b67345948f4f2bf9868b286", size = 3922, upload-time = "2025-05-17T13:52:36.463Z" },]-[[package]]-name = "ebooklib"-version = "0.20"-source = { registry = "https://pypi.org/simple" }-dependencies = [- { name = "lxml" },- { name = "six" },-]-sdist = { url = "https://files.pythonhosted.org/packages/77/85/322e8882a582d4b707220d1929cfb74c125f2ba513991edbce40dbc462de/ebooklib-0.20.tar.gz", hash = "sha256:35e2f9d7d39907be8d39ae2deb261b19848945903ae3dbb6577b187ead69e985", size = 127066, upload-time = "2025-10-26T20:56:20.968Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/bf/ee/aa015c5de8b0dc42a8e507eae8c2de5d1c0e068c896858fec6d502402ed6/ebooklib-0.20-py3-none-any.whl", hash = "sha256:fff5322517a37e31c972d27be7d982cc3928c16b3dcc5fd7e8f7c0f5d7bcf42b", size = 40995, upload-time = "2025-10-26T20:56:19.104Z" },-]-[[package]]name = "ecdsa"version = "0.19.2"@@ -1226,15 +1213,6 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/51/79/119091c98e2bf49e24ed9f3ae69f816d715d2904aefa6a2baa039a2ba0b0/ecdsa-0.19.2-py2.py3-none-any.whl", hash = "sha256:840f5dc5e375c68f36c1a7a5b9caad28f95daa65185c9253c0c08dd952bb7399", size = 150818, upload-time = "2026-03-26T09:58:15.808Z" },]-[[package]]-name = "et-xmlfile"-version = "2.0.0"-source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/d3/38/af70d7ab1ae9d4da450eeec1fa3918940a5fafb9055e934af8d6eb0c2313/et_xmlfile-2.0.0.tar.gz", hash = "sha256:dab3f4764309081ce75662649be815c4c9081e88f0837825f90fd28317d4da54", size = 17234, upload-time = "2024-10-25T17:25:40.039Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/c1/8b/5fe2cc11fee489817272089c4203e679c63b570a5aaeb18d852ae3cbba6a/et_xmlfile-2.0.0-py3-none-any.whl", hash = "sha256:7a91720bc756843502c3b7504c77b8fe44217c85c537d85037f0f536151b2caa", size = 18059, upload-time = "2024-10-25T17:25:39.051Z" },-]-[[package]]name = "exceptiongroup"version = "1.3.1"@@ -1365,6 +1343,21 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/f9/0b/de6f54d4a8bedfe8645c41497f3c18d749f0bd3218170c667bf4b81d0cdd/filelock-3.25.0-py3-none-any.whl", hash = "sha256:5ccf8069f7948f494968fc0713c10e5c182a9c9d9eef3a636307a20c2490f047", size = 26427, upload-time = "2026-03-01T15:08:44.593Z" },]+[[package]]+name = "firecrawl-anydoc"+version = "0.1.9"+source = { registry = "https://pypi.org/simple" }+sdist = { url = "https://files.pythonhosted.org/packages/aa/16/e37d4f284482a4f30e0502ce439a9fd8ccf013e6ae4438b4dfa209e48750/firecrawl_anydoc-0.1.9.tar.gz", hash = "sha256:0dfc64b82b4e971143dd6f0e4f39f8152fcc38be6523663682a241d8f3301b14", size = 197914, upload-time = "2026-08-13T21:48:27.989Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/42/37/866bf6270ad8552eaf6edf3b8e1a360acf3e8764140641361ae7b440514e/firecrawl_anydoc-0.1.9-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:ebd956831822d3ea1831253406736e0e4d9b217e3471eb68c9bf2eba9302ce73", size = 3469321, upload-time = "2026-08-13T21:48:16.775Z" },+ { url = "https://files.pythonhosted.org/packages/b1/9a/8fbe0726cdf69154e1eb05c119332b1fdfd8200eeafcfd1675741ac5f03d/firecrawl_anydoc-0.1.9-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:a2372d97826e8e7e68fd28cd415d08815f6aa66dcc09c6b423ac33dd2dfa91cf", size = 3287520, upload-time = "2026-08-13T21:48:18.37Z" },+ { url = "https://files.pythonhosted.org/packages/b0/ce/86a626224c365d8de65699d6ac8f4626ac05e7e75c3ae76c82f4904a07d2/firecrawl_anydoc-0.1.9-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ba1c19bdaa97f4850daf3fb4ca2855cefa146b5621ce7b22a71d158d3b84e54e", size = 3331094, upload-time = "2026-08-13T21:48:19.961Z" },+ { url = "https://files.pythonhosted.org/packages/0e/28/00d1d48fe205ab66eda4a2a22cce44aadf586ef1495e703349b791c8de32/firecrawl_anydoc-0.1.9-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:deea7a270fccda9be29c73235e938ca32e0f703e1ae120f08bf0fdb39301c6b4", size = 3555945, upload-time = "2026-08-13T21:48:21.822Z" },+ { url = "https://files.pythonhosted.org/packages/c8/25/ac52d98c5d082ef92396b2fcb398745a3b054816c67a12d19de86c1ba535/firecrawl_anydoc-0.1.9-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:de0858decd18188b0b88545689544fb3703ec08ade37f63c2857e4414e99bdf1", size = 3510861, upload-time = "2026-08-13T21:48:23.369Z" },+ { url = "https://files.pythonhosted.org/packages/ca/d3/e80314b5746c4ca85150f3ab2a527b2167012c16e706e148733211cb88b3/firecrawl_anydoc-0.1.9-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:e4e1765ed5574fa502931f8bfaec1f87af913afa5bb432aaaf5f9c094efc7f43", size = 3797896, upload-time = "2026-08-13T21:48:25.061Z" },+ { url = "https://files.pythonhosted.org/packages/8e/d0/e7b35c2365498d5e9f1bdb4b3ded78571b6dafbaba1880538490d27b71c9/firecrawl_anydoc-0.1.9-cp310-abi3-win_amd64.whl", hash = "sha256:aa6a5ca2e10939a87c9bd21c918ef8b146ad0061fedae661d4483ef5a5bbdf60", size = 3642084, upload-time = "2026-08-13T21:48:26.702Z" },+]+[[package]]name = "frozenlist"version = "1.8.0"@@ -3409,15 +3402,6 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/be/9c/92789c596b8df838baa98fa71844d84283302f7604ed565dafe5a6b5041a/oauthlib-3.3.1-py3-none-any.whl", hash = "sha256:88119c938d2b8fb88561af5f6ee0eec8cc8d552b7bb1f712743136eb7523b7a1", size = 160065, upload-time = "2025-06-19T22:48:06.508Z" },]-[[package]]-name = "olefile"-version = "0.47"-source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/69/1b/077b508e3e500e1629d366249c3ccb32f95e50258b231705c09e3c7a4366/olefile-0.47.zip", hash = "sha256:599383381a0bf3dfbd932ca0ca6515acd174ed48870cbf7fee123d698c192c1c", size = 112240, upload-time = "2023-12-01T16:22:53.025Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/17/d3/b64c356a907242d719fc668b71befd73324e47ab46c8ebbbede252c154b2/olefile-0.47-py2.py3-none-any.whl", hash = "sha256:543c7da2a7adadf21214938bb79c83ea12b473a4b6ee4ad4bf854e7715e13d1f", size = 114565, upload-time = "2023-12-01T16:22:51.518Z" },-]-[[package]]name = "openai"version = "2.24.0"@@ -3454,18 +3438,6 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/17/af/0379ba45fd2f4c14be3b62a78714b1b34705d0472cdd8d62df47a8fb1ffd/opencode_ai-0.1.0a36-py3-none-any.whl", hash = "sha256:73bce046ec3d841dadb31bed658a5049f25cf0c2e3ebda2e2d1ff37d792ea1b5", size = 110221, upload-time = "2025-08-27T02:38:57.501Z" },]-[[package]]-name = "openpyxl"-version = "3.1.5"-source = { registry = "https://pypi.org/simple" }-dependencies = [- { name = "et-xmlfile" },-]-sdist = { url = "https://files.pythonhosted.org/packages/3d/f9/88d94a75de065ea32619465d2f77b29a0469500e99012523b91cc4141cd1/openpyxl-3.1.5.tar.gz", hash = "sha256:cf0e3cf56142039133628b5acffe8ef0c12bc902d2aadd3e0fe5878dc08d1050", size = 186464, upload-time = "2024-06-28T14:03:44.161Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/c0/da/977ded879c29cbd04de313843e76868e6e13408a94ed6b987245dc7c8506/openpyxl-3.1.5-py2.py3-none-any.whl", hash = "sha256:5282c12b107bffeef825f4617dc029afaf41d0ea60823bbb665ef3079dc79de2", size = 250910, upload-time = "2024-06-28T14:03:41.161Z" },-]-[[package]]name = "opensandbox"version = "0.1.5"@@ -3640,9 +3612,9 @@ dependencies = [{ name = "charset-normalizer" },{ name = "cryptography" },{ name = "defusedxml" },- { name = "ebooklib" },{ name = "fastapi" },{ name = "feedparser" },+ { name = "firecrawl-anydoc" },{ name = "grep-ast" },{ name = "httpx" },{ name = "jinja2" },@@ -3651,9 +3623,7 @@ dependencies = [{ name = "litellm" },{ name = "loguru" },{ name = "mcp" },- { name = "olefile" },{ name = "openai" },- { name = "openpyxl" },{ name = "opentelemetry-api" },{ name = "opentelemetry-exporter-otlp-proto-grpc" },{ name = "opentelemetry-exporter-otlp-proto-http" },@@ -3665,9 +3635,7 @@ dependencies = [{ name = "pdfplumber" },{ name = "protobuf" },{ name = "pydantic" },- { name = "python-docx" },{ name = "python-multipart" },- { name = "python-pptx" },{ name = "pyyaml" },{ name = "requests" },{ name = "scrapy" },@@ -3691,7 +3659,6 @@ dependencies = [{ name = "uvicorn" },{ name = "volcengine" },{ name = "volcengine-python-sdk", extra = ["ark"] },- { name = "xlrd" },{ name = "xxhash" },]@@ -3708,6 +3675,7 @@ benchmark = [{ name = "langchain-openai" },{ name = "pandas", version = "2.3.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },{ name = "pandas", version = "3.0.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" },+ { name = "python-docx" },{ name = "tiktoken" },]bot = [@@ -3823,9 +3791,9 @@ requires-dist = [{ name = "defusedxml", specifier = ">=0.7.1" },{ name = "diff-match-patch", marker = "extra == 'test'", specifier = ">=20200713" },{ name = "dingtalk-stream", marker = "extra == 'bot'", specifier = ">=0.4.0" },- { name = "ebooklib", specifier = ">=0.18.0" },{ name = "fastapi", specifier = ">=0.128.0" },{ name = "feedparser", specifier = ">=6.0.0" },+ { name = "firecrawl-anydoc", specifier = ">=0.1.8" },{ name = "fusepy", marker = "extra == 'bot'", specifier = ">=3.0.1" },{ name = "google-genai", marker = "extra == 'gemini'", specifier = ">=1.0.0" },{ name = "google-genai", marker = "extra == 'gemini-async'", specifier = ">=1.0.0" },@@ -3852,10 +3820,8 @@ requires-dist = [{ name = "msgpack", marker = "extra == 'bot'", specifier = ">=1.0.8" },{ name = "mypy", marker = "extra == 'dev'", specifier = ">=1.0.0" },{ name = "myst-parser", marker = "extra == 'doc'", specifier = ">=2.0.0" },- { name = "olefile", specifier = ">=0.47" },{ name = "openai", specifier = ">=1.0.0" },{ name = "opencode-ai", marker = "extra == 'bot'", specifier = ">=0.1.0a0" },- { name = "openpyxl", specifier = ">=3.0.0" },{ name = "opensandbox", marker = "extra == 'bot'", specifier = ">=0.1.0" },{ name = "opensandbox-server", marker = "extra == 'bot'", specifier = ">=0.1.0" },{ name = "opentelemetry-api", specifier = ">=1.14" },@@ -3881,13 +3847,12 @@ requires-dist = [{ name = "pytest-asyncio", marker = "extra == 'test'", specifier = ">=0.21.0" },{ name = "pytest-cov", marker = "extra == 'test'", specifier = ">=4.0.0" },{ name = "pytest-xdist", marker = "extra == 'test'", specifier = ">=3.5.0" },- { name = "python-docx", specifier = ">=1.0.0" },+ { name = "python-docx", marker = "extra == 'benchmark'", specifier = ">=1.0.0" },{ name = "python-engineio", marker = "extra == 'bot'", specifier = ">=4.13.2" },{ name = "python-jose", extras = ["cryptography"], marker = "extra == 'auth'", specifier = ">=3.3.0" },{ name = "python-jose", extras = ["cryptography"], marker = "extra == 'bot'", specifier = ">=3.3.0" },{ name = "python-ldap", marker = "extra == 'auth'", specifier = ">=3.4.0" },{ name = "python-multipart", specifier = ">=0.0.31" },- { name = "python-pptx", specifier = ">=1.0.0" },{ name = "python-socketio", marker = "extra == 'bot'", specifier = ">=5.16.2" },{ name = "python-socks", extras = ["asyncio"], marker = "extra == 'bot'", specifier = ">=2.4.0" },{ name = "python-telegram-bot", extras = ["socks"], marker = "extra == 'bot'", specifier = ">=21.0" },@@ -3932,7 +3897,6 @@ requires-dist = [{ name = "websocket-client", marker = "extra == 'bot'", specifier = ">=1.6.0" },{ name = "websockets", marker = "extra == 'bot'", specifier = ">=12.0" },{ name = "wheel", marker = "extra == 'build'" },- { name = "xlrd", specifier = ">=2.0.1" },{ name = "xxhash", specifier = ">=3.0.0" },]provides-extras = ["test", "auth", "dev", "doc", "eval", "gemini", "gemini-async", "ocr", "build", "bot", "benchmark", "local-embed"]@@ -5130,21 +5094,6 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/e1/04/e8135ebd1ad02c56ec633277529b2602ff99ff634be76cdba5744cf554fd/python_multipart-0.0.32-py3-none-any.whl", hash = "sha256:ff6d3f776f16878c894e52e107296ffc890e913c611b1a4ec6c44e2821fe2e23", size = 30042, upload-time = "2026-06-04T16:18:57.319Z" },]-[[package]]-name = "python-pptx"-version = "1.0.2"-source = { registry = "https://pypi.org/simple" }-dependencies = [- { name = "lxml" },- { name = "pillow" },- { name = "typing-extensions" },- { name = "xlsxwriter" },-]-sdist = { url = "https://files.pythonhosted.org/packages/52/a9/0c0db8d37b2b8a645666f7fd8accea4c6224e013c42b1d5c17c93590cd06/python_pptx-1.0.2.tar.gz", hash = "sha256:479a8af0eaf0f0d76b6f00b0887732874ad2e3188230315290cd1f9dd9cc7095", size = 10109297, upload-time = "2024-08-07T17:33:37.772Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/d9/4f/00be2196329ebbff56ce564aa94efb0fbc828d00de250b1980de1a34ab49/python_pptx-1.0.2-py3-none-any.whl", hash = "sha256:160838e0b8565a8b1f67947675886e9fea18aa5e795db7ae531606d68e785cba", size = 472788, upload-time = "2024-08-07T17:33:28.192Z" },-]-[[package]]name = "python-socketio"version = "5.16.3"@@ -7047,24 +6996,6 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/a4/f5/10b68b7b1544245097b2a1b8238f66f2fc6dcaeb24ba5d917f52bd2eed4f/wsproto-1.3.2-py3-none-any.whl", hash = "sha256:61eea322cdf56e8cc904bd3ad7573359a242ba65688716b0710a5eb12beab584", size = 24405, upload-time = "2025-11-20T18:18:00.454Z" },]-[[package]]-name = "xlrd"-version = "2.0.2"-source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/07/5a/377161c2d3538d1990d7af382c79f3b2372e880b65de21b01b1a2b78691e/xlrd-2.0.2.tar.gz", hash = "sha256:08b5e25de58f21ce71dc7db3b3b8106c1fa776f3024c54e45b45b374e89234c9", size = 100167, upload-time = "2025-06-14T08:46:39.039Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/1a/62/c8d562e7766786ba6587d09c5a8ba9f718ed3fa8af7f4553e8f91c36f302/xlrd-2.0.2-py2.py3-none-any.whl", hash = "sha256:ea762c3d29f4cca48d82df517b6d89fbce4db3107f9d78713e48cd321d5c9aa9", size = 96555, upload-time = "2025-06-14T08:46:37.766Z" },-]--[[package]]-name = "xlsxwriter"-version = "3.2.9"-source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/46/2c/c06ef49dc36e7954e55b802a8b231770d286a9758b3d936bd1e04ce5ba88/xlsxwriter-3.2.9.tar.gz", hash = "sha256:254b1c37a368c444eac6e2f867405cc9e461b0ed97a3233b2ac1e574efb4140c", size = 215940, upload-time = "2025-09-16T00:16:21.63Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/3a/0c/3662f4a66880196a590b202f0db82d919dd2f89e99a27fadef91c4a33d41/xlsxwriter-3.2.9-py3-none-any.whl", hash = "sha256:9a5db42bc5dff014806c58a20b9eae7322a134abb6fce3c92c181bfb275ec5b3", size = 175315, upload-time = "2025-09-16T00:16:20.108Z" },-]-[[package]]name = "xxhash"version = "3.6.0"