Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

refactor(parser):Unify Office Document Parsing with AnyDoc and Restore Compatibility Safeguards (#4279)

zgy7ee75611212a
diff --git a/docs/en/api/02-resources.md b/docs/en/api/02-resources.md
index e52e432b9..0ef353724 100644
--- a/docs/en/api/02-resources.md
+++ b/docs/en/api/02-resources.md
@@ -15,16 +15,16 @@ OpenViking supports various resource types, categorized by functionality:
| PDF | `.pdf` | Supports local parsing and MinerU API conversion |
| Markdown | `.md`, `.markdown`, `.mdown`, `.mkd` | Native support, extracts structure and stores in segments |
| HTML | `.html`, `.htm` | Cleans navigation/ads and extracts content, converts to Markdown |
-| Word | `.docx` | Extracts text, headings, tables and converts to Markdown |
+| Word | `.doc`, `.docx`, `.docm`, `.odt`, `.rtf` | Uses anydoc to extract text, headings, tables, and embedded images into Markdown |
| Plain Text | `.txt`, `.text` | Direct import and processing |
-| EPUB | `.epub` | E-book format, supports ebooklib or manual extraction |
+| EPUB | `.epub` | Uses anydoc to convert e-book content and embedded images to Markdown |
**Spreadsheets & Presentations**
| Type | Extensions | Description |
|------|------------|-------------|
-| Excel | `.xlsx`, `.xls`, `.xlsm` | Supports new and legacy Excel formats, converts to Markdown tables by worksheet |
-| PowerPoint | `.pptx` | Extracts content by slide, supports extracting notes |
+| Excel | `.xlsx`, `.xls`, `.xlsm`, `.xlsb`, `.ods`, `.csv` | Uses anydoc to convert worksheets to Markdown tables |
+| PowerPoint | `.pptx`, `.ppt`, `.pptm`, `.pps`, `.ppsx`, `.ppsm`, `.pot`, `.odp` | Uses anydoc to extract slide content and embedded images into Markdown |
**Code**
diff --git a/docs/en/configuration/01-server.md b/docs/en/configuration/01-server.md
index 647984257..88f4fdbe0 100644
--- a/docs/en/configuration/01-server.md
+++ b/docs/en/configuration/01-server.md
@@ -347,7 +347,9 @@ Parsers live under `parsers`:
"audio": {},
"video": {},
"markdown": {},
- "excel": {},
+ "anydoc": {
+ "enabled": true
+ },
"html": {},
"text": {},
"directory": {},
@@ -369,7 +371,7 @@ Parsers live under `parsers`:
| `image` | Image understanding and OCR |
| `audio`, `video` | Audio/video parsing |
| `markdown`, `html`, `text` | Text document chunking |
-| `excel` | Workbook parsing and chunking |
+| `anydoc` | Office and EPUB conversion; `enabled=false` rejects those formats |
| `directory` | Directory scanning and ignore rules |
| `feishu` | Feishu/Lark access and parsing |
| `webfeed` | Sitemap, RSS, and Atom ingestion |
diff --git a/docs/zh/api/02-resources.md b/docs/zh/api/02-resources.md
index 98f4c28dd..8c54824ca 100644
--- a/docs/zh/api/02-resources.md
+++ b/docs/zh/api/02-resources.md
@@ -14,15 +14,15 @@ OpenViking 支持多种资源类型,按照功能分类如下:
| PDF | `.pdf` | 支持本地解析和 MinerU API 转换 |
| Markdown | `.md`, `.markdown`, `.mdown`, `.mkd` | 原生支持,会提取结构并分段存储 |
| HTML | `.html`, `.htm` | 清理导航/广告后提取内容,转换为 Markdown |
-| Word | `.docx` | 提取文本、标题、表格并转换为 Markdown |
+| Word | `.doc`, `.docx`, `.docm`, `.odt`, `.rtf` | 基于 anydoc 提取文本、标题、表格和嵌入图片并转换为 Markdown |
| 纯文本 | `.txt`, `.text` | 直接导入处理 |
-| EPUB | `.epub` | 电子书格式,支持 ebooklib 或手动提取 |
+| EPUB | `.epub` | 基于 anydoc 将电子书内容和嵌入图片转换为 Markdown |
表格类
| 类型 | 扩展名 | 说明 |
|------|--------|------|
-| Excel | `.xlsx`, `.xls`, `.xlsm` | 支持新版和老版 Excel,按工作表转换为 Markdown 表格 |
-| PowerPoint | `.pptx` | 按幻灯片提取内容,支持提取备注 |
+| Excel | `.xlsx`, `.xls`, `.xlsm`, `.xlsb`, `.ods`, `.csv` | 基于 anydoc 按工作表转换为 Markdown 表格 |
+| PowerPoint | `.pptx`, `.ppt`, `.pptm`, `.pps`, `.ppsx`, `.ppsm`, `.pot`, `.odp` | 基于 anydoc 按幻灯片提取内容和嵌入图片并转换为 Markdown |
代码类
| 类型 | 资源名 | 说明 |
diff --git a/docs/zh/configuration/01-server.md b/docs/zh/configuration/01-server.md
index 2e6d98728..f8325eb2f 100644
--- a/docs/zh/configuration/01-server.md
+++ b/docs/zh/configuration/01-server.md
@@ -347,7 +347,9 @@ Provider 和密钥管理配置见[加密指南](../guides/08-encryption.md)。
"audio": {},
"video": {},
"markdown": {},
- "excel": {},
+ "anydoc": {
+ "enabled": true
+ },
"html": {},
"text": {},
"directory": {},
@@ -369,7 +371,7 @@ Provider 和密钥管理配置见[加密指南](../guides/08-encryption.md)。
| `image` | 图片理解和 OCR |
| `audio`、`video` | 音视频内容解析 |
| `markdown`、`html`、`text` | 文本文档分段 |
-| `excel` | Excel 工作表解析与分段 |
+| `anydoc` | Office 和 EPUB 转换;`enabled=false` 时拒绝这些格式 |
| `directory` | 目录扫描和忽略规则 |
| `feishu` | 飞书文档访问与解析 |
| `webfeed` | Sitemap、RSS 和 Atom 导入 |
diff --git a/examples/ov.conf.example b/examples/ov.conf.example
index c8676a5e1..b18146705 100644
--- a/examples/ov.conf.example
+++ b/examples/ov.conf.example
@@ -309,6 +309,9 @@
"include_metadata": true,
"max_heading_depth": 3,
},
+ "anydoc": {
+ "enabled": true,
+ },
"html": {
"extract_text_only": false,
"preserve_structure": true,
diff --git a/openviking/parse/image_validation.py b/openviking/parse/image_validation.py
new file mode 100644
index 000000000..f8ebe4b33
--- /dev/null
+++ b/openviking/parse/image_validation.py
@@ -0,0 +1,58 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Shared acceptance policy for images embedded in parsed documents."""
+
+import io
+from pathlib import Path
+
+from openviking_cli.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+IMAGE_MIN_SIDE = 14
+IMAGE_MIN_PIXELS = 196
+IMAGE_MAX_PIXELS = 36_000_000
+IMAGE_MIN_ASPECT_RATIO = 1 / 150
+IMAGE_MAX_ASPECT_RATIO = 150
+IMAGE_MAX_FILE_BYTES = 10 * 1024 * 1024
+
+
+def is_valid_image(image_bytes: bytes, source_path: Path) -> bool:
+ """Return whether an extracted image is safe and useful to ingest."""
+ if len(image_bytes) > IMAGE_MAX_FILE_BYTES:
+ logger.warning(f"[ImageValidation] Image exceeds 10MB, skipping: {source_path}")
+ return False
+
+ try:
+ from PIL import Image
+
+ with Image.open(io.BytesIO(image_bytes)) as image:
+ width, height = image.size
+ except Exception as exc:
+ logger.warning(
+ f"[ImageValidation] Cannot read image dimensions, skipping {source_path}: {exc}"
+ )
+ return False
+
+ if width <= IMAGE_MIN_SIDE or height <= IMAGE_MIN_SIDE:
+ logger.warning(
+ f"[ImageValidation] Image side too small ({width}x{height}), skipping: {source_path}"
+ )
+ return False
+
+ pixels = width * height
+ if pixels < IMAGE_MIN_PIXELS or pixels > IMAGE_MAX_PIXELS:
+ logger.warning(
+ f"[ImageValidation] Image pixel count out of range ({pixels}), skipping: {source_path}"
+ )
+ return False
+
+ aspect_ratio = width / height
+ if aspect_ratio < IMAGE_MIN_ASPECT_RATIO or aspect_ratio > IMAGE_MAX_ASPECT_RATIO:
+ logger.warning(
+ f"[ImageValidation] Image aspect ratio out of range ({aspect_ratio:.4f}), "
+ f"skipping: {source_path}"
+ )
+ return False
+
+ return True
diff --git a/openviking/parse/parsers/__init__.py b/openviking/parse/parsers/__init__.py
index 5f17740d0..bbbe3aede 100644
--- a/openviking/parse/parsers/__init__.py
+++ b/openviking/parse/parsers/__init__.py
@@ -1,26 +1,20 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
+from .anydoc import AnyDocParser
from .base_parser import BaseParser
-from .epub import EPubParser
-from .excel import ExcelParser
from .html import HTMLParser
from .markdown import MarkdownParser
from .pdf import PDFParser
-from .powerpoint import PowerPointParser
from .text import TextParser
-from .word import WordParser
from .zip_parser import ZipParser
__all__ = [
"BaseParser",
- "EPubParser",
- "ExcelParser",
+ "AnyDocParser",
"HTMLParser",
"MarkdownParser",
"PDFParser",
- "PowerPointParser",
"TextParser",
- "WordParser",
"ZipParser",
]
diff --git a/openviking/parse/parsers/anydoc.py b/openviking/parse/parsers/anydoc.py
new file mode 100644
index 000000000..09de5747d
--- /dev/null
+++ b/openviking/parse/parsers/anydoc.py
@@ -0,0 +1,126 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+"""Unified AnyDoc parser for Office documents and EPUB files."""
+
+import asyncio
+import time
+from pathlib import Path
+from typing import List, Optional, Union
+
+from openviking.parse.base import ParseResult
+from openviking.parse.parsers.base_parser import BaseParser
+from openviking_cli.utils.config.parser_config import AnydocConfig, ParserConfig
+
+_SUPPORTED_EXTENSIONS = [
+ ".doc",
+ ".docx",
+ ".docm",
+ ".odt",
+ ".rtf",
+ ".pptx",
+ ".ppt",
+ ".pptm",
+ ".pps",
+ ".ppsx",
+ ".ppsm",
+ ".pot",
+ ".odp",
+ ".xlsx",
+ ".xls",
+ ".xlsm",
+ ".xlsb",
+ ".ods",
+ ".csv",
+ ".epub",
+]
+
+
+class AnyDocParser(BaseParser):
+ """Parse Office and EPUB files through AnyDoc."""
+
+ def __init__(
+ self,
+ config: Optional[ParserConfig] = None,
+ anydoc_config: Optional[AnydocConfig] = None,
+ ):
+ from openviking.parse.parsers.markdown import MarkdownParser
+
+ self.config = config or ParserConfig()
+ self.anydoc_config = anydoc_config or AnydocConfig()
+ self._md_parser = MarkdownParser(config=self.config)
+
+ @property
+ def supported_extensions(self) -> List[str]:
+ return list(_SUPPORTED_EXTENSIONS)
+
+ async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:
+ started = time.time()
+ path = Path(source)
+ if not path.is_file():
+ raise FileNotFoundError(f"Document file not found: {path}")
+
+ if not self.anydoc_config.enabled:
+ raise RuntimeError(
+ "AnyDoc parser is disabled; Office and EPUB legacy parsers have been removed"
+ )
+
+ from openviking.parse.parsers.anydoc_converter import AnyDocConverter
+ from openviking_cli.utils.storage import get_storage
+
+ storage = get_storage()
+ resource_name = kwargs.get("resource_name") or kwargs.get("source_name") or path.stem
+ conversion = await asyncio.to_thread(
+ AnyDocConverter().convert,
+ path,
+ resource_name=resource_name,
+ storage=storage,
+ max_table_rows=self.anydoc_config.max_table_rows,
+ )
+
+ markdown_kwargs = dict(kwargs)
+ allowed_media_dirs = list(markdown_kwargs.get("allowed_media_dirs") or [])
+ if storage.media_dir not in allowed_media_dirs:
+ allowed_media_dirs.append(storage.media_dir)
+ markdown_kwargs.update(
+ source_path=str(path),
+ base_dir=path.parent,
+ allowed_media_dirs=allowed_media_dirs,
+ )
+ result = await self._md_parser.parse_content(
+ conversion.markdown,
+ instruction=instruction,
+ **markdown_kwargs,
+ )
+ result.source_format = conversion.source_format or path.suffix.lstrip(".").lower()
+ result.parser_name = "AnyDocParser"
+ result.parser_version = "1.0"
+ result.parse_time = time.time() - started
+ result.meta.update(
+ {
+ "converter": "firecrawl-anydoc",
+ "intermediate_markdown_length": len(conversion.markdown),
+ "images_extracted": conversion.images_saved,
+ "assets_referenced": getattr(conversion, "assets_referenced", 0),
+ }
+ )
+ result.warnings.extend(getattr(conversion, "warnings", ()))
+ return result
+
+ async def parse_content(
+ self,
+ content: str,
+ source_path: Optional[str] = None,
+ instruction: str = "",
+ **kwargs,
+ ) -> ParseResult:
+ result = await self._md_parser.parse_content(
+ content,
+ source_path=source_path,
+ instruction=instruction,
+ **kwargs,
+ )
+ suffix = Path(source_path).suffix.lstrip(".").lower() if source_path else ""
+ result.source_format = suffix or "anydoc"
+ result.parser_name = "AnyDocParser"
+ result.parser_version = "1.0"
+ return result
diff --git a/openviking/parse/parsers/anydoc_converter.py b/openviking/parse/parsers/anydoc_converter.py
new file mode 100644
index 000000000..0cbb8e0a2
--- /dev/null
+++ b/openviking/parse/parsers/anydoc_converter.py
@@ -0,0 +1,87 @@
+"""Shared anydoc -> Markdown converter."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+from openviking.parse.parsers.anydoc_renderer import _AnyDocMarkdownRenderer, _attr
+
+
+@dataclass(frozen=True)
+class AnydocConversionResult:
+ markdown: str
+ images_saved: int
+ source_format: str
+ assets_referenced: int = 0
+ warnings: tuple[str, ...] = ()
+
+
+def _format_name(value: Any) -> str | None:
+ if value is None:
+ return None
+ name = _attr(value, "value", "name", default=value)
+ return str(name).split(".")[-1].lower()
+
+
+def _load_document(path: Path, format_hint: str | None = None):
+ """Load an anydoc document while keeping the import optional for unit tests."""
+ if path.suffix.lower() == ".pdf" or _format_name(format_hint) == "pdf":
+ raise ValueError("AnyDocConverter does not support PDF; use PDFParser")
+
+ import anydoc
+
+ data = path.read_bytes()
+ detected = anydoc.format_from_bytes(data)
+ detected_name = _format_name(detected)
+ source_format = (
+ _format_name(format_hint)
+ or detected_name
+ or _format_name(anydoc.format_from_extension(path.suffix))
+ )
+ if source_format == "pdf":
+ raise ValueError("AnyDocConverter does not support PDF; use PDFParser")
+
+ try:
+ document = (
+ anydoc.to_document(data, source_format)
+ if source_format
+ else anydoc.to_document(data)
+ )
+ except anydoc.ConvertError as exc:
+ format_context = f" as {source_format}" if source_format else ""
+ raise type(exc)(f"Failed to convert {path}{format_context}: {exc}") from exc
+ return source_format or path.suffix.lstrip(".").lower() or "unknown", document
+
+
+class AnyDocConverter:
+ def convert(
+ self,
+ path: Path,
+ *,
+ resource_name: str,
+ storage: Any,
+ format_hint: str | None = None,
+ max_table_rows: int = 1000,
+ ) -> AnydocConversionResult:
+ path = Path(path)
+ if path.suffix.lower() == ".pdf" or _format_name(format_hint) == "pdf":
+ raise ValueError("AnyDocConverter does not support PDF; use PDFParser")
+
+ source_format, document = _load_document(path, format_hint=format_hint)
+ renderer = _AnyDocMarkdownRenderer(
+ document,
+ source_format=source_format,
+ resource_name=resource_name,
+ storage=storage,
+ max_table_rows=max_table_rows,
+ )
+ markdown = renderer.render()
+ return AnydocConversionResult(
+ markdown=markdown,
+ images_saved=renderer.images_saved,
+ source_format=str(source_format).lower().removeprefix("format."),
+ assets_referenced=len(renderer.assets_referenced),
+ warnings=tuple(renderer.warnings),
+ )
diff --git a/openviking/parse/parsers/anydoc_renderer.py b/openviking/parse/parsers/anydoc_renderer.py
new file mode 100644
index 000000000..0e5017461
--- /dev/null
+++ b/openviking/parse/parsers/anydoc_renderer.py
@@ -0,0 +1,792 @@
+"""Render firecrawl-anydoc document models into Markdown with OV media rewrites."""
+
+from __future__ import annotations
+
+import html
+import mimetypes
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Iterable, Iterator
+
+from openviking.parse.image_validation import is_valid_image
+from openviking_cli.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+_PRESENTATION_FORMATS = {"ppt", "pptx", "pptm", "pps", "ppsx", "ppsm", "pot", "odp"}
+
+
+def _attr(obj: Any, *names: str, default: Any = None) -> Any:
+ for name in names:
+ if obj is None:
+ break
+ if isinstance(obj, dict) and name in obj:
+ return obj[name]
+ if hasattr(obj, name):
+ return getattr(obj, name)
+ return default
+
+
+def _asset_id(source: Any) -> int | None:
+ raw = _attr(source, "asset_id", "assetId")
+ if raw is None:
+ return None
+ if hasattr(raw, "id"):
+ return int(getattr(raw, "id", raw))
+ try:
+ return int(raw)
+ except (TypeError, ValueError):
+ return None
+
+
+@dataclass(frozen=True)
+class _ResolvedAnchor:
+ fragment: str
+ emit_html: bool
+
+
+@dataclass
+class _TextRun:
+ text: str
+ style: Any
+
+
+def _kind(value: Any) -> str:
+ raw = _attr(value, "kind", default="")
+ raw = _attr(raw, "value", "name", default=raw)
+ return str(raw).split(".")[-1].lower()
+
+
+def _items(value: list[Any] | None) -> list[Any]:
+ return value or []
+
+
+class _AnyDocMarkdownRenderer:
+ """Render AnyDoc's public Document model while materializing image assets."""
+
+ def __init__(
+ self,
+ document: Any,
+ *,
+ source_format: str,
+ resource_name: str,
+ storage: Any,
+ max_table_rows: int = 1000,
+ ):
+ self.document = document
+ self.source_format = str(source_format or "").lower().removeprefix("format.")
+ self.resource_name = resource_name
+ self.storage = storage
+ self.max_table_rows = max_table_rows
+ self.images_saved = 0
+ self.asset_paths: dict[int, str | None] = {}
+ self.assets_referenced: set[int] = set()
+ self.warnings: list[str] = []
+ self.assets = self._collect_assets()
+ self.note_numbers = self._number_notes()
+ self.anchors = self._resolve_anchors()
+
+ def _collect_assets(self) -> dict[int, Any]:
+ assets: dict[int, Any] = {}
+ for index, asset in enumerate(_items(self.document.assets)):
+ raw_id = _attr(asset.id, "id", default=asset.id)
+ try:
+ assets[int(raw_id)] = asset
+ except (TypeError, ValueError):
+ assets[index] = asset
+ return assets
+
+ def render(self) -> str:
+ parts = [part for block in _items(self.document.blocks) if (part := self._render_block(block))]
+ parts.extend(self._render_note_definitions())
+ markdown = "\n\n".join(parts)
+ return f"{markdown}\n" if markdown else ""
+
+ def _render_blocks(self, blocks: list[Any] | None) -> str:
+ return "\n\n".join(
+ rendered for block in _items(blocks) if (rendered := self._render_block(block))
+ )
+
+ def _render_block(self, block: Any) -> str:
+ kind = _kind(block)
+ if kind == "heading":
+ text = self._render_inlines(block.content, context="heading").strip()
+ if not text:
+ return ""
+ level = min(max(int(block.level or 1), 1), 6)
+ return f"{'#' * level} {text}"
+ if kind == "paragraph":
+ return self._render_inlines(block.content, context="block").strip()
+ if kind == "list":
+ return self._render_list(block.list)
+ if kind == "table":
+ if block.table.kind == "layout" and self._table_is_single_cell(block.table):
+ cell = block.table.grid[0][0].cell
+ return self._render_blocks(cell.blocks if cell else [])
+ return self._render_table(block.table)
+ if kind == "block_quote":
+ inner = self._render_blocks(block.blocks)
+ if not inner:
+ return ""
+ if self.source_format in _PRESENTATION_FORMATS:
+ return f"### Speaker Notes\n\n{inner}"
+ return "\n".join(">" if not line else f"> {line}" for line in inner.splitlines())
+ if kind == "code_block":
+ body = (block.text or "").rstrip("\n")
+ fence = self._backtick_fence(body, 3)
+ return f"{fence}{block.lang or ''}\n{body}\n{fence}"
+ if kind == "rule":
+ return "---"
+ raise RuntimeError(f"Unsupported AnyDoc block kind: {kind}")
+
+ def _render_inlines(
+ self,
+ inlines: list[Any] | None,
+ *,
+ context: str,
+ in_label: bool = False,
+ ) -> str:
+ normalized = self._normalize_inlines(_items(inlines))
+ parts: list[str] = []
+ for index, inline in enumerate(normalized):
+ if isinstance(inline, _TextRun):
+ next_inline = normalized[index + 1] if index + 1 < len(normalized) else None
+ trailing_active = self._next_inline_is_active(next_inline)
+ rendered_so_far = "".join(parts)
+ parts.append(
+ self._render_text(
+ inline.text,
+ inline.style,
+ context=context,
+ in_label=in_label,
+ trailing_active=trailing_active,
+ at_line_start=not rendered_so_far or rendered_so_far.endswith("\n"),
+ )
+ )
+ continue
+ parts.append(self._render_inline(inline, context=context, in_label=in_label))
+ return "".join(parts)
+
+ def _render_inline(self, inline: Any, *, context: str, in_label: bool) -> str:
+ kind = _kind(inline)
+ if kind == "link":
+ return self._render_link(inline, context=context)
+ if kind == "image":
+ return self._render_image(inline, context=context, in_label=in_label)
+ if kind == "anchor":
+ resolved = self.anchors.get(inline.anchor)
+ return f'<a id="{html.escape(resolved.fragment, quote=True)}"></a>' if resolved and resolved.emit_html else ""
+ if kind == "note_ref":
+ number = self.note_numbers.get(inline.note_id or "")
+ return f"[^{number}]" if number is not None else ""
+ if kind == "line_break":
+ return "\\\n" if context == "block" else "\n" if context == "table_cell" else " "
+ raise RuntimeError(f"Unsupported AnyDoc inline kind: {kind}")
+
+ def _normalize_inlines(self, inlines: Iterable[Any]) -> list[Any]:
+ normalized: list[Any] = []
+ plain_style = self._style_key(None)
+ for inline in inlines:
+ kind = _kind(inline)
+ if kind == "anchor":
+ resolved = self.anchors.get(inline.anchor)
+ if resolved is None or not resolved.emit_html:
+ continue
+ if kind != "text":
+ normalized.append(inline)
+ continue
+ if not inline.text:
+ continue
+
+ style = None if inline.text.isspace() else inline.style
+ style_key = self._style_key(style)
+ previous = normalized[-1] if normalized else None
+ if isinstance(previous, _TextRun) and self._style_key(previous.style) == style_key:
+ previous.text += inline.text
+ continue
+ if (
+ style_key != plain_style
+ and not style_key[3]
+ and len(normalized) >= 2
+ and isinstance(normalized[-1], _TextRun)
+ and normalized[-1].text.isspace()
+ and self._style_key(normalized[-1].style) == plain_style
+ and isinstance(normalized[-2], _TextRun)
+ and self._style_key(normalized[-2].style) == style_key
+ ):
+ whitespace = normalized.pop().text
+ normalized[-1].text += whitespace + inline.text
+ continue
+ normalized.append(_TextRun(text=inline.text, style=style))
+ return normalized
+
+ def _render_text(
+ self,
+ text: str,
+ style: Any,
+ *,
+ context: str,
+ in_label: bool,
+ trailing_active: bool,
+ at_line_start: bool,
+ ) -> str:
+ bold, italic, strike, code, underline = self._style_key(style)
+ if not any((bold, italic, strike, code, underline)):
+ return self._escape_text(
+ text,
+ context=context,
+ in_label=in_label,
+ trailing_active=trailing_active,
+ at_line_start=at_line_start,
+ )
+
+ leading = text[: len(text) - len(text.lstrip())]
+ trailing = text[len(text.rstrip()) :]
+ core_end = len(text) - len(trailing) if trailing else len(text)
+ core = text[len(leading) : core_end]
+ if not core:
+ return text
+ if code:
+ flattened = core.replace("\n", " ")
+ fence = self._backtick_fence(flattened, 1)
+ pad = " " if flattened.startswith("`") or flattened.endswith("`") else ""
+ rendered = f"{fence}{pad}{flattened}{pad}{fence}"
+ else:
+ opening = ""
+ closing = ""
+ if strike:
+ opening += "~~"
+ closing = "~~" + closing
+ if bold:
+ opening += "**"
+ closing = "**" + closing
+ if italic:
+ opening += "*"
+ closing = "*" + closing
+ escaped = self._escape_text(core, context=context, styled=True, in_label=in_label)
+ rendered = f"{opening}{escaped}{closing}"
+ if underline:
+ rendered = f"<ins>{rendered}</ins>"
+ return f"{leading}{rendered}{trailing}"
+
+ def _render_link(self, inline: Any, *, context: str) -> str:
+ target = inline.target
+ label = self._render_inlines(inline.content, context=context, in_label=True)
+ if not target or not target.value:
+ return label
+ if target.kind == "anchor":
+ resolved = self.anchors.get(target.value)
+ if resolved is None:
+ return label
+ url = f"#{resolved.fragment}"
+ elif target.kind in {"external", "relative"}:
+ url = target.value
+ else:
+ raise RuntimeError(f"Unsupported AnyDoc link target kind: {target.kind}")
+ if label.strip():
+ return f"[{label}]({self._format_url(url)})"
+ escaped = self._escape_text(url, context=context, in_label=True, trailing_active=True)
+ return f"[{escaped}]({self._format_url(url)})"
+
+ def _render_image(self, image: Any, *, context: str, in_label: bool) -> str:
+ alt = (image.alt or "").strip()
+ escaped_alt = self._escape_text(alt, context=context, in_label=True)
+ source = image.source
+ if source is None:
+ return self._escape_text(alt, context=context, in_label=in_label)
+ if source.kind == "external":
+ return f"![{escaped_alt}]({self._format_url(source.url or '')})" if source.url else escaped_alt
+ if source.kind == "unavailable":
+ self.warnings.append(f"Embedded image is unavailable: {alt or '<no alt text>'}")
+ return self._escape_text(alt, context=context, in_label=in_label)
+ if source.kind != "asset":
+ return self._escape_text(alt, context=context, in_label=in_label)
+
+ reference = _asset_id(source)
+ if reference is None:
+ self.warnings.append("AnyDoc image references an asset without an id")
+ return self._escape_text(alt, context=context, in_label=in_label)
+ self.assets_referenced.add(reference)
+ image_ref = self._materialize_asset(reference)
+ if image_ref is None:
+ return self._escape_text(alt, context=context, in_label=in_label)
+ return f"![{escaped_alt or f'anydoc_asset_{reference}'}]({self._format_url(image_ref)})"
+
+ def _materialize_asset(self, reference: int) -> str | None:
+ if reference in self.asset_paths:
+ return self.asset_paths[reference]
+ asset = self.assets.get(reference)
+ if asset is None:
+ self.warnings.append(f"AnyDoc image references missing asset {reference}")
+ self.asset_paths[reference] = None
+ return None
+ if not asset.media_type.lower().startswith("image/"):
+ self.warnings.append(
+ f"AnyDoc asset {reference} is not an image ({asset.media_type}); kept as alt text"
+ )
+ self.asset_paths[reference] = None
+ return None
+
+ image_bytes = bytes(asset.data)
+ extension = self._image_extension(asset)
+ filename = f"anydoc_asset_{reference}"
+ if not is_valid_image(image_bytes, Path(f"{filename}{extension}")):
+ self.warnings.append(
+ f"AnyDoc asset {reference} is not an ingestable image ({asset.media_type})"
+ )
+ self.asset_paths[reference] = None
+ return None
+
+ try:
+ saved_path = self.storage.save_image(
+ self.resource_name,
+ image_bytes,
+ filename=filename,
+ extension=extension,
+ )
+ relative = Path(saved_path).relative_to(Path(self.storage.media_dir)).as_posix()
+ except Exception as exc:
+ self.warnings.append(f"Failed to save AnyDoc image asset {reference}: {exc}")
+ logger.warning(
+ "Failed to save anydoc image asset %s from %s",
+ reference,
+ self.resource_name,
+ exc_info=True,
+ )
+ self.asset_paths[reference] = None
+ return None
+
+ self.images_saved += 1
+ self.asset_paths[reference] = relative
+ return relative
+
+ def _render_list(self, list_model: Any) -> str:
+ if list_model is None or not list_model.items:
+ return ""
+ rendered_items: list[str] = []
+ loose = False
+ for index, item in enumerate(list_model.items):
+ ordinal = int(list_model.start) + index
+ marker = self._list_marker(list_model, item, ordinal)
+ checkbox = "[x] " if item.checked is True else "[ ] " if item.checked is False else ""
+ body = self._render_blocks(item.blocks)
+ if len(item.blocks) > 1:
+ loose = True
+ lines = body.splitlines() or [""]
+ indent = " " * len(marker)
+ rendered = f"{marker}{checkbox}{lines[0]}"
+ for line in lines[1:]:
+ if not line:
+ loose = True
+ rendered += "\n"
+ else:
+ rendered += f"\n{indent}{line}"
+ rendered_items.append(rendered)
+ return ("\n\n" if loose else "\n").join(rendered_items)
+
+ def _render_table(self, table: Any) -> str:
+ if table is None or not table.grid:
+ return ""
+ if table.kind == "layout" and self._table_is_single_cell(table):
+ cell = table.grid[0][0].cell
+ return self._render_blocks(cell.blocks if cell else [])
+
+ grid = _items(table.grid)
+ truncated_rows = 0
+ if self.max_table_rows > 0 and len(grid) > self.max_table_rows:
+ truncated_rows = len(grid) - self.max_table_rows
+ grid = grid[: self.max_table_rows]
+
+ rows = self._render_table_rows(grid)
+ while len(rows) > 1 and all(not text and not covered for text, covered in rows[-1]):
+ rows.pop()
+ width = self._table_width(rows)
+ if width == 0:
+ return ""
+ rows = [row[:width] for row in rows]
+ header = [text for text, _ in rows.pop(0)] if table.header_rows >= 1 else [""] * width
+ lines = [self._format_table_row(header), self._format_table_row(["---"] * width)]
+ lines.extend(self._format_table_row([text for text, _ in row]) for row in rows)
+ if truncated_rows:
+ lines.append(f"\n*... {truncated_rows} more rows truncated ...*")
+ return "\n".join(lines)
+
+ def _render_table_rows(self, grid: list[Any]) -> list[list[tuple[str, bool]]]:
+ width = max((len(row) for row in grid), default=0)
+ rows: list[list[tuple[str, bool]]] = []
+ for row in grid:
+ rendered = [
+ (self._render_cell(slot.cell), False)
+ if slot.kind == "origin"
+ else ("", True)
+ for slot in row
+ ]
+ rendered.extend(("", False) for _ in range(width - len(rendered)))
+ rows.append(rendered)
+ return rows
+
+ def _render_cell(self, cell: Any | None) -> str:
+ if cell is None:
+ return ""
+ parts: list[str] = []
+ for block in cell.blocks:
+ kind = _kind(block)
+ if kind == "heading":
+ text = self._render_inlines(block.content, context="table_cell").strip()
+ if text:
+ parts.append(f"**{text}**")
+ elif kind == "paragraph":
+ text = self._render_inlines(block.content, context="table_cell")
+ if text.strip():
+ parts.append(text)
+ elif kind == "list":
+ text = self._render_list(block.list).replace("\n", " ").strip()
+ if text:
+ parts.append(text)
+ elif kind == "table":
+ parts.extend(self._flatten_nested_table(block.table))
+ elif kind == "block_quote":
+ text = self._render_blocks(block.blocks).replace("\n", " ").strip()
+ if text:
+ parts.append(text)
+ elif kind == "code_block":
+ text = (block.text or "").strip()
+ if text:
+ fence = self._backtick_fence(text, 1)
+ pad = " " if text.startswith("`") or text.endswith("`") else ""
+ parts.append(f"{fence}{pad}{text}{pad}{fence}")
+ elif kind != "rule":
+ raise RuntimeError(f"Unsupported AnyDoc table-cell block kind: {kind}")
+ return "<br>".join(line.strip() for line in "<br>".join(parts).splitlines() if line.strip())
+
+ def _flatten_nested_table(self, table: Any) -> list[str]:
+ lines: list[str] = []
+ for row in table.grid:
+ values = [
+ self._render_cell(slot.cell) if slot.kind == "origin" else "" for slot in row
+ ]
+ if any(values):
+ lines.append(" / ".join(values))
+ return lines
+
+ def _render_note_definitions(self) -> list[str]:
+ rendered: list[str] = []
+ notes_by_id = {note.id: note for note in _items(self.document.notes)}
+ for note_id, number in sorted(self.note_numbers.items(), key=lambda item: item[1]):
+ note = notes_by_id.get(note_id)
+ if note is None:
+ continue
+ body = self._render_blocks(note.blocks)
+ if not body:
+ continue
+ lines = body.splitlines()
+ definition = f"[^{number}]: {lines[0]}"
+ if len(lines) > 1:
+ definition += "\n" + "\n".join(f" {line}" if line else "" for line in lines[1:])
+ rendered.append(definition)
+ return rendered
+
+ def _number_notes(self) -> dict[str, int]:
+ notes = {note.id: note for note in _items(self.document.notes) if note.blocks}
+ order: list[str] = []
+ seen: set[str] = set()
+
+ def add(note_id: str | None) -> None:
+ if note_id and note_id in notes and note_id not in seen:
+ seen.add(note_id)
+ order.append(note_id)
+ for inline in self._walk_inlines(notes[note_id].blocks):
+ if _kind(inline) == "note_ref":
+ add(inline.note_id)
+
+ for inline in self._walk_inlines(self.document.blocks):
+ if _kind(inline) == "note_ref":
+ add(inline.note_id)
+ for note_id in notes:
+ add(note_id)
+ return {note_id: index for index, note_id in enumerate(order, 1)}
+
+ def _resolve_anchors(self) -> dict[str, _ResolvedAnchor]:
+ linked: set[str] = set()
+ headings: list[Any] = []
+ anchors: list[str] = []
+
+ for block in self._walk_blocks(self.document.blocks):
+ if _kind(block) == "heading":
+ headings.append(block)
+ for inline in self._block_inlines(block):
+ kind = _kind(inline)
+ if kind == "link" and inline.target and inline.target.kind == "anchor":
+ linked.add(inline.target.value)
+ elif kind == "anchor" and inline.anchor:
+ anchors.append(inline.anchor)
+ for note in _items(self.document.notes):
+ for inline in self._walk_inlines(note.blocks):
+ if _kind(inline) == "anchor" and inline.anchor:
+ anchors.append(inline.anchor)
+
+ resolved: dict[str, _ResolvedAnchor] = {}
+ used: set[str] = set()
+ for heading in headings:
+ slug = self._claim_anchor(self._gfm_slug(self._plain_text(heading.content)), used)
+ heading_ids = [heading.anchor] if heading.anchor else []
+ heading_ids.extend(self._inline_anchor_ids(heading.content))
+ for anchor_id in heading_ids:
+ resolved.setdefault(anchor_id, _ResolvedAnchor(slug, False))
+ for anchor_id in anchors:
+ if anchor_id in linked and anchor_id not in resolved:
+ fragment = self._claim_anchor(self._sanitize_anchor(anchor_id), used)
+ resolved[anchor_id] = _ResolvedAnchor(fragment, True)
+ return resolved
+
+ def _walk_blocks(self, blocks: list[Any] | None) -> Iterator[Any]:
+ for block in _items(blocks):
+ yield block
+ kind = _kind(block)
+ if kind == "list" and block.list:
+ for item in block.list.items:
+ yield from self._walk_blocks(item.blocks)
+ elif kind == "table" and block.table:
+ for row in block.table.grid:
+ for slot in row:
+ if slot.kind == "origin" and slot.cell:
+ yield from self._walk_blocks(slot.cell.blocks)
+ elif kind == "block_quote":
+ yield from self._walk_blocks(block.blocks)
+
+ def _walk_inlines(self, blocks: list[Any] | None) -> Iterator[Any]:
+ for block in self._walk_blocks(blocks):
+ yield from self._block_inlines(block)
+
+ def _block_inlines(self, block: Any) -> Iterator[Any]:
+ if _kind(block) not in {"heading", "paragraph"}:
+ return
+ yield from self._inline_tree(block.content)
+
+ def _inline_tree(self, inlines: list[Any] | None) -> Iterator[Any]:
+ for inline in _items(inlines):
+ yield inline
+ if _kind(inline) == "link":
+ yield from self._inline_tree(inline.content)
+
+ def _plain_text(self, inlines: list[Any] | None) -> str:
+ parts: list[str] = []
+ for inline in _items(inlines):
+ kind = _kind(inline)
+ if kind == "text":
+ parts.append(inline.text or "")
+ elif kind == "link":
+ parts.append(self._plain_text(inline.content))
+ elif kind == "image":
+ parts.append(inline.alt or "")
+ elif kind == "line_break":
+ parts.append(" ")
+ return "".join(parts)
+
+ def _inline_anchor_ids(self, inlines: list[Any] | None) -> list[str]:
+ ids: list[str] = []
+ for inline in _items(inlines):
+ kind = _kind(inline)
+ if kind == "anchor" and inline.anchor:
+ ids.append(inline.anchor)
+ elif kind == "link":
+ ids.extend(self._inline_anchor_ids(inline.content))
+ return ids
+
+ @staticmethod
+ def _next_inline_is_active(inline: Any) -> bool:
+ if inline is None:
+ return False
+ return not isinstance(inline, _TextRun) or _AnyDocMarkdownRenderer._style_key(inline.style) != (
+ False,
+ False,
+ False,
+ False,
+ False,
+ )
+
+ @staticmethod
+ def _table_is_single_cell(table: Any) -> bool:
+ return len(table.grid) == 1 and len(table.grid[0]) == 1 and table.grid[0][0].kind == "origin"
+
+ @staticmethod
+ def _table_width(rows: list[list[tuple[str, bool]]]) -> int:
+ return max(
+ (
+ max((index + 1 for index, cell in enumerate(row) if cell[0] or cell[1]), default=0)
+ for row in rows
+ ),
+ default=0,
+ )
+
+ @staticmethod
+ def _format_table_row(cells: Iterable[str]) -> str:
+ return "|" + "".join(f" {cell} |" for cell in cells)
+
+ @staticmethod
+ def _style_key(style: Any) -> tuple[bool, bool, bool, bool, bool]:
+ if style is None:
+ return False, False, False, False, False
+ return (
+ bool(_attr(style, "bold", default=False)),
+ bool(_attr(style, "italic", default=False)),
+ bool(_attr(style, "strike", default=False)),
+ bool(_attr(style, "code", default=False)),
+ bool(_attr(style, "underline", "underlined", default=False)),
+ )
+
+ @staticmethod
+ def _list_marker(list_model: Any, item: Any, ordinal: int) -> str:
+ if item.marker_label:
+ return f"- {item.marker_label} "
+ if list_model.marker == "bullet":
+ return "- "
+ if list_model.marker == "decimal":
+ return f"{ordinal}. "
+ return f"- {_AnyDocMarkdownRenderer._marker_label(list_model.marker, ordinal)} "
+
+ @staticmethod
+ def _claim_anchor(base: str, used: set[str]) -> str:
+ if base not in used:
+ used.add(base)
+ return base
+ suffix = 1
+ while f"{base}-{suffix}" in used:
+ suffix += 1
+ claimed = f"{base}-{suffix}"
+ used.add(claimed)
+ return claimed
+
+ @staticmethod
+ def _gfm_slug(text: str) -> str:
+ slug = "".join(
+ "-" if char == " " else char.lower()
+ for char in text.strip()
+ if char == " " or char == "-" or char == "_" or char.isalnum()
+ )
+ return slug or "section"
+
+ @staticmethod
+ def _sanitize_anchor(anchor: str) -> str:
+ sanitized = re.sub(r"[^a-z0-9_-]+", "-", anchor.lower()).strip("-")
+ return sanitized or "anchor"
+
+ @staticmethod
+ def _marker_label(marker: str, number: int) -> str:
+ if marker in {"lower_alpha", "upper_alpha"}:
+ alpha = ""
+ remaining = max(number, 1)
+ while remaining:
+ remaining, offset = divmod(remaining - 1, 26)
+ alpha = chr(ord("a") + offset) + alpha
+ return f"{alpha.upper() if marker == 'upper_alpha' else alpha}."
+ if marker in {"lower_roman", "upper_roman"}:
+ remaining = max(number, 1)
+ result = ""
+ for unit, symbol in (
+ (1000, "M"),
+ (900, "CM"),
+ (500, "D"),
+ (400, "CD"),
+ (100, "C"),
+ (90, "XC"),
+ (50, "L"),
+ (40, "XL"),
+ (10, "X"),
+ (9, "IX"),
+ (5, "V"),
+ (4, "IV"),
+ (1, "I"),
+ ):
+ while remaining >= unit:
+ result += symbol
+ remaining -= unit
+ return f"{result.lower() if marker == 'lower_roman' else result}."
+ raise RuntimeError(f"Unsupported AnyDoc list marker: {marker}")
+
+ @staticmethod
+ def _format_url(url: str) -> str:
+ escaped = "".join(
+ "%7C" if char == "|" else "%3C" if char == "<" else "%3E" if char == ">" else char
+ for char in url
+ if ord(char) >= 32 and ord(char) != 127
+ )
+ return (
+ f"<{escaped}>" if any(char.isspace() or char in "()" for char in escaped) else escaped
+ )
+
+ @staticmethod
+ def _backtick_fence(text: str, minimum: int) -> str:
+ longest = max((len(run) for run in re.findall(r"`+", text)), default=0)
+ return "`" * max(longest + 1, minimum)
+
+ @staticmethod
+ def _escape_text(
+ text: str,
+ *,
+ context: str,
+ styled: bool = False,
+ trailing_active: bool = False,
+ in_label: bool = False,
+ at_line_start: bool = False,
+ ) -> str:
+ characters = list(text)
+ output: list[str] = []
+ line_has_content = not at_line_start
+ for index, char in enumerate(characters):
+ if char == "\n":
+ output.append(char)
+ if context == "block":
+ line_has_content = False
+ continue
+ start_of_line = not line_has_content
+ if not char.isspace():
+ line_has_content = True
+ next_char = characters[index + 1] if index + 1 < len(characters) else None
+ next_nonspace = trailing_active if next_char is None else not next_char.isspace()
+ later = characters[index + 1 :]
+ escape = False
+ if char == "\\":
+ escape = True
+ elif char == "]" and in_label:
+ escape = True
+ elif char == "`":
+ escape = styled or "`" in later
+ elif char == "*":
+ escape = styled or start_of_line or (next_nonspace and "*" in later)
+ elif char == "_":
+ previous_alnum = index > 0 and characters[index - 1].isalnum()
+ next_alnum = bool(next_char and next_char.isalnum())
+ escape = styled or (
+ next_nonspace and not (previous_alnum and next_alnum) and "_" in later
+ )
+ elif char == "~":
+ escape = styled or (next_nonspace and "~" in later)
+ elif char == "[":
+ escape = in_label or "]" in later
+ elif char == "<":
+ escape = bool(next_char and (next_char.isalpha() or next_char in "/!?"))
+ elif char == "!":
+ escape = next_char is None and trailing_active
+ elif char == "|" and context == "table_cell":
+ escape = True
+ elif char in "#>" and start_of_line:
+ escape = True
+ elif char in "-+" and start_of_line:
+ escape = next_char is None or next_char.isspace()
+ if escape:
+ output.append("\\")
+ output.append(char)
+ return "".join(output)
+
+ @staticmethod
+ def _image_extension(asset: Any) -> str:
+ extension = mimetypes.guess_extension(asset.media_type.split(";", 1)[0].strip())
+ if extension == ".jpe":
+ extension = ".jpg"
+ if not extension:
+ extension = Path(asset.origin_part or "").suffix.lower()
+ if not extension or not re.fullmatch(r"\.[a-z0-9]{1,10}", extension):
+ extension = ".png"
+ return extension if extension.startswith(".") else f".{extension}"
diff --git a/openviking/parse/parsers/epub.py b/openviking/parse/parsers/epub.py
deleted file mode 100644
index 6ecc76bbc..000000000
--- a/openviking/parse/parsers/epub.py
+++ /dev/null
@@ -1,215 +0,0 @@
-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
-# SPDX-License-Identifier: AGPL-3.0
-"""
-EPub (.epub) parser for OpenViking.
-
-Converts EPub e-books to Markdown then parses using MarkdownParser.
-Inspired by microsoft/markitdown approach.
-"""
-
-import asyncio
-import html
-import re
-import zipfile
-from html.parser import HTMLParser
-from pathlib import Path
-from typing import List, Optional, Union
-
-from openviking.parse.base import ParseResult
-from openviking.parse.parsers.base_parser import BaseParser
-from openviking_cli.utils.config.parser_config import ParserConfig
-from openviking_cli.utils.logger import get_logger
-
-logger = get_logger(__name__)
-
-
-class _EPubMarkdownParser(HTMLParser):
- """Convert HTML fragments to simple markdown without regex tag stripping."""
-
- _HEADER_PREFIX = {"h1": "# ", "h2": "## ", "h3": "### ", "h4": "#### "}
-
- def __init__(self):
- super().__init__(convert_charrefs=True)
- self._parts: list[str] = []
- self._ignored_tag_stack = 0
-
- def handle_starttag(self, tag: str, attrs):
- normalized_tag = tag.lower()
- if normalized_tag in {"script", "style"}:
- self._ignored_tag_stack += 1
- return
- if self._ignored_tag_stack:
- return
- if normalized_tag in self._HEADER_PREFIX:
- self._ensure_block_break()
- self._parts.append(self._HEADER_PREFIX[normalized_tag])
- elif normalized_tag in {"strong", "b"}:
- self._parts.append("**")
- elif normalized_tag in {"em", "i"}:
- self._parts.append("*")
- elif normalized_tag == "li":
- self._parts.append("\n- ")
- elif normalized_tag == "br":
- self._parts.append("\n")
- elif normalized_tag in {"p", "div", "section", "article"}:
- self._ensure_block_break()
-
- def handle_endtag(self, tag: str):
- normalized_tag = tag.lower()
- if normalized_tag in {"script", "style"}:
- if self._ignored_tag_stack:
- self._ignored_tag_stack -= 1
- return
- if self._ignored_tag_stack:
- return
- if normalized_tag in self._HEADER_PREFIX or normalized_tag in {
- "p",
- "div",
- "section",
- "article",
- }:
- self._parts.append("\n\n")
- elif normalized_tag in {"strong", "b"}:
- self._parts.append("**")
- elif normalized_tag in {"em", "i"}:
- self._parts.append("*")
-
- def handle_data(self, data: str):
- if not self._ignored_tag_stack:
- self._parts.append(data)
-
- def get_markdown(self) -> str:
- return "".join(self._parts)
-
- def _ensure_block_break(self):
- if self._parts and not self._parts[-1].endswith("\n"):
- self._parts.append("\n\n")
-
-
-class EPubParser(BaseParser):
- """
- EPub e-book parser for OpenViking.
-
- Supports: .epub
-
- Converts EPub e-books to Markdown using ebooklib (if available)
- or falls back to manual extraction, then delegates to MarkdownParser.
- """
-
- def __init__(self, config: Optional[ParserConfig] = None):
- """Initialize EPub parser."""
- from openviking.parse.parsers.markdown import MarkdownParser
-
- self._md_parser = MarkdownParser(config=config)
- self.config = config or ParserConfig()
-
- @property
- def supported_extensions(self) -> List[str]:
- return [".epub"]
-
- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:
- """Parse EPub e-book from file path."""
- path = Path(source)
-
- if path.exists():
- markdown_content = await asyncio.to_thread(self._convert_to_markdown, path)
- result = await self._md_parser.parse_content(
- markdown_content, source_path=str(path), instruction=instruction, **kwargs
- )
- else:
- result = await self._md_parser.parse_content(
- str(source), instruction=instruction, **kwargs
- )
- result.source_format = "epub"
- result.parser_name = "EPubParser"
- return result
-
- async def parse_content(
- self, content: str, source_path: Optional[str] = None, instruction: str = "", **kwargs
- ) -> ParseResult:
- """Parse content - delegates to MarkdownParser."""
- result = await self._md_parser.parse_content(content, source_path, **kwargs)
- result.source_format = "epub"
- result.parser_name = "EPubParser"
- return result
-
- def _convert_to_markdown(self, path: Path) -> str:
- """Convert EPub e-book to Markdown string."""
- # Try using ebooklib first
- try:
- import ebooklib
- from ebooklib import epub
-
- return self._convert_with_ebooklib(path, ebooklib, epub)
- except ImportError:
- pass
-
- # Fall back to manual extraction
- return self._convert_manual(path)
-
- def _convert_with_ebooklib(self, path: Path, ebooklib, epub) -> str:
- """Convert EPub using ebooklib."""
- book = epub.read_epub(path)
- markdown_parts = []
-
- title = self._get_metadata(book, "title")
- author = self._get_metadata(book, "creator")
-
- if title:
- markdown_parts.append(f"# {title}")
- if author:
- markdown_parts.append(f"**Author:** {author}")
-
- for item in book.get_items():
- if item.get_type() == ebooklib.ITEM_DOCUMENT:
- content = item.get_content().decode("utf-8", errors="ignore")
- md_content = self._html_to_markdown(content)
- if md_content.strip():
- markdown_parts.append(md_content)
-
- return "\n\n".join(markdown_parts)
-
- def _get_metadata(self, book, key: str) -> str:
- """Get metadata from EPub book."""
- try:
- metadata = book.get_metadata("DC", key)
- if metadata:
- return metadata[0][0]
- except Exception:
- pass
- return ""
-
- def _convert_manual(self, path: Path) -> str:
- """Convert EPub manually using zipfile and HTML parsing."""
- markdown_parts = []
-
- with zipfile.ZipFile(path, "r") as zf:
- html_files = [f for f in zf.namelist() if f.endswith((".html", ".xhtml", ".htm"))]
-
- for html_file in sorted(html_files):
- try:
- content = zf.read(html_file).decode("utf-8", errors="ignore")
- md_content = self._html_to_markdown(content)
- if md_content.strip():
- markdown_parts.append(md_content)
- except Exception as e:
- logger.warning(f"Failed to process {html_file}: {e}")
-
- return (
- "\n\n".join(markdown_parts)
- if markdown_parts
- else "# EPub Content\n\nUnable to extract content."
- )
-
- def _html_to_markdown(self, html_content: str) -> str:
- """Simple HTML to markdown conversion."""
- parser = _EPubMarkdownParser()
- parser.feed(html_content)
- parser.close()
- markdown = html.unescape(parser.get_markdown())
-
- # Normalize whitespace
- markdown = re.sub(r"\n\s*\n", "\n\n", markdown)
- markdown = re.sub(r"[ \t]+", " ", markdown)
-
- return markdown.strip()
diff --git a/openviking/parse/parsers/excel.py b/openviking/parse/parsers/excel.py
deleted file mode 100644
index d8c610c77..000000000
--- a/openviking/parse/parsers/excel.py
+++ /dev/null
@@ -1,440 +0,0 @@
-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
-# SPDX-License-Identifier: AGPL-3.0
-"""
-Excel (.xlsx/.xls/.xlsm) parser for OpenViking.
-
-Converts Excel spreadsheets to Markdown then parses using MarkdownParser.
-Inspired by microsoft/markitdown approach.
-"""
-
-import asyncio
-import concurrent.futures
-import time
-from dataclasses import asdict, fields
-from functools import partial
-from multiprocessing import get_context
-from pathlib import Path
-from typing import Any, Dict, List, Optional, Union
-
-from openviking.parse.base import NodeType, ParseResult, ResourceNode, create_parse_result
-from openviking.parse.parsers.base_parser import BaseParser
-from openviking_cli.utils.config.parser_config import ExcelConfig, ParserConfig
-from openviking_cli.utils.logger import get_logger
-
-logger = get_logger(__name__)
-
-_EXCEL_LAYOUT_EXECUTOR: Optional[concurrent.futures.ProcessPoolExecutor] = None
-_EXCEL_LAYOUT_EXECUTOR_WORKERS: Optional[int] = None
-
-# Non-configurable process-pool internals.
-_EXCEL_PROCESS_POOL_START_METHOD = "spawn"
-_EXCEL_PROCESS_POOL_MIN_BYTES = 200_000
-_EXCEL_PROCESS_POOL_TIMEOUT_S = 120.0
-
-
-def _get_excel_layout_executor(workers: int) -> concurrent.futures.ProcessPoolExecutor:
- global _EXCEL_LAYOUT_EXECUTOR, _EXCEL_LAYOUT_EXECUTOR_WORKERS
- workers = max(1, int(workers))
- if _EXCEL_LAYOUT_EXECUTOR is None:
- _EXCEL_LAYOUT_EXECUTOR = concurrent.futures.ProcessPoolExecutor(
- max_workers=workers,
- mp_context=get_context(_EXCEL_PROCESS_POOL_START_METHOD),
- )
- _EXCEL_LAYOUT_EXECUTOR_WORKERS = workers
- logger.info(
- f"[ExcelParserProcessPool] Started process pool "
- f"workers={workers} start_method={_EXCEL_PROCESS_POOL_START_METHOD}"
- )
- elif _EXCEL_LAYOUT_EXECUTOR_WORKERS != workers:
- logger.warning(
- "[ExcelParserProcessPool] Ignoring process_pool_workers=%s; "
- "pool already started with workers=%s",
- workers,
- _EXCEL_LAYOUT_EXECUTOR_WORKERS,
- )
- return _EXCEL_LAYOUT_EXECUTOR
-
-
-def _build_excel_layout_in_process(
- *,
- path_str: str,
- temp_uri: str,
- instruction: str,
- layout_kwargs: Dict[str, Any],
- config_dict: Dict[str, Any],
- max_rows_per_sheet: int,
-) -> Dict[str, Any]:
- """CPU-only child process worker.
-
- The worker must not touch VikingFS, DB, queues, or RequestContext. It only
- converts Excel to markdown and computes MarkdownParser layout ops.
- """
- import asyncio
-
- from openviking.parse.parsers.excel import ExcelParser
- from openviking_cli.utils.config.parser_config import ParserConfig
-
- started = time.perf_counter()
- path = Path(path_str)
- allowed_config_fields = {field.name for field in fields(ParserConfig)}
- filtered_config = {
- key: value for key, value in config_dict.items() if key in allowed_config_fields
- }
- parser = ExcelParser(
- config=ParserConfig.from_dict(filtered_config),
- max_rows_per_sheet=max_rows_per_sheet,
- )
-
- convert_started = time.perf_counter()
- if path.suffix.lower() == ".xls":
- markdown_content = parser._convert_xls_to_markdown(path)
- else:
- import openpyxl
-
- markdown_content = parser._convert_to_markdown(path, openpyxl)
- convert_s = time.perf_counter() - convert_started
-
- layout_started = time.perf_counter()
- layout = asyncio.run(
- parser._md_parser._compute_layout(
- markdown_content,
- temp_uri,
- source_path=str(path),
- instruction=instruction,
- **layout_kwargs,
- )
- )
- layout_s = time.perf_counter() - layout_started
-
- return {
- "layout": layout,
- "convert_s": convert_s,
- "layout_s": layout_s,
- "total_s": time.perf_counter() - started,
- "markdown_chars": len(markdown_content),
- "layout_ops": len(layout.ops),
- "layout_write_ops": sum(1 for op in layout.ops if op.kind == "write"),
- "layout_mkdir_ops": sum(1 for op in layout.ops if op.kind == "mkdir"),
- }
-
-
-class ExcelParser(BaseParser):
- """
- Excel spreadsheet parser for OpenViking.
-
- Supports: .xlsx, .xls, .xlsm
-
- Converts Excel spreadsheets to Markdown using openpyxl,
- then delegates to MarkdownParser for tree structure creation.
- """
-
- def __init__(
- self, config: Optional[ParserConfig] = None, max_rows_per_sheet: int = 1000
- ):
- """
- Initialize Excel parser.
-
- Args:
- config: Parser configuration (prefer ExcelConfig for process-pool knobs)
- max_rows_per_sheet: Maximum rows to process per sheet (0 = unlimited)
- """
- from openviking.parse.parsers.markdown import MarkdownParser
-
- self._md_parser = MarkdownParser(config=config)
- self.config = config or ExcelConfig()
- self.max_rows_per_sheet = max_rows_per_sheet
-
- def _process_pool_enabled(self) -> bool:
- return bool(getattr(self.config, "enable_process_pool", False))
-
- def _process_pool_workers(self) -> int:
- return max(1, int(getattr(self.config, "process_pool_workers", 2) or 2))
-
- @property
- def supported_extensions(self) -> List[str]:
- return [".xlsx", ".xls", ".xlsm"]
-
- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:
- """Parse Excel spreadsheet from file path."""
- path = Path(source)
-
- if path.exists():
- result = await self._parse_existing_path(path, instruction=instruction, **kwargs)
- else:
- result = await self._md_parser.parse_content(
- str(source), instruction=instruction, **kwargs
- )
- result.source_format = path.suffix.lstrip(".") if path.exists() else "xlsx"
- result.parser_name = "ExcelParser"
- return result
-
- async def _parse_existing_path(
- self, path: Path, instruction: str = "", **kwargs
- ) -> ParseResult:
- if self._should_use_process_pool(path, kwargs):
- try:
- return await self._parse_existing_path_process_pool(
- path, instruction=instruction, **kwargs
- )
- except Exception as exc:
- logger.warning(
- f"[ExcelParserProcessPool] Falling back to in-process parse: {exc}",
- exc_info=True,
- )
-
- # Use xlrd for legacy .xls, openpyxl for .xlsx/.xlsm
- if path.suffix.lower() == ".xls":
- markdown_content = await asyncio.to_thread(self._convert_xls_to_markdown, path)
- else:
- import openpyxl
-
- markdown_content = await asyncio.to_thread(self._convert_to_markdown, path, openpyxl)
- return await self._md_parser.parse_content(
- markdown_content, source_path=str(path), instruction=instruction, **kwargs
- )
-
- def _should_use_process_pool(self, path: Path, kwargs: Dict[str, Any]) -> bool:
- if not self._process_pool_enabled():
- return False
- if path.suffix.lower() == ".xls":
- return False
- if kwargs.get("enable_link_rewrite") or kwargs.get("base_dir") or kwargs.get("allowed_media_dirs"):
- logger.debug("[ExcelParserProcessPool] Disabled for link/media rewrite parse")
- return False
- try:
- return path.stat().st_size >= _EXCEL_PROCESS_POOL_MIN_BYTES
- except OSError:
- return False
-
- async def _parse_existing_path_process_pool(
- self, path: Path, instruction: str = "", **kwargs
- ) -> ParseResult:
- parse_started = time.perf_counter()
- loop = asyncio.get_running_loop()
- temp_uri = self._md_parser._create_temp_uri()
- layout_kwargs: Dict[str, Any] = {
- key: value
- for key, value in kwargs.items()
- if key in {"resource_name", "source_name"} and isinstance(value, str)
- }
- if isinstance(kwargs.get("split_content"), bool):
- layout_kwargs["split_content"] = kwargs["split_content"]
- future = loop.run_in_executor(
- _get_excel_layout_executor(self._process_pool_workers()),
- partial(
- _build_excel_layout_in_process,
- path_str=str(path),
- temp_uri=temp_uri,
- instruction=instruction,
- layout_kwargs=layout_kwargs,
- config_dict=asdict(self.config),
- max_rows_per_sheet=self.max_rows_per_sheet,
- ),
- )
-
- worker_started = time.perf_counter()
- worker_result = await asyncio.wait_for(
- future, timeout=_EXCEL_PROCESS_POOL_TIMEOUT_S
- )
- worker_s = time.perf_counter() - worker_started
- layout = worker_result["layout"]
-
- self._md_parser._rewrite_ctx = {
- "enabled": False,
- "source_path": str(path),
- "doc_name": layout.doc_name,
- "root_dir": layout.root_dir,
- "import_root": None,
- "base_dir": None,
- "allowed_media_dirs": None,
- }
- try:
- apply_started = time.perf_counter()
- await self._md_parser._apply_layout(layout)
- apply_s = time.perf_counter() - apply_started
- finally:
- self._md_parser._rewrite_ctx = None
-
- parse_time = time.perf_counter() - parse_started
- logger.info(
- f"[ExcelParserProcessPool] path={path} total={parse_time:.3f}s "
- f"worker_wall={worker_s:.3f}s convert={worker_result.get('convert_s', -1.0):.3f}s "
- f"layout={worker_result.get('layout_s', -1.0):.3f}s apply={apply_s:.3f}s "
- f"chars={worker_result.get('markdown_chars')} ops={worker_result.get('layout_ops')} "
- f"writes={worker_result.get('layout_write_ops')} mkdirs={worker_result.get('layout_mkdir_ops')}"
- )
-
- root = ResourceNode(
- type=NodeType.ROOT,
- title=layout.doc_title,
- level=0,
- meta=layout.meta.get("frontmatter", {}),
- )
- result = create_parse_result(
- root=root,
- source_path=str(path),
- source_format="markdown",
- parser_name="MarkdownParser",
- parse_time=parse_time,
- meta=layout.meta,
- warnings=layout.warnings,
- )
- result.temp_dir_path = layout.temp_uri
- return result
-
- async def parse_content(
- self, content: str, source_path: Optional[str] = None, instruction: str = "", **kwargs
- ) -> ParseResult:
- """Parse content - delegates to MarkdownParser."""
- result = await self._md_parser.parse_content(content, source_path, **kwargs)
- result.source_format = "xlsx"
- result.parser_name = "ExcelParser"
- return result
-
- def _convert_xls_to_markdown(self, path: Path) -> str:
- """Convert legacy .xls spreadsheet to Markdown using xlrd."""
- import xlrd
-
- # formatting_info=True enables xlrd to detect date cells via XL_CELL_DATE
- # instead of reporting them as XL_CELL_NUMBER with raw float serials
- wb = xlrd.open_workbook(str(path), formatting_info=True, on_demand=True)
- try:
- return self._build_xls_markdown(wb, path, xlrd)
- finally:
- wb.release_resources()
-
- def _build_xls_markdown(self, wb, path: Path, xlrd) -> str:
- """Build markdown from xlrd workbook."""
- markdown_parts = []
- markdown_parts.append(f"# {path.stem}")
- markdown_parts.append(f"**Sheets:** {wb.nsheets}")
-
- for sheet_idx in range(wb.nsheets):
- sheet = wb.sheet_by_index(sheet_idx)
- parts = [f"## Sheet: {sheet.name}"]
-
- if sheet.nrows == 0 or sheet.ncols == 0:
- parts.append("*Empty sheet*")
- markdown_parts.append("\n\n".join(parts))
- continue
-
- parts.append(f"**Dimensions:** {sheet.nrows} rows × {sheet.ncols} columns")
-
- rows_to_process = sheet.nrows
- if self.max_rows_per_sheet > 0:
- rows_to_process = min(sheet.nrows, self.max_rows_per_sheet)
-
- rows = []
- for row_idx in range(rows_to_process):
- row_data = []
- for col_idx in range(sheet.ncols):
- row_data.append(self._format_xls_cell(sheet.cell(row_idx, col_idx), wb, xlrd))
- rows.append(row_data)
-
- if rows:
- from openviking.parse.base import format_table_to_markdown
-
- parts.append(format_table_to_markdown(rows, has_header=True))
-
- if self.max_rows_per_sheet > 0 and sheet.nrows > self.max_rows_per_sheet:
- parts.append(
- f"\n*... {sheet.nrows - self.max_rows_per_sheet} more rows truncated ...*"
- )
-
- markdown_parts.append("\n\n".join(parts))
-
- return "\n\n".join(markdown_parts)
-
- @staticmethod
- def _format_xls_cell(cell, wb, xlrd) -> str:
- """Format a single xlrd cell value with proper type handling."""
- if cell.ctype == xlrd.XL_CELL_EMPTY or cell.ctype == xlrd.XL_CELL_BLANK:
- return ""
- if cell.ctype == xlrd.XL_CELL_DATE:
- try:
- dt = xlrd.xldate_as_tuple(cell.value, wb.datemode)
- # Include time component if non-zero
- if dt[3] or dt[4] or dt[5]:
- return (
- f"{dt[0]:04d}-{dt[1]:02d}-{dt[2]:02d} {dt[3]:02d}:{dt[4]:02d}:{dt[5]:02d}"
- )
- return f"{dt[0]:04d}-{dt[1]:02d}-{dt[2]:02d}"
- except Exception:
- return str(cell.value)
- if cell.ctype == xlrd.XL_CELL_BOOLEAN:
- return "TRUE" if cell.value else "FALSE"
- if cell.ctype == xlrd.XL_CELL_ERROR:
- # xlrd error code map
- error_map = {
- 0x00: "#NULL!",
- 0x07: "#DIV/0!",
- 0x0F: "#VALUE!",
- 0x17: "#REF!",
- 0x1D: "#NAME?",
- 0x24: "#NUM!",
- 0x2A: "#N/A",
- }
- return error_map.get(cell.value, f"#ERR({cell.value})")
- if cell.ctype == xlrd.XL_CELL_NUMBER:
- # Display integers without trailing .0
- if cell.value == int(cell.value):
- return str(int(cell.value))
- return str(cell.value)
- # XL_CELL_TEXT or fallback
- return str(cell.value) if cell.value is not None else ""
-
- def _convert_to_markdown(self, path: Path, openpyxl) -> str:
- """Convert Excel spreadsheet to Markdown string."""
- wb = openpyxl.load_workbook(path, data_only=True)
-
- markdown_parts = []
- markdown_parts.append(f"# {path.stem}")
- markdown_parts.append(f"**Sheets:** {len(wb.sheetnames)}")
-
- for sheet_name in wb.sheetnames:
- sheet = wb[sheet_name]
- sheet_content = self._convert_sheet(sheet, sheet_name)
- markdown_parts.append(sheet_content)
-
- return "\n\n".join(markdown_parts)
-
- def _convert_sheet(self, sheet, sheet_name: str) -> str:
- """Convert a single sheet to markdown."""
- parts = []
- parts.append(f"## Sheet: {sheet_name}")
-
- max_row = sheet.max_row
- max_col = sheet.max_column
-
- if max_row == 0 or max_col == 0:
- parts.append("*Empty sheet*")
- return "\n\n".join(parts)
-
- parts.append(f"**Dimensions:** {max_row} rows × {max_col} columns")
-
- rows_to_process = max_row
- if self.max_rows_per_sheet > 0:
- rows_to_process = min(max_row, self.max_rows_per_sheet)
-
- rows = []
- for _row_idx, row in enumerate(
- sheet.iter_rows(min_row=1, max_row=rows_to_process, values_only=True), 1
- ):
- row_data = []
- for cell in row:
- if cell is None:
- row_data.append("")
- else:
- row_data.append(str(cell))
- rows.append(row_data)
-
- if rows:
- from openviking.parse.base import format_table_to_markdown
-
- table_md = format_table_to_markdown(rows, has_header=True)
- parts.append(table_md)
-
- if self.max_rows_per_sheet > 0 and max_row > self.max_rows_per_sheet:
- parts.append(f"\n*... {max_row - self.max_rows_per_sheet} more rows truncated ...*")
-
- return "\n\n".join(parts)
diff --git a/openviking/parse/parsers/legacy_doc.py b/openviking/parse/parsers/legacy_doc.py
deleted file mode 100644
index 11b48a519..000000000
--- a/openviking/parse/parsers/legacy_doc.py
+++ /dev/null
@@ -1,408 +0,0 @@
-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
-# SPDX-License-Identifier: AGPL-3.0
-"""
-Legacy Word document (.doc) parser for OpenViking.
-
-Extracts text from OLE2 compound binary .doc files using olefile,
-then delegates to MarkdownParser for tree structure creation.
-"""
-
-import asyncio
-import struct
-import zipfile
-from pathlib import Path
-from typing import List, Optional, Union
-
-from openviking.parse.base import ParseResult
-from openviking.parse.parsers.base_parser import BaseParser
-from openviking_cli.utils.config.parser_config import ParserConfig
-from openviking_cli.utils.logger import get_logger
-
-logger = get_logger(__name__)
-
-
-# Max stream size to read (50MB) — prevents DoS from crafted files
-_MAX_STREAM_SIZE = 50 * 1024 * 1024
-# Max character count sanity cap for ccpText
-_MAX_CCP_TEXT = 10_000_000
-_ZIP_SIGNATURES = (b"PK\x03\x04", b"PK\x05\x06", b"PK\x07\x08")
-
-
-class LegacyDocParser(BaseParser):
- """
- Legacy .doc (OLE2 binary) parser.
-
- Extracts text content from Word 97-2003 (.doc) files using olefile
- to read the WordDocument and table streams, then delegates to
- MarkdownParser for tree structure.
- """
-
- def __init__(self, config: Optional[ParserConfig] = None):
- from openviking.parse.parsers.markdown import MarkdownParser
-
- self._md_parser = MarkdownParser(config=config)
- self.config = config or ParserConfig()
-
- @property
- def supported_extensions(self) -> List[str]:
- return [".doc"]
-
- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:
- """Parse legacy .doc file."""
- path = Path(source)
-
- if path.exists():
- if self._has_zip_signature(path):
- package_type = await asyncio.to_thread(self._classify_zip_package, path)
- if package_type == "docx":
- # python-docx writes OOXML regardless of the filename suffix. This
- # also occurs in real uploads whose extension says .doc while the
- # payload is a modern Word ZIP package. Never pass those bytes to
- # the legacy UTF-16 fallback: it turns ZIP data into plausible but
- # meaningless Unicode and silently persists corrupted Markdown.
- from openviking.parse.parsers.word import WordParser
-
- logger.info(
- "Detected OOXML Word content in %s; routing to WordParser",
- path.name,
- )
- return await WordParser(config=self.config).parse(
- path,
- instruction=instruction,
- **kwargs,
- )
-
- raise ValueError(f"{path.name} is a ZIP package, not a legacy Word .doc file")
-
- text = await asyncio.to_thread(self._extract_text, path)
- result = await self._md_parser.parse_content(
- text, source_path=str(path), instruction=instruction, **kwargs
- )
- else:
- result = await self._md_parser.parse_content(
- str(source), instruction=instruction, **kwargs
- )
- result.source_format = "doc"
- result.parser_name = "LegacyDocParser"
- return result
-
- @staticmethod
- def _has_zip_signature(path: Path) -> bool:
- """Return whether the payload starts with a recognized ZIP signature."""
- with path.open("rb") as file_obj:
- return file_obj.read(4) in _ZIP_SIGNATURES
-
- @staticmethod
- def _classify_zip_package(path: Path) -> str:
- """Identify a ZIP-backed Word package without extracting archive members."""
- try:
- with zipfile.ZipFile(path) as archive:
- try:
- archive.getinfo("[Content_Types].xml")
- archive.getinfo("word/document.xml")
- except KeyError:
- return "zip"
- return "docx"
- except (zipfile.BadZipFile, zipfile.LargeZipFile):
- # A payload with ZIP magic is still not a legacy OLE .doc. Classify
- # malformed archives as generic ZIP so callers fail closed instead of
- # feeding binary bytes into the permissive legacy text fallback.
- return "zip"
-
- async def parse_content(
- self, content: str, source_path: Optional[str] = None, instruction: str = "", **kwargs
- ) -> ParseResult:
- """Parse content string — delegates to MarkdownParser."""
- result = await self._md_parser.parse_content(
- content, source_path, instruction=instruction, **kwargs
- )
- result.source_format = "doc"
- result.parser_name = "LegacyDocParser"
- return result
-
- def _extract_text(self, path: Path) -> str:
- """
- Extract text from a legacy .doc OLE2 file.
-
- Reads the WordDocument stream and uses the FIB (File Information Block)
- to locate text in the document body. Falls back to raw byte scanning
- if structured extraction fails.
- """
- import olefile
-
- try:
- ole = olefile.OleFileIO(str(path))
- except Exception as e:
- logger.warning(f"Failed to open .doc as OLE file: {e}")
- return self._fallback_extract(path)
-
- try:
- return self._extract_from_ole(ole)
- except Exception as e:
- logger.warning(f"Structured OLE extraction failed, using fallback: {e}")
- return self._fallback_extract(path)
- finally:
- ole.close()
-
- @staticmethod
- def _read_ole_stream(ole, stream_name: str) -> bytes:
- """Read an OLE stream with size cap to prevent DoS."""
- stream = ole.openstream(stream_name)
- data = stream.read(_MAX_STREAM_SIZE + 1)
- if len(data) > _MAX_STREAM_SIZE:
- raise ValueError(f"OLE stream '{stream_name}' exceeds {_MAX_STREAM_SIZE} bytes")
- return data
-
- def _extract_from_ole(self, ole) -> str:
- """
- Extract text from OLE streams using the Word Binary File Format.
-
- Reads the FIB to determine if text is stored as UTF-16 or compressed
- (CP1252), then extracts the document body text from the appropriate
- stream (WordDocument or table stream).
- """
- if not ole.exists("WordDocument"):
- raise ValueError("No WordDocument stream found")
-
- word_doc = self._read_ole_stream(ole, "WordDocument")
-
- # Minimum FIB size: need at least 0x01A8 bytes for Word 97+ FIB fields
- if len(word_doc) < 0x01A8:
- raise ValueError(f"WordDocument stream too small ({len(word_doc)} bytes)")
-
- # Check FIB version (nFib at offset 0x0002) — require Word 97+ (0x00C1+)
- nfib = struct.unpack_from("<H", word_doc, 0x0002)[0]
- if nfib < 0x00C1:
- raise ValueError(f"Unsupported Word version (nFib=0x{nfib:04X}), need Word 97+")
-
- # Read FIB flags at offset 0x000A
- # Bit 9: table stream selector (0Table vs 1Table)
- # Bit 8: fComplex (complex fast-saved format — does not affect encoding)
- flags = struct.unpack_from("<H", word_doc, 0x000A)[0]
- is_1table = bool(flags & 0x0200)
- table_stream_name = "1Table" if is_1table else "0Table"
-
- # Read ccpText (character count of main document text) at FIB offset 0x004C
- ccp_text = struct.unpack_from("<i", word_doc, 0x004C)[0]
-
- if ccp_text <= 0:
- raise ValueError("ccpText is zero or negative")
- # Cap ccpText to prevent memory exhaustion from crafted files
- ccp_text = min(ccp_text, _MAX_CCP_TEXT)
-
- # Read the Clx from the table stream to find text positions
- if not ole.exists(table_stream_name):
- raise ValueError(f"Table stream '{table_stream_name}' not found")
- table_data = self._read_ole_stream(ole, table_stream_name)
-
- # fcClx offset in FIB (Word 97+ standard location)
- fc_clx = struct.unpack_from("<i", word_doc, 0x01A2)[0]
- lcb_clx = struct.unpack_from("<i", word_doc, 0x01A6)[0]
-
- if fc_clx <= 0 or lcb_clx <= 0 or fc_clx + lcb_clx > len(table_data):
- return self._simple_text_extract(word_doc, ccp_text)
-
- return self._extract_via_clx(word_doc, table_data, fc_clx, lcb_clx, ccp_text)
-
- def _simple_text_extract(self, word_doc: bytes, ccp_text: int) -> str:
- """
- Simple text extraction using FIB text offset.
-
- The main document text starts at offset 0x0800 in the WordDocument stream
- for most Word 97+ files. Tries UTF-16LE first; falls back to CP1252 if
- the stream is too small for UTF-16.
- """
- text_start = 0x0800 # Standard text start offset
-
- if text_start >= len(word_doc):
- raise ValueError("WordDocument stream too small for text extraction")
-
- # Try UTF-16LE first (2 bytes per char)
- if ccp_text * 2 + text_start <= len(word_doc):
- end = text_start + ccp_text * 2
- raw = word_doc[text_start:end]
- text = raw.decode("utf-16-le", errors="replace")
- # Sanity: if mostly printable, it's likely correct
- if (
- sum(1 for c in text[:200] if c.isprintable() or c in "\n\r\t")
- > len(text[:200]) * 0.5
- ):
- return self._clean_word_text(text)
-
- # Fall back to CP1252 single-byte
- end = min(text_start + ccp_text, len(word_doc))
- raw = word_doc[text_start:end]
- return self._clean_word_text(self._decode_cp1252(raw))
-
- def _extract_via_clx(
- self,
- word_doc: bytes,
- table_data: bytes,
- fc_clx: int,
- lcb_clx: int,
- ccp_text: int,
- ) -> str:
- """
- Extract text using the Clx (piece table) structure.
-
- The Clx contains a PiecePLC that maps character positions to file offsets,
- allowing reconstruction of the document text even when pieces are scattered.
- """
- clx = table_data[fc_clx : fc_clx + lcb_clx]
- pos = 0
- text_parts = []
- chars_extracted = 0
-
- # Skip any Grpprl (type 0x01) entries in the Clx
- while pos < len(clx) and clx[pos] == 0x01:
- if pos + 3 > len(clx):
- break
- cb = struct.unpack_from("<H", clx, pos + 1)[0]
- advance = 3 + cb
- if advance <= 0:
- break # Prevent infinite loop on zero-length Grpprl
- pos += advance
-
- # Now we should be at the Pcdt (type 0x02)
- if pos >= len(clx) or clx[pos] != 0x02:
- return self._simple_text_extract(word_doc, ccp_text)
-
- pos += 1 # skip type byte
- if pos + 4 > len(clx):
- return self._simple_text_extract(word_doc, ccp_text)
-
- lcb_pcd = struct.unpack_from("<I", clx, pos)[0]
- pos += 4
-
- # PLC structure: (n+1) CPs followed by n PCDs (each 8 bytes)
- pcd_start = pos
- pcd_end = pos + lcb_pcd
-
- if pcd_end > len(clx):
- return self._simple_text_extract(word_doc, ccp_text)
-
- # Calculate number of pieces: (lcb_pcd - 4) / (4 + 8) per piece,
- # but CPs are (n+1)*4 bytes + n*8 bytes = lcb_pcd
- # So: 4*(n+1) + 8*n = lcb_pcd → 12n + 4 = lcb_pcd → n = (lcb_pcd - 4) / 12
- n_pieces = (lcb_pcd - 4) // 12
- if n_pieces <= 0:
- return self._simple_text_extract(word_doc, ccp_text)
-
- # Read character positions (n+1 values)
- cps = []
- for i in range(n_pieces + 1):
- offset = pcd_start + i * 4
- if offset + 4 > len(clx):
- break
- cps.append(struct.unpack_from("<I", clx, offset)[0])
-
- # Read piece descriptors (start after the CPs)
- pcd_array_start = pcd_start + (n_pieces + 1) * 4
-
- for i in range(min(n_pieces, len(cps) - 1)):
- if chars_extracted >= ccp_text:
- break
-
- pcd_offset = pcd_array_start + i * 8
- if pcd_offset + 8 > len(clx):
- break
-
- # PCD: 2 bytes flags, 4 bytes fc, 2 bytes prm
- fc_value = struct.unpack_from("<I", clx, pcd_offset + 2)[0]
-
- piece_cp_start = cps[i]
- piece_cp_end = cps[i + 1]
- piece_char_count = piece_cp_end - piece_cp_start
-
- # Bit 30 of fc indicates compressed (CP1252) text
- is_compressed = bool(fc_value & 0x40000000)
- fc_real = fc_value & 0x3FFFFFFF
-
- if is_compressed:
- # CP1252: fc_real / 2 is the byte offset
- byte_offset = fc_real // 2
- byte_end = byte_offset + piece_char_count
- if byte_end <= len(word_doc):
- raw = word_doc[byte_offset:byte_end]
- text_parts.append(self._decode_cp1252(raw))
- else:
- logger.warning(
- f"Piece {i} extends beyond stream ({byte_end} > {len(word_doc)})"
- )
- else:
- # UTF-16LE
- byte_offset = fc_real
- byte_end = byte_offset + piece_char_count * 2
- if byte_end <= len(word_doc):
- raw = word_doc[byte_offset:byte_end]
- text_parts.append(raw.decode("utf-16-le", errors="replace"))
- else:
- logger.warning(
- f"Piece {i} extends beyond stream ({byte_end} > {len(word_doc)})"
- )
-
- chars_extracted += piece_char_count
-
- result = self._clean_word_text("".join(text_parts))
- if not result.strip():
- return self._simple_text_extract(word_doc, ccp_text)
- return result
-
- @staticmethod
- def _decode_cp1252(data: bytes) -> str:
- """Decode CP1252 bytes to string."""
- return data.decode("cp1252", errors="replace")
-
- @staticmethod
- def _clean_word_text(text: str) -> str:
- """Normalize Word control characters to readable equivalents."""
- text = text.replace("\r\n", "\n").replace("\r", "\n")
- # \x07 = cell/row end, \x0B = soft line break, \x0C = section break
- text = text.replace("\x07", "\t").replace("\x0b", "\n").replace("\x0c", "\n\n")
- return text
-
- def _fallback_extract(self, path: Path) -> str:
- """
- Last-resort text extraction by scanning raw bytes for readable text runs.
-
- Tries UTF-16LE decoding first (common in .doc), then falls back to CP1252.
- """
- # Cap read size to prevent DoS from large files
- with open(path, "rb") as f:
- raw = f.read(_MAX_STREAM_SIZE)
-
- # Try to find UTF-16LE text (every other byte is often 0x00 for ASCII)
- try:
- decoded = raw.decode("utf-16-le", errors="ignore")
- # Filter to printable text runs
- lines = []
- current = []
- for ch in decoded:
- if ch.isprintable() or ch in "\n\t":
- current.append(ch)
- else:
- if len(current) > 3:
- lines.append("".join(current))
- current = []
- if current and len(current) > 3:
- lines.append("".join(current))
- text = "\n".join(lines)
- if len(text) > 50:
- return text
- except Exception:
- pass
-
- # Fall back to CP1252
- text = raw.decode("cp1252", errors="replace")
- lines = []
- current = []
- for ch in text:
- if ch.isprintable() or ch in "\n\t":
- current.append(ch)
- else:
- if len(current) > 3:
- lines.append("".join(current))
- current = []
- if current and len(current) > 3:
- lines.append("".join(current))
- return "\n".join(lines)
diff --git a/openviking/parse/parsers/markdown.py b/openviking/parse/parsers/markdown.py
index 4d267aa54..a352db2d1 100644
--- a/openviking/parse/parsers/markdown.py
+++ b/openviking/parse/parsers/markdown.py
@@ -19,7 +19,6 @@ The parser handles scenarios:
import asyncio
import hashlib
-import io
import os
import re
import time
@@ -29,6 +28,7 @@ from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
from openviking.parse.accessors.mime_types import IANA_MEDIA_TYPE_TO_EXTENSION
from openviking.parse.base import NodeType, ParseResult, ResourceNode, create_parse_result
+from openviking.parse.image_validation import is_valid_image
from openviking.parse.parsers.base_parser import BaseParser
from openviking.parse.parsers.code.ast.providers import supports_code_skeleton
from openviking.parse.parsers.constants import (
@@ -134,6 +134,25 @@ class _Layout:
warnings: List[str] = field(default_factory=list)
+@dataclass
+class _RewriteContext:
+ """Link-rewrite inputs and memoized results owned by one parse_content call."""
+
+ source_path: str
+ doc_name: str
+ root_dir: str
+ import_root: str
+ base_dir: Optional[Path]
+ allowed_media_dirs: Optional[List[Path]]
+ split_content: bool
+ adaptive_flatten_requested: bool
+ flatten_single_output: bool
+ split_cache: Dict[Tuple[str, bool, bool], Optional[Dict[str, str]]] = field(
+ default_factory=dict
+ )
+ image_probe_cache: Dict[str, bool] = field(default_factory=dict)
+
+
if TYPE_CHECKING:
pass
@@ -158,6 +177,7 @@ class MarkdownParser(BaseParser):
# Worst-case token density used by _estimate_token_count (CJK ~0.7 tok/char).
# Used to bound force-split chunk size so it also respects the token budget.
MAX_TOKENS_PER_CHAR = 0.7
+ TABLE_HEADER_REPEAT_MAX_CHARS = 1024
MAX_MERGED_FILENAME_LENGTH = 32 # Maximum length for merged section filenames
# Image validation constants
@@ -203,9 +223,6 @@ class MarkdownParser(BaseParser):
# Cache for VikingFS instance
self._viking_fs = None
- # Relative-link rewrite context, set per parse_content() call.
- self._rewrite_ctx = None
-
@property
def supported_extensions(self) -> List[str]:
"""Return list of supported file extensions."""
@@ -299,28 +316,28 @@ class MarkdownParser(BaseParser):
**layout_kwargs,
)
- # Set up relative-link rewrite context consumed by _write_section during
- # apply. Link rewriting is opt-in via kwargs (DirectoryParser sets these);
- # single-file ingestion never passes link_rewrite_root, so it never rewrites.
- # base_dir/allowed_media_dirs let the rewrite ask _ingest_will_handle_image
- # which image embeds ingestion will take (those are left for #2429).
- self._rewrite_ctx = {
- "enabled": bool(kwargs.get("enable_link_rewrite", False)),
- "source_path": source_path,
- "doc_name": layout.doc_name,
- "root_dir": layout.root_dir,
- "import_root": kwargs.get("link_rewrite_root"),
- "base_dir": base_dir,
- "allowed_media_dirs": allowed_media_dirs,
- "split_content": split_content,
- "adaptive_flatten_requested": adaptive_flatten_requested,
- "flatten_single_output": flatten_single_output,
- }
+ import_root = kwargs.get("link_rewrite_root")
+ rewrite_ctx = None
+ if kwargs.get("enable_link_rewrite", False) and source_path and import_root:
+ rewrite_ctx = _RewriteContext(
+ source_path=source_path,
+ doc_name=layout.doc_name,
+ root_dir=layout.root_dir,
+ import_root=import_root,
+ base_dir=base_dir,
+ allowed_media_dirs=allowed_media_dirs,
+ split_content=split_content,
+ adaptive_flatten_requested=adaptive_flatten_requested,
+ flatten_single_output=flatten_single_output,
+ )
# Phase 2 — write only: replay the plan against the real VikingFS,
# rewriting links and ingesting local images.
await self._apply_layout(
- layout, base_dir=base_dir, allowed_media_dirs=allowed_media_dirs
+ layout,
+ rewrite_ctx=rewrite_ctx,
+ base_dir=base_dir,
+ allowed_media_dirs=allowed_media_dirs,
)
parse_time = time.time() - start_time
@@ -350,9 +367,6 @@ class MarkdownParser(BaseParser):
except Exception as e:
logger.error(f"[MarkdownParser] Parse failed: {e}", exc_info=True)
raise
- finally:
- # Rewrite context lives exactly one parse_content call.
- self._rewrite_ctx = None
async def _compute_layout(
self,
@@ -438,6 +452,7 @@ class MarkdownParser(BaseParser):
self,
layout: _Layout,
*,
+ rewrite_ctx: Optional[_RewriteContext] = None,
base_dir: Optional[Path] = None,
allowed_media_dirs: Optional[List[Path]] = None,
) -> None:
@@ -450,12 +465,7 @@ class MarkdownParser(BaseParser):
write_ops = [op for op in layout.ops if op.kind == "write"]
write_chars = sum(len(op.content or "") for op in write_ops)
- rewrite_enabled = bool(
- self._rewrite_ctx
- and self._rewrite_ctx.get("enabled")
- and self._rewrite_ctx.get("source_path")
- and self._rewrite_ctx.get("import_root")
- )
+ rewrite_enabled = rewrite_ctx is not None
mkdir_started = time.perf_counter()
mkdir_count = 0
@@ -467,7 +477,7 @@ class MarkdownParser(BaseParser):
write_started = time.perf_counter()
for op in write_ops:
- await self._write_section(op.uri, op.content)
+ await self._write_section(op.uri, op.content, rewrite_ctx=rewrite_ctx)
write_s = time.perf_counter() - write_started
# Ingest local image files, placing each image next to the markdown file
@@ -622,8 +632,10 @@ class MarkdownParser(BaseParser):
"""
Split oversized content by paragraphs, force split single oversized paragraphs.
- Enforces both a token estimate limit (max_size) and a hard character limit
- (self.config.max_section_chars) to guard against token estimation errors.
+ Targets both the token estimate limit (max_size) and the character limit
+ (self.config.max_section_chars). A single oversized Markdown table row is
+ kept intact so downstream semantic and embedding stages can apply their own
+ input limits without destroying the source table structure.
Args:
content: Content to split
@@ -663,8 +675,16 @@ class MarkdownParser(BaseParser):
# to keep every chunk within the token budget.
token_safe_chars = max(1, int(max_size / self.MAX_TOKENS_PER_CHAR))
step = min(max_chars, token_safe_chars)
- for i in range(0, len(para), step):
- parts.append(para[i : i + step].strip())
+ table_parts = self._split_markdown_table_by_rows(
+ para,
+ max_size=max_size,
+ max_chars=step,
+ )
+ if table_parts:
+ parts.extend(table_parts)
+ else:
+ for i in range(0, len(para), step):
+ parts.append(para[i : i + step].strip())
elif (
current_tokens + para_tokens > max_size or len(current) + len(para) + 2 > max_chars
) and current:
@@ -686,6 +706,95 @@ class MarkdownParser(BaseParser):
return parts if parts else [content]
+ @staticmethod
+ def _is_markdown_table_row(line: str) -> bool:
+ stripped = line.strip()
+ return stripped.startswith("|") and stripped.endswith("|") and "|" in stripped[1:-1]
+
+ @staticmethod
+ def _is_markdown_table_separator(line: str) -> bool:
+ stripped = line.strip()
+ if not stripped.startswith("|") or not stripped.endswith("|"):
+ return False
+ cells = [cell.strip() for cell in stripped.strip("|").split("|")]
+ return bool(cells) and all(re.fullmatch(r":?-{3,}:?", cell or "") for cell in cells)
+
+ def _split_markdown_table_by_rows(
+ self,
+ content: str,
+ *,
+ max_size: int,
+ max_chars: int,
+ ) -> Optional[List[str]]:
+ """Split a single oversized Markdown table without cutting rows.
+
+ The first two table rows are treated as the header/separator and repeated
+ in subsequent chunks when they are small enough to fit. Any prefix before
+ the table, such as a section heading merged into the oversized paragraph,
+ is kept on the first chunk only. A single row may exceed the requested
+ limits because row integrity takes precedence at this source-storage stage.
+ """
+ lines = content.splitlines()
+ table_start = -1
+ for index in range(len(lines) - 1):
+ if self._is_markdown_table_row(lines[index]) and self._is_markdown_table_separator(
+ lines[index + 1]
+ ):
+ table_start = index
+ break
+ if table_start < 0:
+ return None
+
+ table_end = table_start + 2
+ while table_end < len(lines) and self._is_markdown_table_row(lines[table_end]):
+ table_end += 1
+
+ if table_end - table_start < 3:
+ return None
+ if any(line.strip() for line in lines[table_end:]):
+ return None
+
+ prefix = "\n".join(lines[:table_start]).strip()
+ header = lines[table_start : table_start + 2]
+ rows = lines[table_start + 2 : table_end]
+ header_text = "\n".join(header)
+ repeat_header = (
+ len(header_text) <= self.TABLE_HEADER_REPEAT_MAX_CHARS
+ and len(header_text) < max_chars
+ and self._estimate_token_count(header_text) < max_size
+ )
+
+ parts: List[str] = []
+ current_lines: List[str] = []
+ current_data_rows = 0
+
+ def flush() -> None:
+ nonlocal current_lines, current_data_rows
+ if current_lines:
+ parts.append("\n".join(current_lines).strip())
+ current_lines = []
+ current_data_rows = 0
+
+ if prefix:
+ current_lines.extend([prefix, ""])
+ current_lines.extend(header)
+
+ for row in rows:
+ candidate_lines = [*current_lines, row]
+ candidate = "\n".join(candidate_lines)
+ if current_lines and (
+ len(candidate) > max_chars or self._estimate_token_count(candidate) > max_size
+ ) and current_data_rows > 0:
+ flush()
+ current_lines = [*header, row] if repeat_header else [row]
+ current_data_rows = 1
+ else:
+ current_lines.append(row)
+ current_data_rows += 1
+
+ flush()
+ return parts or None
+
async def _ingest_local_images(
self,
root_dir: str,
@@ -908,45 +1017,7 @@ class MarkdownParser(BaseParser):
Returns:
True if the image satisfies all requirements, otherwise False
"""
- # File size check (local file path limit: 10 MB)
- if len(image_bytes) > self.IMAGE_MAX_FILE_BYTES:
- logger.warning(f"[MarkdownParser] Image exceeds 10MB, skipping: {source_path}")
- return False
-
- # Pixel size check
- try:
- from PIL import Image
-
- with Image.open(io.BytesIO(image_bytes)) as img:
- width, height = img.size
- except Exception as e:
- logger.warning(
- f"[MarkdownParser] Cannot read image dimensions, skipping {source_path}: {e}"
- )
- return False
-
- if width <= self.IMAGE_MIN_SIDE or height <= self.IMAGE_MIN_SIDE:
- logger.warning(
- f"[MarkdownParser] Image side too small ({width}x{height}), skipping: {source_path}"
- )
- return False
-
- pixels = width * height
- if pixels < self.IMAGE_MIN_PIXELS or pixels > self.IMAGE_MAX_PIXELS:
- logger.warning(
- f"[MarkdownParser] Image pixel count out of range ({pixels}), skipping: {source_path}"
- )
- return False
-
- aspect_ratio = width / height
- if aspect_ratio < self.IMAGE_MIN_ASPECT_RATIO or aspect_ratio > self.IMAGE_MAX_ASPECT_RATIO:
- logger.warning(
- f"[MarkdownParser] Image aspect ratio out of range ({aspect_ratio:.4f}), "
- f"skipping: {source_path}"
- )
- return False
-
- return True
+ return is_valid_image(image_bytes, source_path)
@staticmethod
def _is_remote_uri(path: str) -> bool:
@@ -1021,7 +1092,12 @@ class MarkdownParser(BaseParser):
return keys[0], False
async def _rewrite_single_link(
- self, link: str, src_dir: str, import_root_abs: str, source_new_dir: str
+ self,
+ link: str,
+ src_dir: str,
+ import_root_abs: str,
+ source_new_dir: str,
+ rewrite_ctx: _RewriteContext,
) -> str:
"""Rewrite one link target; return original `link` if it must not change."""
raw = link.strip()
@@ -1065,7 +1141,7 @@ class MarkdownParser(BaseParser):
# (file vs directory), its name and the section files all come straight from
# the parser, so NOTHING about ingest is assumed here. Whatever parse_content
# does, running it in-memory tells us the final result.
- layout = await self._target_split_files(target_disk)
+ layout = await self._target_split_files(target_disk, rewrite_ctx)
if not layout:
return link # target unparsable → leave the link untouched
@@ -1097,19 +1173,17 @@ class MarkdownParser(BaseParser):
# D) Directory landing → point at the directory and drop any suffix.
return _to_rel(os.path.join(target_parent, landing), keep_suffix=False)
- async def _ingest_will_handle_image(self, link: str) -> bool:
+ async def _ingest_will_handle_image(self, link: str, rewrite_ctx: _RewriteContext) -> bool:
"""Whether ``_ingest_local_images`` will take this image reference: it must
resolve within base_dir/allowed_media_dirs AND pass image validation — the
exact conditions ingestion itself applies. Such embeds are left untouched so
#2429 can copy them next to the section and ``rewrite_image_uris`` can turn
them into viking:// URIs; everything else falls back to depth adjustment.
Results are cached per parse_content call."""
- ctx = self._rewrite_ctx or {}
- cache = ctx.setdefault("_image_probe_cache", {})
- if link not in cache:
+ if link not in rewrite_ctx.image_probe_cache:
handled = False
resolved = self._resolve_image_path(
- link, ctx.get("base_dir"), ctx.get("allowed_media_dirs")
+ link, rewrite_ctx.base_dir, rewrite_ctx.allowed_media_dirs
)
if resolved is not None:
try:
@@ -1117,17 +1191,15 @@ class MarkdownParser(BaseParser):
handled = await asyncio.to_thread(self._is_valid_image, image_bytes, resolved)
except Exception:
handled = False
- cache[link] = handled
- return cache[link]
+ rewrite_ctx.image_probe_cache[link] = handled
+ return rewrite_ctx.image_probe_cache[link]
async def _rewrite_relative_links(
self,
content: str,
*,
- source_path: str,
- doc_name: str,
section_subpath: str,
- import_root: str,
+ rewrite_ctx: _RewriteContext,
) -> str:
"""Rewrite relative markdown links in `content` (disk-coordinate).
@@ -1140,23 +1212,23 @@ class MarkdownParser(BaseParser):
"""
from openviking.parse.image_rewrite import HTML_IMG_PATTERN
- src_dir = os.path.dirname(os.path.abspath(source_path))
- import_root_abs = os.path.abspath(import_root)
- if (self._rewrite_ctx or {}).get("flatten_single_output", False):
+ src_dir = os.path.dirname(os.path.abspath(rewrite_ctx.source_path))
+ import_root_abs = os.path.abspath(rewrite_ctx.import_root)
+ if rewrite_ctx.flatten_single_output:
source_new_dir = os.path.join(src_dir, section_subpath)
else:
- source_new_dir = os.path.join(src_dir, doc_name, section_subpath)
+ source_new_dir = os.path.join(src_dir, rewrite_ctx.doc_name, section_subpath)
out: List[str] = []
last = 0
for match in _MD_LINK_RE.finditer(content):
out.append(content[last : match.start()])
bang, text, link = match.group(1), match.group(2), match.group(3)
- if bang and await self._ingest_will_handle_image(link):
+ if bang and await self._ingest_will_handle_image(link, rewrite_ctx):
new_link = link
else:
new_link = await self._rewrite_single_link(
- link, src_dir, import_root_abs, source_new_dir
+ link, src_dir, import_root_abs, source_new_dir, rewrite_ctx
)
out.append(f"{bang}[{text}]({new_link})")
last = match.end()
@@ -1169,32 +1241,33 @@ class MarkdownParser(BaseParser):
for match in HTML_IMG_PATTERN.finditer(rewritten):
pieces.append(rewritten[last : match.start()])
src = match.group(2)
- if await self._ingest_will_handle_image(src):
+ if await self._ingest_will_handle_image(src, rewrite_ctx):
new_src = src
else:
new_src = await self._rewrite_single_link(
- src, src_dir, import_root_abs, source_new_dir
+ src, src_dir, import_root_abs, source_new_dir, rewrite_ctx
)
pieces.append(f"{match.group(1)}{new_src}{match.group(3)}")
last = match.end()
pieces.append(rewritten[last:])
return "".join(pieces)
- async def _target_split_files(self, target_path: str) -> Optional[Dict[str, str]]:
+ async def _target_split_files(
+ self, target_path: str, rewrite_ctx: _RewriteContext
+ ) -> Optional[Dict[str, str]]:
"""The target's real ingest layout {"<doc_dir>/<section...>": text} via a
side-effect-free parse, cached per parse_content() call. None on parse failure."""
- ctx = self._rewrite_ctx or {}
- cache = ctx.setdefault("_split_cache", {})
- split_content = bool(ctx.get("split_content", True))
- flatten_single_output = bool(ctx.get("adaptive_flatten_requested", False))
+ split_content = rewrite_ctx.split_content
+ flatten_single_output = rewrite_ctx.adaptive_flatten_requested
key = (os.path.abspath(target_path), split_content, flatten_single_output)
- if key not in cache:
- cache[key] = await self._probe_split_layout(
+ if key not in rewrite_ctx.split_cache:
+ rewrite_ctx.split_cache[key] = await self._probe_split_layout(
target_path,
split_content=split_content,
flatten_single_output=flatten_single_output,
+ allowed_media_dirs=rewrite_ctx.allowed_media_dirs,
)
- return cache[key]
+ return rewrite_ctx.split_cache[key]
async def _probe_split_layout(
self,
@@ -1202,6 +1275,7 @@ class MarkdownParser(BaseParser):
*,
split_content: bool,
flatten_single_output: bool = False,
+ allowed_media_dirs: Optional[List[Path]] = None,
) -> Optional[Dict[str, str]]:
"""Plan the target's layout WITHOUT writing anything, returning
{"<doc_dir>/<section...>": text} keyed by path relative to the temp root, i.e.
@@ -1215,7 +1289,7 @@ class MarkdownParser(BaseParser):
flatten_single_output = not await self._has_ingestable_local_image(
content,
base_dir=Path(target_path).parent,
- allowed_media_dirs=(self._rewrite_ctx or {}).get("allowed_media_dirs"),
+ allowed_media_dirs=allowed_media_dirs,
)
layout = await self._compute_layout(
content,
@@ -1250,16 +1324,19 @@ class MarkdownParser(BaseParser):
return section_dir[len(root) + 1 :]
return ""
- async def _write_section(self, uri: str, content: str) -> None:
+ async def _write_section(
+ self,
+ uri: str,
+ content: str,
+ *,
+ rewrite_ctx: Optional[_RewriteContext] = None,
+ ) -> None:
"""Write a markdown section file, rewriting relative links when enabled."""
- ctx = self._rewrite_ctx
- if ctx and ctx.get("enabled") and ctx.get("source_path") and ctx.get("import_root"):
+ if rewrite_ctx is not None:
content = await self._rewrite_relative_links(
content,
- source_path=ctx["source_path"],
- doc_name=ctx["doc_name"],
- section_subpath=self._section_subpath(uri, ctx["root_dir"]),
- import_root=ctx["import_root"],
+ section_subpath=self._section_subpath(uri, rewrite_ctx.root_dir),
+ rewrite_ctx=rewrite_ctx,
)
await self._get_viking_fs().write_file(uri, content)
@@ -1605,9 +1682,9 @@ class MarkdownParser(BaseParser):
) -> None:
"""Plan merged sections as a single file with smart naming into ``ops``.
- If the joined content exceeds max_section_chars it is split further
- by _smart_split_content before writing, so no single file ever exceeds
- the hard character limit.
+ If the joined content exceeds max_section_chars it is split further by
+ _smart_split_content before writing. A single oversized Markdown table row
+ remains intact to preserve the source table structure.
"""
name = self._generate_merged_filename(sections)
content = "\n\n".join(c for _, c, _ in sections)
diff --git a/openviking/parse/parsers/powerpoint.py b/openviking/parse/parsers/powerpoint.py
deleted file mode 100644
index 9ef183996..000000000
--- a/openviking/parse/parsers/powerpoint.py
+++ /dev/null
@@ -1,150 +0,0 @@
-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
-# SPDX-License-Identifier: AGPL-3.0
-"""
-PowerPoint (.pptx) parser for OpenViking.
-
-Converts PowerPoint presentations to Markdown then parses using MarkdownParser.
-Inspired by microsoft/markitdown approach.
-"""
-
-import asyncio
-from pathlib import Path
-from typing import List, Optional, Union
-
-from openviking.parse.base import ParseResult
-from openviking.parse.parsers.base_parser import BaseParser
-from openviking_cli.utils.config.parser_config import ParserConfig
-from openviking_cli.utils.logger import get_logger
-
-logger = get_logger(__name__)
-
-
-class PowerPointParser(BaseParser):
- """
- PowerPoint presentation parser for OpenViking.
-
- Supports: .pptx
-
- Converts PowerPoint presentations to Markdown using python-pptx,
- then delegates to MarkdownParser for tree structure creation.
- """
-
- def __init__(self, config: Optional[ParserConfig] = None, extract_notes: bool = False):
- """
- Initialize PowerPoint parser.
-
- Args:
- config: Parser configuration
- extract_notes: Whether to extract speaker notes
- """
- from openviking.parse.parsers.markdown import MarkdownParser
-
- self._md_parser = MarkdownParser(config=config)
- self.config = config or ParserConfig()
- self.extract_notes = extract_notes
-
- @property
- def supported_extensions(self) -> List[str]:
- return [".pptx"]
-
- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:
- """Parse PowerPoint presentation from file path."""
- path = Path(source)
-
- if path.exists():
- import pptx
-
- markdown_content = await asyncio.to_thread(self._convert_to_markdown, path, pptx)
- result = await self._md_parser.parse_content(
- markdown_content, source_path=str(path), instruction=instruction, **kwargs
- )
- else:
- result = await self._md_parser.parse_content(
- str(source), instruction=instruction, **kwargs
- )
- result.source_format = "pptx"
- result.parser_name = "PowerPointParser"
- return result
-
- async def parse_content(
- self, content: str, source_path: Optional[str] = None, instruction: str = "", **kwargs
- ) -> ParseResult:
- """Parse content - delegates to MarkdownParser."""
- result = await self._md_parser.parse_content(content, source_path, **kwargs)
- result.source_format = "pptx"
- result.parser_name = "PowerPointParser"
- return result
-
- def _convert_to_markdown(self, path: Path, pptx) -> str:
- """Convert PowerPoint presentation to Markdown string."""
- prs = pptx.Presentation(path)
- markdown_parts = []
- slide_count = len(prs.slides)
-
- for idx, slide in enumerate(prs.slides, 1):
- slide_parts = []
- slide_parts.append(f"## Slide {idx}/{slide_count}")
-
- title = self._extract_slide_title(slide)
- if title:
- slide_parts.append(f"### {title}")
-
- content = self._extract_slide_content(slide)
- if content:
- slide_parts.append(content)
-
- if self.extract_notes and slide.has_notes_slide:
- notes = slide.notes_slide.notes_text_frame.text.strip()
- if notes:
- slide_parts.append(f"**Notes:** {notes}")
-
- markdown_parts.append("\n\n".join(slide_parts))
-
- return "\n\n---\n\n".join(markdown_parts)
-
- def _extract_slide_title(self, slide) -> str:
- """Extract title from a slide."""
- from pptx.enum.shapes import PP_PLACEHOLDER
-
- for shape in slide.shapes:
- if shape.is_placeholder:
- ph_type = shape.placeholder_format.type
- if ph_type in (PP_PLACEHOLDER.TITLE, PP_PLACEHOLDER.CENTER_TITLE):
- return shape.text.strip()
- return ""
-
- def _extract_slide_content(self, slide) -> str:
- """Extract content from slide shapes."""
- from pptx.enum.shapes import PP_PLACEHOLDER
-
- content_parts = []
-
- for shape in slide.shapes:
- if shape.is_placeholder:
- ph_type = shape.placeholder_format.type
- if ph_type in (PP_PLACEHOLDER.TITLE, PP_PLACEHOLDER.CENTER_TITLE):
- continue
-
- if hasattr(shape, "text") and shape.text.strip():
- if shape.has_table:
- content_parts.append(self._convert_table(shape.table))
- else:
- text = shape.text.strip()
- if text:
- content_parts.append(text)
-
- return "\n\n".join(content_parts)
-
- def _convert_table(self, table) -> str:
- """Convert PowerPoint table to markdown format."""
- if not table.rows:
- return ""
-
- rows = []
- for row in table.rows:
- row_data = [cell.text.strip() for cell in row.cells]
- rows.append(row_data)
-
- from openviking.parse.base import format_table_to_markdown
-
- return format_table_to_markdown(rows, has_header=True)
diff --git a/openviking/parse/parsers/word.py b/openviking/parse/parsers/word.py
deleted file mode 100644
index 0a76f110a..000000000
--- a/openviking/parse/parsers/word.py
+++ /dev/null
@@ -1,231 +0,0 @@
-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
-# SPDX-License-Identifier: AGPL-3.0
-"""
-Word document (.docx) parser for OpenViking.
-
-Converts Word documents to Markdown then parses using MarkdownParser.
-Inspired by microsoft/markitdown approach.
-"""
-
-import asyncio
-from pathlib import Path
-from typing import List, Optional, Union
-
-from openviking.parse.base import ParseResult
-from openviking.parse.parsers.base_parser import BaseParser
-from openviking_cli.utils.config.parser_config import ParserConfig
-from openviking_cli.utils.logger import get_logger
-
-logger = get_logger(__name__)
-
-
-class WordParser(BaseParser):
- """
- Word document parser for OpenViking.
-
- Supports: .docx
-
- Converts Word documents to Markdown using python-docx,
- then delegates to MarkdownParser for tree structure creation.
- """
-
- def __init__(self, config: Optional[ParserConfig] = None):
- """Initialize Word parser."""
- from openviking.parse.parsers.markdown import MarkdownParser
-
- self._md_parser = MarkdownParser(config=config)
- self.config = config or ParserConfig()
-
- @property
- def supported_extensions(self) -> List[str]:
- return [".docx"]
-
- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:
- """Parse Word document from file path."""
- path = Path(source)
-
- if path.exists():
- import docx
-
- from openviking_cli.utils.storage import get_storage
-
- storage = get_storage()
- resource_name = kwargs.get("resource_name") or kwargs.get("source_name") or path.stem
-
- markdown_content = await asyncio.to_thread(
- self._convert_to_markdown, path, docx, resource_name, storage
- )
- result = await self._md_parser.parse_content(
- markdown_content,
- source_path=str(path),
- # Forward the original upload name explicitly (mirrors pdf.py) so
- # the resource is named after it, not the temp upload path. Pass
- # the raw kwargs values rather than the path.stem-backed local
- # resource_name, leaving MarkdownParser's naming logic unchanged.
- resource_name=kwargs.get("resource_name"),
- source_name=kwargs.get("source_name"),
- instruction=instruction,
- base_dir=path.parent,
- # docx images are extracted into storage.media_dir, so (like pdf)
- # that is the only derived-media root needed.
- allowed_media_dirs=[storage.media_dir],
- split_content=kwargs.get("split_content", True),
- )
- else:
- result = await self._md_parser.parse_content(
- str(source),
- instruction=instruction,
- resource_name=kwargs.get("resource_name"),
- source_name=kwargs.get("source_name"),
- split_content=kwargs.get("split_content", True),
- )
- result.source_format = "docx"
- result.parser_name = "WordParser"
- return result
-
- async def parse_content(
- self, content: str, source_path: Optional[str] = None, instruction: str = "", **kwargs
- ) -> ParseResult:
- """Parse content - delegates to MarkdownParser."""
- result = await self._md_parser.parse_content(content, source_path, **kwargs)
- result.source_format = "docx"
- result.parser_name = "WordParser"
- return result
-
- def _convert_to_markdown(self, path: Path, docx, resource_name=None, storage=None) -> str:
- """Convert Word document to Markdown string.
-
- Iterates the document body in order so that tables appear in their
- original position rather than being appended at the end. Embedded
- images are extracted to local storage and referenced inline so the
- MarkdownParser can ingest them into VikingFS.
- """
- doc = docx.Document(path)
- markdown_parts = []
-
- # Map XML table elements to python-docx Table objects for O(1) lookup
- table_by_element = {table._tbl: table for table in doc.tables}
-
- # Track extracted images to deduplicate and number sequentially
- image_counter = [0]
-
- # Walk the document body in order to preserve table positions
- from docx.oxml.ns import qn
-
- for child in doc.element.body:
- if child.tag == qn("w:p"):
- # It's a paragraph
- from docx.text.paragraph import Paragraph
-
- paragraph = Paragraph(child, doc)
-
- # Extract any embedded images in this paragraph first so they
- # keep their original document position.
- image_md = self._convert_paragraph_images(
- paragraph, doc, qn, resource_name, storage, image_counter
- )
- if image_md:
- markdown_parts.append(image_md)
-
- if not paragraph.text.strip():
- continue
-
- style_name = paragraph.style.name if paragraph.style else "Normal"
-
- if style_name.startswith("Heading"):
- level = self._extract_heading_level(style_name)
- markdown_parts.append(f"{'#' * level} {paragraph.text}")
- else:
- text = self._convert_formatted_text(paragraph)
- markdown_parts.append(text)
-
- elif child.tag == qn("w:tbl"):
- # It's a table
- if child in table_by_element:
- markdown_parts.append(self._convert_table(table_by_element[child]))
-
- return "\n\n".join(markdown_parts)
-
- def _convert_paragraph_images(
- self, paragraph, doc, qn, resource_name, storage, image_counter
- ) -> str:
- """Extract embedded images from a paragraph and return markdown references.
-
- Images in .docx are stored as relationship parts (``word/media/*``)
- referenced by ``r:embed`` ids inside ``w:drawing`` elements. We pull the
- binary blob for each and persist it via the shared storage helper so the
- MarkdownParser's image ingestion can pick it up.
- """
- if storage is None:
- return ""
-
- parts = []
- # Each <a:blip r:embed="rId"> inside the paragraph references an image part
- for blip in paragraph._p.findall(".//" + qn("a:blip")):
- rid = blip.get(qn("r:embed"))
- if not rid:
- continue
- try:
- image_part = doc.part.related_parts[rid]
- image_bytes = image_part.blob
- except Exception as e:
- logger.warning(f"[WordParser] Failed to read embedded image {rid}: {e}")
- continue
-
- try:
- extension = Path(image_part.partname).suffix or ".png"
- image_counter[0] += 1
- filename = f"image{image_counter[0]}"
- image_path = storage.save_image(
- resource_name, image_bytes, filename=filename, extension=extension
- )
- # Reference relative to the media root so the MarkdownParser can
- # resolve it within the resource's media lifecycle (never cwd).
- rel_path = image_path.relative_to(storage.media_dir)
- parts.append(f"![{filename}]({rel_path})")
- except Exception as e:
- logger.warning(f"[WordParser] Failed to save embedded image {rid}: {e}")
-
- return "\n\n".join(parts)
-
- def _extract_heading_level(self, style_name: str) -> int:
- """Extract heading level from style name."""
- try:
- if "Heading" in style_name:
- parts = style_name.split()
- for part in parts:
- if part.isdigit():
- return min(int(part), 6)
- except Exception:
- pass
- return 1
-
- def _convert_formatted_text(self, paragraph) -> str:
- """Convert paragraph with formatting to markdown."""
- text_parts = []
- for run in paragraph.runs:
- text = run.text
- if not text:
- continue
- if run.bold:
- text = f"**{text}**"
- if run.italic:
- text = f"*{text}*"
- if run.underline:
- text = f"<ins>{text}</ins>"
- text_parts.append(text)
- return "".join(text_parts)
-
- def _convert_table(self, table) -> str:
- """Convert Word table to markdown format."""
- if not table.rows:
- return ""
-
- rows = []
- for row in table.rows:
- row_data = [cell.text.strip() for cell in row.cells]
- rows.append(row_data)
-
- from openviking.parse.base import format_table_to_markdown
-
- return format_table_to_markdown(rows, has_header=True)
diff --git a/openviking/parse/registry.py b/openviking/parse/registry.py
index dc0140562..8cadb5096 100644
--- a/openviking/parse/registry.py
+++ b/openviking/parse/registry.py
@@ -10,24 +10,18 @@ from pathlib import Path
from typing import Dict, List, Optional, Union
from openviking.parse.base import ParseResult
+from openviking.parse.parsers.anydoc import AnyDocParser
from openviking.parse.parsers.base_parser import BaseParser
from openviking.parse.parsers.constants import TYPESCRIPT_MPEG_TS_EXTENSION
from openviking.parse.parsers.directory import DirectoryParser
-from openviking.parse.parsers.epub import EPubParser
-from openviking.parse.parsers.excel import ExcelParser
# Import will be handled dynamically to avoid dependency issues
from openviking.parse.parsers.html import HTMLParser
-
-# Import markitdown-inspired parsers
-from openviking.parse.parsers.legacy_doc import LegacyDocParser
from openviking.parse.parsers.markdown import MarkdownParser
from openviking.parse.parsers.media import AudioParser, ImageParser, VideoParser
from openviking.parse.parsers.media.utils import is_mpeg_ts, read_mpeg_ts_probe
from openviking.parse.parsers.pdf import PDFParser
-from openviking.parse.parsers.powerpoint import PowerPointParser
from openviking.parse.parsers.text import TextParser
-from openviking.parse.parsers.word import WordParser
from openviking.parse.parsers.zip_parser import ZipParser
from openviking_cli.utils.config.parser_config import ParserConfig
@@ -59,14 +53,14 @@ class ParserRegistry:
self._register("pdf", PDFParser(config=self._parser_configs.get("pdf")))
self._register("html", HTMLParser(config=self._parser_configs.get("html")))
- # Register markitdown-inspired parsers (built-in)
- self._register("word", WordParser(config=self._parser_configs.get("word")))
- self._register("legacy_doc", LegacyDocParser(config=self._parser_configs.get("legacy_doc")))
+ # Register Office/EPUB through the unified AnyDoc parser.
self._register(
- "powerpoint", PowerPointParser(config=self._parser_configs.get("powerpoint"))
+ "anydoc",
+ AnyDocParser(
+ config=self._parser_configs.get("markdown"),
+ anydoc_config=self._parser_configs.get("anydoc"),
+ ),
)
- self._register("excel", ExcelParser(config=self._parser_configs.get("excel")))
- self._register("epub", EPubParser(config=self._parser_configs.get("epub")))
self._register("zip", ZipParser())
self._register("directory", DirectoryParser())
@@ -176,15 +170,7 @@ def get_registry() -> ParserRegistry:
"markdown": config.markdown,
"pdf": config.pdf,
"html": config.html,
- "word": config.markdown,
- "legacy_doc": config.markdown,
- "powerpoint": config.markdown,
- # Excel had no dedicated config section and reused
- # ``config.markdown``. Keep unset sectioning fields following
- # Markdown so existing deployments keep their node structure and
- # stable URIs after this upgrade.
- "excel": config.excel.with_sectioning_defaults_from(config.markdown),
- "epub": config.markdown,
+ "anydoc": getattr(config, "anydoc", None),
"image": config.image,
}
except Exception:
diff --git a/openviking_cli/utils/config/__init__.py b/openviking_cli/utils/config/__init__.py
index b84173610..fedbc5cc5 100644
--- a/openviking_cli/utils/config/__init__.py
+++ b/openviking_cli/utils/config/__init__.py
@@ -65,9 +65,9 @@ from .open_viking_config import (
from .ovcli_config import OVCLIConfig, load_ovcli_config
from .parser_config import (
PARSER_CONFIG_REGISTRY,
+ AnydocConfig,
AudioConfig,
CodeConfig,
- ExcelConfig,
HTMLConfig,
ImageConfig,
MarkdownConfig,
@@ -147,7 +147,7 @@ __all__ = [
"AudioConfig",
"VideoConfig",
"MarkdownConfig",
- "ExcelConfig",
+ "AnydocConfig",
"HTMLConfig",
"TextConfig",
"get_parser_config",
diff --git a/openviking_cli/utils/config/open_viking_config.py b/openviking_cli/utils/config/open_viking_config.py
index 601c9eb42..1c53330dc 100644
--- a/openviking_cli/utils/config/open_viking_config.py
+++ b/openviking_cli/utils/config/open_viking_config.py
@@ -28,10 +28,10 @@ from .log_config import LogConfig
from .memory_config import MemoryConfig
from .oauth_config import OAuthConfig
from .parser_config import (
+ AnydocConfig,
AudioConfig,
CodeConfig,
DirectoryConfig,
- ExcelConfig,
FeishuConfig,
HTMLConfig,
ImageConfig,
@@ -52,7 +52,7 @@ from .telemetry_config import TelemetryConfig
from .vlm_config import VLMConfig
-def _get_config_warning_logger():
+def _get_config_logger():
"""Use stdlib logging during config bootstrap to avoid early logger side effects."""
return logging.getLogger(__name__)
@@ -206,12 +206,9 @@ class OpenVikingConfig(BaseModel):
default_factory=MarkdownConfig, description="Markdown parsing configuration"
)
- excel: ExcelConfig = Field(
- # from_dict on an empty mapping, not the bare constructor: an absent
- # parsers.excel section must record that no key was set, so sectioning
- # still follows parsers.markdown for deployments predating this section.
- default_factory=lambda: ExcelConfig.from_dict({}),
- description="Excel parsing configuration",
+ anydoc: AnydocConfig = Field(
+ default_factory=AnydocConfig,
+ description="Shared anydoc Office conversion configuration",
)
html: HTMLConfig = Field(default_factory=HTMLConfig, description="HTML parsing configuration")
@@ -302,7 +299,7 @@ class OpenVikingConfig(BaseModel):
@model_validator(mode="after")
def _warn_on_deprecated_language_fallback(self) -> "OpenVikingConfig":
if self.language_fallback and self.language_fallback != "en":
- _get_config_warning_logger().warning(
+ _get_config_logger().warning(
"Config field 'language_fallback=%s' is deprecated and has no effect; "
"remove it, or set 'output_language_override' to pin an explicit language.",
self.language_fallback,
@@ -407,7 +404,7 @@ class OpenVikingConfig(BaseModel):
"audio",
"video",
"markdown",
- "excel",
+ "anydoc",
"html",
"text",
"directory",
@@ -432,6 +429,13 @@ class OpenVikingConfig(BaseModel):
parser_configs = {}
if not isinstance(parser_configs, dict):
raise ValueError("Invalid parsers config: 'parsers' section must be an object")
+ parser_configs = parser_configs.copy()
+ if "excel" in parser_configs:
+ parser_configs.pop("excel")
+ _get_config_logger().error(
+ "Config field 'parsers.excel' was removed and is ignored; "
+ "spreadsheet parsing now uses 'parsers.anydoc'."
+ )
raise_unknown_config_fields(
data=parser_configs,
valid_fields=set(parser_types),
diff --git a/openviking_cli/utils/config/parser_config.py b/openviking_cli/utils/config/parser_config.py
index abcdce746..31520326f 100644
--- a/openviking_cli/utils/config/parser_config.py
+++ b/openviking_cli/utils/config/parser_config.py
@@ -8,9 +8,9 @@ scattered across different modules. All configurations inherit from ParserConfig
and can be loaded from ov.conf files.
"""
-from dataclasses import dataclass, replace
+from dataclasses import dataclass
from pathlib import Path
-from typing import Any, Dict, Iterable, Optional, Union
+from typing import Any, Dict, Optional, Union
from openviking_cli.utils.logger import get_logger
@@ -33,7 +33,8 @@ class ParserConfig:
encoding: Default file encoding
max_section_size: Maximum tokens per section before splitting
section_size_flexibility: Allow overflow to maintain coherence (0.0-1.0)
- max_section_chars: Hard character limit per section (guards against token estimation errors)
+ max_section_chars: Target character limit per section. A single oversized
+ Markdown table row may remain intact to preserve table semantics.
"""
enabled: bool = True
@@ -43,9 +44,7 @@ class ParserConfig:
# Smart splitting configuration
max_section_size: int = 2048 # Maximum tokens per section before splitting
section_size_flexibility: float = 0.3 # Allow 30% overflow to maintain coherence
- max_section_chars: int = (
- 6000 # Hard character limit per section (guards against token estimation errors)
- )
+ max_section_chars: int = 6000 # Target character limit per section
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "ParserConfig":
@@ -466,92 +465,21 @@ class MarkdownConfig(ParserConfig):
@dataclass
-class ExcelConfig(ParserConfig):
- """
- Configuration for Excel parsing.
-
- Attributes:
- enable_process_pool: Offload Excel→Markdown conversion and layout
- planning to a ProcessPoolExecutor (default off).
- process_pool_workers: Max worker processes when the pool is enabled.
- """
-
- enable_process_pool: bool = False
- process_pool_workers: int = 2
-
- # Excel is converted to Markdown and then sectioned by MarkdownParser, so
- # these fields decide the resulting node structure and stable URIs.
- _SECTIONING_FIELDS = (
- "max_content_length",
- "encoding",
- "max_section_size",
- "section_size_flexibility",
- "max_section_chars",
- )
+class AnydocConfig(ParserConfig):
+ """Configuration for the shared anydoc Office converter."""
- # Names of keys a config source actually provided. Tracked as a plain
- # instance attribute rather than a dataclass field so it never appears in
- # asdict/model_dump output, cannot be injected from a config file, and does
- # not affect equality. Absent means "provenance unknown".
- _EXPLICIT_ATTR = "_openviking_explicit_keys"
+ max_table_rows: int = 1000
@classmethod
- def from_dict(cls, data: Dict[str, Any]) -> "ExcelConfig":
- """Build the config while remembering which keys were actually present.
-
- ``with_sectioning_defaults_from`` needs to tell "the user wrote this
- value" from "the key was absent". Comparing against class defaults
- cannot do that, so record the provided keys here instead.
- """
- config = super().from_dict(data)
- return config.with_explicit_keys(data)
-
- def with_explicit_keys(self, names: Iterable[str]) -> "ExcelConfig":
- """Return this config marked as having ``names`` explicitly configured."""
- object.__setattr__(self, self._EXPLICIT_ATTR, frozenset(names))
- return self
-
- @property
- def explicit_keys(self) -> Optional[frozenset]:
- """Keys a config source provided, or ``None`` when unknown."""
- return getattr(self, self._EXPLICIT_ATTR, None)
-
- def with_sectioning_defaults_from(self, markdown: "ParserConfig") -> "ExcelConfig":
- """Inherit sectioning fields that ``parsers.excel`` did not set.
-
- Excel used to be registered with ``config.markdown`` directly, so a
- deployment that tuned ``parsers.markdown`` also tuned Excel imports.
- Introducing a dedicated ``parsers.excel`` section must not silently
- change that node structure, so a sectioning field absent from
- ``parsers.excel`` keeps following Markdown. Explicit ``parsers.excel``
- values always win, including one that happens to equal the class
- default.
-
- Configs built without ``from_dict`` carry no key information; those are
- treated as fully explicit so a hand-constructed ``ExcelConfig`` is never
- silently rewritten.
- """
- if markdown is None:
- return self
-
- explicit = self.explicit_keys
- if explicit is None:
- return self
-
- overrides = {
- name: getattr(markdown, name)
- for name in self._SECTIONING_FIELDS
- if hasattr(markdown, name) and name not in explicit
- }
- if not overrides:
- return self
- return replace(self, **overrides).with_explicit_keys(explicit)
+ def from_dict(cls, data: Dict[str, Any]) -> "AnydocConfig":
+ valid_fields = {f.name for f in cls.__dataclass_fields__.values()}
+ raise_unknown_config_fields(data=data, valid_fields=valid_fields, context_name=cls.__name__)
+ return cls(**data)
def validate(self) -> None:
- """Validate Excel-specific configuration."""
super().validate()
- if self.process_pool_workers < 1:
- raise ValueError("process_pool_workers must be at least 1")
+ if self.max_table_rows < 0:
+ raise ValueError("max_table_rows must be non-negative")
@dataclass
@@ -790,7 +718,7 @@ PARSER_CONFIG_REGISTRY = {
"audio": AudioConfig,
"video": VideoConfig,
"markdown": MarkdownConfig,
- "excel": ExcelConfig,
+ "anydoc": AnydocConfig,
"html": HTMLConfig,
"text": TextConfig,
"directory": DirectoryConfig,
diff --git a/pyproject.toml b/pyproject.toml
index 0f1d38aeb..0a4fb028a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -43,12 +43,7 @@ dependencies = [
"openai>=1.0.0",
"requests>=2.33.0",
"charset-normalizer>=3.4,<4",
- "python-docx>=1.0.0",
- "olefile>=0.47",
- "xlrd>=2.0.1",
- "python-pptx>=1.0.0",
- "openpyxl>=3.0.0",
- "ebooklib>=0.18.0",
+ "firecrawl-anydoc>=0.1.8",
"json-repair>=0.25.0",
"apscheduler>=3.11.0",
"volcengine>=1.0.216",
@@ -187,6 +182,7 @@ benchmark = [
"tiktoken>=0.5.0",
"datasets>=2.0.0",
"pandas>=2.0.0",
+ "python-docx>=1.0.0",
]
local-embed = [
"llama-cpp-python>=0.3.0",
diff --git a/tests/parse/test_add_directory.py b/tests/parse/test_add_directory.py
index 3325e5332..542c7db51 100644
--- a/tests/parse/test_add_directory.py
+++ b/tests/parse/test_add_directory.py
@@ -457,7 +457,7 @@ class TestParserDelegation:
@pytest.mark.asyncio
async def test_docx_file_goes_through_parser(self, tmp_path: Path, parser, fake_fs) -> None:
- """Word (.docx) files should be processed by WordParser.parse()."""
+ """Word-compatible (.docx) files should be processed by AnyDocParser.parse()."""
(tmp_path / "report.docx").write_bytes(b"PK\x03\x04")
mock_temp = fake_fs.create_temp_uri()
@@ -470,7 +470,7 @@ class TestParserDelegation:
root=ResourceNode(type=NodeType.ROOT),
source_path=str(tmp_path / "report.docx"),
source_format="docx",
- parser_name="WordParser",
+ parser_name="AnyDocParser",
parse_time=0.1,
)
fake_result.temp_dir_path = mock_temp
@@ -478,14 +478,14 @@ class TestParserDelegation:
with patch(
"openviking.parse.parsers.directory.DirectoryParser._assign_parser",
) as mock_assign:
- from openviking.parse.parsers.word import WordParser as _Word
+ from openviking.parse.parsers.anydoc import AnyDocParser as _AnyDoc
- mock_word = AsyncMock(spec=_Word)
- mock_word.parse = AsyncMock(return_value=fake_result)
+ mock_anydoc = AsyncMock(spec=_AnyDoc)
+ mock_anydoc.parse = AsyncMock(return_value=fake_result)
def assign_side_effect(cf, registry):
if cf.path.suffix == ".docx":
- return mock_word
+ return mock_anydoc
return registry.get_parser_for_file(cf.path)
mock_assign.side_effect = assign_side_effect
@@ -497,128 +497,6 @@ class TestParserDelegation:
)
assert found_md, f"report.md not found. Files: {list(fake_fs.files.keys())}"
- @pytest.mark.asyncio
- async def test_xlsx_file_goes_through_parser(self, tmp_path: Path, parser, fake_fs) -> None:
- """Excel (.xlsx) files should be processed by ExcelParser.parse()."""
- (tmp_path / "data.xlsx").write_bytes(b"PK\x03\x04")
-
- mock_temp = fake_fs.create_temp_uri()
- doc_dir = f"{mock_temp}/data"
- await fake_fs.mkdir(mock_temp)
- await fake_fs.mkdir(doc_dir)
- await fake_fs.write_file(f"{doc_dir}/data.md", "# Converted Excel")
-
- fake_result = create_parse_result(
- root=ResourceNode(type=NodeType.ROOT),
- source_path=str(tmp_path / "data.xlsx"),
- source_format="xlsx",
- parser_name="ExcelParser",
- parse_time=0.1,
- )
- fake_result.temp_dir_path = mock_temp
-
- with patch(
- "openviking.parse.parsers.directory.DirectoryParser._assign_parser",
- ) as mock_assign:
- from openviking.parse.parsers.excel import ExcelParser as _Excel
-
- mock_excel = AsyncMock(spec=_Excel)
- mock_excel.parse = AsyncMock(return_value=fake_result)
-
- def assign_side_effect(cf, registry):
- if cf.path.suffix in {".xlsx", ".xls", ".xlsm"}:
- return mock_excel
- return registry.get_parser_for_file(cf.path)
-
- mock_assign.side_effect = assign_side_effect
- await parser.parse(str(tmp_path))
-
- dir_name = tmp_path.name
- found_md = any(uri.endswith("data.md") and f"/{dir_name}/" in uri for uri in fake_fs.files)
- assert found_md, f"data.md not found. Files: {list(fake_fs.files.keys())}"
-
- @pytest.mark.asyncio
- async def test_epub_file_goes_through_parser(self, tmp_path: Path, parser, fake_fs) -> None:
- """EPub (.epub) files should be processed by EPubParser.parse()."""
- (tmp_path / "book.epub").write_bytes(b"PK\x03\x04")
-
- mock_temp = fake_fs.create_temp_uri()
- doc_dir = f"{mock_temp}/book"
- await fake_fs.mkdir(mock_temp)
- await fake_fs.mkdir(doc_dir)
- await fake_fs.write_file(f"{doc_dir}/book.md", "# Converted EPub")
-
- fake_result = create_parse_result(
- root=ResourceNode(type=NodeType.ROOT),
- source_path=str(tmp_path / "book.epub"),
- source_format="epub",
- parser_name="EPubParser",
- parse_time=0.1,
- )
- fake_result.temp_dir_path = mock_temp
-
- with patch(
- "openviking.parse.parsers.directory.DirectoryParser._assign_parser",
- ) as mock_assign:
- from openviking.parse.parsers.epub import EPubParser as _EPub
-
- mock_epub = AsyncMock(spec=_EPub)
- mock_epub.parse = AsyncMock(return_value=fake_result)
-
- def assign_side_effect(cf, registry):
- if cf.path.suffix == ".epub":
- return mock_epub
- return registry.get_parser_for_file(cf.path)
-
- mock_assign.side_effect = assign_side_effect
- await parser.parse(str(tmp_path))
-
- dir_name = tmp_path.name
- found_md = any(uri.endswith("book.md") and f"/{dir_name}/" in uri for uri in fake_fs.files)
- assert found_md, f"book.md not found. Files: {list(fake_fs.files.keys())}"
-
- @pytest.mark.asyncio
- async def test_pptx_file_goes_through_parser(self, tmp_path: Path, parser, fake_fs) -> None:
- """PowerPoint (.pptx) files should be processed by PowerPointParser.parse()."""
- (tmp_path / "slides.pptx").write_bytes(b"PK\x03\x04")
-
- mock_temp = fake_fs.create_temp_uri()
- doc_dir = f"{mock_temp}/slides"
- await fake_fs.mkdir(mock_temp)
- await fake_fs.mkdir(doc_dir)
- await fake_fs.write_file(f"{doc_dir}/slides.md", "# Converted PowerPoint")
-
- fake_result = create_parse_result(
- root=ResourceNode(type=NodeType.ROOT),
- source_path=str(tmp_path / "slides.pptx"),
- source_format="pptx",
- parser_name="PowerPointParser",
- parse_time=0.1,
- )
- fake_result.temp_dir_path = mock_temp
-
- with patch(
- "openviking.parse.parsers.directory.DirectoryParser._assign_parser",
- ) as mock_assign:
- from openviking.parse.parsers.powerpoint import PowerPointParser as _PPT
-
- mock_ppt = AsyncMock(spec=_PPT)
- mock_ppt.parse = AsyncMock(return_value=fake_result)
-
- def assign_side_effect(cf, registry):
- if cf.path.suffix == ".pptx":
- return mock_ppt
- return registry.get_parser_for_file(cf.path)
-
- mock_assign.side_effect = assign_side_effect
- await parser.parse(str(tmp_path))
-
- dir_name = tmp_path.name
- found_md = any(
- uri.endswith("slides.md") and f"/{dir_name}/" in uri for uri in fake_fs.files
- )
- assert found_md, f"slides.md not found. Files: {list(fake_fs.files.keys())}"
-
@pytest.mark.asyncio
async def test_zip_file_goes_through_parser(self, tmp_path: Path, parser, fake_fs) -> None:
"""ZIP (.zip) files should be processed by ZipParser.parse()."""
diff --git a/tests/parse/test_directory_parser_routing.py b/tests/parse/test_directory_parser_routing.py
index ca75798cf..607e0c240 100644
--- a/tests/parse/test_directory_parser_routing.py
+++ b/tests/parse/test_directory_parser_routing.py
@@ -27,14 +27,11 @@ from openviking.parse.directory_scan import (
DirectoryScanResult,
scan_directory,
)
-from openviking.parse.parsers.epub import EPubParser
-from openviking.parse.parsers.excel import ExcelParser
+from openviking.parse.parsers.anydoc import AnyDocParser
from openviking.parse.parsers.html import HTMLParser
from openviking.parse.parsers.markdown import MarkdownParser
from openviking.parse.parsers.pdf import PDFParser
-from openviking.parse.parsers.powerpoint import PowerPointParser
from openviking.parse.parsers.text import TextParser
-from openviking.parse.parsers.word import WordParser
from openviking.parse.parsers.zip_parser import ZipParser
from openviking.parse.registry import ParserRegistry
@@ -72,13 +69,22 @@ def tmp_all_parsers(tmp_path: Path) -> Path:
notes.txt -> TextParser
log.text -> TextParser
office/
- report.docx -> WordParser
- data.xlsx -> ExcelParser
- legacy.xls -> ExcelParser
- macro.xlsm -> ExcelParser
- slides.pptx -> PowerPointParser
+ legacy.doc -> AnyDocParser
+ report.docx -> AnyDocParser
+ macro.docm -> AnyDocParser
+ open.odt -> AnyDocParser
+ rich.rtf -> AnyDocParser
+ data.xlsx -> AnyDocParser
+ legacy.xls -> AnyDocParser
+ macro.xlsm -> AnyDocParser
+ binary.xlsb -> AnyDocParser
+ sheet.ods -> AnyDocParser
+ table.csv -> AnyDocParser
+ slides.pptx -> AnyDocParser
+ legacy.ppt -> AnyDocParser
+ show.pps -> AnyDocParser
books/
- book.epub -> EPubParser
+ book.epub -> AnyDocParser
archives/
bundle.zip -> ZipParser
code/
@@ -123,11 +129,20 @@ def tmp_all_parsers(tmp_path: Path) -> Path:
(tmp_path / "config" / "rules.toml").write_text("[section]", encoding="utf-8")
(tmp_path / "office").mkdir()
+ (tmp_path / "office" / "legacy.doc").write_bytes(b"\xd0\xcf\x11\xe0")
(tmp_path / "office" / "report.docx").write_bytes(b"PK\x03\x04")
+ (tmp_path / "office" / "macro.docm").write_bytes(b"PK\x03\x04")
+ (tmp_path / "office" / "open.odt").write_bytes(b"PK\x03\x04")
+ (tmp_path / "office" / "rich.rtf").write_text(r"{\rtf1 text}", encoding="utf-8")
(tmp_path / "office" / "data.xlsx").write_bytes(b"PK\x03\x04")
(tmp_path / "office" / "legacy.xls").write_bytes(b"\xd0\xcf\x11\xe0")
(tmp_path / "office" / "macro.xlsm").write_bytes(b"PK\x03\x04")
+ (tmp_path / "office" / "binary.xlsb").write_bytes(b"PK\x03\x04")
+ (tmp_path / "office" / "sheet.ods").write_bytes(b"PK\x03\x04")
+ (tmp_path / "office" / "table.csv").write_text("a,b\n1,2\n", encoding="utf-8")
(tmp_path / "office" / "slides.pptx").write_bytes(b"PK\x03\x04")
+ (tmp_path / "office" / "legacy.ppt").write_bytes(b"\xd0\xcf\x11\xe0")
+ (tmp_path / "office" / "show.pps").write_bytes(b"\xd0\xcf\x11\xe0")
(tmp_path / "books").mkdir()
(tmp_path / "books" / "book.epub").write_bytes(b"PK\x03\x04")
@@ -156,12 +171,21 @@ class TestParserSelection:
".pdf": PDFParser,
".txt": TextParser,
".text": TextParser,
- ".docx": WordParser,
- ".xlsx": ExcelParser,
- ".xls": ExcelParser,
- ".xlsm": ExcelParser,
- ".epub": EPubParser,
- ".pptx": PowerPointParser,
+ ".doc": AnyDocParser,
+ ".docx": AnyDocParser,
+ ".docm": AnyDocParser,
+ ".odt": AnyDocParser,
+ ".rtf": AnyDocParser,
+ ".xlsx": AnyDocParser,
+ ".xls": AnyDocParser,
+ ".xlsm": AnyDocParser,
+ ".xlsb": AnyDocParser,
+ ".ods": AnyDocParser,
+ ".csv": AnyDocParser,
+ ".epub": AnyDocParser,
+ ".pptx": AnyDocParser,
+ ".ppt": AnyDocParser,
+ ".pps": AnyDocParser,
".zip": ZipParser,
}
@@ -230,54 +254,6 @@ class TestParserSelection:
)
-class TestParserCanParse:
- """Parser.can_parse must accept its own supported extensions."""
-
- @pytest.mark.parametrize(
- "parser_cls,filenames",
- [
- (MarkdownParser, ["doc.md", "spec.markdown", "x.mdown", "y.mkd"]),
- (HTMLParser, ["page.html", "site.htm"]),
- (PDFParser, ["paper.pdf"]),
- (TextParser, ["notes.txt", "log.text"]),
- (WordParser, ["report.docx"]),
- (ExcelParser, ["data.xlsx", "legacy.xls", "book.xlsm"]),
- (EPubParser, ["book.epub"]),
- (PowerPointParser, ["slides.pptx"]),
- (ZipParser, ["archive.zip"]),
- ],
- )
- def test_can_parse_returns_true(self, parser_cls: type, filenames: List[str]) -> None:
- parser = parser_cls()
- for name in filenames:
- assert parser.can_parse(Path(name)), (
- f"{parser_cls.__name__}.can_parse('{name}') should be True"
- )
-
- @pytest.mark.parametrize(
- "parser_cls,filenames",
- [
- (MarkdownParser, ["file.py", "file.html", "file.pdf"]),
- (HTMLParser, ["file.md", "file.pdf", "file.txt"]),
- (PDFParser, ["file.md", "file.txt", "file.html"]),
- (TextParser, ["file.md", "file.html", "file.pdf"]),
- (WordParser, ["file.pdf", "file.xlsx", "file.txt"]),
- (ExcelParser, ["file.docx", "file.pdf", "file.txt"]),
- (EPubParser, ["file.pdf", "file.docx", "file.zip"]),
- (PowerPointParser, ["file.pdf", "file.docx", "file.txt"]),
- (ZipParser, ["file.rar", "file.pdf", "file.docx"]),
- ],
- )
- def test_can_parse_returns_false_for_wrong_extension(
- self, parser_cls: type, filenames: List[str]
- ) -> None:
- parser = parser_cls()
- for name in filenames:
- assert not parser.can_parse(Path(name)), (
- f"{parser_cls.__name__}.can_parse('{name}') should be False"
- )
-
-
# ═══════════════════════════════════════════════════════════════════════════
# Part 2 – Relative-path → Viking URI mapping
# ═══════════════════════════════════════════════════════════════════════════
diff --git a/tests/parse/test_directory_scan.py b/tests/parse/test_directory_scan.py
index 09f7ca8e8..039b6a2eb 100644
--- a/tests/parse/test_directory_scan.py
+++ b/tests/parse/test_directory_scan.py
@@ -135,7 +135,7 @@ class TestScanDirectoryClassification:
assert "readme.md" in processable_rel
assert "doc.html" in processable_rel
assert "note.txt" in processable_rel
- # Word, Excel, EPub, PowerPoint, Zip parsers
+ # AnyDoc Office/EPUB and Zip parsers
assert "report.docx" in processable_rel
assert "data.xlsx" in processable_rel
assert "book.epub" in processable_rel
diff --git a/tests/parse/test_document_parser_threading.py b/tests/parse/test_document_parser_threading.py
index 35e9165a0..e54d5021f 100644
--- a/tests/parse/test_document_parser_threading.py
+++ b/tests/parse/test_document_parser_threading.py
@@ -1,9 +1,7 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
-"""Regression tests for offloading synchronous document conversions."""
+"""Regression tests for offloading synchronous AnyDoc conversions."""
-import sys
-import zipfile
from pathlib import Path
from types import SimpleNamespace
from typing import Any, Callable
@@ -11,7 +9,8 @@ from typing import Any, Callable
import pytest
from openviking.parse.base import NodeType, ResourceNode, create_parse_result
-from openviking.parse.parsers import epub, excel, legacy_doc, powerpoint, word
+from openviking.parse.parsers import anydoc, anydoc_converter
+from openviking_cli.utils.config.parser_config import AnydocConfig
def _stub_markdown_parse(parser) -> dict[str, Any]:
@@ -49,229 +48,39 @@ def _patch_to_thread(monkeypatch, module) -> list[tuple[Callable[..., Any], tupl
return calls
+def _conversion(markdown: str = "# converted docx", source_format: str = "docx"):
+ return SimpleNamespace(
+ markdown=markdown,
+ source_format=source_format,
+ images_saved=0,
+ assets_referenced=0,
+ warnings=(),
+ )
+
+
@pytest.mark.asyncio
-async def test_word_parser_offloads_docx_conversion(monkeypatch, tmp_path: Path):
- parser = word.WordParser()
+async def test_anydoc_parser_offloads_docx_conversion(monkeypatch, tmp_path: Path):
+ parser = anydoc.AnyDocParser(anydoc_config=AnydocConfig(max_table_rows=7))
seen = _stub_markdown_parse(parser)
- calls = _patch_to_thread(monkeypatch, word)
- fake_docx = SimpleNamespace()
- monkeypatch.setitem(sys.modules, "docx", fake_docx)
+ calls = _patch_to_thread(monkeypatch, anydoc)
- def convert(path: Path, docx_module, resource_name=None, storage=None) -> str:
- assert docx_module is fake_docx
- return "# converted docx"
+ def convert(self, path: Path, *, resource_name, storage, max_table_rows=1000):
+ return _conversion()
- monkeypatch.setattr(parser, "_convert_to_markdown", convert)
+ monkeypatch.setattr(anydoc_converter.AnyDocConverter, "convert", convert)
source = tmp_path / "sample.docx"
source.write_bytes(b"placeholder")
result = await parser.parse(source)
- # #2429 added resource_name/storage to the offloaded conversion call; assert
- # the conversion was offloaded with the doc + docx module, without pinning the
- # identity of the storage singleton.
assert len(calls) == 1
- func, args, _ = calls[0]
- assert func is convert
- assert args[0] == source
- assert args[1] is fake_docx
+ func, args, call_kwargs = calls[0]
+ assert func.__func__ is convert
+ assert args == (source,)
+ assert call_kwargs["resource_name"] == "sample"
+ assert call_kwargs["max_table_rows"] == 7
+ storage = call_kwargs["storage"]
assert seen["content"] == "# converted docx"
+ assert seen["kwargs"]["allowed_media_dirs"] == [storage.media_dir]
assert result.source_format == "docx"
- assert result.parser_name == "WordParser"
-
-
-async def test_word_parser_forwards_original_name_to_markdown(monkeypatch, tmp_path: Path):
- # On single-file upload the on-disk path is a temp name (upload_<uuid>.docx)
- # and the user's original filename arrives via source_name. WordParser must
- # forward that name to MarkdownParser explicitly; otherwise parse_content can
- # only fall back to the temp path and the resource ends up named upload_<uuid>.
- parser = word.WordParser()
- seen = _stub_markdown_parse(parser)
- _patch_to_thread(monkeypatch, word)
- monkeypatch.setitem(sys.modules, "docx", SimpleNamespace())
- monkeypatch.setattr(parser, "_convert_to_markdown", lambda *args, **kwargs: "# converted docx")
-
- upload = tmp_path / "upload_abc123.docx"
- upload.write_bytes(b"placeholder")
-
- await parser.parse(upload, source_name="季度报告.docx")
-
- forwarded = seen["kwargs"].get("source_name") or seen["kwargs"].get("resource_name")
- assert forwarded == "季度报告.docx", seen["kwargs"]
-
-
-async def test_word_parser_forwards_no_split_to_markdown(monkeypatch, tmp_path: Path):
- parser = word.WordParser()
- seen = _stub_markdown_parse(parser)
- _patch_to_thread(monkeypatch, word)
- monkeypatch.setitem(sys.modules, "docx", SimpleNamespace())
- monkeypatch.setattr(parser, "_convert_to_markdown", lambda *args, **kwargs: "# converted docx")
-
- upload = tmp_path / "screenplay.docx"
- upload.write_bytes(b"placeholder")
-
- await parser.parse(upload, split_content=False)
-
- assert seen["kwargs"]["split_content"] is False
-
-
-async def test_excel_parser_offloads_xlsx_conversion(monkeypatch, tmp_path: Path):
- parser = excel.ExcelParser()
- seen = _stub_markdown_parse(parser)
- calls = _patch_to_thread(monkeypatch, excel)
- fake_openpyxl = SimpleNamespace()
- monkeypatch.setitem(sys.modules, "openpyxl", fake_openpyxl)
-
- def convert(path: Path, openpyxl_module) -> str:
- assert openpyxl_module is fake_openpyxl
- return "# converted xlsx"
-
- monkeypatch.setattr(parser, "_convert_to_markdown", convert)
- source = tmp_path / "sample.xlsx"
- source.write_bytes(b"placeholder")
-
- result = await parser.parse(source)
-
- assert calls == [(convert, (source, fake_openpyxl), {})]
- assert seen["content"] == "# converted xlsx"
- assert result.source_format == "xlsx"
- assert result.parser_name == "ExcelParser"
-
-
-async def test_excel_parser_offloads_xls_conversion(monkeypatch, tmp_path: Path):
- parser = excel.ExcelParser()
- seen = _stub_markdown_parse(parser)
- calls = _patch_to_thread(monkeypatch, excel)
-
- def convert(path: Path) -> str:
- return "# converted xls"
-
- monkeypatch.setattr(parser, "_convert_xls_to_markdown", convert)
- source = tmp_path / "sample.xls"
- source.write_bytes(b"placeholder")
-
- result = await parser.parse(source)
-
- assert calls == [(convert, (source,), {})]
- assert seen["content"] == "# converted xls"
- assert result.source_format == "xls"
- assert result.parser_name == "ExcelParser"
-
-
-async def test_powerpoint_parser_offloads_pptx_conversion(monkeypatch, tmp_path: Path):
- parser = powerpoint.PowerPointParser()
- seen = _stub_markdown_parse(parser)
- calls = _patch_to_thread(monkeypatch, powerpoint)
- fake_pptx = SimpleNamespace()
- monkeypatch.setitem(sys.modules, "pptx", fake_pptx)
-
- def convert(path: Path, pptx_module) -> str:
- assert pptx_module is fake_pptx
- return "# converted pptx"
-
- monkeypatch.setattr(parser, "_convert_to_markdown", convert)
- source = tmp_path / "sample.pptx"
- source.write_bytes(b"placeholder")
-
- result = await parser.parse(source)
-
- assert calls == [(convert, (source, fake_pptx), {})]
- assert seen["content"] == "# converted pptx"
- assert result.source_format == "pptx"
- assert result.parser_name == "PowerPointParser"
-
-
-async def test_epub_parser_offloads_epub_conversion(monkeypatch, tmp_path: Path):
- parser = epub.EPubParser()
- seen = _stub_markdown_parse(parser)
- calls = _patch_to_thread(monkeypatch, epub)
-
- def convert(path: Path) -> str:
- return "# converted epub"
-
- monkeypatch.setattr(parser, "_convert_to_markdown", convert)
- source = tmp_path / "sample.epub"
- source.write_bytes(b"placeholder")
-
- result = await parser.parse(source)
-
- assert calls == [(convert, (source,), {})]
- assert seen["content"] == "# converted epub"
- assert result.source_format == "epub"
- assert result.parser_name == "EPubParser"
-
-
-async def test_legacy_doc_parser_offloads_doc_extraction(monkeypatch, tmp_path: Path):
- parser = legacy_doc.LegacyDocParser()
- seen = _stub_markdown_parse(parser)
- calls = _patch_to_thread(monkeypatch, legacy_doc)
-
- def extract(path: Path) -> str:
- return "# converted doc"
-
- monkeypatch.setattr(parser, "_extract_text", extract)
- source = tmp_path / "sample.doc"
- source.write_bytes(b"placeholder")
-
- result = await parser.parse(source)
-
- assert calls == [(extract, (source,), {})]
- assert seen["content"] == "# converted doc"
- assert result.source_format == "doc"
- assert result.parser_name == "LegacyDocParser"
-
-
-async def test_legacy_doc_parser_routes_ooxml_payload_to_word_parser(monkeypatch, tmp_path: Path):
- parser = legacy_doc.LegacyDocParser()
- _stub_markdown_parse(parser)
- source = tmp_path / "mislabeled.doc"
- with zipfile.ZipFile(source, "w") as archive:
- archive.writestr("[Content_Types].xml", "<Types />")
- archive.writestr("word/document.xml", "<w:document />")
-
- seen: dict[str, Any] = {}
-
- async def parse_word(self, source_path, instruction="", **kwargs):
- seen["source"] = source_path
- seen["instruction"] = instruction
- seen["kwargs"] = kwargs
- return create_parse_result(
- root=ResourceNode(type=NodeType.ROOT),
- source_path=str(source_path),
- source_format="docx",
- parser_name="WordParser",
- )
-
- monkeypatch.setattr(word.WordParser, "parse", parse_word)
-
- result = await parser.parse(
- source,
- instruction="preserve tables",
- source_name="mislabeled.doc",
- )
-
- assert seen == {
- "source": source,
- "instruction": "preserve tables",
- "kwargs": {"source_name": "mislabeled.doc"},
- }
- assert result.source_format == "docx"
- assert result.parser_name == "WordParser"
-
-
-async def test_legacy_doc_parser_rejects_non_word_zip_payload(tmp_path: Path):
- source = tmp_path / "not-a-word-document.doc"
- with zipfile.ZipFile(source, "w") as archive:
- archive.writestr("payload.bin", b"binary")
-
- with pytest.raises(ValueError, match="ZIP package"):
- await legacy_doc.LegacyDocParser().parse(source)
+ assert result.parser_name == "AnyDocParser"
diff --git a/tests/parse/test_excel_process_pool.py b/tests/parse/test_excel_process_pool.py
deleted file mode 100644
index 2f376a7d5..000000000
--- a/tests/parse/test_excel_process_pool.py
+++ /dev/null
@@ -1,290 +0,0 @@
-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
-# SPDX-License-Identifier: AGPL-3.0
-"""Tests for ExcelParser._should_use_process_pool, the config-gated routing decision
-that decides whether Excel→Markdown conversion + layout planning run in a
-ProcessPoolExecutor child process instead of the main process's event loop.
-
-A real ProcessPoolExecutor is not exercised here: spawning a child process per
-test is slow and orthogonal to this decision, and the worker itself
-(_build_excel_layout_in_process) is a thin, side-effect-free function that only
-touches its own arguments. This covers the routing decision that gates it.
-"""
-
-from pathlib import Path
-
-import pytest
-
-from openviking.parse.parsers import excel as excel_module
-from openviking.parse.parsers.excel import (
- ExcelParser,
- _EXCEL_PROCESS_POOL_MIN_BYTES,
-)
-from openviking_cli.utils.config.parser_config import (
- ExcelConfig,
- MarkdownConfig,
- ParserConfig,
-)
-
-
-class TestShouldUseProcessPool:
- def _parser(self, **excel_kwargs) -> ExcelParser:
- return ExcelParser(config=ExcelConfig(**excel_kwargs))
-
- def _make_file(self, tmp_path: Path, suffix: str = ".xlsx", size: int = 10) -> Path:
- path = tmp_path / f"sheet{suffix}"
- path.write_bytes(b"x" * size)
- return path
-
- def test_disabled_by_default(self, tmp_path: Path):
- path = self._make_file(tmp_path, size=_EXCEL_PROCESS_POOL_MIN_BYTES)
- assert self._parser()._should_use_process_pool(path, {}) is False
-
- def test_xls_never_uses_process_pool(self, tmp_path: Path):
- # Legacy .xls goes through xlrd, not the openpyxl path the worker assumes.
- path = self._make_file(tmp_path, suffix=".xls", size=_EXCEL_PROCESS_POOL_MIN_BYTES)
- assert (
- self._parser(enable_process_pool=True)._should_use_process_pool(path, {})
- is False
- )
-
- @pytest.mark.parametrize(
- "kwargs",
- [
- {"enable_link_rewrite": True},
- {"base_dir": Path(".")},
- {"allowed_media_dirs": [Path(".")]},
- ],
- )
- def test_link_or_media_rewrite_kwargs_disable_process_pool(self, tmp_path, kwargs):
- # The worker never touches VikingFS/base_dir-relative media, so any parse
- # that needs link/media rewriting must stay in-process.
- path = self._make_file(tmp_path, size=_EXCEL_PROCESS_POOL_MIN_BYTES)
- assert (
- self._parser(enable_process_pool=True)._should_use_process_pool(path, kwargs)
- is False
- )
-
- def test_below_min_bytes_returns_false(self, tmp_path):
- path = self._make_file(tmp_path, size=_EXCEL_PROCESS_POOL_MIN_BYTES - 1)
- assert (
- self._parser(enable_process_pool=True)._should_use_process_pool(path, {})
- is False
- )
-
- def test_at_or_above_min_bytes_returns_true(self, tmp_path):
- path = self._make_file(tmp_path, size=_EXCEL_PROCESS_POOL_MIN_BYTES)
- assert (
- self._parser(enable_process_pool=True)._should_use_process_pool(path, {})
- is True
- )
-
-
-async def test_process_pool_forwards_no_split_layout_flag(
- monkeypatch: pytest.MonkeyPatch,
- tmp_path: Path,
-):
- path = tmp_path / "sheet.xlsx"
- path.write_bytes(b"xlsx")
- parser = ExcelParser(config=ExcelConfig(enable_process_pool=True))
- monkeypatch.setattr(
- parser._md_parser,
- "_create_temp_uri",
- lambda: "viking://temp/test/excel",
- )
- captured = {}
-
- class _CapturingLoop:
- def run_in_executor(self, _executor, callback):
- captured["callback"] = callback
- raise RuntimeError("layout callback captured")
-
- monkeypatch.setattr(excel_module.asyncio, "get_running_loop", lambda: _CapturingLoop())
- monkeypatch.setattr(excel_module, "_get_excel_layout_executor", lambda _workers: object())
-
- with pytest.raises(RuntimeError, match="layout callback captured"):
- await parser._parse_existing_path_process_pool(path, split_content=False)
-
- callback = captured["callback"]
- assert callback.keywords["layout_kwargs"]["split_content"] is False
-
-
-class TestExcelConfig:
- def test_defaults(self):
- cfg = ExcelConfig()
- assert cfg.enable_process_pool is False
- assert cfg.process_pool_workers == 2
-
- def test_validate_rejects_zero_workers(self):
- cfg = ExcelConfig(process_pool_workers=0)
- with pytest.raises(ValueError, match="process_pool_workers"):
- cfg.validate()
-
- def test_openviking_config_accepts_excel_section(self):
- from openviking_cli.utils.config.open_viking_config import OpenVikingConfig
-
- cfg = OpenVikingConfig.from_dict(
- {
- "excel": {
- "enable_process_pool": True,
- "process_pool_workers": 8,
- }
- }
- )
- assert cfg.excel.enable_process_pool is True
- assert cfg.excel.process_pool_workers == 8
-
-
-class TestExcelSectioningInheritance:
- """Excel used to be registered with ``config.markdown``.
-
- A dedicated ``parsers.excel`` section must not silently change sectioning,
- because section boundaries decide node structure and stable Viking URIs.
- """
-
- def test_unset_sectioning_follows_markdown(self):
- markdown = MarkdownConfig(max_section_size=512, max_section_chars=2000)
- resolved = ExcelConfig.from_dict({}).with_sectioning_defaults_from(markdown)
- assert resolved.max_section_size == 512
- assert resolved.max_section_chars == 2000
-
- def test_explicit_excel_values_win(self):
- markdown = MarkdownConfig(max_section_size=512, max_section_chars=2000)
- resolved = ExcelConfig.from_dict(
- {"max_section_size": 1024}
- ).with_sectioning_defaults_from(markdown)
- assert resolved.max_section_size == 1024
- # Absent from parsers.excel, so this one keeps following Markdown.
- assert resolved.max_section_chars == 2000
-
- def test_explicit_value_equal_to_default_is_not_overwritten(self):
- """"Excel keeps 2048" is a legitimate config even though 2048 is the default.
-
- Comparing against class defaults cannot distinguish it from an absent
- key, which would make the documented "explicit values win" contract
- false for exactly this case.
- """
- markdown = MarkdownConfig(max_section_size=512)
- resolved = ExcelConfig.from_dict(
- {"max_section_size": 2048}
- ).with_sectioning_defaults_from(markdown)
- assert resolved.max_section_size == 2048
-
- def test_process_pool_knobs_are_never_inherited(self):
- markdown = MarkdownConfig(max_section_size=512)
- resolved = ExcelConfig.from_dict(
- {"enable_process_pool": True, "process_pool_workers": 8}
- ).with_sectioning_defaults_from(markdown)
- assert resolved.enable_process_pool is True
- assert resolved.process_pool_workers == 8
- assert resolved.max_section_size == 512
-
- def test_default_markdown_leaves_excel_unchanged(self):
- resolved = ExcelConfig.from_dict({}).with_sectioning_defaults_from(MarkdownConfig())
- assert resolved == ExcelConfig()
-
- def test_missing_markdown_config_is_tolerated(self):
- assert ExcelConfig.from_dict({}).with_sectioning_defaults_from(None) == ExcelConfig()
-
- def test_hand_built_config_is_treated_as_fully_explicit(self):
- """A config built without from_dict carries no key provenance."""
- markdown = MarkdownConfig(max_section_size=512)
- resolved = ExcelConfig(max_section_size=2048).with_sectioning_defaults_from(markdown)
- assert resolved.max_section_size == 2048
-
- def test_provenance_does_not_affect_equality(self):
- assert ExcelConfig.from_dict({}) == ExcelConfig()
- assert ExcelConfig.from_dict({"max_section_size": 2048}) == ExcelConfig()
-
- def test_absent_excel_section_still_inherits_through_full_config(self):
- """Deployments predating parsers.excel must keep their Excel sectioning."""
- from openviking_cli.utils.config.open_viking_config import OpenVikingConfig
-
- cfg = OpenVikingConfig.from_dict({"markdown": {"max_section_size": 512}})
- resolved = cfg.excel.with_sectioning_defaults_from(cfg.markdown)
- assert resolved.max_section_size == 512
-
- def test_key_provenance_does_not_leak_into_serialization(self):
- """Provenance is bookkeeping, not config: it must not reach any output.
-
- A dataclass field would appear in asdict/model_dump and, being a
- frozenset, would break JSON serialization for callers.
- """
- import dataclasses
- import json
-
- cfg = ExcelConfig.from_dict({"max_section_size": 2048})
- dumped = dataclasses.asdict(cfg)
- assert "_explicit_fields" not in dumped
- assert not any(key.startswith("_") for key in dumped), sorted(dumped)
- json.dumps(dumped)
-
- def test_config_data_cannot_forge_key_provenance(self):
- """Provenance must not be settable from a config file."""
- with pytest.raises(ValueError):
- ExcelConfig.from_dict({"_explicit_fields": ["max_section_size"]})
-
- def test_replace_and_copy_preserve_key_provenance(self):
- import copy
- import dataclasses
-
- markdown = MarkdownConfig(max_section_size=512)
- cfg = ExcelConfig.from_dict({"max_section_size": 2048})
-
- for variant in (
- dataclasses.replace(cfg, enable_process_pool=True),
- copy.copy(cfg),
- copy.deepcopy(cfg),
- ):
- resolved = variant.with_sectioning_defaults_from(markdown)
- assert resolved.max_section_size == 2048
-
- def test_absent_excel_section_inherits_through_parser_loader(self):
- """load_parser_configs_from_dict is a second config entry point."""
- from openviking_cli.utils.config.parser_config import (
- load_parser_configs_from_dict,
- )
-
- configs = load_parser_configs_from_dict({"markdown": {"max_section_size": 512}})
- resolved = configs["excel"].with_sectioning_defaults_from(configs["markdown"])
- assert resolved.max_section_size == 512
-
- def test_explicit_excel_section_wins_through_full_config(self):
- from openviking_cli.utils.config.open_viking_config import OpenVikingConfig
-
- cfg = OpenVikingConfig.from_dict(
- {
- "markdown": {"max_section_size": 512},
- "excel": {"max_section_size": 2048, "enable_process_pool": True},
- }
- )
- resolved = cfg.excel.with_sectioning_defaults_from(cfg.markdown)
- assert resolved.max_section_size == 2048
- assert resolved.enable_process_pool is True
-
- def test_registry_resolves_excel_against_markdown(self, monkeypatch):
- from types import SimpleNamespace
-
- from openviking.parse import registry as registry_module
-
- config = SimpleNamespace(
- text=ParserConfig(),
- markdown=MarkdownConfig(max_section_size=512, max_section_chars=2000),
- pdf=ParserConfig(),
- html=ParserConfig(),
- excel=ExcelConfig.from_dict({"enable_process_pool": True}),
- image=ParserConfig(),
- )
- monkeypatch.setattr(
- "openviking_cli.utils.config.get_openviking_config",
- lambda: config,
- )
- monkeypatch.setattr(registry_module, "_default_registry", None)
-
- excel_parser = registry_module.get_registry().get_parser_for_file("book.xlsx")
-
- assert excel_parser.config.enable_process_pool is True
- assert excel_parser.config.max_section_size == 512
- # The inner MarkdownParser is what actually sections the converted sheet.
- assert excel_parser._md_parser.config.max_section_size == 512
- assert excel_parser._md_parser.config.max_section_chars == 2000
diff --git a/tests/parse/test_markdown_apply_layout.py b/tests/parse/test_markdown_apply_layout.py
index 9f1d1279b..a356a14ca 100644
--- a/tests/parse/test_markdown_apply_layout.py
+++ b/tests/parse/test_markdown_apply_layout.py
@@ -1,12 +1,14 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
-"""Tests for MarkdownParser._apply_layout.
+"""Tests for MarkdownParser layout application.
``_apply_layout`` replays ``layout.ops`` verbatim: every mkdir op runs, and each
section is written through ``_write_section`` so it keeps VikingFS's parent-dir
-and encrypted-write handling. These tests pin that contract.
+and encrypted-write handling. Concurrent parses must keep link-rewrite inputs
+isolated while replaying their layouts.
"""
+import asyncio
from unittest.mock import patch
from openviking.parse.parsers.base_parser import BaseParser
@@ -20,6 +22,7 @@ class FakeVikingFS:
self.mkdir_calls = []
self.files = {}
self.raw_write_calls = []
+ self._temp_counter = 0
async def mkdir(self, uri, exist_ok=False, **kw):
self.mkdir_calls.append(uri)
@@ -37,6 +40,27 @@ class FakeVikingFS:
# No images in this layout; let _ingest_local_images short-circuit.
return {"matches": []}
+ def create_temp_uri(self):
+ self._temp_counter += 1
+ return f"viking://temp/concurrent_{self._temp_counter}"
+
+
+class InterleavingFakeVikingFS(FakeVikingFS):
+ """Pause the first parse until a second parse enters layout application."""
+
+ def __init__(self):
+ super().__init__()
+ self._initial_mkdir_count = 0
+ self._both_parses_started = asyncio.Event()
+
+ async def mkdir(self, uri, exist_ok=False, **kw):
+ await super().mkdir(uri, exist_ok=exist_ok, **kw)
+ if self._initial_mkdir_count < 2:
+ self._initial_mkdir_count += 1
+ if self._initial_mkdir_count == 2:
+ self._both_parses_started.set()
+ await self._both_parses_started.wait()
+
class TestApplyLayout:
def _layout(self) -> _Layout:
@@ -69,14 +93,33 @@ class TestApplyLayout:
"viking://temp/root/doc/sec/b.md": "B",
"viking://temp/root/doc/other/c.md": "C",
}
+ assert fake.raw_write_calls == []
- async def test_sections_go_through_write_section_not_raw_write(self):
- """Section writes must keep VikingFS parent-dir/encrypted-write handling."""
- fake = FakeVikingFS()
+ async def test_concurrent_parses_keep_link_rewrite_inputs_isolated(self, tmp_path):
+ fake = InterleavingFakeVikingFS()
parser = MarkdownParser()
+
+ roots = {}
+ for name in ("a", "b"):
+ root = tmp_path / name
+ root.mkdir()
+ (root / f"{name}.md").write_text("source", encoding="utf-8")
+ (root / f"{name}.txt").write_text("target", encoding="utf-8")
+ roots[name] = root
+
+ async def parse(name):
+ root = roots[name]
+ return await parser.parse_content(
+ f"[target](./{name}.txt)",
+ source_path=str(root / f"{name}.md"),
+ base_dir=root,
+ enable_link_rewrite=True,
+ link_rewrite_root=str(root),
+ )
+
with patch.object(BaseParser, "_get_viking_fs", return_value=fake):
- await parser._apply_layout(self._layout())
+ results = await asyncio.gather(parse("a"), parse("b"))
- assert fake.raw_write_calls == [], (
- f"Section writes must not bypass write_file: {fake.raw_write_calls}"
- )
+ for name, result in zip(("a", "b"), results, strict=True):
+ uri = f"{result.temp_dir_path}/{name}/{name}.md"
+ assert fake.files[uri] == f"[target](../{name}.txt)"
diff --git a/tests/parse/test_markdown_link_rewrite.py b/tests/parse/test_markdown_link_rewrite.py
index 9c0ad23fb..f878a2f1b 100644
--- a/tests/parse/test_markdown_link_rewrite.py
+++ b/tests/parse/test_markdown_link_rewrite.py
@@ -9,7 +9,7 @@ import pytest
from openviking.parse.parsers.base_parser import BaseParser
from openviking.parse.parsers.directory import DirectoryParser
-from openviking.parse.parsers.markdown import MarkdownParser
+from openviking.parse.parsers.markdown import MarkdownParser, _RewriteContext
class TestRewriteRelativeLinks:
@@ -28,12 +28,21 @@ class TestRewriteRelativeLinks:
return kb
async def _rewrite(self, parser, kb, content, section_subpath=""):
- return await parser._rewrite_relative_links(
- content,
+ rewrite_ctx = _RewriteContext(
source_path=str(kb / "文档.md"),
doc_name="文档",
- section_subpath=section_subpath,
+ root_dir="viking://temp/test/文档",
import_root=str(kb),
+ base_dir=kb,
+ allowed_media_dirs=None,
+ split_content=True,
+ adaptive_flatten_requested=False,
+ flatten_single_output=False,
+ )
+ return await parser._rewrite_relative_links(
+ content,
+ section_subpath=section_subpath,
+ rewrite_ctx=rewrite_ctx,
)
async def test_md_target_becomes_directory(self, tmp_path: Path):
@@ -172,7 +181,8 @@ class TestRewriteRelativeLinks:
kb = self._make_tree(tmp_path)
p = self._parser()
- async def fake_bare_layout(_path): # 模拟未来:目标入库为单个裸文件,无 <dir>/ 包裹
+ async def fake_bare_layout(_path, _context):
+ # 模拟未来:目标入库为单个裸文件,无 <dir>/ 包裹
return {"文档.md": "# 目标\n\n内容"}
p._target_split_files = fake_bare_layout # type: ignore[method-assign]
diff --git a/tests/test_config_loader.py b/tests/test_config_loader.py
index 9750c64f3..7977b9fcf 100644
--- a/tests/test_config_loader.py
+++ b/tests/test_config_loader.py
@@ -206,9 +206,10 @@ def test_generic_code_hosting_domains_load_from_config():
assert config.code_hosting_domains == ["git.generic.example.com"]
-def test_openviking_config_rejects_unknown_nested_parser_section(monkeypatch):
+def test_openviking_config_handles_nested_parser_compatibility(monkeypatch):
monkeypatch.setenv(OPENVIKING_CONFIG_ENV, "/tmp/codex-no-config.json")
+ from openviking_cli.utils.config import open_viking_config as config_module
from openviking_cli.utils.config.open_viking_config import (
OpenVikingConfig,
OpenVikingConfigSingleton,
@@ -228,6 +229,28 @@ def test_openviking_config_rejects_unknown_nested_parser_section(monkeypatch):
}
)
+ errors: list[str] = []
+ monkeypatch.setattr(
+ config_module._get_config_logger(),
+ "error",
+ lambda message, *args, **kwargs: errors.append(message % args if args else message),
+ )
+ config = OpenVikingConfig.from_dict(
+ {
+ "embedding": {
+ "dense": {
+ "provider": "openai",
+ "api_key": "test-key",
+ "model": "text-embedding-3-small",
+ }
+ },
+ "parsers": {"excel": {"enable_process_pool": True}},
+ }
+ )
+
+ assert config.anydoc.enabled is True
+ assert any("Config field 'parsers.excel' was removed and is ignored" in item for item in errors)
+
OpenVikingConfigSingleton.reset_instance()
diff --git a/uv.lock b/uv.lock
index 988610c7b..1a1d267d5 100644
--- a/uv.lock
+++ b/uv.lock
@@ -1201,19 +1201,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/b0/0d/9feae160378a3553fa9a339b0e9c1a048e147a4127210e286ef18b730f03/durationpy-0.10-py3-none-any.whl", hash = "sha256:3b41e1b601234296b4fb368338fdcd3e13e0b4fb5b67345948f4f2bf9868b286", size = 3922, upload-time = "2025-05-17T13:52:36.463Z" },
]
-[[package]]
-name = "ebooklib"
-version = "0.20"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
- { name = "lxml" },
- { name = "six" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/77/85/322e8882a582d4b707220d1929cfb74c125f2ba513991edbce40dbc462de/ebooklib-0.20.tar.gz", hash = "sha256:35e2f9d7d39907be8d39ae2deb261b19848945903ae3dbb6577b187ead69e985", size = 127066, upload-time = "2025-10-26T20:56:20.968Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/bf/ee/aa015c5de8b0dc42a8e507eae8c2de5d1c0e068c896858fec6d502402ed6/ebooklib-0.20-py3-none-any.whl", hash = "sha256:fff5322517a37e31c972d27be7d982cc3928c16b3dcc5fd7e8f7c0f5d7bcf42b", size = 40995, upload-time = "2025-10-26T20:56:19.104Z" },
-]
-
[[package]]
name = "ecdsa"
version = "0.19.2"
@@ -1226,15 +1213,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/51/79/119091c98e2bf49e24ed9f3ae69f816d715d2904aefa6a2baa039a2ba0b0/ecdsa-0.19.2-py2.py3-none-any.whl", hash = "sha256:840f5dc5e375c68f36c1a7a5b9caad28f95daa65185c9253c0c08dd952bb7399", size = 150818, upload-time = "2026-03-26T09:58:15.808Z" },
]
-[[package]]
-name = "et-xmlfile"
-version = "2.0.0"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/d3/38/af70d7ab1ae9d4da450eeec1fa3918940a5fafb9055e934af8d6eb0c2313/et_xmlfile-2.0.0.tar.gz", hash = "sha256:dab3f4764309081ce75662649be815c4c9081e88f0837825f90fd28317d4da54", size = 17234, upload-time = "2024-10-25T17:25:40.039Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/c1/8b/5fe2cc11fee489817272089c4203e679c63b570a5aaeb18d852ae3cbba6a/et_xmlfile-2.0.0-py3-none-any.whl", hash = "sha256:7a91720bc756843502c3b7504c77b8fe44217c85c537d85037f0f536151b2caa", size = 18059, upload-time = "2024-10-25T17:25:39.051Z" },
-]
-
[[package]]
name = "exceptiongroup"
version = "1.3.1"
@@ -1365,6 +1343,21 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/f9/0b/de6f54d4a8bedfe8645c41497f3c18d749f0bd3218170c667bf4b81d0cdd/filelock-3.25.0-py3-none-any.whl", hash = "sha256:5ccf8069f7948f494968fc0713c10e5c182a9c9d9eef3a636307a20c2490f047", size = 26427, upload-time = "2026-03-01T15:08:44.593Z" },
]
+[[package]]
+name = "firecrawl-anydoc"
+version = "0.1.9"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/aa/16/e37d4f284482a4f30e0502ce439a9fd8ccf013e6ae4438b4dfa209e48750/firecrawl_anydoc-0.1.9.tar.gz", hash = "sha256:0dfc64b82b4e971143dd6f0e4f39f8152fcc38be6523663682a241d8f3301b14", size = 197914, upload-time = "2026-08-13T21:48:27.989Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/42/37/866bf6270ad8552eaf6edf3b8e1a360acf3e8764140641361ae7b440514e/firecrawl_anydoc-0.1.9-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:ebd956831822d3ea1831253406736e0e4d9b217e3471eb68c9bf2eba9302ce73", size = 3469321, upload-time = "2026-08-13T21:48:16.775Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/9a/8fbe0726cdf69154e1eb05c119332b1fdfd8200eeafcfd1675741ac5f03d/firecrawl_anydoc-0.1.9-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:a2372d97826e8e7e68fd28cd415d08815f6aa66dcc09c6b423ac33dd2dfa91cf", size = 3287520, upload-time = "2026-08-13T21:48:18.37Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/ce/86a626224c365d8de65699d6ac8f4626ac05e7e75c3ae76c82f4904a07d2/firecrawl_anydoc-0.1.9-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ba1c19bdaa97f4850daf3fb4ca2855cefa146b5621ce7b22a71d158d3b84e54e", size = 3331094, upload-time = "2026-08-13T21:48:19.961Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/28/00d1d48fe205ab66eda4a2a22cce44aadf586ef1495e703349b791c8de32/firecrawl_anydoc-0.1.9-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:deea7a270fccda9be29c73235e938ca32e0f703e1ae120f08bf0fdb39301c6b4", size = 3555945, upload-time = "2026-08-13T21:48:21.822Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/25/ac52d98c5d082ef92396b2fcb398745a3b054816c67a12d19de86c1ba535/firecrawl_anydoc-0.1.9-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:de0858decd18188b0b88545689544fb3703ec08ade37f63c2857e4414e99bdf1", size = 3510861, upload-time = "2026-08-13T21:48:23.369Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/d3/e80314b5746c4ca85150f3ab2a527b2167012c16e706e148733211cb88b3/firecrawl_anydoc-0.1.9-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:e4e1765ed5574fa502931f8bfaec1f87af913afa5bb432aaaf5f9c094efc7f43", size = 3797896, upload-time = "2026-08-13T21:48:25.061Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/d0/e7b35c2365498d5e9f1bdb4b3ded78571b6dafbaba1880538490d27b71c9/firecrawl_anydoc-0.1.9-cp310-abi3-win_amd64.whl", hash = "sha256:aa6a5ca2e10939a87c9bd21c918ef8b146ad0061fedae661d4483ef5a5bbdf60", size = 3642084, upload-time = "2026-08-13T21:48:26.702Z" },
+]
+
[[package]]
name = "frozenlist"
version = "1.8.0"
@@ -3409,15 +3402,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/be/9c/92789c596b8df838baa98fa71844d84283302f7604ed565dafe5a6b5041a/oauthlib-3.3.1-py3-none-any.whl", hash = "sha256:88119c938d2b8fb88561af5f6ee0eec8cc8d552b7bb1f712743136eb7523b7a1", size = 160065, upload-time = "2025-06-19T22:48:06.508Z" },
]
-[[package]]
-name = "olefile"
-version = "0.47"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/69/1b/077b508e3e500e1629d366249c3ccb32f95e50258b231705c09e3c7a4366/olefile-0.47.zip", hash = "sha256:599383381a0bf3dfbd932ca0ca6515acd174ed48870cbf7fee123d698c192c1c", size = 112240, upload-time = "2023-12-01T16:22:53.025Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/17/d3/b64c356a907242d719fc668b71befd73324e47ab46c8ebbbede252c154b2/olefile-0.47-py2.py3-none-any.whl", hash = "sha256:543c7da2a7adadf21214938bb79c83ea12b473a4b6ee4ad4bf854e7715e13d1f", size = 114565, upload-time = "2023-12-01T16:22:51.518Z" },
-]
-
[[package]]
name = "openai"
version = "2.24.0"
@@ -3454,18 +3438,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/17/af/0379ba45fd2f4c14be3b62a78714b1b34705d0472cdd8d62df47a8fb1ffd/opencode_ai-0.1.0a36-py3-none-any.whl", hash = "sha256:73bce046ec3d841dadb31bed658a5049f25cf0c2e3ebda2e2d1ff37d792ea1b5", size = 110221, upload-time = "2025-08-27T02:38:57.501Z" },
]
-[[package]]
-name = "openpyxl"
-version = "3.1.5"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
- { name = "et-xmlfile" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/3d/f9/88d94a75de065ea32619465d2f77b29a0469500e99012523b91cc4141cd1/openpyxl-3.1.5.tar.gz", hash = "sha256:cf0e3cf56142039133628b5acffe8ef0c12bc902d2aadd3e0fe5878dc08d1050", size = 186464, upload-time = "2024-06-28T14:03:44.161Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/c0/da/977ded879c29cbd04de313843e76868e6e13408a94ed6b987245dc7c8506/openpyxl-3.1.5-py2.py3-none-any.whl", hash = "sha256:5282c12b107bffeef825f4617dc029afaf41d0ea60823bbb665ef3079dc79de2", size = 250910, upload-time = "2024-06-28T14:03:41.161Z" },
-]
-
[[package]]
name = "opensandbox"
version = "0.1.5"
@@ -3640,9 +3612,9 @@ dependencies = [
{ name = "charset-normalizer" },
{ name = "cryptography" },
{ name = "defusedxml" },
- { name = "ebooklib" },
{ name = "fastapi" },
{ name = "feedparser" },
+ { name = "firecrawl-anydoc" },
{ name = "grep-ast" },
{ name = "httpx" },
{ name = "jinja2" },
@@ -3651,9 +3623,7 @@ dependencies = [
{ name = "litellm" },
{ name = "loguru" },
{ name = "mcp" },
- { name = "olefile" },
{ name = "openai" },
- { name = "openpyxl" },
{ name = "opentelemetry-api" },
{ name = "opentelemetry-exporter-otlp-proto-grpc" },
{ name = "opentelemetry-exporter-otlp-proto-http" },
@@ -3665,9 +3635,7 @@ dependencies = [
{ name = "pdfplumber" },
{ name = "protobuf" },
{ name = "pydantic" },
- { name = "python-docx" },
{ name = "python-multipart" },
- { name = "python-pptx" },
{ name = "pyyaml" },
{ name = "requests" },
{ name = "scrapy" },
@@ -3691,7 +3659,6 @@ dependencies = [
{ name = "uvicorn" },
{ name = "volcengine" },
{ name = "volcengine-python-sdk", extra = ["ark"] },
- { name = "xlrd" },
{ name = "xxhash" },
]
@@ -3708,6 +3675,7 @@ benchmark = [
{ name = "langchain-openai" },
{ name = "pandas", version = "2.3.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },
{ name = "pandas", version = "3.0.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" },
+ { name = "python-docx" },
{ name = "tiktoken" },
]
bot = [
@@ -3823,9 +3791,9 @@ requires-dist = [
{ name = "defusedxml", specifier = ">=0.7.1" },
{ name = "diff-match-patch", marker = "extra == 'test'", specifier = ">=20200713" },
{ name = "dingtalk-stream", marker = "extra == 'bot'", specifier = ">=0.4.0" },
- { name = "ebooklib", specifier = ">=0.18.0" },
{ name = "fastapi", specifier = ">=0.128.0" },
{ name = "feedparser", specifier = ">=6.0.0" },
+ { name = "firecrawl-anydoc", specifier = ">=0.1.8" },
{ name = "fusepy", marker = "extra == 'bot'", specifier = ">=3.0.1" },
{ name = "google-genai", marker = "extra == 'gemini'", specifier = ">=1.0.0" },
{ name = "google-genai", marker = "extra == 'gemini-async'", specifier = ">=1.0.0" },
@@ -3852,10 +3820,8 @@ requires-dist = [
{ name = "msgpack", marker = "extra == 'bot'", specifier = ">=1.0.8" },
{ name = "mypy", marker = "extra == 'dev'", specifier = ">=1.0.0" },
{ name = "myst-parser", marker = "extra == 'doc'", specifier = ">=2.0.0" },
- { name = "olefile", specifier = ">=0.47" },
{ name = "openai", specifier = ">=1.0.0" },
{ name = "opencode-ai", marker = "extra == 'bot'", specifier = ">=0.1.0a0" },
- { name = "openpyxl", specifier = ">=3.0.0" },
{ name = "opensandbox", marker = "extra == 'bot'", specifier = ">=0.1.0" },
{ name = "opensandbox-server", marker = "extra == 'bot'", specifier = ">=0.1.0" },
{ name = "opentelemetry-api", specifier = ">=1.14" },
@@ -3881,13 +3847,12 @@ requires-dist = [
{ name = "pytest-asyncio", marker = "extra == 'test'", specifier = ">=0.21.0" },
{ name = "pytest-cov", marker = "extra == 'test'", specifier = ">=4.0.0" },
{ name = "pytest-xdist", marker = "extra == 'test'", specifier = ">=3.5.0" },
- { name = "python-docx", specifier = ">=1.0.0" },
+ { name = "python-docx", marker = "extra == 'benchmark'", specifier = ">=1.0.0" },
{ name = "python-engineio", marker = "extra == 'bot'", specifier = ">=4.13.2" },
{ name = "python-jose", extras = ["cryptography"], marker = "extra == 'auth'", specifier = ">=3.3.0" },
{ name = "python-jose", extras = ["cryptography"], marker = "extra == 'bot'", specifier = ">=3.3.0" },
{ name = "python-ldap", marker = "extra == 'auth'", specifier = ">=3.4.0" },
{ name = "python-multipart", specifier = ">=0.0.31" },
- { name = "python-pptx", specifier = ">=1.0.0" },
{ name = "python-socketio", marker = "extra == 'bot'", specifier = ">=5.16.2" },
{ name = "python-socks", extras = ["asyncio"], marker = "extra == 'bot'", specifier = ">=2.4.0" },
{ name = "python-telegram-bot", extras = ["socks"], marker = "extra == 'bot'", specifier = ">=21.0" },
@@ -3932,7 +3897,6 @@ requires-dist = [
{ name = "websocket-client", marker = "extra == 'bot'", specifier = ">=1.6.0" },
{ name = "websockets", marker = "extra == 'bot'", specifier = ">=12.0" },
{ name = "wheel", marker = "extra == 'build'" },
- { name = "xlrd", specifier = ">=2.0.1" },
{ name = "xxhash", specifier = ">=3.0.0" },
]
provides-extras = ["test", "auth", "dev", "doc", "eval", "gemini", "gemini-async", "ocr", "build", "bot", "benchmark", "local-embed"]
@@ -5130,21 +5094,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/e1/04/e8135ebd1ad02c56ec633277529b2602ff99ff634be76cdba5744cf554fd/python_multipart-0.0.32-py3-none-any.whl", hash = "sha256:ff6d3f776f16878c894e52e107296ffc890e913c611b1a4ec6c44e2821fe2e23", size = 30042, upload-time = "2026-06-04T16:18:57.319Z" },
]
-[[package]]
-name = "python-pptx"
-version = "1.0.2"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
- { name = "lxml" },
- { name = "pillow" },
- { name = "typing-extensions" },
- { name = "xlsxwriter" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/52/a9/0c0db8d37b2b8a645666f7fd8accea4c6224e013c42b1d5c17c93590cd06/python_pptx-1.0.2.tar.gz", hash = "sha256:479a8af0eaf0f0d76b6f00b0887732874ad2e3188230315290cd1f9dd9cc7095", size = 10109297, upload-time = "2024-08-07T17:33:37.772Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/d9/4f/00be2196329ebbff56ce564aa94efb0fbc828d00de250b1980de1a34ab49/python_pptx-1.0.2-py3-none-any.whl", hash = "sha256:160838e0b8565a8b1f67947675886e9fea18aa5e795db7ae531606d68e785cba", size = 472788, upload-time = "2024-08-07T17:33:28.192Z" },
-]
-
[[package]]
name = "python-socketio"
version = "5.16.3"
@@ -7047,24 +6996,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/a4/f5/10b68b7b1544245097b2a1b8238f66f2fc6dcaeb24ba5d917f52bd2eed4f/wsproto-1.3.2-py3-none-any.whl", hash = "sha256:61eea322cdf56e8cc904bd3ad7573359a242ba65688716b0710a5eb12beab584", size = 24405, upload-time = "2025-11-20T18:18:00.454Z" },
]
-[[package]]
-name = "xlrd"
-version = "2.0.2"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/07/5a/377161c2d3538d1990d7af382c79f3b2372e880b65de21b01b1a2b78691e/xlrd-2.0.2.tar.gz", hash = "sha256:08b5e25de58f21ce71dc7db3b3b8106c1fa776f3024c54e45b45b374e89234c9", size = 100167, upload-time = "2025-06-14T08:46:39.039Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/1a/62/c8d562e7766786ba6587d09c5a8ba9f718ed3fa8af7f4553e8f91c36f302/xlrd-2.0.2-py2.py3-none-any.whl", hash = "sha256:ea762c3d29f4cca48d82df517b6d89fbce4db3107f9d78713e48cd321d5c9aa9", size = 96555, upload-time = "2025-06-14T08:46:37.766Z" },
-]
-
-[[package]]
-name = "xlsxwriter"
-version = "3.2.9"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/46/2c/c06ef49dc36e7954e55b802a8b231770d286a9758b3d936bd1e04ce5ba88/xlsxwriter-3.2.9.tar.gz", hash = "sha256:254b1c37a368c444eac6e2f867405cc9e461b0ed97a3233b2ac1e574efb4140c", size = 215940, upload-time = "2025-09-16T00:16:21.63Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/3a/0c/3662f4a66880196a590b202f0db82d919dd2f89e99a27fadef91c4a33d41/xlsxwriter-3.2.9-py3-none-any.whl", hash = "sha256:9a5db42bc5dff014806c58a20b9eae7322a134abb6fce3c92c181bfb275ec5b3", size = 175315, upload-time = "2025-09-16T00:16:20.108Z" },
-]
-
[[package]]
name = "xxhash"
version = "3.6.0"