Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

refactor(parse): 收口资源解析路由 (#3295)

Qin Haojie2be4bb48798f
diff --git a/docs/design/parser-two-layer-refactor-plan.md b/docs/design/parser-two-layer-refactor-plan.md
index 0c9433156..25f01add8 100644
--- a/docs/design/parser-two-layer-refactor-plan.md
+++ b/docs/design/parser-two-layer-refactor-plan.md
@@ -1,5 +1,7 @@
# OpenViking 解析器两层架构重构
+> 本文是 Accessor / Parser 两层拆分的历史重构记录。当前 `add_resource` 的完整入口分流、Understanding、Connector 与异步执行规则,见 [添加资源后的解析路由](./resource-ingestion-routing.md)。
+
| 项目 | 信息 |
|-----|------|
| 状态 | `已完成` |
diff --git a/docs/design/resource-ingestion-routing.md b/docs/design/resource-ingestion-routing.md
new file mode 100644
index 000000000..d0ce5b3f6
--- /dev/null
+++ b/docs/design/resource-ingestion-routing.md
@@ -0,0 +1,310 @@
+# 添加资源后的解析路由
+
+本文描述当前 `add_resource` 从收到资源到落盘、建索引的真实执行链。重点回答四个问题:入口在哪里分流、资源类型在哪里确定、Understanding 与 Connector 分别做什么,以及 `wait` 到底等待什么。
+
+## 先记住五条规则
+
+1. 对外只有一个资源添加入口:`ResourceService.add_resource`。SDK、HTTP API、MCP 最终都应调用它;Worker 不再拿后台任务冒充一次新的资源添加请求。
+2. Connector 是一条独立的端到端导入链;Understanding 只是标准链里的一个 Parser 后端。
+3. 普通文件只做一次 Parser 选择。选择依据是 Accessor 获取资源后冻结的 `resolved_extension`,不是临时文件名,也不会在队列消费者里重新猜;原始飞书 URL 是一个显式例外,可在 Accessor 前按配置直达 Understanding。
+4. 目录、网站目录以及内部 Parser 遍历出的子文件不逐个调用 Understanding,统一留在内置 Parser 链内处理。
+5. Watch 是一次新的来源刷新,会重新走提交分流,但不会创建、取消或覆盖自身的 Watch 任务。
+
+## 总流程
+
+```text
+SDK / HTTP API / MCP
+ |
+ v
+ResourceService.add_resource 对外入口
+ |
+ v
+ResourceService._submit_resource_ingestion 阶段一:对新请求选择执行方式
+ |
+ +-- Connector 命中且参数受支持 ------> Connector doc/add
+ | |
+ | +--> 轮询 Connector task/info
+ | +--> Connector 自己解析并写入资源
+ |
+ +-- Git && wait=false ----------------> 预检 + AddResource 队列 --> 返回 task_id
+ |
+ +-- HTTP 服务远程资源 && wait=false && Understanding 已启用
+ | |
+ | +-- 原始 URL 可直达 -----------> 提交 Understanding --> ExternalParse 队列
+ | | |
+ | | +--> 返回 task_id --> Worker
+ | |
+ | +-- 需要先确定本地类型 --------> Accessor 下载并识别
+ | |
+ | +-- 命中 Understanding --> 上传同一文件 --> ExternalParse 队列
+ | | |
+ | | +--> 返回 task_id --> Worker
+ | |
+ | +-- 未命中 ------------> 复用 LocalResource,进入当前请求标准链
+ |
+ +-- 其余场景(包括所有 wait=true) -----> _execute_resource_ingestion
+
+_execute_resource_ingestion 阶段二:执行标准链
+ |
+ v
+ResourceProcessor.process_resource
+ |
+ v
+UnifiedResourceProcessor
+ |
+ +-- 已有 understanding_response_id ----> ParserRouter --> Understanding 恢复任务
+ |
+ +-- 原始 URL 可直达 Understanding -----> ParserRouter --> Understanding 提交并等待
+ |
+ +-- 原始文本 --------------------------> 内置 ParserRegistry
+ |
+ +-- 路径 / URL --> AccessorRegistry.access(选择数据访问器)
+ |
+ v
+ LocalResource
+ |
+ +-- 目录 --> DirectoryParser(内置)
+ |
+ +-- 文件 --> ParserRouter.parse(选择 Parser,只选一次)
+ |
+ +-- 内置 ParserRegistry
+ +-- Understanding 上传并等待
+ |
+ v
+ ParseResult --> TreeBuilder 落盘
+ |
+ v
+阶段三:返回策略
+ |
+ +-- wait=true --> 继续等待摘要 / 语义队列 / 向量索引后返回
+ |
+ +-- wait=false --> 普通标准链落盘后进入 AddResource 队列并返回
+
+后台队列 Worker
+ |
+ +-- source job -----------------------> _execute_resource_ingestion(wait=true)
+ | 使用消息中冻结的 backend / response_id / extension
+ |
+ +-- prepared job ---------------------> finish_prepared_resource
+ |
+ +-- 不再调用 add_resource,也不再重做 Connector / Git 顶层分类
+
+WatchScheduler
+ |
+ +-- refresh_resource -----------------> _submit_resource_ingestion(manage_watch=false)
+ 重新获取和解析来源,但不修改 Watch 任务
+```
+
+Understanding 不受 `wait=false` 限制。`wait=true` 在当前请求内完成 Understanding 提交、轮询、解析和 TreeBuilder,并继续等待后续队列;`wait=false` 的远程服务路径会先提交 Understanding、将 `response_id` 入队,再由 Worker 恢复任务。
+
+分类发生的位置:
+
+| 分类内容 | 代码位置 | 输出 |
+|---|---|---|
+| Connector、异步 Git、标准执行链 | `ResourceService._submit_resource_ingestion` | 选定新请求的顶层执行方式 |
+| 异步 Understanding 或当前请求内执行 | `ResourceService._execute_resource_ingestion` | 根据 `wait`、配置和已识别类型决定提交还是同步执行 |
+| Feishu、Git、Feed、HTTP、本地文件 | `AccessorRegistry.access` | `LocalResource` |
+| Understanding 能否直接接收原始 URL | `ParserRouter.should_use_understanding_directly` | 直达 Understanding 或继续 Accessor |
+| Understanding 与内置 Parser | `ParserRouter.parse` | `ParseResult` |
+
+后台资源任务统一使用可恢复的 `AddResourceMsg`,但按工作类型进入两个独立队列:Understanding 使用受外部解析并发限制的 `ExternalParse`;Git 和已落盘的本地后处理使用 `AddResource`。两个队列都由 `AddResourceProcessor` 消费,锁接管失败时仍回到原队列。消息已经冻结了生产端做出的选择,例如 `parser_backend`、`understanding_response_id` 和 `resolved_extension`。Worker 直接执行该选择,不再把任务送回公开 `add_resource` 重新分类。
+
+## 顶层路由表
+
+| 输入或场景 | 获取数据 | 解析者 | 是否走标准 `ParseResult -> TreeBuilder` | 返回时机 |
+|---|---|---|---|---|
+| Connector 配置允许的 TOS 或 Git,提供精确 `to` 且参数受支持 | Connector 服务 | Connector 服务 | 否 | 提交成功立即返回 `task_id`,后台轮询状态 |
+| `tos://` 但 Connector 不可用或参数不支持 | 不降级 | 不执行 | 否 | 直接报清晰的参数或配置错误 |
+| Git 未命中 Connector 或无凭证回退,`wait=false` | GitAccessor 在后台 clone | 内置目录/代码仓库 Parser | 是 | 预检仓库并预占 URI 后返回 |
+| Git,`wait=true` | GitAccessor | 内置目录/代码仓库 Parser | 是 | 解析、落盘及语义队列完成后返回 |
+| 飞书 URL,`parser_api.enable_feishu_url=true` 且有 user/app 凭证 | Understanding 直接读取飞书 | Understanding | 是 | `wait=false` 时提交并入队;`wait=true` 时同步解析 |
+| 飞书 URL,直达配置关闭或无可用凭证 | FeishuAccessor | 内置 Markdown Parser | 是 | Accessor 拉取、归一化后走标准链 |
+| HTTP 服务请求,`wait=false`,命中 Understanding | HTTPAccessor 识别类型并上传同一份本地文件 | Understanding | 是 | 类型识别、Understanding 提交、URI 预占和入队后返回 |
+| 其他 URL、文件、目录、原始文本 | 对应 Accessor;原始文本无需 Accessor | 内置 Parser 或同步 Understanding | 是 | 至少完成解析和落盘后返回 |
+
+Git 是 Connector 与标准链共享的来源:未命中 Connector 或参数不受支持时可回退到标准链;一旦请求带有 Connector 专用凭证则禁止回退,避免凭证进入持久化队列。`tos://` 没有标准 Accessor,不能回退,否则只会在更深处得到误导性的解析错误。
+
+## Accessor:先把“数据在哪”变成“本地是什么”
+
+`AccessorRegistry` 按优先级选择数据访问器,当前内置顺序是:
+
+```text
+FeishuAccessor (100)
+GitAccessor (80)
+WebFeedAccessor (60)
+HTTPAccessor (50)
+LocalAccessor (1)
+```
+
+Accessor 的产物统一是 `LocalResource`,包含本地文件或目录路径、`source_type`、原始来源、是否需要清理,以及检测元数据。标准 Parser 不再负责 clone 或下载;只有明确开启的飞书 Understanding 直达链会在 Accessor 前消费原始 URL。
+
+### 飞书资源
+
+飞书是“远程私有数据源”,不是一种本地文件格式。默认使用 `FeishuAccessor` 获取并归一化:
+
+```text
+飞书 URL
+ |
+ v
+FeishuAccessor --调用飞书 OpenAPI--> Markdown + 下载后的图片
+ |
+ v
+LocalResource(document.md)
+ |
+ v
+ParserRouter --> MarkdownParser --> ParseResult --> TreeBuilder
+```
+
+当 `parser_api.enable=true`、`parser_api.enable_feishu_url=true`,并且本次请求有 `args.feishu_access_token` 或服务端已配置飞书应用凭证时,原始飞书 URL 会在 Accessor 前进入统一的 Understanding 链。否则进入上面的 FeishuAccessor 路径。
+
+`enable_feishu_url` 默认是 `false`。Understanding 一旦被选中,提交、轮询或解析失败会直接返回错误,不会在运行时自动重试 FeishuAccessor。
+
+`FeishuAccessor` 预期支持范围如下:
+
+| URL 类型 | Accessor 行为 | 当前边界 |
+|---|---|---|
+| `docx/{token}` | 拉取文档 block,转换标题、列表、代码、表格、图片和内嵌 sheet | 图片下载受 `download_images` 控制;内嵌 sheet 最多读取 100 行、A-Z 列 |
+| `sheets/{token}` | 枚举所有 sheet;普通网格读取单元格,内嵌多维表格根据 `blockInfo` 转走 Bitable API,统一输出 Markdown 表格 | 普通网格最多读取 `max_rows_per_sheet` 行、A-Z 列;内嵌多维表格继承 Base 的记录和图片处理能力 |
+| `base/{app_token}` | 无查询参数时枚举全部数据表;`table` 限定数据表,`table` + `view` 限定视图 | 表和字段完整翻页;每张表最多读取 `max_records_per_table` 条记录;附件字段中的图片会下载,其他附件保留文件名 |
+| `wiki/{token}` | 先解析 wiki 节点的实际类型和 token,再进入上述 `docx`、`sheets` 或 `base` 处理器 | wiki 指向其他飞书对象类型时明确报不支持 |
+
+四类入口都支持应用凭证获取的 tenant token;显式传入 `args.feishu_access_token` 时,同一 user token 会用于本次选中的飞书链路。Accessor 路径中,它用于 wiki 解析、正文、sheet/base 和图片请求;Understanding 直达路径中,它作为 `lark_file.user_access_token` 提交,随后从后台队列参数中删除。
+
+飞书专有的本地归一化逻辑到 `FeishuAccessor` 为止,后面只处理 Markdown,因此不再保留并行的本地 `FeishuParser` 入口。即使 `parser_api.extensions` 包含 `md`,`SourceType.FEISHU` 的 `LocalResource` 也固定使用内置 Markdown Parser,不会把已经拉取的数据再次送 Understanding。传给 Understanding 的本地文件同样优先上传本地内容,`original_source` 只作为来源元数据,不会导致二次抓取原 URL。
+
+Accessor 路径中的文档图片和 Base 附件字段图片使用同一条素材下载链路:先生成内部 `feishu://image/{file_token}` 引用;`download_images` 开启时,再通过飞书 Drive 素材接口下载到 `images/` 并改写 Markdown 相对路径。应用还必须具备 `docs:document.media:download` 权限;关闭下载、权限不足(例如 HTTP 403)或素材请求失败时,不会伪造本地图片文件,也不会把引用改成不存在的本地路径。Understanding 直达路径不调用这套素材接口,而是接收结果 ZIP 中已经包含的图片,并生成 artifact 图片映射供后续 URI 改写。
+
+`UnifiedResourceProcessor.prepare` 随后冻结两个字段:
+
+- `resolved_extension`:本次 Parser 路由唯一使用的扩展名。
+- `resolved_name`:用于展示和默认资源命名,不参与 HTTP 资源的类型覆盖。
+
+HTTP 资源以 HTTPAccessor 检出的 `meta.extension` 为准;本地文件或上传的临时文件可优先使用显式 `source_name` 的扩展名。这样既不会拿随机临时文件名选 Parser,也不会让用户提供的 URL 名称覆盖实际下载内容类型。
+
+## 无后缀 URL 怎么判断类型
+
+HTTPAccessor 按以下顺序收集和修正类型:
+
+1. URL path 中受支持的显式扩展名。
+2. HEAD 响应的 `Content-Disposition` 文件名。
+3. HEAD 响应的 `Content-Type`。
+4. GET 响应的 `Content-Disposition` 和 `Content-Type`,用于修正之前的模糊网页判断。
+5. GET 内容的 magic bytes,例如 PDF、图片、音视频、Office/EPUB/ZIP 签名。
+6. 仍无法识别时按网页处理。
+
+最终扩展名写入 `LocalResource.meta.extension`,然后冻结为 `resolved_extension`。`ParserRouter` 只读取这个结果。URL 上已有明确扩展名时,不用 magic bytes 擅自覆盖它。
+
+因此,`wait=false` 不等于“完全不碰源站就返回”。HTTP 服务收到无后缀远程 URL、且 Understanding 已启用时,必须先下载或探测一次,才能知道应进入外部解析队列还是内置 Parser。命中 Understanding 后,生产端直接上传这份已检测的本地文件并取得 `response_id`,然后才清理临时文件;Worker 只恢复该 response,不会重新下载可能已过期或内容已变化的 URL。
+
+## Parser:只回答“本地内容怎么解析”
+
+标准执行链分四类:
+
+- 原始文本:直接进入内置 `ParserRegistry`。
+- 可由 Understanding 直接接收的原始 URL:在 Accessor 前进入 `ParserRouter`。
+- 本地目录:直接进入 `DirectoryParser`;Git 仓库由目录链委派给代码仓库 Parser。
+- 本地文件:进入 `ParserRouter`,根据 `resolved_extension` 在内置 `ParserRegistry` 与 Understanding 之间选一次。
+
+ParserRegistry 只注册项目内置 Parser,不再提供自定义 Parser 类、回调注册或可选模块注册入口。新增数据源优先实现 Accessor;新增文件格式则直接增加内置 Parser 和对应测试。
+
+目录、网站和压缩包的“子文件遍历”属于当前内置 Parser 的内部实现,不回到顶层 `ResourceService`,也不会给每个子文件重新选择 Understanding。顶层压缩文件本身是否进入 Understanding,仍由其冻结扩展名和 `parser_api.extensions` 决定。
+
+## Understanding 链路
+
+Understanding 是“外部解析器”,不是“外部落盘器”:
+
+```text
+LocalResource / 远程源
+ |
+ v
+Understanding API
+ |
+ v
+解析结果 ZIP
+ |
+ v
+解压到临时 Viking 目录
+ |
+ v
+ParseResult
+ |
+ v
+TreeBuilder + 标准摘要/索引链
+```
+
+同步路径中,Accessor 已下载的本地文件会直接上传给 Understanding,`original_source` 只保留作来源元数据。普通 HTTP 文件的异步路径也先上传已检测文件,再通过统一的 `AddResourceMsg` 持久化 `understanding_response_id`、冻结的 `resolved_extension` 和 `parser_backend="understanding"`;Worker 直接恢复 response,既不重新下载源 URL,也不因配置变化重新选择后端。这些冻结字段是内部任务字段,公共 `args` 不能指定,避免调用方绕过外部解析开关和扩展名白名单。
+
+飞书直达的异步路径也冻结 `parser_backend="understanding"`。生产端统一调用 Understanding 提交:显式 user token 直接转换为 `lark_file`,应用凭证则在 UnderstandingAPI 内获取 tenant token。QueueFS 只持久化 `understanding_response_id`,不保存 token;Worker 从该 response 继续轮询,不重复提交也不重新判断配置。未显式指定资源名的飞书 artifact 会从 ZIP 单一根目录恢复真实标题,目标 URI 因而延迟到解析完成后确定。
+
+Understanding 返回 ZIP 时,本地适配器会安全解压,并根据 Markdown 的相对图片引用生成受控的图片映射 sidecar。TreeBuilder 后续仍使用统一的图片 URI 改写链,不依赖飞书 Accessor 的 `feishu://` 占位符。
+
+这条链的关键特征是:Understanding 只替代 Parser,后面的 `ParseResult`、URI 规划、TreeBuilder 落盘、摘要和索引仍属于 OpenViking。
+
+## Connector 链路
+
+Connector 是另一套端到端导入服务:
+
+```text
+ResourceService
+ |
+ +--> Connector doc/add
+ |
+ +--> Connector 获取源数据
+ +--> Connector 解析
+ +--> Connector 写入目标资源树
+ |
+ +--> 后台轮询 Connector task/info
+ |
+ +--> 更新 OpenViking TaskRecord
+```
+
+Connector 不返回本地 `ParseResult`,也不调用当前进程的 `TreeBuilder`。OpenViking 只负责校验这次请求能否无损委派、提交任务、返回 OpenViking `task_id`,再把 Connector 的终态同步到任务记录。
+
+Connector 当前要求提供精确 `to`,不接受 `parent`;也不支持 `wait=true`、watch、instruction、关闭建索引、摘要、strict、include/exclude 等。无凭证的 Git 请求可回退到标准链;带 Connector 专用凭证的 Git 和 Connector-only 来源会立即报错,避免凭证落入本地持久化任务。
+
+## `wait` 的准确含义
+
+| 路径 | `wait=false` | `wait=true` |
+|---|---|---|
+| Connector | 提交外部任务后返回 | 不支持;不会假装同步等待 |
+| Git | 预检并预占 URI 后启动后台标准链 | 当前请求内完成标准链并等待队列 |
+| 飞书直达 Understanding | 生产端提交后只将 response ID 放入 `ExternalParse`;无显式资源名时可延迟返回 `root_uri` | 当前请求内调用 Understanding,再等待后续队列 |
+| HTTP 服务 + 异步 Understanding | 先识别类型,再预占 URI、入 `ExternalParse` 后返回 | 当前请求内调用 Understanding,再等待后续队列 |
+| 普通标准链 | 解析和落盘完成后返回;摘要/语义处理由任务监控 | 解析和落盘完成后继续等待语义队列 |
+
+所以普通 `wait=false` 不是“所有工作后台化”,而是“资源树已落盘,但不阻塞等待后续语义任务”。Git 与异步 Understanding 是两个明确的例外分支。
+
+## 目标 URI、锁和失败边界
+
+- 需要后台执行的 Git 与普通文件 Understanding 任务会先规划并预占目标 URI,避免返回的 URI 随后台竞态变化。未指定名称的飞书直达任务会延迟目标 URI 解析,以 artifact 的真实根标题作为最终名称,并在完成时回写 TaskRecord。
+- 锁通过 handoff 交给后台任务或队列 Worker;入队失败时立即释放,并把任务标为失败。
+- 临时 `LocalResource` 由拥有它的调用层清理;交给标准处理器后,清理责任随之转移。
+- Parser 产生 `ParseResult` 后才进入 TreeBuilder。没有临时解析产物时标准链返回解析错误;目录允许带 warnings 的部分成功,`strict` 决定是否暴露这些警告。
+- Connector 的失败边界在外部任务终态,OpenViking 不对其内部文件逐个回滚。
+
+## 代码定位
+
+| 职责 | 入口 |
+|---|---|
+| 公开入口、新请求分流与内部执行 | `openviking/service/resource_service.py`:`ResourceService.add_resource`、`_submit_resource_ingestion`、`_execute_resource_ingestion` |
+| Watch 刷新入口 | `openviking/service/resource_service.py`:`ResourceService.refresh_resource`;`openviking/resource/watch_scheduler.py` |
+| 标准解析与落盘编排 | `openviking/utils/resource_processor.py`:`ResourceProcessor.process_resource` |
+| Accessor 与 Parser 两层衔接 | `openviking/utils/media_processor.py`:`UnifiedResourceProcessor` |
+| 文件 Parser 单次选择 | `openviking/parse/parser_router.py`:`ParserRouter` |
+| 内置 Parser 注册 | `openviking/parse/registry.py`:`ParserRegistry` |
+| HTTP 类型识别 | `openviking/parse/accessors/http_accessor.py`:`HTTPAccessor`、`URLTypeDetector` |
+| Understanding 同步适配 | `openviking/parse/understanding_api.py`:`UnderstandingAPI` |
+| 飞书直达开关 | `openviking_cli/utils/config/open_viking_config.py`:`ParserApiConfig.enable_feishu_url` |
+| 可恢复的后台资源消息与双队列 Worker | `openviking/storage/queuefs/add_resource_msg.py`、`add_resource_processor.py`、`queue_manager.py` |
+| Connector 客户端 | `openviking/connector/client.py`:`ConnectorClient` |
+
+## 快速自检
+
+- 无后缀 PDF URL:HTTPAccessor 从响应头或 PDF 签名得到 `.pdf`,ParserRouter 再决定是否使用 Understanding。
+- 飞书默认配置:走 FeishuAccessor;仅开启 `enable_feishu_url` 且有凭证时才绕过 Accessor。
+- 飞书 user token 直达:队列中只有 response ID 和强制后端标记,不出现 token 明文。
+- 已归一化飞书 Markdown:无论 `extensions` 是否包含 `md`,都走内置 Markdown Parser。
+- Understanding 返回结果:先转成 `ParseResult`,仍由本地 TreeBuilder 落盘。
+- Connector 返回结果:只返回任务标识,不经过本地 `ParseResult`。
+- 普通 Markdown 且 `wait=false`:返回前 Markdown 已解析并落盘,只是不等待后续语义队列。
+- 网站抓取出的目录:进入 DirectoryParser,页面子文件不会逐个调用 Understanding。
diff --git a/openviking/eval/datasets/local_doc_example_glm5.jsonl b/openviking/eval/datasets/local_doc_example_glm5.jsonl
index ec42d4834..3e1eb897b 100644
--- a/openviking/eval/datasets/local_doc_example_glm5.jsonl
+++ b/openviking/eval/datasets/local_doc_example_glm5.jsonl
@@ -16,5 +16,4 @@
{"question": "语义处理队列的工作机制是什么?如何监控和调试异步处理状态?", "files": ["OpenViking/docs/en/concepts/01-architecture.md", "OpenViking/openviking/service/resource_service.py:1-80"], "answer": "语义处理队列通过 SemanticQueue 异步生成 L0/L1 内容。\n\n**工作机制**:\n\n1. **处理流水线**:\n```\nInput → Parser → TreeBuilder → AGFS → SemanticQueue → Vector Index\n```\n\n2. **队列处理**:\n - TreeBuilder 将文件移动到 AGFS 后入队\n - SemanticQueue 异步处理队列项\n - 自底向上生成 L0/L1\n\n3. **异步处理优势**:\n - 不阻塞资源添加操作\n - 支持大规模文档处理\n - LLM 调用并行化\n\n**监控和调试**:\n\n1. **等待处理完成**:\n```python\nresult = client.add_resource(\"./docs\", wait=False)\nclient.wait_processed() # 等待队列处理完成\n```\n\n2. **超时控制**:\n```python\nresult = client.add_resource(\n \"./docs\",\n wait=True,\n timeout=300 # 超时时间(秒)\n)\n```\n\n3. **DebugService**:\n - 提供 ObserverService 用于调试\n - 可视化检索轨迹\n\n4. **日志监控**:\n - 通过 logger 获取处理状态\n - 监控队列长度和处理延迟\n\n**Service 层支持**:\n- ResourceService 管理 add_resource 操作\n- wait_processed() 方法等待语义处理完成"}
{"question": "如何通过代码动态创建和管理 Session?Session 与 Memory 的关系是什么?", "files": ["OpenViking/docs/en/concepts/08-session.md", "OpenViking/openviking/session/compressor_v2.py"], "answer": "Session 通过 client API 动态创建,commit 后自动提取 Memory。\n\n**Session 创建和管理**:\n\n1. **创建 Session**:\n```python\n# 创建新会话\nsession = client.session(session_id=\"chat_001\")\n\n# 或使用 create_session\nresult = client.create_session()\n```\n\n2. **添加消息**:\n```python\nsession.add_message(\n \"user\",\n [TextPart(\"How to configure embedding?\")]\n)\n\nsession.add_message(\n \"assistant\",\n [\n TextPart(\"Here's how...\"),\n ContextPart(uri=\"viking://user/memories/profile.md\"),\n ]\n)\n```\n\n3. **记录使用**:\n```python\nsession.used(contexts=[\"viking://user/memories/profile.md\"])\nsession.used(skill={\n \"uri\": \"viking://user/skills/code-search\",\n \"input\": \"search config\",\n \"output\": \"found 3 files\",\n \"success\": True\n})\n```\n\n4. **提交会话**:\n```python\nresult = session.commit()\n# {\"status\": \"committed\", \"memories_extracted\": 5}\n```\n\n**Session 与 Memory 的关系**:\n\n1. **Memory 来源**:\n - Memory 从 Session 消息中提取\n - SessionCompressorV2 在 commit 时触发\n\n2. **提取流程**:\n - 格式化会话消息\n - 根据当前记忆策略调用 LLM 提取已启用的记忆类型\n - 写入当前用户或相关 Peer 的记忆目录并索引\n\n3. **内置记忆类型与默认位置**:\n - profile → `user/memories/profile.md`\n - preferences → `user/memories/preferences/`\n - entities → `user/memories/entities/`\n - events → `user/memories/events/`\n - identity → `user/memories/identity.md`\n - soul → `user/memories/soul.md`\n - cases → `user/memories/cases/`\n - trajectories → `user/memories/trajectories/`\n - experiences → `user/memories/experiences/`\n - tools → `user/memories/tools/`\n - skills → `user/memories/skills/`\n\n这些 `user/...` 短路径会按当前身份解析为 `viking://user/{user_id}/...`,应用也可以通过自定义模板扩展或调整记忆类型。"}
{"question": "OpenViking Server 模式与本地模式的区别是什么?如何选择部署方式?", "files": ["OpenViking/openviking/server/bootstrap.py:1-61", "OpenViking/docs/en/concepts/01-architecture.md"], "answer": "Server 模式提供 HTTP API,本地模式直接嵌入应用。\n\n**模式对比**:\n\n| 维度 | 本地模式 | Server 模式 |\n|------|----------|-------------|\n| 访问方式 | Python SDK | HTTP API |\n| 部署复杂度 | 简单 | 需启动服务 |\n| 多客户端 | 不支持 | 支持 |\n| 资源共享 | 单进程 | 多进程共享 |\n| 适用场景 | 单机应用 | 生产环境 |\n\n**Server 模式启动**:\n\n1. **命令行启动**:\n```bash\nopenviking-server --host 0.0.0.0 --port 1933 --config ./ov.conf\n```\n\n2. **参数说明**:\n- `--host`:绑定地址\n- `--port`:端口号\n- `--config`:配置文件路径\n\n3. **HTTP API 示例**:\n```bash\ncurl -X POST http://localhost:1933/api/v1/resources \\\n -H \"Content-Type: application/json\" \\\n -H \"X-API-Key: your-key\" \\\n -d '{\"path\": \"./documents/guide.md\"}'\n```\n\n**部署方式选择**:\n\n1. **选择本地模式**:\n - 单机应用开发\n - 快速原型验证\n - 无需多客户端访问\n\n2. **选择 Server 模式**:\n - 生产环境部署\n - 多客户端/多服务访问\n - 需要集中式上下文管理\n - 微服务架构集成\n\n**Service 层复用**:\n两种模式共享 Service 层逻辑,支持 HTTP Server 和 CLI 复用。"}
-{"question": "如何扩展自定义的 Parser?例如支持新的文档格式或代码语言?", "files": ["OpenViking/openviking/parse/registry.py", "OpenViking/openviking/parse/parsers/base_parser.py:1-80"], "answer": "通过 ParserRegistry 注册自定义 Parser,支持 Protocol 接口。\n\n**自定义 Parser 步骤**:\n\n1. **实现 BaseParser 接口**:\n```python\nclass BaseParser(ABC):\n @abstractmethod\n async def parse(self, source: Union[str, Path], instruction: str = \"\", **kwargs) -> ParseResult:\n \"\"\"解析文档\"\"\"\n pass\n \n @abstractmethod\n async def parse_content(self, content: str, source_path: Optional[str] = None, **kwargs) -> ParseResult:\n \"\"\"解析内容\"\"\"\n pass\n \n @property\n @abstractmethod\n def supported_extensions(self) -> List[str]:\n \"\"\"支持的文件扩展名\"\"\"\n pass\n```\n\n2. **注册 Parser**:\n```python\nfrom openviking.parse.registry import get_registry\n\nclass MyParser(BaseParser):\n @property\n def supported_extensions(self) -> List[str]:\n return [\".myext\"]\n \n async def parse(self, source, instruction=\"\", **kwargs):\n # 实现解析逻辑\n return ParseResult(...)\n\n# 注册\nregistry = get_registry()\nregistry.register(\"my_parser\", MyParser())\n```\n\n3. **使用 Protocol 接口**:\n```python\nregistry.register_custom(\n handler=my_parser, # 实现 CustomParserProtocol\n extensions=[\".custom\"],\n name=\"custom_parser\"\n)\n```\n\n**内置 Parser**:\n- TextParser:纯文本\n- MarkdownParser:Markdown\n- PDFParser:PDF\n- HTMLParser:HTML\n- CodeRepositoryParser:代码仓库\n- ImageParser:图片(可选)\n\n**扩展点**:\n- `supported_extensions`:定义支持的扩展名\n- `can_parse()`:检查文件是否可解析\n- `_read_file()`:读取文件内容"}
{"question": "eval 模块的 RAGAS 集成是如何实现的?如何自定义评测指标?", "files": ["OpenViking/openviking/eval/ragas.py", "OpenViking/openviking/eval/base.py:1-60"], "answer": "eval 模块通过 RagasEvaluator 适配器集成 RAGAS,支持自定义评测指标。\n\n**RAGAS 集成实现**:\n\n1. **RagasEvaluator 类**:\n```python\nclass RagasEvaluator(BaseEvaluator):\n def __init__(self, metrics=None, llm=None, embeddings=None):\n from ragas.metrics import (\n answer_relevance,\n context_precision,\n context_recall,\n faithfulness,\n )\n self.metrics = metrics or [\n faithfulness,\n answer_relevance,\n context_precision,\n context_recall,\n ]\n```\n\n2. **评测流程**:\n```python\nasync def evaluate_dataset(self, dataset: EvalDataset) -> SummaryResult:\n # 转换为 RAGAS 数据集格式\n data = {\n \"question\": [s.query for s in dataset.samples],\n \"contexts\": [s.context for s in dataset.samples],\n \"answer\": [s.response for s in dataset.samples],\n \"ground_truth\": [s.ground_truth for s in dataset.samples],\n }\n \n # 调用 RAGAS evaluate\n result = evaluate(data, metrics=self.metrics)\n```\n\n**自定义评测指标**:\n\n1. **继承 BaseEvaluator**:\n```python\nclass MyEvaluator(BaseEvaluator):\n async def evaluate_sample(self, sample: EvalSample) -> EvalResult:\n # 实现自定义评测逻辑\n scores = {\"my_metric\": 0.85}\n return EvalResult(sample=sample, scores=scores)\n```\n\n2. **自定义 RAGAS 指标**:\n```python\nfrom ragas import evaluate\nfrom ragas.metrics import Metric\n\nclass MyMetric(Metric):\n name = \"my_metric\"\n # 实现指标计算\n\nevaluator = RagasEvaluator(metrics=[MyMetric()])\n```\n\n3. **评测结果结构**:\n```python\nclass SummaryResult(BaseModel):\n dataset_name: str\n sample_count: int\n mean_scores: Dict[str, float] # 平均分数\n results: List[EvalResult] # 每个样本结果\n```"}
diff --git a/openviking/parse/__init__.py b/openviking/parse/__init__.py
index 380f65288..d4b4f0370 100644
--- a/openviking/parse/__init__.py
+++ b/openviking/parse/__init__.py
@@ -4,8 +4,6 @@
from openviking.core.building_tree import BuildingTree
from openviking.parse.base import NodeType, ParseResult, ResourceNode, create_parse_result
-from openviking.parse.converter import DocumentConverter
-from openviking.parse.custom import CallbackParserWrapper, CustomParserProtocol, CustomParserWrapper
from openviking.parse.directory_scan import (
CLASS_PROCESSABLE,
CLASS_UNSUPPORTED,
@@ -18,7 +16,6 @@ from openviking.parse.parsers.html import HTMLParser
from openviking.parse.parsers.markdown import MarkdownParser
from openviking.parse.parsers.pdf import PDFParser
from openviking.parse.parsers.text import TextParser
-from openviking.parse.registry import ParserRegistry, get_registry, parse
from openviking.parse.tree_builder import TreeBuilder
from openviking.parse.vlm import VLMProcessor
@@ -34,15 +31,6 @@ __all__ = [
"MarkdownParser",
"PDFParser",
"HTMLParser",
- "DocumentConverter",
- # Custom parser support
- "CustomParserProtocol",
- "CustomParserWrapper",
- "CallbackParserWrapper",
- # Registry
- "ParserRegistry",
- "get_registry",
- "parse",
# Tree builder
"TreeBuilder",
"BuildingTree",
diff --git a/openviking/parse/accessors/feishu_accessor.py b/openviking/parse/accessors/feishu_accessor.py
index d8951a10b..df8a30db9 100644
--- a/openviking/parse/accessors/feishu_accessor.py
+++ b/openviking/parse/accessors/feishu_accessor.py
@@ -4,21 +4,23 @@
Feishu/Lark Accessor.
Fetches Feishu/Lark cloud documents using the lark-oapi SDK.
-This is the DataAccessor layer extracted from FeishuParser.
Note: This accessor requires the `lark-oapi` package.
Included by default in `openviking[bot]` installation.
"""
import asyncio
+import json
+import mimetypes
import os
import re
import tempfile
from dataclasses import dataclass
from pathlib import Path
-from typing import Any, Dict, NoReturn, Optional, Tuple, Union
-from urllib.parse import urlparse
+from typing import Any, Dict, List, NoReturn, Optional, Tuple, Union
+from urllib.parse import parse_qs, urlparse
+from openviking.parse.base import format_table_to_markdown
from openviking.utils.exceptions import error_code_from_http_status
from openviking_cli.exceptions import OpenVikingError
from openviking_cli.utils.logger import get_logger
@@ -83,8 +85,7 @@ def _raise_from_lark_response(
if code == _FEISHU_BITABLE_PERMISSION_REQUIRED:
public_code = "FAILED_PRECONDITION"
message = (
- "Feishu application is missing required Bitable permissions: "
- f"code={code}, msg={msg}"
+ f"Feishu application is missing required Bitable permissions: code={code}, msg={msg}"
)
else:
public_code = (
@@ -101,7 +102,7 @@ def _raise_from_lark_response(
class FeishuDocument:
"""Result from fetching a Feishu document."""
- doc_type: str # "docx" only for now
+ doc_type: str
token: str
markdown_content: str
title: str
@@ -114,7 +115,9 @@ class FeishuAccessor(DataAccessor):
Supports:
- Documents: https://*.feishu.cn/docx/{document_id}
- - Wiki pages: https://*.feishu.cn/wiki/{token} (resolves to docx)
+ - Wiki pages: https://*.feishu.cn/wiki/{token}
+ - Spreadsheets: https://*.feishu.cn/sheets/{token}
+ - Bitable: https://*.feishu.cn/base/{app_token}
Requires:
- lark-oapi package
@@ -127,6 +130,11 @@ class FeishuAccessor(DataAccessor):
# Wiki obj_type normalization (API returns short names)
_WIKI_TYPE_MAP = {"doc": "docx", "sheet": "sheets", "bitable": "base"}
+ _DOC_TYPE_HANDLERS = {
+ "docx": "_parse_docx",
+ "sheets": "_parse_sheets",
+ "base": "_parse_bitable",
+ }
# Attributes that skip processing (structural containers or metadata)
_SKIP_ATTRS = {"page", "table_cell", "quote_container", "grid", "grid_column"}
@@ -136,6 +144,7 @@ class FeishuAccessor(DataAccessor):
"divider": "_handle_divider",
"image": "_handle_image",
"table": "_table_block_to_markdown",
+ "sheet": "_embedded_sheet_to_markdown",
}
# Attribute → markdown prefix template for text-bearing blocks.
@@ -169,6 +178,7 @@ class FeishuAccessor(DataAccessor):
19: "callout",
22: "divider",
27: "image",
+ 30: "sheet",
31: "table",
32: "table_cell",
34: "quote_container",
@@ -196,6 +206,7 @@ class FeishuAccessor(DataAccessor):
"callout",
"divider",
"image",
+ "sheet",
"table",
"table_cell",
"quote_container",
@@ -312,9 +323,12 @@ class FeishuAccessor(DataAccessor):
"""
Fetch a Feishu document and convert to Markdown.
- This method extracts and adapts the logic from FeishuParser.parse().
+ The fetched document is materialized as Markdown for the standard parser chain.
"""
doc_type, token = self._parse_feishu_url(url)
+ query = parse_qs(urlparse(url).query)
+ table_id = (query.get("table") or [None])[0]
+ view_id = (query.get("view") or [None])[0]
title = None
meta = {}
@@ -328,24 +342,37 @@ class FeishuAccessor(DataAccessor):
doc_type, token = real_type, real_token
meta["wiki_resolved"] = True
- # Only docx is supported
- if doc_type != "docx":
+ if doc_type != "base":
+ table_id = view_id = None
+
+ handler_name = self._DOC_TYPE_HANDLERS.get(doc_type)
+ if handler_name is None:
raise ValueError(
f"Unsupported Feishu document type: {doc_type}. "
- f"Only docx is supported in this version."
+ f"Supported: {list(self._DOC_TYPE_HANDLERS)}"
)
- # Call the handler (in thread pool since lark-oapi is sync)
+ handler_kwargs = {}
+ if doc_type == "base":
+ handler_kwargs = {"table_id": table_id, "view_id": view_id}
+
+ # Feishu's SDK is synchronous; keep it off the event loop.
markdown, doc_title = await asyncio.to_thread(
- self._parse_docx,
+ getattr(self, handler_name),
token,
feishu_access_token,
+ **handler_kwargs,
)
if title:
- doc_title = title
+ scope = "/".join(value for value in (table_id, view_id) if value)
+ doc_title = f"{title} ({scope})" if scope else title
meta["original_url"] = url
+ if table_id:
+ meta["feishu_table_id"] = table_id
+ if view_id:
+ meta["feishu_view_id"] = view_id
return FeishuDocument(
doc_type=doc_type,
@@ -382,7 +409,7 @@ class FeishuAccessor(DataAccessor):
path_parts = [p for p in parsed.path.split("/") if p]
if len(path_parts) < 2:
raise ValueError(f"Cannot parse Feishu URL: {url}")
- doc_type = path_parts[0] # docx, wiki
+ doc_type = path_parts[0] # docx, wiki, sheets, base
token = path_parts[1]
return doc_type, token
@@ -434,6 +461,10 @@ class FeishuAccessor(DataAccessor):
return RequestOption.builder().user_access_token(feishu_access_token).build()
+ def _call_api(self, method, request, feishu_access_token: Optional[str] = None):
+ option = self._user_request_option(feishu_access_token)
+ return method(request) if option is None else method(request, option)
+
# ========== Wiki Resolution ==========
def _resolve_wiki_node(
@@ -451,11 +482,11 @@ class FeishuAccessor(DataAccessor):
client = self._get_client(use_user_token=bool(feishu_access_token))
request = GetNodeSpaceRequest.builder().token(token).build()
- option = self._user_request_option(feishu_access_token)
- if option is None:
- response = client.wiki.v2.space.get_node(request)
- else:
- response = client.wiki.v2.space.get_node(request, option)
+ response = self._call_api(
+ client.wiki.v2.space.get_node,
+ request,
+ feishu_access_token,
+ )
if not response.success():
_raise_from_lark_response(
response,
@@ -512,7 +543,11 @@ class FeishuAccessor(DataAccessor):
continue # Skip page container
line = self._block_to_markdown(
- block, block_map, ordered_counter, document_id=document_id
+ block,
+ block_map,
+ ordered_counter,
+ document_id=document_id,
+ feishu_access_token=feishu_access_token,
)
if line is not None:
markdown_lines.append(line)
@@ -548,11 +583,11 @@ class FeishuAccessor(DataAccessor):
builder = builder.page_token(page_token)
request = builder.build()
- option = self._user_request_option(feishu_access_token)
- if option is None:
- response = client.docx.v1.document_block.list(request)
- else:
- response = client.docx.v1.document_block.list(request, option)
+ response = self._call_api(
+ client.docx.v1.document_block.list,
+ request,
+ feishu_access_token,
+ )
if not response.success():
_raise_from_lark_response(
@@ -592,7 +627,12 @@ class FeishuAccessor(DataAccessor):
return None
def _block_to_markdown(
- self, block, block_map: Dict, ordered_counter: Dict[str, int], document_id: str = ""
+ self,
+ block,
+ block_map: Dict,
+ ordered_counter: Dict[str, int],
+ document_id: str = "",
+ feishu_access_token: Optional[str] = None,
) -> Optional[str]:
"""Convert a single SDK block object to markdown string.
@@ -618,7 +658,12 @@ class FeishuAccessor(DataAccessor):
# Special blocks (non-text: divider, image, table)
special_handler = self._SPECIAL_BLOCK_HANDLERS.get(attr)
if special_handler:
- return getattr(self, special_handler)(block, block_map, document_id=document_id)
+ return getattr(self, special_handler)(
+ block,
+ block_map,
+ document_id=document_id,
+ feishu_access_token=feishu_access_token,
+ )
# --- Text-bearing blocks: extract elements, apply formatting ---
content_obj = getattr(block, attr, None)
@@ -747,12 +792,8 @@ class FeishuAccessor(DataAccessor):
.token_types({token_type})
.build()
)
- option = self._user_request_option(feishu_access_token)
-
try:
- raw_resp = (
- client.request(raw_req) if option is None else client.request(raw_req, option)
- )
+ raw_resp = self._call_api(client.request, raw_req, feishu_access_token)
except Exception as exc:
logger.warning("[FeishuAccessor] Error downloading image %s: %s", file_token, exc)
return None
@@ -936,8 +977,6 @@ class FeishuAccessor(DataAccessor):
row.append("")
rows.append(row)
- from openviking.parse.base import format_table_to_markdown
-
return format_table_to_markdown(rows, has_header=True) if rows else None
def _extract_cell_text(self, cell_block, block_map: Dict) -> str:
@@ -956,3 +995,374 @@ class FeishuAccessor(DataAccessor):
if text:
texts.append(text)
return " ".join(texts)
+
+ def _embedded_sheet_to_markdown(
+ self,
+ block,
+ block_map: Dict = None,
+ *,
+ document_id: str = "",
+ feishu_access_token: Optional[str] = None,
+ **_,
+ ) -> Optional[str]:
+ """Convert an embedded spreadsheet block in a docx document."""
+ import lark_oapi as lark
+
+ client = self._get_client(use_user_token=bool(feishu_access_token))
+ token_type = (
+ lark.AccessTokenType.USER if feishu_access_token else lark.AccessTokenType.TENANT
+ )
+ request = (
+ lark.BaseRequest.builder()
+ .http_method(lark.HttpMethod.GET)
+ .uri(
+ f"/open-apis/docx/v1/documents/{document_id or block.parent_id}"
+ f"/blocks/{block.block_id}"
+ )
+ .token_types({token_type})
+ .build()
+ )
+ response = self._call_api(client.request, request, feishu_access_token)
+ if not response.success():
+ logger.warning(
+ "[FeishuAccessor] Failed to inspect embedded sheet %s: code=%s msg=%s",
+ block.block_id,
+ getattr(response, "code", None),
+ getattr(response, "msg", None),
+ )
+ return None
+
+ data = json.loads(response.raw.content)
+ sheet_token = data.get("data", {}).get("block", {}).get("sheet", {}).get("token", "")
+ parts = sheet_token.rsplit("_", 1)
+ if len(parts) != 2:
+ return None
+
+ spreadsheet_token, sheet_id = parts
+ try:
+ rows = self._read_sheet_range(
+ spreadsheet_token,
+ sheet_id,
+ max_rows=100,
+ max_cols=26,
+ feishu_access_token=feishu_access_token,
+ )
+ except Exception as exc:
+ logger.warning(
+ "[FeishuAccessor] Failed to read embedded sheet %s: %s",
+ sheet_token,
+ exc,
+ )
+ return None
+
+ rows = self._trim_empty_columns(rows)
+ return format_table_to_markdown(rows, has_header=True) if rows else None
+
+ @staticmethod
+ def _trim_empty_columns(rows: List[List[str]]) -> List[List[str]]:
+ """Remove trailing columns that are empty in every row."""
+ if not rows:
+ return rows
+ last_col = 0
+ for col in range(max(len(row) for row in rows)):
+ if any(col < len(row) and row[col].strip() for row in rows):
+ last_col = col + 1
+ return [row[:last_col] for row in rows] if last_col else []
+
+ def _parse_sheets(
+ self,
+ token: str,
+ feishu_access_token: Optional[str] = None,
+ ) -> Tuple[str, str]:
+ """Fetch a Feishu spreadsheet and convert it to Markdown."""
+ import lark_oapi as lark
+
+ client = self._get_client(use_user_token=bool(feishu_access_token))
+ config = self._get_config()
+ token_type = (
+ lark.AccessTokenType.USER if feishu_access_token else lark.AccessTokenType.TENANT
+ )
+ metadata_request = (
+ lark.BaseRequest.builder()
+ .http_method(lark.HttpMethod.GET)
+ .uri(f"/open-apis/sheets/v2/spreadsheets/{token}/metainfo")
+ .token_types({token_type})
+ .build()
+ )
+ metadata_response = self._call_api(
+ client.request,
+ metadata_request,
+ feishu_access_token,
+ )
+ if not metadata_response.success():
+ _raise_from_lark_response(
+ metadata_response,
+ operation=f"fetch spreadsheet metadata for {token}",
+ resource=token,
+ )
+ metadata = json.loads(metadata_response.raw.content).get("data", {})
+ title = (metadata.get("properties") or {}).get("title") or "Spreadsheet"
+ sheets = metadata.get("sheets") or []
+ markdown_parts = [f"# {title}", f"**Sheets:** {len(sheets)}"]
+ for sheet in sheets:
+ sheet_id = sheet.get("sheetId") or ""
+ sheet_title = sheet.get("title") or sheet_id
+ parts = [f"## Sheet: {sheet_title}"]
+
+ block_info = sheet.get("blockInfo")
+ if block_info:
+ block_type = block_info.get("blockType") or "unknown"
+ if block_type != "BITABLE_BLOCK":
+ parts.append(f"*Unsupported sheet block: {block_type}*")
+ else:
+ block_token = block_info.get("blockToken") or ""
+ tokens = block_token.rsplit("_", 1)
+ if len(tokens) != 2 or not all(tokens):
+ parts.append("*Invalid embedded bitable token*")
+ else:
+ bitable, _ = self._parse_bitable(
+ tokens[0],
+ feishu_access_token,
+ table_id=tokens[1],
+ table_name=sheet_title,
+ )
+ parts.append(bitable or "*Empty bitable*")
+ markdown_parts.append("\n\n".join(parts))
+ continue
+
+ row_count = int(sheet.get("rowCount") or 0)
+ col_count = int(sheet.get("columnCount") or 0)
+ if not row_count or not col_count:
+ parts.append("*Empty sheet*")
+ markdown_parts.append("\n\n".join(parts))
+ continue
+
+ parts.append(f"**Dimensions:** {row_count} rows x {col_count} columns")
+ rows_to_read = min(row_count, config.max_rows_per_sheet)
+ rows = self._read_sheet_range(
+ token,
+ sheet_id,
+ rows_to_read,
+ col_count,
+ feishu_access_token=feishu_access_token,
+ )
+ if rows:
+ parts.append(format_table_to_markdown(rows, has_header=True))
+ if row_count > config.max_rows_per_sheet:
+ parts.append(
+ f"\n*... {row_count - config.max_rows_per_sheet} more rows truncated ...*"
+ )
+ if col_count > 26:
+ parts.append(f"\n*... {col_count - 26} columns after Z omitted ...*")
+ markdown_parts.append("\n\n".join(parts))
+
+ return "\n\n".join(markdown_parts), title
+
+ def _read_sheet_range(
+ self,
+ token: str,
+ sheet_id: str,
+ max_rows: int,
+ max_cols: int,
+ feishu_access_token: Optional[str] = None,
+ ) -> List[List[str]]:
+ """Read a bounded cell range from a Feishu spreadsheet."""
+ import lark_oapi as lark
+
+ client = self._get_client(use_user_token=bool(feishu_access_token))
+ # ponytail: the existing importer reads A:Z only; add chunked ranges if wider
+ # spreadsheet imports become a real requirement.
+ end_col = self._col_number_to_letter(min(max_cols, 26))
+ cell_range = f"{sheet_id}!A1:{end_col}{max_rows}"
+ token_type = (
+ lark.AccessTokenType.USER if feishu_access_token else lark.AccessTokenType.TENANT
+ )
+ request = (
+ lark.BaseRequest.builder()
+ .http_method(lark.HttpMethod.GET)
+ .uri(f"/open-apis/sheets/v2/spreadsheets/{token}/values/{cell_range}")
+ .token_types({token_type})
+ .build()
+ )
+ response = self._call_api(client.request, request, feishu_access_token)
+ if not response.success():
+ _raise_from_lark_response(
+ response,
+ operation=f"read spreadsheet range {cell_range}",
+ resource=token,
+ )
+
+ data = json.loads(response.raw.content)
+ values = data.get("data", {}).get("valueRange", {}).get("values", [])
+ return [[str(cell) if cell is not None else "" for cell in row] for row in values]
+
+ @staticmethod
+ def _col_number_to_letter(number: int) -> str:
+ return chr(ord("A") + number - 1) if 1 <= number <= 26 else "Z"
+
+ def _parse_bitable(
+ self,
+ app_token: str,
+ feishu_access_token: Optional[str] = None,
+ *,
+ table_id: Optional[str] = None,
+ table_name: Optional[str] = None,
+ view_id: Optional[str] = None,
+ ) -> Tuple[str, str]:
+ """Fetch a Feishu bitable app and convert it to Markdown."""
+ if view_id and not table_id:
+ raise ValueError("Feishu Base URL with 'view' must also include 'table'")
+
+ from lark_oapi.api.bitable.v1 import (
+ ListAppTableFieldRequest,
+ ListAppTableRecordRequest,
+ ListAppTableRequest,
+ )
+
+ client = self._get_client(use_user_token=bool(feishu_access_token))
+ config = self._get_config()
+ if table_id:
+ tables = [(table_id, table_name or table_id)]
+ title = table_name or table_id
+ if view_id:
+ title = f"{title} ({view_id})"
+ markdown_parts = []
+ heading = "###"
+ else:
+ table_models = []
+ page_token = None
+ while True:
+ builder = ListAppTableRequest.builder().app_token(app_token).page_size(100)
+ if page_token:
+ builder = builder.page_token(page_token)
+ tables_response = self._call_api(
+ client.bitable.v1.app_table.list,
+ builder.build(),
+ feishu_access_token,
+ )
+ if not tables_response.success():
+ _raise_from_lark_response(
+ tables_response,
+ operation=f"list bitable tables for {app_token}",
+ resource=app_token,
+ )
+ table_models.extend(tables_response.data.items or [])
+ if not getattr(tables_response.data, "has_more", False):
+ break
+ page_token = getattr(tables_response.data, "page_token", None)
+ if not page_token:
+ raise RuntimeError("Feishu returned more bitable tables without a page token")
+
+ tables = [(table.table_id, table.name or table.table_id) for table in table_models]
+ title = f"Bitable ({len(tables)} tables)"
+ markdown_parts = [f"# {title}"]
+ heading = "##"
+
+ for current_table_id, current_table_name in tables:
+ fields = []
+ page_token = None
+ while True:
+ builder = (
+ ListAppTableFieldRequest.builder()
+ .app_token(app_token)
+ .table_id(current_table_id)
+ .page_size(100)
+ )
+ if page_token:
+ builder = builder.page_token(page_token)
+ fields_response = self._call_api(
+ client.bitable.v1.app_table_field.list,
+ builder.build(),
+ feishu_access_token,
+ )
+ if not fields_response.success():
+ _raise_from_lark_response(
+ fields_response,
+ operation=f"list fields for bitable table {current_table_id}",
+ resource=app_token,
+ )
+ fields.extend(fields_response.data.items or [])
+ if not getattr(fields_response.data, "has_more", False):
+ break
+ page_token = getattr(fields_response.data, "page_token", None)
+ if not page_token:
+ raise RuntimeError(
+ f"Feishu returned more fields for table {current_table_id} "
+ "without a page token"
+ )
+ field_names = [field.field_name for field in fields]
+
+ records = []
+ page_token = None
+ records_truncated = False
+ while len(records) < config.max_records_per_table:
+ remaining = config.max_records_per_table - len(records)
+ builder = (
+ ListAppTableRecordRequest.builder()
+ .app_token(app_token)
+ .table_id(current_table_id)
+ .page_size(min(remaining, 500))
+ )
+ if view_id:
+ builder = builder.view_id(view_id)
+ if page_token:
+ builder = builder.page_token(page_token)
+ records_response = self._call_api(
+ client.bitable.v1.app_table_record.list,
+ builder.build(),
+ feishu_access_token,
+ )
+ if not records_response.success():
+ _raise_from_lark_response(
+ records_response,
+ operation=f"list records for bitable table {current_table_id}",
+ resource=app_token,
+ )
+ items = records_response.data.items or []
+ records.extend(items[:remaining])
+ has_more = bool(records_response.data.has_more)
+ if len(items) > remaining:
+ records_truncated = True
+ break
+ if not has_more:
+ break
+ if len(records) >= config.max_records_per_table:
+ records_truncated = True
+ break
+ page_token = records_response.data.page_token
+ if not page_token:
+ raise RuntimeError(
+ f"Feishu returned more records for table {current_table_id} "
+ "without a page token"
+ )
+
+ parts = [f"{heading} {current_table_name}", f"**Records:** {len(records)}"]
+ if field_names and records:
+ rows = [field_names]
+ for record in records:
+ fields = record.fields or {}
+ rows.append(
+ [self._format_bitable_field(fields.get(name, "")) for name in field_names]
+ )
+ parts.append(format_table_to_markdown(rows, has_header=True))
+ if records_truncated:
+ parts.append(f"\n*... records truncated at {config.max_records_per_table} ...*")
+ markdown_parts.append("\n\n".join(parts))
+
+ return "\n\n".join(markdown_parts), title
+
+ @classmethod
+ def _format_bitable_field(cls, value: Any) -> str:
+ """Render the common structured values returned by bitable fields."""
+ if value is None:
+ return ""
+ if isinstance(value, list):
+ return ", ".join(cls._format_bitable_field(item) for item in value)
+ if isinstance(value, dict):
+ file_token = value.get("file_token")
+ name = str(value.get("name") or "image")
+ media_type = value.get("type") or mimetypes.guess_type(name)[0]
+ if file_token and str(media_type).lower().startswith("image/"):
+ return f"![{name}](feishu://image/{file_token})"
+ return str(value.get("text", value.get("name", value)))
+ return str(value)
diff --git a/openviking/parse/converter.py b/openviking/parse/converter.py
deleted file mode 100644
index 2f627ea44..000000000
--- a/openviking/parse/converter.py
+++ /dev/null
@@ -1,83 +0,0 @@
-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
-# SPDX-License-Identifier: AGPL-3.0
-"""Document converters for consistent rendering."""
-
-import asyncio
-import tempfile
-from pathlib import Path
-from typing import Optional
-
-from openviking_cli.utils.logger import get_logger
-
-logger = get_logger(__name__)
-
-
-class DocumentConverter:
- """Converts documents to PDF for consistent rendering (DOCX/MD/PPTX -> PDF)."""
-
- def __init__(self, temp_dir: Optional[Path] = None):
- self.temp_dir = temp_dir
-
- async def to_pdf(self, file_path: Path) -> Optional[Path]:
- """Convert document to PDF."""
- ext = file_path.suffix.lower()
-
- if ext == ".pdf":
- return file_path
- elif ext in (".docx", ".pptx"):
- return await self._convert_with_libreoffice(file_path)
- elif ext in (".md", ".markdown"):
- return await self._convert_markdown_to_pdf(file_path)
-
- logger.warning(f"No converter available for {ext}")
- return None
-
- async def _convert_with_libreoffice(self, file_path: Path) -> Optional[Path]:
- """Convert using LibreOffice (soffice)."""
- output_dir = self.temp_dir or Path(tempfile.gettempdir())
- output_path = output_dir / f"{file_path.stem}.pdf"
-
- try:
- cmd = [
- "soffice",
- "--headless",
- "--convert-to",
- "pdf",
- "--outdir",
- str(output_dir),
- str(file_path),
- ]
- process = await asyncio.create_subprocess_exec(
- *cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE
- )
- await process.communicate()
-
- if process.returncode == 0 and output_path.exists():
- return output_path
- return None
- except FileNotFoundError:
- logger.warning("LibreOffice not found")
- return None
- except Exception as e:
- logger.error(f"Conversion error: {e}")
- return None
-
- async def _convert_markdown_to_pdf(self, file_path: Path) -> Optional[Path]:
- """Convert Markdown to PDF using pandoc."""
- output_dir = self.temp_dir or Path(tempfile.gettempdir())
- output_path = output_dir / f"{file_path.stem}.pdf"
-
- try:
- cmd = ["pandoc", str(file_path), "-o", str(output_path), "--pdf-engine=xelatex"]
- process = await asyncio.create_subprocess_exec(
- *cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE
- )
- await process.communicate()
-
- if process.returncode == 0 and output_path.exists():
- return output_path
- return await self._convert_with_libreoffice(file_path)
- except FileNotFoundError:
- return await self._convert_with_libreoffice(file_path)
- except Exception:
- return None
diff --git a/openviking/parse/custom.py b/openviking/parse/custom.py
deleted file mode 100644
index 601d72099..000000000
--- a/openviking/parse/custom.py
+++ /dev/null
@@ -1,244 +0,0 @@
-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
-# SPDX-License-Identifier: AGPL-3.0
-"""
-Custom parser protocol and wrappers for OpenViking.
-
-Supports two ways to extend parsing:
-1. Protocol-based: Implement CustomParserProtocol
-2. Callback-based: Pass a simple async function
-"""
-
-from pathlib import Path
-from typing import TYPE_CHECKING, Callable, List, Optional, Union
-
-from typing_extensions import Protocol, runtime_checkable
-
-if TYPE_CHECKING:
- from openviking.parse.base import ParseResult
-
-
-@runtime_checkable
-class CustomParserProtocol(Protocol):
- """
- Protocol for custom parsers.
-
- External parsers must implement this interface to be registered
- with ParserRegistry.
-
- Example:
- ```python
- class MyCustomParser:
- @property
- def supported_extensions(self) -> List[str]:
- return [".xyz"]
-
- def can_handle(self, source: Union[str, Path]) -> bool:
- return str(source).endswith(".xyz")
-
- async def parse(self, source: Union[str, Path], **kwargs) -> ParseResult:
- # Custom parsing logic
- ...
- ```
- """
-
- def can_handle(self, source: Union[str, Path]) -> bool:
- """
- Check if this parser can handle the given source.
-
- Args:
- source: File path or content string
-
- Returns:
- True if this parser can handle the source
- """
- ...
-
- async def parse(self, source: Union[str, Path], **kwargs) -> "ParseResult":
- """
- Parse the source and return a ParseResult.
-
- Args:
- source: File path or content string
- **kwargs: Additional parsing options
-
- Returns:
- ParseResult with document tree
- """
- ...
-
- @property
- def supported_extensions(self) -> List[str]:
- """
- List of supported file extensions.
-
- Returns:
- List of extensions (e.g., [".xyz", ".abc"])
- """
- ...
-
-
-class CustomParserWrapper:
- """
- Wrapper to adapt external CustomParserProtocol to BaseParser interface.
-
- This allows external parsers to be registered and used seamlessly
- alongside built-in parsers.
- """
-
- def __init__(
- self,
- custom_parser: CustomParserProtocol,
- extensions: Optional[List[str]] = None,
- ):
- """
- Initialize wrapper.
-
- Args:
- custom_parser: External parser implementing CustomParserProtocol
- extensions: Override supported extensions (optional)
- """
- if not isinstance(custom_parser, CustomParserProtocol):
- raise TypeError(
- f"custom_parser must implement CustomParserProtocol, "
- f"got {type(custom_parser).__name__}"
- )
-
- self.custom_parser = custom_parser
- self._extensions = extensions or custom_parser.supported_extensions
-
- @property
- def supported_extensions(self) -> List[str]:
- """Return supported extensions."""
- return self._extensions
-
- def can_parse(self, path: Union[str, Path]) -> bool:
- """Check if can parse the given file."""
- return self.custom_parser.can_handle(path)
-
- async def parse(self, source: Union[str, Path], **kwargs) -> "ParseResult":
- """
- Parse the source using the custom parser.
-
- Args:
- source: File path or content string
- **kwargs: Additional options
-
- Returns:
- ParseResult from custom parser
-
- Raises:
- ValueError: If the custom parser cannot handle this source
- """
- if not self.custom_parser.can_handle(source):
- raise ValueError(
- f"Parser {type(self.custom_parser).__name__} cannot handle source: {source}"
- )
-
- return await self.custom_parser.parse(source, **kwargs)
-
- async def parse_content(
- self, content: str, source_path: Optional[str] = None, **kwargs
- ) -> "ParseResult":
- """
- Parse content string.
-
- Note: Most custom parsers work with file paths, so this may not
- be supported. Override in custom parser if needed.
-
- Args:
- content: Content string
- source_path: Optional source path for reference
- **kwargs: Additional options
-
- Returns:
- ParseResult
-
- Raises:
- NotImplementedError: If custom parser doesn't support content parsing
- """
- raise NotImplementedError(
- f"Parser {type(self.custom_parser).__name__} "
- "does not support content parsing. Use parse() with file path instead."
- )
-
-
-class CallbackParserWrapper:
- """
- Wrapper for simple callback-based parsers.
-
- Allows registering a simple async function as a parser without
- implementing the full CustomParserProtocol.
-
- Example:
- ```python
- async def my_parser(source: Union[str, Path], **kwargs) -> ParseResult:
- root = ResourceNode(type=NodeType.ROOT, title="My Document")
- return create_parse_result(
- root=root,
- source_path=str(source),
- source_format="custom",
- parser_name="my_parser",
- )
-
- registry.register_callback(".xyz", my_parser)
- ```
- """
-
- def __init__(
- self,
- extension: str,
- parse_fn: Callable[[Union[str, Path]], "ParseResult"],
- name: Optional[str] = None,
- ):
- """
- Initialize callback wrapper.
-
- Args:
- extension: File extension (e.g., ".xyz")
- parse_fn: Async function that takes source and returns ParseResult
- name: Optional parser name for identification
- """
- self.extension = extension
- self.parse_fn = parse_fn
- self.name = name or f"callback_{extension}"
-
- @property
- def supported_extensions(self) -> List[str]:
- """Return supported extension."""
- return [self.extension]
-
- def can_parse(self, path: Union[str, Path]) -> bool:
- """Check if can parse the given file."""
- return str(path).lower().endswith(self.extension.lower())
-
- async def parse(self, source: Union[str, Path], **kwargs) -> "ParseResult":
- """
- Parse using the callback function.
-
- Args:
- source: File path
- **kwargs: Additional options passed to callback
-
- Returns:
- ParseResult from callback
- """
- return await self.parse_fn(source, **kwargs)
-
- async def parse_content(
- self, content: str, source_path: Optional[str] = None, **kwargs
- ) -> "ParseResult":
- """
- Parse content - typically not supported for callback parsers.
-
- Args:
- content: Content string
- source_path: Optional source path
- **kwargs: Additional options
-
- Raises:
- NotImplementedError: Callback parsers typically work with files
- """
- raise NotImplementedError(
- f"Callback parser {self.name} does not support content parsing. "
- "Use parse() with file path instead."
- )
diff --git a/openviking/parse/parser_router.py b/openviking/parse/parser_router.py
index 0c2a4a3ae..d629b78ee 100644
--- a/openviking/parse/parser_router.py
+++ b/openviking/parse/parser_router.py
@@ -7,12 +7,10 @@ Routing is controlled by ov.conf (OpenVikingConfig.parser_api).
"""
from pathlib import Path
-from typing import TYPE_CHECKING, Union
+from typing import Union
from urllib.parse import urlparse
-if TYPE_CHECKING:
- from openviking.parse.accessors.base import LocalResource
-
+from openviking.parse.accessors.base import LocalResource, SourceType
from openviking.parse.base import ParseResult
from openviking.parse.registry import ParserRegistry
from openviking_cli.utils.logger import get_logger
@@ -34,10 +32,28 @@ class ParserRouter:
self._parser_registry = parser_registry
self._understanding_api = None
- def should_use_understanding_api(self, source_path: Union[str, Path]) -> bool:
+ def understanding_api_enabled(self) -> bool:
+ """Return whether the external parser is enabled."""
+ try:
+ from openviking_cli.utils.config.open_viking_config import get_openviking_config
+
+ parser_api = getattr(get_openviking_config(), "parser_api", None)
+ except Exception:
+ return False
+ return bool(parser_api and getattr(parser_api, "enable", False))
+
+ def should_use_understanding_api(
+ self,
+ source: Union[str, Path, LocalResource],
+ resolved_extension: str = "",
+ ) -> bool:
"""
Decide whether to use UnderstandingAPI.
"""
+ # FeishuAccessor has already normalized proprietary content to Markdown.
+ if isinstance(source, LocalResource) and source.source_type == SourceType.FEISHU:
+ return False
+
try:
from openviking_cli.utils.config.open_viking_config import get_openviking_config
@@ -45,21 +61,33 @@ class ParserRouter:
except Exception:
return False
- parser_api = ov_config.parser_api
- if not parser_api.enable:
+ parser_api = getattr(ov_config, "parser_api", None)
+ if not parser_api or not getattr(parser_api, "enable", False):
return False
- if parser_api.enable_feishu_url:
- try:
- from openviking.parse.accessors.feishu_accessor import FeishuAccessor
+ source_path = self._extract_source_path(source)
+ try:
+ from openviking.parse.accessors.feishu_accessor import FeishuAccessor
- if FeishuAccessor._is_feishu_url(str(source_path)):
- return True
- except Exception:
- pass
+ if FeishuAccessor._is_feishu_url(str(source_path)):
+ return bool(getattr(parser_api, "enable_feishu_url", False))
+ except Exception:
+ pass
+
+ ext = self._normalize_extension(resolved_extension) or self._extract_extension(source_path)
+ extensions = getattr(parser_api, "extensions", None) or []
+ return ext in extensions
- ext = self._extract_extension(source_path)
- return ext in parser_api.extensions
+ def should_use_understanding_directly(self, source: str, **kwargs) -> bool:
+ forced = kwargs.get("parser_backend") == "understanding"
+ return bool(
+ (forced or self.should_use_understanding_api(source))
+ and self._get_understanding_api().can_submit_url_directly(source, **kwargs)
+ )
+
+ @staticmethod
+ def _normalize_extension(extension: str) -> str:
+ return str(extension or "").lower().lstrip(".")
def _extract_extension(self, source_path: Union[str, Path]) -> str:
source = str(source_path)
@@ -74,7 +102,25 @@ class ParserRouter:
"""
source_path = self._extract_source_path(source)
- if self.should_use_understanding_api(source_path):
+ parser_backend = kwargs.pop("parser_backend", None)
+ if parser_backend not in {None, "internal", "understanding"}:
+ raise ValueError(f"Unknown parser backend: {parser_backend}")
+
+ normalized_feishu = (
+ isinstance(source, LocalResource) and source.source_type == SourceType.FEISHU
+ )
+ use_understanding = not normalized_feishu and (
+ parser_backend == "understanding"
+ or (
+ parser_backend is None
+ and self.should_use_understanding_api(
+ source,
+ resolved_extension=str(kwargs.get("resolved_extension") or ""),
+ )
+ )
+ )
+
+ if use_understanding:
display = source_path
if isinstance(source_path, str) and source_path.startswith(("http://", "https://")):
display = "<url>"
@@ -93,12 +139,15 @@ class ParserRouter:
logger.info(f"[ParserRouter] Using internal ParserRegistry for {display}")
return await self._parser_registry.parse(source_path, **kwargs)
- async def submit_url(self, source: str, **kwargs) -> str:
- if not self.should_use_understanding_api(source):
+ async def submit(self, source: Union[str, Path, LocalResource], **kwargs) -> str:
+ source_path = self._extract_source_path(source)
+ if Path(source_path).is_file():
+ return await self._get_understanding_api().submit_file(source_path)
+ if not self.should_use_understanding_api(str(source_path)):
raise ValueError("source is not routed to UnderstandingAPI")
- return await self._get_understanding_api().submit_url(source, **kwargs)
+ return await self._get_understanding_api().submit_url(str(source_path), **kwargs)
- def _extract_source_path(self, source: Union[str, Path, "LocalResource"]) -> Union[str, Path]:
+ def _extract_source_path(self, source: Union[str, Path, LocalResource]) -> Union[str, Path]:
"""Extract a filesystem path from the source."""
if hasattr(source, "path"):
return source.path
diff --git a/openviking/parse/parsers/README.md b/openviking/parse/parsers/README.md
index 1508cdad7..935181e75 100644
--- a/openviking/parse/parsers/README.md
+++ b/openviking/parse/parsers/README.md
@@ -1,555 +1,10 @@
-# OpenViking 解析系统
+# Built-in parsers
-OpenViking 的解析系统负责将各种格式的原始文档转换为结构化的上下文信息,遵循 L0/L1/L2 三层信息模型。该系统采用模块化设计,支持多种文档格式,并允许用户通过协议扩展自定义解析器。
+OpenViking only routes resources to built-in parsers. Applications add resources through
+`add_resource`; parser registries are internal implementation details.
-## 核心架构
+For the complete Accessor, Parser, Understanding, Connector, and asynchronous execution
+flow, see [Resource ingestion routing](../../../docs/design/resource-ingestion-routing.md).
-### 三层信息模型 (L0/L1/L2)
-
-OpenViking 采用独特的三层信息模型来组织上下文内容,针对不同使用场景提供不同粒度的信息:
-
-| 层级 | 文件 | 大小 | 用途 |
-|------|------|------|------|
-| **L0** | `.abstract.md` | <200 token | 摘要,支持向量化检索、用于目录下的快速浏览 |
-| **L1** | `.overview.md` | <1000 token | 决策参考、内容导览 |
-| **L2** | `content.md` 或原始文件 | 文件完整内容 | 供详细阅读(仅遍历终点)|
-
-**设计出发点**:不同场景需要不同粒度的内容。通过上下文信息的分层,我们允许 Agent 按需读取需要的信息并节省上下文窗口的 token 消耗。
-
-### 举例说明
-
-一份技术文档的三层内容示例:
-
-```
-L0: "这是一份介绍 OpenViking 上下文数据库的技术文档,该文档主要介绍其中的上下文提取流程,涵盖资源解析和信息提取、存储和索引的流程规范,其中重点介绍了..."
-
-L1: """
-# 目录 (Line 1 ~ Line 10)
-目录部分提供了本文的目录结构,共划分为 4 个一级章节,分别是资源解析、信息提取、存储机制、语义索引机制。
-
-# 资源解析 (Line 11 ~ Line 50)
-资源解析部分介绍了如何解析外部知识,包括文档、图片、视频等,包括具体的用例...
-
-# 信息提取 (Line 51 ~ Line 100)
-信息提取部分介绍了如何从资源中提取摘要和导览信息,以文本文件、图片、视频文件等作为具体示例...
-
-# 存储机制 (Line 101 ~ Line 150)
-存储机制部分介绍了如何将上下文存储到 VikingFS 中,包括文件系统、索引结构等。
-
-# 语义索引机制 (Line 151 ~ Line 200)
-语义索引机制介绍了如何将上下文存储到向量库中,支持语义检索。
-"""
-```
-
-### 解析流程 (v5.0 架构)
-
-```
-┌─────────────────────────────────────────────────────────────────────┐
-│ 资源提取流程 (v5.0) │
-├─────────────────────────────────────────────────────────────────────┤
-│ │
-│ ┌─────────┐ ┌─────────────────────────────────────────────┐ │
-│ │ 文件 │───▶│ Parser │ │
-│ │ PDF/MD │ │ ┌─────────────────────────────────────┐ │ │
-│ │ 图片... │ │ │ 解析 + 创建文件和目录结构 │ │ │
-│ └─────────┘ │ │ (无 LLM 调用) │ │ │
-│ │ └──────────────┬──────────────────────┘ │ │
-│ └─────────────────┼───────────────────────────┘ │
-│ │ │
-│ ▼ ParseResult │
-│ │ (temp_dir_path) │
-│ ┌─────────────────┴───────────────────────────┐ │
-│ │ TreeBuilder │ │
-│ │ ┌─────────┐ ┌──────────────────────────┐ │ │
-│ │ │ 1.移动 │─▶│ 2.入队 SemanticQueue │ │ │
-│ │ │ 到AGFS │ │ (自底向上处理) │ │ │
-│ │ └─────────┘ └──────────────────────────┘ │ │
-│ └─────────────────────┬───────────────────────┘ │
-│ │ │
-│ ▼ BuildingTree │
-│ ┌─────────────────────┴───────────────────────┐ │
-│ │ SemanticQueue (异步) │ │
-│ │ ┌─────────────────────────────────────┐ │ │
-│ │ │ SemanticProcessor (自底向上): │ │ │
-│ │ │ 1. 收集子目录 abstract │ │ │
-│ │ │ 2. 生成文件 summary (并发LLM) │ │ │
-│ │ │ 3. 生成 .abstract.md (L0) │ │ │
-│ │ │ 4. 生成 .overview.md (L1) │ │ │
-│ │ │ 5. 直接调用向量化写入 │ │ │
-│ │ └─────────────────────────────────────┘ │ │
-│ └─────────────────────┬───────────────────────┘ │
-│ │ │
-│ ▼ │
-│ ┌─────────────────────┴───────────────────────┐ │
-│ │ AGFS + 向量库 │ │
-│ │ L0/L1/L2 文件 + 向量化索引 │ │
-│ └─────────────────────────────────────────────┘ │
-│ │
-└─────────────────────────────────────────────────────────────────────┘
-```
-
-## 解析器类型
-
-### 1. MarkdownParser (`markdown.py`)
-**支持格式**: `.md`, `.markdown`, `.mdown`, `.mkd`
-
-核心文档解析器,所有文本类文档最终都通过 MarkdownParser 处理。采用 v5.0 简化架构:
-
-#### 解析逻辑(单阶段)
-- 按标题层级(#, ##, ###)拆分章节,支持混合目录结构(文件 + 子目录)
-- 切分规则:
- - 小文件(< 4000 tokens):保留原文件名,直接保存
- - 大文件(> 4000 tokens):按章节切分
- - 章节有子章节且总大小 > 4000:章节变目录,直接内容视为第一个虚拟子章节参与合并
- - 章节无子章节但超长:创建目录,按段落切分为 `章节名_1.md`, `章节名_2.md`
- - 正常章节:保存为 `章节名.md`
-- 小章节合并规则(< 800 tokens):
- - 连续的小章节会合并,直到总大小 >= 800 tokens 或无法与下一个章节合并
- - 合并后文件名用下划线拼接所有章节名:`章节A_章节B_章节C.md`
- - 直接内容(标题前或子标题前的内容)视为虚拟章节,使用父级名称参与合并
-- 直接写入临时目录的文件和目录结构
-- **无 LLM 调用**:语义生成移到 SemanticQueue 异步处理
-
-#### 语义生成(异步,由 SemanticProcessor 处理)
-- TreeBuilder 将目录移到 AGFS 后,入队到 SemanticQueue
-- SemanticProcessor 自底向上处理每个目录:
- - 收集子目录的 `.abstract.md`
- - 并发生成文件 summary(LLM)
- - 生成当前目录的 `.abstract.md` (L0) 和 `.overview.md` (L1)
- - 直接调用向量化写入
-
-### 2. PDFParser (`pdf.py`)
-**支持格式**: `.pdf`
-
-统一 PDF 解析器,采用双策略转换:
-- **本地策略**: 使用 `pdfplumber` 进行文本和表格提取
-- **远程策略**: 使用 `MinerU API` 进行高级 PDF 处理
-- **自动策略**: 先尝试本地,失败时回退到 MinerU(如果配置了)
-
-处理流程:`PDF → Markdown → ParseResult`,最终通过 MarkdownParser 处理。
-
-### 3. HTMLParser (`html.py`)
-**支持格式**: `.html`, `.htm`
-
-使用 `trafilatura` 抽取正文,去除广告、导航等噪音元素,转换为 Markdown 后通过 MarkdownParser 处理。
-
-### 4. TextParser (`text.py`)
-**支持格式**: `.txt`, `.text`, `.log`, `.csv`, `.tsv`, `.json`, `.yaml`, `.yml`, `.xml`, `.ini`, `.cfg`, `.conf`
-
-纯文本解析器,支持多种文本格式。对于结构化格式(JSON、YAML、XML)会尝试提取结构化信息。
-
-### 5. CodeRepositoryParser (`code/*`)
-**支持来源**: github 代码仓库等
-
-代码解析器,支持语法高亮和代码结构分析。能识别函数、类、方法等代码元素,遵循 `.gitignore` 并忽略常见非代码目录。
-
-### 6. MediaParser (`media/*.py`)
-
-**支持格式**:
-- 图片: `.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`
-- 视频: `.mp4`, `.mov`, `.avi`, `.webm`
-- 音频: `.mp3`, `.wav`, `.m4a`, `.flac`
-
-多媒体解析器,使用 VLM(视觉语言模型)分析图像、视频和音频内容,生成文本描述。多媒体解析器当且仅当 add-resource 调用时只添加上述文件类型时生效。即:
-1. 当添加目录时,系统将对多媒体文件暂不生成单独目录和文本描述,仅存储和进行递归摘要。
-2. 当单独添加多媒体文件时,多媒体解析器会直接解析该文件,然后通过单独目录存放,在目录下生成文本描述。
-
-#### 存储组织策略
-
-多媒体文件的存储采用以下层级结构:
-
-```
-viking://resource/
-├── images/ # 图片文件
-│ └── 20240820/ # 上传日期(YYYYMMDD)
-│ └── 20240820_123456_jpg/ # 文件文件夹(文件名_扩展名)
-│ ├── .abstract.md # L0 摘要
-│ ├── .overview.md # L1 概览
-│ └── 20240820_123456.jpg # 原始文件
-├── audio/ # 音频文件
-│ └── 20240820/
-│ └── my_song_mp3/
-│ ├── .abstract.md
-│ ├── .overview.md
-│ └── my_song.mp3
-└── video/ # 视频文件
- └── 20240820/
- └── my_video_mp4/
- ├── .abstract.md
- ├── .overview.md
- └── my_video.mp4
-```
-
-详细说明:
-
-1. **媒体子目录**: 在 `viking://resource` 下按类型划分为三个子目录
- - `viking://resource/images`: 存储未明确指定目标路径的图片文件
- - `viking://resource/audio`: 存储未明确指定目标路径的音频文件
- - `viking://resource/video`: 存储未明确指定目标路径的视频文件
-
-2. **日期子目录**: 每次上传的文件按当前日期(格式:YYYYMMDD)组织,而非文件内部元信息的时间
- - 例如:`viking://resource/images/20240820/` 存储 2024年8月20日上传的所有图片
-
-3. **文件文件夹**: 为每个多媒体文件创建一个专属文件夹,命名规则为:`文件名_扩展名`(扩展名不含点)
- - 示例:上传 `20240820_123456.jpg` → 创建文件夹 `20240820_123456_jpg`
-
-4. **文件夹内容**: 每个文件文件夹内包含:
- - **原始文件**: 保持原始文件名,空格字符替换为下划线 `_`(因为 OpenViking URI 不允许包含空格)
- - 示例:`photo 1.jpg` → 保存为 `photo_1.jpg`
- - `.abstract.md` (L0 层): 摘要信息(<200 token)
- - 图片:文件名、内容描述、画面风格等
- - 音频:文件名、时长、内容概述等
- - 视频:文件名、时长、内容概述等
- - `.overview.md` (L1 层): 概览信息
- - 图片:除摘要外,还包含尺寸、OCR 识别结果、场景和主体描述等
- - 音频:除摘要外,还包含语音/歌词识别结果、章节时间线等
- - 视频:除摘要外,还包含使用场景等(未来会支持切分视频,递归存储子文件)
-
-
-## 核心组件
-
-### BaseParser (`base_parser.py`)
-所有解析器的抽象基类,定义了解析器的标准接口:
-
-```python
-class BaseParser(ABC):
- @abstractmethod
- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:
- """从文件路径或内容字符串解析文档"""
- pass
-
- @abstractmethod
- async def parse_content(self, content: str, source_path: Optional[str] = None, instruction: str = "", **kwargs) -> ParseResult:
- """直接解析文档内容"""
- pass
-
- @property
- @abstractmethod
- def supported_extensions(self) -> List[str]:
- """支持的文件扩展名列表"""
- pass
-```
-
-### ParserRegistry (`registry.py`)
-解析器注册表,提供自动解析器选择和注册机制:
-
-```python
-class ParserRegistry:
- def __init__(self, register_optional: bool = True):
- """初始化注册表并注册默认解析器"""
- self._parsers: Dict[str, BaseParser] = {}
- self._extension_map: Dict[str, str] = {}
-
- # 注册核心解析器
- self.register("text", TextParser())
- self.register("markdown", MarkdownParser())
- self.register("pdf", PDFParser())
- self.register("html", HTMLParser())
- self.register("code", CodeRepositoryParser())
-
- def get_parser_for_file(self, path: Union[str, Path]) -> Optional[BaseParser]:
- """根据文件扩展名获取合适的解析器"""
- pass
-
- def register_custom(self, handler: "CustomParserProtocol", extensions: Optional[List[str]] = None, name: Optional[str] = None):
- """注册自定义解析器(协议方式)"""
- pass
-```
-
-### TreeBuilder (`tree_builder.py`)
-树构建器,负责将临时目录移动到 AGFS 并入队语义处理:
-
-```python
-class TreeBuilder:
- async def finalize_from_temp(
- self,
- temp_dir_path: str,
- scope: str,
- base_uri: Optional[str] = None,
- source_path: Optional[str] = None,
- source_format: Optional[str] = None,
- ) -> "BuildingTree":
- """
- 从临时目录最终化树结构:
- 1. 移动文件到 AGFS
- 2. 入队到 SemanticQueue(自底向上处理)
- 3. 扫描创建 Resource 对象
- """
- pass
-```
-
-### 资源类型
-解析系统使用统一的 Resource 类型。
-
-## 使用示例
-
-### 基本使用
-
-```python
-from openviking.parse.parsers.markdown import MarkdownParser
-from openviking.parse.registry import get_registry
-
-# 方式1:直接使用解析器
-parser = MarkdownParser()
-result = await parser.parse("document.md")
-
-# 方式2:通过注册表自动选择
-registry = get_registry()
-result = await registry.parse("document.pdf") # 自动选择 PDFParser
-```
-
-### 自定义解析器
-
-```python
-from openviking.parse.parsers.base_parser import BaseParser
-from openviking.parse.base import ParseResult, ResourceNode, NodeType, create_parse_result
-
-class CustomParser(BaseParser):
- @property
- def supported_extensions(self) -> List[str]:
- return [".xyz"]
-
- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:
- content = self._read_file(source)
- root = ResourceNode(type=NodeType.ROOT, title="Custom Document")
- return create_parse_result(root=root, source_path=str(source), source_format="custom")
-
-# 注册自定义解析器
-registry = get_registry()
-registry.register("custom", CustomParser())
-```
-
-### 协议方式扩展
-
-```python
-from openviking.parse.custom import CustomParserProtocol
-from typing import List
-
-class MyParser:
- @property
- def supported_extensions(self) -> List[str]:
- return [".myformat"]
-
- def can_handle(self, source) -> bool:
- return str(source).endswith(".myformat")
-
- async def parse(self, source, **kwargs):
- # 实现解析逻辑
- pass
-
-# 注册(自动包装为 BaseParser)
-registry.register_custom(MyParser(), name="my_parser")
-```
-
-## 扩展指南
-
-### 1. 继承 BaseParser(推荐)
-适合需要完整控制解析流程的场景:
-
-```python
-from openviking.parse.parsers.base_parser import BaseParser
-
-class MyParser(BaseParser):
- @property
- def supported_extensions(self) -> List[str]:
- return [".myformat"]
-
- async def parse(self, source, instruction="", **kwargs):
- # 实现三阶段解析
- # 1. 解析原始内容
- # 2. 生成 L0/L1
- # 3. 写入临时目录
- pass
-
- async def parse_content(self, content, source_path=None, instruction="", **kwargs):
- # 类似实现
- pass
-```
-
-### 2. 实现 CustomParserProtocol
-适合已有解析逻辑,需要快速集成:
-
-```python
-from openviking.parse.custom import CustomParserProtocol
-
-class ExistingParser:
- def parse_file(self, path):
- # 现有解析逻辑
- pass
-
-# 适配器模式
-class Adapter(CustomParserProtocol):
- def __init__(self, existing_parser):
- self.parser = existing_parser
-
- @property
- def supported_extensions(self):
- return [".existing"]
-
- def can_handle(self, source):
- return True
-
- async def parse(self, source, **kwargs):
- result = self.parser.parse_file(source)
- # 转换为 ParseResult
- pass
-```
-
-### 3. 回调函数方式
-适合简单转换场景:
-
-```python
-from openviking.parse.registry import ParserRegistry
-
-async def simple_parser(source, **kwargs):
- content = Path(source).read_text()
- # 简单处理
- return create_parse_result(...)
-
-registry = get_registry()
-registry.register_callback(".simple", simple_parser, name="simple_parser")
-```
-
-## 设计原则
-
-### 1. 解析与语义分离原则
-- 解析器完成:解析 + 文件/目录创建(零 LLM 调用)
-- 语义生成:异步 SemanticQueue 处理(自底向上)
-- TreeBuilder 只负责文件移动和入队
-- 减少解析阻塞,降低内存压力
-
-### 2. 异步语义生成
-- Parser 输出临时目录后即可返回
-- SemanticQueue 后台处理语义信息生成
-- 支持并发文件 summary 生成
-- 直接向量化写入,不经过 EmbeddingQueue
-
-### 3. 混合目录结构
-- 一个目录下可同时包含文件和子目录
-- 章节有子章节时,直接内容保存为同名文件
-- 灵活的内容组织,符合自然文档结构
-
-### 4. 临时目录架构
-- 每个解析器在临时目录中构建完整结构
-- TreeBuilder 只负责移动和入队
-- 支持并发解析,避免内存冲突
-
-### 5. 统一 Markdown 处理
-- 所有文档格式先转成 Markdown
-- 统一通过 MarkdownParser 处理结构
-- 保证一致的解析质量和输出格式
-
-### 6. 轻量级索引
-- Context 对象不存储内容,只存 URI 和元数据
-- 内容通过 `get_abstract()` / `get_overview()` / `get_detail()` 按需加载
-- 加载后缓存在内存中,避免重复读取
-
-## 存储结构
-
-### 最终 AGFS 结构
-
-混合目录结构示例(章节有子章节时,小章节合并):
-
-```
-viking://resources/Python_异步编程指南/
-├── .abstract.md # L0: 目录摘要
-├── .overview.md # L1: 目录概览
-├── Python_异步编程指南_第一章_asyncio_基础.md # 直接内容 + 第一章合并(均 < 800 tokens)
-├── 第二章_高级模式/ # > 4000 tokens 且有子章节,变目录
-│ ├── .abstract.md
-│ ├── .overview.md
-│ ├── 第二章_高级模式_并发控制.md # 直接内容 + 并发控制合并
-│ └── 错误处理.md # 单独保存(>= 800 tokens)
-└── 附录.md # < 4000 tokens,直接文件
-```
-
-超大章节无子章节时的切分:
-
-```
-第二章_高级模式/
-├── .abstract.md
-├── .overview.md
-├── 第二章_高级模式_1.md # 按段落切分
-├── 第二章_高级模式_2.md
-└── 第二章_高级模式_3.md
-```
-
-### 临时目录结构
-
-```
-viking://temp/abc123/
-└── document/
- ├── Python_异步编程指南.md
- ├── 第一章_asyncio_基础.md
- ├── 第二章_高级模式/
- │ ├── 第二章_高级模式.md
- │ ├── 并发控制.md
- │ └── 错误处理.md
- └── 附录.md
-```
-
-注:临时目录只包含文件和目录结构,`.abstract.md` 和 `.overview.md` 由 SemanticProcessor 异步生成。
-
-## 性能优化
-
-### 1. 并发处理
-- 支持多个文档同时解析
-- 异步 I/O 操作
-- 智能资源调度
-
-### 2. 缓存机制
-- 解析结果缓存
-- 向量化结果复用
-- 内容按需加载
-
-### 3. 增量更新
-- 支持文档部分更新
-- 智能重新解析
-- 最小化计算开销
-
-## 故障排除
-
-### 常见问题
-
-1. **解析失败**
- - 检查文件格式是否支持
- - 验证文件完整性
- - 检查依赖库版本
-
-2. **LLM 调用失败**
- - 检查 API 密钥配置
- - 验证网络连接
- - 检查请求配额
-
-3. **内存不足**
- - 启用临时目录模式
- - 减少并发解析数量
- - 优化解析策略
-
-### 调试建议
-
-```python
-import logging
-logging.basicConfig(level=logging.DEBUG)
-
-# 启用详细日志
-parser = MarkdownParser()
-result = await parser.parse("test.md", debug=True)
-```
-
-## 版本历史
-
-| 版本 | 日期 | 更新内容 |
-|------|------|----------|
-| v5.0 | 2026-01-19 | 解析与语义分离,引入 SemanticQueue,支持混合目录结构 |
-| v4.0 | 2026-01-07 | 引入临时目录架构,解析器自包含设计 |
-| v3.0 | 2026-01-07 | 统一 Markdown 处理流程 |
-| v2.0 | 2026-01-05 | 增加多媒体解析器支持 |
-| v1.0 | 2026-01-04 | 初始解析系统架构 |
-
-## 相关文档
-
-- [OpenViking 整体架构](../../../docs/zh/concepts/01-architecture.md)
-- [上下文提取流程](../../../docs//zh/concepts/07-extraction.md)
-- [存储系统设计](../../../docs/zh/concepts/05-storage.md)
-- [配置指南](../../../docs/zh/configuration/configuration.md)
+Each built-in parser implements `BaseParser` and returns a `ParseResult`. New formats are
+added to the repository as built-in parsers and registered in `openviking/parse/registry.py`.
diff --git a/openviking/parse/parsers/feishu.py b/openviking/parse/parsers/feishu.py
deleted file mode 100644
index dfa8ef732..000000000
--- a/openviking/parse/parsers/feishu.py
+++ /dev/null
@@ -1,907 +0,0 @@
-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
-# SPDX-License-Identifier: AGPL-3.0
-"""
-Feishu/Lark cloud document parser for OpenViking.
-
-Supports:
-- Documents: https://*.feishu.cn/docx/{document_id}
-- Wiki pages: https://*.feishu.cn/wiki/{token}
-- Spreadsheets: https://*.feishu.cn/sheets/{token}
-- Bitable: https://*.feishu.cn/base/{app_token}
-
-All types are converted to Markdown then parsed via MarkdownParser.
-"""
-
-import asyncio
-import json
-import os
-import time
-from pathlib import Path
-from typing import Any, Dict, List, Optional, Tuple, Union
-from urllib.parse import urlparse
-
-from openviking.parse.base import (
- NodeType,
- ParseResult,
- ResourceNode,
- create_parse_result,
- format_table_to_markdown,
-)
-from openviking.parse.parsers.base_parser import BaseParser
-from openviking_cli.utils.config.parser_config import FeishuConfig
-from openviking_cli.utils.logger import get_logger
-
-logger = get_logger(__name__)
-
-_ALLOWED_FEISHU_HOSTS = ("feishu.cn", "larksuite.com", "larkoffice.com")
-
-
-def _getattr_safe(obj, key: str, default=None):
- """Get attribute from SDK object or dict, with safe fallback."""
- if isinstance(obj, dict):
- return obj.get(key, default)
- return getattr(obj, key, default)
-
-
-def _is_allowed_feishu_url(source_path: str) -> bool:
- """Return True for Feishu/Lark URLs on allowed hosts."""
- parsed = urlparse(source_path)
- hostname = (parsed.hostname or "").lower().rstrip(".")
- return any(
- hostname == allowed_host or hostname.endswith(f".{allowed_host}")
- for allowed_host in _ALLOWED_FEISHU_HOSTS
- )
-
-
-class FeishuParser(BaseParser):
- """Parser for Feishu/Lark cloud documents.
-
- Block type detection uses a two-tier strategy:
- 1. Primary: block_type integer → attribute name lookup via _BLOCK_TYPE_TO_ATTR
- (reliable, not affected by SDK changes)
- 2. Fallback: scan _KNOWN_CONTENT_ATTRS whitelist for unknown block types
- (auto-compat for new types without code changes)
- """
-
- # Attributes that skip processing (structural containers or metadata)
- _SKIP_ATTRS = {"page", "table_cell", "quote_container", "grid", "grid_column"}
-
- # Attribute → special handler method (non-text blocks)
- _SPECIAL_BLOCK_HANDLERS = {
- "divider": "_handle_divider",
- "image": "_handle_image",
- "table": "_table_block_to_markdown",
- "sheet": "_embedded_sheet_to_markdown",
- }
-
- # Attribute → markdown prefix template for text-bearing blocks.
- # "{text}" is replaced with extracted text content.
- # Headings are handled dynamically (heading1-heading9 → # through #########).
- _TEXT_FORMAT = {
- "bullet": "- {text}",
- "quote": "> {text}",
- }
-
- # Known block_type integer → SDK attribute name mapping.
- # Primary dispatch mechanism for reliable block detection.
- # Source: Feishu OpenAPI documentation + lark-oapi SDK Block class.
- _BLOCK_TYPE_TO_ATTR = {
- 1: "page",
- 2: "text",
- 3: "heading1",
- 4: "heading2",
- 5: "heading3",
- 6: "heading4",
- 7: "heading5",
- 8: "heading6",
- 9: "heading7",
- 10: "heading8",
- 11: "heading9",
- 12: "bullet",
- 13: "ordered",
- 14: "code",
- 15: "quote",
- 17: "todo",
- 18: "bitable",
- 19: "callout",
- 22: "divider",
- 24: "file",
- 27: "image",
- 30: "sheet",
- 31: "table",
- 32: "table_cell",
- 34: "quote_container",
- }
-
- # All known content attribute names on SDK Block objects (for fallback detection).
- _KNOWN_CONTENT_ATTRS = frozenset(
- {
- "page",
- "text",
- "heading1",
- "heading2",
- "heading3",
- "heading4",
- "heading5",
- "heading6",
- "heading7",
- "heading8",
- "heading9",
- "bullet",
- "ordered",
- "code",
- "quote",
- "todo",
- "callout",
- "divider",
- "image",
- "table",
- "table_cell",
- "quote_container",
- "sheet",
- "file",
- "bitable",
- "equation",
- "task",
- "grid",
- "grid_column",
- "iframe",
- "board",
- "chat_card",
- "diagram",
- "agenda",
- "agenda_item",
- "agenda_item_content",
- "agenda_item_title",
- "ai_template",
- "isv",
- "jira_issue",
- "link_preview",
- "meeting_notes_qa",
- "mindnote",
- "okr",
- "okr_key_result",
- "okr_objective",
- "okr_progress",
- "project",
- "reference_base",
- "reference_synced",
- "source_synced",
- "sub_page_list",
- "undefined",
- "view",
- "wiki_catalog",
- }
- )
-
- # Document type → parse method name mapping.
- # Wiki nodes are resolved to one of these types via _resolve_wiki_node.
- # New types can be supported by adding an entry here and the corresponding method.
- _DOC_TYPE_HANDLERS = {
- "docx": "_parse_docx",
- "sheets": "_parse_sheets",
- "base": "_parse_bitable",
- }
-
- # Wiki obj_type normalization (API returns short names)
- _WIKI_TYPE_MAP = {"doc": "docx", "sheet": "sheets", "bitable": "base"}
-
- def __init__(self, config: Optional[FeishuConfig] = None):
- self._client = None
- self._config = config
- self._markdown_parser = None
-
- @property
- def supported_extensions(self) -> List[str]:
- return [] # URL-based parser, no file extensions
-
- # ========== Configuration & Client ==========
-
- def _get_config(self):
- """Get FeishuConfig from OpenViking config."""
- if self._config is None:
- from openviking_cli.utils.config import get_openviking_config
-
- self._config = get_openviking_config().feishu
- return self._config
-
- def _get_markdown_parser(self):
- """Lazy import and create MarkdownParser with Feishu parser config."""
- if self._markdown_parser is None:
- from openviking.parse.parsers.markdown import MarkdownParser
-
- self._markdown_parser = MarkdownParser(config=self._get_config())
- return self._markdown_parser
-
- def _get_client(self):
- """Lazy-init lark-oapi client."""
- if self._client is None:
- try:
- import lark_oapi as lark
- except ImportError:
- raise ImportError(
- "lark-oapi is required for Feishu document parsing. "
- "Install it with: pip install lark-oapi>=1.0.0"
- )
- config = self._get_config()
- app_id = config.app_id or os.getenv("FEISHU_APP_ID", "")
- app_secret = config.app_secret or os.getenv("FEISHU_APP_SECRET", "")
- if not app_id or not app_secret:
- raise ValueError(
- "Feishu credentials not configured. Set FEISHU_APP_ID and "
- "FEISHU_APP_SECRET environment variables, or configure in ov.conf."
- )
- domain = config.domain or "https://open.feishu.cn"
- self._client = (
- lark.Client.builder().app_id(app_id).app_secret(app_secret).domain(domain).build()
- )
- return self._client
-
- # ========== URL Parsing ==========
-
- @staticmethod
- def _parse_feishu_url(url: str) -> Tuple[str, str]:
- """
- Extract doc_type and token from Feishu URL.
-
- Returns:
- (doc_type, token) e.g. ("docx", "doxcnABC123")
- """
- if not _is_allowed_feishu_url(url):
- raise ValueError(f"Feishu host not allowed: {url}")
-
- parsed = urlparse(url)
- path_parts = [p for p in parsed.path.split("/") if p]
- if len(path_parts) < 2:
- raise ValueError(f"Cannot parse Feishu URL: {url}")
- doc_type = path_parts[0] # docx, wiki, sheets, base
- token = path_parts[1]
- return doc_type, token
-
- # ========== Main Parse ==========
-
- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:
- """Parse a Feishu cloud document URL."""
- url = str(source)
- start_time = time.time()
-
- try:
- doc_type, token = self._parse_feishu_url(url)
-
- title = None
- if doc_type == "wiki":
- real_type, real_token, title = await asyncio.to_thread(
- self._resolve_wiki_node, token
- )
- doc_type, token = real_type, real_token
-
- handler_name = self._DOC_TYPE_HANDLERS.get(doc_type)
- if not handler_name:
- raise ValueError(
- f"Unsupported Feishu document type: {doc_type}. "
- f"Supported: {list(self._DOC_TYPE_HANDLERS.keys())}"
- )
- markdown, _doc_title = await asyncio.to_thread(getattr(self, handler_name), token)
-
- md_parser = self._get_markdown_parser()
- result = await md_parser.parse_content(
- markdown, source_path=url, instruction=instruction, **kwargs
- )
- result.source_format = f"feishu_{doc_type}"
- result.parser_name = "FeishuParser"
- result.parse_time = time.time() - start_time
- result.meta["feishu_doc_type"] = doc_type
- result.meta["feishu_token"] = token
- return result
-
- except Exception as e:
- logger.error(f"[FeishuParser] Failed to parse {url}: {e}")
- return create_parse_result(
- root=ResourceNode(type=NodeType.ROOT),
- source_path=url,
- source_format="feishu",
- parser_name="FeishuParser",
- parse_time=time.time() - start_time,
- warnings=[f"Feishu parse failed: {e}"],
- )
-
- async def parse_content(
- self,
- content: str,
- source_path: Optional[str] = None,
- instruction: str = "",
- **kwargs,
- ) -> ParseResult:
- """Not typically used for Feishu (URL-based parser)."""
- if source_path and _is_allowed_feishu_url(source_path):
- return await self.parse(source_path, instruction=instruction, **kwargs)
- raise NotImplementedError("FeishuParser requires a Feishu URL. Use parse() instead.")
-
- # ========== Wiki Resolution ==========
-
- def _resolve_wiki_node(self, token: str) -> Tuple[str, str, Optional[str]]:
- """
- Resolve wiki token to actual document type, token, and title.
-
- Returns:
- (doc_type, obj_token, title)
- """
- from lark_oapi.api.wiki.v2 import GetNodeSpaceRequest
-
- client = self._get_client()
- request = GetNodeSpaceRequest.builder().token(token).build()
- response = client.wiki.v2.space.get_node(request)
- if not response.success():
- raise RuntimeError(
- f"Failed to resolve wiki node {token}: code={response.code}, msg={response.msg}"
- )
- node = response.data.node
- obj_type = node.obj_type or ""
- obj_token = node.obj_token or ""
- title = node.title
-
- # Normalize type names
- doc_type = self._WIKI_TYPE_MAP.get(obj_type, obj_type)
-
- return doc_type, obj_token, title
-
- # ========== Docx Parsing ==========
-
- def _parse_docx(self, document_id: str) -> Tuple[str, str]:
- """
- Fetch all blocks and convert to Markdown.
-
- Returns:
- (markdown_content, document_title)
- """
- blocks = self._fetch_all_blocks(document_id)
- if not blocks:
- return "", "Untitled"
-
- # Build block lookup by block_id
- block_map = {b.block_id: b for b in blocks}
-
- # Find title from page block
- doc_title = "Untitled"
- for b in blocks:
- if b.page is not None:
- if b.page.elements:
- doc_title = self._extract_text_from_elements(b.page.elements)
- break
-
- # Convert blocks to markdown
- markdown_lines = []
- ordered_counter: Dict[str, int] = {}
-
- for block in blocks:
- if block.page is not None:
- continue # Skip page container
-
- line = self._block_to_markdown(
- block, block_map, ordered_counter, document_id=document_id
- )
- if line is not None:
- markdown_lines.append(line)
-
- markdown = "\n\n".join(markdown_lines)
-
- if doc_title and doc_title != "Untitled":
- markdown = f"# {doc_title}\n\n{markdown}"
-
- return markdown, doc_title
-
- def _fetch_all_blocks(self, document_id: str) -> list:
- """Fetch all blocks with pagination. Returns list of SDK block objects."""
- from lark_oapi.api.docx.v1 import ListDocumentBlockRequest
-
- client = self._get_client()
- all_blocks = []
- page_token = None
-
- while True:
- builder = (
- ListDocumentBlockRequest.builder()
- .document_id(document_id)
- .page_size(500)
- .document_revision_id(-1)
- )
- if page_token:
- builder = builder.page_token(page_token)
-
- request = builder.build()
- response = client.docx.v1.document_block.list(request)
-
- if not response.success():
- raise RuntimeError(
- f"Failed to fetch blocks for {document_id}: "
- f"code={response.code}, msg={response.msg}"
- )
-
- items = response.data.items or []
- all_blocks.extend(items)
-
- if not response.data.has_more:
- break
- page_token = response.data.page_token
-
- return all_blocks
-
- # ========== Block -> Markdown Conversion ==========
-
- def _detect_block_attr(self, block) -> Optional[str]:
- """Detect which content attribute is populated on a block object.
-
- Uses block_type integer as the primary dispatch (reliable), falling
- back to attribute inspection over a known whitelist for unknown types.
- """
- # Primary: lookup by block_type integer
- block_type = getattr(block, "block_type", None)
- if block_type is not None:
- attr = self._BLOCK_TYPE_TO_ATTR.get(block_type)
- if attr:
- return attr
-
- # Fallback: scan known content attributes for unknown block types
- for attr in self._KNOWN_CONTENT_ATTRS:
- if getattr(block, attr, None) is not None:
- return attr
- return None
-
- def _block_to_markdown(
- self, block, block_map: Dict, ordered_counter: Dict[str, int], document_id: str = ""
- ) -> Optional[str]:
- """Convert a single SDK block object to markdown string.
-
- Uses block_type integer for primary dispatch, with attribute whitelist
- fallback for unknown types. Formatting is data-driven via _TEXT_FORMAT
- and _SPECIAL_BLOCK_HANDLERS tables.
- """
- attr = self._detect_block_attr(block)
-
- if attr is None:
- return None
-
- # Skip structural containers (processed via their children)
- if attr in self._SKIP_ATTRS:
- return None
-
- # Reset ordered list counter when any non-ordered block appears
- if attr != "ordered":
- parent_id = block.parent_id or ""
- if parent_id in ordered_counter:
- del ordered_counter[parent_id]
-
- # Special blocks (non-text: divider, image, table, sheet)
- special_handler = self._SPECIAL_BLOCK_HANDLERS.get(attr)
- if special_handler:
- return getattr(self, special_handler)(block, block_map, document_id=document_id)
-
- # --- Text-bearing blocks: extract elements, apply formatting ---
- content_obj = getattr(block, attr, None)
- if not content_obj or not hasattr(content_obj, "elements") or not content_obj.elements:
- return None
-
- text = self._extract_text_from_elements(content_obj.elements)
- if not text:
- return None
-
- # Headings: heading1 -> #, heading2 -> ##, ...
- if attr.startswith("heading"):
- level = int(attr.replace("heading", "") or "1")
- return f"{'#' * level} {text}"
-
- # Ordered list (needs counter state)
- if attr == "ordered":
- parent_id = block.parent_id or ""
- counter = ordered_counter.get(parent_id, 0) + 1
- ordered_counter[parent_id] = counter
- return f"{counter}. {text}"
-
- # Code block (needs language from style)
- if attr == "code":
- lang = ""
- if hasattr(content_obj, "style") and content_obj.style:
- lang = str(getattr(content_obj.style, "language", "") or "")
- return f"```{lang}\n{text}\n```"
-
- # Todo (needs done state from style)
- if attr == "todo":
- done = False
- if hasattr(content_obj, "style") and content_obj.style:
- done = getattr(content_obj.style, "done", False)
- checkbox = "[x]" if done else "[ ]"
- return f"- {checkbox} {text}"
-
- # Simple template formatting (bullet, quote, etc.)
- fmt = self._TEXT_FORMAT.get(attr)
- if fmt:
- return fmt.format(text=text)
-
- # Default: return plain text (covers callout, equation, task, unknown, etc.)
- return text
-
- @staticmethod
- def _handle_divider(block, block_map: Dict = None, **_) -> str:
- """Convert divider block to markdown."""
- return "---"
-
- @staticmethod
- def _handle_image(block, block_map: Dict = None, **_) -> Optional[str]:
- """Convert image block to markdown."""
- image = block.image
- if not image:
- return None
- file_token = image.token or ""
- alt_text = getattr(image, "alt", "") or "image"
- return f"![{alt_text}](feishu://image/{file_token})"
-
- def _extract_block_text(self, block, attr_name: str) -> str:
- """Extract text from a block's named attribute (e.g. block.text, block.heading2)."""
- content_obj = getattr(block, attr_name, None)
- if content_obj and hasattr(content_obj, "elements") and content_obj.elements:
- return self._extract_text_from_elements(content_obj.elements)
- return ""
-
- def _extract_text_from_elements(self, elements) -> str:
- """Convert Feishu TextElement SDK objects to formatted text."""
- if not elements:
- return ""
- parts = []
- for element in elements:
- # TextRun
- text_run = element.text_run
- if text_run:
- content = text_run.content or ""
- style = text_run.text_element_style
- content = self._apply_text_style(content, style)
- parts.append(content)
- continue
-
- # MentionUser
- mention_user = element.mention_user
- if mention_user:
- user_id = _getattr_safe(mention_user, "user_id", "user")
- parts.append(f"@{user_id}")
- continue
-
- # MentionDoc
- mention_doc = element.mention_doc
- if mention_doc:
- title = _getattr_safe(mention_doc, "title", "document")
- url = _getattr_safe(mention_doc, "url", "")
- parts.append(f"[{title}]({url})" if url else str(title))
- continue
-
- # Equation
- equation = element.equation
- if equation:
- parts.append(f"${_getattr_safe(equation, 'content', '')}$")
- continue
-
- return "".join(parts)
-
- @staticmethod
- def _apply_text_style(text: str, style) -> str:
- """Apply markdown formatting based on TextElementStyle SDK object."""
- if not text or not style:
- return text
- # inline_code (SDK uses 'inline_code', not 'code_inline')
- if getattr(style, "inline_code", False):
- return f"`{text}`"
- # link
- link = getattr(style, "link", None)
- if link:
- url = _getattr_safe(link, "url", "")
- if url:
- text = f"[{text}]({url})"
- if getattr(style, "bold", False):
- text = f"**{text}**"
- if getattr(style, "italic", False):
- text = f"*{text}*"
- if getattr(style, "strikethrough", False):
- text = f"~~{text}~~"
- return text
-
- def _table_block_to_markdown(self, block, block_map: Dict, **_) -> Optional[str]:
- """Convert table block to markdown table."""
- table = block.table
- children = block.children
- if not table or not children:
- return None
-
- prop = table.property
- if not prop:
- return None
- row_size = prop.row_size or 0
- col_size = prop.column_size or 0
- if not row_size or not col_size:
- return None
-
- rows = []
- for row_idx in range(row_size):
- row = []
- for col_idx in range(col_size):
- cell_idx = row_idx * col_size + col_idx
- if cell_idx < len(children):
- cell_block_id = children[cell_idx]
- cell_block = block_map.get(cell_block_id)
- cell_text = self._extract_cell_text(cell_block, block_map)
- row.append(cell_text)
- else:
- row.append("")
- rows.append(row)
-
- return format_table_to_markdown(rows, has_header=True) if rows else None
-
- def _extract_cell_text(self, cell_block, block_map: Dict) -> str:
- """Extract text from a table cell block by reading its children."""
- if not cell_block or not cell_block.children:
- return ""
- texts = []
- for child_id in cell_block.children:
- child = block_map.get(child_id)
- if not child:
- continue
- # Use attribute-driven detection to find text in any block type
- attr = self._detect_block_attr(child)
- if attr:
- text = self._extract_block_text(child, attr)
- if text:
- texts.append(text)
- return " ".join(texts)
-
- # ========== Embedded Sheet in Docx ==========
-
- def _embedded_sheet_to_markdown(
- self, block, block_map: Dict = None, *, document_id: str = "", **_
- ) -> Optional[str]:
- """Convert an embedded sheet block to markdown table.
-
- These blocks appear in docx documents when a user embeds a spreadsheet
- view. The block contains a sheet token in the format
- ``{spreadsheet_token}_{sheet_id}``.
- """
- import lark_oapi as lark
-
- client = self._get_client()
- block_id = block.block_id
- doc_id = document_id or block.parent_id
-
- raw_req = (
- lark.BaseRequest.builder()
- .http_method(lark.HttpMethod.GET)
- .uri(f"/open-apis/docx/v1/documents/{doc_id}/blocks/{block_id}")
- .token_types({lark.AccessTokenType.TENANT})
- .build()
- )
- raw_resp = client.request(raw_req)
- if not raw_resp.success():
- return None
-
- data = json.loads(raw_resp.raw.content)
- sheet_token = data.get("data", {}).get("block", {}).get("sheet", {}).get("token", "")
- if not sheet_token:
- return None
-
- # Parse token: {spreadsheet_token}_{sheet_id}
- parts = sheet_token.rsplit("_", 1)
- if len(parts) != 2:
- return None
- spreadsheet_token, sheet_id = parts
-
- # Read cell data and trim empty trailing columns
- try:
- rows = self._read_sheet_range(spreadsheet_token, sheet_id, max_rows=100, max_cols=26)
- if rows:
- rows = self._trim_empty_columns(rows)
- if rows:
- return format_table_to_markdown(rows, has_header=True)
- except Exception as e:
- logger.warning(f"[FeishuParser] Failed to read embedded sheet {sheet_token}: {e}")
-
- return None
-
- @staticmethod
- def _trim_empty_columns(rows: List[List[str]]) -> List[List[str]]:
- """Remove trailing columns that are entirely empty across all rows."""
- if not rows:
- return rows
- max_cols = max(len(r) for r in rows)
- # Find rightmost non-empty column
- last_col = 0
- for col in range(max_cols):
- for row in rows:
- if col < len(row) and row[col].strip():
- last_col = col + 1
- if last_col == 0:
- return []
- return [row[:last_col] for row in rows]
-
- # ========== Sheets Parsing ==========
-
- def _parse_sheets(self, token: str) -> Tuple[str, str]:
- """Fetch spreadsheet data and convert to Markdown."""
- from lark_oapi.api.sheets.v3 import (
- GetSpreadsheetRequest,
- QuerySpreadsheetSheetRequest,
- )
-
- client = self._get_client()
- config = self._get_config()
-
- # Get spreadsheet metadata
- meta_request = GetSpreadsheetRequest.builder().spreadsheet_token(token).build()
- meta_response = client.sheets.v3.spreadsheet.get(meta_request)
- title = "Spreadsheet"
- if meta_response.success() and meta_response.data.spreadsheet:
- title = meta_response.data.spreadsheet.title or title
-
- # Get sheet list
- sheets_request = QuerySpreadsheetSheetRequest.builder().spreadsheet_token(token).build()
- sheets_response = client.sheets.v3.spreadsheet_sheet.query(sheets_request)
- if not sheets_response.success():
- raise RuntimeError(
- f"Failed to fetch sheets for {token}: "
- f"code={sheets_response.code}, msg={sheets_response.msg}"
- )
-
- sheets = sheets_response.data.sheets or []
- markdown_parts = [f"# {title}", f"**Sheets:** {len(sheets)}"]
-
- for sheet in sheets:
- sheet_id = sheet.sheet_id
- sheet_title = sheet.title or sheet_id
- row_count = sheet.grid_properties.row_count if sheet.grid_properties else 0
- col_count = sheet.grid_properties.column_count if sheet.grid_properties else 0
-
- parts = [f"## Sheet: {sheet_title}"]
-
- if row_count == 0 or col_count == 0:
- parts.append("*Empty sheet*")
- markdown_parts.append("\n\n".join(parts))
- continue
-
- parts.append(f"**Dimensions:** {row_count} rows x {col_count} columns")
-
- rows_to_read = min(row_count, config.max_rows_per_sheet)
- cell_data = self._read_sheet_range(token, sheet_id, rows_to_read, col_count)
-
- if cell_data:
- table_md = format_table_to_markdown(cell_data, has_header=True)
- parts.append(table_md)
-
- if row_count > config.max_rows_per_sheet:
- parts.append(
- f"\n*... {row_count - config.max_rows_per_sheet} more rows truncated ...*"
- )
-
- markdown_parts.append("\n\n".join(parts))
-
- return "\n\n".join(markdown_parts), title
-
- def _read_sheet_range(
- self, token: str, sheet_id: str, max_rows: int, max_cols: int
- ) -> List[List[str]]:
- """Read cell values from a sheet range using lark-oapi SDK."""
- import lark_oapi as lark
-
- client = self._get_client()
- end_col = self._col_number_to_letter(min(max_cols, 26))
- range_str = f"{sheet_id}!A1:{end_col}{max_rows}"
-
- request = (
- lark.BaseRequest.builder()
- .http_method(lark.HttpMethod.GET)
- .uri(f"/open-apis/sheets/v2/spreadsheets/{token}/values/{range_str}")
- .token_types({lark.AccessTokenType.TENANT})
- .build()
- )
-
- response = client.request(request)
- if not response.success():
- raise RuntimeError(
- f"Failed to read sheet range: code={response.code}, msg={response.msg}"
- )
-
- data = json.loads(response.raw.content)
- values = data.get("data", {}).get("valueRange", {}).get("values", [])
- return [[str(cell) if cell is not None else "" for cell in row] for row in values]
-
- @staticmethod
- def _col_number_to_letter(n: int) -> str:
- """Convert column number (1-based) to letter (A, B, ..., Z)."""
- return chr(ord("A") + n - 1) if 1 <= n <= 26 else "Z"
-
- # ========== Bitable Parsing ==========
-
- def _parse_bitable(self, app_token: str) -> Tuple[str, str]:
- """Fetch bitable data and convert to Markdown."""
- from lark_oapi.api.bitable.v1 import (
- ListAppTableFieldRequest,
- ListAppTableRecordRequest,
- ListAppTableRequest,
- )
-
- client = self._get_client()
- config = self._get_config()
-
- tables_request = ListAppTableRequest.builder().app_token(app_token).build()
- tables_response = client.bitable.v1.app_table.list(tables_request)
- if not tables_response.success():
- raise RuntimeError(
- f"Failed to list bitable tables: "
- f"code={tables_response.code}, msg={tables_response.msg}"
- )
-
- tables = tables_response.data.items or []
- title = f"Bitable ({len(tables)} tables)"
- markdown_parts = [f"# {title}"]
-
- for table in tables:
- table_id = table.table_id
- table_name = table.name or table_id
-
- fields_request = (
- ListAppTableFieldRequest.builder().app_token(app_token).table_id(table_id).build()
- )
- fields_response = client.bitable.v1.app_table_field.list(fields_request)
- field_names: List[str] = []
- if fields_response.success() and fields_response.data.items:
- field_names = [f.field_name for f in fields_response.data.items]
-
- all_records: list = []
- page_token = None
- while len(all_records) < config.max_records_per_table:
- remaining = config.max_records_per_table - len(all_records)
- page_size = min(remaining, 500)
- builder = (
- ListAppTableRecordRequest.builder()
- .app_token(app_token)
- .table_id(table_id)
- .page_size(page_size)
- )
- if page_token:
- builder = builder.page_token(page_token)
- records_response = client.bitable.v1.app_table_record.list(builder.build())
- if not records_response.success():
- break
- items = records_response.data.items or []
- all_records.extend(items)
- if not records_response.data.has_more:
- break
- page_token = records_response.data.page_token
-
- parts = [f"## {table_name}"]
- parts.append(f"**Records:** {len(all_records)}")
-
- if field_names and all_records:
- rows = [field_names]
- for record in all_records:
- fields = record.fields or {}
- row = [self._format_bitable_field(fields.get(fn, "")) for fn in field_names]
- rows.append(row)
- parts.append(format_table_to_markdown(rows, has_header=True))
-
- if len(all_records) >= config.max_records_per_table:
- parts.append(f"\n*... records truncated at {config.max_records_per_table} ...*")
-
- markdown_parts.append("\n\n".join(parts))
-
- return "\n\n".join(markdown_parts), title
-
- @staticmethod
- def _format_bitable_field(value: Any) -> str:
- """Format bitable field value to string."""
- if value is None:
- return ""
- if isinstance(value, list):
- texts = []
- for item in value:
- if isinstance(item, dict):
- texts.append(item.get("text", item.get("name", str(item))))
- else:
- texts.append(str(item))
- return ", ".join(texts)
- if isinstance(value, dict):
- return value.get("text", value.get("name", str(value)))
- return str(value)
diff --git a/openviking/parse/registry.py b/openviking/parse/registry.py
index ad57f891c..398c19e43 100644
--- a/openviking/parse/registry.py
+++ b/openviking/parse/registry.py
@@ -7,7 +7,7 @@ Provides automatic parser selection based on file type.
"""
from pathlib import Path
-from typing import TYPE_CHECKING, Callable, Dict, List, Optional, Union
+from typing import Dict, List, Optional, Union
from openviking.parse.base import ParseResult
from openviking.parse.parsers.base_parser import BaseParser
@@ -27,14 +27,8 @@ from openviking.parse.parsers.powerpoint import PowerPointParser
from openviking.parse.parsers.text import TextParser
from openviking.parse.parsers.word import WordParser
from openviking.parse.parsers.zip_parser import ZipParser
-from openviking_cli.utils import get_logger
from openviking_cli.utils.config.parser_config import ParserConfig
-if TYPE_CHECKING:
- from openviking.parse.custom import CustomParserProtocol
-
-logger = get_logger(__name__)
-
class ParserRegistry:
"""
@@ -45,15 +39,12 @@ class ParserRegistry:
def __init__(
self,
- register_optional: bool = True,
parser_configs: Optional[Dict[str, ParserConfig]] = None,
):
"""
Initialize registry with default parsers.
Args:
- register_optional: Whether to register optional parsers
- that require extra dependencies
parser_configs: Dictionary of parser configurations (from load_parser_configs_from_dict)
"""
self._parsers: Dict[str, BaseParser] = {}
@@ -61,129 +52,33 @@ class ParserRegistry:
self._parser_configs = parser_configs or {}
# Register core parsers
- self.register("text", TextParser(config=self._parser_configs.get("text")))
- self.register("markdown", MarkdownParser(config=self._parser_configs.get("markdown")))
- self.register("pdf", PDFParser(config=self._parser_configs.get("pdf")))
- self.register("html", HTMLParser(config=self._parser_configs.get("html")))
+ self._register("text", TextParser(config=self._parser_configs.get("text")))
+ self._register("markdown", MarkdownParser(config=self._parser_configs.get("markdown")))
+ self._register("pdf", PDFParser(config=self._parser_configs.get("pdf")))
+ self._register("html", HTMLParser(config=self._parser_configs.get("html")))
# Register markitdown-inspired parsers (built-in)
- self.register("word", WordParser(config=self._parser_configs.get("word")))
- self.register("legacy_doc", LegacyDocParser(config=self._parser_configs.get("legacy_doc")))
- self.register("powerpoint", PowerPointParser(config=self._parser_configs.get("powerpoint")))
- self.register("excel", ExcelParser(config=self._parser_configs.get("excel")))
- self.register("epub", EPubParser(config=self._parser_configs.get("epub")))
- self.register("zip", ZipParser())
- self.register("directory", DirectoryParser())
-
- self.register("image", ImageParser(config=self._parser_configs.get("image")))
- self.register("audio", AudioParser())
- self.register("video", VideoParser())
-
- def register(self, name: str, parser: BaseParser) -> None:
- """
- Register a parser.
-
- Args:
- name: Parser name
- parser: Parser instance
- """
+ self._register("word", WordParser(config=self._parser_configs.get("word")))
+ self._register("legacy_doc", LegacyDocParser(config=self._parser_configs.get("legacy_doc")))
+ self._register(
+ "powerpoint", PowerPointParser(config=self._parser_configs.get("powerpoint"))
+ )
+ self._register("excel", ExcelParser(config=self._parser_configs.get("excel")))
+ self._register("epub", EPubParser(config=self._parser_configs.get("epub")))
+ self._register("zip", ZipParser())
+ self._register("directory", DirectoryParser())
+
+ self._register("image", ImageParser(config=self._parser_configs.get("image")))
+ self._register("audio", AudioParser())
+ self._register("video", VideoParser())
+
+ def _register(self, name: str, parser: BaseParser) -> None:
self._parsers[name] = parser
# Map extensions to parser name
for ext in parser.supported_extensions:
self._extension_map[ext.lower()] = name
- def register_custom(
- self,
- handler: "CustomParserProtocol",
- extensions: Optional[List[str]] = None,
- name: Optional[str] = None,
- ) -> None:
- """
- Register a custom parser (Protocol-based).
-
- Args:
- handler: Object implementing CustomParserProtocol
- extensions: Optional list of extensions (overrides handler's extensions)
- name: Optional parser name (default: "custom_N")
-
- Example:
- ```python
- class MyParser:
- @property
- def supported_extensions(self) -> List[str]:
- return [".xyz"]
-
- def can_handle(self, source) -> bool:
- return str(source).endswith(".xyz")
-
- async def parse(self, source, **kwargs) -> ParseResult:
- ...
-
- registry.register_custom(MyParser(), name="xyz_parser")
- ```
- """
- from openviking.parse.custom import CustomParserWrapper
-
- # Generate name if not provided
- if name is None:
- # Count existing custom parsers
- custom_count = sum(1 for n in self._parsers if n.startswith("custom_"))
- name = f"custom_{custom_count}"
-
- # Wrap and register
- wrapper = CustomParserWrapper(handler, extensions=extensions)
- self.register(name, wrapper) # type: ignore
- logger.info(f"Registered custom parser '{name}' for {wrapper.supported_extensions}")
-
- def register_callback(
- self,
- extension: str,
- parse_fn: "Callable[[Union[str, Path]], ParseResult]",
- name: Optional[str] = None,
- ) -> None:
- """
- Register a callback function as a parser.
-
- Args:
- extension: File extension (e.g., ".xyz")
- parse_fn: Async function that parses and returns ParseResult
- name: Optional parser name (default: "callback_<ext>")
-
- Example:
- ```python
- async def my_parser(source: Union[str, Path], **kwargs) -> ParseResult:
- content = Path(source).read_text()
- return create_parse_result(
- root=ResourceNode(type=NodeType.ROOT, content=content),
- source_path=str(source),
- source_format="custom",
- parser_name="my_parser",
- )
-
- registry.register_callback(".xyz", my_parser)
- ```
- """
- from openviking.parse.custom import CallbackParserWrapper
-
- # Generate name if not provided
- if name is None:
- name = f"callback{extension}"
-
- # Wrap and register
- wrapper = CallbackParserWrapper(extension, parse_fn, name=name)
- self.register(name, wrapper) # type: ignore
- logger.info(f"Registered callback parser '{name}' for {extension}")
-
- def unregister(self, name: str) -> None:
- """Remove a parser from registry."""
- if name in self._parsers:
- parser = self._parsers[name]
- for ext in parser.supported_extensions:
- if self._extension_map.get(ext.lower()) == name:
- del self._extension_map[ext.lower()]
- del self._parsers[name]
-
def get_parser_for_file(self, path: Union[str, Path]) -> Optional[BaseParser]:
"""
Get appropriate parser for a file.
diff --git a/openviking/parse/understanding_api.py b/openviking/parse/understanding_api.py
index 7831121d1..57a01be5b 100644
--- a/openviking/parse/understanding_api.py
+++ b/openviking/parse/understanding_api.py
@@ -87,30 +87,41 @@ class UnderstandingAPI(BaseParser):
"""
source_str = str(source)
original_source = kwargs.get("original_source")
- candidate = original_source if isinstance(original_source, str) else source_str
- explicit_name = kwargs.get("resource_name") or kwargs.get("source_name")
+ resolved_extension = str(kwargs.get("resolved_extension") or "").lower().lstrip(".")
+ display_name = kwargs.get("resource_name") or kwargs.get("source_name")
prepared_response_id = kwargs.get(PREPARED_RESPONSE_ID_ARG)
if prepared_response_id is not None:
if not isinstance(prepared_response_id, str) or not prepared_response_id.strip():
raise ValueError(f"{PREPARED_RESPONSE_ID_ARG} must be a non-empty string")
prepared_response_id = prepared_response_id.strip()
- lark_file = self._normalize_lark_file(kwargs)
- # Legacy accessor output may keep a Feishu URL in original_source while
- # the actual parser input is a local Markdown file. Only direct URL
- # routing may use the parser API Lark protocol.
+
+ # Only a directly routed Feishu URL uses the Lark protocol. Accessor
+ # output is an existing local Markdown file and must stay local.
is_feishu_url = self._is_feishu_url(source_str)
- if is_feishu_url and not lark_file and not prepared_response_id:
- raise ValueError(
- "exactly one Feishu user or tenant access token is required for parser API imports"
- )
+ lark_file = (
+ await self._resolve_lark_file(kwargs)
+ if is_feishu_url and not prepared_response_id
+ else None
+ )
url: Optional[str] = None
local_path: Optional[Path] = None
- if isinstance(candidate, str) and candidate.startswith(("http://", "https://")):
- url = candidate
+ source_path = Path(source_str)
+ if source_path.is_file():
+ local_path = source_path
+ elif source_str.startswith(("http://", "https://")):
+ url = source_str
+ elif isinstance(original_source, str) and original_source.startswith(
+ ("http://", "https://")
+ ):
+ url = original_source
+ else:
+ local_path = source_path
+
+ if url is not None:
parsed = urlparse(url)
- doc_name = Path(parsed.path).stem or "resource"
- doc_type = Path(parsed.path).suffix.lower().lstrip(".") or "unknown"
+ inferred_name = Path(parsed.path).name
+ doc_type = resolved_extension or Path(parsed.path).suffix.lower().lstrip(".")
if is_feishu_url:
path_parts = [part for part in parsed.path.split("/") if part]
doc_type = {
@@ -118,17 +129,17 @@ class UnderstandingAPI(BaseParser):
"base": "bitable",
}.get(path_parts[0], path_parts[0])
else:
- local_path = Path(candidate)
- if not local_path.is_file():
+ inferred_name = local_path.name if local_path is not None else ""
+ if local_path is None or not local_path.is_file():
raise ValueError(
"UnderstandingAPI supports http(s) URLs or local files. "
"Got an invalid local file path."
)
- doc_name = local_path.stem or "resource"
- doc_type = local_path.suffix.lower().lstrip(".") or "unknown"
+ doc_type = resolved_extension or local_path.suffix.lower().lstrip(".")
- if explicit_name:
- doc_name = Path(str(explicit_name)).name or doc_name
+ effective_name = str(display_name or inferred_name or "resource")
+ doc_name = Path(effective_name).stem or "resource"
+ doc_type = doc_type or "unknown"
task_meta: Dict[str, Any] = {}
@@ -170,7 +181,7 @@ class UnderstandingAPI(BaseParser):
zip_path = await self._download_zip(zip_url)
try:
- if is_feishu_url and not explicit_name:
+ if is_feishu_url and not display_name:
archive_root = self._single_zip_root_name(zip_path)
if archive_root:
doc_name = archive_root
@@ -211,7 +222,7 @@ class UnderstandingAPI(BaseParser):
result = ParseResult(
root=root_node,
- source_path=url or source_str,
+ source_path=original_source if isinstance(original_source, str) else url or source_str,
source_format=doc_type,
temp_dir_path=temp_dir_path,
parser_name="UnderstandingAPI",
@@ -221,17 +232,32 @@ class UnderstandingAPI(BaseParser):
logger.info("[UnderstandingAPI] done")
return result
+ async def submit_file(self, source: Union[str, Path]) -> str:
+ """Upload a local file and submit it without retaining the local artifact."""
+ local_path = Path(source)
+ if not local_path.is_file():
+ raise ValueError("UnderstandingAPI file submission requires an existing local file")
+
+ file_obj = await self._create_file(local_path=local_path)
+ file_id = file_obj.get("id")
+ if not file_id:
+ raise RuntimeError(f"files api missing file_id: {self._safe_error_summary(file_obj)}")
+
+ response_obj = await self._create_response_for_file(file_id=str(file_id))
+ response_id = response_obj.get("id")
+ if not response_id:
+ raise RuntimeError(
+ f"responses api missing id: {self._safe_error_summary(response_obj)}"
+ )
+ return str(response_id)
+
async def submit_url(self, source: str, **kwargs) -> str:
"""Submit a URL without persisting its credentials in a queue payload."""
if not isinstance(source, str) or not source.startswith(("http://", "https://")):
raise ValueError("UnderstandingAPI URL submission requires an http(s) URL")
is_feishu_url = self._is_feishu_url(source)
- lark_file = self._normalize_lark_file(kwargs)
- if is_feishu_url and not lark_file:
- raise ValueError(
- "exactly one Feishu user or tenant access token is required for parser API imports"
- )
+ lark_file = await self._resolve_lark_file(kwargs) if is_feishu_url else None
parsed = urlparse(source)
doc_type = Path(parsed.path).suffix.lower().lstrip(".") or "unknown"
@@ -254,6 +280,20 @@ class UnderstandingAPI(BaseParser):
)
return str(response_id)
+ def can_submit_url_directly(self, source: str, **kwargs) -> bool:
+ """Return whether this URL can bypass source materialization."""
+ if not source.startswith(("http://", "https://")) or not self._is_feishu_url(source):
+ return False
+ if self._normalize_lark_file(kwargs):
+ return True
+ try:
+ from openviking.resource.feishu_watch_auth import load_feishu_app_credentials
+
+ load_feishu_app_credentials()
+ return True
+ except (FileNotFoundError, ValueError):
+ return False
+
async def parse_content(
self, content: str, source_path: Optional[str] = None, instruction: str = "", **kwargs
) -> ParseResult:
@@ -396,6 +436,20 @@ class UnderstandingAPI(BaseParser):
)
return auth or None
+ async def _resolve_lark_file(self, kwargs: Dict[str, Any]) -> Dict[str, str]:
+ auth = self._normalize_lark_file(kwargs)
+ if auth:
+ return auth
+ try:
+ from openviking.resource.feishu_watch_auth import FeishuOAuthClient
+
+ token = await FeishuOAuthClient.from_config().get_tenant_access_token()
+ except (FileNotFoundError, ValueError) as exc:
+ raise ValueError(
+ "exactly one Feishu user or tenant access token is required for parser API imports"
+ ) from exc
+ return {"tenant_access_token": token}
+
@staticmethod
def _single_zip_root_name(zip_path: Path) -> Optional[str]:
roots = set()
@@ -591,40 +645,51 @@ class UnderstandingAPI(BaseParser):
async def _unpack_zip_to_temp_dir(self, zip_path: Path, resource_name: str) -> str:
viking_fs = get_viking_fs()
temp_uri = viking_fs.create_temp_uri()
- await viking_fs.mkdir(temp_uri)
-
- temp_doc_uri = f"{temp_uri}/{resource_name}"
- await viking_fs.mkdir(temp_doc_uri)
-
- with tempfile.TemporaryDirectory() as extract_dir:
- with zipfile.ZipFile(zip_path, "r") as zf:
- safe_extract_zip(zf, extract_dir)
- extract_path = Path(extract_dir)
- items = [p for p in extract_path.iterdir() if p.name not in {".", ".."}]
- if len(items) == 1 and items[0].is_dir():
- root_dir = items[0]
- else:
- root_dir = extract_path
-
- image_mappings = await asyncio.to_thread(build_artifact_image_mappings, root_dir)
-
- for child in root_dir.iterdir():
- if child.name in {".", "..", IMAGE_MAPPINGS_FILENAME}:
- continue
- if child.is_dir():
- sub_uri = f"{temp_doc_uri}/{child.name}"
- await viking_fs.mkdir(sub_uri)
- await self._copy_dir_to_fs(child, sub_uri)
+ try:
+ await viking_fs.mkdir(temp_uri)
+
+ temp_doc_uri = f"{temp_uri}/{resource_name}"
+ await viking_fs.mkdir(temp_doc_uri)
+
+ with tempfile.TemporaryDirectory() as extract_dir:
+ with zipfile.ZipFile(zip_path, "r") as zf:
+ safe_extract_zip(zf, extract_dir)
+ extract_path = Path(extract_dir)
+ items = [p for p in extract_path.iterdir() if p.name not in {".", ".."}]
+ if len(items) == 1 and items[0].is_dir():
+ root_dir = items[0]
else:
- await viking_fs.write_file_bytes(
- f"{temp_doc_uri}/{child.name}", child.read_bytes()
+ root_dir = extract_path
+
+ image_mappings = await asyncio.to_thread(build_artifact_image_mappings, root_dir)
+
+ for child in root_dir.iterdir():
+ if child.name in {".", "..", IMAGE_MAPPINGS_FILENAME}:
+ continue
+ if child.is_dir():
+ sub_uri = f"{temp_doc_uri}/{child.name}"
+ await viking_fs.mkdir(sub_uri)
+ await self._copy_dir_to_fs(child, sub_uri)
+ else:
+ await viking_fs.write_file_bytes(
+ f"{temp_doc_uri}/{child.name}", child.read_bytes()
+ )
+
+ if image_mappings:
+ await viking_fs.write_file(
+ f"{temp_doc_uri}/{IMAGE_MAPPINGS_FILENAME}",
+ json.dumps(image_mappings, ensure_ascii=False),
)
-
- if image_mappings:
- await viking_fs.write_file(
- f"{temp_doc_uri}/{IMAGE_MAPPINGS_FILENAME}",
- json.dumps(image_mappings, ensure_ascii=False),
+ except BaseException:
+ try:
+ await viking_fs.delete_temp(temp_uri)
+ except Exception as cleanup_exc:
+ logger.warning(
+ "[UnderstandingAPI] Failed to clean temporary artifact %s: %s",
+ temp_uri,
+ cleanup_exc,
)
+ raise
return temp_uri
diff --git a/openviking/resource/watch_scheduler.py b/openviking/resource/watch_scheduler.py
index 6b37d8cfd..d3fa5e063 100644
--- a/openviking/resource/watch_scheduler.py
+++ b/openviking/resource/watch_scheduler.py
@@ -213,7 +213,7 @@ class WatchScheduler:
async def _execute_task(self, task) -> None:
"""Execute a single watch task.
- Calls ResourceService.add_resource to process the resource.
+ Calls ResourceService.refresh_resource to re-process the resource.
Handles errors gracefully and updates execution time regardless of success/failure.
Deactivates tasks when resources no longer exist.
@@ -282,7 +282,7 @@ class WatchScheduler:
raise
if not should_deactivate:
- result = await self._resource_service.add_resource(
+ result = await self._resource_service.refresh_resource(
path=task.path,
ctx=ctx,
to=task.to_uri,
@@ -292,7 +292,6 @@ class WatchScheduler:
build_index=getattr(task, "build_index", True),
summarize=getattr(task, "summarize", False),
watch_interval=task.watch_interval,
- skip_watch_management=True,
**processor_kwargs,
)
diff --git a/openviking/service/resource_service.py b/openviking/service/resource_service.py
index 773bd6f43..343181746 100644
--- a/openviking/service/resource_service.py
+++ b/openviking/service/resource_service.py
@@ -64,6 +64,7 @@ from openviking_cli.exceptions import (
from openviking_cli.utils import get_logger
if TYPE_CHECKING:
+ from openviking.parse.accessors.base import LocalResource
from openviking.resource.watch_manager import WatchManager
from openviking.resource.watch_scheduler import WatchScheduler
from openviking.service.resource_memory_link_service import ResourceMemoryLinkService
@@ -101,10 +102,27 @@ _ADD_RESOURCE_ARGS_RESERVED_FIELDS = frozenset(
"telemetry",
"request_validator",
"understanding_response_id",
+ "parser_backend",
+ "resolved_extension",
"defer_post_processing",
}
)
+_INTERNAL_INGESTION_FIELDS = frozenset(
+ {
+ "manage_watch",
+ "parser_args",
+ "resource_lock",
+ "route_source",
+ "skip_watch_management",
+ "stage_callback",
+ "watch_auth_state",
+ "understanding_response_id",
+ "parser_backend",
+ "resolved_extension",
+ }
+)
+
@dataclass
class _ResourceSourceInfo:
@@ -161,14 +179,6 @@ class ResourceService:
return None
return self._watch_scheduler.watch_manager
- def _get_parser_router(self):
- if not hasattr(self, "_parser_router"):
- from openviking.parse.parser_router import ParserRouter
- from openviking.parse.registry import get_registry
-
- self._parser_router = ParserRouter(get_registry())
- return self._parser_router
-
def _sanitize_watch_processor_kwargs(self, processor_kwargs: Dict[str, Any]) -> Dict[str, Any]:
sanitized: Dict[str, Any] = {}
for key, value in processor_kwargs.items():
@@ -183,7 +193,7 @@ class ResourceService:
self,
*,
watch_manager: Optional["WatchManager"],
- skip_watch_management: bool,
+ manage_watch: bool,
watch_interval: float,
target: ContentTargetSpec,
root_uri: str,
@@ -196,7 +206,7 @@ class ResourceService:
watch_auth_state: Optional[Dict[str, Any]],
ctx: RequestContext,
) -> None:
- if not watch_manager or skip_watch_management:
+ if not watch_manager or not manage_watch:
return
telemetry = get_current_telemetry()
with telemetry.measure("resource.watch"):
@@ -384,6 +394,15 @@ class ResourceService:
target_uri = msg.root_uri
parent_uri = None
internal_kwargs: Dict[str, Any] = {}
+ queued_args = dict(msg.args)
+ for key in ("parser_backend", "resolved_extension"):
+ if key in queued_args:
+ internal_kwargs[key] = queued_args.pop(key)
+ normalized_args = self._normalize_add_resource_args(
+ queued_args,
+ watch_interval=msg.watch_interval,
+ )
+ internal_kwargs.update(normalized_args.processor_kwargs)
if msg.defer_target_resolution:
from openviking_cli.utils.uri import VikingURI
@@ -393,7 +412,7 @@ class ResourceService:
from openviking.parse.understanding_api import PREPARED_RESPONSE_ID_ARG
internal_kwargs[PREPARED_RESPONSE_ID_ARG] = msg.understanding_response_id
- return await self.add_resource(
+ return await self._execute_resource_ingestion(
path=msg.path,
ctx=ctx,
to=target_uri,
@@ -405,11 +424,12 @@ class ResourceService:
build_index=msg.build_index,
summarize=msg.summarize,
watch_interval=msg.watch_interval,
- skip_watch_management=msg.skip_watch_management,
+ manage_watch=not msg.skip_watch_management,
allow_local_path_resolution=msg.allow_local_path_resolution,
enforce_public_remote_targets=msg.enforce_public_remote_targets,
resource_lock=resource_lock,
stage_callback=stage_callback,
+ watch_auth_state=normalized_args.watch_auth_state,
strict=msg.strict,
source_name=msg.source_name,
ignore_dirs=msg.ignore_dirs,
@@ -418,7 +438,6 @@ class ResourceService:
directly_upload_media=msg.directly_upload_media,
preserve_structure=msg.preserve_structure,
create_parent=msg.create_parent,
- args=msg.args,
**internal_kwargs,
)
@@ -486,7 +505,7 @@ class ResourceService:
build_index: bool = True,
summarize: bool = False,
watch_interval: float = 0,
- skip_watch_management: bool = False,
+ manage_watch: bool = True,
allow_local_path_resolution: bool = True,
enforce_public_remote_targets: bool = False,
args: Optional[Dict[str, Any]] = None,
@@ -560,7 +579,7 @@ class ResourceService:
build_index=build_index,
summarize=summarize,
watch_interval=watch_interval,
- skip_watch_management=skip_watch_management,
+ skip_watch_management=not manage_watch,
allow_local_path_resolution=allow_local_path_resolution,
enforce_public_remote_targets=enforce_public_remote_targets,
strict=bool(kwargs.get("strict", False)),
@@ -629,18 +648,6 @@ class ResourceService:
)
return root_uri, resource_lock
- def _should_use_understanding_api(self, path: str) -> bool:
- return self._get_parser_router().should_use_understanding_api(path)
-
- @staticmethod
- def _is_feishu_url(path: str) -> bool:
- try:
- from openviking.parse.accessors.feishu_accessor import FeishuAccessor
-
- return FeishuAccessor._is_feishu_url(path)
- except Exception:
- return False
-
@staticmethod
def _target_doc_name(
path: str,
@@ -703,15 +710,93 @@ class ResourceService:
build_index: bool = True,
summarize: bool = False,
watch_interval: float = 0,
- skip_watch_management: bool = False,
allow_local_path_resolution: bool = True,
enforce_public_remote_targets: bool = False,
- resource_lock: Optional[LockLease] = None,
- stage_callback: Optional[Callable[[str], Any]] = None,
args: Optional[Dict[str, Any]] = None,
**kwargs,
) -> Dict[str, Any]:
- """Add resource to OpenViking (only supports resources scope).
+ """Accept and route a new resource-add request."""
+ internal_fields = sorted(set(kwargs).intersection(_INTERNAL_INGESTION_FIELDS))
+ if internal_fields:
+ raise InvalidArgumentError(
+ "add_resource does not accept internal execution fields: "
+ + ", ".join(internal_fields)
+ )
+ return await self._submit_resource_ingestion(
+ path=path,
+ ctx=ctx,
+ to=to,
+ parent=parent,
+ reason=reason,
+ instruction=instruction,
+ wait=wait,
+ timeout=timeout,
+ build_index=build_index,
+ summarize=summarize,
+ watch_interval=watch_interval,
+ manage_watch=True,
+ allow_local_path_resolution=allow_local_path_resolution,
+ enforce_public_remote_targets=enforce_public_remote_targets,
+ args=args,
+ **kwargs,
+ )
+
+ async def refresh_resource(
+ self,
+ path: str,
+ ctx: RequestContext,
+ to: Optional[str] = None,
+ parent: Optional[str] = None,
+ reason: str = "",
+ instruction: str = "",
+ timeout: Optional[float] = None,
+ build_index: bool = True,
+ summarize: bool = False,
+ watch_interval: float = 0,
+ allow_local_path_resolution: bool = True,
+ enforce_public_remote_targets: bool = False,
+ **kwargs,
+ ) -> Dict[str, Any]:
+ """Submit a scheduled refresh without changing its watch task."""
+ return await self._submit_resource_ingestion(
+ path=path,
+ ctx=ctx,
+ to=to,
+ parent=parent,
+ reason=reason,
+ instruction=instruction,
+ wait=False,
+ timeout=timeout,
+ build_index=build_index,
+ summarize=summarize,
+ watch_interval=watch_interval,
+ manage_watch=False,
+ allow_local_path_resolution=allow_local_path_resolution,
+ enforce_public_remote_targets=enforce_public_remote_targets,
+ args=None,
+ **kwargs,
+ )
+
+ async def _submit_resource_ingestion(
+ self,
+ path: str,
+ ctx: RequestContext,
+ to: Optional[str] = None,
+ parent: Optional[str] = None,
+ reason: str = "",
+ instruction: str = "",
+ wait: bool = False,
+ timeout: Optional[float] = None,
+ build_index: bool = True,
+ summarize: bool = False,
+ watch_interval: float = 0,
+ manage_watch: bool = True,
+ allow_local_path_resolution: bool = True,
+ enforce_public_remote_targets: bool = False,
+ args: Optional[Dict[str, Any]] = None,
+ **kwargs,
+ ) -> Dict[str, Any]:
+ """Validate and route one resource ingestion request.
Args:
path: Resource path (local file or URL)
@@ -734,8 +819,6 @@ class ResourceService:
Note: If the target URI already has an active watch task, a ConflictError will be
raised. You must first cancel the existing watch (set watch_interval <= 0) before
creating a new one.
- skip_watch_management: If True, skip watch task management (used by scheduler to
- avoid recursive watch task creation during scheduled execution)
enforce_public_remote_targets: When True, reject non-public remote hosts and
validate each outbound HTTP request URL during fetch.
args: Parser/accessor-specific options forwarded to the processing chain.
@@ -809,12 +892,56 @@ class ResourceService:
build_index=build_index,
summarize=summarize,
watch_interval=watch_interval,
- skip_watch_management=skip_watch_management,
+ manage_watch=manage_watch,
allow_local_path_resolution=allow_local_path_resolution,
enforce_public_remote_targets=enforce_public_remote_targets,
**kwargs,
)
+ return await self._execute_resource_ingestion(
+ path=path,
+ ctx=ctx,
+ to=to,
+ parent=parent,
+ reason=reason,
+ instruction=instruction,
+ wait=wait,
+ timeout=timeout,
+ build_index=build_index,
+ summarize=summarize,
+ watch_interval=watch_interval,
+ manage_watch=manage_watch,
+ allow_local_path_resolution=allow_local_path_resolution,
+ enforce_public_remote_targets=enforce_public_remote_targets,
+ watch_auth_state=normalized_args.watch_auth_state,
+ parser_args=normalized_args.processor_kwargs,
+ **kwargs,
+ )
+
+ async def _execute_resource_ingestion(
+ self,
+ path: str,
+ ctx: RequestContext,
+ to: Optional[str] = None,
+ parent: Optional[str] = None,
+ reason: str = "",
+ instruction: str = "",
+ wait: bool = False,
+ timeout: Optional[float] = None,
+ build_index: bool = True,
+ summarize: bool = False,
+ watch_interval: float = 0,
+ manage_watch: bool = True,
+ allow_local_path_resolution: bool = True,
+ enforce_public_remote_targets: bool = False,
+ watch_auth_state: Optional[Dict[str, Any]] = None,
+ parser_args: Optional[Dict[str, Any]] = None,
+ resource_lock: Optional[LockLease] = None,
+ stage_callback: Optional[Callable[[str], Any]] = None,
+ **kwargs,
+ ) -> Dict[str, Any]:
+ """Execute an already-routed resource ingestion."""
+ self._ensure_initialized()
request_start = time.perf_counter()
telemetry = get_current_telemetry()
telemetry_id = register_wait_telemetry(wait)
@@ -824,13 +951,14 @@ class ResourceService:
if telemetry_id:
request_wait_tracker.register_request(telemetry_id)
watch_manager = self._get_watch_manager()
- watch_enabled = bool(watch_manager and not skip_watch_management and watch_interval > 0)
+ watch_enabled = bool(watch_manager and manage_watch and watch_interval > 0)
telemetry.set("resource.flags.wait", wait)
telemetry.set("resource.flags.build_index", build_index)
telemetry.set("resource.flags.summarize", summarize)
telemetry.set("resource.flags.watch_enabled", watch_enabled)
+ prepared_resource: Optional["LocalResource"] = None
try:
target = ContentTargetSpec.from_fields(
ctx=ctx,
@@ -845,20 +973,57 @@ class ResourceService:
if resource_lock is not None:
kwargs["resource_lock"] = resource_lock
- if (
+ async_understanding_candidate = (
not wait
and not is_git_repo_url(path)
- and self._should_use_understanding_api(path)
and not allow_local_path_resolution
and self._resource_processor is not None
+ )
+ direct_understanding = bool(
+ async_understanding_candidate
+ and self._resource_processor.should_use_understanding_directly(path, **kwargs)
+ )
+
+ if (
+ async_understanding_candidate
+ and not direct_understanding
+ and self._resource_processor.understanding_api_enabled()
+ ):
+ prepared_resource = await self._resource_processor.prepare_resource(
+ path,
+ ctx,
+ allow_local_path_resolution=allow_local_path_resolution,
+ **kwargs,
+ )
+
+ if (
+ prepared_resource is not None
+ and self._resource_processor is not None
+ and self._resource_processor.should_use_understanding_api(prepared_resource)
):
+ understanding_source = prepared_resource
+ elif direct_understanding:
+ understanding_source = path
+ else:
+ understanding_source = None
+
+ if understanding_source is not None:
from openviking.storage.queuefs.add_resource_msg import AddResourceMsg
- source_name = kwargs.get("source_name")
+ if prepared_resource is not None:
+ resolved_extension = str(prepared_resource.meta.get("resolved_extension") or "")
+ source_name = (
+ kwargs.get("source_name")
+ or prepared_resource.meta.get("original_filename")
+ or prepared_resource.meta.get("resolved_name")
+ )
+ else:
+ resolved_extension = ""
+ source_name = kwargs.get("source_name")
source_info = _ResourceSourceInfo(
source_name=source_name,
source_path=path,
- source_format="file",
+ source_format=resolved_extension.lstrip(".") or "file",
)
doc_name = self._target_doc_name(path, source_name, source_info)
source_path = source_info.source_path or source_name or path
@@ -876,10 +1041,7 @@ class ResourceService:
create_parent=target.create_parent,
)
defer_target_resolution = bool(
- candidate_uri
- and not source_name
- and not watch_enabled
- and self._is_feishu_url(path)
+ candidate_uri and not source_name and not watch_enabled and direct_understanding
)
if self._viking_fs is None:
raise NotInitializedError("VikingFS")
@@ -913,14 +1075,20 @@ class ResourceService:
enqueue_started = False
try:
- queued_args = dict(normalized_args.processor_kwargs)
- feishu_access_token = queued_args.pop(FEISHU_ACCESS_TOKEN_ARG, None)
- understanding_response_id = None
- if self._is_feishu_url(path) and feishu_access_token:
- understanding_response_id = await self._get_parser_router().submit_url(
- path,
- feishu_access_token=feishu_access_token,
- )
+ queued_args = dict(parser_args or {})
+ understanding_response_id = await self._resource_processor.submit_understanding(
+ understanding_source,
+ **queued_args,
+ )
+ queued_args.pop(FEISHU_ACCESS_TOKEN_ARG, None)
+ if prepared_resource is not None:
+ prepared_resource.cleanup()
+ prepared_resource = None
+
+ processor_args = self._sanitize_watch_processor_kwargs(queued_args)
+ processor_args["parser_backend"] = "understanding"
+ if resolved_extension:
+ processor_args["resolved_extension"] = resolved_extension
lock_handoff = lock_lease.to_handoff()
msg = AddResourceMsg(
@@ -946,7 +1114,7 @@ class ResourceService:
create_parent=bool(kwargs.get("create_parent", False)),
allow_local_path_resolution=allow_local_path_resolution,
enforce_public_remote_targets=enforce_public_remote_targets,
- args=self._sanitize_watch_processor_kwargs(queued_args),
+ args=processor_args,
source_name=source_name,
lock_handoff=lock_handoff.to_dict() if lock_handoff else None,
skip_watch_management=True,
@@ -972,7 +1140,7 @@ class ResourceService:
)
await self._manage_watch_if_needed(
watch_manager=watch_manager,
- skip_watch_management=skip_watch_management,
+ manage_watch=manage_watch,
watch_interval=watch_interval,
target=target,
root_uri=root_uri,
@@ -982,7 +1150,7 @@ class ResourceService:
build_index=build_index,
summarize=summarize,
processor_kwargs=kwargs,
- watch_auth_state=normalized_args.watch_auth_state,
+ watch_auth_state=watch_auth_state,
ctx=ctx,
)
response = {
@@ -1005,9 +1173,11 @@ class ResourceService:
summarize=summarize,
stage_callback=stage_callback,
allow_local_path_resolution=allow_local_path_resolution,
+ prepared_resource=prepared_resource,
defer_post_processing=not wait,
**kwargs,
)
+ prepared_resource = None
if result.get("status") == "error":
return result
@@ -1104,7 +1274,7 @@ class ResourceService:
job_enqueued = True
await self._manage_watch_if_needed(
watch_manager=watch_manager,
- skip_watch_management=skip_watch_management,
+ manage_watch=manage_watch,
watch_interval=watch_interval,
target=target,
root_uri=str(result.get("root_uri") or ""),
@@ -1114,7 +1284,7 @@ class ResourceService:
build_index=build_index,
summarize=summarize,
processor_kwargs=kwargs,
- watch_auth_state=normalized_args.watch_auth_state,
+ watch_auth_state=watch_auth_state,
ctx=ctx,
)
if wait:
@@ -1134,6 +1304,8 @@ class ResourceService:
)
raise
finally:
+ if prepared_resource is not None:
+ prepared_resource.cleanup()
telemetry.set(
"resource.request.duration_ms",
round((time.perf_counter() - request_start) * 1000, 3),
diff --git a/openviking/utils/media_processor.py b/openviking/utils/media_processor.py
index ab1785a76..57e3b33aa 100644
--- a/openviking/utils/media_processor.py
+++ b/openviking/utils/media_processor.py
@@ -5,8 +5,7 @@
from pathlib import Path
from typing import TYPE_CHECKING, Optional
-from openviking.parse import parse
-from openviking.parse.accessors.base import SourceType
+from openviking.parse.accessors.base import LocalResource, SourceType
from openviking.parse.accessors.mime_types import IANA_MEDIA_TYPE_TO_EXTENSION
from openviking.parse.base import ParseResult
from openviking.parse.parsers.constants import (
@@ -14,6 +13,7 @@ from openviking.parse.parsers.constants import (
DOCUMENTATION_EXTENSIONS,
IGNORE_EXTENSIONS,
)
+from openviking.parse.registry import parse
from openviking.server.local_input_guard import (
is_remote_resource_source,
looks_like_local_path,
@@ -77,7 +77,6 @@ class UnifiedResourceProcessor:
):
self.storage = storage
self._vlm_processor = vlm_processor
- self._document_converter = None
self._accessor_registry = None
def _get_vlm_processor(self) -> Optional["VLMProcessor"]:
@@ -104,11 +103,66 @@ class UnifiedResourceProcessor:
self._parser_router = ParserRouter(get_registry())
return self._parser_router
+ def understanding_api_enabled(self) -> bool:
+ return self._get_parser_router().understanding_api_enabled()
+
+ def should_use_understanding_api(self, source: str | Path | LocalResource) -> bool:
+ if isinstance(source, LocalResource):
+ if source.path.is_dir():
+ return False
+ return self._get_parser_router().should_use_understanding_api(
+ source,
+ resolved_extension=str(source.meta.get("resolved_extension") or ""),
+ )
+ return self._get_parser_router().should_use_understanding_api(source)
+
+ def should_use_understanding_directly(self, source: str, **kwargs) -> bool:
+ return self._get_parser_router().should_use_understanding_directly(source, **kwargs)
+
+ async def submit_understanding(self, source: str | Path | LocalResource, **kwargs) -> str:
+ return await self._get_parser_router().submit(source, **kwargs)
+
+ @staticmethod
+ def _set_resolved_identity(resource: LocalResource, source_name: Optional[str]) -> None:
+ meta = resource.meta
+ if resource.path.is_dir():
+ extension = ""
+ elif resource.source_type == SourceType.HTTP:
+ extension = str(meta.get("extension") or resource.path.suffix)
+ else:
+ extension = Path(source_name).suffix if source_name else ""
+ extension = extension or str(meta.get("extension") or resource.path.suffix)
+
+ meta["resolved_extension"] = extension.lower()
+ meta["resolved_name"] = source_name or meta.get("original_filename") or resource.path.name
+
+ async def prepare(
+ self,
+ source: str,
+ allow_local_path_resolution: bool = True,
+ **kwargs,
+ ) -> LocalResource:
+ """Fetch a path/URL and freeze the identity used for parser routing."""
+ if (
+ not allow_local_path_resolution
+ and not is_remote_resource_source(source)
+ and looks_like_local_path(source)
+ ):
+ raise PermissionDeniedError(
+ "HTTP server only accepts remote resource URLs or temp-uploaded files; "
+ "direct host filesystem paths are not allowed."
+ )
+
+ resource = await self._get_accessor_registry().access(source, **kwargs)
+ self._set_resolved_identity(resource, kwargs.get("source_name"))
+ return resource
+
async def process(
self,
source: str,
instruction: str = "",
allow_local_path_resolution: bool = True,
+ prepared_resource: Optional[LocalResource] = None,
**kwargs,
) -> ParseResult:
"""Process any source (file/URL/content) with two-layer architecture.
@@ -129,39 +183,47 @@ class UnifiedResourceProcessor:
# Treat as raw content
return await parse(source, instruction=instruction)
- # Block local paths in HTTP server mode, but allow remote URLs
- if (
- not allow_local_path_resolution
- and not is_remote_resource_source(source)
- and looks_like_local_path(source)
- ):
- raise PermissionDeniedError(
- "HTTP server only accepts remote resource URLs or temp-uploaded files; "
- "direct host filesystem paths are not allowed."
- )
-
- if self._should_bypass_feishu_accessor(source, kwargs):
+ if prepared_resource is None and self._has_prepared_understanding_response(kwargs):
parse_kwargs = dict(kwargs)
parse_kwargs["instruction"] = instruction
parse_kwargs["vlm_processor"] = self._get_vlm_processor()
parse_kwargs["storage"] = self.storage
parse_kwargs["_source_meta"] = {
- "source_type": SourceType.FEISHU,
+ "source_type": SourceType.HTTP,
"original_url": source,
}
parse_kwargs["original_source"] = source
+ parse_kwargs["parser_backend"] = "understanding"
parse_kwargs.pop("feishu_access_token", None)
- if not self._has_prepared_understanding_response(kwargs):
- parse_kwargs["lark_file"] = await self._resolve_feishu_parser_auth(kwargs)
explicit_name = kwargs.get("resource_name") or kwargs.get("source_name")
if explicit_name:
parse_kwargs["resource_name"] = _smart_stem(explicit_name)
return await self._get_parser_router().parse(source, **parse_kwargs)
- # Phase 1: Accessor - get local resource
- registry = self._get_accessor_registry()
- local_resource = await registry.access(source, **kwargs)
+ if prepared_resource is None and self.should_use_understanding_directly(source, **kwargs):
+ parse_kwargs = dict(kwargs)
+ parse_kwargs["instruction"] = instruction
+ parse_kwargs["vlm_processor"] = self._get_vlm_processor()
+ parse_kwargs["storage"] = self.storage
+ parse_kwargs["_source_meta"] = {
+ "source_type": SourceType.HTTP,
+ "original_url": source,
+ }
+ parse_kwargs["original_source"] = source
+
+ explicit_name = kwargs.get("resource_name") or kwargs.get("source_name")
+ if explicit_name:
+ parse_kwargs["resource_name"] = _smart_stem(explicit_name)
+ return await self._get_parser_router().parse(source, **parse_kwargs)
+
+ # Phase 1: Accessor - get local resource. A caller may prepare a remote
+ # resource first so async routing uses the same detected file type.
+ local_resource = prepared_resource or await self.prepare(
+ source,
+ allow_local_path_resolution=allow_local_path_resolution,
+ **kwargs,
+ )
# Use context manager for automatic cleanup, but preserve directories for TreeBuilder
try:
@@ -171,6 +233,9 @@ class UnifiedResourceProcessor:
parse_kwargs["vlm_processor"] = self._get_vlm_processor()
parse_kwargs["storage"] = self.storage
parse_kwargs["_source_meta"] = local_resource.meta
+ parse_kwargs["resolved_extension"] = kwargs.get(
+ "resolved_extension"
+ ) or local_resource.meta.get("resolved_extension", "")
# CRITICAL: Pass along the original_source!
# This is the full URL the user provided (e.g. "https://github.com/volcengine/OpenViking")
# CodeRepositoryParser and TreeBuilder need this to extract the org/repo format
@@ -222,48 +287,9 @@ class UnifiedResourceProcessor:
"""Check if source is a URL."""
return is_remote_resource_source(source)
- @staticmethod
- def _is_feishu_url(source: str) -> bool:
- """Backward-compatible Feishu URL detector used by legacy tests/callers."""
- try:
- from openviking.parse.accessors.feishu_accessor import FeishuAccessor
-
- return FeishuAccessor._is_feishu_url(source)
- except Exception:
- return False
-
- def _should_bypass_feishu_accessor(self, source: str, kwargs: dict) -> bool:
- token = kwargs.get("feishu_access_token")
- if not self._is_feishu_url(source):
- return False
- if not self._get_parser_router().should_use_understanding_api(source):
- return False
- if self._has_prepared_understanding_response(kwargs):
- return True
- if isinstance(token, str) and token.strip():
- return True
- try:
- from openviking.resource.feishu_watch_auth import load_feishu_app_credentials
-
- load_feishu_app_credentials()
- return True
- except ValueError:
- return False
-
@staticmethod
def _has_prepared_understanding_response(kwargs: dict) -> bool:
from openviking.parse.understanding_api import PREPARED_RESPONSE_ID_ARG
value = kwargs.get(PREPARED_RESPONSE_ID_ARG)
return isinstance(value, str) and bool(value.strip())
-
- @staticmethod
- async def _resolve_feishu_parser_auth(kwargs: dict) -> dict[str, str]:
- token = kwargs.get("feishu_access_token")
- if isinstance(token, str) and token.strip():
- return {"user_access_token": token.strip()}
-
- from openviking.resource.feishu_watch_auth import FeishuOAuthClient
-
- tenant_access_token = await FeishuOAuthClient.from_config().get_tenant_access_token()
- return {"tenant_access_token": tenant_access_token}
diff --git a/openviking/utils/resource_processor.py b/openviking/utils/resource_processor.py
index 74c64f04f..cf42b59a2 100644
--- a/openviking/utils/resource_processor.py
+++ b/openviking/utils/resource_processor.py
@@ -10,7 +10,7 @@ as described in the OpenViking design document.
import inspect
import time
from collections.abc import Callable
-from typing import TYPE_CHECKING, Any, Dict, List, Optional
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union
from openviking.parse.image_rewrite import rewrite_image_uris
from openviking.parse.tree_builder import TreeBuilder
@@ -33,6 +33,7 @@ from openviking_cli.utils import get_logger
from openviking_cli.utils.storage import StoragePath
if TYPE_CHECKING:
+ from openviking.parse.accessors.base import LocalResource
from openviking.parse.vlm import VLMProcessor
logger = get_logger(__name__)
@@ -92,6 +93,34 @@ class ResourceProcessor:
)
return self._media_processor
+ async def prepare_resource(
+ self,
+ path: str,
+ ctx: RequestContext,
+ *,
+ allow_local_path_resolution: bool = True,
+ **kwargs,
+ ) -> "LocalResource":
+ """Resolve a source once before selecting an async parser route."""
+ with get_viking_fs().bind_request_context(ctx):
+ return await self._get_media_processor().prepare(
+ path,
+ allow_local_path_resolution=allow_local_path_resolution,
+ **kwargs,
+ )
+
+ def understanding_api_enabled(self) -> bool:
+ return self._get_media_processor().understanding_api_enabled()
+
+ def should_use_understanding_api(self, source: Union[str, "LocalResource"]) -> bool:
+ return self._get_media_processor().should_use_understanding_api(source)
+
+ def should_use_understanding_directly(self, source: str, **kwargs) -> bool:
+ return self._get_media_processor().should_use_understanding_directly(source, **kwargs)
+
+ async def submit_understanding(self, source: Union[str, "LocalResource"], **kwargs) -> str:
+ return await self._get_media_processor().submit_understanding(source, **kwargs)
+
async def build_index(
self, resource_uris: List[str], ctx: RequestContext, **kwargs
) -> Dict[str, Any]:
@@ -118,6 +147,7 @@ class ResourceProcessor:
parent: Optional[str] = None,
summarize: bool = False,
stage_callback: Optional[Callable[[str], Any]] = None,
+ prepared_resource: Optional["LocalResource"] = None,
**kwargs,
) -> Dict[str, Any]:
"""
@@ -169,6 +199,7 @@ class ResourceProcessor:
parse_result = await media_processor.process(
source=path,
instruction=effective_instruction,
+ prepared_resource=prepared_resource,
**kwargs,
)
result["source_path"] = parse_result.source_path or path
diff --git a/tests/parse/test_directory_parser_routing.py b/tests/parse/test_directory_parser_routing.py
index 1b94774ba..ca75798cf 100644
--- a/tests/parse/test_directory_parser_routing.py
+++ b/tests/parse/test_directory_parser_routing.py
@@ -46,7 +46,7 @@ from openviking.parse.registry import ParserRegistry
@pytest.fixture
def registry() -> ParserRegistry:
"""Default registry (no optional parsers like ImageParser)."""
- return ParserRegistry(register_optional=False)
+ return ParserRegistry()
# -- directory tree that covers every parser type ----------------------------
diff --git a/tests/parse/test_directory_scan.py b/tests/parse/test_directory_scan.py
index 02fedef52..2e4e28954 100644
--- a/tests/parse/test_directory_scan.py
+++ b/tests/parse/test_directory_scan.py
@@ -85,7 +85,7 @@ def tmp_with_drafts(tmp_path: Path) -> Path:
@pytest.fixture
def registry() -> ParserRegistry:
"""Default parser registry (includes markdown, pdf, html, text, etc.)."""
- return ParserRegistry(register_optional=False)
+ return ParserRegistry()
# ---------------------------------------------------------------------------
diff --git a/tests/parse/test_feishu_accessor.py b/tests/parse/test_feishu_accessor.py
index 9059720d4..505f932cf 100644
--- a/tests/parse/test_feishu_accessor.py
+++ b/tests/parse/test_feishu_accessor.py
@@ -1,12 +1,14 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
-"""Tests for FeishuAccessor user token and image handling."""
+"""Tests for FeishuAccessor supported types, user token, and image handling."""
import asyncio
import sys
from types import ModuleType, SimpleNamespace
from unittest.mock import MagicMock
+import pytest
+
from openviking.parse.accessors.feishu_accessor import FeishuAccessor, _title_as_filename
@@ -110,6 +112,27 @@ class _FakeListDocumentBlockRequestBuilder:
return self._request
+class _FakeTypedRequest:
+ @staticmethod
+ def builder():
+ return _FakeTypedRequestBuilder()
+
+
+class _FakeTypedRequestBuilder:
+ def __init__(self):
+ self._request = SimpleNamespace()
+
+ def __getattr__(self, name):
+ def _set(value):
+ setattr(self._request, name, value)
+ return self
+
+ return _set
+
+ def build(self):
+ return self._request
+
+
def _install_fake_lark_modules(monkeypatch):
lark = ModuleType("lark_oapi")
lark.BaseRequest = _FakeBaseRequest
@@ -117,10 +140,15 @@ def _install_fake_lark_modules(monkeypatch):
lark.AccessTokenType = SimpleNamespace(TENANT="tenant", USER="user")
docx_v1 = ModuleType("lark_oapi.api.docx.v1")
docx_v1.ListDocumentBlockRequest = _FakeListDocumentBlockRequest
+ bitable_v1 = ModuleType("lark_oapi.api.bitable.v1")
+ bitable_v1.ListAppTableRequest = _FakeTypedRequest
+ bitable_v1.ListAppTableFieldRequest = _FakeTypedRequest
+ bitable_v1.ListAppTableRecordRequest = _FakeTypedRequest
core_model = ModuleType("lark_oapi.core.model")
core_model.RequestOption = _FakeRequestOption
monkeypatch.setitem(sys.modules, "lark_oapi", lark)
monkeypatch.setitem(sys.modules, "lark_oapi.api.docx.v1", docx_v1)
+ monkeypatch.setitem(sys.modules, "lark_oapi.api.bitable.v1", bitable_v1)
monkeypatch.setitem(sys.modules, "lark_oapi.core.model", core_model)
@@ -185,9 +213,7 @@ def test_resolve_image_refs_uses_content_type_extension(monkeypatch):
accessor._config = SimpleNamespace(download_images=True)
accessor._client = SimpleNamespace(request=request_media)
- updated, images = accessor._resolve_image_refs(
- "![j](feishu://image/img_token_jpeg)"
- )
+ updated, images = accessor._resolve_image_refs("![j](feishu://image/img_token_jpeg)")
assert updated == "![j](images/img_token_jpeg.jpg)"
assert images == {"images/img_token_jpeg.jpg": b"\xff\xd8\xff\xe0jpeg-bytes"}
@@ -202,9 +228,7 @@ def test_resolve_image_refs_falls_back_to_byte_magic_extension(monkeypatch):
accessor._config = SimpleNamespace(download_images=True)
accessor._client = SimpleNamespace(request=request_media)
- updated, images = accessor._resolve_image_refs(
- "![w](feishu://image/img_token_webp)"
- )
+ updated, images = accessor._resolve_image_refs("![w](feishu://image/img_token_webp)")
assert updated == "![w](images/img_token_webp.webp)"
assert images == {"images/img_token_webp.webp": webp_bytes}
@@ -338,6 +362,323 @@ def test_access_writes_downloaded_images_next_to_markdown(monkeypatch):
assert not resource.path.parent.exists()
+def test_fetch_document_dispatches_all_supported_types(monkeypatch):
+ accessor = FeishuAccessor()
+ handlers = {
+ "_parse_docx": MagicMock(return_value=("docx body", "Doc")),
+ "_parse_sheets": MagicMock(return_value=("sheet body", "Sheet")),
+ "_parse_bitable": MagicMock(return_value=("base body", "Base")),
+ }
+ for name, handler in handlers.items():
+ monkeypatch.setattr(accessor, name, handler)
+
+ docx = asyncio.run(
+ accessor._fetch_document(
+ "https://example.feishu.cn/docx/doc_token",
+ feishu_access_token="u-test",
+ )
+ )
+ sheets = asyncio.run(accessor._fetch_document("https://example.feishu.cn/sheets/sht_token"))
+ base = asyncio.run(accessor._fetch_document("https://example.feishu.cn/base/app_token"))
+ monkeypatch.setattr(
+ accessor,
+ "_resolve_wiki_node",
+ MagicMock(return_value=("base", "wiki_app_token", "Wiki Base")),
+ )
+ wiki = asyncio.run(accessor._fetch_document("https://example.feishu.cn/wiki/wiki_token"))
+
+ assert (docx.doc_type, sheets.doc_type, base.doc_type, wiki.doc_type) == (
+ "docx",
+ "sheets",
+ "base",
+ "base",
+ )
+ assert wiki.title == "Wiki Base"
+ handlers["_parse_docx"].assert_called_once_with("doc_token", "u-test")
+ handlers["_parse_sheets"].assert_called_once_with("sht_token", None)
+ assert handlers["_parse_bitable"].call_args_list[-1].args == ("wiki_app_token", None)
+
+
+def test_fetch_document_honors_bitable_table_and_view(monkeypatch):
+ _install_fake_lark_modules(monkeypatch)
+ list_tables = MagicMock()
+ list_fields = MagicMock(
+ return_value=_SuccessResponse(
+ SimpleNamespace(
+ items=[SimpleNamespace(field_name="Status")],
+ has_more=False,
+ page_token=None,
+ )
+ )
+ )
+ list_records = MagicMock(
+ return_value=_SuccessResponse(
+ SimpleNamespace(
+ items=[SimpleNamespace(fields={"Status": "Published"})],
+ has_more=False,
+ page_token=None,
+ )
+ )
+ )
+ accessor = FeishuAccessor()
+ accessor._config = SimpleNamespace(max_records_per_table=10)
+ accessor._client = SimpleNamespace(
+ bitable=SimpleNamespace(
+ v1=SimpleNamespace(
+ app_table=SimpleNamespace(list=list_tables),
+ app_table_field=SimpleNamespace(list=list_fields),
+ app_table_record=SimpleNamespace(list=list_records),
+ )
+ )
+ )
+
+ document = asyncio.run(
+ accessor._fetch_document(
+ "https://example.feishu.cn/base/app_token?table=tblSales&view=vewPublic"
+ )
+ )
+
+ list_tables.assert_not_called()
+ request = list_records.call_args.args[0]
+ assert (request.table_id, request.view_id) == ("tblSales", "vewPublic")
+ assert document.title == "tblSales (vewPublic)"
+ assert document.meta["feishu_table_id"] == "tblSales"
+ assert document.meta["feishu_view_id"] == "vewPublic"
+
+
+def test_fetch_document_rejects_bitable_view_without_table():
+ accessor = FeishuAccessor()
+
+ with pytest.raises(ValueError, match="'view'.*'table'"):
+ asyncio.run(
+ accessor._fetch_document("https://example.feishu.cn/base/app_token?view=vewPublic")
+ )
+
+
+def test_parse_sheets_handles_grid_and_embedded_bitable(monkeypatch):
+ _install_fake_lark_modules(monkeypatch)
+ request = MagicMock(
+ side_effect=[
+ _FakeMediaResponse(
+ b'{"data":{"properties":{"title":"Budget"},"sheets":['
+ b'{"sheetId":"sheet-1","title":"Q1","rowCount":3,"columnCount":28},'
+ b'{"sheetId":"block-1","title":"Content Calendar","rowCount":0,'
+ b'"columnCount":0,"blockInfo":{"blockType":"BITABLE_BLOCK",'
+ b'"blockToken":"app-token_table-1"}}]}}'
+ ),
+ _FakeMediaResponse(b'{"data":{"valueRange":{"values":[["name","amount"],["A",1]]}}}'),
+ _FakeMediaResponse(b"\x89PNG\r\n"),
+ ]
+ )
+ list_tables = MagicMock()
+ list_fields = MagicMock(
+ return_value=_SuccessResponse(
+ SimpleNamespace(
+ items=[
+ SimpleNamespace(field_name="Topic"),
+ SimpleNamespace(field_name="Cover"),
+ SimpleNamespace(field_name="Brief"),
+ ],
+ has_more=False,
+ page_token=None,
+ )
+ )
+ )
+ list_records = MagicMock(
+ return_value=_SuccessResponse(
+ SimpleNamespace(
+ items=[
+ SimpleNamespace(
+ fields={
+ "Topic": "Welcome",
+ "Cover": [
+ {
+ "file_token": "cover-token",
+ "name": "cover.png",
+ "type": "image/png",
+ }
+ ],
+ "Brief": [
+ {
+ "file_token": "brief-token",
+ "name": "brief.pdf",
+ "type": "application/pdf",
+ }
+ ],
+ }
+ )
+ ],
+ has_more=False,
+ page_token=None,
+ )
+ )
+ )
+ accessor = FeishuAccessor()
+ accessor._config = SimpleNamespace(
+ max_rows_per_sheet=2,
+ max_records_per_table=10,
+ download_images=True,
+ )
+ accessor._user_token_client = SimpleNamespace(
+ request=request,
+ bitable=SimpleNamespace(
+ v1=SimpleNamespace(
+ app_table=SimpleNamespace(list=list_tables),
+ app_table_field=SimpleNamespace(list=list_fields),
+ app_table_record=SimpleNamespace(list=list_records),
+ )
+ ),
+ )
+
+ markdown, title = accessor._parse_sheets("sht_token", "u-test")
+
+ assert title == "Budget"
+ assert "| name | amount |" in markdown
+ assert "1 more rows truncated" in markdown
+ assert "2 columns after Z omitted" in markdown
+ assert "### Content Calendar" in markdown
+ assert "Welcome" in markdown
+ assert "![cover.png](feishu://image/cover-token)" in markdown
+ assert "brief.pdf" in markdown
+ assert "feishu://image/brief-token" not in markdown
+ assert "Empty sheet" not in markdown
+ assert list_tables.call_count == 0
+ assert list_fields.call_args.args[0].table_id == "table-1"
+
+ resolved, images = accessor._resolve_image_refs(
+ markdown,
+ feishu_access_token="u-test",
+ )
+
+ assert "![cover.png](images/cover-token.png)" in resolved
+ assert images == {"images/cover-token.png": b"\x89PNG\r\n"}
+ assert request.call_args_list[-1].args[0].uri == (
+ "/open-apis/drive/v1/medias/cover-token/download"
+ )
+ assert all(call.args[0].token_types == {"user"} for call in request.call_args_list)
+ assert all(call.args[1].user_access_token == "u-test" for call in request.call_args_list)
+
+
+def test_parse_bitable_uses_user_token_and_formats_records(monkeypatch):
+ _install_fake_lark_modules(monkeypatch)
+ list_tables = MagicMock(
+ side_effect=[
+ _SuccessResponse(
+ SimpleNamespace(
+ items=[SimpleNamespace(table_id="table-1", name="Leads")],
+ has_more=True,
+ page_token="tables-2",
+ )
+ ),
+ _SuccessResponse(
+ SimpleNamespace(
+ items=[SimpleNamespace(table_id="table-2", name="Companies")],
+ has_more=False,
+ page_token=None,
+ )
+ ),
+ ]
+ )
+ list_fields = MagicMock(
+ side_effect=[
+ _SuccessResponse(
+ SimpleNamespace(
+ items=[SimpleNamespace(field_name="Owner")],
+ has_more=True,
+ page_token="fields-2",
+ )
+ ),
+ _SuccessResponse(
+ SimpleNamespace(
+ items=[SimpleNamespace(field_name="Status")],
+ has_more=False,
+ page_token=None,
+ )
+ ),
+ _SuccessResponse(
+ SimpleNamespace(
+ items=[SimpleNamespace(field_name="Name")],
+ has_more=False,
+ page_token=None,
+ )
+ ),
+ ]
+ )
+ list_records = MagicMock(
+ side_effect=[
+ _SuccessResponse(
+ SimpleNamespace(
+ items=[SimpleNamespace(fields={"Owner": [{"name": "Alice"}], "Status": "New"})],
+ has_more=False,
+ page_token=None,
+ )
+ ),
+ _SuccessResponse(
+ SimpleNamespace(
+ items=[SimpleNamespace(fields={"Name": "Acme"})],
+ has_more=False,
+ page_token=None,
+ )
+ ),
+ ]
+ )
+ accessor = FeishuAccessor()
+ accessor._config = SimpleNamespace(max_records_per_table=10)
+ accessor._user_token_client = SimpleNamespace(
+ bitable=SimpleNamespace(
+ v1=SimpleNamespace(
+ app_table=SimpleNamespace(list=list_tables),
+ app_table_field=SimpleNamespace(list=list_fields),
+ app_table_record=SimpleNamespace(list=list_records),
+ )
+ )
+ )
+
+ markdown, title = accessor._parse_bitable("app_token", "u-test")
+
+ assert title == "Bitable (2 tables)"
+ assert "## Leads" in markdown
+ assert "## Companies" in markdown
+ assert "Alice" in markdown
+ assert "Acme" in markdown
+ assert "records truncated" not in markdown
+ assert list_tables.call_args_list[1].args[0].page_token == "tables-2"
+ assert list_fields.call_args_list[1].args[0].page_token == "fields-2"
+ assert all(call.args[1].user_access_token == "u-test" for call in list_records.call_args_list)
+
+
+def test_embedded_sheet_uses_same_user_token(monkeypatch):
+ _install_fake_lark_modules(monkeypatch)
+ inspect_block = MagicMock(
+ return_value=_FakeMediaResponse(
+ b'{"data":{"block":{"sheet":{"token":"spreadsheet-1_sheet-1"}}}}'
+ )
+ )
+ accessor = FeishuAccessor()
+ accessor._user_token_client = SimpleNamespace(request=inspect_block)
+ read_range = MagicMock(return_value=[["name", "amount"], ["A", "1"]])
+ monkeypatch.setattr(accessor, "_read_sheet_range", read_range)
+ block = SimpleNamespace(
+ block_id="block-1",
+ block_type=30,
+ parent_id="doc-1",
+ sheet=SimpleNamespace(),
+ )
+
+ markdown = accessor._block_to_markdown(
+ block,
+ {},
+ {},
+ document_id="doc-1",
+ feishu_access_token="u-test",
+ )
+
+ assert "| name | amount |" in markdown
+ assert inspect_block.call_args.args[0].token_types == {"user"}
+ assert inspect_block.call_args.args[1].user_access_token == "u-test"
+ assert read_range.call_args.kwargs["feishu_access_token"] == "u-test"
+
+
def test_access_keeps_raw_title_but_exposes_safe_original_filename(monkeypatch):
accessor = FeishuAccessor()
accessor._config = SimpleNamespace(download_images=False)
diff --git a/tests/parse/test_feishu_parser.py b/tests/parse/test_feishu_parser.py
deleted file mode 100644
index 24272fdfb..000000000
--- a/tests/parse/test_feishu_parser.py
+++ /dev/null
@@ -1,649 +0,0 @@
-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
-# SPDX-License-Identifier: AGPL-3.0
-"""Tests for FeishuParser."""
-
-import asyncio
-from types import SimpleNamespace
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-from openviking.parse.parsers.feishu import FeishuParser
-
-
-def _make_block(**kwargs):
- """Create a mock block object with only the specified attributes populated."""
- # Start with all common attributes as None
- defaults = {
- "block_id": "test_id",
- "block_type": 0,
- "parent_id": "parent_id",
- "children": None,
- "comment_ids": None,
- "add_ons": None,
- "page": None,
- "text": None,
- "heading1": None,
- "heading2": None,
- "heading3": None,
- "heading4": None,
- "heading5": None,
- "heading6": None,
- "heading7": None,
- "heading8": None,
- "heading9": None,
- "bullet": None,
- "ordered": None,
- "code": None,
- "quote": None,
- "todo": None,
- "divider": None,
- "image": None,
- "table": None,
- "table_cell": None,
- "quote_container": None,
- "sheet": None,
- "callout": None,
- "equation": None,
- "task": None,
- }
- defaults.update(kwargs)
- return SimpleNamespace(**defaults)
-
-
-def _make_text_content(
- text: str, bold=False, italic=False, inline_code=False, strikethrough=False, link_url=None
-):
- """Create a mock text content object with elements."""
- style = SimpleNamespace(
- bold=bold,
- italic=italic,
- inline_code=inline_code,
- strikethrough=strikethrough,
- link=SimpleNamespace(url=link_url) if link_url else None,
- )
- element = SimpleNamespace(
- text_run=SimpleNamespace(content=text, text_element_style=style),
- mention_user=None,
- mention_doc=None,
- equation=None,
- )
- return SimpleNamespace(elements=[element], style=None)
-
-
-class TestParseFeishuUrl:
- def test_docx_url(self):
- doc_type, token = FeishuParser._parse_feishu_url(
- "https://example.feishu.cn/docx/doxcnABC123"
- )
- assert doc_type == "docx"
- assert token == "doxcnABC123"
-
- def test_wiki_url(self):
- doc_type, token = FeishuParser._parse_feishu_url("https://example.feishu.cn/wiki/wikiXYZ")
- assert doc_type == "wiki"
- assert token == "wikiXYZ"
-
- def test_sheets_url(self):
- doc_type, token = FeishuParser._parse_feishu_url(
- "https://example.feishu.cn/sheets/shtcn123"
- )
- assert doc_type == "sheets"
-
- def test_base_url(self):
- doc_type, token = FeishuParser._parse_feishu_url(
- "https://example.feishu.cn/base/bascn999?table=tbl1"
- )
- assert doc_type == "base"
- assert token == "bascn999"
-
- def test_larksuite_url(self):
- doc_type, token = FeishuParser._parse_feishu_url(
- "https://example.larksuite.com/docx/abc123"
- )
- assert doc_type == "docx"
-
- def test_invalid_url_raises(self):
- with pytest.raises(ValueError, match="Cannot parse"):
- FeishuParser._parse_feishu_url("https://example.feishu.cn/")
-
- def test_invalid_host_raises(self):
- with pytest.raises(ValueError, match="host not allowed"):
- FeishuParser._parse_feishu_url("https://evil.example/docx/doxcnABC123")
-
-
-class TestBlockToMarkdown:
- """Test attribute-driven block-to-markdown conversion."""
-
- def setup_method(self):
- self.parser = FeishuParser()
-
- def test_text_block(self):
- block = _make_block(text=_make_text_content("Hello world"))
- result = self.parser._block_to_markdown(block, {}, {})
- assert result == "Hello world"
-
- def test_heading_blocks(self):
- for level in range(1, 7):
- content = _make_text_content(f"Heading {level}")
- block = _make_block(**{f"heading{level}": content})
- result = self.parser._block_to_markdown(block, {}, {})
- assert result == f"{'#' * level} Heading {level}"
-
- def test_bullet_list(self):
- block = _make_block(bullet=_make_text_content("Item one"))
- result = self.parser._block_to_markdown(block, {}, {})
- assert result == "- Item one"
-
- def test_ordered_list(self):
- counter: dict = {}
- block = _make_block(ordered=_make_text_content("First"))
- result = self.parser._block_to_markdown(block, {}, counter)
- assert result == "1. First"
-
- block2 = _make_block(ordered=_make_text_content("Second"))
- result2 = self.parser._block_to_markdown(block2, {}, counter)
- assert result2 == "2. Second"
-
- def test_code_block(self):
- code_content = _make_text_content("print('hello')")
- code_content.style = SimpleNamespace(language="python")
- block = _make_block(code=code_content)
- result = self.parser._block_to_markdown(block, {}, {})
- assert result == "```python\nprint('hello')\n```"
-
- def test_quote_block(self):
- block = _make_block(quote=_make_text_content("A quote"))
- result = self.parser._block_to_markdown(block, {}, {})
- assert result == "> A quote"
-
- def test_todo_block(self):
- todo_content = _make_text_content("Buy milk")
- todo_content.style = SimpleNamespace(done=True)
- block = _make_block(todo=todo_content)
- result = self.parser._block_to_markdown(block, {}, {})
- assert result == "- [x] Buy milk"
-
- def test_divider_block(self):
- block = _make_block(divider=SimpleNamespace())
- result = self.parser._block_to_markdown(block, {}, {})
- assert result == "---"
-
- def test_image_block(self):
- block = _make_block(image=SimpleNamespace(token="img_token_123", alt="screenshot"))
- result = self.parser._block_to_markdown(block, {}, {})
- assert result == "![screenshot](feishu://image/img_token_123)"
-
- def test_skip_page(self):
- block = _make_block(page=SimpleNamespace(elements=[]))
- result = self.parser._block_to_markdown(block, {}, {})
- assert result is None # page is in _SKIP_ATTRS
-
- def test_skip_table_cell(self):
- block = _make_block(table_cell=SimpleNamespace())
- result = self.parser._block_to_markdown(block, {}, {})
- assert result is None
-
- def test_unknown_with_elements_extracts_text(self):
- """Unknown block type with text elements should still extract content."""
- block = _make_block(callout=_make_text_content("Important note"))
- result = self.parser._block_to_markdown(block, {}, {})
- assert result == "Important note"
-
-
-class TestApplyTextStyle:
- def test_bold(self):
- assert (
- FeishuParser._apply_text_style(
- "text",
- SimpleNamespace(
- bold=True, italic=False, strikethrough=False, inline_code=False, link=None
- ),
- )
- == "**text**"
- )
-
- def test_italic(self):
- assert (
- FeishuParser._apply_text_style(
- "text",
- SimpleNamespace(
- bold=False, italic=True, strikethrough=False, inline_code=False, link=None
- ),
- )
- == "*text*"
- )
-
- def test_inline_code(self):
- assert (
- FeishuParser._apply_text_style(
- "code",
- SimpleNamespace(
- bold=False, italic=False, strikethrough=False, inline_code=True, link=None
- ),
- )
- == "`code`"
- )
-
- def test_link(self):
- result = FeishuParser._apply_text_style(
- "click",
- SimpleNamespace(
- bold=False,
- italic=False,
- strikethrough=False,
- inline_code=False,
- link=SimpleNamespace(url="https://example.com"),
- ),
- )
- assert result == "[click](https://example.com)"
-
- def test_empty_text(self):
- assert FeishuParser._apply_text_style("", SimpleNamespace(bold=True)) == ""
-
- def test_none_style(self):
- assert FeishuParser._apply_text_style("text", None) == "text"
-
-
-class TestFormatBitableField:
- def test_none(self):
- assert FeishuParser._format_bitable_field(None) == ""
-
- def test_string(self):
- assert FeishuParser._format_bitable_field("hello") == "hello"
-
- def test_number(self):
- assert FeishuParser._format_bitable_field(42) == "42"
-
- def test_list_of_dicts(self):
- result = FeishuParser._format_bitable_field([{"text": "A"}, {"name": "B"}])
- assert result == "A, B"
-
- def test_dict_with_text(self):
- assert FeishuParser._format_bitable_field({"text": "value"}) == "value"
-
-
-class TestTrimEmptyColumns:
- def test_trim(self):
- rows = [["a", "b", "", ""], ["c", "d", "", ""]]
- result = FeishuParser._trim_empty_columns(rows)
- assert result == [["a", "b"], ["c", "d"]]
-
- def test_no_trim_needed(self):
- rows = [["a", "b"], ["c", "d"]]
- assert FeishuParser._trim_empty_columns(rows) == rows
-
- def test_all_empty(self):
- rows = [["", ""], ["", ""]]
- assert FeishuParser._trim_empty_columns(rows) == []
-
-
-# ========== Mock Integration Tests ==========
-
-
-def _mock_list_blocks_response(blocks, has_more=False, page_token=None):
- """Create a mock response for docx.v1.document_block.list()."""
- resp = MagicMock()
- resp.success.return_value = True
- resp.data.items = blocks
- resp.data.has_more = has_more
- resp.data.page_token = page_token
- return resp
-
-
-def _make_sdk_block(block_id, parent_id="doc_id", **attrs):
- """Create a mock SDK block with all attributes defaulting to None."""
- defaults = {
- "block_id": block_id,
- "block_type": 0,
- "parent_id": parent_id,
- "children": None,
- "comment_ids": None,
- "add_ons": None,
- "page": None,
- "text": None,
- "heading1": None,
- "heading2": None,
- "heading3": None,
- "heading4": None,
- "heading5": None,
- "heading6": None,
- "heading7": None,
- "heading8": None,
- "heading9": None,
- "bullet": None,
- "ordered": None,
- "code": None,
- "quote": None,
- "todo": None,
- "divider": None,
- "image": None,
- "table": None,
- "table_cell": None,
- "quote_container": None,
- "sheet": None,
- "callout": None,
- "equation": None,
- "task": None,
- }
- defaults.update(attrs)
- return SimpleNamespace(**defaults)
-
-
-class TestParseDocxIntegration:
- """Integration tests for _parse_docx with mocked lark-oapi client."""
-
- def _make_parser_with_mock_client(self, list_response):
- """Create a FeishuParser with a mocked lark-oapi client."""
- parser = FeishuParser()
- mock_client = MagicMock()
- mock_client.docx.v1.document_block.list.return_value = list_response
- parser._client = mock_client
- return parser
-
- def test_parse_docx_basic(self):
- """Test basic document with page title, heading, and text."""
- blocks = [
- _make_sdk_block(
- "page_id",
- page=SimpleNamespace(
- elements=[
- SimpleNamespace(
- text_run=SimpleNamespace(
- content="My Document", text_element_style=None
- ),
- mention_user=None,
- mention_doc=None,
- equation=None,
- )
- ]
- ),
- ),
- _make_sdk_block("h1_id", heading2=_make_text_content("Introduction")),
- _make_sdk_block("t1_id", text=_make_text_content("Hello world")),
- ]
- response = _mock_list_blocks_response(blocks)
- parser = self._make_parser_with_mock_client(response)
-
- markdown, title = parser._parse_docx("test_doc_id")
-
- assert title == "My Document"
- assert "# My Document" in markdown
- assert "## Introduction" in markdown
- assert "Hello world" in markdown
-
- def test_parse_docx_with_pagination(self):
- """Test document fetching with multiple pages of blocks."""
- page1_blocks = [
- _make_sdk_block(
- "page_id",
- page=SimpleNamespace(
- elements=[
- SimpleNamespace(
- text_run=SimpleNamespace(
- content="Paginated Doc", text_element_style=None
- ),
- mention_user=None,
- mention_doc=None,
- equation=None,
- )
- ]
- ),
- ),
- _make_sdk_block("t1_id", text=_make_text_content("Page 1 content")),
- ]
- page2_blocks = [
- _make_sdk_block("t2_id", text=_make_text_content("Page 2 content")),
- ]
-
- resp1 = _mock_list_blocks_response(page1_blocks, has_more=True, page_token="token2")
- resp2 = _mock_list_blocks_response(page2_blocks, has_more=False)
-
- parser = FeishuParser()
- mock_client = MagicMock()
- mock_client.docx.v1.document_block.list.side_effect = [resp1, resp2]
- parser._client = mock_client
-
- markdown, title = parser._parse_docx("test_doc_id")
-
- assert title == "Paginated Doc"
- assert "Page 1 content" in markdown
- assert "Page 2 content" in markdown
- assert mock_client.docx.v1.document_block.list.call_count == 2
-
- def test_parse_docx_empty(self):
- """Test empty document."""
- response = _mock_list_blocks_response([])
- parser = self._make_parser_with_mock_client(response)
-
- markdown, title = parser._parse_docx("empty_doc")
-
- assert markdown == ""
- assert title == "Untitled"
-
- def test_parse_docx_api_error(self):
- """Test handling of API error."""
- parser = FeishuParser()
- mock_client = MagicMock()
- error_resp = MagicMock()
- error_resp.success.return_value = False
- error_resp.code = 403
- error_resp.msg = "permission denied"
- mock_client.docx.v1.document_block.list.return_value = error_resp
- parser._client = mock_client
-
- with pytest.raises(RuntimeError, match="permission denied"):
- parser._parse_docx("forbidden_doc")
-
- def test_parse_docx_mixed_blocks(self):
- """Test document with various block types."""
- blocks = [
- _make_sdk_block(
- "page_id",
- page=SimpleNamespace(
- elements=[
- SimpleNamespace(
- text_run=SimpleNamespace(content="Report", text_element_style=None),
- mention_user=None,
- mention_doc=None,
- equation=None,
- )
- ]
- ),
- ),
- _make_sdk_block("h_id", heading2=_make_text_content("Section 1")),
- _make_sdk_block("b1_id", bullet=_make_text_content("Item A")),
- _make_sdk_block("b2_id", bullet=_make_text_content("Item B")),
- _make_sdk_block("div_id", divider=SimpleNamespace()),
- _make_sdk_block("q_id", quote=_make_text_content("A wise quote")),
- _make_sdk_block(
- "code_id",
- code=SimpleNamespace(
- elements=[
- SimpleNamespace(
- text_run=SimpleNamespace(content="x = 1", text_element_style=None),
- mention_user=None,
- mention_doc=None,
- equation=None,
- )
- ],
- style=SimpleNamespace(language="python"),
- ),
- ),
- ]
- response = _mock_list_blocks_response(blocks)
- parser = self._make_parser_with_mock_client(response)
-
- markdown, title = parser._parse_docx("test_doc")
-
- assert title == "Report"
- assert "## Section 1" in markdown
- assert "- Item A" in markdown
- assert "- Item B" in markdown
- assert "---" in markdown
- assert "> A wise quote" in markdown
- assert "```python\nx = 1\n```" in markdown
-
- def test_parse_docx_ordered_list_reset(self):
- """Test that ordered list counters reset between separate lists."""
- blocks = [
- _make_sdk_block("page_id", page=SimpleNamespace(elements=[])),
- _make_sdk_block("o1", ordered=_make_text_content("First")),
- _make_sdk_block("o2", ordered=_make_text_content("Second")),
- _make_sdk_block(
- "t1", text=_make_text_content("Break")
- ), # Non-ordered block resets counter
- _make_sdk_block("o3", ordered=_make_text_content("New first")),
- ]
- response = _mock_list_blocks_response(blocks)
- parser = self._make_parser_with_mock_client(response)
-
- markdown, _ = parser._parse_docx("test_doc")
-
- assert "1. First" in markdown
- assert "2. Second" in markdown
- assert "1. New first" in markdown
-
-
-class TestParseAsyncIntegration:
- """Test the async parse() entry point with mocked internals."""
-
- def test_parse_docx_url(self):
- """Test full parse() flow with a docx URL."""
- parser = FeishuParser()
-
- blocks = [
- _make_sdk_block(
- "page_id",
- page=SimpleNamespace(
- elements=[
- SimpleNamespace(
- text_run=SimpleNamespace(content="Test Doc", text_element_style=None),
- mention_user=None,
- mention_doc=None,
- equation=None,
- )
- ]
- ),
- ),
- _make_sdk_block("t1", text=_make_text_content("Content here")),
- ]
- mock_client = MagicMock()
- mock_client.docx.v1.document_block.list.return_value = _mock_list_blocks_response(blocks)
- parser._client = mock_client
-
- # Mock MarkdownParser to avoid VikingFS dependency
- mock_md_result = MagicMock()
- mock_md_result.source_format = "markdown"
- mock_md_result.parser_name = "MarkdownParser"
- mock_md_result.parse_time = 0.1
- mock_md_result.meta = {}
-
- mock_md_instance = MagicMock()
-
- async def _mock_parse_content(*a, **kw):
- return mock_md_result
-
- mock_md_instance.parse_content = _mock_parse_content
-
- with patch.object(parser, "_get_markdown_parser", return_value=mock_md_instance):
- result = asyncio.get_event_loop().run_until_complete(
- parser.parse("https://example.feishu.cn/docx/test123")
- )
-
- assert result.source_format == "feishu_docx"
- assert result.parser_name == "FeishuParser"
- assert result.meta["feishu_doc_type"] == "docx"
- assert result.meta["feishu_token"] == "test123"
-
- def test_parse_wiki_url_resolves(self):
- """Test that wiki URLs are resolved to underlying document type."""
- parser = FeishuParser()
-
- # Mock wiki resolution
- mock_client = MagicMock()
- # Use "doc" (not "docx") to test _WIKI_TYPE_MAP normalization: "doc" -> "docx"
- wiki_node = SimpleNamespace(obj_type="doc", obj_token="real_token", title="Wiki Page")
- wiki_resp = MagicMock()
- wiki_resp.success.return_value = True
- wiki_resp.data.node = wiki_node
- mock_client.wiki.v2.space.get_node.return_value = wiki_resp
-
- # Mock docx blocks
- blocks = [
- _make_sdk_block("page_id", page=SimpleNamespace(elements=[])),
- _make_sdk_block("t1", text=_make_text_content("Wiki content")),
- ]
- mock_client.docx.v1.document_block.list.return_value = _mock_list_blocks_response(blocks)
- parser._client = mock_client
-
- mock_md_result = MagicMock()
- mock_md_result.source_format = "markdown"
- mock_md_result.parser_name = "MarkdownParser"
- mock_md_result.parse_time = 0.1
- mock_md_result.meta = {}
-
- mock_md_instance = MagicMock()
-
- async def _mock_parse_content(*a, **kw):
- return mock_md_result
-
- mock_md_instance.parse_content = _mock_parse_content
-
- with patch.object(parser, "_get_markdown_parser", return_value=mock_md_instance):
- result = asyncio.get_event_loop().run_until_complete(
- parser.parse("https://example.feishu.cn/wiki/wiki_token")
- )
-
- assert result.source_format == "feishu_docx"
- assert result.meta["feishu_doc_type"] == "docx"
- assert result.meta["feishu_token"] == "real_token"
-
- def test_parse_content_routes_larkoffice_url(self):
- """Test that parse_content routes larkoffice.com URLs to FeishuParser.parse() correctly."""
- parser = FeishuParser()
-
- # Mock the underlying parse() method to avoid network/SDK calls
- mock_result = MagicMock()
- mock_result.source_format = "feishu_docx"
- mock_result.meta = {"feishu_token": "larkoffice123"}
-
- async def _mock_parse(*a, **kw):
- return mock_result
-
- parser.parse = _mock_parse
-
- result = asyncio.get_event_loop().run_until_complete(
- parser.parse_content(
- content="", source_path="https://bytedance.larkoffice.com/wiki/larkoffice123"
- )
- )
-
- assert result.source_format == "feishu_docx"
- assert result.meta["feishu_token"] == "larkoffice123"
-
- def test_parse_unsupported_type(self):
- """Test that unsupported document types return error ParseResult."""
- parser = FeishuParser()
- parser._client = MagicMock() # Won't be called
-
- result = asyncio.get_event_loop().run_until_complete(
- parser.parse("https://example.feishu.cn/mindnote/abc123")
- )
-
- assert result.warnings
- assert "Unsupported" in result.warnings[0]
-
- def test_parse_rejects_invalid_host(self):
- """Main parse() entry point must reject non-Feishu hosts."""
- parser = FeishuParser()
-
- result = asyncio.get_event_loop().run_until_complete(
- parser.parse("https://evil.example/docx/doxcn123")
- )
-
- assert result.source_format == "feishu"
- assert result.warnings
- assert "host not allowed" in result.warnings[0]
- assert "feishu_doc_type" not in result.meta
- assert "feishu_token" not in result.meta
diff --git a/tests/parse/test_feishu_parser_api.py b/tests/parse/test_feishu_parser_api.py
index 1de56bf4c..039b9b679 100644
--- a/tests/parse/test_feishu_parser_api.py
+++ b/tests/parse/test_feishu_parser_api.py
@@ -7,6 +7,7 @@ from unittest.mock import AsyncMock, Mock
import pytest
+from openviking.parse.accessors.base import LocalResource, SourceType
from openviking.parse.understanding_api import PREPARED_RESPONSE_ID_ARG, UnderstandingAPI
from openviking.server.identity import RequestContext, Role
from openviking.service.resource_service import ResourceService
@@ -23,7 +24,7 @@ from openviking_cli.session.user_id import UserIdentifier
async def test_feishu_parser_api_bypasses_accessor():
result = object()
router = SimpleNamespace(
- should_use_understanding_api=lambda source: True,
+ should_use_understanding_directly=Mock(return_value=True),
parse=AsyncMock(return_value=result),
)
processor = UnifiedResourceProcessor(vlm_processor=object())
@@ -39,36 +40,22 @@ async def test_feishu_parser_api_bypasses_accessor():
router.parse.assert_awaited_once()
(call_source,) = router.parse.await_args.args
assert call_source == source
- assert router.parse.await_args.kwargs["lark_file"] == {"user_access_token": "u-test"}
+ assert router.parse.await_args.kwargs["feishu_access_token"] == " u-test "
assert "resource_name" not in router.parse.await_args.kwargs
@pytest.mark.asyncio
async def test_feishu_parser_api_uses_app_credentials_for_tenant_token(monkeypatch):
- result = object()
- router = SimpleNamespace(
- should_use_understanding_api=lambda source: True,
- parse=AsyncMock(return_value=result),
- )
oauth_client = SimpleNamespace(get_tenant_access_token=AsyncMock(return_value="t-test"))
- processor = UnifiedResourceProcessor(vlm_processor=object())
- processor._parser_router = router
- processor._accessor_registry = SimpleNamespace(
- access=AsyncMock(side_effect=AssertionError("accessor should not be called"))
- )
- monkeypatch.setattr(
- "openviking.resource.feishu_watch_auth.load_feishu_app_credentials",
- Mock(return_value=object()),
- )
monkeypatch.setattr(
"openviking.resource.feishu_watch_auth.FeishuOAuthClient.from_config",
Mock(return_value=oauth_client),
)
+ api = _understanding_api_for_parse()
- actual = await processor.process("https://example.larkoffice.com/docx/doxcnToken")
+ auth = await api._resolve_lark_file({})
- assert actual is result
- assert router.parse.await_args.kwargs["lark_file"] == {"tenant_access_token": "t-test"}
+ assert auth == {"tenant_access_token": "t-test"}
oauth_client.get_tenant_access_token.assert_awaited_once_with()
@@ -76,7 +63,9 @@ async def test_feishu_parser_api_uses_app_credentials_for_tenant_token(monkeypat
async def test_prepared_feishu_response_bypasses_accessor_without_credentials(monkeypatch):
result = object()
router = SimpleNamespace(
- should_use_understanding_api=lambda source: True,
+ should_use_understanding_api=Mock(
+ side_effect=AssertionError("prepared response must not re-evaluate routing config")
+ ),
parse=AsyncMock(return_value=result),
)
processor = UnifiedResourceProcessor(vlm_processor=object())
@@ -96,6 +85,73 @@ async def test_prepared_feishu_response_bypasses_accessor_without_credentials(mo
assert actual is result
assert router.parse.await_args.kwargs[PREPARED_RESPONSE_ID_ARG] == "response-1"
+ assert router.parse.await_args.kwargs["parser_backend"] == "understanding"
+ assert "lark_file" not in router.parse.await_args.kwargs
+
+
+async def test_feishu_without_understanding_auth_falls_back_to_accessor(monkeypatch, tmp_path):
+ markdown_path = tmp_path / "document.md"
+ markdown_path.write_text("content", encoding="utf-8")
+ resource = LocalResource(
+ path=markdown_path,
+ source_type=SourceType.FEISHU,
+ original_source="https://example.larkoffice.com/docx/doxcnToken",
+ meta={"extension": ".md"},
+ is_temporary=False,
+ )
+ result = object()
+ router = SimpleNamespace(
+ should_use_understanding_directly=Mock(return_value=False),
+ parse=AsyncMock(return_value=result),
+ )
+ accessor = SimpleNamespace(access=AsyncMock(return_value=resource))
+ processor = UnifiedResourceProcessor(vlm_processor=object())
+ processor._parser_router = router
+ processor._accessor_registry = accessor
+ monkeypatch.setattr(
+ "openviking.resource.feishu_watch_auth.load_feishu_app_credentials",
+ Mock(side_effect=ValueError("missing credentials")),
+ )
+
+ actual = await processor.process(resource.original_source)
+
+ assert actual is result
+ accessor.access.assert_awaited_once_with(resource.original_source)
+ router.parse.assert_awaited_once()
+ assert router.parse.await_args.args[0] is resource
+ assert "lark_file" not in router.parse.await_args.kwargs
+
+
+async def test_prepared_feishu_resource_is_not_bypassed(monkeypatch, tmp_path):
+ markdown_path = tmp_path / "document.md"
+ markdown_path.write_text("content", encoding="utf-8")
+ resource = LocalResource(
+ path=markdown_path,
+ source_type=SourceType.FEISHU,
+ original_source="https://example.larkoffice.com/docx/doxcnToken",
+ meta={"resolved_extension": ".md"},
+ is_temporary=False,
+ )
+ result = object()
+ router = SimpleNamespace(
+ should_use_understanding_api=Mock(
+ side_effect=AssertionError("prepared resources must not be routed before parsing")
+ ),
+ parse=AsyncMock(return_value=result),
+ )
+ processor = UnifiedResourceProcessor(vlm_processor=object())
+ processor._parser_router = router
+
+ actual = await processor.process(
+ resource.original_source,
+ prepared_resource=resource,
+ feishu_access_token="u-test",
+ )
+
+ assert actual is result
+ assert router.parse.await_args.args[0] is resource
assert "lark_file" not in router.parse.await_args.kwargs
@@ -217,7 +273,11 @@ async def test_legacy_accessor_output_does_not_enable_lark_protocol(tmp_path: Pa
zf.writestr("parsed/0.md", "content")
api = _understanding_api_for_parse()
- api._create_response_for_url = AsyncMock(return_value={"id": "response-1"})
+ api._create_file = AsyncMock(return_value={"id": "file-1"})
+ api._create_response_for_file = AsyncMock(return_value={"id": "response-1"})
+ api._create_response_for_url = AsyncMock(
+ side_effect=AssertionError("accessor output must stay local")
+ )
api._poll_response = AsyncMock(
return_value={"status": "completed", "result": {"zip_url": "https://tos/result.zip"}}
)
@@ -230,11 +290,9 @@ async def test_legacy_accessor_output_does_not_enable_lark_protocol(tmp_path: Pa
feishu_access_token="u-test",
)
- api._create_response_for_url.assert_awaited_once_with(
- url="https://example.larkoffice.com/docx/doxcnToken",
- doc_type="unknown",
- lark_file=None,
- )
+ api._create_file.assert_awaited_once_with(local_path=markdown_path)
+ api._create_response_for_file.assert_awaited_once_with(file_id="file-1")
+ api._create_response_for_url.assert_not_awaited()
@pytest.mark.asyncio
@@ -336,11 +394,10 @@ def test_add_resource_message_round_trips_internal_fields():
async def test_uat_producer_payload_reaches_worker_without_persisting_token(monkeypatch):
source = "https://example.larkoffice.com/docx/doxcnToken"
root_uri = "viking://resources/lark/doxcnToken"
- parser_router = SimpleNamespace(
- should_use_understanding_api=lambda _source: True,
- submit_url=AsyncMock(return_value="response-1"),
- )
+ submit_understanding = AsyncMock(return_value="response-1")
resource_processor = SimpleNamespace(
+ should_use_understanding_directly=lambda _source, **_kwargs: True,
+ submit_understanding=submit_understanding,
tree_builder=SimpleNamespace(
resolve_target_uri=AsyncMock(return_value=(root_uri, root_uri))
),
@@ -377,14 +434,12 @@ async def test_uat_producer_payload_reaches_worker_without_persisting_token(monk
resource_processor=resource_processor,
skill_processor=SimpleNamespace(),
)
- service._parser_router = parser_router
monkeypatch.setattr(service, "_should_use_connector", Mock(return_value=False))
monkeypatch.setattr(
"openviking.service.resource_service.is_git_repo_url",
Mock(return_value=False),
)
monkeypatch.setattr("openviking.service.resource_service.uuid4", Mock(return_value="task-1"))
- monkeypatch.setattr(service, "_is_feishu_url", Mock(return_value=True))
ctx = RequestContext(
user=UserIdentifier("account-1", "user-1"),
role=Role.USER,
@@ -401,18 +456,22 @@ async def test_uat_producer_payload_reaches_worker_without_persisting_token(monk
assert initial_result == {"status": "success", "task_id": "task-1"}
assert task_tracker.create.await_args.kwargs["resource_id"] is None
- parser_router.submit_url.assert_awaited_once_with(
+ submit_understanding.assert_awaited_once_with(
source,
feishu_access_token="u-secret",
+ custom_option="forwarded",
)
payload = queue_manager.enqueue.await_args.args[1]
assert queue_manager.enqueue.await_args.args[0] == QueueManager.EXTERNAL_PARSE
assert "u-secret" not in json.dumps(payload)
assert payload["understanding_response_id"] == "response-1"
- assert payload["args"] == {"custom_option": "forwarded"}
+ assert payload["args"] == {
+ "custom_option": "forwarded",
+ "parser_backend": "understanding",
+ }
queued_msg = AddResourceMsg.from_dict(payload)
- service.add_resource = AsyncMock(
+ service._execute_resource_ingestion = AsyncMock(
return_value={
"status": "success",
"root_uri": "viking://resources/lark/真实文档标题",
@@ -425,9 +484,10 @@ async def test_uat_producer_payload_reaches_worker_without_persisting_token(monk
stage_callback=AsyncMock(),
)
- call = service.add_resource.await_args
+ call = service._execute_resource_ingestion.await_args
assert call.kwargs[PREPARED_RESPONSE_ID_ARG] == "response-1"
- assert call.kwargs["args"] == {"custom_option": "forwarded"}
+ assert call.kwargs["custom_option"] == "forwarded"
+ assert call.kwargs["parser_backend"] == "understanding"
@pytest.mark.asyncio
@@ -450,11 +510,8 @@ async def test_local_prepared_job_uses_add_resource_queue(monkeypatch):
resource_processor=resource_processor,
skill_processor=SimpleNamespace(),
)
- service._enqueue_add_resource_job = AsyncMock(
- return_value=SimpleNamespace(task_id="task-1")
- )
+ service._enqueue_add_resource_job = AsyncMock(return_value=SimpleNamespace(task_id="task-1"))
monkeypatch.setattr(service, "_should_use_connector", Mock(return_value=False))
- monkeypatch.setattr(service, "_should_use_understanding_api", Mock(return_value=False))
monkeypatch.setattr(
"openviking.service.resource_service.is_git_repo_url",
Mock(return_value=False),
@@ -495,14 +552,10 @@ async def test_uat_producer_cancellation_respects_queue_ownership(
else:
handoff.side_effect = asyncio.CancelledError
- parser_router = SimpleNamespace(
- should_use_understanding_api=lambda _source: True,
- submit_url=submit_url,
- )
resource_processor = SimpleNamespace(
- tree_builder=SimpleNamespace(
- resolve_target_uri=AsyncMock(return_value=(root_uri, None))
- ),
+ should_use_understanding_directly=lambda _source, **_kwargs: True,
+ submit_understanding=submit_url,
+ tree_builder=SimpleNamespace(resolve_target_uri=AsyncMock(return_value=(root_uri, None))),
)
task_tracker = SimpleNamespace(
create=AsyncMock(return_value=SimpleNamespace(task_id="task-1")),
@@ -546,13 +599,11 @@ async def test_uat_producer_cancellation_respects_queue_ownership(
resource_processor=resource_processor,
skill_processor=SimpleNamespace(),
)
- service._parser_router = parser_router
monkeypatch.setattr(service, "_should_use_connector", Mock(return_value=False))
monkeypatch.setattr(
"openviking.service.resource_service.is_git_repo_url",
Mock(return_value=False),
)
- monkeypatch.setattr(service, "_is_feishu_url", Mock(return_value=True))
ctx = RequestContext(
user=UserIdentifier("account-1", "user-1"),
role=Role.USER,
@@ -575,14 +626,23 @@ async def test_uat_producer_cancellation_respects_queue_ownership(
enqueue.assert_not_awaited()
else:
enqueue.assert_awaited_once()
-
-
-def test_prepared_response_id_is_reserved_from_public_args():
+ assert enqueue.await_args.args[0] == QueueManager.EXTERNAL_PARSE
+
+
+ ("field", "value"),
+ [
+ (PREPARED_RESPONSE_ID_ARG, "response-1"),
+ ("parser_backend", "understanding"),
+ ("resolved_extension", ".pdf"),
+ ],
+)
+def test_internal_parser_fields_are_reserved_from_public_args(field, value):
service = ResourceService()
- with pytest.raises(InvalidArgumentError, match=PREPARED_RESPONSE_ID_ARG):
+ with pytest.raises(InvalidArgumentError, match=field):
service._normalize_add_resource_args(
- {PREPARED_RESPONSE_ID_ARG: "response-1"},
+ {field: value},
watch_interval=0,
)
@@ -590,7 +650,7 @@ def test_prepared_response_id_is_reserved_from_public_args():
@pytest.mark.asyncio
async def test_add_resource_job_defers_target_and_expands_prepared_response():
service = ResourceService()
- service.add_resource = AsyncMock(
+ service._execute_resource_ingestion = AsyncMock(
return_value={
"status": "success",
"root_uri": "viking://resources/真实文档标题",
@@ -618,18 +678,17 @@ async def test_add_resource_job_defers_target_and_expands_prepared_response():
stage_callback=AsyncMock(),
)
- call = service.add_resource.await_args
+ call = service._execute_resource_ingestion.await_args
assert call.kwargs["to"] is None
assert call.kwargs["parent"] == "viking://resources/lark"
assert call.kwargs[PREPARED_RESPONSE_ID_ARG] == "response-1"
- assert call.kwargs["args"] == {}
assert result["root_uri"] == "viking://resources/真实文档标题"
@pytest.mark.asyncio
async def test_add_resource_job_expands_parser_args():
service = ResourceService()
- service.add_resource = AsyncMock(
+ service._execute_resource_ingestion = AsyncMock(
return_value={
"status": "success",
"root_uri": "viking://resources/doxcnToken",
@@ -656,10 +715,10 @@ async def test_add_resource_job_expands_parser_args():
stage_callback=AsyncMock(),
)
- call = service.add_resource.await_args
+ call = service._execute_resource_ingestion.await_args
assert call.kwargs["to"] == "viking://resources/doxcnToken"
assert call.kwargs["parent"] is None
- assert call.kwargs["args"] == {"custom_option": "forwarded"}
+ assert call.kwargs["custom_option"] == "forwarded"
@pytest.mark.asyncio
@@ -722,21 +781,18 @@ async def test_add_resource_processor_persists_final_resource_uri(monkeypatch):
assert queue_manager.enqueue.await_args.args[0] == QueueManager.ADD_RESOURCE
-def test_feishu_bypass_requires_configured_auth(monkeypatch):
- router = SimpleNamespace(
- should_use_understanding_api=lambda source: True,
- )
- processor = UnifiedResourceProcessor(vlm_processor=object())
- processor._parser_router = router
+def test_feishu_direct_submission_requires_configured_auth(monkeypatch):
+ api = _understanding_api_for_parse()
+ source = "https://example.larkoffice.com/docx/doxcnToken"
+
+ assert api.can_submit_url_directly(source, feishu_access_token="u-test")
+
monkeypatch.setattr(
"openviking.resource.feishu_watch_auth.load_feishu_app_credentials",
Mock(side_effect=ValueError("missing credentials")),
)
- assert not processor._should_bypass_feishu_accessor(
- "https://example.larkoffice.com/docx/doxcnToken",
- {},
- )
+ assert not api.can_submit_url_directly(source)
def test_normalize_lark_file_accepts_exactly_one_token():
diff --git a/tests/parse/test_parser_config_wiring.py b/tests/parse/test_parser_config_wiring.py
index cf25b215c..aaae5cc4c 100644
--- a/tests/parse/test_parser_config_wiring.py
+++ b/tests/parse/test_parser_config_wiring.py
@@ -1,10 +1,8 @@
"""Tests for parser config propagation through registries and composed parsers."""
-from openviking.parse.parsers.feishu import FeishuParser
from openviking.parse.parsers.html import HTMLParser
from openviking.parse.parsers.pdf import PDFParser
from openviking_cli.utils.config.parser_config import (
- FeishuConfig,
HTMLConfig,
PDFConfig,
)
@@ -26,12 +24,3 @@ def test_html_parser_passes_its_config_to_nested_markdown_parser():
assert markdown_parser.config.max_section_size == 2111
assert markdown_parser.config.max_section_chars == 5444
-
-
-def test_feishu_parser_passes_its_config_to_nested_markdown_parser():
- parser = FeishuParser(config=FeishuConfig(max_section_size=1999, max_section_chars=5333))
-
- markdown_parser = parser._get_markdown_parser()
-
- assert markdown_parser.config.max_section_size == 1999
- assert markdown_parser.config.max_section_chars == 5333
diff --git a/tests/parse/test_parser_router.py b/tests/parse/test_parser_router.py
index 6d289b063..564349213 100644
--- a/tests/parse/test_parser_router.py
+++ b/tests/parse/test_parser_router.py
@@ -1,6 +1,12 @@
from types import SimpleNamespace
+from unittest.mock import AsyncMock
+import pytest
+
+from openviking.parse.accessors.base import LocalResource, SourceType
from openviking.parse.parser_router import ParserRouter
+from openviking.parse.understanding_api import PREPARED_RESPONSE_ID_ARG
+from openviking.utils.media_processor import UnifiedResourceProcessor
def test_should_use_understanding_api_for_signed_video_url(monkeypatch):
@@ -23,6 +29,35 @@ def test_should_use_understanding_api_for_signed_video_url(monkeypatch):
)
+def test_resolved_extension_routes_extensionless_download(monkeypatch, tmp_path):
+ config = SimpleNamespace(
+ parser_api=SimpleNamespace(
+ enable=True,
+ enable_feishu_url=False,
+ extensions=["pdf"],
+ ),
+ )
+ monkeypatch.setattr(
+ "openviking_cli.utils.config.open_viking_config.get_openviking_config",
+ lambda: config,
+ )
+ downloaded = tmp_path / "download"
+ downloaded.write_bytes(b"%PDF-1.7")
+ resource = LocalResource(
+ path=downloaded,
+ source_type=SourceType.HTTP,
+ original_source="https://example.com/download?id=123",
+ meta={"extension": ".pdf"},
+ is_temporary=False,
+ )
+ processor = UnifiedResourceProcessor()
+
+ processor._set_resolved_identity(resource, source_name=None)
+
+ assert resource.meta["resolved_extension"] == ".pdf"
+ assert processor.should_use_understanding_api(resource)
+
+
def test_should_use_understanding_api_for_feishu_url(monkeypatch):
config = SimpleNamespace(
parser_api=SimpleNamespace(
@@ -37,19 +72,26 @@ def test_should_use_understanding_api_for_feishu_url(monkeypatch):
)
router = ParserRouter(parser_registry=object())
+ router._understanding_api = SimpleNamespace(
+ can_submit_url_directly=lambda _source, **kwargs: bool(kwargs.get("feishu_access_token"))
+ )
assert router.should_use_understanding_api("https://example.larkoffice.com/wiki/wikicnToken")
+ assert router.should_use_understanding_directly(
+ "https://example.larkoffice.com/wiki/wikicnToken",
+ feishu_access_token="u-test",
+ )
assert not router.should_use_understanding_api(
"https://larkoffice.com.evil.example/wiki/wikicnToken"
)
-def test_feishu_url_flag_defaults_to_extension_routing(monkeypatch):
+def test_feishu_url_flag_is_required_even_when_extension_matches(monkeypatch):
config = SimpleNamespace(
parser_api=SimpleNamespace(
enable=True,
enable_feishu_url=False,
- extensions=[],
+ extensions=["pdf"],
),
)
monkeypatch.setattr(
@@ -59,4 +101,116 @@ def test_feishu_url_flag_defaults_to_extension_routing(monkeypatch):
router = ParserRouter(parser_registry=object())
- assert not router.should_use_understanding_api("https://example.larkoffice.com/docx/doxcnToken")
+ assert not router.should_use_understanding_api(
+ "https://example.larkoffice.com/docx/doxcnToken.pdf"
+ )
+
+
+def test_directories_never_route_to_understanding(monkeypatch, tmp_path):
+ config = SimpleNamespace(
+ parser_api=SimpleNamespace(
+ enable=True,
+ enable_feishu_url=False,
+ extensions=["pdf"],
+ ),
+ )
+ monkeypatch.setattr(
+ "openviking_cli.utils.config.open_viking_config.get_openviking_config",
+ lambda: config,
+ )
+ resource = LocalResource(
+ path=tmp_path,
+ source_type=SourceType.HTTP,
+ original_source="https://example.com/site",
+ meta={"resolved_extension": ".pdf"},
+ is_temporary=False,
+ )
+
+ assert not UnifiedResourceProcessor().should_use_understanding_api(resource)
+
+
+async def test_normalized_feishu_markdown_stays_internal(monkeypatch, tmp_path):
+ config = SimpleNamespace(
+ parser_api=SimpleNamespace(
+ enable=True,
+ enable_feishu_url=True,
+ extensions=["md"],
+ ),
+ )
+ monkeypatch.setattr(
+ "openviking_cli.utils.config.open_viking_config.get_openviking_config",
+ lambda: config,
+ )
+ resource = LocalResource(
+ path=tmp_path / "document.md",
+ source_type=SourceType.FEISHU,
+ original_source="https://example.feishu.cn/docx/doc-token",
+ meta={"resolved_extension": ".md"},
+ is_temporary=False,
+ )
+ registry = SimpleNamespace(parse=AsyncMock(return_value=object()))
+ understanding = SimpleNamespace(parse=AsyncMock(return_value=object()))
+ router = ParserRouter(registry)
+ router._understanding_api = understanding
+ processor = UnifiedResourceProcessor()
+ processor._parser_router = router
+
+ assert not processor.should_use_understanding_api(resource)
+ await router.parse(
+ resource,
+ resolved_extension=".md",
+ parser_backend="understanding",
+ )
+
+ registry.parse.assert_awaited_once()
+ understanding.parse.assert_not_awaited()
+
+
+async def test_forced_resolved_extension_survives_worker_redownload(tmp_path):
+ downloaded = tmp_path / "redetected.docx"
+ downloaded.write_bytes(b"content")
+ resource = LocalResource(
+ path=downloaded,
+ source_type=SourceType.HTTP,
+ original_source="https://example.com/download",
+ meta={"resolved_extension": ".docx"},
+ is_temporary=False,
+ )
+ parser_router = SimpleNamespace(parse=AsyncMock(return_value=object()))
+ processor = UnifiedResourceProcessor(vlm_processor=object())
+ processor._parser_router = parser_router
+
+ await processor.process(
+ "https://example.com/download",
+ prepared_resource=resource,
+ resolved_extension=".pdf",
+ )
+
+ assert parser_router.parse.await_args.kwargs["resolved_extension"] == ".pdf"
+
+
+async def test_prepared_understanding_response_bypasses_remote_redownload():
+ result = object()
+ parser_router = SimpleNamespace(parse=AsyncMock(return_value=result))
+ accessor = SimpleNamespace(
+ access=AsyncMock(side_effect=AssertionError("prepared response must not redownload source"))
+ )
+ processor = UnifiedResourceProcessor(vlm_processor=object())
+ processor._parser_router = parser_router
+ processor._accessor_registry = accessor
+
+ actual = await processor.process(
+ "https://example.com/expiring-download",
+ resolved_extension=".pdf",
+ **{PREPARED_RESPONSE_ID_ARG: "response-1"},
+ )
+
+ assert actual is result
+ accessor.access.assert_not_awaited()
+ parser_router.parse.assert_awaited_once()
+ assert parser_router.parse.await_args.args == ("https://example.com/expiring-download",)
+ assert parser_router.parse.await_args.kwargs["parser_backend"] == "understanding"
+ assert parser_router.parse.await_args.kwargs[PREPARED_RESPONSE_ID_ARG] == "response-1"
diff --git a/tests/parse/test_understanding_api.py b/tests/parse/test_understanding_api.py
new file mode 100644
index 000000000..021f746c6
--- /dev/null
+++ b/tests/parse/test_understanding_api.py
@@ -0,0 +1,74 @@
+from pathlib import Path
+from unittest.mock import AsyncMock
+
+import pytest
+
+from openviking.parse.understanding_api import UnderstandingAPI
+
+
+async def test_parse_uses_downloaded_file_and_resolved_extension(monkeypatch, tmp_path):
+ downloaded = tmp_path / "download"
+ downloaded.write_bytes(b"%PDF-1.7")
+ zip_path = tmp_path / "result.zip"
+ zip_path.write_bytes(b"zip")
+ uploaded: list[Path] = []
+
+ api = UnderstandingAPI.__new__(UnderstandingAPI)
+ api._video_exts = {"mp4"}
+ api._audio_exts = {"mp3"}
+ api._image_exts = {"png"}
+
+ async def create_file(*, local_path):
+ uploaded.append(local_path)
+ return {"id": "file-1"}
+
+ async def create_response_for_file(*, file_id):
+ assert file_id == "file-1"
+ return {"id": "response-1"}
+
+ async def poll_response(*, response_id):
+ assert response_id == "response-1"
+ return {"status": "completed"}
+
+ monkeypatch.setattr(api, "_create_file", create_file)
+ monkeypatch.setattr(api, "_create_response_for_file", create_response_for_file)
+ monkeypatch.setattr(api, "_poll_response", poll_response)
+ monkeypatch.setattr(api, "_extract_zip_url", lambda _: "https://example.com/result.zip")
+ monkeypatch.setattr(api, "_download_zip", lambda _: _return(zip_path))
+ monkeypatch.setattr(
+ api,
+ "_unpack_zip_to_temp_dir",
+ lambda **_: _return("viking://temp/result"),
+ )
+
+ result = await api.parse(
+ downloaded,
+ original_source="https://example.com/download?id=123",
+ resource_name="report",
+ resolved_extension=".pdf",
+ )
+
+ assert uploaded == [downloaded]
+ assert result.source_path == "https://example.com/download?id=123"
+ assert result.source_format == "pdf"
+ assert result.root.title == "report"
+
+
+async def test_submit_file_returns_response_id(tmp_path):
+ source = tmp_path / "download.pdf"
+ source.write_bytes(b"%PDF-1.7")
+ api = UnderstandingAPI.__new__(UnderstandingAPI)
+ api._create_file = AsyncMock(return_value={"id": "file-1"})
+ api._create_response_for_file = AsyncMock(return_value={"id": "response-1"})
+
+ response_id = await api.submit_file(source)
+
+ assert response_id == "response-1"
+ api._create_file.assert_awaited_once_with(local_path=source)
+ api._create_response_for_file.assert_awaited_once_with(file_id="file-1")
+
+
+async def _return(value):
+ return value
diff --git a/tests/parse/test_understanding_api_artifact_images.py b/tests/parse/test_understanding_api_artifact_images.py
index e4db58118..b9317e799 100644
--- a/tests/parse/test_understanding_api_artifact_images.py
+++ b/tests/parse/test_understanding_api_artifact_images.py
@@ -16,6 +16,7 @@ class _FakeVikingFS:
def __init__(self):
self.files = {}
self.dirs = set()
+ self.deleted_temps = []
def create_temp_uri(self):
return "viking://temp/artifact"
@@ -29,6 +30,9 @@ class _FakeVikingFS:
async def write_file(self, uri, content):
self.files[uri] = content.encode("utf-8")
+ async def delete_temp(self, uri):
+ self.deleted_temps.append(uri)
+
def test_build_artifact_image_mappings_uses_existing_sibling_images(tmp_path: Path):
chapter = tmp_path / "章节"
@@ -80,3 +84,19 @@ async def test_unpack_artifact_writes_image_mapping_sidecar(tmp_path: Path):
"Ov测试_1.md": {"Ov测试_1_img1.png": "Ov测试_1_img1.png"}
}
assert fake_fs.files[f"{temp_uri}/resource/Ov测试_1_img1.png"] == b"png"
+
+
+async def test_unpack_artifact_cleans_temp_on_failure(tmp_path: Path):
+ invalid_zip = tmp_path / "invalid.zip"
+ invalid_zip.write_bytes(b"not-a-zip")
+ fake_fs = _FakeVikingFS()
+ api = UnderstandingAPI.__new__(UnderstandingAPI)
+
+ with (
+ patch("openviking.parse.understanding_api.get_viking_fs", return_value=fake_fs),
+ pytest.raises(zipfile.BadZipFile),
+ ):
+ await api._unpack_zip_to_temp_dir(invalid_zip, "resource")
+
+ assert fake_fs.deleted_temps == ["viking://temp/artifact"]
diff --git a/tests/resource/test_watch_scheduler.py b/tests/resource/test_watch_scheduler.py
index ee49727ed..36557e4cc 100644
--- a/tests/resource/test_watch_scheduler.py
+++ b/tests/resource/test_watch_scheduler.py
@@ -42,7 +42,7 @@ class TestWatchSchedulerResourceExistence:
super().__init__()
self.calls = []
- async def add_resource(self, **kwargs):
+ async def refresh_resource(self, **kwargs):
self.calls.append(kwargs)
return {"root_uri": kwargs.get("to")}
@@ -81,7 +81,7 @@ class TestWatchSchedulerResourceExistence:
super().__init__()
self.calls = []
- async def add_resource(self, **kwargs):
+ async def refresh_resource(self, **kwargs):
self.calls.append(kwargs)
return {"root_uri": kwargs.get("to")}
diff --git a/tests/service/test_resource_service_connector.py b/tests/service/test_resource_service_connector.py
index 66f4e6a97..d0a093253 100644
--- a/tests/service/test_resource_service_connector.py
+++ b/tests/service/test_resource_service_connector.py
@@ -4,7 +4,7 @@
import asyncio
from types import SimpleNamespace
-from unittest.mock import AsyncMock
+from unittest.mock import AsyncMock, Mock
import httpx
import pytest
@@ -12,6 +12,7 @@ import pytest
from openviking.server.identity import RequestContext, Role
from openviking.service import resource_service as resource_service_module
from openviking.service.resource_service import ResourceService
+from openviking.storage.queuefs.add_resource_msg import AddResourceMsg
from openviking_cli.exceptions import InvalidArgumentError
from openviking_cli.session.user_id import UserIdentifier
@@ -720,6 +721,66 @@ async def test_add_resource_falls_back_for_shared_source_with_parent(
service.enqueue_git_add_resource.assert_awaited_once()
+ ("field", "value"),
+ [
+ ("manage_watch", False),
+ ("parser_args", {}),
+ ("resource_lock", object()),
+ ("route_source", False),
+ ("skip_watch_management", True),
+ ("stage_callback", object()),
+ ("watch_auth_state", {}),
+ ("understanding_response_id", "response-1"),
+ ("parser_backend", "understanding"),
+ ("resolved_extension", ".pdf"),
+ ],
+)
+async def test_add_resource_rejects_internal_execution_fields(ctx, service, field, value):
+ with pytest.raises(InvalidArgumentError, match=field):
+ await service.add_resource(
+ path="https://example.com/manual.pdf",
+ ctx=ctx,
+ **{field: value},
+ )
+
+
+async def test_add_resource_job_executes_frozen_route(ctx, service):
+ service._execute_resource_ingestion = AsyncMock(
+ return_value={"status": "success", "root_uri": "root"}
+ )
+ service._should_use_connector = Mock()
+ resource_lock = SimpleNamespace(close=AsyncMock())
+ stage_callback = AsyncMock()
+ msg = AddResourceMsg(
+ task_id="task-1",
+ path="git://example.com/repo.git",
+ root_uri="viking://resources/repo",
+ account_id=ctx.account_id,
+ user_id=ctx.user.user_id,
+ role=str(ctx.role),
+ args={"parser_backend": "understanding"},
+ )
+
+ result = await service.execute_add_resource_job(
+ msg,
+ ctx=ctx,
+ resource_lock=resource_lock,
+ stage_callback=stage_callback,
+ )
+
+ assert result == {"status": "success", "root_uri": "root"}
+ call = service._execute_resource_ingestion.await_args.kwargs
+ assert call["path"] == msg.path
+ assert call["to"] == msg.root_uri
+ assert call["wait"] is True
+ assert call["resource_lock"] is resource_lock
+ assert call["parser_backend"] == "understanding"
+ service._should_use_connector.assert_not_called()
+
+
@pytest.mark.asyncio
async def test_shared_source_create_parent_false_routes_to_connector(
monkeypatch,
diff --git a/tests/service/test_resource_service_understanding_routing.py b/tests/service/test_resource_service_understanding_routing.py
new file mode 100644
index 000000000..7444e87a9
--- /dev/null
+++ b/tests/service/test_resource_service_understanding_routing.py
@@ -0,0 +1,109 @@
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import pytest
+
+from openviking.parse.accessors.base import LocalResource, SourceType
+from openviking.server.identity import RequestContext, Role
+from openviking.service import resource_service as resource_service_module
+from openviking.service.resource_service import ResourceService
+from openviking.storage.queuefs import QueueManager
+from openviking.storage.queuefs.add_resource_msg import AddResourceMsg
+from openviking_cli.session.user_id import UserIdentifier
+
+
+async def test_extensionless_remote_url_queues_frozen_understanding_route(
+ monkeypatch,
+ tmp_path,
+):
+ ctx = RequestContext(
+ user=UserIdentifier("acct", "alice"),
+ role=Role.USER,
+ api_key="secret",
+ )
+ downloaded = tmp_path / "download.pdf"
+ downloaded.write_bytes(b"%PDF-1.7")
+ prepared = LocalResource(
+ path=downloaded,
+ source_type=SourceType.HTTP,
+ original_source="https://example.com/download?id=1",
+ meta={
+ "resolved_extension": ".pdf",
+ "original_filename": "manual.pdf",
+ },
+ is_temporary=True,
+ )
+ lock = SimpleNamespace(
+ to_handoff=lambda: SimpleNamespace(to_dict=lambda: {"handle_id": "lock-1"}),
+ handoff=AsyncMock(),
+ close=AsyncMock(),
+ )
+ processor = SimpleNamespace(
+ understanding_api_enabled=lambda: True,
+ should_use_understanding_directly=lambda _source, **_kwargs: False,
+ prepare_resource=AsyncMock(return_value=prepared),
+ should_use_understanding_api=lambda resource: resource is prepared,
+ submit_understanding=AsyncMock(return_value="response-1"),
+ tree_builder=SimpleNamespace(
+ resolve_target_uri=AsyncMock(
+ return_value=(
+ "viking://resources/manual",
+ "viking://resources/manual",
+ )
+ )
+ ),
+ reserve_unique_candidate=AsyncMock(return_value=("viking://resources/manual", lock)),
+ process_resource=AsyncMock(),
+ )
+ service = ResourceService(
+ vikingdb=object(),
+ viking_fs=object(),
+ resource_processor=processor,
+ skill_processor=object(),
+ )
+ service._should_use_connector = lambda *_args, **_kwargs: False
+ tracker = SimpleNamespace(
+ create=AsyncMock(return_value=SimpleNamespace(task_id="task-1")),
+ update_stage=AsyncMock(),
+ fail=AsyncMock(),
+ )
+ queue_manager = SimpleNamespace(enqueue=AsyncMock())
+ monkeypatch.setattr(resource_service_module, "is_git_repo_url", lambda _path: False)
+ monkeypatch.setattr(
+ "openviking.service.task_tracker.get_task_tracker",
+ lambda: tracker,
+ )
+ monkeypatch.setattr(
+ "openviking.storage.queuefs.get_queue_manager",
+ lambda: queue_manager,
+ )
+ monkeypatch.setattr(
+ "openviking.storage.transaction.get_lock_manager",
+ lambda: object(),
+ )
+
+ result = await service.add_resource(
+ path="https://example.com/download?id=1",
+ ctx=ctx,
+ to="viking://resources/manual",
+ wait=False,
+ allow_local_path_resolution=False,
+ )
+
+ assert result == {
+ "status": "success",
+ "root_uri": "viking://resources/manual",
+ "task_id": "task-1",
+ }
+ _, message = queue_manager.enqueue.await_args.args
+ assert queue_manager.enqueue.await_args.args[0] == QueueManager.EXTERNAL_PARSE
+ queued = AddResourceMsg.from_dict(message)
+ assert queued.args["parser_backend"] == "understanding"
+ assert queued.args["resolved_extension"] == ".pdf"
+ assert queued.understanding_response_id == "response-1"
+ assert queued.source_name == "manual.pdf"
+ assert not downloaded.exists()
+ processor.submit_understanding.assert_awaited_once_with(prepared)
+ processor.process_resource.assert_not_awaited()
+ lock.handoff.assert_awaited_once()