Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
refactor(parse): 收口资源解析路由 (#3295)
Qin Haojie2be4bb48798f
diff --git a/docs/design/parser-two-layer-refactor-plan.md b/docs/design/parser-two-layer-refactor-plan.mdindex 0c9433156..25f01add8 100644--- a/docs/design/parser-two-layer-refactor-plan.md+++ b/docs/design/parser-two-layer-refactor-plan.md@@ -1,5 +1,7 @@# OpenViking 解析器两层架构重构+> 本文是 Accessor / Parser 两层拆分的历史重构记录。当前 `add_resource` 的完整入口分流、Understanding、Connector 与异步执行规则,见 [添加资源后的解析路由](./resource-ingestion-routing.md)。+| 项目 | 信息 ||-----|------|| 状态 | `已完成` |diff --git a/docs/design/resource-ingestion-routing.md b/docs/design/resource-ingestion-routing.mdnew file mode 100644index 000000000..d0ce5b3f6--- /dev/null+++ b/docs/design/resource-ingestion-routing.md@@ -0,0 +1,310 @@+# 添加资源后的解析路由++本文描述当前 `add_resource` 从收到资源到落盘、建索引的真实执行链。重点回答四个问题:入口在哪里分流、资源类型在哪里确定、Understanding 与 Connector 分别做什么,以及 `wait` 到底等待什么。++## 先记住五条规则++1. 对外只有一个资源添加入口:`ResourceService.add_resource`。SDK、HTTP API、MCP 最终都应调用它;Worker 不再拿后台任务冒充一次新的资源添加请求。+2. Connector 是一条独立的端到端导入链;Understanding 只是标准链里的一个 Parser 后端。+3. 普通文件只做一次 Parser 选择。选择依据是 Accessor 获取资源后冻结的 `resolved_extension`,不是临时文件名,也不会在队列消费者里重新猜;原始飞书 URL 是一个显式例外,可在 Accessor 前按配置直达 Understanding。+4. 目录、网站目录以及内部 Parser 遍历出的子文件不逐个调用 Understanding,统一留在内置 Parser 链内处理。+5. Watch 是一次新的来源刷新,会重新走提交分流,但不会创建、取消或覆盖自身的 Watch 任务。++## 总流程++```text+SDK / HTTP API / MCP+ |+ v+ResourceService.add_resource 对外入口+ |+ v+ResourceService._submit_resource_ingestion 阶段一:对新请求选择执行方式+ |+ +-- Connector 命中且参数受支持 ------> Connector doc/add+ | |+ | +--> 轮询 Connector task/info+ | +--> Connector 自己解析并写入资源+ |+ +-- Git && wait=false ----------------> 预检 + AddResource 队列 --> 返回 task_id+ |+ +-- HTTP 服务远程资源 && wait=false && Understanding 已启用+ | |+ | +-- 原始 URL 可直达 -----------> 提交 Understanding --> ExternalParse 队列+ | | |+ | | +--> 返回 task_id --> Worker+ | |+ | +-- 需要先确定本地类型 --------> Accessor 下载并识别+ | |+ | +-- 命中 Understanding --> 上传同一文件 --> ExternalParse 队列+ | | |+ | | +--> 返回 task_id --> Worker+ | |+ | +-- 未命中 ------------> 复用 LocalResource,进入当前请求标准链+ |+ +-- 其余场景(包括所有 wait=true) -----> _execute_resource_ingestion++_execute_resource_ingestion 阶段二:执行标准链+ |+ v+ResourceProcessor.process_resource+ |+ v+UnifiedResourceProcessor+ |+ +-- 已有 understanding_response_id ----> ParserRouter --> Understanding 恢复任务+ |+ +-- 原始 URL 可直达 Understanding -----> ParserRouter --> Understanding 提交并等待+ |+ +-- 原始文本 --------------------------> 内置 ParserRegistry+ |+ +-- 路径 / URL --> AccessorRegistry.access(选择数据访问器)+ |+ v+ LocalResource+ |+ +-- 目录 --> DirectoryParser(内置)+ |+ +-- 文件 --> ParserRouter.parse(选择 Parser,只选一次)+ |+ +-- 内置 ParserRegistry+ +-- Understanding 上传并等待+ |+ v+ ParseResult --> TreeBuilder 落盘+ |+ v+阶段三:返回策略+ |+ +-- wait=true --> 继续等待摘要 / 语义队列 / 向量索引后返回+ |+ +-- wait=false --> 普通标准链落盘后进入 AddResource 队列并返回++后台队列 Worker+ |+ +-- source job -----------------------> _execute_resource_ingestion(wait=true)+ | 使用消息中冻结的 backend / response_id / extension+ |+ +-- prepared job ---------------------> finish_prepared_resource+ |+ +-- 不再调用 add_resource,也不再重做 Connector / Git 顶层分类++WatchScheduler+ |+ +-- refresh_resource -----------------> _submit_resource_ingestion(manage_watch=false)+ 重新获取和解析来源,但不修改 Watch 任务+```++Understanding 不受 `wait=false` 限制。`wait=true` 在当前请求内完成 Understanding 提交、轮询、解析和 TreeBuilder,并继续等待后续队列;`wait=false` 的远程服务路径会先提交 Understanding、将 `response_id` 入队,再由 Worker 恢复任务。++分类发生的位置:++| 分类内容 | 代码位置 | 输出 |+|---|---|---|+| Connector、异步 Git、标准执行链 | `ResourceService._submit_resource_ingestion` | 选定新请求的顶层执行方式 |+| 异步 Understanding 或当前请求内执行 | `ResourceService._execute_resource_ingestion` | 根据 `wait`、配置和已识别类型决定提交还是同步执行 |+| Feishu、Git、Feed、HTTP、本地文件 | `AccessorRegistry.access` | `LocalResource` |+| Understanding 能否直接接收原始 URL | `ParserRouter.should_use_understanding_directly` | 直达 Understanding 或继续 Accessor |+| Understanding 与内置 Parser | `ParserRouter.parse` | `ParseResult` |++后台资源任务统一使用可恢复的 `AddResourceMsg`,但按工作类型进入两个独立队列:Understanding 使用受外部解析并发限制的 `ExternalParse`;Git 和已落盘的本地后处理使用 `AddResource`。两个队列都由 `AddResourceProcessor` 消费,锁接管失败时仍回到原队列。消息已经冻结了生产端做出的选择,例如 `parser_backend`、`understanding_response_id` 和 `resolved_extension`。Worker 直接执行该选择,不再把任务送回公开 `add_resource` 重新分类。++## 顶层路由表++| 输入或场景 | 获取数据 | 解析者 | 是否走标准 `ParseResult -> TreeBuilder` | 返回时机 |+|---|---|---|---|---|+| Connector 配置允许的 TOS 或 Git,提供精确 `to` 且参数受支持 | Connector 服务 | Connector 服务 | 否 | 提交成功立即返回 `task_id`,后台轮询状态 |+| `tos://` 但 Connector 不可用或参数不支持 | 不降级 | 不执行 | 否 | 直接报清晰的参数或配置错误 |+| Git 未命中 Connector 或无凭证回退,`wait=false` | GitAccessor 在后台 clone | 内置目录/代码仓库 Parser | 是 | 预检仓库并预占 URI 后返回 |+| Git,`wait=true` | GitAccessor | 内置目录/代码仓库 Parser | 是 | 解析、落盘及语义队列完成后返回 |+| 飞书 URL,`parser_api.enable_feishu_url=true` 且有 user/app 凭证 | Understanding 直接读取飞书 | Understanding | 是 | `wait=false` 时提交并入队;`wait=true` 时同步解析 |+| 飞书 URL,直达配置关闭或无可用凭证 | FeishuAccessor | 内置 Markdown Parser | 是 | Accessor 拉取、归一化后走标准链 |+| HTTP 服务请求,`wait=false`,命中 Understanding | HTTPAccessor 识别类型并上传同一份本地文件 | Understanding | 是 | 类型识别、Understanding 提交、URI 预占和入队后返回 |+| 其他 URL、文件、目录、原始文本 | 对应 Accessor;原始文本无需 Accessor | 内置 Parser 或同步 Understanding | 是 | 至少完成解析和落盘后返回 |++Git 是 Connector 与标准链共享的来源:未命中 Connector 或参数不受支持时可回退到标准链;一旦请求带有 Connector 专用凭证则禁止回退,避免凭证进入持久化队列。`tos://` 没有标准 Accessor,不能回退,否则只会在更深处得到误导性的解析错误。++## Accessor:先把“数据在哪”变成“本地是什么”++`AccessorRegistry` 按优先级选择数据访问器,当前内置顺序是:++```text+FeishuAccessor (100)+GitAccessor (80)+WebFeedAccessor (60)+HTTPAccessor (50)+LocalAccessor (1)+```++Accessor 的产物统一是 `LocalResource`,包含本地文件或目录路径、`source_type`、原始来源、是否需要清理,以及检测元数据。标准 Parser 不再负责 clone 或下载;只有明确开启的飞书 Understanding 直达链会在 Accessor 前消费原始 URL。++### 飞书资源++飞书是“远程私有数据源”,不是一种本地文件格式。默认使用 `FeishuAccessor` 获取并归一化:++```text+飞书 URL+ |+ v+FeishuAccessor --调用飞书 OpenAPI--> Markdown + 下载后的图片+ |+ v+LocalResource(document.md)+ |+ v+ParserRouter --> MarkdownParser --> ParseResult --> TreeBuilder+```++当 `parser_api.enable=true`、`parser_api.enable_feishu_url=true`,并且本次请求有 `args.feishu_access_token` 或服务端已配置飞书应用凭证时,原始飞书 URL 会在 Accessor 前进入统一的 Understanding 链。否则进入上面的 FeishuAccessor 路径。++`enable_feishu_url` 默认是 `false`。Understanding 一旦被选中,提交、轮询或解析失败会直接返回错误,不会在运行时自动重试 FeishuAccessor。++`FeishuAccessor` 预期支持范围如下:++| URL 类型 | Accessor 行为 | 当前边界 |+|---|---|---|+| `docx/{token}` | 拉取文档 block,转换标题、列表、代码、表格、图片和内嵌 sheet | 图片下载受 `download_images` 控制;内嵌 sheet 最多读取 100 行、A-Z 列 |+| `sheets/{token}` | 枚举所有 sheet;普通网格读取单元格,内嵌多维表格根据 `blockInfo` 转走 Bitable API,统一输出 Markdown 表格 | 普通网格最多读取 `max_rows_per_sheet` 行、A-Z 列;内嵌多维表格继承 Base 的记录和图片处理能力 |+| `base/{app_token}` | 无查询参数时枚举全部数据表;`table` 限定数据表,`table` + `view` 限定视图 | 表和字段完整翻页;每张表最多读取 `max_records_per_table` 条记录;附件字段中的图片会下载,其他附件保留文件名 |+| `wiki/{token}` | 先解析 wiki 节点的实际类型和 token,再进入上述 `docx`、`sheets` 或 `base` 处理器 | wiki 指向其他飞书对象类型时明确报不支持 |++四类入口都支持应用凭证获取的 tenant token;显式传入 `args.feishu_access_token` 时,同一 user token 会用于本次选中的飞书链路。Accessor 路径中,它用于 wiki 解析、正文、sheet/base 和图片请求;Understanding 直达路径中,它作为 `lark_file.user_access_token` 提交,随后从后台队列参数中删除。++飞书专有的本地归一化逻辑到 `FeishuAccessor` 为止,后面只处理 Markdown,因此不再保留并行的本地 `FeishuParser` 入口。即使 `parser_api.extensions` 包含 `md`,`SourceType.FEISHU` 的 `LocalResource` 也固定使用内置 Markdown Parser,不会把已经拉取的数据再次送 Understanding。传给 Understanding 的本地文件同样优先上传本地内容,`original_source` 只作为来源元数据,不会导致二次抓取原 URL。++Accessor 路径中的文档图片和 Base 附件字段图片使用同一条素材下载链路:先生成内部 `feishu://image/{file_token}` 引用;`download_images` 开启时,再通过飞书 Drive 素材接口下载到 `images/` 并改写 Markdown 相对路径。应用还必须具备 `docs:document.media:download` 权限;关闭下载、权限不足(例如 HTTP 403)或素材请求失败时,不会伪造本地图片文件,也不会把引用改成不存在的本地路径。Understanding 直达路径不调用这套素材接口,而是接收结果 ZIP 中已经包含的图片,并生成 artifact 图片映射供后续 URI 改写。++`UnifiedResourceProcessor.prepare` 随后冻结两个字段:++- `resolved_extension`:本次 Parser 路由唯一使用的扩展名。+- `resolved_name`:用于展示和默认资源命名,不参与 HTTP 资源的类型覆盖。++HTTP 资源以 HTTPAccessor 检出的 `meta.extension` 为准;本地文件或上传的临时文件可优先使用显式 `source_name` 的扩展名。这样既不会拿随机临时文件名选 Parser,也不会让用户提供的 URL 名称覆盖实际下载内容类型。++## 无后缀 URL 怎么判断类型++HTTPAccessor 按以下顺序收集和修正类型:++1. URL path 中受支持的显式扩展名。+2. HEAD 响应的 `Content-Disposition` 文件名。+3. HEAD 响应的 `Content-Type`。+4. GET 响应的 `Content-Disposition` 和 `Content-Type`,用于修正之前的模糊网页判断。+5. GET 内容的 magic bytes,例如 PDF、图片、音视频、Office/EPUB/ZIP 签名。+6. 仍无法识别时按网页处理。++最终扩展名写入 `LocalResource.meta.extension`,然后冻结为 `resolved_extension`。`ParserRouter` 只读取这个结果。URL 上已有明确扩展名时,不用 magic bytes 擅自覆盖它。++因此,`wait=false` 不等于“完全不碰源站就返回”。HTTP 服务收到无后缀远程 URL、且 Understanding 已启用时,必须先下载或探测一次,才能知道应进入外部解析队列还是内置 Parser。命中 Understanding 后,生产端直接上传这份已检测的本地文件并取得 `response_id`,然后才清理临时文件;Worker 只恢复该 response,不会重新下载可能已过期或内容已变化的 URL。++## Parser:只回答“本地内容怎么解析”++标准执行链分四类:++- 原始文本:直接进入内置 `ParserRegistry`。+- 可由 Understanding 直接接收的原始 URL:在 Accessor 前进入 `ParserRouter`。+- 本地目录:直接进入 `DirectoryParser`;Git 仓库由目录链委派给代码仓库 Parser。+- 本地文件:进入 `ParserRouter`,根据 `resolved_extension` 在内置 `ParserRegistry` 与 Understanding 之间选一次。++ParserRegistry 只注册项目内置 Parser,不再提供自定义 Parser 类、回调注册或可选模块注册入口。新增数据源优先实现 Accessor;新增文件格式则直接增加内置 Parser 和对应测试。++目录、网站和压缩包的“子文件遍历”属于当前内置 Parser 的内部实现,不回到顶层 `ResourceService`,也不会给每个子文件重新选择 Understanding。顶层压缩文件本身是否进入 Understanding,仍由其冻结扩展名和 `parser_api.extensions` 决定。++## Understanding 链路++Understanding 是“外部解析器”,不是“外部落盘器”:++```text+LocalResource / 远程源+ |+ v+Understanding API+ |+ v+解析结果 ZIP+ |+ v+解压到临时 Viking 目录+ |+ v+ParseResult+ |+ v+TreeBuilder + 标准摘要/索引链+```++同步路径中,Accessor 已下载的本地文件会直接上传给 Understanding,`original_source` 只保留作来源元数据。普通 HTTP 文件的异步路径也先上传已检测文件,再通过统一的 `AddResourceMsg` 持久化 `understanding_response_id`、冻结的 `resolved_extension` 和 `parser_backend="understanding"`;Worker 直接恢复 response,既不重新下载源 URL,也不因配置变化重新选择后端。这些冻结字段是内部任务字段,公共 `args` 不能指定,避免调用方绕过外部解析开关和扩展名白名单。++飞书直达的异步路径也冻结 `parser_backend="understanding"`。生产端统一调用 Understanding 提交:显式 user token 直接转换为 `lark_file`,应用凭证则在 UnderstandingAPI 内获取 tenant token。QueueFS 只持久化 `understanding_response_id`,不保存 token;Worker 从该 response 继续轮询,不重复提交也不重新判断配置。未显式指定资源名的飞书 artifact 会从 ZIP 单一根目录恢复真实标题,目标 URI 因而延迟到解析完成后确定。++Understanding 返回 ZIP 时,本地适配器会安全解压,并根据 Markdown 的相对图片引用生成受控的图片映射 sidecar。TreeBuilder 后续仍使用统一的图片 URI 改写链,不依赖飞书 Accessor 的 `feishu://` 占位符。++这条链的关键特征是:Understanding 只替代 Parser,后面的 `ParseResult`、URI 规划、TreeBuilder 落盘、摘要和索引仍属于 OpenViking。++## Connector 链路++Connector 是另一套端到端导入服务:++```text+ResourceService+ |+ +--> Connector doc/add+ |+ +--> Connector 获取源数据+ +--> Connector 解析+ +--> Connector 写入目标资源树+ |+ +--> 后台轮询 Connector task/info+ |+ +--> 更新 OpenViking TaskRecord+```++Connector 不返回本地 `ParseResult`,也不调用当前进程的 `TreeBuilder`。OpenViking 只负责校验这次请求能否无损委派、提交任务、返回 OpenViking `task_id`,再把 Connector 的终态同步到任务记录。++Connector 当前要求提供精确 `to`,不接受 `parent`;也不支持 `wait=true`、watch、instruction、关闭建索引、摘要、strict、include/exclude 等。无凭证的 Git 请求可回退到标准链;带 Connector 专用凭证的 Git 和 Connector-only 来源会立即报错,避免凭证落入本地持久化任务。++## `wait` 的准确含义++| 路径 | `wait=false` | `wait=true` |+|---|---|---|+| Connector | 提交外部任务后返回 | 不支持;不会假装同步等待 |+| Git | 预检并预占 URI 后启动后台标准链 | 当前请求内完成标准链并等待队列 |+| 飞书直达 Understanding | 生产端提交后只将 response ID 放入 `ExternalParse`;无显式资源名时可延迟返回 `root_uri` | 当前请求内调用 Understanding,再等待后续队列 |+| HTTP 服务 + 异步 Understanding | 先识别类型,再预占 URI、入 `ExternalParse` 后返回 | 当前请求内调用 Understanding,再等待后续队列 |+| 普通标准链 | 解析和落盘完成后返回;摘要/语义处理由任务监控 | 解析和落盘完成后继续等待语义队列 |++所以普通 `wait=false` 不是“所有工作后台化”,而是“资源树已落盘,但不阻塞等待后续语义任务”。Git 与异步 Understanding 是两个明确的例外分支。++## 目标 URI、锁和失败边界++- 需要后台执行的 Git 与普通文件 Understanding 任务会先规划并预占目标 URI,避免返回的 URI 随后台竞态变化。未指定名称的飞书直达任务会延迟目标 URI 解析,以 artifact 的真实根标题作为最终名称,并在完成时回写 TaskRecord。+- 锁通过 handoff 交给后台任务或队列 Worker;入队失败时立即释放,并把任务标为失败。+- 临时 `LocalResource` 由拥有它的调用层清理;交给标准处理器后,清理责任随之转移。+- Parser 产生 `ParseResult` 后才进入 TreeBuilder。没有临时解析产物时标准链返回解析错误;目录允许带 warnings 的部分成功,`strict` 决定是否暴露这些警告。+- Connector 的失败边界在外部任务终态,OpenViking 不对其内部文件逐个回滚。++## 代码定位++| 职责 | 入口 |+|---|---|+| 公开入口、新请求分流与内部执行 | `openviking/service/resource_service.py`:`ResourceService.add_resource`、`_submit_resource_ingestion`、`_execute_resource_ingestion` |+| Watch 刷新入口 | `openviking/service/resource_service.py`:`ResourceService.refresh_resource`;`openviking/resource/watch_scheduler.py` |+| 标准解析与落盘编排 | `openviking/utils/resource_processor.py`:`ResourceProcessor.process_resource` |+| Accessor 与 Parser 两层衔接 | `openviking/utils/media_processor.py`:`UnifiedResourceProcessor` |+| 文件 Parser 单次选择 | `openviking/parse/parser_router.py`:`ParserRouter` |+| 内置 Parser 注册 | `openviking/parse/registry.py`:`ParserRegistry` |+| HTTP 类型识别 | `openviking/parse/accessors/http_accessor.py`:`HTTPAccessor`、`URLTypeDetector` |+| Understanding 同步适配 | `openviking/parse/understanding_api.py`:`UnderstandingAPI` |+| 飞书直达开关 | `openviking_cli/utils/config/open_viking_config.py`:`ParserApiConfig.enable_feishu_url` |+| 可恢复的后台资源消息与双队列 Worker | `openviking/storage/queuefs/add_resource_msg.py`、`add_resource_processor.py`、`queue_manager.py` |+| Connector 客户端 | `openviking/connector/client.py`:`ConnectorClient` |++## 快速自检++- 无后缀 PDF URL:HTTPAccessor 从响应头或 PDF 签名得到 `.pdf`,ParserRouter 再决定是否使用 Understanding。+- 飞书默认配置:走 FeishuAccessor;仅开启 `enable_feishu_url` 且有凭证时才绕过 Accessor。+- 飞书 user token 直达:队列中只有 response ID 和强制后端标记,不出现 token 明文。+- 已归一化飞书 Markdown:无论 `extensions` 是否包含 `md`,都走内置 Markdown Parser。+- Understanding 返回结果:先转成 `ParseResult`,仍由本地 TreeBuilder 落盘。+- Connector 返回结果:只返回任务标识,不经过本地 `ParseResult`。+- 普通 Markdown 且 `wait=false`:返回前 Markdown 已解析并落盘,只是不等待后续语义队列。+- 网站抓取出的目录:进入 DirectoryParser,页面子文件不会逐个调用 Understanding。diff --git a/openviking/eval/datasets/local_doc_example_glm5.jsonl b/openviking/eval/datasets/local_doc_example_glm5.jsonlindex ec42d4834..3e1eb897b 100644--- a/openviking/eval/datasets/local_doc_example_glm5.jsonl+++ b/openviking/eval/datasets/local_doc_example_glm5.jsonl@@ -16,5 +16,4 @@{"question": "语义处理队列的工作机制是什么?如何监控和调试异步处理状态?", "files": ["OpenViking/docs/en/concepts/01-architecture.md", "OpenViking/openviking/service/resource_service.py:1-80"], "answer": "语义处理队列通过 SemanticQueue 异步生成 L0/L1 内容。\n\n**工作机制**:\n\n1. **处理流水线**:\n```\nInput → Parser → TreeBuilder → AGFS → SemanticQueue → Vector Index\n```\n\n2. **队列处理**:\n - TreeBuilder 将文件移动到 AGFS 后入队\n - SemanticQueue 异步处理队列项\n - 自底向上生成 L0/L1\n\n3. **异步处理优势**:\n - 不阻塞资源添加操作\n - 支持大规模文档处理\n - LLM 调用并行化\n\n**监控和调试**:\n\n1. **等待处理完成**:\n```python\nresult = client.add_resource(\"./docs\", wait=False)\nclient.wait_processed() # 等待队列处理完成\n```\n\n2. **超时控制**:\n```python\nresult = client.add_resource(\n \"./docs\",\n wait=True,\n timeout=300 # 超时时间(秒)\n)\n```\n\n3. **DebugService**:\n - 提供 ObserverService 用于调试\n - 可视化检索轨迹\n\n4. **日志监控**:\n - 通过 logger 获取处理状态\n - 监控队列长度和处理延迟\n\n**Service 层支持**:\n- ResourceService 管理 add_resource 操作\n- wait_processed() 方法等待语义处理完成"}{"question": "如何通过代码动态创建和管理 Session?Session 与 Memory 的关系是什么?", "files": ["OpenViking/docs/en/concepts/08-session.md", "OpenViking/openviking/session/compressor_v2.py"], "answer": "Session 通过 client API 动态创建,commit 后自动提取 Memory。\n\n**Session 创建和管理**:\n\n1. **创建 Session**:\n```python\n# 创建新会话\nsession = client.session(session_id=\"chat_001\")\n\n# 或使用 create_session\nresult = client.create_session()\n```\n\n2. **添加消息**:\n```python\nsession.add_message(\n \"user\",\n [TextPart(\"How to configure embedding?\")]\n)\n\nsession.add_message(\n \"assistant\",\n [\n TextPart(\"Here's how...\"),\n ContextPart(uri=\"viking://user/memories/profile.md\"),\n ]\n)\n```\n\n3. **记录使用**:\n```python\nsession.used(contexts=[\"viking://user/memories/profile.md\"])\nsession.used(skill={\n \"uri\": \"viking://user/skills/code-search\",\n \"input\": \"search config\",\n \"output\": \"found 3 files\",\n \"success\": True\n})\n```\n\n4. **提交会话**:\n```python\nresult = session.commit()\n# {\"status\": \"committed\", \"memories_extracted\": 5}\n```\n\n**Session 与 Memory 的关系**:\n\n1. **Memory 来源**:\n - Memory 从 Session 消息中提取\n - SessionCompressorV2 在 commit 时触发\n\n2. **提取流程**:\n - 格式化会话消息\n - 根据当前记忆策略调用 LLM 提取已启用的记忆类型\n - 写入当前用户或相关 Peer 的记忆目录并索引\n\n3. **内置记忆类型与默认位置**:\n - profile → `user/memories/profile.md`\n - preferences → `user/memories/preferences/`\n - entities → `user/memories/entities/`\n - events → `user/memories/events/`\n - identity → `user/memories/identity.md`\n - soul → `user/memories/soul.md`\n - cases → `user/memories/cases/`\n - trajectories → `user/memories/trajectories/`\n - experiences → `user/memories/experiences/`\n - tools → `user/memories/tools/`\n - skills → `user/memories/skills/`\n\n这些 `user/...` 短路径会按当前身份解析为 `viking://user/{user_id}/...`,应用也可以通过自定义模板扩展或调整记忆类型。"}{"question": "OpenViking Server 模式与本地模式的区别是什么?如何选择部署方式?", "files": ["OpenViking/openviking/server/bootstrap.py:1-61", "OpenViking/docs/en/concepts/01-architecture.md"], "answer": "Server 模式提供 HTTP API,本地模式直接嵌入应用。\n\n**模式对比**:\n\n| 维度 | 本地模式 | Server 模式 |\n|------|----------|-------------|\n| 访问方式 | Python SDK | HTTP API |\n| 部署复杂度 | 简单 | 需启动服务 |\n| 多客户端 | 不支持 | 支持 |\n| 资源共享 | 单进程 | 多进程共享 |\n| 适用场景 | 单机应用 | 生产环境 |\n\n**Server 模式启动**:\n\n1. **命令行启动**:\n```bash\nopenviking-server --host 0.0.0.0 --port 1933 --config ./ov.conf\n```\n\n2. **参数说明**:\n- `--host`:绑定地址\n- `--port`:端口号\n- `--config`:配置文件路径\n\n3. **HTTP API 示例**:\n```bash\ncurl -X POST http://localhost:1933/api/v1/resources \\\n -H \"Content-Type: application/json\" \\\n -H \"X-API-Key: your-key\" \\\n -d '{\"path\": \"./documents/guide.md\"}'\n```\n\n**部署方式选择**:\n\n1. **选择本地模式**:\n - 单机应用开发\n - 快速原型验证\n - 无需多客户端访问\n\n2. **选择 Server 模式**:\n - 生产环境部署\n - 多客户端/多服务访问\n - 需要集中式上下文管理\n - 微服务架构集成\n\n**Service 层复用**:\n两种模式共享 Service 层逻辑,支持 HTTP Server 和 CLI 复用。"}-{"question": "如何扩展自定义的 Parser?例如支持新的文档格式或代码语言?", "files": ["OpenViking/openviking/parse/registry.py", "OpenViking/openviking/parse/parsers/base_parser.py:1-80"], "answer": "通过 ParserRegistry 注册自定义 Parser,支持 Protocol 接口。\n\n**自定义 Parser 步骤**:\n\n1. **实现 BaseParser 接口**:\n```python\nclass BaseParser(ABC):\n @abstractmethod\n async def parse(self, source: Union[str, Path], instruction: str = \"\", **kwargs) -> ParseResult:\n \"\"\"解析文档\"\"\"\n pass\n \n @abstractmethod\n async def parse_content(self, content: str, source_path: Optional[str] = None, **kwargs) -> ParseResult:\n \"\"\"解析内容\"\"\"\n pass\n \n @property\n @abstractmethod\n def supported_extensions(self) -> List[str]:\n \"\"\"支持的文件扩展名\"\"\"\n pass\n```\n\n2. **注册 Parser**:\n```python\nfrom openviking.parse.registry import get_registry\n\nclass MyParser(BaseParser):\n @property\n def supported_extensions(self) -> List[str]:\n return [\".myext\"]\n \n async def parse(self, source, instruction=\"\", **kwargs):\n # 实现解析逻辑\n return ParseResult(...)\n\n# 注册\nregistry = get_registry()\nregistry.register(\"my_parser\", MyParser())\n```\n\n3. **使用 Protocol 接口**:\n```python\nregistry.register_custom(\n handler=my_parser, # 实现 CustomParserProtocol\n extensions=[\".custom\"],\n name=\"custom_parser\"\n)\n```\n\n**内置 Parser**:\n- TextParser:纯文本\n- MarkdownParser:Markdown\n- PDFParser:PDF\n- HTMLParser:HTML\n- CodeRepositoryParser:代码仓库\n- ImageParser:图片(可选)\n\n**扩展点**:\n- `supported_extensions`:定义支持的扩展名\n- `can_parse()`:检查文件是否可解析\n- `_read_file()`:读取文件内容"}{"question": "eval 模块的 RAGAS 集成是如何实现的?如何自定义评测指标?", "files": ["OpenViking/openviking/eval/ragas.py", "OpenViking/openviking/eval/base.py:1-60"], "answer": "eval 模块通过 RagasEvaluator 适配器集成 RAGAS,支持自定义评测指标。\n\n**RAGAS 集成实现**:\n\n1. **RagasEvaluator 类**:\n```python\nclass RagasEvaluator(BaseEvaluator):\n def __init__(self, metrics=None, llm=None, embeddings=None):\n from ragas.metrics import (\n answer_relevance,\n context_precision,\n context_recall,\n faithfulness,\n )\n self.metrics = metrics or [\n faithfulness,\n answer_relevance,\n context_precision,\n context_recall,\n ]\n```\n\n2. **评测流程**:\n```python\nasync def evaluate_dataset(self, dataset: EvalDataset) -> SummaryResult:\n # 转换为 RAGAS 数据集格式\n data = {\n \"question\": [s.query for s in dataset.samples],\n \"contexts\": [s.context for s in dataset.samples],\n \"answer\": [s.response for s in dataset.samples],\n \"ground_truth\": [s.ground_truth for s in dataset.samples],\n }\n \n # 调用 RAGAS evaluate\n result = evaluate(data, metrics=self.metrics)\n```\n\n**自定义评测指标**:\n\n1. **继承 BaseEvaluator**:\n```python\nclass MyEvaluator(BaseEvaluator):\n async def evaluate_sample(self, sample: EvalSample) -> EvalResult:\n # 实现自定义评测逻辑\n scores = {\"my_metric\": 0.85}\n return EvalResult(sample=sample, scores=scores)\n```\n\n2. **自定义 RAGAS 指标**:\n```python\nfrom ragas import evaluate\nfrom ragas.metrics import Metric\n\nclass MyMetric(Metric):\n name = \"my_metric\"\n # 实现指标计算\n\nevaluator = RagasEvaluator(metrics=[MyMetric()])\n```\n\n3. **评测结果结构**:\n```python\nclass SummaryResult(BaseModel):\n dataset_name: str\n sample_count: int\n mean_scores: Dict[str, float] # 平均分数\n results: List[EvalResult] # 每个样本结果\n```"}diff --git a/openviking/parse/__init__.py b/openviking/parse/__init__.pyindex 380f65288..d4b4f0370 100644--- a/openviking/parse/__init__.py+++ b/openviking/parse/__init__.py@@ -4,8 +4,6 @@from openviking.core.building_tree import BuildingTreefrom openviking.parse.base import NodeType, ParseResult, ResourceNode, create_parse_result-from openviking.parse.converter import DocumentConverter-from openviking.parse.custom import CallbackParserWrapper, CustomParserProtocol, CustomParserWrapperfrom openviking.parse.directory_scan import (CLASS_PROCESSABLE,CLASS_UNSUPPORTED,@@ -18,7 +16,6 @@ from openviking.parse.parsers.html import HTMLParserfrom openviking.parse.parsers.markdown import MarkdownParserfrom openviking.parse.parsers.pdf import PDFParserfrom openviking.parse.parsers.text import TextParser-from openviking.parse.registry import ParserRegistry, get_registry, parsefrom openviking.parse.tree_builder import TreeBuilderfrom openviking.parse.vlm import VLMProcessor@@ -34,15 +31,6 @@ __all__ = ["MarkdownParser","PDFParser","HTMLParser",- "DocumentConverter",- # Custom parser support- "CustomParserProtocol",- "CustomParserWrapper",- "CallbackParserWrapper",- # Registry- "ParserRegistry",- "get_registry",- "parse",# Tree builder"TreeBuilder","BuildingTree",diff --git a/openviking/parse/accessors/feishu_accessor.py b/openviking/parse/accessors/feishu_accessor.pyindex d8951a10b..df8a30db9 100644--- a/openviking/parse/accessors/feishu_accessor.py+++ b/openviking/parse/accessors/feishu_accessor.py@@ -4,21 +4,23 @@Feishu/Lark Accessor.Fetches Feishu/Lark cloud documents using the lark-oapi SDK.-This is the DataAccessor layer extracted from FeishuParser.Note: This accessor requires the `lark-oapi` package.Included by default in `openviking[bot]` installation."""import asyncio+import json+import mimetypesimport osimport reimport tempfilefrom dataclasses import dataclassfrom pathlib import Path-from typing import Any, Dict, NoReturn, Optional, Tuple, Union-from urllib.parse import urlparse+from typing import Any, Dict, List, NoReturn, Optional, Tuple, Union+from urllib.parse import parse_qs, urlparse+from openviking.parse.base import format_table_to_markdownfrom openviking.utils.exceptions import error_code_from_http_statusfrom openviking_cli.exceptions import OpenVikingErrorfrom openviking_cli.utils.logger import get_logger@@ -83,8 +85,7 @@ def _raise_from_lark_response(if code == _FEISHU_BITABLE_PERMISSION_REQUIRED:public_code = "FAILED_PRECONDITION"message = (- "Feishu application is missing required Bitable permissions: "- f"code={code}, msg={msg}"+ f"Feishu application is missing required Bitable permissions: code={code}, msg={msg}")else:public_code = (@@ -101,7 +102,7 @@ def _raise_from_lark_response(class FeishuDocument:"""Result from fetching a Feishu document."""- doc_type: str # "docx" only for now+ doc_type: strtoken: strmarkdown_content: strtitle: str@@ -114,7 +115,9 @@ class FeishuAccessor(DataAccessor):Supports:- Documents: https://*.feishu.cn/docx/{document_id}- - Wiki pages: https://*.feishu.cn/wiki/{token} (resolves to docx)+ - Wiki pages: https://*.feishu.cn/wiki/{token}+ - Spreadsheets: https://*.feishu.cn/sheets/{token}+ - Bitable: https://*.feishu.cn/base/{app_token}Requires:- lark-oapi package@@ -127,6 +130,11 @@ class FeishuAccessor(DataAccessor):# Wiki obj_type normalization (API returns short names)_WIKI_TYPE_MAP = {"doc": "docx", "sheet": "sheets", "bitable": "base"}+ _DOC_TYPE_HANDLERS = {+ "docx": "_parse_docx",+ "sheets": "_parse_sheets",+ "base": "_parse_bitable",+ }# Attributes that skip processing (structural containers or metadata)_SKIP_ATTRS = {"page", "table_cell", "quote_container", "grid", "grid_column"}@@ -136,6 +144,7 @@ class FeishuAccessor(DataAccessor):"divider": "_handle_divider","image": "_handle_image","table": "_table_block_to_markdown",+ "sheet": "_embedded_sheet_to_markdown",}# Attribute → markdown prefix template for text-bearing blocks.@@ -169,6 +178,7 @@ class FeishuAccessor(DataAccessor):19: "callout",22: "divider",27: "image",+ 30: "sheet",31: "table",32: "table_cell",34: "quote_container",@@ -196,6 +206,7 @@ class FeishuAccessor(DataAccessor):"callout","divider","image",+ "sheet","table","table_cell","quote_container",@@ -312,9 +323,12 @@ class FeishuAccessor(DataAccessor):"""Fetch a Feishu document and convert to Markdown.- This method extracts and adapts the logic from FeishuParser.parse().+ The fetched document is materialized as Markdown for the standard parser chain."""doc_type, token = self._parse_feishu_url(url)+ query = parse_qs(urlparse(url).query)+ table_id = (query.get("table") or [None])[0]+ view_id = (query.get("view") or [None])[0]title = Nonemeta = {}@@ -328,24 +342,37 @@ class FeishuAccessor(DataAccessor):doc_type, token = real_type, real_tokenmeta["wiki_resolved"] = True- # Only docx is supported- if doc_type != "docx":+ if doc_type != "base":+ table_id = view_id = None++ handler_name = self._DOC_TYPE_HANDLERS.get(doc_type)+ if handler_name is None:raise ValueError(f"Unsupported Feishu document type: {doc_type}. "- f"Only docx is supported in this version."+ f"Supported: {list(self._DOC_TYPE_HANDLERS)}")- # Call the handler (in thread pool since lark-oapi is sync)+ handler_kwargs = {}+ if doc_type == "base":+ handler_kwargs = {"table_id": table_id, "view_id": view_id}++ # Feishu's SDK is synchronous; keep it off the event loop.markdown, doc_title = await asyncio.to_thread(- self._parse_docx,+ getattr(self, handler_name),token,feishu_access_token,+ **handler_kwargs,)if title:- doc_title = title+ scope = "/".join(value for value in (table_id, view_id) if value)+ doc_title = f"{title} ({scope})" if scope else titlemeta["original_url"] = url+ if table_id:+ meta["feishu_table_id"] = table_id+ if view_id:+ meta["feishu_view_id"] = view_idreturn FeishuDocument(doc_type=doc_type,@@ -382,7 +409,7 @@ class FeishuAccessor(DataAccessor):path_parts = [p for p in parsed.path.split("/") if p]if len(path_parts) < 2:raise ValueError(f"Cannot parse Feishu URL: {url}")- doc_type = path_parts[0] # docx, wiki+ doc_type = path_parts[0] # docx, wiki, sheets, basetoken = path_parts[1]return doc_type, token@@ -434,6 +461,10 @@ class FeishuAccessor(DataAccessor):return RequestOption.builder().user_access_token(feishu_access_token).build()+ def _call_api(self, method, request, feishu_access_token: Optional[str] = None):+ option = self._user_request_option(feishu_access_token)+ return method(request) if option is None else method(request, option)+# ========== Wiki Resolution ==========def _resolve_wiki_node(@@ -451,11 +482,11 @@ class FeishuAccessor(DataAccessor):client = self._get_client(use_user_token=bool(feishu_access_token))request = GetNodeSpaceRequest.builder().token(token).build()- option = self._user_request_option(feishu_access_token)- if option is None:- response = client.wiki.v2.space.get_node(request)- else:- response = client.wiki.v2.space.get_node(request, option)+ response = self._call_api(+ client.wiki.v2.space.get_node,+ request,+ feishu_access_token,+ )if not response.success():_raise_from_lark_response(response,@@ -512,7 +543,11 @@ class FeishuAccessor(DataAccessor):continue # Skip page containerline = self._block_to_markdown(- block, block_map, ordered_counter, document_id=document_id+ block,+ block_map,+ ordered_counter,+ document_id=document_id,+ feishu_access_token=feishu_access_token,)if line is not None:markdown_lines.append(line)@@ -548,11 +583,11 @@ class FeishuAccessor(DataAccessor):builder = builder.page_token(page_token)request = builder.build()- option = self._user_request_option(feishu_access_token)- if option is None:- response = client.docx.v1.document_block.list(request)- else:- response = client.docx.v1.document_block.list(request, option)+ response = self._call_api(+ client.docx.v1.document_block.list,+ request,+ feishu_access_token,+ )if not response.success():_raise_from_lark_response(@@ -592,7 +627,12 @@ class FeishuAccessor(DataAccessor):return Nonedef _block_to_markdown(- self, block, block_map: Dict, ordered_counter: Dict[str, int], document_id: str = ""+ self,+ block,+ block_map: Dict,+ ordered_counter: Dict[str, int],+ document_id: str = "",+ feishu_access_token: Optional[str] = None,) -> Optional[str]:"""Convert a single SDK block object to markdown string.@@ -618,7 +658,12 @@ class FeishuAccessor(DataAccessor):# Special blocks (non-text: divider, image, table)special_handler = self._SPECIAL_BLOCK_HANDLERS.get(attr)if special_handler:- return getattr(self, special_handler)(block, block_map, document_id=document_id)+ return getattr(self, special_handler)(+ block,+ block_map,+ document_id=document_id,+ feishu_access_token=feishu_access_token,+ )# --- Text-bearing blocks: extract elements, apply formatting ---content_obj = getattr(block, attr, None)@@ -747,12 +792,8 @@ class FeishuAccessor(DataAccessor):.token_types({token_type}).build())- option = self._user_request_option(feishu_access_token)-try:- raw_resp = (- client.request(raw_req) if option is None else client.request(raw_req, option)- )+ raw_resp = self._call_api(client.request, raw_req, feishu_access_token)except Exception as exc:logger.warning("[FeishuAccessor] Error downloading image %s: %s", file_token, exc)return None@@ -936,8 +977,6 @@ class FeishuAccessor(DataAccessor):row.append("")rows.append(row)- from openviking.parse.base import format_table_to_markdown-return format_table_to_markdown(rows, has_header=True) if rows else Nonedef _extract_cell_text(self, cell_block, block_map: Dict) -> str:@@ -956,3 +995,374 @@ class FeishuAccessor(DataAccessor):if text:texts.append(text)return " ".join(texts)++ def _embedded_sheet_to_markdown(+ self,+ block,+ block_map: Dict = None,+ *,+ document_id: str = "",+ feishu_access_token: Optional[str] = None,+ **_,+ ) -> Optional[str]:+ """Convert an embedded spreadsheet block in a docx document."""+ import lark_oapi as lark++ client = self._get_client(use_user_token=bool(feishu_access_token))+ token_type = (+ lark.AccessTokenType.USER if feishu_access_token else lark.AccessTokenType.TENANT+ )+ request = (+ lark.BaseRequest.builder()+ .http_method(lark.HttpMethod.GET)+ .uri(+ f"/open-apis/docx/v1/documents/{document_id or block.parent_id}"+ f"/blocks/{block.block_id}"+ )+ .token_types({token_type})+ .build()+ )+ response = self._call_api(client.request, request, feishu_access_token)+ if not response.success():+ logger.warning(+ "[FeishuAccessor] Failed to inspect embedded sheet %s: code=%s msg=%s",+ block.block_id,+ getattr(response, "code", None),+ getattr(response, "msg", None),+ )+ return None++ data = json.loads(response.raw.content)+ sheet_token = data.get("data", {}).get("block", {}).get("sheet", {}).get("token", "")+ parts = sheet_token.rsplit("_", 1)+ if len(parts) != 2:+ return None++ spreadsheet_token, sheet_id = parts+ try:+ rows = self._read_sheet_range(+ spreadsheet_token,+ sheet_id,+ max_rows=100,+ max_cols=26,+ feishu_access_token=feishu_access_token,+ )+ except Exception as exc:+ logger.warning(+ "[FeishuAccessor] Failed to read embedded sheet %s: %s",+ sheet_token,+ exc,+ )+ return None++ rows = self._trim_empty_columns(rows)+ return format_table_to_markdown(rows, has_header=True) if rows else None++ @staticmethod+ def _trim_empty_columns(rows: List[List[str]]) -> List[List[str]]:+ """Remove trailing columns that are empty in every row."""+ if not rows:+ return rows+ last_col = 0+ for col in range(max(len(row) for row in rows)):+ if any(col < len(row) and row[col].strip() for row in rows):+ last_col = col + 1+ return [row[:last_col] for row in rows] if last_col else []++ def _parse_sheets(+ self,+ token: str,+ feishu_access_token: Optional[str] = None,+ ) -> Tuple[str, str]:+ """Fetch a Feishu spreadsheet and convert it to Markdown."""+ import lark_oapi as lark++ client = self._get_client(use_user_token=bool(feishu_access_token))+ config = self._get_config()+ token_type = (+ lark.AccessTokenType.USER if feishu_access_token else lark.AccessTokenType.TENANT+ )+ metadata_request = (+ lark.BaseRequest.builder()+ .http_method(lark.HttpMethod.GET)+ .uri(f"/open-apis/sheets/v2/spreadsheets/{token}/metainfo")+ .token_types({token_type})+ .build()+ )+ metadata_response = self._call_api(+ client.request,+ metadata_request,+ feishu_access_token,+ )+ if not metadata_response.success():+ _raise_from_lark_response(+ metadata_response,+ operation=f"fetch spreadsheet metadata for {token}",+ resource=token,+ )+ metadata = json.loads(metadata_response.raw.content).get("data", {})+ title = (metadata.get("properties") or {}).get("title") or "Spreadsheet"+ sheets = metadata.get("sheets") or []+ markdown_parts = [f"# {title}", f"**Sheets:** {len(sheets)}"]+ for sheet in sheets:+ sheet_id = sheet.get("sheetId") or ""+ sheet_title = sheet.get("title") or sheet_id+ parts = [f"## Sheet: {sheet_title}"]++ block_info = sheet.get("blockInfo")+ if block_info:+ block_type = block_info.get("blockType") or "unknown"+ if block_type != "BITABLE_BLOCK":+ parts.append(f"*Unsupported sheet block: {block_type}*")+ else:+ block_token = block_info.get("blockToken") or ""+ tokens = block_token.rsplit("_", 1)+ if len(tokens) != 2 or not all(tokens):+ parts.append("*Invalid embedded bitable token*")+ else:+ bitable, _ = self._parse_bitable(+ tokens[0],+ feishu_access_token,+ table_id=tokens[1],+ table_name=sheet_title,+ )+ parts.append(bitable or "*Empty bitable*")+ markdown_parts.append("\n\n".join(parts))+ continue++ row_count = int(sheet.get("rowCount") or 0)+ col_count = int(sheet.get("columnCount") or 0)+ if not row_count or not col_count:+ parts.append("*Empty sheet*")+ markdown_parts.append("\n\n".join(parts))+ continue++ parts.append(f"**Dimensions:** {row_count} rows x {col_count} columns")+ rows_to_read = min(row_count, config.max_rows_per_sheet)+ rows = self._read_sheet_range(+ token,+ sheet_id,+ rows_to_read,+ col_count,+ feishu_access_token=feishu_access_token,+ )+ if rows:+ parts.append(format_table_to_markdown(rows, has_header=True))+ if row_count > config.max_rows_per_sheet:+ parts.append(+ f"\n*... {row_count - config.max_rows_per_sheet} more rows truncated ...*"+ )+ if col_count > 26:+ parts.append(f"\n*... {col_count - 26} columns after Z omitted ...*")+ markdown_parts.append("\n\n".join(parts))++ return "\n\n".join(markdown_parts), title++ def _read_sheet_range(+ self,+ token: str,+ sheet_id: str,+ max_rows: int,+ max_cols: int,+ feishu_access_token: Optional[str] = None,+ ) -> List[List[str]]:+ """Read a bounded cell range from a Feishu spreadsheet."""+ import lark_oapi as lark++ client = self._get_client(use_user_token=bool(feishu_access_token))+ # ponytail: the existing importer reads A:Z only; add chunked ranges if wider+ # spreadsheet imports become a real requirement.+ end_col = self._col_number_to_letter(min(max_cols, 26))+ cell_range = f"{sheet_id}!A1:{end_col}{max_rows}"+ token_type = (+ lark.AccessTokenType.USER if feishu_access_token else lark.AccessTokenType.TENANT+ )+ request = (+ lark.BaseRequest.builder()+ .http_method(lark.HttpMethod.GET)+ .uri(f"/open-apis/sheets/v2/spreadsheets/{token}/values/{cell_range}")+ .token_types({token_type})+ .build()+ )+ response = self._call_api(client.request, request, feishu_access_token)+ if not response.success():+ _raise_from_lark_response(+ response,+ operation=f"read spreadsheet range {cell_range}",+ resource=token,+ )++ data = json.loads(response.raw.content)+ values = data.get("data", {}).get("valueRange", {}).get("values", [])+ return [[str(cell) if cell is not None else "" for cell in row] for row in values]++ @staticmethod+ def _col_number_to_letter(number: int) -> str:+ return chr(ord("A") + number - 1) if 1 <= number <= 26 else "Z"++ def _parse_bitable(+ self,+ app_token: str,+ feishu_access_token: Optional[str] = None,+ *,+ table_id: Optional[str] = None,+ table_name: Optional[str] = None,+ view_id: Optional[str] = None,+ ) -> Tuple[str, str]:+ """Fetch a Feishu bitable app and convert it to Markdown."""+ if view_id and not table_id:+ raise ValueError("Feishu Base URL with 'view' must also include 'table'")++ from lark_oapi.api.bitable.v1 import (+ ListAppTableFieldRequest,+ ListAppTableRecordRequest,+ ListAppTableRequest,+ )++ client = self._get_client(use_user_token=bool(feishu_access_token))+ config = self._get_config()+ if table_id:+ tables = [(table_id, table_name or table_id)]+ title = table_name or table_id+ if view_id:+ title = f"{title} ({view_id})"+ markdown_parts = []+ heading = "###"+ else:+ table_models = []+ page_token = None+ while True:+ builder = ListAppTableRequest.builder().app_token(app_token).page_size(100)+ if page_token:+ builder = builder.page_token(page_token)+ tables_response = self._call_api(+ client.bitable.v1.app_table.list,+ builder.build(),+ feishu_access_token,+ )+ if not tables_response.success():+ _raise_from_lark_response(+ tables_response,+ operation=f"list bitable tables for {app_token}",+ resource=app_token,+ )+ table_models.extend(tables_response.data.items or [])+ if not getattr(tables_response.data, "has_more", False):+ break+ page_token = getattr(tables_response.data, "page_token", None)+ if not page_token:+ raise RuntimeError("Feishu returned more bitable tables without a page token")++ tables = [(table.table_id, table.name or table.table_id) for table in table_models]+ title = f"Bitable ({len(tables)} tables)"+ markdown_parts = [f"# {title}"]+ heading = "##"++ for current_table_id, current_table_name in tables:+ fields = []+ page_token = None+ while True:+ builder = (+ ListAppTableFieldRequest.builder()+ .app_token(app_token)+ .table_id(current_table_id)+ .page_size(100)+ )+ if page_token:+ builder = builder.page_token(page_token)+ fields_response = self._call_api(+ client.bitable.v1.app_table_field.list,+ builder.build(),+ feishu_access_token,+ )+ if not fields_response.success():+ _raise_from_lark_response(+ fields_response,+ operation=f"list fields for bitable table {current_table_id}",+ resource=app_token,+ )+ fields.extend(fields_response.data.items or [])+ if not getattr(fields_response.data, "has_more", False):+ break+ page_token = getattr(fields_response.data, "page_token", None)+ if not page_token:+ raise RuntimeError(+ f"Feishu returned more fields for table {current_table_id} "+ "without a page token"+ )+ field_names = [field.field_name for field in fields]++ records = []+ page_token = None+ records_truncated = False+ while len(records) < config.max_records_per_table:+ remaining = config.max_records_per_table - len(records)+ builder = (+ ListAppTableRecordRequest.builder()+ .app_token(app_token)+ .table_id(current_table_id)+ .page_size(min(remaining, 500))+ )+ if view_id:+ builder = builder.view_id(view_id)+ if page_token:+ builder = builder.page_token(page_token)+ records_response = self._call_api(+ client.bitable.v1.app_table_record.list,+ builder.build(),+ feishu_access_token,+ )+ if not records_response.success():+ _raise_from_lark_response(+ records_response,+ operation=f"list records for bitable table {current_table_id}",+ resource=app_token,+ )+ items = records_response.data.items or []+ records.extend(items[:remaining])+ has_more = bool(records_response.data.has_more)+ if len(items) > remaining:+ records_truncated = True+ break+ if not has_more:+ break+ if len(records) >= config.max_records_per_table:+ records_truncated = True+ break+ page_token = records_response.data.page_token+ if not page_token:+ raise RuntimeError(+ f"Feishu returned more records for table {current_table_id} "+ "without a page token"+ )++ parts = [f"{heading} {current_table_name}", f"**Records:** {len(records)}"]+ if field_names and records:+ rows = [field_names]+ for record in records:+ fields = record.fields or {}+ rows.append(+ [self._format_bitable_field(fields.get(name, "")) for name in field_names]+ )+ parts.append(format_table_to_markdown(rows, has_header=True))+ if records_truncated:+ parts.append(f"\n*... records truncated at {config.max_records_per_table} ...*")+ markdown_parts.append("\n\n".join(parts))++ return "\n\n".join(markdown_parts), title++ @classmethod+ def _format_bitable_field(cls, value: Any) -> str:+ """Render the common structured values returned by bitable fields."""+ if value is None:+ return ""+ if isinstance(value, list):+ return ", ".join(cls._format_bitable_field(item) for item in value)+ if isinstance(value, dict):+ file_token = value.get("file_token")+ name = str(value.get("name") or "image")+ media_type = value.get("type") or mimetypes.guess_type(name)[0]+ if file_token and str(media_type).lower().startswith("image/"):+ return f""+ return str(value.get("text", value.get("name", value)))+ return str(value)diff --git a/openviking/parse/converter.py b/openviking/parse/converter.pydeleted file mode 100644index 2f627ea44..000000000--- a/openviking/parse/converter.py+++ /dev/null@@ -1,83 +0,0 @@-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.-# SPDX-License-Identifier: AGPL-3.0-"""Document converters for consistent rendering."""--import asyncio-import tempfile-from pathlib import Path-from typing import Optional--from openviking_cli.utils.logger import get_logger--logger = get_logger(__name__)---class DocumentConverter:- """Converts documents to PDF for consistent rendering (DOCX/MD/PPTX -> PDF)."""-- def __init__(self, temp_dir: Optional[Path] = None):- self.temp_dir = temp_dir-- async def to_pdf(self, file_path: Path) -> Optional[Path]:- """Convert document to PDF."""- ext = file_path.suffix.lower()-- if ext == ".pdf":- return file_path- elif ext in (".docx", ".pptx"):- return await self._convert_with_libreoffice(file_path)- elif ext in (".md", ".markdown"):- return await self._convert_markdown_to_pdf(file_path)-- logger.warning(f"No converter available for {ext}")- return None-- async def _convert_with_libreoffice(self, file_path: Path) -> Optional[Path]:- """Convert using LibreOffice (soffice)."""- output_dir = self.temp_dir or Path(tempfile.gettempdir())- output_path = output_dir / f"{file_path.stem}.pdf"-- try:- cmd = [- "soffice",- "--headless",- "--convert-to",- "pdf",- "--outdir",- str(output_dir),- str(file_path),- ]- process = await asyncio.create_subprocess_exec(- *cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE- )- await process.communicate()-- if process.returncode == 0 and output_path.exists():- return output_path- return None- except FileNotFoundError:- logger.warning("LibreOffice not found")- return None- except Exception as e:- logger.error(f"Conversion error: {e}")- return None-- async def _convert_markdown_to_pdf(self, file_path: Path) -> Optional[Path]:- """Convert Markdown to PDF using pandoc."""- output_dir = self.temp_dir or Path(tempfile.gettempdir())- output_path = output_dir / f"{file_path.stem}.pdf"-- try:- cmd = ["pandoc", str(file_path), "-o", str(output_path), "--pdf-engine=xelatex"]- process = await asyncio.create_subprocess_exec(- *cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE- )- await process.communicate()-- if process.returncode == 0 and output_path.exists():- return output_path- return await self._convert_with_libreoffice(file_path)- except FileNotFoundError:- return await self._convert_with_libreoffice(file_path)- except Exception:- return Nonediff --git a/openviking/parse/custom.py b/openviking/parse/custom.pydeleted file mode 100644index 601d72099..000000000--- a/openviking/parse/custom.py+++ /dev/null@@ -1,244 +0,0 @@-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.-# SPDX-License-Identifier: AGPL-3.0-"""-Custom parser protocol and wrappers for OpenViking.--Supports two ways to extend parsing:-1. Protocol-based: Implement CustomParserProtocol-2. Callback-based: Pass a simple async function-"""--from pathlib import Path-from typing import TYPE_CHECKING, Callable, List, Optional, Union--from typing_extensions import Protocol, runtime_checkable--if TYPE_CHECKING:- from openviking.parse.base import ParseResult---@runtime_checkable-class CustomParserProtocol(Protocol):- """- Protocol for custom parsers.-- External parsers must implement this interface to be registered- with ParserRegistry.-- Example:- ```python- class MyCustomParser:- @property- def supported_extensions(self) -> List[str]:- return [".xyz"]-- def can_handle(self, source: Union[str, Path]) -> bool:- return str(source).endswith(".xyz")-- async def parse(self, source: Union[str, Path], **kwargs) -> ParseResult:- # Custom parsing logic- ...- ```- """-- def can_handle(self, source: Union[str, Path]) -> bool:- """- Check if this parser can handle the given source.-- Args:- source: File path or content string-- Returns:- True if this parser can handle the source- """- ...-- async def parse(self, source: Union[str, Path], **kwargs) -> "ParseResult":- """- Parse the source and return a ParseResult.-- Args:- source: File path or content string- **kwargs: Additional parsing options-- Returns:- ParseResult with document tree- """- ...-- @property- def supported_extensions(self) -> List[str]:- """- List of supported file extensions.-- Returns:- List of extensions (e.g., [".xyz", ".abc"])- """- ...---class CustomParserWrapper:- """- Wrapper to adapt external CustomParserProtocol to BaseParser interface.-- This allows external parsers to be registered and used seamlessly- alongside built-in parsers.- """-- def __init__(- self,- custom_parser: CustomParserProtocol,- extensions: Optional[List[str]] = None,- ):- """- Initialize wrapper.-- Args:- custom_parser: External parser implementing CustomParserProtocol- extensions: Override supported extensions (optional)- """- if not isinstance(custom_parser, CustomParserProtocol):- raise TypeError(- f"custom_parser must implement CustomParserProtocol, "- f"got {type(custom_parser).__name__}"- )-- self.custom_parser = custom_parser- self._extensions = extensions or custom_parser.supported_extensions-- @property- def supported_extensions(self) -> List[str]:- """Return supported extensions."""- return self._extensions-- def can_parse(self, path: Union[str, Path]) -> bool:- """Check if can parse the given file."""- return self.custom_parser.can_handle(path)-- async def parse(self, source: Union[str, Path], **kwargs) -> "ParseResult":- """- Parse the source using the custom parser.-- Args:- source: File path or content string- **kwargs: Additional options-- Returns:- ParseResult from custom parser-- Raises:- ValueError: If the custom parser cannot handle this source- """- if not self.custom_parser.can_handle(source):- raise ValueError(- f"Parser {type(self.custom_parser).__name__} cannot handle source: {source}"- )-- return await self.custom_parser.parse(source, **kwargs)-- async def parse_content(- self, content: str, source_path: Optional[str] = None, **kwargs- ) -> "ParseResult":- """- Parse content string.-- Note: Most custom parsers work with file paths, so this may not- be supported. Override in custom parser if needed.-- Args:- content: Content string- source_path: Optional source path for reference- **kwargs: Additional options-- Returns:- ParseResult-- Raises:- NotImplementedError: If custom parser doesn't support content parsing- """- raise NotImplementedError(- f"Parser {type(self.custom_parser).__name__} "- "does not support content parsing. Use parse() with file path instead."- )---class CallbackParserWrapper:- """- Wrapper for simple callback-based parsers.-- Allows registering a simple async function as a parser without- implementing the full CustomParserProtocol.-- Example:- ```python- async def my_parser(source: Union[str, Path], **kwargs) -> ParseResult:- root = ResourceNode(type=NodeType.ROOT, title="My Document")- return create_parse_result(- root=root,- source_path=str(source),- source_format="custom",- parser_name="my_parser",- )-- registry.register_callback(".xyz", my_parser)- ```- """-- def __init__(- self,- extension: str,- parse_fn: Callable[[Union[str, Path]], "ParseResult"],- name: Optional[str] = None,- ):- """- Initialize callback wrapper.-- Args:- extension: File extension (e.g., ".xyz")- parse_fn: Async function that takes source and returns ParseResult- name: Optional parser name for identification- """- self.extension = extension- self.parse_fn = parse_fn- self.name = name or f"callback_{extension}"-- @property- def supported_extensions(self) -> List[str]:- """Return supported extension."""- return [self.extension]-- def can_parse(self, path: Union[str, Path]) -> bool:- """Check if can parse the given file."""- return str(path).lower().endswith(self.extension.lower())-- async def parse(self, source: Union[str, Path], **kwargs) -> "ParseResult":- """- Parse using the callback function.-- Args:- source: File path- **kwargs: Additional options passed to callback-- Returns:- ParseResult from callback- """- return await self.parse_fn(source, **kwargs)-- async def parse_content(- self, content: str, source_path: Optional[str] = None, **kwargs- ) -> "ParseResult":- """- Parse content - typically not supported for callback parsers.-- Args:- content: Content string- source_path: Optional source path- **kwargs: Additional options-- Raises:- NotImplementedError: Callback parsers typically work with files- """- raise NotImplementedError(- f"Callback parser {self.name} does not support content parsing. "- "Use parse() with file path instead."- )diff --git a/openviking/parse/parser_router.py b/openviking/parse/parser_router.pyindex 0c2a4a3ae..d629b78ee 100644--- a/openviking/parse/parser_router.py+++ b/openviking/parse/parser_router.py@@ -7,12 +7,10 @@ Routing is controlled by ov.conf (OpenVikingConfig.parser_api)."""from pathlib import Path-from typing import TYPE_CHECKING, Union+from typing import Unionfrom urllib.parse import urlparse-if TYPE_CHECKING:- from openviking.parse.accessors.base import LocalResource-+from openviking.parse.accessors.base import LocalResource, SourceTypefrom openviking.parse.base import ParseResultfrom openviking.parse.registry import ParserRegistryfrom openviking_cli.utils.logger import get_logger@@ -34,10 +32,28 @@ class ParserRouter:self._parser_registry = parser_registryself._understanding_api = None- def should_use_understanding_api(self, source_path: Union[str, Path]) -> bool:+ def understanding_api_enabled(self) -> bool:+ """Return whether the external parser is enabled."""+ try:+ from openviking_cli.utils.config.open_viking_config import get_openviking_config++ parser_api = getattr(get_openviking_config(), "parser_api", None)+ except Exception:+ return False+ return bool(parser_api and getattr(parser_api, "enable", False))++ def should_use_understanding_api(+ self,+ source: Union[str, Path, LocalResource],+ resolved_extension: str = "",+ ) -> bool:"""Decide whether to use UnderstandingAPI."""+ # FeishuAccessor has already normalized proprietary content to Markdown.+ if isinstance(source, LocalResource) and source.source_type == SourceType.FEISHU:+ return False+try:from openviking_cli.utils.config.open_viking_config import get_openviking_config@@ -45,21 +61,33 @@ class ParserRouter:except Exception:return False- parser_api = ov_config.parser_api- if not parser_api.enable:+ parser_api = getattr(ov_config, "parser_api", None)+ if not parser_api or not getattr(parser_api, "enable", False):return False- if parser_api.enable_feishu_url:- try:- from openviking.parse.accessors.feishu_accessor import FeishuAccessor+ source_path = self._extract_source_path(source)+ try:+ from openviking.parse.accessors.feishu_accessor import FeishuAccessor- if FeishuAccessor._is_feishu_url(str(source_path)):- return True- except Exception:- pass+ if FeishuAccessor._is_feishu_url(str(source_path)):+ return bool(getattr(parser_api, "enable_feishu_url", False))+ except Exception:+ pass++ ext = self._normalize_extension(resolved_extension) or self._extract_extension(source_path)+ extensions = getattr(parser_api, "extensions", None) or []+ return ext in extensions- ext = self._extract_extension(source_path)- return ext in parser_api.extensions+ def should_use_understanding_directly(self, source: str, **kwargs) -> bool:+ forced = kwargs.get("parser_backend") == "understanding"+ return bool(+ (forced or self.should_use_understanding_api(source))+ and self._get_understanding_api().can_submit_url_directly(source, **kwargs)+ )++ @staticmethod+ def _normalize_extension(extension: str) -> str:+ return str(extension or "").lower().lstrip(".")def _extract_extension(self, source_path: Union[str, Path]) -> str:source = str(source_path)@@ -74,7 +102,25 @@ class ParserRouter:"""source_path = self._extract_source_path(source)- if self.should_use_understanding_api(source_path):+ parser_backend = kwargs.pop("parser_backend", None)+ if parser_backend not in {None, "internal", "understanding"}:+ raise ValueError(f"Unknown parser backend: {parser_backend}")++ normalized_feishu = (+ isinstance(source, LocalResource) and source.source_type == SourceType.FEISHU+ )+ use_understanding = not normalized_feishu and (+ parser_backend == "understanding"+ or (+ parser_backend is None+ and self.should_use_understanding_api(+ source,+ resolved_extension=str(kwargs.get("resolved_extension") or ""),+ )+ )+ )++ if use_understanding:display = source_pathif isinstance(source_path, str) and source_path.startswith(("http://", "https://")):display = "<url>"@@ -93,12 +139,15 @@ class ParserRouter:logger.info(f"[ParserRouter] Using internal ParserRegistry for {display}")return await self._parser_registry.parse(source_path, **kwargs)- async def submit_url(self, source: str, **kwargs) -> str:- if not self.should_use_understanding_api(source):+ async def submit(self, source: Union[str, Path, LocalResource], **kwargs) -> str:+ source_path = self._extract_source_path(source)+ if Path(source_path).is_file():+ return await self._get_understanding_api().submit_file(source_path)+ if not self.should_use_understanding_api(str(source_path)):raise ValueError("source is not routed to UnderstandingAPI")- return await self._get_understanding_api().submit_url(source, **kwargs)+ return await self._get_understanding_api().submit_url(str(source_path), **kwargs)- def _extract_source_path(self, source: Union[str, Path, "LocalResource"]) -> Union[str, Path]:+ def _extract_source_path(self, source: Union[str, Path, LocalResource]) -> Union[str, Path]:"""Extract a filesystem path from the source."""if hasattr(source, "path"):return source.pathdiff --git a/openviking/parse/parsers/README.md b/openviking/parse/parsers/README.mdindex 1508cdad7..935181e75 100644--- a/openviking/parse/parsers/README.md+++ b/openviking/parse/parsers/README.md@@ -1,555 +1,10 @@-# OpenViking 解析系统+# Built-in parsers-OpenViking 的解析系统负责将各种格式的原始文档转换为结构化的上下文信息,遵循 L0/L1/L2 三层信息模型。该系统采用模块化设计,支持多种文档格式,并允许用户通过协议扩展自定义解析器。+OpenViking only routes resources to built-in parsers. Applications add resources through+`add_resource`; parser registries are internal implementation details.-## 核心架构+For the complete Accessor, Parser, Understanding, Connector, and asynchronous execution+flow, see [Resource ingestion routing](../../../docs/design/resource-ingestion-routing.md).-### 三层信息模型 (L0/L1/L2)--OpenViking 采用独特的三层信息模型来组织上下文内容,针对不同使用场景提供不同粒度的信息:--| 层级 | 文件 | 大小 | 用途 |-|------|------|------|------|-| **L0** | `.abstract.md` | <200 token | 摘要,支持向量化检索、用于目录下的快速浏览 |-| **L1** | `.overview.md` | <1000 token | 决策参考、内容导览 |-| **L2** | `content.md` 或原始文件 | 文件完整内容 | 供详细阅读(仅遍历终点)|--**设计出发点**:不同场景需要不同粒度的内容。通过上下文信息的分层,我们允许 Agent 按需读取需要的信息并节省上下文窗口的 token 消耗。--### 举例说明--一份技术文档的三层内容示例:--```-L0: "这是一份介绍 OpenViking 上下文数据库的技术文档,该文档主要介绍其中的上下文提取流程,涵盖资源解析和信息提取、存储和索引的流程规范,其中重点介绍了..."--L1: """-# 目录 (Line 1 ~ Line 10)-目录部分提供了本文的目录结构,共划分为 4 个一级章节,分别是资源解析、信息提取、存储机制、语义索引机制。--# 资源解析 (Line 11 ~ Line 50)-资源解析部分介绍了如何解析外部知识,包括文档、图片、视频等,包括具体的用例...--# 信息提取 (Line 51 ~ Line 100)-信息提取部分介绍了如何从资源中提取摘要和导览信息,以文本文件、图片、视频文件等作为具体示例...--# 存储机制 (Line 101 ~ Line 150)-存储机制部分介绍了如何将上下文存储到 VikingFS 中,包括文件系统、索引结构等。--# 语义索引机制 (Line 151 ~ Line 200)-语义索引机制介绍了如何将上下文存储到向量库中,支持语义检索。-"""-```--### 解析流程 (v5.0 架构)--```-┌─────────────────────────────────────────────────────────────────────┐-│ 资源提取流程 (v5.0) │-├─────────────────────────────────────────────────────────────────────┤-│ │-│ ┌─────────┐ ┌─────────────────────────────────────────────┐ │-│ │ 文件 │───▶│ Parser │ │-│ │ PDF/MD │ │ ┌─────────────────────────────────────┐ │ │-│ │ 图片... │ │ │ 解析 + 创建文件和目录结构 │ │ │-│ └─────────┘ │ │ (无 LLM 调用) │ │ │-│ │ └──────────────┬──────────────────────┘ │ │-│ └─────────────────┼───────────────────────────┘ │-│ │ │-│ ▼ ParseResult │-│ │ (temp_dir_path) │-│ ┌─────────────────┴───────────────────────────┐ │-│ │ TreeBuilder │ │-│ │ ┌─────────┐ ┌──────────────────────────┐ │ │-│ │ │ 1.移动 │─▶│ 2.入队 SemanticQueue │ │ │-│ │ │ 到AGFS │ │ (自底向上处理) │ │ │-│ │ └─────────┘ └──────────────────────────┘ │ │-│ └─────────────────────┬───────────────────────┘ │-│ │ │-│ ▼ BuildingTree │-│ ┌─────────────────────┴───────────────────────┐ │-│ │ SemanticQueue (异步) │ │-│ │ ┌─────────────────────────────────────┐ │ │-│ │ │ SemanticProcessor (自底向上): │ │ │-│ │ │ 1. 收集子目录 abstract │ │ │-│ │ │ 2. 生成文件 summary (并发LLM) │ │ │-│ │ │ 3. 生成 .abstract.md (L0) │ │ │-│ │ │ 4. 生成 .overview.md (L1) │ │ │-│ │ │ 5. 直接调用向量化写入 │ │ │-│ │ └─────────────────────────────────────┘ │ │-│ └─────────────────────┬───────────────────────┘ │-│ │ │-│ ▼ │-│ ┌─────────────────────┴───────────────────────┐ │-│ │ AGFS + 向量库 │ │-│ │ L0/L1/L2 文件 + 向量化索引 │ │-│ └─────────────────────────────────────────────┘ │-│ │-└─────────────────────────────────────────────────────────────────────┘-```--## 解析器类型--### 1. MarkdownParser (`markdown.py`)-**支持格式**: `.md`, `.markdown`, `.mdown`, `.mkd`--核心文档解析器,所有文本类文档最终都通过 MarkdownParser 处理。采用 v5.0 简化架构:--#### 解析逻辑(单阶段)-- 按标题层级(#, ##, ###)拆分章节,支持混合目录结构(文件 + 子目录)-- 切分规则:- - 小文件(< 4000 tokens):保留原文件名,直接保存- - 大文件(> 4000 tokens):按章节切分- - 章节有子章节且总大小 > 4000:章节变目录,直接内容视为第一个虚拟子章节参与合并- - 章节无子章节但超长:创建目录,按段落切分为 `章节名_1.md`, `章节名_2.md`- - 正常章节:保存为 `章节名.md`-- 小章节合并规则(< 800 tokens):- - 连续的小章节会合并,直到总大小 >= 800 tokens 或无法与下一个章节合并- - 合并后文件名用下划线拼接所有章节名:`章节A_章节B_章节C.md`- - 直接内容(标题前或子标题前的内容)视为虚拟章节,使用父级名称参与合并-- 直接写入临时目录的文件和目录结构-- **无 LLM 调用**:语义生成移到 SemanticQueue 异步处理--#### 语义生成(异步,由 SemanticProcessor 处理)-- TreeBuilder 将目录移到 AGFS 后,入队到 SemanticQueue-- SemanticProcessor 自底向上处理每个目录:- - 收集子目录的 `.abstract.md`- - 并发生成文件 summary(LLM)- - 生成当前目录的 `.abstract.md` (L0) 和 `.overview.md` (L1)- - 直接调用向量化写入--### 2. PDFParser (`pdf.py`)-**支持格式**: `.pdf`--统一 PDF 解析器,采用双策略转换:-- **本地策略**: 使用 `pdfplumber` 进行文本和表格提取-- **远程策略**: 使用 `MinerU API` 进行高级 PDF 处理-- **自动策略**: 先尝试本地,失败时回退到 MinerU(如果配置了)--处理流程:`PDF → Markdown → ParseResult`,最终通过 MarkdownParser 处理。--### 3. HTMLParser (`html.py`)-**支持格式**: `.html`, `.htm`--使用 `trafilatura` 抽取正文,去除广告、导航等噪音元素,转换为 Markdown 后通过 MarkdownParser 处理。--### 4. TextParser (`text.py`)-**支持格式**: `.txt`, `.text`, `.log`, `.csv`, `.tsv`, `.json`, `.yaml`, `.yml`, `.xml`, `.ini`, `.cfg`, `.conf`--纯文本解析器,支持多种文本格式。对于结构化格式(JSON、YAML、XML)会尝试提取结构化信息。--### 5. CodeRepositoryParser (`code/*`)-**支持来源**: github 代码仓库等--代码解析器,支持语法高亮和代码结构分析。能识别函数、类、方法等代码元素,遵循 `.gitignore` 并忽略常见非代码目录。--### 6. MediaParser (`media/*.py`)--**支持格式**:-- 图片: `.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`-- 视频: `.mp4`, `.mov`, `.avi`, `.webm`-- 音频: `.mp3`, `.wav`, `.m4a`, `.flac`--多媒体解析器,使用 VLM(视觉语言模型)分析图像、视频和音频内容,生成文本描述。多媒体解析器当且仅当 add-resource 调用时只添加上述文件类型时生效。即:-1. 当添加目录时,系统将对多媒体文件暂不生成单独目录和文本描述,仅存储和进行递归摘要。-2. 当单独添加多媒体文件时,多媒体解析器会直接解析该文件,然后通过单独目录存放,在目录下生成文本描述。--#### 存储组织策略--多媒体文件的存储采用以下层级结构:--```-viking://resource/-├── images/ # 图片文件-│ └── 20240820/ # 上传日期(YYYYMMDD)-│ └── 20240820_123456_jpg/ # 文件文件夹(文件名_扩展名)-│ ├── .abstract.md # L0 摘要-│ ├── .overview.md # L1 概览-│ └── 20240820_123456.jpg # 原始文件-├── audio/ # 音频文件-│ └── 20240820/-│ └── my_song_mp3/-│ ├── .abstract.md-│ ├── .overview.md-│ └── my_song.mp3-└── video/ # 视频文件- └── 20240820/- └── my_video_mp4/- ├── .abstract.md- ├── .overview.md- └── my_video.mp4-```--详细说明:--1. **媒体子目录**: 在 `viking://resource` 下按类型划分为三个子目录- - `viking://resource/images`: 存储未明确指定目标路径的图片文件- - `viking://resource/audio`: 存储未明确指定目标路径的音频文件- - `viking://resource/video`: 存储未明确指定目标路径的视频文件--2. **日期子目录**: 每次上传的文件按当前日期(格式:YYYYMMDD)组织,而非文件内部元信息的时间- - 例如:`viking://resource/images/20240820/` 存储 2024年8月20日上传的所有图片--3. **文件文件夹**: 为每个多媒体文件创建一个专属文件夹,命名规则为:`文件名_扩展名`(扩展名不含点)- - 示例:上传 `20240820_123456.jpg` → 创建文件夹 `20240820_123456_jpg`--4. **文件夹内容**: 每个文件文件夹内包含:- - **原始文件**: 保持原始文件名,空格字符替换为下划线 `_`(因为 OpenViking URI 不允许包含空格)- - 示例:`photo 1.jpg` → 保存为 `photo_1.jpg`- - `.abstract.md` (L0 层): 摘要信息(<200 token)- - 图片:文件名、内容描述、画面风格等- - 音频:文件名、时长、内容概述等- - 视频:文件名、时长、内容概述等- - `.overview.md` (L1 层): 概览信息- - 图片:除摘要外,还包含尺寸、OCR 识别结果、场景和主体描述等- - 音频:除摘要外,还包含语音/歌词识别结果、章节时间线等- - 视频:除摘要外,还包含使用场景等(未来会支持切分视频,递归存储子文件)---## 核心组件--### BaseParser (`base_parser.py`)-所有解析器的抽象基类,定义了解析器的标准接口:--```python-class BaseParser(ABC):- @abstractmethod- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:- """从文件路径或内容字符串解析文档"""- pass-- @abstractmethod- async def parse_content(self, content: str, source_path: Optional[str] = None, instruction: str = "", **kwargs) -> ParseResult:- """直接解析文档内容"""- pass-- @property- @abstractmethod- def supported_extensions(self) -> List[str]:- """支持的文件扩展名列表"""- pass-```--### ParserRegistry (`registry.py`)-解析器注册表,提供自动解析器选择和注册机制:--```python-class ParserRegistry:- def __init__(self, register_optional: bool = True):- """初始化注册表并注册默认解析器"""- self._parsers: Dict[str, BaseParser] = {}- self._extension_map: Dict[str, str] = {}-- # 注册核心解析器- self.register("text", TextParser())- self.register("markdown", MarkdownParser())- self.register("pdf", PDFParser())- self.register("html", HTMLParser())- self.register("code", CodeRepositoryParser())-- def get_parser_for_file(self, path: Union[str, Path]) -> Optional[BaseParser]:- """根据文件扩展名获取合适的解析器"""- pass-- def register_custom(self, handler: "CustomParserProtocol", extensions: Optional[List[str]] = None, name: Optional[str] = None):- """注册自定义解析器(协议方式)"""- pass-```--### TreeBuilder (`tree_builder.py`)-树构建器,负责将临时目录移动到 AGFS 并入队语义处理:--```python-class TreeBuilder:- async def finalize_from_temp(- self,- temp_dir_path: str,- scope: str,- base_uri: Optional[str] = None,- source_path: Optional[str] = None,- source_format: Optional[str] = None,- ) -> "BuildingTree":- """- 从临时目录最终化树结构:- 1. 移动文件到 AGFS- 2. 入队到 SemanticQueue(自底向上处理)- 3. 扫描创建 Resource 对象- """- pass-```--### 资源类型-解析系统使用统一的 Resource 类型。--## 使用示例--### 基本使用--```python-from openviking.parse.parsers.markdown import MarkdownParser-from openviking.parse.registry import get_registry--# 方式1:直接使用解析器-parser = MarkdownParser()-result = await parser.parse("document.md")--# 方式2:通过注册表自动选择-registry = get_registry()-result = await registry.parse("document.pdf") # 自动选择 PDFParser-```--### 自定义解析器--```python-from openviking.parse.parsers.base_parser import BaseParser-from openviking.parse.base import ParseResult, ResourceNode, NodeType, create_parse_result--class CustomParser(BaseParser):- @property- def supported_extensions(self) -> List[str]:- return [".xyz"]-- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:- content = self._read_file(source)- root = ResourceNode(type=NodeType.ROOT, title="Custom Document")- return create_parse_result(root=root, source_path=str(source), source_format="custom")--# 注册自定义解析器-registry = get_registry()-registry.register("custom", CustomParser())-```--### 协议方式扩展--```python-from openviking.parse.custom import CustomParserProtocol-from typing import List--class MyParser:- @property- def supported_extensions(self) -> List[str]:- return [".myformat"]-- def can_handle(self, source) -> bool:- return str(source).endswith(".myformat")-- async def parse(self, source, **kwargs):- # 实现解析逻辑- pass--# 注册(自动包装为 BaseParser)-registry.register_custom(MyParser(), name="my_parser")-```--## 扩展指南--### 1. 继承 BaseParser(推荐)-适合需要完整控制解析流程的场景:--```python-from openviking.parse.parsers.base_parser import BaseParser--class MyParser(BaseParser):- @property- def supported_extensions(self) -> List[str]:- return [".myformat"]-- async def parse(self, source, instruction="", **kwargs):- # 实现三阶段解析- # 1. 解析原始内容- # 2. 生成 L0/L1- # 3. 写入临时目录- pass-- async def parse_content(self, content, source_path=None, instruction="", **kwargs):- # 类似实现- pass-```--### 2. 实现 CustomParserProtocol-适合已有解析逻辑,需要快速集成:--```python-from openviking.parse.custom import CustomParserProtocol--class ExistingParser:- def parse_file(self, path):- # 现有解析逻辑- pass--# 适配器模式-class Adapter(CustomParserProtocol):- def __init__(self, existing_parser):- self.parser = existing_parser-- @property- def supported_extensions(self):- return [".existing"]-- def can_handle(self, source):- return True-- async def parse(self, source, **kwargs):- result = self.parser.parse_file(source)- # 转换为 ParseResult- pass-```--### 3. 回调函数方式-适合简单转换场景:--```python-from openviking.parse.registry import ParserRegistry--async def simple_parser(source, **kwargs):- content = Path(source).read_text()- # 简单处理- return create_parse_result(...)--registry = get_registry()-registry.register_callback(".simple", simple_parser, name="simple_parser")-```--## 设计原则--### 1. 解析与语义分离原则-- 解析器完成:解析 + 文件/目录创建(零 LLM 调用)-- 语义生成:异步 SemanticQueue 处理(自底向上)-- TreeBuilder 只负责文件移动和入队-- 减少解析阻塞,降低内存压力--### 2. 异步语义生成-- Parser 输出临时目录后即可返回-- SemanticQueue 后台处理语义信息生成-- 支持并发文件 summary 生成-- 直接向量化写入,不经过 EmbeddingQueue--### 3. 混合目录结构-- 一个目录下可同时包含文件和子目录-- 章节有子章节时,直接内容保存为同名文件-- 灵活的内容组织,符合自然文档结构--### 4. 临时目录架构-- 每个解析器在临时目录中构建完整结构-- TreeBuilder 只负责移动和入队-- 支持并发解析,避免内存冲突--### 5. 统一 Markdown 处理-- 所有文档格式先转成 Markdown-- 统一通过 MarkdownParser 处理结构-- 保证一致的解析质量和输出格式--### 6. 轻量级索引-- Context 对象不存储内容,只存 URI 和元数据-- 内容通过 `get_abstract()` / `get_overview()` / `get_detail()` 按需加载-- 加载后缓存在内存中,避免重复读取--## 存储结构--### 最终 AGFS 结构--混合目录结构示例(章节有子章节时,小章节合并):--```-viking://resources/Python_异步编程指南/-├── .abstract.md # L0: 目录摘要-├── .overview.md # L1: 目录概览-├── Python_异步编程指南_第一章_asyncio_基础.md # 直接内容 + 第一章合并(均 < 800 tokens)-├── 第二章_高级模式/ # > 4000 tokens 且有子章节,变目录-│ ├── .abstract.md-│ ├── .overview.md-│ ├── 第二章_高级模式_并发控制.md # 直接内容 + 并发控制合并-│ └── 错误处理.md # 单独保存(>= 800 tokens)-└── 附录.md # < 4000 tokens,直接文件-```--超大章节无子章节时的切分:--```-第二章_高级模式/-├── .abstract.md-├── .overview.md-├── 第二章_高级模式_1.md # 按段落切分-├── 第二章_高级模式_2.md-└── 第二章_高级模式_3.md-```--### 临时目录结构--```-viking://temp/abc123/-└── document/- ├── Python_异步编程指南.md- ├── 第一章_asyncio_基础.md- ├── 第二章_高级模式/- │ ├── 第二章_高级模式.md- │ ├── 并发控制.md- │ └── 错误处理.md- └── 附录.md-```--注:临时目录只包含文件和目录结构,`.abstract.md` 和 `.overview.md` 由 SemanticProcessor 异步生成。--## 性能优化--### 1. 并发处理-- 支持多个文档同时解析-- 异步 I/O 操作-- 智能资源调度--### 2. 缓存机制-- 解析结果缓存-- 向量化结果复用-- 内容按需加载--### 3. 增量更新-- 支持文档部分更新-- 智能重新解析-- 最小化计算开销--## 故障排除--### 常见问题--1. **解析失败**- - 检查文件格式是否支持- - 验证文件完整性- - 检查依赖库版本--2. **LLM 调用失败**- - 检查 API 密钥配置- - 验证网络连接- - 检查请求配额--3. **内存不足**- - 启用临时目录模式- - 减少并发解析数量- - 优化解析策略--### 调试建议--```python-import logging-logging.basicConfig(level=logging.DEBUG)--# 启用详细日志-parser = MarkdownParser()-result = await parser.parse("test.md", debug=True)-```--## 版本历史--| 版本 | 日期 | 更新内容 |-|------|------|----------|-| v5.0 | 2026-01-19 | 解析与语义分离,引入 SemanticQueue,支持混合目录结构 |-| v4.0 | 2026-01-07 | 引入临时目录架构,解析器自包含设计 |-| v3.0 | 2026-01-07 | 统一 Markdown 处理流程 |-| v2.0 | 2026-01-05 | 增加多媒体解析器支持 |-| v1.0 | 2026-01-04 | 初始解析系统架构 |--## 相关文档--- [OpenViking 整体架构](../../../docs/zh/concepts/01-architecture.md)-- [上下文提取流程](../../../docs//zh/concepts/07-extraction.md)-- [存储系统设计](../../../docs/zh/concepts/05-storage.md)-- [配置指南](../../../docs/zh/configuration/configuration.md)+Each built-in parser implements `BaseParser` and returns a `ParseResult`. New formats are+added to the repository as built-in parsers and registered in `openviking/parse/registry.py`.diff --git a/openviking/parse/parsers/feishu.py b/openviking/parse/parsers/feishu.pydeleted file mode 100644index dfa8ef732..000000000--- a/openviking/parse/parsers/feishu.py+++ /dev/null@@ -1,907 +0,0 @@-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.-# SPDX-License-Identifier: AGPL-3.0-"""-Feishu/Lark cloud document parser for OpenViking.--Supports:-- Documents: https://*.feishu.cn/docx/{document_id}-- Wiki pages: https://*.feishu.cn/wiki/{token}-- Spreadsheets: https://*.feishu.cn/sheets/{token}-- Bitable: https://*.feishu.cn/base/{app_token}--All types are converted to Markdown then parsed via MarkdownParser.-"""--import asyncio-import json-import os-import time-from pathlib import Path-from typing import Any, Dict, List, Optional, Tuple, Union-from urllib.parse import urlparse--from openviking.parse.base import (- NodeType,- ParseResult,- ResourceNode,- create_parse_result,- format_table_to_markdown,-)-from openviking.parse.parsers.base_parser import BaseParser-from openviking_cli.utils.config.parser_config import FeishuConfig-from openviking_cli.utils.logger import get_logger--logger = get_logger(__name__)--_ALLOWED_FEISHU_HOSTS = ("feishu.cn", "larksuite.com", "larkoffice.com")---def _getattr_safe(obj, key: str, default=None):- """Get attribute from SDK object or dict, with safe fallback."""- if isinstance(obj, dict):- return obj.get(key, default)- return getattr(obj, key, default)---def _is_allowed_feishu_url(source_path: str) -> bool:- """Return True for Feishu/Lark URLs on allowed hosts."""- parsed = urlparse(source_path)- hostname = (parsed.hostname or "").lower().rstrip(".")- return any(- hostname == allowed_host or hostname.endswith(f".{allowed_host}")- for allowed_host in _ALLOWED_FEISHU_HOSTS- )---class FeishuParser(BaseParser):- """Parser for Feishu/Lark cloud documents.-- Block type detection uses a two-tier strategy:- 1. Primary: block_type integer → attribute name lookup via _BLOCK_TYPE_TO_ATTR- (reliable, not affected by SDK changes)- 2. Fallback: scan _KNOWN_CONTENT_ATTRS whitelist for unknown block types- (auto-compat for new types without code changes)- """-- # Attributes that skip processing (structural containers or metadata)- _SKIP_ATTRS = {"page", "table_cell", "quote_container", "grid", "grid_column"}-- # Attribute → special handler method (non-text blocks)- _SPECIAL_BLOCK_HANDLERS = {- "divider": "_handle_divider",- "image": "_handle_image",- "table": "_table_block_to_markdown",- "sheet": "_embedded_sheet_to_markdown",- }-- # Attribute → markdown prefix template for text-bearing blocks.- # "{text}" is replaced with extracted text content.- # Headings are handled dynamically (heading1-heading9 → # through #########).- _TEXT_FORMAT = {- "bullet": "- {text}",- "quote": "> {text}",- }-- # Known block_type integer → SDK attribute name mapping.- # Primary dispatch mechanism for reliable block detection.- # Source: Feishu OpenAPI documentation + lark-oapi SDK Block class.- _BLOCK_TYPE_TO_ATTR = {- 1: "page",- 2: "text",- 3: "heading1",- 4: "heading2",- 5: "heading3",- 6: "heading4",- 7: "heading5",- 8: "heading6",- 9: "heading7",- 10: "heading8",- 11: "heading9",- 12: "bullet",- 13: "ordered",- 14: "code",- 15: "quote",- 17: "todo",- 18: "bitable",- 19: "callout",- 22: "divider",- 24: "file",- 27: "image",- 30: "sheet",- 31: "table",- 32: "table_cell",- 34: "quote_container",- }-- # All known content attribute names on SDK Block objects (for fallback detection).- _KNOWN_CONTENT_ATTRS = frozenset(- {- "page",- "text",- "heading1",- "heading2",- "heading3",- "heading4",- "heading5",- "heading6",- "heading7",- "heading8",- "heading9",- "bullet",- "ordered",- "code",- "quote",- "todo",- "callout",- "divider",- "image",- "table",- "table_cell",- "quote_container",- "sheet",- "file",- "bitable",- "equation",- "task",- "grid",- "grid_column",- "iframe",- "board",- "chat_card",- "diagram",- "agenda",- "agenda_item",- "agenda_item_content",- "agenda_item_title",- "ai_template",- "isv",- "jira_issue",- "link_preview",- "meeting_notes_qa",- "mindnote",- "okr",- "okr_key_result",- "okr_objective",- "okr_progress",- "project",- "reference_base",- "reference_synced",- "source_synced",- "sub_page_list",- "undefined",- "view",- "wiki_catalog",- }- )-- # Document type → parse method name mapping.- # Wiki nodes are resolved to one of these types via _resolve_wiki_node.- # New types can be supported by adding an entry here and the corresponding method.- _DOC_TYPE_HANDLERS = {- "docx": "_parse_docx",- "sheets": "_parse_sheets",- "base": "_parse_bitable",- }-- # Wiki obj_type normalization (API returns short names)- _WIKI_TYPE_MAP = {"doc": "docx", "sheet": "sheets", "bitable": "base"}-- def __init__(self, config: Optional[FeishuConfig] = None):- self._client = None- self._config = config- self._markdown_parser = None-- @property- def supported_extensions(self) -> List[str]:- return [] # URL-based parser, no file extensions-- # ========== Configuration & Client ==========-- def _get_config(self):- """Get FeishuConfig from OpenViking config."""- if self._config is None:- from openviking_cli.utils.config import get_openviking_config-- self._config = get_openviking_config().feishu- return self._config-- def _get_markdown_parser(self):- """Lazy import and create MarkdownParser with Feishu parser config."""- if self._markdown_parser is None:- from openviking.parse.parsers.markdown import MarkdownParser-- self._markdown_parser = MarkdownParser(config=self._get_config())- return self._markdown_parser-- def _get_client(self):- """Lazy-init lark-oapi client."""- if self._client is None:- try:- import lark_oapi as lark- except ImportError:- raise ImportError(- "lark-oapi is required for Feishu document parsing. "- "Install it with: pip install lark-oapi>=1.0.0"- )- config = self._get_config()- app_id = config.app_id or os.getenv("FEISHU_APP_ID", "")- app_secret = config.app_secret or os.getenv("FEISHU_APP_SECRET", "")- if not app_id or not app_secret:- raise ValueError(- "Feishu credentials not configured. Set FEISHU_APP_ID and "- "FEISHU_APP_SECRET environment variables, or configure in ov.conf."- )- domain = config.domain or "https://open.feishu.cn"- self._client = (- lark.Client.builder().app_id(app_id).app_secret(app_secret).domain(domain).build()- )- return self._client-- # ========== URL Parsing ==========-- @staticmethod- def _parse_feishu_url(url: str) -> Tuple[str, str]:- """- Extract doc_type and token from Feishu URL.-- Returns:- (doc_type, token) e.g. ("docx", "doxcnABC123")- """- if not _is_allowed_feishu_url(url):- raise ValueError(f"Feishu host not allowed: {url}")-- parsed = urlparse(url)- path_parts = [p for p in parsed.path.split("/") if p]- if len(path_parts) < 2:- raise ValueError(f"Cannot parse Feishu URL: {url}")- doc_type = path_parts[0] # docx, wiki, sheets, base- token = path_parts[1]- return doc_type, token-- # ========== Main Parse ==========-- async def parse(self, source: Union[str, Path], instruction: str = "", **kwargs) -> ParseResult:- """Parse a Feishu cloud document URL."""- url = str(source)- start_time = time.time()-- try:- doc_type, token = self._parse_feishu_url(url)-- title = None- if doc_type == "wiki":- real_type, real_token, title = await asyncio.to_thread(- self._resolve_wiki_node, token- )- doc_type, token = real_type, real_token-- handler_name = self._DOC_TYPE_HANDLERS.get(doc_type)- if not handler_name:- raise ValueError(- f"Unsupported Feishu document type: {doc_type}. "- f"Supported: {list(self._DOC_TYPE_HANDLERS.keys())}"- )- markdown, _doc_title = await asyncio.to_thread(getattr(self, handler_name), token)-- md_parser = self._get_markdown_parser()- result = await md_parser.parse_content(- markdown, source_path=url, instruction=instruction, **kwargs- )- result.source_format = f"feishu_{doc_type}"- result.parser_name = "FeishuParser"- result.parse_time = time.time() - start_time- result.meta["feishu_doc_type"] = doc_type- result.meta["feishu_token"] = token- return result-- except Exception as e:- logger.error(f"[FeishuParser] Failed to parse {url}: {e}")- return create_parse_result(- root=ResourceNode(type=NodeType.ROOT),- source_path=url,- source_format="feishu",- parser_name="FeishuParser",- parse_time=time.time() - start_time,- warnings=[f"Feishu parse failed: {e}"],- )-- async def parse_content(- self,- content: str,- source_path: Optional[str] = None,- instruction: str = "",- **kwargs,- ) -> ParseResult:- """Not typically used for Feishu (URL-based parser)."""- if source_path and _is_allowed_feishu_url(source_path):- return await self.parse(source_path, instruction=instruction, **kwargs)- raise NotImplementedError("FeishuParser requires a Feishu URL. Use parse() instead.")-- # ========== Wiki Resolution ==========-- def _resolve_wiki_node(self, token: str) -> Tuple[str, str, Optional[str]]:- """- Resolve wiki token to actual document type, token, and title.-- Returns:- (doc_type, obj_token, title)- """- from lark_oapi.api.wiki.v2 import GetNodeSpaceRequest-- client = self._get_client()- request = GetNodeSpaceRequest.builder().token(token).build()- response = client.wiki.v2.space.get_node(request)- if not response.success():- raise RuntimeError(- f"Failed to resolve wiki node {token}: code={response.code}, msg={response.msg}"- )- node = response.data.node- obj_type = node.obj_type or ""- obj_token = node.obj_token or ""- title = node.title-- # Normalize type names- doc_type = self._WIKI_TYPE_MAP.get(obj_type, obj_type)-- return doc_type, obj_token, title-- # ========== Docx Parsing ==========-- def _parse_docx(self, document_id: str) -> Tuple[str, str]:- """- Fetch all blocks and convert to Markdown.-- Returns:- (markdown_content, document_title)- """- blocks = self._fetch_all_blocks(document_id)- if not blocks:- return "", "Untitled"-- # Build block lookup by block_id- block_map = {b.block_id: b for b in blocks}-- # Find title from page block- doc_title = "Untitled"- for b in blocks:- if b.page is not None:- if b.page.elements:- doc_title = self._extract_text_from_elements(b.page.elements)- break-- # Convert blocks to markdown- markdown_lines = []- ordered_counter: Dict[str, int] = {}-- for block in blocks:- if block.page is not None:- continue # Skip page container-- line = self._block_to_markdown(- block, block_map, ordered_counter, document_id=document_id- )- if line is not None:- markdown_lines.append(line)-- markdown = "\n\n".join(markdown_lines)-- if doc_title and doc_title != "Untitled":- markdown = f"# {doc_title}\n\n{markdown}"-- return markdown, doc_title-- def _fetch_all_blocks(self, document_id: str) -> list:- """Fetch all blocks with pagination. Returns list of SDK block objects."""- from lark_oapi.api.docx.v1 import ListDocumentBlockRequest-- client = self._get_client()- all_blocks = []- page_token = None-- while True:- builder = (- ListDocumentBlockRequest.builder()- .document_id(document_id)- .page_size(500)- .document_revision_id(-1)- )- if page_token:- builder = builder.page_token(page_token)-- request = builder.build()- response = client.docx.v1.document_block.list(request)-- if not response.success():- raise RuntimeError(- f"Failed to fetch blocks for {document_id}: "- f"code={response.code}, msg={response.msg}"- )-- items = response.data.items or []- all_blocks.extend(items)-- if not response.data.has_more:- break- page_token = response.data.page_token-- return all_blocks-- # ========== Block -> Markdown Conversion ==========-- def _detect_block_attr(self, block) -> Optional[str]:- """Detect which content attribute is populated on a block object.-- Uses block_type integer as the primary dispatch (reliable), falling- back to attribute inspection over a known whitelist for unknown types.- """- # Primary: lookup by block_type integer- block_type = getattr(block, "block_type", None)- if block_type is not None:- attr = self._BLOCK_TYPE_TO_ATTR.get(block_type)- if attr:- return attr-- # Fallback: scan known content attributes for unknown block types- for attr in self._KNOWN_CONTENT_ATTRS:- if getattr(block, attr, None) is not None:- return attr- return None-- def _block_to_markdown(- self, block, block_map: Dict, ordered_counter: Dict[str, int], document_id: str = ""- ) -> Optional[str]:- """Convert a single SDK block object to markdown string.-- Uses block_type integer for primary dispatch, with attribute whitelist- fallback for unknown types. Formatting is data-driven via _TEXT_FORMAT- and _SPECIAL_BLOCK_HANDLERS tables.- """- attr = self._detect_block_attr(block)-- if attr is None:- return None-- # Skip structural containers (processed via their children)- if attr in self._SKIP_ATTRS:- return None-- # Reset ordered list counter when any non-ordered block appears- if attr != "ordered":- parent_id = block.parent_id or ""- if parent_id in ordered_counter:- del ordered_counter[parent_id]-- # Special blocks (non-text: divider, image, table, sheet)- special_handler = self._SPECIAL_BLOCK_HANDLERS.get(attr)- if special_handler:- return getattr(self, special_handler)(block, block_map, document_id=document_id)-- # --- Text-bearing blocks: extract elements, apply formatting ---- content_obj = getattr(block, attr, None)- if not content_obj or not hasattr(content_obj, "elements") or not content_obj.elements:- return None-- text = self._extract_text_from_elements(content_obj.elements)- if not text:- return None-- # Headings: heading1 -> #, heading2 -> ##, ...- if attr.startswith("heading"):- level = int(attr.replace("heading", "") or "1")- return f"{'#' * level} {text}"-- # Ordered list (needs counter state)- if attr == "ordered":- parent_id = block.parent_id or ""- counter = ordered_counter.get(parent_id, 0) + 1- ordered_counter[parent_id] = counter- return f"{counter}. {text}"-- # Code block (needs language from style)- if attr == "code":- lang = ""- if hasattr(content_obj, "style") and content_obj.style:- lang = str(getattr(content_obj.style, "language", "") or "")- return f"```{lang}\n{text}\n```"-- # Todo (needs done state from style)- if attr == "todo":- done = False- if hasattr(content_obj, "style") and content_obj.style:- done = getattr(content_obj.style, "done", False)- checkbox = "[x]" if done else "[ ]"- return f"- {checkbox} {text}"-- # Simple template formatting (bullet, quote, etc.)- fmt = self._TEXT_FORMAT.get(attr)- if fmt:- return fmt.format(text=text)-- # Default: return plain text (covers callout, equation, task, unknown, etc.)- return text-- @staticmethod- def _handle_divider(block, block_map: Dict = None, **_) -> str:- """Convert divider block to markdown."""- return "---"-- @staticmethod- def _handle_image(block, block_map: Dict = None, **_) -> Optional[str]:- """Convert image block to markdown."""- image = block.image- if not image:- return None- file_token = image.token or ""- alt_text = getattr(image, "alt", "") or "image"- return f""-- def _extract_block_text(self, block, attr_name: str) -> str:- """Extract text from a block's named attribute (e.g. block.text, block.heading2)."""- content_obj = getattr(block, attr_name, None)- if content_obj and hasattr(content_obj, "elements") and content_obj.elements:- return self._extract_text_from_elements(content_obj.elements)- return ""-- def _extract_text_from_elements(self, elements) -> str:- """Convert Feishu TextElement SDK objects to formatted text."""- if not elements:- return ""- parts = []- for element in elements:- # TextRun- text_run = element.text_run- if text_run:- content = text_run.content or ""- style = text_run.text_element_style- content = self._apply_text_style(content, style)- parts.append(content)- continue-- # MentionUser- mention_user = element.mention_user- if mention_user:- user_id = _getattr_safe(mention_user, "user_id", "user")- parts.append(f"@{user_id}")- continue-- # MentionDoc- mention_doc = element.mention_doc- if mention_doc:- title = _getattr_safe(mention_doc, "title", "document")- url = _getattr_safe(mention_doc, "url", "")- parts.append(f"[{title}]({url})" if url else str(title))- continue-- # Equation- equation = element.equation- if equation:- parts.append(f"${_getattr_safe(equation, 'content', '')}$")- continue-- return "".join(parts)-- @staticmethod- def _apply_text_style(text: str, style) -> str:- """Apply markdown formatting based on TextElementStyle SDK object."""- if not text or not style:- return text- # inline_code (SDK uses 'inline_code', not 'code_inline')- if getattr(style, "inline_code", False):- return f"`{text}`"- # link- link = getattr(style, "link", None)- if link:- url = _getattr_safe(link, "url", "")- if url:- text = f"[{text}]({url})"- if getattr(style, "bold", False):- text = f"**{text}**"- if getattr(style, "italic", False):- text = f"*{text}*"- if getattr(style, "strikethrough", False):- text = f"~~{text}~~"- return text-- def _table_block_to_markdown(self, block, block_map: Dict, **_) -> Optional[str]:- """Convert table block to markdown table."""- table = block.table- children = block.children- if not table or not children:- return None-- prop = table.property- if not prop:- return None- row_size = prop.row_size or 0- col_size = prop.column_size or 0- if not row_size or not col_size:- return None-- rows = []- for row_idx in range(row_size):- row = []- for col_idx in range(col_size):- cell_idx = row_idx * col_size + col_idx- if cell_idx < len(children):- cell_block_id = children[cell_idx]- cell_block = block_map.get(cell_block_id)- cell_text = self._extract_cell_text(cell_block, block_map)- row.append(cell_text)- else:- row.append("")- rows.append(row)-- return format_table_to_markdown(rows, has_header=True) if rows else None-- def _extract_cell_text(self, cell_block, block_map: Dict) -> str:- """Extract text from a table cell block by reading its children."""- if not cell_block or not cell_block.children:- return ""- texts = []- for child_id in cell_block.children:- child = block_map.get(child_id)- if not child:- continue- # Use attribute-driven detection to find text in any block type- attr = self._detect_block_attr(child)- if attr:- text = self._extract_block_text(child, attr)- if text:- texts.append(text)- return " ".join(texts)-- # ========== Embedded Sheet in Docx ==========-- def _embedded_sheet_to_markdown(- self, block, block_map: Dict = None, *, document_id: str = "", **_- ) -> Optional[str]:- """Convert an embedded sheet block to markdown table.-- These blocks appear in docx documents when a user embeds a spreadsheet- view. The block contains a sheet token in the format- ``{spreadsheet_token}_{sheet_id}``.- """- import lark_oapi as lark-- client = self._get_client()- block_id = block.block_id- doc_id = document_id or block.parent_id-- raw_req = (- lark.BaseRequest.builder()- .http_method(lark.HttpMethod.GET)- .uri(f"/open-apis/docx/v1/documents/{doc_id}/blocks/{block_id}")- .token_types({lark.AccessTokenType.TENANT})- .build()- )- raw_resp = client.request(raw_req)- if not raw_resp.success():- return None-- data = json.loads(raw_resp.raw.content)- sheet_token = data.get("data", {}).get("block", {}).get("sheet", {}).get("token", "")- if not sheet_token:- return None-- # Parse token: {spreadsheet_token}_{sheet_id}- parts = sheet_token.rsplit("_", 1)- if len(parts) != 2:- return None- spreadsheet_token, sheet_id = parts-- # Read cell data and trim empty trailing columns- try:- rows = self._read_sheet_range(spreadsheet_token, sheet_id, max_rows=100, max_cols=26)- if rows:- rows = self._trim_empty_columns(rows)- if rows:- return format_table_to_markdown(rows, has_header=True)- except Exception as e:- logger.warning(f"[FeishuParser] Failed to read embedded sheet {sheet_token}: {e}")-- return None-- @staticmethod- def _trim_empty_columns(rows: List[List[str]]) -> List[List[str]]:- """Remove trailing columns that are entirely empty across all rows."""- if not rows:- return rows- max_cols = max(len(r) for r in rows)- # Find rightmost non-empty column- last_col = 0- for col in range(max_cols):- for row in rows:- if col < len(row) and row[col].strip():- last_col = col + 1- if last_col == 0:- return []- return [row[:last_col] for row in rows]-- # ========== Sheets Parsing ==========-- def _parse_sheets(self, token: str) -> Tuple[str, str]:- """Fetch spreadsheet data and convert to Markdown."""- from lark_oapi.api.sheets.v3 import (- GetSpreadsheetRequest,- QuerySpreadsheetSheetRequest,- )-- client = self._get_client()- config = self._get_config()-- # Get spreadsheet metadata- meta_request = GetSpreadsheetRequest.builder().spreadsheet_token(token).build()- meta_response = client.sheets.v3.spreadsheet.get(meta_request)- title = "Spreadsheet"- if meta_response.success() and meta_response.data.spreadsheet:- title = meta_response.data.spreadsheet.title or title-- # Get sheet list- sheets_request = QuerySpreadsheetSheetRequest.builder().spreadsheet_token(token).build()- sheets_response = client.sheets.v3.spreadsheet_sheet.query(sheets_request)- if not sheets_response.success():- raise RuntimeError(- f"Failed to fetch sheets for {token}: "- f"code={sheets_response.code}, msg={sheets_response.msg}"- )-- sheets = sheets_response.data.sheets or []- markdown_parts = [f"# {title}", f"**Sheets:** {len(sheets)}"]-- for sheet in sheets:- sheet_id = sheet.sheet_id- sheet_title = sheet.title or sheet_id- row_count = sheet.grid_properties.row_count if sheet.grid_properties else 0- col_count = sheet.grid_properties.column_count if sheet.grid_properties else 0-- parts = [f"## Sheet: {sheet_title}"]-- if row_count == 0 or col_count == 0:- parts.append("*Empty sheet*")- markdown_parts.append("\n\n".join(parts))- continue-- parts.append(f"**Dimensions:** {row_count} rows x {col_count} columns")-- rows_to_read = min(row_count, config.max_rows_per_sheet)- cell_data = self._read_sheet_range(token, sheet_id, rows_to_read, col_count)-- if cell_data:- table_md = format_table_to_markdown(cell_data, has_header=True)- parts.append(table_md)-- if row_count > config.max_rows_per_sheet:- parts.append(- f"\n*... {row_count - config.max_rows_per_sheet} more rows truncated ...*"- )-- markdown_parts.append("\n\n".join(parts))-- return "\n\n".join(markdown_parts), title-- def _read_sheet_range(- self, token: str, sheet_id: str, max_rows: int, max_cols: int- ) -> List[List[str]]:- """Read cell values from a sheet range using lark-oapi SDK."""- import lark_oapi as lark-- client = self._get_client()- end_col = self._col_number_to_letter(min(max_cols, 26))- range_str = f"{sheet_id}!A1:{end_col}{max_rows}"-- request = (- lark.BaseRequest.builder()- .http_method(lark.HttpMethod.GET)- .uri(f"/open-apis/sheets/v2/spreadsheets/{token}/values/{range_str}")- .token_types({lark.AccessTokenType.TENANT})- .build()- )-- response = client.request(request)- if not response.success():- raise RuntimeError(- f"Failed to read sheet range: code={response.code}, msg={response.msg}"- )-- data = json.loads(response.raw.content)- values = data.get("data", {}).get("valueRange", {}).get("values", [])- return [[str(cell) if cell is not None else "" for cell in row] for row in values]-- @staticmethod- def _col_number_to_letter(n: int) -> str:- """Convert column number (1-based) to letter (A, B, ..., Z)."""- return chr(ord("A") + n - 1) if 1 <= n <= 26 else "Z"-- # ========== Bitable Parsing ==========-- def _parse_bitable(self, app_token: str) -> Tuple[str, str]:- """Fetch bitable data and convert to Markdown."""- from lark_oapi.api.bitable.v1 import (- ListAppTableFieldRequest,- ListAppTableRecordRequest,- ListAppTableRequest,- )-- client = self._get_client()- config = self._get_config()-- tables_request = ListAppTableRequest.builder().app_token(app_token).build()- tables_response = client.bitable.v1.app_table.list(tables_request)- if not tables_response.success():- raise RuntimeError(- f"Failed to list bitable tables: "- f"code={tables_response.code}, msg={tables_response.msg}"- )-- tables = tables_response.data.items or []- title = f"Bitable ({len(tables)} tables)"- markdown_parts = [f"# {title}"]-- for table in tables:- table_id = table.table_id- table_name = table.name or table_id-- fields_request = (- ListAppTableFieldRequest.builder().app_token(app_token).table_id(table_id).build()- )- fields_response = client.bitable.v1.app_table_field.list(fields_request)- field_names: List[str] = []- if fields_response.success() and fields_response.data.items:- field_names = [f.field_name for f in fields_response.data.items]-- all_records: list = []- page_token = None- while len(all_records) < config.max_records_per_table:- remaining = config.max_records_per_table - len(all_records)- page_size = min(remaining, 500)- builder = (- ListAppTableRecordRequest.builder()- .app_token(app_token)- .table_id(table_id)- .page_size(page_size)- )- if page_token:- builder = builder.page_token(page_token)- records_response = client.bitable.v1.app_table_record.list(builder.build())- if not records_response.success():- break- items = records_response.data.items or []- all_records.extend(items)- if not records_response.data.has_more:- break- page_token = records_response.data.page_token-- parts = [f"## {table_name}"]- parts.append(f"**Records:** {len(all_records)}")-- if field_names and all_records:- rows = [field_names]- for record in all_records:- fields = record.fields or {}- row = [self._format_bitable_field(fields.get(fn, "")) for fn in field_names]- rows.append(row)- parts.append(format_table_to_markdown(rows, has_header=True))-- if len(all_records) >= config.max_records_per_table:- parts.append(f"\n*... records truncated at {config.max_records_per_table} ...*")-- markdown_parts.append("\n\n".join(parts))-- return "\n\n".join(markdown_parts), title-- @staticmethod- def _format_bitable_field(value: Any) -> str:- """Format bitable field value to string."""- if value is None:- return ""- if isinstance(value, list):- texts = []- for item in value:- if isinstance(item, dict):- texts.append(item.get("text", item.get("name", str(item))))- else:- texts.append(str(item))- return ", ".join(texts)- if isinstance(value, dict):- return value.get("text", value.get("name", str(value)))- return str(value)diff --git a/openviking/parse/registry.py b/openviking/parse/registry.pyindex ad57f891c..398c19e43 100644--- a/openviking/parse/registry.py+++ b/openviking/parse/registry.py@@ -7,7 +7,7 @@ Provides automatic parser selection based on file type."""from pathlib import Path-from typing import TYPE_CHECKING, Callable, Dict, List, Optional, Union+from typing import Dict, List, Optional, Unionfrom openviking.parse.base import ParseResultfrom openviking.parse.parsers.base_parser import BaseParser@@ -27,14 +27,8 @@ from openviking.parse.parsers.powerpoint import PowerPointParserfrom openviking.parse.parsers.text import TextParserfrom openviking.parse.parsers.word import WordParserfrom openviking.parse.parsers.zip_parser import ZipParser-from openviking_cli.utils import get_loggerfrom openviking_cli.utils.config.parser_config import ParserConfig-if TYPE_CHECKING:- from openviking.parse.custom import CustomParserProtocol--logger = get_logger(__name__)-class ParserRegistry:"""@@ -45,15 +39,12 @@ class ParserRegistry:def __init__(self,- register_optional: bool = True,parser_configs: Optional[Dict[str, ParserConfig]] = None,):"""Initialize registry with default parsers.Args:- register_optional: Whether to register optional parsers- that require extra dependenciesparser_configs: Dictionary of parser configurations (from load_parser_configs_from_dict)"""self._parsers: Dict[str, BaseParser] = {}@@ -61,129 +52,33 @@ class ParserRegistry:self._parser_configs = parser_configs or {}# Register core parsers- self.register("text", TextParser(config=self._parser_configs.get("text")))- self.register("markdown", MarkdownParser(config=self._parser_configs.get("markdown")))- self.register("pdf", PDFParser(config=self._parser_configs.get("pdf")))- self.register("html", HTMLParser(config=self._parser_configs.get("html")))+ self._register("text", TextParser(config=self._parser_configs.get("text")))+ self._register("markdown", MarkdownParser(config=self._parser_configs.get("markdown")))+ self._register("pdf", PDFParser(config=self._parser_configs.get("pdf")))+ self._register("html", HTMLParser(config=self._parser_configs.get("html")))# Register markitdown-inspired parsers (built-in)- self.register("word", WordParser(config=self._parser_configs.get("word")))- self.register("legacy_doc", LegacyDocParser(config=self._parser_configs.get("legacy_doc")))- self.register("powerpoint", PowerPointParser(config=self._parser_configs.get("powerpoint")))- self.register("excel", ExcelParser(config=self._parser_configs.get("excel")))- self.register("epub", EPubParser(config=self._parser_configs.get("epub")))- self.register("zip", ZipParser())- self.register("directory", DirectoryParser())-- self.register("image", ImageParser(config=self._parser_configs.get("image")))- self.register("audio", AudioParser())- self.register("video", VideoParser())-- def register(self, name: str, parser: BaseParser) -> None:- """- Register a parser.-- Args:- name: Parser name- parser: Parser instance- """+ self._register("word", WordParser(config=self._parser_configs.get("word")))+ self._register("legacy_doc", LegacyDocParser(config=self._parser_configs.get("legacy_doc")))+ self._register(+ "powerpoint", PowerPointParser(config=self._parser_configs.get("powerpoint"))+ )+ self._register("excel", ExcelParser(config=self._parser_configs.get("excel")))+ self._register("epub", EPubParser(config=self._parser_configs.get("epub")))+ self._register("zip", ZipParser())+ self._register("directory", DirectoryParser())++ self._register("image", ImageParser(config=self._parser_configs.get("image")))+ self._register("audio", AudioParser())+ self._register("video", VideoParser())++ def _register(self, name: str, parser: BaseParser) -> None:self._parsers[name] = parser# Map extensions to parser namefor ext in parser.supported_extensions:self._extension_map[ext.lower()] = name- def register_custom(- self,- handler: "CustomParserProtocol",- extensions: Optional[List[str]] = None,- name: Optional[str] = None,- ) -> None:- """- Register a custom parser (Protocol-based).-- Args:- handler: Object implementing CustomParserProtocol- extensions: Optional list of extensions (overrides handler's extensions)- name: Optional parser name (default: "custom_N")-- Example:- ```python- class MyParser:- @property- def supported_extensions(self) -> List[str]:- return [".xyz"]-- def can_handle(self, source) -> bool:- return str(source).endswith(".xyz")-- async def parse(self, source, **kwargs) -> ParseResult:- ...-- registry.register_custom(MyParser(), name="xyz_parser")- ```- """- from openviking.parse.custom import CustomParserWrapper-- # Generate name if not provided- if name is None:- # Count existing custom parsers- custom_count = sum(1 for n in self._parsers if n.startswith("custom_"))- name = f"custom_{custom_count}"-- # Wrap and register- wrapper = CustomParserWrapper(handler, extensions=extensions)- self.register(name, wrapper) # type: ignore- logger.info(f"Registered custom parser '{name}' for {wrapper.supported_extensions}")-- def register_callback(- self,- extension: str,- parse_fn: "Callable[[Union[str, Path]], ParseResult]",- name: Optional[str] = None,- ) -> None:- """- Register a callback function as a parser.-- Args:- extension: File extension (e.g., ".xyz")- parse_fn: Async function that parses and returns ParseResult- name: Optional parser name (default: "callback_<ext>")-- Example:- ```python- async def my_parser(source: Union[str, Path], **kwargs) -> ParseResult:- content = Path(source).read_text()- return create_parse_result(- root=ResourceNode(type=NodeType.ROOT, content=content),- source_path=str(source),- source_format="custom",- parser_name="my_parser",- )-- registry.register_callback(".xyz", my_parser)- ```- """- from openviking.parse.custom import CallbackParserWrapper-- # Generate name if not provided- if name is None:- name = f"callback{extension}"-- # Wrap and register- wrapper = CallbackParserWrapper(extension, parse_fn, name=name)- self.register(name, wrapper) # type: ignore- logger.info(f"Registered callback parser '{name}' for {extension}")-- def unregister(self, name: str) -> None:- """Remove a parser from registry."""- if name in self._parsers:- parser = self._parsers[name]- for ext in parser.supported_extensions:- if self._extension_map.get(ext.lower()) == name:- del self._extension_map[ext.lower()]- del self._parsers[name]-def get_parser_for_file(self, path: Union[str, Path]) -> Optional[BaseParser]:"""Get appropriate parser for a file.diff --git a/openviking/parse/understanding_api.py b/openviking/parse/understanding_api.pyindex 7831121d1..57a01be5b 100644--- a/openviking/parse/understanding_api.py+++ b/openviking/parse/understanding_api.py@@ -87,30 +87,41 @@ class UnderstandingAPI(BaseParser):"""source_str = str(source)original_source = kwargs.get("original_source")- candidate = original_source if isinstance(original_source, str) else source_str- explicit_name = kwargs.get("resource_name") or kwargs.get("source_name")+ resolved_extension = str(kwargs.get("resolved_extension") or "").lower().lstrip(".")+ display_name = kwargs.get("resource_name") or kwargs.get("source_name")prepared_response_id = kwargs.get(PREPARED_RESPONSE_ID_ARG)if prepared_response_id is not None:if not isinstance(prepared_response_id, str) or not prepared_response_id.strip():raise ValueError(f"{PREPARED_RESPONSE_ID_ARG} must be a non-empty string")prepared_response_id = prepared_response_id.strip()- lark_file = self._normalize_lark_file(kwargs)- # Legacy accessor output may keep a Feishu URL in original_source while- # the actual parser input is a local Markdown file. Only direct URL- # routing may use the parser API Lark protocol.++ # Only a directly routed Feishu URL uses the Lark protocol. Accessor+ # output is an existing local Markdown file and must stay local.is_feishu_url = self._is_feishu_url(source_str)- if is_feishu_url and not lark_file and not prepared_response_id:- raise ValueError(- "exactly one Feishu user or tenant access token is required for parser API imports"- )+ lark_file = (+ await self._resolve_lark_file(kwargs)+ if is_feishu_url and not prepared_response_id+ else None+ )url: Optional[str] = Nonelocal_path: Optional[Path] = None- if isinstance(candidate, str) and candidate.startswith(("http://", "https://")):- url = candidate+ source_path = Path(source_str)+ if source_path.is_file():+ local_path = source_path+ elif source_str.startswith(("http://", "https://")):+ url = source_str+ elif isinstance(original_source, str) and original_source.startswith(+ ("http://", "https://")+ ):+ url = original_source+ else:+ local_path = source_path++ if url is not None:parsed = urlparse(url)- doc_name = Path(parsed.path).stem or "resource"- doc_type = Path(parsed.path).suffix.lower().lstrip(".") or "unknown"+ inferred_name = Path(parsed.path).name+ doc_type = resolved_extension or Path(parsed.path).suffix.lower().lstrip(".")if is_feishu_url:path_parts = [part for part in parsed.path.split("/") if part]doc_type = {@@ -118,17 +129,17 @@ class UnderstandingAPI(BaseParser):"base": "bitable",}.get(path_parts[0], path_parts[0])else:- local_path = Path(candidate)- if not local_path.is_file():+ inferred_name = local_path.name if local_path is not None else ""+ if local_path is None or not local_path.is_file():raise ValueError("UnderstandingAPI supports http(s) URLs or local files. ""Got an invalid local file path.")- doc_name = local_path.stem or "resource"- doc_type = local_path.suffix.lower().lstrip(".") or "unknown"+ doc_type = resolved_extension or local_path.suffix.lower().lstrip(".")- if explicit_name:- doc_name = Path(str(explicit_name)).name or doc_name+ effective_name = str(display_name or inferred_name or "resource")+ doc_name = Path(effective_name).stem or "resource"+ doc_type = doc_type or "unknown"task_meta: Dict[str, Any] = {}@@ -170,7 +181,7 @@ class UnderstandingAPI(BaseParser):zip_path = await self._download_zip(zip_url)try:- if is_feishu_url and not explicit_name:+ if is_feishu_url and not display_name:archive_root = self._single_zip_root_name(zip_path)if archive_root:doc_name = archive_root@@ -211,7 +222,7 @@ class UnderstandingAPI(BaseParser):result = ParseResult(root=root_node,- source_path=url or source_str,+ source_path=original_source if isinstance(original_source, str) else url or source_str,source_format=doc_type,temp_dir_path=temp_dir_path,parser_name="UnderstandingAPI",@@ -221,17 +232,32 @@ class UnderstandingAPI(BaseParser):logger.info("[UnderstandingAPI] done")return result+ async def submit_file(self, source: Union[str, Path]) -> str:+ """Upload a local file and submit it without retaining the local artifact."""+ local_path = Path(source)+ if not local_path.is_file():+ raise ValueError("UnderstandingAPI file submission requires an existing local file")++ file_obj = await self._create_file(local_path=local_path)+ file_id = file_obj.get("id")+ if not file_id:+ raise RuntimeError(f"files api missing file_id: {self._safe_error_summary(file_obj)}")++ response_obj = await self._create_response_for_file(file_id=str(file_id))+ response_id = response_obj.get("id")+ if not response_id:+ raise RuntimeError(+ f"responses api missing id: {self._safe_error_summary(response_obj)}"+ )+ return str(response_id)+async def submit_url(self, source: str, **kwargs) -> str:"""Submit a URL without persisting its credentials in a queue payload."""if not isinstance(source, str) or not source.startswith(("http://", "https://")):raise ValueError("UnderstandingAPI URL submission requires an http(s) URL")is_feishu_url = self._is_feishu_url(source)- lark_file = self._normalize_lark_file(kwargs)- if is_feishu_url and not lark_file:- raise ValueError(- "exactly one Feishu user or tenant access token is required for parser API imports"- )+ lark_file = await self._resolve_lark_file(kwargs) if is_feishu_url else Noneparsed = urlparse(source)doc_type = Path(parsed.path).suffix.lower().lstrip(".") or "unknown"@@ -254,6 +280,20 @@ class UnderstandingAPI(BaseParser):)return str(response_id)+ def can_submit_url_directly(self, source: str, **kwargs) -> bool:+ """Return whether this URL can bypass source materialization."""+ if not source.startswith(("http://", "https://")) or not self._is_feishu_url(source):+ return False+ if self._normalize_lark_file(kwargs):+ return True+ try:+ from openviking.resource.feishu_watch_auth import load_feishu_app_credentials++ load_feishu_app_credentials()+ return True+ except (FileNotFoundError, ValueError):+ return False+async def parse_content(self, content: str, source_path: Optional[str] = None, instruction: str = "", **kwargs) -> ParseResult:@@ -396,6 +436,20 @@ class UnderstandingAPI(BaseParser):)return auth or None+ async def _resolve_lark_file(self, kwargs: Dict[str, Any]) -> Dict[str, str]:+ auth = self._normalize_lark_file(kwargs)+ if auth:+ return auth+ try:+ from openviking.resource.feishu_watch_auth import FeishuOAuthClient++ token = await FeishuOAuthClient.from_config().get_tenant_access_token()+ except (FileNotFoundError, ValueError) as exc:+ raise ValueError(+ "exactly one Feishu user or tenant access token is required for parser API imports"+ ) from exc+ return {"tenant_access_token": token}+@staticmethoddef _single_zip_root_name(zip_path: Path) -> Optional[str]:roots = set()@@ -591,40 +645,51 @@ class UnderstandingAPI(BaseParser):async def _unpack_zip_to_temp_dir(self, zip_path: Path, resource_name: str) -> str:viking_fs = get_viking_fs()temp_uri = viking_fs.create_temp_uri()- await viking_fs.mkdir(temp_uri)-- temp_doc_uri = f"{temp_uri}/{resource_name}"- await viking_fs.mkdir(temp_doc_uri)-- with tempfile.TemporaryDirectory() as extract_dir:- with zipfile.ZipFile(zip_path, "r") as zf:- safe_extract_zip(zf, extract_dir)- extract_path = Path(extract_dir)- items = [p for p in extract_path.iterdir() if p.name not in {".", ".."}]- if len(items) == 1 and items[0].is_dir():- root_dir = items[0]- else:- root_dir = extract_path-- image_mappings = await asyncio.to_thread(build_artifact_image_mappings, root_dir)-- for child in root_dir.iterdir():- if child.name in {".", "..", IMAGE_MAPPINGS_FILENAME}:- continue- if child.is_dir():- sub_uri = f"{temp_doc_uri}/{child.name}"- await viking_fs.mkdir(sub_uri)- await self._copy_dir_to_fs(child, sub_uri)+ try:+ await viking_fs.mkdir(temp_uri)++ temp_doc_uri = f"{temp_uri}/{resource_name}"+ await viking_fs.mkdir(temp_doc_uri)++ with tempfile.TemporaryDirectory() as extract_dir:+ with zipfile.ZipFile(zip_path, "r") as zf:+ safe_extract_zip(zf, extract_dir)+ extract_path = Path(extract_dir)+ items = [p for p in extract_path.iterdir() if p.name not in {".", ".."}]+ if len(items) == 1 and items[0].is_dir():+ root_dir = items[0]else:- await viking_fs.write_file_bytes(- f"{temp_doc_uri}/{child.name}", child.read_bytes()+ root_dir = extract_path++ image_mappings = await asyncio.to_thread(build_artifact_image_mappings, root_dir)++ for child in root_dir.iterdir():+ if child.name in {".", "..", IMAGE_MAPPINGS_FILENAME}:+ continue+ if child.is_dir():+ sub_uri = f"{temp_doc_uri}/{child.name}"+ await viking_fs.mkdir(sub_uri)+ await self._copy_dir_to_fs(child, sub_uri)+ else:+ await viking_fs.write_file_bytes(+ f"{temp_doc_uri}/{child.name}", child.read_bytes()+ )++ if image_mappings:+ await viking_fs.write_file(+ f"{temp_doc_uri}/{IMAGE_MAPPINGS_FILENAME}",+ json.dumps(image_mappings, ensure_ascii=False),)-- if image_mappings:- await viking_fs.write_file(- f"{temp_doc_uri}/{IMAGE_MAPPINGS_FILENAME}",- json.dumps(image_mappings, ensure_ascii=False),+ except BaseException:+ try:+ await viking_fs.delete_temp(temp_uri)+ except Exception as cleanup_exc:+ logger.warning(+ "[UnderstandingAPI] Failed to clean temporary artifact %s: %s",+ temp_uri,+ cleanup_exc,)+ raisereturn temp_uridiff --git a/openviking/resource/watch_scheduler.py b/openviking/resource/watch_scheduler.pyindex 6b37d8cfd..d3fa5e063 100644--- a/openviking/resource/watch_scheduler.py+++ b/openviking/resource/watch_scheduler.py@@ -213,7 +213,7 @@ class WatchScheduler:async def _execute_task(self, task) -> None:"""Execute a single watch task.- Calls ResourceService.add_resource to process the resource.+ Calls ResourceService.refresh_resource to re-process the resource.Handles errors gracefully and updates execution time regardless of success/failure.Deactivates tasks when resources no longer exist.@@ -282,7 +282,7 @@ class WatchScheduler:raiseif not should_deactivate:- result = await self._resource_service.add_resource(+ result = await self._resource_service.refresh_resource(path=task.path,ctx=ctx,to=task.to_uri,@@ -292,7 +292,6 @@ class WatchScheduler:build_index=getattr(task, "build_index", True),summarize=getattr(task, "summarize", False),watch_interval=task.watch_interval,- skip_watch_management=True,**processor_kwargs,)diff --git a/openviking/service/resource_service.py b/openviking/service/resource_service.pyindex 773bd6f43..343181746 100644--- a/openviking/service/resource_service.py+++ b/openviking/service/resource_service.py@@ -64,6 +64,7 @@ from openviking_cli.exceptions import (from openviking_cli.utils import get_loggerif TYPE_CHECKING:+ from openviking.parse.accessors.base import LocalResourcefrom openviking.resource.watch_manager import WatchManagerfrom openviking.resource.watch_scheduler import WatchSchedulerfrom openviking.service.resource_memory_link_service import ResourceMemoryLinkService@@ -101,10 +102,27 @@ _ADD_RESOURCE_ARGS_RESERVED_FIELDS = frozenset("telemetry","request_validator","understanding_response_id",+ "parser_backend",+ "resolved_extension","defer_post_processing",})+_INTERNAL_INGESTION_FIELDS = frozenset(+ {+ "manage_watch",+ "parser_args",+ "resource_lock",+ "route_source",+ "skip_watch_management",+ "stage_callback",+ "watch_auth_state",+ "understanding_response_id",+ "parser_backend",+ "resolved_extension",+ }+)+@dataclassclass _ResourceSourceInfo:@@ -161,14 +179,6 @@ class ResourceService:return Nonereturn self._watch_scheduler.watch_manager- def _get_parser_router(self):- if not hasattr(self, "_parser_router"):- from openviking.parse.parser_router import ParserRouter- from openviking.parse.registry import get_registry-- self._parser_router = ParserRouter(get_registry())- return self._parser_router-def _sanitize_watch_processor_kwargs(self, processor_kwargs: Dict[str, Any]) -> Dict[str, Any]:sanitized: Dict[str, Any] = {}for key, value in processor_kwargs.items():@@ -183,7 +193,7 @@ class ResourceService:self,*,watch_manager: Optional["WatchManager"],- skip_watch_management: bool,+ manage_watch: bool,watch_interval: float,target: ContentTargetSpec,root_uri: str,@@ -196,7 +206,7 @@ class ResourceService:watch_auth_state: Optional[Dict[str, Any]],ctx: RequestContext,) -> None:- if not watch_manager or skip_watch_management:+ if not watch_manager or not manage_watch:returntelemetry = get_current_telemetry()with telemetry.measure("resource.watch"):@@ -384,6 +394,15 @@ class ResourceService:target_uri = msg.root_uriparent_uri = Noneinternal_kwargs: Dict[str, Any] = {}+ queued_args = dict(msg.args)+ for key in ("parser_backend", "resolved_extension"):+ if key in queued_args:+ internal_kwargs[key] = queued_args.pop(key)+ normalized_args = self._normalize_add_resource_args(+ queued_args,+ watch_interval=msg.watch_interval,+ )+ internal_kwargs.update(normalized_args.processor_kwargs)if msg.defer_target_resolution:from openviking_cli.utils.uri import VikingURI@@ -393,7 +412,7 @@ class ResourceService:from openviking.parse.understanding_api import PREPARED_RESPONSE_ID_ARGinternal_kwargs[PREPARED_RESPONSE_ID_ARG] = msg.understanding_response_id- return await self.add_resource(+ return await self._execute_resource_ingestion(path=msg.path,ctx=ctx,to=target_uri,@@ -405,11 +424,12 @@ class ResourceService:build_index=msg.build_index,summarize=msg.summarize,watch_interval=msg.watch_interval,- skip_watch_management=msg.skip_watch_management,+ manage_watch=not msg.skip_watch_management,allow_local_path_resolution=msg.allow_local_path_resolution,enforce_public_remote_targets=msg.enforce_public_remote_targets,resource_lock=resource_lock,stage_callback=stage_callback,+ watch_auth_state=normalized_args.watch_auth_state,strict=msg.strict,source_name=msg.source_name,ignore_dirs=msg.ignore_dirs,@@ -418,7 +438,6 @@ class ResourceService:directly_upload_media=msg.directly_upload_media,preserve_structure=msg.preserve_structure,create_parent=msg.create_parent,- args=msg.args,**internal_kwargs,)@@ -486,7 +505,7 @@ class ResourceService:build_index: bool = True,summarize: bool = False,watch_interval: float = 0,- skip_watch_management: bool = False,+ manage_watch: bool = True,allow_local_path_resolution: bool = True,enforce_public_remote_targets: bool = False,args: Optional[Dict[str, Any]] = None,@@ -560,7 +579,7 @@ class ResourceService:build_index=build_index,summarize=summarize,watch_interval=watch_interval,- skip_watch_management=skip_watch_management,+ skip_watch_management=not manage_watch,allow_local_path_resolution=allow_local_path_resolution,enforce_public_remote_targets=enforce_public_remote_targets,strict=bool(kwargs.get("strict", False)),@@ -629,18 +648,6 @@ class ResourceService:)return root_uri, resource_lock- def _should_use_understanding_api(self, path: str) -> bool:- return self._get_parser_router().should_use_understanding_api(path)-- @staticmethod- def _is_feishu_url(path: str) -> bool:- try:- from openviking.parse.accessors.feishu_accessor import FeishuAccessor-- return FeishuAccessor._is_feishu_url(path)- except Exception:- return False-@staticmethoddef _target_doc_name(path: str,@@ -703,15 +710,93 @@ class ResourceService:build_index: bool = True,summarize: bool = False,watch_interval: float = 0,- skip_watch_management: bool = False,allow_local_path_resolution: bool = True,enforce_public_remote_targets: bool = False,- resource_lock: Optional[LockLease] = None,- stage_callback: Optional[Callable[[str], Any]] = None,args: Optional[Dict[str, Any]] = None,**kwargs,) -> Dict[str, Any]:- """Add resource to OpenViking (only supports resources scope).+ """Accept and route a new resource-add request."""+ internal_fields = sorted(set(kwargs).intersection(_INTERNAL_INGESTION_FIELDS))+ if internal_fields:+ raise InvalidArgumentError(+ "add_resource does not accept internal execution fields: "+ + ", ".join(internal_fields)+ )+ return await self._submit_resource_ingestion(+ path=path,+ ctx=ctx,+ to=to,+ parent=parent,+ reason=reason,+ instruction=instruction,+ wait=wait,+ timeout=timeout,+ build_index=build_index,+ summarize=summarize,+ watch_interval=watch_interval,+ manage_watch=True,+ allow_local_path_resolution=allow_local_path_resolution,+ enforce_public_remote_targets=enforce_public_remote_targets,+ args=args,+ **kwargs,+ )++ async def refresh_resource(+ self,+ path: str,+ ctx: RequestContext,+ to: Optional[str] = None,+ parent: Optional[str] = None,+ reason: str = "",+ instruction: str = "",+ timeout: Optional[float] = None,+ build_index: bool = True,+ summarize: bool = False,+ watch_interval: float = 0,+ allow_local_path_resolution: bool = True,+ enforce_public_remote_targets: bool = False,+ **kwargs,+ ) -> Dict[str, Any]:+ """Submit a scheduled refresh without changing its watch task."""+ return await self._submit_resource_ingestion(+ path=path,+ ctx=ctx,+ to=to,+ parent=parent,+ reason=reason,+ instruction=instruction,+ wait=False,+ timeout=timeout,+ build_index=build_index,+ summarize=summarize,+ watch_interval=watch_interval,+ manage_watch=False,+ allow_local_path_resolution=allow_local_path_resolution,+ enforce_public_remote_targets=enforce_public_remote_targets,+ args=None,+ **kwargs,+ )++ async def _submit_resource_ingestion(+ self,+ path: str,+ ctx: RequestContext,+ to: Optional[str] = None,+ parent: Optional[str] = None,+ reason: str = "",+ instruction: str = "",+ wait: bool = False,+ timeout: Optional[float] = None,+ build_index: bool = True,+ summarize: bool = False,+ watch_interval: float = 0,+ manage_watch: bool = True,+ allow_local_path_resolution: bool = True,+ enforce_public_remote_targets: bool = False,+ args: Optional[Dict[str, Any]] = None,+ **kwargs,+ ) -> Dict[str, Any]:+ """Validate and route one resource ingestion request.Args:path: Resource path (local file or URL)@@ -734,8 +819,6 @@ class ResourceService:Note: If the target URI already has an active watch task, a ConflictError will beraised. You must first cancel the existing watch (set watch_interval <= 0) beforecreating a new one.- skip_watch_management: If True, skip watch task management (used by scheduler to- avoid recursive watch task creation during scheduled execution)enforce_public_remote_targets: When True, reject non-public remote hosts andvalidate each outbound HTTP request URL during fetch.args: Parser/accessor-specific options forwarded to the processing chain.@@ -809,12 +892,56 @@ class ResourceService:build_index=build_index,summarize=summarize,watch_interval=watch_interval,- skip_watch_management=skip_watch_management,+ manage_watch=manage_watch,allow_local_path_resolution=allow_local_path_resolution,enforce_public_remote_targets=enforce_public_remote_targets,**kwargs,)+ return await self._execute_resource_ingestion(+ path=path,+ ctx=ctx,+ to=to,+ parent=parent,+ reason=reason,+ instruction=instruction,+ wait=wait,+ timeout=timeout,+ build_index=build_index,+ summarize=summarize,+ watch_interval=watch_interval,+ manage_watch=manage_watch,+ allow_local_path_resolution=allow_local_path_resolution,+ enforce_public_remote_targets=enforce_public_remote_targets,+ watch_auth_state=normalized_args.watch_auth_state,+ parser_args=normalized_args.processor_kwargs,+ **kwargs,+ )++ async def _execute_resource_ingestion(+ self,+ path: str,+ ctx: RequestContext,+ to: Optional[str] = None,+ parent: Optional[str] = None,+ reason: str = "",+ instruction: str = "",+ wait: bool = False,+ timeout: Optional[float] = None,+ build_index: bool = True,+ summarize: bool = False,+ watch_interval: float = 0,+ manage_watch: bool = True,+ allow_local_path_resolution: bool = True,+ enforce_public_remote_targets: bool = False,+ watch_auth_state: Optional[Dict[str, Any]] = None,+ parser_args: Optional[Dict[str, Any]] = None,+ resource_lock: Optional[LockLease] = None,+ stage_callback: Optional[Callable[[str], Any]] = None,+ **kwargs,+ ) -> Dict[str, Any]:+ """Execute an already-routed resource ingestion."""+ self._ensure_initialized()request_start = time.perf_counter()telemetry = get_current_telemetry()telemetry_id = register_wait_telemetry(wait)@@ -824,13 +951,14 @@ class ResourceService:if telemetry_id:request_wait_tracker.register_request(telemetry_id)watch_manager = self._get_watch_manager()- watch_enabled = bool(watch_manager and not skip_watch_management and watch_interval > 0)+ watch_enabled = bool(watch_manager and manage_watch and watch_interval > 0)telemetry.set("resource.flags.wait", wait)telemetry.set("resource.flags.build_index", build_index)telemetry.set("resource.flags.summarize", summarize)telemetry.set("resource.flags.watch_enabled", watch_enabled)+ prepared_resource: Optional["LocalResource"] = Nonetry:target = ContentTargetSpec.from_fields(ctx=ctx,@@ -845,20 +973,57 @@ class ResourceService:if resource_lock is not None:kwargs["resource_lock"] = resource_lock- if (+ async_understanding_candidate = (not waitand not is_git_repo_url(path)- and self._should_use_understanding_api(path)and not allow_local_path_resolutionand self._resource_processor is not None+ )+ direct_understanding = bool(+ async_understanding_candidate+ and self._resource_processor.should_use_understanding_directly(path, **kwargs)+ )++ if (+ async_understanding_candidate+ and not direct_understanding+ and self._resource_processor.understanding_api_enabled()+ ):+ prepared_resource = await self._resource_processor.prepare_resource(+ path,+ ctx,+ allow_local_path_resolution=allow_local_path_resolution,+ **kwargs,+ )++ if (+ prepared_resource is not None+ and self._resource_processor is not None+ and self._resource_processor.should_use_understanding_api(prepared_resource)):+ understanding_source = prepared_resource+ elif direct_understanding:+ understanding_source = path+ else:+ understanding_source = None++ if understanding_source is not None:from openviking.storage.queuefs.add_resource_msg import AddResourceMsg- source_name = kwargs.get("source_name")+ if prepared_resource is not None:+ resolved_extension = str(prepared_resource.meta.get("resolved_extension") or "")+ source_name = (+ kwargs.get("source_name")+ or prepared_resource.meta.get("original_filename")+ or prepared_resource.meta.get("resolved_name")+ )+ else:+ resolved_extension = ""+ source_name = kwargs.get("source_name")source_info = _ResourceSourceInfo(source_name=source_name,source_path=path,- source_format="file",+ source_format=resolved_extension.lstrip(".") or "file",)doc_name = self._target_doc_name(path, source_name, source_info)source_path = source_info.source_path or source_name or path@@ -876,10 +1041,7 @@ class ResourceService:create_parent=target.create_parent,)defer_target_resolution = bool(- candidate_uri- and not source_name- and not watch_enabled- and self._is_feishu_url(path)+ candidate_uri and not source_name and not watch_enabled and direct_understanding)if self._viking_fs is None:raise NotInitializedError("VikingFS")@@ -913,14 +1075,20 @@ class ResourceService:enqueue_started = Falsetry:- queued_args = dict(normalized_args.processor_kwargs)- feishu_access_token = queued_args.pop(FEISHU_ACCESS_TOKEN_ARG, None)- understanding_response_id = None- if self._is_feishu_url(path) and feishu_access_token:- understanding_response_id = await self._get_parser_router().submit_url(- path,- feishu_access_token=feishu_access_token,- )+ queued_args = dict(parser_args or {})+ understanding_response_id = await self._resource_processor.submit_understanding(+ understanding_source,+ **queued_args,+ )+ queued_args.pop(FEISHU_ACCESS_TOKEN_ARG, None)+ if prepared_resource is not None:+ prepared_resource.cleanup()+ prepared_resource = None++ processor_args = self._sanitize_watch_processor_kwargs(queued_args)+ processor_args["parser_backend"] = "understanding"+ if resolved_extension:+ processor_args["resolved_extension"] = resolved_extensionlock_handoff = lock_lease.to_handoff()msg = AddResourceMsg(@@ -946,7 +1114,7 @@ class ResourceService:create_parent=bool(kwargs.get("create_parent", False)),allow_local_path_resolution=allow_local_path_resolution,enforce_public_remote_targets=enforce_public_remote_targets,- args=self._sanitize_watch_processor_kwargs(queued_args),+ args=processor_args,source_name=source_name,lock_handoff=lock_handoff.to_dict() if lock_handoff else None,skip_watch_management=True,@@ -972,7 +1140,7 @@ class ResourceService:)await self._manage_watch_if_needed(watch_manager=watch_manager,- skip_watch_management=skip_watch_management,+ manage_watch=manage_watch,watch_interval=watch_interval,target=target,root_uri=root_uri,@@ -982,7 +1150,7 @@ class ResourceService:build_index=build_index,summarize=summarize,processor_kwargs=kwargs,- watch_auth_state=normalized_args.watch_auth_state,+ watch_auth_state=watch_auth_state,ctx=ctx,)response = {@@ -1005,9 +1173,11 @@ class ResourceService:summarize=summarize,stage_callback=stage_callback,allow_local_path_resolution=allow_local_path_resolution,+ prepared_resource=prepared_resource,defer_post_processing=not wait,**kwargs,)+ prepared_resource = Noneif result.get("status") == "error":return result@@ -1104,7 +1274,7 @@ class ResourceService:job_enqueued = Trueawait self._manage_watch_if_needed(watch_manager=watch_manager,- skip_watch_management=skip_watch_management,+ manage_watch=manage_watch,watch_interval=watch_interval,target=target,root_uri=str(result.get("root_uri") or ""),@@ -1114,7 +1284,7 @@ class ResourceService:build_index=build_index,summarize=summarize,processor_kwargs=kwargs,- watch_auth_state=normalized_args.watch_auth_state,+ watch_auth_state=watch_auth_state,ctx=ctx,)if wait:@@ -1134,6 +1304,8 @@ class ResourceService:)raisefinally:+ if prepared_resource is not None:+ prepared_resource.cleanup()telemetry.set("resource.request.duration_ms",round((time.perf_counter() - request_start) * 1000, 3),diff --git a/openviking/utils/media_processor.py b/openviking/utils/media_processor.pyindex ab1785a76..57e3b33aa 100644--- a/openviking/utils/media_processor.py+++ b/openviking/utils/media_processor.py@@ -5,8 +5,7 @@from pathlib import Pathfrom typing import TYPE_CHECKING, Optional-from openviking.parse import parse-from openviking.parse.accessors.base import SourceType+from openviking.parse.accessors.base import LocalResource, SourceTypefrom openviking.parse.accessors.mime_types import IANA_MEDIA_TYPE_TO_EXTENSIONfrom openviking.parse.base import ParseResultfrom openviking.parse.parsers.constants import (@@ -14,6 +13,7 @@ from openviking.parse.parsers.constants import (DOCUMENTATION_EXTENSIONS,IGNORE_EXTENSIONS,)+from openviking.parse.registry import parsefrom openviking.server.local_input_guard import (is_remote_resource_source,looks_like_local_path,@@ -77,7 +77,6 @@ class UnifiedResourceProcessor:):self.storage = storageself._vlm_processor = vlm_processor- self._document_converter = Noneself._accessor_registry = Nonedef _get_vlm_processor(self) -> Optional["VLMProcessor"]:@@ -104,11 +103,66 @@ class UnifiedResourceProcessor:self._parser_router = ParserRouter(get_registry())return self._parser_router+ def understanding_api_enabled(self) -> bool:+ return self._get_parser_router().understanding_api_enabled()++ def should_use_understanding_api(self, source: str | Path | LocalResource) -> bool:+ if isinstance(source, LocalResource):+ if source.path.is_dir():+ return False+ return self._get_parser_router().should_use_understanding_api(+ source,+ resolved_extension=str(source.meta.get("resolved_extension") or ""),+ )+ return self._get_parser_router().should_use_understanding_api(source)++ def should_use_understanding_directly(self, source: str, **kwargs) -> bool:+ return self._get_parser_router().should_use_understanding_directly(source, **kwargs)++ async def submit_understanding(self, source: str | Path | LocalResource, **kwargs) -> str:+ return await self._get_parser_router().submit(source, **kwargs)++ @staticmethod+ def _set_resolved_identity(resource: LocalResource, source_name: Optional[str]) -> None:+ meta = resource.meta+ if resource.path.is_dir():+ extension = ""+ elif resource.source_type == SourceType.HTTP:+ extension = str(meta.get("extension") or resource.path.suffix)+ else:+ extension = Path(source_name).suffix if source_name else ""+ extension = extension or str(meta.get("extension") or resource.path.suffix)++ meta["resolved_extension"] = extension.lower()+ meta["resolved_name"] = source_name or meta.get("original_filename") or resource.path.name++ async def prepare(+ self,+ source: str,+ allow_local_path_resolution: bool = True,+ **kwargs,+ ) -> LocalResource:+ """Fetch a path/URL and freeze the identity used for parser routing."""+ if (+ not allow_local_path_resolution+ and not is_remote_resource_source(source)+ and looks_like_local_path(source)+ ):+ raise PermissionDeniedError(+ "HTTP server only accepts remote resource URLs or temp-uploaded files; "+ "direct host filesystem paths are not allowed."+ )++ resource = await self._get_accessor_registry().access(source, **kwargs)+ self._set_resolved_identity(resource, kwargs.get("source_name"))+ return resource+async def process(self,source: str,instruction: str = "",allow_local_path_resolution: bool = True,+ prepared_resource: Optional[LocalResource] = None,**kwargs,) -> ParseResult:"""Process any source (file/URL/content) with two-layer architecture.@@ -129,39 +183,47 @@ class UnifiedResourceProcessor:# Treat as raw contentreturn await parse(source, instruction=instruction)- # Block local paths in HTTP server mode, but allow remote URLs- if (- not allow_local_path_resolution- and not is_remote_resource_source(source)- and looks_like_local_path(source)- ):- raise PermissionDeniedError(- "HTTP server only accepts remote resource URLs or temp-uploaded files; "- "direct host filesystem paths are not allowed."- )-- if self._should_bypass_feishu_accessor(source, kwargs):+ if prepared_resource is None and self._has_prepared_understanding_response(kwargs):parse_kwargs = dict(kwargs)parse_kwargs["instruction"] = instructionparse_kwargs["vlm_processor"] = self._get_vlm_processor()parse_kwargs["storage"] = self.storageparse_kwargs["_source_meta"] = {- "source_type": SourceType.FEISHU,+ "source_type": SourceType.HTTP,"original_url": source,}parse_kwargs["original_source"] = source+ parse_kwargs["parser_backend"] = "understanding"parse_kwargs.pop("feishu_access_token", None)- if not self._has_prepared_understanding_response(kwargs):- parse_kwargs["lark_file"] = await self._resolve_feishu_parser_auth(kwargs)explicit_name = kwargs.get("resource_name") or kwargs.get("source_name")if explicit_name:parse_kwargs["resource_name"] = _smart_stem(explicit_name)return await self._get_parser_router().parse(source, **parse_kwargs)- # Phase 1: Accessor - get local resource- registry = self._get_accessor_registry()- local_resource = await registry.access(source, **kwargs)+ if prepared_resource is None and self.should_use_understanding_directly(source, **kwargs):+ parse_kwargs = dict(kwargs)+ parse_kwargs["instruction"] = instruction+ parse_kwargs["vlm_processor"] = self._get_vlm_processor()+ parse_kwargs["storage"] = self.storage+ parse_kwargs["_source_meta"] = {+ "source_type": SourceType.HTTP,+ "original_url": source,+ }+ parse_kwargs["original_source"] = source++ explicit_name = kwargs.get("resource_name") or kwargs.get("source_name")+ if explicit_name:+ parse_kwargs["resource_name"] = _smart_stem(explicit_name)+ return await self._get_parser_router().parse(source, **parse_kwargs)++ # Phase 1: Accessor - get local resource. A caller may prepare a remote+ # resource first so async routing uses the same detected file type.+ local_resource = prepared_resource or await self.prepare(+ source,+ allow_local_path_resolution=allow_local_path_resolution,+ **kwargs,+ )# Use context manager for automatic cleanup, but preserve directories for TreeBuildertry:@@ -171,6 +233,9 @@ class UnifiedResourceProcessor:parse_kwargs["vlm_processor"] = self._get_vlm_processor()parse_kwargs["storage"] = self.storageparse_kwargs["_source_meta"] = local_resource.meta+ parse_kwargs["resolved_extension"] = kwargs.get(+ "resolved_extension"+ ) or local_resource.meta.get("resolved_extension", "")# CRITICAL: Pass along the original_source!# This is the full URL the user provided (e.g. "https://github.com/volcengine/OpenViking")# CodeRepositoryParser and TreeBuilder need this to extract the org/repo format@@ -222,48 +287,9 @@ class UnifiedResourceProcessor:"""Check if source is a URL."""return is_remote_resource_source(source)- @staticmethod- def _is_feishu_url(source: str) -> bool:- """Backward-compatible Feishu URL detector used by legacy tests/callers."""- try:- from openviking.parse.accessors.feishu_accessor import FeishuAccessor-- return FeishuAccessor._is_feishu_url(source)- except Exception:- return False-- def _should_bypass_feishu_accessor(self, source: str, kwargs: dict) -> bool:- token = kwargs.get("feishu_access_token")- if not self._is_feishu_url(source):- return False- if not self._get_parser_router().should_use_understanding_api(source):- return False- if self._has_prepared_understanding_response(kwargs):- return True- if isinstance(token, str) and token.strip():- return True- try:- from openviking.resource.feishu_watch_auth import load_feishu_app_credentials-- load_feishu_app_credentials()- return True- except ValueError:- return False-@staticmethoddef _has_prepared_understanding_response(kwargs: dict) -> bool:from openviking.parse.understanding_api import PREPARED_RESPONSE_ID_ARGvalue = kwargs.get(PREPARED_RESPONSE_ID_ARG)return isinstance(value, str) and bool(value.strip())-- @staticmethod- async def _resolve_feishu_parser_auth(kwargs: dict) -> dict[str, str]:- token = kwargs.get("feishu_access_token")- if isinstance(token, str) and token.strip():- return {"user_access_token": token.strip()}-- from openviking.resource.feishu_watch_auth import FeishuOAuthClient-- tenant_access_token = await FeishuOAuthClient.from_config().get_tenant_access_token()- return {"tenant_access_token": tenant_access_token}diff --git a/openviking/utils/resource_processor.py b/openviking/utils/resource_processor.pyindex 74c64f04f..cf42b59a2 100644--- a/openviking/utils/resource_processor.py+++ b/openviking/utils/resource_processor.py@@ -10,7 +10,7 @@ as described in the OpenViking design document.import inspectimport timefrom collections.abc import Callable-from typing import TYPE_CHECKING, Any, Dict, List, Optional+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Unionfrom openviking.parse.image_rewrite import rewrite_image_urisfrom openviking.parse.tree_builder import TreeBuilder@@ -33,6 +33,7 @@ from openviking_cli.utils import get_loggerfrom openviking_cli.utils.storage import StoragePathif TYPE_CHECKING:+ from openviking.parse.accessors.base import LocalResourcefrom openviking.parse.vlm import VLMProcessorlogger = get_logger(__name__)@@ -92,6 +93,34 @@ class ResourceProcessor:)return self._media_processor+ async def prepare_resource(+ self,+ path: str,+ ctx: RequestContext,+ *,+ allow_local_path_resolution: bool = True,+ **kwargs,+ ) -> "LocalResource":+ """Resolve a source once before selecting an async parser route."""+ with get_viking_fs().bind_request_context(ctx):+ return await self._get_media_processor().prepare(+ path,+ allow_local_path_resolution=allow_local_path_resolution,+ **kwargs,+ )++ def understanding_api_enabled(self) -> bool:+ return self._get_media_processor().understanding_api_enabled()++ def should_use_understanding_api(self, source: Union[str, "LocalResource"]) -> bool:+ return self._get_media_processor().should_use_understanding_api(source)++ def should_use_understanding_directly(self, source: str, **kwargs) -> bool:+ return self._get_media_processor().should_use_understanding_directly(source, **kwargs)++ async def submit_understanding(self, source: Union[str, "LocalResource"], **kwargs) -> str:+ return await self._get_media_processor().submit_understanding(source, **kwargs)+async def build_index(self, resource_uris: List[str], ctx: RequestContext, **kwargs) -> Dict[str, Any]:@@ -118,6 +147,7 @@ class ResourceProcessor:parent: Optional[str] = None,summarize: bool = False,stage_callback: Optional[Callable[[str], Any]] = None,+ prepared_resource: Optional["LocalResource"] = None,**kwargs,) -> Dict[str, Any]:"""@@ -169,6 +199,7 @@ class ResourceProcessor:parse_result = await media_processor.process(source=path,instruction=effective_instruction,+ prepared_resource=prepared_resource,**kwargs,)result["source_path"] = parse_result.source_path or pathdiff --git a/tests/parse/test_directory_parser_routing.py b/tests/parse/test_directory_parser_routing.pyindex 1b94774ba..ca75798cf 100644--- a/tests/parse/test_directory_parser_routing.py+++ b/tests/parse/test_directory_parser_routing.py@@ -46,7 +46,7 @@ from openviking.parse.registry import ParserRegistry@pytest.fixturedef registry() -> ParserRegistry:"""Default registry (no optional parsers like ImageParser)."""- return ParserRegistry(register_optional=False)+ return ParserRegistry()# -- directory tree that covers every parser type ----------------------------diff --git a/tests/parse/test_directory_scan.py b/tests/parse/test_directory_scan.pyindex 02fedef52..2e4e28954 100644--- a/tests/parse/test_directory_scan.py+++ b/tests/parse/test_directory_scan.py@@ -85,7 +85,7 @@ def tmp_with_drafts(tmp_path: Path) -> Path:@pytest.fixturedef registry() -> ParserRegistry:"""Default parser registry (includes markdown, pdf, html, text, etc.)."""- return ParserRegistry(register_optional=False)+ return ParserRegistry()# ---------------------------------------------------------------------------diff --git a/tests/parse/test_feishu_accessor.py b/tests/parse/test_feishu_accessor.pyindex 9059720d4..505f932cf 100644--- a/tests/parse/test_feishu_accessor.py+++ b/tests/parse/test_feishu_accessor.py@@ -1,12 +1,14 @@# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.# SPDX-License-Identifier: AGPL-3.0-"""Tests for FeishuAccessor user token and image handling."""+"""Tests for FeishuAccessor supported types, user token, and image handling."""import asyncioimport sysfrom types import ModuleType, SimpleNamespacefrom unittest.mock import MagicMock+import pytest+from openviking.parse.accessors.feishu_accessor import FeishuAccessor, _title_as_filename@@ -110,6 +112,27 @@ class _FakeListDocumentBlockRequestBuilder:return self._request+class _FakeTypedRequest:+ @staticmethod+ def builder():+ return _FakeTypedRequestBuilder()+++class _FakeTypedRequestBuilder:+ def __init__(self):+ self._request = SimpleNamespace()++ def __getattr__(self, name):+ def _set(value):+ setattr(self._request, name, value)+ return self++ return _set++ def build(self):+ return self._request++def _install_fake_lark_modules(monkeypatch):lark = ModuleType("lark_oapi")lark.BaseRequest = _FakeBaseRequest@@ -117,10 +140,15 @@ def _install_fake_lark_modules(monkeypatch):lark.AccessTokenType = SimpleNamespace(TENANT="tenant", USER="user")docx_v1 = ModuleType("lark_oapi.api.docx.v1")docx_v1.ListDocumentBlockRequest = _FakeListDocumentBlockRequest+ bitable_v1 = ModuleType("lark_oapi.api.bitable.v1")+ bitable_v1.ListAppTableRequest = _FakeTypedRequest+ bitable_v1.ListAppTableFieldRequest = _FakeTypedRequest+ bitable_v1.ListAppTableRecordRequest = _FakeTypedRequestcore_model = ModuleType("lark_oapi.core.model")core_model.RequestOption = _FakeRequestOptionmonkeypatch.setitem(sys.modules, "lark_oapi", lark)monkeypatch.setitem(sys.modules, "lark_oapi.api.docx.v1", docx_v1)+ monkeypatch.setitem(sys.modules, "lark_oapi.api.bitable.v1", bitable_v1)monkeypatch.setitem(sys.modules, "lark_oapi.core.model", core_model)@@ -185,9 +213,7 @@ def test_resolve_image_refs_uses_content_type_extension(monkeypatch):accessor._config = SimpleNamespace(download_images=True)accessor._client = SimpleNamespace(request=request_media)- updated, images = accessor._resolve_image_refs(- ""- )+ updated, images = accessor._resolve_image_refs("")assert updated == ""assert images == {"images/img_token_jpeg.jpg": b"\xff\xd8\xff\xe0jpeg-bytes"}@@ -202,9 +228,7 @@ def test_resolve_image_refs_falls_back_to_byte_magic_extension(monkeypatch):accessor._config = SimpleNamespace(download_images=True)accessor._client = SimpleNamespace(request=request_media)- updated, images = accessor._resolve_image_refs(- ""- )+ updated, images = accessor._resolve_image_refs("")assert updated == ""assert images == {"images/img_token_webp.webp": webp_bytes}@@ -338,6 +362,323 @@ def test_access_writes_downloaded_images_next_to_markdown(monkeypatch):assert not resource.path.parent.exists()+def test_fetch_document_dispatches_all_supported_types(monkeypatch):+ accessor = FeishuAccessor()+ handlers = {+ "_parse_docx": MagicMock(return_value=("docx body", "Doc")),+ "_parse_sheets": MagicMock(return_value=("sheet body", "Sheet")),+ "_parse_bitable": MagicMock(return_value=("base body", "Base")),+ }+ for name, handler in handlers.items():+ monkeypatch.setattr(accessor, name, handler)++ docx = asyncio.run(+ accessor._fetch_document(+ "https://example.feishu.cn/docx/doc_token",+ feishu_access_token="u-test",+ )+ )+ sheets = asyncio.run(accessor._fetch_document("https://example.feishu.cn/sheets/sht_token"))+ base = asyncio.run(accessor._fetch_document("https://example.feishu.cn/base/app_token"))+ monkeypatch.setattr(+ accessor,+ "_resolve_wiki_node",+ MagicMock(return_value=("base", "wiki_app_token", "Wiki Base")),+ )+ wiki = asyncio.run(accessor._fetch_document("https://example.feishu.cn/wiki/wiki_token"))++ assert (docx.doc_type, sheets.doc_type, base.doc_type, wiki.doc_type) == (+ "docx",+ "sheets",+ "base",+ "base",+ )+ assert wiki.title == "Wiki Base"+ handlers["_parse_docx"].assert_called_once_with("doc_token", "u-test")+ handlers["_parse_sheets"].assert_called_once_with("sht_token", None)+ assert handlers["_parse_bitable"].call_args_list[-1].args == ("wiki_app_token", None)+++def test_fetch_document_honors_bitable_table_and_view(monkeypatch):+ _install_fake_lark_modules(monkeypatch)+ list_tables = MagicMock()+ list_fields = MagicMock(+ return_value=_SuccessResponse(+ SimpleNamespace(+ items=[SimpleNamespace(field_name="Status")],+ has_more=False,+ page_token=None,+ )+ )+ )+ list_records = MagicMock(+ return_value=_SuccessResponse(+ SimpleNamespace(+ items=[SimpleNamespace(fields={"Status": "Published"})],+ has_more=False,+ page_token=None,+ )+ )+ )+ accessor = FeishuAccessor()+ accessor._config = SimpleNamespace(max_records_per_table=10)+ accessor._client = SimpleNamespace(+ bitable=SimpleNamespace(+ v1=SimpleNamespace(+ app_table=SimpleNamespace(list=list_tables),+ app_table_field=SimpleNamespace(list=list_fields),+ app_table_record=SimpleNamespace(list=list_records),+ )+ )+ )++ document = asyncio.run(+ accessor._fetch_document(+ "https://example.feishu.cn/base/app_token?table=tblSales&view=vewPublic"+ )+ )++ list_tables.assert_not_called()+ request = list_records.call_args.args[0]+ assert (request.table_id, request.view_id) == ("tblSales", "vewPublic")+ assert document.title == "tblSales (vewPublic)"+ assert document.meta["feishu_table_id"] == "tblSales"+ assert document.meta["feishu_view_id"] == "vewPublic"+++def test_fetch_document_rejects_bitable_view_without_table():+ accessor = FeishuAccessor()++ with pytest.raises(ValueError, match="'view'.*'table'"):+ asyncio.run(+ accessor._fetch_document("https://example.feishu.cn/base/app_token?view=vewPublic")+ )+++def test_parse_sheets_handles_grid_and_embedded_bitable(monkeypatch):+ _install_fake_lark_modules(monkeypatch)+ request = MagicMock(+ side_effect=[+ _FakeMediaResponse(+ b'{"data":{"properties":{"title":"Budget"},"sheets":['+ b'{"sheetId":"sheet-1","title":"Q1","rowCount":3,"columnCount":28},'+ b'{"sheetId":"block-1","title":"Content Calendar","rowCount":0,'+ b'"columnCount":0,"blockInfo":{"blockType":"BITABLE_BLOCK",'+ b'"blockToken":"app-token_table-1"}}]}}'+ ),+ _FakeMediaResponse(b'{"data":{"valueRange":{"values":[["name","amount"],["A",1]]}}}'),+ _FakeMediaResponse(b"\x89PNG\r\n"),+ ]+ )+ list_tables = MagicMock()+ list_fields = MagicMock(+ return_value=_SuccessResponse(+ SimpleNamespace(+ items=[+ SimpleNamespace(field_name="Topic"),+ SimpleNamespace(field_name="Cover"),+ SimpleNamespace(field_name="Brief"),+ ],+ has_more=False,+ page_token=None,+ )+ )+ )+ list_records = MagicMock(+ return_value=_SuccessResponse(+ SimpleNamespace(+ items=[+ SimpleNamespace(+ fields={+ "Topic": "Welcome",+ "Cover": [+ {+ "file_token": "cover-token",+ "name": "cover.png",+ "type": "image/png",+ }+ ],+ "Brief": [+ {+ "file_token": "brief-token",+ "name": "brief.pdf",+ "type": "application/pdf",+ }+ ],+ }+ )+ ],+ has_more=False,+ page_token=None,+ )+ )+ )+ accessor = FeishuAccessor()+ accessor._config = SimpleNamespace(+ max_rows_per_sheet=2,+ max_records_per_table=10,+ download_images=True,+ )+ accessor._user_token_client = SimpleNamespace(+ request=request,+ bitable=SimpleNamespace(+ v1=SimpleNamespace(+ app_table=SimpleNamespace(list=list_tables),+ app_table_field=SimpleNamespace(list=list_fields),+ app_table_record=SimpleNamespace(list=list_records),+ )+ ),+ )++ markdown, title = accessor._parse_sheets("sht_token", "u-test")++ assert title == "Budget"+ assert "| name | amount |" in markdown+ assert "1 more rows truncated" in markdown+ assert "2 columns after Z omitted" in markdown+ assert "### Content Calendar" in markdown+ assert "Welcome" in markdown+ assert "" in markdown+ assert "brief.pdf" in markdown+ assert "feishu://image/brief-token" not in markdown+ assert "Empty sheet" not in markdown+ assert list_tables.call_count == 0+ assert list_fields.call_args.args[0].table_id == "table-1"++ resolved, images = accessor._resolve_image_refs(+ markdown,+ feishu_access_token="u-test",+ )++ assert "" in resolved+ assert images == {"images/cover-token.png": b"\x89PNG\r\n"}+ assert request.call_args_list[-1].args[0].uri == (+ "/open-apis/drive/v1/medias/cover-token/download"+ )+ assert all(call.args[0].token_types == {"user"} for call in request.call_args_list)+ assert all(call.args[1].user_access_token == "u-test" for call in request.call_args_list)+++def test_parse_bitable_uses_user_token_and_formats_records(monkeypatch):+ _install_fake_lark_modules(monkeypatch)+ list_tables = MagicMock(+ side_effect=[+ _SuccessResponse(+ SimpleNamespace(+ items=[SimpleNamespace(table_id="table-1", name="Leads")],+ has_more=True,+ page_token="tables-2",+ )+ ),+ _SuccessResponse(+ SimpleNamespace(+ items=[SimpleNamespace(table_id="table-2", name="Companies")],+ has_more=False,+ page_token=None,+ )+ ),+ ]+ )+ list_fields = MagicMock(+ side_effect=[+ _SuccessResponse(+ SimpleNamespace(+ items=[SimpleNamespace(field_name="Owner")],+ has_more=True,+ page_token="fields-2",+ )+ ),+ _SuccessResponse(+ SimpleNamespace(+ items=[SimpleNamespace(field_name="Status")],+ has_more=False,+ page_token=None,+ )+ ),+ _SuccessResponse(+ SimpleNamespace(+ items=[SimpleNamespace(field_name="Name")],+ has_more=False,+ page_token=None,+ )+ ),+ ]+ )+ list_records = MagicMock(+ side_effect=[+ _SuccessResponse(+ SimpleNamespace(+ items=[SimpleNamespace(fields={"Owner": [{"name": "Alice"}], "Status": "New"})],+ has_more=False,+ page_token=None,+ )+ ),+ _SuccessResponse(+ SimpleNamespace(+ items=[SimpleNamespace(fields={"Name": "Acme"})],+ has_more=False,+ page_token=None,+ )+ ),+ ]+ )+ accessor = FeishuAccessor()+ accessor._config = SimpleNamespace(max_records_per_table=10)+ accessor._user_token_client = SimpleNamespace(+ bitable=SimpleNamespace(+ v1=SimpleNamespace(+ app_table=SimpleNamespace(list=list_tables),+ app_table_field=SimpleNamespace(list=list_fields),+ app_table_record=SimpleNamespace(list=list_records),+ )+ )+ )++ markdown, title = accessor._parse_bitable("app_token", "u-test")++ assert title == "Bitable (2 tables)"+ assert "## Leads" in markdown+ assert "## Companies" in markdown+ assert "Alice" in markdown+ assert "Acme" in markdown+ assert "records truncated" not in markdown+ assert list_tables.call_args_list[1].args[0].page_token == "tables-2"+ assert list_fields.call_args_list[1].args[0].page_token == "fields-2"+ assert all(call.args[1].user_access_token == "u-test" for call in list_records.call_args_list)+++def test_embedded_sheet_uses_same_user_token(monkeypatch):+ _install_fake_lark_modules(monkeypatch)+ inspect_block = MagicMock(+ return_value=_FakeMediaResponse(+ b'{"data":{"block":{"sheet":{"token":"spreadsheet-1_sheet-1"}}}}'+ )+ )+ accessor = FeishuAccessor()+ accessor._user_token_client = SimpleNamespace(request=inspect_block)+ read_range = MagicMock(return_value=[["name", "amount"], ["A", "1"]])+ monkeypatch.setattr(accessor, "_read_sheet_range", read_range)+ block = SimpleNamespace(+ block_id="block-1",+ block_type=30,+ parent_id="doc-1",+ sheet=SimpleNamespace(),+ )++ markdown = accessor._block_to_markdown(+ block,+ {},+ {},+ document_id="doc-1",+ feishu_access_token="u-test",+ )++ assert "| name | amount |" in markdown+ assert inspect_block.call_args.args[0].token_types == {"user"}+ assert inspect_block.call_args.args[1].user_access_token == "u-test"+ assert read_range.call_args.kwargs["feishu_access_token"] == "u-test"++def test_access_keeps_raw_title_but_exposes_safe_original_filename(monkeypatch):accessor = FeishuAccessor()accessor._config = SimpleNamespace(download_images=False)diff --git a/tests/parse/test_feishu_parser.py b/tests/parse/test_feishu_parser.pydeleted file mode 100644index 24272fdfb..000000000--- a/tests/parse/test_feishu_parser.py+++ /dev/null@@ -1,649 +0,0 @@-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.-# SPDX-License-Identifier: AGPL-3.0-"""Tests for FeishuParser."""--import asyncio-from types import SimpleNamespace-from unittest.mock import MagicMock, patch--import pytest--from openviking.parse.parsers.feishu import FeishuParser---def _make_block(**kwargs):- """Create a mock block object with only the specified attributes populated."""- # Start with all common attributes as None- defaults = {- "block_id": "test_id",- "block_type": 0,- "parent_id": "parent_id",- "children": None,- "comment_ids": None,- "add_ons": None,- "page": None,- "text": None,- "heading1": None,- "heading2": None,- "heading3": None,- "heading4": None,- "heading5": None,- "heading6": None,- "heading7": None,- "heading8": None,- "heading9": None,- "bullet": None,- "ordered": None,- "code": None,- "quote": None,- "todo": None,- "divider": None,- "image": None,- "table": None,- "table_cell": None,- "quote_container": None,- "sheet": None,- "callout": None,- "equation": None,- "task": None,- }- defaults.update(kwargs)- return SimpleNamespace(**defaults)---def _make_text_content(- text: str, bold=False, italic=False, inline_code=False, strikethrough=False, link_url=None-):- """Create a mock text content object with elements."""- style = SimpleNamespace(- bold=bold,- italic=italic,- inline_code=inline_code,- strikethrough=strikethrough,- link=SimpleNamespace(url=link_url) if link_url else None,- )- element = SimpleNamespace(- text_run=SimpleNamespace(content=text, text_element_style=style),- mention_user=None,- mention_doc=None,- equation=None,- )- return SimpleNamespace(elements=[element], style=None)---class TestParseFeishuUrl:- def test_docx_url(self):- doc_type, token = FeishuParser._parse_feishu_url(- "https://example.feishu.cn/docx/doxcnABC123"- )- assert doc_type == "docx"- assert token == "doxcnABC123"-- def test_wiki_url(self):- doc_type, token = FeishuParser._parse_feishu_url("https://example.feishu.cn/wiki/wikiXYZ")- assert doc_type == "wiki"- assert token == "wikiXYZ"-- def test_sheets_url(self):- doc_type, token = FeishuParser._parse_feishu_url(- "https://example.feishu.cn/sheets/shtcn123"- )- assert doc_type == "sheets"-- def test_base_url(self):- doc_type, token = FeishuParser._parse_feishu_url(- "https://example.feishu.cn/base/bascn999?table=tbl1"- )- assert doc_type == "base"- assert token == "bascn999"-- def test_larksuite_url(self):- doc_type, token = FeishuParser._parse_feishu_url(- "https://example.larksuite.com/docx/abc123"- )- assert doc_type == "docx"-- def test_invalid_url_raises(self):- with pytest.raises(ValueError, match="Cannot parse"):- FeishuParser._parse_feishu_url("https://example.feishu.cn/")-- def test_invalid_host_raises(self):- with pytest.raises(ValueError, match="host not allowed"):- FeishuParser._parse_feishu_url("https://evil.example/docx/doxcnABC123")---class TestBlockToMarkdown:- """Test attribute-driven block-to-markdown conversion."""-- def setup_method(self):- self.parser = FeishuParser()-- def test_text_block(self):- block = _make_block(text=_make_text_content("Hello world"))- result = self.parser._block_to_markdown(block, {}, {})- assert result == "Hello world"-- def test_heading_blocks(self):- for level in range(1, 7):- content = _make_text_content(f"Heading {level}")- block = _make_block(**{f"heading{level}": content})- result = self.parser._block_to_markdown(block, {}, {})- assert result == f"{'#' * level} Heading {level}"-- def test_bullet_list(self):- block = _make_block(bullet=_make_text_content("Item one"))- result = self.parser._block_to_markdown(block, {}, {})- assert result == "- Item one"-- def test_ordered_list(self):- counter: dict = {}- block = _make_block(ordered=_make_text_content("First"))- result = self.parser._block_to_markdown(block, {}, counter)- assert result == "1. First"-- block2 = _make_block(ordered=_make_text_content("Second"))- result2 = self.parser._block_to_markdown(block2, {}, counter)- assert result2 == "2. Second"-- def test_code_block(self):- code_content = _make_text_content("print('hello')")- code_content.style = SimpleNamespace(language="python")- block = _make_block(code=code_content)- result = self.parser._block_to_markdown(block, {}, {})- assert result == "```python\nprint('hello')\n```"-- def test_quote_block(self):- block = _make_block(quote=_make_text_content("A quote"))- result = self.parser._block_to_markdown(block, {}, {})- assert result == "> A quote"-- def test_todo_block(self):- todo_content = _make_text_content("Buy milk")- todo_content.style = SimpleNamespace(done=True)- block = _make_block(todo=todo_content)- result = self.parser._block_to_markdown(block, {}, {})- assert result == "- [x] Buy milk"-- def test_divider_block(self):- block = _make_block(divider=SimpleNamespace())- result = self.parser._block_to_markdown(block, {}, {})- assert result == "---"-- def test_image_block(self):- block = _make_block(image=SimpleNamespace(token="img_token_123", alt="screenshot"))- result = self.parser._block_to_markdown(block, {}, {})- assert result == ""-- def test_skip_page(self):- block = _make_block(page=SimpleNamespace(elements=[]))- result = self.parser._block_to_markdown(block, {}, {})- assert result is None # page is in _SKIP_ATTRS-- def test_skip_table_cell(self):- block = _make_block(table_cell=SimpleNamespace())- result = self.parser._block_to_markdown(block, {}, {})- assert result is None-- def test_unknown_with_elements_extracts_text(self):- """Unknown block type with text elements should still extract content."""- block = _make_block(callout=_make_text_content("Important note"))- result = self.parser._block_to_markdown(block, {}, {})- assert result == "Important note"---class TestApplyTextStyle:- def test_bold(self):- assert (- FeishuParser._apply_text_style(- "text",- SimpleNamespace(- bold=True, italic=False, strikethrough=False, inline_code=False, link=None- ),- )- == "**text**"- )-- def test_italic(self):- assert (- FeishuParser._apply_text_style(- "text",- SimpleNamespace(- bold=False, italic=True, strikethrough=False, inline_code=False, link=None- ),- )- == "*text*"- )-- def test_inline_code(self):- assert (- FeishuParser._apply_text_style(- "code",- SimpleNamespace(- bold=False, italic=False, strikethrough=False, inline_code=True, link=None- ),- )- == "`code`"- )-- def test_link(self):- result = FeishuParser._apply_text_style(- "click",- SimpleNamespace(- bold=False,- italic=False,- strikethrough=False,- inline_code=False,- link=SimpleNamespace(url="https://example.com"),- ),- )- assert result == "[click](https://example.com)"-- def test_empty_text(self):- assert FeishuParser._apply_text_style("", SimpleNamespace(bold=True)) == ""-- def test_none_style(self):- assert FeishuParser._apply_text_style("text", None) == "text"---class TestFormatBitableField:- def test_none(self):- assert FeishuParser._format_bitable_field(None) == ""-- def test_string(self):- assert FeishuParser._format_bitable_field("hello") == "hello"-- def test_number(self):- assert FeishuParser._format_bitable_field(42) == "42"-- def test_list_of_dicts(self):- result = FeishuParser._format_bitable_field([{"text": "A"}, {"name": "B"}])- assert result == "A, B"-- def test_dict_with_text(self):- assert FeishuParser._format_bitable_field({"text": "value"}) == "value"---class TestTrimEmptyColumns:- def test_trim(self):- rows = [["a", "b", "", ""], ["c", "d", "", ""]]- result = FeishuParser._trim_empty_columns(rows)- assert result == [["a", "b"], ["c", "d"]]-- def test_no_trim_needed(self):- rows = [["a", "b"], ["c", "d"]]- assert FeishuParser._trim_empty_columns(rows) == rows-- def test_all_empty(self):- rows = [["", ""], ["", ""]]- assert FeishuParser._trim_empty_columns(rows) == []---# ========== Mock Integration Tests ==========---def _mock_list_blocks_response(blocks, has_more=False, page_token=None):- """Create a mock response for docx.v1.document_block.list()."""- resp = MagicMock()- resp.success.return_value = True- resp.data.items = blocks- resp.data.has_more = has_more- resp.data.page_token = page_token- return resp---def _make_sdk_block(block_id, parent_id="doc_id", **attrs):- """Create a mock SDK block with all attributes defaulting to None."""- defaults = {- "block_id": block_id,- "block_type": 0,- "parent_id": parent_id,- "children": None,- "comment_ids": None,- "add_ons": None,- "page": None,- "text": None,- "heading1": None,- "heading2": None,- "heading3": None,- "heading4": None,- "heading5": None,- "heading6": None,- "heading7": None,- "heading8": None,- "heading9": None,- "bullet": None,- "ordered": None,- "code": None,- "quote": None,- "todo": None,- "divider": None,- "image": None,- "table": None,- "table_cell": None,- "quote_container": None,- "sheet": None,- "callout": None,- "equation": None,- "task": None,- }- defaults.update(attrs)- return SimpleNamespace(**defaults)---class TestParseDocxIntegration:- """Integration tests for _parse_docx with mocked lark-oapi client."""-- def _make_parser_with_mock_client(self, list_response):- """Create a FeishuParser with a mocked lark-oapi client."""- parser = FeishuParser()- mock_client = MagicMock()- mock_client.docx.v1.document_block.list.return_value = list_response- parser._client = mock_client- return parser-- def test_parse_docx_basic(self):- """Test basic document with page title, heading, and text."""- blocks = [- _make_sdk_block(- "page_id",- page=SimpleNamespace(- elements=[- SimpleNamespace(- text_run=SimpleNamespace(- content="My Document", text_element_style=None- ),- mention_user=None,- mention_doc=None,- equation=None,- )- ]- ),- ),- _make_sdk_block("h1_id", heading2=_make_text_content("Introduction")),- _make_sdk_block("t1_id", text=_make_text_content("Hello world")),- ]- response = _mock_list_blocks_response(blocks)- parser = self._make_parser_with_mock_client(response)-- markdown, title = parser._parse_docx("test_doc_id")-- assert title == "My Document"- assert "# My Document" in markdown- assert "## Introduction" in markdown- assert "Hello world" in markdown-- def test_parse_docx_with_pagination(self):- """Test document fetching with multiple pages of blocks."""- page1_blocks = [- _make_sdk_block(- "page_id",- page=SimpleNamespace(- elements=[- SimpleNamespace(- text_run=SimpleNamespace(- content="Paginated Doc", text_element_style=None- ),- mention_user=None,- mention_doc=None,- equation=None,- )- ]- ),- ),- _make_sdk_block("t1_id", text=_make_text_content("Page 1 content")),- ]- page2_blocks = [- _make_sdk_block("t2_id", text=_make_text_content("Page 2 content")),- ]-- resp1 = _mock_list_blocks_response(page1_blocks, has_more=True, page_token="token2")- resp2 = _mock_list_blocks_response(page2_blocks, has_more=False)-- parser = FeishuParser()- mock_client = MagicMock()- mock_client.docx.v1.document_block.list.side_effect = [resp1, resp2]- parser._client = mock_client-- markdown, title = parser._parse_docx("test_doc_id")-- assert title == "Paginated Doc"- assert "Page 1 content" in markdown- assert "Page 2 content" in markdown- assert mock_client.docx.v1.document_block.list.call_count == 2-- def test_parse_docx_empty(self):- """Test empty document."""- response = _mock_list_blocks_response([])- parser = self._make_parser_with_mock_client(response)-- markdown, title = parser._parse_docx("empty_doc")-- assert markdown == ""- assert title == "Untitled"-- def test_parse_docx_api_error(self):- """Test handling of API error."""- parser = FeishuParser()- mock_client = MagicMock()- error_resp = MagicMock()- error_resp.success.return_value = False- error_resp.code = 403- error_resp.msg = "permission denied"- mock_client.docx.v1.document_block.list.return_value = error_resp- parser._client = mock_client-- with pytest.raises(RuntimeError, match="permission denied"):- parser._parse_docx("forbidden_doc")-- def test_parse_docx_mixed_blocks(self):- """Test document with various block types."""- blocks = [- _make_sdk_block(- "page_id",- page=SimpleNamespace(- elements=[- SimpleNamespace(- text_run=SimpleNamespace(content="Report", text_element_style=None),- mention_user=None,- mention_doc=None,- equation=None,- )- ]- ),- ),- _make_sdk_block("h_id", heading2=_make_text_content("Section 1")),- _make_sdk_block("b1_id", bullet=_make_text_content("Item A")),- _make_sdk_block("b2_id", bullet=_make_text_content("Item B")),- _make_sdk_block("div_id", divider=SimpleNamespace()),- _make_sdk_block("q_id", quote=_make_text_content("A wise quote")),- _make_sdk_block(- "code_id",- code=SimpleNamespace(- elements=[- SimpleNamespace(- text_run=SimpleNamespace(content="x = 1", text_element_style=None),- mention_user=None,- mention_doc=None,- equation=None,- )- ],- style=SimpleNamespace(language="python"),- ),- ),- ]- response = _mock_list_blocks_response(blocks)- parser = self._make_parser_with_mock_client(response)-- markdown, title = parser._parse_docx("test_doc")-- assert title == "Report"- assert "## Section 1" in markdown- assert "- Item A" in markdown- assert "- Item B" in markdown- assert "---" in markdown- assert "> A wise quote" in markdown- assert "```python\nx = 1\n```" in markdown-- def test_parse_docx_ordered_list_reset(self):- """Test that ordered list counters reset between separate lists."""- blocks = [- _make_sdk_block("page_id", page=SimpleNamespace(elements=[])),- _make_sdk_block("o1", ordered=_make_text_content("First")),- _make_sdk_block("o2", ordered=_make_text_content("Second")),- _make_sdk_block(- "t1", text=_make_text_content("Break")- ), # Non-ordered block resets counter- _make_sdk_block("o3", ordered=_make_text_content("New first")),- ]- response = _mock_list_blocks_response(blocks)- parser = self._make_parser_with_mock_client(response)-- markdown, _ = parser._parse_docx("test_doc")-- assert "1. First" in markdown- assert "2. Second" in markdown- assert "1. New first" in markdown---class TestParseAsyncIntegration:- """Test the async parse() entry point with mocked internals."""-- def test_parse_docx_url(self):- """Test full parse() flow with a docx URL."""- parser = FeishuParser()-- blocks = [- _make_sdk_block(- "page_id",- page=SimpleNamespace(- elements=[- SimpleNamespace(- text_run=SimpleNamespace(content="Test Doc", text_element_style=None),- mention_user=None,- mention_doc=None,- equation=None,- )- ]- ),- ),- _make_sdk_block("t1", text=_make_text_content("Content here")),- ]- mock_client = MagicMock()- mock_client.docx.v1.document_block.list.return_value = _mock_list_blocks_response(blocks)- parser._client = mock_client-- # Mock MarkdownParser to avoid VikingFS dependency- mock_md_result = MagicMock()- mock_md_result.source_format = "markdown"- mock_md_result.parser_name = "MarkdownParser"- mock_md_result.parse_time = 0.1- mock_md_result.meta = {}-- mock_md_instance = MagicMock()-- async def _mock_parse_content(*a, **kw):- return mock_md_result-- mock_md_instance.parse_content = _mock_parse_content-- with patch.object(parser, "_get_markdown_parser", return_value=mock_md_instance):- result = asyncio.get_event_loop().run_until_complete(- parser.parse("https://example.feishu.cn/docx/test123")- )-- assert result.source_format == "feishu_docx"- assert result.parser_name == "FeishuParser"- assert result.meta["feishu_doc_type"] == "docx"- assert result.meta["feishu_token"] == "test123"-- def test_parse_wiki_url_resolves(self):- """Test that wiki URLs are resolved to underlying document type."""- parser = FeishuParser()-- # Mock wiki resolution- mock_client = MagicMock()- # Use "doc" (not "docx") to test _WIKI_TYPE_MAP normalization: "doc" -> "docx"- wiki_node = SimpleNamespace(obj_type="doc", obj_token="real_token", title="Wiki Page")- wiki_resp = MagicMock()- wiki_resp.success.return_value = True- wiki_resp.data.node = wiki_node- mock_client.wiki.v2.space.get_node.return_value = wiki_resp-- # Mock docx blocks- blocks = [- _make_sdk_block("page_id", page=SimpleNamespace(elements=[])),- _make_sdk_block("t1", text=_make_text_content("Wiki content")),- ]- mock_client.docx.v1.document_block.list.return_value = _mock_list_blocks_response(blocks)- parser._client = mock_client-- mock_md_result = MagicMock()- mock_md_result.source_format = "markdown"- mock_md_result.parser_name = "MarkdownParser"- mock_md_result.parse_time = 0.1- mock_md_result.meta = {}-- mock_md_instance = MagicMock()-- async def _mock_parse_content(*a, **kw):- return mock_md_result-- mock_md_instance.parse_content = _mock_parse_content-- with patch.object(parser, "_get_markdown_parser", return_value=mock_md_instance):- result = asyncio.get_event_loop().run_until_complete(- parser.parse("https://example.feishu.cn/wiki/wiki_token")- )-- assert result.source_format == "feishu_docx"- assert result.meta["feishu_doc_type"] == "docx"- assert result.meta["feishu_token"] == "real_token"-- def test_parse_content_routes_larkoffice_url(self):- """Test that parse_content routes larkoffice.com URLs to FeishuParser.parse() correctly."""- parser = FeishuParser()-- # Mock the underlying parse() method to avoid network/SDK calls- mock_result = MagicMock()- mock_result.source_format = "feishu_docx"- mock_result.meta = {"feishu_token": "larkoffice123"}-- async def _mock_parse(*a, **kw):- return mock_result-- parser.parse = _mock_parse-- result = asyncio.get_event_loop().run_until_complete(- parser.parse_content(- content="", source_path="https://bytedance.larkoffice.com/wiki/larkoffice123"- )- )-- assert result.source_format == "feishu_docx"- assert result.meta["feishu_token"] == "larkoffice123"-- def test_parse_unsupported_type(self):- """Test that unsupported document types return error ParseResult."""- parser = FeishuParser()- parser._client = MagicMock() # Won't be called-- result = asyncio.get_event_loop().run_until_complete(- parser.parse("https://example.feishu.cn/mindnote/abc123")- )-- assert result.warnings- assert "Unsupported" in result.warnings[0]-- def test_parse_rejects_invalid_host(self):- """Main parse() entry point must reject non-Feishu hosts."""- parser = FeishuParser()-- result = asyncio.get_event_loop().run_until_complete(- parser.parse("https://evil.example/docx/doxcn123")- )-- assert result.source_format == "feishu"- assert result.warnings- assert "host not allowed" in result.warnings[0]- assert "feishu_doc_type" not in result.meta- assert "feishu_token" not in result.metadiff --git a/tests/parse/test_feishu_parser_api.py b/tests/parse/test_feishu_parser_api.pyindex 1de56bf4c..039b9b679 100644--- a/tests/parse/test_feishu_parser_api.py+++ b/tests/parse/test_feishu_parser_api.py@@ -7,6 +7,7 @@ from unittest.mock import AsyncMock, Mockimport pytest+from openviking.parse.accessors.base import LocalResource, SourceTypefrom openviking.parse.understanding_api import PREPARED_RESPONSE_ID_ARG, UnderstandingAPIfrom openviking.server.identity import RequestContext, Rolefrom openviking.service.resource_service import ResourceService@@ -23,7 +24,7 @@ from openviking_cli.session.user_id import UserIdentifierasync def test_feishu_parser_api_bypasses_accessor():result = object()router = SimpleNamespace(- should_use_understanding_api=lambda source: True,+ should_use_understanding_directly=Mock(return_value=True),parse=AsyncMock(return_value=result),)processor = UnifiedResourceProcessor(vlm_processor=object())@@ -39,36 +40,22 @@ async def test_feishu_parser_api_bypasses_accessor():router.parse.assert_awaited_once()(call_source,) = router.parse.await_args.argsassert call_source == source- assert router.parse.await_args.kwargs["lark_file"] == {"user_access_token": "u-test"}+ assert router.parse.await_args.kwargs["feishu_access_token"] == " u-test "assert "resource_name" not in router.parse.await_args.kwargs@pytest.mark.asyncioasync def test_feishu_parser_api_uses_app_credentials_for_tenant_token(monkeypatch):- result = object()- router = SimpleNamespace(- should_use_understanding_api=lambda source: True,- parse=AsyncMock(return_value=result),- )oauth_client = SimpleNamespace(get_tenant_access_token=AsyncMock(return_value="t-test"))- processor = UnifiedResourceProcessor(vlm_processor=object())- processor._parser_router = router- processor._accessor_registry = SimpleNamespace(- access=AsyncMock(side_effect=AssertionError("accessor should not be called"))- )- monkeypatch.setattr(- "openviking.resource.feishu_watch_auth.load_feishu_app_credentials",- Mock(return_value=object()),- )monkeypatch.setattr("openviking.resource.feishu_watch_auth.FeishuOAuthClient.from_config",Mock(return_value=oauth_client),)+ api = _understanding_api_for_parse()- actual = await processor.process("https://example.larkoffice.com/docx/doxcnToken")+ auth = await api._resolve_lark_file({})- assert actual is result- assert router.parse.await_args.kwargs["lark_file"] == {"tenant_access_token": "t-test"}+ assert auth == {"tenant_access_token": "t-test"}oauth_client.get_tenant_access_token.assert_awaited_once_with()@@ -76,7 +63,9 @@ async def test_feishu_parser_api_uses_app_credentials_for_tenant_token(monkeypatasync def test_prepared_feishu_response_bypasses_accessor_without_credentials(monkeypatch):result = object()router = SimpleNamespace(- should_use_understanding_api=lambda source: True,+ should_use_understanding_api=Mock(+ side_effect=AssertionError("prepared response must not re-evaluate routing config")+ ),parse=AsyncMock(return_value=result),)processor = UnifiedResourceProcessor(vlm_processor=object())@@ -96,6 +85,73 @@ async def test_prepared_feishu_response_bypasses_accessor_without_credentials(moassert actual is resultassert router.parse.await_args.kwargs[PREPARED_RESPONSE_ID_ARG] == "response-1"+ assert router.parse.await_args.kwargs["parser_backend"] == "understanding"+ assert "lark_file" not in router.parse.await_args.kwargs+++async def test_feishu_without_understanding_auth_falls_back_to_accessor(monkeypatch, tmp_path):+ markdown_path = tmp_path / "document.md"+ markdown_path.write_text("content", encoding="utf-8")+ resource = LocalResource(+ path=markdown_path,+ source_type=SourceType.FEISHU,+ original_source="https://example.larkoffice.com/docx/doxcnToken",+ meta={"extension": ".md"},+ is_temporary=False,+ )+ result = object()+ router = SimpleNamespace(+ should_use_understanding_directly=Mock(return_value=False),+ parse=AsyncMock(return_value=result),+ )+ accessor = SimpleNamespace(access=AsyncMock(return_value=resource))+ processor = UnifiedResourceProcessor(vlm_processor=object())+ processor._parser_router = router+ processor._accessor_registry = accessor+ monkeypatch.setattr(+ "openviking.resource.feishu_watch_auth.load_feishu_app_credentials",+ Mock(side_effect=ValueError("missing credentials")),+ )++ actual = await processor.process(resource.original_source)++ assert actual is result+ accessor.access.assert_awaited_once_with(resource.original_source)+ router.parse.assert_awaited_once()+ assert router.parse.await_args.args[0] is resource+ assert "lark_file" not in router.parse.await_args.kwargs+++async def test_prepared_feishu_resource_is_not_bypassed(monkeypatch, tmp_path):+ markdown_path = tmp_path / "document.md"+ markdown_path.write_text("content", encoding="utf-8")+ resource = LocalResource(+ path=markdown_path,+ source_type=SourceType.FEISHU,+ original_source="https://example.larkoffice.com/docx/doxcnToken",+ meta={"resolved_extension": ".md"},+ is_temporary=False,+ )+ result = object()+ router = SimpleNamespace(+ should_use_understanding_api=Mock(+ side_effect=AssertionError("prepared resources must not be routed before parsing")+ ),+ parse=AsyncMock(return_value=result),+ )+ processor = UnifiedResourceProcessor(vlm_processor=object())+ processor._parser_router = router++ actual = await processor.process(+ resource.original_source,+ prepared_resource=resource,+ feishu_access_token="u-test",+ )++ assert actual is result+ assert router.parse.await_args.args[0] is resourceassert "lark_file" not in router.parse.await_args.kwargs@@ -217,7 +273,11 @@ async def test_legacy_accessor_output_does_not_enable_lark_protocol(tmp_path: Pazf.writestr("parsed/0.md", "content")api = _understanding_api_for_parse()- api._create_response_for_url = AsyncMock(return_value={"id": "response-1"})+ api._create_file = AsyncMock(return_value={"id": "file-1"})+ api._create_response_for_file = AsyncMock(return_value={"id": "response-1"})+ api._create_response_for_url = AsyncMock(+ side_effect=AssertionError("accessor output must stay local")+ )api._poll_response = AsyncMock(return_value={"status": "completed", "result": {"zip_url": "https://tos/result.zip"}})@@ -230,11 +290,9 @@ async def test_legacy_accessor_output_does_not_enable_lark_protocol(tmp_path: Pafeishu_access_token="u-test",)- api._create_response_for_url.assert_awaited_once_with(- url="https://example.larkoffice.com/docx/doxcnToken",- doc_type="unknown",- lark_file=None,- )+ api._create_file.assert_awaited_once_with(local_path=markdown_path)+ api._create_response_for_file.assert_awaited_once_with(file_id="file-1")+ api._create_response_for_url.assert_not_awaited()@pytest.mark.asyncio@@ -336,11 +394,10 @@ def test_add_resource_message_round_trips_internal_fields():async def test_uat_producer_payload_reaches_worker_without_persisting_token(monkeypatch):source = "https://example.larkoffice.com/docx/doxcnToken"root_uri = "viking://resources/lark/doxcnToken"- parser_router = SimpleNamespace(- should_use_understanding_api=lambda _source: True,- submit_url=AsyncMock(return_value="response-1"),- )+ submit_understanding = AsyncMock(return_value="response-1")resource_processor = SimpleNamespace(+ should_use_understanding_directly=lambda _source, **_kwargs: True,+ submit_understanding=submit_understanding,tree_builder=SimpleNamespace(resolve_target_uri=AsyncMock(return_value=(root_uri, root_uri))),@@ -377,14 +434,12 @@ async def test_uat_producer_payload_reaches_worker_without_persisting_token(monkresource_processor=resource_processor,skill_processor=SimpleNamespace(),)- service._parser_router = parser_routermonkeypatch.setattr(service, "_should_use_connector", Mock(return_value=False))monkeypatch.setattr("openviking.service.resource_service.is_git_repo_url",Mock(return_value=False),)monkeypatch.setattr("openviking.service.resource_service.uuid4", Mock(return_value="task-1"))- monkeypatch.setattr(service, "_is_feishu_url", Mock(return_value=True))ctx = RequestContext(user=UserIdentifier("account-1", "user-1"),role=Role.USER,@@ -401,18 +456,22 @@ async def test_uat_producer_payload_reaches_worker_without_persisting_token(monkassert initial_result == {"status": "success", "task_id": "task-1"}assert task_tracker.create.await_args.kwargs["resource_id"] is None- parser_router.submit_url.assert_awaited_once_with(+ submit_understanding.assert_awaited_once_with(source,feishu_access_token="u-secret",+ custom_option="forwarded",)payload = queue_manager.enqueue.await_args.args[1]assert queue_manager.enqueue.await_args.args[0] == QueueManager.EXTERNAL_PARSEassert "u-secret" not in json.dumps(payload)assert payload["understanding_response_id"] == "response-1"- assert payload["args"] == {"custom_option": "forwarded"}+ assert payload["args"] == {+ "custom_option": "forwarded",+ "parser_backend": "understanding",+ }queued_msg = AddResourceMsg.from_dict(payload)- service.add_resource = AsyncMock(+ service._execute_resource_ingestion = AsyncMock(return_value={"status": "success","root_uri": "viking://resources/lark/真实文档标题",@@ -425,9 +484,10 @@ async def test_uat_producer_payload_reaches_worker_without_persisting_token(monkstage_callback=AsyncMock(),)- call = service.add_resource.await_args+ call = service._execute_resource_ingestion.await_argsassert call.kwargs[PREPARED_RESPONSE_ID_ARG] == "response-1"- assert call.kwargs["args"] == {"custom_option": "forwarded"}+ assert call.kwargs["custom_option"] == "forwarded"+ assert call.kwargs["parser_backend"] == "understanding"@pytest.mark.asyncio@@ -450,11 +510,8 @@ async def test_local_prepared_job_uses_add_resource_queue(monkeypatch):resource_processor=resource_processor,skill_processor=SimpleNamespace(),)- service._enqueue_add_resource_job = AsyncMock(- return_value=SimpleNamespace(task_id="task-1")- )+ service._enqueue_add_resource_job = AsyncMock(return_value=SimpleNamespace(task_id="task-1"))monkeypatch.setattr(service, "_should_use_connector", Mock(return_value=False))- monkeypatch.setattr(service, "_should_use_understanding_api", Mock(return_value=False))monkeypatch.setattr("openviking.service.resource_service.is_git_repo_url",Mock(return_value=False),@@ -495,14 +552,10 @@ async def test_uat_producer_cancellation_respects_queue_ownership(else:handoff.side_effect = asyncio.CancelledError- parser_router = SimpleNamespace(- should_use_understanding_api=lambda _source: True,- submit_url=submit_url,- )resource_processor = SimpleNamespace(- tree_builder=SimpleNamespace(- resolve_target_uri=AsyncMock(return_value=(root_uri, None))- ),+ should_use_understanding_directly=lambda _source, **_kwargs: True,+ submit_understanding=submit_url,+ tree_builder=SimpleNamespace(resolve_target_uri=AsyncMock(return_value=(root_uri, None))),)task_tracker = SimpleNamespace(create=AsyncMock(return_value=SimpleNamespace(task_id="task-1")),@@ -546,13 +599,11 @@ async def test_uat_producer_cancellation_respects_queue_ownership(resource_processor=resource_processor,skill_processor=SimpleNamespace(),)- service._parser_router = parser_routermonkeypatch.setattr(service, "_should_use_connector", Mock(return_value=False))monkeypatch.setattr("openviking.service.resource_service.is_git_repo_url",Mock(return_value=False),)- monkeypatch.setattr(service, "_is_feishu_url", Mock(return_value=True))ctx = RequestContext(user=UserIdentifier("account-1", "user-1"),role=Role.USER,@@ -575,14 +626,23 @@ async def test_uat_producer_cancellation_respects_queue_ownership(enqueue.assert_not_awaited()else:enqueue.assert_awaited_once()---def test_prepared_response_id_is_reserved_from_public_args():+ assert enqueue.await_args.args[0] == QueueManager.EXTERNAL_PARSE+++ ("field", "value"),+ [+ (PREPARED_RESPONSE_ID_ARG, "response-1"),+ ("parser_backend", "understanding"),+ ("resolved_extension", ".pdf"),+ ],+)+def test_internal_parser_fields_are_reserved_from_public_args(field, value):service = ResourceService()- with pytest.raises(InvalidArgumentError, match=PREPARED_RESPONSE_ID_ARG):+ with pytest.raises(InvalidArgumentError, match=field):service._normalize_add_resource_args(- {PREPARED_RESPONSE_ID_ARG: "response-1"},+ {field: value},watch_interval=0,)@@ -590,7 +650,7 @@ def test_prepared_response_id_is_reserved_from_public_args():@pytest.mark.asyncioasync def test_add_resource_job_defers_target_and_expands_prepared_response():service = ResourceService()- service.add_resource = AsyncMock(+ service._execute_resource_ingestion = AsyncMock(return_value={"status": "success","root_uri": "viking://resources/真实文档标题",@@ -618,18 +678,17 @@ async def test_add_resource_job_defers_target_and_expands_prepared_response():stage_callback=AsyncMock(),)- call = service.add_resource.await_args+ call = service._execute_resource_ingestion.await_argsassert call.kwargs["to"] is Noneassert call.kwargs["parent"] == "viking://resources/lark"assert call.kwargs[PREPARED_RESPONSE_ID_ARG] == "response-1"- assert call.kwargs["args"] == {}assert result["root_uri"] == "viking://resources/真实文档标题"@pytest.mark.asyncioasync def test_add_resource_job_expands_parser_args():service = ResourceService()- service.add_resource = AsyncMock(+ service._execute_resource_ingestion = AsyncMock(return_value={"status": "success","root_uri": "viking://resources/doxcnToken",@@ -656,10 +715,10 @@ async def test_add_resource_job_expands_parser_args():stage_callback=AsyncMock(),)- call = service.add_resource.await_args+ call = service._execute_resource_ingestion.await_argsassert call.kwargs["to"] == "viking://resources/doxcnToken"assert call.kwargs["parent"] is None- assert call.kwargs["args"] == {"custom_option": "forwarded"}+ assert call.kwargs["custom_option"] == "forwarded"@pytest.mark.asyncio@@ -722,21 +781,18 @@ async def test_add_resource_processor_persists_final_resource_uri(monkeypatch):assert queue_manager.enqueue.await_args.args[0] == QueueManager.ADD_RESOURCE-def test_feishu_bypass_requires_configured_auth(monkeypatch):- router = SimpleNamespace(- should_use_understanding_api=lambda source: True,- )- processor = UnifiedResourceProcessor(vlm_processor=object())- processor._parser_router = router+def test_feishu_direct_submission_requires_configured_auth(monkeypatch):+ api = _understanding_api_for_parse()+ source = "https://example.larkoffice.com/docx/doxcnToken"++ assert api.can_submit_url_directly(source, feishu_access_token="u-test")+monkeypatch.setattr("openviking.resource.feishu_watch_auth.load_feishu_app_credentials",Mock(side_effect=ValueError("missing credentials")),)- assert not processor._should_bypass_feishu_accessor(- "https://example.larkoffice.com/docx/doxcnToken",- {},- )+ assert not api.can_submit_url_directly(source)def test_normalize_lark_file_accepts_exactly_one_token():diff --git a/tests/parse/test_parser_config_wiring.py b/tests/parse/test_parser_config_wiring.pyindex cf25b215c..aaae5cc4c 100644--- a/tests/parse/test_parser_config_wiring.py+++ b/tests/parse/test_parser_config_wiring.py@@ -1,10 +1,8 @@"""Tests for parser config propagation through registries and composed parsers."""-from openviking.parse.parsers.feishu import FeishuParserfrom openviking.parse.parsers.html import HTMLParserfrom openviking.parse.parsers.pdf import PDFParserfrom openviking_cli.utils.config.parser_config import (- FeishuConfig,HTMLConfig,PDFConfig,)@@ -26,12 +24,3 @@ def test_html_parser_passes_its_config_to_nested_markdown_parser():assert markdown_parser.config.max_section_size == 2111assert markdown_parser.config.max_section_chars == 5444---def test_feishu_parser_passes_its_config_to_nested_markdown_parser():- parser = FeishuParser(config=FeishuConfig(max_section_size=1999, max_section_chars=5333))-- markdown_parser = parser._get_markdown_parser()-- assert markdown_parser.config.max_section_size == 1999- assert markdown_parser.config.max_section_chars == 5333diff --git a/tests/parse/test_parser_router.py b/tests/parse/test_parser_router.pyindex 6d289b063..564349213 100644--- a/tests/parse/test_parser_router.py+++ b/tests/parse/test_parser_router.py@@ -1,6 +1,12 @@from types import SimpleNamespace+from unittest.mock import AsyncMock+import pytest++from openviking.parse.accessors.base import LocalResource, SourceTypefrom openviking.parse.parser_router import ParserRouter+from openviking.parse.understanding_api import PREPARED_RESPONSE_ID_ARG+from openviking.utils.media_processor import UnifiedResourceProcessordef test_should_use_understanding_api_for_signed_video_url(monkeypatch):@@ -23,6 +29,35 @@ def test_should_use_understanding_api_for_signed_video_url(monkeypatch):)+def test_resolved_extension_routes_extensionless_download(monkeypatch, tmp_path):+ config = SimpleNamespace(+ parser_api=SimpleNamespace(+ enable=True,+ enable_feishu_url=False,+ extensions=["pdf"],+ ),+ )+ monkeypatch.setattr(+ "openviking_cli.utils.config.open_viking_config.get_openviking_config",+ lambda: config,+ )+ downloaded = tmp_path / "download"+ downloaded.write_bytes(b"%PDF-1.7")+ resource = LocalResource(+ path=downloaded,+ source_type=SourceType.HTTP,+ original_source="https://example.com/download?id=123",+ meta={"extension": ".pdf"},+ is_temporary=False,+ )+ processor = UnifiedResourceProcessor()++ processor._set_resolved_identity(resource, source_name=None)++ assert resource.meta["resolved_extension"] == ".pdf"+ assert processor.should_use_understanding_api(resource)++def test_should_use_understanding_api_for_feishu_url(monkeypatch):config = SimpleNamespace(parser_api=SimpleNamespace(@@ -37,19 +72,26 @@ def test_should_use_understanding_api_for_feishu_url(monkeypatch):)router = ParserRouter(parser_registry=object())+ router._understanding_api = SimpleNamespace(+ can_submit_url_directly=lambda _source, **kwargs: bool(kwargs.get("feishu_access_token"))+ )assert router.should_use_understanding_api("https://example.larkoffice.com/wiki/wikicnToken")+ assert router.should_use_understanding_directly(+ "https://example.larkoffice.com/wiki/wikicnToken",+ feishu_access_token="u-test",+ )assert not router.should_use_understanding_api("https://larkoffice.com.evil.example/wiki/wikicnToken")-def test_feishu_url_flag_defaults_to_extension_routing(monkeypatch):+def test_feishu_url_flag_is_required_even_when_extension_matches(monkeypatch):config = SimpleNamespace(parser_api=SimpleNamespace(enable=True,enable_feishu_url=False,- extensions=[],+ extensions=["pdf"],),)monkeypatch.setattr(@@ -59,4 +101,116 @@ def test_feishu_url_flag_defaults_to_extension_routing(monkeypatch):router = ParserRouter(parser_registry=object())- assert not router.should_use_understanding_api("https://example.larkoffice.com/docx/doxcnToken")+ assert not router.should_use_understanding_api(+ "https://example.larkoffice.com/docx/doxcnToken.pdf"+ )+++def test_directories_never_route_to_understanding(monkeypatch, tmp_path):+ config = SimpleNamespace(+ parser_api=SimpleNamespace(+ enable=True,+ enable_feishu_url=False,+ extensions=["pdf"],+ ),+ )+ monkeypatch.setattr(+ "openviking_cli.utils.config.open_viking_config.get_openviking_config",+ lambda: config,+ )+ resource = LocalResource(+ path=tmp_path,+ source_type=SourceType.HTTP,+ original_source="https://example.com/site",+ meta={"resolved_extension": ".pdf"},+ is_temporary=False,+ )++ assert not UnifiedResourceProcessor().should_use_understanding_api(resource)+++async def test_normalized_feishu_markdown_stays_internal(monkeypatch, tmp_path):+ config = SimpleNamespace(+ parser_api=SimpleNamespace(+ enable=True,+ enable_feishu_url=True,+ extensions=["md"],+ ),+ )+ monkeypatch.setattr(+ "openviking_cli.utils.config.open_viking_config.get_openviking_config",+ lambda: config,+ )+ resource = LocalResource(+ path=tmp_path / "document.md",+ source_type=SourceType.FEISHU,+ original_source="https://example.feishu.cn/docx/doc-token",+ meta={"resolved_extension": ".md"},+ is_temporary=False,+ )+ registry = SimpleNamespace(parse=AsyncMock(return_value=object()))+ understanding = SimpleNamespace(parse=AsyncMock(return_value=object()))+ router = ParserRouter(registry)+ router._understanding_api = understanding+ processor = UnifiedResourceProcessor()+ processor._parser_router = router++ assert not processor.should_use_understanding_api(resource)+ await router.parse(+ resource,+ resolved_extension=".md",+ parser_backend="understanding",+ )++ registry.parse.assert_awaited_once()+ understanding.parse.assert_not_awaited()+++async def test_forced_resolved_extension_survives_worker_redownload(tmp_path):+ downloaded = tmp_path / "redetected.docx"+ downloaded.write_bytes(b"content")+ resource = LocalResource(+ path=downloaded,+ source_type=SourceType.HTTP,+ original_source="https://example.com/download",+ meta={"resolved_extension": ".docx"},+ is_temporary=False,+ )+ parser_router = SimpleNamespace(parse=AsyncMock(return_value=object()))+ processor = UnifiedResourceProcessor(vlm_processor=object())+ processor._parser_router = parser_router++ await processor.process(+ "https://example.com/download",+ prepared_resource=resource,+ resolved_extension=".pdf",+ )++ assert parser_router.parse.await_args.kwargs["resolved_extension"] == ".pdf"+++async def test_prepared_understanding_response_bypasses_remote_redownload():+ result = object()+ parser_router = SimpleNamespace(parse=AsyncMock(return_value=result))+ accessor = SimpleNamespace(+ access=AsyncMock(side_effect=AssertionError("prepared response must not redownload source"))+ )+ processor = UnifiedResourceProcessor(vlm_processor=object())+ processor._parser_router = parser_router+ processor._accessor_registry = accessor++ actual = await processor.process(+ "https://example.com/expiring-download",+ resolved_extension=".pdf",+ **{PREPARED_RESPONSE_ID_ARG: "response-1"},+ )++ assert actual is result+ accessor.access.assert_not_awaited()+ parser_router.parse.assert_awaited_once()+ assert parser_router.parse.await_args.args == ("https://example.com/expiring-download",)+ assert parser_router.parse.await_args.kwargs["parser_backend"] == "understanding"+ assert parser_router.parse.await_args.kwargs[PREPARED_RESPONSE_ID_ARG] == "response-1"diff --git a/tests/parse/test_understanding_api.py b/tests/parse/test_understanding_api.pynew file mode 100644index 000000000..021f746c6--- /dev/null+++ b/tests/parse/test_understanding_api.py@@ -0,0 +1,74 @@+from pathlib import Path+from unittest.mock import AsyncMock++import pytest++from openviking.parse.understanding_api import UnderstandingAPI+++async def test_parse_uses_downloaded_file_and_resolved_extension(monkeypatch, tmp_path):+ downloaded = tmp_path / "download"+ downloaded.write_bytes(b"%PDF-1.7")+ zip_path = tmp_path / "result.zip"+ zip_path.write_bytes(b"zip")+ uploaded: list[Path] = []++ api = UnderstandingAPI.__new__(UnderstandingAPI)+ api._video_exts = {"mp4"}+ api._audio_exts = {"mp3"}+ api._image_exts = {"png"}++ async def create_file(*, local_path):+ uploaded.append(local_path)+ return {"id": "file-1"}++ async def create_response_for_file(*, file_id):+ assert file_id == "file-1"+ return {"id": "response-1"}++ async def poll_response(*, response_id):+ assert response_id == "response-1"+ return {"status": "completed"}++ monkeypatch.setattr(api, "_create_file", create_file)+ monkeypatch.setattr(api, "_create_response_for_file", create_response_for_file)+ monkeypatch.setattr(api, "_poll_response", poll_response)+ monkeypatch.setattr(api, "_extract_zip_url", lambda _: "https://example.com/result.zip")+ monkeypatch.setattr(api, "_download_zip", lambda _: _return(zip_path))+ monkeypatch.setattr(+ api,+ "_unpack_zip_to_temp_dir",+ lambda **_: _return("viking://temp/result"),+ )++ result = await api.parse(+ downloaded,+ original_source="https://example.com/download?id=123",+ resource_name="report",+ resolved_extension=".pdf",+ )++ assert uploaded == [downloaded]+ assert result.source_path == "https://example.com/download?id=123"+ assert result.source_format == "pdf"+ assert result.root.title == "report"+++async def test_submit_file_returns_response_id(tmp_path):+ source = tmp_path / "download.pdf"+ source.write_bytes(b"%PDF-1.7")+ api = UnderstandingAPI.__new__(UnderstandingAPI)+ api._create_file = AsyncMock(return_value={"id": "file-1"})+ api._create_response_for_file = AsyncMock(return_value={"id": "response-1"})++ response_id = await api.submit_file(source)++ assert response_id == "response-1"+ api._create_file.assert_awaited_once_with(local_path=source)+ api._create_response_for_file.assert_awaited_once_with(file_id="file-1")+++async def _return(value):+ return valuediff --git a/tests/parse/test_understanding_api_artifact_images.py b/tests/parse/test_understanding_api_artifact_images.pyindex e4db58118..b9317e799 100644--- a/tests/parse/test_understanding_api_artifact_images.py+++ b/tests/parse/test_understanding_api_artifact_images.py@@ -16,6 +16,7 @@ class _FakeVikingFS:def __init__(self):self.files = {}self.dirs = set()+ self.deleted_temps = []def create_temp_uri(self):return "viking://temp/artifact"@@ -29,6 +30,9 @@ class _FakeVikingFS:async def write_file(self, uri, content):self.files[uri] = content.encode("utf-8")+ async def delete_temp(self, uri):+ self.deleted_temps.append(uri)+def test_build_artifact_image_mappings_uses_existing_sibling_images(tmp_path: Path):chapter = tmp_path / "章节"@@ -80,3 +84,19 @@ async def test_unpack_artifact_writes_image_mapping_sidecar(tmp_path: Path):"Ov测试_1.md": {"Ov测试_1_img1.png": "Ov测试_1_img1.png"}}assert fake_fs.files[f"{temp_uri}/resource/Ov测试_1_img1.png"] == b"png"+++async def test_unpack_artifact_cleans_temp_on_failure(tmp_path: Path):+ invalid_zip = tmp_path / "invalid.zip"+ invalid_zip.write_bytes(b"not-a-zip")+ fake_fs = _FakeVikingFS()+ api = UnderstandingAPI.__new__(UnderstandingAPI)++ with (+ patch("openviking.parse.understanding_api.get_viking_fs", return_value=fake_fs),+ pytest.raises(zipfile.BadZipFile),+ ):+ await api._unpack_zip_to_temp_dir(invalid_zip, "resource")++ assert fake_fs.deleted_temps == ["viking://temp/artifact"]diff --git a/tests/resource/test_watch_scheduler.py b/tests/resource/test_watch_scheduler.pyindex ee49727ed..36557e4cc 100644--- a/tests/resource/test_watch_scheduler.py+++ b/tests/resource/test_watch_scheduler.py@@ -42,7 +42,7 @@ class TestWatchSchedulerResourceExistence:super().__init__()self.calls = []- async def add_resource(self, **kwargs):+ async def refresh_resource(self, **kwargs):self.calls.append(kwargs)return {"root_uri": kwargs.get("to")}@@ -81,7 +81,7 @@ class TestWatchSchedulerResourceExistence:super().__init__()self.calls = []- async def add_resource(self, **kwargs):+ async def refresh_resource(self, **kwargs):self.calls.append(kwargs)return {"root_uri": kwargs.get("to")}diff --git a/tests/service/test_resource_service_connector.py b/tests/service/test_resource_service_connector.pyindex 66f4e6a97..d0a093253 100644--- a/tests/service/test_resource_service_connector.py+++ b/tests/service/test_resource_service_connector.py@@ -4,7 +4,7 @@import asynciofrom types import SimpleNamespace-from unittest.mock import AsyncMock+from unittest.mock import AsyncMock, Mockimport httpximport pytest@@ -12,6 +12,7 @@ import pytestfrom openviking.server.identity import RequestContext, Rolefrom openviking.service import resource_service as resource_service_modulefrom openviking.service.resource_service import ResourceService+from openviking.storage.queuefs.add_resource_msg import AddResourceMsgfrom openviking_cli.exceptions import InvalidArgumentErrorfrom openviking_cli.session.user_id import UserIdentifier@@ -720,6 +721,66 @@ async def test_add_resource_falls_back_for_shared_source_with_parent(service.enqueue_git_add_resource.assert_awaited_once()+ ("field", "value"),+ [+ ("manage_watch", False),+ ("parser_args", {}),+ ("resource_lock", object()),+ ("route_source", False),+ ("skip_watch_management", True),+ ("stage_callback", object()),+ ("watch_auth_state", {}),+ ("understanding_response_id", "response-1"),+ ("parser_backend", "understanding"),+ ("resolved_extension", ".pdf"),+ ],+)+async def test_add_resource_rejects_internal_execution_fields(ctx, service, field, value):+ with pytest.raises(InvalidArgumentError, match=field):+ await service.add_resource(+ path="https://example.com/manual.pdf",+ ctx=ctx,+ **{field: value},+ )+++async def test_add_resource_job_executes_frozen_route(ctx, service):+ service._execute_resource_ingestion = AsyncMock(+ return_value={"status": "success", "root_uri": "root"}+ )+ service._should_use_connector = Mock()+ resource_lock = SimpleNamespace(close=AsyncMock())+ stage_callback = AsyncMock()+ msg = AddResourceMsg(+ task_id="task-1",+ path="git://example.com/repo.git",+ root_uri="viking://resources/repo",+ account_id=ctx.account_id,+ user_id=ctx.user.user_id,+ role=str(ctx.role),+ args={"parser_backend": "understanding"},+ )++ result = await service.execute_add_resource_job(+ msg,+ ctx=ctx,+ resource_lock=resource_lock,+ stage_callback=stage_callback,+ )++ assert result == {"status": "success", "root_uri": "root"}+ call = service._execute_resource_ingestion.await_args.kwargs+ assert call["path"] == msg.path+ assert call["to"] == msg.root_uri+ assert call["wait"] is True+ assert call["resource_lock"] is resource_lock+ assert call["parser_backend"] == "understanding"+ service._should_use_connector.assert_not_called()++@pytest.mark.asyncioasync def test_shared_source_create_parent_false_routes_to_connector(monkeypatch,diff --git a/tests/service/test_resource_service_understanding_routing.py b/tests/service/test_resource_service_understanding_routing.pynew file mode 100644index 000000000..7444e87a9--- /dev/null+++ b/tests/service/test_resource_service_understanding_routing.py@@ -0,0 +1,109 @@+from types import SimpleNamespace+from unittest.mock import AsyncMock++import pytest++from openviking.parse.accessors.base import LocalResource, SourceType+from openviking.server.identity import RequestContext, Role+from openviking.service import resource_service as resource_service_module+from openviking.service.resource_service import ResourceService+from openviking.storage.queuefs import QueueManager+from openviking.storage.queuefs.add_resource_msg import AddResourceMsg+from openviking_cli.session.user_id import UserIdentifier+++async def test_extensionless_remote_url_queues_frozen_understanding_route(+ monkeypatch,+ tmp_path,+):+ ctx = RequestContext(+ user=UserIdentifier("acct", "alice"),+ role=Role.USER,+ api_key="secret",+ )+ downloaded = tmp_path / "download.pdf"+ downloaded.write_bytes(b"%PDF-1.7")+ prepared = LocalResource(+ path=downloaded,+ source_type=SourceType.HTTP,+ original_source="https://example.com/download?id=1",+ meta={+ "resolved_extension": ".pdf",+ "original_filename": "manual.pdf",+ },+ is_temporary=True,+ )+ lock = SimpleNamespace(+ to_handoff=lambda: SimpleNamespace(to_dict=lambda: {"handle_id": "lock-1"}),+ handoff=AsyncMock(),+ close=AsyncMock(),+ )+ processor = SimpleNamespace(+ understanding_api_enabled=lambda: True,+ should_use_understanding_directly=lambda _source, **_kwargs: False,+ prepare_resource=AsyncMock(return_value=prepared),+ should_use_understanding_api=lambda resource: resource is prepared,+ submit_understanding=AsyncMock(return_value="response-1"),+ tree_builder=SimpleNamespace(+ resolve_target_uri=AsyncMock(+ return_value=(+ "viking://resources/manual",+ "viking://resources/manual",+ )+ )+ ),+ reserve_unique_candidate=AsyncMock(return_value=("viking://resources/manual", lock)),+ process_resource=AsyncMock(),+ )+ service = ResourceService(+ vikingdb=object(),+ viking_fs=object(),+ resource_processor=processor,+ skill_processor=object(),+ )+ service._should_use_connector = lambda *_args, **_kwargs: False+ tracker = SimpleNamespace(+ create=AsyncMock(return_value=SimpleNamespace(task_id="task-1")),+ update_stage=AsyncMock(),+ fail=AsyncMock(),+ )+ queue_manager = SimpleNamespace(enqueue=AsyncMock())+ monkeypatch.setattr(resource_service_module, "is_git_repo_url", lambda _path: False)+ monkeypatch.setattr(+ "openviking.service.task_tracker.get_task_tracker",+ lambda: tracker,+ )+ monkeypatch.setattr(+ "openviking.storage.queuefs.get_queue_manager",+ lambda: queue_manager,+ )+ monkeypatch.setattr(+ "openviking.storage.transaction.get_lock_manager",+ lambda: object(),+ )++ result = await service.add_resource(+ path="https://example.com/download?id=1",+ ctx=ctx,+ to="viking://resources/manual",+ wait=False,+ allow_local_path_resolution=False,+ )++ assert result == {+ "status": "success",+ "root_uri": "viking://resources/manual",+ "task_id": "task-1",+ }+ _, message = queue_manager.enqueue.await_args.args+ assert queue_manager.enqueue.await_args.args[0] == QueueManager.EXTERNAL_PARSE+ queued = AddResourceMsg.from_dict(message)+ assert queued.args["parser_backend"] == "understanding"+ assert queued.args["resolved_extension"] == ".pdf"+ assert queued.understanding_response_id == "response-1"+ assert queued.source_name == "manual.pdf"+ assert not downloaded.exists()+ processor.submit_understanding.assert_awaited_once_with(prepared)+ processor.process_resource.assert_not_awaited()+ lock.handoff.assert_awaited_once()