Open-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.
Go agentagenticaichatbotdsh-pluginembeddingsevaluationgenerative-aigolangknowledge-basellmmulti-tenantollamaopenaiquestion-answeringragrerankingsemantic-searchvector-searchwiki
feat(docparser): support MinerU 4.0 V1 API for self-hosted MinerU
wizardchene22a8367efc9
diff --git a/frontend/src/api/system/index.ts b/frontend/src/api/system/index.tsindex dcd3feb2e..fa717354b 100644--- a/frontend/src/api/system/index.ts+++ b/frontend/src/api/system/index.ts@@ -117,7 +117,10 @@ export interface ParserEngineConfig {docreader_transport?: stringmineru_endpoint?: stringmineru_api_key?: string- // MinerU 自建参数+ // MinerU 自建参数(协议自动识别:4.0+ 走 V1 API)+ mineru_server_api_key?: string+ mineru_tier?: string // flash / basic / standard / advanced,空为服务端默认+ // 以下仅对 MinerU 3.x 及更早版本生效mineru_model?: stringmineru_vlm_server_url?: stringmineru_enable_formula?: boolean | nulldiff --git a/frontend/src/i18n/locales/en-US.ts b/frontend/src/i18n/locales/en-US.tsindex 3bbf9c199..94f4c7b8b 100755--- a/frontend/src/i18n/locales/en-US.ts+++ b/frontend/src/i18n/locales/en-US.ts@@ -2054,6 +2054,18 @@ export default {serverUrl: 'Server URL',vlmServerUrlPlaceholder: 'e.g. http://your-vllm-server:8000',vlmServerUrlHint: 'Required when Backend is vlm-http-client or hybrid-http-client',+ mineruEndpointHint: 'The server version is detected automatically: MinerU 4.0+ uses the V1 API, older versions use /file_parse.',+ mineruServerApiKeyPlaceholder: 'Value of the server --api-key flag (leave empty if auth is disabled)',+ mineruServerApiKeyHint: 'Only used by MinerU 4.0+.',+ mineruTierLabel: 'Parsing tier',+ mineruTierDefault: 'Server default (prefers standard)',+ mineruTierFlash: 'flash (fastest, lowest quality)',+ mineruTierBasic: 'basic (small models, runs on CPU)',+ mineruTierStandard: 'standard (VLM, high quality)',+ mineruTierAdvanced: 'advanced (VLM, highest quality, slowest)',+ mineruTierHint: 'Only applies to MinerU 4.0+; available tiers depend on the server --tier flag.',+ mineruLegacySection: 'Legacy options (MinerU 3.x and earlier)',+ mineruLegacySectionHint: 'MinerU 4.0 removed these request parameters, so they are ignored by 4.0+ servers; configure the VLM server on the MinerU side instead.',paddleocrVlEndpointPlaceholder: 'e.g. http://your-paddleocr-vl:8080',paddleocrVlEndpointHint: 'Base URL of the full PaddleOCR-VL pipeline service; no /layout-parsing suffix needed',paddleocrVlCloudTokenPlaceholder: 'PaddleOCR-VL AI Studio Token'diff --git a/frontend/src/i18n/locales/ja-JP.ts b/frontend/src/i18n/locales/ja-JP.tsindex ff5b59439..a47b0a0ba 100644--- a/frontend/src/i18n/locales/ja-JP.ts+++ b/frontend/src/i18n/locales/ja-JP.ts@@ -2054,6 +2054,18 @@ export default {serverUrl: 'サーバURL',vlmServerUrlPlaceholder: '例: http://your-vllm-server:8000',vlmServerUrlHint: 'バックエンドがvlm-http-clientまたはhybrid-http-clientの場合は必須です',+ mineruEndpointHint: 'サーバーのバージョンを自動判別します:MinerU 4.0 以降は V1 API、それ以前は /file_parse を使用します。',+ mineruServerApiKeyPlaceholder: 'サーバー起動引数 --api-key の値(認証なしの場合は空欄)',+ mineruServerApiKeyHint: 'MinerU 4.0 以降でのみ使用されます。',+ mineruTierLabel: '解析ティア',+ mineruTierDefault: 'サーバーのデフォルト(standard 優先)',+ mineruTierFlash: 'flash(最速・最低品質)',+ mineruTierBasic: 'basic(小型モデル・CPU で動作)',+ mineruTierStandard: 'standard(VLM・高品質)',+ mineruTierAdvanced: 'advanced(VLM・最高品質・最も低速)',+ mineruTierHint: 'MinerU 4.0 以降でのみ有効です。利用可能なティアはサーバーの --tier 設定によります。',+ mineruLegacySection: '旧バージョン向け設定(MinerU 3.x 以前)',+ mineruLegacySectionHint: 'MinerU 4.0 ではこれらのリクエストパラメータが廃止されたため、4.0 以降のサーバーでは無視されます。VLM サーバーは MinerU 側で設定してください。',paddleocrVlEndpointPlaceholder: '例: http://your-paddleocr-vl:8080',paddleocrVlEndpointHint: 'PaddleOCR-VLパイプラインサービスのベースURLです。末尾に/layout-parsingを付ける必要はありません',paddleocrVlCloudTokenPlaceholder: 'PaddleOCR-VL AI Studio Token'diff --git a/frontend/src/i18n/locales/ko-KR.ts b/frontend/src/i18n/locales/ko-KR.tsindex 4d496c196..2764b7c71 100755--- a/frontend/src/i18n/locales/ko-KR.ts+++ b/frontend/src/i18n/locales/ko-KR.ts@@ -6354,6 +6354,18 @@ export default {serverUrl: '서버 URL',vlmServerUrlPlaceholder: '예: http://your-vllm-server:8000',vlmServerUrlHint: 'Backend가 vlm-http-client 또는 hybrid-http-client인 경우 필요',+ mineruEndpointHint: '서버 버전을 자동으로 감지합니다: MinerU 4.0 이상은 V1 API, 이전 버전은 /file_parse를 사용합니다.',+ mineruServerApiKeyPlaceholder: '서버 실행 인자 --api-key 값 (인증 미사용 시 비워 두세요)',+ mineruServerApiKeyHint: 'MinerU 4.0 이상에서만 사용됩니다.',+ mineruTierLabel: '파싱 티어',+ mineruTierDefault: '서버 기본값 (standard 우선)',+ mineruTierFlash: 'flash (가장 빠름, 품질 최저)',+ mineruTierBasic: 'basic (소형 모델, CPU 실행 가능)',+ mineruTierStandard: 'standard (VLM, 고품질)',+ mineruTierAdvanced: 'advanced (VLM, 최고 품질, 가장 느림)',+ mineruTierHint: 'MinerU 4.0 이상에서만 적용되며, 사용 가능한 티어는 서버의 --tier 설정에 따라 다릅니다.',+ mineruLegacySection: '레거시 옵션 (MinerU 3.x 이하)',+ mineruLegacySectionHint: 'MinerU 4.0에서 아래 요청 파라미터가 제거되어 4.0 이상 서버에서는 무시됩니다. VLM 서버는 MinerU 쪽에서 설정하세요.',paddleocrVlEndpointPlaceholder: '예: http://your-paddleocr-vl:8080',paddleocrVlEndpointHint: 'PaddleOCR-VL 전체 서비스(pipeline) 주소를 입력하세요. /layout-parsing 접미사는 불필요합니다',paddleocrVlCloudTokenPlaceholder: 'PaddleOCR-VL AI Studio Token'diff --git a/frontend/src/i18n/locales/ru-RU.ts b/frontend/src/i18n/locales/ru-RU.tsindex a1c7f885c..35fa20f2a 100755--- a/frontend/src/i18n/locales/ru-RU.ts+++ b/frontend/src/i18n/locales/ru-RU.ts@@ -6354,6 +6354,18 @@ export default {serverUrl: 'Server URL',vlmServerUrlPlaceholder: 'e.g. http://your-vllm-server:8000',vlmServerUrlHint: 'Required when Backend is vlm-http-client or hybrid-http-client',+ mineruEndpointHint: 'Версия сервера определяется автоматически: MinerU 4.0+ использует V1 API, более ранние версии — /file_parse.',+ mineruServerApiKeyPlaceholder: 'Значение флага сервера --api-key (оставьте пустым, если аутентификация выключена)',+ mineruServerApiKeyHint: 'Используется только MinerU 4.0+.',+ mineruTierLabel: 'Уровень разбора',+ mineruTierDefault: 'По умолчанию сервера (предпочтительно standard)',+ mineruTierFlash: 'flash (самый быстрый, низкое качество)',+ mineruTierBasic: 'basic (малые модели, работает на CPU)',+ mineruTierStandard: 'standard (VLM, высокое качество)',+ mineruTierAdvanced: 'advanced (VLM, наивысшее качество, самый медленный)',+ mineruTierHint: 'Применяется только к MinerU 4.0+; доступные уровни зависят от флага сервера --tier.',+ mineruLegacySection: 'Устаревшие параметры (MinerU 3.x и ранее)',+ mineruLegacySectionHint: 'В MinerU 4.0 эти параметры запроса удалены, поэтому серверы 4.0+ их игнорируют; VLM-сервер настраивается на стороне MinerU.',paddleocrVlEndpointPlaceholder: 'напр. http://your-paddleocr-vl:8080',paddleocrVlEndpointHint: 'Адрес полного сервиса PaddleOCR-VL (pipeline); суффикс /layout-parsing не требуется',paddleocrVlCloudTokenPlaceholder: 'Токен PaddleOCR-VL AI Studio'diff --git a/frontend/src/i18n/locales/zh-CN.ts b/frontend/src/i18n/locales/zh-CN.tsindex dc5bac94e..ef2b7597b 100755--- a/frontend/src/i18n/locales/zh-CN.ts+++ b/frontend/src/i18n/locales/zh-CN.ts@@ -6356,6 +6356,18 @@ export default {serverUrl: '服务器地址',vlmServerUrlPlaceholder: '如 http://your-vllm-server:8000',vlmServerUrlHint: '当 Backend 选择 vlm-http-client 或 hybrid-http-client 时需要填写',+ mineruEndpointHint: '自动识别服务版本:MinerU 4.0 及以上走 V1 API,更早版本走 /file_parse。',+ mineruServerApiKeyPlaceholder: '服务端启动参数 --api-key 的值(未启用鉴权可留空)',+ mineruServerApiKeyHint: '仅 MinerU 4.0 及以上使用。',+ mineruTierLabel: '解析档位',+ mineruTierDefault: '服务端默认(优先 standard)',+ mineruTierFlash: 'flash(最快,质量最低)',+ mineruTierBasic: 'basic(基础模型,CPU 可运行)',+ mineruTierStandard: 'standard(VLM,高质量)',+ mineruTierAdvanced: 'advanced(VLM,质量最高、最慢)',+ mineruTierHint: '仅 MinerU 4.0 及以上生效;可选档位取决于服务端启动时的 --tier。',+ mineruLegacySection: '旧版参数(MinerU 3.x 及更早)',+ mineruLegacySectionHint: 'MinerU 4.0 已移除以下请求参数,连接 4.0 及以上服务时会被忽略;VLM 地址改在 MinerU 服务端配置。',paddleocrVlEndpointPlaceholder: '如 http://your-paddleocr-vl:8080',paddleocrVlEndpointHint: '填写 PaddleOCR-VL 完整服务(pipeline)地址,无需 /layout-parsing 后缀',paddleocrVlCloudTokenPlaceholder: 'PaddleOCR-VL 飞桨星河社区 Token'diff --git a/frontend/src/views/settings/ParserEngineSettings.vue b/frontend/src/views/settings/ParserEngineSettings.vueindex d16465919..436d255ee 100644--- a/frontend/src/views/settings/ParserEngineSettings.vue+++ b/frontend/src/views/settings/ParserEngineSettings.vue@@ -234,7 +234,45 @@:placeholder="$t('settings.parser.mineruEndpointPlaceholder')"clearable/>+ <p class="form-desc">{{ $t('settings.parser.mineruEndpointHint') }}</p></div>+ <div class="form-item">+ <label class="form-label">API Key</label>+ <t-input+ v-model="config.mineru_server_api_key"+ type="password"+ :placeholder="$t('settings.parser.mineruServerApiKeyPlaceholder')"+ clearable+ >+ <template #prefix-icon><t-icon name="lock-on" /></template>+ </t-input>+ <p class="form-desc">{{ $t('settings.parser.mineruServerApiKeyHint') }}</p>+ </div>+ <div class="form-item">+ <label class="form-label">{{ $t('settings.parser.mineruTierLabel') }}</label>+ <t-select v-model="config.mineru_tier" :placeholder="$t('settings.parser.mineruTierDefault')" clearable>+ <t-option value="flash" :label="$t('settings.parser.mineruTierFlash')" />+ <t-option value="basic" :label="$t('settings.parser.mineruTierBasic')" />+ <t-option value="standard" :label="$t('settings.parser.mineruTierStandard')" />+ <t-option value="advanced" :label="$t('settings.parser.mineruTierAdvanced')" />+ </t-select>+ <p class="form-desc">{{ $t('settings.parser.mineruTierHint') }}</p>+ </div>+ <div class="form-item">+ <label class="form-label">{{ $t('settings.parser.parseMethodLabel') }}</label>+ <t-select v-model="config.mineru_parse_method">+ <t-option value="auto" :label="$t('settings.parser.parseMethodAuto')" />+ <t-option value="ocr" :label="$t('settings.parser.parseMethodOCR')" />+ <t-option value="txt" :label="$t('settings.parser.parseMethodText')" />+ </t-select>+ <p class="form-desc">{{ $t('settings.parser.parseMethodHint') }}</p>+ </div>+ </section>++ <section v-if="currentEngine.Name === 'mineru'" class="setting-drawer__section">+ <h4 class="setting-drawer__section-title">{{ $t('settings.parser.mineruLegacySection') }}</h4>+ <p class="form-desc">{{ $t('settings.parser.mineruLegacySectionHint') }}</p>+<div class="form-item"><label class="form-label">Backend</label><t-select v-model="config.mineru_model" :placeholder="$t('settings.parser.defaultPipeline')" clearable>@@ -254,15 +292,6 @@/><p class="form-desc">{{ $t('settings.parser.vlmServerUrlHint') }}</p></div>- <div class="form-item">- <label class="form-label">{{ $t('settings.parser.parseMethodLabel') }}</label>- <t-select v-model="config.mineru_parse_method">- <t-option value="auto" :label="$t('settings.parser.parseMethodAuto')" />- <t-option value="ocr" :label="$t('settings.parser.parseMethodOCR')" />- <t-option value="txt" :label="$t('settings.parser.parseMethodText')" />- </t-select>- <p class="form-desc">{{ $t('settings.parser.parseMethodHint') }}</p>- </div><div class="form-item"><label class="form-label">{{ $t('settings.parser.featuresLabel', '识别选项') }}</label><div class="form-toggles">@@ -418,6 +447,8 @@ const DEFAULT_PARSER_CONFIG: ParserEngineConfig = {docreader_transport: 'grpc',mineru_endpoint: '',mineru_api_key: '',+ mineru_server_api_key: '',+ mineru_tier: '',mineru_model: 'pipeline',mineru_vlm_server_url: '',mineru_enable_formula: true,@@ -557,6 +588,8 @@ async function loadConfig() {docreader_transport: data?.docreader_transport ?? DEFAULT_PARSER_CONFIG.docreader_transport ?? 'grpc',mineru_endpoint: data?.mineru_endpoint ?? DEFAULT_PARSER_CONFIG.mineru_endpoint ?? '',mineru_api_key: data?.mineru_api_key ?? DEFAULT_PARSER_CONFIG.mineru_api_key ?? '',+ mineru_server_api_key: data?.mineru_server_api_key ?? DEFAULT_PARSER_CONFIG.mineru_server_api_key ?? '',+ mineru_tier: data?.mineru_tier ?? DEFAULT_PARSER_CONFIG.mineru_tier ?? '',mineru_model: data?.mineru_model ?? DEFAULT_PARSER_CONFIG.mineru_model ?? '',mineru_vlm_server_url: data?.mineru_vlm_server_url ?? DEFAULT_PARSER_CONFIG.mineru_vlm_server_url ?? '',mineru_enable_formula: data?.mineru_enable_formula ?? DEFAULT_PARSER_CONFIG.mineru_enable_formula ?? true,@@ -595,6 +628,8 @@ function buildConfigPayload(): ParserEngineConfig {docreader_transport: (config.value.docreader_transport ?? 'grpc').trim() || 'grpc',mineru_endpoint: config.value.mineru_endpoint?.trim() ?? '',mineru_api_key: config.value.mineru_api_key?.trim() ?? '',+ mineru_server_api_key: config.value.mineru_server_api_key?.trim() ?? '',+ mineru_tier: config.value.mineru_tier?.trim() ?? '',mineru_model: config.value.mineru_model?.trim() ?? '',mineru_vlm_server_url: config.value.mineru_vlm_server_url?.trim() ?? '',mineru_enable_formula: config.value.mineru_enable_formula,diff --git a/internal/infrastructure/docparser/engines.go b/internal/infrastructure/docparser/engines.goindex 7883094f4..d3b28f8f4 100644--- a/internal/infrastructure/docparser/engines.go+++ b/internal/infrastructure/docparser/engines.go@@ -208,7 +208,7 @@ func (e *mineruEngine) CheckAvailable(_ bool, overrides map[string]string) (boolif endpoint == "" {return false, "MinerU service not configured"}- return PingMinerU(endpoint)+ return PingMinerU(endpoint, overrides["mineru_server_api_key"])}func (e *mineruEngine) NewReader(_ context.Context, deps ReaderDeps) (interfaces.DocReader, error) {diff --git a/internal/infrastructure/docparser/mineru_cloud_converter.go b/internal/infrastructure/docparser/mineru_cloud_converter.goindex 133d2aec0..d3993c965 100644--- a/internal/infrastructure/docparser/mineru_cloud_converter.go+++ b/internal/infrastructure/docparser/mineru_cloud_converter.go@@ -9,8 +9,8 @@ import ("io""mime""net/http"+ "net/url""path/filepath"- "regexp""sort""strings""time"@@ -350,8 +350,6 @@ func (c *MinerUCloudReader) extractDoneResult(_ context.Context, item *extractRe// --- ZIP handling ----var imgRefPattern = regexp.MustCompile(`!\[[^\]]*\]\(([^)]+)\)`)-func downloadAndExtractZip(zipURL string) (string, []types.ImageRef, error) {if err := utils.ValidateURLForSSRF(zipURL); err != nil {return "", nil, fmt.Errorf("zip URL blocked by SSRF check: %v", err)@@ -370,7 +368,12 @@ func downloadAndExtractZip(zipURL string) (string, []types.ImageRef, error) {if err != nil {return "", nil, fmt.Errorf("read zip body: %w", err)}+ return extractMarkdownZip(zipData, "MinerUCloud")+}+// extractMarkdownZip reads the shallowest .md file of a MinerU result package+// and the images it references, via markdown or inline HTML <img> syntax.+func extractMarkdownZip(zipData []byte, logLabel string) (string, []types.ImageRef, error) {zr, err := zip.NewReader(bytes.NewReader(zipData), int64(len(zipData)))if err != nil {return "", nil, fmt.Errorf("open zip: %w", err)@@ -406,8 +409,7 @@ func downloadAndExtractZip(zipURL string) (string, []types.ImageRef, error) {// Extract referenced imagesvar imageRefs []types.ImageRefseen := map[string]bool{}- for _, match := range imgRefPattern.FindAllStringSubmatch(mdText, -1) {- imgPath := match[1]+ for _, imgPath := range extractImageRefsFromContent(mdText) {if strings.HasPrefix(imgPath, "http://") || strings.HasPrefix(imgPath, "https://") || strings.HasPrefix(imgPath, "data:") {continue}@@ -418,13 +420,13 @@ func downloadAndExtractZip(zipURL string) (string, []types.ImageRef, error) {resolved := resolveInZip(imgPath, mdDir, entries)if resolved == nil {- logger.Errorf(context.Background(), "[MinerUCloud] image not found in zip: %s", imgPath)+ logger.Errorf(context.Background(), "[%s] image not found in zip: %s", logLabel, imgPath)continue}imgData, err := readZipEntryBytes(resolved)if err != nil {- logger.Errorf(context.Background(), "[MinerUCloud] failed to read zip image %s: %v", imgPath, err)+ logger.Errorf(context.Background(), "[%s] failed to read zip image %s: %v", logLabel, imgPath, err)continue}@@ -449,15 +451,20 @@ func downloadAndExtractZip(zipURL string) (string, []types.ImageRef, error) {}func resolveInZip(imgPath, mdDir string, entries map[string]*zip.File) *zip.File {- normalized := strings.ReplaceAll(imgPath, "\\", "/")- if f, ok := entries[normalized]; ok {- return f+ candidates := []string{strings.ReplaceAll(imgPath, "\\", "/")}+ if decoded, err := url.PathUnescape(candidates[0]); err == nil && decoded != candidates[0] {+ candidates = append(candidates, decoded)}- if mdDir != "" && mdDir != "." {- joined := mdDir + "/" + normalized- if f, ok := entries[joined]; ok {+ for _, c := range candidates {+ c = strings.TrimPrefix(c, "./")+ if f, ok := entries[c]; ok {return f}+ if mdDir != "" && mdDir != "." {+ if f, ok := entries[mdDir+"/"+c]; ok {+ return f+ }+ }}return nil}diff --git a/internal/infrastructure/docparser/mineru_converter.go b/internal/infrastructure/docparser/mineru_converter.goindex 1c7f1372a..ead9ac5d2 100644--- a/internal/infrastructure/docparser/mineru_converter.go+++ b/internal/infrastructure/docparser/mineru_converter.go@@ -30,14 +30,21 @@ var ()// MinerUReader calls a self-hosted MinerU API to read/convert documents.+// MinerU >= 4.0 is driven through the V1 API; older servers through the+// legacy /file_parse endpoint. The protocol is detected per request, so an+// in-place MinerU upgrade needs no configuration change.type MinerUReader struct {- endpoint string+ endpoint string+ apiKey string // V1 only: the server's --api-key+ tier string // V1 only: flash, basic, standard, advanced; "" = server default+ // Legacy (<= 3.x) only; MinerU 4.0 dropped these request parameters.backend string // "pipeline", "vlm-*", "hybrid-*"vlmServerURL string // vLLM server URL for vlm-http-client / hybrid-http-clientformulaEnable booltableEnable bool- parseMethod stringlanguage string+ // Shared: legacy parse_method / V1 ocr_mode take the same values.+ parseMethod string}// NewMinerUReader creates a reader from ParserEngineOverrides.@@ -50,6 +57,8 @@ func NewMinerUReader(overrides map[string]string) *MinerUReader {c := &MinerUReader{endpoint: strings.TrimRight(overrides["mineru_endpoint"], "/"),+ apiKey: strings.TrimSpace(overrides["mineru_server_api_key"]),+ tier: resolveMinerUTier(overrides["mineru_tier"]),backend: stringOr(overrides["mineru_model"], "pipeline"),vlmServerURL: overrides["mineru_vlm_server_url"],formulaEnable: parseBoolOr(overrides["mineru_enable_formula"], true),@@ -60,6 +69,16 @@ func NewMinerUReader(overrides map[string]string) *MinerUReader {return c}+// resolveMinerUTier keeps only tiers MinerU 4.0 understands; anything else+// falls back to the server's default selection.+func resolveMinerUTier(raw string) string {+ tier := strings.ToLower(strings.TrimSpace(raw))+ if _, ok := mineruV1Tiers[tier]; ok {+ return tier+ }+ return ""+}+func (c *MinerUReader) Read(ctx context.Context, req *types.ReadRequest) (*types.ReadResult, error) {if c.endpoint == "" {return &types.ReadResult{Error: "MinerU endpoint is not configured"}, nil@@ -78,7 +97,21 @@ func (c *MinerUReader) Read(ctx context.Context, req *types.ReadRequest) (*typesreturn &types.ReadResult{Error: "no file content provided"}, nil}- logger.Infof(context.Background(), "[MinerU] Parsing file=%s size=%d via %s", req.FileName, len(content), c.endpoint)+ protocol, err := detectMinerUProtocol(ctx, utils.NewSSRFSafeHTTPClient(utils.SSRFSafeHTTPClientConfig{+ Timeout: mineruV1APITimeout,+ MaxRedirects: 5,+ }), c.endpoint)+ if err != nil {+ return nil, fmt.Errorf("MinerU protocol detection: %w", err)+ }++ if protocol == minerUProtocolV1 {+ logger.Infof(ctx, "[MinerU] Parsing file=%s size=%d via %s (V1 API)", req.FileName, len(content), c.endpoint)+ return c.readV1(ctx, req)+ }++ logger.Infof(ctx, "[MinerU] Parsing file=%s size=%d via %s (legacy /file_parse)",+ req.FileName, len(content), c.endpoint)mdContent, imagesB64, err := c.callFileParse(ctx, content, req.FileName, req.FileType)if err != nil {@@ -103,6 +136,25 @@ func (c *MinerUReader) Read(ctx context.Context, req *types.ReadRequest) (*types}, nil}+func (c *MinerUReader) readV1(ctx context.Context, req *types.ReadRequest) (*types.ReadResult, error) {+ client := newMinerUV1Client(c.endpoint, c.apiKey, mineruTimeout)+ mdContent, imageRefs, err := client.Parse(ctx, req.FileContent, minerUUploadFileName(req.FileName, req.FileType),+ minerUV1ParseOptions{Tier: c.tier, OCRMode: c.parseMethod})+ if err != nil {+ return nil, fmt.Errorf("MinerU V1: %w", err)+ }++ mdContent = normalizeMinerUMarkdown(mdContent)+ mdContent, imageRefs = ensureOriginalImageRef(req, mdContent, imageRefs)++ logger.Infof(ctx, "[MinerU] Parsed successfully (V1), markdown=%d chars, images=%d", len(mdContent), len(imageRefs))++ return &types.ReadResult{+ MarkdownContent: mdContent,+ ImageRefs: imageRefs,+ }, nil+}+type mineruFileEntry struct {MDContent string `json:"md_content"`Images map[string]string `json:"images"` // path -> "data:image/png;base64,..." or raw base64@@ -358,8 +410,10 @@ func validateMinerUOutboundURL(rawURL string) error {return nil}-// PingMinerU checks if the self-hosted MinerU service is reachable.-func PingMinerU(endpoint string) (bool, string) {+// PingMinerU checks if the self-hosted MinerU service is reachable. For+// MinerU 4.0 it also verifies the configured API key, since /v1/health itself+// is served without authentication.+func PingMinerU(endpoint, apiKey string) (bool, string) {endpoint = strings.TrimRight(endpoint, "/")if endpoint == "" {return false, "未配置 MinerU 端点"@@ -371,6 +425,16 @@ func PingMinerU(endpoint string) (bool, string) {Timeout: 5 * time.Second,MaxRedirects: 5,})+ ctx := context.Background()++ protocol, err := detectMinerUProtocol(ctx, client, endpoint)+ if err != nil {+ return false, fmt.Sprintf("MinerU 服务不可用: %v", err)+ }+ if protocol == minerUProtocolV1 {+ return pingMinerUV1Auth(ctx, client, endpoint, apiKey)+ }+resp, err := client.Get(endpoint + "/docs")if err != nil {return false, fmt.Sprintf("MinerU 服务不可达: %v", err)@@ -382,6 +446,30 @@ func PingMinerU(endpoint string) (bool, string) {return true, ""}+// pingMinerUV1Auth hits an authenticated V1 endpoint: a server started with+// --api-key answers 401 when the key is missing or wrong.+func pingMinerUV1Auth(ctx context.Context, client *http.Client, endpoint, apiKey string) (bool, string) {+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint+"/v1/parse/jobs?limit=1", nil)+ if err != nil {+ return false, fmt.Sprintf("构建请求失败: %v", err)+ }+ if apiKey = strings.TrimSpace(apiKey); apiKey != "" {+ req.Header.Set("Authorization", "Bearer "+apiKey)+ }+ resp, err := client.Do(req)+ if err != nil {+ return false, fmt.Sprintf("MinerU 服务不可达: %v", err)+ }+ _ = resp.Body.Close()+ if resp.StatusCode == http.StatusUnauthorized {+ if apiKey == "" {+ return false, "MinerU 服务已启用鉴权,请配置 API Key"+ }+ return false, "MinerU API Key 无效"+ }+ return true, ""+}+// escapedImageSyntaxPattern matches markdown image references whose `[` was// over-escaped to `\[` by html-to-markdown. The URL group mirrors the// downstream image-extraction regex so escapes are only stripped for actualdiff --git a/internal/infrastructure/docparser/mineru_converter_ssrf_test.go b/internal/infrastructure/docparser/mineru_converter_ssrf_test.goindex 133e0be47..cb6d29ceb 100644--- a/internal/infrastructure/docparser/mineru_converter_ssrf_test.go+++ b/internal/infrastructure/docparser/mineru_converter_ssrf_test.go@@ -14,7 +14,7 @@ func TestValidateMinerUOutboundURL_RejectsLoopback(t *testing.T) {}func TestPingMinerU_RejectsPrivateEndpoint(t *testing.T) {- ok, msg := PingMinerU("http://127.0.0.1:8080")+ ok, msg := PingMinerU("http://127.0.0.1:8080", "")assert.False(t, ok)assert.Contains(t, msg, "SSRF")}diff --git a/internal/infrastructure/docparser/mineru_converter_test.go b/internal/infrastructure/docparser/mineru_converter_test.goindex 4310d9fdb..e25dab023 100644--- a/internal/infrastructure/docparser/mineru_converter_test.go+++ b/internal/infrastructure/docparser/mineru_converter_test.go@@ -183,6 +183,11 @@ func TestMinerUReaderPreservesMultipartFilename(t *testing.T) {}server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ if r.URL.Path == "/v1/health" {+ // MinerU <= 3.x has no V1 API.+ http.NotFound(w, r)+ return+ }got := capturedRequest{method: r.Method, path: r.URL.Path}file, header, err := r.FormFile("files")if err != nil {diff --git a/internal/infrastructure/docparser/mineru_v1_client.go b/internal/infrastructure/docparser/mineru_v1_client.gonew file mode 100644index 000000000..0eae3915e--- /dev/null+++ b/internal/infrastructure/docparser/mineru_v1_client.go@@ -0,0 +1,594 @@+package docparser++import (+ "bytes"+ "context"+ "crypto/sha256"+ "encoding/hex"+ "encoding/json"+ "errors"+ "fmt"+ "io"+ "mime"+ "net/http"+ "net/url"+ "path/filepath"+ "strings"+ "time"++ "github.com/Tencent/WeKnora/internal/logger"+ "github.com/Tencent/WeKnora/internal/types"+ "github.com/Tencent/WeKnora/internal/utils"+)++// MinerU 4.0 replaced the synchronous /file_parse endpoint with the V1 API:+// upload → parse job → poll → download artifacts. See+// https://opendatalab.github.io/MinerU/usage/http_api/.++const (+ mineruV1PollInitial = 2 * time.Second+ mineruV1PollMax = 30 * time.Second+ mineruV1APITimeout = 30 * time.Second+ mineruV1MaxZipBytes = 1 << 30+ mineruV1CancelBudget = 10 * time.Second+)++// MinerU V1 tiers accepted by POST /v1/parse/jobs. An empty tier means the+// server's default selection policy (standard → basic for PDF/image).+var mineruV1Tiers = map[string]struct{}{+ "flash": {},+ "basic": {},+ "standard": {},+ "advanced": {},+}++type minerUProtocol int++const (+ minerUProtocolLegacy minerUProtocol = iota // MinerU <= 3.x: POST /file_parse+ minerUProtocolV1 // MinerU >= 4.0: /v1/*+)++// minerUV1Client talks to any MinerU V1 deployment (self-hosted api-server or+// the official mineru.net/api).+type minerUV1Client struct {+ baseURL string // without the /v1 suffix+ apiKey string+ apiClient *http.Client // small JSON calls+ bulkClient *http.Client // byte upload and artifact download+ pollInitial time.Duration+ pollMax time.Duration+ jobTimeout time.Duration+ maxZipBytes int64+ logLabel string+ cancelBudget time.Duration+}++type minerUV1ParseOptions struct {+ Tier string // "" → server default+ OCRMode string // auto, txt, ocr+}++func newMinerUV1Client(baseURL, apiKey string, jobTimeout time.Duration) *minerUV1Client {+ return &minerUV1Client{+ baseURL: strings.TrimRight(baseURL, "/"),+ apiKey: strings.TrimSpace(apiKey),+ apiClient: utils.NewSSRFSafeHTTPClient(utils.SSRFSafeHTTPClientConfig{+ Timeout: mineruV1APITimeout,+ MaxRedirects: 5,+ }),+ bulkClient: utils.NewSSRFSafeHTTPClient(utils.SSRFSafeHTTPClientConfig{+ Timeout: jobTimeout,+ MaxRedirects: 5,+ }),+ pollInitial: mineruV1PollInitial,+ pollMax: mineruV1PollMax,+ jobTimeout: jobTimeout,+ maxZipBytes: mineruV1MaxZipBytes,+ logLabel: "MinerU",+ cancelBudget: mineruV1CancelBudget,+ }+}++// --- wire types ---++type minerUV1ErrorEnvelope struct {+ Error struct {+ Message string `json:"message"`+ Type string `json:"type"`+ Code string `json:"code"`+ } `json:"error"`+}++type minerUV1File struct {+ ID string `json:"id"`+}++type minerUV1Upload struct {+ ID string `json:"id"`+ Status string `json:"status"`+ UploadURL string `json:"upload_url"`+ UploadMethod string `json:"upload_method"`+ UploadHeaders map[string]string `json:"upload_headers"`+ File *minerUV1File `json:"file"`+}++type minerUV1FileRef struct {+ FileID string `json:"file_id"`+ Bytes int64 `json:"bytes"`+}++type minerUV1JobFile struct {+ Name string `json:"name"`+ Status string `json:"status"`+ OutputFiles map[string]minerUV1FileRef `json:"output_files"`+ Error *struct {+ Code string `json:"code"`+ Message string `json:"message"`+ } `json:"error"`+ Parse *struct {+ ModelUsed string `json:"model_used"`+ DurationMS int64 `json:"duration_ms"`+ ParserVersion string `json:"parser_version"`+ } `json:"parse"`+}++type minerUV1Job struct {+ JobID string `json:"job_id"`+ Status string `json:"status"`+ Tier string `json:"tier"`+ Files []minerUV1JobFile `json:"files"`+}++func minerUV1JobTerminal(status string) bool {+ switch status {+ case "completed", "partial", "failed", "canceled":+ return true+ }+ return false+}++// --- protocol detection ---++// detectMinerUProtocol probes GET /v1/health. MinerU 4.0 answers 200 (or 503+// while models are still preloading); older servers have no such route and+// answer 404/405.+func detectMinerUProtocol(ctx context.Context, client *http.Client, endpoint string) (minerUProtocol, error) {+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint+"/v1/health", nil)+ if err != nil {+ return minerUProtocolLegacy, fmt.Errorf("create health request: %w", err)+ }+ resp, err := client.Do(req)+ if err != nil {+ return minerUProtocolLegacy, fmt.Errorf("MinerU service unreachable: %w", err)+ }+ defer func() { _ = resp.Body.Close() }()+ body, _ := io.ReadAll(io.LimitReader(resp.Body, 64<<10))++ switch resp.StatusCode {+ case http.StatusOK:+ var health struct {+ Status string `json:"status"`+ }+ if err := json.Unmarshal(body, &health); err != nil || health.Status != "ok" {+ return minerUProtocolLegacy, fmt.Errorf("unexpected /v1/health response: %s",+ truncateForLog(string(body), 300))+ }+ return minerUProtocolV1, nil+ case http.StatusNotFound, http.StatusMethodNotAllowed:+ return minerUProtocolLegacy, nil+ default:+ if msg := minerUV1ErrorMessage(body); msg != "" {+ return minerUProtocolV1, fmt.Errorf("MinerU service not ready (status %d): %s", resp.StatusCode, msg)+ }+ return minerUProtocolLegacy, fmt.Errorf("MinerU /v1/health status %d: %s",+ resp.StatusCode, truncateForLog(string(body), 300))+ }+}++func minerUV1ErrorMessage(body []byte) string {+ var env minerUV1ErrorEnvelope+ if err := json.Unmarshal(body, &env); err != nil {+ return ""+ }+ if env.Error.Code != "" && env.Error.Message != "" {+ return env.Error.Code + ": " + env.Error.Message+ }+ return firstNonEmpty(env.Error.Message, env.Error.Code)+}++func truncateForLog(s string, n int) string {+ if len(s) <= n {+ return s+ }+ return s[:n] + "..."+}++// --- parse flow ---++// Parse uploads content, runs a parse job and returns the markdown and+// images extracted from the job's zip artifact.+func (c *minerUV1Client) Parse(+ ctx context.Context,+ content []byte,+ fileName string,+ opts minerUV1ParseOptions,+) (string, []types.ImageRef, error) {+ ctx, cancel := context.WithTimeout(ctx, c.jobTimeout)+ defer cancel()++ fileID, err := c.upload(ctx, content, fileName)+ if err != nil {+ return "", nil, fmt.Errorf("upload: %w", err)+ }++ jobID, err := c.createJob(ctx, fileID, opts)+ if err != nil {+ return "", nil, fmt.Errorf("create parse job: %w", err)+ }+ logger.Infof(ctx, "[%s] parse job created: job_id=%s file_id=%s tier=%q ocr_mode=%s",+ c.logLabel, jobID, fileID, opts.Tier, opts.OCRMode)++ job, err := c.waitJob(ctx, jobID)+ if err != nil {+ c.cancelJob(jobID)+ return "", nil, err+ }++ zipRef, err := minerUV1ZipOutput(job)+ if err != nil {+ return "", nil, err+ }++ zipData, err := c.downloadFile(ctx, zipRef.FileID)+ if err != nil {+ return "", nil, fmt.Errorf("download zip artifact: %w", err)+ }++ md, imageRefs, err := extractMarkdownZip(zipData, c.logLabel)+ if err != nil {+ return "", nil, fmt.Errorf("extract zip artifact: %w", err)+ }+ return md, imageRefs, nil+}++func (c *minerUV1Client) upload(ctx context.Context, content []byte, fileName string) (string, error) {+ sum := sha256.Sum256(content)+ sha := hex.EncodeToString(sum[:])++ var created minerUV1Upload+ err := c.doJSON(ctx, http.MethodPost, "/v1/uploads", map[string]interface{}{+ "filename": fileName,+ "bytes": len(content),+ "mime_type": minerUV1MimeType(fileName),+ "purpose": "parse",+ "sha256sum": sha,+ }, &created)+ if err != nil {+ return "", err+ }++ switch created.Status {+ case "completed":+ // Same bytes already stored on the server: no transfer needed.+ if created.File == nil || created.File.ID == "" {+ return "", fmt.Errorf("upload completed without file id")+ }+ return created.File.ID, nil+ case "pending":+ default:+ return "", fmt.Errorf("unexpected upload status %q", created.Status)+ }+ if created.ID == "" || created.UploadURL == "" {+ return "", fmt.Errorf("pending upload missing id or upload_url")+ }++ if err := c.putBytes(ctx, &created, content); err != nil {+ return "", err+ }++ var completed minerUV1Upload+ err = c.doJSON(ctx, http.MethodPost, "/v1/uploads/"+url.PathEscape(created.ID)+"/complete",+ map[string]string{"sha256sum": sha}, &completed)+ if err != nil {+ return "", fmt.Errorf("complete upload: %w", err)+ }+ if completed.File == nil || completed.File.ID == "" {+ return "", fmt.Errorf("complete upload returned no file id (status %q)", completed.Status)+ }+ return completed.File.ID, nil+}++// putBytes sends the file to the URL returned by POST /v1/uploads. The MinerU+// API key is attached only when that URL is same-origin with the configured+// endpoint: the official API hands out pre-signed object-storage URLs that+// carry their own authorization and must never receive our key.+func (c *minerUV1Client) putBytes(ctx context.Context, up *minerUV1Upload, content []byte) error {+ target, err := c.resolveURL(up.UploadURL)+ if err != nil {+ return fmt.Errorf("invalid upload_url: %w", err)+ }+ method := strings.ToUpper(stringOr(up.UploadMethod, http.MethodPut))++ req, err := http.NewRequestWithContext(ctx, method, target.String(), bytes.NewReader(content))+ if err != nil {+ return fmt.Errorf("create upload request: %w", err)+ }+ for k, v := range up.UploadHeaders {+ req.Header.Set(k, v)+ }+ if c.sameOrigin(target) {+ c.setAuth(req)+ }++ resp, err := c.bulkClient.Do(req)+ if err != nil {+ return fmt.Errorf("upload bytes: %w", err)+ }+ defer func() { _ = resp.Body.Close() }()+ if resp.StatusCode >= 300 {+ body, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))+ return fmt.Errorf("upload bytes status %d: %s", resp.StatusCode, truncateForLog(string(body), 300))+ }+ return nil+}++func (c *minerUV1Client) createJob(ctx context.Context, fileID string, opts minerUV1ParseOptions) (string, error) {+ payload := map[string]interface{}{+ "files": []map[string]interface{}{+ {"source": map[string]string{"type": "file_id", "file_id": fileID}},+ },+ "output_formats": []string{"zip"},+ }+ if opts.Tier != "" {+ payload["tier"] = opts.Tier+ }+ if opts.OCRMode != "" {+ payload["ocr_mode"] = opts.OCRMode+ }++ var job minerUV1Job+ if err := c.doJSON(ctx, http.MethodPost, "/v1/parse/jobs", payload, &job); err != nil {+ return "", err+ }+ if job.JobID == "" {+ return "", fmt.Errorf("response has no job_id")+ }+ return job.JobID, nil+}++func (c *minerUV1Client) waitJob(ctx context.Context, jobID string) (*minerUV1Job, error) {+ interval := c.pollInitial+ for poll := 1; ; poll++ {+ var job minerUV1Job+ err := c.doJSON(ctx, http.MethodGet, "/v1/parse/jobs/"+url.PathEscape(jobID), nil, &job)+ if err != nil {+ if ctxErr := ctx.Err(); ctxErr != nil {+ return nil, fmt.Errorf("parse job %s: %w", jobID, ctxErr)+ }+ var apiErr *minerUV1APIError+ if errors.As(err, &apiErr) && apiErr.status == http.StatusNotFound {+ // Job state lives in process memory; a restarted server forgets it.+ return nil, fmt.Errorf("parse job %s no longer exists (MinerU restarted?): %w", jobID, err)+ }+ logger.Warnf(ctx, "[%s] poll #%d for job %s failed: %v", c.logLabel, poll, jobID, err)+ } else {+ if poll == 1 || poll%10 == 0 || minerUV1JobTerminal(job.Status) {+ logger.Infof(ctx, "[%s] poll #%d: job=%s status=%s", c.logLabel, poll, jobID, job.Status)+ }+ if minerUV1JobTerminal(job.Status) {+ return &job, nil+ }+ }++ sleepCtx(ctx, interval)+ if ctxErr := ctx.Err(); ctxErr != nil {+ return nil, fmt.Errorf("parse job %s: %w", jobID, ctxErr)+ }+ interval *= 2+ if interval > c.pollMax {+ interval = c.pollMax+ }+ }+}++// cancelJob best-effort cancels a job we stopped waiting for so the server+// does not keep burning GPU time on it. It runs on a fresh context because the+// caller's context is usually already done.+func (c *minerUV1Client) cancelJob(jobID string) {+ ctx, cancel := context.WithTimeout(context.Background(), c.cancelBudget)+ defer cancel()+ if err := c.doJSON(ctx, http.MethodDelete, "/v1/parse/jobs/"+url.PathEscape(jobID), nil, nil); err != nil {+ logger.Warnf(ctx, "[%s] cancel job %s: %v", c.logLabel, jobID, err)+ }+}++// minerUV1ZipOutput returns the zip artifact of the single submitted file, or+// the per-file error when parsing did not succeed.+func minerUV1ZipOutput(job *minerUV1Job) (minerUV1FileRef, error) {+ if len(job.Files) == 0 {+ return minerUV1FileRef{}, fmt.Errorf("parse job %s ended %s without file results", job.JobID, job.Status)+ }+ f := job.Files[0]+ if f.Status != "completed" {+ reason := job.Status+ if f.Error != nil {+ reason = firstNonEmpty(strings.TrimSpace(f.Error.Code+": "+f.Error.Message), job.Status)+ }+ return minerUV1FileRef{}, fmt.Errorf("parse job %s %s: %s", job.JobID, job.Status, reason)+ }+ ref, ok := f.OutputFiles["zip"]+ if !ok || ref.FileID == "" {+ return minerUV1FileRef{}, fmt.Errorf("parse job %s completed without zip output", job.JobID)+ }+ return ref, nil+}++func (c *minerUV1Client) downloadFile(ctx context.Context, fileID string) ([]byte, error) {+ req, err := http.NewRequestWithContext(ctx, http.MethodGet,+ c.baseURL+"/v1/files/"+url.PathEscape(fileID)+"/content", nil)+ if err != nil {+ return nil, err+ }+ // Go drops Authorization when following a redirect to another host, so a+ // 302 to a CDN (official API) does not leak the key.+ c.setAuth(req)++ resp, err := c.bulkClient.Do(req)+ if err != nil {+ return nil, err+ }+ defer func() { _ = resp.Body.Close() }()+ if resp.StatusCode != http.StatusOK {+ body, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))+ return nil, newMinerUV1APIError(resp.StatusCode, body)+ }+ data, err := io.ReadAll(io.LimitReader(resp.Body, c.maxZipBytes+1))+ if err != nil {+ return nil, err+ }+ if int64(len(data)) > c.maxZipBytes {+ return nil, fmt.Errorf("artifact exceeds %d bytes", c.maxZipBytes)+ }+ return data, nil+}++// --- HTTP helpers ---++type minerUV1APIError struct {+ status int+ msg string+}++func (e *minerUV1APIError) Error() string {+ return fmt.Sprintf("MinerU API status %d: %s", e.status, e.msg)+}++func newMinerUV1APIError(status int, body []byte) *minerUV1APIError {+ msg := minerUV1ErrorMessage(body)+ if msg == "" {+ msg = truncateForLog(string(body), 500)+ }+ return &minerUV1APIError{status: status, msg: msg}+}++func (c *minerUV1Client) doJSON(ctx context.Context, method, path string, in, out interface{}) error {+ var body io.Reader+ if in != nil {+ raw, err := json.Marshal(in)+ if err != nil {+ return fmt.Errorf("marshal request: %w", err)+ }+ body = bytes.NewReader(raw)+ }+ req, err := http.NewRequestWithContext(ctx, method, c.baseURL+path, body)+ if err != nil {+ return fmt.Errorf("create request: %w", err)+ }+ if in != nil {+ req.Header.Set("Content-Type", "application/json")+ }+ c.setAuth(req)++ resp, err := c.apiClient.Do(req)+ if err != nil {+ return fmt.Errorf("%s %s: %w", method, path, err)+ }+ defer func() { _ = resp.Body.Close() }()+ respBody, err := io.ReadAll(io.LimitReader(resp.Body, 16<<20))+ if err != nil {+ return fmt.Errorf("read response: %w", err)+ }+ if resp.StatusCode < 200 || resp.StatusCode >= 300 {+ return newMinerUV1APIError(resp.StatusCode, respBody)+ }+ if out == nil {+ return nil+ }+ if err := json.Unmarshal(respBody, out); err != nil {+ return fmt.Errorf("decode %s %s response: %w", method, path, err)+ }+ return nil+}++func (c *minerUV1Client) setAuth(req *http.Request) {+ if c.apiKey != "" {+ req.Header.Set("Authorization", "Bearer "+c.apiKey)+ }+}++func (c *minerUV1Client) resolveURL(raw string) (*url.URL, error) {+ base, err := url.Parse(c.baseURL + "/")+ if err != nil {+ return nil, err+ }+ ref, err := url.Parse(strings.TrimSpace(raw))+ if err != nil {+ return nil, err+ }+ resolved := base.ResolveReference(ref)+ if resolved.Scheme != "http" && resolved.Scheme != "https" {+ return nil, fmt.Errorf("unsupported scheme %q", resolved.Scheme)+ }+ return resolved, nil+}++// sameOrigin compares scheme, host and effective port with the base URL.+func (c *minerUV1Client) sameOrigin(u *url.URL) bool {+ base, err := url.Parse(c.baseURL)+ if err != nil {+ return false+ }+ return strings.EqualFold(base.Scheme, u.Scheme) &&+ strings.EqualFold(base.Hostname(), u.Hostname()) &&+ effectivePort(base) == effectivePort(u)+}++func effectivePort(u *url.URL) string {+ if p := u.Port(); p != "" {+ return p+ }+ switch strings.ToLower(u.Scheme) {+ case "https":+ return "443"+ case "http":+ return "80"+ }+ return ""+}++// Slim container images often ship without /etc/mime.types, where Go only+// knows a handful of extensions; cover the formats MinerU accepts explicitly.+var minerUV1MimeTypes = map[string]string{+ ".pdf": "application/pdf",+ ".doc": "application/msword",+ ".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",+ ".ppt": "application/vnd.ms-powerpoint",+ ".pptx": "application/vnd.openxmlformats-officedocument.presentationml.presentation",+ ".xls": "application/vnd.ms-excel",+ ".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",+ ".png": "image/png",+ ".jpg": "image/jpeg",+ ".jpeg": "image/jpeg",+ ".bmp": "image/bmp",+ ".tif": "image/tiff",+ ".tiff": "image/tiff",+ ".html": "text/html",+ ".htm": "text/html",+}++func minerUV1MimeType(fileName string) string {+ ext := strings.ToLower(filepath.Ext(fileName))+ if t, ok := minerUV1MimeTypes[ext]; ok {+ return t+ }+ if ext != "" {+ if t := mime.TypeByExtension(ext); t != "" {+ if mediaType, _, err := mime.ParseMediaType(t); err == nil {+ return mediaType+ }+ return t+ }+ }+ return "application/octet-stream"+}diff --git a/internal/infrastructure/docparser/mineru_v1_client_test.go b/internal/infrastructure/docparser/mineru_v1_client_test.gonew file mode 100644index 000000000..4f8c0b915--- /dev/null+++ b/internal/infrastructure/docparser/mineru_v1_client_test.go@@ -0,0 +1,433 @@+package docparser++import (+ "archive/zip"+ "bytes"+ "context"+ "encoding/json"+ "io"+ "net/http"+ "net/http/httptest"+ "strings"+ "sync"+ "testing"+ "time"++ "github.com/Tencent/WeKnora/internal/types"+ "github.com/Tencent/WeKnora/internal/utils"+ "github.com/stretchr/testify/assert"+ "github.com/stretchr/testify/require"+)++// fakeMinerUV1 is a minimal in-memory MinerU 4.0 api-server.+type fakeMinerUV1 struct {+ t *testing.T+ apiKey string++ mu sync.Mutex+ uploadURL string // absolute or relative; defaults to same-origin relative path+ dedupHit bool // POST /v1/uploads answers "completed" directly+ uploaded []byte+ putAuth string+ uploadMime string+ jobPayload map[string]any+ pollStatuses []string // statuses returned by successive GET job calls+ polls int+ fileError *map[string]string+ canceled bool+ zipData []byte+ downloadCalls int+}++func (f *fakeMinerUV1) handler() http.Handler {+ mux := http.NewServeMux()+ writeJSON := func(w http.ResponseWriter, status int, v any) {+ w.Header().Set("Content-Type", "application/json")+ w.WriteHeader(status)+ _ = json.NewEncoder(w).Encode(v)+ }+ authed := func(next http.HandlerFunc) http.HandlerFunc {+ return func(w http.ResponseWriter, r *http.Request) {+ if f.apiKey != "" && r.Header.Get("Authorization") != "Bearer "+f.apiKey {+ writeJSON(w, http.StatusUnauthorized, map[string]any{+ "error": map[string]string{"message": "bad key", "type": "auth_error", "code": "invalid_api_key"},+ })+ return+ }+ next(w, r)+ }+ }++ mux.HandleFunc("GET /v1/health", func(w http.ResponseWriter, _ *http.Request) {+ writeJSON(w, http.StatusOK, map[string]any{"status": "ok", "version": "1.0.0"})+ })+ mux.HandleFunc("POST /v1/uploads", authed(func(w http.ResponseWriter, r *http.Request) {+ var body map[string]any+ require.NoError(f.t, json.NewDecoder(r.Body).Decode(&body))+ f.mu.Lock()+ defer f.mu.Unlock()+ f.uploadMime, _ = body["mime_type"].(string)+ if f.dedupHit {+ writeJSON(w, http.StatusOK, map[string]any{+ "id": "upload_1", "status": "completed", "file": map[string]any{"id": "file-src"},+ })+ return+ }+ uploadURL := f.uploadURL+ if uploadURL == "" {+ uploadURL = "/v1/uploads/upload_1/content"+ }+ writeJSON(w, http.StatusOK, map[string]any{+ "id": "upload_1", "status": "pending", "upload_url": uploadURL, "upload_method": "PUT",+ "upload_headers": map[string]string{"Content-Type": f.uploadMime},+ })+ }))+ mux.HandleFunc("PUT /v1/uploads/upload_1/content", f.handlePut)+ mux.HandleFunc("POST /v1/uploads/upload_1/complete", authed(func(w http.ResponseWriter, _ *http.Request) {+ writeJSON(w, http.StatusOK, map[string]any{+ "id": "upload_1", "status": "completed", "file": map[string]any{"id": "file-src"},+ })+ }))+ mux.HandleFunc("POST /v1/parse/jobs", authed(func(w http.ResponseWriter, r *http.Request) {+ var body map[string]any+ require.NoError(f.t, json.NewDecoder(r.Body).Decode(&body))+ f.mu.Lock()+ f.jobPayload = body+ f.mu.Unlock()+ writeJSON(w, http.StatusAccepted, map[string]any{"job_id": "job_1", "status": "queued"})+ }))+ mux.HandleFunc("GET /v1/parse/jobs/job_1", authed(func(w http.ResponseWriter, _ *http.Request) {+ f.mu.Lock()+ defer f.mu.Unlock()+ status := "completed"+ if f.polls < len(f.pollStatuses) {+ status = f.pollStatuses[f.polls]+ }+ f.polls+++ file := map[string]any{"name": "report.pdf", "status": status}+ switch status {+ case "completed":+ file["output_files"] = map[string]any{"zip": map[string]any{"file_id": "file-zip", "bytes": len(f.zipData)}}+ case "failed":+ file["error"] = *f.fileError+ }+ writeJSON(w, http.StatusOK, map[string]any{"job_id": "job_1", "status": status, "files": []any{file}})+ }))+ mux.HandleFunc("DELETE /v1/parse/jobs/job_1", authed(func(w http.ResponseWriter, _ *http.Request) {+ f.mu.Lock()+ f.canceled = true+ f.mu.Unlock()+ writeJSON(w, http.StatusOK, map[string]any{"job_id": "job_1", "status": "canceled"})+ }))+ mux.HandleFunc("GET /v1/parse/jobs", authed(func(w http.ResponseWriter, _ *http.Request) {+ writeJSON(w, http.StatusOK, map[string]any{"object": "list", "data": []any{}})+ }))+ mux.HandleFunc("GET /v1/files/file-zip/content", authed(func(w http.ResponseWriter, _ *http.Request) {+ f.mu.Lock()+ f.downloadCalls+++ f.mu.Unlock()+ w.Header().Set("Content-Type", "application/zip")+ _, _ = w.Write(f.zipData)+ }))+ return mux+}++func (f *fakeMinerUV1) handlePut(w http.ResponseWriter, r *http.Request) {+ data, err := io.ReadAll(r.Body)+ require.NoError(f.t, err)+ f.mu.Lock()+ f.uploaded = data+ f.putAuth = r.Header.Get("Authorization")+ f.mu.Unlock()+ w.WriteHeader(http.StatusOK)+}++func buildMinerUZip(t *testing.T, files map[string]string) []byte {+ t.Helper()+ var buf bytes.Buffer+ zw := zip.NewWriter(&buf)+ for name, content := range files {+ w, err := zw.Create(name)+ require.NoError(t, err)+ _, err = w.Write([]byte(content))+ require.NoError(t, err)+ }+ require.NoError(t, zw.Close())+ return buf.Bytes()+}++func allowLoopbackSSRF(t *testing.T) {+ t.Helper()+ utils.SetSSRFWhitelistFromRaw("127.0.0.1")+ t.Cleanup(func() { utils.SetSSRFWhitelistFromRaw("") })+}++func fastV1Client(baseURL, apiKey string) *minerUV1Client {+ c := newMinerUV1Client(baseURL, apiKey, 10*time.Second)+ c.pollInitial = time.Millisecond+ c.pollMax = 5 * time.Millisecond+ return c+}++const v1TestMarkdown = "# Report\n\n\n\n" ++ "<table><tr><td><img src=\"images/page_0_table_image_2_1.png\"></td></tr></table>\n"++func v1TestZip(t *testing.T) []byte {+ return buildMinerUZip(t, map[string]string{+ "markdown.md": v1TestMarkdown,+ "middle_json.json": "{}",+ "images/page_0_image_1.png": "png-bytes-1",+ "images/page_0_table_image_2_1.png": "png-bytes-2",+ "images/page_1_image_unreferenced.png": "unused",+ })+}++func TestMinerUV1ClientParseFullFlow(t *testing.T) {+ allowLoopbackSSRF(t)+ fake := &fakeMinerUV1{+ t: t,+ apiKey: "server-key",+ pollStatuses: []string{"queued", "running", "completed"},+ zipData: v1TestZip(t),+ }+ server := httptest.NewServer(fake.handler())+ defer server.Close()++ client := fastV1Client(server.URL, "server-key")+ md, refs, err := client.Parse(context.Background(), []byte("%PDF-1.7 body"), "report.pdf",+ minerUV1ParseOptions{Tier: "advanced", OCRMode: "txt"})+ require.NoError(t, err)++ assert.Equal(t, v1TestMarkdown, md)+ require.Len(t, refs, 2)+ byRef := map[string]types.ImageRef{}+ for _, ref := range refs {+ byRef[ref.OriginalRef] = ref+ }+ assert.Equal(t, []byte("png-bytes-1"), byRef["images/page_0_image_1.png"].ImageData)+ assert.Equal(t, []byte("png-bytes-2"), byRef["images/page_0_table_image_2_1.png"].ImageData)+ assert.Equal(t, "image/png", byRef["images/page_0_image_1.png"].MimeType)++ assert.Equal(t, []byte("%PDF-1.7 body"), fake.uploaded)+ assert.Equal(t, "Bearer server-key", fake.putAuth, "same-origin upload must carry the API key")+ assert.Equal(t, "application/pdf", fake.uploadMime)+ assert.Equal(t, 3, fake.polls)+ assert.False(t, fake.canceled)++ assert.Equal(t, "advanced", fake.jobPayload["tier"])+ assert.Equal(t, "txt", fake.jobPayload["ocr_mode"])+ assert.Equal(t, []any{"zip"}, fake.jobPayload["output_formats"])+ files := fake.jobPayload["files"].([]any)+ source := files[0].(map[string]any)["source"].(map[string]any)+ assert.Equal(t, map[string]any{"type": "file_id", "file_id": "file-src"}, source)+}++func TestMinerUV1ClientOmitsTierForServerDefault(t *testing.T) {+ allowLoopbackSSRF(t)+ fake := &fakeMinerUV1{t: t, zipData: v1TestZip(t)}+ server := httptest.NewServer(fake.handler())+ defer server.Close()++ _, _, err := fastV1Client(server.URL, "").Parse(context.Background(), []byte("x"), "a.pdf",+ minerUV1ParseOptions{OCRMode: "auto"})+ require.NoError(t, err)+ _, hasTier := fake.jobPayload["tier"]+ assert.False(t, hasTier)+}++func TestMinerUV1ClientCrossOriginUploadOmitsAPIKey(t *testing.T) {+ allowLoopbackSSRF(t)+ fake := &fakeMinerUV1{t: t, apiKey: "server-key", zipData: v1TestZip(t)}++ // A different port on the same host is a different origin, like a+ // pre-signed object-storage URL handed out by the official API.+ storage := httptest.NewServer(http.HandlerFunc(fake.handlePut))+ defer storage.Close()+ fake.uploadURL = storage.URL + "/v1/uploads/upload_1/content?Signature=abc"++ server := httptest.NewServer(fake.handler())+ defer server.Close()++ _, _, err := fastV1Client(server.URL, "server-key").Parse(context.Background(), []byte("bytes"), "a.pdf",+ minerUV1ParseOptions{})+ require.NoError(t, err)+ assert.Equal(t, []byte("bytes"), fake.uploaded)+ assert.Empty(t, fake.putAuth, "API key must not leak to a cross-origin upload URL")+}++func TestMinerUV1ClientDedupSkipsByteUpload(t *testing.T) {+ allowLoopbackSSRF(t)+ fake := &fakeMinerUV1{t: t, dedupHit: true, zipData: v1TestZip(t)}+ server := httptest.NewServer(fake.handler())+ defer server.Close()++ _, _, err := fastV1Client(server.URL, "").Parse(context.Background(), []byte("bytes"), "a.pdf",+ minerUV1ParseOptions{})+ require.NoError(t, err)+ assert.Nil(t, fake.uploaded)+}++func TestMinerUV1ClientFailedJobReportsFileError(t *testing.T) {+ allowLoopbackSSRF(t)+ fake := &fakeMinerUV1{+ t: t,+ pollStatuses: []string{"failed"},+ fileError: &map[string]string{"code": "parse_failed", "message": "corrupt pdf"},+ }+ server := httptest.NewServer(fake.handler())+ defer server.Close()++ _, _, err := fastV1Client(server.URL, "").Parse(context.Background(), []byte("x"), "a.pdf",+ minerUV1ParseOptions{})+ require.Error(t, err)+ assert.Contains(t, err.Error(), "parse_failed: corrupt pdf")+ assert.Zero(t, fake.downloadCalls)+}++func TestMinerUV1ClientCancelsJobOnTimeout(t *testing.T) {+ allowLoopbackSSRF(t)+ statuses := make([]string, 1000)+ for i := range statuses {+ statuses[i] = "running"+ }+ fake := &fakeMinerUV1{t: t, pollStatuses: statuses}+ server := httptest.NewServer(fake.handler())+ defer server.Close()++ client := fastV1Client(server.URL, "")+ client.jobTimeout = 100 * time.Millisecond+ _, _, err := client.Parse(context.Background(), []byte("x"), "a.pdf", minerUV1ParseOptions{})+ require.Error(t, err)+ assert.ErrorIs(t, err, context.DeadlineExceeded)+ assert.True(t, fake.canceled, "abandoned job should be canceled on the server")+}++func TestMinerUV1ClientAPIErrorMessage(t *testing.T) {+ allowLoopbackSSRF(t)+ fake := &fakeMinerUV1{t: t, apiKey: "server-key"}+ server := httptest.NewServer(fake.handler())+ defer server.Close()++ _, _, err := fastV1Client(server.URL, "wrong").Parse(context.Background(), []byte("x"), "a.pdf",+ minerUV1ParseOptions{})+ require.Error(t, err)+ assert.Contains(t, err.Error(), "401")+ assert.Contains(t, err.Error(), "invalid_api_key: bad key")+}++func TestDetectMinerUProtocol(t *testing.T) {+ allowLoopbackSSRF(t)+ tests := []struct {+ name string+ status int+ body string+ want minerUProtocol+ wantErr string+ }{+ {name: "v1 healthy", status: 200, body: `{"status":"ok","version":"1.0.0"}`, want: minerUProtocolV1},+ {name: "legacy 404", status: 404, body: `{"detail":"Not Found"}`, want: minerUProtocolLegacy},+ {name: "legacy 405", status: 405, body: ``, want: minerUProtocolLegacy},+ {+ name: "v1 preloading models",+ status: 503,+ body: `{"error":{"message":"weights missing","type":"engine_error",` ++ `"code":"model_preload_files_missing"}}`,+ want: minerUProtocolV1,+ wantErr: "model_preload_files_missing: weights missing",+ },+ {name: "proxy error", status: 502, body: `bad gateway`, wantErr: "status 502"},+ }+ for _, tt := range tests {+ t.Run(tt.name, func(t *testing.T) {+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ assert.Equal(t, "/v1/health", r.URL.Path)+ w.WriteHeader(tt.status)+ _, _ = w.Write([]byte(tt.body))+ }))+ defer server.Close()++ got, err := detectMinerUProtocol(context.Background(), http.DefaultClient, server.URL)+ if tt.wantErr != "" {+ require.Error(t, err)+ assert.Contains(t, err.Error(), tt.wantErr)+ return+ }+ require.NoError(t, err)+ assert.Equal(t, tt.want, got)+ })+ }+}++func TestMinerUReaderUsesV1WhenAvailable(t *testing.T) {+ allowLoopbackSSRF(t)+ fake := &fakeMinerUV1{t: t, apiKey: "server-key", zipData: v1TestZip(t)}+ server := httptest.NewServer(fake.handler())+ defer server.Close()++ reader := NewMinerUReader(map[string]string{+ "mineru_endpoint": server.URL,+ "mineru_server_api_key": "server-key",+ "mineru_tier": "Standard",+ "mineru_parse_method": "ocr",+ // Legacy-only knobs are ignored by the V1 path.+ "mineru_model": "pipeline",+ "mineru_enable_formula": "false",+ })+ result, err := reader.Read(context.Background(), &types.ReadRequest{+ FileContent: []byte("%PDF"),+ FileName: "dir/report.pdf",+ FileType: "pdf",+ })+ require.NoError(t, err)+ require.Empty(t, result.Error)+ assert.True(t, strings.HasPrefix(result.MarkdownContent, "# Report"))+ assert.Len(t, result.ImageRefs, 2)+ assert.Equal(t, "standard", fake.jobPayload["tier"])+ assert.Equal(t, "ocr", fake.jobPayload["ocr_mode"])+}++func TestResolveMinerUTier(t *testing.T) {+ assert.Equal(t, "", resolveMinerUTier(""))+ assert.Equal(t, "flash", resolveMinerUTier(" Flash "))+ assert.Equal(t, "advanced", resolveMinerUTier("advanced"))+ assert.Equal(t, "", resolveMinerUTier("pipeline"), "legacy backend names are not tiers")+ assert.Equal(t, "", resolveMinerUTier("auto"))+}++func TestPingMinerUV1(t *testing.T) {+ allowLoopbackSSRF(t)+ fake := &fakeMinerUV1{t: t, apiKey: "server-key"}+ server := httptest.NewServer(fake.handler())+ defer server.Close()++ ok, msg := PingMinerU(server.URL, "server-key")+ assert.True(t, ok, msg)++ ok, msg = PingMinerU(server.URL, "wrong")+ assert.False(t, ok)+ assert.Contains(t, msg, "API Key 无效")++ ok, msg = PingMinerU(server.URL, "")+ assert.False(t, ok)+ assert.Contains(t, msg, "配置 API Key")++ open := &fakeMinerUV1{t: t}+ openServer := httptest.NewServer(open.handler())+ defer openServer.Close()+ ok, msg = PingMinerU(openServer.URL, "")+ assert.True(t, ok, msg)+}++func TestExtractMarkdownZipResolvesEncodedAndNestedPaths(t *testing.T) {+ data := buildMinerUZip(t, map[string]string{+ "report/report.md": "\n",+ "report/images/第 1 页.png": "p1",+ "report/images/b.jpg": "p2",+ })+ md, refs, err := extractMarkdownZip(data, "test")+ require.NoError(t, err)+ assert.Contains(t, md, "images/")+ require.Len(t, refs, 2)+ assert.Equal(t, []byte("p1"), refs[0].ImageData)+ assert.Equal(t, "image/jpeg", refs[1].MimeType)+}diff --git a/internal/types/config_redaction.go b/internal/types/config_redaction.goindex c7e557761..ff743b745 100644--- a/internal/types/config_redaction.go+++ b/internal/types/config_redaction.go@@ -33,6 +33,9 @@ func ParserEngineConfigForResponse(cfg *ParserEngineConfig, maskSecrets bool) *Pif out.MinerUAPIKey != "" {out.MinerUAPIKey = RedactedSecretPlaceholder}+ if out.MinerUServerAPIKey != "" {+ out.MinerUServerAPIKey = RedactedSecretPlaceholder+ }if out.PaddleOCRVLCloudToken != "" {out.PaddleOCRVLCloudToken = RedactedSecretPlaceholder}@@ -167,6 +170,7 @@ func MergeParserEngineConfigForUpdate(incoming, existing *ParserEngineConfig) *Pprev = *existing}out.MinerUAPIKey = PreserveIfRedacted(out.MinerUAPIKey, prev.MinerUAPIKey)+ out.MinerUServerAPIKey = PreserveIfRedacted(out.MinerUServerAPIKey, prev.MinerUServerAPIKey)out.PaddleOCRVLCloudToken = PreserveIfRedacted(out.PaddleOCRVLCloudToken, prev.PaddleOCRVLCloudToken)// Chat attachment parser rules are configured per agent; preserve any legacy// tenant-level rules when the settings UI omits this field on engine updates.diff --git a/internal/types/config_redaction_test.go b/internal/types/config_redaction_test.goindex 75d885812..b4838ae6c 100644--- a/internal/types/config_redaction_test.go+++ b/internal/types/config_redaction_test.go@@ -43,21 +43,37 @@ func TestMergeWebSearchConfigForUpdate_PreservesRedactedSecrets(t *testing.T) {func TestMergeParserEngineConfigForUpdate_PreservesRedactedSecrets(t *testing.T) {existing := &ParserEngineConfig{MinerUAPIKey: "mineru-secret",+ MinerUServerAPIKey: "mineru-server-secret",PaddleOCRVLCloudToken: "paddle-secret",MinerUEndpoint: "http://mineru",}incoming := &ParserEngineConfig{MinerUAPIKey: RedactedSecretPlaceholder,+ MinerUServerAPIKey: RedactedSecretPlaceholder,PaddleOCRVLCloudToken: RedactedSecretPlaceholder,MinerUEndpoint: "http://mineru-new",}merged := MergeParserEngineConfigForUpdate(incoming, existing)require.NotNil(t, merged)assert.Equal(t, "mineru-secret", merged.MinerUAPIKey)+ assert.Equal(t, "mineru-server-secret", merged.MinerUServerAPIKey)assert.Equal(t, "paddle-secret", merged.PaddleOCRVLCloudToken)assert.Equal(t, "http://mineru-new", merged.MinerUEndpoint)}+func TestParserEngineConfigForResponse_MasksMinerUSecrets(t *testing.T) {+ cfg := &ParserEngineConfig{+ MinerUAPIKey: "mineru-secret",+ MinerUServerAPIKey: "mineru-server-secret",+ MinerUEndpoint: "http://mineru",+ }+ resp := ParserEngineConfigForResponse(cfg, true)+ require.NotNil(t, resp)+ assert.Equal(t, RedactedSecretPlaceholder, resp.MinerUAPIKey)+ assert.Equal(t, RedactedSecretPlaceholder, resp.MinerUServerAPIKey)+ assert.Equal(t, "http://mineru", resp.MinerUEndpoint)+}+func TestMergeParserEngineConfigForUpdate_PreservesLegacyChatParserRules(t *testing.T) {existing := &ParserEngineConfig{ChatParserEngineRules: []ParserEngineRule{diff --git a/internal/types/tenant.go b/internal/types/tenant.goindex d86873dd6..164008a19 100644--- a/internal/types/tenant.go+++ b/internal/types/tenant.go@@ -310,7 +310,10 @@ type ParserEngineConfig struct {MinerUEndpoint string `json:"mineru_endpoint"` // MinerU 自建服务端点MinerUAPIKey string `json:"mineru_api_key"` // MinerU 云 API Key- // MinerU 自建解析参数+ // MinerU 自建解析参数。协议按服务端自动识别:4.0+ 走 V1 API,更早版本走 /file_parse。+ MinerUServerAPIKey string `json:"mineru_server_api_key,omitempty"` // 4.0+:服务端 --api-key+ MinerUTier string `json:"mineru_tier,omitempty"` // 4.0+:flash/basic/standard/advanced,空为服务端默认+ // 以下 backend、vLLM、公式/表格、语言参数仅对 3.x 及更早版本生效。MinerUModel string `json:"mineru_model,omitempty"` // backend: pipeline, vlm-*, hybrid-*MinerUVLMServerURL string `json:"mineru_vlm_server_url,omitempty"` // vLLM 服务器地址 (vlm-http-client / hybrid-http-client)MinerUEnableFormula *bool `json:"mineru_enable_formula,omitempty"`@@ -400,6 +403,12 @@ func (c *ParserEngineConfig) ToOverridesMap() map[string]string {if c.MinerUAPIKey != "" {m["mineru_api_key"] = c.MinerUAPIKey}+ if c.MinerUServerAPIKey != "" {+ m["mineru_server_api_key"] = c.MinerUServerAPIKey+ }+ if c.MinerUTier != "" {+ m["mineru_tier"] = c.MinerUTier+ }if c.MinerUModel != "" {m["mineru_model"] = c.MinerUModel}diff --git a/website-docs/02-architecture/03-document-pipeline.md b/website-docs/02-architecture/03-document-pipeline.mdindex 682cf1c80..9a5199afa 100644--- a/website-docs/02-architecture/03-document-pipeline.md+++ b/website-docs/02-architecture/03-document-pipeline.md@@ -271,7 +271,7 @@ Worker 消费 `TypeDocumentProcess` 后按五个规范化阶段推进,每个- **builtin**:通过 gRPC(`docparser/grpc_parser.go`)或 HTTP(`http_parser.go`)调用 Python **docreader** 服务;- **simple**:Go 原生解析 md/txt/csv/json/图片/音频(`builtin_converter.go`,CSV→Markdown 表格、JSON→递归分割的代码块,图片/音频转占位引用);- **anydoc**:Go 进程内解析 docx/doc/pptx/ppt/xlsx/xls/odf/rtf/epub/csv/pdf(`anydoc_reader.go`),底层是通过 cgo 链接的 anydoc Rust 库。office 文档的嵌入图按文档模型插回 Markdown 原位;无文字层的扫描件 PDF 在 DocReader 可用时回退到 builtin 整页渲染。链接了 anydoc 时,未配置规则的复杂格式默认走 anydoc,但 PDF 默认仍走 builtin。仅在带 `anydoc` 构建标签的二进制中可用,其余构建里该引擎在引擎列表中显示为不可用;- - **weknoracloud / mineru / mineru_cloud / paddleocr_vl / paddleocr_vl_cloud**:HTTP 转换器(`engines.go` 注册,按 `mineru_endpoint`、`mineru_api_key`、`paddleocr_vl_endpoint` 等配置判定可用性)。+ - **weknoracloud / mineru / mineru_cloud / paddleocr_vl / paddleocr_vl_cloud**:HTTP 转换器(`engines.go` 注册,按 `mineru_endpoint`、`mineru_api_key`、`paddleocr_vl_endpoint` 等配置判定可用性;自建 `mineru` 会自动识别 MinerU 4.0 的 V1 API 与旧版 `/file_parse`,见[文档解析服务](../03-features/03-document-parsing.md#mineru-self-hosted))。引擎目录集中在 `internal/infrastructure/docparser/engines.go`:每个引擎同时声明元数据(名称、描述、文件类型、可用性探针)与 `NewReader` 工厂,`docparser.NewReader` 按名字分发,未注册的名字(如只存在于 docreader 的 `markitdown`)落到 docreader 客户端。5. 文件模式:从 `FileService.GetFile(payload.FilePath)` 读回字节填入 `ReadRequest.FileContent`。diff --git a/website-docs/03-features/03-document-parsing.md b/website-docs/03-features/03-document-parsing.mdindex c92e522e6..62dda5f44 100644--- a/website-docs/03-features/03-document-parsing.md+++ b/website-docs/03-features/03-document-parsing.md@@ -460,6 +460,36 @@ docker build -f docker/Dockerfile.app --build-arg WITH_ANYDOC=0 -t weknora-app .| `internal/infrastructure/docparser/engines.go` | 引擎注册(元数据 + Reader 工厂) || `third_party/anydoc-go/` | vendored 的上游 Go 绑定与 C ABI shim(来源与本地改动见该目录 README) |+### MinerU 自建引擎(Go 进程直连,不经 docreader) {#mineru-self-hosted}++`mineru` 引擎由 Go App 直接调用自建的 MinerU 服务(`internal/infrastructure/docparser/mineru_converter.go`)。MinerU 4.0 删除了旧的 `/file_parse` 接口,改为 V1 API,因此 WeKnora 在每次解析前先请求 `GET {mineru_endpoint}/v1/health`,按结果选择协议:++| 探测结果 | 协议 | 流程 |+| --- | --- | --- |+| `200` 且 `status=ok` | V1(MinerU ≥ 4.0,`mineru_v1_client.go`) | `POST /v1/uploads` → 按返回的 `upload_url` 上传字节 → `POST /v1/uploads/{id}/complete` → `POST /v1/parse/jobs`(只请求 `zip` 产物)→ 轮询 `GET /v1/parse/jobs/{id}` → `GET /v1/files/{id}/content` 下载 zip,取其中的 `markdown.md` 与 `images/` |+| `404` / `405` | 旧版(MinerU ≤ 3.x) | `POST /file_parse`,同步返回 Markdown 与 base64 图片 |+| `503` 带错误结构 | V1,但服务未就绪 | 直接报错(常见于模型预加载失败),不回退旧协议 |++升级 MinerU 不需要改 WeKnora 的配置。两套协议的参数对应关系:++| 设置项(`ParserEngineConfig`) | MinerU ≥ 4.0 | MinerU ≤ 3.x |+| --- | --- | --- |+| `mineru_endpoint` | V1 服务地址(如 `http://mineru:8000`) | 同左 |+| `mineru_server_api_key` | 服务以 `--api-key` 启动时作为 `Authorization: Bearer` 发送 | 不使用 |+| `mineru_tier` | `tier`:`flash` / `basic` / `standard` / `advanced`;留空由服务端选默认档位(优先 `standard`) | 不使用 |+| `mineru_parse_method` | `ocr_mode`(`auto` / `txt` / `ocr`) | `parse_method` |+| `mineru_model`、`mineru_vlm_server_url`、`mineru_enable_formula`、`mineru_enable_table`、`mineru_language` | 忽略(4.0 已删除这些参数;VLM 地址改在 MinerU 的 `config.yaml` 里配置) | 原样发送 |++V1 流程的几个细节:++- 上传时附带 `sha256sum`,服务端已有相同文件时直接复用,不再传字节。+- 只有 `upload_url` 与 `mineru_endpoint` 同源(scheme、host、端口都相同)时才附带 API Key;跨源地址(如官方 API 下发的预签名对象存储 URL)不带 Key,并照常经过 SSRF 校验。+- 轮询从 2 秒开始指数退避,最长 30 秒一次;总时长与旧版一样是 1000 秒。超时或调用方取消时,会发 `DELETE /v1/parse/jobs/{id}` 取消服务端任务。+- MinerU V1 服务的上传、任务状态都存在进程内存里,服务重启后正在轮询的任务会返回 404,本次解析直接失败。+- 「测试连接」在 V1 服务上额外请求一次需要鉴权的 `GET /v1/parse/jobs?limit=1`,用来发现 API Key 缺失或错误(`/v1/health` 本身不校验 Key)。++`mineru_cloud`(mineru.net)目前仍走 `/api/v4/file-urls/batch` 批量接口,不受 4.0 自建服务变化影响。+---### 附:关键事实速查