Skip to content

Open-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.

feat(docparser): support MinerU 4.0 V1 API for self-hosted MinerU

wizardchene22a8367efc9
diff --git a/frontend/src/api/system/index.ts b/frontend/src/api/system/index.ts
index dcd3feb2e..fa717354b 100644
--- a/frontend/src/api/system/index.ts
+++ b/frontend/src/api/system/index.ts
@@ -117,7 +117,10 @@ export interface ParserEngineConfig {
docreader_transport?: string
mineru_endpoint?: string
mineru_api_key?: string
- // MinerU 自建参数
+ // MinerU 自建参数(协议自动识别:4.0+ 走 V1 API)
+ mineru_server_api_key?: string
+ mineru_tier?: string // flash / basic / standard / advanced,空为服务端默认
+ // 以下仅对 MinerU 3.x 及更早版本生效
mineru_model?: string
mineru_vlm_server_url?: string
mineru_enable_formula?: boolean | null
diff --git a/frontend/src/i18n/locales/en-US.ts b/frontend/src/i18n/locales/en-US.ts
index 3bbf9c199..94f4c7b8b 100755
--- a/frontend/src/i18n/locales/en-US.ts
+++ b/frontend/src/i18n/locales/en-US.ts
@@ -2054,6 +2054,18 @@ export default {
serverUrl: 'Server URL',
vlmServerUrlPlaceholder: 'e.g. http://your-vllm-server:8000',
vlmServerUrlHint: 'Required when Backend is vlm-http-client or hybrid-http-client',
+ mineruEndpointHint: 'The server version is detected automatically: MinerU 4.0+ uses the V1 API, older versions use /file_parse.',
+ mineruServerApiKeyPlaceholder: 'Value of the server --api-key flag (leave empty if auth is disabled)',
+ mineruServerApiKeyHint: 'Only used by MinerU 4.0+.',
+ mineruTierLabel: 'Parsing tier',
+ mineruTierDefault: 'Server default (prefers standard)',
+ mineruTierFlash: 'flash (fastest, lowest quality)',
+ mineruTierBasic: 'basic (small models, runs on CPU)',
+ mineruTierStandard: 'standard (VLM, high quality)',
+ mineruTierAdvanced: 'advanced (VLM, highest quality, slowest)',
+ mineruTierHint: 'Only applies to MinerU 4.0+; available tiers depend on the server --tier flag.',
+ mineruLegacySection: 'Legacy options (MinerU 3.x and earlier)',
+ mineruLegacySectionHint: 'MinerU 4.0 removed these request parameters, so they are ignored by 4.0+ servers; configure the VLM server on the MinerU side instead.',
paddleocrVlEndpointPlaceholder: 'e.g. http://your-paddleocr-vl:8080',
paddleocrVlEndpointHint: 'Base URL of the full PaddleOCR-VL pipeline service; no /layout-parsing suffix needed',
paddleocrVlCloudTokenPlaceholder: 'PaddleOCR-VL AI Studio Token'
diff --git a/frontend/src/i18n/locales/ja-JP.ts b/frontend/src/i18n/locales/ja-JP.ts
index ff5b59439..a47b0a0ba 100644
--- a/frontend/src/i18n/locales/ja-JP.ts
+++ b/frontend/src/i18n/locales/ja-JP.ts
@@ -2054,6 +2054,18 @@ export default {
serverUrl: 'サーバURL',
vlmServerUrlPlaceholder: '例: http://your-vllm-server:8000',
vlmServerUrlHint: 'バックエンドがvlm-http-clientまたはhybrid-http-clientの場合は必須です',
+ mineruEndpointHint: 'サーバーのバージョンを自動判別します:MinerU 4.0 以降は V1 API、それ以前は /file_parse を使用します。',
+ mineruServerApiKeyPlaceholder: 'サーバー起動引数 --api-key の値(認証なしの場合は空欄)',
+ mineruServerApiKeyHint: 'MinerU 4.0 以降でのみ使用されます。',
+ mineruTierLabel: '解析ティア',
+ mineruTierDefault: 'サーバーのデフォルト(standard 優先)',
+ mineruTierFlash: 'flash(最速・最低品質)',
+ mineruTierBasic: 'basic(小型モデル・CPU で動作)',
+ mineruTierStandard: 'standard(VLM・高品質)',
+ mineruTierAdvanced: 'advanced(VLM・最高品質・最も低速)',
+ mineruTierHint: 'MinerU 4.0 以降でのみ有効です。利用可能なティアはサーバーの --tier 設定によります。',
+ mineruLegacySection: '旧バージョン向け設定(MinerU 3.x 以前)',
+ mineruLegacySectionHint: 'MinerU 4.0 ではこれらのリクエストパラメータが廃止されたため、4.0 以降のサーバーでは無視されます。VLM サーバーは MinerU 側で設定してください。',
paddleocrVlEndpointPlaceholder: '例: http://your-paddleocr-vl:8080',
paddleocrVlEndpointHint: 'PaddleOCR-VLパイプラインサービスのベースURLです。末尾に/layout-parsingを付ける必要はありません',
paddleocrVlCloudTokenPlaceholder: 'PaddleOCR-VL AI Studio Token'
diff --git a/frontend/src/i18n/locales/ko-KR.ts b/frontend/src/i18n/locales/ko-KR.ts
index 4d496c196..2764b7c71 100755
--- a/frontend/src/i18n/locales/ko-KR.ts
+++ b/frontend/src/i18n/locales/ko-KR.ts
@@ -6354,6 +6354,18 @@ export default {
serverUrl: '서버 URL',
vlmServerUrlPlaceholder: '예: http://your-vllm-server:8000',
vlmServerUrlHint: 'Backend가 vlm-http-client 또는 hybrid-http-client인 경우 필요',
+ mineruEndpointHint: '서버 버전을 자동으로 감지합니다: MinerU 4.0 이상은 V1 API, 이전 버전은 /file_parse를 사용합니다.',
+ mineruServerApiKeyPlaceholder: '서버 실행 인자 --api-key 값 (인증 미사용 시 비워 두세요)',
+ mineruServerApiKeyHint: 'MinerU 4.0 이상에서만 사용됩니다.',
+ mineruTierLabel: '파싱 티어',
+ mineruTierDefault: '서버 기본값 (standard 우선)',
+ mineruTierFlash: 'flash (가장 빠름, 품질 최저)',
+ mineruTierBasic: 'basic (소형 모델, CPU 실행 가능)',
+ mineruTierStandard: 'standard (VLM, 고품질)',
+ mineruTierAdvanced: 'advanced (VLM, 최고 품질, 가장 느림)',
+ mineruTierHint: 'MinerU 4.0 이상에서만 적용되며, 사용 가능한 티어는 서버의 --tier 설정에 따라 다릅니다.',
+ mineruLegacySection: '레거시 옵션 (MinerU 3.x 이하)',
+ mineruLegacySectionHint: 'MinerU 4.0에서 아래 요청 파라미터가 제거되어 4.0 이상 서버에서는 무시됩니다. VLM 서버는 MinerU 쪽에서 설정하세요.',
paddleocrVlEndpointPlaceholder: '예: http://your-paddleocr-vl:8080',
paddleocrVlEndpointHint: 'PaddleOCR-VL 전체 서비스(pipeline) 주소를 입력하세요. /layout-parsing 접미사는 불필요합니다',
paddleocrVlCloudTokenPlaceholder: 'PaddleOCR-VL AI Studio Token'
diff --git a/frontend/src/i18n/locales/ru-RU.ts b/frontend/src/i18n/locales/ru-RU.ts
index a1c7f885c..35fa20f2a 100755
--- a/frontend/src/i18n/locales/ru-RU.ts
+++ b/frontend/src/i18n/locales/ru-RU.ts
@@ -6354,6 +6354,18 @@ export default {
serverUrl: 'Server URL',
vlmServerUrlPlaceholder: 'e.g. http://your-vllm-server:8000',
vlmServerUrlHint: 'Required when Backend is vlm-http-client or hybrid-http-client',
+ mineruEndpointHint: 'Версия сервера определяется автоматически: MinerU 4.0+ использует V1 API, более ранние версии — /file_parse.',
+ mineruServerApiKeyPlaceholder: 'Значение флага сервера --api-key (оставьте пустым, если аутентификация выключена)',
+ mineruServerApiKeyHint: 'Используется только MinerU 4.0+.',
+ mineruTierLabel: 'Уровень разбора',
+ mineruTierDefault: 'По умолчанию сервера (предпочтительно standard)',
+ mineruTierFlash: 'flash (самый быстрый, низкое качество)',
+ mineruTierBasic: 'basic (малые модели, работает на CPU)',
+ mineruTierStandard: 'standard (VLM, высокое качество)',
+ mineruTierAdvanced: 'advanced (VLM, наивысшее качество, самый медленный)',
+ mineruTierHint: 'Применяется только к MinerU 4.0+; доступные уровни зависят от флага сервера --tier.',
+ mineruLegacySection: 'Устаревшие параметры (MinerU 3.x и ранее)',
+ mineruLegacySectionHint: 'В MinerU 4.0 эти параметры запроса удалены, поэтому серверы 4.0+ их игнорируют; VLM-сервер настраивается на стороне MinerU.',
paddleocrVlEndpointPlaceholder: 'напр. http://your-paddleocr-vl:8080',
paddleocrVlEndpointHint: 'Адрес полного сервиса PaddleOCR-VL (pipeline); суффикс /layout-parsing не требуется',
paddleocrVlCloudTokenPlaceholder: 'Токен PaddleOCR-VL AI Studio'
diff --git a/frontend/src/i18n/locales/zh-CN.ts b/frontend/src/i18n/locales/zh-CN.ts
index dc5bac94e..ef2b7597b 100755
--- a/frontend/src/i18n/locales/zh-CN.ts
+++ b/frontend/src/i18n/locales/zh-CN.ts
@@ -6356,6 +6356,18 @@ export default {
serverUrl: '服务器地址',
vlmServerUrlPlaceholder: '如 http://your-vllm-server:8000',
vlmServerUrlHint: '当 Backend 选择 vlm-http-client 或 hybrid-http-client 时需要填写',
+ mineruEndpointHint: '自动识别服务版本:MinerU 4.0 及以上走 V1 API,更早版本走 /file_parse。',
+ mineruServerApiKeyPlaceholder: '服务端启动参数 --api-key 的值(未启用鉴权可留空)',
+ mineruServerApiKeyHint: '仅 MinerU 4.0 及以上使用。',
+ mineruTierLabel: '解析档位',
+ mineruTierDefault: '服务端默认(优先 standard)',
+ mineruTierFlash: 'flash(最快,质量最低)',
+ mineruTierBasic: 'basic(基础模型,CPU 可运行)',
+ mineruTierStandard: 'standard(VLM,高质量)',
+ mineruTierAdvanced: 'advanced(VLM,质量最高、最慢)',
+ mineruTierHint: '仅 MinerU 4.0 及以上生效;可选档位取决于服务端启动时的 --tier。',
+ mineruLegacySection: '旧版参数(MinerU 3.x 及更早)',
+ mineruLegacySectionHint: 'MinerU 4.0 已移除以下请求参数,连接 4.0 及以上服务时会被忽略;VLM 地址改在 MinerU 服务端配置。',
paddleocrVlEndpointPlaceholder: '如 http://your-paddleocr-vl:8080',
paddleocrVlEndpointHint: '填写 PaddleOCR-VL 完整服务(pipeline)地址,无需 /layout-parsing 后缀',
paddleocrVlCloudTokenPlaceholder: 'PaddleOCR-VL 飞桨星河社区 Token'
diff --git a/frontend/src/views/settings/ParserEngineSettings.vue b/frontend/src/views/settings/ParserEngineSettings.vue
index d16465919..436d255ee 100644
--- a/frontend/src/views/settings/ParserEngineSettings.vue
+++ b/frontend/src/views/settings/ParserEngineSettings.vue
@@ -234,7 +234,45 @@
:placeholder="$t('settings.parser.mineruEndpointPlaceholder')"
clearable
/>
+ <p class="form-desc">{{ $t('settings.parser.mineruEndpointHint') }}</p>
</div>
+ <div class="form-item">
+ <label class="form-label">API Key</label>
+ <t-input
+ v-model="config.mineru_server_api_key"
+ type="password"
+ :placeholder="$t('settings.parser.mineruServerApiKeyPlaceholder')"
+ clearable
+ >
+ <template #prefix-icon><t-icon name="lock-on" /></template>
+ </t-input>
+ <p class="form-desc">{{ $t('settings.parser.mineruServerApiKeyHint') }}</p>
+ </div>
+ <div class="form-item">
+ <label class="form-label">{{ $t('settings.parser.mineruTierLabel') }}</label>
+ <t-select v-model="config.mineru_tier" :placeholder="$t('settings.parser.mineruTierDefault')" clearable>
+ <t-option value="flash" :label="$t('settings.parser.mineruTierFlash')" />
+ <t-option value="basic" :label="$t('settings.parser.mineruTierBasic')" />
+ <t-option value="standard" :label="$t('settings.parser.mineruTierStandard')" />
+ <t-option value="advanced" :label="$t('settings.parser.mineruTierAdvanced')" />
+ </t-select>
+ <p class="form-desc">{{ $t('settings.parser.mineruTierHint') }}</p>
+ </div>
+ <div class="form-item">
+ <label class="form-label">{{ $t('settings.parser.parseMethodLabel') }}</label>
+ <t-select v-model="config.mineru_parse_method">
+ <t-option value="auto" :label="$t('settings.parser.parseMethodAuto')" />
+ <t-option value="ocr" :label="$t('settings.parser.parseMethodOCR')" />
+ <t-option value="txt" :label="$t('settings.parser.parseMethodText')" />
+ </t-select>
+ <p class="form-desc">{{ $t('settings.parser.parseMethodHint') }}</p>
+ </div>
+ </section>
+
+ <section v-if="currentEngine.Name === 'mineru'" class="setting-drawer__section">
+ <h4 class="setting-drawer__section-title">{{ $t('settings.parser.mineruLegacySection') }}</h4>
+ <p class="form-desc">{{ $t('settings.parser.mineruLegacySectionHint') }}</p>
+
<div class="form-item">
<label class="form-label">Backend</label>
<t-select v-model="config.mineru_model" :placeholder="$t('settings.parser.defaultPipeline')" clearable>
@@ -254,15 +292,6 @@
/>
<p class="form-desc">{{ $t('settings.parser.vlmServerUrlHint') }}</p>
</div>
- <div class="form-item">
- <label class="form-label">{{ $t('settings.parser.parseMethodLabel') }}</label>
- <t-select v-model="config.mineru_parse_method">
- <t-option value="auto" :label="$t('settings.parser.parseMethodAuto')" />
- <t-option value="ocr" :label="$t('settings.parser.parseMethodOCR')" />
- <t-option value="txt" :label="$t('settings.parser.parseMethodText')" />
- </t-select>
- <p class="form-desc">{{ $t('settings.parser.parseMethodHint') }}</p>
- </div>
<div class="form-item">
<label class="form-label">{{ $t('settings.parser.featuresLabel', '识别选项') }}</label>
<div class="form-toggles">
@@ -418,6 +447,8 @@ const DEFAULT_PARSER_CONFIG: ParserEngineConfig = {
docreader_transport: 'grpc',
mineru_endpoint: '',
mineru_api_key: '',
+ mineru_server_api_key: '',
+ mineru_tier: '',
mineru_model: 'pipeline',
mineru_vlm_server_url: '',
mineru_enable_formula: true,
@@ -557,6 +588,8 @@ async function loadConfig() {
docreader_transport: data?.docreader_transport ?? DEFAULT_PARSER_CONFIG.docreader_transport ?? 'grpc',
mineru_endpoint: data?.mineru_endpoint ?? DEFAULT_PARSER_CONFIG.mineru_endpoint ?? '',
mineru_api_key: data?.mineru_api_key ?? DEFAULT_PARSER_CONFIG.mineru_api_key ?? '',
+ mineru_server_api_key: data?.mineru_server_api_key ?? DEFAULT_PARSER_CONFIG.mineru_server_api_key ?? '',
+ mineru_tier: data?.mineru_tier ?? DEFAULT_PARSER_CONFIG.mineru_tier ?? '',
mineru_model: data?.mineru_model ?? DEFAULT_PARSER_CONFIG.mineru_model ?? '',
mineru_vlm_server_url: data?.mineru_vlm_server_url ?? DEFAULT_PARSER_CONFIG.mineru_vlm_server_url ?? '',
mineru_enable_formula: data?.mineru_enable_formula ?? DEFAULT_PARSER_CONFIG.mineru_enable_formula ?? true,
@@ -595,6 +628,8 @@ function buildConfigPayload(): ParserEngineConfig {
docreader_transport: (config.value.docreader_transport ?? 'grpc').trim() || 'grpc',
mineru_endpoint: config.value.mineru_endpoint?.trim() ?? '',
mineru_api_key: config.value.mineru_api_key?.trim() ?? '',
+ mineru_server_api_key: config.value.mineru_server_api_key?.trim() ?? '',
+ mineru_tier: config.value.mineru_tier?.trim() ?? '',
mineru_model: config.value.mineru_model?.trim() ?? '',
mineru_vlm_server_url: config.value.mineru_vlm_server_url?.trim() ?? '',
mineru_enable_formula: config.value.mineru_enable_formula,
diff --git a/internal/infrastructure/docparser/engines.go b/internal/infrastructure/docparser/engines.go
index 7883094f4..d3b28f8f4 100644
--- a/internal/infrastructure/docparser/engines.go
+++ b/internal/infrastructure/docparser/engines.go
@@ -208,7 +208,7 @@ func (e *mineruEngine) CheckAvailable(_ bool, overrides map[string]string) (bool
if endpoint == "" {
return false, "MinerU service not configured"
}
- return PingMinerU(endpoint)
+ return PingMinerU(endpoint, overrides["mineru_server_api_key"])
}
func (e *mineruEngine) NewReader(_ context.Context, deps ReaderDeps) (interfaces.DocReader, error) {
diff --git a/internal/infrastructure/docparser/mineru_cloud_converter.go b/internal/infrastructure/docparser/mineru_cloud_converter.go
index 133d2aec0..d3993c965 100644
--- a/internal/infrastructure/docparser/mineru_cloud_converter.go
+++ b/internal/infrastructure/docparser/mineru_cloud_converter.go
@@ -9,8 +9,8 @@ import (
"io"
"mime"
"net/http"
+ "net/url"
"path/filepath"
- "regexp"
"sort"
"strings"
"time"
@@ -350,8 +350,6 @@ func (c *MinerUCloudReader) extractDoneResult(_ context.Context, item *extractRe
// --- ZIP handling ---
-var imgRefPattern = regexp.MustCompile(`!\[[^\]]*\]\(([^)]+)\)`)
-
func downloadAndExtractZip(zipURL string) (string, []types.ImageRef, error) {
if err := utils.ValidateURLForSSRF(zipURL); err != nil {
return "", nil, fmt.Errorf("zip URL blocked by SSRF check: %v", err)
@@ -370,7 +368,12 @@ func downloadAndExtractZip(zipURL string) (string, []types.ImageRef, error) {
if err != nil {
return "", nil, fmt.Errorf("read zip body: %w", err)
}
+ return extractMarkdownZip(zipData, "MinerUCloud")
+}
+// extractMarkdownZip reads the shallowest .md file of a MinerU result package
+// and the images it references, via markdown or inline HTML <img> syntax.
+func extractMarkdownZip(zipData []byte, logLabel string) (string, []types.ImageRef, error) {
zr, err := zip.NewReader(bytes.NewReader(zipData), int64(len(zipData)))
if err != nil {
return "", nil, fmt.Errorf("open zip: %w", err)
@@ -406,8 +409,7 @@ func downloadAndExtractZip(zipURL string) (string, []types.ImageRef, error) {
// Extract referenced images
var imageRefs []types.ImageRef
seen := map[string]bool{}
- for _, match := range imgRefPattern.FindAllStringSubmatch(mdText, -1) {
- imgPath := match[1]
+ for _, imgPath := range extractImageRefsFromContent(mdText) {
if strings.HasPrefix(imgPath, "http://") || strings.HasPrefix(imgPath, "https://") || strings.HasPrefix(imgPath, "data:") {
continue
}
@@ -418,13 +420,13 @@ func downloadAndExtractZip(zipURL string) (string, []types.ImageRef, error) {
resolved := resolveInZip(imgPath, mdDir, entries)
if resolved == nil {
- logger.Errorf(context.Background(), "[MinerUCloud] image not found in zip: %s", imgPath)
+ logger.Errorf(context.Background(), "[%s] image not found in zip: %s", logLabel, imgPath)
continue
}
imgData, err := readZipEntryBytes(resolved)
if err != nil {
- logger.Errorf(context.Background(), "[MinerUCloud] failed to read zip image %s: %v", imgPath, err)
+ logger.Errorf(context.Background(), "[%s] failed to read zip image %s: %v", logLabel, imgPath, err)
continue
}
@@ -449,15 +451,20 @@ func downloadAndExtractZip(zipURL string) (string, []types.ImageRef, error) {
}
func resolveInZip(imgPath, mdDir string, entries map[string]*zip.File) *zip.File {
- normalized := strings.ReplaceAll(imgPath, "\\", "/")
- if f, ok := entries[normalized]; ok {
- return f
+ candidates := []string{strings.ReplaceAll(imgPath, "\\", "/")}
+ if decoded, err := url.PathUnescape(candidates[0]); err == nil && decoded != candidates[0] {
+ candidates = append(candidates, decoded)
}
- if mdDir != "" && mdDir != "." {
- joined := mdDir + "/" + normalized
- if f, ok := entries[joined]; ok {
+ for _, c := range candidates {
+ c = strings.TrimPrefix(c, "./")
+ if f, ok := entries[c]; ok {
return f
}
+ if mdDir != "" && mdDir != "." {
+ if f, ok := entries[mdDir+"/"+c]; ok {
+ return f
+ }
+ }
}
return nil
}
diff --git a/internal/infrastructure/docparser/mineru_converter.go b/internal/infrastructure/docparser/mineru_converter.go
index 1c7f1372a..ead9ac5d2 100644
--- a/internal/infrastructure/docparser/mineru_converter.go
+++ b/internal/infrastructure/docparser/mineru_converter.go
@@ -30,14 +30,21 @@ var (
)
// MinerUReader calls a self-hosted MinerU API to read/convert documents.
+// MinerU >= 4.0 is driven through the V1 API; older servers through the
+// legacy /file_parse endpoint. The protocol is detected per request, so an
+// in-place MinerU upgrade needs no configuration change.
type MinerUReader struct {
- endpoint string
+ endpoint string
+ apiKey string // V1 only: the server's --api-key
+ tier string // V1 only: flash, basic, standard, advanced; "" = server default
+ // Legacy (<= 3.x) only; MinerU 4.0 dropped these request parameters.
backend string // "pipeline", "vlm-*", "hybrid-*"
vlmServerURL string // vLLM server URL for vlm-http-client / hybrid-http-client
formulaEnable bool
tableEnable bool
- parseMethod string
language string
+ // Shared: legacy parse_method / V1 ocr_mode take the same values.
+ parseMethod string
}
// NewMinerUReader creates a reader from ParserEngineOverrides.
@@ -50,6 +57,8 @@ func NewMinerUReader(overrides map[string]string) *MinerUReader {
c := &MinerUReader{
endpoint: strings.TrimRight(overrides["mineru_endpoint"], "/"),
+ apiKey: strings.TrimSpace(overrides["mineru_server_api_key"]),
+ tier: resolveMinerUTier(overrides["mineru_tier"]),
backend: stringOr(overrides["mineru_model"], "pipeline"),
vlmServerURL: overrides["mineru_vlm_server_url"],
formulaEnable: parseBoolOr(overrides["mineru_enable_formula"], true),
@@ -60,6 +69,16 @@ func NewMinerUReader(overrides map[string]string) *MinerUReader {
return c
}
+// resolveMinerUTier keeps only tiers MinerU 4.0 understands; anything else
+// falls back to the server's default selection.
+func resolveMinerUTier(raw string) string {
+ tier := strings.ToLower(strings.TrimSpace(raw))
+ if _, ok := mineruV1Tiers[tier]; ok {
+ return tier
+ }
+ return ""
+}
+
func (c *MinerUReader) Read(ctx context.Context, req *types.ReadRequest) (*types.ReadResult, error) {
if c.endpoint == "" {
return &types.ReadResult{Error: "MinerU endpoint is not configured"}, nil
@@ -78,7 +97,21 @@ func (c *MinerUReader) Read(ctx context.Context, req *types.ReadRequest) (*types
return &types.ReadResult{Error: "no file content provided"}, nil
}
- logger.Infof(context.Background(), "[MinerU] Parsing file=%s size=%d via %s", req.FileName, len(content), c.endpoint)
+ protocol, err := detectMinerUProtocol(ctx, utils.NewSSRFSafeHTTPClient(utils.SSRFSafeHTTPClientConfig{
+ Timeout: mineruV1APITimeout,
+ MaxRedirects: 5,
+ }), c.endpoint)
+ if err != nil {
+ return nil, fmt.Errorf("MinerU protocol detection: %w", err)
+ }
+
+ if protocol == minerUProtocolV1 {
+ logger.Infof(ctx, "[MinerU] Parsing file=%s size=%d via %s (V1 API)", req.FileName, len(content), c.endpoint)
+ return c.readV1(ctx, req)
+ }
+
+ logger.Infof(ctx, "[MinerU] Parsing file=%s size=%d via %s (legacy /file_parse)",
+ req.FileName, len(content), c.endpoint)
mdContent, imagesB64, err := c.callFileParse(ctx, content, req.FileName, req.FileType)
if err != nil {
@@ -103,6 +136,25 @@ func (c *MinerUReader) Read(ctx context.Context, req *types.ReadRequest) (*types
}, nil
}
+func (c *MinerUReader) readV1(ctx context.Context, req *types.ReadRequest) (*types.ReadResult, error) {
+ client := newMinerUV1Client(c.endpoint, c.apiKey, mineruTimeout)
+ mdContent, imageRefs, err := client.Parse(ctx, req.FileContent, minerUUploadFileName(req.FileName, req.FileType),
+ minerUV1ParseOptions{Tier: c.tier, OCRMode: c.parseMethod})
+ if err != nil {
+ return nil, fmt.Errorf("MinerU V1: %w", err)
+ }
+
+ mdContent = normalizeMinerUMarkdown(mdContent)
+ mdContent, imageRefs = ensureOriginalImageRef(req, mdContent, imageRefs)
+
+ logger.Infof(ctx, "[MinerU] Parsed successfully (V1), markdown=%d chars, images=%d", len(mdContent), len(imageRefs))
+
+ return &types.ReadResult{
+ MarkdownContent: mdContent,
+ ImageRefs: imageRefs,
+ }, nil
+}
+
type mineruFileEntry struct {
MDContent string `json:"md_content"`
Images map[string]string `json:"images"` // path -> "data:image/png;base64,..." or raw base64
@@ -358,8 +410,10 @@ func validateMinerUOutboundURL(rawURL string) error {
return nil
}
-// PingMinerU checks if the self-hosted MinerU service is reachable.
-func PingMinerU(endpoint string) (bool, string) {
+// PingMinerU checks if the self-hosted MinerU service is reachable. For
+// MinerU 4.0 it also verifies the configured API key, since /v1/health itself
+// is served without authentication.
+func PingMinerU(endpoint, apiKey string) (bool, string) {
endpoint = strings.TrimRight(endpoint, "/")
if endpoint == "" {
return false, "未配置 MinerU 端点"
@@ -371,6 +425,16 @@ func PingMinerU(endpoint string) (bool, string) {
Timeout: 5 * time.Second,
MaxRedirects: 5,
})
+ ctx := context.Background()
+
+ protocol, err := detectMinerUProtocol(ctx, client, endpoint)
+ if err != nil {
+ return false, fmt.Sprintf("MinerU 服务不可用: %v", err)
+ }
+ if protocol == minerUProtocolV1 {
+ return pingMinerUV1Auth(ctx, client, endpoint, apiKey)
+ }
+
resp, err := client.Get(endpoint + "/docs")
if err != nil {
return false, fmt.Sprintf("MinerU 服务不可达: %v", err)
@@ -382,6 +446,30 @@ func PingMinerU(endpoint string) (bool, string) {
return true, ""
}
+// pingMinerUV1Auth hits an authenticated V1 endpoint: a server started with
+// --api-key answers 401 when the key is missing or wrong.
+func pingMinerUV1Auth(ctx context.Context, client *http.Client, endpoint, apiKey string) (bool, string) {
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint+"/v1/parse/jobs?limit=1", nil)
+ if err != nil {
+ return false, fmt.Sprintf("构建请求失败: %v", err)
+ }
+ if apiKey = strings.TrimSpace(apiKey); apiKey != "" {
+ req.Header.Set("Authorization", "Bearer "+apiKey)
+ }
+ resp, err := client.Do(req)
+ if err != nil {
+ return false, fmt.Sprintf("MinerU 服务不可达: %v", err)
+ }
+ _ = resp.Body.Close()
+ if resp.StatusCode == http.StatusUnauthorized {
+ if apiKey == "" {
+ return false, "MinerU 服务已启用鉴权,请配置 API Key"
+ }
+ return false, "MinerU API Key 无效"
+ }
+ return true, ""
+}
+
// escapedImageSyntaxPattern matches markdown image references whose `[` was
// over-escaped to `\[` by html-to-markdown. The URL group mirrors the
// downstream image-extraction regex so escapes are only stripped for actual
diff --git a/internal/infrastructure/docparser/mineru_converter_ssrf_test.go b/internal/infrastructure/docparser/mineru_converter_ssrf_test.go
index 133e0be47..cb6d29ceb 100644
--- a/internal/infrastructure/docparser/mineru_converter_ssrf_test.go
+++ b/internal/infrastructure/docparser/mineru_converter_ssrf_test.go
@@ -14,7 +14,7 @@ func TestValidateMinerUOutboundURL_RejectsLoopback(t *testing.T) {
}
func TestPingMinerU_RejectsPrivateEndpoint(t *testing.T) {
- ok, msg := PingMinerU("http://127.0.0.1:8080")
+ ok, msg := PingMinerU("http://127.0.0.1:8080", "")
assert.False(t, ok)
assert.Contains(t, msg, "SSRF")
}
diff --git a/internal/infrastructure/docparser/mineru_converter_test.go b/internal/infrastructure/docparser/mineru_converter_test.go
index 4310d9fdb..e25dab023 100644
--- a/internal/infrastructure/docparser/mineru_converter_test.go
+++ b/internal/infrastructure/docparser/mineru_converter_test.go
@@ -183,6 +183,11 @@ func TestMinerUReaderPreservesMultipartFilename(t *testing.T) {
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if r.URL.Path == "/v1/health" {
+ // MinerU <= 3.x has no V1 API.
+ http.NotFound(w, r)
+ return
+ }
got := capturedRequest{method: r.Method, path: r.URL.Path}
file, header, err := r.FormFile("files")
if err != nil {
diff --git a/internal/infrastructure/docparser/mineru_v1_client.go b/internal/infrastructure/docparser/mineru_v1_client.go
new file mode 100644
index 000000000..0eae3915e
--- /dev/null
+++ b/internal/infrastructure/docparser/mineru_v1_client.go
@@ -0,0 +1,594 @@
+package docparser
+
+import (
+ "bytes"
+ "context"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "mime"
+ "net/http"
+ "net/url"
+ "path/filepath"
+ "strings"
+ "time"
+
+ "github.com/Tencent/WeKnora/internal/logger"
+ "github.com/Tencent/WeKnora/internal/types"
+ "github.com/Tencent/WeKnora/internal/utils"
+)
+
+// MinerU 4.0 replaced the synchronous /file_parse endpoint with the V1 API:
+// upload → parse job → poll → download artifacts. See
+// https://opendatalab.github.io/MinerU/usage/http_api/.
+
+const (
+ mineruV1PollInitial = 2 * time.Second
+ mineruV1PollMax = 30 * time.Second
+ mineruV1APITimeout = 30 * time.Second
+ mineruV1MaxZipBytes = 1 << 30
+ mineruV1CancelBudget = 10 * time.Second
+)
+
+// MinerU V1 tiers accepted by POST /v1/parse/jobs. An empty tier means the
+// server's default selection policy (standard → basic for PDF/image).
+var mineruV1Tiers = map[string]struct{}{
+ "flash": {},
+ "basic": {},
+ "standard": {},
+ "advanced": {},
+}
+
+type minerUProtocol int
+
+const (
+ minerUProtocolLegacy minerUProtocol = iota // MinerU <= 3.x: POST /file_parse
+ minerUProtocolV1 // MinerU >= 4.0: /v1/*
+)
+
+// minerUV1Client talks to any MinerU V1 deployment (self-hosted api-server or
+// the official mineru.net/api).
+type minerUV1Client struct {
+ baseURL string // without the /v1 suffix
+ apiKey string
+ apiClient *http.Client // small JSON calls
+ bulkClient *http.Client // byte upload and artifact download
+ pollInitial time.Duration
+ pollMax time.Duration
+ jobTimeout time.Duration
+ maxZipBytes int64
+ logLabel string
+ cancelBudget time.Duration
+}
+
+type minerUV1ParseOptions struct {
+ Tier string // "" → server default
+ OCRMode string // auto, txt, ocr
+}
+
+func newMinerUV1Client(baseURL, apiKey string, jobTimeout time.Duration) *minerUV1Client {
+ return &minerUV1Client{
+ baseURL: strings.TrimRight(baseURL, "/"),
+ apiKey: strings.TrimSpace(apiKey),
+ apiClient: utils.NewSSRFSafeHTTPClient(utils.SSRFSafeHTTPClientConfig{
+ Timeout: mineruV1APITimeout,
+ MaxRedirects: 5,
+ }),
+ bulkClient: utils.NewSSRFSafeHTTPClient(utils.SSRFSafeHTTPClientConfig{
+ Timeout: jobTimeout,
+ MaxRedirects: 5,
+ }),
+ pollInitial: mineruV1PollInitial,
+ pollMax: mineruV1PollMax,
+ jobTimeout: jobTimeout,
+ maxZipBytes: mineruV1MaxZipBytes,
+ logLabel: "MinerU",
+ cancelBudget: mineruV1CancelBudget,
+ }
+}
+
+// --- wire types ---
+
+type minerUV1ErrorEnvelope struct {
+ Error struct {
+ Message string `json:"message"`
+ Type string `json:"type"`
+ Code string `json:"code"`
+ } `json:"error"`
+}
+
+type minerUV1File struct {
+ ID string `json:"id"`
+}
+
+type minerUV1Upload struct {
+ ID string `json:"id"`
+ Status string `json:"status"`
+ UploadURL string `json:"upload_url"`
+ UploadMethod string `json:"upload_method"`
+ UploadHeaders map[string]string `json:"upload_headers"`
+ File *minerUV1File `json:"file"`
+}
+
+type minerUV1FileRef struct {
+ FileID string `json:"file_id"`
+ Bytes int64 `json:"bytes"`
+}
+
+type minerUV1JobFile struct {
+ Name string `json:"name"`
+ Status string `json:"status"`
+ OutputFiles map[string]minerUV1FileRef `json:"output_files"`
+ Error *struct {
+ Code string `json:"code"`
+ Message string `json:"message"`
+ } `json:"error"`
+ Parse *struct {
+ ModelUsed string `json:"model_used"`
+ DurationMS int64 `json:"duration_ms"`
+ ParserVersion string `json:"parser_version"`
+ } `json:"parse"`
+}
+
+type minerUV1Job struct {
+ JobID string `json:"job_id"`
+ Status string `json:"status"`
+ Tier string `json:"tier"`
+ Files []minerUV1JobFile `json:"files"`
+}
+
+func minerUV1JobTerminal(status string) bool {
+ switch status {
+ case "completed", "partial", "failed", "canceled":
+ return true
+ }
+ return false
+}
+
+// --- protocol detection ---
+
+// detectMinerUProtocol probes GET /v1/health. MinerU 4.0 answers 200 (or 503
+// while models are still preloading); older servers have no such route and
+// answer 404/405.
+func detectMinerUProtocol(ctx context.Context, client *http.Client, endpoint string) (minerUProtocol, error) {
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint+"/v1/health", nil)
+ if err != nil {
+ return minerUProtocolLegacy, fmt.Errorf("create health request: %w", err)
+ }
+ resp, err := client.Do(req)
+ if err != nil {
+ return minerUProtocolLegacy, fmt.Errorf("MinerU service unreachable: %w", err)
+ }
+ defer func() { _ = resp.Body.Close() }()
+ body, _ := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
+
+ switch resp.StatusCode {
+ case http.StatusOK:
+ var health struct {
+ Status string `json:"status"`
+ }
+ if err := json.Unmarshal(body, &health); err != nil || health.Status != "ok" {
+ return minerUProtocolLegacy, fmt.Errorf("unexpected /v1/health response: %s",
+ truncateForLog(string(body), 300))
+ }
+ return minerUProtocolV1, nil
+ case http.StatusNotFound, http.StatusMethodNotAllowed:
+ return minerUProtocolLegacy, nil
+ default:
+ if msg := minerUV1ErrorMessage(body); msg != "" {
+ return minerUProtocolV1, fmt.Errorf("MinerU service not ready (status %d): %s", resp.StatusCode, msg)
+ }
+ return minerUProtocolLegacy, fmt.Errorf("MinerU /v1/health status %d: %s",
+ resp.StatusCode, truncateForLog(string(body), 300))
+ }
+}
+
+func minerUV1ErrorMessage(body []byte) string {
+ var env minerUV1ErrorEnvelope
+ if err := json.Unmarshal(body, &env); err != nil {
+ return ""
+ }
+ if env.Error.Code != "" && env.Error.Message != "" {
+ return env.Error.Code + ": " + env.Error.Message
+ }
+ return firstNonEmpty(env.Error.Message, env.Error.Code)
+}
+
+func truncateForLog(s string, n int) string {
+ if len(s) <= n {
+ return s
+ }
+ return s[:n] + "..."
+}
+
+// --- parse flow ---
+
+// Parse uploads content, runs a parse job and returns the markdown and
+// images extracted from the job's zip artifact.
+func (c *minerUV1Client) Parse(
+ ctx context.Context,
+ content []byte,
+ fileName string,
+ opts minerUV1ParseOptions,
+) (string, []types.ImageRef, error) {
+ ctx, cancel := context.WithTimeout(ctx, c.jobTimeout)
+ defer cancel()
+
+ fileID, err := c.upload(ctx, content, fileName)
+ if err != nil {
+ return "", nil, fmt.Errorf("upload: %w", err)
+ }
+
+ jobID, err := c.createJob(ctx, fileID, opts)
+ if err != nil {
+ return "", nil, fmt.Errorf("create parse job: %w", err)
+ }
+ logger.Infof(ctx, "[%s] parse job created: job_id=%s file_id=%s tier=%q ocr_mode=%s",
+ c.logLabel, jobID, fileID, opts.Tier, opts.OCRMode)
+
+ job, err := c.waitJob(ctx, jobID)
+ if err != nil {
+ c.cancelJob(jobID)
+ return "", nil, err
+ }
+
+ zipRef, err := minerUV1ZipOutput(job)
+ if err != nil {
+ return "", nil, err
+ }
+
+ zipData, err := c.downloadFile(ctx, zipRef.FileID)
+ if err != nil {
+ return "", nil, fmt.Errorf("download zip artifact: %w", err)
+ }
+
+ md, imageRefs, err := extractMarkdownZip(zipData, c.logLabel)
+ if err != nil {
+ return "", nil, fmt.Errorf("extract zip artifact: %w", err)
+ }
+ return md, imageRefs, nil
+}
+
+func (c *minerUV1Client) upload(ctx context.Context, content []byte, fileName string) (string, error) {
+ sum := sha256.Sum256(content)
+ sha := hex.EncodeToString(sum[:])
+
+ var created minerUV1Upload
+ err := c.doJSON(ctx, http.MethodPost, "/v1/uploads", map[string]interface{}{
+ "filename": fileName,
+ "bytes": len(content),
+ "mime_type": minerUV1MimeType(fileName),
+ "purpose": "parse",
+ "sha256sum": sha,
+ }, &created)
+ if err != nil {
+ return "", err
+ }
+
+ switch created.Status {
+ case "completed":
+ // Same bytes already stored on the server: no transfer needed.
+ if created.File == nil || created.File.ID == "" {
+ return "", fmt.Errorf("upload completed without file id")
+ }
+ return created.File.ID, nil
+ case "pending":
+ default:
+ return "", fmt.Errorf("unexpected upload status %q", created.Status)
+ }
+ if created.ID == "" || created.UploadURL == "" {
+ return "", fmt.Errorf("pending upload missing id or upload_url")
+ }
+
+ if err := c.putBytes(ctx, &created, content); err != nil {
+ return "", err
+ }
+
+ var completed minerUV1Upload
+ err = c.doJSON(ctx, http.MethodPost, "/v1/uploads/"+url.PathEscape(created.ID)+"/complete",
+ map[string]string{"sha256sum": sha}, &completed)
+ if err != nil {
+ return "", fmt.Errorf("complete upload: %w", err)
+ }
+ if completed.File == nil || completed.File.ID == "" {
+ return "", fmt.Errorf("complete upload returned no file id (status %q)", completed.Status)
+ }
+ return completed.File.ID, nil
+}
+
+// putBytes sends the file to the URL returned by POST /v1/uploads. The MinerU
+// API key is attached only when that URL is same-origin with the configured
+// endpoint: the official API hands out pre-signed object-storage URLs that
+// carry their own authorization and must never receive our key.
+func (c *minerUV1Client) putBytes(ctx context.Context, up *minerUV1Upload, content []byte) error {
+ target, err := c.resolveURL(up.UploadURL)
+ if err != nil {
+ return fmt.Errorf("invalid upload_url: %w", err)
+ }
+ method := strings.ToUpper(stringOr(up.UploadMethod, http.MethodPut))
+
+ req, err := http.NewRequestWithContext(ctx, method, target.String(), bytes.NewReader(content))
+ if err != nil {
+ return fmt.Errorf("create upload request: %w", err)
+ }
+ for k, v := range up.UploadHeaders {
+ req.Header.Set(k, v)
+ }
+ if c.sameOrigin(target) {
+ c.setAuth(req)
+ }
+
+ resp, err := c.bulkClient.Do(req)
+ if err != nil {
+ return fmt.Errorf("upload bytes: %w", err)
+ }
+ defer func() { _ = resp.Body.Close() }()
+ if resp.StatusCode >= 300 {
+ body, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))
+ return fmt.Errorf("upload bytes status %d: %s", resp.StatusCode, truncateForLog(string(body), 300))
+ }
+ return nil
+}
+
+func (c *minerUV1Client) createJob(ctx context.Context, fileID string, opts minerUV1ParseOptions) (string, error) {
+ payload := map[string]interface{}{
+ "files": []map[string]interface{}{
+ {"source": map[string]string{"type": "file_id", "file_id": fileID}},
+ },
+ "output_formats": []string{"zip"},
+ }
+ if opts.Tier != "" {
+ payload["tier"] = opts.Tier
+ }
+ if opts.OCRMode != "" {
+ payload["ocr_mode"] = opts.OCRMode
+ }
+
+ var job minerUV1Job
+ if err := c.doJSON(ctx, http.MethodPost, "/v1/parse/jobs", payload, &job); err != nil {
+ return "", err
+ }
+ if job.JobID == "" {
+ return "", fmt.Errorf("response has no job_id")
+ }
+ return job.JobID, nil
+}
+
+func (c *minerUV1Client) waitJob(ctx context.Context, jobID string) (*minerUV1Job, error) {
+ interval := c.pollInitial
+ for poll := 1; ; poll++ {
+ var job minerUV1Job
+ err := c.doJSON(ctx, http.MethodGet, "/v1/parse/jobs/"+url.PathEscape(jobID), nil, &job)
+ if err != nil {
+ if ctxErr := ctx.Err(); ctxErr != nil {
+ return nil, fmt.Errorf("parse job %s: %w", jobID, ctxErr)
+ }
+ var apiErr *minerUV1APIError
+ if errors.As(err, &apiErr) && apiErr.status == http.StatusNotFound {
+ // Job state lives in process memory; a restarted server forgets it.
+ return nil, fmt.Errorf("parse job %s no longer exists (MinerU restarted?): %w", jobID, err)
+ }
+ logger.Warnf(ctx, "[%s] poll #%d for job %s failed: %v", c.logLabel, poll, jobID, err)
+ } else {
+ if poll == 1 || poll%10 == 0 || minerUV1JobTerminal(job.Status) {
+ logger.Infof(ctx, "[%s] poll #%d: job=%s status=%s", c.logLabel, poll, jobID, job.Status)
+ }
+ if minerUV1JobTerminal(job.Status) {
+ return &job, nil
+ }
+ }
+
+ sleepCtx(ctx, interval)
+ if ctxErr := ctx.Err(); ctxErr != nil {
+ return nil, fmt.Errorf("parse job %s: %w", jobID, ctxErr)
+ }
+ interval *= 2
+ if interval > c.pollMax {
+ interval = c.pollMax
+ }
+ }
+}
+
+// cancelJob best-effort cancels a job we stopped waiting for so the server
+// does not keep burning GPU time on it. It runs on a fresh context because the
+// caller's context is usually already done.
+func (c *minerUV1Client) cancelJob(jobID string) {
+ ctx, cancel := context.WithTimeout(context.Background(), c.cancelBudget)
+ defer cancel()
+ if err := c.doJSON(ctx, http.MethodDelete, "/v1/parse/jobs/"+url.PathEscape(jobID), nil, nil); err != nil {
+ logger.Warnf(ctx, "[%s] cancel job %s: %v", c.logLabel, jobID, err)
+ }
+}
+
+// minerUV1ZipOutput returns the zip artifact of the single submitted file, or
+// the per-file error when parsing did not succeed.
+func minerUV1ZipOutput(job *minerUV1Job) (minerUV1FileRef, error) {
+ if len(job.Files) == 0 {
+ return minerUV1FileRef{}, fmt.Errorf("parse job %s ended %s without file results", job.JobID, job.Status)
+ }
+ f := job.Files[0]
+ if f.Status != "completed" {
+ reason := job.Status
+ if f.Error != nil {
+ reason = firstNonEmpty(strings.TrimSpace(f.Error.Code+": "+f.Error.Message), job.Status)
+ }
+ return minerUV1FileRef{}, fmt.Errorf("parse job %s %s: %s", job.JobID, job.Status, reason)
+ }
+ ref, ok := f.OutputFiles["zip"]
+ if !ok || ref.FileID == "" {
+ return minerUV1FileRef{}, fmt.Errorf("parse job %s completed without zip output", job.JobID)
+ }
+ return ref, nil
+}
+
+func (c *minerUV1Client) downloadFile(ctx context.Context, fileID string) ([]byte, error) {
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet,
+ c.baseURL+"/v1/files/"+url.PathEscape(fileID)+"/content", nil)
+ if err != nil {
+ return nil, err
+ }
+ // Go drops Authorization when following a redirect to another host, so a
+ // 302 to a CDN (official API) does not leak the key.
+ c.setAuth(req)
+
+ resp, err := c.bulkClient.Do(req)
+ if err != nil {
+ return nil, err
+ }
+ defer func() { _ = resp.Body.Close() }()
+ if resp.StatusCode != http.StatusOK {
+ body, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))
+ return nil, newMinerUV1APIError(resp.StatusCode, body)
+ }
+ data, err := io.ReadAll(io.LimitReader(resp.Body, c.maxZipBytes+1))
+ if err != nil {
+ return nil, err
+ }
+ if int64(len(data)) > c.maxZipBytes {
+ return nil, fmt.Errorf("artifact exceeds %d bytes", c.maxZipBytes)
+ }
+ return data, nil
+}
+
+// --- HTTP helpers ---
+
+type minerUV1APIError struct {
+ status int
+ msg string
+}
+
+func (e *minerUV1APIError) Error() string {
+ return fmt.Sprintf("MinerU API status %d: %s", e.status, e.msg)
+}
+
+func newMinerUV1APIError(status int, body []byte) *minerUV1APIError {
+ msg := minerUV1ErrorMessage(body)
+ if msg == "" {
+ msg = truncateForLog(string(body), 500)
+ }
+ return &minerUV1APIError{status: status, msg: msg}
+}
+
+func (c *minerUV1Client) doJSON(ctx context.Context, method, path string, in, out interface{}) error {
+ var body io.Reader
+ if in != nil {
+ raw, err := json.Marshal(in)
+ if err != nil {
+ return fmt.Errorf("marshal request: %w", err)
+ }
+ body = bytes.NewReader(raw)
+ }
+ req, err := http.NewRequestWithContext(ctx, method, c.baseURL+path, body)
+ if err != nil {
+ return fmt.Errorf("create request: %w", err)
+ }
+ if in != nil {
+ req.Header.Set("Content-Type", "application/json")
+ }
+ c.setAuth(req)
+
+ resp, err := c.apiClient.Do(req)
+ if err != nil {
+ return fmt.Errorf("%s %s: %w", method, path, err)
+ }
+ defer func() { _ = resp.Body.Close() }()
+ respBody, err := io.ReadAll(io.LimitReader(resp.Body, 16<<20))
+ if err != nil {
+ return fmt.Errorf("read response: %w", err)
+ }
+ if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+ return newMinerUV1APIError(resp.StatusCode, respBody)
+ }
+ if out == nil {
+ return nil
+ }
+ if err := json.Unmarshal(respBody, out); err != nil {
+ return fmt.Errorf("decode %s %s response: %w", method, path, err)
+ }
+ return nil
+}
+
+func (c *minerUV1Client) setAuth(req *http.Request) {
+ if c.apiKey != "" {
+ req.Header.Set("Authorization", "Bearer "+c.apiKey)
+ }
+}
+
+func (c *minerUV1Client) resolveURL(raw string) (*url.URL, error) {
+ base, err := url.Parse(c.baseURL + "/")
+ if err != nil {
+ return nil, err
+ }
+ ref, err := url.Parse(strings.TrimSpace(raw))
+ if err != nil {
+ return nil, err
+ }
+ resolved := base.ResolveReference(ref)
+ if resolved.Scheme != "http" && resolved.Scheme != "https" {
+ return nil, fmt.Errorf("unsupported scheme %q", resolved.Scheme)
+ }
+ return resolved, nil
+}
+
+// sameOrigin compares scheme, host and effective port with the base URL.
+func (c *minerUV1Client) sameOrigin(u *url.URL) bool {
+ base, err := url.Parse(c.baseURL)
+ if err != nil {
+ return false
+ }
+ return strings.EqualFold(base.Scheme, u.Scheme) &&
+ strings.EqualFold(base.Hostname(), u.Hostname()) &&
+ effectivePort(base) == effectivePort(u)
+}
+
+func effectivePort(u *url.URL) string {
+ if p := u.Port(); p != "" {
+ return p
+ }
+ switch strings.ToLower(u.Scheme) {
+ case "https":
+ return "443"
+ case "http":
+ return "80"
+ }
+ return ""
+}
+
+// Slim container images often ship without /etc/mime.types, where Go only
+// knows a handful of extensions; cover the formats MinerU accepts explicitly.
+var minerUV1MimeTypes = map[string]string{
+ ".pdf": "application/pdf",
+ ".doc": "application/msword",
+ ".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
+ ".ppt": "application/vnd.ms-powerpoint",
+ ".pptx": "application/vnd.openxmlformats-officedocument.presentationml.presentation",
+ ".xls": "application/vnd.ms-excel",
+ ".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
+ ".png": "image/png",
+ ".jpg": "image/jpeg",
+ ".jpeg": "image/jpeg",
+ ".bmp": "image/bmp",
+ ".tif": "image/tiff",
+ ".tiff": "image/tiff",
+ ".html": "text/html",
+ ".htm": "text/html",
+}
+
+func minerUV1MimeType(fileName string) string {
+ ext := strings.ToLower(filepath.Ext(fileName))
+ if t, ok := minerUV1MimeTypes[ext]; ok {
+ return t
+ }
+ if ext != "" {
+ if t := mime.TypeByExtension(ext); t != "" {
+ if mediaType, _, err := mime.ParseMediaType(t); err == nil {
+ return mediaType
+ }
+ return t
+ }
+ }
+ return "application/octet-stream"
+}
diff --git a/internal/infrastructure/docparser/mineru_v1_client_test.go b/internal/infrastructure/docparser/mineru_v1_client_test.go
new file mode 100644
index 000000000..4f8c0b915
--- /dev/null
+++ b/internal/infrastructure/docparser/mineru_v1_client_test.go
@@ -0,0 +1,433 @@
+package docparser
+
+import (
+ "archive/zip"
+ "bytes"
+ "context"
+ "encoding/json"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/Tencent/WeKnora/internal/types"
+ "github.com/Tencent/WeKnora/internal/utils"
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+// fakeMinerUV1 is a minimal in-memory MinerU 4.0 api-server.
+type fakeMinerUV1 struct {
+ t *testing.T
+ apiKey string
+
+ mu sync.Mutex
+ uploadURL string // absolute or relative; defaults to same-origin relative path
+ dedupHit bool // POST /v1/uploads answers "completed" directly
+ uploaded []byte
+ putAuth string
+ uploadMime string
+ jobPayload map[string]any
+ pollStatuses []string // statuses returned by successive GET job calls
+ polls int
+ fileError *map[string]string
+ canceled bool
+ zipData []byte
+ downloadCalls int
+}
+
+func (f *fakeMinerUV1) handler() http.Handler {
+ mux := http.NewServeMux()
+ writeJSON := func(w http.ResponseWriter, status int, v any) {
+ w.Header().Set("Content-Type", "application/json")
+ w.WriteHeader(status)
+ _ = json.NewEncoder(w).Encode(v)
+ }
+ authed := func(next http.HandlerFunc) http.HandlerFunc {
+ return func(w http.ResponseWriter, r *http.Request) {
+ if f.apiKey != "" && r.Header.Get("Authorization") != "Bearer "+f.apiKey {
+ writeJSON(w, http.StatusUnauthorized, map[string]any{
+ "error": map[string]string{"message": "bad key", "type": "auth_error", "code": "invalid_api_key"},
+ })
+ return
+ }
+ next(w, r)
+ }
+ }
+
+ mux.HandleFunc("GET /v1/health", func(w http.ResponseWriter, _ *http.Request) {
+ writeJSON(w, http.StatusOK, map[string]any{"status": "ok", "version": "1.0.0"})
+ })
+ mux.HandleFunc("POST /v1/uploads", authed(func(w http.ResponseWriter, r *http.Request) {
+ var body map[string]any
+ require.NoError(f.t, json.NewDecoder(r.Body).Decode(&body))
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ f.uploadMime, _ = body["mime_type"].(string)
+ if f.dedupHit {
+ writeJSON(w, http.StatusOK, map[string]any{
+ "id": "upload_1", "status": "completed", "file": map[string]any{"id": "file-src"},
+ })
+ return
+ }
+ uploadURL := f.uploadURL
+ if uploadURL == "" {
+ uploadURL = "/v1/uploads/upload_1/content"
+ }
+ writeJSON(w, http.StatusOK, map[string]any{
+ "id": "upload_1", "status": "pending", "upload_url": uploadURL, "upload_method": "PUT",
+ "upload_headers": map[string]string{"Content-Type": f.uploadMime},
+ })
+ }))
+ mux.HandleFunc("PUT /v1/uploads/upload_1/content", f.handlePut)
+ mux.HandleFunc("POST /v1/uploads/upload_1/complete", authed(func(w http.ResponseWriter, _ *http.Request) {
+ writeJSON(w, http.StatusOK, map[string]any{
+ "id": "upload_1", "status": "completed", "file": map[string]any{"id": "file-src"},
+ })
+ }))
+ mux.HandleFunc("POST /v1/parse/jobs", authed(func(w http.ResponseWriter, r *http.Request) {
+ var body map[string]any
+ require.NoError(f.t, json.NewDecoder(r.Body).Decode(&body))
+ f.mu.Lock()
+ f.jobPayload = body
+ f.mu.Unlock()
+ writeJSON(w, http.StatusAccepted, map[string]any{"job_id": "job_1", "status": "queued"})
+ }))
+ mux.HandleFunc("GET /v1/parse/jobs/job_1", authed(func(w http.ResponseWriter, _ *http.Request) {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ status := "completed"
+ if f.polls < len(f.pollStatuses) {
+ status = f.pollStatuses[f.polls]
+ }
+ f.polls++
+ file := map[string]any{"name": "report.pdf", "status": status}
+ switch status {
+ case "completed":
+ file["output_files"] = map[string]any{"zip": map[string]any{"file_id": "file-zip", "bytes": len(f.zipData)}}
+ case "failed":
+ file["error"] = *f.fileError
+ }
+ writeJSON(w, http.StatusOK, map[string]any{"job_id": "job_1", "status": status, "files": []any{file}})
+ }))
+ mux.HandleFunc("DELETE /v1/parse/jobs/job_1", authed(func(w http.ResponseWriter, _ *http.Request) {
+ f.mu.Lock()
+ f.canceled = true
+ f.mu.Unlock()
+ writeJSON(w, http.StatusOK, map[string]any{"job_id": "job_1", "status": "canceled"})
+ }))
+ mux.HandleFunc("GET /v1/parse/jobs", authed(func(w http.ResponseWriter, _ *http.Request) {
+ writeJSON(w, http.StatusOK, map[string]any{"object": "list", "data": []any{}})
+ }))
+ mux.HandleFunc("GET /v1/files/file-zip/content", authed(func(w http.ResponseWriter, _ *http.Request) {
+ f.mu.Lock()
+ f.downloadCalls++
+ f.mu.Unlock()
+ w.Header().Set("Content-Type", "application/zip")
+ _, _ = w.Write(f.zipData)
+ }))
+ return mux
+}
+
+func (f *fakeMinerUV1) handlePut(w http.ResponseWriter, r *http.Request) {
+ data, err := io.ReadAll(r.Body)
+ require.NoError(f.t, err)
+ f.mu.Lock()
+ f.uploaded = data
+ f.putAuth = r.Header.Get("Authorization")
+ f.mu.Unlock()
+ w.WriteHeader(http.StatusOK)
+}
+
+func buildMinerUZip(t *testing.T, files map[string]string) []byte {
+ t.Helper()
+ var buf bytes.Buffer
+ zw := zip.NewWriter(&buf)
+ for name, content := range files {
+ w, err := zw.Create(name)
+ require.NoError(t, err)
+ _, err = w.Write([]byte(content))
+ require.NoError(t, err)
+ }
+ require.NoError(t, zw.Close())
+ return buf.Bytes()
+}
+
+func allowLoopbackSSRF(t *testing.T) {
+ t.Helper()
+ utils.SetSSRFWhitelistFromRaw("127.0.0.1")
+ t.Cleanup(func() { utils.SetSSRFWhitelistFromRaw("") })
+}
+
+func fastV1Client(baseURL, apiKey string) *minerUV1Client {
+ c := newMinerUV1Client(baseURL, apiKey, 10*time.Second)
+ c.pollInitial = time.Millisecond
+ c.pollMax = 5 * time.Millisecond
+ return c
+}
+
+const v1TestMarkdown = "# Report\n\n![](images/page_0_image_1.png)\n\n" +
+ "<table><tr><td><img src=\"images/page_0_table_image_2_1.png\"></td></tr></table>\n"
+
+func v1TestZip(t *testing.T) []byte {
+ return buildMinerUZip(t, map[string]string{
+ "markdown.md": v1TestMarkdown,
+ "middle_json.json": "{}",
+ "images/page_0_image_1.png": "png-bytes-1",
+ "images/page_0_table_image_2_1.png": "png-bytes-2",
+ "images/page_1_image_unreferenced.png": "unused",
+ })
+}
+
+func TestMinerUV1ClientParseFullFlow(t *testing.T) {
+ allowLoopbackSSRF(t)
+ fake := &fakeMinerUV1{
+ t: t,
+ apiKey: "server-key",
+ pollStatuses: []string{"queued", "running", "completed"},
+ zipData: v1TestZip(t),
+ }
+ server := httptest.NewServer(fake.handler())
+ defer server.Close()
+
+ client := fastV1Client(server.URL, "server-key")
+ md, refs, err := client.Parse(context.Background(), []byte("%PDF-1.7 body"), "report.pdf",
+ minerUV1ParseOptions{Tier: "advanced", OCRMode: "txt"})
+ require.NoError(t, err)
+
+ assert.Equal(t, v1TestMarkdown, md)
+ require.Len(t, refs, 2)
+ byRef := map[string]types.ImageRef{}
+ for _, ref := range refs {
+ byRef[ref.OriginalRef] = ref
+ }
+ assert.Equal(t, []byte("png-bytes-1"), byRef["images/page_0_image_1.png"].ImageData)
+ assert.Equal(t, []byte("png-bytes-2"), byRef["images/page_0_table_image_2_1.png"].ImageData)
+ assert.Equal(t, "image/png", byRef["images/page_0_image_1.png"].MimeType)
+
+ assert.Equal(t, []byte("%PDF-1.7 body"), fake.uploaded)
+ assert.Equal(t, "Bearer server-key", fake.putAuth, "same-origin upload must carry the API key")
+ assert.Equal(t, "application/pdf", fake.uploadMime)
+ assert.Equal(t, 3, fake.polls)
+ assert.False(t, fake.canceled)
+
+ assert.Equal(t, "advanced", fake.jobPayload["tier"])
+ assert.Equal(t, "txt", fake.jobPayload["ocr_mode"])
+ assert.Equal(t, []any{"zip"}, fake.jobPayload["output_formats"])
+ files := fake.jobPayload["files"].([]any)
+ source := files[0].(map[string]any)["source"].(map[string]any)
+ assert.Equal(t, map[string]any{"type": "file_id", "file_id": "file-src"}, source)
+}
+
+func TestMinerUV1ClientOmitsTierForServerDefault(t *testing.T) {
+ allowLoopbackSSRF(t)
+ fake := &fakeMinerUV1{t: t, zipData: v1TestZip(t)}
+ server := httptest.NewServer(fake.handler())
+ defer server.Close()
+
+ _, _, err := fastV1Client(server.URL, "").Parse(context.Background(), []byte("x"), "a.pdf",
+ minerUV1ParseOptions{OCRMode: "auto"})
+ require.NoError(t, err)
+ _, hasTier := fake.jobPayload["tier"]
+ assert.False(t, hasTier)
+}
+
+func TestMinerUV1ClientCrossOriginUploadOmitsAPIKey(t *testing.T) {
+ allowLoopbackSSRF(t)
+ fake := &fakeMinerUV1{t: t, apiKey: "server-key", zipData: v1TestZip(t)}
+
+ // A different port on the same host is a different origin, like a
+ // pre-signed object-storage URL handed out by the official API.
+ storage := httptest.NewServer(http.HandlerFunc(fake.handlePut))
+ defer storage.Close()
+ fake.uploadURL = storage.URL + "/v1/uploads/upload_1/content?Signature=abc"
+
+ server := httptest.NewServer(fake.handler())
+ defer server.Close()
+
+ _, _, err := fastV1Client(server.URL, "server-key").Parse(context.Background(), []byte("bytes"), "a.pdf",
+ minerUV1ParseOptions{})
+ require.NoError(t, err)
+ assert.Equal(t, []byte("bytes"), fake.uploaded)
+ assert.Empty(t, fake.putAuth, "API key must not leak to a cross-origin upload URL")
+}
+
+func TestMinerUV1ClientDedupSkipsByteUpload(t *testing.T) {
+ allowLoopbackSSRF(t)
+ fake := &fakeMinerUV1{t: t, dedupHit: true, zipData: v1TestZip(t)}
+ server := httptest.NewServer(fake.handler())
+ defer server.Close()
+
+ _, _, err := fastV1Client(server.URL, "").Parse(context.Background(), []byte("bytes"), "a.pdf",
+ minerUV1ParseOptions{})
+ require.NoError(t, err)
+ assert.Nil(t, fake.uploaded)
+}
+
+func TestMinerUV1ClientFailedJobReportsFileError(t *testing.T) {
+ allowLoopbackSSRF(t)
+ fake := &fakeMinerUV1{
+ t: t,
+ pollStatuses: []string{"failed"},
+ fileError: &map[string]string{"code": "parse_failed", "message": "corrupt pdf"},
+ }
+ server := httptest.NewServer(fake.handler())
+ defer server.Close()
+
+ _, _, err := fastV1Client(server.URL, "").Parse(context.Background(), []byte("x"), "a.pdf",
+ minerUV1ParseOptions{})
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), "parse_failed: corrupt pdf")
+ assert.Zero(t, fake.downloadCalls)
+}
+
+func TestMinerUV1ClientCancelsJobOnTimeout(t *testing.T) {
+ allowLoopbackSSRF(t)
+ statuses := make([]string, 1000)
+ for i := range statuses {
+ statuses[i] = "running"
+ }
+ fake := &fakeMinerUV1{t: t, pollStatuses: statuses}
+ server := httptest.NewServer(fake.handler())
+ defer server.Close()
+
+ client := fastV1Client(server.URL, "")
+ client.jobTimeout = 100 * time.Millisecond
+ _, _, err := client.Parse(context.Background(), []byte("x"), "a.pdf", minerUV1ParseOptions{})
+ require.Error(t, err)
+ assert.ErrorIs(t, err, context.DeadlineExceeded)
+ assert.True(t, fake.canceled, "abandoned job should be canceled on the server")
+}
+
+func TestMinerUV1ClientAPIErrorMessage(t *testing.T) {
+ allowLoopbackSSRF(t)
+ fake := &fakeMinerUV1{t: t, apiKey: "server-key"}
+ server := httptest.NewServer(fake.handler())
+ defer server.Close()
+
+ _, _, err := fastV1Client(server.URL, "wrong").Parse(context.Background(), []byte("x"), "a.pdf",
+ minerUV1ParseOptions{})
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), "401")
+ assert.Contains(t, err.Error(), "invalid_api_key: bad key")
+}
+
+func TestDetectMinerUProtocol(t *testing.T) {
+ allowLoopbackSSRF(t)
+ tests := []struct {
+ name string
+ status int
+ body string
+ want minerUProtocol
+ wantErr string
+ }{
+ {name: "v1 healthy", status: 200, body: `{"status":"ok","version":"1.0.0"}`, want: minerUProtocolV1},
+ {name: "legacy 404", status: 404, body: `{"detail":"Not Found"}`, want: minerUProtocolLegacy},
+ {name: "legacy 405", status: 405, body: ``, want: minerUProtocolLegacy},
+ {
+ name: "v1 preloading models",
+ status: 503,
+ body: `{"error":{"message":"weights missing","type":"engine_error",` +
+ `"code":"model_preload_files_missing"}}`,
+ want: minerUProtocolV1,
+ wantErr: "model_preload_files_missing: weights missing",
+ },
+ {name: "proxy error", status: 502, body: `bad gateway`, wantErr: "status 502"},
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ assert.Equal(t, "/v1/health", r.URL.Path)
+ w.WriteHeader(tt.status)
+ _, _ = w.Write([]byte(tt.body))
+ }))
+ defer server.Close()
+
+ got, err := detectMinerUProtocol(context.Background(), http.DefaultClient, server.URL)
+ if tt.wantErr != "" {
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), tt.wantErr)
+ return
+ }
+ require.NoError(t, err)
+ assert.Equal(t, tt.want, got)
+ })
+ }
+}
+
+func TestMinerUReaderUsesV1WhenAvailable(t *testing.T) {
+ allowLoopbackSSRF(t)
+ fake := &fakeMinerUV1{t: t, apiKey: "server-key", zipData: v1TestZip(t)}
+ server := httptest.NewServer(fake.handler())
+ defer server.Close()
+
+ reader := NewMinerUReader(map[string]string{
+ "mineru_endpoint": server.URL,
+ "mineru_server_api_key": "server-key",
+ "mineru_tier": "Standard",
+ "mineru_parse_method": "ocr",
+ // Legacy-only knobs are ignored by the V1 path.
+ "mineru_model": "pipeline",
+ "mineru_enable_formula": "false",
+ })
+ result, err := reader.Read(context.Background(), &types.ReadRequest{
+ FileContent: []byte("%PDF"),
+ FileName: "dir/report.pdf",
+ FileType: "pdf",
+ })
+ require.NoError(t, err)
+ require.Empty(t, result.Error)
+ assert.True(t, strings.HasPrefix(result.MarkdownContent, "# Report"))
+ assert.Len(t, result.ImageRefs, 2)
+ assert.Equal(t, "standard", fake.jobPayload["tier"])
+ assert.Equal(t, "ocr", fake.jobPayload["ocr_mode"])
+}
+
+func TestResolveMinerUTier(t *testing.T) {
+ assert.Equal(t, "", resolveMinerUTier(""))
+ assert.Equal(t, "flash", resolveMinerUTier(" Flash "))
+ assert.Equal(t, "advanced", resolveMinerUTier("advanced"))
+ assert.Equal(t, "", resolveMinerUTier("pipeline"), "legacy backend names are not tiers")
+ assert.Equal(t, "", resolveMinerUTier("auto"))
+}
+
+func TestPingMinerUV1(t *testing.T) {
+ allowLoopbackSSRF(t)
+ fake := &fakeMinerUV1{t: t, apiKey: "server-key"}
+ server := httptest.NewServer(fake.handler())
+ defer server.Close()
+
+ ok, msg := PingMinerU(server.URL, "server-key")
+ assert.True(t, ok, msg)
+
+ ok, msg = PingMinerU(server.URL, "wrong")
+ assert.False(t, ok)
+ assert.Contains(t, msg, "API Key 无效")
+
+ ok, msg = PingMinerU(server.URL, "")
+ assert.False(t, ok)
+ assert.Contains(t, msg, "配置 API Key")
+
+ open := &fakeMinerUV1{t: t}
+ openServer := httptest.NewServer(open.handler())
+ defer openServer.Close()
+ ok, msg = PingMinerU(openServer.URL, "")
+ assert.True(t, ok, msg)
+}
+
+func TestExtractMarkdownZipResolvesEncodedAndNestedPaths(t *testing.T) {
+ data := buildMinerUZip(t, map[string]string{
+ "report/report.md": "![](images/%E7%AC%AC%201%20%E9%A1%B5.png)\n![](./images/b.jpg)",
+ "report/images/第 1 页.png": "p1",
+ "report/images/b.jpg": "p2",
+ })
+ md, refs, err := extractMarkdownZip(data, "test")
+ require.NoError(t, err)
+ assert.Contains(t, md, "images/")
+ require.Len(t, refs, 2)
+ assert.Equal(t, []byte("p1"), refs[0].ImageData)
+ assert.Equal(t, "image/jpeg", refs[1].MimeType)
+}
diff --git a/internal/types/config_redaction.go b/internal/types/config_redaction.go
index c7e557761..ff743b745 100644
--- a/internal/types/config_redaction.go
+++ b/internal/types/config_redaction.go
@@ -33,6 +33,9 @@ func ParserEngineConfigForResponse(cfg *ParserEngineConfig, maskSecrets bool) *P
if out.MinerUAPIKey != "" {
out.MinerUAPIKey = RedactedSecretPlaceholder
}
+ if out.MinerUServerAPIKey != "" {
+ out.MinerUServerAPIKey = RedactedSecretPlaceholder
+ }
if out.PaddleOCRVLCloudToken != "" {
out.PaddleOCRVLCloudToken = RedactedSecretPlaceholder
}
@@ -167,6 +170,7 @@ func MergeParserEngineConfigForUpdate(incoming, existing *ParserEngineConfig) *P
prev = *existing
}
out.MinerUAPIKey = PreserveIfRedacted(out.MinerUAPIKey, prev.MinerUAPIKey)
+ out.MinerUServerAPIKey = PreserveIfRedacted(out.MinerUServerAPIKey, prev.MinerUServerAPIKey)
out.PaddleOCRVLCloudToken = PreserveIfRedacted(out.PaddleOCRVLCloudToken, prev.PaddleOCRVLCloudToken)
// Chat attachment parser rules are configured per agent; preserve any legacy
// tenant-level rules when the settings UI omits this field on engine updates.
diff --git a/internal/types/config_redaction_test.go b/internal/types/config_redaction_test.go
index 75d885812..b4838ae6c 100644
--- a/internal/types/config_redaction_test.go
+++ b/internal/types/config_redaction_test.go
@@ -43,21 +43,37 @@ func TestMergeWebSearchConfigForUpdate_PreservesRedactedSecrets(t *testing.T) {
func TestMergeParserEngineConfigForUpdate_PreservesRedactedSecrets(t *testing.T) {
existing := &ParserEngineConfig{
MinerUAPIKey: "mineru-secret",
+ MinerUServerAPIKey: "mineru-server-secret",
PaddleOCRVLCloudToken: "paddle-secret",
MinerUEndpoint: "http://mineru",
}
incoming := &ParserEngineConfig{
MinerUAPIKey: RedactedSecretPlaceholder,
+ MinerUServerAPIKey: RedactedSecretPlaceholder,
PaddleOCRVLCloudToken: RedactedSecretPlaceholder,
MinerUEndpoint: "http://mineru-new",
}
merged := MergeParserEngineConfigForUpdate(incoming, existing)
require.NotNil(t, merged)
assert.Equal(t, "mineru-secret", merged.MinerUAPIKey)
+ assert.Equal(t, "mineru-server-secret", merged.MinerUServerAPIKey)
assert.Equal(t, "paddle-secret", merged.PaddleOCRVLCloudToken)
assert.Equal(t, "http://mineru-new", merged.MinerUEndpoint)
}
+func TestParserEngineConfigForResponse_MasksMinerUSecrets(t *testing.T) {
+ cfg := &ParserEngineConfig{
+ MinerUAPIKey: "mineru-secret",
+ MinerUServerAPIKey: "mineru-server-secret",
+ MinerUEndpoint: "http://mineru",
+ }
+ resp := ParserEngineConfigForResponse(cfg, true)
+ require.NotNil(t, resp)
+ assert.Equal(t, RedactedSecretPlaceholder, resp.MinerUAPIKey)
+ assert.Equal(t, RedactedSecretPlaceholder, resp.MinerUServerAPIKey)
+ assert.Equal(t, "http://mineru", resp.MinerUEndpoint)
+}
+
func TestMergeParserEngineConfigForUpdate_PreservesLegacyChatParserRules(t *testing.T) {
existing := &ParserEngineConfig{
ChatParserEngineRules: []ParserEngineRule{
diff --git a/internal/types/tenant.go b/internal/types/tenant.go
index d86873dd6..164008a19 100644
--- a/internal/types/tenant.go
+++ b/internal/types/tenant.go
@@ -310,7 +310,10 @@ type ParserEngineConfig struct {
MinerUEndpoint string `json:"mineru_endpoint"` // MinerU 自建服务端点
MinerUAPIKey string `json:"mineru_api_key"` // MinerU 云 API Key
- // MinerU 自建解析参数
+ // MinerU 自建解析参数。协议按服务端自动识别:4.0+ 走 V1 API,更早版本走 /file_parse。
+ MinerUServerAPIKey string `json:"mineru_server_api_key,omitempty"` // 4.0+:服务端 --api-key
+ MinerUTier string `json:"mineru_tier,omitempty"` // 4.0+:flash/basic/standard/advanced,空为服务端默认
+ // 以下 backend、vLLM、公式/表格、语言参数仅对 3.x 及更早版本生效。
MinerUModel string `json:"mineru_model,omitempty"` // backend: pipeline, vlm-*, hybrid-*
MinerUVLMServerURL string `json:"mineru_vlm_server_url,omitempty"` // vLLM 服务器地址 (vlm-http-client / hybrid-http-client)
MinerUEnableFormula *bool `json:"mineru_enable_formula,omitempty"`
@@ -400,6 +403,12 @@ func (c *ParserEngineConfig) ToOverridesMap() map[string]string {
if c.MinerUAPIKey != "" {
m["mineru_api_key"] = c.MinerUAPIKey
}
+ if c.MinerUServerAPIKey != "" {
+ m["mineru_server_api_key"] = c.MinerUServerAPIKey
+ }
+ if c.MinerUTier != "" {
+ m["mineru_tier"] = c.MinerUTier
+ }
if c.MinerUModel != "" {
m["mineru_model"] = c.MinerUModel
}
diff --git a/website-docs/02-architecture/03-document-pipeline.md b/website-docs/02-architecture/03-document-pipeline.md
index 682cf1c80..9a5199afa 100644
--- a/website-docs/02-architecture/03-document-pipeline.md
+++ b/website-docs/02-architecture/03-document-pipeline.md
@@ -271,7 +271,7 @@ Worker 消费 `TypeDocumentProcess` 后按五个规范化阶段推进,每个
- **builtin**:通过 gRPC(`docparser/grpc_parser.go`)或 HTTP(`http_parser.go`)调用 Python **docreader** 服务;
- **simple**:Go 原生解析 md/txt/csv/json/图片/音频(`builtin_converter.go`,CSV→Markdown 表格、JSON→递归分割的代码块,图片/音频转占位引用);
- **anydoc**:Go 进程内解析 docx/doc/pptx/ppt/xlsx/xls/odf/rtf/epub/csv/pdf(`anydoc_reader.go`),底层是通过 cgo 链接的 anydoc Rust 库。office 文档的嵌入图按文档模型插回 Markdown 原位;无文字层的扫描件 PDF 在 DocReader 可用时回退到 builtin 整页渲染。链接了 anydoc 时,未配置规则的复杂格式默认走 anydoc,但 PDF 默认仍走 builtin。仅在带 `anydoc` 构建标签的二进制中可用,其余构建里该引擎在引擎列表中显示为不可用;
- - **weknoracloud / mineru / mineru_cloud / paddleocr_vl / paddleocr_vl_cloud**:HTTP 转换器(`engines.go` 注册,按 `mineru_endpoint`、`mineru_api_key`、`paddleocr_vl_endpoint` 等配置判定可用性)。
+ - **weknoracloud / mineru / mineru_cloud / paddleocr_vl / paddleocr_vl_cloud**:HTTP 转换器(`engines.go` 注册,按 `mineru_endpoint`、`mineru_api_key`、`paddleocr_vl_endpoint` 等配置判定可用性;自建 `mineru` 会自动识别 MinerU 4.0 的 V1 API 与旧版 `/file_parse`,见[文档解析服务](../03-features/03-document-parsing.md#mineru-self-hosted))。
引擎目录集中在 `internal/infrastructure/docparser/engines.go`:每个引擎同时声明元数据(名称、描述、文件类型、可用性探针)与 `NewReader` 工厂,`docparser.NewReader` 按名字分发,未注册的名字(如只存在于 docreader 的 `markitdown`)落到 docreader 客户端。
5. 文件模式:从 `FileService.GetFile(payload.FilePath)` 读回字节填入 `ReadRequest.FileContent`。
diff --git a/website-docs/03-features/03-document-parsing.md b/website-docs/03-features/03-document-parsing.md
index c92e522e6..62dda5f44 100644
--- a/website-docs/03-features/03-document-parsing.md
+++ b/website-docs/03-features/03-document-parsing.md
@@ -460,6 +460,36 @@ docker build -f docker/Dockerfile.app --build-arg WITH_ANYDOC=0 -t weknora-app .
| `internal/infrastructure/docparser/engines.go` | 引擎注册(元数据 + Reader 工厂) |
| `third_party/anydoc-go/` | vendored 的上游 Go 绑定与 C ABI shim(来源与本地改动见该目录 README) |
+### MinerU 自建引擎(Go 进程直连,不经 docreader) {#mineru-self-hosted}
+
+`mineru` 引擎由 Go App 直接调用自建的 MinerU 服务(`internal/infrastructure/docparser/mineru_converter.go`)。MinerU 4.0 删除了旧的 `/file_parse` 接口,改为 V1 API,因此 WeKnora 在每次解析前先请求 `GET {mineru_endpoint}/v1/health`,按结果选择协议:
+
+| 探测结果 | 协议 | 流程 |
+| --- | --- | --- |
+| `200` 且 `status=ok` | V1(MinerU ≥ 4.0,`mineru_v1_client.go`) | `POST /v1/uploads` → 按返回的 `upload_url` 上传字节 → `POST /v1/uploads/{id}/complete` → `POST /v1/parse/jobs`(只请求 `zip` 产物)→ 轮询 `GET /v1/parse/jobs/{id}` → `GET /v1/files/{id}/content` 下载 zip,取其中的 `markdown.md` 与 `images/` |
+| `404` / `405` | 旧版(MinerU ≤ 3.x) | `POST /file_parse`,同步返回 Markdown 与 base64 图片 |
+| `503` 带错误结构 | V1,但服务未就绪 | 直接报错(常见于模型预加载失败),不回退旧协议 |
+
+升级 MinerU 不需要改 WeKnora 的配置。两套协议的参数对应关系:
+
+| 设置项(`ParserEngineConfig`) | MinerU ≥ 4.0 | MinerU ≤ 3.x |
+| --- | --- | --- |
+| `mineru_endpoint` | V1 服务地址(如 `http://mineru:8000`) | 同左 |
+| `mineru_server_api_key` | 服务以 `--api-key` 启动时作为 `Authorization: Bearer` 发送 | 不使用 |
+| `mineru_tier` | `tier`:`flash` / `basic` / `standard` / `advanced`;留空由服务端选默认档位(优先 `standard`) | 不使用 |
+| `mineru_parse_method` | `ocr_mode`(`auto` / `txt` / `ocr`) | `parse_method` |
+| `mineru_model`、`mineru_vlm_server_url`、`mineru_enable_formula`、`mineru_enable_table`、`mineru_language` | 忽略(4.0 已删除这些参数;VLM 地址改在 MinerU 的 `config.yaml` 里配置) | 原样发送 |
+
+V1 流程的几个细节:
+
+- 上传时附带 `sha256sum`,服务端已有相同文件时直接复用,不再传字节。
+- 只有 `upload_url` 与 `mineru_endpoint` 同源(scheme、host、端口都相同)时才附带 API Key;跨源地址(如官方 API 下发的预签名对象存储 URL)不带 Key,并照常经过 SSRF 校验。
+- 轮询从 2 秒开始指数退避,最长 30 秒一次;总时长与旧版一样是 1000 秒。超时或调用方取消时,会发 `DELETE /v1/parse/jobs/{id}` 取消服务端任务。
+- MinerU V1 服务的上传、任务状态都存在进程内存里,服务重启后正在轮询的任务会返回 404,本次解析直接失败。
+- 「测试连接」在 V1 服务上额外请求一次需要鉴权的 `GET /v1/parse/jobs?limit=1`,用来发现 API Key 缺失或错误(`/v1/health` 本身不校验 Key)。
+
+`mineru_cloud`(mineru.net)目前仍走 `/api/v4/file-urls/batch` 批量接口,不受 4.0 自建服务变化影响。
+
---
### 附:关键事实速查