RAGFlow is an agent-native database: a Retrieval-Augmented Generation (RAG) engine that grew into context infrastructure.
Go agent-harnessagent-native-databaseagentic-aiagentic-retrievalagentic-searchaiai-agentscontext-enginecontext-engineeringcontext-managementharness-engineeringknowledge-compilationragretrieval-augmented-generationretrieval-harnesssearch-harness
fix(dao): migrate Python-era flat ingestion config onto pipeline components (#20986)
jay77721b859e0460ebe
diff --git a/docs/administrator/migration/database_change_log.md b/docs/administrator/migration/database_change_log.mdindex 919e8d575c..8f2b169b28 100644--- a/docs/administrator/migration/database_change_log.md+++ b/docs/administrator/migration/database_change_log.md@@ -16,7 +16,7 @@ A version-by-version record of the schema and data changes the Go backend applie- The marker is a row in `system_settings` with the key `mysql_migration.database.version`.- The marker prevents a migration step from running again after that step has completed successfully.-- Go defines the versions as constants: `modelMigrationBaseVersion` and `modelMigrationTargetVersion` in [migration_version.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/migration_version.go), `conversationHistoryTargetVersion` in [conversation_history_migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/conversation_history_migration.go), and `knowledgebaseParserConfigTargetVersion` in [migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/migration.go).+- Go defines the versions as constants: `modelMigrationBaseVersion` and `modelMigrationTargetVersion` in [migration_version.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/migration_version.go), `conversationHistoryTargetVersion` in [conversation_history_migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/conversation_history_migration.go), `knowledgebaseParserConfigTargetVersion` in [migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/migration.go), and `ingestionParserConfigTargetVersion` in [ingestion_parser_config_migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/ingestion_parser_config_migration.go). The two parser-config constants share `v1.0.0-rc2.dev1`.- A step runs when the stored version is lower than the step's version. Versions are compared with Python PEP 440 semantics, where a `dev` prerelease sorts *before* the release it precedes (`v1.0.0-rc1.dev1` < `v1.0.0-rc1`). A missing or unparsable marker never skips a step.- `checkDatabaseVersion` in [cmd/ragflow_server.go](https://github.com/infiniflow/ragflow/blob/main/cmd/ragflow_server.go) refuses to start when the code version is older than the stored database version, so a database migrated by a newer release cannot be opened by an older one.@@ -26,7 +26,7 @@ Startup order in `InitDB` ([database.go](https://github.com/infiniflow/ragflow/b2. `migrateIngestionLogRunIdentity` — ingestion log columns and indexes.3. `AutoMigrate` — converges every ORM model.4. `migrateConversationHistory` — runs *after* `AutoMigrate`, which creates the child tables it backfills.-5. `migrateKnowledgebaseParserConfig` — runs last, so the marker does not move past the conversation split's version before that split has run.+5. `migrateIngestionParserConfigs` — converts ingestion settings on datasets and documents, then invokes the existing `migrateKnowledgebaseParserConfig` cleanup. Both run after the conversation split so the shared marker cannot skip that split.## Version overview@@ -35,7 +35,7 @@ Startup order in `InitDB` ([database.go](https://github.com/infiniflow/ragflow/b| `v0.26.0` | Rebuild the tenant model tables from `tenant_llm` and normalize stored model ids | `modelMigrationBaseVersion` || `v1.0.0-rc1` | Seed factory-declared models, merge `model_type` into an integer bitmask, populate the `tenant_*_id` columns | `modelMigrationTargetVersion` || `v1.0.0-rc1.dev1` | Split conversation message and reference payloads into child tables | `conversationHistoryTargetVersion` |-| `v1.0.0-rc2.dev1` | Delete the retired `raptor` and `graphrag` sections from `knowledgebase.parser_config` | `knowledgebaseParserConfigTargetVersion` |+| `v1.0.0-rc2.dev1` | Convert Python ingestion settings in dataset/document `parser_config` onto pipeline components, then remove retired dataset `raptor` / `graphrag` sections | `ingestionParserConfigTargetVersion` / `knowledgebaseParserConfigTargetVersion` |The two tenant model steps are cumulative: a database at `v0.26.0` still runs the `v1.0.0-rc1` step, and a database at or above `v1.0.0-rc1` runs neither.@@ -126,22 +126,26 @@ On success the step writes `v1.0.0-rc1.dev1` into the version marker.## v1.0.0-rc2.dev1-Retires the `raptor` and `graphrag` sections of the dataset parser config. Both moved out of `knowledgebase.parser_config`, so the stored sections are dead data no code path reads any more.+Converts saved Python v0.27.2 ingestion settings into the component-scoped configuration consumed by Go. After conversion, the existing dataset cleanup removes the retired `raptor` and `graphrag` sections.### Data migration-| Table | Change |-|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|-| `knowledgebase` | `parser_config` is parsed as a JSON object; the top level `raptor` and `graphrag` keys are deleted and the remaining object is written back to the same column. |+| Table | Change |+|-------|--------|+| `knowledgebase`, `document` | Move supported top-level Parser, Chunker, Tokenizer and Extractor settings onto the corresponding pipeline component nodes. |+| `knowledgebase` | The existing cleanup deletes top-level `raptor` and `graphrag` sections, preserving other settings. |++The conversion covers parser engines and model references, PDF pages, MinerU method/language, TCADP options, spreadsheet column configuration, supported chunk sizes and delimiters, overlap, media context, parent/child settings, filename embedding weight, keywords, questions and metadata. Fields are converted only where the selected component supports them. Settings without a Go consumer are dropped with table/row/field warnings; tags and early Go node formats are left untouched.Execution details:-- Rows are read in id-ordered batches of 256, each batch in its own transaction, so an interrupted run resumes from the last committed batch.-- Only rows whose config actually carries one of the keys are rewritten; a row without them is left byte-identical.-- A `NULL`, empty or non-object payload is skipped rather than failing the step.-- The step returns without touching the marker when the `knowledgebase` table does not exist.+- Component conversion reads each table in id-ordered batches of 256, each batch in its own transaction. Changed rows are updated once by the conversion; unchanged rows retain their original bytes. The existing dataset cleanup performs its own subsequent scan.+- Each row's `parser_id` determines the builtin component targets. Parser/Chunker/Tokenizer conversion leaves custom pipeline rows and unsupported parser IDs untouched. Extractor conversion can also target existing Extractor nodes in custom pipeline configs.+- Existing component values win, except that dataset-level modular metadata transport remains authoritative under the current service contract.+- SQL `NULL`, empty strings and JSON `null` are left untouched. Invalid JSON, unsupported metadata schemas and unsupported engine selections stop migration with table/row context. A failed batch rolls back; earlier committed batches can be rerun safely. See [Parser-config conversion failures](./upgrade_from_v1_to_later_releases.md#parser-config-conversion-failures) for recovery instructions.+- The marker advances only after both table conversions and the existing dataset cleanup succeed. If neither table exists, the marker is left unchanged.-On success the step writes `v1.0.0-rc2.dev1` into the version marker.+On success the migration records `v1.0.0-rc2.dev1`. Databases already marked `v1.0.0-rc2.dev1` or later skip conversion. This includes databases marked by the earlier cleanup-only implementation: that marker does not prove their ingestion settings were converted.---diff --git a/docs/administrator/migration/upgrade_from_v1_to_later_releases.md b/docs/administrator/migration/upgrade_from_v1_to_later_releases.mdindex fc5af1fe66..7ed02122b0 100644--- a/docs/administrator/migration/upgrade_from_v1_to_later_releases.md+++ b/docs/administrator/migration/upgrade_from_v1_to_later_releases.md@@ -53,7 +53,15 @@ docker compose -p docker -f docker/docker-compose.yml logs -f ragflow-cpuKeep the log view open until the migration finishes and the application starts. Press `Ctrl+C` to leave the log view; the containers continue running. Review warnings as well as errors before allowing users, ingestion workers, or synchronization jobs to write data.-If the logs report a migration failure, stop the deployment, preserve the logs, and follow [Roll back after a failed migration](#roll-back-after-a-failed-migration).+If the logs report a migration failure, stop the deployment and preserve the logs. For a parser-config conversion error, follow the instructions below. For other failures, follow [Roll back after a failed migration](#roll-back-after-a-failed-migration).++#### Parser-config conversion failures++The `v1.0.0-rc2.dev1` step converts saved Python v0.27.2 ingestion settings for Parser, Chunker, Tokenizer, and Extractor together before the existing dataset `raptor` and `graphrag` cleanup records the shared version. Invalid JSON, metadata schemas that Go cannot represent, and unsupported PDF engine or OCR model-reference choices stop `--migrate` and block application startup. The error identifies the table, row ID, and rejected setting.++Keep all writers stopped. Back up the affected row's original `parser_config`, then correct the reported value in that row. For an unsupported engine or OCR model reference, explicitly choose and configure an engine that Go can run; deleting the setting would select the template's default engine instead. Start the target deployment again to rerun `--migrate`, and review its logs before resuming writes.++Each failed batch rolls back and its migration version does not advance. Earlier batches may already be committed; rerunning the conversion after correcting the data is safe. Do not change the migration version manually. To return to the previous release instead, restore the complete pre-upgrade recovery point as described below.### 4. Check the deployment statusdiff --git a/internal/dao/database.go b/internal/dao/database.goindex f687e2f546..3adbd9ffdf 100644--- a/internal/dao/database.go+++ b/internal/dao/database.go@@ -241,12 +241,10 @@ func InitDB(ctx context.Context, migrateDB bool) error {if err = migrateConversationHistory(ctx, DB); err != nil {return fmt.Errorf("failed to migrate conversation history: %w", err)}- // Drop the retired raptor and graphrag sections from the dataset parser- // configs. It runs after the conversation split because the two steps- // share one version marker: this step's version sorts above the split's- // v1.0.0-rc1, so marking it earlier would skip a split that has not run.- if err = migrateKnowledgebaseParserConfig(ctx, DB); err != nil {- return fmt.Errorf("failed to migrate knowledgebase parser config: %w", err)+ // Run ingestion conversion after the conversation split. It invokes the+ // dataset cleanup last because both use the same version marker.+ if err = migrateIngestionParserConfigs(ctx, DB); err != nil {+ return fmt.Errorf("failed to migrate ingestion parser configs: %w", err)}if err = migrateTenantModelMaxTokens(ctx, DB); err != nil {diff --git a/internal/dao/ingestion_parser_config_migration.go b/internal/dao/ingestion_parser_config_migration.gonew file mode 100644index 0000000000..b74c90a76d--- /dev/null+++ b/internal/dao/ingestion_parser_config_migration.go@@ -0,0 +1,1335 @@+//+// Copyright 2026 The InfiniFlow Authors. All Rights Reserved.+//+// Licensed under the Apache License, Version 2.0 (the "License");+// you may not use this file except in compliance with the License.+// You may obtain a copy of the License at+//+// http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing, software+// distributed under the License is distributed on an "AS IS" BASIS,+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+// See the License for the specific language governing permissions and+// limitations under the License.++package dao++import (+ "context"+ "fmt"+ "maps"+ "sort"+ "strconv"+ "strings"++ "go.uber.org/zap"+ "gorm.io/gorm"++ "ragflow/internal/common"+ "ragflow/internal/entity"+ parserchunk "ragflow/internal/parser/chunk"+ parserpkg "ragflow/internal/parser/parser"+ "ragflow/internal/utility"+)++// Use the project version shared with the dataset parser config cleanup.+const ingestionParserConfigTargetVersion = knowledgebaseParserConfigTargetVersion++// Python v0.27.2 stored component settings at the parser_config root. Convert+// them to Go's component-scoped shape before the API discards unscoped keys.+// Target IDs are literal because importing the pipeline registry into DAO would+// introduce a dependency cycle; runtime tests check them against the templates.+type builtinIngestionTarget struct {+ parserNode string+ pdf bool+ spreadsheet bool+ // Overrides replace an entire family; carry template PDF values that differ+ // from the Parser constructor's defaults.+ pdfDefaults map[string]any+ chunker string+ chunkerNode string+ tokenizerNode string+ extractorNode string+}++func (t builtinIngestionTarget) hasFamily(family string) bool {+ switch family {+ case "pdf":+ return t.pdf+ case "spreadsheet":+ return t.spreadsheet+ default:+ return false+ }+}++// acceptsChunkParams reports whether the chunker accepts delimiter, overlap,+// context and parent/child parameters.+func (t builtinIngestionTarget) acceptsChunkParams() bool {+ return t.chunker == "GeneralChunker" || t.chunker == "TokenChunker"+}++// Only PDF and spreadsheet settings are migrated from the Parser's flat keys.+var ingestionTargets = func() map[string]builtinIngestionTarget {+ // "naive" is the historical name of the general pipeline: the embedded+ // general template declares parser_ids ["naive"] and the registry aliases+ // "general" onto it.+ general := builtinIngestionTarget{+ parserNode: "Parser:HipSignsRhyme",+ pdf: true,+ spreadsheet: true,+ chunker: "GeneralChunker",+ chunkerNode: "GeneralChunker:SixApplesFall",+ tokenizerNode: "Tokenizer:LegalReadersDecide",+ extractorNode: "Extractor:AutoExtractDefault",+ }+ return map[string]builtinIngestionTarget{+ "general": general,+ "naive": general,+ "qa": {+ parserNode: "Parser:HipSignsRhyme",+ pdf: true,+ spreadsheet: true,+ chunker: "QAChunker",+ chunkerNode: "QAChunker:TidyCloudsThink",+ tokenizerNode: "Tokenizer:ColdCloudsDream",+ },+ "manual": {+ parserNode: "Parser:HipSignsRhyme",+ pdf: true,+ chunker: "ManualChunker",+ chunkerNode: "ManualChunker:NineInsectsFind",+ tokenizerNode: "Tokenizer:FunnyBalloonsGrin",+ extractorNode: "Extractor:AutoExtractDefault",+ },+ "table": {+ parserNode: "Parser:HipSignsRhyme",+ spreadsheet: true,+ chunker: "TableChunker",+ chunkerNode: "TableChunker:FastFoxesJump",+ tokenizerNode: "Tokenizer:DeepLakesShine",+ },+ "paper": {+ parserNode: "Parser:HipSignsRhyme",+ pdf: true,+ pdfDefaults: map[string]any{"enable_multi_column": true},+ chunker: "TitleChunker",+ chunkerNode: "TitleChunker:SparklySchoolsTravel",+ tokenizerNode: "Tokenizer:GreatCarsWash",+ extractorNode: "Extractor:AutoExtractDefault",+ },+ "book": {+ parserNode: "Parser:HipSignsRhyme",+ pdf: true,+ pdfDefaults: map[string]any{"remove_toc": true},+ chunker: "TitleChunker",+ chunkerNode: "TitleChunker:GrumpyGarlicsBake",+ tokenizerNode: "Tokenizer:HotDonutsRing",+ extractorNode: "Extractor:AutoExtractDefault",+ },+ "laws": {+ parserNode: "Parser:HipSignsRhyme",+ pdf: true,+ pdfDefaults: map[string]any{"remove_toc": true},+ chunker: "TitleChunker",+ chunkerNode: "TitleChunker:SpicyKeysKick",+ tokenizerNode: "Tokenizer:PublicJobsTake",+ extractorNode: "Extractor:AutoExtractDefault",+ },+ "presentation": {+ parserNode: "Parser:HipSignsRhyme",+ pdf: true,+ chunker: "PageChunker",+ chunkerNode: "PageChunker:HappyHillsGlow",+ tokenizerNode: "Tokenizer:TallTreesDance",+ extractorNode: "Extractor:AutoExtractDefault",+ },+ "picture": {+ parserNode: "Parser:ViewsCaptureLight",+ chunker: "OneChunker",+ chunkerNode: "OneChunker:BrightColorsGlow",+ tokenizerNode: "Tokenizer:SharpLensFocus",+ extractorNode: "Extractor:AutoExtractDefault",+ },+ "one": {+ parserNode: "Parser:HipSignsRhyme",+ pdf: true,+ spreadsheet: true,+ chunker: "OneChunker",+ chunkerNode: "OneChunker:DryDrinksVisit",+ tokenizerNode: "Tokenizer:FrankWeeksListen",+ extractorNode: "Extractor:AutoExtractDefault",+ },+ "audio": {+ parserNode: "Parser:SongsFillAir",+ chunker: "OneChunker",+ chunkerNode: "OneChunker:BlueSkiesLaugh",+ tokenizerNode: "Tokenizer:KindEyesWatch",+ extractorNode: "Extractor:AutoExtractDefault",+ },+ "email": {+ parserNode: "Parser:BirdsFlutterHigh",+ chunker: "TokenChunker",+ chunkerNode: "TokenChunker:WarmBreadSmells",+ tokenizerNode: "Tokenizer:NiceWordsSpoken",+ extractorNode: "Extractor:AutoExtractDefault",+ },+ }+}()++// ingestionParserFlatKeys are the Python-era top level keys this step converts.+var (+ ingestionParserFlatKeys = []string{+ "layout_recognize",+ "mineru_parse_method",+ "mineru_lang",+ "pages",+ "table_result_type",+ "markdown_image_response_type",+ "table_column_mode",+ "table_column_roles",+ "table_column_names",+ }+ ingestionChunkerFlatKeys = []string{+ "chunk_token_num",+ "delimiter",+ "overlapped_percent",+ "table_context_size",+ "image_context_size",+ "parent_child",+ "enable_children",+ "children_delimiter",+ }+ ingestionTokenizerFlatKeys = []string{"filename_embd_weight"}+)++// ingestionRetiredFlatKeys are Python settings that have no Go consumer at all.+// They are deleted so the stored config stops advertising behaviour the Go+// runtime does not implement; every drop is logged with table, row and field.+var ingestionRetiredFlatKeys = []string{+ // Spreadsheets are always rendered as segmented HTML; the flag only warns+ // now (internal/parser/parser/office_table_render.go:deprecatedHTML4Excel).+ "html4excel",+ // Splits one parse into several ingestion jobs; it is not a page range and+ // not a chunk size.+ "task_page_size",+ // Follows hyperlinks and indexes the linked content; unimplemented in Go.+ "analyze_hyperlink",+ "hyperlink_urls",+ // Video parsing is rejected at dispatch+ // (internal/ingestion/component/media_dispatch.go:maybeDispatchVideo).+ "video_prompt",+ // These MinerU toggles have no Go reader; method and language are migrated.+ "mineru_formula_enable",+ "mineru_table_enable",+}++// ingestionOwnedFlatKeys is every flat key this step consumes or retires. A row+// carrying none of them is left completely untouched, which is what makes a+// second run a no-op.+func ingestionOwnedFlatKeys() []string {+ owned := make([]string, 0, len(ingestionParserFlatKeys)+len(ingestionChunkerFlatKeys)++ len(ingestionTokenizerFlatKeys)+len(ingestionRetiredFlatKeys))+ owned = append(owned, ingestionParserFlatKeys...)+ owned = append(owned, ingestionChunkerFlatKeys...)+ owned = append(owned, ingestionTokenizerFlatKeys...)+ owned = append(owned, ingestionRetiredFlatKeys...)+ return owned+}++// ingestionConfigMigrationRow is one dataset or document row.+type ingestionConfigMigrationRow struct {+ ID string `gorm:"column:id"`+ ParserID string `gorm:"column:parser_id"`+ PipelineID *string `gorm:"column:pipeline_id"`+ ParserConfig *string `gorm:"column:parser_config"`+}++func migrateIngestionParserConfigs(ctx context.Context, db *gorm.DB) error {+ version, err := GetDatabaseMigrationVersion(ctx, db)+ if err != nil {+ return err+ }+ if shouldSkipMigration(version, ingestionParserConfigTargetVersion) {+ return nil+ }+ found := false+ for _, target := range []struct {+ table string+ model any+ }{+ {"knowledgebase", &entity.Knowledgebase{}}, {"document", &entity.Document{}},+ } {+ if !db.WithContext(ctx).Migrator().HasTable(target.model) {+ continue+ }+ found = true+ count, err := migrateIngestionConfigRows(ctx, db, target.model, target.table)+ if err != nil {+ return fmt.Errorf("migrate %s parser configs: %w", target.table, err)+ }+ common.Info("Migrated Python parser configs", zap.String("table", target.table),+ zap.String("version", ingestionParserConfigTargetVersion), zap.Int("rows", count))+ }+ if !found {+ return nil+ }+ // Cleanup shares this version: run it only after both conversions succeed.+ if err := migrateKnowledgebaseParserConfig(ctx, db); err != nil {+ return err+ }+ return setDatabaseMigrationVersion(ctx, db, ingestionParserConfigTargetVersion)+}++func migrateIngestionConfigRows(ctx context.Context, db *gorm.DB, model any, table string) (int, error) {+ count := 0+ var rows []ingestionConfigMigrationRow+ result := db.WithContext(ctx).Model(model).+ Select("id", "parser_id", "pipeline_id", "parser_config").Order("id").+ FindInBatches(&rows, parserConfigMigrationBatchSize, func(batch *gorm.DB, _ int) error {+ return batch.Transaction(func(tx *gorm.DB) error {+ for _, row := range rows {+ config, ok := parseParserConfig(row.ParserConfig)+ if !ok {+ // An empty or absent payload carries nothing to migrate;+ // a malformed one is rejected so the batch rolls back+ // with the offending row named.+ if row.ParserConfig == nil || strings.TrimSpace(*row.ParserConfig) == "" {+ continue+ }+ return fmt.Errorf("row %s: invalid parser_config JSON", row.ID)+ }+ extractorChanged, err := migrateExtractorConfig(config, row.ParserID, row.PipelineID)+ if err != nil {+ return fmt.Errorf("row %s: %w", row.ID, err)+ }+ ingestionChanged, err := migrateParserChunkerTokenizerConfig(config, row.ParserID, row.PipelineID, table, row.ID)+ if err != nil {+ return fmt.Errorf("row %s: %w", row.ID, err)+ }+ if !extractorChanged && !ingestionChanged {+ continue+ }+ if err := tx.Model(model).Where("id = ?", row.ID).+ Update("parser_config", entity.JSONMap(config)).Error; err != nil {+ return fmt.Errorf("row %s: update parser_config: %w", row.ID, err)+ }+ count+++ }+ return nil+ })+ })+ return count, result.Error+}++// migrateParserChunkerTokenizerConfig rewrites one row's Python-era flat Parser / Chunker /+// Tokenizer settings into the components of the pipeline that row runs. Params+// the row already carries win; a setting with no Go consumer is deleted and+// logged.+func migrateParserChunkerTokenizerConfig(config map[string]any, parserID string, pipelineID *string, table, rowID string) (bool, error) {+ // A custom canvas reads component-scoped params on the Python side too, and+ // a canvas can hold several nodes of the same kind, so no ownership rule for+ // a flat setting is safe.+ if pipelineID != nil && strings.TrimSpace(*pipelineID) != "" {+ return false, nil+ }+ target, ok := ingestionTargets[strings.ToLower(strings.TrimSpace(parserID))]+ if !ok {+ // resume / tag / knowledge_graph and unknown parser ids have no Go+ // template, so their settings are preserved instead of being dropped.+ return false, nil+ }+ owned := false+ for _, key := range ingestionOwnedFlatKeys() {+ if _, ok := config[key]; ok {+ owned = true+ break+ }+ }+ if !owned {+ return false, nil+ }++ row := ingestionConfigRow{table: table, rowID: rowID}+ writes := nodeWrites{}+ if err := convertIngestionParser(config, target, writes, row); err != nil {+ return false, err+ }+ if err := convertIngestionChunker(config, target, writes, row); err != nil {+ return false, err+ }+ if err := convertIngestionTokenizer(config, target, writes, row); err != nil {+ return false, err+ }+ changed, err := applyNodeWrites(config, writes)+ if err != nil {+ return false, err+ }+ for _, key := range ingestionRetiredFlatKeys {+ if _, exists := config[key]; exists {+ row.retire(key, "this Python setting has no Go consumer")+ }+ }+ for _, key := range ingestionOwnedFlatKeys() {+ if _, exists := config[key]; exists {+ delete(config, key)+ changed = true+ }+ }+ return changed, nil+}++// ingestionConfigRow carries the log context of the row being converted.+type ingestionConfigRow struct {+ table string+ rowID string+}++// retire records that a Python setting with an actual value has no Go target.+func (r ingestionConfigRow) retire(field, reason string) {+ common.Warn("Ingestion parser config: dropping setting with no Go consumer",+ zap.String("table", r.table),+ zap.String("row_id", r.rowID),+ zap.String("field", field),+ zap.String("reason", reason))+}++// nodeWrites accumulates the component-scoped params the converters derive.+type nodeWrites map[string]map[string]any++func (w nodeWrites) params(node string) map[string]any {+ if w[node] == nil {+ w[node] = map[string]any{}+ }+ return w[node]+}++func (w nodeWrites) set(node, param string, value any) {+ w.params(node)[param] = value+}++// setSetup writes a param inside a Parser file-family setup, the shape the+// Parser component reads (schema.ParserSetup). An existing family map is merged+// into, never replaced.+func (w nodeWrites) setSetup(node, family, param string, value any) {+ params := w.params(node)+ setup, _ := params[family].(map[string]any)+ if setup == nil {+ setup = map[string]any{}+ params[family] = setup+ }+ setup[param] = value+}++// applyNodeWrites merges the derived params into the stored config. A param the+// row already carries always wins - including an explicit false/0/empty value -+// and a node or family is created only when it has something to receive.+func applyNodeWrites(config map[string]any, writes nodeWrites) (bool, error) {+ nodes := make([]string, 0, len(writes))+ for node := range writes {+ nodes = append(nodes, node)+ }+ sort.Strings(nodes)++ changed := false+ for _, node := range nodes {+ derived := writes[node]+ if len(derived) == 0 {+ continue+ }+ raw, exists := config[node]+ if !exists {+ params := make(map[string]any, len(derived))+ for key, value := range derived {+ params[key] = value+ }+ config[node] = params+ changed = true+ continue+ }+ params, ok := raw.(map[string]any)+ if !ok {+ return false, fmt.Errorf("%s is not an object", node)+ }+ for key, value := range derived {+ existing, present := params[key]+ if !present {+ params[key] = value+ changed = true+ continue+ }+ // Merge missing fields within Parser families and parent_child groups.+ derivedMap, derivedIsMap := value.(map[string]any)+ existingMap, existingIsMap := existing.(map[string]any)+ if !derivedIsMap || !existingIsMap {+ continue+ }+ for subKey, subValue := range derivedMap {+ if _, exists := existingMap[subKey]; exists {+ continue+ }+ existingMap[subKey] = subValue+ changed = true+ }+ }+ }+ return changed, nil+}++// convertIngestionParser moves the flat PDF-parser settings onto the row's+// Parser node. An unsupported engine stops migration before the source is lost.+func convertIngestionParser(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {+ if raw, ok := config["layout_recognize"]; ok {+ if err := convertIngestionLayoutRecognize(raw, config, target, writes, row); err != nil {+ return err+ }+ }+ for _, key := range []string{"mineru_parse_method", "mineru_lang"} {+ if raw, exists := config[key]; exists && raw != nil {+ value, ok := raw.(string)+ if !ok {+ return fmt.Errorf("%s must be a string, got %T", key, raw)+ }+ if target.hasFamily("pdf") {+ writes.setSetup(target.parserNode, "pdf", key, value)+ } else {+ row.retire(key, "the targeted pipeline has no PDF file family")+ }+ }+ }+ if err := convertIngestionParserPages(config, target, writes, row); err != nil {+ return err+ }+ if err := convertIngestionTCADPOptions(config, target, writes, row); err != nil {+ return err+ }+ if err := convertIngestionTableColumns(config, target, writes, row); err != nil {+ return err+ }+ seedCarriedParserFamilyDefaults(target, writes)+ return nil+}++// Carry PDF template defaults whenever an override replaces that family.+func seedCarriedParserFamilyDefaults(target builtinIngestionTarget, writes nodeWrites) {+ pdf, ok := writes[target.parserNode]["pdf"].(map[string]any)+ if !ok {+ return+ }+ for key, value := range target.pdfDefaults {+ if _, exists := pdf[key]; !exists {+ pdf[key] = value+ }+ }+}++// Normalize Python engine spellings using the existing Go runtime vocabulary.+func ingestionPDFParseMethod(raw string) (string, bool) {+ method := strings.ToLower(strings.TrimSpace(raw))+ if method == "tcadp" {+ method = "tcadp parser"+ }+ if !parserpkg.IsPDFParseMethod(method) {+ return "", false+ }+ switch method {+ case "tcadp parser":+ method = "tcadp"+ case "plaintext", "plain text":+ method = "plain_text"+ }+ return method, true+}++// Resolve the flat engine choice without overriding an existing selector.+func convertIngestionLayoutRecognize(raw any, config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {+ // An existing selector owns the engine choice as a whole: filling its+ // sibling could change which dispatcher wins, even if the old value stays.+ if node, ok := config[target.parserNode].(map[string]any); ok {+ if setup, ok := node["pdf"].(map[string]any); ok {+ _, hasMethod := setup["parse_method"]+ _, hasModel := setup["layout_recognizer"]+ if hasMethod || hasModel {+ // A PDF selector does not own the spreadsheet family's engine.+ if value, ok := raw.(string); ok && target.hasFamily("spreadsheet") {+ if method, known := ingestionPDFParseMethod(value); known && method == "tcadp" {+ writes.setSetup(target.parserNode, "spreadsheet", "parse_method", method)+ return nil+ }+ }+ return nil+ }+ }+ }+ switch value := raw.(type) {+ case nil:+ // A null engine means "unset": nothing to derive, nothing lost.+ return nil+ case bool:+ // The dataset UI stored the DeepDOC toggle as a boolean.+ method := "plain_text"+ if value {+ method = "deepdoc"+ }+ if !target.hasFamily("pdf") {+ row.retire("layout_recognize", "the targeted pipeline has no PDF file family")+ return nil+ }+ writes.setSetup(target.parserNode, "pdf", "parse_method", method)+ return nil+ case string:+ trimmed := strings.TrimSpace(value)+ if trimmed == "" {+ // Python's ParserConfig declares layout_recognize str with+ // min_length 1, so an empty string is malformed rather than unset.+ return fmt.Errorf("layout_recognize must not be empty")+ }+ if ingestionLayoutModelReference(trimmed) {+ if !target.hasFamily("pdf") {+ row.retire("layout_recognize", "the targeted pipeline has no PDF file family")+ return nil+ }+ // These OCR selectors cannot be resolved by the Go ingestion dispatcher.+ // Preserve them by stopping rather than sending them to generic VLM chat.+ switch ingestionModelReferenceEngine(trimmed) {+ case "mistral ocr", "somark", "opendataloader", "paddleocr":+ return fmt.Errorf("layout_recognize %q has no supported Go model-reference dispatch", trimmed)+ }+ // Keep supported OCR and VLM model references whole.+ writes.setSetup(target.parserNode, "pdf", "layout_recognizer", trimmed)+ return nil+ }+ method, known := ingestionPDFParseMethod(trimmed)+ if !known {+ return fmt.Errorf("layout_recognize %q is not a supported Go parse method", trimmed)+ }+ if target.pdf {+ pdfMethod := method+ // The existing PDF dispatcher recognizes the full TCADP spelling.+ if method == "tcadp" {+ pdfMethod = "tcadp parser"+ }+ writes.setSetup(target.parserNode, "pdf", "parse_method", pdfMethod)+ }+ if method == "tcadp" && target.hasFamily("spreadsheet") {+ // Python applied one layout_recognizer to every file type, and+ // Go's TCADP engine serves the spreadsheet family too.+ writes.setSetup(target.parserNode, "spreadsheet", "parse_method", "tcadp")+ }+ if !target.hasFamily("pdf") && !(method == "tcadp" && target.hasFamily("spreadsheet")) {+ row.retire("layout_recognize", "the targeted pipeline has no file family this engine serves")+ }+ return nil+ default:+ return fmt.Errorf("layout_recognize must be a string or a boolean, got %T", raw)+ }+}++// convertIngestionParserPages moves the flat page range onto the PDF family. An+// unparseable range fails the batch, exactly as the same invalid range would+// fail a write through the API, rather than being silently dropped.+func convertIngestionParserPages(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {+ raw, ok := config["pages"]+ if !ok {+ return nil+ }+ if !target.hasFamily("pdf") {+ row.retire("pages", "the targeted pipeline has no PDF file family")+ return nil+ }+ pages, err := utility.NormalizePDFPages(raw)+ if err != nil {+ return fmt.Errorf("pages: %w", err)+ }+ if len(pages) == 0 {+ // "no value" and an empty range list both mean "parse every page",+ // which is what the template already does.+ return nil+ }+ out := make([]any, 0, len(pages))+ for _, span := range pages {+ out = append(out, []any{span[0], span[1]})+ }+ writes.setSetup(target.parserNode, "pdf", "pages", out)+ return nil+}++// convertIngestionTCADPOptions moves the TCADP response options onto every+// family whose effective engine is TCADP. Both keys are read from the family+// setup a parser configures itself with+// (internal/parser/parser/{csv,xls,xlsx,ppt,pptx}_parser.go and+// pdf_parser_common.go), so they follow the engine rather than the file type.+func convertIngestionTCADPOptions(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {+ keys := []string{"table_result_type", "markdown_image_response_type"}+ present := make(map[string]string, len(keys))+ for _, key := range keys {+ raw, ok := config[key]+ if !ok || raw == nil {+ continue+ }+ text, ok := raw.(string)+ if !ok {+ return fmt.Errorf("%s must be a string, got %T", key, raw)+ }+ if text != "" {+ present[key] = text+ }+ }+ if len(present) == 0 {+ return nil+ }+ families := make([]string, 0, 2)+ for _, family := range []string{"pdf", "spreadsheet"} {+ if !target.hasFamily(family) {+ continue+ }+ if ingestionEffectiveFamilyEngine(config, writes, target.parserNode, family) == "tcadp" {+ families = append(families, family)+ }+ }+ if len(families) == 0 {+ for key := range present {+ row.retire(key, "no file family of the targeted pipeline parses with TCADP")+ }+ return nil+ }+ for key, text := range present {+ for _, family := range families {+ writes.setSetup(target.parserNode, family, key, text)+ }+ }+ return nil+}++// ingestionEffectiveFamilyEngine resolves the engine a family will actually run+// with, mirroring the runtime precedence: an explicit param on the row wins over+// the value this migration derives.+func ingestionEffectiveFamilyEngine(config map[string]any, writes nodeWrites, parserNode, family string) string {+ if node, ok := config[parserNode].(map[string]any); ok {+ if setup, ok := node[family].(map[string]any); ok {+ if existing, ok := setup["parse_method"].(string); ok && existing != "" {+ if canonical, known := ingestionPDFParseMethod(existing); known {+ return canonical+ }+ return strings.ToLower(existing)+ }+ if model, ok := setup["layout_recognizer"].(string); ok && model != "" {+ return strings.ToLower(model)+ }+ }+ }+ if setup, ok := writes[parserNode][family].(map[string]any); ok {+ if derived, ok := setup["parse_method"].(string); ok && derived != "" {+ if method, known := ingestionPDFParseMethod(derived); known {+ return method+ }+ return strings.ToLower(derived)+ }+ }+ return ""+}++// convertIngestionTableColumns moves the table column role settings onto the+// spreadsheet family. The root-level keys must go: ingestion reads+// parser_config's root-level table_column_* first+// (internal/ingestion/task/indexdoc/process.go:resolveTableColumnConfig) and+// would otherwise shadow anything written to the node.+func convertIngestionTableColumns(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {+ modeKey, rolesKey, namesKey := "table_column_mode", "table_column_roles", "table_column_names"+ _, hasMode := config[modeKey]+ _, hasRoles := config[rolesKey]+ _, hasNames := config[namesKey]+ if !hasMode && !hasRoles && !hasNames {+ return nil+ }+ if !target.hasFamily("spreadsheet") {+ row.retire(modeKey, "the targeted pipeline has no spreadsheet file family")+ return nil+ }+ if raw, ok := config[modeKey]; ok && raw != nil {+ mode, ok := raw.(string)+ if !ok {+ return fmt.Errorf("%s must be a string, got %T", modeKey, raw)+ }+ if mode != "" {+ // Python types this as Literal["auto", "manual"] | None; anything+ // else was rejected by the API, so a stored other value is garbage.+ if mode != "auto" && mode != "manual" {+ row.retire(modeKey, fmt.Sprintf("%q is not a mode the table parser understands", mode))+ } else {+ writes.setSetup(target.parserNode, "spreadsheet", "column_mode", mode)+ }+ }+ }+ if raw, ok := config[rolesKey]; ok && raw != nil {+ roles, ok := raw.(map[string]any)+ if !ok {+ return fmt.Errorf("%s must be an object, got %T", rolesKey, raw)+ }+ normalized := make(map[string]any, len(roles))+ for column, role := range roles {+ text, _ := role.(string)+ switch text {+ case "vectorize":+ // Python normalizes the retired role name at validation time+ // (api/utils/validation_utils.py:legacy_vectorize_table_column_role).+ normalized[column] = "indexing"+ case "indexing", "metadata", "both":+ normalized[column] = text+ default:+ row.retire(rolesKey, fmt.Sprintf("column %q has unsupported role %q", column, text))+ }+ }+ if len(normalized) > 0 {+ writes.setSetup(target.parserNode, "spreadsheet", "column_roles", normalized)+ }+ }+ if raw, ok := config[namesKey]; ok && raw != nil {+ names, ok := raw.([]any)+ if !ok {+ return fmt.Errorf("%s must be a list, got %T", namesKey, raw)+ }+ out := make([]any, 0, len(names))+ for _, name := range names {+ text, ok := name.(string)+ if !ok || text == "" {+ row.retire(namesKey, fmt.Sprintf("dropping non-string column name %v", name))+ continue+ }+ out = append(out, text)+ }+ if len(out) > 0 {+ writes.setSetup(target.parserNode, "spreadsheet", "column_names", out)+ }+ }+ return nil+}++// ingestionLayoutModelReference reports whether a layout_recognize value names a+// VLM model rather than a parse engine: Python's "<llm>@<instance>@<provider>"+// composite (normalize_layout_recognizer) or a bare tenant model id.+func ingestionLayoutModelReference(raw string) bool {+ if strings.Contains(raw, "@") {+ return true+ }+ compact := strings.ReplaceAll(raw, "-", "")+ if len(compact) != 32 {+ return false+ }+ for _, r := range compact {+ if !((r >= '0' && r <= '9') || (r >= 'a' && r <= 'f') || (r >= 'A' && r <= 'F')) {+ return false+ }+ }+ return true+}++// ingestionModelReferenceEngine returns the engine segment of a composite model+// reference ("name@instance@provider" -> "provider"), or "" for a bare id.+func ingestionModelReferenceEngine(raw string) string {+ idx := strings.LastIndex(raw, "@")+ if idx < 0 {+ return ""+ }+ return strings.ToLower(strings.TrimSpace(raw[idx+1:]))+}++// convertIngestionChunker moves the flat chunking settings onto the row's+// chunker node. Which settings have a home depends on the chunker: only+// GeneralChunker and TokenChunker read a token size, delimiter list, overlap and+// context sizes.+func convertIngestionChunker(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {+ convertIngestionChunkTokenNum(config, target, writes, row)+ if err := convertIngestionDelimiter(config, target, writes, row); err != nil {+ return err+ }+ convertIngestionOverlappedPercent(config, target, writes, row)+ convertIngestionContextSizes(config, target, writes, row)+ return convertIngestionParentChild(config, target, writes, row)+}++// convertIngestionChunkTokenNum maps the flat token count onto the target+// chunker's own param name. The meaning is not uniform across chunkers, so each+// case states its own mapping or refuses.+func convertIngestionChunkTokenNum(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) {+ raw, ok := config["chunk_token_num"]+ if !ok || raw == nil {+ // Absent or null means "the default", which the template already+ // supplies: nothing is lost by deleting the flat key.+ return+ }+ size, valid := common.ToInt(raw)+ if !valid {+ row.retire("chunk_token_num", fmt.Sprintf("%v is not an integer", raw))+ return+ }+ switch target.chunker {+ case "GeneralChunker":+ // 0 is legal here and means "no split window", the same thing Python+ // set for spreadsheet and TCADP files at parse time.+ if size < 0 {+ row.retire("chunk_token_num", fmt.Sprintf("%d is negative", size))+ return+ }+ writes.set(target.chunkerNode, "chunk_token_size", size)+ case "TokenChunker":+ // schema/chunker.go rejects a non-positive chunk_token_size.+ if size <= 0 {+ row.retire("chunk_token_num", fmt.Sprintf("%d is not a positive chunk size", size))+ return+ }+ writes.set(target.chunkerNode, "chunk_token_size", size)+ case "TitleChunker":+ // Known semantic difference: Python used chunk_token_num as the merge+ // target of rag/app/{book,laws,paper}.py while Go's chunk_token_cap is a+ // hard ceiling on an emitted chunk. 0 disables the ceiling in Go, which+ // is what Python's "no cap" guard did, so it carries over; a non-zero+ // value must be inside the range the chunker accepts or the component+ // would refuse to build.+ if size != 0 && (size < 128 || size > 8000) {+ row.retire("chunk_token_num", fmt.Sprintf("%d is outside the title chunker's accepted 128..8000 range", size))+ return+ }+ writes.set(target.chunkerNode, "chunk_token_cap", size)+ default:+ row.retire("chunk_token_num", fmt.Sprintf("%s has no token size parameter", target.chunker))+ }+}++// convertIngestionDelimiter maps the flat delimiter string onto the canonical+// delimiter list, using the same parser the write path uses+// (pipeline.normalizeGeneralComponentParams), so a migrated row and an+// API-written row carry byte-identical lists.+func convertIngestionDelimiter(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {+ raw, ok := config["delimiter"]+ if !ok || raw == nil {+ return nil+ }+ if !target.acceptsChunkParams() {+ row.retire("delimiter", fmt.Sprintf("%s splits on its own structure", target.chunker))+ return nil+ }+ text, ok := raw.(string)+ if !ok {+ return fmt.Errorf("delimiter must be a string, got %T", raw)+ }+ if text == "" {+ // Python declares min_length 1; an empty value has no split points.+ return nil+ }+ delimiters := parserchunk.ParseDelimiterField(text)+ if len(delimiters) == 0 {+ return nil+ }+ out := make([]any, 0, len(delimiters))+ for _, delimiter := range delimiters {+ out = append(out, delimiter)+ }+ writes.set(target.chunkerNode, "delimiters", out)+ return nil+}++// convertIngestionOverlappedPercent copies the overlap across unchanged. The+// runtime normalizes it (schema.NormalizeOverlappedPercent turns a 0..1 fraction+// into a whole percentage and clamps it), which is where the API leaves the value+// too, so the stored number stays the user's own.+func convertIngestionOverlappedPercent(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) {+ raw, ok := config["overlapped_percent"]+ if !ok || raw == nil {+ return+ }+ if !target.acceptsChunkParams() {+ row.retire("overlapped_percent", fmt.Sprintf("%s has no overlap parameter", target.chunker))+ return+ }+ if _, ok := common.ToFloat(raw); !ok {+ row.retire("overlapped_percent", fmt.Sprintf("%v is not a number", raw))+ return+ }+ writes.set(target.chunkerNode, "overlapped_percent", raw)+}++// convertIngestionContextSizes moves the table and image context windows. Only+// the general and token chunkers grew them in Go; the title, manual and one+// chunkers Python used them with have no such parameter.+func convertIngestionContextSizes(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) {+ for _, key := range []string{"table_context_size", "image_context_size"} {+ raw, ok := config[key]+ if !ok || raw == nil {+ continue+ }+ if !target.acceptsChunkParams() {+ row.retire(key, fmt.Sprintf("%s has no context size parameter", target.chunker))+ continue+ }+ size, valid := common.ToInt(raw)+ if !valid || size < 0 {+ row.retire(key, fmt.Sprintf("%v is not a non-negative integer", raw))+ continue+ }+ writes.set(target.chunkerNode, key, size)+ }+}++// convertIngestionParentChild folds the three shapes the parent/child setting was+// stored in into the component-scoped group the chunker reads, and derives the+// children_delimiters array from it. Both halves are needed: the group is what+// the write path keeps, children_delimiters is what the chunker reads at run time+// (pipeline.ApplyParentChildChunkerConfig derives it while writing, which never+// runs for a row this migration rewrites).+func convertIngestionParentChild(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {+ groupRaw, hasGroup := config["parent_child"]+ enabledRaw, hasEnabled := config["enable_children"]+ delimiterRaw, hasDelimiter := config["children_delimiter"]+ if !hasGroup && !hasEnabled && !hasDelimiter {+ return nil+ }+ if !target.acceptsChunkParams() {+ row.retire("parent_child", fmt.Sprintf("%s cannot split parent chunks into children", target.chunker))+ return nil+ }++ useParentChild := false+ useFromGroup := false+ childrenDelimiter := ""+ delimiterFromGroup := false+ if hasGroup && groupRaw != nil {+ group, ok := groupRaw.(map[string]any)+ if !ok {+ return fmt.Errorf("parent_child must be an object, got %T", groupRaw)+ }+ if raw, exists := group["use_parent_child"]; exists && raw != nil {+ enabled, ok := raw.(bool)+ if !ok {+ return fmt.Errorf("parent_child.use_parent_child must be a boolean, got %T", raw)+ }+ useParentChild, useFromGroup = enabled, true+ }+ if text, ok := group["children_delimiter"].(string); ok {+ childrenDelimiter, delimiterFromGroup = text, true+ }+ }+ // The flat keys are the Python-era shape and rank below the group: on a row+ // that carries both, the current shape would otherwise be overruled by a+ // stale sibling.+ if hasEnabled && enabledRaw != nil && !useFromGroup {+ enabled, ok := enabledRaw.(bool)+ if !ok {+ return fmt.Errorf("enable_children must be a boolean, got %T", enabledRaw)+ }+ useParentChild = enabled+ }+ if hasDelimiter && delimiterRaw != nil && !delimiterFromGroup {+ text, ok := delimiterRaw.(string)+ if !ok {+ return fmt.Errorf("children_delimiter must be a string, got %T", delimiterRaw)+ }+ childrenDelimiter = text+ }+ if !delimiterFromGroup && (!hasDelimiter || delimiterRaw == nil) {+ childrenDelimiter = "\n"+ }+ if !useFromGroup && (!hasEnabled || enabledRaw == nil) && hasDelimiter {+ useParentChild = childrenDelimiter != ""+ }+ childrenDelimiter = decodePythonChildDelimiter(childrenDelimiter)+ group := map[string]any{"use_parent_child": useParentChild, "children_delimiter": childrenDelimiter}+ if node, ok := config[target.chunkerNode].(map[string]any); ok {+ if raw, present := node["parent_child"]; present {+ existing, ok := raw.(map[string]any)+ if !ok {+ return fmt.Errorf("%s.parent_child is not an object", target.chunkerNode)+ }+ for key, value := range existing {+ group[key] = value+ }+ }+ }+ useParentChild, _ = group["use_parent_child"].(bool)+ childrenDelimiter, _ = group["children_delimiter"].(string)+ writes.set(target.chunkerNode, "parent_child", group)+ children := []any{}+ if useParentChild {+ for _, delimiter := range parserchunk.ParseDelimiterField(childrenDelimiter) {+ children = append(children, delimiter)+ }+ }+ writes.set(target.chunkerNode, "children_delimiters", children)+ return nil+}++// Python decodes escaped control characters before splitting child chunks.+// Unknown escapes retain their backslash, as unicode_escape does.+func decodePythonChildDelimiter(text string) string {+ var out strings.Builder+ for len(text) > 0 {+ if text[0] == '\\' {+ char, _, tail, err := strconv.UnquoteChar(text, '"')+ if err == nil {+ out.WriteRune(char)+ text = tail+ continue+ }+ }+ out.WriteByte(text[0])+ text = text[1:]+ }+ return out.String()+}++// convertIngestionTokenizer moves the filename weight onto the row's tokenizer.+// A null weight is left alone: Go's default (0.1) is Python's default too, and+// the flat key is deleted either way.+func convertIngestionTokenizer(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {+ raw, ok := config["filename_embd_weight"]+ if !ok {+ return nil+ }+ // search_method and fields belong to the flow Tokenizer component in Python+ // too; the flat config never carried them, so they are neither migrated nor+ // deleted here (the extractor migration leaves topn_tags/tag_kb_ids alone for+ // the same reason).+ if raw == nil {+ return nil+ }+ weight, ok := common.ToFloat(raw)+ if !ok {+ return fmt.Errorf("filename_embd_weight must be a number, got %T", raw)+ }+ if _, isText := raw.(string); isText {+ // A numeric string reached the field; store the number the runtime will+ // actually use rather than the string it happens to parse.+ writes.set(target.tokenizerNode, "filename_embd_weight", weight)+ return nil+ }+ writes.set(target.tokenizerNode, "filename_embd_weight", raw)+ return nil+}++var extractorFlatKeys = []string{"auto_keywords", "auto_questions", "enable_metadata", "metadata", "built_in_metadata"}++// migrateExtractorConfig rewrites only Python's top-level extraction settings.+// Existing node fields win over flat Python values. A modular metadata transport+// object replaces node metadata, matching ApplyComponentScopedParserConfig.+// Unsupported inputs return an error before the config is mutated.+func migrateExtractorConfig(config map[string]any, parserID string, pipelineID *string) (bool, error) {+ present := false+ for _, key := range extractorFlatKeys {+ if _, ok := config[key]; ok {+ present = true+ break+ }+ }+ if !present {+ return false, nil+ }+ var targets []map[string]any+ var newNode map[string]any+ var extractorNode string+ if pipelineID == nil || strings.TrimSpace(*pipelineID) == "" {+ target, ok := ingestionTargets[strings.ToLower(strings.TrimSpace(parserID))]+ if !ok || target.extractorNode == "" {+ return false, nil+ }+ extractorNode = target.extractorNode+ if raw, exists := config[extractorNode]; exists {+ node, ok := raw.(map[string]any)+ if !ok {+ return false, fmt.Errorf("%s is not an object", extractorNode)+ }+ targets = append(targets, node)+ } else {+ newNode = map[string]any{}+ targets = append(targets, newNode)+ }+ } else {+ for id, raw := range config {+ id = strings.ToLower(id)+ if strings.HasPrefix(id, "extractor:") || strings.HasPrefix(id, "extractor_") {+ if node, ok := raw.(map[string]any); ok {+ targets = append(targets, node)+ }+ }+ }+ if len(targets) == 0 {+ return false, nil+ }+ }+ groups, replaceMetadata, err := convertPythonExtractorGroups(config)+ if err != nil {+ return false, err+ }+ for _, node := range targets {+ for name := range groups {+ if raw, exists := node[name]; exists && !(name == "metadata" && replaceMetadata) {+ if _, ok := raw.(map[string]any); !ok {+ return false, fmt.Errorf("%s group is not an object", name)+ }+ }+ }+ }+ for _, node := range targets {+ for name, defaults := range groups {+ group, ok := node[name].(map[string]any)+ if !ok || (name == "metadata" && replaceMetadata) {+ node[name] = maps.Clone(defaults)+ continue+ }+ for key, value := range defaults {+ if _, exists := group[key]; !exists {+ group[key] = value+ }+ }+ }+ }+ if newNode != nil {+ config[extractorNode] = newNode+ }+ for _, key := range extractorFlatKeys {+ delete(config, key)+ }+ return true, nil+}++func convertPythonExtractorGroups(config map[string]any) (map[string]map[string]any, bool, error) {+ groups := map[string]map[string]any{}+ for _, field := range []struct{ key, group string }{{"auto_keywords", "keywords"}, {"auto_questions", "questions"}} {+ if raw, exists := config[field.key]; exists {+ value, ok := raw.(float64)+ if raw == nil {+ value, ok = 0, true+ }+ if !ok || value < 0 || value != float64(int(value)) {+ return nil, false, fmt.Errorf("%s must be a non-negative integer", field.key)+ }+ groups[field.group] = map[string]any{"top_n": int(value)}+ }+ }+ source := config+ replaceMetadata := false+ _, hasToggle := config["enable_metadata"]+ _, hasFields := config["metadata"]+ _, hasBuiltIn := config["built_in_metadata"]+ if obj, ok := config["metadata"].(map[string]any); ok {+ if len(obj) == 0 && !hasToggle && !hasBuiltIn {+ source, replaceMetadata = obj, true+ }+ for _, key := range []string{"enabled", "metadata", "built_in_metadata"} {+ if _, exists := obj[key]; exists {+ source = obj+ replaceMetadata = true+ break+ }+ }+ }+ if hasToggle || hasFields || hasBuiltIn {+ flag := source["enable_metadata"]+ if replaceMetadata {+ flag = source["enabled"]+ }+ if flag != nil {+ switch value := flag.(type) {+ case bool:+ case float64:+ if value != 0 && value != 1 {+ return nil, false, fmt.Errorf("metadata enabled must be bool or 0/1")+ }+ default:+ return nil, false, fmt.Errorf("metadata enabled must be bool or 0/1")+ }+ }+ fields, err := convertExtractorMetadataFields(source["metadata"])+ if err != nil {+ return nil, false, fmt.Errorf("metadata: %w", err)+ }+ builtIn, err := convertExtractorMetadataFields(source["built_in_metadata"])+ if err != nil {+ return nil, false, fmt.Errorf("built_in_metadata: %w", err)+ }+ groups["metadata"] = map[string]any{"enabled": common.ParserConfigBool(flag), "metadata": fields, "built_in_metadata": builtIn}+ }+ return groups, replaceMetadata, nil+}++// convertExtractorMetadataFields accepts field lists and flat JSON Schemas that+// fit MetadataFieldDef. Constraints the target cannot express are rejected.+func convertExtractorMetadataFields(raw any) ([]common.MetadataFieldDef, error) {+ if raw == nil {+ return []common.MetadataFieldDef{}, nil+ }+ if schema, ok := raw.(map[string]any); ok {+ if len(schema) == 0 {+ return []common.MetadataFieldDef{}, nil+ }+ for key := range schema {+ if key != "type" && key != "properties" && key != "additionalProperties" {+ return nil, fmt.Errorf("unsupported JSON Schema keyword %s", key)+ }+ }+ properties, ok := schema["properties"].(map[string]any)+ if !ok || schema["type"] != "object" || schema["additionalProperties"] != false {+ return nil, fmt.Errorf("expected flat object schema with additionalProperties=false")+ }+ names := make([]string, 0, len(properties))+ for name := range properties {+ names = append(names, name)+ }+ sort.Strings(names)+ fields := make([]any, 0, len(names))+ for _, name := range names {+ prop, ok := properties[name].(map[string]any)+ if !ok {+ return nil, fmt.Errorf("property %s is not an object", name)+ }+ for key := range prop {+ if key != "type" && key != "description" && key != "enum" {+ return nil, fmt.Errorf("unsupported property constraint %s.%s", name, key)+ }+ }+ field := maps.Clone(prop)+ field["key"] = name+ fields = append(fields, field)+ }+ raw = fields+ }+ list, ok := raw.([]any)+ if !ok {+ return nil, fmt.Errorf("expected field list")+ }+ normalized := make([]any, 0, len(list))+ for _, rawField := range list {+ field, ok := rawField.(map[string]any)+ if !ok {+ return nil, fmt.Errorf("field is not an object")+ }+ key, ok := field["key"].(string)+ if !ok || key == "" || strings.TrimSpace(key) != key {+ return nil, fmt.Errorf("field requires a non-empty key without surrounding whitespace")+ }+ for _, name := range []string{"type", "description", "descriptions"} {+ if value := field[name]; value != nil {+ if _, ok := value.(string); !ok {+ return nil, fmt.Errorf("%s.%s must be a string", key, name)+ }+ }+ }+ if value := field["enum"]; value != nil {+ values, ok := value.([]any)+ if !ok {+ return nil, fmt.Errorf("%s.enum must be a string list", key)+ }+ for _, value := range values {+ if _, ok := value.(string); !ok {+ return nil, fmt.Errorf("%s.enum must be a string list", key)+ }+ }+ }+ item := maps.Clone(field)+ if description, _ := item["description"].(string); description == "" {+ item["description"] = item["descriptions"]+ }+ normalized = append(normalized, item)+ }+ return common.MetadataFieldDefsFromRaw(normalized), nil+}diff --git a/internal/dao/ingestion_parser_config_migration_test.go b/internal/dao/ingestion_parser_config_migration_test.gonew file mode 100644index 0000000000..e0f99cffcc--- /dev/null+++ b/internal/dao/ingestion_parser_config_migration_test.go@@ -0,0 +1,472 @@+package dao++import (+ "encoding/json"+ "fmt"+ "reflect"+ "strings"+ "testing"++ "github.com/glebarez/sqlite"+ "go.uber.org/zap"+ "go.uber.org/zap/zaptest/observer"+ "gorm.io/gorm"+ "ragflow/internal/common"+ "ragflow/internal/entity"+)++// Test bridges keep migration internals out of the production API.+var MigrateIngestionParserConfigsForTest = migrateIngestionParserConfigs+var MigrateParserChunkerTokenizerConfigForTest = migrateParserChunkerTokenizerConfig+var MigrateExtractorConfigForTest = migrateExtractorConfig++func TestMigrateParserChunkerTokenizerConfig(t *testing.T) {+ cases := []struct {+ name, parser, input, want string+ pipeline *string+ }{+ {+ name: "engine and chunk settings",+ input: `{"layout_recognize":true,"chunk_token_num":128,"delimiter":"\n","overlapped_percent":0.29,"table_context_size":100,"image_context_size":50,"filename_embd_weight":0.3}`,+ want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"deepdoc"}},"GeneralChunker:SixApplesFall":{"chunk_token_size":128,"delimiters":["\n"],"overlapped_percent":0.29,"table_context_size":100,"image_context_size":50},"Tokenizer:LegalReadersDecide":{"filename_embd_weight":0.3}}`,+ },+ {name: "engine off", input: `{"layout_recognize":false}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"}}}`},+ {name: "engine spellings", input: `{"layout_recognize":"plain text"}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"}}}`},+ {name: "tcadp reaches both families", input: `{"layout_recognize":"TCADP Parser"}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"tcadp parser"},"spreadsheet":{"parse_method":"tcadp"}}}`},+ {name: "tcadp on a spreadsheet-only pipeline", parser: "table", input: `{"layout_recognize":"TCADP Parser"}`, want: `{"Parser:HipSignsRhyme":{"spreadsheet":{"parse_method":"tcadp"}}}`},+ {name: "tcadp result options follow the engine", parser: "table", input: `{"layout_recognize":"TCADP Parser","table_result_type":"2","markdown_image_response_type":"2"}`, want: `{"Parser:HipSignsRhyme":{"spreadsheet":{"markdown_image_response_type":"2","parse_method":"tcadp","table_result_type":"2"}}}`},+ {name: "tcadp result options without the engine", parser: "table", input: `{"layout_recognize":"DeepDOC","table_result_type":"2"}`, want: `{}`},+ {name: "model reference stays a layout recognizer", input: `{"layout_recognize":"bge-m3@mineru"}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"layout_recognizer":"bge-m3@mineru"}}}`},+ {name: "existing engine wins over old model reference", input: `{"layout_recognize":"foo@MinerU","Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"}}}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"}}}`},+ {name: "existing model reference wins over old engine", input: `{"layout_recognize":"DeepDOC","Parser:HipSignsRhyme":{"pdf":{"layout_recognizer":"foo@MinerU"}}}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"layout_recognizer":"foo@MinerU"}}}`},+ {name: "existing engine supersedes unsupported old engine", input: `{"layout_recognize":"MagicOCR","Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"}}}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"}}}`},+ {name: "PDF selector does not mask spreadsheet engine", input: `{"layout_recognize":"TCADP Parser","Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"}}}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"},"spreadsheet":{"parse_method":"tcadp"}}}`},+ {name: "MinerU options reach the PDF setup", input: `{"layout_recognize":"MinerU","mineru_parse_method":"ocr","mineru_lang":"English"}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"mineru","mineru_parse_method":"ocr","mineru_lang":"English"}}}`},+ {name: "existing MinerU options win", input: `{"mineru_lang":"English","Parser:HipSignsRhyme":{"pdf":{"parse_method":"mineru","mineru_lang":"Chinese"}}}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"mineru","mineru_lang":"Chinese"}}}`},+ {name: "picture has no engine path", parser: "picture", input: `{"layout_recognize":"PaddleOCR"}`, want: `{}`},+ {name: "audio has no engine path", parser: "audio", input: `{"layout_recognize":true}`, want: `{}`},+ {name: "pages are normalized", input: `{"pages":[[3,5],[1,2],[6,7]]}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"pages":[[1,7]]}}}`},+ {name: "pages on a pdf-less pipeline", parser: "table", input: `{"pages":[[1,5]]}`, want: `{}`},+ {+ name: "table columns move to the spreadsheet family",+ parser: "table",+ input: `{"table_column_mode":"manual","table_column_roles":{"a":"vectorize","b":"metadata","c":"nonsense"},"table_column_names":["a","b",7]}`,+ want: `{"Parser:HipSignsRhyme":{"spreadsheet":{"column_mode":"manual","column_roles":{"a":"indexing","b":"metadata"},"column_names":["a","b"]}}}`,+ },+ {name: "table columns need a spreadsheet family", parser: "paper", input: `{"table_column_mode":"manual"}`, want: `{}`},+ {name: "zero chunk size on the general chunker", input: `{"chunk_token_num":0}`, want: `{"GeneralChunker:SixApplesFall":{"chunk_token_size":0}}`},+ {name: "zero chunk size on the token chunker", parser: "email", input: `{"chunk_token_num":0}`, want: `{}`},+ {name: "chunk size reached the email chunker", parser: "email", input: `{"chunk_token_num":256}`, want: `{"TokenChunker:WarmBreadSmells":{"chunk_token_size":256}}`},+ {name: "numeric string chunk size", input: `{"chunk_token_num":"256"}`, want: `{"GeneralChunker:SixApplesFall":{"chunk_token_size":256}}`},+ {name: "unusable chunk size is dropped", input: `{"chunk_token_num":"many"}`, want: `{}`},+ {name: "title chunker cap", parser: "book", input: `{"chunk_token_num":256}`, want: `{"TitleChunker:GrumpyGarlicsBake":{"chunk_token_cap":256}}`},+ {name: "title chunker cap out of range", parser: "book", input: `{"chunk_token_num":64}`, want: `{}`},+ {name: "manual chunker has no cap", parser: "manual", input: `{"chunk_token_num":512}`, want: `{}`},+ {name: "delimiter with backticks and unicode", input: "{\"delimiter\":\"`--`。?\"}", want: `{"GeneralChunker:SixApplesFall":{"delimiters":["` + "`--`" + `","。","?"]}}`},+ {name: "delimiter on a title chunker", parser: "book", input: `{"delimiter":"\n"}`, want: `{}`},+ {name: "context sizes need the general or token chunker", parser: "book", input: `{"table_context_size":10,"image_context_size":20}`, want: `{}`},+ {name: "overlap on a title chunker", parser: "book", input: `{"overlapped_percent":0.29}`, want: `{}`},+ {name: "parent child group", input: `{"parent_child":{"use_parent_child":true,"children_delimiter":"。"}}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"。"},"children_delimiters":["。"]}}`},+ {name: "legacy parent child flags", input: `{"enable_children":true,"children_delimiter":"。;"}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"。;"},"children_delimiters":["。",";"]}}`},+ {name: "parent child switched off", input: `{"enable_children":false}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":false,"children_delimiter":"\n"},"children_delimiters":[]}}`},+ {name: "group shape wins over the legacy flag", input: `{"parent_child":{"use_parent_child":false,"children_delimiter":"|"},"enable_children":true,"children_delimiter":"。"}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":false,"children_delimiter":"|"},"children_delimiters":[]}}`},+ {name: "parent child delimiters default to a newline", input: `{"parent_child":{"use_parent_child":true}}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"\n"},"children_delimiters":["\n"]}}`},+ {name: "parent child on a QA chunker", parser: "qa", input: `{"enable_children":true}`, want: `{}`},+ {name: "effective Python child delimiter", input: `{"children_delimiter":"。"}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"。"},"children_delimiters":["。"]}}`},+ {name: "escaped Python child delimiter", input: `{"enable_children":true,"children_delimiter":"\\n"}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"\n"},"children_delimiters":["\n"]}}`},+ {name: "explicit empty child delimiter stays disabled", input: `{"parent_child":{"use_parent_child":true,"children_delimiter":""}}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":""},"children_delimiters":[]}}`},+ {name: "existing disabled child group wins", input: `{"enable_children":true,"children_delimiter":"。","GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":false,"children_delimiter":"|"}}}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":false,"children_delimiter":"|"},"children_delimiters":[]}}`},+ {name: "existing enabled child group wins", input: `{"enable_children":false,"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"|"}}}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"|"},"children_delimiters":["|"]}}`},+ {name: "explicit runtime child delimiters win", input: `{"enable_children":true,"children_delimiter":"。","GeneralChunker:SixApplesFall":{"children_delimiters":["|"]}}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"。"},"children_delimiters":["|"]}}`},+ {name: "filename weight", parser: "picture", input: `{"filename_embd_weight":0}`, want: `{"Tokenizer:SharpLensFocus":{"filename_embd_weight":0}}`},+ {name: "null filename weight", parser: "picture", input: `{"filename_embd_weight":null}`, want: `{}`},+ {name: "null engine", input: `{"layout_recognize":null}`, want: `{}`},+ {name: "naive alias", parser: "naive", input: `{"layout_recognize":true}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"deepdoc"}}}`},+ {name: "presentation pdf family", parser: "presentation", input: `{"layout_recognize":true}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"deepdoc"}}}`},+ {name: "book carries its remove_toc", parser: "book", input: `{"layout_recognize":true}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"deepdoc","remove_toc":true}}}`},+ {name: "book carries remove_toc alongside pages", parser: "book", input: `{"pages":[[2,4]]}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"pages":[[2,4]],"remove_toc":true}}}`},+ {name: "laws carries its remove_toc", parser: "laws", input: `{"layout_recognize":"plaintext"}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text","remove_toc":true}}}`},+ {name: "paper carries multi column", parser: "paper", input: `{"layout_recognize":"plaintext"}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text","enable_multi_column":true}}}`},+ {+ name: "existing node params win",+ input: `{"layout_recognize":true,"chunk_token_num":4096,"filename_embd_weight":0.9,"Parser:HipSignsRhyme":{"pdf":{"parse_method":"somark"}},"GeneralChunker:SixApplesFall":{"chunk_token_size":512,"delimiters":["\n"]},"Tokenizer:LegalReadersDecide":{"filename_embd_weight":0.1}}`,+ want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"somark"}},"GeneralChunker:SixApplesFall":{"chunk_token_size":512,"delimiters":["\n"]},"Tokenizer:LegalReadersDecide":{"filename_embd_weight":0.1}}`,+ },+ {+ name: "an existing family map is merged, not replaced",+ parser: "book",+ input: `{"layout_recognize":true,"Parser:HipSignsRhyme":{"pdf":{"preprocess":["main_content"],"remove_toc":false}}}`,+ want: `{"Parser:HipSignsRhyme":{"pdf":{"preprocess":["main_content"],"remove_toc":false,"parse_method":"deepdoc"}}}`,+ },+ {name: "retired settings are deleted", input: `{"html4excel":true,"task_page_size":256,"analyze_hyperlink":true,"hyperlink_urls":true,"video_prompt":"x","mineru_formula_enable":true}`, want: `{}`},+ {name: "no owned keys", input: `{"chunk_method":"naive","topn_tags":5}`, want: `{"chunk_method":"naive","topn_tags":5}`},+ {name: "empty config", input: `{}`, want: `{}`},+ {name: "canvas rows are skipped", pipeline: ptrString("canvas-1"), input: `{"layout_recognize":true,"chunk_token_num":128}`, want: `{"layout_recognize":true,"chunk_token_num":128}`},+ {name: "unknown parser ids are preserved", parser: "knowledge_graph", input: `{"layout_recognize":true,"chunk_token_num":128}`, want: `{"layout_recognize":true,"chunk_token_num":128}`},+ }+ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ config, want := decodeIngestionConfig(t, tc.input), decodeIngestionConfig(t, tc.want)+ parser := tc.parser+ if parser == "" {+ parser = "general"+ }+ if _, err := migrateParserChunkerTokenizerConfig(config, parser, tc.pipeline, "knowledgebase", "row-1"); err != nil {+ t.Fatal(err)+ }+ if got := decodeIngestionConfig(t, encodeIngestionConfig(t, config)); !reflect.DeepEqual(got, want) {+ t.Fatalf("got %s; want %s", encodeIngestionConfig(t, config), tc.want)+ }+ snapshot := encodeIngestionConfig(t, config)+ if changed, err := migrateParserChunkerTokenizerConfig(config, parser, tc.pipeline, "knowledgebase", "row-1"); err != nil || changed {+ t.Fatalf("second run changed=%v, err=%v", changed, err)+ }+ if encodeIngestionConfig(t, config) != snapshot {+ t.Fatal("second run changed config")+ }+ })+ }+}++func TestMigrateParserChunkerTokenizerConfigRejectsMalformedValues(t *testing.T) {+ for _, raw := range []string{+ `{"layout_recognize":"MagicOCR"}`,+ `{"layout_recognize":"foo@mistral ocr"}`,+ `{"layout_recognize":"model@instance@somark"}`,+ `{"layout_recognize":"model@instance@opendataloader"}`,+ `{"layout_recognize":"model@instance@paddleocr"}`,+ `{"pages":"1-5"}`,+ `{"pages":[[0,5]]}`,+ `{"pages":[[5,1]]}`,+ `{"layout_recognize":5}`,+ `{"delimiter":5}`,+ `{"parent_child":"yes"}`,+ `{"parent_child":{"use_parent_child":"true"}}`,+ `{"enable_children":"true"}`,+ `{"children_delimiter":5}`,+ `{"table_column_mode":5}`,+ `{"table_column_roles":"a"}`,+ `{"table_column_names":"a"}`,+ `{"filename_embd_weight":"heavy"}`,+ `{"table_result_type":2}`,+ `{"layout_recognize":""}`,+ } {+ t.Run(raw, func(t *testing.T) {+ config := decodeIngestionConfig(t, raw)+ before := encodeIngestionConfig(t, config)+ if _, err := migrateParserChunkerTokenizerConfig(config, "general", nil, "knowledgebase", "row-1"); err == nil {+ t.Fatalf("malformed value was accepted: %s", before)+ }+ if encodeIngestionConfig(t, config) != before {+ t.Fatal("a rejected row was modified")+ }+ })+ }+}++func TestMigrateParserChunkerTokenizerConfigLogsRetiredFields(t *testing.T) {+ core, logs := observer.New(zap.WarnLevel)+ previous := common.Logger+ common.Logger = zap.New(core)+ t.Cleanup(func() { common.Logger = previous })+ config := decodeIngestionConfig(t, `{"html4excel":true,"task_page_size":12,"mineru_formula_enable":false}`)+ if _, err := migrateParserChunkerTokenizerConfig(config, "general", nil, "document", "doc-1"); err != nil {+ t.Fatal(err)+ }+ for _, field := range []string{"html4excel", "task_page_size", "mineru_formula_enable"} {+ entries := logs.FilterField(zap.String("field", field)).All()+ if len(entries) != 1 || entries[0].ContextMap()["table"] != "document" || entries[0].ContextMap()["row_id"] != "doc-1" {+ t.Errorf("missing row-specific warning for %s: %v", field, entries)+ }+ }+}++func TestMigrateIngestionParserConfigsBatches(t *testing.T) {+ db := openIngestionMigrationDB(t)+ for i := 0; i < 257; i++ {+ for _, table := range []string{"knowledgebase", "document"} {+ insertIngestionConfig(t, db, table, fmt.Sprintf("%032d", i), "general", python027IngestionConfig)+ }+ }+ if err := migrateIngestionParserConfigs(t.Context(), db); err != nil {+ t.Fatal(err)+ }+ for _, table := range []string{"knowledgebase", "document"} {+ var rows []ingestionConfigMigrationRow+ if err := db.Table(table).Find(&rows).Error; err != nil {+ t.Fatal(err)+ }+ if len(rows) != 257 {+ t.Fatalf("%s rows=%d", table, len(rows))+ }+ for _, row := range rows {+ config := decodeIngestionConfig(t, *row.ParserConfig)+ extractor, _ := config[ingestionTargets["general"].extractorNode].(map[string]any)+ keywords, _ := extractor["keywords"].(map[string]any)+ if keywords["top_n"] != float64(8) {+ t.Fatalf("%s.%s: extractor settings not migrated: %v", table, row.ID, config)+ }+ parserNode, ok := config["Parser:HipSignsRhyme"].(map[string]any)+ if !ok {+ t.Fatalf("%s.%s: parser node missing: %v", table, row.ID, config)+ }+ pdf, _ := parserNode["pdf"].(map[string]any)+ if pdf["parse_method"] != "plain_text" {+ t.Fatalf("%s.%s: pdf=%v", table, row.ID, pdf)+ }+ chunker, _ := config["GeneralChunker:SixApplesFall"].(map[string]any)+ if chunker["chunk_token_size"] != float64(256) {+ t.Fatalf("%s.%s: chunker=%v", table, row.ID, chunker)+ }+ if _, leftovers := config["chunk_token_num"]; leftovers {+ t.Fatalf("%s.%s: flat key survived: %v", table, row.ID, config)+ }+ }+ }+ if got := databaseVersion(t, db); got != ingestionParserConfigTargetVersion {+ t.Fatalf("version=%s", got)+ }+}++func TestMigrateIngestionParserConfigsRejectsInvalidRows(t *testing.T) {+ for _, raw := range []string{+ `{"pages":[[9,2]]}`,+ `not-json`,+ `{"auto_keywords":"8","auto_questions":2}`,+ `{"enable_metadata":true,"metadata":[{"name":"author"}]}`,+ `{"enable_metadata":true,"metadata":{"type":"object","required":["author"],"properties":{"author":{"type":"object","properties":{"name":{"type":"string"}}}}}}`,+ } {+ t.Run(raw, func(t *testing.T) {+ db := openIngestionMigrationDB(t)+ insertIngestionConfig(t, db, "knowledgebase", "a", "general", python027IngestionConfig)+ insertIngestionConfig(t, db, "knowledgebase", "b", "general", raw)+ if err := migrateIngestionParserConfigs(t.Context(), db); err == nil {+ t.Fatal("invalid input was marked as migrated")+ }+ var rows []ingestionConfigMigrationRow+ if err := db.Table("knowledgebase").Order("id").Find(&rows).Error; err != nil {+ t.Fatal(err)+ }+ if *rows[0].ParserConfig != python027IngestionConfig || *rows[1].ParserConfig != raw {+ t.Fatal("failed batch modified source configs")+ }+ if databaseVersion(t, db) != "" {+ t.Fatal("failed migration advanced version")+ }+ })+ }+}++func TestMigrateIngestionParserConfigsVersionAndEmptyRows(t *testing.T) {+ for _, tc := range []struct {+ version string+ skip bool+ }{+ {"", false}, {"v0.27.2", false}, {"v1.0.0-rc1", false},+ {"v1.0.0-rc2.dev2", true},+ {ingestionParserConfigTargetVersion, true}, {"v1.0.0", true},+ } {+ t.Run(tc.version, func(t *testing.T) {+ db := openIngestionMigrationDB(t)+ if tc.version != "" {+ writeDatabaseVersion(t, db, tc.version)+ }+ untouched := []string{``, `null`, `{}`, `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"somark"}}}`, `{"Extractor:AutoExtractDefault":{"metadata":{"enabled":false}}}`}+ for i, raw := range untouched {+ insertIngestionConfig(t, db, "knowledgebase", fmt.Sprint(i), "general", raw)+ }+ insertIngestionConfig(t, db, "knowledgebase", "old", "general", python027IngestionConfig)+ if err := migrateIngestionParserConfigs(t.Context(), db); err != nil {+ t.Fatal(err)+ }+ var rows []ingestionConfigMigrationRow+ if err := db.Table("knowledgebase").Order("id").Find(&rows).Error; err != nil {+ t.Fatal(err)+ }+ for i, raw := range untouched {+ if *rows[i].ParserConfig != raw {+ t.Fatalf("untouched row %d rewritten: %s", i, *rows[i].ParserConfig)+ }+ }+ unchanged := *rows[len(untouched)].ParserConfig == python027IngestionConfig+ if unchanged != tc.skip {+ t.Fatalf("version gate: skipped=%v; want %v", unchanged, tc.skip)+ }+ wantVersion := ingestionParserConfigTargetVersion+ if tc.skip {+ wantVersion = tc.version+ }+ if databaseVersion(t, db) != wantVersion {+ t.Fatal("unexpected version")+ }+ })+ }+}++// python027IngestionConfig is a dataset parser_config as Python v0.27.2 stored+// it: every value differs from the Go default so a lost field is visible.+const python027IngestionConfig = `{"auto_keywords":8,"layout_recognize":"plain text","chunk_token_num":256,"delimiter":"\n","overlapped_percent":0.29,"filename_embd_weight":0.2,"pages":[[1,3]],"table_context_size":10,"image_context_size":20,"html4excel":false}`++func openIngestionMigrationDB(t *testing.T) *gorm.DB {+ t.Helper()+ db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})+ if err != nil {+ t.Fatal(err)+ }+ sqlDB, err := db.DB()+ if err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { _ = sqlDB.Close() })+ if err := db.AutoMigrate(&entity.Knowledgebase{}, &entity.Document{}, &entity.SystemSettings{}); err != nil {+ t.Fatal(err)+ }+ return db+}++func insertIngestionConfig(t *testing.T, db *gorm.DB, table, id, parserID, raw string) {+ t.Helper()+ values := map[string]any{"id": id, "parser_id": parserID, "parser_config": raw, "created_by": "user"}+ if table == "knowledgebase" {+ values["tenant_id"] = "tenant"+ values["name"] = "dataset"+ values["embd_id"] = "embedding"+ } else {+ values["kb_id"] = "dataset"+ values["source_type"] = "local"+ values["type"] = "doc"+ values["suffix"] = "pdf"+ }+ if err := db.Table(table).Create(values).Error; err != nil {+ t.Fatal(err)+ }+}++func decodeIngestionConfig(t *testing.T, raw string) map[string]any {+ t.Helper()+ var config map[string]any+ if err := json.Unmarshal([]byte(raw), &config); err != nil {+ t.Fatal(err)+ }+ return config+}++func encodeIngestionConfig(t *testing.T, config map[string]any) string {+ t.Helper()+ raw, err := json.Marshal(config)+ if err != nil {+ t.Fatal(err)+ }+ return string(raw)+}++const python027ParserConfig = `{"auto_keywords":8,"auto_questions":3,"enable_metadata":true,"metadata":[{"key":"author","description":"who wrote it"}],"built_in_metadata":[{"key":"update_time"}],"topn_tags":5,"tag_kb_ids":["tag-kb"]}`++func TestMigrateExtractorConfig(t *testing.T) {+ cases := []struct {+ name, parser, input, want string+ pipeline *string+ }{+ {name: "Python fields", parser: "naive", input: python027ParserConfig, want: `{"Extractor:AutoExtractDefault":{"keywords":{"top_n":8},"questions":{"top_n":3},"metadata":{"enabled":true,"metadata":[{"key":"author","description":"who wrote it"}],"built_in_metadata":[{"key":"update_time"}]}},"topn_tags":5,"tag_kb_ids":["tag-kb"]}`},+ {name: "missing metadata toggle stays disabled", input: `{"metadata":[{"key":"author"}]}`, want: `{"Extractor:AutoExtractDefault":{"metadata":{"enabled":false,"metadata":[{"key":"author"}],"built_in_metadata":[]}}}`},+ {name: "explicit zero and false", input: `{"auto_keywords":0,"enable_metadata":false,"metadata":[]}`, want: `{"Extractor:AutoExtractDefault":{"keywords":{"top_n":0},"metadata":{"enabled":false,"metadata":[],"built_in_metadata":[]}}}`},+ {name: "existing node values win over Python fields", input: `{"auto_keywords":8,"enable_metadata":true,"metadata":[{"key":"author"}],"Extractor:AutoExtractDefault":{"keywords":{"top_n":2,"enabled":false},"metadata":{"enabled":false,"metadata":[],"built_in_metadata":[]}}}`, want: `{"Extractor:AutoExtractDefault":{"keywords":{"top_n":2,"enabled":false},"metadata":{"enabled":false,"metadata":[],"built_in_metadata":[]}}}`},+ {name: "modular transport is authoritative", input: `{"metadata":{"enabled":true,"metadata":[{"key":"author"}],"built_in_metadata":[]},"Extractor:AutoExtractDefault":{"metadata":{"enabled":false}}}`, want: `{"Extractor:AutoExtractDefault":{"metadata":{"enabled":true,"metadata":[{"key":"author"}],"built_in_metadata":[]}}}`},+ {name: "empty modular transport disables metadata", input: `{"metadata":{},"Extractor:AutoExtractDefault":{"metadata":{"enabled":true}}}`, want: `{"Extractor:AutoExtractDefault":{"metadata":{"enabled":false,"metadata":[],"built_in_metadata":[]}}}`},+ {name: "flat schema", input: `{"enable_metadata":true,"metadata":{"type":"object","properties":{"author":{"type":"string","description":"who","enum":["A"]}},"additionalProperties":false}}`, want: `{"Extractor:AutoExtractDefault":{"metadata":{"enabled":true,"metadata":[{"key":"author","type":"string","description":"who","enum":["A"]}],"built_in_metadata":[]}}}`},+ {name: "description alias", input: `{"enable_metadata":true,"metadata":[{"key":"author","descriptions":"who"}]}`, want: `{"Extractor:AutoExtractDefault":{"metadata":{"enabled":true,"metadata":[{"key":"author","description":"who"}],"built_in_metadata":[]}}}`},+ {name: "tag settings untouched", input: `{"topn_tags":5,"tag_kb_ids":["tag-kb"],"tags":{"tag_file_id":"file"}}`, want: `{"topn_tags":5,"tag_kb_ids":["tag-kb"],"tags":{"tag_file_id":"file"}}`},+ {name: "early Go nodes untouched", input: `{"Extractor:AutoExtractDefault":{"auto_keywords":8,"keywords_sys_prompt":"saved","enable_summary":true,"sys_prompt":"saved summary","auto_tags":5,"tag_file_id":"file","metadata_config":{"enabled":true}}}`, want: `{"Extractor:AutoExtractDefault":{"auto_keywords":8,"keywords_sys_prompt":"saved","enable_summary":true,"sys_prompt":"saved summary","auto_tags":5,"tag_file_id":"file","metadata_config":{"enabled":true}}}`},+ {name: "custom node", pipeline: ptrString("canvas-1"), input: `{"auto_questions":2,"Extractor:Custom":{"summary":{"enabled":true}}}`, want: `{"Extractor:Custom":{"questions":{"top_n":2},"summary":{"enabled":true}}}`},+ {name: "QA has no target", parser: "qa", input: python027ParserConfig, want: python027ParserConfig},+ {name: "table has no target", parser: "table", input: python027ParserConfig, want: python027ParserConfig},+ {name: "QA ignores stale extractor", parser: "qa", input: `{"auto_keywords":8,"Extractor:Old":{}}`, want: `{"auto_keywords":8,"Extractor:Old":{}}`},+ {name: "table ignores stale extractor", parser: "table", input: `{"auto_keywords":8,"Extractor:Old":{}}`, want: `{"auto_keywords":8,"Extractor:Old":{}}`},+ {name: "canvas has no target", pipeline: ptrString("canvas-1"), input: python027ParserConfig, want: python027ParserConfig},+ {name: "empty row", input: `{}`, want: `{}`},+ }+ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ config, want := decodeIngestionConfig(t, tc.input), decodeIngestionConfig(t, tc.want)+ parser := tc.parser+ if parser == "" {+ parser = "general"+ }+ if _, err := migrateExtractorConfig(config, parser, tc.pipeline); err != nil {+ t.Fatal(err)+ }+ if got := decodeIngestionConfig(t, encodeIngestionConfig(t, config)); !reflect.DeepEqual(got, want) {+ t.Fatalf("got %s; want %s", encodeIngestionConfig(t, config), tc.want)+ }+ snapshot := encodeIngestionConfig(t, config)+ if changed, err := migrateExtractorConfig(config, parser, tc.pipeline); err != nil || changed {+ t.Fatalf("second run changed=%v, err=%v", changed, err)+ }+ if encodeIngestionConfig(t, config) != snapshot {+ t.Fatal("second run changed config")+ }+ })+ }+}++func ptrString(value string) *string { return &value }++func TestMigrateIngestionParserConfigsSharesDev1Cleanup(t *testing.T) {+ if ingestionParserConfigTargetVersion != "v1.0.0-rc2.dev1" {+ t.Fatalf("migration version = %s", ingestionParserConfigTargetVersion)+ }+ db := openIngestionMigrationDB(t)+ insertIngestionConfig(t, db, "knowledgebase", "kb", "general", `{"raptor":{},"auto_keywords":8}`)+ if err := migrateIngestionParserConfigs(t.Context(), db); err != nil {+ t.Fatal(err)+ }+ var row ingestionConfigMigrationRow+ if err := db.Table("knowledgebase").Where("id = ?", "kb").Take(&row).Error; err != nil {+ t.Fatal(err)+ }+ config := decodeIngestionConfig(t, *row.ParserConfig)+ if _, remains := config["raptor"]; remains {+ t.Fatal("dataset cleanup was skipped")+ }+ if _, migrated := config[ingestionTargets["general"].extractorNode]; !migrated {+ t.Fatal("component conversion was skipped")+ }+}++func TestMigrateIngestionParserConfigsUpdateError(t *testing.T) {+ for _, table := range []string{"knowledgebase", "document"} {+ t.Run(table, func(t *testing.T) {+ db := openIngestionMigrationDB(t)+ for _, id := range []string{"row-a", "row-b"} {+ insertIngestionConfig(t, db, table, id, "general", python027IngestionConfig)+ }+ trigger := fmt.Sprintf(`CREATE TRIGGER reject_parser_config_update+ BEFORE UPDATE OF parser_config ON %s WHEN OLD.id = 'row-b'+ BEGIN SELECT RAISE(FAIL, 'forced parser config update failure'); END`, table)+ if err := db.Exec(trigger).Error; err != nil {+ t.Fatal(err)+ }+ err := migrateIngestionParserConfigs(t.Context(), db)+ if err == nil {+ t.Fatal("expected update failure")+ }+ for _, detail := range []string{table, "row-b", "forced parser config update failure"} {+ if !strings.Contains(err.Error(), detail) {+ t.Errorf("error %q does not identify %q", err, detail)+ }+ }+ var rows []ingestionConfigMigrationRow+ if err := db.Table(table).Find(&rows).Error; err != nil {+ t.Fatal(err)+ }+ for _, row := range rows {+ if *row.ParserConfig != python027IngestionConfig {+ t.Fatalf("failed batch changed row %s", row.ID)+ }+ }+ if got := databaseVersion(t, db); got != "" {+ t.Fatalf("failed migration advanced version to %q", got)+ }+ })+ }+}diff --git a/internal/dao/ingestion_parser_config_runtime_test.go b/internal/dao/ingestion_parser_config_runtime_test.gonew file mode 100644index 0000000000..1c9be4f2f0--- /dev/null+++ b/internal/dao/ingestion_parser_config_runtime_test.go@@ -0,0 +1,743 @@+package dao_test++import (+ "encoding/json"+ "reflect"+ "strings"+ "testing"++ "github.com/glebarez/sqlite"+ "gorm.io/gorm"++ "ragflow/internal/dao"+ "ragflow/internal/entity"+ "ragflow/internal/ingestion/component"+ "ragflow/internal/ingestion/component/chunker"+ "ragflow/internal/ingestion/component/schema"+ "ragflow/internal/ingestion/pipeline"+ "ragflow/internal/parser/parser"+ "ragflow/internal/service"+)++// python027MaximalIngestionConfig carries every Python-era key the migration can+// convert, so one run exercises every converter and every target family.+const python027MaximalIngestionConfig = `{+ "layout_recognize":"TCADP Parser",+ "pages":[[1,3]],+ "table_result_type":"2",+ "markdown_image_response_type":"2",+ "table_column_mode":"manual",+ "table_column_roles":{"a":"indexing"},+ "table_column_names":["a"],+ "chunk_token_num":256,+ "delimiter":"\n",+ "overlapped_percent":0.29,+ "table_context_size":10,+ "image_context_size":20,+ "parent_child":{"use_parent_child":true,"children_delimiter":"。"},+ "filename_embd_weight":0.2+}`++// TestMigratedIngestionConfigReachesRuntime drives the real row walk and then+// builds the real components from the stored config, so a migrated setting that+// the runtime cannot read fails here.+func TestMigratedIngestionConfigReachesRuntime(t *testing.T) {+ db := openIngestionRuntimeDB(t)+ insertIngestionRuntimeRow(t, db, "kb-general", "general",+ `{"layout_recognize":"plain text","pages":[[2,3]],"chunk_token_num":256,"delimiter":"\n","overlapped_percent":0.29,"filename_embd_weight":0.2}`)+ insertIngestionRuntimeRow(t, db, "kb-tcadp", "general", `{"layout_recognize":"TCADP Parser"}`)+ insertIngestionRuntimeRow(t, db, "kb-book", "book", `{"layout_recognize":true,"pages":[[1,2]],"chunk_token_num":300}`)+ if err := dao.MigrateIngestionParserConfigsForTest(t.Context(), db); err != nil {+ t.Fatal(err)+ }++ t.Run("general", func(t *testing.T) {+ config := migratedConfig(t, db, "kb-general")+ if _, leftovers := config["layout_recognize"]; leftovers {+ t.Fatalf("flat key survived: %v", config)+ }+ effective := effectiveComponentParams(t, "general", config)++ pdf := parser.NewPDFParser()+ pdf.ConfigureFromSetup(effective["Parser:HipSignsRhyme"]["pdf"].(map[string]any))+ if pdf.ParseMethod != "plain_text" {+ t.Errorf("parse_method=%q; want plain_text", pdf.ParseMethod)+ }+ if !reflect.DeepEqual(pdf.Pages, [][]int{{2, 3}}) {+ t.Errorf("pages=%v; want [[2 3]]", pdf.Pages)+ }++ chunkerParams := effective["GeneralChunker:SixApplesFall"]+ if _, err := chunker.NewGeneralChunker(chunkerParams); err != nil {+ t.Fatalf("migrated chunker params are not runtime-acceptable: %v", err)+ }+ if got := schema.NormalizeOverlappedPercent(chunkerParams["overlapped_percent"]); got != 29 {+ t.Errorf("overlapped_percent normalizes to %v; want 29", got)+ }+ if got := chunkerParams["chunk_token_size"]; got != float64(256) {+ t.Errorf("chunk_token_size=%v; want 256", got)+ }++ tokenizerParams := effective["Tokenizer:LegalReadersDecide"]+ if _, err := component.NewTokenizerComponent(tokenizerParams); err != nil {+ t.Fatalf("migrated tokenizer params are not runtime-acceptable: %v", err)+ }+ if got := tokenizerParams["filename_embd_weight"]; got != 0.2 {+ t.Errorf("filename_embd_weight=%v; want 0.2", got)+ }+ })++ t.Run("TCADP matches the existing runtime", func(t *testing.T) {+ effective := effectiveComponentParams(t, "general", migratedConfig(t, db, "kb-tcadp"))+ node := effective["Parser:HipSignsRhyme"]+ pdf := node["pdf"].(map[string]any)["parse_method"].(string)+ if pdf != "tcadp parser" || !parser.IsPDFParseMethod(pdf) {+ t.Fatalf("migrated PDF engine would be dispatched as a VLM: %q", pdf)+ }+ if got := node["spreadsheet"].(map[string]any)["parse_method"]; got != "tcadp" {+ t.Fatalf("spreadsheet engine=%v; want tcadp", got)+ }+ })++ t.Run("book keeps the defaults its template bakes", func(t *testing.T) {+ config := migratedConfig(t, db, "kb-book")+ node, ok := config["TitleChunker:GrumpyGarlicsBake"].(map[string]any)+ if !ok {+ t.Fatalf("no TitleChunker node written: %v", config)+ }+ if node["chunk_token_cap"] != float64(300) {+ t.Fatalf("chunk_token_cap=%v; want 300", node["chunk_token_cap"])+ }+ effective := effectiveComponentParams(t, "book", config)+ pdf := parser.NewPDFParser()+ pdf.ConfigureFromSetup(effective["Parser:HipSignsRhyme"]["pdf"].(map[string]any))+ // The book template bakes remove_toc=true, which component.defaultSetups()+ // does not: writing the pdf family must carry it, or a migrated dataset+ // would start indexing its own table of contents.+ if !pdf.RemoveTOC {+ t.Error("remove_toc was lost; the migrated pdf family dropped a template default")+ }+ if pdf.EnableMultiColumn {+ t.Error("enable_multi_column leaked in from another template")+ }+ if !reflect.DeepEqual(pdf.Pages, [][]int{{1, 2}}) {+ t.Errorf("pages=%v; want [[1 2]]", pdf.Pages)+ }+ })+}++// TestMigratedParentChildMatchesWritePath asserts the migration derives+// children_delimiters exactly as the API's own ApplyParentChildChunkerConfig+// does for the same setting, so a migrated row and a row saved through the UI+// agree. children_delimiters is the only half the chunker reads at run time.+func TestMigratedParentChildMatchesWritePath(t *testing.T) {+ const nodeID = "GeneralChunker:SixApplesFall"+ for _, tc := range []struct {+ name string+ input string+ want []any+ }{+ {"group with delimiter", `{"parent_child":{"use_parent_child":true,"children_delimiter":"。"}}`, []any{"。"}},+ {"group with multi delimiter", `{"parent_child":{"use_parent_child":true,"children_delimiter":"。;"}}`, []any{"。", ";"}},+ {"legacy flat keys", `{"enable_children":true,"children_delimiter":"。"}`, []any{"。"}},+ {"disabled", `{"enable_children":false,"children_delimiter":"。"}`, []any{}},+ {"disabled group", `{"parent_child":{"use_parent_child":false}}`, []any{}},+ } {+ t.Run(tc.name, func(t *testing.T) {+ config := decodeIngestionConfig(t, tc.input)+ if _, err := dao.MigrateParserChunkerTokenizerConfigForTest(config, "general", nil, "knowledgebase", "row-1"); err != nil {+ t.Fatal(err)+ }+ node, ok := config[nodeID].(map[string]any)+ if !ok {+ t.Fatalf("no chunker node written: %v", config)+ }+ got := node["children_delimiters"]+ if !reflect.DeepEqual(got, tc.want) {+ t.Fatalf("children_delimiters=%v; want %v", got, tc.want)+ }+ // The write path, given the same component config, must derive the+ // same list. It only ever reads the "parent_child" group.+ apiSided := entity.JSONMap{nodeID: cloneConfig(node)}+ pipeline.ApplyParentChildChunkerConfig(apiSided, nil)+ derived := apiSided[nodeID].(map[string]any)["children_delimiters"]+ if !reflect.DeepEqual(jsonNormalize(t, got), jsonNormalize(t, derived)) {+ t.Fatalf("migration wrote %v; the write path derives %v from the same group", got, derived)+ }+ if _, ok := node["parent_child"].(map[string]any); !ok {+ t.Fatalf("the parent_child group was not written: %v", node)+ }+ })+ }+}++// TestMigratedParentChildEnabledWithoutDelimiterDefaultsToNewline pins the one+// place the migration deliberately differs from ApplyParentChildChunkerConfig:+// Python's ParentChildConfig defaults children_delimiter to "\n" and its+// flattening writes that default into the config, while the Go write path+// derives an empty list when the delimiter is absent. Keeping Python's default+// preserves the children the Python-era run produced.+func TestMigratedParentChildEnabledWithoutDelimiterDefaultsToNewline(t *testing.T) {+ config := decodeIngestionConfig(t, `{"parent_child":{"use_parent_child":true}}`)+ if _, err := dao.MigrateParserChunkerTokenizerConfigForTest(config, "general", nil, "knowledgebase", "row-1"); err != nil {+ t.Fatal(err)+ }+ node := config["GeneralChunker:SixApplesFall"].(map[string]any)+ group := node["parent_child"].(map[string]any)+ if group["children_delimiter"] != "\n" {+ t.Fatalf("children_delimiter=%v; want the Python default \"\\n\"", group["children_delimiter"])+ }+ if !reflect.DeepEqual(node["children_delimiters"], []any{"\n"}) {+ t.Fatalf("children_delimiters=%v; want [\"\\n\"]", node["children_delimiters"])+ }+}++// TestIngestionMigrationMatchesBuiltinTemplates pins the migration's literal+// target table against the embedded templates by reading the node IDs and file+// families out of the templates themselves: a renamed component, a dropped+// family, or a new chunker type fails here instead of silently skipping a row or+// thinning a family.+func TestIngestionMigrationMatchesBuiltinTemplates(t *testing.T) {+ registry, err := pipeline.DefaultRegistry()+ if err != nil {+ t.Fatal(err)+ }+ // targetFamilies are the only Parser families the migration can write.+ const targetFamilies = "pdf,spreadsheet"+ for _, ref := range append(registry.Refs(), "naive") {+ t.Run(ref, func(t *testing.T) {+ template, ok := registry.Get(ref)+ if !ok {+ t.Fatal("template missing")+ }+ components := templateComponents(t, template.DSL)+ parserCpn := soleComponent(t, components, "Parser")+ chunkerCpn := soleComponentSuffix(t, components, "Chunker")+ tokenizerCpn := soleComponent(t, components, "Tokenizer")+ templateFamilies := parserFamilies(t, parserCpn.params)+ rawDSL, err := json.Marshal(template.DSL)+ if err != nil {+ t.Fatal(err)+ }++ for _, tc := range []struct {+ name string+ // config is the Python-era row being migrated.+ config string+ // families are the template families the config must land on;+ // every other target family must be left alone.+ families map[string]bool+ // chunker reports whether the config carries a chunker-owned key.+ chunker bool+ // tokenizer reports whether the config carries a tokenizer-owned key.+ tokenizer bool+ }{+ {+ name: "maximal",+ config: python027MaximalIngestionConfig,+ families: map[string]bool{"pdf": true, "spreadsheet": true},+ chunker: true,+ tokenizer: true,+ },+ {+ name: "engine",+ config: `{"layout_recognize":"TCADP Parser"}`,+ families: map[string]bool{"pdf": true, "spreadsheet": true},+ },+ {+ name: "columns",+ config: `{"table_column_mode":"manual","table_column_roles":{"a":"indexing"},"table_column_names":["a"]}`,+ families: map[string]bool{"spreadsheet": true},+ },+ {+ name: "chunking",+ config: `{"chunk_token_num":256,"delimiter":"\n","overlapped_percent":0.29,"enable_children":true}`,+ families: map[string]bool{},+ chunker: true,+ },+ } {+ t.Run(tc.name, func(t *testing.T) {+ config := decodeIngestionConfig(t, tc.config)+ changed, err := dao.MigrateParserChunkerTokenizerConfigForTest(config, ref, nil, "knowledgebase", "row-1")+ if err != nil || !changed {+ t.Fatalf("changed=%v err=%v", changed, err)+ }++ if tc.name == "maximal" && len(config) == 0 {+ t.Fatal("migration wrote nothing for the maximal config")+ }+ // Snapshot before cleaning so mutations cannot hide a lost parameter.+ expected := jsonNormalize(t, config)+ cleaned := pipeline.CleanComponentParams(rawDSL, config)+ if !reflect.DeepEqual(jsonNormalize(t, cleaned), expected) {+ t.Fatalf("API cleaning changed migrated params: got %v; want %v", cleaned, expected)+ }++ // Parser: exactly the template's families the config targets,+ // never a family the template does not declare.+ wantFamilies := map[string]bool{}+ for family := range tc.families {+ if _, present := templateFamilies[family]; present {+ wantFamilies[family] = true+ }+ }+ gotFamilies := map[string]bool{}+ node, nodeWritten := config[parserCpn.cpnID].(map[string]any)+ for family, raw := range node {+ familyParams, isMap := raw.(map[string]any)+ if !isMap {+ t.Fatalf("family %q is not an object: %v", family, raw)+ }+ if !strings.Contains(targetFamilies, family) {+ t.Fatalf("migration wrote family %q, which is not a migration target: %v", family, familyParams)+ }+ gotFamilies[family] = true+ }+ if !reflect.DeepEqual(gotFamilies, wantFamilies) {+ t.Fatalf("Parser families %v; want %v (config=%s)", gotFamilies, wantFamilies, tc.config)+ }+ if len(wantFamilies) == 0 && nodeWritten {+ t.Fatalf("Parser node written for a config that targets no family: %v", node)+ }+ if nodeWritten {+ assertFamiliesPreserved(t, node, templateFamilies, wantFamilies)+ }+ // The Parser the run compiles must accept the override. The+ // merge is applyOverrideParams': the template params with the+ // stored component params on top.+ if _, err := component.NewParserComponent(mergeComponentParams(parserCpn.params, node)); err != nil {+ t.Fatalf("migrated Parser params do not compile: %v", err)+ }++ // Chunker: only the chunkers that own the migrated params+ // may receive a node, on the template's own node ID.+ chunkerNode, chunkerWritten := config[chunkerCpn.cpnID].(map[string]any)+ switch {+ case !tc.chunker:+ if chunkerWritten {+ t.Fatalf("chunker node written for a chunker-agnostic config: %v", chunkerNode)+ }+ case chunkerCpn.name == "GeneralChunker" || chunkerCpn.name == "TokenChunker":+ if !chunkerWritten {+ t.Fatalf("no %s node written", chunkerCpn.cpnID)+ }+ if _, err := chunkerNodeAccepts(chunkerCpn.name, mergeComponentParams(chunkerCpn.params, chunkerNode)); err != nil {+ t.Fatalf("migrated chunker params do not compile: %v", err)+ }+ case chunkerCpn.name == "TitleChunker":+ if !chunkerWritten {+ t.Fatalf("no %s node written", chunkerCpn.cpnID)+ }+ if value, ok := schema.NumericFromAny(chunkerNode["chunk_token_cap"]); !ok || value != 256 {+ t.Fatalf("chunk_token_cap=%v; want 256", chunkerNode["chunk_token_cap"])+ }+ default:+ if chunkerWritten {+ t.Fatalf("%s must not receive migrated chunker params: %v", chunkerCpn.name, chunkerNode)+ }+ }++ // Tokenizer: every template has one, and the filename weight+ // is the only param the migration puts on it.+ tokenizerNode, tokenizerWritten := config[tokenizerCpn.cpnID].(map[string]any)+ if !tc.tokenizer {+ if tokenizerWritten {+ t.Fatalf("tokenizer node written for a config with no tokenizer param: %v", tokenizerNode)+ }+ } else {+ if !tokenizerWritten {+ t.Fatalf("no %s node written", tokenizerCpn.cpnID)+ }+ if value, ok := schema.NumericFromAny(tokenizerNode["filename_embd_weight"]); !ok || value != 0.2 {+ t.Fatalf("filename_embd_weight=%v; want 0.2", tokenizerNode["filename_embd_weight"])+ }+ if _, err := component.NewTokenizerComponent(mergeComponentParams(tokenizerCpn.params, tokenizerNode)); err != nil {+ t.Fatalf("migrated tokenizer params do not compile: %v", err)+ }+ }++ // No node outside the migration's three targets.+ for cpnID := range config {+ if cpnID == parserCpn.cpnID || cpnID == chunkerCpn.cpnID || cpnID == tokenizerCpn.cpnID {+ continue+ }+ t.Fatalf("migration wrote an unexpected component: %q", cpnID)+ }+ })+ }+ })+ }+}++func chunkerNodeAccepts(name string, params map[string]any) (any, error) {+ if name == "TokenChunker" {+ return chunker.NewTokenChunker(params)+ }+ return chunker.NewGeneralChunker(params)+}++// mergeComponentParams overlays the component-scoped overrides on the template's+// baked params the way internal/agent/canvas.applyOverrideParams does at run+// time: the override wins on the top-level key, so an override replaces a whole+// Parser family.+func mergeComponentParams(templateParams, overrides map[string]any) map[string]any {+ out := make(map[string]any, len(templateParams)+len(overrides))+ for key, value := range templateParams {+ out[key] = value+ }+ for key, value := range overrides {+ out[key] = value+ }+ return out+}++// templateComponent is one component of a builtin template.+type templateComponent struct {+ cpnID string+ name string+ params map[string]any+}++func templateComponents(t *testing.T, dsl map[string]any) []templateComponent {+ t.Helper()+ raw, err := json.Marshal(dsl)+ if err != nil {+ t.Fatal(err)+ }+ schemas, err := pipeline.ExtractAllComponentParams(raw)+ if err != nil {+ t.Fatal(err)+ }+ out := make([]templateComponent, 0, len(schemas))+ for _, s := range schemas {+ out = append(out, templateComponent{cpnID: s.CpnID, name: s.ComponentName, params: s.ParamsDefaults})+ }+ return out+}++func soleComponent(t *testing.T, components []templateComponent, name string) templateComponent {+ t.Helper()+ found := templateComponent{}+ count := 0+ for _, c := range components {+ if c.name == name {+ found, count = c, count+1+ }+ }+ if count != 1 {+ t.Fatalf("template has %d %s components; want 1", count, name)+ }+ return found+}++func soleComponentSuffix(t *testing.T, components []templateComponent, suffix string) templateComponent {+ t.Helper()+ found := templateComponent{}+ count := 0+ for _, c := range components {+ if strings.HasSuffix(c.name, suffix) {+ found, count = c, count+1+ }+ }+ if count != 1 {+ t.Fatalf("template has %d *%s components; want 1", count, suffix)+ }+ return found+}++func parserFamilies(t *testing.T, params map[string]any) map[string]map[string]any {+ t.Helper()+ out := map[string]map[string]any{}+ for _, key := range []string{"pdf", "spreadsheet"} {+ if raw, exists := params[key]; exists {+ family, ok := raw.(map[string]any)+ if !ok {+ t.Fatalf("Parser family %q is not an object", key)+ }+ out[key] = family+ }+ }+ return out+}++// overriddenFamilyKeys are family params the migration overwrites by design+// whenever the config supplies them, so the migrated value may differ from the+// template's.+var overriddenFamilyKeys = map[string]bool{+ "parse_method": true,+ "pages": true,+ "column_mode": true,+ "column_roles": true,+ "column_names": true,+ "layout_recognizer": true,+ "table_result_type": true,+ "markdown_image_response_type": true,+}++// inertFamilyKeys are template-baked family keys the migration may leave out,+// with the reason dropping them is harmless. component.NewParserComponent seeds+// every family from defaultSetups() and then overlays the params, and the+// run-level override replaces a family wholesale, so whatever a written family+// leaves out comes from defaultSetups() — equal to the template's value, or+// read by nobody, for every entry below.+//+// A key may appear here and in overriddenFamilyKeys: the two answer different+// questions (this one, "is the template's value safe to lose"; that one, "may+// the value the migration writes differ from the template's").+var inertFamilyKeys = map[string]string{+ "output_format": "normalizeParserOutputFormats forces JSON on every family",+ "preprocess": "the value has no Go reader; the preprocessing stage is selected by the chunker's own options",+ "suffix": "no Go reader; file routing uses the document's own suffix column, not the family list",+ "vlm": "empty in every template; absent falls back to the family llm_id and then the tenant model, and only Markdown and audio read a family vlm at all",+ "flatten_media_to_text": "false in every template, which is also defaultSetups()'s default",+ "pages": "the only template that bakes it lists every page ([[1,100000]]), which is what an absent pages means",+ "parse_method": "a written pdf family always restates it; for the spreadsheet family defaultSetups() re-derives the same deepdoc engine",+ "column_mode": "the table template bakes auto, which is what an absent column_mode means to indexdoc's resolveTableColumnConfig",+ "column_roles": "the table template bakes the empty role map, which is what an absent column_roles means",+ "column_names": "the table template bakes the empty name list, which is what an absent column_names means",+}++// assertFamiliesPreserved checks that no template-baked key of a written family+// is left unaccounted for: a key that would simply disappear from the effective+// config must be either carried or documented above.+func assertFamiliesPreserved(t *testing.T, node map[string]any, templateFamilies map[string]map[string]any, written map[string]bool) {+ t.Helper()+ for family := range written {+ templateFamily, ok := templateFamilies[family]+ if !ok {+ t.Fatalf("migration wrote family %q, which the template does not declare", family)+ }+ familyParams, _ := node[family].(map[string]any)+ for key, templateValue := range templateFamily {+ got, present := familyParams[key]+ if !present {+ if _, inert := inertFamilyKeys[key]; !inert {+ t.Fatalf("family %s: template key %q would be dropped by the override and is not accounted for; carry it or document it as inert", family, key)+ }+ continue+ }+ if overriddenFamilyKeys[key] {+ continue+ }+ if !reflect.DeepEqual(got, templateValue) {+ t.Fatalf("family %s: key %q = %v, but the template bakes %v", family, key, got, templateValue)+ }+ }+ }+}++// effectiveComponentParams reproduces the run-level merge: the template's baked+// component params with the stored parser_config overlaid, the stored value+// winning on the top-level key (internal/agent/canvas.applyOverrideParams), then+// component.NewParserComponent's own defaults underneath.+func effectiveComponentParams(t *testing.T, parserID string, config map[string]any) map[string]map[string]any {+ t.Helper()+ registry, err := pipeline.DefaultRegistry()+ if err != nil {+ t.Fatal(err)+ }+ template, ok := registry.Get(parserID)+ if !ok {+ t.Fatal("template missing")+ }+ out := map[string]map[string]any{}+ for _, c := range templateComponents(t, template.DSL) {+ out[c.cpnID] = c.params+ }+ for cpnID, raw := range config {+ params, ok := raw.(map[string]any)+ if !ok {+ t.Fatalf("stored cpnID %q is not an object", cpnID)+ }+ out[cpnID] = mergeComponentParams(out[cpnID], params)+ }+ return out+}++func openIngestionRuntimeDB(t *testing.T) *gorm.DB {+ t.Helper()+ db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})+ if err != nil {+ t.Fatal(err)+ }+ sqlDB, err := db.DB()+ if err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { _ = sqlDB.Close() })+ if err := db.AutoMigrate(&entity.Knowledgebase{}, &entity.SystemSettings{}); err != nil {+ t.Fatal(err)+ }+ return db+}++func insertIngestionRuntimeRow(t *testing.T, db *gorm.DB, id, parserID, config string) {+ t.Helper()+ err := db.Exec(+ `INSERT INTO knowledgebase (id,tenant_id,name,embd_id,created_by,parser_id,parser_config) VALUES (?,?,?,?,?,?,?)`,+ id, "tenant-1", "dataset", "embedding", "user-1", parserID, config).Error+ if err != nil {+ t.Fatal(err)+ }+}++func migratedConfig(t *testing.T, db *gorm.DB, id string) map[string]any {+ t.Helper()+ var raw string+ if err := db.Table("knowledgebase").Where("id = ?", id).Pluck("parser_config", &raw).Error; err != nil {+ t.Fatal(err)+ }+ return decodeIngestionConfig(t, raw)+}++func decodeIngestionConfig(t *testing.T, raw string) map[string]any {+ t.Helper()+ var config map[string]any+ if err := json.Unmarshal([]byte(raw), &config); err != nil {+ t.Fatal(err)+ }+ return config+}++// jsonNormalize round-trips a value through JSON so two structurally equal+// values that differ only in Go slice type ([]string vs []any, which is what a+// DB round-trip produces either way) compare equal.+func jsonNormalize(t *testing.T, value any) any {+ t.Helper()+ raw, err := json.Marshal(value)+ if err != nil {+ t.Fatal(err)+ }+ var out any+ if err := json.Unmarshal(raw, &out); err != nil {+ t.Fatal(err)+ }+ return out+}++// cloneConfig deep-copies a parser_config so a helper that mutates its argument+// cannot corrupt the caller's fixture.+func cloneConfig(config map[string]any) map[string]any {+ raw, err := json.Marshal(config)+ if err != nil {+ return config+ }+ var out map[string]any+ if err := json.Unmarshal(raw, &out); err != nil {+ return config+ }+ return out+}++func TestMigratedExtractorConfigReachesRuntime(t *testing.T) {+ db := openIngestionRuntimeDB(t)+ input := `{"auto_keywords":8,"auto_questions":2,"enable_metadata":true,"metadata":[{"key":"author","type":"string"}],"built_in_metadata":[{"key":"file_name","type":"string"}],"Extractor:AutoExtractDefault":{"tags":{"top_n":5,"tag_file_id":"tag-file"},"keywords":{"system_prompt":"custom keywords"},"summary":{"enabled":true,"system_prompt":"custom summary"}}}`+ insertIngestionRuntimeRow(t, db, "kb-1", "general", input)+ if err := dao.MigrateIngestionParserConfigsForTest(t.Context(), db); err != nil {+ t.Fatal(err)+ }+ config := migratedConfig(t, db, "kb-1")+ got := extractorFromMigratedConfig(t, config).Param+ if !got.Keywords.Enabled || got.Keywords.TopN != 8 || got.Keywords.SystemPrompt != "custom keywords" || !got.Questions.Enabled || got.Questions.TopN != 2 {+ t.Fatalf("keyword/question settings lost: %#v", got)+ }+ if !got.Metadata.Enabled || len(got.Metadata.Metadata) != 1 || got.Metadata.Metadata[0].Key != "author" || len(got.Metadata.BuiltInMetadata) != 1 || got.Metadata.BuiltInMetadata[0].Key != "file_name" {+ t.Fatalf("metadata=%#v", got.Metadata)+ }+ if !got.Tags.Enabled || got.Tags.TagFileID != "tag-file" || got.Tags.TopN != 5 || !got.Summary.Enabled || got.Summary.SystemPrompt != "custom summary" || got.LLMID != "tenant-llm" {+ t.Fatalf("existing scoped settings lost: %#v", got)+ }+}++func TestExtractorMigrationMatchesBuiltinTemplates(t *testing.T) {+ registry, err := pipeline.DefaultRegistry()+ if err != nil {+ t.Fatal(err)+ }+ for _, ref := range append(registry.Refs(), "naive") {+ t.Run(ref, func(t *testing.T) {+ template, ok := registry.Get(ref)+ if !ok {+ t.Fatal("template missing")+ }+ components := template.DSL["components"].(map[string]any)+ _, hasExtractor := components["Extractor:AutoExtractDefault"]+ config := map[string]any{"auto_keywords": float64(8)}+ changed, err := dao.MigrateExtractorConfigForTest(config, ref, nil)+ if err != nil {+ t.Fatal(err)+ }+ if changed != hasExtractor {+ t.Fatalf("changed=%v; template has Extractor=%v", changed, hasExtractor)+ }+ if hasExtractor {+ if node, ok := config["Extractor:AutoExtractDefault"].(map[string]any); !ok || node["keywords"].(map[string]any)["top_n"] != 8 {+ t.Fatalf("config=%v", config)+ }+ } else if config["auto_keywords"] != float64(8) {+ t.Fatal("settings without target were discarded")+ }+ })+ }+}++func TestExtractorMigrationPreservesRuntimeSettings(t *testing.T) {+ for _, tt := range []struct {+ name, input string+ check func(*testing.T, *component.ExtractorComponent)+ }{+ {+ name: "empty Python metadata keeps builtins",+ input: `{"enable_metadata":true,"metadata":{},"built_in_metadata":[{"key":"file_name"}]}`,+ check: func(t *testing.T, extractor *component.ExtractorComponent) {+ got := extractor.Param.Metadata+ if !got.Enabled || len(got.BuiltInMetadata) != 1 || got.BuiltInMetadata[0].Key != "file_name" {+ t.Fatalf("metadata=%#v", got)+ }+ },+ },+ {+ name: "builtin uses active extractor despite stale node",+ input: `{"auto_keywords":8,"Extractor:Old":{"summary":{"enabled":true}}}`,+ check: func(t *testing.T, extractor *component.ExtractorComponent) {+ got := extractor.Param.Keywords+ if !got.Enabled || got.TopN != 8 {+ t.Fatalf("keywords=%#v", got)+ }+ },+ },+ } {+ t.Run(tt.name, func(t *testing.T) {+ var config map[string]any+ if err := json.Unmarshal([]byte(tt.input), &config); err != nil {+ t.Fatal(err)+ }+ if changed, err := dao.MigrateExtractorConfigForTest(config, "general", nil); err != nil || !changed {+ t.Fatalf("changed=%v, err=%v", changed, err)+ }+ tt.check(t, extractorFromMigratedConfig(t, config))+ })+ }+}++func extractorFromMigratedConfig(t *testing.T, config map[string]any) *component.ExtractorComponent {+ t.Helper()+ registry, err := pipeline.DefaultRegistry()+ if err != nil {+ t.Fatal(err)+ }+ template, ok := registry.Get("general")+ if !ok {+ t.Fatal("general template missing")+ }+ dsl, err := json.Marshal(template.DSL)+ if err != nil {+ t.Fatal(err)+ }+ cleaned := service.ApplyComponentScopedParserConfig(pipeline.BuildParserConfig(dsl, config), "tenant-llm")+ c, err := component.NewExtractorComponent(cleaned["Extractor:AutoExtractDefault"].(map[string]any))+ if err != nil {+ t.Fatal(err)+ }+ return c.(*component.ExtractorComponent)+}