Skip to content

RAGFlow is an agent-native database: a Retrieval-Augmented Generation (RAG) engine that grew into context infrastructure.

fix(dao): migrate Python-era flat ingestion config onto pipeline components (#20986)

jay77721b859e0460ebe
diff --git a/docs/administrator/migration/database_change_log.md b/docs/administrator/migration/database_change_log.md
index 919e8d575c..8f2b169b28 100644
--- a/docs/administrator/migration/database_change_log.md
+++ b/docs/administrator/migration/database_change_log.md
@@ -16,7 +16,7 @@ A version-by-version record of the schema and data changes the Go backend applie
- The marker is a row in `system_settings` with the key `mysql_migration.database.version`.
- The marker prevents a migration step from running again after that step has completed successfully.
-- Go defines the versions as constants: `modelMigrationBaseVersion` and `modelMigrationTargetVersion` in [migration_version.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/migration_version.go), `conversationHistoryTargetVersion` in [conversation_history_migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/conversation_history_migration.go), and `knowledgebaseParserConfigTargetVersion` in [migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/migration.go).
+- Go defines the versions as constants: `modelMigrationBaseVersion` and `modelMigrationTargetVersion` in [migration_version.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/migration_version.go), `conversationHistoryTargetVersion` in [conversation_history_migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/conversation_history_migration.go), `knowledgebaseParserConfigTargetVersion` in [migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/migration.go), and `ingestionParserConfigTargetVersion` in [ingestion_parser_config_migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/ingestion_parser_config_migration.go). The two parser-config constants share `v1.0.0-rc2.dev1`.
- A step runs when the stored version is lower than the step's version. Versions are compared with Python PEP 440 semantics, where a `dev` prerelease sorts *before* the release it precedes (`v1.0.0-rc1.dev1` < `v1.0.0-rc1`). A missing or unparsable marker never skips a step.
- `checkDatabaseVersion` in [cmd/ragflow_server.go](https://github.com/infiniflow/ragflow/blob/main/cmd/ragflow_server.go) refuses to start when the code version is older than the stored database version, so a database migrated by a newer release cannot be opened by an older one.
@@ -26,7 +26,7 @@ Startup order in `InitDB` ([database.go](https://github.com/infiniflow/ragflow/b
2. `migrateIngestionLogRunIdentity` — ingestion log columns and indexes.
3. `AutoMigrate` — converges every ORM model.
4. `migrateConversationHistory` — runs *after* `AutoMigrate`, which creates the child tables it backfills.
-5. `migrateKnowledgebaseParserConfig` — runs last, so the marker does not move past the conversation split's version before that split has run.
+5. `migrateIngestionParserConfigs` — converts ingestion settings on datasets and documents, then invokes the existing `migrateKnowledgebaseParserConfig` cleanup. Both run after the conversation split so the shared marker cannot skip that split.
## Version overview
@@ -35,7 +35,7 @@ Startup order in `InitDB` ([database.go](https://github.com/infiniflow/ragflow/b
| `v0.26.0` | Rebuild the tenant model tables from `tenant_llm` and normalize stored model ids | `modelMigrationBaseVersion` |
| `v1.0.0-rc1` | Seed factory-declared models, merge `model_type` into an integer bitmask, populate the `tenant_*_id` columns | `modelMigrationTargetVersion` |
| `v1.0.0-rc1.dev1` | Split conversation message and reference payloads into child tables | `conversationHistoryTargetVersion` |
-| `v1.0.0-rc2.dev1` | Delete the retired `raptor` and `graphrag` sections from `knowledgebase.parser_config` | `knowledgebaseParserConfigTargetVersion` |
+| `v1.0.0-rc2.dev1` | Convert Python ingestion settings in dataset/document `parser_config` onto pipeline components, then remove retired dataset `raptor` / `graphrag` sections | `ingestionParserConfigTargetVersion` / `knowledgebaseParserConfigTargetVersion` |
The two tenant model steps are cumulative: a database at `v0.26.0` still runs the `v1.0.0-rc1` step, and a database at or above `v1.0.0-rc1` runs neither.
@@ -126,22 +126,26 @@ On success the step writes `v1.0.0-rc1.dev1` into the version marker.
## v1.0.0-rc2.dev1
-Retires the `raptor` and `graphrag` sections of the dataset parser config. Both moved out of `knowledgebase.parser_config`, so the stored sections are dead data no code path reads any more.
+Converts saved Python v0.27.2 ingestion settings into the component-scoped configuration consumed by Go. After conversion, the existing dataset cleanup removes the retired `raptor` and `graphrag` sections.
### Data migration
-| Table | Change |
-|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `knowledgebase` | `parser_config` is parsed as a JSON object; the top level `raptor` and `graphrag` keys are deleted and the remaining object is written back to the same column. |
+| Table | Change |
+|-------|--------|
+| `knowledgebase`, `document` | Move supported top-level Parser, Chunker, Tokenizer and Extractor settings onto the corresponding pipeline component nodes. |
+| `knowledgebase` | The existing cleanup deletes top-level `raptor` and `graphrag` sections, preserving other settings. |
+
+The conversion covers parser engines and model references, PDF pages, MinerU method/language, TCADP options, spreadsheet column configuration, supported chunk sizes and delimiters, overlap, media context, parent/child settings, filename embedding weight, keywords, questions and metadata. Fields are converted only where the selected component supports them. Settings without a Go consumer are dropped with table/row/field warnings; tags and early Go node formats are left untouched.
Execution details:
-- Rows are read in id-ordered batches of 256, each batch in its own transaction, so an interrupted run resumes from the last committed batch.
-- Only rows whose config actually carries one of the keys are rewritten; a row without them is left byte-identical.
-- A `NULL`, empty or non-object payload is skipped rather than failing the step.
-- The step returns without touching the marker when the `knowledgebase` table does not exist.
+- Component conversion reads each table in id-ordered batches of 256, each batch in its own transaction. Changed rows are updated once by the conversion; unchanged rows retain their original bytes. The existing dataset cleanup performs its own subsequent scan.
+- Each row's `parser_id` determines the builtin component targets. Parser/Chunker/Tokenizer conversion leaves custom pipeline rows and unsupported parser IDs untouched. Extractor conversion can also target existing Extractor nodes in custom pipeline configs.
+- Existing component values win, except that dataset-level modular metadata transport remains authoritative under the current service contract.
+- SQL `NULL`, empty strings and JSON `null` are left untouched. Invalid JSON, unsupported metadata schemas and unsupported engine selections stop migration with table/row context. A failed batch rolls back; earlier committed batches can be rerun safely. See [Parser-config conversion failures](./upgrade_from_v1_to_later_releases.md#parser-config-conversion-failures) for recovery instructions.
+- The marker advances only after both table conversions and the existing dataset cleanup succeed. If neither table exists, the marker is left unchanged.
-On success the step writes `v1.0.0-rc2.dev1` into the version marker.
+On success the migration records `v1.0.0-rc2.dev1`. Databases already marked `v1.0.0-rc2.dev1` or later skip conversion. This includes databases marked by the earlier cleanup-only implementation: that marker does not prove their ingestion settings were converted.
---
diff --git a/docs/administrator/migration/upgrade_from_v1_to_later_releases.md b/docs/administrator/migration/upgrade_from_v1_to_later_releases.md
index fc5af1fe66..7ed02122b0 100644
--- a/docs/administrator/migration/upgrade_from_v1_to_later_releases.md
+++ b/docs/administrator/migration/upgrade_from_v1_to_later_releases.md
@@ -53,7 +53,15 @@ docker compose -p docker -f docker/docker-compose.yml logs -f ragflow-cpu
Keep the log view open until the migration finishes and the application starts. Press `Ctrl+C` to leave the log view; the containers continue running. Review warnings as well as errors before allowing users, ingestion workers, or synchronization jobs to write data.
-If the logs report a migration failure, stop the deployment, preserve the logs, and follow [Roll back after a failed migration](#roll-back-after-a-failed-migration).
+If the logs report a migration failure, stop the deployment and preserve the logs. For a parser-config conversion error, follow the instructions below. For other failures, follow [Roll back after a failed migration](#roll-back-after-a-failed-migration).
+
+#### Parser-config conversion failures
+
+The `v1.0.0-rc2.dev1` step converts saved Python v0.27.2 ingestion settings for Parser, Chunker, Tokenizer, and Extractor together before the existing dataset `raptor` and `graphrag` cleanup records the shared version. Invalid JSON, metadata schemas that Go cannot represent, and unsupported PDF engine or OCR model-reference choices stop `--migrate` and block application startup. The error identifies the table, row ID, and rejected setting.
+
+Keep all writers stopped. Back up the affected row's original `parser_config`, then correct the reported value in that row. For an unsupported engine or OCR model reference, explicitly choose and configure an engine that Go can run; deleting the setting would select the template's default engine instead. Start the target deployment again to rerun `--migrate`, and review its logs before resuming writes.
+
+Each failed batch rolls back and its migration version does not advance. Earlier batches may already be committed; rerunning the conversion after correcting the data is safe. Do not change the migration version manually. To return to the previous release instead, restore the complete pre-upgrade recovery point as described below.
### 4. Check the deployment status
diff --git a/internal/dao/database.go b/internal/dao/database.go
index f687e2f546..3adbd9ffdf 100644
--- a/internal/dao/database.go
+++ b/internal/dao/database.go
@@ -241,12 +241,10 @@ func InitDB(ctx context.Context, migrateDB bool) error {
if err = migrateConversationHistory(ctx, DB); err != nil {
return fmt.Errorf("failed to migrate conversation history: %w", err)
}
- // Drop the retired raptor and graphrag sections from the dataset parser
- // configs. It runs after the conversation split because the two steps
- // share one version marker: this step's version sorts above the split's
- // v1.0.0-rc1, so marking it earlier would skip a split that has not run.
- if err = migrateKnowledgebaseParserConfig(ctx, DB); err != nil {
- return fmt.Errorf("failed to migrate knowledgebase parser config: %w", err)
+ // Run ingestion conversion after the conversation split. It invokes the
+ // dataset cleanup last because both use the same version marker.
+ if err = migrateIngestionParserConfigs(ctx, DB); err != nil {
+ return fmt.Errorf("failed to migrate ingestion parser configs: %w", err)
}
if err = migrateTenantModelMaxTokens(ctx, DB); err != nil {
diff --git a/internal/dao/ingestion_parser_config_migration.go b/internal/dao/ingestion_parser_config_migration.go
new file mode 100644
index 0000000000..b74c90a76d
--- /dev/null
+++ b/internal/dao/ingestion_parser_config_migration.go
@@ -0,0 +1,1335 @@
+//
+// Copyright 2026 The InfiniFlow Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package dao
+
+import (
+ "context"
+ "fmt"
+ "maps"
+ "sort"
+ "strconv"
+ "strings"
+
+ "go.uber.org/zap"
+ "gorm.io/gorm"
+
+ "ragflow/internal/common"
+ "ragflow/internal/entity"
+ parserchunk "ragflow/internal/parser/chunk"
+ parserpkg "ragflow/internal/parser/parser"
+ "ragflow/internal/utility"
+)
+
+// Use the project version shared with the dataset parser config cleanup.
+const ingestionParserConfigTargetVersion = knowledgebaseParserConfigTargetVersion
+
+// Python v0.27.2 stored component settings at the parser_config root. Convert
+// them to Go's component-scoped shape before the API discards unscoped keys.
+// Target IDs are literal because importing the pipeline registry into DAO would
+// introduce a dependency cycle; runtime tests check them against the templates.
+type builtinIngestionTarget struct {
+ parserNode string
+ pdf bool
+ spreadsheet bool
+ // Overrides replace an entire family; carry template PDF values that differ
+ // from the Parser constructor's defaults.
+ pdfDefaults map[string]any
+ chunker string
+ chunkerNode string
+ tokenizerNode string
+ extractorNode string
+}
+
+func (t builtinIngestionTarget) hasFamily(family string) bool {
+ switch family {
+ case "pdf":
+ return t.pdf
+ case "spreadsheet":
+ return t.spreadsheet
+ default:
+ return false
+ }
+}
+
+// acceptsChunkParams reports whether the chunker accepts delimiter, overlap,
+// context and parent/child parameters.
+func (t builtinIngestionTarget) acceptsChunkParams() bool {
+ return t.chunker == "GeneralChunker" || t.chunker == "TokenChunker"
+}
+
+// Only PDF and spreadsheet settings are migrated from the Parser's flat keys.
+var ingestionTargets = func() map[string]builtinIngestionTarget {
+ // "naive" is the historical name of the general pipeline: the embedded
+ // general template declares parser_ids ["naive"] and the registry aliases
+ // "general" onto it.
+ general := builtinIngestionTarget{
+ parserNode: "Parser:HipSignsRhyme",
+ pdf: true,
+ spreadsheet: true,
+ chunker: "GeneralChunker",
+ chunkerNode: "GeneralChunker:SixApplesFall",
+ tokenizerNode: "Tokenizer:LegalReadersDecide",
+ extractorNode: "Extractor:AutoExtractDefault",
+ }
+ return map[string]builtinIngestionTarget{
+ "general": general,
+ "naive": general,
+ "qa": {
+ parserNode: "Parser:HipSignsRhyme",
+ pdf: true,
+ spreadsheet: true,
+ chunker: "QAChunker",
+ chunkerNode: "QAChunker:TidyCloudsThink",
+ tokenizerNode: "Tokenizer:ColdCloudsDream",
+ },
+ "manual": {
+ parserNode: "Parser:HipSignsRhyme",
+ pdf: true,
+ chunker: "ManualChunker",
+ chunkerNode: "ManualChunker:NineInsectsFind",
+ tokenizerNode: "Tokenizer:FunnyBalloonsGrin",
+ extractorNode: "Extractor:AutoExtractDefault",
+ },
+ "table": {
+ parserNode: "Parser:HipSignsRhyme",
+ spreadsheet: true,
+ chunker: "TableChunker",
+ chunkerNode: "TableChunker:FastFoxesJump",
+ tokenizerNode: "Tokenizer:DeepLakesShine",
+ },
+ "paper": {
+ parserNode: "Parser:HipSignsRhyme",
+ pdf: true,
+ pdfDefaults: map[string]any{"enable_multi_column": true},
+ chunker: "TitleChunker",
+ chunkerNode: "TitleChunker:SparklySchoolsTravel",
+ tokenizerNode: "Tokenizer:GreatCarsWash",
+ extractorNode: "Extractor:AutoExtractDefault",
+ },
+ "book": {
+ parserNode: "Parser:HipSignsRhyme",
+ pdf: true,
+ pdfDefaults: map[string]any{"remove_toc": true},
+ chunker: "TitleChunker",
+ chunkerNode: "TitleChunker:GrumpyGarlicsBake",
+ tokenizerNode: "Tokenizer:HotDonutsRing",
+ extractorNode: "Extractor:AutoExtractDefault",
+ },
+ "laws": {
+ parserNode: "Parser:HipSignsRhyme",
+ pdf: true,
+ pdfDefaults: map[string]any{"remove_toc": true},
+ chunker: "TitleChunker",
+ chunkerNode: "TitleChunker:SpicyKeysKick",
+ tokenizerNode: "Tokenizer:PublicJobsTake",
+ extractorNode: "Extractor:AutoExtractDefault",
+ },
+ "presentation": {
+ parserNode: "Parser:HipSignsRhyme",
+ pdf: true,
+ chunker: "PageChunker",
+ chunkerNode: "PageChunker:HappyHillsGlow",
+ tokenizerNode: "Tokenizer:TallTreesDance",
+ extractorNode: "Extractor:AutoExtractDefault",
+ },
+ "picture": {
+ parserNode: "Parser:ViewsCaptureLight",
+ chunker: "OneChunker",
+ chunkerNode: "OneChunker:BrightColorsGlow",
+ tokenizerNode: "Tokenizer:SharpLensFocus",
+ extractorNode: "Extractor:AutoExtractDefault",
+ },
+ "one": {
+ parserNode: "Parser:HipSignsRhyme",
+ pdf: true,
+ spreadsheet: true,
+ chunker: "OneChunker",
+ chunkerNode: "OneChunker:DryDrinksVisit",
+ tokenizerNode: "Tokenizer:FrankWeeksListen",
+ extractorNode: "Extractor:AutoExtractDefault",
+ },
+ "audio": {
+ parserNode: "Parser:SongsFillAir",
+ chunker: "OneChunker",
+ chunkerNode: "OneChunker:BlueSkiesLaugh",
+ tokenizerNode: "Tokenizer:KindEyesWatch",
+ extractorNode: "Extractor:AutoExtractDefault",
+ },
+ "email": {
+ parserNode: "Parser:BirdsFlutterHigh",
+ chunker: "TokenChunker",
+ chunkerNode: "TokenChunker:WarmBreadSmells",
+ tokenizerNode: "Tokenizer:NiceWordsSpoken",
+ extractorNode: "Extractor:AutoExtractDefault",
+ },
+ }
+}()
+
+// ingestionParserFlatKeys are the Python-era top level keys this step converts.
+var (
+ ingestionParserFlatKeys = []string{
+ "layout_recognize",
+ "mineru_parse_method",
+ "mineru_lang",
+ "pages",
+ "table_result_type",
+ "markdown_image_response_type",
+ "table_column_mode",
+ "table_column_roles",
+ "table_column_names",
+ }
+ ingestionChunkerFlatKeys = []string{
+ "chunk_token_num",
+ "delimiter",
+ "overlapped_percent",
+ "table_context_size",
+ "image_context_size",
+ "parent_child",
+ "enable_children",
+ "children_delimiter",
+ }
+ ingestionTokenizerFlatKeys = []string{"filename_embd_weight"}
+)
+
+// ingestionRetiredFlatKeys are Python settings that have no Go consumer at all.
+// They are deleted so the stored config stops advertising behaviour the Go
+// runtime does not implement; every drop is logged with table, row and field.
+var ingestionRetiredFlatKeys = []string{
+ // Spreadsheets are always rendered as segmented HTML; the flag only warns
+ // now (internal/parser/parser/office_table_render.go:deprecatedHTML4Excel).
+ "html4excel",
+ // Splits one parse into several ingestion jobs; it is not a page range and
+ // not a chunk size.
+ "task_page_size",
+ // Follows hyperlinks and indexes the linked content; unimplemented in Go.
+ "analyze_hyperlink",
+ "hyperlink_urls",
+ // Video parsing is rejected at dispatch
+ // (internal/ingestion/component/media_dispatch.go:maybeDispatchVideo).
+ "video_prompt",
+ // These MinerU toggles have no Go reader; method and language are migrated.
+ "mineru_formula_enable",
+ "mineru_table_enable",
+}
+
+// ingestionOwnedFlatKeys is every flat key this step consumes or retires. A row
+// carrying none of them is left completely untouched, which is what makes a
+// second run a no-op.
+func ingestionOwnedFlatKeys() []string {
+ owned := make([]string, 0, len(ingestionParserFlatKeys)+len(ingestionChunkerFlatKeys)+
+ len(ingestionTokenizerFlatKeys)+len(ingestionRetiredFlatKeys))
+ owned = append(owned, ingestionParserFlatKeys...)
+ owned = append(owned, ingestionChunkerFlatKeys...)
+ owned = append(owned, ingestionTokenizerFlatKeys...)
+ owned = append(owned, ingestionRetiredFlatKeys...)
+ return owned
+}
+
+// ingestionConfigMigrationRow is one dataset or document row.
+type ingestionConfigMigrationRow struct {
+ ID string `gorm:"column:id"`
+ ParserID string `gorm:"column:parser_id"`
+ PipelineID *string `gorm:"column:pipeline_id"`
+ ParserConfig *string `gorm:"column:parser_config"`
+}
+
+func migrateIngestionParserConfigs(ctx context.Context, db *gorm.DB) error {
+ version, err := GetDatabaseMigrationVersion(ctx, db)
+ if err != nil {
+ return err
+ }
+ if shouldSkipMigration(version, ingestionParserConfigTargetVersion) {
+ return nil
+ }
+ found := false
+ for _, target := range []struct {
+ table string
+ model any
+ }{
+ {"knowledgebase", &entity.Knowledgebase{}}, {"document", &entity.Document{}},
+ } {
+ if !db.WithContext(ctx).Migrator().HasTable(target.model) {
+ continue
+ }
+ found = true
+ count, err := migrateIngestionConfigRows(ctx, db, target.model, target.table)
+ if err != nil {
+ return fmt.Errorf("migrate %s parser configs: %w", target.table, err)
+ }
+ common.Info("Migrated Python parser configs", zap.String("table", target.table),
+ zap.String("version", ingestionParserConfigTargetVersion), zap.Int("rows", count))
+ }
+ if !found {
+ return nil
+ }
+ // Cleanup shares this version: run it only after both conversions succeed.
+ if err := migrateKnowledgebaseParserConfig(ctx, db); err != nil {
+ return err
+ }
+ return setDatabaseMigrationVersion(ctx, db, ingestionParserConfigTargetVersion)
+}
+
+func migrateIngestionConfigRows(ctx context.Context, db *gorm.DB, model any, table string) (int, error) {
+ count := 0
+ var rows []ingestionConfigMigrationRow
+ result := db.WithContext(ctx).Model(model).
+ Select("id", "parser_id", "pipeline_id", "parser_config").Order("id").
+ FindInBatches(&rows, parserConfigMigrationBatchSize, func(batch *gorm.DB, _ int) error {
+ return batch.Transaction(func(tx *gorm.DB) error {
+ for _, row := range rows {
+ config, ok := parseParserConfig(row.ParserConfig)
+ if !ok {
+ // An empty or absent payload carries nothing to migrate;
+ // a malformed one is rejected so the batch rolls back
+ // with the offending row named.
+ if row.ParserConfig == nil || strings.TrimSpace(*row.ParserConfig) == "" {
+ continue
+ }
+ return fmt.Errorf("row %s: invalid parser_config JSON", row.ID)
+ }
+ extractorChanged, err := migrateExtractorConfig(config, row.ParserID, row.PipelineID)
+ if err != nil {
+ return fmt.Errorf("row %s: %w", row.ID, err)
+ }
+ ingestionChanged, err := migrateParserChunkerTokenizerConfig(config, row.ParserID, row.PipelineID, table, row.ID)
+ if err != nil {
+ return fmt.Errorf("row %s: %w", row.ID, err)
+ }
+ if !extractorChanged && !ingestionChanged {
+ continue
+ }
+ if err := tx.Model(model).Where("id = ?", row.ID).
+ Update("parser_config", entity.JSONMap(config)).Error; err != nil {
+ return fmt.Errorf("row %s: update parser_config: %w", row.ID, err)
+ }
+ count++
+ }
+ return nil
+ })
+ })
+ return count, result.Error
+}
+
+// migrateParserChunkerTokenizerConfig rewrites one row's Python-era flat Parser / Chunker /
+// Tokenizer settings into the components of the pipeline that row runs. Params
+// the row already carries win; a setting with no Go consumer is deleted and
+// logged.
+func migrateParserChunkerTokenizerConfig(config map[string]any, parserID string, pipelineID *string, table, rowID string) (bool, error) {
+ // A custom canvas reads component-scoped params on the Python side too, and
+ // a canvas can hold several nodes of the same kind, so no ownership rule for
+ // a flat setting is safe.
+ if pipelineID != nil && strings.TrimSpace(*pipelineID) != "" {
+ return false, nil
+ }
+ target, ok := ingestionTargets[strings.ToLower(strings.TrimSpace(parserID))]
+ if !ok {
+ // resume / tag / knowledge_graph and unknown parser ids have no Go
+ // template, so their settings are preserved instead of being dropped.
+ return false, nil
+ }
+ owned := false
+ for _, key := range ingestionOwnedFlatKeys() {
+ if _, ok := config[key]; ok {
+ owned = true
+ break
+ }
+ }
+ if !owned {
+ return false, nil
+ }
+
+ row := ingestionConfigRow{table: table, rowID: rowID}
+ writes := nodeWrites{}
+ if err := convertIngestionParser(config, target, writes, row); err != nil {
+ return false, err
+ }
+ if err := convertIngestionChunker(config, target, writes, row); err != nil {
+ return false, err
+ }
+ if err := convertIngestionTokenizer(config, target, writes, row); err != nil {
+ return false, err
+ }
+ changed, err := applyNodeWrites(config, writes)
+ if err != nil {
+ return false, err
+ }
+ for _, key := range ingestionRetiredFlatKeys {
+ if _, exists := config[key]; exists {
+ row.retire(key, "this Python setting has no Go consumer")
+ }
+ }
+ for _, key := range ingestionOwnedFlatKeys() {
+ if _, exists := config[key]; exists {
+ delete(config, key)
+ changed = true
+ }
+ }
+ return changed, nil
+}
+
+// ingestionConfigRow carries the log context of the row being converted.
+type ingestionConfigRow struct {
+ table string
+ rowID string
+}
+
+// retire records that a Python setting with an actual value has no Go target.
+func (r ingestionConfigRow) retire(field, reason string) {
+ common.Warn("Ingestion parser config: dropping setting with no Go consumer",
+ zap.String("table", r.table),
+ zap.String("row_id", r.rowID),
+ zap.String("field", field),
+ zap.String("reason", reason))
+}
+
+// nodeWrites accumulates the component-scoped params the converters derive.
+type nodeWrites map[string]map[string]any
+
+func (w nodeWrites) params(node string) map[string]any {
+ if w[node] == nil {
+ w[node] = map[string]any{}
+ }
+ return w[node]
+}
+
+func (w nodeWrites) set(node, param string, value any) {
+ w.params(node)[param] = value
+}
+
+// setSetup writes a param inside a Parser file-family setup, the shape the
+// Parser component reads (schema.ParserSetup). An existing family map is merged
+// into, never replaced.
+func (w nodeWrites) setSetup(node, family, param string, value any) {
+ params := w.params(node)
+ setup, _ := params[family].(map[string]any)
+ if setup == nil {
+ setup = map[string]any{}
+ params[family] = setup
+ }
+ setup[param] = value
+}
+
+// applyNodeWrites merges the derived params into the stored config. A param the
+// row already carries always wins - including an explicit false/0/empty value -
+// and a node or family is created only when it has something to receive.
+func applyNodeWrites(config map[string]any, writes nodeWrites) (bool, error) {
+ nodes := make([]string, 0, len(writes))
+ for node := range writes {
+ nodes = append(nodes, node)
+ }
+ sort.Strings(nodes)
+
+ changed := false
+ for _, node := range nodes {
+ derived := writes[node]
+ if len(derived) == 0 {
+ continue
+ }
+ raw, exists := config[node]
+ if !exists {
+ params := make(map[string]any, len(derived))
+ for key, value := range derived {
+ params[key] = value
+ }
+ config[node] = params
+ changed = true
+ continue
+ }
+ params, ok := raw.(map[string]any)
+ if !ok {
+ return false, fmt.Errorf("%s is not an object", node)
+ }
+ for key, value := range derived {
+ existing, present := params[key]
+ if !present {
+ params[key] = value
+ changed = true
+ continue
+ }
+ // Merge missing fields within Parser families and parent_child groups.
+ derivedMap, derivedIsMap := value.(map[string]any)
+ existingMap, existingIsMap := existing.(map[string]any)
+ if !derivedIsMap || !existingIsMap {
+ continue
+ }
+ for subKey, subValue := range derivedMap {
+ if _, exists := existingMap[subKey]; exists {
+ continue
+ }
+ existingMap[subKey] = subValue
+ changed = true
+ }
+ }
+ }
+ return changed, nil
+}
+
+// convertIngestionParser moves the flat PDF-parser settings onto the row's
+// Parser node. An unsupported engine stops migration before the source is lost.
+func convertIngestionParser(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {
+ if raw, ok := config["layout_recognize"]; ok {
+ if err := convertIngestionLayoutRecognize(raw, config, target, writes, row); err != nil {
+ return err
+ }
+ }
+ for _, key := range []string{"mineru_parse_method", "mineru_lang"} {
+ if raw, exists := config[key]; exists && raw != nil {
+ value, ok := raw.(string)
+ if !ok {
+ return fmt.Errorf("%s must be a string, got %T", key, raw)
+ }
+ if target.hasFamily("pdf") {
+ writes.setSetup(target.parserNode, "pdf", key, value)
+ } else {
+ row.retire(key, "the targeted pipeline has no PDF file family")
+ }
+ }
+ }
+ if err := convertIngestionParserPages(config, target, writes, row); err != nil {
+ return err
+ }
+ if err := convertIngestionTCADPOptions(config, target, writes, row); err != nil {
+ return err
+ }
+ if err := convertIngestionTableColumns(config, target, writes, row); err != nil {
+ return err
+ }
+ seedCarriedParserFamilyDefaults(target, writes)
+ return nil
+}
+
+// Carry PDF template defaults whenever an override replaces that family.
+func seedCarriedParserFamilyDefaults(target builtinIngestionTarget, writes nodeWrites) {
+ pdf, ok := writes[target.parserNode]["pdf"].(map[string]any)
+ if !ok {
+ return
+ }
+ for key, value := range target.pdfDefaults {
+ if _, exists := pdf[key]; !exists {
+ pdf[key] = value
+ }
+ }
+}
+
+// Normalize Python engine spellings using the existing Go runtime vocabulary.
+func ingestionPDFParseMethod(raw string) (string, bool) {
+ method := strings.ToLower(strings.TrimSpace(raw))
+ if method == "tcadp" {
+ method = "tcadp parser"
+ }
+ if !parserpkg.IsPDFParseMethod(method) {
+ return "", false
+ }
+ switch method {
+ case "tcadp parser":
+ method = "tcadp"
+ case "plaintext", "plain text":
+ method = "plain_text"
+ }
+ return method, true
+}
+
+// Resolve the flat engine choice without overriding an existing selector.
+func convertIngestionLayoutRecognize(raw any, config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {
+ // An existing selector owns the engine choice as a whole: filling its
+ // sibling could change which dispatcher wins, even if the old value stays.
+ if node, ok := config[target.parserNode].(map[string]any); ok {
+ if setup, ok := node["pdf"].(map[string]any); ok {
+ _, hasMethod := setup["parse_method"]
+ _, hasModel := setup["layout_recognizer"]
+ if hasMethod || hasModel {
+ // A PDF selector does not own the spreadsheet family's engine.
+ if value, ok := raw.(string); ok && target.hasFamily("spreadsheet") {
+ if method, known := ingestionPDFParseMethod(value); known && method == "tcadp" {
+ writes.setSetup(target.parserNode, "spreadsheet", "parse_method", method)
+ return nil
+ }
+ }
+ return nil
+ }
+ }
+ }
+ switch value := raw.(type) {
+ case nil:
+ // A null engine means "unset": nothing to derive, nothing lost.
+ return nil
+ case bool:
+ // The dataset UI stored the DeepDOC toggle as a boolean.
+ method := "plain_text"
+ if value {
+ method = "deepdoc"
+ }
+ if !target.hasFamily("pdf") {
+ row.retire("layout_recognize", "the targeted pipeline has no PDF file family")
+ return nil
+ }
+ writes.setSetup(target.parserNode, "pdf", "parse_method", method)
+ return nil
+ case string:
+ trimmed := strings.TrimSpace(value)
+ if trimmed == "" {
+ // Python's ParserConfig declares layout_recognize str with
+ // min_length 1, so an empty string is malformed rather than unset.
+ return fmt.Errorf("layout_recognize must not be empty")
+ }
+ if ingestionLayoutModelReference(trimmed) {
+ if !target.hasFamily("pdf") {
+ row.retire("layout_recognize", "the targeted pipeline has no PDF file family")
+ return nil
+ }
+ // These OCR selectors cannot be resolved by the Go ingestion dispatcher.
+ // Preserve them by stopping rather than sending them to generic VLM chat.
+ switch ingestionModelReferenceEngine(trimmed) {
+ case "mistral ocr", "somark", "opendataloader", "paddleocr":
+ return fmt.Errorf("layout_recognize %q has no supported Go model-reference dispatch", trimmed)
+ }
+ // Keep supported OCR and VLM model references whole.
+ writes.setSetup(target.parserNode, "pdf", "layout_recognizer", trimmed)
+ return nil
+ }
+ method, known := ingestionPDFParseMethod(trimmed)
+ if !known {
+ return fmt.Errorf("layout_recognize %q is not a supported Go parse method", trimmed)
+ }
+ if target.pdf {
+ pdfMethod := method
+ // The existing PDF dispatcher recognizes the full TCADP spelling.
+ if method == "tcadp" {
+ pdfMethod = "tcadp parser"
+ }
+ writes.setSetup(target.parserNode, "pdf", "parse_method", pdfMethod)
+ }
+ if method == "tcadp" && target.hasFamily("spreadsheet") {
+ // Python applied one layout_recognizer to every file type, and
+ // Go's TCADP engine serves the spreadsheet family too.
+ writes.setSetup(target.parserNode, "spreadsheet", "parse_method", "tcadp")
+ }
+ if !target.hasFamily("pdf") && !(method == "tcadp" && target.hasFamily("spreadsheet")) {
+ row.retire("layout_recognize", "the targeted pipeline has no file family this engine serves")
+ }
+ return nil
+ default:
+ return fmt.Errorf("layout_recognize must be a string or a boolean, got %T", raw)
+ }
+}
+
+// convertIngestionParserPages moves the flat page range onto the PDF family. An
+// unparseable range fails the batch, exactly as the same invalid range would
+// fail a write through the API, rather than being silently dropped.
+func convertIngestionParserPages(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {
+ raw, ok := config["pages"]
+ if !ok {
+ return nil
+ }
+ if !target.hasFamily("pdf") {
+ row.retire("pages", "the targeted pipeline has no PDF file family")
+ return nil
+ }
+ pages, err := utility.NormalizePDFPages(raw)
+ if err != nil {
+ return fmt.Errorf("pages: %w", err)
+ }
+ if len(pages) == 0 {
+ // "no value" and an empty range list both mean "parse every page",
+ // which is what the template already does.
+ return nil
+ }
+ out := make([]any, 0, len(pages))
+ for _, span := range pages {
+ out = append(out, []any{span[0], span[1]})
+ }
+ writes.setSetup(target.parserNode, "pdf", "pages", out)
+ return nil
+}
+
+// convertIngestionTCADPOptions moves the TCADP response options onto every
+// family whose effective engine is TCADP. Both keys are read from the family
+// setup a parser configures itself with
+// (internal/parser/parser/{csv,xls,xlsx,ppt,pptx}_parser.go and
+// pdf_parser_common.go), so they follow the engine rather than the file type.
+func convertIngestionTCADPOptions(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {
+ keys := []string{"table_result_type", "markdown_image_response_type"}
+ present := make(map[string]string, len(keys))
+ for _, key := range keys {
+ raw, ok := config[key]
+ if !ok || raw == nil {
+ continue
+ }
+ text, ok := raw.(string)
+ if !ok {
+ return fmt.Errorf("%s must be a string, got %T", key, raw)
+ }
+ if text != "" {
+ present[key] = text
+ }
+ }
+ if len(present) == 0 {
+ return nil
+ }
+ families := make([]string, 0, 2)
+ for _, family := range []string{"pdf", "spreadsheet"} {
+ if !target.hasFamily(family) {
+ continue
+ }
+ if ingestionEffectiveFamilyEngine(config, writes, target.parserNode, family) == "tcadp" {
+ families = append(families, family)
+ }
+ }
+ if len(families) == 0 {
+ for key := range present {
+ row.retire(key, "no file family of the targeted pipeline parses with TCADP")
+ }
+ return nil
+ }
+ for key, text := range present {
+ for _, family := range families {
+ writes.setSetup(target.parserNode, family, key, text)
+ }
+ }
+ return nil
+}
+
+// ingestionEffectiveFamilyEngine resolves the engine a family will actually run
+// with, mirroring the runtime precedence: an explicit param on the row wins over
+// the value this migration derives.
+func ingestionEffectiveFamilyEngine(config map[string]any, writes nodeWrites, parserNode, family string) string {
+ if node, ok := config[parserNode].(map[string]any); ok {
+ if setup, ok := node[family].(map[string]any); ok {
+ if existing, ok := setup["parse_method"].(string); ok && existing != "" {
+ if canonical, known := ingestionPDFParseMethod(existing); known {
+ return canonical
+ }
+ return strings.ToLower(existing)
+ }
+ if model, ok := setup["layout_recognizer"].(string); ok && model != "" {
+ return strings.ToLower(model)
+ }
+ }
+ }
+ if setup, ok := writes[parserNode][family].(map[string]any); ok {
+ if derived, ok := setup["parse_method"].(string); ok && derived != "" {
+ if method, known := ingestionPDFParseMethod(derived); known {
+ return method
+ }
+ return strings.ToLower(derived)
+ }
+ }
+ return ""
+}
+
+// convertIngestionTableColumns moves the table column role settings onto the
+// spreadsheet family. The root-level keys must go: ingestion reads
+// parser_config's root-level table_column_* first
+// (internal/ingestion/task/indexdoc/process.go:resolveTableColumnConfig) and
+// would otherwise shadow anything written to the node.
+func convertIngestionTableColumns(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {
+ modeKey, rolesKey, namesKey := "table_column_mode", "table_column_roles", "table_column_names"
+ _, hasMode := config[modeKey]
+ _, hasRoles := config[rolesKey]
+ _, hasNames := config[namesKey]
+ if !hasMode && !hasRoles && !hasNames {
+ return nil
+ }
+ if !target.hasFamily("spreadsheet") {
+ row.retire(modeKey, "the targeted pipeline has no spreadsheet file family")
+ return nil
+ }
+ if raw, ok := config[modeKey]; ok && raw != nil {
+ mode, ok := raw.(string)
+ if !ok {
+ return fmt.Errorf("%s must be a string, got %T", modeKey, raw)
+ }
+ if mode != "" {
+ // Python types this as Literal["auto", "manual"] | None; anything
+ // else was rejected by the API, so a stored other value is garbage.
+ if mode != "auto" && mode != "manual" {
+ row.retire(modeKey, fmt.Sprintf("%q is not a mode the table parser understands", mode))
+ } else {
+ writes.setSetup(target.parserNode, "spreadsheet", "column_mode", mode)
+ }
+ }
+ }
+ if raw, ok := config[rolesKey]; ok && raw != nil {
+ roles, ok := raw.(map[string]any)
+ if !ok {
+ return fmt.Errorf("%s must be an object, got %T", rolesKey, raw)
+ }
+ normalized := make(map[string]any, len(roles))
+ for column, role := range roles {
+ text, _ := role.(string)
+ switch text {
+ case "vectorize":
+ // Python normalizes the retired role name at validation time
+ // (api/utils/validation_utils.py:legacy_vectorize_table_column_role).
+ normalized[column] = "indexing"
+ case "indexing", "metadata", "both":
+ normalized[column] = text
+ default:
+ row.retire(rolesKey, fmt.Sprintf("column %q has unsupported role %q", column, text))
+ }
+ }
+ if len(normalized) > 0 {
+ writes.setSetup(target.parserNode, "spreadsheet", "column_roles", normalized)
+ }
+ }
+ if raw, ok := config[namesKey]; ok && raw != nil {
+ names, ok := raw.([]any)
+ if !ok {
+ return fmt.Errorf("%s must be a list, got %T", namesKey, raw)
+ }
+ out := make([]any, 0, len(names))
+ for _, name := range names {
+ text, ok := name.(string)
+ if !ok || text == "" {
+ row.retire(namesKey, fmt.Sprintf("dropping non-string column name %v", name))
+ continue
+ }
+ out = append(out, text)
+ }
+ if len(out) > 0 {
+ writes.setSetup(target.parserNode, "spreadsheet", "column_names", out)
+ }
+ }
+ return nil
+}
+
+// ingestionLayoutModelReference reports whether a layout_recognize value names a
+// VLM model rather than a parse engine: Python's "<llm>@<instance>@<provider>"
+// composite (normalize_layout_recognizer) or a bare tenant model id.
+func ingestionLayoutModelReference(raw string) bool {
+ if strings.Contains(raw, "@") {
+ return true
+ }
+ compact := strings.ReplaceAll(raw, "-", "")
+ if len(compact) != 32 {
+ return false
+ }
+ for _, r := range compact {
+ if !((r >= '0' && r <= '9') || (r >= 'a' && r <= 'f') || (r >= 'A' && r <= 'F')) {
+ return false
+ }
+ }
+ return true
+}
+
+// ingestionModelReferenceEngine returns the engine segment of a composite model
+// reference ("name@instance@provider" -> "provider"), or "" for a bare id.
+func ingestionModelReferenceEngine(raw string) string {
+ idx := strings.LastIndex(raw, "@")
+ if idx < 0 {
+ return ""
+ }
+ return strings.ToLower(strings.TrimSpace(raw[idx+1:]))
+}
+
+// convertIngestionChunker moves the flat chunking settings onto the row's
+// chunker node. Which settings have a home depends on the chunker: only
+// GeneralChunker and TokenChunker read a token size, delimiter list, overlap and
+// context sizes.
+func convertIngestionChunker(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {
+ convertIngestionChunkTokenNum(config, target, writes, row)
+ if err := convertIngestionDelimiter(config, target, writes, row); err != nil {
+ return err
+ }
+ convertIngestionOverlappedPercent(config, target, writes, row)
+ convertIngestionContextSizes(config, target, writes, row)
+ return convertIngestionParentChild(config, target, writes, row)
+}
+
+// convertIngestionChunkTokenNum maps the flat token count onto the target
+// chunker's own param name. The meaning is not uniform across chunkers, so each
+// case states its own mapping or refuses.
+func convertIngestionChunkTokenNum(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) {
+ raw, ok := config["chunk_token_num"]
+ if !ok || raw == nil {
+ // Absent or null means "the default", which the template already
+ // supplies: nothing is lost by deleting the flat key.
+ return
+ }
+ size, valid := common.ToInt(raw)
+ if !valid {
+ row.retire("chunk_token_num", fmt.Sprintf("%v is not an integer", raw))
+ return
+ }
+ switch target.chunker {
+ case "GeneralChunker":
+ // 0 is legal here and means "no split window", the same thing Python
+ // set for spreadsheet and TCADP files at parse time.
+ if size < 0 {
+ row.retire("chunk_token_num", fmt.Sprintf("%d is negative", size))
+ return
+ }
+ writes.set(target.chunkerNode, "chunk_token_size", size)
+ case "TokenChunker":
+ // schema/chunker.go rejects a non-positive chunk_token_size.
+ if size <= 0 {
+ row.retire("chunk_token_num", fmt.Sprintf("%d is not a positive chunk size", size))
+ return
+ }
+ writes.set(target.chunkerNode, "chunk_token_size", size)
+ case "TitleChunker":
+ // Known semantic difference: Python used chunk_token_num as the merge
+ // target of rag/app/{book,laws,paper}.py while Go's chunk_token_cap is a
+ // hard ceiling on an emitted chunk. 0 disables the ceiling in Go, which
+ // is what Python's "no cap" guard did, so it carries over; a non-zero
+ // value must be inside the range the chunker accepts or the component
+ // would refuse to build.
+ if size != 0 && (size < 128 || size > 8000) {
+ row.retire("chunk_token_num", fmt.Sprintf("%d is outside the title chunker's accepted 128..8000 range", size))
+ return
+ }
+ writes.set(target.chunkerNode, "chunk_token_cap", size)
+ default:
+ row.retire("chunk_token_num", fmt.Sprintf("%s has no token size parameter", target.chunker))
+ }
+}
+
+// convertIngestionDelimiter maps the flat delimiter string onto the canonical
+// delimiter list, using the same parser the write path uses
+// (pipeline.normalizeGeneralComponentParams), so a migrated row and an
+// API-written row carry byte-identical lists.
+func convertIngestionDelimiter(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {
+ raw, ok := config["delimiter"]
+ if !ok || raw == nil {
+ return nil
+ }
+ if !target.acceptsChunkParams() {
+ row.retire("delimiter", fmt.Sprintf("%s splits on its own structure", target.chunker))
+ return nil
+ }
+ text, ok := raw.(string)
+ if !ok {
+ return fmt.Errorf("delimiter must be a string, got %T", raw)
+ }
+ if text == "" {
+ // Python declares min_length 1; an empty value has no split points.
+ return nil
+ }
+ delimiters := parserchunk.ParseDelimiterField(text)
+ if len(delimiters) == 0 {
+ return nil
+ }
+ out := make([]any, 0, len(delimiters))
+ for _, delimiter := range delimiters {
+ out = append(out, delimiter)
+ }
+ writes.set(target.chunkerNode, "delimiters", out)
+ return nil
+}
+
+// convertIngestionOverlappedPercent copies the overlap across unchanged. The
+// runtime normalizes it (schema.NormalizeOverlappedPercent turns a 0..1 fraction
+// into a whole percentage and clamps it), which is where the API leaves the value
+// too, so the stored number stays the user's own.
+func convertIngestionOverlappedPercent(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) {
+ raw, ok := config["overlapped_percent"]
+ if !ok || raw == nil {
+ return
+ }
+ if !target.acceptsChunkParams() {
+ row.retire("overlapped_percent", fmt.Sprintf("%s has no overlap parameter", target.chunker))
+ return
+ }
+ if _, ok := common.ToFloat(raw); !ok {
+ row.retire("overlapped_percent", fmt.Sprintf("%v is not a number", raw))
+ return
+ }
+ writes.set(target.chunkerNode, "overlapped_percent", raw)
+}
+
+// convertIngestionContextSizes moves the table and image context windows. Only
+// the general and token chunkers grew them in Go; the title, manual and one
+// chunkers Python used them with have no such parameter.
+func convertIngestionContextSizes(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) {
+ for _, key := range []string{"table_context_size", "image_context_size"} {
+ raw, ok := config[key]
+ if !ok || raw == nil {
+ continue
+ }
+ if !target.acceptsChunkParams() {
+ row.retire(key, fmt.Sprintf("%s has no context size parameter", target.chunker))
+ continue
+ }
+ size, valid := common.ToInt(raw)
+ if !valid || size < 0 {
+ row.retire(key, fmt.Sprintf("%v is not a non-negative integer", raw))
+ continue
+ }
+ writes.set(target.chunkerNode, key, size)
+ }
+}
+
+// convertIngestionParentChild folds the three shapes the parent/child setting was
+// stored in into the component-scoped group the chunker reads, and derives the
+// children_delimiters array from it. Both halves are needed: the group is what
+// the write path keeps, children_delimiters is what the chunker reads at run time
+// (pipeline.ApplyParentChildChunkerConfig derives it while writing, which never
+// runs for a row this migration rewrites).
+func convertIngestionParentChild(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {
+ groupRaw, hasGroup := config["parent_child"]
+ enabledRaw, hasEnabled := config["enable_children"]
+ delimiterRaw, hasDelimiter := config["children_delimiter"]
+ if !hasGroup && !hasEnabled && !hasDelimiter {
+ return nil
+ }
+ if !target.acceptsChunkParams() {
+ row.retire("parent_child", fmt.Sprintf("%s cannot split parent chunks into children", target.chunker))
+ return nil
+ }
+
+ useParentChild := false
+ useFromGroup := false
+ childrenDelimiter := ""
+ delimiterFromGroup := false
+ if hasGroup && groupRaw != nil {
+ group, ok := groupRaw.(map[string]any)
+ if !ok {
+ return fmt.Errorf("parent_child must be an object, got %T", groupRaw)
+ }
+ if raw, exists := group["use_parent_child"]; exists && raw != nil {
+ enabled, ok := raw.(bool)
+ if !ok {
+ return fmt.Errorf("parent_child.use_parent_child must be a boolean, got %T", raw)
+ }
+ useParentChild, useFromGroup = enabled, true
+ }
+ if text, ok := group["children_delimiter"].(string); ok {
+ childrenDelimiter, delimiterFromGroup = text, true
+ }
+ }
+ // The flat keys are the Python-era shape and rank below the group: on a row
+ // that carries both, the current shape would otherwise be overruled by a
+ // stale sibling.
+ if hasEnabled && enabledRaw != nil && !useFromGroup {
+ enabled, ok := enabledRaw.(bool)
+ if !ok {
+ return fmt.Errorf("enable_children must be a boolean, got %T", enabledRaw)
+ }
+ useParentChild = enabled
+ }
+ if hasDelimiter && delimiterRaw != nil && !delimiterFromGroup {
+ text, ok := delimiterRaw.(string)
+ if !ok {
+ return fmt.Errorf("children_delimiter must be a string, got %T", delimiterRaw)
+ }
+ childrenDelimiter = text
+ }
+ if !delimiterFromGroup && (!hasDelimiter || delimiterRaw == nil) {
+ childrenDelimiter = "\n"
+ }
+ if !useFromGroup && (!hasEnabled || enabledRaw == nil) && hasDelimiter {
+ useParentChild = childrenDelimiter != ""
+ }
+ childrenDelimiter = decodePythonChildDelimiter(childrenDelimiter)
+ group := map[string]any{"use_parent_child": useParentChild, "children_delimiter": childrenDelimiter}
+ if node, ok := config[target.chunkerNode].(map[string]any); ok {
+ if raw, present := node["parent_child"]; present {
+ existing, ok := raw.(map[string]any)
+ if !ok {
+ return fmt.Errorf("%s.parent_child is not an object", target.chunkerNode)
+ }
+ for key, value := range existing {
+ group[key] = value
+ }
+ }
+ }
+ useParentChild, _ = group["use_parent_child"].(bool)
+ childrenDelimiter, _ = group["children_delimiter"].(string)
+ writes.set(target.chunkerNode, "parent_child", group)
+ children := []any{}
+ if useParentChild {
+ for _, delimiter := range parserchunk.ParseDelimiterField(childrenDelimiter) {
+ children = append(children, delimiter)
+ }
+ }
+ writes.set(target.chunkerNode, "children_delimiters", children)
+ return nil
+}
+
+// Python decodes escaped control characters before splitting child chunks.
+// Unknown escapes retain their backslash, as unicode_escape does.
+func decodePythonChildDelimiter(text string) string {
+ var out strings.Builder
+ for len(text) > 0 {
+ if text[0] == '\\' {
+ char, _, tail, err := strconv.UnquoteChar(text, '"')
+ if err == nil {
+ out.WriteRune(char)
+ text = tail
+ continue
+ }
+ }
+ out.WriteByte(text[0])
+ text = text[1:]
+ }
+ return out.String()
+}
+
+// convertIngestionTokenizer moves the filename weight onto the row's tokenizer.
+// A null weight is left alone: Go's default (0.1) is Python's default too, and
+// the flat key is deleted either way.
+func convertIngestionTokenizer(config map[string]any, target builtinIngestionTarget, writes nodeWrites, row ingestionConfigRow) error {
+ raw, ok := config["filename_embd_weight"]
+ if !ok {
+ return nil
+ }
+ // search_method and fields belong to the flow Tokenizer component in Python
+ // too; the flat config never carried them, so they are neither migrated nor
+ // deleted here (the extractor migration leaves topn_tags/tag_kb_ids alone for
+ // the same reason).
+ if raw == nil {
+ return nil
+ }
+ weight, ok := common.ToFloat(raw)
+ if !ok {
+ return fmt.Errorf("filename_embd_weight must be a number, got %T", raw)
+ }
+ if _, isText := raw.(string); isText {
+ // A numeric string reached the field; store the number the runtime will
+ // actually use rather than the string it happens to parse.
+ writes.set(target.tokenizerNode, "filename_embd_weight", weight)
+ return nil
+ }
+ writes.set(target.tokenizerNode, "filename_embd_weight", raw)
+ return nil
+}
+
+var extractorFlatKeys = []string{"auto_keywords", "auto_questions", "enable_metadata", "metadata", "built_in_metadata"}
+
+// migrateExtractorConfig rewrites only Python's top-level extraction settings.
+// Existing node fields win over flat Python values. A modular metadata transport
+// object replaces node metadata, matching ApplyComponentScopedParserConfig.
+// Unsupported inputs return an error before the config is mutated.
+func migrateExtractorConfig(config map[string]any, parserID string, pipelineID *string) (bool, error) {
+ present := false
+ for _, key := range extractorFlatKeys {
+ if _, ok := config[key]; ok {
+ present = true
+ break
+ }
+ }
+ if !present {
+ return false, nil
+ }
+ var targets []map[string]any
+ var newNode map[string]any
+ var extractorNode string
+ if pipelineID == nil || strings.TrimSpace(*pipelineID) == "" {
+ target, ok := ingestionTargets[strings.ToLower(strings.TrimSpace(parserID))]
+ if !ok || target.extractorNode == "" {
+ return false, nil
+ }
+ extractorNode = target.extractorNode
+ if raw, exists := config[extractorNode]; exists {
+ node, ok := raw.(map[string]any)
+ if !ok {
+ return false, fmt.Errorf("%s is not an object", extractorNode)
+ }
+ targets = append(targets, node)
+ } else {
+ newNode = map[string]any{}
+ targets = append(targets, newNode)
+ }
+ } else {
+ for id, raw := range config {
+ id = strings.ToLower(id)
+ if strings.HasPrefix(id, "extractor:") || strings.HasPrefix(id, "extractor_") {
+ if node, ok := raw.(map[string]any); ok {
+ targets = append(targets, node)
+ }
+ }
+ }
+ if len(targets) == 0 {
+ return false, nil
+ }
+ }
+ groups, replaceMetadata, err := convertPythonExtractorGroups(config)
+ if err != nil {
+ return false, err
+ }
+ for _, node := range targets {
+ for name := range groups {
+ if raw, exists := node[name]; exists && !(name == "metadata" && replaceMetadata) {
+ if _, ok := raw.(map[string]any); !ok {
+ return false, fmt.Errorf("%s group is not an object", name)
+ }
+ }
+ }
+ }
+ for _, node := range targets {
+ for name, defaults := range groups {
+ group, ok := node[name].(map[string]any)
+ if !ok || (name == "metadata" && replaceMetadata) {
+ node[name] = maps.Clone(defaults)
+ continue
+ }
+ for key, value := range defaults {
+ if _, exists := group[key]; !exists {
+ group[key] = value
+ }
+ }
+ }
+ }
+ if newNode != nil {
+ config[extractorNode] = newNode
+ }
+ for _, key := range extractorFlatKeys {
+ delete(config, key)
+ }
+ return true, nil
+}
+
+func convertPythonExtractorGroups(config map[string]any) (map[string]map[string]any, bool, error) {
+ groups := map[string]map[string]any{}
+ for _, field := range []struct{ key, group string }{{"auto_keywords", "keywords"}, {"auto_questions", "questions"}} {
+ if raw, exists := config[field.key]; exists {
+ value, ok := raw.(float64)
+ if raw == nil {
+ value, ok = 0, true
+ }
+ if !ok || value < 0 || value != float64(int(value)) {
+ return nil, false, fmt.Errorf("%s must be a non-negative integer", field.key)
+ }
+ groups[field.group] = map[string]any{"top_n": int(value)}
+ }
+ }
+ source := config
+ replaceMetadata := false
+ _, hasToggle := config["enable_metadata"]
+ _, hasFields := config["metadata"]
+ _, hasBuiltIn := config["built_in_metadata"]
+ if obj, ok := config["metadata"].(map[string]any); ok {
+ if len(obj) == 0 && !hasToggle && !hasBuiltIn {
+ source, replaceMetadata = obj, true
+ }
+ for _, key := range []string{"enabled", "metadata", "built_in_metadata"} {
+ if _, exists := obj[key]; exists {
+ source = obj
+ replaceMetadata = true
+ break
+ }
+ }
+ }
+ if hasToggle || hasFields || hasBuiltIn {
+ flag := source["enable_metadata"]
+ if replaceMetadata {
+ flag = source["enabled"]
+ }
+ if flag != nil {
+ switch value := flag.(type) {
+ case bool:
+ case float64:
+ if value != 0 && value != 1 {
+ return nil, false, fmt.Errorf("metadata enabled must be bool or 0/1")
+ }
+ default:
+ return nil, false, fmt.Errorf("metadata enabled must be bool or 0/1")
+ }
+ }
+ fields, err := convertExtractorMetadataFields(source["metadata"])
+ if err != nil {
+ return nil, false, fmt.Errorf("metadata: %w", err)
+ }
+ builtIn, err := convertExtractorMetadataFields(source["built_in_metadata"])
+ if err != nil {
+ return nil, false, fmt.Errorf("built_in_metadata: %w", err)
+ }
+ groups["metadata"] = map[string]any{"enabled": common.ParserConfigBool(flag), "metadata": fields, "built_in_metadata": builtIn}
+ }
+ return groups, replaceMetadata, nil
+}
+
+// convertExtractorMetadataFields accepts field lists and flat JSON Schemas that
+// fit MetadataFieldDef. Constraints the target cannot express are rejected.
+func convertExtractorMetadataFields(raw any) ([]common.MetadataFieldDef, error) {
+ if raw == nil {
+ return []common.MetadataFieldDef{}, nil
+ }
+ if schema, ok := raw.(map[string]any); ok {
+ if len(schema) == 0 {
+ return []common.MetadataFieldDef{}, nil
+ }
+ for key := range schema {
+ if key != "type" && key != "properties" && key != "additionalProperties" {
+ return nil, fmt.Errorf("unsupported JSON Schema keyword %s", key)
+ }
+ }
+ properties, ok := schema["properties"].(map[string]any)
+ if !ok || schema["type"] != "object" || schema["additionalProperties"] != false {
+ return nil, fmt.Errorf("expected flat object schema with additionalProperties=false")
+ }
+ names := make([]string, 0, len(properties))
+ for name := range properties {
+ names = append(names, name)
+ }
+ sort.Strings(names)
+ fields := make([]any, 0, len(names))
+ for _, name := range names {
+ prop, ok := properties[name].(map[string]any)
+ if !ok {
+ return nil, fmt.Errorf("property %s is not an object", name)
+ }
+ for key := range prop {
+ if key != "type" && key != "description" && key != "enum" {
+ return nil, fmt.Errorf("unsupported property constraint %s.%s", name, key)
+ }
+ }
+ field := maps.Clone(prop)
+ field["key"] = name
+ fields = append(fields, field)
+ }
+ raw = fields
+ }
+ list, ok := raw.([]any)
+ if !ok {
+ return nil, fmt.Errorf("expected field list")
+ }
+ normalized := make([]any, 0, len(list))
+ for _, rawField := range list {
+ field, ok := rawField.(map[string]any)
+ if !ok {
+ return nil, fmt.Errorf("field is not an object")
+ }
+ key, ok := field["key"].(string)
+ if !ok || key == "" || strings.TrimSpace(key) != key {
+ return nil, fmt.Errorf("field requires a non-empty key without surrounding whitespace")
+ }
+ for _, name := range []string{"type", "description", "descriptions"} {
+ if value := field[name]; value != nil {
+ if _, ok := value.(string); !ok {
+ return nil, fmt.Errorf("%s.%s must be a string", key, name)
+ }
+ }
+ }
+ if value := field["enum"]; value != nil {
+ values, ok := value.([]any)
+ if !ok {
+ return nil, fmt.Errorf("%s.enum must be a string list", key)
+ }
+ for _, value := range values {
+ if _, ok := value.(string); !ok {
+ return nil, fmt.Errorf("%s.enum must be a string list", key)
+ }
+ }
+ }
+ item := maps.Clone(field)
+ if description, _ := item["description"].(string); description == "" {
+ item["description"] = item["descriptions"]
+ }
+ normalized = append(normalized, item)
+ }
+ return common.MetadataFieldDefsFromRaw(normalized), nil
+}
diff --git a/internal/dao/ingestion_parser_config_migration_test.go b/internal/dao/ingestion_parser_config_migration_test.go
new file mode 100644
index 0000000000..e0f99cffcc
--- /dev/null
+++ b/internal/dao/ingestion_parser_config_migration_test.go
@@ -0,0 +1,472 @@
+package dao
+
+import (
+ "encoding/json"
+ "fmt"
+ "reflect"
+ "strings"
+ "testing"
+
+ "github.com/glebarez/sqlite"
+ "go.uber.org/zap"
+ "go.uber.org/zap/zaptest/observer"
+ "gorm.io/gorm"
+ "ragflow/internal/common"
+ "ragflow/internal/entity"
+)
+
+// Test bridges keep migration internals out of the production API.
+var MigrateIngestionParserConfigsForTest = migrateIngestionParserConfigs
+var MigrateParserChunkerTokenizerConfigForTest = migrateParserChunkerTokenizerConfig
+var MigrateExtractorConfigForTest = migrateExtractorConfig
+
+func TestMigrateParserChunkerTokenizerConfig(t *testing.T) {
+ cases := []struct {
+ name, parser, input, want string
+ pipeline *string
+ }{
+ {
+ name: "engine and chunk settings",
+ input: `{"layout_recognize":true,"chunk_token_num":128,"delimiter":"\n","overlapped_percent":0.29,"table_context_size":100,"image_context_size":50,"filename_embd_weight":0.3}`,
+ want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"deepdoc"}},"GeneralChunker:SixApplesFall":{"chunk_token_size":128,"delimiters":["\n"],"overlapped_percent":0.29,"table_context_size":100,"image_context_size":50},"Tokenizer:LegalReadersDecide":{"filename_embd_weight":0.3}}`,
+ },
+ {name: "engine off", input: `{"layout_recognize":false}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"}}}`},
+ {name: "engine spellings", input: `{"layout_recognize":"plain text"}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"}}}`},
+ {name: "tcadp reaches both families", input: `{"layout_recognize":"TCADP Parser"}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"tcadp parser"},"spreadsheet":{"parse_method":"tcadp"}}}`},
+ {name: "tcadp on a spreadsheet-only pipeline", parser: "table", input: `{"layout_recognize":"TCADP Parser"}`, want: `{"Parser:HipSignsRhyme":{"spreadsheet":{"parse_method":"tcadp"}}}`},
+ {name: "tcadp result options follow the engine", parser: "table", input: `{"layout_recognize":"TCADP Parser","table_result_type":"2","markdown_image_response_type":"2"}`, want: `{"Parser:HipSignsRhyme":{"spreadsheet":{"markdown_image_response_type":"2","parse_method":"tcadp","table_result_type":"2"}}}`},
+ {name: "tcadp result options without the engine", parser: "table", input: `{"layout_recognize":"DeepDOC","table_result_type":"2"}`, want: `{}`},
+ {name: "model reference stays a layout recognizer", input: `{"layout_recognize":"bge-m3@mineru"}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"layout_recognizer":"bge-m3@mineru"}}}`},
+ {name: "existing engine wins over old model reference", input: `{"layout_recognize":"foo@MinerU","Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"}}}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"}}}`},
+ {name: "existing model reference wins over old engine", input: `{"layout_recognize":"DeepDOC","Parser:HipSignsRhyme":{"pdf":{"layout_recognizer":"foo@MinerU"}}}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"layout_recognizer":"foo@MinerU"}}}`},
+ {name: "existing engine supersedes unsupported old engine", input: `{"layout_recognize":"MagicOCR","Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"}}}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"}}}`},
+ {name: "PDF selector does not mask spreadsheet engine", input: `{"layout_recognize":"TCADP Parser","Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"}}}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text"},"spreadsheet":{"parse_method":"tcadp"}}}`},
+ {name: "MinerU options reach the PDF setup", input: `{"layout_recognize":"MinerU","mineru_parse_method":"ocr","mineru_lang":"English"}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"mineru","mineru_parse_method":"ocr","mineru_lang":"English"}}}`},
+ {name: "existing MinerU options win", input: `{"mineru_lang":"English","Parser:HipSignsRhyme":{"pdf":{"parse_method":"mineru","mineru_lang":"Chinese"}}}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"mineru","mineru_lang":"Chinese"}}}`},
+ {name: "picture has no engine path", parser: "picture", input: `{"layout_recognize":"PaddleOCR"}`, want: `{}`},
+ {name: "audio has no engine path", parser: "audio", input: `{"layout_recognize":true}`, want: `{}`},
+ {name: "pages are normalized", input: `{"pages":[[3,5],[1,2],[6,7]]}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"pages":[[1,7]]}}}`},
+ {name: "pages on a pdf-less pipeline", parser: "table", input: `{"pages":[[1,5]]}`, want: `{}`},
+ {
+ name: "table columns move to the spreadsheet family",
+ parser: "table",
+ input: `{"table_column_mode":"manual","table_column_roles":{"a":"vectorize","b":"metadata","c":"nonsense"},"table_column_names":["a","b",7]}`,
+ want: `{"Parser:HipSignsRhyme":{"spreadsheet":{"column_mode":"manual","column_roles":{"a":"indexing","b":"metadata"},"column_names":["a","b"]}}}`,
+ },
+ {name: "table columns need a spreadsheet family", parser: "paper", input: `{"table_column_mode":"manual"}`, want: `{}`},
+ {name: "zero chunk size on the general chunker", input: `{"chunk_token_num":0}`, want: `{"GeneralChunker:SixApplesFall":{"chunk_token_size":0}}`},
+ {name: "zero chunk size on the token chunker", parser: "email", input: `{"chunk_token_num":0}`, want: `{}`},
+ {name: "chunk size reached the email chunker", parser: "email", input: `{"chunk_token_num":256}`, want: `{"TokenChunker:WarmBreadSmells":{"chunk_token_size":256}}`},
+ {name: "numeric string chunk size", input: `{"chunk_token_num":"256"}`, want: `{"GeneralChunker:SixApplesFall":{"chunk_token_size":256}}`},
+ {name: "unusable chunk size is dropped", input: `{"chunk_token_num":"many"}`, want: `{}`},
+ {name: "title chunker cap", parser: "book", input: `{"chunk_token_num":256}`, want: `{"TitleChunker:GrumpyGarlicsBake":{"chunk_token_cap":256}}`},
+ {name: "title chunker cap out of range", parser: "book", input: `{"chunk_token_num":64}`, want: `{}`},
+ {name: "manual chunker has no cap", parser: "manual", input: `{"chunk_token_num":512}`, want: `{}`},
+ {name: "delimiter with backticks and unicode", input: "{\"delimiter\":\"`--`。?\"}", want: `{"GeneralChunker:SixApplesFall":{"delimiters":["` + "`--`" + `","。","?"]}}`},
+ {name: "delimiter on a title chunker", parser: "book", input: `{"delimiter":"\n"}`, want: `{}`},
+ {name: "context sizes need the general or token chunker", parser: "book", input: `{"table_context_size":10,"image_context_size":20}`, want: `{}`},
+ {name: "overlap on a title chunker", parser: "book", input: `{"overlapped_percent":0.29}`, want: `{}`},
+ {name: "parent child group", input: `{"parent_child":{"use_parent_child":true,"children_delimiter":"。"}}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"。"},"children_delimiters":["。"]}}`},
+ {name: "legacy parent child flags", input: `{"enable_children":true,"children_delimiter":"。;"}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"。;"},"children_delimiters":["。",";"]}}`},
+ {name: "parent child switched off", input: `{"enable_children":false}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":false,"children_delimiter":"\n"},"children_delimiters":[]}}`},
+ {name: "group shape wins over the legacy flag", input: `{"parent_child":{"use_parent_child":false,"children_delimiter":"|"},"enable_children":true,"children_delimiter":"。"}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":false,"children_delimiter":"|"},"children_delimiters":[]}}`},
+ {name: "parent child delimiters default to a newline", input: `{"parent_child":{"use_parent_child":true}}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"\n"},"children_delimiters":["\n"]}}`},
+ {name: "parent child on a QA chunker", parser: "qa", input: `{"enable_children":true}`, want: `{}`},
+ {name: "effective Python child delimiter", input: `{"children_delimiter":"。"}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"。"},"children_delimiters":["。"]}}`},
+ {name: "escaped Python child delimiter", input: `{"enable_children":true,"children_delimiter":"\\n"}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"\n"},"children_delimiters":["\n"]}}`},
+ {name: "explicit empty child delimiter stays disabled", input: `{"parent_child":{"use_parent_child":true,"children_delimiter":""}}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":""},"children_delimiters":[]}}`},
+ {name: "existing disabled child group wins", input: `{"enable_children":true,"children_delimiter":"。","GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":false,"children_delimiter":"|"}}}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":false,"children_delimiter":"|"},"children_delimiters":[]}}`},
+ {name: "existing enabled child group wins", input: `{"enable_children":false,"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"|"}}}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"|"},"children_delimiters":["|"]}}`},
+ {name: "explicit runtime child delimiters win", input: `{"enable_children":true,"children_delimiter":"。","GeneralChunker:SixApplesFall":{"children_delimiters":["|"]}}`, want: `{"GeneralChunker:SixApplesFall":{"parent_child":{"use_parent_child":true,"children_delimiter":"。"},"children_delimiters":["|"]}}`},
+ {name: "filename weight", parser: "picture", input: `{"filename_embd_weight":0}`, want: `{"Tokenizer:SharpLensFocus":{"filename_embd_weight":0}}`},
+ {name: "null filename weight", parser: "picture", input: `{"filename_embd_weight":null}`, want: `{}`},
+ {name: "null engine", input: `{"layout_recognize":null}`, want: `{}`},
+ {name: "naive alias", parser: "naive", input: `{"layout_recognize":true}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"deepdoc"}}}`},
+ {name: "presentation pdf family", parser: "presentation", input: `{"layout_recognize":true}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"deepdoc"}}}`},
+ {name: "book carries its remove_toc", parser: "book", input: `{"layout_recognize":true}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"deepdoc","remove_toc":true}}}`},
+ {name: "book carries remove_toc alongside pages", parser: "book", input: `{"pages":[[2,4]]}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"pages":[[2,4]],"remove_toc":true}}}`},
+ {name: "laws carries its remove_toc", parser: "laws", input: `{"layout_recognize":"plaintext"}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text","remove_toc":true}}}`},
+ {name: "paper carries multi column", parser: "paper", input: `{"layout_recognize":"plaintext"}`, want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"plain_text","enable_multi_column":true}}}`},
+ {
+ name: "existing node params win",
+ input: `{"layout_recognize":true,"chunk_token_num":4096,"filename_embd_weight":0.9,"Parser:HipSignsRhyme":{"pdf":{"parse_method":"somark"}},"GeneralChunker:SixApplesFall":{"chunk_token_size":512,"delimiters":["\n"]},"Tokenizer:LegalReadersDecide":{"filename_embd_weight":0.1}}`,
+ want: `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"somark"}},"GeneralChunker:SixApplesFall":{"chunk_token_size":512,"delimiters":["\n"]},"Tokenizer:LegalReadersDecide":{"filename_embd_weight":0.1}}`,
+ },
+ {
+ name: "an existing family map is merged, not replaced",
+ parser: "book",
+ input: `{"layout_recognize":true,"Parser:HipSignsRhyme":{"pdf":{"preprocess":["main_content"],"remove_toc":false}}}`,
+ want: `{"Parser:HipSignsRhyme":{"pdf":{"preprocess":["main_content"],"remove_toc":false,"parse_method":"deepdoc"}}}`,
+ },
+ {name: "retired settings are deleted", input: `{"html4excel":true,"task_page_size":256,"analyze_hyperlink":true,"hyperlink_urls":true,"video_prompt":"x","mineru_formula_enable":true}`, want: `{}`},
+ {name: "no owned keys", input: `{"chunk_method":"naive","topn_tags":5}`, want: `{"chunk_method":"naive","topn_tags":5}`},
+ {name: "empty config", input: `{}`, want: `{}`},
+ {name: "canvas rows are skipped", pipeline: ptrString("canvas-1"), input: `{"layout_recognize":true,"chunk_token_num":128}`, want: `{"layout_recognize":true,"chunk_token_num":128}`},
+ {name: "unknown parser ids are preserved", parser: "knowledge_graph", input: `{"layout_recognize":true,"chunk_token_num":128}`, want: `{"layout_recognize":true,"chunk_token_num":128}`},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ config, want := decodeIngestionConfig(t, tc.input), decodeIngestionConfig(t, tc.want)
+ parser := tc.parser
+ if parser == "" {
+ parser = "general"
+ }
+ if _, err := migrateParserChunkerTokenizerConfig(config, parser, tc.pipeline, "knowledgebase", "row-1"); err != nil {
+ t.Fatal(err)
+ }
+ if got := decodeIngestionConfig(t, encodeIngestionConfig(t, config)); !reflect.DeepEqual(got, want) {
+ t.Fatalf("got %s; want %s", encodeIngestionConfig(t, config), tc.want)
+ }
+ snapshot := encodeIngestionConfig(t, config)
+ if changed, err := migrateParserChunkerTokenizerConfig(config, parser, tc.pipeline, "knowledgebase", "row-1"); err != nil || changed {
+ t.Fatalf("second run changed=%v, err=%v", changed, err)
+ }
+ if encodeIngestionConfig(t, config) != snapshot {
+ t.Fatal("second run changed config")
+ }
+ })
+ }
+}
+
+func TestMigrateParserChunkerTokenizerConfigRejectsMalformedValues(t *testing.T) {
+ for _, raw := range []string{
+ `{"layout_recognize":"MagicOCR"}`,
+ `{"layout_recognize":"foo@mistral ocr"}`,
+ `{"layout_recognize":"model@instance@somark"}`,
+ `{"layout_recognize":"model@instance@opendataloader"}`,
+ `{"layout_recognize":"model@instance@paddleocr"}`,
+ `{"pages":"1-5"}`,
+ `{"pages":[[0,5]]}`,
+ `{"pages":[[5,1]]}`,
+ `{"layout_recognize":5}`,
+ `{"delimiter":5}`,
+ `{"parent_child":"yes"}`,
+ `{"parent_child":{"use_parent_child":"true"}}`,
+ `{"enable_children":"true"}`,
+ `{"children_delimiter":5}`,
+ `{"table_column_mode":5}`,
+ `{"table_column_roles":"a"}`,
+ `{"table_column_names":"a"}`,
+ `{"filename_embd_weight":"heavy"}`,
+ `{"table_result_type":2}`,
+ `{"layout_recognize":""}`,
+ } {
+ t.Run(raw, func(t *testing.T) {
+ config := decodeIngestionConfig(t, raw)
+ before := encodeIngestionConfig(t, config)
+ if _, err := migrateParserChunkerTokenizerConfig(config, "general", nil, "knowledgebase", "row-1"); err == nil {
+ t.Fatalf("malformed value was accepted: %s", before)
+ }
+ if encodeIngestionConfig(t, config) != before {
+ t.Fatal("a rejected row was modified")
+ }
+ })
+ }
+}
+
+func TestMigrateParserChunkerTokenizerConfigLogsRetiredFields(t *testing.T) {
+ core, logs := observer.New(zap.WarnLevel)
+ previous := common.Logger
+ common.Logger = zap.New(core)
+ t.Cleanup(func() { common.Logger = previous })
+ config := decodeIngestionConfig(t, `{"html4excel":true,"task_page_size":12,"mineru_formula_enable":false}`)
+ if _, err := migrateParserChunkerTokenizerConfig(config, "general", nil, "document", "doc-1"); err != nil {
+ t.Fatal(err)
+ }
+ for _, field := range []string{"html4excel", "task_page_size", "mineru_formula_enable"} {
+ entries := logs.FilterField(zap.String("field", field)).All()
+ if len(entries) != 1 || entries[0].ContextMap()["table"] != "document" || entries[0].ContextMap()["row_id"] != "doc-1" {
+ t.Errorf("missing row-specific warning for %s: %v", field, entries)
+ }
+ }
+}
+
+func TestMigrateIngestionParserConfigsBatches(t *testing.T) {
+ db := openIngestionMigrationDB(t)
+ for i := 0; i < 257; i++ {
+ for _, table := range []string{"knowledgebase", "document"} {
+ insertIngestionConfig(t, db, table, fmt.Sprintf("%032d", i), "general", python027IngestionConfig)
+ }
+ }
+ if err := migrateIngestionParserConfigs(t.Context(), db); err != nil {
+ t.Fatal(err)
+ }
+ for _, table := range []string{"knowledgebase", "document"} {
+ var rows []ingestionConfigMigrationRow
+ if err := db.Table(table).Find(&rows).Error; err != nil {
+ t.Fatal(err)
+ }
+ if len(rows) != 257 {
+ t.Fatalf("%s rows=%d", table, len(rows))
+ }
+ for _, row := range rows {
+ config := decodeIngestionConfig(t, *row.ParserConfig)
+ extractor, _ := config[ingestionTargets["general"].extractorNode].(map[string]any)
+ keywords, _ := extractor["keywords"].(map[string]any)
+ if keywords["top_n"] != float64(8) {
+ t.Fatalf("%s.%s: extractor settings not migrated: %v", table, row.ID, config)
+ }
+ parserNode, ok := config["Parser:HipSignsRhyme"].(map[string]any)
+ if !ok {
+ t.Fatalf("%s.%s: parser node missing: %v", table, row.ID, config)
+ }
+ pdf, _ := parserNode["pdf"].(map[string]any)
+ if pdf["parse_method"] != "plain_text" {
+ t.Fatalf("%s.%s: pdf=%v", table, row.ID, pdf)
+ }
+ chunker, _ := config["GeneralChunker:SixApplesFall"].(map[string]any)
+ if chunker["chunk_token_size"] != float64(256) {
+ t.Fatalf("%s.%s: chunker=%v", table, row.ID, chunker)
+ }
+ if _, leftovers := config["chunk_token_num"]; leftovers {
+ t.Fatalf("%s.%s: flat key survived: %v", table, row.ID, config)
+ }
+ }
+ }
+ if got := databaseVersion(t, db); got != ingestionParserConfigTargetVersion {
+ t.Fatalf("version=%s", got)
+ }
+}
+
+func TestMigrateIngestionParserConfigsRejectsInvalidRows(t *testing.T) {
+ for _, raw := range []string{
+ `{"pages":[[9,2]]}`,
+ `not-json`,
+ `{"auto_keywords":"8","auto_questions":2}`,
+ `{"enable_metadata":true,"metadata":[{"name":"author"}]}`,
+ `{"enable_metadata":true,"metadata":{"type":"object","required":["author"],"properties":{"author":{"type":"object","properties":{"name":{"type":"string"}}}}}}`,
+ } {
+ t.Run(raw, func(t *testing.T) {
+ db := openIngestionMigrationDB(t)
+ insertIngestionConfig(t, db, "knowledgebase", "a", "general", python027IngestionConfig)
+ insertIngestionConfig(t, db, "knowledgebase", "b", "general", raw)
+ if err := migrateIngestionParserConfigs(t.Context(), db); err == nil {
+ t.Fatal("invalid input was marked as migrated")
+ }
+ var rows []ingestionConfigMigrationRow
+ if err := db.Table("knowledgebase").Order("id").Find(&rows).Error; err != nil {
+ t.Fatal(err)
+ }
+ if *rows[0].ParserConfig != python027IngestionConfig || *rows[1].ParserConfig != raw {
+ t.Fatal("failed batch modified source configs")
+ }
+ if databaseVersion(t, db) != "" {
+ t.Fatal("failed migration advanced version")
+ }
+ })
+ }
+}
+
+func TestMigrateIngestionParserConfigsVersionAndEmptyRows(t *testing.T) {
+ for _, tc := range []struct {
+ version string
+ skip bool
+ }{
+ {"", false}, {"v0.27.2", false}, {"v1.0.0-rc1", false},
+ {"v1.0.0-rc2.dev2", true},
+ {ingestionParserConfigTargetVersion, true}, {"v1.0.0", true},
+ } {
+ t.Run(tc.version, func(t *testing.T) {
+ db := openIngestionMigrationDB(t)
+ if tc.version != "" {
+ writeDatabaseVersion(t, db, tc.version)
+ }
+ untouched := []string{``, `null`, `{}`, `{"Parser:HipSignsRhyme":{"pdf":{"parse_method":"somark"}}}`, `{"Extractor:AutoExtractDefault":{"metadata":{"enabled":false}}}`}
+ for i, raw := range untouched {
+ insertIngestionConfig(t, db, "knowledgebase", fmt.Sprint(i), "general", raw)
+ }
+ insertIngestionConfig(t, db, "knowledgebase", "old", "general", python027IngestionConfig)
+ if err := migrateIngestionParserConfigs(t.Context(), db); err != nil {
+ t.Fatal(err)
+ }
+ var rows []ingestionConfigMigrationRow
+ if err := db.Table("knowledgebase").Order("id").Find(&rows).Error; err != nil {
+ t.Fatal(err)
+ }
+ for i, raw := range untouched {
+ if *rows[i].ParserConfig != raw {
+ t.Fatalf("untouched row %d rewritten: %s", i, *rows[i].ParserConfig)
+ }
+ }
+ unchanged := *rows[len(untouched)].ParserConfig == python027IngestionConfig
+ if unchanged != tc.skip {
+ t.Fatalf("version gate: skipped=%v; want %v", unchanged, tc.skip)
+ }
+ wantVersion := ingestionParserConfigTargetVersion
+ if tc.skip {
+ wantVersion = tc.version
+ }
+ if databaseVersion(t, db) != wantVersion {
+ t.Fatal("unexpected version")
+ }
+ })
+ }
+}
+
+// python027IngestionConfig is a dataset parser_config as Python v0.27.2 stored
+// it: every value differs from the Go default so a lost field is visible.
+const python027IngestionConfig = `{"auto_keywords":8,"layout_recognize":"plain text","chunk_token_num":256,"delimiter":"\n","overlapped_percent":0.29,"filename_embd_weight":0.2,"pages":[[1,3]],"table_context_size":10,"image_context_size":20,"html4excel":false}`
+
+func openIngestionMigrationDB(t *testing.T) *gorm.DB {
+ t.Helper()
+ db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ sqlDB, err := db.DB()
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = sqlDB.Close() })
+ if err := db.AutoMigrate(&entity.Knowledgebase{}, &entity.Document{}, &entity.SystemSettings{}); err != nil {
+ t.Fatal(err)
+ }
+ return db
+}
+
+func insertIngestionConfig(t *testing.T, db *gorm.DB, table, id, parserID, raw string) {
+ t.Helper()
+ values := map[string]any{"id": id, "parser_id": parserID, "parser_config": raw, "created_by": "user"}
+ if table == "knowledgebase" {
+ values["tenant_id"] = "tenant"
+ values["name"] = "dataset"
+ values["embd_id"] = "embedding"
+ } else {
+ values["kb_id"] = "dataset"
+ values["source_type"] = "local"
+ values["type"] = "doc"
+ values["suffix"] = "pdf"
+ }
+ if err := db.Table(table).Create(values).Error; err != nil {
+ t.Fatal(err)
+ }
+}
+
+func decodeIngestionConfig(t *testing.T, raw string) map[string]any {
+ t.Helper()
+ var config map[string]any
+ if err := json.Unmarshal([]byte(raw), &config); err != nil {
+ t.Fatal(err)
+ }
+ return config
+}
+
+func encodeIngestionConfig(t *testing.T, config map[string]any) string {
+ t.Helper()
+ raw, err := json.Marshal(config)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return string(raw)
+}
+
+const python027ParserConfig = `{"auto_keywords":8,"auto_questions":3,"enable_metadata":true,"metadata":[{"key":"author","description":"who wrote it"}],"built_in_metadata":[{"key":"update_time"}],"topn_tags":5,"tag_kb_ids":["tag-kb"]}`
+
+func TestMigrateExtractorConfig(t *testing.T) {
+ cases := []struct {
+ name, parser, input, want string
+ pipeline *string
+ }{
+ {name: "Python fields", parser: "naive", input: python027ParserConfig, want: `{"Extractor:AutoExtractDefault":{"keywords":{"top_n":8},"questions":{"top_n":3},"metadata":{"enabled":true,"metadata":[{"key":"author","description":"who wrote it"}],"built_in_metadata":[{"key":"update_time"}]}},"topn_tags":5,"tag_kb_ids":["tag-kb"]}`},
+ {name: "missing metadata toggle stays disabled", input: `{"metadata":[{"key":"author"}]}`, want: `{"Extractor:AutoExtractDefault":{"metadata":{"enabled":false,"metadata":[{"key":"author"}],"built_in_metadata":[]}}}`},
+ {name: "explicit zero and false", input: `{"auto_keywords":0,"enable_metadata":false,"metadata":[]}`, want: `{"Extractor:AutoExtractDefault":{"keywords":{"top_n":0},"metadata":{"enabled":false,"metadata":[],"built_in_metadata":[]}}}`},
+ {name: "existing node values win over Python fields", input: `{"auto_keywords":8,"enable_metadata":true,"metadata":[{"key":"author"}],"Extractor:AutoExtractDefault":{"keywords":{"top_n":2,"enabled":false},"metadata":{"enabled":false,"metadata":[],"built_in_metadata":[]}}}`, want: `{"Extractor:AutoExtractDefault":{"keywords":{"top_n":2,"enabled":false},"metadata":{"enabled":false,"metadata":[],"built_in_metadata":[]}}}`},
+ {name: "modular transport is authoritative", input: `{"metadata":{"enabled":true,"metadata":[{"key":"author"}],"built_in_metadata":[]},"Extractor:AutoExtractDefault":{"metadata":{"enabled":false}}}`, want: `{"Extractor:AutoExtractDefault":{"metadata":{"enabled":true,"metadata":[{"key":"author"}],"built_in_metadata":[]}}}`},
+ {name: "empty modular transport disables metadata", input: `{"metadata":{},"Extractor:AutoExtractDefault":{"metadata":{"enabled":true}}}`, want: `{"Extractor:AutoExtractDefault":{"metadata":{"enabled":false,"metadata":[],"built_in_metadata":[]}}}`},
+ {name: "flat schema", input: `{"enable_metadata":true,"metadata":{"type":"object","properties":{"author":{"type":"string","description":"who","enum":["A"]}},"additionalProperties":false}}`, want: `{"Extractor:AutoExtractDefault":{"metadata":{"enabled":true,"metadata":[{"key":"author","type":"string","description":"who","enum":["A"]}],"built_in_metadata":[]}}}`},
+ {name: "description alias", input: `{"enable_metadata":true,"metadata":[{"key":"author","descriptions":"who"}]}`, want: `{"Extractor:AutoExtractDefault":{"metadata":{"enabled":true,"metadata":[{"key":"author","description":"who"}],"built_in_metadata":[]}}}`},
+ {name: "tag settings untouched", input: `{"topn_tags":5,"tag_kb_ids":["tag-kb"],"tags":{"tag_file_id":"file"}}`, want: `{"topn_tags":5,"tag_kb_ids":["tag-kb"],"tags":{"tag_file_id":"file"}}`},
+ {name: "early Go nodes untouched", input: `{"Extractor:AutoExtractDefault":{"auto_keywords":8,"keywords_sys_prompt":"saved","enable_summary":true,"sys_prompt":"saved summary","auto_tags":5,"tag_file_id":"file","metadata_config":{"enabled":true}}}`, want: `{"Extractor:AutoExtractDefault":{"auto_keywords":8,"keywords_sys_prompt":"saved","enable_summary":true,"sys_prompt":"saved summary","auto_tags":5,"tag_file_id":"file","metadata_config":{"enabled":true}}}`},
+ {name: "custom node", pipeline: ptrString("canvas-1"), input: `{"auto_questions":2,"Extractor:Custom":{"summary":{"enabled":true}}}`, want: `{"Extractor:Custom":{"questions":{"top_n":2},"summary":{"enabled":true}}}`},
+ {name: "QA has no target", parser: "qa", input: python027ParserConfig, want: python027ParserConfig},
+ {name: "table has no target", parser: "table", input: python027ParserConfig, want: python027ParserConfig},
+ {name: "QA ignores stale extractor", parser: "qa", input: `{"auto_keywords":8,"Extractor:Old":{}}`, want: `{"auto_keywords":8,"Extractor:Old":{}}`},
+ {name: "table ignores stale extractor", parser: "table", input: `{"auto_keywords":8,"Extractor:Old":{}}`, want: `{"auto_keywords":8,"Extractor:Old":{}}`},
+ {name: "canvas has no target", pipeline: ptrString("canvas-1"), input: python027ParserConfig, want: python027ParserConfig},
+ {name: "empty row", input: `{}`, want: `{}`},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ config, want := decodeIngestionConfig(t, tc.input), decodeIngestionConfig(t, tc.want)
+ parser := tc.parser
+ if parser == "" {
+ parser = "general"
+ }
+ if _, err := migrateExtractorConfig(config, parser, tc.pipeline); err != nil {
+ t.Fatal(err)
+ }
+ if got := decodeIngestionConfig(t, encodeIngestionConfig(t, config)); !reflect.DeepEqual(got, want) {
+ t.Fatalf("got %s; want %s", encodeIngestionConfig(t, config), tc.want)
+ }
+ snapshot := encodeIngestionConfig(t, config)
+ if changed, err := migrateExtractorConfig(config, parser, tc.pipeline); err != nil || changed {
+ t.Fatalf("second run changed=%v, err=%v", changed, err)
+ }
+ if encodeIngestionConfig(t, config) != snapshot {
+ t.Fatal("second run changed config")
+ }
+ })
+ }
+}
+
+func ptrString(value string) *string { return &value }
+
+func TestMigrateIngestionParserConfigsSharesDev1Cleanup(t *testing.T) {
+ if ingestionParserConfigTargetVersion != "v1.0.0-rc2.dev1" {
+ t.Fatalf("migration version = %s", ingestionParserConfigTargetVersion)
+ }
+ db := openIngestionMigrationDB(t)
+ insertIngestionConfig(t, db, "knowledgebase", "kb", "general", `{"raptor":{},"auto_keywords":8}`)
+ if err := migrateIngestionParserConfigs(t.Context(), db); err != nil {
+ t.Fatal(err)
+ }
+ var row ingestionConfigMigrationRow
+ if err := db.Table("knowledgebase").Where("id = ?", "kb").Take(&row).Error; err != nil {
+ t.Fatal(err)
+ }
+ config := decodeIngestionConfig(t, *row.ParserConfig)
+ if _, remains := config["raptor"]; remains {
+ t.Fatal("dataset cleanup was skipped")
+ }
+ if _, migrated := config[ingestionTargets["general"].extractorNode]; !migrated {
+ t.Fatal("component conversion was skipped")
+ }
+}
+
+func TestMigrateIngestionParserConfigsUpdateError(t *testing.T) {
+ for _, table := range []string{"knowledgebase", "document"} {
+ t.Run(table, func(t *testing.T) {
+ db := openIngestionMigrationDB(t)
+ for _, id := range []string{"row-a", "row-b"} {
+ insertIngestionConfig(t, db, table, id, "general", python027IngestionConfig)
+ }
+ trigger := fmt.Sprintf(`CREATE TRIGGER reject_parser_config_update
+ BEFORE UPDATE OF parser_config ON %s WHEN OLD.id = 'row-b'
+ BEGIN SELECT RAISE(FAIL, 'forced parser config update failure'); END`, table)
+ if err := db.Exec(trigger).Error; err != nil {
+ t.Fatal(err)
+ }
+ err := migrateIngestionParserConfigs(t.Context(), db)
+ if err == nil {
+ t.Fatal("expected update failure")
+ }
+ for _, detail := range []string{table, "row-b", "forced parser config update failure"} {
+ if !strings.Contains(err.Error(), detail) {
+ t.Errorf("error %q does not identify %q", err, detail)
+ }
+ }
+ var rows []ingestionConfigMigrationRow
+ if err := db.Table(table).Find(&rows).Error; err != nil {
+ t.Fatal(err)
+ }
+ for _, row := range rows {
+ if *row.ParserConfig != python027IngestionConfig {
+ t.Fatalf("failed batch changed row %s", row.ID)
+ }
+ }
+ if got := databaseVersion(t, db); got != "" {
+ t.Fatalf("failed migration advanced version to %q", got)
+ }
+ })
+ }
+}
diff --git a/internal/dao/ingestion_parser_config_runtime_test.go b/internal/dao/ingestion_parser_config_runtime_test.go
new file mode 100644
index 0000000000..1c9be4f2f0
--- /dev/null
+++ b/internal/dao/ingestion_parser_config_runtime_test.go
@@ -0,0 +1,743 @@
+package dao_test
+
+import (
+ "encoding/json"
+ "reflect"
+ "strings"
+ "testing"
+
+ "github.com/glebarez/sqlite"
+ "gorm.io/gorm"
+
+ "ragflow/internal/dao"
+ "ragflow/internal/entity"
+ "ragflow/internal/ingestion/component"
+ "ragflow/internal/ingestion/component/chunker"
+ "ragflow/internal/ingestion/component/schema"
+ "ragflow/internal/ingestion/pipeline"
+ "ragflow/internal/parser/parser"
+ "ragflow/internal/service"
+)
+
+// python027MaximalIngestionConfig carries every Python-era key the migration can
+// convert, so one run exercises every converter and every target family.
+const python027MaximalIngestionConfig = `{
+ "layout_recognize":"TCADP Parser",
+ "pages":[[1,3]],
+ "table_result_type":"2",
+ "markdown_image_response_type":"2",
+ "table_column_mode":"manual",
+ "table_column_roles":{"a":"indexing"},
+ "table_column_names":["a"],
+ "chunk_token_num":256,
+ "delimiter":"\n",
+ "overlapped_percent":0.29,
+ "table_context_size":10,
+ "image_context_size":20,
+ "parent_child":{"use_parent_child":true,"children_delimiter":"。"},
+ "filename_embd_weight":0.2
+}`
+
+// TestMigratedIngestionConfigReachesRuntime drives the real row walk and then
+// builds the real components from the stored config, so a migrated setting that
+// the runtime cannot read fails here.
+func TestMigratedIngestionConfigReachesRuntime(t *testing.T) {
+ db := openIngestionRuntimeDB(t)
+ insertIngestionRuntimeRow(t, db, "kb-general", "general",
+ `{"layout_recognize":"plain text","pages":[[2,3]],"chunk_token_num":256,"delimiter":"\n","overlapped_percent":0.29,"filename_embd_weight":0.2}`)
+ insertIngestionRuntimeRow(t, db, "kb-tcadp", "general", `{"layout_recognize":"TCADP Parser"}`)
+ insertIngestionRuntimeRow(t, db, "kb-book", "book", `{"layout_recognize":true,"pages":[[1,2]],"chunk_token_num":300}`)
+ if err := dao.MigrateIngestionParserConfigsForTest(t.Context(), db); err != nil {
+ t.Fatal(err)
+ }
+
+ t.Run("general", func(t *testing.T) {
+ config := migratedConfig(t, db, "kb-general")
+ if _, leftovers := config["layout_recognize"]; leftovers {
+ t.Fatalf("flat key survived: %v", config)
+ }
+ effective := effectiveComponentParams(t, "general", config)
+
+ pdf := parser.NewPDFParser()
+ pdf.ConfigureFromSetup(effective["Parser:HipSignsRhyme"]["pdf"].(map[string]any))
+ if pdf.ParseMethod != "plain_text" {
+ t.Errorf("parse_method=%q; want plain_text", pdf.ParseMethod)
+ }
+ if !reflect.DeepEqual(pdf.Pages, [][]int{{2, 3}}) {
+ t.Errorf("pages=%v; want [[2 3]]", pdf.Pages)
+ }
+
+ chunkerParams := effective["GeneralChunker:SixApplesFall"]
+ if _, err := chunker.NewGeneralChunker(chunkerParams); err != nil {
+ t.Fatalf("migrated chunker params are not runtime-acceptable: %v", err)
+ }
+ if got := schema.NormalizeOverlappedPercent(chunkerParams["overlapped_percent"]); got != 29 {
+ t.Errorf("overlapped_percent normalizes to %v; want 29", got)
+ }
+ if got := chunkerParams["chunk_token_size"]; got != float64(256) {
+ t.Errorf("chunk_token_size=%v; want 256", got)
+ }
+
+ tokenizerParams := effective["Tokenizer:LegalReadersDecide"]
+ if _, err := component.NewTokenizerComponent(tokenizerParams); err != nil {
+ t.Fatalf("migrated tokenizer params are not runtime-acceptable: %v", err)
+ }
+ if got := tokenizerParams["filename_embd_weight"]; got != 0.2 {
+ t.Errorf("filename_embd_weight=%v; want 0.2", got)
+ }
+ })
+
+ t.Run("TCADP matches the existing runtime", func(t *testing.T) {
+ effective := effectiveComponentParams(t, "general", migratedConfig(t, db, "kb-tcadp"))
+ node := effective["Parser:HipSignsRhyme"]
+ pdf := node["pdf"].(map[string]any)["parse_method"].(string)
+ if pdf != "tcadp parser" || !parser.IsPDFParseMethod(pdf) {
+ t.Fatalf("migrated PDF engine would be dispatched as a VLM: %q", pdf)
+ }
+ if got := node["spreadsheet"].(map[string]any)["parse_method"]; got != "tcadp" {
+ t.Fatalf("spreadsheet engine=%v; want tcadp", got)
+ }
+ })
+
+ t.Run("book keeps the defaults its template bakes", func(t *testing.T) {
+ config := migratedConfig(t, db, "kb-book")
+ node, ok := config["TitleChunker:GrumpyGarlicsBake"].(map[string]any)
+ if !ok {
+ t.Fatalf("no TitleChunker node written: %v", config)
+ }
+ if node["chunk_token_cap"] != float64(300) {
+ t.Fatalf("chunk_token_cap=%v; want 300", node["chunk_token_cap"])
+ }
+ effective := effectiveComponentParams(t, "book", config)
+ pdf := parser.NewPDFParser()
+ pdf.ConfigureFromSetup(effective["Parser:HipSignsRhyme"]["pdf"].(map[string]any))
+ // The book template bakes remove_toc=true, which component.defaultSetups()
+ // does not: writing the pdf family must carry it, or a migrated dataset
+ // would start indexing its own table of contents.
+ if !pdf.RemoveTOC {
+ t.Error("remove_toc was lost; the migrated pdf family dropped a template default")
+ }
+ if pdf.EnableMultiColumn {
+ t.Error("enable_multi_column leaked in from another template")
+ }
+ if !reflect.DeepEqual(pdf.Pages, [][]int{{1, 2}}) {
+ t.Errorf("pages=%v; want [[1 2]]", pdf.Pages)
+ }
+ })
+}
+
+// TestMigratedParentChildMatchesWritePath asserts the migration derives
+// children_delimiters exactly as the API's own ApplyParentChildChunkerConfig
+// does for the same setting, so a migrated row and a row saved through the UI
+// agree. children_delimiters is the only half the chunker reads at run time.
+func TestMigratedParentChildMatchesWritePath(t *testing.T) {
+ const nodeID = "GeneralChunker:SixApplesFall"
+ for _, tc := range []struct {
+ name string
+ input string
+ want []any
+ }{
+ {"group with delimiter", `{"parent_child":{"use_parent_child":true,"children_delimiter":"。"}}`, []any{"。"}},
+ {"group with multi delimiter", `{"parent_child":{"use_parent_child":true,"children_delimiter":"。;"}}`, []any{"。", ";"}},
+ {"legacy flat keys", `{"enable_children":true,"children_delimiter":"。"}`, []any{"。"}},
+ {"disabled", `{"enable_children":false,"children_delimiter":"。"}`, []any{}},
+ {"disabled group", `{"parent_child":{"use_parent_child":false}}`, []any{}},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ config := decodeIngestionConfig(t, tc.input)
+ if _, err := dao.MigrateParserChunkerTokenizerConfigForTest(config, "general", nil, "knowledgebase", "row-1"); err != nil {
+ t.Fatal(err)
+ }
+ node, ok := config[nodeID].(map[string]any)
+ if !ok {
+ t.Fatalf("no chunker node written: %v", config)
+ }
+ got := node["children_delimiters"]
+ if !reflect.DeepEqual(got, tc.want) {
+ t.Fatalf("children_delimiters=%v; want %v", got, tc.want)
+ }
+ // The write path, given the same component config, must derive the
+ // same list. It only ever reads the "parent_child" group.
+ apiSided := entity.JSONMap{nodeID: cloneConfig(node)}
+ pipeline.ApplyParentChildChunkerConfig(apiSided, nil)
+ derived := apiSided[nodeID].(map[string]any)["children_delimiters"]
+ if !reflect.DeepEqual(jsonNormalize(t, got), jsonNormalize(t, derived)) {
+ t.Fatalf("migration wrote %v; the write path derives %v from the same group", got, derived)
+ }
+ if _, ok := node["parent_child"].(map[string]any); !ok {
+ t.Fatalf("the parent_child group was not written: %v", node)
+ }
+ })
+ }
+}
+
+// TestMigratedParentChildEnabledWithoutDelimiterDefaultsToNewline pins the one
+// place the migration deliberately differs from ApplyParentChildChunkerConfig:
+// Python's ParentChildConfig defaults children_delimiter to "\n" and its
+// flattening writes that default into the config, while the Go write path
+// derives an empty list when the delimiter is absent. Keeping Python's default
+// preserves the children the Python-era run produced.
+func TestMigratedParentChildEnabledWithoutDelimiterDefaultsToNewline(t *testing.T) {
+ config := decodeIngestionConfig(t, `{"parent_child":{"use_parent_child":true}}`)
+ if _, err := dao.MigrateParserChunkerTokenizerConfigForTest(config, "general", nil, "knowledgebase", "row-1"); err != nil {
+ t.Fatal(err)
+ }
+ node := config["GeneralChunker:SixApplesFall"].(map[string]any)
+ group := node["parent_child"].(map[string]any)
+ if group["children_delimiter"] != "\n" {
+ t.Fatalf("children_delimiter=%v; want the Python default \"\\n\"", group["children_delimiter"])
+ }
+ if !reflect.DeepEqual(node["children_delimiters"], []any{"\n"}) {
+ t.Fatalf("children_delimiters=%v; want [\"\\n\"]", node["children_delimiters"])
+ }
+}
+
+// TestIngestionMigrationMatchesBuiltinTemplates pins the migration's literal
+// target table against the embedded templates by reading the node IDs and file
+// families out of the templates themselves: a renamed component, a dropped
+// family, or a new chunker type fails here instead of silently skipping a row or
+// thinning a family.
+func TestIngestionMigrationMatchesBuiltinTemplates(t *testing.T) {
+ registry, err := pipeline.DefaultRegistry()
+ if err != nil {
+ t.Fatal(err)
+ }
+ // targetFamilies are the only Parser families the migration can write.
+ const targetFamilies = "pdf,spreadsheet"
+ for _, ref := range append(registry.Refs(), "naive") {
+ t.Run(ref, func(t *testing.T) {
+ template, ok := registry.Get(ref)
+ if !ok {
+ t.Fatal("template missing")
+ }
+ components := templateComponents(t, template.DSL)
+ parserCpn := soleComponent(t, components, "Parser")
+ chunkerCpn := soleComponentSuffix(t, components, "Chunker")
+ tokenizerCpn := soleComponent(t, components, "Tokenizer")
+ templateFamilies := parserFamilies(t, parserCpn.params)
+ rawDSL, err := json.Marshal(template.DSL)
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ for _, tc := range []struct {
+ name string
+ // config is the Python-era row being migrated.
+ config string
+ // families are the template families the config must land on;
+ // every other target family must be left alone.
+ families map[string]bool
+ // chunker reports whether the config carries a chunker-owned key.
+ chunker bool
+ // tokenizer reports whether the config carries a tokenizer-owned key.
+ tokenizer bool
+ }{
+ {
+ name: "maximal",
+ config: python027MaximalIngestionConfig,
+ families: map[string]bool{"pdf": true, "spreadsheet": true},
+ chunker: true,
+ tokenizer: true,
+ },
+ {
+ name: "engine",
+ config: `{"layout_recognize":"TCADP Parser"}`,
+ families: map[string]bool{"pdf": true, "spreadsheet": true},
+ },
+ {
+ name: "columns",
+ config: `{"table_column_mode":"manual","table_column_roles":{"a":"indexing"},"table_column_names":["a"]}`,
+ families: map[string]bool{"spreadsheet": true},
+ },
+ {
+ name: "chunking",
+ config: `{"chunk_token_num":256,"delimiter":"\n","overlapped_percent":0.29,"enable_children":true}`,
+ families: map[string]bool{},
+ chunker: true,
+ },
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ config := decodeIngestionConfig(t, tc.config)
+ changed, err := dao.MigrateParserChunkerTokenizerConfigForTest(config, ref, nil, "knowledgebase", "row-1")
+ if err != nil || !changed {
+ t.Fatalf("changed=%v err=%v", changed, err)
+ }
+
+ if tc.name == "maximal" && len(config) == 0 {
+ t.Fatal("migration wrote nothing for the maximal config")
+ }
+ // Snapshot before cleaning so mutations cannot hide a lost parameter.
+ expected := jsonNormalize(t, config)
+ cleaned := pipeline.CleanComponentParams(rawDSL, config)
+ if !reflect.DeepEqual(jsonNormalize(t, cleaned), expected) {
+ t.Fatalf("API cleaning changed migrated params: got %v; want %v", cleaned, expected)
+ }
+
+ // Parser: exactly the template's families the config targets,
+ // never a family the template does not declare.
+ wantFamilies := map[string]bool{}
+ for family := range tc.families {
+ if _, present := templateFamilies[family]; present {
+ wantFamilies[family] = true
+ }
+ }
+ gotFamilies := map[string]bool{}
+ node, nodeWritten := config[parserCpn.cpnID].(map[string]any)
+ for family, raw := range node {
+ familyParams, isMap := raw.(map[string]any)
+ if !isMap {
+ t.Fatalf("family %q is not an object: %v", family, raw)
+ }
+ if !strings.Contains(targetFamilies, family) {
+ t.Fatalf("migration wrote family %q, which is not a migration target: %v", family, familyParams)
+ }
+ gotFamilies[family] = true
+ }
+ if !reflect.DeepEqual(gotFamilies, wantFamilies) {
+ t.Fatalf("Parser families %v; want %v (config=%s)", gotFamilies, wantFamilies, tc.config)
+ }
+ if len(wantFamilies) == 0 && nodeWritten {
+ t.Fatalf("Parser node written for a config that targets no family: %v", node)
+ }
+ if nodeWritten {
+ assertFamiliesPreserved(t, node, templateFamilies, wantFamilies)
+ }
+ // The Parser the run compiles must accept the override. The
+ // merge is applyOverrideParams': the template params with the
+ // stored component params on top.
+ if _, err := component.NewParserComponent(mergeComponentParams(parserCpn.params, node)); err != nil {
+ t.Fatalf("migrated Parser params do not compile: %v", err)
+ }
+
+ // Chunker: only the chunkers that own the migrated params
+ // may receive a node, on the template's own node ID.
+ chunkerNode, chunkerWritten := config[chunkerCpn.cpnID].(map[string]any)
+ switch {
+ case !tc.chunker:
+ if chunkerWritten {
+ t.Fatalf("chunker node written for a chunker-agnostic config: %v", chunkerNode)
+ }
+ case chunkerCpn.name == "GeneralChunker" || chunkerCpn.name == "TokenChunker":
+ if !chunkerWritten {
+ t.Fatalf("no %s node written", chunkerCpn.cpnID)
+ }
+ if _, err := chunkerNodeAccepts(chunkerCpn.name, mergeComponentParams(chunkerCpn.params, chunkerNode)); err != nil {
+ t.Fatalf("migrated chunker params do not compile: %v", err)
+ }
+ case chunkerCpn.name == "TitleChunker":
+ if !chunkerWritten {
+ t.Fatalf("no %s node written", chunkerCpn.cpnID)
+ }
+ if value, ok := schema.NumericFromAny(chunkerNode["chunk_token_cap"]); !ok || value != 256 {
+ t.Fatalf("chunk_token_cap=%v; want 256", chunkerNode["chunk_token_cap"])
+ }
+ default:
+ if chunkerWritten {
+ t.Fatalf("%s must not receive migrated chunker params: %v", chunkerCpn.name, chunkerNode)
+ }
+ }
+
+ // Tokenizer: every template has one, and the filename weight
+ // is the only param the migration puts on it.
+ tokenizerNode, tokenizerWritten := config[tokenizerCpn.cpnID].(map[string]any)
+ if !tc.tokenizer {
+ if tokenizerWritten {
+ t.Fatalf("tokenizer node written for a config with no tokenizer param: %v", tokenizerNode)
+ }
+ } else {
+ if !tokenizerWritten {
+ t.Fatalf("no %s node written", tokenizerCpn.cpnID)
+ }
+ if value, ok := schema.NumericFromAny(tokenizerNode["filename_embd_weight"]); !ok || value != 0.2 {
+ t.Fatalf("filename_embd_weight=%v; want 0.2", tokenizerNode["filename_embd_weight"])
+ }
+ if _, err := component.NewTokenizerComponent(mergeComponentParams(tokenizerCpn.params, tokenizerNode)); err != nil {
+ t.Fatalf("migrated tokenizer params do not compile: %v", err)
+ }
+ }
+
+ // No node outside the migration's three targets.
+ for cpnID := range config {
+ if cpnID == parserCpn.cpnID || cpnID == chunkerCpn.cpnID || cpnID == tokenizerCpn.cpnID {
+ continue
+ }
+ t.Fatalf("migration wrote an unexpected component: %q", cpnID)
+ }
+ })
+ }
+ })
+ }
+}
+
+func chunkerNodeAccepts(name string, params map[string]any) (any, error) {
+ if name == "TokenChunker" {
+ return chunker.NewTokenChunker(params)
+ }
+ return chunker.NewGeneralChunker(params)
+}
+
+// mergeComponentParams overlays the component-scoped overrides on the template's
+// baked params the way internal/agent/canvas.applyOverrideParams does at run
+// time: the override wins on the top-level key, so an override replaces a whole
+// Parser family.
+func mergeComponentParams(templateParams, overrides map[string]any) map[string]any {
+ out := make(map[string]any, len(templateParams)+len(overrides))
+ for key, value := range templateParams {
+ out[key] = value
+ }
+ for key, value := range overrides {
+ out[key] = value
+ }
+ return out
+}
+
+// templateComponent is one component of a builtin template.
+type templateComponent struct {
+ cpnID string
+ name string
+ params map[string]any
+}
+
+func templateComponents(t *testing.T, dsl map[string]any) []templateComponent {
+ t.Helper()
+ raw, err := json.Marshal(dsl)
+ if err != nil {
+ t.Fatal(err)
+ }
+ schemas, err := pipeline.ExtractAllComponentParams(raw)
+ if err != nil {
+ t.Fatal(err)
+ }
+ out := make([]templateComponent, 0, len(schemas))
+ for _, s := range schemas {
+ out = append(out, templateComponent{cpnID: s.CpnID, name: s.ComponentName, params: s.ParamsDefaults})
+ }
+ return out
+}
+
+func soleComponent(t *testing.T, components []templateComponent, name string) templateComponent {
+ t.Helper()
+ found := templateComponent{}
+ count := 0
+ for _, c := range components {
+ if c.name == name {
+ found, count = c, count+1
+ }
+ }
+ if count != 1 {
+ t.Fatalf("template has %d %s components; want 1", count, name)
+ }
+ return found
+}
+
+func soleComponentSuffix(t *testing.T, components []templateComponent, suffix string) templateComponent {
+ t.Helper()
+ found := templateComponent{}
+ count := 0
+ for _, c := range components {
+ if strings.HasSuffix(c.name, suffix) {
+ found, count = c, count+1
+ }
+ }
+ if count != 1 {
+ t.Fatalf("template has %d *%s components; want 1", count, suffix)
+ }
+ return found
+}
+
+func parserFamilies(t *testing.T, params map[string]any) map[string]map[string]any {
+ t.Helper()
+ out := map[string]map[string]any{}
+ for _, key := range []string{"pdf", "spreadsheet"} {
+ if raw, exists := params[key]; exists {
+ family, ok := raw.(map[string]any)
+ if !ok {
+ t.Fatalf("Parser family %q is not an object", key)
+ }
+ out[key] = family
+ }
+ }
+ return out
+}
+
+// overriddenFamilyKeys are family params the migration overwrites by design
+// whenever the config supplies them, so the migrated value may differ from the
+// template's.
+var overriddenFamilyKeys = map[string]bool{
+ "parse_method": true,
+ "pages": true,
+ "column_mode": true,
+ "column_roles": true,
+ "column_names": true,
+ "layout_recognizer": true,
+ "table_result_type": true,
+ "markdown_image_response_type": true,
+}
+
+// inertFamilyKeys are template-baked family keys the migration may leave out,
+// with the reason dropping them is harmless. component.NewParserComponent seeds
+// every family from defaultSetups() and then overlays the params, and the
+// run-level override replaces a family wholesale, so whatever a written family
+// leaves out comes from defaultSetups() — equal to the template's value, or
+// read by nobody, for every entry below.
+//
+// A key may appear here and in overriddenFamilyKeys: the two answer different
+// questions (this one, "is the template's value safe to lose"; that one, "may
+// the value the migration writes differ from the template's").
+var inertFamilyKeys = map[string]string{
+ "output_format": "normalizeParserOutputFormats forces JSON on every family",
+ "preprocess": "the value has no Go reader; the preprocessing stage is selected by the chunker's own options",
+ "suffix": "no Go reader; file routing uses the document's own suffix column, not the family list",
+ "vlm": "empty in every template; absent falls back to the family llm_id and then the tenant model, and only Markdown and audio read a family vlm at all",
+ "flatten_media_to_text": "false in every template, which is also defaultSetups()'s default",
+ "pages": "the only template that bakes it lists every page ([[1,100000]]), which is what an absent pages means",
+ "parse_method": "a written pdf family always restates it; for the spreadsheet family defaultSetups() re-derives the same deepdoc engine",
+ "column_mode": "the table template bakes auto, which is what an absent column_mode means to indexdoc's resolveTableColumnConfig",
+ "column_roles": "the table template bakes the empty role map, which is what an absent column_roles means",
+ "column_names": "the table template bakes the empty name list, which is what an absent column_names means",
+}
+
+// assertFamiliesPreserved checks that no template-baked key of a written family
+// is left unaccounted for: a key that would simply disappear from the effective
+// config must be either carried or documented above.
+func assertFamiliesPreserved(t *testing.T, node map[string]any, templateFamilies map[string]map[string]any, written map[string]bool) {
+ t.Helper()
+ for family := range written {
+ templateFamily, ok := templateFamilies[family]
+ if !ok {
+ t.Fatalf("migration wrote family %q, which the template does not declare", family)
+ }
+ familyParams, _ := node[family].(map[string]any)
+ for key, templateValue := range templateFamily {
+ got, present := familyParams[key]
+ if !present {
+ if _, inert := inertFamilyKeys[key]; !inert {
+ t.Fatalf("family %s: template key %q would be dropped by the override and is not accounted for; carry it or document it as inert", family, key)
+ }
+ continue
+ }
+ if overriddenFamilyKeys[key] {
+ continue
+ }
+ if !reflect.DeepEqual(got, templateValue) {
+ t.Fatalf("family %s: key %q = %v, but the template bakes %v", family, key, got, templateValue)
+ }
+ }
+ }
+}
+
+// effectiveComponentParams reproduces the run-level merge: the template's baked
+// component params with the stored parser_config overlaid, the stored value
+// winning on the top-level key (internal/agent/canvas.applyOverrideParams), then
+// component.NewParserComponent's own defaults underneath.
+func effectiveComponentParams(t *testing.T, parserID string, config map[string]any) map[string]map[string]any {
+ t.Helper()
+ registry, err := pipeline.DefaultRegistry()
+ if err != nil {
+ t.Fatal(err)
+ }
+ template, ok := registry.Get(parserID)
+ if !ok {
+ t.Fatal("template missing")
+ }
+ out := map[string]map[string]any{}
+ for _, c := range templateComponents(t, template.DSL) {
+ out[c.cpnID] = c.params
+ }
+ for cpnID, raw := range config {
+ params, ok := raw.(map[string]any)
+ if !ok {
+ t.Fatalf("stored cpnID %q is not an object", cpnID)
+ }
+ out[cpnID] = mergeComponentParams(out[cpnID], params)
+ }
+ return out
+}
+
+func openIngestionRuntimeDB(t *testing.T) *gorm.DB {
+ t.Helper()
+ db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ sqlDB, err := db.DB()
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = sqlDB.Close() })
+ if err := db.AutoMigrate(&entity.Knowledgebase{}, &entity.SystemSettings{}); err != nil {
+ t.Fatal(err)
+ }
+ return db
+}
+
+func insertIngestionRuntimeRow(t *testing.T, db *gorm.DB, id, parserID, config string) {
+ t.Helper()
+ err := db.Exec(
+ `INSERT INTO knowledgebase (id,tenant_id,name,embd_id,created_by,parser_id,parser_config) VALUES (?,?,?,?,?,?,?)`,
+ id, "tenant-1", "dataset", "embedding", "user-1", parserID, config).Error
+ if err != nil {
+ t.Fatal(err)
+ }
+}
+
+func migratedConfig(t *testing.T, db *gorm.DB, id string) map[string]any {
+ t.Helper()
+ var raw string
+ if err := db.Table("knowledgebase").Where("id = ?", id).Pluck("parser_config", &raw).Error; err != nil {
+ t.Fatal(err)
+ }
+ return decodeIngestionConfig(t, raw)
+}
+
+func decodeIngestionConfig(t *testing.T, raw string) map[string]any {
+ t.Helper()
+ var config map[string]any
+ if err := json.Unmarshal([]byte(raw), &config); err != nil {
+ t.Fatal(err)
+ }
+ return config
+}
+
+// jsonNormalize round-trips a value through JSON so two structurally equal
+// values that differ only in Go slice type ([]string vs []any, which is what a
+// DB round-trip produces either way) compare equal.
+func jsonNormalize(t *testing.T, value any) any {
+ t.Helper()
+ raw, err := json.Marshal(value)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var out any
+ if err := json.Unmarshal(raw, &out); err != nil {
+ t.Fatal(err)
+ }
+ return out
+}
+
+// cloneConfig deep-copies a parser_config so a helper that mutates its argument
+// cannot corrupt the caller's fixture.
+func cloneConfig(config map[string]any) map[string]any {
+ raw, err := json.Marshal(config)
+ if err != nil {
+ return config
+ }
+ var out map[string]any
+ if err := json.Unmarshal(raw, &out); err != nil {
+ return config
+ }
+ return out
+}
+
+func TestMigratedExtractorConfigReachesRuntime(t *testing.T) {
+ db := openIngestionRuntimeDB(t)
+ input := `{"auto_keywords":8,"auto_questions":2,"enable_metadata":true,"metadata":[{"key":"author","type":"string"}],"built_in_metadata":[{"key":"file_name","type":"string"}],"Extractor:AutoExtractDefault":{"tags":{"top_n":5,"tag_file_id":"tag-file"},"keywords":{"system_prompt":"custom keywords"},"summary":{"enabled":true,"system_prompt":"custom summary"}}}`
+ insertIngestionRuntimeRow(t, db, "kb-1", "general", input)
+ if err := dao.MigrateIngestionParserConfigsForTest(t.Context(), db); err != nil {
+ t.Fatal(err)
+ }
+ config := migratedConfig(t, db, "kb-1")
+ got := extractorFromMigratedConfig(t, config).Param
+ if !got.Keywords.Enabled || got.Keywords.TopN != 8 || got.Keywords.SystemPrompt != "custom keywords" || !got.Questions.Enabled || got.Questions.TopN != 2 {
+ t.Fatalf("keyword/question settings lost: %#v", got)
+ }
+ if !got.Metadata.Enabled || len(got.Metadata.Metadata) != 1 || got.Metadata.Metadata[0].Key != "author" || len(got.Metadata.BuiltInMetadata) != 1 || got.Metadata.BuiltInMetadata[0].Key != "file_name" {
+ t.Fatalf("metadata=%#v", got.Metadata)
+ }
+ if !got.Tags.Enabled || got.Tags.TagFileID != "tag-file" || got.Tags.TopN != 5 || !got.Summary.Enabled || got.Summary.SystemPrompt != "custom summary" || got.LLMID != "tenant-llm" {
+ t.Fatalf("existing scoped settings lost: %#v", got)
+ }
+}
+
+func TestExtractorMigrationMatchesBuiltinTemplates(t *testing.T) {
+ registry, err := pipeline.DefaultRegistry()
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, ref := range append(registry.Refs(), "naive") {
+ t.Run(ref, func(t *testing.T) {
+ template, ok := registry.Get(ref)
+ if !ok {
+ t.Fatal("template missing")
+ }
+ components := template.DSL["components"].(map[string]any)
+ _, hasExtractor := components["Extractor:AutoExtractDefault"]
+ config := map[string]any{"auto_keywords": float64(8)}
+ changed, err := dao.MigrateExtractorConfigForTest(config, ref, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if changed != hasExtractor {
+ t.Fatalf("changed=%v; template has Extractor=%v", changed, hasExtractor)
+ }
+ if hasExtractor {
+ if node, ok := config["Extractor:AutoExtractDefault"].(map[string]any); !ok || node["keywords"].(map[string]any)["top_n"] != 8 {
+ t.Fatalf("config=%v", config)
+ }
+ } else if config["auto_keywords"] != float64(8) {
+ t.Fatal("settings without target were discarded")
+ }
+ })
+ }
+}
+
+func TestExtractorMigrationPreservesRuntimeSettings(t *testing.T) {
+ for _, tt := range []struct {
+ name, input string
+ check func(*testing.T, *component.ExtractorComponent)
+ }{
+ {
+ name: "empty Python metadata keeps builtins",
+ input: `{"enable_metadata":true,"metadata":{},"built_in_metadata":[{"key":"file_name"}]}`,
+ check: func(t *testing.T, extractor *component.ExtractorComponent) {
+ got := extractor.Param.Metadata
+ if !got.Enabled || len(got.BuiltInMetadata) != 1 || got.BuiltInMetadata[0].Key != "file_name" {
+ t.Fatalf("metadata=%#v", got)
+ }
+ },
+ },
+ {
+ name: "builtin uses active extractor despite stale node",
+ input: `{"auto_keywords":8,"Extractor:Old":{"summary":{"enabled":true}}}`,
+ check: func(t *testing.T, extractor *component.ExtractorComponent) {
+ got := extractor.Param.Keywords
+ if !got.Enabled || got.TopN != 8 {
+ t.Fatalf("keywords=%#v", got)
+ }
+ },
+ },
+ } {
+ t.Run(tt.name, func(t *testing.T) {
+ var config map[string]any
+ if err := json.Unmarshal([]byte(tt.input), &config); err != nil {
+ t.Fatal(err)
+ }
+ if changed, err := dao.MigrateExtractorConfigForTest(config, "general", nil); err != nil || !changed {
+ t.Fatalf("changed=%v, err=%v", changed, err)
+ }
+ tt.check(t, extractorFromMigratedConfig(t, config))
+ })
+ }
+}
+
+func extractorFromMigratedConfig(t *testing.T, config map[string]any) *component.ExtractorComponent {
+ t.Helper()
+ registry, err := pipeline.DefaultRegistry()
+ if err != nil {
+ t.Fatal(err)
+ }
+ template, ok := registry.Get("general")
+ if !ok {
+ t.Fatal("general template missing")
+ }
+ dsl, err := json.Marshal(template.DSL)
+ if err != nil {
+ t.Fatal(err)
+ }
+ cleaned := service.ApplyComponentScopedParserConfig(pipeline.BuildParserConfig(dsl, config), "tenant-llm")
+ c, err := component.NewExtractorComponent(cleaned["Extractor:AutoExtractDefault"].(map[string]any))
+ if err != nil {
+ t.Fatal(err)
+ }
+ return c.(*component.ExtractorComponent)
+}