RAGFlow is an agent-native database: a Retrieval-Augmented Generation (RAG) engine that grew into context infrastructure.
Go agent-harnessagent-native-databaseagentic-aiagentic-retrievalagentic-searchaiai-agentscontext-enginecontext-engineeringcontext-managementharness-engineeringknowledge-compilationragretrieval-augmented-generationretrieval-harnesssearch-harness
Fix: remove raptor and graphrag when migrate parse config (#20599)
Lynn1413972e09a2
diff --git a/docs/administrator/migration/database_change_log.md b/docs/administrator/migration/database_change_log.mdindex 8b7591961e..919e8d575c 100644--- a/docs/administrator/migration/database_change_log.md+++ b/docs/administrator/migration/database_change_log.md@@ -16,7 +16,7 @@ A version-by-version record of the schema and data changes the Go backend applie- The marker is a row in `system_settings` with the key `mysql_migration.database.version`.- The marker prevents a migration step from running again after that step has completed successfully.-- Go defines the versions as constants: `modelMigrationBaseVersion` and `modelMigrationTargetVersion` in [migration_version.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/migration_version.go), and `conversationHistoryTargetVersion` in [conversation_history_migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/conversation_history_migration.go).+- Go defines the versions as constants: `modelMigrationBaseVersion` and `modelMigrationTargetVersion` in [migration_version.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/migration_version.go), `conversationHistoryTargetVersion` in [conversation_history_migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/conversation_history_migration.go), and `knowledgebaseParserConfigTargetVersion` in [migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/migration.go).- A step runs when the stored version is lower than the step's version. Versions are compared with Python PEP 440 semantics, where a `dev` prerelease sorts *before* the release it precedes (`v1.0.0-rc1.dev1` < `v1.0.0-rc1`). A missing or unparsable marker never skips a step.- `checkDatabaseVersion` in [cmd/ragflow_server.go](https://github.com/infiniflow/ragflow/blob/main/cmd/ragflow_server.go) refuses to start when the code version is older than the stored database version, so a database migrated by a newer release cannot be opened by an older one.@@ -26,6 +26,7 @@ Startup order in `InitDB` ([database.go](https://github.com/infiniflow/ragflow/b2. `migrateIngestionLogRunIdentity` — ingestion log columns and indexes.3. `AutoMigrate` — converges every ORM model.4. `migrateConversationHistory` — runs *after* `AutoMigrate`, which creates the child tables it backfills.+5. `migrateKnowledgebaseParserConfig` — runs last, so the marker does not move past the conversation split's version before that split has run.## Version overview@@ -34,6 +35,7 @@ Startup order in `InitDB` ([database.go](https://github.com/infiniflow/ragflow/b| `v0.26.0` | Rebuild the tenant model tables from `tenant_llm` and normalize stored model ids | `modelMigrationBaseVersion` || `v1.0.0-rc1` | Seed factory-declared models, merge `model_type` into an integer bitmask, populate the `tenant_*_id` columns | `modelMigrationTargetVersion` || `v1.0.0-rc1.dev1` | Split conversation message and reference payloads into child tables | `conversationHistoryTargetVersion` |+| `v1.0.0-rc2.dev1` | Delete the retired `raptor` and `graphrag` sections from `knowledgebase.parser_config` | `knowledgebaseParserConfigTargetVersion` |The two tenant model steps are cumulative: a database at `v0.26.0` still runs the `v1.0.0-rc1` step, and a database at or above `v1.0.0-rc1` runs neither.@@ -122,6 +124,27 @@ On success the step writes `v1.0.0-rc1.dev1` into the version marker.---+## v1.0.0-rc2.dev1++Retires the `raptor` and `graphrag` sections of the dataset parser config. Both moved out of `knowledgebase.parser_config`, so the stored sections are dead data no code path reads any more.++### Data migration++| Table | Change |+|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|+| `knowledgebase` | `parser_config` is parsed as a JSON object; the top level `raptor` and `graphrag` keys are deleted and the remaining object is written back to the same column. |++Execution details:++- Rows are read in id-ordered batches of 256, each batch in its own transaction, so an interrupted run resumes from the last committed batch.+- Only rows whose config actually carries one of the keys are rewritten; a row without them is left byte-identical.+- A `NULL`, empty or non-object payload is skipped rather than failing the step.+- The step returns without touching the marker when the `knowledgebase` table does not exist.++On success the step writes `v1.0.0-rc2.dev1` into the version marker.++---+## Version-less stepsThese steps are not gated by the version marker. They are idempotent and run on every startup, either through `RunMigrations` or through the ingestion helpers in `InitDB`.diff --git a/internal/dao/database.go b/internal/dao/database.goindex f5593462d6..4587596eae 100644--- a/internal/dao/database.go+++ b/internal/dao/database.go@@ -246,6 +246,14 @@ func InitDB(ctx context.Context, migrateDB bool) error {if err = migrateConversationHistory(ctx, DB); err != nil {return fmt.Errorf("failed to migrate conversation history: %w", err)}+ // Drop the retired raptor and graphrag sections from the dataset parser+ // configs. It runs after the conversation split because the two steps+ // share one version marker: this step's version sorts above the split's+ // v1.0.0-rc1, so marking it earlier would skip a split that has not run.+ if err = migrateKnowledgebaseParserConfig(ctx, DB); err != nil {+ return fmt.Errorf("failed to migrate knowledgebase parser config: %w", err)+ }+if err = migrateTenantModelMaxTokens(ctx, DB); err != nil {return fmt.Errorf("failed to backfill tenant model max_tokens: %w", err)}diff --git a/internal/dao/knowledgebase_parser_config_test.go b/internal/dao/knowledgebase_parser_config_test.gonew file mode 100644index 0000000000..0550a5b93d--- /dev/null+++ b/internal/dao/knowledgebase_parser_config_test.go@@ -0,0 +1,174 @@+//+// Copyright 2026 The InfiniFlow Authors. All Rights Reserved.+//+// Licensed under the Apache License, Version 2.0 (the "License");+// you may not use this file except in compliance with the License.+// You may obtain a copy of the License at+//+// http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing, software+// distributed under the License is distributed on an "AS IS" BASIS,+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+// See the License for the specific language governing permissions and+// limitations under the License.++package dao++import (+ "encoding/json"+ "testing"++ "github.com/glebarez/sqlite"+ "gorm.io/gorm"++ "ragflow/internal/entity"+)++// legacyParserConfig is a dataset parser config as written by a release that+// still carried the raptor and graphrag sections.+const legacyParserConfig = `{"llm_id":"Qwen/Qwen3-8B@qq@SILICONFLOW","raptor":{"use_raptor":true,"max_token":256,"threshold":0.1,"max_cluster":64,"random_seed":0},"graphrag":{"method":"light","use_graphrag":true,"entity_types":["organization","person"],"retry_attempts":2},"delimiter":"\n","topn_tags":3,"html4excel":false,"chunk_token_num":512,"layout_recognize":"DeepDOC","auto_keywords":0,"auto_questions":0}`++func setupKnowledgebaseParserConfigDB(t *testing.T) *gorm.DB {+ t.Helper()++ db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{TranslateError: true})+ if err != nil {+ t.Fatalf("failed to open sqlite: %v", err)+ }+ if err := db.AutoMigrate(&entity.Knowledgebase{}, &entity.SystemSettings{}); err != nil {+ t.Fatalf("failed to migrate: %v", err)+ }+ return db+}++func insertKnowledgebase(t *testing.T, db *gorm.DB, id, parserConfig string) {+ t.Helper()+ if err := db.Exec("INSERT INTO knowledgebase (id, tenant_id, name, embd_id, created_by, parser_config) VALUES (?, ?, ?, ?, ?, ?)",+ id, "tenant-"+id, "kb-"+id, "embd-"+id, "user-"+id, parserConfig).Error; err != nil {+ t.Fatalf("failed to insert knowledgebase %s: %v", id, err)+ }+}++func knowledgebaseParserConfigRaw(t *testing.T, db *gorm.DB, id string) string {+ t.Helper()+ var raw string+ if err := db.Table("knowledgebase").Where("id = ?", id).Pluck("parser_config", &raw).Error; err != nil {+ t.Fatalf("failed to read parser config %s: %v", id, err)+ }+ return raw+}++func knowledgebaseParserConfig(t *testing.T, db *gorm.DB, id string) map[string]interface{} {+ t.Helper()+ raw := knowledgebaseParserConfigRaw(t, db, id)+ config := make(map[string]interface{})+ if err := json.Unmarshal([]byte(raw), &config); err != nil {+ t.Fatalf("failed to parse parser config %s: %v", id, err)+ }+ return config+}++func TestMigrateKnowledgebaseParserConfigDropsRaptorAndGraphrag(t *testing.T) {+ db := setupKnowledgebaseParserConfigDB(t)+ insertKnowledgebase(t, db, "kb1", legacyParserConfig)++ if err := migrateKnowledgebaseParserConfig(t.Context(), db); err != nil {+ t.Fatalf("migrateKnowledgebaseParserConfig: %v", err)+ }++ config := knowledgebaseParserConfig(t, db, "kb1")+ for _, key := range removedParserConfigKeys {+ if _, ok := config[key]; ok {+ t.Fatalf("%s survived the migration: %#v", key, config)+ }+ }+ if got := config["llm_id"]; got != "Qwen/Qwen3-8B@qq@SILICONFLOW" {+ t.Fatalf("llm_id = %v, want the original value", got)+ }+ if got := config["chunk_token_num"]; got != float64(512) {+ t.Fatalf("chunk_token_num = %v, want 512", got)+ }+ if got := config["layout_recognize"]; got != "DeepDOC" {+ t.Fatalf("layout_recognize = %v, want DeepDOC", got)+ }+ if got := databaseVersion(t, db); got != knowledgebaseParserConfigTargetVersion {+ t.Fatalf("database version = %q, want %q", got, knowledgebaseParserConfigTargetVersion)+ }+}++func TestMigrateKnowledgebaseParserConfigKeepsCleanConfig(t *testing.T) {+ clean := `{"llm_id":"model","chunk_token_num":128}`+ db := setupKnowledgebaseParserConfigDB(t)+ insertKnowledgebase(t, db, "kb1", clean)++ if err := migrateKnowledgebaseParserConfig(t.Context(), db); err != nil {+ t.Fatalf("migrateKnowledgebaseParserConfig: %v", err)+ }++ config := knowledgebaseParserConfig(t, db, "kb1")+ if len(config) != 2 || config["llm_id"] != "model" {+ t.Fatalf("clean config changed: %#v", config)+ }+ if got := databaseVersion(t, db); got != knowledgebaseParserConfigTargetVersion {+ t.Fatalf("database version = %q, want %q", got, knowledgebaseParserConfigTargetVersion)+ }+}++func TestMigrateKnowledgebaseParserConfigSkipsMigratedDatabase(t *testing.T) {+ db := setupKnowledgebaseParserConfigDB(t)+ insertKnowledgebase(t, db, "kb1", legacyParserConfig)+ writeDatabaseVersion(t, db, knowledgebaseParserConfigTargetVersion)++ if err := migrateKnowledgebaseParserConfig(t.Context(), db); err != nil {+ t.Fatalf("migrateKnowledgebaseParserConfig: %v", err)+ }++ config := knowledgebaseParserConfig(t, db, "kb1")+ if _, ok := config["raptor"]; !ok {+ t.Fatalf("raptor removed on an already migrated database: %#v", config)+ }+}++func TestMigrateKnowledgebaseParserConfigToleratesMalformedPayload(t *testing.T) {+ db := setupKnowledgebaseParserConfigDB(t)+ insertKnowledgebase(t, db, "kb1", "not-json")+ insertKnowledgebase(t, db, "kb2", legacyParserConfig)+ insertKnowledgebase(t, db, "kb3", "")++ if err := migrateKnowledgebaseParserConfig(t.Context(), db); err != nil {+ t.Fatalf("migrateKnowledgebaseParserConfig: %v", err)+ }++ // A payload the step cannot parse is left exactly as it was stored.+ if got := knowledgebaseParserConfigRaw(t, db, "kb1"); got != "not-json" {+ t.Fatalf("malformed config rewritten as %q", got)+ }+ if got := knowledgebaseParserConfigRaw(t, db, "kb3"); got != "" {+ t.Fatalf("empty config rewritten as %q", got)+ }+ config := knowledgebaseParserConfig(t, db, "kb2")+ if _, ok := config["graphrag"]; ok {+ t.Fatalf("graphrag survived the migration: %#v", config)+ }+ if got := databaseVersion(t, db); got != knowledgebaseParserConfigTargetVersion {+ t.Fatalf("database version = %q, want %q", got, knowledgebaseParserConfigTargetVersion)+ }+}++func TestMigrateKnowledgebaseParserConfigWithoutTable(t *testing.T) {+ db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{TranslateError: true})+ if err != nil {+ t.Fatalf("failed to open sqlite: %v", err)+ }+ if err := db.AutoMigrate(&entity.SystemSettings{}); err != nil {+ t.Fatalf("failed to migrate: %v", err)+ }++ if err := migrateKnowledgebaseParserConfig(t.Context(), db); err != nil {+ t.Fatalf("migrateKnowledgebaseParserConfig: %v", err)+ }+ if got := databaseVersion(t, db); got != "" {+ t.Fatalf("database version = %q, want it left unset", got)+ }+}diff --git a/internal/dao/migration.go b/internal/dao/migration.goindex 28f8621411..da661a0e5c 100644--- a/internal/dao/migration.go+++ b/internal/dao/migration.go@@ -18,6 +18,7 @@ package daoimport ("context"+ "encoding/json""fmt""ragflow/internal/common""ragflow/internal/entity"@@ -492,6 +493,120 @@ func migrateUserCanvasTitleUnique(ctx context.Context, db *gorm.DB) error {return nil}+// knowledgebaseParserConfigTargetVersion is the database version a completed+// knowledgebase parser config cleanup brings the database to. It is stored in+// system_settings under mysql_migration.database.version, the marker shared with+// the other versioned steps, so work already performed by a previous run is+// skipped.+const knowledgebaseParserConfigTargetVersion = "v1.0.0-rc2.dev1"++// removedParserConfigKeys are the top level knowledgebase parser config keys the+// v1.0.0-rc2.dev1 step deletes. raptor and graphrag moved out of the dataset+// parser config, so the stored sections are dead data.+var removedParserConfigKeys = []string{"raptor", "graphrag"}++// migrateKnowledgebaseParserConfig deletes the retired raptor and graphrag+// sections from knowledgebase.parser_config, leaving every other key of the+// stored JSON untouched.+//+// It runs after AutoMigrate: the shared version marker is monotonic, and this+// step's version sorts above the conversation split's, so running it any+// earlier would mark that split as done on a database that has not run it yet.+func migrateKnowledgebaseParserConfig(ctx context.Context, db *gorm.DB) error {+ currentVersion, err := GetDatabaseMigrationVersion(ctx, db)+ if err != nil {+ return fmt.Errorf("read database migration version: %w", err)+ }+ if shouldSkipMigration(currentVersion, knowledgebaseParserConfigTargetVersion) {+ common.Info("Knowledgebase parser configs already cleaned, skipping",+ zap.String("current_version", currentVersion),+ zap.String("target_version", knowledgebaseParserConfigTargetVersion))+ return nil+ }+ if !db.WithContext(ctx).Migrator().HasTable("knowledgebase") {+ return nil+ }++ common.Info("Running knowledgebase parser config migration",+ zap.String("current_version", currentVersion),+ zap.String("target_version", knowledgebaseParserConfigTargetVersion))+ migrated, err := removeParserConfigKeys(ctx, db)+ if err != nil {+ return err+ }+ if err := setDatabaseMigrationVersion(ctx, db, knowledgebaseParserConfigTargetVersion); err != nil {+ return fmt.Errorf("mark database migration version %s: %w", knowledgebaseParserConfigTargetVersion, err)+ }+ common.Info("Knowledgebase parser config migration completed",+ zap.Int("knowledgebases", migrated),+ zap.String("version", knowledgebaseParserConfigTargetVersion))+ return nil+}++// knowledgebaseParserConfigRow reads one knowledgebase row's parser config as+// raw text, so a malformed payload is skipped instead of failing the stage.+type knowledgebaseParserConfigRow struct {+ ID string `gorm:"column:id"`+ ParserConfig *string `gorm:"column:parser_config"`+}++func removeParserConfigKeys(ctx context.Context, db *gorm.DB) (int, error) {+ var migrated int+ var rows []knowledgebaseParserConfigRow+ result := db.WithContext(ctx).Model(&entity.Knowledgebase{}).+ Select("id", "parser_config").Order("id").+ FindInBatches(&rows, parserConfigMigrationBatchSize, func(batchDB *gorm.DB, _ int) error {+ return batchDB.Transaction(func(tx *gorm.DB) error {+ for _, row := range rows {+ config, ok := parseParserConfig(row.ParserConfig)+ if !ok || !dropRemovedParserConfigKeys(config) {+ continue+ }+ if err := tx.Model(&entity.Knowledgebase{}).Where("id = ?", row.ID).+ Update("parser_config", entity.JSONMap(config)).Error; err != nil {+ return fmt.Errorf("update knowledgebase parser config %q: %w", row.ID, err)+ }+ migrated+++ }+ return nil+ })+ })+ if result.Error != nil {+ return migrated, fmt.Errorf("read knowledgebase parser configs: %w", result.Error)+ }+ return migrated, nil+}++// parseParserConfig decodes one stored parser config. It reports false for a+// missing, empty or non-object payload, which no cleanup can apply to.+func parseParserConfig(raw *string) (map[string]interface{}, bool) {+ if raw == nil {+ return nil, false+ }+ trimmed := strings.TrimSpace(*raw)+ if trimmed == "" {+ return nil, false+ }+ config := make(map[string]interface{})+ if err := json.Unmarshal([]byte(trimmed), &config); err != nil {+ return nil, false+ }+ return config, true+}++// dropRemovedParserConfigKeys deletes the retired sections and reports whether+// the config changed, so only rows that actually carry them are rewritten.+func dropRemovedParserConfigKeys(config map[string]interface{}) bool {+ removed := false+ for _, key := range removedParserConfigKeys {+ if _, ok := config[key]; ok {+ delete(config, key)+ removed = true+ }+ }+ return removed+}+// modifyColumnTypes aligns columns whose stored type must match the Python// models in api/db/db_models.py. The target types are also declared on the Go// entities, so AutoMigrate converges on the same definition and the two paths