Skip to content

RAGFlow is an agent-native database: a Retrieval-Augmented Generation (RAG) engine that grew into context infrastructure.

Fix: remove raptor and graphrag when migrate parse config (#20599)

Lynn1413972e09a2
diff --git a/docs/administrator/migration/database_change_log.md b/docs/administrator/migration/database_change_log.md
index 8b7591961e..919e8d575c 100644
--- a/docs/administrator/migration/database_change_log.md
+++ b/docs/administrator/migration/database_change_log.md
@@ -16,7 +16,7 @@ A version-by-version record of the schema and data changes the Go backend applie
- The marker is a row in `system_settings` with the key `mysql_migration.database.version`.
- The marker prevents a migration step from running again after that step has completed successfully.
-- Go defines the versions as constants: `modelMigrationBaseVersion` and `modelMigrationTargetVersion` in [migration_version.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/migration_version.go), and `conversationHistoryTargetVersion` in [conversation_history_migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/conversation_history_migration.go).
+- Go defines the versions as constants: `modelMigrationBaseVersion` and `modelMigrationTargetVersion` in [migration_version.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/migration_version.go), `conversationHistoryTargetVersion` in [conversation_history_migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/conversation_history_migration.go), and `knowledgebaseParserConfigTargetVersion` in [migration.go](https://github.com/infiniflow/ragflow/blob/main/internal/dao/migration.go).
- A step runs when the stored version is lower than the step's version. Versions are compared with Python PEP 440 semantics, where a `dev` prerelease sorts *before* the release it precedes (`v1.0.0-rc1.dev1` < `v1.0.0-rc1`). A missing or unparsable marker never skips a step.
- `checkDatabaseVersion` in [cmd/ragflow_server.go](https://github.com/infiniflow/ragflow/blob/main/cmd/ragflow_server.go) refuses to start when the code version is older than the stored database version, so a database migrated by a newer release cannot be opened by an older one.
@@ -26,6 +26,7 @@ Startup order in `InitDB` ([database.go](https://github.com/infiniflow/ragflow/b
2. `migrateIngestionLogRunIdentity` — ingestion log columns and indexes.
3. `AutoMigrate` — converges every ORM model.
4. `migrateConversationHistory` — runs *after* `AutoMigrate`, which creates the child tables it backfills.
+5. `migrateKnowledgebaseParserConfig` — runs last, so the marker does not move past the conversation split's version before that split has run.
## Version overview
@@ -34,6 +35,7 @@ Startup order in `InitDB` ([database.go](https://github.com/infiniflow/ragflow/b
| `v0.26.0` | Rebuild the tenant model tables from `tenant_llm` and normalize stored model ids | `modelMigrationBaseVersion` |
| `v1.0.0-rc1` | Seed factory-declared models, merge `model_type` into an integer bitmask, populate the `tenant_*_id` columns | `modelMigrationTargetVersion` |
| `v1.0.0-rc1.dev1` | Split conversation message and reference payloads into child tables | `conversationHistoryTargetVersion` |
+| `v1.0.0-rc2.dev1` | Delete the retired `raptor` and `graphrag` sections from `knowledgebase.parser_config` | `knowledgebaseParserConfigTargetVersion` |
The two tenant model steps are cumulative: a database at `v0.26.0` still runs the `v1.0.0-rc1` step, and a database at or above `v1.0.0-rc1` runs neither.
@@ -122,6 +124,27 @@ On success the step writes `v1.0.0-rc1.dev1` into the version marker.
---
+## v1.0.0-rc2.dev1
+
+Retires the `raptor` and `graphrag` sections of the dataset parser config. Both moved out of `knowledgebase.parser_config`, so the stored sections are dead data no code path reads any more.
+
+### Data migration
+
+| Table | Change |
+|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `knowledgebase` | `parser_config` is parsed as a JSON object; the top level `raptor` and `graphrag` keys are deleted and the remaining object is written back to the same column. |
+
+Execution details:
+
+- Rows are read in id-ordered batches of 256, each batch in its own transaction, so an interrupted run resumes from the last committed batch.
+- Only rows whose config actually carries one of the keys are rewritten; a row without them is left byte-identical.
+- A `NULL`, empty or non-object payload is skipped rather than failing the step.
+- The step returns without touching the marker when the `knowledgebase` table does not exist.
+
+On success the step writes `v1.0.0-rc2.dev1` into the version marker.
+
+---
+
## Version-less steps
These steps are not gated by the version marker. They are idempotent and run on every startup, either through `RunMigrations` or through the ingestion helpers in `InitDB`.
diff --git a/internal/dao/database.go b/internal/dao/database.go
index f5593462d6..4587596eae 100644
--- a/internal/dao/database.go
+++ b/internal/dao/database.go
@@ -246,6 +246,14 @@ func InitDB(ctx context.Context, migrateDB bool) error {
if err = migrateConversationHistory(ctx, DB); err != nil {
return fmt.Errorf("failed to migrate conversation history: %w", err)
}
+ // Drop the retired raptor and graphrag sections from the dataset parser
+ // configs. It runs after the conversation split because the two steps
+ // share one version marker: this step's version sorts above the split's
+ // v1.0.0-rc1, so marking it earlier would skip a split that has not run.
+ if err = migrateKnowledgebaseParserConfig(ctx, DB); err != nil {
+ return fmt.Errorf("failed to migrate knowledgebase parser config: %w", err)
+ }
+
if err = migrateTenantModelMaxTokens(ctx, DB); err != nil {
return fmt.Errorf("failed to backfill tenant model max_tokens: %w", err)
}
diff --git a/internal/dao/knowledgebase_parser_config_test.go b/internal/dao/knowledgebase_parser_config_test.go
new file mode 100644
index 0000000000..0550a5b93d
--- /dev/null
+++ b/internal/dao/knowledgebase_parser_config_test.go
@@ -0,0 +1,174 @@
+//
+// Copyright 2026 The InfiniFlow Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package dao
+
+import (
+ "encoding/json"
+ "testing"
+
+ "github.com/glebarez/sqlite"
+ "gorm.io/gorm"
+
+ "ragflow/internal/entity"
+)
+
+// legacyParserConfig is a dataset parser config as written by a release that
+// still carried the raptor and graphrag sections.
+const legacyParserConfig = `{"llm_id":"Qwen/Qwen3-8B@qq@SILICONFLOW","raptor":{"use_raptor":true,"max_token":256,"threshold":0.1,"max_cluster":64,"random_seed":0},"graphrag":{"method":"light","use_graphrag":true,"entity_types":["organization","person"],"retry_attempts":2},"delimiter":"\n","topn_tags":3,"html4excel":false,"chunk_token_num":512,"layout_recognize":"DeepDOC","auto_keywords":0,"auto_questions":0}`
+
+func setupKnowledgebaseParserConfigDB(t *testing.T) *gorm.DB {
+ t.Helper()
+
+ db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{TranslateError: true})
+ if err != nil {
+ t.Fatalf("failed to open sqlite: %v", err)
+ }
+ if err := db.AutoMigrate(&entity.Knowledgebase{}, &entity.SystemSettings{}); err != nil {
+ t.Fatalf("failed to migrate: %v", err)
+ }
+ return db
+}
+
+func insertKnowledgebase(t *testing.T, db *gorm.DB, id, parserConfig string) {
+ t.Helper()
+ if err := db.Exec("INSERT INTO knowledgebase (id, tenant_id, name, embd_id, created_by, parser_config) VALUES (?, ?, ?, ?, ?, ?)",
+ id, "tenant-"+id, "kb-"+id, "embd-"+id, "user-"+id, parserConfig).Error; err != nil {
+ t.Fatalf("failed to insert knowledgebase %s: %v", id, err)
+ }
+}
+
+func knowledgebaseParserConfigRaw(t *testing.T, db *gorm.DB, id string) string {
+ t.Helper()
+ var raw string
+ if err := db.Table("knowledgebase").Where("id = ?", id).Pluck("parser_config", &raw).Error; err != nil {
+ t.Fatalf("failed to read parser config %s: %v", id, err)
+ }
+ return raw
+}
+
+func knowledgebaseParserConfig(t *testing.T, db *gorm.DB, id string) map[string]interface{} {
+ t.Helper()
+ raw := knowledgebaseParserConfigRaw(t, db, id)
+ config := make(map[string]interface{})
+ if err := json.Unmarshal([]byte(raw), &config); err != nil {
+ t.Fatalf("failed to parse parser config %s: %v", id, err)
+ }
+ return config
+}
+
+func TestMigrateKnowledgebaseParserConfigDropsRaptorAndGraphrag(t *testing.T) {
+ db := setupKnowledgebaseParserConfigDB(t)
+ insertKnowledgebase(t, db, "kb1", legacyParserConfig)
+
+ if err := migrateKnowledgebaseParserConfig(t.Context(), db); err != nil {
+ t.Fatalf("migrateKnowledgebaseParserConfig: %v", err)
+ }
+
+ config := knowledgebaseParserConfig(t, db, "kb1")
+ for _, key := range removedParserConfigKeys {
+ if _, ok := config[key]; ok {
+ t.Fatalf("%s survived the migration: %#v", key, config)
+ }
+ }
+ if got := config["llm_id"]; got != "Qwen/Qwen3-8B@qq@SILICONFLOW" {
+ t.Fatalf("llm_id = %v, want the original value", got)
+ }
+ if got := config["chunk_token_num"]; got != float64(512) {
+ t.Fatalf("chunk_token_num = %v, want 512", got)
+ }
+ if got := config["layout_recognize"]; got != "DeepDOC" {
+ t.Fatalf("layout_recognize = %v, want DeepDOC", got)
+ }
+ if got := databaseVersion(t, db); got != knowledgebaseParserConfigTargetVersion {
+ t.Fatalf("database version = %q, want %q", got, knowledgebaseParserConfigTargetVersion)
+ }
+}
+
+func TestMigrateKnowledgebaseParserConfigKeepsCleanConfig(t *testing.T) {
+ clean := `{"llm_id":"model","chunk_token_num":128}`
+ db := setupKnowledgebaseParserConfigDB(t)
+ insertKnowledgebase(t, db, "kb1", clean)
+
+ if err := migrateKnowledgebaseParserConfig(t.Context(), db); err != nil {
+ t.Fatalf("migrateKnowledgebaseParserConfig: %v", err)
+ }
+
+ config := knowledgebaseParserConfig(t, db, "kb1")
+ if len(config) != 2 || config["llm_id"] != "model" {
+ t.Fatalf("clean config changed: %#v", config)
+ }
+ if got := databaseVersion(t, db); got != knowledgebaseParserConfigTargetVersion {
+ t.Fatalf("database version = %q, want %q", got, knowledgebaseParserConfigTargetVersion)
+ }
+}
+
+func TestMigrateKnowledgebaseParserConfigSkipsMigratedDatabase(t *testing.T) {
+ db := setupKnowledgebaseParserConfigDB(t)
+ insertKnowledgebase(t, db, "kb1", legacyParserConfig)
+ writeDatabaseVersion(t, db, knowledgebaseParserConfigTargetVersion)
+
+ if err := migrateKnowledgebaseParserConfig(t.Context(), db); err != nil {
+ t.Fatalf("migrateKnowledgebaseParserConfig: %v", err)
+ }
+
+ config := knowledgebaseParserConfig(t, db, "kb1")
+ if _, ok := config["raptor"]; !ok {
+ t.Fatalf("raptor removed on an already migrated database: %#v", config)
+ }
+}
+
+func TestMigrateKnowledgebaseParserConfigToleratesMalformedPayload(t *testing.T) {
+ db := setupKnowledgebaseParserConfigDB(t)
+ insertKnowledgebase(t, db, "kb1", "not-json")
+ insertKnowledgebase(t, db, "kb2", legacyParserConfig)
+ insertKnowledgebase(t, db, "kb3", "")
+
+ if err := migrateKnowledgebaseParserConfig(t.Context(), db); err != nil {
+ t.Fatalf("migrateKnowledgebaseParserConfig: %v", err)
+ }
+
+ // A payload the step cannot parse is left exactly as it was stored.
+ if got := knowledgebaseParserConfigRaw(t, db, "kb1"); got != "not-json" {
+ t.Fatalf("malformed config rewritten as %q", got)
+ }
+ if got := knowledgebaseParserConfigRaw(t, db, "kb3"); got != "" {
+ t.Fatalf("empty config rewritten as %q", got)
+ }
+ config := knowledgebaseParserConfig(t, db, "kb2")
+ if _, ok := config["graphrag"]; ok {
+ t.Fatalf("graphrag survived the migration: %#v", config)
+ }
+ if got := databaseVersion(t, db); got != knowledgebaseParserConfigTargetVersion {
+ t.Fatalf("database version = %q, want %q", got, knowledgebaseParserConfigTargetVersion)
+ }
+}
+
+func TestMigrateKnowledgebaseParserConfigWithoutTable(t *testing.T) {
+ db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{TranslateError: true})
+ if err != nil {
+ t.Fatalf("failed to open sqlite: %v", err)
+ }
+ if err := db.AutoMigrate(&entity.SystemSettings{}); err != nil {
+ t.Fatalf("failed to migrate: %v", err)
+ }
+
+ if err := migrateKnowledgebaseParserConfig(t.Context(), db); err != nil {
+ t.Fatalf("migrateKnowledgebaseParserConfig: %v", err)
+ }
+ if got := databaseVersion(t, db); got != "" {
+ t.Fatalf("database version = %q, want it left unset", got)
+ }
+}
diff --git a/internal/dao/migration.go b/internal/dao/migration.go
index 28f8621411..da661a0e5c 100644
--- a/internal/dao/migration.go
+++ b/internal/dao/migration.go
@@ -18,6 +18,7 @@ package dao
import (
"context"
+ "encoding/json"
"fmt"
"ragflow/internal/common"
"ragflow/internal/entity"
@@ -492,6 +493,120 @@ func migrateUserCanvasTitleUnique(ctx context.Context, db *gorm.DB) error {
return nil
}
+// knowledgebaseParserConfigTargetVersion is the database version a completed
+// knowledgebase parser config cleanup brings the database to. It is stored in
+// system_settings under mysql_migration.database.version, the marker shared with
+// the other versioned steps, so work already performed by a previous run is
+// skipped.
+const knowledgebaseParserConfigTargetVersion = "v1.0.0-rc2.dev1"
+
+// removedParserConfigKeys are the top level knowledgebase parser config keys the
+// v1.0.0-rc2.dev1 step deletes. raptor and graphrag moved out of the dataset
+// parser config, so the stored sections are dead data.
+var removedParserConfigKeys = []string{"raptor", "graphrag"}
+
+// migrateKnowledgebaseParserConfig deletes the retired raptor and graphrag
+// sections from knowledgebase.parser_config, leaving every other key of the
+// stored JSON untouched.
+//
+// It runs after AutoMigrate: the shared version marker is monotonic, and this
+// step's version sorts above the conversation split's, so running it any
+// earlier would mark that split as done on a database that has not run it yet.
+func migrateKnowledgebaseParserConfig(ctx context.Context, db *gorm.DB) error {
+ currentVersion, err := GetDatabaseMigrationVersion(ctx, db)
+ if err != nil {
+ return fmt.Errorf("read database migration version: %w", err)
+ }
+ if shouldSkipMigration(currentVersion, knowledgebaseParserConfigTargetVersion) {
+ common.Info("Knowledgebase parser configs already cleaned, skipping",
+ zap.String("current_version", currentVersion),
+ zap.String("target_version", knowledgebaseParserConfigTargetVersion))
+ return nil
+ }
+ if !db.WithContext(ctx).Migrator().HasTable("knowledgebase") {
+ return nil
+ }
+
+ common.Info("Running knowledgebase parser config migration",
+ zap.String("current_version", currentVersion),
+ zap.String("target_version", knowledgebaseParserConfigTargetVersion))
+ migrated, err := removeParserConfigKeys(ctx, db)
+ if err != nil {
+ return err
+ }
+ if err := setDatabaseMigrationVersion(ctx, db, knowledgebaseParserConfigTargetVersion); err != nil {
+ return fmt.Errorf("mark database migration version %s: %w", knowledgebaseParserConfigTargetVersion, err)
+ }
+ common.Info("Knowledgebase parser config migration completed",
+ zap.Int("knowledgebases", migrated),
+ zap.String("version", knowledgebaseParserConfigTargetVersion))
+ return nil
+}
+
+// knowledgebaseParserConfigRow reads one knowledgebase row's parser config as
+// raw text, so a malformed payload is skipped instead of failing the stage.
+type knowledgebaseParserConfigRow struct {
+ ID string `gorm:"column:id"`
+ ParserConfig *string `gorm:"column:parser_config"`
+}
+
+func removeParserConfigKeys(ctx context.Context, db *gorm.DB) (int, error) {
+ var migrated int
+ var rows []knowledgebaseParserConfigRow
+ result := db.WithContext(ctx).Model(&entity.Knowledgebase{}).
+ Select("id", "parser_config").Order("id").
+ FindInBatches(&rows, parserConfigMigrationBatchSize, func(batchDB *gorm.DB, _ int) error {
+ return batchDB.Transaction(func(tx *gorm.DB) error {
+ for _, row := range rows {
+ config, ok := parseParserConfig(row.ParserConfig)
+ if !ok || !dropRemovedParserConfigKeys(config) {
+ continue
+ }
+ if err := tx.Model(&entity.Knowledgebase{}).Where("id = ?", row.ID).
+ Update("parser_config", entity.JSONMap(config)).Error; err != nil {
+ return fmt.Errorf("update knowledgebase parser config %q: %w", row.ID, err)
+ }
+ migrated++
+ }
+ return nil
+ })
+ })
+ if result.Error != nil {
+ return migrated, fmt.Errorf("read knowledgebase parser configs: %w", result.Error)
+ }
+ return migrated, nil
+}
+
+// parseParserConfig decodes one stored parser config. It reports false for a
+// missing, empty or non-object payload, which no cleanup can apply to.
+func parseParserConfig(raw *string) (map[string]interface{}, bool) {
+ if raw == nil {
+ return nil, false
+ }
+ trimmed := strings.TrimSpace(*raw)
+ if trimmed == "" {
+ return nil, false
+ }
+ config := make(map[string]interface{})
+ if err := json.Unmarshal([]byte(trimmed), &config); err != nil {
+ return nil, false
+ }
+ return config, true
+}
+
+// dropRemovedParserConfigKeys deletes the retired sections and reports whether
+// the config changed, so only rows that actually carry them are rewritten.
+func dropRemovedParserConfigKeys(config map[string]interface{}) bool {
+ removed := false
+ for _, key := range removedParserConfigKeys {
+ if _, ok := config[key]; ok {
+ delete(config, key)
+ removed = true
+ }
+ }
+ return removed
+}
+
// modifyColumnTypes aligns columns whose stored type must match the Python
// models in api/db/db_models.py. The target types are also declared on the Go
// entities, so AutoMigrate converges on the same definition and the two paths