Skip to content

RAGFlow is an agent-native database: a Retrieval-Augmented Generation (RAG) engine that grew into context infrastructure.

Fix: not drop uncreated unique index (#20118)

Lynn3af32e6f406d
diff --git a/AGENTS.md b/AGENTS.md
index d5102485e1..e506e490d3 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -51,6 +51,20 @@ Use this file as the local operating guide for the current codebase. Prefer the
- Remove commented-out Go code instead of leaving recovery notes in place.
- Keep package comments and doc comments aligned with the current runtime path, not with migration history.
+## Shared database schema (Go + Python)
+The Go services and the Python API write to the same MySQL/PostgreSQL schema, and both own parts of it. Neither is authoritative over the whole.
+
+Each ORM derives index names differently:
+- Go (`internal/entity/`): GORM derives `idx_<table>_<column>` for indexes and `uni_<table>_<column>` for unique constraints. Tags in this repo name unique indexes explicitly, e.g. `uniqueIndex:idx_commit_file`.
+- Python (`api/db/db_models.py`): Peewee derives `<db_table>_<col1>_<col2>` (truncated with an md5 suffix past 64 chars, see `playhouse.migrate.make_index_name`). There is no `idx_` prefix.
+
+Both sides reconcile by **column set, not by name**: Go's `hasUniqueIndex` (`internal/dao/migration.go`) accepts an existing index only when its full column set equals the requested one, and Python's `ensure_model_indexes` (`api/db/db_models.py`) keys `DB.get_indexes()` on the column tuple. Consequences:
+
+- Divergent names covering the same columns are expected, not a defect. Whichever runtime reaches the database first decides the name. Do not "fix" this by renaming inside a migration; it rewrites live databases for no functional gain.
+- Add a shared unique index to **both** sides (Go tag + Peewee `Meta.indexes` or `unique=True`), so each runtime recreates it when absent. Do not add a second index over a column set an existing one already covers.
+- In Go, declare single-column uniqueness with a named `uniqueIndex:`, not a bare `unique` tag. A bare `unique` becomes a table-level constraint in DDL, which SQLite materializes as an implicit index that cannot be dropped by name, and which GORM's `MigrateColumnUnique` then tries to drop under its own `uni_<table>_<column>` name — MySQL answers 1091 every startup. `namedIndexMigrator` (`internal/dao/database.go`) suppresses exactly that phantom drop; see `TestNamedIndexMigratorSkipsPhantomUniqueDrop`.
+- A Python model existing does not mean Python owns that table's constraints. `pipeline_operation_log` has no `run_count` column and no unique index in `db_models.py`; both are added only by `internal/dao/migration.go`.
+
## Go Test Tiers
Go tests are classified by build tag so the default `go test ./...` run stays self-contained. Tag a test file with `//go:build <tier>` placed before the `package` clause.
diff --git a/internal/dao/database.go b/internal/dao/database.go
index a458177e70..a7ce337a78 100644
--- a/internal/dao/database.go
+++ b/internal/dao/database.go
@@ -35,12 +35,85 @@ import (
"gorm.io/driver/mysql"
"gorm.io/gorm"
+ "gorm.io/gorm/migrator"
+ "gorm.io/gorm/schema"
)
var DB *gorm.DB
var modelProviderManager *models.ProviderManager
var modelProviderManagerMu sync.Mutex
+// migrationAwareDialector hands out a namedIndexMigrator instead of the stock
+// one.
+//
+// Both wrappers embed the concrete driver types, not gorm's interfaces. Embedding
+// gorm.Dialector or gorm.Migrator promotes only the methods those interfaces
+// declare, and gorm reaches capabilities such as ErrorTranslator,
+// SavePointerDialectorInterface and BuildIndexOptionsInterface by type
+// assertion on db.Dialector and db.Migrator() instead of through the
+// interfaces. The storage driver exposes twenty migrator methods beyond
+// gorm.Migrator, so forwarding them by hand loses one on every upgrade --
+// usually surfacing as a runtime panic mid-migration.
+type migrationAwareDialector struct {
+ *mysql.Dialector
+}
+
+// The capabilities gorm discovers by type assertion on db.Dialector.
+var (
+ _ gorm.Dialector = migrationAwareDialector{}
+ _ gorm.ErrorTranslator = migrationAwareDialector{}
+ _ gorm.SavePointerDialectorInterface = migrationAwareDialector{}
+)
+
+func newMigrationAwareDialector(dsn string) gorm.Dialector {
+ base := mysql.Open(dsn)
+ if dialector, ok := base.(*mysql.Dialector); ok {
+ return migrationAwareDialector{Dialector: dialector}
+ }
+ return base
+}
+
+func (d migrationAwareDialector) Migrator(db *gorm.DB) gorm.Migrator {
+ migrator, ok := d.Dialector.Migrator(db).(mysql.Migrator)
+ if !ok {
+ // An unsupported driver shape degrades to the stock behaviour rather
+ // than panicking: a foreign migrator means we get the redundant drops
+ // back, which is what shipped before this wrapper existed.
+ return d.Dialector.Migrator(db)
+ }
+ return namedIndexMigrator{Migrator: migrator}
+}
+
+// namedIndexMigrator leaves uniqueness to the named indexes declared with
+// uniqueIndex tags and created by the manual migrations.
+type namedIndexMigrator struct {
+ mysql.Migrator
+}
+
+var _ migrator.BuildIndexOptionsInterface = namedIndexMigrator{}
+
+// MigrateColumnUnique drops a unique constraint only once it exists. The stock
+// implementation equates "this column carries some single-column UNIQUE index"
+// with "this column carries a UNIQUE constraint", derives the matching default
+// name (uni_<table>_<column>) and drops it. Our named indexes are created as
+// indexes and never under that name, so the DROP always targets a missing
+// object and MySQL answers 1091 -- once per column, on every startup. The
+// add-constraint branch is left alone: it names its own object, so it cannot
+// hit the same mismatch.
+// phantomUniqueDrop reports whether the stock migrator is about to drop a
+// unique constraint this schema never created. See MigrateColumnUnique.
+func phantomUniqueDrop(field *schema.Field, columnType gorm.ColumnType) bool {
+ unique, _ := columnType.Unique()
+ return unique && !field.Unique
+}
+
+func (m namedIndexMigrator) MigrateColumnUnique(dst interface{}, field *schema.Field, columnType gorm.ColumnType) error {
+ if phantomUniqueDrop(field, columnType) {
+ return nil
+ }
+ return m.Migrator.MigrateColumnUnique(dst, field, columnType)
+}
+
// LLMFactoryConfig represents a single LLM factory configuration
type LLMFactoryConfig struct {
Name string `json:"name"`
@@ -89,7 +162,7 @@ func InitDB(ctx context.Context, migrateDB bool) error {
// Connect to database
var err error
- DB, err = gorm.Open(mysql.Open(dsn), &gorm.Config{
+ DB, err = gorm.Open(newMigrationAwareDialector(dsn), &gorm.Config{
Logger: gormLogger.Default.LogMode(gormLogLevel),
NowFunc: func() time.Time {
return time.Now().Local()
diff --git a/internal/dao/database_test.go b/internal/dao/database_test.go
index 880a98a4a4..3ef0319ef8 100644
--- a/internal/dao/database_test.go
+++ b/internal/dao/database_test.go
@@ -19,13 +19,16 @@ package dao
import (
"context"
"errors"
+ "reflect"
"strings"
"testing"
"ragflow/internal/entity"
"github.com/glebarez/sqlite"
+ "gorm.io/driver/mysql"
"gorm.io/gorm"
+ "gorm.io/gorm/schema"
)
func TestAutoMigrateRuntimeModelsCreatesGoRuntimeTables(t *testing.T) {
@@ -240,6 +243,81 @@ func TestCreateIngestionLogSchemaObjectsRechecksAfterCreateError(t *testing.T) {
}
}
+// uniqueColumnType is a gorm.ColumnType that answers Unique() and nothing
+// else, which is all MigrateColumnUnique branches on.
+type uniqueColumnType struct {
+ unique bool
+}
+
+func (c uniqueColumnType) Name() string { return "document_id" }
+func (c uniqueColumnType) DatabaseTypeName() string { return "VARCHAR" }
+func (c uniqueColumnType) ColumnType() (string, bool) { return "varchar(32)", true }
+func (c uniqueColumnType) PrimaryKey() (bool, bool) { return false, true }
+func (c uniqueColumnType) AutoIncrement() (bool, bool) { return false, true }
+func (c uniqueColumnType) Length() (int64, bool) { return 32, true }
+func (c uniqueColumnType) DecimalSize() (int64, int64, bool) { return 0, 0, false }
+func (c uniqueColumnType) Nullable() (bool, bool) { return false, true }
+func (c uniqueColumnType) Unique() (bool, bool) { return c.unique, true }
+func (c uniqueColumnType) ScanType() reflect.Type { return reflect.TypeOf("") }
+func (c uniqueColumnType) Comment() (string, bool) { return "", false }
+func (c uniqueColumnType) DefaultValue() (string, bool) { return "", false }
+
+// TestNamedIndexMigratorSkipsPhantomUniqueDrop pins the behaviour that kept
+// failing every startup on MySQL: a unique index created as an index is not a
+// column-level unique constraint, so nothing exists to drop. The other three
+// combinations must fall through to the stock migrator, which decides whether
+// the constraint it names has to be created.
+func TestNamedIndexMigratorSkipsPhantomUniqueDrop(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ unique bool
+ tag bool
+ wantSkip bool
+ }{
+ {"named index with no unique tag", true, false, true},
+ {"column is not unique", false, false, false},
+ {"unique tag already set", false, true, false},
+ {"index and tag agree", true, true, false},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ got := phantomUniqueDrop(&schema.Field{Unique: tc.tag}, uniqueColumnType{unique: tc.unique})
+ if got != tc.wantSkip {
+ t.Fatalf("phantomUniqueDrop(unique=%v, tag=%v) = %v, want %v", tc.unique, tc.tag, got, tc.wantSkip)
+ }
+ })
+ }
+}
+
+// TestWrappersExposeEverythingWrappedTypeDoes is the guard that made the
+// concrete embedding necessary: embedding gorm's interfaces instead would drop
+// the twenty migrator methods and the three dialector methods that gorm reaches
+// by type assertion, and losing one surfaces as a panic part-way through
+// AutoMigrate. Embedding the concrete types promotes all of them by
+// construction; this catches a future refactor that reverts to the interfaces.
+func TestWrappersExposeEverythingWrappedTypeDoes(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ wrapped reflect.Type
+ wrapper reflect.Type
+ override string
+ }{
+ {"dialector", reflect.TypeOf(&mysql.Dialector{}), reflect.TypeOf(migrationAwareDialector{}), "Migrator"},
+ {"migrator", reflect.TypeOf(mysql.Migrator{}), reflect.TypeOf(namedIndexMigrator{}), "MigrateColumnUnique"},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ for i := 0; i < tc.wrapped.NumMethod(); i++ {
+ name := tc.wrapped.Method(i).Name
+ if name == tc.override {
+ continue
+ }
+ if _, ok := tc.wrapper.MethodByName(name); !ok {
+ t.Errorf("wrapper does not expose %s; gorm reaches some capabilities by type assertion on the wrapped type rather than through its interfaces", name)
+ }
+ }
+ })
+ }
+}
+
func TestAutoMigrateRuntimeModelsNamesFailingTable(t *testing.T) {
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
if err != nil {
diff --git a/internal/dao/migration.go b/internal/dao/migration.go
index 37dcb839f5..c2efad68cd 100644
--- a/internal/dao/migration.go
+++ b/internal/dao/migration.go
@@ -253,18 +253,11 @@ func migrateIngestionTaskDocumentIDUnique(ctx context.Context, db *gorm.DB) erro
// migrateIngestionTaskPipelineLogID adds ingestion_task.pipeline_log_id when
// the column is missing.
//
-// It cannot be left to AutoMigrate. The model declares document_id's unique key
-// as a named unique index (uniqueIndex:idx_ingestion_task_document_id), while
-// the column is UNIQUE at the database level. GORM's MigrateColumnUnique reads
-// that pairing as a stray unique constraint: it issues
-// `ALTER TABLE ingestion_task DROP FOREIGN KEY uni_ingestion_task_document_id`
-// for the default constraint name, which does not exist, so MySQL fails with
-// 1091. AutoMigrate aborts on the first error, and because autoMigrateSafely
-// treats 1091 as a benign "already dropped" case, the failure is swallowed and
-// the migration reports success -- before ever reaching the missing column.
-// A column added to this table would therefore never be created, and every
-// ingestion_task query (the DAO selects all columns) would fail with
-// Error 1054.
+// It runs after AutoMigrate rather than being left to it. AutoMigrate reaches a
+// new column only once every earlier column has converged, so any failure on
+// the way (a duplicate key, a rejected ALTER) silently skips it, and every
+// ingestion_task query selects all columns: a missing column then fails the
+// whole API with Error 1054.
func migrateIngestionTaskPipelineLogID(ctx context.Context, db *gorm.DB) error {
// Use the model, not the bare table name: HasColumn resolves the field
// against the statement schema and dereferences it, which a plain string
diff --git a/internal/entity/file_commit.go b/internal/entity/file_commit.go
index 1f262180a7..21c08216cb 100644
--- a/internal/entity/file_commit.go
+++ b/internal/entity/file_commit.go
@@ -38,6 +38,12 @@ func (FileCommit) TableName() string {
}
// FileCommitItem represents a single file change within a commit.
+//
+// Python declares the same pair unique and Peewee names it
+// file_commit_item_commit_id_file_id. Both runtimes match an existing index by
+// column set rather than by name, so the two names are equivalent and one of
+// them appearing in the database is not a bug. See the "Shared database schema"
+// section of AGENTS.md before renaming either.
type FileCommitItem struct {
ID string `gorm:"column:id;primaryKey;size:32" json:"id"`
Seq uint `gorm:"column:seq;index" json:"seq,omitempty"`