RAGFlow is an agent-native database: a Retrieval-Augmented Generation (RAG) engine that grew into context infrastructure.
Go agent-harnessagent-native-databaseagentic-aiagentic-retrievalagentic-searchaiai-agentscontext-enginecontext-engineeringcontext-managementharness-engineeringknowledge-compilationragretrieval-augmented-generationretrieval-harnesssearch-harness
Fix: not drop uncreated unique index (#20118)
Lynn3af32e6f406d
diff --git a/AGENTS.md b/AGENTS.mdindex d5102485e1..e506e490d3 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -51,6 +51,20 @@ Use this file as the local operating guide for the current codebase. Prefer the- Remove commented-out Go code instead of leaving recovery notes in place.- Keep package comments and doc comments aligned with the current runtime path, not with migration history.+## Shared database schema (Go + Python)+The Go services and the Python API write to the same MySQL/PostgreSQL schema, and both own parts of it. Neither is authoritative over the whole.++Each ORM derives index names differently:+- Go (`internal/entity/`): GORM derives `idx_<table>_<column>` for indexes and `uni_<table>_<column>` for unique constraints. Tags in this repo name unique indexes explicitly, e.g. `uniqueIndex:idx_commit_file`.+- Python (`api/db/db_models.py`): Peewee derives `<db_table>_<col1>_<col2>` (truncated with an md5 suffix past 64 chars, see `playhouse.migrate.make_index_name`). There is no `idx_` prefix.++Both sides reconcile by **column set, not by name**: Go's `hasUniqueIndex` (`internal/dao/migration.go`) accepts an existing index only when its full column set equals the requested one, and Python's `ensure_model_indexes` (`api/db/db_models.py`) keys `DB.get_indexes()` on the column tuple. Consequences:++- Divergent names covering the same columns are expected, not a defect. Whichever runtime reaches the database first decides the name. Do not "fix" this by renaming inside a migration; it rewrites live databases for no functional gain.+- Add a shared unique index to **both** sides (Go tag + Peewee `Meta.indexes` or `unique=True`), so each runtime recreates it when absent. Do not add a second index over a column set an existing one already covers.+- In Go, declare single-column uniqueness with a named `uniqueIndex:`, not a bare `unique` tag. A bare `unique` becomes a table-level constraint in DDL, which SQLite materializes as an implicit index that cannot be dropped by name, and which GORM's `MigrateColumnUnique` then tries to drop under its own `uni_<table>_<column>` name — MySQL answers 1091 every startup. `namedIndexMigrator` (`internal/dao/database.go`) suppresses exactly that phantom drop; see `TestNamedIndexMigratorSkipsPhantomUniqueDrop`.+- A Python model existing does not mean Python owns that table's constraints. `pipeline_operation_log` has no `run_count` column and no unique index in `db_models.py`; both are added only by `internal/dao/migration.go`.+## Go Test TiersGo tests are classified by build tag so the default `go test ./...` run stays self-contained. Tag a test file with `//go:build <tier>` placed before the `package` clause.diff --git a/internal/dao/database.go b/internal/dao/database.goindex a458177e70..a7ce337a78 100644--- a/internal/dao/database.go+++ b/internal/dao/database.go@@ -35,12 +35,85 @@ import ("gorm.io/driver/mysql""gorm.io/gorm"+ "gorm.io/gorm/migrator"+ "gorm.io/gorm/schema")var DB *gorm.DBvar modelProviderManager *models.ProviderManagervar modelProviderManagerMu sync.Mutex+// migrationAwareDialector hands out a namedIndexMigrator instead of the stock+// one.+//+// Both wrappers embed the concrete driver types, not gorm's interfaces. Embedding+// gorm.Dialector or gorm.Migrator promotes only the methods those interfaces+// declare, and gorm reaches capabilities such as ErrorTranslator,+// SavePointerDialectorInterface and BuildIndexOptionsInterface by type+// assertion on db.Dialector and db.Migrator() instead of through the+// interfaces. The storage driver exposes twenty migrator methods beyond+// gorm.Migrator, so forwarding them by hand loses one on every upgrade --+// usually surfacing as a runtime panic mid-migration.+type migrationAwareDialector struct {+ *mysql.Dialector+}++// The capabilities gorm discovers by type assertion on db.Dialector.+var (+ _ gorm.Dialector = migrationAwareDialector{}+ _ gorm.ErrorTranslator = migrationAwareDialector{}+ _ gorm.SavePointerDialectorInterface = migrationAwareDialector{}+)++func newMigrationAwareDialector(dsn string) gorm.Dialector {+ base := mysql.Open(dsn)+ if dialector, ok := base.(*mysql.Dialector); ok {+ return migrationAwareDialector{Dialector: dialector}+ }+ return base+}++func (d migrationAwareDialector) Migrator(db *gorm.DB) gorm.Migrator {+ migrator, ok := d.Dialector.Migrator(db).(mysql.Migrator)+ if !ok {+ // An unsupported driver shape degrades to the stock behaviour rather+ // than panicking: a foreign migrator means we get the redundant drops+ // back, which is what shipped before this wrapper existed.+ return d.Dialector.Migrator(db)+ }+ return namedIndexMigrator{Migrator: migrator}+}++// namedIndexMigrator leaves uniqueness to the named indexes declared with+// uniqueIndex tags and created by the manual migrations.+type namedIndexMigrator struct {+ mysql.Migrator+}++var _ migrator.BuildIndexOptionsInterface = namedIndexMigrator{}++// MigrateColumnUnique drops a unique constraint only once it exists. The stock+// implementation equates "this column carries some single-column UNIQUE index"+// with "this column carries a UNIQUE constraint", derives the matching default+// name (uni_<table>_<column>) and drops it. Our named indexes are created as+// indexes and never under that name, so the DROP always targets a missing+// object and MySQL answers 1091 -- once per column, on every startup. The+// add-constraint branch is left alone: it names its own object, so it cannot+// hit the same mismatch.+// phantomUniqueDrop reports whether the stock migrator is about to drop a+// unique constraint this schema never created. See MigrateColumnUnique.+func phantomUniqueDrop(field *schema.Field, columnType gorm.ColumnType) bool {+ unique, _ := columnType.Unique()+ return unique && !field.Unique+}++func (m namedIndexMigrator) MigrateColumnUnique(dst interface{}, field *schema.Field, columnType gorm.ColumnType) error {+ if phantomUniqueDrop(field, columnType) {+ return nil+ }+ return m.Migrator.MigrateColumnUnique(dst, field, columnType)+}+// LLMFactoryConfig represents a single LLM factory configurationtype LLMFactoryConfig struct {Name string `json:"name"`@@ -89,7 +162,7 @@ func InitDB(ctx context.Context, migrateDB bool) error {// Connect to databasevar err error- DB, err = gorm.Open(mysql.Open(dsn), &gorm.Config{+ DB, err = gorm.Open(newMigrationAwareDialector(dsn), &gorm.Config{Logger: gormLogger.Default.LogMode(gormLogLevel),NowFunc: func() time.Time {return time.Now().Local()diff --git a/internal/dao/database_test.go b/internal/dao/database_test.goindex 880a98a4a4..3ef0319ef8 100644--- a/internal/dao/database_test.go+++ b/internal/dao/database_test.go@@ -19,13 +19,16 @@ package daoimport ("context""errors"+ "reflect""strings""testing""ragflow/internal/entity""github.com/glebarez/sqlite"+ "gorm.io/driver/mysql""gorm.io/gorm"+ "gorm.io/gorm/schema")func TestAutoMigrateRuntimeModelsCreatesGoRuntimeTables(t *testing.T) {@@ -240,6 +243,81 @@ func TestCreateIngestionLogSchemaObjectsRechecksAfterCreateError(t *testing.T) {}}+// uniqueColumnType is a gorm.ColumnType that answers Unique() and nothing+// else, which is all MigrateColumnUnique branches on.+type uniqueColumnType struct {+ unique bool+}++func (c uniqueColumnType) Name() string { return "document_id" }+func (c uniqueColumnType) DatabaseTypeName() string { return "VARCHAR" }+func (c uniqueColumnType) ColumnType() (string, bool) { return "varchar(32)", true }+func (c uniqueColumnType) PrimaryKey() (bool, bool) { return false, true }+func (c uniqueColumnType) AutoIncrement() (bool, bool) { return false, true }+func (c uniqueColumnType) Length() (int64, bool) { return 32, true }+func (c uniqueColumnType) DecimalSize() (int64, int64, bool) { return 0, 0, false }+func (c uniqueColumnType) Nullable() (bool, bool) { return false, true }+func (c uniqueColumnType) Unique() (bool, bool) { return c.unique, true }+func (c uniqueColumnType) ScanType() reflect.Type { return reflect.TypeOf("") }+func (c uniqueColumnType) Comment() (string, bool) { return "", false }+func (c uniqueColumnType) DefaultValue() (string, bool) { return "", false }++// TestNamedIndexMigratorSkipsPhantomUniqueDrop pins the behaviour that kept+// failing every startup on MySQL: a unique index created as an index is not a+// column-level unique constraint, so nothing exists to drop. The other three+// combinations must fall through to the stock migrator, which decides whether+// the constraint it names has to be created.+func TestNamedIndexMigratorSkipsPhantomUniqueDrop(t *testing.T) {+ for _, tc := range []struct {+ name string+ unique bool+ tag bool+ wantSkip bool+ }{+ {"named index with no unique tag", true, false, true},+ {"column is not unique", false, false, false},+ {"unique tag already set", false, true, false},+ {"index and tag agree", true, true, false},+ } {+ t.Run(tc.name, func(t *testing.T) {+ got := phantomUniqueDrop(&schema.Field{Unique: tc.tag}, uniqueColumnType{unique: tc.unique})+ if got != tc.wantSkip {+ t.Fatalf("phantomUniqueDrop(unique=%v, tag=%v) = %v, want %v", tc.unique, tc.tag, got, tc.wantSkip)+ }+ })+ }+}++// TestWrappersExposeEverythingWrappedTypeDoes is the guard that made the+// concrete embedding necessary: embedding gorm's interfaces instead would drop+// the twenty migrator methods and the three dialector methods that gorm reaches+// by type assertion, and losing one surfaces as a panic part-way through+// AutoMigrate. Embedding the concrete types promotes all of them by+// construction; this catches a future refactor that reverts to the interfaces.+func TestWrappersExposeEverythingWrappedTypeDoes(t *testing.T) {+ for _, tc := range []struct {+ name string+ wrapped reflect.Type+ wrapper reflect.Type+ override string+ }{+ {"dialector", reflect.TypeOf(&mysql.Dialector{}), reflect.TypeOf(migrationAwareDialector{}), "Migrator"},+ {"migrator", reflect.TypeOf(mysql.Migrator{}), reflect.TypeOf(namedIndexMigrator{}), "MigrateColumnUnique"},+ } {+ t.Run(tc.name, func(t *testing.T) {+ for i := 0; i < tc.wrapped.NumMethod(); i++ {+ name := tc.wrapped.Method(i).Name+ if name == tc.override {+ continue+ }+ if _, ok := tc.wrapper.MethodByName(name); !ok {+ t.Errorf("wrapper does not expose %s; gorm reaches some capabilities by type assertion on the wrapped type rather than through its interfaces", name)+ }+ }+ })+ }+}+func TestAutoMigrateRuntimeModelsNamesFailingTable(t *testing.T) {db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})if err != nil {diff --git a/internal/dao/migration.go b/internal/dao/migration.goindex 37dcb839f5..c2efad68cd 100644--- a/internal/dao/migration.go+++ b/internal/dao/migration.go@@ -253,18 +253,11 @@ func migrateIngestionTaskDocumentIDUnique(ctx context.Context, db *gorm.DB) erro// migrateIngestionTaskPipelineLogID adds ingestion_task.pipeline_log_id when// the column is missing.//-// It cannot be left to AutoMigrate. The model declares document_id's unique key-// as a named unique index (uniqueIndex:idx_ingestion_task_document_id), while-// the column is UNIQUE at the database level. GORM's MigrateColumnUnique reads-// that pairing as a stray unique constraint: it issues-// `ALTER TABLE ingestion_task DROP FOREIGN KEY uni_ingestion_task_document_id`-// for the default constraint name, which does not exist, so MySQL fails with-// 1091. AutoMigrate aborts on the first error, and because autoMigrateSafely-// treats 1091 as a benign "already dropped" case, the failure is swallowed and-// the migration reports success -- before ever reaching the missing column.-// A column added to this table would therefore never be created, and every-// ingestion_task query (the DAO selects all columns) would fail with-// Error 1054.+// It runs after AutoMigrate rather than being left to it. AutoMigrate reaches a+// new column only once every earlier column has converged, so any failure on+// the way (a duplicate key, a rejected ALTER) silently skips it, and every+// ingestion_task query selects all columns: a missing column then fails the+// whole API with Error 1054.func migrateIngestionTaskPipelineLogID(ctx context.Context, db *gorm.DB) error {// Use the model, not the bare table name: HasColumn resolves the field// against the statement schema and dereferences it, which a plain stringdiff --git a/internal/entity/file_commit.go b/internal/entity/file_commit.goindex 1f262180a7..21c08216cb 100644--- a/internal/entity/file_commit.go+++ b/internal/entity/file_commit.go@@ -38,6 +38,12 @@ func (FileCommit) TableName() string {}// FileCommitItem represents a single file change within a commit.+//+// Python declares the same pair unique and Peewee names it+// file_commit_item_commit_id_file_id. Both runtimes match an existing index by+// column set rather than by name, so the two names are equivalent and one of+// them appearing in the database is not a bug. See the "Shared database schema"+// section of AGENTS.md before renaming either.type FileCommitItem struct {ID string `gorm:"column:id;primaryKey;size:32" json:"id"`Seq uint `gorm:"column:seq;index" json:"seq,omitempty"`