Skip to content

RAGFlow is an agent-native database: a Retrieval-Augmented Generation (RAG) engine that grew into context infrastructure.

Docs: v0.27.2 -> v1.0.0-rc1 (#20338)

Wang Qi1be9590a4a9f
diff --git a/docker/.env b/docker/.env
index 15f3f42362..4ff07f4a52 100644
--- a/docker/.env
+++ b/docker/.env
@@ -15,9 +15,9 @@
#
# CI (.github/workflows/sep-tests.yml) appends `true` here on every run: the
# image stamps its own git-describe output into /ragflow/VERSION, so a CI build
-# reports v0.27.x against the v1.0.0-rc1.dev1 marker it migrated to moments
+# reports v0.27.x against the v1.0.0-rc1 marker it migrated to moments
# earlier. Do not treat that generated value as the default.
-RAGFLOW_DEV_MODE=false
+# RAGFLOW_DEV_MODE=false
# ------------------------------
# docker env var for specifying vector db type at startup
@@ -261,7 +261,7 @@ SVR_MCP_PORT=9382
ADMIN_DEFAULT_PASSWORD=admin
# API_PROXY_SCHEME=hybrid # go and python hybrid deploy mode
-API_PROXY_SCHEME=go # use go server deployment
+API_PROXY_SCHEME=go
#API_PROXY_SCHEME=python # use pure python server deployment
# Development-only: set to 1 to bypass host safety checks for test_db_connection and allow private/local database hosts.
@@ -269,11 +269,11 @@ API_PROXY_SCHEME=go # use go server deployment
ALLOW_ANY_HOST=0
# The RAGFlow Docker image to download. v0.22+ doesn't include embedding models.
-RAGFLOW_IMAGE=infiniflow/ragflow:v1.0.0
+RAGFLOW_IMAGE=infiniflow/ragflow:v1.0.0-rc1
# If you cannot download the RAGFlow Docker image:
-# RAGFLOW_IMAGE=swr.cn-north-4.myhuaweicloud.com/infiniflow/ragflow:v1.0.0
-# RAGFLOW_IMAGE=registry.cn-hangzhou.aliyuncs.com/infiniflow/ragflow:v1.0.0
+# RAGFLOW_IMAGE=swr.cn-north-4.myhuaweicloud.com/infiniflow/ragflow:v1.0.0-rc1
+# RAGFLOW_IMAGE=registry.cn-hangzhou.aliyuncs.com/infiniflow/ragflow:v1.0.0-rc1
#
# - For the `nightly` edition, uncomment either of the following:
# RAGFLOW_IMAGE=swr.cn-north-4.myhuaweicloud.com/infiniflow/ragflow:nightly
diff --git a/docs/administrator/admin/ragflow_cli.md b/docs/administrator/admin/ragflow_cli.md
index 0800242521..9c1d3bddb7 100644
--- a/docs/administrator/admin/ragflow_cli.md
+++ b/docs/administrator/admin/ragflow_cli.md
@@ -25,7 +25,7 @@ When `VERSION` is omitted, the installer resolves the latest GitHub Release and
```bash
curl -fsSL https://raw.githubusercontent.com/infiniflow/ragflow/main/tools/scripts/install.sh \
- | VERSION=v0.27.2 sh
+ | VERSION=v1.0.0-rc1 sh
```
Both forms are supported. Pin a version for production or reproducible installations so that a later Release does not change the installed version.
@@ -46,7 +46,7 @@ When no version is specified, the Windows installer also uses the latest GitHub
```powershell
irm https://raw.githubusercontent.com/infiniflow/ragflow/main/tools/scripts/install.ps1 -OutFile install.ps1
-./install.ps1 -Version v0.27.2
+./install.ps1 -Version v1.0.0-rc1
```
The Windows installer installs `ragflow-cli.exe` under `%LOCALAPPDATA%\Programs\RAGFlow` by default and adds that directory to the user `PATH`. Restart the terminal if the installer reports that `PATH` was updated.
diff --git a/docs/administrator/migration/database_change_log.md b/docs/administrator/migration/database_change_log.md
index 628cbbcf43..2da92a62bc 100644
--- a/docs/administrator/migration/database_change_log.md
+++ b/docs/administrator/migration/database_change_log.md
@@ -32,10 +32,10 @@ Startup order in `InitDB` ([database.go](https://github.com/infiniflow/ragflow/b
| Database version | Scope | Gate constant |
|---|---|---|
| `v0.26.0` | Rebuild the tenant model tables from `tenant_llm` and normalize stored model ids | `modelMigrationBaseVersion` |
-| `v0.27.2` | Seed factory-declared models, merge `model_type` into an integer bitmask, populate the `tenant_*_id` columns | `modelMigrationTargetVersion` |
+| `v1.0.0-rc1` | Seed factory-declared models, merge `model_type` into an integer bitmask, populate the `tenant_*_id` columns | `modelMigrationTargetVersion` |
| `v1.0.0-rc1.dev1` | Split conversation message and reference payloads into child tables | `conversationHistoryTargetVersion` |
-The two tenant model steps are cumulative: a database at `v0.26.0` still runs the `v0.27.2` step, and a database at or above `v0.27.2` runs neither.
+The two tenant model steps are cumulative: a database at `v0.26.0` still runs the `v1.0.0-rc1` step, and a database at or above `v1.0.0-rc1` runs neither.
---
@@ -50,7 +50,7 @@ The two tenant model steps are cumulative: a database at `v0.26.0` still runs th
| `tenant_model_instance` | Created if missing. Columns: `id` (PK), `instance_name`, `provider_id`, `api_key`, `status`, `extra`, base timestamps. |
| `tenant_model` | Created if missing. Columns: `id` (PK), `model_name`, `provider_id`, `instance_id`, `model_type`, `status`, `extra`, base timestamps. |
-The three tables are created only when the legacy `tenant_llm` table exists and the target table does not, mirroring the `CREATE TABLE IF NOT EXISTS` the Python stages run. Existing tables are never altered by this step: `tenant_model.model_type` deliberately keeps the text shape the Python migration left behind so the `v0.27.2` merge can still read the legacy model names.
+The three tables are created only when the legacy `tenant_llm` table exists and the target table does not, mirroring the `CREATE TABLE IF NOT EXISTS` the Python stages run. Existing tables are never altered by this step: `tenant_model.model_type` deliberately keeps the text shape the Python migration left behind so the `v1.0.0-rc1` merge can still read the legacy model names.
### Data migration
diff --git a/docs/administrator/migration/database_schema_and_migration.md b/docs/administrator/migration/database_schema_and_migration.md
index 7f50ad39d9..07d86e0869 100644
--- a/docs/administrator/migration/database_schema_and_migration.md
+++ b/docs/administrator/migration/database_schema_and_migration.md
@@ -28,7 +28,7 @@ The standalone command runs the full sequence. Ordinary server startup connects
## Database version and startup check
-The Go migration marker is stored in `system_settings` under the fixed key `mysql_migration.database.version`. The key name is retained for database compatibility and does not select another migration implementation. The Go implementation writes `v0.26.0` after the base tenant-model step and `v0.27.2` after the follow-up model-data step. When the stored version is below `v1.0.0-rc1.dev1`, the conversation-history migration writes that value after it completes, even if the old history columns or rows are absent and no data needs copying. A step already covered by the stored version is skipped. The last completed step therefore determines the value that remains in the row. This marker is **not** a complete version number for every table or schema change. A missing or unparseable marker does not prove that the database is up to date.
+The Go migration marker is stored in `system_settings` under the fixed key `mysql_migration.database.version`. The key name is retained for database compatibility and does not select another migration implementation. The Go implementation writes `v0.26.0` after the base tenant-model step and `v0.27.2` after the follow-up model-data step. When the stored version is below `v1.0.0-rc1`, the conversation-history migration writes that value after it completes, even if the old history columns or rows are absent and no data needs copying. A step already covered by the stored version is skipped. The last completed step therefore determines the value that remains in the row. This marker is **not** a complete version number for every table or schema change. A missing or unparseable marker does not prove that the database is up to date.
Each Go server process compares its code version with this marker after database initialization. That initialization may already have updated runtime tables, indexes, or built-in templates before the check runs. The comparison uses the release-number portion of each version; development suffixes are not used to establish ordering. If the database version is newer, startup fails with `Refusing to start: database was migrated by a newer version`. If the marker is absent or the versions cannot be compared, this downgrade check does not block startup. The check does not apply pending migrations, and the standalone `--migrate` command does not run the downgrade check. Verify the binary and target database before invoking it.
diff --git a/docs/administrator/upgrade_ragflow.mdx b/docs/administrator/upgrade_ragflow.mdx
index 2608ff5127..e54afb29e5 100644
--- a/docs/administrator/upgrade_ragflow.mdx
+++ b/docs/administrator/upgrade_ragflow.mdx
@@ -64,16 +64,16 @@ To upgrade RAGFlow, you must upgrade **both** your code **and** your Docker imag
git pull
```
-3. Switch to the latest, officially published release, e.g., `v0.27.2`:
+3. Switch to the latest, officially published release, e.g., `v1.0.0-rc1`:
```bash
- git checkout -f v0.27.2
+ git checkout -f v1.0.0-rc1
```
4. Update **docker/.env**:
```bash
- RAGFLOW_IMAGE=infiniflow/ragflow:v0.27.2
+ RAGFLOW_IMAGE=infiniflow/ragflow:v1.0.0-rc1
```
5. Update the RAGFlow image and restart RAGFlow:
@@ -94,10 +94,10 @@ No, you do not need to. Upgrading RAGFlow in itself will *not* remove your uploa
1. From an environment with Internet access, pull the required Docker image.
2. Save the Docker image to a **.tar** file.
```bash
- docker save -o ragflow.v0.27.2.tar infiniflow/ragflow:v0.27.2
+ docker save -o ragflow.v1.0.0-rc1.tar infiniflow/ragflow:v1.0.0-rc1
```
3. Copy the **.tar** file to the target server.
4. Load the **.tar** file into Docker:
```bash
- docker load -i ragflow.v0.27.2.tar
+ docker load -i ragflow.v1.0.0-rc1.tar
```
diff --git a/internal/development.md b/internal/development.md
index 7b22cf1b0f..fb7f137518 100644
--- a/internal/development.md
+++ b/internal/development.md
@@ -252,7 +252,7 @@ otherwise, they will encounter errors when sending heartbeats.
`--migrate` writes the database version marker that server modes check on
startup, and a development branch regularly records a version for a release
that has not been tagged yet — a build from a `v0.27.x` commit that writes
-`v1.0.0-rc1.dev1` refuses to start afterwards, because the recorded version
+`v1.0.0-rc1` refuses to start afterwards, because the recorded version
looks newer than the code. Set `RAGFLOW_DEV_MODE=true` (see `docker/.env`) for
such a checkout: it turns the "code version must not be older than the database
version" guard off. Leave it off in production.
diff --git a/pyproject.toml b/pyproject.toml
index 251afd3683..02b991cc61 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
[project]
name = "ragflow"
-version = "0.27.2"
+version = "1.0.0-rc1"
description = "[RAGFlow](https://ragflow.io/) is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding. It offers a streamlined RAG workflow for businesses of any scale, combining LLM (Large Language Models) to provide truthful question-answering capabilities, backed by well-founded citations from various complex formatted data."
authors = [{ name = "Zhichang Yu", email = "[email protected]" }]
license-files = ["LICENSE"]
diff --git a/sdk/python/pyproject.toml b/sdk/python/pyproject.toml
index 2ed1485d49..73cb437b63 100644
--- a/sdk/python/pyproject.toml
+++ b/sdk/python/pyproject.toml
@@ -1,6 +1,6 @@
[project]
name = "ragflow-sdk"
-version = "0.27.2"
+version = "1.0.0-rc1"
description = "Python client sdk of [RAGFlow](https://github.com/infiniflow/ragflow). RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding."
authors = [{ name = "Zhichang Yu", email = "[email protected]" }]
license = { text = "Apache License, Version 2.0" }
diff --git a/sdk/python/uv.lock b/sdk/python/uv.lock
index bfd9447eb3..86deb7f50e 100644
--- a/sdk/python/uv.lock
+++ b/sdk/python/uv.lock
@@ -250,7 +250,7 @@ wheels = [
[[package]]
name = "ragflow-sdk"
-version = "0.27.2"
+version = "1.0.0-rc1"
source = { virtual = "." }
dependencies = [
{ name = "beartype" },
diff --git a/test/README.md b/test/README.md
index 724236cd5d..ca6fd17357 100644
--- a/test/README.md
+++ b/test/README.md
@@ -33,7 +33,7 @@ uv pip install sdk/python
```env
COMPOSE_PROFILES=${COMPOSE_PROFILES},tei-cpu
TEI_MODEL=BAAI/bge-small-en-v1.5
-RAGFLOW_IMAGE=infiniflow/ragflow:v0.27.2 #Replace with the image you are using
+RAGFLOW_IMAGE=infiniflow/ragflow:v1.0.0-rc1 #Replace with the image you are using
```
diff --git a/tools/scripts/README.md b/tools/scripts/README.md
deleted file mode 100644
index 286f12f8ad..0000000000
--- a/tools/scripts/README.md
+++ /dev/null
@@ -1,346 +0,0 @@
-# Database Scripts
-
-This directory contains database-related utility scripts for RAGFlow.
-
-- **mysql_migration.py**: Data migration between tables with stage-based execution
-- **db_schema_sync.py**: Database schema synchronization using peewee-migrate
-
----
-
-# mysql_migration.py
-
-A flexible MySQL data migration tool for migrating data between tables with stage-based execution.
-
-## Overview
-
-This script provides stage-based data migration between MySQL tables. Currently supports:
-- `tenant_model_provider`
-- `tenant_model_instance`
-- `tenant_model`
-
-### Migration Stages
-
-| Stage | Source Table | Target Table | Description |
-|-------|-------------|--------------|-------------|
-| `tenant_model_provider` | `tenant_llm` | `tenant_model_provider` | Extracts distinct `(tenant_id, llm_factory)` pairs |
-| `tenant_model_instance` | `tenant_llm` + `tenant_model_provider` | `tenant_model_instance` | Creates instances with distinct `(tenant_id, llm_factory, api_key)` |
-| `tenant_model` | `tenant_llm` + `tenant_model_provider` + `tenant_model_instance` | `tenant_model` | Migrates model configurations (only `status='0'` records) |
-
-### Stage Dependencies
-
-```
-tenant_model_provider (no dependencies)
- ↓
-tenant_model_instance (depends on tenant_model_provider)
- ↓
-tenant_model (depends on tenant_model_provider and tenant_model_instance)
-```
-
-### Field Mapping Rules
-
-#### tenant_model_provider
-
-| Target Field | Source | Rule |
-|--------------|--------|------|
-| `id` | - | Random 32-character UUID1 |
-| `provider_name` | `tenant_llm.llm_factory` | Direct mapping |
-| `tenant_id` | `tenant_llm.tenant_id` | Direct mapping |
-
-- **Deduplication**: Groups by `(tenant_id, llm_factory)` and takes distinct pairs
-
-#### tenant_model_instance
-
-| Target Field | Source | Rule |
-|--------------|--------|------|
-| `id` | - | Random 32-character UUID1 |
-| `instance_name` | `tenant_llm.llm_factory` | Direct mapping |
-| `provider_id` | `tenant_model_provider.id` | JOIN on `tenant_id` and `provider_name=llm_factory` |
-| `api_key` | `tenant_llm.api_key` | Direct mapping |
-| `status` | `tenant_llm.status` | Direct mapping |
-
-- **Deduplication**: Groups by `(tenant_id, llm_factory, api_key)` and takes distinct records
-
-#### tenant_model
-
-| Target Field | Source | Rule |
-|--------------|--------|------|
-| `id` | - | Random 32-character UUID1 |
-| `model_name` | `tenant_llm.llm_name` | Direct mapping |
-| `provider_id` | `tenant_model_provider.id` | JOIN on `tenant_id` and `provider_name=llm_factory` |
-| `instance_id` | `tenant_model_instance.id` | JOIN on `provider_id` and `api_key` |
-| `model_type` | `tenant_llm.model_type` | Direct mapping |
-| `status` | `tenant_llm.status` | Direct mapping |
-
-- **Filter**: Only migrates records where `tenant_llm.status='0'`
-
-## Usage
-
-### Command Line Arguments
-
-```
-python mysql_migration.py [OPTIONS]
-```
-
-| Option | Short | Description | Default |
-|--------|-------|-------------|---------|
-| `--host` | - | MySQL host | `localhost` |
-| `--port` | - | MySQL port | `3306` |
-| `--user` | - | MySQL user | `root` |
-| `--password` | - | MySQL password | (empty) |
-| `--database` | - | MySQL database name | `rag_flow` |
-| `--config` | `-c` | Path to YAML config file | - |
-| `--stages` | `-s` | Comma-separated list of stages to run | - |
-| `--list-stages` | `-l` | List available stages and exit | - |
-| `--execute` | `-e` | Execute full migration (create tables and migrate data) | `False` |
-| `--create-table-only` | - | Only create target tables, skip data migration | `False` |
-
-> **Note**: MySQL connection can be configured via command line arguments (`--host`, `--port`, `--user`, `--password`, `--database`) or via a YAML config file (`--config`). Command line arguments take precedence over config file values.
-
-### Execution Modes
-
-The script has three mutually exclusive modes:
-
-1. **Dry-Run Mode** (default): Check only, no database writes
- ```bash
- # Using config file
- python mysql_migration.py --stages tenant_model_provider --config config.yaml
-
- # Using command line MySQL connection
- python mysql_migration.py --stages tenant_model_provider --host localhost --port 3306 --user root
- ```
-
-2. **Create Table Only Mode**: Create target tables without migrating data
- ```bash
- python mysql_migration.py --stages tenant_model_provider --config config.yaml --create-table-only
- ```
-
-3. **Execute Mode**: Create tables and migrate data
- ```bash
- python mysql_migration.py --stages tenant_model_provider --config config.yaml --execute
- ```
-
-### Configuration File
-
-Create a YAML configuration file with MySQL connection settings:
-
-```yaml
-database:
- host: localhost
- port: 3306
- user: root
- password: your_password
- name: rag_flow
-```
-
-Alternative keys are also supported:
-
-```yaml
-mysql:
- host: localhost
- port: 3306
- user: root
- password: your_password
- database: rag_flow
-```
-
-### Examples
-
-```bash
-# List all available stages
-python mysql_migration.py --list-stages
-
-# Dry run single stage using command line MySQL connection
-python mysql_migration.py --stages tenant_model_provider --host localhost --port 3306 --user root --password secret
-
-# Dry run single stage using config file
-python mysql_migration.py --stages tenant_model_provider --config /path/to/config.yaml
-
-# Create tables only for multiple stages
-python mysql_migration.py --stages tenant_model_provider,tenant_model_instance --config /path/to/config.yaml --create-table-only
-
-# Execute full migration for all stages (in dependency order)
-python mysql_migration.py --stages tenant_model_provider,tenant_model_instance,tenant_model --config /path/to/config.yaml --execute
-
-# Use config file with command line password override
-python mysql_migration.py --stages tenant_model_provider --config /path/to/config.yaml --password mypassword --execute
-```
-
-## Output Interpretation
-
-### Stage Execution Log
-
-Each stage displays a header showing progress:
-
-```
-============================================================
-Stage [1/3]: tenant_model_provider
-============================================================
-```
-
-The stage then performs:
-1. Check phase: Verifies source/target tables exist and counts records to migrate
-2. Execute phase: Creates tables (if needed) and migrates data in batches
-
-### Dry-Run Output
-
-In dry-run mode, the script outputs what it would do without writing:
-
-```
-[DRY RUN] Would insert 150 records
- instance_name=OpenAI, provider_id=abc123, api_key=***
- ... and 145 more records
-```
-
-### Migration Summary
-
-After all stages complete, a summary is printed:
-
-```
-============================================================
-Migration Summary
-============================================================
-Total Duration: 2.45s
-Total Rows Processed: 350
-Tables Operated: tenant_model_provider, tenant_model_instance
-------------------------------------------------------------
-Stage Details:
- [tenant_model_provider] Tables: tenant_model_provider, Rows: 50, Duration: 0.82s
- [tenant_model_instance] Tables: tenant_model_instance, Rows: 300, Duration: 1.63s
-============================================================
-```
-
-### Common Messages
-
-| Message | Meaning |
-|---------|-------------------------------------------------------------------------|
-| `No new data to migrate` | All records already exist in target table |
-| `[DRY RUN] Target table does not exist` | Target table missing, use `--execute` or `--create-table-only`to create |
-| `Dependency table does not exist` | Required table from previous stage missing |
-| `Inserted batch X: Y records` | Successfully inserted batch of records |
-
----
-
-# db_schema_sync.py
-
-A database schema synchronization tool that uses peewee-migrate to detect and manage schema changes.
-
-## Overview
-
-This script:
-1. Reads model definitions from `api/db/db_models.py`
-2. Compares with existing database tables specified via command line
-3. Generates migration files in `tools/migrate/{version}/`
-
-### Detected Change Types
-
-| Change Type | Description | Auto-included? |
-|-------------|-------------|----------------|
-| New table | Model class with no corresponding DB table | Yes |
-| New field | Model field not present in DB table | Yes |
-| Field type change | Model field type differs from DB column type | Yes |
-| Removed field | DB column not present in model definition | No (requires `--drop`) |
-
-> **Warning**: Removed fields are **not** included in migrations by default. You must explicitly use `--drop` to generate `DROP COLUMN` statements, as this operation permanently deletes data.
-
-## Prerequisites
-
-Install peewee-migrate:
-```bash
-pip install peewee-migrate
-```
-
-## Usage
-
-### Command Line Arguments
-
-```
-python db_schema_sync.py [OPTIONS]
-```
-
-| Option | Short | Description |
-|--------|-------|-------------|
-| `--host` | - | MySQL host (required) |
-| `--port` | - | MySQL port (default: 3306) |
-| `--user` | - | MySQL user (required) |
-| `--password` | - | MySQL password (required) |
-| `--database` | - | MySQL database name (required) |
-| `--version` | `-v` | Version number in format `vxx.xx.xx` (required) |
-| `--list` | `-l` | List all migrations |
-| `--create` | - | Create a new migration (auto-detect changes) |
-| `--migrate` | `-m` | Run pending migrations |
-| `--diff` | `-d` | Show schema differences |
-| `--name` | `-n` | Migration name (default: auto) |
-| `--drop` | - | Include `DROP COLUMN` for fields removed from models (destructive - permanently deletes data!) |
-
-### Version Format
-
-Version must be in format `vxx.xx.xx` where `xx` are digits:
-- Valid: `v0.27.2`, `v1.0.0`, `v10.20.30`
-- Invalid: `0.27.2`, `v0.25`, `v0.27.2.1`
-
-### Migration File Location
-
-Migration files are stored in:
-```
-tools/migrate/{version_dir}/
-```
-
-Where `{version_dir}` is the version with `.` replaced by `_`.
-
-Example: Version `v0.27.2` → Directory `tools/migrate/v0_27_2/`
-
-### Examples
-
-```bash
-# List all migrations
-python db_schema_sync.py --list \
- --host localhost --port 3306 --user root --password xxx --database rag_flow \
- --version v0.27.2
-
-# Create a new auto-detected migration (new tables, new fields, type changes only)
-python db_schema_sync.py --create \
- --host localhost --port 3306 --user root --password xxx --database rag_flow \
- --version v0.27.2
-
-# Create a migration including dropped fields (destructive!)
-python db_schema_sync.py --create --drop \
- --host localhost --port 3306 --user root --password xxx --database rag_flow \
- --version v0.27.2
-
-# Create a named migration
-python db_schema_sync.py --create --name add_user_table \
- --host localhost --port 3306 --user root --password xxx --database rag_flow \
- --version v0.27.2
-
-# Run all pending migrations
-python db_schema_sync.py --migrate \
- --host localhost --port 3306 --user root --password xxx --database rag_flow \
- --version v0.27.2
-
-# Show schema differences (including removed fields)
-python db_schema_sync.py --diff \
- --host localhost --port 3306 --user root --password xxx --database rag_flow \
- --version v0.27.2
-```
-
-## How It Works
-
-1. **Load Models**: Imports all model classes from `api/db/db_models.py`
-2. **Connect Database**: Creates MySQL connection from command line arguments
-3. **Detect Changes**: Compares model definitions with actual database schema:
- - New tables → `create_model`
- - New fields → `ALTER TABLE ADD COLUMN`
- - Field type changes → `ALTER TABLE MODIFY COLUMN`
- - Removed fields → `ALTER TABLE DROP COLUMN` (only with `--drop`)
-4. **Generate Migration**: Creates Python migration file with `migrate()` and `rollback()` functions
-
-### Rollback Behavior
-
-| Forward Operation | Rollback Operation |
-|-------------------|--------------------|
-| `CREATE TABLE` | `remove_model` |
-| `ADD COLUMN` | `DROP COLUMN` |
-| `MODIFY COLUMN` | `MODIFY COLUMN` (restore original type) |
-| `DROP COLUMN` | `ADD COLUMN` (restore column definition; **data is lost**) |
-
-> **Note**: Rolling back a `DROP COLUMN` will re-add the column structure, but the data that was in it cannot be recovered.
diff --git a/tools/scripts/db_schema_sync.py b/tools/scripts/db_schema_sync.py
deleted file mode 100644
index 469e78a4b6..0000000000
--- a/tools/scripts/db_schema_sync.py
+++ /dev/null
@@ -1,942 +0,0 @@
-#
-# Copyright 2026 The InfiniFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-"""
-Database Schema Sync Script
-
-This script synchronizes database models defined in api/db/db_models.py
-with the actual database schema using peewee-migrate.
-
-Features:
-1. Reads model definitions from api/db/db_models.py
-2. Compares with existing database tables specified via command line
-3. Generates migration files in tools/migrate/{version}/
-"""
-
-import argparse
-import importlib.util
-import inspect
-import logging
-import os
-import re
-import sys
-
-from peewee import Field, Model, MySQLDatabase
-from peewee_migrate import Router
-
-# Add project root to path for imports
-PROJECT_BASE = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-sys.path.insert(0, PROJECT_BASE)
-
-# Configure logging
-logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")
-logger = logging.getLogger(__name__)
-
-
-def validate_version(version: str) -> bool:
- """Validate version format: vxx.xx.xx where xx are digits"""
- pattern = r"^v\d+\.\d+\.\d+$"
- return bool(re.match(pattern, version))
-
-
-def version_to_dirname(version: str) -> str:
- """Convert version string to valid directory name (e.g., 'v0.27.2' -> 'v0_27_2')"""
- return version.replace(".", "_")
-
-
-def load_db_models():
- """Load database models from api/db/db_models.py"""
- models_path = os.path.join(PROJECT_BASE, "api", "db", "db_models.py")
-
- if not os.path.exists(models_path):
- raise FileNotFoundError(f"db_models.py not found at {models_path}")
-
- # Import the module
- spec = importlib.util.spec_from_file_location("db_models", models_path)
- db_models = importlib.util.module_from_spec(spec)
- spec.loader.exec_module(db_models)
-
- # Get all Model subclasses
- models = []
- for name, obj in inspect.getmembers(db_models):
- if inspect.isclass(obj) and issubclass(obj, Model) and obj is not Model:
- # Skip base model classes
- if obj.__name__ in ["BaseModel", "DataBaseModel"]:
- continue
- # Check if it has a database attribute (is a proper model)
- if hasattr(obj._meta, "database"):
- models.append(obj)
-
- return models, db_models
-
-
-def create_database_connection(host: str, port: int, user: str, password: str, database: str):
- """Create MySQL database connection from command line arguments"""
- db = MySQLDatabase(database, host=host, port=port, user=user, password=password, charset="utf8mb4")
- return db
-
-
-# MySQL type to Peewee field type mapping
-# Fields peewee/BaseModel manage on their own. compare_fields() and both of its
-# callers must exclude the same set, or columns filtered out on one side get
-# reported as removed on the other.
-AUTO_GENERATED_FIELDS = {"id", "create_time", "create_date", "update_time", "update_date"}
-
-MYSQL_TO_PEEWEE_TYPE = {
- "varchar": "CharField",
- "char": "CharField",
- "text": "TextField",
- "longtext": "TextField",
- "mediumtext": "TextField",
- "int": "IntegerField",
- "integer": "IntegerField",
- "bigint": "BigIntegerField",
- "float": "FloatField",
- "double": "FloatField",
- "decimal": "FloatField",
- "datetime": "DateTimeField",
- "timestamp": "DateTimeField",
- "tinyint(1)": "BooleanField",
- "tinyint": "IntegerField",
- "smallint": "IntegerField",
- "mediumint": "IntegerField",
-}
-
-PEEWEE_TO_MYSQL_TYPE = {
- "CharField": "varchar",
- "TextField": "text",
- "IntegerField": "int",
- "BigIntegerField": "bigint",
- "FloatField": "float",
- "BooleanField": "tinyint",
- "DateTimeField": "datetime",
-}
-
-
-def get_table_columns(db, table_name: str) -> dict:
- """Get column information from database table
-
- Returns:
- dict: {column_name: {type, nullable, default, ...}}
- """
- cursor = db.execute_sql(
- """
- SELECT
- column_name,
- data_type,
- column_type,
- is_nullable,
- column_default,
- column_key,
- extra
- FROM information_schema.columns
- WHERE table_schema = %s AND table_name = %s
- ORDER BY ordinal_position
- """,
- (db.database, table_name),
- )
-
- columns = {}
- for row in cursor.fetchall():
- col_name = row[0]
- data_type = row[1].lower()
- column_type = row[2].lower()
- is_nullable = row[3] == "YES"
- column_default = row[4]
- column_key = row[5]
- extra = row[6] or ""
-
- # Determine peewee type
- if column_type.startswith("tinyint(1)"):
- peewee_type = "BooleanField"
- else:
- peewee_type = MYSQL_TO_PEEWEE_TYPE.get(data_type, "TextField")
-
- columns[col_name] = {
- "data_type": data_type,
- "column_type": column_type,
- "peewee_type": peewee_type,
- "nullable": is_nullable,
- "default": column_default,
- "is_primary": column_key == "PRI",
- "extra": extra,
- }
-
- return columns
-
-
-def get_peewee_field_type(field: Field) -> str:
- """Get peewee field type name"""
- field_class = field.__class__.__name__
- return field_class
-
-
-def get_base_field_type(field: Field) -> str:
- """Get base peewee field type by walking the MRO chain.
-
- Custom field types (like DateTimeTzField, JSONField) inherit from standard types.
- This function returns the underlying standard type for comparison.
- """
- # Standard peewee field types we consider as "base" types
- STANDARD_TYPES = {
- "CharField",
- "TextField",
- "IntegerField",
- "BigIntegerField",
- "FloatField",
- "BooleanField",
- "DateTimeField",
- "DateField",
- "TimeField",
- "DecimalField",
- "ForeignKeyField",
- "ManyToManyField",
- "PrimaryKeyField",
- "AutoField",
- }
-
- # Walk through the MRO (Method Resolution Order) to find standard type
- for cls in field.__class__.__mro__:
- class_name = cls.__name__
- if class_name in STANDARD_TYPES:
- return class_name
-
- # Fallback to TextField if no standard type found
- return "TextField"
-
-
-def normalize_field_type(field: Field) -> str:
- """Normalize field type for comparison using base type"""
- return get_base_field_type(field)
-
-
-def compare_fields(model_fields: dict, db_columns: dict) -> dict:
- """Compare model fields with database columns
-
- Returns:
- dict: {
- 'added': {field_name: field_obj}, # New fields not in DB
- 'changed': {field_name: (old_info, new_field)}, # Type changed
- 'removed': {field_name: col_info}, # Fields in DB but not in model
- }
- """
- result = {
- "added": {},
- "changed": {},
- "removed": {},
- }
-
- skip_fields = AUTO_GENERATED_FIELDS
- logger.debug("Excluding auto-generated fields from schema comparison: %s", sorted(skip_fields))
-
- for field_name, field in model_fields.items():
- if field_name in skip_fields:
- continue
-
- # Check if field exists in database
- if field_name not in db_columns:
- result["added"][field_name] = field
- logger.info(f" New field detected: {field_name} ({field.__class__.__name__})")
- else:
- # Check if type changed
- db_col = db_columns[field_name]
- model_base_type = normalize_field_type(field)
- db_type = db_col["peewee_type"]
-
- # Type mismatch
- if model_base_type != db_type:
- result["changed"][field_name] = (db_col, field)
- logger.info(f" Field type changed: {field_name} ({db_type} -> {model_base_type}, actual: {field.__class__.__name__})")
-
- # Detect removed fields: columns in DB but not in model
- for col_name, col_info in db_columns.items():
- if col_name in skip_fields:
- continue
- if col_name not in model_fields:
- result["removed"][col_name] = col_info
- logger.info(f" Removed field detected: {col_name} ({col_info['column_type']})")
-
- return result
-
-
-def generate_field_code(field: Field, field_name: str) -> str:
- """Generate peewee field definition code"""
- field_class = field.__class__.__name__
-
- # Map custom field types to standard peewee types for migration
- # These custom types will be stored as their underlying standard type
- custom_to_standard = {
- "LongTextField": "TextField",
- "JSONField": "TextField",
- "ListField": "TextField",
- "SerializedField": "TextField",
- "DateTimeTzField": "CharField",
- }
-
- # Use standard type for custom fields
- pw_field_class = custom_to_standard.get(field_class, field_class)
-
- # Build field arguments
- args = []
-
- # max_length for CharField
- if pw_field_class == "CharField" and hasattr(field, "max_length") and field.max_length is not None:
- args.append(f"max_length={field.max_length}")
-
- # null
- if field.null:
- args.append("null=True")
-
- # default
- if field.default is not None:
- default_val = field.default
- if isinstance(default_val, str):
- # Escape quotes in string
- escaped = default_val.replace("'", "\\'")
- args.append(f"default='{escaped}'")
- elif isinstance(default_val, bool):
- args.append(f"default={'True' if default_val else 'False'}")
- elif isinstance(default_val, (int, float)):
- args.append(f"default={default_val}")
- elif isinstance(default_val, dict):
- args.append(f"default={default_val}")
- elif isinstance(default_val, list):
- args.append(f"default={default_val}")
-
- # index
- if getattr(field, "index", False):
- args.append("index=True")
-
- # unique
- if getattr(field, "unique", False):
- args.append("unique=True")
-
- args_str = ", ".join(args)
- return f"pw.{pw_field_class}({args_str})"
-
-
-def generate_add_field_sql(table_name: str, field: Field, field_name: str) -> str:
- """Generate raw SQL for adding a field to MySQL table.
-
- This is used for existing tables where migrator.add_fields doesn't work
- because the model is not registered in migrator.orm.
- """
- field_class = field.__class__.__name__
-
- # Determine MySQL column type
- mysql_type_map = {
- "CharField": f"VARCHAR({field.max_length})" if hasattr(field, "max_length") and field.max_length else "VARCHAR(255)",
- "TextField": "LONGTEXT",
- "LongTextField": "LONGTEXT",
- "JSONField": "LONGTEXT",
- "ListField": "LONGTEXT",
- "SerializedField": "LONGTEXT",
- "IntegerField": "INT",
- "BigIntegerField": "BIGINT",
- "FloatField": "DOUBLE",
- "BooleanField": "TINYINT(1)",
- "DateTimeField": "DATETIME",
- "DateTimeTzField": f"VARCHAR({field.max_length})" if hasattr(field, "max_length") and field.max_length else "VARCHAR(255)",
- }
-
- mysql_type = mysql_type_map.get(field_class, "LONGTEXT")
-
- # Build column definition
- parts = [f"`{field_name}`", mysql_type]
-
- # NULL/NOT NULL
- if field.null:
- parts.append("NULL")
- else:
- parts.append("NOT NULL")
-
- # DEFAULT
- if field.default is not None:
- default_val = field.default
- if isinstance(default_val, str):
- escaped = default_val.replace("'", "''")
- parts.append(f"DEFAULT '{escaped}'")
- elif isinstance(default_val, bool):
- parts.append(f"DEFAULT {1 if default_val else 0}")
- elif isinstance(default_val, (int, float)):
- parts.append(f"DEFAULT {default_val}")
- elif isinstance(default_val, dict) or isinstance(default_val, list):
- import json
-
- escaped = json.dumps(default_val).replace("'", "''")
- parts.append(f"DEFAULT '{escaped}'")
-
- # COMMENT
- if hasattr(field, "help_text") and field.help_text:
- escaped = field.help_text.replace("'", "''")
- parts.append(f"COMMENT '{escaped}'")
-
- sql = f"ALTER TABLE `{table_name}` ADD COLUMN {' '.join(parts)}"
-
- # Add index if needed
- index_sql = None
- if getattr(field, "index", False):
- index_sql = f"CREATE INDEX `idx_{table_name}_{field_name}` ON `{table_name}` (`{field_name}`)"
-
- return sql, index_sql
-
-
-def generate_drop_field_sql(table_name: str, field_name: str) -> str:
- """Generate SQL for dropping a field from a table."""
- return f"ALTER TABLE `{table_name}` DROP COLUMN `{field_name}`"
-
-
-def generate_rollback_field_sql(table_name: str, field_name: str) -> str:
- """Generate SQL for removing a field."""
- return f"ALTER TABLE `{table_name}` DROP COLUMN `{field_name}`"
-
-
-def generate_rollback_add_field_sql(table_name: str, col_info: dict, field_name: str) -> str:
- """Generate SQL for rolling back a dropped field (re-adding it).
-
- This reconstructs the ADD COLUMN statement from the column info
- that was captured before the field was dropped.
- """
- mysql_type = col_info.get("column_type", "LONGTEXT")
-
- parts = [f"`{field_name}`", mysql_type]
-
- # NULL/NOT NULL
- if col_info.get("nullable", True):
- parts.append("NULL")
- else:
- parts.append("NOT NULL")
-
- # DEFAULT
- default_val = col_info.get("default")
- if default_val is not None:
- if isinstance(default_val, str):
- escaped = default_val.replace("'", "''")
- parts.append(f"DEFAULT '{escaped}'")
- elif isinstance(default_val, bool):
- parts.append(f"DEFAULT {1 if default_val else 0}")
- elif isinstance(default_val, (int, float)):
- parts.append(f"DEFAULT {default_val}")
-
- sql = f"ALTER TABLE `{table_name}` ADD COLUMN {' '.join(parts)}"
-
- # Re-add index if it was a non-primary key
- index_sql = None
- if col_info.get("column_key") == "MUL":
- index_sql = f"CREATE INDEX `idx_{table_name}_{field_name}` ON `{table_name}` (`{field_name}`)"
-
- return sql, index_sql
-
-
-def generate_rollback_modify_sql(table_name: str, old_info: dict, field_name: str) -> str:
- """Generate SQL for rolling back a field type change.
-
- Note: This restores the column type, but data values may need manual handling
- if the type conversion caused data loss or transformation.
- """
- # Reconstruct MySQL type from old_info
- mysql_type = old_info.get("column_type", "LONGTEXT")
-
- # Build column definition
- parts = [f"`{field_name}`", mysql_type]
-
- # NULL/NOT NULL
- if old_info.get("nullable", True):
- parts.append("NULL")
- else:
- parts.append("NOT NULL")
-
- # DEFAULT (if available)
- if old_info.get("default") is not None:
- default_val = old_info["default"]
- if isinstance(default_val, str):
- escaped = default_val.replace("'", "''")
- parts.append(f"DEFAULT '{escaped}'")
- elif isinstance(default_val, bool):
- parts.append(f"DEFAULT {1 if default_val else 0}")
- elif isinstance(default_val, (int, float)):
- parts.append(f"DEFAULT {default_val}")
-
- return f"ALTER TABLE `{table_name}` MODIFY COLUMN {' '.join(parts)}"
-
-
-def generate_modify_field_sql(table_name: str, field: Field, field_name: str) -> str:
- """Generate SQL for modifying a field in MySQL table."""
- field_class = field.__class__.__name__
-
- # Determine MySQL column type
- mysql_type_map = {
- "CharField": f"VARCHAR({field.max_length})" if hasattr(field, "max_length") and field.max_length else "VARCHAR(255)",
- "TextField": "LONGTEXT",
- "LongTextField": "LONGTEXT",
- "JSONField": "LONGTEXT",
- "ListField": "LONGTEXT",
- "SerializedField": "LONGTEXT",
- "IntegerField": "INT",
- "BigIntegerField": "BIGINT",
- "FloatField": "DOUBLE",
- "BooleanField": "TINYINT(1)",
- "DateTimeField": "DATETIME",
- "DateTimeTzField": f"VARCHAR({field.max_length})" if hasattr(field, "max_length") and field.max_length else "VARCHAR(255)",
- }
-
- mysql_type = mysql_type_map.get(field_class, "LONGTEXT")
-
- # Build column definition
- parts = [f"`{field_name}`", mysql_type]
-
- # NULL/NOT NULL
- if field.null:
- parts.append("NULL")
- else:
- parts.append("NOT NULL")
-
- # DEFAULT
- if field.default is not None:
- default_val = field.default
- if isinstance(default_val, str):
- escaped = default_val.replace("'", "''")
- parts.append(f"DEFAULT '{escaped}'")
- elif isinstance(default_val, bool):
- parts.append(f"DEFAULT {1 if default_val else 0}")
- elif isinstance(default_val, (int, float)):
- parts.append(f"DEFAULT {default_val}")
- elif isinstance(default_val, dict) or isinstance(default_val, list):
- import json
-
- escaped = json.dumps(default_val).replace("'", "''")
- parts.append(f"DEFAULT '{escaped}'")
-
- # COMMENT
- if hasattr(field, "help_text") and field.help_text:
- escaped = field.help_text.replace("'", "''")
- parts.append(f"COMMENT '{escaped}'")
-
- return f"ALTER TABLE `{table_name}` MODIFY COLUMN {' '.join(parts)}"
-
-
-def generate_migration_content(new_tables: list, field_changes: dict, migrate_dir: str, migration_name: str, drop_fields: bool = False) -> str:
- """Generate migration file content"""
- lines = [
- '"""Peewee migrations."""',
- "",
- "from contextlib import suppress",
- "",
- "import peewee as pw",
- "from peewee_migrate import Migrator",
- "",
- "",
- "with suppress(ImportError):",
- " import playhouse.postgres_ext as pw_pext",
- "",
- "",
- "def migrate(migrator: Migrator, database: pw.Database, *, fake=False):",
- ' """Write your migrations here."""',
- "",
- ]
-
- # Generate create_model for new tables
- for model in new_tables:
- table_name = model._meta.table_name
- model_name = model.__name__
-
- lines.append(" @migrator.create_model")
- lines.append(f" class {model_name}(pw.Model):")
-
- # Get all fields
- fields = model._meta.fields
- for field_name, field in fields.items():
- field_code = generate_field_code(field, field_name)
- lines.append(f" {field_name} = {field_code}")
-
- lines.append("")
- lines.append(" class Meta:")
- lines.append(f' table_name = "{table_name}"')
-
- # Add indexes if defined
- indexes = getattr(model._meta, "indexes", None)
- if indexes:
- lines.append(f" indexes = {indexes}")
-
- lines.append("")
-
- # Generate SQL for adding new fields to existing tables
- for table_name, changes in field_changes.items():
- if changes.get("added"):
- for field_name, field in changes["added"].items():
- sql, index_sql = generate_add_field_sql(table_name, field, field_name)
- lines.append(f' migrator.sql("{sql}")')
- if index_sql:
- lines.append(f' migrator.sql("{index_sql}")')
- lines.append("")
-
- # Generate SQL for modifying fields in existing tables
- for table_name, changes in field_changes.items():
- if changes.get("changed"):
- for field_name, (old_info, field) in changes["changed"].items():
- modify_sql = generate_modify_field_sql(table_name, field, field_name)
- lines.append(f' migrator.sql("{modify_sql}")')
- lines.append("")
-
- # Generate SQL for dropping removed fields from existing tables
- if drop_fields:
- for table_name, changes in field_changes.items():
- if changes.get("removed"):
- for field_name, col_info in changes["removed"].items():
- drop_sql = generate_drop_field_sql(table_name, field_name)
- lines.append(f" # WARNING: Dropping column `{field_name}` from `{table_name}` - this will permanently delete data!")
- lines.append(f' migrator.sql("{drop_sql}")')
- lines.append("")
-
- # Generate rollback
- lines.append("")
- lines.append("def rollback(migrator: Migrator, database: pw.Database, *, fake=False):")
- lines.append(' """Write your rollback migrations here."""')
- lines.append("")
-
- # Rollback: re-add dropped fields (before other rollbacks, since they may depend on these fields)
- if drop_fields:
- for table_name, changes in field_changes.items():
- if changes.get("removed"):
- for field_name, col_info in changes["removed"].items():
- add_sql, index_sql = generate_rollback_add_field_sql(table_name, col_info, field_name)
- lines.append(f" # Re-add dropped column `{field_name}` to `{table_name}` (data is lost)")
- lines.append(f' migrator.sql("{add_sql}")')
- if index_sql:
- lines.append(f' migrator.sql("{index_sql}")')
-
- # Rollback: reverse field type changes first (before removing added fields)
- for table_name, changes in field_changes.items():
- if changes.get("changed"):
- for field_name, (old_info, field) in changes["changed"].items():
- rollback_modify_sql = generate_rollback_modify_sql(table_name, old_info, field_name)
- lines.append(" # Note: Data values may need manual handling if type conversion caused data loss")
- lines.append(f' migrator.sql("{rollback_modify_sql}")')
-
- # Rollback: remove added fields using SQL
- for table_name, changes in field_changes.items():
- if changes.get("added"):
- for field_name in changes["added"].keys():
- rollback_sql = generate_rollback_field_sql(table_name, field_name)
- lines.append(f' migrator.sql("{rollback_sql}")')
-
- # Rollback: remove tables (in reverse order)
- for model in reversed(new_tables):
- table_name = model._meta.table_name
- lines.append(f' migrator.remove_model("{table_name}")')
-
- lines.append("")
-
- return "\n".join(lines)
-
-
-def create_migration(router: Router, models: list, db, name: str = "auto", drop_fields: bool = False):
- """Create a new migration by auto-detecting model changes
-
- Detects:
- 1. New tables -> generate create_model
- 2. New fields in existing tables -> generate add_fields
- 3. Field type changes -> generate change_fields
- 4. Removed fields (only when --drop is specified) -> generate drop_fields
-
- Args:
- router: peewee-migrate Router instance
- models: List of model classes to compare against database
- db: Database connection
- name: Migration name
- drop_fields: Whether to include DROP COLUMN for removed fields
- """
- try:
- # Get existing tables from database
- cursor = db.execute_sql("SELECT table_name FROM information_schema.tables WHERE table_schema = %s", (db.database,))
- existing_tables = {row[0] for row in cursor.fetchall()}
-
- new_tables = []
- field_changes = {}
-
- for model in models:
- table_name = model._meta.table_name
-
- if table_name not in existing_tables:
- # New table
- new_tables.append(model)
- logger.info(f"New table detected: {table_name}")
- else:
- # Existing table - check for field changes
- logger.info(f"Checking existing table: {table_name}")
-
- # Get model fields (exclude auto-generated)
- model_fields = {}
- for field_name, field in model._meta.fields.items():
- # Skip id and base model fields
- if field_name in AUTO_GENERATED_FIELDS:
- continue
- if hasattr(field, "_auto_created") and field._auto_created:
- continue
- model_fields[field_name] = field
-
- # Get database columns
- db_columns = get_table_columns(db, table_name)
-
- # Compare
- changes = compare_fields(model_fields, db_columns)
-
- if changes["added"] or changes["changed"] or changes["removed"]:
- field_changes[table_name] = changes
-
- # Check if any changes detected
- has_removed = any(changes.get("removed") for changes in field_changes.values())
- if not drop_fields and has_removed:
- removed_details = []
- for table_name, changes in field_changes.items():
- if changes.get("removed"):
- for col_name in changes["removed"]:
- removed_details.append(f"{table_name}.{col_name}")
- logger.warning(f"Removed fields detected (not included in migration, use --drop to include): {', '.join(removed_details)}")
- # Remove 'removed' from changes since we're not acting on them
- for table_name in field_changes:
- field_changes[table_name]["removed"] = {}
-
- if not new_tables and not any(changes["added"] or changes["changed"] for changes in field_changes.values()):
- if not (drop_fields and has_removed):
- logger.info("No schema changes detected, migration not created")
- return None
-
- # Generate migration file content
- migration_content = generate_migration_content(new_tables, field_changes, router.migrate_dir, name, drop_fields=drop_fields)
-
- # Get next migration number (count existing migration files)
- existing_migrations = [f for f in os.listdir(router.migrate_dir) if f.endswith(".py") and not f.startswith("_")]
- migration_num = len(existing_migrations) + 1
- migration_file = os.path.join(router.migrate_dir, f"{migration_num:03d}_{name}.py")
-
- with open(migration_file, "w") as f:
- f.write(migration_content)
-
- logger.info(f"Created migration: {migration_file}")
- return migration_file
-
- except Exception as e:
- logger.error(f"Failed to create migration: {e}")
- raise
-
-
-def run_migrations(router: Router):
- """Run all pending migrations"""
- try:
- diff = router.diff
- if not diff:
- logger.info("No pending migrations to run")
- return
-
- router.run()
- logger.info("Migrations completed successfully")
- except Exception as e:
- logger.error(f"Failed to run migrations: {e}")
- raise
-
-
-def list_migrations(router: Router):
- """List all migrations"""
- todo = router.todo
- if not todo:
- logger.info("No migration files found")
- return
-
- logger.info("Available migrations:")
- done = set(router.done)
- for migration in todo:
- status = "applied" if migration in done else "pending"
- logger.info(f" [{status}] {migration}")
-
-
-def diff_schema(models: list, db):
- """Show schema differences between models and database"""
- logger.info("Checking schema differences...")
-
- # Tables to ignore (managed by peewee-migrate)
- IGNORE_TABLES = {"migratehistory"}
-
- # Get all model table names
- model_tables = set()
- for model in models:
- table_name = model._meta.table_name
- model_tables.add(table_name)
-
- logger.info(f"Found {len(model_tables)} model tables")
-
- # Get existing tables from database
- cursor = db.execute_sql("SELECT table_name FROM information_schema.tables WHERE table_schema = %s", (db.database,))
- existing_tables = {row[0] for row in cursor.fetchall() if row[0] not in IGNORE_TABLES}
-
- # Find tables that exist in models but not in database
- missing_tables = model_tables - existing_tables
- if missing_tables:
- logger.warning(f"Tables not in database ({len(missing_tables)}): {', '.join(sorted(missing_tables))}")
-
- # Find tables that exist in database but not in models
- extra_tables = existing_tables - model_tables
- if extra_tables:
- logger.info(f"Tables in database but not in models: {', '.join(sorted(extra_tables))}")
-
- # Check field differences for existing tables
- common_tables = model_tables & existing_tables
- if common_tables:
- logger.info(f"\nChecking field differences for {len(common_tables)} existing tables...")
-
- total_added = 0
- total_changed = 0
- total_removed = 0
-
- for model in models:
- table_name = model._meta.table_name
- if table_name not in common_tables:
- continue
-
- # Get model fields
- model_fields = {}
- for field_name, field in model._meta.fields.items():
- if field_name in AUTO_GENERATED_FIELDS:
- continue
- model_fields[field_name] = field
-
- # Get database columns
- db_columns = get_table_columns(db, table_name)
-
- # Compare
- changes = compare_fields(model_fields, db_columns)
-
- if changes["added"]:
- total_added += len(changes["added"])
- field_details = [f"{k}:{v.__class__.__name__}" for k, v in changes["added"].items()]
- logger.info(f" {table_name}: {len(changes['added'])} new field(s) - {field_details}")
-
- if changes["changed"]:
- total_changed += len(changes["changed"])
- field_details = [f"{k}:{v[1].__class__.__name__}" for k, v in changes["changed"].items()]
- logger.info(f" {table_name}: {len(changes['changed'])} changed field(s) - {field_details}")
-
- if changes["removed"]:
- total_removed += len(changes["removed"])
- field_details = [f"{k}:{v['column_type']}" for k, v in changes["removed"].items()]
- logger.warning(f" {table_name}: {len(changes['removed'])} removed field(s) - {field_details}")
-
- logger.info(f"\nSummary: {total_added} new fields, {total_changed} changed fields, {total_removed} removed fields")
-
-
-def main():
- parser = argparse.ArgumentParser(
- description="Database Schema Synchronization Tool using peewee-migrate",
- formatter_class=argparse.RawDescriptionHelpFormatter,
- epilog="""
-Examples:
- # List all migrations
- python db_schema_sync.py --list --host localhost --port 3306 --user root --password xxx --database rag_flow --version v0.27.2
-
- # Create migration from model changes
- python db_schema_sync.py --create --host localhost --port 3306 --user root --password xxx --database rag_flow --version v0.27.2
-
- # Create migration including dropped fields (destructive!)
- python db_schema_sync.py --create --drop --host localhost --port 3306 --user root --password xxx --database rag_flow --version v0.27.2
-
- # Run all pending migrations
- python db_schema_sync.py --migrate --host localhost --port 3306 --user root --password xxx --database rag_flow --version v0.27.2
-
- # Show schema differences
- python db_schema_sync.py --diff --host localhost --port 3306 --user root --password xxx --database rag_flow --version v0.27.2
-""",
- )
-
- # Database connection options
- parser.add_argument("--host", type=str, required=True, help="MySQL host")
- parser.add_argument("--port", type=int, default=3306, help="MySQL port (default: 3306)")
- parser.add_argument("--user", type=str, required=True, help="MySQL user")
- parser.add_argument("--password", type=str, required=True, help="MySQL password")
- parser.add_argument("--database", type=str, required=True, help="MySQL database name")
-
- # Version option
- parser.add_argument("--version", "-v", type=str, required=True, help="Version number in format vxx.xx.xx (e.g., v0.27.2)")
-
- # Action options
- parser.add_argument("--list", "-l", action="store_true", help="List all migrations")
- parser.add_argument("--create", "-c", action="store_true", help="Create migration from model changes (auto-detect)")
- parser.add_argument("--migrate", "-m", action="store_true", help="Run pending migrations")
- parser.add_argument("--diff", "-d", action="store_true", help="Show schema differences")
-
- # Migration options
- parser.add_argument("--name", "-n", type=str, default="auto", help="Migration name")
- parser.add_argument("--drop", action="store_true", help="Include DROP COLUMN for fields removed from models (destructive - will permanently delete data!)")
-
- args = parser.parse_args()
-
- # Validate version format
- if not validate_version(args.version):
- logger.error(f"Invalid version format: {args.version}. Expected format: vxx.xx.xx (e.g., v0.27.2)")
- sys.exit(1)
-
- # Validate at least one action is specified
- if not any([args.list, args.create, args.migrate, args.diff]):
- parser.print_help()
- logger.error("Please specify at least one action: --list, --create, --migrate, or --diff")
- sys.exit(1)
-
- # Convert version to directory name
- version_dir = version_to_dirname(args.version)
- migrate_dir = os.path.join(PROJECT_BASE, "tools", "migrate", version_dir)
-
- logger.info(f"Version: {args.version}")
- logger.info(f"Migration directory: {migrate_dir}")
-
- # Create migration directory if it doesn't exist
- os.makedirs(migrate_dir, exist_ok=True)
-
- # Load database models
- logger.info("Loading database models from api/db/db_models.py...")
- models, _ = load_db_models()
- logger.info(f"Found {len(models)} model classes")
-
- # Create database connection
- db = create_database_connection(host=args.host, port=args.port, user=args.user, password=args.password, database=args.database)
-
- try:
- db.connect()
- logger.info(f"Connected to database: {args.database}")
-
- # Create router
- router = Router(db, migrate_dir, ignore=["basemodel", "base_model", "migratehistory"])
-
- # Execute requested actions
- if args.list:
- list_migrations(router)
-
- if args.create:
- create_migration(router, models, db, args.name, drop_fields=args.drop)
-
- if args.migrate:
- run_migrations(router)
-
- if args.diff:
- diff_schema(models, db)
-
- finally:
- if not db.is_closed():
- db.close()
- logger.info("Database connection closed")
-
- logger.info("Done.")
-
-
-if __name__ == "__main__":
- main()
diff --git a/tools/scripts/mysql_migration.py b/tools/scripts/mysql_migration.py
deleted file mode 100644
index 0cdaeaadcd..0000000000
--- a/tools/scripts/mysql_migration.py
+++ /dev/null
@@ -1,2286 +0,0 @@
-#
-# Copyright 2026 The InfiniFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-"""
-MySQL Data Migration Script
-
-This script provides a flexible MySQL data migration tool that supports:
-1. MySQL configuration via config file or command line arguments
-2. Direct peewee operations without importing api.db.services
-3. Configurable migration stages via command line
-4. Migration logging with table names, row counts, and duration
-"""
-
-import argparse
-import json
-import logging
-import os
-import sys
-import time
-import uuid
-
-from packaging.version import InvalidVersion, Version
-from peewee import (
- BigIntegerField,
- CharField,
- DateTimeField,
- IntegerField,
- Model,
- MySQLDatabase,
- PrimaryKeyField,
- TextField,
-)
-from playhouse.migrate import MySQLMigrator
-
-# Add project root to path for imports
-PROJECT_BASE = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-sys.path.insert(0, PROJECT_BASE)
-
-# Configure logging
-logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")
-logger = logging.getLogger(__name__)
-
-
-MIGRATION_DB_VERSION_MARKER = "mysql_migration.database.version"
-
-
-class MigrationConfig:
- """Configuration for MySQL connection"""
-
- def __init__(self, host: str = "localhost", port: int = 3306, user: str = "root", password: str = "", database: str = "rag_flow"):
- self.host = host
- self.port = port
- self.user = user
- self.password = password
- self.database = database
-
- @classmethod
- def from_config_file(cls, config_path: str) -> "MigrationConfig":
- """Load configuration from YAML config file"""
- try:
- from ruamel.yaml import YAML
-
- yaml = YAML(typ="safe", pure=True)
-
- with open(config_path, "r") as f:
- config = yaml.load(f)
-
- # Try to get database config
- db_config = config.get("database", config.get("mysql", {}))
-
- return cls(
- host=db_config.get("host", "localhost"),
- port=db_config.get("port", 3306),
- user=db_config.get("user", "root"),
- password=db_config.get("password", ""),
- database=db_config.get("name", db_config.get("database", "rag_flow")),
- )
- except Exception as e:
- logger.warning(f"Failed to load config file: {e}, using defaults")
- return cls()
-
-
-class MigrationStats:
- """Track migration statistics"""
-
- def __init__(self):
- self.tables_operated = []
- self.rows_processed = 0
- self.start_time = None
- self.end_time = None
- self.stage_stats = []
-
- def start(self):
- self.start_time = time.time()
-
- def end(self):
- self.end_time = time.time()
-
- def add_stage_stats(self, stage_name: str, tables: list, rows: int, duration: float):
- self.stage_stats.append({"stage": stage_name, "tables": tables, "rows": rows, "duration": duration})
- self.tables_operated.extend(tables)
- self.rows_processed += rows
-
- def print_summary(self):
- duration = self.end_time - self.start_time if self.end_time and self.start_time else 0
- logger.info("=" * 60)
- logger.info("Migration Summary")
- logger.info("=" * 60)
- logger.info(f"Total Duration: {duration:.2f}s")
- logger.info(f"Total Rows Processed: {self.rows_processed}")
- logger.info(f"Tables Operated: {', '.join(set(self.tables_operated))}")
- logger.info("-" * 60)
- logger.info("Stage Details:")
- for stat in self.stage_stats:
- logger.info(f" [{stat['stage']}] Tables: {', '.join(stat['tables'])}, Rows: {stat['rows']}, Duration: {stat['duration']:.2f}s")
- logger.info("=" * 60)
-
-
-class MigrationDatabase:
- """Database wrapper for migrations"""
-
- def __init__(self, config: MigrationConfig):
- self.config = config
- self.db = MySQLDatabase(config.database, host=config.host, port=config.port, user=config.user, password=config.password, charset="utf8mb4")
- self.migrator = MySQLMigrator(self.db)
-
- def connect(self):
- self.db.connect()
- logger.info(f"Connected to MySQL database: {self.config.database}")
-
- def close(self):
- if not self.db.is_closed():
- self.db.close()
- logger.info("Database connection closed")
-
- def execute_sql(self, sql: str, params=None):
- return self.db.execute_sql(sql, params)
-
- def table_exists(self, table_name: str) -> bool:
- cursor = self.execute_sql("SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = %s AND table_name = %s", (self.config.database, table_name))
- return cursor.fetchone()[0] > 0
-
- def column_exists(self, table_name: str, column_name: str) -> bool:
- cursor = self.execute_sql("SELECT COUNT(*) FROM information_schema.columns WHERE table_schema = %s AND table_name = %s AND column_name = %s", (self.config.database, table_name, column_name))
- return cursor.fetchone()[0] > 0
-
- def get_column_type(self, table_name: str, column_name: str) -> str | None:
- """Get the DATA_TYPE of a column from information_schema, returns None if column does not exist"""
- cursor = self.execute_sql("SELECT DATA_TYPE FROM information_schema.columns WHERE table_schema = %s AND table_name = %s AND column_name = %s", (self.config.database, table_name, column_name))
- row = cursor.fetchone()
- return row[0] if row else None
-
- def get_system_setting_value(self, name: str) -> str | None:
- if not self.table_exists("system_settings"):
- logger.info("Table 'system_settings' does not exist, migration marker is unavailable")
- return None
- cursor = self.execute_sql(
- "SELECT `value` FROM `system_settings` WHERE `name` = %s",
- (name,),
- )
- row = cursor.fetchone()
- return row[0] if row else None
-
- def upsert_system_setting(self, name: str, value: str, source: str = "migration", data_type: str = "string"):
- if not self.table_exists("system_settings"):
- logger.warning("Table 'system_settings' does not exist, migration marker was not saved")
- return
-
- current_ts = int(time.time())
- self.execute_sql(
- """
- INSERT INTO `system_settings`
- (`name`, `source`, `data_type`, `value`, `create_time`, `create_date`, `update_time`, `update_date`)
- VALUES (%s, %s, %s, %s, %s, FROM_UNIXTIME(%s), %s, FROM_UNIXTIME(%s))
- ON DUPLICATE KEY UPDATE
- `source` = VALUES(`source`),
- `data_type` = VALUES(`data_type`),
- `value` = VALUES(`value`),
- `update_time` = VALUES(`update_time`),
- `update_date` = VALUES(`update_date`)
- """,
- (
- name,
- source,
- data_type,
- value,
- current_ts * 1000,
- current_ts,
- current_ts * 1000,
- current_ts,
- ),
- )
-
- def get_database_version(self) -> str | None:
- return self.get_system_setting_value(MIGRATION_DB_VERSION_MARKER)
-
- def set_database_version(self, version: str):
- self.upsert_system_setting(MIGRATION_DB_VERSION_MARKER, version)
-
-
-def parse_migration_version(version: str | None) -> Version | None:
- if not version:
- return None
- normalized = version.strip()
- if normalized.startswith(("v", "V")):
- normalized = normalized[1:]
- try:
- return Version(normalized)
- except InvalidVersion:
- logger.warning("Invalid migration version format: %s", version)
- return None
-
-
-def should_skip_migration(current_db_version: str | None, target_version: str) -> bool:
- current = parse_migration_version(current_db_version)
- target = parse_migration_version(target_version)
- if current is None or target is None:
- return False
- return current >= target
-
-
-# Define model classes for migration (not importing from api.db.db_models)
-class BaseModel(Model):
- """Base model for migration tables"""
-
- create_time = BigIntegerField(null=True, index=True)
- create_date = DateTimeField(null=True, index=True)
- update_time = BigIntegerField(null=True, index=True)
- update_date = DateTimeField(null=True, index=True)
-
- class Meta:
- database = None # Will be set dynamically
-
-
-class TenantLLM(BaseModel):
- """Tenant LLM model (source table)"""
-
- id = PrimaryKeyField()
- tenant_id = CharField(max_length=32, null=False, index=True)
- llm_factory = CharField(max_length=128, null=False, index=True)
- model_type = CharField(max_length=128, null=True, index=True)
- llm_name = CharField(max_length=128, null=True, default="", index=True)
- api_key = TextField(null=True)
- api_base = CharField(max_length=255, null=True)
- max_tokens = IntegerField(default=8192, index=True)
- used_tokens = IntegerField(default=0, index=True)
- status = CharField(max_length=1, null=False, default="1", index=True)
-
- class Meta:
- table_name = "tenant_llm"
- database = None
-
-
-class TenantModelProvider(BaseModel):
- """Tenant Model Provider model (target table)"""
-
- id = CharField(max_length=32, primary_key=True)
- provider_name = CharField(max_length=128, null=False, index=True)
- tenant_id = CharField(max_length=32, null=False, index=True)
-
- class Meta:
- table_name = "tenant_model_provider"
- database = None
-
-
-class MigrationStage:
- """Base class for migration stages"""
-
- name = "base_stage"
- description = "Base migration stage"
- source_tables = []
- target_tables = []
-
- def __init__(self, db: MigrationDatabase, dry_run: bool = True, create_table_only: bool = False):
- self.db = db
- self.dry_run = dry_run
- self.create_table_only = create_table_only
- self._noop_completes_migration = False
-
- def check(self) -> bool:
- """Check if migration is needed"""
- raise NotImplementedError
-
- def execute(self) -> tuple[int, list]:
- """Execute migration, returns (rows_affected, tables_operated)"""
- raise NotImplementedError
-
- def create_target_table(self):
- """Create target table (override in subclass if needed)"""
- pass
-
- def mark_noop_completes_migration(self):
- self._noop_completes_migration = True
-
- def noop_completes_migration(self) -> bool:
- return self._noop_completes_migration
-
-
-class TenantModelProviderStage(MigrationStage):
- """Migrate tenant_llm to tenant_model_provider"""
-
- name = "tenant_model_provider"
- description = "Migrate tenant_llm.llm_factory to tenant_model_provider.provider_name"
- source_tables = ["tenant_llm"]
- target_tables = ["tenant_model_provider"]
-
- def current_timestamp(self) -> int:
- return int(time.time())
-
- def generate_uuid(self) -> str:
- """Generate 32-character UUID1"""
- return uuid.uuid1().hex
-
- def check(self) -> bool:
- """Check if migration is needed"""
- # Check if source table exists
- if not self.db.table_exists("tenant_llm"):
- logger.warning("Source table 'tenant_llm' does not exist")
- return False
-
- # Check if target table exists
- if not self.db.table_exists("tenant_model_provider"):
- if self.dry_run:
- logger.info("[DRY RUN] Target table 'tenant_model_provider' does not exist. Use --execute to create and populate the table.")
- return False
- logger.info("Target table 'tenant_model_provider' does not exist, will create")
- return True
-
- # Check if there's data to migrate
- cursor = self.db.execute_sql(
- "SELECT COUNT(*) FROM tenant_llm t1 WHERE NOT EXISTS ( SELECT 1 FROM tenant_model_provider t2 WHERE t2.tenant_id = t1.tenant_id AND t2.provider_name = t1.llm_factory)"
- )
- count = cursor.fetchone()[0]
-
- if count == 0:
- self.mark_noop_completes_migration()
- logger.info("No new data to migrate from tenant_llm to tenant_model_provider")
- return False
-
- logger.info(f"Found {count} rows to migrate from tenant_llm to tenant_model_provider")
- return True
-
- def execute(self) -> tuple[int, list]:
- """Execute migration"""
- current_ts = self.current_timestamp()
- rows_inserted = 0
-
- # Check if target table exists
- if not self.db.table_exists("tenant_model_provider"):
- if self.dry_run:
- logger.info("[DRY RUN] Target table 'tenant_model_provider' does not exist. Use --execute to create and populate the table.")
- return 0, []
- logger.info("Target table 'tenant_model_provider' does not exist, will create")
- self.create_target_table()
-
- # If create_table_only mode, skip data migration
- if self.create_table_only:
- logger.info("[CREATE TABLE ONLY] Target table created/verified, skipping data migration")
- return 0, self.target_tables
-
- # Get distinct tenant_id, llm_factory pairs that don't exist in target
- cursor = self.db.execute_sql(
- "SELECT DISTINCT tenant_id, llm_factory FROM tenant_llm t1 "
- "WHERE NOT EXISTS ("
- " SELECT 1 FROM tenant_model_provider t2 "
- " WHERE t2.tenant_id = t1.tenant_id AND t2.provider_name = t1.llm_factory"
- ")"
- )
-
- records = cursor.fetchall()
-
- if not records:
- logger.info("No records to migrate")
- return 0, []
-
- logger.info(f"Migrating {len(records)} unique tenant_id/llm_factory pairs...")
-
- if self.dry_run:
- logger.info(f"[DRY RUN] Would insert {len(records)} records")
- return len(records), self.target_tables
-
- # Insert records in batches with parameterized SQL to avoid quote breakage/injection
- batch_size = 100
- for i in range(0, len(records), batch_size):
- batch = records[i : i + batch_size]
- placeholders = []
- params = []
- for tenant_id, llm_factory in batch:
- record_id = self.generate_uuid()
- placeholders.append("(%s, %s, %s, %s, FROM_UNIXTIME(%s), %s, FROM_UNIXTIME(%s))")
- params.extend(
- [
- record_id,
- llm_factory,
- tenant_id,
- current_ts * 1000,
- current_ts,
- current_ts * 1000,
- current_ts,
- ]
- )
- insert_sql = f"""
- INSERT INTO tenant_model_provider
- (id, provider_name, tenant_id, create_time, create_date, update_time, update_date)
- VALUES {", ".join(placeholders)}
- """
- self.db.execute_sql(insert_sql, params)
- rows_inserted += len(batch)
- logger.info(f"Inserted batch {i // batch_size + 1}: {len(batch)} records")
-
- return rows_inserted, self.target_tables
-
- def create_target_table(self):
- """Create tenant_model_provider table"""
- create_sql = """
- CREATE TABLE IF NOT EXISTS tenant_model_provider (
- id VARCHAR(32) NOT NULL PRIMARY KEY,
- provider_name VARCHAR(128) NOT NULL,
- tenant_id VARCHAR(32) NOT NULL,
- create_time BIGINT,
- create_date DATETIME,
- update_time BIGINT,
- update_date DATETIME,
- INDEX idx_provider_name (provider_name),
- INDEX idx_tenant_id (tenant_id),
- UNIQUE INDEX idx_tenant_provider_unique (tenant_id, provider_name)
- ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
- """
- self.db.execute_sql(create_sql)
- logger.info("Created tenant_model_provider table")
-
-
-class TenantModelInstanceStage(MigrationStage):
- """Migrate tenant_llm to tenant_model_instance"""
-
- name = "tenant_model_instance"
- description = "Migrate tenant_llm to tenant_model_instance with provider_id lookup"
- source_tables = ["tenant_llm", "tenant_model_provider"]
- target_tables = ["tenant_model_instance"]
-
- def current_timestamp(self) -> int:
- return int(time.time())
-
- def generate_uuid(self) -> str:
- """Generate 32-character UUID1"""
- return uuid.uuid1().hex
-
- def check(self) -> bool:
- """Check if migration is needed"""
- # Check if source table exists
- if not self.db.table_exists("tenant_llm"):
- logger.warning("Source table 'tenant_llm' does not exist")
- return False
-
- # Check if tenant_model_provider exists (dependency)
- if not self.db.table_exists("tenant_model_provider"):
- if self.dry_run:
- logger.info("[DRY RUN] Dependency table 'tenant_model_provider' does not exist. Run 'tenant_model_provider' stage first or use --execute.")
- return False
- logger.warning("Dependency table 'tenant_model_provider' does not exist. Please run 'tenant_model_provider' stage first.")
- return False
-
- # Check if target table exists
- if not self.db.table_exists("tenant_model_instance"):
- if self.dry_run:
- logger.info("[DRY RUN] Target table 'tenant_model_instance' does not exist. Use --execute to create and populate the table.")
- return False
- logger.info("Target table 'tenant_model_instance' does not exist, will create")
- return True
-
- # Check if there's data to migrate (distinct by tenant_id, llm_factory, api_key)
- cursor = self.db.execute_sql(
- "SELECT COUNT(*) FROM ("
- " SELECT tl.tenant_id, tl.llm_factory, tl.api_key, tmp.id as provider_id "
- " FROM tenant_llm tl "
- " INNER JOIN tenant_model_provider tmp ON tmp.tenant_id = tl.tenant_id AND tmp.provider_name = tl.llm_factory "
- " WHERE NOT EXISTS ("
- " SELECT 1 FROM tenant_model_instance tmi "
- " WHERE tmi.provider_id = tmp.id AND tmi.api_key = tl.api_key"
- " ) "
- " GROUP BY tl.tenant_id, tl.llm_factory, tl.api_key, tmp.id"
- ") AS distinct_records"
- )
- count = cursor.fetchone()[0]
-
- if count == 0:
- self.mark_noop_completes_migration()
- logger.info("No new data to migrate from tenant_llm to tenant_model_instance")
- return False
-
- logger.info(f"Found {count} rows to migrate from tenant_llm to tenant_model_instance")
- return True
-
- def execute(self) -> tuple[int, list]:
- """Execute migration"""
- current_ts = self.current_timestamp()
- rows_inserted = 0
-
- # Check if tenant_model_provider exists (dependency)
- if not self.db.table_exists("tenant_model_provider"):
- logger.error("Dependency table 'tenant_model_provider' does not exist. Please run 'tenant_model_provider' stage first.")
- return 0, []
-
- # Check if target table exists
- if not self.db.table_exists("tenant_model_instance"):
- if self.dry_run:
- logger.info("[DRY RUN] Target table 'tenant_model_instance' does not exist. Use --execute to create and populate the table.")
- return 0, []
- logger.info("Target table 'tenant_model_instance' does not exist, will create")
- self.create_target_table()
-
- # If create_table_only mode, skip data migration
- if self.create_table_only:
- logger.info("[CREATE TABLE ONLY] Target table created/verified, skipping data migration")
- return 0, self.target_tables
-
- # Get records from tenant_llm with provider_id lookup
- # Group by tenant_id, llm_factory, api_key to get distinct records
- # instance_name = llm_factory, provider_id from tenant_model_provider, api_key from tenant_llm
- cursor = self.db.execute_sql(
- "SELECT tl.tenant_id, tl.llm_factory, tl.api_key, MAX(tl.status) as status, tmp.id as provider_id "
- "FROM tenant_llm tl "
- "INNER JOIN tenant_model_provider tmp ON tmp.tenant_id = tl.tenant_id AND tmp.provider_name = tl.llm_factory "
- "WHERE NOT EXISTS ("
- " SELECT 1 FROM tenant_model_instance tmi "
- " WHERE tmi.provider_id = tmp.id AND tmi.api_key = tl.api_key"
- ") "
- "GROUP BY tl.tenant_id, tl.llm_factory, tl.api_key, tmp.id"
- )
-
- records = cursor.fetchall()
-
- if not records:
- logger.info("No records to migrate")
- return 0, []
-
- # Deduplicate records where api_keys differ only by is_tools encoding.
- # When _encode_api_key_config wraps a plain api_key into {"api_key": "...", "is_tools": true/false},
- # multiple tenant_llm rows for the same provider can have logically identical api_keys that
- # only differ in the is_tools field. We merge these by stripping is_tools for comparison.
- records = self._dedup_api_key_records(records)
-
- logger.info(f"Migrating {len(records)} tenant_model_instance records...")
-
- if self.dry_run:
- logger.info(f"[DRY RUN] Would insert {len(records)} records")
- for tenant_id, llm_factory, api_key, status, provider_id in records[:5]:
- logger.info(f" instance_name=default, provider_id={provider_id}, api_key=***")
- if len(records) > 5:
- logger.info(f" ... and {len(records) - 5} more records")
- return len(records), self.target_tables
-
- # Insert records in batches
- batch_size = 100
- for i in range(0, len(records), batch_size):
- batch = records[i : i + batch_size]
- values = []
- for tenant_id, llm_factory, api_key, status, provider_id in batch:
- record_id = self.generate_uuid()
- instance_name = "default"
- api_key_escaped = api_key.replace("'", "''") if api_key else ""
- status_val = "active" if status in ["1", "active", "enable"] else "inactive"
- values.append(
- f"('{record_id}', '{instance_name}', '{provider_id}', "
- f"'{api_key_escaped}', '{status_val}', "
- f"{current_ts * 1000}, FROM_UNIXTIME({current_ts}), "
- f"{current_ts * 1000}, FROM_UNIXTIME({current_ts}))"
- )
-
- insert_sql = f"""
- INSERT INTO tenant_model_instance
- (id, instance_name, provider_id, api_key, status, create_time, create_date, update_time, update_date)
- VALUES {", ".join(values)}
- """
- self.db.execute_sql(insert_sql)
- rows_inserted += len(batch)
- logger.info(f"Inserted batch {i // batch_size + 1}: {len(batch)} records")
-
- return rows_inserted, self.target_tables
-
- @staticmethod
- def _strip_is_tools_from_api_key(api_key: str) -> str:
- """Strip is_tools from api_key for dedup comparison.
-
- Handles three api_key formats:
- 1. Plain string (e.g. "sk-xxx" or "x") — returned as-is.
- 2. JSON with only {"api_key": "...", "is_tools": true/false} — extract the inner api_key value.
- 3. JSON with factory-specific fields + optional "is_tools" — remove only the "is_tools" key.
-
- For format 3, the factory-specific JSON structures are:
- VolcEngine: {"ark_api_key": ..., "endpoint_id": ...}
- Tencent Cloud: {"tencent_cloud_sid": ..., "tencent_cloud_sk": ...}
- Bedrock: {"auth_mode": ..., "bedrock_ak": ..., "bedrock_sk": ..., "bedrock_region": ..., "aws_role_arn": ...}
- XunFei Spark (tts): {"spark_app_id": ..., "spark_api_secret": ..., "spark_api_key": ...}
- BaiduYiyan: {"yiyan_ak": ..., "yiyan_sk": ...}
- Fish Audio: {"fish_audio_ak": ..., "fish_audio_refid": ...}
- Google Cloud: {"google_project_id": ..., "google_region": ..., "google_service_account_key": ...}
- Azure-OpenAI: {"api_key": ..., "api_version": ...}
- OpenRouter: {"api_key": ..., "provider_order": ...}
- MinerU: {"api_key": ..., "provider_order": ...}
- PaddleOCR: {"api_key": ..., "provider_order": ...}
- OpenDataLoader: {"api_key": ..., "provider_order": ...}
- """
- if not api_key:
- return api_key
-
- try:
- parsed = json.loads(api_key)
- except (json.JSONDecodeError, TypeError, ValueError):
- return api_key
-
- if not isinstance(parsed, dict):
- return api_key
-
- # Case 2: {"api_key": "...", "is_tools": true/false} — extract inner api_key
- if set(parsed.keys()) <= {"api_key", "is_tools"}:
- return parsed.get("api_key", "")
-
- # Case 3: factory-specific JSON with is_tools appended — remove is_tools key
- if "is_tools" in parsed:
- payload = {k: v for k, v in parsed.items() if k != "is_tools"}
- return json.dumps(payload, sort_keys=True)
-
- # Already a JSON dict without is_tools — return as-is
- return json.dumps(parsed, sort_keys=True)
-
- def _dedup_api_key_records(self, records: list) -> list:
- """Deduplicate records whose api_keys are logically identical after stripping is_tools.
-
- Groups by (tenant_id, llm_factory, provider_id). Within each group, if multiple
- records share the same canonical api_key (with is_tools removed), only one is kept.
- The kept record uses the original api_key value from the first occurrence; is_tools
- information is not needed in tenant_model_instance (it is stored in tenant_model instead).
- """
- from collections import defaultdict
-
- groups = defaultdict(list)
- for rec in records:
- tenant_id, llm_factory, api_key, status, provider_id = rec
- groups[(tenant_id, llm_factory, provider_id)].append(rec)
-
- deduped = []
- dup_count = 0
- for (tenant_id, llm_factory, provider_id), group in groups.items():
- if len(group) <= 1:
- deduped.extend(group)
- continue
-
- # Multiple records in group — dedup by canonical api_key
- seen = {} # canonical_key -> first record
- for rec in group:
- _, _, api_key, _, _ = rec
- canonical = self._strip_is_tools_from_api_key(api_key)
- if canonical not in seen:
- seen[canonical] = rec
- else:
- dup_count += 1
- logger.debug(f"Dedup api_key for tenant={tenant_id}, factory={llm_factory}, provider={provider_id}: keeping '{api_key[:20]}...', dropping '{seen[canonical][2][:20]}...'")
- deduped.extend(seen.values())
-
- if dup_count > 0:
- logger.info(f"Deduplicated {dup_count} api_key records (is_tools-only differences)")
-
- return deduped
-
- def create_target_table(self):
- """Create tenant_model_instance table"""
- create_sql = """
- CREATE TABLE IF NOT EXISTS tenant_model_instance (
- id VARCHAR(32) NOT NULL PRIMARY KEY,
- instance_name VARCHAR(128) NOT NULL,
- provider_id VARCHAR(32) NOT NULL,
- api_key VARCHAR(512) NOT NULL,
- status VARCHAR(32) DEFAULT 'active',
- extra VARCHAR(512) DEFAULT '{}',
- create_time BIGINT,
- create_date DATETIME,
- update_time BIGINT,
- update_date DATETIME,
- INDEX idx_provider_id (provider_id)
- ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
- """
- self.db.execute_sql(create_sql)
- logger.info("Created tenant_model_instance table")
-
-
-class TenantModelStage(MigrationStage):
- """Migrate tenant_llm to tenant_model"""
-
- name = "tenant_model"
- description = "Migrate tenant_llm to tenant_model (status='0' records, plus status='1' for empty-llm factories)"
- source_tables = ["tenant_llm", "tenant_model_provider", "tenant_model_instance"]
- target_tables = ["tenant_model"]
-
- @staticmethod
- def _get_empty_llm_factories() -> list[str]:
- """Load factory names whose llm field is an empty list from conf/llm_factories.json"""
- conf_path = os.path.join(PROJECT_BASE, "conf", "llm_factories.json")
- with open(conf_path, "r") as f:
- data = json.load(f)
- factories = []
- for key, items in data.items():
- if isinstance(items, list):
- for item in items:
- if isinstance(item, dict):
- llm = item.get("llm")
- if isinstance(llm, list) and len(llm) == 0:
- factories.append(item["name"])
- return factories
-
- def _build_status_condition(self) -> str:
- """Build SQL WHERE condition for status filtering"""
- empty_factories = self._get_empty_llm_factories()
- if empty_factories:
- placeholders = ", ".join(f"'{f}'" for f in empty_factories)
- return f"(tl.status = '0' OR (tl.status = '1' AND tl.llm_factory IN ({placeholders})))"
- return "tl.status = '0'"
-
- def current_timestamp(self) -> int:
- return int(time.time())
-
- def generate_uuid(self) -> str:
- """Generate 32-character UUID1"""
- return uuid.uuid1().hex
-
- def check(self) -> bool:
- """Check if migration is needed"""
- # Check if source table exists
- if not self.db.table_exists("tenant_llm"):
- logger.warning("Source table 'tenant_llm' does not exist")
- return False
-
- # Check if tenant_model_provider exists (dependency)
- if not self.db.table_exists("tenant_model_provider"):
- if self.dry_run:
- logger.info("[DRY RUN] Dependency table 'tenant_model_provider' does not exist. Run 'tenant_model_provider' stage first or use --execute.")
- return False
- logger.warning("Dependency table 'tenant_model_provider' does not exist. Please run 'tenant_model_provider' stage first.")
- return False
-
- # Check if tenant_model_instance exists (dependency)
- if not self.db.table_exists("tenant_model_instance"):
- if self.dry_run:
- logger.info("[DRY RUN] Dependency table 'tenant_model_instance' does not exist. Run 'tenant_model_instance' stage first or use --execute.")
- return False
- logger.warning("Dependency table 'tenant_model_instance' does not exist. Please run 'tenant_model_instance' stage first.")
- return False
-
- # Check if target table exists
- if not self.db.table_exists("tenant_model"):
- if self.dry_run:
- logger.info("[DRY RUN] Target table 'tenant_model' does not exist. Use --execute to create and populate the table.")
- return False
- logger.info("Target table 'tenant_model' does not exist, will create")
- return True
-
- status_condition = self._build_status_condition()
-
- # Check if there's data to migrate
- # We cannot JOIN tenant_model_instance on api_key directly because the instance
- # stage deduped api_keys (stripping is_tools), so a plain SQL equality won't
- # match records whose api_key was merged. Count at the provider level instead.
- # Use a constant instead of tl.id: very old tenant_llm tables may lack the
- # id column, and the count only cares about the number of rows.
- cursor = self.db.execute_sql(
- f"SELECT COUNT(*) FROM ("
- f" SELECT 1 "
- f" FROM tenant_llm tl "
- f" INNER JOIN tenant_model_provider tmp ON tmp.tenant_id = tl.tenant_id AND tmp.provider_name = tl.llm_factory "
- f" WHERE {status_condition} "
- f") AS source_records"
- )
- count = cursor.fetchone()[0]
-
- if count == 0:
- self.mark_noop_completes_migration()
- logger.info("No new data to migrate from tenant_llm to tenant_model")
- return False
-
- logger.info(f"Found {count} rows to migrate from tenant_llm to tenant_model")
- return True
-
- def execute(self) -> tuple[int, list]:
- """Execute migration"""
- current_ts = self.current_timestamp()
- rows_inserted = 0
-
- # Check if tenant_model_provider exists (dependency)
- if not self.db.table_exists("tenant_model_provider"):
- logger.error("Dependency table 'tenant_model_provider' does not exist. Please run 'tenant_model_provider' stage first.")
- return 0, []
-
- # Check if tenant_model_instance exists (dependency)
- if not self.db.table_exists("tenant_model_instance"):
- logger.error("Dependency table 'tenant_model_instance' does not exist. Please run 'tenant_model_instance' stage first.")
- return 0, []
-
- # Check if target table exists
- if not self.db.table_exists("tenant_model"):
- if self.dry_run:
- logger.info("[DRY RUN] Target table 'tenant_model' does not exist. Use --execute to create and populate the table.")
- return 0, []
- logger.info("Target table 'tenant_model' does not exist, will create")
- self.create_target_table()
-
- # If create_table_only mode, skip data migration
- if self.create_table_only:
- logger.info("[CREATE TABLE ONLY] Target table created/verified, skipping data migration")
- return 0, self.target_tables
-
- # Very old tenant_llm tables may lack the id column, while the migration
- # SELECT below relies on it (source_id for audit logging). Add an
- # auto-increment id column in place before running the data comparison
- # and inserts. Fail hard when it cannot be ensured: silently returning
- # would let the runner mark this stage as completed without migrating.
- if not self._ensure_tenant_llm_id_column():
- raise RuntimeError("tenant_llm has no 'id' column and it could not be added; aborting tenant_model stage")
-
- status_condition = self._build_status_condition()
-
- # Load all tenant_model_instance records into memory for Python-level matching.
- # We cannot JOIN on api_key in SQL because the instance stage deduped api_keys
- # (stripping is_tools), so a plain SQL equality won't match records whose
- # api_key was merged during dedup.
- instance_lookup = self._build_instance_lookup()
-
- # Get records from tenant_llm with provider_id lookup (no instance JOIN)
- # Migrate status='0' records, plus status='1' for empty-llm factories
- cursor = self.db.execute_sql(
- f"SELECT tl.id, tl.llm_name, tmp.id as provider_id, "
- f" tl.model_type, tl.status, tl.api_key "
- f"FROM tenant_llm tl "
- f"INNER JOIN tenant_model_provider tmp ON tmp.tenant_id = tl.tenant_id AND tmp.provider_name = tl.llm_factory "
- f"WHERE {status_condition} "
- f"AND NOT EXISTS ("
- f" SELECT 1 FROM tenant_model tm "
- f" WHERE tm.provider_id = tmp.id AND tm.model_name = tl.llm_name"
- f")"
- )
-
- records = cursor.fetchall()
-
- if not records:
- logger.info("No records to migrate")
- return 0, []
-
- # Resolve instance_id for each record using Python-level canonical matching
- resolved_records = self._resolve_instance_ids(records, instance_lookup)
-
- if not resolved_records:
- logger.info("No records with matching instance_id to migrate")
- return 0, []
-
- logger.info(f"Migrating {len(resolved_records)} tenant_model records...")
-
- if self.dry_run:
- logger.info(f"[DRY RUN] Would insert {len(resolved_records)} records")
- for source_id, llm_name, provider_id, instance_id, model_type, status, api_key in resolved_records[:5]:
- logger.info(f" model_name={llm_name}, provider_id={provider_id}, instance_id={instance_id}, model_type={model_type}")
- if len(resolved_records) > 5:
- logger.info(f" ... and {len(resolved_records) - 5} more records")
- return len(resolved_records), self.target_tables
-
- # Insert records in batches
- batch_size = 100
- for i in range(0, len(resolved_records), batch_size):
- batch = resolved_records[i : i + batch_size]
- values = []
- for source_id, llm_name, provider_id, instance_id, model_type, status, api_key in batch:
- record_id = self.generate_uuid()
- model_name_escaped = llm_name.replace("'", "''") if llm_name else ""
- model_type_escaped = model_type.replace("'", "''") if model_type else ""
- status_val = "active" if status in ["1", "active", "enable"] else "inactive"
- # Extract is_tools from api_key JSON and put it in extra
- extra = self._extract_extra_from_api_key(api_key)
- extra_escaped = extra.replace("'", "''") if extra else "{}"
- values.append(
- f"('{record_id}', '{model_name_escaped}', '{provider_id}', "
- f"'{instance_id}', '{model_type_escaped}', '{status_val}', "
- f"'{extra_escaped}', "
- f"{current_ts * 1000}, FROM_UNIXTIME({current_ts}), "
- f"{current_ts * 1000}, FROM_UNIXTIME({current_ts}))"
- )
-
- insert_sql = f"""
- INSERT INTO tenant_model
- (id, model_name, provider_id, instance_id, model_type, status, extra,
- create_time, create_date, update_time, update_date)
- VALUES {", ".join(values)}
- """
- self.db.execute_sql(insert_sql)
- rows_inserted += len(batch)
- logger.info(f"Inserted batch {i // batch_size + 1}: {len(batch)} records")
-
- return rows_inserted, self.target_tables
-
- def _ensure_tenant_llm_id_column(self) -> bool:
- """Ensure tenant_llm has an id column, adding an auto-increment one if missing.
-
- Very old tenant_llm tables may predate the id column, while the migration
- SELECT relies on it (tl.id is carried as source_id for audit logging).
- The id is only used as a unique audit key, so it is added as an
- AUTO_INCREMENT UNIQUE column rather than PRIMARY KEY: old tables may
- already have another primary key. MySQL back-fills existing rows with
- sequential values when adding an AUTO_INCREMENT column.
-
- Returns:
- True if the column exists (or was added successfully), False otherwise.
- """
- if self.db.column_exists("tenant_llm", "id"):
- return True
-
- logger.warning("Source table 'tenant_llm' has no 'id' column, adding auto-increment id column")
- if self.dry_run:
- logger.info("[DRY RUN] Would ALTER TABLE tenant_llm to add auto-increment id column")
- return False
-
- try:
- self.db.execute_sql("ALTER TABLE tenant_llm ADD COLUMN id BIGINT AUTO_INCREMENT UNIQUE")
- except Exception as e:
- logger.error(f"Failed to add auto-increment id column to tenant_llm: {e}")
- return False
- logger.info("Added auto-increment id column to tenant_llm")
- return True
-
- def _build_instance_lookup(self) -> dict:
- """Load all tenant_model_instance records, indexed by (provider_id, canonical_api_key).
-
- The canonical_api_key is computed by stripping is_tools from the stored api_key,
- matching the dedup logic used during the instance migration stage.
-
- Returns:
- dict mapping (provider_id, canonical_api_key) -> instance_id
- """
- cursor = self.db.execute_sql("SELECT id, provider_id, api_key FROM tenant_model_instance")
- lookup = {}
- for instance_id, provider_id, api_key in cursor.fetchall():
- canonical = TenantModelInstanceStage._strip_is_tools_from_api_key(api_key)
- lookup[(provider_id, canonical)] = instance_id
- logger.info(f"Loaded {len(lookup)} instance records for lookup")
- return lookup
-
- @staticmethod
- def _resolve_instance_ids(records: list, instance_lookup: dict) -> list:
- """Resolve instance_id for each tenant_llm record using canonical api_key matching.
-
- Args:
- records: list of tuples (source_id, llm_name, provider_id, model_type, status, api_key)
- instance_lookup: dict mapping (provider_id, canonical_api_key) -> instance_id
-
- Returns:
- list of tuples (source_id, llm_name, provider_id, instance_id, model_type, status, api_key)
- Only records with a matching instance_id are included.
- """
- resolved = []
- skipped = 0
- for source_id, llm_name, provider_id, model_type, status, api_key in records:
- canonical = TenantModelInstanceStage._strip_is_tools_from_api_key(api_key)
- instance_id = instance_lookup.get((provider_id, canonical))
- if instance_id:
- resolved.append((source_id, llm_name, provider_id, instance_id, model_type, status, api_key))
- else:
- skipped += 1
- # Don't include the API key (even truncated) in the log:
- # CodeQL flags this as clear-text-logging-sensitive-data,
- # and the first 30 chars of an API key often carry enough
- # entropy to be useful to an attacker who reads the log.
- logger.warning(
- "No matching instance for tenant_llm id=%s provider_id=%s llm_name=%s",
- source_id,
- provider_id,
- llm_name,
- )
-
- if skipped > 0:
- logger.warning(f"Skipped {skipped} records with no matching instance_id")
-
- return resolved
-
- @staticmethod
- def _extract_extra_from_api_key(api_key: str) -> str:
- """Extract is_tools from api_key JSON and return an extra JSON string for tenant_model.
-
- If api_key is a JSON dict containing "is_tools": true, return '{"is_tools": true}'.
- Otherwise return '{}' (empty dict).
- """
- if not api_key:
- return "{}"
-
- try:
- parsed = json.loads(api_key)
- except (json.JSONDecodeError, TypeError, ValueError):
- return "{}"
-
- if not isinstance(parsed, dict):
- return "{}"
-
- if parsed.get("is_tools") is True:
- return json.dumps({"is_tools": True})
-
- return "{}"
-
- def create_target_table(self):
- """Create tenant_model table"""
- create_sql = """
- CREATE TABLE IF NOT EXISTS tenant_model (
- id VARCHAR(32) NOT NULL PRIMARY KEY,
- model_name VARCHAR(128),
- provider_id VARCHAR(32) NOT NULL,
- instance_id VARCHAR(32) NOT NULL,
- model_type VARCHAR(32) NOT NULL,
- status VARCHAR(32) DEFAULT 'active',
- extra VARCHAR(1024) DEFAULT '{}',
- create_time BIGINT,
- create_date DATETIME,
- update_time BIGINT,
- update_date DATETIME,
- INDEX idx_instance_id (instance_id)
- ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
- """
- self.db.execute_sql(create_sql)
- logger.info("Created tenant_model table")
-
-
-class ModelIdConfigStage(MigrationStage):
- """Normalize stored model IDs from model@provider to model@default@provider."""
-
- name = "model_id_config"
- description = "Normalize stored model IDs in config columns to model@default@provider"
- source_tables = [
- "tenant",
- "knowledgebase",
- "document",
- "dialog",
- "memory",
- "search",
- "user_canvas",
- "canvas_template",
- "user_canvas_version",
- "api_4_conversation",
- "pipeline_operation_log",
- "connector",
- "evaluation_runs",
- ]
- target_tables = source_tables
-
- model_id_fields = {
- "llm_id",
- "embd_id",
- "embedding_model",
- "rerank_id",
- "asr_id",
- "img2txt_id",
- "tts_id",
- "ocr_id",
- }
- search_config_model_id_fields = {"chat_id"}
- scan_batch_size = 500
- string_columns = {
- "tenant": ("llm_id", "embd_id", "asr_id", "img2txt_id", "rerank_id", "tts_id", "ocr_id"),
- "knowledgebase": ("embd_id",),
- "dialog": ("llm_id", "rerank_id"),
- "memory": ("embd_id", "llm_id"),
- }
- json_columns = {
- "knowledgebase": ("parser_config",),
- "document": ("parser_config",),
- "search": ("search_config",),
- "user_canvas": ("dsl",),
- "canvas_template": ("dsl",),
- "user_canvas_version": ("dsl",),
- "api_4_conversation": ("dsl",),
- "pipeline_operation_log": ("dsl",),
- "connector": ("config",),
- "evaluation_runs": ("config_snapshot",),
- }
-
- def normalize_model_id(self, value):
- if not isinstance(value, str) or not value:
- return value, False
-
- parts = value.split("@")
- if len(parts) != 2:
- return value, False
-
- model_name, provider_name = parts
- if not model_name or not provider_name:
- return value, False
-
- return f"{model_name}@default@{provider_name}", True
-
- def normalize_config(self, value, path=None):
- path = path or ()
-
- if isinstance(value, dict):
- changed = False
- normalized = {}
- for key, item in value.items():
- key_path = path + (str(key),)
- should_normalize = key in self.model_id_fields or (key in self.search_config_model_id_fields and "search_config" in path)
- if should_normalize:
- normalized_item, item_changed = self.normalize_model_id(item)
- else:
- normalized_item, item_changed = self.normalize_config(item, key_path)
- normalized[key] = normalized_item
- changed = changed or item_changed
- return normalized, changed
-
- if isinstance(value, list):
- changed = False
- normalized = []
- for index, item in enumerate(value):
- normalized_item, item_changed = self.normalize_config(item, path + (str(index),))
- normalized.append(normalized_item)
- changed = changed or item_changed
- return normalized, changed
-
- return value, False
-
- def existing_columns(self, table_columns):
- for table_name, columns in table_columns.items():
- if not self.db.table_exists(table_name):
- logger.info("Table '%s' does not exist, skipping", table_name)
- continue
- for column_name in columns:
- if not self.db.column_exists(table_name, column_name):
- logger.info("Column '%s.%s' does not exist, skipping", table_name, column_name)
- continue
- yield table_name, column_name
-
- def load_json_value(self, raw_value, table_name, column_name, row_id):
- if raw_value in (None, ""):
- return None, False
- if isinstance(raw_value, (dict, list)):
- return raw_value, True
- try:
- return json.loads(raw_value), True
- except (TypeError, json.JSONDecodeError):
- logger.warning(
- "Failed to parse JSON in %s.%s id=%s, skipping",
- table_name,
- column_name,
- row_id,
- )
- return None, False
-
- def iter_string_changes(self):
- for table_name, column_name in self.existing_columns(self.string_columns):
- cursor = self.db.execute_sql(
- f"SELECT id, `{column_name}` FROM `{table_name}` WHERE `{column_name}` IS NOT NULL AND `{column_name}` != '' AND `{column_name}` LIKE %s",
- ("%@%",),
- )
- while True:
- rows = cursor.fetchmany(self.scan_batch_size)
- if not rows:
- break
- for row_id, value in rows:
- normalized, changed = self.normalize_model_id(value)
- if changed:
- yield table_name, column_name, row_id, normalized
-
- def iter_json_changes(self):
- for table_name, column_name in self.existing_columns(self.json_columns):
- cursor = self.db.execute_sql(
- f"SELECT id, `{column_name}` FROM `{table_name}` WHERE `{column_name}` IS NOT NULL AND `{column_name}` != '' AND `{column_name}` LIKE %s",
- ("%@%",),
- )
- while True:
- rows = cursor.fetchmany(self.scan_batch_size)
- if not rows:
- break
- for row_id, raw_value in rows:
- config, loaded = self.load_json_value(raw_value, table_name, column_name, row_id)
- if not loaded:
- continue
- normalized, changed = self.normalize_config(config, (column_name,))
- if changed:
- normalized_json = json.dumps(
- normalized,
- ensure_ascii=False,
- separators=(",", ":"),
- )
- yield table_name, column_name, row_id, normalized_json
-
- def count_changes(self) -> tuple[int, set]:
- rows = 0
- tables = set()
- for table_name, _, _, _ in self.iter_string_changes():
- rows += 1
- tables.add(table_name)
- for table_name, _, _, _ in self.iter_json_changes():
- rows += 1
- tables.add(table_name)
- return rows, tables
-
- def check(self) -> bool:
- rows, tables = self.count_changes()
- if rows == 0:
- self.mark_noop_completes_migration()
- logger.info("No stored model IDs need normalization")
- return False
- logger.info(
- "Found %s rows to normalize across tables: %s",
- rows,
- ", ".join(sorted(tables)),
- )
- return True
-
- def execute(self) -> tuple[int, list]:
- if self.create_table_only:
- logger.info("[CREATE TABLE ONLY] No tables are created for this data migration")
- return 0, []
-
- rows_updated = 0
- tables_operated = set()
-
- for table_name, column_name, row_id, normalized in self.iter_string_changes():
- tables_operated.add(table_name)
- rows_updated += 1
- if rows_updated <= 10:
- logger.info(
- "%s %s.%s id=%s -> %s",
- "[DRY RUN] Would update" if self.dry_run else "Updating",
- table_name,
- column_name,
- row_id,
- normalized,
- )
- if not self.dry_run:
- self.db.execute_sql(
- f"UPDATE `{table_name}` SET `{column_name}` = %s WHERE id = %s",
- (normalized, row_id),
- )
-
- for table_name, column_name, row_id, normalized_json in self.iter_json_changes():
- tables_operated.add(table_name)
- rows_updated += 1
- if rows_updated <= 10:
- logger.info(
- "%s %s.%s id=%s",
- "[DRY RUN] Would update" if self.dry_run else "Updating",
- table_name,
- column_name,
- row_id,
- )
- if not self.dry_run:
- self.db.execute_sql(
- f"UPDATE `{table_name}` SET `{column_name}` = %s WHERE id = %s",
- (normalized_json, row_id),
- )
-
- if rows_updated > 10:
- logger.info("... and %s more row updates", rows_updated - 10)
-
- if self.dry_run:
- logger.info("[DRY RUN] Would update %s rows", rows_updated)
- else:
- logger.info("Updated %s rows", rows_updated)
-
- return rows_updated, sorted(tables_operated)
-
-
-class TenantModelSeedingStage(MigrationStage):
- """Seed tenant_model table with models from conf/llm_factories.json"""
-
- name = "tenant_model_seeding"
- description = "Seed tenant_model with models from conf/llm_factories.json for existing providers/instances"
- source_tables = ["tenant_model_provider", "tenant_model_instance", "tenant_model"]
- target_tables = ["tenant_model"]
-
- def current_timestamp(self) -> int:
- return int(time.time())
-
- def generate_uuid(self) -> str:
- """Generate 32-character UUID1"""
- return uuid.uuid1().hex
-
- def _load_llm_factories(self) -> list:
- """Load factory_llm_infos from conf/llm_factories.json"""
- conf_path = os.path.join(PROJECT_BASE, "conf", "llm_factories.json")
- with open(conf_path, "r") as f:
- data = json.load(f)
- return data.get("factory_llm_infos", [])
-
- def check(self) -> bool:
- """Check if migration is needed"""
- if not self.db.table_exists("tenant_model_provider"):
- logger.warning("Dependency table 'tenant_model_provider' does not exist")
- return False
-
- if not self.db.table_exists("tenant_model_instance"):
- logger.warning("Dependency table 'tenant_model_instance' does not exist")
- return False
-
- if not self.db.table_exists("tenant_model"):
- if self.dry_run:
- logger.info("[DRY RUN] Target table 'tenant_model' does not exist")
- return False
- return True
-
- # If model_type is already INT, the merge stage has been executed — seeding is not applicable
- model_type_dtype = self.db.get_column_type("tenant_model", "model_type")
- if model_type_dtype and model_type_dtype.lower() == "int":
- self.mark_noop_completes_migration()
- logger.info("tenant_model.model_type is already INT, seeding stage is not applicable (merge already done)")
- return False
-
- # Check if there are any models to seed
- factories = self._load_llm_factories()
- total_missing = 0
- for factory in factories:
- llm_list = factory.get("llm", [])
- if not llm_list:
- continue
- factory_name = factory["name"]
- # Determine the provider_name to query and optional instance extra filter
- if factory_name == "siliconflow_intl":
- provider_name_filter = "SILICONFLOW"
- instance_extra_include = '%"region": "intl"%'
- instance_extra_exclude = None
- elif factory_name == "SILICONFLOW":
- provider_name_filter = "SILICONFLOW"
- instance_extra_include = None
- instance_extra_exclude = '%"region": "intl"%'
- else:
- provider_name_filter = factory_name
- instance_extra_include = None
- instance_extra_exclude = None
- # Query provider for this factory
- cursor = self.db.execute_sql("SELECT id FROM tenant_model_provider WHERE provider_name = %s", (provider_name_filter,))
- providers = cursor.fetchall()
- for (provider_id,) in providers:
- # Query instances for this provider, with optional extra include/exclude filter
- if instance_extra_include and instance_extra_exclude:
- cursor = self.db.execute_sql(
- "SELECT id FROM tenant_model_instance WHERE provider_id = %s AND extra LIKE %s AND extra NOT LIKE %s", (provider_id, instance_extra_include, instance_extra_exclude)
- )
- elif instance_extra_include:
- cursor = self.db.execute_sql("SELECT id FROM tenant_model_instance WHERE provider_id = %s AND extra LIKE %s", (provider_id, instance_extra_include))
- elif instance_extra_exclude:
- cursor = self.db.execute_sql("SELECT id FROM tenant_model_instance WHERE provider_id = %s AND extra NOT LIKE %s", (provider_id, instance_extra_exclude))
- else:
- cursor = self.db.execute_sql("SELECT id FROM tenant_model_instance WHERE provider_id = %s", (provider_id,))
- instances = cursor.fetchall()
- for (instance_id,) in instances:
- for llm in llm_list:
- model_name = llm.get("llm_name", "")
- model_types = llm.get("model_type", "chat")
- if isinstance(model_types, str):
- model_types = [model_types]
- for mt in model_types:
- cursor = self.db.execute_sql(
- "SELECT COUNT(*) FROM tenant_model WHERE provider_id = %s AND instance_id = %s AND model_name = %s AND model_type = %s", (provider_id, instance_id, model_name, mt)
- )
- if cursor.fetchone()[0] == 0:
- total_missing += 1
-
- if total_missing == 0:
- self.mark_noop_completes_migration()
- logger.info("No missing models to seed in tenant_model")
- return False
-
- logger.info(f"Found {total_missing} missing models to seed in tenant_model")
- return True
-
- def execute(self) -> tuple[int, list]:
- """Execute migration"""
- current_ts = self.current_timestamp()
- rows_inserted = 0
-
- if not self.db.table_exists("tenant_model_provider"):
- logger.error("Dependency table 'tenant_model_provider' does not exist")
- return 0, []
-
- if not self.db.table_exists("tenant_model_instance"):
- logger.error("Dependency table 'tenant_model_instance' does not exist")
- return 0, []
-
- if not self.db.table_exists("tenant_model"):
- if self.dry_run:
- logger.info("[DRY RUN] Target table 'tenant_model' does not exist")
- return 0, []
- logger.info("Target table 'tenant_model' does not exist, will create")
- self.create_target_table()
-
- if self.create_table_only:
- logger.info("[CREATE TABLE ONLY] Target table created/verified, skipping data seeding")
- return 0, self.target_tables
-
- # Pre-load existing extra values for model_names from tenant_model
- # to reuse non-default extra for same model_name across providers
- cursor = self.db.execute_sql("SELECT model_name, extra FROM tenant_model WHERE extra != '{}' AND extra IS NOT NULL")
- existing_extra_map = {}
- for model_name, extra in cursor.fetchall():
- if model_name and extra and extra != "{}":
- existing_extra_map[model_name] = extra
-
- # Pre-load existing status values for model_names from tenant_model
- # Prefer non-unsupported status when seeding new records for the same model_name
- cursor = self.db.execute_sql("SELECT model_name, status FROM tenant_model WHERE status != 'unsupported' AND status IS NOT NULL")
- existing_status_map = {}
- for model_name, status in cursor.fetchall():
- if model_name and status:
- existing_status_map[model_name] = status
-
- factories = self._load_llm_factories()
- all_records = []
-
- for factory in factories:
- llm_list = factory.get("llm", [])
- if not llm_list:
- continue
- factory_name = factory["name"]
-
- # Determine the provider_name to query and optional instance extra filter
- if factory_name == "siliconflow_intl":
- provider_name_filter = "SILICONFLOW"
- instance_extra_include = '%"region": "intl"%'
- instance_extra_exclude = None
- elif factory_name == "SILICONFLOW":
- provider_name_filter = "SILICONFLOW"
- instance_extra_include = None
- instance_extra_exclude = '%"region": "intl"%'
- else:
- provider_name_filter = factory_name
- instance_extra_include = None
- instance_extra_exclude = None
-
- # Query provider for this factory
- cursor = self.db.execute_sql("SELECT id FROM tenant_model_provider WHERE provider_name = %s", (provider_name_filter,))
- providers = cursor.fetchall()
-
- for (provider_id,) in providers:
- # Query instances for this provider, with optional extra include/exclude filter
- if instance_extra_include and instance_extra_exclude:
- cursor = self.db.execute_sql(
- "SELECT id FROM tenant_model_instance WHERE provider_id = %s AND extra LIKE %s AND extra NOT LIKE %s", (provider_id, instance_extra_include, instance_extra_exclude)
- )
- elif instance_extra_include:
- cursor = self.db.execute_sql("SELECT id FROM tenant_model_instance WHERE provider_id = %s AND extra LIKE %s", (provider_id, instance_extra_include))
- elif instance_extra_exclude:
- cursor = self.db.execute_sql("SELECT id FROM tenant_model_instance WHERE provider_id = %s AND extra NOT LIKE %s", (provider_id, instance_extra_exclude))
- else:
- cursor = self.db.execute_sql("SELECT id FROM tenant_model_instance WHERE provider_id = %s", (provider_id,))
- instances = cursor.fetchall()
- if not instances:
- logger.warning(f"No instances found for provider '{provider_name_filter}' (id={provider_id}), skipping")
- continue
-
- for (instance_id,) in instances:
- for llm in llm_list:
- model_name = llm.get("llm_name", "")
- if not model_name:
- continue
- model_types = llm.get("model_type", "chat")
- if isinstance(model_types, str):
- model_types = [model_types]
-
- # Determine extra: reuse existing non-default extra for same model_name;
- # otherwise build default from is_tools and max_tokens in the llm entry
- if model_name in existing_extra_map:
- extra = existing_extra_map[model_name]
- else:
- extra_dict = {}
- if llm.get("is_tools") is True:
- extra_dict["is_tools"] = True
- max_tokens = llm.get("max_tokens")
- if max_tokens is not None:
- extra_dict["max_tokens"] = max_tokens
- extra = json.dumps(extra_dict) if extra_dict else "{}"
-
- # Determine status: reuse existing non-unsupported status for same model_name, else "active"
- status = existing_status_map.get(model_name, "active")
-
- for mt in model_types:
- # Check if record already exists
- cursor = self.db.execute_sql(
- "SELECT COUNT(*) FROM tenant_model WHERE provider_id = %s AND instance_id = %s AND model_name = %s AND model_type = %s", (provider_id, instance_id, model_name, mt)
- )
- if cursor.fetchone()[0] > 0:
- continue
- all_records.append((model_name, provider_id, instance_id, mt, status, extra))
-
- if not all_records:
- logger.info("No missing models to seed")
- return 0, []
-
- logger.info(f"Seeding {len(all_records)} tenant_model records...")
-
- if self.dry_run:
- logger.info(f"[DRY RUN] Would insert {len(all_records)} records")
- for model_name, provider_id, instance_id, mt, status, extra in all_records[:5]:
- logger.info(f" model_name={model_name}, provider_id={provider_id}, instance_id={instance_id}, model_type={mt}, status={status}, extra={extra}")
- if len(all_records) > 5:
- logger.info(f" ... and {len(all_records) - 5} more records")
- return len(all_records), self.target_tables
-
- # Insert records in batches
- batch_size = 100
- for i in range(0, len(all_records), batch_size):
- batch = all_records[i : i + batch_size]
- values = []
- for model_name, provider_id, instance_id, mt, status, extra in batch:
- record_id = self.generate_uuid()
- model_name_escaped = model_name.replace("'", "''") if model_name else ""
- mt_escaped = mt.replace("'", "''") if mt else ""
- extra_escaped = extra.replace("'", "''") if extra else "{}"
- status_escaped = status.replace("'", "''") if status else "active"
- values.append(
- f"('{record_id}', '{model_name_escaped}', '{provider_id}', "
- f"'{instance_id}', '{mt_escaped}', '{status_escaped}', "
- f"'{extra_escaped}', "
- f"{current_ts * 1000}, FROM_UNIXTIME({current_ts}), "
- f"{current_ts * 1000}, FROM_UNIXTIME({current_ts}))"
- )
-
- insert_sql = f"""
- INSERT INTO tenant_model
- (id, model_name, provider_id, instance_id, model_type, status, extra,
- create_time, create_date, update_time, update_date)
- VALUES {", ".join(values)}
- """
- self.db.execute_sql(insert_sql)
- rows_inserted += len(batch)
- logger.info(f"Inserted batch {i // batch_size + 1}: {len(batch)} records")
-
- return rows_inserted, self.target_tables
-
- def create_target_table(self):
- """Create tenant_model table"""
- create_sql = """
- CREATE TABLE IF NOT EXISTS tenant_model (
- id VARCHAR(32) NOT NULL PRIMARY KEY,
- model_name VARCHAR(128),
- provider_id VARCHAR(32) NOT NULL,
- instance_id VARCHAR(32) NOT NULL,
- model_type VARCHAR(32) NOT NULL,
- status VARCHAR(32) DEFAULT 'active',
- extra VARCHAR(1024) DEFAULT '{}',
- create_time BIGINT,
- create_date DATETIME,
- update_time BIGINT,
- update_date DATETIME,
- INDEX idx_instance_id (instance_id)
- ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
- """
- self.db.execute_sql(create_sql)
- logger.info("Created tenant_model table")
-
-
-class ModelTypeMergeStage(MigrationStage):
- """Merge tenant_model rows by (provider_id, instance_id, model_name), converting model_type to binary integer"""
-
- name = "model_type_merge"
- description = "Merge tenant_model rows by provider_id/instance_id/model_name, converting model_type to binary integer"
- source_tables = ["tenant_model"]
- target_tables = ["tenant_model_merge_tmp"]
-
- # Mapping from LLMType string values to ModelTypeBinary integer values
- MODEL_TYPE_STR_TO_INT = {
- "chat": 1, # 0b0000001
- "embedding": 2, # 0b0000010
- "asr": 4, # 0b0000100
- "speech2text": 4, # 0b0000100 legacy alias
- "vision": 8, # 0b0001000
- "image2text": 8, # 0b0001000 legacy alias
- "rerank": 16, # 0b0010000
- "tts": 32, # 0b0100000
- "ocr": 64, # 0b1000000
- }
-
- def current_timestamp(self) -> int:
- return int(time.time())
-
- def generate_uuid(self) -> str:
- """Generate 32-character UUID1"""
- return uuid.uuid1().hex
-
- def check(self) -> bool:
- """Check if migration is needed"""
- if not self.db.table_exists("tenant_model"):
- logger.warning("Source table 'tenant_model' does not exist")
- return False
-
- # If model_type is already INT, the merge has already been executed — no work needed
- model_type_dtype = self.db.get_column_type("tenant_model", "model_type")
- if model_type_dtype and model_type_dtype.lower() == "int":
- self.mark_noop_completes_migration()
- logger.info("tenant_model.model_type is already INT, merge stage is not applicable (already done)")
- return False
-
- return True
-
- def execute(self) -> tuple[int, list]:
- """Execute migration"""
- current_ts = self.current_timestamp()
- rows_inserted = 0
-
- if not self.db.table_exists("tenant_model"):
- logger.error("Source table 'tenant_model' does not exist")
- return 0, []
-
- # Create temporary table with model_type as INTEGER
- if not self.dry_run:
- self.create_target_table()
-
- if self.create_table_only:
- logger.info("[CREATE TABLE ONLY] Temporary table created, skipping data merge")
- return 0, self.target_tables
-
- # Load all records from tenant_model
- cursor = self.db.execute_sql("SELECT id, model_name, provider_id, instance_id, model_type, status, extra, create_time, create_date, update_time, update_date FROM tenant_model ORDER BY id")
- all_rows = cursor.fetchall()
-
- # Group by (provider_id, instance_id, model_name)
- from collections import defaultdict
-
- groups = defaultdict(list)
- for row in all_rows:
- id_, model_name, provider_id, instance_id, model_type_str, status, extra, create_time, create_date, update_time, update_date = row
- groups[(provider_id, instance_id, model_name)].append(row)
-
- # Merge each group into a single record
- merged_records = []
- for (provider_id, instance_id, model_name), rows in groups.items():
- # Compute binary model_type:
- # OR all model_types where status != 'unsupported'
- # Then AND with complement of OR of all model_types where status == 'unsupported'
- supported_bits = 0
- unsupported_bits = 0
- first_non_unsupported_status = None
- first_row = rows[0]
-
- for row in rows:
- _, _, _, _, model_type_str, status, _, _, _, _, _ = row
- type_bit = self.MODEL_TYPE_STR_TO_INT.get(model_type_str, 0)
- if status == "unsupported":
- unsupported_bits |= type_bit
- else:
- supported_bits |= type_bit
- if first_non_unsupported_status is None:
- first_non_unsupported_status = status
-
- merged_type = supported_bits & ~unsupported_bits
- merged_status = first_non_unsupported_status or "active"
-
- # Use first row's values for all other fields, but replace id, model_type, status
- id_, model_name, provider_id, instance_id, _, _, extra, create_time, create_date, update_time, update_date = first_row
-
- # Only include records whose merged status is "active"
- if merged_status != "active":
- continue
-
- merged_records.append((self.generate_uuid(), model_name, provider_id, instance_id, merged_type, merged_status, extra, create_time, create_date, update_time, update_date))
-
- if not merged_records:
- logger.info("No records to merge")
- return 0, []
-
- logger.info(f"Merging {len(all_rows)} rows into {len(merged_records)} rows...")
-
- if self.dry_run:
- logger.info(f"[DRY RUN] Would insert {len(merged_records)} merged rows, then rename tables")
- for rec in merged_records[:5]:
- _, model_name, provider_id, instance_id, merged_type, status, extra, *_ = rec
- logger.info(f" model_name={model_name}, provider_id={provider_id}, instance_id={instance_id}, model_type={merged_type}, status={status}, extra={extra}")
- if len(merged_records) > 5:
- logger.info(f" ... and {len(merged_records) - 5} more rows")
- return len(merged_records), self.target_tables
-
- # Insert merged records into temporary table
- batch_size = 100
- for i in range(0, len(merged_records), batch_size):
- batch = merged_records[i : i + batch_size]
- values = []
- for rec in batch:
- id_, model_name, provider_id, instance_id, merged_type, status, extra, create_time, create_date, update_time, update_date = rec
- model_name_escaped = (model_name.replace("'", "''") if model_name else "") or None
- extra_escaped = extra.replace("'", "''") if extra else "{}"
- status_escaped = status.replace("'", "''") if status else "active"
- # Handle NULL date fields
- create_time_val = create_time if create_time is not None else current_ts * 1000
- update_time_val = update_time if update_time is not None else current_ts * 1000
- create_date_sql = f"FROM_UNIXTIME({int(create_time_val / 1000)})" if create_time_val else "NULL"
- update_date_sql = f"FROM_UNIXTIME({int(update_time_val / 1000)})" if update_time_val else "NULL"
- values.append(
- f"('{id_}', '{model_name_escaped}', '{provider_id}', "
- f"'{instance_id}', {merged_type}, '{status_escaped}', "
- f"'{extra_escaped}', "
- f"{create_time_val}, {create_date_sql}, "
- f"{update_time_val}, {update_date_sql})"
- )
-
- insert_sql = f"""
- INSERT INTO tenant_model_merge_tmp
- (id, model_name, provider_id, instance_id, model_type, status, extra,
- create_time, create_date, update_time, update_date)
- VALUES {", ".join(values)}
- """
- self.db.execute_sql(insert_sql)
- rows_inserted += len(batch)
- logger.info(f"Inserted batch {i // batch_size + 1}: {len(batch)} merged rows")
-
- # Rename original table to backup, then rename temp table to tenant_model
- logger.info("Renaming tables: tenant_model -> tenant_model_backup, tenant_model_merge_tmp -> tenant_model")
- self.db.execute_sql("DROP TABLE IF EXISTS tenant_model_backup")
- self.db.execute_sql("ALTER TABLE tenant_model RENAME TO tenant_model_backup")
- self.db.execute_sql("ALTER TABLE tenant_model_merge_tmp RENAME TO tenant_model")
- logger.info("Table rename completed successfully")
-
- return rows_inserted, ["tenant_model_merge_tmp", "tenant_model", "tenant_model_backup"]
-
- def create_target_table(self):
- """Create temporary table tenant_model_merge_tmp with model_type as INTEGER"""
- create_sql = """
- CREATE TABLE IF NOT EXISTS tenant_model_merge_tmp (
- id VARCHAR(32) NOT NULL PRIMARY KEY,
- model_name VARCHAR(128),
- provider_id VARCHAR(32) NOT NULL,
- instance_id VARCHAR(32) NOT NULL,
- model_type INT NOT NULL,
- status VARCHAR(32) DEFAULT 'active',
- extra VARCHAR(1024) DEFAULT '{}',
- create_time BIGINT,
- create_date DATETIME,
- update_time BIGINT,
- update_date DATETIME,
- INDEX idx_instance_id (instance_id)
- ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
- """
- self.db.execute_sql(create_sql)
- logger.info("Created tenant_model_merge_tmp table")
-
-
-class TenantModelIdMigrationStage(MigrationStage):
- """Populate tenant_*_id columns in tenant, knowledgebase, dialog, memory with tenant_model.id values."""
-
- name = "tenant_model_id_migration"
- description = "Populate tenant_*_id columns with tenant_model.id, converting from IntegerField to CharField (VARCHAR(32))"
- source_tables = ["tenant_model", "tenant_model_provider", "tenant_model_instance", "tenant", "knowledgebase", "dialog", "memory"]
- target_tables = ["tenant", "knowledgebase", "dialog", "memory"]
-
- # Maps table -> list of (tenant_*_id column, legacy *_id column, model_type for lookup)
- # The legacy column holds the model_name (e.g. "gpt-4o@default@OpenAI"), we use it to find tenant_model.id
- TENANT_ID_FIELDS = {
- "tenant": [
- ("tenant_llm_id", "llm_id", "chat"),
- ("tenant_embd_id", "embd_id", "embedding"),
- ("tenant_asr_id", "asr_id", "speech2text"),
- ("tenant_img2txt_id", "img2txt_id", "image2text"),
- ("tenant_rerank_id", "rerank_id", "rerank"),
- ("tenant_tts_id", "tts_id", "tts"),
- ("tenant_ocr_id", "ocr_id", "ocr"),
- ],
- "knowledgebase": [
- ("tenant_embd_id", "embd_id", "embedding"),
- ],
- "dialog": [
- ("tenant_llm_id", "llm_id", "chat"),
- ("tenant_rerank_id", "rerank_id", "rerank"),
- ],
- "memory": [
- ("tenant_embd_id", "embd_id", "embedding"),
- ("tenant_llm_id", "llm_id", "chat"),
- ],
- }
-
- # model_type string -> binary integer mapping
- MODEL_TYPE_TO_INT = {
- "chat": 1,
- "embedding": 2,
- "asr": 4,
- "speech2text": 4,
- "vision": 8,
- "image2text": 8,
- "rerank": 16,
- "tts": 32,
- "ocr": 64,
- }
-
- scan_batch_size = 500
-
- def check(self) -> bool:
- """Check if migration is needed - any tenant_*_id column is still INT or empty."""
- if not self.db.table_exists("tenant_model"):
- logger.warning("Dependency table 'tenant_model' does not exist")
- return False
-
- if not self.db.table_exists("tenant_model_provider"):
- logger.warning("Dependency table 'tenant_model_provider' does not exist")
- return False
-
- if not self.db.table_exists("tenant_model_instance"):
- logger.warning("Dependency table 'tenant_model_instance' does not exist")
- return False
-
- has_work = False
- for table_name, fields in self.TENANT_ID_FIELDS.items():
- if not self.db.table_exists(table_name):
- continue
- for tenant_id_col, _, _ in fields:
- if not self.db.column_exists(table_name, tenant_id_col):
- # Column does not exist yet — needs to be added
- has_work = True
- break
- # Check if column is still INT type -> needs conversion
- col_type = self.db.get_column_type(table_name, tenant_id_col)
- if col_type and col_type.lower() in ("int", "bigint", "integer"):
- has_work = True
- break
- # Check if there are NULL values that should be populated
- cursor = self.db.execute_sql(f"SELECT COUNT(*) FROM `{table_name}` WHERE `{tenant_id_col}` IS NULL OR `{tenant_id_col}` = '' OR LENGTH(`{tenant_id_col}`) <> 32")
- null_count = cursor.fetchone()[0]
- if null_count > 0:
- has_work = True
- break
- if has_work:
- break
-
- if not has_work:
- self.mark_noop_completes_migration()
- logger.info("All tenant_*_id columns are already VARCHAR(32) and populated, no migration needed")
- return False
-
- return True
-
- def execute(self) -> tuple[int, list]:
- """Execute migration: alter column types and populate tenant_*_id values."""
- rows_updated = 0
- tables_operated = set()
-
- if self.create_table_only:
- logger.info("[CREATE TABLE ONLY] No tables are created for this data migration")
- return 0, []
-
- # Step 1: Add missing columns as VARCHAR(32) or alter existing INT columns to VARCHAR(32)
- for table_name, fields in self.TENANT_ID_FIELDS.items():
- if not self.db.table_exists(table_name):
- continue
- for tenant_id_col, _, _ in fields:
- if not self.db.column_exists(table_name, tenant_id_col):
- # Column does not exist yet — add it as VARCHAR(32)
- logger.info(f"Adding column {table_name}.{tenant_id_col} as VARCHAR(32) NULL")
- if not self.dry_run:
- self.db.execute_sql(f"ALTER TABLE `{table_name}` ADD COLUMN `{tenant_id_col}` VARCHAR(32) NULL")
- tables_operated.add(table_name)
- continue
- col_type = self.db.get_column_type(table_name, tenant_id_col)
- if col_type and col_type.lower() in ("int", "bigint", "integer"):
- logger.info(f"Converting {table_name}.{tenant_id_col} from {col_type} to VARCHAR(32)")
- if not self.dry_run:
- self.db.execute_sql(f"ALTER TABLE `{table_name}` MODIFY COLUMN `{tenant_id_col}` VARCHAR(32) NULL")
- tables_operated.add(table_name)
-
- # Step 2: Build lookup cache from tenant_model joined with provider + instance
- # to resolve model_name@instance@provider -> tenant_model.id
- model_lookup = self._build_model_lookup()
- logger.info(f"Built model lookup with {len(model_lookup)} entries")
-
- # Step 3: Populate tenant_*_id from model_name
- for table_name, fields in self.TENANT_ID_FIELDS.items():
- if not self.db.table_exists(table_name):
- continue
- for tenant_id_col, legacy_col, model_type_str in fields:
- if not self.db.column_exists(table_name, tenant_id_col):
- logger.info(f"Column {table_name}.{tenant_id_col} does not exist, skipping")
- continue
- if not self.db.column_exists(table_name, legacy_col):
- logger.info(f"Column {table_name}.{legacy_col} does not exist, skipping")
- continue
-
- # Also need tenant_id for model lookup
- # For tenant table, the PK is the tenant_id
- # For knowledgebase, dialog, memory we also need tenant_id
- if table_name == "tenant":
- cursor = self.db.execute_sql(
- f"SELECT id, `{legacy_col}` FROM `{table_name}` WHERE (`{tenant_id_col}` IS NULL OR `{tenant_id_col}` = '' OR LENGTH(`{tenant_id_col}`) <> 32) AND `{legacy_col}` IS NOT NULL AND `{legacy_col}` != ''"
- )
- while True:
- rows = cursor.fetchmany(self.scan_batch_size)
- if not rows:
- break
- for row_id, model_name in rows:
- tenant_id = row_id # For tenant table, id == tenant_id
- resolved_id = self._resolve_model_id(model_lookup, tenant_id, model_name, model_type_str)
- if resolved_id:
- if not self.dry_run:
- self.db.execute_sql(
- f"UPDATE `{table_name}` SET `{tenant_id_col}` = %s WHERE id = %s",
- (resolved_id, row_id),
- )
- else:
- if not self.dry_run:
- self.db.execute_sql(
- f"UPDATE `{table_name}` SET `{tenant_id_col}` = '' WHERE id = %s",
- (row_id,),
- )
- rows_updated += 1
- tables_operated.add(table_name)
- else:
- cursor = self.db.execute_sql(
- f"SELECT id, tenant_id, `{legacy_col}` FROM `{table_name}` WHERE (`{tenant_id_col}` IS NULL OR `{tenant_id_col}` = '' OR LENGTH(`{tenant_id_col}`) <> 32) AND `{legacy_col}` IS NOT NULL AND `{legacy_col}` != ''"
- )
- while True:
- rows = cursor.fetchmany(self.scan_batch_size)
- if not rows:
- break
- for row_id, tenant_id, model_name in rows:
- resolved_id = self._resolve_model_id(model_lookup, tenant_id, model_name, model_type_str)
- if resolved_id:
- if not self.dry_run:
- self.db.execute_sql(
- f"UPDATE `{table_name}` SET `{tenant_id_col}` = %s WHERE id = %s",
- (resolved_id, row_id),
- )
- else:
- if not self.dry_run:
- self.db.execute_sql(
- f"UPDATE `{table_name}` SET `{tenant_id_col}` = '' WHERE id = %s",
- (row_id,),
- )
- rows_updated += 1
- tables_operated.add(table_name)
-
- if self.dry_run:
- logger.info(f"[DRY RUN] Would update {rows_updated} rows")
- else:
- logger.info(f"Updated {rows_updated} rows with tenant_model.id values")
-
- return rows_updated, sorted(tables_operated)
-
- def _split_model_name(self, model_name: str) -> tuple[str, str, str]:
- """Parse model_name: {model_name}@{factory_name} or {model_name}@{instance_name}@{factory_name}"""
- if not model_name:
- return "", "", ""
- parts = model_name.split("@")
- if len(parts) == 1:
- return parts[0], "default", ""
- elif len(parts) == 2:
- return parts[0], "default", parts[1]
- else:
- return parts[0], parts[1], parts[2]
-
- def _build_model_lookup(self) -> dict:
- """Build a lookup dict: (tenant_id, model_name_pure, provider_name, model_type_int) -> tenant_model.id"""
- cursor = self.db.execute_sql(
- "SELECT tm.id, tm.model_name, tm.model_type, tmp.tenant_id, tmp.provider_name "
- "FROM tenant_model tm "
- "INNER JOIN tenant_model_provider tmp ON tm.provider_id = tmp.id "
- "WHERE tm.status = 'active'"
- )
- lookup = {}
- for model_id, model_name, model_type, tenant_id, provider_name in cursor.fetchall():
- # model_type is a binary integer; we check each bit
- for type_str, type_bit in self.MODEL_TYPE_TO_INT.items():
- if model_type & type_bit:
- key = (tenant_id, model_name, provider_name, type_str)
- lookup[key] = model_id
- return lookup
-
- def _resolve_model_id(self, model_lookup: dict, tenant_id: str, model_name: str, model_type_str: str) -> str | None:
- """Look up tenant_model.id from the model_lookup cache."""
- pure_name, instance_name, provider_name = self._split_model_name(model_name)
- if not pure_name or not provider_name:
- logger.warning(f"Cannot resolve model id for {model_name}: missing model_name or provider")
- return None
-
- # Try exact lookup with the provider name from the model_name
- key = (tenant_id, pure_name, provider_name, model_type_str)
- result = model_lookup.get(key)
- if result:
- return result
-
- # Try with instance_name as part of provider (shouldn't normally happen)
- # Try without specific provider (any provider matching model_name + type for this tenant)
- for (t_id, m_name, p_name, m_type), m_id in model_lookup.items():
- if t_id == tenant_id and m_name == pure_name and m_type == model_type_str:
- return m_id
-
- logger.warning(f"No tenant_model.id found for tenant={tenant_id}, model={model_name}, type={model_type_str}")
- return None
-
-
-# Registry of available migration stages
-MIGRATION_STAGES = {
- "tenant_model_provider": TenantModelProviderStage,
- "tenant_model_instance": TenantModelInstanceStage,
- "tenant_model": TenantModelStage,
- "model_id_config": ModelIdConfigStage,
- "tenant_model_seeding": TenantModelSeedingStage,
- "model_type_merge": ModelTypeMergeStage,
- "tenant_model_id_migration": TenantModelIdMigrationStage,
-}
-
-
-def list_available_stages():
- """List all available migration stages"""
- logger.info("Available migration stages:")
- for name, stage_cls in MIGRATION_STAGES.items():
- logger.info(f" - {name}: {stage_cls.description}")
- logger.info(f" Source tables: {stage_cls.source_tables}")
- logger.info(f" Target tables: {stage_cls.target_tables}")
-
-
-def run_migration(
- config: MigrationConfig,
- stages: list,
- dry_run: bool = True,
- create_table_only: bool = False,
- database_version: str | None = None,
- mark_database_version_on_success: bool = False,
-):
- """Run migration with specified stages"""
- stats = MigrationStats()
- stats.start()
-
- db = MigrationDatabase(config)
-
- try:
- db.connect()
-
- if database_version:
- current_db_version = db.get_database_version()
- if should_skip_migration(current_db_version, database_version):
- logger.info(
- "Database migration version is %s, target version is %s, skipping all stages",
- current_db_version,
- database_version,
- )
- return
-
- if current_db_version:
- logger.info(
- "Current database migration version is %s, target version is %s",
- current_db_version,
- database_version,
- )
- else:
- logger.info(
- "Database migration version marker is not set, target version is %s",
- database_version,
- )
-
- total_stages = len(stages)
- all_stages_completed = True
-
- for idx, stage_name in enumerate(stages, 1):
- logger.info(f"{'=' * 60}")
- logger.info(f"Stage [{idx}/{total_stages}]: {stage_name}")
- logger.info(f"{'=' * 60}")
-
- if stage_name not in MIGRATION_STAGES:
- logger.error(f"Unknown stage: {stage_name}")
- stats.add_stage_stats(stage_name, [], 0, 0)
- all_stages_completed = False
- continue
-
- stage_cls = MIGRATION_STAGES[stage_name]
- stage = stage_cls(db, dry_run=dry_run, create_table_only=create_table_only)
-
- stage_start = time.time()
-
- # For create_table_only mode, skip check and directly execute
- if create_table_only:
- logger.info("[CREATE TABLE ONLY] Skipping check, will create/verify target table")
- rows, tables = stage.execute()
- else:
- # Check if migration is needed
- if not stage.check():
- logger.info(f"Stage '{stage_name}' check: no migration needed")
- stats.add_stage_stats(stage_name, [], 0, time.time() - stage_start)
- continue
-
- # Execute migration
- rows, tables = stage.execute()
-
- stage_duration = time.time() - stage_start
-
- stats.add_stage_stats(stage_name, tables, rows, stage_duration)
- logger.info(f"Stage '{stage_name}' completed: {rows} rows in {stage_duration:.2f}s")
-
- if mark_database_version_on_success and not dry_run and not create_table_only and database_version and all_stages_completed:
- db.set_database_version(database_version)
- logger.info("Marked database migration version as %s", database_version)
-
- finally:
- db.close()
- stats.end()
- stats.print_summary()
-
-
-def check_database_version(config: MigrationConfig, target_version: str) -> int:
- db = MigrationDatabase(config)
- try:
- db.connect()
- current_db_version = db.get_database_version()
- if should_skip_migration(current_db_version, target_version):
- logger.info(
- "Database migration version is %s, target version is %s, migration is not needed",
- current_db_version,
- target_version,
- )
- return 0
-
- if current_db_version:
- logger.info(
- "Database migration version is %s, target version is %s, migration is needed",
- current_db_version,
- target_version,
- )
- else:
- logger.info(
- "Database migration version marker is not set, target version is %s, migration is needed",
- target_version,
- )
- return 1
- finally:
- db.close()
-
-
-def mark_database_version(config: MigrationConfig, version: str) -> None:
- db = MigrationDatabase(config)
- try:
- db.connect()
- db.set_database_version(version)
- logger.info("Marked database migration version as %s", version)
- finally:
- db.close()
-
-
-def main():
- parser = argparse.ArgumentParser(
- description="MySQL Data Migration Tool",
- formatter_class=argparse.RawDescriptionHelpFormatter,
- epilog="""
-Examples:
- # List available stages
- python mysql_migration.py --list-stages
-
- # Check whether migration is needed for a target version
- python mysql_migration.py --check-database-version --database-version v0.27.2 --config /path/to/config.yaml
-
- # Mark database version separately
- python mysql_migration.py --mark-database-version --database-version v0.27.2 --config /path/to/config.yaml
-
- # Dry run (default - check only, no write) with config file
- python mysql_migration.py --stages tenant_model_provider --config /path/to/config.yaml
-
- # Dry run with command line MySQL connection
- python mysql_migration.py --stages tenant_model_provider --host localhost --port 3306 --user root --password secret
-
- # Create target tables only (no data migration)
- python mysql_migration.py --stages tenant_model_provider --config /path/to/config.yaml --create-table-only
-
- # Execute full migration (create tables and migrate data)
- python mysql_migration.py --stages tenant_model_provider --config /path/to/config.yaml --execute
-
- # Execute migration only when database version is lower than v0.27.2
- python mysql_migration.py --stages tenant_model_provider --config /path/to/config.yaml --execute --database-version v0.27.2
-
- # Execute migration and mark the database version when all stages succeed
- python mysql_migration.py --stages tenant_model_provider,tenant_model_instance,tenant_model,model_id_config --config /path/to/config.yaml --execute --database-version v0.27.2 --mark-database-version-on-success
-
- # Normalize legacy model IDs in stored configs
- python mysql_migration.py --stages model_id_config --config /path/to/config.yaml --execute
-
- # Run multiple stages
- python mysql_migration.py --stages stage1,stage2,stage3 --config /path/to/config.yaml --execute
-""",
- )
-
- # MySQL connection options
- parser.add_argument("--host", type=str, default="localhost", help="MySQL host (default: localhost)")
- parser.add_argument("--port", type=int, default=3306, help="MySQL port (default: 3306)")
- parser.add_argument("--user", type=str, default="root", help="MySQL user (default: root)")
- parser.add_argument("--password", type=str, default="", help="MySQL password (default: empty)")
- parser.add_argument("--database", type=str, default="rag_flow", help="MySQL database name (default: rag_flow)")
-
- # Configuration options
- parser.add_argument("--config", "-c", type=str, help="Path to YAML config file")
-
- # Migration options
- parser.add_argument("--stages", "-s", type=str, help="Comma-separated list of stages to run")
- parser.add_argument("--list-stages", "-l", action="store_true", help="List available stages")
- parser.add_argument("--check-database-version", action="store_true", help="Check whether migration is needed for the target database version")
- parser.add_argument("--mark-database-version", action="store_true", help="Write the database migration version marker and exit")
- parser.add_argument("--database-version", type=str, metavar="VERSION", help="Database migration version used by check/mark commands and as the migration threshold for --stages")
- parser.add_argument("--mark-database-version-on-success", action="store_true", help="When used with --stages and --execute, write --database-version after all stages succeed")
- parser.add_argument("--execute", "-e", action="store_true", default=False, help="Execute full migration: create tables and migrate data")
- parser.add_argument("--create-table-only", action="store_true", default=False, help="Only create target tables, skip data migration")
-
- args = parser.parse_args()
-
- # List stages and exit
- if args.list_stages:
- list_available_stages()
- return
-
- # Load configuration: command line args take precedence over config file
- if args.config:
- config = MigrationConfig.from_config_file(args.config)
- # Override with command line args if provided
- if args.host != "localhost":
- config.host = args.host
- if args.port != 3306:
- config.port = args.port
- if args.user != "root":
- config.user = args.user
- if args.password != "":
- config.password = args.password
- if args.database != "rag_flow":
- config.database = args.database
- else:
- # Use command line args directly
- config = MigrationConfig(host=args.host, port=args.port, user=args.user, password=args.password, database=args.database)
-
- logger.info(f"MySQL Configuration: host={config.host}, port={config.port}, user={config.user}, database={config.database}")
-
- if args.check_database_version and args.mark_database_version:
- logger.error("--check-database-version and --mark-database-version are mutually exclusive")
- sys.exit(1)
-
- if args.check_database_version:
- if not args.database_version:
- logger.error("--check-database-version requires --database-version")
- sys.exit(1)
- sys.exit(check_database_version(config, args.database_version))
-
- if args.mark_database_version:
- if not args.database_version:
- logger.error("--mark-database-version requires --database-version")
- sys.exit(1)
- mark_database_version(config, args.database_version)
- return
-
- if args.mark_database_version_on_success and not args.database_version:
- logger.error("--mark-database-version-on-success requires --database-version")
- sys.exit(1)
-
- # Three mutually exclusive modes: dry-run (default), create-table-only, execute
- if args.execute and args.create_table_only:
- logger.error("--execute and --create-table-only are mutually exclusive")
- sys.exit(1)
-
- if not args.stages:
- logger.error("No stages specified. Use --stages to specify stages or --list-stages to see available stages.")
- sys.exit(1)
-
- stages = [s.strip() for s in args.stages.split(",")]
- dry_run = True
- create_table_only = False
-
- if args.create_table_only:
- logger.info("Running in CREATE TABLE ONLY mode (create tables, no data migration)")
- dry_run = False
- create_table_only = True
- elif args.execute:
- logger.info("Running in EXECUTE mode (create tables and migrate data)")
- dry_run = False
- else:
- logger.info("Running in DRY-RUN mode (check only, no write). Use --create-table-only to create tables, or --execute for full migration.")
-
- run_migration(
- config=config,
- stages=stages,
- dry_run=dry_run,
- create_table_only=create_table_only,
- database_version=args.database_version,
- mark_database_version_on_success=args.mark_database_version_on_success,
- )
-
-
-if __name__ == "__main__":
- main()
diff --git a/tools/scripts/rename_models_providers.py b/tools/scripts/rename_models_providers.py
deleted file mode 100644
index 6e66f9123e..0000000000
--- a/tools/scripts/rename_models_providers.py
+++ /dev/null
@@ -1,59 +0,0 @@
-#!/usr/bin/env python3
-"""Rename provider name fields in conf/models/*.json to match llm_factories.json."""
-
-import json
-import os
-
-REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
-MODELS_DIR = os.path.join(REPO_ROOT, "conf", "models")
-
-# Valid mappings: models current name → llm_factories name
-CHANGES = {
- "avian": "Avian",
- "Baichuan": "BaiChuan",
- "Baidu": "BaiduYiyan",
- "CoHere": "Cohere",
- "FishAudio": "Fish Audio",
- "Gitee": "GiteeAI",
- "Google": "Gemini",
- "JieKouAI": "Jiekou.AI",
- "lmstudio": "LM-Studio",
- "localai": "LocalAI",
- "modelscope": "ModelScope",
- "Novita": "NovitaAI",
- "Nvidia": "NVIDIA",
- "ollama": "Ollama",
- "SiliconFlow": "SILICONFLOW",
- "HunYuan": "Tencent Hunyuan",
- "Aliyun": "Tongyi-Qianwen",
- "vllm": "VLLM",
- "Voyage": "Voyage AI",
- "xinference": "Xinference",
- "XunFei": "XunFei Spark",
-}
-
-
-def main():
- # Build current name → filename mapping
- file_map = {}
- for fn in os.listdir(MODELS_DIR):
- if not fn.endswith(".json"):
- continue
- with open(os.path.join(MODELS_DIR, fn)) as f:
- d = json.load(f)
- file_map[d["name"]] = fn
-
- for old_name, new_name in CHANGES.items():
- fn = file_map[old_name]
- path = os.path.join(MODELS_DIR, fn)
- with open(path) as f:
- d = json.load(f)
- d["name"] = new_name
- with open(path, "w") as f:
- json.dump(d, f, indent=2, ensure_ascii=False)
- f.write("\n")
- print(f'{fn}: "{old_name}" -> "{new_name}"')
-
-
-if __name__ == "__main__":
- main()
diff --git a/tools/scripts/run_migrations.sh b/tools/scripts/run_migrations.sh
deleted file mode 100755
index f2a91d9ade..0000000000
--- a/tools/scripts/run_migrations.sh
+++ /dev/null
@@ -1,61 +0,0 @@
-#!/bin/bash
-# -----------------------------------------------------------------------------
-# Shared migration script for model provider tables.
-#
-# Called by docker/entrypoint.sh and docker/launch_backend_service.sh.
-# Keeps migration stages and versions in one place to avoid divergence.
-#
-# Usage:
-# PY=python3 tools/scripts/run_migrations.sh [--config CONFIG_PATH]
-#
-# Environment variables:
-# PY - Python interpreter path (default: python3)
-# -----------------------------------------------------------------------------
-
-set -e
-
-PY="${PY:-python3}"
-CONFIG="conf/service_conf.yaml"
-
-while [ $# -gt 0 ]; do
- case "$1" in
- --config)
- # Require a real file. mysql_migration.py:90 turns any config-load
- # failure into a warning and falls back to host=localhost user=root
- # database=rag_flow, and this script also passes --execute, so an
- # empty, whitespace, flag-shaped or missing path would silently
- # migrate the default database instead of failing.
- if [ $# -lt 2 ] || [ -z "${2//[[:space:]]/}" ] || [ ! -f "$2" ] || [ "${2#-}" != "$2" ]; then
- echo "Error: --config requires the path of an existing file" >&2
- exit 1
- fi
- CONFIG="$2"
- shift 2
- ;;
- *)
- echo "Error: unknown argument: $1" >&2
- echo "Usage: PY=python3 tools/scripts/run_migrations.sh [--config CONFIG_PATH]" >&2
- exit 1
- ;;
- esac
-done
-
-echo "Running model provider table migrations..."
-
-# Step 1: Create base model provider tables
-"$PY" tools/scripts/mysql_migration.py \
- --stages tenant_model_provider,tenant_model_instance,tenant_model,model_id_config \
- --config "$CONFIG" \
- --execute \
- --database-version "v0.26.0" \
- --mark-database-version-on-success
-
-# Step 2: Seed, merge model types, and migrate model IDs
-"$PY" tools/scripts/mysql_migration.py \
- --stages tenant_model_seeding,model_type_merge,tenant_model_id_migration \
- --config "$CONFIG" \
- --execute \
- --database-version "v0.27.2" \
- --mark-database-version-on-success
-
-echo "Model provider table migrations completed."
diff --git a/uv.lock b/uv.lock
index fac3933707..f7e777d004 100644
--- a/uv.lock
+++ b/uv.lock
@@ -1924,7 +1924,7 @@ wheels = [
[[package]]
name = "ragflow"
-version = "0.27.2"
+version = "1.0.0-rc1"
source = { virtual = "." }
dependencies = [
{ name = "agentrun-sdk" },