LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
V2 Migration Guide
This folder contains scripts and management commands for performing data migrations from v1 to v2 for the Unstract Multitenancy system.
From version v0.93.0 onwards, there have been signifcant changes in how data is stored. This warrants a data migration and the below steps can come in handy.
[!WARNING] If you're updating from a very old version of Unstract and you face issues, its recommended to update to
v0.92.1with./run-platform.sh -u -v v0.92.1first and then perform this data migration.
Performing the migration
[!TIP] If the existing data is important to you, consider backing up and restoring to a new database beforehand.
Run the migrate-to-v2.sh script to perform schema and data migrations in one shot.
./migrate-to-v2.sh
If you prefer explicitly executing the migration commands, refer these steps below
Preparing for the migration (safest approach)
Follow below steps to backup existing data in case of any issues during the migration
Backup existing data
- Make sure
pg_dumpis installed. This would usually be bundled with postgres and can be installed with
sudo apt-get install postgresql-client
- Take a backup of the existing data with the below command (provide the appropriate options)
pg_dump -h localhost -p 5432 -U unstract_dev -d unstract_db -f unstract-backup-$(date +"%Y-%m-%d_%H-%M-%S").sql
Create a new database
- Create a new database, for example
unstract_db_v2
Restore backup to the new database
- Exec into the DB if required (from /docker)
docker exec -it unstract-db /bin/bash
- Execute the below command to restore the backup in the new DB
psql -h localhost -p 5432 -U unstract_dev -d unstract_db_v2 -f unstract-backup-<DateTime>.sql
Manually performing V2 migration
Configuring the env
- Make sure to update all the necessary
.envfiles - For example, in case of
v2ensurebackend,prompt-service, andplatform-service.envcontainDB_SCHEMA=unstract_v2
Create a new schema
- Create a new schema within the above DB, for example
unstract_v2 - This can be done directly in the DB or
- Run the command supported from /backend/manage.py, for example in case of
v2
python manage.py create_schema
Run schema migrations
- Schema migrations are run by django, this gets invoked when running the backend service
- Or run it explicitly with
python manage.py migrate
- This ensures that the required tables are created or altered
Run data migrations
- This helps move the existing data from your current schema to the new schema
- It can either be run locally or within the backend container
docker exec -it unstract-backend /bin/bash
- Make sure to activate the virtual environment
source .venv/bin/activate
- Perform the data migration
python manage.py migrate_to_v2
- Test to see if the applied migrations work as expected.
Django Commands Reference
The commands provided here help you manage the migration process, including creating and dropping schemas and applying the necessary migrations to transition from v1 to v2.
Pre-requisites:
Ensure that migrating.v2 is added to the SHARED_APPS in your Django settings.
1. List Commands
To view all available management commands, use:
python manage.py help
2. Create V2 Schema
To set up the v2 schema in your database, execute:
python manage.py create_schema
This command initializes the schema required for v2 migrations.
3. Drop V2 Schema
To remove the v2 schema from your database, run:
python manage.py drop_schema
Note: Use this command with caution as it will permanently delete the v2 schema and its associated data.
4. Migrate to V2 Schema
To apply migrations and transition data from v1 to v2, use:
python manage.py migrate_to_v2
This command performs the migration, updating your database to the v2 schema and transforming the data accordingly.
Notes
- Ensure you have backed up your data before attempting any migration.
- Verify your database configuration and environment variables are correctly set.
- Review the migration logs for any issues or errors during the migration process.
For more details on each command and its options, refer to the command help output using
python manage.py help <command_name>