Skip to content

LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows

4.6 KB
Raw Blame

V2 Migration Guide

This folder contains scripts and management commands for performing data migrations from v1 to v2 for the Unstract Multitenancy system. From version v0.93.0 onwards, there have been signifcant changes in how data is stored. This warrants a data migration and the below steps can come in handy.

[!WARNING] If you're updating from a very old version of Unstract and you face issues, its recommended to update to v0.92.1 with ./run-platform.sh -u -v v0.92.1 first and then perform this data migration.

Performing the migration

[!TIP] If the existing data is important to you, consider backing up and restoring to a new database beforehand.

Run the migrate-to-v2.sh script to perform schema and data migrations in one shot.

./migrate-to-v2.sh

If you prefer explicitly executing the migration commands, refer these steps below

Preparing for the migration (safest approach)

Follow below steps to backup existing data in case of any issues during the migration

Backup existing data

  • Make sure pg_dump is installed. This would usually be bundled with postgres and can be installed with
sudo apt-get install postgresql-client
  • Take a backup of the existing data with the below command (provide the appropriate options)
pg_dump -h localhost -p 5432 -U unstract_dev -d unstract_db -f unstract-backup-$(date +"%Y-%m-%d_%H-%M-%S").sql

Create a new database

  • Create a new database, for example unstract_db_v2

Restore backup to the new database

  • Exec into the DB if required (from /docker)
docker exec -it unstract-db /bin/bash
  • Execute the below command to restore the backup in the new DB
psql -h localhost -p 5432 -U unstract_dev -d unstract_db_v2 -f unstract-backup-<DateTime>.sql

Manually performing V2 migration

Configuring the env

  • Make sure to update all the necessary .env files
  • For example, in case of v2 ensure backend, prompt-service, and platform-service .env contain DB_SCHEMA=unstract_v2

Create a new schema

  • Create a new schema within the above DB, for example unstract_v2
  • This can be done directly in the DB or
  • Run the command supported from /backend/manage.py, for example in case of v2
python manage.py create_schema

Run schema migrations

  • Schema migrations are run by django, this gets invoked when running the backend service
  • Or run it explicitly with
python manage.py migrate
  • This ensures that the required tables are created or altered

Run data migrations

  • This helps move the existing data from your current schema to the new schema
  • It can either be run locally or within the backend container
docker exec -it unstract-backend /bin/bash
  • Make sure to activate the virtual environment
source .venv/bin/activate
  • Perform the data migration
python manage.py migrate_to_v2
  • Test to see if the applied migrations work as expected.

Django Commands Reference

The commands provided here help you manage the migration process, including creating and dropping schemas and applying the necessary migrations to transition from v1 to v2.

Pre-requisites: Ensure that migrating.v2 is added to the SHARED_APPS in your Django settings.

1. List Commands

To view all available management commands, use:

python manage.py help

2. Create V2 Schema

To set up the v2 schema in your database, execute:

python manage.py create_schema

This command initializes the schema required for v2 migrations.

3. Drop V2 Schema

To remove the v2 schema from your database, run:

python manage.py drop_schema

Note: Use this command with caution as it will permanently delete the v2 schema and its associated data.

4. Migrate to V2 Schema

To apply migrations and transition data from v1 to v2, use:

python manage.py migrate_to_v2

This command performs the migration, updating your database to the v2 schema and transforming the data accordingly.

Notes

  • Ensure you have backed up your data before attempting any migration.
  • Verify your database configuration and environment variables are correctly set.
  • Review the migration logs for any issues or errors during the migration process.

For more details on each command and its options, refer to the command help output using

python manage.py help <command_name>