LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Python ai-agentsdata-engineeringdocument-aigenerative-aiidpjson-extractionllmmcp-serverocrpdf-extractionprompt-engineeringstructured-output
Script to run v2 migration commands (#791)
Chandrasekharan Mf4f51a26dfe9
diff --git a/backend/account_v2/templates/login.html b/backend/account_v2/templates/login.htmlindex 4edf8e3f1..b9b2fda2e 100644--- a/backend/account_v2/templates/login.html+++ b/backend/account_v2/templates/login.html@@ -104,7 +104,7 @@</style></head><body>- <form method="post" action="{% url 'login' %}">+ <form method="post" action="{% url 'public:login' %}"><div class="spinner"><!-- Spinner animation --><div class="lds-dual-ring"></div>diff --git a/backend/migrating/v2/README.md b/backend/migrating/v2/README.mdindex 3c5867d35..81cd1e061 100644--- a/backend/migrating/v2/README.md+++ b/backend/migrating/v2/README.md@@ -1,17 +1,119 @@-# V2 Migrations and Commands+# V2 Migration GuideThis folder contains scripts and management commands for performing data migrations from v1 to v2 for the Unstract Multitenancy system.+From version `v0.93.0` onwards, there have been signifcant changes in how data is stored. This warrants a data migration and the below steps can come in handy.-## Overview+> [!WARNING]+> If you're updating from a very old version of Unstract and you face issues, its recommended to update to `v0.92.1` with+`./run-platform.sh -u -v v0.92.1` first and then perform this data migration.-The commands provided here help you manage the migration process, including creating and dropping schemas and applying the necessary migrations to transition from v1 to v2.+## Performing the migration++> [!TIP]+> If the existing data is important to you, consider [backing up and restoring to a new database](/backend/migrating/v2/README.md#preparing-for-the-migration-safest-approach) beforehand.++Run the `migrate-to-v2.sh` script to perform schema and data migrations in one shot.++```shell+./migrate-to-v2.sh+```++If you prefer explicitly executing the migration commands, refer [these steps below](/backend/migrating/v2/README.md#manually-performing-V2-migration)++## Preparing for the migration (safest approach)++Follow below steps to backup existing data in case of any issues during the migration++### Backup existing data++- Make sure `pg_dump` is installed. This would usually be bundled with postgres and can be installed with++```shell+sudo apt-get install postgresql-client+```++- Take a backup of the existing data with the below command (provide the appropriate options)++```shell+pg_dump -h localhost -p 5432 -U unstract_dev -d unstract_db -f unstract-backup-$(date +"%Y-%m-%d_%H-%M-%S").sql+```++### Create a new database++- Create a new database, for example `unstract_db_v2`++### Restore backup to the new database++- Exec into the DB if required (from [/docker](/docker/))++```shell+docker exec -it unstract-db /bin/bash+```++- Execute the below command to restore the backup in the new DB++```shell+psql -h localhost -p 5432 -U unstract_dev -d unstract_db_v2 -f unstract-backup-<DateTime>.sql+```++## Manually performing V2 migration++### Configuring the env++- Make sure to update all the necessary `.env` files+- For example, in case of `v2` ensure `backend`, `prompt-service`, and `platform-service` `.env` contain `DB_SCHEMA=unstract_v2`++### Create a new schema++- Create a new schema within the above DB, for example `unstract_v2`+- This can be done directly in the DB or+- Run the command supported from [/backend/manage.py](/backend/manage.py), for example in case of `v2`+```shell+python manage.py create_v2_schema+```++### Run schema migrations++- Schema migrations are run by django, this gets invoked when running the backend service+- Or run it explicitly with++```shell+python manage.py migrate+```++- This ensures that the required tables are created or altered++### Run data migrations++- This helps move the existing data from your current schema to the new schema+- It can either be run locally or within the backend container++```shell+docker exec -it unstract-backend /bin/bash+```++- Make sure to activate the virtual environment++```shell+source .venv/bin/activate+```++- Perform the data migration++```shell+python manage.py migrate_to_v2+```++- Test to see if the applied migrations work as expected.++## Django Commands Reference++The commands provided here help you manage the migration process, including creating and dropping schemas and applying the necessary migrations to transition from v1 to v2.**Pre-requisites:**Ensure that `migrating.v2` is added to the `SHARED_APPS` in your Django settings.-## Available Commands-### 1. List CommandsTo view all available management commands, use:@@ -27,6 +129,7 @@ To set up the v2 schema in your database, execute:```bashpython manage.py create_v2_schema```+This command initializes the schema required for v2 migrations.### 3. Drop V2 Schema@@ -36,9 +139,11 @@ To remove the v2 schema from your database, run:```bashpython manage.py drop_v2_schema```+Note: Use this command with caution as it will permanently delete the v2 schema and its associated data.### 4. Migrate to V2 Schema+To apply migrations and transition data from v1 to v2, use:```bash@@ -47,13 +152,14 @@ python manage.py migrate_to_v2This command performs the migration, updating your database to the v2 schema and transforming the data accordingly.-## Notes-- Ensure you have backups of your data before running migration commands.++- Ensure you have backed up your data before attempting any migration.- Verify your database configuration and environment variables are correctly set.- Review the migration logs for any issues or errors during the migration process.-For more details on each command and its options, refer to the command help output using+For more details on each command and its options, refer to the command help output using+```bashpython manage.py help <command_name>```diff --git a/backend/migrating/v2/migrate-to-v2.sh b/backend/migrating/v2/migrate-to-v2.shnew file mode 100755index 000000000..18e07a50e--- /dev/null+++ b/backend/migrating/v2/migrate-to-v2.sh@@ -0,0 +1,47 @@+#!/usr/bin/env bash++# Usage: ./migrate-to-v2.sh++set -o nounset # exit if a variable is not set+set -o errexit # exit for any command failure"++# text color escape codes (\033 == \e but OSX doesn't respect the \e)+blue_text='\033[94m'+green_text='\033[32m'+red_text='\033[31m'+default_text='\033[39m'+yellow_text='\033[33m'++# set -x/xtrace uses PS4 for more info+PS4="$blue_text""${0}:${LINENO}: ""$default_text"++# Function to check and create virtual environment+activate_venv() {+ if [[ ! -d ".venv" ]]; then+ echo "No .venv found, please create one and install dependencies. Refer 'run-platform.sh' or 'dev-env-cli.sh";+ exit 1;+ else+ echo "Virtual environment found. Activating..."+ source .venv/bin/activate || { echo "Failed to activate virtual environment"; exit 1; }+ fi+}++script_dir=$(dirname "$(readlink -f "$BASH_SOURCE")")+manage_script_dir=$(readlink -f "$script_dir/../../")+root_dir=$(readlink -f "$manage_script_dir/..")++# Configure environment variables for all services+echo -e "${blue_text}Configuring environment variables for the migration${default_text}"+"$root_dir/run-platform.sh" -e || { echo "Failed to configure environment"; exit 1; }++cd $manage_script_dir+activate_venv++echo -e "${blue_text}Running schema creation command...${default_text}"+python manage.py create_v2_schema || { echo "Schema creation failed"; exit 1; }++echo -e "${blue_text}Running schema migration command...${default_text}"+python manage.py migrate || { echo "Schema migration failed"; exit 1; }++echo -e "${blue_text}Running data migration command...${default_text}"+SCHEMAS_TO_MIGRATE=_ALL_ python manage.py migrate_to_v2 || { echo "Data migration failed"; exit 1; }diff --git a/run-platform.sh b/run-platform.shindex fea90dbc8..4b4a39880 100755--- a/run-platform.sh+++ b/run-platform.sh@@ -11,7 +11,8 @@ default_text='\033[39m'yellow_text='\033[33m'# set -x/xtrace uses PS4 for more info-PS4="$blue_text""${0}:${LINENO}: ""$default_text"+PS4="$blue_text"'${0}:${LINENO}: '"$default_text"+debug() {if [ "$opt_verbose" = true ]; then@@ -136,30 +137,33 @@ do_git_pull() {returnfi+ current_version=$(git describe --tags --abbrev=0)echo "Fetching release tags."git fetch --quiet --tagsif [[ "$opt_version" == "latest" ]]; then- branch=`git describe --tags --abbrev=0`+ target_branch=`git ls-remote --tags origin | awk -F/ '{print $3}' | sort -V | tail -n1`elif [[ "$opt_version" == "main" ]]; then- branch="main"+ target_branch="main"opt_build_local=trueecho -e "Choosing ""$blue_text""local build""$default_text"" of Docker images from ""$blue_text""main""$default_text"" branch."elif [ -z $(git tag -l "$opt_version") ]; thenecho -e "$red_text""Version not found.""$default_text"- if [[ ! $opt_version == v* ]]; then- echo -e "$red_text""Version must be provided with a 'v' prefix (e.g. v0.47.0).""$default_text"++ version_regex="^v([0-9]+)\.([0-9]+)\.([0-9]+)(-[a-zA-Z0-9]+(\.[0-9]+)?)?$"+ if [[ ! $opt_version =~ $version_regex ]]; then+ echo -e "$red_text""Version must be provided with a 'v' prefix and follow SemVer (e.g. v0.47.0).""$default_text"fiexit 1else- branch="$opt_version"+ target_branch="$opt_version"fi- echo -e "Performing ""$blue_text""git checkout""$default_text"" to ""$blue_text""$branch""$default_text""."- git checkout --quiet $branch+ echo -e "Performing ""$blue_text""git checkout""$default_text"" to ""$blue_text""$target_branch""$default_text""."+ git checkout --quiet $target_branch- echo -e "Performing ""$blue_text""git pull""$default_text"" on ""$blue_text""$branch""$default_text""."- git pull --quiet $(git remote) $branch+ echo -e "Performing ""$blue_text""git pull""$default_text"" on ""$blue_text""$target_branch""$default_text""."+ git pull --quiet $(git remote) $target_branch}copy_or_merge_envs() {@@ -277,6 +281,9 @@ run_services() {elseecho -e "$green_text""Updated platform to $opt_version version.""$default_text"fi++ # Show release notes on version update if applicable+ python3 "$script_dir/scripts/release-notes/print_release_notes.py" "$current_version" "$target_branch"fiecho -e "\nOnce the services are up, visit ""$blue_text""http://frontend.unstract.localhost""$default_text"" in your browser."echo -e "\nSee logs with:"@@ -314,6 +321,8 @@ script_dir=$(dirname "$(readlink -f "$BASH_SOURCE")")first_setup=false# Extract service names from docker compose file.services=($(VERSION=$opt_version $docker_compose_cmd -f $script_dir/docker/docker-compose.build.yaml config --services))+current_version=""+target_branch=""display_bannerparse_args $*diff --git a/scripts/release-notes/README.md b/scripts/release-notes/README.mdnew file mode 100644index 000000000..4b9311851--- /dev/null+++ b/scripts/release-notes/README.md@@ -0,0 +1,16 @@+# Print Release Notes++Accepts `current_version` and `target_version` as inputs to print the necessary release notes / warnings to be shown while updating a release++Run below to get more information on the usage++```shell+python print_release_notes.py -h+```++Make sure to create / activate a virtual environment with the below commands++```shell+pdm venv create -w virtualenv --with-pip+eval $(pdm venv activate in-project)+```diff --git a/scripts/release-notes/print_release_notes.py b/scripts/release-notes/print_release_notes.pynew file mode 100644index 000000000..4e15c735a--- /dev/null+++ b/scripts/release-notes/print_release_notes.py@@ -0,0 +1,120 @@+# type: ignore+import argparse+import json+import re+import sys+from pathlib import Path++from packaging import version++# Sample JSON data (in a separate file 'warnings.json')+# {+# "v0.93.0": {+# "message": "This a WARNING note",+# "warning": true+# },+# "v0.95.0": {+# "message": "This is an INFO note",+# "warning": false+# }+# }++RELEASE_NOTES_FILE_PATH = Path(__file__).with_name("release_notes.json").absolute()+++class Colour:+ BLUE = "\033[94m"+ GREEN = "\033[32m"+ RED = "\033[31m"+ DEFAULT = "\033[39m"+ YELLOW = "\033[33m"+++def colour(text, colour: Colour):+ return f"{colour}{text}{Colour.DEFAULT}"+++def load_release_notes(json_file):+ """Load warnings from a JSON file."""+ with open(json_file) as file:+ return json.load(file)+++def is_valid_semver(ver):+ """Check if a version string follows semantic versioning (vX.X.X)."""+ semver_pattern = r"^v\d+\.\d+\.\d+(-[a-zA-Z0-9]+(\.\d+)?)?$"+ return re.match(semver_pattern, ver) is not None+++def print_release_notes(current_version, target_version, release_notes):+ """Print relevant warnings if current_version < warning_version <=+ target_version."""+ messages_to_print = []+ for note_version, release_note in release_notes.items():+ if (+ version.parse(current_version)+ < version.parse(note_version)+ <= version.parse(target_version)+ ):+ # Check release note and display warning in yellow release_note+ message = (+ f"{colour('WARNING', Colour.YELLOW)}: "+ if release_note["warning"]+ else ""+ )+ message += (+ f"[from {colour(note_version, Colour.BLUE)}]\n"+ f"{release_note['message']}"+ )+ messages_to_print.append(message)++ if not messages_to_print:+ return++ print("\n########################## RELEASE NOTES ##########################")+ for message in messages_to_print:+ print("\n" + message + "\n")+ print("###################################################################")+++def main():+ """Main function to load warnings and check versions."""+ parser = argparse.ArgumentParser(+ description="Check for version upgrade warnings based on semantic versioning."+ )+ parser.add_argument(+ "current_version",+ type=str,+ help="Current version of Unstract (e.g., v0.85.0)",+ )+ parser.add_argument(+ "target_version",+ type=str,+ help="Target version for the upgrade (e.g., v0.90.0)",+ )++ args = parser.parse_args()++ current_version = args.current_version+ target_version = args.target_version++ # Load the warnings from the JSON file+ release_notes = load_release_notes(RELEASE_NOTES_FILE_PATH)++ # Validate the current_version and target_version+ if not is_valid_semver(current_version) or not is_valid_semver(target_version):+ print(+ f"\n{colour('WARNING', Colour.YELLOW)}: "+ "You are trying to update between versions\n"+ f"\t({colour(current_version, Colour.BLUE)} -> "+ f"{colour(target_version, Colour.BLUE)})\n"+ f"Please check release notes at "+ f"'{colour(RELEASE_NOTES_FILE_PATH, Colour.BLUE)}'\n"+ )+ sys.exit(1)++ print_release_notes(current_version, target_version, release_notes)+++if __name__ == "__main__":+ main()diff --git a/scripts/release-notes/release_notes.json b/scripts/release-notes/release_notes.jsonnew file mode 100644index 000000000..4c7aa1a7a--- /dev/null+++ b/scripts/release-notes/release_notes.json@@ -0,0 +1,6 @@+{+ "v0.93.0": {+ "message": "A data migration is required to continue using the platform with your existing data.\nFor detailed instructions on how to perform this migration, please check:\n `backend/migrating/v2/README.md`",+ "warning": true+ }+}