Skip to content

LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows

Script to run v2 migration commands (#791)

Chandrasekharan Mf4f51a26dfe9
diff --git a/backend/account_v2/templates/login.html b/backend/account_v2/templates/login.html
index 4edf8e3f1..b9b2fda2e 100644
--- a/backend/account_v2/templates/login.html
+++ b/backend/account_v2/templates/login.html
@@ -104,7 +104,7 @@
</style>
</head>
<body>
- <form method="post" action="{% url 'login' %}">
+ <form method="post" action="{% url 'public:login' %}">
<div class="spinner">
<!-- Spinner animation -->
<div class="lds-dual-ring"></div>
diff --git a/backend/migrating/v2/README.md b/backend/migrating/v2/README.md
index 3c5867d35..81cd1e061 100644
--- a/backend/migrating/v2/README.md
+++ b/backend/migrating/v2/README.md
@@ -1,17 +1,119 @@
-# V2 Migrations and Commands
+# V2 Migration Guide
This folder contains scripts and management commands for performing data migrations from v1 to v2 for the Unstract Multitenancy system.
+From version `v0.93.0` onwards, there have been signifcant changes in how data is stored. This warrants a data migration and the below steps can come in handy.
-## Overview
+> [!WARNING]
+> If you're updating from a very old version of Unstract and you face issues, its recommended to update to `v0.92.1` with
+`./run-platform.sh -u -v v0.92.1` first and then perform this data migration.
-The commands provided here help you manage the migration process, including creating and dropping schemas and applying the necessary migrations to transition from v1 to v2.
+## Performing the migration
+
+> [!TIP]
+> If the existing data is important to you, consider [backing up and restoring to a new database](/backend/migrating/v2/README.md#preparing-for-the-migration-safest-approach) beforehand.
+
+Run the `migrate-to-v2.sh` script to perform schema and data migrations in one shot.
+
+```shell
+./migrate-to-v2.sh
+```
+
+If you prefer explicitly executing the migration commands, refer [these steps below](/backend/migrating/v2/README.md#manually-performing-V2-migration)
+
+## Preparing for the migration (safest approach)
+
+Follow below steps to backup existing data in case of any issues during the migration
+
+### Backup existing data
+
+- Make sure `pg_dump` is installed. This would usually be bundled with postgres and can be installed with
+
+```shell
+sudo apt-get install postgresql-client
+```
+
+- Take a backup of the existing data with the below command (provide the appropriate options)
+
+```shell
+pg_dump -h localhost -p 5432 -U unstract_dev -d unstract_db -f unstract-backup-$(date +"%Y-%m-%d_%H-%M-%S").sql
+```
+
+### Create a new database
+
+- Create a new database, for example `unstract_db_v2`
+
+### Restore backup to the new database
+
+- Exec into the DB if required (from [/docker](/docker/))
+
+```shell
+docker exec -it unstract-db /bin/bash
+```
+
+- Execute the below command to restore the backup in the new DB
+
+```shell
+psql -h localhost -p 5432 -U unstract_dev -d unstract_db_v2 -f unstract-backup-<DateTime>.sql
+```
+
+## Manually performing V2 migration
+
+### Configuring the env
+
+- Make sure to update all the necessary `.env` files
+- For example, in case of `v2` ensure `backend`, `prompt-service`, and `platform-service` `.env` contain `DB_SCHEMA=unstract_v2`
+
+### Create a new schema
+
+- Create a new schema within the above DB, for example `unstract_v2`
+- This can be done directly in the DB or
+- Run the command supported from [/backend/manage.py](/backend/manage.py), for example in case of `v2`
+```shell
+python manage.py create_v2_schema
+```
+
+### Run schema migrations
+
+- Schema migrations are run by django, this gets invoked when running the backend service
+- Or run it explicitly with
+
+```shell
+python manage.py migrate
+```
+
+- This ensures that the required tables are created or altered
+
+### Run data migrations
+
+- This helps move the existing data from your current schema to the new schema
+- It can either be run locally or within the backend container
+
+```shell
+docker exec -it unstract-backend /bin/bash
+```
+
+- Make sure to activate the virtual environment
+
+```shell
+source .venv/bin/activate
+```
+
+- Perform the data migration
+
+```shell
+python manage.py migrate_to_v2
+```
+
+- Test to see if the applied migrations work as expected.
+
+## Django Commands Reference
+
+The commands provided here help you manage the migration process, including creating and dropping schemas and applying the necessary migrations to transition from v1 to v2.
**Pre-requisites:**
Ensure that `migrating.v2` is added to the `SHARED_APPS` in your Django settings.
-## Available Commands
-
### 1. List Commands
To view all available management commands, use:
@@ -27,6 +129,7 @@ To set up the v2 schema in your database, execute:
```bash
python manage.py create_v2_schema
```
+
This command initializes the schema required for v2 migrations.
### 3. Drop V2 Schema
@@ -36,9 +139,11 @@ To remove the v2 schema from your database, run:
```bash
python manage.py drop_v2_schema
```
+
Note: Use this command with caution as it will permanently delete the v2 schema and its associated data.
### 4. Migrate to V2 Schema
+
To apply migrations and transition data from v1 to v2, use:
```bash
@@ -47,13 +152,14 @@ python manage.py migrate_to_v2
This command performs the migration, updating your database to the v2 schema and transforming the data accordingly.
-
## Notes
-- Ensure you have backups of your data before running migration commands.
+
+- Ensure you have backed up your data before attempting any migration.
- Verify your database configuration and environment variables are correctly set.
- Review the migration logs for any issues or errors during the migration process.
-For more details on each command and its options, refer to the command help output using
+For more details on each command and its options, refer to the command help output using
+
```bash
python manage.py help <command_name>
```
diff --git a/backend/migrating/v2/migrate-to-v2.sh b/backend/migrating/v2/migrate-to-v2.sh
new file mode 100755
index 000000000..18e07a50e
--- /dev/null
+++ b/backend/migrating/v2/migrate-to-v2.sh
@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+
+# Usage: ./migrate-to-v2.sh
+
+set -o nounset # exit if a variable is not set
+set -o errexit # exit for any command failure"
+
+# text color escape codes (\033 == \e but OSX doesn't respect the \e)
+blue_text='\033[94m'
+green_text='\033[32m'
+red_text='\033[31m'
+default_text='\033[39m'
+yellow_text='\033[33m'
+
+# set -x/xtrace uses PS4 for more info
+PS4="$blue_text""${0}:${LINENO}: ""$default_text"
+
+# Function to check and create virtual environment
+activate_venv() {
+ if [[ ! -d ".venv" ]]; then
+ echo "No .venv found, please create one and install dependencies. Refer 'run-platform.sh' or 'dev-env-cli.sh";
+ exit 1;
+ else
+ echo "Virtual environment found. Activating..."
+ source .venv/bin/activate || { echo "Failed to activate virtual environment"; exit 1; }
+ fi
+}
+
+script_dir=$(dirname "$(readlink -f "$BASH_SOURCE")")
+manage_script_dir=$(readlink -f "$script_dir/../../")
+root_dir=$(readlink -f "$manage_script_dir/..")
+
+# Configure environment variables for all services
+echo -e "${blue_text}Configuring environment variables for the migration${default_text}"
+"$root_dir/run-platform.sh" -e || { echo "Failed to configure environment"; exit 1; }
+
+cd $manage_script_dir
+activate_venv
+
+echo -e "${blue_text}Running schema creation command...${default_text}"
+python manage.py create_v2_schema || { echo "Schema creation failed"; exit 1; }
+
+echo -e "${blue_text}Running schema migration command...${default_text}"
+python manage.py migrate || { echo "Schema migration failed"; exit 1; }
+
+echo -e "${blue_text}Running data migration command...${default_text}"
+SCHEMAS_TO_MIGRATE=_ALL_ python manage.py migrate_to_v2 || { echo "Data migration failed"; exit 1; }
diff --git a/run-platform.sh b/run-platform.sh
index fea90dbc8..4b4a39880 100755
--- a/run-platform.sh
+++ b/run-platform.sh
@@ -11,7 +11,8 @@ default_text='\033[39m'
yellow_text='\033[33m'
# set -x/xtrace uses PS4 for more info
-PS4="$blue_text""${0}:${LINENO}: ""$default_text"
+PS4="$blue_text"'${0}:${LINENO}: '"$default_text"
+
debug() {
if [ "$opt_verbose" = true ]; then
@@ -136,30 +137,33 @@ do_git_pull() {
return
fi
+ current_version=$(git describe --tags --abbrev=0)
echo "Fetching release tags."
git fetch --quiet --tags
if [[ "$opt_version" == "latest" ]]; then
- branch=`git describe --tags --abbrev=0`
+ target_branch=`git ls-remote --tags origin | awk -F/ '{print $3}' | sort -V | tail -n1`
elif [[ "$opt_version" == "main" ]]; then
- branch="main"
+ target_branch="main"
opt_build_local=true
echo -e "Choosing ""$blue_text""local build""$default_text"" of Docker images from ""$blue_text""main""$default_text"" branch."
elif [ -z $(git tag -l "$opt_version") ]; then
echo -e "$red_text""Version not found.""$default_text"
- if [[ ! $opt_version == v* ]]; then
- echo -e "$red_text""Version must be provided with a 'v' prefix (e.g. v0.47.0).""$default_text"
+
+ version_regex="^v([0-9]+)\.([0-9]+)\.([0-9]+)(-[a-zA-Z0-9]+(\.[0-9]+)?)?$"
+ if [[ ! $opt_version =~ $version_regex ]]; then
+ echo -e "$red_text""Version must be provided with a 'v' prefix and follow SemVer (e.g. v0.47.0).""$default_text"
fi
exit 1
else
- branch="$opt_version"
+ target_branch="$opt_version"
fi
- echo -e "Performing ""$blue_text""git checkout""$default_text"" to ""$blue_text""$branch""$default_text""."
- git checkout --quiet $branch
+ echo -e "Performing ""$blue_text""git checkout""$default_text"" to ""$blue_text""$target_branch""$default_text""."
+ git checkout --quiet $target_branch
- echo -e "Performing ""$blue_text""git pull""$default_text"" on ""$blue_text""$branch""$default_text""."
- git pull --quiet $(git remote) $branch
+ echo -e "Performing ""$blue_text""git pull""$default_text"" on ""$blue_text""$target_branch""$default_text""."
+ git pull --quiet $(git remote) $target_branch
}
copy_or_merge_envs() {
@@ -277,6 +281,9 @@ run_services() {
else
echo -e "$green_text""Updated platform to $opt_version version.""$default_text"
fi
+
+ # Show release notes on version update if applicable
+ python3 "$script_dir/scripts/release-notes/print_release_notes.py" "$current_version" "$target_branch"
fi
echo -e "\nOnce the services are up, visit ""$blue_text""http://frontend.unstract.localhost""$default_text"" in your browser."
echo -e "\nSee logs with:"
@@ -314,6 +321,8 @@ script_dir=$(dirname "$(readlink -f "$BASH_SOURCE")")
first_setup=false
# Extract service names from docker compose file.
services=($(VERSION=$opt_version $docker_compose_cmd -f $script_dir/docker/docker-compose.build.yaml config --services))
+current_version=""
+target_branch=""
display_banner
parse_args $*
diff --git a/scripts/release-notes/README.md b/scripts/release-notes/README.md
new file mode 100644
index 000000000..4b9311851
--- /dev/null
+++ b/scripts/release-notes/README.md
@@ -0,0 +1,16 @@
+# Print Release Notes
+
+Accepts `current_version` and `target_version` as inputs to print the necessary release notes / warnings to be shown while updating a release
+
+Run below to get more information on the usage
+
+```shell
+python print_release_notes.py -h
+```
+
+Make sure to create / activate a virtual environment with the below commands
+
+```shell
+pdm venv create -w virtualenv --with-pip
+eval $(pdm venv activate in-project)
+```
diff --git a/scripts/release-notes/print_release_notes.py b/scripts/release-notes/print_release_notes.py
new file mode 100644
index 000000000..4e15c735a
--- /dev/null
+++ b/scripts/release-notes/print_release_notes.py
@@ -0,0 +1,120 @@
+# type: ignore
+import argparse
+import json
+import re
+import sys
+from pathlib import Path
+
+from packaging import version
+
+# Sample JSON data (in a separate file 'warnings.json')
+# {
+# "v0.93.0": {
+# "message": "This a WARNING note",
+# "warning": true
+# },
+# "v0.95.0": {
+# "message": "This is an INFO note",
+# "warning": false
+# }
+# }
+
+RELEASE_NOTES_FILE_PATH = Path(__file__).with_name("release_notes.json").absolute()
+
+
+class Colour:
+ BLUE = "\033[94m"
+ GREEN = "\033[32m"
+ RED = "\033[31m"
+ DEFAULT = "\033[39m"
+ YELLOW = "\033[33m"
+
+
+def colour(text, colour: Colour):
+ return f"{colour}{text}{Colour.DEFAULT}"
+
+
+def load_release_notes(json_file):
+ """Load warnings from a JSON file."""
+ with open(json_file) as file:
+ return json.load(file)
+
+
+def is_valid_semver(ver):
+ """Check if a version string follows semantic versioning (vX.X.X)."""
+ semver_pattern = r"^v\d+\.\d+\.\d+(-[a-zA-Z0-9]+(\.\d+)?)?$"
+ return re.match(semver_pattern, ver) is not None
+
+
+def print_release_notes(current_version, target_version, release_notes):
+ """Print relevant warnings if current_version < warning_version <=
+ target_version."""
+ messages_to_print = []
+ for note_version, release_note in release_notes.items():
+ if (
+ version.parse(current_version)
+ < version.parse(note_version)
+ <= version.parse(target_version)
+ ):
+ # Check release note and display warning in yellow release_note
+ message = (
+ f"{colour('WARNING', Colour.YELLOW)}: "
+ if release_note["warning"]
+ else ""
+ )
+ message += (
+ f"[from {colour(note_version, Colour.BLUE)}]\n"
+ f"{release_note['message']}"
+ )
+ messages_to_print.append(message)
+
+ if not messages_to_print:
+ return
+
+ print("\n########################## RELEASE NOTES ##########################")
+ for message in messages_to_print:
+ print("\n" + message + "\n")
+ print("###################################################################")
+
+
+def main():
+ """Main function to load warnings and check versions."""
+ parser = argparse.ArgumentParser(
+ description="Check for version upgrade warnings based on semantic versioning."
+ )
+ parser.add_argument(
+ "current_version",
+ type=str,
+ help="Current version of Unstract (e.g., v0.85.0)",
+ )
+ parser.add_argument(
+ "target_version",
+ type=str,
+ help="Target version for the upgrade (e.g., v0.90.0)",
+ )
+
+ args = parser.parse_args()
+
+ current_version = args.current_version
+ target_version = args.target_version
+
+ # Load the warnings from the JSON file
+ release_notes = load_release_notes(RELEASE_NOTES_FILE_PATH)
+
+ # Validate the current_version and target_version
+ if not is_valid_semver(current_version) or not is_valid_semver(target_version):
+ print(
+ f"\n{colour('WARNING', Colour.YELLOW)}: "
+ "You are trying to update between versions\n"
+ f"\t({colour(current_version, Colour.BLUE)} -> "
+ f"{colour(target_version, Colour.BLUE)})\n"
+ f"Please check release notes at "
+ f"'{colour(RELEASE_NOTES_FILE_PATH, Colour.BLUE)}'\n"
+ )
+ sys.exit(1)
+
+ print_release_notes(current_version, target_version, release_notes)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/release-notes/release_notes.json b/scripts/release-notes/release_notes.json
new file mode 100644
index 000000000..4c7aa1a7a
--- /dev/null
+++ b/scripts/release-notes/release_notes.json
@@ -0,0 +1,6 @@
+{
+ "v0.93.0": {
+ "message": "A data migration is required to continue using the platform with your existing data.\nFor detailed instructions on how to perform this migration, please check:\n `backend/migrating/v2/README.md`",
+ "warning": true
+ }
+}