LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Python ai-agentsdata-engineeringdocument-aigenerative-aiidpjson-extractionllmmcp-serverocrpdf-extractionprompt-engineeringstructured-output
Feat/platform launch script (#28)
Jaseem Jas4f784bfc263d
diff --git a/README.md b/README.mdindex bf0d32ff9..8a0f28b5b 100644--- a/README.md+++ b/README.md@@ -2,32 +2,51 @@[](https://pdm-project.org)-TODO: Write few lines about the project.+Use LLMs to eliminate manual processes involving unstructured data.## System Requirements-- docker-- git+- `docker` (see [instructions](https://docs.docker.com/engine/install/))+- `git`+- `pdm` (see below)+- `pyenv` (recommended to manage multiple Python versions)-## Running with docker compose+## Quick Start-- All services needed by the backend can be run with+Just run the `run-platform.sh` launch script to get started in few minutes.-```-cd docker/-VERSION=test docker compose -f docker-compose.build.yaml build-VERSION=test docker compose -f docker-compose.yaml up -d-```+The launch script does env setup with default values, pulls public Docker images or builds them locally and finally runs them in containers.-Additional information on running with Docker can be found in [DOCKERISING.md](/DOCKERISING.md)+```bash+# Pull and run entire Unstract platform with default env config.+./run-platform.sh-- Use the `-f` flag to run all dependencies necessary for development, this runs containers needed for testing as well such as Minio.+# Pull and run docker containers with a specific version tag.+./run-platform.sh -v v0.1.0+# Build docker images and run as containers with a specific version tag.+./run-platform.sh -B -v v0.1.0++# Display the help information.+./run-platform.sh -h++# Only do setup of environment files.+./run-platform.sh -e++# Only do docker images pull/build with a specific version tag.+./run-platform.sh -b v0.1.0++# Pull and run Docker containers in detached mode.+./run-platform.sh -d -v v0.1.0```-docker compose -f docker-compose-dev-essentials.yaml up-```-- It might take sometime on the first run to pull the images.+Now visit [http://frontend.unstract.localhost](http://frontend.unstract.localhost) in your browser.++That's all. Enjoy!++## Running with docker compose++See [Docker README.md](docker/README.md).## Running locally@@ -36,75 +55,82 @@ docker compose -f docker-compose-dev-essentials.yaml up- Install the below libraries which are needed to run Unstract- Linux- ```- sudo apt install build-essential pkg-config libpoppler-cpp-dev libmagic-dev python3-dev+ ```bash+ apt install build-essential libmagic-dev pandoc pkg-config tesseract-ocr```- Mac- ```- brew install pkg-config poppler freetds libmagic+ ```bash+ brew install freetds libmagic pkg-config poppler```### Create your virtual env-- In order to install dependencies and run a package, ensure that you've sourced a virtual environment within that package. All commands in this repository assumes that you have sourced your required venv.+All commands assumes that you have activated your `venv`.-```-cd <package_to_use>-python -m venv .venv-source ./venv/bin/activate+```bash+cd <service>++# Create venv+pdm venv create -w virtualenv --with-pip+eval "$(pdm venv activate in-project)"++# Remove venv+pdm venv remove in-project```### Install dependencies with PDM-- This repository makes use of [PDM](https://github.com/pdm-project/pdm) for managing dependencies with the help of a virtual-environment.-- If you haven't installed PDM in your machine yet,- - Install it using the below command- ```- curl -sSL https://pdm.fming.dev/install-pdm.py | python3 -- ```- - Or install it from PyPI using `pip`- ```- pip install pdm- ```+[PDM](https://github.com/pdm-project/pdm) is used for dependency management.-Ensure you're running the PDM commands from the corresponding package root-- Install dependencies for running the package with+```bash+# Install via script+curl -sSL https://pdm.fming.dev/install-pdm.py | python3 -+# Install via pip+pip install pdm```++Go to service dir and install dependencies listed in corresponding `pyproject.toml`.++```bash+# Install dependenciespdm install-```-This install dev dependencies as well by default-- For production, install the requirements with-```-pdm install --prod-```+# Install specific dev dependency group+pdm install --dev -G lint-- With PDM its possible to run some services from any directory within this-repository. To list the possible scripts that can be executed+# Install production dependencies only+pdm install --prod --no-editable```++PDM allows you to run scripts applicable within the service dir.++```bash+# List the possible scripts that can be executedpdm run -l```-- Add a new dependency with (ensure you're running it from the correct project's root)-Perform an editable install with `-e` only for local development.-```+Add dependencies as follows.++```bash+# Add a new service dependency to ts pyproject.toml.pdm add <package_from_PyPI>+# Add a relative path as an editable install.pdm add -e <relative_path_to_local_package>-```-- List all dependencies with-```+# List all dependencies.pdm list```-- After updating `pyproject.toml`s with a newly added dependency, the lock file can be updated with++After modifying `pyproject.toml`, the lock file can be updated as below.+```pdm lock```-- Refer [PDM's documentation](https://pdm.fming.dev/latest/reference/cli/) for further details.++See [PDM's documentation](https://pdm.fming.dev/latest/reference/cli/) for further details.### Configuring Postgres@@ -152,14 +178,14 @@ If you require a different config, make sure the necessary envs from [backend/sa### Backend-- Check [backend/README.md](/backend/README.md) for running the backend.+- Check [backend/README.md](backend/README.md) for running the backend.### Frontend- Install dependencies with `npm install`- Start the server with `npm start`-### Traefik Proxy Overrides+### Traefik Proxy Overrides for Local + Docker RunsIt is possible to simultaneously run few services directly on docker host while others are run as docker containers via docker compose.This enables seamless development without worrying about deployment of other services which you are not concerned with.diff --git a/backend/sample.env b/backend/sample.envindex 5fca2ee4d..363212649 100644--- a/backend/sample.env+++ b/backend/sample.env@@ -26,7 +26,7 @@ GOOGLE_OAUTH2_KEY=GOOGLE_OAUTH2_SECRET=# User session-SESSION_EXPIRATION_TIME_IN_SECOND=+SESSION_EXPIRATION_TIME_IN_SECOND=7200# FE Web Application DependenciesLOGIN_NEXT_URL="http://frontend.unstract.localhost/org"@@ -73,7 +73,7 @@ UNSTRACT_WORKER_PORT=5002# Document ServiceDOCUMENT_PROCESSOR_URL=http://unstract-document-service:3002-DOCUMENT_PROCESSOR_API_KEY=+DOCUMENT_PROCESSOR_API_KEY="0vpepz5CyzRIAKuy+9ZJXLOCXO+TYOAt+55M+ewxGMk="# Workflow executionWORKFLOW_DATA_DIR = "/data/execution"diff --git a/docker/docker-compose.yaml b/docker/docker-compose.yamlindex 79cf942a0..ed2e772c1 100644--- a/docker/docker-compose.yaml+++ b/docker/docker-compose.yaml@@ -49,7 +49,7 @@ services:celery-flower:image: unstract/backend:${VERSION}container_name: unstract-celery-flower- entrypoint: /app/.venv/bin/celery+ entrypoint: .venv/bin/celerycommand: "-A backend flower --port=5555"env_file:- ../backend/.env@@ -69,7 +69,7 @@ services:celery-beat:image: unstract/backend:${VERSION}container_name: unstract-celery-beat- entrypoint: /app/.venv/bin/celery+ entrypoint: .venv/bin/celerycommand: "-A backend beat --scheduler django_celery_beat.schedulers:DatabaseScheduler -l INFO"env_file:- ../backend/.env@@ -110,6 +110,8 @@ services:prompt-service:image: unstract/prompt-service:${VERSION}container_name: unstract-prompt-service+ depends_on:+ - dbports:- "3003:3003"env_file:diff --git a/docker/scripts/deploy.sh b/docker/scripts/deploy.shdeleted file mode 100755index c23663277..000000000--- a/docker/scripts/deploy.sh+++ /dev/null@@ -1,145 +0,0 @@-#!/bin/bash--display_help() {- echo "Usage: $0 [options]"- echo "Options:"- echo " -h, --help Display this help message"- echo " -v, --version Version tag for built Docker images (default \"dev\")"- echo " Ignored if --no-build is specified"- echo " -E, --no-setup-env Do not setup env"- echo " -B, --no-build Do not build Docker images"- echo " -l, --local Deploy in local instead of a Docker container"- echo " Requires --services (accepts only one service)"- echo " -s, --services <names> Deploys the specified services only"- echo " Expects service names as a comma separated list"- echo " -V, --verbose Print verbose logs"-}--debug() {- if [ "$opt_verbose" = true ]; then- echo $1- fi-}--log() {- echo $1-}--parse_args() {- while [[ $# -gt 0 ]]; do- arg="$1"- case $arg in- -h|--help)- display_help- exit 0- ;;- -v|--version)- opt_version="$2"- shift- ;;- -E|--no-setup-env)- opt_no_setup_env=true- ;;- -l|--local)- opt_local=true- ;;- -s|--services)- opt_services="$2"- shift- ;;- -V|--verbose)- opt_verbose=true- ;;- *)- echo "Unknown option: $arg"- display_help- exit 1- ;;- esac- shift- done-- debug "OPTION version: $opt_version"- debug "OPTION no_setup_env: $opt_no_setup_env"- debug "OPTION local: $opt_local"- debug "OPTION services: $opt_services"- debug "OPTION verbose: $opt_verbose"-}--check_system_requirements() {- system_requirements=("git" "docker")- for sr in "${system_requirements[@]}"; do- command -v $sr >/dev/null 2>&1 || { echo >&2 "$sr is not installed. Exiting."; exit 1; }- done-}--_deploy_selected_services() {- debug "Deploying selected services"-- script_dir=$(dirname "$(readlink -f "$BASH_SOURCE")")- project_dir="${script_dir}/../.."- IFS=',' read -r -a deploy_services <<< "$opt_services"-- not_services=("docker" "document_display_service" "tools" "unstract")-- debug "Services to deploy: ${deploy_services[@]}"- debug "Not services list: ${not_services[@]}"-- for service_path in "$project_dir"/*; do- if [ ! -d "$service_path" ]; then- continue- fi-- service="$(basename "$service_path")"-- # Check if path is included in not services list.- if [[ $(echo ${not_services[@]} | grep -F -w $service) ]]- then- debug "Not a service, skipping: $service"- continue- fi-- if [ "${#deploy_services[@]}" -gt 0 ] &&- ! [[ $(echo ${deploy_services[@]} | grep -F -w $service) ]]- then- debug "Service not opted for deploy: $service"- continue- fi-- log "Deploying service: $service"- done-}--_setup_env_all_services() {- debug "Setting up env for all services"-}--_build_all_services() {- debug "Building all services"- VERSION=$opt_version docker compose -f docker-compose.build.yaml build-}--_deploy_all_services() {- debug "Deploying all services"- VERSION=$opt_version docker compose -f docker-compose.yaml up -d-}--deploy_services() {- if [ -n "$opt_services" ]; then- _deploy_selected_services- else- _setup_env_all_services- _build_all_services- _deploy_all_services- fi-}--opt_version="dev"-opt_no_setup_env=false-opt_local=false-opt_services=""-opt_verbose=false--parse_args "$@"-check_system_requirements-deploy_servicesdiff --git a/document-service/sample.env b/document-service/sample.envindex 52bdae823..272e10beb 100644--- a/document-service/sample.env+++ b/document-service/sample.env@@ -1,4 +1,4 @@-SERVICE_API_TOKEN=""+SERVICE_API_TOKEN="0vpepz5CyzRIAKuy+9ZJXLOCXO+TYOAt+55M+ewxGMk="UPLOAD_FOLDER="/app/uploads"PROCESS_FOLDER="/app/processed"LIBREOFFICE_PYTHON="/usr/bin/python3"diff --git a/run-platform.sh b/run-platform.shnew file mode 100755index 000000000..b82b23b1e--- /dev/null+++ b/run-platform.sh@@ -0,0 +1,213 @@+#!/usr/bin/env bash++set -o nounset # exit if a variable is not set+set -o errexit # exit for any command failure"++# text color escape codes (\033 == \e but OSX doesn't respect the \e)+blue_text='\033[94m'+red_text='\033[31m'+default_text='\033[39m'++# set -x/xtrace uses PS4 for more info+PS4="$blue_text""${0}:${LINENO}: ""$default_text"++debug() {+ if [ "$opt_verbose" = true ]; then+ echo $1+ fi+}++display_banner() {+ # Make sure the console is huge+ if test $(tput cols) -ge 64; then+ # Make it green!+ echo -e "\033[32m"+ echo -e " _ _ _ _ _____ _______ _____ _____ _______"+ echo -e "| | | | \ | |/ ____|__ __| __ \ /\ / ____|__ __|"+ echo -e "| | | | \| | (___ | | | |__) | / \ | | | |"+ echo -e "| | | | . |\___ \ | | | _ / / /\ \| | | |"+ echo -e "| |__| | |\ |____) | | | | | \ \ / ____ \ |____ | |"+ echo -e " \____/|_| \_|_____/ |_| |_| \_\/_/ \_\_____| |_|"+ echo -e " "+ # Make it less green+ echo -e "\033[0m"+ sleep 1+ fi+}++display_help() {+ printf "Run Unstract platform in docker containers\n"+ echo+ echo -e "Syntax: $0 [options]"+ echo -e "Options:"+ echo -e " -h, --help Displays the help information"+ echo -e " -e, --only-env Only do env files setup"+ echo -e " -b, --only-build Only do docker images build"+ echo -e " -B, --build-local Build docker images locally"+ echo -e " -d, --detach Run docker containers in detached mode"+ echo -e " -x, --trace Enables trace mode"+ echo -e " -V, --verbose Print verbose logs"+ echo -e " -v, --version Docker images version tag (default \"dev\")"+ echo -e ""+}++parse_args() {+ while [[ $# -gt 0 ]]; do+ arg="$1"+ case $arg in+ -h | --help)+ display_help+ exit+ ;;+ -e | --only-env)+ opt_only_env=true+ ;;+ -b | --only-build)+ opt_only_build=true+ ;;+ -B | --build-local)+ opt_build_local=true+ ;;+ -d | --detach)+ opt_detach="-d"+ ;;+ -x | --trace)+ set -o xtrace # display every line before execution; enables PS4+ ;;+ -V | --verbose)+ opt_verbose=true+ ;;+ -v | --version)+ opt_version="$2"+ shift+ ;;+ *)+ echo "'$1' is not a known command."+ echo+ display_help+ exit+ ;;+ esac+ shift+ done++ debug "OPTION only_env: $opt_only_env"+ debug "OPTION only_build: $opt_only_build"+ debug "OPTION build_local: $opt_build_local"+ debug "OPTION detach: $opt_detach"+ debug "OPTION verbose: $opt_verbose"+ debug "OPTION version: $opt_version"+}++setup_env() {+ for service in "${services[@]}"; do+ sample_env_path="$script_dir/$service/sample.env"+ env_path="$script_dir/$service/.env"++ # Generate Fernet Key. Refer https://pypi.org/project/cryptography/.+ ENCRYPTION_KEY=$(python -c "import secrets, base64; print(base64.urlsafe_b64encode(secrets.token_bytes(32)).decode())")++ if [ -e "$sample_env_path" ] && [ ! -e "$env_path" ]; then+ cp "$sample_env_path" "$env_path"+ # Add encryption secret for backend and platform-service.+ if [[ "$service" == "backend" || "$service" == "platform-service" ]]; then+ echo -e "$blue_text""Adding encryption secret to $service""$default_text"+ echo "ENCRYPTION_KEY=\"$ENCRYPTION_KEY\"" >> $env_path+ fi+ echo -e "Created env for ""$blue_text""$service""$default_text" at ""$blue_text""$env_path""$default_text"."+ else+ echo -e "Found existing env for ""$blue_text""$service""$default_text" at ""$blue_text""$env_path""$default_text"."+ fi+ done++ if [ ! -e "$script_dir/docker/essentials.env" ]; then+ cp "$script_dir/docker/sample.essentials.env" "$script_dir/docker/essentials.env"+ echo -e "Created env for ""$blue_text""essential services""$default_text"" at ""$blue_text""$script_dir/docker/essentials.env""$default_text""."+ else+ echo -e "Found existing env for ""$blue_text""essential services""$default_text"" at ""$blue_text""$script_dir/docker/essentials.env""$default_text""."+ fi++ if [ ! -e "$script_dir/docker/proxy_overrides.yaml" ]; then+ echo -e "NOTE: Proxy behaviour can be overridden via ""$blue_text""$script_dir/docker/proxy_overrides.yaml""$default_text""."+ else+ echo -e "Found ""$blue_text""$script_dir/docker/proxy_overrides.yaml""$default_text"". Proxy behaviour will be overridden."+ fi++ if [ "$opt_only_env" = true ]; then+ echo "Done." && exit 0+ fi+}++build_services() {+ pushd ${script_dir}/docker 1>/dev/null++ if [ "$opt_build_local" = true ]; then+ echo "Building docker images locally."+ VERSION=$opt_version docker-compose -f $script_dir/docker/docker-compose.build.yaml build || {+ echo -e "$red_text""Failed to build docker images.""$default_text"+ exit 1+ }+ else+ for service in "${services[@]}"; do+ if ! docker image inspect "unstract/${service}:$opt_version" &> /dev/null; then+ echo -e "$red_text""Docker image 'unstract/${service}:$opt_version' not found. Pulling...""$default_text"+ docker pull unstract/${service}:$opt_version || {+ echo -e "$red_text""Failed to pull 'unstract/${service}:$opt_version'. Check if version exists.""$default_text"+ exit 1+ }+ else+ echo -e "Found docker image ""$blue_text""'unstract/${service}:$opt_version'""$default_text""."+ fi+ done+ fi++ popd 1>/dev/null++ if [ "$opt_only_build" = true ]; then+ echo "Done." && exit 0+ fi+}++run_services() {+ pushd ${script_dir}/docker 1>/dev/null++ if [ -z "$opt_detach" ]; then+ echo -e "$blue_text""Starting docker containers""$default_text"+ else+ echo -e "$blue_text""Starting docker containers in detach mode""$default_text"+ fi+ echo -e "Once the services are up, visit ""$blue_text""http://frontend.unstract.localhost""$default_text"" in your browser."++ VERSION=$opt_version docker compose up $opt_detach++ popd 1>/dev/null+}++#+# Run Unstract platform - BEGIN+#+if ! command -v docker compose &> /dev/null; then+ echo "$red_text""docker compose not found. Please install it and try again.""$default_text"+ exit 1+fi++opt_only_env=false+opt_only_build=false+opt_build_local=false+opt_detach=""+opt_verbose=false+opt_version="dev"++script_dir=$(dirname "$(readlink -f "$BASH_SOURCE")")+# Extract service names from docker compose file.+services=($(VERSION=$opt_version docker compose -f $script_dir/docker/docker-compose.build.yaml config --services))++display_banner+parse_args $*++setup_env+build_services+run_services+#+# Run Unstract platform - END+#\ No newline at end of file