LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Docker Commands
Docker Build
# Build all services
VERSION=dev docker compose -f docker-compose.build.yaml build
# Build a specific service alone
VERSION=dev docker compose -f docker-compose.build.yaml build frontend
# Build optional services also
VERSION=dev docker compose -f docker-compose.build.yaml --profile optional build
Docker Run
NOTE: First copy sample.*.env files to *.env and update as required.
# Up all services
VERSION=dev docker compose -f docker-compose.yaml up -d
# Up a specific service alone
VERSION=dev docker compose -f docker-compose.yaml up -d frontend
# Up optional services also
VERSION=dev docker compose -f docker-compose.yaml --profile optional up -d
Now access frontend at http://frontend.unstract.localhost
Overriding a service's config
By making use of the merge compose files feature its possible to override some configuration that's used by the services.
Copy and rename the sample.compose.override.yaml to compose.override.yaml and update it as necessary.
cp sample.compose.override.yaml compose.override.yaml
# Configuration in docker-compose.yaml gets overridden
VERSION=dev docker compose -f docker-compose.yaml -f compose.override.yaml up -d
This can be useful during development to:
- Not run some memory intensive services
- Use commands with different arguments to save resources
- Mount additional volumes or define additional env to configure behaviour
Development with Docker Compose Watch
Docker Compose Watch (available in Docker Compose v2.22.0+) enables a streamlined development workflow by automatically syncing code changes to containers and restarting services as needed.
Setting Up Watch Mode
Ensure you're using Docker Compose v2.22.0 or higher
docker compose versionCreate your
compose.override.yamlwith watch configurationscp sample.compose.override.yaml compose.override.yamlStart services with watch mode enabled
VERSION=dev docker compose -f docker-compose.yaml -f compose.override.yaml watch
NOTE: Make sure to specify the build definitions also in your
compose.override.yamlfile or specify docker-compose.build.yaml while running the above command.
Example Workflow
Start services with watch mode:
VERSION=dev docker compose -f docker-compose.yaml -f compose.override.yaml watchMake changes to your code - they're automatically synced and services restart as needed
View logs:
docker compose logs -f [service_name]
Debugging Containers
Enable debugpy by adding compose.debug.yaml:
VERSION=dev docker compose -f docker-compose.yaml -f compose.override.yaml -f compose.debug.yaml watch
Debug ports per service:
| Service | Port |
|---|---|
| backend | 5678 |
| runner | 5679 |
| platform-service | 5680 |
| prompt-service | 5681 |
| V2 Workers | |
| worker-pg-fileproc | 5682 |
| worker-pg-callback | 5683 |
| worker-pg-orchestrator-api | 5684 |
| worker-pg-orchestrator-general | 5685 |
| worker-pg-notification | 5686 |
| worker-log-stream-consumer | 5687 |
| worker-pg-scheduler | 5688 |
VSCode Configuration
Example launch.json to attach to the backend container:
{
"name": "Docker: Backend Remote Debug",
"type": "debugpy",
"request": "attach",
"connect": { "host": "localhost", "port": 5678 },
"pathMappings": [
{ "localRoot": "${workspaceFolder:unstract}/backend", "remoteRoot": "/app" },
{ "localRoot": "${workspaceFolder:unstract}/unstract", "remoteRoot": "/unstract" }
],
"justMyCode": false,
"django": true
}
See VSCode docs for more details.
src Folder Layout and gunicorn
For the following project structure:
scheduler
|- src
| |- unstract
| |- scheduler
| |- main.py
|- uv.lock
|- pyproject.toml
This will install the project to:
.venv/lib/python3.12/site-packages/unstract/scheduler/main.py
This will allow gunicorn to refer the package directly as:
$ gunicorn "-c" "python:unstract.scheduler.config.gunicorn" "unstract.scheduler.main:app"