LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Unstract Workers - Operations Guide
This guide consolidates deployment, monitoring, and troubleshooting information for Unstract workers.
Deployment
Development Setup
# Quick setup for development
cd /home/ali/projects/unstract/workers
./scripts/setup.sh --environment development
# Install dependencies
uv sync
for dir in api-deployment general file-processing callback; do
cd $dir && uv sync && cd ..
done
Production Deployment
Docker Deployment
# Build images
VERSION=local docker compose -f docker-compose.build.yaml build \
worker-api-deployment worker-callback worker-file-processing worker-general
# Run workers
VERSION=local docker compose --profile workers-new up -d
Kubernetes Deployment
# See docker/kubernetes/ for complete manifests
kubectl apply -f docker/kubernetes/workers/
Environment Configuration
Copy sample.env to .env and configure:
Required:
INTERNAL_API_BASE_URL: Backend internal API URLINTERNAL_SERVICE_API_KEY: Authentication keyCELERY_BROKER_URL: Message broker URLCELERY_RESULT_BACKEND: Result storage backend
Worker-Specific:
[WORKER]_MAX_CONCURRENT_TASKS: Concurrency limits[WORKER]_HEALTH_PORT: Health check ports[WORKER]_QUEUE: Queue names
Monitoring
Health Checks
# Check all workers health
./scripts/monitor.sh health
# Individual worker health
curl http://localhost:8080/health # API deployment worker
curl http://localhost:8081/health # General worker
curl http://localhost:8082/health # File processing worker
curl http://localhost:8083/health # Callback worker
Metrics (Prometheus)
PG-queue processes serve Prometheus text-format metrics at /metrics on
their existing health port (WORKER_PG_QUEUE_CONSUMER_HEALTH_PORT for
consumers, WORKER_PG_REAPER_HEALTH_PORT for the reaper — same server as
/health; no port configured means neither endpoint):
- Every PG consumer / the fleet supervisor (per-pod):
pg_consumer_heartbeat_age_seconds(poll-loop freshness — the same signal/healthverdicts on); the supervisor addspg_consumer_alive_children/pg_consumer_configured_concurrency. - Reaper only (the leader-elected singleton — queue-WIDE state comes from one
process):
pg_reaper_heartbeat_age_seconds(tick-loop freshness),pg_queue_depth{queue},pg_queue_oldest_message_age_seconds{queue},pg_barrier_live/pg_barrier_stranded, outcome + failure counters (pg_reaper_barrier_*_total,pg_reaper_claim_*_total,pg_reaper_sweep_failures_total{table},pg_reaper_tick_failures_total,pg_reaper_gauge_refresh_failures_total) andpg_reaper_is_leader. Queue gauges are cached snapshots refreshed on the reaper's own cadence (60s —_GAUGE_REFRESH_INTERVAL_SECONDSinreaper.py) — scrapes never touch the DB;pg_queue_gauges_age_secondsexposes snapshot staleness and keeps growing while refreshes fail or the pod is a standby (alert on it together withpg_reaper_is_leader == 1).
curl http://localhost:8090/metrics # PG consumer / supervisor
curl http://localhost:8086/metrics # PG reaper
Nothing scrapes these in cloud yet (no PodMonitoring/alerts — deferred); they're for direct curl during incidents, dashboards-to-come, and the load-test harness.
Celery workers expose no app-level /metrics; their queue observability
comes from RabbitMQ's Prometheus plugin and Flower (below).
Logging
Workers use structured logging with configurable levels:
export LOG_LEVEL=DEBUG # DEBUG, INFO, WARNING, ERROR
export LOG_FORMAT=json # json or simple
Flower Dashboard
# Start Flower for task monitoring
celery -A backend flower --port=5555
# Access at http://localhost:5555
Troubleshooting
Common Issues
Workers Not Processing Tasks
Check connectivity:
# Test Redis/RabbitMQ connection redis-cli ping rabbitmqctl statusVerify API access:
curl -H "X-API-Key: $INTERNAL_SERVICE_API_KEY" \ $INTERNAL_API_BASE_URL/v1/health/Check worker logs:
docker logs unstract-worker-api-deployment-new # Or for local development tail -f logs/api-deployment-worker.log
Memory Issues
- Adjust
CELERY_WORKER_MAX_TASKS_PER_CHILD(default: 1000) - Configure
[WORKER]_MAX_CONCURRENT_TASKSbased on available memory - Enable memory profiling with
ENABLE_MEMORY_PROFILING=true
Task Timeouts
- Increase
CELERY_TASK_TIMEOUT(default: 300s) - Adjust
CELERY_TASK_SOFT_TIMEOUT(default: 270s) - Configure worker-specific timeouts in environment
Circuit Breaker Trips
When API calls fail repeatedly:
- Check
CIRCUIT_BREAKER_FAILURE_THRESHOLD(default: 5) - Adjust
CIRCUIT_BREAKER_RECOVERY_TIMEOUT(default: 60s) - Review backend API logs for errors
Debug Mode
Enable detailed debugging:
export DEBUG=true
export LOG_LEVEL=DEBUG
export CELERY_TASK_ALWAYS_EAGER=true # Run tasks synchronously
Performance Tuning
Concurrency Settings
# Per worker type
API_DEPLOYMENT_MAX_CONCURRENT_TASKS=5
GENERAL_MAX_CONCURRENT_TASKS=10
FILE_PROCESSING_MAX_CONCURRENT_TASKS=4
CALLBACK_MAX_CONCURRENT_TASKS=3
Autoscaling
# Format: max,min
API_DEPLOYMENT_AUTOSCALE=4,1
GENERAL_AUTOSCALE=6,2
FILE_PROCESSING_AUTOSCALE=8,2
CALLBACK_AUTOSCALE=4,1
Connection Pooling
CONNECTION_POOL_SIZE=10
CONNECTION_POOL_MAX_OVERFLOW=20
Maintenance
Log Rotation
Configure log rotation in production:
# /etc/logrotate.d/unstract-workers
/var/log/unstract/workers/*.log {
daily
rotate 7
compress
missingok
notifempty
}
Backup and Recovery
- Task results are stored in Redis/PostgreSQL
- Configure
EXECUTION_RESULT_TTL_SECONDSfor retention - Enable
ENABLE_TASK_BACKUP=truefor critical workflows
Upgrading Workers
Rolling update (recommended):
./scripts/deploy.sh --environment production --action rolling-updateBlue-green deployment:
# Deploy new version ./scripts/deploy.sh --environment production --action deploy --version new # Switch traffic ./scripts/deploy.sh --environment production --action switch --to new # Remove old version ./scripts/deploy.sh --environment production --action cleanup --version old
Support
- Documentation: See README.md for architecture details
- Issues: Report at https://github.com/unstract/unstract/issues
- Logs: Check
/var/log/unstract/workers/or Docker logs - Metrics: Access Prometheus at
:9090and Grafana at:3000