BrainAPI is a knowledge graph–powered AI memory layer that transforms unstructured data into structured knowledge, enabling intelligent search, recommendations, and contextual memory for AI agents and applications.
Python ai-agentsai-memoryknowledge-graphknowledge-managementknowledge-representationreccomendation-system
Add heavy and light Docker deploy profiles.
Christianfd02fac33fd9
diff --git a/README.md b/README.mdindex 0de1d4a..882429b 100644--- a/README.md+++ b/README.md@@ -120,12 +120,17 @@ make start-console-dev # terminal 2 → http://localhost:5173/console/**Other handy targets:** `make start-api`, `make start-mcp`, `make stop-all`, and `make install-extras` (auto-installs heavy ML extras like local OCR based on your `.env`).-**Docker only:**+**Docker only** (pick a [deploy profile](deploy/README.md)):```sh-docker compose -f example-docker-compose.yaml up -d-```+# light — Postgres + DeepSeek + OpenAI embeddings+cp deploy/env.light.example .env # fill in secrets, then:+docker compose -f deploy/docker-compose.light.yaml up -d+# heavy — Neo4j + Milvus + Mongo + GCP/Azure (full resources)+cp deploy/env.heavy.example .env+docker compose -f deploy/docker-compose.heavy.yaml up -d+```Then ingest your first data point:```shdiff --git a/deploy/README.md b/deploy/README.mdnew file mode 100644index 0000000..97741a6--- /dev/null+++ b/deploy/README.md@@ -0,0 +1,46 @@+# BrainAPI deploy profiles++Choose **heavy** or **light** when bringing up a new deployment.++| | Heavy | Light |+| --- | --- | --- |+| Compose | [`docker-compose.heavy.yaml`](docker-compose.heavy.yaml) | [`docker-compose.light.yaml`](docker-compose.light.yaml) |+| Env template | [`env.heavy.example`](env.heavy.example) | [`env.light.example`](env.light.example) |+| Graph | Neo4j | NetworkX on Postgres |+| Data | MongoDB | PostgreSQL |+| Vectors | Milvus | PostgreSQL + pgvector |+| LLMs | GCP Vertex (small) + Azure (large) | DeepSeek (small + large) |+| Embeddings | Azure | OpenAI (`text-embedding-3-large`) |+| Container limits | Full (current production) | Smaller footprint (`mem_limit` / `cpus`) |+| Suggested VM | ~8–16 GB RAM / 4+ vCPU | ~2 GB RAM / 1 vCPU |++## Quick start++```bash+# 1. Pick a profile and install env+cp deploy/env.light.example /root/.env # or env.heavy.example+# edit secrets: BRAINPAT_TOKEN, DEEPSEEK_API_KEY, OPENAI_API_KEY, …++# 2. Bring up the matching compose+docker compose -f deploy/docker-compose.light.yaml up -d+# or+docker compose -f deploy/docker-compose.heavy.yaml up -d+```++## Light sizing++| Service | Heavy | Light |+| --- | --- | --- |+| nginx | 128m / 0.2 CPU | 16m / 0.025 CPU |+| redis | 2048m / 0.5 CPU (`maxmemory 1500mb`) | 256m / 0.0625 CPU (`maxmemory 192mb`) |+| postgres | — (not in heavy compose) | 512m / 0.25 CPU |+| neo4j / milvus / mongo / etcd / minio | present | omitted |+| api / worker / mcp | 4g / 2 CPU each | 512m / 0.25 CPU each |+| celery `--concurrency` | 2 | 1 |+| `CELERY_WORKER_CONCURRENCY` | 4 | 1 |++## Notes++- Both profiles expect `/root/.env` and nginx TLS paths under `/srv/nginx` (same as the legacy root `example-docker-compose.yaml`).+- Heavy still mounts `/root/gcp_credentials.json` for Vertex; light does not need GCP credentials.+- Do not mix profiles on the same host without wiping volumes — DB backends differ.diff --git a/deploy/docker-compose.heavy.yaml b/deploy/docker-compose.heavy.yamlnew file mode 100644index 0000000..99370f6--- /dev/null+++ b/deploy/docker-compose.heavy.yaml@@ -0,0 +1,383 @@+# BrainAPI deploy profile: heavy+# Full stack (Redis + Neo4j + Milvus + Mongo + API/worker/MCP).+# Pair with deploy/env.heavy.example → /root/.env+# See deploy/README.md++version: "3.8"++services:+ nginx:+ image: nginx:stable-alpine+ container_name: brainapi-nginx+ ports:+ - "80:80"+ - "443:443"+ volumes:+ - /srv/nginx/conf/nginx.conf:/etc/nginx/nginx.conf:ro+ - /srv/nginx/conf/conf.d:/etc/nginx/conf.d:ro+ - /etc/letsencrypt:/etc/letsencrypt:ro+ - /srv/nginx/logs:/var/log/nginx+ networks:+ - appnet+ restart: unless-stopped+ depends_on:+ - redis+ - mongo+ - neo4j+ - milvus+ mem_limit: 128m+ cpus: 0.2++ redis:+ image: redis:7-alpine+ container_name: brainapi-redis+ ports:+ - "6379:6379"+ volumes:+ - redis-data:/data+ command: >+ redis-server+ --appendonly yes+ --maxmemory 1500mb+ --maxmemory-policy allkeys-lru+ --save 900 1+ --save 300 10+ --save 60 10000+ --tcp-backlog 511+ --timeout 300+ networks:+ - appnet+ restart: unless-stopped+ mem_limit: 2048m+ cpus: 0.5+ healthcheck:+ test: ["CMD", "redis-cli", "ping"]+ interval: 10s+ timeout: 3s+ retries: 3++ neo4j:+ image: neo4j:5-enterprise+ container_name: brainapi-neo4j+ ports:+ - "7474:7474"+ - "7687:7687"+ volumes:+ - neo4j-data:/data+ - neo4j-logs:/logs+ - neo4j-plugins:/plugins+ - ./src/config/neo4j:/var/lib/neo4j/conf+ environment:+ - NEO4J_AUTH=neo4j/your_password+ - NEO4J_ACCEPT_LICENSE_AGREEMENT=yes+ - NEO4J_PLUGINS=["apoc"]+ - NEO4J_server_memory_heap_max__size=2g+ - NEO4J_server_memory_pagecache_size=1g+ - NEO4J_server_jvm_additional=-XX:+UseG1GC -XX:MaxGCPauseMillis=200+ networks:+ - appnet+ restart: unless-stopped+ mem_limit: 3g+ cpus: 1+ healthcheck:+ test: ["CMD-SHELL", "wget -q --spider http://localhost:7474 || exit 1"]+ interval: 15s+ timeout: 10s+ retries: 5+ start_period: 120s++ etcd:+ image: quay.io/coreos/etcd:v3.5.18+ container_name: brainapi-etcd+ environment:+ - ETCD_AUTO_COMPACTION_MODE=revision+ - ETCD_AUTO_COMPACTION_RETENTION=1000+ - ETCD_QUOTA_BACKEND_BYTES=4294967296+ - ETCD_SNAPSHOT_COUNT=50000+ volumes:+ - etcd-data:/etcd+ command: etcd -advertise-client-urls=http://etcd:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd+ networks:+ - appnet+ restart: unless-stopped+ mem_limit: 512m+ cpus: 0.2+ healthcheck:+ test: ["CMD", "etcdctl", "endpoint", "health"]+ interval: 30s+ timeout: 20s+ retries: 3++ minio:+ image: minio/minio:RELEASE.2024-12-18T13-15-44Z+ container_name: brainapi-minio+ environment:+ MINIO_ACCESS_KEY: minioadmin+ MINIO_SECRET_KEY: minioadmin+ ports:+ - "9001:9001"+ - "9000:9000"+ volumes:+ - minio-data:/minio_data+ command: minio server /minio_data --console-address ":9001"+ networks:+ - appnet+ restart: unless-stopped+ mem_limit: 1g+ cpus: 0.4+ healthcheck:+ test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]+ interval: 30s+ timeout: 20s+ retries: 3++ milvus:+ image: milvusdb/milvus:v2.6.6+ container_name: brainapi-milvus+ command: ["milvus", "run", "standalone"]+ security_opt:+ - seccomp:unconfined+ environment:+ ETCD_ENDPOINTS: etcd:2379+ MINIO_ADDRESS: minio:9000+ MINIO_ACCESS_KEY: minioadmin+ MINIO_SECRET_KEY: minioadmin+ MQ_TYPE: woodpecker+ MILVUS_COMMON_LOG_LEVEL: warn+ volumes:+ - milvus-data:/var/lib/milvus+ ports:+ - "19530:19530"+ - "9091:9091"+ networks:+ - appnet+ depends_on:+ etcd:+ condition: service_healthy+ minio:+ condition: service_healthy+ restart: unless-stopped+ mem_limit: 2g+ cpus: 0.6+ healthcheck:+ test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]+ interval: 30s+ start_period: 90s+ timeout: 20s+ retries: 3++ mongo:+ image: mongo:8.0+ container_name: brainapi-mongo+ ports:+ - "27017:27017"+ environment:+ MONGO_INITDB_ROOT_USERNAME: root+ MONGO_INITDB_ROOT_PASSWORD: password+ volumes:+ - mongo-data:/data/db+ - mongo-config:/data/configdb+ command: >+ mongod+ --wiredTigerCacheSizeGB=5+ --wiredTigerCollectionBlockCompressor=snappy+ --wiredTigerIndexPrefixCompression=true+ --logpath=/var/log/mongodb/mongod.log+ --logappend+ --bind_ip_all+ networks:+ - appnet+ restart: unless-stopped+ mem_limit: 1.5g+ cpus: 0.4+ healthcheck:+ test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"]+ interval: 10s+ timeout: 5s+ retries: 5++ brainapi:+ image: ghcr.io/lumen-labs/brainapi:v2.9.7-dev+ container_name: brainapi-api+ ports:+ - "8000:8000"+ env_file:+ - /root/.env+ environment:+ PLUGIN_REGISTRY_URL: ${PLUGIN_REGISTRY_URL:-}+ PLUGIN_PUBLISHER_ID: ${PLUGIN_PUBLISHER_ID:-}+ PLUGIN_PUBLISHER_API_KEY: ${PLUGIN_PUBLISHER_API_KEY:-}+ BRAINAPI_PLUGINS: my-plugin:1.0.0,analytics-plugin+ volumes:+ - /root/gcp_credentials.json:/app/gcp_credentials.json+ - brainapi-plugins:/app/plugins+ command:+ [+ "-m",+ "uvicorn",+ "src.services.api.app:app",+ "--host",+ "0.0.0.0",+ "--port",+ "8000",+ "--workers",+ "1",+ "--timeout-keep-alive",+ "30",+ "--limit-concurrency",+ "200",+ "--backlog",+ "2048",+ ]+ depends_on:+ redis:+ condition: service_healthy+ neo4j:+ condition: service_started+ mongo:+ condition: service_healthy+ milvus:+ condition: service_healthy+ networks:+ - appnet+ restart: unless-stopped+ mem_limit: 4g+ cpus: 2+ healthcheck:+ test:+ [+ "CMD",+ "curl",+ "-f",+ "http://localhost:8000/",+ "-H",+ "BrainPAT: ${BRAINPAT_TOKEN}",+ ]+ interval: 30s+ timeout: 10s+ retries: 5+ start_period: 900s+ start_interval: 30s++ brainapi-worker:+ image: ghcr.io/lumen-labs/brainapi:v2.9.7-dev+ container_name: brainapi-worker+ env_file:+ - /root/.env+ environment:+ BRAINAPI_PLUGINS: ${BRAINAPI_PLUGINS:-}+ PLUGIN_REGISTRY_URL: ${PLUGIN_REGISTRY_URL:-}+ PLUGIN_PUBLISHER_ID: ${PLUGIN_PUBLISHER_ID:-}+ PLUGIN_PUBLISHER_API_KEY: ${PLUGIN_PUBLISHER_API_KEY:-}+ volumes:+ - /root/gcp_credentials.json:/app/gcp_credentials.json+ - brainapi-plugins:/app/plugins+ command:+ [+ "-m",+ "celery",+ "-A",+ "src.workers.app",+ "worker",+ "--loglevel=info",+ "--pool=threads",+ "--without-gossip",+ "--without-mingle",+ "--concurrency=2",+ "--max-tasks-per-child=50",+ ]+ depends_on:+ redis:+ condition: service_healthy+ neo4j:+ condition: service_started+ mongo:+ condition: service_healthy+ milvus:+ condition: service_healthy+ networks:+ - appnet+ restart: unless-stopped+ mem_limit: 4g+ cpus: 2+ healthcheck:+ test:+ [+ "CMD-SHELL",+ "python -m celery -A src.workers.app inspect ping --timeout 5 >/dev/null 2>&1 && q=$(redis-cli -h redis -p 6379 LLEN ingest_data 2>/dev/null || echo 0); [ \"$q\" -lt \"${CELERY_QUEUE_HEALTH_THRESHOLD:-50}\" ]",+ ]+ interval: 30s+ timeout: 10s+ retries: 5+ start_period: 120s+ start_interval: 30s++ brainapi-mcp:+ image: ghcr.io/lumen-labs/brainapi:v2.9.7-dev+ container_name: brainapi-mcp+ ports:+ - "8001:8001"+ env_file:+ - /root/.env+ environment:+ MCP_OAUTH_ISSUER_URL: ${MCP_OAUTH_ISSUER_URL:-}+ MCP_RESOURCE_SERVER_URL: ${MCP_RESOURCE_SERVER_URL:-}+ BRAINAPI_PLUGINS: ${BRAINAPI_PLUGINS:-}+ PLUGIN_REGISTRY_URL: ${PLUGIN_REGISTRY_URL:-}+ PLUGIN_PUBLISHER_ID: ${PLUGIN_PUBLISHER_ID:-}+ PLUGIN_PUBLISHER_API_KEY: ${PLUGIN_PUBLISHER_API_KEY:-}+ volumes:+ - /root/gcp_credentials.json:/app/gcp_credentials.json+ - brainapi-plugins:/app/plugins+ command:+ [+ "-m",+ "uvicorn",+ "src.services.mcp.app:app",+ "--host",+ "0.0.0.0",+ "--port",+ "8001",+ "--timeout-keep-alive",+ "30",+ ]+ depends_on:+ redis:+ condition: service_healthy+ neo4j:+ condition: service_started+ mongo:+ condition: service_healthy+ milvus:+ condition: service_healthy+ networks:+ - appnet+ restart: unless-stopped+ mem_limit: 4g+ cpus: 2+ healthcheck:+ test:+ [+ "CMD-SHELL",+ 'code=$(curl -s -o /dev/null -w ''%{http_code}'' http://localhost:8001/); test "$code" = ''200'' || test "$code" = ''405''',+ ]+ interval: 30s+ timeout: 10s+ retries: 3+ start_period: 60s++networks:+ appnet:+ driver: bridge++volumes:+ redis-data:+ neo4j-data:+ neo4j-logs:+ neo4j-plugins:+ etcd-data:+ minio-data:+ milvus-data:+ mongo-data:+ mongo-config:+ brainapi-plugins:\ No newline at end of filediff --git a/deploy/docker-compose.light.yaml b/deploy/docker-compose.light.yamlnew file mode 100644index 0000000..1646ab7--- /dev/null+++ b/deploy/docker-compose.light.yaml@@ -0,0 +1,240 @@+# BrainAPI deploy profile: light+# Slim stack (Redis + Postgres/pgvector + API/worker/MCP).+# Pair with deploy/env.light.example → /root/.env+# See deploy/README.md++version: "3.8"++services:+ nginx:+ image: nginx:stable-alpine+ container_name: brainapi-nginx+ ports:+ - "80:80"+ - "443:443"+ volumes:+ - /srv/nginx/conf/nginx.conf:/etc/nginx/nginx.conf:ro+ - /srv/nginx/conf/conf.d:/etc/nginx/conf.d:ro+ - /etc/letsencrypt:/etc/letsencrypt:ro+ - /srv/nginx/logs:/var/log/nginx+ networks:+ - appnet+ restart: unless-stopped+ depends_on:+ - redis+ - postgres+ mem_limit: 16m+ cpus: 0.025++ redis:+ image: redis:7-alpine+ container_name: brainapi-redis+ ports:+ - "6379:6379"+ volumes:+ - redis-data:/data+ command: >+ redis-server+ --appendonly yes+ --maxmemory 192mb+ --maxmemory-policy allkeys-lru+ --save 900 1+ --save 300 10+ --save 60 10000+ --tcp-backlog 511+ --timeout 300+ networks:+ - appnet+ restart: unless-stopped+ mem_limit: 256m+ cpus: 0.0625+ healthcheck:+ test: ["CMD", "redis-cli", "ping"]+ interval: 10s+ timeout: 3s+ retries: 3++ postgres:+ image: pgvector/pgvector:pg16+ container_name: brainapi-postgres+ ports:+ - "5432:5432"+ environment:+ POSTGRES_USER: postgres+ POSTGRES_PASSWORD: password+ volumes:+ - postgres-data:/var/lib/postgresql/data+ networks:+ - appnet+ restart: unless-stopped+ mem_limit: 512m+ cpus: 0.25+ healthcheck:+ test: ["CMD-SHELL", "pg_isready -U postgres"]+ interval: 5s+ timeout: 5s+ retries: 10++ brainapi:+ image: ghcr.io/lumen-labs/brainapi:v2.9.7-dev+ container_name: brainapi-api+ ports:+ - "8000:8000"+ env_file:+ - /root/.env+ environment:+ PLUGIN_REGISTRY_URL: ${PLUGIN_REGISTRY_URL:-}+ PLUGIN_PUBLISHER_ID: ${PLUGIN_PUBLISHER_ID:-}+ PLUGIN_PUBLISHER_API_KEY: ${PLUGIN_PUBLISHER_API_KEY:-}+ BRAINAPI_PLUGINS: ${BRAINAPI_PLUGINS:-}+ volumes:+ - brainapi-plugins:/app/plugins+ command:+ [+ "-m",+ "uvicorn",+ "src.services.api.app:app",+ "--host",+ "0.0.0.0",+ "--port",+ "8000",+ "--workers",+ "1",+ "--timeout-keep-alive",+ "30",+ "--limit-concurrency",+ "50",+ "--backlog",+ "256",+ ]+ depends_on:+ redis:+ condition: service_healthy+ postgres:+ condition: service_healthy+ networks:+ - appnet+ restart: unless-stopped+ mem_limit: 512m+ cpus: 0.25+ healthcheck:+ test:+ [+ "CMD",+ "curl",+ "-f",+ "http://localhost:8000/",+ "-H",+ "BrainPAT: ${BRAINPAT_TOKEN}",+ ]+ interval: 30s+ timeout: 10s+ retries: 5+ start_period: 300s+ start_interval: 30s++ brainapi-worker:+ image: ghcr.io/lumen-labs/brainapi:v2.9.7-dev+ container_name: brainapi-worker+ env_file:+ - /root/.env+ environment:+ BRAINAPI_PLUGINS: ${BRAINAPI_PLUGINS:-}+ PLUGIN_REGISTRY_URL: ${PLUGIN_REGISTRY_URL:-}+ PLUGIN_PUBLISHER_ID: ${PLUGIN_PUBLISHER_ID:-}+ PLUGIN_PUBLISHER_API_KEY: ${PLUGIN_PUBLISHER_API_KEY:-}+ volumes:+ - brainapi-plugins:/app/plugins+ command:+ [+ "-m",+ "celery",+ "-A",+ "src.workers.app",+ "worker",+ "--loglevel=info",+ "--pool=threads",+ "--without-gossip",+ "--without-mingle",+ "--concurrency=1",+ "--max-tasks-per-child=50",+ ]+ depends_on:+ redis:+ condition: service_healthy+ postgres:+ condition: service_healthy+ networks:+ - appnet+ restart: unless-stopped+ mem_limit: 512m+ cpus: 0.25+ healthcheck:+ test:+ [+ "CMD-SHELL",+ "python -m celery -A src.workers.app inspect ping --timeout 5 >/dev/null 2>&1 && q=$(redis-cli -h redis -p 6379 LLEN ingest_data 2>/dev/null || echo 0); [ \"$q\" -lt \"${CELERY_QUEUE_HEALTH_THRESHOLD:-50}\" ]",+ ]+ interval: 30s+ timeout: 10s+ retries: 5+ start_period: 120s+ start_interval: 30s++ brainapi-mcp:+ image: ghcr.io/lumen-labs/brainapi:v2.9.7-dev+ container_name: brainapi-mcp+ ports:+ - "8001:8001"+ env_file:+ - /root/.env+ environment:+ MCP_OAUTH_ISSUER_URL: ${MCP_OAUTH_ISSUER_URL:-}+ MCP_RESOURCE_SERVER_URL: ${MCP_RESOURCE_SERVER_URL:-}+ BRAINAPI_PLUGINS: ${BRAINAPI_PLUGINS:-}+ PLUGIN_REGISTRY_URL: ${PLUGIN_REGISTRY_URL:-}+ PLUGIN_PUBLISHER_ID: ${PLUGIN_PUBLISHER_ID:-}+ PLUGIN_PUBLISHER_API_KEY: ${PLUGIN_PUBLISHER_API_KEY:-}+ volumes:+ - brainapi-plugins:/app/plugins+ command:+ [+ "-m",+ "uvicorn",+ "src.services.mcp.app:app",+ "--host",+ "0.0.0.0",+ "--port",+ "8001",+ "--timeout-keep-alive",+ "30",+ ]+ depends_on:+ redis:+ condition: service_healthy+ postgres:+ condition: service_healthy+ networks:+ - appnet+ restart: unless-stopped+ mem_limit: 512m+ cpus: 0.25+ healthcheck:+ test:+ [+ "CMD-SHELL",+ 'code=$(curl -s -o /dev/null -w ''%{http_code}'' http://localhost:8001/); test "$code" = ''200'' || test "$code" = ''405''',+ ]+ interval: 30s+ timeout: 10s+ retries: 3+ start_period: 60s++networks:+ appnet:+ driver: bridge++volumes:+ redis-data:+ postgres-data:+ brainapi-plugins:diff --git a/deploy/env.heavy.example b/deploy/env.heavy.examplenew file mode 100644index 0000000..47a9801--- /dev/null+++ b/deploy/env.heavy.example@@ -0,0 +1,209 @@+# BrainAPI deploy profile: heavy+# Pair with deploy/docker-compose.heavy.yaml+# Stack: Neo4j + Milvus + Mongo + Redis; LLMs: GCP small + Azure large/embeddings++# LLM Implementation envs+LLM_SMALL_PROVIDER="gcp_vertex"+LLM_LARGE_PROVIDER="azure"+EMBEDDINGS_PROVIDER="azure"+AZURE_SMALL_LLM_MODEL="gpt-4o-mini"+AZURE_LARGE_LLM_MODEL="gpt-4o"+AZURE_LLM_API_VERSION="2024-12-01-preview"+AZURE_LLM_ENDPOINT="https://yourproject.openai.azure.com"+AZURE_LLM_SUBSCRIPTION_KEY="xyz"+GCP_EXTRA_SMALL_LLM_MODEL="gemini-3-flash-preview"+GCP_SMALL_LLM_MODEL="gemini-3-flash-preview"+GCP_LARGE_LLM_MODEL="gemini-3-flash-preview"+GCP_EMBEDDING_MODEL="text-embedding-005"+GCP_PROJECT_ID="your-project-id"+GCP_CREDENTIALS_PATH=".path-to-credentials-file.json"+BEDROCK_REGION="us-east-1"+BEDROCK_ACCESS_KEY_ID="your-access-key-id"+BEDROCK_SECRET_ACCESS_KEY="your-secret-access-key"+BEDROCK_SESSION_TOKEN=""+BEDROCK_SMALL_LLM_MODEL="us.anthropic.claude-3-5-haiku-20241022-v1:0"+BEDROCK_LARGE_LLM_MODEL="us.anthropic.claude-3-7-sonnet-20250219-v1:0"+BEDROCK_EMBEDDING_MODEL="amazon.titan-embed-text-v2:0"+OPENAI_API_KEY="sk-..."+OPENAI_BASE_URL=""+OPENAI_SMALL_LLM_MODEL="gpt-4o-mini"+OPENAI_LARGE_LLM_MODEL="gpt-4o"+OPENAI_EMBEDDING_MODEL="text-embedding-3-large"+# DeepSeek chat (OpenAI-compatible). No embeddings — keep EMBEDDINGS_PROVIDER on azure/openai/gcp_vertex/etc.+# LLM_SMALL_PROVIDER="deepseek"+# LLM_LARGE_PROVIDER="deepseek"+DEEPSEEK_API_KEY="sk-..."+DEEPSEEK_SMALL_LLM_MODEL="deepseek-v4-flash"+DEEPSEEK_LARGE_LLM_MODEL="deepseek-v4-pro"++# Embedding envs+# The following are just examples of the adapter implementation, you can have completely different envs+AZURE_EMBEDDING_FULL_ENDPOINT="https://yourproject.openai.azure.com/openai/deployments/text-embedding-3-large/embeddings?api-version=2023-05-15"+AZURE_EMBEDDING_KEY="xyz"+AZURE_EMBEDDING_MODEL="text-embedding-3-large"+EMBEDDINGS_SMALL_MODEL="paraphrase-multilingual-MiniLM-L12-v2"+EMBEDDINGS_LOCAL_MODEL="intfloat/e5-small"+EMBEDDING_NODES_DIMENSION=3072+EMBEDDING_TRIPLETS_DIMENSION=3072+EMBEDDING_OBSERVATIONS_DIMENSION=3072+EMBEDDING_DATA_DIMENSION=3072+EMBEDDING_RELATIONSHIPS_DIMENSION=3072++# Cache envs+# The following are just examples of the adapter implementation, you can have completely different envs+REDIS_HOST="redis"+REDIS_PORT=6379++# Worker envs+# Can be whatever you prefer+# CELERY_BACKEND="rabbitmq"+# CELERY_UNBLOCKED_WINDOW_SECONDS is the window in seconds during which the number of unblocked events is counted+# CELERY_UNBLOCKED_MAX_RETRIES is the maximum number of unblocked events that can occur during the window+# CELERY_QUEUE_HEALTH_THRESHOLD is the maximum number of tasks that can be in the queue before the worker is considered unhealthy+CELERY_BACKEND="redis"+CELERY_UNBLOCKED_WINDOW_SECONDS=60+CELERY_UNBLOCKED_MAX_RETRIES=20+CELERY_QUEUE_HEALTH_THRESHOLD=50++# GraphDB envs+# The following are just examples of the adapter implementation, you can have completely different envs+NEO4J_HOST="neo4j"+NEO4J_PORT=7687+NEO4J_USERNAME="neo4j"+NEO4J_PASSWORD="your_password"++# VectorDB envs+# The following are just examples of the adapter implementation, you can have completely different envs+MILVUS_HOST="milvus"+MILVUS_PORT=19530++# PostgreSQL envs+# Used for data / vector / graph storage when DATA_DB, VECTOR_DB, or GRAPH_DB+# point at postgresql / pgvector / networkx.+# Each brain owns its own Postgres database (brain_<sanitized_id>).+# POSTGRES_SYSTEM_DATABASE holds the brains registry (data_brains table).+# POSTGRES_MAINTENANCE_DATABASE is used only for CREATE DATABASE statements.+POSTGRES_HOST="localhost"+POSTGRES_PORT=5432+POSTGRES_USERNAME="postgres"+POSTGRES_PASSWORD="password"+POSTGRES_SYSTEM_DATABASE="brainapi"+POSTGRES_MAINTENANCE_DATABASE="postgres"++# DataDB envs (Mongo alternative to PostgreSQL data layer)+# The following are just examples of the adapter implementation, you can have completely different envs+MONGO_HOST="mongo"+MONGO_PORT=27017+MONGO_USERNAME="root"+MONGO_PASSWORD="password"+MONGO_SYSTEM_DATABASE="system"++# Auth+# Used to handle authentication+BRAINPAT_TOKEN="your_token"++# MultiBrain+# Choose to allow or block automatic creation of new brains on requests with non existing new brain_ids+BRAIN_CREATION_ALLOWED="true"+# Choose whether to fallback to default brain if not provided+DEFAULT_BRAIN_FALLBACK="true"++# GCP+# Used on the oss project for the small LLM but can be changed to any other model+GCP_EXTRA_SMALL_LLM_MODEL="gemini-3-flash-preview"+GCP_SMALL_LLM_MODEL="gemini-3-flash-preview"+GCP_PROJECT_ID="your-project-id"+GCP_CREDENTIALS_PATH=".path-to-credentials-file.json"++# Celery+# Used to run the worker tasks+CELERY_WORKER_CONCURRENCY=4++# DocParser+# Used to parse the documents into markdown text+# and by forwarding the webhook (our app host) and the task identifier,+# we can be called back from the DocParser API to start the ingestion process+# with the created markdown text.+DOCPARSER_ENDPOINT="https://docparser.lumen-labs.ai"+APP_HOST="https://your-app-host.com/ingest"+DOCPARSER_TOKEN="your-docparser-token"++# Spacy+# Used to keep the spacy models in memory or reinitialize them on each request+SPACY_KEEP_MODELS_IN_MEMORY="true"++# Pipeline+# Used to configure the pipeline+# PIPELINE_MODE => lightweight = faster but less accurate, accurate = slower but more accurate+# MODELS_MODE => local = uses local models for llm and embeddings, remote = uses remote models for llm and embeddings+# OCR_MODE => docling = uses local docling ocr engine, docparser = uses remote docparser ocr pipeline engine+# AGENTIC_ARCHITECTURE => custom = uses the custom agentic architecture (faster and token efficient), langchain = uses the langchain agentic architecture+# RUN_GRAPH_CONSOLIDATOR => true = consolidates after ingest (LLM mutations; keep false until audited)+# RUN_OBSERVATIONS => true = ObservationsAgent LLM notes (does not write the graph)+# JANITOR_BATCH_SIZE => relationships per Janitor LLM call when batching (default 20)+# INGEST_JANITOR_MAX_LLM_CALLS => hard cap on Janitor LLM batches/session (default 2); remainder dropped with audit+# INGEST_ARCHITECT_PER_UNIT => true = Architect sees Scout chunks (+ prior window); false = full text (legacy)+# INGEST_DEFER_JANITOR => true = batched post-Architect Janitor; false = per-create Janitor (legacy)+# INGEST_ARCHITECT_MODE => batch (default; pure tools=[] extract; escalate to tooler on fail) | schema | tooler (ReAct rollback)+# INGEST_ARCHITECT_MAX_SCHEMA_CALLS => max LLM calls/unit in schema|batch before escalate (default 3)+# INGEST_ARCHITECT_ESCALATE => true (default) allow tooler escalate on empty schema; false = partial/empty unit+# INGEST_ARCHITECT_ESCALATE_MAX_TURNS => max tool turns on escalate tooler path (default 3)+# INGEST_ARCHITECT_DENSE_ENTITY_THRESHOLD => rechunk architect unit when Scout entities >= N (default 12)+# INGEST_ARCHITECT_DENSE_MAX_CHARS => max chars per dense rechunk part (default 1200)+# INGEST_ARCHITECT_PRIOR_CONTEXT => auto (default; scratchpad for batch/schema, raw for tooler) | scratchpad | raw+# INGEST_ARCHITECT_SCRATCHPAD_TOKEN_CAP => max reference tokens for prior scratchpad (default 500)+PIPELINE_MODE="accurate"+MODELS_MODE="remote"+OCR_MODE="docparser"+AGENTIC_ARCHITECTURE="custom"+RUN_GRAPH_CONSOLIDATOR="false"+RUN_OBSERVATIONS="false"+JANITOR_BATCH_SIZE="20"+INGEST_JANITOR_MAX_LLM_CALLS="2"+INGEST_ARCHITECT_PER_UNIT="true"+INGEST_DEFER_JANITOR="true"+INGEST_ARCHITECT_MODE="batch"+INGEST_ARCHITECT_MAX_SCHEMA_CALLS="3"+INGEST_ARCHITECT_ESCALATE="true"+INGEST_ARCHITECT_ESCALATE_MAX_TURNS="3"+INGEST_ARCHITECT_DENSE_ENTITY_THRESHOLD="12"+INGEST_ARCHITECT_DENSE_MAX_CHARS="1200"+INGEST_ARCHITECT_PRIOR_CONTEXT="auto"+INGEST_ARCHITECT_SCRATCHPAD_TOKEN_CAP="500"+GRAPH_DB="neo4j"+DATA_DB="mongo"+VECTOR_DB="milvus"++# Ollama+# Used to configure the ollama models+# Tests with models using OpenAI's function tool calling are not currently compatible with BrainAPI+OLLAMA_HOST="localhost"+OLLAMA_PORT=11434+# OLLAMA_LLM_SMALL_MODEL="hf.co/unsloth/gpt-oss-20b-GGUF:Q4_K_M"+# OLLAMA_LLM_LARGE_MODEL="hf.co/unsloth/gpt-oss-20b-GGUF:Q4_K_M"+# OLLAMA_LLM_LARGE_MODEL="hf.co/MaziyarPanahi/Qwen3-8B-GGUF:Q4_K_M"+# OLLAMA_LLM_SMALL_MODEL="hf.co/MaziyarPanahi/Qwen3-8B-GGUF:Q4_K_M"+OLLAMA_LLM_SMALL_MODEL="hf.co/unsloth/Qwen3-14B-GGUF:Q4_K_M"+OLLAMA_LLM_LARGE_MODEL="hf.co/unsloth/Qwen3-14B-GGUF:Q4_K_M"++# Plugins+PLUGINS_DIR="plugins"+PLUGIN_REGISTRY_URL="https://registry.brain-api.dev"+PLUGIN_PUBLISHER_API_KEY="your-publisher-api-key"+PLUGIN_PUBLISHER_ID="your-publisher-id"++# MCP+MCP_OAUTH_ISSUER_URL="https://your-mcp-oauth-issuer-url.com"+MCP_RESOURCE_SERVER_URL="https://your-mcp-resource-server-url.com"++# Tracing+TRACE_TRACKER_ENABLED=false+TRACE_QUEUE_MAX_SIZE=10000+TRACE_SLOW_REQUEST_MS=1000+TRACE_AGENT_SLOW_OPERATION_MS=30000+TRACE_HEARTBEAT_INTERVAL_SECONDS=30+TRACE_HEALTH_INTERVAL_SECONDS=30+TRACE_RESOURCE_INTERVAL_SECONDS=30+TRACE_HEALTH_TIMEOUT_SECONDS=1+TRACE_AGENT_LOOP_ITERATIONS=20+TRACE_AGENT_TOOL_LOOP_ITERATIONS=20\ No newline at end of filediff --git a/deploy/env.light.example b/deploy/env.light.examplenew file mode 100644index 0000000..7ab76ef--- /dev/null+++ b/deploy/env.light.example@@ -0,0 +1,207 @@+# BrainAPI deploy profile: light+# Pair with deploy/docker-compose.light.yaml+# Stack: Postgres/pgvector + NetworkX + Redis; LLMs: DeepSeek; embeddings: OpenAI++# LLM Implementation envs+LLM_SMALL_PROVIDER="deepseek"+LLM_LARGE_PROVIDER="deepseek"+EMBEDDINGS_PROVIDER="openai"+AZURE_SMALL_LLM_MODEL="gpt-4o-mini"+AZURE_LARGE_LLM_MODEL="gpt-4o"+AZURE_LLM_API_VERSION="2024-12-01-preview"+AZURE_LLM_ENDPOINT="https://yourproject.openai.azure.com"+AZURE_LLM_SUBSCRIPTION_KEY="xyz"+GCP_EXTRA_SMALL_LLM_MODEL="gemini-3-flash-preview"+GCP_SMALL_LLM_MODEL="gemini-3-flash-preview"+GCP_LARGE_LLM_MODEL="gemini-3-flash-preview"+GCP_EMBEDDING_MODEL="text-embedding-005"+GCP_PROJECT_ID="your-project-id"+GCP_CREDENTIALS_PATH=".path-to-credentials-file.json"+BEDROCK_REGION="us-east-1"+BEDROCK_ACCESS_KEY_ID="your-access-key-id"+BEDROCK_SECRET_ACCESS_KEY="your-secret-access-key"+BEDROCK_SESSION_TOKEN=""+BEDROCK_SMALL_LLM_MODEL="us.anthropic.claude-3-5-haiku-20241022-v1:0"+BEDROCK_LARGE_LLM_MODEL="us.anthropic.claude-3-7-sonnet-20250219-v1:0"+BEDROCK_EMBEDDING_MODEL="amazon.titan-embed-text-v2:0"+OPENAI_API_KEY="sk-..."+OPENAI_BASE_URL=""+OPENAI_SMALL_LLM_MODEL="gpt-4o-mini"+OPENAI_LARGE_LLM_MODEL="gpt-4o"+OPENAI_EMBEDDING_MODEL="text-embedding-3-large"+# DeepSeek chat (OpenAI-compatible). Embeddings stay on OpenAI above.+DEEPSEEK_API_KEY="sk-..."+DEEPSEEK_SMALL_LLM_MODEL="deepseek-v4-flash"+DEEPSEEK_LARGE_LLM_MODEL="deepseek-v4-pro"++# Embedding envs+# The following are just examples of the adapter implementation, you can have completely different envs+AZURE_EMBEDDING_FULL_ENDPOINT="https://yourproject.openai.azure.com/openai/deployments/text-embedding-3-large/embeddings?api-version=2023-05-15"+AZURE_EMBEDDING_KEY="xyz"+AZURE_EMBEDDING_MODEL="text-embedding-3-large"+EMBEDDINGS_SMALL_MODEL="paraphrase-multilingual-MiniLM-L12-v2"+EMBEDDINGS_LOCAL_MODEL="intfloat/e5-small"+EMBEDDING_NODES_DIMENSION=3072+EMBEDDING_TRIPLETS_DIMENSION=3072+EMBEDDING_OBSERVATIONS_DIMENSION=3072+EMBEDDING_DATA_DIMENSION=3072+EMBEDDING_RELATIONSHIPS_DIMENSION=3072++# Cache envs+# The following are just examples of the adapter implementation, you can have completely different envs+REDIS_HOST="redis"+REDIS_PORT=6379++# Worker envs+# Can be whatever you prefer+# CELERY_BACKEND="rabbitmq"+# CELERY_UNBLOCKED_WINDOW_SECONDS is the window in seconds during which the number of unblocked events is counted+# CELERY_UNBLOCKED_MAX_RETRIES is the maximum number of unblocked events that can occur during the window+# CELERY_QUEUE_HEALTH_THRESHOLD is the maximum number of tasks that can be in the queue before the worker is considered unhealthy+CELERY_BACKEND="redis"+CELERY_UNBLOCKED_WINDOW_SECONDS=60+CELERY_UNBLOCKED_MAX_RETRIES=20+CELERY_QUEUE_HEALTH_THRESHOLD=50++# GraphDB envs+# The following are just examples of the adapter implementation, you can have completely different envs+NEO4J_HOST="localhost"+NEO4J_PORT=7687+NEO4J_USERNAME="neo4j"+NEO4J_PASSWORD="your_password"++# VectorDB envs+# The following are just examples of the adapter implementation, you can have completely different envs+MILVUS_HOST="localhost"+MILVUS_PORT=19530++# PostgreSQL envs+# Used for data / vector / graph storage when DATA_DB, VECTOR_DB, or GRAPH_DB+# point at postgresql / pgvector / networkx.+# Each brain owns its own Postgres database (brain_<sanitized_id>).+# POSTGRES_SYSTEM_DATABASE holds the brains registry (data_brains table).+# POSTGRES_MAINTENANCE_DATABASE is used only for CREATE DATABASE statements.+POSTGRES_HOST="postgres"+POSTGRES_PORT=5432+POSTGRES_USERNAME="postgres"+POSTGRES_PASSWORD="password"+POSTGRES_SYSTEM_DATABASE="brainapi"+POSTGRES_MAINTENANCE_DATABASE="postgres"++# DataDB envs (Mongo alternative to PostgreSQL data layer)+# The following are just examples of the adapter implementation, you can have completely different envs+MONGO_HOST="localhost"+MONGO_PORT=27017+MONGO_USERNAME="root"+MONGO_PASSWORD="password"+MONGO_SYSTEM_DATABASE="system"++# Auth+# Used to handle authentication+BRAINPAT_TOKEN="your_token"++# MultiBrain+# Choose to allow or block automatic creation of new brains on requests with non existing new brain_ids+BRAIN_CREATION_ALLOWED="true"+# Choose whether to fallback to default brain if not provided+DEFAULT_BRAIN_FALLBACK="true"++# GCP+# Used on the oss project for the small LLM but can be changed to any other model+GCP_EXTRA_SMALL_LLM_MODEL="gemini-3-flash-preview"+GCP_SMALL_LLM_MODEL="gemini-3-flash-preview"+GCP_PROJECT_ID="your-project-id"+GCP_CREDENTIALS_PATH=".path-to-credentials-file.json"++# Celery+# Used to run the worker tasks+CELERY_WORKER_CONCURRENCY=1++# DocParser+# Used to parse the documents into markdown text+# and by forwarding the webhook (our app host) and the task identifier,+# we can be called back from the DocParser API to start the ingestion process+# with the created markdown text.+DOCPARSER_ENDPOINT="https://docparser.lumen-labs.ai"+APP_HOST="https://your-app-host.com/ingest"+DOCPARSER_TOKEN="your-docparser-token"++# Spacy+# Used to keep the spacy models in memory or reinitialize them on each request+SPACY_KEEP_MODELS_IN_MEMORY="true"++# Pipeline+# Used to configure the pipeline+# PIPELINE_MODE => lightweight = faster but less accurate, accurate = slower but more accurate+# MODELS_MODE => local = uses local models for llm and embeddings, remote = uses remote models for llm and embeddings+# OCR_MODE => docling = uses local docling ocr engine, docparser = uses remote docparser ocr pipeline engine+# AGENTIC_ARCHITECTURE => custom = uses the custom agentic architecture (faster and token efficient), langchain = uses the langchain agentic architecture+# RUN_GRAPH_CONSOLIDATOR => true = consolidates after ingest (LLM mutations; keep false until audited)+# RUN_OBSERVATIONS => true = ObservationsAgent LLM notes (does not write the graph)+# JANITOR_BATCH_SIZE => relationships per Janitor LLM call when batching (default 20)+# INGEST_JANITOR_MAX_LLM_CALLS => hard cap on Janitor LLM batches/session (default 2); remainder dropped with audit+# INGEST_ARCHITECT_PER_UNIT => true = Architect sees Scout chunks (+ prior window); false = full text (legacy)+# INGEST_DEFER_JANITOR => true = batched post-Architect Janitor; false = per-create Janitor (legacy)+# INGEST_ARCHITECT_MODE => batch (default; pure tools=[] extract; escalate to tooler on fail) | schema | tooler (ReAct rollback)+# INGEST_ARCHITECT_MAX_SCHEMA_CALLS => max LLM calls/unit in schema|batch before escalate (default 3)+# INGEST_ARCHITECT_ESCALATE => true (default) allow tooler escalate on empty schema; false = partial/empty unit+# INGEST_ARCHITECT_ESCALATE_MAX_TURNS => max tool turns on escalate tooler path (default 3)+# INGEST_ARCHITECT_DENSE_ENTITY_THRESHOLD => rechunk architect unit when Scout entities >= N (default 12)+# INGEST_ARCHITECT_DENSE_MAX_CHARS => max chars per dense rechunk part (default 1200)+# INGEST_ARCHITECT_PRIOR_CONTEXT => auto (default; scratchpad for batch/schema, raw for tooler) | scratchpad | raw+# INGEST_ARCHITECT_SCRATCHPAD_TOKEN_CAP => max reference tokens for prior scratchpad (default 500)+PIPELINE_MODE="accurate"+MODELS_MODE="remote"+OCR_MODE="docparser"+AGENTIC_ARCHITECTURE="custom"+RUN_GRAPH_CONSOLIDATOR="false"+RUN_OBSERVATIONS="false"+JANITOR_BATCH_SIZE="20"+INGEST_JANITOR_MAX_LLM_CALLS="2"+INGEST_ARCHITECT_PER_UNIT="true"+INGEST_DEFER_JANITOR="true"+INGEST_ARCHITECT_MODE="batch"+INGEST_ARCHITECT_MAX_SCHEMA_CALLS="3"+INGEST_ARCHITECT_ESCALATE="true"+INGEST_ARCHITECT_ESCALATE_MAX_TURNS="3"+INGEST_ARCHITECT_DENSE_ENTITY_THRESHOLD="12"+INGEST_ARCHITECT_DENSE_MAX_CHARS="1200"+INGEST_ARCHITECT_PRIOR_CONTEXT="auto"+INGEST_ARCHITECT_SCRATCHPAD_TOKEN_CAP="500"+GRAPH_DB="networkx"+DATA_DB="postgresql"+VECTOR_DB="postgresql"++# Ollama+# Used to configure the ollama models+# Tests with models using OpenAI's function tool calling are not currently compatible with BrainAPI+OLLAMA_HOST="localhost"+OLLAMA_PORT=11434+# OLLAMA_LLM_SMALL_MODEL="hf.co/unsloth/gpt-oss-20b-GGUF:Q4_K_M"+# OLLAMA_LLM_LARGE_MODEL="hf.co/unsloth/gpt-oss-20b-GGUF:Q4_K_M"+# OLLAMA_LLM_LARGE_MODEL="hf.co/MaziyarPanahi/Qwen3-8B-GGUF:Q4_K_M"+# OLLAMA_LLM_SMALL_MODEL="hf.co/MaziyarPanahi/Qwen3-8B-GGUF:Q4_K_M"+OLLAMA_LLM_SMALL_MODEL="hf.co/unsloth/Qwen3-14B-GGUF:Q4_K_M"+OLLAMA_LLM_LARGE_MODEL="hf.co/unsloth/Qwen3-14B-GGUF:Q4_K_M"++# Plugins+PLUGINS_DIR="plugins"+PLUGIN_REGISTRY_URL="https://registry.brain-api.dev"+PLUGIN_PUBLISHER_API_KEY="your-publisher-api-key"+PLUGIN_PUBLISHER_ID="your-publisher-id"++# MCP+MCP_OAUTH_ISSUER_URL="https://your-mcp-oauth-issuer-url.com"+MCP_RESOURCE_SERVER_URL="https://your-mcp-resource-server-url.com"++# Tracing+TRACE_TRACKER_ENABLED=false+TRACE_QUEUE_MAX_SIZE=10000+TRACE_SLOW_REQUEST_MS=1000+TRACE_AGENT_SLOW_OPERATION_MS=30000+TRACE_HEARTBEAT_INTERVAL_SECONDS=30+TRACE_HEALTH_INTERVAL_SECONDS=30+TRACE_RESOURCE_INTERVAL_SECONDS=30+TRACE_HEALTH_TIMEOUT_SECONDS=1+TRACE_AGENT_LOOP_ITERATIONS=20+TRACE_AGENT_TOOL_LOOP_ITERATIONS=20\ No newline at end of filediff --git a/example-docker-compose.yaml b/example-docker-compose.yamlindex e8a9d2d..87b8728 100644--- a/example-docker-compose.yaml+++ b/example-docker-compose.yaml@@ -1,3 +1,8 @@+# Legacy path — prefer deploy/ profiles for new installs:+# heavy → deploy/docker-compose.heavy.yaml + deploy/env.heavy.example+# light → deploy/docker-compose.light.yaml + deploy/env.light.example+# See deploy/README.md+version: "3.8"services: