Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

3.6 KB
Raw Blame
# AiSOC — NVIDIA GPU overlay for the bundled Ollama
#
# Usage:
#   make up-gpu
#
# or by hand:
#   docker compose -f docker-compose.yml -f infra/compose/docker-compose.gpu.yml up -d
#
# What this changes
# -----------------
# One thing: it reserves an NVIDIA device for the `ollama` service. Nothing
# else moves. The model, the gateway, the aliases and every other service are
# whatever the base file says.
#
# Why it is an overlay and not a line in the base file
# ----------------------------------------------------
# A `deploy.resources.reservations.devices` block naming the nvidia driver is
# not advisory. On a host without an NVIDIA GPU *and* the container toolkit,
# the daemon refuses to start the service at all:
#
#   Error response from daemon: could not select device driver "nvidia" with
#   capabilities: [[gpu]]
#
# `make up` is the documented first-run path and CORE ships
# `llama3.2:3b-instruct-q4_K_M` to run CPU-only in 8 GB, so putting this in the
# base file takes out first run for every Mac, every CPU-only Linux box and
# every CI runner. It has to be opt-in, and `make up-gpu` runs
# `scripts/check_gpu_runtime.py` first so the failure names a cause and a next
# command rather than handing over the daemon's error.
#
# Apple Silicon
# -------------
# This overlay does nothing for you. Docker Desktop cannot pass the Metal GPU
# into a Linux container, so a bundled Ollama is CPU-only on a Mac no matter
# what is reserved here. Run Ollama natively instead -- it uses Metal -- and
# point the stack at it with `make up-host-llm`.
#
# Sizing
# ------
# The pinned model needs roughly 3 GB of VRAM. `count: 1` takes the first
# visible device; set `AISOC_GPU_COUNT=all` to expose every device, or pin
# specific ones with `device_ids` instead of `count` if you share the host.

# Model
# -----
# This overlay moves the default off the 3B the CPU profile ships. That is a
# measurement, not a preference: graded on the 72-item balanced verdict
# corpus, `llama3.1:8b-instruct-q4_K_M` scores 0.357 balanced accuracy
# against the 3B's 0.285 and a 0.250 constant-answer floor, and is the only
# model measured whose calibration error falls below 0.40. The table and the
# four rows it beat are published on the benchmark page under "Verdict
# quality"; `scripts/run_verdict_eval.py` reproduces it.
#
# It needs roughly 6 GB of VRAM rather than the 3B's 3 GB. Override with
# AISOC_LLM_LOCAL_MODEL plus the two routing variables if your card is
# smaller -- all three have to move together, or the gateway routes to a
# model nothing pulled.

services:
  ollama:
    environment:
      # Ollama decides layer placement itself and logs what it did. Left
      # unset rather than forced, because a hardcoded layer count is wrong
      # on every card except the one it was measured on.
      OLLAMA_NUM_PARALLEL: "${OLLAMA_NUM_PARALLEL:-1}"
    # The reservation belongs to the service that runs the model. Ollama is
    # the only one here that does any inference: litellm proxies and
    # ollama-pull downloads, so a device reserved on either of those is a
    # GPU held by something that cannot use it while generation runs on the
    # CPU -- the overlay reporting success having done nothing.
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: ${AISOC_GPU_COUNT:-1}
              capabilities: [gpu]

  ollama-pull:
    entrypoint: ["ollama", "pull", "${AISOC_LLM_LOCAL_MODEL:-llama3.1:8b-instruct-q4_K_M}"]

  litellm:
    environment:
      AISOC_LLM_MODEL_FAST: ${AISOC_LLM_MODEL_FAST:-ollama_chat/llama3.1:8b-instruct-q4_K_M}
      AISOC_LLM_MODEL_DEEP: ${AISOC_LLM_MODEL_DEEP:-ollama_chat/llama3.1:8b-instruct-q4_K_M}