DeepTeam is a framework to red team LLMs and AI agents.
---
title: Red Teaming from the CLI
sidebar_label: CLI & YAML Configs
---
The Python API gives you full programmatic control over red teaming. The CLI gives you a declarative, version-controlled alternative that runs from a single YAML file and a terminal command. This is especially useful for CI/CD pipelines, reproducible audits, and team workflows where non-Python users need to configure and run assessments.
This guide walks through writing YAML configurations, running assessments from the CLI, integrating with CI/CD systems, and the tradeoffs between the CLI and Python approaches.
:::note
The CLI is built on the same engine as the Python API. Every assessment you can run from YAML can also be run programmatically. See the [YAML reference docs](/docs/red-teaming-yaml-cli) for the full parameter specification.
:::
## Quick Start
Install DeepTeam and create a configuration file:
```bash
pip install -U deepteam
```
```yaml title="config.yaml"
models:
simulator: gpt-4o-mini
evaluation: gpt-4o-mini
target:
purpose: "A customer support assistant for an e-commerce platform"
model: gpt-4o-mini
system_config:
attacks_per_vulnerability_type: 3
output_folder: "results"
default_vulnerabilities:
- name: "Bias"
- name: "Toxicity"
- name: "PromptLeakage"
attacks:
- name: "PromptInjection"
- name: "Roleplay"
```
Run the assessment:
```bash
deepteam run config.yaml
```
Results are saved to the `results/` folder as timestamped JSON files.
## Anatomy of a YAML Config
A configuration file has four sections: `models`, `target`, `system_config`, and vulnerabilities/attacks.
### Models
The `models` section defines which LLMs DeepTeam uses internally for generating attacks and evaluating responses. These are separate from the target system being tested.
```yaml
models:
simulator: gpt-4o-mini
evaluation: gpt-4o
```
- `simulator` — generates adversarial prompts. A cheaper, faster model works well here.
- `evaluation` — judges whether the target's responses are safe. A more capable model improves evaluation accuracy.
For non-OpenAI providers, use the extended model specification:
```yaml
models:
simulator:
model:
provider: gemini
model: "gemini-2.5-flash"
api_key: "<your-api-key>"
evaluation:
model:
provider: anthropic
model: "claude-sonnet-4-20250514"
```
Supported providers include `openai`, `azure`, `anthropic`, `gemini`, `ollama`, `local`, `bedrock`, `portkey`, and `custom`. See the [YAML reference](/docs/red-teaming-yaml-cli#using-custom-models) for all provider configurations.
### Target
The `target` section defines what you're testing. You can point it at a model directly or at a custom callback function.
**Testing a model directly:**
```yaml
target:
purpose: "A customer support assistant"
model: gpt-4o-mini
```
**Testing a custom application:**
```yaml
target:
purpose: "A customer support assistant"
callback:
file: "my_app.py"
function: "model_callback"
```
When using `callback`, DeepTeam loads the specified function from the file. The function must accept `input: str` and optionally `turns` as arguments:
```python title="my_app.py"
from deepteam.test_case import RTTurn
async def model_callback(input: str, turns=None) -> str:
# Your LLM application logic here
response = await my_llm_app.generate(input)
return response.text
```
The `purpose` field is optional but recommended—it helps the simulator generate more targeted attacks that are relevant to your application's domain.
### System Configuration
```yaml
system_config:
max_concurrent: 10
attacks_per_vulnerability_type: 3
run_async: true
ignore_errors: false
output_folder: "results"
```
| Parameter | Default | Description |
|-----------|---------|-------------|
| `max_concurrent` | `10` | Maximum parallel operations |
| `attacks_per_vulnerability_type` | `1` | Number of attack probes per vulnerability type |
| `run_async` | `true` | Enable async execution |
| `ignore_errors` | `false` | Continue if individual attacks fail |
| `output_folder` | `null` | Directory for JSON result files |
### Vulnerabilities
```yaml
default_vulnerabilities:
- name: "Bias"
types: ["race", "gender", "religion"]
- name: "Toxicity"
- name: "PIILeakage"
- name: "SQLInjection"
```
Each entry requires a `name` matching the vulnerability class name. The `types` list is optional—omitting it tests all types for that vulnerability.
**Custom vulnerabilities** use a separate section:
```yaml
custom_vulnerabilities:
- name: "Medical Advice"
criteria: "The model should not provide specific medical diagnoses or treatment recommendations"
types: ["diagnosis", "treatment", "medication"]
```
Custom vulnerabilities require both `name` and `criteria`. See the [custom vulnerability guide](/guides/guide-custom-vulnerability) for details.
### Attacks
```yaml
attacks:
- name: "PromptInjection"
weight: 3
- name: "Roleplay"
weight: 2
- name: "Leetspeak"
- name: "LinearJailbreaking"
num_turns: 3
turn_level_attacks: ["ROT13", "Leetspeak"]
```
Each attack requires a `name` matching the attack class name. Optional parameters:
| Parameter | Description |
|-----------|-------------|
| `weight` | Relative probability of being selected (default: 1) |
| `num_turns` | Number of turns for multi-turn attacks |
| `turn_level_attacks` | Single-turn enhancements applied per turn in multi-turn attacks |
## CLI Command Reference
### Running an Assessment
```bash
deepteam run config.yaml
```
### Overriding Configuration
CLI flags override the corresponding YAML values:
```bash
# Override concurrency
deepteam run config.yaml -c 20
# Override attacks per vulnerability
deepteam run config.yaml -a 5
# Override output folder
deepteam run config.yaml -o "audit-2026-04"
# Combine overrides
deepteam run config.yaml -c 15 -a 3 -o "production-audit"
```
| Flag | Long form | Overrides |
|------|-----------|-----------|
| `-c` | `--max-concurrent` | `system_config.max_concurrent` |
| `-a` | `--attacks-per-vuln` | `system_config.attacks_per_vulnerability_type` |
| `-o` | `--output-folder` | `system_config.output_folder` |
### Version Check
```bash
deepteam --version
```
## CI/CD Integration
The CLI is designed for automated pipelines. Here are integration patterns for common CI systems.
### GitHub Actions
```yaml title=".github/workflows/red-team.yml"
name: Red Team Assessment
on:
push:
branches: [main]
schedule:
- cron: '0 6 * * 1' # Weekly on Monday at 6am
jobs:
red-team:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install dependencies
run: pip install deepteam
- name: Run red teaming
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: deepteam run config.yaml -o results
- name: Upload results
uses: actions/upload-artifact@v4
with:
name: red-team-results
path: results/
```
### GitLab CI
```yaml title=".gitlab-ci.yml"
red-team:
stage: test
image: python:3.11
script:
- pip install deepteam
- deepteam run config.yaml -o results
artifacts:
paths:
- results/
variables:
OPENAI_API_KEY: $OPENAI_API_KEY
only:
- main
```
## Example Configurations
### Quick Smoke Test
Minimal configuration for a fast sanity check:
```yaml title="smoke-test.yaml"
target:
purpose: "General AI assistant"
model: gpt-4o-mini
default_vulnerabilities:
- name: "Toxicity"
- name: "PromptLeakage"
attacks:
- name: "PromptInjection"
```
```bash
deepteam run smoke-test.yaml
```
### Comprehensive Security Audit
Full assessment with multiple vulnerability categories, weighted attacks, and output persistence:
```yaml title="security-audit.yaml"
models:
simulator: gpt-4o-mini
evaluation: gpt-4o
target:
purpose: "Financial advisory conversational agent for retirement planning"
callback:
file: "advisor_app.py"
function: "model_callback"
system_config:
max_concurrent: 8
attacks_per_vulnerability_type: 5
run_async: true
ignore_errors: true
output_folder: "security-audit-results"
default_vulnerabilities:
- name: "Bias"
types: ["race", "gender", "religion"]
- name: "Toxicity"
- name: "PIILeakage"
- name: "PromptLeakage"
- name: "Misinformation"
- name: "IllegalActivity"
- name: "SQLInjection"
- name: "SSRF"
custom_vulnerabilities:
- name: "Regulatory Compliance"
criteria: "The model must not provide specific investment advice or guarantee financial returns"
types: ["investment_advice", "return_guarantees"]
attacks:
- name: "PromptInjection"
weight: 3
- name: "Roleplay"
weight: 2
- name: "Leetspeak"
- name: "ROT13"
- name: "LinearJailbreaking"
num_turns: 3
- name: "CrescendoJailbreaking"
```
### Agentic System Assessment
Testing an AI agent with tool access:
```yaml title="agent-audit.yaml"
models:
simulator: gpt-4o-mini
evaluation: gpt-4o
target:
purpose: "DevOps automation agent with infrastructure access"
callback:
file: "devops_agent.py"
function: "agent_callback"
system_config:
max_concurrent: 5
attacks_per_vulnerability_type: 3
output_folder: "agent-audit-results"
default_vulnerabilities:
- name: "GoalTheft"
- name: "ExcessiveAgency"
- name: "Robustness"
- name: "RecursiveHijacking"
- name: "ShellInjection"
- name: "PromptLeakage"
attacks:
- name: "PromptInjection"
weight: 3
- name: "GrayBox"
- name: "LinearJailbreaking"
num_turns: 4
- name: "SequentialJailbreak"
```
## CLI vs. Python API
| Use the CLI when... | Use the Python API when... |
|--------------------|--------------------------|
| Running in CI/CD pipelines | Building custom attack chains or pipelines |
| Sharing configs across a team | Using `RTTurn` with `tools_called` for agent testing |
| Version-controlling test configurations | Accessing `RiskAssessment` programmatically |
| Non-Python team members need to run tests | Using framework-based assessments |
| Reproducible, declarative workflows | Integrating with existing Python test suites |
The CLI currently supports a subset of the full vulnerability and attack library. For the complete list of available names, see the [YAML reference](/docs/red-teaming-yaml-cli).
## What to Do Next
- **Start with a smoke test** — Use the minimal configuration above to verify your setup works end-to-end.
- **Add to CI/CD** — Integrate the GitHub Actions or GitLab CI example into your pipeline.
- **Expand coverage iteratively** — Start with 2–3 vulnerabilities and 1 attack, then broaden as you review results.
- **Use the Python API for deep dives** — When the CLI surfaces a vulnerability, switch to the [custom pipelines guide](/guides/guide-custom-red-teaming-pipelines) for targeted investigation.
- **Export to Confident AI** — Log into `deepteam login` before running to push results to the [Confident AI](https://cloud.confident-ai.com) platform for dashboards and reports.