Fully autonomous AI Agents system capable of performing complex penetration testing tasks
Go ai-agentsai-security-toolanthropicautonomous-agentsgolanggptgraphqlmulti-agent-systemoffensive-securityopen-sourceopenaipenetration-testingpenetration-testing-toolsreactsecurity-automationsecurity-testingsecurity-toolsself-hosted
feat(providers): add xAI provider support and configuration details
Dmitry Ng2da929d38d27
diff --git a/.vscode/launch.json b/.vscode/launch.jsonindex 10253c9d..dfa9ebed 100644--- a/.vscode/launch.json+++ b/.vscode/launch.json@@ -82,6 +82,7 @@// "-config", "${workspaceFolder}/examples/configs/openrouter.provider.yml",// "-config", "${workspaceFolder}/examples/configs/deepinfra.provider.yml",// "-config", "${workspaceFolder}/examples/configs/novita.provider.yml",+ // "-config", "${workspaceFolder}/examples/configs/xai.provider.yml",// "-report", "${workspaceFolder}/examples/tests/openai-report.md",// "-report", "${workspaceFolder}/examples/tests/anthropic-report.md",// "-report", "${workspaceFolder}/examples/tests/gemini-report.md",@@ -106,6 +107,7 @@// "-report", "${workspaceFolder}/examples/tests/openrouter-report.md",// "-report", "${workspaceFolder}/examples/tests/deepinfra-report.md",// "-report", "${workspaceFolder}/examples/tests/novita-report.md",+ // "-report", "${workspaceFolder}/examples/tests/xai-report.md","-agents", "all","-groups", "all","-workers", "8",diff --git a/README.md b/README.mdindex 32bbf3b8..83775430 100644--- a/README.md+++ b/README.md@@ -3236,7 +3236,7 @@ docker run --rm \#### Using Pre-configured Providers-The Docker image comes with built-in support for major providers (OpenAI, Anthropic, Gemini, Ollama) and pre-configured provider files for additional services (OpenRouter, OrcaRouter, DeepInfra, DeepSeek, Moonshot, Novita):+The Docker image comes with built-in support for major providers (OpenAI, Anthropic, Gemini, Ollama) and pre-configured provider files for additional services (OpenRouter, OrcaRouter, DeepInfra, DeepSeek, Moonshot, Novita, xAI):```bash# Test with OpenRouter configuration@@ -3269,6 +3269,9 @@ docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/moons# Test with Novita configurationdocker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/novita.provider.yml+# Test with xAI configuration+docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/xai.provider.yml+# Test with OpenAI configurationdocker exec -it pentagi /opt/pentagi/bin/ctester -type openai@@ -3297,11 +3300,11 @@ docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/ollamTo use these configurations, your `.env` file only needs to contain:```-LLM_SERVER_URL=https://openrouter.ai/api/v1 # or https://api.orcarouter.ai/v1 or https://api.deepinfra.com/v1/openai or https://api.openai.com/v1 or https://api.novita.ai/openai or https://api.atlascloud.ai/v1+LLM_SERVER_URL=https://openrouter.ai/api/v1 # or https://api.deepinfra.com/v1/openai or https://api.openai.com/v1 or https://api.novita.ai/openai or https://api.atlascloud.ai/v1 or https://api.orcarouter.ai/v1 or https://api.x.ai/v1LLM_SERVER_KEY=your_api_keyLLM_SERVER_MODEL= # Leave empty, as models are specified in the config-LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/openrouter.provider.yml # or orcarouter.provider.yml or deepinfra.provider.ymll or custom-openai.provider.yml or novita.provider.yml or atlas.provider.yml-LLM_SERVER_PROVIDER= # Provider name for LiteLLM proxy (e.g., openrouter, orcarouter, deepseek, moonshot, novita)+LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/openrouter.provider.yml # or deepinfra.provider.ymll or custom-openai.provider.yml or novita.provider.yml or atlas.provider.yml or orcarouter.provider.yml or xai.provider.yml+LLM_SERVER_PROVIDER= # Provider name for LiteLLM proxy (e.g., openrouter, deepseek, moonshot, novita, orcarouter, xai)LLM_SERVER_LEGACY_REASONING=false # Controls reasoning format, for OpenAI must be true (default: false)LLM_SERVER_PRESERVE_REASONING=false # Preserve reasoning content in multi-turn conversations (required by Moonshot, default: false)@@ -3424,6 +3427,7 @@ When using LiteLLM proxy, set the corresponding `*_PROVIDER` variable to enable- `orcarouter` - for OrcaRouter aggregator- `deepinfra` - for DeepInfra hosting- `novita` - for Novita AI+- `xai` - for xAI (`grok-*` models)- Any other provider name configured in your LiteLLM instance**Example with LiteLLM:**diff --git a/examples/configs/xai.provider.yml b/examples/configs/xai.provider.ymlnew file mode 100644index 00000000..06e0ec33--- /dev/null+++ b/examples/configs/xai.provider.yml@@ -0,0 +1,143 @@+simple:+ model: grok-4.20-0309-non-reasoning+ temperature: 0.3+ top_p: 0.9+ n: 1+ max_tokens: 8192+ price:+ input: 1.25+ output: 2.50+ cache_read: 0.20++simple_json:+ model: grok-4.20-0309-non-reasoning+ temperature: 0.3+ top_p: 0.9+ n: 1+ max_tokens: 4096+ json: true+ price:+ input: 1.25+ output: 2.50+ cache_read: 0.20++primary_agent:+ model: grok-4.5+ n: 1+ max_tokens: 16384+ reasoning:+ effort: high+ price:+ input: 2.00+ output: 6.00+ cache_read: 0.30++assistant:+ model: grok-4.5+ n: 1+ max_tokens: 16384+ reasoning:+ effort: high+ price:+ input: 2.00+ output: 6.00+ cache_read: 0.30++generator:+ model: grok-4.5+ n: 1+ max_tokens: 32768+ reasoning:+ effort: high+ price:+ input: 2.00+ output: 6.00+ cache_read: 0.30++refiner:+ model: grok-4.5+ n: 1+ max_tokens: 32768+ reasoning:+ effort: high+ price:+ input: 2.00+ output: 6.00+ cache_read: 0.30++adviser:+ model: grok-4.5+ n: 1+ max_tokens: 16384+ reasoning:+ effort: high+ price:+ input: 2.00+ output: 6.00+ cache_read: 0.30++reflector:+ model: grok-4.20-0309-non-reasoning+ temperature: 0.5+ top_p: 0.9+ n: 1+ max_tokens: 8192+ price:+ input: 1.25+ output: 2.50+ cache_read: 0.20++searcher:+ model: grok-4.20-0309-non-reasoning+ temperature: 0.5+ top_p: 0.9+ n: 1+ max_tokens: 4096+ price:+ input: 1.25+ output: 2.50+ cache_read: 0.20++enricher:+ model: grok-4.20-0309-non-reasoning+ temperature: 0.5+ top_p: 0.9+ n: 1+ max_tokens: 4096+ price:+ input: 1.25+ output: 2.50+ cache_read: 0.20++coder:+ model: grok-build-0.1+ n: 1+ max_tokens: 20480+ reasoning:+ effort: high+ price:+ input: 1.00+ output: 2.00+ cache_read: 0.20++installer:+ model: grok-4.20-0309-reasoning+ n: 1+ max_tokens: 12288+ reasoning:+ effort: high+ price:+ input: 1.25+ output: 2.50+ cache_read: 0.20++pentester:+ model: grok-4.5+ n: 1+ max_tokens: 16384+ reasoning:+ effort: high+ price:+ input: 2.00+ output: 6.00+ cache_read: 0.30diff --git a/examples/tests/xai-report.md b/examples/tests/xai-report.mdnew file mode 100644index 00000000..b6a122bb--- /dev/null+++ b/examples/tests/xai-report.md@@ -0,0 +1,556 @@+# LLM Agent Testing Report++Generated: Mon, 03 Aug 2026 21:25:49 UTC++## Overall Results++| Agent | Model | Reasoning | Success Rate | Average Latency |+|-------|-------|-----------|--------------|-----------------|+| simple | grok-4.20-0309-non-reasoning | false | 24/24 (100.00%) | 0.807s |+| simple_json | grok-4.20-0309-non-reasoning | false | 7/7 (100.00%) | 0.637s |+| primary_agent | grok-4.5 | true | 24/24 (100.00%) | 1.876s |+| assistant | grok-4.5 | true | 24/24 (100.00%) | 1.934s |+| generator | grok-4.5 | true | 24/24 (100.00%) | 2.239s |+| refiner | grok-4.5 | true | 24/24 (100.00%) | 2.133s |+| adviser | grok-4.5 | true | 24/24 (100.00%) | 1.788s |+| reflector | grok-4.20-0309-non-reasoning | true | 24/24 (100.00%) | 0.726s |+| searcher | grok-4.20-0309-non-reasoning | true | 24/24 (100.00%) | 0.771s |+| enricher | grok-4.20-0309-non-reasoning | true | 24/24 (100.00%) | 0.721s |+| coder | grok-build-0.1 | true | 24/24 (100.00%) | 2.738s |+| installer | grok-4.20-0309-reasoning | true | 23/24 (95.83%) | 2.326s |+| pentester | grok-4.5 | true | 24/24 (100.00%) | 1.867s |++**Total**: 294/295 (99.66%) successful tests+**Overall average latency**: 1.636s++## Detailed Results++### simple (grok-4.20-0309-non-reasoning)++#### Basic Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| Simple Math | ✅ Pass | 1.221s | |+| Text Transform Uppercase | ✅ Pass | 0.484s | |+| Count from 1 to 5 | ✅ Pass | 0.525s | |+| Math Calculation | ✅ Pass | 0.514s | |+| Basic Echo Function | ✅ Pass | 0.723s | |+| Streaming Simple Math Streaming | ✅ Pass | 0.506s | |+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.607s | |+| Streaming Basic Echo Function Streaming | ✅ Pass | 0.767s | |++#### Advanced Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| JSON Response Function | ✅ Pass | 1.004s | |+| Search Query Function | ✅ Pass | 0.670s | |+| Ask Advice Function | ✅ Pass | 0.688s | |+| Streaming Search Query Function Streaming | ✅ Pass | 0.679s | |+| Basic Context Memory Test | ✅ Pass | 0.639s | |+| Function Argument Memory Test | ✅ Pass | 0.521s | |+| Function Response Memory Test | ✅ Pass | 0.490s | |+| Penetration Testing Memory with Tool Call | ✅ Pass | 0.802s | |+| Cybersecurity Workflow Memory Test | ✅ Pass | 0.497s | |+| Read a file, then edit it via unified diff | ✅ Pass | 1.495s | |+| Penetration Testing Methodology | ✅ Pass | 1.504s | |+| Vulnerability Assessment Tools | ✅ Pass | 1.706s | |+| SQL Injection Attack Type | ✅ Pass | 0.646s | |+| Penetration Testing Framework | ✅ Pass | 1.137s | |+| Web Application Security Scanner | ✅ Pass | 0.809s | |+| Penetration Testing Tool Selection | ✅ Pass | 0.732s | |++**Summary**: 24/24 (100.00%) successful tests++**Average latency**: 0.807s++---++### simple_json (grok-4.20-0309-non-reasoning)++#### Advanced Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| Vulnerability Report Memory Test | ✅ Pass | 0.633s | |+| Person Information JSON | ✅ Pass | 0.624s | |+| Project Information JSON | ✅ Pass | 0.538s | |+| User Profile JSON | ✅ Pass | 0.609s | |+| JSON Array Response Without Schema | ✅ Pass | 0.622s | |+| Streaming Person Information JSON Streaming | ✅ Pass | 0.700s | |++#### Capability Tests++| Test | Capability | Result | Latency | Note |+|------|------------|--------|---------|------|+| Structured Output With JSON Schema | structured_output | ✅ Pass | 0.732s | |++**Summary**: 7/7 (100.00%) successful tests++**Average latency**: 0.637s++---++### primary_agent (grok-4.5)++#### Basic Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| Simple Math | ✅ Pass | 1.845s | |+| Text Transform Uppercase | ✅ Pass | 1.289s | |+| Count from 1 to 5 | ✅ Pass | 1.296s | |+| Math Calculation | ✅ Pass | 1.127s | |+| Basic Echo Function | ✅ Pass | 0.930s | |+| Streaming Simple Math Streaming | ✅ Pass | 1.185s | |+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.273s | |+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.144s | |++#### Advanced Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| JSON Response Function | ✅ Pass | 1.484s | |+| Search Query Function | ✅ Pass | 1.134s | |+| Ask Advice Function | ✅ Pass | 1.496s | |+| Streaming Search Query Function Streaming | ✅ Pass | 1.137s | |+| Basic Context Memory Test | ✅ Pass | 2.021s | |+| Function Argument Memory Test | ✅ Pass | 1.550s | |+| Function Response Memory Test | ✅ Pass | 1.563s | |+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.265s | |+| Cybersecurity Workflow Memory Test | ✅ Pass | 1.709s | |+| Read a file, then edit it via unified diff | ✅ Pass | 3.104s | |+| Penetration Testing Methodology | ✅ Pass | 3.804s | |+| Vulnerability Assessment Tools | ✅ Pass | 4.579s | |+| SQL Injection Attack Type | ✅ Pass | 1.892s | |+| Penetration Testing Framework | ✅ Pass | 3.664s | |+| Web Application Security Scanner | ✅ Pass | 2.335s | |+| Penetration Testing Tool Selection | ✅ Pass | 1.181s | |++**Summary**: 24/24 (100.00%) successful tests++**Average latency**: 1.876s++---++### assistant (grok-4.5)++#### Basic Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| Simple Math | ✅ Pass | 1.168s | |+| Text Transform Uppercase | ✅ Pass | 1.116s | |+| Count from 1 to 5 | ✅ Pass | 1.350s | |+| Math Calculation | ✅ Pass | 1.217s | |+| Basic Echo Function | ✅ Pass | 1.782s | |+| Streaming Simple Math Streaming | ✅ Pass | 1.366s | |+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.477s | |+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.087s | |++#### Advanced Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| JSON Response Function | ✅ Pass | 1.519s | |+| Search Query Function | ✅ Pass | 0.971s | |+| Ask Advice Function | ✅ Pass | 1.364s | |+| Streaming Search Query Function Streaming | ✅ Pass | 1.039s | |+| Basic Context Memory Test | ✅ Pass | 1.753s | |+| Function Argument Memory Test | ✅ Pass | 1.434s | |+| Function Response Memory Test | ✅ Pass | 0.848s | |+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.493s | |+| Cybersecurity Workflow Memory Test | ✅ Pass | 1.561s | |+| Read a file, then edit it via unified diff | ✅ Pass | 2.970s | |+| Penetration Testing Methodology | ✅ Pass | 3.787s | |+| Vulnerability Assessment Tools | ✅ Pass | 6.609s | |+| SQL Injection Attack Type | ✅ Pass | 1.340s | |+| Penetration Testing Framework | ✅ Pass | 3.854s | |+| Web Application Security Scanner | ✅ Pass | 3.004s | |+| Penetration Testing Tool Selection | ✅ Pass | 1.286s | |++**Summary**: 24/24 (100.00%) successful tests++**Average latency**: 1.934s++---++### generator (grok-4.5)++#### Basic Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| Simple Math | ✅ Pass | 1.356s | |+| Text Transform Uppercase | ✅ Pass | 1.209s | |+| Count from 1 to 5 | ✅ Pass | 1.193s | |+| Math Calculation | ✅ Pass | 1.054s | |+| Basic Echo Function | ✅ Pass | 1.548s | |+| Streaming Simple Math Streaming | ✅ Pass | 1.583s | |+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.422s | |+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.043s | |++#### Advanced Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| JSON Response Function | ✅ Pass | 2.088s | |+| Search Query Function | ✅ Pass | 0.986s | |+| Ask Advice Function | ✅ Pass | 1.701s | |+| Streaming Search Query Function Streaming | ✅ Pass | 1.023s | |+| Basic Context Memory Test | ✅ Pass | 5.516s | |+| Function Argument Memory Test | ✅ Pass | 1.519s | |+| Function Response Memory Test | ✅ Pass | 1.451s | |+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.068s | |+| Cybersecurity Workflow Memory Test | ✅ Pass | 1.650s | |+| Read a file, then edit it via unified diff | ✅ Pass | 3.279s | |+| Penetration Testing Methodology | ✅ Pass | 7.178s | |+| Vulnerability Assessment Tools | ✅ Pass | 5.469s | |+| SQL Injection Attack Type | ✅ Pass | 1.346s | |+| Penetration Testing Framework | ✅ Pass | 3.533s | |+| Web Application Security Scanner | ✅ Pass | 3.128s | |+| Penetration Testing Tool Selection | ✅ Pass | 1.376s | |++**Summary**: 24/24 (100.00%) successful tests++**Average latency**: 2.239s++---++### refiner (grok-4.5)++#### Basic Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| Simple Math | ✅ Pass | 1.262s | |+| Text Transform Uppercase | ✅ Pass | 1.209s | |+| Count from 1 to 5 | ✅ Pass | 1.154s | |+| Math Calculation | ✅ Pass | 1.090s | |+| Basic Echo Function | ✅ Pass | 1.655s | |+| Streaming Simple Math Streaming | ✅ Pass | 1.109s | |+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.352s | |+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.094s | |++#### Advanced Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| JSON Response Function | ✅ Pass | 1.427s | |+| Search Query Function | ✅ Pass | 1.081s | |+| Ask Advice Function | ✅ Pass | 1.280s | |+| Streaming Search Query Function Streaming | ✅ Pass | 1.093s | |+| Basic Context Memory Test | ✅ Pass | 7.274s | |+| Function Argument Memory Test | ✅ Pass | 1.311s | |+| Function Response Memory Test | ✅ Pass | 0.883s | |+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.285s | |+| Cybersecurity Workflow Memory Test | ✅ Pass | 1.773s | |+| Read a file, then edit it via unified diff | ✅ Pass | 3.301s | |+| Penetration Testing Methodology | ✅ Pass | 4.107s | |+| Vulnerability Assessment Tools | ✅ Pass | 4.685s | |+| SQL Injection Attack Type | ✅ Pass | 1.599s | |+| Penetration Testing Framework | ✅ Pass | 5.428s | |+| Web Application Security Scanner | ✅ Pass | 2.458s | |+| Penetration Testing Tool Selection | ✅ Pass | 1.257s | |++**Summary**: 24/24 (100.00%) successful tests++**Average latency**: 2.133s++---++### adviser (grok-4.5)++#### Basic Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| Simple Math | ✅ Pass | 1.145s | |+| Text Transform Uppercase | ✅ Pass | 1.152s | |+| Count from 1 to 5 | ✅ Pass | 1.330s | |+| Math Calculation | ✅ Pass | 1.096s | |+| Basic Echo Function | ✅ Pass | 1.467s | |+| Streaming Simple Math Streaming | ✅ Pass | 1.185s | |+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.319s | |+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.115s | |++#### Advanced Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| JSON Response Function | ✅ Pass | 1.273s | |+| Search Query Function | ✅ Pass | 0.986s | |+| Ask Advice Function | ✅ Pass | 1.185s | |+| Streaming Search Query Function Streaming | ✅ Pass | 1.192s | |+| Basic Context Memory Test | ✅ Pass | 1.718s | |+| Function Argument Memory Test | ✅ Pass | 1.361s | |+| Function Response Memory Test | ✅ Pass | 0.872s | |+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.029s | |+| Cybersecurity Workflow Memory Test | ✅ Pass | 1.603s | |+| Read a file, then edit it via unified diff | ✅ Pass | 3.113s | |+| Penetration Testing Methodology | ✅ Pass | 4.391s | |+| Vulnerability Assessment Tools | ✅ Pass | 3.740s | |+| SQL Injection Attack Type | ✅ Pass | 1.516s | |+| Penetration Testing Framework | ✅ Pass | 4.216s | |+| Web Application Security Scanner | ✅ Pass | 2.738s | |+| Penetration Testing Tool Selection | ✅ Pass | 1.166s | |++**Summary**: 24/24 (100.00%) successful tests++**Average latency**: 1.788s++---++### reflector (grok-4.20-0309-non-reasoning)++#### Basic Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| Simple Math | ✅ Pass | 0.496s | |+| Text Transform Uppercase | ✅ Pass | 0.556s | |+| Count from 1 to 5 | ✅ Pass | 0.711s | |+| Math Calculation | ✅ Pass | 0.540s | |+| Basic Echo Function | ✅ Pass | 0.501s | |+| Streaming Simple Math Streaming | ✅ Pass | 0.578s | |+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.602s | |+| Streaming Basic Echo Function Streaming | ✅ Pass | 0.580s | |++#### Advanced Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| JSON Response Function | ✅ Pass | 0.859s | |+| Search Query Function | ✅ Pass | 0.447s | |+| Ask Advice Function | ✅ Pass | 0.544s | |+| Streaming Search Query Function Streaming | ✅ Pass | 0.693s | |+| Basic Context Memory Test | ✅ Pass | 0.527s | |+| Function Argument Memory Test | ✅ Pass | 0.497s | |+| Function Response Memory Test | ✅ Pass | 0.586s | |+| Penetration Testing Memory with Tool Call | ✅ Pass | 0.628s | |+| Cybersecurity Workflow Memory Test | ✅ Pass | 0.470s | |+| Read a file, then edit it via unified diff | ✅ Pass | 1.326s | |+| Penetration Testing Methodology | ✅ Pass | 1.583s | |+| Vulnerability Assessment Tools | ✅ Pass | 1.584s | |+| SQL Injection Attack Type | ✅ Pass | 0.635s | |+| Penetration Testing Framework | ✅ Pass | 1.050s | |+| Web Application Security Scanner | ✅ Pass | 0.822s | |+| Penetration Testing Tool Selection | ✅ Pass | 0.600s | |++**Summary**: 24/24 (100.00%) successful tests++**Average latency**: 0.726s++---++### searcher (grok-4.20-0309-non-reasoning)++#### Basic Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| Simple Math | ✅ Pass | 0.494s | |+| Text Transform Uppercase | ✅ Pass | 0.539s | |+| Count from 1 to 5 | ✅ Pass | 0.475s | |+| Math Calculation | ✅ Pass | 0.482s | |+| Basic Echo Function | ✅ Pass | 0.788s | |+| Streaming Simple Math Streaming | ✅ Pass | 0.539s | |+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.595s | |+| Streaming Basic Echo Function Streaming | ✅ Pass | 0.619s | |++#### Advanced Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| JSON Response Function | ✅ Pass | 1.240s | |+| Search Query Function | ✅ Pass | 0.518s | |+| Ask Advice Function | ✅ Pass | 0.631s | |+| Streaming Search Query Function Streaming | ✅ Pass | 0.648s | |+| Basic Context Memory Test | ✅ Pass | 0.542s | |+| Function Argument Memory Test | ✅ Pass | 0.501s | |+| Function Response Memory Test | ✅ Pass | 0.574s | |+| Penetration Testing Memory with Tool Call | ✅ Pass | 0.638s | |+| Cybersecurity Workflow Memory Test | ✅ Pass | 0.532s | |+| Read a file, then edit it via unified diff | ✅ Pass | 1.399s | |+| Penetration Testing Methodology | ✅ Pass | 1.575s | |+| Vulnerability Assessment Tools | ✅ Pass | 1.853s | |+| SQL Injection Attack Type | ✅ Pass | 0.649s | |+| Penetration Testing Framework | ✅ Pass | 1.129s | |+| Web Application Security Scanner | ✅ Pass | 0.908s | |+| Penetration Testing Tool Selection | ✅ Pass | 0.624s | |++**Summary**: 24/24 (100.00%) successful tests++**Average latency**: 0.771s++---++### enricher (grok-4.20-0309-non-reasoning)++#### Basic Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| Simple Math | ✅ Pass | 0.483s | |+| Text Transform Uppercase | ✅ Pass | 0.452s | |+| Count from 1 to 5 | ✅ Pass | 0.519s | |+| Math Calculation | ✅ Pass | 0.496s | |+| Basic Echo Function | ✅ Pass | 0.795s | |+| Streaming Simple Math Streaming | ✅ Pass | 0.617s | |+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.580s | |+| Streaming Basic Echo Function Streaming | ✅ Pass | 0.689s | |++#### Advanced Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| JSON Response Function | ✅ Pass | 0.549s | |+| Search Query Function | ✅ Pass | 0.527s | |+| Ask Advice Function | ✅ Pass | 0.573s | |+| Streaming Search Query Function Streaming | ✅ Pass | 0.610s | |+| Basic Context Memory Test | ✅ Pass | 0.566s | |+| Function Argument Memory Test | ✅ Pass | 0.478s | |+| Function Response Memory Test | ✅ Pass | 0.515s | |+| Penetration Testing Memory with Tool Call | ✅ Pass | 0.593s | |+| Cybersecurity Workflow Memory Test | ✅ Pass | 0.458s | |+| Read a file, then edit it via unified diff | ✅ Pass | 1.230s | |+| Penetration Testing Methodology | ✅ Pass | 1.456s | |+| Vulnerability Assessment Tools | ✅ Pass | 1.870s | |+| SQL Injection Attack Type | ✅ Pass | 0.638s | |+| Penetration Testing Framework | ✅ Pass | 1.157s | |+| Web Application Security Scanner | ✅ Pass | 0.872s | |+| Penetration Testing Tool Selection | ✅ Pass | 0.573s | |++**Summary**: 24/24 (100.00%) successful tests++**Average latency**: 0.721s++---++### coder (grok-build-0.1)++#### Basic Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| Simple Math | ✅ Pass | 1.649s | |+| Text Transform Uppercase | ✅ Pass | 1.563s | |+| Count from 1 to 5 | ✅ Pass | 2.030s | |+| Math Calculation | ✅ Pass | 1.293s | |+| Basic Echo Function | ✅ Pass | 2.304s | |+| Streaming Simple Math Streaming | ✅ Pass | 1.025s | |+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.856s | |+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.875s | |++#### Advanced Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| JSON Response Function | ✅ Pass | 1.841s | |+| Search Query Function | ✅ Pass | 1.673s | |+| Ask Advice Function | ✅ Pass | 1.683s | |+| Streaming Search Query Function Streaming | ✅ Pass | 2.231s | |+| Basic Context Memory Test | ✅ Pass | 1.904s | |+| Function Argument Memory Test | ✅ Pass | 2.705s | |+| Function Response Memory Test | ✅ Pass | 2.171s | |+| Penetration Testing Memory with Tool Call | ✅ Pass | 3.996s | |+| Cybersecurity Workflow Memory Test | ✅ Pass | 6.073s | |+| Read a file, then edit it via unified diff | ✅ Pass | 3.945s | |+| Penetration Testing Methodology | ✅ Pass | 3.424s | |+| Vulnerability Assessment Tools | ✅ Pass | 7.914s | |+| SQL Injection Attack Type | ✅ Pass | 2.726s | |+| Penetration Testing Framework | ✅ Pass | 3.132s | |+| Web Application Security Scanner | ✅ Pass | 4.601s | |+| Penetration Testing Tool Selection | ✅ Pass | 2.079s | |++**Summary**: 24/24 (100.00%) successful tests++**Average latency**: 2.738s++---++### installer (grok-4.20-0309-reasoning)++#### Basic Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| Simple Math | ✅ Pass | 1.064s | |+| Text Transform Uppercase | ✅ Pass | 1.107s | |+| Count from 1 to 5 | ✅ Pass | 1.737s | |+| Math Calculation | ✅ Pass | 1.186s | |+| Basic Echo Function | ✅ Pass | 2.403s | |+| Streaming Simple Math Streaming | ✅ Pass | 3.255s | |+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.509s | |+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.742s | |++#### Advanced Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| JSON Response Function | ✅ Pass | 1.758s | |+| Search Query Function | ✅ Pass | 1.338s | |+| Ask Advice Function | ✅ Pass | 1.663s | |+| Streaming Search Query Function Streaming | ✅ Pass | 1.077s | |+| Basic Context Memory Test | ✅ Pass | 1.257s | |+| Function Argument Memory Test | ✅ Pass | 1.405s | |+| Function Response Memory Test | ✅ Pass | 1.645s | |+| Penetration Testing Memory with Tool Call | ✅ Pass | 1.746s | |+| Cybersecurity Workflow Memory Test | ✅ Pass | 2.975s | |+| Read a file, then edit it via unified diff | ❌ Fail | 3.645s | edit\_file's diff did not apply: invalid diff line \(must start with ' ', '\-', or '+'\): "Status: draft" |+| Penetration Testing Methodology | ✅ Pass | 2.628s | |+| Vulnerability Assessment Tools | ✅ Pass | 9.348s | |+| SQL Injection Attack Type | ✅ Pass | 2.879s | |+| Penetration Testing Framework | ✅ Pass | 3.056s | |+| Web Application Security Scanner | ✅ Pass | 3.682s | |+| Penetration Testing Tool Selection | ✅ Pass | 1.696s | |++**Summary**: 23/24 (95.83%) successful tests++**Average latency**: 2.326s++---++### pentester (grok-4.5)++#### Basic Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| Simple Math | ✅ Pass | 1.260s | |+| Text Transform Uppercase | ✅ Pass | 1.169s | |+| Count from 1 to 5 | ✅ Pass | 1.384s | |+| Math Calculation | ✅ Pass | 0.933s | |+| Basic Echo Function | ✅ Pass | 1.732s | |+| Streaming Simple Math Streaming | ✅ Pass | 1.352s | |+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.393s | |+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.114s | |++#### Advanced Tests++| Test | Result | Latency | Error |+|------|--------|---------|-------|+| JSON Response Function | ✅ Pass | 1.519s | |+| Search Query Function | ✅ Pass | 1.005s | |+| Ask Advice Function | ✅ Pass | 1.883s | |+| Streaming Search Query Function Streaming | ✅ Pass | 1.194s | |+| Basic Context Memory Test | ✅ Pass | 1.896s | |+| Function Argument Memory Test | ✅ Pass | 1.463s | |+| Function Response Memory Test | ✅ Pass | 0.841s | |+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.286s | |+| Cybersecurity Workflow Memory Test | ✅ Pass | 1.429s | |+| Read a file, then edit it via unified diff | ✅ Pass | 3.019s | |+| Penetration Testing Methodology | ✅ Pass | 3.691s | |+| Vulnerability Assessment Tools | ✅ Pass | 4.333s | |+| SQL Injection Attack Type | ✅ Pass | 1.306s | |+| Penetration Testing Framework | ✅ Pass | 5.069s | |+| Web Application Security Scanner | ✅ Pass | 2.253s | |+| Penetration Testing Tool Selection | ✅ Pass | 1.271s | |++**Summary**: 24/24 (100.00%) successful tests++**Average latency**: 1.867s++---+