Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

20.9 KB
Raw Blame

LLM Agent Testing Report

Generated: Sat, 26 Sep 2026 12:51:40 UTC

Overall Results

Agent Model Reasoning Success Rate Average Latency
simple grok-4.20-0309-non-reasoning false 25/25 (100.00%) 1.026s
simple_json grok-4.20-0309-non-reasoning false 8/8 (100.00%) 0.880s
primary_agent grok-4.3 true 25/25 (100.00%) 2.002s
assistant grok-4.3 true 25/25 (100.00%) 1.499s
generator grok-4.3 true 25/25 (100.00%) 1.944s
refiner grok-4.3 true 25/25 (100.00%) 1.443s
adviser grok-4.3 true 16/16 (100.00%) 0.572s
reflector grok-4.20-0309-non-reasoning false 16/16 (100.00%) 0.304s
searcher grok-4.3 true 25/25 (100.00%) 1.872s
enricher grok-4.20-0309-non-reasoning false 25/25 (100.00%) 0.369s
coder grok-4.3 true 25/25 (100.00%) 1.813s
installer grok-4.3 true 25/25 (100.00%) 0.611s
pentester grok-4.3 true 25/25 (100.00%) 0.556s

Total: 290/290 (100.00%) successful tests Overall average latency: 1.205s

Detailed Results

simple (grok-4.20-0309-non-reasoning)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 0.210s
Streaming Simple Math Streaming ✅ Pass 0.255s
Answer Stops At The Output Limit ✅ Pass 0.256s
Count from 1 to 5 ✅ Pass 0.256s
Basic Echo Function ✅ Pass 0.280s
Math Calculation ✅ Pass 0.285s
Simple Math ✅ Pass 0.290s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.299s
Streaming Basic Echo Function Streaming ✅ Pass 0.194s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.978s
Basic Context Memory Test ✅ Pass 0.825s
Streaming Search Query Function Streaming ✅ Pass 0.938s
Ask Advice Function ✅ Pass 1.053s
Search Query Function ✅ Pass 1.156s
Function Response Memory Test ✅ Pass 0.949s
Function Argument Memory Test ✅ Pass 1.163s
Penetration Testing Memory with Tool Call ✅ Pass 1.396s
Cybersecurity Workflow Memory Test ✅ Pass 0.776s
Read a file, then edit it via unified diff ✅ Pass 2.652s
SQL Injection Attack Type ✅ Pass 1.142s
Penetration Testing Tool Selection ✅ Pass 1.039s
Penetration Testing Framework ✅ Pass 1.589s
Web Application Security Scanner ✅ Pass 1.382s
Vulnerability Assessment Tools ✅ Pass 2.706s
Penetration Testing Methodology ✅ Pass 3.580s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.026s


simple_json (grok-4.20-0309-non-reasoning)

Advanced Tests

Test Result Latency Error
Vulnerability Report Memory Test ✅ Pass 1.243s
Person Information JSON ✅ Pass 0.966s
Project Information JSON ✅ Pass 0.994s
User Profile JSON ✅ Pass 0.986s
Streaming Person Information JSON Streaming ✅ Pass 0.880s
JSON Array Response Without Schema ✅ Pass 0.992s

Capability Tests

Test Capability Result Latency Note
Structured Output Refuses A Non-Object Schema structured_output ✅ Pass 0.000s
Structured Output With JSON Schema structured_output ✅ Pass 0.979s

Summary: 8/8 (100.00%) successful tests

Average latency: 0.880s


primary_agent (grok-4.3)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 0.189s
Simple Math ✅ Pass 0.215s
Answer Stops At The Output Limit ✅ Pass 0.220s
Count from 1 to 5 ✅ Pass 0.230s
Math Calculation ✅ Pass 0.239s
Basic Echo Function ✅ Pass 0.236s
Streaming Simple Math Streaming ✅ Pass 0.231s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.190s
Streaming Basic Echo Function Streaming ✅ Pass 0.185s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 1.878s
Search Query Function ✅ Pass 2.178s
Function Argument Memory Test ✅ Pass 1.687s
JSON Response Function ✅ Pass 2.284s
Function Response Memory Test ✅ Pass 1.869s
Streaming Search Query Function Streaming ✅ Pass 2.531s
Ask Advice Function ✅ Pass 2.609s
Penetration Testing Memory with Tool Call ✅ Pass 3.107s
Cybersecurity Workflow Memory Test ✅ Pass 3.836s
Read a file, then edit it via unified diff ✅ Pass 3.265s
Penetration Testing Methodology ✅ Pass 3.085s
SQL Injection Attack Type ✅ Pass 2.519s
Penetration Testing Framework ✅ Pass 3.383s
Penetration Testing Tool Selection ✅ Pass 3.453s
Web Application Security Scanner ✅ Pass 4.170s
Vulnerability Assessment Tools ✅ Pass 6.239s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.002s


assistant (grok-4.3)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.191s
Simple Math ✅ Pass 0.198s
Text Transform Uppercase ✅ Pass 0.208s
Math Calculation ✅ Pass 0.212s
Count from 1 to 5 ✅ Pass 0.215s
Basic Echo Function ✅ Pass 0.213s
Streaming Simple Math Streaming ✅ Pass 0.194s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.188s
Streaming Basic Echo Function Streaming ✅ Pass 0.199s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.188s
Search Query Function ✅ Pass 0.187s
Basic Context Memory Test ✅ Pass 0.206s
Function Response Memory Test ✅ Pass 1.681s
Cybersecurity Workflow Memory Test ✅ Pass 2.026s
Ask Advice Function ✅ Pass 2.461s
Function Argument Memory Test ✅ Pass 2.333s
Penetration Testing Memory with Tool Call ✅ Pass 2.400s
Streaming Search Query Function Streaming ✅ Pass 2.938s
Read a file, then edit it via unified diff ✅ Pass 3.216s
SQL Injection Attack Type ✅ Pass 0.203s
Penetration Testing Tool Selection ✅ Pass 0.355s
Penetration Testing Methodology ✅ Pass 3.612s
Penetration Testing Framework ✅ Pass 3.494s
Web Application Security Scanner ✅ Pass 4.825s
Vulnerability Assessment Tools ✅ Pass 5.529s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.499s


generator (grok-4.3)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.187s
Simple Math ✅ Pass 0.190s
Text Transform Uppercase ✅ Pass 0.201s
Math Calculation ✅ Pass 0.209s
Count from 1 to 5 ✅ Pass 0.213s
Basic Echo Function ✅ Pass 0.193s
Streaming Simple Math Streaming ✅ Pass 0.186s
Streaming Basic Echo Function Streaming ✅ Pass 0.201s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.211s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 2.111s
JSON Response Function ✅ Pass 2.442s
Function Response Memory Test ✅ Pass 1.606s
Basic Context Memory Test ✅ Pass 2.315s
Ask Advice Function ✅ Pass 2.556s
Streaming Search Query Function Streaming ✅ Pass 2.554s
Function Argument Memory Test ✅ Pass 2.455s
Penetration Testing Memory with Tool Call ✅ Pass 2.040s
Cybersecurity Workflow Memory Test ✅ Pass 1.912s
Read a file, then edit it via unified diff ✅ Pass 3.606s
SQL Injection Attack Type ✅ Pass 2.894s
Penetration Testing Methodology ✅ Pass 4.001s
Penetration Testing Tool Selection ✅ Pass 2.505s
Penetration Testing Framework ✅ Pass 3.540s
Web Application Security Scanner ✅ Pass 3.499s
Vulnerability Assessment Tools ✅ Pass 6.752s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.944s


refiner (grok-4.3)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.189s
Simple Math ✅ Pass 0.189s
Count from 1 to 5 ✅ Pass 0.194s
Text Transform Uppercase ✅ Pass 0.206s
Math Calculation ✅ Pass 0.204s
Basic Echo Function ✅ Pass 0.203s
Streaming Simple Math Streaming ✅ Pass 0.194s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.200s
Streaming Basic Echo Function Streaming ✅ Pass 0.204s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.190s
Search Query Function ✅ Pass 0.191s
Basic Context Memory Test ✅ Pass 0.190s
Function Response Memory Test ✅ Pass 1.606s
Cybersecurity Workflow Memory Test ✅ Pass 1.857s
Ask Advice Function ✅ Pass 2.400s
Streaming Search Query Function Streaming ✅ Pass 2.295s
Function Argument Memory Test ✅ Pass 2.108s
Penetration Testing Memory with Tool Call ✅ Pass 2.130s
Read a file, then edit it via unified diff ✅ Pass 3.054s
SQL Injection Attack Type ✅ Pass 0.183s
Penetration Testing Tool Selection ✅ Pass 0.195s
Penetration Testing Methodology ✅ Pass 3.103s
Web Application Security Scanner ✅ Pass 3.237s
Penetration Testing Framework ✅ Pass 4.244s
Vulnerability Assessment Tools ✅ Pass 7.292s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.443s


adviser (grok-4.3)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.193s
Simple Math ✅ Pass 0.184s
Text Transform Uppercase ✅ Pass 0.186s
Count from 1 to 5 ✅ Pass 0.194s
Math Calculation ✅ Pass 0.188s
Streaming Simple Math Streaming ✅ Pass 0.193s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.191s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 0.201s
Function Argument Memory Test ✅ Pass 1.667s
Function Response Memory Test ✅ Pass 2.109s
Cybersecurity Workflow Memory Test ✅ Pass 2.841s
Penetration Testing Methodology ✅ Pass 0.191s
Vulnerability Assessment Tools ✅ Pass 0.199s
SQL Injection Attack Type ✅ Pass 0.201s
Penetration Testing Framework ✅ Pass 0.203s
Web Application Security Scanner ✅ Pass 0.198s

Summary: 16/16 (100.00%) successful tests

Average latency: 0.572s


reflector (grok-4.20-0309-non-reasoning)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.187s
Simple Math ✅ Pass 0.201s
Count from 1 to 5 ✅ Pass 0.197s
Text Transform Uppercase ✅ Pass 0.212s
Math Calculation ✅ Pass 0.183s
Streaming Simple Math Streaming ✅ Pass 0.186s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.212s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 0.199s
Function Response Memory Test ✅ Pass 0.747s
Cybersecurity Workflow Memory Test ✅ Pass 0.724s
Function Argument Memory Test ✅ Pass 0.804s
Penetration Testing Methodology ✅ Pass 0.232s
Vulnerability Assessment Tools ✅ Pass 0.193s
SQL Injection Attack Type ✅ Pass 0.190s
Penetration Testing Framework ✅ Pass 0.201s
Web Application Security Scanner ✅ Pass 0.194s

Summary: 16/16 (100.00%) successful tests

Average latency: 0.304s


searcher (grok-4.3)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.198s
Simple Math ✅ Pass 0.191s
Text Transform Uppercase ✅ Pass 0.196s
Count from 1 to 5 ✅ Pass 0.205s
Math Calculation ✅ Pass 0.205s
Basic Echo Function ✅ Pass 0.202s
Streaming Simple Math Streaming ✅ Pass 0.202s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.226s
Streaming Basic Echo Function Streaming ✅ Pass 0.228s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.866s
Search Query Function ✅ Pass 2.244s
Basic Context Memory Test ✅ Pass 1.903s
Ask Advice Function ✅ Pass 2.100s
Function Argument Memory Test ✅ Pass 1.916s
Function Response Memory Test ✅ Pass 1.735s
Streaming Search Query Function Streaming ✅ Pass 2.294s
Penetration Testing Memory with Tool Call ✅ Pass 2.143s
Cybersecurity Workflow Memory Test ✅ Pass 2.402s
Read a file, then edit it via unified diff ✅ Pass 3.096s
SQL Injection Attack Type ✅ Pass 2.659s
Penetration Testing Methodology ✅ Pass 3.652s
Web Application Security Scanner ✅ Pass 2.926s
Penetration Testing Framework ✅ Pass 4.071s
Penetration Testing Tool Selection ✅ Pass 3.559s
Vulnerability Assessment Tools ✅ Pass 6.360s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.872s


enricher (grok-4.20-0309-non-reasoning)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.200s
Simple Math ✅ Pass 0.202s
Text Transform Uppercase ✅ Pass 0.203s
Count from 1 to 5 ✅ Pass 0.205s
Math Calculation ✅ Pass 0.204s
Basic Echo Function ✅ Pass 0.198s
Streaming Simple Math Streaming ✅ Pass 0.197s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.229s
Streaming Basic Echo Function Streaming ✅ Pass 0.226s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.196s
Search Query Function ✅ Pass 0.202s
Ask Advice Function ✅ Pass 0.198s
Streaming Search Query Function Streaming ✅ Pass 0.184s
Basic Context Memory Test ✅ Pass 0.185s
Function Argument Memory Test ✅ Pass 0.855s
Function Response Memory Test ✅ Pass 0.844s
Cybersecurity Workflow Memory Test ✅ Pass 0.868s
Penetration Testing Memory with Tool Call ✅ Pass 1.147s
Read a file, then edit it via unified diff ✅ Pass 1.514s
Penetration Testing Methodology ✅ Pass 0.190s
Vulnerability Assessment Tools ✅ Pass 0.194s
SQL Injection Attack Type ✅ Pass 0.190s
Penetration Testing Framework ✅ Pass 0.194s
Web Application Security Scanner ✅ Pass 0.191s
Penetration Testing Tool Selection ✅ Pass 0.192s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.369s


coder (grok-4.3)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.196s
Simple Math ✅ Pass 0.205s
Text Transform Uppercase ✅ Pass 0.209s
Count from 1 to 5 ✅ Pass 0.217s
Math Calculation ✅ Pass 0.216s
Basic Echo Function ✅ Pass 0.198s
Streaming Simple Math Streaming ✅ Pass 0.190s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.210s
Streaming Basic Echo Function Streaming ✅ Pass 0.216s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 1.793s
JSON Response Function ✅ Pass 2.116s
Function Argument Memory Test ✅ Pass 1.684s
Function Response Memory Test ✅ Pass 1.623s
Ask Advice Function ✅ Pass 2.263s
Streaming Search Query Function Streaming ✅ Pass 2.337s
Basic Context Memory Test ✅ Pass 2.424s
Cybersecurity Workflow Memory Test ✅ Pass 1.983s
Penetration Testing Memory with Tool Call ✅ Pass 2.557s
Read a file, then edit it via unified diff ✅ Pass 3.626s
SQL Injection Attack Type ✅ Pass 2.796s
Penetration Testing Methodology ✅ Pass 3.477s
Penetration Testing Framework ✅ Pass 3.862s
Penetration Testing Tool Selection ✅ Pass 2.851s
Vulnerability Assessment Tools ✅ Pass 4.885s
Web Application Security Scanner ✅ Pass 3.183s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.813s


installer (grok-4.3)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.202s
Simple Math ✅ Pass 0.198s
Text Transform Uppercase ✅ Pass 0.194s
Count from 1 to 5 ✅ Pass 0.196s
Math Calculation ✅ Pass 0.200s
Basic Echo Function ✅ Pass 0.185s
Streaming Simple Math Streaming ✅ Pass 0.200s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.244s
Streaming Basic Echo Function Streaming ✅ Pass 0.225s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.201s
Search Query Function ✅ Pass 0.203s
Ask Advice Function ✅ Pass 0.206s
Streaming Search Query Function Streaming ✅ Pass 0.198s
Basic Context Memory Test ✅ Pass 0.206s
Function Response Memory Test ✅ Pass 1.815s
Penetration Testing Memory with Tool Call ✅ Pass 2.026s
Cybersecurity Workflow Memory Test ✅ Pass 2.458s
Function Argument Memory Test ✅ Pass 3.265s
Read a file, then edit it via unified diff ✅ Pass 1.695s
Penetration Testing Methodology ✅ Pass 0.191s
Vulnerability Assessment Tools ✅ Pass 0.182s
SQL Injection Attack Type ✅ Pass 0.197s
Penetration Testing Framework ✅ Pass 0.186s
Web Application Security Scanner ✅ Pass 0.191s
Penetration Testing Tool Selection ✅ Pass 0.189s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.611s


pentester (grok-4.3)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.219s
Simple Math ✅ Pass 0.203s
Text Transform Uppercase ✅ Pass 0.206s
Count from 1 to 5 ✅ Pass 0.188s
Math Calculation ✅ Pass 0.237s
Basic Echo Function ✅ Pass 0.211s
Streaming Simple Math Streaming ✅ Pass 0.252s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.255s
Streaming Basic Echo Function Streaming ✅ Pass 0.252s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.203s
Search Query Function ✅ Pass 0.189s
Ask Advice Function ✅ Pass 0.191s
Streaming Search Query Function Streaming ✅ Pass 0.220s
Basic Context Memory Test ✅ Pass 0.208s
Function Response Memory Test ✅ Pass 1.709s
Function Argument Memory Test ✅ Pass 1.815s
Penetration Testing Memory with Tool Call ✅ Pass 2.052s
Cybersecurity Workflow Memory Test ✅ Pass 2.394s
Read a file, then edit it via unified diff ✅ Pass 1.753s
Penetration Testing Methodology ✅ Pass 0.183s
Vulnerability Assessment Tools ✅ Pass 0.190s
SQL Injection Attack Type ✅ Pass 0.195s
Penetration Testing Framework ✅ Pass 0.196s
Web Application Security Scanner ✅ Pass 0.181s
Penetration Testing Tool Selection ✅ Pass 0.181s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.556s