Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

22.2 KB
Raw Blame

LLM Agent Testing Report

Generated: Thu, 23 Jul 2026 13:52:24 UTC

Overall Results

Agent Model Reasoning Success Rate Average Latency
simple openai.gpt-oss-120b-1:0 true 24/24 (100.00%) 0.925s
simple_json openai.gpt-oss-120b-1:0 true 6/7 (85.71%) 1.165s
primary_agent us.anthropic.claude-sonnet-4-5-20250929-v1:0 true 24/24 (100.00%) 5.064s
assistant us.anthropic.claude-sonnet-4-5-20250929-v1:0 true 24/24 (100.00%) 5.458s
generator us.anthropic.claude-sonnet-4-5-20250929-v1:0 true 24/24 (100.00%) 5.905s
refiner us.anthropic.claude-sonnet-4-5-20250929-v1:0 true 24/24 (100.00%) 5.909s
adviser us.anthropic.claude-opus-4-6-v1 true 25/25 (100.00%) 3.157s
reflector us.anthropic.claude-haiku-4-5-20251001-v1:0 true 24/24 (100.00%) 2.100s
searcher us.anthropic.claude-haiku-4-5-20251001-v1:0 true 24/24 (100.00%) 1.996s
enricher us.anthropic.claude-haiku-4-5-20251001-v1:0 true 24/24 (100.00%) 1.924s
coder us.anthropic.claude-sonnet-4-5-20250929-v1:0 true 24/24 (100.00%) 3.952s
installer us.anthropic.claude-sonnet-4-5-20250929-v1:0 true 24/24 (100.00%) 3.825s
pentester us.anthropic.claude-sonnet-4-5-20250929-v1:0 true 24/24 (100.00%) 5.473s

Total: 295/296 (99.66%) successful tests Overall average latency: 3.743s

Detailed Results

simple (openai.gpt-oss-120b-1:0)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.186s
Text Transform Uppercase ✅ Pass 0.663s
Count from 1 to 5 ✅ Pass 2.572s
Math Calculation ✅ Pass 0.507s
Basic Echo Function ✅ Pass 0.921s
Streaming Simple Math Streaming ✅ Pass 0.562s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.587s
Streaming Basic Echo Function Streaming ✅ Pass 1.478s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.048s
Search Query Function ✅ Pass 0.941s
Ask Advice Function ✅ Pass 0.622s
Streaming Search Query Function Streaming ✅ Pass 0.689s
Basic Context Memory Test ✅ Pass 0.487s
Function Argument Memory Test ✅ Pass 0.327s
Function Response Memory Test ✅ Pass 0.415s
Penetration Testing Memory with Tool Call ✅ Pass 1.690s
Cybersecurity Workflow Memory Test ✅ Pass 0.449s
Read a file, then edit it via unified diff ✅ Pass 1.678s
Penetration Testing Methodology ✅ Pass 0.482s
Vulnerability Assessment Tools ✅ Pass 0.860s
SQL Injection Attack Type ✅ Pass 0.811s
Penetration Testing Framework ✅ Pass 1.515s
Web Application Security Scanner ✅ Pass 0.503s
Penetration Testing Tool Selection ✅ Pass 1.185s

Summary: 24/24 (100.00%) successful tests

Average latency: 0.925s


simple_json (openai.gpt-oss-120b-1:0)

Advanced Tests

Test Result Latency Error
Vulnerability Report Memory Test ✅ Pass 2.278s
Person Information JSON ✅ Pass 0.519s
Project Information JSON ✅ Pass 1.633s
User Profile JSON ✅ Pass 1.150s
JSON Array Response Without Schema ✅ Pass 0.482s
Streaming Person Information JSON Streaming ✅ Pass 1.352s

Capability Tests

Test Capability Result Latency Note
Structured Output With JSON Schema structured_output ❌ Fail 0.737s structured output: response validation failed (provider=bedrock model=openai.gpt-oss-120b-1:0 choice=0 stop_reason=end_turn): response is n...

Summary: 6/7 (85.71%) successful tests

Average latency: 1.165s


primary_agent (us.anthropic.claude-sonnet-4-5-20250929-v1:0)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.329s
Text Transform Uppercase ✅ Pass 3.413s
Count from 1 to 5 ✅ Pass 2.544s
Math Calculation ✅ Pass 1.546s
Basic Echo Function ✅ Pass 6.180s
Streaming Simple Math Streaming ✅ Pass 6.043s
Streaming Count from 1 to 3 Streaming ✅ Pass 7.734s
Streaming Basic Echo Function Streaming ✅ Pass 2.355s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 5.141s
Search Query Function ✅ Pass 3.702s
Ask Advice Function ✅ Pass 2.272s
Streaming Search Query Function Streaming ✅ Pass 11.213s
Basic Context Memory Test ✅ Pass 10.009s
Function Argument Memory Test ✅ Pass 2.917s
Function Response Memory Test ✅ Pass 5.162s
Penetration Testing Memory with Tool Call ✅ Pass 4.143s
Cybersecurity Workflow Memory Test ✅ Pass 3.283s
Read a file, then edit it via unified diff ✅ Pass 6.530s
Penetration Testing Methodology ✅ Pass 9.574s
Vulnerability Assessment Tools ✅ Pass 7.992s
SQL Injection Attack Type ✅ Pass 3.212s
Penetration Testing Framework ✅ Pass 5.664s
Web Application Security Scanner ✅ Pass 5.351s
Penetration Testing Tool Selection ✅ Pass 3.205s

Summary: 24/24 (100.00%) successful tests

Average latency: 5.064s


assistant (us.anthropic.claude-sonnet-4-5-20250929-v1:0)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.367s
Text Transform Uppercase ✅ Pass 2.837s
Count from 1 to 5 ✅ Pass 2.651s
Math Calculation ✅ Pass 9.216s
Basic Echo Function ✅ Pass 4.094s
Streaming Simple Math Streaming ✅ Pass 6.681s
Streaming Count from 1 to 3 Streaming ✅ Pass 5.003s
Streaming Basic Echo Function Streaming ✅ Pass 5.792s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 9.983s
Search Query Function ✅ Pass 5.538s
Ask Advice Function ✅ Pass 7.430s
Streaming Search Query Function Streaming ✅ Pass 2.729s
Basic Context Memory Test ✅ Pass 2.832s
Function Argument Memory Test ✅ Pass 5.509s
Function Response Memory Test ✅ Pass 9.934s
Penetration Testing Memory with Tool Call ✅ Pass 5.064s
Cybersecurity Workflow Memory Test ✅ Pass 3.274s
Read a file, then edit it via unified diff ✅ Pass 6.139s
Penetration Testing Methodology ✅ Pass 7.628s
Vulnerability Assessment Tools ✅ Pass 7.695s
SQL Injection Attack Type ✅ Pass 3.696s
Penetration Testing Framework ✅ Pass 6.685s
Web Application Security Scanner ✅ Pass 5.280s
Penetration Testing Tool Selection ✅ Pass 2.920s

Summary: 24/24 (100.00%) successful tests

Average latency: 5.458s


generator (us.anthropic.claude-sonnet-4-5-20250929-v1:0)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.816s
Text Transform Uppercase ✅ Pass 2.134s
Count from 1 to 5 ✅ Pass 2.502s
Math Calculation ✅ Pass 2.185s
Basic Echo Function ✅ Pass 7.081s
Streaming Simple Math Streaming ✅ Pass 4.785s
Streaming Count from 1 to 3 Streaming ✅ Pass 6.945s
Streaming Basic Echo Function Streaming ✅ Pass 5.815s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 3.219s
Search Query Function ✅ Pass 10.109s
Ask Advice Function ✅ Pass 4.937s
Streaming Search Query Function Streaming ✅ Pass 4.710s
Basic Context Memory Test ✅ Pass 6.858s
Function Argument Memory Test ✅ Pass 4.102s
Function Response Memory Test ✅ Pass 3.286s
Penetration Testing Memory with Tool Call ✅ Pass 6.096s
Cybersecurity Workflow Memory Test ✅ Pass 7.840s
Read a file, then edit it via unified diff ✅ Pass 13.576s
Penetration Testing Methodology ✅ Pass 8.999s
Vulnerability Assessment Tools ✅ Pass 9.065s
SQL Injection Attack Type ✅ Pass 5.445s
Penetration Testing Framework ✅ Pass 10.560s
Web Application Security Scanner ✅ Pass 7.129s
Penetration Testing Tool Selection ✅ Pass 2.512s

Summary: 24/24 (100.00%) successful tests

Average latency: 5.905s


refiner (us.anthropic.claude-sonnet-4-5-20250929-v1:0)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.868s
Text Transform Uppercase ✅ Pass 2.619s
Count from 1 to 5 ✅ Pass 2.149s
Math Calculation ✅ Pass 10.627s
Basic Echo Function ✅ Pass 7.943s
Streaming Simple Math Streaming ✅ Pass 9.873s
Streaming Count from 1 to 3 Streaming ✅ Pass 3.635s
Streaming Basic Echo Function Streaming ✅ Pass 6.441s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 6.119s
Search Query Function ✅ Pass 2.248s
Ask Advice Function ✅ Pass 6.612s
Streaming Search Query Function Streaming ✅ Pass 4.235s
Basic Context Memory Test ✅ Pass 4.851s
Function Argument Memory Test ✅ Pass 7.741s
Function Response Memory Test ✅ Pass 7.798s
Penetration Testing Memory with Tool Call ✅ Pass 5.301s
Cybersecurity Workflow Memory Test ✅ Pass 3.042s
Read a file, then edit it via unified diff ✅ Pass 11.847s
Penetration Testing Methodology ✅ Pass 9.992s
Vulnerability Assessment Tools ✅ Pass 7.838s
SQL Injection Attack Type ✅ Pass 2.739s
Penetration Testing Framework ✅ Pass 6.891s
Web Application Security Scanner ✅ Pass 5.807s
Penetration Testing Tool Selection ✅ Pass 2.592s

Summary: 24/24 (100.00%) successful tests

Average latency: 5.909s


adviser (us.anthropic.claude-opus-4-6-v1)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.796s
Text Transform Uppercase ✅ Pass 1.961s
Count from 1 to 5 ✅ Pass 1.989s
Math Calculation ✅ Pass 2.222s
Basic Echo Function ✅ Pass 2.073s
Streaming Simple Math Streaming ✅ Pass 2.119s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.883s
Streaming Basic Echo Function Streaming ✅ Pass 2.116s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 2.285s
Search Query Function ✅ Pass 2.387s
Ask Advice Function ✅ Pass 2.132s
Streaming Search Query Function Streaming ✅ Pass 2.310s
Basic Context Memory Test ✅ Pass 2.382s
Function Argument Memory Test ✅ Pass 2.671s
Function Response Memory Test ✅ Pass 2.469s
Penetration Testing Memory with Tool Call ✅ Pass 3.786s
Cybersecurity Workflow Memory Test ✅ Pass 2.676s
Read a file, then edit it via unified diff ✅ Pass 6.302s
Penetration Testing Methodology ✅ Pass 7.591s
Vulnerability Assessment Tools ✅ Pass 6.459s
SQL Injection Attack Type ✅ Pass 2.692s
Penetration Testing Framework ✅ Pass 5.051s
Web Application Security Scanner ✅ Pass 4.501s
Penetration Testing Tool Selection ✅ Pass 2.532s

Capability Tests

Test Capability Result Latency Note
Adaptive Thinking Produces Reasoning adaptive_thinking ✅ Pass 4.535s

Summary: 25/25 (100.00%) successful tests

Average latency: 3.157s


reflector (us.anthropic.claude-haiku-4-5-20251001-v1:0)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.679s
Text Transform Uppercase ✅ Pass 1.102s
Count from 1 to 5 ✅ Pass 1.182s
Math Calculation ✅ Pass 1.150s
Basic Echo Function ✅ Pass 2.008s
Streaming Simple Math Streaming ✅ Pass 1.068s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.144s
Streaming Basic Echo Function Streaming ✅ Pass 1.523s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.484s
Search Query Function ✅ Pass 1.366s
Ask Advice Function ✅ Pass 1.509s
Streaming Search Query Function Streaming ✅ Pass 1.413s
Basic Context Memory Test ✅ Pass 1.398s
Function Argument Memory Test ✅ Pass 1.560s
Function Response Memory Test ✅ Pass 1.405s
Penetration Testing Memory with Tool Call ✅ Pass 1.864s
Cybersecurity Workflow Memory Test ✅ Pass 1.677s
Read a file, then edit it via unified diff ✅ Pass 6.126s
Penetration Testing Methodology ✅ Pass 4.944s
Vulnerability Assessment Tools ✅ Pass 4.513s
SQL Injection Attack Type ✅ Pass 1.965s
Penetration Testing Framework ✅ Pass 3.471s
Web Application Security Scanner ✅ Pass 2.792s
Penetration Testing Tool Selection ✅ Pass 2.042s

Summary: 24/24 (100.00%) successful tests

Average latency: 2.100s


searcher (us.anthropic.claude-haiku-4-5-20251001-v1:0)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.461s
Text Transform Uppercase ✅ Pass 1.179s
Count from 1 to 5 ✅ Pass 1.594s
Math Calculation ✅ Pass 0.974s
Basic Echo Function ✅ Pass 1.311s
Streaming Simple Math Streaming ✅ Pass 1.281s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.202s
Streaming Basic Echo Function Streaming ✅ Pass 1.288s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.384s
Search Query Function ✅ Pass 1.233s
Ask Advice Function ✅ Pass 1.522s
Streaming Search Query Function Streaming ✅ Pass 1.505s
Basic Context Memory Test ✅ Pass 1.488s
Function Argument Memory Test ✅ Pass 1.488s
Function Response Memory Test ✅ Pass 1.483s
Penetration Testing Memory with Tool Call ✅ Pass 3.077s
Cybersecurity Workflow Memory Test ✅ Pass 1.505s
Read a file, then edit it via unified diff ✅ Pass 3.537s
Penetration Testing Methodology ✅ Pass 4.214s
Vulnerability Assessment Tools ✅ Pass 4.317s
SQL Injection Attack Type ✅ Pass 1.680s
Penetration Testing Framework ✅ Pass 4.428s
Web Application Security Scanner ✅ Pass 3.003s
Penetration Testing Tool Selection ✅ Pass 1.741s

Summary: 24/24 (100.00%) successful tests

Average latency: 1.996s


enricher (us.anthropic.claude-haiku-4-5-20251001-v1:0)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.076s
Text Transform Uppercase ✅ Pass 1.069s
Count from 1 to 5 ✅ Pass 1.426s
Math Calculation ✅ Pass 1.292s
Basic Echo Function ✅ Pass 1.226s
Streaming Simple Math Streaming ✅ Pass 0.987s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.501s
Streaming Basic Echo Function Streaming ✅ Pass 2.189s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.353s
Search Query Function ✅ Pass 1.247s
Ask Advice Function ✅ Pass 1.457s
Streaming Search Query Function Streaming ✅ Pass 1.302s
Basic Context Memory Test ✅ Pass 1.146s
Function Argument Memory Test ✅ Pass 1.467s
Function Response Memory Test ✅ Pass 1.281s
Penetration Testing Memory with Tool Call ✅ Pass 2.432s
Cybersecurity Workflow Memory Test ✅ Pass 1.489s
Read a file, then edit it via unified diff ✅ Pass 3.384s
Penetration Testing Methodology ✅ Pass 3.548s
Vulnerability Assessment Tools ✅ Pass 4.859s
SQL Injection Attack Type ✅ Pass 1.556s
Penetration Testing Framework ✅ Pass 3.209s
Web Application Security Scanner ✅ Pass 4.042s
Penetration Testing Tool Selection ✅ Pass 1.632s

Summary: 24/24 (100.00%) successful tests

Average latency: 1.924s


coder (us.anthropic.claude-sonnet-4-5-20250929-v1:0)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.576s
Text Transform Uppercase ✅ Pass 2.114s
Count from 1 to 5 ✅ Pass 2.913s
Math Calculation ✅ Pass 2.111s
Basic Echo Function ✅ Pass 3.436s
Streaming Simple Math Streaming ✅ Pass 2.611s
Streaming Count from 1 to 3 Streaming ✅ Pass 2.983s
Streaming Basic Echo Function Streaming ✅ Pass 2.376s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 2.710s
Search Query Function ✅ Pass 2.976s
Ask Advice Function ✅ Pass 2.751s
Streaming Search Query Function Streaming ✅ Pass 2.175s
Basic Context Memory Test ✅ Pass 2.772s
Function Argument Memory Test ✅ Pass 3.752s
Function Response Memory Test ✅ Pass 3.655s
Penetration Testing Memory with Tool Call ✅ Pass 4.272s
Cybersecurity Workflow Memory Test ✅ Pass 2.688s
Read a file, then edit it via unified diff ✅ Pass 8.360s
Penetration Testing Methodology ✅ Pass 9.056s
Vulnerability Assessment Tools ✅ Pass 8.946s
SQL Injection Attack Type ✅ Pass 4.097s
Penetration Testing Framework ✅ Pass 5.506s
Web Application Security Scanner ✅ Pass 6.770s
Penetration Testing Tool Selection ✅ Pass 3.232s

Summary: 24/24 (100.00%) successful tests

Average latency: 3.952s


installer (us.anthropic.claude-sonnet-4-5-20250929-v1:0)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.592s
Text Transform Uppercase ✅ Pass 2.380s
Count from 1 to 5 ✅ Pass 1.810s
Math Calculation ✅ Pass 2.042s
Basic Echo Function ✅ Pass 2.297s
Streaming Simple Math Streaming ✅ Pass 1.833s
Streaming Count from 1 to 3 Streaming ✅ Pass 2.224s
Streaming Basic Echo Function Streaming ✅ Pass 2.641s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 2.645s
Search Query Function ✅ Pass 2.456s
Ask Advice Function ✅ Pass 3.426s
Streaming Search Query Function Streaming ✅ Pass 6.833s
Basic Context Memory Test ✅ Pass 2.283s
Function Argument Memory Test ✅ Pass 3.039s
Function Response Memory Test ✅ Pass 2.601s
Penetration Testing Memory with Tool Call ✅ Pass 4.714s
Cybersecurity Workflow Memory Test ✅ Pass 2.684s
Read a file, then edit it via unified diff ✅ Pass 7.480s
Penetration Testing Methodology ✅ Pass 9.298s
Vulnerability Assessment Tools ✅ Pass 7.915s
SQL Injection Attack Type ✅ Pass 4.091s
Penetration Testing Framework ✅ Pass 5.788s
Web Application Security Scanner ✅ Pass 5.974s
Penetration Testing Tool Selection ✅ Pass 2.749s

Summary: 24/24 (100.00%) successful tests

Average latency: 3.825s


pentester (us.anthropic.claude-sonnet-4-5-20250929-v1:0)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.288s
Text Transform Uppercase ✅ Pass 2.883s
Count from 1 to 5 ✅ Pass 2.553s
Math Calculation ✅ Pass 4.614s
Basic Echo Function ✅ Pass 3.006s
Streaming Simple Math Streaming ✅ Pass 4.374s
Streaming Count from 1 to 3 Streaming ✅ Pass 10.242s
Streaming Basic Echo Function Streaming ✅ Pass 4.492s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 5.880s
Search Query Function ✅ Pass 10.373s
Ask Advice Function ✅ Pass 3.626s
Streaming Search Query Function Streaming ✅ Pass 2.263s
Basic Context Memory Test ✅ Pass 11.123s
Function Argument Memory Test ✅ Pass 3.534s
Function Response Memory Test ✅ Pass 3.349s
Penetration Testing Memory with Tool Call ✅ Pass 4.539s
Cybersecurity Workflow Memory Test ✅ Pass 6.869s
Read a file, then edit it via unified diff ✅ Pass 7.278s
Penetration Testing Methodology ✅ Pass 9.884s
Vulnerability Assessment Tools ✅ Pass 8.628s
SQL Injection Attack Type ✅ Pass 3.626s
Penetration Testing Framework ✅ Pass 6.146s
Web Application Security Scanner ✅ Pass 6.964s
Penetration Testing Tool Selection ✅ Pass 2.808s

Summary: 24/24 (100.00%) successful tests

Average latency: 5.473s