Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

22.6 KB
Raw Blame

LLM Agent Testing Report

Generated: Fri, 25 Sep 2026 00:15:09 UTC

Overall Results

Agent Model Reasoning Success Rate Average Latency
simple claude-haiku-4-5 false 25/25 (100.00%) 1.981s
simple_json claude-haiku-4-5 false 8/8 (100.00%) 1.036s
primary_agent claude-sonnet-5 true 26/26 (100.00%) 2.799s
assistant claude-sonnet-5 true 26/26 (100.00%) 2.936s
generator claude-opus-4-8 true 26/26 (100.00%) 4.156s
refiner claude-opus-4-8 true 26/26 (100.00%) 3.751s
adviser claude-sonnet-5 true 17/17 (100.00%) 3.577s
reflector claude-haiku-4-5 true 16/16 (100.00%) 3.148s
searcher claude-haiku-4-5 true 25/25 (100.00%) 2.677s
enricher claude-haiku-4-5 false 25/25 (100.00%) 2.044s
coder claude-sonnet-5 true 26/26 (100.00%) 2.848s
installer claude-sonnet-5 true 26/26 (100.00%) 2.841s
pentester claude-sonnet-5 true 26/26 (100.00%) 3.007s

Total: 298/298 (100.00%) successful tests Overall average latency: 2.912s

Detailed Results

simple (claude-haiku-4-5)

Basic Tests

Test Result Latency Error
Math Calculation ✅ Pass 0.874s
Simple Math ✅ Pass 0.888s
Streaming Simple Math Streaming ✅ Pass 0.890s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.891s
Text Transform Uppercase ✅ Pass 0.909s
Count from 1 to 5 ✅ Pass 1.241s
Basic Echo Function ✅ Pass 1.271s
Streaming Basic Echo Function Streaming ✅ Pass 1.166s
Answer Stops At The Output Limit ✅ Pass 15.809s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.270s
Search Query Function ✅ Pass 0.850s
Basic Context Memory Test ✅ Pass 0.808s
Streaming Search Query Function Streaming ✅ Pass 0.913s
Function Argument Memory Test ✅ Pass 0.661s
Function Response Memory Test ✅ Pass 0.625s
Cybersecurity Workflow Memory Test ✅ Pass 0.718s
Ask Advice Function ✅ Pass 1.991s
Penetration Testing Memory with Tool Call ✅ Pass 1.547s
Read a file, then edit it via unified diff ✅ Pass 3.774s
SQL Injection Attack Type ✅ Pass 0.796s
Penetration Testing Tool Selection ✅ Pass 1.178s
Penetration Testing Methodology ✅ Pass 2.452s
Web Application Security Scanner ✅ Pass 2.119s
Vulnerability Assessment Tools ✅ Pass 3.221s
Penetration Testing Framework ✅ Pass 2.641s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.981s


simple_json (claude-haiku-4-5)

Advanced Tests

Test Result Latency Error
Vulnerability Report Memory Test ✅ Pass 0.952s
Person Information JSON ✅ Pass 0.824s
Project Information JSON ✅ Pass 0.964s
User Profile JSON ✅ Pass 1.240s
JSON Array Response Without Schema ✅ Pass 0.906s
Streaming Person Information JSON Streaming ✅ Pass 0.997s

Capability Tests

Test Capability Result Latency Note
Structured Output Refuses A Non-Object Schema structured_output ✅ Pass 0.000s
Structured Output With JSON Schema structured_output ✅ Pass 2.397s

Summary: 8/8 (100.00%) successful tests

Average latency: 1.036s


primary_agent (claude-sonnet-5)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 1.263s
Math Calculation ✅ Pass 1.000s
Simple Math ✅ Pass 1.492s
Basic Echo Function ✅ Pass 1.726s
Streaming Simple Math Streaming ✅ Pass 1.184s
Count from 1 to 5 ✅ Pass 2.454s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.510s
Streaming Basic Echo Function Streaming ✅ Pass 1.779s
Answer Stops At The Output Limit ✅ Pass 18.677s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.508s
Search Query Function ✅ Pass 1.754s
Streaming Search Query Function Streaming ✅ Pass 1.458s
Ask Advice Function ✅ Pass 1.833s
Basic Context Memory Test ✅ Pass 1.390s
Function Argument Memory Test ✅ Pass 1.116s
Function Response Memory Test ✅ Pass 0.964s
Cybersecurity Workflow Memory Test ✅ Pass 0.804s
Penetration Testing Memory with Tool Call ✅ Pass 2.535s
Read a file, then edit it via unified diff ✅ Pass 4.239s
SQL Injection Attack Type ✅ Pass 1.923s
Penetration Testing Tool Selection ✅ Pass 1.583s
Penetration Testing Methodology ✅ Pass 3.646s
Web Application Security Scanner ✅ Pass 3.270s
Penetration Testing Framework ✅ Pass 4.360s
Vulnerability Assessment Tools ✅ Pass 6.727s

Capability Tests

Test Capability Result Latency Note
Adaptive Thinking Produces Reasoning adaptive_thinking ✅ Pass 2.562s

Summary: 26/26 (100.00%) successful tests

Average latency: 2.799s


assistant (claude-sonnet-5)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 1.132s
Simple Math ✅ Pass 1.370s
Math Calculation ✅ Pass 0.917s
Count from 1 to 5 ✅ Pass 2.172s
Basic Echo Function ✅ Pass 1.757s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.449s
Streaming Simple Math Streaming ✅ Pass 1.480s
Streaming Basic Echo Function Streaming ✅ Pass 1.993s
Answer Stops At The Output Limit ✅ Pass 18.052s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.527s
Search Query Function ✅ Pass 1.589s
Ask Advice Function ✅ Pass 1.690s
Streaming Search Query Function Streaming ✅ Pass 1.568s
Function Response Memory Test ✅ Pass 1.024s
Function Argument Memory Test ✅ Pass 1.072s
Basic Context Memory Test ✅ Pass 1.572s
Cybersecurity Workflow Memory Test ✅ Pass 1.147s
Penetration Testing Memory with Tool Call ✅ Pass 2.984s
Read a file, then edit it via unified diff ✅ Pass 4.620s
SQL Injection Attack Type ✅ Pass 2.002s
Penetration Testing Tool Selection ✅ Pass 1.855s
Vulnerability Assessment Tools ✅ Pass 5.407s
Penetration Testing Methodology ✅ Pass 5.796s
Penetration Testing Framework ✅ Pass 4.730s
Web Application Security Scanner ✅ Pass 4.508s

Capability Tests

Test Capability Result Latency Note
Adaptive Thinking Produces Reasoning adaptive_thinking ✅ Pass 2.895s

Summary: 26/26 (100.00%) successful tests

Average latency: 2.936s


generator (claude-opus-4-8)

Basic Tests

Test Result Latency Error
Count from 1 to 5 ✅ Pass 1.350s
Text Transform Uppercase ✅ Pass 1.769s
Simple Math ✅ Pass 1.836s
Math Calculation ✅ Pass 1.167s
Streaming Simple Math Streaming ✅ Pass 1.171s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.174s
Basic Echo Function ✅ Pass 2.307s
Streaming Basic Echo Function Streaming ✅ Pass 1.868s
Answer Stops At The Output Limit ✅ Pass 30.394s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.745s
Search Query Function ✅ Pass 1.789s
Streaming Search Query Function Streaming ✅ Pass 1.644s
Ask Advice Function ✅ Pass 2.367s
Function Argument Memory Test ✅ Pass 1.388s
Function Response Memory Test ✅ Pass 1.073s
Basic Context Memory Test ✅ Pass 1.567s
Cybersecurity Workflow Memory Test ✅ Pass 1.302s
Penetration Testing Memory with Tool Call ✅ Pass 3.787s
Read a file, then edit it via unified diff ✅ Pass 6.241s
Penetration Testing Methodology ✅ Pass 5.411s
Vulnerability Assessment Tools ✅ Pass 7.381s
Penetration Testing Tool Selection ✅ Pass 5.637s
SQL Injection Attack Type ✅ Pass 6.556s
Web Application Security Scanner ✅ Pass 6.281s
Penetration Testing Framework ✅ Pass 6.726s

Capability Tests

Test Capability Result Latency Note
Adaptive Thinking Produces Reasoning adaptive_thinking ✅ Pass 4.122s

Summary: 26/26 (100.00%) successful tests

Average latency: 4.156s


refiner (claude-opus-4-8)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.328s
Count from 1 to 5 ✅ Pass 1.165s
Text Transform Uppercase ✅ Pass 1.331s
Math Calculation ✅ Pass 1.117s
Streaming Simple Math Streaming ✅ Pass 1.116s
Basic Echo Function ✅ Pass 1.967s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.919s
Streaming Basic Echo Function Streaming ✅ Pass 1.480s
Answer Stops At The Output Limit ✅ Pass 30.239s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 2.356s
Search Query Function ✅ Pass 2.388s
Ask Advice Function ✅ Pass 2.508s
Function Argument Memory Test ✅ Pass 1.091s
Basic Context Memory Test ✅ Pass 1.218s
Streaming Search Query Function Streaming ✅ Pass 3.793s
Function Response Memory Test ✅ Pass 1.794s
Cybersecurity Workflow Memory Test ✅ Pass 1.549s
Penetration Testing Memory with Tool Call ✅ Pass 4.052s
Read a file, then edit it via unified diff ✅ Pass 4.725s
SQL Injection Attack Type ✅ Pass 3.770s
Penetration Testing Methodology ✅ Pass 6.680s
Penetration Testing Tool Selection ✅ Pass 1.856s
Web Application Security Scanner ✅ Pass 2.248s
Vulnerability Assessment Tools ✅ Pass 7.482s
Penetration Testing Framework ✅ Pass 4.295s

Capability Tests

Test Capability Result Latency Note
Adaptive Thinking Produces Reasoning adaptive_thinking ✅ Pass 4.056s

Summary: 26/26 (100.00%) successful tests

Average latency: 3.751s


adviser (claude-sonnet-5)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.265s
Text Transform Uppercase ✅ Pass 1.288s
Math Calculation ✅ Pass 1.340s
Count from 1 to 5 ✅ Pass 2.118s
Streaming Simple Math Streaming ✅ Pass 1.464s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.340s
Answer Stops At The Output Limit ✅ Pass 19.117s

Advanced Tests

Test Result Latency Error
Function Argument Memory Test ✅ Pass 1.522s
Function Response Memory Test ✅ Pass 0.948s
Basic Context Memory Test ✅ Pass 2.910s
Cybersecurity Workflow Memory Test ✅ Pass 2.748s
SQL Injection Attack Type ✅ Pass 1.909s
Web Application Security Scanner ✅ Pass 3.889s
Penetration Testing Methodology ✅ Pass 5.324s
Penetration Testing Framework ✅ Pass 4.514s
Vulnerability Assessment Tools ✅ Pass 5.654s

Capability Tests

Test Capability Result Latency Note
Adaptive Thinking Produces Reasoning adaptive_thinking ✅ Pass 3.446s

Summary: 17/17 (100.00%) successful tests

Average latency: 3.577s


reflector (claude-haiku-4-5)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.924s
Text Transform Uppercase ✅ Pass 0.816s
Count from 1 to 5 ✅ Pass 1.037s
Math Calculation ✅ Pass 1.155s
Streaming Simple Math Streaming ✅ Pass 1.025s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.125s
Answer Stops At The Output Limit ✅ Pass 18.402s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 1.152s
Function Argument Memory Test ✅ Pass 1.203s
Function Response Memory Test ✅ Pass 1.470s
Cybersecurity Workflow Memory Test ✅ Pass 1.313s
SQL Injection Attack Type ✅ Pass 2.528s
Vulnerability Assessment Tools ✅ Pass 5.107s
Penetration Testing Methodology ✅ Pass 5.361s
Penetration Testing Framework ✅ Pass 4.245s
Web Application Security Scanner ✅ Pass 3.492s

Summary: 16/16 (100.00%) successful tests

Average latency: 3.148s


searcher (claude-haiku-4-5)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.048s
Text Transform Uppercase ✅ Pass 1.072s
Math Calculation ✅ Pass 0.877s
Count from 1 to 5 ✅ Pass 2.131s
Basic Echo Function ✅ Pass 1.452s
Streaming Simple Math Streaming ✅ Pass 0.919s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.236s
Streaming Basic Echo Function Streaming ✅ Pass 1.561s
Answer Stops At The Output Limit ✅ Pass 17.417s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 1.128s
JSON Response Function ✅ Pass 1.634s
Ask Advice Function ✅ Pass 1.243s
Streaming Search Query Function Streaming ✅ Pass 1.479s
Basic Context Memory Test ✅ Pass 1.282s
Function Argument Memory Test ✅ Pass 1.207s
Function Response Memory Test ✅ Pass 1.456s
Cybersecurity Workflow Memory Test ✅ Pass 1.312s
Penetration Testing Memory with Tool Call ✅ Pass 1.934s
Read a file, then edit it via unified diff ✅ Pass 4.047s
SQL Injection Attack Type ✅ Pass 2.336s
Penetration Testing Tool Selection ✅ Pass 1.881s
Penetration Testing Framework ✅ Pass 3.955s
Penetration Testing Methodology ✅ Pass 4.970s
Vulnerability Assessment Tools ✅ Pass 5.260s
Web Application Security Scanner ✅ Pass 4.075s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.677s


enricher (claude-haiku-4-5)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.677s
Text Transform Uppercase ✅ Pass 0.751s
Count from 1 to 5 ✅ Pass 1.019s
Math Calculation ✅ Pass 0.746s
Basic Echo Function ✅ Pass 1.230s
Streaming Simple Math Streaming ✅ Pass 0.773s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.796s
Streaming Basic Echo Function Streaming ✅ Pass 0.917s
Answer Stops At The Output Limit ✅ Pass 17.005s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 0.956s
JSON Response Function ✅ Pass 1.233s
Ask Advice Function ✅ Pass 0.881s
Basic Context Memory Test ✅ Pass 0.817s
Streaming Search Query Function Streaming ✅ Pass 0.970s
Function Response Memory Test ✅ Pass 0.763s
Cybersecurity Workflow Memory Test ✅ Pass 0.783s
Function Argument Memory Test ✅ Pass 1.784s
Penetration Testing Memory with Tool Call ✅ Pass 1.700s
Read a file, then edit it via unified diff ✅ Pass 4.046s
SQL Injection Attack Type ✅ Pass 0.813s
Vulnerability Assessment Tools ✅ Pass 2.650s
Web Application Security Scanner ✅ Pass 2.158s
Penetration Testing Methodology ✅ Pass 3.528s
Penetration Testing Tool Selection ✅ Pass 1.460s
Penetration Testing Framework ✅ Pass 2.630s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.044s


coder (claude-sonnet-5)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.298s
Text Transform Uppercase ✅ Pass 1.407s
Count from 1 to 5 ✅ Pass 1.131s
Math Calculation ✅ Pass 1.020s
Basic Echo Function ✅ Pass 1.673s
Streaming Simple Math Streaming ✅ Pass 1.335s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.451s
Streaming Basic Echo Function Streaming ✅ Pass 1.662s
Answer Stops At The Output Limit ✅ Pass 19.048s

Advanced Tests

Test Result Latency Error
Streaming Search Query Function Streaming ✅ Pass 1.422s
Basic Context Memory Test ✅ Pass 1.311s
JSON Response Function ✅ Pass 1.702s
Function Argument Memory Test ✅ Pass 1.102s
Search Query Function ✅ Pass 1.626s
Ask Advice Function ✅ Pass 1.748s
Function Response Memory Test ✅ Pass 1.056s
Cybersecurity Workflow Memory Test ✅ Pass 1.173s
Penetration Testing Memory with Tool Call ✅ Pass 2.974s
Read a file, then edit it via unified diff ✅ Pass 4.579s
SQL Injection Attack Type ✅ Pass 2.046s
Penetration Testing Tool Selection ✅ Pass 2.171s
Penetration Testing Methodology ✅ Pass 4.087s
Web Application Security Scanner ✅ Pass 3.726s
Vulnerability Assessment Tools ✅ Pass 5.611s
Penetration Testing Framework ✅ Pass 5.141s

Capability Tests

Test Capability Result Latency Note
Adaptive Thinking Produces Reasoning adaptive_thinking ✅ Pass 2.532s

Summary: 26/26 (100.00%) successful tests

Average latency: 2.848s


installer (claude-sonnet-5)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.268s
Text Transform Uppercase ✅ Pass 0.963s
Count from 1 to 5 ✅ Pass 0.999s
Math Calculation ✅ Pass 1.077s
Basic Echo Function ✅ Pass 1.651s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.397s
Streaming Simple Math Streaming ✅ Pass 1.450s
Streaming Basic Echo Function Streaming ✅ Pass 1.630s
Answer Stops At The Output Limit ✅ Pass 19.376s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 1.595s
JSON Response Function ✅ Pass 1.612s
Ask Advice Function ✅ Pass 1.742s
Streaming Search Query Function Streaming ✅ Pass 1.674s
Function Argument Memory Test ✅ Pass 0.898s
Basic Context Memory Test ✅ Pass 1.485s
Function Response Memory Test ✅ Pass 1.092s
Cybersecurity Workflow Memory Test ✅ Pass 1.249s
Penetration Testing Memory with Tool Call ✅ Pass 2.692s
Read a file, then edit it via unified diff ✅ Pass 5.661s
SQL Injection Attack Type ✅ Pass 2.723s
Penetration Testing Methodology ✅ Pass 5.018s
Penetration Testing Tool Selection ✅ Pass 1.630s
Web Application Security Scanner ✅ Pass 2.487s
Penetration Testing Framework ✅ Pass 3.700s
Vulnerability Assessment Tools ✅ Pass 6.283s

Capability Tests

Test Capability Result Latency Note
Adaptive Thinking Produces Reasoning adaptive_thinking ✅ Pass 2.490s

Summary: 26/26 (100.00%) successful tests

Average latency: 2.841s


pentester (claude-sonnet-5)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 1.372s
Count from 1 to 5 ✅ Pass 0.934s
Simple Math ✅ Pass 1.536s
Math Calculation ✅ Pass 1.060s
Streaming Simple Math Streaming ✅ Pass 1.354s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.360s
Basic Echo Function ✅ Pass 1.548s
Streaming Basic Echo Function Streaming ✅ Pass 1.608s
Answer Stops At The Output Limit ✅ Pass 19.345s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.562s
Search Query Function ✅ Pass 1.580s
Function Argument Memory Test ✅ Pass 0.896s
Basic Context Memory Test ✅ Pass 1.441s
Streaming Search Query Function Streaming ✅ Pass 1.662s
Ask Advice Function ✅ Pass 1.878s
Function Response Memory Test ✅ Pass 0.990s
Cybersecurity Workflow Memory Test ✅ Pass 1.093s
Penetration Testing Memory with Tool Call ✅ Pass 2.511s
Read a file, then edit it via unified diff ✅ Pass 5.174s
Penetration Testing Tool Selection ✅ Pass 1.559s
SQL Injection Attack Type ✅ Pass 2.172s
Web Application Security Scanner ✅ Pass 4.206s
Penetration Testing Framework ✅ Pass 5.018s
Penetration Testing Methodology ✅ Pass 6.336s
Vulnerability Assessment Tools ✅ Pass 7.407s

Capability Tests

Test Capability Result Latency Note
Adaptive Thinking Produces Reasoning adaptive_thinking ✅ Pass 2.559s

Summary: 26/26 (100.00%) successful tests

Average latency: 3.007s