Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

21.2 KB
Raw Blame

LLM Agent Testing Report

Generated: Thu, 23 Jul 2026 14:01:26 UTC

Overall Results

Agent Model Reasoning Success Rate Average Latency
simple gpt-4.1-mini false 24/24 (100.00%) 1.125s
simple_json gpt-4.1-mini false 7/7 (100.00%) 1.248s
primary_agent o3-mini true 24/24 (100.00%) 1.538s
assistant o3-mini true 23/24 (95.83%) 2.135s
generator o3-mini true 24/24 (100.00%) 2.359s
refiner gpt-4.1 false 24/24 (100.00%) 1.136s
adviser o3-mini true 24/24 (100.00%) 2.425s
reflector o3-mini true 24/24 (100.00%) 2.018s
searcher gpt-4.1-mini false 24/24 (100.00%) 1.491s
enricher gpt-4.1-mini false 24/24 (100.00%) 1.101s
coder gpt-4.1 false 24/24 (100.00%) 1.090s
installer gpt-4.1 false 24/24 (100.00%) 1.192s
pentester o3-mini true 23/24 (95.83%) 1.392s

Total: 293/295 (99.32%) successful tests Overall average latency: 1.576s

Detailed Results

simple (gpt-4.1-mini)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.352s
Text Transform Uppercase ✅ Pass 0.810s
Count from 1 to 5 ✅ Pass 0.572s
Math Calculation ✅ Pass 0.526s
Basic Echo Function ✅ Pass 0.980s
Streaming Simple Math Streaming ✅ Pass 0.735s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.845s
Streaming Basic Echo Function Streaming ✅ Pass 1.065s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.985s
Search Query Function ✅ Pass 0.950s
Ask Advice Function ✅ Pass 1.058s
Streaming Search Query Function Streaming ✅ Pass 0.741s
Basic Context Memory Test ✅ Pass 0.743s
Function Argument Memory Test ✅ Pass 0.867s
Function Response Memory Test ✅ Pass 0.556s
Penetration Testing Memory with Tool Call ✅ Pass 1.461s
Cybersecurity Workflow Memory Test ✅ Pass 0.729s
Read a file, then edit it via unified diff ✅ Pass 2.892s
Penetration Testing Methodology ✅ Pass 1.063s
Vulnerability Assessment Tools ✅ Pass 2.046s
SQL Injection Attack Type ✅ Pass 0.789s
Penetration Testing Framework ✅ Pass 0.923s
Web Application Security Scanner ✅ Pass 1.040s
Penetration Testing Tool Selection ✅ Pass 1.268s

Summary: 24/24 (100.00%) successful tests

Average latency: 1.125s


simple_json (gpt-4.1-mini)

Advanced Tests

Test Result Latency Error
Vulnerability Report Memory Test ✅ Pass 1.547s
Project Information JSON ✅ Pass 0.996s
User Profile JSON ✅ Pass 1.236s
Person Information JSON ✅ Pass 1.343s
Streaming Person Information JSON Streaming ✅ Pass 1.012s
JSON Array Response Without Schema ✅ Pass 1.777s

Capability Tests

Test Capability Result Latency Note
Structured Output With JSON Schema structured_output ✅ Pass 0.822s

Summary: 7/7 (100.00%) successful tests

Average latency: 1.248s


primary_agent (o3-mini)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.273s
Text Transform Uppercase ✅ Pass 0.927s
Count from 1 to 5 ✅ Pass 1.382s
Math Calculation ✅ Pass 0.754s
Basic Echo Function ✅ Pass 0.955s
Streaming Simple Math Streaming ✅ Pass 1.388s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.855s
Streaming Basic Echo Function Streaming ✅ Pass 1.404s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.122s
Search Query Function ✅ Pass 1.488s
Ask Advice Function ✅ Pass 1.547s
Streaming Search Query Function Streaming ✅ Pass 1.168s
Basic Context Memory Test ✅ Pass 1.456s
Function Argument Memory Test ✅ Pass 1.237s
Function Response Memory Test ✅ Pass 0.945s
Penetration Testing Memory with Tool Call ✅ Pass 1.739s
Cybersecurity Workflow Memory Test ✅ Pass 1.114s
Read a file, then edit it via unified diff ✅ Pass 3.412s
Penetration Testing Methodology ✅ Pass 1.630s
Vulnerability Assessment Tools ✅ Pass 2.166s
SQL Injection Attack Type ✅ Pass 2.493s
Penetration Testing Framework ✅ Pass 1.705s
Web Application Security Scanner ✅ Pass 2.348s
Penetration Testing Tool Selection ✅ Pass 1.396s

Summary: 24/24 (100.00%) successful tests

Average latency: 1.538s


assistant (o3-mini)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.883s
Text Transform Uppercase ✅ Pass 1.371s
Count from 1 to 5 ✅ Pass 1.620s
Math Calculation ✅ Pass 1.671s
Basic Echo Function ✅ Pass 1.565s
Streaming Simple Math Streaming ✅ Pass 1.021s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.551s
Streaming Basic Echo Function Streaming ✅ Pass 1.436s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.148s
Search Query Function ✅ Pass 1.201s
Ask Advice Function ✅ Pass 1.688s
Streaming Search Query Function Streaming ✅ Pass 3.910s
Function Argument Memory Test ✅ Pass 1.419s
Function Response Memory Test ✅ Pass 1.205s
Basic Context Memory Test ✅ Pass 6.839s
Penetration Testing Memory with Tool Call ✅ Pass 2.682s
Cybersecurity Workflow Memory Test ✅ Pass 1.122s
Read a file, then edit it via unified diff ❌ Fail 2.188s expected the first call to be read_file, got action="edit_file"
Penetration Testing Methodology ✅ Pass 2.388s
Vulnerability Assessment Tools ✅ Pass 3.516s
SQL Injection Attack Type ✅ Pass 2.124s
Penetration Testing Framework ✅ Pass 2.319s
Web Application Security Scanner ✅ Pass 2.368s
Penetration Testing Tool Selection ✅ Pass 2.001s

Summary: 23/24 (95.83%) successful tests

Average latency: 2.135s


generator (o3-mini)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 3.148s
Text Transform Uppercase ✅ Pass 1.903s
Count from 1 to 5 ✅ Pass 1.631s
Math Calculation ✅ Pass 1.127s
Basic Echo Function ✅ Pass 1.430s
Streaming Simple Math Streaming ✅ Pass 1.430s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.397s
Streaming Basic Echo Function Streaming ✅ Pass 1.260s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.305s
Search Query Function ✅ Pass 1.771s
Ask Advice Function ✅ Pass 1.206s
Streaming Search Query Function Streaming ✅ Pass 1.727s
Basic Context Memory Test ✅ Pass 1.504s
Function Response Memory Test ✅ Pass 1.381s
Penetration Testing Memory with Tool Call ✅ Pass 2.323s
Function Argument Memory Test ✅ Pass 7.217s
Cybersecurity Workflow Memory Test ✅ Pass 1.352s
Read a file, then edit it via unified diff ✅ Pass 7.915s
Penetration Testing Methodology ✅ Pass 3.334s
Vulnerability Assessment Tools ✅ Pass 4.136s
SQL Injection Attack Type ✅ Pass 3.268s
Penetration Testing Framework ✅ Pass 1.594s
Web Application Security Scanner ✅ Pass 1.956s
Penetration Testing Tool Selection ✅ Pass 1.297s

Summary: 24/24 (100.00%) successful tests

Average latency: 2.359s


refiner (gpt-4.1)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 3.127s
Text Transform Uppercase ✅ Pass 0.854s
Count from 1 to 5 ✅ Pass 0.921s
Math Calculation ✅ Pass 0.599s
Basic Echo Function ✅ Pass 0.787s
Streaming Simple Math Streaming ✅ Pass 0.835s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.686s
Streaming Basic Echo Function Streaming ✅ Pass 0.923s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.094s
Search Query Function ✅ Pass 1.007s
Ask Advice Function ✅ Pass 0.795s
Streaming Search Query Function Streaming ✅ Pass 1.574s
Basic Context Memory Test ✅ Pass 0.887s
Function Argument Memory Test ✅ Pass 1.092s
Function Response Memory Test ✅ Pass 0.750s
Penetration Testing Memory with Tool Call ✅ Pass 1.222s
Cybersecurity Workflow Memory Test ✅ Pass 1.077s
Read a file, then edit it via unified diff ✅ Pass 2.805s
Penetration Testing Methodology ✅ Pass 0.930s
Vulnerability Assessment Tools ✅ Pass 1.961s
SQL Injection Attack Type ✅ Pass 0.697s
Penetration Testing Framework ✅ Pass 0.921s
Web Application Security Scanner ✅ Pass 0.766s
Penetration Testing Tool Selection ✅ Pass 0.931s

Summary: 24/24 (100.00%) successful tests

Average latency: 1.136s


adviser (o3-mini)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.854s
Text Transform Uppercase ✅ Pass 1.580s
Count from 1 to 5 ✅ Pass 1.507s
Math Calculation ✅ Pass 1.288s
Basic Echo Function ✅ Pass 1.047s
Streaming Simple Math Streaming ✅ Pass 2.252s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.216s
Streaming Basic Echo Function Streaming ✅ Pass 1.517s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.046s
Search Query Function ✅ Pass 1.199s
Ask Advice Function ✅ Pass 1.455s
Streaming Search Query Function Streaming ✅ Pass 0.975s
Basic Context Memory Test ✅ Pass 2.491s
Function Argument Memory Test ✅ Pass 1.167s
Function Response Memory Test ✅ Pass 1.107s
Cybersecurity Workflow Memory Test ✅ Pass 1.411s
Read a file, then edit it via unified diff ✅ Pass 4.286s
Penetration Testing Methodology ✅ Pass 1.978s
Penetration Testing Memory with Tool Call ✅ Pass 15.972s
Vulnerability Assessment Tools ✅ Pass 3.418s
SQL Injection Attack Type ✅ Pass 2.451s
Penetration Testing Framework ✅ Pass 2.078s
Web Application Security Scanner ✅ Pass 2.585s
Penetration Testing Tool Selection ✅ Pass 1.321s

Summary: 24/24 (100.00%) successful tests

Average latency: 2.425s


reflector (o3-mini)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 4.059s
Text Transform Uppercase ✅ Pass 2.572s
Count from 1 to 5 ✅ Pass 1.482s
Math Calculation ✅ Pass 1.047s
Basic Echo Function ✅ Pass 1.414s
Streaming Simple Math Streaming ✅ Pass 1.626s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.106s
Streaming Basic Echo Function Streaming ✅ Pass 2.268s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 2.897s
Search Query Function ✅ Pass 1.714s
Ask Advice Function ✅ Pass 1.650s
Streaming Search Query Function Streaming ✅ Pass 0.901s
Basic Context Memory Test ✅ Pass 1.388s
Function Argument Memory Test ✅ Pass 1.074s
Function Response Memory Test ✅ Pass 1.172s
Penetration Testing Memory with Tool Call ✅ Pass 2.452s
Cybersecurity Workflow Memory Test ✅ Pass 2.427s
Read a file, then edit it via unified diff ✅ Pass 5.565s
Penetration Testing Methodology ✅ Pass 2.198s
Vulnerability Assessment Tools ✅ Pass 2.768s
SQL Injection Attack Type ✅ Pass 2.052s
Penetration Testing Framework ✅ Pass 1.360s
Web Application Security Scanner ✅ Pass 1.889s
Penetration Testing Tool Selection ✅ Pass 1.342s

Summary: 24/24 (100.00%) successful tests

Average latency: 2.018s


searcher (gpt-4.1-mini)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.653s
Text Transform Uppercase ✅ Pass 1.217s
Math Calculation ✅ Pass 0.538s
Count from 1 to 5 ✅ Pass 4.304s
Basic Echo Function ✅ Pass 0.898s
Streaming Simple Math Streaming ✅ Pass 0.718s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.637s
Streaming Basic Echo Function Streaming ✅ Pass 1.408s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.328s
Search Query Function ✅ Pass 1.137s
Ask Advice Function ✅ Pass 1.124s
Streaming Search Query Function Streaming ✅ Pass 0.672s
Basic Context Memory Test ✅ Pass 0.982s
Function Argument Memory Test ✅ Pass 0.726s
Function Response Memory Test ✅ Pass 0.589s
Penetration Testing Memory with Tool Call ✅ Pass 2.775s
Cybersecurity Workflow Memory Test ✅ Pass 0.597s
Read a file, then edit it via unified diff ✅ Pass 5.608s
Penetration Testing Methodology ✅ Pass 1.062s
Vulnerability Assessment Tools ✅ Pass 1.697s
SQL Injection Attack Type ✅ Pass 0.712s
Penetration Testing Framework ✅ Pass 0.950s
Web Application Security Scanner ✅ Pass 1.121s
Penetration Testing Tool Selection ✅ Pass 1.309s

Summary: 24/24 (100.00%) successful tests

Average latency: 1.491s


enricher (gpt-4.1-mini)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.906s
Text Transform Uppercase ✅ Pass 0.924s
Count from 1 to 5 ✅ Pass 0.756s
Math Calculation ✅ Pass 0.769s
Basic Echo Function ✅ Pass 1.063s
Streaming Simple Math Streaming ✅ Pass 0.619s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.556s
Streaming Basic Echo Function Streaming ✅ Pass 1.273s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.995s
Search Query Function ✅ Pass 0.782s
Ask Advice Function ✅ Pass 1.336s
Streaming Search Query Function Streaming ✅ Pass 0.768s
Basic Context Memory Test ✅ Pass 0.834s
Function Argument Memory Test ✅ Pass 0.612s
Function Response Memory Test ✅ Pass 0.698s
Penetration Testing Memory with Tool Call ✅ Pass 1.485s
Cybersecurity Workflow Memory Test ✅ Pass 0.676s
Read a file, then edit it via unified diff ✅ Pass 3.228s
Penetration Testing Methodology ✅ Pass 1.038s
Vulnerability Assessment Tools ✅ Pass 1.831s
SQL Injection Attack Type ✅ Pass 0.975s
Penetration Testing Framework ✅ Pass 0.937s
Web Application Security Scanner ✅ Pass 1.424s
Penetration Testing Tool Selection ✅ Pass 0.916s

Summary: 24/24 (100.00%) successful tests

Average latency: 1.101s


coder (gpt-4.1)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.828s
Text Transform Uppercase ✅ Pass 0.856s
Count from 1 to 5 ✅ Pass 0.930s
Math Calculation ✅ Pass 0.617s
Basic Echo Function ✅ Pass 0.743s
Streaming Simple Math Streaming ✅ Pass 0.584s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.661s
Streaming Basic Echo Function Streaming ✅ Pass 1.033s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.146s
Search Query Function ✅ Pass 1.128s
Ask Advice Function ✅ Pass 0.960s
Streaming Search Query Function Streaming ✅ Pass 2.265s
Basic Context Memory Test ✅ Pass 0.779s
Function Argument Memory Test ✅ Pass 0.584s
Function Response Memory Test ✅ Pass 1.038s
Penetration Testing Memory with Tool Call ✅ Pass 1.155s
Cybersecurity Workflow Memory Test ✅ Pass 1.005s
Read a file, then edit it via unified diff ✅ Pass 2.478s
Penetration Testing Methodology ✅ Pass 1.631s
Vulnerability Assessment Tools ✅ Pass 2.536s
SQL Injection Attack Type ✅ Pass 0.665s
Penetration Testing Framework ✅ Pass 0.921s
Web Application Security Scanner ✅ Pass 0.727s
Penetration Testing Tool Selection ✅ Pass 0.890s

Summary: 24/24 (100.00%) successful tests

Average latency: 1.090s


installer (gpt-4.1)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.566s
Text Transform Uppercase ✅ Pass 0.781s
Count from 1 to 5 ✅ Pass 0.812s
Math Calculation ✅ Pass 0.761s
Basic Echo Function ✅ Pass 1.194s
Streaming Simple Math Streaming ✅ Pass 0.725s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.017s
Streaming Basic Echo Function Streaming ✅ Pass 1.138s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.743s
Search Query Function ✅ Pass 1.143s
Ask Advice Function ✅ Pass 1.276s
Streaming Search Query Function Streaming ✅ Pass 1.655s
Basic Context Memory Test ✅ Pass 0.874s
Function Argument Memory Test ✅ Pass 0.585s
Function Response Memory Test ✅ Pass 0.579s
Penetration Testing Memory with Tool Call ✅ Pass 1.199s
Cybersecurity Workflow Memory Test ✅ Pass 0.769s
Read a file, then edit it via unified diff ✅ Pass 2.397s
Penetration Testing Methodology ✅ Pass 0.988s
Vulnerability Assessment Tools ✅ Pass 2.887s
Penetration Testing Framework ✅ Pass 1.089s
SQL Injection Attack Type ✅ Pass 3.809s
Web Application Security Scanner ✅ Pass 0.892s
Penetration Testing Tool Selection ✅ Pass 0.718s

Summary: 24/24 (100.00%) successful tests

Average latency: 1.192s


pentester (o3-mini)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.192s
Text Transform Uppercase ✅ Pass 1.077s
Count from 1 to 5 ✅ Pass 1.850s
Math Calculation ✅ Pass 0.896s
Basic Echo Function ✅ Pass 0.810s
Streaming Simple Math Streaming ✅ Pass 0.987s
Streaming Basic Echo Function Streaming ✅ Pass 0.940s
Streaming Count from 1 to 3 Streaming ✅ Pass 3.016s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.926s
Search Query Function ✅ Pass 0.849s
Ask Advice Function ✅ Pass 1.222s
Streaming Search Query Function Streaming ✅ Pass 0.960s
Basic Context Memory Test ✅ Pass 1.939s
Function Argument Memory Test ✅ Pass 0.804s
Function Response Memory Test ✅ Pass 0.827s
Penetration Testing Memory with Tool Call ✅ Pass 1.353s
Cybersecurity Workflow Memory Test ✅ Pass 1.153s
Read a file, then edit it via unified diff ❌ Fail 2.900s expected the first call to be read_file, got action="edit_file"
Penetration Testing Methodology ✅ Pass 1.118s
Vulnerability Assessment Tools ✅ Pass 2.815s
SQL Injection Attack Type ✅ Pass 2.276s
Penetration Testing Framework ✅ Pass 1.421s
Web Application Security Scanner ✅ Pass 0.949s
Penetration Testing Tool Selection ✅ Pass 1.125s

Summary: 23/24 (95.83%) successful tests

Average latency: 1.392s