Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

21.2 KB
Raw Blame

LLM Agent Testing Report

Generated: Sat, 26 Sep 2026 12:50:38 UTC

Overall Results

Agent Model Reasoning Success Rate Average Latency
simple gemini-3.1-flash-lite false 25/25 (100.00%) 0.916s
simple_json gemini-3.1-flash-lite false 8/8 (100.00%) 0.783s
primary_agent gemini-3.5-flash-lite true 25/25 (100.00%) 2.728s
assistant gemini-3.5-flash-lite true 25/25 (100.00%) 2.341s
generator gemini-3.5-flash-lite true 25/25 (100.00%) 2.748s
refiner gemini-3.5-flash-lite true 24/25 (96.00%) 2.621s
adviser gemini-3.5-flash-lite true 16/16 (100.00%) 2.985s
reflector gemini-3.1-flash-lite false 16/16 (100.00%) 0.923s
searcher gemini-3.1-flash-lite false 25/25 (100.00%) 0.875s
enricher gemini-3.1-flash-lite false 25/25 (100.00%) 0.921s
coder gemini-3.5-flash-lite true 25/25 (100.00%) 2.387s
installer gemini-3.5-flash-lite true 25/25 (100.00%) 2.383s
pentester gemini-3.5-flash-lite true 25/25 (100.00%) 2.426s

Total: 289/290 (99.66%) successful tests Overall average latency: 1.991s

Detailed Results

simple (gemini-3.1-flash-lite)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.753s
Math Calculation ✅ Pass 0.815s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.826s
Count from 1 to 5 ✅ Pass 1.041s
Basic Echo Function ✅ Pass 1.041s
Text Transform Uppercase ✅ Pass 1.041s
Streaming Simple Math Streaming ✅ Pass 1.044s
Streaming Basic Echo Function Streaming ✅ Pass 0.875s
Answer Stops At The Output Limit ✅ Pass 1.942s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.622s
Ask Advice Function ✅ Pass 0.633s
Search Query Function ✅ Pass 0.756s
Streaming Search Query Function Streaming ✅ Pass 0.635s
Basic Context Memory Test ✅ Pass 0.695s
Function Argument Memory Test ✅ Pass 0.626s
Function Response Memory Test ✅ Pass 0.750s
Cybersecurity Workflow Memory Test ✅ Pass 0.750s
Penetration Testing Memory with Tool Call ✅ Pass 0.811s
Read a file, then edit it via unified diff ✅ Pass 1.935s
Penetration Testing Methodology ✅ Pass 0.874s
SQL Injection Attack Type ✅ Pass 0.751s
Vulnerability Assessment Tools ✅ Pass 1.128s
Web Application Security Scanner ✅ Pass 0.662s
Penetration Testing Tool Selection ✅ Pass 0.821s
Penetration Testing Framework ✅ Pass 1.070s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.916s


simple_json (gemini-3.1-flash-lite)

Advanced Tests

Test Result Latency Error
Vulnerability Report Memory Test ✅ Pass 1.124s
Person Information JSON ✅ Pass 0.774s
Project Information JSON ✅ Pass 0.814s
User Profile JSON ✅ Pass 0.883s
JSON Array Response Without Schema ✅ Pass 1.009s
Streaming Person Information JSON Streaming ✅ Pass 0.838s

Capability Tests

Test Capability Result Latency Note
Structured Output Refuses A Non-Object Schema structured_output ✅ Pass 0.000s
Structured Output With JSON Schema structured_output ✅ Pass 0.818s

Summary: 8/8 (100.00%) successful tests

Average latency: 0.783s


primary_agent (gemini-3.5-flash-lite)

Basic Tests

Test Result Latency Error
Streaming Count from 1 to 3 Streaming ✅ Pass 1.262s
Math Calculation ✅ Pass 2.168s
Basic Echo Function ✅ Pass 2.222s
Simple Math ✅ Pass 2.671s
Streaming Simple Math Streaming ✅ Pass 2.224s
Text Transform Uppercase ✅ Pass 2.864s
Count from 1 to 5 ✅ Pass 2.923s
Streaming Basic Echo Function Streaming ✅ Pass 1.320s
Answer Stops At The Output Limit ✅ Pass 3.957s

Advanced Tests

Test Result Latency Error
Streaming Search Query Function Streaming ✅ Pass 1.011s
JSON Response Function ✅ Pass 1.871s
Ask Advice Function ✅ Pass 2.188s
Search Query Function ✅ Pass 2.440s
Basic Context Memory Test ✅ Pass 2.126s
Function Argument Memory Test ✅ Pass 2.193s
Function Response Memory Test ✅ Pass 2.958s
Cybersecurity Workflow Memory Test ✅ Pass 2.638s
Penetration Testing Memory with Tool Call ✅ Pass 3.032s
Read a file, then edit it via unified diff ✅ Pass 4.435s
SQL Injection Attack Type ✅ Pass 2.890s
Penetration Testing Framework ✅ Pass 3.025s
Penetration Testing Tool Selection ✅ Pass 2.956s
Vulnerability Assessment Tools ✅ Pass 3.899s
Penetration Testing Methodology ✅ Pass 4.775s
Web Application Security Scanner ✅ Pass 4.148s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.728s


assistant (gemini-3.5-flash-lite)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 2.198s
Basic Echo Function ✅ Pass 1.944s
Simple Math ✅ Pass 2.642s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.192s
Math Calculation ✅ Pass 2.264s
Answer Stops At The Output Limit ✅ Pass 3.152s
Count from 1 to 5 ✅ Pass 2.631s
Streaming Simple Math Streaming ✅ Pass 1.960s
Streaming Basic Echo Function Streaming ✅ Pass 1.164s

Advanced Tests

Test Result Latency Error
Streaming Search Query Function Streaming ✅ Pass 0.631s
JSON Response Function ✅ Pass 1.714s
Search Query Function ✅ Pass 1.652s
Ask Advice Function ✅ Pass 1.653s
Basic Context Memory Test ✅ Pass 2.469s
Function Response Memory Test ✅ Pass 2.136s
Function Argument Memory Test ✅ Pass 2.759s
Cybersecurity Workflow Memory Test ✅ Pass 2.509s
Penetration Testing Memory with Tool Call ✅ Pass 2.755s
Read a file, then edit it via unified diff ✅ Pass 4.503s
Vulnerability Assessment Tools ✅ Pass 2.471s
Penetration Testing Methodology ✅ Pass 3.397s
SQL Injection Attack Type ✅ Pass 2.449s
Penetration Testing Tool Selection ✅ Pass 2.073s
Web Application Security Scanner ✅ Pass 2.693s
Penetration Testing Framework ✅ Pass 3.513s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.341s


generator (gemini-3.5-flash-lite)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 2.027s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.131s
Simple Math ✅ Pass 2.463s
Math Calculation ✅ Pass 2.077s
Basic Echo Function ✅ Pass 2.079s
Count from 1 to 5 ✅ Pass 2.776s
Streaming Simple Math Streaming ✅ Pass 2.584s
Answer Stops At The Output Limit ✅ Pass 3.420s
Streaming Basic Echo Function Streaming ✅ Pass 1.393s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.816s
Search Query Function ✅ Pass 1.998s
Streaming Search Query Function Streaming ✅ Pass 1.060s
Ask Advice Function ✅ Pass 2.757s
Function Argument Memory Test ✅ Pass 2.668s
Function Response Memory Test ✅ Pass 2.496s
Basic Context Memory Test ✅ Pass 2.858s
Cybersecurity Workflow Memory Test ✅ Pass 2.436s
Penetration Testing Memory with Tool Call ✅ Pass 3.015s
Read a file, then edit it via unified diff ✅ Pass 4.752s
Penetration Testing Tool Selection ✅ Pass 2.337s
Vulnerability Assessment Tools ✅ Pass 4.053s
SQL Injection Attack Type ✅ Pass 3.438s
Web Application Security Scanner ✅ Pass 3.402s
Penetration Testing Methodology ✅ Pass 4.856s
Penetration Testing Framework ✅ Pass 4.802s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.748s


refiner (gemini-3.5-flash-lite)

Basic Tests

Test Result Latency Error
Streaming Count from 1 to 3 Streaming ✅ Pass 1.196s
Simple Math ✅ Pass 2.329s
Text Transform Uppercase ✅ Pass 2.263s
Math Calculation ✅ Pass 1.819s
Count from 1 to 5 ✅ Pass 2.700s
Answer Stops At The Output Limit ✅ Pass 3.458s
Streaming Simple Math Streaming ✅ Pass 2.315s
Basic Echo Function ✅ Pass 2.573s
Streaming Basic Echo Function Streaming ✅ Pass 1.124s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 2.372s
JSON Response Function ✅ Pass 2.394s
Ask Advice Function ✅ Pass 1.889s
Streaming Search Query Function Streaming ✅ Pass 0.904s
Basic Context Memory Test ✅ Pass 2.490s
Cybersecurity Workflow Memory Test ✅ Pass 2.271s
Function Argument Memory Test ✅ Pass 3.059s
Penetration Testing Memory with Tool Call ✅ Pass 2.853s
Function Response Memory Test ✅ Pass 3.231s
Read a file, then edit it via unified diff ❌ Fail 4.205s expected the second call to be edit_file, got action="write_file"; stop reason: STOP
Penetration Testing Methodology ✅ Pass 4.475s
Penetration Testing Framework ✅ Pass 2.757s
Vulnerability Assessment Tools ✅ Pass 4.147s
Penetration Testing Tool Selection ✅ Pass 2.250s
SQL Injection Attack Type ✅ Pass 3.204s
Web Application Security Scanner ✅ Pass 3.234s

Summary: 24/25 (96.00%) successful tests

Average latency: 2.621s


adviser (gemini-3.5-flash-lite)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.018s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.268s
Text Transform Uppercase ✅ Pass 2.460s
Math Calculation ✅ Pass 1.831s
Streaming Simple Math Streaming ✅ Pass 2.014s
Answer Stops At The Output Limit ✅ Pass 3.139s
Count from 1 to 5 ✅ Pass 3.017s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 2.206s
Function Argument Memory Test ✅ Pass 2.534s
Function Response Memory Test ✅ Pass 3.258s
Cybersecurity Workflow Memory Test ✅ Pass 3.035s
Penetration Testing Methodology ✅ Pass 4.530s
Vulnerability Assessment Tools ✅ Pass 4.153s
Penetration Testing Framework ✅ Pass 3.959s
SQL Injection Attack Type ✅ Pass 4.224s
Web Application Security Scanner ✅ Pass 4.107s

Summary: 16/16 (100.00%) successful tests

Average latency: 2.985s


reflector (gemini-3.1-flash-lite)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 0.683s
Simple Math ✅ Pass 0.877s
Answer Stops At The Output Limit ✅ Pass 2.263s
Count from 1 to 5 ✅ Pass 0.806s
Math Calculation ✅ Pass 0.997s
Streaming Simple Math Streaming ✅ Pass 0.686s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.807s

Advanced Tests

Test Result Latency Error
Function Argument Memory Test ✅ Pass 0.626s
Basic Context Memory Test ✅ Pass 0.759s
Function Response Memory Test ✅ Pass 0.705s
Cybersecurity Workflow Memory Test ✅ Pass 0.960s
Penetration Testing Methodology ✅ Pass 0.902s
SQL Injection Attack Type ✅ Pass 0.703s
Vulnerability Assessment Tools ✅ Pass 1.265s
Web Application Security Scanner ✅ Pass 0.826s
Penetration Testing Framework ✅ Pass 0.892s

Summary: 16/16 (100.00%) successful tests

Average latency: 0.923s


searcher (gemini-3.1-flash-lite)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.752s
Text Transform Uppercase ✅ Pass 0.750s
Count from 1 to 5 ✅ Pass 0.750s
Streaming Simple Math Streaming ✅ Pass 0.754s
Math Calculation ✅ Pass 0.825s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.701s
Streaming Basic Echo Function Streaming ✅ Pass 0.625s
Basic Echo Function ✅ Pass 0.941s
Answer Stops At The Output Limit ✅ Pass 2.008s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 0.698s
JSON Response Function ✅ Pass 0.896s
Streaming Search Query Function Streaming ✅ Pass 0.724s
Ask Advice Function ✅ Pass 1.076s
Basic Context Memory Test ✅ Pass 0.688s
Function Argument Memory Test ✅ Pass 0.628s
Function Response Memory Test ✅ Pass 0.681s
Penetration Testing Memory with Tool Call ✅ Pass 0.879s
Cybersecurity Workflow Memory Test ✅ Pass 0.858s
Read a file, then edit it via unified diff ✅ Pass 1.688s
Penetration Testing Methodology ✅ Pass 0.748s
SQL Injection Attack Type ✅ Pass 0.558s
Penetration Testing Framework ✅ Pass 0.816s
Penetration Testing Tool Selection ✅ Pass 0.618s
Web Application Security Scanner ✅ Pass 0.826s
Vulnerability Assessment Tools ✅ Pass 1.372s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.875s


enricher (gemini-3.1-flash-lite)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.816s
Text Transform Uppercase ✅ Pass 0.691s
Math Calculation ✅ Pass 0.682s
Streaming Simple Math Streaming ✅ Pass 0.628s
Count from 1 to 5 ✅ Pass 0.747s
Basic Echo Function ✅ Pass 0.881s
Streaming Basic Echo Function Streaming ✅ Pass 0.627s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.763s
Answer Stops At The Output Limit ✅ Pass 2.021s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 0.687s
JSON Response Function ✅ Pass 0.921s
Ask Advice Function ✅ Pass 0.692s
Streaming Search Query Function Streaming ✅ Pass 0.943s
Function Argument Memory Test ✅ Pass 0.750s
Basic Context Memory Test ✅ Pass 1.012s
Function Response Memory Test ✅ Pass 0.735s
Penetration Testing Memory with Tool Call ✅ Pass 0.821s
Cybersecurity Workflow Memory Test ✅ Pass 0.818s
Read a file, then edit it via unified diff ✅ Pass 1.672s
SQL Injection Attack Type ✅ Pass 0.885s
Penetration Testing Methodology ✅ Pass 1.259s
Web Application Security Scanner ✅ Pass 0.752s
Penetration Testing Framework ✅ Pass 0.933s
Vulnerability Assessment Tools ✅ Pass 1.264s
Penetration Testing Tool Selection ✅ Pass 1.001s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.921s


coder (gemini-3.5-flash-lite)

Basic Tests

Test Result Latency Error
Streaming Count from 1 to 3 Streaming ✅ Pass 1.148s
Simple Math ✅ Pass 2.161s
Text Transform Uppercase ✅ Pass 2.100s
Basic Echo Function ✅ Pass 1.723s
Math Calculation ✅ Pass 2.043s
Streaming Simple Math Streaming ✅ Pass 1.925s
Count from 1 to 5 ✅ Pass 2.551s
Streaming Basic Echo Function Streaming ✅ Pass 0.826s
Answer Stops At The Output Limit ✅ Pass 3.305s

Advanced Tests

Test Result Latency Error
Streaming Search Query Function Streaming ✅ Pass 0.818s
JSON Response Function ✅ Pass 1.636s
Ask Advice Function ✅ Pass 1.828s
Search Query Function ✅ Pass 2.462s
Function Argument Memory Test ✅ Pass 2.290s
Penetration Testing Memory with Tool Call ✅ Pass 2.459s
Basic Context Memory Test ✅ Pass 2.713s
Function Response Memory Test ✅ Pass 3.020s
Cybersecurity Workflow Memory Test ✅ Pass 2.521s
Read a file, then edit it via unified diff ✅ Pass 4.767s
Penetration Testing Tool Selection ✅ Pass 2.117s
Web Application Security Scanner ✅ Pass 2.389s
Penetration Testing Framework ✅ Pass 2.570s
SQL Injection Attack Type ✅ Pass 2.884s
Vulnerability Assessment Tools ✅ Pass 3.520s
Penetration Testing Methodology ✅ Pass 3.882s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.387s


installer (gemini-3.5-flash-lite)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 2.014s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.123s
Simple Math ✅ Pass 2.398s
Math Calculation ✅ Pass 2.190s
Count from 1 to 5 ✅ Pass 2.452s
Streaming Simple Math Streaming ✅ Pass 1.883s
Basic Echo Function ✅ Pass 2.269s
Streaming Basic Echo Function Streaming ✅ Pass 1.192s
Answer Stops At The Output Limit ✅ Pass 3.522s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.963s
Streaming Search Query Function Streaming ✅ Pass 0.753s
Search Query Function ✅ Pass 2.197s
Ask Advice Function ✅ Pass 1.691s
Basic Context Memory Test ✅ Pass 2.070s
Function Argument Memory Test ✅ Pass 2.821s
Cybersecurity Workflow Memory Test ✅ Pass 2.761s
Penetration Testing Memory with Tool Call ✅ Pass 2.825s
Function Response Memory Test ✅ Pass 2.940s
Read a file, then edit it via unified diff ✅ Pass 3.758s
Vulnerability Assessment Tools ✅ Pass 2.943s
Penetration Testing Methodology ✅ Pass 3.694s
SQL Injection Attack Type ✅ Pass 2.253s
Penetration Testing Tool Selection ✅ Pass 1.810s
Web Application Security Scanner ✅ Pass 2.512s
Penetration Testing Framework ✅ Pass 3.521s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.383s


pentester (gemini-3.5-flash-lite)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.953s
Basic Echo Function ✅ Pass 1.627s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.015s
Math Calculation ✅ Pass 2.011s
Streaming Basic Echo Function Streaming ✅ Pass 0.684s
Text Transform Uppercase ✅ Pass 2.576s
Answer Stops At The Output Limit ✅ Pass 3.149s
Streaming Simple Math Streaming ✅ Pass 2.272s
Count from 1 to 5 ✅ Pass 2.904s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.821s
Streaming Search Query Function Streaming ✅ Pass 0.745s
Ask Advice Function ✅ Pass 1.880s
Search Query Function ✅ Pass 2.447s
Basic Context Memory Test ✅ Pass 1.756s
Function Argument Memory Test ✅ Pass 2.128s
Cybersecurity Workflow Memory Test ✅ Pass 2.303s
Function Response Memory Test ✅ Pass 2.932s
Penetration Testing Memory with Tool Call ✅ Pass 3.196s
Read a file, then edit it via unified diff ✅ Pass 4.134s
SQL Injection Attack Type ✅ Pass 2.590s
Vulnerability Assessment Tools ✅ Pass 2.963s
Penetration Testing Tool Selection ✅ Pass 2.334s
Web Application Security Scanner ✅ Pass 2.850s
Penetration Testing Framework ✅ Pass 3.840s
Penetration Testing Methodology ✅ Pass 4.521s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.426s