Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

21.5 KB
Raw Blame

LLM Agent Testing Report

Generated: Fri, 25 Sep 2026 00:07:47 UTC

Overall Results

Agent Model Reasoning Success Rate Average Latency
simple deepseek-ai/DeepSeek-V4-Flash false 24/25 (96.00%) 3.070s
simple_json google/gemma-4-31B-it false 8/8 (100.00%) 2.699s
primary_agent deepseek-ai/DeepSeek-V4.1-Flash false 25/25 (100.00%) 3.173s
assistant deepseek-ai/DeepSeek-V4.1-Flash false 25/25 (100.00%) 2.625s
generator deepseek-ai/DeepSeek-V4.1-Flash false 25/25 (100.00%) 2.555s
refiner deepseek-ai/DeepSeek-V4.1-Flash false 25/25 (100.00%) 2.993s
adviser deepseek-ai/DeepSeek-V4.1-Flash false 16/16 (100.00%) 2.895s
reflector deepseek-ai/DeepSeek-V4-Flash false 16/16 (100.00%) 3.892s
searcher deepseek-ai/DeepSeek-V4-Flash false 25/25 (100.00%) 3.177s
enricher deepseek-ai/DeepSeek-V4.1-Flash false 25/25 (100.00%) 3.221s
coder deepseek-ai/DeepSeek-V4.1-Flash false 25/25 (100.00%) 2.894s
installer deepseek-ai/DeepSeek-V4.1-Flash false 25/25 (100.00%) 3.633s
pentester deepseek-ai/DeepSeek-V4.1-Flash false 25/25 (100.00%) 3.137s

Total: 289/290 (99.66%) successful tests Overall average latency: 3.076s

Detailed Results

simple (deepseek-ai/DeepSeek-V4-Flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.368s
Text Transform Uppercase ✅ Pass 0.651s
Count from 1 to 5 ✅ Pass 0.721s
Math Calculation ✅ Pass 0.458s
Basic Echo Function ✅ Pass 1.196s
Streaming Simple Math Streaming ✅ Pass 0.568s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.379s
Streaming Basic Echo Function Streaming ✅ Pass 1.054s
Answer Stops At The Output Limit ✅ Pass 38.147s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.656s
Search Query Function ✅ Pass 2.020s
Ask Advice Function ✅ Pass 1.255s
Streaming Search Query Function Streaming ✅ Pass 1.594s
Function Argument Memory Test ✅ Pass 0.710s
Function Response Memory Test ✅ Pass 0.606s
Basic Context Memory Test ✅ Pass 1.963s
Cybersecurity Workflow Memory Test ✅ Pass 0.483s
Penetration Testing Memory with Tool Call ❌ Fail 1.119s expected function 'generate_report' not found in tool calls: expected function generate_report not found in tool calls; stop reason: tool_calls
Read a file, then edit it via unified diff ✅ Pass 5.828s
Penetration Testing Methodology ✅ Pass 2.583s
SQL Injection Attack Type ✅ Pass 0.847s
Vulnerability Assessment Tools ✅ Pass 5.752s
Penetration Testing Framework ✅ Pass 3.694s
Web Application Security Scanner ✅ Pass 1.091s
Penetration Testing Tool Selection ✅ Pass 1.991s

Summary: 24/25 (96.00%) successful tests

Average latency: 3.070s


simple_json (google/gemma-4-31B-it)

Advanced Tests

Test Result Latency Error
Vulnerability Report Memory Test ✅ Pass 3.930s
Person Information JSON ✅ Pass 1.228s
Project Information JSON ✅ Pass 1.148s
User Profile JSON ✅ Pass 7.100s
JSON Array Response Without Schema ✅ Pass 5.049s
Streaming Person Information JSON Streaming ✅ Pass 1.499s

Capability Tests

Test Capability Result Latency Note
Structured Output With JSON Schema structured_output ✅ Pass 1.637s
Structured Output Refuses A Non-Object Schema structured_output ✅ Pass 0.000s

Summary: 8/8 (100.00%) successful tests

Average latency: 2.699s


primary_agent (deepseek-ai/DeepSeek-V4.1-Flash)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 28.826s
Simple Math ✅ Pass 0.445s
Text Transform Uppercase ✅ Pass 1.652s
Count from 1 to 5 ✅ Pass 0.573s
Math Calculation ✅ Pass 0.429s
Streaming Simple Math Streaming ✅ Pass 2.493s
Basic Echo Function ✅ Pass 3.472s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.502s
Streaming Basic Echo Function Streaming ✅ Pass 0.681s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.167s
Search Query Function ✅ Pass 1.265s
Ask Advice Function ✅ Pass 0.698s
Streaming Search Query Function Streaming ✅ Pass 1.065s
Basic Context Memory Test ✅ Pass 1.130s
Function Argument Memory Test ✅ Pass 1.472s
Function Response Memory Test ✅ Pass 0.776s
Cybersecurity Workflow Memory Test ✅ Pass 1.440s
Penetration Testing Memory with Tool Call ✅ Pass 1.486s
Read a file, then edit it via unified diff ✅ Pass 10.336s
Penetration Testing Methodology ✅ Pass 8.468s
Vulnerability Assessment Tools ✅ Pass 7.797s
SQL Injection Attack Type ✅ Pass 0.615s
Penetration Testing Framework ✅ Pass 0.759s
Web Application Security Scanner ✅ Pass 0.772s
Penetration Testing Tool Selection ✅ Pass 0.998s

Summary: 25/25 (100.00%) successful tests

Average latency: 3.173s


assistant (deepseek-ai/DeepSeek-V4.1-Flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.845s
Text Transform Uppercase ✅ Pass 0.985s
Count from 1 to 5 ✅ Pass 0.490s
Math Calculation ✅ Pass 0.544s
Basic Echo Function ✅ Pass 0.921s
Streaming Simple Math Streaming ✅ Pass 1.298s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.446s
Streaming Basic Echo Function Streaming ✅ Pass 1.175s
Answer Stops At The Output Limit ✅ Pass 29.539s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 1.530s
JSON Response Function ✅ Pass 1.577s
Streaming Search Query Function Streaming ✅ Pass 0.890s
Basic Context Memory Test ✅ Pass 1.065s
Function Argument Memory Test ✅ Pass 0.502s
Ask Advice Function ✅ Pass 2.825s
Function Response Memory Test ✅ Pass 0.840s
Cybersecurity Workflow Memory Test ✅ Pass 0.808s
Penetration Testing Memory with Tool Call ✅ Pass 1.354s
Read a file, then edit it via unified diff ✅ Pass 6.801s
Penetration Testing Methodology ✅ Pass 3.505s
SQL Injection Attack Type ✅ Pass 0.516s
Vulnerability Assessment Tools ✅ Pass 3.340s
Penetration Testing Framework ✅ Pass 0.713s
Web Application Security Scanner ✅ Pass 1.471s
Penetration Testing Tool Selection ✅ Pass 1.631s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.625s


generator (deepseek-ai/DeepSeek-V4.1-Flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.474s
Text Transform Uppercase ✅ Pass 0.439s
Count from 1 to 5 ✅ Pass 0.478s
Math Calculation ✅ Pass 0.509s
Basic Echo Function ✅ Pass 0.752s
Streaming Simple Math Streaming ✅ Pass 0.460s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.176s
Streaming Basic Echo Function Streaming ✅ Pass 1.141s
Answer Stops At The Output Limit ✅ Pass 32.333s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 2.381s
JSON Response Function ✅ Pass 2.646s
Ask Advice Function ✅ Pass 0.780s
Basic Context Memory Test ✅ Pass 0.608s
Streaming Search Query Function Streaming ✅ Pass 1.221s
Function Response Memory Test ✅ Pass 0.871s
Function Argument Memory Test ✅ Pass 0.914s
Cybersecurity Workflow Memory Test ✅ Pass 0.669s
Penetration Testing Memory with Tool Call ✅ Pass 1.471s
Read a file, then edit it via unified diff ✅ Pass 5.597s
Penetration Testing Methodology ✅ Pass 2.498s
SQL Injection Attack Type ✅ Pass 0.412s
Vulnerability Assessment Tools ✅ Pass 2.478s
Web Application Security Scanner ✅ Pass 0.723s
Penetration Testing Framework ✅ Pass 1.511s
Penetration Testing Tool Selection ✅ Pass 1.333s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.555s


refiner (deepseek-ai/DeepSeek-V4.1-Flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.517s
Text Transform Uppercase ✅ Pass 0.433s
Count from 1 to 5 ✅ Pass 1.494s
Math Calculation ✅ Pass 0.742s
Basic Echo Function ✅ Pass 0.748s
Streaming Simple Math Streaming ✅ Pass 0.503s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.672s
Streaming Basic Echo Function Streaming ✅ Pass 1.822s
Answer Stops At The Output Limit ✅ Pass 35.430s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 1.502s
JSON Response Function ✅ Pass 2.529s
Streaming Search Query Function Streaming ✅ Pass 0.807s
Ask Advice Function ✅ Pass 1.385s
Basic Context Memory Test ✅ Pass 0.974s
Function Response Memory Test ✅ Pass 0.469s
Function Argument Memory Test ✅ Pass 1.130s
Cybersecurity Workflow Memory Test ✅ Pass 0.665s
Penetration Testing Memory with Tool Call ✅ Pass 2.849s
Read a file, then edit it via unified diff ✅ Pass 9.630s
Penetration Testing Methodology ✅ Pass 1.505s
SQL Injection Attack Type ✅ Pass 0.709s
Vulnerability Assessment Tools ✅ Pass 3.307s
Web Application Security Scanner ✅ Pass 0.583s
Penetration Testing Framework ✅ Pass 3.127s
Penetration Testing Tool Selection ✅ Pass 1.287s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.993s


adviser (deepseek-ai/DeepSeek-V4.1-Flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.577s
Text Transform Uppercase ✅ Pass 0.490s
Count from 1 to 5 ✅ Pass 0.504s
Math Calculation ✅ Pass 0.507s
Streaming Simple Math Streaming ✅ Pass 0.549s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.450s
Answer Stops At The Output Limit ✅ Pass 32.386s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 0.599s
Function Argument Memory Test ✅ Pass 0.769s
Function Response Memory Test ✅ Pass 0.909s
Cybersecurity Workflow Memory Test ✅ Pass 2.599s
Penetration Testing Methodology ✅ Pass 2.471s
Vulnerability Assessment Tools ✅ Pass 1.884s
SQL Injection Attack Type ✅ Pass 0.432s
Penetration Testing Framework ✅ Pass 0.427s
Web Application Security Scanner ✅ Pass 0.757s

Summary: 16/16 (100.00%) successful tests

Average latency: 2.895s


reflector (deepseek-ai/DeepSeek-V4-Flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.365s
Text Transform Uppercase ✅ Pass 0.455s
Count from 1 to 5 ✅ Pass 0.989s
Math Calculation ✅ Pass 1.298s
Streaming Simple Math Streaming ✅ Pass 0.505s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.962s
Answer Stops At The Output Limit ✅ Pass 37.503s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 0.940s
Function Argument Memory Test ✅ Pass 0.309s
Function Response Memory Test ✅ Pass 0.655s
Cybersecurity Workflow Memory Test ✅ Pass 0.650s
Penetration Testing Methodology ✅ Pass 1.782s
SQL Injection Attack Type ✅ Pass 0.505s
Penetration Testing Framework ✅ Pass 1.815s
Web Application Security Scanner ✅ Pass 0.940s
Vulnerability Assessment Tools ✅ Pass 12.588s

Summary: 16/16 (100.00%) successful tests

Average latency: 3.892s


searcher (deepseek-ai/DeepSeek-V4-Flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.324s
Text Transform Uppercase ✅ Pass 0.334s
Count from 1 to 5 ✅ Pass 0.543s
Math Calculation ✅ Pass 0.304s
Basic Echo Function ✅ Pass 0.958s
Streaming Simple Math Streaming ✅ Pass 0.305s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.487s
Streaming Basic Echo Function Streaming ✅ Pass 1.302s
Answer Stops At The Output Limit ✅ Pass 51.153s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 0.861s
JSON Response Function ✅ Pass 1.813s
Ask Advice Function ✅ Pass 1.758s
Basic Context Memory Test ✅ Pass 0.467s
Streaming Search Query Function Streaming ✅ Pass 1.488s
Function Argument Memory Test ✅ Pass 0.732s
Function Response Memory Test ✅ Pass 0.644s
Cybersecurity Workflow Memory Test ✅ Pass 0.395s
Penetration Testing Memory with Tool Call ✅ Pass 1.960s
Read a file, then edit it via unified diff ✅ Pass 3.725s
Penetration Testing Methodology ✅ Pass 0.776s
Vulnerability Assessment Tools ✅ Pass 3.933s
SQL Injection Attack Type ✅ Pass 0.621s
Penetration Testing Framework ✅ Pass 1.394s
Web Application Security Scanner ✅ Pass 0.592s
Penetration Testing Tool Selection ✅ Pass 1.547s

Summary: 25/25 (100.00%) successful tests

Average latency: 3.177s


enricher (deepseek-ai/DeepSeek-V4.1-Flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.552s
Text Transform Uppercase ✅ Pass 0.345s
Count from 1 to 5 ✅ Pass 0.798s
Math Calculation ✅ Pass 0.603s
Basic Echo Function ✅ Pass 0.750s
Streaming Simple Math Streaming ✅ Pass 0.570s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.822s
Streaming Basic Echo Function Streaming ✅ Pass 1.099s
Answer Stops At The Output Limit ✅ Pass 50.441s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 2.643s
Ask Advice Function ✅ Pass 0.720s
Search Query Function ✅ Pass 2.289s
Streaming Search Query Function Streaming ✅ Pass 1.397s
Basic Context Memory Test ✅ Pass 1.014s
Function Argument Memory Test ✅ Pass 1.062s
Function Response Memory Test ✅ Pass 1.081s
Penetration Testing Memory with Tool Call ✅ Pass 1.262s
Cybersecurity Workflow Memory Test ✅ Pass 1.222s
Read a file, then edit it via unified diff ✅ Pass 2.980s
Penetration Testing Methodology ✅ Pass 1.489s
SQL Injection Attack Type ✅ Pass 0.481s
Vulnerability Assessment Tools ✅ Pass 1.984s
Penetration Testing Framework ✅ Pass 0.753s
Web Application Security Scanner ✅ Pass 1.274s
Penetration Testing Tool Selection ✅ Pass 2.885s

Summary: 25/25 (100.00%) successful tests

Average latency: 3.221s


coder (deepseek-ai/DeepSeek-V4.1-Flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.548s
Text Transform Uppercase ✅ Pass 0.492s
Count from 1 to 5 ✅ Pass 0.496s
Math Calculation ✅ Pass 0.525s
Basic Echo Function ✅ Pass 1.114s
Streaming Simple Math Streaming ✅ Pass 0.416s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.459s
Streaming Basic Echo Function Streaming ✅ Pass 1.132s
Answer Stops At The Output Limit ✅ Pass 46.224s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.924s
Search Query Function ✅ Pass 0.977s
Ask Advice Function ✅ Pass 0.732s
Streaming Search Query Function Streaming ✅ Pass 1.057s
Basic Context Memory Test ✅ Pass 0.632s
Function Argument Memory Test ✅ Pass 0.731s
Function Response Memory Test ✅ Pass 0.807s
Penetration Testing Memory with Tool Call ✅ Pass 1.192s
Cybersecurity Workflow Memory Test ✅ Pass 0.931s
Read a file, then edit it via unified diff ✅ Pass 5.438s
Penetration Testing Methodology ✅ Pass 1.456s
SQL Injection Attack Type ✅ Pass 0.470s
Vulnerability Assessment Tools ✅ Pass 0.839s
Penetration Testing Framework ✅ Pass 1.272s
Penetration Testing Tool Selection ✅ Pass 1.044s
Web Application Security Scanner ✅ Pass 2.435s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.894s


installer (deepseek-ai/DeepSeek-V4.1-Flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.402s
Text Transform Uppercase ✅ Pass 0.487s
Count from 1 to 5 ✅ Pass 0.558s
Math Calculation ✅ Pass 1.064s
Basic Echo Function ✅ Pass 2.027s
Streaming Simple Math Streaming ✅ Pass 0.835s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.386s
Streaming Basic Echo Function Streaming ✅ Pass 1.276s
Answer Stops At The Output Limit ✅ Pass 49.310s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 0.748s
JSON Response Function ✅ Pass 0.841s
Streaming Search Query Function Streaming ✅ Pass 1.098s
Ask Advice Function ✅ Pass 1.723s
Basic Context Memory Test ✅ Pass 0.806s
Function Response Memory Test ✅ Pass 0.488s
Function Argument Memory Test ✅ Pass 1.184s
Cybersecurity Workflow Memory Test ✅ Pass 1.191s
Penetration Testing Memory with Tool Call ✅ Pass 4.060s
Read a file, then edit it via unified diff ✅ Pass 11.418s
Vulnerability Assessment Tools ✅ Pass 1.693s
SQL Injection Attack Type ✅ Pass 1.171s
Penetration Testing Methodology ✅ Pass 4.311s
Web Application Security Scanner ✅ Pass 0.436s
Penetration Testing Framework ✅ Pass 1.778s
Penetration Testing Tool Selection ✅ Pass 1.528s

Summary: 25/25 (100.00%) successful tests

Average latency: 3.633s


pentester (deepseek-ai/DeepSeek-V4.1-Flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.444s
Text Transform Uppercase ✅ Pass 0.491s
Count from 1 to 5 ✅ Pass 0.478s
Math Calculation ✅ Pass 0.416s
Basic Echo Function ✅ Pass 0.694s
Streaming Simple Math Streaming ✅ Pass 1.427s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.486s
Streaming Basic Echo Function Streaming ✅ Pass 2.485s
Answer Stops At The Output Limit ✅ Pass 41.521s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.041s
Search Query Function ✅ Pass 0.976s
Streaming Search Query Function Streaming ✅ Pass 1.018s
Ask Advice Function ✅ Pass 1.793s
Basic Context Memory Test ✅ Pass 0.783s
Function Argument Memory Test ✅ Pass 0.710s
Function Response Memory Test ✅ Pass 0.579s
Cybersecurity Workflow Memory Test ✅ Pass 0.710s
Penetration Testing Memory with Tool Call ✅ Pass 2.117s
Read a file, then edit it via unified diff ✅ Pass 10.713s
Penetration Testing Methodology ✅ Pass 2.313s
SQL Injection Attack Type ✅ Pass 0.549s
Penetration Testing Framework ✅ Pass 0.958s
Vulnerability Assessment Tools ✅ Pass 2.642s
Web Application Security Scanner ✅ Pass 0.971s
Penetration Testing Tool Selection ✅ Pass 2.094s

Summary: 25/25 (100.00%) successful tests

Average latency: 3.137s