Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

21.3 KB
Raw Blame

LLM Agent Testing Report

Generated: Tue, 04 Aug 2026 20:10:25 UTC

Overall Results

Agent Model Reasoning Success Rate Average Latency
simple mimo-v2.5 true 24/24 (100.00%) 3.713s
simple_json mimo-v2.5 false 7/7 (100.00%) 2.993s
primary_agent deepseek-v4-pro true 24/24 (100.00%) 2.459s
assistant deepseek-v4-pro true 24/24 (100.00%) 2.577s
generator deepseek-v4-pro true 24/24 (100.00%) 2.544s
refiner deepseek-v4-pro true 24/24 (100.00%) 2.571s
adviser qwen3.7-plus true 24/24 (100.00%) 6.186s
reflector deepseek-v4-flash true 24/24 (100.00%) 1.789s
searcher deepseek-v4-flash true 24/24 (100.00%) 1.782s
enricher deepseek-v4-flash true 24/24 (100.00%) 1.697s
coder minimax-m3 true 23/24 (95.83%) 1.892s
installer deepseek-v4-flash true 24/24 (100.00%) 1.906s
pentester deepseek-v4-flash true 24/24 (100.00%) 2.008s

Total: 294/295 (99.66%) successful tests Overall average latency: 2.603s

Detailed Results

simple (mimo-v2.5)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 2.843s
Simple Math ✅ Pass 7.262s
Count from 1 to 5 ✅ Pass 2.157s
Math Calculation ✅ Pass 1.826s
Basic Echo Function ✅ Pass 1.861s
Streaming Simple Math Streaming ✅ Pass 3.729s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.266s
Streaming Basic Echo Function Streaming ✅ Pass 2.107s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.934s
Search Query Function ✅ Pass 1.953s
Ask Advice Function ✅ Pass 5.166s
Streaming Search Query Function Streaming ✅ Pass 7.402s
Basic Context Memory Test ✅ Pass 4.311s
Function Argument Memory Test ✅ Pass 1.958s
Function Response Memory Test ✅ Pass 1.940s
Penetration Testing Memory with Tool Call ✅ Pass 3.350s
Cybersecurity Workflow Memory Test ✅ Pass 4.064s
Read a file, then edit it via unified diff ✅ Pass 8.963s
Penetration Testing Methodology ✅ Pass 4.230s
Vulnerability Assessment Tools ✅ Pass 6.164s
SQL Injection Attack Type ✅ Pass 4.165s
Penetration Testing Framework ✅ Pass 3.052s
Web Application Security Scanner ✅ Pass 3.987s
Penetration Testing Tool Selection ✅ Pass 3.404s

Summary: 24/24 (100.00%) successful tests

Average latency: 3.713s


simple_json (mimo-v2.5)

Advanced Tests

Test Result Latency Error
Vulnerability Report Memory Test ✅ Pass 4.877s
Person Information JSON ✅ Pass 2.944s
Project Information JSON ✅ Pass 2.369s
User Profile JSON ✅ Pass 3.358s
JSON Array Response Without Schema ✅ Pass 3.220s
Streaming Person Information JSON Streaming ✅ Pass 3.122s

Capability Tests

Test Capability Result Latency Note
Structured Output With JSON Schema structured_output ✅ Pass 1.060s

Summary: 7/7 (100.00%) successful tests

Average latency: 2.993s


primary_agent (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.190s
Text Transform Uppercase ✅ Pass 1.829s
Count from 1 to 5 ✅ Pass 1.928s
Math Calculation ✅ Pass 1.894s
Basic Echo Function ✅ Pass 1.851s
Streaming Simple Math Streaming ✅ Pass 1.585s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.695s
Streaming Basic Echo Function Streaming ✅ Pass 2.054s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 2.716s
Search Query Function ✅ Pass 2.367s
Ask Advice Function ✅ Pass 2.249s
Streaming Search Query Function Streaming ✅ Pass 1.819s
Basic Context Memory Test ✅ Pass 2.530s
Function Argument Memory Test ✅ Pass 1.934s
Function Response Memory Test ✅ Pass 2.199s
Penetration Testing Memory with Tool Call ✅ Pass 2.946s
Cybersecurity Workflow Memory Test ✅ Pass 2.406s
Read a file, then edit it via unified diff ✅ Pass 5.338s
Penetration Testing Methodology ✅ Pass 3.498s
Vulnerability Assessment Tools ✅ Pass 4.697s
SQL Injection Attack Type ✅ Pass 2.377s
Penetration Testing Framework ✅ Pass 2.428s
Web Application Security Scanner ✅ Pass 2.540s
Penetration Testing Tool Selection ✅ Pass 1.924s

Summary: 24/24 (100.00%) successful tests

Average latency: 2.459s


assistant (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.139s
Text Transform Uppercase ✅ Pass 2.014s
Count from 1 to 5 ✅ Pass 1.832s
Math Calculation ✅ Pass 1.560s
Basic Echo Function ✅ Pass 1.986s
Streaming Simple Math Streaming ✅ Pass 1.981s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.861s
Streaming Basic Echo Function Streaming ✅ Pass 2.498s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 2.482s
Search Query Function ✅ Pass 2.020s
Ask Advice Function ✅ Pass 2.770s
Streaming Search Query Function Streaming ✅ Pass 1.853s
Basic Context Memory Test ✅ Pass 2.040s
Function Argument Memory Test ✅ Pass 2.032s
Function Response Memory Test ✅ Pass 2.106s
Penetration Testing Memory with Tool Call ✅ Pass 2.957s
Cybersecurity Workflow Memory Test ✅ Pass 2.265s
Read a file, then edit it via unified diff ✅ Pass 5.237s
Penetration Testing Methodology ✅ Pass 3.523s
Vulnerability Assessment Tools ✅ Pass 5.750s
SQL Injection Attack Type ✅ Pass 3.153s
Penetration Testing Framework ✅ Pass 3.118s
Web Application Security Scanner ✅ Pass 2.198s
Penetration Testing Tool Selection ✅ Pass 2.471s

Summary: 24/24 (100.00%) successful tests

Average latency: 2.577s


generator (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.088s
Text Transform Uppercase ✅ Pass 1.740s
Count from 1 to 5 ✅ Pass 2.220s
Math Calculation ✅ Pass 1.708s
Basic Echo Function ✅ Pass 1.918s
Streaming Simple Math Streaming ✅ Pass 1.807s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.792s
Streaming Basic Echo Function Streaming ✅ Pass 1.970s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.998s
Search Query Function ✅ Pass 1.964s
Ask Advice Function ✅ Pass 2.670s
Streaming Search Query Function Streaming ✅ Pass 1.780s
Basic Context Memory Test ✅ Pass 3.211s
Function Argument Memory Test ✅ Pass 2.128s
Function Response Memory Test ✅ Pass 2.162s
Penetration Testing Memory with Tool Call ✅ Pass 2.919s
Cybersecurity Workflow Memory Test ✅ Pass 1.929s
Read a file, then edit it via unified diff ✅ Pass 4.732s
Penetration Testing Methodology ✅ Pass 3.777s
Vulnerability Assessment Tools ✅ Pass 6.549s
SQL Injection Attack Type ✅ Pass 2.130s
Penetration Testing Framework ✅ Pass 2.684s
Web Application Security Scanner ✅ Pass 2.727s
Penetration Testing Tool Selection ✅ Pass 2.440s

Summary: 24/24 (100.00%) successful tests

Average latency: 2.544s


refiner (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.062s
Text Transform Uppercase ✅ Pass 1.854s
Count from 1 to 5 ✅ Pass 1.989s
Math Calculation ✅ Pass 1.321s
Basic Echo Function ✅ Pass 1.748s
Streaming Simple Math Streaming ✅ Pass 1.654s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.947s
Streaming Basic Echo Function Streaming ✅ Pass 1.812s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 2.266s
Search Query Function ✅ Pass 1.919s
Ask Advice Function ✅ Pass 2.668s
Streaming Search Query Function Streaming ✅ Pass 1.946s
Basic Context Memory Test ✅ Pass 2.427s
Function Argument Memory Test ✅ Pass 1.932s
Function Response Memory Test ✅ Pass 1.962s
Penetration Testing Memory with Tool Call ✅ Pass 3.524s
Cybersecurity Workflow Memory Test ✅ Pass 1.940s
Read a file, then edit it via unified diff ✅ Pass 5.215s
Penetration Testing Methodology ✅ Pass 3.689s
Vulnerability Assessment Tools ✅ Pass 6.487s
SQL Injection Attack Type ✅ Pass 3.090s
Penetration Testing Framework ✅ Pass 3.394s
Web Application Security Scanner ✅ Pass 2.363s
Penetration Testing Tool Selection ✅ Pass 2.492s

Summary: 24/24 (100.00%) successful tests

Average latency: 2.571s


adviser (qwen3.7-plus)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 3.453s
Text Transform Uppercase ✅ Pass 6.036s
Count from 1 to 5 ✅ Pass 5.461s
Math Calculation ✅ Pass 4.543s
Basic Echo Function ✅ Pass 5.292s
Streaming Simple Math Streaming ✅ Pass 3.027s
Streaming Basic Echo Function Streaming ✅ Pass 4.731s
Streaming Count from 1 to 3 Streaming ✅ Pass 8.427s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 2.548s
Ask Advice Function ✅ Pass 6.060s
Streaming Search Query Function Streaming ✅ Pass 4.008s
Basic Context Memory Test ✅ Pass 4.596s
Search Query Function ✅ Pass 16.296s
Function Argument Memory Test ✅ Pass 4.500s
Function Response Memory Test ✅ Pass 5.568s
Cybersecurity Workflow Memory Test ✅ Pass 4.007s
Penetration Testing Memory with Tool Call ✅ Pass 9.534s
Read a file, then edit it via unified diff ✅ Pass 5.928s
Penetration Testing Methodology ✅ Pass 7.995s
Vulnerability Assessment Tools ✅ Pass 11.838s
SQL Injection Attack Type ✅ Pass 9.801s
Penetration Testing Framework ✅ Pass 7.005s
Web Application Security Scanner ✅ Pass 5.210s
Penetration Testing Tool Selection ✅ Pass 2.585s

Summary: 24/24 (100.00%) successful tests

Average latency: 6.186s


reflector (deepseek-v4-flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.728s
Text Transform Uppercase ✅ Pass 3.622s
Count from 1 to 5 ✅ Pass 1.807s
Math Calculation ✅ Pass 1.388s
Basic Echo Function ✅ Pass 1.791s
Streaming Simple Math Streaming ✅ Pass 1.440s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.345s
Streaming Basic Echo Function Streaming ✅ Pass 1.886s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.424s
Search Query Function ✅ Pass 1.430s
Ask Advice Function ✅ Pass 1.501s
Streaming Search Query Function Streaming ✅ Pass 1.633s
Basic Context Memory Test ✅ Pass 1.176s
Function Argument Memory Test ✅ Pass 1.317s
Function Response Memory Test ✅ Pass 1.435s
Penetration Testing Memory with Tool Call ✅ Pass 2.016s
Cybersecurity Workflow Memory Test ✅ Pass 1.374s
Read a file, then edit it via unified diff ✅ Pass 4.120s
Penetration Testing Methodology ✅ Pass 1.528s
Vulnerability Assessment Tools ✅ Pass 3.710s
SQL Injection Attack Type ✅ Pass 1.378s
Penetration Testing Framework ✅ Pass 1.265s
Web Application Security Scanner ✅ Pass 1.144s
Penetration Testing Tool Selection ✅ Pass 1.473s

Summary: 24/24 (100.00%) successful tests

Average latency: 1.789s


searcher (deepseek-v4-flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.503s
Text Transform Uppercase ✅ Pass 3.224s
Count from 1 to 5 ✅ Pass 1.959s
Math Calculation ✅ Pass 1.419s
Basic Echo Function ✅ Pass 1.743s
Streaming Simple Math Streaming ✅ Pass 1.493s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.151s
Streaming Basic Echo Function Streaming ✅ Pass 1.544s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.301s
Search Query Function ✅ Pass 1.534s
Ask Advice Function ✅ Pass 1.379s
Streaming Search Query Function Streaming ✅ Pass 1.606s
Basic Context Memory Test ✅ Pass 1.167s
Function Argument Memory Test ✅ Pass 1.551s
Function Response Memory Test ✅ Pass 1.303s
Penetration Testing Memory with Tool Call ✅ Pass 1.799s
Cybersecurity Workflow Memory Test ✅ Pass 1.664s
Read a file, then edit it via unified diff ✅ Pass 3.595s
Penetration Testing Methodology ✅ Pass 1.626s
Vulnerability Assessment Tools ✅ Pass 4.019s
SQL Injection Attack Type ✅ Pass 1.457s
Penetration Testing Framework ✅ Pass 1.149s
Web Application Security Scanner ✅ Pass 1.164s
Penetration Testing Tool Selection ✅ Pass 1.409s

Summary: 24/24 (100.00%) successful tests

Average latency: 1.782s


enricher (deepseek-v4-flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.617s
Text Transform Uppercase ✅ Pass 2.610s
Count from 1 to 5 ✅ Pass 1.770s
Math Calculation ✅ Pass 1.612s
Basic Echo Function ✅ Pass 1.670s
Streaming Simple Math Streaming ✅ Pass 1.229s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.867s
Streaming Basic Echo Function Streaming ✅ Pass 1.760s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.479s
Search Query Function ✅ Pass 1.527s
Ask Advice Function ✅ Pass 1.515s
Streaming Search Query Function Streaming ✅ Pass 1.670s
Basic Context Memory Test ✅ Pass 1.797s
Function Argument Memory Test ✅ Pass 1.206s
Function Response Memory Test ✅ Pass 1.115s
Penetration Testing Memory with Tool Call ✅ Pass 1.794s
Cybersecurity Workflow Memory Test ✅ Pass 1.449s
Read a file, then edit it via unified diff ✅ Pass 3.797s
Penetration Testing Methodology ✅ Pass 1.360s
Vulnerability Assessment Tools ✅ Pass 2.952s
SQL Injection Attack Type ✅ Pass 1.278s
Penetration Testing Framework ✅ Pass 1.183s
Web Application Security Scanner ✅ Pass 1.097s
Penetration Testing Tool Selection ✅ Pass 1.365s

Summary: 24/24 (100.00%) successful tests

Average latency: 1.697s


coder (minimax-m3)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.134s
Text Transform Uppercase ✅ Pass 1.002s
Count from 1 to 5 ✅ Pass 1.186s
Math Calculation ✅ Pass 1.072s
Basic Echo Function ✅ Pass 1.217s
Streaming Simple Math Streaming ✅ Pass 1.034s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.733s
Streaming Basic Echo Function Streaming ✅ Pass 2.039s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.153s
Search Query Function ✅ Pass 1.322s
Ask Advice Function ✅ Pass 1.559s
Streaming Search Query Function Streaming ✅ Pass 1.922s
Basic Context Memory Test ✅ Pass 1.357s
Function Argument Memory Test ✅ Pass 1.183s
Function Response Memory Test ✅ Pass 1.159s
Penetration Testing Memory with Tool Call ❌ Fail 1.749s expected function 'generate_report' not found in tool calls: expected function generate_report not found in tool calls
Cybersecurity Workflow Memory Test ✅ Pass 0.982s
Read a file, then edit it via unified diff ✅ Pass 5.263s
Penetration Testing Methodology ✅ Pass 4.101s
Vulnerability Assessment Tools ✅ Pass 3.509s
SQL Injection Attack Type ✅ Pass 1.767s
Penetration Testing Framework ✅ Pass 4.416s
Web Application Security Scanner ✅ Pass 2.111s
Penetration Testing Tool Selection ✅ Pass 1.437s

Summary: 23/24 (95.83%) successful tests

Average latency: 1.892s


installer (deepseek-v4-flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.994s
Text Transform Uppercase ✅ Pass 2.642s
Count from 1 to 5 ✅ Pass 1.896s
Math Calculation ✅ Pass 2.042s
Basic Echo Function ✅ Pass 1.909s
Streaming Simple Math Streaming ✅ Pass 1.611s
Streaming Count from 1 to 3 Streaming ✅ Pass 2.163s
Streaming Basic Echo Function Streaming ✅ Pass 1.951s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.582s
Search Query Function ✅ Pass 1.525s
Ask Advice Function ✅ Pass 1.512s
Streaming Search Query Function Streaming ✅ Pass 1.641s
Basic Context Memory Test ✅ Pass 2.317s
Function Argument Memory Test ✅ Pass 1.634s
Function Response Memory Test ✅ Pass 1.345s
Penetration Testing Memory with Tool Call ✅ Pass 2.281s
Cybersecurity Workflow Memory Test ✅ Pass 1.808s
Read a file, then edit it via unified diff ✅ Pass 4.274s
Penetration Testing Methodology ✅ Pass 1.342s
Vulnerability Assessment Tools ✅ Pass 2.158s
SQL Injection Attack Type ✅ Pass 1.707s
Penetration Testing Framework ✅ Pass 1.398s
Web Application Security Scanner ✅ Pass 1.314s
Penetration Testing Tool Selection ✅ Pass 1.680s

Summary: 24/24 (100.00%) successful tests

Average latency: 1.906s


pentester (deepseek-v4-flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.183s
Text Transform Uppercase ✅ Pass 3.149s
Count from 1 to 5 ✅ Pass 1.765s
Math Calculation ✅ Pass 1.944s
Basic Echo Function ✅ Pass 1.904s
Streaming Simple Math Streaming ✅ Pass 1.358s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.945s
Streaming Basic Echo Function Streaming ✅ Pass 1.824s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.577s
Search Query Function ✅ Pass 1.392s
Ask Advice Function ✅ Pass 1.713s
Streaming Search Query Function Streaming ✅ Pass 1.555s
Basic Context Memory Test ✅ Pass 2.392s
Function Argument Memory Test ✅ Pass 1.696s
Function Response Memory Test ✅ Pass 1.686s
Penetration Testing Memory with Tool Call ✅ Pass 1.942s
Cybersecurity Workflow Memory Test ✅ Pass 2.330s
Read a file, then edit it via unified diff ✅ Pass 3.631s
Penetration Testing Methodology ✅ Pass 1.553s
Vulnerability Assessment Tools ✅ Pass 3.893s
SQL Injection Attack Type ✅ Pass 2.072s
Penetration Testing Framework ✅ Pass 1.467s
Web Application Security Scanner ✅ Pass 1.436s
Penetration Testing Tool Selection ✅ Pass 1.772s

Summary: 24/24 (100.00%) successful tests

Average latency: 2.008s