Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

21.1 KB
Raw Blame

LLM Agent Testing Report

Generated: Wed, 30 Sep 2026 15:55:05 UTC

Overall Results

Agent Model Reasoning Success Rate Average Latency
simple Qwen/Qwen3.8-27B-FP8 false 25/25 (100.00%) 2.660s
simple_json Qwen/Qwen3.8-27B-FP8 false 8/8 (100.00%) 1.269s
primary_agent DeepSeek-V4-Flash true 25/25 (100.00%) 0.926s
assistant DeepSeek-V4-Flash true 25/25 (100.00%) 1.070s
generator DeepSeek-V4-Flash true 25/25 (100.00%) 1.045s
refiner DeepSeek-V4-Flash true 25/25 (100.00%) 0.907s
adviser DeepSeek-V4-Flash true 16/16 (100.00%) 1.014s
reflector Qwen/Qwen3.8-27B-FP8 false 15/16 (93.75%) 2.427s
searcher Qwen/Qwen3.8-27B-FP8 false 25/25 (100.00%) 2.041s
enricher Qwen/Qwen3.8-27B-FP8 false 25/25 (100.00%) 2.134s
coder Qwen/Qwen3.8-27B-FP8 true 25/25 (100.00%) 2.966s
installer Qwen/Qwen3.8-27B-FP8 true 25/25 (100.00%) 2.979s
pentester DeepSeek-V4-Flash true 25/25 (100.00%) 0.807s

Total: 289/290 (99.66%) successful tests Overall average latency: 1.737s

Detailed Results

simple (Qwen/Qwen3.8-27B-FP8)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.877s
Streaming Simple Math Streaming ✅ Pass 1.113s
Math Calculation ✅ Pass 1.152s
Count from 1 to 5 ✅ Pass 1.181s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.327s
Basic Echo Function ✅ Pass 1.364s
Text Transform Uppercase ✅ Pass 1.488s
Streaming Basic Echo Function Streaming ✅ Pass 0.956s
Answer Stops At The Output Limit ✅ Pass 25.684s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 0.859s
JSON Response Function ✅ Pass 1.128s
Streaming Search Query Function Streaming ✅ Pass 1.460s
Basic Context Memory Test ✅ Pass 0.855s
Ask Advice Function ✅ Pass 1.806s
Function Argument Memory Test ✅ Pass 0.970s
Function Response Memory Test ✅ Pass 0.775s
Cybersecurity Workflow Memory Test ✅ Pass 1.440s
Penetration Testing Memory with Tool Call ✅ Pass 1.887s
Read a file, then edit it via unified diff ✅ Pass 6.458s
Penetration Testing Methodology ✅ Pass 2.078s
SQL Injection Attack Type ✅ Pass 0.882s
Penetration Testing Tool Selection ✅ Pass 1.574s
Web Application Security Scanner ✅ Pass 2.437s
Penetration Testing Framework ✅ Pass 2.941s
Vulnerability Assessment Tools ✅ Pass 3.807s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.660s


simple_json (Qwen/Qwen3.8-27B-FP8)

Advanced Tests

Test Result Latency Error
Vulnerability Report Memory Test ✅ Pass 2.105s
Person Information JSON ✅ Pass 1.025s
Project Information JSON ✅ Pass 1.282s
User Profile JSON ✅ Pass 1.190s
Streaming Person Information JSON Streaming ✅ Pass 1.392s
JSON Array Response Without Schema ✅ Pass 2.205s

Capability Tests

Test Capability Result Latency Note
Structured Output Refuses A Non-Object Schema structured_output ✅ Pass 0.000s
Structured Output With JSON Schema structured_output ✅ Pass 0.947s

Summary: 8/8 (100.00%) successful tests

Average latency: 1.269s


primary_agent (DeepSeek-V4-Flash)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 0.340s
Simple Math ✅ Pass 0.388s
Math Calculation ✅ Pass 0.316s
Count from 1 to 5 ✅ Pass 0.532s
Streaming Simple Math Streaming ✅ Pass 0.421s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.450s
Basic Echo Function ✅ Pass 0.526s
Streaming Basic Echo Function Streaming ✅ Pass 0.562s
Answer Stops At The Output Limit ✅ Pass 8.126s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.536s
Search Query Function ✅ Pass 0.457s
Ask Advice Function ✅ Pass 0.608s
Streaming Search Query Function Streaming ✅ Pass 0.408s
Function Response Memory Test ✅ Pass 0.740s
Function Argument Memory Test ✅ Pass 1.000s
Basic Context Memory Test ✅ Pass 1.449s
Penetration Testing Memory with Tool Call ✅ Pass 0.902s
Cybersecurity Workflow Memory Test ✅ Pass 0.577s
Read a file, then edit it via unified diff ✅ Pass 2.069s
Penetration Testing Methodology ✅ Pass 0.308s
SQL Injection Attack Type ✅ Pass 0.324s
Web Application Security Scanner ✅ Pass 0.308s
Penetration Testing Framework ✅ Pass 0.372s
Vulnerability Assessment Tools ✅ Pass 0.700s
Penetration Testing Tool Selection ✅ Pass 0.717s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.926s


assistant (DeepSeek-V4-Flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.417s
Text Transform Uppercase ✅ Pass 0.444s
Math Calculation ✅ Pass 0.505s
Streaming Simple Math Streaming ✅ Pass 0.400s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.409s
Basic Echo Function ✅ Pass 0.582s
Count from 1 to 5 ✅ Pass 0.865s
Streaming Basic Echo Function Streaming ✅ Pass 0.570s
Answer Stops At The Output Limit ✅ Pass 8.308s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.590s
Search Query Function ✅ Pass 0.585s
Ask Advice Function ✅ Pass 0.664s
Streaming Search Query Function Streaming ✅ Pass 0.647s
Function Argument Memory Test ✅ Pass 0.627s
Function Response Memory Test ✅ Pass 0.927s
Basic Context Memory Test ✅ Pass 1.178s
Cybersecurity Workflow Memory Test ✅ Pass 0.586s
Penetration Testing Memory with Tool Call ✅ Pass 1.270s
Read a file, then edit it via unified diff ✅ Pass 1.836s
SQL Injection Attack Type ✅ Pass 0.618s
Penetration Testing Methodology ✅ Pass 0.795s
Penetration Testing Framework ✅ Pass 0.401s
Web Application Security Scanner ✅ Pass 0.417s
Penetration Testing Tool Selection ✅ Pass 0.734s
Vulnerability Assessment Tools ✅ Pass 2.371s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.070s


generator (DeepSeek-V4-Flash)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 0.368s
Simple Math ✅ Pass 0.459s
Count from 1 to 5 ✅ Pass 0.469s
Math Calculation ✅ Pass 0.443s
Streaming Simple Math Streaming ✅ Pass 0.383s
Basic Echo Function ✅ Pass 0.619s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.510s
Streaming Basic Echo Function Streaming ✅ Pass 0.618s
Answer Stops At The Output Limit ✅ Pass 9.096s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 0.600s
JSON Response Function ✅ Pass 0.667s
Ask Advice Function ✅ Pass 0.645s
Streaming Search Query Function Streaming ✅ Pass 0.625s
Function Argument Memory Test ✅ Pass 0.654s
Function Response Memory Test ✅ Pass 0.623s
Cybersecurity Workflow Memory Test ✅ Pass 0.603s
Basic Context Memory Test ✅ Pass 1.508s
Penetration Testing Memory with Tool Call ✅ Pass 1.606s
Read a file, then edit it via unified diff ✅ Pass 2.044s
Penetration Testing Methodology ✅ Pass 0.407s
SQL Injection Attack Type ✅ Pass 0.531s
Penetration Testing Framework ✅ Pass 0.385s
Vulnerability Assessment Tools ✅ Pass 0.855s
Web Application Security Scanner ✅ Pass 0.320s
Penetration Testing Tool Selection ✅ Pass 1.077s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.045s


refiner (DeepSeek-V4-Flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.338s
Text Transform Uppercase ✅ Pass 0.463s
Math Calculation ✅ Pass 0.508s
Count from 1 to 5 ✅ Pass 0.568s
Basic Echo Function ✅ Pass 0.579s
Streaming Simple Math Streaming ✅ Pass 0.379s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.431s
Streaming Basic Echo Function Streaming ✅ Pass 0.510s
Answer Stops At The Output Limit ✅ Pass 6.741s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 0.617s
JSON Response Function ✅ Pass 0.721s
Ask Advice Function ✅ Pass 0.626s
Streaming Search Query Function Streaming ✅ Pass 0.597s
Basic Context Memory Test ✅ Pass 0.516s
Function Response Memory Test ✅ Pass 0.640s
Function Argument Memory Test ✅ Pass 0.680s
Cybersecurity Workflow Memory Test ✅ Pass 0.529s
Penetration Testing Memory with Tool Call ✅ Pass 1.261s
Read a file, then edit it via unified diff ✅ Pass 1.732s
Penetration Testing Methodology ✅ Pass 0.511s
Penetration Testing Framework ✅ Pass 0.415s
Vulnerability Assessment Tools ✅ Pass 0.752s
Web Application Security Scanner ✅ Pass 0.549s
SQL Injection Attack Type ✅ Pass 1.130s
Penetration Testing Tool Selection ✅ Pass 0.865s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.907s


adviser (DeepSeek-V4-Flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.363s
Text Transform Uppercase ✅ Pass 0.430s
Count from 1 to 5 ✅ Pass 0.593s
Math Calculation ✅ Pass 0.345s
Streaming Simple Math Streaming ✅ Pass 0.386s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.403s
Answer Stops At The Output Limit ✅ Pass 7.857s

Advanced Tests

Test Result Latency Error
Cybersecurity Workflow Memory Test ✅ Pass 0.502s
Function Response Memory Test ✅ Pass 0.529s
Function Argument Memory Test ✅ Pass 0.606s
Basic Context Memory Test ✅ Pass 1.222s
Penetration Testing Framework ✅ Pass 0.341s
Penetration Testing Methodology ✅ Pass 0.764s
SQL Injection Attack Type ✅ Pass 0.642s
Vulnerability Assessment Tools ✅ Pass 0.806s
Web Application Security Scanner ✅ Pass 0.426s

Summary: 16/16 (100.00%) successful tests

Average latency: 1.014s


reflector (Qwen/Qwen3.8-27B-FP8)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.123s
Text Transform Uppercase ✅ Pass 0.812s
Math Calculation ✅ Pass 0.630s
Count from 1 to 5 ✅ Pass 1.419s
Streaming Simple Math Streaming ✅ Pass 0.832s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.948s
Answer Stops At The Output Limit ✅ Pass 20.378s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 0.615s
Function Response Memory Test ✅ Pass 0.571s
Cybersecurity Workflow Memory Test ✅ Pass 0.600s
Function Argument Memory Test ❌ Fail 1.188s expected text 'Go programming language' not found; stop reason: stop
SQL Injection Attack Type ✅ Pass 0.700s
Penetration Testing Framework ✅ Pass 1.591s
Web Application Security Scanner ✅ Pass 1.752s
Vulnerability Assessment Tools ✅ Pass 2.773s
Penetration Testing Methodology ✅ Pass 2.897s

Summary: 15/16 (93.75%) successful tests

Average latency: 2.427s


searcher (Qwen/Qwen3.8-27B-FP8)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.877s
Text Transform Uppercase ✅ Pass 0.752s
Count from 1 to 5 ✅ Pass 0.747s
Math Calculation ✅ Pass 0.637s
Streaming Simple Math Streaming ✅ Pass 0.843s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.230s
Basic Echo Function ✅ Pass 1.550s
Streaming Basic Echo Function Streaming ✅ Pass 1.175s
Answer Stops At The Output Limit ✅ Pass 17.939s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.071s
Search Query Function ✅ Pass 0.892s
Function Argument Memory Test ✅ Pass 0.721s
Basic Context Memory Test ✅ Pass 0.830s
Streaming Search Query Function Streaming ✅ Pass 0.871s
Ask Advice Function ✅ Pass 1.197s
Function Response Memory Test ✅ Pass 0.940s
Cybersecurity Workflow Memory Test ✅ Pass 0.668s
Penetration Testing Memory with Tool Call ✅ Pass 1.880s
Read a file, then edit it via unified diff ✅ Pass 5.749s
SQL Injection Attack Type ✅ Pass 0.675s
Penetration Testing Methodology ✅ Pass 1.930s
Penetration Testing Framework ✅ Pass 1.777s
Web Application Security Scanner ✅ Pass 1.710s
Penetration Testing Tool Selection ✅ Pass 1.133s
Vulnerability Assessment Tools ✅ Pass 3.226s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.041s


enricher (Qwen/Qwen3.8-27B-FP8)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.616s
Count from 1 to 5 ✅ Pass 0.682s
Text Transform Uppercase ✅ Pass 0.725s
Math Calculation ✅ Pass 0.694s
Streaming Simple Math Streaming ✅ Pass 0.605s
Basic Echo Function ✅ Pass 1.007s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.401s
Streaming Basic Echo Function Streaming ✅ Pass 1.608s
Answer Stops At The Output Limit ✅ Pass 18.074s

Advanced Tests

Test Result Latency Error
Function Argument Memory Test ✅ Pass 0.623s
JSON Response Function ✅ Pass 1.110s
Basic Context Memory Test ✅ Pass 0.649s
Streaming Search Query Function Streaming ✅ Pass 1.039s
Search Query Function ✅ Pass 1.151s
Ask Advice Function ✅ Pass 1.367s
Function Response Memory Test ✅ Pass 0.554s
Cybersecurity Workflow Memory Test ✅ Pass 1.726s
Penetration Testing Memory with Tool Call ✅ Pass 3.037s
Read a file, then edit it via unified diff ✅ Pass 4.871s
SQL Injection Attack Type ✅ Pass 0.927s
Penetration Testing Methodology ✅ Pass 2.076s
Penetration Testing Tool Selection ✅ Pass 1.288s
Web Application Security Scanner ✅ Pass 1.457s
Penetration Testing Framework ✅ Pass 1.922s
Vulnerability Assessment Tools ✅ Pass 4.124s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.134s


coder (Qwen/Qwen3.8-27B-FP8)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.977s
Text Transform Uppercase ✅ Pass 1.107s
Math Calculation ✅ Pass 1.026s
Count from 1 to 5 ✅ Pass 1.237s
Basic Echo Function ✅ Pass 1.649s
Streaming Simple Math Streaming ✅ Pass 1.559s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.477s
Streaming Basic Echo Function Streaming ✅ Pass 1.346s
Answer Stops At The Output Limit ✅ Pass 26.162s

Advanced Tests

Test Result Latency Error
Streaming Search Query Function Streaming ✅ Pass 1.766s
Search Query Function ✅ Pass 2.263s
Basic Context Memory Test ✅ Pass 2.019s
JSON Response Function ✅ Pass 2.585s
Ask Advice Function ✅ Pass 2.462s
Function Argument Memory Test ✅ Pass 1.874s
Function Response Memory Test ✅ Pass 1.283s
Cybersecurity Workflow Memory Test ✅ Pass 2.481s
Penetration Testing Memory with Tool Call ✅ Pass 4.669s
Read a file, then edit it via unified diff ✅ Pass 4.858s
Penetration Testing Methodology ✅ Pass 1.243s
SQL Injection Attack Type ✅ Pass 1.810s
Web Application Security Scanner ✅ Pass 1.516s
Vulnerability Assessment Tools ✅ Pass 2.377s
Penetration Testing Tool Selection ✅ Pass 1.995s
Penetration Testing Framework ✅ Pass 2.401s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.966s


installer (Qwen/Qwen3.8-27B-FP8)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.335s
Text Transform Uppercase ✅ Pass 3.156s
Count from 1 to 5 ✅ Pass 4.504s
Math Calculation ✅ Pass 4.474s
Streaming Simple Math Streaming ✅ Pass 1.184s
Basic Echo Function ✅ Pass 2.944s
Streaming Basic Echo Function Streaming ✅ Pass 1.145s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.793s
Answer Stops At The Output Limit ✅ Pass 18.439s

Advanced Tests

Test Result Latency Error
Streaming Search Query Function Streaming ✅ Pass 1.158s
Search Query Function ✅ Pass 1.575s
JSON Response Function ✅ Pass 1.767s
Ask Advice Function ✅ Pass 1.812s
Function Response Memory Test ✅ Pass 1.251s
Basic Context Memory Test ✅ Pass 2.580s
Cybersecurity Workflow Memory Test ✅ Pass 1.543s
Penetration Testing Memory with Tool Call ✅ Pass 3.639s
Function Argument Memory Test ✅ Pass 4.864s
Read a file, then edit it via unified diff ✅ Pass 4.589s
Penetration Testing Methodology ✅ Pass 1.344s
SQL Injection Attack Type ✅ Pass 1.632s
Web Application Security Scanner ✅ Pass 1.454s
Penetration Testing Framework ✅ Pass 1.655s
Vulnerability Assessment Tools ✅ Pass 2.692s
Penetration Testing Tool Selection ✅ Pass 1.941s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.979s


pentester (DeepSeek-V4-Flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.336s
Text Transform Uppercase ✅ Pass 0.324s
Count from 1 to 5 ✅ Pass 0.401s
Math Calculation ✅ Pass 0.304s
Streaming Simple Math Streaming ✅ Pass 0.325s
Basic Echo Function ✅ Pass 0.393s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.418s
Streaming Basic Echo Function Streaming ✅ Pass 0.412s
Answer Stops At The Output Limit ✅ Pass 6.242s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.486s
Search Query Function ✅ Pass 0.448s
Ask Advice Function ✅ Pass 0.483s
Streaming Search Query Function Streaming ✅ Pass 0.425s
Function Argument Memory Test ✅ Pass 0.681s
Function Response Memory Test ✅ Pass 0.696s
Basic Context Memory Test ✅ Pass 0.979s
Cybersecurity Workflow Memory Test ✅ Pass 0.512s
Penetration Testing Memory with Tool Call ✅ Pass 1.127s
Read a file, then edit it via unified diff ✅ Pass 1.576s
Penetration Testing Methodology ✅ Pass 0.612s
SQL Injection Attack Type ✅ Pass 0.461s
Penetration Testing Framework ✅ Pass 0.361s
Web Application Security Scanner ✅ Pass 0.392s
Vulnerability Assessment Tools ✅ Pass 1.194s
Penetration Testing Tool Selection ✅ Pass 0.572s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.807s