Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

21.8 KB
Raw Blame

LLM Agent Testing Report

Generated: Thu, 24 Sep 2026 21:30:59 UTC

Overall Results

Agent Model Reasoning Success Rate Average Latency
simple deepseek-flash false 26/26 (100.00%) 1.415s
simple_json deepseek-flash false 8/8 (100.00%) 0.775s
primary_agent deepseek-v4-pro true 25/25 (100.00%) 3.173s
assistant deepseek-v4-pro true 25/25 (100.00%) 2.063s
generator deepseek-v4-pro true 25/25 (100.00%) 2.706s
refiner deepseek-v4-pro true 25/25 (100.00%) 1.992s
adviser deepseek-v4-pro true 16/16 (100.00%) 2.095s
reflector deepseek-flash false 17/17 (100.00%) 1.485s
searcher deepseek-flash false 26/26 (100.00%) 1.381s
enricher deepseek-flash false 26/26 (100.00%) 0.472s
coder deepseek-v4-pro true 25/25 (100.00%) 2.906s
installer deepseek-flash true 25/25 (100.00%) 1.374s
pentester deepseek-v4-pro true 25/25 (100.00%) 0.674s

Total: 294/294 (100.00%) successful tests Overall average latency: 1.776s

Detailed Results

simple (deepseek-flash)

Basic Tests

Test Result Latency Error
Math Calculation ✅ Pass 0.698s
Simple Math ✅ Pass 0.698s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.706s
Count from 1 to 5 ✅ Pass 0.782s
Streaming Simple Math Streaming ✅ Pass 0.963s
Text Transform Uppercase ✅ Pass 0.998s
Basic Echo Function ✅ Pass 1.106s
Streaming Basic Echo Function Streaming ✅ Pass 0.880s
Answer Stops At The Output Limit ✅ Pass 10.748s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.990s
Ask Advice Function ✅ Pass 1.082s
Search Query Function ✅ Pass 1.146s
Streaming Search Query Function Streaming ✅ Pass 0.950s
Basic Context Memory Test ✅ Pass 1.081s
Function Argument Memory Test ✅ Pass 1.032s
Function Response Memory Test ✅ Pass 1.160s
Penetration Testing Memory with Tool Call ✅ Pass 1.168s
Cybersecurity Workflow Memory Test ✅ Pass 0.920s
Read a file, then edit it via unified diff ✅ Pass 3.049s
Penetration Testing Methodology ✅ Pass 0.822s
Penetration Testing Framework ✅ Pass 0.877s
SQL Injection Attack Type ✅ Pass 1.014s
Web Application Security Scanner ✅ Pass 0.689s
Vulnerability Assessment Tools ✅ Pass 1.267s
Penetration Testing Tool Selection ✅ Pass 1.199s

Capability Tests

Test Capability Result Latency Note
Explicit Reasoning Off Suppresses Reasoning reasoning_off ✅ Pass 0.750s

Summary: 26/26 (100.00%) successful tests

Average latency: 1.415s


simple_json (deepseek-flash)

Advanced Tests

Test Result Latency Error
Vulnerability Report Memory Test ✅ Pass 0.919s
Person Information JSON ✅ Pass 0.822s
Project Information JSON ✅ Pass 0.839s
User Profile JSON ✅ Pass 1.044s
JSON Array Response Without Schema ✅ Pass 0.795s
Streaming Person Information JSON Streaming ✅ Pass 0.866s

Capability Tests

Test Capability Result Latency Note
Structured Output Refuses A Non-Object Schema structured_output ✅ Pass 0.000s
Structured Output With JSON Schema structured_output ✅ Pass 0.910s

Summary: 8/8 (100.00%) successful tests

Average latency: 0.775s


primary_agent (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Math Calculation ✅ Pass 1.323s
Text Transform Uppercase ✅ Pass 1.663s
Count from 1 to 5 ✅ Pass 1.542s
Simple Math ✅ Pass 1.971s
Basic Echo Function ✅ Pass 1.669s
Streaming Simple Math Streaming ✅ Pass 1.516s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.925s
Streaming Basic Echo Function Streaming ✅ Pass 1.801s
Answer Stops At The Output Limit ✅ Pass 26.161s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.986s
Ask Advice Function ✅ Pass 2.007s
Search Query Function ✅ Pass 2.031s
Function Argument Memory Test ✅ Pass 1.578s
Streaming Search Query Function Streaming ✅ Pass 2.362s
Basic Context Memory Test ✅ Pass 2.379s
Function Response Memory Test ✅ Pass 1.543s
Cybersecurity Workflow Memory Test ✅ Pass 1.657s
Penetration Testing Memory with Tool Call ✅ Pass 2.586s
Read a file, then edit it via unified diff ✅ Pass 5.222s
SQL Injection Attack Type ✅ Pass 1.823s
Penetration Testing Methodology ✅ Pass 2.289s
Penetration Testing Framework ✅ Pass 2.154s
Web Application Security Scanner ✅ Pass 2.113s
Penetration Testing Tool Selection ✅ Pass 2.223s
Vulnerability Assessment Tools ✅ Pass 5.790s

Summary: 25/25 (100.00%) successful tests

Average latency: 3.173s


assistant (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.180s
Text Transform Uppercase ✅ Pass 0.177s
Count from 1 to 5 ✅ Pass 0.186s
Math Calculation ✅ Pass 0.234s
Basic Echo Function ✅ Pass 0.276s
Streaming Simple Math Streaming ✅ Pass 0.260s
Streaming Basic Echo Function Streaming ✅ Pass 1.429s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.607s
Answer Stops At The Output Limit ✅ Pass 24.979s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.189s
Search Query Function ✅ Pass 0.175s
Ask Advice Function ✅ Pass 0.188s
Streaming Search Query Function Streaming ✅ Pass 0.186s
Basic Context Memory Test ✅ Pass 0.206s
Function Response Memory Test ✅ Pass 1.672s
Function Argument Memory Test ✅ Pass 1.813s
Cybersecurity Workflow Memory Test ✅ Pass 1.732s
Penetration Testing Memory with Tool Call ✅ Pass 2.093s
Read a file, then edit it via unified diff ✅ Pass 3.744s
SQL Injection Attack Type ✅ Pass 0.193s
Penetration Testing Methodology ✅ Pass 1.730s
Penetration Testing Tool Selection ✅ Pass 1.503s
Penetration Testing Framework ✅ Pass 1.786s
Web Application Security Scanner ✅ Pass 1.791s
Vulnerability Assessment Tools ✅ Pass 3.247s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.063s


generator (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.672s
Count from 1 to 5 ✅ Pass 1.488s
Text Transform Uppercase ✅ Pass 2.137s
Math Calculation ✅ Pass 1.596s
Streaming Simple Math Streaming ✅ Pass 1.064s
Basic Echo Function ✅ Pass 1.738s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.360s
Streaming Basic Echo Function Streaming ✅ Pass 1.576s
Answer Stops At The Output Limit ✅ Pass 21.727s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.726s
Search Query Function ✅ Pass 1.739s
Function Argument Memory Test ✅ Pass 1.451s
Streaming Search Query Function Streaming ✅ Pass 1.685s
Ask Advice Function ✅ Pass 1.722s
Function Response Memory Test ✅ Pass 1.610s
Basic Context Memory Test ✅ Pass 1.915s
Cybersecurity Workflow Memory Test ✅ Pass 1.587s
Penetration Testing Memory with Tool Call ✅ Pass 2.361s
Read a file, then edit it via unified diff ✅ Pass 4.040s
SQL Injection Attack Type ✅ Pass 2.149s
Penetration Testing Methodology ✅ Pass 2.408s
Penetration Testing Framework ✅ Pass 1.261s
Penetration Testing Tool Selection ✅ Pass 1.707s
Web Application Security Scanner ✅ Pass 1.815s
Vulnerability Assessment Tools ✅ Pass 4.101s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.706s


refiner (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.179s
Text Transform Uppercase ✅ Pass 0.175s
Count from 1 to 5 ✅ Pass 0.185s
Math Calculation ✅ Pass 0.174s
Basic Echo Function ✅ Pass 0.181s
Streaming Simple Math Streaming ✅ Pass 0.186s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.182s
Streaming Basic Echo Function Streaming ✅ Pass 0.188s
Answer Stops At The Output Limit ✅ Pass 25.811s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.182s
Search Query Function ✅ Pass 0.180s
Ask Advice Function ✅ Pass 0.180s
Streaming Search Query Function Streaming ✅ Pass 0.185s
Basic Context Memory Test ✅ Pass 0.182s
Cybersecurity Workflow Memory Test ✅ Pass 1.642s
Function Response Memory Test ✅ Pass 1.807s
Function Argument Memory Test ✅ Pass 1.977s
Penetration Testing Memory with Tool Call ✅ Pass 2.552s
Read a file, then edit it via unified diff ✅ Pass 4.386s
SQL Injection Attack Type ✅ Pass 0.177s
Penetration Testing Framework ✅ Pass 0.176s
Penetration Testing Methodology ✅ Pass 1.642s
Web Application Security Scanner ✅ Pass 1.751s
Penetration Testing Tool Selection ✅ Pass 1.957s
Vulnerability Assessment Tools ✅ Pass 3.545s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.992s


adviser (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.183s
Text Transform Uppercase ✅ Pass 0.183s
Count from 1 to 5 ✅ Pass 0.178s
Math Calculation ✅ Pass 0.178s
Streaming Simple Math Streaming ✅ Pass 0.180s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.248s
Answer Stops At The Output Limit ✅ Pass 26.372s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 0.175s
Function Argument Memory Test ✅ Pass 1.539s
Function Response Memory Test ✅ Pass 1.573s
Cybersecurity Workflow Memory Test ✅ Pass 1.778s
Vulnerability Assessment Tools ✅ Pass 0.189s
Penetration Testing Methodology ✅ Pass 0.195s
SQL Injection Attack Type ✅ Pass 0.178s
Penetration Testing Framework ✅ Pass 0.183s
Web Application Security Scanner ✅ Pass 0.174s

Summary: 16/16 (100.00%) successful tests

Average latency: 2.095s


reflector (deepseek-flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.775s
Text Transform Uppercase ✅ Pass 1.022s
Count from 1 to 5 ✅ Pass 0.760s
Math Calculation ✅ Pass 1.285s
Streaming Simple Math Streaming ✅ Pass 0.790s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.838s
Answer Stops At The Output Limit ✅ Pass 10.654s

Advanced Tests

Test Result Latency Error
Function Argument Memory Test ✅ Pass 0.672s
Function Response Memory Test ✅ Pass 0.868s
Basic Context Memory Test ✅ Pass 0.962s
Cybersecurity Workflow Memory Test ✅ Pass 0.841s
SQL Injection Attack Type ✅ Pass 0.782s
Penetration Testing Methodology ✅ Pass 0.965s
Penetration Testing Framework ✅ Pass 0.954s
Web Application Security Scanner ✅ Pass 0.672s
Vulnerability Assessment Tools ✅ Pass 1.550s

Capability Tests

Test Capability Result Latency Note
Explicit Reasoning Off Suppresses Reasoning reasoning_off ✅ Pass 0.846s

Summary: 17/17 (100.00%) successful tests

Average latency: 1.485s


searcher (deepseek-flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.697s
Text Transform Uppercase ✅ Pass 0.943s
Count from 1 to 5 ✅ Pass 0.767s
Math Calculation ✅ Pass 0.862s
Basic Echo Function ✅ Pass 1.090s
Streaming Simple Math Streaming ✅ Pass 1.016s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.904s
Streaming Basic Echo Function Streaming ✅ Pass 1.295s
Answer Stops At The Output Limit ✅ Pass 11.091s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.782s
Function Argument Memory Test ✅ Pass 0.609s
Basic Context Memory Test ✅ Pass 0.698s
Ask Advice Function ✅ Pass 0.742s
Search Query Function ✅ Pass 0.873s
Streaming Search Query Function Streaming ✅ Pass 0.846s
Function Response Memory Test ✅ Pass 0.762s
Cybersecurity Workflow Memory Test ✅ Pass 0.764s
Penetration Testing Memory with Tool Call ✅ Pass 1.192s
Read a file, then edit it via unified diff ✅ Pass 2.538s
SQL Injection Attack Type ✅ Pass 0.768s
Vulnerability Assessment Tools ✅ Pass 1.082s
Penetration Testing Framework ✅ Pass 0.949s
Penetration Testing Tool Selection ✅ Pass 0.944s
Web Application Security Scanner ✅ Pass 1.102s
Penetration Testing Methodology ✅ Pass 1.648s

Capability Tests

Test Capability Result Latency Note
Explicit Reasoning Off Suppresses Reasoning reasoning_off ✅ Pass 0.919s

Summary: 26/26 (100.00%) successful tests

Average latency: 1.381s


enricher (deepseek-flash)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.185s
Simple Math ✅ Pass 0.181s
Text Transform Uppercase ✅ Pass 0.175s
Count from 1 to 5 ✅ Pass 0.180s
Math Calculation ✅ Pass 0.186s
Basic Echo Function ✅ Pass 0.181s
Streaming Simple Math Streaming ✅ Pass 0.179s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.199s
Streaming Basic Echo Function Streaming ✅ Pass 0.190s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.195s
Search Query Function ✅ Pass 0.233s
Ask Advice Function ✅ Pass 0.233s
Streaming Search Query Function Streaming ✅ Pass 0.250s
Basic Context Memory Test ✅ Pass 0.178s
Function Response Memory Test ✅ Pass 0.595s
Function Argument Memory Test ✅ Pass 0.808s
Cybersecurity Workflow Memory Test ✅ Pass 0.754s
Penetration Testing Memory with Tool Call ✅ Pass 1.016s
Read a file, then edit it via unified diff ✅ Pass 2.281s
SQL Injection Attack Type ✅ Pass 0.183s
Penetration Testing Framework ✅ Pass 0.177s
Penetration Testing Tool Selection ✅ Pass 0.180s
Vulnerability Assessment Tools ✅ Pass 1.132s
Penetration Testing Methodology ✅ Pass 1.330s
Web Application Security Scanner ✅ Pass 0.853s

Capability Tests

Test Capability Result Latency Note
Explicit Reasoning Off Suppresses Reasoning reasoning_off ✅ Pass 0.202s

Summary: 26/26 (100.00%) successful tests

Average latency: 0.472s


coder (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.926s
Text Transform Uppercase ✅ Pass 1.521s
Count from 1 to 5 ✅ Pass 1.246s
Math Calculation ✅ Pass 1.556s
Basic Echo Function ✅ Pass 1.712s
Streaming Simple Math Streaming ✅ Pass 1.540s
Streaming Count from 1 to 3 Streaming ✅ Pass 2.029s
Streaming Basic Echo Function Streaming ✅ Pass 1.942s
Answer Stops At The Output Limit ✅ Pass 22.987s

Advanced Tests

Test Result Latency Error
Search Query Function ✅ Pass 1.550s
JSON Response Function ✅ Pass 1.740s
Function Argument Memory Test ✅ Pass 1.788s
Function Response Memory Test ✅ Pass 1.940s
Streaming Search Query Function Streaming ✅ Pass 2.213s
Basic Context Memory Test ✅ Pass 2.157s
Ask Advice Function ✅ Pass 2.297s
Penetration Testing Memory with Tool Call ✅ Pass 2.103s
Cybersecurity Workflow Memory Test ✅ Pass 1.941s
Read a file, then edit it via unified diff ✅ Pass 4.611s
SQL Injection Attack Type ✅ Pass 1.642s
Penetration Testing Methodology ✅ Pass 1.821s
Penetration Testing Framework ✅ Pass 1.798s
Web Application Security Scanner ✅ Pass 1.738s
Penetration Testing Tool Selection ✅ Pass 2.112s
Vulnerability Assessment Tools ✅ Pass 4.725s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.906s


installer (deepseek-flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.023s
Text Transform Uppercase ✅ Pass 0.996s
Count from 1 to 5 ✅ Pass 1.041s
Math Calculation ✅ Pass 1.115s
Basic Echo Function ✅ Pass 0.907s
Streaming Simple Math Streaming ✅ Pass 0.798s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.682s
Streaming Basic Echo Function Streaming ✅ Pass 1.128s
Answer Stops At The Output Limit ✅ Pass 7.256s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.072s
Search Query Function ✅ Pass 1.035s
Streaming Search Query Function Streaming ✅ Pass 1.059s
Ask Advice Function ✅ Pass 1.114s
Basic Context Memory Test ✅ Pass 1.001s
Function Response Memory Test ✅ Pass 0.794s
Function Argument Memory Test ✅ Pass 1.150s
Cybersecurity Workflow Memory Test ✅ Pass 1.312s
Penetration Testing Memory with Tool Call ✅ Pass 1.633s
Read a file, then edit it via unified diff ✅ Pass 2.086s
Penetration Testing Methodology ✅ Pass 1.089s
Vulnerability Assessment Tools ✅ Pass 1.811s
SQL Injection Attack Type ✅ Pass 1.051s
Penetration Testing Framework ✅ Pass 1.018s
Penetration Testing Tool Selection ✅ Pass 0.981s
Web Application Security Scanner ✅ Pass 1.180s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.374s


pentester (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.179s
Simple Math ✅ Pass 0.178s
Text Transform Uppercase ✅ Pass 0.177s
Count from 1 to 5 ✅ Pass 0.175s
Math Calculation ✅ Pass 0.177s
Basic Echo Function ✅ Pass 0.171s
Streaming Simple Math Streaming ✅ Pass 0.182s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.177s
Streaming Basic Echo Function Streaming ✅ Pass 0.183s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.221s
Ask Advice Function ✅ Pass 0.210s
Search Query Function ✅ Pass 0.223s
Streaming Search Query Function Streaming ✅ Pass 0.197s
Basic Context Memory Test ✅ Pass 0.180s
Function Argument Memory Test ✅ Pass 2.428s
Cybersecurity Workflow Memory Test ✅ Pass 2.427s
Function Response Memory Test ✅ Pass 2.461s
Penetration Testing Memory with Tool Call ✅ Pass 3.062s
Read a file, then edit it via unified diff ✅ Pass 2.759s
Penetration Testing Methodology ✅ Pass 0.179s
Vulnerability Assessment Tools ✅ Pass 0.182s
SQL Injection Attack Type ✅ Pass 0.182s
Penetration Testing Framework ✅ Pass 0.177s
Web Application Security Scanner ✅ Pass 0.176s
Penetration Testing Tool Selection ✅ Pass 0.183s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.674s