Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

21.0 KB
Raw Blame

LLM Agent Testing Report

Generated: Thu, 24 Sep 2026 21:36:36 UTC

Overall Results

Agent Model Reasoning Success Rate Average Latency
simple qwen3.6-flash true 25/25 (100.00%) 5.997s
simple_json qwen3.6-flash true 8/8 (100.00%) 4.357s
primary_agent deepseek-v4.1-flash true 25/25 (100.00%) 0.405s
assistant deepseek-v4.1-flash true 25/25 (100.00%) 0.389s
generator deepseek-v4-pro true 25/25 (100.00%) 4.838s
refiner deepseek-v4-pro true 25/25 (100.00%) 4.778s
adviser deepseek-v4-pro true 16/16 (100.00%) 5.322s
reflector qwen3.6-flash true 16/16 (100.00%) 4.372s
searcher qwen3.6-flash true 25/25 (100.00%) 3.003s
enricher qwen3.6-flash true 25/25 (100.00%) 2.174s
coder deepseek-v4.1-flash true 25/25 (100.00%) 0.403s
installer deepseek-v4.1-flash true 25/25 (100.00%) 0.396s
pentester deepseek-v4.1-flash true 25/25 (100.00%) 0.397s

Total: 290/290 (100.00%) successful tests Overall average latency: 2.619s

Detailed Results

simple (qwen3.6-flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.642s
Basic Echo Function ✅ Pass 1.864s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.934s
Math Calculation ✅ Pass 1.951s
Streaming Simple Math Streaming ✅ Pass 2.072s
Text Transform Uppercase ✅ Pass 2.296s
Count from 1 to 5 ✅ Pass 2.362s
Streaming Basic Echo Function Streaming ✅ Pass 1.177s
Answer Stops At The Output Limit ✅ Pass 90.027s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 2.037s
Search Query Function ✅ Pass 1.691s
Ask Advice Function ✅ Pass 1.144s
Streaming Search Query Function Streaming ✅ Pass 1.574s
Basic Context Memory Test ✅ Pass 2.366s
Function Argument Memory Test ✅ Pass 1.961s
Function Response Memory Test ✅ Pass 1.427s
Penetration Testing Memory with Tool Call ✅ Pass 1.913s
Cybersecurity Workflow Memory Test ✅ Pass 2.383s
Read a file, then edit it via unified diff ✅ Pass 2.724s
Penetration Testing Methodology ✅ Pass 4.131s
SQL Injection Attack Type ✅ Pass 3.063s
Penetration Testing Tool Selection ✅ Pass 1.444s
Web Application Security Scanner ✅ Pass 3.597s
Vulnerability Assessment Tools ✅ Pass 6.585s
Penetration Testing Framework ✅ Pass 6.546s

Summary: 25/25 (100.00%) successful tests

Average latency: 5.997s


simple_json (qwen3.6-flash)

Advanced Tests

Test Result Latency Error
Vulnerability Report Memory Test ✅ Pass 2.110s
Person Information JSON ✅ Pass 4.881s
Project Information JSON ✅ Pass 5.137s
User Profile JSON ✅ Pass 7.394s
JSON Array Response Without Schema ✅ Pass 5.891s
Streaming Person Information JSON Streaming ✅ Pass 6.453s

Capability Tests

Test Capability Result Latency Note
Structured Output Refuses A Non-Object Schema structured_output ✅ Pass 0.000s
Structured Output With JSON Schema structured_output ✅ Pass 2.987s

Summary: 8/8 (100.00%) successful tests

Average latency: 4.357s


primary_agent (deepseek-v4.1-flash)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.187s
Simple Math ✅ Pass 0.175s
Text Transform Uppercase ✅ Pass 0.189s
Count from 1 to 5 ✅ Pass 0.172s
Math Calculation ✅ Pass 0.173s
Basic Echo Function ✅ Pass 0.174s
Streaming Simple Math Streaming ✅ Pass 0.176s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.175s
Streaming Basic Echo Function Streaming ✅ Pass 0.184s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.308s
Search Query Function ✅ Pass 0.188s
Ask Advice Function ✅ Pass 0.180s
Streaming Search Query Function Streaming ✅ Pass 0.195s
Basic Context Memory Test ✅ Pass 0.181s
Function Argument Memory Test ✅ Pass 0.942s
Cybersecurity Workflow Memory Test ✅ Pass 0.983s
Function Response Memory Test ✅ Pass 1.713s
Penetration Testing Memory with Tool Call ✅ Pass 2.381s
Read a file, then edit it via unified diff ✅ Pass 0.361s
Penetration Testing Methodology ✅ Pass 0.185s
Vulnerability Assessment Tools ✅ Pass 0.175s
SQL Injection Attack Type ✅ Pass 0.178s
Penetration Testing Framework ✅ Pass 0.174s
Web Application Security Scanner ✅ Pass 0.176s
Penetration Testing Tool Selection ✅ Pass 0.177s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.405s


assistant (deepseek-v4.1-flash)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.176s
Simple Math ✅ Pass 0.181s
Text Transform Uppercase ✅ Pass 0.174s
Count from 1 to 5 ✅ Pass 0.171s
Math Calculation ✅ Pass 0.182s
Basic Echo Function ✅ Pass 0.185s
Streaming Simple Math Streaming ✅ Pass 0.243s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.269s
Streaming Basic Echo Function Streaming ✅ Pass 0.280s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.179s
Search Query Function ✅ Pass 0.184s
Ask Advice Function ✅ Pass 0.183s
Streaming Search Query Function Streaming ✅ Pass 0.180s
Basic Context Memory Test ✅ Pass 0.172s
Function Argument Memory Test ✅ Pass 1.230s
Function Response Memory Test ✅ Pass 1.567s
Penetration Testing Memory with Tool Call ✅ Pass 1.451s
Cybersecurity Workflow Memory Test ✅ Pass 1.265s
Read a file, then edit it via unified diff ✅ Pass 0.354s
Penetration Testing Methodology ✅ Pass 0.180s
Vulnerability Assessment Tools ✅ Pass 0.174s
SQL Injection Attack Type ✅ Pass 0.174s
Penetration Testing Framework ✅ Pass 0.184s
Web Application Security Scanner ✅ Pass 0.186s
Penetration Testing Tool Selection ✅ Pass 0.181s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.389s


generator (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.015s
Math Calculation ✅ Pass 2.101s
Streaming Simple Math Streaming ✅ Pass 2.170s
Count from 1 to 5 ✅ Pass 2.456s
Basic Echo Function ✅ Pass 2.468s
Text Transform Uppercase ✅ Pass 2.849s
Streaming Count from 1 to 3 Streaming ✅ Pass 2.228s
Streaming Basic Echo Function Streaming ✅ Pass 3.018s
Answer Stops At The Output Limit ✅ Pass 37.841s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 2.655s
Ask Advice Function ✅ Pass 2.860s
Search Query Function ✅ Pass 3.199s
Streaming Search Query Function Streaming ✅ Pass 2.779s
Basic Context Memory Test ✅ Pass 3.069s
Function Argument Memory Test ✅ Pass 2.279s
Function Response Memory Test ✅ Pass 1.953s
Cybersecurity Workflow Memory Test ✅ Pass 2.284s
Penetration Testing Memory with Tool Call ✅ Pass 4.690s
Read a file, then edit it via unified diff ✅ Pass 7.796s
Web Application Security Scanner ✅ Pass 2.965s
Penetration Testing Methodology ✅ Pass 4.736s
Penetration Testing Framework ✅ Pass 4.376s
Penetration Testing Tool Selection ✅ Pass 4.317s
SQL Injection Attack Type ✅ Pass 6.643s
Vulnerability Assessment Tools ✅ Pass 7.201s

Summary: 25/25 (100.00%) successful tests

Average latency: 4.838s


refiner (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 2.007s
Simple Math ✅ Pass 2.171s
Count from 1 to 5 ✅ Pass 2.601s
Math Calculation ✅ Pass 2.099s
Streaming Simple Math Streaming ✅ Pass 1.922s
Basic Echo Function ✅ Pass 2.399s
Streaming Count from 1 to 3 Streaming ✅ Pass 2.741s
Streaming Basic Echo Function Streaming ✅ Pass 2.802s
Answer Stops At The Output Limit ✅ Pass 37.255s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 2.544s
Search Query Function ✅ Pass 2.655s
Streaming Search Query Function Streaming ✅ Pass 2.371s
Basic Context Memory Test ✅ Pass 2.537s
Ask Advice Function ✅ Pass 3.439s
Function Argument Memory Test ✅ Pass 2.160s
Function Response Memory Test ✅ Pass 2.499s
Cybersecurity Workflow Memory Test ✅ Pass 2.742s
Penetration Testing Memory with Tool Call ✅ Pass 4.828s
Read a file, then edit it via unified diff ✅ Pass 8.050s
SQL Injection Attack Type ✅ Pass 2.962s
Penetration Testing Framework ✅ Pass 3.494s
Penetration Testing Methodology ✅ Pass 6.160s
Web Application Security Scanner ✅ Pass 3.240s
Penetration Testing Tool Selection ✅ Pass 3.253s
Vulnerability Assessment Tools ✅ Pass 10.502s

Summary: 25/25 (100.00%) successful tests

Average latency: 4.778s


adviser (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.267s
Text Transform Uppercase ✅ Pass 2.207s
Math Calculation ✅ Pass 1.987s
Count from 1 to 5 ✅ Pass 2.708s
Streaming Simple Math Streaming ✅ Pass 2.202s
Streaming Count from 1 to 3 Streaming ✅ Pass 2.477s
Answer Stops At The Output Limit ✅ Pass 36.965s

Advanced Tests

Test Result Latency Error
Function Argument Memory Test ✅ Pass 2.083s
Basic Context Memory Test ✅ Pass 2.582s
Function Response Memory Test ✅ Pass 2.629s
Cybersecurity Workflow Memory Test ✅ Pass 2.829s
Penetration Testing Methodology ✅ Pass 3.659s
Web Application Security Scanner ✅ Pass 2.776s
SQL Injection Attack Type ✅ Pass 4.390s
Penetration Testing Framework ✅ Pass 4.162s
Vulnerability Assessment Tools ✅ Pass 9.218s

Summary: 16/16 (100.00%) successful tests

Average latency: 5.322s


reflector (qwen3.6-flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.634s
Text Transform Uppercase ✅ Pass 2.049s
Count from 1 to 5 ✅ Pass 1.789s
Math Calculation ✅ Pass 1.898s
Streaming Simple Math Streaming ✅ Pass 1.335s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.729s
Answer Stops At The Output Limit ✅ Pass 29.101s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 1.644s
Function Response Memory Test ✅ Pass 1.194s
Function Argument Memory Test ✅ Pass 1.246s
Cybersecurity Workflow Memory Test ✅ Pass 1.925s
Penetration Testing Methodology ✅ Pass 4.958s
SQL Injection Attack Type ✅ Pass 3.243s
Vulnerability Assessment Tools ✅ Pass 6.019s
Penetration Testing Framework ✅ Pass 4.203s
Web Application Security Scanner ✅ Pass 5.970s

Summary: 16/16 (100.00%) successful tests

Average latency: 4.372s


searcher (qwen3.6-flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.173s
Text Transform Uppercase ✅ Pass 0.173s
Count from 1 to 5 ✅ Pass 0.172s
Math Calculation ✅ Pass 0.175s
Streaming Simple Math Streaming ✅ Pass 0.171s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.180s
Basic Echo Function ✅ Pass 1.159s
Streaming Basic Echo Function Streaming ✅ Pass 1.096s
Answer Stops At The Output Limit ✅ Pass 32.820s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 0.175s
JSON Response Function ✅ Pass 1.742s
Search Query Function ✅ Pass 1.656s
Streaming Search Query Function Streaming ✅ Pass 1.350s
Ask Advice Function ✅ Pass 1.657s
Function Argument Memory Test ✅ Pass 1.611s
Function Response Memory Test ✅ Pass 1.936s
Cybersecurity Workflow Memory Test ✅ Pass 1.516s
Penetration Testing Memory with Tool Call ✅ Pass 1.763s
Read a file, then edit it via unified diff ✅ Pass 3.870s
Penetration Testing Methodology ✅ Pass 0.178s
SQL Injection Attack Type ✅ Pass 3.555s
Web Application Security Scanner ✅ Pass 3.068s
Penetration Testing Tool Selection ✅ Pass 1.950s
Penetration Testing Framework ✅ Pass 6.009s
Vulnerability Assessment Tools ✅ Pass 6.916s

Summary: 25/25 (100.00%) successful tests

Average latency: 3.003s


enricher (qwen3.6-flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.177s
Text Transform Uppercase ✅ Pass 0.173s
Count from 1 to 5 ✅ Pass 0.173s
Math Calculation ✅ Pass 0.170s
Basic Echo Function ✅ Pass 0.174s
Streaming Simple Math Streaming ✅ Pass 0.189s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.192s
Streaming Basic Echo Function Streaming ✅ Pass 0.196s
Answer Stops At The Output Limit ✅ Pass 30.426s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.226s
Search Query Function ✅ Pass 0.180s
Ask Advice Function ✅ Pass 0.171s
Streaming Search Query Function Streaming ✅ Pass 0.179s
Basic Context Memory Test ✅ Pass 0.178s
Function Argument Memory Test ✅ Pass 1.316s
Function Response Memory Test ✅ Pass 1.572s
Cybersecurity Workflow Memory Test ✅ Pass 2.876s
Penetration Testing Memory with Tool Call ✅ Pass 4.238s
Read a file, then edit it via unified diff ✅ Pass 3.747s
Penetration Testing Methodology ✅ Pass 0.176s
Vulnerability Assessment Tools ✅ Pass 0.185s
SQL Injection Attack Type ✅ Pass 0.179s
Penetration Testing Framework ✅ Pass 0.172s
Penetration Testing Tool Selection ✅ Pass 1.655s
Web Application Security Scanner ✅ Pass 5.413s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.174s


coder (deepseek-v4.1-flash)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.186s
Simple Math ✅ Pass 0.179s
Text Transform Uppercase ✅ Pass 0.170s
Count from 1 to 5 ✅ Pass 0.170s
Math Calculation ✅ Pass 0.168s
Basic Echo Function ✅ Pass 0.176s
Streaming Simple Math Streaming ✅ Pass 0.183s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.192s
Streaming Basic Echo Function Streaming ✅ Pass 0.182s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.187s
Search Query Function ✅ Pass 0.201s
Ask Advice Function ✅ Pass 0.220s
Streaming Search Query Function Streaming ✅ Pass 0.178s
Basic Context Memory Test ✅ Pass 0.198s
Function Response Memory Test ✅ Pass 0.879s
Function Argument Memory Test ✅ Pass 1.081s
Cybersecurity Workflow Memory Test ✅ Pass 1.925s
Penetration Testing Memory with Tool Call ✅ Pass 2.152s
Read a file, then edit it via unified diff ✅ Pass 0.361s
Penetration Testing Methodology ✅ Pass 0.180s
Vulnerability Assessment Tools ✅ Pass 0.172s
SQL Injection Attack Type ✅ Pass 0.176s
Penetration Testing Framework ✅ Pass 0.177s
Web Application Security Scanner ✅ Pass 0.182s
Penetration Testing Tool Selection ✅ Pass 0.192s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.403s


installer (deepseek-v4.1-flash)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.194s
Simple Math ✅ Pass 0.181s
Text Transform Uppercase ✅ Pass 0.180s
Count from 1 to 5 ✅ Pass 0.187s
Math Calculation ✅ Pass 0.175s
Basic Echo Function ✅ Pass 0.180s
Streaming Simple Math Streaming ✅ Pass 0.178s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.178s
Streaming Basic Echo Function Streaming ✅ Pass 0.196s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.178s
Search Query Function ✅ Pass 0.175s
Ask Advice Function ✅ Pass 0.182s
Streaming Search Query Function Streaming ✅ Pass 0.175s
Basic Context Memory Test ✅ Pass 0.181s
Function Argument Memory Test ✅ Pass 1.154s
Function Response Memory Test ✅ Pass 1.588s
Cybersecurity Workflow Memory Test ✅ Pass 1.210s
Penetration Testing Memory with Tool Call ✅ Pass 1.952s
Read a file, then edit it via unified diff ✅ Pass 0.379s
Penetration Testing Methodology ✅ Pass 0.184s
Vulnerability Assessment Tools ✅ Pass 0.177s
SQL Injection Attack Type ✅ Pass 0.171s
Penetration Testing Framework ✅ Pass 0.177s
Web Application Security Scanner ✅ Pass 0.183s
Penetration Testing Tool Selection ✅ Pass 0.184s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.396s


pentester (deepseek-v4.1-flash)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.179s
Simple Math ✅ Pass 0.186s
Text Transform Uppercase ✅ Pass 0.176s
Count from 1 to 5 ✅ Pass 0.185s
Math Calculation ✅ Pass 0.180s
Basic Echo Function ✅ Pass 0.183s
Streaming Simple Math Streaming ✅ Pass 0.183s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.184s
Streaming Basic Echo Function Streaming ✅ Pass 0.181s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.191s
Search Query Function ✅ Pass 0.275s
Ask Advice Function ✅ Pass 0.246s
Streaming Search Query Function Streaming ✅ Pass 0.263s
Basic Context Memory Test ✅ Pass 0.257s
Function Response Memory Test ✅ Pass 1.248s
Cybersecurity Workflow Memory Test ✅ Pass 1.151s
Function Argument Memory Test ✅ Pass 1.473s
Penetration Testing Memory with Tool Call ✅ Pass 1.587s
Read a file, then edit it via unified diff ✅ Pass 0.373s
Penetration Testing Methodology ✅ Pass 0.176s
Vulnerability Assessment Tools ✅ Pass 0.174s
SQL Injection Attack Type ✅ Pass 0.176s
Penetration Testing Framework ✅ Pass 0.211s
Web Application Security Scanner ✅ Pass 0.247s
Penetration Testing Tool Selection ✅ Pass 0.232s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.397s