Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

21.0 KB
Raw Blame

LLM Agent Testing Report

Generated: Thu, 24 Sep 2026 21:36:36 UTC

Overall Results

Agent Model Reasoning Success Rate Average Latency
simple qwen3.6-flash true 25/25 (100.00%) 4.036s
simple_json qwen3.6-flash true 8/8 (100.00%) 4.661s
primary_agent deepseek-v4.1-flash true 25/25 (100.00%) 0.445s
assistant deepseek-v4.1-flash true 25/25 (100.00%) 0.365s
generator deepseek-v4-pro true 25/25 (100.00%) 4.895s
refiner deepseek-v4-pro true 25/25 (100.00%) 4.816s
adviser deepseek-v4-pro true 16/16 (100.00%) 5.506s
reflector qwen3.6-flash true 16/16 (100.00%) 4.939s
searcher qwen3.6-flash true 25/25 (100.00%) 4.423s
enricher qwen3.6-flash true 25/25 (100.00%) 2.024s
coder deepseek-v4.1-flash true 25/25 (100.00%) 0.410s
installer deepseek-v4.1-flash true 25/25 (100.00%) 0.473s
pentester deepseek-v4.1-flash true 25/25 (100.00%) 0.528s

Total: 290/290 (100.00%) successful tests Overall average latency: 2.637s

Detailed Results

simple (qwen3.6-flash)

Basic Tests

Test Result Latency Error
Math Calculation ✅ Pass 1.663s
Streaming Simple Math Streaming ✅ Pass 1.773s
Basic Echo Function ✅ Pass 1.873s
Simple Math ✅ Pass 1.981s
Streaming Count from 1 to 3 Streaming ✅ Pass 2.119s
Count from 1 to 5 ✅ Pass 2.156s
Text Transform Uppercase ✅ Pass 2.501s
Streaming Basic Echo Function Streaming ✅ Pass 0.894s
Answer Stops At The Output Limit ✅ Pass 34.420s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.397s
Search Query Function ✅ Pass 1.267s
Ask Advice Function ✅ Pass 1.279s
Streaming Search Query Function Streaming ✅ Pass 1.522s
Function Argument Memory Test ✅ Pass 1.873s
Basic Context Memory Test ✅ Pass 2.137s
Function Response Memory Test ✅ Pass 1.033s
Cybersecurity Workflow Memory Test ✅ Pass 1.759s
Penetration Testing Memory with Tool Call ✅ Pass 4.779s
Read a file, then edit it via unified diff ✅ Pass 3.338s
SQL Injection Attack Type ✅ Pass 3.959s
Penetration Testing Tool Selection ✅ Pass 2.186s
Penetration Testing Methodology ✅ Pass 5.237s
Penetration Testing Framework ✅ Pass 5.403s
Vulnerability Assessment Tools ✅ Pass 7.659s
Web Application Security Scanner ✅ Pass 6.669s

Summary: 25/25 (100.00%) successful tests

Average latency: 4.036s


simple_json (qwen3.6-flash)

Advanced Tests

Test Result Latency Error
Vulnerability Report Memory Test ✅ Pass 3.028s
Person Information JSON ✅ Pass 5.970s
Project Information JSON ✅ Pass 5.782s
User Profile JSON ✅ Pass 6.000s
JSON Array Response Without Schema ✅ Pass 6.405s
Streaming Person Information JSON Streaming ✅ Pass 3.324s

Capability Tests

Test Capability Result Latency Note
Structured Output Refuses A Non-Object Schema structured_output ✅ Pass 0.000s
Structured Output With JSON Schema structured_output ✅ Pass 6.778s

Summary: 8/8 (100.00%) successful tests

Average latency: 4.661s


primary_agent (deepseek-v4.1-flash)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.276s
Simple Math ✅ Pass 0.278s
Text Transform Uppercase ✅ Pass 0.204s
Count from 1 to 5 ✅ Pass 0.172s
Math Calculation ✅ Pass 0.185s
Basic Echo Function ✅ Pass 0.187s
Streaming Simple Math Streaming ✅ Pass 0.190s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.186s
Streaming Basic Echo Function Streaming ✅ Pass 0.171s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.183s
Search Query Function ✅ Pass 0.182s
Ask Advice Function ✅ Pass 0.175s
Streaming Search Query Function Streaming ✅ Pass 0.188s
Basic Context Memory Test ✅ Pass 0.202s
Function Argument Memory Test ✅ Pass 1.065s
Penetration Testing Memory with Tool Call ✅ Pass 1.379s
Cybersecurity Workflow Memory Test ✅ Pass 1.175s
Function Response Memory Test ✅ Pass 3.262s
Read a file, then edit it via unified diff ✅ Pass 0.351s
Penetration Testing Methodology ✅ Pass 0.191s
Vulnerability Assessment Tools ✅ Pass 0.186s
SQL Injection Attack Type ✅ Pass 0.186s
Penetration Testing Framework ✅ Pass 0.174s
Web Application Security Scanner ✅ Pass 0.174s
Penetration Testing Tool Selection ✅ Pass 0.178s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.445s


assistant (deepseek-v4.1-flash)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.175s
Simple Math ✅ Pass 0.183s
Text Transform Uppercase ✅ Pass 0.185s
Count from 1 to 5 ✅ Pass 0.202s
Math Calculation ✅ Pass 0.195s
Basic Echo Function ✅ Pass 0.189s
Streaming Simple Math Streaming ✅ Pass 0.190s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.191s
Streaming Basic Echo Function Streaming ✅ Pass 0.189s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.191s
Search Query Function ✅ Pass 0.180s
Ask Advice Function ✅ Pass 0.174s
Streaming Search Query Function Streaming ✅ Pass 0.214s
Basic Context Memory Test ✅ Pass 0.172s
Function Argument Memory Test ✅ Pass 1.075s
Function Response Memory Test ✅ Pass 1.159s
Penetration Testing Memory with Tool Call ✅ Pass 1.802s
Cybersecurity Workflow Memory Test ✅ Pass 1.032s
Read a file, then edit it via unified diff ✅ Pass 0.359s
Penetration Testing Methodology ✅ Pass 0.179s
Vulnerability Assessment Tools ✅ Pass 0.171s
SQL Injection Attack Type ✅ Pass 0.172s
Penetration Testing Framework ✅ Pass 0.173s
Web Application Security Scanner ✅ Pass 0.179s
Penetration Testing Tool Selection ✅ Pass 0.184s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.365s


generator (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Math Calculation ✅ Pass 2.014s
Streaming Simple Math Streaming ✅ Pass 2.251s
Text Transform Uppercase ✅ Pass 2.398s
Simple Math ✅ Pass 2.697s
Basic Echo Function ✅ Pass 2.564s
Count from 1 to 5 ✅ Pass 2.891s
Streaming Count from 1 to 3 Streaming ✅ Pass 2.444s
Streaming Basic Echo Function Streaming ✅ Pass 2.617s
Answer Stops At The Output Limit ✅ Pass 38.434s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 2.909s
Search Query Function ✅ Pass 2.952s
Streaming Search Query Function Streaming ✅ Pass 3.085s
Ask Advice Function ✅ Pass 3.255s
Function Argument Memory Test ✅ Pass 2.488s
Function Response Memory Test ✅ Pass 2.593s
Basic Context Memory Test ✅ Pass 3.587s
Cybersecurity Workflow Memory Test ✅ Pass 2.752s
Penetration Testing Memory with Tool Call ✅ Pass 4.463s
Read a file, then edit it via unified diff ✅ Pass 8.599s
SQL Injection Attack Type ✅ Pass 3.052s
Web Application Security Scanner ✅ Pass 2.668s
Penetration Testing Framework ✅ Pass 4.282s
Penetration Testing Methodology ✅ Pass 5.527s
Penetration Testing Tool Selection ✅ Pass 3.487s
Vulnerability Assessment Tools ✅ Pass 8.355s

Summary: 25/25 (100.00%) successful tests

Average latency: 4.895s


refiner (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 2.098s
Simple Math ✅ Pass 2.240s
Count from 1 to 5 ✅ Pass 2.959s
Math Calculation ✅ Pass 2.459s
Streaming Count from 1 to 3 Streaming ✅ Pass 2.270s
Streaming Simple Math Streaming ✅ Pass 2.473s
Basic Echo Function ✅ Pass 2.976s
Streaming Basic Echo Function Streaming ✅ Pass 3.131s
Answer Stops At The Output Limit ✅ Pass 37.894s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 3.253s
Search Query Function ✅ Pass 2.860s
Ask Advice Function ✅ Pass 3.151s
Streaming Search Query Function Streaming ✅ Pass 3.234s
Function Argument Memory Test ✅ Pass 2.201s
Basic Context Memory Test ✅ Pass 3.360s
Function Response Memory Test ✅ Pass 2.393s
Cybersecurity Workflow Memory Test ✅ Pass 2.554s
Penetration Testing Memory with Tool Call ✅ Pass 5.153s
Read a file, then edit it via unified diff ✅ Pass 8.596s
Vulnerability Assessment Tools ✅ Pass 3.236s
SQL Injection Attack Type ✅ Pass 3.242s
Web Application Security Scanner ✅ Pass 3.363s
Penetration Testing Methodology ✅ Pass 5.412s
Penetration Testing Tool Selection ✅ Pass 4.122s
Penetration Testing Framework ✅ Pass 5.761s

Summary: 25/25 (100.00%) successful tests

Average latency: 4.816s


adviser (deepseek-v4-pro)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.563s
Text Transform Uppercase ✅ Pass 2.611s
Count from 1 to 5 ✅ Pass 2.436s
Math Calculation ✅ Pass 2.575s
Streaming Simple Math Streaming ✅ Pass 2.611s
Streaming Count from 1 to 3 Streaming ✅ Pass 2.721s
Answer Stops At The Output Limit ✅ Pass 37.543s

Advanced Tests

Test Result Latency Error
Function Argument Memory Test ✅ Pass 2.603s
Basic Context Memory Test ✅ Pass 3.501s
Function Response Memory Test ✅ Pass 2.622s
Cybersecurity Workflow Memory Test ✅ Pass 2.873s
Penetration Testing Methodology ✅ Pass 4.078s
Web Application Security Scanner ✅ Pass 2.632s
SQL Injection Attack Type ✅ Pass 3.766s
Penetration Testing Framework ✅ Pass 4.184s
Vulnerability Assessment Tools ✅ Pass 8.769s

Summary: 16/16 (100.00%) successful tests

Average latency: 5.506s


reflector (qwen3.6-flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.581s
Math Calculation ✅ Pass 1.543s
Text Transform Uppercase ✅ Pass 2.471s
Count from 1 to 5 ✅ Pass 2.436s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.540s
Streaming Simple Math Streaming ✅ Pass 1.735s
Answer Stops At The Output Limit ✅ Pass 35.745s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 2.094s
Function Argument Memory Test ✅ Pass 1.553s
Function Response Memory Test ✅ Pass 2.463s
Cybersecurity Workflow Memory Test ✅ Pass 2.082s
SQL Injection Attack Type ✅ Pass 2.363s
Penetration Testing Methodology ✅ Pass 4.835s
Web Application Security Scanner ✅ Pass 4.791s
Vulnerability Assessment Tools ✅ Pass 6.381s
Penetration Testing Framework ✅ Pass 5.406s

Summary: 16/16 (100.00%) successful tests

Average latency: 4.939s


searcher (qwen3.6-flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.180s
Text Transform Uppercase ✅ Pass 0.171s
Count from 1 to 5 ✅ Pass 0.178s
Math Calculation ✅ Pass 0.176s
Streaming Simple Math Streaming ✅ Pass 0.172s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.177s
Basic Echo Function ✅ Pass 1.256s
Streaming Basic Echo Function Streaming ✅ Pass 1.495s
Answer Stops At The Output Limit ✅ Pass 64.922s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 0.197s
Function Argument Memory Test ✅ Pass 1.052s
JSON Response Function ✅ Pass 1.950s
Search Query Function ✅ Pass 1.442s
Ask Advice Function ✅ Pass 1.546s
Streaming Search Query Function Streaming ✅ Pass 1.643s
Function Response Memory Test ✅ Pass 1.223s
Cybersecurity Workflow Memory Test ✅ Pass 1.796s
Penetration Testing Memory with Tool Call ✅ Pass 2.916s
Read a file, then edit it via unified diff ✅ Pass 3.697s
SQL Injection Attack Type ✅ Pass 0.180s
Penetration Testing Methodology ✅ Pass 4.381s
Penetration Testing Tool Selection ✅ Pass 1.415s
Web Application Security Scanner ✅ Pass 4.321s
Vulnerability Assessment Tools ✅ Pass 6.114s
Penetration Testing Framework ✅ Pass 7.960s

Summary: 25/25 (100.00%) successful tests

Average latency: 4.423s


enricher (qwen3.6-flash)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.179s
Text Transform Uppercase ✅ Pass 0.180s
Count from 1 to 5 ✅ Pass 0.179s
Math Calculation ✅ Pass 0.171s
Basic Echo Function ✅ Pass 0.170s
Streaming Simple Math Streaming ✅ Pass 0.170s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.181s
Streaming Basic Echo Function Streaming ✅ Pass 0.184s
Answer Stops At The Output Limit ✅ Pass 36.049s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.181s
Search Query Function ✅ Pass 0.173s
Ask Advice Function ✅ Pass 0.198s
Streaming Search Query Function Streaming ✅ Pass 0.178s
Basic Context Memory Test ✅ Pass 0.175s
Function Argument Memory Test ✅ Pass 1.161s
Function Response Memory Test ✅ Pass 1.389s
Cybersecurity Workflow Memory Test ✅ Pass 1.446s
Penetration Testing Memory with Tool Call ✅ Pass 2.673s
Read a file, then edit it via unified diff ✅ Pass 3.092s
Penetration Testing Methodology ✅ Pass 0.182s
Vulnerability Assessment Tools ✅ Pass 0.176s
SQL Injection Attack Type ✅ Pass 0.178s
Penetration Testing Framework ✅ Pass 0.176s
Web Application Security Scanner ✅ Pass 0.174s
Penetration Testing Tool Selection ✅ Pass 1.561s

Summary: 25/25 (100.00%) successful tests

Average latency: 2.024s


coder (deepseek-v4.1-flash)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.191s
Simple Math ✅ Pass 0.181s
Text Transform Uppercase ✅ Pass 0.197s
Count from 1 to 5 ✅ Pass 0.182s
Math Calculation ✅ Pass 0.181s
Basic Echo Function ✅ Pass 0.183s
Streaming Simple Math Streaming ✅ Pass 0.184s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.182s
Streaming Basic Echo Function Streaming ✅ Pass 0.180s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.180s
Search Query Function ✅ Pass 0.182s
Ask Advice Function ✅ Pass 0.179s
Streaming Search Query Function Streaming ✅ Pass 0.189s
Basic Context Memory Test ✅ Pass 0.180s
Function Response Memory Test ✅ Pass 1.068s
Function Argument Memory Test ✅ Pass 1.124s
Penetration Testing Memory with Tool Call ✅ Pass 1.797s
Cybersecurity Workflow Memory Test ✅ Pass 2.228s
Read a file, then edit it via unified diff ✅ Pass 0.379s
Penetration Testing Methodology ✅ Pass 0.184s
Vulnerability Assessment Tools ✅ Pass 0.168s
SQL Injection Attack Type ✅ Pass 0.177s
Penetration Testing Framework ✅ Pass 0.180s
Web Application Security Scanner ✅ Pass 0.176s
Penetration Testing Tool Selection ✅ Pass 0.183s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.410s


installer (deepseek-v4.1-flash)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.175s
Simple Math ✅ Pass 0.185s
Text Transform Uppercase ✅ Pass 0.181s
Count from 1 to 5 ✅ Pass 0.182s
Math Calculation ✅ Pass 0.178s
Basic Echo Function ✅ Pass 0.191s
Streaming Simple Math Streaming ✅ Pass 0.179s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.187s
Streaming Basic Echo Function Streaming ✅ Pass 0.187s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.171s
Search Query Function ✅ Pass 0.179s
Ask Advice Function ✅ Pass 0.177s
Streaming Search Query Function Streaming ✅ Pass 0.187s
Basic Context Memory Test ✅ Pass 0.280s
Function Response Memory Test ✅ Pass 1.289s
Function Argument Memory Test ✅ Pass 1.413s
Penetration Testing Memory with Tool Call ✅ Pass 2.242s
Cybersecurity Workflow Memory Test ✅ Pass 2.787s
Read a file, then edit it via unified diff ✅ Pass 0.365s
Penetration Testing Methodology ✅ Pass 0.189s
Vulnerability Assessment Tools ✅ Pass 0.187s
SQL Injection Attack Type ✅ Pass 0.177s
Penetration Testing Framework ✅ Pass 0.175s
Web Application Security Scanner ✅ Pass 0.170s
Penetration Testing Tool Selection ✅ Pass 0.177s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.473s


pentester (deepseek-v4.1-flash)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.174s
Simple Math ✅ Pass 0.180s
Text Transform Uppercase ✅ Pass 0.184s
Count from 1 to 5 ✅ Pass 0.180s
Math Calculation ✅ Pass 0.174s
Basic Echo Function ✅ Pass 0.177s
Streaming Simple Math Streaming ✅ Pass 0.177s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.177s
Streaming Basic Echo Function Streaming ✅ Pass 1.413s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.197s
Search Query Function ✅ Pass 0.185s
Ask Advice Function ✅ Pass 0.174s
Streaming Search Query Function Streaming ✅ Pass 0.189s
Basic Context Memory Test ✅ Pass 0.181s
Function Argument Memory Test ✅ Pass 1.151s
Function Response Memory Test ✅ Pass 1.194s
Penetration Testing Memory with Tool Call ✅ Pass 1.973s
Read a file, then edit it via unified diff ✅ Pass 0.365s
Cybersecurity Workflow Memory Test ✅ Pass 3.481s
Penetration Testing Methodology ✅ Pass 0.174s
Vulnerability Assessment Tools ✅ Pass 0.241s
SQL Injection Attack Type ✅ Pass 0.248s
Penetration Testing Framework ✅ Pass 0.237s
Web Application Security Scanner ✅ Pass 0.198s
Penetration Testing Tool Selection ✅ Pass 0.174s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.528s