Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

21.1 KB
Raw Blame

LLM Agent Testing Report

Generated: Thu, 24 Sep 2026 15:03:55 UTC

Overall Results

Agent Model Reasoning Success Rate Average Latency
simple minimax-m2.7:cloud true 25/25 (100.00%) 5.001s
simple_json deepseek-v4.1-flash:cloud true 8/8 (100.00%) 0.866s
primary_agent deepseek-v4.1-flash:cloud true 25/25 (100.00%) 1.464s
assistant deepseek-v4.1-flash:cloud true 25/25 (100.00%) 1.601s
generator kimi-k3:cloud true 25/25 (100.00%) 4.148s
refiner glm-5.3:cloud true 25/25 (100.00%) 3.219s
adviser kimi-k3:cloud true 16/16 (100.00%) 4.620s
reflector minimax-m2.7:cloud true 16/16 (100.00%) 6.385s
searcher deepseek-v4.1-flash:cloud true 25/25 (100.00%) 1.513s
enricher minimax-m2.7:cloud true 25/25 (100.00%) 4.791s
coder kimi-k2.7-code:cloud true 25/25 (100.00%) 3.480s
installer kimi-k2.7-code:cloud true 25/25 (100.00%) 3.590s
pentester deepseek-v4.1-flash:cloud true 25/25 (100.00%) 1.670s

Total: 290/290 (100.00%) successful tests Overall average latency: 3.258s

Detailed Results

simple (minimax-m2.7:cloud)

Basic Tests

Test Result Latency Error
Math Calculation ✅ Pass 1.957s
Simple Math ✅ Pass 2.207s
Basic Echo Function ✅ Pass 2.697s
Text Transform Uppercase ✅ Pass 3.162s
Streaming Simple Math Streaming ✅ Pass 3.544s
Streaming Count from 1 to 3 Streaming ✅ Pass 4.148s
Streaming Basic Echo Function Streaming ✅ Pass 2.380s
Count from 1 to 5 ✅ Pass 4.995s
Answer Stops At The Output Limit ✅ Pass 40.062s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.971s
Search Query Function ✅ Pass 2.439s
Streaming Search Query Function Streaming ✅ Pass 1.911s
Function Argument Memory Test ✅ Pass 1.563s
Ask Advice Function ✅ Pass 4.634s
Basic Context Memory Test ✅ Pass 3.336s
Function Response Memory Test ✅ Pass 2.085s
Cybersecurity Workflow Memory Test ✅ Pass 2.431s
Penetration Testing Memory with Tool Call ✅ Pass 4.455s
Read a file, then edit it via unified diff ✅ Pass 6.708s
SQL Injection Attack Type ✅ Pass 2.331s
Penetration Testing Methodology ✅ Pass 3.296s
Penetration Testing Tool Selection ✅ Pass 1.966s
Web Application Security Scanner ✅ Pass 3.739s
Penetration Testing Framework ✅ Pass 6.330s
Vulnerability Assessment Tools ✅ Pass 10.669s

Summary: 25/25 (100.00%) successful tests

Average latency: 5.001s


simple_json (deepseek-v4.1-flash:cloud)

Advanced Tests

Test Result Latency Error
Vulnerability Report Memory Test ✅ Pass 1.258s
Person Information JSON ✅ Pass 1.262s
Project Information JSON ✅ Pass 0.905s
User Profile JSON ✅ Pass 0.983s
JSON Array Response Without Schema ✅ Pass 0.994s
Streaming Person Information JSON Streaming ✅ Pass 0.854s

Capability Tests

Test Capability Result Latency Note
Structured Output Refuses A Non-Object Schema structured_output ✅ Pass 0.000s
Structured Output With JSON Schema structured_output ✅ Pass 0.670s

Summary: 8/8 (100.00%) successful tests

Average latency: 0.866s


primary_agent (deepseek-v4.1-flash:cloud)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 0.709s
Count from 1 to 5 ✅ Pass 0.537s
Math Calculation ✅ Pass 0.486s
Simple Math ✅ Pass 1.699s
Streaming Simple Math Streaming ✅ Pass 1.087s
Basic Echo Function ✅ Pass 1.170s
Streaming Basic Echo Function Streaming ✅ Pass 0.920s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.307s
Answer Stops At The Output Limit ✅ Pass 8.808s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.881s
Search Query Function ✅ Pass 0.916s
Streaming Search Query Function Streaming ✅ Pass 1.064s
Ask Advice Function ✅ Pass 1.190s
Basic Context Memory Test ✅ Pass 0.913s
Function Argument Memory Test ✅ Pass 0.873s
Function Response Memory Test ✅ Pass 1.004s
Penetration Testing Memory with Tool Call ✅ Pass 1.162s
Cybersecurity Workflow Memory Test ✅ Pass 1.208s
Read a file, then edit it via unified diff ✅ Pass 2.617s
Penetration Testing Methodology ✅ Pass 1.198s
SQL Injection Attack Type ✅ Pass 0.957s
Penetration Testing Framework ✅ Pass 0.794s
Web Application Security Scanner ✅ Pass 0.827s
Penetration Testing Tool Selection ✅ Pass 0.979s
Vulnerability Assessment Tools ✅ Pass 3.292s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.464s


assistant (deepseek-v4.1-flash:cloud)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.994s
Math Calculation ✅ Pass 0.810s
Count from 1 to 5 ✅ Pass 0.850s
Text Transform Uppercase ✅ Pass 0.883s
Basic Echo Function ✅ Pass 0.898s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.065s
Streaming Basic Echo Function Streaming ✅ Pass 1.126s
Streaming Simple Math Streaming ✅ Pass 1.872s
Answer Stops At The Output Limit ✅ Pass 11.557s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.862s
Search Query Function ✅ Pass 0.965s
Ask Advice Function ✅ Pass 0.898s
Function Argument Memory Test ✅ Pass 0.878s
Basic Context Memory Test ✅ Pass 1.056s
Streaming Search Query Function Streaming ✅ Pass 1.422s
Penetration Testing Memory with Tool Call ✅ Pass 1.021s
Cybersecurity Workflow Memory Test ✅ Pass 0.937s
Function Response Memory Test ✅ Pass 1.923s
Read a file, then edit it via unified diff ✅ Pass 2.543s
Penetration Testing Methodology ✅ Pass 0.880s
SQL Injection Attack Type ✅ Pass 1.084s
Penetration Testing Framework ✅ Pass 1.084s
Web Application Security Scanner ✅ Pass 1.022s
Penetration Testing Tool Selection ✅ Pass 1.014s
Vulnerability Assessment Tools ✅ Pass 2.373s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.601s


generator (kimi-k3:cloud)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 1.654s
Text Transform Uppercase ✅ Pass 1.859s
Math Calculation ✅ Pass 1.850s
Count from 1 to 5 ✅ Pass 2.583s
Basic Echo Function ✅ Pass 2.677s
Streaming Simple Math Streaming ✅ Pass 1.915s
Streaming Count from 1 to 3 Streaming ✅ Pass 2.049s
Streaming Basic Echo Function Streaming ✅ Pass 2.437s
Answer Stops At The Output Limit ✅ Pass 28.241s

Advanced Tests

Test Result Latency Error
Ask Advice Function ✅ Pass 1.701s
Search Query Function ✅ Pass 2.054s
JSON Response Function ✅ Pass 3.022s
Streaming Search Query Function Streaming ✅ Pass 2.799s
Function Argument Memory Test ✅ Pass 2.445s
Basic Context Memory Test ✅ Pass 3.030s
Function Response Memory Test ✅ Pass 2.246s
Cybersecurity Workflow Memory Test ✅ Pass 2.040s
Penetration Testing Memory with Tool Call ✅ Pass 3.903s
Read a file, then edit it via unified diff ✅ Pass 7.926s
SQL Injection Attack Type ✅ Pass 2.150s
Penetration Testing Tool Selection ✅ Pass 2.769s
Web Application Security Scanner ✅ Pass 3.868s
Vulnerability Assessment Tools ✅ Pass 4.743s
Penetration Testing Framework ✅ Pass 5.495s
Penetration Testing Methodology ✅ Pass 8.240s

Summary: 25/25 (100.00%) successful tests

Average latency: 4.148s


refiner (glm-5.3:cloud)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.891s
Text Transform Uppercase ✅ Pass 0.841s
Basic Echo Function ✅ Pass 0.939s
Streaming Simple Math Streaming ✅ Pass 0.798s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.767s
Streaming Basic Echo Function Streaming ✅ Pass 0.567s
Math Calculation ✅ Pass 2.075s
Count from 1 to 5 ✅ Pass 2.967s
Answer Stops At The Output Limit ✅ Pass 18.425s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 1.095s
Search Query Function ✅ Pass 1.395s
Ask Advice Function ✅ Pass 1.287s
Streaming Search Query Function Streaming ✅ Pass 1.168s
Function Response Memory Test ✅ Pass 1.210s
Basic Context Memory Test ✅ Pass 1.838s
Function Argument Memory Test ✅ Pass 1.660s
Cybersecurity Workflow Memory Test ✅ Pass 1.624s
Penetration Testing Memory with Tool Call ✅ Pass 2.276s
Read a file, then edit it via unified diff ✅ Pass 12.822s
Penetration Testing Tool Selection ✅ Pass 1.415s
SQL Injection Attack Type ✅ Pass 3.337s
Vulnerability Assessment Tools ✅ Pass 5.064s
Penetration Testing Methodology ✅ Pass 6.966s
Web Application Security Scanner ✅ Pass 3.481s
Penetration Testing Framework ✅ Pass 5.556s

Summary: 25/25 (100.00%) successful tests

Average latency: 3.219s


adviser (kimi-k3:cloud)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 1.408s
Simple Math ✅ Pass 1.719s
Count from 1 to 5 ✅ Pass 2.224s
Streaming Simple Math Streaming ✅ Pass 2.103s
Streaming Count from 1 to 3 Streaming ✅ Pass 2.224s
Math Calculation ✅ Pass 3.490s
Answer Stops At The Output Limit ✅ Pass 20.467s

Advanced Tests

Test Result Latency Error
Function Response Memory Test ✅ Pass 1.418s
Function Argument Memory Test ✅ Pass 1.726s
Basic Context Memory Test ✅ Pass 2.429s
Cybersecurity Workflow Memory Test ✅ Pass 1.995s
Penetration Testing Framework ✅ Pass 3.657s
SQL Injection Attack Type ✅ Pass 5.194s
Web Application Security Scanner ✅ Pass 5.370s
Vulnerability Assessment Tools ✅ Pass 8.448s
Penetration Testing Methodology ✅ Pass 10.040s

Summary: 16/16 (100.00%) successful tests

Average latency: 4.620s


reflector (minimax-m2.7:cloud)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 3.125s
Text Transform Uppercase ✅ Pass 4.134s
Math Calculation ✅ Pass 2.719s
Count from 1 to 5 ✅ Pass 4.735s
Streaming Simple Math Streaming ✅ Pass 2.812s
Streaming Count from 1 to 3 Streaming ✅ Pass 3.435s
Answer Stops At The Output Limit ✅ Pass 38.390s

Advanced Tests

Test Result Latency Error
Function Argument Memory Test ✅ Pass 2.488s
Function Response Memory Test ✅ Pass 2.181s
Basic Context Memory Test ✅ Pass 2.838s
Cybersecurity Workflow Memory Test ✅ Pass 2.884s
Penetration Testing Methodology ✅ Pass 4.957s
Web Application Security Scanner ✅ Pass 3.049s
Penetration Testing Framework ✅ Pass 6.725s
SQL Injection Attack Type ✅ Pass 7.663s
Vulnerability Assessment Tools ✅ Pass 10.013s

Summary: 16/16 (100.00%) successful tests

Average latency: 6.385s


searcher (deepseek-v4.1-flash:cloud)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.696s
Text Transform Uppercase ✅ Pass 0.913s
Count from 1 to 5 ✅ Pass 0.881s
Math Calculation ✅ Pass 1.050s
Basic Echo Function ✅ Pass 0.790s
Streaming Simple Math Streaming ✅ Pass 1.262s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.740s
Streaming Basic Echo Function Streaming ✅ Pass 0.874s
Answer Stops At The Output Limit ✅ Pass 11.051s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.883s
Search Query Function ✅ Pass 0.875s
Ask Advice Function ✅ Pass 0.877s
Streaming Search Query Function Streaming ✅ Pass 0.941s
Basic Context Memory Test ✅ Pass 1.016s
Function Argument Memory Test ✅ Pass 0.853s
Function Response Memory Test ✅ Pass 0.865s
Cybersecurity Workflow Memory Test ✅ Pass 1.337s
Penetration Testing Memory with Tool Call ✅ Pass 1.423s
Read a file, then edit it via unified diff ✅ Pass 2.767s
Penetration Testing Methodology ✅ Pass 0.893s
Vulnerability Assessment Tools ✅ Pass 1.638s
Penetration Testing Framework ✅ Pass 1.075s
SQL Injection Attack Type ✅ Pass 1.672s
Web Application Security Scanner ✅ Pass 1.191s
Penetration Testing Tool Selection ✅ Pass 1.239s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.513s


enricher (minimax-m2.7:cloud)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 3.365s
Text Transform Uppercase ✅ Pass 5.042s
Math Calculation ✅ Pass 1.790s
Basic Echo Function ✅ Pass 2.931s
Streaming Simple Math Streaming ✅ Pass 2.882s
Streaming Basic Echo Function Streaming ✅ Pass 3.117s
Count from 1 to 5 ✅ Pass 6.409s
Streaming Count from 1 to 3 Streaming ✅ Pass 4.471s
Answer Stops At The Output Limit ✅ Pass 32.604s

Advanced Tests

Test Result Latency Error
Streaming Search Query Function Streaming ✅ Pass 1.646s
JSON Response Function ✅ Pass 2.072s
Ask Advice Function ✅ Pass 1.935s
Search Query Function ✅ Pass 2.183s
Basic Context Memory Test ✅ Pass 1.978s
Function Argument Memory Test ✅ Pass 2.070s
Function Response Memory Test ✅ Pass 2.421s
Cybersecurity Workflow Memory Test ✅ Pass 2.660s
Penetration Testing Memory with Tool Call ✅ Pass 3.430s
Read a file, then edit it via unified diff ✅ Pass 6.451s
Penetration Testing Framework ✅ Pass 4.251s
Web Application Security Scanner ✅ Pass 4.026s
SQL Injection Attack Type ✅ Pass 4.742s
Penetration Testing Tool Selection ✅ Pass 3.310s
Penetration Testing Methodology ✅ Pass 5.391s
Vulnerability Assessment Tools ✅ Pass 8.581s

Summary: 25/25 (100.00%) successful tests

Average latency: 4.791s


coder (kimi-k2.7-code:cloud)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.218s
Text Transform Uppercase ✅ Pass 1.726s
Math Calculation ✅ Pass 1.268s
Count from 1 to 5 ✅ Pass 2.062s
Basic Echo Function ✅ Pass 1.658s
Streaming Simple Math Streaming ✅ Pass 1.872s
Streaming Count from 1 to 3 Streaming ✅ Pass 1.624s
Streaming Basic Echo Function Streaming ✅ Pass 1.979s
Answer Stops At The Output Limit ✅ Pass 28.503s

Advanced Tests

Test Result Latency Error
Streaming Search Query Function Streaming ✅ Pass 1.394s
Basic Context Memory Test ✅ Pass 1.889s
Search Query Function ✅ Pass 3.072s
JSON Response Function ✅ Pass 3.297s
Function Argument Memory Test ✅ Pass 1.666s
Ask Advice Function ✅ Pass 3.150s
Function Response Memory Test ✅ Pass 1.500s
Cybersecurity Workflow Memory Test ✅ Pass 3.135s
Penetration Testing Memory with Tool Call ✅ Pass 3.549s
Read a file, then edit it via unified diff ✅ Pass 4.901s
Penetration Testing Methodology ✅ Pass 2.853s
SQL Injection Attack Type ✅ Pass 1.727s
Web Application Security Scanner ✅ Pass 2.009s
Penetration Testing Tool Selection ✅ Pass 2.031s
Vulnerability Assessment Tools ✅ Pass 4.971s
Penetration Testing Framework ✅ Pass 2.941s

Summary: 25/25 (100.00%) successful tests

Average latency: 3.480s


installer (kimi-k2.7-code:cloud)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 2.114s
Text Transform Uppercase ✅ Pass 1.722s
Count from 1 to 5 ✅ Pass 1.883s
Math Calculation ✅ Pass 2.081s
Streaming Simple Math Streaming ✅ Pass 1.384s
Basic Echo Function ✅ Pass 2.200s
Streaming Count from 1 to 3 Streaming ✅ Pass 2.295s
Streaming Basic Echo Function Streaming ✅ Pass 2.295s
Answer Stops At The Output Limit ✅ Pass 25.733s

Advanced Tests

Test Result Latency Error
Streaming Search Query Function Streaming ✅ Pass 1.785s
Search Query Function ✅ Pass 1.864s
JSON Response Function ✅ Pass 2.264s
Function Argument Memory Test ✅ Pass 1.679s
Basic Context Memory Test ✅ Pass 2.324s
Ask Advice Function ✅ Pass 2.448s
Penetration Testing Memory with Tool Call ✅ Pass 1.888s
Cybersecurity Workflow Memory Test ✅ Pass 1.914s
Function Response Memory Test ✅ Pass 2.119s
Read a file, then edit it via unified diff ✅ Pass 6.766s
Penetration Testing Methodology ✅ Pass 3.105s
Penetration Testing Tool Selection ✅ Pass 2.039s
SQL Injection Attack Type ✅ Pass 3.989s
Penetration Testing Framework ✅ Pass 3.590s
Web Application Security Scanner ✅ Pass 3.788s
Vulnerability Assessment Tools ✅ Pass 6.472s

Summary: 25/25 (100.00%) successful tests

Average latency: 3.590s


pentester (deepseek-v4.1-flash:cloud)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.813s
Count from 1 to 5 ✅ Pass 0.829s
Text Transform Uppercase ✅ Pass 1.220s
Math Calculation ✅ Pass 1.242s
Basic Echo Function ✅ Pass 1.170s
Streaming Simple Math Streaming ✅ Pass 1.136s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.854s
Streaming Basic Echo Function Streaming ✅ Pass 1.165s
Answer Stops At The Output Limit ✅ Pass 13.063s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.870s
Streaming Search Query Function Streaming ✅ Pass 1.140s
Ask Advice Function ✅ Pass 1.415s
Function Argument Memory Test ✅ Pass 1.036s
Basic Context Memory Test ✅ Pass 1.157s
Search Query Function ✅ Pass 1.822s
Function Response Memory Test ✅ Pass 0.971s
Cybersecurity Workflow Memory Test ✅ Pass 1.154s
Penetration Testing Memory with Tool Call ✅ Pass 1.518s
Read a file, then edit it via unified diff ✅ Pass 2.106s
Penetration Testing Methodology ✅ Pass 0.937s
Penetration Testing Framework ✅ Pass 0.773s
SQL Injection Attack Type ✅ Pass 1.258s
Web Application Security Scanner ✅ Pass 0.740s
Vulnerability Assessment Tools ✅ Pass 2.062s
Penetration Testing Tool Selection ✅ Pass 1.275s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.670s