Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

21.7 KB
Raw Blame

LLM Agent Testing Report

Generated: Thu, 24 Sep 2026 08:15:45 UTC

Overall Results

Agent Model Reasoning Success Rate Average Latency
simple kimi-k2.6 false 26/26 (100.00%) 0.484s
simple_json kimi-k2.6 false 8/8 (100.00%) 0.467s
primary_agent kimi-k2.7-code-highspeed true 25/25 (100.00%) 0.420s
assistant kimi-k2.7-code-highspeed true 25/25 (100.00%) 0.401s
generator kimi-k2.7-code true 25/25 (100.00%) 0.573s
refiner kimi-k2.7-code true 25/25 (100.00%) 0.519s
adviser kimi-k3 true 16/16 (100.00%) 1.082s
reflector kimi-k2.6 false 17/17 (100.00%) 0.663s
searcher kimi-k2.6 false 26/26 (100.00%) 0.499s
enricher kimi-k2.6 false 26/26 (100.00%) 0.481s
coder kimi-k2.7-code true 25/25 (100.00%) 0.613s
installer kimi-k2.6 true 25/25 (100.00%) 0.847s
pentester kimi-k2.7-code true 25/25 (100.00%) 0.564s

Total: 294/294 (100.00%) successful tests Overall average latency: 0.574s

Detailed Results

simple (kimi-k2.6)

Basic Tests

Test Result Latency Error
Count from 1 to 5 ✅ Pass 0.356s
Basic Echo Function ✅ Pass 0.356s
Streaming Simple Math Streaming ✅ Pass 0.364s
Math Calculation ✅ Pass 0.367s
Answer Stops At The Output Limit ✅ Pass 0.368s
Simple Math ✅ Pass 0.388s
Text Transform Uppercase ✅ Pass 0.388s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.407s
Streaming Basic Echo Function Streaming ✅ Pass 0.300s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.224s
Search Query Function ✅ Pass 0.225s
Ask Advice Function ✅ Pass 0.222s
Streaming Search Query Function Streaming ✅ Pass 0.228s
Basic Context Memory Test ✅ Pass 0.228s
Function Response Memory Test ✅ Pass 0.857s
Cybersecurity Workflow Memory Test ✅ Pass 1.041s
Function Argument Memory Test ✅ Pass 1.131s
Penetration Testing Memory with Tool Call ✅ Pass 2.945s
Read a file, then edit it via unified diff ✅ Pass 0.546s
Penetration Testing Methodology ✅ Pass 0.222s
Vulnerability Assessment Tools ✅ Pass 0.217s
SQL Injection Attack Type ✅ Pass 0.233s
Penetration Testing Framework ✅ Pass 0.238s
Web Application Security Scanner ✅ Pass 0.228s
Penetration Testing Tool Selection ✅ Pass 0.234s

Capability Tests

Test Capability Result Latency Note
Explicit Reasoning Off Suppresses Reasoning reasoning_off ✅ Pass 0.250s

Summary: 26/26 (100.00%) successful tests

Average latency: 0.484s


simple_json (kimi-k2.6)

Advanced Tests

Test Result Latency Error
Vulnerability Report Memory Test ✅ Pass 2.286s
Person Information JSON ✅ Pass 0.226s
Project Information JSON ✅ Pass 0.233s
User Profile JSON ✅ Pass 0.241s
JSON Array Response Without Schema ✅ Pass 0.251s
Streaming Person Information JSON Streaming ✅ Pass 0.256s

Capability Tests

Test Capability Result Latency Note
Structured Output Refuses A Non-Object Schema structured_output ✅ Pass 0.000s
Structured Output With JSON Schema structured_output ✅ Pass 0.237s

Summary: 8/8 (100.00%) successful tests

Average latency: 0.467s


primary_agent (kimi-k2.7-code-highspeed)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.267s
Count from 1 to 5 ✅ Pass 0.288s
Text Transform Uppercase ✅ Pass 0.289s
Simple Math ✅ Pass 0.292s
Basic Echo Function ✅ Pass 0.294s
Math Calculation ✅ Pass 0.305s
Streaming Simple Math Streaming ✅ Pass 0.285s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.222s
Streaming Basic Echo Function Streaming ✅ Pass 0.304s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.215s
Search Query Function ✅ Pass 0.221s
Ask Advice Function ✅ Pass 0.219s
Streaming Search Query Function Streaming ✅ Pass 0.216s
Basic Context Memory Test ✅ Pass 0.232s
Function Response Memory Test ✅ Pass 1.081s
Function Argument Memory Test ✅ Pass 1.151s
Penetration Testing Memory with Tool Call ✅ Pass 1.486s
Cybersecurity Workflow Memory Test ✅ Pass 1.335s
Read a file, then edit it via unified diff ✅ Pass 0.461s
Penetration Testing Methodology ✅ Pass 0.225s
Vulnerability Assessment Tools ✅ Pass 0.229s
SQL Injection Attack Type ✅ Pass 0.217s
Penetration Testing Framework ✅ Pass 0.217s
Web Application Security Scanner ✅ Pass 0.214s
Penetration Testing Tool Selection ✅ Pass 0.228s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.420s


assistant (kimi-k2.7-code-highspeed)

Basic Tests

Test Result Latency Error
Text Transform Uppercase ✅ Pass 0.261s
Simple Math ✅ Pass 0.267s
Answer Stops At The Output Limit ✅ Pass 0.269s
Count from 1 to 5 ✅ Pass 0.268s
Math Calculation ✅ Pass 0.268s
Basic Echo Function ✅ Pass 0.268s
Streaming Simple Math Streaming ✅ Pass 0.216s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.243s
Streaming Basic Echo Function Streaming ✅ Pass 0.240s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.228s
Search Query Function ✅ Pass 0.226s
Ask Advice Function ✅ Pass 0.222s
Streaming Search Query Function Streaming ✅ Pass 0.225s
Basic Context Memory Test ✅ Pass 0.231s
Function Argument Memory Test ✅ Pass 1.141s
Penetration Testing Memory with Tool Call ✅ Pass 1.117s
Function Response Memory Test ✅ Pass 1.203s
Cybersecurity Workflow Memory Test ✅ Pass 1.281s
Read a file, then edit it via unified diff ✅ Pass 0.462s
Penetration Testing Methodology ✅ Pass 0.227s
Vulnerability Assessment Tools ✅ Pass 0.237s
SQL Injection Attack Type ✅ Pass 0.227s
Penetration Testing Framework ✅ Pass 0.231s
Web Application Security Scanner ✅ Pass 0.222s
Penetration Testing Tool Selection ✅ Pass 0.225s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.401s


generator (kimi-k2.7-code)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.231s
Simple Math ✅ Pass 0.252s
Text Transform Uppercase ✅ Pass 0.246s
Math Calculation ✅ Pass 0.246s
Count from 1 to 5 ✅ Pass 0.250s
Basic Echo Function ✅ Pass 0.215s
Streaming Simple Math Streaming ✅ Pass 0.227s
Streaming Basic Echo Function Streaming ✅ Pass 0.234s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.237s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.226s
Search Query Function ✅ Pass 0.230s
Ask Advice Function ✅ Pass 0.235s
Streaming Search Query Function Streaming ✅ Pass 0.217s
Basic Context Memory Test ✅ Pass 0.219s
Function Argument Memory Test ✅ Pass 1.184s
Function Response Memory Test ✅ Pass 2.319s
Penetration Testing Memory with Tool Call ✅ Pass 2.611s
Cybersecurity Workflow Memory Test ✅ Pass 3.084s
Read a file, then edit it via unified diff ✅ Pass 0.439s
Penetration Testing Methodology ✅ Pass 0.218s
Vulnerability Assessment Tools ✅ Pass 0.226s
SQL Injection Attack Type ✅ Pass 0.238s
Penetration Testing Framework ✅ Pass 0.243s
Web Application Security Scanner ✅ Pass 0.248s
Penetration Testing Tool Selection ✅ Pass 0.242s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.573s


refiner (kimi-k2.7-code)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.245s
Simple Math ✅ Pass 0.242s
Text Transform Uppercase ✅ Pass 0.242s
Count from 1 to 5 ✅ Pass 0.243s
Math Calculation ✅ Pass 0.222s
Basic Echo Function ✅ Pass 0.237s
Streaming Simple Math Streaming ✅ Pass 0.233s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.240s
Streaming Basic Echo Function Streaming ✅ Pass 0.267s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.215s
Search Query Function ✅ Pass 0.219s
Ask Advice Function ✅ Pass 0.223s
Streaming Search Query Function Streaming ✅ Pass 0.221s
Basic Context Memory Test ✅ Pass 0.236s
Cybersecurity Workflow Memory Test ✅ Pass 1.043s
Function Response Memory Test ✅ Pass 1.986s
Penetration Testing Memory with Tool Call ✅ Pass 2.299s
Function Argument Memory Test ✅ Pass 2.587s
Read a file, then edit it via unified diff ✅ Pass 0.437s
Penetration Testing Methodology ✅ Pass 0.235s
Vulnerability Assessment Tools ✅ Pass 0.216s
SQL Injection Attack Type ✅ Pass 0.217s
Penetration Testing Framework ✅ Pass 0.220s
Web Application Security Scanner ✅ Pass 0.214s
Penetration Testing Tool Selection ✅ Pass 0.221s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.519s


adviser (kimi-k3)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.240s
Text Transform Uppercase ✅ Pass 0.252s
Answer Stops At The Output Limit ✅ Pass 0.257s
Count from 1 to 5 ✅ Pass 0.229s
Math Calculation ✅ Pass 0.216s
Streaming Simple Math Streaming ✅ Pass 0.225s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.236s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 0.222s
Function Response Memory Test ✅ Pass 3.922s
Function Argument Memory Test ✅ Pass 5.336s
Cybersecurity Workflow Memory Test ✅ Pass 4.419s
Penetration Testing Methodology ✅ Pass 0.232s
Vulnerability Assessment Tools ✅ Pass 0.228s
SQL Injection Attack Type ✅ Pass 0.231s
Penetration Testing Framework ✅ Pass 0.217s
Web Application Security Scanner ✅ Pass 0.848s

Summary: 16/16 (100.00%) successful tests

Average latency: 1.082s


reflector (kimi-k2.6)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.225s
Simple Math ✅ Pass 0.235s
Count from 1 to 5 ✅ Pass 0.245s
Text Transform Uppercase ✅ Pass 0.256s
Math Calculation ✅ Pass 0.246s
Streaming Simple Math Streaming ✅ Pass 0.245s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.348s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 0.221s
Function Response Memory Test ✅ Pass 0.964s
Cybersecurity Workflow Memory Test ✅ Pass 0.923s
Function Argument Memory Test ✅ Pass 3.387s
Vulnerability Assessment Tools ✅ Pass 0.781s
Web Application Security Scanner ✅ Pass 0.709s
Penetration Testing Framework ✅ Pass 0.719s
Penetration Testing Methodology ✅ Pass 0.788s
SQL Injection Attack Type ✅ Pass 0.737s

Capability Tests

Test Capability Result Latency Note
Explicit Reasoning Off Suppresses Reasoning reasoning_off ✅ Pass 0.225s

Summary: 17/17 (100.00%) successful tests

Average latency: 0.663s


searcher (kimi-k2.6)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.356s
Simple Math ✅ Pass 0.326s
Text Transform Uppercase ✅ Pass 0.364s
Count from 1 to 5 ✅ Pass 0.358s
Math Calculation ✅ Pass 0.359s
Basic Echo Function ✅ Pass 0.353s
Streaming Simple Math Streaming ✅ Pass 0.278s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.217s
Streaming Basic Echo Function Streaming ✅ Pass 0.229s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.226s
Search Query Function ✅ Pass 0.218s
Ask Advice Function ✅ Pass 0.226s
Streaming Search Query Function Streaming ✅ Pass 0.218s
Basic Context Memory Test ✅ Pass 0.233s
Function Response Memory Test ✅ Pass 1.032s
Cybersecurity Workflow Memory Test ✅ Pass 1.054s
Function Argument Memory Test ✅ Pass 1.322s
Penetration Testing Memory with Tool Call ✅ Pass 2.346s
Read a file, then edit it via unified diff ✅ Pass 0.450s
Penetration Testing Methodology ✅ Pass 0.699s
Vulnerability Assessment Tools ✅ Pass 0.691s
SQL Injection Attack Type ✅ Pass 0.286s
Penetration Testing Framework ✅ Pass 0.285s
Web Application Security Scanner ✅ Pass 0.285s
Penetration Testing Tool Selection ✅ Pass 0.311s

Capability Tests

Test Capability Result Latency Note
Explicit Reasoning Off Suppresses Reasoning reasoning_off ✅ Pass 0.225s

Summary: 26/26 (100.00%) successful tests

Average latency: 0.499s


enricher (kimi-k2.6)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.225s
Simple Math ✅ Pass 0.232s
Count from 1 to 5 ✅ Pass 0.241s
Text Transform Uppercase ✅ Pass 0.246s
Math Calculation ✅ Pass 0.247s
Basic Echo Function ✅ Pass 0.227s
Streaming Simple Math Streaming ✅ Pass 0.219s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.229s
Streaming Basic Echo Function Streaming ✅ Pass 0.234s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.217s
Search Query Function ✅ Pass 0.226s
Ask Advice Function ✅ Pass 0.235s
Streaming Search Query Function Streaming ✅ Pass 0.222s
Basic Context Memory Test ✅ Pass 0.225s
Function Response Memory Test ✅ Pass 0.949s
Cybersecurity Workflow Memory Test ✅ Pass 1.021s
Function Argument Memory Test ✅ Pass 2.457s
Penetration Testing Memory with Tool Call ✅ Pass 2.343s
Read a file, then edit it via unified diff ✅ Pass 0.443s
Vulnerability Assessment Tools ✅ Pass 0.297s
Penetration Testing Methodology ✅ Pass 0.311s
SQL Injection Attack Type ✅ Pass 0.315s
Penetration Testing Framework ✅ Pass 0.315s
Penetration Testing Tool Selection ✅ Pass 0.284s
Web Application Security Scanner ✅ Pass 0.311s

Capability Tests

Test Capability Result Latency Note
Explicit Reasoning Off Suppresses Reasoning reasoning_off ✅ Pass 0.222s

Summary: 26/26 (100.00%) successful tests

Average latency: 0.481s


coder (kimi-k2.7-code)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.217s
Text Transform Uppercase ✅ Pass 0.234s
Simple Math ✅ Pass 0.234s
Count from 1 to 5 ✅ Pass 0.242s
Math Calculation ✅ Pass 0.234s
Basic Echo Function ✅ Pass 0.221s
Streaming Simple Math Streaming ✅ Pass 0.220s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.217s
Streaming Basic Echo Function Streaming ✅ Pass 0.221s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.218s
Search Query Function ✅ Pass 0.218s
Ask Advice Function ✅ Pass 0.224s
Basic Context Memory Test ✅ Pass 0.215s
Streaming Search Query Function Streaming ✅ Pass 0.288s
Function Response Memory Test ✅ Pass 2.250s
Cybersecurity Workflow Memory Test ✅ Pass 2.347s
Penetration Testing Memory with Tool Call ✅ Pass 2.383s
Function Argument Memory Test ✅ Pass 2.552s
Read a file, then edit it via unified diff ✅ Pass 0.435s
Vulnerability Assessment Tools ✅ Pass 0.269s
Penetration Testing Framework ✅ Pass 0.329s
Web Application Security Scanner ✅ Pass 0.329s
SQL Injection Attack Type ✅ Pass 0.336s
Penetration Testing Tool Selection ✅ Pass 0.315s
Penetration Testing Methodology ✅ Pass 0.557s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.613s


installer (kimi-k2.6)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.227s
Simple Math ✅ Pass 0.243s
Text Transform Uppercase ✅ Pass 0.239s
Count from 1 to 5 ✅ Pass 0.235s
Math Calculation ✅ Pass 0.224s
Basic Echo Function ✅ Pass 0.225s
Streaming Simple Math Streaming ✅ Pass 0.224s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.220s
Streaming Basic Echo Function Streaming ✅ Pass 0.217s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.222s
Ask Advice Function ✅ Pass 0.216s
Search Query Function ✅ Pass 0.469s
Streaming Search Query Function Streaming ✅ Pass 0.227s
Basic Context Memory Test ✅ Pass 0.221s
Cybersecurity Workflow Memory Test ✅ Pass 3.065s
Function Response Memory Test ✅ Pass 3.755s
Function Argument Memory Test ✅ Pass 4.364s
Read a file, then edit it via unified diff ✅ Pass 0.437s
Penetration Testing Memory with Tool Call ✅ Pass 4.606s
Penetration Testing Methodology ✅ Pass 0.251s
Vulnerability Assessment Tools ✅ Pass 0.251s
SQL Injection Attack Type ✅ Pass 0.236s
Penetration Testing Framework ✅ Pass 0.263s
Web Application Security Scanner ✅ Pass 0.268s
Penetration Testing Tool Selection ✅ Pass 0.268s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.847s


pentester (kimi-k2.7-code)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.222s
Simple Math ✅ Pass 0.226s
Text Transform Uppercase ✅ Pass 0.231s
Count from 1 to 5 ✅ Pass 0.220s
Math Calculation ✅ Pass 0.224s
Basic Echo Function ✅ Pass 0.226s
Streaming Simple Math Streaming ✅ Pass 0.229s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.219s
Streaming Basic Echo Function Streaming ✅ Pass 0.230s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.221s
Search Query Function ✅ Pass 0.218s
Ask Advice Function ✅ Pass 0.228s
Streaming Search Query Function Streaming ✅ Pass 0.225s
Basic Context Memory Test ✅ Pass 0.223s
Function Response Memory Test ✅ Pass 1.915s
Penetration Testing Memory with Tool Call ✅ Pass 2.037s
Function Argument Memory Test ✅ Pass 2.348s
Cybersecurity Workflow Memory Test ✅ Pass 2.711s
Read a file, then edit it via unified diff ✅ Pass 0.443s
Penetration Testing Methodology ✅ Pass 0.264s
SQL Injection Attack Type ✅ Pass 0.251s
Vulnerability Assessment Tools ✅ Pass 0.251s
Penetration Testing Framework ✅ Pass 0.240s
Web Application Security Scanner ✅ Pass 0.246s
Penetration Testing Tool Selection ✅ Pass 0.240s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.564s