Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

21.2 KB
Raw Blame

LLM Agent Testing Report

Generated: Thu, 24 Sep 2026 23:48:06 UTC

Overall Results

Agent Model Reasoning Success Rate Average Latency
simple mistral-small-latest false 25/25 (100.00%) 0.385s
simple_json mistral-small-latest false 8/8 (100.00%) 0.296s
primary_agent mistral-large-latest false 25/25 (100.00%) 0.384s
assistant mistral-large-latest false 25/25 (100.00%) 0.389s
generator mistral-medium-latest true 24/25 (96.00%) 0.454s
refiner mistral-medium-latest true 25/25 (100.00%) 0.440s
adviser mistral-medium-latest true 16/16 (100.00%) 0.386s
reflector mistral-small-latest false 16/16 (100.00%) 0.285s
searcher mistral-small-latest false 25/25 (100.00%) 0.315s
enricher mistral-small-latest false 25/25 (100.00%) 0.291s
coder mistral-large-latest false 25/25 (100.00%) 0.332s
installer mistral-small-latest true 25/25 (100.00%) 0.366s
pentester mistral-large-latest false 25/25 (100.00%) 1.038s

Total: 289/290 (99.66%) successful tests Overall average latency: 0.424s

Detailed Results

simple (mistral-small-latest)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.231s
Answer Stops At The Output Limit ✅ Pass 0.325s
Text Transform Uppercase ✅ Pass 0.381s
Count from 1 to 5 ✅ Pass 0.286s
Math Calculation ✅ Pass 0.430s
Basic Echo Function ✅ Pass 0.459s
Streaming Simple Math Streaming ✅ Pass 0.223s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.302s
Streaming Basic Echo Function Streaming ✅ Pass 0.308s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.215s
Search Query Function ✅ Pass 0.305s
Ask Advice Function ✅ Pass 0.307s
Streaming Search Query Function Streaming ✅ Pass 0.230s
Basic Context Memory Test ✅ Pass 0.222s
Function Response Memory Test ✅ Pass 1.079s
Function Argument Memory Test ✅ Pass 1.363s
Cybersecurity Workflow Memory Test ✅ Pass 0.462s
Penetration Testing Memory with Tool Call ✅ Pass 0.757s
Read a file, then edit it via unified diff ✅ Pass 0.431s
Penetration Testing Methodology ✅ Pass 0.219s
Vulnerability Assessment Tools ✅ Pass 0.222s
SQL Injection Attack Type ✅ Pass 0.216s
Penetration Testing Framework ✅ Pass 0.212s
Web Application Security Scanner ✅ Pass 0.217s
Penetration Testing Tool Selection ✅ Pass 0.217s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.385s


simple_json (mistral-small-latest)

Advanced Tests

Test Result Latency Error
Vulnerability Report Memory Test ✅ Pass 1.004s
Person Information JSON ✅ Pass 0.216s
Project Information JSON ✅ Pass 0.235s
User Profile JSON ✅ Pass 0.220s
JSON Array Response Without Schema ✅ Pass 0.214s
Streaming Person Information JSON Streaming ✅ Pass 0.227s

Capability Tests

Test Capability Result Latency Note
Structured Output With JSON Schema structured_output ✅ Pass 0.247s
Structured Output Refuses A Non-Object Schema structured_output ✅ Pass 0.000s

Summary: 8/8 (100.00%) successful tests

Average latency: 0.296s


primary_agent (mistral-large-latest)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.367s
Simple Math ✅ Pass 0.213s
Count from 1 to 5 ✅ Pass 0.303s
Text Transform Uppercase ✅ Pass 0.347s
Basic Echo Function ✅ Pass 0.230s
Math Calculation ✅ Pass 0.236s
Streaming Simple Math Streaming ✅ Pass 0.383s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.384s
Streaming Basic Echo Function Streaming ✅ Pass 0.239s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.438s
Search Query Function ✅ Pass 0.220s
Ask Advice Function ✅ Pass 0.217s
Streaming Search Query Function Streaming ✅ Pass 0.217s
Basic Context Memory Test ✅ Pass 0.221s
Function Argument Memory Test ✅ Pass 0.630s
Function Response Memory Test ✅ Pass 0.595s
Cybersecurity Workflow Memory Test ✅ Pass 0.710s
Penetration Testing Memory with Tool Call ✅ Pass 1.828s
Read a file, then edit it via unified diff ✅ Pass 0.430s
Penetration Testing Methodology ✅ Pass 0.219s
Vulnerability Assessment Tools ✅ Pass 0.208s
SQL Injection Attack Type ✅ Pass 0.215s
Penetration Testing Framework ✅ Pass 0.224s
Web Application Security Scanner ✅ Pass 0.294s
Penetration Testing Tool Selection ✅ Pass 0.222s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.384s


assistant (mistral-large-latest)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.322s
Simple Math ✅ Pass 0.295s
Text Transform Uppercase ✅ Pass 0.267s
Count from 1 to 5 ✅ Pass 0.368s
Math Calculation ✅ Pass 0.328s
Basic Echo Function ✅ Pass 0.220s
Streaming Simple Math Streaming ✅ Pass 0.219s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.251s
Streaming Basic Echo Function Streaming ✅ Pass 0.298s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.218s
Search Query Function ✅ Pass 0.223s
Ask Advice Function ✅ Pass 0.221s
Streaming Search Query Function Streaming ✅ Pass 0.220s
Basic Context Memory Test ✅ Pass 0.212s
Function Response Memory Test ✅ Pass 0.580s
Function Argument Memory Test ✅ Pass 0.662s
Cybersecurity Workflow Memory Test ✅ Pass 0.868s
Penetration Testing Memory with Tool Call ✅ Pass 1.811s
Read a file, then edit it via unified diff ✅ Pass 0.464s
Penetration Testing Methodology ✅ Pass 0.590s
Vulnerability Assessment Tools ✅ Pass 0.221s
SQL Injection Attack Type ✅ Pass 0.218s
Penetration Testing Framework ✅ Pass 0.213s
Web Application Security Scanner ✅ Pass 0.215s
Penetration Testing Tool Selection ✅ Pass 0.217s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.389s


generator (mistral-medium-latest)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.220s
Simple Math ✅ Pass 0.215s
Text Transform Uppercase ✅ Pass 0.300s
Count from 1 to 5 ✅ Pass 0.304s
Math Calculation ✅ Pass 0.269s
Basic Echo Function ✅ Pass 0.234s
Streaming Simple Math Streaming ✅ Pass 0.253s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.295s
Streaming Basic Echo Function Streaming ✅ Pass 0.212s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.232s
Search Query Function ✅ Pass 0.726s
Ask Advice Function ✅ Pass 0.217s
Streaming Search Query Function Streaming ✅ Pass 0.215s
Basic Context Memory Test ✅ Pass 0.218s
Function Argument Memory Test ❌ Fail 1.274s expected text 'Go programming language' not found; stop reason: tool_calls
Function Response Memory Test ✅ Pass 1.347s
Cybersecurity Workflow Memory Test ✅ Pass 0.856s
Penetration Testing Memory with Tool Call ✅ Pass 2.224s
Read a file, then edit it via unified diff ✅ Pass 0.435s
Penetration Testing Methodology ✅ Pass 0.216s
Vulnerability Assessment Tools ✅ Pass 0.215s
SQL Injection Attack Type ✅ Pass 0.217s
Penetration Testing Framework ✅ Pass 0.215s
Web Application Security Scanner ✅ Pass 0.216s
Penetration Testing Tool Selection ✅ Pass 0.216s

Summary: 24/25 (96.00%) successful tests

Average latency: 0.454s


refiner (mistral-medium-latest)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.214s
Simple Math ✅ Pass 0.353s
Text Transform Uppercase ✅ Pass 0.346s
Count from 1 to 5 ✅ Pass 0.219s
Math Calculation ✅ Pass 0.219s
Basic Echo Function ✅ Pass 0.219s
Streaming Simple Math Streaming ✅ Pass 0.219s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.219s
Streaming Basic Echo Function Streaming ✅ Pass 0.238s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.226s
Search Query Function ✅ Pass 0.227s
Ask Advice Function ✅ Pass 0.218s
Streaming Search Query Function Streaming ✅ Pass 0.215s
Basic Context Memory Test ✅ Pass 0.219s
Function Response Memory Test ✅ Pass 0.847s
Function Argument Memory Test ✅ Pass 1.153s
Cybersecurity Workflow Memory Test ✅ Pass 1.040s
Penetration Testing Memory with Tool Call ✅ Pass 2.835s
Read a file, then edit it via unified diff ✅ Pass 0.425s
Penetration Testing Methodology ✅ Pass 0.211s
Vulnerability Assessment Tools ✅ Pass 0.218s
SQL Injection Attack Type ✅ Pass 0.217s
Penetration Testing Framework ✅ Pass 0.225s
Web Application Security Scanner ✅ Pass 0.247s
Penetration Testing Tool Selection ✅ Pass 0.223s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.440s


adviser (mistral-medium-latest)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.227s
Simple Math ✅ Pass 0.225s
Text Transform Uppercase ✅ Pass 0.240s
Count from 1 to 5 ✅ Pass 0.225s
Math Calculation ✅ Pass 0.212s
Streaming Simple Math Streaming ✅ Pass 0.216s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.330s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 0.225s
Function Argument Memory Test ✅ Pass 0.945s
Function Response Memory Test ✅ Pass 1.004s
Cybersecurity Workflow Memory Test ✅ Pass 1.176s
Penetration Testing Methodology ✅ Pass 0.225s
Vulnerability Assessment Tools ✅ Pass 0.240s
SQL Injection Attack Type ✅ Pass 0.253s
Penetration Testing Framework ✅ Pass 0.218s
Web Application Security Scanner ✅ Pass 0.213s

Summary: 16/16 (100.00%) successful tests

Average latency: 0.386s


reflector (mistral-small-latest)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.272s
Simple Math ✅ Pass 0.220s
Text Transform Uppercase ✅ Pass 0.217s
Count from 1 to 5 ✅ Pass 0.218s
Math Calculation ✅ Pass 0.224s
Streaming Simple Math Streaming ✅ Pass 0.213s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.219s

Advanced Tests

Test Result Latency Error
Basic Context Memory Test ✅ Pass 0.220s
Function Argument Memory Test ✅ Pass 0.474s
Function Response Memory Test ✅ Pass 0.543s
Cybersecurity Workflow Memory Test ✅ Pass 0.495s
Penetration Testing Methodology ✅ Pass 0.215s
Vulnerability Assessment Tools ✅ Pass 0.236s
SQL Injection Attack Type ✅ Pass 0.219s
Penetration Testing Framework ✅ Pass 0.359s
Web Application Security Scanner ✅ Pass 0.214s

Summary: 16/16 (100.00%) successful tests

Average latency: 0.285s


searcher (mistral-small-latest)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.216s
Simple Math ✅ Pass 0.209s
Text Transform Uppercase ✅ Pass 0.234s
Count from 1 to 5 ✅ Pass 0.223s
Math Calculation ✅ Pass 0.225s
Basic Echo Function ✅ Pass 0.229s
Streaming Simple Math Streaming ✅ Pass 0.241s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.257s
Streaming Basic Echo Function Streaming ✅ Pass 0.216s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.215s
Search Query Function ✅ Pass 0.222s
Ask Advice Function ✅ Pass 0.221s
Streaming Search Query Function Streaming ✅ Pass 0.217s
Basic Context Memory Test ✅ Pass 0.223s
Function Response Memory Test ✅ Pass 0.726s
Function Argument Memory Test ✅ Pass 0.938s
Cybersecurity Workflow Memory Test ✅ Pass 0.482s
Penetration Testing Memory with Tool Call ✅ Pass 0.778s
Read a file, then edit it via unified diff ✅ Pass 0.484s
Penetration Testing Methodology ✅ Pass 0.226s
Vulnerability Assessment Tools ✅ Pass 0.218s
SQL Injection Attack Type ✅ Pass 0.223s
Penetration Testing Framework ✅ Pass 0.208s
Web Application Security Scanner ✅ Pass 0.215s
Penetration Testing Tool Selection ✅ Pass 0.213s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.315s


enricher (mistral-small-latest)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.216s
Simple Math ✅ Pass 0.226s
Text Transform Uppercase ✅ Pass 0.215s
Count from 1 to 5 ✅ Pass 0.218s
Math Calculation ✅ Pass 0.217s
Basic Echo Function ✅ Pass 0.214s
Streaming Simple Math Streaming ✅ Pass 0.217s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.217s
Streaming Basic Echo Function Streaming ✅ Pass 0.214s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.210s
Search Query Function ✅ Pass 0.220s
Ask Advice Function ✅ Pass 0.212s
Streaming Search Query Function Streaming ✅ Pass 0.210s
Basic Context Memory Test ✅ Pass 0.218s
Function Argument Memory Test ✅ Pass 0.482s
Function Response Memory Test ✅ Pass 0.561s
Cybersecurity Workflow Memory Test ✅ Pass 0.586s
Penetration Testing Memory with Tool Call ✅ Pass 0.758s
Read a file, then edit it via unified diff ✅ Pass 0.438s
Penetration Testing Methodology ✅ Pass 0.237s
Vulnerability Assessment Tools ✅ Pass 0.219s
SQL Injection Attack Type ✅ Pass 0.254s
Penetration Testing Framework ✅ Pass 0.249s
Web Application Security Scanner ✅ Pass 0.225s
Penetration Testing Tool Selection ✅ Pass 0.223s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.291s


coder (mistral-large-latest)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.216s
Simple Math ✅ Pass 0.210s
Text Transform Uppercase ✅ Pass 0.226s
Count from 1 to 5 ✅ Pass 0.222s
Math Calculation ✅ Pass 0.220s
Basic Echo Function ✅ Pass 0.223s
Streaming Simple Math Streaming ✅ Pass 0.227s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.225s
Streaming Basic Echo Function Streaming ✅ Pass 0.230s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.219s
Search Query Function ✅ Pass 0.216s
Ask Advice Function ✅ Pass 0.213s
Streaming Search Query Function Streaming ✅ Pass 0.215s
Basic Context Memory Test ✅ Pass 0.219s
Function Argument Memory Test ✅ Pass 0.682s
Function Response Memory Test ✅ Pass 0.645s
Cybersecurity Workflow Memory Test ✅ Pass 0.689s
Penetration Testing Memory with Tool Call ✅ Pass 1.328s
Read a file, then edit it via unified diff ✅ Pass 0.427s
Vulnerability Assessment Tools ✅ Pass 0.217s
Penetration Testing Methodology ✅ Pass 0.224s
Penetration Testing Framework ✅ Pass 0.276s
SQL Injection Attack Type ✅ Pass 0.284s
Web Application Security Scanner ✅ Pass 0.221s
Penetration Testing Tool Selection ✅ Pass 0.222s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.332s


installer (mistral-small-latest)

Basic Tests

Test Result Latency Error
Answer Stops At The Output Limit ✅ Pass 0.219s
Text Transform Uppercase ✅ Pass 0.223s
Simple Math ✅ Pass 0.228s
Count from 1 to 5 ✅ Pass 0.229s
Math Calculation ✅ Pass 0.229s
Streaming Simple Math Streaming ✅ Pass 0.250s
Basic Echo Function ✅ Pass 0.257s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.230s
Streaming Basic Echo Function Streaming ✅ Pass 0.228s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.215s
Search Query Function ✅ Pass 0.226s
Ask Advice Function ✅ Pass 0.216s
Streaming Search Query Function Streaming ✅ Pass 0.214s
Basic Context Memory Test ✅ Pass 0.212s
Function Response Memory Test ✅ Pass 0.893s
Function Argument Memory Test ✅ Pass 1.081s
Cybersecurity Workflow Memory Test ✅ Pass 0.842s
Penetration Testing Memory with Tool Call ✅ Pass 1.330s
Read a file, then edit it via unified diff ✅ Pass 0.451s
Penetration Testing Methodology ✅ Pass 0.229s
Vulnerability Assessment Tools ✅ Pass 0.229s
SQL Injection Attack Type ✅ Pass 0.226s
Penetration Testing Framework ✅ Pass 0.226s
Web Application Security Scanner ✅ Pass 0.222s
Penetration Testing Tool Selection ✅ Pass 0.223s

Summary: 25/25 (100.00%) successful tests

Average latency: 0.366s


pentester (mistral-large-latest)

Basic Tests

Test Result Latency Error
Simple Math ✅ Pass 0.231s
Answer Stops At The Output Limit ✅ Pass 0.234s
Count from 1 to 5 ✅ Pass 0.223s
Text Transform Uppercase ✅ Pass 0.229s
Math Calculation ✅ Pass 0.226s
Basic Echo Function ✅ Pass 0.227s
Streaming Simple Math Streaming ✅ Pass 0.221s
Streaming Count from 1 to 3 Streaming ✅ Pass 0.227s
Streaming Basic Echo Function Streaming ✅ Pass 0.234s

Advanced Tests

Test Result Latency Error
JSON Response Function ✅ Pass 0.222s
Search Query Function ✅ Pass 0.267s
Ask Advice Function ✅ Pass 0.249s
Streaming Search Query Function Streaming ✅ Pass 0.370s
Basic Context Memory Test ✅ Pass 1.015s
Function Argument Memory Test ✅ Pass 1.338s
Function Response Memory Test ✅ Pass 1.319s
Penetration Testing Memory with Tool Call ✅ Pass 1.243s
Read a file, then edit it via unified diff ✅ Pass 0.454s
Cybersecurity Workflow Memory Test ✅ Pass 16.102s
Penetration Testing Methodology ✅ Pass 0.216s
Vulnerability Assessment Tools ✅ Pass 0.226s
SQL Injection Attack Type ✅ Pass 0.218s
Penetration Testing Framework ✅ Pass 0.214s
Web Application Security Scanner ✅ Pass 0.216s
Penetration Testing Tool Selection ✅ Pass 0.212s

Summary: 25/25 (100.00%) successful tests

Average latency: 1.038s