Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

Merge branch 'integrate/open-prs-v2' into feature/next-release

Dmitry Ng74c266311438
diff --cc .env.example
index bc82bc71,35ee89b8..7fb6dcf8
--- a/.env.example
+++ b/.env.example
@@@ -26,7 -26,8 +26,7 @@@ BEDROCK_ACCESS_KEY_ID
BEDROCK_SECRET_ACCESS_KEY=
BEDROCK_SESSION_TOKEN=
BEDROCK_SERVER_URL=
- BEDROCK_CONFIG_PATH= # Optional: path to a custom YAML provider config on the host (e.g. ./glm_flash_bedrock.yml next to docker-compose.yml); the name field inside the YAML sets the provider name in the UI
+ BEDROCK_CONFIG_PATH=
-BEDROCK_MODELS_PATH=
## DeepSeek LLM provider
DEEPSEEK_API_KEY=
@@@ -143,6 -152,8 +148,7 @@@ PENTAGI_DOCKER_SOCKET
PENTAGI_DOCKER_CERT_PATH=
PENTAGI_LLM_SERVER_CONFIG_PATH=
PENTAGI_OLLAMA_SERVER_CONFIG_PATH=
+ PENTAGI_BEDROCK_CONFIG_PATH=
-PENTAGI_BEDROCK_MODELS_PATH=
## PentAGI security settings
PUBLIC_URL=https://localhost:8443
diff --cc .vscode/launch.json
index 43dd5172,43dd5172..f68b9116
--- a/.vscode/launch.json
+++ b/.vscode/launch.json
@@@ -65,6 -65,6 +65,7 @@@
// "-type", "glm",
// "-type", "kimi",
// "-type", "qwen",
++ // "-type", "minimax",
"-config", "${workspaceFolder}/examples/configs/moonshot.provider.yml",
// "-config", "${workspaceFolder}/examples/configs/deepseek.provider.yml",
// "-config", "${workspaceFolder}/examples/configs/ollama-cloud.provider.yml",
@@@ -89,11 -89,11 +90,14 @@@
// "-report", "${workspaceFolder}/examples/tests/ollama-qwen332b-fp16-tc-report.md",
// "-report", "${workspaceFolder}/examples/tests/vllm-qwen3.5-27b-fp8.report.md",
// "-report", "${workspaceFolder}/examples/tests/vllm-qwen332b-fp16-report.md",
++ // "-report", "${workspaceFolder}/examples/tests/vllm-qwen3.6-27b-fp8.report.md",
++ // "-report", "${workspaceFolder}/examples/tests/vllm-mixed.report.md",
"-report", "${workspaceFolder}/examples/tests/moonshot-report.md",
// "-report", "${workspaceFolder}/examples/tests/deepseek-report.md",
// "-report", "${workspaceFolder}/examples/tests/glm-report.md",
// "-report", "${workspaceFolder}/examples/tests/kimi-report.md",
// "-report", "${workspaceFolder}/examples/tests/qwen-report.md",
++ // "-report", "${workspaceFolder}/examples/tests/minimax-report.md",
// "-report", "${workspaceFolder}/examples/tests/custom-openai-report.md",
// "-report", "${workspaceFolder}/examples/tests/openrouter-report.md",
// "-report", "${workspaceFolder}/examples/tests/deepinfra-report.md",
diff --cc .vscode/settings.json
index dc6687bb,dc6687bb..981d3928
--- a/.vscode/settings.json
+++ b/.vscode/settings.json
@@@ -4,7 -4,7 +4,8 @@@
],
"go.testTimeout": "300s",
"go.testEnvVars": {
-- "INSTALLER_LOG_FILE": "${workspaceFolder}/build/log.json"
++ "INSTALLER_LOG_FILE": "${workspaceFolder}/build/log.json",
++ "DEBUG": "false"
},
"go.testExplorer.enable": true,
"search.useIgnoreFiles": true,
diff --cc README.md
index cc6f774a,487a16bd..945f4960
--- a/README.md
+++ b/README.md
@@@ -33,6 -33,6 +33,7 @@@
- [GLM](#glm-provider-configuration)
- [Kimi](#kimi-provider-configuration)
- [Qwen](#qwen-provider-configuration)
++ - [MiniMax](#minimax-provider-configuration)
- [Advanced Setup](#advanced-setup)
- [Langfuse Integration](#langfuse-integration)
- [Monitoring and Observability](#monitoring-and-observability)
@@@ -1607,90 -1605,90 +1608,115 @@@ PROXY_URL=http://your-proxy:808
#### Supported Models
--PentAGI supports 31 OpenAI models with tool calling, streaming, reasoning modes, and prompt caching. Models marked with `*` are used in default configuration.
++PentAGI supports 32 OpenAI models with tool calling, streaming, reasoning modes, and prompt caching. Models marked with `*` are used in default configuration. Models marked `⚠️` are deprecated by OpenAI and kept only for backward compatibility with agent configs already pinned to those names — avoid them for new assignments.
--**GPT-5.2 Series - Latest Flagship Agentic (December 2025)**
++**GPT-5.6 Series - Latest Frontier (Feb 2026 knowledge cutoff, 1.05M context, 128K max output)**
--| Model ID | Thinking | Price (Input/Output/Cache) | Use Case |
--| --------------------- | -------- | -------------------------- | ----------------------------------------------- |
--| `gpt-5.2`* | ✅ | $1.75/$14.00/$0.18 | Latest flagship with enhanced reasoning and tool integration, autonomous security research |
--| `gpt-5.2-pro` | ✅ | $21.00/$168.00/$0.00 | Premium version with superior agentic coding, mission-critical security research, zero-day discovery |
--| `gpt-5.2-codex` | ✅ | $1.75/$14.00/$0.18 | Most advanced code-specialized, context compaction, strong cybersecurity capabilities |
++| Model ID | Thinking | Reasoning Effort | Price (Input/Output/Cache) | Use Case |
++| --------------------- | -------- | ---------------------------- | -------------------------- | ----------------------------------------------- |
++| `gpt-5.6-sol` | ✅ | low/medium/high/xhigh | $5.00/$30.00/$0.50 | Frontier model for complex professional work, most demanding autonomous pentesting, sophisticated exploit chain development, deep multi-stage attack simulation |
++| `gpt-5.6-terra`* | ✅ | low/medium/high/xhigh | $2.50/$15.00/$0.25 | Balances intelligence and cost; multi-phase security assessments, coordinated multi-tool pentesting (generator/refiner/adviser/coder default) |
++| `gpt-5.6-luna` | ✅ | low/medium/high/xhigh | $1.00/$6.00/$0.10 | Optimized for cost-sensitive, high-volume workloads; rapid reconnaissance, bulk vulnerability scanning, real-time monitoring |
--**GPT-5/5.1 Series - Advanced Agentic Models**
++**GPT-5.5 Series - Frontier (Dec 2025 knowledge cutoff, 1.05M context, 128K max output)**
--| Model ID | Thinking | Price (Input/Output/Cache) | Use Case |
--| --------------------- | -------- | -------------------------- | ----------------------------------------------- |
--| `gpt-5` | ✅ | $1.25/$10.00/$0.13 | Premier agentic with advanced reasoning, autonomous security research, exploit chain development |
--| `gpt-5.1` | ✅ | $1.25/$10.00/$0.13 | Enhanced agentic with adaptive reasoning, balanced penetration testing with strong tool coordination |
--| `gpt-5-pro` | ✅ | $15.00/$120.00/$0.00 | Premium version with major reasoning improvements, reduced hallucinations, critical security operations |
--| `gpt-5-mini` | ✅ | $0.25/$2.00/$0.03 | Efficient balancing speed and intelligence, automated vulnerability analysis, exploit generation |
--| `gpt-5-nano` | ✅ | $0.05/$0.40/$0.01 | Fastest for high-throughput scanning, reconnaissance, bulk vulnerability detection |
++| Model ID | Thinking | Reasoning Effort | Price (Input/Output/Cache) | Use Case |
++| --------------------- | -------- | ------------------------------ | -------------------------- | ----------------------------------------------- |
++| `gpt-5.5` | ✅ | none/low/medium/high/xhigh | $5.00/$30.00/$0.50 | New class of intelligence for coding and professional work; complex security research, advanced autonomous pentesting |
++| `gpt-5.5-pro` | ✅ | medium/high/xhigh | $30.00/$180.00/$0.00 | Uses more compute for smarter, more precise responses; no cached-input discount; mission-critical security research, zero-day discovery |
++
++**GPT-5.4 Series - Advanced Reasoning at Scale (1M context)**
--**GPT-5/5.1 Codex Series - Code-Specialized**
++| Model ID | Thinking | Reasoning Effort | Price (Input/Output/Cache) | Use Case |
++| --------------------- | -------- | -------------------------- | -------------------------- | ----------------------------------------------- |
++| `gpt-5.4` | ✅ | low/medium/high/xhigh | $2.50/$15.00/$0.25 | Best intelligence at scale for agentic, coding, and professional workflows; maximum cognitive depth for pentesting |
++| `gpt-5.4-mini`* | ✅ | low/medium/high/xhigh | $0.75/$4.50/$0.075 | Strongest mini model for coding, computer use, subagents (primary_agent/assistant/reflector/installer/pentester default) |
++| `gpt-5.4-nano`* | ✅ | low/medium/high/xhigh | $0.20/$1.25/$0.02 | Cheapest GPT-5.4-class model for simple, high-volume tasks (simple/simple_json/searcher/enricher default) |
++
++**GPT-5.2 Series - Previous Flagship Agentic**
++
++| Model ID | Thinking | Reasoning Effort | Price (Input/Output/Cache) | Use Case |
++| --------------------- | -------- | ------------------------ | -------------------------- | ----------------------------------------------- |
++| `gpt-5.2` | ✅ | low/medium/high/xhigh | $1.75/$14.00/$0.175 | Superseded by 5.4/5.6; autonomous security research, complex exploit chain development |
++| `gpt-5.2-pro` | ✅ | medium/high/xhigh | $21.00/$168.00/$0.00 | Superior agentic coding and long-context performance, mission-critical security research, zero-day discovery |
++
++**GPT-5/5.1 Series - Advanced Agentic Models**
| Model ID | Thinking | Price (Input/Output/Cache) | Use Case |
| --------------------- | -------- | -------------------------- | ----------------------------------------------- |
--| `gpt-5.1-codex-max` | ✅ | $1.25/$10.00/$0.13 | Enhanced reasoning for sophisticated coding, proven CVE findings, systematic exploit development |
--| `gpt-5.1-codex` | ✅ | $1.25/$10.00/$0.13 | Standard code-optimized with strong reasoning, exploit generation, vulnerability analysis |
--| `gpt-5-codex` | ✅ | $1.25/$10.00/$0.13 | Foundational code-specialized, vulnerability scanning, basic exploit generation |
--| `gpt-5.1-codex-mini` | ✅ | $0.25/$2.00/$0.03 | Compact high-performance, 4x higher capacity, rapid vulnerability detection |
--| `codex-mini-latest` | ✅ | $1.50/$6.00/$0.38 | Latest compact code model, automated code review, basic vulnerability analysis |
++| `gpt-5` | ✅ | $1.25/$10.00/$0.125 | Autonomous security research, exploit chain development, coordinating multi-tool pentesting workflows |
++| `gpt-5.1` | ✅ | $1.25/$10.00/$0.125 | Bridges GPT-5 and GPT-5.2 with faster responses; balanced penetration testing with strong tool coordination |
++| `gpt-5-pro` | ✅ (high) | $15.00/$120.00/$0.00 | Reduced hallucinations, exceptional accuracy, critical security operations |
++| `gpt-5-mini` | ✅ | $0.25/$2.00/$0.025 | Automated vulnerability analysis, exploit generation with strong function calling |
++| `gpt-5-nano` | ✅ | $0.05/$0.40/$0.005 | High-throughput security scanning, reconnaissance, real-time monitoring |
--**GPT-4.1 Series - Enhanced Intelligence**
++**GPT-4.1 Series - Enhanced Intelligence (Non-Reasoning)**
| Model ID | Thinking | Price (Input/Output/Cache) | Use Case |
| --------------------- | -------- | -------------------------- | ----------------------------------------------- |
--| `gpt-4.1` | ❌ | $2.00/$8.00/$0.50 | Enhanced flagship with superior function calling, complex threat analysis, sophisticated exploit development |
--| `gpt-4.1-mini`* | ❌ | $0.40/$1.60/$0.10 | Balanced performance with improved efficiency, routine security assessments, automated code analysis |
--| `gpt-4.1-nano` | ❌ | $0.10/$0.40/$0.03 | Ultra-fast lightweight, bulk security scanning, rapid reconnaissance, continuous monitoring |
++| `gpt-4.1` | ❌ | $2.00/$8.00/$0.50 | Superior function calling, complex threat analysis, sophisticated exploit development |
++| `gpt-4.1-mini` | ❌ | $0.40/$1.60/$0.10 | Routine security assessments, automated code analysis (no longer used in default configuration) |
--**GPT-4o Series - Multimodal Flagship**
++**GPT-4o Series - Multimodal (Non-Reasoning)**
| Model ID | Thinking | Price (Input/Output/Cache) | Use Case |
| --------------------- | -------- | -------------------------- | ----------------------------------------------- |
--| `gpt-4o` | ❌ | $2.50/$10.00/$1.25 | Multimodal flagship with vision, image analysis, web UI assessment, multi-tool orchestration |
--| `gpt-4o-mini` | ❌ | $0.15/$0.60/$0.08 | Compact multimodal with strong function calling, high-frequency scanning, cost-effective bulk operations |
++| `gpt-4o-mini` | ❌ | $0.15/$0.60/$0.075 | Compact multimodal with strong function calling, high-frequency scanning, cost-effective bulk operations |
--**o-Series - Advanced Reasoning Models**
++**o-Series - Advanced Reasoning Models (Current)**
| Model ID | Thinking | Price (Input/Output/Cache) | Use Case |
| --------------------- | -------- | -------------------------- | ----------------------------------------------- |
--| `o4-mini`* | ✅ | $1.10/$4.40/$0.28 | Next-gen reasoning with enhanced speed, methodical security assessments, systematic exploit development |
--| `o3`* | ✅ | $2.00/$8.00/$0.50 | Advanced reasoning powerhouse, multi-stage attack chains, deep vulnerability analysis |
--| `o3-mini` | ✅ | $1.10/$4.40/$0.55 | Compact reasoning with extended thinking, step-by-step attack planning, logical vulnerability chaining |
--| `o1` | ✅ | $15.00/$60.00/$7.50 | Premier reasoning with maximum depth, advanced penetration testing, novel exploit research |
--| `o3-pro` | ✅ | $20.00/$80.00/$0.00 | Most advanced reasoning, 80% cheaper than o1-pro, zero-day research, critical security investigations |
--| `o1-pro` | ✅ | $150.00/$600.00/$0.00 | Previous-gen premium reasoning, exhaustive security analysis, mission-critical challenges |
++| `o3` | ✅ | $2.00/$8.00/$0.50 | Succeeded by GPT-5; multi-stage attack chains, deep vulnerability analysis |
++| `o3-pro` | ✅ | $20.00/$80.00/$0.00 | More compute for better responses; zero-day research, critical security investigations |
++
++**Deprecated Models - Kept for Backward Compatibility ⚠️**
++
++These models were marked deprecated by OpenAI. PentAGI keeps them defined only so that pre-existing agent configs pinned to these names keep working; do not assign them to new agents.
++
++| Model ID | Thinking | Price (Input/Output/Cache) | Notes |
++| --------------------- | -------- | -------------------------- | --------------------------------------------- |
++| `gpt-5.2-codex` | ✅ | $1.75/$14.00/$0.175 | Superseded code-specialized model; use `gpt-5.6-terra`/`gpt-5.4-mini` instead |
++| `gpt-5.1-codex-max` | ✅ | $1.25/$10.00/$0.125 | Superseded; enhanced reasoning for coding workflows |
++| `gpt-5.1-codex` | ✅ | $1.25/$10.00/$0.125 | Superseded standard code-optimized model |
++| `gpt-5-codex` | ✅ | $1.25/$10.00/$0.125 | Superseded foundational code-specialized model |
++| `gpt-5.1-codex-mini` | ✅ | $0.25/$2.00/$0.025 | Superseded compact code model |
++| `codex-mini-latest` | ✅ | $1.50/$6.00/$0.375 | Superseded compact code model |
++| `gpt-4o` | ❌ | $2.50/$10.00/$1.25 | Superseded by GPT-5.x/5.6 series multimodal flagship |
++| `gpt-4.1-nano` | ❌ | $0.10/$0.40/$0.025 | Superseded ultra-fast lightweight model |
++| `o3-mini` | ✅ | $1.10/$4.40/$0.55 | Superseded compact reasoning model |
++| `o4-mini` | ✅ | $1.10/$4.40/$0.275 | Succeeded by `gpt-5-mini` |
++| `o1` | ✅ | $15.00/$60.00/$7.50 | Superseded premier reasoning model |
++| `o1-pro` | ✅ | $150.00/$600.00/$0.00 | Superseded, highest cost point of the o-series |
**Prices**: Per 1M tokens. Reasoning models include thinking tokens in output pricing.
> [!WARNING]
--> **GPT-5* Models - Trusted Access Required**
++> **GPT-5/5.1/5.2 Models - Trusted Access Required**
>
--> All GPT-5 series models (`gpt-5`, `gpt-5.1`, `gpt-5.2`, `gpt-5-pro`, `gpt-5.2-pro`, and all Codex variants) work **unstably with PentAGI** and may trigger OpenAI's cybersecurity safety mechanisms without verified access.
++> The original GPT-5, GPT-5.1, and GPT-5.2 models (`gpt-5`, `gpt-5.1`, `gpt-5.2`, `gpt-5-pro`, `gpt-5.2-pro`, and all deprecated Codex variants) work **unstably with PentAGI** and may trigger OpenAI's cybersecurity safety mechanisms without verified access. This does not affect the newer GPT-5.4/5.5/5.6 series used in PentAGI's default configuration below.
>
--> **To use GPT-5* models reliably:**
++> **To use these models reliably:**
> 1. **Individual users**: Verify your identity at [chatgpt.com/cyber](https://chatgpt.com/cyber)
> 2. **Enterprise teams**: Request trusted access through your OpenAI representative
> 3. **Security researchers**: Apply for the [Cybersecurity Grant Program](https://openai.com/form/cybersecurity-grant-program/) (includes $10M in API credits)
>
> **Recommended alternatives without verification:**
--> - Use `o-series` models (o3, o4-mini, o1) for reasoning tasks
--> - Use `gpt-4.1` series for general intelligence and function calling
--> - All o-series and gpt-4.x models work reliably without special access
++> - Use PentAGI's defaults — `gpt-5.4-mini`/`gpt-5.4-nano`/`gpt-5.6-terra` — which work out of the box
++> - Use `o3`/`o3-pro` for reasoning tasks
++> - Use `gpt-4.1` series for general intelligence and function calling without reasoning
--**Reasoning Effort Levels**:
--- **High**: Maximum reasoning depth (refiner - o3 with high effort)
--- **Medium**: Balanced reasoning (primary_agent, assistant, reflector - o4-mini/o3 with medium effort)
--- **Low**: Efficient targeted reasoning (coder, installer, pentester - o3/o4-mini with low effort; adviser - gpt-5.2 with low effort)
++**Reasoning Configuration**:
++- **Reasoning forced off by default**: every default agent assigned `gpt-5.4-mini` or `gpt-5.6-terra` (primary_agent, assistant, generator, refiner, adviser, reflector, coder, installer, pentester) sets `reasoning: {mode: off}` — this genuinely disables reasoning, it is not simply "low effort". PentAGI calls OpenAI exclusively through `/v1/chat/completions` (never `/v1/responses`), and this endpoint rejects requests that combine function tools with these models' default-on thinking; forcing thinking off is required for tool calls to work reliably (see the investigation notes in [`backend/pkg/providers/openai/config.yml`](backend/pkg/providers/openai/config.yml)).
++- **No override needed for `gpt-5.4-nano`**: used for simple, simple_json, searcher, and enricher, this tier does not default to thinking on, so tools attach without conflict and no `reasoning` override is required.
++- **Manual tuning available**: outside the default assignments, GPT-5.6/5.5/5.4/5.2 series models expose explicit reasoning effort levels (`low`/`medium`/`high`/`xhigh`, plus `none` on GPT-5.5) for custom agent configs that need variable reasoning depth with tool calling disabled or via `/v1/responses`.
**Key Features**:
--- **Extended Reasoning**: o-series models with chain-of-thought for complex security analysis
--- **Agentic Intelligence**: GPT-5/5.1/5.2 series with enhanced tool integration and autonomous capabilities
++- **Extended Reasoning**: GPT-5.4/5.5/5.6 and o-series models with chain-of-thought for complex security analysis
++- **Agentic Intelligence**: GPT-5.4/5.5/5.6 series with enhanced tool integration, million-token context windows, and autonomous capabilities
- **Prompt Caching**: Cost reduction on repeated context (10-50% of input price)
--- **Code Specialization**: Dedicated Codex models for vulnerability discovery and exploit development
--- **Multimodal Support**: GPT-4o series for vision-based security assessments
++- **Code Specialization**: Legacy Codex models remain available (deprecated) for vulnerability discovery and exploit development in pinned configs
++- **Multimodal Support**: `gpt-4o-mini` for vision-based security assessments
- **Tool Calling**: Robust function calling across all models for pentesting tool orchestration
- **Streaming**: Real-time response streaming for interactive workflows
- **Proven Track Record**: Industry-leading models with CVE discoveries and real-world security applications
@@@ -1730,43 -1728,45 +1756,43 @@@ PROXY_URL=http://your-proxy:808
#### Supported Models
- PentAGI supports 10 Claude models with tool calling, streaming, extended thinking, adaptive thinking, and prompt caching. Models marked with `*` are used in default configuration.
-PentAGI supports 11 Claude models with tool calling, streaming, extended thinking, adaptive thinking, and prompt caching. Models marked with `*` are used in default configuration.
++PentAGI supports 9 Claude models with tool calling, streaming, extended thinking, adaptive thinking, and prompt caching. Models marked with `*` are used in default configuration.
--**Claude 4 Series - Latest Models (2025-2026)**
++**Claude 5 Series - Newest Models (2026)**
| Model ID | Thinking | Release Date | Price (Input/Output/Cache R/W) | Use Case |
| ------------------------ | -------- | ------------ | ------------------------------ | ----------------------------------------------- |
-| `claude-opus-4-8` | ✅ | May 2026 | $5.00/$25.00/$0.50/$6.25 | Flagship for coding, agents, and deep reasoning. Adaptive thinking only — budget thinking and sampling params (temperature/top_p/top_k) are rejected. Most demanding exploit development and multi-stage attack simulation |
-| `claude-opus-4-7` | ✅ | Apr 2026 | $5.00/$25.00/$0.50/$6.25 | Advanced software engineering and long-running agentic security analysis. Adaptive thinking only (manual budget thinking rejected) |
--| `claude-opus-4-6`* | ✅ | May 2025 | $5.00/$25.00/$0.50/$6.25 | Most intelligent model for autonomous agents and coding. Extended + adaptive thinking for complex exploit development, multi-stage attack simulation |
--| `claude-sonnet-4-6`* | ✅ | Aug 2025 | $3.00/$15.00/$0.30/$3.75 | Best speed/intelligence balance with adaptive thinking. Multi-phase security assessments, intelligent vulnerability analysis, real-time threat hunting |
--| `claude-haiku-4-5`* | ✅ | Oct 2025 | $1.00/$5.00/$0.10/$1.25 | Fastest model with near-frontier intelligence. High-frequency scanning, real-time monitoring, bulk automated testing |
++| `claude-sonnet-5`* | ✅ | Jul 2026 | $3.00/$15.00/$0.30/$3.75 | Best combination of speed and intelligence for coding, agents, and professional work at scale. Adaptive thinking only (manual budget thinking rejected); sampling parameters not supported. Default model for primary agent, assistant, coder, adviser, installer, pentester |
++| `claude-fable-5` | ✅ | Jun 2026 | $10.00/$50.00/$1.00/$12.50 | Anthropic's most capable widely released model for long-running agents and the most demanding reasoning workloads. Adaptive thinking always on (budget thinking and an explicit disable are rejected); sampling parameters not supported |
--**Legacy Models - Still Supported**
++**Claude 4 Series**
| Model ID | Thinking | Release Date | Price (Input/Output/Cache R/W) | Use Case |
| ------------------------ | -------- | ------------ | ------------------------------ | ----------------------------------------------- |
--| `claude-sonnet-4-5` | ✅ | Sep 2025 | $3.00/$15.00/$0.30/$3.75 | State-of-the-art reasoning (superseded by 4-6). Sophisticated penetration testing, advanced threat analysis |
--| `claude-opus-4-5` | ✅ | Nov 2025 | $5.00/$25.00/$0.50/$6.25 | Ultimate reasoning (superseded by opus-4-6). Critical security research, zero-day discovery, red team operations |
--| `claude-opus-4-1` | ✅ | Aug 2025 | $15.00/$75.00/$1.50/$18.75 | Advanced reasoning (superseded). Complex penetration testing, sophisticated threat modeling |
--| `claude-sonnet-4-0` | ✅ | May 2025 | $3.00/$15.00/$0.30/$3.75 | High-performance reasoning (superseded). Complex threat modeling, multi-tool coordination |
--| `claude-opus-4-0` | ✅ | May 2025 | $15.00/$75.00/$1.50/$18.75 | First generation Opus (superseded). Multi-step exploit development, autonomous pentesting workflows |
++| `claude-opus-4-8`* | ✅ | May 2026 | $5.00/$25.00/$0.50/$6.25 | Flagship for coding, agents, and deep reasoning in enterprise security workflows. Adaptive thinking only — budget thinking and sampling params (temperature/top_p/top_k) are rejected. Default model for generator and refiner; most demanding exploit development and multi-stage attack simulation |
++| `claude-opus-4-7` | ✅ | Apr 2026 | $5.00/$25.00/$0.50/$6.25 | Advanced software engineering and long-running agentic security analysis. Adaptive thinking only (manual budget thinking rejected) |
++| `claude-sonnet-4-6` | ✅ | Feb 2026 | $3.00/$15.00/$0.30/$3.75 | Best speed/intelligence balance with adaptive thinking. Multi-phase security assessments, intelligent vulnerability analysis, real-time threat hunting |
++| `claude-opus-4-6` | ✅ | Feb 2026 | $5.00/$25.00/$0.50/$6.25 | Most intelligent model for autonomous agents and coding. Extended + adaptive thinking for complex exploit development, multi-stage attack simulation |
++| `claude-haiku-4-5`* | ❌ | Oct 2025 | $1.00/$5.00/$0.10/$1.25 | Fast and efficient model with exceptional function calling and low latency, no thinking support. Default model for simple, simple_json, reflector, searcher, enricher; high-frequency scanning, real-time monitoring, bulk automated testing |
--**Deprecated Models - Migrate to Current Models**
++**Legacy Models - Still Supported**
--| Model ID | Thinking | Release Date | Price (Input/Output/Cache R/W) | Notes |
--| ---------------------------- | -------- | ------------ | ------------------------------ | -------------------------------------------- |
--| `claude-3-haiku-20240307` | ❌ | Mar 2024 | $0.25/$1.25/$0.03/$0.30 | Will be retired April 19, 2026. Migrate to claude-haiku-4-5 |
++| Model ID | Thinking | Release Date | Price (Input/Output/Cache R/W) | Use Case |
++| ------------------------ | -------- | ------------ | ------------------------------ | ----------------------------------------------- |
++| `claude-sonnet-4-5` | ✅ | Sep 2025 | $3.00/$15.00/$0.30/$3.75 | State-of-the-art reasoning (superseded by sonnet-4-6/sonnet-5). Sophisticated penetration testing, advanced threat analysis |
++| `claude-opus-4-5` | ✅ | Nov 2025 | $5.00/$25.00/$0.50/$6.25 | Ultimate reasoning (superseded by opus-4-6/4-7/4-8). Critical security research, zero-day discovery, red team operations |
**Prices**: Per 1M tokens. Cache pricing includes both Read and Write costs.
--**Extended Thinking Configuration**:
--- **Max Tokens 4096**: Generator (claude-opus-4-6) for maximum reasoning depth on complex exploit development
--- **Max Tokens 2048**: Coder (claude-sonnet-4-6) for balanced code analysis and vulnerability research
--- **Max Tokens 1024**: Primary agent, assistant, refiner, adviser, reflector, searcher, installer, pentester for focused reasoning on specific tasks
--- **Extended Thinking**: All Claude 4.5+ and 4.6 models support configurable extended thinking for deep reasoning tasks
++**Extended Thinking Configuration** (default agent config, see [`backend/pkg/providers/anthropic/config.yml`](backend/pkg/providers/anthropic/config.yml)):
++- **Generator / Refiner** (`claude-opus-4-8`): adaptive reasoning at `xhigh`/`high` effort for maximum reasoning depth on complex exploit development
++- **Primary agent, assistant, coder, adviser, installer, pentester** (`claude-sonnet-5`): adaptive reasoning (adviser at `xhigh` effort) for balanced code analysis and vulnerability research
++- **Reflector, searcher** (`claude-haiku-4-5`): fixed reasoning budget of 1024 tokens for focused reasoning on specific tasks
++- **Simple, simple_json, enricher** (`claude-haiku-4-5`): no thinking, optimized for speed
**Key Features**:
--- **Extended Thinking**: All Claude 4.5+ and 4.6 models with configurable chain-of-thought reasoning depths for complex security analysis
- - **Adaptive Thinking**: Claude 4.6 series (Opus/Sonnet) dynamically adjusts reasoning depth based on task complexity for optimal performance
-- **Adaptive Thinking**: Claude 4.6 series (Opus/Sonnet) dynamically adjusts reasoning depth based on task complexity; Claude Opus 4.7/4.8 are adaptive-thinking-only (manual budget thinking and sampling parameters are rejected with HTTP 400)
++- **Extended Thinking**: All Claude 4.5+ models support configurable chain-of-thought reasoning depths for complex security analysis
++- **Adaptive Thinking**: Claude 4.6 series (Opus/Sonnet) dynamically adjusts reasoning depth based on task complexity; Claude Opus 4.7/4.8 and the Claude 5 series (Sonnet/Fable) are adaptive-thinking-only (manual budget thinking and sampling parameters are rejected with HTTP 400)
- **Prompt Caching**: Significant cost reduction with separate read/write pricing (10% read, 125% write of input)
- **Extended Context Window**: 200K tokens standard, up to 1M tokens (beta) for Claude Opus/Sonnet 4.6 for comprehensive codebase analysis
- **Tool Calling**: Robust function calling with exceptional accuracy for security tool orchestration
@@@ -1907,9 -1908,29 +1933,26 @@@ BEDROCK_SERVER_URL=https://bedrock-runt
PROXY_URL=http://your-proxy:8080
```
+ #### Custom Provider Config and Models (advanced)
+
+ By default the Bedrock provider uses a per-agent config and model catalog compiled into the binary. Two optional paths override them without rebuilding:
+
+ - `BEDROCK_CONFIG_PATH` — a YAML file (same shape as the other provider configs) that **replaces** the built-in per-agent model/price assignments. See [`examples/configs/bedrock-glm-flash.provider.yml`](examples/configs/bedrock-glm-flash.provider.yml).
-- `BEDROCK_MODELS_PATH` — a YAML model catalog that is **merged** onto the built-in models: new model ids are added and become selectable under Settings → Providers, while a matching id overrides the built-in entry. See [`examples/configs/bedrock-glm-flash.models.yml`](examples/configs/bedrock-glm-flash.models.yml).
+
+ This is useful to expose a Bedrock model newer than the compiled-in catalog — for example Z.AI's `zai.glm-4.7-flash`. Use the exact Model ID from the model's AWS Bedrock detail page; add a `us.`/`eu.`/`apac.` inference-profile prefix only when that page marks the model as requiring cross-region inference (`zai.glm-4.7-flash` is In-Region, so it is used as-is, with no prefix).
+
+ With Docker Compose, set the host-side mount source and the in-container path together:
+
+ ```bash
+ # host files mounted into the container
+ PENTAGI_BEDROCK_CONFIG_PATH=./examples/configs/bedrock-glm-flash.provider.yml
-PENTAGI_BEDROCK_MODELS_PATH=./examples/configs/bedrock-glm-flash.models.yml
+ # tell the backend to read the mounted files
+ BEDROCK_CONFIG_PATH=/opt/pentagi/conf/bedrock.provider.yml
-BEDROCK_MODELS_PATH=/opt/pentagi/conf/bedrock.models.yml
+ ```
+
#### Supported Models
- PentAGI supports 21 AWS Bedrock models with tool calling, streaming, and multimodal capabilities. Models marked with `*` are used in default configuration.
+ PentAGI supports 24 AWS Bedrock models with tool calling, streaming, and multimodal capabilities. Models marked with `*` are used in default configuration.
| Model ID | Provider | Thinking | Multimodal | Price (Input/Output) | Use Case |
| ------------------------------------------------ | --------------- | -------- | ---------- | -------------------- | --------------------------------------- |
@@@ -2087,13 -2110,13 +2132,14 @@@ GLM_PROVIDER=zai # Adds prefix to mode
#### Supported Models
--PentAGI supports 13 GLM models with tool calling, streaming, hybrid thinking modes, and prompt caching. Models marked with `*` are used in default configuration. Thinking is controlled via `extra_body.thinking.type` ("enabled"/"disabled"); unlike Kimi, GLM is permissive about temperature in either mode.
++PentAGI supports 14 GLM models with tool calling, streaming, hybrid thinking modes, and prompt caching. Models marked with `*` are used in default configuration. Thinking is controlled via `extra_body.thinking.type` ("enabled"/"disabled"); unlike Kimi, GLM is permissive about temperature in either mode.
**GLM-5.x Series - Latest Generation (200K context, 128K max output)**
| Model ID | Thinking | Context | Max Output | Price (Input/Output/Cache) | Use Case |
| ---------------- | -------- | ------- | ---------- | -------------------------- | ------------------------------------------------------------------- |
--| `glm-5.1`* | ✅ Hybrid | 200K | 128K | $1.40/$4.40/$0.26 | Newest flagship: 8h sustained autonomous execution, Claude Opus 4.6-aligned (generator/refiner/adviser/coder/pentester default) |
++| `glm-5.2`* | ✅ Hybrid | 200K | 128K | $1.40/$4.40/$0.26 | Newest flagship, improves on GLM-5.1. Supports explicit `reasoning_effort` (high/max) (generator/refiner/adviser/coder/pentester default) |
++| `glm-5.1` | ✅ Hybrid | 200K | 128K | $1.40/$4.40/$0.26 | Long-horizon tasks: 8h sustained autonomous execution, Claude Opus 4.6-aligned coding |
| `glm-5` | ✅ Hybrid | 200K | 128K | $1.00/$3.20/$0.20 | Foundation for Agentic Engineering, MoE 744B/40B active, Claude Opus 4.5-level coding |
| `glm-5-turbo`* | ✅ Hybrid | 200K | 128K | $1.20/$4.00/$0.24 | OpenClaw-native: optimized for tool invocation, persistent tasks, long-chain execution (primary_agent/assistant default) |
@@@ -2131,23 -2154,23 +2177,26 @@@
**Default Agent Configuration**:
--Strategy: `glm-5.1` (newest flagship, $1.40 input) for critical reasoning, `glm-5-turbo` (OpenClaw-native, agent-optimized) for orchestration, `glm-4.5-air` (cheap MoE with hybrid thinking and reliable RPM) for all utility/installer agents. `glm-4.7-flashx` is avoided as default due to lower RPM limits causing frequent 429 errors at high frequency.
++Strategy: `glm-5.2` (newest flagship, $1.40 input) for critical reasoning, `glm-5-turbo` (OpenClaw-native, agent-optimized) for orchestration, `glm-4.5-air` (cheap MoE with hybrid thinking and reliable RPM) for all utility/installer agents. `glm-4.7-flashx` is avoided as default due to lower RPM limits causing frequent 429 errors at high frequency.
| Agent Role | Default Model | Thinking | Temperature | Top P | Max Output |
| ----------------------------------- | ------------- | -------- | ----------- | ----- | ---------- |
--| Generator / Refiner | `glm-5.1` | Enabled | 1.0 | 0.95 | 32768 |
--| Coder | `glm-5.1` | Enabled | 1.0 | 0.95 | 20480 |
--| Adviser / Pentester | `glm-5.1` | Enabled | 1.0 | 0.95 | 16384 |
++| Generator / Refiner | `glm-5.2` | Enabled | 1.0 | 0.95 | 32768 |
++| Coder | `glm-5.2` | Enabled | 1.0 | 0.95 | 20480 |
++| Adviser / Pentester | `glm-5.2` | Enabled | 1.0 | 0.95 | 16384 |
| Primary Agent / Assistant | `glm-5-turbo` | Enabled | 1.0 | 0.95 | 16384 |
| Installer | `glm-4.5-air` | Enabled | 1.0 | 0.95 | 16384 |
| Simple / Reflector | `glm-4.5-air` | Disabled | 0.6 | 0.9 | 8192 |
| Searcher / Enricher / Simple JSON | `glm-4.5-air` | Disabled | 0.6 | 0.9 | 4096 |
--> **Note on temperature**: GLM accepts both `1.0` and `0.6` in either thinking/non-thinking mode (per Z.AI docs). langchaingo's `IsReasoningModel` matches `glm-4.5*`/`glm-4.6*`/`glm-4.7*` prefixes and force-overrides temperature to 1.0 in `createChatRequest` — this is harmless for GLM (unlike Kimi) but means temperature values for those models in YAML are advisory. `glm-5`/`glm-5.1`/`glm-5-turbo` are not matched, so explicit values pass through unchanged.
++Generator, refiner, and adviser additionally set `reasoning.effort: max`, which layers `llms.WithReasoning(ReasoningMax, 0)` on top of `extra_body.thinking.type=enabled` — this `reasoning_effort` parameter is only supported by `glm-5.2`.
++
++> **Note on temperature**: GLM accepts both `1.0` and `0.6` in either thinking/non-thinking mode (per Z.AI docs). langchaingo's `IsReasoningModel` matches `glm-4.5*`/`glm-4.6*`/`glm-4.7*` prefixes and force-overrides temperature to 1.0 in `createChatRequest` — this is harmless for GLM (unlike Kimi) but means temperature values for those models in YAML are advisory. `glm-5`/`glm-5.1`/`glm-5.2`/`glm-5-turbo` are not matched, so explicit values pass through unchanged.
**Thinking Modes**:
- **Hybrid** (GLM-5.x, GLM-4.7): Explicit toggle via `extra_body.thinking.type`
- **Auto** (GLM-4.6, GLM-4.5 series): Model automatically determines when reasoning is needed
++- **Reasoning Effort** (GLM-5.2 only): supports an explicit `reasoning_effort` parameter (`high`/`max`) on top of hybrid thinking, for finer control over reasoning depth than the other GLM-5.x models
- **Preserved Thinking** (Z.AI Coding capability): all thinking-enabled agents in PentAGI also pass `extra_body.thinking.clear_thinking: false` so that `reasoning_content` from previous assistant turns is retained across the conversation. This is required on the standard API endpoint (`/api/paas/v4`) — on the Coding Plan endpoint it would be enabled by default. Improves reasoning continuity and cache hit rates in multi-turn tool call chains.
- All thinking-enabled agents also pass `extra_body.tool_choice: auto` defensively
@@@ -2194,14 -2217,14 +2243,27 @@@ KIMI_PROVIDER=moonshot # Adds prefix t
#### Supported Models
--PentAGI supports 8 Kimi/Moonshot models with tool calling, streaming, hybrid thinking modes, and multimodal capabilities (text/image/video for K2.x). All `kimi-k2-*` legacy models (turbo-preview, 0905-preview, 0711-preview, thinking, thinking-turbo) were deprecated by Moonshot on 2026-05-25 and are NOT included. Models marked with `*` are used in default configuration.
++PentAGI supports 11 Kimi/Moonshot models with tool calling, streaming, hybrid thinking modes, and multimodal capabilities (text/image/video for K2.x). All `kimi-k2-*` legacy models (turbo-preview, 0905-preview, 0711-preview, thinking, thinking-turbo) were deprecated by Moonshot on 2026-05-25 and are NOT included. Models marked with `*` are used in default configuration.
++
++**Kimi K3 - Flagship (Always Reasoning)**
++
++| Model ID | Thinking | Multimodal | Context | Price (Input Miss / Output / Cache Hit) | Use Case |
++| ---------- | -------- | ---------- | ------- | --------------------------------------- | ------------------------------------------------------- |
++| `kimi-k3`* | ✅ always | ❌ | 1M | $3.00 / $15.00 / $0.30 | Flagship for long-horizon coding and end-to-end knowledge work. Always reasons — no `thinking` toggle, depth set via top-level `reasoning_effort` (currently `max` only) (generator/refiner/adviser default) |
++
++**Kimi K2.7 Code Series - Coding-Focused**
++
++| Model ID | Thinking | Multimodal | Context | Price (Input Miss / Output / Cache Hit) | Use Case |
++| ---------------------------- | -------- | ---------- | ------- | --------------------------------------- | ------------------------------------------------------- |
++| `kimi-k2.7-code`* | ✅ hybrid | ✅ | 256K | $0.95 / $4.00 / $0.19 | Coding-focused, higher success rates on long-context programming tasks (coder/pentester default) |
++| `kimi-k2.7-code-highspeed`* | ✅ hybrid | ✅ | 256K | $1.90 / $8.00 / $0.38 | Same model as `kimi-k2.7-code` with higher output throughput (~180-260 tokens/s) (primary_agent/assistant default) |
**Kimi K2.x Series - Multimodal Flagship**
| Model ID | Thinking | Multimodal | Context | Price (Input Miss / Output / Cache Hit) | Use Case |
| ---------------- | -------- | ---------- | ------- | --------------------------------------- | ------------------------------------------------------- |
--| `kimi-k2.6`* | ✅ hybrid | ✅ | 256K | $0.95 / $4.00 / $0.16 | Latest flagship: native multimodal, stronger code, improved instruction compliance (generator/refiner/adviser/coder/pentester default) |
--| `kimi-k2.5`* | ✅ hybrid | ✅ | 256K | $0.60 / $3.00 / $0.10 | Previous-gen: 36% cheaper input, same architecture (primary/assistant/installer/utility default) |
++| `kimi-k2.6` | ✅ hybrid | ✅ | 256K | $0.95 / $4.00 / $0.16 | Latest multimodal flagship: native architecture, stronger code, improved instruction compliance (not used by default config) |
++| `kimi-k2.5`* | ✅ hybrid | ✅ | 256K | $0.60 / $3.00 / $0.10 | Previous-gen: 36% cheaper input than K2.6 (simple/simple_json/reflector/searcher/enricher/installer default) |
**Moonshot V1 Series - Generation Models (Flexible Parameters)**
@@@ -2219,39 -2242,39 +2281,44 @@@
| `moonshot-v1-32k-vision-preview` | ❌ | ✅ | 32K | $1.00 / $3.00 | Vision + medium context |
| `moonshot-v1-128k-vision-preview` | ❌ | ✅ | 128K | $2.00 / $5.00 | Vision + long context |
--**Prices**: Per 1M tokens. Cache pricing applies to prompt tokens served from automatic context cache (only Kimi K2.x models support cache).
++**Prices**: Per 1M tokens. Cache pricing applies to prompt tokens served from automatic context cache (only Kimi K3/K2.7/K2.x models support cache).
--> **CRITICAL — Kimi K2.6/K2.5 parameter constraints**: API returns `invalid_request_error` for any deviation:
--> - `temperature`: MUST be `1.0` in thinking mode, MUST be `0.6` in non-thinking mode
--> - `top_p`: MUST be `0.95`
--> - `n`: MUST be `1`
--> - `presence_penalty` and `frequency_penalty`: MUST be `0` (not modifiable)
++> **CRITICAL — Kimi parameter constraints per model family**: API returns `invalid_request_error` for any deviation:
++> - `kimi-k3`: always reasons, no `thinking` param at all; reasoning depth is set via the top-level `reasoning_effort` field (`low`/`high`/`max`, default `max`) — PentAGI pins it to `max` for all agents using this model. `temperature` MUST be `1.0`, `top_p` MUST be `0.95`, `n` MUST be `1`, `presence_penalty`/`frequency_penalty` MUST be `0`. Do not switch effort per call — it invalidates the prefix cache.
++> - `kimi-k2.7-code` / `kimi-k2.7-code-highspeed`: `thinking` may be omitted; if set explicitly, only `{"type":"enabled","keep":"all"}` is accepted (`type: disabled` is rejected). `reasoning_effort` is not supported. `temperature` MUST be `1.0`, `top_p` MUST be `0.95`, `n` MUST be `1`; `tool_choice: required` is not supported (use `auto`).
++> - `kimi-k2.6`: thinking mode needs `temperature=1.0`, `top_p=0.95`, `n=1`, `thinking.keep="all"`; non-thinking mode needs `temperature=0.6`, `top_p=0.95`, `n=1`.
++> - `kimi-k2.5`: thinking mode needs `temperature=1.0`, `top_p=0.95`, `n=1` (no `keep` support); non-thinking mode needs `temperature=0.6`, `top_p=0.95`, `n=1`.
++> - All Kimi models: `presence_penalty=0`, `frequency_penalty=0`, `tool_choice` in `{auto, none}`.
>
> Moonshot V1 models use standard OpenAI-compatible parameters with no such constraints.
**Default Agent Configuration**:
--Strategy: prefer `kimi-k2.5` as cost-effective workhorse (36% cheaper input vs `kimi-k2.6`); reserve `kimi-k2.6` for critical reasoning. All `kimi-k2.x` agents are configured with the API-required fixed parameters (temp/top_p/n) and explicit `extra_body.thinking.type`. For thinking-enabled agents, `extra_body.thinking.keep: "all"` is set to preserve historical `reasoning_content` in multi-turn tool call chains (without it Moonshot returns "thinking is enabled but reasoning_content is missing").
++Strategy: `kimi-k2.5` for utility/orchestration, `kimi-k2.7-code-highspeed` for the primary/assistant loop, `kimi-k3` (always-thinking flagship) for critical reasoning (generator/refiner/adviser), `kimi-k2.7-code` for coder/pentester. `kimi-k2.6` is not used in the default configuration. All `kimi-k2.x`/`k2.7` agents are configured with the API-required fixed parameters (temp/top_p/n) and explicit `extra_body.thinking.type`. For thinking-enabled agents, `extra_body.thinking.keep: "all"` is set (where supported) to preserve historical `reasoning_content` in multi-turn tool call chains (without it Moonshot returns "thinking is enabled but reasoning_content is missing").
++
++| Agent Role | Default Model | Thinking | Temperature | Top P | Max Output |
++| -------------------------------------------- | --------------------------- | --------------------- | ----------- | ----- | ---------- |
++| Generator / Refiner | `kimi-k3` | Always (effort=max) | 1.0 | 0.95 | 32768 |
++| Adviser (mentor/planner) | `kimi-k3` | Always (effort=max) | 1.0 | 0.95 | 8192 |
++| Coder | `kimi-k2.7-code` | Enabled (keep=all) | 1.0 | 0.95 | 20480 |
++| Pentester | `kimi-k2.7-code` | Enabled (keep=all) | 1.0 | 0.95 | 16384 |
++| Primary Agent / Assistant | `kimi-k2.7-code-highspeed` | Enabled (keep=all) | 1.0 | 0.95 | 16384 |
++| Installer | `kimi-k2.5` | Enabled | 1.0 | 0.95 | 16384 |
++| Reflector / Searcher / Enricher | `kimi-k2.5` | Disabled | 0.6 | 0.95 | 4096 |
++| Simple | `kimi-k2.5` | Disabled | 0.6 | 0.95 | 8192 |
++| Simple JSON | `kimi-k2.5` | Disabled | 0.6 | 0.95 | 4096 |
--| Agent Role | Default Model | Thinking | Temperature | Top P | Max Output |
--| -------------------------------------------- | ------------- | -------- | ----------- | ----- | ---------- |
--| Generator / Refiner | `kimi-k2.6` | Enabled (keep=all) | 1.0 | 0.95 | 32768 |
--| Coder | `kimi-k2.6` | Enabled (keep=all) | 1.0 | 0.95 | 20480 |
--| Pentester | `kimi-k2.6` | Enabled (keep=all) | 1.0 | 0.95 | 16384 |
--| Adviser (mentor/planner) | `kimi-k2.6` | Enabled (keep=all) | 1.0 | 0.95 | 8192 |
--| Primary Agent / Assistant | `kimi-k2.5` | Enabled (keep=all) | 1.0 | 0.95 | 16384 |
--| Installer | `kimi-k2.5` | Enabled (keep=all) | 1.0 | 0.95 | 12288 |
--| Reflector / Searcher / Enricher | `kimi-k2.5` | Disabled | 0.6 | 0.95 | 4096 |
--| Simple / Simple JSON | `kimi-k2.5` | Disabled | 0.6 | 0.95 | 2048 |
++> **Note**: for `kimi-k2.5` non-thinking agents, PentAGI also duplicates `temperature: 0.6` into `extra_body` as a workaround — langchaingo's `IsReasoningModel` matches the substring `2.5` and force-overrides temperature to `1.0`, and `extra_body` bypasses that override.
**Key Features**:
--- **Ultra-Long Context**: Up to 256K tokens (K2.x) for comprehensive codebase/documentation analysis
--- **Native Multimodal**: K2.6/K2.5 support text + image + video input out of the box
--- **Hybrid Thinking**: K2.6/K2.5 toggle between thinking and non-thinking via `extra_body.thinking.type`
--- **Preserved Thinking** (K2.6): `thinking.keep: "all"` preserves historical `reasoning_content` across turns — required for multi-turn tool call chains
--- **Automatic Context Caching**: K2.x models cache repeated prefixes (~17% of miss price for K2.6, ~17% for K2.5)
--- **Tool Calling**: Full function-calling support for K2.x and Moonshot V1
--- **Self-Correction**: K2.6 features improved instruction compliance and self-correction
++- **Always-On Reasoning Flagship**: `kimi-k3` never disables thinking and offers a 1M token context for the most demanding long-horizon coding and knowledge work
++- **Ultra-Long Context**: Up to 256K tokens (K2.7/K2.x) or 1M tokens (K3) for comprehensive codebase/documentation analysis
++- **Native Multimodal**: K2.7/K2.6/K2.5 support text + image + video input out of the box
++- **Hybrid Thinking**: K2.7/K2.6/K2.5 toggle between thinking and non-thinking via `extra_body.thinking.type`
++- **Preserved Thinking** (K2.7, K2.6): `thinking.keep: "all"` preserves historical `reasoning_content` across turns — required for multi-turn tool call chains
++- **Automatic Context Caching**: K3/K2.7/K2.x models cache repeated prefixes
++- **Tool Calling**: Full function-calling support for K3, K2.7, K2.x, and Moonshot V1
++- **Coding-Optimized Variants**: `kimi-k2.7-code`/`kimi-k2.7-code-highspeed` target higher success rates on long-context programming tasks, with the highspeed variant tuned for throughput
- **Multilingual**: Strong Chinese, English, and multi-language support
**Multi-turn with thinking + tool calls**: PentAGI's universal reasoning preservation pattern (`TextPartWithReasoning` + `WithPreserveReasoningContent`) automatically ensures `reasoning_content` is sent back in the required TextContent → ToolCall order, satisfying Moonshot's "thinking is enabled but reasoning_content is missing in assistant tool call message" requirement.
@@@ -2418,6 -2441,43 +2485,43 @@@ EMBEDDING_STRIP_NEW_LINES= # optiona
**As OpenAI-typed custom LLM provider**: instead of the dedicated `QWEN_*` variables, you can wire any Qwen chat model through PentAGI's custom OpenAI-compatible provider by pointing `OPENAI_SERVER_URL` (or a custom provider entry) to the DashScope `/compatible-mode/v1` endpoint and selecting the desired Qwen model name. Useful when you already manage all model traffic through a single OpenAI-shaped client (e.g. shared with LiteLLM/OneAPI proxies).
+ ### MiniMax Provider Configuration
+
+ PentAGI integrates with MiniMax's M-series through the OpenAI-compatible `https://api.minimax.io/v1` endpoint: large-context agentic models with tool calling, JSON output, and streaming.
+
+ #### Configuration Variables
+
+ | Variable | Default Value | Description |
+ | -------------------- | --------------------------- | -------------------------------------------------- |
+ | `MINIMAX_API_KEY` | | MiniMax API key for authentication |
+ | `MINIMAX_SERVER_URL` | `https://api.minimax.io/v1` | MiniMax API endpoint URL |
+ | `MINIMAX_PROVIDER` | | Provider prefix for LiteLLM integration (optional) |
+
+ #### Configuration Examples
+
+ ```bash
+ # Direct API usage
+ MINIMAX_API_KEY=your_minimax_api_key
+ MINIMAX_SERVER_URL=https://api.minimax.io/v1
+
+ # With LiteLLM proxy
+ MINIMAX_API_KEY=your_litellm_key
+ MINIMAX_SERVER_URL=http://litellm-proxy:4000
+ MINIMAX_PROVIDER=minimax # Adds prefix to model names (minimax/MiniMax-M3) for LiteLLM
+ ```
+
+ #### Supported Models
+
+ PentAGI ships 3 MiniMax models with tool calling, JSON output, and streaming. `MiniMax-M3` is the default for all agent types.
+
-| Model ID | Context | Price (Input/Output) | Use Case |
-| ------------------------ | ------- | -------------------- | --------------------------------------------------------------------------------------------------- |
-| `MiniMax-M3`* | ~1M | $0.60/$2.40 | Latest flagship for agentic reasoning, tool use, code generation, and long-context tasks (default) |
-| `MiniMax-M2.7` | 204K | $0.40/$1.10 | Previous-generation model with strong reasoning and coding |
-| `MiniMax-M2.7-highspeed` | 204K | $0.40/$1.10 | Low-latency variant of M2.7 for fast-response scenarios |
++| Model ID | Context | Price (Input/Output, ≤512K context) | Use Case |
++| ------------------------ | ------- | ------------------------------------ | --------------------------------------------------------------------------------------------------- |
++| `MiniMax-M3`* | ~1M | $0.30/$1.20 (2x above 512K tokens) | Latest flagship for agentic reasoning, tool use, code generation, and long-context tasks (default) |
++| `MiniMax-M2.7` | 204K | $0.30/$1.20 | Previous-generation model with strong reasoning and coding |
++| `MiniMax-M2.7-highspeed` | 204K | $0.60/$2.40 | Low-latency variant of M2.7 for fast-response scenarios |
+
+ **LiteLLM Integration**: Set `MINIMAX_PROVIDER=minimax` to enable model name prefixing when using default PentAGI configurations with LiteLLM proxy. Leave empty for direct API usage.
+
## Advanced Setup
### Langfuse Integration
@@@ -3007,11 -3063,11 +3110,11 @@@ docker exec -it pentagi /opt/pentagi/bi
To use these configurations, your `.env` file only needs to contain:
```
- LLM_SERVER_URL=https://openrouter.ai/api/v1 # or https://api.orcarouter.ai/v1 or https://api.deepinfra.com/v1/openai or https://api.openai.com/v1 or https://api.novita.ai/openai
-LLM_SERVER_URL=https://openrouter.ai/api/v1 # or https://api.deepinfra.com/v1/openai or https://api.openai.com/v1 or https://api.novita.ai/openai or https://api.atlascloud.ai/v1
++LLM_SERVER_URL=https://openrouter.ai/api/v1 # or https://api.orcarouter.ai/v1 or https://api.deepinfra.com/v1/openai or https://api.openai.com/v1 or https://api.novita.ai/openai or https://api.atlascloud.ai/v1
LLM_SERVER_KEY=your_api_key
LLM_SERVER_MODEL= # Leave empty, as models are specified in the config
- LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/openrouter.provider.yml # or orcarouter.provider.yml or deepinfra.provider.ymll or custom-openai.provider.yml or novita.provider.yml
-LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/openrouter.provider.yml # or deepinfra.provider.yml or custom-openai.provider.yml or novita.provider.yml or atlas.provider.yml
-LLM_SERVER_PROVIDER= # Provider name for LiteLLM proxy (e.g., openrouter, deepseek, moonshot, novita)
++LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/openrouter.provider.yml # or orcarouter.provider.yml or deepinfra.provider.ymll or custom-openai.provider.yml or novita.provider.yml or atlas.provider.yml
+LLM_SERVER_PROVIDER= # Provider name for LiteLLM proxy (e.g., openrouter, orcarouter, deepseek, moonshot, novita)
LLM_SERVER_LEGACY_REASONING=false # Controls reasoning format, for OpenAI must be true (default: false)
LLM_SERVER_PRESERVE_REASONING=false # Preserve reasoning content in multi-turn conversations (required by Moonshot, default: false)
diff --cc backend/cmd/ctester/main.go
index 85f48253,eb5ad29a..1f02e476
--- a/backend/cmd/ctester/main.go
+++ b/backend/cmd/ctester/main.go
@@@ -324,21 -335,21 +335,34 @@@ func convertToAgentResults(results test
var totalLatency time.Duration
for _, testResult := range agentTestResults {
oldResult := TestResult{
-- Name: testResult.Name,
-- Type: string(testResult.Type),
-- Success: testResult.Success,
-- Error: testResult.Error,
-- Streaming: testResult.Streaming,
-- Reasoning: testResult.Reasoning,
-- LatencyMs: testResult.Latency.Milliseconds(),
++ Name: testResult.Name,
++ Type: string(testResult.Type),
++ Capability: string(testResult.Capability),
++ Success: testResult.Success,
++ Unsupported: testResult.Unsupported,
++ Error: testResult.Error,
++ Streaming: testResult.Streaming,
++ Reasoning: testResult.Reasoning,
++ LatencyMs: testResult.Latency.Milliseconds(),
}
-- if testResult.Group == testdata.TestGroupBasic {
++ switch {
++ case testResult.Capability != "":
++ result.CapabilityTests = append(result.CapabilityTests, oldResult)
++ case testResult.Group == testdata.TestGroupBasic:
result.BasicTests = append(result.BasicTests, oldResult)
-- } else {
++ default:
result.AdvancedTests = append(result.AdvancedTests, oldResult)
}
++ // an unsupported optional capability doesn't count against the
++ // overall success rate — it's not attempted-and-failed, it's
++ // attempted-and-not-available on this model.
++ if testResult.Unsupported {
++ totalLatency += testResult.Latency
++ continue
++ }
++
result.TotalTests++
if testResult.Success {
result.TotalSuccess++
diff --cc backend/cmd/ctester/models.go
index d30f4bd0,d30f4bd0..99e50ca3
--- a/backend/cmd/ctester/models.go
+++ b/backend/cmd/ctester/models.go
@@@ -4,24 -4,24 +4,36 @@@ import "time
// TestResult represents the result of a single test for CLI compatibility
type TestResult struct {
-- Name string
-- Type string
-- Success bool
-- Error error
-- Streaming bool
-- Reasoning bool
-- LatencyMs int64
-- Response string
-- Expected string
++ Name string
++ Type string
++ Capability string
++ Success bool
++ // Unsupported means the provider/model proactively rejected the requested
++ // capability (e.g. structured output, or disabling reasoning) with a typed
++ // SDK error rather than the call failing for an unrelated reason. It does
++ // not count against TotalSuccess/TotalTests: an optional capability being
++ // unavailable on a given model is informative, not a defect in the config.
++ Unsupported bool
++ Error error
++ Streaming bool
++ Reasoning bool
++ LatencyMs int64
++ Response string
++ Expected string
}
// AgentTestResult collects test results for each agent type for CLI compatibility
type AgentTestResult struct {
-- AgentType string
-- ModelName string
-- Reasoning bool
-- BasicTests []TestResult
-- AdvancedTests []TestResult
++ AgentType string
++ ModelName string
++ Reasoning bool
++ BasicTests []TestResult
++ AdvancedTests []TestResult
++ // CapabilityTests holds results for capability-gated tests (adaptive
++ // thinking, reasoning off, structured output) kept separate from
++ // Basic/AdvancedTests so the report can call out "does this optional
++ // capability work" without it affecting the primary pass/fail tally.
++ CapabilityTests []TestResult
TotalSuccess int
TotalTests int
AverageLatency time.Duration
diff --cc backend/cmd/ctester/report.go
index 8fe16613,8fe16613..d4900680
--- a/backend/cmd/ctester/report.go
+++ b/backend/cmd/ctester/report.go
@@@ -52,6 -52,6 +52,29 @@@ func PrintAgentResults(result AgentTest
fmt.Printf(" %s\n", result.SkippedReason)
}
++ // Capability tests section (adaptive thinking / reasoning off / structured
++ // output): only present when the model was actually classified as
++ // supporting the capability, so its absence here means "not applicable to
++ // this model", not "not tested".
++ if len(result.CapabilityTests) > 0 {
++ fmt.Println("\nCapability Tests:")
++ for _, test := range result.CapabilityTests {
++ status := "✓"
++ switch {
++ case test.Unsupported:
++ status = "⊘"
++ case !test.Success:
++ status = "✗"
++ }
++ fmt.Printf("[%s] %s (%s) (%.3fs)\n", status, test.Name, test.Capability, float64(test.LatencyMs)/1000)
++ if test.Unsupported {
++ fmt.Printf(" Not supported by this model/provider: %v\n", test.Error)
++ } else if !test.Success && test.Error != nil {
++ fmt.Printf(" Error: %v\n", test.Error)
++ }
++ }
++ }
++
// Summary
successRate := float64(result.TotalSuccess) / float64(result.TotalTests) * 100
fmt.Printf("\nSummary: %d/%d (%.2f%%) successful tests\n",
@@@ -214,6 -214,6 +237,41 @@@ func WriteReportToFile(results []AgentT
file.WriteString(fmt.Sprintf("*%s*\n\n", result.SkippedReason))
}
++ // Capability tests (adaptive thinking / reasoning off / structured
++ // output) - absent entirely for a model not classified as supporting
++ // the capability at all, distinguished from "attempted but rejected"
++ // (Unsupported) and from a genuine failure.
++ if len(result.CapabilityTests) > 0 {
++ file.WriteString("#### Capability Tests\n\n")
++ file.WriteString("| Test | Capability | Result | Latency | Note |\n")
++ file.WriteString("|------|------------|--------|---------|------|\n")
++
++ for _, test := range result.CapabilityTests {
++ status := "✅ Pass"
++ note := ""
++ switch {
++ case test.Unsupported:
++ status = "⊘ Not Supported"
++ if test.Error != nil {
++ note = TruncateString(EscapeMarkdown(test.Error.Error()), 150)
++ }
++ case !test.Success:
++ status = "❌ Fail"
++ if test.Error != nil {
++ note = TruncateString(EscapeMarkdown(test.Error.Error()), 150)
++ }
++ }
++
++ file.WriteString(fmt.Sprintf("| %s | %s | %s | %.3fs | %s |\n",
++ test.Name,
++ test.Capability,
++ status,
++ float64(test.LatencyMs)/1000,
++ note))
++ }
++ file.WriteString("\n")
++ }
++
// Summary
successRate := float64(result.TotalSuccess) / float64(result.TotalTests) * 100
file.WriteString(fmt.Sprintf("**Summary**: %d/%d (%.2f%%) successful tests\n\n",
diff --cc backend/cmd/ftester/worker/args.go
index b1d00d8e,475be317..7308676c
--- a/backend/cmd/ftester/worker/args.go
+++ b/backend/cmd/ftester/worker/args.go
@@@ -238,8 -238,8 +238,21 @@@ func ParseFunctionArgs(funcName string
// Check if there's a value for the argument
if i+1 < len(args) && !strings.HasPrefix(args[i+1], "-") {
// Next arg is the value
-- parsedArgs[argName] = args[i+1]
++ value := args[i+1]
i++ // Skip the value in the next iteration
++
++ if argInfo.Type == "array" {
++ // Array arguments (e.g. []string) can't be passed as a single CLI value,
++ // so each occurrence of the flag appends one element to the resulting slice.
++ // This lets the user fill in at least one element via the command line.
++ if existing, ok := parsedArgs[argName].([]string); ok {
++ parsedArgs[argName] = append(existing, value)
++ } else {
++ parsedArgs[argName] = []string{value}
++ }
++ } else {
++ parsedArgs[argName] = value
++ }
} else {
// Boolean flag (no value)
parsedArgs[argName] = true
diff --cc backend/cmd/ftester/worker/interactive.go
index 4042da0f,4042da0f..1636f229
--- a/backend/cmd/ftester/worker/interactive.go
+++ b/backend/cmd/ftester/worker/interactive.go
@@@ -106,6 -106,6 +106,30 @@@ func InteractiveFillArgs(ctx context.Co
break
}
++ case "array":
++ var items []string
++ for {
++ itemTitle := fmt.Sprintf("%s item #%d (leave empty to finish)", title, len(items)+1)
++ strValue, err := terminal.InteractivePromptContext(ctx, itemTitle, os.Stdin)
++ if err != nil {
++ return nil, fmt.Errorf("input cancelled for '%s': %w", arg.Name, err)
++ }
++
++ if strValue == "" {
++ if len(items) == 0 && arg.Required {
++ terminal.PrintError("At least one value is required")
++ continue
++ }
++ break
++ }
++
++ items = append(items, strValue)
++ }
++
++ if len(items) > 0 {
++ value = items
++ }
++
default: // string and other types
strValue, err := terminal.InteractivePromptContext(ctx, title, os.Stdin)
if err != nil {
@@@ -170,6 -170,6 +194,15 @@@ func fillStructFromMap(structPtr any, d
fieldValue.SetInt(int64(value.(int)))
case reflect.Bool:
fieldValue.SetBool(value.(bool))
++ case reflect.Slice:
++ if fieldValue.Type().Elem().Kind() == reflect.String {
++ switch v := value.(type) {
++ case []string:
++ fieldValue.Set(reflect.ValueOf(v))
++ case string:
++ fieldValue.Set(reflect.ValueOf([]string{v}))
++ }
++ }
case reflect.Struct:
// For special types that may be in the tools package
// This is a simplified version that may require refinement
diff --cc backend/docs/config.md
index c7ab50ae,aa44c640..5670affd
--- a/backend/docs/config.md
+++ b/backend/docs/config.md
@@@ -66,6 -66,6 +67,7 @@@ This document serves as a comprehensiv
- [Usage Details](#usage-details-11)
- [Graphiti Knowledge Graph Settings](#graphiti-knowledge-graph-settings)
- [Usage Details](#usage-details-12)
++ - [Current Limitations (Beta)](#current-limitations-beta)
- [Agent Supervision Settings](#agent-supervision-settings)
- [Usage Details](#usage-details-13)
- [Supervision System Integration](#supervision-system-integration)
@@@ -119,7 -119,7 +121,7 @@@ These web-console features do not repla
The environment variables documented below remain the source of truth for configuration that is not currently editable from the web console:
- - **LLM credentials and connection settings**: API keys, base URLs, auth modes, and provider-specific connection settings for OpenAI, Anthropic, Bedrock, Ollama, custom providers, and similar backends; config-path settings apply only where supported, such as `OLLAMA_SERVER_CONFIG_PATH` and `LLM_SERVER_CONFIG_PATH`.
-- **LLM credentials and connection settings**: API keys, base URLs, auth modes, and provider-specific connection settings for OpenAI, Anthropic, Bedrock, Ollama, custom providers, and similar backends; config-path settings apply only where supported, such as `OLLAMA_SERVER_CONFIG_PATH`, `LLM_SERVER_CONFIG_PATH`, and `BEDROCK_CONFIG_PATH`/`BEDROCK_MODELS_PATH`.
++- **LLM credentials and connection settings**: API keys, base URLs, auth modes, and provider-specific connection settings for OpenAI, Anthropic, Bedrock, Ollama, custom providers, and similar backends; config-path settings apply only where supported, such as `OLLAMA_SERVER_CONFIG_PATH`, `LLM_SERVER_CONFIG_PATH`, and `BEDROCK_CONFIG_PATH`.
- **Search provider credentials and options**: DuckDuckGo, Google, Tavily, Traversaal, Perplexity, Searxng, Sploitus, and related search configuration.
- **Third-party integrations**: Langfuse, Graphiti, and other external observability or knowledge services.
- **MCP server management**: MCP settings are not currently exposed as a live web-console feature.
@@@ -607,6 -608,8 +609,7 @@@ There is no `VERTEX_API_KEY` or `GOOGLE
| BedrockSecretKey | `BEDROCK_SECRET_ACCESS_KEY` | *(none)* | AWS secret access key for static credentials authentication |
| BedrockSessionToken | `BEDROCK_SESSION_TOKEN` | *(none)* | AWS session token for temporary credentials (optional, used with static credentials for STS/assumed roles) |
| BedrockServerURL | `BEDROCK_SERVER_URL` | *(none)* | Optional custom endpoint URL for Bedrock service (VPC endpoints, local testing) |
+ | BedrockConfig | `BEDROCK_CONFIG_PATH` | *(none)* | Path to a custom YAML config that replaces the built-in Bedrock per-agent config (model assignments, prices) |
-| BedrockModels | `BEDROCK_MODELS_PATH` | *(none)* | Path to a custom YAML model catalog merged onto the built-in Bedrock models (adds new ids; matching names override) |
**Authentication Priority**: `BedrockDefaultAuth` (highest) → `BedrockBearerToken` → `BedrockAccessKey`+`BedrockSecretKey` (lowest)
diff --cc backend/go.mod
index 4a31f836,d091d3b7..7301cacc
--- a/backend/go.mod
+++ b/backend/go.mod
@@@ -52,7 -52,7 +52,7 @@@ require
github.com/vektah/gqlparser/v2 v2.5.19
github.com/vxcontrol/cloud v0.9.0
github.com/vxcontrol/graphiti-go-client v0.9.0
- github.com/vxcontrol/langchaingo v0.1.14-update.5
- github.com/vxcontrol/langchaingo v0.1.14-update.6
++ github.com/vxcontrol/langchaingo v0.1.15-0.20260717093433-cf5f5f41f4eb
github.com/wasilibs/go-re2 v1.10.0
github.com/xeipuuv/gojsonschema v1.2.0
go.opentelemetry.io/otel v1.39.0
@@@ -65,9 -65,9 +65,11 @@@
go.opentelemetry.io/otel/sdk/log v0.14.0
go.opentelemetry.io/otel/sdk/metric v1.39.0
go.opentelemetry.io/otel/trace v1.39.0
++ go.opentelemetry.io/proto/otlp v1.9.0
golang.org/x/crypto v0.46.0
golang.org/x/net v0.48.0
golang.org/x/oauth2 v0.34.0
++ golang.org/x/sync v0.19.0
golang.org/x/sys v0.40.0
google.golang.org/api v0.238.0
google.golang.org/grpc v1.79.3
@@@ -189,6 -189,6 +191,7 @@@ require
github.com/richardlehane/msoleps v1.0.1 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/sahilm/fuzzy v0.1.1 // indirect
++ github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 // indirect
github.com/sethvargo/go-retry v0.2.4 // indirect
github.com/shoenig/go-m1cpu v0.1.6 // indirect
github.com/sosodev/duration v1.3.1 // indirect
@@@ -218,11 -218,11 +221,9 @@@
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.39.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.33.0 // indirect
-- go.opentelemetry.io/proto/otlp v1.9.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/arch v0.8.0 // indirect
golang.org/x/mod v0.30.0 // indirect
-- golang.org/x/sync v0.19.0 // indirect
golang.org/x/term v0.38.0 // indirect
golang.org/x/text v0.32.0 // indirect
golang.org/x/time v0.12.0 // indirect
diff --cc backend/go.sum
index ee265fa3,582cc75f..46df62b9
--- a/backend/go.sum
+++ b/backend/go.sum
@@@ -533,6 -533,6 +533,8 @@@ github.com/russross/blackfriday/v2 v2.1
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sahilm/fuzzy v0.1.1 h1:ceu5RHF8DGgoi+/dR5PsECjCDH1BE3Fnmpo7aVXOdRA=
github.com/sahilm/fuzzy v0.1.1/go.mod h1:VFvziUEIMCrT6A6tw2RFIXPXXmzXbOsSHF0DOI8ZK9Y=
++github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
++github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
github.com/segmentio/asm v1.2.0 h1:9BQrFxC+YOHJlTlHGkTrFWf59nbL3XnCoFLTwDCI7ys=
github.com/segmentio/asm v1.2.0/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
github.com/sergi/go-diff v1.3.1 h1:xkr+Oxo4BOQKmkn/B9eMK0g5Kg/983T9DqqPHwYqD+8=
@@@ -630,8 -630,8 +632,8 @@@ github.com/vxcontrol/cloud v0.9.0 h1:p7
github.com/vxcontrol/cloud v0.9.0/go.mod h1:AeiQFqiMgJJAXy6FYXtDS2a3P/PMB56iiBNY2vGrZhQ=
github.com/vxcontrol/graphiti-go-client v0.9.0 h1:3GxpFmQoHmz/d7/9tyEqD8+S99v2cuqG1UEmrbAFrLU=
github.com/vxcontrol/graphiti-go-client v0.9.0/go.mod h1:6UHL5uqAKp4KAdziva4qgcAxFtBzU05Hm/BAo4NkAuo=
- github.com/vxcontrol/langchaingo v0.1.14-update.5 h1:QIib3znyGg/YnRSRB3ZMxwwfRE2vy+xZ2gDH6zwj9fk=
- github.com/vxcontrol/langchaingo v0.1.14-update.5/go.mod h1:fJal4XqJsYXRFTbAPJpwcJdztea9+1174fSDYacgctU=
-github.com/vxcontrol/langchaingo v0.1.15-0.20260621133759-91d8dfdd9c89 h1:voo1O4CrwWnWxiHDbkpSugMMZtzB3nDda10Ligesz5k=
-github.com/vxcontrol/langchaingo v0.1.15-0.20260621133759-91d8dfdd9c89/go.mod h1:fJal4XqJsYXRFTbAPJpwcJdztea9+1174fSDYacgctU=
++github.com/vxcontrol/langchaingo v0.1.15-0.20260717093433-cf5f5f41f4eb h1:rufkyNpHdmdXXw2YcBZRLocgNMAHrdDDrQyosGubhr4=
++github.com/vxcontrol/langchaingo v0.1.15-0.20260717093433-cf5f5f41f4eb/go.mod h1:VcRflCqD/8y/qPEZaqdtDFCMX378SZwChm5XXuV8xQk=
github.com/wasilibs/go-re2 v1.10.0 h1:vQZEBYZOCA9jdBMmrO4+CvqyCj0x4OomXTJ4a5/urQ0=
github.com/wasilibs/go-re2 v1.10.0/go.mod h1:k+5XqO2bCJS+QpGOnqugyfwC04nw0jaglmjrrkG8U6o=
github.com/wasilibs/wazero-helpers v0.0.0-20240620070341-3dff1577cd52 h1:OvLBa8SqJnZ6P+mjlzc2K7PM22rRUPE1x32G9DTPrC4=
diff --cc backend/pkg/database/converter/converter_test.go
index 266cf391,88617227..fd99d22a
--- a/backend/pkg/database/converter/converter_test.go
+++ b/backend/pkg/database/converter/converter_test.go
@@@ -56,3 -64,112 +64,123 @@@ func TestIsAgentTool(t *testing.T)
})
}
}
+
+ func TestConvertModelsReasoning(t *testing.T) {
+ models := pconfig.ModelsConfig{
+ {Name: "ao", Reasoning: &pconfig.ModelReasoningInfo{
+ Mode: pconfig.ModelReasoningAdaptiveOnly,
+ Efforts: []llms.ReasoningEffort{llms.ReasoningLow, llms.ReasoningEffort("xhigh")},
+ }},
+ {Name: "ad", Reasoning: &pconfig.ModelReasoningInfo{Mode: pconfig.ModelReasoningAdaptive}},
+ {Name: "bd", Reasoning: &pconfig.ModelReasoningInfo{Mode: pconfig.ModelReasoningBudget}},
+ {Name: "none"},
+ }
+
+ byName := make(map[string]*model.ModelConfig)
+ for _, m := range ConvertModels(models, reasoning.ProviderUnknown) {
+ byName[m.Name] = m
+ }
+
+ require.NotNil(t, byName["ao"].Reasoning)
+ require.NotNil(t, byName["ao"].Reasoning.Mode)
+ assert.Equal(t, model.ModelReasoningModeAdaptiveOnly, *byName["ao"].Reasoning.Mode, "hyphenated pconfig value must map to the GraphQL underscore enum")
+ assert.Equal(t, []model.ReasoningEffort{model.ReasoningEffort("low"), model.ReasoningEffort("xhigh")}, byName["ao"].Reasoning.Efforts)
+
+ require.NotNil(t, byName["ad"].Reasoning.Mode)
+ assert.Equal(t, model.ModelReasoningModeAdaptive, *byName["ad"].Reasoning.Mode)
+ require.NotNil(t, byName["bd"].Reasoning.Mode)
+ assert.Equal(t, model.ModelReasoningModeBudget, *byName["bd"].Reasoning.Mode)
+
+ assert.Nil(t, byName["none"].Reasoning, "model without a reasoning descriptor maps to nil")
+ }
+
+ func TestConvertModels_OffCapability(t *testing.T) {
+ tr := func(b bool) *bool { return &b }
+ models := pconfig.ModelsConfig{
+ // Gemini: thinking:true, no reasoning block -> capability still surfaced,
+ // Off effective via thinkingBudget:0.
+ {Name: "gemini-2.5-flash", Thinking: tr(true)},
- // Unclassified OpenAI-family reasoning model: Off is a silent no-op (OffOmit,
- // default-on unknown) -> cannotDisable must be true so the UI hides Off.
++ // Reasoning model with no provider-specific disable wire (OffOmit) and
++ // unknown default-on -> Off is a silent no-op, so cannotDisable is true.
+ {Name: "glm-5.2", Reasoning: &pconfig.ModelReasoningInfo{Mode: pconfig.ModelReasoningBudget}},
+ // Adaptive-only but off by default: Off works by omission -> disablable.
+ {Name: "claude-opus-4-8", Reasoning: &pconfig.ModelReasoningInfo{Mode: pconfig.ModelReasoningAdaptiveOnly}},
+ // Always-on Claude: Off is rejected (OffUnsupported) -> not disablable.
+ {Name: "claude-fable-5", Reasoning: &pconfig.ModelReasoningInfo{Mode: pconfig.ModelReasoningAdaptiveOnly}},
+ // Plain non-thinking model: no reasoning capability at all.
+ {Name: "gpt-4.1"},
+ }
+
+ byProvider := map[string]reasoning.Provider{
+ "gemini-2.5-flash": reasoning.ProviderGoogleAI,
- "glm-5.2": reasoning.ProviderOpenAI,
- "claude-opus-4-8": reasoning.ProviderAnthropic,
- "claude-fable-5": reasoning.ProviderAnthropic,
- "gpt-4.1": reasoning.ProviderOpenAI,
++ // ProviderUnknown: ResolveOff returns OffOmit (no clean disable signal).
++ // ProviderOpenAI would instead return OffEffortNone (optimistic).
++ "glm-5.2": reasoning.ProviderUnknown,
++ "claude-opus-4-8": reasoning.ProviderAnthropic,
++ "claude-fable-5": reasoning.ProviderAnthropic,
++ "gpt-4.1": reasoning.ProviderOpenAI,
+ }
+
+ get := func(name string) *model.ModelReasoningInfo {
+ for _, mc := range ConvertModels(pconfig.ModelsConfig{lookup(models, name)}, byProvider[name]) {
+ if mc.Name == name {
+ return mc.Reasoning
+ }
+ }
+ return nil
+ }
+
+ // Gemini: capability present, Off effective.
+ gem := get("gemini-2.5-flash")
+ require.NotNil(t, gem, "thinking-capable model must surface reasoning capability")
+ require.NotNil(t, gem.CannotDisable)
+ assert.False(t, *gem.CannotDisable, "Gemini Off works via thinkingBudget:0")
+
- // glm-5.2: Off is a no-op -> cannotDisable true (UI hides Off).
++ // glm-5.2: OffOmit + unknown default-on -> no-op, hide Off.
+ glm := get("glm-5.2")
+ require.NotNil(t, glm)
+ require.NotNil(t, glm.CannotDisable)
- assert.True(t, *glm.CannotDisable, "unclassified default-on model: Off is a no-op, must hide it")
++ assert.True(t, *glm.CannotDisable, "OffOmit with unknown default-on is a no-op, must hide Off")
++
++ // Same model on OpenAI: ResolveOff is optimistic (OffEffortNone) -> Off offered.
++ glmOA := ConvertModels(
++ pconfig.ModelsConfig{lookup(models, "glm-5.2")},
++ reasoning.ProviderOpenAI,
++ )[0].Reasoning
++ require.NotNil(t, glmOA)
++ require.NotNil(t, glmOA.CannotDisable)
++ assert.False(t, *glmOA.CannotDisable, "OpenAI ResolveOff attempts effort none")
+
+ // opus-4-8: off by default -> disablable.
+ opus := get("claude-opus-4-8")
+ require.NotNil(t, opus.CannotDisable)
+ assert.False(t, *opus.CannotDisable, "off-by-default model: Off works by omission")
+
+ // fable-5: always on -> not disablable.
+ fable := get("claude-fable-5")
+ require.NotNil(t, fable.CannotDisable)
+ assert.True(t, *fable.CannotDisable, "always-on model rejects Off")
+
+ // non-thinking model: no capability.
+ assert.Nil(t, get("gpt-4.1"), "non-thinking model has no reasoning capability")
+ }
+
+ func lookup(ms pconfig.ModelsConfig, name string) pconfig.ModelConfig {
+ for _, m := range ms {
+ if m.Name == name {
+ return m
+ }
+ }
+ return pconfig.ModelConfig{}
+ }
+
+ func TestOffEffective(t *testing.T) {
+ tr := func(b bool) *bool { return &b }
+ assert.True(t, offEffective(reasoning.OffDisableClaude, nil))
+ assert.True(t, offEffective(reasoning.OffZeroBudget, nil))
+ assert.True(t, offEffective(reasoning.OffEffortNone, nil))
+ assert.True(t, offEffective(reasoning.OffOmit, tr(false)), "off-by-default: omit disables")
+ assert.False(t, offEffective(reasoning.OffOmit, tr(true)), "default-on + omit = no-op")
+ assert.False(t, offEffective(reasoning.OffOmit, nil), "unknown default + omit = assume no-op")
+ assert.False(t, offEffective(reasoning.OffUnsupported, nil), "rejected")
+ }
diff --cc backend/pkg/database/knowledge.sql.go
index c11337d3,7f454205..4e7c69a5
--- a/backend/pkg/database/knowledge.sql.go
+++ b/backend/pkg/database/knowledge.sql.go
@@@ -453,3 -453,37 +453,36 @@@ func (q *Queries) UpdateKnowledgeDocume
err := row.Scan(&i.ID, &i.Document, &i.Cmetadata)
return i, err
}
+
+ const updateKnowledgeDocumentMetadata = `-- name: UpdateKnowledgeDocumentMetadata :one
+ UPDATE langchain_pg_embedding
+ SET cmetadata = $1::json
+ WHERE uuid::text = $2
+ AND collection_id = (SELECT uuid FROM langchain_pg_collection WHERE name = 'langchain')
+ RETURNING
+ uuid::text AS id,
+ COALESCE(document, '') AS document,
+ COALESCE(cmetadata::text, '{}') AS cmetadata
+ `
+
+ type UpdateKnowledgeDocumentMetadataParams struct {
+ Cmetadata pqtype.NullRawMessage `json:"cmetadata"`
+ Uuid sql.NullString `json:"uuid"`
+ }
+
+ type UpdateKnowledgeDocumentMetadataRow struct {
+ ID string `json:"id"`
+ Document string `json:"document"`
+ Cmetadata sql.NullString `json:"cmetadata"`
+ }
+
+ // Update only the document's metadata, leaving its text and embedding intact.
-// Used when an update changes no content (e.g. rename), so no embedder is needed.
++// Used when an update changes no content (e.g. rename) so re-embedding — and
++// therefore a configured embedder — is unnecessary.
++// cmetadata must be valid JSON text.
+ func (q *Queries) UpdateKnowledgeDocumentMetadata(ctx context.Context, arg UpdateKnowledgeDocumentMetadataParams) (UpdateKnowledgeDocumentMetadataRow, error) {
- row := q.db.QueryRowContext(ctx, updateKnowledgeDocumentMetadata,
- arg.Cmetadata,
- arg.Uuid,
- )
++ row := q.db.QueryRowContext(ctx, updateKnowledgeDocumentMetadata, arg.Cmetadata, arg.Uuid)
+ var i UpdateKnowledgeDocumentMetadataRow
+ err := row.Scan(&i.ID, &i.Document, &i.Cmetadata)
+ return i, err
+ }
diff --cc backend/pkg/database/querier.go
index 8d80eb45,36d3143b..2f065052
--- a/backend/pkg/database/querier.go
+++ b/backend/pkg/database/querier.go
@@@ -286,6 -286,7 +286,11 @@@ type Querier interface
// embedding must be formatted as a PostgreSQL vector literal: '[f1,f2,...]'
// cmetadata must be valid JSON text.
UpdateKnowledgeDocument(ctx context.Context, arg UpdateKnowledgeDocumentParams) (UpdateKnowledgeDocumentRow, error)
++ // Update only the document's metadata, leaving its text and embedding intact.
++ // Used when an update changes no content (e.g. rename) so re-embedding — and
++ // therefore a configured embedder — is unnecessary.
++ // cmetadata must be valid JSON text.
+ UpdateKnowledgeDocumentMetadata(ctx context.Context, arg UpdateKnowledgeDocumentMetadataParams) (UpdateKnowledgeDocumentMetadataRow, error)
UpdateMsgChain(ctx context.Context, arg UpdateMsgChainParams) (Msgchain, error)
UpdateMsgChainUsage(ctx context.Context, arg UpdateMsgChainUsageParams) (Msgchain, error)
UpdateMsgLogResult(ctx context.Context, arg UpdateMsgLogResultParams) (Msglog, error)
diff --cc backend/pkg/docker/client_test.go
index 00000000,8d8a616c..f7ae3a23
mode 000000,100644..100644
--- a/backend/pkg/docker/client_test.go
+++ b/backend/pkg/docker/client_test.go
@@@ -1,0 -1,198 +1,262 @@@
+ package docker
+
+ import (
++ "bytes"
+ "context"
++ "encoding/binary"
+ "errors"
+ "fmt"
+ "math/rand"
++ "strings"
+ "sync/atomic"
+ "testing"
+ "time"
+
+ "github.com/docker/docker/api/types/container"
+ )
+
-func okStat(_ context.Context, name string) (container.PathStat, error) {
- return container.PathStat{Name: name, Size: int64(len(name))}, nil
-}
-
-func failNames(failures []statFailure) map[string]bool {
- m := make(map[string]bool, len(failures))
- for _, f := range failures {
- m[f.name] = true
- }
- return m
-}
-
+ func TestStatContainerEntries_AllSucceed(t *testing.T) {
+ names := []string{"c", "a", "b", "z", "m"}
+ stats, failures := statContainerEntries(context.Background(), names, 20, okStat)
+ if len(failures) != 0 {
+ t.Fatalf("unexpected failures: %v", failures)
+ }
+ if len(stats) != len(names) {
+ t.Fatalf("got %d stats, want %d", len(stats), len(names))
+ }
+ }
+
+ func TestStatContainerEntries_Empty(t *testing.T) {
+ stats, failures := statContainerEntries(context.Background(), nil, 20, okStat)
+ if len(stats) != 0 || len(failures) != 0 {
+ t.Fatalf("got %d stats / %d failures, want 0/0", len(stats), len(failures))
+ }
+ }
+
+ // A per-entry stat error must NOT abort the batch: the readable entries are
+ // returned and the failing ones come back as failures — partial success.
+ func TestStatContainerEntries_PartialSuccessNotAborting(t *testing.T) {
+ names := []string{"e0", "e1", "e2", "e3", "e4"}
+ fail := map[string]bool{"e1": true, "e3": true}
+ statFn := func(_ context.Context, name string) (container.PathStat, error) {
+ if fail[name] {
+ return container.PathStat{}, fmt.Errorf("boom %s", name)
+ }
+ return container.PathStat{Name: name}, nil
+ }
+ stats, failures := statContainerEntries(context.Background(), names, 20, statFn)
+
+ fn := failNames(failures)
+ if len(fn) != 2 || !fn["e1"] || !fn["e3"] {
+ t.Fatalf("want failures {e1,e3}, got %v", fn)
+ }
+ wantOK := []string{"e0", "e2", "e4"} // successes, preserved in input order
+ if len(stats) != len(wantOK) {
+ t.Fatalf("want %d readable entries, got %d", len(wantOK), len(stats))
+ }
+ for i, s := range stats {
+ if s.Name != wantOK[i] {
+ t.Fatalf("success order at %d: got %q want %q", i, s.Name, wantOK[i])
+ }
+ }
+ }
+
+ // The pool never runs more than `workers` stat calls at once — the load-bearing
+ // bound against the Docker daemon.
+ func TestStatContainerEntries_RespectsConcurrencyLimit(t *testing.T) {
+ const workers = 5
+ names := make([]string, 100)
+ for i := range names {
+ names[i] = fmt.Sprintf("e%d", i)
+ }
+ var cur, max atomic.Int64
+ statFn := func(_ context.Context, name string) (container.PathStat, error) {
+ c := cur.Add(1)
+ for {
+ m := max.Load()
+ if c <= m || max.CompareAndSwap(m, c) {
+ break
+ }
+ }
+ time.Sleep(2 * time.Millisecond)
+ cur.Add(-1)
+ return container.PathStat{Name: name}, nil
+ }
+ if _, failures := statContainerEntries(context.Background(), names, workers, statFn); len(failures) != 0 {
+ t.Fatalf("unexpected failures: %v", failures)
+ }
+ if got := max.Load(); got > workers {
+ t.Fatalf("concurrency exceeded limit: peak=%d, limit=%d", got, workers)
+ } else if got < 2 {
+ t.Fatalf("statFn never overlapped (peak=%d) — test is not exercising concurrency", got)
+ }
+ }
+
+ // The caller's context reaches each stat call; a cancellation turns entries into
+ // failures rather than blanking the whole batch.
+ func TestStatContainerEntries_ContextPropagates(t *testing.T) {
+ ctx, cancel := context.WithCancel(context.Background())
+ names := []string{"a", "b", "c", "d", "e"}
+ var once atomic.Bool
+ statFn := func(ctx context.Context, name string) (container.PathStat, error) {
+ if once.CompareAndSwap(false, true) {
+ cancel()
+ }
+ select {
+ case <-ctx.Done():
+ return container.PathStat{}, ctx.Err()
+ case <-time.After(2 * time.Second):
+ return container.PathStat{Name: name}, nil
+ }
+ }
+ _, failures := statContainerEntries(ctx, names, 20, statFn)
+ if len(failures) == 0 {
+ t.Fatal("expected failures once the context is cancelled")
+ }
+ for _, f := range failures {
+ if !errors.Is(f.err, context.Canceled) {
+ t.Fatalf("failure %q: want context.Canceled, got %v", f.name, f.err)
+ }
+ }
+ }
+
+ // A non-positive worker count must fall back to a safe bound rather than
+ // deadlock (errgroup SetLimit(0)) or run unbounded (SetLimit(<0)).
+ func TestStatContainerEntries_NonPositiveWorkersFallBack(t *testing.T) {
+ names := make([]string, 50)
+ for i := range names {
+ names[i] = fmt.Sprintf("e%d", i)
+ }
+ for _, workers := range []int{0, -1} {
+ t.Run(fmt.Sprintf("workers=%d", workers), func(t *testing.T) {
+ done := make(chan int, 1)
+ go func() {
+ stats, _ := statContainerEntries(context.Background(), names, workers, okStat)
+ done <- len(stats)
+ }()
+ select {
+ case n := <-done:
+ if n != len(names) {
+ t.Fatalf("got %d stats, want %d", n, len(names))
+ }
+ case <-time.After(3 * time.Second):
+ t.Fatal("statContainerEntries hung with a non-positive worker count")
+ }
+ })
+ }
+ }
+
+ // Property-based: over randomized shapes nothing is lost — every failing name
+ // lands in failures, every other name yields a stat, and stats+failures == n.
+ func TestStatContainerEntries_PropertyFuzz(t *testing.T) {
+ rng := rand.New(rand.NewSource(2))
+ for iter := 0; iter < 300; iter++ {
+ n := rng.Intn(50)
+ names := make([]string, n)
+ wantFail := make(map[string]bool, n)
+ for i := range names {
+ names[i] = fmt.Sprintf("it%d-e%d", iter, i)
+ if rng.Intn(3) == 0 {
+ wantFail[names[i]] = true
+ }
+ }
+ statFn := func(_ context.Context, name string) (container.PathStat, error) {
+ if wantFail[name] {
+ return container.PathStat{}, fmt.Errorf("boom %s", name)
+ }
+ return container.PathStat{Name: name}, nil
+ }
+ stats, failures := statContainerEntries(context.Background(), names, 20, statFn)
+
+ if len(stats)+len(failures) != n {
+ t.Fatalf("iter %d: lost data — %d stats + %d failures != %d names", iter, len(stats), len(failures), n)
+ }
+ gotFail := failNames(failures)
+ if len(gotFail) != len(wantFail) {
+ t.Fatalf("iter %d: want %d failures, got %d", iter, len(wantFail), len(gotFail))
+ }
+ for nm := range wantFail {
+ if !gotFail[nm] {
+ t.Fatalf("iter %d: missing failure for %q", iter, nm)
+ }
+ }
+ for _, s := range stats {
+ if wantFail[s.Name] {
+ t.Fatalf("iter %d: %q failed but appears in stats", iter, s.Name)
+ }
+ }
+ }
+ }
++
++func TestParseFindEntries(t *testing.T) {
++ entries, truncated := parseFindEntries([]byte("a\x00b\x00\x00c\x00"))
++ if truncated {
++ t.Error("small input must not truncate")
++ }
++ if len(entries) != 3 || entries[0] != "a" || entries[1] != "b" || entries[2] != "c" {
++ t.Fatalf("split/skip-empty wrong: %q", entries)
++ }
++
++ // exactly at the cap → not truncated
++ e, tr := parseFindEntries([]byte(strings.Repeat("x\x00", maxListEntries)))
++ if tr || len(e) != maxListEntries {
++ t.Fatalf("at cap: truncated=%v len=%d (want false, %d)", tr, len(e), maxListEntries)
++ }
++
++ // cap+1 → truncated, capped to maxListEntries
++ e, tr = parseFindEntries([]byte(strings.Repeat("x\x00", maxListEntries+1)))
++ if !tr || len(e) != maxListEntries {
++ t.Fatalf("over cap: truncated=%v len=%d (want true, %d)", tr, len(e), maxListEntries)
++ }
++}
++
++func TestDemuxExecStdout_StdoutOnly_AndByteCap(t *testing.T) {
++ // stdout is returned; stderr (id 2) is discarded
++ in := append(listingFrame(1, "hello"), listingFrame(2, "diagnostic")...)
++ out, err := demuxExecStdout(bytes.NewReader(in), 1<<20)
++ if err != nil {
++ t.Fatalf("unexpected err: %v", err)
++ }
++ if string(out) != "hello" {
++ t.Fatalf("want stdout only, got %q", out)
++ }
++
++ // stdout exceeding the cap → error before materializing the full buffer
++ _, err = demuxExecStdout(bytes.NewReader(listingFrame(1, strings.Repeat("x", 50))), 10)
++ if err == nil || !strings.Contains(err.Error(), "listing output exceeded") {
++ t.Fatalf("want cap error, got %v", err)
++ }
++}
++
++func TestDemuxExecStdout_TruncatedAndSystemerr(t *testing.T) {
++ // a header cut short mid-frame must error, not silently drop the tail
++ torn := append(listingFrame(1, "a.txt\x00"), 0x01, 0x00, 0x00) // 3 stray header bytes
++ if _, err := demuxExecStdout(bytes.NewReader(torn), 1<<20); err == nil || !strings.Contains(err.Error(), "truncated") {
++ t.Fatalf("want truncated-stream error, got %v", err)
++ }
++
++ // a systemerr (stream id 3) daemon error must surface, not be discarded
++ sys := append(listingFrame(1, "ok"), listingFrame(3, "daemon connection reset")...)
++ if _, err := demuxExecStdout(bytes.NewReader(sys), 1<<20); err == nil || !strings.Contains(err.Error(), "systemerr") {
++ t.Fatalf("want systemerr, got %v", err)
++ }
++}
++
++func listingFrame(streamID byte, payload string) []byte {
++ h := make([]byte, 8)
++ h[0] = streamID
++ binary.BigEndian.PutUint32(h[4:8], uint32(len(payload)))
++ return append(h, []byte(payload)...)
++}
++
++func okStat(_ context.Context, name string) (container.PathStat, error) {
++ return container.PathStat{Name: name, Size: int64(len(name))}, nil
++}
++
++func failNames(failures []statFailure) map[string]bool {
++ m := make(map[string]bool, len(failures))
++ for _, f := range failures {
++ m[f.name] = true
++ }
++ return m
++}
diff --cc backend/pkg/graph/context.go
index e415d453,e415d453..a29c365c
--- a/backend/pkg/graph/context.go
+++ b/backend/pkg/graph/context.go
@@@ -162,6 -162,6 +162,35 @@@ func validateUserResources
return result, nil
}
++// Knowledge-document field limits. MUST stay in sync with the REST request models
++// (server/models/knowledge.go `validate` tags) and the frontend zod schema.
++const (
++ maxKnowledgeContentLen = 65536
++ maxKnowledgeQuestionLen = 2048
++ maxKnowledgeDescriptionLen = 1000
++ maxKnowledgeCodeLangLen = 100
++)
++
++// maxAPITokenNameLen MUST stay in sync with the REST model (server/models/api_tokens.go
++// `validate` tag) and the frontend tokenNameSchema.
++const maxAPITokenNameLen = 100
++
++func validateKnowledgeFieldLengths(content string, question, description, codeLang *string) error {
++ if len(content) > maxKnowledgeContentLen {
++ return fmt.Errorf("content must not exceed %d characters", maxKnowledgeContentLen)
++ }
++ if question != nil && len(*question) > maxKnowledgeQuestionLen {
++ return fmt.Errorf("question must not exceed %d characters", maxKnowledgeQuestionLen)
++ }
++ if description != nil && len(*description) > maxKnowledgeDescriptionLen {
++ return fmt.Errorf("description must not exceed %d characters", maxKnowledgeDescriptionLen)
++ }
++ if codeLang != nil && len(*codeLang) > maxKnowledgeCodeLangLen {
++ return fmt.Errorf("code language must not exceed %d characters", maxKnowledgeCodeLangLen)
++ }
++ return nil
++}
++
func convertFlowFiles(files flowfiles.Files) []*model.FlowFile {
converted := make([]*model.FlowFile, 0, len(files.Files))
for _, file := range files.Files {
diff --cc backend/pkg/graph/context_test.go
index 724e0f81,724e0f81..31821c84
--- a/backend/pkg/graph/context_test.go
+++ b/backend/pkg/graph/context_test.go
@@@ -2,6 -2,6 +2,7 @@@ package grap
import (
"context"
++ "strings"
"testing"
"github.com/stretchr/testify/assert"
@@@ -386,3 -386,3 +387,44 @@@ func TestGetUserPermissions_NilSlice(t
require.NoError(t, err)
assert.Nil(t, perms)
}
++
++func TestValidateKnowledgeFieldLengths(t *testing.T) {
++ tests := []struct {
++ name string
++ content string
++ question *string
++ description *string
++ codeLang *string
++ wantErr bool
++ }{
++ {name: "all empty/nil is valid", content: ""},
++ {name: "all fields at their max is valid",
++ content: strings.Repeat("a", maxKnowledgeContentLen),
++ question: strOfLen(maxKnowledgeQuestionLen),
++ description: strOfLen(maxKnowledgeDescriptionLen),
++ codeLang: strOfLen(maxKnowledgeCodeLangLen),
++ },
++ {name: "content one over max", content: strings.Repeat("a", maxKnowledgeContentLen+1), wantErr: true},
++ {name: "question one over max", question: strOfLen(maxKnowledgeQuestionLen + 1), wantErr: true},
++ {name: "description one over max", description: strOfLen(maxKnowledgeDescriptionLen + 1), wantErr: true},
++ {name: "code language one over max", codeLang: strOfLen(maxKnowledgeCodeLangLen + 1), wantErr: true},
++ {name: "nil optionals with content at max", content: strings.Repeat("a", maxKnowledgeContentLen)},
++ }
++
++ for _, tt := range tests {
++ t.Run(tt.name, func(t *testing.T) {
++ err := validateKnowledgeFieldLengths(tt.content, tt.question, tt.description, tt.codeLang)
++ if tt.wantErr && err == nil {
++ t.Fatalf("expected an error, got nil")
++ }
++ if !tt.wantErr && err != nil {
++ t.Fatalf("expected no error, got %v", err)
++ }
++ })
++ }
++}
++
++func strOfLen(n int) *string {
++ s := strings.Repeat("a", n)
++ return &s
++}
diff --cc backend/pkg/graph/schema.resolvers.go
index b5c049dd,360b6a56..d285d0ef
--- a/backend/pkg/graph/schema.resolvers.go
+++ b/backend/pkg/graph/schema.resolvers.go
@@@ -2059,8 -2138,11 +2113,8 @@@ func (r *queryResolver) SettingsProvide
}
case provider.ProviderBedrock:
config.Default.Bedrock = mpcfg
- if models, err := bedrock.DefaultModels(r.Config); err == nil {
+ if models, err := bedrock.DefaultModels(); err == nil {
- config.Models.Bedrock = converter.ConvertModels(models)
+ config.Models.Bedrock = converter.ConvertModels(models, prvtype.ReasoningProvider())
- } else {
- // A bad BEDROCK_MODELS_PATH otherwise yields an empty model list with no signal.
- r.Logger.WithError(err).Warn("failed to load bedrock models")
}
case provider.ProviderOllama:
config.Default.Ollama = mpcfg
@@@ -2100,6 -2187,6 +2159,9 @@@
config.Enabled.Gemini = true
case provider.ProviderBedrock:
config.Enabled.Bedrock = true
++ if p, ok := defaultProviders[provider.DefaultProviderNameBedrock]; ok {
++ config.Models.Bedrock = converter.ConvertModels(p.GetModels(), prvtype.ReasoningProvider())
++ }
case provider.ProviderOllama:
config.Enabled.Ollama = true
if p, ok := defaultProviders[provider.DefaultProviderNameOllama]; ok {
diff --cc backend/pkg/observability/langfuse/observer_test.go
index 00000000,95835470..95835470
mode 000000,100644..100644
--- a/backend/pkg/observability/langfuse/observer_test.go
+++ b/backend/pkg/observability/langfuse/observer_test.go
diff --cc backend/pkg/observability/obs_test.go
index 00000000,e5bb2ffb..459bd5ca
mode 000000,100644..100644
--- a/backend/pkg/observability/obs_test.go
+++ b/backend/pkg/observability/obs_test.go
@@@ -1,0 -1,114 +1,222 @@@
+ package observability
+
+ import (
+ "context"
++ "errors"
+ "net"
+ "sync/atomic"
+ "testing"
+ "time"
+
+ "pentagi/pkg/config"
+ "pentagi/pkg/observability/langfuse"
+
++ otellog "go.opentelemetry.io/otel/log"
++ otelloggernoop "go.opentelemetry.io/otel/log/noop"
++ otelmetric "go.opentelemetry.io/otel/metric"
++ otelmetricnoop "go.opentelemetry.io/otel/metric/noop"
++ oteltrace "go.opentelemetry.io/otel/trace"
++ oteltracenoop "go.opentelemetry.io/otel/trace/noop"
+ collogsv1 "go.opentelemetry.io/proto/otlp/collector/logs/v1"
+ colmetricsv1 "go.opentelemetry.io/proto/otlp/collector/metrics/v1"
+ coltracev1 "go.opentelemetry.io/proto/otlp/collector/trace/v1"
+ "google.golang.org/grpc"
+ )
+
++type fakeLangfuseClient struct {
++ shutdownCalled bool
++ flushCalled bool
++ shutdownErr error
++ flushErr error
++}
++
++func (f *fakeLangfuseClient) API() langfuse.Client { return langfuse.Client{} }
++func (f *fakeLangfuseClient) Observer() langfuse.Observer { return langfuse.NewNoopObserver() }
++func (f *fakeLangfuseClient) Shutdown(context.Context) error {
++ f.shutdownCalled = true
++ return f.shutdownErr
++}
++func (f *fakeLangfuseClient) ForceFlush(context.Context) error {
++ f.flushCalled = true
++ return f.flushErr
++}
++
++type fakeTelemetryClient struct {
++ shutdownCalled bool
++ flushCalled bool
++ shutdownErr error
++ flushErr error
++}
++
++func (f *fakeTelemetryClient) Logger() otellog.LoggerProvider {
++ return otelloggernoop.NewLoggerProvider()
++}
++func (f *fakeTelemetryClient) Tracer() oteltrace.TracerProvider {
++ return oteltracenoop.NewTracerProvider()
++}
++func (f *fakeTelemetryClient) Meter() otelmetric.MeterProvider {
++ return otelmetricnoop.NewMeterProvider()
++}
++func (f *fakeTelemetryClient) Shutdown(context.Context) error {
++ f.shutdownCalled = true
++ return f.shutdownErr
++}
++func (f *fakeTelemetryClient) ForceFlush(context.Context) error {
++ f.flushCalled = true
++ return f.flushErr
++}
++
++func TestObserverShutdownDrainsBothClients(t *testing.T) {
++ lf := &fakeLangfuseClient{}
++ ot := &fakeTelemetryClient{}
++ obs := &observer{lfclient: lf, otelclient: ot}
++
++ if err := obs.Shutdown(context.Background()); err != nil {
++ t.Fatalf("Shutdown: %v", err)
++ }
++ if !lf.shutdownCalled {
++ t.Error("langfuse client was not shut down")
++ }
++ if !ot.shutdownCalled {
++ t.Error("otel client was not shut down")
++ }
++}
++
++func TestObserverFlushDrainsBothClients(t *testing.T) {
++ lf := &fakeLangfuseClient{}
++ ot := &fakeTelemetryClient{}
++ obs := &observer{lfclient: lf, otelclient: ot}
++
++ if err := obs.Flush(context.Background()); err != nil {
++ t.Fatalf("Flush: %v", err)
++ }
++ if !lf.flushCalled {
++ t.Error("langfuse client was not flushed")
++ }
++ if !ot.flushCalled {
++ t.Error("otel client was not flushed")
++ }
++}
++
++func TestObserverShutdownAggregatesErrorsAndDrainsBoth(t *testing.T) {
++ lfErr := errors.New("langfuse boom")
++ otErr := errors.New("otel boom")
++ lf := &fakeLangfuseClient{shutdownErr: lfErr}
++ ot := &fakeTelemetryClient{shutdownErr: otErr}
++ obs := &observer{lfclient: lf, otelclient: ot}
++
++ err := obs.Shutdown(context.Background())
++ if !errors.Is(err, lfErr) || !errors.Is(err, otErr) {
++ t.Fatalf("want both errors joined, got %v", err)
++ }
++ if !lf.shutdownCalled || !ot.shutdownCalled {
++ t.Fatal("a client was skipped despite the other erroring")
++ }
++}
++
++func TestObserverShutdownFlushNilClientsAreNoops(t *testing.T) {
++ obs := &observer{}
++ if err := obs.Shutdown(context.Background()); err != nil {
++ t.Errorf("Shutdown with no clients: %v", err)
++ }
++ if err := obs.Flush(context.Background()); err != nil {
++ t.Errorf("Flush with no clients: %v", err)
++ }
++}
++
+ type recordingTraceCollector struct {
+ coltracev1.UnimplementedTraceServiceServer
+ spanBatches int32
+ }
+
+ func (r *recordingTraceCollector) Export(context.Context, *coltracev1.ExportTraceServiceRequest) (*coltracev1.ExportTraceServiceResponse, error) {
+ atomic.AddInt32(&r.spanBatches, 1)
+ return &coltracev1.ExportTraceServiceResponse{}, nil
+ }
+
+ // blockingLangfuse models the real langfuse leg: ForceFlush/Shutdown ignore the
+ // passed ctx and block on the observer's own machinery, so a pending batch against
+ // a down endpoint blocks past the drain budget.
+ type blockingLangfuse struct {
+ release chan struct{}
+ flushCalls int32
+ }
+
+ func (b *blockingLangfuse) API() langfuse.Client { return langfuse.Client{} }
+ func (b *blockingLangfuse) Observer() langfuse.Observer { return langfuse.NewNoopObserver() }
+ func (b *blockingLangfuse) ForceFlush(context.Context) error {
+ atomic.AddInt32(&b.flushCalls, 1)
+ <-b.release
+ return nil
+ }
+ func (b *blockingLangfuse) Shutdown(context.Context) error {
+ <-b.release
+ return nil
+ }
+
+ func newRecordingOtel(t *testing.T) (*telemetryClient, *recordingTraceCollector) {
+ t.Helper()
+ ln, err := net.Listen("tcp", "127.0.0.1:0")
+ if err != nil {
+ t.Fatalf("listen: %v", err)
+ }
+ rec := &recordingTraceCollector{}
+ srv := grpc.NewServer()
+ collogsv1.RegisterLogsServiceServer(srv, fakeLogsCollector{})
+ colmetricsv1.RegisterMetricsServiceServer(srv, fakeMetricsCollector{})
+ coltracev1.RegisterTraceServiceServer(srv, rec)
+ go func() { _ = srv.Serve(ln) }()
+ t.Cleanup(srv.Stop)
+
+ tc, err := NewTelemetryClient(context.Background(), &config.Config{TelemetryEndpoint: ln.Addr().String()})
+ if err != nil {
+ t.Fatalf("NewTelemetryClient: %v", err)
+ }
+ return tc.(*telemetryClient), rec
+ }
+
+ func bufferOneSpan(o *observer) {
+ _, span := o.otelclient.Tracer().Tracer("drain-test").Start(context.Background(), "s")
+ span.End()
+ }
+
+ // A healthy otel sink is delivered even while an unhealthy langfuse sink blocks.
+ func TestObserverDrain_DeliversOtelDespiteBlockedLangfuse(t *testing.T) {
+ otel, rec := newRecordingOtel(t)
+ lf := &blockingLangfuse{release: make(chan struct{})}
+ t.Cleanup(func() { close(lf.release) })
+
+ obs := &observer{lfclient: lf, otelclient: otel}
+ bufferOneSpan(obs)
+
+ ctx, cancel := context.WithTimeout(context.Background(), 800*time.Millisecond)
+ defer cancel()
+ _ = obs.Drain(ctx)
+
+ if atomic.LoadInt32(&lf.flushCalls) == 0 {
+ t.Fatal("precondition: langfuse ForceFlush should have been attempted")
+ }
+ if got := atomic.LoadInt32(&rec.spanBatches); got != 1 {
+ t.Fatalf("otel span must be delivered despite a blocked langfuse; got %d export batches", got)
+ }
+ }
+
+ func TestObserverDrain_ReturnsWithinDeadlineWhenSinkBlocks(t *testing.T) {
+ otel, _ := newRecordingOtel(t)
+ lf := &blockingLangfuse{release: make(chan struct{})}
+ t.Cleanup(func() { close(lf.release) })
+
+ obs := &observer{lfclient: lf, otelclient: otel}
+
+ start := time.Now()
+ ctx, cancel := context.WithTimeout(context.Background(), 300*time.Millisecond)
+ defer cancel()
+ err := obs.Drain(ctx)
+
+ if elapsed := time.Since(start); elapsed > time.Second {
+ t.Fatalf("Drain hung past its deadline: %v", elapsed)
+ }
+ if err == nil {
+ t.Fatal("Drain should report the deadline was hit while a sink was blocked")
+ }
+ }
diff --cc backend/pkg/providers/anthropic/anthropic.go
index 914901b2,8bda8f7f..d9a26aeb
--- a/backend/pkg/providers/anthropic/anthropic.go
+++ b/backend/pkg/providers/anthropic/anthropic.go
@@@ -184,6 -189,6 +189,28 @@@ func (p *anthropicProvider) CallWithToo
)
}
++// CallWithExtraOptions: extra wins over both the config and the auto-adaptive
++// append below, since it's appended last.
++func (p *anthropicProvider) CallWithExtraOptions(
++ ctx context.Context,
++ opt pconfig.ProviderOptionsType,
++ chain []llms.MessageContent,
++ tools []llms.Tool,
++ streamCb streaming.Callback,
++ extra ...llms.CallOption,
++) (*llms.ContentResponse, error) {
++ base := []llms.CallOption{llms.WithStreamingFunc(streamCb)}
++ if len(tools) > 0 {
++ base = append(base, llms.WithTools(tools))
++ }
++ base = append(base, p.providerConfig.GetOptionsForType(opt)...)
++
++ ctx, options := p.providerConfig.PrepareAdaptiveCallOptions(ctx, p.models, opt, base)
++ options = append(options, extra...)
++
++ return provider.WrapGenerateContent(ctx, p, opt, p.llm.GenerateContent, chain, options...)
++}
++
func (p *anthropicProvider) GetUsage(info map[string]any) pconfig.CallUsage {
return pconfig.NewCallUsage(info)
}
diff --cc backend/pkg/providers/anthropic/config.yml
index 1da02d64,1da02d64..ca7200b3
--- a/backend/pkg/providers/anthropic/config.yml
+++ b/backend/pkg/providers/anthropic/config.yml
@@@ -22,12 -22,12 +22,12 @@@ simple_json
cache_write: 1.25
primary_agent:
-- model: claude-sonnet-4-5
++ model: claude-sonnet-5
temperature: 1.0
n: 1
max_tokens: 16384
reasoning:
-- max_tokens: 1024
++ mode: adaptive
price:
input: 3.0
output: 15.0
@@@ -35,12 -35,12 +35,12 @@@
cache_write: 3.75
assistant:
-- model: claude-sonnet-4-5
++ model: claude-sonnet-5
temperature: 1.0
n: 1
max_tokens: 16384
reasoning:
-- max_tokens: 1024
++ mode: adaptive
price:
input: 3.0
output: 15.0
@@@ -48,12 -48,12 +48,13 @@@
cache_write: 3.75
generator:
-- model: claude-opus-4-5
++ model: claude-opus-4-8
temperature: 1.0
n: 1
max_tokens: 32768
reasoning:
-- max_tokens: 4096
++ mode: adaptive
++ effort: xhigh
price:
input: 5.0
output: 25.0
@@@ -61,25 -61,25 +62,27 @@@
cache_write: 6.25
refiner:
-- model: claude-sonnet-4-5
++ model: claude-opus-4-8
temperature: 1.0
n: 1
max_tokens: 20480
reasoning:
-- max_tokens: 1024
++ mode: adaptive
++ effort: high
price:
-- input: 3.0
-- output: 15.0
-- cache_read: 0.3
-- cache_write: 3.75
++ input: 5.0
++ output: 25.0
++ cache_read: 0.5
++ cache_write: 6.25
adviser:
-- model: claude-sonnet-4-5
++ model: claude-sonnet-5
temperature: 1.0
n: 1
max_tokens: 8192
reasoning:
-- max_tokens: 4096
++ mode: adaptive
++ effort: xhigh
price:
input: 3.0
output: 15.0
@@@ -124,12 -124,12 +127,12 @@@ enricher
cache_write: 1.25
coder:
-- model: claude-sonnet-4-5
++ model: claude-sonnet-5
temperature: 1.0
n: 1
max_tokens: 20480
reasoning:
-- max_tokens: 2048
++ mode: adaptive
price:
input: 3.0
output: 15.0
@@@ -137,12 -137,12 +140,12 @@@
cache_write: 3.75
installer:
-- model: claude-sonnet-4-5
++ model: claude-sonnet-5
temperature: 1.0
n: 1
max_tokens: 16384
reasoning:
-- max_tokens: 1024
++ mode: adaptive
price:
input: 3.0
output: 15.0
@@@ -150,12 -150,12 +153,12 @@@
cache_write: 3.75
pentester:
-- model: claude-sonnet-4-5
++ model: claude-sonnet-5
temperature: 1.0
n: 1
max_tokens: 8192
reasoning:
-- max_tokens: 1024
++ mode: adaptive
price:
input: 3.0
output: 15.0
diff --cc backend/pkg/providers/anthropic/models.yml
index 19de257f,d26528d5..ab91d206
--- a/backend/pkg/providers/anthropic/models.yml
+++ b/backend/pkg/providers/anthropic/models.yml
@@@ -1,4 -1,4 +1,30 @@@
--# Claude 4 series - Most capable models for advanced security operations
++# Claude 5 / 4 series - Most capable models for advanced security operations
++- name: claude-fable-5
++ description: Anthropic's most capable widely released model for long-running agents and the most demanding reasoning workloads. Adaptive thinking is always on (budget thinking and an explicit disable are rejected); sampling parameters are not supported.
++ thinking: true
++ reasoning:
++ mode: adaptive-only
++ efforts: [low, medium, high, xhigh, max]
++ release_date: 2026-06-09
++ price:
++ input: 10.0
++ output: 50.0
++ cache_read: 1.0
++ cache_write: 12.5
++
++- name: claude-sonnet-5
++ description: Best combination of speed and intelligence for coding, agents, and professional work at scale. Adaptive thinking only (manual budget thinking is rejected); sampling parameters are not supported.
++ thinking: true
++ reasoning:
++ mode: adaptive-only
++ efforts: [low, medium, high, xhigh, max]
++ release_date: 2026-07-01
++ price:
++ input: 3.0
++ output: 15.0
++ cache_read: 0.3
++ cache_write: 3.75
++
- name: claude-haiku-4-5
description: Fast and efficient model with exceptional function calling and low latency. Ideal for high-frequency security scanning, rapid vulnerability detection, real-time monitoring, and bulk automated testing where speed is paramount. Strong tool orchestration capabilities at minimal cost.
thinking: false
@@@ -19,16 -22,45 +48,45 @@@
cache_read: 0.3
cache_write: 3.75
-- name: claude-opus-4-6
- description: The most intelligent model for building autonomous agents and advanced coding. Unmatched capabilities in complex exploit development, sophisticated penetration testing automation, multi-stage attack simulation, and intelligent security research. Features extended and adaptive thinking for maximum reasoning depth in critical security operations.
- thinking: true
- reasoning:
- mode: adaptive
- efforts: [low, medium, high, max]
- release_date: 2026-02-05
- price:
- input: 5.0
- output: 25.0
- cache_read: 0.5
- cache_write: 6.25
-
+ - name: claude-opus-4-8
+ description: Anthropic Opus 4.8 - flagship model for coding, agents, and deep reasoning in enterprise security workflows. Adaptive thinking only (manual budget thinking is rejected); sampling parameters are not supported.
+ thinking: true
+ reasoning:
+ mode: adaptive-only
+ efforts: [low, medium, high, xhigh, max]
+ release_date: 2026-05-28
+ price:
+ input: 5.0
+ output: 25.0
+ cache_read: 0.5
+ cache_write: 6.25
+
+ - name: claude-opus-4-7
+ description: Anthropic Opus 4.7 for advanced software engineering, long-running agentic tasks, and rigorous security analysis. Adaptive thinking only (manual budget thinking is rejected); sampling parameters are not supported.
+ thinking: true
+ reasoning:
+ mode: adaptive-only
+ efforts: [low, medium, high, xhigh, max]
+ release_date: 2026-04-16
+ price:
+ input: 5.0
+ output: 25.0
+ cache_read: 0.5
+ cache_write: 6.25
+
+- name: claude-opus-4-6
+ description: The most intelligent model for building autonomous agents and advanced coding. Unmatched capabilities in complex exploit development, sophisticated penetration testing automation, multi-stage attack simulation, and intelligent security research. Features extended and adaptive thinking for maximum reasoning depth in critical security operations.
+ thinking: true
++ reasoning:
++ mode: adaptive
++ efforts: [low, medium, high, max]
+ release_date: 2026-02-05
+ price:
+ input: 5.0
+ output: 25.0
+ cache_read: 0.5
+ cache_write: 6.25
+
# Legacy models - Still supported but consider migrating to newer versions
- name: claude-sonnet-4-5
description: State-of-the-art reasoning model with superior analytical depth and enhanced tool integration (superseded by sonnet-4-6). Premier choice for sophisticated penetration testing, advanced threat analysis, complex exploit development, and autonomous security research requiring deep reasoning and precise tool orchestration.
@@@ -49,44 -81,44 +107,3 @@@
output: 25.0
cache_read: 0.5
cache_write: 6.25
--
--- name: claude-opus-4-1
-- description: Advanced reasoning model with strong analytical capabilities (superseded by opus-4-5 and opus-4-6). Excellent for complex penetration testing scenarios, sophisticated threat modeling, and autonomous security operations requiring deep reasoning and tool integration.
-- thinking: true
-- release_date: 2025-08-05
-- price:
-- input: 15.0
-- output: 75.0
-- cache_read: 1.5
-- cache_write: 18.75
--
--- name: claude-sonnet-4-0
-- description: High-performance reasoning model with exceptional analytical capabilities (superseded by sonnet-4-5 and sonnet-4-6). Strong at complex threat modeling, multi-tool coordination, and advanced penetration testing. Maintains excellent function calling and autonomous agent performance.
-- thinking: true
-- release_date: 2025-05-22
-- price:
-- input: 3.0
-- output: 15.0
-- cache_read: 0.3
-- cache_write: 3.75
--
--- name: claude-opus-4-0
-- description: First generation Opus model with strong reasoning capabilities (superseded by newer Opus versions). Capable of handling complex security tasks, multi-step exploit development, and autonomous penetration testing workflows. Consider migrating to opus-4-6 for improved performance.
-- thinking: true
-- release_date: 2025-05-22
-- price:
-- input: 15.0
-- output: 75.0
-- cache_read: 1.5
-- cache_write: 18.75
--
--# Deprecated models - Will be retired, migrate to current models
--- name: claude-3-haiku-20240307
-- description: Legacy fast model from Claude 3 series (DEPRECATED - will be retired on April 19, 2026). Basic capabilities for simple security scanning and automated testing. Migrate to claude-haiku-4-5 for significantly improved performance, extended context, and modern function calling.
-- thinking: false
-- release_date: 2024-03-07
-- price:
-- input: 0.25
-- output: 1.25
-- cache_read: 0.03
-- cache_write: 0.30
diff --cc backend/pkg/providers/bedrock/bedrock.go
index cd484db2,cc217de6..9a844f58
--- a/backend/pkg/providers/bedrock/bedrock.go
+++ b/backend/pkg/providers/bedrock/bedrock.go
@@@ -29,7 -29,7 +29,7 @@@ import
//go:embed config.yml models.yml
var configFS embed.FS
--const BedrockAgentModel = bedrock.ModelAnthropicClaudeSonnet4
++const BedrockAgentModel = bedrock.ModelAnthropicClaudeSonnet46
const BedrockToolCallIDTemplate = "tooluse_{r:22:x}"
@@@ -72,7 -73,47 +73,12 @@@ func DefaultModels() (pconfig.ModelsCon
return nil, err
}
- return pconfig.LoadModelsConfigData(configData)
+ models, err := pconfig.LoadModelsConfigData(configData)
+ if err != nil {
+ return nil, err
+ }
+
- if cfg.BedrockModels == "" {
- return models, nil
- }
-
- externalData, err := os.ReadFile(cfg.BedrockModels)
- if err != nil {
- return nil, err
- }
-
- externalModels, err := pconfig.LoadModelsConfigData(externalData)
- if err != nil {
- return nil, err
- }
-
- return mergeModels(models, externalModels), nil
-}
-
-// mergeModels returns base plus override; on a name collision the override entry wins.
-func mergeModels(base, override pconfig.ModelsConfig) pconfig.ModelsConfig {
- indexByName := make(map[string]int, len(base))
- merged := make(pconfig.ModelsConfig, len(base))
- for i, m := range base {
- indexByName[m.Name] = i
- merged[i] = m
- }
-
- for _, m := range override {
- if i, ok := indexByName[m.Name]; ok {
- merged[i] = m
- } else {
- indexByName[m.Name] = len(merged)
- merged = append(merged, m)
- }
- }
-
- return merged
++ return models, nil
}
type bedrockProvider struct {
@@@ -258,12 -303,13 +268,40 @@@ func (p *bedrockProvider) CallWithTools
configOptions := p.providerConfig.GetOptionsForType(opt)
- // Build final options: config + streaming + cleaned tools LAST (to override any dirty tools from config)
+ // Put cleaned tools after config to override any dirty tools restored from config.
options := append(configOptions, llms.WithStreamingFunc(streamCb), llms.WithTools(tools))
+ ctx, options = p.providerConfig.PrepareAdaptiveCallOptions(ctx, p.models, opt, options)
+
+ return provider.WrapGenerateContent(ctx, p, opt, p.llm.GenerateContent, chain, options...)
+ }
+
++// CallWithExtraOptions: extra wins over both the config and the auto-adaptive
++// append below, since it's appended last.
++func (p *bedrockProvider) CallWithExtraOptions(
++ ctx context.Context,
++ opt pconfig.ProviderOptionsType,
++ chain []llms.MessageContent,
++ tools []llms.Tool,
++ streamCb streaming.Callback,
++ extra ...llms.CallOption,
++) (*llms.ContentResponse, error) {
++ tools = restoreMissedToolsFromChain(chain, tools)
++ tools = cleanToolSchemas(tools)
++
++ configOptions := p.providerConfig.GetOptionsForType(opt)
++
++ base := []llms.CallOption{llms.WithStreamingFunc(streamCb)}
++ base = append(base, configOptions...)
++ if len(tools) > 0 {
++ base = append(base, llms.WithTools(tools))
++ }
++
++ ctx, options := p.providerConfig.PrepareAdaptiveCallOptions(ctx, p.models, opt, base)
++ options = append(options, extra...)
+
+ return provider.WrapGenerateContent(ctx, p, opt, p.llm.GenerateContent, chain, options...)
+}
+
func (p *bedrockProvider) GetUsage(info map[string]any) pconfig.CallUsage {
return pconfig.NewCallUsage(info)
}
diff --cc backend/pkg/providers/bedrock/bedrock_test.go
index a0421564,42e0c52c..463381d3
--- a/backend/pkg/providers/bedrock/bedrock_test.go
+++ b/backend/pkg/providers/bedrock/bedrock_test.go
@@@ -113,19 -115,83 +115,55 @@@ func TestModelsLoading(t *testing.T)
}
}
+ func TestBedrockProviderConfigPathOverride(t *testing.T) {
+ dir := t.TempDir()
+ path := filepath.Join(dir, "bedrock.provider.yml")
+ external := "simple:\n model: zai.glm-4.7-flash\n temperature: 0.5\n n: 1\n max_tokens: 1000\n price:\n input: 0.15\n output: 0.6\n"
+ if err := os.WriteFile(path, []byte(external), 0o600); err != nil {
+ t.Fatalf("Failed to write external config: %v", err)
+ }
+
+ cfg := &config.Config{
+ BedrockRegion: "us-east-1",
+ BedrockAccessKey: "test-key",
+ BedrockSecretKey: "test-key",
+ BedrockConfig: path,
+ }
+
+ providerConfig, err := DefaultProviderConfig(cfg)
+ if err != nil {
+ t.Fatalf("Failed to load external provider config: %v", err)
+ }
+
+ prov, err := New(cfg, provider.DefaultProviderNameBedrock, providerConfig)
+ if err != nil {
+ t.Fatalf("Failed to create provider: %v", err)
+ }
+
+ if got := prov.Model(pconfig.OptionsTypeSimple); got != "zai.glm-4.7-flash" {
+ t.Errorf("simple agent model = %q, want zai.glm-4.7-flash (external override)", got)
+ }
+ }
+
-func TestBedrockModelsPathMerge(t *testing.T) {
- embedded, err := DefaultModels(&config.Config{})
- if err != nil {
- t.Fatalf("Failed to load embedded models: %v", err)
- }
-
- dir := t.TempDir()
- path := filepath.Join(dir, "bedrock.models.yml")
- external := "- name: zai.glm-4.7-flash\n description: GLM 4.7 Flash on Bedrock\n thinking: false\n price:\n input: 0.15\n output: 0.6\n"
- if err := os.WriteFile(path, []byte(external), 0o600); err != nil {
- t.Fatalf("Failed to write external models: %v", err)
- }
-
- merged, err := DefaultModels(&config.Config{BedrockModels: path})
- if err != nil {
- t.Fatalf("Failed to load merged models: %v", err)
- }
-
- if len(merged) != len(embedded)+1 {
- t.Errorf("merged models = %d, want %d (embedded + 1 external)", len(merged), len(embedded)+1)
- }
-
- found := false
- for _, m := range merged {
- if m.Name == "zai.glm-4.7-flash" {
- found = true
- break
- }
- }
- if !found {
- t.Error("external model zai.glm-4.7-flash not found in merged catalog")
- }
-}
-
func TestBedrockSpecificFeatures(t *testing.T) {
- models, err := DefaultModels(&config.Config{})
+ models, err := DefaultModels()
if err != nil {
t.Fatalf("Failed to load models: %v", err)
}
-- // Test that we have current Bedrock models
++ // Models present in langchaingo/llms/bedrock models_list (and this catalog)
expectedModels := []string{
-- "us.anthropic.claude-sonnet-4-20250514-v1:0",
-- "us.anthropic.claude-3-5-haiku-20241022-v1:0",
-- "us.amazon.nova-premier-v1:0",
++ "us.amazon.nova-2-lite-v1:0",
"us.amazon.nova-pro-v1:0",
-- "us.amazon.nova-lite-v1:0",
++ "us.anthropic.claude-fable-5",
++ "us.anthropic.claude-sonnet-5",
++ "us.anthropic.claude-opus-4-8",
++ "us.anthropic.claude-opus-4-6-v1",
++ "us.meta.llama4-maverick-17b-instruct-v1:0",
++ "deepseek.v3.2",
++ "zai.glm-4.7-flash",
++ "minimax.minimax-m2.5",
++ "nvidia.nemotron-super-3-120b",
}
for _, expectedModel := range expectedModels {
found := false
@@@ -141,9 -207,9 +179,9 @@@
}
// Test default agent model
-- if BedrockAgentModel != bedrock.ModelAnthropicClaudeSonnet4 {
++ if BedrockAgentModel != bedrock.ModelAnthropicClaudeSonnet46 {
t.Errorf("Expected default agent model to be %s, got %s",
-- bedrock.ModelAnthropicClaudeSonnet4, BedrockAgentModel)
++ bedrock.ModelAnthropicClaudeSonnet46, BedrockAgentModel)
}
}
diff --cc backend/pkg/providers/bedrock/models.yml
index 20f3c706,cad8141c..07568d1e
--- a/backend/pkg/providers/bedrock/models.yml
+++ b/backend/pkg/providers/bedrock/models.yml
@@@ -1,4 -1,4 +1,4 @@@
--# Amazon Nova Series - Advanced multimodal models with state-of-the-art performance
++# Amazon Nova Series - Multimodal understanding models
- name: us.amazon.nova-2-lite-v1:0
description: Advanced multimodal model with adaptive reasoning and efficient thinking, intelligently balances performance and efficiency by dynamically adjusting reasoning depth based on task complexity
thinking: false
@@@ -7,14 -7,14 +7,6 @@@
input: 0.33
output: 2.75
--- name: us.amazon.nova-premier-v1:0
-- description: Most capable multimodal model for complex reasoning tasks, frontier intelligence for advanced analysis, and the best teacher for distilling custom models with exceptional problem-solving capabilities
-- thinking: false
-- release_date: 2025-04-30
-- price:
-- input: 2.5
-- output: 12.5
--
- name: us.amazon.nova-pro-v1:0
description: Highly capable multimodal model with optimal balance of accuracy, speed, and cost for wide range of penetration testing tasks and complex security analysis workflows
thinking: false
@@@ -39,7 -39,34 +31,59 @@@
input: 0.035
output: 0.14
- # Anthropic Claude 4.6 Series - Latest generation with world-class coding and agentic capabilities
-# Anthropic Claude 4.7 Series - Latest Opus generation with adaptive thinking
-- name: us.anthropic.claude-opus-4-7
- description: Most capable Opus model for advanced software engineering, long-running agentic tasks, professional work, and rigorous security analysis with adaptive thinking
++# Anthropic Claude Series
++- name: us.anthropic.claude-fable-5
++ description: Anthropic's most capable widely released model for demanding reasoning and long-horizon agentic work; adaptive thinking is always on (raw chain of thought is never returned); requires Bedrock provider data-share retention mode
+ thinking: true
+ reasoning:
+ mode: adaptive-only
+ efforts: [low, medium, high, xhigh, max]
- release_date: 2026-04-16
++ release_date: 2026-06-09
++ price:
++ input: 10.0
++ output: 50.0
++ cache_read: 1.0
++ cache_write: 12.5
++
++- name: us.anthropic.claude-opus-4-8
++ description: Anthropic Opus 4.8 - flagship model for coding, agents, and deep reasoning in enterprise security workflows, with adaptive thinking only
++ thinking: true
++ reasoning:
++ mode: adaptive-only
++ efforts: [low, medium, high, xhigh, max]
++ release_date: 2026-05-28
+ price:
+ input: 5.0
+ output: 25.0
+ cache_read: 0.5
+ cache_write: 6.25
+
-- name: us.anthropic.claude-opus-4-8
- description: Anthropic Opus 4.8 - flagship model for coding, agents, and deep reasoning in enterprise security workflows, with adaptive thinking only
++- name: us.anthropic.claude-opus-4-7
++ description: Most capable Opus model for advanced software engineering, long-running agentic tasks, professional work, and rigorous security analysis with adaptive thinking
+ thinking: true
+ reasoning:
+ mode: adaptive-only
+ efforts: [low, medium, high, xhigh, max]
- release_date: 2026-05-28
++ release_date: 2026-04-16
+ price:
+ input: 5.0
+ output: 25.0
+ cache_read: 0.5
+ cache_write: 6.25
+
-# Anthropic Claude 4.6 Series - Latest generation with world-class coding and agentic capabilities
++- name: us.anthropic.claude-sonnet-5
++ description: Most capable Sonnet for coding, agents, and professional work at scale with near-Opus intelligence at Sonnet cost; on Bedrock adaptive thinking is always on and cannot be disabled
++ thinking: true
++ reasoning:
++ mode: adaptive-only
++ efforts: [low, medium, high, xhigh, max]
++ release_date: 2026-07-01
++ price:
++ input: 3.0
++ output: 15.0
++ cache_read: 0.3
++ cache_write: 3.75
++
- name: us.anthropic.claude-opus-4-6-v1
description: World's best model for coding, enterprise agents, and professional work with industry-leading reliability for agentic workflows and security analysis
thinking: true
@@@ -60,7 -93,7 +110,7 @@@
cache_read: 0.3
cache_write: 3.75
--# Anthropic Claude 4.5 Series - Advanced reasoning models with extended thinking
++# Anthropic Claude 4.5 Series - Extended thinking models
- name: us.anthropic.claude-opus-4-5-20251101-v1:0
description: Next generation most intelligent model delivering multi-day software development projects in hours with frontier intelligence and deep technical capabilities
thinking: true
@@@ -91,45 -124,45 +141,79 @@@
cache_read: 0.3
cache_write: 3.75
--# Anthropic Claude 4 Series - State-of-the-art coding and agentic capabilities
--- name: us.anthropic.claude-sonnet-4-20250514-v1:0
-- description: Balanced performance for coding with optimal speed and cost for high-volume use cases, handles production-ready AI assistants and efficient research
-- thinking: true
-- release_date: 2025-05-22
++# Meta Llama Series - Open models for multilingual dialogue and multimodal understanding
++- name: us.meta.llama4-maverick-17b-instruct-v1:0
++ description: Industry-leading image and text understanding across 12 languages; strong for general assistant, chat, precise image understanding, and creative writing
++ thinking: false
++ release_date: 2025-04-05
price:
-- input: 3.0
-- output: 15.0
-- cache_read: 0.3
-- cache_write: 3.75
++ input: 0.20
++ output: 0.80
--# Anthropic Claude 3.5 Series - Enhanced performance and cost-effectiveness
--- name: us.anthropic.claude-3-5-haiku-20241022-v1:0
-- description: Fastest and most cost-effective model perfect for rapid security scanning, automated vulnerability detection, and high-throughput threat intelligence processing
++- name: us.meta.llama4-scout-17b-instruct-v1:0
++ description: General-purpose MoE model with industry-leading multi-million-token context for multi-document summarization, activity parsing, and reasoning over large codebases
thinking: false
-- release_date: 2024-11-04
++ release_date: 2025-04-05
price:
-- input: 0.8
-- output: 4.0
-- cache_read: 0.08
-- cache_write: 1
++ input: 0.17
++ output: 0.66
--# Cohere Command Series - Advanced retrieval and tool use for enterprise security
--- name: cohere.command-r-plus-v1:0
-- description: Highly performant generative model optimized for large-scale security operations, advanced threat research, and complex penetration testing with superior RAG capabilities
++- name: us.meta.llama3-3-70b-instruct-v1:0
++ description: Near-405B performance at lower cost with tool use, code generation, advanced reasoning, and steerability for multilingual dialogue
thinking: false
-- release_date: 2024-04-29
++ release_date: 2024-12-06
price:
-- input: 3.0
-- output: 15.0
++ input: 0.72
++ output: 0.72
+
-# DeepSeek Series - Advanced reasoning and efficiency models
++- name: us.meta.llama3-1-70b-instruct-v1:0
++ description: Multilingual Llama 3.1 70B with 128K context and improved reasoning for dialogue and complex instruction following
++ thinking: false
++ release_date: 2024-07-23
++ price:
++ input: 0.72
++ output: 0.72
++
++- name: meta.llama3-1-8b-instruct-v1:0
++ description: Compact multilingual Llama 3.1 8B with 128K context for efficient dialogue and on-budget agent workloads
++ thinking: false
++ release_date: 2024-07-23
++ price:
++ input: 0.22
++ output: 0.22
++
++- name: meta.llama3-70b-instruct-v1:0
++ description: Accessible open Llama 3 70B for content creation, conversational AI, language understanding, and enterprise applications
++ thinking: false
++ release_date: 2024-04-18
++ price:
++ input: 2.65
++ output: 3.5
+
- # DeepSeek Series - Advanced reasoning and efficiency models
++- name: meta.llama3-8b-instruct-v1:0
++ description: Compact open Llama 3 8B for limited compute, edge-friendly deployments, and faster iteration
++ thinking: false
++ release_date: 2024-04-18
++ price:
++ input: 0.30
++ output: 0.60
++
++# DeepSeek Series - Reasoning and agent efficiency models
- name: deepseek.v3.2
description: Harmonizes high computational efficiency with superior reasoning and agent performance, excels at long-context reasoning and agentic tasks with sparse attention design
thinking: false
release_date: 2025-12-01
price:
-- input: 0.58
-- output: 1.68
++ input: 0.62
++ output: 1.85
++
++- name: us.deepseek.r1-v1:0
++ description: State-of-the-art reasoning model optimized for general reasoning, math, science, and code generation; text-only English and Chinese
++ thinking: true
++ release_date: 2025-01-20
++ price:
++ input: 1.35
++ output: 5.40
# OpenAI GPT OSS Series - Open-source reasoning models
- name: openai.gpt-oss-120b-1:0
@@@ -148,7 -181,7 +232,7 @@@
input: 0.07
output: 0.3
--# Qwen3 Series - Cutting-edge MoE models with ultra-long context
++# Qwen3 Series - MoE, vision-language, and coding models
- name: qwen.qwen3-next-80b-a3b
description: Cutting-edge MoE and hybrid attention for ultra-long-context workflows, flagship-level reasoning and coding with only 3B active parameters per token
thinking: false
@@@ -157,6 -190,6 +241,14 @@@
input: 0.15
output: 1.2
++- name: qwen.qwen3-vl-235b-a22b
++ description: Frontier vision-language MoE for OCR, layout, multimodal RAG, visual QA, and UI/scene understanding across images, documents, and long videos
++ thinking: false
++ release_date: 2025-09-23
++ price:
++ input: 0.53
++ output: 2.66
++
- name: qwen.qwen3-32b-v1:0
description: Balanced dense model with strong reasoning and general-purpose performance, surpasses many larger models in reasoning, coding, and research use cases
thinking: false
@@@ -178,18 -211,18 +270,42 @@@
thinking: false
release_date: 2026-02-02
price:
-- input: 0.45
-- output: 1.8
++ input: 0.50
++ output: 1.20
--# Mistral Series - Advanced multimodal models with long context
++# Mistral Series - Multimodal, coding, and instruction models
- name: mistral.mistral-large-3-675b-instruct
description: Most advanced open-weight multimodal model with granular MoE architecture, state-of-the-art reliability and long-context reasoning for production assistants
thinking: false
release_date: 2025-12-02
+ price:
+ input: 0.5
+ output: 1.5
+
++- name: mistral.devstral-2-123b
++ description: Agentic software-engineering model for autonomous coding workflows, multi-file edits, and native tool-calling across repositories
++ thinking: false
++ release_date: 2025-12-09
++ price:
++ input: 0.40
++ output: 2.00
++
++- name: mistral.magistral-small-2509
++ description: Small dense multimodal model optimized for fast, cost-efficient instruction following, reasoning, and coding
++ thinking: true
++ release_date: 2025-09-17
++ price:
++ input: 0.50
++ output: 1.50
++
++- name: mistral.mistral-large-2402-v1:0
++ description: Advanced Mistral Large for complex multilingual reasoning, text understanding, transformation, and code generation
++ thinking: false
++ release_date: 2024-02-26
+ price:
+ input: 4.0
+ output: 12.0
+
# Moonshot Kimi Series - Multimodal and thinking agent models
- name: moonshotai.kimi-k2.5
description: Strong vision, language, and code capabilities in single natively multimodal architecture, handles complex tasks mixing images and text with high accuracy
@@@ -198,3 -231,3 +314,70 @@@
price:
input: 0.6
output: 3.0
++
++- name: moonshot.kimi-k2-thinking
++ description: Flagship thinking-agent MoE for deep tool-augmented reasoning, long-horizon planning, complex coding, and research agents over large corpora
++ thinking: true
++ release_date: 2025-11-06
++ price:
++ input: 0.60
++ output: 2.50
++
++# Z.AI GLM Series - Reasoning, coding, and agent models
++- name: zai.glm-4.7
++ description: General-purpose GLM focused on clean front-end code and web UI generation alongside standard text and reasoning tasks
++ thinking: true
++ release_date: 2025-12-22
++ price:
++ input: 0.60
++ output: 2.20
++
++- name: zai.glm-4.7-flash
++ description: Lightweight MoE variant of GLM-4.7 for low-latency, cost-efficient interactive assistants and high-traffic services
++ thinking: true
++ release_date: 2025-12-22
++ price:
++ input: 0.07
++ output: 0.40
++
++- name: zai.glm-5
++ description: Frontier reasoning and agentic model with strong tool use and long-context planning for production agent systems
++ thinking: true
++ release_date: 2026-02-11
++ price:
++ input: 1.00
++ output: 3.20
++
++# MiniMax Series - Agent-native coding and planning models
++- name: minimax.minimax-m2.5
++ description: Agent-native frontier model for efficient reasoning, task decomposition, and complex workflows under real-world time and cost constraints
++ thinking: false
++ release_date: 2026-02-12
++ price:
++ input: 0.30
++ output: 1.20
++
++- name: minimax.minimax-m2.1
++ description: Open-weight model focused on coding, tool use, and long-horizon task planning for agent-based applications
++ thinking: false
++ release_date: 2025-12-23
++ price:
++ input: 0.30
++ output: 1.20
++
++- name: minimax.minimax-m2
++ description: Efficient MoE for AI agents with strong reasoning, coding, and multilingual performance at competitive cost
++ thinking: false
++ release_date: 2025-10-27
++ price:
++ input: 0.30
++ output: 1.20
++
++# NVIDIA Nemotron Series
++- name: nvidia.nemotron-super-3-120b
++ description: Open hybrid MoE (about 12B active) built for reasoning, coding, and agentic tasks with strong cost efficiency
++ thinking: true
++ release_date: 2026-01-15
++ price:
++ input: 0.15
++ output: 0.65
diff --cc backend/pkg/providers/custom/custom.go
index 2a178586,2a178586..327822a9
--- a/backend/pkg/providers/custom/custom.go
+++ b/backend/pkg/providers/custom/custom.go
@@@ -188,6 -188,6 +188,25 @@@ func (p *customProvider) CallWithTools
)
}
++// CallWithExtraOptions: extra is appended last, so it overrides the config.
++func (p *customProvider) CallWithExtraOptions(
++ ctx context.Context,
++ opt pconfig.ProviderOptionsType,
++ chain []llms.MessageContent,
++ tools []llms.Tool,
++ streamCb streaming.Callback,
++ extra ...llms.CallOption,
++) (*llms.ContentResponse, error) {
++ options := []llms.CallOption{llms.WithStreamingFunc(streamCb)}
++ if len(tools) > 0 {
++ options = append(options, llms.WithTools(tools))
++ }
++ options = append(options, p.providerConfig.GetOptionsForType(opt)...)
++ options = append(options, extra...)
++
++ return provider.WrapGenerateContent(ctx, p, opt, p.llm.GenerateContent, chain, options...)
++}
++
func (p *customProvider) GetUsage(info map[string]any) pconfig.CallUsage {
return pconfig.NewCallUsage(info)
}
diff --cc backend/pkg/providers/gemini/gemini.go
index 891fef14,891fef14..eb4a97bf
--- a/backend/pkg/providers/gemini/gemini.go
+++ b/backend/pkg/providers/gemini/gemini.go
@@@ -193,6 -193,6 +193,25 @@@ func (p *geminiProvider) CallWithTools
)
}
++// CallWithExtraOptions: extra is appended last, so it overrides the config.
++func (p *geminiProvider) CallWithExtraOptions(
++ ctx context.Context,
++ opt pconfig.ProviderOptionsType,
++ chain []llms.MessageContent,
++ tools []llms.Tool,
++ streamCb streaming.Callback,
++ extra ...llms.CallOption,
++) (*llms.ContentResponse, error) {
++ options := []llms.CallOption{llms.WithStreamingFunc(streamCb)}
++ if len(tools) > 0 {
++ options = append(options, llms.WithTools(tools))
++ }
++ options = append(options, p.providerConfig.GetOptionsForType(opt)...)
++ options = append(options, extra...)
++
++ return provider.WrapGenerateContent(ctx, p, opt, p.llm.GenerateContent, chain, options...)
++}
++
func (p *geminiProvider) GetUsage(info map[string]any) pconfig.CallUsage {
return pconfig.NewCallUsage(info)
}
diff --cc backend/pkg/providers/glm/config.yml
index 9e4da043,3f8ca8dc..342208e4
--- a/backend/pkg/providers/glm/config.yml
+++ b/backend/pkg/providers/glm/config.yml
@@@ -1,6 -1,6 +1,6 @@@
# Z.AI GLM agent configuration.
#
--# Strategy: glm-5.1 (newest flagship) for critical reasoning, glm-5-turbo (OpenClaw-
++# Strategy: glm-5.2 (newest flagship) for critical reasoning, glm-5-turbo (OpenClaw-
# native, agent-optimized) for orchestration, glm-4.5-air for cheap utility.
#
# Thinking is controlled via extra_body.thinking.type ("enabled" / "disabled"). GLM is
@@@ -10,12 -10,12 +10,17 @@@
# on the standard API endpoint Z.AI defaults to clearing reasoning_content between
# turns, which hurts agent loops with tool calls. Setting clear_thinking=false
# preserves reasoning_content across turns, improving reasoning continuity and cache
- # hit rates. Required together with WithPreserveReasoningContent() in glm.go so that
+ # hit rates. Required together with WithPreserveReasoningContent() in openaicompat.go so that
# langchaingo serializes reasoning_content back to the API.
#
++# glm-5.2 additionally supports the reasoning_effort parameter (high/max, see
++# models.yml). For generator/refiner/adviser it is set to "max" via reasoning.effort
++# below, which AgentConfig.BuildOptions turns into llms.WithReasoning(ReasoningMax, 0)
++# (see pconfig/config.go), on top of the existing thinking.type=enabled wire.
++#
# Note: langchaingo's IsReasoningModel matches glm-4.5/4.6/4.7 prefixes and force-
# overrides temperature to 1.0; harmless for GLM but temp values for those models in
--# YAML are advisory. glm-5/glm-5.1/glm-5-turbo are not matched and pass through.
++# YAML are advisory. glm-5/glm-5.1/glm-5.2/glm-5-turbo are not matched and pass through.
simple:
model: "glm-4.5-air"
@@@ -81,11 -81,11 +86,13 @@@ assistant
cache_read: 0.24
generator:
-- model: "glm-5.1"
++ model: "glm-5.2"
temperature: 1.0
top_p: 0.95
n: 1
max_tokens: 32768
++ reasoning:
++ effort: max
extra_body:
thinking:
type: enabled
@@@ -97,11 -97,11 +104,13 @@@
cache_read: 0.26
refiner:
-- model: "glm-5.1"
++ model: "glm-5.2"
temperature: 1.0
top_p: 0.95
n: 1
max_tokens: 32768
++ reasoning:
++ effort: max
extra_body:
thinking:
type: enabled
@@@ -113,11 -113,11 +122,13 @@@
cache_read: 0.26
adviser:
-- model: "glm-5.1"
++ model: "glm-5.2"
temperature: 1.0
top_p: 0.95
n: 1
max_tokens: 16384
++ reasoning:
++ effort: max
extra_body:
thinking:
type: enabled
@@@ -174,7 -174,7 +185,7 @@@ enricher
cache_read: 0.03
coder:
-- model: "glm-5.1"
++ model: "glm-5.2"
temperature: 1.0
top_p: 0.95
n: 1
@@@ -206,7 -206,7 +217,7 @@@ installer
cache_read: 0.03
pentester:
-- model: "glm-5.1"
++ model: "glm-5.2"
temperature: 1.0
top_p: 0.95
n: 1
diff --cc backend/pkg/providers/kimi/config.yml
index 66cf51e4,66cf51e4..235dfedf
--- a/backend/pkg/providers/kimi/config.yml
+++ b/backend/pkg/providers/kimi/config.yml
@@@ -1,10 -1,10 +1,27 @@@
# Kimi (Moonshot) agent configuration.
#
--# Strategy: kimi-k2.5 for utility/orchestration, kimi-k2.6 for critical reasoning.
++# Strategy: kimi-k2.5 for utility/orchestration, kimi-k2.7-code-highspeed for the
++# primary/assistant loop, kimi-k3 (always-thinking flagship) for critical reasoning
++# (generator/refiner/adviser), kimi-k2.7-code for coder/pentester.
#
--# K2.6/K2.5 API constraints (any deviation -> invalid_request_error):
--# thinking: temperature=1.0, top_p=0.95, n=1, thinking.keep="all" (k2.6 only)
--# non-thinking: temperature=0.6, top_p=0.95, n=1
++# API constraints per model family (any deviation -> invalid_request_error):
++# kimi-k3: always reasons, no `thinking` param at all; reasoning depth is set
++# via the top-level `reasoning_effort` field (low/high/max, default
++# max). Here it is pinned to "max" via reasoning.effort below, which
++# AgentConfig.BuildOptions turns into llms.WithReasoning(ReasoningMax,
++# 0) (see pconfig/config.go), emitted as the legacy top-level
++# reasoning_effort string (ModernReasoningFormat is off for this
++# shared openaicompat base). temperature=1.0, top_p=0.95, n=1,
++# presence/frequency_penalty=0 are fixed by the API. Do not switch
++# effort per call: it invalidates the prefix cache.
++# kimi-k2.7-code / -highspeed: `thinking` may be omitted; if set explicitly, only
++# {"type":"enabled","keep":"all"} is accepted (no `type: disabled`).
++# `reasoning_effort` is not supported. temperature=1.0, top_p=0.95,
++# n=1 fixed; tool_choice="required" not supported (use auto).
++# kimi-k2.6: thinking: temperature=1.0, top_p=0.95, n=1, thinking.keep="all"
++# non-thinking: temperature=0.6, top_p=0.95, n=1
++# kimi-k2.5: thinking: temperature=1.0, top_p=0.95, n=1 (no `keep` support)
++# non-thinking: temperature=0.6, top_p=0.95, n=1
# always: presence_penalty=0, frequency_penalty=0, tool_choice in {auto, none}
#
# WORKAROUND (drop after fix in langchaingo): for kimi-k2.5 non-thinking agents we
@@@ -51,7 -51,7 +68,7 @@@ simple_json
cache_read: 0.10
primary_agent:
-- model: kimi-k2.5
++ model: kimi-k2.7-code-highspeed
temperature: 1.0
top_p: 0.95
n: 1
@@@ -59,14 -59,14 +76,15 @@@
extra_body:
thinking:
type: enabled
++ keep: all
tool_choice: auto
price:
-- input: 0.60
-- output: 3.00
-- cache_read: 0.10
++ input: 1.90
++ output: 8.00
++ cache_read: 0.38
assistant:
-- model: kimi-k2.5
++ model: kimi-k2.7-code-highspeed
temperature: 1.0
top_p: 0.95
n: 1
@@@ -74,59 -74,59 +92,63 @@@
extra_body:
thinking:
type: enabled
++ keep: all
tool_choice: auto
price:
-- input: 0.60
-- output: 3.00
-- cache_read: 0.10
++ input: 1.90
++ output: 8.00
++ cache_read: 0.38
generator:
-- model: kimi-k2.6
++ model: kimi-k3
temperature: 1.0
top_p: 0.95
++ presence_penalty: 0
++ frequency_penalty: 0
n: 1
max_tokens: 32768
++ reasoning:
++ effort: max
extra_body:
-- thinking:
-- type: enabled
-- keep: all
tool_choice: auto
price:
-- input: 0.95
-- output: 4.00
-- cache_read: 0.16
++ input: 3.00
++ output: 15.00
++ cache_read: 0.30
refiner:
-- model: kimi-k2.6
++ model: kimi-k3
temperature: 1.0
top_p: 0.95
++ presence_penalty: 0
++ frequency_penalty: 0
n: 1
max_tokens: 32768
++ reasoning:
++ effort: max
extra_body:
-- thinking:
-- type: enabled
-- keep: all
tool_choice: auto
price:
-- input: 0.95
-- output: 4.00
-- cache_read: 0.16
++ input: 3.00
++ output: 15.00
++ cache_read: 0.30
adviser:
-- model: kimi-k2.6
++ model: kimi-k3
temperature: 1.0
top_p: 0.95
++ presence_penalty: 0
++ frequency_penalty: 0
n: 1
max_tokens: 8192
++ reasoning:
++ effort: max
extra_body:
-- thinking:
-- type: enabled
-- keep: all
tool_choice: auto
price:
-- input: 0.95
-- output: 4.00
-- cache_read: 0.16
++ input: 3.00
++ output: 15.00
++ cache_read: 0.30
reflector:
model: kimi-k2.5
@@@ -177,7 -177,7 +199,7 @@@ enricher
cache_read: 0.10
coder:
-- model: kimi-k2.6
++ model: kimi-k2.7-code
temperature: 1.0
top_p: 0.95
n: 1
@@@ -190,7 -190,7 +212,7 @@@
price:
input: 0.95
output: 4.00
-- cache_read: 0.16
++ cache_read: 0.19
installer:
model: kimi-k2.5
@@@ -208,7 -208,7 +230,7 @@@
cache_read: 0.10
pentester:
-- model: kimi-k2.6
++ model: kimi-k2.7-code
temperature: 1.0
top_p: 0.95
n: 1
@@@ -221,4 -221,4 +243,4 @@@
price:
input: 0.95
output: 4.00
-- cache_read: 0.16
++ cache_read: 0.19
diff --cc backend/pkg/providers/kimi/models.yml
index b18d33e3,80d1a5c7..a867015a
--- a/backend/pkg/providers/kimi/models.yml
+++ b/backend/pkg/providers/kimi/models.yml
@@@ -1,3 -1,12 +1,31 @@@
-# Kimi K2.7 Code - Code-specialized flagship model
++# Kimi K3 - Flagship for long-horizon coding and end-to-end knowledge work
++- name: kimi-k3
++ description: Kimi K3 - Flagship model for long-horizon coding and end-to-end knowledge work with 1M context. Always reasons; reasoning_effort currently supports max only. ToolCalls, JSON Mode, structured output, automatic context caching, tool_choice constraints, and dynamically loaded tools
++ thinking: true
++ reasoning:
++ efforts: [max]
++ price:
++ input: 3.00
++ output: 15.00
++ cache_read: 0.30
++
++# Kimi K2.7 Code - Coding-focused model (and high-speed variant)
+ - name: kimi-k2.7-code
- description: Kimi K2.7 Code - Code-specialist variant of K2.7 with native architecture. Stronger code generation and long-term reasoning than K2.6; supports thinking/non-thinking modes, 256k context, automatic context caching, ToolCalls, JSON Mode
++ description: Kimi K2.7 Code - Coding-focused model with higher success rates on long-context programming tasks. Supports text/image/video input, thinking mode, dialogue and agent tasks, 256k context, automatic context caching, ToolCalls, JSON Mode, Partial Mode
+ thinking: true
+ price:
+ input: 0.95
+ output: 4.00
+ cache_read: 0.19
+
++- name: kimi-k2.7-code-highspeed
++ description: Kimi K2.7 Code HighSpeed - Same model as kimi-k2.7-code with higher output throughput (~180 tokens/s, up to ~260 tokens/s in short context). Supports text/image/video input, thinking mode, 256k context, automatic context caching, ToolCalls, JSON Mode, Partial Mode
++ thinking: true
++ price:
++ input: 1.90
++ output: 8.00
++ cache_read: 0.38
++
# Kimi K2.6 - Latest flagship multimodal model with native architecture
- name: kimi-k2.6
description: Kimi K2.6 - Latest and most intelligent multimodal model with native architecture, stronger long-term code writing, improved instruction compliance and self-correction. Supports text/image/video input, thinking/non-thinking modes, 256k context, automatic context caching, ToolCalls, JSON Mode, internet search
diff --cc backend/pkg/providers/minimax/config.yml
index c9ddde53,0bdcb964..c7058291
--- a/backend/pkg/providers/minimax/config.yml
+++ b/backend/pkg/providers/minimax/config.yml
@@@ -1,17 -1,18 +1,20 @@@
simple:
-- model: MiniMax-M3
-- temperature: 0.5
-- top_p: 0.5
++ model: MiniMax-M2.7
++ temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
max_tokens: 8192
price:
- input: 0.60
- output: 2.40
+ input: 0.30
+ output: 1.20
+ cache_read: 0.06
simple_json:
-- model: MiniMax-M3
-- temperature: 0.5
-- top_p: 0.5
++ model: MiniMax-M2.7
++ temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
max_tokens: 4096
json: true
@@@ -21,8 -23,8 +25,9 @@@
primary_agent:
model: MiniMax-M3
-- temperature: 0.7
-- top_p: 0.8
++ temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
max_tokens: 16384
price:
@@@ -31,8 -34,8 +37,9 @@@
assistant:
model: MiniMax-M3
-- temperature: 0.7
-- top_p: 0.8
++ temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
max_tokens: 16384
price:
@@@ -41,8 -45,8 +49,9 @@@
generator:
model: MiniMax-M3
-- temperature: 0.7
-- top_p: 0.8
++ temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
max_tokens: 32768
price:
@@@ -51,8 -56,8 +61,9 @@@
refiner:
model: MiniMax-M3
-- temperature: 0.7
-- top_p: 0.8
++ temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
max_tokens: 20480
price:
@@@ -61,8 -67,8 +73,9 @@@
adviser:
model: MiniMax-M3
-- temperature: 0.7
-- top_p: 0.8
++ temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
max_tokens: 8192
price:
@@@ -71,8 -78,8 +85,9 @@@
reflector:
model: MiniMax-M3
-- temperature: 0.5
-- top_p: 0.5
++ temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
max_tokens: 4096
price:
@@@ -81,8 -89,8 +97,9 @@@
searcher:
model: MiniMax-M3
-- temperature: 0.7
-- top_p: 0.8
++ temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
max_tokens: 4096
price:
@@@ -91,8 -100,8 +109,9 @@@
enricher:
model: MiniMax-M3
-- temperature: 0.7
-- top_p: 0.8
++ temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
max_tokens: 4096
price:
@@@ -101,8 -111,8 +121,9 @@@
coder:
model: MiniMax-M3
-- temperature: 0.5
-- top_p: 0.5
++ temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
max_tokens: 20480
price:
@@@ -111,8 -122,8 +133,9 @@@
installer:
model: MiniMax-M3
-- temperature: 0.5
-- top_p: 0.5
++ temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
max_tokens: 16384
price:
@@@ -121,8 -133,8 +145,9 @@@
pentester:
model: MiniMax-M3
-- temperature: 0.5
-- top_p: 0.5
++ temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
max_tokens: 16384
price:
diff --cc backend/pkg/providers/minimax/models.yml
index 5ede908e,9be49760..6591f8c3
--- a/backend/pkg/providers/minimax/models.yml
+++ b/backend/pkg/providers/minimax/models.yml
@@@ -9,12 -10,14 +10,16 @@@
description: MiniMax-M2.7 - Previous flagship model with enhanced reasoning and coding capabilities. Supports tool calling, JSON output, and streaming.
thinking: false
price:
- input: 0.40
- output: 1.10
+ input: 0.30
+ output: 1.20
+ cache_read: 0.06
++ cache_write: 0.375
- name: MiniMax-M2.7-highspeed
-- description: MiniMax-M2.7-highspeed - High-speed version of M2.7 for low-latency scenarios. Same capabilities as M2.7 with improved response latency.
++ description: MiniMax-M2.7-highspeed - High-speed version of M2.7 for low-latency scenarios. Same capabilities as M2.7; priced higher for improved response latency.
thinking: false
price:
- input: 0.40
- output: 1.10
+ input: 0.60
+ output: 2.40
+ cache_read: 0.06
++ cache_write: 0.375
diff --cc backend/pkg/providers/ollama/ollama.go
index fb7c9bc2,fb7c9bc2..a267d82c
--- a/backend/pkg/providers/ollama/ollama.go
+++ b/backend/pkg/providers/ollama/ollama.go
@@@ -300,6 -300,6 +300,25 @@@ func (p *ollamaProvider) CallWithTools
)
}
++// CallWithExtraOptions: extra is appended last, so it overrides the config.
++func (p *ollamaProvider) CallWithExtraOptions(
++ ctx context.Context,
++ opt pconfig.ProviderOptionsType,
++ chain []llms.MessageContent,
++ tools []llms.Tool,
++ streamCb streaming.Callback,
++ extra ...llms.CallOption,
++) (*llms.ContentResponse, error) {
++ options := []llms.CallOption{llms.WithStreamingFunc(streamCb)}
++ if len(tools) > 0 {
++ options = append(options, llms.WithTools(tools))
++ }
++ options = append(options, p.providerConfig.GetOptionsForType(opt)...)
++ options = append(options, extra...)
++
++ return provider.WrapGenerateContent(ctx, p, opt, p.llm.GenerateContent, chain, options...)
++}
++
func (p *ollamaProvider) GetUsage(info map[string]any) pconfig.CallUsage {
return pconfig.NewCallUsage(info)
}
diff --cc backend/pkg/providers/openai/config.yml
index 60254562,60254562..088ab8fe
--- a/backend/pkg/providers/openai/config.yml
+++ b/backend/pkg/providers/openai/config.yml
@@@ -1,3 -1,3 +1,60 @@@
++# OpenAI agent configuration.
++#
++# Strategy: gpt-5.4-nano for utility/orchestration (simple/searcher/enricher),
++# gpt-5.4-mini for the primary/assistant loop and lightweight security agents,
++# gpt-5.6-terra (newest mid-tier flagship, same price as gpt-5.4) for critical
++# reasoning (generator/refiner/adviser/coder).
++#
++# All models below are current (non-deprecated) per OpenAI's model catalog. Do not
++# reintroduce o3/o4-mini/gpt-4.1/gpt-4o-class models here: see models.yml for the
++# full list of models OpenAI has marked deprecated (kept there only for backward
++# compatibility with pre-existing agent configs pinned to those names).
++#
++# Every gpt-5.4-mini / gpt-5.6-terra agent below sets `reasoning: {mode: off}`.
++# PentAGI calls OpenAI through /v1/chat/completions (never /v1/responses), and
++# for these two models that endpoint rejects ANY request that both attaches
++# function tools and leaves reasoning at its (default-on) level — even when no
++# `reasoning_effort` field is sent at all:
++# API returned unexpected status code: 400: Function tools with
++# reasoning_effort are not supported for <model> in /v1/chat/completions. To
++# use function tools, use /v1/responses or set reasoning_effort to 'none'.
++#
++# This was verified live against the real OpenAI API with `ctester` (see
++# `.vscode/launch.json` "Launch Agents Tests", `-type openai -agents <name>`):
++# - Omitting `reasoning:` entirely (the previous state of this file) is NOT
++# enough: it still 400s on every test of `type: tool` for gpt-5.6-terra
++# (verified on the `coder` agent, 15/23 passing — every failure was a
++# tool-attached call, every pass was a plain completion). These models
++# default to thinking ON even with no reasoning CallOption set, and the
++# API refuses to combine that default-on thinking with tools on this
++# endpoint.
++# - `reasoning: {effort: none}` is ALSO not a fix: `pconfig.AgentConfig.
++# BuildOptions` (pkg/providers/pconfig/config.go) only forwards the typed
++# constants llms.ReasoningLow/Medium/High/XHigh/Max to llms.WithReasoning;
++# the plain string "none" matches none of them, so it silently emits no
++# CallOption — byte-for-byte the same (and byte-for-byte the same failure)
++# as omitting `reasoning:` altogether.
++# - `reasoning: {mode: off}` is the only config value that actually forces
++# the wire to send "none": it calls llms.WithReasoningDisabled() ->
++# reasoning.ResolveOff() (langchaingo v0.1.14-update.6, llms/reasoning/off.go).
++# gpt-5.4-mini/gpt-5.6-terra are reasoning models (IsReasoningModel's
++# "gpt-5" prefix match) that are NOT in the hard-mandatory o1/o3/o4-mini/
++# gpt-5-pro set, so ResolveOff returns OffEffortNone and openaillm.go's
++# setReasoningOff sends a real "reasoning_effort":"none". Re-tested the
++# `coder` agent with this set: 24/24 (100%) passing, including every
++# `type: tool` and streaming test.
++# - gpt-5.4-nano (simple/simple_json/searcher/enricher) needs no such
++# override: re-tested live and it is 100% passing (69/69) as-is — the
++# tester's "Reasoning" column reports `false` for it, i.e. this tier does
++# not default to thinking ON, so tools attach without conflict.
++#
++# Net effect: every gpt-5.4-mini/gpt-5.6-terra agent below runs with thinking
++# fully OFF (not "low effort" — genuinely disabled), because that model
++# family cannot combine any nonzero reasoning with function tools on
++# /v1/chat/completions. Do not remove `mode: off` or replace it with an
++# `effort:` value on these agents without re-verifying against a live account
++# that every `type: tool` test still passes.
++
simple:
model: gpt-5.4-nano
temperature: 0.5
@@@ -22,122 -22,122 +79,122 @@@ simple_json
cache_read: 0.02
primary_agent:
-- model: o4-mini
++ model: gpt-5.4-mini
n: 1
max_tokens: 16384
reasoning:
-- effort: medium
++ mode: off
price:
-- input: 1.1
-- output: 4.4
-- cache_read: 0.275
++ input: 0.75
++ output: 4.5
++ cache_read: 0.075
assistant:
-- model: o4-mini
++ model: gpt-5.4-mini
n: 1
max_tokens: 16384
reasoning:
-- effort: medium
++ mode: off
price:
-- input: 1.1
-- output: 4.4
-- cache_read: 0.275
++ input: 0.75
++ output: 4.5
++ cache_read: 0.075
generator:
-- model: o3
++ model: gpt-5.6-terra
n: 1
max_tokens: 32768
reasoning:
-- effort: medium
++ mode: off
price:
-- input: 2.0
-- output: 8.0
-- cache_read: 0.5
++ input: 2.5
++ output: 15.0
++ cache_read: 0.25
refiner:
-- model: o3
++ model: gpt-5.6-terra
n: 1
max_tokens: 20480
reasoning:
-- effort: high
++ mode: off
price:
-- input: 2.0
-- output: 8.0
-- cache_read: 0.5
++ input: 2.5
++ output: 15.0
++ cache_read: 0.25
adviser:
-- model: gpt-5.4
++ model: gpt-5.6-terra
n: 1
max_tokens: 8192
reasoning:
-- effort: low
++ mode: off
price:
input: 2.5
output: 15.0
cache_read: 0.25
reflector:
-- model: o4-mini
++ model: gpt-5.4-mini
n: 1
max_tokens: 4096
reasoning:
-- effort: medium
++ mode: off
price:
-- input: 1.1
-- output: 4.4
-- cache_read: 0.275
++ input: 0.75
++ output: 4.5
++ cache_read: 0.075
searcher:
-- model: gpt-4.1-mini
++ model: gpt-5.4-nano
temperature: 0.7
top_p: 0.8
n: 1
max_tokens: 8192
price:
-- input: 0.4
-- output: 1.6
-- cache_read: 0.1
++ input: 0.2
++ output: 1.25
++ cache_read: 0.02
enricher:
-- model: gpt-4.1-mini
++ model: gpt-5.4-nano
temperature: 0.7
top_p: 0.8
n: 1
max_tokens: 4096
price:
-- input: 0.4
-- output: 1.6
-- cache_read: 0.1
++ input: 0.2
++ output: 1.25
++ cache_read: 0.02
coder:
-- model: o3
++ model: gpt-5.6-terra
n: 1
max_tokens: 20480
reasoning:
-- effort: low
++ mode: off
price:
-- input: 2.0
-- output: 8.0
-- cache_read: 0.5
++ input: 2.5
++ output: 15.0
++ cache_read: 0.25
installer:
-- model: o4-mini
++ model: gpt-5.4-mini
n: 1
max_tokens: 16384
reasoning:
-- effort: low
++ mode: off
price:
-- input: 1.1
-- output: 4.4
-- cache_read: 0.275
++ input: 0.75
++ output: 4.5
++ cache_read: 0.075
pentester:
-- model: o4-mini
++ model: gpt-5.4-mini
n: 1
max_tokens: 8192
reasoning:
-- effort: low
++ mode: off
price:
-- input: 1.1
-- output: 4.4
-- cache_read: 0.275
++ input: 0.75
++ output: 4.5
++ cache_read: 0.075
diff --cc backend/pkg/providers/openai/models.yml
index d040cae3,91539dc2..86a3bc0c
--- a/backend/pkg/providers/openai/models.yml
+++ b/backend/pkg/providers/openai/models.yml
@@@ -1,3 -1,3 +1,59 @@@
++# GPT-5.6 series - Latest frontier models (Feb 16, 2026 knowledge cutoff, 1M context)
++- name: gpt-5.6-sol
++ description: Frontier model for complex professional work. Roughly corresponds to the unsuffixed model tier used in earlier GPT-5 families (successor to gpt-5.4). 1.05M context window, 128K max output tokens, configurable reasoning effort. Best for the most demanding autonomous penetration testing, sophisticated exploit chain development, and deep multi-stage attack simulation.
++ thinking: true
++ reasoning:
++ efforts: [low, medium, high, xhigh]
++ release_date: 2026-05-15
++ price:
++ input: 5.0
++ output: 30.0
++ cache_read: 0.5
++
++- name: gpt-5.6-terra
++ description: GPT-5.6 model balancing intelligence and cost. Roughly corresponds to the mini model tier used in earlier GPT-5 families. 1.05M context window, 128K max output tokens, configurable reasoning effort. Strong fit for multi-phase security assessments and coordinated multi-tool penetration testing at a lower cost than gpt-5.6-sol.
++ thinking: true
++ reasoning:
++ efforts: [low, medium, high, xhigh]
++ release_date: 2026-05-15
++ price:
++ input: 2.5
++ output: 15.0
++ cache_read: 0.25
++
++- name: gpt-5.6-luna
++ description: GPT-5.6 model optimized for cost-sensitive, high-volume workloads. Roughly corresponds to the nano model tier used in earlier GPT-5 families. 1.05M context window, 128K max output tokens, configurable reasoning effort. Ideal for rapid reconnaissance, bulk vulnerability scanning, and real-time security monitoring.
++ thinking: true
++ reasoning:
++ efforts: [low, medium, high, xhigh]
++ release_date: 2026-05-15
++ price:
++ input: 1.0
++ output: 6.0
++ cache_read: 0.1
++
++# GPT-5.5 series - Frontier models for complex professional work (Dec 01, 2025 knowledge cutoff, 1M context)
++- name: gpt-5.5
++ description: New class of intelligence for coding and professional work. 1.05M context window, 128K max output tokens, reasoning effort supports none/low/medium(default)/high/xhigh. Excels at complex professional work, sophisticated security research, and advanced autonomous penetration testing.
++ thinking: true
++ reasoning:
++ efforts: [none, low, medium, high, xhigh]
++ release_date: 2026-02-10
++ price:
++ input: 5.0
++ output: 30.0
++ cache_read: 0.5
++
++- name: gpt-5.5-pro
++ description: Version of GPT-5.5 that uses more compute to think harder and produce smarter, more precise responses. 1.05M context window, 128K max output tokens, reasoning effort supports medium/high(default)/xhigh. No cached-input discount. Designed for mission-critical security research, advanced zero-day discovery, and complex autonomous penetration testing requiring maximum reasoning depth and accuracy.
++ thinking: true
++ reasoning:
++ efforts: [medium, high, xhigh]
++ release_date: 2026-02-10
++ price:
++ input: 30.0
++ output: 180.0
++
# Latest GPT-5.4 series - Frontier models with advanced reasoning and professional workflows
- name: gpt-5.4
description: Best intelligence at scale for agentic, coding, and professional workflows. Flagship model with 1M context window and configurable reasoning effort (none/low/medium/high/xhigh). Excels at complex professional work, sophisticated security research, and advanced autonomous penetration testing requiring maximum cognitive depth and accuracy.
@@@ -28,8 -34,10 +90,10 @@@
# Latest GPT-5.2 series - Enhanced agentic models with improved reasoning and tool integration
- name: gpt-5.2
-- description: Latest flagship agentic model with enhanced reasoning and tool integration. Excels at autonomous security research, complex exploit chain development, and coordinating multi-tool penetration testing workflows. Optimal for sophisticated threat modeling and adaptive attack strategies.
++ description: Previous frontier model for professional work with configurable reasoning effort. Excels at autonomous security research, complex exploit chain development, and coordinating multi-tool penetration testing workflows. Optimal for sophisticated threat modeling and adaptive attack strategies.
thinking: true
+ reasoning:
+ efforts: [low, medium, high, xhigh]
release_date: 2025-12-11
price:
input: 1.75
@@@ -37,8 -45,10 +101,10 @@@
cache_read: 0.175
- name: gpt-5.2-pro
-- description: Premium version of GPT-5.2 with superior agentic coding capabilities and long-context performance. Designed for mission-critical security research, advanced zero-day discovery, and complex autonomous penetration testing requiring maximum reasoning depth, accuracy, and reduced hallucinations in high-stakes scenarios.
++ description: Previous pro model for professional work that produces smarter and more precise responses. Superior agentic coding capabilities and long-context performance. Designed for mission-critical security research, advanced zero-day discovery, and complex autonomous penetration testing requiring maximum reasoning depth, accuracy, and reduced hallucinations in high-stakes scenarios.
thinking: true
+ reasoning:
+ efforts: [medium, high, xhigh]
release_date: 2025-12-11
price:
input: 21.0
@@@ -46,7 -56,7 +112,7 @@@
# Latest GPT-5 series - Advanced agentic models with native function calling and reasoning
- name: gpt-5
-- description: Premier agentic model with advanced reasoning and native tool integration. Excels at autonomous security research, complex exploit chain development, and coordinating multi-tool penetration testing workflows. Optimal for sophisticated threat modeling and adaptive attack strategies.
++ description: Previous intelligent reasoning model for coding and agentic tasks with configurable reasoning effort. Excels at autonomous security research, complex exploit chain development, and coordinating multi-tool penetration testing workflows. Optimal for sophisticated threat modeling and adaptive attack strategies.
thinking: true
release_date: 2025-08-07
price:
@@@ -55,7 -65,7 +121,7 @@@
cache_read: 0.125
- name: gpt-5.1
-- description: Enhanced agentic model with adaptive reasoning and improved conversational capabilities. Bridges the gap between GPT-5 and GPT-5.2 with faster responses, better personality presets, and refined security analysis. Excellent for balanced penetration testing requiring strong tool coordination with enhanced contextual understanding.
++ description: The best model for coding and agentic tasks with configurable reasoning effort. Bridges the gap between GPT-5 and GPT-5.2 with faster responses, better personality presets, and refined security analysis. Excellent for balanced penetration testing requiring strong tool coordination with enhanced contextual understanding.
thinking: true
release_date: 2025-11-12
price:
@@@ -64,15 -74,17 +130,17 @@@
cache_read: 0.125
- name: gpt-5-pro
-- description: Premium version of GPT-5 with major improvements in reasoning depth and code quality. Optimized for complex security tasks requiring step-by-step reasoning, reduced hallucinations, and exceptional accuracy in high-stakes penetration testing. Superior instruction following and advanced prompt understanding for critical security operations.
++ description: Version of GPT-5 that produces smarter and more precise responses. Optimized for complex security tasks requiring step-by-step reasoning, reduced hallucinations, and exceptional accuracy in high-stakes penetration testing. Superior instruction following and advanced prompt understanding for critical security operations.
thinking: true
+ reasoning:
+ efforts: [high]
release_date: 2025-08-07
price:
input: 15.0
output: 120.0
- name: gpt-5-mini
-- description: Efficient agentic model balancing speed and intelligence for well-scoped security tasks. Ideal for automated vulnerability analysis, exploit generation, and systematic penetration testing with strong function calling capabilities for security tool orchestration.
++ description: Near-frontier intelligence for cost sensitive, low latency, high volume workloads. Ideal for automated vulnerability analysis, exploit generation, and systematic penetration testing with strong function calling capabilities for security tool orchestration.
thinking: true
release_date: 2025-08-07
price:
@@@ -81,7 -93,7 +149,7 @@@
cache_read: 0.025
- name: gpt-5-nano
-- description: Fastest agentic model optimized for high-throughput security scanning and rapid tool execution. Perfect for reconnaissance phases, bulk vulnerability detection, and real-time security monitoring with minimal latency in autonomous agent workflows.
++ description: Fastest, most cost-efficient version of GPT-5, optimized for high-throughput security scanning and rapid tool execution. Perfect for reconnaissance phases, bulk vulnerability detection, and real-time security monitoring with minimal latency in autonomous agent workflows.
thinking: true
release_date: 2025-08-07
price:
@@@ -90,8 -102,8 +158,10 @@@
cache_read: 0.005
# Flagship chat models - Versatile, high-intelligence models for complex tasks
++# DEPRECATED by OpenAI (superseded by GPT-5.x/5.6 series). Kept only for backward
++# compatibility with existing agent configs pinned to this model; avoid for new assignments.
- name: gpt-4o
-- description: Multimodal flagship model with vision capabilities and robust function calling. Excellent for comprehensive penetration testing requiring image analysis, web UI assessment, and complex multi-tool orchestration. Strong balance of speed and intelligence for real-time security operations.
++ description: "[DEPRECATED by OpenAI] Multimodal flagship model with vision capabilities and robust function calling. Excellent for comprehensive penetration testing requiring image analysis, web UI assessment, and complex multi-tool orchestration. Strong balance of speed and intelligence for real-time security operations."
thinking: false
release_date: 2024-05-13
price:
@@@ -100,7 -112,7 +170,7 @@@
cache_read: 1.25
- name: gpt-4o-mini
-- description: Compact multimodal model with strong function calling and fast inference. Optimal for high-frequency security scanning, automated vulnerability checks, and routine penetration testing tasks. Cost-effective choice for bulk operations and continuous security monitoring.
++ description: Fast, affordable small multimodal model with strong function calling and fast inference. Optimal for high-frequency security scanning, automated vulnerability checks, and routine penetration testing tasks. Cost-effective choice for bulk operations and continuous security monitoring.
thinking: false
release_date: 2024-07-18
price:
@@@ -110,7 -122,7 +180,7 @@@
# Latest GPT-4.1 series - Enhanced intelligence models with improved function calling
- name: gpt-4.1
-- description: Enhanced flagship model with superior function calling accuracy and deeper security domain knowledge. Excels at complex threat analysis, sophisticated exploit development, and comprehensive penetration testing requiring extensive tool coordination and adaptive attack planning.
++ description: Smartest non-reasoning model, with superior function calling accuracy and deeper security domain knowledge. Excels at complex threat analysis, sophisticated exploit development, and comprehensive penetration testing requiring extensive tool coordination and adaptive attack planning.
thinking: false
release_date: 2025-04-14
price:
@@@ -119,7 -131,7 +189,7 @@@
cache_read: 0.5
- name: gpt-4.1-mini
-- description: Balanced performance model with improved efficiency and strong function calling. Excellent for routine security assessments, automated code analysis, and systematic vulnerability testing with optimal cost-to-intelligence ratio for production workloads.
++ description: Smaller, faster version of GPT-4.1 with improved efficiency and strong function calling. Excellent for routine security assessments, automated code analysis, and systematic vulnerability testing with optimal cost-to-intelligence ratio for production workloads.
thinking: false
release_date: 2025-04-14
price:
@@@ -127,8 -139,8 +197,10 @@@
output: 1.6
cache_read: 0.1
++# DEPRECATED by OpenAI. Kept only for backward compatibility with existing agent configs
++# pinned to this model; avoid for new assignments.
- name: gpt-4.1-nano
-- description: Ultra-fast lightweight model optimized for high-throughput operations. Perfect for bulk security scanning, rapid reconnaissance, continuous monitoring, and basic vulnerability detection where speed and cost efficiency are critical.
++ description: "[DEPRECATED by OpenAI] Fastest, most cost-efficient version of GPT-4.1. Optimized for high-throughput operations, bulk security scanning, rapid reconnaissance, continuous monitoring, and basic vulnerability detection where speed and cost efficiency are critical."
thinking: false
release_date: 2025-04-14
price:
@@@ -137,9 -149,11 +209,13 @@@
cache_read: 0.025
# Codex series - Specialized models optimized for code analysis, exploit development, and cybersecurity
++# All models below are DEPRECATED by OpenAI. Kept only for backward compatibility with
++# existing agent configs pinned to these models; avoid for new assignments.
- name: gpt-5.2-codex
-- description: Most advanced code-specialized model optimized for agentic security coding. Features context compaction for long-horizon work, superior performance on large code refactors and migrations, enhanced Windows environment support, and significantly stronger cybersecurity capabilities. Ideal for vulnerability discovery, exploit chain development, and complex code analysis in large repositories.
++ description: "[DEPRECATED by OpenAI] Most advanced code-specialized model optimized for agentic security coding. Features context compaction for long-horizon work, superior performance on large code refactors and migrations, enhanced Windows environment support, and significantly stronger cybersecurity capabilities. Ideal for vulnerability discovery, exploit chain development, and complex code analysis in large repositories."
thinking: true
+ reasoning:
+ efforts: [low, medium, high, xhigh]
release_date: 2025-12-11
price:
input: 1.75
@@@ -147,7 -161,7 +223,7 @@@
cache_read: 0.175
- name: gpt-5.1-codex-max
-- description: Enhanced reasoning model for sophisticated coding workflows with superior long-horizon task performance. Proven track record in real-world vulnerability discovery (CVE findings). Excels at systematic exploit development, complex code analysis, and agentic penetration testing requiring extended reasoning chains and deep code comprehension.
++ description: "[DEPRECATED by OpenAI] Enhanced reasoning model for sophisticated coding workflows with superior long-horizon task performance. Proven track record in real-world vulnerability discovery (CVE findings). Excels at systematic exploit development, complex code analysis, and agentic penetration testing requiring extended reasoning chains and deep code comprehension."
thinking: true
release_date: 2025-11-01
price:
@@@ -156,7 -170,7 +232,7 @@@
cache_read: 0.125
- name: gpt-5.1-codex
-- description: Standard code-optimized model with strong reasoning capabilities for security engineering. Balanced performance for exploit generation, vulnerability analysis, and automated security code review. Excellent choice for systematic penetration testing workflows requiring reliable code understanding and tool orchestration.
++ description: "[DEPRECATED by OpenAI] Standard code-optimized model with strong reasoning capabilities for security engineering. Balanced performance for exploit generation, vulnerability analysis, and automated security code review. Excellent choice for systematic penetration testing workflows requiring reliable code understanding and tool orchestration."
thinking: true
release_date: 2025-11-01
price:
@@@ -165,7 -179,7 +241,7 @@@
cache_read: 0.125
- name: gpt-5-codex
-- description: Foundational code-specialized model for security-focused development tasks. Strong at vulnerability scanning, basic exploit generation, and security code analysis. Cost-effective option for routine penetration testing workflows requiring solid code comprehension and tool integration.
++ description: "[DEPRECATED by OpenAI] Foundational code-specialized model for security-focused development tasks. Strong at vulnerability scanning, basic exploit generation, and security code analysis. Cost-effective option for routine penetration testing workflows requiring solid code comprehension and tool integration."
thinking: true
release_date: 2025-08-07
price:
@@@ -174,7 -188,7 +250,7 @@@
cache_read: 0.125
- name: gpt-5.1-codex-mini
-- description: Compact high-performance code model with 4x higher usage capacity compared to full Codex variants. Optimized for high-frequency security code analysis, rapid vulnerability detection, and bulk exploit scanning where speed and cost efficiency are paramount while maintaining strong coding capabilities.
++ description: "[DEPRECATED by OpenAI] Compact high-performance code model with 4x higher usage capacity compared to full Codex variants. Optimized for high-frequency security code analysis, rapid vulnerability detection, and bulk exploit scanning where speed and cost efficiency are paramount while maintaining strong coding capabilities."
thinking: true
release_date: 2025-11-01
price:
@@@ -183,7 -197,7 +259,7 @@@
cache_read: 0.025
- name: codex-mini-latest
-- description: Latest compact code model offering balanced performance for routine security coding tasks. Ideal for automated code review, basic vulnerability analysis, and continuous security monitoring requiring efficient code understanding at minimal cost.
++ description: "[DEPRECATED by OpenAI] Latest compact code model offering balanced performance for routine security coding tasks. Ideal for automated code review, basic vulnerability analysis, and continuous security monitoring requiring efficient code understanding at minimal cost."
thinking: true
release_date: 2025-01-01
price:
@@@ -192,8 -206,8 +268,10 @@@
cache_read: 0.375
# Reasoning models - o-series with extended chain-of-thought for complex security tasks
++# o3-mini, o4-mini, o1, and o1-pro are DEPRECATED by OpenAI. Kept only for backward
++# compatibility with existing agent configs pinned to these models; avoid for new assignments.
- name: o3-mini
-- description: Compact reasoning model with extended thinking capabilities for methodical security analysis. Excellent at step-by-step attack planning, logical vulnerability chaining, and systematic penetration testing. Strong deliberative reasoning for complex security scenarios at an efficient cost point.
++ description: "[DEPRECATED by OpenAI] Compact reasoning model with extended thinking capabilities for methodical security analysis. Excellent at step-by-step attack planning, logical vulnerability chaining, and systematic penetration testing. Strong deliberative reasoning for complex security scenarios at an efficient cost point."
thinking: true
release_date: 2025-01-31
price:
@@@ -202,7 -216,7 +280,7 @@@
cache_read: 0.55
- name: o4-mini
-- description: Next-generation reasoning model with enhanced speed and accuracy. Ideal for methodical security assessments, systematic exploit development, and structured vulnerability analysis. Balances deep reasoning with faster inference for production penetration testing workflows.
++ description: "[DEPRECATED by OpenAI, succeeded by GPT-5 mini] Next-generation reasoning model with enhanced speed and accuracy. Ideal for methodical security assessments, systematic exploit development, and structured vulnerability analysis. Balances deep reasoning with faster inference for production penetration testing workflows."
thinking: true
release_date: 2025-04-16
price:
@@@ -211,7 -225,7 +289,7 @@@
cache_read: 0.275
- name: o3
-- description: Advanced reasoning powerhouse for sophisticated security research and complex threat modeling. Excels at multi-stage attack chain development, deep vulnerability analysis, and intricate exploit construction requiring extensive deliberative thinking and strategic planning.
++ description: Reasoning model for complex tasks, succeeded by GPT-5. Excels at multi-stage attack chain development, deep vulnerability analysis, and intricate exploit construction requiring extensive deliberative thinking and strategic planning.
thinking: true
release_date: 2025-04-16
price:
@@@ -220,7 -234,7 +298,7 @@@
cache_read: 0.5
- name: o1
-- description: Premier reasoning model with maximum thinking depth for highly complex security challenges. Specialized in advanced penetration testing methodologies, novel exploit research, and sophisticated attack vector discovery. Best for critical security research requiring exhaustive analysis.
++ description: "[DEPRECATED by OpenAI] Premier reasoning model with maximum thinking depth for highly complex security challenges. Specialized in advanced penetration testing methodologies, novel exploit research, and sophisticated attack vector discovery. Best for critical security research requiring exhaustive analysis."
thinking: true
release_date: 2024-12-17
price:
@@@ -229,7 -243,7 +307,7 @@@
cache_read: 7.5
- name: o3-pro
-- description: Most advanced reasoning powerhouse with deep slow-thinking capabilities. Delivers exceptional performance in complex mathematical analysis, scientific security research, and intricate coding challenges. Massive 80% price reduction from previous o1-pro while maintaining superior reasoning depth. Ideal for novel zero-day research, sophisticated attack chain analysis, and critical security investigations requiring maximum cognitive depth.
++ description: Version of o3 with more compute for better responses. Delivers exceptional performance in complex mathematical analysis, scientific security research, and intricate coding challenges. Ideal for novel zero-day research, sophisticated attack chain analysis, and critical security investigations requiring maximum cognitive depth.
thinking: true
release_date: 2025-06-10
price:
@@@ -237,7 -251,7 +315,7 @@@
output: 80.0
- name: o1-pro
-- description: Previous-generation premium reasoning model with maximum deliberation capabilities. Specialized in exhaustive security analysis, advanced cryptographic research, and complex threat modeling. Highest cost point but unmatched reasoning depth for mission-critical security challenges where budget is not a constraint and absolute thoroughness is required.
++ description: "[DEPRECATED by OpenAI] Previous-generation premium reasoning model with maximum deliberation capabilities. Specialized in exhaustive security analysis, advanced cryptographic research, and complex threat modeling. Highest cost point but unmatched reasoning depth for mission-critical security challenges where budget is not a constraint and absolute thoroughness is required."
thinking: true
release_date: 2024-12-17
price:
diff --cc backend/pkg/providers/openai/openai.go
index 675004a0,675004a0..f5f0bf1c
--- a/backend/pkg/providers/openai/openai.go
+++ b/backend/pkg/providers/openai/openai.go
@@@ -18,7 -18,7 +18,7 @@@ import
//go:embed config.yml models.yml
var configFS embed.FS
--const OpenAIAgentModel = "o4-mini"
++const OpenAIAgentModel = "gpt-5.4-nano"
const OpenAIToolCallIDTemplate = "call_{r:24:b}"
@@@ -176,6 -176,6 +176,25 @@@ func (p *openaiProvider) CallWithTools
)
}
++// CallWithExtraOptions: extra is appended last, so it overrides the config.
++func (p *openaiProvider) CallWithExtraOptions(
++ ctx context.Context,
++ opt pconfig.ProviderOptionsType,
++ chain []llms.MessageContent,
++ tools []llms.Tool,
++ streamCb streaming.Callback,
++ extra ...llms.CallOption,
++) (*llms.ContentResponse, error) {
++ options := []llms.CallOption{llms.WithStreamingFunc(streamCb)}
++ if len(tools) > 0 {
++ options = append(options, llms.WithTools(tools))
++ }
++ options = append(options, p.providerConfig.GetOptionsForType(opt)...)
++ options = append(options, extra...)
++
++ return provider.WrapGenerateContent(ctx, p, opt, p.llm.GenerateContent, chain, options...)
++}
++
func (p *openaiProvider) GetUsage(info map[string]any) pconfig.CallUsage {
return pconfig.NewCallUsage(info)
}
diff --cc backend/pkg/providers/openaicompat/openaicompat.go
index 00000000,b5f7ad70..3161737e
mode 000000,100644..100644
--- a/backend/pkg/providers/openaicompat/openaicompat.go
+++ b/backend/pkg/providers/openaicompat/openaicompat.go
@@@ -1,0 -1,173 +1,193 @@@
+ // Package openaicompat is the shared implementation for the OpenAI-compatible
+ // providers that differ only in constants and credentials (deepseek, glm, kimi,
+ // qwen, minimax). The native OpenAI provider is intentionally NOT built on this:
+ // it has no model-prefix support and no reasoning-preservation option.
+ package openaicompat
+
+ import (
+ "context"
+ "fmt"
+ "net/http"
+
+ "pentagi/pkg/providers/pconfig"
+ "pentagi/pkg/providers/provider"
+ "pentagi/pkg/templates"
+
+ "github.com/vxcontrol/langchaingo/llms"
+ "github.com/vxcontrol/langchaingo/llms/openai"
+ "github.com/vxcontrol/langchaingo/llms/streaming"
+ )
+
+ // Spec is the per-provider data the shared base needs.
+ type Spec struct {
+ Type provider.ProviderType
+ Model string // fallback agent model
+ ToolCallIDTemplate string
+ APIKey string
+ ServerURL string
+ Prefix string // LiteLLM model-name prefix (empty disables prefixing)
+ PreserveReasoning bool // adds openai.WithPreserveReasoningContent() for thinking models
+ }
+
+ // Provider implements provider.Provider over an OpenAI-compatible endpoint.
+ type Provider struct {
+ llm *openai.LLM
+ models pconfig.ModelsConfig
+ providerName provider.ProviderName
+ providerConfig *pconfig.ProviderConfig
+ providerType provider.ProviderType
+ prefix string
+ agentModel string
+ toolCallIDTemplate string
+ }
+
+ // New builds an OpenAI-compatible provider from spec. The caller supplies the
+ // shared HTTP client and the already-loaded model catalog.
+ func New(
+ spec Spec,
+ httpClient *http.Client,
+ models pconfig.ModelsConfig,
+ providerName provider.ProviderName,
+ providerConfig *pconfig.ProviderConfig,
+ ) (provider.Provider, error) {
+ if spec.APIKey == "" {
+ return nil, fmt.Errorf("missing API key for provider %q", spec.Type)
+ }
+
+ opts := []openai.Option{
+ openai.WithToken(spec.APIKey),
+ openai.WithModel(spec.Model),
+ openai.WithBaseURL(spec.ServerURL),
+ openai.WithHTTPClient(httpClient),
+ }
+ // Do NOT add openai.WithModernReasoningFormat() to these shared opts: DeepSeek's
+ // API requires the legacy top-level "reasoning_effort" string, which langchaingo
+ // emits only while ModernReasoningFormat is off (the default). The modern
+ // "reasoning":{} object form breaks DeepSeek thinking mode.
+ if spec.PreserveReasoning {
+ opts = append(opts, openai.WithPreserveReasoningContent())
+ }
+
+ client, err := openai.New(opts...)
+ if err != nil {
+ return nil, err
+ }
+
+ return &Provider{
+ llm: client,
+ models: models,
+ providerName: providerName,
+ providerConfig: providerConfig,
+ providerType: spec.Type,
+ prefix: spec.Prefix,
+ agentModel: spec.Model,
+ toolCallIDTemplate: spec.ToolCallIDTemplate,
+ }, nil
+ }
+
+ func (p *Provider) Type() provider.ProviderType {
+ return p.providerType
+ }
+
+ func (p *Provider) Name() provider.ProviderName {
+ return p.providerName
+ }
+
+ func (p *Provider) GetRawConfig() []byte {
+ return p.providerConfig.GetRawConfig()
+ }
+
+ func (p *Provider) GetProviderConfig() *pconfig.ProviderConfig {
+ return p.providerConfig
+ }
+
+ func (p *Provider) GetPriceInfo(opt pconfig.ProviderOptionsType) *pconfig.PriceInfo {
+ return p.providerConfig.GetPriceInfoForType(opt)
+ }
+
+ func (p *Provider) GetModels() pconfig.ModelsConfig {
+ return p.models
+ }
+
+ func (p *Provider) Model(opt pconfig.ProviderOptionsType) string {
+ model := p.agentModel
+ opts := llms.CallOptions{Model: &model}
+ for _, option := range p.providerConfig.GetOptionsForType(opt) {
+ option(&opts)
+ }
+
+ return opts.GetModel()
+ }
+
+ func (p *Provider) ModelWithPrefix(opt pconfig.ProviderOptionsType) string {
+ return provider.ApplyModelPrefix(p.Model(opt), p.prefix)
+ }
+
+ func (p *Provider) Call(
+ ctx context.Context,
+ opt pconfig.ProviderOptionsType,
+ prompt string,
+ ) (string, error) {
+ return provider.WrapGenerateFromSinglePrompt(
+ ctx, p, opt, p.llm, prompt,
+ p.providerConfig.GetOptionsForType(opt)...,
+ )
+ }
+
+ func (p *Provider) CallEx(
+ ctx context.Context,
+ opt pconfig.ProviderOptionsType,
+ chain []llms.MessageContent,
+ streamCb streaming.Callback,
+ ) (*llms.ContentResponse, error) {
+ return provider.WrapGenerateContent(
+ ctx, p, opt, p.llm.GenerateContent, chain,
+ append([]llms.CallOption{
+ llms.WithStreamingFunc(streamCb),
+ }, p.providerConfig.GetOptionsForType(opt)...)...,
+ )
+ }
+
+ func (p *Provider) CallWithTools(
+ ctx context.Context,
+ opt pconfig.ProviderOptionsType,
+ chain []llms.MessageContent,
+ tools []llms.Tool,
+ streamCb streaming.Callback,
+ ) (*llms.ContentResponse, error) {
+ return provider.WrapGenerateContent(
+ ctx, p, opt, p.llm.GenerateContent, chain,
+ append([]llms.CallOption{
+ llms.WithTools(tools),
+ llms.WithStreamingFunc(streamCb),
+ }, p.providerConfig.GetOptionsForType(opt)...)...,
+ )
+ }
+
++// CallWithExtraOptions: extra is appended last, so it overrides the config.
++// Shared by deepseek/glm/kimi/qwen/minimax.
++func (p *Provider) CallWithExtraOptions(
++ ctx context.Context,
++ opt pconfig.ProviderOptionsType,
++ chain []llms.MessageContent,
++ tools []llms.Tool,
++ streamCb streaming.Callback,
++ extra ...llms.CallOption,
++) (*llms.ContentResponse, error) {
++ options := []llms.CallOption{llms.WithStreamingFunc(streamCb)}
++ if len(tools) > 0 {
++ options = append(options, llms.WithTools(tools))
++ }
++ options = append(options, p.providerConfig.GetOptionsForType(opt)...)
++ options = append(options, extra...)
++
++ return provider.WrapGenerateContent(ctx, p, opt, p.llm.GenerateContent, chain, options...)
++}
++
+ func (p *Provider) GetUsage(info map[string]any) pconfig.CallUsage {
+ return pconfig.NewCallUsage(info)
+ }
+
+ func (p *Provider) GetToolCallIDTemplate(ctx context.Context, prompter templates.Prompter) (string, error) {
+ return provider.DetermineToolCallIDTemplate(ctx, p, pconfig.OptionsTypeSimple, prompter, p.toolCallIDTemplate)
+ }
diff --cc backend/pkg/providers/pconfig/config.go
index 0a85f9a7,f68a3213..fa5c6bdf
--- a/backend/pkg/providers/pconfig/config.go
+++ b/backend/pkg/providers/pconfig/config.go
@@@ -232,6 -281,71 +282,71 @@@ type ProviderConfig struct
rawConfig []byte `json:"-" yaml:"-"`
}
+ // Validate rejects universally-invalid agent values (negatives where a value is
+ // physically meaningless, an inverted length window, an out-of-range probability
+ // or temperature, a reasoning budget over the engine cap). It stays permissive —
+ // 0 means "unset", and provider-specific tuning ranges are the LLM API's job —
+ // so it never rejects a valid provider-specific config.
+ func (pc *ProviderConfig) Validate() error {
+ agents := map[string]*AgentConfig{
+ "simple": pc.Simple, "simple_json": pc.SimpleJSON, "primary_agent": pc.PrimaryAgent,
+ "assistant": pc.Assistant, "generator": pc.Generator, "refiner": pc.Refiner,
+ "adviser": pc.Adviser, "reflector": pc.Reflector, "searcher": pc.Searcher,
+ "enricher": pc.Enricher, "coder": pc.Coder, "installer": pc.Installer,
+ "pentester": pc.Pentester,
+ }
+ for name, ac := range agents {
+ if ac == nil {
+ continue
+ }
+ if err := ac.Validate(); err != nil {
+ return fmt.Errorf("%s: %w", name, err)
+ }
+ }
+ return nil
+ }
+
+ func (ac *AgentConfig) Validate() error {
+ if ac.Temperature < 0 || ac.Temperature > 2 {
+ return fmt.Errorf("temperature %v out of range [0, 2]", ac.Temperature)
+ }
+ if ac.TopP < 0 || ac.TopP > 1 {
+ return fmt.Errorf("top_p %v out of range [0, 1]", ac.TopP)
+ }
+ if ac.TopK < 0 {
+ return fmt.Errorf("top_k %d must be >= 0", ac.TopK)
+ }
+ if ac.MaxTokens < 0 {
+ return fmt.Errorf("max_tokens %d must be >= 0", ac.MaxTokens)
+ }
+ if ac.MinLength < 0 {
+ return fmt.Errorf("min_length %d must be >= 0", ac.MinLength)
+ }
+ if ac.MaxLength < 0 {
+ return fmt.Errorf("max_length %d must be >= 0", ac.MaxLength)
+ }
+ if ac.MinLength > 0 && ac.MaxLength > 0 && ac.MinLength > ac.MaxLength {
+ return fmt.Errorf("min_length %d must not exceed max_length %d", ac.MinLength, ac.MaxLength)
+ }
+ if ac.RepetitionPenalty < 0 || ac.RepetitionPenalty > 2 {
+ return fmt.Errorf("repetition_penalty %v out of range [0, 2]", ac.RepetitionPenalty)
+ }
+ if ac.FrequencyPenalty < -2 || ac.FrequencyPenalty > 2 {
+ return fmt.Errorf("frequency_penalty %v out of range [-2, 2]", ac.FrequencyPenalty)
+ }
+ if ac.PresencePenalty < -2 || ac.PresencePenalty > 2 {
+ return fmt.Errorf("presence_penalty %v out of range [-2, 2]", ac.PresencePenalty)
+ }
- if ac.Reasoning.MaxTokens < 0 || ac.Reasoning.MaxTokens > 32000 {
- return fmt.Errorf("reasoning.max_tokens %d out of range [0, 32000]", ac.Reasoning.MaxTokens)
++ if ac.Reasoning.MaxTokens < 0 || ac.Reasoning.MaxTokens > 32768 {
++ return fmt.Errorf("reasoning.max_tokens %d out of range [0, 32768]", ac.Reasoning.MaxTokens)
+ }
+ if ac.Price != nil &&
+ (ac.Price.Input < 0 || ac.Price.Output < 0 || ac.Price.CacheRead < 0 || ac.Price.CacheWrite < 0) {
+ return fmt.Errorf("price values must be >= 0")
+ }
+ return nil
+ }
+
const EmptyProviderConfigRaw = `{
"simple": {},
"simple_json": {},
@@@ -573,14 -717,22 +718,37 @@@ func (ac *AgentConfig) BuildOptions() [
if _, ok := ac.raw["response_mime_type"]; ok && ac.ResponseMIMEType != "" {
options = append(options, llms.WithResponseMIMEType(ac.ResponseMIMEType))
}
- if _, ok := ac.raw["reasoning"]; ok && (ac.Reasoning.Effort != llms.ReasoningNone || ac.Reasoning.MaxTokens != 0) {
- switch ac.Reasoning.Effort {
- case llms.ReasoningLow, llms.ReasoningMedium, llms.ReasoningHigh:
- options = append(options, llms.WithReasoning(ac.Reasoning.Effort, 0))
- default:
- if ac.Reasoning.MaxTokens > 0 && ac.Reasoning.MaxTokens <= 32000 {
++ // This "reasoning" block is the generic, provider-agnostic thinking control:
++ // on the wire it only ever produces the standard OpenAI-shaped top-level
++ // `reasoning_effort` string (or its disable token), via langchaingo's shared
++ // openai adapter. GLM (Z.AI), Kimi (Moonshot), Qwen (DashScope), and DeepSeek
++ // additionally expose their OWN vendor-specific thinking toggle through
++ // `extra_body` (`thinking.type` for GLM/Kimi/DeepSeek, `enable_thinking` for
++ // Qwen) — see each provider's config.yml for the exact keys. The two
++ // mechanisms are independent and are not reconciled here: `extra_body` is
++ // raw passthrough (below) that this switch never inspects, so for those
++ // vendors, whether the model actually thinks at all is decided by
++ // `extra_body`, while `reasoning.effort` (where the vendor documents support
++ // for it, e.g. GLM-5.2+/Kimi K3) only tunes depth once thinking is already
++ // running. Setting only `reasoning: {mode: off}` on those providers does NOT
++ // guarantee thinking is disabled unless the matching `extra_body` toggle is
++ // also set to disabled.
+ if _, ok := ac.raw["reasoning"]; ok && !ac.Reasoning.IsZero() {
+ switch ac.Reasoning.EffectiveMode() {
+ case ReasoningModeOff:
+ options = append(options, llms.WithReasoningDisabled())
+ case ReasoningModeAdaptive:
+ // Adaptive thinking is applied per-call by PrepareAdaptiveCallOptions
+ // (it appends llms.WithAdaptiveReasoning); no CallOption is emitted here.
+ case ReasoningModeBudget:
- if ac.Reasoning.MaxTokens > 0 && ac.Reasoning.MaxTokens <= 32000 {
++ if ac.Reasoning.MaxTokens > 0 && ac.Reasoning.MaxTokens <= 32768 {
options = append(options, llms.WithReasoning(llms.ReasoningNone, ac.Reasoning.MaxTokens))
}
+ default:
+ switch ac.Reasoning.Effort {
+ case llms.ReasoningLow, llms.ReasoningMedium, llms.ReasoningHigh, llms.ReasoningXHigh, llms.ReasoningMax:
+ options = append(options, llms.WithReasoning(ac.Reasoning.Effort, 0))
+ }
}
}
if _, ok := ac.raw["extra_body"]; ok && ac.ExtraBody != nil {
@@@ -821,6 -975,68 +991,76 @@@ func (pc *ProviderConfig) GetPriceInfoF
return nil
}
+ func (pc *ProviderConfig) modelReasoningMode(models ModelsConfig, opt ProviderOptionsType) ModelReasoningMode {
+ agentConfig := pc.AgentConfigForType(opt)
+ if agentConfig == nil || agentConfig.Model == "" {
+ return ModelReasoningNone
+ }
+
+ for _, m := range models {
+ if m.Name == agentConfig.Model && m.Reasoning != nil {
+ return m.Reasoning.Mode
+ }
+ }
+
+ return ModelReasoningNone
+ }
+
+ func (pc *ProviderConfig) reasoningConfigForType(opt ProviderOptionsType) (ReasoningConfig, bool) {
+ agentConfig := pc.AgentConfigForType(opt)
+ if agentConfig == nil || agentConfig.Reasoning.IsZero() {
+ return ReasoningConfig{}, false
+ }
+
+ return agentConfig.Reasoning, true
+ }
+
+ // UsesAdaptiveThinking reports whether this agent's call must use adaptive
+ // thinking: the agent selected adaptive mode, or the model only supports adaptive
+ // (e.g. Opus 4.7/4.8, where budget thinking returns a 400).
++//
++// Caveat: an adaptive-only model forces this to true even when the agent has no
++// reasoning config at all (see the first branch below), and an empty effort
++// defaults to "high" on the wire (PrepareAdaptiveCallOptions). Assigning such a
++// model to a fast/cheap utility role (simple, simple_json, reflector, searcher,
++// ...) therefore silently adds real latency/cost to what is meant to be a quick
++// call, unless the operator explicitly sets `reasoning: {mode: off}` for that
++// agent. There is currently no lower-effort default for these roles.
+ func (pc *ProviderConfig) UsesAdaptiveThinking(models ModelsConfig, opt ProviderOptionsType) bool {
+ // An explicit off wins over the adaptive-only auto-adaptive below: the caller
+ // disabled thinking, so no adaptive CallOption may be appended (the disable is
+ // emitted in BuildOptions instead).
+ if reasoning, ok := pc.reasoningConfigForType(opt); ok && reasoning.EffectiveMode() == ReasoningModeOff {
+ return false
+ }
+ if pc.modelReasoningMode(models, opt) == ModelReasoningAdaptiveOnly {
+ return true
+ }
+
+ reasoning, ok := pc.reasoningConfigForType(opt)
+ return ok && reasoning.EffectiveMode() == ReasoningModeAdaptive
+ }
+
+ // PrepareAdaptiveCallOptions appends the adaptive-thinking CallOption for agents
+ // that require it. No-op for non-adaptive calls. The langchaingo provider emits
+ // thinking.type=adaptive + output_config.effort and omits sampling params; an
+ // empty effort defaults to "high" there.
+ func (pc *ProviderConfig) PrepareAdaptiveCallOptions(
+ ctx context.Context,
+ models ModelsConfig,
+ opt ProviderOptionsType,
+ options []llms.CallOption,
+ ) (context.Context, []llms.CallOption) {
+ if !pc.UsesAdaptiveThinking(models, opt) {
+ return ctx, options
+ }
+
+ reasoning, _ := pc.reasoningConfigForType(opt)
+ options = append(options, llms.WithAdaptiveReasoning(reasoning.Effort))
+
+ return ctx, options
+ }
+
func (pc *ProviderConfig) BuildOptionsMap() map[ProviderOptionsType][]llms.CallOption {
if pc == nil {
return nil
diff --cc backend/pkg/providers/pconfig/config_test.go
index e899de7b,2f25b85a..565e4c94
--- a/backend/pkg/providers/pconfig/config_test.go
+++ b/backend/pkg/providers/pconfig/config_test.go
@@@ -13,6 -14,91 +14,265 @@@ import
"gopkg.in/yaml.v3"
)
+ func TestProviderConfig_UsesAdaptiveThinking(t *testing.T) {
+ adaptiveOnly := ModelsConfig{{Name: "opus-4-8", Reasoning: &ModelReasoningInfo{Mode: ModelReasoningAdaptiveOnly}}}
+ adaptiveCapable := ModelsConfig{{Name: "opus-4-6", Reasoning: &ModelReasoningInfo{Mode: ModelReasoningAdaptive}}}
+ budgetOnly := ModelsConfig{{Name: "sonnet-4-5", Reasoning: &ModelReasoningInfo{Mode: ModelReasoningBudget}}}
+
+ tests := []struct {
+ name string
+ models ModelsConfig
+ agent *AgentConfig
+ want bool
+ }{
+ {"adaptive-only model forces adaptive with no agent reasoning",
+ adaptiveOnly, &AgentConfig{Model: "opus-4-8"}, true},
+ {"adaptive-only model overrides an agent budget choice",
+ adaptiveOnly, &AgentConfig{Model: "opus-4-8", Reasoning: ReasoningConfig{Mode: ReasoningModeBudget, MaxTokens: 4096}}, true},
+ {"explicit off wins over the adaptive-only auto-adaptive",
+ adaptiveOnly, &AgentConfig{Model: "opus-4-8", Reasoning: ReasoningConfig{Mode: ReasoningModeOff}}, false},
+ {"agent selects adaptive on an adaptive-capable model",
+ adaptiveCapable, &AgentConfig{Model: "opus-4-6", Reasoning: ReasoningConfig{Mode: ReasoningModeAdaptive}}, true},
+ {"agent selects budget on an adaptive-capable model",
+ adaptiveCapable, &AgentConfig{Model: "opus-4-6", Reasoning: ReasoningConfig{Mode: ReasoningModeBudget, MaxTokens: 4096}}, false},
++ {"unknown model still honors an explicit adaptive agent choice",
++ ModelsConfig{}, &AgentConfig{Model: "unknown", Reasoning: ReasoningConfig{Mode: ReasoningModeAdaptive}}, true},
+ {"budget-only model with a budget agent",
+ budgetOnly, &AgentConfig{Model: "sonnet-4-5", Reasoning: ReasoningConfig{Mode: ReasoningModeBudget, MaxTokens: 4096}}, false},
+ {"no reasoning anywhere",
+ budgetOnly, &AgentConfig{Model: "sonnet-4-5"}, false},
++ {"empty model cannot match adaptive-only catalog entry",
++ adaptiveOnly, &AgentConfig{Model: ""}, false},
++ {"nil agent is not adaptive",
++ adaptiveOnly, nil, false},
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ pc := &ProviderConfig{Simple: tt.agent}
+ assert.Equal(t, tt.want, pc.UsesAdaptiveThinking(tt.models, OptionsTypeSimple))
+ })
+ }
+ }
+
++// PrepareAdaptiveCallOptions is a thin wrapper around UsesAdaptiveThinking; these
++// cases cover option shape only (append / effort / preserve). Predicate branches
++// live in TestProviderConfig_UsesAdaptiveThinking.
+ func TestProviderConfig_PrepareAdaptiveCallOptions(t *testing.T) {
+ adaptiveOnly := ModelsConfig{{Name: "opus-4-8", Reasoning: &ModelReasoningInfo{Mode: ModelReasoningAdaptiveOnly}}}
++ budgetOnly := ModelsConfig{{Name: "sonnet-4-5", Reasoning: &ModelReasoningInfo{Mode: ModelReasoningBudget}}}
+
- // adaptiveReasoning applies the returned options to a CallOptions and reports
- // the reasoning the provider would emit (nil if no adaptive option was added).
- adaptiveReasoning := func(options []llms.CallOption) *llms.ReasoningConfig {
++ appliedReasoning := func(options []llms.CallOption) *llms.ReasoningConfig {
+ var co llms.CallOptions
+ for _, o := range options {
+ o(&co)
+ }
+ return co.Reasoning
+ }
+
- t.Run("overrides an agent budget choice on an adaptive-only model", func(t *testing.T) {
++ t.Run("appends adaptive option and forwards effort", func(t *testing.T) {
+ pc := &ProviderConfig{Simple: &AgentConfig{
+ Model: "opus-4-8",
- Reasoning: ReasoningConfig{Mode: ReasoningModeBudget, Effort: llms.ReasoningHigh, MaxTokens: 4096},
++ Reasoning: ReasoningConfig{Effort: llms.ReasoningHigh},
+ }}
- _, options := pc.PrepareAdaptiveCallOptions(context.Background(), adaptiveOnly, OptionsTypeSimple, nil)
- require.Len(t, options, 1, "the adaptive call option must be appended")
++ existing := []llms.CallOption{llms.WithTemperature(0.5)}
++ _, options := pc.PrepareAdaptiveCallOptions(context.Background(), adaptiveOnly, OptionsTypeSimple, existing)
++ require.Len(t, options, 2, "adaptive option must be appended after existing options")
+
- r := adaptiveReasoning(options)
++ r := appliedReasoning(options)
+ require.NotNil(t, r)
- assert.True(t, r.Adaptive, "a budget choice must be overridden to adaptive for an adaptive-only model")
++ assert.True(t, r.Adaptive)
+ assert.Equal(t, llms.ReasoningHigh, r.Effort)
+ })
+
- t.Run("forces adaptive on an adaptive-only model with no agent reasoning block", func(t *testing.T) {
++ t.Run("empty effort when agent has no reasoning block", func(t *testing.T) {
+ pc := &ProviderConfig{Simple: &AgentConfig{Model: "opus-4-8"}}
+ _, options := pc.PrepareAdaptiveCallOptions(context.Background(), adaptiveOnly, OptionsTypeSimple, nil)
+ require.Len(t, options, 1)
+
- r := adaptiveReasoning(options)
++ r := appliedReasoning(options)
+ require.NotNil(t, r)
+ assert.True(t, r.Adaptive)
+ assert.Equal(t, llms.ReasoningNone, r.Effort, "empty effort is left for the provider to default to high")
+ })
+
- t.Run("no-op for a budget-only model, preserving existing options", func(t *testing.T) {
- budgetOnly := ModelsConfig{{Name: "sonnet-4-5", Reasoning: &ModelReasoningInfo{Mode: ModelReasoningBudget}}}
++ t.Run("off on adaptive-only does not append", func(t *testing.T) {
++ pc := &ProviderConfig{Simple: &AgentConfig{
++ Model: "opus-4-8",
++ Reasoning: ReasoningConfig{Mode: ReasoningModeOff},
++ }}
++ existing := []llms.CallOption{llms.WithTemperature(1)}
++ _, options := pc.PrepareAdaptiveCallOptions(context.Background(), adaptiveOnly, OptionsTypeSimple, existing)
++ require.Len(t, options, 1, "existing options must be preserved unchanged")
++ assert.Nil(t, appliedReasoning(options), "off must not append adaptive reasoning")
++ })
++
++ t.Run("no-op preserves existing options", func(t *testing.T) {
+ pc := &ProviderConfig{Simple: &AgentConfig{
+ Model: "sonnet-4-5",
+ Reasoning: ReasoningConfig{Mode: ReasoningModeBudget, MaxTokens: 4096},
+ }}
+ existing := []llms.CallOption{llms.WithTemperature(1)}
+ _, options := pc.PrepareAdaptiveCallOptions(context.Background(), budgetOnly, OptionsTypeSimple, existing)
+ require.Len(t, options, 1, "a non-adaptive call must not append the adaptive option")
- assert.Nil(t, adaptiveReasoning(options), "no adaptive reasoning option should be present")
++ assert.Nil(t, appliedReasoning(options), "no adaptive reasoning option should be present")
+ })
+ }
+
++// An explicit off mode must emit llms.WithReasoningDisabled() so the langchaingo
++// adapter sends the provider's disable wire. Helper semantics for Mode/IsZero live
++// in TestReasoningConfig_EffectiveMode / TestReasoningConfig_IsZero.
++func TestProviderConfig_EmitsDisableOnOffMode(t *testing.T) {
++ var ac AgentConfig
++ require.NoError(t, yaml.Unmarshal([]byte("model: gpt-5.5\nreasoning:\n mode: \"off\"\n"), &ac))
++
++ pc := &ProviderConfig{Simple: &ac}
++ var applied llms.CallOptions
++ for _, opt := range pc.GetOptionsForType(OptionsTypeSimple) {
++ opt(&applied)
++ }
++
++ require.NotNil(t, applied.Reasoning, "off must emit a reasoning option")
++ assert.True(t, applied.Reasoning.IsDisabled(), "off must disable reasoning on the call options")
++}
++
++// Dependency contract: langchaingo must keep xhigh/max unclamped so OpenAI-compatible
++// reasoning_effort carries the real level. Not a pconfig.ReasoningConfig unit test.
++func TestLangchaingoReasoningConfig_GetEffortPreservesXHighAndMax(t *testing.T) {
++ for _, effort := range []llms.ReasoningEffort{llms.ReasoningXHigh, llms.ReasoningMax} {
++ rc := llms.ReasoningConfig{Effort: effort}
++ if got := rc.GetEffort(8192); got != effort {
++ t.Errorf("GetEffort(effort=%q) = %q, want %q", effort, got, effort)
++ }
++ }
++}
++
++func TestProviderConfig_EmitsTopReasoningEffortOnEffortPath(t *testing.T) {
++ for _, effort := range []string{"low", "medium", "high", "xhigh", "max"} {
++ t.Run(effort, func(t *testing.T) {
++ var ac AgentConfig
++ require.NoError(t, yaml.Unmarshal([]byte("model: gpt-5.5\nreasoning:\n effort: "+effort+"\n"), &ac))
++
++ pc := &ProviderConfig{Simple: &ac}
++ var applied llms.CallOptions
++ for _, opt := range pc.GetOptionsForType(OptionsTypeSimple) {
++ opt(&applied)
++ }
++
++ require.NotNil(t, applied.Reasoning, "effort %q must emit a reasoning option on the non-adaptive path", effort)
++ assert.Equal(t, llms.ReasoningEffort(effort), applied.Reasoning.Effort, "the configured effort must reach the call options unchanged")
++ assert.False(t, applied.Reasoning.Adaptive)
++ assert.False(t, applied.Reasoning.IsDisabled())
++ })
++ }
++}
++
++// gopkg.in/yaml.v3 follows YAML 1.2 core (only true/false are booleans), so an
++// unquoted off in a user-authored provider YAML parses as the string mode, not a
++// boolean. This guards the user-editable config path against the YAML 1.1 off gotcha.
++func TestReasoningConfig_UnquotedOffParsesAsMode(t *testing.T) {
++ var ac AgentConfig
++ require.NoError(t, yaml.Unmarshal([]byte("model: gpt-5.5\nreasoning:\n mode: off\n"), &ac))
++ assert.Equal(t, ReasoningModeOff, ac.Reasoning.Mode)
++}
++
++// TestModelConfigReasoningCapability guards the custom ModelConfig (un)marshalers:
++// the model reasoning capability descriptor must survive YAML and JSON round-trips,
++// because providers rely on it to force adaptive thinking for adaptive-only models.
++func TestModelConfigReasoningCapability(t *testing.T) {
++ yamlData := []byte("- name: m1\n reasoning:\n mode: adaptive-only\n efforts: [low, high, xhigh]\n")
++ models, err := LoadModelsConfigData(yamlData)
++ require.NoError(t, err)
++ require.Len(t, models, 1)
++ require.NotNil(t, models[0].Reasoning, "reasoning must parse from YAML")
++ assert.Equal(t, ModelReasoningAdaptiveOnly, models[0].Reasoning.Mode)
++ assert.Equal(t, []llms.ReasoningEffort{"low", "high", "xhigh"}, models[0].Reasoning.Efforts)
++
++ jsonBytes, err := json.Marshal(models[0])
++ require.NoError(t, err)
++ var backJSON ModelConfig
++ require.NoError(t, json.Unmarshal(jsonBytes, &backJSON))
++ require.NotNil(t, backJSON.Reasoning, "reasoning must survive JSON round-trip")
++ assert.Equal(t, ModelReasoningAdaptiveOnly, backJSON.Reasoning.Mode)
++ assert.Equal(t, []llms.ReasoningEffort{"low", "high", "xhigh"}, backJSON.Reasoning.Efforts)
++
++ yamlBytes, err := yaml.Marshal(models[0])
++ require.NoError(t, err)
++ var backYAML ModelConfig
++ require.NoError(t, yaml.Unmarshal(yamlBytes, &backYAML))
++ require.NotNil(t, backYAML.Reasoning, "reasoning must survive YAML round-trip")
++ assert.Equal(t, ModelReasoningAdaptiveOnly, backYAML.Reasoning.Mode)
++ assert.Equal(t, []llms.ReasoningEffort{"low", "high", "xhigh"}, backYAML.Reasoning.Efforts)
++}
++
++func TestReasoningConfig_EffectiveMode(t *testing.T) {
++ tests := []struct {
++ name string
++ rc ReasoningConfig
++ want ReasoningMode
++ }{
++ {"empty -> default", ReasoningConfig{}, ReasoningModeDefault},
++ {"explicit adaptive", ReasoningConfig{Mode: ReasoningModeAdaptive}, ReasoningModeAdaptive},
++ {"explicit budget", ReasoningConfig{Mode: ReasoningModeBudget}, ReasoningModeBudget},
++ {"explicit off", ReasoningConfig{Mode: ReasoningModeOff}, ReasoningModeOff},
++ {"off wins over max_tokens", ReasoningConfig{Mode: ReasoningModeOff, MaxTokens: 5000}, ReasoningModeOff},
++ {"effort only, no mode/tokens -> default", ReasoningConfig{Effort: llms.ReasoningHigh}, ReasoningModeDefault},
++ {"max_tokens, no effort -> budget", ReasoningConfig{MaxTokens: 5000}, ReasoningModeBudget},
++ {"max_tokens + none effort -> budget", ReasoningConfig{Effort: llms.ReasoningNone, MaxTokens: 5000}, ReasoningModeBudget},
++ {"max_tokens + string none effort -> budget", ReasoningConfig{Effort: llms.ReasoningEffort("none"), MaxTokens: 5000}, ReasoningModeBudget},
++ {"max_tokens + non-none effort -> default", ReasoningConfig{Effort: llms.ReasoningHigh, MaxTokens: 5000}, ReasoningModeDefault},
++ }
++ for _, tt := range tests {
++ t.Run(tt.name, func(t *testing.T) {
++ assert.Equal(t, tt.want, tt.rc.EffectiveMode())
++ })
++ }
++}
++
++func TestReasoningConfig_IsZero(t *testing.T) {
++ assert.True(t, ReasoningConfig{}.IsZero())
++ assert.False(t, ReasoningConfig{Mode: ReasoningModeAdaptive}.IsZero())
++ assert.False(t, ReasoningConfig{Mode: ReasoningModeBudget}.IsZero())
++ assert.False(t, ReasoningConfig{Mode: ReasoningModeOff}.IsZero())
++ assert.False(t, ReasoningConfig{Effort: llms.ReasoningLow}.IsZero())
++ assert.False(t, ReasoningConfig{MaxTokens: 1}.IsZero())
++}
++
++func TestAgentConfigValidate(t *testing.T) {
++ require.NoError(t, (&AgentConfig{
++ Temperature: 2, TopP: 1, TopK: 40, MaxTokens: 8192,
++ MinLength: 1, MaxLength: 100,
++ RepetitionPenalty: 1.1, FrequencyPenalty: -2, PresencePenalty: 2,
++ Reasoning: ReasoningConfig{MaxTokens: 32768},
++ Price: &PriceInfo{Input: 5, Output: 25},
++ }).Validate())
++ require.NoError(t, (&AgentConfig{}).Validate(), "zero value (all unset) is valid")
++
++ cases := map[string]*AgentConfig{
++ "temperature too high": {Temperature: 5},
++ "temperature negative": {Temperature: -1},
++ "top_p too high": {TopP: 1.5},
++ "top_k negative": {TopK: -1},
++ "max_tokens negative": {MaxTokens: -100},
++ "inverted length window": {MinLength: 5000, MaxLength: 10},
++ "frequency out of range": {FrequencyPenalty: 9},
++ "presence out of range": {PresencePenalty: -3},
++ "repetition out of range": {RepetitionPenalty: 3},
++ "reasoning budget over cap": {Reasoning: ReasoningConfig{MaxTokens: 40000}},
++ "reasoning budget negative": {Reasoning: ReasoningConfig{MaxTokens: -1}},
++ "negative price": {Price: &PriceInfo{Input: -1}},
++ "negative cache_read price": {Price: &PriceInfo{CacheRead: -0.1}},
++ }
++ for name, ac := range cases {
++ t.Run(name, func(t *testing.T) {
++ assert.Error(t, ac.Validate())
++ })
++ }
++
++ err := (&ProviderConfig{Adviser: &AgentConfig{Temperature: 5}}).Validate()
++ require.Error(t, err)
++ assert.Contains(t, err.Error(), "adviser")
++}
++
func TestReasoningConfig_UnmarshalJSON(t *testing.T) {
tests := []struct {
name string
@@@ -934,7 -1063,7 +1237,7 @@@ func TestAgentConfig_BuildOptions(t *te
"max_tokens": 50000
}
}`,
-- wantLen: 2, // shouldn't include reasoning option because tokens > 32000
++ wantLen: 2, // shouldn't include reasoning option because tokens > 32768
},
{
name: "with invalid reasoning tokens negative",
diff --cc backend/pkg/providers/provider/provider.go
index b80b4aef,5e89bb83..20d7e464
--- a/backend/pkg/providers/provider/provider.go
+++ b/backend/pkg/providers/provider/provider.go
@@@ -19,6 -20,24 +20,34 @@@ func (p ProviderType) String() string
return string(p)
}
+ // ReasoningProvider maps the provider type to the langchaingo reasoning.Provider
-// consumed by capability introspection (llms.ReasoningSupportFor) and the disable
-// resolver. OpenAI-compatible providers all speak the OpenAI reasoning wire.
++// consumed ONLY by capability introspection for the settings UI (CannotDisable /
++// Supported hints via llms.ReasoningSupportFor / reasoning.ResolveOff) — it has no
++// effect on the actual wire call, which each provider builds independently.
++//
++// DeepSeek/GLM/Kimi/Qwen/MiniMax/Custom are folded into reasoning.ProviderOpenAI
++// as a best-effort approximation, not because they share OpenAI's real disable
++// semantics: GLM/Kimi/DeepSeek's actual thinking on/off switch is their own
++// extra_body toggle (see BuildOptions' reasoning-block comment), and Custom can
++// front literally any backend behind LLM_SERVER_URL. So the resulting hint may
++// say "reasoning can be disabled via effort=none" for a model whose real API
++// ignores that field entirely and only obeys extra_body. Treat this mapping as
++// a rough default for the UI, not a guarantee of correct wire behavior.
+ func (p ProviderType) ReasoningProvider() reasoning.Provider {
+ switch p {
+ case ProviderAnthropic:
+ return reasoning.ProviderAnthropic
+ case ProviderBedrock:
+ return reasoning.ProviderBedrock
+ case ProviderGemini:
+ return reasoning.ProviderGoogleAI
+ case ProviderOpenAI, ProviderDeepSeek, ProviderGLM, ProviderKimi, ProviderQwen, ProviderMiniMax, ProviderCustom:
+ return reasoning.ProviderOpenAI
+ default: // ProviderOllama and anything unrecognized
+ return reasoning.ProviderUnknown
+ }
+ }
+
const (
ProviderOpenAI ProviderType = "openai"
ProviderAnthropic ProviderType = "anthropic"
@@@ -75,6 -110,6 +120,19 @@@ type Provider interface
tools []llms.Tool,
streamCb streaming.Callback,
) (*llms.ContentResponse, error)
++ // CallWithExtraOptions is CallWithTools with extra appended after the
++ // agent's own configured options, so it always wins. Lets a caller force a
++ // wire behavior (e.g. adaptive thinking, reasoning off, structured output)
++ // the agent's static config doesn't already request — used by the
++ // provider tester, but not limited to it.
++ CallWithExtraOptions(
++ ctx context.Context,
++ opt pconfig.ProviderOptionsType,
++ chain []llms.MessageContent,
++ tools []llms.Tool,
++ streamCb streaming.Callback,
++ extra ...llms.CallOption,
++ ) (*llms.ContentResponse, error)
// Configuration access methods
GetRawConfig() []byte
diff --cc backend/pkg/providers/providers.go
index 2c4a8e3c,94d18d48..a9a69c7d
--- a/backend/pkg/providers/providers.go
+++ b/backend/pkg/providers/providers.go
@@@ -158,9 -151,26 +153,38 @@@ type providerController struct
defaultConfigs provider.ProvidersConfig
++ // defaultConfigErrors records, per provider type, why buildDefaultConfigs
++ // tolerated a load failure (type disabled + unreadable custom config path).
++ // patchProviderConfig surfaces this instead of a bare "not found" so a later
++ // CreateProvider/UpdateProvider attempt on that type reports the real root
++ // cause (e.g. a bad BEDROCK_CONFIG_PATH) rather than an ambiguous message
++ // that reads the same as "this provider type does not exist at all".
++ defaultConfigErrors map[provider.ProviderType]error
++
provider.Providers
}
-func buildDefaultConfigs(cfg *config.Config) (provider.ProvidersConfig, error) {
++func buildDefaultConfigs(
++ cfg *config.Config,
++) (provider.ProvidersConfig, map[provider.ProviderType]error, error) {
+ defaultConfigs := make(provider.ProvidersConfig)
++ skipReasons := make(map[provider.ProviderType]error)
+ for _, e := range providerRegistry {
+ config, err := e.NewConfig(cfg)
+ if err != nil {
+ // A returned error here aborts startup; tolerate it for disabled providers.
+ if !e.Enabled(cfg) {
+ logrus.WithError(err).Warnf("skipping config for disabled %s provider", e.Type)
++ skipReasons[e.Type] = err
+ continue
+ }
- return nil, fmt.Errorf("failed to create %s provider config: %w", e.Type, err)
++ return nil, nil, fmt.Errorf("failed to create %s provider config: %w", e.Type, err)
+ }
+ defaultConfigs[e.Type] = config
+ }
- return defaultConfigs, nil
++ return defaultConfigs, skipReasons, nil
+ }
+
func NewProviderController(
cfg *config.Config,
db database.Querier,
@@@ -176,174 -186,23 +200,23 @@@
}
providers := make(provider.Providers)
- defaultConfigs := make(provider.ProvidersConfig)
-
- if config, err := openai.DefaultProviderConfig(); err != nil {
- return nil, fmt.Errorf("failed to create openai provider config: %w", err)
- } else {
- defaultConfigs[provider.ProviderOpenAI] = config
- }
-
- if config, err := anthropic.DefaultProviderConfig(); err != nil {
- return nil, fmt.Errorf("failed to create anthropic provider config: %w", err)
- } else {
- defaultConfigs[provider.ProviderAnthropic] = config
- }
-
- if config, err := gemini.DefaultProviderConfig(); err != nil {
- return nil, fmt.Errorf("failed to create gemini provider config: %w", err)
- } else {
- defaultConfigs[provider.ProviderGemini] = config
- }
-
- if config, err := bedrock.DefaultProviderConfig(cfg); err != nil {
- return nil, fmt.Errorf("failed to create bedrock provider config: %w", err)
- } else {
- defaultConfigs[provider.ProviderBedrock] = config
- }
-
- if config, err := ollama.DefaultProviderConfig(cfg); err != nil {
- return nil, fmt.Errorf("failed to create ollama provider config: %w", err)
- } else {
- defaultConfigs[provider.ProviderOllama] = config
- }
-
- if config, err := custom.DefaultProviderConfig(cfg); err != nil {
- return nil, fmt.Errorf("failed to create custom provider config: %w", err)
- } else {
- defaultConfigs[provider.ProviderCustom] = config
- }
-
- if config, err := deepseek.DefaultProviderConfig(); err != nil {
- return nil, fmt.Errorf("failed to create deepseek provider config: %w", err)
- } else {
- defaultConfigs[provider.ProviderDeepSeek] = config
- }
-
- if config, err := glm.DefaultProviderConfig(); err != nil {
- return nil, fmt.Errorf("failed to create glm provider config: %w", err)
- } else {
- defaultConfigs[provider.ProviderGLM] = config
- }
-
- if config, err := kimi.DefaultProviderConfig(); err != nil {
- return nil, fmt.Errorf("failed to create kimi provider config: %w", err)
- } else {
- defaultConfigs[provider.ProviderKimi] = config
- }
-
- if config, err := qwen.DefaultProviderConfig(); err != nil {
- return nil, fmt.Errorf("failed to create qwen provider config: %w", err)
- } else {
- defaultConfigs[provider.ProviderQwen] = config
- }
-
- if config, err := minimax.DefaultProviderConfig(); err != nil {
- return nil, fmt.Errorf("failed to create minimax provider config: %w", err)
- } else {
- defaultConfigs[provider.ProviderMiniMax] = config
- }
-
- if cfg.OpenAIKey != "" {
- p, err := openai.New(cfg, provider.DefaultProviderNameOpenAI, defaultConfigs[provider.ProviderOpenAI])
- if err != nil {
- return nil, fmt.Errorf("failed to create openai provider: %w", err)
- }
-
- providers[provider.DefaultProviderNameOpenAI] = p
- }
-
- if cfg.AnthropicAPIKey != "" {
- p, err := anthropic.New(cfg, provider.DefaultProviderNameAnthropic, defaultConfigs[provider.ProviderAnthropic])
- if err != nil {
- return nil, fmt.Errorf("failed to create anthropic provider: %w", err)
- }
-
- providers[provider.DefaultProviderNameAnthropic] = p
- }
- if cfg.GeminiAPIKey != "" {
- p, err := gemini.New(cfg, provider.DefaultProviderNameGemini, defaultConfigs[provider.ProviderGemini])
- if err != nil {
- return nil, fmt.Errorf("failed to create gemini provider: %w", err)
- }
-
- providers[provider.DefaultProviderNameGemini] = p
- }
-
- // Bedrock supports three authentication strategies:
- // 1. Default AWS SDK auth (BedrockDefaultAuth=true)
- // 2. Bearer token (BedrockBearerToken set)
- // 3. Static credentials (BedrockAccessKey + BedrockSecretKey)
- if cfg.BedrockDefaultAuth || cfg.BedrockBearerToken != "" ||
- (cfg.BedrockAccessKey != "" && cfg.BedrockSecretKey != "") {
- p, err := bedrock.New(cfg, provider.DefaultProviderNameBedrock, defaultConfigs[provider.ProviderBedrock])
- if err != nil {
- return nil, fmt.Errorf("failed to create bedrock provider: %w", err)
- }
- providers[provider.DefaultProviderNameBedrock] = p
- }
-
- if cfg.OllamaServerURL != "" {
- p, err := ollama.New(cfg, provider.DefaultProviderNameOllama, defaultConfigs[provider.ProviderOllama])
- if err != nil {
- return nil, fmt.Errorf("failed to create ollama provider: %w", err)
- }
- providers[provider.DefaultProviderNameOllama] = p
- }
-
- if cfg.LLMServerURL != "" && (cfg.LLMServerModel != "" || cfg.LLMServerConfig != "") {
- p, err := custom.New(cfg, provider.DefaultProviderNameCustom, defaultConfigs[provider.ProviderCustom])
- if err != nil {
- return nil, fmt.Errorf("failed to create custom provider: %w", err)
- }
-
- providers[provider.DefaultProviderNameCustom] = p
- }
-
- if cfg.DeepSeekAPIKey != "" {
- p, err := deepseek.New(cfg, provider.DefaultProviderNameDeepSeek, defaultConfigs[provider.ProviderDeepSeek])
- if err != nil {
- return nil, fmt.Errorf("failed to create deepseek provider: %w", err)
- }
-
- providers[provider.DefaultProviderNameDeepSeek] = p
- }
-
- if cfg.GLMAPIKey != "" {
- p, err := glm.New(cfg, provider.DefaultProviderNameGLM, defaultConfigs[provider.ProviderGLM])
- if err != nil {
- return nil, fmt.Errorf("failed to create glm provider: %w", err)
- }
-
- providers[provider.DefaultProviderNameGLM] = p
- }
-
- if cfg.KimiAPIKey != "" {
- p, err := kimi.New(cfg, provider.DefaultProviderNameKimi, defaultConfigs[provider.ProviderKimi])
- if err != nil {
- return nil, fmt.Errorf("failed to create kimi provider: %w", err)
- }
-
- providers[provider.DefaultProviderNameKimi] = p
- defaultConfigs, err := buildDefaultConfigs(cfg)
++ defaultConfigs, defaultConfigErrors, err := buildDefaultConfigs(cfg)
+ if err != nil {
+ return nil, err
}
- if cfg.QwenAPIKey != "" {
- p, err := qwen.New(cfg, provider.DefaultProviderNameQwen, defaultConfigs[provider.ProviderQwen])
- if err != nil {
- return nil, fmt.Errorf("failed to create qwen provider: %w", err)
+ for _, e := range providerRegistry {
+ if !e.Enabled(cfg) {
+ continue
}
- providers[provider.DefaultProviderNameQwen] = p
- }
-
- if cfg.MiniMaxAPIKey != "" {
- p, err := minimax.New(cfg, provider.DefaultProviderNameMiniMax, defaultConfigs[provider.ProviderMiniMax])
+ p, err := e.New(cfg, e.Name, defaultConfigs[e.Type])
if err != nil {
- return nil, fmt.Errorf("failed to create minimax provider: %w", err)
+ return nil, fmt.Errorf("failed to create %s provider: %w", e.Type, err)
}
- providers[provider.DefaultProviderNameMiniMax] = p
+ providers[e.Name] = p
}
summarizerAgent := csum.NewSummarizer(csum.SummarizerConfig{
@@@ -394,27 -253,10 +267,28 @@@
summarizerAgent: summarizerAgent,
summarizerAssistant: summarizerAssistant,
-- defaultConfigs: defaultConfigs,
++ defaultConfigs: defaultConfigs,
++ defaultConfigErrors: defaultConfigErrors,
Providers: providers,
- }, nil
+ }
+
+ // Seed configured system providers into the DB for all existing users so
+ // they are immediately available without any UI interaction. This runs on
+ // every startup, so editing a YAML config file and restarting PentAGI
+ // automatically propagates new values.
+ ctx := context.Background()
+ if users, err := db.GetUsers(ctx); err != nil {
+ logrus.WithError(err).Warn("failed to fetch users for provider seeding")
+ } else {
+ for _, u := range users {
+ if err := pc.SeedDefaultProviders(ctx, u.ID); err != nil {
+ logrus.WithError(err).Warnf("failed to seed default providers for user %d", u.ID)
+ }
+ }
+ }
+
+ return pc, nil
}
func (pc *providerController) NewFlowProvider(
@@@ -796,7 -613,12 +645,14 @@@ func (pc *providerController) GetProvid
for _, prv := range providers {
p, err := pc.NewProvider(prv)
if err != nil {
- return nil, fmt.Errorf("failed to build provider: %w", err)
+ // Any unbuildable saved provider (its type is disabled, or its stored
+ // config is stale/invalid) is skipped, not propagated — one bad row must
- // not take down the whole list. Debug, not Warn: it re-fires on every
- // providers fetch; WithError carries the specific reason.
- logrus.WithError(err).Debugf("skipping unusable user provider '%s' (type '%s')", prv.Name, prv.Type)
++ // not take down the whole list. Logged at Warn (not Error) since this is
++ // an expected, recoverable condition (e.g. a type disabled via env vars),
++ // but it must stay visible in production logs rather than silently
++ // disappearing the provider from the list on every fetch.
++ logrus.WithError(err).Warnf("skipping unusable user provider '%s' (type '%s')", prv.Name, prv.Type)
+ continue
}
providersMap[provider.ProviderName(prv.Name)] = p
}
@@@ -816,124 -638,19 +672,65 @@@ func (pc *providerController) NewProvid
return nil, fmt.Errorf("provider type '%s' is not available", prv.Type)
}
- switch providerType {
- case provider.ProviderOpenAI:
- openaiConfig, err := openai.BuildProviderConfig(prv.Config)
- if err != nil {
- return nil, fmt.Errorf("failed to build openai provider config: %w", err)
- }
- return openai.New(pc.cfg, providerName, openaiConfig)
- case provider.ProviderAnthropic:
- anthropicConfig, err := anthropic.BuildProviderConfig(prv.Config)
- if err != nil {
- return nil, fmt.Errorf("failed to build anthropic provider config: %w", err)
- }
- return anthropic.New(pc.cfg, providerName, anthropicConfig)
- case provider.ProviderGemini:
- geminiConfig, err := gemini.BuildProviderConfig(prv.Config)
- if err != nil {
- return nil, fmt.Errorf("failed to build gemini provider config: %w", err)
- }
- return gemini.New(pc.cfg, providerName, geminiConfig)
- case provider.ProviderBedrock:
- bedrockConfig, err := bedrock.BuildProviderConfig(prv.Config)
- if err != nil {
- return nil, fmt.Errorf("failed to build bedrock provider config: %w", err)
- }
- return bedrock.New(pc.cfg, providerName, bedrockConfig)
- case provider.ProviderOllama:
- ollamaConfig, err := ollama.BuildProviderConfig(pc.cfg, prv.Config)
- if err != nil {
- return nil, fmt.Errorf("failed to build ollama provider config: %w", err)
- }
- return ollama.New(pc.cfg, providerName, ollamaConfig)
- case provider.ProviderCustom:
- customConfig, err := custom.BuildProviderConfig(pc.cfg, prv.Config)
- if err != nil {
- return nil, fmt.Errorf("failed to build custom provider config: %w", err)
- }
- return custom.New(pc.cfg, providerName, customConfig)
- case provider.ProviderDeepSeek:
- deepseekConfig, err := deepseek.BuildProviderConfig(prv.Config)
- if err != nil {
- return nil, fmt.Errorf("failed to build deepseek provider config: %w", err)
- }
- return deepseek.New(pc.cfg, providerName, deepseekConfig)
- case provider.ProviderGLM:
- glmConfig, err := glm.BuildProviderConfig(prv.Config)
- if err != nil {
- return nil, fmt.Errorf("failed to build glm provider config: %w", err)
- }
- return glm.New(pc.cfg, providerName, glmConfig)
- case provider.ProviderKimi:
- kimiConfig, err := kimi.BuildProviderConfig(prv.Config)
- if err != nil {
- return nil, fmt.Errorf("failed to build kimi provider config: %w", err)
- }
- return kimi.New(pc.cfg, providerName, kimiConfig)
- case provider.ProviderQwen:
- qwenConfig, err := qwen.BuildProviderConfig(prv.Config)
- if err != nil {
- return nil, fmt.Errorf("failed to build qwen provider config: %w", err)
- }
- return qwen.New(pc.cfg, providerName, qwenConfig)
- case provider.ProviderMiniMax:
- minimaxConfig, err := minimax.BuildProviderConfig(prv.Config)
- if err != nil {
- return nil, fmt.Errorf("failed to build minimax provider config: %w", err)
- }
- return minimax.New(pc.cfg, providerName, minimaxConfig)
- default:
+ e, ok := entryForType(providerType)
+ if !ok {
return nil, fmt.Errorf("unknown provider type: %s", prv.Type)
}
+
+ config, err := e.BuildConfig(pc.cfg, prv.Config)
+ if err != nil {
+ return nil, fmt.Errorf("failed to build %s provider config: %w", providerType, err)
+ }
+
+ return e.New(pc.cfg, providerName, config)
}
+func (pc *providerController) SeedDefaultProviders(ctx context.Context, userID int64) error {
+ if pc.cfg.BedrockConfig == "" {
+ return nil
+ }
+ if !pc.cfg.BedrockDefaultAuth && pc.cfg.BedrockBearerToken == "" &&
+ (pc.cfg.BedrockAccessKey == "" || pc.cfg.BedrockSecretKey == "") {
+ return nil
+ }
+
+ bedrockCfg, ok := pc.defaultConfigs[provider.ProviderBedrock]
+ if !ok {
+ return nil
+ }
+
+ rawConfig, err := json.Marshal(bedrockCfg)
+ if err != nil {
+ return fmt.Errorf("failed to marshal bedrock config: %w", err)
+ }
+
+ prvname := bedrockCfg.Name
+ if prvname == "" {
+ prvname = string(provider.DefaultProviderNameBedrock)
+ }
+ existing, err := pc.db.GetUserProviderByName(ctx, database.GetUserProviderByNameParams{
+ Name: prvname,
+ UserID: userID,
+ })
+ if err != nil {
+ _, err = pc.db.CreateProvider(ctx, database.CreateProviderParams{
+ UserID: userID,
+ Type: database.ProviderType(provider.ProviderBedrock),
+ Name: prvname,
+ Config: rawConfig,
+ })
+ return err
+ }
+
+ _, err = pc.db.UpdateUserProvider(ctx, database.UpdateUserProviderParams{
+ ID: existing.ID,
+ UserID: userID,
+ Config: rawConfig,
+ Name: existing.Name,
+ })
+ return err
+}
+
func (pc *providerController) CreateProvider(
ctx context.Context,
userID int64,
@@@ -1189,6 -914,6 +994,12 @@@ func (pc *providerController) patchProv
)
if defaultCfg, ok = pc.defaultConfigs[prvtype]; !ok {
++ if reason, hasReason := pc.defaultConfigErrors[prvtype]; hasReason {
++ return nil, fmt.Errorf(
++ "provider type '%s' has no default config because it failed to load at startup: %w",
++ prvtype.String(), reason,
++ )
++ }
return nil, fmt.Errorf("default provider config not found for type: %s", prvtype.String())
}
diff --cc backend/pkg/providers/providers_test.go
index 00000000,00000000..f988363d
new file mode 100644
--- /dev/null
+++ b/backend/pkg/providers/providers_test.go
@@@ -1,0 -1,0 +1,226 @@@
++package providers
++
++import (
++ "context"
++ "path/filepath"
++ "testing"
++
++ "pentagi/pkg/config"
++ "pentagi/pkg/database"
++ "pentagi/pkg/providers/anthropic"
++ "pentagi/pkg/providers/bedrock"
++ "pentagi/pkg/providers/deepseek"
++ "pentagi/pkg/providers/gemini"
++ "pentagi/pkg/providers/glm"
++ "pentagi/pkg/providers/kimi"
++ "pentagi/pkg/providers/minimax"
++ "pentagi/pkg/providers/openai"
++ "pentagi/pkg/providers/pconfig"
++ "pentagi/pkg/providers/provider"
++ "pentagi/pkg/providers/qwen"
++
++ "github.com/stretchr/testify/assert"
++ "github.com/stretchr/testify/require"
++)
++
++// stubTypedProvider reports only its type — enough for Providers.ListTypes(),
++// which is all GetProviders touches before it rejects an unavailable type.
++type stubTypedProvider struct {
++ provider.Provider
++ ptype provider.ProviderType
++}
++
++func (s stubTypedProvider) Type() provider.ProviderType { return s.ptype }
++
++type stubProvidersQuerier struct {
++ database.Querier
++ rows []database.Provider
++}
++
++func (s stubProvidersQuerier) GetUserProviders(context.Context, int64) ([]database.Provider, error) {
++ return s.rows, nil
++}
++
++func TestGetProviders_SkipsUserProviderOfDisabledType(t *testing.T) {
++ pc := &providerController{
++ cfg: &config.Config{},
++ db: stubProvidersQuerier{rows: []database.Provider{
++ {Name: "stale-minimax", Type: "minimax"}, // not in ListTypes() below
++ }},
++ Providers: provider.Providers{
++ "openai-default": stubTypedProvider{ptype: provider.ProviderOpenAI},
++ },
++ }
++
++ got, err := pc.GetProviders(context.Background(), 1)
++
++ require.NoError(t, err, "one user provider of a disabled type must not fail the whole query")
++ assert.Contains(t, got, provider.ProviderName("openai-default"), "enabled providers stay reachable")
++ assert.NotContains(t, got, provider.ProviderName("stale-minimax"), "the unavailable provider is skipped")
++}
++
++func TestGetProviders_StaleUserRowSpansValidSibling(t *testing.T) {
++ pc := &providerController{
++ cfg: &config.Config{},
++ db: stubProvidersQuerier{rows: []database.Provider{
++ // ollama.New builds with no API key and makes no network call (pull/load
++ // off under the empty config), so this row clears the real NewProvider.
++ {Name: "ollama-user", Type: "ollama"},
++ {Name: "stale-minimax", Type: "minimax"}, // type absent from ListTypes()
++ }},
++ Providers: provider.Providers{
++ "ollama-default": stubTypedProvider{ptype: provider.ProviderOllama},
++ },
++ }
++
++ got, err := pc.GetProviders(context.Background(), 1)
++
++ require.NoError(t, err, "a stale sibling row must not fail the whole query")
++ assert.Contains(t, got, provider.ProviderName("ollama-user"), "a valid user provider survives a stale sibling")
++ assert.NotContains(t, got, provider.ProviderName("stale-minimax"), "the unbuildable sibling is skipped")
++}
++
++func TestBuildDefaultConfigs_DisabledProviderBadPathIsNotFatal(t *testing.T) {
++ cfg := &config.Config{
++ BedrockConfig: filepath.Join(t.TempDir(), "missing.yml"),
++ }
++
++ configs, skipReasons, err := buildDefaultConfigs(cfg)
++
++ require.NoError(t, err)
++ _, hasBedrock := configs[provider.ProviderBedrock]
++ assert.False(t, hasBedrock, "disabled provider with an unreadable config path is skipped")
++ assert.Error(t, skipReasons[provider.ProviderBedrock], "the skip reason must be recorded, not just swallowed")
++ _, hasOpenAI := configs[provider.ProviderOpenAI]
++ assert.True(t, hasOpenAI, "providers with a readable (embedded) default config still load")
++}
++
++func TestBuildDefaultConfigs_EnabledProviderBadPathIsFatal(t *testing.T) {
++ cfg := &config.Config{
++ BedrockConfig: filepath.Join(t.TempDir(), "missing.yml"),
++ BedrockBearerToken: "token",
++ }
++
++ _, _, err := buildDefaultConfigs(cfg)
++
++ require.Error(t, err)
++}
++
++// A provider type whose default config failed to load at startup (e.g. an
++// unreadable BEDROCK_CONFIG_PATH while Bedrock is otherwise disabled) must
++// report that specific cause from patchProviderConfig, not the generic "not
++// found" message that reads identically to "this type does not exist".
++func TestPatchProviderConfig_SurfacesSkipReasonInsteadOfAmbiguousNotFound(t *testing.T) {
++ cfg := &config.Config{
++ BedrockConfig: filepath.Join(t.TempDir(), "missing.yml"),
++ }
++ configs, skipReasons, err := buildDefaultConfigs(cfg)
++ require.NoError(t, err)
++
++ pc := &providerController{
++ cfg: cfg,
++ defaultConfigs: configs,
++ defaultConfigErrors: skipReasons,
++ }
++
++ _, err = pc.patchProviderConfig(provider.ProviderBedrock, nil)
++ require.Error(t, err)
++ assert.Contains(t, err.Error(), "failed to load at startup", "must surface the real cause, not just 'not found'")
++}
++
++func TestOpenAICompatProvidersDoNotUseAdaptiveThinking(t *testing.T) {
++ t.Parallel()
++
++ providers := []struct {
++ name string
++ config func() (*pconfig.ProviderConfig, error)
++ models func() (pconfig.ModelsConfig, error)
++ }{
++ {"qwen", qwen.DefaultProviderConfig, qwen.DefaultModels},
++ {"glm", glm.DefaultProviderConfig, glm.DefaultModels},
++ {"deepseek", deepseek.DefaultProviderConfig, deepseek.DefaultModels},
++ {"kimi", kimi.DefaultProviderConfig, kimi.DefaultModels},
++ {"minimax", minimax.DefaultProviderConfig, minimax.DefaultModels},
++ }
++
++ for _, p := range providers {
++ cfg, err := p.config()
++ if err != nil {
++ t.Fatalf("%s: DefaultProviderConfig() error: %v", p.name, err)
++ }
++ models, err := p.models()
++ if err != nil {
++ t.Fatalf("%s: DefaultModels() error: %v", p.name, err)
++ }
++
++ for _, opt := range pconfig.AllAgentTypes {
++ if cfg.UsesAdaptiveThinking(models, opt) {
++ t.Errorf("%s/%s resolves to adaptive thinking, but the openaicompat transport never emits it (M7): adaptive thinking is Anthropic/Bedrock-only", p.name, opt)
++ }
++ }
++
++ for _, m := range models {
++ if m.Reasoning == nil {
++ continue
++ }
++ if mode := m.Reasoning.Mode; mode == pconfig.ModelReasoningAdaptive || mode == pconfig.ModelReasoningAdaptiveOnly {
++ t.Errorf("%s catalog model %q declares reasoning mode %q, but the openaicompat transport never emits adaptive thinking (M7): adaptive thinking is Anthropic/Bedrock-only", p.name, m.Name, mode)
++ }
++ }
++ }
++}
++
++// The default agent config (config.yml) carries its own per-agent price, and
++// GetPriceInfoForType returns it verbatim (no catalog fallback), so a drift from
++// the model catalog silently mis-prices cost telemetry for the default config.
++func TestAgentConfigPricesMatchCatalog(t *testing.T) {
++ t.Parallel()
++
++ providers := []struct {
++ name string
++ config func() (*pconfig.ProviderConfig, error)
++ models func() (pconfig.ModelsConfig, error)
++ }{
++ {"anthropic", anthropic.DefaultProviderConfig, anthropic.DefaultModels},
++ {"bedrock",
++ func() (*pconfig.ProviderConfig, error) { return bedrock.DefaultProviderConfig(&config.Config{}) },
++ func() (pconfig.ModelsConfig, error) { return bedrock.DefaultModels() }},
++ {"gemini", gemini.DefaultProviderConfig, gemini.DefaultModels},
++ {"deepseek", deepseek.DefaultProviderConfig, deepseek.DefaultModels},
++ {"glm", glm.DefaultProviderConfig, glm.DefaultModels},
++ {"kimi", kimi.DefaultProviderConfig, kimi.DefaultModels},
++ {"minimax", minimax.DefaultProviderConfig, minimax.DefaultModels},
++ {"openai", openai.DefaultProviderConfig, openai.DefaultModels},
++ {"qwen", qwen.DefaultProviderConfig, qwen.DefaultModels},
++ }
++
++ for _, p := range providers {
++ cfg, err := p.config()
++ if err != nil {
++ t.Fatalf("%s: DefaultProviderConfig() error: %v", p.name, err)
++ }
++ models, err := p.models()
++ if err != nil {
++ t.Fatalf("%s: DefaultModels() error: %v", p.name, err)
++ }
++
++ catalog := make(map[string]*pconfig.PriceInfo, len(models))
++ for i := range models {
++ catalog[models[i].Name] = models[i].Price
++ }
++
++ for _, opt := range pconfig.AllAgentTypes {
++ ac := cfg.AgentConfigForType(opt)
++ if ac == nil || ac.Model == "" || ac.Price == nil {
++ continue
++ }
++ cat, ok := catalog[ac.Model]
++ if !ok || cat == nil {
++ continue
++ }
++ if *ac.Price != *cat {
++ t.Errorf("%s/%s model %q: config.yml price %+v != catalog price %+v", p.name, opt, ac.Model, *ac.Price, *cat)
++ }
++ }
++ }
++}
diff --cc backend/pkg/providers/tester/capability.go
index 00000000,00000000..be49d998
new file mode 100644
--- /dev/null
+++ b/backend/pkg/providers/tester/capability.go
@@@ -1,0 -1,0 +1,60 @@@
++package tester
++
++import (
++ "pentagi/pkg/providers/pconfig"
++ "pentagi/pkg/providers/provider"
++ "pentagi/pkg/providers/tester/testdata"
++)
++
++// capabilitySupported reports whether opt's agent config, exactly as loaded
++// (models.yml + the operator's provider YAML), would actually exercise
++// capability during a real PentAGI flow. ctester exists to answer "will this
++// configuration work inside PentAGI", so a capability test must only run when
++// PentAGI's own runtime — pconfig.AgentConfig.BuildOptions /
++// ProviderConfig.UsesAdaptiveThinking, the same code CallWithTools/Call use on
++// every real request — would make this exact call for this exact agent.
++// Forcing the wire behavior through some other path the app never takes
++// (e.g. via CallWithExtraOptions) would test a call that can never happen in
++// production, so it is deliberately not done here: pass or fail, the result
++// wouldn't say anything about a flow that will ever actually run.
++func capabilitySupported(prv provider.Provider, opt pconfig.ProviderOptionsType, capability testdata.TestCapability) bool {
++ switch capability {
++ case testdata.CapabilityNone:
++ return true
++ case testdata.CapabilityStructuredOutput:
++ // Deliberate exception to the "only test a call PentAGI already
++ // makes" rule above: AgentConfig.BuildOptions doesn't wire
++ // llms.WithStructuredOutput yet, but that wiring is landing soon for
++ // the simple_json agent specifically. This runs now, ahead of it, as
++ // a readiness check — is this LLM/backend even capable of
++ // schema-constrained output at all — so there's already 100%
++ // confidence in the backend once the real call path ships. Only ever
++ // reachable for OptionsTypeSimpleJSON in practice: isTestCompatibleWithAgent
++ // restricts TestTypeJSON (what structured_output tests are) to that
++ // agent alone.
++ return true
++ }
++
++ agentConfig := prv.GetProviderConfig().AgentConfigForType(opt)
++ if agentConfig == nil {
++ return false
++ }
++
++ switch capability {
++ case testdata.CapabilityAdaptiveThinking:
++ // Same gate PrepareAdaptiveCallOptions uses to decide whether to
++ // append llms.WithAdaptiveReasoning for this opt: either the agent
++ // explicitly chose adaptive mode, or its assigned model only supports
++ // adaptive thinking (e.g. Claude Opus 4.7/4.8), in which case PentAGI
++ // forces it on regardless of what the agent's reasoning config says.
++ return prv.GetProviderConfig().UsesAdaptiveThinking(prv.GetModels(), opt)
++ case testdata.CapabilityReasoningOff:
++ // Same gate AgentConfig.BuildOptions uses to decide whether to append
++ // llms.WithReasoningDisabled: only when this agent's own config sets
++ // reasoning mode to off. If it doesn't, PentAGI never sends a disable
++ // signal for this agent at all, so there is nothing to verify.
++ return agentConfig.Reasoning.EffectiveMode() == pconfig.ReasoningModeOff
++ default:
++ return false
++ }
++}
diff --cc backend/pkg/providers/tester/mock/provider.go
index c0c65641,c0c65641..d020a718
--- a/backend/pkg/providers/tester/mock/provider.go
+++ b/backend/pkg/providers/tester/mock/provider.go
@@@ -23,6 -23,6 +23,8 @@@ type Provider struct
responses map[string]interface{} // key -> response mapping
defaultResp string
streamingDelay time.Duration
++ providerConfig *pconfig.ProviderConfig
++ models pconfig.ModelsConfig
}
// ResponseConfig configures mock responses
@@@ -60,6 -60,6 +62,21 @@@ func (p *Provider) SetStreamingDelay(de
p.streamingDelay = delay
}
++// SetProviderConfig configures the per-agent config GetProviderConfig
++// returns, so capability gating (tester.capabilitySupported) can be exercised
++// against a mock the same way it reads a real provider's loaded YAML — e.g.
++// setting AgentConfig.Reasoning.Mode to off/adaptive for a given agent type.
++func (p *Provider) SetProviderConfig(pc *pconfig.ProviderConfig) {
++ p.providerConfig = pc
++}
++
++// SetModels configures the model catalog GetModels returns, so tests can
++// simulate an adaptive-only (or otherwise reasoning-classified) model without
++// depending on the SDK's real model-name tables.
++func (p *Provider) SetModels(models pconfig.ModelsConfig) {
++ p.models = models
++}
++
// Type implements provider.Provider
func (p *Provider) Type() provider.ProviderType {
return p.providerType
@@@ -87,6 -87,6 +104,9 @@@ func (p *Provider) GetUsage(info map[st
// GetModels implements provider.Provider
func (p *Provider) GetModels() pconfig.ModelsConfig {
++ if p.models != nil {
++ return p.models
++ }
return pconfig.ModelsConfig{}
}
@@@ -213,6 -213,6 +233,77 @@@ func (p *Provider) CallWithTools
return p.handleContentResponse(respInterface)
}
++// CallWithExtraOptions implements provider.Provider for completeness. The
++// capability test pipeline no longer routes through it for adaptive
++// thinking/reasoning off (those are gated to configs that already produce the
++// behavior via the plain CallWithTools/CallEx path below, so tests should set
++// up the mock's canned response accordingly rather than relying on this
++// method to synthesize it); it is reachable by the CapabilityStructuredOutput
++// path. Uses partial-match lookup like CallEx (CallWithTools only matches
++// exact content, too strict for multi-sentence prompts).
++func (p *Provider) CallWithExtraOptions(
++ ctx context.Context,
++ opt pconfig.ProviderOptionsType,
++ chain []llms.MessageContent,
++ tools []llms.Tool,
++ streamCb streaming.Callback,
++ extra ...llms.CallOption,
++) (*llms.ContentResponse, error) {
++ var applied llms.CallOptions
++ for _, o := range extra {
++ o(&applied)
++ }
++
++ var content string
++ for _, msg := range chain {
++ for _, part := range msg.Parts {
++ if textContent, ok := part.(llms.TextContent); ok {
++ content += textContent.Text + " "
++ }
++ }
++ }
++ content = strings.TrimSpace(content)
++
++ var respInterface interface{}
++ if resp, ok := p.responses[content]; ok {
++ respInterface = resp
++ } else {
++ for key, resp := range p.responses {
++ if strings.Contains(content, key) {
++ respInterface = resp
++ break
++ }
++ }
++ }
++ if respInterface == nil {
++ respInterface = p.defaultResp
++ }
++
++ resp, err := p.handleContentResponse(respInterface)
++ if err != nil {
++ return nil, err
++ }
++
++ switch {
++ case applied.Reasoning != nil && applied.Reasoning.IsDisabled():
++ for _, choice := range resp.Choices {
++ choice.Reasoning = nil
++ }
++ case applied.Reasoning != nil && applied.Reasoning.Adaptive:
++ for _, choice := range resp.Choices {
++ if choice.Reasoning.IsEmpty() {
++ choice.Reasoning = &reasoning.ContentReasoning{Content: "mock adaptive reasoning trace"}
++ }
++ }
++ }
++
++ if streamCb == nil {
++ return resp, nil
++ }
++
++ return p.handleStreamingResponse(ctx, resp, streamCb)
++}
++
// GetRawConfig implements provider.Provider
func (p *Provider) GetRawConfig() []byte {
return []byte(`{"mock": true}`)
@@@ -220,6 -220,6 +311,9 @@@
// GetProviderConfig implements provider.Provider
func (p *Provider) GetProviderConfig() *pconfig.ProviderConfig {
++ if p.providerConfig != nil {
++ return p.providerConfig
++ }
return &pconfig.ProviderConfig{}
}
diff --cc backend/pkg/providers/tester/runner.go
index 917cb566,917cb566..8d14a91f
--- a/backend/pkg/providers/tester/runner.go
+++ b/backend/pkg/providers/tester/runner.go
@@@ -2,6 -2,6 +2,7 @@@ package teste
import (
"context"
++ "errors"
"fmt"
"log"
"sync"
@@@ -10,6 -10,6 +11,9 @@@
"pentagi/pkg/providers/pconfig"
"pentagi/pkg/providers/provider"
"pentagi/pkg/providers/tester/testdata"
++
++ "github.com/vxcontrol/langchaingo/llms"
++ "github.com/vxcontrol/langchaingo/llms/reasoning"
)
// testRequest represents a test execution request
@@@ -93,6 -93,6 +97,14 @@@ func collectTestRequests(registry *test
continue
}
++ // skip capability-gated tests (adaptive thinking, reasoning
++ // off, structured output) whose wire behavior this agent's
++ // ACTUAL config wouldn't produce in a real PentAGI flow — see
++ // capabilitySupported for why.
++ if !capabilitySupported(prv, agentType, testCase.Capability()) {
++ continue
++ }
++
requests = append(requests, testRequest{
agentType: agentType,
testCase: testCase,
@@@ -179,8 -179,8 +191,23 @@@ func executeTest(ctx context.Context, r
var response interface{}
var err error
++ extra := req.testCase.ExtraOptions()
++
// execute based on test type and available data
switch {
++ case len(extra) > 0:
++ // Only reachable by a CapabilityStructuredOutput test today — see
++ // TestCase.ExtraOptions. adaptive_thinking/reasoning_off are gated to
++ // configs that already produce the wire behavior through the plain
++ // CallWithTools/CallEx path below, so they never populate extra.
++ response, err = req.provider.CallWithExtraOptions(
++ ctx,
++ req.agentType,
++ req.testCase.Messages(),
++ req.testCase.Tools(),
++ req.testCase.StreamingCallback(),
++ extra...,
++ )
case len(req.testCase.Messages()) > 0 && len(req.testCase.Tools()) > 0:
// tool calling with messages
response, err = req.provider.CallWithTools(
@@@ -209,18 -209,18 +236,37 @@@
if err != nil {
return testdata.TestResult{
-- ID: req.testCase.ID(),
-- Name: req.testCase.Name(),
-- Type: req.testCase.Type(),
-- Group: req.testCase.Group(),
-- Success: false,
-- Error: err,
-- Latency: latency,
++ ID: req.testCase.ID(),
++ Name: req.testCase.Name(),
++ Type: req.testCase.Type(),
++ Group: req.testCase.Group(),
++ Capability: req.testCase.Capability(),
++ Success: false,
++ Unsupported: isUnsupportedCapabilityError(err),
++ Error: err,
++ Latency: latency,
}, nil
}
// let test case validate and produce result
-- return req.testCase.Execute(response, latency), nil
++ result := req.testCase.Execute(response, latency)
++ result.Capability = req.testCase.Capability()
++ return result, nil
++}
++
++// isUnsupportedCapabilityError reports whether err is one of the SDK's typed
++// "this model/provider does not support the requested capability" sentinels,
++// proactively returned by the langchaingo adapters before any network call —
++// distinguishing "the capability isn't available here" (informative, not a
++// defect in the tested configuration) from a genuine call failure.
++func isUnsupportedCapabilityError(err error) bool {
++ var structuredUnsupported *llms.ErrStructuredOutputUnsupported
++ var structuredConflict *llms.ErrStructuredOutputConflict
++ var reasoningOffUnsupported *reasoning.ErrReasoningOffUnsupported
++
++ return errors.As(err, &structuredUnsupported) ||
++ errors.As(err, &structuredConflict) ||
++ errors.As(err, &reasoningOffUnsupported)
}
// groupResults organizes test results by agent type
diff --cc backend/pkg/providers/tester/runner_test.go
index 47eda9d6,47eda9d6..cef36cae
--- a/backend/pkg/providers/tester/runner_test.go
+++ b/backend/pkg/providers/tester/runner_test.go
@@@ -11,6 -11,6 +11,7 @@@ import
"pentagi/pkg/providers/tester/testdata"
"github.com/vxcontrol/langchaingo/llms"
++ "github.com/vxcontrol/langchaingo/llms/reasoning"
)
func TestTestProvider(t *testing.T) {
@@@ -325,6 -325,6 +326,148 @@@ func TestTestProviderGroups(t *testing.
}
}
++// TestTestProviderCapabilityGating_AdaptiveThinking verifies that
++// adaptive_thinking_reasoning_present runs — via the plain CallWithTools/
++// CallEx path, not a forced extra option — only when the agent's assigned
++// model is adaptive-only, exactly the condition
++// ProviderConfig.UsesAdaptiveThinking uses in production to decide whether to
++// append llms.WithAdaptiveReasoning. reasoning_off_suppresses_reasoning must
++// NOT run: this config never sets reasoning mode to off, so PentAGI's runtime
++// would never send that disable signal for this agent either.
++func TestTestProviderCapabilityGating_AdaptiveThinking(t *testing.T) {
++ const model = "claude-opus-4-8"
++ mockProvider := mock.NewProvider(provider.ProviderAnthropic, provider.DefaultProviderNameAnthropic, model)
++ mockProvider.SetProviderConfig(&pconfig.ProviderConfig{
++ Simple: &pconfig.AgentConfig{Model: model},
++ })
++ mockProvider.SetModels(pconfig.ModelsConfig{
++ {Name: model, Reasoning: &pconfig.ModelReasoningInfo{Mode: pconfig.ModelReasoningAdaptiveOnly}},
++ })
++ mockProvider.SetResponses([]mock.ResponseConfig{
++ {Key: "farmer has 17 sheep", Response: &llms.ContentResponse{
++ Choices: []*llms.ContentChoice{{
++ Content: "The answer is 13",
++ Reasoning: &reasoning.ContentReasoning{Content: "mock reasoning trace"},
++ }},
++ }},
++ })
++
++ results, err := TestProvider(
++ t.Context(),
++ mockProvider,
++ WithAgentTypes(pconfig.OptionsTypeSimple),
++ WithGroups(testdata.TestGroupAdvanced),
++ WithStreamingMode(false),
++ )
++ if err != nil {
++ t.Fatalf("TestProvider capability gating failed: %v", err)
++ }
++
++ var foundAdaptive, foundOff bool
++ for _, result := range results.Simple {
++ switch result.ID {
++ case "adaptive_thinking_reasoning_present":
++ foundAdaptive = true
++ if !result.Success {
++ t.Errorf("adaptive_thinking test failed: %v", result.Error)
++ }
++ if result.Capability != testdata.CapabilityAdaptiveThinking {
++ t.Errorf("expected capability %q, got %q", testdata.CapabilityAdaptiveThinking, result.Capability)
++ }
++ case "reasoning_off_suppresses_reasoning":
++ foundOff = true
++ }
++ }
++
++ if !foundAdaptive {
++ t.Errorf("expected adaptive_thinking_reasoning_present to run for an adaptive-only model")
++ }
++ if foundOff {
++ t.Errorf("reasoning_off_suppresses_reasoning must not run: this config never sets reasoning mode to off")
++ }
++}
++
++// TestTestProviderCapabilityGating_ReasoningOff verifies that
++// reasoning_off_suppresses_reasoning runs — via the plain CallWithTools/
++// CallEx path — only when the agent's own config explicitly sets
++// `reasoning: {mode: off}`, exactly the condition AgentConfig.BuildOptions
++// uses in production to decide whether to append llms.WithReasoningDisabled.
++// adaptive_thinking_reasoning_present must NOT run for this config.
++func TestTestProviderCapabilityGating_ReasoningOff(t *testing.T) {
++ mockProvider := mock.NewProvider(provider.ProviderCustom, provider.DefaultProviderNameCustom, "some-model")
++ mockProvider.SetProviderConfig(&pconfig.ProviderConfig{
++ Simple: &pconfig.AgentConfig{
++ Model: "some-model",
++ Reasoning: pconfig.ReasoningConfig{Mode: pconfig.ReasoningModeOff},
++ },
++ })
++ mockProvider.SetResponses([]mock.ResponseConfig{
++ {Key: "capital of France", Response: &llms.ContentResponse{
++ Choices: []*llms.ContentChoice{{Content: "Paris"}},
++ }},
++ })
++
++ results, err := TestProvider(
++ t.Context(),
++ mockProvider,
++ WithAgentTypes(pconfig.OptionsTypeSimple),
++ WithGroups(testdata.TestGroupAdvanced),
++ WithStreamingMode(false),
++ )
++ if err != nil {
++ t.Fatalf("TestProvider capability gating failed: %v", err)
++ }
++
++ var foundAdaptive, foundOff bool
++ for _, result := range results.Simple {
++ switch result.ID {
++ case "reasoning_off_suppresses_reasoning":
++ foundOff = true
++ if !result.Success {
++ t.Errorf("reasoning_off test failed: %v", result.Error)
++ }
++ if result.Capability != testdata.CapabilityReasoningOff {
++ t.Errorf("expected capability %q, got %q", testdata.CapabilityReasoningOff, result.Capability)
++ }
++ case "adaptive_thinking_reasoning_present":
++ foundAdaptive = true
++ }
++ }
++
++ if !foundOff {
++ t.Errorf("expected reasoning_off_suppresses_reasoning to run when the agent's config sets reasoning mode to off")
++ }
++ if foundAdaptive {
++ t.Errorf("adaptive_thinking_reasoning_present must not run: this config doesn't request adaptive thinking")
++ }
++}
++
++// TestTestProviderCapabilityGating_SkipsDefaultConfig verifies that capability
++// tests are entirely absent (not failed) for a provider whose config sets
++// neither an adaptive mode nor an explicit reasoning-off — i.e. a flow that
++// would never trigger either CallOption in production.
++func TestTestProviderCapabilityGating_SkipsDefaultConfig(t *testing.T) {
++ mockProvider := mock.NewProvider(provider.ProviderCustom, provider.DefaultProviderNameCustom, "test-model")
++ mockProvider.SetDefaultResponse("Mock response")
++
++ results, err := TestProvider(
++ t.Context(),
++ mockProvider,
++ WithAgentTypes(pconfig.OptionsTypeSimple),
++ WithGroups(testdata.TestGroupAdvanced),
++ WithStreamingMode(false),
++ )
++ if err != nil {
++ t.Fatalf("TestProvider capability gating failed: %v", err)
++ }
++
++ for _, result := range results.Simple {
++ if result.Capability != testdata.CapabilityNone {
++ t.Errorf("default config must not run capability test %s (capability=%s)", result.ID, result.Capability)
++ }
++ }
++}
++
func TestApplyOptions(t *testing.T) {
// test default configuration
config := applyOptions(nil)
diff --cc backend/pkg/providers/tester/testdata/completion.go
index d064c345,d064c345..5fc1430b
--- a/backend/pkg/providers/tester/testdata/completion.go
+++ b/backend/pkg/providers/tester/testdata/completion.go
@@@ -49,6 -49,6 +49,15 @@@ func (t *testCaseCompletion) Streaming(
func (t *testCaseCompletion) Prompt() string { return t.def.Prompt }
func (t *testCaseCompletion) Messages() []llms.MessageContent { return t.messages }
func (t *testCaseCompletion) Tools() []llms.Tool { return nil }
++func (t *testCaseCompletion) Capability() TestCapability { return t.def.Capability }
++
++// ExtraOptions is always nil for completion tests: CapabilityAdaptiveThinking
++// and CapabilityReasoningOff are gated (see tester.capabilitySupported) to
++// only run when the agent's own static config already requests that wire
++// behavior, so the plain CallWithTools/CallEx path — the same one PentAGI's
++// runtime uses — produces it naturally. Forcing it here on top would test a
++// CallOption combination the real app never sends.
++func (t *testCaseCompletion) ExtraOptions() []llms.CallOption { return nil }
func (t *testCaseCompletion) StreamingCallback() streaming.Callback {
if !t.def.Streaming {
@@@ -126,6 -126,6 +135,20 @@@ func (t *testCaseCompletion) Execute(re
result.Success = success
if !success {
result.Error = fmt.Errorf("expected text '%s' not found", t.expected)
++ return result
++ }
++
++ // for capability tests (adaptive thinking / reasoning off), the content
++ // match alone isn't enough: verify the capability actually took effect on
++ // the wire, since a model that silently ignores the request would still
++ // answer correctly while telling us nothing useful about the capability.
++ if t.def.RequireReasoning != nil && *t.def.RequireReasoning != hasReasoning {
++ result.Success = false
++ if *t.def.RequireReasoning {
++ result.Error = fmt.Errorf("expected reasoning content for capability %q, but none was returned", t.def.Capability)
++ } else {
++ result.Error = fmt.Errorf("expected no reasoning content for capability %q, but reasoning was returned", t.def.Capability)
++ }
}
return result
diff --cc backend/pkg/providers/tester/testdata/json.go
index 20aafa0f,20aafa0f..e258c0da
--- a/backend/pkg/providers/tester/testdata/json.go
+++ b/backend/pkg/providers/tester/testdata/json.go
@@@ -16,11 -16,11 +16,12 @@@ type testCaseJSON struct
def TestDefinition
// state for streaming and response collection
-- mu sync.Mutex
-- content strings.Builder
-- reasoning strings.Builder
-- messages []llms.MessageContent
-- expected map[string]any
++ mu sync.Mutex
++ content strings.Builder
++ reasoning strings.Builder
++ messages []llms.MessageContent
++ expected map[string]any
++ structuredOutput *llms.StructuredOutputConfig
}
func newJSONTestCase(def TestDefinition) (TestCase, error) {
@@@ -40,10 -40,10 +41,26 @@@
return nil, fmt.Errorf("failed to convert messages: %v", err)
}
++ var structuredOutput *llms.StructuredOutputConfig
++ if def.Capability == CapabilityStructuredOutput {
++ if def.Schema == nil {
++ return nil, fmt.Errorf("structured_output test %q requires a schema", def.ID)
++ }
++ schemaBytes, err := json.Marshal(def.Schema)
++ if err != nil {
++ return nil, fmt.Errorf("failed to marshal schema for test %q: %v", def.ID, err)
++ }
++ structuredOutput = &llms.StructuredOutputConfig{
++ Name: def.SchemaName,
++ Schema: schemaBytes,
++ }
++ }
++
return &testCaseJSON{
-- def: def,
-- expected: expected,
-- messages: messages,
++ def: def,
++ expected: expected,
++ messages: messages,
++ structuredOutput: structuredOutput,
}, nil
}
@@@ -55,6 -55,6 +72,21 @@@ func (t *testCaseJSON) Streaming() boo
func (t *testCaseJSON) Prompt() string { return "" }
func (t *testCaseJSON) Messages() []llms.MessageContent { return t.messages }
func (t *testCaseJSON) Tools() []llms.Tool { return nil }
++func (t *testCaseJSON) Capability() TestCapability { return t.def.Capability }
++
++// ExtraOptions attaches the per-call JSON Schema for a CapabilityStructuredOutput
++// test. AgentConfig.BuildOptions doesn't wire llms.WithStructuredOutput into
++// simple_json yet, but that's landing soon, so this call is forced here ahead
++// of time to confirm the LLM/backend can actually honor schema-constrained
++// output before the real integration depends on it. llms.WithStructuredOutput
++// also implies JSONMode, so the agent's own `json: true` config setting (if
++// present) is redundant but harmless.
++func (t *testCaseJSON) ExtraOptions() []llms.CallOption {
++ if t.structuredOutput == nil {
++ return nil
++ }
++ return []llms.CallOption{llms.WithStructuredOutput(*t.structuredOutput)}
++}
func (t *testCaseJSON) StreamingCallback() streaming.Callback {
if !t.def.Streaming {
diff --cc backend/pkg/providers/tester/testdata/models.go
index aca604f7,aca604f7..fb3183fd
--- a/backend/pkg/providers/tester/testdata/models.go
+++ b/backend/pkg/providers/tester/testdata/models.go
@@@ -27,6 -27,6 +27,38 @@@ const
TestGroupKnowledge TestGroup = "knowledge"
)
++// TestCapability names a wire capability a test case exercises. A test with a
++// non-empty Capability only runs against an agent whose ACTUAL loaded config
++// would make this exact call in a real PentAGI flow (see
++// tester.capabilitySupported, which mirrors pconfig.AgentConfig.BuildOptions /
++// ProviderConfig.UsesAdaptiveThinking exactly) — ctester answers "will this
++// configuration work inside PentAGI", so testing a CallOption combination the
++// app's runtime would never actually send for this config would produce a
++// pass or fail that says nothing about a flow that will ever really happen.
++type TestCapability string
++
++const (
++ // CapabilityNone is the default: the test runs unconditionally for every
++ // agent/model, exactly like the pre-existing basic/advanced/knowledge tests.
++ CapabilityNone TestCapability = ""
++ // CapabilityAdaptiveThinking exercises llms.WithAdaptiveReasoning, gated to
++ // agents whose config actually triggers it in production: explicit
++ // `reasoning: {mode: adaptive}`, or an adaptive-only model (Claude Opus
++ // 4.7/4.8), which PrepareAdaptiveCallOptions forces on regardless of config.
++ CapabilityAdaptiveThinking TestCapability = "adaptive_thinking"
++ // CapabilityReasoningOff exercises llms.WithReasoningDisabled, gated to
++ // agents whose config explicitly sets `reasoning: {mode: off}` — the only
++ // case where AgentConfig.BuildOptions actually emits it.
++ CapabilityReasoningOff TestCapability = "reasoning_off"
++ // CapabilityStructuredOutput exercises llms.WithStructuredOutput with the
++ // test's Schema. Deliberately run ahead of AgentConfig.BuildOptions
++ // actually wiring it in: that integration is landing soon for the
++ // simple_json agent, so this validates the LLM/backend can honor
++ // schema-constrained output before the real call path depends on it. Only
++ // ever scheduled for OptionsTypeSimpleJSON (see isTestCompatibleWithAgent).
++ CapabilityStructuredOutput TestCapability = "structured_output"
++)
++
// MessagesData represents a collection of message data with conversion capabilities
type MessagesData []MessageData
@@@ -99,15 -99,15 +131,30 @@@ func (md MessagesData) ToMessageContent
// TestDefinition represents immutable test configuration from YAML
type TestDefinition struct {
-- ID string `yaml:"id"`
-- Name string `yaml:"name"`
-- Type TestType `yaml:"type"`
-- Group TestGroup `yaml:"group"`
-- Prompt string `yaml:"prompt,omitempty"`
-- Messages MessagesData `yaml:"messages,omitempty"`
-- Tools []ToolData `yaml:"tools,omitempty"`
-- Expected any `yaml:"expected"`
-- Streaming bool `yaml:"streaming"`
++ ID string `yaml:"id"`
++ Name string `yaml:"name"`
++ Type TestType `yaml:"type"`
++ Group TestGroup `yaml:"group"`
++ Prompt string `yaml:"prompt,omitempty"`
++ Messages MessagesData `yaml:"messages,omitempty"`
++ Tools []ToolData `yaml:"tools,omitempty"`
++ Expected any `yaml:"expected"`
++ Streaming bool `yaml:"streaming"`
++ Capability TestCapability `yaml:"capability,omitempty"`
++
++ // RequireReasoning, when non-nil, asserts the presence (true) or absence
++ // (false) of reasoning content in the response — used by
++ // CapabilityAdaptiveThinking (must reason) and CapabilityReasoningOff
++ // (must NOT reason) tests to verify the capability actually took effect on
++ // the wire, not just that the call happened not to error.
++ RequireReasoning *bool `yaml:"require_reasoning,omitempty"`
++
++ // Schema and SchemaName configure a CapabilityStructuredOutput json test:
++ // Schema is the raw JSON Schema document (any YAML-decoded value, marshaled
++ // to JSON when building the test case) passed verbatim to
++ // llms.WithStructuredOutput.
++ Schema any `yaml:"schema,omitempty"`
++ SchemaName string `yaml:"schema_name,omitempty"`
}
type MessageData struct {
@@@ -154,6 -154,6 +201,24 @@@ type TestCase interface
Tools() []llms.Tool
StreamingCallback() streaming.Callback
++ // Capability reports the wire behavior this test is meant to exercise
++ // (CapabilityNone for every pre-existing test type). The runner uses it
++ // to gate whether this agent's ACTUAL config would make this exact call
++ // in production before spending a request on it (see
++ // tester.capabilitySupported): Capability is informational/gating only,
++ // it does not by itself force anything onto the wire.
++ Capability() TestCapability
++ // ExtraOptions returns extra llms.CallOption values to layer on top of
++ // the agent's configured options. Nil for CapabilityAdaptiveThinking/
++ // CapabilityReasoningOff: those are gated to configs that already produce
++ // that behavior via the plain CallWithTools/CallEx path, so forcing it
++ // here would be redundant at best and, for a config that doesn't
++ // naturally produce it, would test a call PentAGI's runtime never makes.
++ // Non-nil only for CapabilityStructuredOutput, which forces
++ // llms.WithStructuredOutput ahead of AgentConfig.BuildOptions wiring it in
++ // for real — see CapabilityStructuredOutput's doc comment.
++ ExtraOptions() []llms.CallOption
++
// result validation and state management
Execute(response any, latency time.Duration) TestResult
}
diff --cc backend/pkg/providers/tester/testdata/result.go
index ad0d9018,ad0d9018..4b5571a2
--- a/backend/pkg/providers/tester/testdata/result.go
+++ b/backend/pkg/providers/tester/testdata/result.go
@@@ -4,13 -4,13 +4,22 @@@ import "time
// TestResult represents the result of a single test execution
type TestResult struct {
-- ID string `json:"id"`
-- Name string `json:"name"`
-- Type TestType `json:"type"`
-- Group TestGroup `json:"group"`
-- Success bool `json:"success"`
-- Error error `json:"error"`
-- Streaming bool `json:"streaming"`
-- Reasoning bool `json:"reasoning"`
-- Latency time.Duration `json:"latency"`
++ ID string `json:"id"`
++ Name string `json:"name"`
++ Type TestType `json:"type"`
++ Group TestGroup `json:"group"`
++ Capability TestCapability `json:"capability,omitempty"`
++ Success bool `json:"success"`
++ // Unsupported reports that the provider/model proactively rejected the
++ // requested capability with a typed SDK error (ErrStructuredOutputUnsupported,
++ // ErrStructuredOutputConflict, reasoning.ErrReasoningOffUnsupported) rather
++ // than the call failing for an unrelated reason. It is distinct from a hard
++ // failure: the config is fine, this particular optional capability simply
++ // isn't available on this model, which is itself a useful production
++ // readiness signal rather than a bug in the tested configuration.
++ Unsupported bool `json:"unsupported,omitempty"`
++ Error error `json:"error"`
++ Streaming bool `json:"streaming"`
++ Reasoning bool `json:"reasoning"`
++ Latency time.Duration `json:"latency"`
}
diff --cc backend/pkg/providers/tester/testdata/tests.yml
index 8ed19043,8ed19043..d09efaef
--- a/backend/pkg/providers/tester/testdata/tests.yml
+++ b/backend/pkg/providers/tester/testdata/tests.yml
@@@ -89,6 -89,6 +89,72 @@@
active: true
streaming: false
++# Structured Output tests (llms.WithStructuredOutput) - provider-native
++# schema-constrained decoding, distinct from the plain best-effort JSON-mode
++# tests above. Gated by capability: structured_output (optimistic - attempted
++# on every provider, a proactive ErrStructuredOutputUnsupported/Conflict is
++# reported as Unsupported rather than a hard failure). Runs ahead of
++# AgentConfig.BuildOptions actually wiring this into simple_json (landing
++# soon) so the LLM/backend is already verified capable of it by the time that
++# integration ships.
++#
++# The prompt explicitly forbids markdown/code fences, matching how the SDK's
++# own passing structured-output tests are written for weak-compliance
++# backends (see langchaingo/llms/ollama/ollama_cloud_test.go's
++# TestCloudStructuredOutput): some backends only apply the schema as a soft
++# hint rather than a hard grammar constraint, so a model that isn't told not
++# to wrap its answer in ```json fences may do so out of habit even though it
++# received the schema. structuredoutput.ValidateText deliberately does not
++# strip markdown before validating (schema used verbatim, nothing stripped),
++# so a backend/model that still wraps output after this explicit instruction
++# is a genuine ErrStructuredOutputValidation failure, not a test artifact —
++# it means this configuration is not yet ready for real schema-constrained
++# output in production.
++- id: "structured_output_with_schema"
++ name: "Structured Output With JSON Schema"
++ type: "json"
++ group: "json"
++ capability: "structured_output"
++ schema_name: "person_info"
++ schema:
++ type: "object"
++ properties:
++ name:
++ type: "string"
++ age:
++ type: "integer"
++ city:
++ type: "string"
++ required: ["name", "age", "city"]
++ additionalProperties: false
++ messages:
++ - role: "system"
++ content: "You must respond only with valid JSON. No explanations, no markdown, no code fences — just the bare JSON value."
++ - role: "user"
++ content: |
++ Return a JSON with a person's information with correct types:
++ name='Alice Smith', age=28, city='Seattle'
++ expected:
++ name: "Alice Smith"
++ age: 28
++ city: "Seattle"
++ streaming: false
++
++# No-schema counterpart: NOT a duplicate of the plain-JSON tests above (which
++# already cover the no-schema/object-shaped case) - this instead exercises the
++# array-shaped response path via the empty/nil `expected` handling in
++# extractJSON/validateArgumentValue, which no other test in this file reaches.
++- id: "color_list_json_array_no_schema"
++ name: "JSON Array Response Without Schema"
++ type: "json"
++ group: "json"
++ messages:
++ - role: "system"
++ content: "You must respond only with valid JSON. No explanations or additional text."
++ - role: "user"
++ content: "Return a JSON array with exactly two color objects, each having 'name' and 'hex' fields: red (#FF0000) and blue (#0000FF)"
++ streaming: false
++
# Basic function calling tests
- id: "echo_function_basic"
name: "Basic Echo Function"
@@@ -654,6 -654,6 +720,45 @@@
expected: "example.com"
streaming: false
++# Reasoning capability tests - gated by capability so they only run against an
++# agent whose ACTUAL loaded config already triggers this wire behavior in a
++# real PentAGI flow (explicit reasoning:{mode: adaptive|off} in the provider
++# YAML, or - for adaptive_thinking - an adaptive-only model): see
++# tester.capabilitySupported. An agent whose config doesn't request it never
++# gets these requests at all: they are absent from the results rather than a
++# false pass/fail on a call PentAGI's runtime would never actually make.
++- id: "adaptive_thinking_reasoning_present"
++ name: "Adaptive Thinking Produces Reasoning"
++ type: "completion"
++ group: "advanced"
++ capability: "adaptive_thinking"
++ require_reasoning: true
++ messages:
++ - role: "system"
++ content: "You are a careful assistant that thinks through problems step by step before answering."
++ - role: "user"
++ content: |
++ A farmer has 17 sheep. All but 9 run away. He then buys back exactly
++ half of the sheep that ran away. How many sheep does the farmer have
++ now? Think it through, then give only the final number as your very
++ last line, with no other text after it.
++ expected: "13"
++ streaming: false
++
++- id: "reasoning_off_suppresses_reasoning"
++ name: "Explicit Reasoning Off Suppresses Reasoning"
++ type: "completion"
++ group: "advanced"
++ capability: "reasoning_off"
++ require_reasoning: false
++ messages:
++ - role: "system"
++ content: "You are a helpful assistant that follows instructions precisely."
++ - role: "user"
++ content: "What is the capital of France? Answer with only the city name, no other text."
++ expected: "Paris"
++ streaming: false
++
- id: "vulnerability_report_memory"
name: "Vulnerability Report Memory Test"
type: "json"
diff --cc backend/pkg/providers/tester/testdata/tool.go
index 41b51658,41b51658..b76d9cb5
--- a/backend/pkg/providers/tester/testdata/tool.go
+++ b/backend/pkg/providers/tester/testdata/tool.go
@@@ -90,6 -90,6 +90,8 @@@ func (t *testCaseTool) Streaming() boo
func (t *testCaseTool) Prompt() string { return "" }
func (t *testCaseTool) Messages() []llms.MessageContent { return t.messages }
func (t *testCaseTool) Tools() []llms.Tool { return t.tools }
++func (t *testCaseTool) Capability() TestCapability { return CapabilityNone }
++func (t *testCaseTool) ExtraOptions() []llms.CallOption { return nil }
func (t *testCaseTool) StreamingCallback() streaming.Callback {
if !t.def.Streaming {
diff --cc backend/pkg/server/router_test.go
index 00000000,c124772d..c124772d
mode 000000,100644..100644
--- a/backend/pkg/server/router_test.go
+++ b/backend/pkg/server/router_test.go
diff --cc backend/pkg/tools/executor.go
index 8320940c,a0fa02d0..f9385adc
--- a/backend/pkg/tools/executor.go
+++ b/backend/pkg/tools/executor.go
@@@ -299,9 -301,11 +299,12 @@@ func (ce *customExecutor) Execute
wrapHandler := func(ctx context.Context, name string, args json.RawMessage) (string, database.MsglogResultFormat, error) {
resultFormat := getMessageResultFormat(name)
result, err := handler(ctx, name, args)
++ persistCtx := context.WithoutCancel(ctx)
++
if err != nil {
durationDelta := time.Since(startTime).Seconds()
- _ = ce.tclp.UpdateLogFailed(ctx, tcID, fmt.Sprintf("failed to execute handler: %s", err.Error()), durationDelta)
+ failureResult := fmt.Sprintf("failed to execute handler: %s", err.Error())
- _ = ce.tclp.UpdateLogFailed(ctx, tcID, failureResult, durationDelta)
- ce.recordEvidenceReceipt(ctx, tcID, id, name, args, evidenceReceiptStatusFailed, failureResult)
++ _ = ce.tclp.UpdateLogFailed(persistCtx, tcID, failureResult, durationDelta)
return "", resultFormat, fmt.Errorf("failed to execute handler: %w", err)
}
@@@ -312,10 -316,12 +315,11 @@@
if err != nil {
return "", resultFormat, fmt.Errorf("failed to get summarize prompt: %w", err)
}
-- result, err = ce.summarizer(ctx, summarizePrompt)
++ result, err = ce.summarizer(persistCtx, summarizePrompt)
if err != nil {
durationDelta := time.Since(startTime).Seconds()
- _ = ce.tclp.UpdateLogFailed(ctx, tcID, fmt.Sprintf("failed to summarize result: %s", err.Error()), durationDelta)
+ failureResult := fmt.Sprintf("failed to summarize result: %s", err.Error())
- _ = ce.tclp.UpdateLogFailed(ctx, tcID, failureResult, durationDelta)
- ce.recordEvidenceReceipt(ctx, tcID, id, name, args, evidenceReceiptStatusFailed, failureResult)
++ _ = ce.tclp.UpdateLogFailed(persistCtx, tcID, failureResult, durationDelta)
return "", resultFormat, fmt.Errorf("failed to summarize result: %w", err)
}
resultFormat = database.MsglogResultFormatMarkdown
@@@ -330,7 -336,7 +334,7 @@@
}
durationDelta := time.Since(startTime).Seconds()
-- err = ce.tclp.UpdateLogSuccess(ctx, tcID, result, durationDelta)
++ err = ce.tclp.UpdateLogSuccess(persistCtx, tcID, result, durationDelta)
if err != nil {
return "", resultFormat, fmt.Errorf("failed to update toolcall result: %w", err)
}
diff --cc backend/pkg/tools/tools.go
index db6cc0e2,395034fd..1afa672e
--- a/backend/pkg/tools/tools.go
+++ b/backend/pkg/tools/tools.go
@@@ -371,8 -387,9 +386,8 @@@ func NewFlowToolsExecutor
db: db,
docker: docker,
functions: functions,
- replacer: replacer,
+ replacer: sharedReplacer,
cfg: cfg,
- evidence: newEvidenceReceiptRecorder(cfg.DataDir, flowID, cfg.EvidenceReceiptsEnabled),
flowID: flowID,
userID: userID,
definitions: make(map[string]llms.FunctionDefinition),
diff --cc docker-compose.yml
index df228237,efa21fbf..ff0b56cf
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@@ -55,7 -55,8 +55,7 @@@ services
- BEDROCK_SECRET_ACCESS_KEY=${BEDROCK_SECRET_ACCESS_KEY:-}
- BEDROCK_SESSION_TOKEN=${BEDROCK_SESSION_TOKEN:-}
- BEDROCK_SERVER_URL=${BEDROCK_SERVER_URL:-}
- - BEDROCK_CONFIG_PATH=${BEDROCK_CONFIG_PATH:+/opt/pentagi/conf/bedrock.provider.yml}
+ - BEDROCK_CONFIG_PATH=${BEDROCK_CONFIG_PATH:-}
- - BEDROCK_MODELS_PATH=${BEDROCK_MODELS_PATH:-}
- DEEPSEEK_API_KEY=${DEEPSEEK_API_KEY:-}
- DEEPSEEK_SERVER_URL=${DEEPSEEK_SERVER_URL:-}
- DEEPSEEK_PROVIDER=${DEEPSEEK_PROVIDER:-}
@@@ -184,7 -187,8 +187,7 @@@
- ${PENTAGI_DOCKER_SOCKET:-/var/run/docker.sock}:/var/run/docker.sock
- ${PENTAGI_LLM_SERVER_CONFIG_PATH:-./example.custom.provider.yml}:/opt/pentagi/conf/custom.provider.yml
- ${PENTAGI_OLLAMA_SERVER_CONFIG_PATH:-./example.ollama.provider.yml}:/opt/pentagi/conf/ollama.provider.yml
- - ${BEDROCK_CONFIG_PATH:-./glm_flash_bedrock.yml}:/opt/pentagi/conf/bedrock.provider.yml
- - ${PENTAGI_BEDROCK_CONFIG_PATH:-./examples/configs/bedrock-glm-flash.provider.yml}:/opt/pentagi/conf/bedrock.provider.yml
- - ${PENTAGI_BEDROCK_MODELS_PATH:-./examples/configs/bedrock-glm-flash.models.yml}:/opt/pentagi/conf/bedrock.models.yml
++ - ${BEDROCK_CONFIG_PATH:-./example.bedrock.provider.yml}:/opt/pentagi/conf/bedrock.provider.yml
- ${PENTAGI_DOCKER_CERT_PATH:-./docker-ssl}:/opt/pentagi/docker/ssl
user: root:root # while using docker.sock
networks:
diff --cc examples/configs/deepinfra.provider.yml
index 20e14ac0,20e14ac0..85ec3d86
--- a/examples/configs/deepinfra.provider.yml
+++ b/examples/configs/deepinfra.provider.yml
@@@ -1,119 -1,119 +1,173 @@@
++# DeepInfra config. Direct-hosted models only - excludes Partner-tagged entries
++# (Qwen3-Max-Thinking, Seed-1.6/2.0-*, MiniMax-M2.7-Turbo) and Google/Gemma
++# (guardrail false positives on offensive-security content).
++#
++# MiniMaxAI/MiniMax-M3 -> simple/reflector/searcher/enricher/adviser
++# moonshotai/Kimi-K2.6 -> primary_agent/assistant/pentester/simple_json
++# zai-org/GLM-5.2 -> generator/refiner (effort: xhigh only; "max" = HTTP 422)
++# moonshotai/Kimi-K2.7-Code -> coder/installer (occasional HTTP 429, transient)
++#
++# Dropped: DeepSeek-V4-flash (HTTP 400, no deployment), GLM-4.7-flash (hangs,
++# never errors). Kimi-K2.5 rejected for simple_json: cheaper but corrupts plain
++# JSON (turned "10.1.1.50" into 10.11). MiniMax-M3 rejects `thinking` (HTTP 400)
++# and `json_schema` (HTTP 405) - no reasoning fields on it, not used for
++# simple_json.
++#
++# Prices from DeepInfra's model API: $/1M = cents_per_token * 10000; cache_read
++# = input_price * rate_per_input_token_cached.
++#
++# Test: HTTP_CLIENT_TIMEOUT=30 ctester -config deepinfra.provider.yml -agents
++# <type> -workers 1 (default 600s timeout otherwise blocks on hanging models).
++
simple:
-- model: "Qwen/Qwen3-Next-80B-A3B-Instruct"
-- temperature: 0.7
++ model: "MiniMaxAI/MiniMax-M3"
++ temperature: 1.0
top_p: 0.95
++ top_k: 40
n: 1
-- max_tokens: 4000
++ max_tokens: 8192
price:
-- input: 0.14
-- output: 1.4
++ input: 0.30
++ output: 1.20
++ cache_read: 0.06
simple_json:
-- model: "Qwen/Qwen3-Next-80B-A3B-Instruct"
-- temperature: 0.7
-- top_p: 1.0
++ model: "moonshotai/Kimi-K2.6"
++ temperature: 0.6
++ top_p: 0.95
n: 1
-- max_tokens: 4000
++ max_tokens: 4096
json: true
price:
-- input: 0.14
-- output: 1.4
++ input: 0.75
++ output: 3.50
++ cache_read: 0.15
primary_agent:
-- model: "moonshotai/Kimi-K2-Instruct-0905"
++ model: "moonshotai/Kimi-K2.6"
temperature: 1.0
++ top_p: 0.95
n: 1
-- max_tokens: 6000
++ max_tokens: 16384
price:
-- input: 0.4
-- output: 2.0
++ input: 0.75
++ output: 3.50
++ cache_read: 0.15
assistant:
-- model: "moonshotai/Kimi-K2-Instruct-0905"
++ model: "moonshotai/Kimi-K2.6"
temperature: 1.0
++ top_p: 0.95
n: 1
-- max_tokens: 8000
++ max_tokens: 16384
price:
-- input: 0.4
-- output: 2.0
++ input: 0.75
++ output: 3.50
++ cache_read: 0.15
generator:
-- model: "google/gemini-2.5-pro"
++ model: "zai-org/GLM-5.2"
temperature: 1.0
++ top_p: 0.95
n: 1
-- max_tokens: 8000
++ max_tokens: 32768
++ reasoning:
++ effort: xhigh
price:
-- input: 1.25
-- output: 10.0
++ input: 0.93
++ output: 3.00
++ cache_read: 0.18
refiner:
-- model: "deepseek-ai/DeepSeek-R1-0528-Turbo"
++ model: "zai-org/GLM-5.2"
temperature: 1.0
++ top_p: 0.95
n: 1
-- max_tokens: 8000
++ max_tokens: 32768
++ reasoning:
++ effort: xhigh
price:
-- input: 1.0
-- output: 3.0
++ input: 0.93
++ output: 3.00
++ cache_read: 0.18
adviser:
-- model: "google/gemini-2.5-pro"
++ model: "MiniMaxAI/MiniMax-M3"
temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
-- max_tokens: 4000
++ max_tokens: 16384
price:
-- input: 1.25
-- output: 10.0
++ input: 0.30
++ output: 1.20
++ cache_read: 0.06
reflector:
-- model: "Qwen/Qwen3-Next-80B-A3B-Instruct"
++ model: "MiniMaxAI/MiniMax-M3"
temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
-- max_tokens: 4000
++ max_tokens: 4096
price:
-- input: 0.14
-- output: 1.4
++ input: 0.30
++ output: 1.20
++ cache_read: 0.06
searcher:
-- model: "Qwen/Qwen3-32B"
++ model: "MiniMaxAI/MiniMax-M3"
temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
-- max_tokens: 4000
++ max_tokens: 4096
price:
-- input: 0.1
-- output: 0.3
++ input: 0.30
++ output: 1.20
++ cache_read: 0.06
enricher:
-- model: "Qwen/Qwen3-32B"
++ model: "MiniMaxAI/MiniMax-M3"
temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
-- max_tokens: 6000
++ max_tokens: 8192
price:
-- input: 0.1
-- output: 0.3
++ input: 0.30
++ output: 1.20
++ cache_read: 0.06
coder:
-- model: "anthropic/claude-4-sonnet"
++ model: "moonshotai/Kimi-K2.7-Code"
temperature: 1.0
++ top_p: 0.95
n: 1
-- max_tokens: 8000
++ max_tokens: 20480
price:
-- input: 3.3
-- output: 16.5
++ input: 0.74
++ output: 3.50
++ cache_read: 0.15
installer:
-- model: "google/gemini-2.5-flash"
++ model: "moonshotai/Kimi-K2.7-Code"
temperature: 1.0
++ top_p: 0.95
n: 1
-- max_tokens: 6000
++ max_tokens: 16384
price:
-- input: 0.3
-- output: 2.5
++ input: 0.74
++ output: 3.50
++ cache_read: 0.15
pentester:
-- model: "moonshotai/Kimi-K2-Instruct-0905"
++ model: "moonshotai/Kimi-K2.6"
temperature: 1.0
++ top_p: 0.95
n: 1
-- max_tokens: 6000
++ max_tokens: 16384
price:
-- input: 0.4
-- output: 2.0
++ input: 0.75
++ output: 3.50
++ cache_read: 0.15
diff --cc examples/configs/nvidia-glm-5.1.provider.yml
index 00000000,00000000..d8659fe1
new file mode 100644
--- /dev/null
+++ b/examples/configs/nvidia-glm-5.1.provider.yml
@@@ -1,0 -1,0 +1,171 @@@
++# NVIDIA NIM Provider Configuration — z-ai/glm-5.1 (GLM-5.1)
++# Endpoint: https://integrate.api.nvidia.com/v1 (OpenAI-compatible)
++#
++# Intended environment:
++# LLM_SERVER_URL=https://integrate.api.nvidia.com/v1
++# LLM_SERVER_KEY=nvapi-...
++# LLM_SERVER_MODEL= # left empty: model is taken from this file
++# LLM_SERVER_PROVIDER= # left empty: no model-name prefix filtering
++# LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/custom.provider.yml
++# LLM_SERVER_LEGACY_REASONING=true # GLM emits inline <think> reasoning
++# LLM_SERVER_PRESERVE_REASONING=true # keep reasoning content in history
++#
++# GLM-5.1 facts (from the model card):
++# - MoE, 754B params, agentic engineering / long-horizon tool use
++# - Input/Output context length: 131,072 tokens
++#
++# Reasoning ("thinking") on this endpoint is NOT controlled by the OpenAI
++# `reasoning_effort` field, but by the GLM chat template flags passed through
++# `extra_body.chat_template_kwargs`:
++# - enable_thinking: true|false — turn the thinking phase on/off
++# - clear_thinking: false — keep the reasoning trace (pairs with
++# LLM_SERVER_PRESERVE_REASONING=true)
++#
++# Only OpenAI-compatible sampling fields are used (temperature, top_p, n,
++# max_tokens). vLLM-only fields (top_k, min_p, repetition_penalty) are
++# intentionally omitted: the NVIDIA OpenAI-compatible API does not need them
++# and the confirmed-working request used only temperature + top_p.
++#
++# Sampling profiles:
++# - reasoning / general tasks: temperature=1.0, top_p=0.95
++# - precise coding tasks: temperature=0.6, top_p=0.95
++# - lightweight non-thinking: temperature=0.6, top_p=0.95
++
++simple:
++ model: "z-ai/glm-5.1"
++ temperature: 0.6
++ top_p: 0.95
++ n: 1
++ max_tokens: 16384
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: false
++
++simple_json:
++ model: "z-ai/glm-5.1"
++ temperature: 0.6
++ top_p: 0.95
++ n: 1
++ max_tokens: 16384
++ json: true
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: false
++
++primary_agent:
++ model: "z-ai/glm-5.1"
++ temperature: 1.0
++ top_p: 0.95
++ n: 1
++ max_tokens: 16384
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: true
++ clear_thinking: false
++
++assistant:
++ model: "z-ai/glm-5.1"
++ temperature: 1.0
++ top_p: 0.95
++ n: 1
++ max_tokens: 16384
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: true
++ clear_thinking: false
++
++generator:
++ model: "z-ai/glm-5.1"
++ temperature: 1.0
++ top_p: 0.95
++ n: 1
++ max_tokens: 16384
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: true
++ clear_thinking: false
++
++refiner:
++ model: "z-ai/glm-5.1"
++ temperature: 1.0
++ top_p: 0.95
++ n: 1
++ max_tokens: 16384
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: true
++ clear_thinking: false
++
++adviser:
++ model: "z-ai/glm-5.1"
++ temperature: 1.0
++ top_p: 0.95
++ n: 1
++ max_tokens: 16384
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: true
++ clear_thinking: false
++
++reflector:
++ model: "z-ai/glm-5.1"
++ temperature: 0.6
++ top_p: 0.95
++ n: 1
++ max_tokens: 16384
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: false
++
++searcher:
++ model: "z-ai/glm-5.1"
++ temperature: 0.6
++ top_p: 0.95
++ n: 1
++ max_tokens: 16384
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: false
++
++enricher:
++ model: "z-ai/glm-5.1"
++ temperature: 0.6
++ top_p: 0.95
++ n: 1
++ max_tokens: 16384
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: false
++
++coder:
++ model: "z-ai/glm-5.1"
++ temperature: 0.6
++ top_p: 0.95
++ n: 1
++ max_tokens: 16384
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: true
++ clear_thinking: false
++
++installer:
++ model: "z-ai/glm-5.1"
++ temperature: 0.6
++ top_p: 0.95
++ n: 1
++ max_tokens: 16384
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: true
++ clear_thinking: false
++
++pentester:
++ model: "z-ai/glm-5.1"
++ temperature: 0.6
++ top_p: 0.95
++ n: 1
++ max_tokens: 16384
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: true
++ clear_thinking: false
diff --cc examples/configs/ollama-cloud.provider.yml
index f507ee04,f507ee04..bc23dfd6
--- a/examples/configs/ollama-cloud.provider.yml
+++ b/examples/configs/ollama-cloud.provider.yml
@@@ -1,93 -1,93 +1,145 @@@
++# Ollama Cloud agent configuration.
++#
++# Ollama Cloud is a flat-rate subscription (no per-token billing), so model
++# selection here is driven purely by capability fit per role, not price - unlike
++# the OpenRouter/native provider configs in this directory, there is no `price:`
++# block and no need to trade quality for cost on any agent.
++#
++# Excluded from the available model list on purpose: `gemini-3-flash-preview` and
++# `gpt-oss:120b`/`gpt-oss:20b`. This product runs cybersecurity/pentesting
++# workloads, and Google/OpenAI-trained models (hosted or open-weight) carry
++# built-in safety alignment prone to false-positive refusals on legitimate
++# exploit-development and offensive security content - the same reasoning applied
++# when curating examples/configs/openrouter.provider.yml.
++#
++# Role assignments (from the models actually offered on this Ollama Cloud
++# account):
++# - nemotron-3-nano:30b : simple/simple_json/reflector - NVIDIA's smallest
++# Nemotron-3 tier (30B/3B-active MoE), used for fast, low-latency utility
++# calls (docker image selection, tool-call retry nudging).
++# - qwen3.5:397b : searcher - largest available Qwen3.5 tier, giving the
++# most context/recall headroom for information-gathering and query
++# construction.
++# - minimax-m2.7 : enricher - smallest/fastest MiniMax M-series tier,
++# sufficient for cheap note enrichment with low latency.
++# - kimi-k2.6 : primary_agent/assistant - Moonshot's latest flagship
++# multimodal model, described by Moonshot as having "improved instruction
++# compliance and self-correction" - a direct fit for a reliable long-chain
++# tool-calling orchestration loop.
++# - kimi-k2.7-code : coder/installer - coding-specialized Kimi variant with
++# strong long-context programming performance; also covers installer's
++# environment/build-script setup tasks.
++# - glm-5.1 : adviser - Z.AI's own positioning for this model is
++# "designed for long-horizon tasks... best for planning, mentor, and complex
++# agentic engineering" - a literal match for the adviser/mentor role, which is
++# invoked frequently (execution monitor + planning).
++# - mistral-large-3:675b: generator - Mistral's largest available dense model,
++# used for the highest-stakes step in a flow (initial attack-plan generation).
++# - nemotron-3-ultra : refiner - NVIDIA's largest Nemotron-3 tier (550B/55B-
++# active MoE), used to review/refine the plan generator produced; a different
++# architecture (MoE vs. dense) from generator's model to catch different
++# classes of planning errors.
++# - deepseek-v4-pro : pentester - "higher-tier reasoning... security
++# analysis" per its own description; pentester is the most frequently invoked
++# agent in a flow, so a well-balanced (not maximal) MoE flagship is used to
++# keep per-call latency reasonable.
++#
++# Reasoning/thinking control note: this file intentionally omits the `reasoning:`
++# field entirely. For the native Ollama provider (pkg/providers/ollama, backed by
++# langchaingo's llms/ollama client, not openaicompat), `reasoning:` is a no-op -
++# llms/ollama's createChatRequest/makeOllamaOptionsFromOptions never reads
++# opts.Reasoning; "reasoning" support in that client is purely passive parsing of
++# `<thinking>` tags already present in a model's response text, not an active
++# request parameter. Each model's own hybrid-thinking default is used as-is.
++
simple:
-- model: "nemotron-3-super:cloud"
++ model: "nemotron-3-nano:30b-cloud"
temperature: 1
top_p: 0.9
n: 1
-- max_tokens: 65536
++ max_tokens: 8192
simple_json:
-- model: "nemotron-3-super:cloud"
++ model: "nemotron-3-nano:30b-cloud"
temperature: 1
-- top_p: 0.95
++ top_p: 0.9
n: 1
-- max_tokens: 65536
++ max_tokens: 8192
json: true
primary_agent:
-- model: "qwen3-coder-next:cloud"
++ model: "kimi-k2.6:cloud"
temperature: 1
-- top_p: 0.9
++ top_p: 0.95
n: 1
-- max_tokens: 32768
-- reasoning:
-- effort: high
++ max_tokens: 16384
assistant:
-- model: "nemotron-3-super:cloud"
++ model: "kimi-k2.6:cloud"
temperature: 1
-- top_p: 0.9
++ top_p: 0.95
n: 1
-- max_tokens: 65536
++ max_tokens: 16384
generator:
-- model: "qwen3-coder-next:cloud"
++ model: "mistral-large-3:675b-cloud"
temperature: 1
top_p: 0.95
n: 1
max_tokens: 32768
refiner:
-- model: "glm-5:cloud"
++ model: "nemotron-3-ultra:cloud"
temperature: 1
top_p: 0.95
n: 1
-- max_tokens: 65536
++ max_tokens: 32768
adviser:
-- model: "minimax-m2.7:cloud"
++ model: "glm-5.1:cloud"
temperature: 1
top_p: 0.95
n: 1
-- max_tokens: 65536
++ max_tokens: 16384
reflector:
-- model: "glm-5:cloud"
++ model: "nemotron-3-nano:30b-cloud"
temperature: 1
-- top_p: 0.95
++ top_p: 0.9
n: 1
-- max_tokens: 65536
++ max_tokens: 8192
searcher:
model: "qwen3.5:397b-cloud"
temperature: 1
top_p: 0.95
n: 1
-- max_tokens: 65536
++ max_tokens: 8192
enricher:
model: "minimax-m2.7:cloud"
temperature: 1
top_p: 0.95
n: 1
-- max_tokens: 65536
++ max_tokens: 4096
coder:
-- model: "qwen3-coder-next:cloud"
++ model: "kimi-k2.7-code:cloud"
temperature: 1
top_p: 0.95
n: 1
-- max_tokens: 32768
++ max_tokens: 20480
installer:
-- model: "devstral-2:123b-cloud"
++ model: "kimi-k2.7-code:cloud"
temperature: 1
-- top_p: 0.9
++ top_p: 0.95
n: 1
-- max_tokens: 65536
++ max_tokens: 16384
pentester:
-- model: "qwen3-coder-next:cloud"
++ model: "deepseek-v4-pro:cloud"
temperature: 1
top_p: 0.95
n: 1
-- max_tokens: 32768
++ max_tokens: 16384
diff --cc examples/configs/openrouter.provider.yml
index cb35056b,cb35056b..568228b1
--- a/examples/configs/openrouter.provider.yml
+++ b/examples/configs/openrouter.provider.yml
@@@ -1,128 -1,128 +1,214 @@@
++# OpenRouter agent configuration.
++#
++# Strategy: route every agent through cheap/strong open-weight models available on
++# OpenRouter, deliberately avoiding Google/Anthropic/OpenAI models. This product
++# runs cybersecurity/pentesting workloads, and those vendors' safety guardrails are
++# prone to false-positive refusals on legitimate exploit-development and offensive
++# security content. Models are drawn from 4 families (DeepSeek, Z.AI/GLM, MiniMax,
++# Moonshot/Kimi) chosen per-agent for the best price/intelligence fit for that role,
++# not forced onto a single vendor:
++# - deepseek-v4-flash : utility tier (simple/simple_json/reflector/searcher) and
++# pentester (kept on the flash tier for cost since it's the most frequently
++# invoked agent in a flow, but with thinking left on for tool-call quality).
++# - z-ai/glm-5-turbo : primary_agent/assistant - vendor-described as "OpenClaw-
++# native... optimized for tool invocation... long-chain execution", a direct
++# match for the main orchestration loop.
++# - z-ai/glm-5.2 : generator/refiner - newest GLM flagship, explicitly
++# supports reasoning_effort (high/max); used at max effort for plan generation
++# and refinement, the most consequential decisions in a flow.
++# - minimax/minimax-m3 : adviser/enricher - latest MiniMax flagship with adaptive
++# thinking (toggled via extra_body.thinking.type, see below). Used at adaptive
++# depth for adviser (mentor/planner, invoked frequently: execution monitor +
++# planning) and with thinking disabled for enricher (cheap note enrichment,
++# no deep reasoning needed).
++# - moonshotai/kimi-k2.7-code : coder/installer - coding-specialized model with
++# strong long-context programming performance; also covers installer's
++# environment/build-script setup tasks, keeping the model roster small.
++#
++# Reasoning control notes:
++# - `reasoning: {mode: off}` (generic field, no extra_body) is used for the true
++# fast/deterministic utility agents on deepseek-v4-flash; it maps to a legacy
++# `reasoning_effort:"none"` on the wire (verified in langchaingo's
++# reasoning.ResolveOff/IsReasoningModel against this repo's pinned langchaingo
++# version).
++# - `reasoning: {effort: ...}` is only set where the model's own catalog
++# (see z-ai/glm/models.yml) documents graduated effort support (glm-5.2).
++# glm-5-turbo and kimi-k2.7-code do NOT document reasoning_effort support
++# (binary thinking toggle only per their native provider configs), so
++# `reasoning` is left unset for them and their own hybrid-thinking default
++# (thinking on) is used instead.
++# - minimax-m3's thinking depth is NOT controlled via the generic `reasoning`
++# field: MiniMax exposes an adaptive-thinking toggle only through
++# `extra_body.thinking.type` (adaptive/disabled), which OpenRouter forwards
++# verbatim to the MiniMax backend.
++# - DeepSeek agents with `reasoning.effort` set omit temperature/top_p: DeepSeek
++# silently ignores sampling params in thinking mode (same constraint as the
++# native deepseek/config.yml).
++#
++# Validated with `ctester -config openrouter.provider.yml -agents all`: 287/287
++# tests passing (100%) across all 13 agent types as of 2026-07-21. Re-validate
++# after any model/parameter change, since OpenRouter's routing behavior for
++# non-OpenAI reasoning models can shift with backend updates.
++
simple:
-- model: "openai/gpt-4.1-mini"
-- temperature: 0.6
-- top_p: 0.95
++ model: "deepseek/deepseek-v4-flash"
++ temperature: 1.0
n: 1
-- max_tokens: 4000
++ max_tokens: 8192
++ reasoning:
++ mode: off
price:
-- input: 0.4
-- output: 1.6
++ input: 0.14
++ output: 0.28
++ cache_read: 0.0028
simple_json:
-- model: "openai/gpt-4.1-mini"
-- temperature: 0.7
-- top_p: 1.0
++ model: "deepseek/deepseek-v4-flash"
++ temperature: 1.0
n: 1
-- max_tokens: 4000
++ max_tokens: 4096
json: true
++ reasoning:
++ mode: off
price:
-- input: 0.4
-- output: 1.6
++ input: 0.14
++ output: 0.28
++ cache_read: 0.0028
primary_agent:
-- model: "openai/gpt-5" # x-ai/grok-3-mini, openai/o4-mini
++ model: "z-ai/glm-5-turbo"
++ temperature: 1.0
++ top_p: 0.95
n: 1
-- max_tokens: 6000
-- reasoning:
-- effort: medium
++ max_tokens: 16384
price:
-- input: 1.25
-- output: 10.0
++ input: 1.20
++ output: 4.00
++ cache_read: 0.24
assistant:
-- model: "openai/gpt-5" # x-ai/grok-4, x-ai/grok-3-mini, google/gemini-2.5-flash, openai/o4-mini
++ model: "z-ai/glm-5-turbo"
++ temperature: 1.0
++ top_p: 0.95
n: 1
-- max_tokens: 6000
-- reasoning:
-- effort: medium
++ max_tokens: 16384
price:
-- input: 1.25
-- output: 10.0
++ input: 1.20
++ output: 4.00
++ cache_read: 0.24
generator:
-- model: "anthropic/claude-sonnet-4.5"
++ model: "z-ai/glm-5.2"
++ temperature: 1.0
++ top_p: 0.95
n: 1
-- max_tokens: 12000
++ max_tokens: 32768
reasoning:
-- max_tokens: 4000
++ effort: max
price:
-- input: 3.0
-- output: 15.0
++ input: 1.40
++ output: 4.40
++ cache_read: 0.26
refiner:
-- model: "google/gemini-2.5-pro"
++ model: "z-ai/glm-5.2"
++ temperature: 1.0
++ top_p: 0.95
n: 1
-- max_tokens: 10000
++ max_tokens: 32768
reasoning:
-- effort: medium
++ effort: max
price:
-- input: 1.25
-- output: 10.0
++ input: 1.40
++ output: 4.40
++ cache_read: 0.26
adviser:
-- model: "google/gemini-2.5-pro"
++ model: "minimax/minimax-m3"
++ temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
-- max_tokens: 6000
-- reasoning:
-- effort: high
++ max_tokens: 16384
++ extra_body:
++ thinking:
++ type: adaptive
price:
-- input: 1.25
-- output: 10.0
++ input: 0.30
++ output: 1.20
++ cache_read: 0.06
reflector:
-- model: "openai/gpt-4.1-mini"
-- temperature: 0.8
-- top_p: 1.0
++ model: "deepseek/deepseek-v4-flash"
++ temperature: 1.0
n: 1
-- max_tokens: 4000
++ max_tokens: 4096
++ reasoning:
++ mode: off
price:
-- input: 0.4
-- output: 1.6
++ input: 0.14
++ output: 0.28
++ cache_read: 0.0028
searcher:
-- model: "x-ai/grok-3-mini"
++ model: "deepseek/deepseek-v4-flash"
++ temperature: 1.0
n: 1
-- max_tokens: 4000
++ max_tokens: 4096
reasoning:
-- max_tokens: 1024
++ mode: off
price:
-- input: 0.3
-- output: 0.5
++ input: 0.14
++ output: 0.28
++ cache_read: 0.0028
enricher:
-- model: "openai/gpt-4.1-mini"
-- temperature: 0.95
-- top_p: 1.0
++ model: "minimax/minimax-m3"
++ temperature: 1.0
++ top_p: 0.95
++ top_k: 40
n: 1
-- max_tokens: 6000
++ max_tokens: 8192
++ extra_body:
++ thinking:
++ type: disabled
++ reasoning:
++ mode: off
price:
-- input: 0.4
-- output: 1.6
++ input: 0.30
++ output: 1.20
++ cache_read: 0.06
coder:
-- model: "anthropic/claude-sonnet-4.5"
++ model: "moonshotai/kimi-k2.7-code"
++ temperature: 1.0
++ top_p: 0.95
n: 1
-- max_tokens: 8000
-- reasoning:
-- max_tokens: 2000
++ max_tokens: 20480
price:
-- input: 3.0
-- output: 15.0
++ input: 0.95
++ output: 4.00
++ cache_read: 0.19
installer:
-- model: "google/gemini-2.5-flash"
++ model: "moonshotai/kimi-k2.7-code"
++ temperature: 1.0
++ top_p: 0.95
n: 1
-- max_tokens: 4000
-- reasoning:
-- max_tokens: 1024
++ max_tokens: 16384
price:
-- input: 0.3
-- output: 2.5
++ input: 0.95
++ output: 4.00
++ cache_read: 0.19
pentester:
-- model: "moonshotai/kimi-k2-0905"
++ model: "deepseek/deepseek-v4-flash"
++ temperature: 1.0
n: 1
-- max_tokens: 6000
++ max_tokens: 16384
++ reasoning:
++ effort: high
price:
-- input: 0.4
-- output: 2.0
++ input: 0.14
++ output: 0.28
++ cache_read: 0.0028
diff --cc examples/configs/vllm-mixed.provider.yml
index 00000000,00000000..441966e1
new file mode 100644
--- /dev/null
+++ b/examples/configs/vllm-mixed.provider.yml
@@@ -1,0 -1,0 +1,163 @@@
++# Mixed vLLM Provider Configuration
++# Qwen3.6-27B-FP8 for lightweight / coding agents; DeepSeek-V4-Flash for planning & reasoning agents
++#
++# Qwen3.6-27B-FP8 (official Qwen recommendations for vLLM):
++# - Thinking mode enabled by default (no extra_body needed)
++# - General tasks: temp=1.0, top_p=0.95, top_k=20, min_p=0.0, pp=1.5, rp=1.0
++# - Precise coding: temp=0.6, top_p=0.95, top_k=20, min_p=0.0, pp=0.0, rp=1.0
++# - Non-thinking: temp=0.7, top_p=0.8 (+ enable_thinking: false via extra_body)
++#
++# DeepSeek-V4-Flash (official HF README for local deploy):
++# - Sampling: temperature=1.0, top_p=1.0 (unified; ignored when thinking is enabled)
++# - Default: thinking enabled, reasoning_effort=high (Think High)
++# - Think Max: reasoning_effort=max; context window should be >= 384K
++# - Control via reasoning.effort (top-level reasoning_effort) + extra_body.thinking
++
++simple:
++ model: "Qwen/Qwen3.6-27B-FP8"
++ temperature: 0.7
++ top_k: 20
++ top_p: 0.8
++ min_p: 0.0
++ presence_penalty: 1.5
++ repetition_penalty: 1.0
++ n: 1
++ max_tokens: 32768
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: false
++
++simple_json:
++ model: "Qwen/Qwen3.6-27B-FP8"
++ temperature: 0.7
++ top_k: 20
++ top_p: 0.8
++ min_p: 0.0
++ presence_penalty: 1.5
++ repetition_penalty: 1.0
++ n: 1
++ max_tokens: 32768
++ json: true
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: false
++
++primary_agent:
++ model: "DeepSeek-V4-Flash"
++ temperature: 1.0
++ top_p: 1.0
++ n: 1
++ max_tokens: 32768
++ reasoning:
++ effort: high
++
++assistant:
++ model: "DeepSeek-V4-Flash"
++ temperature: 1.0
++ top_p: 1.0
++ n: 1
++ max_tokens: 32768
++ reasoning:
++ effort: high
++
++generator:
++ model: "DeepSeek-V4-Flash"
++ temperature: 1.0
++ top_p: 1.0
++ n: 1
++ max_tokens: 32768
++ reasoning:
++ effort: max
++
++refiner:
++ model: "DeepSeek-V4-Flash"
++ temperature: 1.0
++ top_p: 1.0
++ n: 1
++ max_tokens: 32768
++ reasoning:
++ effort: max
++
++adviser:
++ model: "DeepSeek-V4-Flash"
++ temperature: 1.0
++ top_p: 1.0
++ n: 1
++ max_tokens: 32768
++ reasoning:
++ effort: max
++
++reflector:
++ model: "Qwen/Qwen3.6-27B-FP8"
++ temperature: 1.0
++ top_k: 20
++ top_p: 0.95
++ min_p: 0.0
++ presence_penalty: 1.5
++ repetition_penalty: 1.0
++ n: 1
++ max_tokens: 32768
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: false
++
++searcher:
++ model: "Qwen/Qwen3.6-27B-FP8"
++ temperature: 0.7
++ top_k: 20
++ top_p: 0.8
++ min_p: 0.0
++ presence_penalty: 1.5
++ repetition_penalty: 1.0
++ n: 1
++ max_tokens: 32768
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: false
++
++enricher:
++ model: "Qwen/Qwen3.6-27B-FP8"
++ temperature: 0.7
++ top_k: 20
++ top_p: 0.8
++ min_p: 0.0
++ presence_penalty: 1.5
++ repetition_penalty: 1.0
++ n: 1
++ max_tokens: 32768
++ extra_body:
++ chat_template_kwargs:
++ enable_thinking: false
++
++coder:
++ model: "Qwen/Qwen3.6-27B-FP8"
++ temperature: 0.6
++ top_k: 20
++ top_p: 0.95
++ min_p: 0.0
++ presence_penalty: 0.0
++ repetition_penalty: 1.0
++ n: 1
++ max_tokens: 32768
++
++installer:
++ model: "Qwen/Qwen3.6-27B-FP8"
++ temperature: 0.6
++ top_k: 20
++ top_p: 0.95
++ min_p: 0.0
++ presence_penalty: 0.0
++ repetition_penalty: 1.0
++ n: 1
++ max_tokens: 32768
++
++pentester:
++ model: "Qwen/Qwen3.6-27B-FP8"
++ temperature: 0.6
++ top_k: 20
++ top_p: 0.95
++ min_p: 0.0
++ presence_penalty: 0.0
++ repetition_penalty: 1.0
++ n: 1
++ max_tokens: 32768
diff --cc examples/tests/anthropic-report.md
index c13e1c82,c13e1c82..a598828e
--- a/examples/tests/anthropic-report.md
+++ b/examples/tests/anthropic-report.md
@@@ -1,27 -1,27 +1,27 @@@
# LLM Agent Testing Report
--Generated: Thu, 29 Jan 2026 17:36:55 UTC
++Generated: Tue, 21 Jul 2026 17:29:51 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
--| simple | claude-haiku-4-5 | false | 23/23 (100.00%) | 1.239s |
--| simple_json | claude-haiku-4-5 | false | 5/5 (100.00%) | 1.181s |
--| primary_agent | claude-sonnet-4-5 | true | 23/23 (100.00%) | 3.542s |
--| assistant | claude-sonnet-4-5 | true | 23/23 (100.00%) | 3.484s |
--| generator | claude-opus-4-5 | true | 22/23 (95.65%) | 3.806s |
--| refiner | claude-sonnet-4-5 | true | 23/23 (100.00%) | 3.512s |
--| adviser | claude-sonnet-4-5 | true | 23/23 (100.00%) | 3.846s |
--| reflector | claude-haiku-4-5 | true | 23/23 (100.00%) | 1.750s |
--| searcher | claude-haiku-4-5 | true | 23/23 (100.00%) | 2.005s |
--| enricher | claude-haiku-4-5 | true | 23/23 (100.00%) | 1.274s |
--| coder | claude-sonnet-4-5 | true | 23/23 (100.00%) | 3.591s |
--| installer | claude-sonnet-4-5 | true | 23/23 (100.00%) | 3.467s |
--| pentester | claude-sonnet-4-5 | true | 23/23 (100.00%) | 3.390s |
--
--**Total**: 280/281 (99.64%) successful tests
--**Overall average latency**: 2.878s
++| simple | claude-haiku-4-5 | false | 23/23 (100.00%) | 1.551s |
++| simple_json | claude-haiku-4-5 | false | 7/7 (100.00%) | 1.066s |
++| primary_agent | claude-sonnet-5 | true | 24/24 (100.00%) | 2.675s |
++| assistant | claude-sonnet-5 | true | 24/24 (100.00%) | 2.445s |
++| generator | claude-opus-4-8 | true | 24/24 (100.00%) | 3.121s |
++| refiner | claude-opus-4-8 | true | 24/24 (100.00%) | 3.083s |
++| adviser | claude-sonnet-5 | true | 24/24 (100.00%) | 2.793s |
++| reflector | claude-haiku-4-5 | true | 23/23 (100.00%) | 2.148s |
++| searcher | claude-haiku-4-5 | true | 23/23 (100.00%) | 2.070s |
++| enricher | claude-haiku-4-5 | true | 23/23 (100.00%) | 1.460s |
++| coder | claude-sonnet-5 | true | 24/24 (100.00%) | 2.304s |
++| installer | claude-sonnet-5 | true | 24/24 (100.00%) | 2.395s |
++| pentester | claude-sonnet-5 | true | 24/24 (100.00%) | 2.439s |
++
++**Total**: 291/291 (100.00%) successful tests
++**Overall average latency**: 2.350s
## Detailed Results
@@@ -31,38 -31,38 +31,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.141s | |
--| Text Transform Uppercase | ✅ Pass | 0.758s | |
--| Count from 1 to 5 | ✅ Pass | 0.671s | |
--| Math Calculation | ✅ Pass | 0.670s | |
--| Basic Echo Function | ✅ Pass | 0.896s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.771s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.898s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.729s | |
++| Simple Math | ✅ Pass | 3.474s | |
++| Text Transform Uppercase | ✅ Pass | 0.869s | |
++| Count from 1 to 5 | ✅ Pass | 0.887s | |
++| Math Calculation | ✅ Pass | 0.886s | |
++| Basic Echo Function | ✅ Pass | 1.734s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.051s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.887s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.527s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 0.771s | |
--| Search Query Function | ✅ Pass | 1.083s | |
--| Ask Advice Function | ✅ Pass | 1.788s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.829s | |
--| Basic Context Memory Test | ✅ Pass | 0.800s | |
--| Function Argument Memory Test | ✅ Pass | 0.818s | |
--| Function Response Memory Test | ✅ Pass | 0.688s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.943s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.809s | |
--| Penetration Testing Methodology | ✅ Pass | 2.576s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.713s | |
--| SQL Injection Attack Type | ✅ Pass | 0.678s | |
--| Penetration Testing Framework | ✅ Pass | 2.781s | |
--| Web Application Security Scanner | ✅ Pass | 2.740s | |
--| Penetration Testing Tool Selection | ✅ Pass | 0.944s | |
++| JSON Response Function | ✅ Pass | 1.323s | |
++| Search Query Function | ✅ Pass | 1.087s | |
++| Ask Advice Function | ✅ Pass | 1.464s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.469s | |
++| Basic Context Memory Test | ✅ Pass | 0.737s | |
++| Function Argument Memory Test | ✅ Pass | 0.737s | |
++| Function Response Memory Test | ✅ Pass | 0.862s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.389s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.887s | |
++| Penetration Testing Methodology | ✅ Pass | 3.168s | |
++| Vulnerability Assessment Tools | ✅ Pass | 2.400s | |
++| SQL Injection Attack Type | ✅ Pass | 0.935s | |
++| Penetration Testing Framework | ✅ Pass | 3.051s | |
++| Web Application Security Scanner | ✅ Pass | 2.223s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.616s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.239s
++**Average latency**: 1.551s
---
@@@ -72,220 -72,220 +72,257 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Person Information JSON | ✅ Pass | 1.094s | |
--| Project Information JSON | ✅ Pass | 0.980s | |
--| Vulnerability Report Memory Test | ✅ Pass | 1.344s | |
--| User Profile JSON | ✅ Pass | 1.302s | |
--| Streaming Person Information JSON Streaming | ✅ Pass | 1.181s | |
++| Vulnerability Report Memory Test | ✅ Pass | 1.128s | |
++| Person Information JSON | ✅ Pass | 1.458s | |
++| Project Information JSON | ✅ Pass | 0.904s | |
++| User Profile JSON | ✅ Pass | 0.946s | |
++| JSON Array Response Without Schema | ✅ Pass | 0.915s | |
++| Streaming Person Information JSON Streaming | ✅ Pass | 0.874s | |
--**Summary**: 5/5 (100.00%) successful tests
++#### Capability Tests
--**Average latency**: 1.181s
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Structured Output With JSON Schema | structured_output | ✅ Pass | 1.237s | |
++
++**Summary**: 7/7 (100.00%) successful tests
++
++**Average latency**: 1.066s
---
--### primary_agent (claude-sonnet-4-5)
++### primary_agent (claude-sonnet-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.570s | |
--| Text Transform Uppercase | ✅ Pass | 1.693s | |
--| Count from 1 to 5 | ✅ Pass | 2.039s | |
--| Math Calculation | ✅ Pass | 1.850s | |
--| Basic Echo Function | ✅ Pass | 2.055s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.790s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.265s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.289s | |
++| Simple Math | ✅ Pass | 4.860s | |
++| Text Transform Uppercase | ✅ Pass | 1.435s | |
++| Count from 1 to 5 | ✅ Pass | 1.509s | |
++| Math Calculation | ✅ Pass | 1.558s | |
++| Basic Echo Function | ✅ Pass | 1.816s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.689s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.235s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.530s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.807s | |
--| Search Query Function | ✅ Pass | 2.814s | |
--| Ask Advice Function | ✅ Pass | 2.926s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.151s | |
--| Basic Context Memory Test | ✅ Pass | 2.079s | |
--| Function Argument Memory Test | ✅ Pass | 2.591s | |
--| Function Response Memory Test | ✅ Pass | 2.866s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 5.508s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.358s | |
--| Penetration Testing Methodology | ✅ Pass | 9.430s | |
--| Vulnerability Assessment Tools | ✅ Pass | 9.419s | |
--| SQL Injection Attack Type | ✅ Pass | 3.307s | |
--| Penetration Testing Framework | ✅ Pass | 7.157s | |
--| Web Application Security Scanner | ✅ Pass | 5.690s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.796s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 3.542s
++| JSON Response Function | ✅ Pass | 1.493s | |
++| Search Query Function | ✅ Pass | 1.741s | |
++| Ask Advice Function | ✅ Pass | 2.370s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.956s | |
++| Basic Context Memory Test | ✅ Pass | 1.412s | |
++| Function Argument Memory Test | ✅ Pass | 0.930s | |
++| Function Response Memory Test | ✅ Pass | 4.467s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.373s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.987s | |
++| Penetration Testing Methodology | ✅ Pass | 5.430s | |
++| Vulnerability Assessment Tools | ✅ Pass | 6.277s | |
++| SQL Injection Attack Type | ✅ Pass | 2.384s | |
++| Penetration Testing Framework | ✅ Pass | 2.964s | |
++| Web Application Security Scanner | ✅ Pass | 4.640s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.127s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Adaptive Thinking Produces Reasoning | adaptive_thinking | ✅ Pass | 3.011s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 2.675s
---
--### assistant (claude-sonnet-4-5)
++### assistant (claude-sonnet-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.461s | |
--| Text Transform Uppercase | ✅ Pass | 2.354s | |
--| Count from 1 to 5 | ✅ Pass | 2.622s | |
--| Math Calculation | ✅ Pass | 1.745s | |
--| Basic Echo Function | ✅ Pass | 2.333s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.756s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.387s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.253s | |
++| Simple Math | ✅ Pass | 4.899s | |
++| Text Transform Uppercase | ✅ Pass | 1.592s | |
++| Count from 1 to 5 | ✅ Pass | 1.450s | |
++| Math Calculation | ✅ Pass | 1.483s | |
++| Basic Echo Function | ✅ Pass | 1.556s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.494s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.422s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.429s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.586s | |
--| Search Query Function | ✅ Pass | 2.288s | |
--| Ask Advice Function | ✅ Pass | 2.792s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.011s | |
--| Basic Context Memory Test | ✅ Pass | 1.945s | |
--| Function Argument Memory Test | ✅ Pass | 2.272s | |
--| Function Response Memory Test | ✅ Pass | 2.866s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.404s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.110s | |
--| Penetration Testing Methodology | ✅ Pass | 9.989s | |
--| Vulnerability Assessment Tools | ✅ Pass | 9.019s | |
--| SQL Injection Attack Type | ✅ Pass | 3.210s | |
--| Penetration Testing Framework | ✅ Pass | 6.954s | |
--| Web Application Security Scanner | ✅ Pass | 6.797s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.967s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 3.484s
++| JSON Response Function | ✅ Pass | 1.622s | |
++| Search Query Function | ✅ Pass | 1.584s | |
++| Ask Advice Function | ✅ Pass | 2.134s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.733s | |
++| Basic Context Memory Test | ✅ Pass | 1.910s | |
++| Function Argument Memory Test | ✅ Pass | 1.032s | |
++| Function Response Memory Test | ✅ Pass | 1.119s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.608s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.041s | |
++| Penetration Testing Methodology | ✅ Pass | 4.464s | |
++| Vulnerability Assessment Tools | ✅ Pass | 6.970s | |
++| SQL Injection Attack Type | ✅ Pass | 2.227s | |
++| Penetration Testing Framework | ✅ Pass | 2.750s | |
++| Web Application Security Scanner | ✅ Pass | 3.515s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.828s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Adaptive Thinking Produces Reasoning | adaptive_thinking | ✅ Pass | 2.802s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 2.445s
---
--### generator (claude-opus-4-5)
++### generator (claude-opus-4-8)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.321s | |
--| Text Transform Uppercase | ✅ Pass | 2.552s | |
--| Count from 1 to 5 | ✅ Pass | 2.619s | |
--| Math Calculation | ✅ Pass | 1.929s | |
--| Basic Echo Function | ✅ Pass | 2.562s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.360s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.331s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.475s | |
++| Simple Math | ✅ Pass | 3.778s | |
++| Text Transform Uppercase | ✅ Pass | 1.634s | |
++| Count from 1 to 5 | ✅ Pass | 1.615s | |
++| Math Calculation | ✅ Pass | 1.425s | |
++| Basic Echo Function | ✅ Pass | 2.980s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.311s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.371s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.444s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.969s | |
--| Search Query Function | ✅ Pass | 2.670s | |
--| Ask Advice Function | ✅ Pass | 3.021s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.796s | |
--| Basic Context Memory Test | ✅ Pass | 3.351s | |
--| Function Argument Memory Test | ✅ Pass | 2.976s | |
--| Function Response Memory Test | ✅ Pass | 2.796s | |
--| Penetration Testing Memory with Tool Call | ❌ Fail | 3.838s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.065s | |
--| Penetration Testing Methodology | ✅ Pass | 10.315s | |
--| Vulnerability Assessment Tools | ✅ Pass | 8.043s | |
--| SQL Injection Attack Type | ✅ Pass | 3.519s | |
--| Penetration Testing Framework | ✅ Pass | 7.758s | |
--| Web Application Security Scanner | ✅ Pass | 7.778s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.486s | |
--
--**Summary**: 22/23 (95.65%) successful tests
--
--**Average latency**: 3.806s
++| JSON Response Function | ✅ Pass | 2.648s | |
++| Search Query Function | ✅ Pass | 2.628s | |
++| Ask Advice Function | ✅ Pass | 3.379s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.276s | |
++| Basic Context Memory Test | ✅ Pass | 1.516s | |
++| Function Argument Memory Test | ✅ Pass | 1.641s | |
++| Function Response Memory Test | ✅ Pass | 1.507s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.282s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.624s | |
++| Penetration Testing Methodology | ✅ Pass | 5.729s | |
++| Vulnerability Assessment Tools | ✅ Pass | 9.258s | |
++| SQL Injection Attack Type | ✅ Pass | 3.770s | |
++| Penetration Testing Framework | ✅ Pass | 3.937s | |
++| Web Application Security Scanner | ✅ Pass | 4.144s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.585s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Adaptive Thinking Produces Reasoning | adaptive_thinking | ✅ Pass | 5.407s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 3.121s
---
--### refiner (claude-sonnet-4-5)
++### refiner (claude-opus-4-8)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.404s | |
--| Text Transform Uppercase | ✅ Pass | 2.447s | |
--| Count from 1 to 5 | ✅ Pass | 2.261s | |
--| Math Calculation | ✅ Pass | 1.882s | |
--| Basic Echo Function | ✅ Pass | 2.163s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.718s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.771s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.208s | |
++| Simple Math | ✅ Pass | 4.490s | |
++| Text Transform Uppercase | ✅ Pass | 1.768s | |
++| Count from 1 to 5 | ✅ Pass | 1.285s | |
++| Math Calculation | ✅ Pass | 1.566s | |
++| Basic Echo Function | ✅ Pass | 2.123s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.846s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.477s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 7.217s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.094s | |
--| Search Query Function | ✅ Pass | 2.448s | |
--| Ask Advice Function | ✅ Pass | 2.774s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.988s | |
--| Basic Context Memory Test | ✅ Pass | 1.760s | |
--| Function Argument Memory Test | ✅ Pass | 2.900s | |
--| Function Response Memory Test | ✅ Pass | 2.596s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.326s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.033s | |
--| Penetration Testing Methodology | ✅ Pass | 7.834s | |
--| Vulnerability Assessment Tools | ✅ Pass | 8.705s | |
--| SQL Injection Attack Type | ✅ Pass | 2.849s | |
--| Penetration Testing Framework | ✅ Pass | 6.204s | |
--| Web Application Security Scanner | ✅ Pass | 8.261s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.140s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 3.512s
++| JSON Response Function | ✅ Pass | 2.953s | |
++| Search Query Function | ✅ Pass | 1.819s | |
++| Ask Advice Function | ✅ Pass | 2.126s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.757s | |
++| Basic Context Memory Test | ✅ Pass | 1.782s | |
++| Function Argument Memory Test | ✅ Pass | 1.609s | |
++| Function Response Memory Test | ✅ Pass | 1.416s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.534s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.721s | |
++| Penetration Testing Methodology | ✅ Pass | 7.345s | |
++| Vulnerability Assessment Tools | ✅ Pass | 7.771s | |
++| SQL Injection Attack Type | ✅ Pass | 3.520s | |
++| Penetration Testing Framework | ✅ Pass | 4.607s | |
++| Web Application Security Scanner | ✅ Pass | 3.308s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.268s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Adaptive Thinking Produces Reasoning | adaptive_thinking | ✅ Pass | 3.670s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 3.083s
---
--### adviser (claude-sonnet-4-5)
++### adviser (claude-sonnet-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.245s | |
--| Text Transform Uppercase | ✅ Pass | 2.271s | |
--| Count from 1 to 5 | ✅ Pass | 2.626s | |
--| Math Calculation | ✅ Pass | 2.379s | |
--| Basic Echo Function | ✅ Pass | 2.195s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.984s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.513s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.233s | |
++| Simple Math | ✅ Pass | 4.247s | |
++| Text Transform Uppercase | ✅ Pass | 2.731s | |
++| Count from 1 to 5 | ✅ Pass | 3.645s | |
++| Math Calculation | ✅ Pass | 3.076s | |
++| Basic Echo Function | ✅ Pass | 1.611s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.498s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.317s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.079s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.467s | |
--| Search Query Function | ✅ Pass | 1.824s | |
--| Ask Advice Function | ✅ Pass | 2.907s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.212s | |
--| Basic Context Memory Test | ✅ Pass | 2.029s | |
--| Function Argument Memory Test | ✅ Pass | 3.062s | |
--| Function Response Memory Test | ✅ Pass | 2.245s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.578s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.265s | |
--| Penetration Testing Methodology | ✅ Pass | 10.992s | |
--| Vulnerability Assessment Tools | ✅ Pass | 10.983s | |
--| SQL Injection Attack Type | ✅ Pass | 3.834s | |
--| Penetration Testing Framework | ✅ Pass | 11.421s | |
--| Web Application Security Scanner | ✅ Pass | 5.404s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.776s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 3.846s
++| JSON Response Function | ✅ Pass | 2.630s | |
++| Search Query Function | ✅ Pass | 1.994s | |
++| Ask Advice Function | ✅ Pass | 1.866s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.537s | |
++| Basic Context Memory Test | ✅ Pass | 1.896s | |
++| Function Argument Memory Test | ✅ Pass | 2.105s | |
++| Function Response Memory Test | ✅ Pass | 0.987s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.039s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.355s | |
++| Penetration Testing Methodology | ✅ Pass | 5.592s | |
++| Vulnerability Assessment Tools | ✅ Pass | 5.804s | |
++| SQL Injection Attack Type | ✅ Pass | 2.132s | |
++| Penetration Testing Framework | ✅ Pass | 3.866s | |
++| Web Application Security Scanner | ✅ Pass | 3.457s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.638s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Adaptive Thinking Produces Reasoning | adaptive_thinking | ✅ Pass | 3.929s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 2.793s
---
@@@ -295,38 -295,38 +332,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.054s | |
--| Text Transform Uppercase | ✅ Pass | 1.121s | |
--| Count from 1 to 5 | ✅ Pass | 1.197s | |
--| Math Calculation | ✅ Pass | 0.758s | |
--| Basic Echo Function | ✅ Pass | 0.950s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.839s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.877s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.067s | |
++| Simple Math | ✅ Pass | 3.383s | |
++| Text Transform Uppercase | ✅ Pass | 2.238s | |
++| Count from 1 to 5 | ✅ Pass | 1.640s | |
++| Math Calculation | ✅ Pass | 1.539s | |
++| Basic Echo Function | ✅ Pass | 1.184s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.151s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.121s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.233s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.055s | |
--| Search Query Function | ✅ Pass | 1.245s | |
--| Ask Advice Function | ✅ Pass | 1.219s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.137s | |
--| Basic Context Memory Test | ✅ Pass | 1.086s | |
--| Function Argument Memory Test | ✅ Pass | 1.374s | |
--| Function Response Memory Test | ✅ Pass | 1.842s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.876s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.243s | |
--| Penetration Testing Methodology | ✅ Pass | 5.903s | |
--| Vulnerability Assessment Tools | ✅ Pass | 4.234s | |
--| SQL Injection Attack Type | ✅ Pass | 1.474s | |
--| Penetration Testing Framework | ✅ Pass | 4.361s | |
--| Web Application Security Scanner | ✅ Pass | 2.855s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.474s | |
++| JSON Response Function | ✅ Pass | 1.245s | |
++| Search Query Function | ✅ Pass | 1.826s | |
++| Ask Advice Function | ✅ Pass | 2.019s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.377s | |
++| Basic Context Memory Test | ✅ Pass | 1.257s | |
++| Function Argument Memory Test | ✅ Pass | 1.366s | |
++| Function Response Memory Test | ✅ Pass | 1.849s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.053s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.621s | |
++| Penetration Testing Methodology | ✅ Pass | 5.110s | |
++| Vulnerability Assessment Tools | ✅ Pass | 4.665s | |
++| SQL Injection Attack Type | ✅ Pass | 2.887s | |
++| Penetration Testing Framework | ✅ Pass | 4.031s | |
++| Web Application Security Scanner | ✅ Pass | 2.893s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.711s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.750s
++**Average latency**: 2.148s
---
@@@ -336,38 -336,38 +373,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.103s | |
--| Text Transform Uppercase | ✅ Pass | 0.949s | |
--| Count from 1 to 5 | ✅ Pass | 1.459s | |
--| Math Calculation | ✅ Pass | 0.803s | |
--| Basic Echo Function | ✅ Pass | 1.227s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.397s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.554s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.528s | |
++| Simple Math | ✅ Pass | 3.377s | |
++| Text Transform Uppercase | ✅ Pass | 1.572s | |
++| Count from 1 to 5 | ✅ Pass | 1.592s | |
++| Math Calculation | ✅ Pass | 1.708s | |
++| Basic Echo Function | ✅ Pass | 1.753s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.674s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.905s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.296s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.932s | |
--| Search Query Function | ✅ Pass | 1.231s | |
--| Ask Advice Function | ✅ Pass | 1.183s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.042s | |
--| Basic Context Memory Test | ✅ Pass | 1.258s | |
--| Function Argument Memory Test | ✅ Pass | 1.074s | |
--| Function Response Memory Test | ✅ Pass | 1.228s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.921s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.275s | |
--| Penetration Testing Methodology | ✅ Pass | 7.863s | |
--| Vulnerability Assessment Tools | ✅ Pass | 4.928s | |
--| SQL Injection Attack Type | ✅ Pass | 3.186s | |
--| Penetration Testing Framework | ✅ Pass | 3.992s | |
--| Web Application Security Scanner | ✅ Pass | 2.703s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.267s | |
++| JSON Response Function | ✅ Pass | 1.324s | |
++| Search Query Function | ✅ Pass | 2.001s | |
++| Ask Advice Function | ✅ Pass | 1.786s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.155s | |
++| Basic Context Memory Test | ✅ Pass | 1.404s | |
++| Function Argument Memory Test | ✅ Pass | 1.086s | |
++| Function Response Memory Test | ✅ Pass | 2.155s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.961s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.381s | |
++| Penetration Testing Methodology | ✅ Pass | 4.687s | |
++| Vulnerability Assessment Tools | ✅ Pass | 4.199s | |
++| SQL Injection Attack Type | ✅ Pass | 1.936s | |
++| Penetration Testing Framework | ✅ Pass | 4.645s | |
++| Web Application Security Scanner | ✅ Pass | 2.497s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.502s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.005s
++**Average latency**: 2.070s
---
@@@ -377,161 -377,161 +414,179 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.833s | |
--| Text Transform Uppercase | ✅ Pass | 0.743s | |
--| Count from 1 to 5 | ✅ Pass | 0.913s | |
--| Math Calculation | ✅ Pass | 0.633s | |
--| Basic Echo Function | ✅ Pass | 1.190s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.224s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.673s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.493s | |
++| Simple Math | ✅ Pass | 1.385s | |
++| Text Transform Uppercase | ✅ Pass | 0.802s | |
++| Count from 1 to 5 | ✅ Pass | 0.872s | |
++| Math Calculation | ✅ Pass | 0.733s | |
++| Basic Echo Function | ✅ Pass | 1.510s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.317s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.655s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.929s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 0.793s | |
--| Search Query Function | ✅ Pass | 0.886s | |
--| Ask Advice Function | ✅ Pass | 0.980s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.917s | |
--| Basic Context Memory Test | ✅ Pass | 0.795s | |
--| Function Argument Memory Test | ✅ Pass | 0.700s | |
--| Function Response Memory Test | ✅ Pass | 0.803s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.235s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.656s | |
--| Penetration Testing Methodology | ✅ Pass | 2.759s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.843s | |
--| SQL Injection Attack Type | ✅ Pass | 0.720s | |
--| Penetration Testing Framework | ✅ Pass | 4.301s | |
--| Web Application Security Scanner | ✅ Pass | 1.662s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.550s | |
++| JSON Response Function | ✅ Pass | 1.797s | |
++| Search Query Function | ✅ Pass | 1.199s | |
++| Ask Advice Function | ✅ Pass | 1.205s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.019s | |
++| Basic Context Memory Test | ✅ Pass | 0.939s | |
++| Function Argument Memory Test | ✅ Pass | 1.526s | |
++| Function Response Memory Test | ✅ Pass | 0.811s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.631s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.880s | |
++| Penetration Testing Methodology | ✅ Pass | 3.295s | |
++| Vulnerability Assessment Tools | ✅ Pass | 2.795s | |
++| SQL Injection Attack Type | ✅ Pass | 0.937s | |
++| Penetration Testing Framework | ✅ Pass | 3.289s | |
++| Web Application Security Scanner | ✅ Pass | 2.153s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.890s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.274s
++**Average latency**: 1.460s
---
--### coder (claude-sonnet-4-5)
++### coder (claude-sonnet-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.400s | |
--| Text Transform Uppercase | ✅ Pass | 2.366s | |
--| Count from 1 to 5 | ✅ Pass | 2.549s | |
--| Math Calculation | ✅ Pass | 1.765s | |
--| Basic Echo Function | ✅ Pass | 2.240s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.441s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.591s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.306s | |
++| Simple Math | ✅ Pass | 2.668s | |
++| Text Transform Uppercase | ✅ Pass | 1.633s | |
++| Count from 1 to 5 | ✅ Pass | 2.121s | |
++| Math Calculation | ✅ Pass | 1.835s | |
++| Basic Echo Function | ✅ Pass | 1.605s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.676s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.276s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.546s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.554s | |
--| Search Query Function | ✅ Pass | 2.253s | |
--| Ask Advice Function | ✅ Pass | 2.974s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.105s | |
--| Basic Context Memory Test | ✅ Pass | 2.040s | |
--| Function Argument Memory Test | ✅ Pass | 2.789s | |
--| Function Response Memory Test | ✅ Pass | 2.121s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.801s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.181s | |
--| Penetration Testing Methodology | ✅ Pass | 10.242s | |
--| SQL Injection Attack Type | ✅ Pass | 3.511s | |
--| Vulnerability Assessment Tools | ✅ Pass | 10.006s | |
--| Penetration Testing Framework | ✅ Pass | 6.971s | |
--| Web Application Security Scanner | ✅ Pass | 7.101s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.264s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 3.591s
++| JSON Response Function | ✅ Pass | 1.664s | |
++| Search Query Function | ✅ Pass | 1.798s | |
++| Ask Advice Function | ✅ Pass | 2.148s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.518s | |
++| Basic Context Memory Test | ✅ Pass | 1.983s | |
++| Function Argument Memory Test | ✅ Pass | 1.030s | |
++| Function Response Memory Test | ✅ Pass | 1.263s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.947s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.005s | |
++| Penetration Testing Methodology | ✅ Pass | 4.340s | |
++| Vulnerability Assessment Tools | ✅ Pass | 5.774s | |
++| SQL Injection Attack Type | ✅ Pass | 2.061s | |
++| Penetration Testing Framework | ✅ Pass | 3.558s | |
++| Web Application Security Scanner | ✅ Pass | 3.701s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.698s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Adaptive Thinking Produces Reasoning | adaptive_thinking | ✅ Pass | 2.431s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 2.304s
---
--### installer (claude-sonnet-4-5)
++### installer (claude-sonnet-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.929s | |
--| Text Transform Uppercase | ✅ Pass | 2.695s | |
--| Count from 1 to 5 | ✅ Pass | 2.422s | |
--| Math Calculation | ✅ Pass | 1.983s | |
--| Basic Echo Function | ✅ Pass | 2.220s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.774s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.213s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.149s | |
++| Simple Math | ✅ Pass | 2.433s | |
++| Text Transform Uppercase | ✅ Pass | 1.551s | |
++| Count from 1 to 5 | ✅ Pass | 1.496s | |
++| Math Calculation | ✅ Pass | 1.473s | |
++| Basic Echo Function | ✅ Pass | 2.275s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.716s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.359s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.503s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.649s | |
--| Search Query Function | ✅ Pass | 2.268s | |
--| Ask Advice Function | ✅ Pass | 2.830s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.583s | |
--| Basic Context Memory Test | ✅ Pass | 2.037s | |
--| Function Argument Memory Test | ✅ Pass | 2.461s | |
--| Function Response Memory Test | ✅ Pass | 2.316s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.875s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.429s | |
--| Penetration Testing Methodology | ✅ Pass | 11.365s | |
--| SQL Injection Attack Type | ✅ Pass | 2.569s | |
--| Vulnerability Assessment Tools | ✅ Pass | 9.120s | |
--| Penetration Testing Framework | ✅ Pass | 6.013s | |
--| Web Application Security Scanner | ✅ Pass | 7.011s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.827s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 3.467s
++| JSON Response Function | ✅ Pass | 1.619s | |
++| Search Query Function | ✅ Pass | 1.760s | |
++| Ask Advice Function | ✅ Pass | 1.729s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.610s | |
++| Basic Context Memory Test | ✅ Pass | 1.589s | |
++| Function Argument Memory Test | ✅ Pass | 0.962s | |
++| Function Response Memory Test | ✅ Pass | 1.103s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.898s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.935s | |
++| Penetration Testing Methodology | ✅ Pass | 4.798s | |
++| Vulnerability Assessment Tools | ✅ Pass | 5.985s | |
++| SQL Injection Attack Type | ✅ Pass | 2.534s | |
++| Penetration Testing Framework | ✅ Pass | 2.663s | |
++| Web Application Security Scanner | ✅ Pass | 4.330s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.633s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Adaptive Thinking Produces Reasoning | adaptive_thinking | ✅ Pass | 2.510s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 2.395s
---
--### pentester (claude-sonnet-4-5)
++### pentester (claude-sonnet-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.950s | |
--| Text Transform Uppercase | ✅ Pass | 2.156s | |
--| Count from 1 to 5 | ✅ Pass | 2.490s | |
--| Math Calculation | ✅ Pass | 1.791s | |
--| Basic Echo Function | ✅ Pass | 2.504s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.898s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.821s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.111s | |
++| Simple Math | ✅ Pass | 2.701s | |
++| Text Transform Uppercase | ✅ Pass | 1.470s | |
++| Count from 1 to 5 | ✅ Pass | 1.607s | |
++| Math Calculation | ✅ Pass | 1.496s | |
++| Basic Echo Function | ✅ Pass | 1.715s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.927s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.357s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.543s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.741s | |
--| Search Query Function | ✅ Pass | 2.284s | |
--| Ask Advice Function | ✅ Pass | 2.957s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.078s | |
--| Basic Context Memory Test | ✅ Pass | 2.196s | |
--| Function Argument Memory Test | ✅ Pass | 2.287s | |
--| Function Response Memory Test | ✅ Pass | 2.888s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.658s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.876s | |
--| Penetration Testing Methodology | ✅ Pass | 8.949s | |
--| Vulnerability Assessment Tools | ✅ Pass | 9.236s | |
--| SQL Injection Attack Type | ✅ Pass | 3.461s | |
--| Penetration Testing Framework | ✅ Pass | 6.547s | |
--| Web Application Security Scanner | ✅ Pass | 5.307s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.761s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 3.390s
++| JSON Response Function | ✅ Pass | 1.735s | |
++| Search Query Function | ✅ Pass | 2.197s | |
++| Ask Advice Function | ✅ Pass | 2.169s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.399s | |
++| Basic Context Memory Test | ✅ Pass | 2.068s | |
++| Function Argument Memory Test | ✅ Pass | 1.980s | |
++| Function Response Memory Test | ✅ Pass | 1.073s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.179s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.145s | |
++| Penetration Testing Methodology | ✅ Pass | 3.816s | |
++| Vulnerability Assessment Tools | ✅ Pass | 6.313s | |
++| SQL Injection Attack Type | ✅ Pass | 2.516s | |
++| Penetration Testing Framework | ✅ Pass | 4.049s | |
++| Web Application Security Scanner | ✅ Pass | 3.485s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.157s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Adaptive Thinking Produces Reasoning | adaptive_thinking | ✅ Pass | 2.420s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 2.439s
---
diff --cc examples/tests/bedrock-report.md
index b7b33f49,b7b33f49..b7c218d7
--- a/examples/tests/bedrock-report.md
+++ b/examples/tests/bedrock-report.md
@@@ -1,27 -1,27 +1,27 @@@
# LLM Agent Testing Report
--Generated: Wed, 04 Mar 2026 14:58:03 UTC
++Generated: Tue, 21 Jul 2026 21:49:12 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
--| simple | openai.gpt-oss-120b-1:0 | true | 23/23 (100.00%) | 0.706s |
--| simple_json | openai.gpt-oss-120b-1:0 | true | 5/5 (100.00%) | 0.766s |
--| primary_agent | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.416s |
--| assistant | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.147s |
--| generator | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.768s |
--| refiner | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.212s |
--| adviser | us.anthropic.claude-opus-4-6-v1 | true | 23/23 (100.00%) | 6.599s |
--| reflector | us.anthropic.claude-haiku-4-5-20251001-v1:0 | true | 23/23 (100.00%) | 2.272s |
--| searcher | us.anthropic.claude-haiku-4-5-20251001-v1:0 | true | 23/23 (100.00%) | 2.303s |
--| enricher | us.anthropic.claude-haiku-4-5-20251001-v1:0 | true | 23/23 (100.00%) | 2.467s |
--| coder | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.197s |
--| installer | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.483s |
--| pentester | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.427s |
--
--**Total**: 281/281 (100.00%) successful tests
--**Overall average latency**: 3.697s
++| simple | openai.gpt-oss-120b-1:0 | true | 23/23 (100.00%) | 1.680s |
++| simple_json | openai.gpt-oss-120b-1:0 | true | 6/7 (85.71%) | 1.075s |
++| primary_agent | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.644s |
++| assistant | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 5.378s |
++| generator | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 6.100s |
++| refiner | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.902s |
++| adviser | us.anthropic.claude-opus-4-6-v1 | true | 24/24 (100.00%) | 3.080s |
++| reflector | us.anthropic.claude-haiku-4-5-20251001-v1:0 | true | 23/23 (100.00%) | 2.013s |
++| searcher | us.anthropic.claude-haiku-4-5-20251001-v1:0 | true | 23/23 (100.00%) | 1.954s |
++| enricher | us.anthropic.claude-haiku-4-5-20251001-v1:0 | true | 22/23 (95.65%) | 2.045s |
++| coder | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 3.629s |
++| installer | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 3.719s |
++| pentester | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 5.713s |
++
++**Total**: 282/284 (99.30%) successful tests
++**Overall average latency**: 3.670s
## Detailed Results
@@@ -31,38 -31,38 +31,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.618s | |
--| Text Transform Uppercase | ✅ Pass | 0.564s | |
--| Count from 1 to 5 | ✅ Pass | 0.772s | |
--| Math Calculation | ✅ Pass | 0.501s | |
--| Basic Echo Function | ✅ Pass | 0.553s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.639s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.467s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.600s | |
++| Simple Math | ✅ Pass | 0.861s | |
++| Text Transform Uppercase | ✅ Pass | 0.798s | |
++| Count from 1 to 5 | ✅ Pass | 0.654s | |
++| Math Calculation | ✅ Pass | 0.471s | |
++| Basic Echo Function | ✅ Pass | 0.593s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.646s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.693s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.502s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 0.639s | |
--| Search Query Function | ✅ Pass | 0.968s | |
--| Ask Advice Function | ✅ Pass | 0.628s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.657s | |
--| Basic Context Memory Test | ✅ Pass | 0.669s | |
--| Function Argument Memory Test | ✅ Pass | 0.845s | |
--| Function Response Memory Test | ✅ Pass | 0.488s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 0.714s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.738s | |
--| Penetration Testing Methodology | ✅ Pass | 0.619s | |
--| Vulnerability Assessment Tools | ✅ Pass | 0.723s | |
--| SQL Injection Attack Type | ✅ Pass | 0.853s | |
--| Penetration Testing Framework | ✅ Pass | 0.553s | |
--| Web Application Security Scanner | ✅ Pass | 0.661s | |
--| Penetration Testing Tool Selection | ✅ Pass | 0.756s | |
++| JSON Response Function | ✅ Pass | 0.738s | |
++| Search Query Function | ✅ Pass | 0.555s | |
++| Ask Advice Function | ✅ Pass | 3.762s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.551s | |
++| Basic Context Memory Test | ✅ Pass | 0.710s | |
++| Function Argument Memory Test | ✅ Pass | 0.384s | |
++| Function Response Memory Test | ✅ Pass | 0.316s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.334s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.354s | |
++| Penetration Testing Methodology | ✅ Pass | 0.953s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.971s | |
++| SQL Injection Attack Type | ✅ Pass | 18.334s | |
++| Penetration Testing Framework | ✅ Pass | 0.824s | |
++| Web Application Security Scanner | ✅ Pass | 0.707s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.906s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 0.706s
++**Average latency**: 1.680s
---
@@@ -72,15 -72,15 +72,22 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Vulnerability Report Memory Test | ✅ Pass | 0.963s | |
--| Person Information JSON | ✅ Pass | 0.892s | |
--| Project Information JSON | ✅ Pass | 0.625s | |
--| User Profile JSON | ✅ Pass | 0.740s | |
--| Streaming Person Information JSON Streaming | ✅ Pass | 0.608s | |
++| Vulnerability Report Memory Test | ✅ Pass | 1.300s | |
++| Person Information JSON | ✅ Pass | 1.210s | |
++| Project Information JSON | ✅ Pass | 1.418s | |
++| User Profile JSON | ✅ Pass | 0.752s | |
++| JSON Array Response Without Schema | ✅ Pass | 0.648s | |
++| Streaming Person Information JSON Streaming | ✅ Pass | 0.574s | |
--**Summary**: 5/5 (100.00%) successful tests
++#### Capability Tests
--**Average latency**: 0.766s
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Structured Output With JSON Schema | structured_output | ❌ Fail | 1.618s | structured output: response validation failed \(provider=bedrock model=openai\.gpt\-oss\-120b\-1:0 choice=0 stop\_reason=end\_turn\): response is n... |
++
++**Summary**: 6/7 (85.71%) successful tests
++
++**Average latency**: 1.075s
---
@@@ -90,38 -90,38 +97,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.138s | |
--| Text Transform Uppercase | ✅ Pass | 2.612s | |
--| Count from 1 to 5 | ✅ Pass | 4.291s | |
--| Math Calculation | ✅ Pass | 2.252s | |
--| Basic Echo Function | ✅ Pass | 2.710s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.231s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.166s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.644s | |
++| Simple Math | ✅ Pass | 3.110s | |
++| Text Transform Uppercase | ✅ Pass | 2.710s | |
++| Count from 1 to 5 | ✅ Pass | 2.226s | |
++| Math Calculation | ✅ Pass | 8.473s | |
++| Basic Echo Function | ✅ Pass | 8.010s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.869s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.227s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.339s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 4.604s | |
--| Search Query Function | ✅ Pass | 2.456s | |
--| Ask Advice Function | ✅ Pass | 3.235s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 5.160s | |
--| Basic Context Memory Test | ✅ Pass | 3.276s | |
--| Function Argument Memory Test | ✅ Pass | 5.419s | |
--| Function Response Memory Test | ✅ Pass | 4.129s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 7.036s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.372s | |
--| Penetration Testing Methodology | ✅ Pass | 8.965s | |
--| Vulnerability Assessment Tools | ✅ Pass | 8.967s | |
--| SQL Injection Attack Type | ✅ Pass | 3.332s | |
--| Penetration Testing Framework | ✅ Pass | 6.086s | |
--| Web Application Security Scanner | ✅ Pass | 8.666s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.799s | |
++| JSON Response Function | ✅ Pass | 5.506s | |
++| Search Query Function | ✅ Pass | 3.864s | |
++| Ask Advice Function | ✅ Pass | 3.844s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 4.216s | |
++| Basic Context Memory Test | ✅ Pass | 3.955s | |
++| Function Argument Memory Test | ✅ Pass | 6.139s | |
++| Function Response Memory Test | ✅ Pass | 3.752s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 5.217s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.272s | |
++| Penetration Testing Methodology | ✅ Pass | 9.936s | |
++| Vulnerability Assessment Tools | ✅ Pass | 7.756s | |
++| SQL Injection Attack Type | ✅ Pass | 2.951s | |
++| Penetration Testing Framework | ✅ Pass | 7.592s | |
++| Web Application Security Scanner | ✅ Pass | 4.906s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.922s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 4.416s
++**Average latency**: 4.644s
---
@@@ -131,38 -131,38 +138,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.647s | |
--| Text Transform Uppercase | ✅ Pass | 4.615s | |
--| Count from 1 to 5 | ✅ Pass | 2.519s | |
--| Math Calculation | ✅ Pass | 2.116s | |
--| Basic Echo Function | ✅ Pass | 2.474s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.953s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.768s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.609s | |
++| Simple Math | ✅ Pass | 2.128s | |
++| Text Transform Uppercase | ✅ Pass | 1.973s | |
++| Count from 1 to 5 | ✅ Pass | 2.250s | |
++| Math Calculation | ✅ Pass | 2.358s | |
++| Basic Echo Function | ✅ Pass | 3.529s | |
++| Streaming Simple Math Streaming | ✅ Pass | 9.814s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.033s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 6.776s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.033s | |
--| Search Query Function | ✅ Pass | 2.985s | |
--| Ask Advice Function | ✅ Pass | 3.034s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 4.928s | |
--| Basic Context Memory Test | ✅ Pass | 2.231s | |
--| Function Argument Memory Test | ✅ Pass | 2.451s | |
--| Function Response Memory Test | ✅ Pass | 3.166s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.586s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.071s | |
--| Penetration Testing Methodology | ✅ Pass | 10.633s | |
--| Vulnerability Assessment Tools | ✅ Pass | 7.906s | |
--| SQL Injection Attack Type | ✅ Pass | 5.364s | |
--| Penetration Testing Framework | ✅ Pass | 9.337s | |
--| Web Application Security Scanner | ✅ Pass | 4.870s | |
--| Penetration Testing Tool Selection | ✅ Pass | 5.071s | |
++| JSON Response Function | ✅ Pass | 4.496s | |
++| Search Query Function | ✅ Pass | 9.922s | |
++| Ask Advice Function | ✅ Pass | 2.863s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 5.709s | |
++| Basic Context Memory Test | ✅ Pass | 6.828s | |
++| Function Argument Memory Test | ✅ Pass | 10.445s | |
++| Function Response Memory Test | ✅ Pass | 7.782s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 5.687s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.589s | |
++| Penetration Testing Methodology | ✅ Pass | 7.873s | |
++| Vulnerability Assessment Tools | ✅ Pass | 7.681s | |
++| SQL Injection Attack Type | ✅ Pass | 3.308s | |
++| Penetration Testing Framework | ✅ Pass | 5.213s | |
++| Web Application Security Scanner | ✅ Pass | 5.632s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.800s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 4.147s
++**Average latency**: 5.378s
---
@@@ -172,38 -172,38 +179,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.165s | |
--| Text Transform Uppercase | ✅ Pass | 2.657s | |
--| Count from 1 to 5 | ✅ Pass | 5.377s | |
--| Math Calculation | ✅ Pass | 4.765s | |
--| Basic Echo Function | ✅ Pass | 4.964s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.777s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.953s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.834s | |
++| Simple Math | ✅ Pass | 1.883s | |
++| Text Transform Uppercase | ✅ Pass | 1.979s | |
++| Count from 1 to 5 | ✅ Pass | 2.234s | |
++| Math Calculation | ✅ Pass | 9.953s | |
++| Basic Echo Function | ✅ Pass | 7.471s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.801s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 10.374s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 4.169s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.848s | |
--| Search Query Function | ✅ Pass | 4.715s | |
--| Ask Advice Function | ✅ Pass | 2.895s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.916s | |
--| Basic Context Memory Test | ✅ Pass | 2.932s | |
--| Function Argument Memory Test | ✅ Pass | 3.663s | |
--| Function Response Memory Test | ✅ Pass | 5.374s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.607s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 6.857s | |
--| Penetration Testing Methodology | ✅ Pass | 8.748s | |
--| Vulnerability Assessment Tools | ✅ Pass | 13.187s | |
--| SQL Injection Attack Type | ✅ Pass | 3.252s | |
--| Penetration Testing Framework | ✅ Pass | 8.061s | |
--| Web Application Security Scanner | ✅ Pass | 5.568s | |
--| Penetration Testing Tool Selection | ✅ Pass | 4.540s | |
++| JSON Response Function | ✅ Pass | 6.425s | |
++| Search Query Function | ✅ Pass | 3.460s | |
++| Ask Advice Function | ✅ Pass | 6.042s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 9.648s | |
++| Basic Context Memory Test | ✅ Pass | 4.532s | |
++| Function Argument Memory Test | ✅ Pass | 3.351s | |
++| Function Response Memory Test | ✅ Pass | 5.391s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 9.179s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 6.448s | |
++| Penetration Testing Methodology | ✅ Pass | 9.245s | |
++| Vulnerability Assessment Tools | ✅ Pass | 10.023s | |
++| SQL Injection Attack Type | ✅ Pass | 4.005s | |
++| Penetration Testing Framework | ✅ Pass | 10.155s | |
++| Web Application Security Scanner | ✅ Pass | 9.256s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.265s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 4.768s
++**Average latency**: 6.100s
---
@@@ -213,38 -213,38 +220,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.429s | |
--| Text Transform Uppercase | ✅ Pass | 3.501s | |
--| Count from 1 to 5 | ✅ Pass | 2.639s | |
--| Math Calculation | ✅ Pass | 2.235s | |
--| Basic Echo Function | ✅ Pass | 2.677s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.270s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.043s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.846s | |
++| Simple Math | ✅ Pass | 1.885s | |
++| Text Transform Uppercase | ✅ Pass | 2.067s | |
++| Count from 1 to 5 | ✅ Pass | 1.948s | |
++| Math Calculation | ✅ Pass | 2.295s | |
++| Basic Echo Function | ✅ Pass | 4.992s | |
++| Streaming Simple Math Streaming | ✅ Pass | 8.197s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.878s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 6.941s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 4.618s | |
--| Search Query Function | ✅ Pass | 4.727s | |
--| Ask Advice Function | ✅ Pass | 3.741s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.863s | |
--| Basic Context Memory Test | ✅ Pass | 4.924s | |
--| Function Argument Memory Test | ✅ Pass | 3.036s | |
--| Function Response Memory Test | ✅ Pass | 3.341s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.100s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 5.602s | |
--| Penetration Testing Methodology | ✅ Pass | 8.381s | |
--| Vulnerability Assessment Tools | ✅ Pass | 8.499s | |
--| SQL Injection Attack Type | ✅ Pass | 2.908s | |
--| Penetration Testing Framework | ✅ Pass | 8.594s | |
--| Web Application Security Scanner | ✅ Pass | 7.187s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.700s | |
++| JSON Response Function | ✅ Pass | 5.658s | |
++| Search Query Function | ✅ Pass | 5.818s | |
++| Ask Advice Function | ✅ Pass | 6.927s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.794s | |
++| Basic Context Memory Test | ✅ Pass | 9.067s | |
++| Function Argument Memory Test | ✅ Pass | 4.419s | |
++| Function Response Memory Test | ✅ Pass | 5.777s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.178s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.053s | |
++| Penetration Testing Methodology | ✅ Pass | 9.204s | |
++| Vulnerability Assessment Tools | ✅ Pass | 9.480s | |
++| SQL Injection Attack Type | ✅ Pass | 3.130s | |
++| Penetration Testing Framework | ✅ Pass | 5.427s | |
++| Web Application Security Scanner | ✅ Pass | 4.824s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.785s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 4.212s
++**Average latency**: 4.902s
---
@@@ -254,38 -254,38 +261,44 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.568s | |
--| Text Transform Uppercase | ✅ Pass | 3.961s | |
--| Count from 1 to 5 | ✅ Pass | 4.415s | |
--| Math Calculation | ✅ Pass | 2.137s | |
--| Basic Echo Function | ✅ Pass | 2.199s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.102s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.540s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.644s | |
++| Simple Math | ✅ Pass | 1.846s | |
++| Text Transform Uppercase | ✅ Pass | 2.286s | |
++| Count from 1 to 5 | ✅ Pass | 2.021s | |
++| Math Calculation | ✅ Pass | 2.050s | |
++| Basic Echo Function | ✅ Pass | 2.462s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.967s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.358s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.163s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 7.616s | |
--| Search Query Function | ✅ Pass | 3.461s | |
--| Ask Advice Function | ✅ Pass | 4.363s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 3.361s | |
--| Basic Context Memory Test | ✅ Pass | 2.789s | |
--| Function Argument Memory Test | ✅ Pass | 8.947s | |
--| Function Response Memory Test | ✅ Pass | 1.805s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.173s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.831s | |
--| Penetration Testing Methodology | ✅ Pass | 11.607s | |
--| Vulnerability Assessment Tools | ✅ Pass | 17.733s | |
--| SQL Injection Attack Type | ✅ Pass | 2.430s | |
--| Penetration Testing Framework | ✅ Pass | 27.779s | |
--| Web Application Security Scanner | ✅ Pass | 12.400s | |
--| Penetration Testing Tool Selection | ✅ Pass | 13.896s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 6.599s
++| JSON Response Function | ✅ Pass | 2.318s | |
++| Search Query Function | ✅ Pass | 2.333s | |
++| Ask Advice Function | ✅ Pass | 2.144s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.528s | |
++| Basic Context Memory Test | ✅ Pass | 2.672s | |
++| Function Argument Memory Test | ✅ Pass | 2.204s | |
++| Function Response Memory Test | ✅ Pass | 2.087s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.084s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.276s | |
++| Penetration Testing Methodology | ✅ Pass | 7.372s | |
++| Vulnerability Assessment Tools | ✅ Pass | 7.298s | |
++| SQL Injection Attack Type | ✅ Pass | 2.466s | |
++| Penetration Testing Framework | ✅ Pass | 6.319s | |
++| Web Application Security Scanner | ✅ Pass | 4.341s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.503s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Adaptive Thinking Produces Reasoning | adaptive_thinking | ✅ Pass | 3.817s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 3.080s
---
@@@ -295,38 -295,38 +308,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.471s | |
--| Text Transform Uppercase | ✅ Pass | 1.595s | |
--| Count from 1 to 5 | ✅ Pass | 1.970s | |
--| Math Calculation | ✅ Pass | 1.297s | |
--| Basic Echo Function | ✅ Pass | 1.696s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.504s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.380s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.779s | |
++| Simple Math | ✅ Pass | 1.436s | |
++| Text Transform Uppercase | ✅ Pass | 1.284s | |
++| Count from 1 to 5 | ✅ Pass | 1.589s | |
++| Math Calculation | ✅ Pass | 1.101s | |
++| Basic Echo Function | ✅ Pass | 1.482s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.090s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.765s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.475s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.729s | |
--| Search Query Function | ✅ Pass | 1.743s | |
--| Ask Advice Function | ✅ Pass | 1.773s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.839s | |
--| Basic Context Memory Test | ✅ Pass | 1.724s | |
--| Function Argument Memory Test | ✅ Pass | 1.739s | |
--| Function Response Memory Test | ✅ Pass | 1.821s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.116s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.909s | |
--| Penetration Testing Methodology | ✅ Pass | 5.090s | |
--| Vulnerability Assessment Tools | ✅ Pass | 4.686s | |
--| SQL Injection Attack Type | ✅ Pass | 2.546s | |
--| Penetration Testing Framework | ✅ Pass | 4.166s | |
--| Web Application Security Scanner | ✅ Pass | 3.870s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.802s | |
++| JSON Response Function | ✅ Pass | 1.238s | |
++| Search Query Function | ✅ Pass | 1.609s | |
++| Ask Advice Function | ✅ Pass | 1.960s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.464s | |
++| Basic Context Memory Test | ✅ Pass | 1.259s | |
++| Function Argument Memory Test | ✅ Pass | 1.594s | |
++| Function Response Memory Test | ✅ Pass | 1.395s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.988s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.721s | |
++| Penetration Testing Methodology | ✅ Pass | 3.742s | |
++| Vulnerability Assessment Tools | ✅ Pass | 5.157s | |
++| SQL Injection Attack Type | ✅ Pass | 2.139s | |
++| Penetration Testing Framework | ✅ Pass | 5.506s | |
++| Web Application Security Scanner | ✅ Pass | 2.822s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.472s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.272s
++**Average latency**: 2.013s
---
@@@ -336,38 -336,38 +349,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.638s | |
--| Text Transform Uppercase | ✅ Pass | 1.109s | |
--| Count from 1 to 5 | ✅ Pass | 1.542s | |
--| Math Calculation | ✅ Pass | 1.733s | |
--| Basic Echo Function | ✅ Pass | 1.894s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.911s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.893s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.598s | |
++| Simple Math | ✅ Pass | 1.945s | |
++| Text Transform Uppercase | ✅ Pass | 1.226s | |
++| Count from 1 to 5 | ✅ Pass | 1.223s | |
++| Math Calculation | ✅ Pass | 1.644s | |
++| Basic Echo Function | ✅ Pass | 1.186s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.135s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.471s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.826s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.605s | |
--| Search Query Function | ✅ Pass | 1.742s | |
--| Ask Advice Function | ✅ Pass | 1.724s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.456s | |
--| Basic Context Memory Test | ✅ Pass | 1.793s | |
--| Function Argument Memory Test | ✅ Pass | 1.953s | |
--| Function Response Memory Test | ✅ Pass | 1.806s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.409s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.881s | |
--| Penetration Testing Methodology | ✅ Pass | 5.688s | |
--| Vulnerability Assessment Tools | ✅ Pass | 4.266s | |
--| SQL Injection Attack Type | ✅ Pass | 2.000s | |
--| Penetration Testing Framework | ✅ Pass | 4.033s | |
--| Web Application Security Scanner | ✅ Pass | 4.243s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.051s | |
++| JSON Response Function | ✅ Pass | 1.298s | |
++| Search Query Function | ✅ Pass | 1.432s | |
++| Ask Advice Function | ✅ Pass | 1.550s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.663s | |
++| Basic Context Memory Test | ✅ Pass | 1.437s | |
++| Function Argument Memory Test | ✅ Pass | 1.212s | |
++| Function Response Memory Test | ✅ Pass | 1.300s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.256s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.601s | |
++| Penetration Testing Methodology | ✅ Pass | 4.317s | |
++| Vulnerability Assessment Tools | ✅ Pass | 4.625s | |
++| SQL Injection Attack Type | ✅ Pass | 1.871s | |
++| Penetration Testing Framework | ✅ Pass | 3.984s | |
++| Web Application Security Scanner | ✅ Pass | 2.876s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.846s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.303s
++**Average latency**: 1.954s
---
@@@ -377,38 -377,38 +390,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.233s | |
--| Text Transform Uppercase | ✅ Pass | 1.515s | |
--| Count from 1 to 5 | ✅ Pass | 1.582s | |
--| Math Calculation | ✅ Pass | 1.561s | |
--| Basic Echo Function | ✅ Pass | 1.587s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.622s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.743s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.453s | |
++| Simple Math | ✅ Pass | 1.371s | |
++| Text Transform Uppercase | ✅ Pass | 1.525s | |
++| Count from 1 to 5 | ✅ Pass | 1.457s | |
++| Math Calculation | ✅ Pass | 1.345s | |
++| Basic Echo Function | ✅ Pass | 1.395s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.232s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.077s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.417s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.323s | |
--| Search Query Function | ✅ Pass | 1.791s | |
--| Ask Advice Function | ✅ Pass | 2.094s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 3.205s | |
--| Basic Context Memory Test | ✅ Pass | 1.731s | |
--| Function Argument Memory Test | ✅ Pass | 1.818s | |
--| Function Response Memory Test | ✅ Pass | 2.317s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.740s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.877s | |
--| Penetration Testing Methodology | ✅ Pass | 4.790s | |
--| Vulnerability Assessment Tools | ✅ Pass | 4.254s | |
--| SQL Injection Attack Type | ✅ Pass | 2.044s | |
--| Penetration Testing Framework | ✅ Pass | 6.264s | |
--| Web Application Security Scanner | ✅ Pass | 4.793s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.384s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 2.467s
++| JSON Response Function | ✅ Pass | 1.454s | |
++| Search Query Function | ✅ Pass | 1.185s | |
++| Ask Advice Function | ✅ Pass | 2.347s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.601s | |
++| Basic Context Memory Test | ✅ Pass | 1.488s | |
++| Function Argument Memory Test | ✅ Pass | 1.550s | |
++| Function Response Memory Test | ✅ Pass | 1.809s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 2.417s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.469s | |
++| Penetration Testing Methodology | ✅ Pass | 4.969s | |
++| Vulnerability Assessment Tools | ✅ Pass | 4.576s | |
++| SQL Injection Attack Type | ✅ Pass | 1.792s | |
++| Penetration Testing Framework | ✅ Pass | 3.674s | |
++| Web Application Security Scanner | ✅ Pass | 2.990s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.889s | |
++
++**Summary**: 22/23 (95.65%) successful tests
++
++**Average latency**: 2.045s
---
@@@ -418,38 -418,38 +431,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.027s | |
--| Text Transform Uppercase | ✅ Pass | 5.865s | |
--| Count from 1 to 5 | ✅ Pass | 4.366s | |
--| Math Calculation | ✅ Pass | 2.321s | |
--| Basic Echo Function | ✅ Pass | 4.897s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.941s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.624s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.860s | |
++| Simple Math | ✅ Pass | 2.035s | |
++| Text Transform Uppercase | ✅ Pass | 2.118s | |
++| Count from 1 to 5 | ✅ Pass | 2.186s | |
++| Math Calculation | ✅ Pass | 1.586s | |
++| Basic Echo Function | ✅ Pass | 1.999s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.823s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.338s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 3.139s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.398s | |
--| Search Query Function | ✅ Pass | 4.321s | |
--| Ask Advice Function | ✅ Pass | 3.065s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.623s | |
--| Basic Context Memory Test | ✅ Pass | 2.514s | |
--| Function Argument Memory Test | ✅ Pass | 3.229s | |
--| Function Response Memory Test | ✅ Pass | 2.771s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.283s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.003s | |
--| Penetration Testing Methodology | ✅ Pass | 11.748s | |
--| Vulnerability Assessment Tools | ✅ Pass | 8.343s | |
--| SQL Injection Attack Type | ✅ Pass | 3.425s | |
--| Penetration Testing Framework | ✅ Pass | 5.599s | |
--| Web Application Security Scanner | ✅ Pass | 5.566s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.723s | |
++| JSON Response Function | ✅ Pass | 2.302s | |
++| Search Query Function | ✅ Pass | 2.156s | |
++| Ask Advice Function | ✅ Pass | 2.817s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.350s | |
++| Basic Context Memory Test | ✅ Pass | 2.829s | |
++| Function Argument Memory Test | ✅ Pass | 2.727s | |
++| Function Response Memory Test | ✅ Pass | 3.028s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.871s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.942s | |
++| Penetration Testing Methodology | ✅ Pass | 10.503s | |
++| Vulnerability Assessment Tools | ✅ Pass | 8.797s | |
++| SQL Injection Attack Type | ✅ Pass | 3.950s | |
++| Penetration Testing Framework | ✅ Pass | 8.605s | |
++| Web Application Security Scanner | ✅ Pass | 6.065s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.288s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 4.197s
++**Average latency**: 3.629s
---
@@@ -459,38 -459,38 +472,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.495s | |
--| Text Transform Uppercase | ✅ Pass | 2.385s | |
--| Count from 1 to 5 | ✅ Pass | 2.851s | |
--| Math Calculation | ✅ Pass | 4.008s | |
--| Basic Echo Function | ✅ Pass | 3.840s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.631s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.243s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.520s | |
++| Simple Math | ✅ Pass | 1.758s | |
++| Text Transform Uppercase | ✅ Pass | 2.647s | |
++| Count from 1 to 5 | ✅ Pass | 1.854s | |
++| Math Calculation | ✅ Pass | 3.306s | |
++| Basic Echo Function | ✅ Pass | 2.677s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.142s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.545s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.636s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.584s | |
--| Search Query Function | ✅ Pass | 5.006s | |
--| Ask Advice Function | ✅ Pass | 3.081s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 5.535s | |
--| Basic Context Memory Test | ✅ Pass | 5.053s | |
--| Function Argument Memory Test | ✅ Pass | 2.839s | |
--| Function Response Memory Test | ✅ Pass | 5.648s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.765s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.324s | |
--| Penetration Testing Methodology | ✅ Pass | 6.697s | |
--| Vulnerability Assessment Tools | ✅ Pass | 8.405s | |
--| SQL Injection Attack Type | ✅ Pass | 7.206s | |
--| Penetration Testing Framework | ✅ Pass | 9.822s | |
--| Web Application Security Scanner | ✅ Pass | 6.015s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.145s | |
++| JSON Response Function | ✅ Pass | 7.368s | |
++| Search Query Function | ✅ Pass | 2.425s | |
++| Ask Advice Function | ✅ Pass | 3.898s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.308s | |
++| Basic Context Memory Test | ✅ Pass | 2.803s | |
++| Function Argument Memory Test | ✅ Pass | 3.239s | |
++| Function Response Memory Test | ✅ Pass | 2.782s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.988s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.995s | |
++| Penetration Testing Methodology | ✅ Pass | 8.775s | |
++| Vulnerability Assessment Tools | ✅ Pass | 8.485s | |
++| SQL Injection Attack Type | ✅ Pass | 3.628s | |
++| Penetration Testing Framework | ✅ Pass | 4.818s | |
++| Web Application Security Scanner | ✅ Pass | 5.829s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.611s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 4.483s
++**Average latency**: 3.719s
---
@@@ -500,38 -500,38 +513,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.531s | |
--| Text Transform Uppercase | ✅ Pass | 4.248s | |
--| Count from 1 to 5 | ✅ Pass | 2.429s | |
--| Math Calculation | ✅ Pass | 2.792s | |
--| Basic Echo Function | ✅ Pass | 3.709s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.008s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.826s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.912s | |
++| Simple Math | ✅ Pass | 2.756s | |
++| Text Transform Uppercase | ✅ Pass | 2.343s | |
++| Count from 1 to 5 | ✅ Pass | 4.949s | |
++| Math Calculation | ✅ Pass | 6.273s | |
++| Basic Echo Function | ✅ Pass | 9.546s | |
++| Streaming Simple Math Streaming | ✅ Pass | 9.831s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 10.218s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 4.962s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.994s | |
--| Search Query Function | ✅ Pass | 2.333s | |
--| Ask Advice Function | ✅ Pass | 6.841s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 4.218s | |
--| Basic Context Memory Test | ✅ Pass | 4.731s | |
--| Function Argument Memory Test | ✅ Pass | 3.151s | |
--| Function Response Memory Test | ✅ Pass | 3.061s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 6.495s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.016s | |
--| Penetration Testing Methodology | ✅ Pass | 11.347s | |
--| Vulnerability Assessment Tools | ✅ Pass | 7.938s | |
--| SQL Injection Attack Type | ✅ Pass | 3.653s | |
--| Penetration Testing Framework | ✅ Pass | 9.077s | |
--| Web Application Security Scanner | ✅ Pass | 6.831s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.679s | |
++| JSON Response Function | ✅ Pass | 3.745s | |
++| Search Query Function | ✅ Pass | 8.772s | |
++| Ask Advice Function | ✅ Pass | 2.384s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 7.628s | |
++| Basic Context Memory Test | ✅ Pass | 2.202s | |
++| Function Argument Memory Test | ✅ Pass | 7.457s | |
++| Function Response Memory Test | ✅ Pass | 4.223s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.302s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 5.027s | |
++| Penetration Testing Methodology | ✅ Pass | 8.014s | |
++| Vulnerability Assessment Tools | ✅ Pass | 7.896s | |
++| SQL Injection Attack Type | ✅ Pass | 2.703s | |
++| Penetration Testing Framework | ✅ Pass | 7.406s | |
++| Web Application Security Scanner | ✅ Pass | 5.370s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.377s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 4.427s
++**Average latency**: 5.713s
---
diff --cc examples/tests/deepinfra-report.md
index 1a52cef9,1a52cef9..2ed96b93
--- a/examples/tests/deepinfra-report.md
+++ b/examples/tests/deepinfra-report.md
@@@ -1,537 -1,537 +1,544 @@@
# LLM Agent Testing Report
--Generated: Tue, 30 Sep 2025 19:10:56 UTC
++Generated: Tue, 21 Jul 2026 21:30:02 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
--| simple | Qwen/Qwen3-Next-80B-A3B-Instruct | false | 23/23 (100.00%) | 1.284s |
--| simple_json | Qwen/Qwen3-Next-80B-A3B-Instruct | false | 5/5 (100.00%) | 1.261s |
--| primary_agent | moonshotai/Kimi-K2-Instruct-0905 | false | 22/23 (95.65%) | 1.406s |
--| assistant | moonshotai/Kimi-K2-Instruct-0905 | true | 21/23 (91.30%) | 1.397s |
--| generator | google/gemini-2.5-pro | true | 22/23 (95.65%) | 7.349s |
--| refiner | deepseek-ai/DeepSeek-R1-0528-Turbo | true | 22/23 (95.65%) | 4.424s |
--| adviser | google/gemini-2.5-pro | true | 23/23 (100.00%) | 6.986s |
--| reflector | Qwen/Qwen3-Next-80B-A3B-Instruct | true | 23/23 (100.00%) | 1.277s |
--| searcher | Qwen/Qwen3-32B | true | 23/23 (100.00%) | 6.780s |
--| enricher | Qwen/Qwen3-32B | true | 23/23 (100.00%) | 6.705s |
--| coder | anthropic/claude-4-sonnet | true | 23/23 (100.00%) | 2.953s |
--| installer | google/gemini-2.5-flash | true | 23/23 (100.00%) | 2.703s |
--| pentester | moonshotai/Kimi-K2-Instruct-0905 | true | 22/23 (95.65%) | 1.303s |
--
--**Total**: 275/281 (97.86%) successful tests
--**Overall average latency**: 3.670s
++| simple | MiniMaxAI/MiniMax-M3 | true | 23/23 (100.00%) | 0.213s |
++| simple_json | moonshotai/Kimi-K2.6 | true | 7/7 (100.00%) | 0.213s |
++| primary_agent | moonshotai/Kimi-K2.6 | true | 23/23 (100.00%) | 0.214s |
++| assistant | moonshotai/Kimi-K2.6 | true | 23/23 (100.00%) | 0.213s |
++| generator | zai-org/GLM-5.2 | true | 23/23 (100.00%) | 0.207s |
++| refiner | zai-org/GLM-5.2 | true | 23/23 (100.00%) | 0.214s |
++| adviser | MiniMaxAI/MiniMax-M3 | true | 23/23 (100.00%) | 0.216s |
++| reflector | MiniMaxAI/MiniMax-M3 | true | 23/23 (100.00%) | 0.217s |
++| searcher | MiniMaxAI/MiniMax-M3 | true | 23/23 (100.00%) | 0.217s |
++| enricher | MiniMaxAI/MiniMax-M3 | true | 23/23 (100.00%) | 0.213s |
++| coder | moonshotai/Kimi-K2.7-Code | true | 23/23 (100.00%) | 0.212s |
++| installer | moonshotai/Kimi-K2.7-Code | true | 23/23 (100.00%) | 0.214s |
++| pentester | moonshotai/Kimi-K2.6 | true | 23/23 (100.00%) | 0.210s |
++
++**Total**: 283/283 (100.00%) successful tests
++**Overall average latency**: 0.213s
## Detailed Results
--### simple (Qwen/Qwen3-Next-80B-A3B-Instruct)
++### simple (MiniMaxAI/MiniMax-M3)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.578s | |
--| Text Transform Uppercase | ✅ Pass | 2.580s | |
--| Count from 1 to 5 | ✅ Pass | 0.964s | |
--| Math Calculation | ✅ Pass | 0.900s | |
--| Basic Echo Function | ✅ Pass | 1.061s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.022s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.949s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.736s | |
++| Simple Math | ✅ Pass | 0.254s | |
++| Text Transform Uppercase | ✅ Pass | 0.218s | |
++| Count from 1 to 5 | ✅ Pass | 0.223s | |
++| Math Calculation | ✅ Pass | 0.210s | |
++| Basic Echo Function | ✅ Pass | 0.214s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.204s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.204s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.208s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.425s | |
--| Search Query Function | ✅ Pass | 1.348s | |
--| Ask Advice Function | ✅ Pass | 1.016s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.939s | |
--| Basic Context Memory Test | ✅ Pass | 1.248s | |
--| Function Argument Memory Test | ✅ Pass | 1.044s | |
--| Function Response Memory Test | ✅ Pass | 0.887s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.120s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.931s | |
--| Penetration Testing Methodology | ✅ Pass | 1.612s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.659s | |
--| SQL Injection Attack Type | ✅ Pass | 0.930s | |
--| Penetration Testing Framework | ✅ Pass | 1.154s | |
--| Web Application Security Scanner | ✅ Pass | 1.297s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.111s | |
++| JSON Response Function | ✅ Pass | 0.207s | |
++| Search Query Function | ✅ Pass | 0.203s | |
++| Ask Advice Function | ✅ Pass | 0.206s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.225s | |
++| Basic Context Memory Test | ✅ Pass | 0.210s | |
++| Function Argument Memory Test | ✅ Pass | 0.211s | |
++| Function Response Memory Test | ✅ Pass | 0.203s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.207s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.216s | |
++| Penetration Testing Methodology | ✅ Pass | 0.215s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.210s | |
++| SQL Injection Attack Type | ✅ Pass | 0.220s | |
++| Penetration Testing Framework | ✅ Pass | 0.201s | |
++| Web Application Security Scanner | ✅ Pass | 0.202s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.212s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.284s
++**Average latency**: 0.213s
---
--### simple_json (Qwen/Qwen3-Next-80B-A3B-Instruct)
++### simple_json (moonshotai/Kimi-K2.6)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Vulnerability Report Memory Test | ✅ Pass | 1.469s | |
--| Person Information JSON | ✅ Pass | 1.179s | |
--| User Profile JSON | ✅ Pass | 1.124s | |
--| Project Information JSON | ✅ Pass | 1.246s | |
--| Streaming Person Information JSON Streaming | ✅ Pass | 1.283s | |
++| Vulnerability Report Memory Test | ✅ Pass | 0.214s | |
++| Person Information JSON | ✅ Pass | 0.215s | |
++| Project Information JSON | ✅ Pass | 0.214s | |
++| User Profile JSON | ✅ Pass | 0.216s | |
++| JSON Array Response Without Schema | ✅ Pass | 0.207s | |
++| Streaming Person Information JSON Streaming | ✅ Pass | 0.209s | |
--**Summary**: 5/5 (100.00%) successful tests
++#### Capability Tests
--**Average latency**: 1.261s
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Structured Output With JSON Schema | structured_output | ✅ Pass | 0.211s | |
++
++**Summary**: 7/7 (100.00%) successful tests
++
++**Average latency**: 0.213s
---
--### primary_agent (moonshotai/Kimi-K2-Instruct-0905)
++### primary_agent (moonshotai/Kimi-K2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.603s | |
--| Text Transform Uppercase | ✅ Pass | 2.651s | |
--| Count from 1 to 5 | ✅ Pass | 0.755s | |
--| Math Calculation | ✅ Pass | 0.793s | |
--| Basic Echo Function | ✅ Pass | 1.418s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.783s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.653s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.644s | |
++| Simple Math | ✅ Pass | 0.261s | |
++| Text Transform Uppercase | ✅ Pass | 0.214s | |
++| Count from 1 to 5 | ✅ Pass | 0.211s | |
++| Math Calculation | ✅ Pass | 0.202s | |
++| Basic Echo Function | ✅ Pass | 0.202s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.206s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.226s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.215s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.157s | |
--| Search Query Function | ✅ Pass | 1.301s | |
--| Ask Advice Function | ✅ Pass | 1.400s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.985s | |
--| Basic Context Memory Test | ✅ Pass | 1.052s | |
--| Function Argument Memory Test | ✅ Pass | 1.118s | |
--| Function Response Memory Test | ✅ Pass | 0.731s | |
--| Penetration Testing Memory with Tool Call | ❌ Fail | 2.041s | no tool calls found, expected at least 1 |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.632s | |
--| Penetration Testing Methodology | ✅ Pass | 1.588s | |
--| Vulnerability Assessment Tools | ✅ Pass | 0.885s | |
--| SQL Injection Attack Type | ✅ Pass | 0.796s | |
--| Penetration Testing Framework | ✅ Pass | 4.317s | |
--| Web Application Security Scanner | ✅ Pass | 0.679s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.336s | |
--
--**Summary**: 22/23 (95.65%) successful tests
--
--**Average latency**: 1.406s
++| JSON Response Function | ✅ Pass | 0.203s | |
++| Search Query Function | ✅ Pass | 0.207s | |
++| Ask Advice Function | ✅ Pass | 0.212s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.230s | |
++| Basic Context Memory Test | ✅ Pass | 0.210s | |
++| Function Argument Memory Test | ✅ Pass | 0.209s | |
++| Function Response Memory Test | ✅ Pass | 0.210s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.208s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.210s | |
++| Penetration Testing Methodology | ✅ Pass | 0.215s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.212s | |
++| SQL Injection Attack Type | ✅ Pass | 0.221s | |
++| Penetration Testing Framework | ✅ Pass | 0.203s | |
++| Web Application Security Scanner | ✅ Pass | 0.207s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.207s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.214s
---
--### assistant (moonshotai/Kimi-K2-Instruct-0905)
++### assistant (moonshotai/Kimi-K2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.648s | |
--| Text Transform Uppercase | ✅ Pass | 2.672s | |
--| Count from 1 to 5 | ✅ Pass | 0.759s | |
--| Math Calculation | ✅ Pass | 0.772s | |
--| Basic Echo Function | ❌ Fail | 1.552s | no tool calls found, expected at least 1 |
--| Streaming Simple Math Streaming | ✅ Pass | 0.702s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.661s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.752s | |
++| Simple Math | ✅ Pass | 0.240s | |
++| Text Transform Uppercase | ✅ Pass | 0.219s | |
++| Count from 1 to 5 | ✅ Pass | 0.213s | |
++| Math Calculation | ✅ Pass | 0.207s | |
++| Basic Echo Function | ✅ Pass | 0.201s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.206s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.225s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.220s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.377s | |
--| Search Query Function | ✅ Pass | 2.002s | |
--| Ask Advice Function | ✅ Pass | 1.351s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.864s | |
--| Basic Context Memory Test | ✅ Pass | 1.075s | |
--| Function Argument Memory Test | ✅ Pass | 0.667s | |
--| Function Response Memory Test | ✅ Pass | 0.603s | |
--| Penetration Testing Memory with Tool Call | ❌ Fail | 2.146s | no tool calls found, expected at least 1 |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.626s | |
--| Penetration Testing Methodology | ✅ Pass | 1.616s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.152s | |
--| SQL Injection Attack Type | ✅ Pass | 0.886s | |
--| Penetration Testing Framework | ✅ Pass | 3.288s | |
--| Web Application Security Scanner | ✅ Pass | 0.714s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.229s | |
--
--**Summary**: 21/23 (91.30%) successful tests
--
--**Average latency**: 1.397s
++| JSON Response Function | ✅ Pass | 0.198s | |
++| Search Query Function | ✅ Pass | 0.208s | |
++| Ask Advice Function | ✅ Pass | 0.206s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.241s | |
++| Basic Context Memory Test | ✅ Pass | 0.219s | |
++| Function Argument Memory Test | ✅ Pass | 0.206s | |
++| Function Response Memory Test | ✅ Pass | 0.205s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.208s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.210s | |
++| Penetration Testing Methodology | ✅ Pass | 0.215s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.209s | |
++| SQL Injection Attack Type | ✅ Pass | 0.214s | |
++| Penetration Testing Framework | ✅ Pass | 0.208s | |
++| Web Application Security Scanner | ✅ Pass | 0.210s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.207s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.213s
---
--### generator (google/gemini-2.5-pro)
++### generator (zai-org/GLM-5.2)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 5.339s | |
--| Text Transform Uppercase | ✅ Pass | 8.288s | |
--| Math Calculation | ✅ Pass | 3.081s | |
--| Count from 1 to 5 | ✅ Pass | 5.671s | |
--| Streaming Simple Math Streaming | ✅ Pass | 4.258s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.547s | |
--| Basic Echo Function | ✅ Pass | 8.848s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.689s | |
++| Simple Math | ✅ Pass | 0.080s | |
++| Text Transform Uppercase | ✅ Pass | 0.217s | |
++| Count from 1 to 5 | ✅ Pass | 0.212s | |
++| Math Calculation | ✅ Pass | 0.222s | |
++| Basic Echo Function | ✅ Pass | 0.202s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.209s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.229s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.221s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Search Query Function | ✅ Pass | 4.440s | |
--| JSON Response Function | ✅ Pass | 8.128s | |
--| Ask Advice Function | ✅ Pass | 4.194s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 3.103s | |
--| Function Argument Memory Test | ✅ Pass | 4.669s | |
--| Basic Context Memory Test | ✅ Pass | 7.525s | |
--| Function Response Memory Test | ✅ Pass | 7.255s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 6.236s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 5.003s | |
--| Penetration Testing Methodology | ✅ Pass | 10.267s | |
--| SQL Injection Attack Type | ✅ Pass | 6.586s | |
--| Vulnerability Assessment Tools | ❌ Fail | 21.490s | expected text 'network' not found |
--| Penetration Testing Tool Selection | ✅ Pass | 4.211s | |
--| Penetration Testing Framework | ✅ Pass | 18.186s | |
--| Web Application Security Scanner | ✅ Pass | 15.008s | |
--
--**Summary**: 22/23 (95.65%) successful tests
--
--**Average latency**: 7.349s
++| JSON Response Function | ✅ Pass | 0.200s | |
++| Search Query Function | ✅ Pass | 0.206s | |
++| Ask Advice Function | ✅ Pass | 0.221s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.239s | |
++| Basic Context Memory Test | ✅ Pass | 0.212s | |
++| Function Argument Memory Test | ✅ Pass | 0.213s | |
++| Function Response Memory Test | ✅ Pass | 0.211s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.206s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.208s | |
++| Penetration Testing Methodology | ✅ Pass | 0.213s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.210s | |
++| SQL Injection Attack Type | ✅ Pass | 0.208s | |
++| Penetration Testing Framework | ✅ Pass | 0.205s | |
++| Web Application Security Scanner | ✅ Pass | 0.204s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.207s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.207s
---
--### refiner (deepseek-ai/DeepSeek-R1-0528-Turbo)
++### refiner (zai-org/GLM-5.2)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Text Transform Uppercase | ✅ Pass | 3.027s | |
--| Count from 1 to 5 | ✅ Pass | 1.407s | |
--| Simple Math | ✅ Pass | 5.472s | |
--| Math Calculation | ✅ Pass | 2.814s | |
--| Basic Echo Function | ✅ Pass | 2.816s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.297s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.343s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.435s | |
++| Simple Math | ✅ Pass | 0.219s | |
++| Text Transform Uppercase | ✅ Pass | 0.219s | |
++| Count from 1 to 5 | ✅ Pass | 0.225s | |
++| Math Calculation | ✅ Pass | 0.207s | |
++| Basic Echo Function | ✅ Pass | 0.209s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.205s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.241s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.221s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.203s | |
--| Search Query Function | ✅ Pass | 3.924s | |
--| Ask Advice Function | ✅ Pass | 2.326s | |
--| Streaming Search Query Function Streaming | ❌ Fail | 3.341s | no tool calls found, expected at least 1 |
--| Basic Context Memory Test | ✅ Pass | 4.284s | |
--| Function Argument Memory Test | ✅ Pass | 2.489s | |
--| Function Response Memory Test | ✅ Pass | 1.821s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.045s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 9.472s | |
--| Vulnerability Assessment Tools | ✅ Pass | 4.717s | |
--| Penetration Testing Methodology | ✅ Pass | 8.188s | |
--| SQL Injection Attack Type | ✅ Pass | 6.380s | |
--| Penetration Testing Framework | ✅ Pass | 10.979s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.450s | |
--| Web Application Security Scanner | ✅ Pass | 10.505s | |
--
--**Summary**: 22/23 (95.65%) successful tests
--
--**Average latency**: 4.424s
++| JSON Response Function | ✅ Pass | 0.198s | |
++| Search Query Function | ✅ Pass | 0.210s | |
++| Ask Advice Function | ✅ Pass | 0.220s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.235s | |
++| Basic Context Memory Test | ✅ Pass | 0.202s | |
++| Function Argument Memory Test | ✅ Pass | 0.210s | |
++| Function Response Memory Test | ✅ Pass | 0.211s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.213s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.206s | |
++| Penetration Testing Methodology | ✅ Pass | 0.217s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.205s | |
++| SQL Injection Attack Type | ✅ Pass | 0.213s | |
++| Penetration Testing Framework | ✅ Pass | 0.212s | |
++| Web Application Security Scanner | ✅ Pass | 0.205s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.212s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.214s
---
--### adviser (google/gemini-2.5-pro)
++### adviser (MiniMaxAI/MiniMax-M3)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 5.136s | |
--| Text Transform Uppercase | ✅ Pass | 5.091s | |
--| Count from 1 to 5 | ✅ Pass | 5.334s | |
--| Math Calculation | ✅ Pass | 3.554s | |
--| Streaming Simple Math Streaming | ✅ Pass | 4.277s | |
--| Basic Echo Function | ✅ Pass | 5.468s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.383s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 4.941s | |
++| Simple Math | ✅ Pass | 0.272s | |
++| Text Transform Uppercase | ✅ Pass | 0.219s | |
++| Count from 1 to 5 | ✅ Pass | 0.205s | |
++| Math Calculation | ✅ Pass | 0.215s | |
++| Basic Echo Function | ✅ Pass | 0.203s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.206s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.239s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.210s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Search Query Function | ✅ Pass | 4.618s | |
--| JSON Response Function | ✅ Pass | 8.534s | |
--| Ask Advice Function | ✅ Pass | 4.124s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 3.168s | |
--| Basic Context Memory Test | ✅ Pass | 5.123s | |
--| Function Argument Memory Test | ✅ Pass | 3.921s | |
--| Function Response Memory Test | ✅ Pass | 6.008s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.767s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 8.001s | |
--| Penetration Testing Methodology | ✅ Pass | 11.466s | |
--| SQL Injection Attack Type | ✅ Pass | 8.174s | |
--| Vulnerability Assessment Tools | ✅ Pass | 15.468s | |
--| Penetration Testing Tool Selection | ✅ Pass | 4.422s | |
--| Web Application Security Scanner | ✅ Pass | 15.610s | |
--| Penetration Testing Framework | ✅ Pass | 20.072s | |
++| JSON Response Function | ✅ Pass | 0.202s | |
++| Search Query Function | ✅ Pass | 0.234s | |
++| Ask Advice Function | ✅ Pass | 0.218s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.208s | |
++| Basic Context Memory Test | ✅ Pass | 0.203s | |
++| Function Argument Memory Test | ✅ Pass | 0.212s | |
++| Function Response Memory Test | ✅ Pass | 0.213s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.216s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.207s | |
++| Penetration Testing Methodology | ✅ Pass | 0.213s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.208s | |
++| SQL Injection Attack Type | ✅ Pass | 0.217s | |
++| Penetration Testing Framework | ✅ Pass | 0.208s | |
++| Web Application Security Scanner | ✅ Pass | 0.206s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.215s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 6.986s
++**Average latency**: 0.216s
---
--### reflector (Qwen/Qwen3-Next-80B-A3B-Instruct)
++### reflector (MiniMaxAI/MiniMax-M3)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.579s | |
--| Text Transform Uppercase | ✅ Pass | 1.261s | |
--| Count from 1 to 5 | ✅ Pass | 0.888s | |
--| Math Calculation | ✅ Pass | 0.900s | |
--| Basic Echo Function | ✅ Pass | 0.943s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.933s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.168s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.156s | |
++| Simple Math | ✅ Pass | 0.269s | |
++| Text Transform Uppercase | ✅ Pass | 0.204s | |
++| Count from 1 to 5 | ✅ Pass | 0.211s | |
++| Math Calculation | ✅ Pass | 0.210s | |
++| Basic Echo Function | ✅ Pass | 0.203s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.210s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.236s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.216s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.828s | |
--| Search Query Function | ✅ Pass | 1.027s | |
--| Ask Advice Function | ✅ Pass | 0.974s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.438s | |
--| Basic Context Memory Test | ✅ Pass | 0.978s | |
--| Function Argument Memory Test | ✅ Pass | 0.914s | |
--| Function Response Memory Test | ✅ Pass | 0.950s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.307s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.194s | |
--| Penetration Testing Methodology | ✅ Pass | 1.691s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.906s | |
--| SQL Injection Attack Type | ✅ Pass | 1.504s | |
--| Penetration Testing Framework | ✅ Pass | 1.183s | |
--| Web Application Security Scanner | ✅ Pass | 1.622s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.013s | |
++| JSON Response Function | ✅ Pass | 0.197s | |
++| Search Query Function | ✅ Pass | 0.245s | |
++| Ask Advice Function | ✅ Pass | 0.218s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.208s | |
++| Basic Context Memory Test | ✅ Pass | 0.207s | |
++| Function Argument Memory Test | ✅ Pass | 0.215s | |
++| Function Response Memory Test | ✅ Pass | 0.206s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.269s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.208s | |
++| Penetration Testing Methodology | ✅ Pass | 0.210s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.204s | |
++| SQL Injection Attack Type | ✅ Pass | 0.207s | |
++| Penetration Testing Framework | ✅ Pass | 0.210s | |
++| Web Application Security Scanner | ✅ Pass | 0.210s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.210s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.277s
++**Average latency**: 0.217s
---
--### searcher (Qwen/Qwen3-32B)
++### searcher (MiniMaxAI/MiniMax-M3)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 5.632s | |
--| Text Transform Uppercase | ✅ Pass | 5.010s | |
--| Count from 1 to 5 | ✅ Pass | 4.092s | |
--| Basic Echo Function | ✅ Pass | 3.654s | |
--| Math Calculation | ✅ Pass | 6.467s | |
--| Streaming Simple Math Streaming | ✅ Pass | 5.738s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.966s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.840s | |
++| Simple Math | ✅ Pass | 0.251s | |
++| Text Transform Uppercase | ✅ Pass | 0.218s | |
++| Count from 1 to 5 | ✅ Pass | 0.227s | |
++| Math Calculation | ✅ Pass | 0.217s | |
++| Basic Echo Function | ✅ Pass | 0.199s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.225s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.239s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.213s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 5.198s | |
--| Search Query Function | ✅ Pass | 3.664s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 4.344s | |
--| Ask Advice Function | ✅ Pass | 6.628s | |
--| Function Argument Memory Test | ✅ Pass | 4.112s | |
--| Basic Context Memory Test | ✅ Pass | 7.697s | |
--| Function Response Memory Test | ✅ Pass | 4.331s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.918s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 7.309s | |
--| Penetration Testing Methodology | ✅ Pass | 12.251s | |
--| SQL Injection Attack Type | ✅ Pass | 9.998s | |
--| Vulnerability Assessment Tools | ✅ Pass | 17.638s | |
--| Penetration Testing Framework | ✅ Pass | 12.965s | |
--| Web Application Security Scanner | ✅ Pass | 10.203s | |
--| Penetration Testing Tool Selection | ✅ Pass | 5.265s | |
++| JSON Response Function | ✅ Pass | 0.209s | |
++| Search Query Function | ✅ Pass | 0.236s | |
++| Ask Advice Function | ✅ Pass | 0.212s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.207s | |
++| Basic Context Memory Test | ✅ Pass | 0.214s | |
++| Function Argument Memory Test | ✅ Pass | 0.214s | |
++| Function Response Memory Test | ✅ Pass | 0.211s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.205s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.209s | |
++| Penetration Testing Methodology | ✅ Pass | 0.213s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.210s | |
++| SQL Injection Attack Type | ✅ Pass | 0.211s | |
++| Penetration Testing Framework | ✅ Pass | 0.215s | |
++| Web Application Security Scanner | ✅ Pass | 0.209s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.211s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 6.780s
++**Average latency**: 0.217s
---
--### enricher (Qwen/Qwen3-32B)
++### enricher (MiniMaxAI/MiniMax-M3)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Text Transform Uppercase | ✅ Pass | 5.822s | |
--| Simple Math | ✅ Pass | 8.812s | |
--| Count from 1 to 5 | ✅ Pass | 4.942s | |
--| Math Calculation | ✅ Pass | 5.295s | |
--| Basic Echo Function | ✅ Pass | 3.634s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.727s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.957s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 7.226s | |
++| Simple Math | ✅ Pass | 0.203s | |
++| Text Transform Uppercase | ✅ Pass | 0.222s | |
++| Count from 1 to 5 | ✅ Pass | 0.218s | |
++| Math Calculation | ✅ Pass | 0.209s | |
++| Basic Echo Function | ✅ Pass | 0.206s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.212s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.228s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.214s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Search Query Function | ✅ Pass | 3.308s | |
--| JSON Response Function | ✅ Pass | 7.863s | |
--| Ask Advice Function | ✅ Pass | 4.381s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 4.571s | |
--| Basic Context Memory Test | ✅ Pass | 7.343s | |
--| Function Response Memory Test | ✅ Pass | 4.464s | |
--| Function Argument Memory Test | ✅ Pass | 6.395s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 7.066s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 5.124s | |
--| Penetration Testing Methodology | ✅ Pass | 11.162s | |
--| Vulnerability Assessment Tools | ✅ Pass | 11.898s | |
--| SQL Injection Attack Type | ✅ Pass | 8.524s | |
--| Penetration Testing Framework | ✅ Pass | 11.747s | |
--| Web Application Security Scanner | ✅ Pass | 10.317s | |
--| Penetration Testing Tool Selection | ✅ Pass | 6.616s | |
++| JSON Response Function | ✅ Pass | 0.210s | |
++| Search Query Function | ✅ Pass | 0.232s | |
++| Ask Advice Function | ✅ Pass | 0.205s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.206s | |
++| Basic Context Memory Test | ✅ Pass | 0.216s | |
++| Function Argument Memory Test | ✅ Pass | 0.214s | |
++| Function Response Memory Test | ✅ Pass | 0.213s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.212s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.205s | |
++| Penetration Testing Methodology | ✅ Pass | 0.209s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.210s | |
++| SQL Injection Attack Type | ✅ Pass | 0.205s | |
++| Penetration Testing Framework | ✅ Pass | 0.207s | |
++| Web Application Security Scanner | ✅ Pass | 0.209s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.211s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 6.705s
++**Average latency**: 0.213s
---
--### coder (anthropic/claude-4-sonnet)
++### coder (moonshotai/Kimi-K2.7-Code)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 4.038s | |
--| Text Transform Uppercase | ✅ Pass | 1.810s | |
--| Count from 1 to 5 | ✅ Pass | 1.872s | |
--| Math Calculation | ✅ Pass | 2.400s | |
--| Basic Echo Function | ✅ Pass | 2.111s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.348s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.517s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.253s | |
++| Simple Math | ✅ Pass | 0.207s | |
++| Text Transform Uppercase | ✅ Pass | 0.219s | |
++| Count from 1 to 5 | ✅ Pass | 0.214s | |
++| Math Calculation | ✅ Pass | 0.211s | |
++| Basic Echo Function | ✅ Pass | 0.219s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.220s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.217s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.206s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.561s | |
--| Search Query Function | ✅ Pass | 2.016s | |
--| Ask Advice Function | ✅ Pass | 2.363s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.742s | |
--| Basic Context Memory Test | ✅ Pass | 2.941s | |
--| Function Argument Memory Test | ✅ Pass | 1.764s | |
--| Function Response Memory Test | ✅ Pass | 1.999s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.912s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.101s | |
--| Penetration Testing Methodology | ✅ Pass | 5.195s | |
--| Vulnerability Assessment Tools | ✅ Pass | 5.804s | |
--| SQL Injection Attack Type | ✅ Pass | 2.188s | |
--| Penetration Testing Framework | ✅ Pass | 5.985s | |
--| Web Application Security Scanner | ✅ Pass | 4.381s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.602s | |
++| JSON Response Function | ✅ Pass | 0.207s | |
++| Search Query Function | ✅ Pass | 0.216s | |
++| Ask Advice Function | ✅ Pass | 0.217s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.208s | |
++| Basic Context Memory Test | ✅ Pass | 0.215s | |
++| Function Argument Memory Test | ✅ Pass | 0.213s | |
++| Function Response Memory Test | ✅ Pass | 0.214s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.213s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.217s | |
++| Penetration Testing Methodology | ✅ Pass | 0.207s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.210s | |
++| SQL Injection Attack Type | ✅ Pass | 0.206s | |
++| Penetration Testing Framework | ✅ Pass | 0.204s | |
++| Web Application Security Scanner | ✅ Pass | 0.203s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.211s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.953s
++**Average latency**: 0.212s
---
--### installer (google/gemini-2.5-flash)
++### installer (moonshotai/Kimi-K2.7-Code)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.469s | |
--| Text Transform Uppercase | ✅ Pass | 2.376s | |
--| Count from 1 to 5 | ✅ Pass | 1.039s | |
--| Math Calculation | ✅ Pass | 1.579s | |
--| Basic Echo Function | ✅ Pass | 1.417s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.214s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.954s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.251s | |
++| Simple Math | ✅ Pass | 0.208s | |
++| Text Transform Uppercase | ✅ Pass | 0.226s | |
++| Count from 1 to 5 | ✅ Pass | 0.224s | |
++| Math Calculation | ✅ Pass | 0.211s | |
++| Basic Echo Function | ✅ Pass | 0.213s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.216s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.216s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.211s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.778s | |
--| Search Query Function | ✅ Pass | 2.765s | |
--| Ask Advice Function | ✅ Pass | 2.787s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.896s | |
--| Basic Context Memory Test | ✅ Pass | 3.006s | |
--| Function Argument Memory Test | ✅ Pass | 1.224s | |
--| Function Response Memory Test | ✅ Pass | 1.743s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.311s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.248s | |
--| Penetration Testing Methodology | ✅ Pass | 2.712s | |
--| SQL Injection Attack Type | ✅ Pass | 1.453s | |
--| Vulnerability Assessment Tools | ✅ Pass | 14.593s | |
--| Penetration Testing Framework | ✅ Pass | 4.916s | |
--| Web Application Security Scanner | ✅ Pass | 3.753s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.673s | |
++| JSON Response Function | ✅ Pass | 0.209s | |
++| Search Query Function | ✅ Pass | 0.218s | |
++| Ask Advice Function | ✅ Pass | 0.213s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.232s | |
++| Basic Context Memory Test | ✅ Pass | 0.212s | |
++| Function Argument Memory Test | ✅ Pass | 0.203s | |
++| Function Response Memory Test | ✅ Pass | 0.209s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.215s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.219s | |
++| Penetration Testing Methodology | ✅ Pass | 0.207s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.208s | |
++| SQL Injection Attack Type | ✅ Pass | 0.213s | |
++| Penetration Testing Framework | ✅ Pass | 0.216s | |
++| Web Application Security Scanner | ✅ Pass | 0.204s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.212s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.703s
++**Average latency**: 0.214s
---
--### pentester (moonshotai/Kimi-K2-Instruct-0905)
++### pentester (moonshotai/Kimi-K2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.661s | |
--| Text Transform Uppercase | ✅ Pass | 0.641s | |
--| Count from 1 to 5 | ✅ Pass | 1.036s | |
--| Math Calculation | ✅ Pass | 0.594s | |
--| Basic Echo Function | ✅ Pass | 1.246s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.594s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.479s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.881s | |
++| Simple Math | ✅ Pass | 0.208s | |
++| Text Transform Uppercase | ✅ Pass | 0.228s | |
++| Count from 1 to 5 | ✅ Pass | 0.212s | |
++| Math Calculation | ✅ Pass | 0.219s | |
++| Basic Echo Function | ✅ Pass | 0.202s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.207s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.207s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.217s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.316s | |
--| Search Query Function | ✅ Pass | 1.443s | |
--| Ask Advice Function | ✅ Pass | 1.316s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.617s | |
--| Basic Context Memory Test | ✅ Pass | 0.840s | |
--| Function Argument Memory Test | ✅ Pass | 0.659s | |
--| Function Response Memory Test | ✅ Pass | 0.611s | |
--| Penetration Testing Memory with Tool Call | ❌ Fail | 2.006s | no tool calls found, expected at least 1 |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.633s | |
--| Penetration Testing Methodology | ✅ Pass | 1.000s | |
--| Vulnerability Assessment Tools | ✅ Pass | 0.791s | |
--| SQL Injection Attack Type | ✅ Pass | 5.163s | |
--| Penetration Testing Framework | ✅ Pass | 0.761s | |
--| Web Application Security Scanner | ✅ Pass | 0.598s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.073s | |
--
--**Summary**: 22/23 (95.65%) successful tests
--
--**Average latency**: 1.303s
++| JSON Response Function | ✅ Pass | 0.203s | |
++| Search Query Function | ✅ Pass | 0.205s | |
++| Ask Advice Function | ✅ Pass | 0.211s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.207s | |
++| Basic Context Memory Test | ✅ Pass | 0.214s | |
++| Function Argument Memory Test | ✅ Pass | 0.204s | |
++| Function Response Memory Test | ✅ Pass | 0.206s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.212s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.210s | |
++| Penetration Testing Methodology | ✅ Pass | 0.208s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.212s | |
++| SQL Injection Attack Type | ✅ Pass | 0.206s | |
++| Penetration Testing Framework | ✅ Pass | 0.205s | |
++| Web Application Security Scanner | ✅ Pass | 0.210s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.209s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.210s
---
diff --cc examples/tests/deepseek-report.md
index 662aa93c,662aa93c..88f20760
--- a/examples/tests/deepseek-report.md
+++ b/examples/tests/deepseek-report.md
@@@ -1,27 -1,27 +1,27 @@@
# LLM Agent Testing Report
--Generated: Wed, 27 May 2026 22:59:39 UTC
++Generated: Tue, 21 Jul 2026 16:50:46 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
--| simple | deepseek-v4-flash | false | 23/23 (100.00%) | 1.039s |
--| simple_json | deepseek-v4-flash | false | 5/5 (100.00%) | 0.989s |
--| primary_agent | deepseek-v4-pro | true | 23/23 (100.00%) | 2.896s |
--| assistant | deepseek-v4-pro | true | 22/23 (95.65%) | 2.929s |
--| generator | deepseek-v4-pro | true | 23/23 (100.00%) | 3.046s |
--| refiner | deepseek-v4-pro | true | 23/23 (100.00%) | 2.932s |
--| adviser | deepseek-v4-pro | true | 23/23 (100.00%) | 2.667s |
--| reflector | deepseek-v4-flash | true | 23/23 (100.00%) | 1.288s |
--| searcher | deepseek-v4-flash | true | 23/23 (100.00%) | 0.991s |
--| enricher | deepseek-v4-flash | true | 23/23 (100.00%) | 0.981s |
--| coder | deepseek-v4-pro | true | 23/23 (100.00%) | 2.817s |
--| installer | deepseek-v4-flash | true | 23/23 (100.00%) | 1.246s |
--| pentester | deepseek-v4-pro | true | 23/23 (100.00%) | 3.086s |
--
--**Total**: 280/281 (99.64%) successful tests
--**Overall average latency**: 2.139s
++| simple | deepseek-v4-flash | true | 23/23 (100.00%) | 0.225s |
++| simple_json | deepseek-v4-flash | true | 6/7 (85.71%) | 0.359s |
++| primary_agent | deepseek-v4-pro | true | 23/23 (100.00%) | 0.222s |
++| assistant | deepseek-v4-pro | true | 23/23 (100.00%) | 0.219s |
++| generator | deepseek-v4-pro | true | 23/23 (100.00%) | 0.222s |
++| refiner | deepseek-v4-pro | true | 23/23 (100.00%) | 0.220s |
++| adviser | deepseek-v4-pro | true | 23/23 (100.00%) | 0.213s |
++| reflector | deepseek-v4-flash | true | 23/23 (100.00%) | 0.218s |
++| searcher | deepseek-v4-flash | true | 23/23 (100.00%) | 0.211s |
++| enricher | deepseek-v4-flash | true | 23/23 (100.00%) | 0.213s |
++| coder | deepseek-v4-pro | true | 23/23 (100.00%) | 0.217s |
++| installer | deepseek-v4-flash | true | 23/23 (100.00%) | 0.217s |
++| pentester | deepseek-v4-pro | true | 23/23 (100.00%) | 0.224s |
++
++**Total**: 282/283 (99.65%) successful tests
++**Overall average latency**: 0.222s
## Detailed Results
@@@ -31,38 -31,38 +31,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.973s | |
--| Text Transform Uppercase | ✅ Pass | 0.709s | |
--| Count from 1 to 5 | ✅ Pass | 0.903s | |
--| Math Calculation | ✅ Pass | 0.698s | |
--| Basic Echo Function | ✅ Pass | 1.075s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.775s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.871s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.081s | |
++| Simple Math | ✅ Pass | 0.389s | |
++| Text Transform Uppercase | ✅ Pass | 0.231s | |
++| Count from 1 to 5 | ✅ Pass | 0.209s | |
++| Math Calculation | ✅ Pass | 0.207s | |
++| Basic Echo Function | ✅ Pass | 0.210s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.205s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.222s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.210s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.254s | |
--| Search Query Function | ✅ Pass | 0.963s | |
--| Ask Advice Function | ✅ Pass | 1.142s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.976s | |
--| Basic Context Memory Test | ✅ Pass | 0.768s | |
--| Function Argument Memory Test | ✅ Pass | 0.865s | |
--| Function Response Memory Test | ✅ Pass | 0.782s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.414s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.847s | |
--| Penetration Testing Methodology | ✅ Pass | 1.116s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.258s | |
--| SQL Injection Attack Type | ✅ Pass | 0.876s | |
--| Penetration Testing Framework | ✅ Pass | 1.179s | |
--| Web Application Security Scanner | ✅ Pass | 1.190s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.180s | |
++| JSON Response Function | ✅ Pass | 0.212s | |
++| Search Query Function | ✅ Pass | 0.211s | |
++| Ask Advice Function | ✅ Pass | 0.206s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.210s | |
++| Basic Context Memory Test | ✅ Pass | 0.205s | |
++| Function Argument Memory Test | ✅ Pass | 0.206s | |
++| Function Response Memory Test | ✅ Pass | 0.205s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.281s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.210s | |
++| Penetration Testing Methodology | ✅ Pass | 0.267s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.165s | |
++| SQL Injection Attack Type | ✅ Pass | 0.199s | |
++| Penetration Testing Framework | ✅ Pass | 0.208s | |
++| Web Application Security Scanner | ✅ Pass | 0.173s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.334s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.039s
++**Average latency**: 0.225s
---
@@@ -72,15 -72,15 +72,22 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Person Information JSON | ✅ Pass | 0.872s | |
--| Project Information JSON | ✅ Pass | 0.944s | |
--| Vulnerability Report Memory Test | ✅ Pass | 1.059s | |
--| User Profile JSON | ✅ Pass | 0.976s | |
--| Streaming Person Information JSON Streaming | ✅ Pass | 1.093s | |
++| Vulnerability Report Memory Test | ✅ Pass | 0.374s | |
++| Person Information JSON | ✅ Pass | 0.368s | |
++| Project Information JSON | ✅ Pass | 0.365s | |
++| User Profile JSON | ✅ Pass | 0.309s | |
++| JSON Array Response Without Schema | ✅ Pass | 0.248s | |
++| Streaming Person Information JSON Streaming | ✅ Pass | 0.287s | |
--**Summary**: 5/5 (100.00%) successful tests
++#### Capability Tests
--**Average latency**: 0.989s
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Structured Output With JSON Schema | structured_output | ❌ Fail | 0.556s | API returned unexpected status code: 400: litellm\.BadRequestError: DeepseekException \- \{"error":\{"message":"This response\_format type is unava... |
++
++**Summary**: 6/7 (85.71%) successful tests
++
++**Average latency**: 0.359s
---
@@@ -90,38 -90,38 +97,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.178s | |
--| Text Transform Uppercase | ✅ Pass | 1.700s | |
--| Count from 1 to 5 | ✅ Pass | 4.825s | |
--| Math Calculation | ✅ Pass | 1.798s | |
--| Basic Echo Function | ✅ Pass | 2.080s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.931s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.996s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.133s | |
++| Simple Math | ✅ Pass | 0.422s | |
++| Text Transform Uppercase | ✅ Pass | 0.236s | |
++| Count from 1 to 5 | ✅ Pass | 0.205s | |
++| Math Calculation | ✅ Pass | 0.206s | |
++| Basic Echo Function | ✅ Pass | 0.205s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.203s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.213s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.210s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.657s | |
--| Search Query Function | ✅ Pass | 2.248s | |
--| Ask Advice Function | ✅ Pass | 2.200s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.060s | |
--| Basic Context Memory Test | ✅ Pass | 3.547s | |
--| Function Argument Memory Test | ✅ Pass | 2.111s | |
--| Function Response Memory Test | ✅ Pass | 2.269s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.453s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.267s | |
--| Penetration Testing Methodology | ✅ Pass | 4.575s | |
--| Vulnerability Assessment Tools | ✅ Pass | 7.988s | |
--| SQL Injection Attack Type | ✅ Pass | 2.190s | |
--| Penetration Testing Framework | ✅ Pass | 5.697s | |
--| Web Application Security Scanner | ✅ Pass | 2.219s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.468s | |
++| JSON Response Function | ✅ Pass | 0.204s | |
++| Search Query Function | ✅ Pass | 0.207s | |
++| Ask Advice Function | ✅ Pass | 0.206s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.211s | |
++| Basic Context Memory Test | ✅ Pass | 0.203s | |
++| Function Argument Memory Test | ✅ Pass | 0.213s | |
++| Function Response Memory Test | ✅ Pass | 0.208s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.212s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.204s | |
++| Penetration Testing Methodology | ✅ Pass | 0.250s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.188s | |
++| SQL Injection Attack Type | ✅ Pass | 0.198s | |
++| Penetration Testing Framework | ✅ Pass | 0.185s | |
++| Web Application Security Scanner | ✅ Pass | 0.178s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.315s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.896s
++**Average latency**: 0.222s
---
@@@ -131,38 -131,38 +138,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.123s | |
--| Text Transform Uppercase | ✅ Pass | 1.972s | |
--| Count from 1 to 5 | ✅ Pass | 6.415s | |
--| Math Calculation | ✅ Pass | 1.814s | |
--| Basic Echo Function | ✅ Pass | 2.143s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.970s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.243s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.283s | |
++| Simple Math | ✅ Pass | 0.399s | |
++| Text Transform Uppercase | ✅ Pass | 0.239s | |
++| Count from 1 to 5 | ✅ Pass | 0.204s | |
++| Math Calculation | ✅ Pass | 0.207s | |
++| Basic Echo Function | ✅ Pass | 0.205s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.205s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.205s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.203s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.509s | |
--| Search Query Function | ✅ Pass | 2.421s | |
--| Ask Advice Function | ✅ Pass | 2.316s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.957s | |
--| Basic Context Memory Test | ✅ Pass | 2.203s | |
--| Function Argument Memory Test | ✅ Pass | 2.293s | |
--| Function Response Memory Test | ✅ Pass | 2.042s | |
--| Penetration Testing Memory with Tool Call | ❌ Fail | 3.318s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.571s | |
--| Penetration Testing Methodology | ✅ Pass | 3.431s | |
--| SQL Injection Attack Type | ✅ Pass | 2.289s | |
--| Vulnerability Assessment Tools | ✅ Pass | 10.808s | |
--| Penetration Testing Framework | ✅ Pass | 3.233s | |
--| Web Application Security Scanner | ✅ Pass | 3.258s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.752s | |
--
--**Summary**: 22/23 (95.65%) successful tests
--
--**Average latency**: 2.929s
++| JSON Response Function | ✅ Pass | 0.210s | |
++| Search Query Function | ✅ Pass | 0.210s | |
++| Ask Advice Function | ✅ Pass | 0.206s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.209s | |
++| Basic Context Memory Test | ✅ Pass | 0.208s | |
++| Function Argument Memory Test | ✅ Pass | 0.209s | |
++| Function Response Memory Test | ✅ Pass | 0.200s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.212s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.213s | |
++| Penetration Testing Methodology | ✅ Pass | 0.236s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.209s | |
++| SQL Injection Attack Type | ✅ Pass | 0.193s | |
++| Penetration Testing Framework | ✅ Pass | 0.137s | |
++| Web Application Security Scanner | ✅ Pass | 0.202s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.312s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.219s
---
@@@ -172,38 -172,38 +179,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.071s | |
--| Text Transform Uppercase | ✅ Pass | 2.231s | |
--| Count from 1 to 5 | ✅ Pass | 6.367s | |
--| Math Calculation | ✅ Pass | 1.514s | |
--| Basic Echo Function | ✅ Pass | 1.995s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.534s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.143s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.014s | |
++| Simple Math | ✅ Pass | 0.389s | |
++| Text Transform Uppercase | ✅ Pass | 0.238s | |
++| Count from 1 to 5 | ✅ Pass | 0.206s | |
++| Math Calculation | ✅ Pass | 0.209s | |
++| Basic Echo Function | ✅ Pass | 0.205s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.204s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.220s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.206s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.431s | |
--| Search Query Function | ✅ Pass | 2.231s | |
--| Ask Advice Function | ✅ Pass | 2.042s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.115s | |
--| Basic Context Memory Test | ✅ Pass | 2.382s | |
--| Function Argument Memory Test | ✅ Pass | 2.525s | |
--| Function Response Memory Test | ✅ Pass | 2.067s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.271s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.383s | |
--| Penetration Testing Methodology | ✅ Pass | 4.247s | |
--| SQL Injection Attack Type | ✅ Pass | 2.293s | |
--| Vulnerability Assessment Tools | ✅ Pass | 12.485s | |
--| Penetration Testing Framework | ✅ Pass | 4.695s | |
--| Web Application Security Scanner | ✅ Pass | 2.208s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.794s | |
++| JSON Response Function | ✅ Pass | 0.205s | |
++| Search Query Function | ✅ Pass | 0.210s | |
++| Ask Advice Function | ✅ Pass | 0.206s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.206s | |
++| Basic Context Memory Test | ✅ Pass | 0.210s | |
++| Function Argument Memory Test | ✅ Pass | 0.206s | |
++| Function Response Memory Test | ✅ Pass | 0.208s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.214s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.208s | |
++| Penetration Testing Methodology | ✅ Pass | 0.265s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.204s | |
++| SQL Injection Attack Type | ✅ Pass | 0.197s | |
++| Penetration Testing Framework | ✅ Pass | 0.169s | |
++| Web Application Security Scanner | ✅ Pass | 0.210s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.300s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 3.046s
++**Average latency**: 0.222s
---
@@@ -213,38 -213,38 +220,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.115s | |
--| Text Transform Uppercase | ✅ Pass | 1.996s | |
--| Count from 1 to 5 | ✅ Pass | 4.321s | |
--| Math Calculation | ✅ Pass | 1.722s | |
--| Basic Echo Function | ✅ Pass | 2.028s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.742s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.044s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.170s | |
++| Simple Math | ✅ Pass | 0.419s | |
++| Text Transform Uppercase | ✅ Pass | 0.204s | |
++| Count from 1 to 5 | ✅ Pass | 0.210s | |
++| Math Calculation | ✅ Pass | 0.210s | |
++| Basic Echo Function | ✅ Pass | 0.205s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.214s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.212s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.212s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.374s | |
--| Search Query Function | ✅ Pass | 2.454s | |
--| Ask Advice Function | ✅ Pass | 2.166s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.974s | |
--| Basic Context Memory Test | ✅ Pass | 3.341s | |
--| Function Argument Memory Test | ✅ Pass | 2.086s | |
--| Function Response Memory Test | ✅ Pass | 2.400s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.446s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.015s | |
--| Penetration Testing Methodology | ✅ Pass | 3.273s | |
--| SQL Injection Attack Type | ✅ Pass | 3.515s | |
--| Vulnerability Assessment Tools | ✅ Pass | 11.480s | |
--| Penetration Testing Framework | ✅ Pass | 4.129s | |
--| Web Application Security Scanner | ✅ Pass | 2.298s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.323s | |
++| JSON Response Function | ✅ Pass | 0.206s | |
++| Search Query Function | ✅ Pass | 0.206s | |
++| Ask Advice Function | ✅ Pass | 0.207s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.207s | |
++| Basic Context Memory Test | ✅ Pass | 0.206s | |
++| Function Argument Memory Test | ✅ Pass | 0.210s | |
++| Function Response Memory Test | ✅ Pass | 0.203s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.217s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.205s | |
++| Penetration Testing Methodology | ✅ Pass | 0.256s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.194s | |
++| SQL Injection Attack Type | ✅ Pass | 0.183s | |
++| Penetration Testing Framework | ✅ Pass | 0.170s | |
++| Web Application Security Scanner | ✅ Pass | 0.203s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.287s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.932s
++**Average latency**: 0.220s
---
@@@ -254,38 -254,38 +261,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.074s | |
--| Text Transform Uppercase | ✅ Pass | 1.958s | |
--| Count from 1 to 5 | ✅ Pass | 4.874s | |
--| Math Calculation | ✅ Pass | 1.681s | |
--| Basic Echo Function | ✅ Pass | 2.290s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.667s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.040s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.949s | |
++| Simple Math | ✅ Pass | 0.389s | |
++| Text Transform Uppercase | ✅ Pass | 0.208s | |
++| Count from 1 to 5 | ✅ Pass | 0.214s | |
++| Math Calculation | ✅ Pass | 0.210s | |
++| Basic Echo Function | ✅ Pass | 0.208s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.202s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.218s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.213s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.484s | |
--| Search Query Function | ✅ Pass | 2.248s | |
--| Ask Advice Function | ✅ Pass | 1.988s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.986s | |
--| Basic Context Memory Test | ✅ Pass | 2.873s | |
--| Function Argument Memory Test | ✅ Pass | 2.211s | |
--| Function Response Memory Test | ✅ Pass | 1.874s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.502s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.064s | |
--| Penetration Testing Methodology | ✅ Pass | 3.537s | |
--| Vulnerability Assessment Tools | ✅ Pass | 5.184s | |
--| SQL Injection Attack Type | ✅ Pass | 4.424s | |
--| Penetration Testing Framework | ✅ Pass | 3.939s | |
--| Web Application Security Scanner | ✅ Pass | 2.206s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.267s | |
++| JSON Response Function | ✅ Pass | 0.207s | |
++| Search Query Function | ✅ Pass | 0.206s | |
++| Ask Advice Function | ✅ Pass | 0.207s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.203s | |
++| Basic Context Memory Test | ✅ Pass | 0.207s | |
++| Function Argument Memory Test | ✅ Pass | 0.209s | |
++| Function Response Memory Test | ✅ Pass | 0.209s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.221s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.214s | |
++| Penetration Testing Methodology | ✅ Pass | 0.213s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.200s | |
++| SQL Injection Attack Type | ✅ Pass | 0.136s | |
++| Penetration Testing Framework | ✅ Pass | 0.193s | |
++| Web Application Security Scanner | ✅ Pass | 0.197s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.215s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.667s
++**Average latency**: 0.213s
---
@@@ -295,38 -295,38 +302,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.798s | |
--| Text Transform Uppercase | ✅ Pass | 0.888s | |
--| Count from 1 to 5 | ✅ Pass | 7.083s | |
--| Math Calculation | ✅ Pass | 0.887s | |
--| Basic Echo Function | ✅ Pass | 1.023s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.759s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.762s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.272s | |
++| Simple Math | ✅ Pass | 0.423s | |
++| Text Transform Uppercase | ✅ Pass | 0.221s | |
++| Count from 1 to 5 | ✅ Pass | 0.216s | |
++| Math Calculation | ✅ Pass | 0.213s | |
++| Basic Echo Function | ✅ Pass | 0.208s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.219s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.226s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.217s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.026s | |
--| Search Query Function | ✅ Pass | 0.935s | |
--| Ask Advice Function | ✅ Pass | 1.016s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.064s | |
--| Basic Context Memory Test | ✅ Pass | 0.733s | |
--| Function Argument Memory Test | ✅ Pass | 0.869s | |
--| Function Response Memory Test | ✅ Pass | 0.791s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.358s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.853s | |
--| Penetration Testing Methodology | ✅ Pass | 1.117s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.876s | |
--| SQL Injection Attack Type | ✅ Pass | 0.844s | |
--| Penetration Testing Framework | ✅ Pass | 1.286s | |
--| Web Application Security Scanner | ✅ Pass | 1.159s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.207s | |
++| JSON Response Function | ✅ Pass | 0.204s | |
++| Search Query Function | ✅ Pass | 0.206s | |
++| Ask Advice Function | ✅ Pass | 0.203s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.203s | |
++| Basic Context Memory Test | ✅ Pass | 0.203s | |
++| Function Argument Memory Test | ✅ Pass | 0.202s | |
++| Function Response Memory Test | ✅ Pass | 0.209s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.221s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.201s | |
++| Penetration Testing Methodology | ✅ Pass | 0.228s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.208s | |
++| SQL Injection Attack Type | ✅ Pass | 0.168s | |
++| Penetration Testing Framework | ✅ Pass | 0.192s | |
++| Web Application Security Scanner | ✅ Pass | 0.202s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.196s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.288s
++**Average latency**: 0.218s
---
@@@ -336,38 -336,38 +343,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.813s | |
--| Text Transform Uppercase | ✅ Pass | 0.770s | |
--| Count from 1 to 5 | ✅ Pass | 0.893s | |
--| Math Calculation | ✅ Pass | 0.742s | |
--| Basic Echo Function | ✅ Pass | 0.971s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.695s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.929s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.123s | |
++| Simple Math | ✅ Pass | 0.413s | |
++| Text Transform Uppercase | ✅ Pass | 0.219s | |
++| Count from 1 to 5 | ✅ Pass | 0.217s | |
++| Math Calculation | ✅ Pass | 0.210s | |
++| Basic Echo Function | ✅ Pass | 0.210s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.218s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.219s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.215s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.010s | |
--| Search Query Function | ✅ Pass | 0.922s | |
--| Ask Advice Function | ✅ Pass | 1.081s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.909s | |
--| Basic Context Memory Test | ✅ Pass | 0.755s | |
--| Function Argument Memory Test | ✅ Pass | 0.789s | |
--| Function Response Memory Test | ✅ Pass | 0.742s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.220s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.870s | |
--| Penetration Testing Methodology | ✅ Pass | 0.981s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.004s | |
--| SQL Injection Attack Type | ✅ Pass | 0.899s | |
--| Penetration Testing Framework | ✅ Pass | 1.223s | |
--| Web Application Security Scanner | ✅ Pass | 1.287s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.151s | |
++| JSON Response Function | ✅ Pass | 0.205s | |
++| Search Query Function | ✅ Pass | 0.204s | |
++| Ask Advice Function | ✅ Pass | 0.211s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.209s | |
++| Basic Context Memory Test | ✅ Pass | 0.210s | |
++| Function Argument Memory Test | ✅ Pass | 0.201s | |
++| Function Response Memory Test | ✅ Pass | 0.207s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.216s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.208s | |
++| Penetration Testing Methodology | ✅ Pass | 0.222s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.184s | |
++| SQL Injection Attack Type | ✅ Pass | 0.179s | |
++| Penetration Testing Framework | ✅ Pass | 0.197s | |
++| Web Application Security Scanner | ✅ Pass | 0.193s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.071s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 0.991s
++**Average latency**: 0.211s
---
@@@ -377,38 -377,38 +384,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.759s | |
--| Text Transform Uppercase | ✅ Pass | 0.897s | |
--| Count from 1 to 5 | ✅ Pass | 0.872s | |
--| Math Calculation | ✅ Pass | 0.710s | |
--| Basic Echo Function | ✅ Pass | 0.948s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.870s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.746s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.899s | |
++| Simple Math | ✅ Pass | 0.237s | |
++| Text Transform Uppercase | ✅ Pass | 0.218s | |
++| Count from 1 to 5 | ✅ Pass | 0.212s | |
++| Math Calculation | ✅ Pass | 0.209s | |
++| Basic Echo Function | ✅ Pass | 0.210s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.213s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.204s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.210s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.144s | |
--| Search Query Function | ✅ Pass | 0.972s | |
--| Ask Advice Function | ✅ Pass | 0.989s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.015s | |
--| Basic Context Memory Test | ✅ Pass | 0.858s | |
--| Function Argument Memory Test | ✅ Pass | 0.726s | |
--| Function Response Memory Test | ✅ Pass | 0.754s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.414s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.851s | |
--| Penetration Testing Methodology | ✅ Pass | 0.991s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.597s | |
--| SQL Injection Attack Type | ✅ Pass | 0.857s | |
--| Penetration Testing Framework | ✅ Pass | 1.269s | |
--| Web Application Security Scanner | ✅ Pass | 1.036s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.377s | |
++| JSON Response Function | ✅ Pass | 0.208s | |
++| Search Query Function | ✅ Pass | 0.203s | |
++| Ask Advice Function | ✅ Pass | 0.204s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.205s | |
++| Basic Context Memory Test | ✅ Pass | 0.208s | |
++| Function Argument Memory Test | ✅ Pass | 0.199s | |
++| Function Response Memory Test | ✅ Pass | 0.209s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.213s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.434s | |
++| Penetration Testing Methodology | ✅ Pass | 0.222s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.130s | |
++| SQL Injection Attack Type | ✅ Pass | 0.204s | |
++| Penetration Testing Framework | ✅ Pass | 0.195s | |
++| Web Application Security Scanner | ✅ Pass | 0.136s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.215s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 0.981s
++**Average latency**: 0.213s
---
@@@ -418,38 -418,38 +425,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.791s | |
--| Text Transform Uppercase | ✅ Pass | 6.028s | |
--| Count from 1 to 5 | ✅ Pass | 1.873s | |
--| Math Calculation | ✅ Pass | 1.707s | |
--| Basic Echo Function | ✅ Pass | 2.051s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.951s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.989s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.446s | |
++| Simple Math | ✅ Pass | 0.208s | |
++| Text Transform Uppercase | ✅ Pass | 0.219s | |
++| Count from 1 to 5 | ✅ Pass | 0.209s | |
++| Math Calculation | ✅ Pass | 0.208s | |
++| Basic Echo Function | ✅ Pass | 0.204s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.208s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.214s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.210s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.494s | |
--| Search Query Function | ✅ Pass | 2.010s | |
--| Ask Advice Function | ✅ Pass | 2.242s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.133s | |
--| Basic Context Memory Test | ✅ Pass | 3.207s | |
--| Function Argument Memory Test | ✅ Pass | 2.075s | |
--| Function Response Memory Test | ✅ Pass | 1.921s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.561s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.345s | |
--| Penetration Testing Methodology | ✅ Pass | 3.598s | |
--| Vulnerability Assessment Tools | ✅ Pass | 8.780s | |
--| SQL Injection Attack Type | ✅ Pass | 2.581s | |
--| Penetration Testing Framework | ✅ Pass | 2.960s | |
--| Web Application Security Scanner | ✅ Pass | 2.493s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.548s | |
++| JSON Response Function | ✅ Pass | 0.209s | |
++| Search Query Function | ✅ Pass | 0.203s | |
++| Ask Advice Function | ✅ Pass | 0.207s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.211s | |
++| Basic Context Memory Test | ✅ Pass | 0.207s | |
++| Function Argument Memory Test | ✅ Pass | 0.204s | |
++| Function Response Memory Test | ✅ Pass | 0.207s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.212s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.438s | |
++| Penetration Testing Methodology | ✅ Pass | 0.232s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.164s | |
++| SQL Injection Attack Type | ✅ Pass | 0.206s | |
++| Penetration Testing Framework | ✅ Pass | 0.208s | |
++| Web Application Security Scanner | ✅ Pass | 0.169s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.233s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.817s
++**Average latency**: 0.217s
---
@@@ -459,38 -459,38 +466,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.185s | |
--| Text Transform Uppercase | ✅ Pass | 1.103s | |
--| Count from 1 to 5 | ✅ Pass | 1.222s | |
--| Math Calculation | ✅ Pass | 0.962s | |
--| Basic Echo Function | ✅ Pass | 0.990s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.908s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.216s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.106s | |
++| Simple Math | ✅ Pass | 0.235s | |
++| Text Transform Uppercase | ✅ Pass | 0.218s | |
++| Count from 1 to 5 | ✅ Pass | 0.208s | |
++| Math Calculation | ✅ Pass | 0.208s | |
++| Basic Echo Function | ✅ Pass | 0.201s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.210s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.208s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.211s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.264s | |
--| Search Query Function | ✅ Pass | 1.140s | |
--| Ask Advice Function | ✅ Pass | 1.158s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.171s | |
--| Basic Context Memory Test | ✅ Pass | 1.188s | |
--| Function Argument Memory Test | ✅ Pass | 0.905s | |
--| Function Response Memory Test | ✅ Pass | 1.120s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.758s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.140s | |
--| Penetration Testing Methodology | ✅ Pass | 1.382s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.060s | |
--| SQL Injection Attack Type | ✅ Pass | 1.298s | |
--| Penetration Testing Framework | ✅ Pass | 1.423s | |
--| Web Application Security Scanner | ✅ Pass | 1.586s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.352s | |
++| JSON Response Function | ✅ Pass | 0.208s | |
++| Search Query Function | ✅ Pass | 0.216s | |
++| Ask Advice Function | ✅ Pass | 0.206s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.202s | |
++| Basic Context Memory Test | ✅ Pass | 0.208s | |
++| Function Argument Memory Test | ✅ Pass | 0.205s | |
++| Function Response Memory Test | ✅ Pass | 0.207s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.208s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.418s | |
++| Penetration Testing Methodology | ✅ Pass | 0.227s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.182s | |
++| SQL Injection Attack Type | ✅ Pass | 0.211s | |
++| Penetration Testing Framework | ✅ Pass | 0.193s | |
++| Web Application Security Scanner | ✅ Pass | 0.172s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.219s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.246s
++**Average latency**: 0.217s
---
@@@ -500,38 -500,38 +507,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.845s | |
--| Text Transform Uppercase | ✅ Pass | 6.297s | |
--| Count from 1 to 5 | ✅ Pass | 1.982s | |
--| Math Calculation | ✅ Pass | 1.653s | |
--| Basic Echo Function | ✅ Pass | 1.838s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.884s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.004s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.209s | |
++| Simple Math | ✅ Pass | 0.235s | |
++| Text Transform Uppercase | ✅ Pass | 0.224s | |
++| Count from 1 to 5 | ✅ Pass | 0.210s | |
++| Math Calculation | ✅ Pass | 0.208s | |
++| Basic Echo Function | ✅ Pass | 0.208s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.216s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.204s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.209s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.702s | |
--| Search Query Function | ✅ Pass | 1.898s | |
--| Ask Advice Function | ✅ Pass | 2.049s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.145s | |
--| Basic Context Memory Test | ✅ Pass | 3.227s | |
--| Function Argument Memory Test | ✅ Pass | 1.892s | |
--| Function Response Memory Test | ✅ Pass | 2.334s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.631s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.276s | |
--| Penetration Testing Methodology | ✅ Pass | 3.126s | |
--| SQL Injection Attack Type | ✅ Pass | 2.551s | |
--| Vulnerability Assessment Tools | ✅ Pass | 14.848s | |
--| Penetration Testing Framework | ✅ Pass | 3.944s | |
--| Web Application Security Scanner | ✅ Pass | 1.612s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.012s | |
++| JSON Response Function | ✅ Pass | 0.210s | |
++| Search Query Function | ✅ Pass | 0.216s | |
++| Ask Advice Function | ✅ Pass | 0.202s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.210s | |
++| Basic Context Memory Test | ✅ Pass | 0.204s | |
++| Function Argument Memory Test | ✅ Pass | 0.207s | |
++| Function Response Memory Test | ✅ Pass | 0.206s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.265s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.413s | |
++| Penetration Testing Methodology | ✅ Pass | 0.231s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.198s | |
++| SQL Injection Attack Type | ✅ Pass | 0.203s | |
++| Penetration Testing Framework | ✅ Pass | 0.140s | |
++| Web Application Security Scanner | ✅ Pass | 0.297s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.234s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 3.086s
++**Average latency**: 0.224s
---
diff --cc examples/tests/gemini-report.md
index ff39febd,ff39febd..fa36ef87
--- a/examples/tests/gemini-report.md
+++ b/examples/tests/gemini-report.md
@@@ -1,27 -1,27 +1,27 @@@
# LLM Agent Testing Report
--Generated: Wed, 27 May 2026 22:33:44 UTC
++Generated: Tue, 21 Jul 2026 17:10:09 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
--| simple | gemini-3.1-flash-lite | true | 23/23 (100.00%) | 0.768s |
--| simple_json | gemini-3.1-flash-lite | true | 5/5 (100.00%) | 0.715s |
--| primary_agent | gemini-3.1-pro-preview | true | 22/23 (95.65%) | 4.190s |
--| assistant | gemini-3.1-pro-preview | true | 22/23 (95.65%) | 4.239s |
--| generator | gemini-3.1-pro-preview | true | 21/23 (91.30%) | 4.434s |
--| refiner | gemini-3.1-pro-preview | true | 23/23 (100.00%) | 4.221s |
--| adviser | gemini-3.1-pro-preview | true | 21/23 (91.30%) | 4.304s |
--| reflector | gemini-3.5-flash | true | 23/23 (100.00%) | 1.692s |
--| searcher | gemini-3.5-flash | true | 23/23 (100.00%) | 1.817s |
--| enricher | gemini-3.5-flash | true | 23/23 (100.00%) | 1.724s |
--| coder | gemini-3.1-pro-preview | true | 23/23 (100.00%) | 4.103s |
--| installer | gemini-3.5-flash | true | 22/23 (95.65%) | 2.891s |
--| pentester | gemini-3.1-pro-preview | true | 23/23 (100.00%) | 4.197s |
--
--**Total**: 274/281 (97.51%) successful tests
--**Overall average latency**: 3.171s
++| simple | gemini-3.1-flash-lite | true | 23/23 (100.00%) | 0.572s |
++| simple_json | gemini-3.1-flash-lite | true | 7/7 (100.00%) | 0.595s |
++| primary_agent | gemini-3.1-pro-preview | true | 21/23 (91.30%) | 4.793s |
++| assistant | gemini-3.1-pro-preview | true | 21/23 (91.30%) | 9.515s |
++| generator | gemini-3.1-pro-preview | true | 19/23 (82.61%) | 6.179s |
++| refiner | gemini-3.1-pro-preview | true | 22/23 (95.65%) | 5.690s |
++| adviser | gemini-3.1-pro-preview | true | 19/23 (82.61%) | 4.937s |
++| reflector | gemini-3.5-flash | true | 23/23 (100.00%) | 1.967s |
++| searcher | gemini-3.5-flash | true | 23/23 (100.00%) | 1.885s |
++| enricher | gemini-3.5-flash | true | 23/23 (100.00%) | 1.830s |
++| coder | gemini-3.1-pro-preview | true | 22/23 (95.65%) | 5.014s |
++| installer | gemini-3.5-flash | true | 23/23 (100.00%) | 2.520s |
++| pentester | gemini-3.1-pro-preview | true | 22/23 (95.65%) | 4.466s |
++
++**Total**: 268/283 (94.70%) successful tests
++**Overall average latency**: 4.027s
## Detailed Results
@@@ -31,38 -31,38 +31,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.367s | |
--| Text Transform Uppercase | ✅ Pass | 0.517s | |
--| Count from 1 to 5 | ✅ Pass | 0.704s | |
--| Math Calculation | ✅ Pass | 0.530s | |
--| Basic Echo Function | ✅ Pass | 0.524s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.506s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.626s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.663s | |
++| Simple Math | ✅ Pass | 1.175s | |
++| Text Transform Uppercase | ✅ Pass | 0.496s | |
++| Count from 1 to 5 | ✅ Pass | 0.551s | |
++| Math Calculation | ✅ Pass | 0.504s | |
++| Basic Echo Function | ✅ Pass | 0.434s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.505s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.488s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.483s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 0.578s | |
--| Search Query Function | ✅ Pass | 0.544s | |
--| Ask Advice Function | ✅ Pass | 0.576s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.529s | |
--| Basic Context Memory Test | ✅ Pass | 0.551s | |
--| Function Argument Memory Test | ✅ Pass | 0.490s | |
--| Function Response Memory Test | ✅ Pass | 0.688s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.597s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.546s | |
--| Penetration Testing Methodology | ✅ Pass | 1.218s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.050s | |
--| SQL Injection Attack Type | ✅ Pass | 0.661s | |
--| Penetration Testing Framework | ✅ Pass | 0.762s | |
--| Web Application Security Scanner | ✅ Pass | 0.750s | |
--| Penetration Testing Tool Selection | ✅ Pass | 0.673s | |
++| JSON Response Function | ✅ Pass | 0.559s | |
++| Search Query Function | ✅ Pass | 0.492s | |
++| Ask Advice Function | ✅ Pass | 0.529s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.430s | |
++| Basic Context Memory Test | ✅ Pass | 0.440s | |
++| Function Argument Memory Test | ✅ Pass | 0.431s | |
++| Function Response Memory Test | ✅ Pass | 0.438s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.691s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.492s | |
++| Penetration Testing Methodology | ✅ Pass | 0.680s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.928s | |
++| SQL Injection Attack Type | ✅ Pass | 0.649s | |
++| Penetration Testing Framework | ✅ Pass | 0.685s | |
++| Web Application Security Scanner | ✅ Pass | 0.498s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.554s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 0.768s
++**Average latency**: 0.572s
---
@@@ -72,15 -72,15 +72,22 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Person Information JSON | ✅ Pass | 0.664s | |
--| Vulnerability Report Memory Test | ✅ Pass | 0.961s | |
--| User Profile JSON | ✅ Pass | 0.594s | |
--| Project Information JSON | ✅ Pass | 0.766s | |
--| Streaming Person Information JSON Streaming | ✅ Pass | 0.590s | |
++| Vulnerability Report Memory Test | ✅ Pass | 0.680s | |
++| Person Information JSON | ✅ Pass | 0.569s | |
++| Project Information JSON | ✅ Pass | 0.567s | |
++| User Profile JSON | ✅ Pass | 0.561s | |
++| JSON Array Response Without Schema | ✅ Pass | 0.678s | |
++| Streaming Person Information JSON Streaming | ✅ Pass | 0.556s | |
--**Summary**: 5/5 (100.00%) successful tests
++#### Capability Tests
--**Average latency**: 0.715s
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Structured Output With JSON Schema | structured_output | ✅ Pass | 0.553s | |
++
++**Summary**: 7/7 (100.00%) successful tests
++
++**Average latency**: 0.595s
---
@@@ -90,38 -90,38 +97,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.812s | |
--| Text Transform Uppercase | ✅ Pass | 3.313s | |
--| Count from 1 to 5 | ✅ Pass | 4.086s | |
--| Math Calculation | ✅ Pass | 2.632s | |
--| Basic Echo Function | ❌ Fail | 4.651s | no tool calls found, expected at least 1 |
--| Streaming Simple Math Streaming | ✅ Pass | 3.243s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.803s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.800s | |
++| Simple Math | ✅ Pass | 4.736s | |
++| Text Transform Uppercase | ✅ Pass | 3.911s | |
++| Count from 1 to 5 | ✅ Pass | 4.733s | |
++| Math Calculation | ✅ Pass | 3.294s | |
++| Basic Echo Function | ✅ Pass | 4.043s | |
++| Streaming Simple Math Streaming | ✅ Pass | 3.061s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.591s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.864s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 5.060s | |
--| Search Query Function | ✅ Pass | 4.701s | |
--| Ask Advice Function | ✅ Pass | 6.206s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.818s | |
--| Basic Context Memory Test | ✅ Pass | 3.449s | |
--| Function Argument Memory Test | ✅ Pass | 4.006s | |
--| Function Response Memory Test | ✅ Pass | 4.055s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.735s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 4.365s | |
--| Penetration Testing Methodology | ✅ Pass | 5.392s | |
--| Vulnerability Assessment Tools | ✅ Pass | 5.336s | |
--| SQL Injection Attack Type | ✅ Pass | 3.569s | |
--| Penetration Testing Framework | ✅ Pass | 4.948s | |
--| Web Application Security Scanner | ✅ Pass | 5.728s | |
--| Penetration Testing Tool Selection | ✅ Pass | 5.643s | |
++| JSON Response Function | ✅ Pass | 5.274s | |
++| Search Query Function | ✅ Pass | 5.765s | |
++| Ask Advice Function | ✅ Pass | 4.155s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.039s | |
++| Basic Context Memory Test | ✅ Pass | 4.656s | |
++| Function Argument Memory Test | ✅ Pass | 3.736s | |
++| Function Response Memory Test | ✅ Pass | 4.103s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 8.039s | no tool calls found, expected at least 1 |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.921s | |
++| Penetration Testing Methodology | ✅ Pass | 6.699s | |
++| Vulnerability Assessment Tools | ✅ Pass | 6.705s | |
++| SQL Injection Attack Type | ✅ Pass | 3.375s | |
++| Penetration Testing Framework | ✅ Pass | 4.664s | |
++| Web Application Security Scanner | ✅ Pass | 5.470s | |
++| Penetration Testing Tool Selection | ❌ Fail | 12.396s | no tool calls found, expected at least 1 |
--**Summary**: 22/23 (95.65%) successful tests
++**Summary**: 21/23 (91.30%) successful tests
--**Average latency**: 4.190s
++**Average latency**: 4.793s
---
@@@ -131,38 -131,38 +138,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.818s | |
--| Text Transform Uppercase | ✅ Pass | 3.634s | |
--| Count from 1 to 5 | ✅ Pass | 4.193s | |
--| Math Calculation | ✅ Pass | 3.305s | |
--| Basic Echo Function | ❌ Fail | 5.872s | no tool calls found, expected at least 1 |
--| Streaming Simple Math Streaming | ✅ Pass | 3.279s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.548s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.377s | |
++| Simple Math | ✅ Pass | 4.922s | |
++| Text Transform Uppercase | ✅ Pass | 4.325s | |
++| Count from 1 to 5 | ✅ Pass | 7.273s | |
++| Math Calculation | ✅ Pass | 2.922s | |
++| Basic Echo Function | ✅ Pass | 5.052s | |
++| Streaming Simple Math Streaming | ✅ Pass | 3.366s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.613s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.735s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.867s | |
--| Search Query Function | ✅ Pass | 3.219s | |
--| Ask Advice Function | ✅ Pass | 5.921s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.671s | |
--| Basic Context Memory Test | ✅ Pass | 3.188s | |
--| Function Argument Memory Test | ✅ Pass | 4.451s | |
--| Function Response Memory Test | ✅ Pass | 3.309s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 6.425s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 4.503s | |
--| Penetration Testing Methodology | ✅ Pass | 5.211s | |
--| Vulnerability Assessment Tools | ✅ Pass | 5.501s | |
--| SQL Injection Attack Type | ✅ Pass | 4.066s | |
--| Penetration Testing Framework | ✅ Pass | 5.491s | |
--| Web Application Security Scanner | ✅ Pass | 5.037s | |
--| Penetration Testing Tool Selection | ✅ Pass | 6.604s | |
++| JSON Response Function | ✅ Pass | 4.776s | |
++| Search Query Function | ✅ Pass | 3.675s | |
++| Ask Advice Function | ✅ Pass | 4.402s | |
++| Basic Context Memory Test | ✅ Pass | 4.029s | |
++| Function Argument Memory Test | ✅ Pass | 3.728s | |
++| Function Response Memory Test | ✅ Pass | 4.033s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 16.108s | no tool calls found, expected at least 1 |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.484s | |
++| Penetration Testing Methodology | ✅ Pass | 4.964s | |
++| Vulnerability Assessment Tools | ✅ Pass | 6.272s | |
++| SQL Injection Attack Type | ✅ Pass | 4.569s | |
++| Penetration Testing Framework | ✅ Pass | 4.902s | |
++| Web Application Security Scanner | ✅ Pass | 5.659s | |
++| Penetration Testing Tool Selection | ❌ Fail | 16.806s | no tool calls found, expected at least 1 |
++| Streaming Search Query Function Streaming | ✅ Pass | 98.207s | |
--**Summary**: 22/23 (95.65%) successful tests
++**Summary**: 21/23 (91.30%) successful tests
--**Average latency**: 4.239s
++**Average latency**: 9.515s
---
@@@ -172,38 -172,38 +179,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.066s | |
--| Text Transform Uppercase | ✅ Pass | 4.138s | |
--| Count from 1 to 5 | ✅ Pass | 3.613s | |
--| Math Calculation | ✅ Pass | 3.089s | |
--| Basic Echo Function | ❌ Fail | 4.109s | no tool calls found, expected at least 1 |
--| Streaming Simple Math Streaming | ✅ Pass | 3.430s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.411s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.347s | |
++| Simple Math | ✅ Pass | 3.806s | |
++| Text Transform Uppercase | ✅ Pass | 4.008s | |
++| Count from 1 to 5 | ✅ Pass | 5.580s | |
++| Math Calculation | ✅ Pass | 3.740s | |
++| Basic Echo Function | ✅ Pass | 3.731s | |
++| Streaming Simple Math Streaming | ✅ Pass | 3.053s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.228s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 6.705s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 4.487s | |
--| Search Query Function | ✅ Pass | 3.946s | |
--| Ask Advice Function | ✅ Pass | 4.281s | |
--| Streaming Search Query Function Streaming | ❌ Fail | 3.204s | no tool calls found, expected at least 1 |
--| Basic Context Memory Test | ✅ Pass | 4.219s | |
--| Function Argument Memory Test | ✅ Pass | 4.124s | |
--| Function Response Memory Test | ✅ Pass | 3.339s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 7.760s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.648s | |
--| Penetration Testing Methodology | ✅ Pass | 7.310s | |
--| Vulnerability Assessment Tools | ✅ Pass | 9.153s | |
--| SQL Injection Attack Type | ✅ Pass | 3.768s | |
--| Penetration Testing Framework | ✅ Pass | 3.997s | |
--| Web Application Security Scanner | ✅ Pass | 4.903s | |
--| Penetration Testing Tool Selection | ✅ Pass | 6.634s | |
--
--**Summary**: 21/23 (91.30%) successful tests
--
--**Average latency**: 4.434s
++| JSON Response Function | ✅ Pass | 3.933s | |
++| Search Query Function | ❌ Fail | 6.695s | no tool calls found, expected at least 1 |
++| Ask Advice Function | ❌ Fail | 6.017s | no tool calls found, expected at least 1 |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.922s | |
++| Basic Context Memory Test | ✅ Pass | 3.532s | |
++| Function Argument Memory Test | ✅ Pass | 3.905s | |
++| Function Response Memory Test | ✅ Pass | 3.967s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 4.051s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 20.272s | no tool calls found, expected at least 1 |
++| Penetration Testing Methodology | ✅ Pass | 7.631s | |
++| Vulnerability Assessment Tools | ✅ Pass | 7.752s | |
++| SQL Injection Attack Type | ✅ Pass | 3.358s | |
++| Penetration Testing Framework | ✅ Pass | 6.881s | |
++| Web Application Security Scanner | ✅ Pass | 5.409s | |
++| Penetration Testing Tool Selection | ❌ Fail | 22.927s | no tool calls found, expected at least 1 |
++
++**Summary**: 19/23 (82.61%) successful tests
++
++**Average latency**: 6.179s
---
@@@ -213,38 -213,38 +220,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.559s | |
--| Text Transform Uppercase | ✅ Pass | 4.160s | |
--| Count from 1 to 5 | ✅ Pass | 3.893s | |
--| Math Calculation | ✅ Pass | 2.809s | |
--| Basic Echo Function | ✅ Pass | 4.024s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.314s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.412s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.594s | |
++| Simple Math | ✅ Pass | 4.555s | |
++| Text Transform Uppercase | ✅ Pass | 3.360s | |
++| Count from 1 to 5 | ✅ Pass | 4.623s | |
++| Math Calculation | ✅ Pass | 2.988s | |
++| Streaming Simple Math Streaming | ✅ Pass | 3.184s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.860s | |
++| Basic Echo Function | ✅ Pass | 14.128s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 3.895s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 4.043s | |
--| Search Query Function | ✅ Pass | 3.295s | |
--| Ask Advice Function | ✅ Pass | 4.504s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.448s | |
--| Basic Context Memory Test | ✅ Pass | 3.640s | |
--| Function Argument Memory Test | ✅ Pass | 3.824s | |
--| Function Response Memory Test | ✅ Pass | 4.427s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 7.450s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.576s | |
--| Penetration Testing Methodology | ✅ Pass | 5.822s | |
--| Vulnerability Assessment Tools | ✅ Pass | 6.900s | |
--| SQL Injection Attack Type | ✅ Pass | 3.912s | |
--| Penetration Testing Framework | ✅ Pass | 4.930s | |
--| Web Application Security Scanner | ✅ Pass | 5.459s | |
--| Penetration Testing Tool Selection | ✅ Pass | 6.079s | |
++| JSON Response Function | ✅ Pass | 5.466s | |
++| Search Query Function | ✅ Pass | 2.973s | |
++| Ask Advice Function | ✅ Pass | 3.722s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.918s | |
++| Basic Context Memory Test | ✅ Pass | 4.045s | |
++| Function Argument Memory Test | ✅ Pass | 3.300s | |
++| Function Response Memory Test | ✅ Pass | 4.094s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 14.605s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 6.032s | |
++| Vulnerability Assessment Tools | ✅ Pass | 5.394s | |
++| Penetration Testing Methodology | ✅ Pass | 14.148s | |
++| SQL Injection Attack Type | ✅ Pass | 3.666s | |
++| Penetration Testing Framework | ✅ Pass | 4.038s | |
++| Web Application Security Scanner | ✅ Pass | 4.975s | |
++| Penetration Testing Tool Selection | ❌ Fail | 12.892s | no tool calls found, expected at least 1 |
--**Summary**: 23/23 (100.00%) successful tests
++**Summary**: 22/23 (95.65%) successful tests
--**Average latency**: 4.221s
++**Average latency**: 5.690s
---
@@@ -254,38 -254,38 +261,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.811s | |
--| Text Transform Uppercase | ✅ Pass | 3.618s | |
--| Count from 1 to 5 | ✅ Pass | 5.524s | |
--| Math Calculation | ✅ Pass | 3.096s | |
--| Basic Echo Function | ✅ Pass | 5.575s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.725s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.862s | |
--| Streaming Basic Echo Function Streaming | ❌ Fail | 3.410s | no tool calls found, expected at least 1 |
++| Simple Math | ✅ Pass | 4.242s | |
++| Text Transform Uppercase | ✅ Pass | 3.971s | |
++| Count from 1 to 5 | ✅ Pass | 4.435s | |
++| Math Calculation | ✅ Pass | 3.617s | |
++| Basic Echo Function | ✅ Pass | 4.109s | |
++| Streaming Simple Math Streaming | ✅ Pass | 3.995s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.637s | |
++| Streaming Basic Echo Function Streaming | ❌ Fail | 4.687s | no tool calls found, expected at least 1 |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.542s | |
--| Search Query Function | ❌ Fail | 4.056s | no tool calls found, expected at least 1 |
--| Ask Advice Function | ✅ Pass | 3.966s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.561s | |
--| Basic Context Memory Test | ✅ Pass | 3.647s | |
--| Function Argument Memory Test | ✅ Pass | 4.782s | |
--| Function Response Memory Test | ✅ Pass | 4.070s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 6.209s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.835s | |
--| Penetration Testing Methodology | ✅ Pass | 6.464s | |
--| Vulnerability Assessment Tools | ✅ Pass | 6.427s | |
--| SQL Injection Attack Type | ✅ Pass | 3.244s | |
--| Penetration Testing Framework | ✅ Pass | 5.763s | |
--| Web Application Security Scanner | ✅ Pass | 5.714s | |
--| Penetration Testing Tool Selection | ✅ Pass | 5.082s | |
--
--**Summary**: 21/23 (91.30%) successful tests
--
--**Average latency**: 4.304s
++| JSON Response Function | ✅ Pass | 4.164s | |
++| Search Query Function | ❌ Fail | 3.340s | no tool calls found, expected at least 1 |
++| Ask Advice Function | ✅ Pass | 4.271s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.594s | |
++| Basic Context Memory Test | ✅ Pass | 3.187s | |
++| Function Argument Memory Test | ✅ Pass | 2.980s | |
++| Function Response Memory Test | ✅ Pass | 3.858s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 15.658s | no tool calls found, expected at least 1 |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.167s | |
++| Penetration Testing Methodology | ✅ Pass | 6.449s | |
++| Vulnerability Assessment Tools | ✅ Pass | 4.278s | |
++| SQL Injection Attack Type | ✅ Pass | 3.853s | |
++| Penetration Testing Framework | ✅ Pass | 5.700s | |
++| Web Application Security Scanner | ✅ Pass | 4.230s | |
++| Penetration Testing Tool Selection | ❌ Fail | 14.125s | no tool calls found, expected at least 1 |
++
++**Summary**: 19/23 (82.61%) successful tests
++
++**Average latency**: 4.937s
---
@@@ -295,38 -295,38 +302,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.016s | |
--| Text Transform Uppercase | ✅ Pass | 1.398s | |
--| Count from 1 to 5 | ✅ Pass | 1.549s | |
--| Math Calculation | ✅ Pass | 1.023s | |
--| Basic Echo Function | ✅ Pass | 1.100s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.953s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.477s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.172s | |
++| Simple Math | ✅ Pass | 1.856s | |
++| Text Transform Uppercase | ✅ Pass | 1.177s | |
++| Count from 1 to 5 | ✅ Pass | 1.626s | |
++| Math Calculation | ✅ Pass | 0.995s | |
++| Basic Echo Function | ✅ Pass | 1.200s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.000s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.423s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.056s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.330s | |
--| Search Query Function | ✅ Pass | 0.972s | |
--| Ask Advice Function | ✅ Pass | 1.386s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.149s | |
--| Basic Context Memory Test | ✅ Pass | 1.303s | |
--| Function Argument Memory Test | ✅ Pass | 1.155s | |
--| Function Response Memory Test | ✅ Pass | 1.319s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.053s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.048s | |
--| Penetration Testing Methodology | ✅ Pass | 2.124s | |
--| Vulnerability Assessment Tools | ✅ Pass | 3.923s | |
--| SQL Injection Attack Type | ✅ Pass | 1.678s | |
--| Penetration Testing Framework | ✅ Pass | 2.756s | |
--| Web Application Security Scanner | ✅ Pass | 1.608s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.421s | |
++| JSON Response Function | ✅ Pass | 1.257s | |
++| Search Query Function | ✅ Pass | 1.050s | |
++| Ask Advice Function | ✅ Pass | 1.228s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.870s | |
++| Basic Context Memory Test | ✅ Pass | 1.409s | |
++| Function Argument Memory Test | ✅ Pass | 1.798s | |
++| Function Response Memory Test | ✅ Pass | 1.306s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.170s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.111s | |
++| Penetration Testing Methodology | ✅ Pass | 2.948s | |
++| Vulnerability Assessment Tools | ✅ Pass | 3.927s | |
++| SQL Injection Attack Type | ✅ Pass | 2.370s | |
++| Penetration Testing Framework | ✅ Pass | 4.652s | |
++| Web Application Security Scanner | ✅ Pass | 1.685s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.114s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.692s
++**Average latency**: 1.967s
---
@@@ -336,38 -336,38 +343,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.992s | |
--| Text Transform Uppercase | ✅ Pass | 1.363s | |
--| Count from 1 to 5 | ✅ Pass | 1.590s | |
--| Math Calculation | ✅ Pass | 0.971s | |
--| Basic Echo Function | ✅ Pass | 1.336s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.916s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.542s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.124s | |
++| Simple Math | ✅ Pass | 2.171s | |
++| Text Transform Uppercase | ✅ Pass | 1.366s | |
++| Count from 1 to 5 | ✅ Pass | 1.447s | |
++| Math Calculation | ✅ Pass | 1.371s | |
++| Basic Echo Function | ✅ Pass | 1.324s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.439s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.443s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.493s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.079s | |
--| Search Query Function | ✅ Pass | 1.121s | |
--| Ask Advice Function | ✅ Pass | 1.387s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.260s | |
--| Basic Context Memory Test | ✅ Pass | 1.399s | |
--| Function Argument Memory Test | ✅ Pass | 1.557s | |
--| Function Response Memory Test | ✅ Pass | 1.474s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.472s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.838s | |
--| Penetration Testing Methodology | ✅ Pass | 2.772s | |
--| Vulnerability Assessment Tools | ✅ Pass | 4.313s | |
--| SQL Injection Attack Type | ✅ Pass | 1.202s | |
--| Penetration Testing Framework | ✅ Pass | 2.830s | |
--| Web Application Security Scanner | ✅ Pass | 2.120s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.116s | |
++| JSON Response Function | ✅ Pass | 1.152s | |
++| Search Query Function | ✅ Pass | 1.112s | |
++| Ask Advice Function | ✅ Pass | 1.490s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.065s | |
++| Basic Context Memory Test | ✅ Pass | 1.277s | |
++| Function Argument Memory Test | ✅ Pass | 1.805s | |
++| Function Response Memory Test | ✅ Pass | 1.433s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.231s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.482s | |
++| Penetration Testing Methodology | ✅ Pass | 2.539s | |
++| Vulnerability Assessment Tools | ✅ Pass | 3.027s | |
++| SQL Injection Attack Type | ✅ Pass | 1.613s | |
++| Penetration Testing Framework | ✅ Pass | 2.977s | |
++| Web Application Security Scanner | ✅ Pass | 2.852s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.242s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.817s
++**Average latency**: 1.885s
---
@@@ -377,38 -377,38 +384,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.981s | |
--| Text Transform Uppercase | ✅ Pass | 1.215s | |
--| Count from 1 to 5 | ✅ Pass | 1.425s | |
--| Math Calculation | ✅ Pass | 0.932s | |
--| Basic Echo Function | ✅ Pass | 1.084s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.958s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.604s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.237s | |
++| Simple Math | ✅ Pass | 0.931s | |
++| Text Transform Uppercase | ✅ Pass | 1.375s | |
++| Count from 1 to 5 | ✅ Pass | 1.435s | |
++| Math Calculation | ✅ Pass | 1.295s | |
++| Basic Echo Function | ✅ Pass | 1.140s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.995s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.424s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.549s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.128s | |
--| Search Query Function | ✅ Pass | 0.993s | |
--| Ask Advice Function | ✅ Pass | 1.090s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.009s | |
--| Basic Context Memory Test | ✅ Pass | 1.461s | |
--| Function Argument Memory Test | ✅ Pass | 0.955s | |
--| Function Response Memory Test | ✅ Pass | 1.856s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.277s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.771s | |
--| Penetration Testing Methodology | ✅ Pass | 2.691s | |
--| Vulnerability Assessment Tools | ✅ Pass | 3.802s | |
--| SQL Injection Attack Type | ✅ Pass | 1.611s | |
--| Penetration Testing Framework | ✅ Pass | 2.182s | |
--| Web Application Security Scanner | ✅ Pass | 2.253s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.135s | |
++| JSON Response Function | ✅ Pass | 1.419s | |
++| Search Query Function | ✅ Pass | 1.175s | |
++| Ask Advice Function | ✅ Pass | 1.304s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.055s | |
++| Basic Context Memory Test | ✅ Pass | 1.304s | |
++| Function Argument Memory Test | ✅ Pass | 1.741s | |
++| Function Response Memory Test | ✅ Pass | 1.550s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.407s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.805s | |
++| Penetration Testing Methodology | ✅ Pass | 2.351s | |
++| Vulnerability Assessment Tools | ✅ Pass | 3.706s | |
++| SQL Injection Attack Type | ✅ Pass | 1.863s | |
++| Penetration Testing Framework | ✅ Pass | 1.976s | |
++| Web Application Security Scanner | ✅ Pass | 2.727s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.552s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.724s
++**Average latency**: 1.830s
---
@@@ -418,38 -418,38 +425,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.599s | |
--| Text Transform Uppercase | ✅ Pass | 3.774s | |
--| Count from 1 to 5 | ✅ Pass | 3.470s | |
--| Math Calculation | ✅ Pass | 3.170s | |
--| Basic Echo Function | ✅ Pass | 4.147s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.895s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.859s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.385s | |
++| Simple Math | ✅ Pass | 3.937s | |
++| Text Transform Uppercase | ✅ Pass | 3.855s | |
++| Count from 1 to 5 | ✅ Pass | 3.427s | |
++| Math Calculation | ✅ Pass | 3.287s | |
++| Basic Echo Function | ✅ Pass | 3.881s | |
++| Streaming Simple Math Streaming | ✅ Pass | 4.231s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.236s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.602s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 5.047s | |
--| Search Query Function | ✅ Pass | 3.097s | |
--| Ask Advice Function | ✅ Pass | 3.666s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.431s | |
--| Basic Context Memory Test | ✅ Pass | 3.474s | |
--| Function Argument Memory Test | ✅ Pass | 3.706s | |
--| Function Response Memory Test | ✅ Pass | 3.504s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 6.181s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.421s | |
--| Penetration Testing Methodology | ✅ Pass | 5.411s | |
--| Vulnerability Assessment Tools | ✅ Pass | 8.503s | |
--| SQL Injection Attack Type | ✅ Pass | 3.996s | |
--| Penetration Testing Framework | ✅ Pass | 4.476s | |
--| Web Application Security Scanner | ✅ Pass | 4.807s | |
--| Penetration Testing Tool Selection | ✅ Pass | 6.343s | |
++| JSON Response Function | ✅ Pass | 4.478s | |
++| Search Query Function | ✅ Pass | 3.723s | |
++| Ask Advice Function | ✅ Pass | 4.353s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 3.942s | |
++| Basic Context Memory Test | ✅ Pass | 3.981s | |
++| Function Argument Memory Test | ✅ Pass | 2.978s | |
++| Function Response Memory Test | ✅ Pass | 3.865s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 11.925s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 4.649s | |
++| Vulnerability Assessment Tools | ✅ Pass | 4.950s | |
++| Penetration Testing Methodology | ✅ Pass | 19.930s | |
++| SQL Injection Attack Type | ✅ Pass | 4.029s | |
++| Penetration Testing Framework | ✅ Pass | 5.204s | |
++| Web Application Security Scanner | ✅ Pass | 4.520s | |
++| Penetration Testing Tool Selection | ❌ Fail | 5.328s | no tool calls found, expected at least 1 |
--**Summary**: 23/23 (100.00%) successful tests
++**Summary**: 22/23 (95.65%) successful tests
--**Average latency**: 4.103s
++**Average latency**: 5.014s
---
@@@ -459,38 -459,38 +466,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.355s | |
--| Text Transform Uppercase | ✅ Pass | 3.024s | |
--| Count from 1 to 5 | ✅ Pass | 2.930s | |
--| Math Calculation | ✅ Pass | 1.667s | |
--| Basic Echo Function | ✅ Pass | 2.377s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.014s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.497s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.267s | |
++| Simple Math | ✅ Pass | 2.758s | |
++| Text Transform Uppercase | ✅ Pass | 3.107s | |
++| Count from 1 to 5 | ✅ Pass | 2.562s | |
++| Math Calculation | ✅ Pass | 2.153s | |
++| Basic Echo Function | ✅ Pass | 1.689s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.054s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.487s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.991s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.773s | |
--| Search Query Function | ✅ Pass | 2.193s | |
--| Ask Advice Function | ✅ Pass | 1.839s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.064s | |
--| Basic Context Memory Test | ✅ Pass | 2.661s | |
--| Function Argument Memory Test | ✅ Pass | 2.603s | |
--| Function Response Memory Test | ✅ Pass | 2.563s | |
--| Penetration Testing Memory with Tool Call | ❌ Fail | 5.230s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 5.456s | |
--| Penetration Testing Methodology | ✅ Pass | 4.745s | |
--| Vulnerability Assessment Tools | ✅ Pass | 5.977s | |
--| SQL Injection Attack Type | ✅ Pass | 2.499s | |
--| Penetration Testing Framework | ✅ Pass | 4.620s | |
--| Web Application Security Scanner | ✅ Pass | 4.105s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.022s | |
++| JSON Response Function | ✅ Pass | 1.981s | |
++| Search Query Function | ✅ Pass | 2.041s | |
++| Ask Advice Function | ✅ Pass | 0.928s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.933s | |
++| Basic Context Memory Test | ✅ Pass | 2.175s | |
++| Function Argument Memory Test | ✅ Pass | 3.162s | |
++| Function Response Memory Test | ✅ Pass | 2.878s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.772s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.602s | |
++| Penetration Testing Methodology | ✅ Pass | 3.909s | |
++| Vulnerability Assessment Tools | ✅ Pass | 4.209s | |
++| SQL Injection Attack Type | ✅ Pass | 2.117s | |
++| Penetration Testing Framework | ✅ Pass | 3.217s | |
++| Web Application Security Scanner | ✅ Pass | 2.360s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.856s | |
--**Summary**: 22/23 (95.65%) successful tests
++**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.891s
++**Average latency**: 2.520s
---
@@@ -500,38 -500,38 +507,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.995s | |
--| Text Transform Uppercase | ✅ Pass | 3.980s | |
--| Count from 1 to 5 | ✅ Pass | 3.836s | |
--| Math Calculation | ✅ Pass | 3.256s | |
--| Basic Echo Function | ✅ Pass | 4.078s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.645s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.757s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.314s | |
++| Simple Math | ✅ Pass | 3.191s | |
++| Text Transform Uppercase | ✅ Pass | 3.663s | |
++| Count from 1 to 5 | ✅ Pass | 4.359s | |
++| Math Calculation | ✅ Pass | 2.914s | |
++| Basic Echo Function | ✅ Pass | 3.995s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.860s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.531s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.597s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 4.023s | |
--| Search Query Function | ✅ Pass | 3.725s | |
--| Ask Advice Function | ✅ Pass | 4.173s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.589s | |
--| Basic Context Memory Test | ✅ Pass | 3.997s | |
--| Function Argument Memory Test | ✅ Pass | 4.753s | |
--| Function Response Memory Test | ✅ Pass | 4.861s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 5.704s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.880s | |
--| Penetration Testing Methodology | ✅ Pass | 6.182s | |
--| Vulnerability Assessment Tools | ✅ Pass | 5.410s | |
--| SQL Injection Attack Type | ✅ Pass | 3.714s | |
--| Penetration Testing Framework | ✅ Pass | 4.977s | |
--| Web Application Security Scanner | ✅ Pass | 5.953s | |
--| Penetration Testing Tool Selection | ✅ Pass | 5.708s | |
++| JSON Response Function | ✅ Pass | 4.768s | |
++| Search Query Function | ✅ Pass | 3.387s | |
++| Ask Advice Function | ✅ Pass | 4.440s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.496s | |
++| Basic Context Memory Test | ✅ Pass | 3.669s | |
++| Function Argument Memory Test | ✅ Pass | 3.852s | |
++| Function Response Memory Test | ✅ Pass | 3.675s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 8.701s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.324s | |
++| Penetration Testing Methodology | ✅ Pass | 5.095s | |
++| Vulnerability Assessment Tools | ✅ Pass | 6.883s | |
++| SQL Injection Attack Type | ✅ Pass | 3.547s | |
++| Penetration Testing Framework | ✅ Pass | 4.535s | |
++| Web Application Security Scanner | ✅ Pass | 5.704s | |
++| Penetration Testing Tool Selection | ❌ Fail | 12.517s | no tool calls found, expected at least 1 |
--**Summary**: 23/23 (100.00%) successful tests
++**Summary**: 22/23 (95.65%) successful tests
--**Average latency**: 4.197s
++**Average latency**: 4.466s
---
diff --cc examples/tests/glm-report.md
index 8d362137,8d362137..42893b69
--- a/examples/tests/glm-report.md
+++ b/examples/tests/glm-report.md
@@@ -1,27 -1,27 +1,27 @@@
# LLM Agent Testing Report
--Generated: Thu, 28 May 2026 12:25:41 UTC
++Generated: Tue, 21 Jul 2026 17:03:13 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
--| simple | glm-4.5-air | false | 23/23 (100.00%) | 3.213s |
--| simple_json | glm-4.5-air | false | 5/5 (100.00%) | 3.923s |
--| primary_agent | glm-5-turbo | true | 23/23 (100.00%) | 4.488s |
--| assistant | glm-5-turbo | true | 22/23 (95.65%) | 3.383s |
--| generator | glm-5.1 | true | 23/23 (100.00%) | 7.081s |
--| refiner | glm-5.1 | true | 22/23 (95.65%) | 6.864s |
--| adviser | glm-5.1 | true | 23/23 (100.00%) | 6.553s |
--| reflector | glm-4.5-air | true | 23/23 (100.00%) | 2.760s |
--| searcher | glm-4.5-air | true | 23/23 (100.00%) | 2.889s |
--| enricher | glm-4.5-air | true | 23/23 (100.00%) | 3.001s |
--| coder | glm-5.1 | true | 22/23 (95.65%) | 7.172s |
--| installer | glm-4.5-air | true | 23/23 (100.00%) | 6.146s |
--| pentester | glm-5.1 | true | 23/23 (100.00%) | 6.772s |
--
--**Total**: 278/281 (98.93%) successful tests
--**Overall average latency**: 5.007s
++| simple | glm-4.5-air | false | 22/23 (95.65%) | 0.236s |
++| simple_json | glm-4.5-air | false | 6/7 (85.71%) | 1.870s |
++| primary_agent | glm-5-turbo | true | 22/23 (95.65%) | 0.232s |
++| assistant | glm-5-turbo | true | 22/23 (95.65%) | 0.233s |
++| generator | glm-5.2 | true | 23/23 (100.00%) | 0.233s |
++| refiner | glm-5.2 | true | 23/23 (100.00%) | 0.231s |
++| adviser | glm-5.2 | true | 22/23 (95.65%) | 0.228s |
++| reflector | glm-4.5-air | false | 22/23 (95.65%) | 0.230s |
++| searcher | glm-4.5-air | false | 23/23 (100.00%) | 0.231s |
++| enricher | glm-4.5-air | false | 23/23 (100.00%) | 0.210s |
++| coder | glm-5.2 | true | 23/23 (100.00%) | 0.214s |
++| installer | glm-4.5-air | true | 23/23 (100.00%) | 0.213s |
++| pentester | glm-5.2 | true | 23/23 (100.00%) | 0.210s |
++
++**Total**: 277/283 (97.88%) successful tests
++**Overall average latency**: 0.266s
## Detailed Results
@@@ -31,38 -31,38 +31,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 4.356s | |
--| Text Transform Uppercase | ✅ Pass | 1.140s | |
--| Count from 1 to 5 | ✅ Pass | 1.534s | |
--| Math Calculation | ✅ Pass | 1.975s | |
--| Basic Echo Function | ✅ Pass | 1.529s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.911s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.859s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.170s | |
++| Simple Math | ✅ Pass | 0.428s | |
++| Text Transform Uppercase | ✅ Pass | 0.216s | |
++| Count from 1 to 5 | ✅ Pass | 0.214s | |
++| Math Calculation | ✅ Pass | 0.201s | |
++| Basic Echo Function | ✅ Pass | 0.205s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.203s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.215s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.220s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.415s | |
--| Search Query Function | ✅ Pass | 3.528s | |
--| Ask Advice Function | ✅ Pass | 1.866s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.467s | |
--| Basic Context Memory Test | ✅ Pass | 1.478s | |
--| Function Argument Memory Test | ✅ Pass | 1.072s | |
--| Function Response Memory Test | ✅ Pass | 2.755s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.953s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.161s | |
--| Penetration Testing Methodology | ✅ Pass | 7.480s | |
--| Vulnerability Assessment Tools | ✅ Pass | 18.361s | |
--| SQL Injection Attack Type | ✅ Pass | 1.326s | |
--| Penetration Testing Framework | ✅ Pass | 7.132s | |
--| Web Application Security Scanner | ✅ Pass | 3.189s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.219s | |
++| JSON Response Function | ✅ Pass | 0.212s | |
++| Search Query Function | ✅ Pass | 0.207s | |
++| Ask Advice Function | ✅ Pass | 0.206s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.249s | |
++| Basic Context Memory Test | ✅ Pass | 0.208s | |
++| Function Argument Memory Test | ✅ Pass | 0.218s | |
++| Function Response Memory Test | ✅ Pass | 0.496s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 0.256s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.206s | |
++| Penetration Testing Methodology | ✅ Pass | 0.205s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.213s | |
++| SQL Injection Attack Type | ✅ Pass | 0.208s | |
++| Penetration Testing Framework | ✅ Pass | 0.208s | |
++| Web Application Security Scanner | ✅ Pass | 0.205s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.220s | |
--**Summary**: 23/23 (100.00%) successful tests
++**Summary**: 22/23 (95.65%) successful tests
--**Average latency**: 3.213s
++**Average latency**: 0.236s
---
@@@ -72,15 -72,15 +72,22 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Person Information JSON | ✅ Pass | 4.516s | |
--| Vulnerability Report Memory Test | ✅ Pass | 5.454s | |
--| Project Information JSON | ✅ Pass | 1.830s | |
--| User Profile JSON | ✅ Pass | 3.002s | |
--| Streaming Person Information JSON Streaming | ✅ Pass | 4.808s | |
++| Person Information JSON | ✅ Pass | 1.448s | |
++| Streaming Person Information JSON Streaming | ✅ Pass | 1.375s | |
++| Project Information JSON | ✅ Pass | 1.654s | |
++| User Profile JSON | ✅ Pass | 1.684s | |
++| JSON Array Response Without Schema | ✅ Pass | 2.093s | |
++| Vulnerability Report Memory Test | ✅ Pass | 3.363s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Structured Output With JSON Schema | structured_output | ❌ Fail | 1.471s | structured output: response validation failed \(provider=openai model=zai/glm\-4\.5\-air choice=0 stop\_reason=stop\): response is not a single JSO... |
--**Summary**: 5/5 (100.00%) successful tests
++**Summary**: 6/7 (85.71%) successful tests
--**Average latency**: 3.923s
++**Average latency**: 1.870s
---
@@@ -90,38 -90,38 +97,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 5.935s | |
--| Text Transform Uppercase | ✅ Pass | 3.695s | |
--| Count from 1 to 5 | ✅ Pass | 3.836s | |
--| Math Calculation | ✅ Pass | 3.798s | |
--| Basic Echo Function | ✅ Pass | 2.191s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.538s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.337s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.491s | |
++| Simple Math | ✅ Pass | 0.423s | |
++| Text Transform Uppercase | ✅ Pass | 0.214s | |
++| Count from 1 to 5 | ✅ Pass | 0.219s | |
++| Math Calculation | ✅ Pass | 0.206s | |
++| Basic Echo Function | ✅ Pass | 0.205s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.202s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.219s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.221s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.857s | |
--| Search Query Function | ✅ Pass | 1.869s | |
--| Ask Advice Function | ✅ Pass | 1.275s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.561s | |
--| Basic Context Memory Test | ✅ Pass | 5.012s | |
--| Function Argument Memory Test | ✅ Pass | 3.120s | |
--| Function Response Memory Test | ✅ Pass | 1.293s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.664s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 4.496s | |
--| Penetration Testing Methodology | ✅ Pass | 22.967s | |
--| Vulnerability Assessment Tools | ✅ Pass | 9.455s | |
--| SQL Injection Attack Type | ✅ Pass | 5.215s | |
--| Penetration Testing Framework | ✅ Pass | 5.249s | |
--| Web Application Security Scanner | ✅ Pass | 5.172s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.180s | |
++| JSON Response Function | ✅ Pass | 0.209s | |
++| Search Query Function | ✅ Pass | 0.205s | |
++| Ask Advice Function | ✅ Pass | 0.204s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.213s | |
++| Basic Context Memory Test | ✅ Pass | 0.205s | |
++| Function Argument Memory Test | ✅ Pass | 0.219s | |
++| Function Response Memory Test | ✅ Pass | 0.480s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 0.239s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.206s | |
++| Penetration Testing Methodology | ✅ Pass | 0.215s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.205s | |
++| SQL Injection Attack Type | ✅ Pass | 0.206s | |
++| Penetration Testing Framework | ✅ Pass | 0.204s | |
++| Web Application Security Scanner | ✅ Pass | 0.207s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.208s | |
--**Summary**: 23/23 (100.00%) successful tests
++**Summary**: 22/23 (95.65%) successful tests
--**Average latency**: 4.488s
++**Average latency**: 0.232s
---
@@@ -131,161 -131,161 +138,161 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.980s | |
--| Text Transform Uppercase | ✅ Pass | 2.896s | |
--| Count from 1 to 5 | ✅ Pass | 2.572s | |
--| Math Calculation | ✅ Pass | 3.321s | |
--| Basic Echo Function | ✅ Pass | 1.431s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.935s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.138s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.242s | |
++| Simple Math | ✅ Pass | 0.399s | |
++| Text Transform Uppercase | ✅ Pass | 0.226s | |
++| Count from 1 to 5 | ✅ Pass | 0.216s | |
++| Math Calculation | ✅ Pass | 0.215s | |
++| Basic Echo Function | ✅ Pass | 0.207s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.204s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.219s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.225s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.378s | |
--| Search Query Function | ✅ Pass | 1.954s | |
--| Ask Advice Function | ✅ Pass | 1.638s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.390s | |
--| Basic Context Memory Test | ✅ Pass | 4.105s | |
--| Function Argument Memory Test | ✅ Pass | 1.651s | |
--| Function Response Memory Test | ✅ Pass | 2.104s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 6.155s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 5.861s | |
--| Penetration Testing Methodology | ✅ Pass | 5.751s | |
--| Vulnerability Assessment Tools | ❌ Fail | 6.224s | expected text 'network' not found |
--| SQL Injection Attack Type | ✅ Pass | 4.402s | |
--| Penetration Testing Framework | ✅ Pass | 6.187s | |
--| Web Application Security Scanner | ✅ Pass | 5.577s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.904s | |
++| JSON Response Function | ✅ Pass | 0.212s | |
++| Search Query Function | ✅ Pass | 0.203s | |
++| Ask Advice Function | ✅ Pass | 0.199s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.217s | |
++| Basic Context Memory Test | ✅ Pass | 0.207s | |
++| Function Argument Memory Test | ✅ Pass | 0.213s | |
++| Function Response Memory Test | ✅ Pass | 0.466s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 0.284s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.205s | |
++| Penetration Testing Methodology | ✅ Pass | 0.217s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.208s | |
++| SQL Injection Attack Type | ✅ Pass | 0.204s | |
++| Penetration Testing Framework | ✅ Pass | 0.204s | |
++| Web Application Security Scanner | ✅ Pass | 0.206s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.201s | |
**Summary**: 22/23 (95.65%) successful tests
--**Average latency**: 3.383s
++**Average latency**: 0.233s
---
--### generator (glm-5.1)
++### generator (glm-5.2)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 5.598s | |
--| Text Transform Uppercase | ✅ Pass | 4.686s | |
--| Count from 1 to 5 | ✅ Pass | 4.534s | |
--| Math Calculation | ✅ Pass | 4.466s | |
--| Basic Echo Function | ✅ Pass | 5.033s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.667s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.827s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.858s | |
++| Simple Math | ✅ Pass | 0.389s | |
++| Text Transform Uppercase | ✅ Pass | 0.223s | |
++| Count from 1 to 5 | ✅ Pass | 0.219s | |
++| Math Calculation | ✅ Pass | 0.208s | |
++| Basic Echo Function | ✅ Pass | 0.202s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.204s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.231s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.218s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 5.855s | |
--| Search Query Function | ✅ Pass | 5.443s | |
--| Ask Advice Function | ✅ Pass | 3.380s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 4.654s | |
--| Basic Context Memory Test | ✅ Pass | 7.745s | |
--| Function Argument Memory Test | ✅ Pass | 9.781s | |
--| Function Response Memory Test | ✅ Pass | 6.321s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 6.979s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 5.237s | |
--| Penetration Testing Methodology | ✅ Pass | 13.391s | |
--| Vulnerability Assessment Tools | ✅ Pass | 19.520s | |
--| SQL Injection Attack Type | ✅ Pass | 14.260s | |
--| Penetration Testing Framework | ✅ Pass | 7.151s | |
--| Web Application Security Scanner | ✅ Pass | 12.338s | |
--| Penetration Testing Tool Selection | ✅ Pass | 4.118s | |
++| JSON Response Function | ✅ Pass | 0.207s | |
++| Search Query Function | ✅ Pass | 0.203s | |
++| Ask Advice Function | ✅ Pass | 0.202s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.252s | |
++| Basic Context Memory Test | ✅ Pass | 0.214s | |
++| Function Argument Memory Test | ✅ Pass | 0.205s | |
++| Function Response Memory Test | ✅ Pass | 0.458s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.243s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.216s | |
++| Penetration Testing Methodology | ✅ Pass | 0.211s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.204s | |
++| SQL Injection Attack Type | ✅ Pass | 0.208s | |
++| Penetration Testing Framework | ✅ Pass | 0.205s | |
++| Web Application Security Scanner | ✅ Pass | 0.207s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.210s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 7.081s
++**Average latency**: 0.233s
---
--### refiner (glm-5.1)
++### refiner (glm-5.2)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 7.346s | |
--| Text Transform Uppercase | ✅ Pass | 7.388s | |
--| Count from 1 to 5 | ✅ Pass | 7.160s | |
--| Math Calculation | ✅ Pass | 4.476s | |
--| Basic Echo Function | ✅ Pass | 7.189s | |
--| Streaming Simple Math Streaming | ✅ Pass | 9.466s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.994s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.243s | |
++| Simple Math | ✅ Pass | 0.418s | |
++| Text Transform Uppercase | ✅ Pass | 0.207s | |
++| Count from 1 to 5 | ✅ Pass | 0.215s | |
++| Math Calculation | ✅ Pass | 0.209s | |
++| Basic Echo Function | ✅ Pass | 0.201s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.211s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.225s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.205s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 6.498s | |
--| Search Query Function | ✅ Pass | 5.493s | |
--| Ask Advice Function | ✅ Pass | 4.021s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 3.686s | |
--| Basic Context Memory Test | ✅ Pass | 5.153s | |
--| Function Argument Memory Test | ✅ Pass | 5.468s | |
--| Function Response Memory Test | ✅ Pass | 3.245s | |
--| Penetration Testing Memory with Tool Call | ❌ Fail | 7.345s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 4.404s | |
--| Penetration Testing Methodology | ✅ Pass | 12.575s | |
--| Vulnerability Assessment Tools | ✅ Pass | 16.644s | |
--| SQL Injection Attack Type | ✅ Pass | 5.347s | |
--| Penetration Testing Framework | ✅ Pass | 11.479s | |
--| Web Application Security Scanner | ✅ Pass | 12.258s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.992s | |
++| JSON Response Function | ✅ Pass | 0.201s | |
++| Search Query Function | ✅ Pass | 0.213s | |
++| Ask Advice Function | ✅ Pass | 0.199s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.252s | |
++| Basic Context Memory Test | ✅ Pass | 0.210s | |
++| Function Argument Memory Test | ✅ Pass | 0.206s | |
++| Function Response Memory Test | ✅ Pass | 0.444s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.206s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.243s | |
++| Penetration Testing Methodology | ✅ Pass | 0.216s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.205s | |
++| SQL Injection Attack Type | ✅ Pass | 0.209s | |
++| Penetration Testing Framework | ✅ Pass | 0.207s | |
++| Web Application Security Scanner | ✅ Pass | 0.208s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.200s | |
--**Summary**: 22/23 (95.65%) successful tests
++**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 6.864s
++**Average latency**: 0.231s
---
--### adviser (glm-5.1)
++### adviser (glm-5.2)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 7.436s | |
--| Text Transform Uppercase | ✅ Pass | 4.739s | |
--| Count from 1 to 5 | ✅ Pass | 7.323s | |
--| Math Calculation | ✅ Pass | 3.884s | |
--| Basic Echo Function | ✅ Pass | 4.145s | |
--| Streaming Simple Math Streaming | ✅ Pass | 4.102s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.763s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 5.518s | |
++| Simple Math | ✅ Pass | 0.389s | |
++| Text Transform Uppercase | ✅ Pass | 0.208s | |
++| Count from 1 to 5 | ✅ Pass | 0.213s | |
++| Math Calculation | ✅ Pass | 0.203s | |
++| Basic Echo Function | ✅ Pass | 0.201s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.221s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.228s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.207s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.657s | |
--| Search Query Function | ✅ Pass | 3.205s | |
--| Ask Advice Function | ✅ Pass | 3.559s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 3.959s | |
--| Basic Context Memory Test | ✅ Pass | 6.817s | |
--| Function Argument Memory Test | ✅ Pass | 6.147s | |
--| Function Response Memory Test | ✅ Pass | 3.902s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 5.443s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 5.905s | |
--| Penetration Testing Methodology | ✅ Pass | 14.182s | |
--| Vulnerability Assessment Tools | ✅ Pass | 19.240s | |
--| SQL Injection Attack Type | ✅ Pass | 5.849s | |
--| Penetration Testing Framework | ✅ Pass | 11.161s | |
--| Web Application Security Scanner | ✅ Pass | 11.757s | |
--| Penetration Testing Tool Selection | ✅ Pass | 4.019s | |
++| JSON Response Function | ✅ Pass | 0.198s | |
++| Search Query Function | ✅ Pass | 0.202s | |
++| Ask Advice Function | ✅ Pass | 0.196s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.215s | |
++| Basic Context Memory Test | ✅ Pass | 0.208s | |
++| Function Argument Memory Test | ✅ Pass | 0.208s | |
++| Function Response Memory Test | ✅ Pass | 0.431s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 0.210s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.251s | |
++| Penetration Testing Methodology | ✅ Pass | 0.212s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.215s | |
++| SQL Injection Attack Type | ✅ Pass | 0.213s | |
++| Penetration Testing Framework | ✅ Pass | 0.202s | |
++| Web Application Security Scanner | ✅ Pass | 0.208s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.199s | |
--**Summary**: 23/23 (100.00%) successful tests
++**Summary**: 22/23 (95.65%) successful tests
--**Average latency**: 6.553s
++**Average latency**: 0.228s
---
@@@ -295,38 -295,38 +302,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.349s | |
--| Text Transform Uppercase | ✅ Pass | 0.952s | |
--| Count from 1 to 5 | ✅ Pass | 3.336s | |
--| Math Calculation | ✅ Pass | 0.828s | |
--| Basic Echo Function | ✅ Pass | 1.318s | |
--| Streaming Simple Math Streaming | ✅ Pass | 8.738s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.624s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.163s | |
++| Simple Math | ✅ Pass | 0.418s | |
++| Text Transform Uppercase | ✅ Pass | 0.211s | |
++| Count from 1 to 5 | ✅ Pass | 0.220s | |
++| Math Calculation | ✅ Pass | 0.207s | |
++| Basic Echo Function | ✅ Pass | 0.204s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.212s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.215s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.210s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.864s | |
--| Search Query Function | ✅ Pass | 1.759s | |
--| Ask Advice Function | ✅ Pass | 1.985s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.501s | |
--| Basic Context Memory Test | ✅ Pass | 1.776s | |
--| Function Argument Memory Test | ✅ Pass | 1.869s | |
--| Function Response Memory Test | ✅ Pass | 1.430s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.434s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.338s | |
--| Penetration Testing Methodology | ✅ Pass | 6.325s | |
--| Vulnerability Assessment Tools | ✅ Pass | 5.048s | |
--| SQL Injection Attack Type | ✅ Pass | 2.589s | |
--| Penetration Testing Framework | ✅ Pass | 7.158s | |
--| Web Application Security Scanner | ✅ Pass | 5.490s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.597s | |
++| JSON Response Function | ✅ Pass | 0.201s | |
++| Search Query Function | ✅ Pass | 0.202s | |
++| Ask Advice Function | ✅ Pass | 0.197s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.253s | |
++| Basic Context Memory Test | ✅ Pass | 0.207s | |
++| Function Argument Memory Test | ✅ Pass | 0.224s | |
++| Function Response Memory Test | ✅ Pass | 0.410s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 0.216s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.237s | |
++| Penetration Testing Methodology | ✅ Pass | 0.208s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.213s | |
++| SQL Injection Attack Type | ✅ Pass | 0.204s | |
++| Penetration Testing Framework | ✅ Pass | 0.208s | |
++| Web Application Security Scanner | ✅ Pass | 0.211s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.199s | |
--**Summary**: 23/23 (100.00%) successful tests
++**Summary**: 22/23 (95.65%) successful tests
--**Average latency**: 2.760s
++**Average latency**: 0.230s
---
@@@ -336,38 -336,38 +343,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.144s | |
--| Text Transform Uppercase | ✅ Pass | 1.230s | |
--| Count from 1 to 5 | ✅ Pass | 2.526s | |
--| Math Calculation | ✅ Pass | 0.975s | |
--| Basic Echo Function | ✅ Pass | 1.962s | |
--| Streaming Simple Math Streaming | ✅ Pass | 6.438s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.872s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.207s | |
++| Simple Math | ✅ Pass | 0.418s | |
++| Text Transform Uppercase | ✅ Pass | 0.212s | |
++| Count from 1 to 5 | ✅ Pass | 0.218s | |
++| Math Calculation | ✅ Pass | 0.215s | |
++| Basic Echo Function | ✅ Pass | 0.214s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.209s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.215s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.203s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.837s | |
--| Search Query Function | ✅ Pass | 2.366s | |
--| Ask Advice Function | ✅ Pass | 1.564s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.558s | |
--| Basic Context Memory Test | ✅ Pass | 1.415s | |
--| Function Argument Memory Test | ✅ Pass | 1.913s | |
--| Function Response Memory Test | ✅ Pass | 1.785s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.208s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.507s | |
--| Penetration Testing Methodology | ✅ Pass | 5.352s | |
--| Vulnerability Assessment Tools | ✅ Pass | 14.510s | |
--| SQL Injection Attack Type | ✅ Pass | 2.434s | |
--| Penetration Testing Framework | ✅ Pass | 5.262s | |
--| Web Application Security Scanner | ✅ Pass | 2.488s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.891s | |
++| JSON Response Function | ✅ Pass | 0.204s | |
++| Search Query Function | ✅ Pass | 0.201s | |
++| Ask Advice Function | ✅ Pass | 0.203s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.229s | |
++| Basic Context Memory Test | ✅ Pass | 0.218s | |
++| Function Argument Memory Test | ✅ Pass | 0.207s | |
++| Function Response Memory Test | ✅ Pass | 0.400s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.243s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.239s | |
++| Penetration Testing Methodology | ✅ Pass | 0.210s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.208s | |
++| SQL Injection Attack Type | ✅ Pass | 0.206s | |
++| Penetration Testing Framework | ✅ Pass | 0.208s | |
++| Web Application Security Scanner | ✅ Pass | 0.204s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.209s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.889s
++**Average latency**: 0.231s
---
@@@ -377,79 -377,79 +384,79 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.166s | |
--| Text Transform Uppercase | ✅ Pass | 1.054s | |
--| Count from 1 to 5 | ✅ Pass | 1.600s | |
--| Math Calculation | ✅ Pass | 1.054s | |
--| Basic Echo Function | ✅ Pass | 1.737s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.306s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.486s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.403s | |
++| Simple Math | ✅ Pass | 0.211s | |
++| Text Transform Uppercase | ✅ Pass | 0.212s | |
++| Count from 1 to 5 | ✅ Pass | 0.206s | |
++| Math Calculation | ✅ Pass | 0.216s | |
++| Basic Echo Function | ✅ Pass | 0.208s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.201s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.208s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.204s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.452s | |
--| Search Query Function | ✅ Pass | 1.895s | |
--| Ask Advice Function | ✅ Pass | 1.631s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.771s | |
--| Basic Context Memory Test | ✅ Pass | 1.640s | |
--| Function Argument Memory Test | ✅ Pass | 2.034s | |
--| Function Response Memory Test | ✅ Pass | 1.623s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.927s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.506s | |
--| Penetration Testing Methodology | ✅ Pass | 8.173s | |
--| Vulnerability Assessment Tools | ✅ Pass | 12.181s | |
--| SQL Injection Attack Type | ✅ Pass | 1.488s | |
--| Penetration Testing Framework | ✅ Pass | 14.886s | |
--| Web Application Security Scanner | ✅ Pass | 2.242s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.759s | |
++| JSON Response Function | ✅ Pass | 0.202s | |
++| Search Query Function | ✅ Pass | 0.200s | |
++| Ask Advice Function | ✅ Pass | 0.202s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.219s | |
++| Basic Context Memory Test | ✅ Pass | 0.207s | |
++| Function Argument Memory Test | ✅ Pass | 0.210s | |
++| Function Response Memory Test | ✅ Pass | 0.208s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.239s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.210s | |
++| Penetration Testing Methodology | ✅ Pass | 0.215s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.204s | |
++| SQL Injection Attack Type | ✅ Pass | 0.209s | |
++| Penetration Testing Framework | ✅ Pass | 0.208s | |
++| Web Application Security Scanner | ✅ Pass | 0.208s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.209s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 3.001s
++**Average latency**: 0.210s
---
--### coder (glm-5.1)
++### coder (glm-5.2)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 4.213s | |
--| Text Transform Uppercase | ✅ Pass | 4.692s | |
--| Count from 1 to 5 | ✅ Pass | 5.795s | |
--| Math Calculation | ✅ Pass | 9.208s | |
--| Basic Echo Function | ✅ Pass | 5.473s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.389s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 8.238s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.129s | |
++| Simple Math | ✅ Pass | 0.221s | |
++| Text Transform Uppercase | ✅ Pass | 0.216s | |
++| Count from 1 to 5 | ✅ Pass | 0.204s | |
++| Math Calculation | ✅ Pass | 0.215s | |
++| Basic Echo Function | ✅ Pass | 0.202s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.202s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.211s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.204s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.931s | |
--| Search Query Function | ✅ Pass | 3.264s | |
--| Ask Advice Function | ✅ Pass | 7.299s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 3.841s | |
--| Basic Context Memory Test | ✅ Pass | 5.589s | |
--| Function Argument Memory Test | ✅ Pass | 6.503s | |
--| Function Response Memory Test | ✅ Pass | 3.301s | |
--| Penetration Testing Memory with Tool Call | ❌ Fail | 6.407s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 6.279s | |
--| Penetration Testing Methodology | ✅ Pass | 17.570s | |
--| Vulnerability Assessment Tools | ✅ Pass | 19.116s | |
--| SQL Injection Attack Type | ✅ Pass | 5.596s | |
--| Penetration Testing Framework | ✅ Pass | 16.889s | |
--| Web Application Security Scanner | ✅ Pass | 11.733s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.497s | |
++| JSON Response Function | ✅ Pass | 0.206s | |
++| Search Query Function | ✅ Pass | 0.205s | |
++| Ask Advice Function | ✅ Pass | 0.206s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.218s | |
++| Basic Context Memory Test | ✅ Pass | 0.212s | |
++| Function Argument Memory Test | ✅ Pass | 0.220s | |
++| Function Response Memory Test | ✅ Pass | 0.259s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.237s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.207s | |
++| Penetration Testing Methodology | ✅ Pass | 0.212s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.208s | |
++| SQL Injection Attack Type | ✅ Pass | 0.209s | |
++| Penetration Testing Framework | ✅ Pass | 0.210s | |
++| Web Application Security Scanner | ✅ Pass | 0.211s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.202s | |
--**Summary**: 22/23 (95.65%) successful tests
++**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 7.172s
++**Average latency**: 0.214s
---
@@@ -459,79 -459,79 +466,79 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 6.253s | |
--| Text Transform Uppercase | ✅ Pass | 2.297s | |
--| Count from 1 to 5 | ✅ Pass | 3.713s | |
--| Math Calculation | ✅ Pass | 2.951s | |
--| Basic Echo Function | ✅ Pass | 2.057s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.884s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.290s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.606s | |
++| Simple Math | ✅ Pass | 0.219s | |
++| Text Transform Uppercase | ✅ Pass | 0.223s | |
++| Count from 1 to 5 | ✅ Pass | 0.202s | |
++| Math Calculation | ✅ Pass | 0.209s | |
++| Basic Echo Function | ✅ Pass | 0.205s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.207s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.210s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.203s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 4.104s | |
--| Search Query Function | ✅ Pass | 3.696s | |
--| Ask Advice Function | ✅ Pass | 3.538s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 3.188s | |
--| Basic Context Memory Test | ✅ Pass | 3.317s | |
--| Function Argument Memory Test | ✅ Pass | 1.915s | |
--| Function Response Memory Test | ✅ Pass | 3.620s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 5.784s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.012s | |
--| Penetration Testing Methodology | ✅ Pass | 14.559s | |
--| Vulnerability Assessment Tools | ✅ Pass | 16.449s | |
--| SQL Injection Attack Type | ✅ Pass | 8.793s | |
--| Penetration Testing Framework | ✅ Pass | 22.365s | |
--| Web Application Security Scanner | ✅ Pass | 15.217s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.740s | |
++| JSON Response Function | ✅ Pass | 0.205s | |
++| Search Query Function | ✅ Pass | 0.202s | |
++| Ask Advice Function | ✅ Pass | 0.210s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.260s | |
++| Basic Context Memory Test | ✅ Pass | 0.205s | |
++| Function Argument Memory Test | ✅ Pass | 0.204s | |
++| Function Response Memory Test | ✅ Pass | 0.247s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.237s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.205s | |
++| Penetration Testing Methodology | ✅ Pass | 0.207s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.206s | |
++| SQL Injection Attack Type | ✅ Pass | 0.206s | |
++| Penetration Testing Framework | ✅ Pass | 0.205s | |
++| Web Application Security Scanner | ✅ Pass | 0.217s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.202s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 6.146s
++**Average latency**: 0.213s
---
--### pentester (glm-5.1)
++### pentester (glm-5.2)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 5.261s | |
--| Text Transform Uppercase | ✅ Pass | 6.631s | |
--| Count from 1 to 5 | ✅ Pass | 4.222s | |
--| Math Calculation | ✅ Pass | 3.124s | |
--| Basic Echo Function | ✅ Pass | 3.221s | |
--| Streaming Simple Math Streaming | ✅ Pass | 6.244s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.420s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.163s | |
++| Simple Math | ✅ Pass | 0.234s | |
++| Text Transform Uppercase | ✅ Pass | 0.221s | |
++| Count from 1 to 5 | ✅ Pass | 0.213s | |
++| Math Calculation | ✅ Pass | 0.212s | |
++| Basic Echo Function | ✅ Pass | 0.199s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.199s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.206s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.208s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 4.696s | |
--| Search Query Function | ✅ Pass | 2.983s | |
--| Ask Advice Function | ✅ Pass | 5.813s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 9.633s | |
--| Basic Context Memory Test | ✅ Pass | 4.770s | |
--| Function Argument Memory Test | ✅ Pass | 7.549s | |
--| Function Response Memory Test | ✅ Pass | 3.946s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.723s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.120s | |
--| Penetration Testing Methodology | ✅ Pass | 17.474s | |
--| Vulnerability Assessment Tools | ✅ Pass | 21.649s | |
--| SQL Injection Attack Type | ✅ Pass | 6.494s | |
--| Penetration Testing Framework | ✅ Pass | 12.428s | |
--| Web Application Security Scanner | ✅ Pass | 8.468s | |
--| Penetration Testing Tool Selection | ✅ Pass | 4.718s | |
++| JSON Response Function | ✅ Pass | 0.206s | |
++| Search Query Function | ✅ Pass | 0.200s | |
++| Ask Advice Function | ✅ Pass | 0.207s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.203s | |
++| Basic Context Memory Test | ✅ Pass | 0.207s | |
++| Function Argument Memory Test | ✅ Pass | 0.203s | |
++| Function Response Memory Test | ✅ Pass | 0.222s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.218s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.210s | |
++| Penetration Testing Methodology | ✅ Pass | 0.212s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.204s | |
++| SQL Injection Attack Type | ✅ Pass | 0.208s | |
++| Penetration Testing Framework | ✅ Pass | 0.210s | |
++| Web Application Security Scanner | ✅ Pass | 0.218s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.207s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 6.772s
++**Average latency**: 0.210s
---
diff --cc examples/tests/kimi-report.md
index b6effc43,b6effc43..e668a033
--- a/examples/tests/kimi-report.md
+++ b/examples/tests/kimi-report.md
@@@ -1,27 -1,27 +1,27 @@@
# LLM Agent Testing Report
--Generated: Wed, 27 May 2026 23:34:01 UTC
++Generated: Tue, 21 Jul 2026 16:45:17 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
--| simple | kimi-k2.5 | true | 23/23 (100.00%) | 1.452s |
--| simple_json | kimi-k2.5 | false | 5/5 (100.00%) | 1.241s |
--| primary_agent | kimi-k2.5 | true | 23/23 (100.00%) | 3.602s |
--| assistant | kimi-k2.5 | true | 23/23 (100.00%) | 3.660s |
--| generator | kimi-k2.6 | true | 23/23 (100.00%) | 3.471s |
--| refiner | kimi-k2.6 | true | 23/23 (100.00%) | 3.700s |
--| adviser | kimi-k2.6 | true | 23/23 (100.00%) | 3.886s |
--| reflector | kimi-k2.5 | true | 23/23 (100.00%) | 1.526s |
--| searcher | kimi-k2.5 | true | 23/23 (100.00%) | 1.383s |
--| enricher | kimi-k2.5 | true | 23/23 (100.00%) | 1.271s |
--| coder | kimi-k2.6 | true | 23/23 (100.00%) | 3.321s |
--| installer | kimi-k2.5 | true | 23/23 (100.00%) | 3.381s |
--| pentester | kimi-k2.6 | true | 23/23 (100.00%) | 3.432s |
--
--**Total**: 281/281 (100.00%) successful tests
--**Overall average latency**: 2.812s
++| simple | kimi-k2.5 | true | 23/23 (100.00%) | 1.748s |
++| simple_json | kimi-k2.5 | true | 7/7 (100.00%) | 2.417s |
++| primary_agent | kimi-k2.7-code-highspeed | true | 23/23 (100.00%) | 0.918s |
++| assistant | kimi-k2.7-code-highspeed | true | 23/23 (100.00%) | 0.859s |
++| generator | kimi-k3 | true | 22/23 (95.65%) | 6.887s |
++| refiner | kimi-k3 | true | 22/23 (95.65%) | 5.991s |
++| adviser | kimi-k3 | true | 22/23 (95.65%) | 6.501s |
++| reflector | kimi-k2.5 | true | 23/23 (100.00%) | 3.374s |
++| searcher | kimi-k2.5 | true | 23/23 (100.00%) | 3.541s |
++| enricher | kimi-k2.5 | true | 23/23 (100.00%) | 2.676s |
++| coder | kimi-k2.7-code | true | 23/23 (100.00%) | 1.857s |
++| installer | kimi-k2.5 | true | 23/23 (100.00%) | 4.023s |
++| pentester | kimi-k2.7-code | true | 23/23 (100.00%) | 1.658s |
++
++**Total**: 280/283 (98.94%) successful tests
++**Overall average latency**: 3.313s
## Detailed Results
@@@ -31,38 -31,38 +31,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.967s | |
--| Text Transform Uppercase | ✅ Pass | 0.754s | |
--| Count from 1 to 5 | ✅ Pass | 0.898s | |
--| Math Calculation | ✅ Pass | 0.592s | |
--| Basic Echo Function | ✅ Pass | 1.073s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.716s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.857s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.202s | |
++| Simple Math | ✅ Pass | 1.168s | |
++| Text Transform Uppercase | ✅ Pass | 0.847s | |
++| Count from 1 to 5 | ✅ Pass | 1.035s | |
++| Math Calculation | ✅ Pass | 0.850s | |
++| Basic Echo Function | ✅ Pass | 1.267s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.791s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.979s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.140s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.030s | |
--| Search Query Function | ✅ Pass | 0.991s | |
--| Ask Advice Function | ✅ Pass | 1.341s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.123s | |
--| Basic Context Memory Test | ✅ Pass | 0.844s | |
--| Function Argument Memory Test | ✅ Pass | 0.811s | |
--| Function Response Memory Test | ✅ Pass | 0.827s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.086s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.772s | |
--| Penetration Testing Methodology | ✅ Pass | 2.927s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.957s | |
--| SQL Injection Attack Type | ✅ Pass | 0.878s | |
--| Penetration Testing Framework | ✅ Pass | 3.484s | |
--| Web Application Security Scanner | ✅ Pass | 3.183s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.066s | |
++| JSON Response Function | ✅ Pass | 1.398s | |
++| Search Query Function | ✅ Pass | 1.141s | |
++| Ask Advice Function | ✅ Pass | 3.642s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.218s | |
++| Basic Context Memory Test | ✅ Pass | 1.036s | |
++| Function Argument Memory Test | ✅ Pass | 0.924s | |
++| Function Response Memory Test | ✅ Pass | 0.819s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.141s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.905s | |
++| Penetration Testing Methodology | ✅ Pass | 6.074s | |
++| Vulnerability Assessment Tools | ✅ Pass | 3.232s | |
++| SQL Injection Attack Type | ✅ Pass | 1.155s | |
++| Penetration Testing Framework | ✅ Pass | 4.287s | |
++| Web Application Security Scanner | ✅ Pass | 2.473s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.677s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.452s
++**Average latency**: 1.748s
---
@@@ -72,220 -72,220 +72,227 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Project Information JSON | ✅ Pass | 1.047s | |
--| Person Information JSON | ✅ Pass | 1.194s | |
--| Vulnerability Report Memory Test | ✅ Pass | 2.029s | |
--| User Profile JSON | ✅ Pass | 0.951s | |
--| Streaming Person Information JSON Streaming | ✅ Pass | 0.984s | |
++| Project Information JSON | ✅ Pass | 1.108s | |
++| User Profile JSON | ✅ Pass | 1.267s | |
++| Person Information JSON | ✅ Pass | 3.657s | |
++| Vulnerability Report Memory Test | ✅ Pass | 3.801s | |
++| Streaming Person Information JSON Streaming | ✅ Pass | 3.193s | |
++| JSON Array Response Without Schema | ✅ Pass | 3.663s | |
++
++#### Capability Tests
--**Summary**: 5/5 (100.00%) successful tests
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Structured Output With JSON Schema | structured_output | ✅ Pass | 0.225s | |
--**Average latency**: 1.241s
++**Summary**: 7/7 (100.00%) successful tests
++
++**Average latency**: 2.417s
---
--### primary_agent (kimi-k2.5)
++### primary_agent (kimi-k2.7-code-highspeed)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.805s | |
--| Text Transform Uppercase | ✅ Pass | 2.413s | |
--| Count from 1 to 5 | ✅ Pass | 2.075s | |
--| Math Calculation | ✅ Pass | 1.364s | |
--| Basic Echo Function | ✅ Pass | 1.724s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.629s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.851s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.895s | |
++| Simple Math | ✅ Pass | 0.475s | |
++| Text Transform Uppercase | ✅ Pass | 0.213s | |
++| Count from 1 to 5 | ✅ Pass | 0.217s | |
++| Math Calculation | ✅ Pass | 0.218s | |
++| Basic Echo Function | ✅ Pass | 0.234s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.215s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.243s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.008s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.622s | |
--| Search Query Function | ✅ Pass | 1.869s | |
--| Ask Advice Function | ✅ Pass | 2.412s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.302s | |
--| Basic Context Memory Test | ✅ Pass | 2.121s | |
--| Function Argument Memory Test | ✅ Pass | 2.266s | |
--| Function Response Memory Test | ✅ Pass | 1.705s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.334s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.341s | |
--| Penetration Testing Methodology | ✅ Pass | 11.752s | |
--| Vulnerability Assessment Tools | ✅ Pass | 7.759s | |
--| SQL Injection Attack Type | ✅ Pass | 2.492s | |
--| Penetration Testing Framework | ✅ Pass | 12.965s | |
--| Web Application Security Scanner | ✅ Pass | 8.206s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.925s | |
++| JSON Response Function | ✅ Pass | 1.081s | |
++| Search Query Function | ✅ Pass | 0.896s | |
++| Ask Advice Function | ✅ Pass | 1.040s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.019s | |
++| Basic Context Memory Test | ✅ Pass | 1.074s | |
++| Function Argument Memory Test | ✅ Pass | 1.040s | |
++| Function Response Memory Test | ✅ Pass | 0.944s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.317s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.104s | |
++| Penetration Testing Methodology | ✅ Pass | 1.063s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.716s | |
++| SQL Injection Attack Type | ✅ Pass | 1.002s | |
++| Penetration Testing Framework | ✅ Pass | 1.607s | |
++| Web Application Security Scanner | ✅ Pass | 1.204s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.160s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 3.602s
++**Average latency**: 0.918s
---
--### assistant (kimi-k2.5)
++### assistant (kimi-k2.7-code-highspeed)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.792s | |
--| Text Transform Uppercase | ✅ Pass | 1.857s | |
--| Count from 1 to 5 | ✅ Pass | 3.005s | |
--| Math Calculation | ✅ Pass | 1.790s | |
--| Basic Echo Function | ✅ Pass | 1.660s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.064s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.648s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.884s | |
++| Simple Math | ✅ Pass | 0.473s | |
++| Text Transform Uppercase | ✅ Pass | 0.213s | |
++| Count from 1 to 5 | ✅ Pass | 0.208s | |
++| Math Calculation | ✅ Pass | 0.212s | |
++| Basic Echo Function | ✅ Pass | 0.216s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.217s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.166s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.120s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.934s | |
--| Search Query Function | ✅ Pass | 1.837s | |
--| Ask Advice Function | ✅ Pass | 2.343s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.786s | |
--| Basic Context Memory Test | ✅ Pass | 2.777s | |
--| Function Argument Memory Test | ✅ Pass | 2.055s | |
--| Function Response Memory Test | ✅ Pass | 1.940s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.760s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.796s | |
--| Penetration Testing Methodology | ✅ Pass | 10.671s | |
--| Vulnerability Assessment Tools | ✅ Pass | 10.371s | |
--| SQL Injection Attack Type | ✅ Pass | 2.529s | |
--| Penetration Testing Framework | ✅ Pass | 11.063s | |
--| Web Application Security Scanner | ✅ Pass | 6.830s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.769s | |
++| JSON Response Function | ✅ Pass | 1.086s | |
++| Search Query Function | ✅ Pass | 1.038s | |
++| Ask Advice Function | ✅ Pass | 2.175s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.014s | |
++| Basic Context Memory Test | ✅ Pass | 1.012s | |
++| Function Argument Memory Test | ✅ Pass | 1.053s | |
++| Function Response Memory Test | ✅ Pass | 1.089s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.377s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.055s | |
++| Penetration Testing Methodology | ✅ Pass | 1.270s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.209s | |
++| SQL Injection Attack Type | ✅ Pass | 0.224s | |
++| Penetration Testing Framework | ✅ Pass | 1.119s | |
++| Web Application Security Scanner | ✅ Pass | 0.997s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.199s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 3.660s
++**Average latency**: 0.859s
---
--### generator (kimi-k2.6)
++### generator (kimi-k3)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.818s | |
--| Text Transform Uppercase | ✅ Pass | 4.989s | |
--| Count from 1 to 5 | ✅ Pass | 2.068s | |
--| Math Calculation | ✅ Pass | 1.499s | |
--| Basic Echo Function | ✅ Pass | 1.577s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.753s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.802s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.221s | |
++| Simple Math | ✅ Pass | 3.409s | |
++| Text Transform Uppercase | ✅ Pass | 3.522s | |
++| Count from 1 to 5 | ✅ Pass | 3.907s | |
++| Math Calculation | ✅ Pass | 2.995s | |
++| Basic Echo Function | ✅ Pass | 4.075s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.937s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.215s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 3.155s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.911s | |
--| Search Query Function | ✅ Pass | 1.693s | |
--| Ask Advice Function | ✅ Pass | 2.063s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.638s | |
--| Basic Context Memory Test | ✅ Pass | 2.781s | |
--| Function Argument Memory Test | ✅ Pass | 1.929s | |
--| Function Response Memory Test | ✅ Pass | 1.722s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.330s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 6.770s | |
--| Penetration Testing Methodology | ✅ Pass | 5.593s | |
--| Vulnerability Assessment Tools | ✅ Pass | 9.689s | |
--| SQL Injection Attack Type | ✅ Pass | 3.238s | |
--| Penetration Testing Framework | ✅ Pass | 6.540s | |
--| Web Application Security Scanner | ✅ Pass | 4.246s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.942s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 3.471s
++| JSON Response Function | ✅ Pass | 4.522s | |
++| Search Query Function | ✅ Pass | 5.932s | |
++| Ask Advice Function | ✅ Pass | 3.868s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 4.012s | |
++| Basic Context Memory Test | ✅ Pass | 5.962s | |
++| Function Argument Memory Test | ✅ Pass | 4.789s | |
++| Function Response Memory Test | ✅ Pass | 4.603s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 9.806s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 4.822s | |
++| Penetration Testing Methodology | ✅ Pass | 20.204s | |
++| Vulnerability Assessment Tools | ✅ Pass | 15.992s | |
++| SQL Injection Attack Type | ✅ Pass | 5.997s | |
++| Penetration Testing Framework | ✅ Pass | 19.052s | |
++| Web Application Security Scanner | ✅ Pass | 11.354s | |
++| Penetration Testing Tool Selection | ✅ Pass | 9.255s | |
++
++**Summary**: 22/23 (95.65%) successful tests
++
++**Average latency**: 6.887s
---
--### refiner (kimi-k2.6)
++### refiner (kimi-k3)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.930s | |
--| Text Transform Uppercase | ✅ Pass | 1.828s | |
--| Count from 1 to 5 | ✅ Pass | 2.079s | |
--| Math Calculation | ✅ Pass | 2.422s | |
--| Basic Echo Function | ✅ Pass | 1.638s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.739s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.457s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.056s | |
++| Simple Math | ✅ Pass | 3.383s | |
++| Text Transform Uppercase | ✅ Pass | 3.330s | |
++| Count from 1 to 5 | ✅ Pass | 4.068s | |
++| Math Calculation | ✅ Pass | 5.388s | |
++| Basic Echo Function | ✅ Pass | 4.278s | |
++| Streaming Simple Math Streaming | ✅ Pass | 3.022s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.832s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 3.464s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.574s | |
--| Search Query Function | ✅ Pass | 3.115s | |
--| Ask Advice Function | ✅ Pass | 1.961s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.807s | |
--| Basic Context Memory Test | ✅ Pass | 1.837s | |
--| Function Argument Memory Test | ✅ Pass | 2.727s | |
--| Function Response Memory Test | ✅ Pass | 2.079s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.531s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.579s | |
--| Penetration Testing Methodology | ✅ Pass | 12.756s | |
--| Vulnerability Assessment Tools | ✅ Pass | 10.127s | |
--| SQL Injection Attack Type | ✅ Pass | 4.953s | |
--| Penetration Testing Framework | ✅ Pass | 8.007s | |
--| Web Application Security Scanner | ✅ Pass | 4.581s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.309s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 3.700s
++| JSON Response Function | ✅ Pass | 4.040s | |
++| Search Query Function | ✅ Pass | 3.786s | |
++| Ask Advice Function | ✅ Pass | 4.272s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 3.479s | |
++| Basic Context Memory Test | ✅ Pass | 4.717s | |
++| Function Argument Memory Test | ✅ Pass | 6.360s | |
++| Function Response Memory Test | ✅ Pass | 3.190s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 7.865s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 4.861s | |
++| Penetration Testing Methodology | ✅ Pass | 19.021s | |
++| Vulnerability Assessment Tools | ✅ Pass | 10.633s | |
++| SQL Injection Attack Type | ✅ Pass | 5.304s | |
++| Penetration Testing Framework | ✅ Pass | 11.172s | |
++| Web Application Security Scanner | ✅ Pass | 12.213s | |
++| Penetration Testing Tool Selection | ✅ Pass | 6.113s | |
++
++**Summary**: 22/23 (95.65%) successful tests
++
++**Average latency**: 5.991s
---
--### adviser (kimi-k2.6)
++### adviser (kimi-k3)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 4.341s | |
--| Text Transform Uppercase | ✅ Pass | 1.662s | |
--| Count from 1 to 5 | ✅ Pass | 2.051s | |
--| Basic Echo Function | ✅ Pass | 1.700s | |
--| Math Calculation | ✅ Pass | 4.090s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.907s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.154s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.487s | |
++| Simple Math | ✅ Pass | 2.838s | |
++| Text Transform Uppercase | ✅ Pass | 3.910s | |
++| Count from 1 to 5 | ✅ Pass | 4.158s | |
++| Math Calculation | ✅ Pass | 2.513s | |
++| Basic Echo Function | ✅ Pass | 4.136s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.902s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.926s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 5.287s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.825s | |
--| Search Query Function | ✅ Pass | 1.781s | |
--| Ask Advice Function | ✅ Pass | 2.140s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 4.836s | |
--| Basic Context Memory Test | ✅ Pass | 2.497s | |
--| Function Argument Memory Test | ✅ Pass | 1.844s | |
--| Function Response Memory Test | ✅ Pass | 2.066s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 5.170s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 6.002s | |
--| Penetration Testing Methodology | ✅ Pass | 7.523s | |
--| Vulnerability Assessment Tools | ✅ Pass | 7.854s | |
--| SQL Injection Attack Type | ✅ Pass | 2.703s | |
--| Penetration Testing Framework | ✅ Pass | 10.072s | |
--| Web Application Security Scanner | ✅ Pass | 5.844s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.817s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 3.886s
++| JSON Response Function | ❌ Fail | 4.621s | no tool calls found, expected at least 1 |
++| Search Query Function | ✅ Pass | 5.229s | |
++| Ask Advice Function | ✅ Pass | 4.019s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 4.333s | |
++| Basic Context Memory Test | ✅ Pass | 7.273s | |
++| Function Argument Memory Test | ✅ Pass | 4.714s | |
++| Function Response Memory Test | ✅ Pass | 3.870s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 8.639s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 6.006s | |
++| Penetration Testing Methodology | ✅ Pass | 18.671s | |
++| Vulnerability Assessment Tools | ✅ Pass | 14.315s | |
++| SQL Injection Attack Type | ✅ Pass | 5.862s | |
++| Penetration Testing Framework | ✅ Pass | 14.244s | |
++| Web Application Security Scanner | ✅ Pass | 11.567s | |
++| Penetration Testing Tool Selection | ✅ Pass | 7.487s | |
++
++**Summary**: 22/23 (95.65%) successful tests
++
++**Average latency**: 6.501s
---
@@@ -295,38 -295,38 +302,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.557s | |
--| Text Transform Uppercase | ✅ Pass | 0.691s | |
--| Count from 1 to 5 | ✅ Pass | 0.821s | |
--| Math Calculation | ✅ Pass | 0.788s | |
--| Basic Echo Function | ✅ Pass | 1.170s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.658s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.917s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.215s | |
++| Simple Math | ✅ Pass | 1.784s | |
++| Text Transform Uppercase | ✅ Pass | 0.986s | |
++| Count from 1 to 5 | ✅ Pass | 1.347s | |
++| Math Calculation | ✅ Pass | 2.671s | |
++| Basic Echo Function | ✅ Pass | 4.613s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.516s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.414s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 3.371s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.349s | |
--| Search Query Function | ✅ Pass | 1.016s | |
--| Ask Advice Function | ✅ Pass | 1.553s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.116s | |
--| Basic Context Memory Test | ✅ Pass | 1.271s | |
++| JSON Response Function | ✅ Pass | 5.791s | |
++| Search Query Function | ✅ Pass | 1.454s | |
++| Ask Advice Function | ✅ Pass | 5.769s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 3.161s | |
++| Basic Context Memory Test | ✅ Pass | 2.177s | |
| Function Argument Memory Test | ✅ Pass | 0.825s | |
--| Function Response Memory Test | ✅ Pass | 0.693s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.984s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.656s | |
--| Penetration Testing Methodology | ✅ Pass | 3.017s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.796s | |
--| SQL Injection Attack Type | ✅ Pass | 0.961s | |
--| Penetration Testing Framework | ✅ Pass | 4.904s | |
--| Web Application Security Scanner | ✅ Pass | 2.988s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.130s | |
++| Function Response Memory Test | ✅ Pass | 0.815s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.192s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.839s | |
++| Penetration Testing Methodology | ✅ Pass | 9.492s | |
++| Vulnerability Assessment Tools | ✅ Pass | 8.875s | |
++| SQL Injection Attack Type | ✅ Pass | 1.054s | |
++| Penetration Testing Framework | ✅ Pass | 8.044s | |
++| Web Application Security Scanner | ✅ Pass | 2.056s | |
++| Penetration Testing Tool Selection | ✅ Pass | 5.347s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.526s
++**Average latency**: 3.374s
---
@@@ -336,38 -336,38 +343,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.629s | |
--| Text Transform Uppercase | ✅ Pass | 0.720s | |
--| Count from 1 to 5 | ✅ Pass | 0.826s | |
--| Math Calculation | ✅ Pass | 1.210s | |
--| Basic Echo Function | ✅ Pass | 1.057s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.705s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.748s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.239s | |
++| Simple Math | ✅ Pass | 1.660s | |
++| Text Transform Uppercase | ✅ Pass | 1.039s | |
++| Count from 1 to 5 | ✅ Pass | 1.295s | |
++| Math Calculation | ✅ Pass | 1.552s | |
++| Basic Echo Function | ✅ Pass | 4.212s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.500s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.993s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.287s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.191s | |
--| Search Query Function | ✅ Pass | 1.057s | |
--| Ask Advice Function | ✅ Pass | 1.340s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.069s | |
--| Basic Context Memory Test | ✅ Pass | 1.075s | |
--| Function Argument Memory Test | ✅ Pass | 0.814s | |
--| Function Response Memory Test | ✅ Pass | 0.680s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.213s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.775s | |
--| Penetration Testing Methodology | ✅ Pass | 3.708s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.575s | |
--| SQL Injection Attack Type | ✅ Pass | 0.897s | |
--| Penetration Testing Framework | ✅ Pass | 2.486s | |
--| Web Application Security Scanner | ✅ Pass | 1.508s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.267s | |
++| JSON Response Function | ✅ Pass | 5.724s | |
++| Search Query Function | ✅ Pass | 5.527s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.268s | |
++| Ask Advice Function | ✅ Pass | 5.755s | |
++| Basic Context Memory Test | ✅ Pass | 2.214s | |
++| Function Argument Memory Test | ✅ Pass | 0.813s | |
++| Function Response Memory Test | ✅ Pass | 0.886s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.317s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.290s | |
++| Penetration Testing Methodology | ✅ Pass | 9.712s | |
++| Vulnerability Assessment Tools | ✅ Pass | 8.599s | |
++| SQL Injection Attack Type | ✅ Pass | 1.166s | |
++| Penetration Testing Framework | ✅ Pass | 12.429s | |
++| Web Application Security Scanner | ✅ Pass | 6.200s | |
++| Penetration Testing Tool Selection | ✅ Pass | 4.004s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.383s
++**Average latency**: 3.541s
---
@@@ -377,79 -377,79 +384,79 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.721s | |
--| Text Transform Uppercase | ✅ Pass | 0.799s | |
--| Count from 1 to 5 | ✅ Pass | 0.791s | |
--| Math Calculation | ✅ Pass | 0.712s | |
--| Basic Echo Function | ✅ Pass | 1.013s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.621s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.699s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.240s | |
++| Simple Math | ✅ Pass | 1.194s | |
++| Text Transform Uppercase | ✅ Pass | 0.984s | |
++| Count from 1 to 5 | ✅ Pass | 1.359s | |
++| Math Calculation | ✅ Pass | 1.392s | |
++| Basic Echo Function | ✅ Pass | 2.918s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.439s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.296s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 3.038s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.281s | |
--| Search Query Function | ✅ Pass | 1.034s | |
--| Ask Advice Function | ✅ Pass | 1.261s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.968s | |
--| Basic Context Memory Test | ✅ Pass | 0.962s | |
--| Function Argument Memory Test | ✅ Pass | 0.781s | |
--| Function Response Memory Test | ✅ Pass | 0.746s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.290s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.715s | |
--| Penetration Testing Methodology | ✅ Pass | 3.325s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.531s | |
--| SQL Injection Attack Type | ✅ Pass | 1.343s | |
--| Penetration Testing Framework | ✅ Pass | 2.634s | |
--| Web Application Security Scanner | ✅ Pass | 1.549s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.217s | |
++| JSON Response Function | ✅ Pass | 5.582s | |
++| Search Query Function | ✅ Pass | 5.275s | |
++| Ask Advice Function | ✅ Pass | 1.614s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.364s | |
++| Basic Context Memory Test | ✅ Pass | 1.956s | |
++| Function Argument Memory Test | ✅ Pass | 0.816s | |
++| Function Response Memory Test | ✅ Pass | 0.827s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.101s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.220s | |
++| Penetration Testing Methodology | ✅ Pass | 5.329s | |
++| SQL Injection Attack Type | ✅ Pass | 1.034s | |
++| Vulnerability Assessment Tools | ✅ Pass | 11.682s | |
++| Penetration Testing Framework | ✅ Pass | 6.363s | |
++| Web Application Security Scanner | ✅ Pass | 0.217s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.527s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.271s
++**Average latency**: 2.676s
---
--### coder (kimi-k2.6)
++### coder (kimi-k2.7-code)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.077s | |
--| Text Transform Uppercase | ✅ Pass | 1.523s | |
--| Count from 1 to 5 | ✅ Pass | 1.973s | |
--| Math Calculation | ✅ Pass | 1.502s | |
--| Basic Echo Function | ✅ Pass | 1.630s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.642s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.130s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.123s | |
++| Simple Math | ✅ Pass | 0.222s | |
++| Text Transform Uppercase | ✅ Pass | 0.210s | |
++| Count from 1 to 5 | ✅ Pass | 0.208s | |
++| Math Calculation | ✅ Pass | 0.214s | |
++| Basic Echo Function | ✅ Pass | 0.210s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.214s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.737s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.779s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.829s | |
--| Search Query Function | ✅ Pass | 1.558s | |
--| Ask Advice Function | ✅ Pass | 1.854s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.841s | |
--| Basic Context Memory Test | ✅ Pass | 2.435s | |
--| Function Argument Memory Test | ✅ Pass | 1.975s | |
--| Function Response Memory Test | ✅ Pass | 2.004s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.342s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.180s | |
--| Penetration Testing Methodology | ✅ Pass | 8.157s | |
--| SQL Injection Attack Type | ✅ Pass | 3.469s | |
--| Vulnerability Assessment Tools | ✅ Pass | 18.587s | |
--| Penetration Testing Framework | ✅ Pass | 3.799s | |
--| Web Application Security Scanner | ✅ Pass | 3.946s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.792s | |
++| JSON Response Function | ✅ Pass | 1.980s | |
++| Search Query Function | ✅ Pass | 1.959s | |
++| Ask Advice Function | ✅ Pass | 1.882s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.525s | |
++| Basic Context Memory Test | ✅ Pass | 1.567s | |
++| Function Argument Memory Test | ✅ Pass | 1.594s | |
++| Function Response Memory Test | ✅ Pass | 1.481s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.682s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.334s | |
++| Penetration Testing Methodology | ✅ Pass | 2.137s | |
++| Vulnerability Assessment Tools | ✅ Pass | 4.048s | |
++| SQL Injection Attack Type | ✅ Pass | 1.848s | |
++| Penetration Testing Framework | ✅ Pass | 3.963s | |
++| Web Application Security Scanner | ✅ Pass | 2.175s | |
++| Penetration Testing Tool Selection | ✅ Pass | 4.730s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 3.321s
++**Average latency**: 1.857s
---
@@@ -459,79 -459,79 +466,79 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.393s | |
--| Text Transform Uppercase | ✅ Pass | 2.517s | |
--| Count from 1 to 5 | ✅ Pass | 1.722s | |
--| Math Calculation | ✅ Pass | 1.790s | |
--| Basic Echo Function | ✅ Pass | 1.364s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.430s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.340s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.897s | |
++| Simple Math | ✅ Pass | 2.195s | |
++| Text Transform Uppercase | ✅ Pass | 1.955s | |
++| Count from 1 to 5 | ✅ Pass | 6.687s | |
++| Math Calculation | ✅ Pass | 3.855s | |
++| Basic Echo Function | ✅ Pass | 1.849s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.962s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.923s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.689s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.757s | |
--| Search Query Function | ✅ Pass | 1.746s | |
--| Ask Advice Function | ✅ Pass | 2.257s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.568s | |
--| Basic Context Memory Test | ✅ Pass | 2.824s | |
--| Function Argument Memory Test | ✅ Pass | 2.347s | |
--| Function Response Memory Test | ✅ Pass | 1.724s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.100s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.024s | |
--| Penetration Testing Methodology | ✅ Pass | 11.036s | |
--| Vulnerability Assessment Tools | ✅ Pass | 5.959s | |
--| SQL Injection Attack Type | ✅ Pass | 5.710s | |
--| Penetration Testing Framework | ✅ Pass | 10.269s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.241s | |
--| Web Application Security Scanner | ✅ Pass | 6.747s | |
++| JSON Response Function | ✅ Pass | 1.772s | |
++| Search Query Function | ✅ Pass | 1.840s | |
++| Ask Advice Function | ✅ Pass | 1.972s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.081s | |
++| Basic Context Memory Test | ✅ Pass | 3.618s | |
++| Function Argument Memory Test | ✅ Pass | 2.526s | |
++| Function Response Memory Test | ✅ Pass | 2.659s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.096s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.482s | |
++| Penetration Testing Methodology | ✅ Pass | 12.085s | |
++| SQL Injection Attack Type | ✅ Pass | 2.480s | |
++| Vulnerability Assessment Tools | ✅ Pass | 9.315s | |
++| Penetration Testing Framework | ✅ Pass | 9.074s | |
++| Web Application Security Scanner | ✅ Pass | 11.971s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.424s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 3.381s
++**Average latency**: 4.023s
---
--### pentester (kimi-k2.6)
++### pentester (kimi-k2.7-code)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 4.828s | |
--| Text Transform Uppercase | ✅ Pass | 1.994s | |
--| Count from 1 to 5 | ✅ Pass | 2.194s | |
--| Math Calculation | ✅ Pass | 1.846s | |
--| Basic Echo Function | ✅ Pass | 1.777s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.363s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.947s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.943s | |
++| Simple Math | ✅ Pass | 0.210s | |
++| Text Transform Uppercase | ✅ Pass | 0.208s | |
++| Count from 1 to 5 | ✅ Pass | 0.214s | |
++| Math Calculation | ✅ Pass | 0.215s | |
++| Basic Echo Function | ✅ Pass | 0.209s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.220s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.391s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.889s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.911s | |
--| Search Query Function | ✅ Pass | 2.339s | |
--| Ask Advice Function | ✅ Pass | 2.164s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.754s | |
--| Basic Context Memory Test | ✅ Pass | 2.243s | |
--| Function Argument Memory Test | ✅ Pass | 1.688s | |
--| Function Response Memory Test | ✅ Pass | 2.078s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.638s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.884s | |
--| Penetration Testing Methodology | ✅ Pass | 7.438s | |
--| SQL Injection Attack Type | ✅ Pass | 2.465s | |
--| Vulnerability Assessment Tools | ✅ Pass | 16.920s | |
--| Penetration Testing Framework | ✅ Pass | 6.907s | |
--| Web Application Security Scanner | ✅ Pass | 4.049s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.557s | |
++| JSON Response Function | ✅ Pass | 2.085s | |
++| Search Query Function | ✅ Pass | 2.094s | |
++| Ask Advice Function | ✅ Pass | 1.860s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.555s | |
++| Basic Context Memory Test | ✅ Pass | 1.725s | |
++| Function Argument Memory Test | ✅ Pass | 1.524s | |
++| Function Response Memory Test | ✅ Pass | 1.178s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.426s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.003s | |
++| Penetration Testing Methodology | ✅ Pass | 1.798s | |
++| Vulnerability Assessment Tools | ✅ Pass | 5.907s | |
++| SQL Injection Attack Type | ✅ Pass | 2.046s | |
++| Penetration Testing Framework | ✅ Pass | 2.358s | |
++| Web Application Security Scanner | ✅ Pass | 2.833s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.177s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 3.432s
++**Average latency**: 1.658s
---
diff --cc examples/tests/minimax-report.md
index 00000000,00000000..cfbb14c1
new file mode 100644
--- /dev/null
+++ b/examples/tests/minimax-report.md
@@@ -1,0 -1,0 +1,544 @@@
++# LLM Agent Testing Report
++
++Generated: Tue, 21 Jul 2026 19:11:05 UTC
++
++## Overall Results
++
++| Agent | Model | Reasoning | Success Rate | Average Latency |
++|-------|-------|-----------|--------------|-----------------|
++| simple | MiniMax-M2.7 | true | 23/23 (100.00%) | 3.034s |
++| simple_json | MiniMax-M2.7 | true | 6/7 (85.71%) | 2.505s |
++| primary_agent | MiniMax-M3 | true | 23/23 (100.00%) | 1.784s |
++| assistant | MiniMax-M3 | true | 22/23 (95.65%) | 1.863s |
++| generator | MiniMax-M3 | true | 22/23 (95.65%) | 1.774s |
++| refiner | MiniMax-M3 | true | 22/23 (95.65%) | 1.857s |
++| adviser | MiniMax-M3 | true | 22/23 (95.65%) | 1.618s |
++| reflector | MiniMax-M3 | true | 23/23 (100.00%) | 3.042s |
++| searcher | MiniMax-M3 | true | 23/23 (100.00%) | 2.713s |
++| enricher | MiniMax-M3 | true | 23/23 (100.00%) | 2.287s |
++| coder | MiniMax-M3 | true | 22/23 (95.65%) | 1.399s |
++| installer | MiniMax-M3 | true | 23/23 (100.00%) | 0.224s |
++| pentester | MiniMax-M3 | true | 23/23 (100.00%) | 0.222s |
++
++**Total**: 277/283 (97.88%) successful tests
++**Overall average latency**: 1.835s
++
++## Detailed Results
++
++### simple (MiniMax-M2.7)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 4.424s | |
++| Math Calculation | ✅ Pass | 1.493s | |
++| Count from 1 to 5 | ✅ Pass | 2.932s | |
++| Text Transform Uppercase | ✅ Pass | 5.049s | |
++| Basic Echo Function | ✅ Pass | 1.800s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.018s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.595s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.591s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 1.775s | |
++| Search Query Function | ✅ Pass | 1.895s | |
++| Ask Advice Function | ✅ Pass | 2.342s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.684s | |
++| Basic Context Memory Test | ✅ Pass | 2.271s | |
++| Function Argument Memory Test | ✅ Pass | 2.541s | |
++| Function Response Memory Test | ✅ Pass | 2.198s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.435s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.963s | |
++| Penetration Testing Methodology | ✅ Pass | 4.513s | |
++| Vulnerability Assessment Tools | ✅ Pass | 6.186s | |
++| SQL Injection Attack Type | ✅ Pass | 4.290s | |
++| Penetration Testing Framework | ✅ Pass | 6.556s | |
++| Web Application Security Scanner | ✅ Pass | 2.842s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.374s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 3.034s
++
++---
++
++### simple_json (MiniMax-M2.7)
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Project Information JSON | ✅ Pass | 2.120s | |
++| Person Information JSON | ✅ Pass | 2.365s | |
++| User Profile JSON | ✅ Pass | 2.581s | |
++| Vulnerability Report Memory Test | ✅ Pass | 2.818s | |
++| Streaming Person Information JSON Streaming | ❌ Fail | 2.079s | invalid JSON response: invalid character '<' after top\-level value |
++| JSON Array Response Without Schema | ✅ Pass | 2.834s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Structured Output With JSON Schema | structured_output | ✅ Pass | 2.732s | |
++
++**Summary**: 6/7 (85.71%) successful tests
++
++**Average latency**: 2.505s
++
++---
++
++### primary_agent (MiniMax-M3)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 2.002s | |
++| Text Transform Uppercase | ✅ Pass | 0.225s | |
++| Count from 1 to 5 | ✅ Pass | 0.207s | |
++| Math Calculation | ✅ Pass | 0.211s | |
++| Basic Echo Function | ✅ Pass | 0.211s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.205s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.265s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.278s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.211s | |
++| Search Query Function | ✅ Pass | 0.202s | |
++| Ask Advice Function | ✅ Pass | 0.208s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.257s | |
++| Basic Context Memory Test | ✅ Pass | 0.211s | |
++| Function Argument Memory Test | ✅ Pass | 1.470s | |
++| Function Response Memory Test | ✅ Pass | 1.604s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.260s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.735s | |
++| Penetration Testing Methodology | ✅ Pass | 4.491s | |
++| Vulnerability Assessment Tools | ✅ Pass | 9.840s | |
++| SQL Injection Attack Type | ✅ Pass | 3.323s | |
++| Penetration Testing Framework | ✅ Pass | 7.383s | |
++| Web Application Security Scanner | ✅ Pass | 1.496s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.715s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 1.784s
++
++---
++
++### assistant (MiniMax-M3)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 2.002s | |
++| Text Transform Uppercase | ✅ Pass | 0.208s | |
++| Count from 1 to 5 | ✅ Pass | 0.207s | |
++| Math Calculation | ✅ Pass | 0.211s | |
++| Basic Echo Function | ✅ Pass | 0.207s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.207s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.261s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.273s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.217s | |
++| Search Query Function | ✅ Pass | 0.207s | |
++| Ask Advice Function | ✅ Pass | 0.207s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.257s | |
++| Basic Context Memory Test | ✅ Pass | 0.211s | |
++| Function Argument Memory Test | ✅ Pass | 1.350s | |
++| Function Response Memory Test | ✅ Pass | 1.193s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.322s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.172s | |
++| Penetration Testing Methodology | ✅ Pass | 5.842s | |
++| Vulnerability Assessment Tools | ✅ Pass | 5.280s | |
++| SQL Injection Attack Type | ❌ Fail | 4.207s | expected text 'injection' not found |
++| Penetration Testing Framework | ✅ Pass | 7.334s | |
++| Web Application Security Scanner | ✅ Pass | 4.036s | |
++| Penetration Testing Tool Selection | ✅ Pass | 4.427s | |
++
++**Summary**: 22/23 (95.65%) successful tests
++
++**Average latency**: 1.863s
++
++---
++
++### generator (MiniMax-M3)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 2.022s | |
++| Text Transform Uppercase | ✅ Pass | 0.215s | |
++| Count from 1 to 5 | ✅ Pass | 0.209s | |
++| Math Calculation | ✅ Pass | 0.214s | |
++| Basic Echo Function | ✅ Pass | 0.208s | |
++| Streaming Simple Math Streaming | ❌ Fail | 0.207s | expected text '6' not found |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.262s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.261s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.213s | |
++| Search Query Function | ✅ Pass | 0.204s | |
++| Ask Advice Function | ✅ Pass | 0.209s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.256s | |
++| Basic Context Memory Test | ✅ Pass | 0.213s | |
++| Function Argument Memory Test | ✅ Pass | 1.599s | |
++| Function Response Memory Test | ✅ Pass | 1.137s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.304s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.192s | |
++| Penetration Testing Methodology | ✅ Pass | 4.910s | |
++| Vulnerability Assessment Tools | ✅ Pass | 9.593s | |
++| SQL Injection Attack Type | ✅ Pass | 2.922s | |
++| Penetration Testing Framework | ✅ Pass | 5.884s | |
++| Web Application Security Scanner | ✅ Pass | 2.006s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.560s | |
++
++**Summary**: 22/23 (95.65%) successful tests
++
++**Average latency**: 1.774s
++
++---
++
++### refiner (MiniMax-M3)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 2.000s | |
++| Text Transform Uppercase | ✅ Pass | 0.207s | |
++| Count from 1 to 5 | ✅ Pass | 0.208s | |
++| Math Calculation | ✅ Pass | 0.211s | |
++| Basic Echo Function | ✅ Pass | 0.206s | |
++| Streaming Simple Math Streaming | ❌ Fail | 0.208s | expected text '6' not found |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.261s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.260s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.211s | |
++| Search Query Function | ✅ Pass | 0.207s | |
++| Ask Advice Function | ✅ Pass | 0.208s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.258s | |
++| Basic Context Memory Test | ✅ Pass | 0.207s | |
++| Function Argument Memory Test | ✅ Pass | 1.150s | |
++| Function Response Memory Test | ✅ Pass | 1.505s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.291s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.101s | |
++| Penetration Testing Methodology | ✅ Pass | 4.023s | |
++| Vulnerability Assessment Tools | ✅ Pass | 8.803s | |
++| SQL Injection Attack Type | ✅ Pass | 2.038s | |
++| Penetration Testing Framework | ✅ Pass | 6.667s | |
++| Web Application Security Scanner | ✅ Pass | 3.783s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.675s | |
++
++**Summary**: 22/23 (95.65%) successful tests
++
++**Average latency**: 1.857s
++
++---
++
++### adviser (MiniMax-M3)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 2.002s | |
++| Text Transform Uppercase | ✅ Pass | 0.211s | |
++| Count from 1 to 5 | ✅ Pass | 0.209s | |
++| Math Calculation | ✅ Pass | 0.218s | |
++| Basic Echo Function | ✅ Pass | 0.217s | |
++| Streaming Simple Math Streaming | ❌ Fail | 0.207s | expected text '6' not found |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.261s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.262s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.210s | |
++| Search Query Function | ✅ Pass | 0.207s | |
++| Ask Advice Function | ✅ Pass | 0.206s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.258s | |
++| Basic Context Memory Test | ✅ Pass | 0.204s | |
++| Function Argument Memory Test | ✅ Pass | 1.427s | |
++| Function Response Memory Test | ✅ Pass | 0.980s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.853s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.013s | |
++| Penetration Testing Methodology | ✅ Pass | 2.843s | |
++| Vulnerability Assessment Tools | ✅ Pass | 8.423s | |
++| SQL Injection Attack Type | ✅ Pass | 1.870s | |
++| Penetration Testing Framework | ✅ Pass | 7.770s | |
++| Web Application Security Scanner | ✅ Pass | 1.755s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.600s | |
++
++**Summary**: 22/23 (95.65%) successful tests
++
++**Average latency**: 1.618s
++
++---
++
++### reflector (MiniMax-M3)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 3.207s | |
++| Text Transform Uppercase | ✅ Pass | 1.806s | |
++| Count from 1 to 5 | ✅ Pass | 1.716s | |
++| Math Calculation | ✅ Pass | 1.479s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.874s | |
++| Basic Echo Function | ✅ Pass | 2.816s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.577s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.428s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 1.800s | |
++| Ask Advice Function | ✅ Pass | 1.774s | |
++| Search Query Function | ✅ Pass | 3.854s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.568s | |
++| Basic Context Memory Test | ✅ Pass | 1.478s | |
++| Function Argument Memory Test | ✅ Pass | 1.394s | |
++| Function Response Memory Test | ✅ Pass | 2.723s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.058s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.080s | |
++| Penetration Testing Methodology | ✅ Pass | 4.896s | |
++| Vulnerability Assessment Tools | ✅ Pass | 8.835s | |
++| SQL Injection Attack Type | ✅ Pass | 2.803s | |
++| Penetration Testing Framework | ✅ Pass | 7.024s | |
++| Web Application Security Scanner | ✅ Pass | 2.798s | |
++| Penetration Testing Tool Selection | ✅ Pass | 4.965s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 3.042s
++
++---
++
++### searcher (MiniMax-M3)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 3.522s | |
++| Text Transform Uppercase | ✅ Pass | 1.797s | |
++| Count from 1 to 5 | ✅ Pass | 1.528s | |
++| Math Calculation | ✅ Pass | 1.399s | |
++| Basic Echo Function | ✅ Pass | 2.405s | |
++| Streaming Simple Math Streaming | ✅ Pass | 3.506s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.981s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.416s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 1.665s | |
++| Ask Advice Function | ✅ Pass | 1.628s | |
++| Search Query Function | ✅ Pass | 3.944s | |
++| Basic Context Memory Test | ✅ Pass | 1.394s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.683s | |
++| Function Argument Memory Test | ✅ Pass | 1.794s | |
++| Function Response Memory Test | ✅ Pass | 1.223s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.969s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.941s | |
++| Penetration Testing Methodology | ✅ Pass | 3.698s | |
++| Vulnerability Assessment Tools | ✅ Pass | 6.506s | |
++| SQL Injection Attack Type | ✅ Pass | 2.416s | |
++| Penetration Testing Framework | ✅ Pass | 6.985s | |
++| Web Application Security Scanner | ✅ Pass | 2.666s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.328s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 2.713s
++
++---
++
++### enricher (MiniMax-M3)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 1.371s | |
++| Text Transform Uppercase | ✅ Pass | 2.029s | |
++| Count from 1 to 5 | ✅ Pass | 1.483s | |
++| Math Calculation | ✅ Pass | 1.719s | |
++| Basic Echo Function | ✅ Pass | 1.897s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.594s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.228s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.384s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 1.659s | |
++| Ask Advice Function | ✅ Pass | 1.517s | |
++| Search Query Function | ✅ Pass | 2.571s | |
++| Basic Context Memory Test | ✅ Pass | 1.365s | |
++| Function Argument Memory Test | ✅ Pass | 0.254s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 5.597s | |
++| Function Response Memory Test | ✅ Pass | 1.071s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.096s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.652s | |
++| Penetration Testing Methodology | ✅ Pass | 2.176s | |
++| Vulnerability Assessment Tools | ✅ Pass | 5.249s | |
++| SQL Injection Attack Type | ✅ Pass | 3.192s | |
++| Penetration Testing Framework | ✅ Pass | 3.008s | |
++| Web Application Security Scanner | ✅ Pass | 2.955s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.519s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 2.287s
++
++---
++
++### coder (MiniMax-M3)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.237s | |
++| Text Transform Uppercase | ✅ Pass | 0.219s | |
++| Count from 1 to 5 | ✅ Pass | 0.209s | |
++| Math Calculation | ✅ Pass | 0.213s | |
++| Basic Echo Function | ✅ Pass | 0.216s | |
++| Streaming Simple Math Streaming | ❌ Fail | 0.211s | expected text '6' not found |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.255s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.257s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.204s | |
++| Search Query Function | ✅ Pass | 0.208s | |
++| Ask Advice Function | ✅ Pass | 0.208s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.255s | |
++| Basic Context Memory Test | ✅ Pass | 0.214s | |
++| Function Argument Memory Test | ✅ Pass | 0.250s | |
++| Function Response Memory Test | ✅ Pass | 0.838s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.135s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.314s | |
++| Penetration Testing Methodology | ✅ Pass | 3.940s | |
++| SQL Injection Attack Type | ✅ Pass | 0.209s | |
++| Vulnerability Assessment Tools | ✅ Pass | 8.416s | |
++| Penetration Testing Framework | ✅ Pass | 3.656s | |
++| Web Application Security Scanner | ✅ Pass | 2.992s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.512s | |
++
++**Summary**: 22/23 (95.65%) successful tests
++
++**Average latency**: 1.399s
++
++---
++
++### installer (MiniMax-M3)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.238s | |
++| Text Transform Uppercase | ✅ Pass | 0.208s | |
++| Count from 1 to 5 | ✅ Pass | 0.204s | |
++| Math Calculation | ✅ Pass | 0.208s | |
++| Basic Echo Function | ✅ Pass | 0.222s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.212s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.260s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.267s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.213s | |
++| Search Query Function | ✅ Pass | 0.209s | |
++| Ask Advice Function | ✅ Pass | 0.219s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.260s | |
++| Basic Context Memory Test | ✅ Pass | 0.221s | |
++| Function Argument Memory Test | ✅ Pass | 0.244s | |
++| Function Response Memory Test | ✅ Pass | 0.203s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.216s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.201s | |
++| Penetration Testing Methodology | ✅ Pass | 0.207s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.290s | |
++| SQL Injection Attack Type | ✅ Pass | 0.214s | |
++| Penetration Testing Framework | ✅ Pass | 0.212s | |
++| Web Application Security Scanner | ✅ Pass | 0.212s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.209s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.224s
++
++---
++
++### pentester (MiniMax-M3)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.241s | |
++| Text Transform Uppercase | ✅ Pass | 0.209s | |
++| Count from 1 to 5 | ✅ Pass | 0.212s | |
++| Math Calculation | ✅ Pass | 0.214s | |
++| Basic Echo Function | ✅ Pass | 0.202s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.212s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.251s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.272s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.211s | |
++| Search Query Function | ✅ Pass | 0.218s | |
++| Ask Advice Function | ✅ Pass | 0.212s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.262s | |
++| Basic Context Memory Test | ✅ Pass | 0.217s | |
++| Function Argument Memory Test | ✅ Pass | 0.201s | |
++| Function Response Memory Test | ✅ Pass | 0.257s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.211s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.205s | |
++| Penetration Testing Methodology | ✅ Pass | 0.208s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.215s | |
++| SQL Injection Attack Type | ✅ Pass | 0.212s | |
++| Penetration Testing Framework | ✅ Pass | 0.210s | |
++| Web Application Security Scanner | ✅ Pass | 0.223s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.226s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.222s
++
++---
++
diff --cc examples/tests/moonshot-report.md
index f8657530,f8657530..5e370561
--- a/examples/tests/moonshot-report.md
+++ b/examples/tests/moonshot-report.md
@@@ -1,86 -1,86 +1,93 @@@
# LLM Agent Testing Report
--Generated: Thu, 29 Jan 2026 17:23:17 UTC
++Generated: Tue, 21 Jul 2026 16:50:01 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
--| simple | kimi-k2-0905-preview | true | 23/23 (100.00%) | 2.527s |
--| simple_json | kimi-k2-0905-preview | false | 5/5 (100.00%) | 3.434s |
--| primary_agent | kimi-k2.5 | true | 23/23 (100.00%) | 4.712s |
--| assistant | kimi-k2.5 | true | 23/23 (100.00%) | 4.800s |
--| generator | kimi-k2.5 | true | 23/23 (100.00%) | 4.819s |
--| refiner | kimi-k2.5 | true | 23/23 (100.00%) | 5.105s |
--| adviser | kimi-k2.5 | true | 23/23 (100.00%) | 4.209s |
--| reflector | kimi-k2-0905-preview | true | 23/23 (100.00%) | 2.616s |
--| searcher | kimi-k2-0905-preview | true | 23/23 (100.00%) | 2.564s |
--| enricher | kimi-k2-0905-preview | true | 23/23 (100.00%) | 2.497s |
--| coder | kimi-k2.5 | true | 23/23 (100.00%) | 5.042s |
--| installer | kimi-k2-turbo-preview | true | 23/23 (100.00%) | 1.057s |
--| pentester | kimi-k2-turbo-preview | true | 23/23 (100.00%) | 1.050s |
--
--**Total**: 281/281 (100.00%) successful tests
--**Overall average latency**: 3.417s
++| simple | kimi-k2.5 | true | 23/23 (100.00%) | 0.229s |
++| simple_json | kimi-k2.5 | true | 7/7 (100.00%) | 0.209s |
++| primary_agent | kimi-k2.5 | true | 23/23 (100.00%) | 0.230s |
++| assistant | kimi-k2.5 | true | 23/23 (100.00%) | 0.230s |
++| generator | kimi-k2.6 | true | 23/23 (100.00%) | 3.664s |
++| refiner | kimi-k2.6 | true | 23/23 (100.00%) | 3.490s |
++| adviser | kimi-k2.6 | true | 23/23 (100.00%) | 3.168s |
++| reflector | kimi-k2.5 | true | 23/23 (100.00%) | 0.232s |
++| searcher | kimi-k2.5 | true | 23/23 (100.00%) | 0.231s |
++| enricher | kimi-k2.5 | true | 23/23 (100.00%) | 0.208s |
++| coder | kimi-k2.6 | true | 23/23 (100.00%) | 3.672s |
++| installer | kimi-k2.5 | true | 23/23 (100.00%) | 0.216s |
++| pentester | kimi-k2.6 | true | 23/23 (100.00%) | 3.397s |
++
++**Total**: 283/283 (100.00%) successful tests
++**Overall average latency**: 1.547s
## Detailed Results
--### simple (kimi-k2-0905-preview)
++### simple (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.969s | |
--| Text Transform Uppercase | ✅ Pass | 1.409s | |
--| Count from 1 to 5 | ✅ Pass | 2.185s | |
--| Math Calculation | ✅ Pass | 1.264s | |
--| Basic Echo Function | ✅ Pass | 3.142s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.245s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.640s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.451s | |
++| Simple Math | ✅ Pass | 0.564s | |
++| Text Transform Uppercase | ✅ Pass | 0.222s | |
++| Count from 1 to 5 | ✅ Pass | 0.208s | |
++| Math Calculation | ✅ Pass | 0.212s | |
++| Basic Echo Function | ✅ Pass | 0.214s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.207s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.219s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.214s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 4.030s | |
--| Search Query Function | ✅ Pass | 3.010s | |
--| Ask Advice Function | ✅ Pass | 4.312s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.796s | |
--| Basic Context Memory Test | ✅ Pass | 2.255s | |
--| Function Argument Memory Test | ✅ Pass | 1.492s | |
--| Function Response Memory Test | ✅ Pass | 1.159s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 8.136s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.327s | |
--| Penetration Testing Methodology | ✅ Pass | 1.652s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.114s | |
--| SQL Injection Attack Type | ✅ Pass | 2.276s | |
--| Penetration Testing Framework | ✅ Pass | 1.798s | |
--| Web Application Security Scanner | ✅ Pass | 1.280s | |
--| Penetration Testing Tool Selection | ✅ Pass | 4.168s | |
++| JSON Response Function | ✅ Pass | 0.208s | |
++| Search Query Function | ✅ Pass | 0.211s | |
++| Ask Advice Function | ✅ Pass | 0.211s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.212s | |
++| Basic Context Memory Test | ✅ Pass | 0.211s | |
++| Function Argument Memory Test | ✅ Pass | 0.210s | |
++| Function Response Memory Test | ✅ Pass | 0.208s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.213s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.219s | |
++| Penetration Testing Methodology | ✅ Pass | 0.207s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.217s | |
++| SQL Injection Attack Type | ✅ Pass | 0.216s | |
++| Penetration Testing Framework | ✅ Pass | 0.212s | |
++| Web Application Security Scanner | ✅ Pass | 0.212s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.221s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.527s
++**Average latency**: 0.229s
---
--### simple_json (kimi-k2-0905-preview)
++### simple_json (kimi-k2.5)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Person Information JSON | ✅ Pass | 2.695s | |
--| Project Information JSON | ✅ Pass | 2.609s | |
--| Streaming Person Information JSON Streaming | ✅ Pass | 2.682s | |
--| User Profile JSON | ✅ Pass | 3.029s | |
--| Vulnerability Report Memory Test | ✅ Pass | 6.151s | |
++| Vulnerability Report Memory Test | ✅ Pass | 0.204s | |
++| Person Information JSON | ✅ Pass | 0.216s | |
++| Project Information JSON | ✅ Pass | 0.211s | |
++| User Profile JSON | ✅ Pass | 0.207s | |
++| JSON Array Response Without Schema | ✅ Pass | 0.208s | |
++| Streaming Person Information JSON Streaming | ✅ Pass | 0.204s | |
--**Summary**: 5/5 (100.00%) successful tests
++#### Capability Tests
--**Average latency**: 3.434s
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Structured Output With JSON Schema | structured_output | ✅ Pass | 0.212s | |
++
++**Summary**: 7/7 (100.00%) successful tests
++
++**Average latency**: 0.209s
---
@@@ -90,38 -90,38 +97,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.156s | |
--| Text Transform Uppercase | ✅ Pass | 2.189s | |
--| Count from 1 to 5 | ✅ Pass | 4.424s | |
--| Math Calculation | ✅ Pass | 1.450s | |
--| Basic Echo Function | ✅ Pass | 1.899s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.228s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.028s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.610s | |
++| Simple Math | ✅ Pass | 0.581s | |
++| Text Transform Uppercase | ✅ Pass | 0.207s | |
++| Count from 1 to 5 | ✅ Pass | 0.208s | |
++| Math Calculation | ✅ Pass | 0.216s | |
++| Basic Echo Function | ✅ Pass | 0.207s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.210s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.223s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.215s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.485s | |
--| Search Query Function | ✅ Pass | 1.712s | |
--| Ask Advice Function | ✅ Pass | 2.803s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.520s | |
--| Basic Context Memory Test | ✅ Pass | 2.999s | |
--| Function Argument Memory Test | ✅ Pass | 2.119s | |
--| Function Response Memory Test | ✅ Pass | 2.689s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.852s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.575s | |
--| Penetration Testing Methodology | ✅ Pass | 15.432s | |
--| Vulnerability Assessment Tools | ✅ Pass | 15.437s | |
--| SQL Injection Attack Type | ✅ Pass | 6.549s | |
--| Penetration Testing Framework | ✅ Pass | 9.912s | |
--| Web Application Security Scanner | ✅ Pass | 13.061s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.227s | |
++| JSON Response Function | ✅ Pass | 0.217s | |
++| Search Query Function | ✅ Pass | 0.219s | |
++| Ask Advice Function | ✅ Pass | 0.213s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.210s | |
++| Basic Context Memory Test | ✅ Pass | 0.211s | |
++| Function Argument Memory Test | ✅ Pass | 0.214s | |
++| Function Response Memory Test | ✅ Pass | 0.208s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.210s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.209s | |
++| Penetration Testing Methodology | ✅ Pass | 0.209s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.220s | |
++| SQL Injection Attack Type | ✅ Pass | 0.217s | |
++| Penetration Testing Framework | ✅ Pass | 0.214s | |
++| Web Application Security Scanner | ✅ Pass | 0.216s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.221s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 4.712s
++**Average latency**: 0.230s
---
@@@ -131,407 -131,407 +138,407 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.533s | |
--| Text Transform Uppercase | ✅ Pass | 2.086s | |
--| Count from 1 to 5 | ✅ Pass | 4.770s | |
--| Math Calculation | ✅ Pass | 2.369s | |
--| Basic Echo Function | ✅ Pass | 2.024s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.364s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.988s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.295s | |
++| Simple Math | ✅ Pass | 0.576s | |
++| Text Transform Uppercase | ✅ Pass | 0.211s | |
++| Count from 1 to 5 | ✅ Pass | 0.213s | |
++| Math Calculation | ✅ Pass | 0.211s | |
++| Basic Echo Function | ✅ Pass | 0.213s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.208s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.232s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.215s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.751s | |
--| Search Query Function | ✅ Pass | 1.407s | |
--| Ask Advice Function | ✅ Pass | 2.156s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.947s | |
--| Basic Context Memory Test | ✅ Pass | 3.082s | |
--| Function Argument Memory Test | ✅ Pass | 2.787s | |
--| Function Response Memory Test | ✅ Pass | 2.670s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.721s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.700s | |
--| Penetration Testing Methodology | ✅ Pass | 15.716s | |
--| Vulnerability Assessment Tools | ✅ Pass | 17.163s | |
--| SQL Injection Attack Type | ✅ Pass | 4.081s | |
--| Penetration Testing Framework | ✅ Pass | 10.991s | |
--| Web Application Security Scanner | ✅ Pass | 14.581s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.207s | |
++| JSON Response Function | ✅ Pass | 0.216s | |
++| Search Query Function | ✅ Pass | 0.213s | |
++| Ask Advice Function | ✅ Pass | 0.226s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.214s | |
++| Basic Context Memory Test | ✅ Pass | 0.220s | |
++| Function Argument Memory Test | ✅ Pass | 0.211s | |
++| Function Response Memory Test | ✅ Pass | 0.221s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.214s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.204s | |
++| Penetration Testing Methodology | ✅ Pass | 0.207s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.218s | |
++| SQL Injection Attack Type | ✅ Pass | 0.212s | |
++| Penetration Testing Framework | ✅ Pass | 0.217s | |
++| Web Application Security Scanner | ✅ Pass | 0.207s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.210s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 4.800s
++**Average latency**: 0.230s
---
--### generator (kimi-k2.5)
++### generator (kimi-k2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.573s | |
--| Text Transform Uppercase | ✅ Pass | 3.412s | |
--| Count from 1 to 5 | ✅ Pass | 4.818s | |
--| Math Calculation | ✅ Pass | 1.616s | |
--| Basic Echo Function | ✅ Pass | 1.914s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.830s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.596s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.960s | |
++| Text Transform Uppercase | ✅ Pass | 2.428s | |
++| Simple Math | ✅ Pass | 3.484s | |
++| Count from 1 to 5 | ✅ Pass | 2.986s | |
++| Math Calculation | ✅ Pass | 1.980s | |
++| Basic Echo Function | ✅ Pass | 1.899s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.660s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.251s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.262s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.316s | |
--| Search Query Function | ✅ Pass | 1.623s | |
--| Ask Advice Function | ✅ Pass | 1.644s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.135s | |
--| Basic Context Memory Test | ✅ Pass | 3.843s | |
--| Function Argument Memory Test | ✅ Pass | 1.947s | |
--| Function Response Memory Test | ✅ Pass | 4.476s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.291s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.153s | |
--| Penetration Testing Methodology | ✅ Pass | 11.999s | |
--| SQL Injection Attack Type | ✅ Pass | 4.011s | |
--| Vulnerability Assessment Tools | ✅ Pass | 17.057s | |
--| Penetration Testing Framework | ✅ Pass | 12.720s | |
--| Web Application Security Scanner | ✅ Pass | 11.890s | |
--| Penetration Testing Tool Selection | ✅ Pass | 5.008s | |
++| JSON Response Function | ✅ Pass | 3.614s | |
++| Search Query Function | ✅ Pass | 2.205s | |
++| Ask Advice Function | ✅ Pass | 1.962s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.098s | |
++| Basic Context Memory Test | ✅ Pass | 3.120s | |
++| Function Argument Memory Test | ✅ Pass | 2.717s | |
++| Function Response Memory Test | ✅ Pass | 2.522s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.499s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.459s | |
++| Vulnerability Assessment Tools | ✅ Pass | 8.993s | |
++| Penetration Testing Methodology | ✅ Pass | 13.232s | |
++| SQL Injection Attack Type | ✅ Pass | 2.385s | |
++| Penetration Testing Framework | ✅ Pass | 4.996s | |
++| Web Application Security Scanner | ✅ Pass | 5.140s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.376s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 4.819s
++**Average latency**: 3.664s
---
--### refiner (kimi-k2.5)
++### refiner (kimi-k2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.575s | |
--| Text Transform Uppercase | ✅ Pass | 1.467s | |
--| Count from 1 to 5 | ✅ Pass | 2.710s | |
--| Math Calculation | ✅ Pass | 2.063s | |
--| Basic Echo Function | ✅ Pass | 2.177s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.719s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.366s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.311s | |
++| Text Transform Uppercase | ✅ Pass | 1.532s | |
++| Simple Math | ✅ Pass | 3.010s | |
++| Count from 1 to 5 | ✅ Pass | 1.915s | |
++| Math Calculation | ✅ Pass | 1.968s | |
++| Basic Echo Function | ✅ Pass | 2.093s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.647s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.763s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.509s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.040s | |
--| Search Query Function | ✅ Pass | 1.610s | |
--| Ask Advice Function | ✅ Pass | 2.109s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.672s | |
--| Basic Context Memory Test | ✅ Pass | 3.430s | |
--| Function Argument Memory Test | ✅ Pass | 2.247s | |
--| Function Response Memory Test | ✅ Pass | 3.155s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 6.300s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.164s | |
--| Penetration Testing Methodology | ✅ Pass | 11.814s | |
--| SQL Injection Attack Type | ✅ Pass | 4.782s | |
--| Vulnerability Assessment Tools | ✅ Pass | 19.726s | |
--| Penetration Testing Framework | ✅ Pass | 17.103s | |
--| Web Application Security Scanner | ✅ Pass | 14.709s | |
--| Penetration Testing Tool Selection | ✅ Pass | 4.152s | |
++| JSON Response Function | ✅ Pass | 2.231s | |
++| Search Query Function | ✅ Pass | 2.320s | |
++| Ask Advice Function | ✅ Pass | 1.975s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.176s | |
++| Basic Context Memory Test | ✅ Pass | 3.605s | |
++| Function Argument Memory Test | ✅ Pass | 2.709s | |
++| Function Response Memory Test | ✅ Pass | 2.090s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.044s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.128s | |
++| Penetration Testing Methodology | ✅ Pass | 11.961s | |
++| Vulnerability Assessment Tools | ✅ Pass | 12.566s | |
++| SQL Injection Attack Type | ✅ Pass | 3.213s | |
++| Penetration Testing Framework | ✅ Pass | 5.740s | |
++| Web Application Security Scanner | ✅ Pass | 4.530s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.544s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 5.105s
++**Average latency**: 3.490s
---
--### adviser (kimi-k2.5)
++### adviser (kimi-k2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.087s | |
--| Text Transform Uppercase | ✅ Pass | 2.282s | |
--| Count from 1 to 5 | ✅ Pass | 1.617s | |
--| Math Calculation | ✅ Pass | 2.105s | |
--| Basic Echo Function | ✅ Pass | 2.211s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.083s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.229s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.895s | |
++| Simple Math | ✅ Pass | 2.083s | |
++| Text Transform Uppercase | ✅ Pass | 2.149s | |
++| Count from 1 to 5 | ✅ Pass | 2.700s | |
++| Math Calculation | ✅ Pass | 2.092s | |
++| Basic Echo Function | ✅ Pass | 1.864s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.302s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.366s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 3.381s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.561s | |
--| Search Query Function | ✅ Pass | 1.690s | |
--| Ask Advice Function | ✅ Pass | 2.217s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.371s | |
--| Basic Context Memory Test | ✅ Pass | 2.929s | |
--| Function Argument Memory Test | ✅ Pass | 1.864s | |
--| Function Response Memory Test | ✅ Pass | 3.357s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.003s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.202s | |
--| Penetration Testing Methodology | ✅ Pass | 13.754s | |
--| SQL Injection Attack Type | ✅ Pass | 2.961s | |
--| Vulnerability Assessment Tools | ✅ Pass | 16.462s | |
--| Penetration Testing Framework | ✅ Pass | 6.220s | |
--| Web Application Security Scanner | ✅ Pass | 12.801s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.884s | |
++| JSON Response Function | ✅ Pass | 3.084s | |
++| Search Query Function | ✅ Pass | 1.596s | |
++| Ask Advice Function | ✅ Pass | 2.853s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.541s | |
++| Basic Context Memory Test | ✅ Pass | 2.088s | |
++| Function Argument Memory Test | ✅ Pass | 2.092s | |
++| Function Response Memory Test | ✅ Pass | 3.063s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.249s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.498s | |
++| Penetration Testing Methodology | ✅ Pass | 5.485s | |
++| Vulnerability Assessment Tools | ✅ Pass | 9.568s | |
++| SQL Injection Attack Type | ✅ Pass | 3.220s | |
++| Penetration Testing Framework | ✅ Pass | 4.217s | |
++| Web Application Security Scanner | ✅ Pass | 3.356s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.002s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 4.209s
++**Average latency**: 3.168s
---
--### reflector (kimi-k2-0905-preview)
++### reflector (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.968s | |
--| Text Transform Uppercase | ✅ Pass | 1.353s | |
--| Count from 1 to 5 | ✅ Pass | 2.131s | |
--| Math Calculation | ✅ Pass | 1.275s | |
--| Basic Echo Function | ✅ Pass | 2.887s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.255s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.752s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.544s | |
++| Simple Math | ✅ Pass | 0.544s | |
++| Text Transform Uppercase | ✅ Pass | 0.208s | |
++| Count from 1 to 5 | ✅ Pass | 0.219s | |
++| Math Calculation | ✅ Pass | 0.217s | |
++| Basic Echo Function | ✅ Pass | 0.205s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.217s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.217s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.239s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 4.087s | |
--| Search Query Function | ✅ Pass | 3.010s | |
--| Ask Advice Function | ✅ Pass | 4.280s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 3.324s | |
--| Basic Context Memory Test | ✅ Pass | 2.626s | |
--| Function Argument Memory Test | ✅ Pass | 1.436s | |
--| Function Response Memory Test | ✅ Pass | 1.267s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 7.401s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.327s | |
--| Penetration Testing Methodology | ✅ Pass | 2.447s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.007s | |
--| SQL Injection Attack Type | ✅ Pass | 2.523s | |
--| Penetration Testing Framework | ✅ Pass | 1.611s | |
--| Web Application Security Scanner | ✅ Pass | 2.019s | |
--| Penetration Testing Tool Selection | ✅ Pass | 4.628s | |
++| JSON Response Function | ✅ Pass | 0.216s | |
++| Search Query Function | ✅ Pass | 0.209s | |
++| Ask Advice Function | ✅ Pass | 0.207s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.227s | |
++| Basic Context Memory Test | ✅ Pass | 0.208s | |
++| Function Argument Memory Test | ✅ Pass | 0.211s | |
++| Function Response Memory Test | ✅ Pass | 0.212s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.272s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.210s | |
++| Penetration Testing Methodology | ✅ Pass | 0.212s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.209s | |
++| SQL Injection Attack Type | ✅ Pass | 0.214s | |
++| Penetration Testing Framework | ✅ Pass | 0.211s | |
++| Web Application Security Scanner | ✅ Pass | 0.213s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.217s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.616s
++**Average latency**: 0.232s
---
--### searcher (kimi-k2-0905-preview)
++### searcher (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.976s | |
--| Text Transform Uppercase | ✅ Pass | 1.436s | |
--| Count from 1 to 5 | ✅ Pass | 2.454s | |
--| Math Calculation | ✅ Pass | 1.724s | |
--| Basic Echo Function | ✅ Pass | 2.867s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.358s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.621s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.471s | |
++| Simple Math | ✅ Pass | 0.563s | |
++| Text Transform Uppercase | ✅ Pass | 0.212s | |
++| Count from 1 to 5 | ✅ Pass | 0.210s | |
++| Math Calculation | ✅ Pass | 0.214s | |
++| Basic Echo Function | ✅ Pass | 0.208s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.226s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.220s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.212s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.997s | |
--| Search Query Function | ✅ Pass | 2.949s | |
--| Ask Advice Function | ✅ Pass | 4.267s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.800s | |
--| Basic Context Memory Test | ✅ Pass | 2.723s | |
--| Function Argument Memory Test | ✅ Pass | 1.476s | |
--| Function Response Memory Test | ✅ Pass | 1.480s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 7.167s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.430s | |
--| Penetration Testing Methodology | ✅ Pass | 2.091s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.125s | |
--| SQL Injection Attack Type | ✅ Pass | 2.275s | |
--| Penetration Testing Framework | ✅ Pass | 1.584s | |
--| Web Application Security Scanner | ✅ Pass | 1.475s | |
--| Penetration Testing Tool Selection | ✅ Pass | 4.217s | |
++| JSON Response Function | ✅ Pass | 0.214s | |
++| Search Query Function | ✅ Pass | 0.225s | |
++| Ask Advice Function | ✅ Pass | 0.225s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.220s | |
++| Basic Context Memory Test | ✅ Pass | 0.207s | |
++| Function Argument Memory Test | ✅ Pass | 0.213s | |
++| Function Response Memory Test | ✅ Pass | 0.225s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.211s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.217s | |
++| Penetration Testing Methodology | ✅ Pass | 0.209s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.211s | |
++| SQL Injection Attack Type | ✅ Pass | 0.215s | |
++| Penetration Testing Framework | ✅ Pass | 0.213s | |
++| Web Application Security Scanner | ✅ Pass | 0.218s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.217s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.564s
++**Average latency**: 0.231s
---
--### enricher (kimi-k2-0905-preview)
++### enricher (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.380s | |
--| Text Transform Uppercase | ✅ Pass | 1.480s | |
--| Count from 1 to 5 | ✅ Pass | 2.090s | |
--| Math Calculation | ✅ Pass | 1.219s | |
--| Basic Echo Function | ✅ Pass | 2.964s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.157s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.274s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.458s | |
++| Simple Math | ✅ Pass | 0.070s | |
++| Text Transform Uppercase | ✅ Pass | 0.209s | |
++| Count from 1 to 5 | ✅ Pass | 0.216s | |
++| Math Calculation | ✅ Pass | 0.207s | |
++| Basic Echo Function | ✅ Pass | 0.212s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.216s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.213s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.212s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.441s | |
--| Search Query Function | ✅ Pass | 2.873s | |
--| Ask Advice Function | ✅ Pass | 4.240s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 3.152s | |
--| Basic Context Memory Test | ✅ Pass | 2.943s | |
--| Function Argument Memory Test | ✅ Pass | 1.419s | |
--| Function Response Memory Test | ✅ Pass | 1.301s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.490s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 7.617s | |
--| Penetration Testing Methodology | ✅ Pass | 2.430s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.518s | |
--| SQL Injection Attack Type | ✅ Pass | 1.380s | |
--| Penetration Testing Framework | ✅ Pass | 1.676s | |
--| Web Application Security Scanner | ✅ Pass | 1.834s | |
--| Penetration Testing Tool Selection | ✅ Pass | 4.091s | |
++| JSON Response Function | ✅ Pass | 0.219s | |
++| Search Query Function | ✅ Pass | 0.212s | |
++| Ask Advice Function | ✅ Pass | 0.216s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.218s | |
++| Basic Context Memory Test | ✅ Pass | 0.213s | |
++| Function Argument Memory Test | ✅ Pass | 0.217s | |
++| Function Response Memory Test | ✅ Pass | 0.215s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.210s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.219s | |
++| Penetration Testing Methodology | ✅ Pass | 0.212s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.212s | |
++| SQL Injection Attack Type | ✅ Pass | 0.215s | |
++| Penetration Testing Framework | ✅ Pass | 0.212s | |
++| Web Application Security Scanner | ✅ Pass | 0.221s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.214s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.497s
++**Average latency**: 0.208s
---
--### coder (kimi-k2.5)
++### coder (kimi-k2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.846s | |
--| Text Transform Uppercase | ✅ Pass | 1.852s | |
--| Count from 1 to 5 | ✅ Pass | 5.834s | |
--| Math Calculation | ✅ Pass | 2.142s | |
--| Basic Echo Function | ✅ Pass | 2.158s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.971s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.713s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.089s | |
++| Simple Math | ✅ Pass | 1.711s | |
++| Text Transform Uppercase | ✅ Pass | 2.812s | |
++| Count from 1 to 5 | ✅ Pass | 1.740s | |
++| Math Calculation | ✅ Pass | 2.105s | |
++| Basic Echo Function | ✅ Pass | 1.990s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.762s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.696s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.610s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.249s | |
--| Search Query Function | ✅ Pass | 2.192s | |
--| Ask Advice Function | ✅ Pass | 3.064s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.303s | |
--| Basic Context Memory Test | ✅ Pass | 3.178s | |
--| Function Argument Memory Test | ✅ Pass | 1.985s | |
--| Function Response Memory Test | ✅ Pass | 3.064s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.572s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.607s | |
--| Penetration Testing Methodology | ✅ Pass | 17.249s | |
--| SQL Injection Attack Type | ✅ Pass | 2.996s | |
--| Vulnerability Assessment Tools | ✅ Pass | 17.738s | |
--| Penetration Testing Framework | ✅ Pass | 14.365s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.291s | |
--| Web Application Security Scanner | ✅ Pass | 14.502s | |
++| JSON Response Function | ✅ Pass | 2.562s | |
++| Search Query Function | ✅ Pass | 1.632s | |
++| Ask Advice Function | ✅ Pass | 2.824s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.205s | |
++| Basic Context Memory Test | ✅ Pass | 2.604s | |
++| Function Argument Memory Test | ✅ Pass | 1.864s | |
++| Function Response Memory Test | ✅ Pass | 2.314s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.615s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.214s | |
++| Penetration Testing Methodology | ✅ Pass | 10.963s | |
++| SQL Injection Attack Type | ✅ Pass | 4.259s | |
++| Vulnerability Assessment Tools | ✅ Pass | 11.520s | |
++| Web Application Security Scanner | ✅ Pass | 3.507s | |
++| Penetration Testing Framework | ✅ Pass | 7.736s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.207s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 5.042s
++**Average latency**: 3.672s
---
--### installer (kimi-k2-turbo-preview)
++### installer (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.799s | |
--| Text Transform Uppercase | ✅ Pass | 0.831s | |
--| Count from 1 to 5 | ✅ Pass | 0.949s | |
--| Math Calculation | ✅ Pass | 0.844s | |
--| Basic Echo Function | ✅ Pass | 1.011s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.789s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.455s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.465s | |
++| Simple Math | ✅ Pass | 0.212s | |
++| Text Transform Uppercase | ✅ Pass | 0.218s | |
++| Count from 1 to 5 | ✅ Pass | 0.211s | |
++| Math Calculation | ✅ Pass | 0.217s | |
++| Basic Echo Function | ✅ Pass | 0.217s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.217s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.212s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.210s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.008s | |
--| Search Query Function | ✅ Pass | 1.117s | |
--| Ask Advice Function | ✅ Pass | 1.262s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.063s | |
--| Basic Context Memory Test | ✅ Pass | 0.915s | |
--| Function Argument Memory Test | ✅ Pass | 0.787s | |
--| Function Response Memory Test | ✅ Pass | 0.798s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.997s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.827s | |
--| Penetration Testing Methodology | ✅ Pass | 0.989s | |
--| Vulnerability Assessment Tools | ✅ Pass | 0.864s | |
--| SQL Injection Attack Type | ✅ Pass | 0.902s | |
--| Penetration Testing Framework | ✅ Pass | 1.020s | |
--| Web Application Security Scanner | ✅ Pass | 1.263s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.345s | |
++| JSON Response Function | ✅ Pass | 0.211s | |
++| Search Query Function | ✅ Pass | 0.225s | |
++| Ask Advice Function | ✅ Pass | 0.207s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.209s | |
++| Basic Context Memory Test | ✅ Pass | 0.214s | |
++| Function Argument Memory Test | ✅ Pass | 0.214s | |
++| Function Response Memory Test | ✅ Pass | 0.239s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.216s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.215s | |
++| Penetration Testing Methodology | ✅ Pass | 0.216s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.216s | |
++| SQL Injection Attack Type | ✅ Pass | 0.208s | |
++| Penetration Testing Framework | ✅ Pass | 0.220s | |
++| Web Application Security Scanner | ✅ Pass | 0.211s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.214s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.057s
++**Average latency**: 0.216s
---
--### pentester (kimi-k2-turbo-preview)
++### pentester (kimi-k2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.856s | |
--| Text Transform Uppercase | ✅ Pass | 0.816s | |
--| Count from 1 to 5 | ✅ Pass | 0.893s | |
--| Math Calculation | ✅ Pass | 0.829s | |
--| Basic Echo Function | ✅ Pass | 0.923s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.820s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.375s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.587s | |
++| Simple Math | ✅ Pass | 1.675s | |
++| Text Transform Uppercase | ✅ Pass | 2.616s | |
++| Count from 1 to 5 | ✅ Pass | 2.692s | |
++| Math Calculation | ✅ Pass | 2.204s | |
++| Basic Echo Function | ✅ Pass | 2.184s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.315s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.624s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.971s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.088s | |
--| Search Query Function | ✅ Pass | 0.976s | |
--| Ask Advice Function | ✅ Pass | 1.218s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.231s | |
--| Basic Context Memory Test | ✅ Pass | 0.975s | |
--| Function Argument Memory Test | ✅ Pass | 0.927s | |
--| Function Response Memory Test | ✅ Pass | 0.830s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.959s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.795s | |
--| Penetration Testing Methodology | ✅ Pass | 1.035s | |
--| Vulnerability Assessment Tools | ✅ Pass | 0.987s | |
--| SQL Injection Attack Type | ✅ Pass | 0.972s | |
--| Penetration Testing Framework | ✅ Pass | 1.222s | |
--| Web Application Security Scanner | ✅ Pass | 0.834s | |
--| Penetration Testing Tool Selection | ✅ Pass | 0.989s | |
++| JSON Response Function | ✅ Pass | 3.288s | |
++| Search Query Function | ✅ Pass | 1.558s | |
++| Ask Advice Function | ✅ Pass | 3.152s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.146s | |
++| Basic Context Memory Test | ✅ Pass | 2.631s | |
++| Function Argument Memory Test | ✅ Pass | 1.841s | |
++| Function Response Memory Test | ✅ Pass | 2.501s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 5.078s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.673s | |
++| Penetration Testing Methodology | ✅ Pass | 6.215s | |
++| Vulnerability Assessment Tools | ✅ Pass | 8.888s | |
++| Penetration Testing Framework | ✅ Pass | 4.000s | |
++| SQL Injection Attack Type | ✅ Pass | 7.689s | |
++| Web Application Security Scanner | ✅ Pass | 3.694s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.490s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.050s
++**Average latency**: 3.397s
---
diff --cc examples/tests/ollama-cloud-report.md
index 976a7086,976a7086..c0a2200e
--- a/examples/tests/ollama-cloud-report.md
+++ b/examples/tests/ollama-cloud-report.md
@@@ -1,332 -1,332 +1,339 @@@
# LLM Agent Testing Report
--Generated: Mon, 23 Mar 2026 16:29:35 UTC
++Generated: Tue, 21 Jul 2026 20:33:49 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
--| simple | nemotron-3-super:cloud | false | 22/23 (95.65%) | 22.627s |
--| simple_json | nemotron-3-super:cloud | false | 5/5 (100.00%) | 27.380s |
--| primary_agent | qwen3-coder-next:cloud | false | 23/23 (100.00%) | 4.101s |
--| assistant | nemotron-3-super:cloud | false | 23/23 (100.00%) | 22.773s |
--| generator | qwen3-coder-next:cloud | false | 23/23 (100.00%) | 2.348s |
--| refiner | glm-5:cloud | false | 23/23 (100.00%) | 7.964s |
--| adviser | minimax-m2.7:cloud | false | 23/23 (100.00%) | 3.990s |
--| reflector | glm-5:cloud | false | 23/23 (100.00%) | 9.201s |
--| searcher | qwen3.5:397b-cloud | false | 20/23 (86.96%) | 46.174s |
--| enricher | minimax-m2.7:cloud | false | 23/23 (100.00%) | 6.153s |
--| coder | qwen3-coder-next:cloud | false | 23/23 (100.00%) | 3.223s |
--| installer | devstral-2:123b-cloud | false | 23/23 (100.00%) | 8.049s |
--| pentester | qwen3-coder-next:cloud | false | 23/23 (100.00%) | 2.227s |
--
--**Total**: 277/281 (98.58%) successful tests
--**Overall average latency**: 11.851s
++| simple | nemotron-3-nano:30b-cloud | false | 23/23 (100.00%) | 2.553s |
++| simple_json | nemotron-3-nano:30b-cloud | false | 7/7 (100.00%) | 2.410s |
++| primary_agent | kimi-k2.6:cloud | false | 23/23 (100.00%) | 2.718s |
++| assistant | kimi-k2.6:cloud | false | 23/23 (100.00%) | 2.847s |
++| generator | mistral-large-3:675b-cloud | false | 23/23 (100.00%) | 1.886s |
++| refiner | nemotron-3-ultra:cloud | false | 23/23 (100.00%) | 5.631s |
++| adviser | glm-5.1:cloud | false | 23/23 (100.00%) | 3.260s |
++| reflector | nemotron-3-nano:30b-cloud | false | 23/23 (100.00%) | 2.096s |
++| searcher | qwen3.5:397b-cloud | false | 23/23 (100.00%) | 5.856s |
++| enricher | minimax-m2.7:cloud | false | 23/23 (100.00%) | 3.425s |
++| coder | kimi-k2.7-code:cloud | false | 22/23 (95.65%) | 2.322s |
++| installer | kimi-k2.7-code:cloud | false | 23/23 (100.00%) | 2.550s |
++| pentester | deepseek-v4-pro:cloud | false | 23/23 (100.00%) | 1.824s |
++
++**Total**: 282/283 (99.65%) successful tests
++**Overall average latency**: 3.064s
## Detailed Results
--### simple (nemotron-3-super:cloud)
++### simple (nemotron-3-nano:30b-cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.444s | |
--| Text Transform Uppercase | ✅ Pass | 10.749s | |
--| Count from 1 to 5 | ✅ Pass | 19.076s | |
--| Math Calculation | ✅ Pass | 6.549s | |
--| Basic Echo Function | ✅ Pass | 16.980s | |
--| Streaming Simple Math Streaming | ✅ Pass | 18.192s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 8.274s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 10.723s | |
++| Simple Math | ✅ Pass | 1.189s | |
++| Count from 1 to 5 | ✅ Pass | 1.408s | |
++| Text Transform Uppercase | ✅ Pass | 9.093s | |
++| Math Calculation | ✅ Pass | 2.943s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.950s | |
++| Basic Echo Function | ✅ Pass | 6.563s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.039s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.664s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 20.851s | |
--| Search Query Function | ✅ Pass | 16.304s | |
--| Ask Advice Function | ✅ Pass | 33.178s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 14.129s | |
--| Basic Context Memory Test | ✅ Pass | 25.373s | |
--| Function Argument Memory Test | ✅ Pass | 34.212s | |
--| Function Response Memory Test | ✅ Pass | 22.402s | |
--| Penetration Testing Memory with Tool Call | ❌ Fail | 39.672s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 14.158s | |
--| Penetration Testing Methodology | ✅ Pass | 57.038s | |
--| Vulnerability Assessment Tools | ✅ Pass | 40.968s | |
--| SQL Injection Attack Type | ✅ Pass | 9.713s | |
--| Penetration Testing Framework | ✅ Pass | 65.247s | |
--| Web Application Security Scanner | ✅ Pass | 15.760s | |
--| Penetration Testing Tool Selection | ✅ Pass | 18.418s | |
++| JSON Response Function | ✅ Pass | 3.877s | |
++| Search Query Function | ✅ Pass | 2.516s | |
++| Ask Advice Function | ✅ Pass | 1.608s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 3.250s | |
++| Basic Context Memory Test | ✅ Pass | 2.386s | |
++| Function Argument Memory Test | ✅ Pass | 2.295s | |
++| Function Response Memory Test | ✅ Pass | 1.072s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.079s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.494s | |
++| Penetration Testing Methodology | ✅ Pass | 1.433s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.904s | |
++| SQL Injection Attack Type | ✅ Pass | 2.914s | |
++| Penetration Testing Framework | ✅ Pass | 2.560s | |
++| Web Application Security Scanner | ✅ Pass | 1.253s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.227s | |
--**Summary**: 22/23 (95.65%) successful tests
++**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 22.627s
++**Average latency**: 2.553s
---
--### simple_json (nemotron-3-super:cloud)
++### simple_json (nemotron-3-nano:30b-cloud)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Vulnerability Report Memory Test | ✅ Pass | 30.553s | |
--| Person Information JSON | ✅ Pass | 23.124s | |
--| Project Information JSON | ✅ Pass | 25.124s | |
--| User Profile JSON | ✅ Pass | 31.109s | |
--| Streaming Person Information JSON Streaming | ✅ Pass | 26.988s | |
++| Person Information JSON | ✅ Pass | 1.468s | |
++| Project Information JSON | ✅ Pass | 1.558s | |
++| User Profile JSON | ✅ Pass | 1.822s | |
++| Vulnerability Report Memory Test | ✅ Pass | 3.089s | |
++| Streaming Person Information JSON Streaming | ✅ Pass | 1.354s | |
++| JSON Array Response Without Schema | ✅ Pass | 5.023s | |
++
++#### Capability Tests
--**Summary**: 5/5 (100.00%) successful tests
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Structured Output With JSON Schema | structured_output | ✅ Pass | 2.552s | |
--**Average latency**: 27.380s
++**Summary**: 7/7 (100.00%) successful tests
++
++**Average latency**: 2.410s
---
--### primary_agent (qwen3-coder-next:cloud)
++### primary_agent (kimi-k2.6:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.931s | |
--| Text Transform Uppercase | ✅ Pass | 0.763s | |
--| Count from 1 to 5 | ✅ Pass | 3.641s | |
--| Math Calculation | ✅ Pass | 0.932s | |
--| Basic Echo Function | ✅ Pass | 0.869s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.700s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.713s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.812s | |
++| Simple Math | ✅ Pass | 1.671s | |
++| Text Transform Uppercase | ✅ Pass | 1.436s | |
++| Count from 1 to 5 | ✅ Pass | 2.245s | |
++| Math Calculation | ✅ Pass | 1.513s | |
++| Basic Echo Function | ✅ Pass | 1.420s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.155s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.994s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 3.485s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 0.959s | |
--| Search Query Function | ✅ Pass | 0.933s | |
--| Ask Advice Function | ✅ Pass | 33.999s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.998s | |
--| Basic Context Memory Test | ✅ Pass | 8.687s | |
--| Function Argument Memory Test | ✅ Pass | 6.186s | |
--| Function Response Memory Test | ✅ Pass | 2.615s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.527s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 4.732s | |
--| Penetration Testing Methodology | ✅ Pass | 6.404s | |
--| Vulnerability Assessment Tools | ✅ Pass | 3.203s | |
--| SQL Injection Attack Type | ✅ Pass | 2.766s | |
--| Penetration Testing Framework | ✅ Pass | 8.059s | |
--| Web Application Security Scanner | ✅ Pass | 1.546s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.339s | |
++| JSON Response Function | ✅ Pass | 2.924s | |
++| Search Query Function | ✅ Pass | 1.882s | |
++| Ask Advice Function | ✅ Pass | 1.658s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.927s | |
++| Basic Context Memory Test | ✅ Pass | 2.056s | |
++| Function Argument Memory Test | ✅ Pass | 3.303s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.341s | |
++| Function Response Memory Test | ✅ Pass | 6.369s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.381s | |
++| Penetration Testing Methodology | ✅ Pass | 3.671s | |
++| Vulnerability Assessment Tools | ✅ Pass | 5.404s | |
++| SQL Injection Attack Type | ✅ Pass | 2.150s | |
++| Penetration Testing Framework | ✅ Pass | 6.441s | |
++| Web Application Security Scanner | ✅ Pass | 2.062s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.018s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 4.101s
++**Average latency**: 2.718s
---
--### assistant (nemotron-3-super:cloud)
++### assistant (kimi-k2.6:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 13.460s | |
--| Text Transform Uppercase | ✅ Pass | 8.769s | |
--| Count from 1 to 5 | ✅ Pass | 6.944s | |
--| Math Calculation | ✅ Pass | 31.133s | |
--| Basic Echo Function | ✅ Pass | 10.710s | |
--| Streaming Simple Math Streaming | ✅ Pass | 16.396s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 7.617s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 13.783s | |
++| Simple Math | ✅ Pass | 2.077s | |
++| Text Transform Uppercase | ✅ Pass | 1.443s | |
++| Count from 1 to 5 | ✅ Pass | 1.803s | |
++| Math Calculation | ✅ Pass | 1.713s | |
++| Basic Echo Function | ✅ Pass | 1.231s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.716s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.366s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 3.403s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 20.992s | |
--| Search Query Function | ✅ Pass | 13.170s | |
--| Ask Advice Function | ✅ Pass | 17.042s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 18.533s | |
--| Basic Context Memory Test | ✅ Pass | 23.059s | |
--| Function Argument Memory Test | ✅ Pass | 25.397s | |
--| Function Response Memory Test | ✅ Pass | 24.149s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 39.970s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 8.683s | |
--| Penetration Testing Methodology | ✅ Pass | 63.505s | |
--| Vulnerability Assessment Tools | ✅ Pass | 34.659s | |
--| SQL Injection Attack Type | ✅ Pass | 26.478s | |
--| Penetration Testing Framework | ✅ Pass | 58.539s | |
--| Web Application Security Scanner | ✅ Pass | 33.683s | |
--| Penetration Testing Tool Selection | ✅ Pass | 7.095s | |
++| JSON Response Function | ✅ Pass | 2.741s | |
++| Search Query Function | ✅ Pass | 1.544s | |
++| Ask Advice Function | ✅ Pass | 1.553s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.748s | |
++| Basic Context Memory Test | ✅ Pass | 1.659s | |
++| Function Argument Memory Test | ✅ Pass | 3.101s | |
++| Function Response Memory Test | ✅ Pass | 1.532s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.992s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.628s | |
++| Penetration Testing Methodology | ✅ Pass | 4.080s | |
++| SQL Injection Attack Type | ✅ Pass | 1.767s | |
++| Vulnerability Assessment Tools | ✅ Pass | 15.714s | |
++| Penetration Testing Framework | ✅ Pass | 7.542s | |
++| Web Application Security Scanner | ✅ Pass | 2.199s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.912s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 22.773s
++**Average latency**: 2.847s
---
--### generator (qwen3-coder-next:cloud)
++### generator (mistral-large-3:675b-cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.633s | |
--| Text Transform Uppercase | ✅ Pass | 0.664s | |
--| Count from 1 to 5 | ✅ Pass | 1.159s | |
--| Math Calculation | ✅ Pass | 7.640s | |
--| Basic Echo Function | ✅ Pass | 1.056s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.638s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.870s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.083s | |
++| Simple Math | ✅ Pass | 1.057s | |
++| Text Transform Uppercase | ✅ Pass | 0.906s | |
++| Count from 1 to 5 | ✅ Pass | 1.016s | |
++| Math Calculation | ✅ Pass | 0.843s | |
++| Basic Echo Function | ✅ Pass | 1.010s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.334s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.931s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.129s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 0.946s | |
--| Search Query Function | ✅ Pass | 0.880s | |
--| Ask Advice Function | ✅ Pass | 1.013s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.946s | |
--| Basic Context Memory Test | ✅ Pass | 1.315s | |
--| Function Argument Memory Test | ✅ Pass | 0.722s | |
--| Function Response Memory Test | ✅ Pass | 3.881s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 6.597s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.642s | |
--| Penetration Testing Methodology | ✅ Pass | 9.073s | |
--| Vulnerability Assessment Tools | ✅ Pass | 3.598s | |
--| SQL Injection Attack Type | ✅ Pass | 5.544s | |
--| Penetration Testing Framework | ✅ Pass | 1.304s | |
--| Web Application Security Scanner | ✅ Pass | 1.261s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.537s | |
++| JSON Response Function | ✅ Pass | 1.149s | |
++| Search Query Function | ✅ Pass | 1.324s | |
++| Ask Advice Function | ✅ Pass | 1.236s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.226s | |
++| Basic Context Memory Test | ✅ Pass | 0.929s | |
++| Function Argument Memory Test | ✅ Pass | 0.803s | |
++| Function Response Memory Test | ✅ Pass | 0.882s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.688s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.908s | |
++| Penetration Testing Methodology | ✅ Pass | 1.975s | |
++| SQL Injection Attack Type | ✅ Pass | 0.980s | |
++| Vulnerability Assessment Tools | ✅ Pass | 15.025s | |
++| Penetration Testing Framework | ✅ Pass | 2.884s | |
++| Web Application Security Scanner | ✅ Pass | 2.011s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.118s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.348s
++**Average latency**: 1.886s
---
--### refiner (glm-5:cloud)
++### refiner (nemotron-3-ultra:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.032s | |
--| Text Transform Uppercase | ✅ Pass | 5.144s | |
--| Count from 1 to 5 | ✅ Pass | 1.324s | |
--| Math Calculation | ✅ Pass | 22.879s | |
--| Basic Echo Function | ✅ Pass | 2.930s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.628s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.727s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.291s | |
++| Simple Math | ✅ Pass | 1.812s | |
++| Count from 1 to 5 | ✅ Pass | 4.445s | |
++| Math Calculation | ✅ Pass | 2.275s | |
++| Text Transform Uppercase | ✅ Pass | 12.666s | |
++| Basic Echo Function | ✅ Pass | 3.633s | |
++| Streaming Simple Math Streaming | ✅ Pass | 7.361s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 5.497s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 10.178s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.312s | |
--| Search Query Function | ✅ Pass | 1.351s | |
--| Ask Advice Function | ✅ Pass | 1.539s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.405s | |
--| Basic Context Memory Test | ✅ Pass | 5.605s | |
--| Function Argument Memory Test | ✅ Pass | 2.879s | |
--| Function Response Memory Test | ✅ Pass | 1.468s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 15.864s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.101s | |
--| Penetration Testing Methodology | ✅ Pass | 16.016s | |
--| Vulnerability Assessment Tools | ✅ Pass | 24.194s | |
--| SQL Injection Attack Type | ✅ Pass | 7.143s | |
--| Penetration Testing Framework | ✅ Pass | 22.774s | |
--| Web Application Security Scanner | ✅ Pass | 34.765s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.788s | |
++| Search Query Function | ✅ Pass | 2.298s | |
++| Ask Advice Function | ✅ Pass | 4.959s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 4.867s | |
++| Basic Context Memory Test | ✅ Pass | 1.962s | |
++| Function Argument Memory Test | ✅ Pass | 1.149s | |
++| Function Response Memory Test | ✅ Pass | 1.863s | |
++| JSON Response Function | ✅ Pass | 31.697s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 7.205s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 6.164s | |
++| Penetration Testing Methodology | ✅ Pass | 4.731s | |
++| Vulnerability Assessment Tools | ✅ Pass | 4.192s | |
++| SQL Injection Attack Type | ✅ Pass | 1.133s | |
++| Penetration Testing Framework | ✅ Pass | 6.045s | |
++| Web Application Security Scanner | ✅ Pass | 1.541s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.820s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 7.964s
++**Average latency**: 5.631s
---
--### adviser (minimax-m2.7:cloud)
++### adviser (glm-5.1:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.791s | |
--| Text Transform Uppercase | ✅ Pass | 1.907s | |
--| Count from 1 to 5 | ✅ Pass | 5.236s | |
--| Math Calculation | ✅ Pass | 4.146s | |
--| Basic Echo Function | ✅ Pass | 1.855s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.809s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.696s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 7.184s | |
++| Simple Math | ✅ Pass | 1.888s | |
++| Text Transform Uppercase | ✅ Pass | 3.135s | |
++| Count from 1 to 5 | ✅ Pass | 2.208s | |
++| Math Calculation | ✅ Pass | 1.384s | |
++| Basic Echo Function | ✅ Pass | 2.010s | |
++| Streaming Simple Math Streaming | ✅ Pass | 3.185s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.989s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.622s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.898s | |
--| Search Query Function | ✅ Pass | 4.785s | |
--| Ask Advice Function | ✅ Pass | 1.973s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.405s | |
--| Basic Context Memory Test | ✅ Pass | 2.504s | |
--| Function Argument Memory Test | ✅ Pass | 2.930s | |
--| Function Response Memory Test | ✅ Pass | 2.279s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.874s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.765s | |
--| Penetration Testing Methodology | ✅ Pass | 3.184s | |
--| Vulnerability Assessment Tools | ✅ Pass | 14.047s | |
--| SQL Injection Attack Type | ✅ Pass | 5.317s | |
--| Penetration Testing Framework | ✅ Pass | 5.817s | |
--| Web Application Security Scanner | ✅ Pass | 4.591s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.773s | |
++| JSON Response Function | ✅ Pass | 1.383s | |
++| Search Query Function | ✅ Pass | 3.991s | |
++| Ask Advice Function | ✅ Pass | 1.618s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.838s | |
++| Basic Context Memory Test | ✅ Pass | 4.070s | |
++| Function Argument Memory Test | ✅ Pass | 2.376s | |
++| Function Response Memory Test | ✅ Pass | 3.116s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.410s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.902s | |
++| Penetration Testing Methodology | ✅ Pass | 4.680s | |
++| Vulnerability Assessment Tools | ✅ Pass | 7.330s | |
++| SQL Injection Attack Type | ✅ Pass | 2.247s | |
++| Penetration Testing Framework | ✅ Pass | 9.490s | |
++| Web Application Security Scanner | ✅ Pass | 4.576s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.529s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 3.990s
++**Average latency**: 3.260s
---
--### reflector (glm-5:cloud)
++### reflector (nemotron-3-nano:30b-cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 18.770s | |
--| Text Transform Uppercase | ✅ Pass | 3.024s | |
--| Count from 1 to 5 | ✅ Pass | 1.798s | |
--| Math Calculation | ✅ Pass | 2.176s | |
--| Basic Echo Function | ✅ Pass | 4.612s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.214s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 17.066s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.913s | |
++| Simple Math | ✅ Pass | 1.271s | |
++| Text Transform Uppercase | ✅ Pass | 1.096s | |
++| Count from 1 to 5 | ✅ Pass | 1.697s | |
++| Math Calculation | ✅ Pass | 1.010s | |
++| Basic Echo Function | ✅ Pass | 1.577s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.141s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.362s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.766s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.675s | |
--| Search Query Function | ✅ Pass | 1.432s | |
--| Ask Advice Function | ✅ Pass | 1.765s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 11.441s | |
--| Basic Context Memory Test | ✅ Pass | 5.525s | |
--| Function Argument Memory Test | ✅ Pass | 10.351s | |
--| Function Response Memory Test | ✅ Pass | 3.459s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 7.899s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.117s | |
--| Penetration Testing Methodology | ✅ Pass | 10.196s | |
--| Vulnerability Assessment Tools | ✅ Pass | 63.102s | |
--| SQL Injection Attack Type | ✅ Pass | 4.516s | |
--| Penetration Testing Framework | ✅ Pass | 17.855s | |
--| Web Application Security Scanner | ✅ Pass | 9.839s | |
--| Penetration Testing Tool Selection | ✅ Pass | 6.867s | |
++| JSON Response Function | ✅ Pass | 2.121s | |
++| Search Query Function | ✅ Pass | 1.656s | |
++| Ask Advice Function | ✅ Pass | 1.843s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.452s | |
++| Basic Context Memory Test | ✅ Pass | 2.101s | |
++| Function Argument Memory Test | ✅ Pass | 2.782s | |
++| Function Response Memory Test | ✅ Pass | 2.649s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 5.311s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.175s | |
++| Penetration Testing Methodology | ✅ Pass | 1.141s | |
++| Vulnerability Assessment Tools | ✅ Pass | 5.713s | |
++| SQL Injection Attack Type | ✅ Pass | 1.929s | |
++| Penetration Testing Framework | ✅ Pass | 1.143s | |
++| Web Application Security Scanner | ✅ Pass | 1.367s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.884s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 9.201s
++**Average latency**: 2.096s
---
@@@ -336,38 -336,38 +343,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.317s | |
--| Text Transform Uppercase | ✅ Pass | 1.459s | |
--| Count from 1 to 5 | ✅ Pass | 3.248s | |
--| Math Calculation | ✅ Pass | 1.474s | |
--| Basic Echo Function | ✅ Pass | 2.060s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.405s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.521s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.976s | |
++| Simple Math | ✅ Pass | 3.078s | |
++| Text Transform Uppercase | ✅ Pass | 5.277s | |
++| Count from 1 to 5 | ✅ Pass | 5.322s | |
++| Math Calculation | ✅ Pass | 2.751s | |
++| Basic Echo Function | ✅ Pass | 1.776s | |
++| Streaming Simple Math Streaming | ✅ Pass | 3.460s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.329s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.876s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.263s | |
--| Search Query Function | ✅ Pass | 0.955s | |
--| Ask Advice Function | ✅ Pass | 1.196s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.967s | |
--| Basic Context Memory Test | ✅ Pass | 2.587s | |
--| Function Argument Memory Test | ✅ Pass | 1.596s | |
--| Function Response Memory Test | ✅ Pass | 1.408s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.083s | |
--| Penetration Testing Methodology | ✅ Pass | 5.672s | |
--| Penetration Testing Memory with Tool Call | ❌ Fail | 600.006s | Post "https://ollama\.com/api/chat?ts=1774282315": net/http: request canceled \(Client\.Timeout exceeded while awaiting headers\) |
--| SQL Injection Attack Type | ✅ Pass | 4.083s | |
--| Vulnerability Assessment Tools | ❌ Fail | 391.303s | 500 Internal Server Error: Internal Server Error \(ref: 9d3370b7\-e2d2\-4086\-9f8b\-948f29fcd567\) |
--| Penetration Testing Framework | ❌ Fail | 30.619s | expected text 'exploitation' not found |
--| Web Application Security Scanner | ✅ Pass | 2.647s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.136s | |
--
--**Summary**: 20/23 (86.96%) successful tests
--
--**Average latency**: 46.174s
++| JSON Response Function | ✅ Pass | 2.141s | |
++| Search Query Function | ✅ Pass | 1.866s | |
++| Ask Advice Function | ✅ Pass | 2.291s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.823s | |
++| Basic Context Memory Test | ✅ Pass | 4.063s | |
++| Function Argument Memory Test | ✅ Pass | 1.898s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.962s | |
++| Function Response Memory Test | ✅ Pass | 11.805s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 5.704s | |
++| Penetration Testing Methodology | ✅ Pass | 10.296s | |
++| SQL Injection Attack Type | ✅ Pass | 8.448s | |
++| Penetration Testing Framework | ✅ Pass | 7.163s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.838s | |
++| Web Application Security Scanner | ✅ Pass | 8.118s | |
++| Vulnerability Assessment Tools | ✅ Pass | 34.393s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 5.856s
---
@@@ -377,161 -377,161 +384,161 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.381s | |
--| Text Transform Uppercase | ✅ Pass | 2.794s | |
--| Count from 1 to 5 | ✅ Pass | 4.367s | |
--| Math Calculation | ✅ Pass | 2.078s | |
--| Basic Echo Function | ✅ Pass | 1.977s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.884s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.745s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.099s | |
++| Simple Math | ✅ Pass | 2.714s | |
++| Text Transform Uppercase | ✅ Pass | 3.532s | |
++| Count from 1 to 5 | ✅ Pass | 5.140s | |
++| Math Calculation | ✅ Pass | 1.952s | |
++| Basic Echo Function | ✅ Pass | 1.707s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.469s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.203s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.477s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.883s | |
--| Search Query Function | ✅ Pass | 3.617s | |
--| Ask Advice Function | ✅ Pass | 1.873s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.564s | |
--| Basic Context Memory Test | ✅ Pass | 1.803s | |
--| Function Argument Memory Test | ✅ Pass | 2.661s | |
--| Function Response Memory Test | ✅ Pass | 2.227s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 5.442s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.429s | |
--| Penetration Testing Methodology | ✅ Pass | 5.753s | |
--| Vulnerability Assessment Tools | ✅ Pass | 70.743s | |
--| SQL Injection Attack Type | ✅ Pass | 4.696s | |
--| Penetration Testing Framework | ✅ Pass | 6.033s | |
--| Web Application Security Scanner | ✅ Pass | 3.398s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.064s | |
++| JSON Response Function | ✅ Pass | 2.636s | |
++| Search Query Function | ✅ Pass | 2.099s | |
++| Ask Advice Function | ✅ Pass | 2.682s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.241s | |
++| Basic Context Memory Test | ✅ Pass | 1.996s | |
++| Function Argument Memory Test | ✅ Pass | 2.087s | |
++| Function Response Memory Test | ✅ Pass | 2.030s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.810s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.592s | |
++| Penetration Testing Methodology | ✅ Pass | 4.070s | |
++| Vulnerability Assessment Tools | ✅ Pass | 10.512s | |
++| SQL Injection Attack Type | ✅ Pass | 6.386s | |
++| Penetration Testing Framework | ✅ Pass | 6.082s | |
++| Web Application Security Scanner | ✅ Pass | 3.166s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.176s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 6.153s
++**Average latency**: 3.425s
---
--### coder (qwen3-coder-next:cloud)
++### coder (kimi-k2.7-code:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.654s | |
--| Text Transform Uppercase | ✅ Pass | 12.005s | |
--| Count from 1 to 5 | ✅ Pass | 0.667s | |
--| Math Calculation | ✅ Pass | 0.633s | |
--| Basic Echo Function | ✅ Pass | 0.819s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.612s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.699s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.923s | |
++| Simple Math | ✅ Pass | 1.628s | |
++| Text Transform Uppercase | ✅ Pass | 1.266s | |
++| Count from 1 to 5 | ✅ Pass | 2.849s | |
++| Math Calculation | ✅ Pass | 4.151s | |
++| Basic Echo Function | ✅ Pass | 1.318s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.619s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.497s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.220s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 4.229s | |
--| Search Query Function | ✅ Pass | 0.873s | |
--| Ask Advice Function | ✅ Pass | 1.075s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.897s | |
--| Basic Context Memory Test | ✅ Pass | 1.072s | |
--| Function Argument Memory Test | ✅ Pass | 1.312s | |
--| Function Response Memory Test | ✅ Pass | 0.642s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.623s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.710s | |
--| Penetration Testing Methodology | ✅ Pass | 15.597s | |
--| Vulnerability Assessment Tools | ✅ Pass | 12.591s | |
--| SQL Injection Attack Type | ✅ Pass | 1.284s | |
--| Penetration Testing Framework | ✅ Pass | 2.334s | |
--| Web Application Security Scanner | ✅ Pass | 10.475s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.387s | |
++| JSON Response Function | ✅ Pass | 1.828s | |
++| Search Query Function | ✅ Pass | 1.327s | |
++| Ask Advice Function | ✅ Pass | 1.355s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.443s | |
++| Basic Context Memory Test | ✅ Pass | 1.541s | |
++| Function Argument Memory Test | ✅ Pass | 1.299s | |
++| Function Response Memory Test | ✅ Pass | 1.218s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 6.061s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 3.109s | |
++| Penetration Testing Methodology | ✅ Pass | 2.873s | |
++| Vulnerability Assessment Tools | ✅ Pass | 4.599s | |
++| SQL Injection Attack Type | ✅ Pass | 1.728s | |
++| Penetration Testing Framework | ✅ Pass | 2.782s | |
++| Web Application Security Scanner | ✅ Pass | 1.747s | |
++| Penetration Testing Tool Selection | ❌ Fail | 4.947s | expected function 'nmap' not found in tool calls: expected function nmap not found in tool calls |
--**Summary**: 23/23 (100.00%) successful tests
++**Summary**: 22/23 (95.65%) successful tests
--**Average latency**: 3.223s
++**Average latency**: 2.322s
---
--### installer (devstral-2:123b-cloud)
++### installer (kimi-k2.7-code:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.698s | |
--| Text Transform Uppercase | ✅ Pass | 0.738s | |
--| Count from 1 to 5 | ✅ Pass | 0.793s | |
--| Math Calculation | ✅ Pass | 0.709s | |
--| Basic Echo Function | ✅ Pass | 0.842s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.028s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.740s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 158.484s | |
++| Simple Math | ✅ Pass | 1.531s | |
++| Text Transform Uppercase | ✅ Pass | 1.436s | |
++| Count from 1 to 5 | ✅ Pass | 2.494s | |
++| Math Calculation | ✅ Pass | 3.434s | |
++| Basic Echo Function | ✅ Pass | 1.372s | |
++| Streaming Simple Math Streaming | ✅ Pass | 4.327s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.399s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.221s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 0.955s | |
--| Search Query Function | ✅ Pass | 0.851s | |
--| Ask Advice Function | ✅ Pass | 1.002s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.813s | |
--| Basic Context Memory Test | ✅ Pass | 0.851s | |
--| Function Argument Memory Test | ✅ Pass | 0.757s | |
--| Function Response Memory Test | ✅ Pass | 0.761s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.261s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.730s | |
--| Penetration Testing Methodology | ✅ Pass | 2.598s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.323s | |
--| SQL Injection Attack Type | ✅ Pass | 1.026s | |
--| Penetration Testing Framework | ✅ Pass | 4.639s | |
--| Web Application Security Scanner | ✅ Pass | 1.504s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.016s | |
++| JSON Response Function | ✅ Pass | 1.597s | |
++| Search Query Function | ✅ Pass | 1.715s | |
++| Ask Advice Function | ✅ Pass | 1.459s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.567s | |
++| Basic Context Memory Test | ✅ Pass | 1.722s | |
++| Function Argument Memory Test | ✅ Pass | 1.202s | |
++| Function Response Memory Test | ✅ Pass | 1.433s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 5.963s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.737s | |
++| Penetration Testing Methodology | ✅ Pass | 2.059s | |
++| Vulnerability Assessment Tools | ✅ Pass | 10.085s | |
++| SQL Injection Attack Type | ✅ Pass | 1.453s | |
++| Penetration Testing Framework | ✅ Pass | 2.092s | |
++| Web Application Security Scanner | ✅ Pass | 1.891s | |
++| Penetration Testing Tool Selection | ✅ Pass | 4.455s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 8.049s
++**Average latency**: 2.550s
---
--### pentester (qwen3-coder-next:cloud)
++### pentester (deepseek-v4-pro:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.654s | |
--| Text Transform Uppercase | ✅ Pass | 1.256s | |
--| Count from 1 to 5 | ✅ Pass | 0.716s | |
--| Math Calculation | ✅ Pass | 0.993s | |
--| Basic Echo Function | ✅ Pass | 0.798s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.657s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.658s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.956s | |
++| Simple Math | ✅ Pass | 1.062s | |
++| Text Transform Uppercase | ✅ Pass | 1.226s | |
++| Count from 1 to 5 | ✅ Pass | 1.536s | |
++| Math Calculation | ✅ Pass | 1.281s | |
++| Basic Echo Function | ✅ Pass | 1.337s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.349s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.193s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.361s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.019s | |
--| Search Query Function | ✅ Pass | 0.841s | |
--| Ask Advice Function | ✅ Pass | 8.525s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 7.950s | |
--| Basic Context Memory Test | ✅ Pass | 10.081s | |
--| Function Argument Memory Test | ✅ Pass | 0.815s | |
--| Function Response Memory Test | ✅ Pass | 0.675s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.659s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.804s | |
--| Penetration Testing Methodology | ✅ Pass | 1.960s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.314s | |
++| JSON Response Function | ✅ Pass | 1.357s | |
++| Search Query Function | ✅ Pass | 1.301s | |
++| Ask Advice Function | ✅ Pass | 1.337s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.356s | |
++| Basic Context Memory Test | ✅ Pass | 2.433s | |
++| Function Argument Memory Test | ✅ Pass | 1.422s | |
++| Function Response Memory Test | ✅ Pass | 1.598s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.345s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.396s | |
++| Penetration Testing Methodology | ✅ Pass | 6.055s | |
++| Vulnerability Assessment Tools | ✅ Pass | 4.079s | |
| SQL Injection Attack Type | ✅ Pass | 1.424s | |
--| Penetration Testing Framework | ✅ Pass | 1.673s | |
--| Web Application Security Scanner | ✅ Pass | 1.148s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.646s | |
++| Penetration Testing Framework | ✅ Pass | 2.163s | |
++| Web Application Security Scanner | ✅ Pass | 1.613s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.724s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 2.227s
++**Average latency**: 1.824s
---
diff --cc examples/tests/openai-report.md
index 64481b75,64481b75..77f5d797
--- a/examples/tests/openai-report.md
+++ b/examples/tests/openai-report.md
@@@ -1,537 -1,537 +1,598 @@@
# LLM Agent Testing Report
--Generated: Thu, 29 Jan 2026 17:38:42 UTC
++Generated: Tue, 21 Jul 2026 19:13:07 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
--| simple | gpt-4.1-mini | false | 23/23 (100.00%) | 0.995s |
--| simple_json | gpt-4.1-mini | false | 5/5 (100.00%) | 1.027s |
--| primary_agent | o4-mini | true | 23/23 (100.00%) | 2.302s |
--| assistant | o4-mini | true | 23/23 (100.00%) | 2.415s |
--| generator | o3 | true | 23/23 (100.00%) | 2.079s |
--| refiner | o3 | true | 23/23 (100.00%) | 3.682s |
--| adviser | gpt-5.2 | true | 23/23 (100.00%) | 1.193s |
--| reflector | o4-mini | true | 23/23 (100.00%) | 2.591s |
--| searcher | gpt-4.1-mini | false | 23/23 (100.00%) | 0.855s |
--| enricher | gpt-4.1-mini | false | 23/23 (100.00%) | 0.874s |
--| coder | o3 | true | 23/23 (100.00%) | 1.798s |
--| installer | o4-mini | true | 23/23 (100.00%) | 1.432s |
--| pentester | o4-mini | true | 23/23 (100.00%) | 1.506s |
--
--**Total**: 281/281 (100.00%) successful tests
--**Overall average latency**: 1.796s
++| simple | gpt-5.4-nano | false | 23/23 (100.00%) | 0.930s |
++| simple_json | gpt-5.4-nano | false | 7/7 (100.00%) | 1.392s |
++| primary_agent | gpt-5.4-mini | false | 23/24 (95.83%) | 0.911s |
++| assistant | gpt-5.4-mini | false | 24/24 (100.00%) | 0.937s |
++| generator | gpt-5.6-terra | false | 24/24 (100.00%) | 1.295s |
++| refiner | gpt-5.6-terra | false | 23/24 (95.83%) | 1.417s |
++| adviser | gpt-5.6-terra | false | 24/24 (100.00%) | 1.269s |
++| reflector | gpt-5.4-mini | false | 24/24 (100.00%) | 0.890s |
++| searcher | gpt-5.4-nano | false | 23/23 (100.00%) | 0.976s |
++| enricher | gpt-5.4-nano | false | 23/23 (100.00%) | 0.852s |
++| coder | gpt-5.6-terra | false | 23/24 (95.83%) | 1.773s |
++| installer | gpt-5.4-mini | false | 24/24 (100.00%) | 0.868s |
++| pentester | gpt-5.4-mini | false | 24/24 (100.00%) | 0.831s |
++
++**Total**: 289/292 (98.97%) successful tests
++**Overall average latency**: 1.088s
## Detailed Results
--### simple (gpt-4.1-mini)
++### simple (gpt-5.4-nano)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.368s | |
--| Text Transform Uppercase | ✅ Pass | 0.724s | |
--| Math Calculation | ✅ Pass | 0.571s | |
--| Count from 1 to 5 | ✅ Pass | 3.392s | |
--| Basic Echo Function | ✅ Pass | 0.888s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.704s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.664s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.938s | |
++| Simple Math | ✅ Pass | 1.141s | |
++| Text Transform Uppercase | ✅ Pass | 0.900s | |
++| Count from 1 to 5 | ✅ Pass | 0.838s | |
++| Math Calculation | ✅ Pass | 0.683s | |
++| Basic Echo Function | ✅ Pass | 0.817s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.798s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.873s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.876s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 0.968s | |
--| Search Query Function | ✅ Pass | 0.878s | |
--| Ask Advice Function | ✅ Pass | 1.225s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.808s | |
--| Basic Context Memory Test | ✅ Pass | 0.777s | |
--| Function Argument Memory Test | ✅ Pass | 0.666s | |
--| Function Response Memory Test | ✅ Pass | 0.620s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.191s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.649s | |
--| Penetration Testing Methodology | ✅ Pass | 0.931s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.566s | |
--| SQL Injection Attack Type | ✅ Pass | 0.782s | |
--| Penetration Testing Framework | ✅ Pass | 0.904s | |
--| Web Application Security Scanner | ✅ Pass | 0.751s | |
--| Penetration Testing Tool Selection | ✅ Pass | 0.919s | |
++| JSON Response Function | ✅ Pass | 1.186s | |
++| Search Query Function | ✅ Pass | 1.038s | |
++| Ask Advice Function | ✅ Pass | 1.009s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.858s | |
++| Basic Context Memory Test | ✅ Pass | 0.868s | |
++| Function Argument Memory Test | ✅ Pass | 0.789s | |
++| Function Response Memory Test | ✅ Pass | 0.701s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.342s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.826s | |
++| Penetration Testing Methodology | ✅ Pass | 0.823s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.043s | |
++| SQL Injection Attack Type | ✅ Pass | 0.815s | |
++| Penetration Testing Framework | ✅ Pass | 0.965s | |
++| Web Application Security Scanner | ✅ Pass | 0.852s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.341s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 0.995s
++**Average latency**: 0.930s
---
--### simple_json (gpt-4.1-mini)
++### simple_json (gpt-5.4-nano)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Person Information JSON | ✅ Pass | 0.926s | |
--| Project Information JSON | ✅ Pass | 0.859s | |
--| Vulnerability Report Memory Test | ✅ Pass | 1.562s | |
--| User Profile JSON | ✅ Pass | 0.883s | |
--| Streaming Person Information JSON Streaming | ✅ Pass | 0.901s | |
++| Person Information JSON | ✅ Pass | 0.999s | |
++| Project Information JSON | ✅ Pass | 0.980s | |
++| User Profile JSON | ✅ Pass | 0.929s | |
++| Streaming Person Information JSON Streaming | ✅ Pass | 0.788s | |
++| JSON Array Response Without Schema | ✅ Pass | 1.126s | |
++| Vulnerability Report Memory Test | ✅ Pass | 3.973s | |
--**Summary**: 5/5 (100.00%) successful tests
++#### Capability Tests
--**Average latency**: 1.027s
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Structured Output With JSON Schema | structured_output | ✅ Pass | 0.943s | |
++
++**Summary**: 7/7 (100.00%) successful tests
++
++**Average latency**: 1.392s
---
--### primary_agent (o4-mini)
++### primary_agent (gpt-5.4-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.376s | |
--| Text Transform Uppercase | ✅ Pass | 1.929s | |
--| Count from 1 to 5 | ✅ Pass | 1.718s | |
--| Math Calculation | ✅ Pass | 1.156s | |
--| Basic Echo Function | ✅ Pass | 2.535s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.765s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.355s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.773s | |
++| Simple Math | ✅ Pass | 0.900s | |
++| Text Transform Uppercase | ✅ Pass | 0.979s | |
++| Count from 1 to 5 | ✅ Pass | 0.844s | |
++| Math Calculation | ✅ Pass | 0.893s | |
++| Basic Echo Function | ✅ Pass | 0.920s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.667s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.614s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.751s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Search Query Function | ✅ Pass | 1.626s | |
--| JSON Response Function | ✅ Pass | 4.824s | |
--| Ask Advice Function | ✅ Pass | 2.918s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.077s | |
--| Basic Context Memory Test | ✅ Pass | 2.291s | |
--| Function Argument Memory Test | ✅ Pass | 1.976s | |
--| Function Response Memory Test | ✅ Pass | 1.534s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.414s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 2.189s | |
--| Penetration Testing Methodology | ✅ Pass | 2.109s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.882s | |
--| SQL Injection Attack Type | ✅ Pass | 3.378s | |
--| Penetration Testing Framework | ✅ Pass | 1.863s | |
--| Web Application Security Scanner | ✅ Pass | 2.422s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.821s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 2.302s
++| JSON Response Function | ✅ Pass | 1.107s | |
++| Search Query Function | ❌ Fail | 0.898s | expected function 'search' not found in tool calls: expected function search not found in tool calls |
++| Ask Advice Function | ✅ Pass | 0.906s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.765s | |
++| Basic Context Memory Test | ✅ Pass | 0.934s | |
++| Function Argument Memory Test | ✅ Pass | 0.780s | |
++| Function Response Memory Test | ✅ Pass | 0.725s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.403s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.747s | |
++| Penetration Testing Methodology | ✅ Pass | 0.974s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.261s | |
++| SQL Injection Attack Type | ✅ Pass | 0.767s | |
++| Penetration Testing Framework | ✅ Pass | 0.800s | |
++| Web Application Security Scanner | ✅ Pass | 1.355s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.910s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Explicit Reasoning Off Suppresses Reasoning | reasoning_off | ✅ Pass | 0.964s | |
++
++**Summary**: 23/24 (95.83%) successful tests
++
++**Average latency**: 0.911s
---
--### assistant (o4-mini)
++### assistant (gpt-5.4-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.010s | |
--| Text Transform Uppercase | ✅ Pass | 1.451s | |
--| Count from 1 to 5 | ✅ Pass | 1.825s | |
--| Math Calculation | ✅ Pass | 1.186s | |
--| Basic Echo Function | ✅ Pass | 3.803s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.108s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.409s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.680s | |
++| Simple Math | ✅ Pass | 1.253s | |
++| Text Transform Uppercase | ✅ Pass | 0.941s | |
++| Count from 1 to 5 | ✅ Pass | 0.832s | |
++| Math Calculation | ✅ Pass | 0.720s | |
++| Basic Echo Function | ✅ Pass | 0.956s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.779s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.728s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.280s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 5.114s | |
--| Search Query Function | ✅ Pass | 2.948s | |
--| Ask Advice Function | ✅ Pass | 1.913s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.845s | |
--| Basic Context Memory Test | ✅ Pass | 1.961s | |
--| Function Argument Memory Test | ✅ Pass | 1.367s | |
--| Function Response Memory Test | ✅ Pass | 1.961s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.599s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.941s | |
--| Penetration Testing Methodology | ✅ Pass | 2.459s | |
--| Vulnerability Assessment Tools | ✅ Pass | 4.370s | |
--| SQL Injection Attack Type | ✅ Pass | 3.904s | |
--| Penetration Testing Framework | ✅ Pass | 2.310s | |
--| Web Application Security Scanner | ✅ Pass | 2.158s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.206s | |
++| JSON Response Function | ✅ Pass | 0.921s | |
++| Search Query Function | ✅ Pass | 0.944s | |
++| Ask Advice Function | ✅ Pass | 0.927s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.737s | |
++| Basic Context Memory Test | ✅ Pass | 0.744s | |
++| Function Argument Memory Test | ✅ Pass | 0.681s | |
++| Function Response Memory Test | ✅ Pass | 0.988s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.584s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.807s | |
++| Penetration Testing Methodology | ✅ Pass | 0.921s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.526s | |
++| SQL Injection Attack Type | ✅ Pass | 0.696s | |
++| Penetration Testing Framework | ✅ Pass | 0.720s | |
++| Web Application Security Scanner | ✅ Pass | 0.834s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.017s | |
--**Summary**: 23/23 (100.00%) successful tests
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Explicit Reasoning Off Suppresses Reasoning | reasoning_off | ✅ Pass | 0.943s | |
--**Average latency**: 2.415s
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 0.937s
---
--### generator (o3)
++### generator (gpt-5.6-terra)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.007s | |
--| Text Transform Uppercase | ✅ Pass | 2.139s | |
--| Count from 1 to 5 | ✅ Pass | 1.782s | |
--| Math Calculation | ✅ Pass | 2.060s | |
--| Basic Echo Function | ✅ Pass | 2.894s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.271s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.244s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.827s | |
++| Simple Math | ✅ Pass | 1.199s | |
++| Text Transform Uppercase | ✅ Pass | 0.882s | |
++| Count from 1 to 5 | ✅ Pass | 0.969s | |
++| Math Calculation | ✅ Pass | 0.910s | |
++| Basic Echo Function | ✅ Pass | 1.023s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.890s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.885s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.995s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.864s | |
--| Search Query Function | ✅ Pass | 1.262s | |
--| Ask Advice Function | ✅ Pass | 1.421s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.493s | |
--| Basic Context Memory Test | ✅ Pass | 3.737s | |
--| Function Argument Memory Test | ✅ Pass | 1.326s | |
--| Function Response Memory Test | ✅ Pass | 1.881s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.361s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.761s | |
--| Penetration Testing Methodology | ✅ Pass | 2.348s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.881s | |
--| SQL Injection Attack Type | ✅ Pass | 2.790s | |
--| Penetration Testing Framework | ✅ Pass | 2.106s | |
--| Web Application Security Scanner | ✅ Pass | 1.683s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.678s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 2.079s
++| JSON Response Function | ✅ Pass | 0.945s | |
++| Search Query Function | ✅ Pass | 1.301s | |
++| Ask Advice Function | ✅ Pass | 0.987s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.093s | |
++| Basic Context Memory Test | ✅ Pass | 1.064s | |
++| Function Argument Memory Test | ✅ Pass | 0.911s | |
++| Function Response Memory Test | ✅ Pass | 0.971s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.185s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 6.472s | |
++| Penetration Testing Methodology | ✅ Pass | 1.001s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.438s | |
++| SQL Injection Attack Type | ✅ Pass | 1.052s | |
++| Penetration Testing Framework | ✅ Pass | 1.182s | |
++| Web Application Security Scanner | ✅ Pass | 1.532s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.104s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Explicit Reasoning Off Suppresses Reasoning | reasoning_off | ✅ Pass | 1.081s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 1.295s
---
--### refiner (o3)
++### refiner (gpt-5.6-terra)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.448s | |
--| Text Transform Uppercase | ✅ Pass | 2.546s | |
--| Count from 1 to 5 | ✅ Pass | 5.522s | |
--| Math Calculation | ✅ Pass | 3.212s | |
--| Basic Echo Function | ✅ Pass | 1.892s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.309s | |
--| Streaming Simple Math Streaming | ✅ Pass | 4.889s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.371s | |
++| Simple Math | ✅ Pass | 1.421s | |
++| Text Transform Uppercase | ✅ Pass | 0.934s | |
++| Count from 1 to 5 | ✅ Pass | 0.943s | |
++| Math Calculation | ✅ Pass | 0.939s | |
++| Basic Echo Function | ✅ Pass | 1.035s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.965s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.867s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.965s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.058s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.136s | |
--| Search Query Function | ✅ Pass | 10.011s | |
--| Basic Context Memory Test | ✅ Pass | 4.091s | |
--| Function Argument Memory Test | ✅ Pass | 1.994s | |
--| Ask Advice Function | ✅ Pass | 14.955s | |
--| Function Response Memory Test | ✅ Pass | 3.540s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.963s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.079s | |
--| Penetration Testing Methodology | ✅ Pass | 2.170s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.531s | |
--| SQL Injection Attack Type | ✅ Pass | 1.760s | |
--| Penetration Testing Framework | ✅ Pass | 1.550s | |
--| Web Application Security Scanner | ✅ Pass | 2.946s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.708s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 3.682s
++| JSON Response Function | ✅ Pass | 0.971s | |
++| Search Query Function | ✅ Pass | 1.151s | |
++| Ask Advice Function | ✅ Pass | 1.125s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.899s | |
++| Basic Context Memory Test | ✅ Pass | 0.941s | |
++| Function Argument Memory Test | ✅ Pass | 0.892s | |
++| Function Response Memory Test | ✅ Pass | 0.912s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.217s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 9.853s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
++| Penetration Testing Methodology | ✅ Pass | 1.114s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.322s | |
++| SQL Injection Attack Type | ✅ Pass | 1.305s | |
++| Penetration Testing Framework | ✅ Pass | 1.122s | |
++| Web Application Security Scanner | ✅ Pass | 1.158s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.023s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Explicit Reasoning Off Suppresses Reasoning | reasoning_off | ✅ Pass | 0.925s | |
++
++**Summary**: 23/24 (95.83%) successful tests
++
++**Average latency**: 1.417s
---
--### adviser (gpt-5.2)
++### adviser (gpt-5.6-terra)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.572s | |
--| Text Transform Uppercase | ✅ Pass | 0.817s | |
--| Count from 1 to 5 | ✅ Pass | 0.921s | |
--| Math Calculation | ✅ Pass | 0.793s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.662s | |
--| Basic Echo Function | ✅ Pass | 4.649s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.657s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.768s | |
++| Simple Math | ✅ Pass | 1.447s | |
++| Text Transform Uppercase | ✅ Pass | 1.037s | |
++| Count from 1 to 5 | ✅ Pass | 0.942s | |
++| Math Calculation | ✅ Pass | 1.212s | |
++| Basic Echo Function | ✅ Pass | 0.953s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.098s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.029s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.420s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.151s | |
--| Search Query Function | ✅ Pass | 0.899s | |
--| Ask Advice Function | ✅ Pass | 0.991s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.171s | |
--| Basic Context Memory Test | ✅ Pass | 0.786s | |
--| Function Argument Memory Test | ✅ Pass | 0.761s | |
--| Function Response Memory Test | ✅ Pass | 0.904s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.756s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.567s | |
--| Penetration Testing Methodology | ✅ Pass | 0.936s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.874s | |
--| SQL Injection Attack Type | ✅ Pass | 0.945s | |
--| Penetration Testing Framework | ✅ Pass | 0.835s | |
--| Web Application Security Scanner | ✅ Pass | 0.841s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.185s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 1.193s
++| JSON Response Function | ✅ Pass | 1.026s | |
++| Search Query Function | ✅ Pass | 1.116s | |
++| Ask Advice Function | ✅ Pass | 1.290s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.010s | |
++| Basic Context Memory Test | ✅ Pass | 1.302s | |
++| Function Argument Memory Test | ✅ Pass | 0.883s | |
++| Function Response Memory Test | ✅ Pass | 0.963s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.027s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.912s | |
++| Penetration Testing Methodology | ✅ Pass | 0.926s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.431s | |
++| SQL Injection Attack Type | ✅ Pass | 1.108s | |
++| Penetration Testing Framework | ✅ Pass | 0.944s | |
++| Web Application Security Scanner | ✅ Pass | 1.219s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.228s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Explicit Reasoning Off Suppresses Reasoning | reasoning_off | ✅ Pass | 0.924s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 1.269s
---
--### reflector (o4-mini)
++### reflector (gpt-5.4-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.239s | |
--| Text Transform Uppercase | ✅ Pass | 1.427s | |
--| Count from 1 to 5 | ✅ Pass | 1.759s | |
--| Math Calculation | ✅ Pass | 1.513s | |
--| Basic Echo Function | ✅ Pass | 1.532s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.474s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.386s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 8.476s | |
++| Simple Math | ✅ Pass | 1.105s | |
++| Text Transform Uppercase | ✅ Pass | 0.695s | |
++| Count from 1 to 5 | ✅ Pass | 1.044s | |
++| Math Calculation | ✅ Pass | 0.783s | |
++| Basic Echo Function | ✅ Pass | 0.697s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.638s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.739s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.787s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.630s | |
--| Search Query Function | ✅ Pass | 1.996s | |
--| Ask Advice Function | ✅ Pass | 2.403s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.610s | |
--| Basic Context Memory Test | ✅ Pass | 2.136s | |
--| Function Argument Memory Test | ✅ Pass | 2.317s | |
--| Function Response Memory Test | ✅ Pass | 1.938s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.983s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.886s | |
--| Penetration Testing Methodology | ✅ Pass | 2.069s | |
--| Vulnerability Assessment Tools | ✅ Pass | 3.294s | |
--| SQL Injection Attack Type | ✅ Pass | 1.435s | |
--| Web Application Security Scanner | ✅ Pass | 1.750s | |
--| Penetration Testing Framework | ✅ Pass | 5.447s | |
--| Penetration Testing Tool Selection | ✅ Pass | 7.874s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 2.591s
++| JSON Response Function | ✅ Pass | 0.831s | |
++| Search Query Function | ✅ Pass | 1.198s | |
++| Ask Advice Function | ✅ Pass | 0.877s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.726s | |
++| Basic Context Memory Test | ✅ Pass | 0.886s | |
++| Function Argument Memory Test | ✅ Pass | 0.985s | |
++| Function Response Memory Test | ✅ Pass | 0.755s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.656s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.691s | |
++| Penetration Testing Methodology | ✅ Pass | 1.362s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.279s | |
++| SQL Injection Attack Type | ✅ Pass | 0.652s | |
++| Penetration Testing Framework | ✅ Pass | 0.689s | |
++| Web Application Security Scanner | ✅ Pass | 0.644s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.786s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Explicit Reasoning Off Suppresses Reasoning | reasoning_off | ✅ Pass | 0.834s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 0.890s
---
--### searcher (gpt-4.1-mini)
++### searcher (gpt-5.4-nano)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.365s | |
--| Text Transform Uppercase | ✅ Pass | 0.696s | |
--| Count from 1 to 5 | ✅ Pass | 0.633s | |
--| Math Calculation | ✅ Pass | 0.560s | |
--| Basic Echo Function | ✅ Pass | 0.908s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.632s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.704s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.772s | |
++| Simple Math | ✅ Pass | 0.938s | |
++| Text Transform Uppercase | ✅ Pass | 0.772s | |
++| Count from 1 to 5 | ✅ Pass | 0.748s | |
++| Math Calculation | ✅ Pass | 0.717s | |
++| Basic Echo Function | ✅ Pass | 0.720s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.718s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.776s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.761s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 0.944s | |
--| Search Query Function | ✅ Pass | 0.715s | |
--| Ask Advice Function | ✅ Pass | 0.996s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.768s | |
--| Basic Context Memory Test | ✅ Pass | 0.698s | |
--| Function Argument Memory Test | ✅ Pass | 0.701s | |
--| Function Response Memory Test | ✅ Pass | 0.602s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.197s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.595s | |
--| Penetration Testing Methodology | ✅ Pass | 1.064s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.512s | |
--| SQL Injection Attack Type | ✅ Pass | 0.747s | |
--| Penetration Testing Framework | ✅ Pass | 1.084s | |
--| Web Application Security Scanner | ✅ Pass | 0.797s | |
--| Penetration Testing Tool Selection | ✅ Pass | 0.973s | |
++| JSON Response Function | ✅ Pass | 0.791s | |
++| Search Query Function | ✅ Pass | 0.815s | |
++| Ask Advice Function | ✅ Pass | 0.814s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.757s | |
++| Basic Context Memory Test | ✅ Pass | 0.710s | |
++| Function Argument Memory Test | ✅ Pass | 0.746s | |
++| Function Response Memory Test | ✅ Pass | 0.782s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.452s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 4.369s | |
++| Penetration Testing Methodology | ✅ Pass | 0.724s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.027s | |
++| SQL Injection Attack Type | ✅ Pass | 0.661s | |
++| Penetration Testing Framework | ✅ Pass | 1.033s | |
++| Web Application Security Scanner | ✅ Pass | 0.622s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.977s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 0.855s
++**Average latency**: 0.976s
---
--### enricher (gpt-4.1-mini)
++### enricher (gpt-5.4-nano)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.673s | |
--| Text Transform Uppercase | ✅ Pass | 0.599s | |
--| Count from 1 to 5 | ✅ Pass | 0.696s | |
--| Math Calculation | ✅ Pass | 0.686s | |
--| Basic Echo Function | ✅ Pass | 0.969s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.575s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.699s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.797s | |
++| Simple Math | ✅ Pass | 0.755s | |
++| Text Transform Uppercase | ✅ Pass | 0.673s | |
++| Count from 1 to 5 | ✅ Pass | 0.695s | |
++| Math Calculation | ✅ Pass | 0.650s | |
++| Basic Echo Function | ✅ Pass | 0.756s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.757s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.672s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.039s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 0.863s | |
--| Search Query Function | ✅ Pass | 2.113s | |
--| Ask Advice Function | ✅ Pass | 0.978s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.747s | |
++| JSON Response Function | ✅ Pass | 0.877s | |
++| Search Query Function | ✅ Pass | 0.777s | |
++| Ask Advice Function | ✅ Pass | 0.851s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.770s | |
| Basic Context Memory Test | ✅ Pass | 0.677s | |
--| Function Argument Memory Test | ✅ Pass | 0.922s | |
--| Function Response Memory Test | ✅ Pass | 0.611s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.221s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.708s | |
--| Penetration Testing Methodology | ✅ Pass | 0.905s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.199s | |
--| SQL Injection Attack Type | ✅ Pass | 0.759s | |
--| Penetration Testing Framework | ✅ Pass | 0.752s | |
--| Web Application Security Scanner | ✅ Pass | 0.834s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.106s | |
++| Function Argument Memory Test | ✅ Pass | 0.801s | |
++| Function Response Memory Test | ✅ Pass | 0.595s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.261s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.744s | |
++| Penetration Testing Methodology | ✅ Pass | 1.387s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.043s | |
++| SQL Injection Attack Type | ✅ Pass | 0.956s | |
++| Penetration Testing Framework | ✅ Pass | 0.707s | |
++| Web Application Security Scanner | ✅ Pass | 0.915s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.226s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 0.874s
++**Average latency**: 0.852s
---
--### coder (o3)
++### coder (gpt-5.6-terra)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.846s | |
--| Text Transform Uppercase | ✅ Pass | 1.455s | |
--| Count from 1 to 5 | ✅ Pass | 1.774s | |
--| Math Calculation | ✅ Pass | 1.376s | |
--| Basic Echo Function | ✅ Pass | 1.224s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.248s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.365s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.026s | |
++| Simple Math | ✅ Pass | 1.088s | |
++| Text Transform Uppercase | ✅ Pass | 1.223s | |
++| Count from 1 to 5 | ✅ Pass | 0.936s | |
++| Math Calculation | ✅ Pass | 0.914s | |
++| Basic Echo Function | ✅ Pass | 1.066s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.611s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.955s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.973s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.378s | |
--| Search Query Function | ✅ Pass | 2.455s | |
--| Ask Advice Function | ✅ Pass | 1.263s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.306s | |
--| Basic Context Memory Test | ✅ Pass | 2.486s | |
--| Function Argument Memory Test | ✅ Pass | 1.768s | |
--| Function Response Memory Test | ✅ Pass | 2.899s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.468s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.163s | |
--| Penetration Testing Methodology | ✅ Pass | 1.939s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.276s | |
--| SQL Injection Attack Type | ✅ Pass | 3.775s | |
--| Penetration Testing Framework | ✅ Pass | 1.902s | |
--| Web Application Security Scanner | ✅ Pass | 1.195s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.757s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 1.798s
++| JSON Response Function | ✅ Pass | 1.198s | |
++| Search Query Function | ✅ Pass | 2.118s | |
++| Ask Advice Function | ✅ Pass | 1.242s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.004s | |
++| Basic Context Memory Test | ✅ Pass | 0.970s | |
++| Function Argument Memory Test | ✅ Pass | 0.883s | |
++| Function Response Memory Test | ✅ Pass | 1.617s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.098s | |
++| Penetration Testing Methodology | ✅ Pass | 0.947s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.328s | |
++| SQL Injection Attack Type | ✅ Pass | 1.130s | |
++| Penetration Testing Framework | ✅ Pass | 1.465s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 15.827s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
++| Web Application Security Scanner | ✅ Pass | 1.003s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.004s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Explicit Reasoning Off Suppresses Reasoning | reasoning_off | ✅ Pass | 0.942s | |
++
++**Summary**: 23/24 (95.83%) successful tests
++
++**Average latency**: 1.773s
---
--### installer (o4-mini)
++### installer (gpt-5.4-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.259s | |
--| Text Transform Uppercase | ✅ Pass | 1.103s | |
--| Count from 1 to 5 | ✅ Pass | 1.479s | |
--| Math Calculation | ✅ Pass | 1.098s | |
--| Basic Echo Function | ✅ Pass | 1.458s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.221s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.150s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.093s | |
++| Simple Math | ✅ Pass | 0.642s | |
++| Text Transform Uppercase | ✅ Pass | 0.663s | |
++| Count from 1 to 5 | ✅ Pass | 0.743s | |
++| Math Calculation | ✅ Pass | 0.787s | |
++| Basic Echo Function | ✅ Pass | 0.738s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.743s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.826s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.739s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.167s | |
--| Search Query Function | ✅ Pass | 1.190s | |
--| Ask Advice Function | ✅ Pass | 1.344s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.060s | |
--| Basic Context Memory Test | ✅ Pass | 1.780s | |
--| Function Argument Memory Test | ✅ Pass | 1.371s | |
--| Function Response Memory Test | ✅ Pass | 1.473s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.580s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.675s | |
--| Penetration Testing Methodology | ✅ Pass | 1.638s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.012s | |
--| SQL Injection Attack Type | ✅ Pass | 1.645s | |
--| Penetration Testing Framework | ✅ Pass | 1.624s | |
--| Web Application Security Scanner | ✅ Pass | 1.865s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.639s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 1.432s
++| JSON Response Function | ✅ Pass | 0.756s | |
++| Search Query Function | ✅ Pass | 0.761s | |
++| Ask Advice Function | ✅ Pass | 0.847s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.762s | |
++| Basic Context Memory Test | ✅ Pass | 0.784s | |
++| Function Argument Memory Test | ✅ Pass | 0.736s | |
++| Function Response Memory Test | ✅ Pass | 1.352s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.424s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.946s | |
++| Penetration Testing Methodology | ✅ Pass | 1.208s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.230s | |
++| SQL Injection Attack Type | ✅ Pass | 0.822s | |
++| Penetration Testing Framework | ✅ Pass | 0.802s | |
++| Web Application Security Scanner | ✅ Pass | 0.697s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.896s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Explicit Reasoning Off Suppresses Reasoning | reasoning_off | ✅ Pass | 0.925s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 0.868s
---
--### pentester (o4-mini)
++### pentester (gpt-5.4-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.229s | |
--| Text Transform Uppercase | ✅ Pass | 1.321s | |
--| Count from 1 to 5 | ✅ Pass | 1.642s | |
--| Math Calculation | ✅ Pass | 1.335s | |
--| Basic Echo Function | ✅ Pass | 1.047s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.165s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.275s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.970s | |
++| Simple Math | ✅ Pass | 0.883s | |
++| Text Transform Uppercase | ✅ Pass | 0.729s | |
++| Count from 1 to 5 | ✅ Pass | 0.683s | |
++| Math Calculation | ✅ Pass | 0.698s | |
++| Basic Echo Function | ✅ Pass | 0.742s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.736s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.654s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.700s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.298s | |
--| Search Query Function | ✅ Pass | 1.158s | |
--| Ask Advice Function | ✅ Pass | 1.220s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.040s | |
--| Basic Context Memory Test | ✅ Pass | 1.583s | |
--| Function Argument Memory Test | ✅ Pass | 1.313s | |
--| Function Response Memory Test | ✅ Pass | 1.448s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.786s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.637s | |
--| Penetration Testing Methodology | ✅ Pass | 1.640s | |
--| Vulnerability Assessment Tools | ✅ Pass | 4.050s | |
--| SQL Injection Attack Type | ✅ Pass | 1.678s | |
--| Penetration Testing Framework | ✅ Pass | 1.841s | |
--| Web Application Security Scanner | ✅ Pass | 1.649s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.307s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 1.506s
++| JSON Response Function | ✅ Pass | 0.776s | |
++| Search Query Function | ✅ Pass | 0.764s | |
++| Ask Advice Function | ✅ Pass | 0.734s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.748s | |
++| Basic Context Memory Test | ✅ Pass | 0.760s | |
++| Function Argument Memory Test | ✅ Pass | 0.787s | |
++| Function Response Memory Test | ✅ Pass | 1.284s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.299s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.965s | |
++| Penetration Testing Methodology | ✅ Pass | 0.695s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.438s | |
++| SQL Injection Attack Type | ✅ Pass | 0.675s | |
++| Penetration Testing Framework | ✅ Pass | 0.888s | |
++| Web Application Security Scanner | ✅ Pass | 0.685s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.884s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Explicit Reasoning Off Suppresses Reasoning | reasoning_off | ✅ Pass | 0.727s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 0.831s
---
diff --cc examples/tests/openrouter-report.md
index e8dc7d62,e8dc7d62..52f564df
--- a/examples/tests/openrouter-report.md
+++ b/examples/tests/openrouter-report.md
@@@ -1,537 -1,537 +1,568 @@@
# LLM Agent Testing Report
--Generated: Tue, 30 Sep 2025 18:46:00 UTC
++Generated: Tue, 21 Jul 2026 19:32:12 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
--| simple | openai/gpt-4.1-mini | false | 23/23 (100.00%) | 1.594s |
--| simple_json | openai/gpt-4.1-mini | false | 5/5 (100.00%) | 1.682s |
--| primary_agent | openai/gpt-5 | true | 23/23 (100.00%) | 7.285s |
--| assistant | openai/gpt-5 | true | 23/23 (100.00%) | 8.135s |
--| generator | anthropic/claude-sonnet-4.5 | true | 23/23 (100.00%) | 4.525s |
--| refiner | google/gemini-2.5-pro | true | 21/23 (91.30%) | 5.576s |
--| adviser | google/gemini-2.5-pro | true | 22/23 (95.65%) | 5.532s |
--| reflector | openai/gpt-4.1-mini | false | 23/23 (100.00%) | 1.556s |
--| searcher | x-ai/grok-3-mini | true | 22/23 (95.65%) | 4.511s |
--| enricher | openai/gpt-4.1-mini | true | 23/23 (100.00%) | 1.597s |
--| coder | anthropic/claude-sonnet-4.5 | true | 23/23 (100.00%) | 4.445s |
--| installer | google/gemini-2.5-flash | true | 23/23 (100.00%) | 3.276s |
--| pentester | moonshotai/kimi-k2-0905 | true | 22/23 (95.65%) | 2.301s |
--
--**Total**: 276/281 (98.22%) successful tests
--**Overall average latency**: 4.150s
++| simple | deepseek/deepseek-v4-flash | false | 24/24 (100.00%) | 0.209s |
++| simple_json | deepseek/deepseek-v4-flash | false | 7/7 (100.00%) | 0.222s |
++| primary_agent | z-ai/glm-5-turbo | true | 23/23 (100.00%) | 0.212s |
++| assistant | z-ai/glm-5-turbo | true | 23/23 (100.00%) | 0.204s |
++| generator | z-ai/glm-5.2 | true | 23/23 (100.00%) | 0.212s |
++| refiner | z-ai/glm-5.2 | true | 23/23 (100.00%) | 0.210s |
++| adviser | minimax/minimax-m3 | true | 23/23 (100.00%) | 0.210s |
++| reflector | deepseek/deepseek-v4-flash | false | 24/24 (100.00%) | 0.212s |
++| searcher | deepseek/deepseek-v4-flash | false | 24/24 (100.00%) | 0.214s |
++| enricher | minimax/minimax-m3 | true | 24/24 (100.00%) | 0.208s |
++| coder | moonshotai/kimi-k2.7-code | true | 23/23 (100.00%) | 0.213s |
++| installer | moonshotai/kimi-k2.7-code | true | 23/23 (100.00%) | 0.212s |
++| pentester | deepseek/deepseek-v4-flash | true | 23/23 (100.00%) | 0.208s |
++
++**Total**: 287/287 (100.00%) successful tests
++**Overall average latency**: 0.211s
## Detailed Results
--### simple (openai/gpt-4.1-mini)
++### simple (deepseek/deepseek-v4-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Text Transform Uppercase | ✅ Pass | 2.727s | |
--| Simple Math | ✅ Pass | 2.809s | |
--| Count from 1 to 5 | ✅ Pass | 3.158s | |
--| Math Calculation | ✅ Pass | 1.255s | |
--| Basic Echo Function | ✅ Pass | 1.112s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.109s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.179s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.270s | |
++| Simple Math | ✅ Pass | 0.228s | |
++| Text Transform Uppercase | ✅ Pass | 0.210s | |
++| Count from 1 to 5 | ✅ Pass | 0.213s | |
++| Math Calculation | ✅ Pass | 0.213s | |
++| Basic Echo Function | ✅ Pass | 0.203s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.214s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.228s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.200s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.334s | |
--| Search Query Function | ✅ Pass | 1.375s | |
--| Ask Advice Function | ✅ Pass | 1.433s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.436s | |
--| Basic Context Memory Test | ✅ Pass | 1.293s | |
--| Function Argument Memory Test | ✅ Pass | 1.326s | |
--| Function Response Memory Test | ✅ Pass | 1.378s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.802s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.454s | |
--| Penetration Testing Methodology | ✅ Pass | 1.216s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.509s | |
--| SQL Injection Attack Type | ✅ Pass | 2.427s | |
--| Penetration Testing Framework | ✅ Pass | 1.526s | |
--| Web Application Security Scanner | ✅ Pass | 1.093s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.419s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 1.594s
++| JSON Response Function | ✅ Pass | 0.214s | |
++| Search Query Function | ✅ Pass | 0.202s | |
++| Ask Advice Function | ✅ Pass | 0.202s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.206s | |
++| Basic Context Memory Test | ✅ Pass | 0.216s | |
++| Function Argument Memory Test | ✅ Pass | 0.206s | |
++| Function Response Memory Test | ✅ Pass | 0.203s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.208s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.205s | |
++| Penetration Testing Methodology | ✅ Pass | 0.217s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.215s | |
++| SQL Injection Attack Type | ✅ Pass | 0.205s | |
++| Penetration Testing Framework | ✅ Pass | 0.209s | |
++| Web Application Security Scanner | ✅ Pass | 0.198s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.201s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Explicit Reasoning Off Suppresses Reasoning | reasoning_off | ✅ Pass | 0.199s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 0.209s
---
--### simple_json (openai/gpt-4.1-mini)
++### simple_json (deepseek/deepseek-v4-flash)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Project Information JSON | ✅ Pass | 1.574s | |
--| User Profile JSON | ✅ Pass | 1.531s | |
--| Person Information JSON | ✅ Pass | 1.706s | |
--| Vulnerability Report Memory Test | ✅ Pass | 2.108s | |
--| Streaming Person Information JSON Streaming | ✅ Pass | 1.488s | |
++| Vulnerability Report Memory Test | ✅ Pass | 0.204s | |
++| Person Information JSON | ✅ Pass | 0.206s | |
++| Project Information JSON | ✅ Pass | 0.223s | |
++| User Profile JSON | ✅ Pass | 0.234s | |
++| JSON Array Response Without Schema | ✅ Pass | 0.223s | |
++| Streaming Person Information JSON Streaming | ✅ Pass | 0.236s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Structured Output With JSON Schema | structured_output | ✅ Pass | 0.227s | |
--**Summary**: 5/5 (100.00%) successful tests
++**Summary**: 7/7 (100.00%) successful tests
--**Average latency**: 1.682s
++**Average latency**: 0.222s
---
--### primary_agent (openai/gpt-5)
++### primary_agent (z-ai/glm-5-turbo)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Text Transform Uppercase | ✅ Pass | 5.678s | |
--| Simple Math | ✅ Pass | 6.979s | |
--| Math Calculation | ✅ Pass | 4.546s | |
--| Count from 1 to 5 | ✅ Pass | 8.078s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.289s | |
--| Basic Echo Function | ✅ Pass | 7.959s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.885s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 12.785s | |
++| Simple Math | ✅ Pass | 0.271s | |
++| Text Transform Uppercase | ✅ Pass | 0.218s | |
++| Count from 1 to 5 | ✅ Pass | 0.206s | |
++| Math Calculation | ✅ Pass | 0.209s | |
++| Basic Echo Function | ✅ Pass | 0.207s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.215s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.215s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.203s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 5.854s | |
--| Ask Advice Function | ✅ Pass | 2.945s | |
--| Basic Context Memory Test | ✅ Pass | 6.477s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 10.439s | |
--| Function Argument Memory Test | ✅ Pass | 5.706s | |
--| Search Query Function | ✅ Pass | 17.551s | |
--| Function Response Memory Test | ✅ Pass | 6.284s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 5.072s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 12.418s | |
--| Penetration Testing Methodology | ✅ Pass | 8.316s | |
--| SQL Injection Attack Type | ✅ Pass | 5.413s | |
--| Vulnerability Assessment Tools | ✅ Pass | 11.698s | |
--| Penetration Testing Framework | ✅ Pass | 5.109s | |
--| Web Application Security Scanner | ✅ Pass | 4.251s | |
--| Penetration Testing Tool Selection | ✅ Pass | 5.821s | |
++| JSON Response Function | ✅ Pass | 0.209s | |
++| Search Query Function | ✅ Pass | 0.205s | |
++| Ask Advice Function | ✅ Pass | 0.212s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.206s | |
++| Basic Context Memory Test | ✅ Pass | 0.209s | |
++| Function Argument Memory Test | ✅ Pass | 0.203s | |
++| Function Response Memory Test | ✅ Pass | 0.205s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.207s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.209s | |
++| Penetration Testing Methodology | ✅ Pass | 0.222s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.212s | |
++| SQL Injection Attack Type | ✅ Pass | 0.208s | |
++| Penetration Testing Framework | ✅ Pass | 0.214s | |
++| Web Application Security Scanner | ✅ Pass | 0.206s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.202s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 7.285s
++**Average latency**: 0.212s
---
--### assistant (openai/gpt-5)
++### assistant (z-ai/glm-5-turbo)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Text Transform Uppercase | ✅ Pass | 4.176s | |
--| Simple Math | ✅ Pass | 4.241s | |
--| Count from 1 to 5 | ✅ Pass | 4.418s | |
--| Math Calculation | ✅ Pass | 2.466s | |
--| Streaming Simple Math Streaming | ✅ Pass | 4.288s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.402s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 4.997s | |
--| Basic Echo Function | ✅ Pass | 14.115s | |
++| Simple Math | ✅ Pass | 0.080s | |
++| Text Transform Uppercase | ✅ Pass | 0.221s | |
++| Count from 1 to 5 | ✅ Pass | 0.206s | |
++| Math Calculation | ✅ Pass | 0.214s | |
++| Basic Echo Function | ✅ Pass | 0.202s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.219s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.210s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.211s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Ask Advice Function | ✅ Pass | 3.039s | |
--| Search Query Function | ✅ Pass | 9.098s | |
--| Function Argument Memory Test | ✅ Pass | 3.562s | |
--| Basic Context Memory Test | ✅ Pass | 8.180s | |
--| Function Response Memory Test | ✅ Pass | 4.814s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 15.423s | |
--| JSON Response Function | ✅ Pass | 24.602s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 7.121s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 6.563s | |
--| SQL Injection Attack Type | ✅ Pass | 7.029s | |
--| Penetration Testing Methodology | ✅ Pass | 16.605s | |
--| Vulnerability Assessment Tools | ✅ Pass | 17.711s | |
--| Web Application Security Scanner | ✅ Pass | 3.749s | |
--| Penetration Testing Framework | ✅ Pass | 7.171s | |
--| Penetration Testing Tool Selection | ✅ Pass | 9.317s | |
++| JSON Response Function | ✅ Pass | 0.205s | |
++| Search Query Function | ✅ Pass | 0.201s | |
++| Ask Advice Function | ✅ Pass | 0.211s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.210s | |
++| Basic Context Memory Test | ✅ Pass | 0.206s | |
++| Function Argument Memory Test | ✅ Pass | 0.202s | |
++| Function Response Memory Test | ✅ Pass | 0.204s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.207s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.213s | |
++| Penetration Testing Methodology | ✅ Pass | 0.217s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.206s | |
++| SQL Injection Attack Type | ✅ Pass | 0.208s | |
++| Penetration Testing Framework | ✅ Pass | 0.214s | |
++| Web Application Security Scanner | ✅ Pass | 0.203s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.204s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 8.135s
++**Average latency**: 0.204s
---
--### generator (anthropic/claude-sonnet-4.5)
++### generator (z-ai/glm-5.2)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 4.796s | |
--| Text Transform Uppercase | ✅ Pass | 4.900s | |
--| Count from 1 to 5 | ✅ Pass | 3.211s | |
--| Math Calculation | ✅ Pass | 2.543s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.894s | |
--| Basic Echo Function | ✅ Pass | 3.969s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.810s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.255s | |
++| Simple Math | ✅ Pass | 0.237s | |
++| Text Transform Uppercase | ✅ Pass | 0.220s | |
++| Count from 1 to 5 | ✅ Pass | 0.207s | |
++| Math Calculation | ✅ Pass | 0.213s | |
++| Basic Echo Function | ✅ Pass | 0.206s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.214s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.210s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.217s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.649s | |
--| Search Query Function | ✅ Pass | 3.659s | |
--| Ask Advice Function | ✅ Pass | 3.011s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 3.007s | |
--| Basic Context Memory Test | ✅ Pass | 2.584s | |
--| Function Argument Memory Test | ✅ Pass | 3.795s | |
--| Function Response Memory Test | ✅ Pass | 3.613s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.593s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.289s | |
--| Penetration Testing Methodology | ✅ Pass | 11.070s | |
--| Vulnerability Assessment Tools | ✅ Pass | 9.953s | |
--| SQL Injection Attack Type | ✅ Pass | 4.623s | |
--| Web Application Security Scanner | ✅ Pass | 6.242s | |
--| Penetration Testing Framework | ✅ Pass | 9.207s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.393s | |
++| JSON Response Function | ✅ Pass | 0.211s | |
++| Search Query Function | ✅ Pass | 0.204s | |
++| Ask Advice Function | ✅ Pass | 0.206s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.204s | |
++| Basic Context Memory Test | ✅ Pass | 0.200s | |
++| Function Argument Memory Test | ✅ Pass | 0.208s | |
++| Function Response Memory Test | ✅ Pass | 0.204s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.262s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.207s | |
++| Penetration Testing Methodology | ✅ Pass | 0.203s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.209s | |
++| SQL Injection Attack Type | ✅ Pass | 0.200s | |
++| Penetration Testing Framework | ✅ Pass | 0.204s | |
++| Web Application Security Scanner | ✅ Pass | 0.212s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.204s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 4.525s
++**Average latency**: 0.212s
---
--### refiner (google/gemini-2.5-pro)
++### refiner (z-ai/glm-5.2)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 4.765s | |
--| Text Transform Uppercase | ✅ Pass | 8.021s | |
--| Count from 1 to 5 | ✅ Pass | 5.828s | |
--| Math Calculation | ✅ Pass | 3.337s | |
--| Basic Echo Function | ✅ Pass | 3.749s | |
--| Streaming Simple Math Streaming | ✅ Pass | 4.356s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.657s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.911s | |
++| Simple Math | ✅ Pass | 0.251s | |
++| Text Transform Uppercase | ✅ Pass | 0.215s | |
++| Count from 1 to 5 | ✅ Pass | 0.204s | |
++| Math Calculation | ✅ Pass | 0.214s | |
++| Basic Echo Function | ✅ Pass | 0.207s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.213s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.213s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.218s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.989s | |
--| Search Query Function | ✅ Pass | 4.489s | |
--| Ask Advice Function | ✅ Pass | 3.256s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 3.329s | |
--| Function Argument Memory Test | ❌ Fail | 1.329s | expected text 'Go programming language' not found |
--| Basic Context Memory Test | ✅ Pass | 4.987s | |
--| Function Response Memory Test | ❌ Fail | 1.624s | expected text '22' not found |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 6.209s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.357s | |
--| Penetration Testing Methodology | ✅ Pass | 9.174s | |
--| Vulnerability Assessment Tools | ✅ Pass | 11.362s | |
--| SQL Injection Attack Type | ✅ Pass | 7.982s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.862s | |
--| Web Application Security Scanner | ✅ Pass | 11.705s | |
--| Penetration Testing Framework | ✅ Pass | 15.968s | |
--
--**Summary**: 21/23 (91.30%) successful tests
--
--**Average latency**: 5.576s
++| JSON Response Function | ✅ Pass | 0.212s | |
++| Search Query Function | ✅ Pass | 0.206s | |
++| Ask Advice Function | ✅ Pass | 0.203s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.210s | |
++| Basic Context Memory Test | ✅ Pass | 0.199s | |
++| Function Argument Memory Test | ✅ Pass | 0.206s | |
++| Function Response Memory Test | ✅ Pass | 0.208s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.198s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.201s | |
++| Penetration Testing Methodology | ✅ Pass | 0.206s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.214s | |
++| SQL Injection Attack Type | ✅ Pass | 0.207s | |
++| Penetration Testing Framework | ✅ Pass | 0.202s | |
++| Web Application Security Scanner | ✅ Pass | 0.201s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.203s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.210s
---
--### adviser (google/gemini-2.5-pro)
++### adviser (minimax/minimax-m3)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 4.749s | |
--| Count from 1 to 5 | ✅ Pass | 4.983s | |
--| Text Transform Uppercase | ✅ Pass | 7.715s | |
--| Math Calculation | ✅ Pass | 3.160s | |
--| Basic Echo Function | ✅ Pass | 3.491s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.304s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.330s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.641s | |
++| Simple Math | ✅ Pass | 0.264s | |
++| Text Transform Uppercase | ✅ Pass | 0.204s | |
++| Count from 1 to 5 | ✅ Pass | 0.207s | |
++| Math Calculation | ✅ Pass | 0.211s | |
++| Basic Echo Function | ✅ Pass | 0.202s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.210s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.207s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.204s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.319s | |
--| Search Query Function | ✅ Pass | 3.352s | |
--| Ask Advice Function | ✅ Pass | 2.876s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 3.286s | |
--| Basic Context Memory Test | ✅ Pass | 5.184s | |
--| Function Argument Memory Test | ✅ Pass | 3.338s | |
--| Function Response Memory Test | ❌ Fail | 1.962s | expected text '22' not found |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 5.316s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.781s | |
--| Penetration Testing Methodology | ✅ Pass | 10.426s | |
--| Vulnerability Assessment Tools | ✅ Pass | 12.932s | |
--| SQL Injection Attack Type | ✅ Pass | 6.701s | |
--| Penetration Testing Tool Selection | ✅ Pass | 4.242s | |
--| Penetration Testing Framework | ✅ Pass | 13.500s | |
--| Web Application Security Scanner | ✅ Pass | 12.631s | |
--
--**Summary**: 22/23 (95.65%) successful tests
--
--**Average latency**: 5.532s
++| JSON Response Function | ✅ Pass | 0.218s | |
++| Search Query Function | ✅ Pass | 0.203s | |
++| Ask Advice Function | ✅ Pass | 0.202s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.226s | |
++| Basic Context Memory Test | ✅ Pass | 0.198s | |
++| Function Argument Memory Test | ✅ Pass | 0.214s | |
++| Function Response Memory Test | ✅ Pass | 0.207s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.208s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.201s | |
++| Penetration Testing Methodology | ✅ Pass | 0.210s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.213s | |
++| SQL Injection Attack Type | ✅ Pass | 0.210s | |
++| Penetration Testing Framework | ✅ Pass | 0.202s | |
++| Web Application Security Scanner | ✅ Pass | 0.203s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.204s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.210s
---
--### reflector (openai/gpt-4.1-mini)
++### reflector (deepseek/deepseek-v4-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.664s | |
--| Text Transform Uppercase | ✅ Pass | 3.352s | |
--| Count from 1 to 5 | ✅ Pass | 1.470s | |
--| Math Calculation | ✅ Pass | 1.184s | |
--| Basic Echo Function | ✅ Pass | 1.459s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.206s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.110s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.144s | |
++| Simple Math | ✅ Pass | 0.262s | |
++| Text Transform Uppercase | ✅ Pass | 0.204s | |
++| Count from 1 to 5 | ✅ Pass | 0.204s | |
++| Math Calculation | ✅ Pass | 0.217s | |
++| Basic Echo Function | ✅ Pass | 0.208s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.213s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.231s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.211s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.294s | |
--| Search Query Function | ✅ Pass | 1.555s | |
--| Ask Advice Function | ✅ Pass | 1.328s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.209s | |
--| Basic Context Memory Test | ✅ Pass | 1.465s | |
--| Function Argument Memory Test | ✅ Pass | 1.186s | |
--| Function Response Memory Test | ✅ Pass | 1.476s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.031s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.505s | |
--| Penetration Testing Methodology | ✅ Pass | 1.356s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.687s | |
--| SQL Injection Attack Type | ✅ Pass | 1.316s | |
--| Penetration Testing Framework | ✅ Pass | 1.093s | |
--| Web Application Security Scanner | ✅ Pass | 1.298s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.387s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 1.556s
++| JSON Response Function | ✅ Pass | 0.205s | |
++| Search Query Function | ✅ Pass | 0.204s | |
++| Ask Advice Function | ✅ Pass | 0.211s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.232s | |
++| Basic Context Memory Test | ✅ Pass | 0.198s | |
++| Function Argument Memory Test | ✅ Pass | 0.215s | |
++| Function Response Memory Test | ✅ Pass | 0.208s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.209s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.203s | |
++| Penetration Testing Methodology | ✅ Pass | 0.201s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.210s | |
++| SQL Injection Attack Type | ✅ Pass | 0.205s | |
++| Penetration Testing Framework | ✅ Pass | 0.209s | |
++| Web Application Security Scanner | ✅ Pass | 0.204s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.208s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Explicit Reasoning Off Suppresses Reasoning | reasoning_off | ✅ Pass | 0.205s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 0.212s
---
--### searcher (x-ai/grok-3-mini)
++### searcher (deepseek/deepseek-v4-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 4.840s | |
--| Text Transform Uppercase | ✅ Pass | 5.601s | |
--| Count from 1 to 5 | ✅ Pass | 4.014s | |
--| Math Calculation | ✅ Pass | 3.175s | |
--| Basic Echo Function | ✅ Pass | 4.012s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.994s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.596s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.705s | |
++| Simple Math | ✅ Pass | 0.267s | |
++| Text Transform Uppercase | ✅ Pass | 0.203s | |
++| Count from 1 to 5 | ✅ Pass | 0.206s | |
++| Math Calculation | ✅ Pass | 0.210s | |
++| Basic Echo Function | ✅ Pass | 0.206s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.215s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.211s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.217s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 5.163s | |
--| Search Query Function | ✅ Pass | 3.663s | |
--| Ask Advice Function | ✅ Pass | 4.934s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 5.816s | |
--| Basic Context Memory Test | ✅ Pass | 3.479s | |
--| Function Argument Memory Test | ✅ Pass | 3.226s | |
--| Function Response Memory Test | ✅ Pass | 3.040s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 5.192s | |
--| Penetration Testing Methodology | ✅ Pass | 4.935s | |
--| Cybersecurity Workflow Memory Test | ❌ Fail | 8.973s | expected text 'example\.com' not found |
--| Vulnerability Assessment Tools | ✅ Pass | 6.358s | |
--| SQL Injection Attack Type | ✅ Pass | 3.042s | |
--| Penetration Testing Framework | ✅ Pass | 5.377s | |
--| Web Application Security Scanner | ✅ Pass | 4.338s | |
--| Penetration Testing Tool Selection | ✅ Pass | 4.267s | |
--
--**Summary**: 22/23 (95.65%) successful tests
--
--**Average latency**: 4.511s
++| JSON Response Function | ✅ Pass | 0.210s | |
++| Search Query Function | ✅ Pass | 0.205s | |
++| Ask Advice Function | ✅ Pass | 0.201s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.239s | |
++| Basic Context Memory Test | ✅ Pass | 0.204s | |
++| Function Argument Memory Test | ✅ Pass | 0.219s | |
++| Function Response Memory Test | ✅ Pass | 0.209s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.267s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.204s | |
++| Penetration Testing Methodology | ✅ Pass | 0.209s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.206s | |
++| SQL Injection Attack Type | ✅ Pass | 0.204s | |
++| Penetration Testing Framework | ✅ Pass | 0.204s | |
++| Web Application Security Scanner | ✅ Pass | 0.204s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.203s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Explicit Reasoning Off Suppresses Reasoning | reasoning_off | ✅ Pass | 0.203s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 0.214s
---
--### enricher (openai/gpt-4.1-mini)
++### enricher (minimax/minimax-m3)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.796s | |
--| Text Transform Uppercase | ✅ Pass | 3.117s | |
--| Count from 1 to 5 | ✅ Pass | 1.902s | |
--| Math Calculation | ✅ Pass | 0.887s | |
--| Basic Echo Function | ✅ Pass | 1.260s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.943s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.273s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.393s | |
++| Simple Math | ✅ Pass | 0.203s | |
++| Text Transform Uppercase | ✅ Pass | 0.205s | |
++| Count from 1 to 5 | ✅ Pass | 0.206s | |
++| Math Calculation | ✅ Pass | 0.209s | |
++| Basic Echo Function | ✅ Pass | 0.208s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.213s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.217s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.221s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.519s | |
--| Search Query Function | ✅ Pass | 1.304s | |
--| Ask Advice Function | ✅ Pass | 1.661s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.592s | |
--| Basic Context Memory Test | ✅ Pass | 1.266s | |
--| Function Argument Memory Test | ✅ Pass | 1.239s | |
--| Function Response Memory Test | ✅ Pass | 1.617s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.076s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.278s | |
--| Penetration Testing Methodology | ✅ Pass | 1.934s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.300s | |
--| SQL Injection Attack Type | ✅ Pass | 1.211s | |
--| Penetration Testing Framework | ✅ Pass | 1.614s | |
--| Web Application Security Scanner | ✅ Pass | 1.195s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.334s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 1.597s
++| JSON Response Function | ✅ Pass | 0.205s | |
++| Search Query Function | ✅ Pass | 0.203s | |
++| Ask Advice Function | ✅ Pass | 0.201s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.235s | |
++| Basic Context Memory Test | ✅ Pass | 0.203s | |
++| Function Argument Memory Test | ✅ Pass | 0.214s | |
++| Function Response Memory Test | ✅ Pass | 0.207s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.200s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.204s | |
++| Penetration Testing Methodology | ✅ Pass | 0.211s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.208s | |
++| SQL Injection Attack Type | ✅ Pass | 0.205s | |
++| Penetration Testing Framework | ✅ Pass | 0.206s | |
++| Web Application Security Scanner | ✅ Pass | 0.203s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.205s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Explicit Reasoning Off Suppresses Reasoning | reasoning_off | ✅ Pass | 0.202s | |
++
++**Summary**: 24/24 (100.00%) successful tests
++
++**Average latency**: 0.208s
---
--### coder (anthropic/claude-sonnet-4.5)
++### coder (moonshotai/kimi-k2.7-code)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 5.233s | |
--| Text Transform Uppercase | ✅ Pass | 5.161s | |
--| Count from 1 to 5 | ✅ Pass | 3.227s | |
--| Math Calculation | ✅ Pass | 2.882s | |
--| Basic Echo Function | ✅ Pass | 3.143s | |
--| Streaming Simple Math Streaming | ✅ Pass | 2.506s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.003s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.763s | |
++| Simple Math | ✅ Pass | 0.207s | |
++| Text Transform Uppercase | ✅ Pass | 0.203s | |
++| Count from 1 to 5 | ✅ Pass | 0.214s | |
++| Math Calculation | ✅ Pass | 0.212s | |
++| Basic Echo Function | ✅ Pass | 0.210s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.216s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.209s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.212s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 3.419s | |
--| Search Query Function | ✅ Pass | 3.017s | |
--| Ask Advice Function | ✅ Pass | 2.999s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.992s | |
--| Basic Context Memory Test | ✅ Pass | 3.126s | |
--| Function Argument Memory Test | ✅ Pass | 3.670s | |
--| Function Response Memory Test | ✅ Pass | 3.248s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.631s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.598s | |
--| Penetration Testing Methodology | ✅ Pass | 11.220s | |
--| Vulnerability Assessment Tools | ✅ Pass | 9.139s | |
--| SQL Injection Attack Type | ✅ Pass | 4.317s | |
--| Penetration Testing Framework | ✅ Pass | 7.797s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.140s | |
--| Web Application Security Scanner | ✅ Pass | 8.004s | |
++| JSON Response Function | ✅ Pass | 0.205s | |
++| Search Query Function | ✅ Pass | 0.203s | |
++| Ask Advice Function | ✅ Pass | 0.214s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.239s | |
++| Basic Context Memory Test | ✅ Pass | 0.205s | |
++| Function Argument Memory Test | ✅ Pass | 0.216s | |
++| Function Response Memory Test | ✅ Pass | 0.204s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.259s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.210s | |
++| Penetration Testing Methodology | ✅ Pass | 0.212s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.208s | |
++| SQL Injection Attack Type | ✅ Pass | 0.212s | |
++| Penetration Testing Framework | ✅ Pass | 0.203s | |
++| Web Application Security Scanner | ✅ Pass | 0.199s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.207s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 4.445s
++**Average latency**: 0.213s
---
--### installer (google/gemini-2.5-flash)
++### installer (moonshotai/kimi-k2.7-code)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.212s | |
--| Text Transform Uppercase | ✅ Pass | 2.828s | |
--| Count from 1 to 5 | ✅ Pass | 0.800s | |
--| Math Calculation | ✅ Pass | 1.529s | |
--| Basic Echo Function | ✅ Pass | 1.841s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.011s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.480s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.747s | |
++| Simple Math | ✅ Pass | 0.209s | |
++| Text Transform Uppercase | ✅ Pass | 0.207s | |
++| Count from 1 to 5 | ✅ Pass | 0.214s | |
++| Math Calculation | ✅ Pass | 0.204s | |
++| Basic Echo Function | ✅ Pass | 0.210s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.221s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.210s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.203s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.678s | |
--| Search Query Function | ✅ Pass | 1.535s | |
--| Ask Advice Function | ✅ Pass | 2.439s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 3.017s | |
--| Basic Context Memory Test | ✅ Pass | 2.790s | |
--| Function Response Memory Test | ✅ Pass | 0.868s | |
--| Function Argument Memory Test | ✅ Pass | 3.503s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.933s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.969s | |
--| Penetration Testing Methodology | ✅ Pass | 6.046s | |
--| Vulnerability Assessment Tools | ✅ Pass | 8.005s | |
--| SQL Injection Attack Type | ✅ Pass | 2.731s | |
--| Web Application Security Scanner | ✅ Pass | 4.804s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.752s | |
--| Penetration Testing Framework | ✅ Pass | 13.820s | |
++| JSON Response Function | ✅ Pass | 0.203s | |
++| Search Query Function | ✅ Pass | 0.205s | |
++| Ask Advice Function | ✅ Pass | 0.206s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.227s | |
++| Basic Context Memory Test | ✅ Pass | 0.210s | |
++| Function Argument Memory Test | ✅ Pass | 0.209s | |
++| Function Response Memory Test | ✅ Pass | 0.204s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.262s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.215s | |
++| Penetration Testing Methodology | ✅ Pass | 0.204s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.207s | |
++| SQL Injection Attack Type | ✅ Pass | 0.212s | |
++| Penetration Testing Framework | ✅ Pass | 0.207s | |
++| Web Application Security Scanner | ✅ Pass | 0.201s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.212s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 3.276s
++**Average latency**: 0.212s
---
--### pentester (moonshotai/kimi-k2-0905)
++### pentester (deepseek/deepseek-v4-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 2.485s | |
--| Text Transform Uppercase | ✅ Pass | 1.845s | |
--| Count from 1 to 5 | ✅ Pass | 1.481s | |
--| Math Calculation | ✅ Pass | 1.625s | |
--| Basic Echo Function | ✅ Pass | 1.611s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.693s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.843s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.580s | |
++| Simple Math | ✅ Pass | 0.204s | |
++| Text Transform Uppercase | ✅ Pass | 0.213s | |
++| Count from 1 to 5 | ✅ Pass | 0.218s | |
++| Math Calculation | ✅ Pass | 0.204s | |
++| Basic Echo Function | ✅ Pass | 0.211s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.225s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.204s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.207s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.278s | |
--| Search Query Function | ✅ Pass | 1.168s | |
--| Search Query Function Streaming | ❌ Fail | 0.747s | streaming tool call func returned an error: tool call name is required |
--| Ask Advice Function | ✅ Pass | 4.458s | |
--| Basic Context Memory Test | ✅ Pass | 1.566s | |
--| Function Argument Memory Test | ✅ Pass | 1.148s | |
--| Function Response Memory Test | ✅ Pass | 0.811s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.292s | |
--| Penetration Testing Methodology | ✅ Pass | 2.511s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 11.318s | |
--| Vulnerability Assessment Tools | ✅ Pass | 0.730s | |
--| SQL Injection Attack Type | ✅ Pass | 3.724s | |
--| Penetration Testing Framework | ✅ Pass | 0.782s | |
--| Web Application Security Scanner | ✅ Pass | 3.533s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.674s | |
--
--**Summary**: 22/23 (95.65%) successful tests
--
--**Average latency**: 2.301s
++| JSON Response Function | ✅ Pass | 0.204s | |
++| Search Query Function | ✅ Pass | 0.204s | |
++| Ask Advice Function | ✅ Pass | 0.204s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.222s | |
++| Basic Context Memory Test | ✅ Pass | 0.204s | |
++| Function Argument Memory Test | ✅ Pass | 0.200s | |
++| Function Response Memory Test | ✅ Pass | 0.202s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 0.203s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.211s | |
++| Penetration Testing Methodology | ✅ Pass | 0.208s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.203s | |
++| SQL Injection Attack Type | ✅ Pass | 0.208s | |
++| Penetration Testing Framework | ✅ Pass | 0.202s | |
++| Web Application Security Scanner | ✅ Pass | 0.199s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.204s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.208s
---
diff --cc examples/tests/qwen-report.md
index 56c2f14e,56c2f14e..af7b2b32
--- a/examples/tests/qwen-report.md
+++ b/examples/tests/qwen-report.md
@@@ -1,27 -1,27 +1,27 @@@
# LLM Agent Testing Report
--Generated: Thu, 28 May 2026 16:06:51 UTC
++Generated: Tue, 21 Jul 2026 17:06:14 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
--| simple | qwen3.5-flash | false | 23/23 (100.00%) | 1.194s |
--| simple_json | qwen3.5-flash | false | 4/5 (80.00%) | 0.945s |
--| primary_agent | qwen3.6-plus | true | 23/23 (100.00%) | 6.079s |
--| assistant | qwen3.6-plus | true | 23/23 (100.00%) | 5.512s |
--| generator | qwen3.7-max | true | 23/23 (100.00%) | 5.172s |
--| refiner | qwen3.7-max | true | 23/23 (100.00%) | 5.455s |
--| adviser | qwen3.7-max | true | 23/23 (100.00%) | 4.750s |
--| reflector | qwen3.5-flash | true | 23/23 (100.00%) | 1.222s |
--| searcher | qwen3.5-flash | true | 23/23 (100.00%) | 1.083s |
--| enricher | qwen3.5-flash | true | 23/23 (100.00%) | 0.972s |
--| coder | qwen3-coder-plus | true | 23/23 (100.00%) | 1.702s |
--| installer | qwen3-coder-flash | true | 23/23 (100.00%) | 1.124s |
--| pentester | qwen3.6-plus | true | 23/23 (100.00%) | 9.207s |
--
--**Total**: 280/281 (99.64%) successful tests
--**Overall average latency**: 3.575s
++| simple | qwen3.5-flash | false | 23/23 (100.00%) | 1.089s |
++| simple_json | qwen3.5-flash | false | 7/7 (100.00%) | 1.073s |
++| primary_agent | qwen3.6-plus | true | 23/23 (100.00%) | 5.687s |
++| assistant | qwen3.6-plus | true | 23/23 (100.00%) | 6.443s |
++| generator | qwen3.7-max | true | 23/23 (100.00%) | 5.511s |
++| refiner | qwen3.7-max | true | 23/23 (100.00%) | 5.634s |
++| adviser | qwen3.7-max | true | 23/23 (100.00%) | 5.838s |
++| reflector | qwen3.5-flash | true | 23/23 (100.00%) | 1.094s |
++| searcher | qwen3.5-flash | true | 23/23 (100.00%) | 1.055s |
++| enricher | qwen3.5-flash | true | 22/23 (95.65%) | 0.423s |
++| coder | qwen3-coder-plus | true | 23/23 (100.00%) | 1.732s |
++| installer | qwen3-coder-flash | true | 21/23 (91.30%) | 1.399s |
++| pentester | qwen3.6-plus | true | 23/23 (100.00%) | 4.053s |
++
++**Total**: 280/283 (98.94%) successful tests
++**Overall average latency**: 3.274s
## Detailed Results
@@@ -31,38 -31,38 +31,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.515s | |
--| Text Transform Uppercase | ✅ Pass | 1.075s | |
--| Count from 1 to 5 | ✅ Pass | 1.066s | |
--| Math Calculation | ✅ Pass | 1.122s | |
--| Basic Echo Function | ✅ Pass | 1.252s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.603s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.569s | |
++| Simple Math | ✅ Pass | 1.206s | |
++| Text Transform Uppercase | ✅ Pass | 0.678s | |
++| Count from 1 to 5 | ✅ Pass | 0.704s | |
++| Math Calculation | ✅ Pass | 0.688s | |
++| Basic Echo Function | ✅ Pass | 0.793s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.576s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.800s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.800s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.795s | |
--| Search Query Function | ✅ Pass | 1.148s | |
--| Ask Advice Function | ✅ Pass | 1.347s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.834s | |
--| Basic Context Memory Test | ✅ Pass | 0.655s | |
--| Function Argument Memory Test | ✅ Pass | 1.019s | |
--| Function Response Memory Test | ✅ Pass | 1.023s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.142s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.996s | |
--| Penetration Testing Methodology | ✅ Pass | 2.497s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.849s | |
--| SQL Injection Attack Type | ✅ Pass | 0.552s | |
--| Penetration Testing Framework | ✅ Pass | 1.971s | |
--| Web Application Security Scanner | ✅ Pass | 1.461s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.152s | |
++| JSON Response Function | ✅ Pass | 0.891s | |
++| Search Query Function | ✅ Pass | 0.779s | |
++| Ask Advice Function | ✅ Pass | 0.904s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.989s | |
++| Basic Context Memory Test | ✅ Pass | 1.027s | |
++| Function Argument Memory Test | ✅ Pass | 1.263s | |
++| Function Response Memory Test | ✅ Pass | 0.818s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.280s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.760s | |
++| Penetration Testing Methodology | ✅ Pass | 2.263s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.873s | |
++| SQL Injection Attack Type | ✅ Pass | 0.970s | |
++| Penetration Testing Framework | ✅ Pass | 1.986s | |
++| Web Application Security Scanner | ✅ Pass | 1.829s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.171s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.194s
++**Average latency**: 1.089s
---
@@@ -72,15 -72,15 +72,22 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Project Information JSON | ✅ Pass | 0.751s | |
--| User Profile JSON | ✅ Pass | 0.767s | |
--| Person Information JSON | ✅ Pass | 1.137s | |
--| Vulnerability Report Memory Test | ❌ Fail | 1.318s | got map\[string\]interface \{\}\{"$schema":"http://json\-schema\.org/draft\-07/schema\#", "properties":map\[string\]interface \{\}\{"open\_ports":\... |
--| Streaming Person Information JSON Streaming | ✅ Pass | 0.747s | |
++| Person Information JSON | ✅ Pass | 1.133s | |
++| Project Information JSON | ✅ Pass | 0.909s | |
++| Vulnerability Report Memory Test | ✅ Pass | 1.825s | |
++| User Profile JSON | ✅ Pass | 1.141s | |
++| JSON Array Response Without Schema | ✅ Pass | 0.796s | |
++| Streaming Person Information JSON Streaming | ✅ Pass | 0.753s | |
--**Summary**: 4/5 (80.00%) successful tests
++#### Capability Tests
--**Average latency**: 0.945s
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Structured Output With JSON Schema | structured_output | ✅ Pass | 0.948s | |
++
++**Summary**: 7/7 (100.00%) successful tests
++
++**Average latency**: 1.073s
---
@@@ -90,38 -90,38 +97,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 5.447s | |
--| Text Transform Uppercase | ✅ Pass | 5.747s | |
--| Count from 1 to 5 | ✅ Pass | 6.848s | |
--| Math Calculation | ✅ Pass | 3.912s | |
--| Basic Echo Function | ✅ Pass | 2.791s | |
--| Streaming Simple Math Streaming | ✅ Pass | 4.355s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.205s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.995s | |
++| Simple Math | ✅ Pass | 5.698s | |
++| Text Transform Uppercase | ✅ Pass | 4.584s | |
++| Count from 1 to 5 | ✅ Pass | 5.365s | |
++| Math Calculation | ✅ Pass | 3.524s | |
++| Basic Echo Function | ✅ Pass | 5.089s | |
++| Streaming Simple Math Streaming | ✅ Pass | 4.090s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.484s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.239s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 4.066s | |
--| Search Query Function | ✅ Pass | 2.232s | |
--| Ask Advice Function | ✅ Pass | 2.860s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.094s | |
--| Basic Context Memory Test | ✅ Pass | 4.240s | |
--| Function Argument Memory Test | ✅ Pass | 2.635s | |
--| Function Response Memory Test | ✅ Pass | 9.775s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 7.318s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 4.524s | |
--| Penetration Testing Methodology | ✅ Pass | 13.685s | |
--| Vulnerability Assessment Tools | ✅ Pass | 12.372s | |
--| SQL Injection Attack Type | ✅ Pass | 11.677s | |
--| Penetration Testing Framework | ✅ Pass | 12.500s | |
--| Web Application Security Scanner | ✅ Pass | 9.014s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.509s | |
++| JSON Response Function | ✅ Pass | 3.262s | |
++| Search Query Function | ✅ Pass | 5.559s | |
++| Ask Advice Function | ✅ Pass | 3.116s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 3.530s | |
++| Basic Context Memory Test | ✅ Pass | 5.453s | |
++| Function Argument Memory Test | ✅ Pass | 2.121s | |
++| Function Response Memory Test | ✅ Pass | 5.466s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.208s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.918s | |
++| Penetration Testing Methodology | ✅ Pass | 10.966s | |
++| Vulnerability Assessment Tools | ✅ Pass | 11.440s | |
++| SQL Injection Attack Type | ✅ Pass | 5.690s | |
++| Penetration Testing Framework | ✅ Pass | 14.808s | |
++| Web Application Security Scanner | ✅ Pass | 13.232s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.953s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 6.079s
++**Average latency**: 5.687s
---
@@@ -131,38 -131,38 +138,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 4.896s | |
--| Text Transform Uppercase | ✅ Pass | 4.750s | |
--| Count from 1 to 5 | ✅ Pass | 5.467s | |
--| Math Calculation | ✅ Pass | 3.192s | |
--| Basic Echo Function | ✅ Pass | 2.627s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.657s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.401s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 5.018s | |
++| Simple Math | ✅ Pass | 5.941s | |
++| Text Transform Uppercase | ✅ Pass | 5.097s | |
++| Count from 1 to 5 | ✅ Pass | 5.431s | |
++| Math Calculation | ✅ Pass | 3.949s | |
++| Basic Echo Function | ✅ Pass | 3.107s | |
++| Streaming Simple Math Streaming | ✅ Pass | 3.569s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.479s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 3.581s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.385s | |
--| Search Query Function | ✅ Pass | 2.864s | |
--| Ask Advice Function | ✅ Pass | 3.599s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.866s | |
--| Basic Context Memory Test | ✅ Pass | 4.909s | |
--| Function Argument Memory Test | ✅ Pass | 3.381s | |
--| Function Response Memory Test | ✅ Pass | 4.146s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 5.778s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 5.842s | |
--| Penetration Testing Methodology | ✅ Pass | 9.568s | |
--| SQL Injection Attack Type | ✅ Pass | 6.860s | |
--| Vulnerability Assessment Tools | ✅ Pass | 18.097s | |
--| Penetration Testing Framework | ✅ Pass | 10.065s | |
--| Web Application Security Scanner | ✅ Pass | 9.724s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.665s | |
++| JSON Response Function | ✅ Pass | 2.656s | |
++| Search Query Function | ✅ Pass | 4.222s | |
++| Ask Advice Function | ✅ Pass | 2.965s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.803s | |
++| Basic Context Memory Test | ✅ Pass | 5.119s | |
++| Function Argument Memory Test | ✅ Pass | 3.266s | |
++| Function Response Memory Test | ✅ Pass | 3.152s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 5.938s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.738s | |
++| Penetration Testing Methodology | ✅ Pass | 11.137s | |
++| SQL Injection Attack Type | ✅ Pass | 5.517s | |
++| Vulnerability Assessment Tools | ✅ Pass | 30.782s | |
++| Penetration Testing Framework | ✅ Pass | 16.326s | |
++| Web Application Security Scanner | ✅ Pass | 11.357s | |
++| Penetration Testing Tool Selection | ✅ Pass | 4.043s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 5.512s
++**Average latency**: 6.443s
---
@@@ -172,38 -172,38 +179,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 5.569s | |
--| Text Transform Uppercase | ✅ Pass | 4.279s | |
--| Math Calculation | ✅ Pass | 2.508s | |
--| Basic Echo Function | ✅ Pass | 2.450s | |
--| Count from 1 to 5 | ✅ Pass | 13.152s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.691s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.566s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.392s | |
++| Simple Math | ✅ Pass | 4.825s | |
++| Text Transform Uppercase | ✅ Pass | 4.570s | |
++| Math Calculation | ✅ Pass | 2.269s | |
++| Count from 1 to 5 | ✅ Pass | 8.635s | |
++| Basic Echo Function | ✅ Pass | 2.687s | |
++| Streaming Simple Math Streaming | ✅ Pass | 3.671s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.068s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 7.738s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.320s | |
--| Search Query Function | ✅ Pass | 3.027s | |
--| Ask Advice Function | ✅ Pass | 4.213s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.036s | |
--| Basic Context Memory Test | ✅ Pass | 7.051s | |
--| Function Argument Memory Test | ✅ Pass | 3.966s | |
--| Function Response Memory Test | ✅ Pass | 7.543s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 3.481s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 4.203s | |
--| Penetration Testing Methodology | ✅ Pass | 6.768s | |
--| SQL Injection Attack Type | ✅ Pass | 4.673s | |
--| Vulnerability Assessment Tools | ✅ Pass | 18.743s | |
--| Penetration Testing Framework | ✅ Pass | 5.278s | |
--| Web Application Security Scanner | ✅ Pass | 3.018s | |
--| Penetration Testing Tool Selection | ✅ Pass | 4.024s | |
++| JSON Response Function | ✅ Pass | 3.230s | |
++| Search Query Function | ✅ Pass | 2.312s | |
++| Ask Advice Function | ✅ Pass | 2.780s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.880s | |
++| Basic Context Memory Test | ✅ Pass | 5.618s | |
++| Function Argument Memory Test | ✅ Pass | 5.056s | |
++| Function Response Memory Test | ✅ Pass | 6.050s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.941s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 5.151s | |
++| Penetration Testing Methodology | ✅ Pass | 8.502s | |
++| SQL Injection Attack Type | ✅ Pass | 6.876s | |
++| Penetration Testing Framework | ✅ Pass | 5.269s | |
++| Vulnerability Assessment Tools | ✅ Pass | 25.061s | |
++| Web Application Security Scanner | ✅ Pass | 4.410s | |
++| Penetration Testing Tool Selection | ✅ Pass | 3.144s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 5.172s
++**Average latency**: 5.511s
---
@@@ -213,38 -213,38 +220,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 4.724s | |
--| Text Transform Uppercase | ✅ Pass | 4.732s | |
--| Count from 1 to 5 | ✅ Pass | 8.230s | |
--| Math Calculation | ✅ Pass | 2.684s | |
--| Basic Echo Function | ✅ Pass | 2.505s | |
--| Streaming Simple Math Streaming | ✅ Pass | 5.229s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.093s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.982s | |
++| Simple Math | ✅ Pass | 4.902s | |
++| Text Transform Uppercase | ✅ Pass | 4.827s | |
++| Math Calculation | ✅ Pass | 2.847s | |
++| Count from 1 to 5 | ✅ Pass | 8.910s | |
++| Basic Echo Function | ✅ Pass | 2.504s | |
++| Streaming Simple Math Streaming | ✅ Pass | 3.818s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.238s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.566s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.442s | |
--| Search Query Function | ✅ Pass | 2.913s | |
--| Ask Advice Function | ✅ Pass | 3.594s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 3.382s | |
--| Basic Context Memory Test | ✅ Pass | 4.152s | |
--| Function Argument Memory Test | ✅ Pass | 3.718s | |
--| Function Response Memory Test | ✅ Pass | 5.162s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 4.590s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 4.049s | |
--| Penetration Testing Methodology | ✅ Pass | 5.504s | |
--| SQL Injection Attack Type | ✅ Pass | 5.041s | |
--| Vulnerability Assessment Tools | ✅ Pass | 20.343s | |
--| Penetration Testing Framework | ✅ Pass | 10.747s | |
--| Web Application Security Scanner | ✅ Pass | 10.091s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.551s | |
++| Search Query Function | ✅ Pass | 2.115s | |
++| JSON Response Function | ✅ Pass | 7.494s | |
++| Ask Advice Function | ✅ Pass | 3.308s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.229s | |
++| Basic Context Memory Test | ✅ Pass | 4.650s | |
++| Function Argument Memory Test | ✅ Pass | 5.215s | |
++| Function Response Memory Test | ✅ Pass | 5.734s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 5.895s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 4.718s | |
++| Penetration Testing Methodology | ✅ Pass | 8.927s | |
++| Vulnerability Assessment Tools | ✅ Pass | 18.822s | |
++| SQL Injection Attack Type | ✅ Pass | 5.127s | |
++| Penetration Testing Framework | ✅ Pass | 5.584s | |
++| Web Application Security Scanner | ✅ Pass | 7.260s | |
++| Penetration Testing Tool Selection | ✅ Pass | 5.882s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 5.455s
++**Average latency**: 5.634s
---
@@@ -254,38 -254,38 +261,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.761s | |
--| Text Transform Uppercase | ✅ Pass | 6.663s | |
--| Count from 1 to 5 | ✅ Pass | 8.046s | |
--| Math Calculation | ✅ Pass | 2.987s | |
--| Basic Echo Function | ✅ Pass | 2.306s | |
--| Streaming Simple Math Streaming | ✅ Pass | 3.074s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 2.033s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 7.517s | |
++| Simple Math | ✅ Pass | 4.759s | |
++| Text Transform Uppercase | ✅ Pass | 4.685s | |
++| Count from 1 to 5 | ✅ Pass | 7.905s | |
++| Math Calculation | ✅ Pass | 4.080s | |
++| Basic Echo Function | ✅ Pass | 2.272s | |
++| Streaming Simple Math Streaming | ✅ Pass | 3.686s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.681s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.691s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.833s | |
--| Search Query Function | ✅ Pass | 2.622s | |
--| Ask Advice Function | ✅ Pass | 3.735s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.859s | |
--| Basic Context Memory Test | ✅ Pass | 4.274s | |
--| Function Argument Memory Test | ✅ Pass | 5.488s | |
--| Function Response Memory Test | ✅ Pass | 7.424s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 2.711s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 4.780s | |
--| Penetration Testing Methodology | ✅ Pass | 6.058s | |
--| Vulnerability Assessment Tools | ✅ Pass | 13.545s | |
--| SQL Injection Attack Type | ✅ Pass | 3.451s | |
--| Penetration Testing Framework | ✅ Pass | 5.348s | |
--| Web Application Security Scanner | ✅ Pass | 6.648s | |
--| Penetration Testing Tool Selection | ✅ Pass | 3.077s | |
++| JSON Response Function | ✅ Pass | 3.129s | |
++| Search Query Function | ✅ Pass | 2.579s | |
++| Ask Advice Function | ✅ Pass | 2.710s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.434s | |
++| Basic Context Memory Test | ✅ Pass | 6.015s | |
++| Function Argument Memory Test | ✅ Pass | 6.112s | |
++| Function Response Memory Test | ✅ Pass | 5.051s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 5.913s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 7.863s | |
++| Penetration Testing Methodology | ✅ Pass | 6.513s | |
++| SQL Injection Attack Type | ✅ Pass | 4.288s | |
++| Vulnerability Assessment Tools | ✅ Pass | 23.066s | |
++| Penetration Testing Framework | ✅ Pass | 7.124s | |
++| Web Application Security Scanner | ✅ Pass | 9.098s | |
++| Penetration Testing Tool Selection | ✅ Pass | 5.607s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 4.750s
++**Average latency**: 5.838s
---
@@@ -295,38 -295,38 +302,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.664s | |
--| Text Transform Uppercase | ✅ Pass | 0.999s | |
--| Count from 1 to 5 | ✅ Pass | 0.706s | |
--| Math Calculation | ✅ Pass | 0.808s | |
--| Basic Echo Function | ✅ Pass | 1.256s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.597s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.636s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 1.226s | |
++| Simple Math | ✅ Pass | 1.435s | |
++| Text Transform Uppercase | ✅ Pass | 0.857s | |
++| Count from 1 to 5 | ✅ Pass | 0.930s | |
++| Math Calculation | ✅ Pass | 0.622s | |
++| Basic Echo Function | ✅ Pass | 0.872s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.657s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.622s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.439s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 0.959s | |
--| Search Query Function | ✅ Pass | 1.138s | |
--| Ask Advice Function | ✅ Pass | 1.422s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.120s | |
--| Basic Context Memory Test | ✅ Pass | 0.691s | |
--| Function Argument Memory Test | ✅ Pass | 0.528s | |
--| Function Response Memory Test | ✅ Pass | 0.553s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.283s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.122s | |
--| Penetration Testing Methodology | ✅ Pass | 2.639s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.019s | |
--| SQL Injection Attack Type | ✅ Pass | 1.036s | |
--| Penetration Testing Framework | ✅ Pass | 1.655s | |
--| Web Application Security Scanner | ✅ Pass | 3.147s | |
--| Penetration Testing Tool Selection | ✅ Pass | 0.902s | |
++| JSON Response Function | ✅ Pass | 0.847s | |
++| Search Query Function | ✅ Pass | 0.758s | |
++| Ask Advice Function | ✅ Pass | 0.927s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.968s | |
++| Basic Context Memory Test | ✅ Pass | 1.015s | |
++| Function Argument Memory Test | ✅ Pass | 0.950s | |
++| Function Response Memory Test | ✅ Pass | 0.617s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.238s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.620s | |
++| Penetration Testing Methodology | ✅ Pass | 1.926s | |
++| Vulnerability Assessment Tools | ✅ Pass | 2.342s | |
++| SQL Injection Attack Type | ✅ Pass | 0.764s | |
++| Penetration Testing Framework | ✅ Pass | 1.709s | |
++| Web Application Security Scanner | ✅ Pass | 1.950s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.090s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.222s
++**Average latency**: 1.094s
---
@@@ -336,38 -336,38 +343,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.541s | |
--| Text Transform Uppercase | ✅ Pass | 0.541s | |
--| Count from 1 to 5 | ✅ Pass | 0.660s | |
--| Math Calculation | ✅ Pass | 0.556s | |
--| Basic Echo Function | ✅ Pass | 0.701s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.537s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.012s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.886s | |
++| Simple Math | ✅ Pass | 1.237s | |
++| Text Transform Uppercase | ✅ Pass | 0.869s | |
++| Count from 1 to 5 | ✅ Pass | 0.895s | |
++| Math Calculation | ✅ Pass | 0.217s | |
++| Basic Echo Function | ✅ Pass | 0.802s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.225s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.991s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.370s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 0.744s | |
--| Search Query Function | ✅ Pass | 1.350s | |
--| Ask Advice Function | ✅ Pass | 1.986s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.726s | |
--| Basic Context Memory Test | ✅ Pass | 1.089s | |
--| Function Argument Memory Test | ✅ Pass | 0.558s | |
--| Function Response Memory Test | ✅ Pass | 0.620s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.284s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.607s | |
--| Penetration Testing Methodology | ✅ Pass | 2.167s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.673s | |
--| SQL Injection Attack Type | ✅ Pass | 1.089s | |
--| Penetration Testing Framework | ✅ Pass | 1.859s | |
--| Web Application Security Scanner | ✅ Pass | 1.590s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.127s | |
++| JSON Response Function | ✅ Pass | 1.451s | |
++| Search Query Function | ✅ Pass | 0.221s | |
++| Ask Advice Function | ✅ Pass | 0.919s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.839s | |
++| Basic Context Memory Test | ✅ Pass | 1.333s | |
++| Function Argument Memory Test | ✅ Pass | 0.685s | |
++| Function Response Memory Test | ✅ Pass | 0.755s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.358s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.639s | |
++| Penetration Testing Methodology | ✅ Pass | 2.105s | |
++| Vulnerability Assessment Tools | ✅ Pass | 2.267s | |
++| SQL Injection Attack Type | ✅ Pass | 0.217s | |
++| Penetration Testing Framework | ✅ Pass | 1.696s | |
++| Web Application Security Scanner | ✅ Pass | 2.174s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.987s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.083s
++**Average latency**: 1.055s
---
@@@ -377,38 -377,38 +384,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.582s | |
--| Text Transform Uppercase | ✅ Pass | 1.107s | |
--| Count from 1 to 5 | ✅ Pass | 0.693s | |
--| Math Calculation | ✅ Pass | 0.516s | |
--| Basic Echo Function | ✅ Pass | 0.700s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.523s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.544s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.772s | |
++| Simple Math | ✅ Pass | 0.216s | |
++| Text Transform Uppercase | ✅ Pass | 0.642s | |
++| Count from 1 to 5 | ✅ Pass | 0.210s | |
++| Math Calculation | ✅ Pass | 0.220s | |
++| Basic Echo Function | ✅ Pass | 0.214s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.216s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.204s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.417s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 0.830s | |
--| Search Query Function | ✅ Pass | 1.114s | |
--| Ask Advice Function | ✅ Pass | 0.989s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.729s | |
--| Basic Context Memory Test | ✅ Pass | 0.571s | |
--| Function Argument Memory Test | ✅ Pass | 0.820s | |
--| Function Response Memory Test | ✅ Pass | 0.683s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.762s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.029s | |
--| Penetration Testing Methodology | ✅ Pass | 2.050s | |
--| Vulnerability Assessment Tools | ✅ Pass | 1.990s | |
--| SQL Injection Attack Type | ✅ Pass | 0.651s | |
--| Penetration Testing Framework | ✅ Pass | 1.320s | |
--| Web Application Security Scanner | ✅ Pass | 1.317s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.060s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 0.972s
++| JSON Response Function | ✅ Pass | 1.179s | |
++| Search Query Function | ✅ Pass | 0.216s | |
++| Ask Advice Function | ✅ Pass | 0.211s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.214s | |
++| Basic Context Memory Test | ✅ Pass | 0.279s | |
++| Function Argument Memory Test | ✅ Pass | 0.213s | |
++| Function Response Memory Test | ✅ Pass | 0.228s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 1.207s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.217s | |
++| Penetration Testing Methodology | ✅ Pass | 0.211s | |
++| Vulnerability Assessment Tools | ✅ Pass | 0.240s | |
++| SQL Injection Attack Type | ✅ Pass | 0.213s | |
++| Penetration Testing Framework | ✅ Pass | 1.309s | |
++| Web Application Security Scanner | ✅ Pass | 0.217s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.213s | |
++
++**Summary**: 22/23 (95.65%) successful tests
++
++**Average latency**: 0.423s
---
@@@ -418,38 -418,38 +425,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 1.889s | |
--| Text Transform Uppercase | ✅ Pass | 1.323s | |
--| Count from 1 to 5 | ✅ Pass | 0.900s | |
--| Math Calculation | ✅ Pass | 1.026s | |
--| Basic Echo Function | ✅ Pass | 1.016s | |
--| Streaming Simple Math Streaming | ✅ Pass | 1.920s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.298s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.987s | |
++| Simple Math | ✅ Pass | 1.002s | |
++| Text Transform Uppercase | ✅ Pass | 0.987s | |
++| Count from 1 to 5 | ✅ Pass | 1.064s | |
++| Math Calculation | ✅ Pass | 1.040s | |
++| Basic Echo Function | ✅ Pass | 1.214s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.038s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.502s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.672s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 1.585s | |
--| Search Query Function | ✅ Pass | 1.567s | |
--| Ask Advice Function | ✅ Pass | 1.272s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 0.972s | |
--| Basic Context Memory Test | ✅ Pass | 1.434s | |
--| Function Argument Memory Test | ✅ Pass | 0.764s | |
--| Function Response Memory Test | ✅ Pass | 1.068s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.771s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 0.781s | |
--| Penetration Testing Methodology | ✅ Pass | 3.191s | |
--| Vulnerability Assessment Tools | ✅ Pass | 3.556s | |
--| SQL Injection Attack Type | ✅ Pass | 0.965s | |
--| Penetration Testing Framework | ✅ Pass | 5.216s | |
--| Web Application Security Scanner | ✅ Pass | 3.193s | |
--| Penetration Testing Tool Selection | ✅ Pass | 1.441s | |
++| JSON Response Function | ✅ Pass | 1.275s | |
++| Search Query Function | ✅ Pass | 1.327s | |
++| Ask Advice Function | ✅ Pass | 1.423s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.873s | |
++| Basic Context Memory Test | ✅ Pass | 1.080s | |
++| Function Argument Memory Test | ✅ Pass | 1.052s | |
++| Function Response Memory Test | ✅ Pass | 1.020s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.004s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.048s | |
++| Penetration Testing Methodology | ✅ Pass | 3.898s | |
++| Vulnerability Assessment Tools | ✅ Pass | 4.180s | |
++| SQL Injection Attack Type | ✅ Pass | 1.730s | |
++| Penetration Testing Framework | ✅ Pass | 4.312s | |
++| Web Application Security Scanner | ✅ Pass | 2.572s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.520s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 1.702s
++**Average latency**: 1.732s
---
@@@ -459,38 -459,38 +466,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 0.558s | |
--| Text Transform Uppercase | ✅ Pass | 0.527s | |
--| Count from 1 to 5 | ✅ Pass | 0.651s | |
--| Math Calculation | ✅ Pass | 0.965s | |
--| Basic Echo Function | ✅ Pass | 1.248s | |
--| Streaming Simple Math Streaming | ✅ Pass | 0.919s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.591s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 0.761s | |
++| Simple Math | ✅ Pass | 0.925s | |
++| Text Transform Uppercase | ✅ Pass | 0.924s | |
++| Count from 1 to 5 | ✅ Pass | 1.112s | |
++| Math Calculation | ✅ Pass | 0.917s | |
++| Basic Echo Function | ✅ Pass | 1.052s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.991s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.480s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.999s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 0.869s | |
--| Search Query Function | ✅ Pass | 0.856s | |
--| Ask Advice Function | ✅ Pass | 1.283s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 1.259s | |
--| Basic Context Memory Test | ✅ Pass | 0.981s | |
--| Function Argument Memory Test | ✅ Pass | 0.649s | |
--| Function Response Memory Test | ✅ Pass | 1.007s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 1.940s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 1.070s | |
--| Penetration Testing Methodology | ✅ Pass | 2.822s | |
--| Vulnerability Assessment Tools | ✅ Pass | 2.937s | |
--| SQL Injection Attack Type | ✅ Pass | 0.620s | |
--| Penetration Testing Framework | ✅ Pass | 1.331s | |
--| Web Application Security Scanner | ✅ Pass | 1.018s | |
--| Penetration Testing Tool Selection | ✅ Pass | 0.972s | |
--
--**Summary**: 23/23 (100.00%) successful tests
--
--**Average latency**: 1.124s
++| JSON Response Function | ✅ Pass | 1.082s | |
++| Search Query Function | ❌ Fail | 1.016s | no tool calls found, expected at least 1 |
++| Ask Advice Function | ✅ Pass | 1.419s | |
++| Streaming Search Query Function Streaming | ❌ Fail | 1.374s | no tool calls found, expected at least 1 |
++| Basic Context Memory Test | ✅ Pass | 1.308s | |
++| Function Argument Memory Test | ✅ Pass | 0.864s | |
++| Function Response Memory Test | ✅ Pass | 1.073s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.342s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.194s | |
++| Penetration Testing Methodology | ✅ Pass | 3.036s | |
++| Vulnerability Assessment Tools | ✅ Pass | 3.095s | |
++| SQL Injection Attack Type | ✅ Pass | 1.001s | |
++| Penetration Testing Framework | ✅ Pass | 1.386s | |
++| Web Application Security Scanner | ✅ Pass | 1.717s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.854s | |
++
++**Summary**: 21/23 (91.30%) successful tests
++
++**Average latency**: 1.399s
---
@@@ -500,38 -500,38 +507,38 @@@
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| Simple Math | ✅ Pass | 3.761s | |
--| Text Transform Uppercase | ✅ Pass | 4.980s | |
--| Count from 1 to 5 | ✅ Pass | 6.066s | |
--| Math Calculation | ✅ Pass | 5.125s | |
--| Basic Echo Function | ✅ Pass | 4.145s | |
--| Streaming Simple Math Streaming | ✅ Pass | 4.829s | |
--| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.837s | |
--| Streaming Basic Echo Function Streaming | ✅ Pass | 3.264s | |
++| Simple Math | ✅ Pass | 4.094s | |
++| Text Transform Uppercase | ✅ Pass | 4.964s | |
++| Math Calculation | ✅ Pass | 0.212s | |
++| Count from 1 to 5 | ✅ Pass | 5.474s | |
++| Basic Echo Function | ✅ Pass | 3.864s | |
++| Streaming Simple Math Streaming | ✅ Pass | 4.113s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.273s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.204s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
--| JSON Response Function | ✅ Pass | 2.894s | |
--| Search Query Function | ✅ Pass | 3.922s | |
--| Ask Advice Function | ✅ Pass | 3.311s | |
--| Streaming Search Query Function Streaming | ✅ Pass | 2.728s | |
--| Basic Context Memory Test | ✅ Pass | 4.075s | |
--| Function Argument Memory Test | ✅ Pass | 5.545s | |
--| Function Response Memory Test | ✅ Pass | 10.448s | |
--| Penetration Testing Memory with Tool Call | ✅ Pass | 5.993s | |
--| Cybersecurity Workflow Memory Test | ✅ Pass | 3.695s | |
--| Penetration Testing Methodology | ✅ Pass | 9.881s | |
--| Vulnerability Assessment Tools | ✅ Pass | 10.935s | |
--| Penetration Testing Framework | ✅ Pass | 12.705s | |
--| Web Application Security Scanner | ✅ Pass | 8.109s | |
--| Penetration Testing Tool Selection | ✅ Pass | 2.886s | |
--| SQL Injection Attack Type | ✅ Pass | 86.622s | |
++| JSON Response Function | ✅ Pass | 2.828s | |
++| Search Query Function | ✅ Pass | 2.215s | |
++| Ask Advice Function | ✅ Pass | 3.054s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 3.363s | |
++| Function Argument Memory Test | ✅ Pass | 0.216s | |
++| Basic Context Memory Test | ✅ Pass | 4.736s | |
++| Function Response Memory Test | ✅ Pass | 0.212s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.218s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 5.134s | |
++| Penetration Testing Methodology | ✅ Pass | 11.298s | |
++| Vulnerability Assessment Tools | ✅ Pass | 8.796s | |
++| SQL Injection Attack Type | ✅ Pass | 0.220s | |
++| Penetration Testing Framework | ✅ Pass | 13.884s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.216s | |
++| Web Application Security Scanner | ✅ Pass | 8.616s | |
**Summary**: 23/23 (100.00%) successful tests
--**Average latency**: 9.207s
++**Average latency**: 4.053s
---
diff --cc examples/tests/vllm-mixed.report.md
index 00000000,00000000..f29047a9
new file mode 100644
--- /dev/null
+++ b/examples/tests/vllm-mixed.report.md
@@@ -1,0 -1,0 +1,544 @@@
++# LLM Agent Testing Report
++
++Generated: Tue, 21 Jul 2026 15:31:43 UTC
++
++## Overall Results
++
++| Agent | Model | Reasoning | Success Rate | Average Latency |
++|-------|-------|-----------|--------------|-----------------|
++| simple | Qwen/Qwen3.6-27B-FP8 | false | 23/23 (100.00%) | 0.804s |
++| simple_json | Qwen/Qwen3.6-27B-FP8 | false | 7/7 (100.00%) | 0.773s |
++| primary_agent | DeepSeek-V4-Flash | true | 22/23 (95.65%) | 1.232s |
++| assistant | DeepSeek-V4-Flash | true | 23/23 (100.00%) | 1.231s |
++| generator | DeepSeek-V4-Flash | true | 23/23 (100.00%) | 1.105s |
++| refiner | DeepSeek-V4-Flash | true | 23/23 (100.00%) | 1.070s |
++| adviser | DeepSeek-V4-Flash | true | 23/23 (100.00%) | 0.912s |
++| reflector | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 0.875s |
++| searcher | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 0.898s |
++| enricher | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 0.929s |
++| coder | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 3.311s |
++| installer | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 3.519s |
++| pentester | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 2.947s |
++
++**Total**: 282/283 (99.65%) successful tests
++**Overall average latency**: 1.550s
++
++## Detailed Results
++
++### simple (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.325s | |
++| Text Transform Uppercase | ✅ Pass | 0.389s | |
++| Count from 1 to 5 | ✅ Pass | 0.464s | |
++| Math Calculation | ✅ Pass | 0.360s | |
++| Basic Echo Function | ✅ Pass | 0.664s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.334s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.395s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.662s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.800s | |
++| Search Query Function | ✅ Pass | 0.593s | |
++| Ask Advice Function | ✅ Pass | 0.863s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.589s | |
++| Basic Context Memory Test | ✅ Pass | 0.482s | |
++| Function Argument Memory Test | ✅ Pass | 0.392s | |
++| Function Response Memory Test | ✅ Pass | 0.409s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.202s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.392s | |
++| Penetration Testing Methodology | ✅ Pass | 2.327s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.898s | |
++| SQL Injection Attack Type | ✅ Pass | 0.368s | |
++| Penetration Testing Framework | ✅ Pass | 2.080s | |
++| Web Application Security Scanner | ✅ Pass | 1.688s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.803s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.804s
++
++---
++
++### simple_json (Qwen/Qwen3.6-27B-FP8)
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Person Information JSON | ✅ Pass | 0.488s | |
++| Project Information JSON | ✅ Pass | 0.664s | |
++| User Profile JSON | ✅ Pass | 0.518s | |
++| Vulnerability Report Memory Test | ✅ Pass | 1.415s | |
++| Streaming Person Information JSON Streaming | ✅ Pass | 0.715s | |
++| JSON Array Response Without Schema | ✅ Pass | 1.123s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Structured Output With JSON Schema | structured_output | ✅ Pass | 0.484s | |
++
++**Summary**: 7/7 (100.00%) successful tests
++
++**Average latency**: 0.773s
++
++---
++
++### primary_agent (DeepSeek-V4-Flash)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.718s | |
++| Text Transform Uppercase | ✅ Pass | 0.878s | |
++| Count from 1 to 5 | ✅ Pass | 1.240s | |
++| Math Calculation | ✅ Pass | 0.766s | |
++| Basic Echo Function | ✅ Pass | 0.896s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.952s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.039s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.878s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 1.380s | |
++| Search Query Function | ✅ Pass | 1.391s | |
++| Ask Advice Function | ✅ Pass | 1.405s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.228s | |
++| Basic Context Memory Test | ✅ Pass | 1.470s | |
++| Function Argument Memory Test | ✅ Pass | 0.992s | |
++| Function Response Memory Test | ✅ Pass | 0.809s | |
++| Penetration Testing Memory with Tool Call | ❌ Fail | 1.927s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.175s | |
++| Penetration Testing Methodology | ✅ Pass | 1.148s | |
++| Vulnerability Assessment Tools | ✅ Pass | 2.327s | |
++| SQL Injection Attack Type | ✅ Pass | 2.227s | |
++| Penetration Testing Framework | ✅ Pass | 1.028s | |
++| Web Application Security Scanner | ✅ Pass | 1.359s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.089s | |
++
++**Summary**: 22/23 (95.65%) successful tests
++
++**Average latency**: 1.232s
++
++---
++
++### assistant (DeepSeek-V4-Flash)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.751s | |
++| Text Transform Uppercase | ✅ Pass | 0.844s | |
++| Count from 1 to 5 | ✅ Pass | 1.218s | |
++| Math Calculation | ✅ Pass | 0.758s | |
++| Basic Echo Function | ✅ Pass | 1.046s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.964s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.124s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.855s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 1.338s | |
++| Search Query Function | ✅ Pass | 1.337s | |
++| Ask Advice Function | ✅ Pass | 1.452s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.184s | |
++| Basic Context Memory Test | ✅ Pass | 1.431s | |
++| Function Argument Memory Test | ✅ Pass | 0.893s | |
++| Function Response Memory Test | ✅ Pass | 0.986s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.093s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.954s | |
++| Penetration Testing Methodology | ✅ Pass | 1.423s | |
++| Vulnerability Assessment Tools | ✅ Pass | 2.987s | |
++| SQL Injection Attack Type | ✅ Pass | 1.466s | |
++| Penetration Testing Framework | ✅ Pass | 1.117s | |
++| Web Application Security Scanner | ✅ Pass | 1.026s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.043s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 1.231s
++
++---
++
++### generator (DeepSeek-V4-Flash)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.751s | |
++| Text Transform Uppercase | ✅ Pass | 0.892s | |
++| Count from 1 to 5 | ✅ Pass | 1.239s | |
++| Math Calculation | ✅ Pass | 0.529s | |
++| Basic Echo Function | ✅ Pass | 0.931s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.930s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.001s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.803s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 1.227s | |
++| Search Query Function | ✅ Pass | 1.143s | |
++| Ask Advice Function | ✅ Pass | 1.388s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.190s | |
++| Basic Context Memory Test | ✅ Pass | 1.301s | |
++| Function Argument Memory Test | ✅ Pass | 0.814s | |
++| Function Response Memory Test | ✅ Pass | 0.751s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.916s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.925s | |
++| Penetration Testing Methodology | ✅ Pass | 1.189s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.771s | |
++| SQL Injection Attack Type | ✅ Pass | 1.098s | |
++| Penetration Testing Framework | ✅ Pass | 1.380s | |
++| Web Application Security Scanner | ✅ Pass | 0.901s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.345s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 1.105s
++
++---
++
++### refiner (DeepSeek-V4-Flash)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.751s | |
++| Text Transform Uppercase | ✅ Pass | 0.830s | |
++| Count from 1 to 5 | ✅ Pass | 1.033s | |
++| Math Calculation | ✅ Pass | 0.712s | |
++| Basic Echo Function | ✅ Pass | 0.933s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.859s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.916s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.925s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 1.245s | |
++| Search Query Function | ✅ Pass | 1.121s | |
++| Ask Advice Function | ✅ Pass | 1.264s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.069s | |
++| Basic Context Memory Test | ✅ Pass | 1.078s | |
++| Function Argument Memory Test | ✅ Pass | 0.742s | |
++| Function Response Memory Test | ✅ Pass | 0.770s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.902s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.831s | |
++| Penetration Testing Methodology | ✅ Pass | 1.419s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.832s | |
++| SQL Injection Attack Type | ✅ Pass | 0.764s | |
++| Penetration Testing Framework | ✅ Pass | 1.384s | |
++| Web Application Security Scanner | ✅ Pass | 0.969s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.244s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 1.070s
++
++---
++
++### adviser (DeepSeek-V4-Flash)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.679s | |
++| Text Transform Uppercase | ✅ Pass | 0.630s | |
++| Count from 1 to 5 | ✅ Pass | 0.857s | |
++| Math Calculation | ✅ Pass | 0.643s | |
++| Basic Echo Function | ✅ Pass | 0.729s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.544s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.801s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.858s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 1.020s | |
++| Search Query Function | ✅ Pass | 1.030s | |
++| Ask Advice Function | ✅ Pass | 1.093s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.834s | |
++| Basic Context Memory Test | ✅ Pass | 0.950s | |
++| Function Argument Memory Test | ✅ Pass | 0.713s | |
++| Function Response Memory Test | ✅ Pass | 0.678s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.634s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.687s | |
++| Penetration Testing Methodology | ✅ Pass | 1.078s | |
++| Vulnerability Assessment Tools | ✅ Pass | 2.025s | |
++| SQL Injection Attack Type | ✅ Pass | 0.666s | |
++| Penetration Testing Framework | ✅ Pass | 1.095s | |
++| Web Application Security Scanner | ✅ Pass | 0.852s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.876s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.912s
++
++---
++
++### reflector (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.325s | |
++| Text Transform Uppercase | ✅ Pass | 0.362s | |
++| Count from 1 to 5 | ✅ Pass | 0.503s | |
++| Math Calculation | ✅ Pass | 0.357s | |
++| Basic Echo Function | ✅ Pass | 0.850s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.337s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.433s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.753s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.869s | |
++| Search Query Function | ✅ Pass | 0.975s | |
++| Ask Advice Function | ✅ Pass | 0.934s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.085s | |
++| Basic Context Memory Test | ✅ Pass | 0.444s | |
++| Function Argument Memory Test | ✅ Pass | 0.441s | |
++| Function Response Memory Test | ✅ Pass | 0.382s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.539s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.513s | |
++| Penetration Testing Methodology | ✅ Pass | 2.543s | |
++| Vulnerability Assessment Tools | ✅ Pass | 2.088s | |
++| SQL Injection Attack Type | ✅ Pass | 0.346s | |
++| Penetration Testing Framework | ✅ Pass | 1.457s | |
++| Web Application Security Scanner | ✅ Pass | 1.632s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.940s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.875s
++
++---
++
++### searcher (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.242s | |
++| Text Transform Uppercase | ✅ Pass | 0.523s | |
++| Count from 1 to 5 | ✅ Pass | 0.511s | |
++| Math Calculation | ✅ Pass | 0.356s | |
++| Basic Echo Function | ✅ Pass | 0.847s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.415s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.474s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.922s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.955s | |
++| Search Query Function | ✅ Pass | 0.924s | |
++| Ask Advice Function | ✅ Pass | 0.947s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.082s | |
++| Basic Context Memory Test | ✅ Pass | 0.439s | |
++| Function Argument Memory Test | ✅ Pass | 0.473s | |
++| Function Response Memory Test | ✅ Pass | 0.406s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.599s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.480s | |
++| Penetration Testing Methodology | ✅ Pass | 2.319s | |
++| Vulnerability Assessment Tools | ✅ Pass | 2.374s | |
++| SQL Injection Attack Type | ✅ Pass | 0.371s | |
++| Penetration Testing Framework | ✅ Pass | 1.580s | |
++| Web Application Security Scanner | ✅ Pass | 1.439s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.956s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.898s
++
++---
++
++### enricher (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.158s | |
++| Text Transform Uppercase | ✅ Pass | 0.572s | |
++| Count from 1 to 5 | ✅ Pass | 0.489s | |
++| Math Calculation | ✅ Pass | 0.355s | |
++| Basic Echo Function | ✅ Pass | 0.844s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.413s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.461s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.749s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.951s | |
++| Search Query Function | ✅ Pass | 0.857s | |
++| Ask Advice Function | ✅ Pass | 0.876s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.056s | |
++| Basic Context Memory Test | ✅ Pass | 0.462s | |
++| Function Argument Memory Test | ✅ Pass | 0.495s | |
++| Function Response Memory Test | ✅ Pass | 0.407s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.392s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.461s | |
++| Penetration Testing Methodology | ✅ Pass | 2.876s | |
++| Vulnerability Assessment Tools | ✅ Pass | 2.880s | |
++| SQL Injection Attack Type | ✅ Pass | 0.361s | |
++| Penetration Testing Framework | ✅ Pass | 1.738s | |
++| Web Application Security Scanner | ✅ Pass | 1.509s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.996s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.929s
++
++---
++
++### coder (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 2.082s | |
++| Text Transform Uppercase | ✅ Pass | 2.219s | |
++| Count from 1 to 5 | ✅ Pass | 2.938s | |
++| Math Calculation | ✅ Pass | 2.041s | |
++| Basic Echo Function | ✅ Pass | 1.559s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.969s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.695s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.369s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 2.001s | |
++| Search Query Function | ✅ Pass | 1.574s | |
++| Ask Advice Function | ✅ Pass | 2.571s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.505s | |
++| Basic Context Memory Test | ✅ Pass | 2.486s | |
++| Function Argument Memory Test | ✅ Pass | 1.336s | |
++| Function Response Memory Test | ✅ Pass | 1.431s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.872s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.931s | |
++| Penetration Testing Methodology | ✅ Pass | 7.704s | |
++| SQL Injection Attack Type | ✅ Pass | 2.423s | |
++| Vulnerability Assessment Tools | ✅ Pass | 10.358s | |
++| Penetration Testing Framework | ✅ Pass | 7.475s | |
++| Web Application Security Scanner | ✅ Pass | 8.320s | |
++| Penetration Testing Tool Selection | ✅ Pass | 5.290s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 3.311s
++
++---
++
++### installer (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 1.807s | |
++| Text Transform Uppercase | ✅ Pass | 4.032s | |
++| Math Calculation | ✅ Pass | 1.855s | |
++| Basic Echo Function | ✅ Pass | 2.188s | |
++| Count from 1 to 5 | ✅ Pass | 6.222s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.774s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.077s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.594s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 2.802s | |
++| Search Query Function | ✅ Pass | 1.382s | |
++| Ask Advice Function | ✅ Pass | 2.326s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.422s | |
++| Basic Context Memory Test | ✅ Pass | 2.410s | |
++| Function Argument Memory Test | ✅ Pass | 1.547s | |
++| Function Response Memory Test | ✅ Pass | 1.870s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.757s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 4.061s | |
++| Penetration Testing Methodology | ✅ Pass | 6.035s | |
++| SQL Injection Attack Type | ✅ Pass | 3.266s | |
++| Vulnerability Assessment Tools | ✅ Pass | 7.942s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.049s | |
++| Web Application Security Scanner | ✅ Pass | 7.245s | |
++| Penetration Testing Framework | ✅ Pass | 12.260s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 3.519s
++
++---
++
++### pentester (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 1.800s | |
++| Text Transform Uppercase | ✅ Pass | 2.974s | |
++| Math Calculation | ✅ Pass | 1.781s | |
++| Count from 1 to 5 | ✅ Pass | 3.352s | |
++| Basic Echo Function | ✅ Pass | 1.788s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.382s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.549s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.524s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 1.694s | |
++| Search Query Function | ✅ Pass | 1.305s | |
++| Ask Advice Function | ✅ Pass | 3.077s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.206s | |
++| Function Argument Memory Test | ✅ Pass | 1.352s | |
++| Basic Context Memory Test | ✅ Pass | 3.221s | |
++| Function Response Memory Test | ✅ Pass | 1.188s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.963s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.485s | |
++| Penetration Testing Methodology | ✅ Pass | 6.373s | |
++| SQL Injection Attack Type | ✅ Pass | 3.415s | |
++| Vulnerability Assessment Tools | ✅ Pass | 6.750s | |
++| Penetration Testing Framework | ✅ Pass | 7.276s | |
++| Web Application Security Scanner | ✅ Pass | 6.321s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.003s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 2.947s
++
++---
++
diff --cc examples/tests/vllm-qwen3.6-27b-fp8.report.md
index 00000000,00000000..cfef8c35
new file mode 100644
--- /dev/null
+++ b/examples/tests/vllm-qwen3.6-27b-fp8.report.md
@@@ -1,0 -1,0 +1,544 @@@
++# LLM Agent Testing Report
++
++Generated: Tue, 21 Jul 2026 15:26:19 UTC
++
++## Overall Results
++
++| Agent | Model | Reasoning | Success Rate | Average Latency |
++|-------|-------|-----------|--------------|-----------------|
++| simple | Qwen/Qwen3.6-27B-FP8 | false | 23/23 (100.00%) | 0.877s |
++| simple_json | Qwen/Qwen3.6-27B-FP8 | false | 7/7 (100.00%) | 0.781s |
++| primary_agent | Qwen/Qwen3.6-27B-FP8 | true | 22/23 (95.65%) | 3.168s |
++| assistant | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 3.532s |
++| generator | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 3.354s |
++| refiner | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 3.250s |
++| adviser | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 5.741s |
++| reflector | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 0.835s |
++| searcher | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 0.875s |
++| enricher | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 0.936s |
++| coder | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 3.361s |
++| installer | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 6.277s |
++| pentester | Qwen/Qwen3.6-27B-FP8 | true | 23/23 (100.00%) | 2.999s |
++
++**Total**: 282/283 (99.65%) successful tests
++**Overall average latency**: 2.881s
++
++## Detailed Results
++
++### simple (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.624s | |
++| Text Transform Uppercase | ✅ Pass | 0.357s | |
++| Count from 1 to 5 | ✅ Pass | 0.476s | |
++| Math Calculation | ✅ Pass | 0.345s | |
++| Basic Echo Function | ✅ Pass | 0.661s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.336s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.418s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.602s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.783s | |
++| Search Query Function | ✅ Pass | 0.637s | |
++| Ask Advice Function | ✅ Pass | 0.924s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.604s | |
++| Basic Context Memory Test | ✅ Pass | 0.490s | |
++| Function Argument Memory Test | ✅ Pass | 0.395s | |
++| Function Response Memory Test | ✅ Pass | 0.369s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.372s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.393s | |
++| Penetration Testing Methodology | ✅ Pass | 2.749s | |
++| Vulnerability Assessment Tools | ✅ Pass | 2.405s | |
++| SQL Injection Attack Type | ✅ Pass | 0.390s | |
++| Penetration Testing Framework | ✅ Pass | 2.549s | |
++| Web Application Security Scanner | ✅ Pass | 1.275s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.998s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.877s
++
++---
++
++### simple_json (Qwen/Qwen3.6-27B-FP8)
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Person Information JSON | ✅ Pass | 0.454s | |
++| Project Information JSON | ✅ Pass | 0.656s | |
++| Vulnerability Report Memory Test | ✅ Pass | 1.517s | |
++| User Profile JSON | ✅ Pass | 0.559s | |
++| Streaming Person Information JSON Streaming | ✅ Pass | 0.678s | |
++| JSON Array Response Without Schema | ✅ Pass | 1.107s | |
++
++#### Capability Tests
++
++| Test | Capability | Result | Latency | Note |
++|------|------------|--------|---------|------|
++| Structured Output With JSON Schema | structured_output | ✅ Pass | 0.495s | |
++
++**Summary**: 7/7 (100.00%) successful tests
++
++**Average latency**: 0.781s
++
++---
++
++### primary_agent (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 2.314s | |
++| Text Transform Uppercase | ✅ Pass | 2.568s | |
++| Count from 1 to 5 | ✅ Pass | 2.903s | |
++| Math Calculation | ✅ Pass | 1.699s | |
++| Basic Echo Function | ✅ Pass | 2.587s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.727s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.023s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.244s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 1.301s | |
++| Search Query Function | ✅ Pass | 0.896s | |
++| Ask Advice Function | ✅ Pass | 2.751s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.396s | |
++| Basic Context Memory Test | ✅ Pass | 3.293s | |
++| Function Argument Memory Test | ✅ Pass | 2.259s | |
++| Function Response Memory Test | ✅ Pass | 1.446s | |
++| Cybersecurity Workflow Memory Test | ❌ Fail | 0.247s | API returned unexpected status code: 502 |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 6.357s | |
++| Penetration Testing Methodology | ✅ Pass | 7.403s | |
++| Vulnerability Assessment Tools | ✅ Pass | 8.894s | |
++| SQL Injection Attack Type | ✅ Pass | 3.029s | |
++| Penetration Testing Framework | ✅ Pass | 7.308s | |
++| Web Application Security Scanner | ✅ Pass | 6.238s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.982s | |
++
++**Summary**: 22/23 (95.65%) successful tests
++
++**Average latency**: 3.168s
++
++---
++
++### assistant (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 2.041s | |
++| Text Transform Uppercase | ✅ Pass | 2.711s | |
++| Count from 1 to 5 | ✅ Pass | 3.078s | |
++| Math Calculation | ✅ Pass | 1.944s | |
++| Basic Echo Function | ✅ Pass | 1.700s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.641s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.790s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.759s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 4.203s | |
++| Search Query Function | ✅ Pass | 1.706s | |
++| Ask Advice Function | ✅ Pass | 2.062s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.181s | |
++| Basic Context Memory Test | ✅ Pass | 3.017s | |
++| Function Argument Memory Test | ✅ Pass | 1.716s | |
++| Function Response Memory Test | ✅ Pass | 2.329s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.284s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 7.690s | |
++| Penetration Testing Methodology | ✅ Pass | 7.370s | |
++| Vulnerability Assessment Tools | ✅ Pass | 7.975s | |
++| SQL Injection Attack Type | ✅ Pass | 3.552s | |
++| Penetration Testing Framework | ✅ Pass | 7.924s | |
++| Web Application Security Scanner | ✅ Pass | 6.412s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.139s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 3.532s
++
++---
++
++### generator (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 1.677s | |
++| Text Transform Uppercase | ✅ Pass | 2.245s | |
++| Count from 1 to 5 | ✅ Pass | 3.411s | |
++| Math Calculation | ✅ Pass | 1.499s | |
++| Basic Echo Function | ✅ Pass | 2.209s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.010s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.073s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.806s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 2.294s | |
++| Search Query Function | ✅ Pass | 2.087s | |
++| Ask Advice Function | ✅ Pass | 2.326s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 2.099s | |
++| Basic Context Memory Test | ✅ Pass | 2.675s | |
++| Function Argument Memory Test | ✅ Pass | 2.940s | |
++| Function Response Memory Test | ✅ Pass | 4.411s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.787s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.556s | |
++| Penetration Testing Methodology | ✅ Pass | 7.681s | |
++| Vulnerability Assessment Tools | ✅ Pass | 9.115s | |
++| SQL Injection Attack Type | ✅ Pass | 3.597s | |
++| Penetration Testing Framework | ✅ Pass | 7.387s | |
++| Web Application Security Scanner | ✅ Pass | 5.811s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.429s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 3.354s
++
++---
++
++### refiner (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 2.045s | |
++| Text Transform Uppercase | ✅ Pass | 2.766s | |
++| Count from 1 to 5 | ✅ Pass | 3.423s | |
++| Math Calculation | ✅ Pass | 1.984s | |
++| Basic Echo Function | ✅ Pass | 2.168s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.710s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.720s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.724s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 2.737s | |
++| Search Query Function | ✅ Pass | 1.641s | |
++| Ask Advice Function | ✅ Pass | 1.517s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.751s | |
++| Basic Context Memory Test | ✅ Pass | 1.921s | |
++| Function Argument Memory Test | ✅ Pass | 3.008s | |
++| Function Response Memory Test | ✅ Pass | 1.607s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.890s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.124s | |
++| Penetration Testing Methodology | ✅ Pass | 6.267s | |
++| Vulnerability Assessment Tools | ✅ Pass | 8.601s | |
++| SQL Injection Attack Type | ✅ Pass | 3.598s | |
++| Penetration Testing Framework | ✅ Pass | 7.453s | |
++| Web Application Security Scanner | ✅ Pass | 5.439s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.652s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 3.250s
++
++---
++
++### adviser (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 1.796s | |
++| Text Transform Uppercase | ✅ Pass | 2.022s | |
++| Count from 1 to 5 | ✅ Pass | 2.724s | |
++| Math Calculation | ✅ Pass | 2.223s | |
++| Basic Echo Function | ✅ Pass | 1.523s | |
++| Streaming Simple Math Streaming | ✅ Pass | 1.985s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.725s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.368s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 1.680s | |
++| Search Query Function | ✅ Pass | 1.935s | |
++| Ask Advice Function | ✅ Pass | 1.873s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.135s | |
++| Basic Context Memory Test | ✅ Pass | 2.274s | |
++| Function Argument Memory Test | ✅ Pass | 1.882s | |
++| Function Response Memory Test | ✅ Pass | 1.363s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 5.059s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.751s | |
++| Penetration Testing Methodology | ✅ Pass | 8.470s | |
++| SQL Injection Attack Type | ✅ Pass | 3.770s | |
++| Penetration Testing Framework | ✅ Pass | 7.895s | |
++| Web Application Security Scanner | ✅ Pass | 4.795s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.278s | |
++| Vulnerability Assessment Tools | ✅ Pass | 69.509s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 5.741s
++
++---
++
++### reflector (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.624s | |
++| Text Transform Uppercase | ✅ Pass | 0.518s | |
++| Count from 1 to 5 | ✅ Pass | 0.484s | |
++| Math Calculation | ✅ Pass | 0.394s | |
++| Basic Echo Function | ✅ Pass | 0.672s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.380s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.453s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.751s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.903s | |
++| Search Query Function | ✅ Pass | 0.751s | |
++| Ask Advice Function | ✅ Pass | 0.954s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.844s | |
++| Basic Context Memory Test | ✅ Pass | 0.489s | |
++| Function Argument Memory Test | ✅ Pass | 0.435s | |
++| Function Response Memory Test | ✅ Pass | 0.405s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.411s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.476s | |
++| Penetration Testing Methodology | ✅ Pass | 2.687s | |
++| Vulnerability Assessment Tools | ✅ Pass | 1.877s | |
++| SQL Injection Attack Type | ✅ Pass | 0.364s | |
++| Penetration Testing Framework | ✅ Pass | 1.148s | |
++| Web Application Security Scanner | ✅ Pass | 1.275s | |
++| Penetration Testing Tool Selection | ✅ Pass | 0.904s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.835s
++
++---
++
++### searcher (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.259s | |
++| Text Transform Uppercase | ✅ Pass | 0.397s | |
++| Count from 1 to 5 | ✅ Pass | 0.503s | |
++| Math Calculation | ✅ Pass | 0.367s | |
++| Basic Echo Function | ✅ Pass | 0.772s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.366s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.453s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.690s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.852s | |
++| Search Query Function | ✅ Pass | 0.829s | |
++| Ask Advice Function | ✅ Pass | 0.950s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.866s | |
++| Basic Context Memory Test | ✅ Pass | 0.521s | |
++| Function Argument Memory Test | ✅ Pass | 0.431s | |
++| Function Response Memory Test | ✅ Pass | 0.387s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.551s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.657s | |
++| Penetration Testing Methodology | ✅ Pass | 2.595s | |
++| Vulnerability Assessment Tools | ✅ Pass | 2.407s | |
++| SQL Injection Attack Type | ✅ Pass | 0.366s | |
++| Penetration Testing Framework | ✅ Pass | 1.379s | |
++| Web Application Security Scanner | ✅ Pass | 1.466s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.048s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.875s
++
++---
++
++### enricher (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 0.166s | |
++| Text Transform Uppercase | ✅ Pass | 0.501s | |
++| Count from 1 to 5 | ✅ Pass | 0.497s | |
++| Math Calculation | ✅ Pass | 0.366s | |
++| Basic Echo Function | ✅ Pass | 0.734s | |
++| Streaming Simple Math Streaming | ✅ Pass | 0.352s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.461s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 0.837s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 0.827s | |
++| Search Query Function | ✅ Pass | 0.790s | |
++| Ask Advice Function | ✅ Pass | 0.995s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 0.740s | |
++| Basic Context Memory Test | ✅ Pass | 0.533s | |
++| Function Argument Memory Test | ✅ Pass | 0.451s | |
++| Function Response Memory Test | ✅ Pass | 0.409s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 1.504s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 0.580s | |
++| Penetration Testing Methodology | ✅ Pass | 2.463s | |
++| Vulnerability Assessment Tools | ✅ Pass | 2.674s | |
++| SQL Injection Attack Type | ✅ Pass | 0.346s | |
++| Penetration Testing Framework | ✅ Pass | 2.681s | |
++| Web Application Security Scanner | ✅ Pass | 1.613s | |
++| Penetration Testing Tool Selection | ✅ Pass | 1.001s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 0.936s
++
++---
++
++### coder (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 2.135s | |
++| Text Transform Uppercase | ✅ Pass | 2.539s | |
++| Count from 1 to 5 | ✅ Pass | 3.738s | |
++| Math Calculation | ✅ Pass | 1.360s | |
++| Basic Echo Function | ✅ Pass | 2.107s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.081s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.350s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 4.815s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 1.649s | |
++| Search Query Function | ✅ Pass | 1.346s | |
++| Ask Advice Function | ✅ Pass | 2.244s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.088s | |
++| Basic Context Memory Test | ✅ Pass | 3.039s | |
++| Function Argument Memory Test | ✅ Pass | 1.380s | |
++| Function Response Memory Test | ✅ Pass | 3.934s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 2.733s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 2.180s | |
++| Penetration Testing Methodology | ✅ Pass | 6.927s | |
++| Vulnerability Assessment Tools | ✅ Pass | 8.605s | |
++| SQL Injection Attack Type | ✅ Pass | 3.516s | |
++| Penetration Testing Framework | ✅ Pass | 7.560s | |
++| Web Application Security Scanner | ✅ Pass | 4.755s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.213s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 3.361s
++
++---
++
++### installer (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 2.076s | |
++| Text Transform Uppercase | ✅ Pass | 2.101s | |
++| Count from 1 to 5 | ✅ Pass | 4.749s | |
++| Math Calculation | ✅ Pass | 1.669s | |
++| Basic Echo Function | ✅ Pass | 1.741s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.022s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.826s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 2.107s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 1.669s | |
++| Search Query Function | ✅ Pass | 1.274s | |
++| Ask Advice Function | ✅ Pass | 2.619s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.557s | |
++| Basic Context Memory Test | ✅ Pass | 2.334s | |
++| Function Argument Memory Test | ✅ Pass | 1.931s | |
++| Function Response Memory Test | ✅ Pass | 1.398s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 3.201s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.667s | |
++| Penetration Testing Methodology | ✅ Pass | 7.674s | |
++| SQL Injection Attack Type | ✅ Pass | 5.422s | |
++| Penetration Testing Framework | ✅ Pass | 11.689s | |
++| Web Application Security Scanner | ✅ Pass | 5.730s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.548s | |
++| Vulnerability Assessment Tools | ✅ Pass | 74.361s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 6.277s
++
++---
++
++### pentester (Qwen/Qwen3.6-27B-FP8)
++
++#### Basic Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| Simple Math | ✅ Pass | 1.616s | |
++| Text Transform Uppercase | ✅ Pass | 2.141s | |
++| Count from 1 to 5 | ✅ Pass | 2.635s | |
++| Math Calculation | ✅ Pass | 1.734s | |
++| Basic Echo Function | ✅ Pass | 3.264s | |
++| Streaming Simple Math Streaming | ✅ Pass | 2.129s | |
++| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.503s | |
++| Streaming Basic Echo Function Streaming | ✅ Pass | 1.289s | |
++
++#### Advanced Tests
++
++| Test | Result | Latency | Error |
++|------|--------|---------|-------|
++| JSON Response Function | ✅ Pass | 2.096s | |
++| Search Query Function | ✅ Pass | 1.516s | |
++| Ask Advice Function | ✅ Pass | 1.979s | |
++| Streaming Search Query Function Streaming | ✅ Pass | 1.287s | |
++| Basic Context Memory Test | ✅ Pass | 2.120s | |
++| Function Argument Memory Test | ✅ Pass | 2.736s | |
++| Function Response Memory Test | ✅ Pass | 1.932s | |
++| Cybersecurity Workflow Memory Test | ✅ Pass | 1.794s | |
++| Penetration Testing Memory with Tool Call | ✅ Pass | 5.208s | |
++| Penetration Testing Methodology | ✅ Pass | 6.298s | |
++| Vulnerability Assessment Tools | ✅ Pass | 7.496s | |
++| SQL Injection Attack Type | ✅ Pass | 3.620s | |
++| Penetration Testing Framework | ✅ Pass | 6.025s | |
++| Web Application Security Scanner | ✅ Pass | 5.315s | |
++| Penetration Testing Tool Selection | ✅ Pass | 2.221s | |
++
++**Summary**: 23/23 (100.00%) successful tests
++
++**Average latency**: 2.999s
++
++---
++