Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

feat(providers): Update model configurations for various LLM providers: qwen, kimi, glm, deepseek, gemini

Dmitry Ng643ff7d21855
diff --git a/README.md b/README.md
index 99ee92dd..fc5ca587 100644
--- a/README.md
+++ b/README.md
@@ -1772,62 +1772,57 @@ PROXY_URL=http://your-proxy:8080
#### Supported Models
-PentAGI supports 13 Gemini models with tool calling, streaming, thinking modes, and context caching. Models marked with `*` are used in default configuration.
+PentAGI supports 9 Gemini models with tool calling, streaming, thinking modes, and context caching. Models marked with `*` are used in default configuration.
-**Gemini 3.1 Series - Latest Flagship (February 2026)**
+**Gemini 3.5 Series - Latest Stable Flash (May 2026)**
| Model ID | Thinking | Context | Price (Input/Output/Cache) | Use Case |
| ------------------------------------- | -------- | ------- | -------------------------- | ----------------------------------------------- |
-| `gemini-3.1-pro-preview`* | ✅ | 1M | $2.00/$12.00/$0.20 | Latest flagship with refined thinking, improved token efficiency, optimized for software engineering and agentic workflows |
-| `gemini-3.1-pro-preview-customtools` | ✅ | 1M | $2.00/$12.00/$0.20 | Custom tools endpoint optimized for bash and custom tools (view_file, search_code) prioritization |
-| `gemini-3.1-flash-lite-preview`* | ✅ | 1M | $0.25/$1.50/$0.03 | Most cost-efficient with fastest performance for high-volume agentic tasks and low-latency applications |
+| `gemini-3.5-flash`* | ✅ | 1M | $1.50/$9.00/$0.15 | Most intelligent Flash model with sustained frontier performance on agentic and coding tasks, superior search and grounding |
-**Gemini 3 Series (⚠️ gemini-3-pro-preview DEPRECATED - Shutdown March 9, 2026)**
+**Gemini 3.1 Series - Stable Flash-Lite + Pro Preview (Feb-May 2026)**
| Model ID | Thinking | Context | Price (Input/Output/Cache) | Use Case |
| ------------------------------------- | -------- | ------- | -------------------------- | ----------------------------------------------- |
-| `gemini-3-pro-preview` | ✅ | 1M | $2.00/$12.00/$0.20 | ⚠️ DEPRECATED - Migrate to gemini-3.1-pro-preview before March 9, 2026 |
-| `gemini-3-flash-preview`* | ✅ | 1M | $0.50/$3.00/$0.05 | Frontier intelligence with superior search grounding, high-throughput security scanning |
+| `gemini-3.1-pro-preview`* | ✅ | 1M | $2.00/$12.00/$0.20 | Latest flagship with refined thinking, improved token efficiency, optimized for software engineering and agentic workflows |
+| `gemini-3.1-pro-preview-customtools` | ✅ | 1M | $2.00/$12.00/$0.20 | Custom tools endpoint optimized for bash and custom tools (view_file, search_code) prioritization |
+| `gemini-3.1-flash-lite`* | ✅ | 1M | $0.25/$1.50/$0.025 | Most cost-efficient stable multimodal model, frontier-class performance for high-volume agentic tasks and low-latency applications |
-**Gemini 2.5 Series - Advanced Thinking Models**
+**Gemini 2.5 Series - Advanced Thinking Models (active until October 16, 2026)**
| Model ID | Thinking | Context | Price (Input/Output/Cache) | Use Case |
| ---------------------------------------- | -------- | ------- | -------------------------- | ----------------------------------------------- |
-| `gemini-2.5-pro` | ✅ | 1M | $1.25/$10.00/$0.13 | State-of-the-art for complex coding and reasoning, sophisticated threat modeling |
+| `gemini-2.5-pro` | ✅ | 1M | $1.25/$10.00/$0.125 | State-of-the-art for complex coding and reasoning, sophisticated threat modeling |
| `gemini-2.5-flash` | ✅ | 1M | $0.30/$2.50/$0.03 | First hybrid reasoning model with thinking budgets, best price-performance for large-scale assessments |
| `gemini-2.5-flash-lite` | ✅ | 1M | $0.10/$0.40/$0.01 | Smallest and most cost-effective for at-scale usage, high-throughput scanning |
-| `gemini-2.5-flash-lite-preview-09-2025` | ✅ | 1M | $0.10/$0.40/$0.01 | Latest preview optimized for cost-efficiency, high throughput, and quality |
-
-**Gemini 2.0 Series - Balanced Multimodal for Agents**
-
-| Model ID | Thinking | Context | Price (Input/Output/Cache) | Use Case |
-| ------------------------------------- | -------- | ------- | -------------------------- | ----------------------------------------------- |
-| `gemini-2.0-flash` | ❌ | 1M | $0.10/$0.40/$0.03 | Balanced multimodal built for agents era, diverse security tasks and real-time monitoring |
-| `gemini-2.0-flash-lite` | ❌ | 1M | $0.08/$0.30/$0.00 | Lightweight for continuous monitoring, basic scanning, automated alert processing |
-**Specialized Open-Source Models (Free)**
+**Gemma 4 Open-Source Models (Apache 2.0, Free Tier)**
| Model ID | Thinking | Context | Price (Input/Output/Cache) | Use Case |
| ------------------------------------- | -------- | ------- | -------------------------- | ----------------------------------------------- |
-| `gemma-3-27b-it` | ❌ | 128K | Free/Free/Free | Open-source from Gemini tech, on-premises security operations, privacy-sensitive testing |
-| `gemma-3n-4b-it` | ❌ | 128K | Free/Free/Free | Efficient for edge devices (mobile/laptops/tablets), offline vulnerability scanning |
+| `gemma-4-31b-it` | ✅ | 256K | Free/Free/Free | Largest open-source Gemma 4 dense model (~31B params), multimodal text+image, 140+ languages, on-premises security operations |
+| `gemma-4-26b-a4b-it` | ✅ | 256K | Free/Free/Free | MoE architecture (~26B total / ~3.8B active params), highly efficient inference on consumer GPUs for on-premises high-throughput scanning |
**Prices**: Per 1M tokens (Standard Paid tier). Context window is input token limit.
-> [!WARNING]
-> **Gemini 3 Pro Preview Deprecation**
+> [!NOTE]
+> **Gemini 2.5 Series Shutdown**
>
-> `gemini-3-pro-preview` will be **shut down on March 9, 2026**. Migrate to `gemini-3.1-pro-preview` to avoid service disruption. The new model offers:
+> `gemini-2.5-pro`, `gemini-2.5-flash`, and `gemini-2.5-flash-lite` will be **shut down on October 16, 2026**. Recommended migrations:
>
-> - Refined performance and reliability
-> - Improved thinking and token efficiency
-> - Better grounded, factually consistent responses
-> - Enhanced software engineering behavior
+> - `gemini-2.5-pro` → `gemini-3.1-pro-preview` (same $2.00 input pricing tier)
+> - `gemini-2.5-flash` → `gemini-3.5-flash` (improved frontier capabilities)
+> - `gemini-2.5-flash-lite` → `gemini-3.1-flash-lite` (same $0.25 input pricing)
+
+**Default Model Assignments (config.yml)**:
+- **`gemini-3.1-pro-preview`** - `primary_agent`, `assistant`, `generator`, `refiner`, `adviser`, `coder`, `pentester`
+- **`gemini-3.5-flash`** - `reflector`, `searcher`, `enricher`, `installer`
+- **`gemini-3.1-flash-lite`** - `simple`, `simple_json`
**Key Features**:
-- **Extended Thinking**: Step-by-step reasoning for complex security analysis (all Gemini 3.x and 2.5 series)
-- **Context Caching**: Significant cost reduction on repeated context (10-90% of input price)
-- **Ultra-Long Context**: 1M tokens for comprehensive codebase analysis and documentation review
+- **Extended Thinking**: Step-by-step reasoning for complex security analysis (all Gemini 3.x, 2.5 series, and Gemma 4 with toggleable thinking)
+- **Context Caching**: Significant cost reduction on repeated context (10% of input price for most models)
+- **Ultra-Long Context**: 1M tokens for Gemini chat models, 256K tokens for Gemma 4 open-source models
- **Multimodal Support**: Text, image, video, audio, and PDF processing for comprehensive assessments
- **Tool Calling**: Seamless integration with 20+ pentesting tools via function calling
- **Streaming**: Real-time response streaming for interactive security workflows
@@ -1835,6 +1830,7 @@ PentAGI supports 13 Gemini models with tool calling, streaming, thinking modes,
- **Search Grounding**: Google Search integration for threat intelligence and CVE research
- **File Search**: Document retrieval and RAG capabilities for knowledge-based assessments
- **Batch API**: 50% cost reduction for non-real-time batch processing
+- **Custom Tools Endpoint**: Dedicated `gemini-3.1-pro-preview-customtools` route for tool-heavy agentic workflows that prefer registered tools over bash
**Reasoning Effort Levels**:
- **High**: Maximum thinking depth for complex multi-step analysis (generator)
@@ -1983,28 +1979,51 @@ DEEPSEEK_PROVIDER=deepseek # Adds prefix to model names (deepseek/deepseek-v4-f
#### Supported Models
-PentAGI supports 2 DeepSeek V4 models with tool calling, streaming, thinking modes, and context caching. Models marked with `*` are used in default configuration.
+PentAGI supports 2 DeepSeek V4 models with tool calling, streaming, hybrid thinking/non-thinking modes, and context caching. Both models support thinking mode by default and can be switched to non-thinking mode via `extra_body`. Models marked with `*` are used in default configuration.
+
+| Model ID | Thinking | Max Output | Context | Price (Input/Output/Cache) | Use Case |
+| --------------------- | -------- | ---------- | ------- | -------------------------- | ---------------------------------------------------- |
+| `deepseek-v4-flash`* | ✅ hybrid | 384K | 1M | $0.14/$0.28/$0.0028 | Utility agents, general dialogue, fast tool calling |
+| `deepseek-v4-pro`* | ✅ hybrid | 384K | 1M | $1.74/$3.48/$0.0145 | Advanced reasoning, complex logic, security analysis |
-| Model ID | Thinking | Context | Price (Input/Output/Cache) | Use Case |
-| --------------------- | -------- | ------- | -------------------------- | ---------------------------------------------------- |
-| `deepseek-v4-flash`* | ❌ | 1M | $0.14/$0.28/$0.0028 | General dialogue, code generation, tool calling |
-| `deepseek-v4-pro`* | ✅ | 1M | $0.435/$0.87/$0.003625 | Advanced reasoning, complex logic, security analysis |
+**Prices**: Per 1M tokens. Cache pricing applies to prompt tokens served from cache (input cache hit, reduced to 1/10 of launch price since 2026-04-26). Both models support hybrid thinking — `thinking` mode is enabled by default; pass `extra_body.thinking.type: disabled` to switch to non-thinking mode for faster/cheaper responses.
-**Prices**: Per 1M tokens. Cache pricing applies to prompt tokens served from cache and is heavily discounted versus input price. Models with thinking support include reinforcement learning chain-of-thought reasoning.
+> **Pricing Note (deepseek-v4-pro)**: The 75% promotional discount on `deepseek-v4-pro` officially ended on 2026-05-31 15:59 UTC. The prices above reflect the standard post-promotional pricing. If you have legacy configurations using the discounted prices ($0.435/$0.87/$0.003625), update them to the current rates for accurate cost tracking.
> The legacy model names `deepseek-chat` and `deepseek-reasoner` are scheduled
> for deprecation by DeepSeek on 2026-07-24. Existing user configurations
> referencing the legacy names continue to work until then; the defaults above
-> use the current V4 names.
+> use the current V4 names. `deepseek-chat` maps to `deepseek-v4-flash`
+> non-thinking mode; `deepseek-reasoner` maps to `deepseek-v4-flash` thinking mode.
+
+**Default Agent Configuration**:
+
+Strategy: prefer `deepseek-v4-flash` (12x cheaper input, 12x cheaper output) as the workhorse for utility/lightweight agents; reserve `deepseek-v4-pro` for complex multi-step reasoning. The `installer` agent runs on Flash with thinking enabled because environment setup tasks (shell commands, config edits) rarely require pro-level reasoning. Run A/B tests on your own workloads before promoting more agents to Pro.
+
+| Agent Role | Default Model | Thinking | Reasoning Effort | Max Output | Temperature | Top P |
+| ------------------------------------------- | -------------------- | -------- | ---------------- | ---------- | ----------- | ----- |
+| Generator / Refiner | `deepseek-v4-pro` | Enabled | High | 32768 | (auto) | (auto) |
+| Coder | `deepseek-v4-pro` | Enabled | High | 20480 | (auto) | (auto) |
+| Primary Agent / Assistant / Pentester | `deepseek-v4-pro` | Enabled | High | 16384 | (auto) | (auto) |
+| Adviser (mentor/planner) | `deepseek-v4-pro` | Enabled | High | 8192 | (auto) | (auto) |
+| Installer | `deepseek-v4-flash` | Enabled | High | 12288 | (auto) | (auto) |
+| Reflector / Searcher / Enricher | `deepseek-v4-flash` | Disabled | — | 4096 | 0.5 | 0.9 |
+| Simple / Simple JSON | `deepseek-v4-flash` | Disabled | — | 2048 | 0.3 | 0.9 |
+
+> **Note**: When thinking mode is enabled, DeepSeek silently ignores `temperature`, `top_p`, `presence_penalty`, and `frequency_penalty`. The langchaingo client automatically nullifies `temperature`/`top_p` when `reasoning_effort` is set, so they appear as "(auto)" in the table above. All thinking-enabled agents also explicitly pass `extra_body.thinking.type: enabled` as defensive coding against future provider default changes.
**Key Features**:
-- **Automatic Prompt Caching**: Significant cost reduction on repeated context via cache-hit pricing far below input price
-- **Extended Thinking**: Reinforcement learning CoT for complex security analysis (deepseek-v4-pro)
+- **Hybrid Thinking Modes**: Switch between thinking (deep reasoning) and non-thinking (fast) modes via `extra_body.thinking.type`
+- **Automatic Prompt Caching**: Significant cost reduction on repeated context via cache-hit pricing (1/10 of launch price)
+- **Extended Thinking**: Reinforcement learning CoT for complex security analysis (both V4 models)
- **Strong Coding**: Optimized for code generation and exploit development
+- **Long Context**: 1M token context window with up to 384K output tokens
- **Tool Calling**: Seamless integration with 20+ pentesting tools via function calling
- **Streaming**: Real-time response streaming for interactive workflows
- **Multilingual**: Strong Chinese and English support
-- **Additional Features**: JSON Output, Chat Prefix Completion, FIM (Fill-in-the-Middle) Completion
+- **Additional Features**: JSON Output, Chat Prefix Completion (beta), FIM/Fill-in-the-Middle Completion (non-thinking mode only)
+
+**Concurrency Limits**: `deepseek-v4-flash`: 2500 concurrent requests; `deepseek-v4-pro`: 500 concurrent requests.
**LiteLLM Integration**: Set `DEEPSEEK_PROVIDER=deepseek` to enable model name prefixing when using default PentAGI configurations with LiteLLM proxy. Leave empty for direct API usage.
@@ -2039,70 +2058,78 @@ GLM_PROVIDER=zai # Adds prefix to model names (zai/glm-4) for LiteLLM
#### Supported Models
-PentAGI supports 12 GLM models with tool calling, streaming, thinking modes, and prompt caching. Models marked with `*` are used in default configuration.
+PentAGI supports 13 GLM models with tool calling, streaming, hybrid thinking modes, and prompt caching. Models marked with `*` are used in default configuration. Thinking is controlled via `extra_body.thinking.type` ("enabled"/"disabled"); unlike Kimi, GLM is permissive about temperature in either mode.
-**GLM-5 Series - Flagship MoE (744B/40B active)**
+**GLM-5.x Series - Latest Generation (200K context, 128K max output)**
-| Model ID | Thinking | Context | Max Output | Price (Input/Output/Cache) | Use Case |
-| ----------------------- | ------------- | ------- | ---------- | -------------------------- | ----------------------------------------------- |
-| `glm-5`* | ✅ Forced | 200K | 128K | $1.00/$3.20/$0.20 | Flagship agentic engineering, complex multi-stage tasks |
-| `glm-5-code`† | ✅ Forced | 200K | 128K | $1.20/$5.00/$0.30 | Code-specialized, exploit development (requires Coding Plan) |
+| Model ID | Thinking | Context | Max Output | Price (Input/Output/Cache) | Use Case |
+| ---------------- | -------- | ------- | ---------- | -------------------------- | ------------------------------------------------------------------- |
+| `glm-5.1`* | ✅ Hybrid | 200K | 128K | $1.40/$4.40/$0.26 | Newest flagship: 8h sustained autonomous execution, Claude Opus 4.6-aligned (generator/refiner/adviser/coder/pentester default) |
+| `glm-5` | ✅ Hybrid | 200K | 128K | $1.00/$3.20/$0.20 | Foundation for Agentic Engineering, MoE 744B/40B active, Claude Opus 4.5-level coding |
+| `glm-5-turbo`* | ✅ Hybrid | 200K | 128K | $1.20/$4.00/$0.24 | OpenClaw-native: optimized for tool invocation, persistent tasks, long-chain execution (primary_agent/assistant default) |
**GLM-4.7 Series - Premium with Interleaved Thinking**
-| Model ID | Thinking | Context | Max Output | Price (Input/Output/Cache) | Use Case |
-| ----------------------- | ------------- | ------- | ---------- | -------------------------- | ----------------------------------------------- |
-| `glm-4.7`* | ✅ Forced | 200K | 128K | $0.60/$2.20/$0.11 | Premium with thinking before each response/tool call |
-| `glm-4.7-flashx`* | ✅ Hybrid | 200K | 128K | $0.07/$0.40/$0.01 | High-speed with priority GPU, best price/performance |
-| `glm-4.7-flash` | ✅ Hybrid | 200K | 128K | Free/Free/Free | Free ~30B SOTA model, 1 concurrent request |
+| Model ID | Thinking | Context | Max Output | Price (Input/Output/Cache) | Use Case |
+| ----------------- | -------- | ------- | ---------- | -------------------------- | --------------------------------------------------- |
+| `glm-4.7` | ✅ Hybrid | 200K | 128K | $0.60/$2.20/$0.11 | Enhanced programming, stable multi-step reasoning |
+| `glm-4.7-flashx` | ✅ Hybrid | 200K | 128K | $0.07/$0.40/$0.01 | Ultra-cheap with priority GPU, but lower RPM limits (avoid for high-frequency use) |
+| `glm-4.7-flash` | ✅ Hybrid | 200K | 128K | Free/Free/Free | Free ~30B SOTA model, 1 concurrent request |
**GLM-4.6 Series - Balanced with Auto-Thinking**
-| Model ID | Thinking | Context | Max Output | Price (Input/Output/Cache) | Use Case |
-| ----------------------- | ------------- | ------- | ---------- | -------------------------- | ----------------------------------------------- |
-| `glm-4.6` | ✅ Auto | 200K | 128K | $0.60/$2.20/$0.11 | Balanced, streaming tool calls, 30% token efficient |
+| Model ID | Thinking | Context | Max Output | Price (Input/Output/Cache) | Use Case |
+| --------- | -------- | ------- | ---------- | -------------------------- | ------------------------------------------------- |
+| `glm-4.6` | ✅ Auto | 200K | 128K | $0.60/$2.20/$0.11 | Balanced, streaming tool calls, token-efficient |
**GLM-4.5 Series - Unified Reasoning/Coding/Agents**
-| Model ID | Thinking | Context | Max Output | Price (Input/Output/Cache) | Use Case |
-| ----------------------- | ------------- | ------- | ---------- | -------------------------- | ----------------------------------------------- |
-| `glm-4.5` | ✅ Auto | 128K | 96K | $0.60/$2.20/$0.11 | Unified model, MoE 355B/32B active |
-| `glm-4.5-x` | ✅ Auto | 128K | 96K | $2.20/$8.90/$0.45 | Ultra-fast premium, lowest latency |
-| `glm-4.5-air`* | ✅ Auto | 128K | 96K | $0.20/$1.10/$0.03 | Cost-effective, MoE 106B/12B, best price/quality |
-| `glm-4.5-airx` | ✅ Auto | 128K | 96K | $1.10/$4.50/$0.22 | Accelerated Air with priority GPU |
-| `glm-4.5-flash` | ✅ Auto | 128K | 96K | Free/Free/Free | Free with reasoning/coding/agents support |
+| Model ID | Thinking | Context | Max Output | Price (Input/Output/Cache) | Use Case |
+| ---------------- | -------- | ------- | ---------- | -------------------------- | ------------------------------------------------- |
+| `glm-4.5` | ✅ Auto | 128K | 96K | $0.60/$2.20/$0.11 | Unified, MoE 355B/32B active |
+| `glm-4.5-x` | ✅ Auto | 128K | 96K | $2.20/$8.90/$0.45 | Ultra-fast premium, lowest latency |
+| `glm-4.5-air`* | ✅ Auto | 128K | 96K | $0.20/$1.10/$0.03 | Cost-effective MoE 106B/12B (simple/simple_json/reflector/searcher/enricher/installer default) |
+| `glm-4.5-airx` | ✅ Auto | 128K | 96K | $1.10/$4.50/$0.22 | Accelerated Air with priority GPU |
+| `glm-4.5-flash` | ✅ Auto | 128K | 96K | Free/Free/Free | Free with reasoning/coding/agents support |
**GLM-4 Legacy - Dense Architecture**
-| Model ID | Thinking | Context | Max Output | Price (Input/Output/Cache) | Use Case |
-| ----------------------- | ------------- | ------- | ---------- | -------------------------- | ----------------------------------------------- |
-| `glm-4-32b-0414-128k` | ❌ | 128K | 16K | $0.10/$0.10/$0.00 | Ultra-budget dense 32B, high-volume parsing |
+| Model ID | Thinking | Context | Max Output | Price (Input/Output) | Use Case |
+| --------------------- | -------- | ------- | ---------- | -------------------- | --------------------------------------------- |
+| `glm-4-32b-0414-128k` | ❌ | 128K | 16K | $0.10/$0.10 | Ultra-budget dense 32B, parsing without reasoning |
-**Prices**: Per 1M tokens. Cache pricing is for prompt caching. † Model requires **Coding Plan subscription**.
+**Prices**: Per 1M tokens. Cache pricing is for prompt cache hit; cache storage is currently free per Z.AI promotion. GLM-4-32B has no cache support.
-> [!WARNING]
-> **Coding Plan Requirement**
->
-> The `glm-5-code` model requires an active **Coding Plan subscription**. Attempting to use this model without the subscription will result in:
->
-> ```
-> API returned unexpected status code: 403: You do not have permission to access glm-5-code
-> ```
->
-> For code-specialized tasks without Coding Plan, use `glm-5` (general flagship) or `glm-4.7` (premium with interleaved thinking) instead.
+**Default Agent Configuration**:
+
+Strategy: `glm-5.1` (newest flagship, $1.40 input) for critical reasoning, `glm-5-turbo` (OpenClaw-native, agent-optimized) for orchestration, `glm-4.5-air` (cheap MoE with hybrid thinking and reliable RPM) for all utility/installer agents. `glm-4.7-flashx` is avoided as default due to lower RPM limits causing frequent 429 errors at high frequency.
+
+| Agent Role | Default Model | Thinking | Temperature | Top P | Max Output |
+| ----------------------------------- | ------------- | -------- | ----------- | ----- | ---------- |
+| Generator / Refiner | `glm-5.1` | Enabled | 1.0 | 0.95 | 32768 |
+| Coder | `glm-5.1` | Enabled | 1.0 | 0.95 | 20480 |
+| Adviser / Pentester | `glm-5.1` | Enabled | 1.0 | 0.95 | 16384 |
+| Primary Agent / Assistant | `glm-5-turbo` | Enabled | 1.0 | 0.95 | 16384 |
+| Installer | `glm-4.5-air` | Enabled | 1.0 | 0.95 | 16384 |
+| Simple / Reflector | `glm-4.5-air` | Disabled | 0.6 | 0.9 | 8192 |
+| Searcher / Enricher / Simple JSON | `glm-4.5-air` | Disabled | 0.6 | 0.9 | 4096 |
+
+> **Note on temperature**: GLM accepts both `1.0` and `0.6` in either thinking/non-thinking mode (per Z.AI docs). langchaingo's `IsReasoningModel` matches `glm-4.5*`/`glm-4.6*`/`glm-4.7*` prefixes and force-overrides temperature to 1.0 in `createChatRequest` — this is harmless for GLM (unlike Kimi) but means temperature values for those models in YAML are advisory. `glm-5`/`glm-5.1`/`glm-5-turbo` are not matched, so explicit values pass through unchanged.
**Thinking Modes**:
-- **Forced**: Model always uses thinking mode before responding (GLM-5, GLM-4.7)
-- **Hybrid**: Model intelligently decides when to use thinking (GLM-4.7-FlashX, GLM-4.7-Flash)
-- **Auto**: Model automatically determines when reasoning is needed (GLM-4.6, GLM-4.5 series)
+- **Hybrid** (GLM-5.x, GLM-4.7): Explicit toggle via `extra_body.thinking.type`
+- **Auto** (GLM-4.6, GLM-4.5 series): Model automatically determines when reasoning is needed
+- **Preserved Thinking** (Z.AI Coding capability): all thinking-enabled agents in PentAGI also pass `extra_body.thinking.clear_thinking: false` so that `reasoning_content` from previous assistant turns is retained across the conversation. This is required on the standard API endpoint (`/api/paas/v4`) — on the Coding Plan endpoint it would be enabled by default. Improves reasoning continuity and cache hit rates in multi-turn tool call chains.
+- All thinking-enabled agents also pass `extra_body.tool_choice: auto` defensively
**Key Features**:
+- **Long-Horizon Tasks**: GLM-5.1 supports 8-hour sustained autonomous execution, ideal for complex multi-stage agentic workflows
+- **OpenClaw-Native Orchestration**: GLM-5-Turbo is specifically optimized for tool invocation, instruction following, and long-chain execution
- **Prompt Caching**: Significant cost reduction on repeated context (cached input pricing shown)
-- **Interleaved Thinking**: GLM-4.7 thinks before each response and tool call with preserved reasoning across multi-turn dialogues
-- **Ultra-Long Context**: 200K tokens for GLM-5 and GLM-4.7/4.6 series for massive codebase analysis
-- **MoE Architecture**: Efficient 744B parameters with 40B active (GLM-5), 355B/32B (GLM-4.5), 106B/12B (GLM-4.5-Air)
+- **Ultra-Long Context**: 200K tokens for GLM-5.x/4.7/4.6 series
+- **MoE Architecture**: Efficient 744B/40B active (GLM-5/5.1), 355B/32B (GLM-4.5), 106B/12B (GLM-4.5-Air)
- **Tool Calling**: Seamless integration with 20+ pentesting tools via function calling
-- **Streaming**: Real-time response streaming with streaming tool calls support (GLM-4.6+)
+- **Streaming**: Real-time streaming with streaming tool calls support (GLM-4.6+)
- **Multilingual**: Exceptional Chinese and English NLP capabilities
- **Free Options**: GLM-4.7-Flash and GLM-4.5-Flash for prototyping and experimentation
@@ -2138,51 +2165,67 @@ KIMI_PROVIDER=moonshot # Adds prefix to model names (moonshot/kimi-k2.5) for Li
#### Supported Models
-PentAGI supports 11 Kimi/Moonshot models with tool calling, streaming, thinking modes, and multimodal capabilities. Models marked with `*` are used in default configuration.
+PentAGI supports 8 Kimi/Moonshot models with tool calling, streaming, hybrid thinking modes, and multimodal capabilities (text/image/video for K2.x). All `kimi-k2-*` legacy models (turbo-preview, 0905-preview, 0711-preview, thinking, thinking-turbo) were deprecated by Moonshot on 2026-05-25 and are NOT included. Models marked with `*` are used in default configuration.
-**Kimi K2.5 Series - Advanced Multimodal**
+**Kimi K2.x Series - Multimodal Flagship**
-| Model ID | Thinking | Multimodal | Context | Speed | Price (Input/Output) | Use Case |
-| -------------------------- | -------- | ---------- | ------- | ---------- | -------------------- | ----------------------------------------------- |
-| `kimi-k2.5`* | ✅ | ✅ | 256K | Standard | $0.60/$3.00 | Most intelligent, versatile, vision+text+code |
+| Model ID | Thinking | Multimodal | Context | Price (Input Miss / Output / Cache Hit) | Use Case |
+| ---------------- | -------- | ---------- | ------- | --------------------------------------- | ------------------------------------------------------- |
+| `kimi-k2.6`* | ✅ hybrid | ✅ | 256K | $0.95 / $4.00 / $0.16 | Latest flagship: native multimodal, stronger code, improved instruction compliance (generator/refiner/adviser/coder/pentester default) |
+| `kimi-k2.5`* | ✅ hybrid | ✅ | 256K | $0.60 / $3.00 / $0.10 | Previous-gen: 36% cheaper input, same architecture (primary/assistant/installer/utility default) |
-**Kimi K2 Series - MoE Foundation (1T params, 32B activated)**
+**Moonshot V1 Series - Generation Models (Flexible Parameters)**
-| Model ID | Thinking | Multimodal | Context | Speed | Price (Input/Output) | Use Case |
-| -------------------------- | -------- | ---------- | ------- | ---------- | -------------------- | ----------------------------------------------- |
-| `kimi-k2-0905-preview`* | ❌ | ❌ | 256K | Standard | $0.60/$2.50 | Enhanced agentic coding, improved frontend |
-| `kimi-k2-0711-preview` | ❌ | ❌ | 128K | Standard | $0.60/$2.50 | Powerful code and agent capabilities |
-| `kimi-k2-turbo-preview`* | ❌ | ❌ | 256K | Turbo | $1.15/$8.00 | High-speed version, 60-100 tokens/sec |
-| `kimi-k2-thinking` | ✅ | ❌ | 256K | Standard | $0.60/$2.50 | Long-term thinking, multi-step tool usage |
-| `kimi-k2-thinking-turbo` | ✅ | ❌ | 256K | Turbo | $1.15/$8.00 | High-speed thinking, deep reasoning |
+| Model ID | Thinking | Multimodal | Context | Price (Input / Output) | Use Case |
+| ------------------- | -------- | ---------- | ------- | ---------------------- | ---------------------------------------------- |
+| `moonshot-v1-8k` | ❌ | ❌ | 8K | $0.20 / $2.00 | Short text generation, ultra-cheap |
+| `moonshot-v1-32k` | ❌ | ❌ | 32K | $1.00 / $3.00 | Long text generation |
+| `moonshot-v1-128k` | ❌ | ❌ | 128K | $2.00 / $5.00 | Very long context |
-**Moonshot V1 Series - General Text Generation**
+**Moonshot V1 Vision Series - Image Understanding**
-| Model ID | Thinking | Multimodal | Context | Speed | Price (Input/Output) | Use Case |
-| -------------------------- | -------- | ---------- | ------- | ---------- | -------------------- | ----------------------------------------------- |
-| `moonshot-v1-8k` | ❌ | ❌ | 8K | Standard | $0.20/$2.00 | Short text generation, cost-effective |
-| `moonshot-v1-32k` | ❌ | ❌ | 32K | Standard | $1.00/$3.00 | Long text generation, balanced |
-| `moonshot-v1-128k` | ❌ | ❌ | 128K | Standard | $2.00/$5.00 | Very long text generation, extensive context |
+| Model ID | Thinking | Multimodal | Context | Price (Input / Output) | Use Case |
+| --------------------------------- | -------- | ---------- | ------- | ---------------------- | --------------------------------------- |
+| `moonshot-v1-8k-vision-preview` | ❌ | ✅ | 8K | $0.20 / $2.00 | Vision + short context |
+| `moonshot-v1-32k-vision-preview` | ❌ | ✅ | 32K | $1.00 / $3.00 | Vision + medium context |
+| `moonshot-v1-128k-vision-preview` | ❌ | ✅ | 128K | $2.00 / $5.00 | Vision + long context |
-**Moonshot V1 Vision Series - Multimodal**
+**Prices**: Per 1M tokens. Cache pricing applies to prompt tokens served from automatic context cache (only Kimi K2.x models support cache).
+
+> **CRITICAL — Kimi K2.6/K2.5 parameter constraints**: API returns `invalid_request_error` for any deviation:
+> - `temperature`: MUST be `1.0` in thinking mode, MUST be `0.6` in non-thinking mode
+> - `top_p`: MUST be `0.95`
+> - `n`: MUST be `1`
+> - `presence_penalty` and `frequency_penalty`: MUST be `0` (not modifiable)
+>
+> Moonshot V1 models use standard OpenAI-compatible parameters with no such constraints.
-| Model ID | Thinking | Multimodal | Context | Speed | Price (Input/Output) | Use Case |
-| ----------------------------- | -------- | ---------- | ------- | ---------- | -------------------- | ----------------------------------------------- |
-| `moonshot-v1-8k-vision-preview` | ❌ | ✅ | 8K | Standard | $0.20/$2.00 | Vision understanding, short context |
-| `moonshot-v1-32k-vision-preview` | ❌ | ✅ | 32K | Standard | $1.00/$3.00 | Vision understanding, medium context |
-| `moonshot-v1-128k-vision-preview` | ❌ | ✅ | 128K | Standard | $2.00/$5.00 | Vision understanding, long context |
+**Default Agent Configuration**:
-**Prices**: Per 1M tokens. Turbo models offer 60-100 tokens/sec output speed with higher pricing.
+Strategy: prefer `kimi-k2.5` as cost-effective workhorse (36% cheaper input vs `kimi-k2.6`); reserve `kimi-k2.6` for critical reasoning. All `kimi-k2.x` agents are configured with the API-required fixed parameters (temp/top_p/n) and explicit `extra_body.thinking.type`. For thinking-enabled agents, `extra_body.thinking.keep: "all"` is set to preserve historical `reasoning_content` in multi-turn tool call chains (without it Moonshot returns "thinking is enabled but reasoning_content is missing").
+
+| Agent Role | Default Model | Thinking | Temperature | Top P | Max Output |
+| -------------------------------------------- | ------------- | -------- | ----------- | ----- | ---------- |
+| Generator / Refiner | `kimi-k2.6` | Enabled (keep=all) | 1.0 | 0.95 | 32768 |
+| Coder | `kimi-k2.6` | Enabled (keep=all) | 1.0 | 0.95 | 20480 |
+| Pentester | `kimi-k2.6` | Enabled (keep=all) | 1.0 | 0.95 | 16384 |
+| Adviser (mentor/planner) | `kimi-k2.6` | Enabled (keep=all) | 1.0 | 0.95 | 8192 |
+| Primary Agent / Assistant | `kimi-k2.5` | Enabled (keep=all) | 1.0 | 0.95 | 16384 |
+| Installer | `kimi-k2.5` | Enabled (keep=all) | 1.0 | 0.95 | 12288 |
+| Reflector / Searcher / Enricher | `kimi-k2.5` | Disabled | 0.6 | 0.95 | 4096 |
+| Simple / Simple JSON | `kimi-k2.5` | Disabled | 0.6 | 0.95 | 2048 |
**Key Features**:
-- **Ultra-Long Context**: Up to 256K tokens for comprehensive codebase analysis
-- **Multimodal Capabilities**: Vision models support image understanding for screenshot analysis (Kimi K2.5, V1 Vision series)
-- **Extended Thinking**: Deep reasoning with multi-step tool usage (kimi-k2.5, kimi-k2-thinking models)
-- **High-Speed Turbo**: 60-100 tokens/sec output for real-time workflows (Turbo variants)
-- **Tool Calling**: Seamless integration with 20+ pentesting tools via function calling
-- **Streaming**: Real-time response streaming for interactive security assessment
-- **Multilingual**: Strong Chinese and English language support
-- **MoE Architecture**: Efficient 1T total parameters with 32B activated for K2 series
+- **Ultra-Long Context**: Up to 256K tokens (K2.x) for comprehensive codebase/documentation analysis
+- **Native Multimodal**: K2.6/K2.5 support text + image + video input out of the box
+- **Hybrid Thinking**: K2.6/K2.5 toggle between thinking and non-thinking via `extra_body.thinking.type`
+- **Preserved Thinking** (K2.6): `thinking.keep: "all"` preserves historical `reasoning_content` across turns — required for multi-turn tool call chains
+- **Automatic Context Caching**: K2.x models cache repeated prefixes (~17% of miss price for K2.6, ~17% for K2.5)
+- **Tool Calling**: Full function-calling support for K2.x and Moonshot V1
+- **Self-Correction**: K2.6 features improved instruction compliance and self-correction
+- **Multilingual**: Strong Chinese, English, and multi-language support
+
+**Multi-turn with thinking + tool calls**: PentAGI's universal reasoning preservation pattern (`TextPartWithReasoning` + `WithPreserveReasoningContent`) automatically ensures `reasoning_content` is sent back in the required TextContent → ToolCall order, satisfying Moonshot's "thinking is enabled but reasoning_content is missing in assistant tool call message" requirement.
**LiteLLM Integration**: Set `KIMI_PROVIDER=moonshot` to enable model name prefixing when using default PentAGI configurations with LiteLLM proxy. Leave empty for direct API usage.
@@ -2217,83 +2260,135 @@ QWEN_PROVIDER=dashscope # Adds prefix to model names (dashscope/qwen-plus) for
#### Supported Models
-PentAGI supports 32 Qwen models with tool calling, streaming, thinking modes, and context caching. Models marked with `*` are used in default configuration.
-
-**Wide Availability Models (All Regions)**
-
-| Model ID | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case |
-| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ----------------------------------------------- |
-| `qwen3-max`* | ✅ | ✅ | ✅ | ✅ | $2.40/$12.00/$0.48 | Flagship reasoning, complex security analysis |
-| `qwen3-max-preview` | ✅ | ✅ | ✅ | ✅ | $2.40/$12.00/$0.48 | Preview version with extended thinking |
-| `qwen-max` | ❌ | ✅ | ❌ | ✅ | $1.60/$6.40/$0.32 | Strong instruction following, legacy flagship |
-| `qwen3.5-plus`* | ✅ | ✅ | ✅ | ✅ | $0.40/$2.40/$0.08 | Balanced reasoning, general dialogue, coding |
-| `qwen-plus` | ✅ | ✅ | ✅ | ✅ | $0.40/$4.00/$0.08 | Cost-effective balanced performance |
-| `qwen3.5-flash`* | ✅ | ✅ | ✅ | ✅ | $0.10/$0.40/$0.02 | Ultra-fast lightweight, high-throughput |
-| `qwen-flash` | ❌ | ✅ | ✅ | ✅ | $0.05/$0.40/$0.01 | Fast with context caching, cost-optimized |
-| `qwen-turbo` | ✅ | ✅ | ❌ | ✅ | $0.05/$0.50/$0.01 | Deprecated, use qwen-flash instead |
-| `qwq-plus` | ✅ | ✅ | ❌ | ✅ | $0.80/$2.40/$0.16 | Deep reasoning, chain-of-thought analysis |
-
-**Region-Specific Models**
-
-| Model ID | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case |
-| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ----------------------------------------------- |
-| `qwen-plus-us` | ✅ | ❌ | ✅ | ❌ | $0.40/$4.00/$0.08 | US region optimized balanced model |
-| `qwen-long-latest` | ❌ | ❌ | ❌ | ✅ | $0.07/$0.29/$0.01 | Ultra-long context (10M tokens) |
-
-**Open Source - Qwen3.5 Series**
-
-| Model ID | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case |
-| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ----------------------------------------------- |
-| `qwen3.5-397b-a17b` | ✅ | ✅ | ✅ | ✅ | $0.60/$3.60/$0.12 | Largest 397B parameters, exceptional reasoning |
-| `qwen3.5-122b-a10b` | ✅ | ✅ | ✅ | ✅ | $0.40/$3.20/$0.08 | Large 122B parameters, strong performance |
-| `qwen3.5-27b` | ✅ | ✅ | ✅ | ✅ | $0.30/$2.40/$0.06 | Medium 27B parameters, balanced |
-| `qwen3.5-35b-a3b` | ✅ | ✅ | ✅ | ✅ | $0.25/$2.00/$0.05 | Efficient 35B with 3B active MoE |
-
-**Open Source - Qwen3 Series**
-
-| Model ID | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case |
-| ------------------------------ | -------- | ---- | --------- | ----- | -------------------------- | ----------------------------------------------- |
-| `qwen3-next-80b-a3b-thinking` | ✅ | ✅ | ✅ | ✅ | $0.15/$1.43/$0.03 | Next-gen 80B thinking-only mode |
-| `qwen3-next-80b-a3b-instruct` | ❌ | ✅ | ✅ | ✅ | $0.15/$1.20/$0.03 | Next-gen 80B instruction following |
-| `qwen3-235b-a22b` | ✅ | ✅ | ✅ | ✅ | $0.70/$8.40/$0.14 | Dual-mode 235B with 22B active |
-| `qwen3-32b` | ✅ | ✅ | ✅ | ✅ | $0.29/$2.87/$0.06 | Versatile 32B dual-mode |
-| `qwen3-30b-a3b` | ✅ | ✅ | ✅ | ✅ | $0.20/$2.40/$0.04 | Efficient 30B MoE architecture |
-| `qwen3-14b` | ✅ | ✅ | ✅ | ✅ | $0.35/$4.20/$0.07 | Medium 14B performance-cost balance |
-| `qwen3-8b` | ✅ | ✅ | ✅ | ✅ | $0.18/$2.10/$0.04 | Compact 8B efficiency optimized |
-| `qwen3-4b` | ✅ | ✅ | ❌ | ✅ | $0.11/$1.26/$0.02 | Lightweight 4B for simple tasks |
-| `qwen3-1.7b` | ✅ | ✅ | ❌ | ✅ | $0.11/$1.26/$0.02 | Ultra-compact 1.7B basic tasks |
-| `qwen3-0.6b` | ✅ | ✅ | ❌ | ✅ | $0.11/$1.26/$0.02 | Smallest 0.6B minimal resources |
-
-**Open Source - QwQ & Qwen2.5 Series**
-
-| Model ID | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case |
-| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ----------------------------------------------- |
-| `qwq-32b` | ✅ | ✅ | ✅ | ✅ | $0.29/$0.86/$0.06 | Open 32B reasoning, deep research |
-| `qwen2.5-14b-instruct-1m` | ❌ | ✅ | ❌ | ✅ | $0.81/$3.22/$0.16 | Extended 1M context, 14B parameters |
-| `qwen2.5-7b-instruct-1m` | ❌ | ✅ | ❌ | ✅ | $0.37/$1.47/$0.07 | Extended 1M context, 7B parameters |
-| `qwen2.5-72b-instruct` | ❌ | ✅ | ❌ | ✅ | $1.40/$5.60/$0.28 | Large 72B instruction following |
-| `qwen2.5-32b-instruct` | ❌ | ✅ | ❌ | ✅ | $0.70/$2.80/$0.14 | Medium 32B instruction following |
-| `qwen2.5-14b-instruct` | ❌ | ✅ | ❌ | ✅ | $0.35/$1.40/$0.07 | Compact 14B instruction following |
-| `qwen2.5-7b-instruct` | ❌ | ✅ | ❌ | ✅ | $0.18/$0.70/$0.04 | Small 7B instruction following |
-| `qwen2.5-3b-instruct` | ❌ | ❌ | ❌ | ✅ | $0.04/$0.13/$0.01 | Lightweight 3B Chinese Mainland only |
-
-**Prices**: Per 1M tokens. Cache pricing is for implicit context caching (20% of input cost). Models with thinking support include additional reasoning computation during CoT phase.
+PentAGI supports 33 Qwen models curated for agent workflows: text reasoning, code generation, and vision-language (browser screenshots). All models are non-snapshot main aliases with tool calling, streaming, thinking modes, and context caching. Models marked with `*` are used in default configuration.
+
+**Flagship Models (Top-tier Reasoning)**
+
+| Model ID | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case |
+| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
+| `qwen3.7-max`* | ✅ | ✅ | ✅ | ✅ | $2.50/$7.50/$0.50 | Next-gen flagship for agent-centric era (generator/refiner/adviser default) |
+| `qwen3.6-max-preview` | ✅ | ✅ | ✅ | ✅ | $1.30/$7.80/$0.13 | Preview Max with enhanced vibe coding & front-end skills |
+| `qwen3-max` | ✅ | ✅ | ✅ | ✅ | $1.20/$6.00/$0.24 | Previous-gen flagship with agent programming upgrades |
+| `qwen-plus` | ✅ | ✅ | ✅ | ✅ | $0.40/$4.00/$0.08 | Qwen3-backbone Plus with switchable thinking modes |
+
+**Balanced Plus Models (Mid-tier)**
+
+| Model ID | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case |
+| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
+| `qwen3.6-plus`* | ✅ | ✅ | ✅ | ✅ | $0.50/$3.00/$0.05 | Native VL Plus with agentic coding (primary/assistant/pentester default) |
+| `qwen3.5-plus` | ✅ | ✅ | ✅ | ✅ | $0.40/$2.40/$0.04 | Previous-gen native VL with strong multimodal capabilities |
+
+**Fast Flash Models (Cost-optimized)**
+
+| Model ID | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case |
+| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
+| `qwen3.6-flash` | ✅ | ✅ | ✅ | ✅ | $0.25/$1.50/$0.025 | Latest Flash with significant agentic-coding boost |
+| `qwen3.5-flash`* | ✅ | ✅ | ✅ | ✅ | $0.10/$0.40/$0.01 | Ultra-fast lightweight (simple/reflector/searcher/enricher default) |
+| `qwen-flash` | ✅ | ✅ | ✅ | ✅ | $0.05/$0.40/$0.01 | Qwen3-series Flash with 1M context, tiered pricing |
+
+**Code-Specialized Models**
+
+| Model ID | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case |
+| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
+| `qwen3-coder-plus`* | ❌ | ✅ | ✅ | ✅ | $1.00/$5.00/$0.20 | Strong coding agent with autonomous programming (coder default) |
+| `qwen3-coder-flash`* | ❌ | ✅ | ✅ | ✅ | $0.30/$1.50/$0.06 | Fast code-gen with multi-turn tool stability (installer default) |
+| `qwen3-coder-next` | ❌ | ✅ | ✅ | ✅ | $0.30/$1.50/— | Open-source code generation, SOTA at same scale |
+
+**Vision-Language Models (Browser & Screenshot Analysis)**
+
+| Model ID | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case |
+| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
+| `qwen3-vl-plus` | ✅ | ✅ | ✅ | ✅ | $0.20/$1.60/$0.04 | VL with visual agent capabilities, ultra-long video understanding |
+| `qwen3-vl-flash` | ✅ | ✅ | ✅ | ✅ | $0.05/$0.40/$0.01 | Small VL with 2D/3D localization for browser triage |
+| `qvq-max` | ✅ | ✅ | ✅ | ✅ | $1.20/$4.80/— | Visual reasoning with chain-of-thought |
+
+**Open-Source Qwen3.6 Series**
+
+| Model ID | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case |
+| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
+| `qwen3.6-27b` | ✅ | ✅ | ✅ | ✅ | $0.60/$3.60/— | Native VL on hybrid architecture, on-premises ready |
+| `qwen3.6-35b-a3b` | ✅ | ✅ | ✅ | ✅ | $0.25/$1.49/— | Efficient 35B MoE (~3B active) for continuous monitoring |
+
+**Open-Source Qwen3.5 Series**
+
+| Model ID | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case |
+| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
+| `qwen3.5-397b-a17b` | ✅ | ✅ | ✅ | ✅ | $0.60/$3.60/— | Largest 397B params (~17B active), exceptional reasoning |
+| `qwen3.5-122b-a10b` | ✅ | ✅ | ✅ | ✅ | $0.40/$3.20/— | Large 122B params (~10B active), strong balance |
+| `qwen3.5-35b-a3b` | ✅ | ✅ | ✅ | ✅ | $0.25/$2.00/— | Efficient 35B MoE (~3B active), cost-effective |
+| `qwen3.5-27b` | ✅ | ✅ | ✅ | ✅ | $0.30/$2.40/— | Medium 27B with hybrid linear attention + sparse MoE |
+
+**Open-Source Qwen3 Coder Series**
+
+| Model ID | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case |
+| ------------------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
+| `qwen3-coder-480b-a35b-instruct` | ❌ | ✅ | ✅ | ✅ | $1.50/$7.50/— | Largest open coder MoE (480B/~35B active) |
+| `qwen3-coder-30b-a3b-instruct` | ❌ | ✅ | ✅ | ✅ | $0.45/$2.25/— | Efficient 30B MoE (~3B active), repository-scale |
+
+**Open-Source Qwen3 Dense & MoE Series**
+
+| Model ID | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case |
+| ------------------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
+| `qwen3-next-80b-a3b-thinking` | ✅ | ✅ | ✅ | ✅ | $0.15/$1.20/— | Next-gen 80B MoE (~3B active) thinking-only |
+| `qwen3-next-80b-a3b-instruct` | ❌ | ✅ | ✅ | ✅ | $0.15/$1.20/— | Next-gen 80B MoE instruction-following |
+| `qwen3-235b-a22b` | ✅ | ✅ | ✅ | ✅ | $0.70/$8.40/— | Dual-mode 235B MoE (~22B active) |
+| `qwen3-32b` | ✅ | ✅ | ✅ | ✅ | $0.16/$0.64/— | Versatile 32B dense dual-mode |
+| `qwen3-30b-a3b` | ✅ | ✅ | ✅ | ✅ | $0.20/$2.40/— | Efficient 30B MoE (~3B active) |
+| `qwen3-14b` | ✅ | ✅ | ✅ | ✅ | $0.35/$4.20/— | Medium 14B dense performance-cost balance |
+| `qwen3-8b` | ✅ | ✅ | ✅ | ✅ | $0.18/$2.10/— | Compact 8B dense efficiency |
+| `qwen3-4b` | ✅ | ✅ | ✅ | ✅ | $0.11/$1.26/— | Lightweight 4B dense for simple tasks |
+| `qwen3-1.7b` | ✅ | ✅ | ✅ | ✅ | $0.11/$1.26/— | Ultra-compact 1.7B basic checks |
+| `qwen3-0.6b` | ✅ | ✅ | ✅ | ✅ | $0.11/$1.26/— | Smallest 0.6B for edge monitoring |
+
+**Prices**: Per 1M tokens. Cache pricing reflects implicit cache hit (when available); MoE/dense open-source models do not expose cache pricing. Tiered models (Max/Plus) show lowest-tier pricing (typically ≤32k or ≤256k input); larger contexts incur higher rates per Alibaba Cloud pricing.
**Region Availability**:
- **Intl** (International): Singapore region (`dashscope-intl.aliyuncs.com`)
- **Global/US**: US Virginia region (`dashscope-us.aliyuncs.com`)
- **China**: Chinese Mainland Beijing region (`dashscope.aliyuncs.com`)
+**Default Agent Configuration**:
+| Agent Role | Default Model | Tier |
+| ------------------------------------------------ | -------------------- | --------- |
+| Generator / Refiner / Adviser (planning, mentor) | `qwen3.7-max` | Flagship |
+| Primary / Assistant / Pentester | `qwen3.6-plus` | Balanced |
+| Coder (exploit development) | `qwen3-coder-plus` | Code+ |
+| Installer (env setup) | `qwen3-coder-flash` | Code Fast |
+| Simple / Reflector / Searcher / Enricher | `qwen3.5-flash` | Fast |
+
**Key Features**:
-- **Automatic Context Caching**: 30-50% cost reduction on repeated context with implicit cache (20% of input price)
-- **Extended Thinking**: Chain-of-thought reasoning for complex security analysis (Qwen3-Max, QwQ, Qwen3.5-Plus)
+- **Agent-Centric Design**: Qwen3.7-Max is purpose-built for long-horizon autonomous execution and tool invocation
+- **Automatic Context Caching**: 30-50% cost reduction on repeated context with implicit cache
+- **Extended Thinking**: Chain-of-thought reasoning for complex security analysis (Qwen3.7/3.6/3.5/3-Max, QVQ-Max)
+- **Code Specialization**: Qwen3-Coder series with multi-turn tool interaction and repository-level understanding
+- **Vision-Language**: Qwen3-VL series for browser screenshot triage, 2D/3D localization, OCR-level analysis
- **Tool Calling**: Seamless integration with 20+ pentesting tools via function calling
- **Streaming**: Real-time response streaming for interactive workflows
- **Multilingual**: Strong Chinese, English, and multi-language support
-- **Ultra-Long Context**: Up to 10M tokens with qwen-long-latest for massive codebase analysis
+- **Open-Source Variants**: Dense and MoE models from 0.6B to 480B for on-premises/air-gapped deployments
**LiteLLM Integration**: Set `QWEN_PROVIDER=dashscope` to enable model name prefixing when using default PentAGI configurations with LiteLLM proxy. Leave empty for direct API usage.
+#### Alternative Integrations
+
+DashScope is fully OpenAI-compatible, so Qwen can also power two other PentAGI subsystems through the standard OpenAI client.
+
+**As embedding provider** (`text-embedding-v4`, see [Alibaba Cloud Model Studio pricing](https://modelstudio.console.alibabacloud.com/ap-southeast-1?tab=doc#/doc/?type=model&url=prices)):
+
+```bash
+EMBEDDING_PROVIDER=openai
+EMBEDDING_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1 # International (Singapore)
+# EMBEDDING_URL=https://dashscope.aliyuncs.com/compatible-mode/v1 # Chinese Mainland
+EMBEDDING_KEY=sk-*******
+EMBEDDING_MODEL=text-embedding-v4
+EMBEDDING_BATCH_SIZE= # optional, default applies
+EMBEDDING_STRIP_NEW_LINES= # optional, default applies
+```
+
+> Note: the Global/US DashScope endpoint (`dashscope-us.aliyuncs.com`) does **not** expose embedding APIs — use the International or China endpoints for `text-embedding-v4`.
+
+**As OpenAI-typed custom LLM provider**: instead of the dedicated `QWEN_*` variables, you can wire any Qwen chat model through PentAGI's custom OpenAI-compatible provider by pointing `OPENAI_SERVER_URL` (or a custom provider entry) to the DashScope `/compatible-mode/v1` endpoint and selecting the desired Qwen model name. Useful when you already manage all model traffic through a single OpenAI-shaped client (e.g. shared with LiteLLM/OneAPI proxies).
+
## Advanced Setup
### Langfuse Integration
@@ -3119,6 +3214,8 @@ PentAGI supports the following embedding providers:
- **GoogleAI**: Google's embedding models
- **VoyageAI**: VoyageAI's embedding models
+> **OpenAI-compatible third parties**: any provider exposing OpenAI's `/embeddings` API can be plugged in via `EMBEDDING_PROVIDER=openai` with a custom `EMBEDDING_URL`. For example, **Qwen DashScope** offers `text-embedding-v4` through the `/compatible-mode/v1` endpoint (International and Chinese Mainland regions only — the US region does not expose embeddings). See the [Qwen Alternative Integrations](#alternative-integrations) subsection for the full configuration snippet.
+
<details>
<summary><b>Embedding Provider Configuration</b> (click to expand)</summary>
diff --git a/backend/pkg/providers/deepseek/config.yml b/backend/pkg/providers/deepseek/config.yml
index 2f3b85ab..6fe8caa2 100644
--- a/backend/pkg/providers/deepseek/config.yml
+++ b/backend/pkg/providers/deepseek/config.yml
@@ -1,7 +1,22 @@
+# DeepSeek V4 agent configuration.
+#
+# Strategy: prefer deepseek-v4-flash (12x cheaper) for the "workhorse" agents,
+# reserve deepseek-v4-pro for complex multi-step reasoning (planning, mentor,
+# code generation, vulnerability analysis, orchestration).
+#
+# Hyperparameters note:
+# - v4-pro agents with `reasoning.effort` set MUST NOT specify temperature/top_p
+# (langchaingo nullifies them, and DeepSeek silently ignores them in thinking
+# mode together with presence_penalty/frequency_penalty).
+# - v4-flash utility agents explicitly disable thinking via extra_body to keep
+# them fast and cheap.
+# - All thinking-enabled agents also pass extra_body.thinking.type explicitly
+# as a defensive measure against future DeepSeek default changes.
+
simple:
model: deepseek-v4-flash
- temperature: 0.5
- top_p: 0.5
+ temperature: 0.3
+ top_p: 0.9
n: 1
max_tokens: 8192
extra_body:
@@ -14,8 +29,8 @@ simple:
simple_json:
model: deepseek-v4-flash
- temperature: 0.5
- top_p: 0.5
+ temperature: 0.3
+ top_p: 0.9
n: 1
max_tokens: 4096
json: true
@@ -31,42 +46,76 @@ primary_agent:
model: deepseek-v4-pro
n: 1
max_tokens: 16384
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
+ input: 1.74
+ output: 3.48
+ cache_read: 0.0145
assistant:
model: deepseek-v4-pro
n: 1
max_tokens: 16384
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
+ input: 1.74
+ output: 3.48
+ cache_read: 0.0145
generator:
model: deepseek-v4-pro
n: 1
max_tokens: 32768
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
+ input: 1.74
+ output: 3.48
+ cache_read: 0.0145
refiner:
model: deepseek-v4-pro
n: 1
- max_tokens: 20480
+ max_tokens: 32768
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
+ input: 1.74
+ output: 3.48
+ cache_read: 0.0145
adviser:
+ model: deepseek-v4-pro
+ n: 1
+ max_tokens: 16384
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
+ price:
+ input: 1.74
+ output: 3.48
+ cache_read: 0.0145
+
+reflector:
model: deepseek-v4-flash
- temperature: 0.7
- top_p: 0.8
+ temperature: 0.5
+ top_p: 0.9
n: 1
max_tokens: 8192
extra_body:
@@ -77,19 +126,10 @@ adviser:
output: 0.28
cache_read: 0.0028
-reflector:
- model: deepseek-v4-pro
- n: 1
- max_tokens: 4096
- price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
-
searcher:
model: deepseek-v4-flash
- temperature: 0.7
- top_p: 0.8
+ temperature: 0.5
+ top_p: 0.9
n: 1
max_tokens: 4096
extra_body:
@@ -102,8 +142,8 @@ searcher:
enricher:
model: deepseek-v4-flash
- temperature: 0.7
- top_p: 0.8
+ temperature: 0.5
+ top_p: 0.9
n: 1
max_tokens: 4096
extra_body:
@@ -118,25 +158,40 @@ coder:
model: deepseek-v4-pro
n: 1
max_tokens: 20480
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
+ input: 1.74
+ output: 3.48
+ cache_read: 0.0145
installer:
- model: deepseek-v4-pro
+ model: deepseek-v4-flash
n: 1
- max_tokens: 16384
+ max_tokens: 12288
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
+ input: 0.14
+ output: 0.28
+ cache_read: 0.0028
pentester:
model: deepseek-v4-pro
n: 1
max_tokens: 16384
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
+ input: 1.74
+ output: 3.48
+ cache_read: 0.0145
diff --git a/backend/pkg/providers/deepseek/deepseek.go b/backend/pkg/providers/deepseek/deepseek.go
index 915397cf..743554a9 100644
--- a/backend/pkg/providers/deepseek/deepseek.go
+++ b/backend/pkg/providers/deepseek/deepseek.go
@@ -83,6 +83,12 @@ func New(
return nil, err
}
+ // DeepSeek V4 OpenAI-compatible API expects the legacy string form
+ // "reasoning_effort": "high|medium|low" rather than the modern object form
+ // "reasoning": {"effort": "..."}. The absence of WithModernReasoningFormat()
+ // ensures langchaingo serializes reasoning_effort as a top-level string.
+ // WithPreserveReasoningContent() keeps reasoning_content in multi-turn
+ // assistant messages with tool calls (required for DeepSeek thinking mode).
client, err := openai.New(
openai.WithToken(cfg.DeepSeekAPIKey),
openai.WithModel(DeepSeekAgentModel),
diff --git a/backend/pkg/providers/deepseek/models.yml b/backend/pkg/providers/deepseek/models.yml
index 79b465ee..2b9acf48 100644
--- a/backend/pkg/providers/deepseek/models.yml
+++ b/backend/pkg/providers/deepseek/models.yml
@@ -1,15 +1,15 @@
- name: deepseek-v4-flash
- description: DeepSeek V4 Flash - Cost-efficient general-purpose model suitable for dialogue, code generation, and tool calling. Supports JSON output and tool calls. 1M context, up to 384K output tokens.
- thinking: false
+ description: DeepSeek V4 Flash - Cost-efficient general-purpose model with hybrid thinking/non-thinking modes (default thinking, switchable via extra_body). Suitable for dialogue, code generation, and tool calling. Supports JSON output, tool calls, chat prefix completion (beta), and FIM completion (non-thinking only). 1M context, up to 384K output tokens.
+ thinking: true
price:
input: 0.14
output: 0.28
cache_read: 0.0028
- name: deepseek-v4-pro
- description: DeepSeek V4 Pro - Higher-tier reasoning model suitable for complex logic, mathematical reasoning, and security analysis. 1M context, up to 384K output tokens.
+ description: DeepSeek V4 Pro - Higher-tier reasoning model with hybrid thinking/non-thinking modes (default thinking, switchable via extra_body). Suitable for complex logic, mathematical reasoning, and security analysis. Supports JSON output, tool calls, chat prefix completion (beta), and FIM completion (non-thinking only). 1M context, up to 384K output tokens.
thinking: true
price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
+ input: 1.74
+ output: 3.48
+ cache_read: 0.0145
diff --git a/backend/pkg/providers/gemini/config.yml b/backend/pkg/providers/gemini/config.yml
index 68130d7d..09f4e230 100644
--- a/backend/pkg/providers/gemini/config.yml
+++ b/backend/pkg/providers/gemini/config.yml
@@ -1,5 +1,5 @@
simple:
- model: gemini-3.1-flash-lite-preview
+ model: gemini-3.1-flash-lite
temperature: 0.7
top_p: 0.95
n: 1
@@ -10,7 +10,7 @@ simple:
cache_read: 0.025
simple_json:
- model: gemini-3.1-flash-lite-preview
+ model: gemini-3.1-flash-lite
temperature: 0.7
top_p: 0.95
n: 1
@@ -87,37 +87,37 @@ adviser:
cache_read: 0.2
reflector:
- model: gemini-3-flash-preview
+ model: gemini-3.5-flash
temperature: 1.0
top_p: 0.95
n: 1
max_tokens: 4096
price:
- input: 0.5
- output: 3.0
- cache_read: 0.05
+ input: 1.5
+ output: 9.0
+ cache_read: 0.15
searcher:
- model: gemini-3-flash-preview
+ model: gemini-3.5-flash
temperature: 1.0
top_p: 0.95
n: 1
max_tokens: 8192
price:
- input: 0.5
- output: 3.0
- cache_read: 0.05
+ input: 1.5
+ output: 9.0
+ cache_read: 0.15
enricher:
- model: gemini-3-flash-preview
+ model: gemini-3.5-flash
temperature: 1.0
top_p: 0.95
n: 1
max_tokens: 4096
price:
- input: 0.5
- output: 3.0
- cache_read: 0.05
+ input: 1.5
+ output: 9.0
+ cache_read: 0.15
coder:
model: gemini-3.1-pro-preview
@@ -133,7 +133,7 @@ coder:
cache_read: 0.2
installer:
- model: gemini-3-flash-preview
+ model: gemini-3.5-flash
temperature: 1.0
top_p: 0.95
n: 1
@@ -141,9 +141,9 @@ installer:
reasoning:
effort: low
price:
- input: 0.5
- output: 3.0
- cache_read: 0.05
+ input: 1.5
+ output: 9.0
+ cache_read: 0.15
pentester:
model: gemini-3.1-pro-preview
diff --git a/backend/pkg/providers/gemini/gemini_test.go b/backend/pkg/providers/gemini/gemini_test.go
index 1a3433bd..04f81c44 100644
--- a/backend/pkg/providers/gemini/gemini_test.go
+++ b/backend/pkg/providers/gemini/gemini_test.go
@@ -121,7 +121,7 @@ func TestGeminiSpecificFeatures(t *testing.T) {
}
// Test that we have current Gemini models
- expectedModels := []string{"gemini-2.5-flash", "gemini-2.5-pro", "gemini-2.0-flash"}
+ expectedModels := []string{"gemini-2.5-flash", "gemini-2.5-pro", "gemini-3.5-flash", "gemini-3.1-pro-preview"}
for _, expectedModel := range expectedModels {
found := false
for _, model := range models {
diff --git a/backend/pkg/providers/gemini/models.yml b/backend/pkg/providers/gemini/models.yml
index f446fc14..5070541d 100644
--- a/backend/pkg/providers/gemini/models.yml
+++ b/backend/pkg/providers/gemini/models.yml
@@ -1,6 +1,16 @@
-# Gemini 3.1 Series - Latest Flagship (February 2026)
+# Gemini 3.5 Series - Latest Stable Flash (May 2026)
+- name: gemini-3.5-flash
+ description: Gemini 3.5 Flash - Most intelligent Flash model with sustained frontier performance on agentic and coding tasks, superior search and grounding. Optimal for large-scale autonomous penetration testing, high-throughput security scanning, and continuous vulnerability analysis with hybrid reasoning and 1M token context
+ thinking: true
+ release_date: 2026-05-19
+ price:
+ input: 1.5
+ output: 9.0
+ cache_read: 0.15
+
+# Gemini 3.1 Series - Stable Flash-Lite + Pro Preview (Feb-May 2026)
- name: gemini-3.1-pro-preview
- description: Gemini 3.1 Pro - Latest flagship with refined performance, improved thinking, better token efficiency, and grounded factual consistency. Optimized for software engineering and agentic workflows with precise tool usage
+ description: Gemini 3.1 Pro - Latest flagship with refined performance, improved thinking, better token efficiency, and grounded factual consistency. Optimized for software engineering and agentic workflows with precise tool usage, sophisticated threat modeling, and deep multimodal reasoning for advanced penetration testing scenarios. 1M input / 65K output tokens, knowledge cutoff January 2025
thinking: true
release_date: 2026-02-19
price:
@@ -9,47 +19,26 @@
cache_read: 0.2
- name: gemini-3.1-pro-preview-customtools
- description: Gemini 3.1 Pro Custom Tools - Specialized endpoint optimized for agentic workflows using custom tools and bash, better at prioritizing custom tools (view_file, search_code) over standard tools
+ description: Gemini 3.1 Pro Custom Tools - Specialized endpoint optimized for agentic workflows mixing custom tools and bash, better at prioritizing registered custom tools (view_file, search_code) over bash commands. Ideal for autonomous security agents that orchestrate bespoke pentesting tool ecosystems. Same pricing as gemini-3.1-pro-preview
thinking: true
- release_date: 2026-02-19
+ release_date: 2026-02-23
price:
input: 2.0
output: 12.0
cache_read: 0.2
-- name: gemini-3.1-flash-lite-preview
- description: Gemini 3.1 Flash-Lite - Most cost-efficient multimodal model with fastest performance for high-frequency lightweight tasks, high-volume agentic tasks, simple data extraction, and extremely low-latency applications
+- name: gemini-3.1-flash-lite
+ description: Gemini 3.1 Flash-Lite - Most cost-efficient stable multimodal model with frontier-class performance rivaling larger models at a fraction of the cost. Perfect for high-volume agentic tasks, simple data extraction, high-frequency lightweight pentesting checks, and extremely low-latency security monitoring
thinking: true
- release_date: 2026-03-03
+ release_date: 2026-05-07
price:
input: 0.25
output: 1.5
cache_read: 0.025
-# Gemini 3 Series (DEPRECATED - Shutdown March 9, 2026)
-- name: gemini-3-pro-preview
- description: "[DEPRECATED] Gemini 3 Pro - Shutdown March 9, 2026. Migrate to Gemini 3.1 Pro Preview to avoid service disruption"
- thinking: true
- deprecated: true
- shutdown_date: 2026-03-09
- release_date: 2025-11-20
- price:
- input: 2.0
- output: 12.0
- cache_read: 0.2
-
-- name: gemini-3-flash-preview
- description: Gemini 3 Flash - Frontier intelligence with superior search and grounding, designed for speed and high-throughput security scanning
- thinking: true
- release_date: 2025-11-20
- price:
- input: 0.5
- output: 3.0
- cache_read: 0.05
-
# Gemini 2.5 Series - Advanced thinking models with enhanced reasoning capabilities
- name: gemini-2.5-pro
- description: Gemini 2.5 Pro - State-of-the-art multipurpose model excelling at coding and complex reasoning tasks, sophisticated threat modeling, and comprehensive penetration testing methodologies
+ description: Gemini 2.5 Pro - State-of-the-art multipurpose model excelling at coding and complex reasoning tasks, sophisticated threat modeling, and comprehensive penetration testing methodologies. Strong choice for deep exploit research and advanced code analysis (shutdown October 16, 2026)
thinking: true
release_date: 2025-06-17
price:
@@ -58,7 +47,7 @@
cache_read: 0.125
- name: gemini-2.5-flash
- description: Gemini 2.5 Flash - First hybrid reasoning model with 1M token context window and thinking budgets, best price-performance for large-scale security assessments and automated vulnerability analysis
+ description: Gemini 2.5 Flash - First hybrid reasoning model with 1M token context window and thinking budgets, best price-performance for large-scale security assessments and automated vulnerability analysis (shutdown October 16, 2026, recommended replacement gemini-3.5-flash)
thinking: true
release_date: 2025-06-17
price:
@@ -67,7 +56,7 @@
cache_read: 0.03
- name: gemini-2.5-flash-lite
- description: Gemini 2.5 Flash-Lite - Smallest and most cost-effective model built for at-scale usage, high-throughput security scanning and rapid vulnerability classification
+ description: Gemini 2.5 Flash-Lite - Smallest and most cost-effective 2.5 model built for at-scale usage, high-throughput security scanning, and rapid vulnerability classification (shutdown October 16, 2026, recommended replacement gemini-3.1-flash-lite)
thinking: true
release_date: 2025-07-22
price:
@@ -75,38 +64,19 @@
output: 0.4
cache_read: 0.01
-- name: gemini-2.5-flash-lite-preview-09-2025
- description: Gemini 2.5 Flash-Lite Preview - Latest preview optimized for cost-efficiency, high throughput, and high quality for continuous security monitoring
+# Gemma 4 Open-Source Series - Multimodal open-weight models from Google DeepMind (March 2026)
+- name: gemma-4-31b-it
+ description: Gemma 4 31B Instruction-Tuned - Largest open-source Gemma 4 dense model (~31B params) with 256K context window, multimodal text+image input, support for 140+ languages, and toggleable thinking process. Apache 2.0 license enables unrestricted on-premises deployment for privacy-sensitive penetration testing, customizable security analysis workflows, and air-gapped offensive research. Free of charge on Gemini API
thinking: true
- release_date: 2025-09-01
+ release_date: 2026-03-31
price:
- input: 0.1
- output: 0.4
- cache_read: 0.01
-
-# Gemini 2.0 Series - Balanced multimodal models for agent-based security operations
-- name: gemini-2.0-flash
- description: Gemini 2.0 Flash - Most balanced multimodal model with 1M token context window, built for the era of Agents, optimized for diverse security tasks and real-time threat monitoring
- thinking: false
- release_date: 2025-01-30
- price:
- input: 0.1
- output: 0.4
- cache_read: 0.025
-
-- name: gemini-2.0-flash-lite
- description: Gemini 2.0 Flash-Lite - Lightweight model perfect for continuous security monitoring, basic vulnerability scanning, and automated security alert processing
- thinking: false
- release_date: 2025-02-05
- price:
- input: 0.075
- output: 0.3
+ input: 0.0
+ output: 0.0
-# Specialized Open-Source Models
-- name: gemma-3-27b-it
- description: Gemma 3 - Open-source lightweight model built from Gemini technology, ideal for on-premises security operations, privacy-sensitive penetration testing, and customizable security analysis workflows
- thinking: false
- release_date: 2024-02-21
+- name: gemma-4-26b-a4b-it
+ description: Gemma 4 26B A4B Instruction-Tuned - Open-source Mixture-of-Experts model (~26B total / ~3.8B active params) with 256K context window, multimodal input, and configurable thinking. Highly efficient inference suitable for consumer GPUs, ideal for on-premises high-throughput security scanning, local vulnerability triage, and privacy-preserving offensive workflows. Apache 2.0 license, free of charge on Gemini API
+ thinking: true
+ release_date: 2026-03-31
price:
input: 0.0
output: 0.0
diff --git a/backend/pkg/providers/glm/config.yml b/backend/pkg/providers/glm/config.yml
index 215eee62..9e4da043 100644
--- a/backend/pkg/providers/glm/config.yml
+++ b/backend/pkg/providers/glm/config.yml
@@ -1,70 +1,143 @@
+# Z.AI GLM agent configuration.
+#
+# Strategy: glm-5.1 (newest flagship) for critical reasoning, glm-5-turbo (OpenClaw-
+# native, agent-optimized) for orchestration, glm-4.5-air for cheap utility.
+#
+# Thinking is controlled via extra_body.thinking.type ("enabled" / "disabled"). GLM is
+# permissive about temperature in either mode (no Kimi-style strict constraint).
+#
+# extra_body.thinking.clear_thinking=false enables Preserved Thinking (Z.AI docs):
+# on the standard API endpoint Z.AI defaults to clearing reasoning_content between
+# turns, which hurts agent loops with tool calls. Setting clear_thinking=false
+# preserves reasoning_content across turns, improving reasoning continuity and cache
+# hit rates. Required together with WithPreserveReasoningContent() in glm.go so that
+# langchaingo serializes reasoning_content back to the API.
+#
+# Note: langchaingo's IsReasoningModel matches glm-4.5/4.6/4.7 prefixes and force-
+# overrides temperature to 1.0; harmless for GLM but temp values for those models in
+# YAML are advisory. glm-5/glm-5.1/glm-5-turbo are not matched and pass through.
+
simple:
- model: "glm-4.7-flashx"
- temperature: 1.0
+ model: "glm-4.5-air"
+ temperature: 0.6
+ top_p: 0.9
n: 1
+ max_tokens: 8192
+ extra_body:
+ thinking:
+ type: disabled
+ tool_choice: auto
price:
- input: 0.07
- output: 0.40
- cache_read: 0.01
+ input: 0.20
+ output: 1.10
+ cache_read: 0.03
simple_json:
- model: "glm-4.7-flashx"
- temperature: 1.0
+ model: "glm-4.5-air"
+ temperature: 0.6
+ top_p: 0.9
n: 1
+ max_tokens: 4096
+ json: true
+ extra_body:
+ thinking:
+ type: disabled
+ tool_choice: auto
price:
- input: 0.07
- output: 0.40
- cache_read: 0.01
+ input: 0.20
+ output: 1.10
+ cache_read: 0.03
primary_agent:
- model: "glm-5"
+ model: "glm-5-turbo"
temperature: 1.0
+ top_p: 0.95
n: 1
+ max_tokens: 16384
+ extra_body:
+ thinking:
+ type: enabled
+ clear_thinking: false
+ tool_choice: auto
price:
- input: 1.00
- output: 3.20
- cache_read: 0.20
+ input: 1.20
+ output: 4.00
+ cache_read: 0.24
assistant:
- model: "glm-5"
+ model: "glm-5-turbo"
temperature: 1.0
+ top_p: 0.95
n: 1
+ max_tokens: 16384
+ extra_body:
+ thinking:
+ type: enabled
+ clear_thinking: false
+ tool_choice: auto
price:
- input: 1.00
- output: 3.20
- cache_read: 0.20
+ input: 1.20
+ output: 4.00
+ cache_read: 0.24
generator:
- model: "glm-5"
+ model: "glm-5.1"
temperature: 1.0
+ top_p: 0.95
n: 1
+ max_tokens: 32768
+ extra_body:
+ thinking:
+ type: enabled
+ clear_thinking: false
+ tool_choice: auto
price:
- input: 1.00
- output: 3.20
- cache_read: 0.20
+ input: 1.40
+ output: 4.40
+ cache_read: 0.26
refiner:
- model: "glm-5"
+ model: "glm-5.1"
temperature: 1.0
+ top_p: 0.95
n: 1
+ max_tokens: 32768
+ extra_body:
+ thinking:
+ type: enabled
+ clear_thinking: false
+ tool_choice: auto
price:
- input: 1.00
- output: 3.20
- cache_read: 0.20
+ input: 1.40
+ output: 4.40
+ cache_read: 0.26
adviser:
- model: "glm-5"
+ model: "glm-5.1"
temperature: 1.0
+ top_p: 0.95
n: 1
+ max_tokens: 16384
+ extra_body:
+ thinking:
+ type: enabled
+ clear_thinking: false
+ tool_choice: auto
price:
- input: 1.00
- output: 3.20
- cache_read: 0.20
+ input: 1.40
+ output: 4.40
+ cache_read: 0.26
reflector:
model: "glm-4.5-air"
- temperature: 0.7
+ temperature: 0.6
+ top_p: 0.9
n: 1
+ max_tokens: 8192
+ extra_body:
+ thinking:
+ type: disabled
+ tool_choice: auto
price:
input: 0.20
output: 1.10
@@ -72,8 +145,14 @@ reflector:
searcher:
model: "glm-4.5-air"
- temperature: 0.7
+ temperature: 0.6
+ top_p: 0.9
n: 1
+ max_tokens: 4096
+ extra_body:
+ thinking:
+ type: disabled
+ tool_choice: auto
price:
input: 0.20
output: 1.10
@@ -81,36 +160,63 @@ searcher:
enricher:
model: "glm-4.5-air"
- temperature: 0.7
+ temperature: 0.6
+ top_p: 0.9
n: 1
+ max_tokens: 4096
+ extra_body:
+ thinking:
+ type: disabled
+ tool_choice: auto
price:
input: 0.20
output: 1.10
cache_read: 0.03
coder:
- model: "glm-5"
+ model: "glm-5.1"
temperature: 1.0
+ top_p: 0.95
n: 1
+ max_tokens: 20480
+ extra_body:
+ thinking:
+ type: enabled
+ clear_thinking: false
+ tool_choice: auto
price:
- input: 1.00
- output: 3.20
- cache_read: 0.20
+ input: 1.40
+ output: 4.40
+ cache_read: 0.26
installer:
- model: "glm-4.7"
+ model: "glm-4.5-air"
temperature: 1.0
+ top_p: 0.95
n: 1
+ max_tokens: 16384
+ extra_body:
+ thinking:
+ type: enabled
+ clear_thinking: false
+ tool_choice: auto
price:
- input: 0.60
- output: 2.20
- cache_read: 0.11
+ input: 0.20
+ output: 1.10
+ cache_read: 0.03
pentester:
- model: "glm-4.7"
+ model: "glm-5.1"
temperature: 1.0
+ top_p: 0.95
n: 1
+ max_tokens: 16384
+ extra_body:
+ thinking:
+ type: enabled
+ clear_thinking: false
+ tool_choice: auto
price:
- input: 0.60
- output: 2.20
- cache_read: 0.11
+ input: 1.40
+ output: 4.40
+ cache_read: 0.26
diff --git a/backend/pkg/providers/glm/glm.go b/backend/pkg/providers/glm/glm.go
index 49431ee7..d4de0b46 100644
--- a/backend/pkg/providers/glm/glm.go
+++ b/backend/pkg/providers/glm/glm.go
@@ -82,11 +82,23 @@ func New(
return nil, err
}
+ // Z.AI GLM OpenAI-compatible API. Thinking mode is controlled via extra_body.thinking.type
+ // ("enabled"/"disabled") in the per-agent config. Unlike Kimi/DeepSeek, GLM does not use
+ // reasoning_effort and is permissive about temperature (accepts both 1.0 and 0.6 in
+ // thinking mode per Z.AI docs), so no WithModernReasoningFormat is needed.
+ // Note: langchaingo's reasoning.IsReasoningModel matches glm-4.5/4.6/4.7 prefixes and
+ // force-overrides temperature to 1.0, but GLM accepts any temperature so this is harmless.
+ // WithPreserveReasoningContent() is required for Preserved Thinking (clear_thinking=false
+ // in per-agent extra_body): on the standard API endpoint Z.AI defaults to clearing
+ // reasoning_content across turns, so we must both enable preservation server-side
+ // AND have langchaingo serialize reasoning_content back into assistant messages with
+ // tool calls. Without it the API will reject the request or silently drop history.
client, err := openai.New(
openai.WithToken(cfg.GLMAPIKey),
openai.WithModel(GLMAgentModel),
openai.WithBaseURL(cfg.GLMServerURL),
openai.WithHTTPClient(httpClient),
+ openai.WithPreserveReasoningContent(),
)
if err != nil {
return nil, err
diff --git a/backend/pkg/providers/glm/models.yml b/backend/pkg/providers/glm/models.yml
index 5506c6d4..76626eab 100644
--- a/backend/pkg/providers/glm/models.yml
+++ b/backend/pkg/providers/glm/models.yml
@@ -1,23 +1,31 @@
-# GLM-5 Series - Flagship Models
+# GLM-5.x Series - Latest generation (200K context, 128K max output, hybrid thinking)
+- name: glm-5.1
+ description: GLM-5.1 - Latest flagship designed for long-horizon tasks (8h sustained autonomous execution), Claude Opus 4.6-aligned coding, 200K context, hybrid thinking. Best for planning, mentor, and complex agentic engineering
+ thinking: true
+ price:
+ input: 1.40
+ output: 4.40
+ cache_read: 0.26
+
- name: glm-5
- description: GLM-5 - Flagship model with MoE architecture (744B/40B active), agentic engineering, 200K context, forced thinking mode, best for complex multi-stage tasks
+ description: GLM-5 - Foundation model for Agentic Engineering (MoE 744B/40B active), Claude Opus 4.5-level coding (SWE-bench Verified 77.8), 200K context, hybrid thinking. Cost-effective alternative to 5.1 for heavy reasoning
thinking: true
price:
input: 1.00
output: 3.20
cache_read: 0.20
-- name: glm-5-code
- description: GLM-5-Code - Code-specialized variant optimized for programming, 200K context, forced thinking, ideal for exploit development and offensive tools
+- name: glm-5-turbo
+ description: GLM-5-Turbo - OpenClaw-native model optimized for tool invocation, instruction following, scheduled/persistent tasks, and long-chain execution. 200K context, hybrid thinking. Ideal for orchestrator and assistant roles
thinking: true
price:
input: 1.20
- output: 5.00
- cache_read: 0.30
+ output: 4.00
+ cache_read: 0.24
# GLM-4.7 Series - Premium with Interleaved Thinking
- name: glm-4.7
- description: GLM-4.7 - Premium model with Interleaved Thinking before each response and tool call, 200K context, preserved thinking across multi-turn dialogues
+ description: GLM-4.7 - Premium with enhanced programming, stable multi-step reasoning/execution, better complex agent task performance, 200K context, preserved thinking across multi-turn
thinking: true
price:
input: 0.60
@@ -25,7 +33,7 @@
cache_read: 0.11
- name: glm-4.7-flashx
- description: GLM-4.7-FlashX - High-speed paid version with priority GPU access, 200K context, hybrid thinking, best price/performance for batch tasks
+ description: GLM-4.7-FlashX - Paid high-speed version with priority GPU access, 200K context, hybrid thinking, best price/performance for batch utility tasks
thinking: true
price:
input: 0.07
@@ -42,7 +50,7 @@
# GLM-4.6 Series - Balanced with Auto Thinking
- name: glm-4.6
- description: GLM-4.6 - Balanced model with 200K context, auto-thinking (model decides when to reason), streaming tool calls support, 30% token efficient
+ description: GLM-4.6 - Balanced model with 200K context, auto-thinking, streaming tool calls support, 30% token efficient
thinking: true
price:
input: 0.60
@@ -59,7 +67,7 @@
cache_read: 0.11
- name: glm-4.5-x
- description: GLM-4.5-X - Ultra-fast premium version with lowest latency, 128K context, auto-thinking, most expensive for real-time critical operations
+ description: GLM-4.5-X - Ultra-fast premium with lowest latency, 128K context, auto-thinking. Most expensive for real-time critical operations
thinking: true
price:
input: 2.20
@@ -67,7 +75,7 @@
cache_read: 0.45
- name: glm-4.5-air
- description: GLM-4.5-Air - Cost-effective lightweight model, MoE 106B/12B active, 128K context, auto-thinking, best price/quality ratio for batch processing
+ description: GLM-4.5-Air - Cost-effective lightweight MoE 106B/12B active, 128K context, auto-thinking, best price/quality ratio for utility agents and continuous monitoring
thinking: true
price:
input: 0.20
@@ -90,11 +98,10 @@
output: 0.00
cache_read: 0.00
-# GLM-4 Legacy - Dense Architecture
+# GLM-4 Legacy - Dense Architecture (no thinking)
- name: glm-4-32b-0414-128k
- description: GLM-4-32B - Ultra-budget dense 32B model, 128K context, NO thinking mode, max output 16K, cheapest for high-volume parsing without reasoning
+ description: GLM-4-32B - Ultra-budget dense 32B model, 128K context, NO thinking mode, max output 16K. Cheapest for high-volume parsing/structured tasks without reasoning
thinking: false
price:
input: 0.10
output: 0.10
- cache_read: 0.00
diff --git a/backend/pkg/providers/kimi/config.yml b/backend/pkg/providers/kimi/config.yml
index e07764f1..66cf51e4 100644
--- a/backend/pkg/providers/kimi/config.yml
+++ b/backend/pkg/providers/kimi/config.yml
@@ -1,139 +1,224 @@
+# Kimi (Moonshot) agent configuration.
+#
+# Strategy: kimi-k2.5 for utility/orchestration, kimi-k2.6 for critical reasoning.
+#
+# K2.6/K2.5 API constraints (any deviation -> invalid_request_error):
+# thinking: temperature=1.0, top_p=0.95, n=1, thinking.keep="all" (k2.6 only)
+# non-thinking: temperature=0.6, top_p=0.95, n=1
+# always: presence_penalty=0, frequency_penalty=0, tool_choice in {auto, none}
+#
+# WORKAROUND (drop after fix in langchaingo): for kimi-k2.5 non-thinking agents we
+# duplicate temperature=0.6 into extra_body. Reason: langchaingo's
+# reasoning.IsReasoningModel matches substring "2.5" and force-overrides temperature
+# to 1.0 in createChatRequest. extra_body merges with priority via maps.Copy in
+# openaiclient.createChat, so it bypasses the override. Remove the duplicate
+# extra_body.temperature once langchaingo stops matching k2.5/k2.6 as reasoning.
+#
+# Legacy kimi-k2-* models (turbo-preview, 0905-preview, 0711-preview, thinking,
+# thinking-turbo) were deprecated by Moonshot on 2026-05-25. Do not reintroduce.
+
simple:
- model: "kimi-k2-turbo-preview"
- temperature: 0.3
+ model: kimi-k2.5
+ temperature: 0.6
+ top_p: 0.95
n: 1
max_tokens: 8192
extra_body:
- tool_choice: "auto"
+ temperature: 0.6
+ thinking:
+ type: disabled
+ tool_choice: auto
price:
- input: 1.15
- output: 8.0
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
simple_json:
- model: "kimi-k2-turbo-preview"
- temperature: 0.3
+ model: kimi-k2.5
+ temperature: 0.6
+ top_p: 0.95
n: 1
max_tokens: 4096
json: true
+ extra_body:
+ temperature: 0.6
+ thinking:
+ type: disabled
+ tool_choice: auto
price:
- input: 1.15
- output: 8.0
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
primary_agent:
- model: "kimi-k2.5"
+ model: kimi-k2.5
temperature: 1.0
+ top_p: 0.95
n: 1
max_tokens: 16384
extra_body:
- tool_choice: "auto"
+ thinking:
+ type: enabled
+ tool_choice: auto
price:
- input: 0.6
- output: 3.0
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
assistant:
- model: "kimi-k2.5"
+ model: kimi-k2.5
temperature: 1.0
+ top_p: 0.95
n: 1
max_tokens: 16384
extra_body:
- tool_choice: "auto"
+ thinking:
+ type: enabled
+ tool_choice: auto
price:
- input: 0.6
- output: 3.0
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
generator:
- model: "kimi-k2.5"
+ model: kimi-k2.6
temperature: 1.0
+ top_p: 0.95
n: 1
max_tokens: 32768
extra_body:
- tool_choice: "auto"
+ thinking:
+ type: enabled
+ keep: all
+ tool_choice: auto
price:
- input: 0.6
- output: 3.0
+ input: 0.95
+ output: 4.00
+ cache_read: 0.16
refiner:
- model: "kimi-k2.5"
+ model: kimi-k2.6
temperature: 1.0
+ top_p: 0.95
n: 1
- max_tokens: 20480
+ max_tokens: 32768
extra_body:
- tool_choice: "auto"
+ thinking:
+ type: enabled
+ keep: all
+ tool_choice: auto
price:
- input: 0.6
- output: 3.0
+ input: 0.95
+ output: 4.00
+ cache_read: 0.16
adviser:
- model: "kimi-k2.5"
+ model: kimi-k2.6
temperature: 1.0
+ top_p: 0.95
n: 1
max_tokens: 8192
+ extra_body:
+ thinking:
+ type: enabled
+ keep: all
+ tool_choice: auto
price:
- input: 0.6
- output: 3.0
+ input: 0.95
+ output: 4.00
+ cache_read: 0.16
reflector:
- model: "kimi-k2-0905-preview"
- temperature: 0.7
+ model: kimi-k2.5
+ temperature: 0.6
+ top_p: 0.95
n: 1
max_tokens: 4096
extra_body:
- tool_choice: "auto"
+ temperature: 0.6
+ thinking:
+ type: disabled
+ tool_choice: auto
price:
- input: 0.6
- output: 2.5
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
searcher:
- model: "kimi-k2-0905-preview"
- temperature: 0.7
+ model: kimi-k2.5
+ temperature: 0.6
+ top_p: 0.95
n: 1
max_tokens: 4096
extra_body:
- tool_choice: "auto"
+ temperature: 0.6
+ thinking:
+ type: disabled
+ tool_choice: auto
price:
- input: 0.6
- output: 2.5
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
enricher:
- model: "kimi-k2-0905-preview"
- temperature: 0.7
+ model: kimi-k2.5
+ temperature: 0.6
+ top_p: 0.95
n: 1
max_tokens: 4096
extra_body:
- tool_choice: "auto"
+ temperature: 0.6
+ thinking:
+ type: disabled
+ tool_choice: auto
price:
- input: 0.6
- output: 2.5
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
coder:
- model: "kimi-k2.5"
+ model: kimi-k2.6
temperature: 1.0
+ top_p: 0.95
n: 1
max_tokens: 20480
extra_body:
- tool_choice: "auto"
+ thinking:
+ type: enabled
+ keep: all
+ tool_choice: auto
price:
- input: 0.6
- output: 3.0
+ input: 0.95
+ output: 4.00
+ cache_read: 0.16
installer:
- model: "kimi-k2-turbo-preview"
- temperature: 0.7
+ model: kimi-k2.5
+ temperature: 1.0
+ top_p: 0.95
n: 1
max_tokens: 16384
extra_body:
- tool_choice: "auto"
+ thinking:
+ type: enabled
+ tool_choice: auto
price:
- input: 1.15
- output: 8.0
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
pentester:
- model: "kimi-k2-turbo-preview"
- temperature: 0.8
+ model: kimi-k2.6
+ temperature: 1.0
+ top_p: 0.95
n: 1
max_tokens: 16384
extra_body:
- tool_choice: "auto"
+ thinking:
+ type: enabled
+ keep: all
+ tool_choice: auto
price:
- input: 1.15
- output: 8.0
+ input: 0.95
+ output: 4.00
+ cache_read: 0.16
diff --git a/backend/pkg/providers/kimi/kimi.go b/backend/pkg/providers/kimi/kimi.go
index 8fe3a541..a57672f3 100644
--- a/backend/pkg/providers/kimi/kimi.go
+++ b/backend/pkg/providers/kimi/kimi.go
@@ -19,7 +19,11 @@ import (
//go:embed config.yml models.yml
var configFS embed.FS
-const KimiAgentModel = "kimi-k2-turbo-preview"
+// KimiAgentModel is the fallback model used when no agent-specific configuration exists.
+// kimi-k2.5 is chosen as cost-effective default ($0.60/$3.00 input/output vs $0.95/$4.00 for k2.6).
+// All legacy kimi-k2-* models (turbo-preview, 0905-preview, 0711-preview, thinking, thinking-turbo)
+// were deprecated by Moonshot on 2026-05-25 and must not be used.
+const KimiAgentModel = "kimi-k2.5"
const KimiToolCallIDTemplate = "{f}:{r:1:d}"
@@ -83,6 +87,14 @@ func New(
return nil, err
}
+ // Kimi K2.6/K2.5 OpenAI-compatible API requires legacy string form for reasoning
+ // (no WithModernReasoningFormat). Thinking mode is controlled via
+ // extra_body.thinking.type ("enabled"/"disabled") in the per-agent config, with
+ // thinking.keep="all" required for multi-turn tool calls to preserve historical
+ // reasoning_content (otherwise Moonshot returns "thinking is enabled but
+ // reasoning_content is missing in assistant tool call message").
+ // WithPreserveReasoningContent() keeps reasoning_content in TextContent parts
+ // before ToolCall in assistant messages — Kimi rejects requests without it.
client, err := openai.New(
openai.WithToken(cfg.KimiAPIKey),
openai.WithModel(KimiAgentModel),
diff --git a/backend/pkg/providers/kimi/models.yml b/backend/pkg/providers/kimi/models.yml
index 72fe3def..b18d33e3 100644
--- a/backend/pkg/providers/kimi/models.yml
+++ b/backend/pkg/providers/kimi/models.yml
@@ -1,86 +1,60 @@
-# Kimi K2.5 series - Most versatile multimodal model
-- name: kimi-k2.5
- description: Kimi K2.5 - Most intelligent and versatile model with native multimodal architecture supporting visual/text input, thinking/non-thinking modes, 256k context, cache support
+# Kimi K2.6 - Latest flagship multimodal model with native architecture
+- name: kimi-k2.6
+ description: Kimi K2.6 - Latest and most intelligent multimodal model with native architecture, stronger long-term code writing, improved instruction compliance and self-correction. Supports text/image/video input, thinking/non-thinking modes, 256k context, automatic context caching, ToolCalls, JSON Mode, internet search
thinking: true
price:
- input: 0.60
- output: 3.00
-
-# Kimi K2 series - MoE foundation models with 1T total params, 32B activated
-- name: kimi-k2-0905-preview
- description: Kimi K2-0905 - Enhanced agentic coding, improved frontend code quality and context understanding, 256k context
- thinking: false
- price:
- input: 0.60
- output: 2.50
+ input: 0.95
+ output: 4.00
+ cache_read: 0.16
-- name: kimi-k2-0711-preview
- description: Kimi K2-0711 - MoE base model with powerful code and agent capabilities, 128k context
- thinking: false
- price:
- input: 0.60
- output: 2.50
-
-- name: kimi-k2-turbo-preview
- description: Kimi K2 Turbo - High-speed version of K2-0905, 60-100 tokens/sec output speed, 256k context
- thinking: false
- price:
- input: 1.15
- output: 8.00
-
-- name: kimi-k2-thinking
- description: Kimi K2 Thinking - Long-term thinking model with multi-step tool usage and deep reasoning, 256k context
+# Kimi K2.5 - Previous-gen multimodal model (cost-optimized alternative to K2.6)
+- name: kimi-k2.5
+ description: Kimi K2.5 - Multimodal model supporting text/image/video input, thinking/non-thinking modes, 256k context, automatic context caching, ToolCalls, JSON Mode, Partial Mode, internet search. 36% cheaper input than K2.6
thinking: true
price:
input: 0.60
- output: 2.50
-
-- name: kimi-k2-thinking-turbo
- description: Kimi K2 Thinking Turbo - High-speed thinking model, 60-100 tokens/sec, excels at deep reasoning, 256k context
- thinking: true
- price:
- input: 1.15
- output: 8.00
+ output: 3.00
+ cache_read: 0.10
-# Moonshot V1 series - General text generation models
+# Moonshot V1 - Generation models with flexible parameters (no thinking, free temp/top_p)
- name: moonshot-v1-8k
- description: Moonshot V1-8K - Suitable for generating short texts, 8k context
+ description: Moonshot V1-8K - Short text generation, 8k context window (input+output combined). Flexible OpenAI-standard parameters
thinking: false
price:
input: 0.20
output: 2.00
- name: moonshot-v1-32k
- description: Moonshot V1-32K - Suitable for generating long texts, 32k context
+ description: Moonshot V1-32K - Long text generation, 32k context window. Flexible OpenAI-standard parameters
thinking: false
price:
input: 1.00
output: 3.00
- name: moonshot-v1-128k
- description: Moonshot V1-128K - Suitable for generating very long texts, 128k context
+ description: Moonshot V1-128K - Very long text generation, 128k context window. Flexible OpenAI-standard parameters
thinking: false
price:
input: 2.00
output: 5.00
-# Moonshot V1 Vision series - Multimodal models with image understanding
+# Moonshot V1 Vision - Multimodal generation models with image understanding
- name: moonshot-v1-8k-vision-preview
- description: Moonshot V1-8K Vision - Vision model that understands image content and outputs text, 8k context
+ description: Moonshot V1-8K Vision - Image understanding with text output, 8k context window
thinking: false
price:
input: 0.20
output: 2.00
- name: moonshot-v1-32k-vision-preview
- description: Moonshot V1-32K Vision - Vision model that understands image content and outputs text, 32k context
+ description: Moonshot V1-32K Vision - Image understanding with text output, 32k context window
thinking: false
price:
input: 1.00
output: 3.00
- name: moonshot-v1-128k-vision-preview
- description: Moonshot V1-128K Vision - Vision model that understands image content and outputs text, 128k context
+ description: Moonshot V1-128K Vision - Image understanding with text output, 128k context window
thinking: false
price:
input: 2.00
diff --git a/backend/pkg/providers/qwen/config.yml b/backend/pkg/providers/qwen/config.yml
index 2e98534a..96d00ce6 100644
--- a/backend/pkg/providers/qwen/config.yml
+++ b/backend/pkg/providers/qwen/config.yml
@@ -6,7 +6,7 @@ simple:
price:
input: 0.1
output: 0.4
- cache_read: 0.02
+ cache_read: 0.01
simple_json:
model: "qwen3.5-flash"
@@ -17,57 +17,57 @@ simple_json:
price:
input: 0.1
output: 0.4
- cache_read: 0.02
+ cache_read: 0.01
primary_agent:
- model: "qwen3.5-plus"
+ model: "qwen3.6-plus"
temperature: 1.0
n: 1
max_tokens: 16384
price:
- input: 0.4
- output: 2.4
- cache_read: 0.08
+ input: 0.5
+ output: 3.0
+ cache_read: 0.05
assistant:
- model: "qwen3.5-plus"
+ model: "qwen3.6-plus"
temperature: 1.0
n: 1
max_tokens: 16384
price:
- input: 0.4
- output: 2.4
- cache_read: 0.08
+ input: 0.5
+ output: 3.0
+ cache_read: 0.05
generator:
- model: "qwen3-max"
+ model: "qwen3.7-max"
temperature: 1.0
n: 1
max_tokens: 32768
price:
- input: 2.4
- output: 12.0
- cache_read: 0.48
+ input: 2.5
+ output: 7.5
+ cache_read: 0.5
refiner:
- model: "qwen3-max"
+ model: "qwen3.7-max"
temperature: 1.0
n: 1
max_tokens: 20480
price:
- input: 2.4
- output: 12.0
- cache_read: 0.48
+ input: 2.5
+ output: 7.5
+ cache_read: 0.5
adviser:
- model: "qwen3-max"
+ model: "qwen3.7-max"
temperature: 1.0
n: 1
max_tokens: 8192
price:
- input: 2.4
- output: 12.0
- cache_read: 0.48
+ input: 2.5
+ output: 7.5
+ cache_read: 0.5
reflector:
model: "qwen3.5-flash"
@@ -77,7 +77,7 @@ reflector:
price:
input: 0.1
output: 0.4
- cache_read: 0.02
+ cache_read: 0.01
searcher:
model: "qwen3.5-flash"
@@ -87,7 +87,7 @@ searcher:
price:
input: 0.1
output: 0.4
- cache_read: 0.02
+ cache_read: 0.01
enricher:
model: "qwen3.5-flash"
@@ -97,34 +97,34 @@ enricher:
price:
input: 0.1
output: 0.4
- cache_read: 0.02
+ cache_read: 0.01
coder:
- model: "qwen3.5-plus"
+ model: "qwen3-coder-plus"
temperature: 1.0
n: 1
max_tokens: 20480
price:
- input: 0.4
- output: 2.4
- cache_read: 0.08
+ input: 1.0
+ output: 5.0
+ cache_read: 0.2
installer:
- model: "qwen3.5-plus"
+ model: "qwen3-coder-flash"
temperature: 0.7
n: 1
max_tokens: 16384
price:
- input: 0.4
- output: 2.4
- cache_read: 0.08
+ input: 0.3
+ output: 1.5
+ cache_read: 0.06
pentester:
- model: "qwen3.5-plus"
+ model: "qwen3.6-plus"
temperature: 0.8
n: 1
max_tokens: 16384
price:
- input: 0.4
- output: 2.4
- cache_read: 0.08
+ input: 0.5
+ output: 3.0
+ cache_read: 0.05
diff --git a/backend/pkg/providers/qwen/models.yml b/backend/pkg/providers/qwen/models.yml
index c99208a8..6771f9d7 100644
--- a/backend/pkg/providers/qwen/models.yml
+++ b/backend/pkg/providers/qwen/models.yml
@@ -1,269 +1,272 @@
-# Wide availability models (available in most regions)
-- name: qwen3-max
- description: Wide (Intr/Global/CH). Latest flagship reasoning model with thinking capability, strong instruction following and complex task handling
+# Qwen3.7 Series - Latest flagship Max (May 2026)
+- name: qwen3.7-max
+ description: Qwen3.7 Max - Next-generation flagship designed for the agent-centric era with excellence in programming, productivity tasks, and long-term autonomous execution. Pure-text interface optimal for sophisticated penetration testing automation and complex agentic security workflows
thinking: true
+ release_date: 2026-05-21
price:
- input: 2.4
- output: 12.0
- cache_read: 0.48
+ input: 2.5
+ output: 7.5
+ cache_read: 0.5
-- name: qwen3-max-preview
- description: Wide (Intr/Global/CH). Preview version with extended thinking capabilities for complex analysis
+# Qwen3.6 Series - Native vision-language models with enhanced agentic coding (April 2026)
+- name: qwen3.6-max-preview
+ description: Qwen3.6 Max Preview - Builds on Qwen3-Max and Qwen3.6-Plus with enhanced vibe coding, more efficient coding agent execution, significantly improved front-end skills, and upgraded long-tail knowledge retention. Ideal for advanced security code analysis and complex multi-step penetration testing tasks (lowest <=128k tier pricing)
thinking: true
+ release_date: 2026-04-14
price:
- input: 2.4
- output: 12.0
- cache_read: 0.48
+ input: 1.3
+ output: 7.8
+ cache_read: 0.13
-- name: qwen-max
- description: Intr/CH. Flagship reasoning model with strong instruction following, suitable for complex security analysis
- thinking: false
+- name: qwen3.6-plus
+ description: Qwen3.6 Plus - Native vision-language model with state-of-the-art performance and significant improvements over 3.5 series in agentic coding, front-end programming, OCR, and object localization. Ideal for multimodal pentesting scenarios requiring screen analysis and code reasoning (lowest <=256k tier pricing)
+ thinking: true
+ release_date: 2026-04-01
price:
- input: 1.6
- output: 6.4
- cache_read: 0.32
+ input: 0.5
+ output: 3.0
+ cache_read: 0.05
-- name: qwen3.5-plus
- description: Wide (Intr/Global/CH). Advanced balanced model with thinking mode support, excellent for complex dialogue and analysis
+- name: qwen3.6-flash
+ description: Qwen3.6 Flash - Native vision-language Flash model with significant performance boost over 3.5-Flash, excels in agentic coding, math/code reasoning, spatial intelligence, and object detection. Optimal for high-throughput security scanning with multimodal awareness
thinking: true
+ release_date: 2026-04-17
price:
- input: 0.4
- output: 2.4
- cache_read: 0.08
+ input: 0.25
+ output: 1.5
+ cache_read: 0.025
-- name: qwen-plus
- description: Wide (Intr/Global/US/CH). Balanced performance model with thinking mode, suitable for general dialogue and code generation
+- name: qwen3.6-35b-a3b
+ description: Qwen3.6 35B Open-Source - MoE variant (35B params, ~3B active) of the Qwen3.6 native vision-language series. Cost-effective on-premises option for continuous security monitoring with multimodal awareness
+ thinking: true
+ release_date: 2026-04-22
+ price:
+ input: 0.248
+ output: 1.485
+
+- name: qwen3.6-27b
+ description: Qwen3.6 27B Open-Source - Native vision-language model on hybrid architecture, significantly improved performance over 3.5 series at the same scale. Ideal for on-premises pentesting with privacy-sensitive multimodal workflows
+ thinking: true
+ release_date: 2026-04-22
+ price:
+ input: 0.6
+ output: 3.6
+
+# Qwen3.5 Series - Native vision-language with hybrid linear attention + sparse MoE
+- name: qwen3.5-plus
+ description: Qwen3.5 Plus - Native vision-language model with outstanding performance comparable to latest SOTA models, significant leaps in both pure-text and multimodal capabilities. Excellent for balanced complex pentesting analysis with multimodal context
thinking: true
+ release_date: 2026-04-23
price:
input: 0.4
- output: 4.0
- cache_read: 0.08
+ output: 2.4
+ cache_read: 0.04
- name: qwen3.5-flash
- description: Wide (Intr/Global/CH). Ultra-fast lightweight model optimized for high-throughput scenarios
+ description: Qwen3.5 Flash - Native vision-language Flash with outstanding performance and pure-text capabilities. Ultra-fast lightweight model optimized for high-throughput security scanning and rapid vulnerability classification
thinking: true
+ release_date: 2026-02-23
price:
input: 0.1
output: 0.4
- cache_read: 0.02
+ cache_read: 0.01
-- name: qwen-flash
- description: Wide (Intr/Global/US/CH). Fast lightweight model with context caching support for efficient processing
- thinking: false
+- name: qwen3.5-397b-a17b
+ description: Qwen3.5 397B Open-Source - Largest open-source model in 3.5 series with hybrid architecture (397B params, ~17B active) and exceptional reasoning capabilities. Suitable for deep on-premises security research and sophisticated threat modeling
+ thinking: true
+ release_date: 2026-02-23
price:
- input: 0.05
- output: 0.4
- cache_read: 0.01
+ input: 0.6
+ output: 3.6
-- name: qwen-turbo
- description: Intr/CH. Fastest lightweight model with thinking mode, suitable for simple tasks and real-time response (deprecated, use qwen-flash)
+- name: qwen3.5-122b-a10b
+ description: Qwen3.5 122B Open-Source - Large open-source MoE model (122B params, ~10B active) with strong performance balance for complex on-premises security analysis
thinking: true
+ release_date: 2026-02-23
price:
- input: 0.05
- output: 0.5
- cache_read: 0.01
+ input: 0.4
+ output: 3.2
-- name: qwq-plus
- description: Intr/CH. Deep reasoning model with extended chain-of-thought capabilities for complex logic and security analysis
+- name: qwen3.5-35b-a3b
+ description: Qwen3.5 35B Open-Source - Efficient open-source MoE model (35B params, ~3B active) optimized for cost-effective inference in continuous security monitoring
+ thinking: true
+ release_date: 2026-02-23
+ price:
+ input: 0.25
+ output: 2.0
+
+- name: qwen3.5-27b
+ description: Qwen3.5 27B Open-Source - Native vision-language model with hybrid linear attention and sparse MoE architecture for higher inference efficiency. Ideal for on-premises security analysis with multimodal capabilities
thinking: true
+ release_date: 2026-02-23
price:
- input: 0.8
+ input: 0.3
output: 2.4
- cache_read: 0.16
-# Region-specific models
-- name: qwen-plus-us
- description: US only. Balanced performance model optimized for US region deployment
+# Qwen3 Series - Flagship hybrid thinking models
+- name: qwen3-max
+ description: Qwen3 Max - Flagship reasoning model with specialized upgrades in agent programming and tool invocation, SOTA performance in its field. Optimized for complex agent scenarios requiring sophisticated multi-step pentesting workflows (lowest <=32k tier pricing)
+ thinking: true
+ release_date: 2026-01-23
+ price:
+ input: 1.2
+ output: 6.0
+ cache_read: 0.24
+
+- name: qwen-plus
+ description: Qwen Plus - Enhanced super-large-scale language model with Qwen3 backbone, integrated thinking/non-thinking modes switchable mid-conversation. Versatile choice for general pentesting dialogue, code generation, and balanced security analysis (thinking-mode output pricing shown)
thinking: true
+ release_date: 2025-12-01
price:
input: 0.4
output: 4.0
cache_read: 0.08
-- name: qwen-long-latest
- description: CH only. Specialized model for ultra-long context processing up to 10M tokens
+- name: qwen-flash
+ description: Qwen Flash - Qwen3-series Flash model with seamless thinking/non-thinking mode switching, excels at complex thinking tasks with enhanced instruction adherence. Supports 1M context with tiered pricing, ideal for high-volume security scanning
+ thinking: true
+ release_date: 2025-08-01
+ price:
+ input: 0.05
+ output: 0.4
+ cache_read: 0.01
+
+# Qwen3 Coder Series - Code-specialized models with agentic capabilities
+- name: qwen3-coder-plus
+ description: Qwen3 Coder Plus - Qwen3-based code generation model with strong coding agent capability, excels at tool invocation and environment interaction, enables autonomous programming. Ideal for exploit development, security code review, and CVE research workflows (lowest <=32k tier pricing)
thinking: false
+ release_date: 2025-09-23
price:
- input: 0.072
- output: 0.287
- cache_read: 0.014
+ input: 1.0
+ output: 5.0
+ cache_read: 0.2
-# Open source models - Qwen3.5 series
-- name: qwen3.5-397b-a17b
- description: Wide (Intr/Global/CH). Largest open-source model with 397B parameters, exceptional reasoning capabilities
- thinking: true
+- name: qwen3-coder-flash
+ description: Qwen3 Coder Flash - Fast code generation model inheriting Qwen3-Coder-Plus capabilities with multi-turn tool interaction, repository-level understanding, and enhanced tool-calling stability. Cost-effective for systematic security code analysis and automated vulnerability triage
+ thinking: false
+ release_date: 2025-07-29
price:
- input: 0.6
- output: 3.6
- cache_read: 0.12
+ input: 0.3
+ output: 1.5
+ cache_read: 0.06
-- name: qwen3.5-122b-a10b
- description: Wide (Intr/Global/CH). Large open-source model with 122B parameters, strong performance
+- name: qwen3-coder-next
+ description: Qwen3 Coder Next - Open-source code generation model from Qwen3 series with hybrid/thinking/non-thinking variants achieving SOTA capabilities at the same scale. Suitable for on-premises security code analysis and air-gapped offensive research
+ thinking: false
+ release_date: 2026-02-20
+ price:
+ input: 0.3
+ output: 1.5
+
+- name: qwen3-coder-30b-a3b-instruct
+ description: Qwen3 Coder 30B Open-Source - MoE code model (30B params, ~3B active) optimized for efficient repository-scale code understanding. Good for cost-effective on-premises security code review and exploit generation
+ thinking: false
+ price:
+ input: 0.45
+ output: 2.25
+
+- name: qwen3-coder-480b-a35b-instruct
+ description: Qwen3 Coder 480B Open-Source - Largest open-source coder MoE (480B params, ~35B active) with maximum coding capability for advanced exploit development, complex vulnerability discovery, and large-scale code refactoring in security research
+ thinking: false
+ price:
+ input: 1.5
+ output: 7.5
+
+# Qwen3-VL Series - Vision-language models for browser screenshot analysis and visual reconnaissance
+- name: qwen3-vl-plus
+ description: Qwen3-VL Plus - Vision-language model integrating thinking and non-thinking modes with world-leading visual agent capabilities. Features upgrades in visual coding, spatial perception, multimodal reasoning, and ultra-long video understanding. Ideal for Searcher agent browser screenshot analysis and visual reconnaissance (lowest <=32k tier pricing)
thinking: true
+ release_date: 2025-12-18
price:
- input: 0.4
- output: 3.2
- cache_read: 0.08
+ input: 0.2
+ output: 1.6
+ cache_read: 0.04
-- name: qwen3.5-27b
- description: Wide (Intr/Global/CH). Medium open-source model with 27B parameters, balanced performance
+- name: qwen3-vl-flash
+ description: Qwen3-VL Flash - Small-scale vision understanding model with thinking/non-thinking modes, outperforms open-source Qwen3-VL-30B-A3B at faster response speeds. Equipped with 2D/3D visual localization for complex real-world tasks like browser automation and screenshot triage (lowest <=32k tier pricing)
thinking: true
+ release_date: 2026-01-22
price:
- input: 0.3
- output: 2.4
- cache_read: 0.06
+ input: 0.05
+ output: 0.4
+ cache_read: 0.01
-- name: qwen3.5-35b-a3b
- description: Wide (Intr/Global/CH). Efficient open-source model with 35B parameters and 3B active
+# QVQ Series - Visual reasoning models with chain-of-thought
+- name: qvq-max
+ description: QVQ Max - Visual reasoning model with visual input and chain-of-thought output, stronger capabilities in mathematics, programming, visual analysis, creation, and general tasks. Suitable for deep visual security analysis requiring step-by-step reasoning on screenshots or diagrams
thinking: true
+ release_date: 2025-03-26
price:
- input: 0.25
- output: 2.0
- cache_read: 0.05
+ input: 1.2
+ output: 4.8
-# Open source models - Qwen3 series
+# Qwen3 Open Source Series - Hybrid thinking dense and MoE models
- name: qwen3-next-80b-a3b-thinking
- description: Wide (Intr/Global/CH). Next-gen 80B model optimized for thinking mode only
+ description: Qwen3-Next 80B Open-Source - Next-gen MoE thinking model (80B params, ~3B active) with SOTA reasoning capabilities. Suitable for deep security analysis and methodical attack planning
thinking: true
price:
input: 0.15
- output: 1.434
- cache_read: 0.03
+ output: 1.2
- name: qwen3-next-80b-a3b-instruct
- description: Wide (Intr/Global/CH). Next-gen 80B instruction-following model, non-thinking mode
+ description: Qwen3-Next 80B Instruct Open-Source - Non-thinking variant for efficient instruction-following, ideal for high-throughput security automation tasks
thinking: false
price:
input: 0.15
output: 1.2
- cache_read: 0.03
- name: qwen3-235b-a22b
- description: Wide (Intr/Global/CH). Dual-mode 235B model supporting both thinking and non-thinking modes
+ description: Qwen3 235B Open-Source - Dual-mode MoE model (235B params, ~22B active) supporting both thinking and non-thinking modes for versatile security workloads (thinking-mode output pricing shown)
thinking: true
price:
input: 0.7
output: 8.4
- cache_read: 0.14
- name: qwen3-32b
- description: Wide (Intr/Global/CH). Versatile 32B model with dual-mode capabilities
+ description: Qwen3 32B Open-Source - Versatile dense model with dual-mode capabilities, suitable for balanced on-premises pentesting workflows
thinking: true
price:
- input: 0.287
- output: 2.868
- cache_read: 0.057
+ input: 0.16
+ output: 0.64
- name: qwen3-30b-a3b
- description: Wide (Intr/Global/CH). Efficient 30B model with MoE architecture
+ description: Qwen3 30B Open-Source - Efficient MoE model (30B params, ~3B active) for cost-effective continuous security monitoring (thinking-mode output pricing shown)
thinking: true
price:
input: 0.2
output: 2.4
- cache_read: 0.04
- name: qwen3-14b
- description: Wide (Intr/Global/CH). Medium-sized 14B model with good performance-cost balance
+ description: Qwen3 14B Open-Source - Medium-sized dense model with good performance-cost balance for routine security assessments (thinking-mode output pricing shown)
thinking: true
price:
input: 0.35
output: 4.2
- cache_read: 0.07
- name: qwen3-8b
- description: Wide (Intr/Global/CH). Compact 8B model optimized for efficiency
+ description: Qwen3 8B Open-Source - Compact dense model optimized for efficient inference in lightweight security tasks (thinking-mode output pricing shown)
thinking: true
price:
input: 0.18
output: 2.1
- cache_read: 0.036
- name: qwen3-4b
- description: Intr/CH. Lightweight 4B model for simple tasks
+ description: Qwen3 4B Open-Source - Lightweight model for simple security tasks and high-frequency scanning (thinking-mode output pricing shown)
thinking: true
price:
input: 0.11
output: 1.26
- cache_read: 0.022
- name: qwen3-1.7b
- description: Intr/CH. Ultra-compact 1.7B model for basic tasks
+ description: Qwen3 1.7B Open-Source - Ultra-compact model for basic security checks at minimal resource cost (thinking-mode output pricing shown)
thinking: true
price:
input: 0.11
output: 1.26
- cache_read: 0.022
- name: qwen3-0.6b
- description: Intr/CH. Smallest 0.6B model for minimal resource usage
+ description: Qwen3 0.6B Open-Source - Smallest Qwen3 model for minimal resource usage in edge security monitoring scenarios (thinking-mode output pricing shown)
thinking: true
price:
input: 0.11
output: 1.26
- cache_read: 0.022
-
-# Open source QwQ reasoning models
-- name: qwq-32b
- description: Wide. Open-source 32B reasoning model with powerful logic capabilities for deep security research
- thinking: true
- price:
- input: 0.287
- output: 0.861
- cache_read: 0.057
-
-# Open source Qwen2.5 series
-- name: qwen2.5-14b-instruct-1m
- description: Wide (Intr/CH). Extended context 14B model supporting up to 1M tokens
- thinking: false
- price:
- input: 0.805
- output: 3.22
- cache_read: 0.161
-
-- name: qwen2.5-7b-instruct-1m
- description: Wide (Intr/CH). Extended context 7B model supporting up to 1M tokens
- thinking: false
- price:
- input: 0.368
- output: 1.47
- cache_read: 0.074
-
-- name: qwen2.5-72b-instruct
- description: Wide (Intr/CH). Large 72B instruction-following model
- thinking: false
- price:
- input: 1.4
- output: 5.6
- cache_read: 0.28
-
-- name: qwen2.5-32b-instruct
- description: Wide (Intr/CH). Medium 32B instruction-following model
- thinking: false
- price:
- input: 0.7
- output: 2.8
- cache_read: 0.14
-
-- name: qwen2.5-14b-instruct
- description: Wide (Intr/CH). Compact 14B instruction-following model
- thinking: false
- price:
- input: 0.35
- output: 1.4
- cache_read: 0.07
-
-- name: qwen2.5-7b-instruct
- description: Wide (Intr/CH). Small 7B instruction-following model
- thinking: false
- price:
- input: 0.175
- output: 0.7
- cache_read: 0.035
-
-- name: qwen2.5-3b-instruct
- description: CH only. Lightweight 3B instruction model for Chinese Mainland
- thinking: false
- price:
- input: 0.044
- output: 0.130
- cache_read: 0.009
diff --git a/examples/configs/deepseek.provider.yml b/examples/configs/deepseek.provider.yml
index 485d6f39..b11c9c89 100644
--- a/examples/configs/deepseek.provider.yml
+++ b/examples/configs/deepseek.provider.yml
@@ -1,9 +1,9 @@
simple:
model: deepseek-v4-flash
- temperature: 0.5
- top_p: 0.5
+ temperature: 0.3
+ top_p: 0.9
n: 1
- max_tokens: 4000
+ max_tokens: 8192
extra_body:
thinking:
type: disabled
@@ -14,10 +14,10 @@ simple:
simple_json:
model: deepseek-v4-flash
- temperature: 0.5
- top_p: 0.5
+ temperature: 0.3
+ top_p: 0.9
n: 1
- max_tokens: 4000
+ max_tokens: 4096
json: true
extra_body:
thinking:
@@ -30,45 +30,79 @@ simple_json:
primary_agent:
model: deepseek-v4-pro
n: 1
- max_tokens: 8000
+ max_tokens: 16384
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
+ input: 1.74
+ output: 3.48
+ cache_read: 0.0145
assistant:
model: deepseek-v4-pro
n: 1
- max_tokens: 8000
+ max_tokens: 16384
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
+ input: 1.74
+ output: 3.48
+ cache_read: 0.0145
generator:
model: deepseek-v4-pro
n: 1
- max_tokens: 8000
+ max_tokens: 32768
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
+ input: 1.74
+ output: 3.48
+ cache_read: 0.0145
refiner:
model: deepseek-v4-pro
n: 1
- max_tokens: 8000
+ max_tokens: 32768
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
+ input: 1.74
+ output: 3.48
+ cache_read: 0.0145
adviser:
+ model: deepseek-v4-pro
+ n: 1
+ max_tokens: 16384
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
+ price:
+ input: 1.74
+ output: 3.48
+ cache_read: 0.0145
+
+reflector:
model: deepseek-v4-flash
- temperature: 0.7
- top_p: 0.8
+ temperature: 0.5
+ top_p: 0.9
n: 1
- max_tokens: 6000
+ max_tokens: 8192
extra_body:
thinking:
type: disabled
@@ -77,21 +111,12 @@ adviser:
output: 0.28
cache_read: 0.0028
-reflector:
- model: deepseek-v4-pro
- n: 1
- max_tokens: 4000
- price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
-
searcher:
model: deepseek-v4-flash
- temperature: 0.7
- top_p: 0.8
+ temperature: 0.5
+ top_p: 0.9
n: 1
- max_tokens: 4000
+ max_tokens: 4096
extra_body:
thinking:
type: disabled
@@ -102,10 +127,10 @@ searcher:
enricher:
model: deepseek-v4-flash
- temperature: 0.7
- top_p: 0.8
+ temperature: 0.5
+ top_p: 0.9
n: 1
- max_tokens: 4000
+ max_tokens: 4096
extra_body:
thinking:
type: disabled
@@ -117,26 +142,41 @@ enricher:
coder:
model: deepseek-v4-pro
n: 1
- max_tokens: 16384
+ max_tokens: 20480
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
+ input: 1.74
+ output: 3.48
+ cache_read: 0.0145
installer:
- model: deepseek-v4-pro
+ model: deepseek-v4-flash
n: 1
- max_tokens: 8192
+ max_tokens: 12288
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
+ input: 0.14
+ output: 0.28
+ cache_read: 0.0028
pentester:
model: deepseek-v4-pro
n: 1
- max_tokens: 8192
+ max_tokens: 16384
+ reasoning:
+ effort: high
+ extra_body:
+ thinking:
+ type: enabled
price:
- input: 0.435
- output: 0.87
- cache_read: 0.003625
+ input: 1.74
+ output: 3.48
+ cache_read: 0.0145
diff --git a/examples/configs/moonshot.provider.yml b/examples/configs/moonshot.provider.yml
index 34b96f73..6011341b 100644
--- a/examples/configs/moonshot.provider.yml
+++ b/examples/configs/moonshot.provider.yml
@@ -1,129 +1,205 @@
simple:
- model: "kimi-k2-turbo-preview"
+ model: kimi-k2.5
temperature: 0.6
+ top_p: 0.95
n: 1
max_tokens: 8192
+ extra_body:
+ temperature: 0.6
+ thinking:
+ type: disabled
+ tool_choice: auto
price:
- input: 1.15
- output: 8.0
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
simple_json:
- model: "kimi-k2-turbo-preview"
+ model: kimi-k2.5
temperature: 0.6
+ top_p: 0.95
n: 1
max_tokens: 4096
json: true
+ extra_body:
+ temperature: 0.6
+ thinking:
+ type: disabled
+ tool_choice: auto
price:
- input: 1.15
- output: 8.0
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
primary_agent:
- model: "kimi-k2.5"
+ model: kimi-k2.5
temperature: 1.0
+ top_p: 0.95
n: 1
max_tokens: 16384
- reasoning:
- effort: high
+ extra_body:
+ thinking:
+ type: enabled
+ tool_choice: auto
price:
- input: 0.6
- output: 3.0
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
assistant:
- model: "kimi-k2.5"
+ model: kimi-k2.5
temperature: 1.0
+ top_p: 0.95
n: 1
max_tokens: 16384
- reasoning:
- effort: high
+ extra_body:
+ thinking:
+ type: enabled
+ tool_choice: auto
price:
- input: 0.6
- output: 3.0
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
generator:
- model: "kimi-k2.5"
+ model: kimi-k2.6
temperature: 1.0
+ top_p: 0.95
n: 1
max_tokens: 32768
- reasoning:
- effort: high
+ extra_body:
+ thinking:
+ type: enabled
+ keep: all
+ tool_choice: auto
price:
- input: 0.6
- output: 3.0
+ input: 0.95
+ output: 4.00
+ cache_read: 0.16
refiner:
- model: "kimi-k2.5"
+ model: kimi-k2.6
temperature: 1.0
+ top_p: 0.95
n: 1
- max_tokens: 20480
- reasoning:
- effort: medium
+ max_tokens: 32768
+ extra_body:
+ thinking:
+ type: enabled
+ keep: all
+ tool_choice: auto
price:
- input: 0.6
- output: 3.0
+ input: 0.95
+ output: 4.00
+ cache_read: 0.16
adviser:
- model: "kimi-k2.5"
+ model: kimi-k2.6
temperature: 1.0
+ top_p: 0.95
n: 1
max_tokens: 8192
- reasoning:
- effort: high
+ extra_body:
+ thinking:
+ type: enabled
+ keep: all
+ tool_choice: auto
price:
- input: 0.6
- output: 3.0
+ input: 0.95
+ output: 4.00
+ cache_read: 0.16
reflector:
- model: "kimi-k2-0905-preview"
- temperature: 0.7
+ model: kimi-k2.5
+ temperature: 0.6
+ top_p: 0.95
n: 1
max_tokens: 4096
+ extra_body:
+ temperature: 0.6
+ thinking:
+ type: disabled
+ tool_choice: auto
price:
- input: 0.6
- output: 2.5
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
searcher:
- model: "kimi-k2-0905-preview"
- temperature: 0.7
+ model: kimi-k2.5
+ temperature: 0.6
+ top_p: 0.95
n: 1
max_tokens: 4096
+ extra_body:
+ temperature: 0.6
+ thinking:
+ type: disabled
+ tool_choice: auto
price:
- input: 0.6
- output: 2.5
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
enricher:
- model: "kimi-k2-0905-preview"
- temperature: 0.7
+ model: kimi-k2.5
+ temperature: 0.6
+ top_p: 0.95
n: 1
max_tokens: 4096
+ extra_body:
+ temperature: 0.6
+ thinking:
+ type: disabled
+ tool_choice: auto
price:
- input: 0.6
- output: 2.5
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
coder:
- model: "kimi-k2.5"
+ model: kimi-k2.6
temperature: 1.0
+ top_p: 0.95
n: 1
max_tokens: 20480
- reasoning:
- effort: high
+ extra_body:
+ thinking:
+ type: enabled
+ keep: all
+ tool_choice: auto
price:
- input: 0.6
- output: 3.0
+ input: 0.95
+ output: 4.00
+ cache_read: 0.16
installer:
- model: "kimi-k2-turbo-preview"
- temperature: 0.7
+ model: kimi-k2.5
+ temperature: 1.0
+ top_p: 0.95
n: 1
max_tokens: 16384
+ extra_body:
+ thinking:
+ type: enabled
+ tool_choice: auto
price:
- input: 1.15
- output: 8.0
+ input: 0.60
+ output: 3.00
+ cache_read: 0.10
pentester:
- model: "kimi-k2-turbo-preview"
- temperature: 0.8
+ model: kimi-k2.6
+ temperature: 1.0
+ top_p: 0.95
n: 1
max_tokens: 16384
+ extra_body:
+ thinking:
+ type: enabled
+ keep: all
+ tool_choice: auto
price:
- input: 1.15
- output: 8.0
+ input: 0.95
+ output: 4.00
+ cache_read: 0.16
diff --git a/examples/tests/deepseek-report.md b/examples/tests/deepseek-report.md
index 6d5b4a88..662aa93c 100644
--- a/examples/tests/deepseek-report.md
+++ b/examples/tests/deepseek-report.md
@@ -1,537 +1,537 @@
# LLM Agent Testing Report
-Generated: Thu, 05 Mar 2026 12:37:31 UTC
+Generated: Wed, 27 May 2026 22:59:39 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
-| simple | deepseek-chat | true | 23/23 (100.00%) | 3.290s |
-| simple_json | deepseek-chat | false | 5/5 (100.00%) | 3.141s |
-| primary_agent | deepseek-reasoner | true | 23/23 (100.00%) | 8.280s |
-| assistant | deepseek-reasoner | true | 23/23 (100.00%) | 8.055s |
-| generator | deepseek-reasoner | true | 23/23 (100.00%) | 7.539s |
-| refiner | deepseek-reasoner | true | 23/23 (100.00%) | 7.474s |
-| adviser | deepseek-chat | true | 23/23 (100.00%) | 3.167s |
-| reflector | deepseek-reasoner | true | 23/23 (100.00%) | 7.533s |
-| searcher | deepseek-chat | true | 23/23 (100.00%) | 3.306s |
-| enricher | deepseek-chat | true | 23/23 (100.00%) | 3.386s |
-| coder | deepseek-reasoner | true | 23/23 (100.00%) | 8.082s |
-| installer | deepseek-reasoner | true | 23/23 (100.00%) | 7.726s |
-| pentester | deepseek-reasoner | true | 23/23 (100.00%) | 8.148s |
-
-**Total**: 281/281 (100.00%) successful tests
-**Overall average latency**: 6.275s
+| simple | deepseek-v4-flash | false | 23/23 (100.00%) | 1.039s |
+| simple_json | deepseek-v4-flash | false | 5/5 (100.00%) | 0.989s |
+| primary_agent | deepseek-v4-pro | true | 23/23 (100.00%) | 2.896s |
+| assistant | deepseek-v4-pro | true | 22/23 (95.65%) | 2.929s |
+| generator | deepseek-v4-pro | true | 23/23 (100.00%) | 3.046s |
+| refiner | deepseek-v4-pro | true | 23/23 (100.00%) | 2.932s |
+| adviser | deepseek-v4-pro | true | 23/23 (100.00%) | 2.667s |
+| reflector | deepseek-v4-flash | true | 23/23 (100.00%) | 1.288s |
+| searcher | deepseek-v4-flash | true | 23/23 (100.00%) | 0.991s |
+| enricher | deepseek-v4-flash | true | 23/23 (100.00%) | 0.981s |
+| coder | deepseek-v4-pro | true | 23/23 (100.00%) | 2.817s |
+| installer | deepseek-v4-flash | true | 23/23 (100.00%) | 1.246s |
+| pentester | deepseek-v4-pro | true | 23/23 (100.00%) | 3.086s |
+
+**Total**: 280/281 (99.64%) successful tests
+**Overall average latency**: 2.139s
## Detailed Results
-### simple (deepseek-chat)
+### simple (deepseek-v4-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 2.330s | |
-| Text Transform Uppercase | ✅ Pass | 2.323s | |
-| Count from 1 to 5 | ✅ Pass | 1.662s | |
-| Math Calculation | ✅ Pass | 2.946s | |
-| Basic Echo Function | ✅ Pass | 3.734s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.328s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.450s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 3.727s | |
+| Simple Math | ✅ Pass | 0.973s | |
+| Text Transform Uppercase | ✅ Pass | 0.709s | |
+| Count from 1 to 5 | ✅ Pass | 0.903s | |
+| Math Calculation | ✅ Pass | 0.698s | |
+| Basic Echo Function | ✅ Pass | 1.075s | |
+| Streaming Simple Math Streaming | ✅ Pass | 0.775s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.871s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.081s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 4.530s | |
-| Search Query Function | ✅ Pass | 3.903s | |
-| Ask Advice Function | ✅ Pass | 4.649s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 4.041s | |
-| Basic Context Memory Test | ✅ Pass | 1.679s | |
-| Function Argument Memory Test | ✅ Pass | 1.373s | |
-| Function Response Memory Test | ✅ Pass | 1.616s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 6.403s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.826s | |
-| Penetration Testing Methodology | ✅ Pass | 4.009s | |
-| Vulnerability Assessment Tools | ✅ Pass | 9.493s | |
-| SQL Injection Attack Type | ✅ Pass | 2.073s | |
-| Penetration Testing Framework | ✅ Pass | 4.079s | |
-| Web Application Security Scanner | ✅ Pass | 2.274s | |
-| Penetration Testing Tool Selection | ✅ Pass | 4.213s | |
+| JSON Response Function | ✅ Pass | 1.254s | |
+| Search Query Function | ✅ Pass | 0.963s | |
+| Ask Advice Function | ✅ Pass | 1.142s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 0.976s | |
+| Basic Context Memory Test | ✅ Pass | 0.768s | |
+| Function Argument Memory Test | ✅ Pass | 0.865s | |
+| Function Response Memory Test | ✅ Pass | 0.782s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 1.414s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 0.847s | |
+| Penetration Testing Methodology | ✅ Pass | 1.116s | |
+| Vulnerability Assessment Tools | ✅ Pass | 2.258s | |
+| SQL Injection Attack Type | ✅ Pass | 0.876s | |
+| Penetration Testing Framework | ✅ Pass | 1.179s | |
+| Web Application Security Scanner | ✅ Pass | 1.190s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.180s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 3.290s
+**Average latency**: 1.039s
---
-### simple_json (deepseek-chat)
+### simple_json (deepseek-v4-flash)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Project Information JSON | ✅ Pass | 2.607s | |
-| Person Information JSON | ✅ Pass | 2.909s | |
-| Vulnerability Report Memory Test | ✅ Pass | 4.268s | |
-| User Profile JSON | ✅ Pass | 2.996s | |
-| Streaming Person Information JSON Streaming | ✅ Pass | 2.924s | |
+| Person Information JSON | ✅ Pass | 0.872s | |
+| Project Information JSON | ✅ Pass | 0.944s | |
+| Vulnerability Report Memory Test | ✅ Pass | 1.059s | |
+| User Profile JSON | ✅ Pass | 0.976s | |
+| Streaming Person Information JSON Streaming | ✅ Pass | 1.093s | |
**Summary**: 5/5 (100.00%) successful tests
-**Average latency**: 3.141s
+**Average latency**: 0.989s
---
-### primary_agent (deepseek-reasoner)
+### primary_agent (deepseek-v4-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 5.213s | |
-| Text Transform Uppercase | ✅ Pass | 9.683s | |
-| Count from 1 to 5 | ✅ Pass | 8.563s | |
-| Math Calculation | ✅ Pass | 6.101s | |
-| Basic Echo Function | ✅ Pass | 6.564s | |
-| Streaming Simple Math Streaming | ✅ Pass | 5.017s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 9.689s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 4.930s | |
+| Simple Math | ✅ Pass | 2.178s | |
+| Text Transform Uppercase | ✅ Pass | 1.700s | |
+| Count from 1 to 5 | ✅ Pass | 4.825s | |
+| Math Calculation | ✅ Pass | 1.798s | |
+| Basic Echo Function | ✅ Pass | 2.080s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.931s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.996s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.133s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 5.385s | |
-| Search Query Function | ✅ Pass | 4.589s | |
-| Ask Advice Function | ✅ Pass | 6.765s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 4.260s | |
-| Basic Context Memory Test | ✅ Pass | 7.070s | |
-| Function Argument Memory Test | ✅ Pass | 9.349s | |
-| Function Response Memory Test | ✅ Pass | 9.335s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 9.068s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 6.482s | |
-| Penetration Testing Methodology | ✅ Pass | 12.515s | |
-| Vulnerability Assessment Tools | ✅ Pass | 16.438s | |
-| SQL Injection Attack Type | ✅ Pass | 8.380s | |
-| Penetration Testing Framework | ✅ Pass | 13.453s | |
-| Web Application Security Scanner | ✅ Pass | 14.130s | |
-| Penetration Testing Tool Selection | ✅ Pass | 7.451s | |
+| JSON Response Function | ✅ Pass | 2.657s | |
+| Search Query Function | ✅ Pass | 2.248s | |
+| Ask Advice Function | ✅ Pass | 2.200s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 2.060s | |
+| Basic Context Memory Test | ✅ Pass | 3.547s | |
+| Function Argument Memory Test | ✅ Pass | 2.111s | |
+| Function Response Memory Test | ✅ Pass | 2.269s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 3.453s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 2.267s | |
+| Penetration Testing Methodology | ✅ Pass | 4.575s | |
+| Vulnerability Assessment Tools | ✅ Pass | 7.988s | |
+| SQL Injection Attack Type | ✅ Pass | 2.190s | |
+| Penetration Testing Framework | ✅ Pass | 5.697s | |
+| Web Application Security Scanner | ✅ Pass | 2.219s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.468s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 8.280s
+**Average latency**: 2.896s
---
-### assistant (deepseek-reasoner)
+### assistant (deepseek-v4-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 5.701s | |
-| Text Transform Uppercase | ✅ Pass | 3.974s | |
-| Count from 1 to 5 | ✅ Pass | 14.601s | |
-| Math Calculation | ✅ Pass | 4.861s | |
-| Basic Echo Function | ✅ Pass | 5.742s | |
-| Streaming Simple Math Streaming | ✅ Pass | 6.503s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 8.940s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 5.407s | |
+| Simple Math | ✅ Pass | 2.123s | |
+| Text Transform Uppercase | ✅ Pass | 1.972s | |
+| Count from 1 to 5 | ✅ Pass | 6.415s | |
+| Math Calculation | ✅ Pass | 1.814s | |
+| Basic Echo Function | ✅ Pass | 2.143s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.970s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.243s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.283s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 5.176s | |
-| Search Query Function | ✅ Pass | 4.823s | |
-| Ask Advice Function | ✅ Pass | 5.939s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 4.775s | |
-| Basic Context Memory Test | ✅ Pass | 8.351s | |
-| Function Argument Memory Test | ✅ Pass | 5.517s | |
-| Function Response Memory Test | ✅ Pass | 9.465s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 16.343s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 9.128s | |
-| Penetration Testing Methodology | ✅ Pass | 7.649s | |
-| Vulnerability Assessment Tools | ✅ Pass | 22.114s | |
-| SQL Injection Attack Type | ✅ Pass | 6.896s | |
-| Penetration Testing Framework | ✅ Pass | 11.455s | |
-| Web Application Security Scanner | ✅ Pass | 5.729s | |
-| Penetration Testing Tool Selection | ✅ Pass | 6.166s | |
-
-**Summary**: 23/23 (100.00%) successful tests
-
-**Average latency**: 8.055s
+| JSON Response Function | ✅ Pass | 2.509s | |
+| Search Query Function | ✅ Pass | 2.421s | |
+| Ask Advice Function | ✅ Pass | 2.316s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.957s | |
+| Basic Context Memory Test | ✅ Pass | 2.203s | |
+| Function Argument Memory Test | ✅ Pass | 2.293s | |
+| Function Response Memory Test | ✅ Pass | 2.042s | |
+| Penetration Testing Memory with Tool Call | ❌ Fail | 3.318s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 1.571s | |
+| Penetration Testing Methodology | ✅ Pass | 3.431s | |
+| SQL Injection Attack Type | ✅ Pass | 2.289s | |
+| Vulnerability Assessment Tools | ✅ Pass | 10.808s | |
+| Penetration Testing Framework | ✅ Pass | 3.233s | |
+| Web Application Security Scanner | ✅ Pass | 3.258s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.752s | |
+
+**Summary**: 22/23 (95.65%) successful tests
+
+**Average latency**: 2.929s
---
-### generator (deepseek-reasoner)
+### generator (deepseek-v4-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 6.719s | |
-| Text Transform Uppercase | ✅ Pass | 7.568s | |
-| Count from 1 to 5 | ✅ Pass | 11.023s | |
-| Math Calculation | ✅ Pass | 5.914s | |
-| Basic Echo Function | ✅ Pass | 4.075s | |
-| Streaming Simple Math Streaming | ✅ Pass | 6.838s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 9.104s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 5.719s | |
+| Simple Math | ✅ Pass | 2.071s | |
+| Text Transform Uppercase | ✅ Pass | 2.231s | |
+| Count from 1 to 5 | ✅ Pass | 6.367s | |
+| Math Calculation | ✅ Pass | 1.514s | |
+| Basic Echo Function | ✅ Pass | 1.995s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.534s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.143s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.014s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 4.957s | |
-| Search Query Function | ✅ Pass | 4.612s | |
-| Ask Advice Function | ✅ Pass | 5.846s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 4.085s | |
-| Basic Context Memory Test | ✅ Pass | 8.809s | |
-| Function Argument Memory Test | ✅ Pass | 6.380s | |
-| Function Response Memory Test | ✅ Pass | 9.236s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 8.062s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 5.693s | |
-| Penetration Testing Methodology | ✅ Pass | 14.874s | |
-| Vulnerability Assessment Tools | ✅ Pass | 8.056s | |
-| SQL Injection Attack Type | ✅ Pass | 8.753s | |
-| Penetration Testing Framework | ✅ Pass | 13.774s | |
-| Web Application Security Scanner | ✅ Pass | 7.330s | |
-| Penetration Testing Tool Selection | ✅ Pass | 5.956s | |
+| JSON Response Function | ✅ Pass | 2.431s | |
+| Search Query Function | ✅ Pass | 2.231s | |
+| Ask Advice Function | ✅ Pass | 2.042s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 2.115s | |
+| Basic Context Memory Test | ✅ Pass | 2.382s | |
+| Function Argument Memory Test | ✅ Pass | 2.525s | |
+| Function Response Memory Test | ✅ Pass | 2.067s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 3.271s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 2.383s | |
+| Penetration Testing Methodology | ✅ Pass | 4.247s | |
+| SQL Injection Attack Type | ✅ Pass | 2.293s | |
+| Vulnerability Assessment Tools | ✅ Pass | 12.485s | |
+| Penetration Testing Framework | ✅ Pass | 4.695s | |
+| Web Application Security Scanner | ✅ Pass | 2.208s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.794s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 7.539s
+**Average latency**: 3.046s
---
-### refiner (deepseek-reasoner)
+### refiner (deepseek-v4-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 6.303s | |
-| Text Transform Uppercase | ✅ Pass | 2.714s | |
-| Count from 1 to 5 | ✅ Pass | 10.440s | |
-| Math Calculation | ✅ Pass | 5.162s | |
-| Basic Echo Function | ✅ Pass | 6.387s | |
-| Streaming Simple Math Streaming | ✅ Pass | 5.790s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 9.962s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 6.327s | |
+| Simple Math | ✅ Pass | 2.115s | |
+| Text Transform Uppercase | ✅ Pass | 1.996s | |
+| Count from 1 to 5 | ✅ Pass | 4.321s | |
+| Math Calculation | ✅ Pass | 1.722s | |
+| Basic Echo Function | ✅ Pass | 2.028s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.742s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.044s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.170s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 5.209s | |
-| Search Query Function | ✅ Pass | 4.401s | |
-| Ask Advice Function | ✅ Pass | 6.108s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 4.824s | |
-| Basic Context Memory Test | ✅ Pass | 7.066s | |
-| Function Argument Memory Test | ✅ Pass | 7.766s | |
-| Function Response Memory Test | ✅ Pass | 7.438s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 9.727s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 7.073s | |
-| Penetration Testing Methodology | ✅ Pass | 7.321s | |
-| Vulnerability Assessment Tools | ✅ Pass | 16.650s | |
-| SQL Injection Attack Type | ✅ Pass | 8.742s | |
-| Penetration Testing Framework | ✅ Pass | 10.606s | |
-| Web Application Security Scanner | ✅ Pass | 9.430s | |
-| Penetration Testing Tool Selection | ✅ Pass | 6.448s | |
+| JSON Response Function | ✅ Pass | 2.374s | |
+| Search Query Function | ✅ Pass | 2.454s | |
+| Ask Advice Function | ✅ Pass | 2.166s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.974s | |
+| Basic Context Memory Test | ✅ Pass | 3.341s | |
+| Function Argument Memory Test | ✅ Pass | 2.086s | |
+| Function Response Memory Test | ✅ Pass | 2.400s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 3.446s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 2.015s | |
+| Penetration Testing Methodology | ✅ Pass | 3.273s | |
+| SQL Injection Attack Type | ✅ Pass | 3.515s | |
+| Vulnerability Assessment Tools | ✅ Pass | 11.480s | |
+| Penetration Testing Framework | ✅ Pass | 4.129s | |
+| Web Application Security Scanner | ✅ Pass | 2.298s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.323s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 7.474s
+**Average latency**: 2.932s
---
-### adviser (deepseek-chat)
+### adviser (deepseek-v4-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 2.417s | |
-| Text Transform Uppercase | ✅ Pass | 1.474s | |
-| Count from 1 to 5 | ✅ Pass | 1.638s | |
-| Math Calculation | ✅ Pass | 2.544s | |
-| Basic Echo Function | ✅ Pass | 4.205s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.209s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.555s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 3.583s | |
+| Simple Math | ✅ Pass | 2.074s | |
+| Text Transform Uppercase | ✅ Pass | 1.958s | |
+| Count from 1 to 5 | ✅ Pass | 4.874s | |
+| Math Calculation | ✅ Pass | 1.681s | |
+| Basic Echo Function | ✅ Pass | 2.290s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.667s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.040s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.949s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 4.354s | |
-| Search Query Function | ✅ Pass | 3.786s | |
-| Ask Advice Function | ✅ Pass | 4.394s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 4.019s | |
-| Basic Context Memory Test | ✅ Pass | 2.260s | |
-| Function Argument Memory Test | ✅ Pass | 1.329s | |
-| Function Response Memory Test | ✅ Pass | 1.721s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 6.484s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.417s | |
-| Penetration Testing Methodology | ✅ Pass | 4.292s | |
-| Vulnerability Assessment Tools | ✅ Pass | 7.449s | |
-| SQL Injection Attack Type | ✅ Pass | 2.212s | |
-| Penetration Testing Framework | ✅ Pass | 4.318s | |
-| Web Application Security Scanner | ✅ Pass | 1.906s | |
-| Penetration Testing Tool Selection | ✅ Pass | 4.259s | |
+| JSON Response Function | ✅ Pass | 2.484s | |
+| Search Query Function | ✅ Pass | 2.248s | |
+| Ask Advice Function | ✅ Pass | 1.988s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.986s | |
+| Basic Context Memory Test | ✅ Pass | 2.873s | |
+| Function Argument Memory Test | ✅ Pass | 2.211s | |
+| Function Response Memory Test | ✅ Pass | 1.874s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 3.502s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 2.064s | |
+| Penetration Testing Methodology | ✅ Pass | 3.537s | |
+| Vulnerability Assessment Tools | ✅ Pass | 5.184s | |
+| SQL Injection Attack Type | ✅ Pass | 4.424s | |
+| Penetration Testing Framework | ✅ Pass | 3.939s | |
+| Web Application Security Scanner | ✅ Pass | 2.206s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.267s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 3.167s
+**Average latency**: 2.667s
---
-### reflector (deepseek-reasoner)
+### reflector (deepseek-v4-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 5.139s | |
-| Text Transform Uppercase | ✅ Pass | 4.301s | |
-| Count from 1 to 5 | ✅ Pass | 15.920s | |
-| Math Calculation | ✅ Pass | 7.810s | |
-| Basic Echo Function | ✅ Pass | 5.190s | |
-| Streaming Simple Math Streaming | ✅ Pass | 5.922s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 10.554s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 4.339s | |
+| Simple Math | ✅ Pass | 0.798s | |
+| Text Transform Uppercase | ✅ Pass | 0.888s | |
+| Count from 1 to 5 | ✅ Pass | 7.083s | |
+| Math Calculation | ✅ Pass | 0.887s | |
+| Basic Echo Function | ✅ Pass | 1.023s | |
+| Streaming Simple Math Streaming | ✅ Pass | 0.759s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.762s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.272s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 6.430s | |
-| Search Query Function | ✅ Pass | 5.103s | |
-| Ask Advice Function | ✅ Pass | 7.119s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 4.705s | |
-| Basic Context Memory Test | ✅ Pass | 7.071s | |
-| Function Argument Memory Test | ✅ Pass | 7.197s | |
-| Function Response Memory Test | ✅ Pass | 7.445s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 8.926s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 7.443s | |
-| Penetration Testing Methodology | ✅ Pass | 6.513s | |
-| Vulnerability Assessment Tools | ✅ Pass | 18.854s | |
-| SQL Injection Attack Type | ✅ Pass | 5.673s | |
-| Penetration Testing Framework | ✅ Pass | 7.492s | |
-| Web Application Security Scanner | ✅ Pass | 7.812s | |
-| Penetration Testing Tool Selection | ✅ Pass | 6.289s | |
+| JSON Response Function | ✅ Pass | 1.026s | |
+| Search Query Function | ✅ Pass | 0.935s | |
+| Ask Advice Function | ✅ Pass | 1.016s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.064s | |
+| Basic Context Memory Test | ✅ Pass | 0.733s | |
+| Function Argument Memory Test | ✅ Pass | 0.869s | |
+| Function Response Memory Test | ✅ Pass | 0.791s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 1.358s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 0.853s | |
+| Penetration Testing Methodology | ✅ Pass | 1.117s | |
+| Vulnerability Assessment Tools | ✅ Pass | 1.876s | |
+| SQL Injection Attack Type | ✅ Pass | 0.844s | |
+| Penetration Testing Framework | ✅ Pass | 1.286s | |
+| Web Application Security Scanner | ✅ Pass | 1.159s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.207s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 7.533s
+**Average latency**: 1.288s
---
-### searcher (deepseek-chat)
+### searcher (deepseek-v4-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 2.813s | |
-| Text Transform Uppercase | ✅ Pass | 1.684s | |
-| Count from 1 to 5 | ✅ Pass | 1.832s | |
-| Math Calculation | ✅ Pass | 2.485s | |
-| Basic Echo Function | ✅ Pass | 3.975s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.939s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.701s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 3.943s | |
+| Simple Math | ✅ Pass | 0.813s | |
+| Text Transform Uppercase | ✅ Pass | 0.770s | |
+| Count from 1 to 5 | ✅ Pass | 0.893s | |
+| Math Calculation | ✅ Pass | 0.742s | |
+| Basic Echo Function | ✅ Pass | 0.971s | |
+| Streaming Simple Math Streaming | ✅ Pass | 0.695s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.929s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.123s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 4.516s | |
-| Search Query Function | ✅ Pass | 4.164s | |
-| Ask Advice Function | ✅ Pass | 4.516s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 4.193s | |
-| Basic Context Memory Test | ✅ Pass | 1.903s | |
-| Function Argument Memory Test | ✅ Pass | 1.618s | |
-| Function Response Memory Test | ✅ Pass | 1.745s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 6.457s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.534s | |
-| Penetration Testing Methodology | ✅ Pass | 3.980s | |
-| Vulnerability Assessment Tools | ✅ Pass | 7.738s | |
-| SQL Injection Attack Type | ✅ Pass | 1.721s | |
-| Penetration Testing Framework | ✅ Pass | 3.966s | |
-| Web Application Security Scanner | ✅ Pass | 2.065s | |
-| Penetration Testing Tool Selection | ✅ Pass | 4.550s | |
+| JSON Response Function | ✅ Pass | 1.010s | |
+| Search Query Function | ✅ Pass | 0.922s | |
+| Ask Advice Function | ✅ Pass | 1.081s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 0.909s | |
+| Basic Context Memory Test | ✅ Pass | 0.755s | |
+| Function Argument Memory Test | ✅ Pass | 0.789s | |
+| Function Response Memory Test | ✅ Pass | 0.742s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 1.220s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 0.870s | |
+| Penetration Testing Methodology | ✅ Pass | 0.981s | |
+| Vulnerability Assessment Tools | ✅ Pass | 2.004s | |
+| SQL Injection Attack Type | ✅ Pass | 0.899s | |
+| Penetration Testing Framework | ✅ Pass | 1.223s | |
+| Web Application Security Scanner | ✅ Pass | 1.287s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.151s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 3.306s
+**Average latency**: 0.991s
---
-### enricher (deepseek-chat)
+### enricher (deepseek-v4-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 1.383s | |
-| Text Transform Uppercase | ✅ Pass | 2.289s | |
-| Count from 1 to 5 | ✅ Pass | 1.905s | |
-| Math Calculation | ✅ Pass | 1.825s | |
-| Basic Echo Function | ✅ Pass | 4.084s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.590s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.770s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 3.511s | |
+| Simple Math | ✅ Pass | 0.759s | |
+| Text Transform Uppercase | ✅ Pass | 0.897s | |
+| Count from 1 to 5 | ✅ Pass | 0.872s | |
+| Math Calculation | ✅ Pass | 0.710s | |
+| Basic Echo Function | ✅ Pass | 0.948s | |
+| Streaming Simple Math Streaming | ✅ Pass | 0.870s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.746s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 0.899s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 4.759s | |
-| Search Query Function | ✅ Pass | 4.164s | |
-| Ask Advice Function | ✅ Pass | 4.303s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 3.761s | |
-| Basic Context Memory Test | ✅ Pass | 1.935s | |
-| Function Argument Memory Test | ✅ Pass | 1.462s | |
-| Function Response Memory Test | ✅ Pass | 1.946s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 6.452s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.717s | |
-| Penetration Testing Methodology | ✅ Pass | 7.758s | |
-| Vulnerability Assessment Tools | ✅ Pass | 7.600s | |
-| SQL Injection Attack Type | ✅ Pass | 1.774s | |
-| Penetration Testing Framework | ✅ Pass | 3.895s | |
-| Web Application Security Scanner | ✅ Pass | 3.355s | |
-| Penetration Testing Tool Selection | ✅ Pass | 4.627s | |
+| JSON Response Function | ✅ Pass | 1.144s | |
+| Search Query Function | ✅ Pass | 0.972s | |
+| Ask Advice Function | ✅ Pass | 0.989s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.015s | |
+| Basic Context Memory Test | ✅ Pass | 0.858s | |
+| Function Argument Memory Test | ✅ Pass | 0.726s | |
+| Function Response Memory Test | ✅ Pass | 0.754s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 1.414s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 0.851s | |
+| Penetration Testing Methodology | ✅ Pass | 0.991s | |
+| Vulnerability Assessment Tools | ✅ Pass | 1.597s | |
+| SQL Injection Attack Type | ✅ Pass | 0.857s | |
+| Penetration Testing Framework | ✅ Pass | 1.269s | |
+| Web Application Security Scanner | ✅ Pass | 1.036s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.377s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 3.386s
+**Average latency**: 0.981s
---
-### coder (deepseek-reasoner)
+### coder (deepseek-v4-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 5.972s | |
-| Text Transform Uppercase | ✅ Pass | 4.363s | |
-| Count from 1 to 5 | ✅ Pass | 12.156s | |
-| Math Calculation | ✅ Pass | 7.242s | |
-| Basic Echo Function | ✅ Pass | 4.592s | |
-| Streaming Simple Math Streaming | ✅ Pass | 5.389s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 9.696s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 5.745s | |
+| Simple Math | ✅ Pass | 1.791s | |
+| Text Transform Uppercase | ✅ Pass | 6.028s | |
+| Count from 1 to 5 | ✅ Pass | 1.873s | |
+| Math Calculation | ✅ Pass | 1.707s | |
+| Basic Echo Function | ✅ Pass | 2.051s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.951s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.989s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.446s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 4.971s | |
-| Search Query Function | ✅ Pass | 4.594s | |
-| Ask Advice Function | ✅ Pass | 6.151s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 4.278s | |
-| Basic Context Memory Test | ✅ Pass | 5.950s | |
-| Function Argument Memory Test | ✅ Pass | 8.863s | |
-| Function Response Memory Test | ✅ Pass | 8.061s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 11.734s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 8.433s | |
-| Penetration Testing Methodology | ✅ Pass | 13.948s | |
-| Vulnerability Assessment Tools | ✅ Pass | 15.857s | |
-| SQL Injection Attack Type | ✅ Pass | 8.389s | |
-| Penetration Testing Framework | ✅ Pass | 10.210s | |
-| Web Application Security Scanner | ✅ Pass | 13.036s | |
-| Penetration Testing Tool Selection | ✅ Pass | 6.235s | |
+| JSON Response Function | ✅ Pass | 2.494s | |
+| Search Query Function | ✅ Pass | 2.010s | |
+| Ask Advice Function | ✅ Pass | 2.242s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 2.133s | |
+| Basic Context Memory Test | ✅ Pass | 3.207s | |
+| Function Argument Memory Test | ✅ Pass | 2.075s | |
+| Function Response Memory Test | ✅ Pass | 1.921s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 3.561s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 2.345s | |
+| Penetration Testing Methodology | ✅ Pass | 3.598s | |
+| Vulnerability Assessment Tools | ✅ Pass | 8.780s | |
+| SQL Injection Attack Type | ✅ Pass | 2.581s | |
+| Penetration Testing Framework | ✅ Pass | 2.960s | |
+| Web Application Security Scanner | ✅ Pass | 2.493s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.548s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 8.082s
+**Average latency**: 2.817s
---
-### installer (deepseek-reasoner)
+### installer (deepseek-v4-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 6.160s | |
-| Text Transform Uppercase | ✅ Pass | 3.599s | |
-| Count from 1 to 5 | ✅ Pass | 11.234s | |
-| Math Calculation | ✅ Pass | 5.105s | |
-| Basic Echo Function | ✅ Pass | 4.928s | |
-| Streaming Simple Math Streaming | ✅ Pass | 5.469s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 11.351s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 5.075s | |
+| Simple Math | ✅ Pass | 1.185s | |
+| Text Transform Uppercase | ✅ Pass | 1.103s | |
+| Count from 1 to 5 | ✅ Pass | 1.222s | |
+| Math Calculation | ✅ Pass | 0.962s | |
+| Basic Echo Function | ✅ Pass | 0.990s | |
+| Streaming Simple Math Streaming | ✅ Pass | 0.908s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.216s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.106s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 5.490s | |
-| Search Query Function | ✅ Pass | 5.225s | |
-| Ask Advice Function | ✅ Pass | 7.047s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 4.168s | |
-| Basic Context Memory Test | ✅ Pass | 8.173s | |
-| Function Argument Memory Test | ✅ Pass | 4.750s | |
-| Function Response Memory Test | ✅ Pass | 12.370s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 11.038s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 6.812s | |
-| Penetration Testing Methodology | ✅ Pass | 10.908s | |
-| Vulnerability Assessment Tools | ✅ Pass | 19.308s | |
-| SQL Injection Attack Type | ✅ Pass | 6.246s | |
-| Penetration Testing Framework | ✅ Pass | 9.980s | |
-| Web Application Security Scanner | ✅ Pass | 6.419s | |
-| Penetration Testing Tool Selection | ✅ Pass | 6.823s | |
+| JSON Response Function | ✅ Pass | 1.264s | |
+| Search Query Function | ✅ Pass | 1.140s | |
+| Ask Advice Function | ✅ Pass | 1.158s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.171s | |
+| Basic Context Memory Test | ✅ Pass | 1.188s | |
+| Function Argument Memory Test | ✅ Pass | 0.905s | |
+| Function Response Memory Test | ✅ Pass | 1.120s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 1.758s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 1.140s | |
+| Penetration Testing Methodology | ✅ Pass | 1.382s | |
+| Vulnerability Assessment Tools | ✅ Pass | 2.060s | |
+| SQL Injection Attack Type | ✅ Pass | 1.298s | |
+| Penetration Testing Framework | ✅ Pass | 1.423s | |
+| Web Application Security Scanner | ✅ Pass | 1.586s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.352s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 7.726s
+**Average latency**: 1.246s
---
-### pentester (deepseek-reasoner)
+### pentester (deepseek-v4-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 6.298s | |
-| Text Transform Uppercase | ✅ Pass | 9.756s | |
-| Count from 1 to 5 | ✅ Pass | 11.009s | |
-| Math Calculation | ✅ Pass | 5.242s | |
-| Basic Echo Function | ✅ Pass | 4.391s | |
-| Streaming Simple Math Streaming | ✅ Pass | 5.650s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 13.565s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 5.658s | |
+| Simple Math | ✅ Pass | 1.845s | |
+| Text Transform Uppercase | ✅ Pass | 6.297s | |
+| Count from 1 to 5 | ✅ Pass | 1.982s | |
+| Math Calculation | ✅ Pass | 1.653s | |
+| Basic Echo Function | ✅ Pass | 1.838s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.884s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.004s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.209s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 5.360s | |
-| Search Query Function | ✅ Pass | 5.013s | |
-| Ask Advice Function | ✅ Pass | 7.211s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 4.264s | |
-| Basic Context Memory Test | ✅ Pass | 8.040s | |
-| Function Argument Memory Test | ✅ Pass | 8.709s | |
-| Function Response Memory Test | ✅ Pass | 6.345s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 13.720s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 7.283s | |
-| Penetration Testing Methodology | ✅ Pass | 13.123s | |
-| Vulnerability Assessment Tools | ✅ Pass | 16.003s | |
-| SQL Injection Attack Type | ✅ Pass | 7.168s | |
-| Penetration Testing Framework | ✅ Pass | 9.592s | |
-| Web Application Security Scanner | ✅ Pass | 6.683s | |
-| Penetration Testing Tool Selection | ✅ Pass | 7.318s | |
+| JSON Response Function | ✅ Pass | 2.702s | |
+| Search Query Function | ✅ Pass | 1.898s | |
+| Ask Advice Function | ✅ Pass | 2.049s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 2.145s | |
+| Basic Context Memory Test | ✅ Pass | 3.227s | |
+| Function Argument Memory Test | ✅ Pass | 1.892s | |
+| Function Response Memory Test | ✅ Pass | 2.334s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 3.631s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 2.276s | |
+| Penetration Testing Methodology | ✅ Pass | 3.126s | |
+| SQL Injection Attack Type | ✅ Pass | 2.551s | |
+| Vulnerability Assessment Tools | ✅ Pass | 14.848s | |
+| Penetration Testing Framework | ✅ Pass | 3.944s | |
+| Web Application Security Scanner | ✅ Pass | 1.612s | |
+| Penetration Testing Tool Selection | ✅ Pass | 3.012s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 8.148s
+**Average latency**: 3.086s
---
diff --git a/examples/tests/gemini-report.md b/examples/tests/gemini-report.md
index 10799e97..ff39febd 100644
--- a/examples/tests/gemini-report.md
+++ b/examples/tests/gemini-report.md
@@ -1,86 +1,86 @@
# LLM Agent Testing Report
-Generated: Thu, 05 Mar 2026 17:08:56 UTC
+Generated: Wed, 27 May 2026 22:33:44 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
-| simple | gemini-3.1-flash-lite-preview | true | 23/23 (100.00%) | 1.105s |
-| simple_json | gemini-3.1-flash-lite-preview | true | 5/5 (100.00%) | 1.603s |
-| primary_agent | gemini-3.1-pro-preview | true | 23/23 (100.00%) | 5.646s |
-| assistant | gemini-3.1-pro-preview | true | 21/23 (91.30%) | 6.289s |
-| generator | gemini-3.1-pro-preview | true | 23/23 (100.00%) | 7.440s |
-| refiner | gemini-3.1-pro-preview | true | 22/23 (95.65%) | 12.764s |
-| adviser | gemini-3.1-pro-preview | true | 21/23 (91.30%) | 6.169s |
-| reflector | gemini-3-flash-preview | true | 23/23 (100.00%) | 2.045s |
-| searcher | gemini-3-flash-preview | true | 23/23 (100.00%) | 1.992s |
-| enricher | gemini-3-flash-preview | true | 23/23 (100.00%) | 2.107s |
-| coder | gemini-3.1-pro-preview | true | 23/23 (100.00%) | 5.779s |
-| installer | gemini-3-flash-preview | true | 23/23 (100.00%) | 2.763s |
-| pentester | gemini-3.1-pro-preview | true | 21/23 (91.30%) | 5.733s |
+| simple | gemini-3.1-flash-lite | true | 23/23 (100.00%) | 0.768s |
+| simple_json | gemini-3.1-flash-lite | true | 5/5 (100.00%) | 0.715s |
+| primary_agent | gemini-3.1-pro-preview | true | 22/23 (95.65%) | 4.190s |
+| assistant | gemini-3.1-pro-preview | true | 22/23 (95.65%) | 4.239s |
+| generator | gemini-3.1-pro-preview | true | 21/23 (91.30%) | 4.434s |
+| refiner | gemini-3.1-pro-preview | true | 23/23 (100.00%) | 4.221s |
+| adviser | gemini-3.1-pro-preview | true | 21/23 (91.30%) | 4.304s |
+| reflector | gemini-3.5-flash | true | 23/23 (100.00%) | 1.692s |
+| searcher | gemini-3.5-flash | true | 23/23 (100.00%) | 1.817s |
+| enricher | gemini-3.5-flash | true | 23/23 (100.00%) | 1.724s |
+| coder | gemini-3.1-pro-preview | true | 23/23 (100.00%) | 4.103s |
+| installer | gemini-3.5-flash | true | 22/23 (95.65%) | 2.891s |
+| pentester | gemini-3.1-pro-preview | true | 23/23 (100.00%) | 4.197s |
**Total**: 274/281 (97.51%) successful tests
-**Overall average latency**: 4.926s
+**Overall average latency**: 3.171s
## Detailed Results
-### simple (gemini-3.1-flash-lite-preview)
+### simple (gemini-3.1-flash-lite)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 1.997s | |
-| Text Transform Uppercase | ✅ Pass | 0.678s | |
-| Count from 1 to 5 | ✅ Pass | 1.306s | |
-| Math Calculation | ✅ Pass | 0.788s | |
-| Basic Echo Function | ✅ Pass | 1.675s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.154s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.903s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 0.944s | |
+| Simple Math | ✅ Pass | 1.367s | |
+| Text Transform Uppercase | ✅ Pass | 0.517s | |
+| Count from 1 to 5 | ✅ Pass | 0.704s | |
+| Math Calculation | ✅ Pass | 0.530s | |
+| Basic Echo Function | ✅ Pass | 0.524s | |
+| Streaming Simple Math Streaming | ✅ Pass | 0.506s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.626s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 0.663s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 0.733s | |
-| Search Query Function | ✅ Pass | 1.855s | |
-| Ask Advice Function | ✅ Pass | 0.980s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 0.825s | |
-| Basic Context Memory Test | ✅ Pass | 0.683s | |
-| Function Argument Memory Test | ✅ Pass | 0.889s | |
-| Function Response Memory Test | ✅ Pass | 2.236s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 1.009s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 0.596s | |
-| Penetration Testing Methodology | ✅ Pass | 0.980s | |
-| Vulnerability Assessment Tools | ✅ Pass | 1.341s | |
-| SQL Injection Attack Type | ✅ Pass | 0.655s | |
-| Penetration Testing Framework | ✅ Pass | 1.067s | |
-| Web Application Security Scanner | ✅ Pass | 0.735s | |
-| Penetration Testing Tool Selection | ✅ Pass | 1.376s | |
+| JSON Response Function | ✅ Pass | 0.578s | |
+| Search Query Function | ✅ Pass | 0.544s | |
+| Ask Advice Function | ✅ Pass | 0.576s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 0.529s | |
+| Basic Context Memory Test | ✅ Pass | 0.551s | |
+| Function Argument Memory Test | ✅ Pass | 0.490s | |
+| Function Response Memory Test | ✅ Pass | 0.688s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.597s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 0.546s | |
+| Penetration Testing Methodology | ✅ Pass | 1.218s | |
+| Vulnerability Assessment Tools | ✅ Pass | 1.050s | |
+| SQL Injection Attack Type | ✅ Pass | 0.661s | |
+| Penetration Testing Framework | ✅ Pass | 0.762s | |
+| Web Application Security Scanner | ✅ Pass | 0.750s | |
+| Penetration Testing Tool Selection | ✅ Pass | 0.673s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 1.105s
+**Average latency**: 0.768s
---
-### simple_json (gemini-3.1-flash-lite-preview)
+### simple_json (gemini-3.1-flash-lite)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Vulnerability Report Memory Test | ✅ Pass | 1.034s | |
-| Person Information JSON | ✅ Pass | 0.715s | |
-| User Profile JSON | ✅ Pass | 0.761s | |
-| Streaming Person Information JSON Streaming | ✅ Pass | 0.657s | |
-| Project Information JSON | ✅ Pass | 4.845s | |
+| Person Information JSON | ✅ Pass | 0.664s | |
+| Vulnerability Report Memory Test | ✅ Pass | 0.961s | |
+| User Profile JSON | ✅ Pass | 0.594s | |
+| Project Information JSON | ✅ Pass | 0.766s | |
+| Streaming Person Information JSON Streaming | ✅ Pass | 0.590s | |
**Summary**: 5/5 (100.00%) successful tests
-**Average latency**: 1.603s
+**Average latency**: 0.715s
---
@@ -90,38 +90,38 @@ Generated: Thu, 05 Mar 2026 17:08:56 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 5.034s | |
-| Text Transform Uppercase | ✅ Pass | 6.435s | |
-| Count from 1 to 5 | ✅ Pass | 5.477s | |
-| Math Calculation | ✅ Pass | 3.502s | |
-| Basic Echo Function | ✅ Pass | 5.140s | |
-| Streaming Simple Math Streaming | ✅ Pass | 3.687s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.316s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 6.680s | |
+| Simple Math | ✅ Pass | 3.812s | |
+| Text Transform Uppercase | ✅ Pass | 3.313s | |
+| Count from 1 to 5 | ✅ Pass | 4.086s | |
+| Math Calculation | ✅ Pass | 2.632s | |
+| Basic Echo Function | ❌ Fail | 4.651s | no tool calls found, expected at least 1 |
+| Streaming Simple Math Streaming | ✅ Pass | 3.243s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.803s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.800s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 5.011s | |
-| Search Query Function | ✅ Pass | 4.770s | |
-| Ask Advice Function | ✅ Pass | 7.735s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 7.586s | |
-| Basic Context Memory Test | ✅ Pass | 4.296s | |
-| Function Argument Memory Test | ✅ Pass | 5.851s | |
-| Function Response Memory Test | ✅ Pass | 3.933s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 7.244s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 4.651s | |
-| Penetration Testing Methodology | ✅ Pass | 6.143s | |
-| Vulnerability Assessment Tools | ✅ Pass | 8.006s | |
-| SQL Injection Attack Type | ✅ Pass | 3.981s | |
-| Penetration Testing Framework | ✅ Pass | 5.996s | |
-| Web Application Security Scanner | ✅ Pass | 4.363s | |
-| Penetration Testing Tool Selection | ✅ Pass | 9.017s | |
+| JSON Response Function | ✅ Pass | 5.060s | |
+| Search Query Function | ✅ Pass | 4.701s | |
+| Ask Advice Function | ✅ Pass | 6.206s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 2.818s | |
+| Basic Context Memory Test | ✅ Pass | 3.449s | |
+| Function Argument Memory Test | ✅ Pass | 4.006s | |
+| Function Response Memory Test | ✅ Pass | 4.055s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 3.735s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 4.365s | |
+| Penetration Testing Methodology | ✅ Pass | 5.392s | |
+| Vulnerability Assessment Tools | ✅ Pass | 5.336s | |
+| SQL Injection Attack Type | ✅ Pass | 3.569s | |
+| Penetration Testing Framework | ✅ Pass | 4.948s | |
+| Web Application Security Scanner | ✅ Pass | 5.728s | |
+| Penetration Testing Tool Selection | ✅ Pass | 5.643s | |
-**Summary**: 23/23 (100.00%) successful tests
+**Summary**: 22/23 (95.65%) successful tests
-**Average latency**: 5.646s
+**Average latency**: 4.190s
---
@@ -131,38 +131,38 @@ Generated: Thu, 05 Mar 2026 17:08:56 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 4.311s | |
-| Text Transform Uppercase | ✅ Pass | 10.801s | |
-| Count from 1 to 5 | ✅ Pass | 10.225s | |
-| Math Calculation | ✅ Pass | 3.895s | |
-| Basic Echo Function | ✅ Pass | 8.776s | |
-| Streaming Simple Math Streaming | ✅ Pass | 3.328s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.836s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 11.157s | |
+| Simple Math | ✅ Pass | 2.818s | |
+| Text Transform Uppercase | ✅ Pass | 3.634s | |
+| Count from 1 to 5 | ✅ Pass | 4.193s | |
+| Math Calculation | ✅ Pass | 3.305s | |
+| Basic Echo Function | ❌ Fail | 5.872s | no tool calls found, expected at least 1 |
+| Streaming Simple Math Streaming | ✅ Pass | 3.279s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.548s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.377s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 4.437s | |
-| Search Query Function | ✅ Pass | 4.580s | |
-| Ask Advice Function | ✅ Pass | 4.888s | |
-| Streaming Search Query Function Streaming | ❌ Fail | 11.694s | no tool calls found, expected at least 1 |
-| Basic Context Memory Test | ✅ Pass | 4.081s | |
-| Function Argument Memory Test | ✅ Pass | 4.616s | |
-| Function Response Memory Test | ✅ Pass | 4.995s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 7.145s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 5.072s | |
-| Penetration Testing Methodology | ✅ Pass | 7.007s | |
-| Vulnerability Assessment Tools | ✅ Pass | 6.281s | |
-| SQL Injection Attack Type | ✅ Pass | 4.479s | |
-| Penetration Testing Framework | ✅ Pass | 6.102s | |
-| Web Application Security Scanner | ✅ Pass | 6.151s | |
-| Penetration Testing Tool Selection | ❌ Fail | 6.783s | no tool calls found, expected at least 1 |
+| JSON Response Function | ✅ Pass | 3.867s | |
+| Search Query Function | ✅ Pass | 3.219s | |
+| Ask Advice Function | ✅ Pass | 5.921s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 2.671s | |
+| Basic Context Memory Test | ✅ Pass | 3.188s | |
+| Function Argument Memory Test | ✅ Pass | 4.451s | |
+| Function Response Memory Test | ✅ Pass | 3.309s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 6.425s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 4.503s | |
+| Penetration Testing Methodology | ✅ Pass | 5.211s | |
+| Vulnerability Assessment Tools | ✅ Pass | 5.501s | |
+| SQL Injection Attack Type | ✅ Pass | 4.066s | |
+| Penetration Testing Framework | ✅ Pass | 5.491s | |
+| Web Application Security Scanner | ✅ Pass | 5.037s | |
+| Penetration Testing Tool Selection | ✅ Pass | 6.604s | |
-**Summary**: 21/23 (91.30%) successful tests
+**Summary**: 22/23 (95.65%) successful tests
-**Average latency**: 6.289s
+**Average latency**: 4.239s
---
@@ -172,38 +172,38 @@ Generated: Thu, 05 Mar 2026 17:08:56 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 4.409s | |
-| Text Transform Uppercase | ✅ Pass | 5.281s | |
-| Count from 1 to 5 | ✅ Pass | 5.887s | |
-| Math Calculation | ✅ Pass | 4.106s | |
-| Basic Echo Function | ✅ Pass | 12.134s | |
-| Streaming Simple Math Streaming | ✅ Pass | 3.296s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.472s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 4.108s | |
+| Simple Math | ✅ Pass | 3.066s | |
+| Text Transform Uppercase | ✅ Pass | 4.138s | |
+| Count from 1 to 5 | ✅ Pass | 3.613s | |
+| Math Calculation | ✅ Pass | 3.089s | |
+| Basic Echo Function | ❌ Fail | 4.109s | no tool calls found, expected at least 1 |
+| Streaming Simple Math Streaming | ✅ Pass | 3.430s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.411s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.347s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 17.754s | |
-| Search Query Function | ✅ Pass | 22.188s | |
-| Ask Advice Function | ✅ Pass | 7.185s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 6.614s | |
-| Basic Context Memory Test | ✅ Pass | 4.104s | |
-| Function Argument Memory Test | ✅ Pass | 5.608s | |
-| Function Response Memory Test | ✅ Pass | 4.502s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 7.979s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 6.700s | |
-| Penetration Testing Methodology | ✅ Pass | 8.708s | |
-| Vulnerability Assessment Tools | ✅ Pass | 8.459s | |
-| SQL Injection Attack Type | ✅ Pass | 3.890s | |
-| Penetration Testing Framework | ✅ Pass | 10.137s | |
-| Web Application Security Scanner | ✅ Pass | 7.074s | |
-| Penetration Testing Tool Selection | ✅ Pass | 7.520s | |
+| JSON Response Function | ✅ Pass | 4.487s | |
+| Search Query Function | ✅ Pass | 3.946s | |
+| Ask Advice Function | ✅ Pass | 4.281s | |
+| Streaming Search Query Function Streaming | ❌ Fail | 3.204s | no tool calls found, expected at least 1 |
+| Basic Context Memory Test | ✅ Pass | 4.219s | |
+| Function Argument Memory Test | ✅ Pass | 4.124s | |
+| Function Response Memory Test | ✅ Pass | 3.339s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 7.760s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.648s | |
+| Penetration Testing Methodology | ✅ Pass | 7.310s | |
+| Vulnerability Assessment Tools | ✅ Pass | 9.153s | |
+| SQL Injection Attack Type | ✅ Pass | 3.768s | |
+| Penetration Testing Framework | ✅ Pass | 3.997s | |
+| Web Application Security Scanner | ✅ Pass | 4.903s | |
+| Penetration Testing Tool Selection | ✅ Pass | 6.634s | |
-**Summary**: 23/23 (100.00%) successful tests
+**Summary**: 21/23 (91.30%) successful tests
-**Average latency**: 7.440s
+**Average latency**: 4.434s
---
@@ -213,38 +213,38 @@ Generated: Thu, 05 Mar 2026 17:08:56 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 3.836s | |
-| Text Transform Uppercase | ✅ Pass | 4.510s | |
-| Count from 1 to 5 | ✅ Pass | 4.798s | |
-| Math Calculation | ✅ Pass | 3.319s | |
-| Basic Echo Function | ✅ Pass | 8.214s | |
-| Streaming Simple Math Streaming | ✅ Pass | 4.405s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.426s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 3.710s | |
+| Simple Math | ✅ Pass | 3.559s | |
+| Text Transform Uppercase | ✅ Pass | 4.160s | |
+| Count from 1 to 5 | ✅ Pass | 3.893s | |
+| Math Calculation | ✅ Pass | 2.809s | |
+| Basic Echo Function | ✅ Pass | 4.024s | |
+| Streaming Simple Math Streaming | ✅ Pass | 3.314s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.412s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.594s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 12.893s | |
-| Search Query Function | ✅ Pass | 5.456s | |
-| Ask Advice Function | ✅ Pass | 14.030s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 4.218s | |
-| Basic Context Memory Test | ✅ Pass | 4.220s | |
-| Function Argument Memory Test | ✅ Pass | 4.692s | |
-| Function Response Memory Test | ✅ Pass | 4.569s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 6.465s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 4.908s | |
-| Penetration Testing Methodology | ✅ Pass | 6.765s | |
-| Vulnerability Assessment Tools | ✅ Pass | 6.448s | |
-| Penetration Testing Framework | ✅ Pass | 5.388s | |
-| Web Application Security Scanner | ✅ Pass | 8.114s | |
-| SQL Injection Attack Type | ✅ Pass | 163.281s | |
-| Penetration Testing Tool Selection | ❌ Fail | 3.896s | no tool calls found, expected at least 1 |
+| JSON Response Function | ✅ Pass | 4.043s | |
+| Search Query Function | ✅ Pass | 3.295s | |
+| Ask Advice Function | ✅ Pass | 4.504s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 2.448s | |
+| Basic Context Memory Test | ✅ Pass | 3.640s | |
+| Function Argument Memory Test | ✅ Pass | 3.824s | |
+| Function Response Memory Test | ✅ Pass | 4.427s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 7.450s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.576s | |
+| Penetration Testing Methodology | ✅ Pass | 5.822s | |
+| Vulnerability Assessment Tools | ✅ Pass | 6.900s | |
+| SQL Injection Attack Type | ✅ Pass | 3.912s | |
+| Penetration Testing Framework | ✅ Pass | 4.930s | |
+| Web Application Security Scanner | ✅ Pass | 5.459s | |
+| Penetration Testing Tool Selection | ✅ Pass | 6.079s | |
-**Summary**: 22/23 (95.65%) successful tests
+**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 12.764s
+**Average latency**: 4.221s
---
@@ -254,161 +254,161 @@ Generated: Thu, 05 Mar 2026 17:08:56 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 4.233s | |
-| Text Transform Uppercase | ✅ Pass | 5.863s | |
-| Count from 1 to 5 | ✅ Pass | 5.006s | |
-| Math Calculation | ✅ Pass | 3.472s | |
-| Basic Echo Function | ✅ Pass | 9.962s | |
-| Streaming Simple Math Streaming | ✅ Pass | 6.602s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 9.473s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 3.990s | |
+| Simple Math | ✅ Pass | 2.811s | |
+| Text Transform Uppercase | ✅ Pass | 3.618s | |
+| Count from 1 to 5 | ✅ Pass | 5.524s | |
+| Math Calculation | ✅ Pass | 3.096s | |
+| Basic Echo Function | ✅ Pass | 5.575s | |
+| Streaming Simple Math Streaming | ✅ Pass | 2.725s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.862s | |
+| Streaming Basic Echo Function Streaming | ❌ Fail | 3.410s | no tool calls found, expected at least 1 |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 10.251s | |
-| Search Query Function | ❌ Fail | 5.857s | no tool calls found, expected at least 1 |
-| Ask Advice Function | ✅ Pass | 4.049s | |
-| Streaming Search Query Function Streaming | ❌ Fail | 5.435s | no tool calls found, expected at least 1 |
-| Basic Context Memory Test | ✅ Pass | 4.114s | |
-| Function Argument Memory Test | ✅ Pass | 4.434s | |
-| Function Response Memory Test | ✅ Pass | 4.202s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 7.379s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 6.014s | |
-| Penetration Testing Methodology | ✅ Pass | 9.402s | |
-| Vulnerability Assessment Tools | ✅ Pass | 6.589s | |
-| SQL Injection Attack Type | ✅ Pass | 6.598s | |
-| Penetration Testing Framework | ✅ Pass | 7.364s | |
-| Web Application Security Scanner | ✅ Pass | 5.184s | |
-| Penetration Testing Tool Selection | ✅ Pass | 6.397s | |
+| JSON Response Function | ✅ Pass | 3.542s | |
+| Search Query Function | ❌ Fail | 4.056s | no tool calls found, expected at least 1 |
+| Ask Advice Function | ✅ Pass | 3.966s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 2.561s | |
+| Basic Context Memory Test | ✅ Pass | 3.647s | |
+| Function Argument Memory Test | ✅ Pass | 4.782s | |
+| Function Response Memory Test | ✅ Pass | 4.070s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 6.209s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.835s | |
+| Penetration Testing Methodology | ✅ Pass | 6.464s | |
+| Vulnerability Assessment Tools | ✅ Pass | 6.427s | |
+| SQL Injection Attack Type | ✅ Pass | 3.244s | |
+| Penetration Testing Framework | ✅ Pass | 5.763s | |
+| Web Application Security Scanner | ✅ Pass | 5.714s | |
+| Penetration Testing Tool Selection | ✅ Pass | 5.082s | |
**Summary**: 21/23 (91.30%) successful tests
-**Average latency**: 6.169s
+**Average latency**: 4.304s
---
-### reflector (gemini-3-flash-preview)
+### reflector (gemini-3.5-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 2.522s | |
-| Text Transform Uppercase | ✅ Pass | 1.702s | |
-| Count from 1 to 5 | ✅ Pass | 2.115s | |
-| Math Calculation | ✅ Pass | 1.125s | |
-| Basic Echo Function | ✅ Pass | 1.679s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.487s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.506s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 1.450s | |
+| Simple Math | ✅ Pass | 1.016s | |
+| Text Transform Uppercase | ✅ Pass | 1.398s | |
+| Count from 1 to 5 | ✅ Pass | 1.549s | |
+| Math Calculation | ✅ Pass | 1.023s | |
+| Basic Echo Function | ✅ Pass | 1.100s | |
+| Streaming Simple Math Streaming | ✅ Pass | 0.953s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.477s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.172s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 1.182s | |
-| Search Query Function | ✅ Pass | 1.515s | |
-| Ask Advice Function | ✅ Pass | 1.298s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.354s | |
-| Basic Context Memory Test | ✅ Pass | 1.174s | |
-| Function Argument Memory Test | ✅ Pass | 1.423s | |
-| Function Response Memory Test | ✅ Pass | 1.403s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 3.036s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.681s | |
-| Penetration Testing Methodology | ✅ Pass | 3.639s | |
-| Vulnerability Assessment Tools | ✅ Pass | 5.827s | |
-| SQL Injection Attack Type | ✅ Pass | 1.681s | |
-| Penetration Testing Framework | ✅ Pass | 2.840s | |
-| Web Application Security Scanner | ✅ Pass | 2.972s | |
+| JSON Response Function | ✅ Pass | 1.330s | |
+| Search Query Function | ✅ Pass | 0.972s | |
+| Ask Advice Function | ✅ Pass | 1.386s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.149s | |
+| Basic Context Memory Test | ✅ Pass | 1.303s | |
+| Function Argument Memory Test | ✅ Pass | 1.155s | |
+| Function Response Memory Test | ✅ Pass | 1.319s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 4.053s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.048s | |
+| Penetration Testing Methodology | ✅ Pass | 2.124s | |
+| Vulnerability Assessment Tools | ✅ Pass | 3.923s | |
+| SQL Injection Attack Type | ✅ Pass | 1.678s | |
+| Penetration Testing Framework | ✅ Pass | 2.756s | |
+| Web Application Security Scanner | ✅ Pass | 1.608s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.421s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 2.045s
+**Average latency**: 1.692s
---
-### searcher (gemini-3-flash-preview)
+### searcher (gemini-3.5-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 2.055s | |
-| Text Transform Uppercase | ✅ Pass | 1.362s | |
-| Count from 1 to 5 | ✅ Pass | 1.617s | |
-| Math Calculation | ✅ Pass | 1.431s | |
-| Basic Echo Function | ✅ Pass | 1.369s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.326s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.519s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 1.820s | |
+| Simple Math | ✅ Pass | 0.992s | |
+| Text Transform Uppercase | ✅ Pass | 1.363s | |
+| Count from 1 to 5 | ✅ Pass | 1.590s | |
+| Math Calculation | ✅ Pass | 0.971s | |
+| Basic Echo Function | ✅ Pass | 1.336s | |
+| Streaming Simple Math Streaming | ✅ Pass | 0.916s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.542s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.124s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 0.993s | |
-| Search Query Function | ✅ Pass | 1.155s | |
-| Ask Advice Function | ✅ Pass | 1.018s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.403s | |
-| Basic Context Memory Test | ✅ Pass | 2.049s | |
-| Function Argument Memory Test | ✅ Pass | 1.272s | |
-| Function Response Memory Test | ✅ Pass | 1.256s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 2.351s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.467s | |
-| Penetration Testing Methodology | ✅ Pass | 3.546s | |
-| Vulnerability Assessment Tools | ✅ Pass | 6.066s | |
-| SQL Injection Attack Type | ✅ Pass | 1.849s | |
-| Penetration Testing Framework | ✅ Pass | 3.731s | |
-| Web Application Security Scanner | ✅ Pass | 3.150s | |
-| Penetration Testing Tool Selection | ✅ Pass | 1.988s | |
+| JSON Response Function | ✅ Pass | 1.079s | |
+| Search Query Function | ✅ Pass | 1.121s | |
+| Ask Advice Function | ✅ Pass | 1.387s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.260s | |
+| Basic Context Memory Test | ✅ Pass | 1.399s | |
+| Function Argument Memory Test | ✅ Pass | 1.557s | |
+| Function Response Memory Test | ✅ Pass | 1.474s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 4.472s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.838s | |
+| Penetration Testing Methodology | ✅ Pass | 2.772s | |
+| Vulnerability Assessment Tools | ✅ Pass | 4.313s | |
+| SQL Injection Attack Type | ✅ Pass | 1.202s | |
+| Penetration Testing Framework | ✅ Pass | 2.830s | |
+| Web Application Security Scanner | ✅ Pass | 2.120s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.116s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 1.992s
+**Average latency**: 1.817s
---
-### enricher (gemini-3-flash-preview)
+### enricher (gemini-3.5-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 2.531s | |
-| Text Transform Uppercase | ✅ Pass | 1.052s | |
-| Count from 1 to 5 | ✅ Pass | 1.923s | |
-| Math Calculation | ✅ Pass | 1.989s | |
-| Basic Echo Function | ✅ Pass | 1.358s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.571s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.678s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 1.817s | |
+| Simple Math | ✅ Pass | 0.981s | |
+| Text Transform Uppercase | ✅ Pass | 1.215s | |
+| Count from 1 to 5 | ✅ Pass | 1.425s | |
+| Math Calculation | ✅ Pass | 0.932s | |
+| Basic Echo Function | ✅ Pass | 1.084s | |
+| Streaming Simple Math Streaming | ✅ Pass | 0.958s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.604s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.237s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 1.947s | |
-| Search Query Function | ✅ Pass | 1.491s | |
-| Ask Advice Function | ✅ Pass | 1.126s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 2.128s | |
-| Basic Context Memory Test | ✅ Pass | 1.206s | |
-| Function Argument Memory Test | ✅ Pass | 1.426s | |
-| Function Response Memory Test | ✅ Pass | 1.258s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 2.798s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.380s | |
-| Penetration Testing Methodology | ✅ Pass | 3.086s | |
-| Vulnerability Assessment Tools | ✅ Pass | 6.220s | |
-| SQL Injection Attack Type | ✅ Pass | 1.592s | |
-| Penetration Testing Framework | ✅ Pass | 3.472s | |
-| Web Application Security Scanner | ✅ Pass | 3.306s | |
-| Penetration Testing Tool Selection | ✅ Pass | 2.093s | |
+| JSON Response Function | ✅ Pass | 1.128s | |
+| Search Query Function | ✅ Pass | 0.993s | |
+| Ask Advice Function | ✅ Pass | 1.090s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.009s | |
+| Basic Context Memory Test | ✅ Pass | 1.461s | |
+| Function Argument Memory Test | ✅ Pass | 0.955s | |
+| Function Response Memory Test | ✅ Pass | 1.856s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 4.277s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.771s | |
+| Penetration Testing Methodology | ✅ Pass | 2.691s | |
+| Vulnerability Assessment Tools | ✅ Pass | 3.802s | |
+| SQL Injection Attack Type | ✅ Pass | 1.611s | |
+| Penetration Testing Framework | ✅ Pass | 2.182s | |
+| Web Application Security Scanner | ✅ Pass | 2.253s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.135s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 2.107s
+**Average latency**: 1.724s
---
@@ -418,79 +418,79 @@ Generated: Thu, 05 Mar 2026 17:08:56 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 6.564s | |
-| Text Transform Uppercase | ✅ Pass | 9.168s | |
-| Count from 1 to 5 | ✅ Pass | 4.297s | |
-| Math Calculation | ✅ Pass | 12.848s | |
-| Basic Echo Function | ✅ Pass | 4.367s | |
-| Streaming Simple Math Streaming | ✅ Pass | 4.170s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.534s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 3.830s | |
+| Simple Math | ✅ Pass | 3.599s | |
+| Text Transform Uppercase | ✅ Pass | 3.774s | |
+| Count from 1 to 5 | ✅ Pass | 3.470s | |
+| Math Calculation | ✅ Pass | 3.170s | |
+| Basic Echo Function | ✅ Pass | 4.147s | |
+| Streaming Simple Math Streaming | ✅ Pass | 2.895s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.859s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.385s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 9.196s | |
-| Search Query Function | ✅ Pass | 4.121s | |
-| Ask Advice Function | ✅ Pass | 5.221s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 2.875s | |
-| Basic Context Memory Test | ✅ Pass | 4.935s | |
-| Function Argument Memory Test | ✅ Pass | 4.348s | |
-| Function Response Memory Test | ✅ Pass | 4.011s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 7.053s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 3.931s | |
-| Penetration Testing Methodology | ✅ Pass | 8.298s | |
-| Vulnerability Assessment Tools | ✅ Pass | 5.146s | |
-| SQL Injection Attack Type | ✅ Pass | 4.431s | |
-| Penetration Testing Framework | ✅ Pass | 5.921s | |
-| Web Application Security Scanner | ✅ Pass | 4.735s | |
-| Penetration Testing Tool Selection | ✅ Pass | 6.905s | |
+| JSON Response Function | ✅ Pass | 5.047s | |
+| Search Query Function | ✅ Pass | 3.097s | |
+| Ask Advice Function | ✅ Pass | 3.666s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 2.431s | |
+| Basic Context Memory Test | ✅ Pass | 3.474s | |
+| Function Argument Memory Test | ✅ Pass | 3.706s | |
+| Function Response Memory Test | ✅ Pass | 3.504s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 6.181s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.421s | |
+| Penetration Testing Methodology | ✅ Pass | 5.411s | |
+| Vulnerability Assessment Tools | ✅ Pass | 8.503s | |
+| SQL Injection Attack Type | ✅ Pass | 3.996s | |
+| Penetration Testing Framework | ✅ Pass | 4.476s | |
+| Web Application Security Scanner | ✅ Pass | 4.807s | |
+| Penetration Testing Tool Selection | ✅ Pass | 6.343s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 5.779s
+**Average latency**: 4.103s
---
-### installer (gemini-3-flash-preview)
+### installer (gemini-3.5-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 4.176s | |
-| Text Transform Uppercase | ✅ Pass | 2.361s | |
-| Count from 1 to 5 | ✅ Pass | 3.194s | |
-| Math Calculation | ✅ Pass | 2.707s | |
-| Basic Echo Function | ✅ Pass | 2.371s | |
-| Streaming Simple Math Streaming | ✅ Pass | 2.318s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.116s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 1.306s | |
+| Simple Math | ✅ Pass | 2.355s | |
+| Text Transform Uppercase | ✅ Pass | 3.024s | |
+| Count from 1 to 5 | ✅ Pass | 2.930s | |
+| Math Calculation | ✅ Pass | 1.667s | |
+| Basic Echo Function | ✅ Pass | 2.377s | |
+| Streaming Simple Math Streaming | ✅ Pass | 2.014s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.497s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.267s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 1.202s | |
-| Search Query Function | ✅ Pass | 2.480s | |
-| Ask Advice Function | ✅ Pass | 1.455s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.719s | |
-| Basic Context Memory Test | ✅ Pass | 2.621s | |
-| Function Argument Memory Test | ✅ Pass | 2.249s | |
-| Function Response Memory Test | ✅ Pass | 2.472s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 3.575s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 2.560s | |
-| Penetration Testing Methodology | ✅ Pass | 5.055s | |
-| Vulnerability Assessment Tools | ✅ Pass | 4.685s | |
-| SQL Injection Attack Type | ✅ Pass | 2.319s | |
-| Penetration Testing Framework | ✅ Pass | 5.229s | |
-| Web Application Security Scanner | ✅ Pass | 4.249s | |
-| Penetration Testing Tool Selection | ✅ Pass | 1.111s | |
+| JSON Response Function | ✅ Pass | 1.773s | |
+| Search Query Function | ✅ Pass | 2.193s | |
+| Ask Advice Function | ✅ Pass | 1.839s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.064s | |
+| Basic Context Memory Test | ✅ Pass | 2.661s | |
+| Function Argument Memory Test | ✅ Pass | 2.603s | |
+| Function Response Memory Test | ✅ Pass | 2.563s | |
+| Penetration Testing Memory with Tool Call | ❌ Fail | 5.230s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 5.456s | |
+| Penetration Testing Methodology | ✅ Pass | 4.745s | |
+| Vulnerability Assessment Tools | ✅ Pass | 5.977s | |
+| SQL Injection Attack Type | ✅ Pass | 2.499s | |
+| Penetration Testing Framework | ✅ Pass | 4.620s | |
+| Web Application Security Scanner | ✅ Pass | 4.105s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.022s | |
-**Summary**: 23/23 (100.00%) successful tests
+**Summary**: 22/23 (95.65%) successful tests
-**Average latency**: 2.763s
+**Average latency**: 2.891s
---
@@ -500,38 +500,38 @@ Generated: Thu, 05 Mar 2026 17:08:56 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 3.938s | |
-| Text Transform Uppercase | ✅ Pass | 5.110s | |
-| Count from 1 to 5 | ✅ Pass | 4.386s | |
-| Math Calculation | ✅ Pass | 4.925s | |
-| Basic Echo Function | ❌ Fail | 10.105s | no tool calls found, expected at least 1 |
-| Streaming Simple Math Streaming | ✅ Pass | 7.901s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.401s | |
-| Streaming Basic Echo Function Streaming | ❌ Fail | 3.443s | no tool calls found, expected at least 1 |
+| Simple Math | ✅ Pass | 2.995s | |
+| Text Transform Uppercase | ✅ Pass | 3.980s | |
+| Count from 1 to 5 | ✅ Pass | 3.836s | |
+| Math Calculation | ✅ Pass | 3.256s | |
+| Basic Echo Function | ✅ Pass | 4.078s | |
+| Streaming Simple Math Streaming | ✅ Pass | 3.645s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.757s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.314s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 5.250s | |
-| Search Query Function | ✅ Pass | 8.325s | |
-| Ask Advice Function | ✅ Pass | 4.344s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 3.315s | |
-| Basic Context Memory Test | ✅ Pass | 5.149s | |
-| Function Argument Memory Test | ✅ Pass | 3.930s | |
-| Function Response Memory Test | ✅ Pass | 4.254s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 7.142s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 3.946s | |
-| Penetration Testing Methodology | ✅ Pass | 6.441s | |
-| Vulnerability Assessment Tools | ✅ Pass | 8.660s | |
-| SQL Injection Attack Type | ✅ Pass | 5.839s | |
-| Penetration Testing Framework | ✅ Pass | 6.380s | |
-| Web Application Security Scanner | ✅ Pass | 8.225s | |
-| Penetration Testing Tool Selection | ✅ Pass | 6.434s | |
+| JSON Response Function | ✅ Pass | 4.023s | |
+| Search Query Function | ✅ Pass | 3.725s | |
+| Ask Advice Function | ✅ Pass | 4.173s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 2.589s | |
+| Basic Context Memory Test | ✅ Pass | 3.997s | |
+| Function Argument Memory Test | ✅ Pass | 4.753s | |
+| Function Response Memory Test | ✅ Pass | 4.861s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 5.704s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.880s | |
+| Penetration Testing Methodology | ✅ Pass | 6.182s | |
+| Vulnerability Assessment Tools | ✅ Pass | 5.410s | |
+| SQL Injection Attack Type | ✅ Pass | 3.714s | |
+| Penetration Testing Framework | ✅ Pass | 4.977s | |
+| Web Application Security Scanner | ✅ Pass | 5.953s | |
+| Penetration Testing Tool Selection | ✅ Pass | 5.708s | |
-**Summary**: 21/23 (91.30%) successful tests
+**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 5.733s
+**Average latency**: 4.197s
---
diff --git a/examples/tests/glm-report.md b/examples/tests/glm-report.md
index 2fd80a0a..8d362137 100644
--- a/examples/tests/glm-report.md
+++ b/examples/tests/glm-report.md
@@ -1,291 +1,291 @@
# LLM Agent Testing Report
-Generated: Thu, 05 Mar 2026 16:50:23 UTC
+Generated: Thu, 28 May 2026 12:25:41 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
-| simple | glm-4.7-flashx | true | 22/23 (95.65%) | 20.579s |
-| simple_json | glm-4.7-flashx | true | 5/5 (100.00%) | 7.107s |
-| primary_agent | glm-5 | true | 23/23 (100.00%) | 7.050s |
-| assistant | glm-5 | true | 23/23 (100.00%) | 7.197s |
-| generator | glm-5 | true | 23/23 (100.00%) | 6.794s |
-| refiner | glm-5 | true | 23/23 (100.00%) | 7.235s |
-| adviser | glm-5 | true | 23/23 (100.00%) | 7.876s |
-| reflector | glm-4.5-air | true | 23/23 (100.00%) | 6.347s |
-| searcher | glm-4.5-air | true | 23/23 (100.00%) | 5.492s |
-| enricher | glm-4.5-air | true | 23/23 (100.00%) | 6.488s |
-| coder | glm-5 | true | 23/23 (100.00%) | 6.128s |
-| installer | glm-4.7 | true | 23/23 (100.00%) | 3.903s |
-| pentester | glm-4.7 | true | 22/23 (95.65%) | 5.350s |
-
-**Total**: 279/281 (99.29%) successful tests
-**Overall average latency**: 7.529s
+| simple | glm-4.5-air | false | 23/23 (100.00%) | 3.213s |
+| simple_json | glm-4.5-air | false | 5/5 (100.00%) | 3.923s |
+| primary_agent | glm-5-turbo | true | 23/23 (100.00%) | 4.488s |
+| assistant | glm-5-turbo | true | 22/23 (95.65%) | 3.383s |
+| generator | glm-5.1 | true | 23/23 (100.00%) | 7.081s |
+| refiner | glm-5.1 | true | 22/23 (95.65%) | 6.864s |
+| adviser | glm-5.1 | true | 23/23 (100.00%) | 6.553s |
+| reflector | glm-4.5-air | true | 23/23 (100.00%) | 2.760s |
+| searcher | glm-4.5-air | true | 23/23 (100.00%) | 2.889s |
+| enricher | glm-4.5-air | true | 23/23 (100.00%) | 3.001s |
+| coder | glm-5.1 | true | 22/23 (95.65%) | 7.172s |
+| installer | glm-4.5-air | true | 23/23 (100.00%) | 6.146s |
+| pentester | glm-5.1 | true | 23/23 (100.00%) | 6.772s |
+
+**Total**: 278/281 (98.93%) successful tests
+**Overall average latency**: 5.007s
## Detailed Results
-### simple (glm-4.7-flashx)
+### simple (glm-4.5-air)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 3.350s | |
-| Text Transform Uppercase | ✅ Pass | 3.344s | |
-| Count from 1 to 5 | ✅ Pass | 16.684s | |
-| Math Calculation | ✅ Pass | 21.074s | |
-| Streaming Simple Math Streaming | ✅ Pass | 3.029s | |
-| Basic Echo Function | ✅ Pass | 83.870s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.848s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 1.779s | |
+| Simple Math | ✅ Pass | 4.356s | |
+| Text Transform Uppercase | ✅ Pass | 1.140s | |
+| Count from 1 to 5 | ✅ Pass | 1.534s | |
+| Math Calculation | ✅ Pass | 1.975s | |
+| Basic Echo Function | ✅ Pass | 1.529s | |
+| Streaming Simple Math Streaming | ✅ Pass | 0.911s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.859s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.170s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 32.834s | |
-| Search Query Function | ✅ Pass | 1.665s | |
-| Ask Advice Function | ✅ Pass | 4.198s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 38.178s | |
-| Basic Context Memory Test | ✅ Pass | 4.826s | |
-| Function Argument Memory Test | ✅ Pass | 51.598s | |
-| Function Response Memory Test | ✅ Pass | 59.462s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 18.542s | |
-| Cybersecurity Workflow Memory Test | ❌ Fail | 2.568s | expected text 'example\.com' not found |
-| Penetration Testing Methodology | ✅ Pass | 11.113s | |
-| Vulnerability Assessment Tools | ✅ Pass | 53.941s | |
-| SQL Injection Attack Type | ✅ Pass | 7.778s | |
-| Penetration Testing Framework | ✅ Pass | 32.387s | |
-| Web Application Security Scanner | ✅ Pass | 16.517s | |
-| Penetration Testing Tool Selection | ✅ Pass | 1.715s | |
+| JSON Response Function | ✅ Pass | 1.415s | |
+| Search Query Function | ✅ Pass | 3.528s | |
+| Ask Advice Function | ✅ Pass | 1.866s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.467s | |
+| Basic Context Memory Test | ✅ Pass | 1.478s | |
+| Function Argument Memory Test | ✅ Pass | 1.072s | |
+| Function Response Memory Test | ✅ Pass | 2.755s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 3.953s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 2.161s | |
+| Penetration Testing Methodology | ✅ Pass | 7.480s | |
+| Vulnerability Assessment Tools | ✅ Pass | 18.361s | |
+| SQL Injection Attack Type | ✅ Pass | 1.326s | |
+| Penetration Testing Framework | ✅ Pass | 7.132s | |
+| Web Application Security Scanner | ✅ Pass | 3.189s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.219s | |
-**Summary**: 22/23 (95.65%) successful tests
+**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 20.579s
+**Average latency**: 3.213s
---
-### simple_json (glm-4.7-flashx)
+### simple_json (glm-4.5-air)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Person Information JSON | ✅ Pass | 3.598s | |
-| Project Information JSON | ✅ Pass | 4.589s | |
-| User Profile JSON | ✅ Pass | 2.576s | |
-| Streaming Person Information JSON Streaming | ✅ Pass | 2.393s | |
-| Vulnerability Report Memory Test | ✅ Pass | 22.377s | |
+| Person Information JSON | ✅ Pass | 4.516s | |
+| Vulnerability Report Memory Test | ✅ Pass | 5.454s | |
+| Project Information JSON | ✅ Pass | 1.830s | |
+| User Profile JSON | ✅ Pass | 3.002s | |
+| Streaming Person Information JSON Streaming | ✅ Pass | 4.808s | |
**Summary**: 5/5 (100.00%) successful tests
-**Average latency**: 7.107s
+**Average latency**: 3.923s
---
-### primary_agent (glm-5)
+### primary_agent (glm-5-turbo)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 3.481s | |
-| Text Transform Uppercase | ✅ Pass | 2.408s | |
-| Count from 1 to 5 | ✅ Pass | 5.159s | |
-| Math Calculation | ✅ Pass | 1.981s | |
-| Basic Echo Function | ✅ Pass | 4.224s | |
-| Streaming Simple Math Streaming | ✅ Pass | 4.030s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.049s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 4.907s | |
+| Simple Math | ✅ Pass | 5.935s | |
+| Text Transform Uppercase | ✅ Pass | 3.695s | |
+| Count from 1 to 5 | ✅ Pass | 3.836s | |
+| Math Calculation | ✅ Pass | 3.798s | |
+| Basic Echo Function | ✅ Pass | 2.191s | |
+| Streaming Simple Math Streaming | ✅ Pass | 3.538s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.337s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.491s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 3.821s | |
-| Search Query Function | ✅ Pass | 2.699s | |
-| Ask Advice Function | ✅ Pass | 4.990s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 3.747s | |
-| Basic Context Memory Test | ✅ Pass | 1.676s | |
-| Function Argument Memory Test | ✅ Pass | 3.306s | |
-| Function Response Memory Test | ✅ Pass | 1.959s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 5.072s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 3.079s | |
-| Penetration Testing Methodology | ✅ Pass | 18.268s | |
-| Vulnerability Assessment Tools | ✅ Pass | 39.499s | |
-| SQL Injection Attack Type | ✅ Pass | 3.957s | |
-| Penetration Testing Framework | ✅ Pass | 22.103s | |
-| Web Application Security Scanner | ✅ Pass | 11.489s | |
-| Penetration Testing Tool Selection | ✅ Pass | 5.231s | |
+| JSON Response Function | ✅ Pass | 1.857s | |
+| Search Query Function | ✅ Pass | 1.869s | |
+| Ask Advice Function | ✅ Pass | 1.275s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.561s | |
+| Basic Context Memory Test | ✅ Pass | 5.012s | |
+| Function Argument Memory Test | ✅ Pass | 3.120s | |
+| Function Response Memory Test | ✅ Pass | 1.293s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.664s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 4.496s | |
+| Penetration Testing Methodology | ✅ Pass | 22.967s | |
+| Vulnerability Assessment Tools | ✅ Pass | 9.455s | |
+| SQL Injection Attack Type | ✅ Pass | 5.215s | |
+| Penetration Testing Framework | ✅ Pass | 5.249s | |
+| Web Application Security Scanner | ✅ Pass | 5.172s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.180s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 7.050s
+**Average latency**: 4.488s
---
-### assistant (glm-5)
+### assistant (glm-5-turbo)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 2.425s | |
-| Text Transform Uppercase | ✅ Pass | 5.846s | |
-| Count from 1 to 5 | ✅ Pass | 3.686s | |
-| Math Calculation | ✅ Pass | 2.497s | |
-| Basic Echo Function | ✅ Pass | 3.883s | |
-| Streaming Simple Math Streaming | ✅ Pass | 3.891s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.354s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 4.570s | |
+| Simple Math | ✅ Pass | 1.980s | |
+| Text Transform Uppercase | ✅ Pass | 2.896s | |
+| Count from 1 to 5 | ✅ Pass | 2.572s | |
+| Math Calculation | ✅ Pass | 3.321s | |
+| Basic Echo Function | ✅ Pass | 1.431s | |
+| Streaming Simple Math Streaming | ✅ Pass | 3.935s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.138s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.242s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 3.925s | |
-| Search Query Function | ✅ Pass | 2.993s | |
-| Ask Advice Function | ✅ Pass | 4.531s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 3.684s | |
-| Basic Context Memory Test | ✅ Pass | 2.207s | |
-| Function Argument Memory Test | ✅ Pass | 2.674s | |
-| Function Response Memory Test | ✅ Pass | 2.899s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 9.218s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 2.481s | |
-| Penetration Testing Methodology | ✅ Pass | 19.841s | |
-| Vulnerability Assessment Tools | ✅ Pass | 30.548s | |
-| SQL Injection Attack Type | ✅ Pass | 7.352s | |
-| Penetration Testing Framework | ✅ Pass | 26.431s | |
-| Web Application Security Scanner | ✅ Pass | 13.787s | |
-| Penetration Testing Tool Selection | ✅ Pass | 3.800s | |
+| JSON Response Function | ✅ Pass | 1.378s | |
+| Search Query Function | ✅ Pass | 1.954s | |
+| Ask Advice Function | ✅ Pass | 1.638s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.390s | |
+| Basic Context Memory Test | ✅ Pass | 4.105s | |
+| Function Argument Memory Test | ✅ Pass | 1.651s | |
+| Function Response Memory Test | ✅ Pass | 2.104s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 6.155s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 5.861s | |
+| Penetration Testing Methodology | ✅ Pass | 5.751s | |
+| Vulnerability Assessment Tools | ❌ Fail | 6.224s | expected text 'network' not found |
+| SQL Injection Attack Type | ✅ Pass | 4.402s | |
+| Penetration Testing Framework | ✅ Pass | 6.187s | |
+| Web Application Security Scanner | ✅ Pass | 5.577s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.904s | |
-**Summary**: 23/23 (100.00%) successful tests
+**Summary**: 22/23 (95.65%) successful tests
-**Average latency**: 7.197s
+**Average latency**: 3.383s
---
-### generator (glm-5)
+### generator (glm-5.1)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 3.151s | |
-| Text Transform Uppercase | ✅ Pass | 5.508s | |
-| Count from 1 to 5 | ✅ Pass | 3.995s | |
-| Math Calculation | ✅ Pass | 4.557s | |
-| Basic Echo Function | ✅ Pass | 6.516s | |
-| Streaming Simple Math Streaming | ✅ Pass | 4.617s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.486s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 3.614s | |
+| Simple Math | ✅ Pass | 5.598s | |
+| Text Transform Uppercase | ✅ Pass | 4.686s | |
+| Count from 1 to 5 | ✅ Pass | 4.534s | |
+| Math Calculation | ✅ Pass | 4.466s | |
+| Basic Echo Function | ✅ Pass | 5.033s | |
+| Streaming Simple Math Streaming | ✅ Pass | 3.667s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.827s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 3.858s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 4.064s | |
-| Search Query Function | ✅ Pass | 3.644s | |
-| Ask Advice Function | ✅ Pass | 4.685s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 3.579s | |
-| Basic Context Memory Test | ✅ Pass | 2.774s | |
-| Function Argument Memory Test | ✅ Pass | 3.489s | |
-| Function Response Memory Test | ✅ Pass | 5.024s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 5.929s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 2.633s | |
-| Penetration Testing Methodology | ✅ Pass | 14.089s | |
-| Vulnerability Assessment Tools | ✅ Pass | 23.320s | |
-| SQL Injection Attack Type | ✅ Pass | 5.590s | |
-| Penetration Testing Framework | ✅ Pass | 21.081s | |
-| Web Application Security Scanner | ✅ Pass | 16.597s | |
-| Penetration Testing Tool Selection | ✅ Pass | 5.296s | |
+| JSON Response Function | ✅ Pass | 5.855s | |
+| Search Query Function | ✅ Pass | 5.443s | |
+| Ask Advice Function | ✅ Pass | 3.380s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 4.654s | |
+| Basic Context Memory Test | ✅ Pass | 7.745s | |
+| Function Argument Memory Test | ✅ Pass | 9.781s | |
+| Function Response Memory Test | ✅ Pass | 6.321s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 6.979s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 5.237s | |
+| Penetration Testing Methodology | ✅ Pass | 13.391s | |
+| Vulnerability Assessment Tools | ✅ Pass | 19.520s | |
+| SQL Injection Attack Type | ✅ Pass | 14.260s | |
+| Penetration Testing Framework | ✅ Pass | 7.151s | |
+| Web Application Security Scanner | ✅ Pass | 12.338s | |
+| Penetration Testing Tool Selection | ✅ Pass | 4.118s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 6.794s
+**Average latency**: 7.081s
---
-### refiner (glm-5)
+### refiner (glm-5.1)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 3.857s | |
-| Text Transform Uppercase | ✅ Pass | 3.328s | |
-| Count from 1 to 5 | ✅ Pass | 4.175s | |
-| Math Calculation | ✅ Pass | 1.979s | |
-| Basic Echo Function | ✅ Pass | 3.519s | |
-| Streaming Simple Math Streaming | ✅ Pass | 4.409s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.773s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 3.607s | |
+| Simple Math | ✅ Pass | 7.346s | |
+| Text Transform Uppercase | ✅ Pass | 7.388s | |
+| Count from 1 to 5 | ✅ Pass | 7.160s | |
+| Math Calculation | ✅ Pass | 4.476s | |
+| Basic Echo Function | ✅ Pass | 7.189s | |
+| Streaming Simple Math Streaming | ✅ Pass | 9.466s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.994s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 3.243s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 3.429s | |
-| Search Query Function | ✅ Pass | 2.801s | |
-| Ask Advice Function | ✅ Pass | 3.807s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 3.450s | |
-| Basic Context Memory Test | ✅ Pass | 2.899s | |
-| Function Argument Memory Test | ✅ Pass | 2.724s | |
-| Function Response Memory Test | ✅ Pass | 7.180s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 10.559s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 3.838s | |
-| Penetration Testing Methodology | ✅ Pass | 17.640s | |
-| Vulnerability Assessment Tools | ✅ Pass | 35.316s | |
-| SQL Injection Attack Type | ✅ Pass | 4.652s | |
-| Penetration Testing Framework | ✅ Pass | 19.140s | |
-| Web Application Security Scanner | ✅ Pass | 15.485s | |
-| Penetration Testing Tool Selection | ✅ Pass | 4.825s | |
+| JSON Response Function | ✅ Pass | 6.498s | |
+| Search Query Function | ✅ Pass | 5.493s | |
+| Ask Advice Function | ✅ Pass | 4.021s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 3.686s | |
+| Basic Context Memory Test | ✅ Pass | 5.153s | |
+| Function Argument Memory Test | ✅ Pass | 5.468s | |
+| Function Response Memory Test | ✅ Pass | 3.245s | |
+| Penetration Testing Memory with Tool Call | ❌ Fail | 7.345s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 4.404s | |
+| Penetration Testing Methodology | ✅ Pass | 12.575s | |
+| Vulnerability Assessment Tools | ✅ Pass | 16.644s | |
+| SQL Injection Attack Type | ✅ Pass | 5.347s | |
+| Penetration Testing Framework | ✅ Pass | 11.479s | |
+| Web Application Security Scanner | ✅ Pass | 12.258s | |
+| Penetration Testing Tool Selection | ✅ Pass | 3.992s | |
-**Summary**: 23/23 (100.00%) successful tests
+**Summary**: 22/23 (95.65%) successful tests
-**Average latency**: 7.235s
+**Average latency**: 6.864s
---
-### adviser (glm-5)
+### adviser (glm-5.1)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 3.839s | |
-| Text Transform Uppercase | ✅ Pass | 5.472s | |
-| Count from 1 to 5 | ✅ Pass | 4.924s | |
-| Math Calculation | ✅ Pass | 3.169s | |
-| Basic Echo Function | ✅ Pass | 3.077s | |
-| Streaming Simple Math Streaming | ✅ Pass | 3.900s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.411s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 3.469s | |
+| Simple Math | ✅ Pass | 7.436s | |
+| Text Transform Uppercase | ✅ Pass | 4.739s | |
+| Count from 1 to 5 | ✅ Pass | 7.323s | |
+| Math Calculation | ✅ Pass | 3.884s | |
+| Basic Echo Function | ✅ Pass | 4.145s | |
+| Streaming Simple Math Streaming | ✅ Pass | 4.102s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.763s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 5.518s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 4.821s | |
-| Search Query Function | ✅ Pass | 3.395s | |
-| Ask Advice Function | ✅ Pass | 6.539s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 6.834s | |
-| Basic Context Memory Test | ✅ Pass | 1.888s | |
-| Function Argument Memory Test | ✅ Pass | 2.962s | |
-| Function Response Memory Test | ✅ Pass | 4.197s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 6.934s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 2.397s | |
-| Penetration Testing Methodology | ✅ Pass | 18.101s | |
-| Vulnerability Assessment Tools | ✅ Pass | 46.457s | |
-| SQL Injection Attack Type | ✅ Pass | 9.365s | |
-| Penetration Testing Framework | ✅ Pass | 17.170s | |
-| Web Application Security Scanner | ✅ Pass | 16.017s | |
-| Penetration Testing Tool Selection | ✅ Pass | 3.804s | |
+| JSON Response Function | ✅ Pass | 3.657s | |
+| Search Query Function | ✅ Pass | 3.205s | |
+| Ask Advice Function | ✅ Pass | 3.559s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 3.959s | |
+| Basic Context Memory Test | ✅ Pass | 6.817s | |
+| Function Argument Memory Test | ✅ Pass | 6.147s | |
+| Function Response Memory Test | ✅ Pass | 3.902s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 5.443s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 5.905s | |
+| Penetration Testing Methodology | ✅ Pass | 14.182s | |
+| Vulnerability Assessment Tools | ✅ Pass | 19.240s | |
+| SQL Injection Attack Type | ✅ Pass | 5.849s | |
+| Penetration Testing Framework | ✅ Pass | 11.161s | |
+| Web Application Security Scanner | ✅ Pass | 11.757s | |
+| Penetration Testing Tool Selection | ✅ Pass | 4.019s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 7.876s
+**Average latency**: 6.553s
---
@@ -295,38 +295,38 @@ Generated: Thu, 05 Mar 2026 16:50:23 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 6.143s | |
-| Text Transform Uppercase | ✅ Pass | 1.870s | |
-| Count from 1 to 5 | ✅ Pass | 3.652s | |
-| Math Calculation | ✅ Pass | 1.387s | |
-| Basic Echo Function | ✅ Pass | 2.077s | |
-| Streaming Simple Math Streaming | ✅ Pass | 6.175s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.212s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 1.591s | |
+| Simple Math | ✅ Pass | 1.349s | |
+| Text Transform Uppercase | ✅ Pass | 0.952s | |
+| Count from 1 to 5 | ✅ Pass | 3.336s | |
+| Math Calculation | ✅ Pass | 0.828s | |
+| Basic Echo Function | ✅ Pass | 1.318s | |
+| Streaming Simple Math Streaming | ✅ Pass | 8.738s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.624s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.163s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 2.164s | |
-| Search Query Function | ✅ Pass | 2.576s | |
-| Ask Advice Function | ✅ Pass | 2.395s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 2.058s | |
-| Basic Context Memory Test | ✅ Pass | 2.424s | |
-| Function Argument Memory Test | ✅ Pass | 1.993s | |
-| Function Response Memory Test | ✅ Pass | 2.025s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 4.138s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 2.171s | |
-| Penetration Testing Methodology | ✅ Pass | 24.940s | |
-| Vulnerability Assessment Tools | ✅ Pass | 35.170s | |
-| SQL Injection Attack Type | ✅ Pass | 4.671s | |
-| Penetration Testing Framework | ✅ Pass | 22.360s | |
-| Web Application Security Scanner | ✅ Pass | 9.550s | |
-| Penetration Testing Tool Selection | ✅ Pass | 2.231s | |
+| JSON Response Function | ✅ Pass | 1.864s | |
+| Search Query Function | ✅ Pass | 1.759s | |
+| Ask Advice Function | ✅ Pass | 1.985s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.501s | |
+| Basic Context Memory Test | ✅ Pass | 1.776s | |
+| Function Argument Memory Test | ✅ Pass | 1.869s | |
+| Function Response Memory Test | ✅ Pass | 1.430s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.434s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 1.338s | |
+| Penetration Testing Methodology | ✅ Pass | 6.325s | |
+| Vulnerability Assessment Tools | ✅ Pass | 5.048s | |
+| SQL Injection Attack Type | ✅ Pass | 2.589s | |
+| Penetration Testing Framework | ✅ Pass | 7.158s | |
+| Web Application Security Scanner | ✅ Pass | 5.490s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.597s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 6.347s
+**Average latency**: 2.760s
---
@@ -336,38 +336,38 @@ Generated: Thu, 05 Mar 2026 16:50:23 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 1.419s | |
-| Text Transform Uppercase | ✅ Pass | 1.759s | |
-| Count from 1 to 5 | ✅ Pass | 3.229s | |
-| Math Calculation | ✅ Pass | 1.062s | |
-| Basic Echo Function | ✅ Pass | 1.886s | |
-| Streaming Simple Math Streaming | ✅ Pass | 6.286s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.933s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 1.588s | |
+| Simple Math | ✅ Pass | 1.144s | |
+| Text Transform Uppercase | ✅ Pass | 1.230s | |
+| Count from 1 to 5 | ✅ Pass | 2.526s | |
+| Math Calculation | ✅ Pass | 0.975s | |
+| Basic Echo Function | ✅ Pass | 1.962s | |
+| Streaming Simple Math Streaming | ✅ Pass | 6.438s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.872s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.207s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 3.103s | |
-| Search Query Function | ✅ Pass | 2.430s | |
-| Ask Advice Function | ✅ Pass | 2.791s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 2.399s | |
-| Basic Context Memory Test | ✅ Pass | 1.652s | |
-| Function Argument Memory Test | ✅ Pass | 1.487s | |
-| Function Response Memory Test | ✅ Pass | 2.492s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 6.473s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.894s | |
-| Penetration Testing Methodology | ✅ Pass | 11.508s | |
-| Vulnerability Assessment Tools | ✅ Pass | 26.143s | |
-| SQL Injection Attack Type | ✅ Pass | 5.851s | |
-| Penetration Testing Framework | ✅ Pass | 16.044s | |
-| Web Application Security Scanner | ✅ Pass | 20.166s | |
-| Penetration Testing Tool Selection | ✅ Pass | 2.711s | |
+| JSON Response Function | ✅ Pass | 1.837s | |
+| Search Query Function | ✅ Pass | 2.366s | |
+| Ask Advice Function | ✅ Pass | 1.564s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 2.558s | |
+| Basic Context Memory Test | ✅ Pass | 1.415s | |
+| Function Argument Memory Test | ✅ Pass | 1.913s | |
+| Function Response Memory Test | ✅ Pass | 1.785s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.208s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 1.507s | |
+| Penetration Testing Methodology | ✅ Pass | 5.352s | |
+| Vulnerability Assessment Tools | ✅ Pass | 14.510s | |
+| SQL Injection Attack Type | ✅ Pass | 2.434s | |
+| Penetration Testing Framework | ✅ Pass | 5.262s | |
+| Web Application Security Scanner | ✅ Pass | 2.488s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.891s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 5.492s
+**Average latency**: 2.889s
---
@@ -377,161 +377,161 @@ Generated: Thu, 05 Mar 2026 16:50:23 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 3.025s | |
-| Text Transform Uppercase | ✅ Pass | 2.244s | |
-| Count from 1 to 5 | ✅ Pass | 2.697s | |
-| Math Calculation | ✅ Pass | 1.304s | |
-| Basic Echo Function | ✅ Pass | 1.865s | |
-| Streaming Simple Math Streaming | ✅ Pass | 4.939s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.881s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 1.618s | |
+| Simple Math | ✅ Pass | 1.166s | |
+| Text Transform Uppercase | ✅ Pass | 1.054s | |
+| Count from 1 to 5 | ✅ Pass | 1.600s | |
+| Math Calculation | ✅ Pass | 1.054s | |
+| Basic Echo Function | ✅ Pass | 1.737s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.306s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.486s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.403s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 2.218s | |
-| Search Query Function | ✅ Pass | 2.579s | |
-| Ask Advice Function | ✅ Pass | 2.049s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.947s | |
-| Basic Context Memory Test | ✅ Pass | 1.765s | |
-| Function Argument Memory Test | ✅ Pass | 1.733s | |
-| Function Response Memory Test | ✅ Pass | 1.646s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 4.578s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 4.063s | |
-| Penetration Testing Methodology | ✅ Pass | 15.977s | |
-| Vulnerability Assessment Tools | ✅ Pass | 46.410s | |
-| SQL Injection Attack Type | ✅ Pass | 10.036s | |
-| Penetration Testing Framework | ✅ Pass | 20.568s | |
-| Web Application Security Scanner | ✅ Pass | 10.759s | |
-| Penetration Testing Tool Selection | ✅ Pass | 3.314s | |
+| JSON Response Function | ✅ Pass | 1.452s | |
+| Search Query Function | ✅ Pass | 1.895s | |
+| Ask Advice Function | ✅ Pass | 1.631s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.771s | |
+| Basic Context Memory Test | ✅ Pass | 1.640s | |
+| Function Argument Memory Test | ✅ Pass | 2.034s | |
+| Function Response Memory Test | ✅ Pass | 1.623s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.927s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 2.506s | |
+| Penetration Testing Methodology | ✅ Pass | 8.173s | |
+| Vulnerability Assessment Tools | ✅ Pass | 12.181s | |
+| SQL Injection Attack Type | ✅ Pass | 1.488s | |
+| Penetration Testing Framework | ✅ Pass | 14.886s | |
+| Web Application Security Scanner | ✅ Pass | 2.242s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.759s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 6.488s
+**Average latency**: 3.001s
---
-### coder (glm-5)
+### coder (glm-5.1)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 3.028s | |
-| Text Transform Uppercase | ✅ Pass | 2.695s | |
-| Count from 1 to 5 | ✅ Pass | 4.099s | |
-| Math Calculation | ✅ Pass | 2.054s | |
-| Basic Echo Function | ✅ Pass | 4.083s | |
-| Streaming Simple Math Streaming | ✅ Pass | 2.808s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.021s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 3.917s | |
+| Simple Math | ✅ Pass | 4.213s | |
+| Text Transform Uppercase | ✅ Pass | 4.692s | |
+| Count from 1 to 5 | ✅ Pass | 5.795s | |
+| Math Calculation | ✅ Pass | 9.208s | |
+| Basic Echo Function | ✅ Pass | 5.473s | |
+| Streaming Simple Math Streaming | ✅ Pass | 3.389s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 8.238s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 3.129s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 3.616s | |
-| Search Query Function | ✅ Pass | 4.091s | |
-| Ask Advice Function | ✅ Pass | 4.418s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 4.970s | |
-| Basic Context Memory Test | ✅ Pass | 2.142s | |
-| Function Argument Memory Test | ✅ Pass | 2.669s | |
-| Function Response Memory Test | ✅ Pass | 4.727s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 8.417s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 3.213s | |
-| Penetration Testing Methodology | ✅ Pass | 13.789s | |
-| Vulnerability Assessment Tools | ✅ Pass | 17.248s | |
-| SQL Injection Attack Type | ✅ Pass | 7.931s | |
-| Penetration Testing Framework | ✅ Pass | 18.277s | |
-| Web Application Security Scanner | ✅ Pass | 15.769s | |
-| Penetration Testing Tool Selection | ✅ Pass | 4.949s | |
+| JSON Response Function | ✅ Pass | 3.931s | |
+| Search Query Function | ✅ Pass | 3.264s | |
+| Ask Advice Function | ✅ Pass | 7.299s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 3.841s | |
+| Basic Context Memory Test | ✅ Pass | 5.589s | |
+| Function Argument Memory Test | ✅ Pass | 6.503s | |
+| Function Response Memory Test | ✅ Pass | 3.301s | |
+| Penetration Testing Memory with Tool Call | ❌ Fail | 6.407s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 6.279s | |
+| Penetration Testing Methodology | ✅ Pass | 17.570s | |
+| Vulnerability Assessment Tools | ✅ Pass | 19.116s | |
+| SQL Injection Attack Type | ✅ Pass | 5.596s | |
+| Penetration Testing Framework | ✅ Pass | 16.889s | |
+| Web Application Security Scanner | ✅ Pass | 11.733s | |
+| Penetration Testing Tool Selection | ✅ Pass | 3.497s | |
-**Summary**: 23/23 (100.00%) successful tests
+**Summary**: 22/23 (95.65%) successful tests
-**Average latency**: 6.128s
+**Average latency**: 7.172s
---
-### installer (glm-4.7)
+### installer (glm-4.5-air)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 1.639s | |
-| Text Transform Uppercase | ✅ Pass | 2.362s | |
-| Count from 1 to 5 | ✅ Pass | 2.142s | |
-| Math Calculation | ✅ Pass | 1.261s | |
-| Basic Echo Function | ✅ Pass | 2.065s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.972s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.666s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 24.519s | |
+| Simple Math | ✅ Pass | 6.253s | |
+| Text Transform Uppercase | ✅ Pass | 2.297s | |
+| Count from 1 to 5 | ✅ Pass | 3.713s | |
+| Math Calculation | ✅ Pass | 2.951s | |
+| Basic Echo Function | ✅ Pass | 2.057s | |
+| Streaming Simple Math Streaming | ✅ Pass | 3.884s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.290s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 3.606s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 1.922s | |
-| Search Query Function | ✅ Pass | 1.170s | |
-| Ask Advice Function | ✅ Pass | 1.321s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.223s | |
-| Basic Context Memory Test | ✅ Pass | 2.865s | |
-| Function Argument Memory Test | ✅ Pass | 6.698s | |
-| Function Response Memory Test | ✅ Pass | 1.635s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 2.691s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 2.232s | |
-| Penetration Testing Methodology | ✅ Pass | 4.972s | |
-| Vulnerability Assessment Tools | ✅ Pass | 3.719s | |
-| SQL Injection Attack Type | ✅ Pass | 3.134s | |
-| Penetration Testing Framework | ✅ Pass | 7.910s | |
-| Web Application Security Scanner | ✅ Pass | 8.168s | |
-| Penetration Testing Tool Selection | ✅ Pass | 1.464s | |
+| JSON Response Function | ✅ Pass | 4.104s | |
+| Search Query Function | ✅ Pass | 3.696s | |
+| Ask Advice Function | ✅ Pass | 3.538s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 3.188s | |
+| Basic Context Memory Test | ✅ Pass | 3.317s | |
+| Function Argument Memory Test | ✅ Pass | 1.915s | |
+| Function Response Memory Test | ✅ Pass | 3.620s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 5.784s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.012s | |
+| Penetration Testing Methodology | ✅ Pass | 14.559s | |
+| Vulnerability Assessment Tools | ✅ Pass | 16.449s | |
+| SQL Injection Attack Type | ✅ Pass | 8.793s | |
+| Penetration Testing Framework | ✅ Pass | 22.365s | |
+| Web Application Security Scanner | ✅ Pass | 15.217s | |
+| Penetration Testing Tool Selection | ✅ Pass | 3.740s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 3.903s
+**Average latency**: 6.146s
---
-### pentester (glm-4.7)
+### pentester (glm-5.1)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 1.664s | |
-| Text Transform Uppercase | ✅ Pass | 5.869s | |
-| Count from 1 to 5 | ✅ Pass | 2.157s | |
-| Math Calculation | ✅ Pass | 2.004s | |
-| Basic Echo Function | ✅ Pass | 1.038s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.588s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.166s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 20.024s | |
+| Simple Math | ✅ Pass | 5.261s | |
+| Text Transform Uppercase | ✅ Pass | 6.631s | |
+| Count from 1 to 5 | ✅ Pass | 4.222s | |
+| Math Calculation | ✅ Pass | 3.124s | |
+| Basic Echo Function | ✅ Pass | 3.221s | |
+| Streaming Simple Math Streaming | ✅ Pass | 6.244s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.420s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 3.163s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 1.388s | |
-| Search Query Function | ✅ Pass | 1.291s | |
-| Ask Advice Function | ✅ Pass | 1.762s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.613s | |
-| Basic Context Memory Test | ✅ Pass | 2.043s | |
-| Function Argument Memory Test | ✅ Pass | 1.674s | |
-| Function Response Memory Test | ✅ Pass | 2.169s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 2.299s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 2.199s | |
-| Penetration Testing Methodology | ✅ Pass | 5.637s | |
-| Vulnerability Assessment Tools | ❌ Fail | 36.737s | expected text 'network' not found |
-| SQL Injection Attack Type | ✅ Pass | 4.446s | |
-| Penetration Testing Framework | ✅ Pass | 6.258s | |
-| Web Application Security Scanner | ✅ Pass | 15.067s | |
-| Penetration Testing Tool Selection | ✅ Pass | 1.940s | |
+| JSON Response Function | ✅ Pass | 4.696s | |
+| Search Query Function | ✅ Pass | 2.983s | |
+| Ask Advice Function | ✅ Pass | 5.813s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 9.633s | |
+| Basic Context Memory Test | ✅ Pass | 4.770s | |
+| Function Argument Memory Test | ✅ Pass | 7.549s | |
+| Function Response Memory Test | ✅ Pass | 3.946s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 4.723s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.120s | |
+| Penetration Testing Methodology | ✅ Pass | 17.474s | |
+| Vulnerability Assessment Tools | ✅ Pass | 21.649s | |
+| SQL Injection Attack Type | ✅ Pass | 6.494s | |
+| Penetration Testing Framework | ✅ Pass | 12.428s | |
+| Web Application Security Scanner | ✅ Pass | 8.468s | |
+| Penetration Testing Tool Selection | ✅ Pass | 4.718s | |
-**Summary**: 22/23 (95.65%) successful tests
+**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 5.350s
+**Average latency**: 6.772s
---
diff --git a/examples/tests/kimi-report.md b/examples/tests/kimi-report.md
index 0c74db3c..b6effc43 100644
--- a/examples/tests/kimi-report.md
+++ b/examples/tests/kimi-report.md
@@ -1,86 +1,86 @@
# LLM Agent Testing Report
-Generated: Wed, 04 Mar 2026 22:36:05 UTC
+Generated: Wed, 27 May 2026 23:34:01 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
-| simple | kimi-k2-turbo-preview | false | 23/23 (100.00%) | 1.029s |
-| simple_json | kimi-k2-turbo-preview | false | 5/5 (100.00%) | 1.090s |
-| primary_agent | kimi-k2.5 | true | 23/23 (100.00%) | 4.379s |
-| assistant | kimi-k2.5 | true | 23/23 (100.00%) | 4.599s |
-| generator | kimi-k2.5 | true | 23/23 (100.00%) | 4.054s |
-| refiner | kimi-k2.5 | true | 23/23 (100.00%) | 4.773s |
-| adviser | kimi-k2.5 | true | 23/23 (100.00%) | 4.786s |
-| reflector | kimi-k2-0905-preview | true | 23/23 (100.00%) | 2.573s |
-| searcher | kimi-k2-0905-preview | true | 22/23 (95.65%) | 2.907s |
-| enricher | kimi-k2-0905-preview | true | 23/23 (100.00%) | 2.275s |
-| coder | kimi-k2.5 | true | 23/23 (100.00%) | 4.206s |
-| installer | kimi-k2-turbo-preview | true | 23/23 (100.00%) | 0.918s |
-| pentester | kimi-k2-turbo-preview | true | 23/23 (100.00%) | 0.901s |
-
-**Total**: 280/281 (99.64%) successful tests
-**Overall average latency**: 3.081s
+| simple | kimi-k2.5 | true | 23/23 (100.00%) | 1.452s |
+| simple_json | kimi-k2.5 | false | 5/5 (100.00%) | 1.241s |
+| primary_agent | kimi-k2.5 | true | 23/23 (100.00%) | 3.602s |
+| assistant | kimi-k2.5 | true | 23/23 (100.00%) | 3.660s |
+| generator | kimi-k2.6 | true | 23/23 (100.00%) | 3.471s |
+| refiner | kimi-k2.6 | true | 23/23 (100.00%) | 3.700s |
+| adviser | kimi-k2.6 | true | 23/23 (100.00%) | 3.886s |
+| reflector | kimi-k2.5 | true | 23/23 (100.00%) | 1.526s |
+| searcher | kimi-k2.5 | true | 23/23 (100.00%) | 1.383s |
+| enricher | kimi-k2.5 | true | 23/23 (100.00%) | 1.271s |
+| coder | kimi-k2.6 | true | 23/23 (100.00%) | 3.321s |
+| installer | kimi-k2.5 | true | 23/23 (100.00%) | 3.381s |
+| pentester | kimi-k2.6 | true | 23/23 (100.00%) | 3.432s |
+
+**Total**: 281/281 (100.00%) successful tests
+**Overall average latency**: 2.812s
## Detailed Results
-### simple (kimi-k2-turbo-preview)
+### simple (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 3.911s | |
-| Text Transform Uppercase | ✅ Pass | 0.743s | |
-| Count from 1 to 5 | ✅ Pass | 0.800s | |
-| Math Calculation | ✅ Pass | 0.691s | |
-| Basic Echo Function | ✅ Pass | 0.943s | |
-| Streaming Simple Math Streaming | ✅ Pass | 0.763s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.610s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 1.095s | |
+| Simple Math | ✅ Pass | 2.967s | |
+| Text Transform Uppercase | ✅ Pass | 0.754s | |
+| Count from 1 to 5 | ✅ Pass | 0.898s | |
+| Math Calculation | ✅ Pass | 0.592s | |
+| Basic Echo Function | ✅ Pass | 1.073s | |
+| Streaming Simple Math Streaming | ✅ Pass | 0.716s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.857s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.202s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 1.258s | |
-| Search Query Function | ✅ Pass | 0.699s | |
-| Ask Advice Function | ✅ Pass | 0.925s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 0.914s | |
-| Basic Context Memory Test | ✅ Pass | 0.908s | |
-| Function Argument Memory Test | ✅ Pass | 0.770s | |
-| Function Response Memory Test | ✅ Pass | 0.750s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 1.579s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.014s | |
-| Penetration Testing Methodology | ✅ Pass | 0.932s | |
-| Vulnerability Assessment Tools | ✅ Pass | 0.830s | |
-| SQL Injection Attack Type | ✅ Pass | 0.853s | |
-| Penetration Testing Framework | ✅ Pass | 1.045s | |
-| Web Application Security Scanner | ✅ Pass | 0.615s | |
-| Penetration Testing Tool Selection | ✅ Pass | 1.001s | |
+| JSON Response Function | ✅ Pass | 1.030s | |
+| Search Query Function | ✅ Pass | 0.991s | |
+| Ask Advice Function | ✅ Pass | 1.341s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.123s | |
+| Basic Context Memory Test | ✅ Pass | 0.844s | |
+| Function Argument Memory Test | ✅ Pass | 0.811s | |
+| Function Response Memory Test | ✅ Pass | 0.827s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.086s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 0.772s | |
+| Penetration Testing Methodology | ✅ Pass | 2.927s | |
+| Vulnerability Assessment Tools | ✅ Pass | 2.957s | |
+| SQL Injection Attack Type | ✅ Pass | 0.878s | |
+| Penetration Testing Framework | ✅ Pass | 3.484s | |
+| Web Application Security Scanner | ✅ Pass | 3.183s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.066s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 1.029s
+**Average latency**: 1.452s
---
-### simple_json (kimi-k2-turbo-preview)
+### simple_json (kimi-k2.5)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Person Information JSON | ✅ Pass | 0.962s | |
-| Project Information JSON | ✅ Pass | 0.950s | |
-| Streaming Person Information JSON Streaming | ✅ Pass | 0.834s | |
-| User Profile JSON | ✅ Pass | 1.014s | |
-| Vulnerability Report Memory Test | ✅ Pass | 1.687s | |
+| Project Information JSON | ✅ Pass | 1.047s | |
+| Person Information JSON | ✅ Pass | 1.194s | |
+| Vulnerability Report Memory Test | ✅ Pass | 2.029s | |
+| User Profile JSON | ✅ Pass | 0.951s | |
+| Streaming Person Information JSON Streaming | ✅ Pass | 0.984s | |
**Summary**: 5/5 (100.00%) successful tests
-**Average latency**: 1.090s
+**Average latency**: 1.241s
---
@@ -90,38 +90,38 @@ Generated: Wed, 04 Mar 2026 22:36:05 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 4.913s | |
-| Text Transform Uppercase | ✅ Pass | 2.090s | |
-| Count from 1 to 5 | ✅ Pass | 4.081s | |
-| Math Calculation | ✅ Pass | 1.913s | |
-| Basic Echo Function | ✅ Pass | 1.683s | |
-| Streaming Simple Math Streaming | ✅ Pass | 2.689s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.210s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 2.376s | |
+| Simple Math | ✅ Pass | 3.805s | |
+| Text Transform Uppercase | ✅ Pass | 2.413s | |
+| Count from 1 to 5 | ✅ Pass | 2.075s | |
+| Math Calculation | ✅ Pass | 1.364s | |
+| Basic Echo Function | ✅ Pass | 1.724s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.629s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.851s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.895s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 1.979s | |
-| Search Query Function | ✅ Pass | 1.847s | |
-| Ask Advice Function | ✅ Pass | 3.195s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.966s | |
-| Basic Context Memory Test | ✅ Pass | 2.056s | |
-| Function Argument Memory Test | ✅ Pass | 3.404s | |
-| Function Response Memory Test | ✅ Pass | 2.744s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 5.534s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 2.884s | |
-| Penetration Testing Methodology | ✅ Pass | 13.876s | |
-| Vulnerability Assessment Tools | ✅ Pass | 9.282s | |
-| SQL Injection Attack Type | ✅ Pass | 3.591s | |
-| Penetration Testing Framework | ✅ Pass | 13.475s | |
-| Web Application Security Scanner | ✅ Pass | 10.430s | |
-| Penetration Testing Tool Selection | ✅ Pass | 2.495s | |
+| JSON Response Function | ✅ Pass | 2.622s | |
+| Search Query Function | ✅ Pass | 1.869s | |
+| Ask Advice Function | ✅ Pass | 2.412s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.302s | |
+| Basic Context Memory Test | ✅ Pass | 2.121s | |
+| Function Argument Memory Test | ✅ Pass | 2.266s | |
+| Function Response Memory Test | ✅ Pass | 1.705s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 4.334s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 2.341s | |
+| Penetration Testing Methodology | ✅ Pass | 11.752s | |
+| Vulnerability Assessment Tools | ✅ Pass | 7.759s | |
+| SQL Injection Attack Type | ✅ Pass | 2.492s | |
+| Penetration Testing Framework | ✅ Pass | 12.965s | |
+| Web Application Security Scanner | ✅ Pass | 8.206s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.925s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 4.379s
+**Average latency**: 3.602s
---
@@ -131,407 +131,407 @@ Generated: Wed, 04 Mar 2026 22:36:05 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 5.791s | |
-| Text Transform Uppercase | ✅ Pass | 1.895s | |
-| Count from 1 to 5 | ✅ Pass | 2.390s | |
-| Math Calculation | ✅ Pass | 1.880s | |
-| Basic Echo Function | ✅ Pass | 2.028s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.733s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.070s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 2.019s | |
+| Simple Math | ✅ Pass | 3.792s | |
+| Text Transform Uppercase | ✅ Pass | 1.857s | |
+| Count from 1 to 5 | ✅ Pass | 3.005s | |
+| Math Calculation | ✅ Pass | 1.790s | |
+| Basic Echo Function | ✅ Pass | 1.660s | |
+| Streaming Simple Math Streaming | ✅ Pass | 2.064s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.648s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.884s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 2.252s | |
-| Search Query Function | ✅ Pass | 1.543s | |
-| Ask Advice Function | ✅ Pass | 2.579s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.891s | |
-| Basic Context Memory Test | ✅ Pass | 3.971s | |
-| Function Argument Memory Test | ✅ Pass | 3.501s | |
-| Function Response Memory Test | ✅ Pass | 2.208s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 4.784s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 4.306s | |
-| Vulnerability Assessment Tools | ✅ Pass | 10.359s | |
-| Penetration Testing Methodology | ✅ Pass | 17.251s | |
-| SQL Injection Attack Type | ✅ Pass | 2.463s | |
-| Penetration Testing Framework | ✅ Pass | 11.586s | |
-| Web Application Security Scanner | ✅ Pass | 13.252s | |
-| Penetration Testing Tool Selection | ✅ Pass | 2.006s | |
+| JSON Response Function | ✅ Pass | 2.934s | |
+| Search Query Function | ✅ Pass | 1.837s | |
+| Ask Advice Function | ✅ Pass | 2.343s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.786s | |
+| Basic Context Memory Test | ✅ Pass | 2.777s | |
+| Function Argument Memory Test | ✅ Pass | 2.055s | |
+| Function Response Memory Test | ✅ Pass | 1.940s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 3.760s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 2.796s | |
+| Penetration Testing Methodology | ✅ Pass | 10.671s | |
+| Vulnerability Assessment Tools | ✅ Pass | 10.371s | |
+| SQL Injection Attack Type | ✅ Pass | 2.529s | |
+| Penetration Testing Framework | ✅ Pass | 11.063s | |
+| Web Application Security Scanner | ✅ Pass | 6.830s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.769s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 4.599s
+**Average latency**: 3.660s
---
-### generator (kimi-k2.5)
+### generator (kimi-k2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 6.203s | |
-| Text Transform Uppercase | ✅ Pass | 1.889s | |
-| Count from 1 to 5 | ✅ Pass | 2.280s | |
-| Math Calculation | ✅ Pass | 1.852s | |
-| Basic Echo Function | ✅ Pass | 1.704s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.734s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.883s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 1.803s | |
+| Simple Math | ✅ Pass | 3.818s | |
+| Text Transform Uppercase | ✅ Pass | 4.989s | |
+| Count from 1 to 5 | ✅ Pass | 2.068s | |
+| Math Calculation | ✅ Pass | 1.499s | |
+| Basic Echo Function | ✅ Pass | 1.577s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.753s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.802s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.221s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 1.665s | |
-| Search Query Function | ✅ Pass | 1.833s | |
-| Ask Advice Function | ✅ Pass | 1.824s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.725s | |
-| Basic Context Memory Test | ✅ Pass | 2.329s | |
-| Function Argument Memory Test | ✅ Pass | 2.637s | |
-| Function Response Memory Test | ✅ Pass | 2.065s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 3.415s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 3.119s | |
-| Penetration Testing Methodology | ✅ Pass | 12.633s | |
-| SQL Injection Attack Type | ✅ Pass | 3.027s | |
-| Vulnerability Assessment Tools | ✅ Pass | 13.788s | |
-| Penetration Testing Framework | ✅ Pass | 11.570s | |
-| Web Application Security Scanner | ✅ Pass | 8.272s | |
-| Penetration Testing Tool Selection | ✅ Pass | 2.982s | |
+| JSON Response Function | ✅ Pass | 1.911s | |
+| Search Query Function | ✅ Pass | 1.693s | |
+| Ask Advice Function | ✅ Pass | 2.063s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.638s | |
+| Basic Context Memory Test | ✅ Pass | 2.781s | |
+| Function Argument Memory Test | ✅ Pass | 1.929s | |
+| Function Response Memory Test | ✅ Pass | 1.722s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 2.330s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 6.770s | |
+| Penetration Testing Methodology | ✅ Pass | 5.593s | |
+| Vulnerability Assessment Tools | ✅ Pass | 9.689s | |
+| SQL Injection Attack Type | ✅ Pass | 3.238s | |
+| Penetration Testing Framework | ✅ Pass | 6.540s | |
+| Web Application Security Scanner | ✅ Pass | 4.246s | |
+| Penetration Testing Tool Selection | ✅ Pass | 3.942s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 4.054s
+**Average latency**: 3.471s
---
-### refiner (kimi-k2.5)
+### refiner (kimi-k2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 5.172s | |
-| Text Transform Uppercase | ✅ Pass | 3.542s | |
-| Count from 1 to 5 | ✅ Pass | 3.905s | |
-| Math Calculation | ✅ Pass | 2.205s | |
-| Basic Echo Function | ✅ Pass | 1.896s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.793s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.588s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 2.001s | |
+| Simple Math | ✅ Pass | 3.930s | |
+| Text Transform Uppercase | ✅ Pass | 1.828s | |
+| Count from 1 to 5 | ✅ Pass | 2.079s | |
+| Math Calculation | ✅ Pass | 2.422s | |
+| Basic Echo Function | ✅ Pass | 1.638s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.739s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.457s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.056s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 2.247s | |
-| Search Query Function | ✅ Pass | 1.763s | |
-| Ask Advice Function | ✅ Pass | 2.343s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.959s | |
-| Basic Context Memory Test | ✅ Pass | 2.718s | |
-| Function Argument Memory Test | ✅ Pass | 2.372s | |
-| Function Response Memory Test | ✅ Pass | 2.732s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 5.729s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 2.771s | |
-| Penetration Testing Methodology | ✅ Pass | 14.859s | |
-| Vulnerability Assessment Tools | ✅ Pass | 11.455s | |
-| SQL Injection Attack Type | ✅ Pass | 7.561s | |
-| Penetration Testing Framework | ✅ Pass | 11.828s | |
-| Web Application Security Scanner | ✅ Pass | 10.862s | |
-| Penetration Testing Tool Selection | ✅ Pass | 3.473s | |
+| JSON Response Function | ✅ Pass | 2.574s | |
+| Search Query Function | ✅ Pass | 3.115s | |
+| Ask Advice Function | ✅ Pass | 1.961s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.807s | |
+| Basic Context Memory Test | ✅ Pass | 1.837s | |
+| Function Argument Memory Test | ✅ Pass | 2.727s | |
+| Function Response Memory Test | ✅ Pass | 2.079s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 3.531s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.579s | |
+| Penetration Testing Methodology | ✅ Pass | 12.756s | |
+| Vulnerability Assessment Tools | ✅ Pass | 10.127s | |
+| SQL Injection Attack Type | ✅ Pass | 4.953s | |
+| Penetration Testing Framework | ✅ Pass | 8.007s | |
+| Web Application Security Scanner | ✅ Pass | 4.581s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.309s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 4.773s
+**Average latency**: 3.700s
---
-### adviser (kimi-k2.5)
+### adviser (kimi-k2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 5.235s | |
-| Text Transform Uppercase | ✅ Pass | 2.341s | |
-| Count from 1 to 5 | ✅ Pass | 3.830s | |
-| Math Calculation | ✅ Pass | 2.093s | |
-| Basic Echo Function | ✅ Pass | 1.943s | |
-| Streaming Simple Math Streaming | ✅ Pass | 2.038s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.933s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 2.271s | |
+| Simple Math | ✅ Pass | 4.341s | |
+| Text Transform Uppercase | ✅ Pass | 1.662s | |
+| Count from 1 to 5 | ✅ Pass | 2.051s | |
+| Basic Echo Function | ✅ Pass | 1.700s | |
+| Math Calculation | ✅ Pass | 4.090s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.907s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.154s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.487s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 2.077s | |
-| Search Query Function | ✅ Pass | 1.605s | |
-| Ask Advice Function | ✅ Pass | 2.604s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.891s | |
-| Basic Context Memory Test | ✅ Pass | 3.162s | |
-| Function Argument Memory Test | ✅ Pass | 2.505s | |
-| Function Response Memory Test | ✅ Pass | 2.594s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 3.564s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 2.689s | |
-| Penetration Testing Methodology | ✅ Pass | 18.499s | |
-| SQL Injection Attack Type | ✅ Pass | 3.531s | |
-| Vulnerability Assessment Tools | ✅ Pass | 12.883s | |
-| Penetration Testing Framework | ✅ Pass | 16.455s | |
-| Web Application Security Scanner | ✅ Pass | 9.418s | |
-| Penetration Testing Tool Selection | ✅ Pass | 2.912s | |
+| JSON Response Function | ✅ Pass | 1.825s | |
+| Search Query Function | ✅ Pass | 1.781s | |
+| Ask Advice Function | ✅ Pass | 2.140s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 4.836s | |
+| Basic Context Memory Test | ✅ Pass | 2.497s | |
+| Function Argument Memory Test | ✅ Pass | 1.844s | |
+| Function Response Memory Test | ✅ Pass | 2.066s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 5.170s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 6.002s | |
+| Penetration Testing Methodology | ✅ Pass | 7.523s | |
+| Vulnerability Assessment Tools | ✅ Pass | 7.854s | |
+| SQL Injection Attack Type | ✅ Pass | 2.703s | |
+| Penetration Testing Framework | ✅ Pass | 10.072s | |
+| Web Application Security Scanner | ✅ Pass | 5.844s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.817s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 4.786s
+**Average latency**: 3.886s
---
-### reflector (kimi-k2-0905-preview)
+### reflector (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 4.417s | |
-| Text Transform Uppercase | ✅ Pass | 1.378s | |
-| Count from 1 to 5 | ✅ Pass | 2.084s | |
-| Math Calculation | ✅ Pass | 1.037s | |
-| Basic Echo Function | ✅ Pass | 2.442s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.130s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.451s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 2.794s | |
+| Simple Math | ✅ Pass | 2.557s | |
+| Text Transform Uppercase | ✅ Pass | 0.691s | |
+| Count from 1 to 5 | ✅ Pass | 0.821s | |
+| Math Calculation | ✅ Pass | 0.788s | |
+| Basic Echo Function | ✅ Pass | 1.170s | |
+| Streaming Simple Math Streaming | ✅ Pass | 0.658s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.917s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.215s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 2.659s | |
-| Search Query Function | ✅ Pass | 2.429s | |
-| Ask Advice Function | ✅ Pass | 3.548s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 4.580s | |
-| Basic Context Memory Test | ✅ Pass | 1.911s | |
-| Function Argument Memory Test | ✅ Pass | 1.232s | |
-| Function Response Memory Test | ✅ Pass | 1.283s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 8.395s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.562s | |
-| Penetration Testing Methodology | ✅ Pass | 3.007s | |
-| Vulnerability Assessment Tools | ✅ Pass | 2.142s | |
-| SQL Injection Attack Type | ✅ Pass | 3.116s | |
-| Penetration Testing Framework | ✅ Pass | 1.818s | |
-| Web Application Security Scanner | ✅ Pass | 1.452s | |
-| Penetration Testing Tool Selection | ✅ Pass | 3.308s | |
+| JSON Response Function | ✅ Pass | 1.349s | |
+| Search Query Function | ✅ Pass | 1.016s | |
+| Ask Advice Function | ✅ Pass | 1.553s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.116s | |
+| Basic Context Memory Test | ✅ Pass | 1.271s | |
+| Function Argument Memory Test | ✅ Pass | 0.825s | |
+| Function Response Memory Test | ✅ Pass | 0.693s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 1.984s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 0.656s | |
+| Penetration Testing Methodology | ✅ Pass | 3.017s | |
+| Vulnerability Assessment Tools | ✅ Pass | 2.796s | |
+| SQL Injection Attack Type | ✅ Pass | 0.961s | |
+| Penetration Testing Framework | ✅ Pass | 4.904s | |
+| Web Application Security Scanner | ✅ Pass | 2.988s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.130s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 2.573s
+**Average latency**: 1.526s
---
-### searcher (kimi-k2-0905-preview)
+### searcher (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 4.338s | |
-| Text Transform Uppercase | ✅ Pass | 1.335s | |
-| Count from 1 to 5 | ✅ Pass | 1.871s | |
-| Math Calculation | ✅ Pass | 1.084s | |
-| Basic Echo Function | ✅ Pass | 2.404s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.071s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.100s | |
-| Streaming Basic Echo Function Streaming | ❌ Fail | 12.939s | no tool calls found, expected at least 1 |
+| Simple Math | ✅ Pass | 2.629s | |
+| Text Transform Uppercase | ✅ Pass | 0.720s | |
+| Count from 1 to 5 | ✅ Pass | 0.826s | |
+| Math Calculation | ✅ Pass | 1.210s | |
+| Basic Echo Function | ✅ Pass | 1.057s | |
+| Streaming Simple Math Streaming | ✅ Pass | 0.705s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.748s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.239s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 3.022s | |
-| Search Query Function | ✅ Pass | 2.434s | |
-| Ask Advice Function | ✅ Pass | 3.536s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 2.624s | |
-| Basic Context Memory Test | ✅ Pass | 2.379s | |
-| Function Argument Memory Test | ✅ Pass | 1.240s | |
-| Function Response Memory Test | ✅ Pass | 1.115s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 9.066s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.630s | |
-| Penetration Testing Methodology | ✅ Pass | 1.995s | |
-| Vulnerability Assessment Tools | ✅ Pass | 2.026s | |
-| SQL Injection Attack Type | ✅ Pass | 2.258s | |
-| Penetration Testing Framework | ✅ Pass | 1.442s | |
-| Web Application Security Scanner | ✅ Pass | 1.659s | |
-| Penetration Testing Tool Selection | ✅ Pass | 3.274s | |
-
-**Summary**: 22/23 (95.65%) successful tests
-
-**Average latency**: 2.907s
+| JSON Response Function | ✅ Pass | 1.191s | |
+| Search Query Function | ✅ Pass | 1.057s | |
+| Ask Advice Function | ✅ Pass | 1.340s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.069s | |
+| Basic Context Memory Test | ✅ Pass | 1.075s | |
+| Function Argument Memory Test | ✅ Pass | 0.814s | |
+| Function Response Memory Test | ✅ Pass | 0.680s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.213s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 0.775s | |
+| Penetration Testing Methodology | ✅ Pass | 3.708s | |
+| Vulnerability Assessment Tools | ✅ Pass | 2.575s | |
+| SQL Injection Attack Type | ✅ Pass | 0.897s | |
+| Penetration Testing Framework | ✅ Pass | 2.486s | |
+| Web Application Security Scanner | ✅ Pass | 1.508s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.267s | |
+
+**Summary**: 23/23 (100.00%) successful tests
+
+**Average latency**: 1.383s
---
-### enricher (kimi-k2-0905-preview)
+### enricher (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 1.185s | |
-| Text Transform Uppercase | ✅ Pass | 1.643s | |
-| Count from 1 to 5 | ✅ Pass | 2.241s | |
-| Math Calculation | ✅ Pass | 1.032s | |
-| Basic Echo Function | ✅ Pass | 2.775s | |
-| Streaming Simple Math Streaming | ✅ Pass | 0.915s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.541s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 2.984s | |
+| Simple Math | ✅ Pass | 0.721s | |
+| Text Transform Uppercase | ✅ Pass | 0.799s | |
+| Count from 1 to 5 | ✅ Pass | 0.791s | |
+| Math Calculation | ✅ Pass | 0.712s | |
+| Basic Echo Function | ✅ Pass | 1.013s | |
+| Streaming Simple Math Streaming | ✅ Pass | 0.621s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.699s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.240s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 3.105s | |
-| Search Query Function | ✅ Pass | 2.448s | |
-| Ask Advice Function | ✅ Pass | 3.450s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 3.292s | |
-| Basic Context Memory Test | ✅ Pass | 2.211s | |
-| Function Argument Memory Test | ✅ Pass | 1.394s | |
-| Function Response Memory Test | ✅ Pass | 1.085s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 7.307s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.757s | |
-| Penetration Testing Methodology | ✅ Pass | 1.596s | |
-| Vulnerability Assessment Tools | ✅ Pass | 1.403s | |
-| SQL Injection Attack Type | ✅ Pass | 2.025s | |
-| Penetration Testing Framework | ✅ Pass | 1.477s | |
-| Web Application Security Scanner | ✅ Pass | 2.069s | |
-| Penetration Testing Tool Selection | ✅ Pass | 3.374s | |
+| JSON Response Function | ✅ Pass | 1.281s | |
+| Search Query Function | ✅ Pass | 1.034s | |
+| Ask Advice Function | ✅ Pass | 1.261s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 0.968s | |
+| Basic Context Memory Test | ✅ Pass | 0.962s | |
+| Function Argument Memory Test | ✅ Pass | 0.781s | |
+| Function Response Memory Test | ✅ Pass | 0.746s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.290s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 0.715s | |
+| Penetration Testing Methodology | ✅ Pass | 3.325s | |
+| Vulnerability Assessment Tools | ✅ Pass | 2.531s | |
+| SQL Injection Attack Type | ✅ Pass | 1.343s | |
+| Penetration Testing Framework | ✅ Pass | 2.634s | |
+| Web Application Security Scanner | ✅ Pass | 1.549s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.217s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 2.275s
+**Average latency**: 1.271s
---
-### coder (kimi-k2.5)
+### coder (kimi-k2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 1.468s | |
-| Text Transform Uppercase | ✅ Pass | 2.235s | |
-| Count from 1 to 5 | ✅ Pass | 2.196s | |
-| Math Calculation | ✅ Pass | 2.012s | |
-| Basic Echo Function | ✅ Pass | 1.991s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.718s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.004s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 2.130s | |
+| Simple Math | ✅ Pass | 2.077s | |
+| Text Transform Uppercase | ✅ Pass | 1.523s | |
+| Count from 1 to 5 | ✅ Pass | 1.973s | |
+| Math Calculation | ✅ Pass | 1.502s | |
+| Basic Echo Function | ✅ Pass | 1.630s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.642s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.130s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.123s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 2.385s | |
-| Search Query Function | ✅ Pass | 2.347s | |
-| Ask Advice Function | ✅ Pass | 2.341s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 2.115s | |
-| Basic Context Memory Test | ✅ Pass | 2.458s | |
-| Function Argument Memory Test | ✅ Pass | 2.221s | |
-| Function Response Memory Test | ✅ Pass | 3.387s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 3.680s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 3.346s | |
-| Penetration Testing Methodology | ✅ Pass | 15.380s | |
-| Vulnerability Assessment Tools | ✅ Pass | 8.039s | |
-| SQL Injection Attack Type | ✅ Pass | 7.011s | |
-| Penetration Testing Framework | ✅ Pass | 10.099s | |
-| Penetration Testing Tool Selection | ✅ Pass | 2.757s | |
-| Web Application Security Scanner | ✅ Pass | 11.407s | |
+| JSON Response Function | ✅ Pass | 1.829s | |
+| Search Query Function | ✅ Pass | 1.558s | |
+| Ask Advice Function | ✅ Pass | 1.854s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.841s | |
+| Basic Context Memory Test | ✅ Pass | 2.435s | |
+| Function Argument Memory Test | ✅ Pass | 1.975s | |
+| Function Response Memory Test | ✅ Pass | 2.004s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 3.342s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.180s | |
+| Penetration Testing Methodology | ✅ Pass | 8.157s | |
+| SQL Injection Attack Type | ✅ Pass | 3.469s | |
+| Vulnerability Assessment Tools | ✅ Pass | 18.587s | |
+| Penetration Testing Framework | ✅ Pass | 3.799s | |
+| Web Application Security Scanner | ✅ Pass | 3.946s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.792s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 4.206s
+**Average latency**: 3.321s
---
-### installer (kimi-k2-turbo-preview)
+### installer (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 0.695s | |
-| Text Transform Uppercase | ✅ Pass | 0.708s | |
-| Count from 1 to 5 | ✅ Pass | 0.789s | |
-| Math Calculation | ✅ Pass | 0.718s | |
-| Basic Echo Function | ✅ Pass | 1.103s | |
-| Streaming Simple Math Streaming | ✅ Pass | 0.709s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.758s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 0.985s | |
+| Simple Math | ✅ Pass | 2.393s | |
+| Text Transform Uppercase | ✅ Pass | 2.517s | |
+| Count from 1 to 5 | ✅ Pass | 1.722s | |
+| Math Calculation | ✅ Pass | 1.790s | |
+| Basic Echo Function | ✅ Pass | 1.364s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.430s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.340s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.897s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 0.910s | |
-| Search Query Function | ✅ Pass | 0.811s | |
-| Ask Advice Function | ✅ Pass | 1.036s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.047s | |
-| Basic Context Memory Test | ✅ Pass | 0.968s | |
-| Function Argument Memory Test | ✅ Pass | 0.729s | |
-| Function Response Memory Test | ✅ Pass | 0.775s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 1.667s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 0.733s | |
-| Penetration Testing Methodology | ✅ Pass | 0.866s | |
-| Vulnerability Assessment Tools | ✅ Pass | 1.244s | |
-| SQL Injection Attack Type | ✅ Pass | 1.423s | |
-| Penetration Testing Framework | ✅ Pass | 0.793s | |
-| Web Application Security Scanner | ✅ Pass | 0.720s | |
-| Penetration Testing Tool Selection | ✅ Pass | 0.920s | |
+| JSON Response Function | ✅ Pass | 1.757s | |
+| Search Query Function | ✅ Pass | 1.746s | |
+| Ask Advice Function | ✅ Pass | 2.257s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.568s | |
+| Basic Context Memory Test | ✅ Pass | 2.824s | |
+| Function Argument Memory Test | ✅ Pass | 2.347s | |
+| Function Response Memory Test | ✅ Pass | 1.724s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 3.100s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 2.024s | |
+| Penetration Testing Methodology | ✅ Pass | 11.036s | |
+| Vulnerability Assessment Tools | ✅ Pass | 5.959s | |
+| SQL Injection Attack Type | ✅ Pass | 5.710s | |
+| Penetration Testing Framework | ✅ Pass | 10.269s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.241s | |
+| Web Application Security Scanner | ✅ Pass | 6.747s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 0.918s
+**Average latency**: 3.381s
---
-### pentester (kimi-k2-turbo-preview)
+### pentester (kimi-k2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 0.723s | |
-| Text Transform Uppercase | ✅ Pass | 0.775s | |
-| Count from 1 to 5 | ✅ Pass | 0.871s | |
-| Math Calculation | ✅ Pass | 0.756s | |
-| Basic Echo Function | ✅ Pass | 1.174s | |
-| Streaming Simple Math Streaming | ✅ Pass | 0.561s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.938s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 1.195s | |
+| Simple Math | ✅ Pass | 4.828s | |
+| Text Transform Uppercase | ✅ Pass | 1.994s | |
+| Count from 1 to 5 | ✅ Pass | 2.194s | |
+| Math Calculation | ✅ Pass | 1.846s | |
+| Basic Echo Function | ✅ Pass | 1.777s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.363s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.947s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.943s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 0.878s | |
-| Search Query Function | ✅ Pass | 0.856s | |
-| Ask Advice Function | ✅ Pass | 1.018s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.012s | |
-| Basic Context Memory Test | ✅ Pass | 0.775s | |
-| Function Argument Memory Test | ✅ Pass | 0.752s | |
-| Function Response Memory Test | ✅ Pass | 0.762s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 1.378s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 0.791s | |
-| Penetration Testing Methodology | ✅ Pass | 0.874s | |
-| Vulnerability Assessment Tools | ✅ Pass | 0.982s | |
-| SQL Injection Attack Type | ✅ Pass | 0.833s | |
-| Penetration Testing Framework | ✅ Pass | 0.822s | |
-| Web Application Security Scanner | ✅ Pass | 0.987s | |
-| Penetration Testing Tool Selection | ✅ Pass | 1.006s | |
+| JSON Response Function | ✅ Pass | 1.911s | |
+| Search Query Function | ✅ Pass | 2.339s | |
+| Ask Advice Function | ✅ Pass | 2.164s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.754s | |
+| Basic Context Memory Test | ✅ Pass | 2.243s | |
+| Function Argument Memory Test | ✅ Pass | 1.688s | |
+| Function Response Memory Test | ✅ Pass | 2.078s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 3.638s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 2.884s | |
+| Penetration Testing Methodology | ✅ Pass | 7.438s | |
+| SQL Injection Attack Type | ✅ Pass | 2.465s | |
+| Vulnerability Assessment Tools | ✅ Pass | 16.920s | |
+| Penetration Testing Framework | ✅ Pass | 6.907s | |
+| Web Application Security Scanner | ✅ Pass | 4.049s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.557s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 0.901s
+**Average latency**: 3.432s
---
diff --git a/examples/tests/qwen-report.md b/examples/tests/qwen-report.md
index fee903b8..34b41008 100644
--- a/examples/tests/qwen-report.md
+++ b/examples/tests/qwen-report.md
@@ -1,27 +1,27 @@
# LLM Agent Testing Report
-Generated: Thu, 05 Mar 2026 15:23:06 UTC
+Generated: Wed, 27 May 2026 23:05:00 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
-| simple | qwen3.5-flash | true | 23/23 (100.00%) | 3.985s |
-| simple_json | qwen3.5-flash | true | 5/5 (100.00%) | 7.246s |
-| primary_agent | qwen3.5-plus | true | 23/23 (100.00%) | 6.614s |
-| assistant | qwen3.5-plus | true | 23/23 (100.00%) | 7.055s |
-| generator | qwen3-max | true | 23/23 (100.00%) | 2.869s |
-| refiner | qwen3-max | true | 23/23 (100.00%) | 3.214s |
-| adviser | qwen3-max | true | 23/23 (100.00%) | 2.760s |
-| reflector | qwen3.5-flash | true | 23/23 (100.00%) | 2.902s |
-| searcher | qwen3.5-flash | true | 23/23 (100.00%) | 3.041s |
-| enricher | qwen3.5-flash | true | 23/23 (100.00%) | 2.903s |
-| coder | qwen3.5-plus | true | 23/23 (100.00%) | 6.767s |
-| installer | qwen3.5-plus | true | 23/23 (100.00%) | 6.970s |
-| pentester | qwen3.5-plus | true | 23/23 (100.00%) | 6.877s |
-
-**Total**: 281/281 (100.00%) successful tests
-**Overall average latency**: 4.709s
+| simple | qwen3.5-flash | true | 23/23 (100.00%) | 3.014s |
+| simple_json | qwen3.5-flash | true | 5/5 (100.00%) | 7.088s |
+| primary_agent | qwen3.6-plus | true | 23/23 (100.00%) | 5.366s |
+| assistant | qwen3.6-plus | true | 23/23 (100.00%) | 5.758s |
+| generator | qwen3.7-max | true | 23/23 (100.00%) | 3.473s |
+| refiner | qwen3.7-max | true | 23/23 (100.00%) | 3.352s |
+| adviser | qwen3.7-max | true | 22/23 (95.65%) | 2.941s |
+| reflector | qwen3.5-flash | true | 23/23 (100.00%) | 3.377s |
+| searcher | qwen3.5-flash | true | 23/23 (100.00%) | 4.025s |
+| enricher | qwen3.5-flash | true | 23/23 (100.00%) | 2.857s |
+| coder | qwen3-coder-plus | true | 23/23 (100.00%) | 1.556s |
+| installer | qwen3-coder-flash | true | 20/23 (86.96%) | 1.060s |
+| pentester | qwen3.6-plus | true | 23/23 (100.00%) | 5.504s |
+
+**Total**: 277/281 (98.58%) successful tests
+**Overall average latency**: 3.587s
## Detailed Results
@@ -31,38 +31,38 @@ Generated: Thu, 05 Mar 2026 15:23:06 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 2.732s | |
-| Text Transform Uppercase | ✅ Pass | 2.621s | |
-| Count from 1 to 5 | ✅ Pass | 2.621s | |
-| Math Calculation | ✅ Pass | 1.976s | |
-| Basic Echo Function | ✅ Pass | 1.258s | |
-| Streaming Simple Math Streaming | ✅ Pass | 2.289s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.988s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 1.438s | |
+| Simple Math | ✅ Pass | 2.746s | |
+| Text Transform Uppercase | ✅ Pass | 2.043s | |
+| Count from 1 to 5 | ✅ Pass | 3.729s | |
+| Math Calculation | ✅ Pass | 1.686s | |
+| Basic Echo Function | ✅ Pass | 1.282s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.606s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.200s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 0.928s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 1.548s | |
-| Search Query Function | ✅ Pass | 1.440s | |
-| Ask Advice Function | ✅ Pass | 1.522s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.453s | |
-| Basic Context Memory Test | ✅ Pass | 3.117s | |
-| Function Argument Memory Test | ✅ Pass | 1.429s | |
-| Function Response Memory Test | ✅ Pass | 1.201s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 1.852s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.456s | |
-| Penetration Testing Methodology | ✅ Pass | 8.537s | |
-| SQL Injection Attack Type | ✅ Pass | 2.840s | |
-| Vulnerability Assessment Tools | ✅ Pass | 38.650s | |
-| Penetration Testing Framework | ✅ Pass | 4.082s | |
-| Web Application Security Scanner | ✅ Pass | 3.694s | |
-| Penetration Testing Tool Selection | ✅ Pass | 1.896s | |
+| JSON Response Function | ✅ Pass | 1.163s | |
+| Search Query Function | ✅ Pass | 1.264s | |
+| Ask Advice Function | ✅ Pass | 1.232s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 0.884s | |
+| Basic Context Memory Test | ✅ Pass | 2.803s | |
+| Function Argument Memory Test | ✅ Pass | 0.750s | |
+| Function Response Memory Test | ✅ Pass | 0.951s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 1.953s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 1.255s | |
+| Penetration Testing Methodology | ✅ Pass | 7.005s | |
+| Vulnerability Assessment Tools | ✅ Pass | 19.557s | |
+| SQL Injection Attack Type | ✅ Pass | 2.703s | |
+| Penetration Testing Framework | ✅ Pass | 5.521s | |
+| Web Application Security Scanner | ✅ Pass | 4.520s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.533s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 3.985s
+**Average latency**: 3.014s
---
@@ -72,220 +72,220 @@ Generated: Thu, 05 Mar 2026 15:23:06 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Vulnerability Report Memory Test | ✅ Pass | 5.818s | |
-| Streaming Person Information JSON Streaming | ✅ Pass | 4.840s | |
-| User Profile JSON | ✅ Pass | 6.102s | |
-| Project Information JSON | ✅ Pass | 7.050s | |
-| Person Information JSON | ✅ Pass | 12.418s | |
+| Person Information JSON | ✅ Pass | 4.643s | |
+| User Profile JSON | ✅ Pass | 5.759s | |
+| Streaming Person Information JSON Streaming | ✅ Pass | 6.514s | |
+| Project Information JSON | ✅ Pass | 7.046s | |
+| Vulnerability Report Memory Test | ✅ Pass | 11.475s | |
**Summary**: 5/5 (100.00%) successful tests
-**Average latency**: 7.246s
+**Average latency**: 7.088s
---
-### primary_agent (qwen3.5-plus)
+### primary_agent (qwen3.6-plus)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 5.394s | |
-| Text Transform Uppercase | ✅ Pass | 4.513s | |
-| Count from 1 to 5 | ✅ Pass | 6.846s | |
-| Math Calculation | ✅ Pass | 3.753s | |
-| Basic Echo Function | ✅ Pass | 2.614s | |
-| Streaming Simple Math Streaming | ✅ Pass | 4.509s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.269s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 2.698s | |
+| Simple Math | ✅ Pass | 5.300s | |
+| Text Transform Uppercase | ✅ Pass | 3.870s | |
+| Math Calculation | ✅ Pass | 2.999s | |
+| Count from 1 to 5 | ✅ Pass | 12.409s | |
+| Basic Echo Function | ✅ Pass | 3.121s | |
+| Streaming Simple Math Streaming | ✅ Pass | 4.737s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.527s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 4.412s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 2.813s | |
-| Search Query Function | ✅ Pass | 2.489s | |
-| Ask Advice Function | ✅ Pass | 2.975s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 2.427s | |
-| Basic Context Memory Test | ✅ Pass | 5.721s | |
-| Function Argument Memory Test | ✅ Pass | 2.442s | |
-| Function Response Memory Test | ✅ Pass | 2.232s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 4.758s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 3.045s | |
-| Penetration Testing Methodology | ✅ Pass | 12.097s | |
-| SQL Injection Attack Type | ✅ Pass | 5.733s | |
-| Vulnerability Assessment Tools | ✅ Pass | 45.256s | |
-| Penetration Testing Framework | ✅ Pass | 12.601s | |
-| Web Application Security Scanner | ✅ Pass | 8.284s | |
-| Penetration Testing Tool Selection | ✅ Pass | 3.646s | |
+| JSON Response Function | ✅ Pass | 2.506s | |
+| Search Query Function | ✅ Pass | 2.580s | |
+| Ask Advice Function | ✅ Pass | 3.013s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 2.157s | |
+| Basic Context Memory Test | ✅ Pass | 4.461s | |
+| Function Argument Memory Test | ✅ Pass | 4.159s | |
+| Function Response Memory Test | ✅ Pass | 6.988s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 5.212s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.572s | |
+| Penetration Testing Methodology | ✅ Pass | 11.000s | |
+| Vulnerability Assessment Tools | ✅ Pass | 11.996s | |
+| SQL Injection Attack Type | ✅ Pass | 5.396s | |
+| Penetration Testing Framework | ✅ Pass | 11.164s | |
+| Web Application Security Scanner | ✅ Pass | 4.543s | |
+| Penetration Testing Tool Selection | ✅ Pass | 3.287s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 6.614s
+**Average latency**: 5.366s
---
-### assistant (qwen3.5-plus)
+### assistant (qwen3.6-plus)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 5.779s | |
-| Text Transform Uppercase | ✅ Pass | 4.719s | |
-| Count from 1 to 5 | ✅ Pass | 8.844s | |
-| Math Calculation | ✅ Pass | 3.883s | |
-| Basic Echo Function | ✅ Pass | 2.546s | |
-| Streaming Simple Math Streaming | ✅ Pass | 4.775s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.000s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 2.827s | |
+| Simple Math | ✅ Pass | 5.339s | |
+| Text Transform Uppercase | ✅ Pass | 5.104s | |
+| Count from 1 to 5 | ✅ Pass | 5.229s | |
+| Math Calculation | ✅ Pass | 4.509s | |
+| Basic Echo Function | ✅ Pass | 3.068s | |
+| Streaming Simple Math Streaming | ✅ Pass | 4.267s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.656s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.860s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 3.587s | |
-| Search Query Function | ✅ Pass | 2.333s | |
-| Ask Advice Function | ✅ Pass | 2.948s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 2.618s | |
-| Basic Context Memory Test | ✅ Pass | 5.818s | |
-| Function Argument Memory Test | ✅ Pass | 2.564s | |
-| Function Response Memory Test | ✅ Pass | 4.368s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 4.378s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 3.191s | |
-| Penetration Testing Methodology | ✅ Pass | 13.516s | |
-| SQL Injection Attack Type | ✅ Pass | 7.948s | |
-| Vulnerability Assessment Tools | ✅ Pass | 47.228s | |
-| Penetration Testing Framework | ✅ Pass | 9.612s | |
-| Web Application Security Scanner | ✅ Pass | 10.246s | |
-| Penetration Testing Tool Selection | ✅ Pass | 3.522s | |
+| JSON Response Function | ✅ Pass | 3.200s | |
+| Search Query Function | ✅ Pass | 1.780s | |
+| Ask Advice Function | ✅ Pass | 2.841s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 2.020s | |
+| Basic Context Memory Test | ✅ Pass | 5.116s | |
+| Function Argument Memory Test | ✅ Pass | 3.908s | |
+| Function Response Memory Test | ✅ Pass | 4.534s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 8.970s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 7.041s | |
+| Penetration Testing Methodology | ✅ Pass | 11.914s | |
+| Vulnerability Assessment Tools | ✅ Pass | 17.492s | |
+| SQL Injection Attack Type | ✅ Pass | 5.510s | |
+| Penetration Testing Framework | ✅ Pass | 9.591s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.796s | |
+| Web Application Security Scanner | ✅ Pass | 10.669s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 7.055s
+**Average latency**: 5.758s
---
-### generator (qwen3-max)
+### generator (qwen3.7-max)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 2.382s | |
-| Text Transform Uppercase | ✅ Pass | 1.534s | |
-| Count from 1 to 5 | ✅ Pass | 3.406s | |
-| Math Calculation | ✅ Pass | 2.066s | |
-| Basic Echo Function | ✅ Pass | 1.871s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.919s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.433s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 2.361s | |
+| Simple Math | ✅ Pass | 3.289s | |
+| Text Transform Uppercase | ✅ Pass | 2.103s | |
+| Count from 1 to 5 | ✅ Pass | 4.641s | |
+| Math Calculation | ✅ Pass | 1.319s | |
+| Basic Echo Function | ✅ Pass | 1.852s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.998s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.591s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.928s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 2.966s | |
-| Search Query Function | ✅ Pass | 1.943s | |
-| Ask Advice Function | ✅ Pass | 2.544s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 2.371s | |
-| Basic Context Memory Test | ✅ Pass | 2.034s | |
-| Function Argument Memory Test | ✅ Pass | 1.764s | |
-| Function Response Memory Test | ✅ Pass | 2.797s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 5.606s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 5.562s | |
-| Penetration Testing Methodology | ✅ Pass | 5.060s | |
-| Vulnerability Assessment Tools | ✅ Pass | 3.424s | |
-| SQL Injection Attack Type | ✅ Pass | 2.197s | |
-| Penetration Testing Framework | ✅ Pass | 3.445s | |
-| Web Application Security Scanner | ✅ Pass | 1.614s | |
-| Penetration Testing Tool Selection | ✅ Pass | 4.678s | |
+| JSON Response Function | ✅ Pass | 1.388s | |
+| Search Query Function | ✅ Pass | 2.131s | |
+| Ask Advice Function | ✅ Pass | 1.544s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.977s | |
+| Basic Context Memory Test | ✅ Pass | 4.750s | |
+| Function Argument Memory Test | ✅ Pass | 3.913s | |
+| Function Response Memory Test | ✅ Pass | 4.594s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 3.141s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.417s | |
+| Penetration Testing Methodology | ✅ Pass | 3.566s | |
+| Vulnerability Assessment Tools | ✅ Pass | 15.497s | |
+| SQL Injection Attack Type | ✅ Pass | 4.163s | |
+| Penetration Testing Framework | ✅ Pass | 2.677s | |
+| Web Application Security Scanner | ✅ Pass | 4.448s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.928s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 2.869s
+**Average latency**: 3.473s
---
-### refiner (qwen3-max)
+### refiner (qwen3.7-max)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 3.583s | |
-| Text Transform Uppercase | ✅ Pass | 1.230s | |
-| Count from 1 to 5 | ✅ Pass | 1.545s | |
-| Math Calculation | ✅ Pass | 3.298s | |
-| Basic Echo Function | ✅ Pass | 1.863s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.304s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.383s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 2.649s | |
+| Simple Math | ✅ Pass | 4.088s | |
+| Text Transform Uppercase | ✅ Pass | 3.518s | |
+| Count from 1 to 5 | ✅ Pass | 4.139s | |
+| Math Calculation | ✅ Pass | 1.941s | |
+| Basic Echo Function | ✅ Pass | 1.601s | |
+| Streaming Simple Math Streaming | ✅ Pass | 2.624s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.881s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 2.028s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 2.799s | |
-| Search Query Function | ✅ Pass | 5.225s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 2.741s | |
-| Ask Advice Function | ✅ Pass | 9.108s | |
-| Basic Context Memory Test | ✅ Pass | 1.642s | |
-| Function Argument Memory Test | ✅ Pass | 2.789s | |
-| Function Response Memory Test | ✅ Pass | 2.780s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 5.472s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.503s | |
-| Penetration Testing Methodology | ✅ Pass | 5.761s | |
-| Vulnerability Assessment Tools | ✅ Pass | 3.824s | |
-| SQL Injection Attack Type | ✅ Pass | 1.506s | |
-| Penetration Testing Framework | ✅ Pass | 3.866s | |
-| Web Application Security Scanner | ✅ Pass | 4.553s | |
-| Penetration Testing Tool Selection | ✅ Pass | 3.486s | |
+| JSON Response Function | ✅ Pass | 2.901s | |
+| Search Query Function | ✅ Pass | 2.251s | |
+| Ask Advice Function | ✅ Pass | 1.898s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 2.260s | |
+| Basic Context Memory Test | ✅ Pass | 2.202s | |
+| Function Argument Memory Test | ✅ Pass | 2.319s | |
+| Function Response Memory Test | ✅ Pass | 4.750s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.590s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.847s | |
+| Penetration Testing Methodology | ✅ Pass | 4.130s | |
+| Vulnerability Assessment Tools | ✅ Pass | 12.315s | |
+| SQL Injection Attack Type | ✅ Pass | 4.225s | |
+| Penetration Testing Framework | ✅ Pass | 2.234s | |
+| Web Application Security Scanner | ✅ Pass | 2.434s | |
+| Penetration Testing Tool Selection | ✅ Pass | 2.917s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 3.214s
+**Average latency**: 3.352s
---
-### adviser (qwen3-max)
+### adviser (qwen3.7-max)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 3.255s | |
-| Text Transform Uppercase | ✅ Pass | 1.503s | |
-| Count from 1 to 5 | ✅ Pass | 2.459s | |
-| Math Calculation | ✅ Pass | 1.292s | |
-| Basic Echo Function | ✅ Pass | 2.233s | |
-| Streaming Simple Math Streaming | ✅ Pass | 1.909s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.740s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 3.197s | |
+| Simple Math | ✅ Pass | 3.094s | |
+| Text Transform Uppercase | ✅ Pass | 3.196s | |
+| Count from 1 to 5 | ✅ Pass | 3.210s | |
+| Math Calculation | ✅ Pass | 2.542s | |
+| Basic Echo Function | ✅ Pass | 1.431s | |
+| Streaming Simple Math Streaming | ✅ Pass | 2.476s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.746s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.526s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 3.978s | |
-| Search Query Function | ✅ Pass | 2.562s | |
-| Ask Advice Function | ✅ Pass | 4.336s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 2.548s | |
-| Basic Context Memory Test | ✅ Pass | 2.117s | |
-| Function Argument Memory Test | ✅ Pass | 2.020s | |
-| Function Response Memory Test | ✅ Pass | 2.799s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 4.940s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.440s | |
-| Penetration Testing Methodology | ✅ Pass | 2.778s | |
-| Vulnerability Assessment Tools | ✅ Pass | 3.114s | |
-| SQL Injection Attack Type | ✅ Pass | 4.323s | |
-| Penetration Testing Framework | ✅ Pass | 3.297s | |
-| Web Application Security Scanner | ✅ Pass | 1.453s | |
-| Penetration Testing Tool Selection | ✅ Pass | 4.171s | |
-
-**Summary**: 23/23 (100.00%) successful tests
-
-**Average latency**: 2.760s
+| JSON Response Function | ✅ Pass | 2.530s | |
+| Search Query Function | ✅ Pass | 1.782s | |
+| Ask Advice Function | ✅ Pass | 1.862s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.915s | |
+| Basic Context Memory Test | ✅ Pass | 3.090s | |
+| Function Argument Memory Test | ❌ Fail | 3.063s | expected text 'Go programming language' not found |
+| Function Response Memory Test | ✅ Pass | 4.037s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.505s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 3.618s | |
+| Penetration Testing Methodology | ✅ Pass | 2.493s | |
+| Vulnerability Assessment Tools | ✅ Pass | 6.921s | |
+| SQL Injection Attack Type | ✅ Pass | 3.418s | |
+| Penetration Testing Framework | ✅ Pass | 4.994s | |
+| Web Application Security Scanner | ✅ Pass | 1.890s | |
+| Penetration Testing Tool Selection | ✅ Pass | 3.285s | |
+
+**Summary**: 22/23 (95.65%) successful tests
+
+**Average latency**: 2.941s
---
@@ -295,38 +295,38 @@ Generated: Thu, 05 Mar 2026 15:23:06 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 2.791s | |
-| Text Transform Uppercase | ✅ Pass | 2.826s | |
-| Count from 1 to 5 | ✅ Pass | 3.055s | |
-| Math Calculation | ✅ Pass | 2.041s | |
-| Basic Echo Function | ✅ Pass | 1.482s | |
-| Streaming Simple Math Streaming | ✅ Pass | 2.351s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.763s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 1.381s | |
+| Simple Math | ✅ Pass | 3.446s | |
+| Text Transform Uppercase | ✅ Pass | 2.472s | |
+| Count from 1 to 5 | ✅ Pass | 3.109s | |
+| Math Calculation | ✅ Pass | 2.244s | |
+| Basic Echo Function | ✅ Pass | 1.191s | |
+| Streaming Simple Math Streaming | ✅ Pass | 2.377s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.817s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 0.952s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 1.757s | |
-| Search Query Function | ✅ Pass | 1.583s | |
-| Ask Advice Function | ✅ Pass | 1.569s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.409s | |
-| Basic Context Memory Test | ✅ Pass | 3.085s | |
-| Function Argument Memory Test | ✅ Pass | 1.815s | |
-| Function Response Memory Test | ✅ Pass | 2.675s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 2.251s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.755s | |
-| Penetration Testing Methodology | ✅ Pass | 5.117s | |
-| Vulnerability Assessment Tools | ✅ Pass | 10.283s | |
-| SQL Injection Attack Type | ✅ Pass | 2.764s | |
-| Penetration Testing Framework | ✅ Pass | 4.701s | |
-| Web Application Security Scanner | ✅ Pass | 4.535s | |
-| Penetration Testing Tool Selection | ✅ Pass | 1.755s | |
+| JSON Response Function | ✅ Pass | 1.376s | |
+| Search Query Function | ✅ Pass | 1.034s | |
+| Ask Advice Function | ✅ Pass | 1.177s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.399s | |
+| Basic Context Memory Test | ✅ Pass | 2.150s | |
+| Function Argument Memory Test | ✅ Pass | 0.884s | |
+| Function Response Memory Test | ✅ Pass | 0.900s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.927s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 1.201s | |
+| Penetration Testing Methodology | ✅ Pass | 8.294s | |
+| SQL Injection Attack Type | ✅ Pass | 2.431s | |
+| Vulnerability Assessment Tools | ✅ Pass | 24.680s | |
+| Penetration Testing Framework | ✅ Pass | 4.850s | |
+| Web Application Security Scanner | ✅ Pass | 4.195s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.546s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 2.902s
+**Average latency**: 3.377s
---
@@ -336,38 +336,38 @@ Generated: Thu, 05 Mar 2026 15:23:06 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 2.772s | |
-| Text Transform Uppercase | ✅ Pass | 2.426s | |
-| Count from 1 to 5 | ✅ Pass | 2.708s | |
-| Math Calculation | ✅ Pass | 1.732s | |
-| Basic Echo Function | ✅ Pass | 1.420s | |
-| Streaming Simple Math Streaming | ✅ Pass | 2.133s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.056s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 1.490s | |
+| Simple Math | ✅ Pass | 3.959s | |
+| Text Transform Uppercase | ✅ Pass | 1.535s | |
+| Count from 1 to 5 | ✅ Pass | 3.085s | |
+| Math Calculation | ✅ Pass | 1.883s | |
+| Basic Echo Function | ✅ Pass | 1.491s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.364s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.694s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.338s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 1.600s | |
-| Search Query Function | ✅ Pass | 1.553s | |
-| Ask Advice Function | ✅ Pass | 1.595s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.494s | |
-| Basic Context Memory Test | ✅ Pass | 3.296s | |
-| Function Argument Memory Test | ✅ Pass | 1.393s | |
-| Function Response Memory Test | ✅ Pass | 1.213s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 2.218s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 1.685s | |
-| Penetration Testing Methodology | ✅ Pass | 7.378s | |
-| Vulnerability Assessment Tools | ✅ Pass | 12.756s | |
-| SQL Injection Attack Type | ✅ Pass | 4.852s | |
-| Penetration Testing Framework | ✅ Pass | 6.239s | |
-| Web Application Security Scanner | ✅ Pass | 4.217s | |
-| Penetration Testing Tool Selection | ✅ Pass | 1.702s | |
+| JSON Response Function | ✅ Pass | 1.277s | |
+| Search Query Function | ✅ Pass | 1.249s | |
+| Ask Advice Function | ✅ Pass | 1.653s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 0.925s | |
+| Basic Context Memory Test | ✅ Pass | 3.426s | |
+| Function Argument Memory Test | ✅ Pass | 1.552s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.090s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 1.792s | |
+| Function Response Memory Test | ✅ Pass | 22.405s | |
+| Penetration Testing Methodology | ✅ Pass | 6.872s | |
+| SQL Injection Attack Type | ✅ Pass | 2.512s | |
+| Vulnerability Assessment Tools | ✅ Pass | 19.206s | |
+| Penetration Testing Framework | ✅ Pass | 4.480s | |
+| Web Application Security Scanner | ✅ Pass | 4.976s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.793s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 3.041s
+**Average latency**: 4.025s
---
@@ -377,161 +377,161 @@ Generated: Thu, 05 Mar 2026 15:23:06 UTC
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 2.027s | |
-| Text Transform Uppercase | ✅ Pass | 2.165s | |
-| Count from 1 to 5 | ✅ Pass | 4.274s | |
-| Math Calculation | ✅ Pass | 1.642s | |
-| Basic Echo Function | ✅ Pass | 1.393s | |
-| Streaming Simple Math Streaming | ✅ Pass | 2.071s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.683s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 1.455s | |
+| Simple Math | ✅ Pass | 1.626s | |
+| Text Transform Uppercase | ✅ Pass | 2.623s | |
+| Count from 1 to 5 | ✅ Pass | 2.794s | |
+| Math Calculation | ✅ Pass | 1.657s | |
+| Basic Echo Function | ✅ Pass | 1.406s | |
+| Streaming Simple Math Streaming | ✅ Pass | 2.086s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.390s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.279s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 1.553s | |
-| Search Query Function | ✅ Pass | 1.346s | |
-| Ask Advice Function | ✅ Pass | 1.686s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 1.385s | |
-| Basic Context Memory Test | ✅ Pass | 2.964s | |
-| Function Argument Memory Test | ✅ Pass | 1.430s | |
-| Function Response Memory Test | ✅ Pass | 1.515s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 2.056s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 2.501s | |
-| Penetration Testing Methodology | ✅ Pass | 4.478s | |
-| Vulnerability Assessment Tools | ✅ Pass | 13.331s | |
-| SQL Injection Attack Type | ✅ Pass | 3.656s | |
-| Penetration Testing Framework | ✅ Pass | 4.639s | |
-| Web Application Security Scanner | ✅ Pass | 4.488s | |
-| Penetration Testing Tool Selection | ✅ Pass | 2.019s | |
+| JSON Response Function | ✅ Pass | 1.314s | |
+| Search Query Function | ✅ Pass | 1.160s | |
+| Ask Advice Function | ✅ Pass | 1.645s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 0.948s | |
+| Basic Context Memory Test | ✅ Pass | 3.003s | |
+| Function Argument Memory Test | ✅ Pass | 2.254s | |
+| Function Response Memory Test | ✅ Pass | 4.521s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.225s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 1.494s | |
+| Penetration Testing Methodology | ✅ Pass | 5.008s | |
+| Vulnerability Assessment Tools | ✅ Pass | 11.717s | |
+| SQL Injection Attack Type | ✅ Pass | 4.587s | |
+| Penetration Testing Framework | ✅ Pass | 4.199s | |
+| Web Application Security Scanner | ✅ Pass | 4.499s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.261s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 2.903s
+**Average latency**: 2.857s
---
-### coder (qwen3.5-plus)
+### coder (qwen3-coder-plus)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 4.500s | |
-| Text Transform Uppercase | ✅ Pass | 5.010s | |
-| Count from 1 to 5 | ✅ Pass | 4.886s | |
-| Math Calculation | ✅ Pass | 4.225s | |
-| Basic Echo Function | ✅ Pass | 2.490s | |
-| Streaming Simple Math Streaming | ✅ Pass | 6.589s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.992s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 2.747s | |
+| Simple Math | ✅ Pass | 0.873s | |
+| Text Transform Uppercase | ✅ Pass | 0.824s | |
+| Count from 1 to 5 | ✅ Pass | 0.953s | |
+| Math Calculation | ✅ Pass | 0.842s | |
+| Basic Echo Function | ✅ Pass | 1.514s | |
+| Streaming Simple Math Streaming | ✅ Pass | 1.386s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.346s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 1.009s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 3.030s | |
-| Search Query Function | ✅ Pass | 2.563s | |
-| Ask Advice Function | ✅ Pass | 2.716s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 2.678s | |
-| Basic Context Memory Test | ✅ Pass | 5.383s | |
-| Function Argument Memory Test | ✅ Pass | 4.272s | |
-| Function Response Memory Test | ✅ Pass | 5.055s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 4.332s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 2.919s | |
-| Penetration Testing Methodology | ✅ Pass | 15.407s | |
-| SQL Injection Attack Type | ✅ Pass | 7.833s | |
-| Vulnerability Assessment Tools | ✅ Pass | 40.369s | |
-| Penetration Testing Framework | ✅ Pass | 10.080s | |
-| Web Application Security Scanner | ✅ Pass | 8.710s | |
-| Penetration Testing Tool Selection | ✅ Pass | 3.844s | |
+| JSON Response Function | ✅ Pass | 1.687s | |
+| Search Query Function | ✅ Pass | 0.994s | |
+| Ask Advice Function | ✅ Pass | 1.286s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 1.041s | |
+| Basic Context Memory Test | ✅ Pass | 0.905s | |
+| Function Argument Memory Test | ✅ Pass | 0.845s | |
+| Function Response Memory Test | ✅ Pass | 0.806s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 2.240s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 0.862s | |
+| Penetration Testing Methodology | ✅ Pass | 4.261s | |
+| Vulnerability Assessment Tools | ✅ Pass | 3.398s | |
+| SQL Injection Attack Type | ✅ Pass | 0.869s | |
+| Penetration Testing Framework | ✅ Pass | 3.691s | |
+| Web Application Security Scanner | ✅ Pass | 2.796s | |
+| Penetration Testing Tool Selection | ✅ Pass | 1.349s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 6.767s
+**Average latency**: 1.556s
---
-### installer (qwen3.5-plus)
+### installer (qwen3-coder-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 5.606s | |
-| Text Transform Uppercase | ✅ Pass | 4.408s | |
-| Count from 1 to 5 | ✅ Pass | 7.002s | |
-| Math Calculation | ✅ Pass | 4.185s | |
-| Basic Echo Function | ✅ Pass | 2.654s | |
-| Streaming Simple Math Streaming | ✅ Pass | 6.477s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.567s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 2.616s | |
+| Simple Math | ✅ Pass | 1.007s | |
+| Text Transform Uppercase | ✅ Pass | 0.566s | |
+| Count from 1 to 5 | ✅ Pass | 1.100s | |
+| Math Calculation | ✅ Pass | 1.011s | |
+| Basic Echo Function | ❌ Fail | 0.727s | no tool calls found, expected at least 1 |
+| Streaming Simple Math Streaming | ✅ Pass | 0.542s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.560s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 0.774s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 3.637s | |
-| Search Query Function | ✅ Pass | 2.200s | |
-| Ask Advice Function | ✅ Pass | 3.108s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 2.464s | |
-| Basic Context Memory Test | ✅ Pass | 4.485s | |
-| Function Argument Memory Test | ✅ Pass | 2.547s | |
-| Function Response Memory Test | ✅ Pass | 10.408s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 4.454s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 3.142s | |
-| Penetration Testing Methodology | ✅ Pass | 6.524s | |
-| SQL Injection Attack Type | ✅ Pass | 7.733s | |
-| Vulnerability Assessment Tools | ✅ Pass | 48.454s | |
-| Penetration Testing Framework | ✅ Pass | 11.429s | |
-| Web Application Security Scanner | ✅ Pass | 8.263s | |
-| Penetration Testing Tool Selection | ✅ Pass | 3.931s | |
-
-**Summary**: 23/23 (100.00%) successful tests
-
-**Average latency**: 6.970s
+| JSON Response Function | ✅ Pass | 0.830s | |
+| Search Query Function | ❌ Fail | 1.222s | no tool calls found, expected at least 1 |
+| Ask Advice Function | ✅ Pass | 0.877s | |
+| Streaming Search Query Function Streaming | ❌ Fail | 1.146s | no tool calls found, expected at least 1 |
+| Basic Context Memory Test | ✅ Pass | 0.973s | |
+| Function Argument Memory Test | ✅ Pass | 0.613s | |
+| Function Response Memory Test | ✅ Pass | 0.583s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 1.826s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 0.625s | |
+| Penetration Testing Methodology | ✅ Pass | 1.940s | |
+| Vulnerability Assessment Tools | ✅ Pass | 3.151s | |
+| SQL Injection Attack Type | ✅ Pass | 0.588s | |
+| Penetration Testing Framework | ✅ Pass | 1.175s | |
+| Web Application Security Scanner | ✅ Pass | 1.550s | |
+| Penetration Testing Tool Selection | ✅ Pass | 0.973s | |
+
+**Summary**: 20/23 (86.96%) successful tests
+
+**Average latency**: 1.060s
---
-### pentester (qwen3.5-plus)
+### pentester (qwen3.6-plus)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| Simple Math | ✅ Pass | 4.201s | |
-| Text Transform Uppercase | ✅ Pass | 4.717s | |
-| Count from 1 to 5 | ✅ Pass | 4.946s | |
-| Math Calculation | ✅ Pass | 3.891s | |
-| Basic Echo Function | ✅ Pass | 2.769s | |
-| Streaming Simple Math Streaming | ✅ Pass | 4.423s | |
-| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.584s | |
-| Streaming Basic Echo Function Streaming | ✅ Pass | 2.257s | |
+| Simple Math | ✅ Pass | 4.527s | |
+| Text Transform Uppercase | ✅ Pass | 4.599s | |
+| Count from 1 to 5 | ✅ Pass | 5.743s | |
+| Math Calculation | ✅ Pass | 3.509s | |
+| Basic Echo Function | ✅ Pass | 4.851s | |
+| Streaming Simple Math Streaming | ✅ Pass | 3.707s | |
+| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.781s | |
+| Streaming Basic Echo Function Streaming | ✅ Pass | 5.300s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
-| JSON Response Function | ✅ Pass | 3.128s | |
-| Search Query Function | ✅ Pass | 2.577s | |
-| Ask Advice Function | ✅ Pass | 2.910s | |
-| Streaming Search Query Function Streaming | ✅ Pass | 2.265s | |
-| Basic Context Memory Test | ✅ Pass | 5.007s | |
-| Function Argument Memory Test | ✅ Pass | 2.492s | |
-| Function Response Memory Test | ✅ Pass | 5.037s | |
-| Penetration Testing Memory with Tool Call | ✅ Pass | 4.579s | |
-| Cybersecurity Workflow Memory Test | ✅ Pass | 3.124s | |
-| Penetration Testing Methodology | ✅ Pass | 8.577s | |
-| SQL Injection Attack Type | ✅ Pass | 5.255s | |
-| Penetration Testing Framework | ✅ Pass | 11.867s | |
-| Vulnerability Assessment Tools | ✅ Pass | 56.554s | |
-| Web Application Security Scanner | ✅ Pass | 9.145s | |
-| Penetration Testing Tool Selection | ✅ Pass | 3.846s | |
+| JSON Response Function | ✅ Pass | 2.456s | |
+| Search Query Function | ✅ Pass | 3.989s | |
+| Ask Advice Function | ✅ Pass | 3.255s | |
+| Streaming Search Query Function Streaming | ✅ Pass | 2.430s | |
+| Basic Context Memory Test | ✅ Pass | 4.780s | |
+| Function Argument Memory Test | ✅ Pass | 5.338s | |
+| Function Response Memory Test | ✅ Pass | 2.911s | |
+| Penetration Testing Memory with Tool Call | ✅ Pass | 5.094s | |
+| Cybersecurity Workflow Memory Test | ✅ Pass | 6.163s | |
+| Penetration Testing Methodology | ✅ Pass | 12.475s | |
+| Vulnerability Assessment Tools | ✅ Pass | 11.406s | |
+| SQL Injection Attack Type | ✅ Pass | 6.216s | |
+| Penetration Testing Framework | ✅ Pass | 11.490s | |
+| Web Application Security Scanner | ✅ Pass | 8.229s | |
+| Penetration Testing Tool Selection | ✅ Pass | 3.336s | |
**Summary**: 23/23 (100.00%) successful tests
-**Average latency**: 6.877s
+**Average latency**: 5.504s
---