Fully autonomous AI Agents system capable of performing complex penetration testing tasks
Analytics API
GraphQL API for system analytics and statistics monitoring. This document covers three main use cases for developers building dashboards and analytics tools.
Important Data Notes
Duration Storage and Calculation
Database columns (as of migration 20260129_120000):
toolcalls.duration_seconds: Pre-calculated duration for completed toolcalls- Automatically set during migration:
EXTRACT(EPOCH FROM (updated_at - created_at)) - Set to 0.0 for incomplete toolcalls (
received,running) - Column is
NOT NULL DEFAULT 0.0 - Incremental updates: Backend calculates delta time and passes it to SQL
- Formula:
duration_seconds = duration_seconds + duration_delta(parameter from Go code) - Updated by SQL queries:
UpdateToolcallStatus,UpdateToolcallFinishedResult,UpdateToolcallFailedResult
- Automatically set during migration:
msgchains.duration_seconds: Pre-calculated duration for message chains- Automatically set during migration:
EXTRACT(EPOCH FROM (updated_at - created_at)) - Set to 0.0 for incomplete msgchains
- Column is
NOT NULL DEFAULT 0.0 - Incremental updates: Backend calculates delta time and passes it to SQL
- Formula:
duration_seconds = duration_seconds + duration_delta(parameter from Go code) - Updated by SQL queries:
UpdateMsgChain,UpdateMsgChainUsage
- Automatically set during migration:
Benefits of pre-calculated durations:
- Faster query performance (no real-time
EXTRACT(EPOCH FROM ...)calculations) - Consistent duration values (accumulated over multiple updates)
- Simpler SQL queries for analytics (just
tc.duration_secondsinstead of complex expressions) - Reduced database load for dashboard queries
- Incremental tracking: accurately captures time even when records are updated multiple times
- All analytics SQL queries updated to use
duration_secondscolumn directly
Incremental update logic:
- Backend code tracks execution time and calculates delta in seconds
- Delta is passed as parameter to SQL update queries
- SQL adds delta to existing
duration_secondsvalue - Prevents overwrites and ensures accurate cumulative duration
- Works correctly for long-running operations with multiple status updates
- Backend has full control over time measurement (can use high-precision timers)
Execution Time Metrics (totalDurationSeconds)
What the data shows:
Subtask duration: Linear wall-clock time from subtask start to finish (created_at → updated_at)
- Excludes subtasks in
createdorwaitingstatus (not yet started) - Includes only
running,finished,failedsubtasks - NOT the sum of toolcalls (avoids double-counting nested agent calls)
- Excludes subtasks in
Task duration: Total execution time including:
- Generator Agent execution (runs before subtasks)
- All subtasks execution (with overlap compensation for batch-created subtasks)
- Refiner Agent execution(s) (runs between subtasks)
Flow duration: Complete flow execution time including:
- All tasks duration (generator + subtasks + refiner)
- Flow-level toolcalls (e.g., Assistant toolcalls without task/subtask binding)
- EXCLUDES Assistant msgchain lifetime (only active toolcall time counted)
Why not sum of toolcalls:
- Primary Agent calls Coder Agent → Coder calls Terminal
- Summing toolcalls would count Terminal execution twice (in Coder time AND separately)
- Linear time gives accurate wall-clock duration
Overlap compensation:
- Generator creates multiple subtasks simultaneously (same created_at)
- Subtasks execute sequentially, not in parallel
- System compensates for this overlap to show real execution time
Toolcalls Count Metrics (totalCount, totalToolcallsCount)
What the data shows:
- Only completed toolcalls (status =
finishedorfailed) - Excludes
received(not started) andrunning(in progress) toolcalls - Represents actual completed operations, not attempted or pending
Use this for:
- Counting successful/failed operations
- Calculating success rate:
finished_count / (finished_count + failed_count) - Understanding actual work performed
Agent Type Breakdown
Important distinction:
primary_agent: Main orchestrator for subtasksgenerator: Creates subtask plans (runs once per task, before subtasks)refiner: Updates subtask plans (runs between subtasks, can run multiple times)reporter: Generates final task reportsassistant: Interactive mode (can have long idle periods)- Specialist agents:
coder,pentester,installer,searcher,memorist,adviser
Usage interpretation:
- High
generatorusage = complex task decomposition - High
refinerusage = adaptive planning (many plan adjustments) - High specialist usage = delegated work (Primary Agent using team members)
Agent vs Non-Agent Tools (isAgent field)
The isAgent field in FunctionToolcallsStats categorizes tools into two types:
Agent Tools (isAgent: true):
- Delegation to AI agents (e.g.,
coder,pentester,searcher) - Store results from agent execution (e.g.,
coder_result,hack_result) - Execution time represents delegation overhead + agent decision time
- Does NOT include nested tool calls made by the agent
- Examples:
coder,pentester,installer,searcher,memorist,adviser
Non-Agent Tools (isAgent: false):
- Direct tool execution (e.g.,
terminal,browser,file) - Search engines (e.g.,
google,duckduckgo,tavily,firecrawl,sploitus,searxng) - Vector database operations (e.g.,
search_in_memory,store_guide) - Environment operations (e.g.,
terminal,file) - Execution time represents actual operation duration
- Examples:
terminal,browser,file,google,search_in_memory
Use this distinction for:
- Analyzing delegation overhead vs direct execution time
- Identifying which agents are most frequently used
- Optimizing agent selection strategies
- Understanding the balance between AI-driven and direct operations
- Dashboard visualizations: color-code or separate agent vs non-agent tools
Common Fragments
Define these fragments once and reuse across queries:
fragment UsageStats on UsageStats {
totalUsageIn
totalUsageOut
totalUsageCacheIn
totalUsageCacheOut
totalUsageCostIn
totalUsageCostOut
}
fragment ToolcallsStats on ToolcallsStats {
totalCount
totalDurationSeconds
}
fragment FlowsStats on FlowsStats {
totalFlowsCount
totalTasksCount
totalSubtasksCount
totalAssistantsCount
}
fragment FlowStats on FlowStats {
totalTasksCount
totalSubtasksCount
totalAssistantsCount
}
fragment FunctionToolcallsStats on FunctionToolcallsStats {
functionName
isAgent
totalCount
totalDurationSeconds
avgDurationSeconds
}
fragment SubtaskExecutionStats on SubtaskExecutionStats {
subtaskId
subtaskTitle
totalDurationSeconds
totalToolcallsCount
}
fragment TaskExecutionStats on TaskExecutionStats {
taskId
taskTitle
totalDurationSeconds
totalToolcallsCount
subtasks {
...SubtaskExecutionStats
}
}
fragment FlowExecutionStats on FlowExecutionStats {
flowId
flowTitle
totalDurationSeconds
totalToolcallsCount
totalAssistantsCount
tasks {
...TaskExecutionStats
}
}
Use Case 1: Time-Filtered Dashboard
Purpose: Display system activity over time periods (week/month/quarter) with day-by-day breakdowns and execution metrics.
Query
query TimeDashboard($period: UsageStatsPeriod!) {
# LLM token usage over time
usageStatsByPeriod(period: $period) {
date
stats { ...UsageStats }
}
# Toolcalls activity over time
toolcallsStatsByPeriod(period: $period) {
date
stats { ...ToolcallsStats }
}
# Flows/tasks/subtasks created over time
flowsStatsByPeriod(period: $period) {
date
stats { ...FlowsStats }
}
# Flow execution times with full hierarchy
flowsExecutionStatsByPeriod(period: $period) {
...FlowExecutionStats
}
}
Variables:
{ "period": "week" } // or "month", "quarter"
Data Interpretation
Daily Usage Trends:
usageStatsByPeriod: Track token consumption patterns (input/output, cache hits, costs)- Chart: Line graph showing daily token usage and costs
- Insights: Identify peak usage days, cost optimization opportunities
Toolcalls Performance:
toolcallsStatsByPeriod: Monitor tool execution frequency and duration- Count: Only completed operations (finished/failed), excludes pending/running
- Duration: Sum of individual toolcall execution times (created_at → updated_at)
- Chart: Dual-axis chart (count bars + duration line)
- Insights: Detect performance degradation, identify bottlenecks
- Note: Duration here IS sum of toolcalls (unlike execution stats which use wall-clock)
Flow Activity:
flowsStatsByPeriod: Track system load (flows/tasks/subtasks/assistants created per day)- Chart: Stacked bar chart showing hierarchy depth
- Insights: Understand workload distribution, capacity planning, assistant usage patterns
Execution Breakdown:
flowsExecutionStatsByPeriod: Hierarchical view of execution times- Chart: Treemap or sunburst showing time distribution across flows → tasks → subtasks
- Important: Duration is wall-clock time, not sum of toolcalls
- Subtask: Linear time from start to finish (excludes created/waiting subtasks)
- Task: Subtasks + Generator + Refiner agents
- Flow: Tasks + Flow-level toolcalls (Assistant active time, NOT lifetime)
- Count: Only completed toolcalls (finished/failed status)
- Insights: Identify slow flows/tasks, optimize critical paths
- Note: Batch-created subtasks have overlap compensation applied
Cross-Correlations:
- Compare token usage vs execution time (efficiency metric)
- Correlate toolcall count with flow complexity
- Cost per flow:
totalUsageCost / flowsCount - Average toolcalls per task:
totalToolcallsCount / totalTasksCount
Use Case 2: Overall System Statistics
Purpose: Get comprehensive system-wide metrics without time filters.
Query
query SystemOverview {
# Total LLM usage
usageStatsTotal { ...UsageStats }
# Usage breakdown by provider
usageStatsByProvider {
provider
stats { ...UsageStats }
}
# Usage breakdown by model
usageStatsByModel {
model
provider
stats { ...UsageStats }
}
# Usage breakdown by agent type
usageStatsByAgentType {
agentType
stats { ...UsageStats }
}
# Total toolcalls stats
toolcallsStatsTotal { ...ToolcallsStats }
# Toolcalls breakdown by function
toolcallsStatsByFunction {
...FunctionToolcallsStats
}
# Total flows/tasks/subtasks
flowsStatsTotal { ...FlowsStats }
}
Data Interpretation
Token Economics:
usageStatsTotal: Overall system cost and token consumption- Metrics: Total spend, cache efficiency (
cacheIn / (cacheIn + usageIn)) - Dashboard KPI: Display as headline metrics
Provider Distribution:
usageStatsByProvider: Cost and usage per LLM provider- Chart: Pie chart showing provider share by cost
- Insights: Identify most/least cost-effective providers
Model Efficiency:
usageStatsByModel: Granular per-model breakdown- Chart: Table sorted by cost with usage metrics
- Metrics:
- Cost per token:
(costIn + costOut) / (usageIn + usageOut) - Cache hit rate per model
- Cost per token:
- Insights: Choose optimal models for different tasks
Agent Performance:
usageStatsByAgentType: Resource consumption by agent role- Chart: Horizontal bar chart showing usage by agent
- Insights: Understand which agents consume most resources
Tool Usage Patterns:
toolcallsStatsByFunction: Top tool functions by usage and duration- Agent Classification: The
isAgentfield indicates if the function is an agent delegation tool- Agent tools (e.g.,
coder,pentester) show their own execution time - Does NOT include time of nested calls they make
- Example:
codertoolcall = time for Coder Agent to decide + delegate, not terminal commands - Non-agent tools (e.g.,
terminal,browser) show direct execution time
- Agent tools (e.g.,
- Chart: Table with sortable columns (count, duration, average, agent type)
- Metrics:
- Slowest tools: Sort by
avgDurationSeconds - Most used tools: Sort by
totalCount(only completed) - Time sinks: Sort by
totalDurationSeconds - Agent vs non-agent breakdown: Filter by
isAgent
- Slowest tools: Sort by
- Insights: Optimize frequently-used slow tools, identify unused tools, distinguish delegation overhead from direct execution
System Scale:
flowsStatsTotal: Total entities in system- Metrics:
- Tasks per flow:
totalTasksCount / totalFlowsCount - Subtasks per task:
totalSubtasksCount / totalTasksCount - Assistants per flow:
totalAssistantsCount / totalFlowsCount
- Tasks per flow:
- Insights: Understand average flow complexity and assistant usage
Use Case 3: Flow-Specific Dashboard
Purpose: Deep dive into a specific flow's metrics.
Query
query FlowAnalytics($flowId: ID!) {
# Basic flow info
flow(flowId: $flowId) {
id
title
status
createdAt
updatedAt
}
# LLM usage for this flow
usageStatsByFlow(flowId: $flowId) { ...UsageStats }
# Agent usage breakdown
usageStatsByAgentTypeForFlow(flowId: $flowId) {
agentType
stats { ...UsageStats }
}
# Toolcalls stats
toolcallsStatsByFlow(flowId: $flowId) { ...ToolcallsStats }
# Tool function breakdown
toolcallsStatsByFunctionForFlow(flowId: $flowId) {
...FunctionToolcallsStats
}
# Example: Separate agent and non-agent tools
# Filter client-side: stats.filter(s => s.isAgent) for agent tools
# Filter client-side: stats.filter(s => !s.isAgent) for direct execution tools
# Flow structure stats
flowStatsByFlow(flowId: $flowId) { ...FlowStats }
}
Variables:
{ "flowId": "123" }
Data Interpretation
Flow Performance Summary:
usageStatsByFlow: Total LLM costs for this flow (all msgchains)toolcallsStatsByFlow: Execution metrics (duration, toolcall count)- Count: Only completed toolcalls (finished/failed)
- Duration: Sum of individual toolcall times
- KPIs:
- Cost per toolcall:
totalUsageCost / totalToolcallsCount - Average toolcall duration:
totalDurationSeconds / totalCount - Cost efficiency: tokens per second
- Cost per toolcall:
- Note: For actual flow wall-clock time, use
flowsExecutionStatsByPeriod
Agent Activity:
usageStatsByAgentTypeForFlow: Which agents were most active- Chart: Donut chart showing token distribution by agent
- Insights: Understand which agents drive flow execution
Tool Usage Analysis:
toolcallsStatsByFunctionForFlow: Detailed breakdown per tool- Chart: Bubble chart (x=count, y=avgDuration, size=totalDuration, color=isAgent)
- Metrics:
- Identify bottleneck tools (high avgDuration)
- Find frequently-used tools (high totalCount)
- Calculate tool efficiency scores
- Separate agent delegation overhead from direct execution time using
isAgentfield
Flow Complexity:
flowStatsByFlow: Structural metrics- Metrics:
- Subtasks per task:
totalSubtasksCount / totalTasksCount - Toolcalls per task:
toolcallsCount / tasksCount - Assistants count:
totalAssistantsCount
- Subtasks per task:
- Insights: Compare against average complexity to identify outliers, track assistant usage per flow
Cross-Flow Comparisons: Fetch multiple flows and compare:
- Cost efficiency (cost per task)
- Execution speed (duration per task)
- Tool utilization patterns
- Agent composition differences
Understanding Metric Differences
Execution Stats vs Toolcalls Stats
These two metric types measure different aspects of system performance:
Execution Stats (flowsExecutionStatsByPeriod):
- Purpose: Measure real wall-clock time for flows/tasks/subtasks
- Duration calculation: Linear time (start → end timestamp)
- What it shows: How long a flow/task/subtask actually ran
- Use for: Performance analysis, SLA monitoring, user-facing progress
- Example: Subtask ran for 100 seconds (even if it made 10 toolcalls)
Toolcalls Stats (toolcallsStatsByPeriod, toolcallsStatsByFunction):
- Purpose: Measure individual tool execution metrics
- Duration calculation: Sum of toolcall durations (each toolcall's created_at → updated_at)
- What it shows: Aggregate time spent in specific tools
- Use for: Tool optimization, identifying slow functions, resource attribution
- Example: 50 terminal toolcalls totaling 300 seconds
Key difference:
Flow execution time = 100 seconds (wall-clock)
Toolcalls total time = 150 seconds (sum of all toolcalls)
Why different?
- Flow time is LINEAR (real time elapsed)
- Toolcalls time INCLUDES OVERLAPS (nested agent calls counted in parent time)
When to use which:
- User wants to know "how long did my pentest take?" → Use Execution Stats
- Developer wants to optimize slow tools → Use Toolcalls Stats
- Manager wants to see system utilization → Use Toolcalls Stats
- SLA monitoring → Use Execution Stats
Subtask Status and Inclusion
Included in metrics (counted):
running: Currently executing (duration = created_at → now)finished: Completed successfully (duration = created_at → updated_at)failed: Terminated with error (duration = created_at → updated_at)
Excluded from metrics (NOT counted):
created: Generated but not started yet (duration = 0)waiting: Paused for user input (duration = 0)
Why this matters:
- Generator creates 10 subtasks at once, only 1 starts executing
- You'll see 1 subtask in stats, not 10
- As more subtasks execute, they appear in metrics
- Final stats include only executed subtasks
Assistant Time Accounting
Assistant msgchains:
- Can exist for days/weeks (created once, used intermittently)
- Their lifetime is NOT counted in flow duration
- Only their active toolcalls are counted
Example:
Assistant created: Monday 9 AM
User asks question: Monday 10 AM (toolcall 1: 5 seconds)
User asks question: Tuesday 3 PM (toolcall 2: 3 seconds)
Flow duration contribution: 8 seconds (5 + 3)
NOT: 30+ hours (Monday to Tuesday)
Generator and Refiner Inclusion
Task execution includes:
Generator Agent (runs once at task start):
- Creates initial subtask plan
- Has msgchain with task_id, NO subtask_id
- Time is added to task duration
Subtasks (execute sequentially):
- Each has Primary Agent msgchain
- Individual durations with overlap compensation
Refiner Agent (runs between subtasks):
- Updates subtask plan based on results
- Can run multiple times per task
- Each run has msgchain with task_id, NO subtask_id
- Total refiner time added to task duration
Example task timeline:
Generator (5s) → Subtask 1 (10s) → Refiner (3s) → Subtask 2 (15s) → Refiner (2s) → Subtask 3 (8s)
Task duration = 5 + 10 + 3 + 15 + 2 + 8 = 43 seconds
Advanced Analytics Patterns
1. Cost Optimization Dashboard
Combine queries to identify cost reduction opportunities:
query CostOptimization {
usageStatsByModel {
model
provider
stats { ...UsageStats }
}
toolcallsStatsByFunction {
functionName
isAgent
totalCount
avgDurationSeconds
}
}
Analysis:
- Expensive models with high usage → candidates for cheaper alternatives
- Slow tools called frequently → optimization targets
- Calculate ROI per model:
performance_gain / cost_increase
2. Performance Monitoring
Track system responsiveness:
query PerformanceMetrics($period: UsageStatsPeriod!) {
toolcallsStatsByPeriod(period: $period) {
date
stats { ...ToolcallsStats }
}
flowsExecutionStatsByPeriod(period: $period) {
flowTitle
totalDurationSeconds
totalToolcallsCount
}
}
Metrics:
- Average execution time per flow:
totalDurationSeconds / flowsCount - Toolcalls per day trend (detect performance degradation)
- P95/P99 flow durations (for SLA monitoring)
Important for performance analysis:
- Use
flowsExecutionStatsByPeriodfor wall-clock time (what users experience) - Compare with
toolcallsStatsByPeriodto detect overhead (high ratio = optimization needed) - Ratio > 2.0 suggests significant nested agent call overhead
3. Resource Attribution
Understand resource consumption patterns:
query ResourceAttribution {
usageStatsByAgentType {
agentType
stats { ...UsageStats }
}
toolcallsStatsByFunction {
functionName
isAgent
totalDurationSeconds
}
}
Analysis:
- Which agents consume most resources
- Tool time distribution (execution time budget)
- Cost attribution by capability (pentesting vs coding vs searching)
Implementation Tips
Caching Strategy:
- Cache
*StatsTotalqueries (update every 5-10 minutes) - Cache
*StatsByPeriodper period (update hourly) - Real-time for flow-specific queries
Visualization Libraries:
- Time series: Recharts, Chart.js (daily trends)
- Hierarchical: D3 treemap/sunburst (execution breakdown)
- Tables: TanStack Table with sorting/filtering (function stats)
Performance Optimization:
- Use query batching for multiple flows
- Implement pagination for large datasets
- Add loading states for slow queries (execution stats can be heavy)
Data Refresh:
- Overall stats: Manual refresh or 10min polling
- Time-filtered: Auto-refresh on period change
- Flow-specific: Subscribe to flow updates for real-time metrics
Practical Examples
Example 1: Understanding a Slow Flow
Scenario: Flow took 300 seconds but only has 50 toolcalls
Investigation:
query InvestigateSlowFlow($flowId: ID!) {
flowsExecutionStatsByPeriod(period: week) {
flowId
flowTitle
totalDurationSeconds
totalToolcallsCount
tasks {
taskTitle
totalDurationSeconds
totalToolcallsCount
subtasks {
subtaskTitle
totalDurationSeconds
totalToolcallsCount
}
}
}
}
Analysis:
- Check task breakdown: Which task took longest?
- Check subtask breakdown: Which subtasks in slow task took longest?
- Check toolcall count: High duration + low count = slow individual operations
- Check toolcalls by function: Which tools are slow?
Common causes:
- Long-running terminal commands (compilation, scanning)
- Slow search engine responses (tavily, perplexity)
- Large file operations
- Network latency for browser tool
Example 2: Cost Attribution
Scenario: Need to understand cost per capability
Query:
query CostAttribution {
usageStatsByAgentType {
agentType
stats {
totalUsageCostIn
totalUsageCostOut
}
}
toolcallsStatsByFunction {
functionName
totalCount
avgDurationSeconds
}
}
Interpretation:
primary_agent: Orchestration overheadgenerator: Planning cost (usually low, runs once per task)refiner: Replanning cost (high value = many adjustments)pentester: Security testing operationscoder: Development worksearcher: Research and information gathering
Cost optimization:
- High generator cost → simplify task descriptions
- High refiner cost → improve initial planning (fewer adjustments needed)
- High searcher cost → use memory tools more (cheaper than web search)
Example 3: Detecting Inefficient Flows
Red flags:
Flow A: 100 toolcalls, 500 seconds → 5s per toolcall (GOOD)
Flow B: 100 toolcalls, 5000 seconds → 50s per toolcall (INVESTIGATE)
Check:
query DetectInefficiency {
toolcallsStatsByFunction {
functionName
isAgent
totalCount
avgDurationSeconds
}
}
Common issues:
- High
terminalavg → long commands, consider timeout tuning - High
browseravg → slow websites, consider caching - High
tavily/perplexityavg → deep research, optimize queries - High agent tools (
coder,pentester) avg → complex delegated work
Example 4: Understanding Task Complexity
Scenario: Why do some tasks take so long?
Metrics to check:
Task complexity indicators:
- High subtask count → decomposition into many steps
- High refiner calls → adaptive planning (many plan changes)
- High generator time → complex initial planning
- Low subtask count + high duration → individual subtasks are slow
Query:
query TaskComplexity($flowId: ID!) {
flowsExecutionStatsByPeriod(period: week) {
flowId
tasks {
taskTitle
totalDurationSeconds
totalToolcallsCount
subtasks {
subtaskTitle
totalDurationSeconds
}
}
}
usageStatsByAgentTypeForFlow(flowId: $flowId) {
agentType
stats { totalUsageIn totalUsageOut }
}
}
Analysis patterns:
- Many subtasks + low generator usage → simple decomposition
- Many subtasks + high generator usage → complex planning
- High refiner usage → dynamic adaptation (plan changed during execution)
- Few subtasks + high duration → intensive work per subtask
Data Quality Guarantees
Accuracy
Time measurements:
- ✅ No double-counting of nested agent calls
- ✅ Overlap compensation for batch-created subtasks
- ✅ Excludes non-started subtasks (created/waiting)
- ✅ Includes all execution phases (generator, subtasks, refiner)
Count measurements:
- ✅ Only completed operations (finished/failed)
- ✅ Excludes pending (received) and in-progress (running)
- ✅ Consistent across all queries
Cost measurements:
- ✅ Aggregated from msgchains (source of truth for LLM calls)
- ✅ Includes cache hits and misses
- ✅ Separate input/output costs
Known Limitations
Current limitations:
Historical data: Subtasks created before this update may have:
- Missing primary_agent subtask_id (known bug, now fixed)
- Use linear time fallback (still accurate)
Running entities: Duration calculated as
created_at → now- Updates as entity continues execution
- Final duration set when status changes to finished/failed
Assistant lifetime: Long-lived assistants
- Only active toolcall time counted (correct behavior)
- Msgchain lifetime NOT included in flow duration
Edge cases handled:
- ✅ Batch-created subtasks (overlap compensation)
- ✅ Missing primary_agent msgchain (fallback to linear time)
- ✅ Subtasks in waiting status (excluded from duration)
- ✅ Flow-level toolcalls without task binding (counted separately)
- ✅ Generator/Refiner without subtask binding (counted in task duration)
Troubleshooting
"My flow shows 0 duration but has toolcalls"
Possible causes:
- All subtasks are in
createdorwaitingstatus - Flow just started (no completed subtasks yet)
Check:
query CheckFlowStatus($flowId: ID!) {
flow(flowId: $flowId) {
status
}
tasks(flowId: $flowId) {
status
subtasks {
status
}
}
}
"Task duration seems low compared to subtasks"
This is normal if:
- Subtasks were created in batch (overlap compensation applied)
- Example: 3 subtasks created at 10:00:00, finished at 10:00:10, 10:00:20, 10:00:30
- Naive sum: 10 + 20 + 30 = 60 seconds
- Actual time: 30 seconds (overlap compensated)
"Toolcalls duration > execution duration"
This is expected:
- Toolcalls duration: Sum of all toolcall times (includes nested calls)
- Execution duration: Wall-clock time (linear)
- Nested agent calls cause toolcalls > execution
- Example: Primary Agent (100s) calls Coder (30s), toolcalls = 130s, execution = 100s
"Count doesn't match my expectations"
Remember:
- Only completed toolcalls counted (finished/failed)
- Received (pending) and running (in-progress) excluded
- Check toolcall status distribution if counts seem low
Best Practices
Dashboard Design
Real-time monitoring:
- Use execution stats for user-facing progress
- Show flow/task/subtask hierarchy with durations
- Update as status changes (subscribe to updates)
Historical analysis:
- Use toolcalls stats for tool performance
- Use usage stats for cost tracking
- Group by period for trend analysis
Cost optimization:
- Compare cost per agent type
- Identify expensive models with low value
- Track cache hit rates for efficiency
Query Optimization
For large datasets:
# Don't fetch full hierarchy if not needed
query LightweightStats {
toolcallsStatsTotal { totalCount totalDurationSeconds }
usageStatsTotal { totalUsageCostIn totalUsageCostOut }
}
# Instead of:
query HeavyStats($period: UsageStatsPeriod!) {
flowsExecutionStatsByPeriod(period: $period) {
# Full hierarchy - expensive for many flows
tasks { subtasks { ... } }
}
}
Batch requests:
query BatchedAnalytics {
# Fetch multiple metrics in one request
usageStatsTotal { ... }
toolcallsStatsTotal { ... }
flowsStatsTotal { ... }
}
Data Refresh Strategy
Real-time (WebSocket subscriptions)
- Flow status changes
- Task/subtask creation
- Toolcall completion
- Use for: Live flow monitoring
Polling (every 1-5 minutes)
- Execution stats for running flows
- Toolcalls stats for active periods
- Use for: Dashboard auto-refresh
Cached (refresh every 10-30 minutes)
- Historical period stats
- Total stats (system-wide)
- Provider/model breakdowns
- Use for: Reports and analytics
On-demand (user action)
- Flow-specific deep dives
- Custom period queries
- Export operations
- Use for: Detailed investigation
Migration Notes
Duration Calculation Changes (20260129_120000)
Previous behavior:
- Durations calculated on-the-fly in SQL queries:
EXTRACT(EPOCH FROM (updated_at - created_at)) - Slower query performance due to real-time calculations
- Required complex SQL expressions in every analytics query
New behavior (improved performance):
- Pre-calculated
duration_secondscolumns intoolcallsandmsgchainstables - Duration calculated once during migration for existing records
- Analytics queries use simple column references:
tc.duration_seconds - Significant performance improvement for dashboard queries (simpler execution plans)
Migration steps:
- Add
duration_seconds DOUBLE PRECISION NULLcolumn - Calculate duration for existing records:
EXTRACT(EPOCH FROM (updated_at - created_at)) - Set remaining NULL values to 0.0
- Alter column to
NOT NULL - Set default value to 0.0 for future records
For developers:
- All SQL queries in
backend/sqlc/models/toolcalls.sqlupdated to useduration_seconds - Update queries accept
duration_deltaparameter from Go code - Updated queries with new signatures:
UpdateToolcallStatus(status, duration_delta, id): adds delta to duration when status changesUpdateToolcallFinishedResult(result, duration_delta, id): adds final delta when toolcall finishesUpdateToolcallFailedResult(result, duration_delta, id): adds final delta when toolcall failsUpdateMsgChain(chain, duration_delta, id): adds delta when chain is updatedUpdateMsgChainUsage(usage_in, usage_out, usage_cache_in, usage_cache_out, usage_cost_in, usage_cost_out, duration_delta, id): adds delta when usage updated
- Backend code must calculate
duration_deltain seconds before calling update - Use
time.Since(startTime).Seconds()or similar for accurate measurements - SQL simply adds the delta:
duration_seconds = duration_seconds + $duration_delta