Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

fix(tier3): count edits, searches and MCP calls as productive and leave planning calls out of tool efficiency (#428)

Narendran Raghavan44ee844dbde9
diff --git a/CHANGELOG.md b/CHANGELOG.md
index da8b7a29..a96ae992 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -291,6 +291,31 @@ All notable changes to SkillEvaluator are documented in this file.
2 can wait for readiness while authored image healthchecks remain suppressed.
Keep custom service healthchecks and their readiness dependencies intact.
+- Tier 3 tool efficiency counts writing, editing and patching files, scoped
+ native searches and listings (`Grep`, `Glob`, `LS`, OpenCode `list`, Codex
+ `list_dir` and `grep_files`) and MCP tools named `mcp__server__tool` (Claude
+ Code) as productive. An MCP filesystem search or listing whose `path` or
+ `dir_path` is `/` is a root search and counts as waste. Most of these
+ counted as wasted before, so a with-skill trial that read the skill, edited
+ four files and ran the tests scored tool efficiency 0.33. Codex and OpenCode
+ MCP calls, which are recorded under the bare tool name or as `server_tool`,
+ are not recognized as MCP calls yet. Codex names with a namespace, such as
+ `functions.apply_patch`, are classified by the bare name. Planning and
+ bookkeeping calls (`TodoWrite`, `update_plan`, `TaskCreate` and
+ `TaskUpdate`, background-shell polling such as `BashOutput`, and sub-agent
+ calls) are left out of the score, and the reason says how many were not
+ counted. A trial with only such calls scores 0.0, so a trial whose only
+ calls were `BashOutput` or `KillShell` drops from 1.0 to 0.0. Polling a
+ background shell no longer counts as running a command, while a command
+ typed into a Codex `write_stdin` session or run by `Monitor` is classified
+ like any other command. Help, version and install words are matched in
+ commands only, so a file path or patch text that mentions `pip install` is
+ not waste. Efficiency scores rise for write-, edit-, patch-, search- and
+ MCP-heavy runs. Root searches in commands recorded as an argument list, such
+ as Codex `["bash", "-lc", "find / -name x"]`, now count as waste, and so do
+ help, version and install commands recorded that way, like the same command
+ written as text.
+
### Documentation
- The 0.8.0 entry for refused `claude-code` attempts said the arm reports its
diff --git a/docs/reports.mdx b/docs/reports.mdx
index db384028..2897b0ee 100644
--- a/docs/reports.mdx
+++ b/docs/reports.mdx
@@ -711,12 +711,49 @@ dimensions, each answering one question:
| Effectiveness | Did the skill help complete the task? | `goal_accuracy` from an LLM judge; `behavior_check` from its judge and directly evaluated declared checks (checks-only cases make no judge call) |
| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency`, from deterministic trajectory checks |
+`skill_efficiency` averages skill routing with tool efficiency, the share of
+scored tool calls that did productive work. For Claude Code, Codex and OpenCode
+tool names, these calls are productive: reading, searching, listing, writing,
+editing or patching files (`Write`, `Edit`, `MultiEdit`, `apply_patch`, `LS`,
+OpenCode `list`, Codex `list_dir` and `grep_files`, and similar), running
+commands and activating a skill. MCP tools named `mcp__server__tool`, as Claude
+Code records them, are productive too. Codex records an MCP call under the bare
+tool name and OpenCode as `server_tool`, so these calls are classified by their
+name like any other tool, and most such names count as wasted. A Codex name
+with a namespace, such as `functions.apply_patch`, is classified by the part
+after the last dot. Help, version and install commands are wasted, whether
+written as text or recorded as an argument list, as is any tool outside these
+classes. Only commands are checked for help, version and install words, so a
+file path or patch text that mentions `pip install` is not waste.
+
+Planning and bookkeeping calls are neutral and left out of the score:
+`TodoWrite`, `update_plan`, task lists (`TaskCreate`, `TaskUpdate`, `TaskList`,
+`TaskGet`), `EnterPlanMode` and `ExitPlanMode`, background-shell polling such
+as `BashOutput` and `KillShell`, `TaskStop`, `ToolSearch`, and sub-agent calls
+such as `Task` and Codex `spawn_agent`, `wait` or `wait_agent`. Codex
+`write_stdin` with no input, or only Enter or Ctrl-C, polls its session and is
+neutral, as is a `Monitor` watch with no command. A command typed with
+`write_stdin` or run by `Monitor` is classified like any other command. The
+reason names how many calls were not counted. A trial whose only calls are
+neutral did no work: it scores tool efficiency 0.0, so its `skill_efficiency`
+is 0.0, the same as a trial with no tool calls.
+
Tool-efficiency checks classify explicit searches rooted at `/` as wasted work,
-whether or not the command names the expected skill or script. Scoped searches
-and `/` used as a search pattern retain their existing classification. This rule
-recognizes common search commands, including ones with redirections or wrapped in
-`sh -c`; it does not interpret expansions, comments, here-documents or other
-dynamically constructed shell programs.
+whether or not the command names the expected skill or script. This includes
+native `Grep`, `Glob`, `LS`, OpenCode `list`, Codex `list_dir` and `grep_files`
+calls on `/`, MCP filesystem searches and listings whose `path` or `dir_path`
+is `/`, and commands recorded as an argument list, such as Codex
+`["bash", "-lc", "find / -name x"]`, which are read as the command they run.
+An MCP call counts as a filesystem search or listing when its tool name
+contains `search`, `list`, `tree`, `find`, `glob`, `grep` or `directory`, or
+when its server name contains `filesystem`. Other MCP tools use `path` for
+other things: `/` in a GitHub `get_file_contents` call is the repository root,
+and that call is productive. Scoped searches and `/` used as a search pattern
+are productive. This rule recognizes common search commands, including ones with
+redirections or wrapped in `sh -c`; it does not interpret expansions, comments,
+here-documents or other dynamically constructed shell programs. An argument
+list runs without a shell, so `$`, backticks and `#` in its own words are
+literal.
Every evaluator and dimension score is on a 0.0 to 1.0 scale. The generated
diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdx
index 20d1a758..bb188c17 100644
--- a/docs/tier3-live-evaluation.mdx
+++ b/docs/tier3-live-evaluation.mdx
@@ -859,6 +859,17 @@ grades each trial after the agent finishes. `security`, `skill_execution`, and
checks; a checks-only case makes no behavior judge call. `custom_only` runs
none of them.
+`skill_efficiency` averages skill routing with tool efficiency, the share of
+scored tool calls that did productive work. For Claude Code, Codex and OpenCode
+tool names, reading, searching, listing, writing, editing or patching files,
+running commands and activating a skill are productive, and so are MCP tools
+named `mcp__server__tool` (Claude Code). Codex and OpenCode MCP calls are not
+recognized as MCP calls yet, so most of them count as wasted. Help, version and
+install commands and searches rooted at `/` are wasted. Planning and
+bookkeeping calls, such as `TodoWrite`, `update_plan`, task lists,
+background-shell polling and sub-agent calls, are not counted, and a trial with
+only such calls scores 0.0. [Reports](/skills/skillevaluator/reports) lists the tool classes.
+
The three LLM judges return verdicts only, and the verifier computes the
scores. A `score` field in a judge reply is ignored.
diff --git a/src/skillevaluator/tier3/eval_core/checks.py b/src/skillevaluator/tier3/eval_core/checks.py
index 257152bd..de306fbc 100644
--- a/src/skillevaluator/tier3/eval_core/checks.py
+++ b/src/skillevaluator/tier3/eval_core/checks.py
@@ -5201,13 +5201,15 @@ def _mask_non_assignment_equals(text: str) -> str:
return "".join(out)
-def _tool_searches_root(action: str, args: dict, text: str, depth: int = 0) -> bool:
+def _tool_searches_root(action: str, args: dict, text: str, depth: int = 0, *, argv: bool = False) -> bool:
"""Recognize explicit root search operands, not '/' inside a pattern.
This is bounded top-level command classification. It does not execute or
- interpret dynamically constructed shell programs.
+ interpret dynamically constructed shell programs. ``argv`` marks text
+ joined from an argument list, which runs without a shell: its own words
+ are literal, while a ``sh -c`` payload inside it is still shell text.
"""
- if action in {"grep", "glob", "grep_files", "list_dir", "ls"}:
+ if action in {"grep", "glob", "grep_files", "list_dir", "ls", "list"}:
for key in ("path", "dir_path"):
value = args.get(key)
if isinstance(value, str) and _search_path_is_root(value.strip()):
@@ -5222,7 +5224,7 @@ def _tool_searches_root(action: str, args: dict, text: str, depth: int = 0) -> b
# Without a shell AST, comments, expansions, here-documents and process
# substitutions cannot be classified reliably. Keep their prior score
# instead of inventing a path.
- if any(char in text for char in "$`#"):
+ if not argv and any(char in text for char in "$`#"):
return False
tokens = _shell_tokens(text)
if any(_is_heredoc_redirect(token) or token in {"<(", ">("} for token in tokens):
@@ -5414,12 +5416,175 @@ def _tool_searches_root(action: str, args: dict, text: str, depth: int = 0) -> b
return False
+# Tool productivity classes for check_tool_efficiency. Neutral calls (planning,
+# background-shell control, tool discovery and sub-agents) are left out of the
+# score. Reading, searching, writing, patching, running commands and MCP tools
+# count as productive work.
+_NEUTRAL_TOOL_NAMES = frozenset(
+ {
+ "todowrite",
+ "todoread",
+ "update_plan",
+ "bashoutput",
+ "killshell",
+ "killbash",
+ "kill_shell",
+ "taskstop",
+ "taskoutput",
+ "toolsearch",
+ "task",
+ "agent",
+ # Claude Code task lists and plan mode.
+ "taskcreate",
+ "taskget",
+ "tasklist",
+ "taskupdate",
+ "enterplanmode",
+ "exitplanmode",
+ # Codex multi-agent tools.
+ "spawn_agent",
+ "wait",
+ "wait_agent",
+ "close_agent",
+ "send_input",
+ "resume_agent",
+ }
+)
+_PRODUCTIVE_TOOL_NAMES = frozenset(
+ {
+ "skill",
+ "ls",
+ # OpenCode's directory listing.
+ "list",
+ "list_dir",
+ "grep",
+ "grep_files",
+ "glob",
+ "write",
+ "edit",
+ "multiedit",
+ "notebookedit",
+ "write_file",
+ "edit_file",
+ "apply_patch",
+ "applypatch",
+ }
+)
+# Tools that send text to a running shell session (Codex write_stdin) or start
+# a background watch (Claude Code Monitor), and the argument with that text.
+# Without text they only poll or watch and are neutral; with text they run it,
+# and it is classified like any other command.
+_SESSION_INPUT_TOOL_ARGS = {"write_stdin": "chars", "monitor": "command"}
+# Words in an MCP tool's own name that mark a filesystem search or listing.
+_MCP_FILESYSTEM_SEARCH_WORDS = ("search", "list", "tree", "find", "glob", "grep", "directory")
+
+
+def _tool_efficiency_args(tool_call: dict[str, Any]) -> dict[str, Any]:
+ args = tool_call.get("action_input")
+ return args if isinstance(args, dict) else {}
+
+
+def _tool_efficiency_target(name: str, args: dict[str, Any]) -> Any:
+ """The command, code or path a tool call acts on, as recorded."""
+ if name in _SESSION_INPUT_TOOL_ARGS:
+ return args.get(_SESSION_INPUT_TOOL_ARGS[name], "")
+ return (
+ args.get("command", "")
+ or args.get("cmd", "")
+ or args.get("code", "")
+ or args.get("file_path", "")
+ or args.get("path", "")
+ or args.get("raw", "")
+ )
+
+
+def _is_argv(target: Any) -> bool:
+ return isinstance(target, list) and bool(target) and all(isinstance(item, str) for item in target)
+
+
+def _tool_efficiency_text(target: Any) -> str:
+ """Shell text for a tool call target.
+
+ Codex records some shell calls as an argv list such as
+ ``["bash", "-lc", "find / -name x"]``. Joining it the way a shell reads it
+ lets the root-search and expected-script checks see the command instead of
+ a Python list repr.
+ """
+ if _is_argv(target):
+ return shlex.join(target)
+ return str(target)
+
+
+def _tool_efficiency_name(action: Any) -> str:
+ """Lowercased tool name without a Codex namespace such as ``functions.``.
+
+ MCP tool names keep every part, so ``mcp__tasks__task`` stays an MCP call.
+ """
+ name = str(action).lower()
+ return name if name.startswith("mcp__") else name.rsplit(".", 1)[-1]
+
+
+def _session_input_is_text(value: Any) -> bool:
+ """Whether session input types something; Enter or Ctrl-C alone only polls or interrupts."""
+ return isinstance(value, str) and any(char.isprintable() and not char.isspace() for char in value)
+
+
+def _mcp_call_targets_root(name: str, args: dict[str, Any]) -> bool:
+ """Whether an MCP call searches or lists the whole filesystem.
+
+ Only filesystem tools count: a tool of a server whose name contains
+ ``filesystem``, or a tool whose own name says it searches or lists
+ (``_MCP_FILESYSTEM_SEARCH_WORDS``). Other servers use ``path`` for other
+ things, such as a file in a repository, where "/" is the repository root.
+ """
+ if not name.startswith("mcp__"):
+ return False
+ server, _, tool = name[len("mcp__") :].partition("__")
+ if "filesystem" not in server and not any(word in tool for word in _MCP_FILESYSTEM_SEARCH_WORDS):
+ return False
+ return any(
+ isinstance(args.get(key), str) and _search_path_is_root(args[key].strip()) for key in ("path", "dir_path")
+ )
+
+
+def _tool_call_productivity(tc: dict[str, Any], expected_skill: Any, expected_script: Any) -> str:
+ """Classify one tool call as ``productive``, ``wasted`` or ``neutral`` (not scored)."""
+ name = _tool_efficiency_name(tc["action"])
+ if name in _NEUTRAL_TOOL_NAMES:
+ return "neutral"
+ args = _tool_efficiency_args(tc)
+ target = _tool_efficiency_target(name, args)
+ runs_command = _is_execution_action(name)
+ search_action = name
+ if name in _SESSION_INPUT_TOOL_ARGS:
+ if not _session_input_is_text(target):
+ return "neutral"
+ # The text is a command typed into a shell or run as a watch script.
+ runs_command = True
+ search_action = "exec_command"
+ cmd = _tool_efficiency_text(target)
+ # A search rooted at "/" is waste even when it names the expected skill or script.
+ if _tool_searches_root(search_action, args, cmd, argv=_is_argv(target)) or _mcp_call_targets_root(name, args):
+ return "wasted"
+ if ("read" in name and expected_skill and expected_skill in cmd) or (
+ runs_command and expected_script and expected_script in cmd
+ ):
+ return "productive"
+ # Help, version and install commands are waste. Only a command is checked:
+ # the path, content or patch text of any other tool is data, not a command.
+ if runs_command and any(waste in f"{name} {cmd}".lower() for waste in WASTE_INDICATORS):
+ return "wasted"
+ if "read" in name or name in _PRODUCTIVE_TOOL_NAMES or name.startswith("mcp__") or runs_command:
+ return "productive"
+ return "wasted"
+
+
def check_tool_efficiency(
tool_calls: list[dict[str, Any]],
expected_skill: str | None = None,
expected_script: str | None = None,
) -> dict[str, Any]:
- """Measure what fraction of tool calls were productive vs wasted."""
+ """Measure what fraction of scored tool calls were productive (planning calls are not scored)."""
expected_script = normalize_optional_reference(expected_script)
if not tool_calls:
return {"passed": True, "score": 1.0, "reason": "No tool calls", "details": {}}
@@ -5429,55 +5594,37 @@ def check_tool_efficiency(
"Tool efficiency could not be evaluated because a native Codex exec wrapper was unsupported"
)
- productive = 0
- wasted = 0
+ counts = {"productive": 0, "wasted": 0, "neutral": 0}
wasted_details: list[str] = []
for tc in tool_calls:
- action = tc["action"].lower()
- args = tc.get("action_input", {}) if isinstance(tc.get("action_input"), dict) else {}
- cmd = str(
- args.get("command", "")
- or args.get("cmd", "")
- or args.get("code", "")
- or args.get("file_path", "")
- or args.get("path", "")
- or args.get("raw", "")
- )
- full_text = f"{action} {cmd}".lower()
-
- is_productive = False
-
- if _tool_searches_root(action, args, cmd):
- is_productive = False
- elif ("read" in action and expected_skill and expected_skill in cmd) or (
- _is_execution_action(action) and expected_script and expected_script in cmd
- ):
- is_productive = True
- elif any(waste in full_text for waste in WASTE_INDICATORS):
- is_productive = False
- elif "read" in action or _is_execution_action(action) or action.lower() == "skill":
- is_productive = True
-
- if is_productive:
- productive += 1
- else:
- wasted += 1
- wasted_details.append(f"{tc['action']}({cmd[:60]})")
+ kind = _tool_call_productivity(tc, expected_skill, expected_script)
+ counts[kind] += 1
+ if kind == "wasted":
+ label = str(_tool_efficiency_target(_tool_efficiency_name(tc["action"]), _tool_efficiency_args(tc)))
+ wasted_details.append(f"{tc['action']}({label[:60]})")
+ productive, wasted, neutral = counts["productive"], counts["wasted"], counts["neutral"]
total = productive + wasted
- score = productive / total if total > 0 else 1.0
+ details = {
+ "productive": productive,
+ "wasted": wasted,
+ "neutral": neutral,
+ "total": total,
+ "wasted_calls": wasted_details[:5],
+ }
+ not_counted = f"; {neutral} planning or bookkeeping call(s) not counted" if neutral else ""
+ if total == 0:
+ # Only planning or bookkeeping calls: no work was done, so score 0.0.
+ # skill_efficiency then matches a trajectory with no tool calls (0.0).
+ return {"passed": False, "score": 0.0, "reason": f"No scored tool calls{not_counted}", "details": details}
+ score = productive / total
return {
"passed": score >= 0.5,
"score": round(score, 4),
- "reason": f"{productive}/{total} productive calls ({score:.0%})",
- "details": {
- "productive": productive,
- "wasted": wasted,
- "total": total,
- "wasted_calls": wasted_details[:5],
- },
+ "reason": f"{productive}/{total} productive calls ({score:.0%}){not_counted}",
+ "details": details,
}
diff --git a/tests/tier3/test_root_search_efficiency.py b/tests/tier3/test_root_search_efficiency.py
index 057aa026..b6986a95 100644
--- a/tests/tier3/test_root_search_efficiency.py
+++ b/tests/tier3/test_root_search_efficiency.py
@@ -63,11 +63,10 @@ def test_root_search_does_not_gain_productivity_from_expected_name(module, call)
{"action": "Grep", "action_input": {"path": "/workspace", "pattern": "/"}},
],
)
-def test_scoped_searches_patterns_and_ordinary_calls_keep_existing_classification(module, call):
+def test_scoped_searches_patterns_and_ordinary_calls_stay_productive(module, call):
result = module.check_tool_efficiency([call], expected_skill="task", expected_script="task.py")
- # Public native Grep/Glob calls already receive no productivity credit.
- expected = 0.0 if call["action"] in {"Glob", "Grep"} else 1.0
- assert result["score"] == expected, result
+ # Scoped native Grep/Glob searches are productive work, like scoped shell searches.
+ assert result["score"] == 1.0, result
@pytest.mark.parametrize("module", [checks, template], ids=["host", "bundled"])
@@ -177,3 +176,93 @@ def test_scoped_quoted_heredoc_and_redirect_targets_keep_old_credit(module, comm
[{"action": "exec_command", "action_input": {"cmd": command}}], expected_script="task.py"
)
assert result["score"] == 1.0, result
+
+
[email protected]("module", [checks, template], ids=["host", "bundled"])
+ "argv",
+ [
+ ["bash", "-lc", "find / -name task.py"],
+ ["find", "/", "-name", "task.py"],
+ ["sh", "-c", "grep -r task.py / 2>/dev/null"],
+ ["rg", "--files", "/"],
+ ],
+)
+def test_argv_list_root_searches_are_waste(module, argv):
+ # Codex records shell calls as argv lists. The list is read as the command
+ # it runs, not as its Python repr.
+ for key in ("command", "cmd"):
+ result = module.check_tool_efficiency(
+ [{"action": "shell", "action_input": {key: argv}}], expected_skill="task", expected_script="task.py"
+ )
+ assert result["score"] == 0.0, (key, result)
+
+
[email protected]("module", [checks, template], ids=["host", "bundled"])
+ "argv",
+ [
+ ["python", "task.py"],
+ ["bash", "-lc", "python task.py"],
+ ["bash", "-lc", "find /workspace -name task.py"],
+ ["find", ".", "-name", "/"],
+ ["python", "task.py", "--root", "/"],
+ ],
+)
+def test_argv_list_scoped_commands_and_script_runs_stay_productive(module, argv):
+ result = module.check_tool_efficiency(
+ [{"action": "shell", "action_input": {"command": argv}}], expected_skill="task", expected_script="task.py"
+ )
+ assert result["score"] == 1.0, result
+
+
[email protected]("module", [checks, template], ids=["host", "bundled"])
+def test_argv_list_root_search_and_script_run_score_half(module):
+ calls = [
+ {"action": "shell", "action_input": {"command": ["bash", "-lc", "find / -name task.py"]}},
+ {"action": "shell", "action_input": {"command": ["python", "task.py"]}},
+ ]
+ result = module.check_tool_efficiency(calls, expected_skill="task", expected_script="task.py")
+ assert result["score"] == 0.5, result
+
+
[email protected]("module", [checks, template], ids=["host", "bundled"])
+ "argv",
+ [
+ ["grep", "-r", "#TODO", "/"],
+ ["find", "/", "-name", "$HOME"],
+ ["rg", "`task`", "/"],
+ ["env", "LC_ALL=C", "grep", "-r", "#TODO", "/"],
+ ],
+)
+def test_argv_list_words_are_literal(module, argv):
+ # An argv list runs without a shell, so "$", backticks and "#" in its
+ # words are literal and do not hide the root operand.
+ result = module.check_tool_efficiency(
+ [{"action": "shell", "action_input": {"command": argv}}], expected_skill="task", expected_script="task.py"
+ )
+ assert result["score"] == 0.0, result
+
+
[email protected]("module", [checks, template], ids=["host", "bundled"])
[email protected]("argv", [["grep", "-r", "#TODO", "src"], ["find", ".", "-name", "$HOME"]])
+def test_scoped_argv_lists_with_shell_characters_stay_productive(module, argv):
+ result = module.check_tool_efficiency(
+ [{"action": "shell", "action_input": {"command": argv}}], expected_skill="task", expected_script="task.py"
+ )
+ assert result["score"] == 1.0, result
+
+
[email protected]("module", [checks, template], ids=["host", "bundled"])
[email protected]("payload", ["find / -name task.py # locate", "echo $(find / -name task.py)"])
+def test_shell_payload_in_an_argv_list_scores_like_the_same_text(module, payload):
+ # The payload of "bash -lc" is shell text, so the expansion and comment
+ # rule applies to it exactly as to the same command written as text.
+ as_text = module.check_tool_efficiency(
+ [{"action": "exec_command", "action_input": {"cmd": payload}}], expected_skill="task"
+ )
+ as_argv = module.check_tool_efficiency(
+ [{"action": "shell", "action_input": {"command": ["bash", "-lc", payload]}}], expected_skill="task"
+ )
+ assert as_argv["score"] == as_text["score"], (as_argv, as_text)
diff --git a/tests/tier3/test_tool_productivity.py b/tests/tier3/test_tool_productivity.py
new file mode 100644
index 00000000..eed9e1ae
--- /dev/null
+++ b/tests/tier3/test_tool_productivity.py
@@ -0,0 +1,624 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+"""Tool efficiency counts real work, wastes root searches and setup noise, and ignores bookkeeping."""
+
+from __future__ import annotations
+
+import importlib.util
+import json
+from pathlib import Path
+
+import pytest
+
+from skillevaluator.tier3.eval_core import checks
+
+_TEMPLATE_PATH = Path(__file__).resolve().parents[2] / "src/skillevaluator/tier3/harbor/templates/eval.py"
+_SPEC = importlib.util.spec_from_file_location("tool_productivity_template", _TEMPLATE_PATH)
+assert _SPEC is not None and _SPEC.loader is not None
+template = importlib.util.module_from_spec(_SPEC)
+_SPEC.loader.exec_module(template)
+
+_MODULES = [pytest.param(checks, id="host"), pytest.param(template, id="bundled")]
+_SKILL = "release-review"
+
+
+def _call(action: str, **args) -> dict:
+ return {"action": action, "action_input": args}
+
+
+def _classify(module, call: dict, expected_script: str | None = None) -> str:
+ """Classify one call through check_tool_efficiency, which both copies expose."""
+ result = module.check_tool_efficiency([call], _SKILL, expected_script)
+ if result["reason"].startswith("No scored tool calls"):
+ kind = "neutral"
+ else:
+ kind = {1.0: "productive", 0.0: "wasted"}[result["score"]]
+ if module is checks:
+ assert checks._tool_call_productivity(call, _SKILL, expected_script) == kind
+ return kind
+
+
[email protected]("module", _MODULES)
+ "call",
+ [
+ _call("Read", file_path="src/app.py"),
+ _call("Write", file_path="report.md", content="x"),
+ _call("Edit", file_path="src/app.py"),
+ _call("MultiEdit", file_path="src/app.py"),
+ _call("NotebookEdit", notebook_path="a.ipynb"),
+ _call("Grep", pattern="TODO", path="src"),
+ _call("Glob", pattern="**/*.py"),
+ _call("LS", path="src"),
+ _call("list", path="/workspace"),
+ _call("Bash", command="pytest -q"),
+ _call("Skill", skill=_SKILL),
+ _call("apply_patch", input="*** Begin Patch\n*** End Patch"),
+ _call("exec_command", cmd="make test"),
+ _call("shell", command=["bash", "-lc", "ls src"]),
+ _call("list_dir", dir_path="src"),
+ _call("grep_files", pattern="TODO"),
+ _call("mcp__github__get_issue", issue=1),
+ ],
+ ids=lambda call: call["action"],
+)
+def test_real_work_is_productive(module, call):
+ assert _classify(module, call) == "productive"
+
+
[email protected]("module", _MODULES)
+ "action",
+ [
+ "TodoWrite",
+ "TodoRead",
+ "update_plan",
+ "write_stdin",
+ "BashOutput",
+ "KillShell",
+ "KillBash",
+ "kill_shell",
+ "TaskStop",
+ "TaskOutput",
+ "ToolSearch",
+ "Task",
+ "Agent",
+ "TaskCreate",
+ "TaskGet",
+ "TaskList",
+ "TaskUpdate",
+ "EnterPlanMode",
+ "ExitPlanMode",
+ "spawn_agent",
+ "wait",
+ "wait_agent",
+ "close_agent",
+ "send_input",
+ "resume_agent",
+ ],
+)
+def test_bookkeeping_is_neutral(module, action):
+ assert _classify(module, _call(action)) == "neutral"
+
+
[email protected]("module", _MODULES)
+ ("call", "kind"),
+ [
+ (_call("write_stdin", session_id=1), "neutral"),
+ (_call("write_stdin", session_id=1, chars=""), "neutral"),
+ (_call("write_stdin", session_id=1, chars="\n"), "neutral"),
+ (_call("write_stdin", session_id=1, chars="\x03"), "neutral"),
+ (_call("write_stdin", session_id=1, chars="python review.py\n"), "productive"),
+ (_call("write_stdin", session_id=1, chars="pytest -q\n"), "productive"),
+ (_call("write_stdin", session_id=1, chars="find / -name review.py\n"), "wasted"),
+ (_call("write_stdin", session_id=1, chars="pip install requests\n"), "wasted"),
+ (_call("Monitor", description="deploy events", ws={"url": "wss://events.example.test"}), "neutral"),
+ (
+ _call("Monitor", description="errors", command="tail -f build.log | grep --line-buffered ERROR"),
+ "productive",
+ ),
+ (_call("Monitor", description="skills", command="find / -name SKILL.md"), "wasted"),
+ ],
+ ids=lambda value: value if isinstance(value, str) else f"{value['action']}:{value['action_input']}",
+)
+def test_session_input_is_classified_by_the_text_it_sends(module, call, kind):
+ # Polling a session or watching a socket is bookkeeping. Typing a command
+ # into a shell, or starting a watch script, runs that command.
+ assert _classify(module, call, "review.py") == kind
+
+
[email protected]("module", _MODULES)
+def test_root_search_typed_into_an_interactive_shell_is_wasted(module):
+ calls = [
+ _call("exec_command", cmd="bash", tty=True),
+ _call("write_stdin", session_id=1, chars="find / -name review.py\n"),
+ _call("write_stdin", session_id=1, chars=""),
+ ]
+
+ result = module.check_tool_efficiency(calls, _SKILL, "review.py")
+
+ assert result["score"] == 0.5
+ assert result["reason"] == "1/2 productive calls (50%); 1 planning or bookkeeping call(s) not counted"
+
+
+def test_wasted_session_input_is_labelled_with_its_text():
+ calls = [_call("write_stdin", session_id=1, chars="find / -name review.py\n")]
+
+ result = checks.check_tool_efficiency(calls, _SKILL)
+
+ assert result["details"]["wasted_calls"] == ["write_stdin(find / -name review.py\n)"]
+
+
[email protected]("module", _MODULES)
+ ("call", "kind"),
+ [
+ (_call("mcp__filesystem__search_files", path="/", pattern="SKILL.md"), "wasted"),
+ (_call("mcp__filesystem__list_directory", path="/"), "wasted"),
+ (_call("mcp__filesystem__directory_tree", path="//"), "wasted"),
+ (_call("mcp__workspace__list", dir_path="/"), "wasted"),
+ (_call("mcp__my_filesystem__stat", path="/"), "wasted"),
+ (_call("mcp__filesystem__search_files", path="/workspace", pattern="SKILL.md"), "productive"),
+ (_call("mcp__filesystem__list_directory", path="/workspace/src"), "productive"),
+ (_call("mcp__github__search_code", query="path:/ release"), "productive"),
+ ],
+ ids=lambda value: value if isinstance(value, str) else f"{value['action']}:{value['action_input']}",
+)
+def test_mcp_calls_rooted_at_filesystem_root_are_wasted(module, call, kind):
+ assert _classify(module, call) == kind
+
+
[email protected]("module", _MODULES)
+ "call",
+ [
+ _call("mcp__github__get_file_contents", owner="o", repo="r", path="/"),
+ _call("mcp__github__get_file_contents", owner="o", repo="r", path="src/"),
+ _call("mcp__docs__get_page", path="/"),
+ _call("mcp__wiki__read_page", dir_path="/"),
+ ],
+ ids=lambda call: f"{call['action']}:{call['action_input']}",
+)
+def test_mcp_path_root_of_other_servers_is_productive(module, call):
+ # Other servers use path for something other than the local filesystem:
+ # path "/" on the GitHub server lists a repository root, not the disk.
+ assert _classify(module, call) == "productive"
+
+
[email protected]("module", _MODULES)
+def test_claude_code_task_list_calls_are_left_out_of_the_score(module):
+ calls = [
+ _call("Read", file_path=f"/workspace/skills/{_SKILL}/SKILL.md"),
+ *[_call("TaskCreate", subject=f"step {index}") for index in range(3)],
+ _call("Edit", file_path="src/a.py"),
+ _call("Edit", file_path="src/b.py"),
+ *[_call("TaskUpdate", taskId=str(index), status="completed") for index in range(3)],
+ _call("Bash", command="pytest -q"),
+ ]
+
+ result = module.check_tool_efficiency(calls, _SKILL)
+
+ assert result["passed"] is True
+ assert result["score"] == 1.0
+ assert result["reason"] == "4/4 productive calls (100%); 6 planning or bookkeeping call(s) not counted"
+
+
[email protected]("module", _MODULES)
+def test_codex_wait_is_left_out_like_wait_agent(module):
+ for waiter in ("wait", "wait_agent"):
+ calls = [_call("spawn_agent", message="review"), _call(waiter), _call("exec_command", cmd="pytest -q")]
+ result = module.check_tool_efficiency(calls, _SKILL)
+ assert result["score"] == 1.0, (waiter, result)
+ assert result["reason"] == "1/1 productive calls (100%); 2 planning or bookkeeping call(s) not counted"
+
+
[email protected]("module", _MODULES)
+ ("call", "kind"),
+ [
+ (_call("functions.apply_patch", input="*** Begin Patch\n*** End Patch"), "productive"),
+ (_call("functions.list_dir", dir_path="src"), "productive"),
+ (_call("functions.grep_files", pattern="TODO", path="src"), "productive"),
+ (_call("functions.exec_command", cmd="pytest -q"), "productive"),
+ (_call("functions.update_plan", plan=[]), "neutral"),
+ (_call("functions.spawn_agent", message="review"), "neutral"),
+ (_call("functions.list_dir", dir_path="/"), "wasted"),
+ (_call("functions.grep_files", pattern="release", path="/"), "wasted"),
+ (_call("functions.exec_command", cmd="find / -name review.py"), "wasted"),
+ ],
+ ids=lambda value: value if isinstance(value, str) else f"{value['action']}:{value['action_input']}",
+)
+def test_namespaced_codex_tools_are_classified_by_their_name(module, call, kind):
+ assert _classify(module, call, "review.py") == kind
+
+
[email protected]("module", _MODULES)
+def test_namespaced_codex_patch_run_keeps_full_tool_efficiency(module):
+ calls = [
+ _call("functions.exec_command", cmd="pytest -q"),
+ _call("functions.apply_patch", input="*** Begin Patch\n*** End Patch"),
+ _call("functions.apply_patch", input="*** Begin Patch\n*** End Patch"),
+ _call("functions.update_plan", plan=[]),
+ ]
+
+ result = module.check_tool_efficiency(calls, _SKILL)
+
+ assert result["score"] == 1.0
+ assert result["reason"] == "3/3 productive calls (100%); 1 planning or bookkeeping call(s) not counted"
+
+
[email protected]("module", _MODULES)
[email protected]("action", ["mcp__tasks__task", "mcp__planner__update_plan", "mcp__agents__wait"])
+def test_mcp_tools_named_like_bookkeeping_tools_stay_mcp_calls(module, action):
+ assert _classify(module, _call(action, id="1")) == "productive"
+
+
[email protected]("module", _MODULES)
+def test_background_shell_polling_gets_no_execution_credit(module):
+ # BashOutput and KillShell contain "bash" and "shell", which used to make
+ # them execution calls. Polling a background shell is not new work.
+ result = module.check_tool_efficiency(
+ [_call("BashOutput", bash_id="1"), _call("KillShell", shell_id="1"), _call("Bash", command="grep -r x /")],
+ _SKILL,
+ )
+ assert result["score"] == 0.0
+ assert result["reason"] == "0/1 productive calls (0%); 2 planning or bookkeeping call(s) not counted"
+
+
[email protected]("module", _MODULES)
+ "command",
+ [
+ "grep -r SKILL.md /",
+ "grep -rl 'RELEASE|' / 2>/dev/null",
+ "find / -name policy.json",
+ "rg policy.json / | head",
+ "du -sh /",
+ "pip install requests",
+ "python --version",
+ "which jq",
+ ],
+)
+def test_filesystem_scans_and_setup_noise_are_wasted(module, command):
+ assert _classify(module, _call("Bash", command=command)) == "wasted"
+
+
[email protected]("module", _MODULES)
+ "call",
+ [
+ _call("Bash", command="find / -name review.py"),
+ _call("Bash", command="grep -r review.py /"),
+ _call("exec_command", cmd="find / -name review.py 2>/dev/null"),
+ _call("shell", command=["bash", "-lc", "find / -name review.py"]),
+ _call("shell", command=["find", "/", "-name", "review.py"]),
+ _call("shell", cmd=["sh", "-c", "grep -r review.py /"]),
+ ],
+ ids=lambda call: f"{call['action']}:{call['action_input']}",
+)
+def test_root_searches_naming_the_expected_script_are_wasted(module, call):
+ assert _classify(module, call, "review.py") == "wasted"
+
+
[email protected]("module", _MODULES)
+ "call",
+ [
+ _call("Bash", command="python review.py"),
+ _call("Bash", command="cd /work && python scripts/review.py --input /work/pr.diff"),
+ _call("shell", command=["python", "review.py"]),
+ _call("shell", command=["bash", "-lc", "python review.py"]),
+ ],
+ ids=lambda call: f"{call['action']}:{call['action_input']}",
+)
+def test_running_the_expected_script_stays_productive(module, call):
+ assert _classify(module, call, "review.py") == "productive"
+
+
[email protected]("module", _MODULES)
+ "search",
+ [
+ _call("Bash", command="find / -name review.py"),
+ _call("shell", command=["bash", "-lc", "find / -name review.py"]),
+ _call("shell", command=["find", "/", "-name", "review.py"]),
+ ],
+ ids=["string", "argv-shell", "argv-find"],
+)
+def test_root_search_for_the_expected_script_scores_like_any_root_search(module, search):
+ run = _call("Bash", command="python review.py")
+ other = _call("Bash", command="find / -name other.py")
+
+ named_result = module.check_tool_efficiency([search, run], _SKILL, "review.py")
+ other_result = module.check_tool_efficiency([other, run], _SKILL, "review.py")
+ only_search = module.check_tool_efficiency([search], _SKILL, "review.py")
+
+ assert named_result["score"] == other_result["score"] == 0.5
+ assert only_search["score"] == 0.0
+
+
[email protected]("module", _MODULES)
+ "call",
+ [
+ _call("Glob", pattern="**/SKILL.md", path="/"),
+ _call("Glob", pattern="/**/SKILL.md"),
+ _call("Glob", pattern="/*/policy.json"),
+ _call("Grep", pattern="release", path="/"),
+ _call("Grep", pattern="release", path="//"),
+ _call("LS", path="/"),
+ _call("list", path="/"),
+ _call("grep_files", pattern="release", path="/"),
+ _call("list_dir", dir_path="/", depth=5),
+ ],
+ ids=lambda call: f"{call['action']}:{call['action_input']}",
+)
+def test_native_searches_rooted_at_filesystem_root_are_wasted(module, call):
+ assert _classify(module, call) == "wasted"
+
+
[email protected]("module", _MODULES)
+ "call",
+ [
+ _call("Grep", pattern="release", path="/work"),
+ _call("Glob", pattern="/work/**/*.yaml"),
+ _call("Glob", pattern="**/*.py", path="/workspace"),
+ _call("LS", path="/tmp"),
+ _call("list", path="/workspace/src"),
+ _call("grep_files", pattern="release", path="/work"),
+ _call("list_dir", dir_path="/work", depth=2),
+ ],
+ ids=lambda call: f"{call['action']}:{call['action_input']}",
+)
+def test_scoped_native_searches_stay_productive(module, call):
+ assert _classify(module, call) == "productive"
+
+
[email protected]("module", _MODULES)
+def test_native_and_shell_root_scans_score_the_same(module):
+ read = _call("Read", file_path="/work/manifest.yaml")
+ native = [_call("Glob", pattern="**/SKILL.md", path="/"), _call("Grep", pattern="release", path="/"), read]
+ shell = [_call("Bash", command="find / -name SKILL.md"), _call("Bash", command="grep -r release /"), read]
+
+ native_result = module.check_tool_efficiency(native, _SKILL)
+ shell_result = module.check_tool_efficiency(shell, _SKILL)
+
+ assert native_result["score"] == shell_result["score"] == round(1 / 3, 4)
+ assert native_result["passed"] is shell_result["passed"] is False
+
+
[email protected]("module", _MODULES)
[email protected]("command", ["grep -r TODO src/", "find . -name '*.py'", "ls /tmp", "grep -n x /etc/hosts"])
+def test_scoped_searches_stay_productive(module, command):
+ assert _classify(module, _call("Bash", command=command)) == "productive"
+
+
[email protected]("module", _MODULES)
+ "argv",
+ [
+ ["pip", "install", "requests"],
+ ["pip", "install", "-e", "."],
+ ["brew", "install", "jq"],
+ ["npm", "install"],
+ ["which", "jq"],
+ ["python", "--version"],
+ ],
+ ids=" ".join,
+)
+def test_setup_noise_in_an_argv_command_is_wasted(module, argv):
+ # Help, version and install commands recorded as an argument list are
+ # waste, like the same command written as text.
+ assert _classify(module, _call("shell", command=argv)) == "wasted"
+ assert _classify(module, _call("Bash", command=" ".join(argv))) == "wasted"
+
+
+_README_PATCH = (
+ "*** Begin Patch\n*** Update File: README.md\n@@\n-Install it.\n"
+ "+Run `pip install -e .`, then pick the parser which matches the input.\n"
+ "+See `--help` to adapt the defaults, or `apt-get` and `brew ` for system packages.\n"
+ "*** End Patch"
+)
+
+
[email protected]("module", _MODULES)
[email protected]("action", ["apply_patch", "functions.apply_patch", "applypatch"])
+def test_patch_text_is_not_read_as_a_command(module, action):
+ # The converter records a non-object tool input under "raw". A patch that
+ # mentions an install or help command is still a patch, so it scores the
+ # same under every patch argument.
+ for key in ("input", "raw", "patch", "patchText"):
+ assert _classify(module, _call(action, **{key: _README_PATCH})) == "productive", key
+
+
[email protected]("module", _MODULES)
+ "call",
+ [
+ _call("Edit", raw=_README_PATCH),
+ _call("Write", file_path="docs/which version.md", content="pip install -e ."),
+ _call("Read", file_path="docs/brew install.md"),
+ _call("mcp__docs__search", query="pip install --help"),
+ ],
+ ids=lambda call: f"{call['action']}:{sorted(call['action_input'])}",
+)
+def test_install_words_in_file_text_are_not_commands(module, call):
+ assert _classify(module, call) == "productive"
+
+
[email protected]("module", _MODULES)
+def test_a_command_recorded_under_raw_is_still_checked(module):
+ assert _classify(module, _call("exec_command", raw="pip install requests")) == "wasted"
+ assert _classify(module, _call("exec_command", raw="pytest -q")) == "productive"
+
+
[email protected]("module", _MODULES)
+def test_opencode_run_scores_like_the_same_claude_code_run(module):
+ # OpenCode lists a directory with "list"; Claude Code uses "LS".
+ skill_md = f"/workspace/skills/{_SKILL}/SKILL.md"
+ opencode = [
+ _call("read", filePath=skill_md, path=skill_md),
+ _call("list", path="/workspace"),
+ _call("edit", filePath="/workspace/src/app.py", path="/workspace/src/app.py"),
+ _call("bash", command="pytest -q"),
+ ]
+ claude_code = [
+ _call("Read", file_path=skill_md),
+ _call("LS", path="/workspace"),
+ _call("Edit", file_path="/workspace/src/app.py"),
+ _call("Bash", command="pytest -q"),
+ ]
+
+ opencode_result = module.check_tool_efficiency(opencode, _SKILL)
+ claude_code_result = module.check_tool_efficiency(claude_code, _SKILL)
+
+ assert opencode_result == claude_code_result
+ assert opencode_result["score"] == 1.0
+ assert opencode_result["reason"] == "4/4 productive calls (100%)"
+
+
[email protected]("module", _MODULES)
+def test_neutral_calls_are_left_out_of_the_score(module):
+ calls = [
+ _call("TodoWrite"),
+ _call("Read", file_path="SKILL.md"),
+ _call("Bash", command="grep -r x /"),
+ _call("TodoWrite"),
+ _call("Write", file_path="out.md"),
+ ]
+
+ result = module.check_tool_efficiency(calls, _SKILL)
+
+ assert result["score"] == round(2 / 3, 4)
+ assert result["passed"] is True
+ assert result["reason"] == "2/3 productive calls (67%); 2 planning or bookkeeping call(s) not counted"
+
+
+def test_details_count_neutral_calls_and_label_wasted_argv_calls():
+ calls = [
+ _call("update_plan"),
+ _call("shell", command=["find", "/", "-name", "review.py"]),
+ _call("apply_patch", input="*** Begin Patch\n*** End Patch"),
+ ]
+
+ result = checks.check_tool_efficiency(calls, _SKILL, "review.py")
+
+ assert result["details"] == {
+ "productive": 1,
+ "wasted": 1,
+ "neutral": 1,
+ "total": 2,
+ "wasted_calls": ["shell(['find', '/', '-name', 'review.py'])"],
+ }
+
+
[email protected]("module", _MODULES)
+def test_codex_sub_agent_calls_are_left_out_of_the_score(module):
+ actions = ["update_plan", "spawn_agent", "spawn_agent", "spawn_agent", "exec_command", "wait_agent"]
+ calls = [_call(action, cmd="make test") if action == "exec_command" else _call(action) for action in actions]
+ calls += [_call("update_plan"), _call("update_plan"), _call("exec_command", cmd="pytest -q"), _call("update_plan")]
+
+ result = module.check_tool_efficiency(calls, "review-pr")
+
+ assert result["score"] == 1.0
+ assert result["passed"] is True
+ assert result["reason"] == "2/2 productive calls (100%); 8 planning or bookkeeping call(s) not counted"
+
+
[email protected]("module", _MODULES)
+def test_only_bookkeeping_calls_score_zero(module):
+ # A trial that only plans did no work. It must not outscore a trial with
+ # no tool calls, whose skill_efficiency is 0.0.
+ result = module.check_tool_efficiency([_call("TodoWrite"), _call("update_plan")], _SKILL)
+
+ assert result["passed"] is False
+ assert result["score"] == 0.0
+ assert result["reason"] == "No scored tool calls; 2 planning or bookkeeping call(s) not counted"
+
+
+def test_only_bookkeeping_calls_record_zero_scored_calls():
+ result = checks.check_tool_efficiency([_call("TodoWrite"), _call("update_plan")], _SKILL)
+
+ assert result["details"] == {"productive": 0, "wasted": 0, "neutral": 2, "total": 0, "wasted_calls": []}
+
+
+def _tool_step(index: int, name: str, arguments: dict) -> dict:
+ call_id = f"call-{index}"
+ return {
+ "source": "agent",
+ "tool_calls": [{"tool_call_id": call_id, "function_name": name, "arguments": arguments}],
+ "observation": {"results": [{"source_call_id": call_id, "content": "ok"}]},
+ }
+
+
+def _run_verifier(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, calls: list[tuple[str, dict]]) -> dict:
+ """Run the bundled verifier's main() on a no-reference with-skill case."""
+ trajectory = {
+ "steps": [
+ {"source": "user", "message": "Review the release notes and fix the modules."},
+ *[_tool_step(index, name, arguments) for index, (name, arguments) in enumerate(calls)],
+ {"source": "agent", "message": "Fixed the modules and the tests pass."},
+ ]
+ }
+ agent_dir = tmp_path / "logs" / "agent"
+ verifier_dir = tmp_path / "logs" / "verifier"
+ agent_dir.mkdir(parents=True)
+ verifier_dir.mkdir(parents=True)
+ atif = agent_dir / "trajectory.json"
+ atif.write_text(json.dumps(trajectory), encoding="utf-8")
+ entry = tmp_path / "entry.json"
+ entry.write_text(
+ json.dumps({"id": "case-1", "question": "Fix the modules", "has_skill": True, "expected_skill": _SKILL}),
+ encoding="utf-8",
+ )
+ monkeypatch.setattr(template, "ATIF_PATH", atif)
+ monkeypatch.setattr(template, "ENTRY_PATH", entry)
+ monkeypatch.setattr(template, "VERIFIER_DIR", verifier_dir)
+ monkeypatch.setattr(template, "REWARD_JSON", verifier_dir / "reward.json")
+ monkeypatch.setattr(template, "REWARD_TXT", verifier_dir / "reward.txt")
+ monkeypatch.setattr(template, "SKILL_EVALUATOR_REWARD_JSON", verifier_dir / "skill_evaluator_reward.json")
+ monkeypatch.setattr(template, "call_public_llm", lambda *_a, **_k: pytest.fail("No-reference case called an LLM"))
+
+ template.main()
+
+ return json.loads((verifier_dir / "skill_evaluator_reward.json").read_text(encoding="utf-8"))
+
+
+def test_edit_heavy_run_keeps_full_tool_efficiency_in_skill_efficiency(tmp_path, monkeypatch):
+ calls = [
+ ("Read", {"file_path": f"/workspace/skills/{_SKILL}/SKILL.md"}),
+ ("TodoWrite", {"todos": []}),
+ *[("Edit", {"file_path": f"/workspace/src/module_{index}.py"}) for index in range(4)],
+ ("Bash", {"command": "pytest -q"}),
+ ]
+
+ rich = _run_verifier(tmp_path, monkeypatch, calls)
+
+ efficiency = rich["details"]["skill_efficiency"]
+ assert efficiency["tool_efficiency"] == {
+ "passed": True,
+ "score": 1.0,
+ "reason": "6/6 productive calls (100%); 1 planning or bookkeeping call(s) not counted",
+ }
+ routing = efficiency["routing"]["score"]
+ assert rich["skill_efficiency"] == round((routing + 1.0) / 2, 4)
+
+
+ "calls",
+ [
+ [],
+ [("TodoWrite", {"todos": []})],
+ [("update_plan", {"plan": []})],
+ [("Task", {"prompt": "Fix the modules"})],
+ ],
+ ids=["no-calls", "TodoWrite", "update_plan", "Task"],
+)
+def test_planning_only_run_scores_skill_efficiency_like_no_tool_calls(tmp_path, monkeypatch, calls):
+ rich = _run_verifier(tmp_path, monkeypatch, calls)
+
+ assert rich["skill_efficiency"] == 0.0
+ host = checks.score_skill_efficiency([_call(name, **arguments) for name, arguments in calls], _SKILL)
+ assert host["score"] == 0.0