Skip to content

One 8 KB skill that replaces REA (Reverse Engineer Anything): plain CLI tools, no MCP. Benchmarked: 38% faster, 35% cheaper.

Python
4.0 KB
Raw Blame

REA vs a lean skill: benchmark (2026-10-16)

Setup: Claude Code 2.1.301 headless (claude -p), Sonnet 5.5, macOS arm64. REA is [email protected] (207 MCP tools, run as rea --mcp) plus its own skill (/reverse-engineer-anything). Every run starts in a fresh copy of the targets. Answers were graded by regex against answer keys I made by hand.

Arms:

  • none: no skill, no MCP.
  • leanrea (shown as lean in results/final.jsonl): /leanrea, the 8 KB leanrea/SKILL.md, no MCP.
  • rea: REA MCP plus the REA skill.

Tasks (3 reps per arm per task, 36 runs, arms run side by side):

  • electron: Notion app.asar. Trace mic/meeting detection: native fn, class, IPC channels.
  • native: mac_utils.node. Exports, the TCC check, the single-instance lock.
  • go: gh. Toolchain version, main path, two dependency versions.
  • deep: mac_utils.node, from the disassembly. Socket path format, wire protocol, peer authentication.

Results (median wall time; means for turns, cost, tokens)

task arm wall s turns cost $ tokens k score
electron leanrea 35.3 4.7 0.171 196 14/15
electron none 52.8 9.7 0.173 342 15/15
electron rea 82.9 13.3 0.341 725 15/15
native leanrea 43.4 5.3 0.201 239 24/24
native none 36.7 6.7 0.173 245 24/24
native rea 46.0 11.3 0.268 511 24/24
go leanrea 12.4 2.3 0.110 87 12/12
go none 13.8 2.0 0.092 67 12/12*
go rea 25.9 6.0 0.171 233 12/12
deep leanrea 38.6 6.0 0.241 278 24/24
deep none 45.3 8.7 0.298 383 16/24 (1 refusal)
deep rea 45.7 10.7 0.339 509 24/24

Totals over 12 runs per arm:

arm wall cost turns tokens score
leanrea 388 s $2.17 55 2.40 M 74/75
none 479 s $2.21 81 3.11 M 67/75
rea 627 s $3.36 124 5.93 M 75/75

* One "none/go" answer wrote "Go 1.27.1" and the regex wanted "go1.27.1". That is a grader false negative, corrected above.

The lean skill against REA:

  • 38% less wall time.
  • 35% lower cost.
  • 56% fewer turns.
  • 60% fewer tokens.
  • Same accuracy. The lean skill's only miss: one run named the v1 change channel instead of the v2 one. Both channels exist.

The lean skill against no skill:

  • 19% faster, with 32% fewer turns and 23% fewer tokens.
  • Cost is the same.
  • It avoided the one refusal the bare model gave on the deep task.

Context overhead ("Reply with exactly OK")

config context tokens cost
nothing 31.4k $0.08
lean skill invoked 34.5k (+3.1k) $0.09
REA MCP, deferred tool search (CC default) 34.9k (+3.5k) $0.09
REA skill invoked + MCP 42.7k (+11.3k) $0.13
REA MCP, tool search off (all tools loaded) 400.7k (+369k) $1.60

REA's raw tools/list: 207 tools, 3.35 MB JSON (745 KB input schemas, 2.46 MB output schemas, 90 KB descriptions).

Tool-level microbenchmark (same operation, same file)

operation REA CLI raw command
Go build info, gh 0.56 s, 62.5 KB go version -m: 0.06 s, 15.6 KB
Mach-O overview, mac_utils.node 3.32 s, 52.8 KB otool/dyld_info/nm: 0.04 s, 28.3 KB
Electron app analysis, Notion fails artifact_integrity_mismatch unless you add --integrity-policy record-and-continue; then 47.9 s, 140 MB asar extract + rg -uu IPC sweep: 4.8 s, 24 KB

Caveats

  • n=3 per cell, one model (Sonnet), one machine, three targets. The REA arm used Claude Code's deferred tool search, which is REA's best case. Agents that load every tool pay the 369k-token row above.
  • Native providers (Ghidra, Hopper, IDA) are not installed, so REA used its built-in native path. The lean arm used otool.

Reproduce: ./setup.sh, then ./run_all.sh (or python3 -I run.py <none|leanrea|rea> <task> <rep>) and ./ctx.sh.