The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
Native Release interaction benchmark
Start with the September 22 wrap-up and resume notes for the ready PRs, measured improvements, rejected experiments and remaining priorities.
Connected local workspace resources
The older fixtures below deliberately suppress real connections and several
background services. Their component timings cannot establish total app energy
savings. The connected fixture runs the Release desktop, its full local daemon,
and real tmux terminals with a private home, identity, control port and socket.
FLUTTER_TEST is absent: local background services and daemon supervision
remain active. Environment installation, sign-in and update acquisition are
substituted; the workspace is signed out. The injected discovery uses the
externally owned test daemon and disables ownership-triggered restarts. It does
not use real conversations, model accounts or the installed daemon.
On macOS, build the CLI with node build-bundle.mjs in cli/, then:
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
python3 tool/native_benchmark/prepare.py --flutter /path/to/pinned/flutter \
--connected-resource
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
python3 tool/native_benchmark/test_process_forest_usage.py
HARNESS_BENCH_NODE=/absolute/path/to/pinned/node \
HARNESS_BENCH_TMUX=/absolute/path/to/tmux \
python3 tool/native_benchmark/test_connected_stack.py
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
xcrun swiftc tool/native_benchmark/process_forest_usage.swift \
-o /private/tmp/harness-process-forest-usage
python3 tool/native_benchmark/connected_run.py \
--app '/private/tmp/harness-native-benchmark-PRINTED/desktop/build/macos/Build/Products/Release/Harness Benchmark.app' \
--root /private/tmp/harness-connected-UNUSED_NAME \
--node /absolute/path/to/pinned/node --tmux /absolute/path/to/tmux \
--sampler /private/tmp/harness-process-forest-usage \
--label current --terminals 10 --seconds 30
The launcher refuses an existing root or a normal installed app. Its additional
macOS sandbox allows writes only inside that run and terminal devices, and
outbound networking only over loopback or its private Unix sockets. macOS's
setuid /bin/ps has a narrow execution exception for read-only discovery.
Keep the desktop unlocked for native runs. Before creating any fixture process,
the launcher checks console lock metadata. A known locked session produces a
sibling RUN_NAME.preflight.json rejection receipt without creating the run
directory. This registry field is not a public API contract: unavailable metadata
is recorded as unknown, and never substitutes for the framework/native visibility
checks below. Console account names and identifiers are not retained.
Closing the owning launcher's stdin stops its daemon and its explicitly named
tmux server; cleanup.json verifies the private socket has stopped listening.
The app also refuses startup without the matching private-home environment.
Every retained terminal runs a deterministic Node worker with 1,000 seeded
lines. Four panes are visible; the remaining panes live in other tabs. The five
phases are foreground idle with a visible cursor, foreground idle with a hidden
cursor, foreground output, hidden-app idle, and hidden-app output. Active
workers redraw eight rows at 20 Hz. Output must reach every retained terminal,
connections must remain controlling, and actual output counts and skipped ticks
are retained. A focus, visibility or geometry change during a phase rejects it.
Native visibility alone is insufficient: foreground phases require Flutter
resumed with frames enabled, and hidden phases require Flutter hidden with
frames disabled and no new drawn frames after settling. The fixture records
framework lifecycle transitions/frame counts and native occlusion independently.
Rebuild older fixture apps that lack those observations. A missing or contradictory
lifecycle rejects the run and retains the rejected state and cleanup outcome.
Run python3 tool/native_benchmark/test_connected_visibility.py for that guard's
regression checks. An October 3 diagnostic
reproduced a null framework lifecycle with continued hidden rendering; older
native-only background checks do not establish normal product background behavior.
This controls terminal traffic, not model behavior, inference or cloud latency.
Use --background-only for the two hidden-app phases when foreground focus
cannot be held. Compare it only with the same mode: it starts idle directly,
without the foreground phases' preceding allocation and output history.
process_forest_usage.swift samples the app, daemon and tmux process trees at
1 Hz. CPU and wakeup accounting includes kernel-recorded exited children, so
short-lived ps and Git helpers count. The process inventory includes zombies:
their CPU remains attributed to them until their parent reaps them and receives
the child counters. The regression calibration deliberately delays reaping;
the earlier proc_listallpids inventory dropped that interval and was rejected.
See Apple's child-accounting implementation.
Root overlap, root replacement, unstable
membership and regressing counters reject a measurement. Calibration compares
live and reaped child CPU against independent getrusage counters. Processes
that escape/reparent out of these trees are outside its boundary. Summed
physical footprints can include shared mappings. Neither this sampler nor the
single-process sampler measures GPU joules, battery discharge, or macOS's
“Using Significant Energy” classification.
On supported macOS kernels the sampler also records per-process instructions,
cycles, performance-core time and kernel-accounted CPU energy through
RUSAGE_INFO_V6. These are raw diagnostic counters, not additions to the forest
summary: unlike CPU time, they have no exited-child rollup in this API. Compare
matching PID/birth identities that persist throughout the measured interval.
Older kernels fall back to V4 and omit the V6 fields; zero hardware counters can
also mean unsupported hardware. CPU energy excludes GPU, shared system services
and battery discharge. See Apple's counter mapping.
Use prepare.py --source /path/to/baseline/desktop and connected_run.py --bundle /path/to/baseline/cli/dist/cli.js to compare a second source variant
with the same tooling. The copied build records its source revision, production
patch digest and fixture hashes; the stack records the exact CLI bundle hash.
Finish all builds before recording measurements. Run at least three paired
trials, alternate their order, keep failed runs, and record host contention.
Treat five-second runs as calibration only. Raw phase samples, output counters,
native state, source identities, failures and cleanup results remain in the run
directory. Never extrapolate one workload's CPU improvement to whole-device
energy or unrelated signed-in workloads.
The October 2 connected comparison records three pairs, the rejected initial accounting experiment, and both the hidden-cursor idle improvement and higher foreground active-output CPU. It does not establish overall energy savings.
The macOS discovery experiment records a cheaper executable-image probe, lower idle daemon CPU in three matched pairs, inconsistent connected CPU results, and a diagnostic that keeps the same app and worker processes alive. The daemon result does not establish whole-app energy savings; native comparisons still require a valid framework lifecycle.
Core experiences and process resources
python3 tool/native_benchmark/prepare.py --flutter /path/to/flutter \
--core-workflows --terminals 16 --samples 120 --hold
Open the printed isolated app through normal application controls. Use three
samples for calibration first, then 120 for reporting. run-config.json in the
printed temporary root selects terminals, samples, hold, and a fresh JSON
output filename. After quitting the fixture, edit that file and reopen the same
bundle to repeat without changing its compiled production code. Existing results
are never overwritten. active-run.json identifies the latest invocation. The core mode
supports 1, 16, or 48 retained terminals, with up to four visible and 1,000
seeded lines each. Set seedLines: 10000 to exercise full scrollback buffers;
wrapping and the production buffer limit affect retained physical rows, whose
counts are recorded in the result. It visits every retained tab before measuring,
then repeats at idle and while every terminal receives an eight-row ANSI redraw
at 20 Hz. Actual bytes, duration, and skipped output bursts are retained.
It validates eleven common actions: text commit with a synthetic terminal echo; Cmd+N; Cmd+O; picker query; picker selection followed by immediate input; Cmd+T; tab change followed by input; pane focus followed by input; zoom; Find through a completed index scan; and a page-sized scroll. Inapplicable navigation actions are skipped with one terminal. Five warmups precede each series. Every input commit must arrive exactly once in the focused terminal. Picker queries and Find must produce the expected real results before their frame is accepted.
These are framework dispatch to completed Flutter raster measurements.
Text uses the attached TextInputClient; shortcuts use the normal framework
keyboard/focus dispatcher. Text echo passes through the production terminal
binary decoder and parser with zero simulated network delay. Navigation-plus-
input measurements include the navigation frame and the following verified echo
frame; they are not interchangeable with a shortcut's first-frame time. Scrolling
uses the terminal's scroll position controller and excludes native wheel-event
delivery. AppKit keyboard delivery, actual networking, model inference, GPU
presentation, and native titlebar paint completion remain outside this boundary.
The native host records loss of foreground focus; those runs fail rather than
mixing foreground and background timings. .progress identifies the current
invocation and last completed operation. Preserve failed runs for diagnosis and
use the result's fresh timestamps rather than accepting an old result file.
--hold leaves the idle fixture open after the workload for resource sampling:
xcrun swiftc tool/native_benchmark/process_usage.swift -o /private/tmp/harness-process-usage
/private/tmp/harness-process-usage PID 30 foreground-idle /private/tmp/usage-foreground.json
Use the PID in the completed result's metadata. Hide the fixture through the app's
normal controls and repeat with a new label/file for background idle. This sampler
reads proc_pid_rusage once per second: CPU deltas (100% means one core), physical
memory footprint, resident memory, interrupt/package-idle wakeups, and disk I/O.
CPU counters are Mach absolute ticks,
converted with mach_timebase_info.
Schema 2 records the raw ticks, timebase and process birth stamp; a replaced
process fails the sample. Earlier schema 1 CPU percentages and seconds omitted
that conversion and are invalid on machines whose timebase is not 1:1. Do not
reuse them without the original machine's clock metadata. Memory, wakeup and
disk counters use separate units and are unaffected.
Calibrate the sampler before interpreting CPU results:
python3 tool/native_benchmark/test_process_usage.py
This compiles the sampler and compares a private busy worker against Python's
independent getrusage CPU accounting. It also checks idle CPU and preservation
of existing output, including a file created during sampling. It does not start
or control any app or agent. The October 1 calibration
records the original failure and corrected measurements.
It does not control windows. Its totals cover that process, excluding daemon, agent processes, and GPU energy. The fixture has no real network connections, so its background cost cannot establish that the connected product has zero timers or zero wakeups. Record other app activity and host contention; these are real workstation measurements, not results from an otherwise isolated machine.
To sample sustained redraw cost separately, set holdOutput: true with hold
in run-config.json. After the timing sweep the same 20 Hz workload continues;
.resource-load records achieved bytes and skipped bursts every ten seconds.
That diagnostic timer exists only for this active-output resource mode. Use a
fresh process with holdOutput: false for idle measurements.
For actual loopback/direct/TURN/relay terminal round trips, use the separate real terminal probe. Never add independently measured p95 values and label the sum end-to-end latency.
Pool a published data directory with python3 tool/native_benchmark/summarize_results.py /path/to/data. It computes nearest-rank percentiles from individual measured
observations and lists contributing files, excluding warmups. Failed runs must
be reviewed separately; it does not silently discard them.
Framework-dispatch comparison and manual feature checks
The fixture supports three separate modes:
python3 tool/native_benchmark/prepare.py --flutter /path/to/flutter --flutter-dispatch
python3 tool/native_benchmark/prepare.py --flutter /path/to/flutter --primary-workflows
python3 tool/native_benchmark/prepare.py --flutter /path/to/flutter --interactive
Open the printed BENCHMARK_APP through normal application controls. These
copies embed only their own temporary fixture environment in the copied host.
The dispatch mode runs and exits automatically, writing interactive.json in
BENCHMARK_ROOT; the interactive mode remains open for manual checks. They
cannot be combined. Neither mode opens a real transport or uses saved sessions.
--flutter-dispatch measures Cmd+N/O/T/P/comma/slash/F, tab changes, pane focus,
and zoom in the macOS Release renderer. It calls the framework's keyboard-state
and focus dispatch stages, validates the resulting widget/state, and joins the
exact first frame and fully opened route frame to FrameTiming. Each operation
has five warmups and 40 measured observations. All observations are retained.
The fixture has 16 sessions, four visible panes and 1,000 scrollback rows per
session. It measures idle-terminal interactions only. Run baseline and edited
production sources with identical fixture tooling, sequentially, with builds
and other tests finished. Preserve every completed run.
This mode excludes AppKit input delivery, physical keyboard latency, GPU presentation, network latency and native titlebar paint completion. A fully opened frame includes any route fade; the first frame is reported separately. The reported display maximum is metadata, not an asserted refresh rate. The driver also records the preceding frame, dispatch offset from its vsync, waiting time until the requested frame starts building, and build-start-to-raster time. A preceding busy frame can change the input's position within a refresh interval; inspect these components before attributing elapsed-time differences to CPU work. Wall-clock phase timestamps are correlated using the engine's paired monotonic and wall-clock raster-finish timestamps.
--primary-workflows runs only Cmd+N/O/T and tab switching, with five warmups
and 120 measured samples per action. It inserts a repeatable sequence of 0–20 ms
delays before the measured dispatch to spread input across refresh phases.
The requested delay is retained on every sample. This supplements the original
post-frame cadence; preserve and report both if they give different results.
The recorded preceding frame is the frame awaited before that optional delay.
Use manual mode for visual/interaction QA, not timing claims. See the
September 22 results.
For primary-workflow debug CPU comparisons at 16/48 retained terminals:
flutter test --no-pub test/benchmarks/primary_workflows_benchmark.dart --concurrency=1 --reporter expanded
The fixture exercises Cmd+N/O/T and workspace switching with synthetic metadata
replies and 1,000 scrollback rows per terminal. It checks the destination and
terminal input isolation for every action. Five warmups and a separate rebuild
instrumentation pass precede 40 timed samples per action. Optional
HARNESS_PRIMARY_CPU_PROFILE=/private/tmp/primary with --enable-vmservice
records profiles; HARNESS_PRIMARY_OPERATION=cmd_t restricts the action.
See the primary-workflow results
for the isolated Cmd+O comparison, unchanged controls and rejected new-tab experiment.
The pane-caching experiment
reports both input cadences and frame-phase diagnostics; less build work did not
translate into faster tab switching, so that production change was rejected.
Original AppKit event-queue runner
The original path below has no accepted calibration in this performance pass. Its Cmd+1…4 pane-focus workload also predates the current default keymap; it must be updated and recalibrated before reporting native-event latency.
Status, September 14, 2026: the builder supports the current Harness name,
validates the copied product identity and verifies the resulting bundle. It
accepts both the current ai.autonomous.harness and legacy .v2 source IDs.
Since the preview and installed app now share their ID, the runner also checks
bundle location: only the release identity in /Applications or the current
user's Applications folder is exempt. Development copies, legacy previews
and other benchmark processes still stop preflight. Initial native focus uses
the same Flutter controller as the production titlebar. Nine Python isolation
checks include the actual production config and this renamed-build regression. The current accepted measurements
use a separate framework-dispatch boundary.
Foreground/key-window guards remain intact; no p50/p95/p99 result has been accepted.
This macOS fixture measures AppKit-queued input through the production Swarm screen, terminal session parser and Flutter renderer. It runs as Harness Benchmark, in an isolated copy with its own bundle ID, synthetic transports, blocked HTTP and temporary state. It never reads the user's saved Swarms or sends input to an agent. The benchmark bridge is appended only to the copied native host; it is absent from the production Runner and Harness bundle.
From desktop/, build with a compatible Flutter SDK and Xcode:
python3 tool/native_benchmark/prepare.py --flutter /path/to/flutter
Use the BENCHMARK_APP path printed by the build. Unlock the Mac, normally close
the workspace preview and finish other builds/tests first. A locked desktop
cannot supply the active/key window required for valid samples. Keep this fixture in the foreground during a run; it
exits on focus loss instead of reclaiming focus between observations. The runner
refuses to start alongside another preview or benchmark process and never quits
them. Its error names the exact bundle path. A development copy outside the
standard installation folders is not exempt just because it shares the installed
Harness app's name and bundle identifier.
The installed app may remain open; record other app activity and host load when
reporting timings. That is not a guarantee of an otherwise idle workstation.
Run preflight checks without launching an app:
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tool/native_benchmark -p 'test_*.py' -v
python3 tool/native_benchmark/run.py \
--app '/private/tmp/harness-native-benchmark-EXAMPLE/desktop/build/macos/Build/Products/Release/Harness Benchmark.app' \
--terminals 16 --samples 120 \
--output /private/tmp/harness-native-16.json
Repeat with --terminals 48 and a fresh output path. Use --samples 3 only for
calibration, not percentile claims. The fixture closes itself normally when it
finishes; reopen the workspace preview afterward. Launch through run.py, which uses Launch Services
and supplies the required environment. Opening the fixture without that
environment fails immediately.
Workload and timing boundary
- A 1280 × 800 content area, four visible terminals, 16 or 48 retained sessions, and 1,000 initial scrollback rows each. Every Swarm is visited before warming.
- Typing posts
xthrough AppKit and echoes it through the production binary output handler with zero simulated network RTT. The exact destination session and input count are checked. Focus uses Cmd+1…4; tabs use Cmd+Shift+]. - One cold interaction per operation, then 20 warmups and the requested measured observations for each operation, both idle and with output to every terminal at 20 Hz. Each burst repaints eight rows with ANSI cursor save/restore. Reported byte counts, phase duration and skipped bursts expose the achieved output load.
- Input is posted only to this fixture's own
NSApplicationqueue and window. Both aggregate and device-side modifier flags, with press/release events, match AppKit's keyboard representation. Initial content focus is established once; subsequent navigation must perform its own focus handoff. - The first post-frame callback after the expected state/output change captures
the engine's frame number. It joins that exact ID to
FrameTiming, including its wall-clock raster-finish timestamp. Results retain every sample, including warmup and cold observations, and summarize p50/p95/p99/max in milliseconds. - The fixture uses the real layout file store in a fresh temporary directory. Normal discovery, networking, telemetry and background services are disabled.
These are native event-queue-to-Flutter-raster timings, not physical keyboard-to-photon measurements. They exclude device scanning, real transport and agent response time, GPU/display presentation, and the completion of separate AppKit titlebar drawing. The configured display maximum is recorded; it does not establish a fixed refresh rate. This does not measure app startup, IME, paste, scrolling, reconnect, or every output pattern. Preserve slower runs and report the machine, build revision, host load and sampling limits with results.