Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

perf: measure core UI workflows under output and process resource use

dee2fe8e6fc49af
diff --git a/desktop/tool/native_benchmark/NativeBenchmark.swift b/desktop/tool/native_benchmark/NativeBenchmark.swift
index 965c2cfcf..f067d8e32 100644
--- a/desktop/tool/native_benchmark/NativeBenchmark.swift
+++ b/desktop/tool/native_benchmark/NativeBenchmark.swift
@@ -3,6 +3,8 @@
// Runner sources and the user's V2 bundle never contain this channel.
private enum NativeBenchmark {
private static var channel: FlutterMethodChannel?
+ private static var focusLosses = 0
+ private static var focusObserver: NSObjectProtocol?
static func install(window: NSWindow, messenger: FlutterBinaryMessenger) {
guard Bundle.main.bundleIdentifier == "ai.autonomous.harness.benchmark",
@@ -46,6 +48,18 @@ private enum NativeBenchmark {
// Once, after the synthetic UI is mounted. Never reset native or Dart
// focus between observations: navigation must perform its own handoff.
result(String(describing: type(of: window.firstResponder!)))
+ case "captureState":
+ if focusObserver == nil {
+ focusObserver = NotificationCenter.default.addObserver(
+ forName: NSWindow.didResignKeyNotification, object: window, queue: .main
+ ) { _ in focusLosses += 1 }
+ }
+ result([
+ "key": window.isKeyWindow,
+ "active": NSApp.isActive,
+ "visible": window.isVisible,
+ "focusLosses": focusLosses,
+ ])
case "key":
guard window.isKeyWindow, NSApp.isActive,
let args = call.arguments as? [String: Any],
diff --git a/desktop/tool/native_benchmark/README.md b/desktop/tool/native_benchmark/README.md
index 6f1f5a8f0..f98e554a8 100644
--- a/desktop/tool/native_benchmark/README.md
+++ b/desktop/tool/native_benchmark/README.md
@@ -3,6 +3,67 @@
Start with the [September 22 wrap-up and resume notes](../../../docs/performance/2026-09-22-wrap-up.md)
for the ready PRs, measured improvements, rejected experiments and remaining priorities.
+## Core experiences and process resources
+
+```sh
+python3 tool/native_benchmark/prepare.py --flutter /path/to/flutter \
+ --core-workflows --terminals 16 --samples 120 --hold
+```
+
+Open the printed isolated app through normal application controls. Use three
+samples for calibration first, then 120 for reporting. `run-config.json` in the
+printed temporary root selects `terminals`, `samples`, `hold`, and a fresh JSON
+`output` filename. After quitting the fixture, edit that file and reopen the same
+bundle to repeat without changing its compiled production code. Existing results
+are never overwritten. `active-run.json` identifies the latest invocation. The core mode
+supports 1, 16, or 48 retained terminals, with up to four visible and 1,000
+initial scrollback rows each. It visits every retained tab before measuring,
+then repeats at idle and while every terminal receives an eight-row ANSI redraw
+at 20 Hz. Actual bytes, duration, and skipped output bursts are retained.
+
+It validates eleven common actions: text commit with a synthetic terminal echo;
+Cmd+N; Cmd+O; picker query; picker selection followed by immediate input; Cmd+T;
+tab change followed by input; pane focus followed by input; zoom; Find through a
+completed index scan; and a page-sized scroll. Inapplicable navigation actions
+are skipped with one terminal. Five warmups precede each series. Every input
+commit must arrive exactly once in the focused terminal. Picker queries and Find
+must produce the expected real results before their frame is accepted.
+
+These are **framework dispatch to completed Flutter raster** measurements.
+Text uses the attached `TextInputClient`; shortcuts use the normal framework
+keyboard/focus dispatcher. Text echo passes through the production terminal
+binary decoder and parser with zero simulated network delay. Navigation-plus-
+input measurements include the navigation frame and the following verified echo
+frame; they are not interchangeable with a shortcut's first-frame time. Scrolling
+uses the terminal's scroll position controller and excludes native wheel-event
+delivery. AppKit keyboard delivery, actual networking, model inference, GPU
+presentation, and native titlebar paint completion remain outside this boundary.
+
+The native host records loss of foreground focus; those runs fail rather than
+mixing foreground and background timings. `.progress` identifies the current
+invocation and last completed operation. Preserve failed runs for diagnosis and
+use the result's fresh timestamps rather than accepting an old result file.
+`--hold` leaves the idle fixture open after the workload for resource sampling:
+
+```sh
+xcrun swiftc tool/native_benchmark/process_usage.swift -o /private/tmp/harness-process-usage
+/private/tmp/harness-process-usage PID 30 foreground-idle /private/tmp/usage-foreground.json
+```
+
+Use the PID in the completed result's metadata. Hide the fixture through the app's
+normal controls and repeat with a new label/file for background idle. This sampler
+reads `proc_pid_rusage` once per second: CPU deltas (100% means one core), physical
+memory footprint, resident memory, interrupt/package-idle wakeups, and disk I/O.
+It does not control windows. Its totals cover that process, excluding daemon,
+agent processes, and GPU energy. The fixture has no real network connections, so
+its background cost cannot establish that the connected product has zero timers
+or zero wakeups. Record other app activity and host contention; these are real
+workstation measurements, not results from an otherwise isolated machine.
+
+For actual loopback/direct/TURN/relay terminal round trips, use the separate
+[real terminal probe](../../../cli/scripts/benchmark-terminal-latency.md). Never
+add independently measured p95 values and label the sum end-to-end latency.
+
## Framework-dispatch comparison and manual feature checks
The fixture supports three separate modes:
diff --git a/desktop/tool/native_benchmark/core_driver.dart b/desktop/tool/native_benchmark/core_driver.dart
new file mode 100644
index 000000000..966674005
--- /dev/null
+++ b/desktop/tool/native_benchmark/core_driver.dart
@@ -0,0 +1,485 @@
+// Framework text/key dispatch through the production parser and Release renderer.
+// No OS input, network, model, or display-presentation latency is claimed here.
+import 'dart:async';
+import 'dart:convert';
+import 'dart:io';
+import 'dart:ui' as ui;
+
+import 'package:flutter/material.dart';
+import 'package:flutter/services.dart';
+import 'package:harness/state/app_state.dart';
+import 'package:harness/state/new_harness.dart';
+import 'package:harness/widgets/new_harness_box.dart';
+import 'package:harness/widgets/swarm_search_input.dart';
+import 'package:harness/widgets/terminal_find_bar.dart';
+import 'package:harness/widgets/workspace_welcome.dart';
+import 'package:xterm/src/ui/custom_text_edit.dart';
+import 'package:xterm/xterm.dart';
+
+import 'flutter_driver.dart';
+
+const _escape = (LogicalKeyboardKey.escape, PhysicalKeyboardKey.escape);
+const _enter = (LogicalKeyboardKey.enter, PhysicalKeyboardKey.enter);
+
+void _check(bool ok, String message) {
+ if (!ok) throw StateError(message);
+}
+
+TextEditingValue _editingValue() {
+ final terminal = benchmarkFind(
+ (w) => w is CustomTextEdit && w.focusNode.hasFocus,
+ );
+ if (terminal != null) {
+ final state = (terminal as StatefulElement).state as CustomTextEditState;
+ _check(
+ state.hasInputConnection,
+ 'Focused terminal has no text input connection',
+ );
+ return state.currentTextEditingValue!;
+ }
+ final field = benchmarkFind((w) => w is EditableText && w.focusNode.hasFocus);
+ if (field != null) return (field.widget as EditableText).controller.value;
+ throw StateError(
+ 'No focused text input: ${FocusManager.instance.primaryFocus}',
+ );
+}
+
+void _insert(String text, {bool replace = false}) {
+ final value = _editingValue();
+ final selection = replace
+ ? TextSelection(baseOffset: 0, extentOffset: value.text.length)
+ : value.selection;
+ _check(selection.isValid, 'Text input has no valid selection');
+ final updated = value.text.replaceRange(selection.start, selection.end, text);
+ TextInput.updateEditingValue(
+ TextEditingValue(
+ text: updated,
+ selection: TextSelection.collapsed(offset: selection.start + text.length),
+ ),
+ );
+}
+
+Future<void> runCoreBenchmark(
+ AppNotifier app,
+ String output,
+ Map<String, Object?> metadata,
+ Map<int, ui.FrameTiming> timings, {
+ required int samples,
+ required List<(String, String)> inputs,
+ required Future<void> Function(bool) setOutput,
+ required Map<String, int> Function() outputCounters,
+}) async {
+ final rows = <Map<String, Object?>>[];
+ final phases = <Map<String, Object?>>[];
+ final fixtureCount = app.allPanes.length;
+ final firstTab = app.activeSwarmId;
+ final startedAt = DateTime.now().toUtc().toIso8601String();
+ const host = MethodChannel('harness/isolated_benchmark');
+ var initialWindow = await host.invokeMapMethod<String, Object?>(
+ 'captureState',
+ );
+ // Normal app launch can initially be backgrounded. Let the operator raise
+ // this fixture once; never reclaim focus during the measured series.
+ for (
+ var attempt = 0;
+ attempt < 120 &&
+ (initialWindow?['key'] != true || initialWindow?['active'] != true);
+ attempt++
+ ) {
+ await Future<void>.delayed(const Duration(milliseconds: 250));
+ initialWindow = await host.invokeMapMethod<String, Object?>('captureState');
+ }
+ // Warm every retained renderer once, not just the first workspace.
+ for (final tab in app.swarms.toList()) {
+ app.selectSwarm(tab.id, attachPending: false);
+ await benchmarkFrame();
+ }
+ app.selectSwarm(firstTab, attachPending: false);
+ await benchmarkFrame();
+
+ Future<void> terminalInput() async {
+ final before = inputs.length;
+ final expected = app.focusedPane?.session?.agentId;
+ _check(expected != null, 'No destination terminal');
+ _insert('x');
+ await benchmarkFrame();
+ _check(
+ inputs.length == before + 1 && inputs.last == (expected, 'x'),
+ 'Input was lost, duplicated, or sent to the wrong terminal',
+ );
+ }
+
+ Future<void> measure(
+ String operation,
+ String load,
+ int sample,
+ Future<void> Function() action,
+ ) async {
+ final previousFrame = await benchmarkFrame();
+ final delay = (sample + 5) * 11003 % 20000;
+ await Future<void>.delayed(Duration(microseconds: delay));
+ final began = DateTime.now().microsecondsSinceEpoch;
+ await action();
+ // action has awaited the frame that validates its state. Capture that exact
+ // frame, not a later unrelated redraw, and join the renderer's timing below.
+ rows.add({
+ 'operation': operation,
+ 'load': load,
+ 'phase': sample < 0 ? 'warmup' : 'measured',
+ 'startedWallMicros': began,
+ 'previousFrame': previousFrame,
+ 'readyFrame': ui.PlatformDispatcher.instance.frameData.frameNumber,
+ 'phaseDelayMicros': delay,
+ });
+ }
+
+ Future<void> dismissNew() async {
+ final element = benchmarkFind((w) => w is NewHarnessBox);
+ if (element == null) return;
+ final controller = (element.widget as NewHarnessBox).controller;
+ for (
+ var step = 0;
+ step < 8 && controller.field != NewHarnessField.launch;
+ step++
+ ) {
+ benchmarkKey(_escape, command: false);
+ await benchmarkFrame();
+ }
+ benchmarkKey(_escape, command: false);
+ await benchmarkFrame();
+ _check(
+ benchmarkFind((w) => w is NewHarnessBox) == null,
+ 'New harness did not close',
+ );
+ }
+
+ try {
+ _check(
+ initialWindow?['key'] == true && initialWindow?['active'] == true,
+ 'Core measurement requires the foreground benchmark window: $initialWindow',
+ );
+ for (final load in ['idle', 'all_terminals_20hz']) {
+ await setOutput(load != 'idle');
+ final phaseBegan = DateTime.now().microsecondsSinceEpoch;
+ final countersBefore = outputCounters();
+ for (final operation in [
+ 'typing_echo',
+ 'cmd_n',
+ 'cmd_o',
+ 'picker_query',
+ 'picker_accept_input',
+ 'cmd_t',
+ if (fixtureCount > 4) 'tab_input',
+ if (fixtureCount > 1) ...['focus_input', 'zoom'],
+ 'find_query',
+ 'scroll_page',
+ ]) {
+ for (var sample = -5; sample < samples; sample++) {
+ final originalTab = app.activeSwarmId;
+ final tabsBefore = app.swarms.length;
+ SwarmSearchInput? picker;
+ if (operation == 'picker_query' ||
+ operation == 'picker_accept_input') {
+ benchmarkKey((LogicalKeyboardKey.keyO, PhysicalKeyboardKey.keyO));
+ await benchmarkFrame();
+ picker =
+ benchmarkFind((w) => w is SwarmSearchInput && w.search != null)
+ ?.widget
+ as SwarmSearchInput?;
+ _check(
+ picker != null && picker.focusNode.hasFocus,
+ 'Picker did not take input focus',
+ );
+ if (operation == 'picker_accept_input') {
+ // Picking an agent from a different workspace can add a tile.
+ // Focus another retained tile here so the declared fixture size
+ // and visible count remain constant through the entire series.
+ final target = app.panes
+ .firstWhere(
+ (pane) => pane.id != app.focusedPaneId,
+ orElse: () => app.panes.first,
+ )
+ .agentId!;
+ _insert('Fixture agent ${target.split('-').last}', replace: true);
+ await benchmarkFrame();
+ _check(
+ picker!.search!.selected?.agentId == target,
+ 'Picker selected an unexpected terminal',
+ );
+ }
+ } else if (operation == 'find_query') {
+ benchmarkKey((LogicalKeyboardKey.keyF, PhysicalKeyboardKey.keyF));
+ await benchmarkFrame();
+ _insert('', replace: true);
+ await benchmarkFrame();
+ } else if (operation == 'focus_input') {
+ app.focusPaneByIndex(0);
+ await benchmarkFrame();
+ }
+
+ await measure(operation, load, sample, () async {
+ switch (operation) {
+ case 'typing_echo':
+ await terminalInput();
+ case 'cmd_n':
+ benchmarkKey((
+ LogicalKeyboardKey.keyN,
+ PhysicalKeyboardKey.keyN,
+ ));
+ await benchmarkFrame();
+ _check(
+ benchmarkFind((w) => w is NewHarnessBox) != null,
+ 'Cmd+N did not open',
+ );
+ _editingValue(); // The visible surface must also own editable focus.
+ case 'cmd_o':
+ benchmarkKey((
+ LogicalKeyboardKey.keyO,
+ PhysicalKeyboardKey.keyO,
+ ));
+ await benchmarkFrame();
+ final input =
+ benchmarkFind(
+ (w) => w is SwarmSearchInput && w.search != null,
+ )?.widget
+ as SwarmSearchInput?;
+ _check(
+ input != null && input.focusNode.hasFocus,
+ 'Cmd+O is not input-ready',
+ );
+ case 'picker_query':
+ _insert('Fixture agent 0', replace: true);
+ await benchmarkFrame();
+ _check(
+ picker!.search!.query == 'Fixture agent 0' &&
+ picker.search!.selected?.agentId == 'fixture-0',
+ 'Query did not update results',
+ );
+ case 'picker_accept_input':
+ final expected = picker!.search!.selected!.agentId;
+ benchmarkKey(_enter, command: false);
+ await benchmarkFrame();
+ _check(
+ app.focusedPane?.agentId == expected,
+ 'Picker did not focus the selected terminal',
+ );
+ await terminalInput();
+ case 'cmd_t':
+ benchmarkKey((
+ LogicalKeyboardKey.keyT,
+ PhysicalKeyboardKey.keyT,
+ ));
+ await benchmarkFrame();
+ _check(
+ app.activeSwarmId != originalTab &&
+ app.swarms.length == tabsBefore + 1 &&
+ app.panes.isEmpty &&
+ benchmarkFind((w) => w is WorkspaceWelcome) != null,
+ 'Cmd+T did not create its welcome tab',
+ );
+ case 'tab_input':
+ benchmarkKey((
+ LogicalKeyboardKey.bracketRight,
+ PhysicalKeyboardKey.bracketRight,
+ ), shift: true);
+ await benchmarkFrame();
+ _check(app.activeSwarmId != originalTab, 'Tab did not change');
+ await terminalInput();
+ case 'focus_input':
+ benchmarkKey((
+ LogicalKeyboardKey.arrowRight,
+ PhysicalKeyboardKey.arrowRight,
+ ));
+ await benchmarkFrame();
+ _check(
+ app.focusedPaneId == app.panes[1].id,
+ 'Pane focus did not move',
+ );
+ await terminalInput();
+ case 'zoom':
+ benchmarkKey(_enter);
+ await benchmarkFrame();
+ _check(app.zoomedPaneId != null, 'Pane did not zoom');
+ case 'find_query':
+ _insert('project context', replace: true);
+ // The text callback owns query changes; await the real index's
+ // completion before claiming that the result frame is ready.
+ final find =
+ benchmarkFind(
+ (w) => w is TerminalFindBar && w.search != null,
+ )?.widget
+ as TerminalFindBar?;
+ _check(
+ find?.search?.query == 'project context',
+ 'Find query did not reach its index',
+ );
+ await find!.search!.settled.timeout(const Duration(seconds: 5));
+ await benchmarkFrame();
+ _check(
+ find.search!.count > 0,
+ 'Find did not return a real match',
+ );
+ case 'scroll_page':
+ final view =
+ benchmarkFind(
+ (w) =>
+ w is TerminalView &&
+ w.focusNode?.hasFocus == true,
+ )?.widget
+ as TerminalView?;
+ final controller = view?.scrollController;
+ _check(
+ controller != null && controller.hasClients,
+ 'Focused terminal has no scroll position',
+ );
+ final position = controller!.position;
+ final target = (position.pixels - position.viewportDimension)
+ .clamp(position.minScrollExtent, position.maxScrollExtent);
+ _check(target != position.pixels, 'Scroll did not move');
+ controller.jumpTo(target);
+ await benchmarkFrame();
+ _check(
+ (controller.offset - target).abs() < 1,
+ 'Scroll destination is wrong',
+ );
+ }
+ });
+
+ if (operation == 'cmd_n') {
+ await dismissNew();
+ } else if ([
+ 'cmd_o',
+ 'picker_query',
+ 'find_query',
+ ].contains(operation)) {
+ benchmarkKey(_escape, command: false);
+ await benchmarkFrame();
+ } else if (operation == 'cmd_t') {
+ benchmarkKey((LogicalKeyboardKey.keyW, PhysicalKeyboardKey.keyW));
+ await benchmarkFrame();
+ _check(
+ app.swarms.length == tabsBefore,
+ 'Benchmark tab was not closed',
+ );
+ app.selectSwarm(originalTab, attachPending: false);
+ await benchmarkFrame();
+ } else if (operation == 'zoom') {
+ benchmarkKey(_enter);
+ await benchmarkFrame();
+ _check(app.zoomedPaneId == null, 'Zoom did not restore');
+ } else if (operation == 'scroll_page') {
+ final view =
+ benchmarkFind(
+ (w) => w is TerminalView && w.focusNode?.hasFocus == true,
+ )!.widget
+ as TerminalView;
+ view.scrollController!.jumpTo(
+ view.scrollController!.position.maxScrollExtent,
+ );
+ await benchmarkFrame();
+ }
+ _check(
+ app.allPanes.length == fixtureCount &&
+ app.panes.length == (fixtureCount < 4 ? fixtureCount : 4),
+ 'Operation changed the declared terminal or visible-pane count',
+ );
+ }
+ // A progress artifact distinguishes this invocation from any stale file.
+ await File('$output.progress').writeAsString(
+ jsonEncode({
+ 'startedAt': startedAt,
+ 'load': load,
+ 'completed': operation,
+ 'observations': rows.length,
+ }),
+ );
+ }
+ await setOutput(false);
+ final after = outputCounters();
+ phases.add({
+ 'load': load,
+ 'durationMicros': DateTime.now().microsecondsSinceEpoch - phaseBegan,
+ for (final key in after.keys) key: after[key]! - countersBefore[key]!,
+ });
+ }
+ for (
+ var attempt = 0;
+ attempt < 30 && rows.any((r) => !timings.containsKey(r['readyFrame']));
+ attempt++
+ ) {
+ await Future<void>.delayed(const Duration(milliseconds: 100));
+ await benchmarkFrame();
+ }
+ for (final row in rows) {
+ final timing = timings[row['readyFrame']];
+ _check(timing != null, 'Missing exact renderer frame');
+ row['readyRasterMicros'] =
+ timing!.timestampInMicroseconds(ui.FramePhase.rasterFinishWallTime) -
+ (row['startedWallMicros'] as int);
+ row['buildMicros'] = timing.buildDuration.inMicroseconds;
+ row['rasterMicros'] = timing.rasterDuration.inMicroseconds;
+ _check((row['readyRasterMicros'] as int) >= 0, 'Invalid frame timestamp');
+ }
+ final finalWindow = await host.invokeMapMethod<String, Object?>(
+ 'captureState',
+ );
+ _check(
+ finalWindow?['focusLosses'] == initialWindow?['focusLosses'] &&
+ finalWindow?['key'] == true &&
+ finalWindow?['active'] == true,
+ 'Benchmark lost foreground focus; preserve this run but do not accept timings',
+ );
+ metadata['initialWindow'] = initialWindow;
+ metadata['finalWindow'] = finalWindow;
+ await File(output).writeAsString(
+ const JsonEncoder.withIndent(' ').convert({
+ 'success': true,
+ 'kind': 'release_core_framework_dispatch_to_raster',
+ 'startedAt': startedAt,
+ 'finishedAt': DateTime.now().toUtc().toIso8601String(),
+ 'metadata': metadata,
+ 'terminals': fixtureCount,
+ 'visible': fixtureCount < 4 ? fixtureCount : 4,
+ 'initialScrollbackRows': 1000,
+ 'samplesPerOperationPerLoad': samples,
+ 'warmups': 5,
+ 'inputCommitsVerified': inputs.length,
+ 'phases': phases,
+ 'summaries': [
+ for (final load in ['idle', 'all_terminals_20hz'])
+ for (final operation in rows.map((r) => r['operation']).toSet())
+ {
+ 'load': load,
+ 'operation': operation,
+ for (final metric in [
+ 'readyRasterMicros',
+ 'buildMicros',
+ 'rasterMicros',
+ ])
+ metric: benchmarkDistribution([
+ for (final row in rows)
+ if (row['load'] == load &&
+ row['operation'] == operation &&
+ row['phase'] == 'measured')
+ row[metric] as int,
+ ]),
+ },
+ ],
+ 'observations': rows,
+ }),
+ );
+ } catch (error, stack) {
+ await File(output).writeAsString(
+ jsonEncode({
+ 'success': false,
+ 'startedAt': startedAt,
+ 'error': '$error',
+ 'stack': '$stack',
+ 'observations': rows,
+ }),
+ );
+ rethrow;
+ } finally {
+ await setOutput(false);
+ }
+}
diff --git a/desktop/tool/native_benchmark/flutter_driver.dart b/desktop/tool/native_benchmark/flutter_driver.dart
index 085be8592..4a117525e 100644
--- a/desktop/tool/native_benchmark/flutter_driver.dart
+++ b/desktop/tool/native_benchmark/flutter_driver.dart
@@ -92,6 +92,18 @@ Map<String, num> _distribution(List<int> values) {
};
}
+// Shared by the core-experience fixture; the original benchmark cadence and
+// operations remain unchanged for comparisons with its published results.
+void benchmarkKey(
+ (LogicalKeyboardKey, PhysicalKeyboardKey) key, {
+ bool command = true,
+ bool shift = false,
+}) => _key(key, command: command, shift: shift);
+Future<int> benchmarkFrame() => _frame();
+Element? benchmarkFind(bool Function(Widget) matches) => _find(matches);
+Map<String, num> benchmarkDistribution(List<int> values) =>
+ _distribution(values);
+
Future<void> runFlutterDispatchBenchmark(
AppNotifier app,
String output,
diff --git a/desktop/tool/native_benchmark/main.dart b/desktop/tool/native_benchmark/main.dart
index f0d9a1b30..6ab905020 100644
--- a/desktop/tool/native_benchmark/main.dart
+++ b/desktop/tool/native_benchmark/main.dart
@@ -24,6 +24,7 @@ import 'package:harness/terminal/terminal_binary.dart';
import 'package:harness/terminal/terminal_session.dart';
import 'flutter_driver.dart';
+import 'core_driver.dart';
const _host = MethodChannel('harness/isolated_benchmark');
final _timings = <int, ui.FrameTiming>{};
@@ -37,6 +38,8 @@ int _skippedBursts = 0;
int _inputBytes = 0;
final _interactive = Platform.environment['HARNESS_BENCH_MANUAL'] == '1';
final _flutterDriven = Platform.environment['HARNESS_BENCH_FLUTTER'] == '1';
+final _coreDriven = Platform.environment['HARNESS_BENCH_CORE'] == '1';
+final _coreInputs = <(String, String)>[];
class _Sample {
_Sample(
@@ -146,8 +149,14 @@ Future<AppNotifier> _fixture(int count, Directory stateDirectory) async {
send: (_, _) async => true,
sendBinary: (packet) async {
if (packet.kind != TerminalBinaryKind.input) return true;
- if (_interactive || _flutterDriven) {
+ if (_interactive || _flutterDriven || _coreDriven) {
+ if (_coreDriven && !identical(app.focusedPane?.session, session)) {
+ throw StateError('Input reached an unfocused terminal');
+ }
await _output(session, packet.bytes);
+ if (_coreDriven) {
+ _coreInputs.add((session.agentId, utf8.decode(packet.bytes)));
+ }
return true;
}
final sample = _pending;
@@ -277,13 +286,42 @@ Future<void> _run() async {
!env.containsKey('FLUTTER_TEST')) {
throw StateError('Use the isolated native benchmark runner');
}
- final output = env['HARNESS_BENCH_OUTPUT'];
+ var output = env['HARNESS_BENCH_OUTPUT'];
if (output == null || !output.startsWith('/private/tmp/')) {
throw StateError('A private temporary output path is required');
}
- final count = int.parse(env['HARNESS_BENCH_TERMINALS'] ?? '16');
- final observations = int.parse(env['HARNESS_BENCH_SAMPLES'] ?? '120');
- if (![16, 48].contains(count) || observations < 1 || observations > 500) {
+ var count = int.parse(env['HARNESS_BENCH_TERMINALS'] ?? '16');
+ var observations = int.parse(env['HARNESS_BENCH_SAMPLES'] ?? '120');
+ var hold = env['HARNESS_BENCH_HOLD'] == '1';
+ if (_coreDriven) {
+ final root = File(output).parent;
+ final config = File('${root.path}/run-config.json');
+ if (config.existsSync()) {
+ final values =
+ jsonDecode(await config.readAsString()) as Map<String, dynamic>;
+ count = values['terminals'] as int;
+ observations = values['samples'] as int;
+ hold = values['hold'] as bool;
+ final name = values['output'] as String;
+ if (!RegExp(r'^[a-zA-Z0-9_-]+\.json$').hasMatch(name)) {
+ throw StateError('Core output must be a simple JSON filename');
+ }
+ output = '${root.path}/$name';
+ }
+ if (File(output).existsSync()) {
+ throw StateError('Use a fresh core output filename');
+ }
+ await File('${root.path}/active-run.json').writeAsString(
+ jsonEncode({
+ 'output': output,
+ 'terminals': count,
+ 'samples': observations,
+ 'pid': pid,
+ 'startedAt': DateTime.now().toUtc().toIso8601String(),
+ }),
+ );
+ }
+ if (![1, 16, 48].contains(count) || observations < 1 || observations > 500) {
throw StateError('Invalid fixture size');
}
final binding = WidgetsFlutterBinding.ensureInitialized();
@@ -298,6 +336,8 @@ Future<void> _run() async {
if (metadata?['bundle'] != 'ai.autonomous.harness.benchmark') {
throw StateError('Wrong native host');
}
+ metadata!['pid'] = pid;
+ metadata['sourceRevision'] = env['HARNESS_BENCH_REVISION'] ?? 'unspecified';
final app = await _fixture(
count,
Directory(
@@ -324,9 +364,75 @@ Future<void> _run() async {
await _frame();
if (_interactive) return;
await Future<void>.delayed(const Duration(seconds: 1));
+ if (_coreDriven) {
+ final packet = utf8.encode(
+ '\x1b7\x1b[1;1H${List.filled(8, '${List.filled(110, 'o').join()}\x1b[K\r\n').join()}\x1b8',
+ );
+ Future<void> setOutput(bool active) async {
+ _outputTimer?.cancel();
+ _outputTimer = null;
+ while (_burstPending) {
+ await Future<void>.delayed(Duration.zero);
+ }
+ if (!active) return;
+ _outputTimer = Timer.periodic(const Duration(milliseconds: 50), (
+ _,
+ ) async {
+ if (_burstPending) {
+ _skippedBursts++;
+ return;
+ }
+ _burstPending = true;
+ try {
+ final sessions = app.allPanes
+ .map((p) => p.session)
+ .whereType<TerminalSession>()
+ .toList();
+ await Future.wait([
+ for (final session in sessions) _output(session, packet),
+ ]);
+ _outputBytes += packet.length * sessions.length;
+ } finally {
+ _burstPending = false;
+ }
+ });
+ }
+
+ try {
+ await runCoreBenchmark(
+ app,
+ output,
+ metadata,
+ _timings,
+ samples: observations,
+ inputs: _coreInputs,
+ setOutput: setOutput,
+ outputCounters: () => {
+ 'outputBytes': _outputBytes,
+ 'skippedBursts': _skippedBursts,
+ },
+ );
+ } catch (error) {
+ stderr.writeln('Core benchmark failed: $error');
+ await File('$output.failure').writeAsString(
+ jsonEncode({
+ 'success': false,
+ 'at': DateTime.now().toUtc().toIso8601String(),
+ 'error': '$error',
+ 'metadata': metadata,
+ }),
+ );
+ } finally {
+ await setOutput(false);
+ if (!hold) {
+ await _host.invokeMethod<void>('finish');
+ }
+ }
+ return;
+ }
if (_flutterDriven) {
try {
- await runFlutterDispatchBenchmark(app, output, metadata!, _timings);
+ await runFlutterDispatchBenchmark(app, output, metadata, _timings);
} catch (error, stack) {
await File(output).writeAsString(
jsonEncode({'success': false, 'error': '$error', 'stack': '$stack'}),
@@ -339,7 +445,7 @@ Future<void> _run() async {
final random = Random(77);
final phases = <Map<String, Object?>>[];
try {
- metadata!['initialResponder'] = await _host.invokeMethod<String>('begin');
+ metadata['initialResponder'] = await _host.invokeMethod<String>('begin');
for (final op in ['typing', 'focus', 'tab']) {
await _sample(app, op, 'cold_interaction', 'idle');
}
diff --git a/desktop/tool/native_benchmark/prepare.py b/desktop/tool/native_benchmark/prepare.py
index 3ea788b9c..0834c83c6 100644
--- a/desktop/tool/native_benchmark/prepare.py
+++ b/desktop/tool/native_benchmark/prepare.py
@@ -1,6 +1,7 @@
#!/usr/bin/env python3
"""Build an isolated Release benchmark; never rewrite the user's V2 bundle."""
import argparse
+import json
import os
from pathlib import Path
import shutil
@@ -18,7 +19,16 @@ mode.add_argument('--flutter-dispatch', action='store_true',
help='Benchmark Flutter key dispatch and release raster, excluding OS input delivery')
mode.add_argument('--primary-workflows', action='store_true',
help='Benchmark Cmd+N/O/T and tab switching with varied input phase')
+mode.add_argument('--core-workflows', action='store_true',
+ help='Validate and measure typing, navigation, search, and scrolling at idle and under output')
+parser.add_argument('--terminals', type=int, choices=[1, 16, 48], default=16)
+parser.add_argument('--samples', type=int, default=120)
+parser.add_argument('--hold', action='store_true', help='Keep core fixture open after measurements for process resource sampling')
args = parser.parse_args()
+if not 1 <= args.samples <= 500:
+ parser.error('--samples must be 1–500')
+if args.hold and not args.core_workflows:
+ parser.error('--hold requires --core-workflows')
source = Path(__file__).resolve().parents[2]
root = Path(tempfile.mkdtemp(prefix='harness-native-benchmark-', dir='/private/tmp'))
desktop = root / 'desktop'
@@ -29,15 +39,20 @@ marker = ' super.awakeFromNib()'
if code.count(marker) != 1:
raise RuntimeError('Native benchmark insertion point changed')
code = code.replace(marker, ' NativeBenchmark.install(window: self, messenger: flutterViewController.engine.binaryMessenger)\n' + marker)
-if args.interactive or args.flutter_dispatch or args.primary_workflows:
+if args.interactive or args.flutter_dispatch or args.primary_workflows or args.core_workflows:
# Only this copied host gets fixture state. Launching through normal app
# controls needs no shell environment and never opens a real transport.
setup = '\n'.join(f' setenv("{key}", "{value}", 1)' for key, value in {
'FLUTTER_TEST': '1',
'HARNESS_NATIVE_BENCHMARK': '1',
'HARNESS_BENCH_MANUAL': '1' if args.interactive else '0',
- 'HARNESS_BENCH_FLUTTER': '0' if args.interactive else '1',
+ 'HARNESS_BENCH_FLUTTER': '1' if args.flutter_dispatch or args.primary_workflows else '0',
'HARNESS_BENCH_PRIMARY': '1' if args.primary_workflows else '0',
+ 'HARNESS_BENCH_CORE': '1' if args.core_workflows else '0',
+ 'HARNESS_BENCH_TERMINALS': str(args.terminals),
+ 'HARNESS_BENCH_SAMPLES': str(args.samples),
+ 'HARNESS_BENCH_HOLD': '1' if args.hold else '0',
+ 'HARNESS_BENCH_REVISION': subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=source, text=True).strip(),
'HARNESS_BENCH_OUTPUT': str(root / 'interactive.json'),
}.items())
engine = ' let flutterViewController = FlutterViewController()'
@@ -46,6 +61,11 @@ if args.interactive or args.flutter_dispatch or args.primary_workflows:
code = code.replace(engine, setup + '\n' + engine)
code += '\n' + (source / 'tool/native_benchmark/NativeBenchmark.swift').read_text()
window.write_text(code)
+if args.core_workflows:
+ (root / 'run-config.json').write_text(json.dumps({
+ 'terminals': args.terminals, 'samples': args.samples,
+ 'hold': args.hold, 'output': 'core-first.json',
+ }, indent=2) + '\n')
info = desktop / 'macos/Runner/Configs/AppInfo.xcconfig'
info.write_text(benchmark_configuration(info.read_text()))
env = dict(os.environ, XDG_CONFIG_HOME=str(root / 'tool-config'))
diff --git a/desktop/tool/native_benchmark/process_usage.swift b/desktop/tool/native_benchmark/process_usage.swift
new file mode 100644
index 000000000..857bdad0f
--- /dev/null
+++ b/desktop/tool/native_benchmark/process_usage.swift
@@ -0,0 +1,94 @@
+// Standalone macOS process accounting. Does not launch, focus, or control apps.
+// Build: xcrun swiftc process_usage.swift -o /private/tmp/harness-process-usage
+// Run: /private/tmp/harness-process-usage PID SECONDS LABEL OUTPUT.json
+import Foundation
+import Darwin
+
+func fail(_ message: String) -> Never {
+ FileHandle.standardError.write(Data((message + "\n").utf8))
+ exit(1)
+}
+
+let args = CommandLine.arguments
+guard args.count == 5, let target = Int32(args[1]), target > 1,
+ let seconds = Int(args[2]), (5...1800).contains(seconds) else {
+ fail("Usage: process_usage PID SECONDS(5–1800) LABEL OUTPUT.json")
+}
+guard !FileManager.default.fileExists(atPath: args[4]) else {
+ fail("Output already exists; preserve previous observations")
+}
+
+struct Sample: Codable {
+ let elapsedSeconds: Double
+ let userNanoseconds: UInt64
+ let systemNanoseconds: UInt64
+ let physicalFootprintBytes: UInt64
+ let residentBytes: UInt64
+ let interruptWakeups: UInt64
+ let packageIdleWakeups: UInt64
+ let diskReadBytes: UInt64
+ let diskWriteBytes: UInt64
+}
+
+let began = ProcessInfo.processInfo.systemUptime
+let startedAt = ISO8601DateFormatter().string(from: Date())
+func sample() -> Sample {
+ var usage = rusage_info_v4()
+ let status = withUnsafeMutablePointer(to: &usage) { pointer in
+ pointer.withMemoryRebound(to: rusage_info_t?.self, capacity: 1) {
+ proc_pid_rusage(target, RUSAGE_INFO_V4, $0)
+ }
+ }
+ guard status == 0 else { fail("proc_pid_rusage failed (errno \(errno)); process may have exited") }
+ return Sample(
+ elapsedSeconds: ProcessInfo.processInfo.systemUptime - began,
+ userNanoseconds: usage.ri_user_time,
+ systemNanoseconds: usage.ri_system_time,
+ physicalFootprintBytes: usage.ri_phys_footprint,
+ residentBytes: usage.ri_resident_size,
+ interruptWakeups: usage.ri_interrupt_wkups,
+ packageIdleWakeups: usage.ri_pkg_idle_wkups,
+ diskReadBytes: usage.ri_diskio_bytesread,
+ diskWriteBytes: usage.ri_diskio_byteswritten)
+}
+
+var samples = [sample()]
+for tick in 1...seconds {
+ let remaining = began + Double(tick) - ProcessInfo.processInfo.systemUptime
+ if remaining > 0 { Thread.sleep(forTimeInterval: remaining) }
+ samples.append(sample())
+}
+let first = samples.first!
+let last = samples.last!
+let elapsed = last.elapsedSeconds - first.elapsedSeconds
+let cpuSeconds = Double(last.userNanoseconds - first.userNanoseconds
+ + last.systemNanoseconds - first.systemNanoseconds) / 1_000_000_000
+let footprints = samples.map { Double($0.physicalFootprintBytes) / 1_048_576 }.sorted()
+let summary: [String: Any] = [
+ "elapsedSeconds": elapsed,
+ "cpuPercentOneCore": cpuSeconds / elapsed * 100,
+ "cpuSeconds": cpuSeconds,
+ "physicalFootprintMiBMedian": footprints[footprints.count / 2],
+ "physicalFootprintMiBPeak": footprints.last!,
+ "physicalFootprintMiBStart": Double(first.physicalFootprintBytes) / 1_048_576,
+ "physicalFootprintMiBEnd": Double(last.physicalFootprintBytes) / 1_048_576,
+ "interruptWakeupsPerSecond": Double(last.interruptWakeups - first.interruptWakeups) / elapsed,
+ "packageIdleWakeupsPerSecond": Double(last.packageIdleWakeups - first.packageIdleWakeups) / elapsed,
+ "diskReadBytes": last.diskReadBytes - first.diskReadBytes,
+ "diskWriteBytes": last.diskWriteBytes - first.diskWriteBytes,
+]
+let encodedSamples = try JSONEncoder().encode(samples)
+let output: [String: Any] = [
+ "schema": 1, "success": true, "label": args[3], "pid": target,
+ "startedAt": startedAt, "os": ProcessInfo.processInfo.operatingSystemVersionString,
+ "boundary": "proc_pid_rusage for this process only; CPU 100% = one core; excludes daemon, agents, and GPU energy",
+ "sampleIntervalSeconds": 1, "summary": summary,
+ "samples": try JSONSerialization.jsonObject(with: encodedSamples),
+]
+let data = try JSONSerialization.data(withJSONObject: output, options: [.prettyPrinted, .sortedKeys])
+guard FileManager.default.createFile(atPath: args[4], contents: data,
+ attributes: [.posixPermissions: 0o600]) else {
+ fail("Could not write result")
+}
+let summaryData = try JSONSerialization.data(withJSONObject: summary, options: [.sortedKeys])
+print(String(data: summaryData, encoding: .utf8)!)