Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

Keep native benchmark isolated after Harness bundle rename

deefe42f2ebad08
diff --git a/desktop/tool/native_benchmark/README.md b/desktop/tool/native_benchmark/README.md
index 58d33f099..4acee37c5 100644
--- a/desktop/tool/native_benchmark/README.md
+++ b/desktop/tool/native_benchmark/README.md
@@ -1,11 +1,14 @@
# Native Release interaction benchmark
**Status, September 14, 2026:** the builder supports the current Harness name,
-validates the copied product identity and verifies the resulting bundle. The
-runner identifies current/legacy previews by bundle ID, and initial native
-focus uses the same Flutter controller as the production titlebar. Six Python
-isolation checks cover current/legacy names, unknown/duplicate configuration,
-running previews and the built bundle identity. See the
+validates the copied product identity and verifies the resulting bundle. It
+accepts both the current `ai.autonomous.harness` and legacy `.v2` source IDs.
+Since the preview and installed app now share their ID, the runner also checks
+bundle location: only the release identity in `/Applications` or the current
+user's `Applications` folder is exempt. Development copies, legacy previews
+and other benchmark processes still stop preflight. Initial native focus uses
+the same Flutter controller as the production titlebar. Nine Python isolation
+checks include the actual production config and this renamed-build regression. See the
[handoff](../../../docs/harness-v2-handoff.md) and
[failed calibration notes](../../../docs/harness-v2-performance.md#native-calibration-remains-unmeasured-2026-09-13).
Foreground/key-window guards remain intact; no p50/p95/p99 result has been accepted.
@@ -15,7 +18,7 @@ screen, terminal session parser and Flutter renderer. It runs as **Harness
Benchmark**, in an isolated copy with its own bundle ID, synthetic transports,
blocked HTTP and temporary state. It never reads the user's saved Swarms or sends
input to an agent. The benchmark bridge is appended only to the copied native
-host; it is absent from the production Runner and V2 bundle.
+host; it is absent from the production Runner and Harness bundle.
From `desktop/`, build with a compatible Flutter SDK and Xcode:
@@ -27,8 +30,9 @@ Use the `BENCHMARK_APP` path printed by the build. Normally close the workspace
other builds/tests first. Keep this fixture in the foreground during a run; it
exits on focus loss instead of reclaiming focus between observations. The runner
refuses to start alongside another preview or benchmark process and never quits
-them. Its error names the exact bundle path: the installed Harness app and the
-workspace preview share the name Harness but have distinct bundle identities.
+them. Its error names the exact bundle path. A development copy outside the
+standard installation folders is not exempt just because it shares the installed
+Harness app's name and bundle identifier.
The installed app may remain open; record other app activity and host load when
reporting timings. That is not a guarantee of an otherwise idle workstation.
@@ -47,7 +51,7 @@ python3 tool/native_benchmark/run.py \
Repeat with `--terminals 48` and a fresh output path. Use `--samples 3` only for
calibration, not percentile claims. The fixture closes itself normally when it
-finishes; reopen V2 afterward. Launch through `run.py`, which uses Launch Services
+finishes; reopen the workspace preview afterward. Launch through `run.py`, which uses Launch Services
and supplies the required environment. Opening the fixture without that
environment fails immediately.
diff --git a/desktop/tool/native_benchmark/isolation.py b/desktop/tool/native_benchmark/isolation.py
index 09b06bfec..5230367bd 100644
--- a/desktop/tool/native_benchmark/isolation.py
+++ b/desktop/tool/native_benchmark/isolation.py
@@ -15,7 +15,7 @@ def benchmark_configuration(source):
"""Replace exactly one known product/identifier in the copied config."""
replacements = (
('PRODUCT_NAME', ('Harness', 'Harness V2'), BENCHMARK_NAME),
- ('PRODUCT_BUNDLE_IDENTIFIER', (PREVIEW_ID,), BENCHMARK_ID),
+ ('PRODUCT_BUNDLE_IDENTIFIER', (PREVIEW_ID, RELEASE_ID), BENCHMARK_ID),
)
for key, expected, replacement in replacements:
pattern = re.compile(rf'^{key}[ \t]*=[ \t]*(.*)$', re.MULTILINE)
@@ -40,7 +40,7 @@ def validate_benchmark_bundle(app):
def conflicting_previews(executable_paths):
- """Read bundle identities, since the installed app and preview share a name."""
+ """Identify builds by bundle and location; production now shares their ID."""
conflicts = []
for line in executable_paths.splitlines():
executable = Path(line.strip())
@@ -55,6 +55,16 @@ def conflicting_previews(executable_paths):
raise ValueError(f'Cannot identify running {app.name}: {error}') from error
if identifier in (PREVIEW_ID, BENCHMARK_ID):
conflicts.append(str(app))
- elif identifier != RELEASE_ID:
+ elif identifier == RELEASE_ID:
+ installed_directories = {
+ Path('/Applications').resolve(),
+ (Path.home() / 'Applications').resolve(),
+ }
+ # A renamed development build must not become exempt merely by
+ # gaining the installed app's bundle identifier. Standard install
+ # locations remain allowed, as before; report other copies.
+ if app.parent.resolve() not in installed_directories:
+ conflicts.append(str(app))
+ else:
raise ValueError(f'Unrecognized running Harness identity: {identifier}')
return conflicts
diff --git a/desktop/tool/native_benchmark/test_isolation.py b/desktop/tool/native_benchmark/test_isolation.py
index 16d352c98..4756ae46c 100644
--- a/desktop/tool/native_benchmark/test_isolation.py
+++ b/desktop/tool/native_benchmark/test_isolation.py
@@ -2,6 +2,7 @@ from pathlib import Path
import plistlib
import tempfile
import unittest
+from unittest.mock import patch
from isolation import (
BENCHMARK_ID, BENCHMARK_NAME, PREVIEW_ID, RELEASE_ID,
@@ -25,17 +26,23 @@ class IsolationTests(unittest.TestCase):
def test_current_and_legacy_product_names_are_replaced(self):
for name in ('Harness', 'Harness V2'):
- with self.subTest(name=name):
- source = f'// product\nPRODUCT_NAME = {name}\nPRODUCT_BUNDLE_IDENTIFIER = {PREVIEW_ID}\n'
- result = benchmark_configuration(source)
- self.assertIn(f'PRODUCT_NAME = {BENCHMARK_NAME}\n', result)
- self.assertIn(f'PRODUCT_BUNDLE_IDENTIFIER = {BENCHMARK_ID}\n', result)
- self.assertTrue(result.startswith('// product\n'))
+ for identifier in (PREVIEW_ID, RELEASE_ID):
+ with self.subTest(name=name, identifier=identifier):
+ source = f'// product\nPRODUCT_NAME = {name}\nPRODUCT_BUNDLE_IDENTIFIER = {identifier}\n'
+ result = benchmark_configuration(source)
+ self.assertIn(f'PRODUCT_NAME = {BENCHMARK_NAME}\n', result)
+ self.assertIn(f'PRODUCT_BUNDLE_IDENTIFIER = {BENCHMARK_ID}\n', result)
+ self.assertTrue(result.startswith('// product\n'))
+
+ def test_production_configuration_can_build_an_isolated_benchmark(self):
+ source = Path(__file__).resolve().parents[2] / 'macos/Runner/Configs/AppInfo.xcconfig'
+ result = benchmark_configuration(source.read_text())
+ self.assertIn(f'PRODUCT_BUNDLE_IDENTIFIER = {BENCHMARK_ID}\n', result)
def test_unknown_or_duplicate_configuration_fails_closed(self):
known = f'PRODUCT_NAME = Harness\nPRODUCT_BUNDLE_IDENTIFIER = {PREVIEW_ID}\n'
for source in (known.replace('Harness', 'Other'),
- known.replace(PREVIEW_ID, RELEASE_ID),
+ known.replace(PREVIEW_ID, 'unknown.harness'),
known + 'PRODUCT_NAME = Harness\n',
known.replace('PRODUCT_NAME = Harness\n', '')):
with self.subTest(source=source), self.assertRaises(ValueError):
@@ -49,8 +56,21 @@ class IsolationTests(unittest.TestCase):
self.assertEqual(conflicting_previews(str(binary)), [str(bundle)])
def test_installed_app_and_non_app_commands_are_not_preview_processes(self):
- _, binary = self.app('installed', 'Harness', RELEASE_ID)
- self.assertEqual(conflicting_previews(f'{binary}\n/usr/bin/python3\n/bin/zsh'), [])
+ _, binary = self.app('Applications', 'Harness', RELEASE_ID)
+ with patch('isolation.Path.home', return_value=self.root):
+ self.assertEqual(conflicting_previews(f'{binary}\n/usr/bin/python3\n/bin/zsh'), [])
+
+ def test_renamed_development_build_is_not_mistaken_for_installed_app(self):
+ bundle, binary = self.app('code/desktop/build/macos/Build/Products/Release',
+ 'Harness', RELEASE_ID)
+ with patch('isolation.Path.home', return_value=self.root):
+ self.assertEqual(conflicting_previews(str(binary)), [str(bundle)])
+
+ def test_install_location_cannot_exempt_legacy_or_benchmark_identity(self):
+ for name, identifier in (('Harness V2', PREVIEW_ID), (BENCHMARK_NAME, BENCHMARK_ID)):
+ bundle, binary = self.app('Applications', name, identifier)
+ with patch('isolation.Path.home', return_value=self.root):
+ self.assertEqual(conflicting_previews(str(binary)), [str(bundle)])
def test_unknown_or_unreadable_running_harness_fails_closed(self):
bundle, binary = self.app('unknown', 'Harness', 'unexpected')
diff --git a/docs/harness-v2-handoff.md b/docs/harness-v2-handoff.md
index ce502fcf0..19fde8b50 100644
--- a/docs/harness-v2-handoff.md
+++ b/docs/harness-v2-handoff.md
@@ -350,10 +350,14 @@ that its isolated window can become active/key.
local/remote agents: startup, delayed snapshots, resizing, returning to a
scrolled view, reconnect, paste, selection, and IME.
4. **Measure native responsiveness** under representative retained/output load,
- first updating the isolated benchmark's identity checks for the team's
- new `ai.autonomous.harness` macOS bundle identifier (Linux now uses
- `com.autonomous.harness` and executable `harness`). Earlier compatibility
- checks used the old internal `.v2` identifier; do not reuse that calibration.
+ using the repaired benchmark identity checks for the team's new
+ `ai.autonomous.harness` macOS bundle identifier (Linux now uses
+ `com.autonomous.harness` and executable `harness`). Nine isolation tests pass;
+ a current disposable Release fixture is ready at
+ `/private/tmp/harness-native-benchmark-7hv6voyr`. Its runner correctly refuses
+ the running workspace preview. A request to briefly close/reopen that preview
+ for calibration is pending; elapsed time is not approval. No native samples
+ have been accepted. See the current performance record for exact artifacts.
Then close remaining visual/platform/release qualification gaps. Maintain
the main-branch handoff and keep it honest about what's still unverified.
diff --git a/docs/harness-v2-performance.md b/docs/harness-v2-performance.md
index 7e6a1e9e3..a1bb2a286 100644
--- a/docs/harness-v2-performance.md
+++ b/docs/harness-v2-performance.md
@@ -12,6 +12,26 @@ It prints `SWARM_BENCH` JSON records. Its filename deliberately does not end in
## Current continuation benchmark (2026-09-14)
+### Native fixture after the public bundle-ID change
+
+The team's `76a469b` gave the workspace preview the installed app's
+`ai.autonomous.harness` identity. Two regressions reproduced the resulting
+benchmark problems: the builder rejected current source, and preflight mistook
+a development copy for the exempt installed app. The builder now accepts the
+current and legacy source identities but still produces only the distinct
+`Harness Benchmark` product. Preflight additionally requires the release
+identity to be in `/Applications` or the current user's `Applications` folder
+before exempting it. Unknown identities, development copies, legacy previews
+and other benchmark processes remain blocked.
+
+Nine isolated Python tests pass. The current disposable arm64 Release fixture
+built at `/private/tmp/harness-native-benchmark-7hv6voyr`. Its actual runner
+correctly refused the still-running workspace preview at its exact build path;
+the installed copy is classified separately. No native samples were taken.
+The session requested a brief preview-close/idle window for calibration and
+continues independent performance work while that request is pending.
+Artifacts: `/private/tmp/harness-native-identity-{before,tests,prepare,preflight}.log`.
+
### History focus hot path
History now formats only agents with open views. Previously each focus change