Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

11.7 KB
Raw Blame

How it works

One execution system, with several interfaces. Work belongs to the machine running it; opening or closing a view does not determine whether that work runs.

System overview

Conceptual architecture, recorded September 27, 2026. The founder identifies the TUI as part of current development; this diagram describes its intended client role. The web app is now a browser target of the existing desktop Flutter package, available as a public preview at harness.autonomous.ai. Public shared-session pages remain the next acquisition experiment beyond authenticated workspace access and private invitations. The product direction and roadmap describe the larger bet and its proposed sequence.

                            HUMAN
                              |
          +-------------------+-------------------+
          |                   |                   |
     DEEP WORK           QUICK CONTROL        AUTOMATION
          |                   |                   |
   +------+------+     +------+------+         +-----+
   | Desktop     |     | Mobile      |         | CLI |
   | TUI (dev)   |     | Web         |         +--+--+
   +------+------+     +------+------+            |
          |                   |                   |
          +-------------------+-------------------+
                              |
                SHARED COMMAND + EVENT CONTRACT
                start / resume / stop / answer
                status / questions / results
                              |
             +----------------+----------------+
             |                                 |
        Local socket                  Encrypted remote access
             |                        P2P for terminal traffic
             |                        Relay path / fallback
             |                                 |
             v                                 v
   +-----------------------+       +-----------------------+
   | DAEMON: laptop        |       | DAEMON: workstation   |
   |                       |       |                       |
   | Work + harnesses       |       | Work + harnesses       |
   | Status + questions    |       | Status + questions    |
   | Engine adapters       |       | Engine adapters       |
   | Tool/viewer lifecycle |       | Tool/viewer lifecycle |
   +-----------+-----------+       +-----------+-----------+
       ^       |                               |
       |       v                               v
       |  +--------------------+    +--------------------+
       |  | Agent processes    |    | Agent processes    |
       |  | Domain tools       |    | Domain tools       |
       |  | Project files      |    | Project files      |
       |  | Artifacts + checks |    | Artifacts + checks |
       |  +--------------------+    +--------------------+
       |
      USB
       |
   +-----------------------+
   | HARNESS DEVICE        |
   | Glance at progress    |
   | Answer questions      |
   | Speak instructions    |
   +-----------------------+


   SUPPORTING SERVICES
   +----------------------------------------------------+
   | Backend: accounts, machine discovery, shared swarms |
   | Relay: connection signaling + encrypted forwarding |
   | Store: domain packages, tools, skills, viewers      |
   +----------------------------------------------------+

The contract in the diagram is an API boundary, not an extra server. The daemon already exposes commands and events through its local socket. A common contract does not require every client to expose every operation.

The diagram groups interfaces by their main job; the CLI is also useful for interactive administration, and desktop work includes quick decisions. Actual connection paths differ:

  • Desktop and local CLI commands reach the local daemon. Desktop access to another machine goes through that local daemon's encrypted link.
  • Mobile is a standalone client that authenticates and terminates encryption itself. It hosts no agents. See the mobile architecture.
  • Web shares desktop's Flutter screens, state, and terminal renderer in desktop/. It authenticates and terminates encryption in the browser, reaching machines through the relay. Browser sign-in, storage, and native capability adapters are separate; there is no browser daemon. This first target does not negotiate WebRTC.
  • Remote terminal traffic can use direct WebRTC or TURN; the encrypted relay path remains available. Other command/event traffic uses the supported daemon/relay paths.
  • The device communicates over USB with its host daemon. It does not connect to the backend or execute agents.

Ownership boundaries

Component Responsibility
Daemon Authoritative machine execution state, agent lifecycle, question state, engine integration, and tool/viewer processes.
CLI Scriptable operations and machine administration over the execution system.
Desktop Rich interaction, terminal presentation, artifact inspection, and local window layout.
Mobile Remote access, focused interaction, and decisions while away from the desktop.
Device Desk status, questions, and voice input through the host.
TUI, in development Interactive access inside a terminal, using the same operations.
Web, public preview The shared desktop workspace of swarms and harnesses in a browser, including private read-only invitations; public session pages for discovery and acquisition follow separately.
Backend and relay Identity, discovery, selected shared metadata, signaling, and encrypted forwarding.
Domain packages Instructions, skills, toolchain setup, project templates, checks, and viewers.

Window layouts belong to clients; account-level swarm membership is shared through the backend. Runtime truth belongs to the daemon owning the work. A question answered through one interface must resolve for the others. Closing an interface leaves execution running on an available host; it does not make a sleeping or disconnected host capable of continuing work elsewhere.

Companions and Memory have separate implementations in the optional companions package. Core does not construct either subsystem or bundle their workers. Old optional requests receive an unsupported response without changing their transport security boundaries. The package currently builds as libraries; its application startup and client integration remain disconnected. The extraction plan records those remaining requirements. This extraction does not establish a working companion-memory experience.

The current orchestrator already persists local project runs, task dependencies, and artifact handoffs. Cross-machine orchestration, project migration, and universal engine-session portability are separate capabilities; access to a remote terminal does not provide them automatically.

Daemon, sessions, and transport

The daemon runs detached under your account. Every five seconds it reconciles the harness-* tmux sessions with its registry; a pane has to be missing on two scans before its agent is marked gone. It tails each agent's transcript with a byte offset (JSONL for most engines, SQLite for OpenCode, Kilo, Hermes and Devin) and turns lines into a normalized event stream: turn started, tool call, sub-agent, question, turn ended. Hooks it installs into the vendor CLI tell it about session start, prompt submit and stop. Every five minutes it re-reads engine configs and pane footers to keep model and effort right. On start it restores panes that died with the tmux server, and it self-updates from a signed manifest, swapping the bundle atomically.

The session model. The registry (~/.harness/cli/data/registry.json, mode 0600) is the source of truth for agents: engine, working directory, tmux pane, bound transcript, process identity. Layouts belong to the app. The relay stores machine records, agent names and daily counters, never a transcript, a recap or a keystroke.

Transport. Each daemon holds one WebSocket to the relay, authenticated with its SSO token. Terminal bytes ride a binary channel on that socket until a WebRTC data channel negotiates, then move to it. Encryption is on for every path and has no switch:

  • Ed25519 identity keys, pinned at first pairing and signing every ephemeral after it.
  • A CPace-style PAKE over ristretto255 — the six-character pairing code bootstraps a shared secret across the untrusted relay, and an attacker gets one online guess.
  • X25519 ephemeral Diffie–Hellman per connection, through HKDF to pairwise session keys.
  • A per-process group key so one event encrypts once for many readers.
  • ChaCha20-Poly1305 on every frame, with the associated data binding frame type and session.

The crypto core lives in cli/src/lib/e2ee/ and is a byte-identical twin of the browser's copy, with a drift-guard test and committed self-vectors.

Providers, relay, web

  • provider/ — the spec (spec/README.md), the deterministic reference-provider with the conformance runner on port 4319, and example-provider, a real one backed by the local claude CLI on port 4502 (read its README before running it; it skips permissions). provider/e2e runs both.
  • desktop/ — one Flutter package for native desktop and web, with shared product UI and platform adapters. See web development.
  • backend/ — the relay: Node, MongoDB via Prisma, Redis. Its WebSocket paths include /api/adapter-ws for daemons, /api/web-ws, /api/observer-ws for private sharing, /api/device-ws, and /api/manager-ws. It signals WebRTC and hands out STUN/TURN, and persists machines, agent names and counters. npm install && npm run dev on :8085; backend/README.md and .env.example for the rest. harness-api.autonomous.ai is the hosted instance.
  • harness.autonomous.ai is the download page: it hosts the CLI installer and the desktop downloads. It is not in this repository.

The Harness device

A round 466×466 AMOLED with touch and a far-field microphone, USB-C on the bottom edge. It has no WiFi and holds no credential. It is served entirely over the cable by the daemon on the computer it is plugged into; plugging it in is the authorization. The wire is one USB serial device (303a:1001), framed as A5 5A | ver | type | len | payload | crc16 with a JSON vocabulary, a five-second ping, and a hard 8 KiB frame ceiling.

What it shows: your agents as tiles in the order of the window's panes, with what each is doing and for how long; a wheel of your machines; the agent's own question when it asks one, answerable with a tap; the recap when a turn finishes, with one quiet tone. Scroll the face to scroll the terminal. Double-tap and speak to send a task: the audio goes to the daemon as PCM, comes back as a transcript, and Boss mode routes it. A voice turn can carry a mode — /goal runs an instruction to done, /loop on a schedule — adapted per engine; /loop is Claude Code only today.

Firmware updates travel over the same cable in 16 KB credit windows, offered from the published metadata and never for a dev build. harness flash re-flashes a device from a USB port. The firmware is ESP-IDF ≥ 5.5 under devices/harness-device/firmware/ (idf.py set-target esp32s3 && idf.py build); make device-test runs the host-side tests with no board attached.