The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
autonomous-harness-backend — central product API + single-port reverse proxy
The central, public-facing backend for the distributed deployment. It is the one port the web connects to. It:
- Manages users + auth —
POST /api/auth/login {email,password}issues a JWT. An unknown email is created on the spot (login doubles as signup). Admin-only user CRUD under/api/users. - Provisions agent-nodes —
POST /api/agentsgenerates a per-agent api-key, picks a manager by capacity (readsmaxNodes/nodeCountfrom the sharedmanagerscollection; best-fit = the live manager with the least free room, 503 if all full), calls that manager's control API to create the node, and stores theuser → agentId/apiKey/managerUrlbinding.DELETE /api/agents/:agentIdtears it down. The data-plane proxy then targets the binding's manager. - Reverse-proxies the data plane — every other
/api/*request and the/api/wsWebSocket are authenticated by the agent api key (NOT the JWT): the client sends it (x-api-keyfor HTTP, the WS subprotocol) on a/proxy/...path, the backend derivesagentId = sha256(key)[:32], looks up the owning manager, and forwards the path verbatim to it (the manager mounts its data plane under/proxyand strips it before the node). So any client (a CLI, another app, the web chat page) can reach an agent with only its api key. The owner obtains that key from the control plane (#2); possessing it is the auth (128-bit preimage).
control: web ──(JWT)──────────────────────▶ backend :8085 /api/auth|users|agents ──(x-api-key: MANAGER_API_KEY)──▶ manager :8090/api
data: client ──(agent api key: x-api-key / WS subprotocol)──▶ backend :8085 /proxy/api/* ──▶ manager :8090/proxy/api/* ──▶ agent-node/api/*
One port, two planes
A single http.Server (server.ts serverFactory) splits traffic by an explicit path prefix:
- Data plane (reverse-proxied): everything under
/proxy/*(incl. the/proxy/api/wsupgrade) →lib/proxy.ts. - Local (Fastify control API): everything else —
/api/health,/api/auth/*,/api/users/*,/api/agents*. A stray/api/*that isn't a control route now 404s here instead of being proxied.
Run
npm install # runs `prisma generate`
# Needs MongoDB (shared with the agent-manager) + a running agent-manager:
# cd ../autonomous-code/apps/mongodb && docker compose up -d
# cd ../autonomous-code/apps/machine-manager && npm run dev
npm run dev # tsx watch, :8085 by default
Copy .env.example → .env and set at least MANAGER_API_KEY (to match the manager) and a
DATABASE_URL pointing at the same MongoDB the manager uses. Production SSO/profile URLs are
the defaults; override SSO_ISSUER, SSO_PROFILE_URL and SSO_IDENTITY_URL with the staging hosts for local testing
(the identity URL is asked first — leave it on production and staging tokens are refused there).
| dev | build | other |
|---|---|---|
npm run dev (tsx watch) |
npm run build (esbuild via build.mjs) |
npm run typecheck; npm run test (vitest) |
CLI terminal P2P canary
Remote CLI-to-CLI terminals can use an ordered WebRTC DataChannel while authentication, encrypted
signaling, chat/RPC, and fallback remain on the existing WebSocket path. This mode is STUN-only and
does not require or use TURN. Configure TERMINAL_P2P_ROLLOUT_PERCENT (default 100; set 0 as the kill switch),
TERMINAL_P2P_STUN_URLS (default Cloudflare STUN), and TERMINAL_P2P_OPEN_WAIT_MS (default 1500).
When direct ICE fails, the terminal opens on WebSocket relay automatically.
Deploy
../.github/workflows/production-be-build.yaml builds and pushes the production Docker image on
tag push. Backend releases are tagged vX.Y.Z_backend — the _backend suffix is what routes the push
to this workflow instead of the CLI's ../.github/workflows/release.yml (which only reacts to
vX.Y.Z_cli), now that both live in this one repo. Cut one with:
make release-backend # patch bump
make release-backend ARGS=minor # or major
make release-backend ARGS="1.4.1" # exact version
make release-backend ARGS=--dry-run # preview, tags/pushes nothing
(equivalent to bash backend/scripts/release-be.sh, see that script for details).
The script refuses a checkout that lacks the last vX.Y.Z_backend release's commit: it would deploy
older code under a newer version. Update the checkout first; ARGS=--allow-older is for a rollback.
Memory sizing and Redis limits
The backend app runs as a pm2 cluster: every worker has its own V8 heap. ecosystem.config.cjs
pins the heap with --max-old-space-size (BACKEND_HEAP_MB, default 1024) and recycles a worker at
BACKEND_MAX_MEMORY (default 1536M), so size the container roughly as
instances × BACKEND_MAX_MEMORY × 1.25. Dockerfile.k8s runs one plain node per container — pass
NODE_OPTIONS=--max-old-space-size=… there and set the pod limit ~1.5× that.
Per-process ceilings that bound WebSocket memory (all in src/lib): wsServer.ts WS_LIMITS
(inbound frame size per endpoint), wsSend.ts SEND_HIGH_WATER / SEND_KILL_WATER (outbound
backpressure: droppable frames skipped, slow consumers closed 1013), orderedInbox.ts
DEFAULT_MAX_INFLIGHT (queued frames per socket), VOICE_INFLIGHT_MAX_BYTES (PCM held for STT across
all device sockets) and APP_PROXY_MAX_RELAY_FRAME_BYTES (largest app-proxy frame relayed cross-instance).
App-proxy bodies can cross instances through Redis pub/sub in multi-MiB messages. Redis' default
client-output-buffer-limit pubsub 32mb 8mb 60 disconnects a subscriber that falls behind, so on the
Redis side set it to at least 64mb 16mb 60 and alert on client_recent_max_output_buffer. The
app-proxy channels ride their own subscriber connection (appSub in bus.ts), so a disconnect there
cannot take the chat / presence subscriptions down with it.
Liveness, readiness and drain
/api/health is liveness: the process answers, with no dependency calls, so a Mongo or Redis outage
never makes Kubernetes restart pods. /api/ready is readiness (src/lib/readiness.ts): 200 while Mongo
answers a ping and Redis a PING (Redis only where REDIS_URL is set), each within 1 s; 503 naming
the failed or timed-out one otherwise. One answer serves every probe for 1 s, so probes add at most one
ping per dependency per second per process. From the moment a process gets SIGTERM, /api/ready answers
503 draining, and it keeps serving for DRAIN_DELAY_MS (default 0, closing at once as before) before it
closes its listeners, drains its WebSockets and disconnects Mongo and Redis.
Set DRAIN_DELAY_MS only on the Kubernetes image (Dockerfile.k8s), never under pm2: in cluster mode a probe
reaches whichever worker the master picks, so one worker's 503 says nothing, and pm2 kills a worker that has not
exited within kill_timeout (10 s, ecosystem.config.cjs).
Settings for the api Deployment (the job Deployment serves no HTTP and has no probes):
terminationGracePeriodSeconds: 30 # DRAIN_DELAY_MS + up to ~10 s of socket drain and close, plus margin
containers:
- name: backend
env:
- name: DRAIN_DELAY_MS
value: "15000" # readiness needs up to 10 s to fail, then the ingress a few more to drop
# the pod. No preStop sleep: the process keeps serving on its own
startupProbe:
httpGet: { path: /api/health, port: 8085 }
periodSeconds: 2
timeoutSeconds: 2
failureThreshold: 30 # up to 60 s to start listening
livenessProbe:
httpGet: { path: /api/health, port: 8085 }
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3 # restart only after ~30 s without an answer (a stuck event loop)
readinessProbe:
httpGet: { path: /api/ready, port: 8085 }
periodSeconds: 5
timeoutSeconds: 2 # above the 1 s dependency timeout
successThreshold: 1
failureThreshold: 2 # out of rotation within ~10 s of an outage or a drain
Connector gateway
routes/connectors.ts, lib/connectorGateway.ts: sign-in to GitHub, Slack, Asana, HubSpot, Gmail,
Google Calendar, Google Drive and Figma for harness connections and the desktop app's Settings ▸
Connectors. These services let no computer register its own OAuth client, so the OAuth apps Autonomous
registered with them are used, with the same requests and answers as the Grid control plane
(autonomous-grid-be grid_networks/connectors.py). Every other service signs in from the computer.
To turn it on:
CONNECTOR_REDIRECT_URI: the page every one of those apps allows. Production keeps the default,https://www.autonomous.ai/connector/callback; staging,https://staging.autonomousdev.xyz/connector/callback.- The collections:
prisma db push(run at startup) createsconnector_appsandconnector_credentialswith their unique indexes. Both are kept as they are, not encrypted: the database is private. - The apps, one row per service in
connector_apps(code,clientId,clientSecret,authUrl,tokenUrl,scopes,extra: {mcp_url, token_field, mcp_auth_header, rest_entry, transport},enabled…). A service with no MCP server the grant can use (Gmail, Google Drive, Calendar, Figma) hasrest_entry, its REST tools: the gateway hands it over with the token, untouched, and the computer's bridge serves those tools to agents as MCP, calling the service itself. The rows are edited in Compass, or written from a file withnpm run connectors:import <config-connector-auth.json>(the backend'sDATABASE_URL). The file is the Grid control plane's shape; onlyauth_type: appentries are written. Production's are theappentries of Grid prod's/var/lib/grid-apis/config-connector-auth.json; staging's, Grid dev's. The file is never in git. Running it again replaces those services; the gateway sees any change within a minute.enabled: falseswitches a service off. - The Autonomous web callback page forwards a state that starts
harness_toPOST /api/connectors/callback, as it forwardsgrid_to Grid (ecm-website v1.28.18).
With no row in connector_apps, the gateway lists no service and starts no sign-in; nothing else changes.
Persistence
Prisma over the same MongoDB as the agent-manager (no migration files — prisma db push at
startup). The backend owns the users + agent_bindings collections; it never manages the
manager-owned agent_nodes/managers (read those via raw queries when status enrichment is needed).
Product usage with an account-wide privacy control
Settings → Privacy → Share product usage data on desktop and phone, or
harness privacy status|on|off [--json], controls the signed-in account. New accounts default on;
an existing legacy opt-out stays off until the person explicitly enables sharing. Guest computers
send no product reports. This control does not disable operational records needed to run the service.
The gateway reduces lifecycle signals locally to a fixed vocabulary: active/working installation days, first measured work and response, repeat work, D1/D7 and consecutive-week returns, app opens, launch attempts and outcomes, request refusals, response waits, matched turn ends/interruption, error-event counts, and coarse duration buckets. A first-launch cohort counts work and a text response within 24 hours of an installation's first measured launch attempt. Only day and counts leave the computer; its bounded local state keeps the follow-up flags. These are updated installations, not necessarily new people, and a response is not a judgment of task success.
Launch outcomes follow the existing durable creation receipt. Status polling adds no attempts; install/model/native waits stay pending until an observed created, failed, cancelled or unconfirmed reply. A 15-minute observation bound or memory-cap eviction records an unknown outcome, not a failure. A daemon restart loses in-flight matches. Fixed request counters also cover resume, restart, model changes and history. Only exact known error codes select a fixed failure category; messages are never read. The response timer measures the first text event after an observed turn start, not model tokens or output quality; tool-only turns can have no text response.
hn (the tui surface) adds fixed ui_tui_* flow and health_tui_* milestone counters through its
local daemon's queue, under the same account choice: connections, computer linking, viewer opening,
app start, workspace ready, resume recovery, and once-per-session workspace-ready, first-work and
first-response milestones. It records only while signed in and sends nothing observed before sign-in.
Its counters go in reports of their own, so a backend without the tui names rejects only those; the
daemon then drops them until the next UTC day or its restart, and records no delivery failure for it.
No prompt, response text, transcript, code, file or project name, path, tool argument, error text, engine name, or stable user/device/agent identifier is a report field. Agent and launch identifiers exist only in bounded in-memory maps (1,000 entries each) and never enter disk queues or HTTP reports. The server authenticates requests to check the preference; this is not anonymous network traffic. No analytics SDK, session replay, fingerprinting, or third-party destination is added.
With sharing on, Privacy settings also offers optional feedback: Useful, Couldn't finish, Slow, Confusing, or Missing a feature. A deliberate button press sends only the fixed choice and a random retry receipt. It cannot include text, a task identifier or content. Ambiguous retries keep the same receipt and answer. This is self-selected feedback, not the task success rate of all users.
The product usage collections have distinct purposes:
| Collection | Fields retained | Lifetime |
|---|---|---|
product_usage_preferences |
Account ID, enabled flag, revision | Saved account preference; no usage history |
product_usage_totals |
UTC day, allowlisted metric, sum (plus DB primary key) | 396 days, approximately 13 months |
product_usage_receipts |
Random report UUID, expiration date and its life in days (required on every new write once the worker's receipt sweep installs the collection rule) | Scheduled expiry 4 days from the report day (36 until 2026-10-29) |
product_usage_activity |
HMAC subject (PRODUCT_USAGE_SUBJECT_KEY), UTC day, surfaces, worked flag, sign-up day, staff flag, expiry |
10 days (TTL); deleted on opt-out and /api/analytics/delete; written only while sharing is on and the key is set |
No individual report body or arrival timestamp is stored. Receipt insertion and increments share a Mongo transaction so ambiguous retries cannot count twice. Saving a preference advances its revision; an old queued report is refused even after an off/on cycle. The client checks the choice before each upload, polls at most once a minute, and stops collecting if it cannot confirm the choice. The server checks every ingestion without caching an enabled answer. Already admitted in-flight operations can finish; previously combined totals cannot be selectively removed for one person. Turning off clears the local queue on its next successful preference check. Queued counters older than 35 days are discarded when the app next runs; no task content is read to reconstruct missing metrics.
The worker installs Mongo TTL indexes after schema setup. Expiry is asynchronous: the Mongo TTL background worker may delete later than the configured threshold. It also expires the eight legacy daily history collections after 35 days. The old collector endpoint accepts no new rows, relay tracking writes global event totals and, while sharing is on, the 10-day people marks, and the existing account deletion control covers the six former presence collections as well as the old collector tables. These database lifetimes do not assert anything about infrastructure access logs or backup retention; verify those separately before rollout.
Read daily totals with a read-only database identity, supplying DATABASE_URL through the environment:
cd backend
npm run usage:daily -- 14
The JSON includes daily headlines, the last complete week's return rate, and a review section
with launch and response friction, fixed feedback counts, matured first-launch and D1/D7 cohorts,
and equal seven-day descriptive comparisons. Each comparison includes its numerator, denominator,
coverage limits and evidence caveats. Today and immature cohorts are excluded from comparisons.
Low counts or incomplete windows are explicitly marked; no causal improvement is inferred.
Suggested investigations use synthetic reproductions and benchmarks, never user content. Missing client data is
unknown, and today is explicitly incomplete. Population is participating signed-in computers,
including staff and both service environments, not unique people or an external-user cohort.
Relay connection/turn totals from old clients stay separate from new client counters. Background
windows can count as active, and human/automated work is combined. First measured work is not signup
activation; a turn ending is not task success. The report deliberately makes no signup-to-value
activation claim. Weekly return compares locally counted consecutive-week working computers with
the preceding complete week's working computers; opt-outs, reinstallations, preference changes,
offline gaps and client upgrades affect coverage. Do not compare this estimate directly with the
legacy account-cohort retention percentage or an industry benchmark.
Ship the backend and its worker together, verify the unique and TTL indexes using synthetic reports,
then ship the CLI and settings surfaces with a visible release note explaining the default and switch.
The worker installs measurement keys explicitly even when the general Prisma schema push declines
an unrelated change. It transactionally combines duplicate day/metric totals from a deployment race
before creating their unique index; this bounded repair touches only aggregate counters and preserves
their sum. A missing key or failed repair stops worker startup rather than claiming readiness.
The daily decision workflow and release note are in the measurement guide.
Mixed-version coverage remains explicit until clients update. The existing usage:report command
below reads the bounded historical tables; its current-day counts cease when new history writes stop.
Client country (Cloudflare)
Production DNS is proxied through Cloudflare, so every request and WebSocket upgrade arrives with a
CF-IPCountry header. src/lib/clientGeo.ts is the only reader; it feeds:
users.lastCountryCode/lastCountryAt— from the owner's own control-plane calls (middlewares/authMiddleware.ts), rate-floored to one write per user per hour unless the country changes. This is the "where is the user" answer.machines.countryCode— the computer's country on its last/api/adapter-wsconnect.- Legacy
machine_daily_presence.countryCode/user_daily_device_presence.countryCode— same header on the daily presence rows.user_daily_presencedeliberately has none: it is keyed by (user, machine, day) and its socket is the daemon's, so the machine's country is the one onmachine_daily_presencefor thatmachineId.
Off Cloudflare (local dev, direct origin hits) the header is absent and nothing is written; a value is only ever set, never cleared.
Historical architecture notes still live in autonomous-code/docs.
Product usage report
npm run usage:report reads the existing relay presence and turn counters and prints an
aggregate JSON report. It covers observed app users, owners doing agent work, surfaces,
complete UTC day/week/month windows, weekly return, exact-day return after first observed
work, seven-day signup activation, and workload concentration. It also shows collection
coverage and reconciles machine and agent turn counters, which must never be added together.
Use a database account with read-only privileges. In Bash, from backend/:
read -r -s -p 'Read-only MongoDB URL: ' READ_ONLY_DATABASE_URL
export READ_ONLY_DATABASE_URL
mkdir -p ../.harness/usage
npm run usage:report -- --days 30 --out ../.harness/usage/report.json
unset READ_ONLY_DATABASE_URL
The command deliberately requires READ_ONLY_DATABASE_URL, does not load .env, and does
not import server startup, migrations, or the application database singleton. It issues
read-only aggregates with a 15-second server limit, bounded connection timeouts, and a
100,000-row result ceiling; exceeding the ceiling fails instead of returning partial figures.
It exports no emails, account IDs, machine IDs, agent IDs, hostnames, or credentials. Keep
production reports under ignored .harness/usage/, not in source control. Output files use
mode 0600 and are never overwritten.
Options:
--as-of YYYY-MM-DD: exclusive UTC cutoff, default today; the current partial day is excluded.--days N: reported daily range, 1–365 days. The loader reads at least 90 days for historical context;range.historyFromrecords the exact boundary.--exclude-email-domain example.com: repeatable, case-insensitive exact account-domain filter. Exclusions apply before all metrics and denominators. This is an account-domain segment, not a verified employee/customer classification; unmatched historical owners stay in the report and are counted underobservedUsersWithoutAccountRow.
Interpretation limits are part of the output. App presence can mean an idle window, connected computers can run without an open app, and turns include automation. Surface user counts overlap. P2P offers measure attempts, not confirmed sessions or all remote usage. A cohort's first observed turn is not necessarily its first-ever use. Retention excludes the initial observation day and only includes accounts old enough for the requested return day; signup activation also requires a full seven-day follow-up and signup after the observation boundary.
Coverage describes earliest/latest stored rows across all accounts, including accounts removed
by a segment filter; it does not certify uninterrupted collection. A missing row means no observed
activity, not proven inactivity. In particular, do not compare new surface instrumentation with
older desktop history as if both had equal coverage. The collector-based account dashboard
(src/lib/analyticsQuery.ts) is separate: stale collector rows cannot establish current engine
mix, completion rate, runtime, or token cost.
The expanded measurement contract adds fixed app-flow and feature-use counts, D28 and calendar-month
installation returns, and voluntary task outcomes. Native signed-out setup is a separate explicit
session opt-in at POST /api/product-usage/setup; it rejects credentials and arbitrary fields and
has a process-wide admission budget without storing IP addresses. Web app signed-out setup reporting
is disabled. Anonymous Hub totals, shared links' (/s/:id) included, arrive at POST /api/product-usage/hub, forwarded by the website's
server without cookies or auth; the route accepts only the Hub vocabulary (src/lib/hubMetricsContract.ts)
and has its own admission budget.
Daily reports keep these populations separate, show unmatched outcomes and measurement loss, and
compare only mature periods. See measurement definitions and runtime bounds.
Deploy this additive allowlist before releasing the new clients. This change does not add identity
markers or make installation counts comparable to people-based DAU or retention benchmarks.