Skip to content

Router that virtually distributes inference across connected devices in the home.

Go
11.3 KB
Raw Blame
---
description: Current PAIR runtime architecture, communication paths, modular services, and security boundaries
alwaysApply: true
---
<!--
SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
-->

# System Architecture

`docs/architecture.md` is the rendered current-state reference.

## Process model

PAIR is an Electron application backed by the sibling `services/` tree.

- Electron starts only `nvpair-ui-broker`.
- The broker supervises every runtime worker.
- Backend processes communicate with their parent over stdio JSON-RPC.
- Development binaries live in `cli-bin/`.
- Packaged binaries live in `process.resourcesPath/cli-bin`.

The canonical inventory is `src/shared/constants/modular-binaries.ts`.

Broker-owned workers:

- `nvpair-proxy`, one process hosting a facade per enabled engine. It starts
 with no engine and no listener; the broker sends a `facade/enable` per engine
 carrying that engine's port. Clients still address each facade as
 `ollama-proxy:`, `lmstudio-proxy:`, or `llamacpp-proxy:`, and one supervisor
 covers them all, so a crash is reported against `nvpair-proxy` and restarts
 every facade together;
- `nvpair-node-scanner`;
- `nvpair-node-info`;
- `nvpair-workload-manager`;
- `nvpair-cluster-manager`;
- `nvpair-node-settings`;
- `nvpair-manual-nodes`;
- `nvpair-engine-manager`;
- `nvpair-errors`;
- `nvpair-job-scheduler`.

`nvpair-tui` is bundled but not supervised by Electron or the broker.

Never launch one worker from both Electron and the broker. There is no
broker-absent fallback.

The Inference Demo's `inference-dispatcher` is the one non-broker executable
Electron launches, and the only child `nvpair-tui` spawns besides its own broker.
It is not a worker: its source lives in `scripts/inference-dispatcher` at the
monorepo root, and it is absent from `services/versions.json` and
`modular-binaries.ts`. It nonetheless ships inside `cli-bin` — beside the
binaries it is not one of — so both front ends resolve it the same way. See
"Inference Demo" below.

The connector remains `connecting` until broker `app:ready` and the required
Ollama proxy readiness arrive. A startup deadline routes failures to
Settings > Service with the error and log/restart controls; late readiness
transitions the existing process tree to connected.

## Communication

### Renderer to service

`window.pairApi` is the only renderer service-control API. It uses
`window.__PAIR_TRANSPORT__`, the `service-bridge:*` IPC envelope, Electron main
handlers, and the broker.

Logical invoke and push types live in `src/shared/types/ws-channels.ts`.
Browser mode is unsupported.

### Renderer to Electron

`window.windowApi` is reserved for:

- window and tray management;
- clipboard;
- service lifecycle and logs;
- updater control;
- first-run state.

IPC types live in `src/shared/types/ipc-channels.ts`. All handlers use
`safeHandle()`.

### Terminal

PAIR has no Electron-backed console client. The `nvpair` command on PATH is a
launcher for the bundled Go `nvpair-tui`, which owns its own broker and worker
tree. Electron generates that launcher on packaged builds
(`src/electron/nvpair-command.ts`); the Debian package installs the equivalent
wrapper at `/usr/bin/nvpair`. Do not add a TypeScript command surface, control
socket, or local control server.

### Node telemetry

Electron directly polls each discovered node's advertised `/v1/node-info`
endpoint every two seconds while it is healthy. Consecutive failures back off to
4, 8, 16, then 30 seconds and reset on success or an endpoint change. This is the
only service-data HTTP exception because rich telemetry is not exposed by the
broker JSON-RPC contract.

This renderer-facing metrics path is display-only. Independently, the backend
scanner provides compact maximum-GPU utilization on the same healthy cadence and
capped failure backoff; the manual-node worker supplies the same payload shape.
The broker uses both for scheduling.

### Inference HTTP

Ollama, LM Studio, and llama.cpp proxies expose local client-compatible HTTP
endpoints. Chat behaves as a local third-party client. Prompts, messages,
chunks, and response bodies must never be logged.

## Runtime defaults

Backend-coupled constants belong in
`src/shared/constants/modular-runtime.ts`.

- Prefer live broker or discovery values.
- Do not assume a proxy port before the broker reports it.
- Native chat fallbacks are Ollama `127.0.0.1:11434`, LM Studio
  `127.0.0.1:1234`, and llama.cpp `127.0.0.1:8080`.
- Cluster pairing currently uses port `14321`.
- Node telemetry path is `/v1/node-info`.

Backend discovery service names and TXT layouts are not PAIR contracts. Consume
the broker's reported discovery snapshots instead of duplicating those details.

## State flow

The service is authoritative.

1. The UI sends commands.
2. The broker and workers execute them.
3. Backend notifications update `ModularBridgeState`.
4. Typed push events update renderer stores.

Stores fetch domain snapshots on startup and subscribe to push events. Engine
commands return no state. The centralized pending-actions store is the only
optimistic engine/model state in the renderer.

Current push channels are declared in `WsPushChannelMap`; do not maintain a
second handwritten channel list.

## Discovery

`nvpair-node-scanner` is the discovery authority. The broker exposes canonical
addresses, trust state, model inventory, and per-engine model attribution in
`AvailableNode`.

Electron merges discovery, proxy presence, engine-manager facts, and node-info
metrics into the renderer node model.

Manual nodes go through the broker's `node/add`, `node/remove`, and `nodes/list`
relay. Electron persists and replays user entries.

## Routing

Routing is backend-owned:

1. for model-bearing inference, filter to nodes whose per-engine inventory
   advertises the requested model;
2. within that owner set, apply explicit manual selection;
3. apply `nvpair-job-scheduler` priority;
4. use deterministic proxy default order.

Empty and non-matching inventories are excluded. If no advertised owner is
routable, the proxy returns a local `502`; retryable upstream failures continue
only through other advertised owners.

PAIR does not call `node/select` and must not implement a TypeScript router.
The broker relays pending workload and source-aware GPU telemetry. The scheduler
uses EWMA-smoothed fresh utilization to derive pressure 0–3, with neutral
pressure 1 for invalid, missing, or stale data, then ranks by
`pending + gpuPressure`, pressure, and UUID. Each proxy minimizes
`pending + gpuPressure + localReservations` for capable nodes.

## Inference Demo

A fixed sixty-second burst of synthetic traffic sent through the local proxies
so job activity is visible.

Both front ends run it, each owning its own schedule:

- Electron main (`src/electron/inference-demo.ts`,
 `src/electron/inference-demo-schedule.ts`), surfaced on Settings > Service and
 visible on Overview.
- `nvpair-tui` (`ui/demo.go`, `ui/demoschedule.go`), on the Jobs tab, driven by
 the shell's one-second tick because every submission offset is a whole second.

The two schedules must stay identical — cohorts, stages, offsets, and the
ceiling. They are a contract between the front ends, not an implementation
detail of either; a change to one is a change to both. Neither drives the other.

- Each request spawns the bundled `inference-dispatcher`, which behaves as an
 ordinary third-party HTTP client.
- Requests target a broker-reported proxy port, never an engine's own port. The
 backend decides placement; PAIR must not describe the demo as distributing
 work itself.
- State is node-local and is not synchronized. Electron broadcasts it on the
 `demo:state` Electron push channel (`IpcPushChannelMap`); the terminal keeps it
 in view state. Neither goes near the service push bus.
- Stopping cancels only unsent requests. Never cancel, await, or report on one
 already submitted.
- Children run with stdio discarded and with `INFERENCE_DISPATCHER_*` stripped
 from the environment. Never surface or log prompts or responses.

## Engines

`nvpair-engine-manager` owns lifecycle, port, desired-state, and model
operations.

- Local lifecycle supports install, start, stop, uninstall, update, and port
  changes.
- Remote lifecycle uses the engine manager's remote control methods where
  available.
- `engine:prepare-shutdown` stops processes without clearing persisted desired
  state. The broker calls it during its own shutdown, after stopping the proxy;
  clients leave it to the broker.
- Engine and proxy ports are persisted by their backend owners.
- Running adopted engines reject operations that require process ownership.
- Per-engine launch arguments and environment are editable through
 `engine:get-settings`, `engine:preview-settings`, and `engine:apply-settings`,
 which write the server port, the proxy port, and the launch text together
 against a revision. Do not write any of the three by another route: a second
 writer with no shared revision cannot tell that it lost. Validate through the
 preview and commit the settings it returns, not the draft that was sent.
- Editability is the snapshot's `Editable` and `Reason`, not a rule restated in
 a front end. The backend relays these to a peer, so a front end that refuses
 remote editing on its own is wrong rather than cautious.

The model catalogue is owned by `nvpair-engine-manager` and served over
`engine:catalog`, so the desktop app and the terminal interface browse one
implementation. Electron relays it through
`src/electron/service-bridge/model-catalog.ts`. It provides curated Ollama,
LM Studio, and llama.cpp catalogs, filtered for the OS and CPU a model will
install on; model operations go through the engine manager. llama.cpp's is
populated from approved Hugging Face publishers and also answers a bounded
search across public publishers. Every llama.cpp result is an exact
`owner/repository:Q4_K_M` pull ID.

Managed llama.cpp installation uses CUDA archives on Windows and Linux and
standard Metal-capable archives on macOS. Package choice is based on platform
and architecture, not the installed driver; CPU fallback remains available.
Checksums and required runtime artifacts are verified before installation.

## Pairing and security

`nvpair-cluster-manager` and the other backend workers own cluster identity,
PIN pairing, trusted certificates, membership, and every node-to-node channel.
PAIR implements **no** security or cryptography: it relays pairing and
membership commands, renders the resulting notifications, and never reads keys
or builds transport security.

The PIN is a convenience pairing code, not a strong authenticator. See
`.cursor/rules/security.mdc`.

## Packaging

`npm run build:modular-binaries` builds the backend for the target platform and
writes `cli-bin/manifest.json`. Packaging rejects stale or unexpected `cli-bin`
files.

The macOS privileged helper firewall list is derived from
`needsFirewallAccess` in `MODULAR_RUNTIME_BINARIES`.
`npm run service-contracts:check` verifies parity with the Swift helper.

Build and release details live in `docs/macos-privileged-helper.md`.

## Documentation

Keep these current when contracts change:

- `docs/architecture.md`;
- `docs/frontend-api.md`;
- `docs/services-backend.md`;
- `docs/services-parity.md`;
- this rule and `architecture.mdc`.

Do not add migration diaries or completed historical narratives. Git history is
the history.