Router that virtually distributes inference across connected devices in the home.
nvpair-engine-manager
A config-driven control plane for local inference engines. The bundled manifests support Ollama, LM Studio, and llama.cpp. It manages everything about an engine except serving inference: detect, user-mode install, start/stop/restart, health, and config-declared actions. Adding an engine is a JSON manifest, not code.
The bundled manifests under manifests/ are the working reference for manifest
authoring.
llama.cpp backend checkpoint
The llamacpp manifest runs llama-server in router mode on loopback port
8081. It can list, download, load, unload, and delete exact model ids such as
owner/repository:Q4_K_M. Downloads use /models/sse for progress; deletion
uses the router's native DELETE /models cache operation and does not restart
the router. LLAMA_CACHE points at the manifest's models_dir, ~/.llamacpp,
which no removal this service performs will delete.
Readiness requires /props to report role:"router", so model-selection
arguments that switch llama-server to single-model mode and incompatible
listeners already occupying the port are rejected rather than adopted.
After five minutes without inference work, a loaded model enters llama.cpp sleep
mode and releases its model and KV-cache memory; the next request wakes it. The
router child remains alive and can retain a residual backend GPU context.
Windows and Linux installs download checksum-pinned server and CUDA-runtime
archive pairs (CUDA 12.x for x64 and CUDA 13.4 for arm64); macOS uses the
standard Metal-capable archive. These on-demand downloads are roughly
0.6–0.8 GiB and do not enlarge the PAIR installer. llama.cpp keeps its auto
GPU-layer policy: supported NVIDIA/Metal devices can accelerate, while the
dynamic CPU backend remains the fallback. Hardware acceptance, not /health
alone, is required to claim GPU activation.
Communication
Bidirectional newline-delimited JSON-RPC 2.0 — the same conventions as
every other NVPAIR subprocess. Stdio by default; --ipc <path> dials a Unix
domain socket or Windows named pipe instead. The service is
parent-agnostic: it speaks to whatever owns its pipe (in practice
nvpair-ui-broker) and carries no front-end dependency, so the
same binary runs under any supervisor with zero code change. The engine:*
namespace is the surface the orchestrator/broker forwards from the UI.
JSON-RPC surface
Requests (caller → service):
| Method | Params | Result |
|---|---|---|
engine:get-installed |
— | { engines: [EngineStatus] } |
engine:describe |
{ engine } |
the engine's manifest |
engine:status |
{ engine } |
EngineStatus |
engine:install |
{ engine, start?, port?, bind? } |
EngineStatus (after install; also starts it if start:true) |
engine:uninstall |
{ engine } |
EngineStatus (after removal) |
engine:uninstall-managed |
— | { engines: [{ engine, removed, error? }] } — removes every engine this service installed, selected by its install marker, each through the engine:uninstall path. Backs "reset all data" in both clients; downloaded models are kept. Neither removed nor error means the engine was left alone as not ours |
engine:start |
{ engine, port?, bind? } |
EngineStatus (after readiness) |
engine:stop |
{ engine } |
EngineStatus |
engine:restart |
{ engine } |
EngineStatus |
engine:set-port |
{ engine, port } |
EngineStatus (after rebind) |
engine:action |
{ engine, action, params } |
the engine's raw response. action:"pull_model" is streamed: it emits live engine:pull-progress notifications and returns the pull's terminal result (see below). An action whose manifest declares restart_after (LM Studio's delete_model) restarts a running engine before replying, so the response also means the engine is back and healthy |
engine:logs |
{ engine } |
{ lines: [LogLine] } |
engine:errors |
— | { errors: [ServiceError] } |
engine:catalog |
{ engine, platform?, arch?, query? } |
{ models: [{ id, name, author, url, size?, downloads, likes, updatedAt, tags, family?, parameterSize?, appleOnly? }], source, platform?, arch?, fetchedAt?, searchable?, query? } — the models an engine can download, as opposed to engine:models, which reports what is already installed. Each engine has one curated source. Ollama's has no public API, so it is a locked list compiled into this binary from catalog/ollama-models.json, regenerated on demand by a developer (npm run scrape:ollama-models in desktop/) who reviews the diff; serving it needs no network. LM Studio's catalog is the lmstudio-community Hugging Face org, whose repo ids are exactly what lms get accepts, so it is fetched live and cached for six hours, with concurrent callers coalesced onto one request, a failed attempt — with or without a list to fall back on — followed by a backoff before the next. llama.cpp downloads any GGUF repo as repo:quantization; its browse list is the GGUF repos of ggml-org, bartowski, and unsloth, fetched and cached the same way, keeping only open, text-generating repos that ship a Q4_K_M file, each offered as repo:Q4_K_M. It is the one searchable source: a query searches all of Hugging Face under the same rules, each query cached on its own (the 20 most recently used are kept), and the reply echoes it. A query longer than 100 characters is refused rather than shortened, since a shortened one searches for something the operator did not type. Other sources ignore a query and return their whole list for the client to filter. platform and arch are the GOOS and GOARCH the models will be installed on, which is not always this host — a client driving a peer should name that peer. Omitting both means this host; naming a platform without an arch leaves the arch unknown rather than borrowing this host's. MLX quantizations only install on Apple Silicon, so they are marked appleOnly and kept only for a darwin/arm64 target — an Intel Mac does not get them; the reply echoes the platform and arch it filtered for. name is pull-ready: it can be handed to pull_model verbatim. An engine with no curated source is an error, not an empty list. The Ollama reply is a single multi-megabyte frame, so every hop on its path must allow jsonrpc.WorkerFrameBytes. |
engine:models |
— | { models: [string], modelsByEngine: { <engine>: [string] }, loadedByEngine: { <engine>: [string] } } — the flat de-duplicated union of every running engine's models, the per-engine breakdown keyed by engine name, and the per-engine set of models currently loaded in memory (all normalized from each engine's list_models / loaded_models action result spec). modelsByEngine carries a key for every running engine whose inventory was successfully queried, including an empty list = "running, no models available"; a missing key means not running / not queryable / invalid response. loadedByEngine uses the same known-empty distinction for residency and also omits engines with no loaded endpoint. The /v1/models HTTP surface returns the same shape. |
engine:remote-get-installed |
{ node } |
{ engines: [EngineStatus] } fetched from the remote node over ec mTLS |
engine:remote-install |
{ node, engine, start? } |
{ opId, status: EngineStatus } after the remote install (live progress via engine:remote-progress) |
engine:remote-pull-model |
{ node, engine, model?, params? } |
{ opId, result } after the remote pull (live progress via engine:remote-progress) |
engine:remote-start |
{ node, engine, port? } |
EngineStatus from the remote node (always the manifest's runtime.bind; no per-call bind override on the remote path) |
engine:remote-stop |
{ node, engine } |
EngineStatus from the remote node |
shutdown |
— | null |
log/set-level |
{ level } |
{ level } |
EngineStatus = { engine, display_name, installed, running, healthy, port }.
Notifications (service → caller): engine:ready{version},
engine:state-changed{EngineStatus},
engine:models-changed{engine, models} — pushed when an engine's set of
loaded (in-memory) models changes (explicit load/unload, JIT auto-load, or
TTL/idle eviction); models is the full engine:models shape (incl.
loadedByEngine) so a consumer swaps its whole snapshot,
engine:install-progress{engine, stage, percent?, error?} (error on the
terminal failed frame),
engine:pull-progress{engine, op, stage, percent?, message} (live progress for a
local model pull driven via engine:action{action:"pull_model"} — the local
counterpart of engine:remote-progress; frames are coalesced to changes in
stage/percent, the engine's terminal success surfaces as stage:"success", and
a failed pull emits a terminal stage:"error", percent:-1, message frame so a
UI converges even if its synchronous call already timed out),
engine:remote-progress{opId, node, engine, op, stage, percent?, message}
(relayed live progress for a remote install/pull), and — for the error
pipeline — errors:report / errors:clear (consumed by nvpair-errors
via the broker; see below).
Streaming HTTP pulls have a 30-minute inactivity watchdog. Each increase in an Ollama layer's or llama.cpp file's completed byte count refreshes that watchdog, so an active download may run longer than 30 minutes; duplicate progress frames and heartbeats do not extend a stalled pull. CLI-driven pulls without structured byte progress retain the fixed 30-minute action timeout.
For llama.cpp, an accepted pull that ends before a matching download_finished
or download_failed event stops the active download. This includes caller
cancellation, remote caller disconnect, inactivity timeout, and premature SSE
termination. Cleanup checks GET /models and sends POST /models/unload only
when the exact model is still downloading; cached files are retained, and
models that have already completed are left alone.
The initial POST /models handshake has a separate 30-second total timeout and
continues through caller cancellation so its acceptance can still be read.
The pull then waits for cleanup, which has a separate five-second budget for
the inventory check and stop request together. A failed cleanup reports that
the download could not be confirmed stopped alongside the original pull error.
If the start acknowledgement is lost or unreadable, acceptance and cancellation
are reported as unconfirmed without unloading an unowned download. The monitored
model must match params.model; mismatches are rejected before subscribing.
percent on all three progress notifications is present only when the step is
measurable (download bytes, byte-progress pulls) or terminal (100 on install
done / already-installed, -1 on install failed and pull error). An
indeterminate step omits it — the install verified and installing stages
always do — so a UI renders the stage alone.
The engine:remote-* methods are the client half of remote engine
management: engine-manager resolves the target node in an ec peer
directory (fed by its own discovery:subscribe{services:[ec]} to the broker
relay), dials that peer's ec surface over pin-based cluster mTLS, and — for
remote-install / remote-pull-model — mints an opId, relays each streamed
progress frame up as engine:remote-progress, and settles the request with the
terminal result. They fail if this node isn't clustered or the target isn't a
pinned cluster peer. See "Remote engine management" below.
engine:install, engine:start, engine:stop, engine:restart,
engine:set-port, and engine:action run in their own goroutine on the
service side, so the read loop never blocks and their responses arrive when
the op finishes.
engine:set-port is the persistent port setter (distinct from the
one-shot engine:start {port} override, which reverts on the next restart).
It validates 1-65535, persists the choice as a manifest override (a
{ engine, runtime: { port } } delta written
to the per-user engines/ dir that deep-merges onto the bundled manifest, so
runtime.port becomes the single source of truth and the port is restored on
the next start with no separate store), and applies it — bouncing the engine
onto the new port if it was running. Setting the port back to the bundled
default removes only the shared and host-platform port overrides; the file is
removed only when no other overrides remain. Arguments, environment, install
settings and other platform overrides are preserved. A host-platform port is
updated when necessary so it cannot shadow the saved value on restart.
Malformed override files fail the save instead of being replaced.
Because the chosen port lives in the
effective manifest, restore is automatic: a normal engine:start (no explicit
port) and the adopt-on-fixed-port path both come up on the retained port.
Moving a running, adopted engine is refused with an error (nothing is
persisted) — NVPAIR can't relocate a process it didn't start; see Adoption below.
Lifecycle
Combined launch/server/proxy edits use the broker's
engine settings protocol. The worker
checks basic argument syntax and adapter-declared networking/CORS controls
using pair-arguments-v1, persists host-platform launch_args and launch_env
with the port, and holds its operation lock through stop/rebind/start. The
paired engine-control surface relays settings to its local broker and streams
full authoritative snapshots to pinned peers.
NotInstalled --engine:install--> (HTTPS download + verify-if-pinned + user-mode run) --> Stopped
Stopped --engine:start----> (adopt if already serving the port, else spawn) --> Running --health--> Running
Running --engine:stop-----> (stop signal, bounded grace, force if needed) --> Stopped
Detect uses the manifest's detect paths. Install is one-shot and
user-mode — an HTTPS download, verified against the manifest's sha256
when one is pinned (an unpinned fetch runs with a loud warning). Start
waits for the readiness probe, then runs a periodic health probe; an
unexpected exit is reported. The bundled Ollama manifest allows up to ten
minutes for startup because GPU discovery can exceed the previous 30-second
allowance on supported Windows systems. The deadline remains finite: if Ollama
never serves its readiness endpoint, engine-manager stops the owned process and
reports the failed start. On Unix, stopping an owned process sends SIGTERM to
its process group, waits the manifest's stop.grace_s (five seconds by
default), then escalates to SIGKILL; signal:"kill" skips the grace. Failed
startup cleanup uses the same policy. On Windows, the windowless engines we
spawn cannot receive a graceful (non-/F) close, so stopping uses immediate
taskkill /T /F.
Adoption — start may attach to an engine it didn't launch
Before spawning, engine:start probes the chosen port's readiness
endpoint, including any manifest-declared JSON identity. If a compatible
service answers there — the engine's own desktop
app (e.g. the Ollama tray app on 11434), or an instance left running from a
previous session — the service adopts that instance: it marks the engine
running without launching its own, rather than spawning a duplicate that
would only collide on the port. Consequences worth knowing:
- An adopted engine has no child process the service owns, so
engine:stop(andengine:restart's stop phase, and shutdown's cleanup) resolves the PID bound to the engine's port and terminates it only when that process is running the very binary NVPAIR manages for the engine — reclaiming an orphan a prior run left on our own managed port (e.g. anollama serveon11435whose handle was lost after a crash). This is precise to the port, so a genuine third-party listener on a different port (Ollama's own desktop app on11434while NVPAIR manages11435) is never touched. A listener whose image is not our managed binary is declined with an actionable error naming the offending PID and image path — the user / desktop app owns that process, and NVPAIR won't terminate it out from under them. engine:stopmay return an error while still saving OFF. When stop declines a foreign listener it returns an actionable error, but the user's OFF choice is persisted anyway — UI layers should treat the saved desired state as authoritative (the engine will not restore on restart) and surface the error as guidance, not as proof the OFF intent was lost. The same applies to clusterPOST /v1/engines/stop, which may answer HTTP 500 even though OFF was recorded.engine:set-porton a running adopted engine is refused (returns an error, persists nothing). Moving it would mean killing the old listener and spawning a new one; since NVPAIR can't kill what it didn't start, it errors rather than leaving a duplicate serving the new port while the original keeps serving the old one. Stop the engine in its own app first, then set the port.engine:installshort-circuits the same way.detecthonors existing system installs (Ollama's detect paths include/Applications/Ollama.app,%LOCALAPPDATA%\Programs\Ollama, etc.), so "installing" an engine that's already present downloads nothing and reportsinstalled: true— and a followingstart:truethen adopts the running instance.- Liveness reconciliation uses the same probe. A fixed-port engine found
already serving is reported
running: trueeven though NVPAIR never started it.
To get a NVPAIR-owned, stoppable instance, start it on a port nothing is
already serving — the probe misses, so the service spawns and owns the child
(tracked in st.proc), and a later engine:stop / engine:restart actually
terminates it. engine:set-port to a free port does exactly this; quitting the
external app first and re-starting on its usual port works too. Auto-assigned
ports (manifest runtime.port: 0) never adopt — there's no fixed address to
probe — so they always spawn an owned process.
Remote engine management
When the parent passes --control-port, engine-manager serves the ec
surface — a cluster-scoped remote-control endpoint over pin-based mutual
TLS (nvpair-shared/clustertrust, the same identity + per-peer pins
nvpair-cluster-manager mints). It presents this node's cluster leaf, requires a
client cert, and rejects any caller that isn't a byte-for-byte pinned cluster
peer with a 403. It differs from em (--http-port) only in what it permits:
ec performs privileged operations, while em is a read-only inventory. Both are
locked to pinned cluster peers on the LAN — a node's model list tells a caller
which models that machine holds, which is cluster data like any other. em also
keeps a plaintext personality on loopback, because this node's own scanner reads
its own inventory that way and must be able to while unclustered.
Membership is evaluated live, per handshake and per request, from
--cluster-dir. While this node is not a cluster member it presents no leaf, so
every handshake is refused and nothing privileged is reachable; the moment it
becomes a member the same listener serves pinned peers. The listener is therefore
bound for the life of the process rather than only when clustered at startup — a
node joins and leaves a cluster while engine-manager runs, and a surface chosen
at bind time would stay dark until the process was restarted.
Endpoints (all under /v1):
| Route | Shape | Purpose |
|---|---|---|
GET /v1/engines |
JSON | remote engine:get-installed |
POST /v1/engines/install |
NDJSON stream | remote install (+ optional start) with live progress |
POST /v1/models/pull |
NDJSON stream | remote model pull with live progress |
POST /v1/engines/start |
JSON | remote start → EngineStatus |
POST /v1/engines/stop |
JSON | remote stop → EngineStatus |
The streaming routes emit zero or more {"type":"progress",...} frames followed
by exactly one terminal {"type":"result",...} or {"type":"error",...} frame.
The initiating node's engine-manager consumes that stream, relays each progress
frame up as engine:remote-progress, and settles the originating
engine:remote-* request on the terminal frame — so a UI gets one synchronous
response plus a live progress feed keyed by opId.
The broker wires both directions: it passes --control-port/--cluster-dir,
registers ec with the discovery daemon whenever a cluster dir is configured, and
relays engine-manager's discovery:subscribe{services:[ec]} into the relay
directory so the peer directory stays current. It does not restart
engine-manager on cluster:identity-changed — the surface follows membership on
its own.
CLI flags
| Flag | Default | Description |
|---|---|---|
--ipc <path> |
(stdio) | IPC endpoint: Unix socket or Windows named pipe |
--http-port <port> |
0 (off) |
Serve the plain LAN model-list surface (GET /v1/models, the em service) on this port; the broker passes :14322 |
--control-port <port> |
0 (off) |
Serve the cluster-scoped mTLS remote-control surface (the ec service) on this port; callers are admitted only while this node is a cluster member. The broker passes :14323 |
--reserved-port <port> |
0 (off) |
Refuse local or remote engine starts and persisted port changes on a parent-owned proxy alias; the broker configures this from OLLAMA_HOST |
--cluster-dir <dir> |
(none) | Cluster identity/pin directory; gates the ec surface on and supplies the leaf/pins used to serve it and to dial peers |
--loaded-poll-interval <sec> |
5 |
Seconds between loaded-model polls that drive engine:models-changed; 0 disables the watcher |
--log-level <level> |
(env NVPAIR_LOG_LEVEL or info) |
debug | info | warn | error |
--uninstall-managed |
Remove every engine PAIR installed, then exit; for the platform uninstallers, which have no broker to call engine:uninstall-managed through. Same selection and safety path as that method, loading only the manifests compiled into this binary. Always exits 0 so an uninstaller cannot stall |
|
--version |
Print version and exit |
Logs go to stderr via the shared nvpair-shared/applog format; stdout is
reserved for JSON-RPC frames in stdio mode.
Logging & errors
Each managed engine's stdout/stderr is captured into a bounded ring
(queryable via engine:logs). Operational failures (install/start/health)
are recorded and surfaced as errors:report / errors:clear
notifications using the nvpair-shared/errors wire shape. The broker
(nvpair-ui-broker) forwards them to the nvpair-errors registry.
Ids follow engine-manager:<class>:<engine>.
Security posture
Runs user mode only — no admin/sudo at runtime (privilege escalation
is reserved for NVPAIR's own install time). It has two optional LAN listeners, and
both are pin-based cluster mutual TLS with a live membership check — every
caller is authorized against a per-peer pin, a non-member is refused with a 403,
and while this node belongs to no cluster it presents no leaf so no handshake
completes:
--http-port— the model-list surface (em,GET /v1/models; the broker passes:14322). A node's model inventory is cluster data, so LAN callers must be pinned peers. This port additionally serves plaintext on loopback only, which is how this node's own scanner enriches its own card — including when the node belongs to no cluster, so a standalone machine still shows its own models.--control-port— the remote-control surface (ec, theengine:remote-*targets; the broker passes:14323). mTLS only, no plaintext personality, since every route performs a privileged operation.
Unlike the rest of NVPAIR, engine-manager therefore does terminate inter-node
mTLS itself (and dials peers' ec surfaces with the same pinned identity),
matching how nvpair-errors and nvpair-workload-manager handle their own
cluster traffic. Managed engines bind loopback by default, but a manifest's
runtime.bind may open an inference engine to the LAN — Ollama ships
0.0.0.0 to serve the cluster — overridable per call via
engine:start {bind}; readiness/health probes always target loopback.
Downloads are HTTPS-only (plain http only from loopback) and verified
against the manifest's sha256 when one is pinned; an unpinned fetch runs
with a loud warning.
Cross-platform
One binary compiles and runs on Windows, Linux, and macOS × amd64/arm64.
Per-OS variance lives in the manifest first; OS primitives (process
termination, console hiding) are the only build-tagged Go
(proc_windows.go / proc_unix.go).
Shutdown
Shuts down on stdin EOF (parent closed the pipe), SIGINT/SIGTERM, or a
shutdown JSON-RPC request — stopping any running engines first so none
are orphaned.