Router that virtually distributes inference across connected devices in the home.
Go
One engine proxy binary for Ollama and LM Studio
Noah (Terve) Tervalon6de9953be238
diff --git a/.cursor/rules/architecture.mdc b/.cursor/rules/architecture.mdcindex 44555e6..3b14aa4 100644--- a/.cursor/rules/architecture.mdc+++ b/.cursor/rules/architecture.mdc@@ -72,8 +72,9 @@ Inference HTTP endpoints are for local client-compatible traffic.- Hide operations the backend cannot perform.Routing is owned by the proxies, broker, and `nvpair-job-scheduler`. The backend-combines pending work, compact scanner/manual GPU telemetry, and per-proxy burst-reservations. PAIR must not pin routes, derive policy from renderer metrics, or+combines pending work, compact scanner/manual GPU telemetry, and the proxy+process's burst reservations, which every engine facade shares. PAIR must not+pin routes, derive policy from renderer metrics, orimplement a TypeScript scheduling policy.For model-bearing inference, each proxy admits only nodes whose per-enginediff --git a/.cursor/rules/proxy-inference-routing.mdc b/.cursor/rules/proxy-inference-routing.mdcindex 8414469..9dabf40 100644--- a/.cursor/rules/proxy-inference-routing.mdc+++ b/.cursor/rules/proxy-inference-routing.mdc@@ -9,10 +9,11 @@ SPDX-License-Identifier: Apache-2.0# Proxy and Inference Routing-Routing is owned by `ollama-proxy`, `lmstudio-proxy`,+Routing is owned by `nvpair-proxy` (one process hosting a facade per engine,+addressed by clients as `ollama-proxy:` / `lmstudio-proxy:`),`nvpair-job-scheduler`, and `nvpair-ui-broker`.-For model-bearing inference, each proxy first filters a request-local discovery+For model-bearing inference, each facade first filters a request-local discoverysnapshot to nodes whose per-engine inventory advertises the requested model.Empty and non-matching inventories are ineligible; an empty owner set returns alocal `502` without contacting an engine. Ollama normalizes the implicit@@ -37,9 +38,17 @@ The scheduler smooths fresh utilization into pressure 0–3, treats invalid,missing, or older-than-10-second telemetry as neutral pressure 1, and ranks by`pending + gpuPressure`, then pressure, then stable UUID. It emits`schedule:priority` snapshots containing order, pending counts, and pressure.-Each proxy chooses with `pending + gpuPressure + localReservations`, so+Each facade chooses with `pending + gpuPressure + localReservations`, soconcurrent bursts do not wait for workload feedback.+Those reservations are **process-wide, shared by every facade**. Two facades+bursting at once compete for the same node's GPU, so a dispatch through either+must be visible to the other; this is the reason the engines share one process.+A reservation is stamped with the snapshot generation it was counted against,+released when its request ends, and moved to the node a failover actually+lands on. A snapshot supersedes the reservations taken before it, so a release+arriving after one is ignored rather than double-counted.+## PAIR responsibilities- Register and pass the scheduler path through the canonical modular binarydiff --git a/.cursor/rules/services-change-propagation.mdc b/.cursor/rules/services-change-propagation.mdcindex 2742cf1..d11ea1e 100644--- a/.cursor/rules/services-change-propagation.mdc+++ b/.cursor/rules/services-change-propagation.mdc@@ -38,8 +38,10 @@ SPDX-License-Identifier: Apache-2.0- Contract or payload changes: update every producer and consumer, shared types,protocol docs, and cross-process tests. Update `desktop/` bridge handlers,shared channel types, and run `npm --prefix desktop run service-contracts:check`.-- Proxy behavior: inspect both `ollama-proxy` and `lmstudio-proxy`; preserve their- intentional parity unless the behavior is engine-specific.+- Proxy behavior: `nvpair-proxy` serves every engine, so a change there applies+ to all of them at once. Check the per-engine descriptors in+ `services/nvpair-proxy/engines.go` and the broker's `engineProxyProfile`+ before assuming a change is engine-neutral.- Discovery or identity changes: inspect `shared/noderec`, node scanner, brokerrelay, manual nodes, scheduler/proxies, and UI-facing snapshots.- New, renamed, or removed binaries/services: update broker supervision, alldiff --git a/.cursor/rules/system-architecture.mdc b/.cursor/rules/system-architecture.mdcindex 44e97f6..7b3b0f3 100644--- a/.cursor/rules/system-architecture.mdc+++ b/.cursor/rules/system-architecture.mdc@@ -25,8 +25,11 @@ The canonical inventory is `src/shared/constants/modular-binaries.ts`.Broker-owned workers:-- `ollama-proxy`;-- `lmstudio-proxy`;+- `nvpair-proxy`, one process hosting a facade per enabled engine. It starts+ with no engine and no listener; the broker sends a `facade/enable` per engine+ carrying that engine's port. Clients still address each facade as+ `ollama-proxy:` / `lmstudio-proxy:`, and one supervisor covers them all, so a+ crash is reported against `nvpair-proxy` and restarts every facade together;- `nvpair-node-scanner`;- `nvpair-node-info`;- `nvpair-workload-manager`;diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.mdindex 6aca556..d51203c 100644--- a/THIRD_PARTY_NOTICES.md+++ b/THIRD_PARTY_NOTICES.md@@ -15,7 +15,7 @@ distributed under its own license; the full license text for each component isreproduced in this file.Scope: npm dependencies and the Electron runtime shipped in the desktop-application, plus the Go modules linked into the thirteen shipped service+application, plus the Go modules linked into the twelve shipped servicebinaries across the Windows, Linux, and macOS targets. First-party modules(`nvpair-shared`, `eapnoob`) are excluded.@@ -400,26 +400,26 @@ License: MITRepository: https://github.com/asamuzaK/cssColor```text-MIT License--Copyright (c) 2024 asamuzaK (Kazz)--Permission is hereby granted, free of charge, to any person obtaining a copy-of this software and associated documentation files (the "Software"), to deal-in the Software without restriction, including without limitation the rights-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell-copies of the Software, and to permit persons to whom the Software is-furnished to do so, subject to the following conditions:--The above copyright notice and this permission notice shall be included in all-copies or substantial portions of the Software.--THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+MIT License++Copyright (c) 2024 asamuzaK (Kazz)++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THESOFTWARE.```@@ -430,26 +430,26 @@ License: MITRepository: https://github.com/asamuzaK/domSelector```text-MIT License--Copyright (c) 2023 asamuzaK (Kazz)--Permission is hereby granted, free of charge, to any person obtaining a copy-of this software and associated documentation files (the "Software"), to deal-in the Software without restriction, including without limitation the rights-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell-copies of the Software, and to permit persons to whom the Software is-furnished to do so, subject to the following conditions:--The above copyright notice and this permission notice shall be included in all-copies or substantial portions of the Software.--THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+MIT License++Copyright (c) 2023 asamuzaK (Kazz)++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THESOFTWARE.```@@ -4847,27 +4847,27 @@ License: MITRepository: https://github.com/MikeMcl/decimal.js```text-The MIT Licence.--Copyright (c) 2025 Michael Mclaughlin--Permission is hereby granted, free of charge, to any person obtaining-a copy of this software and associated documentation files (the-'Software'), to deal in the Software without restriction, including-without limitation the rights to use, copy, modify, merge, publish,-distribute, sublicense, and/or sell copies of the Software, and to-permit persons to whom the Software is furnished to do so, subject to-the following conditions:--The above copyright notice and this permission notice shall be-included in all copies or substantial portions of the Software.--THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.-IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY-CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,-TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE+The MIT Licence.++Copyright (c) 2025 Michael Mclaughlin++Permission is hereby granted, free of charge, to any person obtaining+a copy of this software and associated documentation files (the+'Software'), to deal in the Software without restriction, including+without limitation the rights to use, copy, modify, merge, publish,+distribute, sublicense, and/or sell copies of the Software, and to+permit persons to whom the Software is furnished to do so, subject to+the following conditions:++The above copyright notice and this permission notice shall be+included in all copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THESOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.```@@ -10603,26 +10603,26 @@ License: MITRepository: https://github.com/Borewit/node-readable-to-web-readable-stream```text-MIT License--Copyright (c) 2025 Borewit--Permission is hereby granted, free of charge, to any person obtaining a copy-of this software and associated documentation files (the "Software"), to deal-in the Software without restriction, including without limitation the rights-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell-copies of the Software, and to permit persons to whom the Software is-furnished to do so, subject to the following conditions:--The above copyright notice and this permission notice shall be included in all-copies or substantial portions of the Software.--THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+MIT License++Copyright (c) 2025 Borewit++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THESOFTWARE.```@@ -15328,17 +15328,17 @@ License: 0BSDRepository: https://github.com/Microsoft/tslib```text-Copyright (c) Microsoft Corporation.--Permission to use, copy, modify, and/or distribute this software for any-purpose with or without fee is hereby granted.--THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH-REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY-AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,-INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM-LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR-OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR+Copyright (c) Microsoft Corporation.++Permission to use, copy, modify, and/or distribute this software for any+purpose with or without fee is hereby granted.++THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH+REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY+AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,+INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM+LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR+OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE ORPERFORMANCE OF THIS SOFTWARE.```diff --git a/desktop/docs/architecture.md b/desktop/docs/architecture.mdindex e7c72ab..32e1aa2 100644--- a/desktop/docs/architecture.md+++ b/desktop/docs/architecture.md@@ -42,8 +42,7 @@ The canonical runtime inventory is| Binary | Owner | Purpose || ------------------------- | ---------------- | ------------------------------------------- || `nvpair-ui-broker` | Electron | Worker supervision and control-plane relay |-| `ollama-proxy` | Broker | Ollama-compatible proxy and cluster routing |-| `lmstudio-proxy` | Broker, optional | LM Studio OpenAI-compatible proxy |+| `nvpair-proxy` | Broker | Engine proxy and cluster routing; one process hosting a facade per enabled engine || `nvpair-node-scanner` | Broker | LAN discovery and announcement || `nvpair-node-info` | Broker | Node metadata and telemetry endpoint || `nvpair-workload-manager` | Broker, optional | Workload replication |diff --git a/desktop/docs/macos-privileged-helper.md b/desktop/docs/macos-privileged-helper.mdindex 78f7893..7db727b 100644--- a/desktop/docs/macos-privileged-helper.md+++ b/desktop/docs/macos-privileged-helper.md@@ -83,11 +83,11 @@ The set in `native/PrivilegedHelper/main.swift` must stay in sync with the(`src/shared/constants/modular-binaries.ts`) and the manual uninstaller.`npm run service-contracts:check` fails when the Swift list differs from that canonical set:-`ollama-proxy`, `lmstudio-proxy`, `nvpair-node-info`, `nvpair-node-scanner`,+`nvpair-proxy`, `nvpair-node-info`, `nvpair-node-scanner`,`nvpair-workload-manager`, `nvpair-errors`, `nvpair-cluster-manager`,`nvpair-engine-manager`.-> These eight mirror the per-program/per-port `netsh` rules in+> These seven mirror the per-program/per-port `netsh` rules in> `scripts/build/installer.nsh` on Windows. macOS's Application Firewall is> per-application and inbound-only, so one `--add`/`--unblockapp` per binary> collapses Windows's per-port rules.diff --git a/desktop/docs/service-contract-exceptions.json b/desktop/docs/service-contract-exceptions.jsonindex 516fb2b..b5e638b 100644--- a/desktop/docs/service-contract-exceptions.json+++ b/desktop/docs/service-contract-exceptions.json@@ -1,9 +1,10 @@{"$comment": "Intentionally unintegrated JSON-RPC methods for desktop ↔ services contract checks. Each ignoredMethods entry needs a current technical reason. Edit by hand; run npm run service-contracts:write to refresh docs/services-api.md after services/ changes.","ignoredMethods": {- "schedule:priority": "Broker-internal. nvpair-job-scheduler emits a per-engine priority snapshot ({engine, nodes, ranks?} — ordered node ids plus each node's pending count) to nvpair-ui-broker, which forwards it to ollama-proxy and lmstudio-proxy via node/set-priority; each proxy then layers its own optimistic burst reservations on that baseline. PAIR consumes the routing result rather than this notification, and must not mirror the pending/reservation accounting.",+ "schedule:priority": "Broker-internal. nvpair-job-scheduler emits a per-engine priority snapshot ({engine, nodes, ranks?} — ordered node ids plus each node's pending count) to nvpair-ui-broker, which forwards it to each nvpair-proxy instance via node/set-priority; each proxy then layers its own optimistic burst reservations on that baseline. PAIR consumes the routing result rather than this notification, and must not mirror the pending/reservation accounting.","engine:restore-enabled": "Broker-internal startup restoration. nvpair-ui-broker emits engine:restore-enabled directly to its supervised engine-manager after the managed Ollama port gate and on manager respawn; it is not a renderer/UI notification.",- "proxy:ready": "Consumed, not missing: the broker relays it and normalizeBrokerProxy (modular-supervisor.ts) strips the `proxy:` prefix, so the bridge handles the de-prefixed `ready` (sets proxyPort). The literal `proxy:ready` is intentionally absent from our TS — extractor limitation, not a gap.",+ "ollama-proxy:ready": "Consumed, not missing: the broker relays it and normalizeBrokerProxy (modular-supervisor.ts) strips the `ollama-proxy:` prefix, so the bridge handles the de-prefixed `ready` (sets proxyPort). The literal `ollama-proxy:ready` is intentionally absent from our TS — extractor limitation, not a gap.",+ "lmstudio-proxy:ready": "Consumed, not missing: the LM Studio counterpart of ollama-proxy:ready, de-prefixed by the same normalizeBrokerProxy loop. It only became visible to the checker when METHOD_RE started accepting hyphens in a namespace segment; before that the whole lmstudio-proxy:* surface was silently unmatched.","node/selection-changed": "Automatic routing has no selected-node UI, so PAIR deliberately does not consume proxy selection changes.","proxy/request": "Per-request proxy telemetry is not rendered; workload lifecycle uses the broker workloads stream.","proxy/request-started": "Per-request proxy telemetry is not rendered; see proxy/request.",@@ -11,7 +12,7 @@"workload:started": "Proxy-to-broker lifecycle event translated by the broker into workloads:upsert, which PAIR consumes.","workload:completed": "Proxy-to-broker terminal lifecycle event translated by the broker into workloads:upsert.","workload:errored": "Proxy-to-broker terminal lifecycle event translated by the broker into workloads:upsert.",- "node/activity": "Proxy-to-broker liveness evidence, consumed entirely inside the backend. ollama-proxy and lmstudio-proxy raise it (coalesced to one report per node per 2s by nvpair-shared/nodeactivity) whenever a peer's engine returns inference response bytes; nvpair-ui-broker relays it to nvpair-node-scanner as discovery:node-activity, where it keeps a node that is busy serving inference from being evicted for failing a liveness probe it had no spare CPU to answer. PAIR consumes the result — the node staying in the discovery snapshot — not this per-request notification, which carries no state a UI could render. Not currently surfaced by the extractor either (it is emitted through a shared noderec constant, like nodeinfo:observed-addresses); this entry records the intent regardless.",+ "node/activity": "Proxy-to-broker liveness evidence, consumed entirely inside the backend. Each nvpair-proxy instance raises it (coalesced to one report per node per 2s by nvpair-shared/nodeactivity) whenever a peer's engine returns inference response bytes; nvpair-ui-broker relays it to nvpair-node-scanner as discovery:node-activity, where it keeps a node that is busy serving inference from being evicted for failing a liveness probe it had no spare CPU to answer. PAIR consumes the result — the node staying in the discovery snapshot — not this per-request notification, which carries no state a UI could render. Not currently surfaced by the extractor either (it is emitted through a shared noderec constant, like nodeinfo:observed-addresses); this entry records the intent regardless.","errors:report": "Producer-to-broker event demultiplexed into broker-owned nvpair-errors. PAIR consumes the resulting errors:update snapshot.","errors:clear": "Producer-to-broker event demultiplexed into broker-owned nvpair-errors. PAIR consumes the resulting errors:update snapshot.","cluster:trust-changed": "Broker-internal. nvpair-cluster-manager announces it from its trust store after a pin write lands, and nvpair-ui-broker answers it by calling discovery:reload-trust on nvpair-node-scanner so the scanner re-derives each peer's trusted annotation. PAIR consumes the result — the refreshed AvailableNode.trusted carried on the discovery snapshot — not this notification. It carries no payload by design: recipients re-read the cluster dir rather than trusting a diff on the wire.",diff --git a/desktop/docs/services-api.md b/desktop/docs/services-api.mdindex 590e726..e7fea69 100644--- a/desktop/docs/services-api.md+++ b/desktop/docs/services-api.md@@ -16,8 +16,6 @@- none ✅### Requests the backend handles but the bridge never calls (unused capability)-- ⚠️ lmstudio-proxy → node/selected-- ⚠️ lmstudio-proxy → node/set-local-backend- ⚠️ nvpair-engine-manager → engine:describe- ⚠️ nvpair-engine-manager → engine:errors- ⚠️ nvpair-engine-manager → engine:logs@@ -32,44 +30,23 @@- ⚠️ nvpair-node-settings → settings/get-force-ports- ⚠️ nvpair-node-settings → settings/set-cluster-auto-sync- ⚠️ nvpair-node-settings → settings/set-force-ports+- ⚠️ nvpair-proxy → facade/enable+- ⚠️ nvpair-proxy → node/selected+- ⚠️ nvpair-proxy → node/set-local-backend- ⚠️ nvpair-ui-broker → discovery:unsubscribe- ⚠️ nvpair-ui-broker → engine:set-reserved-port- ⚠️ nvpair-ui-broker → engine:unsubscribe- ⚠️ nvpair-ui-broker → internal:set-reserved-port-- ⚠️ nvpair-ui-broker → proxy:get-status-- ⚠️ nvpair-ui-broker → proxy:unsubscribe+- ⚠️ nvpair-ui-broker → lmstudio-proxy:get-status+- ⚠️ nvpair-ui-broker → lmstudio-proxy:set-port+- ⚠️ nvpair-ui-broker → lmstudio-proxy:unsubscribe+- ⚠️ nvpair-ui-broker → ollama-proxy:get-status+- ⚠️ nvpair-ui-broker → ollama-proxy:unsubscribe- ⚠️ nvpair-ui-broker → workloads:unsubscribe-- ⚠️ ollama-proxy → node/selected-- ⚠️ ollama-proxy → node/set-local-backend### Backend binaries not listed in `modular-binaries.ts`- none ✅-## lmstudio-proxy--| Method | Direction | In bridge? |-|---|---|---|-| `error` | notification (we consume) | ✅ yes |-| `errors:clear` | notification (we consume) | ✅ yes |-| `errors:report` | notification (we consume) | ✅ yes |-| `node/discovered` | notification (we consume) | ✅ yes |-| `node/removed` | notification (we consume) | ✅ yes |-| `node/selection-changed` | notification (we consume) | ➖ ignored |-| `node/updated` | notification (we consume) | ✅ yes |-| `proxy/request` | notification (we consume) | ✅ yes |-| `proxy/request-started` | notification (we consume) | ➖ ignored |-| `ready` | notification (we consume) | ✅ yes |-| `node/add-manual` | request (we call) | ✅ yes |-| `node/remove-manual` | request (we call) | ✅ yes |-| `node/select` | request (we call) | ✅ yes |-| `node/selected` | request (we call) | ⚠️ not called |-| `node/set-local-backend` | request (we call) | ⚠️ not called |-| `node/set-priority` | request (we call) | ✅ yes |-| `nodes/list` | request (we call) | ✅ yes |--**Dynamic / unresolved notify sites (verify by hand — `npm run service-contracts` prints the line numbers):**-- `method (var) (proxy.go)`-## nvpair-cluster-manager| Method | Direction | In bridge? |@@ -203,6 +180,32 @@| `settings/set-cluster-id` | request (we call) | ✅ yes || `settings/set-force-ports` | request (we call) | ⚠️ not called |+## nvpair-proxy++| Method | Direction | In bridge? |+|---|---|---|+| `error` | notification (we consume) | ✅ yes |+| `errors:clear` | notification (we consume) | ✅ yes |+| `errors:report` | notification (we consume) | ✅ yes |+| `node/discovered` | notification (we consume) | ✅ yes |+| `node/removed` | notification (we consume) | ✅ yes |+| `node/selection-changed` | notification (we consume) | ➖ ignored |+| `node/updated` | notification (we consume) | ✅ yes |+| `proxy/request` | notification (we consume) | ✅ yes |+| `proxy/request-started` | notification (we consume) | ➖ ignored |+| `ready` | notification (we consume) | ✅ yes |+| `facade/enable` | request (we call) | ⚠️ not called |+| `node/add-manual` | request (we call) | ✅ yes |+| `node/remove-manual` | request (we call) | ✅ yes |+| `node/select` | request (we call) | ✅ yes |+| `node/selected` | request (we call) | ⚠️ not called |+| `node/set-local-backend` | request (we call) | ⚠️ not called |+| `node/set-priority` | request (we call) | ✅ yes |+| `nodes/list` | request (we call) | ✅ yes |++**Dynamic / unresolved notify sites (verify by hand — `npm run service-contracts` prints the line numbers):**+- `method (var) (proxy.go)`+## nvpair-tui| Method | Direction | In bridge? |@@ -227,7 +230,8 @@| `errors:clear` | notification (we consume) | ✅ yes || `errors:report` | notification (we consume) | ✅ yes || `errors:update` | notification (we consume) | ✅ yes |-| `proxy:ready` | notification (we consume) | ✅ yes |+| `lmstudio-proxy:ready` | notification (we consume) | ➖ ignored |+| `ollama-proxy:ready` | notification (we consume) | ✅ yes || `workloads:upsert` | notification (we consume) | ✅ yes || `connection/cluster-auto-sync` | request (we call) | ➖ ignored || `connection/cluster-identity` | request (we call) | ✅ yes |@@ -242,16 +246,20 @@| `engine:unsubscribe` | request (we call) | ⚠️ not called || `errors:get-initial` | request (we call) | ✅ yes || `internal:set-reserved-port` | request (we call) | ⚠️ not called |+| `lmstudio-proxy:get-status` | request (we call) | ⚠️ not called |+| `lmstudio-proxy:set-port` | request (we call) | ⚠️ not called |+| `lmstudio-proxy:subscribe` | request (we call) | ✅ yes |+| `lmstudio-proxy:unsubscribe` | request (we call) | ⚠️ not called || `node/add` | request (we call) | ✅ yes || `node/discovered` | request (we call) | ✅ yes || `node/remove` | request (we call) | ✅ yes || `node/removed` | request (we call) | ✅ yes || `node/updated` | request (we call) | ✅ yes || `nodes/list` | request (we call) | ✅ yes |-| `proxy:get-status` | request (we call) | ⚠️ not called |-| `proxy:set-port` | request (we call) | ✅ yes |-| `proxy:subscribe` | request (we call) | ✅ yes |-| `proxy:unsubscribe` | request (we call) | ⚠️ not called |+| `ollama-proxy:get-status` | request (we call) | ⚠️ not called |+| `ollama-proxy:set-port` | request (we call) | ✅ yes |+| `ollama-proxy:subscribe` | request (we call) | ✅ yes |+| `ollama-proxy:unsubscribe` | request (we call) | ⚠️ not called || `ready` | request (we call) | ✅ yes || `workloads:get-initial` | request (we call) | ✅ yes || `workloads:remove` | request (we call) | ✅ yes |@@ -260,11 +268,9 @@**Dynamic / unresolved notify sites (verify by hand — `npm run service-contracts` prints the line numbers):**- `method (var) (broker.go, 5 sites)`-- `proxy:* (broker.go)`- `method (var) (clustermanager.go)`- `method (var) (errors.go)`-- `lmstudio-proxy:* (lmstudioproxy.go)`-- `method (var) (proxy.go)`+- `method (var) (proxy.go, 2 sites)`- `method (var) (rpcworker.go, 2 sites)`## nvpair-workload-manager@@ -275,28 +281,3 @@| `workloads:remove` | notification (we consume) | ✅ yes || `workloads:upsert` | notification (we consume) | ✅ yes |-## ollama-proxy--| Method | Direction | In bridge? |-|---|---|---|-| `error` | notification (we consume) | ✅ yes |-| `errors:clear` | notification (we consume) | ✅ yes |-| `errors:report` | notification (we consume) | ✅ yes |-| `node/discovered` | notification (we consume) | ✅ yes |-| `node/removed` | notification (we consume) | ✅ yes |-| `node/selection-changed` | notification (we consume) | ➖ ignored |-| `node/updated` | notification (we consume) | ✅ yes |-| `proxy/request` | notification (we consume) | ✅ yes |-| `proxy/request-started` | notification (we consume) | ➖ ignored |-| `ready` | notification (we consume) | ✅ yes |-| `node/add-manual` | request (we call) | ✅ yes |-| `node/remove-manual` | request (we call) | ✅ yes |-| `node/select` | request (we call) | ✅ yes |-| `node/selected` | request (we call) | ⚠️ not called |-| `node/set-local-backend` | request (we call) | ⚠️ not called |-| `node/set-priority` | request (we call) | ✅ yes |-| `nodes/list` | request (we call) | ✅ yes |--**Dynamic / unresolved notify sites (verify by hand — `npm run service-contracts` prints the line numbers):**-- `method (var) (proxy.go)`-diff --git a/desktop/docs/services-backend.md b/desktop/docs/services-backend.mdindex 0d54c4c..fe7ca6c 100644--- a/desktop/docs/services-backend.md+++ b/desktop/docs/services-backend.md@@ -21,8 +21,7 @@ broker supervises every worker and relays its control plane.| Binary | Runtime role || ------------------------- | --------------------------------------------------------- || `nvpair-ui-broker` | Worker supervision and relay |-| `ollama-proxy` | Ollama-compatible routing proxy with cluster-mTLS ingress |-| `lmstudio-proxy` | LM Studio routing proxy with cluster-mTLS ingress |+| `nvpair-proxy` | Engine routing proxy with cluster-mTLS ingress; one process hosting a facade per enabled engine || `nvpair-node-scanner` | Discovery and node announcement || `nvpair-node-info` | Node metadata and telemetry || `nvpair-manual-nodes` | User-managed node entries |@@ -45,7 +44,7 @@ flowchart TBBroker["nvpair-ui-broker"]Scanner["nvpair-node-scanner"]NodeInfo["nvpair-node-info"]- Proxies["ollama-proxy / lmstudio-proxy"]+ Proxies["nvpair-proxy (one process, a facade per engine)"]Engines["nvpair-engine-manager"]Cluster["nvpair-cluster-manager"]Settings["nvpair-node-settings"]@@ -117,7 +116,7 @@ reserved for inference clients.| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- || `app:ready` | Complete broker startup and refresh snapshots | `state:request-refresh` || `discovery:nodes-changed` | Replace discovery snapshot and diff nodes | `discovery:nodes-changed`, `nodes:upsert`, `nodes:remove` |-| `proxy:ready` / `lmstudio-proxy:ready` | Record engine proxy port | `engines:state-changed` |+| `ollama-proxy:ready` / `lmstudio-proxy:ready` | Record engine proxy port | `engines:state-changed` || proxy `node/*` | Update per-engine node presence; the advertised port is the peer's promoted proxy port (not the engine's private loopback port) | node and engine pushes || `engine:ready` / `engine:state-changed` | Update engine facts and models | `engines:state-changed` || `engine:install-progress` / `engine:remote-progress` | Update operation progress | engine progress pushes |@@ -262,8 +261,8 @@ resolved back to the hostname the entry was keyed by.### Secure inference (backend-owned)An NVPAIR-launched engine binds to loopback only and is never directly-LAN-reachable. Each node fronts its engine with its `ollama-proxy` /-`lmstudio-proxy`, whose LAN ingress is gated by cluster mTLS: only a pinned+LAN-reachable. Each node fronts all of its engines with one `nvpair-proxy`+process, and every facade's LAN ingress is gated by cluster mTLS: only a pinnedcluster member can send it work. Discovery advertises the promoted **proxy**port (never the engine port), and the broker hands the private loopback engine tothe local proxy via `node/set-local-backend`. Every cluster-scoped worker derivesdiff --git a/desktop/docs/services-parity.md b/desktop/docs/services-parity.mdindex 93d1329..6e50054 100644--- a/desktop/docs/services-parity.md+++ b/desktop/docs/services-parity.md@@ -39,8 +39,8 @@ history.`nvpair-ui-broker` is the sole Electron child. It owns:-- `ollama-proxy`;-- `lmstudio-proxy`;+- `nvpair-proxy`, one process hosting a facade per enabled engine, under one+ supervisor;- `nvpair-node-scanner`;- `nvpair-node-info`;- `nvpair-manual-nodes`;@@ -99,10 +99,16 @@ they survive worker restarts.## Routing and inference-Both text-engine proxies are broker-owned and cluster-aware:+Both text-engine facades are broker-owned and cluster-aware. They live in one+`nvpair-proxy` process, each enabled after spawn on its own port, and each+serves its engine's dialect:-- `ollama-proxy` serves the Ollama-compatible surface;-- `lmstudio-proxy` serves the LM Studio/OpenAI-compatible surface.+- the Ollama facade serves the Ollama-compatible surface;+- the LM Studio facade serves the LM Studio/OpenAI-compatible surface.++Sharing a process is what lets them share the burst reservations the scheduler+depends on: two facades bursting at once compete for the same node's GPU, so a+dispatch through either has to be visible to the other.Routing precedence is manual selection, scheduler priority, then deterministicproxy ordering. Personal AI Router leaves proxies in automatic mode.diff --git a/desktop/native/PrivilegedHelper/main.swift b/desktop/native/PrivilegedHelper/main.swiftindex a0c4e2e..03ba72e 100644--- a/desktop/native/PrivilegedHelper/main.swift+++ b/desktop/native/PrivilegedHelper/main.swift@@ -36,8 +36,7 @@ enum Firewall {/// verifies this) and the manual uninstaller/// (scripts/build/macos/uninstall.sh).static let networkedBinaries = [- "ollama-proxy",- "lmstudio-proxy",+ "nvpair-proxy","nvpair-node-info","nvpair-node-scanner","nvpair-workload-manager",@@ -46,16 +45,39 @@ enum Firewall {"nvpair-engine-manager"]+ /// Names an earlier version granted access to and this one no longer+ /// ships. Removal-only: never added, only revoked, so an upgrade does not+ /// leave allow entries behind for binaries that are gone. Deliberately not+ /// part of `networkedBinaries`, which must keep matching the canonical+ /// firewall metadata that `service-contracts:check` verifies.+ static let legacyNetworkedBinaries = [+ "ollama-proxy",+ "lmstudio-proxy"+ ]+static func apply(cliBinDir: String, unblock: Bool) throws {guard FileManager.default.isExecutableFile(atPath: socketfilterfw) else {throw FirewallError.socketfilterfwMissing}let fm = FileManager.default- for name in networkedBinaries {- let binPath = (cliBinDir as NSString).appendingPathComponent(name)- guard fm.fileExists(atPath: binPath) else { continue }- if unblock {+ if unblock {+ // Revoke what an earlier version granted and this one no longer+ // ships, BEFORE granting the current set.+ //+ // This has to happen here rather than on the removal path, because+ // configure-firewall is the only firewall entry point the app ever+ // invokes — nothing calls remove-firewall — and it is what runs+ // after a stale-daemon re-register on upgrade. A removal-only+ // legacy pass would never fire.+ for name in legacyNetworkedBinaries {+ revoke(cliBinDir: cliBinDir, name: name)+ }++ for name in networkedBinaries {+ let binPath = (cliBinDir as NSString).appendingPathComponent(name)+ guard fm.fileExists(atPath: binPath) else { continue }+// Granting network access — be certain this is exactly our// binary. TOCTOU guard: re-check identity of this object// immediately before socketfilterfw. Reject symlinks (so a@@ -73,18 +95,45 @@ enum Firewall {}try runChecked(socketfilterfw, ["--add", binPath])try runChecked(socketfilterfw, ["--unblockapp", binPath])- } else {- // Removal only revokes access, so it is best-effort: still reject- // a symlink (never touch a swapped target), but do NOT abort the- // whole cleanup if one rule was never added — `socketfilterfw- // --remove` of an absent entry returns non-zero, which must not- // strand the remaining removals (e.g. during uninstall).- do { try assertRealFile(binPath) } catch { continue }- _ = run(socketfilterfw, ["--remove", binPath])}+ return+ }++ for name in networkedBinaries + legacyNetworkedBinaries {+ revoke(cliBinDir: cliBinDir, name: name)}}+ /// Best-effort revoke of one binary's firewall entry.+ ///+ /// Never aborts the caller: `socketfilterfw --remove` of an entry that was+ /// never added exits non-zero, which must not strand the remaining+ /// removals during an uninstall.+ ///+ /// Revokes when the path holds a real regular file, or holds nothing at+ /// all — an upgrade deletes the old binary before this runs, and that+ /// stale rule is exactly what needs clearing. A symlink is never touched:+ /// `assertRealFile` opens with `O_NOFOLLOW`, so it rejects a symlink and an+ /// absent path alike, and absence is therefore established with `lstat`,+ /// which (unlike `FileManager.fileExists`) does not follow the link.+ static func revoke(cliBinDir: String, name: String) {+ let binPath = (cliBinDir as NSString).appendingPathComponent(name)+ do {+ try assertRealFile(binPath)+ } catch {+ guard pathIsAbsent(binPath) else { return }+ }+ _ = run(socketfilterfw, ["--remove", binPath])+ }++ /// True when nothing exists at `path`. Uses `lstat`, so a symlink —+ /// dangling or not — counts as present rather than being mistaken for an+ /// empty path.+ static func pathIsAbsent(_ path: String) -> Bool {+ var info = stat()+ return lstat(path, &info) != 0 && errno == ENOENT+ }+/// Validate that `appPath` (already derived from the *verified* connecting/// process, never from caller argv) is the canonical, NVIDIA-signed host/// bundle and return its cli-bin directory. Defense in depth on top of thediff --git a/desktop/scripts/build/installer.nsh b/desktop/scripts/build/installer.nshindex 9cb87ed..1517393 100644--- a/desktop/scripts/build/installer.nsh+++ b/desktop/scripts/build/installer.nsh@@ -17,6 +17,9 @@; src/shared/constants/modular-binaries.ts.nsExec::ExecToLog 'taskkill /F /T /IM "${APP_EXECUTABLE_FILENAME}"'nsExec::ExecToLog 'taskkill /F /T /IM "nvpair-tui.exe"'+ nsExec::ExecToLog 'taskkill /F /T /IM "nvpair-proxy.exe"'+ ; Pre-unification names. An orphan still holding 11434 or 1234 is exactly+ ; what the managed-facade planner has to block on, so kill it here too.nsExec::ExecToLog 'taskkill /F /T /IM "ollama-proxy.exe"'nsExec::ExecToLog 'taskkill /F /T /IM "lmstudio-proxy.exe"'nsExec::ExecToLog 'taskkill /F /T /IM "nvpair-node-info.exe"'@@ -120,16 +123,18 @@; expose the node on untrusted public networks.!macro pairAddFirewallRulesDetailPrint "Adding Personal AI Router firewall rules..."- nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router Ollama Proxy" dir=in action=allow program="$INSTDIR\resources\cli-bin\ollama-proxy.exe" enable=yes profile=any remoteip=localsubnet'- nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router LM Studio Proxy" dir=in action=allow program="$INSTDIR\resources\cli-bin\lmstudio-proxy.exe" enable=yes profile=any remoteip=localsubnet'+ nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router Engine Proxy" dir=in action=allow program="$INSTDIR\resources\cli-bin\nvpair-proxy.exe" enable=yes profile=any remoteip=localsubnet'nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router Node Info" dir=in action=allow program="$INSTDIR\resources\cli-bin\nvpair-node-info.exe" enable=yes profile=any remoteip=localsubnet'nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router Node Scanner" dir=in action=allow program="$INSTDIR\resources\cli-bin\nvpair-node-scanner.exe" enable=yes profile=any remoteip=localsubnet'nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router Workload Manager" dir=in action=allow program="$INSTDIR\resources\cli-bin\nvpair-workload-manager.exe" enable=yes profile=any remoteip=localsubnet'nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router Errors" dir=in action=allow program="$INSTDIR\resources\cli-bin\nvpair-errors.exe" enable=yes profile=any remoteip=localsubnet'nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router Cluster Manager" dir=in action=allow program="$INSTDIR\resources\cli-bin\nvpair-cluster-manager.exe" enable=yes profile=any remoteip=localsubnet'nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router Engine Manager" dir=in action=allow program="$INSTDIR\resources\cli-bin\nvpair-engine-manager.exe" enable=yes profile=any remoteip=localsubnet'- nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router mDNS (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\resources\cli-bin\ollama-proxy.exe" enable=yes profile=any remoteip=localsubnet'- nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router mDNS LM Studio Proxy (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\resources\cli-bin\lmstudio-proxy.exe" enable=yes profile=any remoteip=localsubnet'+ ; No mDNS rule for nvpair-proxy: it opens no UDP socket. Its routing targets+ ; arrive over the broker's discovery relay, and zeroconf/miekg are indirect+ ; dependencies it never browses with. The pre-unification ollama-proxy did+ ; browse, which is where this rule came from; the delete below stays so an+ ; upgrade clears the inherited entry.nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router mDNS Node Info (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\resources\cli-bin\nvpair-node-info.exe" enable=yes profile=any remoteip=localsubnet'nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router mDNS Node Scanner (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\resources\cli-bin\nvpair-node-scanner.exe" enable=yes profile=any remoteip=localsubnet'nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router mDNS Workload Manager (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\resources\cli-bin\nvpair-workload-manager.exe" enable=yes profile=any remoteip=localsubnet'@@ -141,6 +146,10 @@!macro pairRemoveFirewallRulesDetailPrint "Removing Personal AI Router firewall rules..."+ nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router Engine Proxy"'+ ; The per-engine rules a pre-unification install created. Deleting a rule+ ; that does not exist is a harmless no-op, and without these an upgrade+ ; leaves rules pointing at binaries this version no longer ships.nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router Ollama Proxy"'nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router LM Studio Proxy"'nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router Node Info"'@@ -150,6 +159,7 @@nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router Cluster Manager"'nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router Engine Manager"'nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS (UDP 5353)"'+ ; Pre-unification; see above.nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS LM Studio Proxy (UDP 5353)"'nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS Node Info (UDP 5353)"'nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS Node Scanner (UDP 5353)"'diff --git a/desktop/scripts/build/macos/uninstall.sh b/desktop/scripts/build/macos/uninstall.shindex ce88d22..ec948cc 100644--- a/desktop/scripts/build/macos/uninstall.sh+++ b/desktop/scripts/build/macos/uninstall.sh@@ -56,6 +56,7 @@ echo "Stopping Personal AI Router processes..."for proc in \"PAIR" \"nvpair-tui" \+ "nvpair-proxy" \"ollama-proxy" \"lmstudio-proxy" \"nvpair-node-info" \@@ -74,7 +75,10 @@ sleep 1FW=/usr/libexec/ApplicationFirewall/socketfilterfwif [ -x "$FW" ]; then- for bin in ollama-proxy lmstudio-proxy nvpair-node-info nvpair-node-scanner \+ # ollama-proxy and lmstudio-proxy are pre-unification names, kept so an+ # upgrade's leftover firewall entries are removed too; --remove on a path+ # that was never added is a no-op.+ for bin in nvpair-proxy ollama-proxy lmstudio-proxy nvpair-node-info nvpair-node-scanner \nvpair-workload-manager nvpair-errors nvpair-cluster-manager nvpair-engine-manager; do"$FW" --remove "$APP_PATH/Contents/Resources/cli-bin/$bin" >/dev/null 2>&1 || truedonediff --git a/desktop/scripts/collect-go-licenses.ts b/desktop/scripts/collect-go-licenses.tsindex c42323b..56be4c3 100644--- a/desktop/scripts/collect-go-licenses.ts+++ b/desktop/scripts/collect-go-licenses.ts@@ -5,7 +5,7 @@* Collects license information for the Go modules linked into the shipped* service binaries.*- * The installer bundles all thirteen Go executables from `../services` as+ * The installer bundles all twelve Go executables from `../services` as* `extraResources`, so their dependencies are redistributed and need attribution* in the notice file that ships inside the application.*diff --git a/desktop/scripts/generate-licenses.ts b/desktop/scripts/generate-licenses.tsindex 65a2aee..eb1ab9c 100644--- a/desktop/scripts/generate-licenses.ts+++ b/desktop/scripts/generate-licenses.ts@@ -7,7 +7,7 @@* - every runtime (production) npm dependency + all their transitives* - the Electron runtime itself (declared as a devDependency, but shipped* with the packaged app, so legally it has to appear in the report)- * - the Go modules linked into the thirteen service binaries, which the+ * - the Go modules linked into the twelve service binaries, which the* installer ships from `../services` as extraResources** Dev-only tooling (vite, tsx, typescript, prettier, knip, etc.) is deliberately@@ -145,7 +145,7 @@ function renderMarkdown(entries: Entry[]): string {'reproduced in this file.','','Scope: npm dependencies and the Electron runtime shipped in the desktop',- 'application, plus the Go modules linked into the thirteen shipped service',+ 'application, plus the Go modules linked into the twelve shipped service','binaries across the Windows, Linux, and macOS targets. First-party modules','(`nvpair-shared`, `eapnoob`) are excluded.','',diff --git a/desktop/scripts/verify-service-contracts.ts b/desktop/scripts/verify-service-contracts.tsindex c3cd9c6..5323a45 100644--- a/desktop/scripts/verify-service-contracts.ts+++ b/desktop/scripts/verify-service-contracts.ts@@ -91,7 +91,11 @@ interface BinarySurface {dynamic: DynamicSite[]}-const METHOD_RE = /^(?:[a-z][a-zA-Z0-9]*(?:[:/][a-zA-Z0-9.-]+)+|ready|error)$/+// The leading segment allows hyphens because every engine relay namespace is+// its proxy's component id — "ollama-proxy:get-status", "lmstudio-proxy:ready".+// Without the hyphen this silently matched nothing under either namespace and+// the drift gate passed by seeing no methods at all.+const METHOD_RE = /^(?:[a-z][a-zA-Z0-9-]*(?:[:/][a-zA-Z0-9.-]+)+|ready|error)$/function isMethodish(s: string): boolean {return METHOD_RE.test(s)diff --git a/desktop/src/electron/service-bridge/empty-handlers.ts b/desktop/src/electron/service-bridge/empty-handlers.tsindex dc5c2dd..bc8bf60 100644--- a/desktop/src/electron/service-bridge/empty-handlers.ts+++ b/desktop/src/electron/service-bridge/empty-handlers.ts@@ -211,10 +211,11 @@ async function applyEnginePort(/*** Set the local Ollama proxy's listen port. Routed through the broker's- * `proxy:set-port` relay, which persists the port (`proxy-port.json`, restored on- * restart), live-rebinds the proxy, and **steers it clear of any running engine- * port** (engines win — a bumped proxy surfaces a sticky `warning` on the errors- * pipeline). The proxy re-emits `proxy:ready` with the actually-bound port, which+ * `ollama-proxy:set-port` relay, which persists the port (`proxy-port.json`,+ * restored on restart), live-rebinds the proxy, and **steers it clear of any+ * running engine port** (engines win — a bumped proxy surfaces a sticky+ * `warning` on the errors pipeline). The proxy re-emits `ollama-proxy:ready`+ * with the actually-bound port, which* the bridge already folds into `status.proxyPort`, so success needs no further* action here beyond surfacing a failure.*/@@ -256,7 +257,7 @@ async function applyProxyPort(proxyEngine: ProxyEngine, port: number): Promise<v* the proxy (the engine's old port is now free for the proxy's target),* persist the engine's new port, then start it back up.*- * Every success path ends in an `engine:set-port` / `doStart` / `proxy:ready`+ * Every success path ends in an `engine:set-port` / `doStart` / `<engine>-proxy:ready`* that re-emits authoritative state, which clears the optimistic op and snaps* the UI inputs to backend truth. On failure we clear the optimistic op and* report, so the UI reverts to the real ports.diff --git a/desktop/src/electron/service-bridge/modular-state.ts b/desktop/src/electron/service-bridge/modular-state.tsindex ab6c5fb..ee178d3 100644--- a/desktop/src/electron/service-bridge/modular-state.ts+++ b/desktop/src/electron/service-bridge/modular-state.ts@@ -36,6 +36,14 @@ import { serviceLogLevel } from './service-log-level'type ProxyNodeSource = 'ollama-proxy' | 'lmstudio-proxy'type BrokerNodeSource = ProxyNodeSource | 'broker'+/**+ * The engine proxies, by the one name that identifies each of them everywhere:+ * the broker's relay prefix, the errors-pipeline source, and the node source+ * recorded here. That is `ComponentName` in `services/shared/engines`, always+ * `<engine>-proxy`.+ */+export const PROXY_NODE_SOURCES: readonly ProxyNodeSource[] = ['ollama-proxy', 'lmstudio-proxy']+/*** Engines surfaced by the broker's proxy plane. Other engine-manager engines* are not currently routed across nodes.@@ -328,7 +336,7 @@ export function parseServiceErrors(value: JsonValue | undefined): ServiceError[]* Per-node proxy "upstream unreachable" warnings duplicate the node list's* offline group, so they are never surfaced in the error UI. Backend id shape:* `<engine>-proxy:upstream-unreachable:<nodeId>`- * (services/ollama-proxy/proxy.go `upstreamUnreachableID`).+ * (services/nvpair-proxy/proxy.go `upstreamUnreachableID`).*/export function isUpstreamUnreachableError(error: ServiceError): boolean {return error.id.includes(':upstream-unreachable:')@@ -2279,7 +2287,7 @@ class ModularBridgeState {}handleNotification(notification: JsonRpcNotification): void {- if (notification.source === 'proxy') {+ if (notification.source === 'ollama-proxy') {this.handleProxyNotification(notification, 'ollama')return}diff --git a/desktop/src/electron/service-bridge/modular-supervisor.ts b/desktop/src/electron/service-bridge/modular-supervisor.tsindex 943d0f8..fac10a0 100644--- a/desktop/src/electron/service-bridge/modular-supervisor.ts+++ b/desktop/src/electron/service-bridge/modular-supervisor.ts@@ -22,6 +22,7 @@ import {parseServiceErrors,parseWorkloadsInitial,PROXY_ENGINES,+ PROXY_NODE_SOURCES,type ProxyEngine} from './modular-state'import { emitBridgePush } from './broadcaster'@@ -304,7 +305,7 @@ function proxyEngineFromManagerId(id: string): ProxyEngine | null {/** The broker relay namespace fronting an engine's reverse proxy. */function proxyRelayPrefix(engine: ProxyEngine): string {- return engine === 'ollama' ? 'proxy' : 'lmstudio-proxy'+ return engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy'}/**@@ -319,7 +320,7 @@ function proxyRelayPrefix(engine: ProxyEngine): string {* `nvpair-cluster-manager`, `nvpair-node-settings`, `nvpair-manual-nodes`,* `nvpair-engine-manager`, `nvpair-errors`, `nvpair-job-scheduler`). Electron passes their resolved paths to* the broker (see `brokerStartupArgs`) and reaches each through a broker relay:- * `proxy:` / `lmstudio-proxy:` for the two engine proxies, `engine:` for the+ * `ollama-proxy:` / `lmstudio-proxy:` for the two engine proxies, `engine:` for the* engine-manager, `errors:` for the error pipeline, `node/*` for manual nodes,* `settings/*` and `cluster:` for the rest. Local inference jobs arrive on the* broker's `workloads:subscribe` stream.@@ -825,8 +826,9 @@ class ModularSupervisor {}passPath('--scanner-path', 'scanner')passPath('--node-info-path', 'node-info')- passPath('--proxy-path', 'proxy')- passPath('--lmstudio-proxy-path', 'lmstudio-proxy')+ // One process fronts every engine; the broker starts it once and enables+ // a facade per entry in its own --proxy-engines default.+ passPath('--proxy-path', 'nvpair-proxy')passPath('--workload-manager-path', 'workload-manager')passPath('--cluster-manager-path', 'cluster-manager')passPath('--settings-path', 'node-settings')@@ -876,7 +878,7 @@ class ModularSupervisor {}}await subscribe('discovery:subscribe', 'subscribe to broker discovery')- await subscribe('proxy:subscribe', 'subscribe to broker ollama-proxy relay')+ await subscribe('ollama-proxy:subscribe', 'subscribe to broker ollama-proxy relay')await subscribe('lmstudio-proxy:subscribe', 'subscribe to broker lmstudio-proxy relay')// Engine events are opt-in and replay no baseline — subscribe then hydrate.await subscribe('engine:subscribe', 'subscribe to broker engine relay')@@ -1079,7 +1081,7 @@ class ModularSupervisor {const obj = objectValue(result)if (obj && booleanValue(obj.ready)) {getModularBridgeState().handleNotification({- source: engine === 'ollama' ? 'proxy' : 'lmstudio-proxy',+ source: engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy',method: 'ready',params: { port: numberValue(obj.port) }})@@ -1100,7 +1102,7 @@ class ModularSupervisor {if (!obj || !Array.isArray(obj.nodes)) returnfor (const node of obj.nodes) {getModularBridgeState().handleNotification({- source: engine === 'ollama' ? 'proxy' : 'lmstudio-proxy',+ source: engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy',method: 'node/discovered',params: node})@@ -1267,7 +1269,7 @@ class ModularSupervisor {}const proxyEngine: ProxyEngine | null =- event.source === 'proxy'+ event.source === 'ollama-proxy'? 'ollama': event.source === 'lmstudio-proxy'? 'lm-studio'@@ -1316,21 +1318,23 @@ class ModularSupervisor {this.readinessWaiters.clear()}- /** Rewrite broker `proxy:`/`lmstudio-proxy:` relay frames into proxy-source events. */+ /**+ * Rewrite a broker `<engine>-proxy:` relay frame into a proxy-source event.+ *+ * The prefix and the resulting source are the same string — the engine's+ * component id — so this stays a loop over the known proxy sources rather+ * than a branch per engine.+ */private normalizeBrokerProxy(notification: JsonRpcNotification): JsonRpcNotification {if (notification.source !== 'broker') return notification- if (notification.method.startsWith('lmstudio-proxy:')) {- return {- source: 'lmstudio-proxy',- method: notification.method.slice('lmstudio-proxy:'.length),- params: notification.params- }- }- if (notification.method.startsWith('proxy:')) {- return {- source: 'proxy',- method: notification.method.slice('proxy:'.length),- params: notification.params+ for (const source of PROXY_NODE_SOURCES) {+ const prefix = `${source}:`+ if (notification.method.startsWith(prefix)) {+ return {+ source,+ method: notification.method.slice(prefix.length),+ params: notification.params+ }}}return notificationdiff --git a/desktop/src/shared/constants/modular-binaries.ts b/desktop/src/shared/constants/modular-binaries.tsindex 8f24c8d..cc37210 100644--- a/desktop/src/shared/constants/modular-binaries.ts+++ b/desktop/src/shared/constants/modular-binaries.ts@@ -4,8 +4,7 @@import type { SupportedPlatform } from '@/shared/types/platform'export type ModularProcessName =- | 'proxy'- | 'lmstudio-proxy'+ | 'nvpair-proxy'| 'broker'| 'node-info'| 'scanner'@@ -56,25 +55,23 @@ export const MODULAR_RUNTIME_BINARIES: ModularRuntimeBinary[] = [// Electron only passes their resolved paths to the broker (see// `brokerStartupArgs` in modular-supervisor.ts) and never spawns them.{- processName: 'proxy',- baseName: 'ollama-proxy',+ // One reverse-proxy process fronts every engine: the broker starts it+ // once and enables a facade per engine over `facade/enable`, so this+ // single entry is the whole proxy tier. Each facade binds its HTTP+ // listener on all interfaces, so it needs firewall access to be+ // reachable.+ //+ // Named for the binary, which is also the process: there is one, and+ // naming it after an engine would claim a per-engine process that does+ // not exist.+ // The per-engine identities are the relay sources (`ollama-proxy` /+ // `lmstudio-proxy`), which live in modular-state.ts.+ processName: 'nvpair-proxy',+ baseName: 'nvpair-proxy',args: [],launchOwner: 'broker',needsFirewallAccess: true},- {- // LM Studio reverse proxy — the LM Studio counterpart of `ollama-proxy`,- // supervised the same way and relayed under the `lmstudio-proxy:`- // namespace (broker 0.18.0, `--lmstudio-proxy-path`). Like `ollama-proxy`- // it binds its HTTP listener on all interfaces, so it needs firewall- // access to be reachable.- processName: 'lmstudio-proxy',- baseName: 'lmstudio-proxy',- args: [],- launchOwner: 'broker',- needsFirewallAccess: true,- optional: true- },{processName: 'scanner',baseName: 'nvpair-node-scanner',diff --git a/desktop/src/shared/constants/modular-runtime.ts b/desktop/src/shared/constants/modular-runtime.tsindex 9b833e9..2fdaefd 100644--- a/desktop/src/shared/constants/modular-runtime.ts+++ b/desktop/src/shared/constants/modular-runtime.ts@@ -1,11 +1,11 @@// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.// SPDX-License-Identifier: Apache-2.0-// The proxy port is never assumed: the broker owns ollama-proxy and reports its-// bound listener via `proxy:ready` / `proxy:get-status`. Until it does, the-// proxy port is unknown (surfaced as null) — we do not fabricate a default,-// since a wrong port misleads the user and can route requests to the wrong-// place.+// The proxy port is never assumed: the broker owns the engine proxies and+// reports each bound listener via that engine's `<engine>-proxy:ready` /+// `<engine>-proxy:get-status` pair. Until it does, the proxy port is unknown+// (surfaced as null) — we do not fabricate a default, since a wrong port+// misleads the user and can route requests to the wrong place.// nvpair-cluster-manager pairing (EAP-NOOB) HTTP port. `cluster:invite-node` and// the PIN handshake are driven against this port. Hardcoded pending backenddiff --git a/desktop/src/shared/types/inference-demo.ts b/desktop/src/shared/types/inference-demo.tsindex 246addf..4af2f35 100644--- a/desktop/src/shared/types/inference-demo.ts+++ b/desktop/src/shared/types/inference-demo.ts@@ -38,13 +38,13 @@ export const DEMO_MAX_SUBMIT_SECONDS = 60export const DEMO_REQUEST_TIMEOUT_SECONDS = 120/**- * Engines the demo can drive, paired with the proxy each one sits behind.+ * Engines the demo can drive, paired with the proxy facade each one sits behind.*- * Ports are deliberately absent. The broker owns `ollama-proxy` and- * `lmstudio-proxy` and reports their bound listeners; PAIR never fabricates a- * port (see the note at the top of `@/shared/constants/modular-runtime`). The- * demo resolves each port at start via `getProxyPort()` and skips any engine- * whose proxy has not reported.+ * Ports are deliberately absent. The broker owns one `nvpair-proxy` process+ * hosting a facade per engine, and reports each facade's bound listener; PAIR+ * never fabricates a port (see the note at the top of+ * `@/shared/constants/modular-runtime`). The demo resolves each port at start+ * via `getProxyPort()` and skips any engine whose facade has not reported.** Targeting a proxy rather than the engine itself is what makes the demo a* demo: requests enter PAIR's router and are placed by it. Hitting an engine'sdiff --git a/desktop/tests/modular/modular-supervisor-readiness.test.ts b/desktop/tests/modular/modular-supervisor-readiness.test.tsindex a476be9..931562b 100644--- a/desktop/tests/modular/modular-supervisor-readiness.test.ts+++ b/desktop/tests/modular/modular-supervisor-readiness.test.ts@@ -54,7 +54,8 @@ vi.mock('@/electron/service-bridge/modular-state', () => ({isUpstreamUnreachableError: () => false,parseServiceErrors: () => [],parseWorkloadsInitial: () => [],- PROXY_ENGINES: ['ollama', 'lm-studio']+ PROXY_ENGINES: ['ollama', 'lm-studio'],+ PROXY_NODE_SOURCES: ['ollama-proxy', 'lmstudio-proxy']}))import {@@ -119,7 +120,7 @@ describe('modular supervisor readiness', () => {settled = true})- notify('proxy:ready', { port: 11434 })+ notify('ollama-proxy:ready', { port: 11434 })await Promise.resolve()expect(supervisor.ready).toBe(false)@@ -142,13 +143,13 @@ describe('modular supervisor readiness', () => {mocks.emitBridgePush.mockClear()supervisor.brokerHydrationDone = true- notify('proxy:error', { message: 'address already in use' })- notify('proxy:ready', { port: 11435 })+ notify('ollama-proxy:error', { message: 'address already in use' })+ notify('ollama-proxy:ready', { port: 11435 })expect(supervisor.ready).toBe(true)expect(onReady).toHaveBeenCalledOnce()expect(mocks.bridgeState.handleNotification).toHaveBeenLastCalledWith({- source: 'proxy',+ source: 'ollama-proxy',method: 'ready',params: { port: 11435 }})diff --git a/desktop/tests/modular/node-address-rerank.test.ts b/desktop/tests/modular/node-address-rerank.test.tsindex 454df5e..8e61610 100644--- a/desktop/tests/modular/node-address-rerank.test.ts+++ b/desktop/tests/modular/node-address-rerank.test.ts@@ -33,7 +33,7 @@ describe('broker address re-rank', () => {}})state.handleNotification({- source: 'proxy',+ source: 'ollama-proxy',method: 'node/discovered',params: {id: 'uuid-rerank',diff --git a/desktop/tests/modular/uuid-node-keying.test.ts b/desktop/tests/modular/uuid-node-keying.test.tsindex 5632b53..7009297 100644--- a/desktop/tests/modular/uuid-node-keying.test.ts+++ b/desktop/tests/modular/uuid-node-keying.test.ts@@ -36,7 +36,7 @@ describe('UUID node keying', () => {}})state.handleNotification({- source: 'proxy',+ source: 'ollama-proxy',method: 'node/discovered',params: {id: 'uuid-merge-1',diff --git a/docs/overview.mdx b/docs/overview.mdxindex 95ea262..d7a9d4b 100644--- a/docs/overview.mdx+++ b/docs/overview.mdx@@ -163,10 +163,11 @@ It stages the generated binaries in `desktop/cli-bin/`.### `services/`-The Go tree contains 13 build outputs:+The Go tree contains 12 build outputs:- `nvpair-ui-broker`: service entry point, worker supervisor, and JSON-RPC API-- `ollama-proxy` and `lmstudio-proxy`: compatible inference proxies+- `nvpair-proxy`: the compatible inference proxy — one process hosting a facade+ for each enabled engine- `nvpair-node-scanner` and `nvpair-node-info`: discovery and host telemetry- `nvpair-manual-nodes`: user-specified nodes- `nvpair-engine-manager`: local engine and model lifecyclediff --git a/scripts/collectlogs/testdata/sample-bundle.txt b/scripts/collectlogs/testdata/sample-bundle.txtindex fcdbb47..e5d5385 100644--- a/scripts/collectlogs/testdata/sample-bundle.txt+++ b/scripts/collectlogs/testdata/sample-bundle.txt@@ -41,7 +41,7 @@{"level":"verbose","time":"2026-07-21T22:10:43.304Z","source":"service-bridge","sublevel":"broker","message":"15:10:43.304 [nvpair-node-settings] INFO settings loaded path=\"C:\\\\Users\\\\testuser\\\\AppData\\\\Local\\\\Nvidia Corporation\\\\Personal AI Router\\\\settings.json\" force_ports=true","data":{"stream":"stderr"}}{"level":"warn","time":"2026-07-21T22:10:43.305Z","source":"service-bridge","sublevel":"broker","message":"15:10:43.304 [nvpair-node-settings] INFO settings loaded path=\"C:\\\\Users\\\\testuser\\\\AppData\\\\Local\\\\Nvidia Corporation\\\\Personal AI Router\\\\settings.json\" force_ports=true"}{"level":"verbose","time":"2026-07-21T22:11:30.322Z","source":"service-bridge","sublevel":"broker","message":"15:11:30.322 [nvpair-errors] INFO http: TLS handshake error from 192.168.50.11:62152: client sent an HTTP request to an HTTPS server","data":{"stream":"stderr"}}-{"level":"info","time":"2026-07-21T22:11:31.000Z","source":"service-bridge","sublevel":"proxy","message":"proxy listening","data":{"addr":"127.0.0.1","port":11434}}+{"level":"info","time":"2026-07-21T22:11:31.000Z","source":"service-bridge","sublevel":"ollama-proxy","message":"proxy listening","data":{"addr":"127.0.0.1","port":11434}}{"level":"info","time":"2026-07-21T22:11:32.000Z","source":"service-bridge","sublevel":"cluster","message":"peer certificate","data":{"fingerprint":"AA:BB:CC:DD:EE:FF:00:11:22:33:44:55:66:77:88:99","mac":"aa:bb:cc:dd:ee:ff"}}{"level":"info","time":"2026-07-21T22:11:35.000Z","source":"service-bridge","sublevel":"lifecycle","message":"spawning worker","data":{"binDir":"C:\\Program Files\\Seth Custom PAIR\\resources\\cli-bin","binaryPath":"C:\\Program Files\\Seth Custom PAIR\\resources\\cli-bin\\nvpair-ui-broker.exe"}}{"level":"verbose","time":"2026-07-21T22:11:36.000Z","source":"service-bridge","sublevel":"broker","message":"15:11:36.000 [nvpair-ui-broker] INFO launching worker path=\"C:\\\\Program Files\\\\Seth Custom PAIR\\\\resources\\\\cli-bin\\\\ollama-proxy.exe\"","data":{"stream":"stderr"}}diff --git a/scripts/inference-dispatcher/config.go b/scripts/inference-dispatcher/config.goindex 427d8ce..2598dcb 100644--- a/scripts/inference-dispatcher/config.go+++ b/scripts/inference-dispatcher/config.go@@ -18,8 +18,8 @@ const (defaultHost = "127.0.0.1"defaultBackend = "ollama"defaultOllamaPort = 11434- // LM Studio's own default, which is also the port PAIR's lmstudio-proxy- // facade claims (services/lmstudio-proxy/portstore.go defaultProxyPort).+ // LM Studio's own default, which is also the port PAIR's LM Studio proxy+ // facade claims (the FacadePort in services/shared/engines).// PAIR's managed LM Studio backend is moved behind it starting at 1235;// pass --port explicitly to reach that directly, which bypasses routing.defaultLMStudioPort = 1234diff --git a/scripts/wipe-app-data.ps1 b/scripts/wipe-app-data.ps1index c6cfc5b..4f2caeb 100644--- a/scripts/wipe-app-data.ps1+++ b/scripts/wipe-app-data.ps1@@ -134,6 +134,8 @@ $Targets = @($PairProcs = @('PAIR','nvpair-ui-broker',+ 'nvpair-proxy',+ # Pre-unification names, so this still cleans up after an older install.'ollama-proxy','lmstudio-proxy','nvpair-node-info',diff --git a/scripts/wipe-app-data.sh b/scripts/wipe-app-data.shindex 335415d..659393d 100755--- a/scripts/wipe-app-data.sh+++ b/scripts/wipe-app-data.sh@@ -146,6 +146,8 @@ fiPAIR_PROCS=(PAIRnvpair-ui-broker+ nvpair-proxy+ # Pre-unification names, so this still cleans up after an older install.ollama-proxylmstudio-proxynvpair-node-infodiff --git a/services/.gitignore b/services/.gitignoreindex 4ecef15..24aade8 100644--- a/services/.gitignore+++ b/services/.gitignore@@ -10,8 +10,7 @@ dist/# Linux/macOS subprocess build artifacts. Explicit per-component patterns# instead of a broader glob so we never accidentally ignore source files that# happen to share a name with a component directory.-ollama-proxy/ollama-proxy-lmstudio-proxy/lmstudio-proxy+nvpair-proxy/nvpair-proxynvpair-node-info/nvpair-node-infonvpair-node-scanner/nvpair-node-scannernvpair-manual-nodes/nvpair-manual-nodesdiff --git a/services/THIRD_PARTY_NOTICES.md b/services/THIRD_PARTY_NOTICES.mdindex 8f65d07..35a75b4 100644--- a/services/THIRD_PARTY_NOTICES.md+++ b/services/THIRD_PARTY_NOTICES.md@@ -2,7 +2,7 @@The NVIDIA Personal AI Router is distributed under the Apache License 2.0 (see `LICENSE`). It incorporates the third-party open-source software listed below. Each component is the property of its respective copyright holders and is distributed under its own license; the full license text for each component is reproduced in this file.-Scope: dependencies linked into the thirteen shipped binaries (`ollama-proxy`, `lmstudio-proxy`, `nvpair-node-info`, `nvpair-node-scanner`, `nvpair-manual-nodes`, `nvpair-workload-manager`, `nvpair-errors`, `nvpair-node-settings`, `nvpair-ui-broker`, `nvpair-engine-manager`, `nvpair-cluster-manager`, `nvpair-job-scheduler`, `nvpair-tui`) across Windows, Linux, and macOS builds. First-party modules (`nvpair-shared`, `eapnoob`) are excluded.+Scope: dependencies linked into the twelve shipped binaries (`nvpair-proxy`, `nvpair-node-info`, `nvpair-node-scanner`, `nvpair-manual-nodes`, `nvpair-workload-manager`, `nvpair-errors`, `nvpair-node-settings`, `nvpair-ui-broker`, `nvpair-engine-manager`, `nvpair-cluster-manager`, `nvpair-job-scheduler`, `nvpair-tui`) across Windows, Linux, and macOS builds. First-party modules (`nvpair-shared`, `eapnoob`) are excluded.## Componentsdiff --git a/services/VERSIONING.md b/services/VERSIONING.mdindex 915d73f..446bf4c 100644--- a/services/VERSIONING.md+++ b/services/VERSIONING.md@@ -65,11 +65,11 @@ see the version decision without inferring it from the diff.Every binary supports `--version`:```powershell-.\build\bin\ollama-proxy.exe --version+.\build\bin\nvpair-proxy.exe --version``````bash-./build/bin/ollama-proxy --version+./build/bin/nvpair-proxy --version```A binary built outside `build.bat` / `build.sh` reports `dev` by design.diff --git a/services/bom.md b/services/bom.mdindex 63497a3..db5a945 100644--- a/services/bom.md+++ b/services/bom.md@@ -5,7 +5,7 @@ SPDX-License-Identifier: Apache-2.0# Bill of Materials — Third-Party Go Libraries-Scope: dependencies linked into the thirteen shipped binaries (`ollama-proxy`, `lmstudio-proxy`, `nvpair-node-info`, `nvpair-node-scanner`, `nvpair-manual-nodes`, `nvpair-workload-manager`, `nvpair-errors`, `nvpair-node-settings`, `nvpair-cluster-manager`, `nvpair-ui-broker`, `nvpair-engine-manager`, `nvpair-job-scheduler`, `nvpair-tui`). The local modules `nvpair-shared` and `eapnoob` (the EAP-NOOB implementation under `eap-noob/`, linked by `nvpair-cluster-manager`) are first-party and excluded. The `tests/`, `mdns-test/`, and `broker-test-driver/` modules are development-only and excluded.+Scope: dependencies linked into the twelve shipped binaries (`nvpair-proxy`, `nvpair-node-info`, `nvpair-node-scanner`, `nvpair-manual-nodes`, `nvpair-workload-manager`, `nvpair-errors`, `nvpair-node-settings`, `nvpair-cluster-manager`, `nvpair-ui-broker`, `nvpair-engine-manager`, `nvpair-job-scheduler`, `nvpair-tui`). The local modules `nvpair-shared` and `eapnoob` (the EAP-NOOB implementation under `eap-noob/`, linked by `nvpair-cluster-manager`) are first-party and excluded. The `tests/`, `mdns-test/`, and `broker-test-driver/` modules are development-only and excluded.`nvpair-tui` is the only component that links the Bubble Tea terminal-UI stack (`charmbracelet/bubbletea` + `lipgloss` + `bubbles`); its transitive `charmbracelet/*`, `muesli/*`, `mattn/*`, `clipperhouse/*`, `atotto/clipboard`, `aymanbagabas/go-osc52`, `lucasb-eyer/go-colorful`, `erikgeiser/coninput`, and `xo/terminfo` dependencies are unique to it.@@ -21,15 +21,15 @@ As of the mDNS dedup, `grandcat/zeroconf`, `miekg/dns`, and `golang.org/x/net` a| Library | Version | Used By | License | License URL ||---------|---------|---------|---------|-------------|-| `github.com/Microsoft/go-winio` | v0.6.2 | ollama-proxy, lmstudio-proxy, nvpair-node-scanner, nvpair-manual-nodes, nvpair-node-settings, nvpair-cluster-manager, nvpair-ui-broker, nvpair-workload-manager, nvpair-errors, nvpair-engine-manager, nvpair-job-scheduler | MIT | [LICENSE](https://github.com/microsoft/go-winio/blob/main/LICENSE) |+| `github.com/Microsoft/go-winio` | v0.6.2 | nvpair-proxy, nvpair-node-scanner, nvpair-manual-nodes, nvpair-node-settings, nvpair-cluster-manager, nvpair-ui-broker, nvpair-workload-manager, nvpair-errors, nvpair-engine-manager, nvpair-job-scheduler | MIT | [LICENSE](https://github.com/microsoft/go-winio/blob/main/LICENSE) || `github.com/charmbracelet/bubbles` | v1.0.0 | nvpair-tui | MIT | [LICENSE](https://github.com/charmbracelet/bubbles/blob/master/LICENSE) || `github.com/charmbracelet/bubbletea` | v1.3.10 | nvpair-tui | MIT | [LICENSE](https://github.com/charmbracelet/bubbletea/blob/master/LICENSE) || `github.com/charmbracelet/lipgloss` | v1.1.0 | nvpair-tui | MIT | [LICENSE](https://github.com/charmbracelet/lipgloss/blob/master/LICENSE) |-| `github.com/grandcat/zeroconf` | v1.0.0 | ollama-proxy, lmstudio-proxy, nvpair-node-scanner, nvpair-errors, nvpair-cluster-manager | MIT | [LICENSE](https://github.com/grandcat/zeroconf/blob/master/LICENSE) |+| `github.com/grandcat/zeroconf` | v1.0.0 | nvpair-proxy, nvpair-node-scanner, nvpair-errors, nvpair-cluster-manager | MIT | [LICENSE](https://github.com/grandcat/zeroconf/blob/master/LICENSE) || `github.com/jaypipes/ghw` | v0.24.0 | nvpair-node-info | Apache-2.0 | [COPYING](https://github.com/jaypipes/ghw/blob/main/COPYING) |-| `github.com/miekg/dns` | v1.1.55 / v1.1.72 | ollama-proxy, lmstudio-proxy, nvpair-node-scanner, nvpair-errors, nvpair-cluster-manager | BSD-3-Clause | [LICENSE](https://github.com/miekg/dns/blob/master/LICENSE) |+| `github.com/miekg/dns` | v1.1.55 / v1.1.72 | nvpair-proxy, nvpair-node-scanner, nvpair-errors, nvpair-cluster-manager | BSD-3-Clause | [LICENSE](https://github.com/miekg/dns/blob/master/LICENSE) || `github.com/shirou/gopsutil/v4` | v4.26.7 | nvpair-node-info (macOS) | BSD-3-Clause | [LICENSE](https://github.com/shirou/gopsutil/blob/master/LICENSE) |-| `golang.org/x/net` | v0.58.0 | ollama-proxy, lmstudio-proxy, nvpair-node-scanner, nvpair-errors, nvpair-cluster-manager | BSD-3-Clause | [LICENSE](https://cs.opensource.google/go/x/net/+/master:LICENSE) |+| `golang.org/x/net` | v0.58.0 | nvpair-proxy, nvpair-node-scanner, nvpair-errors, nvpair-cluster-manager | BSD-3-Clause | [LICENSE](https://cs.opensource.google/go/x/net/+/master:LICENSE) || `golang.org/x/sys` | v0.47.0 | nvpair-node-info, nvpair-cluster-manager, nvpair-engine-manager | BSD-3-Clause | [LICENSE](https://cs.opensource.google/go/x/sys/+/master:LICENSE) || `howett.net/plist` | v1.0.2-0.20250314 | nvpair-node-info | BSD-2-Clause | [LICENSE](https://github.com/DHowett/go-plist/blob/main/LICENSE) |diff --git a/services/build.bat b/services/build.batindex 6311e42..ee5e3a0 100644--- a/services/build.bat+++ b/services/build.bat@@ -38,8 +38,7 @@ REM a string variable, which sidesteps cmd's hostility toward embeddedREM double quotes inside the jq filter (component keys contain hyphens, soREM bare .components.nvpair-ui-broker would parse as subtraction).for /f "delims=" %%V in ('jq -r ".product" "%VERSIONS_FILE%"') do set "V_PRODUCT=%%V"-for /f "delims=" %%V in ('jq -r --arg k "ollama-proxy" ".components[$k]" "%VERSIONS_FILE%"') do set "V_PROXY=%%V"-for /f "delims=" %%V in ('jq -r --arg k "lmstudio-proxy" ".components[$k]" "%VERSIONS_FILE%"') do set "V_LMPROXY=%%V"+for /f "delims=" %%V in ('jq -r --arg k "nvpair-proxy" ".components[$k]" "%VERSIONS_FILE%"') do set "V_PROXY=%%V"for /f "delims=" %%V in ('jq -r --arg k "nvpair-node-info" ".components[$k]" "%VERSIONS_FILE%"') do set "V_NINFO=%%V"for /f "delims=" %%V in ('jq -r --arg k "nvpair-node-scanner" ".components[$k]" "%VERSIONS_FILE%"') do set "V_NSCAN=%%V"for /f "delims=" %%V in ('jq -r --arg k "nvpair-manual-nodes" ".components[$k]" "%VERSIONS_FILE%"') do set "V_MNODES=%%V"@@ -59,8 +58,7 @@ if "%V_PRODUCT%"=="" ()echo product = %V_PRODUCT%-echo ollama-proxy = %V_PROXY%-echo lmstudio-proxy = %V_LMPROXY%+echo nvpair-proxy = %V_PROXY%echo nvpair-node-info = %V_NINFO%echo nvpair-node-scanner = %V_NSCAN%echo nvpair-manual-nodes = %V_MNODES%@@ -79,67 +77,62 @@ echo Building all componentsecho ========================================echo.-echo [1/13] Building ollama-proxy (v%V_PROXY%)...-cd /d "%ROOT%ollama-proxy"-go build -ldflags "-X main.Version=%V_PROXY%" -o ollama-proxy.exe . || goto :fail+echo [1/12] Building nvpair-proxy (v%V_PROXY%)...+cd /d "%ROOT%nvpair-proxy"+go build -ldflags "-X main.Version=%V_PROXY%" -o nvpair-proxy.exe . || goto :failecho OK-echo [2/13] Building lmstudio-proxy (v%V_LMPROXY%)...-cd /d "%ROOT%lmstudio-proxy"-go build -ldflags "-X main.Version=%V_LMPROXY%" -o lmstudio-proxy.exe . || goto :fail-echo OK--echo [3/13] Building nvpair-node-info (v%V_NINFO%)...+echo [2/12] Building nvpair-node-info (v%V_NINFO%)...cd /d "%ROOT%nvpair-node-info"go build -ldflags "-X main.Version=%V_NINFO%" -o nvpair-node-info.exe . || goto :failecho OK-echo [4/13] Building nvpair-node-scanner (v%V_NSCAN%)...+echo [3/12] Building nvpair-node-scanner (v%V_NSCAN%)...cd /d "%ROOT%nvpair-node-scanner"go build -ldflags "-X main.Version=%V_NSCAN%" -o nvpair-node-scanner.exe . || goto :failecho OK-echo [5/13] Building nvpair-manual-nodes (v%V_MNODES%)...+echo [4/12] Building nvpair-manual-nodes (v%V_MNODES%)...cd /d "%ROOT%nvpair-manual-nodes"go build -ldflags "-X main.Version=%V_MNODES%" -o nvpair-manual-nodes.exe . || goto :failecho OK-echo [6/13] Building nvpair-workload-manager (v%V_WLMGR%)...+echo [5/12] Building nvpair-workload-manager (v%V_WLMGR%)...cd /d "%ROOT%nvpair-workload-manager"go build -ldflags "-X main.Version=%V_WLMGR%" -o nvpair-workload-manager.exe . || goto :failecho OK-echo [7/13] Building nvpair-errors (v%V_ERRORS%)...+echo [6/12] Building nvpair-errors (v%V_ERRORS%)...cd /d "%ROOT%nvpair-errors"go build -ldflags "-X main.Version=%V_ERRORS%" -o nvpair-errors.exe . || goto :failecho OK-echo [8/13] Building nvpair-engine-manager (v%V_ENGMGR%)...+echo [7/12] Building nvpair-engine-manager (v%V_ENGMGR%)...cd /d "%ROOT%nvpair-engine-manager"go build -ldflags "-X main.Version=%V_ENGMGR%" -o nvpair-engine-manager.exe . || goto :failecho OK-echo [9/13] Building nvpair-node-settings (v%V_NSETTINGS%)...+echo [8/12] Building nvpair-node-settings (v%V_NSETTINGS%)...cd /d "%ROOT%nvpair-node-settings"go build -ldflags "-X main.Version=%V_NSETTINGS%" -o nvpair-node-settings.exe . || goto :failecho OK-echo [10/13] Building nvpair-ui-broker (v%V_BROKER%)...+echo [9/12] Building nvpair-ui-broker (v%V_BROKER%)...cd /d "%ROOT%nvpair-ui-broker"go build -ldflags "-X main.Version=%V_BROKER%" -o nvpair-ui-broker.exe . || goto :failecho OK-echo [11/13] Building nvpair-cluster-manager (v%V_CLUMGR%)...+echo [10/12] Building nvpair-cluster-manager (v%V_CLUMGR%)...cd /d "%ROOT%nvpair-cluster-manager"go build -ldflags "-X main.Version=%V_CLUMGR%" -o nvpair-cluster-manager.exe . || goto :failecho OK-echo [12/13] Building nvpair-job-scheduler (v%V_SCHED%)...+echo [11/12] Building nvpair-job-scheduler (v%V_SCHED%)...cd /d "%ROOT%nvpair-job-scheduler"go build -ldflags "-X main.Version=%V_SCHED%" -o nvpair-job-scheduler.exe . || goto :failecho OK-echo [13/13] Building nvpair-tui (v%V_TUI%)...+echo [12/12] Building nvpair-tui (v%V_TUI%)...cd /d "%ROOT%nvpair-tui"go build -ldflags "-X main.Version=%V_TUI%" -o nvpair-tui.exe . || goto :failecho OK@@ -155,8 +148,7 @@ REM that doesn't exist at this commit) can't linger and ride into the installerREM as a stale stray.if exist "%BIN_OUT%" rmdir /s /q "%BIN_OUT%"mkdir "%BIN_OUT%"-copy /y "%ROOT%ollama-proxy\ollama-proxy.exe" "%BIN_OUT%\ollama-proxy.exe" >nul || goto :fail-copy /y "%ROOT%lmstudio-proxy\lmstudio-proxy.exe" "%BIN_OUT%\lmstudio-proxy.exe" >nul || goto :fail+copy /y "%ROOT%nvpair-proxy\nvpair-proxy.exe" "%BIN_OUT%\nvpair-proxy.exe" >nul || goto :failcopy /y "%ROOT%nvpair-node-info\nvpair-node-info.exe" "%BIN_OUT%\nvpair-node-info.exe" >nul || goto :failcopy /y "%ROOT%nvpair-node-scanner\nvpair-node-scanner.exe" "%BIN_OUT%\nvpair-node-scanner.exe" >nul || goto :failcopy /y "%ROOT%nvpair-manual-nodes\nvpair-manual-nodes.exe" "%BIN_OUT%\nvpair-manual-nodes.exe" >nul || goto :fail@@ -174,8 +166,7 @@ echo ========================================echo Build complete (product v%V_PRODUCT%)echo ========================================echo.-echo Proxy: %BIN_OUT%\ollama-proxy.exe-echo LM Studio Proxy: %BIN_OUT%\lmstudio-proxy.exe+echo Proxy: %BIN_OUT%\nvpair-proxy.exeecho Node Info: %BIN_OUT%\nvpair-node-info.exeecho Node Scanner: %BIN_OUT%\nvpair-node-scanner.exeecho Manual Nodes: %BIN_OUT%\nvpair-manual-nodes.exediff --git a/services/build.sh b/services/build.shindex feadff1..ef314ec 100755--- a/services/build.sh+++ b/services/build.sh@@ -4,7 +4,7 @@# build.sh — NVIDIA Personal AI Router build script for Linux and macOS.#-# Mirrors build.bat. Reads versions.json with jq, builds the thirteen worker+# Mirrors build.bat. Reads versions.json with jq, builds the twelve worker# binaries with -X main.Version=... ldflags, then copies them into the# repo-root staging bundle at:#@@ -55,8 +55,7 @@ echo# bare-identifier syntax would parse as subtraction. Passing the key as a# string variable sidesteps the ambiguity.V_PRODUCT=$(jq -r '.product' "$VERSIONS_FILE")-V_PROXY=$( jq -r --arg k 'ollama-proxy' '.components[$k]' "$VERSIONS_FILE")-V_LMPROXY=$(jq -r --arg k 'lmstudio-proxy' '.components[$k]' "$VERSIONS_FILE")+V_PROXY=$( jq -r --arg k 'nvpair-proxy' '.components[$k]' "$VERSIONS_FILE")V_NINFO=$( jq -r --arg k 'nvpair-node-info' '.components[$k]' "$VERSIONS_FILE")V_NSCAN=$( jq -r --arg k 'nvpair-node-scanner' '.components[$k]' "$VERSIONS_FILE")V_MNODES=$( jq -r --arg k 'nvpair-manual-nodes' '.components[$k]' "$VERSIONS_FILE")@@ -75,8 +74,7 @@ if [[ -z "$V_PRODUCT" || "$V_PRODUCT" == "null" ]]; thenfiprintf ' product = %s\n' "$V_PRODUCT"-printf ' ollama-proxy = %s\n' "$V_PROXY"-printf ' lmstudio-proxy = %s\n' "$V_LMPROXY"+printf ' nvpair-proxy = %s\n' "$V_PROXY"printf ' nvpair-node-info = %s\n' "$V_NINFO"printf ' nvpair-node-scanner = %s\n' "$V_NSCAN"printf ' nvpair-manual-nodes = %s\n' "$V_MNODES"@@ -97,23 +95,22 @@ echobuild_subbinary() {local idx="$1" name="$2" version="$3"- echo "[$idx/13] Building $name (v$version)..."+ echo "[$idx/12] Building $name (v$version)..."(cd "$ROOT/$name" && go build -ldflags "-X main.Version=$version" -o "$name" .)echo " OK"}-build_subbinary 1 ollama-proxy "$V_PROXY"-build_subbinary 2 lmstudio-proxy "$V_LMPROXY"-build_subbinary 3 nvpair-node-info "$V_NINFO"-build_subbinary 4 nvpair-node-scanner "$V_NSCAN"-build_subbinary 5 nvpair-manual-nodes "$V_MNODES"-build_subbinary 6 nvpair-workload-manager "$V_WLMGR"-build_subbinary 7 nvpair-errors "$V_ERRORS"-build_subbinary 8 nvpair-engine-manager "$V_ENGMGR"-build_subbinary 9 nvpair-node-settings "$V_NSETTINGS"-build_subbinary 10 nvpair-ui-broker "$V_BROKER"-build_subbinary 11 nvpair-cluster-manager "$V_CLUMGR"-build_subbinary 12 nvpair-job-scheduler "$V_SCHED"-build_subbinary 13 nvpair-tui "$V_TUI"+build_subbinary 1 nvpair-proxy "$V_PROXY"+build_subbinary 2 nvpair-node-info "$V_NINFO"+build_subbinary 3 nvpair-node-scanner "$V_NSCAN"+build_subbinary 4 nvpair-manual-nodes "$V_MNODES"+build_subbinary 5 nvpair-workload-manager "$V_WLMGR"+build_subbinary 6 nvpair-errors "$V_ERRORS"+build_subbinary 7 nvpair-engine-manager "$V_ENGMGR"+build_subbinary 8 nvpair-node-settings "$V_NSETTINGS"+build_subbinary 9 nvpair-ui-broker "$V_BROKER"+build_subbinary 10 nvpair-cluster-manager "$V_CLUMGR"+build_subbinary 11 nvpair-job-scheduler "$V_SCHED"+build_subbinary 12 nvpair-tui "$V_TUI"BIN_OUT="$ROOT/build/bin"@@ -130,8 +127,7 @@ echo# components.rm -rf "$BIN_OUT"mkdir -p "$BIN_OUT"-cp "$ROOT/ollama-proxy/ollama-proxy" "$BIN_OUT/ollama-proxy"-cp "$ROOT/lmstudio-proxy/lmstudio-proxy" "$BIN_OUT/lmstudio-proxy"+cp "$ROOT/nvpair-proxy/nvpair-proxy" "$BIN_OUT/nvpair-proxy"cp "$ROOT/nvpair-node-info/nvpair-node-info" "$BIN_OUT/nvpair-node-info"cp "$ROOT/nvpair-node-scanner/nvpair-node-scanner" "$BIN_OUT/nvpair-node-scanner"cp "$ROOT/nvpair-manual-nodes/nvpair-manual-nodes" "$BIN_OUT/nvpair-manual-nodes"@@ -149,8 +145,7 @@ echo "========================================"echo " Build complete (product v$V_PRODUCT)"echo "========================================"echo-printf ' Proxy: %s\n' "$BIN_OUT/ollama-proxy"-printf ' LM Studio Proxy: %s\n' "$BIN_OUT/lmstudio-proxy"+printf ' Proxy: %s\n' "$BIN_OUT/nvpair-proxy"printf ' Node Info: %s\n' "$BIN_OUT/nvpair-node-info"printf ' Node Scanner: %s\n' "$BIN_OUT/nvpair-node-scanner"printf ' Manual Nodes: %s\n' "$BIN_OUT/nvpair-manual-nodes"diff --git a/services/installer/linux/INSTALL.md b/services/installer/linux/INSTALL.mdindex 7e50420..a5f414a 100644--- a/services/installer/linux/INSTALL.md+++ b/services/installer/linux/INSTALL.md@@ -23,8 +23,7 @@ You should end up with this layout:NVIDIA-Personal-AI-Router-<version>/├── bin/│ ├── nvpair-ui-broker # primary entry point (JSON-RPC over stdio / IPC)-│ ├── ollama-proxy-│ ├── lmstudio-proxy+│ ├── nvpair-proxy│ ├── nvpair-node-info│ ├── nvpair-node-scanner│ ├── nvpair-manual-nodesdiff --git a/services/installer/nvpair-setup.nsi b/services/installer/nvpair-setup.nsiindex eaeb5a0..474f524 100644--- a/services/installer/nvpair-setup.nsi+++ b/services/installer/nvpair-setup.nsi@@ -107,6 +107,9 @@ FunctionEnd;---------------------------------------!macro CloseRunningInstanceDetailPrint "Checking for running ${PRODUCT_NAME} processes..."+ nsExec::ExecToLog 'taskkill /F /IM "nvpair-proxy.exe"'+ ; Pre-unification names. An orphan still holding 11434 or 1234 is exactly+ ; what the managed-facade planner has to block on, so kill it here too.nsExec::ExecToLog 'taskkill /F /IM "ollama-proxy.exe"'nsExec::ExecToLog 'taskkill /F /IM "lmstudio-proxy.exe"'nsExec::ExecToLog 'taskkill /F /IM "nvpair-node-info.exe"'@@ -144,21 +147,44 @@ Section "Install"Delete "$PrevInstallDir\uninstall.exe"skip_prev_uninstall:+ ; Revoke the pre-unification rules unconditionally, not only when a previous+ ; uninstaller was found and ran. The cases where that matters most are exactly+ ; the ones the branch above skips: no recorded UninstallString, or an ExecWait+ ; that failed (nothing checks $R3). On those paths the old rules survive and+ ; pre-approve inbound traffic to whatever later lands at the old binary paths.+ ;+ ; "NVPAIR mDNS (UDP 5353)" is included even though the old uninstaller did+ ; delete it: on the skip paths it never ran, and this installer never+ ; re-creates it because nvpair-proxy opens no UDP socket. The leftover+ ; executables go with the rules for the same reason — a stale allow entry+ ; plus the binary it names is the combination worth removing.+ ;+ ; None of these names is re-added by this installer, and deleting an absent+ ; rule is a documented no-op.+ nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR Ollama Proxy"'+ nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR LM Studio Proxy"'+ nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR mDNS LM Studio Proxy (UDP 5353)"'+ nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR mDNS (UDP 5353)"'+ ; $INSTDIR, not $PrevInstallDir: the latter is empty on the no-UninstallString+ ; path this block exists for, and $INSTDIR is where a leftover would sit+ ; alongside the binaries about to be written.+ Delete "$INSTDIR\bin\ollama-proxy.exe"+ Delete "$INSTDIR\bin\lmstudio-proxy.exe"+SetOutPath "$INSTDIR"File "EULA.txt"SetOutPath "$INSTDIR\bin"- ; ollama-proxy's listen port is now a dual-protocol endpoint: loopback- ; plaintext for local clients, and pin-gated cluster mTLS for peers (it+ ; nvpair-proxy fronts every engine from one process, hosting a facade per+ ; enabled engine. Each facade's listen port is a dual-protocol endpoint:+ ; loopback plaintext for local clients, and pin-gated cluster mTLS for peers (it; forwards a validated peer straight to the loopback engine). Plaintext; requests from the LAN are rejected in-process, so only pinned cluster; members can run inference — the firewall rule below still allows the- ; executable because that same user-configurable port carries the mTLS ingress.- File "..\build\bin\ollama-proxy.exe"- ; lmstudio-proxy fronts the cluster's LM Studio engines (OpenAI API) with the- ; same dual-protocol port (loopback plaintext + cluster mTLS ingress); it- ; listens for clients and browses mDNS, so it gets firewall rules below.- File "..\build\bin\lmstudio-proxy.exe"+ ; executable because that same user-configurable port carries the mTLS+ ; ingress. It does not browse mDNS: its routing targets arrive over the+ ; broker's discovery relay, so it opens no UDP socket of its own.+ File "..\build\bin\nvpair-proxy.exe"File "..\build\bin\nvpair-node-info.exe"File "..\build\bin\nvpair-node-scanner.exe"File "..\build\bin\nvpair-manual-nodes.exe"@@ -227,15 +253,17 @@ Section "Install"; nodes"). Scoping to localsubnet keeps the ports closed to anything off the; local link, so covering all profiles does not expose the node on untrusted; public networks.- nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR Ollama Proxy" dir=in action=allow program="$INSTDIR\bin\ollama-proxy.exe" enable=yes profile=any remoteip=localsubnet'- nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR LM Studio Proxy" dir=in action=allow program="$INSTDIR\bin\lmstudio-proxy.exe" enable=yes profile=any remoteip=localsubnet'+ nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR Engine Proxy" dir=in action=allow program="$INSTDIR\bin\nvpair-proxy.exe" enable=yes profile=any remoteip=localsubnet'nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR Node Info" dir=in action=allow program="$INSTDIR\bin\nvpair-node-info.exe" enable=yes profile=any remoteip=localsubnet'nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR Node Scanner" dir=in action=allow program="$INSTDIR\bin\nvpair-node-scanner.exe" enable=yes profile=any remoteip=localsubnet'; mDNS needs UDP 5353 inbound- nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR mDNS (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\bin\ollama-proxy.exe" enable=yes profile=any remoteip=localsubnet'- nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR mDNS LM Studio Proxy (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\bin\lmstudio-proxy.exe" enable=yes profile=any remoteip=localsubnet'+ ; No mDNS rule for nvpair-proxy: it opens no UDP socket. Its routing targets+ ; arrive over the broker's discovery relay, and zeroconf/miekg are indirect+ ; dependencies it never browses with. The pre-unification ollama-proxy did+ ; browse, which is where this rule came from; the delete below stays so an+ ; upgrade clears the inherited entry.nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR mDNS Node Info (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\bin\nvpair-node-info.exe" enable=yes profile=any remoteip=localsubnet'nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR mDNS Node Scanner (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\bin\nvpair-node-scanner.exe" enable=yes profile=any remoteip=localsubnet'nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR mDNS Workload Manager (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\bin\nvpair-workload-manager.exe" enable=yes profile=any remoteip=localsubnet'@@ -271,7 +299,13 @@ Section "Uninstall"!insertmacro CloseRunningInstance; Remove firewall exceptions+ nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR Engine Proxy"'+ ; The per-engine rules a pre-unification install created. Deleting a rule+ ; that does not exist is a harmless no-op, and without these an upgrade+ ; leaves rules pointing at binaries this version no longer ships.nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR Ollama Proxy"'+ nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR LM Studio Proxy"'+ nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR mDNS LM Studio Proxy (UDP 5353)"'nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR Node Info"'nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR mDNS (UDP 5353)"'nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR Node Scanner"'@@ -288,6 +322,8 @@ Section "Uninstall"nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR Engine Manager Control (TCP 14323)"'; Remove files+ Delete "$INSTDIR\bin\nvpair-proxy.exe"+ ; Shipped by a pre-unification install; Delete is a no-op when absent.Delete "$INSTDIR\bin\ollama-proxy.exe"Delete "$INSTDIR\bin\lmstudio-proxy.exe"Delete "$INSTDIR\bin\nvpair-node-info.exe"diff --git a/services/installer_build.sh b/services/installer_build.shindex cba2c44..a6698cf 100755--- a/services/installer_build.sh+++ b/services/installer_build.sh@@ -137,8 +137,7 @@ if [[ ! -x "$BIN_SRC/nvpair-ui-broker" ]]; thenexit 1fi-cp "$BIN_SRC/ollama-proxy" "$STAGE/bin/"-cp "$BIN_SRC/lmstudio-proxy" "$STAGE/bin/"+cp "$BIN_SRC/nvpair-proxy" "$STAGE/bin/"cp "$BIN_SRC/nvpair-node-info" "$STAGE/bin/"cp "$BIN_SRC/nvpair-node-scanner" "$STAGE/bin/"cp "$BIN_SRC/nvpair-manual-nodes" "$STAGE/bin/"diff --git a/services/lmstudio-proxy/README.md b/services/lmstudio-proxy/README.mddeleted file mode 100644index 71a8b70..0000000--- a/services/lmstudio-proxy/README.md+++ /dev/null@@ -1,444 +0,0 @@-<!---SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-SPDX-License-Identifier: Apache-2.0--->--# LM Studio Proxy--A discovery-aware HTTP reverse proxy for LM Studio nodes on the local network. It runs no mDNS browse of its own: its routing targets come from the broker's discovery relay (it sends `discovery:subscribe {services:[lm]}` and replaces its routing overlay from each pushed `discovery:nodes` snapshot) plus user-added manual nodes. It forwards HTTP requests to the selected node, aggregates the model-list route across candidate nodes, and exposes a bidirectional JSON-RPC 2.0 control channel over stdio (or an IPC socket).--> **Clone of `ollama-proxy`.** This proxy is a deliberate clone of [`ollama-proxy`](../ollama-proxy/README.md) so the two share identical routing, failover, CORS, and node-selection behavior — the CORS policy is literally the same code, `nvpair-shared/cors`, and is documented [there](../ollama-proxy/README.md#http-reverse-proxy). The differences are engine-specific: it subscribes to the discovery relay for `lm` nodes, forwards the OpenAI-compatible inference routes (`/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`), tags workloads `lmstudio`, and persists its port to its own file. It has no `--alias-address`, so its self-forward guard covers only its own listener.--## Build--```bash-go build -o lmstudio-proxy .-```--## Usage--```-lmstudio-proxy [flags]-```--### Flags--| Flag | Default | Description |-|------|---------|-------------|-| `--port` | `1234` | HTTP listen port for request forwarding |-| `--ignore-persisted-port` | `false` | Use `--port` even when a prior runtime port was saved |-| `--ipc` | *(empty — use stdio)* | Path to a Unix domain socket or Windows named pipe for IPC |-| `--cluster-dir` | *(empty)* | Cluster trust directory (`node.crt`/`node.key` plus trusted pins). Enables the LAN mTLS inference ingress while this node is a cluster member; empty means no ingress and no peer candidates. |-| `--log-level` | *(`$NVPAIR_LOG_LEVEL`, else `info`)* | Initial log level: `debug`, `info`, `warn`, or `error`. Changeable at runtime with `log/set-level`. |-| `--version` | | Print version and exit |--### HTTP Reverse Proxy--The proxy listens on `--port` (default 1234) and forwards incoming HTTP requests to the currently active LM Studio node — except the model-list route `GET /v1/models`, which is queried across every candidate node concurrently and merged into one de-duplicated inventory. Point your OpenAI-compatible client at `http://localhost:1234` and the proxy handles routing.--**Cluster ingress.** The listener carries two personalities, demultiplexed by each connection's first byte. Plaintext HTTP is accepted only from loopback; a LAN caller is refused. When `--cluster-dir` shows this node is a cluster member, the same listener also terminates cluster mTLS: a peer whose client certificate matches one of this node's pins is forwarded straight to the local engine reported by `node/set-local-backend`, and is never re-routed onward to another node. Membership and pins are re-derived per request, so joining or leaving a cluster needs no restart.--**Persisted port.** A port chosen at runtime via the `set-port` request (see below) is saved as `lmstudio-proxy-port.json` in the per-user data dir (`%LocalAppData%\Nvidia Corporation\Personal AI Router` on Windows, `~/.config/Nvidia Corporation/Personal AI Router` on Linux) and **restored on startup**, taking precedence over `--port`/the default. One value is exempt: a stored `1235` is discarded and `--port` is used instead, so that port cannot be restored even when it was chosen deliberately via `set-port`. Any other stored port is honoured. The broker uses `--ignore-persisted-port` while reserving the managed `1234` facade.--Node selection:-- **Eligibility**: Before routing model-bearing inference, the proxy keeps only nodes whose current LM Studio inventory advertises the exact requested model ID. An empty or non-matching inventory is excluded until a later discovery update; if no advertised owner is routable, the proxy returns a local `502`.-- **Auto**: When no eligible node is explicitly selected, the proxy follows `node/set-priority` (see below), then discovered nodes in stable ID order.-- **Priority (scheduler-driven)**: The Job Scheduler ranks the cluster least-loaded-first by pending workload plus smoothed GPU pressure and, via `nvpair-ui-broker`, pushes the ordered node list with those per-node counts to this proxy with `node/set-priority`. Auto routing sends the request to the listed node carrying the least estimated load. See [`nvpair-job-scheduler`](../nvpair-job-scheduler/README.md).-- **Manual**: Use the `node/select` JSON-RPC method to pin traffic to a specific node. A manual pin **overrides the priority list only when that node is eligible** for the requested model.-- **Failover**: If the selected node disappears from the discovery set, the proxy falls back to auto-select and emits a `node/selection-changed` notification. A transport error or retryable status, including a model `404` from an advertised owner with stale inventory, steps to the next eligible owner.--### IPC Transport--By default the proxy communicates over **stdin/stdout** using newline-delimited JSON-RPC 2.0 (one message per line). All diagnostic logging goes to **stderr**.--For environments where stdout may conflict with the host process (e.g. Electron), pass `--ipc` to redirect the JSON-RPC channel to a named socket or pipe. The parent process should create and listen on the endpoint before spawning the proxy.--```bash-# Default: stdin/stdout-lmstudio-proxy--# Unix domain socket-lmstudio-proxy --ipc /tmp/lmstudio-proxy.sock--# Windows named pipe-lmstudio-proxy --ipc \\.\pipe\lmstudio-proxy-```--## JSON-RPC 2.0 Protocol--All messages conform to the [JSON-RPC 2.0 specification](https://www.jsonrpc.org/specification). Messages are newline-delimited (one JSON object per `\n`).--### Node Object--Nodes are represented throughout the protocol with this shape:--```json-{- "id": "22222222-2222-2222-2222-222222222222",- "host": "my-workstation",- "port": 1234,- "addresses": ["192.168.1.50"],- "txt": ["uuid=22222222-2222-2222-2222-222222222222", "lm=1234"],- "models": ["qwen2.5-7b-instruct"],- "ip": "192.168.1.50"-}-```--| Field | Type | Description |-|-------|------|-------------|-| `id` | string | Stable per-host UUID from the discovery record (the ID you supply, for a manual node) |-| `host` | string | Hostname, for display — routing never keys on it |-| `port` | int | LM Studio port from the discovery record's `lm` service entry |-| `addresses` | string[] | Addresses to dial. A node fed by the discovery relay always carries exactly one canonical address; several only ever appear on a manual node |-| `txt` | string[] | The discovery record's TXT pairs, carried verbatim |-| `models` | string[] | The node's LM Studio model inventory from the discovery snapshot. Model-bearing inference is eligible only when this list advertises the exact requested model ID. An omitted or empty list excludes the node from that request until inventory updates; it remains available for non-inference routes and model-list aggregation |-| `ip` | string | The single canonical LAN address to dial or display, resolved from the node's `ip=` TXT if present and otherwise the best-scored advertised IPv4. Stamped onto outbound `node/*` notifications so consumers agree with the address the proxy routes to |-------### Notifications (proxy → client)--Notifications have no `id` field and do not expect a response.--#### `ready`--Sent after startup, before discovery begins, and again after every successful `set-port` rebind — `port` carries the port now bound.--```json-{"jsonrpc":"2.0","method":"ready","params":{"version":"0.1.0","port":1234}}-```--| Param | Type | Description |-|-------|------|-------------|-| `version` | string | Proxy version |-| `port` | int | HTTP listen port |--#### `error`--Sent when a fatal startup condition stops the proxy from serving — currently only a failed bind — immediately before the process exits non-zero.--```json-{"jsonrpc":"2.0","method":"error","params":{"code":"bind-failed","message":"failed to bind port 1234: ...","port":1234}}-```--#### `node/discovered`--A new LM Studio node appeared on the network.--```json-{"jsonrpc":"2.0","method":"node/discovered","params":{"id":"22222222-2222-2222-2222-222222222222","host":"my-workstation","port":1234,"addresses":["192.168.1.50"]}}-```--#### `node/updated`--A previously discovered node changed its host, port, or addresses.--```json-{"jsonrpc":"2.0","method":"node/updated","params":{"id":"22222222-2222-2222-2222-222222222222","host":"my-workstation","port":1234,"addresses":["192.168.1.51"]}}-```--#### `node/removed`--A node is no longer present in the discovery set (it left the relay's `lm` nodes, or a manual node was removed).--```json-{"jsonrpc":"2.0","method":"node/removed","params":{"id":"22222222-2222-2222-2222-222222222222","host":"my-workstation","port":1234,"addresses":["192.168.1.50"]}}-```--#### `node/selection-changed`--The active node selection changed (either explicitly via `node/select` or because the selected node was removed).--```json-{"jsonrpc":"2.0","method":"node/selection-changed","params":{"id":"22222222-2222-2222-2222-222222222222"}}-```--An empty `id` means the proxy has reverted to auto-select mode.--#### `proxy/request-started`--A request has been committed to a target and its response body is about to stream. It pairs by `id` with the matching `proxy/request`, so a consumer can keep an in-flight count per node. Model-list aggregation has no single target, so it reports `"target":"cluster"` with no `node_id`; a request rejected before forwarding was never in flight and gets no started event.--```json-{"jsonrpc":"2.0","method":"proxy/request-started","params":{"id":"17","node_id":"22222222-2222-2222-2222-222222222222","method":"POST","path":"/v1/chat/completions","target":"192.168.1.50:1234"}}-```--#### `proxy/request`--A proxied request finished, or was rejected before forwarding. `duration_ms` covers the whole request; `ttfb_ms` is the time to the upstream's status line and is omitted where no response header arrived (rejection and transport-error paths). `error` carries normalized error text. This is operational metadata only — request and response bodies are never reported.--```json-{"jsonrpc":"2.0","method":"proxy/request","params":{"id":"17","node_id":"22222222-2222-2222-2222-222222222222","method":"POST","path":"/v1/chat/completions","target":"192.168.1.50:1234","status":200,"duration_ms":6120,"ttfb_ms":95}}-```--#### `workload:started` / `workload:completed` / `workload:errored`--One lifecycle transition per forwarded inference request, carrying a single `workloadInfo`. `engine` is always `lmstudio`; `originatedFrom` is left empty for the broker to stamp, and `scheduledOn` names the node that actually served (re-pointed if failover moved the request). The broker relays these to `nvpair-workload-manager`. The proxy never emits `workload:submitted` — it forwards immediately rather than queueing.--```json-{"jsonrpc":"2.0","method":"workload:started","params":{"workloadInfo":{"id":"17","model":"lmstudio-community/Qwen3-8B-GGUF","engine":"lmstudio","runId":"3ce8a1740b62df95","state":"running","originatedFrom":"","scheduledOn":"22222222-2222-2222-2222-222222222222","createdAt":1716998400000,"startedAt":1716998400000,"completedAt":null,"error":null,"requesterId":null}}}-```--#### `node/activity`--Raised while a node's engine is streaming a response back through the proxy: every successful write of upstream body bytes reports the node that produced them. The broker relays it to `nvpair-node-scanner`, which treats it as proof of life and cancels that node's eviction — a node saturated by inference cannot answer a liveness probe, but it is demonstrably alive precisely because it is streaming. Coalesced to one report per node per 2s (`nvpair-shared/nodeactivity`), since a generation writes hundreds of chunks and the scanner treats one report as good for a minute. `msSince` is the age of the observation; the broker adds its own relay delay before passing it on.--Only bytes that came from the upstream count. The proxy's own error bodies travel through the same writer and are never reported, because they say nothing about the node.--```json-{"jsonrpc":"2.0","method":"node/activity","params":{"hostUuid":"22222222-2222-2222-2222-222222222222","msSince":0}}-```--#### `errors:report` / `errors:clear`--Entries for the `nvpair-errors` pipeline, keyed by a stable `id` so a report and its clear cannot drift. A node dropping out of the discovery set raises `lmstudio-proxy:upstream-unreachable:<nodeId>`; its reappearance clears the same id. `nodeId` and `timestamp` are left unset for the broker to stamp.--```json-{"jsonrpc":"2.0","method":"errors:report","params":{"id":"lmstudio-proxy:upstream-unreachable:22222222-2222-2222-2222-222222222222","message":"Upstream node \"my-workstation\" is no longer reachable (dropped from discovery)","severity":"warning","action":"none"}}-```-------### Requests (client → proxy)--Requests carry an `id` and receive a response.--#### `nodes/list`--Returns all currently discovered nodes.--**Request:**-```json-{"jsonrpc":"2.0","id":1,"method":"nodes/list"}-```--**Response:**-```json-{"jsonrpc":"2.0","id":1,"result":{"nodes":[{"id":"22222222-2222-2222-2222-222222222222","host":"my-workstation","port":1234,"addresses":["192.168.1.50"]}]}}-```--#### `node/select`--Pin the proxy to route HTTP traffic to a specific node. Pass an empty `id` to return to auto-select.--**Request:**-```json-{"jsonrpc":"2.0","id":2,"method":"node/select","params":{"id":"22222222-2222-2222-2222-222222222222"}}-```--**Response:**-```json-{"jsonrpc":"2.0","id":2,"result":{"id":"22222222-2222-2222-2222-222222222222"}}-```--**Error** (node not found):-```json-{"jsonrpc":"2.0","id":2,"error":{"code":-32602,"message":"node \"xyz\" not found"}}-```--#### `node/selected`--Query the currently selected node.--**Request:**-```json-{"jsonrpc":"2.0","id":3,"method":"node/selected"}-```--**Response:**-```json-{"jsonrpc":"2.0","id":3,"result":{"id":"22222222-2222-2222-2222-222222222222"}}-```--An empty `id` means auto-select is active.--#### `node/set-priority`--Set the **auto-routing priority order** — an ordered list of node IDs, highest-priority first, optionally with each node's pending-work count and GPU pressure-in `ranks`. Delivered by `nvpair-ui-broker` on behalf of `nvpair-job-scheduler`,-which ranks the cluster least-loaded-first by total pending workload across-engines plus smoothed GPU pressure (see-[`nvpair-job-scheduler`](../nvpair-job-scheduler/README.md)). The snapshot is-stored verbatim and applied at request time. A `nodes`-only payload is valid and-supplies zero pending and GPU-pressure baselines.--**Request:**-```json-{"jsonrpc":"2.0","id":8,"method":"node/set-priority","params":{"nodes":["MY-PC","LAB-DESK-B","GPU-RIG"],"ranks":[{"id":"MY-PC","pending":0,"gpuPressure":0,"rank":0},{"id":"LAB-DESK-B","pending":1,"gpuPressure":1,"rank":1},{"id":"GPU-RIG","pending":3,"gpuPressure":3,"rank":2}]}}-```--**Response:**-```json-{"jsonrpc":"2.0","id":8,"result":{"count":3}}-```--| Field | Type | Description |-|-------|------|-------------|-| `count` | int | Number of node IDs stored (the length of the accepted list) |--Semantics:-- **Capability gate.** A model-bearing inference request first intersects the- discovery snapshot with nodes advertising that exact LM Studio model ID.- Selection, priority, reservations, and failover operate only on that- request-local owner set. If it is empty, the proxy returns `502` without- contacting an engine.-- **Auto ordering.** Within the eligible owner set, the proxy picks the listed- node carrying the least estimated load —- `pending + gpuPressure` from the last snapshot plus the proxy's own- reservations for requests it has already dispatched but whose workload feedback- has not come back yet — breaking ties by position in `nodes`. It increments the- chosen node's reservation before forwarding, so a concurrent burst spreads- instead of repeatedly choosing from the same stale snapshot. That node moves to- the front of this request's failover list and the rest keeps its order, so a- transport error or retryable status (the existing failover trigger) steps to- the next candidate.-- **Snapshot reset.** Each new snapshot replaces the pending and GPU-pressure- baselines and clears the reservations. GPU pressure is clamped to the- scheduler's 0–3 range.-- **Eligible manual pin wins.** An active `node/select` pin takes precedence when- it is in the request's owner set. An ineligible pin is ignored for that request,- so automatic reservations still apply among eligible owners. Clearing the pin- (`node/select` with an empty `id`) activates the most-recently-set list. Setting- a priority list does **not** emit `node/selection-changed` (the manual selection- is unchanged).-- **Unknown IDs are ignored.** IDs not currently in discovery are kept in the- stored list (a node may appear later) but contribute nothing until discovered.-- **Eligible unlisted nodes are a lowest-priority fallback.** An advertised owner- absent from the list stays routable, but only after every listed owner —- ordered among themselves by the default stable ID sort. This ensures an- eligible manually-added node the scheduler never saw is never stranded.-- **Empty list reverts to default.** `{"nodes":[]}` clears the scheduler's- influence and returns the proxy to its default auto ordering (eligible- discovered nodes by stable ID).--The list persists only in memory for the proxy's lifetime; it is not saved across-restarts. On restart the proxy comes back with an empty list, and the broker-re-pushes the last order once the proxy re-announces `ready`.--#### `set-port`--Change the HTTP listen port at runtime and persist the choice. The proxy-binds the new port first (so a bind failure leaves the current listener-serving), starts serving on it, then closes the old listener — in-flight-connections on the old port drain naturally. The new port is saved to-`lmstudio-proxy-port.json` and a fresh `ready` notification announces it.--**Request:**-```json-{"jsonrpc":"2.0","id":7,"method":"set-port","params":{"port":1300}}-```--**Response:**-```json-{"jsonrpc":"2.0","id":7,"result":{"version":"0.9.0","port":1300}}-```--**Error** (port in use / out of range):-```json-{"jsonrpc":"2.0","id":7,"error":{"code":-32000,"message":"failed to bind port 1300: ..."}}-```--When supervised by `nvpair-ui-broker`, callers reach this as `proxy:set-port`,-and the broker first steers the port clear of any running engine's port-(engines take precedence) before handing it down — see the broker README.--#### `node/add-manual`--Add a node manually (for networks where mDNS is blocked). If the node ID already exists as a manual node, it is updated.--**Request:**-```json-{"jsonrpc":"2.0","id":5,"method":"node/add-manual","params":{"id":"remote-server","host":"remote-server","port":1234,"addresses":["10.0.1.50"]}}-```--**Response:**-```json-{"jsonrpc":"2.0","id":5,"result":{"added":true}}-```--The proxy emits a `node/discovered` notification (or `node/updated` if the node was already registered). Manual nodes are a separate overlay that discovery snapshots never touch — they persist until explicitly removed.--#### `node/remove-manual`--Remove a previously added manual node.--**Request:**-```json-{"jsonrpc":"2.0","id":6,"method":"node/remove-manual","params":{"id":"remote-server"}}-```--**Response:**-```json-{"jsonrpc":"2.0","id":6,"result":{"removed":true}}-```--The proxy emits a `node/removed` notification and clears the active selection if it pointed to this node.--#### `node/set-local-backend`--Tell the proxy which loopback engine this node's own traffic terminates on. The broker sends it once the local LM Studio address and health are known. It is the target the cluster mTLS ingress forwards to, and the substitute used when discovery advertises this node's own proxy endpoint as a candidate. A zero `port` or `"healthy":false` effectively clears it, and the ingress then answers `503`.--**Request:**-```json-{"jsonrpc":"2.0","id":9,"method":"node/set-local-backend","params":{"engine":"lmstudio","host":"127.0.0.1","port":1235,"healthy":true}}-```--**Response:**-```json-{"jsonrpc":"2.0","id":9,"result":{"ok":true}}-```--#### `log/set-level`--Change the active log level at runtime (`debug`, `info`, `warn`, `error`). Accepted as a request or a notification; as a request it responds with the resolved level and rejects an unknown one with `-32602`.--**Request:**-```json-{"jsonrpc":"2.0","id":10,"method":"log/set-level","params":{"level":"debug"}}-```--**Response:**-```json-{"jsonrpc":"2.0","id":10,"result":{"level":"debug"}}-```--#### `shutdown`--Gracefully shuts down the proxy.--**Request:**-```json-{"jsonrpc":"2.0","id":4,"method":"shutdown"}-```--**Response:**-```json-{"jsonrpc":"2.0","id":4,"result":null}-```-------### Error Codes--Standard JSON-RPC 2.0 error codes apply:--| Code | Meaning |-|------|---------|-| `-32601` | Method not found |-| `-32602` | Invalid params |-| `-32000` | The request was well-formed but could not be carried out — returned by `set-port` when the new port cannot be bound |-------## Shutdown--The proxy shuts down gracefully on any of:--1. **stdin EOF** — parent closes stdin (stdio mode only)-2. **`shutdown` JSON-RPC request** — programmatic shutdown-3. **SIGINT / SIGTERM** — standard OS signals--## Discovery--The proxy does not browse mDNS. On startup it subscribes to the broker's discovery relay for `lm` (LM Studio) nodes (`discovery:subscribe {services:[lm]}`). Targets then arrive as `discovery:nodes` notifications carrying the relay's full filtered node set, and each snapshot replaces the routing overlay wholesale — a departed node is simply absent from the next one — while the diff against the previous overlay is what produces the `node/discovered`, `node/updated`, and `node/removed` notifications. User-added manual nodes are merged on top. Nodes are keyed by the discovery record's stable per-host UUID, so routing survives a machine being renamed. The single `_nvpair-node` browse that feeds the relay lives in the `nvpair-node-scanner` daemon (see its README) — this proxy is a pure consumer of the resulting routing set.diff --git a/services/lmstudio-proxy/activity_test.go b/services/lmstudio-proxy/activity_test.godeleted file mode 100644index 49a6354..0000000--- a/services/lmstudio-proxy/activity_test.go+++ /dev/null@@ -1,80 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "net/http"- "net/http/httptest"- "strings"- "testing"- "time"-)--// The two inference proxies are held to deliberate parity, so the liveness-// report a streaming response raises is asserted on both. See ollama-proxy's-// activity_test.go for the reasoning behind the signal itself.--// TestStreamedBytesReportNodeActivity mirrors the ollama-proxy test of the same-// name.-func TestStreamedBytesReportNodeActivity(t *testing.T) {- upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.WriteHeader(http.StatusOK)- _, _ = w.Write([]byte(`{"choices":[]}`))- }))- defer upstream.Close()-- rec := &prRec{}- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "serving-node", upstream.URL, "qwen"))- p := NewProxy(NewCodec(rec), disc, 1235)-- p.handleHTTP(httptest.NewRecorder(),- httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"qwen"}`)))-- deadline := time.Now().Add(2 * time.Second)- for time.Now().Before(deadline) && !rec.has(`"method":"node/activity"`) {- time.Sleep(5 * time.Millisecond)- }- if !rec.has(`"method":"node/activity"`) {- t.Fatal("no node/activity was reported after the upstream streamed a response")- }- if !rec.has(`"hostUuid":"serving-node"`) {- t.Fatal("node/activity did not name the node that served the request")- }-}--// A node that never wrote a response byte has proved nothing and must not be-// vouched for.-func TestNoActivityReportedWithoutUpstreamBytes(t *testing.T) {- rec := &prRec{}- disc := NewDiscovery()- // A port nothing is listening on: the dial fails, so no upstream byte can- // ever reach the client. It still has to advertise the requested model, or- // candidate pruning rejects the request before anything is dialled and the- // test passes without exercising the dial failure at all.- disc.AddManual(Node{- ID: "dead-node",- Addresses: []string{"127.0.0.1"},- Port: closedPortFor(t),- Models: []string{"qwen"},- })- p := NewProxy(NewCodec(rec), disc, 1235)-- p.handleHTTP(httptest.NewRecorder(),- httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"qwen"}`)))-- if rec.has(`"method":"node/activity"`) {- t.Fatal("activity was reported for a node that never answered")- }-}--// closedPortFor returns a port nothing is listening on, by binding and releasing-// it, so the dial is a prompt refusal rather than a timeout.-func closedPortFor(t *testing.T) int {- t.Helper()- srv := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))- port := nodeFor(t, "probe", srv.URL).Port- srv.Close()- return port-}diff --git a/services/lmstudio-proxy/choose_reachable_test.go b/services/lmstudio-proxy/choose_reachable_test.godeleted file mode 100644index 1ca29b0..0000000--- a/services/lmstudio-proxy/choose_reachable_test.go+++ /dev/null@@ -1,271 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "net"- "net/http"- "net/http/httptest"- "path/filepath"- "strconv"- "strings"- "sync"- "sync/atomic"- "testing"- "time"-- "nvpair-shared/clustertrust"- "nvpair-shared/clustertrusttest"- "nvpair-shared/reach"-)--// waitForTarget polls targetURL until it settles on want.-//-// Routing never waits on a handshake — reach.Prefer answers with the node's own-// ranking and confirms behind the request — so the address a multi-homed peer-// settles on is what the requests after the first see.-func waitForTarget(t *testing.T, p *Proxy, n Node, want string) {- t.Helper()- deadline := time.Now().Add(2 * time.Second)- for {- u := p.targetURL(n)- if u != nil && u.Host == want {- return- }- if time.Now().After(deadline) {- got := "<nil>"- if u != nil {- got = u.Host- }- t.Fatalf("targetURL settled on %s, want %s", got, want)- }- time.Sleep(time.Millisecond)- }-}--// countingChooser installs a target chooser that records how many connection-// attempts routing makes and whether they succeed, so a test can assert on-// confirmation behaviour without opening sockets.-func countingChooser(p *Proxy, accept bool) *atomic.Int32 {- var dials atomic.Int32- p.targets = reach.NewChooser(reach.WithDial(- func(_, _ string, _ time.Duration) (net.Conn, error) {- dials.Add(1)- if !accept {- return nil, net.ErrClosed- }- c1, c2 := net.Pipe()- _ = c2.Close()- return c1, nil- }))- return &dials-}--func TestChooseReachableFailsOverForPinnedPeer(t *testing.T) {- const peerUUID = "principal-peer"- const reachable = "192.0.2.11"- clusterDir := filepath.Join(t.TempDir(), "cluster")- clustertrusttest.Join(t, clusterDir, "cluster-xyz", "principal-self", peerUUID)-- p := testProxy(NewDiscovery(), 1235)- p.mesh = clustertrust.Open(clusterDir)- var dials atomic.Int32- p.targets = reach.NewChooser(reach.WithDial(- func(_, address string, _ time.Duration) (net.Conn, error) {- dials.Add(1)- host, _, _ := net.SplitHostPort(address)- if host != reachable {- return nil, net.ErrClosed- }- c1, c2 := net.Pipe()- _ = c2.Close()- return c1, nil- }))-- n := Node{- ID: "peer-a",- Port: 1234,- Addresses: []string{"192.0.2.10", "192.0.2.11"},- ClusterUUID: peerUUID,- }- // The first request is not made to wait for the confirmation, so it uses the- // node's own top-ranked address; the ones behind it use the one that answers.- if u := p.targetURL(n); u == nil || u.Host != net.JoinHostPort("192.0.2.10", "1234") {- t.Fatalf("first selection = %v, want the published ranking without waiting", u)- }- waitForTarget(t, p, n, net.JoinHostPort(reachable, "1234"))- if dials.Load() != 2 {- t.Fatalf("pinned peer triggered %d TCP probes, want both candidates tried", dials.Load())- }-}--func TestChooseReachableProbesPlainMultiHomed(t *testing.T) {- p := testProxy(NewDiscovery(), 1235)- dials := countingChooser(p, true)- n := Node{- ID: "manual-a",- Port: 1234,- Addresses: []string{"192.0.2.10", "192.0.2.11"},- }- if u := p.targetURL(n); u == nil {- t.Fatal("targetURL returned nil")- }- deadline := time.Now().Add(2 * time.Second)- for dials.Load() == 0 {- if time.Now().After(deadline) {- t.Fatal("plain multi-homed target did not confirm reachability")- }- time.Sleep(time.Millisecond)- }-}--// TestTargetURLFailsOverToAReachableAddress is the reported defect at the routing-// layer: a peer whose canonical address is a direct-connect link this host cannot-// reach must still be routed to, at the address that answers.-func TestTargetURLFailsOverToAReachableAddress(t *testing.T) {- const reachable = "10.172.55.129"- p := testProxy(NewDiscovery(), 1235)- p.targets = reach.NewChooser(reach.WithDial(- func(_, address string, _ time.Duration) (net.Conn, error) {- host, _, _ := net.SplitHostPort(address)- if host != reachable {- return nil, net.ErrClosed- }- c1, c2 := net.Pipe()- _ = c2.Close()- return c1, nil- }))-- n := Node{- ID: "spark",- Port: 1234,- // The node's own ranking leads with a link only its cabled neighbour can- // reach; this host is not that neighbour.- Addresses: []string{"192.168.240.1", reachable},- TXT: []string{"ip=192.168.240.1", "ips=192.168.240.1," + reachable},- }- waitForTarget(t, p, n, net.JoinHostPort(reachable, "1234"))-}--// TestNodeCandidatesKeepsPublishedOrder: the node ranked its addresses from-// evidence no observer has, so routing must try them in that order rather than-// re-sorting by address class.-func TestNodeCandidatesKeepsPublishedOrder(t *testing.T) {- n := Node{- ID: "spark",- Port: 1234,- Addresses: []string{"192.168.240.1", "10.172.55.129"},- TXT: []string{"ip=10.172.55.129", "ips=10.172.55.129,192.168.240.1"},- }- got := nodeCandidates(n)- want := []string{- net.JoinHostPort("10.172.55.129", "1234"),- net.JoinHostPort("192.168.240.1", "1234"),- }- if len(got) != len(want) || got[0] != want[0] || got[1] != want[1] {- t.Fatalf("nodeCandidates = %v, want %v", got, want)- }-}--// fakeNetwork is a chooser dialer whose accepting address can be moved, so a test-// can describe an address that stops answering and another that starts. Probes run-// on background goroutines, so both fields are read concurrently with the test.-type fakeNetwork struct {- mu sync.Mutex- accepting string- dials atomic.Int32-}--func (f *fakeNetwork) accept(address string) {- f.mu.Lock()- defer f.mu.Unlock()- f.accepting = address-}--func (f *fakeNetwork) install(p *Proxy) {- p.targets = reach.NewChooser(reach.WithDial(- func(_, address string, _ time.Duration) (net.Conn, error) {- f.dials.Add(1)- host, _, _ := net.SplitHostPort(address)- f.mu.Lock()- accepting := f.accepting- f.mu.Unlock()- if host != accepting {- return nil, net.ErrClosed- }- c1, c2 := net.Pipe()- _ = c2.Close()- return c1, nil- }))-}--// confirmedDeadPeer builds a proxy whose only routing target is a multi-homed peer-// whose confirmed address no longer answers: the address is a loopback endpoint-// whose server is already closed, so a forwarded request fails at the transport-// the way an unplugged link does, while the chooser still believes in it.-func confirmedDeadPeer(t *testing.T, replacement string) (*Proxy, Node, *fakeNetwork) {- t.Helper()- dead := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))- n := nodeForModel(t, "peer-a", dead.URL, "llama")- dead.Close()- confirmed := n.Addresses[0]- n.Addresses = append(n.Addresses, replacement)-- disc := NewDiscovery()- disc.AddManual(n)- p := testProxy(disc, 1235)- fake := &fakeNetwork{accepting: confirmed}- fake.install(p)-- waitForTarget(t, p, n, net.JoinHostPort(confirmed, strconv.Itoa(n.Port)))- return p, n, fake-}--// assertReprobed moves the accepting address and requires the selections after the-// failure to confirm again and land on the replacement. A cached winner that-// outlived the failure keeps answering with the old address and dials nothing.-func assertReprobed(t *testing.T, p *Proxy, n Node, fake *fakeNetwork, replacement string) {- t.Helper()- probesBefore := fake.dials.Load()- fake.accept(replacement)-- waitForTarget(t, p, n, net.JoinHostPort(replacement, strconv.Itoa(n.Port)))- if fake.dials.Load() == probesBefore {- t.Fatal("selection probed nothing: the failed address is still cached")- }-}--// TestUpstreamTransportFailureReprobesTheNextSelection: a dial failure against a-// multi-homed peer must retire the confirmed address. Without that, every later-// request keeps being sent to the address that just failed, and the peer's other-// published addresses are never tried — which is the whole reason routing confirms-// reachability in the first place.-func TestUpstreamTransportFailureReprobesTheNextSelection(t *testing.T) {- const replacement = "192.0.2.11"- p, n, fake := confirmedDeadPeer(t, replacement)-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`)))- if rec.Code != http.StatusBadGateway {- t.Fatalf("status = %d, want 502 from the only, unreachable candidate", rec.Code)- }-- assertReprobed(t, p, n, fake, replacement)-}--// TestModelListTransportFailureReprobesTheNextSelection: the aggregated model list-// reaches every candidate directly, so it learns about a dead address before any-// inference request does, and must retire it on the same evidence.-func TestModelListTransportFailureReprobesTheNextSelection(t *testing.T) {- const replacement = "192.0.2.11"- p, n, fake := confirmedDeadPeer(t, replacement)-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/v1/models", nil))- if rec.Code != http.StatusServiceUnavailable {- t.Fatalf("status = %d, want 503 when the only inventory source is unreachable", rec.Code)- }-- assertReprobed(t, p, n, fake, replacement)-}diff --git a/services/lmstudio-proxy/cluster_trust_test.go b/services/lmstudio-proxy/cluster_trust_test.godeleted file mode 100644index 39f1d18..0000000--- a/services/lmstudio-proxy/cluster_trust_test.go+++ /dev/null@@ -1,81 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "path/filepath"- "testing"-- "nvpair-shared/clustertrust"- "nvpair-shared/clustertrusttest"-)--// TestResolveCandidatesFollowsLivePinSet reproduces the join ordering that made-// a freshly-joined node route to nobody. The peer is discovered BEFORE this node-// has any cluster identity, and its discovery record never changes again — which-// is the normal steady state, because the relay only re-sends a node when its-// mDNS record actually moves. Routing must still pick the peer up the moment the-// pin lands, and drop it again the moment the pin is removed, because it reads-// the live pin set rather than a trust flag cached at discovery time.-func TestResolveCandidatesFollowsLivePinSet(t *testing.T) {- const peerUUID = "principal-peer"- clusterDir := filepath.Join(t.TempDir(), "cluster")-- disc := NewDiscovery()- disc.SetSubscribed([]Node{{- ID: "peer-a", Host: "peer-a", Port: 1234,- Addresses: []string{"192.0.2.10"},- IP: "192.0.2.10",- ClusterUUID: peerUUID,- }})- p := testProxy(disc, 1235)- p.mesh = clustertrust.Open(clusterDir)-- // Pre-join: no identity, no pins, so the peer is not a routable target.- if got := p.resolveCandidates(""); len(got) != 0 {- t.Fatalf("pre-join candidates = %+v, want none", got)- }-- // The cluster-manager lands the join on disk while the proxy is running. No- // new discovery snapshot arrives — the peer's mDNS record has not changed.- clustertrusttest.Join(t, clusterDir, "cluster-xyz", "principal-self", peerUUID)-- cands := p.resolveCandidates("")- if len(cands) != 1 {- t.Fatalf("post-join candidates = %+v, want the peer", cands)- }- if cands[0].id != "peer-a" || cands[0].peerUUID != peerUUID || cands[0].url.Scheme != "https" {- t.Fatalf("post-join candidate = %+v, want peer-a over https pinned to %s", cands[0], peerUUID)- }-- // Removing the peer from the cluster retires it as a target just as promptly,- // again with no discovery event involved.- clustertrusttest.RemovePeerPin(t, clusterDir, peerUUID)- if got := p.resolveCandidates(""); len(got) != 0 {- t.Fatalf("post-removal candidates = %+v, want none", got)- }-}--// TestResolveCandidatesRejectsUnpinnedClusteredPeer keeps the isolation property-// honest now that trust is read locally: a peer that advertises a cluster-// principal we hold no pin for is not routable, even though this node is itself-// a healthy cluster member. Two clusters on one LAN must not route to each other.-func TestResolveCandidatesRejectsUnpinnedClusteredPeer(t *testing.T) {- clusterDir := filepath.Join(t.TempDir(), "cluster")- clustertrusttest.Join(t, clusterDir, "cluster-ours", "principal-self", "principal-ourpeer")-- disc := NewDiscovery()- disc.SetSubscribed([]Node{{- ID: "stranger", Host: "stranger", Port: 1234,- Addresses: []string{"192.0.2.30"},- IP: "192.0.2.30",- ClusterUUID: "principal-stranger",- }})- p := testProxy(disc, 1235)- p.mesh = clustertrust.Open(clusterDir)-- if got := p.resolveCandidates(""); len(got) != 0 {- t.Fatalf("candidates = %+v, want none for a peer in another cluster", got)- }-}diff --git a/services/lmstudio-proxy/discovery.go b/services/lmstudio-proxy/discovery.godeleted file mode 100644index 9b05581..0000000--- a/services/lmstudio-proxy/discovery.go+++ /dev/null@@ -1,165 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--// Discovery is the proxy's routing-target set. The proxy runs no mDNS of its-// own: routing targets are pushed down from the broker's discovery relay-// (discovery:nodes snapshots for the lm service) into the subscribed overlay,-// merged with user-added manual nodes. The proxy is itself advertised — as an lm-// service — by the node-scanner daemon's single _nvpair-node record, keyed off-// the engine port the broker's poller registers.-//-// The routable Node projection (IP / withPrimaryIP) and the manual-node-// overlay live here; request-path reachability (TCP-probe + failover) lives in-// proxy.go.--import (- "slices"- "sync"-- "nvpair-shared/discovery"- "nvpair-shared/netpick"-)--// uuidFromTXT extracts a node's stable uuid= from its TXT records. Kept as a-// thin re-export of the shared helper (it was triplicated across the two proxies-// and the scanner before consolidation).-var uuidFromTXT = discovery.UUIDFromTXT--// Node is the proxy's routable view of a node. It adds a canonical dialable IP-// field over the discovered node shape.-type Node struct {- ID string `json:"id"`- Host string `json:"host"`- Port int `json:"port"`- Addresses []string `json:"addresses"`- TXT []string `json:"txt"`- // Models is the latest model inventory carried by the broker's discovery- // snapshot. Model-bearing inference is eligible only when this list- // advertises the requested model; an empty list stays in discovery but is- // not an inference candidate until a later inventory update.- Models []string `json:"models,omitempty"`- // IP is the single canonical LAN address a consumer should dial/display for- // this node, resolved via the shared netpick ranker: the node's- // own ip= TXT if present, else the best-scored advertised IPv4. It is- // stamped onto outbound node/* notifications only (see withPrimaryIP) so a- // downstream consumer agrees on the same address the proxy routes to.- IP string `json:"ip,omitempty"`- // ClusterUUID is the relay peer's cluster principal (its mTLS cert UUID),- // carried from the discovery DirectoryNode. Non-empty only for a clustered- // peer; it is the key used to pin the peer's server cert when dialing its- // promoted proxy over cluster mTLS. Whether we actually hold that pin is- // resolved against the live mesh at routing time (resolveCandidates), never- // cached here: a cached answer goes stale the moment a peer is paired or- // removed. Internal routing metadata, not part of the proxy's outward node- // contract.- ClusterUUID string `json:"-"`-}--// withPrimaryIP returns a copy of the node with IP resolved by the shared ranker-// (netpick.Primary over its TXT + Addresses).-func (n Node) withPrimaryIP() Node {- n.IP = netpick.Primary(n.TXT, n.Addresses)- return n-}--// Discovery holds the proxy's routing targets: the relay-fed subscribed overlay-// and the user-added manual overlay.-type Discovery struct {- mu sync.RWMutex- manualNodes map[string]Node- // subscribedNodes are routing targets pushed down by the broker's discovery- // relay (discovery:nodes snapshots for the lm service), keyed by node ID (the- // directory instance name).- subscribedNodes map[string]Node-}--func NewDiscovery() *Discovery {- return &Discovery{- manualNodes: make(map[string]Node),- subscribedNodes: make(map[string]Node),- }-}--// Nodes returns the merged subscribed + manual node set (manual entries that-// aren't also present via the relay are appended).-func (d *Discovery) Nodes() []Node {- d.mu.RLock()- defer d.mu.RUnlock()- out := make([]Node, 0, len(d.manualNodes)+len(d.subscribedNodes))- seen := make(map[string]struct{}, len(d.subscribedNodes))- for id, n := range d.subscribedNodes {- out = append(out, n)- seen[id] = struct{}{}- }- for id, n := range d.manualNodes {- if _, exists := seen[id]; !exists {- out = append(out, n)- }- }- return out-}--// SetSubscribed replaces the relay-fed routing overlay with the given set and-// reports what changed versus the previous set (keyed by node ID): nodes newly-// present, nodes whose routable details changed, and nodes that dropped out. The-// broker pushes the full filtered snapshot on every change, so the overlay is-// replaced wholesale; the returned diff lets the caller emit node/discovered|-// updated|removed so a consumer (the UI) learns which peers currently run this-// engine. Manual nodes are a separate overlay and are untouched.-func (d *Discovery) SetSubscribed(nodes []Node) (discovered, updated, removed []Node) {- d.mu.Lock()- defer d.mu.Unlock()- next := make(map[string]Node, len(nodes))- for _, n := range nodes {- next[n.ID] = n- switch prev, ok := d.subscribedNodes[n.ID]; {- case !ok:- discovered = append(discovered, n)- case !nodeEqual(prev, n):- updated = append(updated, n)- }- }- for id, prev := range d.subscribedNodes {- if _, ok := next[id]; !ok {- removed = append(removed, prev)- }- }- d.subscribedNodes = next- return discovered, updated, removed-}--// nodeEqual reports whether two routable Nodes carry the same routing/display-// identity — the fields a consumer dials or renders. A change in any of them-// warrants a node/updated.-func nodeEqual(a, b Node) bool {- return a.ID == b.ID && a.Host == b.Host && a.Port == b.Port && a.IP == b.IP &&- slices.Equal(a.Addresses, b.Addresses) && slices.Equal(a.TXT, b.TXT) &&- slices.Equal(a.Models, b.Models)-}--func (d *Discovery) AddManual(node Node) (added bool) {- d.mu.Lock()- defer d.mu.Unlock()- _, exists := d.manualNodes[node.ID]- d.manualNodes[node.ID] = node- return !exists-}--func (d *Discovery) RemoveManual(id string) (removed bool) {- d.mu.Lock()- defer d.mu.Unlock()- _, exists := d.manualNodes[id]- if exists {- delete(d.manualNodes, id)- }- return exists-}--func (d *Discovery) IsManual(id string) bool {- d.mu.RLock()- defer d.mu.RUnlock()- _, exists := d.manualNodes[id]- return exists-}diff --git a/services/lmstudio-proxy/e2e_test.go b/services/lmstudio-proxy/e2e_test.godeleted file mode 100644index 5412ab7..0000000--- a/services/lmstudio-proxy/e2e_test.go+++ /dev/null@@ -1,217 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "bufio"- "encoding/json"- "fmt"- "io"- "net"- "net/http"- "net/http/httptest"- "os"- "os/exec"- "path/filepath"- "runtime"- "strconv"- "strings"- "testing"- "time"-)--// proxyBin is the real lmstudio-proxy binary, built once in TestMain so the-// e2e test exercises the shipped artifact (not just in-process handlers).-var proxyBin string--func TestMain(m *testing.M) {- tmp, err := os.MkdirTemp("", "nvpair-lmproxy-e2e-*")- if err != nil {- panic(err)- }- suffix := ""- if runtime.GOOS == "windows" {- suffix = ".exe"- }- proxyBin = filepath.Join(tmp, "lmstudio-proxy"+suffix)- if out, err := exec.Command("go", "build", "-o", proxyBin, ".").CombinedOutput(); err != nil {- panic("build lmstudio-proxy: " + err.Error() + "\n" + string(out))- }- code := m.Run()- _ = os.RemoveAll(tmp)- os.Exit(code)-}--type e2eFrame struct {- ID json.RawMessage `json:"id"`- Method string `json:"method"`- Params json.RawMessage `json:"params"`- Result json.RawMessage `json:"result"`- Error json.RawMessage `json:"error"`-}--func e2eReadFrames(r io.Reader, out chan<- e2eFrame) {- sc := bufio.NewScanner(r)- sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)- for sc.Scan() {- var f e2eFrame- if err := json.Unmarshal(sc.Bytes(), &f); err != nil {- continue- }- out <- f- }-}--func e2eSend(t *testing.T, w io.Writer, id int, method string, params any) {- t.Helper()- msg := map[string]any{"jsonrpc": "2.0", "id": id, "method": method}- if params != nil {- msg["params"] = params- }- data, _ := json.Marshal(msg)- if _, err := w.Write(append(data, '\n')); err != nil {- t.Fatalf("send %s: %v", method, err)- }-}--func e2eWaitResult(t *testing.T, frames <-chan e2eFrame, id string, timeout time.Duration) {- t.Helper()- deadline := time.After(timeout)- for {- select {- case f := <-frames:- if string(f.ID) != id {- continue- }- if len(f.Error) > 0 && string(f.Error) != "null" {- t.Fatalf("rpc id %s returned error: %s", id, f.Error)- }- return- case <-deadline:- t.Fatalf("timed out waiting for response id %s", id)- }- }-}--func e2eWaitReadyPort(t *testing.T, frames <-chan e2eFrame, timeout time.Duration) int {- t.Helper()- deadline := time.After(timeout)- for {- select {- case f := <-frames:- if f.Method != "ready" {- continue- }- var p struct {- Port int `json:"port"`- }- if err := json.Unmarshal(f.Params, &p); err != nil {- t.Fatalf("parse ready params: %v", err)- }- return p.Port- case <-deadline:- t.Fatalf("timed out waiting for ready notification")- return 0- }- }-}--func e2eFreePort(t *testing.T) int {- t.Helper()- l, err := net.Listen("tcp", "127.0.0.1:0")- if err != nil {- t.Fatal(err)- }- defer l.Close()- return l.Addr().(*net.TCPAddr).Port-}--func e2eSplitHostPort(t *testing.T, serverURL string) (string, int) {- t.Helper()- host, portStr, err := net.SplitHostPort(strings.TrimPrefix(serverURL, "http://"))- if err != nil {- t.Fatalf("split %q: %v", serverURL, err)- }- port, err := strconv.Atoi(portStr)- if err != nil {- t.Fatalf("port %q: %v", portStr, err)- }- return host, port-}--// TestE2EFailoverOverRealBinary spawns the real lmstudio-proxy binary and-// drives it the way the broker/UI does: register a busy (503) and a healthy-// (200) upstream as manual nodes over JSON-RPC stdio, then send a genuine-// OpenAI inference POST to the proxy's real HTTP port. It asserts the request-// fails over from the busy node to the healthy one, the original body is-// replayed, and CORS headers are present — the whole shipped path (binary +-// stdio control plane + HTTP forwarding + failover) end-to-end, no mocks.-func TestE2EFailoverOverRealBinary(t *testing.T) {- var gotBody string- busy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.WriteHeader(http.StatusServiceUnavailable)- }))- defer busy.Close()- good := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- b, _ := io.ReadAll(r.Body)- gotBody = string(b)- w.WriteHeader(http.StatusOK)- io.WriteString(w, `{"ok":true}`)- }))- defer good.Close()-- port := e2eFreePort(t)- cmd := exec.Command(proxyBin, "--port", strconv.Itoa(port))- stdin, err := cmd.StdinPipe()- if err != nil {- t.Fatal(err)- }- stdout, err := cmd.StdoutPipe()- if err != nil {- t.Fatal(err)- }- if err := cmd.Start(); err != nil {- t.Fatal(err)- }- defer func() {- _ = stdin.Close()- _ = cmd.Process.Kill()- _ = cmd.Wait()- }()-- frames := make(chan e2eFrame, 256)- go e2eReadFrames(stdout, frames)-- if got := e2eWaitReadyPort(t, frames, 10*time.Second); got != port {- t.Fatalf("ready port = %d, want %d", got, port)- }-- busyHost, busyPort := e2eSplitHostPort(t, busy.URL)- goodHost, goodPort := e2eSplitHostPort(t, good.URL)- e2eSend(t, stdin, 1, "node/add-manual", map[string]any{"id": "busy", "host": busyHost, "port": busyPort, "addresses": []string{busyHost}, "models": []string{"m"}})- e2eWaitResult(t, frames, "1", 5*time.Second)- e2eSend(t, stdin, 2, "node/add-manual", map[string]any{"id": "good", "host": goodHost, "port": goodPort, "addresses": []string{goodHost}, "models": []string{"m"}})- e2eWaitResult(t, frames, "2", 5*time.Second)- // Select the busy node so the failover path is deterministic.- e2eSend(t, stdin, 3, "node/select", map[string]any{"id": "busy"})- e2eWaitResult(t, frames, "3", 5*time.Second)-- resp, err := http.Post(fmt.Sprintf("http://127.0.0.1:%d/v1/chat/completions", port), "application/json", strings.NewReader(`{"model":"m"}`))- if err != nil {- t.Fatalf("inference POST: %v", err)- }- defer resp.Body.Close()- if resp.StatusCode != http.StatusOK {- t.Fatalf("status = %d, want 200 (should fail over from the 503 node)", resp.StatusCode)- }- if got := resp.Header.Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want *", got)- }- if gotBody != `{"model":"m"}` {- t.Errorf("healthy upstream got body %q, want the original request body", gotBody)- }-- e2eSend(t, stdin, 9, "shutdown", nil)- e2eWaitResult(t, frames, "9", 5*time.Second)-}diff --git a/services/lmstudio-proxy/failover_test.go b/services/lmstudio-proxy/failover_test.godeleted file mode 100644index 02e5361..0000000--- a/services/lmstudio-proxy/failover_test.go+++ /dev/null@@ -1,600 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "encoding/json"- "io"- "net"- "net/http"- "net/http/httptest"- "net/url"- "strconv"- "strings"- "testing"- "time"-)--// rwNop is a no-op io.ReadWriter so a Codec can be constructed in tests-// without a real transport: reads hit EOF immediately and writes are-// discarded. handleHTTP only ever writes (notifications), so this is enough.-type rwNop struct{}--func (rwNop) Read([]byte) (int, error) { return 0, io.EOF }-func (rwNop) Write(p []byte) (int, error) { return len(p), nil }--func testProxy(disc *Discovery, port int) *Proxy {- return NewProxy(NewCodec(rwNop{}), disc, port)-}--// nodeFor turns an httptest server URL into a discovery Node pointing at it.-func nodeFor(t *testing.T, id, serverURL string) Node {- t.Helper()- u, err := url.Parse(serverURL)- if err != nil {- t.Fatalf("parse %q: %v", serverURL, err)- }- host, portStr, err := net.SplitHostPort(u.Host)- if err != nil {- t.Fatalf("split %q: %v", u.Host, err)- }- port, err := strconv.Atoi(portStr)- if err != nil {- t.Fatalf("port %q: %v", portStr, err)- }- return Node{ID: id, Addresses: []string{host}, Port: port}-}--func nodeForModel(t *testing.T, id, serverURL, model string) Node {- t.Helper()- node := nodeFor(t, id, serverURL)- node.Models = []string{model}- return node-}--// TestHandlePlain_OptionsPreflight: a CORS preflight is answered locally with-// 204 + permissive headers and never forwarded.-func TestHandlePlain_OptionsPreflight(t *testing.T) {- p := testProxy(NewDiscovery(), 11434)- rec := httptest.NewRecorder()- req := httptest.NewRequest(http.MethodOptions, "/v1/chat/completions", nil)- req.RemoteAddr = "127.0.0.1:40000"- req.Header.Set("Access-Control-Request-Headers", "X-Custom-Token")- p.handlePlain(rec, req)-- if rec.Code != http.StatusNoContent {- t.Fatalf("status = %d, want 204", rec.Code)- }- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want *", got)- }- if rec.Header().Get("Access-Control-Allow-Methods") == "" {- t.Errorf("missing Access-Control-Allow-Methods")- }- if got := rec.Header().Get("Access-Control-Expose-Headers"); got != "*" {- t.Errorf("Access-Control-Expose-Headers = %q, want *", got)- }- // The browser's requested headers are echoed so an arbitrary header clears preflight.- if got := rec.Header().Get("Access-Control-Allow-Headers"); got != "X-Custom-Token" {- t.Errorf("Access-Control-Allow-Headers = %q, want echoed X-Custom-Token", got)- }-}--// TestHandlePlain_EngineCredentialedPreflightPreserved: when an engine opts an-// exact origin into credentialed CORS, its preflight policy reaches the browser-// instead of being replaced by the proxy's uncredentialed wildcard fallback.-func TestHandlePlain_EngineCredentialedPreflightPreserved(t *testing.T) {- preflightSeen := make(chan struct{}, 1)- engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- if r.Method != http.MethodOptions {- t.Errorf("engine method = %s, want OPTIONS", r.Method)- }- preflightSeen <- struct{}{}- w.Header().Set("Access-Control-Allow-Origin", "https://app.example")- w.Header().Set("Access-Control-Allow-Credentials", "true")- w.Header().Set("Access-Control-Allow-Methods", "POST")- w.Header().Set("Access-Control-Allow-Headers", "Content-Type")- w.WriteHeader(http.StatusNoContent)- }))- defer engine.Close()-- disc := NewDiscovery()- disc.AddManual(nodeFor(t, "engine", engine.URL))- p := testProxy(disc, 11434)- req := httptest.NewRequest(http.MethodOptions, "/v1/chat/completions", nil)- req.RemoteAddr = "127.0.0.1:40000"- req.Header.Set("Origin", "https://app.example")- req.Header.Set("Access-Control-Request-Method", http.MethodPost)- req.Header.Set("Access-Control-Request-Headers", "Content-Type")- rec := httptest.NewRecorder()-- p.handlePlain(rec, req)-- select {- case <-preflightSeen:- default:- t.Fatal("engine did not receive the credentialed preflight")- }- if rec.Code != http.StatusNoContent {- t.Errorf("status = %d, want %d", rec.Code, http.StatusNoContent)- }- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "https://app.example" {- t.Errorf("Access-Control-Allow-Origin = %q, want the engine's exact origin", got)- }- if got := rec.Header().Get("Access-Control-Allow-Credentials"); got != "true" {- t.Errorf("Access-Control-Allow-Credentials = %q, want the engine's true", got)- }-}--// TestHandleHTTP_EngineCORSPolicyPreserved: an engine that declares its own-// origin policy keeps it. Replacing it with the proxy's wildcard would widen-// what the user configured, and would break a credentialed response outright.-func TestHandleHTTP_EngineCORSPolicyPreserved(t *testing.T) {- engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- w.Header().Set("Access-Control-Allow-Origin", "https://app.example")- w.Header().Set("Access-Control-Allow-Credentials", "true")- w.WriteHeader(http.StatusOK)- io.WriteString(w, `{"done":true}`)- }))- defer engine.Close()-- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "engine", engine.URL, "llama"))- p := testProxy(disc, 11434)-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`)))-- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "https://app.example" {- t.Errorf("Access-Control-Allow-Origin = %q, want the engine's own origin", got)- }- if got := rec.Header().Get("Access-Control-Allow-Credentials"); got != "true" {- t.Errorf("Access-Control-Allow-Credentials = %q, want the engine's true", got)- }-}--// TestHandleHTTP_EngineCredentialsWithoutOriginDropped: an engine (or an-// intermediary in front of it) that sends Allow-Credentials but no origin has-// declared no policy to keep, so the proxy supplies its own. The wildcard it-// writes is invalid next to Allow-Credentials: true, and a browser rejects that-// pair, so the inherited header must not survive the forward.-func TestHandleHTTP_EngineCredentialsWithoutOriginDropped(t *testing.T) {- engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- w.Header().Set("Access-Control-Allow-Credentials", "true")- w.WriteHeader(http.StatusOK)- io.WriteString(w, `{"done":true}`)- }))- defer engine.Close()-- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "engine", engine.URL, "llama"))- p := testProxy(disc, 11434)-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`)))-- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want the proxy's wildcard", got)- }- if got := rec.Header().Get("Access-Control-Allow-Credentials"); got != "" {- t.Errorf("Access-Control-Allow-Credentials = %q, want cleared alongside the wildcard origin", got)- }-}--// TestHandleHTTP_HappyPathSingleNode: the common case — one healthy node-// answers directly, body forwarded, CORS present on the success response.-func TestHandleHTTP_HappyPathSingleNode(t *testing.T) {- var gotBody string- good := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- b, _ := io.ReadAll(r.Body)- gotBody = string(b)- w.WriteHeader(http.StatusOK)- io.WriteString(w, `{"done":true}`)- }))- defer good.Close()-- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "good", good.URL, "llama"))- p := testProxy(disc, 11434)-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`)))-- if rec.Code != http.StatusOK {- t.Fatalf("status = %d, want 200", rec.Code)- }- if gotBody != `{"model":"llama"}` {- t.Errorf("node got body %q, want the original request body", gotBody)- }- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want * on success", got)- }-}--// TestHandleHTTP_NoRetryOn400: a client error (400) is returned as-is and not-// failed over — retrying elsewhere would return the same error and mask it.-func TestHandleHTTP_NoRetryOn400(t *testing.T) {- hits := 0- bad := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- hits++- w.WriteHeader(http.StatusBadRequest)- io.WriteString(w, `{"error":"bad request"}`)- }))- defer bad.Close()- other := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.WriteHeader(http.StatusOK)- }))- defer other.Close()-- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "bad", bad.URL, "llama"))- disc.AddManual(nodeForModel(t, "other", other.URL, "llama"))- p := testProxy(disc, 11434)- p.SetSelected("bad")-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`)))-- if rec.Code != http.StatusBadRequest {- t.Fatalf("status = %d, want 400 (client errors must not fail over)", rec.Code)- }- if hits != 1 {- t.Errorf("bad node hit %d times, want exactly 1 (no retry on 400)", hits)- }-}--// TestHandleHTTP_RejectionHasCORS: even the no-node rejection carries CORS so a-// browser sees the real 502 instead of an opaque CORS error.-func TestHandleHTTP_RejectionHasCORS(t *testing.T) {- p := testProxy(NewDiscovery(), 11434)- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"x"}`)))-- if rec.Code != http.StatusBadGateway {- t.Fatalf("status = %d, want 502", rec.Code)- }- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want * on rejection", got)- }-}--// TestHandleHTTP_FailoverOn503: a busy first node (503) is skipped and the-// request is filled by the next node, with the original body replayed.-func TestHandleHTTP_FailoverOn503(t *testing.T) {- busy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.WriteHeader(http.StatusServiceUnavailable)- io.WriteString(w, `{"error":"loading model"}`)- }))- defer busy.Close()-- var gotBody string- good := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- b, _ := io.ReadAll(r.Body)- gotBody = string(b)- w.WriteHeader(http.StatusOK)- io.WriteString(w, `{"done":true}`)- }))- defer good.Close()-- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "busy", busy.URL, "llama"))- disc.AddManual(nodeForModel(t, "good", good.URL, "llama"))- p := testProxy(disc, 11434)- p.SetSelected("busy") // deterministic: busy is tried first-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`)))-- if rec.Code != http.StatusOK {- t.Fatalf("status = %d, want 200 (should have failed over past the 503)", rec.Code)- }- if gotBody != `{"model":"llama"}` {- t.Errorf("failover node got body %q, want the original request body", gotBody)- }- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want * on proxied success", got)- }-}--// TestHandleHTTP_AllNodesDownReturnsError: when every candidate fails at the-// transport, the client gets one clean 502 (not a hang), still with CORS.-func TestHandleHTTP_AllNodesDownReturnsError(t *testing.T) {- // Two servers we immediately close so dials fail.- a := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))- b := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))- na := nodeForModel(t, "a", a.URL, "llama")- nb := nodeForModel(t, "b", b.URL, "llama")- a.Close()- b.Close()-- disc := NewDiscovery()- disc.AddManual(na)- disc.AddManual(nb)- p := testProxy(disc, 11434)-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`)))-- if rec.Code != http.StatusBadGateway {- t.Fatalf("status = %d, want 502 when all nodes are down", rec.Code)- }- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want * on exhausted error", got)- }-}--// TestHandleHTTP_404FailoverInferenceOnly: a 404 (model-not-found) on an-// inference call fails over to the next advertised owner, but a 404 on a-// non-inference path is returned as-is.-func TestHandleHTTP_404FailoverInferenceOnly(t *testing.T) {- missing := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.WriteHeader(http.StatusNotFound)- io.WriteString(w, `{"error":"model not found"}`)- }))- defer missing.Close()- has := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.WriteHeader(http.StatusOK)- io.WriteString(w, `{"done":true}`)- }))- defer has.Close()-- newProxy := func() *Proxy {- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "missing", missing.URL, "llama"))- disc.AddManual(nodeForModel(t, "has", has.URL, "llama"))- p := testProxy(disc, 11434)- p.SetSelected("missing")- return p- }-- // Inference POST: 404 on first → fail over → 200.- rec := httptest.NewRecorder()- newProxy().handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`)))- if rec.Code != http.StatusOK {- t.Fatalf("inference 404: status = %d, want 200 (should fail over)", rec.Code)- }-- // An ordinary non-inference GET still returns the first node's 404.- rec = httptest.NewRecorder()- newProxy().handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/v1/unknown", nil))- if rec.Code != http.StatusNotFound {- t.Fatalf("non-inference 404: status = %d, want 404 (must NOT fail over)", rec.Code)- }-}--func TestHandleHTTP_AggregatesModelList(t *testing.T) {- entered := make(chan struct{}, 2)- release := make(chan struct{})- server := func(body string) *httptest.Server {- return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- if r.Method != http.MethodGet || r.URL.Path != "/v1/models" {- t.Errorf("upstream request = %s %s, want GET /v1/models", r.Method, r.URL.Path)- }- if r.Header.Get("Authorization") != "" || r.Header.Get("Cookie") != "" {- t.Errorf("client credentials leaked to fan-out target")- }- entered <- struct{}{}- <-release- _, _ = io.WriteString(w, body)- }))- }- a := server(`{"object":"list","data":[{"id":"a","owned_by":"a-only"},{"id":"shared","owned_by":"first"}]}`)- defer a.Close()- b := server(`{"object":"list","data":[{"id":"shared","owned_by":"second"},{"id":"c","owned_by":"c-only"}]}`)- defer b.Close()- malformed := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- _, _ = io.WriteString(w, `{"object":"list","data":null}`)- }))- defer malformed.Close()- down := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))- downNode := nodeFor(t, "down", down.URL)- down.Close()-- disc := NewDiscovery()- disc.AddManual(nodeFor(t, "a", a.URL))- disc.AddManual(nodeFor(t, "b", b.URL))- disc.AddManual(downNode)- disc.AddManual(nodeFor(t, "malformed", malformed.URL))- p := testProxy(disc, 1234)- p.SetSelected("a")- rec := httptest.NewRecorder()- req := httptest.NewRequest(http.MethodGet, "/v1/models", nil)- req.Header.Set("Authorization", "Bearer client-secret")- req.Header.Set("Cookie", "session=client-secret")- done := make(chan struct{})- go func() {- p.handleHTTP(rec, req)- close(done)- }()-- for range 2 {- select {- case <-entered:- case <-time.After(5 * time.Second):- close(release)- t.Fatal("model-list requests were not issued concurrently")- }- }- close(release)- select {- case <-done:- case <-time.After(5 * time.Second):- t.Fatal("aggregate request did not finish")- }-- if rec.Code != http.StatusOK {- t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())- }- var got struct {- Object string `json:"object"`- Data []struct {- ID string `json:"id"`- OwnedBy string `json:"owned_by"`- } `json:"data"`- }- if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {- t.Fatal(err)- }- if got.Object != "list" {- t.Errorf("object = %q, want list", got.Object)- }- if len(got.Data) != 3 || got.Data[0].ID != "a" || got.Data[1].ID != "shared" || got.Data[2].ID != "c" {- t.Fatalf("models = %+v, want a, shared, c", got.Data)- }- if got.Data[1].OwnedBy != "first" {- t.Errorf("duplicate metadata = %q, want deterministic first candidate", got.Data[1].OwnedBy)- }- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want *", got)- }-}--func TestHandleHTTP_ModelListEmptyAndUnavailable(t *testing.T) {- empty := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- _, _ = io.WriteString(w, `{"object":"list","data":[]}`)- }))- emptyNode := nodeFor(t, "empty", empty.URL)- empty.Close()-- disc := NewDiscovery()- disc.AddManual(emptyNode)- rec := httptest.NewRecorder()- testProxy(disc, 1234).handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/v1/models", nil))- if rec.Code != http.StatusServiceUnavailable {- t.Fatalf("unavailable status = %d, want 503", rec.Code)- }-- empty = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- _, _ = io.WriteString(w, `{"object":"list","data":[]}`)- }))- defer empty.Close()- disc = NewDiscovery()- disc.AddManual(nodeFor(t, "empty", empty.URL))- rec = httptest.NewRecorder()- testProxy(disc, 1234).handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/v1/models", nil))- if rec.Code != http.StatusOK || rec.Body.String() != `{"object":"list","data":[]}` {- t.Fatalf("empty response = %d %s, want 200 native empty list", rec.Code, rec.Body.String())- }-}--// TestHandleHTTP_StrictModelRouting proves capability is a gate before-// selection and priority. Unknown and known-missing nodes are excluded from-// inference but remain available for non-inference routes.-func TestHandleHTTP_StrictModelRouting(t *testing.T) {- missHits, unknownHits, matchHits := 0, 0, 0- miss := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- missHits++- w.WriteHeader(http.StatusOK)- }))- defer miss.Close()- unknown := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- unknownHits++- w.WriteHeader(http.StatusOK)- }))- defer unknown.Close()- match := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- matchHits++- body, _ := io.ReadAll(r.Body)- if string(body) != `{"model":"llama"}` {- t.Errorf("matching node got body %q", body)- }- w.WriteHeader(http.StatusOK)- }))- defer match.Close()-- disc := NewDiscovery()- missNode := nodeFor(t, "selected-miss", miss.URL)- missNode.Models = []string{"mistral"}- unknownNode := nodeFor(t, "a-unknown", unknown.URL)- matchNode := nodeFor(t, "z-match", match.URL)- matchNode.Models = []string{"llama"}- disc.AddManual(missNode)- disc.AddManual(unknownNode)- disc.AddManual(matchNode)- p := testProxy(disc, 11434)- p.SetSelected("selected-miss")- p.SetPriority([]string{"a-unknown", "selected-miss", "z-match"})- candidates := p.resolveCandidates("llama")- if len(candidates) != 1 || candidates[0].id != "z-match" {- t.Fatalf("model candidates = %v, want only z-match", candidates)- }-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`)))-- if rec.Code != http.StatusOK {- t.Fatalf("status = %d, want 200", rec.Code)- }- if missHits != 0 || unknownHits != 0 || matchHits != 1 {- t.Fatalf("hits miss=%d unknown=%d match=%d, want 0/0/1", missHits, unknownHits, matchHits)- }-- // Capability filtering applies only to model-bearing inference.- rec = httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/v1/unknown", nil))- if rec.Code != http.StatusOK {- t.Fatalf("non-inference status = %d, want 200", rec.Code)- }- if missHits != 1 || unknownHits != 0 || matchHits != 1 {- t.Fatalf("non-inference hits miss=%d unknown=%d match=%d, want 1/0/1", missHits, unknownHits, matchHits)- }-}--func TestHandleHTTP_NoAdvertisedModelRejectsLocally(t *testing.T) {- hits := 0- upstream := func() *httptest.Server {- return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- hits++- w.WriteHeader(http.StatusOK)- }))- }- missing := upstream()- defer missing.Close()- unknown := upstream()- defer unknown.Close()-- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "missing", missing.URL, "mistral"))- disc.AddManual(nodeFor(t, "unknown", unknown.URL))- events := &prRec{}- p := NewProxy(NewCodec(events), disc, 1234)- p.SetSelected("missing")-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`)))-- if rec.Code != http.StatusBadGateway ||- !strings.Contains(rec.Body.String(), "no available node advertises the requested model") {- t.Fatalf("response = %d %s, want actionable local 502", rec.Code, rec.Body.String())- }- if hits != 0 {- t.Fatalf("ineligible upstreams received %d requests, want 0", hits)- }- if !events.has("no node advertises requested model") {- t.Fatalf("missing rejected request event: %s", events.b)- }-}--// TestResolveCandidates_SelfGuard: a node resolving to the proxy's own-// listen address is dropped so we never self-forward.-func TestResolveCandidates_SelfGuard(t *testing.T) {- disc := NewDiscovery()- disc.AddManual(Node{ID: "self", Addresses: []string{"127.0.0.1"}, Port: 11434})- disc.AddManual(Node{ID: "real", Addresses: []string{"192.0.2.10"}, Port: 11434})- p := testProxy(disc, 11434)-- cands := p.resolveCandidates("")- var haveReal bool- for _, c := range cands {- if c.id == "self" {- t.Errorf("self-target node must be excluded, got candidate %+v", c)- }- if c.id == "real" {- haveReal = true- }- }- if !haveReal {- t.Errorf("expected the real node to survive the self-guard, candidates = %+v", cands)- }-}diff --git a/services/lmstudio-proxy/go.mod b/services/lmstudio-proxy/go.moddeleted file mode 100644index ccc82a1..0000000--- a/services/lmstudio-proxy/go.mod+++ /dev/null@@ -1,19 +0,0 @@-module lmstudio-proxy--go 1.25.0--require nvpair-shared v0.0.0-00010101000000-000000000000--replace nvpair-shared => ../shared--require (- github.com/Microsoft/go-winio v0.6.2 // indirect- github.com/cenkalti/backoff v2.2.1+incompatible // indirect- github.com/grandcat/zeroconf v1.0.0 // indirect- github.com/miekg/dns v1.1.55 // indirect- golang.org/x/mod v0.17.0 // indirect- golang.org/x/net v0.58.0 // indirect- golang.org/x/sync v0.10.0 // indirect- golang.org/x/sys v0.47.0 // indirect- golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect-)diff --git a/services/lmstudio-proxy/ingress.go b/services/lmstudio-proxy/ingress.godeleted file mode 100644index 2b70e69..0000000--- a/services/lmstudio-proxy/ingress.go+++ /dev/null@@ -1,165 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "encoding/json"- "log/slog"- "net"- "net/http"- "net/http/httputil"- "net/url"- "strconv"-- "nvpair-shared/cors"-)--const engineIdentityProbeHeader = "X-NVPAIR-Engine-Identity-Probe"--// localBackend is the explicit loopback engine the cluster mTLS ingress-// forwards to. It is supplied by the broker over node/set-local-backend and is-// deliberately NOT sourced from the discovery overlay: a request that arrived-// over the LAN mTLS ingress can only ever be dumped on this node's own local-// engine, never re-routed to a peer, so the ingress path is strictly terminal-// and cannot recurse or amplify.-type localBackend struct {- Engine string `json:"engine"`- Host string `json:"host"`- Port int `json:"port"`- Healthy bool `json:"healthy"`-}--// setLocalBackend records (or, with a zero port / unhealthy flag, effectively-// clears) the local engine the ingress serves.-func (p *Proxy) setLocalBackend(b localBackend) {- p.backendMu.Lock()- p.backend = b- p.backendMu.Unlock()-}--// localBackendTarget returns the loopback URL of the current local engine, and-// false when none is set/healthy (the ingress then answers 503 rather than-// forwarding). The host defaults to 127.0.0.1 and is always loopback.-func (p *Proxy) localBackendTarget() (*url.URL, bool) {- p.backendMu.RLock()- b := p.backend- p.backendMu.RUnlock()- if b.Port <= 0 || !b.Healthy {- return nil, false- }- host := b.Host- if host == "" {- host = "127.0.0.1"- }- return &url.URL{Scheme: "http", Host: net.JoinHostPort(host, strconv.Itoa(b.Port))}, true-}--// handlePlain is the plaintext personality: it accepts requests only from-// loopback and hands them to the full local router (handleHTTP). A non-loopback-// caller — any LAN peer — is refused; peers must use the mTLS ingress. This is-// what closes the former open-relay exposure (the listener still binds all-// interfaces for the TLS personality, but plaintext is loopback-only).-func (p *Proxy) handlePlain(w http.ResponseWriter, r *http.Request) {- if !isLoopbackRemote(r.RemoteAddr) {- // Answer a non-loopback preflight ahead of the gate. It grants no access- // on its own; the request that follows still receives the real 403. A- // loopback preflight continues into handleHTTP so an available engine's- // exact origin and credentials policy can be preserved.- if cors.WritePreflight(w, r) {- return- }- slog.Warn("rejected non-loopback plaintext request; cluster peers must use mTLS",- "remote", r.RemoteAddr, "method", r.Method, "path", r.URL.Path)- writeIngressError(w, http.StatusForbidden, "loopback-only",- "plaintext requests are accepted only from loopback; cluster peers must use the mTLS ingress")- return- }- // Engine-manager marks identity/action requests so the federated model-list- // facade can never satisfy LM Studio's own /v1/models readiness probe.- if r.Header.Get(engineIdentityProbeHeader) == "1" {- writeIngressError(w, http.StatusConflict, "proxy-facade", "the compatibility facade is not an LM Studio engine")- return- }- p.handleHTTP(w, r)-}--// handleClusterIngress is the LAN mTLS personality: it authenticates the caller-// against this node's cluster pins and, once the peer is a trusted cluster-// member, forwards the request straight to the local loopback engine — exactly-// like the local plaintext path, with no route filtering. The mTLS pin is the-// sole authorization boundary (a trusted peer is treated like a local client),-// so the two personalities stay behaviorally identical toward the engine. It-// never calls resolveCandidates, so a peer request cannot be re-routed onward.-func (p *Proxy) handleClusterIngress(w http.ResponseWriter, r *http.Request) {- // Re-derive membership and pins per request so a cluster left, or a peer- // paired or removed, after startup is reflected immediately without a proxy- // restart — a removed peer must stop being accepted right away, which is the- // whole point of the gate.- p.mesh.Refresh()- peer, ok := p.mesh.VerifyClientPin(r)- if !ok {- writeIngressError(w, http.StatusForbidden, "cluster-auth",- "client certificate is not a pinned member of this node's cluster")- return- }- target, ok := p.localBackendTarget()- if !ok {- writeIngressError(w, http.StatusServiceUnavailable, "no-local-backend",- "no local inference backend is available on this node")- return- }- slog.Debug("cluster ingress forwarding to local backend",- "peer", peer, "method", r.Method, "path", r.URL.Path, "target", target.Host)- p.reverseProxyToLocal(w, r, target)-}--// reverseProxyToLocal streams the request to the local engine, preserving-// cancellation (the request context is the proxy's root context, so a client-// disconnect or shutdown tears down the upstream call and stops generation).-func (p *Proxy) reverseProxyToLocal(w http.ResponseWriter, r *http.Request, target *url.URL) {- p.newLocalReverseProxy(target).ServeHTTP(w, r)-}--func (p *Proxy) newLocalReverseProxy(target *url.URL) *httputil.ReverseProxy {- return &httputil.ReverseProxy{- Director: func(req *http.Request) {- req.URL.Scheme = target.Scheme- req.URL.Host = target.Host- req.Host = target.Host- },- Transport: p.plainHTTPTransport(),- ErrorHandler: func(ew http.ResponseWriter, _ *http.Request, err error) {- slog.Warn("cluster ingress upstream error", "target", target.Host, "err", err)- writeIngressError(ew, http.StatusBadGateway, "backend-error", "local inference backend error")- },- }-}--// isLoopbackRemote reports whether an http.Request RemoteAddr (host:port) is a-// loopback address (127.0.0.0/8 or ::1). An unparseable/empty RemoteAddr is not-// loopback, so it fails closed.-func isLoopbackRemote(remoteAddr string) bool {- host, _, err := net.SplitHostPort(remoteAddr)- if err != nil {- host = remoteAddr- }- ip := net.ParseIP(host)- return ip != nil && ip.IsLoopback()-}--// writeIngressError writes a small structured JSON error. It never echoes the-// request body or any generated output. CORS headers are included because these-// are the proxy's own rejections: without them a browser client cannot read the-// status or reason, and every one of them looks like a generic CORS failure.-func writeIngressError(w http.ResponseWriter, status int, code, msg string) {- cors.Apply(w.Header())- w.Header().Set("Content-Type", "application/json")- w.Header().Set("X-Content-Type-Options", "nosniff")- w.WriteHeader(status)- body, err := json.Marshal(map[string]string{"error": msg, "code": code})- if err != nil {- body = []byte(`{"error":"ingress error"}`)- }- _, _ = w.Write(body)-}diff --git a/services/lmstudio-proxy/ingress_test.go b/services/lmstudio-proxy/ingress_test.godeleted file mode 100644index 2f1e186..0000000--- a/services/lmstudio-proxy/ingress_test.go+++ /dev/null@@ -1,111 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "net/http"- "net/http/httptest"- "net/url"- "testing"-)--// TestResolveCandidatesUnclusteredDropsRelayPeers is the core isolation-// assertion: an unclustered node (nil mesh) must not route inference to-// relay-discovered peers, only to explicit user-added manual nodes.-func TestResolveCandidatesUnclusteredDropsRelayPeers(t *testing.T) {- disc := NewDiscovery()- disc.SetSubscribed([]Node{{- ID: "peer-a", Host: "peer-a", Port: 1234,- Addresses: []string{"192.0.2.10"},- IP: "192.0.2.10",- ClusterUUID: "cluster-uuid-a",- }})- disc.AddManual(Node{- ID: "manual-x", Host: "manual-x", Port: 1234,- Addresses: []string{"192.0.2.20"}, IP: "192.0.2.20",- })- p := testProxy(disc, 1235) // mesh nil => unclustered-- cands := p.resolveCandidates("")- if len(cands) != 1 {- t.Fatalf("unclustered candidate set = %+v, want exactly the manual node", cands)- }- if cands[0].id != "manual-x" || cands[0].peerUUID != "" || cands[0].url.Scheme != "http" {- t.Fatalf("unclustered candidate = %+v, want plaintext manual-x with no peerUUID", cands[0])- }-}--func TestHandlePlainRejectsNonLoopback(t *testing.T) {- p := testProxy(NewDiscovery(), 1235)- req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", nil)- req.RemoteAddr = "192.0.2.50:40000"- rec := httptest.NewRecorder()-- p.handlePlain(rec, req)- if rec.Code != http.StatusForbidden {- t.Fatalf("non-loopback plaintext status = %d, want %d", rec.Code, http.StatusForbidden)- }- // The refusal carries CORS so a browser client reads this 403 and its reason- // instead of an opaque "CORS error" that hides why the call failed.- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want * on the refusal", got)- }-}--// TestHandlePlainAnswersPreflightBeforeLoopbackGate: the preflight is answered-// even for a caller the gate will refuse. It authorizes nothing — the request-// that follows is still rejected — but without it the browser never sends that-// request and reports the refusal as a generic CORS failure.-func TestHandlePlainAnswersPreflightBeforeLoopbackGate(t *testing.T) {- p := testProxy(NewDiscovery(), 1235)- req := httptest.NewRequest(http.MethodOptions, "/v1/chat/completions", nil)- req.RemoteAddr = "192.0.2.50:40000"- rec := httptest.NewRecorder()-- p.handlePlain(rec, req)- if rec.Code != http.StatusNoContent {- t.Fatalf("preflight status = %d, want %d", rec.Code, http.StatusNoContent)- }- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want *", got)- }-}--func TestHandlePlainRejectsEngineIdentityProbe(t *testing.T) {- p := testProxy(NewDiscovery(), 1235)- req := httptest.NewRequest(http.MethodGet, "/v1/models", nil)- req.RemoteAddr = "127.0.0.1:40000"- req.Header.Set("X-NVPAIR-Engine-Identity-Probe", "1")- rec := httptest.NewRecorder()-- p.handlePlain(rec, req)- if rec.Code != http.StatusConflict {- t.Fatalf("identity probe status = %d, want %d", rec.Code, http.StatusConflict)- }-}--func TestHandleClusterIngressUnclusteredForbids(t *testing.T) {- p := testProxy(NewDiscovery(), 1235)- req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", nil)- rec := httptest.NewRecorder()-- p.handleClusterIngress(rec, req)- if rec.Code != http.StatusForbidden {- t.Fatalf("unclustered ingress status = %d, want %d", rec.Code, http.StatusForbidden)- }-}--func TestLocalReverseProxyUsesSharedPlainTransport(t *testing.T) {- p := testProxy(NewDiscovery(), 1235)- shared := p.plainHTTPTransport()- target := &url.URL{Scheme: "http", Host: "127.0.0.1:1"}- rp := p.newLocalReverseProxy(target)- tr, ok := rp.Transport.(*http.Transport)- if !ok {- t.Fatalf("Transport type = %T, want *http.Transport", rp.Transport)- }- if tr != shared {- t.Fatal("ingress reverse proxy did not use the shared plain Transport")- }-}diff --git a/services/lmstudio-proxy/portstore_test.go b/services/lmstudio-proxy/portstore_test.godeleted file mode 100644index 99993c7..0000000--- a/services/lmstudio-proxy/portstore_test.go+++ /dev/null@@ -1,145 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "bytes"- "context"- "fmt"- "net"- "os"- "strings"- "testing"- "time"-)--// redirectConfigDir points os.UserConfigDir() at a temp dir for the test, so-// proxy-port.json reads/writes don't touch the real per-user config. Sets all-// three env vars os.UserConfigDir() consults across platforms: XDG_CONFIG_HOME-// on Linux, $HOME/Library on macOS, and APPDATA on Windows. Missing APPDATA-// meant the Windows-first-class path read/wrote the real %AppData% file —-// clobbering the user's saved port and making the test fail on repeat runs.-func redirectConfigDir(t *testing.T) {- t.Helper()- dir := t.TempDir()- t.Setenv("XDG_CONFIG_HOME", dir)- t.Setenv("HOME", dir)- t.Setenv("APPDATA", dir)- t.Setenv("LOCALAPPDATA", dir)-}--func freeTCPPort(t *testing.T) int {- t.Helper()- l, err := net.Listen("tcp", "127.0.0.1:0")- if err != nil {- t.Fatalf("reserve free port: %v", err)- }- defer l.Close()- return l.Addr().(*net.TCPAddr).Port-}--func TestDefaultProxyPort(t *testing.T) {- if defaultProxyPort != 1234 {- t.Fatalf("default proxy port = %d, want 1234", defaultProxyPort)- }-}--func TestChooseStartupPort(t *testing.T) {- for _, tc := range []struct {- name string- flagPort, persisted int- ignorePersisted, hasPersisted bool- want int- }{- {"new default", 1234, 0, false, false, 1234},- {"legacy default migrates", 1234, 1235, false, true, 1234},- {"custom survives opt-out", 1234, 12400, false, true, 12400},- {"managed flag wins", 1234, 12400, true, true, 1234},- } {- t.Run(tc.name, func(t *testing.T) {- if got := chooseStartupPort(tc.flagPort, tc.ignorePersisted, tc.persisted, tc.hasPersisted); got != tc.want {- t.Fatalf("chooseStartupPort() = %d, want %d", got, tc.want)- }- })- }-}--func TestPersistedPortRoundTrip(t *testing.T) {- redirectConfigDir(t)-- if _, ok := loadPersistedPort(); ok {- t.Fatal("expected no persisted port before any save")- }- if err := savePersistedPort(11500); err != nil {- t.Fatalf("savePersistedPort: %v", err)- }- if p, ok := loadPersistedPort(); !ok || p != 11500 {- t.Errorf("round-trip: got %d ok=%v, want 11500", p, ok)- }-- // An out-of-range stored value is treated as "none" so startup falls- // back to the flag/default rather than trying to bind port 0.- path, err := proxyPortPath()- if err != nil {- t.Fatalf("proxyPortPath: %v", err)- }- if err := os.WriteFile(path, []byte(`{"port":0}`), 0o644); err != nil {- t.Fatal(err)- }- if _, ok := loadPersistedPort(); ok {- t.Error("port 0 should be treated as none")- }-}--// TestSetPortRebinds drives a live rebind: the proxy starts serving on one-// port, set-port moves it to another, and afterward the new port accepts-// connections, the old one doesn't, the choice is persisted, and a fresh-// ready notification carries the new port.-func TestSetPortRebinds(t *testing.T) {- redirectConfigDir(t)-- buf := &bytes.Buffer{}- codec := NewCodec(buf)- disc := NewDiscovery()-- portA := freeTCPPort(t)- proxy := NewProxy(codec, disc, portA)-- lnA, err := net.Listen("tcp", fmt.Sprintf(":%d", portA))- if err != nil {- t.Fatalf("listen on port A: %v", err)- }- ctx, cancel := context.WithCancel(context.Background())- defer cancel()- proxy.serveHTTP(ctx, lnA)- defer proxy.shutdown(context.Background())-- portB := freeTCPPort(t)- if err := proxy.setPort(portB); err != nil {- t.Fatalf("setPort: %v", err)- }-- // New port is now serving.- conn, err := net.DialTimeout("tcp", fmt.Sprintf("127.0.0.1:%d", portB), 2*time.Second)- if err != nil {- t.Fatalf("new port %d not listening after rebind: %v", portB, err)- }- conn.Close()-- // Old port stopped accepting.- if c, err := net.DialTimeout("tcp", fmt.Sprintf("127.0.0.1:%d", portA), 500*time.Millisecond); err == nil {- c.Close()- t.Errorf("old port %d should be closed after rebind", portA)- }-- // Persisted for next startup.- if p, ok := loadPersistedPort(); !ok || p != portB {- t.Errorf("persisted port: got %d ok=%v, want %d", p, ok, portB)- }-- // A fresh ready notification announced the new port.- if !strings.Contains(buf.String(), fmt.Sprintf("\"port\":%d", portB)) {- t.Errorf("expected ready notification carrying port %d, got %q", portB, buf.String())- }-}diff --git a/services/lmstudio-proxy/priority_test.go b/services/lmstudio-proxy/priority_test.godeleted file mode 100644index 6661b30..0000000--- a/services/lmstudio-proxy/priority_test.go+++ /dev/null@@ -1,177 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "encoding/json"- "io"- "strconv"- "sync"- "testing"-)--// prRec is a thread-safe io.ReadWriter that records codec writes so a test can-// assert the response emitted for a request. Reads hit EOF immediately.-type prRec struct {- mu sync.Mutex- b []byte-}--func (r *prRec) Read([]byte) (int, error) { return 0, io.EOF }--func (r *prRec) Write(p []byte) (int, error) {- r.mu.Lock()- defer r.mu.Unlock()- r.b = append(r.b, p...)- return len(p), nil-}--func (r *prRec) has(s string) bool {- r.mu.Lock()- defer r.mu.Unlock()- return string(r.b) != "" && contains(string(r.b), s)-}--func contains(hay, needle string) bool {- for i := 0; i+len(needle) <= len(hay); i++ {- if hay[i:i+len(needle)] == needle {- return true- }- }- return false-}--// prNode builds a discovery Node with a single non-local address so-// resolveCandidates resolves it deterministically (single candidate → no TCP-// probe) and it never trips the loopback rewrite or self-forward guard. The-// octet is derived from the id's first byte so each id gets a distinct valid IP.-func prNode(id string) Node {- return Node{ID: id, Addresses: []string{"192.0.2." + strconv.Itoa(int(id[0]))}, Port: 1234}-}--// prProxy returns a proxy whose discovery holds the given node ids.-func prProxy(t *testing.T, ids ...string) *Proxy {- t.Helper()- disc := NewDiscovery()- for _, id := range ids {- disc.AddManual(prNode(id))- }- return testProxy(disc, 1235)-}--func candidateIDs(p *Proxy) []string {- return candidateIDsForModel(p, "")-}--func candidateIDsForModel(p *Proxy, model string) []string {- cands := p.resolveCandidates(model)- out := make([]string, 0, len(cands))- for _, c := range cands {- out = append(out, c.id)- }- return out-}--func assertOrder(t *testing.T, got, want []string) {- t.Helper()- if len(got) != len(want) {- t.Fatalf("candidate order = %v, want %v", got, want)- }- for i := range want {- if got[i] != want[i] {- t.Fatalf("candidate order = %v, want %v", got, want)- }- }-}--// TestResolveCandidates_PriorityOrder: the priority list dictates auto order.-func TestResolveCandidates_PriorityOrder(t *testing.T) {- p := prProxy(t, "a", "b", "c")- p.SetPriority([]string{"c", "a", "b"})- assertOrder(t, candidateIDs(p), []string{"c", "a", "b"})-}--// TestResolveCandidates_UnlistedFallback: nodes absent from the priority list-// come last, in stable ID order.-func TestResolveCandidates_UnlistedFallback(t *testing.T) {- p := prProxy(t, "a", "b", "c")- p.SetPriority([]string{"b"})- assertOrder(t, candidateIDs(p), []string{"b", "a", "c"})-}--// TestResolveCandidates_UnknownIgnored: an id not in discovery is skipped.-func TestResolveCandidates_UnknownIgnored(t *testing.T) {- p := prProxy(t, "a", "b", "c")- p.SetPriority([]string{"zzz", "c"})- assertOrder(t, candidateIDs(p), []string{"c", "a", "b"})-}--// TestResolveCandidates_ManualPinOverridesPriority: an explicit node/select pin-// wins over the priority list; the rest follow priority order.-func TestResolveCandidates_ManualPinOverridesPriority(t *testing.T) {- p := prProxy(t, "a", "b", "c")- p.SetPriority([]string{"a", "b", "c"})- p.SetSelected("b")- assertOrder(t, candidateIDs(p), []string{"b", "a", "c"})-}--func TestResolveCandidates_FiltersBeforeSelectionAndPriority(t *testing.T) {- disc := NewDiscovery()- a, c, d := prNode("a"), prNode("c"), prNode("d")- a.Models, c.Models, d.Models = []string{"llama"}, []string{"llama"}, []string{"mistral"}- for _, n := range []Node{a, prNode("b"), c, d} {- disc.AddManual(n)- }- p := testProxy(disc, 1235)- p.SetSelected("d")- p.SetPriority([]string{"d", "b", "c", "a"})- assertOrder(t, candidateIDsForModel(p, "llama"), []string{"c", "a"})-- p.SetSelected("a")- assertOrder(t, candidateIDsForModel(p, "llama"), []string{"a", "c"})-}--// TestResolveCandidates_EmptyReverts: an empty priority list reverts to the-// default stable ID order.-func TestResolveCandidates_EmptyReverts(t *testing.T) {- p := prProxy(t, "c", "a", "b")- p.SetPriority([]string{"c", "a"})- p.SetPriority(nil) // clear- assertOrder(t, candidateIDs(p), []string{"a", "b", "c"})-}--// TestSetPriority_CountAndCopy: SetPriority returns the stored length and-// PriorityList hands back an independent copy.-func TestSetPriority_CountAndCopy(t *testing.T) {- p := prProxy(t, "a")- if n := p.SetPriority([]string{"a", "b", "c"}); n != 3 {- t.Fatalf("SetPriority count = %d, want 3", n)- }- got := p.PriorityList()- got[0] = "mutated"- if again := p.PriorityList(); again[0] != "a" {- t.Fatalf("PriorityList returned an aliased slice: %v", again)- }-}--// TestHandleSetPriority_Response: the node/set-priority request returns {count}.-func TestHandleSetPriority_Response(t *testing.T) {- rec := &prRec{}- p := NewProxy(NewCodec(rec), NewDiscovery(), 1235)-- id := json.RawMessage(`7`)- p.handleMessage(&Message{- JSONRPC: "2.0",- ID: &id,- Method: "node/set-priority",- Params: json.RawMessage(`{"nodes":["x","y"]}`),- })-- if !rec.has(`"count":2`) {- t.Fatalf("expected response with count=2, got: %s", rec.b)- }- if got := p.PriorityList(); len(got) != 2 || got[0] != "x" || got[1] != "y" {- t.Fatalf("stored priority = %v, want [x y]", got)- }-}diff --git a/services/lmstudio-proxy/proxy.go b/services/lmstudio-proxy/proxy.godeleted file mode 100644index 6e619e7..0000000--- a/services/lmstudio-proxy/proxy.go+++ /dev/null@@ -1,2043 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "bytes"- "context"- "crypto/rand"- "crypto/tls"- "encoding/hex"- "encoding/json"- stderrors "errors"- "fmt"- "io"- "log"- "log/slog"- "net"- "net/http"- "net/http/httputil"- "net/url"- "sort"- "strconv"- "sync"- "sync/atomic"- "time"-- "nvpair-shared/applog"- "nvpair-shared/clustertrust"- "nvpair-shared/cors"- "nvpair-shared/errors"- "nvpair-shared/netmon"- "nvpair-shared/netpick"- "nvpair-shared/nodeactivity"- "nvpair-shared/noderec"- "nvpair-shared/reach"- "nvpair-shared/schedulerwire"- "nvpair-shared/splitlisten"-)--// Version is stamped at build time via -ldflags "-X main.Version=...".-// See versions.json at the repo root for the source of truth.-var Version = "dev"--type ReadyParams struct {- Version string `json:"version"`- Port int `json:"port"`-}--// ErrorParams is sent as a JSON-RPC "error" notification when the-// proxy encounters a fatal startup-time condition it wants to surface-// to the orchestrator before exiting. Code is a short machine-readable-// tag ("bind-failed" today); Message is a human-friendly string suitable-// for an error bar.-type ErrorParams struct {- Code string `json:"code"`- Message string `json:"message"`- Port int `json:"port,omitempty"`-}--type NodesResult struct {- Nodes []Node `json:"nodes"`-}--type SelectParams struct {- ID string `json:"id"`-}--type SelectedResult struct {- ID string `json:"id"`-}--// RequestStartedEvent is emitted as a `proxy/request-started`-// notification the moment we've resolved a target node and are about-// to forward the request to it. Pairs with the existing-// `proxy/request` completion event by ID so the orchestrator can-// track in-flight requests per target — increment on start, decrement-// on the matching completion. Rejection-path requests (no active-// node) never get a started event because they were never in flight;-// they go straight to a completion event with an unmatched ID.-//-// NodeID is the chosen node's identifier in the discovery list. It's-// the authoritative way to attribute activity to a node card in the-// UI: Target (host:port) would be ambiguous whenever the proxy-// rewrote a local-interface address to 127.0.0.1 (see nodeURL), so-// multiple nodes could plausibly match the same Target string.-// Cluster model-list fan-out has no single node, so it reports an empty-// NodeID and the explicit Target "cluster".-type RequestStartedEvent struct {- ID string `json:"id"`- NodeID string `json:"node_id,omitempty"`- Method string `json:"method"`- Path string `json:"path"`- Target string `json:"target"`-}--// RequestEvent is emitted as a `proxy/request` notification when a-// proxied request finishes (or is rejected before forwarding). The-// ID is unique within one proxy process lifetime — paired with the-// matching RequestStartedEvent so consumers can pop it from an-// in-flight map. ID is always populated even on the rejection path-// (where no Started event was emitted) so consumers don't need a-// separate code path for ID-less completions.-//-// NodeID is empty on the rejection path (no target was resolved) and-// on cluster model-list fan-out, and is the chosen node's identifier otherwise. See RequestStartedEvent-// for why attribution by NodeID is needed instead of by Target.-//-// TTFB is the time-to-first-byte: milliseconds from the moment we-// started forwarding to the node until its HTTP response status line-// came back, captured via ReverseProxy.ModifyResponse. Omitted-// (serialized as absent rather than zero) when not applicable:-// rejection path (no forward happened) and upstream-error path-// (ModifyResponse is never called on connection/dial failures). This-// is the "is the node snappy?" signal — distinct from Duration,-// which for streaming LM Studio responses is dominated by token-// generation time and so doesn't really reflect latency at all.-type RequestEvent struct {- ID string `json:"id"`- NodeID string `json:"node_id,omitempty"`- Method string `json:"method"`- Path string `json:"path"`- Target string `json:"target"`- Status int `json:"status"`- Duration int64 `json:"duration_ms"`- TTFB int64 `json:"ttfb_ms,omitempty"`- Error string `json:"error,omitempty"`-}--// Workload lifecycle method names (workload-manager spec 7). The proxy is-// a workload *producer*: it emits one of these per forwarded inference-// request so the broker can stamp the origin (originatedFrom) and forward it to the-// workload-manager, which broadcasts it cluster-wide. We don't emit-// workload:submitted (the proxy never queues — it forwards immediately) or-// workloads:remove (retirement is a broker concern).-const (- workloadStartedMethod = "workload:started"- workloadCompletedMethod = "workload:completed"- workloadErroredMethod = "workload:errored"-- // workloadEngine is the opaque engine identifier carried in every- // workload this proxy produces. This proxy only ever fronts LM Studio.- workloadEngine = "lmstudio"-)--// inferenceEndpoints is the set of request paths that count as cluster-// workloads. Health checks, model listings (/v1/models), and other control-// traffic are deliberately excluded so we don't flood the cluster with-// non-inference noise. LM Studio serves the OpenAI-compatible API, so these-// are the OpenAI inference routes.-var inferenceEndpoints = map[string]bool{- "/v1/chat/completions": true,- "/v1/completions": true,- "/v1/embeddings": true,-}--// isInferenceRequest reports whether a request should be tracked as a-// workload — a POST to one of the known inference endpoints.-func isInferenceRequest(method, path string) bool {- return method == http.MethodPost && inferenceEndpoints[path]-}--// Workload mirrors the workload-manager spec 6 object. The proxy populates-// the fields it can observe: originatedFrom is intentionally left empty for-// the broker to stamp with the authoritative local node id (exactly like-// errors:report), while scheduledOn is set to the node this proxy actually-// routed the request to (the served candidate's node id — the same-// authoritative attribution handle), so a consumer can attribute the workload-// to where it ran rather than to where it came from. requesterId is omitted.-// Pointer fields serialize as JSON null when unset, matching the spec's-// nullable columns.-type Workload struct {- ID string `json:"id"`- Model string `json:"model"`- Engine string `json:"engine"`- RunID string `json:"runId"`- State string `json:"state"`- OriginatedFrom string `json:"originatedFrom"`- ScheduledOn string `json:"scheduledOn,omitempty"`- CreatedAt int64 `json:"createdAt"`- StartedAt *int64 `json:"startedAt"`- CompletedAt *int64 `json:"completedAt"`- Error *string `json:"error"`- RequesterID *string `json:"requesterId"`-}--// workloadParams is the params envelope for a workload:* notification-// (spec 7.1): a single workloadInfo carrying the full Workload.-type workloadParams struct {- WorkloadInfo Workload `json:"workloadInfo"`-}--// bufferBodyAndModel reads the request body once and returns the raw bytes-// (so each failover attempt can replay it — see the loop in handleHTTP) along-// with the JSON "model" field for workload tracking. Inference bodies are-// small (prompt + model), so full buffering is cheap. Returns (nil, "") when-// the body is absent and an empty model when none is parseable. The caller-// restores r.Body from the returned bytes before each forward attempt.-func bufferBodyAndModel(r *http.Request) ([]byte, string) {- if r.Body == nil {- return nil, ""- }- body, err := io.ReadAll(r.Body)- _ = r.Body.Close()- if err != nil {- return body, ""- }- var probe struct {- Model string `json:"model"`- }- if err := json.Unmarshal(body, &probe); err != nil {- return body, ""- }- return body, probe.Model-}--type statusCapture struct {- http.ResponseWriter- status int-- // idle bounds how long a single write of streamed bytes to the client may- // block before it's abandoned. Zero disables the deadline. See- // idleClientWriteTimeout for the rationale (killed-client / half-open- // socket zombie jobs).- idle time.Duration- rc *http.ResponseController- // wroteErr retains the first error returned when writing the response body- // to the client (e.g. a dead client's send buffer filling and the write- // deadline tripping), so handleHTTP can mark the workload failed rather- // than misreporting the truncated stream as completed.- wroteErr error-- // upstreamAlive is called after each successful body write, but only once the- // upstream has committed — handleHTTP sets it in ModifyResponse, so it stays- // nil while the only thing this writer could carry is the proxy's own error- // body. Past that point every byte written came from the node serving the- // request, which is proof that node is working: the liveness evidence- // discovery cannot obtain for itself while the node is too busy to answer a- // probe. Called on the reverse proxy's copy goroutine, so it must be cheap.- upstreamAlive func()-}--// Unwrap exposes the underlying ResponseWriter so http.ResponseController can-// reach the connection for SetWriteDeadline (and Flush) through this wrapper.-func (sc *statusCapture) Unwrap() http.ResponseWriter { return sc.ResponseWriter }--func (sc *statusCapture) WriteHeader(code int) {- sc.status = code- sc.ResponseWriter.WriteHeader(code)-}--// Write bounds each streamed write to the client with a deadline so a write-// blocked on a dead/half-open client fails promptly instead of hanging the-// reverse-proxy copy indefinitely. The deadline is cleared after every-// successful write, so a legitimately slow generation with long gaps between-// tokens is never penalized — only a write actively stuck on a gone client-// trips it. The first write error is retained (wroteErr) for the caller.-func (sc *statusCapture) Write(b []byte) (int, error) {- if sc.idle > 0 {- if sc.rc == nil {- sc.rc = http.NewResponseController(sc.ResponseWriter)- }- _ = sc.rc.SetWriteDeadline(time.Now().Add(sc.idle))- }- n, err := sc.ResponseWriter.Write(b)- if err != nil {- if sc.wroteErr == nil {- sc.wroteErr = err- }- } else if sc.rc != nil {- _ = sc.rc.SetWriteDeadline(time.Time{})- }- // Reported on every chunk rather than once per response so a long generation- // keeps vouching for its node for as long as it streams. The reporter- // coalesces, so the cost of calling this per chunk is a mutex and a clock- // read.- if err == nil && sc.upstreamAlive != nil {- sc.upstreamAlive()- }- return n, err-}--// FlushError makes the streamed flush deadline-aware. For a streaming-// (chunked) upstream, ReverseProxy flushes after every write via-// http.NewResponseController(w).Flush — and because a small chunk buffers on-// Write without touching the socket, the actual network write for it happens-// here in Flush, not in Write. Without this method that flush reaches the-// underlying connection through Unwrap with no deadline and blocks unbounded on-// a stalled client (the same zombie the Write deadline guards against). So arm-// the same idle deadline around the flush, clear it on success, and retain a-// real flush error so the workload is classified failed. Implementing-// FlushError (which also satisfies the Flusher path via the ResponseController)-// means the flush routes through here instead of unwrapping past us.-func (sc *statusCapture) FlushError() error {- if sc.rc == nil {- sc.rc = http.NewResponseController(sc.ResponseWriter)- }- if sc.idle > 0 {- _ = sc.rc.SetWriteDeadline(time.Now().Add(sc.idle))- }- err := sc.rc.Flush()- if err != nil {- // A ResponseWriter that genuinely can't flush is not a client failure;- // only retain real I/O errors (e.g. the deadline tripping on a dead- // client) so we don't misreport an unsupported-flush as a failed write.- if !stderrors.Is(err, http.ErrNotSupported) && sc.wroteErr == nil {- sc.wroteErr = err- }- return err- }- if sc.idle > 0 {- _ = sc.rc.SetWriteDeadline(time.Time{})- }- return nil-}--type Proxy struct {- codec *Codec- discovery *Discovery- cancel context.CancelFunc-- // httpMu guards port, the servers, the split listener, and ln across a- // live set-port rebind. The HTTP handlers never read port (they route by- // upstream node), so the only contention is set-port vs set-port- // (serialized) and the initial serveHTTP store vs a later rebind.- httpMu sync.Mutex- port int- plainSrv *http.Server- tlsSrv *http.Server- split *splitlisten.Splitter- ln net.Listener-- // mesh is this node's cluster mTLS trust fabric, loaded from --cluster-dir.- // nil = unclustered: the LAN TLS ingress accepts nothing and the node does- // only loopback-plaintext local routing. Read-only after startup.- mesh *clustertrust.Mesh-- // backendMu guards backend, the explicit loopback engine the cluster mTLS- // ingress forwards to. The broker sets/clears it via node/set-local-backend;- // it is never sourced from discovery, so an ingress request can only ever- // reach this node's own local engine and can never be re-routed to a peer.- backendMu sync.RWMutex- backend localBackend-- selectedMu sync.RWMutex- selectedID string-- // activity coalesces the liveness reports raised when a peer's engine streams- // response bytes back through us (see reportActivity).- activity *nodeactivity.Reporter-- // priorityMu guards the scheduler's authoritative baseline and the- // optimistic reservations made since that snapshot arrived. resolveCandidates- // reads priority to form the failover list; reserveCandidate atomically adds- // local dispatches before forwarding so a concurrent burst cannot repeatedly- // choose from the same stale scheduler state.- priorityMu sync.RWMutex- priority []string- priorityPending map[string]int- priorityGPUPressure map[string]int- priorityReservations map[string]int-- // targets remembers, per node, which of its published addresses accepted a- // connection, so a repeated forward costs no confirmation. An entry is- // re-confirmed when the node's candidate list changes and forgotten on an- // upstream error, so the next request fails over to another address.- targets *reach.Chooser-- // transportMu guards the long-lived HTTP transports reused across forwards- // and model-list fetches. Allocating a new http.Transport per request- // defeats connection pooling and leaks idle sockets until GC.- transportMu sync.Mutex- plainTransport *http.Transport- peerTransports map[string]*http.Transport-- // nextRequestID is a monotonic counter for tagging RequestStarted /- // RequestEvent pairs. Atomic add returns the new value, so request- // IDs start at 1 and never collide within a single proxy lifetime.- // IDs deliberately reset across restarts — they're only meaningful- // while the orchestrator's in-flight map is also alive, and a- // fresh proxy session always starts that map empty on the- // orchestrator side via the proxy:stopped → proxy:ready event- // pair.- nextRequestID atomic.Uint64-- // runID is a per-process nonce minted at startup and stamped on every- // workload this proxy emits. It makes a workload's identity- // (originatedFrom, engine, runId, id) globally unique even though- // nextRequestID resets to 1 on restart and the Ollama proxy also counts- // from 1 — without it, two concurrent cross-engine jobs, or a reused id- // after a restart, would collide in the broker's store.- runID string-}--func NewProxy(codec *Codec, discovery *Discovery, port int) *Proxy {- return &Proxy{- codec: codec,- discovery: discovery,- port: port,- targets: reach.NewChooser(),- runID: newRunID(),- activity: nodeactivity.NewReporter(activityReportInterval),- }-}--// activityReportInterval is how often a single node's streaming may raise a-// liveness report. A generation writes hundreds of chunks and the scanner treats-// a report as good for a minute, so anything finer is pure noise on the broker-// pipe.-const activityReportInterval = 2 * time.Second--// reportActivity tells the broker a node's engine just returned response bytes,-// so discovery can keep that node even while it is too busy to answer a liveness-// probe. This is the only liveness signal that strengthens under load, which is-// exactly when the probe-based ones fail.-//-// Reports are not filtered to remote nodes here: this proxy knows targets by URL-// and port, not by whether a uuid is its own. The scanner holds that identity and-// drops its own (see noteActivity).-func (p *Proxy) reportActivity(nodeID string) {- if !p.activity.Due(nodeID) {- return- }- if err := p.codec.Notify(noderec.NotifyNodeActivity, noderec.NodeActivityParams{HostUUID: nodeID}); err != nil {- slog.Debug("failed to report node activity", "node_id", nodeID, "err", err)- }-}--// newRunID returns a short random per-process nonce (hex). A crypto/rand read-// failure falls back to a timestamp — uniqueness matters more than-// unpredictability here.-func newRunID() string {- var b [8]byte- if _, err := rand.Read(b[:]); err != nil {- return strconv.FormatInt(time.Now().UnixNano(), 16)- }- return hex.EncodeToString(b[:])-}--func (p *Proxy) Run(ctx context.Context) error {- ctx, cancel := context.WithCancel(ctx)- p.cancel = cancel- defer cancel()-- // Keep the "is this address local?" set fresh as interfaces come and go,- // so the loopback rewrite in nodeURL stays correct after VPN/dock changes- // or a sleep/wake IP reassignment.- startLocalAddrWatch(ctx)-- // Bind synchronously before announcing "ready": if the port is- // already in use, we want the UI to see a real error reason- // instead of being stuck on "Proxy running" while ListenAndServe- // silently fails in a goroutine.- ln, err := p.listen()- if err != nil {- // Best-effort: notify the orchestrator with a structured- // reason. The process will exit non-zero regardless (main.go- // log.Fatalf's on a non-nil Run error), so a failed Notify- // here is not worth surfacing separately.- _ = p.codec.Notify("error", ErrorParams{- Code: "bind-failed",- Message: fmt.Sprintf("failed to bind port %d: %v", p.port, err),- Port: p.port,- })- return fmt.Errorf("failed to bind port %d: %w", p.port, err)- }-- if err := p.codec.Notify("ready", ReadyParams{- Version: Version,- Port: p.port,- }); err != nil {- // Don't leave a dangling listener holding the port if we- // couldn't even tell the orchestrator about it.- _ = ln.Close()- return fmt.Errorf("failed to send ready notification: %w", err)- }-- // Routing targets come from the broker's discovery relay. Subscribe- // for lm nodes; they arrive as discovery:nodes snapshots (handled in- // handleMessage), each replacing the subscribed overlay. Non-fatal: if the- // parent isn't a relay-aware broker the proxy still routes to manual nodes.- slog.Debug("subscribing to discovery relay for routing targets", "service", string(noderec.ServiceLMStudio))- if err := p.codec.Notify(noderec.MethodSubscribe, noderec.SubscribeParams{Services: []noderec.ServiceKey{noderec.ServiceLMStudio}}); err != nil {- slog.Warn("failed to subscribe to discovery relay", "err", err)- }-- p.serveHTTP(ctx, ln)-- err = p.readLoop(ctx)-- // The app is going away (stdin closed or ctx cancelled). Stop any- // inference requests still in flight rather than letting them run to- // completion: cancelling the proxy's root context propagates to every- // in-flight request context — and thus the upstream reverse-proxy- // connection — so the target LM Studio sees the client disconnect and stops- // generating instead of burning the GPU on a result nobody will read.- cancel()-- // srv.Shutdown then waits for the handlers to unwind (now fast, since- // their upstream calls were just cancelled). As each returns it emits its- // own terminal workload:errored, so peers don't keep showing the workload- // as a "running" ghost. A hard kill (SIGKILL) bypasses all of this.- shutCtx, shutCancel := context.WithTimeout(context.Background(), 3*time.Second)- defer shutCancel()- p.shutdown(shutCtx)-- return err-}--// Timeouts for upstream connections. Logged at startup so they're always-// present in any captured log for post-mortem analysis.-const (- proxyDialTimeout = 10 * time.Second- proxyKeepAlive = 30 * time.Second- proxyResponseTimeout = 120 * time.Second- proxyMaxIdleConns = 50- proxyIdleConnTimeout = 90 * time.Second- // Inbound http.Server limits — keep IdleTimeout aligned with client- // IdleConnTimeout so idle keep-alives are reaped on both sides.- proxyReadHeaderTimeout = 10 * time.Second- proxyServerIdleTimeout = 90 * time.Second- maxModelListBytes = 16 << 20-)--// idleClientWriteTimeout bounds how long a single write of streamed response-// bytes to the client may block. A killed client can leave a half-open socket-// whose kernel send buffer fills and never drains; without this deadline the-// reverse-proxy copy blocks indefinitely (TCP retransmit backoff runs into-// minutes, and r.Context() never fires when no FIN/RST arrives), so the request-// handler never returns and its terminal workload event is never emitted — the-// "zombie job" left showing as running until PAIR restarts. statusCapture.Write-// resets the deadline after every successful write, so this only trips a write-// that is actively stuck on a gone client, never a slow-but-live generation.-//-// It is a var (not a const) only so a test can shorten it to exercise the-// deadline against a real socket; production never reassigns it.-var idleClientWriteTimeout = 30 * time.Second--var modelListClient = &http.Client{- Transport: &http.Transport{- DialContext: (&net.Dialer{- Timeout: proxyDialTimeout,- KeepAlive: proxyKeepAlive,- }).DialContext,- ResponseHeaderTimeout: proxyDialTimeout,- MaxIdleConns: proxyMaxIdleConns,- IdleConnTimeout: proxyIdleConnTimeout,- },- Timeout: proxyDialTimeout,-}--// listen binds the proxy's TCP listener synchronously so bind failures-// (EADDRINUSE and friends) can be reported through a structured error-// notification before the process exits. The caller is responsible for-// closing the returned listener if it doesn't hand it to serveHTTP.-func (p *Proxy) listen() (net.Listener, error) {- return net.Listen("tcp", fmt.Sprintf(":%d", p.port))-}--// serveHTTP takes the already-bound base listener and drives the two proxy-// personalities over it: a plaintext HTTP server (loopback-only, full local-// router) and a LAN mTLS ingress (pin-gated, forwards to the local engine),-// split by the connection's first byte via nvpair-shared/splitlisten. The two-// http.Servers are recorded so set-port can rebind both onto a fresh split-// without tearing the servers down.-func (p *Proxy) serveHTTP(ctx context.Context, ln net.Listener) {- base := func(_ net.Listener) context.Context { return ctx }- plainSrv := &http.Server{- Handler: http.HandlerFunc(p.handlePlain),- BaseContext: base,- ReadHeaderTimeout: proxyReadHeaderTimeout,- IdleTimeout: proxyServerIdleTimeout,- }- tlsSrv := &http.Server{- Handler: http.HandlerFunc(p.handleClusterIngress),- BaseContext: base,- ReadHeaderTimeout: proxyReadHeaderTimeout,- IdleTimeout: proxyServerIdleTimeout,- }-- p.httpMu.Lock()- p.plainSrv = plainSrv- p.tlsSrv = tlsSrv- p.ln = ln- p.startSplitLocked(ln)- p.httpMu.Unlock()-- slog.Info("proxy timeouts configured",- "dial_timeout", proxyDialTimeout,- "keep_alive", proxyKeepAlive,- "response_header_timeout", proxyResponseTimeout,- "max_idle_conns", proxyMaxIdleConns,- "idle_conn_timeout", proxyIdleConnTimeout,- )- slog.Info("HTTP proxy listening", "port", p.port, "addr", ln.Addr().String(),- "cluster_ingress", p.mesh.Clustered())-}--// startSplitLocked wraps base in a first-byte splitter and starts both servers-// on its sub-listeners. Caller holds httpMu. Reuses the persistent plainSrv /-// tlsSrv so set-port can call it repeatedly on fresh listeners.-func (p *Proxy) startSplitLocked(base net.Listener) {- split := splitlisten.New(base)- p.split = split- go func() {- if err := p.plainSrv.Serve(split.Plain()); err != nil && err != http.ErrServerClosed {- slog.Error("plaintext HTTP server exited", "err", err)- }- }()- go p.serveTLS(split.TLS())-}--// serveTLS terminates cluster mTLS on the split's TLS sub-listener. The server-// certificate is resolved per handshake from the live mesh, so this one-// sub-listener covers both states: while this node is unclustered there is no-// leaf to present and the handshake is refused (it exposes no LAN inference-// surface), and the moment the node becomes a member the same sub-listener-// serves the pin-gated ingress — no rebind, and no process restart to pick up a-// freshly-minted identity.-func (p *Proxy) serveTLS(l net.Listener) {- if err := p.tlsSrv.Serve(tls.NewListener(l, p.mesh.ServerTLSConfig())); err != nil && err != http.ErrServerClosed {- slog.Error("cluster mTLS ingress exited", "err", err)- }-}--// shutdown gracefully stops both personalities and closes the split listener.-func (p *Proxy) shutdown(ctx context.Context) {- p.httpMu.Lock()- plainSrv, tlsSrv, split := p.plainSrv, p.tlsSrv, p.split- p.httpMu.Unlock()- if plainSrv != nil {- _ = plainSrv.Shutdown(ctx)- }- if tlsSrv != nil {- _ = tlsSrv.Shutdown(ctx)- }- if split != nil {- _ = split.Close()- }- p.closeIdleTransports()-}--// setPort live-rebinds the HTTP listener onto newPort and persists the choice-// so it survives a restart. It binds the new listener first (so a bind-// failure leaves the current one serving), starts the same server on it, then-// closes the old listener — in-flight connections on the old port drain-// naturally. A fresh `ready` notification announces the new port so the-// orchestrator/UI learn where the proxy is now listening.-func (p *Proxy) setPort(newPort int) error {- p.httpMu.Lock()- defer p.httpMu.Unlock()-- if newPort == p.port {- return nil- }- newLn, err := net.Listen("tcp", fmt.Sprintf(":%d", newPort))- if err != nil {- return fmt.Errorf("failed to bind port %d: %w", newPort, err)- }- oldSplit := p.split- p.ln = newLn- p.port = newPort-- // Re-serve both personalities on a fresh split over the new listener, then- // close the old split (and its base listener) so in-flight connections on- // the old port drain naturally. The plaintext and mTLS personalities always- // move together as one unit.- slog.Info("HTTP proxy listening", "port", newPort, "addr", newLn.Addr().String(),- "cluster_ingress", p.mesh.Clustered())- p.startSplitLocked(newLn)- if oldSplit != nil {- _ = oldSplit.Close()- }-- if err := savePersistedPort(newPort); err != nil {- slog.Warn("failed to persist proxy port", "port", newPort, "err", err)- }- if err := p.codec.Notify("ready", ReadyParams{Version: Version, Port: newPort}); err != nil {- slog.Warn("failed to emit ready after rebind", "err", err)- }- return nil-}--// emitWorkload sends a workload:* lifecycle notification to the-// orchestrator. The broker stamps the origin (originatedFrom) and forwards it to the-// workload-manager; a failed write is logged but never blocks the request.-func (p *Proxy) emitWorkload(method string, w Workload) {- if err := p.codec.Notify(method, workloadParams{WorkloadInfo: w}); err != nil {- slog.Warn("failed to emit workload notification", "method", method, "err", err)- }-}--// candidate is one forwarding target: the node's discovery ID (the-// authoritative attribution handle, stable across nodeURL's 127.0.0.1-// rewrite) and its resolved URL. peerUUID is set for a remote cluster peer:-// the request is dialed over cluster mTLS to the peer's promoted proxy (https),-// pinned to that peer's exact server cert. Empty peerUUID means a plain-HTTP-// dial — the local backend (self) or an explicit manual node.-type candidate struct {- id string- url *url.URL- peerUUID string-}--// candidateTransport returns the reverse-proxy / model-list transport for a-// candidate. Plain/self/manual candidates share one long-lived Transport.-// Cluster peers share one long-lived mTLS Transport per peerUUID. Callers must-// not CloseIdleConnections on the returned value except via closeIdleTransports.-func (p *Proxy) candidateTransport(c candidate) *http.Transport {- if c.peerUUID == "" {- return p.plainHTTPTransport()- }- return p.peerHTTPTransport(c.peerUUID)-}--func newProxyTransport(tlsCfg *tls.Config) *http.Transport {- tr := &http.Transport{- DialContext: (&net.Dialer{- Timeout: proxyDialTimeout,- KeepAlive: proxyKeepAlive,- }).DialContext,- ResponseHeaderTimeout: proxyResponseTimeout,- MaxIdleConns: proxyMaxIdleConns,- MaxIdleConnsPerHost: proxyMaxIdleConns,- IdleConnTimeout: proxyIdleConnTimeout,- }- if tlsCfg != nil {- tr.TLSClientConfig = tlsCfg- }- return tr-}--func (p *Proxy) plainHTTPTransport() *http.Transport {- p.transportMu.Lock()- defer p.transportMu.Unlock()- if p.plainTransport == nil {- p.plainTransport = newProxyTransport(nil)- }- return p.plainTransport-}--func (p *Proxy) peerHTTPTransport(peerUUID string) *http.Transport {- p.transportMu.Lock()- defer p.transportMu.Unlock()- if tr, ok := p.peerTransports[peerUUID]; ok {- if p.mesh != nil && p.mesh.HasPin(peerUUID) {- return tr- }- tr.CloseIdleConnections()- delete(p.peerTransports, peerUUID)- }- if p.mesh == nil {- return newProxyTransport(nil)- }- cfg, ok := p.mesh.ClientTLSConfig(peerUUID)- if !ok {- return newProxyTransport(nil)- }- tr := newProxyTransport(cfg)- if p.peerTransports == nil {- p.peerTransports = make(map[string]*http.Transport)- }- p.peerTransports[peerUUID] = tr- return tr-}--// dropUnpinnedPeerTransports closes idle conns for peer Transports whose pins-// are gone. Safe to call from the mesh Watch callback.-func (p *Proxy) dropUnpinnedPeerTransports() {- p.transportMu.Lock()- defer p.transportMu.Unlock()- for uuid, tr := range p.peerTransports {- if p.mesh != nil && p.mesh.HasPin(uuid) {- continue- }- tr.CloseIdleConnections()- delete(p.peerTransports, uuid)- }-}--func (p *Proxy) closeIdleTransports() {- p.transportMu.Lock()- defer p.transportMu.Unlock()- if p.plainTransport != nil {- p.plainTransport.CloseIdleConnections()- }- for uuid, tr := range p.peerTransports {- tr.CloseIdleConnections()- delete(p.peerTransports, uuid)- }-}--// retrySignal is returned from ModifyResponse to abort a retryable upstream-// response before its body streams to the client, so handleHTTP can fail over-// to the next candidate. It's a distinct type rather than errors.New(...)-// because this package aliases nvpair-shared/errors as `errors` (which has no New).-type retrySignal struct{}--func (retrySignal) Error() string { return "lmstudio-proxy: retry next candidate" }--type modelListItem struct {- key string- raw json.RawMessage-}--type modelListResult struct {- items []modelListItem- ok bool- err error-}--// serveModelList queries every LM Studio candidate concurrently and returns-// the native /v1/models envelope with duplicate IDs removed. Results are-// merged in candidate order, not completion order, so duplicate metadata is-// deterministic while an unavailable peer cannot hide healthy inventories.-func (p *Proxy) serveModelList(w http.ResponseWriter, r *http.Request, candidates []candidate) (int, error) {- writeJSON := func(status int, body []byte) {- cors.Apply(w.Header())- w.Header().Set("Content-Type", "application/json")- w.Header().Set("X-Content-Type-Options", "nosniff")- w.WriteHeader(status)- _, _ = w.Write(body)- }- results := make([]modelListResult, len(candidates))- var wg sync.WaitGroup- for i, cand := range candidates {- target := *cand.url- target.Path = r.URL.Path- target.RawPath = r.URL.RawPath- target.RawQuery = r.URL.RawQuery- upstream, err := http.NewRequestWithContext(r.Context(), http.MethodGet, target.String(), nil)- if err != nil {- results[i].err = err- continue- }- upstream.Header.Set("Accept", "application/json")-- // A cluster-peer candidate is queried over mTLS to its promoted proxy;- // self/manual candidates use the shared plain client.- client := modelListClient- if cand.peerUUID != "" {- client = &http.Client{Timeout: modelListClient.Timeout, Transport: p.candidateTransport(cand)}- }-- wg.Add(1)- go func(i int, cand candidate, req *http.Request, client *http.Client) {- defer wg.Done()- resp, err := client.Do(req)- if err != nil {- p.targets.Forget(cand.id)- results[i].err = err- return- }- defer resp.Body.Close()- if resp.StatusCode != http.StatusOK {- results[i].err = fmt.Errorf("upstream returned %s", resp.Status)- return- }- body, err := io.ReadAll(io.LimitReader(resp.Body, maxModelListBytes+1))- if err != nil {- results[i].err = err- return- }- if len(body) > maxModelListBytes {- results[i].err = fmt.Errorf("model list exceeds %d bytes", maxModelListBytes)- return- }- var envelope struct {- Data *[]json.RawMessage `json:"data"`- }- if err := json.Unmarshal(body, &envelope); err != nil {- results[i].err = err- return- }- if envelope.Data == nil {- results[i].err = fmt.Errorf("upstream response has no data array")- return- }- models := *envelope.Data- items := make([]modelListItem, 0, len(models))- for _, raw := range models {- var identity struct {- ID string `json:"id"`- }- if err := json.Unmarshal(raw, &identity); err != nil {- results[i].err = fmt.Errorf("invalid model record: %w", err)- return- }- if identity.ID == "" {- results[i].err = fmt.Errorf("model record has no id")- return- }- items = append(items, modelListItem{key: identity.ID, raw: raw})- }- results[i] = modelListResult{items: items, ok: true}- }(i, cand, upstream, client)- }- wg.Wait()-- success := false- models := make([]json.RawMessage, 0)- seen := make(map[string]bool)- for i, result := range results {- if !result.ok {- slog.Debug("model list candidate unavailable",- "node_id", candidates[i].id, "target", candidates[i].url.Host, "err", result.err)- continue- }- success = true- for _, item := range result.items {- if !seen[item.key] {- seen[item.key] = true- models = append(models, item.raw)- }- }- }- if !success {- err := fmt.Errorf("no valid model list from %d candidate(s)", len(candidates))- writeJSON(http.StatusServiceUnavailable, []byte(`{"error":"model inventory unavailable"}`))- return http.StatusServiceUnavailable, err- }- body, err := json.Marshal(struct {- Object string `json:"object"`- Data []json.RawMessage `json:"data"`- }{Object: "list", Data: models})- if err != nil {- writeJSON(http.StatusInternalServerError, []byte(`{"error":"failed to encode model inventory"}`))- return http.StatusInternalServerError, err- }- writeJSON(http.StatusOK, body)- return http.StatusOK, nil-}--func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) {- start := time.Now()-- // Allocate the request ID up front so both code paths (rejection- // and forward) can stamp the same value into their notification.- // The rejection path never emits a Started event, so its ID won't- // appear in any orchestrator in-flight map — that's fine; the- // completion event still bumps the failed counter regardless of- // whether a matching Started was seen.- reqID := strconv.FormatUint(p.nextRequestID.Add(1), 10)-- // Parse the request's model before choosing a node. Model eligibility only- // applies to inference routes; control endpoints retain their existing- // routing behavior even when their JSON happens to contain a model field.- bodyBytes, model := bufferBodyAndModel(r)- isInf := isInferenceRequest(r.Method, r.URL.Path)- routingModel := ""- if isInf {- routingModel = model- }- candidates := p.resolveCandidates(routingModel)- if isInf && model != "" {- candidates = p.reserveCandidate(candidates)- }- if r.Method == http.MethodGet && r.URL.Path == "/v1/models" {- if len(candidates) > 0 {- p.codec.Notify("proxy/request-started", RequestStartedEvent{- ID: reqID, Method: r.Method, Path: r.URL.Path, Target: "cluster",- })- }- status, err := p.serveModelList(w, r, candidates)- errText := ""- if err != nil {- errText = err.Error()- }- p.codec.Notify("proxy/request", RequestEvent{- ID: reqID, Method: r.Method, Path: r.URL.Path, Target: "cluster",- Status: status, Duration: time.Since(start).Milliseconds(), Error: errText,- })- return- }- if len(candidates) == 0 {- // With no engine to consult, retain the local permissive preflight used- // for engines that do not publish a CORS policy.- if cors.WritePreflight(w, r) {- return- }- cors.Apply(w.Header())- rejectionBody := `{"error":"no active node selected or available"}`- rejectionError := "no active node"- if isInf && model != "" {- rejectionBody = `{"error":"no available node advertises the requested model"}`- rejectionError = "no node advertises requested model"- }- slog.Warn("proxy request rejected",- "id", reqID, "method", r.Method, "path", r.URL.Path,- "remote", r.RemoteAddr, "reason", rejectionError)- http.Error(w, rejectionBody, http.StatusBadGateway)- p.codec.Notify("proxy/request", RequestEvent{- ID: reqID,- Method: r.Method,- Path: r.URL.Path,- Status: http.StatusBadGateway,- Duration: time.Since(start).Milliseconds(),- Error: rejectionError,- })- return- }- // shouldRetry reports whether an upstream status warrants failing over to- // the next candidate: busy/unavailable/gateway statuses, plus a 404 on an- // inference call (an advertised owner's inventory may have become stale).- // Genuine client errors (400/401/422…) are not retried — they'd fail- // identically on every node.- shouldRetry := func(code int) bool {- switch code {- case http.StatusRequestTimeout,- http.StatusTooManyRequests,- http.StatusBadGateway,- http.StatusServiceUnavailable,- http.StatusGatewayTimeout:- return true- case http.StatusNotFound:- return isInf- }- return code >= 500- }-- var (- servedNodeID string- servedTarget string- ttfbMs int64- proxyErr string- finalStatus int- started bool- wl *Workload- )-- // Emit workload:started up front, the moment we begin forwarding, naming- // the first candidate we'll try. A burst of concurrent inference requests- // must surface as job cards immediately; the upstream engine serializes- // concurrent requests on a single GPU slot, so gating "started" on the- // upstream response headers (the commit point) left every queued-but-- // forwarded job invisible until the node dequeued it — only one card at a- // time (a prior regression). If failover later commits a different- // node, the commit block re-points scheduledOn; the terminal- // completed/errored transition is emitted once at the end regardless.- if isInf && model != "" {- createdMs := start.UnixMilli()- wl = &Workload{- ID: reqID,- Model: model,- Engine: workloadEngine,- RunID: p.runID,- State: "running",- ScheduledOn: candidates[0].id,- CreatedAt: createdMs,- StartedAt: &createdMs,- }- p.emitWorkload(workloadStartedMethod, *wl)- }-- // The terminal workload transition (completed/errored) can be reached from- // two places: the normal path after the stream copy unwinds below, and the- // disconnect watcher that fires while the copy is still blocked. terminalOnce- // guarantees exactly one is emitted; wlMu guards the shared wl fields the- // watcher (a separate goroutine) and ModifyResponse's failover re-point both- // touch; terminated suppresses a late started re-point once we've finalized.- var (- terminalOnce sync.Once- wlMu sync.Mutex- terminated bool- )- emitTerminal := func(state, errMsg string) {- if wl == nil {- return- }- terminalOnce.Do(func() {- now := time.Now().UnixMilli()- wlMu.Lock()- terminated = true- wl.CompletedAt = &now- wl.State = state- if errMsg != "" {- wl.Error = &errMsg- }- snapshot := *wl- wlMu.Unlock()- method := workloadCompletedMethod- if state != "completed" {- method = workloadErroredMethod- }- p.emitWorkload(method, snapshot)- })- }-- // Watch for the client going away while the request is in flight. The- // terminal event is otherwise emitted only after the stream copy returns;- // a client that disconnects mid-stream can leave the copy blocked, so we- // emit the terminal here the moment r.Context() is cancelled instead of- // waiting for the unwind. Cancelling r.Context() (client close, or our own- // shutdown) also propagates to the ReverseProxy's upstream request, so the- // engine stops generating. terminalOnce keeps this from double-emitting- // with the normal path. The half-open case (no FIN, r.Context() never- // fires) is caught instead by statusCapture's write deadline below.- if wl != nil {- reqCtx := r.Context()- finished := make(chan struct{})- defer close(finished)- go func() {- select {- case <-reqCtx.Done():- emitTerminal("failed", "client disconnected before completion")- case <-finished:- }- }()- }-- // committedSC is the statusCapture of the candidate we committed to- // streaming; its wroteErr tells us after the fact whether the client write- // failed (dead/half-open client) so we can mark the workload failed.- var committedSC *statusCapture-- // Failover loop: try candidates in order until one returns a- // usable response or the list is exhausted. We can only retry before the- // first byte reaches the client; once a response starts streaming we're- // committed. proxy/request-started fires at that commit point so it names- // the node that actually serves the request, not one we failed over from;- // workload:started was already emitted above (and is re-pointed there on a- // failover). The self-forward guard lives in resolveCandidates.- for i := range candidates {- cand := candidates[i]- last := i == len(candidates)-1- if bodyBytes != nil {- r.Body = io.NopCloser(bytes.NewReader(bodyBytes))- }- retry := false- sc := &statusCapture{ResponseWriter: w, status: http.StatusOK, idle: idleClientWriteTimeout}-- proxy := &httputil.ReverseProxy{- Director: func(req *http.Request) {- req.URL.Scheme = cand.url.Scheme- req.URL.Host = cand.url.Host- req.Host = cand.url.Host- },- // A remote cluster peer is dialed over mTLS (per-peer pinned config);- // self/manual candidates use the plain transport. See candidateTransport.- Transport: p.candidateTransport(cand),- // ModifyResponse fires when the upstream's status line + headers- // have arrived but before the body streams. That's both the retry- // decision point and, on commit, the time-to-first-byte boundary.- ModifyResponse: func(resp *http.Response) error {- if !last && shouldRetry(resp.StatusCode) {- // Abort before streaming: ReverseProxy closes resp.Body and- // calls ErrorHandler with our sentinel, then we try next.- retry = true- return retrySignal{}- }- // Prefer an engine-declared preflight policy so an exact origin plus- // Allow-Credentials can pass a credentialed browser fetch. Engines- // that publish no policy retain the proxy's permissive 204 fallback.- cors.CompletePreflightFallback(resp)- // Committing to this candidate — body stream is about to begin.- ttfbMs = time.Since(start).Milliseconds()- servedNodeID = cand.id- servedTarget = cand.url.Host- proxyErr = "" // clear any error recorded from a failed-over candidate- // Arm the liveness report only now. statusCapture also carries- // the proxy's OWN error bodies — ReverseProxy's ErrorHandler- // writes a failed dial's message through it — and those bytes- // prove nothing about the node. Reaching here means the upstream- // returned a status line, so everything written from this point- // came from the node. Same goroutine as the body copy, so no- // synchronization is needed.- sc.upstreamAlive = func() { p.reportActivity(cand.id) }- // The engine may enforce its own origin policy. Honor it:- // overwriting a declared Access-Control-Allow-Origin would- // silently widen the user's policy, and a wildcard is invalid- // alongside Allow-Credentials, so it would break a credentialed- // response outright. An engine that omits the header has- // expressed nothing to preserve, so the proxy supplies its own.- if resp.Header.Get("Access-Control-Allow-Origin") == "" {- cors.Apply(resp.Header)- }- if !started {- started = true- p.codec.Notify("proxy/request-started", RequestStartedEvent{- ID: reqID,- NodeID: cand.id,- Method: r.Method,- Path: r.URL.Path,- Target: cand.url.Host,- })- // workload:started was already emitted up front naming the- // first candidate. If failover landed us on a different- // node, re-point scheduledOn so the card — and the terminal- // completed/errored event, which carries the same wl — name- // the node that actually served. Guarded by wlMu against the- // disconnect watcher, and skipped once terminated so a late- // re-point can't resurrect a workload we've already failed.- if wl != nil {- wlMu.Lock()- if !terminated && wl.ScheduledOn != cand.id {- wl.ScheduledOn = cand.id- snapshot := *wl- wlMu.Unlock()- p.emitWorkload(workloadStartedMethod, snapshot)- } else {- wlMu.Unlock()- }- }- }- return nil- },- ErrorHandler: func(ew http.ResponseWriter, _ *http.Request, err error) {- if _, ok := err.(retrySignal); ok {- return // retryable status — the loop advances to the next candidate- }- // Transport/dial error (not a status-based retry): forget this- // node's confirmed address so the next request re-confirms and- // can fail over to another of its published addresses- // (multi-homed peer). The in-request failover below moves on to- // the next node.- p.targets.Forget(cand.id)- if !last {- // Transport/dial error with candidates left: fail over.- retry = true- proxyErr = err.Error()- slog.Warn("proxy upstream error, failing over",- "id", reqID, "node_id", cand.id, "target", cand.url.Host,- "path", r.URL.Path, "err", err)- return- }- // Last candidate failed at the transport: terminal, surface it.- servedNodeID = cand.id- servedTarget = cand.url.Host- if cors.WritePreflight(ew, r) {- proxyErr = ""- return- }- proxyErr = err.Error()- slog.Warn("proxy upstream error, candidates exhausted",- "id", reqID, "node_id", cand.id, "target", cand.url.Host,- "method", r.Method, "path", r.URL.Path,- "duration_ms", time.Since(start).Milliseconds(), "err", err)- body, mErr := json.Marshal(map[string]string{- "error": "upstream error: " + err.Error(),- })- if mErr != nil {- body = []byte(`{"error":"upstream error"}`)- }- cors.Apply(ew.Header())- ew.Header().Set("Content-Type", "application/json")- ew.Header().Set("X-Content-Type-Options", "nosniff")- ew.WriteHeader(http.StatusBadGateway)- ew.Write(body)- },- }-- proxy.ServeHTTP(sc, r)- if !retry {- finalStatus = sc.status- committedSC = sc- break- }- }-- slog.Debug("proxy request complete",- "id", reqID,- "node_id", servedNodeID,- "method", r.Method,- "path", r.URL.Path,- "target", servedTarget,- "status", finalStatus,- "duration_ms", time.Since(start).Milliseconds(),- "ttfb_ms", ttfbMs,- "err", proxyErr,- )-- p.codec.Notify("proxy/request", RequestEvent{- ID: reqID,- NodeID: servedNodeID,- Method: r.Method,- Path: r.URL.Path,- Target: servedTarget,- Status: finalStatus,- Duration: time.Since(start).Milliseconds(),- TTFB: ttfbMs,- Error: proxyErr,- })-- // Terminal workload transition pairs with the workload:started emitted at- // the commit point above. Cancellation, an upstream/transport error, a- // failed client write (dead/half-open client), or any non-2xx status is a- // failure; a clean 2xx is a completion. The Workload carries the same id so- // the broker (and peers) can collapse the start/finish pair. Routed through- // emitTerminal so the disconnect watcher and this path emit exactly once.- if wl != nil {- switch {- case r.Context().Err() != nil:- // The request was cancelled before it finished — either the- // client disconnected or, on shutdown, we cancelled it to stop- // the in-flight inference. A mid-stream cancel never reaches- // ErrorHandler (the 200 headers are already sent), so without this- // branch it would be misreported as completed. (The watcher above- // usually beats us to it; emitTerminal makes that a no-op.)- emitTerminal("failed", "request cancelled before completion")- case committedSC != nil && committedSC.wroteErr != nil:- // The response committed but a write to (or flush toward) the- // client failed — typically the idle deadline tripping on a- // dead/half-open client. The stream is truncated, so this is a- // failure, not the completion the 200 status would otherwise- // suggest.- emitTerminal("failed", "client connection lost: "+committedSC.wroteErr.Error())- case proxyErr != "" || finalStatus >= http.StatusBadRequest:- msg := proxyErr- if msg == "" {- msg = fmt.Sprintf("upstream returned HTTP %d", finalStatus)- }- emitTerminal("failed", msg)- default:- emitTerminal("completed", "")- }- }-}--// resolveCandidates returns the ordered list of nodes to try for the current-// request. A model-bearing request first filters a request-local node copy to-// advertised owners. A user-selected eligible node then leads, followed by-// scheduler priority and stable ID fallback. The failover loop walks the-// resulting owner list until a node returns a usable response.-//-// A node that resolves to this proxy's own listen address is dropped-// (self-forward guard): a local LM Studio advertisement could otherwise-// resolve back to this proxy's own port and loop.-//-// Returns an empty slice when no forwarding target is available; the caller-// treats that as the rejection path.-func (p *Proxy) resolveCandidates(model string) []candidate {- p.selectedMu.RLock()- id := p.selectedID- p.selectedMu.RUnlock()-- priority := p.PriorityList()-- p.httpMu.Lock()- selfPort := p.port- p.httpMu.Unlock()-- // Re-derive membership and pins before resolving so a cluster joined or left,- // and a peer paired or removed, since the last request is reflected without a- // restart: a removed peer stops being a routable candidate immediately, and a- // freshly-paired one becomes one.- p.mesh.Refresh()-- nodes := p.discovery.Nodes()- known := len(nodes)- if model != "" {- owners := make([]Node, 0, len(nodes))- for _, node := range nodes {- if nodeAdvertisesModel(node, model) {- owners = append(owners, node)- }- }- nodes = owners- }- // Sort by ID so candidate order is stable across calls — Discovery.Nodes()- // iterates a map, whose order is randomized per call, which otherwise bounced- // back-to-back requests between nodes. The ID sort is also the- // fallback order for nodes the scheduler's priority list doesn't mention.- sort.Slice(nodes, func(i, j int) bool { return nodes[i].ID < nodes[j].ID })- byID := make(map[string]Node, len(nodes))- for _, n := range nodes {- byID[n.ID] = n- }-- out := make([]candidate, 0, len(nodes))- // Dedup by resolved backend host: the same physical node can appear under two- // IDs (e.g. a manually-added entry and its relay-discovered record), and- // routing to the same engine twice is wasteful.- seenHost := make(map[string]bool, len(nodes))- // placed tracks node IDs already considered so the priority-ordered and- // fallback passes don't reconsider one (scheduler ordering).- placed := make(map[string]bool, len(nodes))- add := func(n Node) {- placed[n.ID] = true- // targetURL picks a reachable address for a multi-homed node (cached,- // TCP-probed), falling back to the first candidate; nil only when the- // node advertises no usable address.- u := p.targetURL(n)- if u == nil {- return- }- peerUUID := ""- switch {- case isSelfTarget(u, selfPort):- // Our own advertised endpoint (lm now points at this proxy). Serve- // it from the explicit local backend — the loopback engine — rather- // than dialing our own mTLS ingress, which would recurse. Ranking- // still used this node's real (discovered) model list above.- lb, ok := p.localBackendTarget()- if !ok {- slog.Debug("resolveCandidates: no local backend for self", "node_id", n.ID)- return- }- u = lb- case p.mesh.HasPin(n.ClusterUUID):- // A pinned cluster peer: reach it only over mTLS to its promoted- // proxy (the lm port now advertises the proxy, not the engine).- // The pin is read from the live mesh refreshed above, not from the- // relayed n.Trusted: that flag is the scanner's answer from whenever- // it last saw this peer's mDNS record, so a peer discovered before- // this node's pins were written stays false until its record next- // changes. It is also strictly weaker than what we hold here — the- // dial itself is gated on ClientTLSConfig finding the same pin — so- // the relayed value can only ever disagree by being stale.- u.Scheme = "https"- peerUUID = n.ClusterUUID- case p.discovery.IsManual(n.ID):- // An explicit user-added manual node: dialed plain to the address- // the user supplied (a deliberate, separately-labeled bypass).- default:- // A relay peer we don't hold a pin for (untrusted, or this node is- // unclustered). Its engine is loopback-only and its proxy refuses- // plaintext from the LAN, so it is not a routable target.- slog.Debug("resolveCandidates: dropping unpinned relay peer",- "node_id", n.ID, "cluster_uuid", n.ClusterUUID)- return- }- // Defensive: the local backend must never resolve back to this proxy.- if isSelfTarget(u, selfPort) {- slog.Debug("resolveCandidates: skipping self-target node",- "node_id", n.ID, "target", u.Host, "self_port", selfPort)- return- }- if seenHost[u.Host] {- return- }- seenHost[u.Host] = true- out = append(out, candidate{- id: n.ID,- url: u,- peerUUID: peerUUID,- })- }-- // Capability has already been enforced. An eligible explicit selection wins,- // then the scheduler's least-loaded order, then unlisted owners by stable ID.- if id != "" {- if n, ok := byID[id]; ok && !placed[id] {- add(n)- }- }- for _, pid := range priority {- if n, ok := byID[pid]; ok && !placed[pid] {- add(n)- }- }- for _, n := range nodes {- if !placed[n.ID] {- add(n)- }- }-- slog.Debug("resolveCandidates resolved",- "selected", id, "priority", len(priority), "candidates", len(out),- "eligible", len(nodes), "known", known)- return out-}--// reserveCandidate atomically moves the least estimated loaded scheduler-listed-// candidate to the front of this request's failover list. The scheduler's-// pending count and GPU pressure form the authoritative baseline; reservations-// are local dispatches made since that snapshot arrived and have not necessarily-// completed the proxy→broker→scheduler→proxy feedback loop yet.-//-// Model eligibility was enforced before this function receives the list. An-// explicit node/select pin bypasses reservations, and unlisted/manual owners-// retain their existing fallback position.-func (p *Proxy) reserveCandidate(candidates []candidate) []candidate {- if len(candidates) == 0 {- return candidates- }- if selectedID := p.SelectedID(); selectedID != "" {- for _, cand := range candidates {- if cand.id == selectedID {- return candidates- }- }- }-- candidateIndex := make(map[string]int, len(candidates))- for i, cand := range candidates {- candidateIndex[cand.id] = i- }-- p.priorityMu.Lock()- defer p.priorityMu.Unlock()- if len(p.priority) == 0 {- return candidates- }- if p.priorityReservations == nil {- p.priorityReservations = make(map[string]int)- }-- bestIndex := -1- bestOrder := len(p.priority)- var bestLoad uint64- for order, id := range p.priority {- index, ok := candidateIndex[id]- if !ok {- continue- }- load := uint64(p.priorityPending[id]) +- uint64(p.priorityGPUPressure[id]) +- uint64(p.priorityReservations[id])- if bestIndex < 0 || load < bestLoad || (load == bestLoad && order < bestOrder) {- bestIndex = index- bestOrder = order- bestLoad = load- }- }- if bestIndex < 0 {- return candidates- }-- chosen := candidates[bestIndex]- p.priorityReservations[chosen.id]++- if bestIndex > 0 {- copy(candidates[1:bestIndex+1], candidates[:bestIndex])- candidates[0] = chosen- }- return candidates-}--func nodeAdvertisesModel(n Node, model string) bool {- if model == "" {- return false- }- for _, available := range n.Models {- if available == model {- return true- }- }- return false-}--// isSelfTarget reports whether u points back at this proxy's own listener.-// nodeURL has already rewritten local-interface addresses to 127.0.0.1, so a-// loopback host on our own port is us.-func isSelfTarget(u *url.URL, selfPort int) bool {- host, portStr, err := net.SplitHostPort(u.Host)- if err != nil {- return false- }- port, err := strconv.Atoi(portStr)- if err != nil || port != selfPort {- return false- }- switch host {- case "127.0.0.1", "::1", "localhost":- return true- }- return false-}--// nodeURL returns the single best forward URL for a node (the first candidate-// in deterministic, loopback-first order). It does no reachability probing —-// p.targetURL is the request-path entry point. Kept as a free function so the-// URL-construction unit test can exercise it without a Proxy.-func nodeURL(n Node) *url.URL {- candidates := nodeCandidates(n)- if len(candidates) == 0 {- return nil- }- return &url.URL{Scheme: "http", Host: candidates[0]}-}--// targetURL picks the forward URL for a node, preferring an address we can-// actually reach. With a single candidate it's just that candidate; with-// several (a multi-homed peer) the shared reach.Chooser returns the confirmed-// last-good address, or the node's own top-ranked one while it confirms in the-// background. The confirmation is transport-neutral: a TCP accept proves the-// address is reachable, while the real pinned mTLS request still authenticates-// which peer answered there.-//-// reach.Prefer, not a blocking confirmation: this runs once per discovered node per-// request, including nodes this request will not be routed to, so a handshake here-// would charge every request for every node's connectivity. An address that is-// wrong is caught by the ErrorHandler below, which forgets it and fails over.-func (p *Proxy) targetURL(n Node) *url.URL {- candidates := nodeCandidates(n)- if len(candidates) == 0 {- return nil- }- host := p.targets.Prefer(n.ID, candidates)- return &url.URL{Scheme: "http", Host: host}-}--// nodeCandidates returns the ordered, de-duplicated host:port targets for a node.-//-// Order comes from the node itself: netpick.Candidates keeps the node's published-// ranking, which it derived from evidence no observer has, and appends anything-// else it advertised. Re-sorting here by address class is what previously put a-// two-host direct-connect link ahead of a peer's real LAN address.-//-// Any local-interface address is rewritten to loopback (the engine binds loopback-// only) and floated to the front because it's unambiguously reachable.-func nodeCandidates(n Node) []string {- port := strconv.Itoa(n.Port)- sorted := netpick.Candidates(n.TXT, n.Addresses)- if len(sorted) == 0 {- // A non-IP entry (a .local hostname) that netpick cannot parse.- hosts := n.Addresses- if len(hosts) == 0 {- if n.Host == "" {- return nil- }- hosts = []string{n.Host}- }- sorted = append([]string(nil), hosts...)- }-- seen := make(map[string]bool, len(sorted))- var loopback, rest []string- for _, h := range sorted {- // If the address belongs to a local interface, use loopback instead;- // connecting via the machine's own external IP would be refused.- if isLocalAddress(h) {- h = "127.0.0.1"- }- // net.JoinHostPort bracket-wraps IPv6 literals (fe80::1 -> [fe80::1]).- hp := net.JoinHostPort(h, port)- if seen[hp] {- continue- }- seen[hp] = true- if ip := net.ParseIP(h); ip != nil && ip.IsLoopback() {- loopback = append(loopback, hp)- } else {- rest = append(rest, hp)- }- }- return append(loopback, rest...)-}--var (- localAddrsMu sync.RWMutex- // localAddrs is the set of IPs currently bound to this host's interfaces.- // It's used to decide whether a discovered node is actually us, so we can- // dial loopback instead of our own external IP (the engine binds loopback- // only). The initial value is a one-shot enumeration; startLocalAddrWatch- // then keeps it in sync with live interface changes, so a late VPN/dock- // interface or a sleep/wake IP reassignment can't strand us dialing a- // stale address.- localAddrs = netmon.Enumerate().LocalIPs-)--func setLocalAddrs(s map[string]bool) {- localAddrsMu.Lock()- localAddrs = s- localAddrsMu.Unlock()-}--func isLocalAddress(addr string) bool {- localAddrsMu.RLock()- defer localAddrsMu.RUnlock()- return localAddrs[addr]-}--// startLocalAddrWatch keeps localAddrs in sync with the host's live interface-// set for the lifetime of ctx. If the network monitor can't start, the set-// stays at its initial enumeration rather than failing the proxy.-func startLocalAddrWatch(ctx context.Context) {- mon, err := netmon.Watch(ctx)- if err != nil {- slog.Warn("proxy: network monitor unavailable; local address set is static", "err", err)- return- }- setLocalAddrs(mon.LocalIPs())- ch := mon.Subscribe()- go func() {- for range ch {- setLocalAddrs(mon.LocalIPs())- slog.Debug("proxy: refreshed local address set after network change")- }- }()-}--func (p *Proxy) SelectedID() string {- p.selectedMu.RLock()- defer p.selectedMu.RUnlock()- return p.selectedID-}--func (p *Proxy) SetSelected(id string) {- p.selectedMu.Lock()- p.selectedID = id- p.selectedMu.Unlock()-}--// clearSelectionIfNotPresent resets the user-selected node (and notifies the-// client) when it's neither in the relay-fed set nor a manual node, so a stale-// selection can't pin routing to a departed target.-func (p *Proxy) clearSelectionIfNotPresent(present map[string]bool) {- p.selectedMu.Lock()- sel := p.selectedID- p.selectedMu.Unlock()- if sel == "" || present[sel] || p.discovery.IsManual(sel) {- return- }- p.selectedMu.Lock()- cleared := p.selectedID == sel- if cleared {- p.selectedID = ""- }- p.selectedMu.Unlock()- if cleared {- p.codec.Notify("node/selection-changed", SelectedResult{ID: ""})- }-}--// PriorityList returns a copy of the current scheduler-supplied priority order.-func (p *Proxy) PriorityList() []string {- p.priorityMu.RLock()- defer p.priorityMu.RUnlock()- return append([]string(nil), p.priority...)-}--// SetPriority stores the auto-routing priority order (highest first) and returns-// the number of ids stored. The list is kept verbatim — unknown ids are retained-// (a node may appear in discovery later) and only consulted at request time. An-// empty list clears the scheduler's influence.-func (p *Proxy) SetPriority(nodes []string) int {- return p.SetPrioritySnapshot(schedulerwire.Priority{Nodes: nodes})-}--// SetPrioritySnapshot replaces the scheduler baseline and clears optimistic-// reservations made against the previous snapshot. Nodes-only callers remain-// valid: a missing rank supplies zero pending and GPU-pressure baselines.-func (p *Proxy) SetPrioritySnapshot(priority schedulerwire.Priority) int {- cleaned := append([]string(nil), priority.Nodes...)- pending := make(map[string]int, len(priority.Ranks))- gpuPressure := make(map[string]int, len(priority.Ranks))- for _, rank := range priority.Ranks {- if rank.ID == "" {- continue- }- if rank.Pending < 0 {- rank.Pending = 0- }- if rank.GPUPressure < 0 {- rank.GPUPressure = 0- } else if rank.GPUPressure > schedulerwire.MaxGPUPressure {- rank.GPUPressure = schedulerwire.MaxGPUPressure- }- pending[rank.ID] = rank.Pending- gpuPressure[rank.ID] = rank.GPUPressure- }-- p.priorityMu.Lock()- p.priority = cleaned- p.priorityPending = pending- p.priorityGPUPressure = gpuPressure- p.priorityReservations = make(map[string]int)- p.priorityMu.Unlock()- return len(cleaned)-}--// replaceSubscribed replaces the proxy's relay-fed routing overlay from a-// discovery:nodes snapshot: it projects every node advertising lm with a dialable-// IP into the overlay (dropping the rest) and clears a user selection pinned to a-// node that's no longer routable. The broker sends the full filtered set on every-// change, so this is a wholesale replace, not a per-node apply — a departed node-// is simply absent from the next snapshot.-func (p *Proxy) replaceSubscribed(params json.RawMessage) {- var res noderec.GetNodesResult- if err := json.Unmarshal(params, &res); err != nil {- slog.Warn("invalid discovery:nodes snapshot", "err", err)- return- }- nodes := make([]Node, 0, len(res.Nodes))- present := make(map[string]bool, len(res.Nodes))- for _, dn := range res.Nodes {- n, ok := subscribedToNode(dn)- if !ok {- continue- }- nodes = append(nodes, n)- present[n.ID] = true- }- discovered, updated, removed := p.discovery.SetSubscribed(nodes)- // Surface the relay-fed set to the client as node/* events — the signal a- // consumer (the UI) uses to show which peers run this engine — mirroring how- // manual nodes are announced. Without this the routing overlay updates- // silently and peers appear engine-less. A node dropping out is also the- // proxy's "this upstream is gone" signal, surfaced through the errors- // pipeline (the broker forwards these to nvpair-errors); a re-appearance clears- // it. NodeID/Timestamp are left unset so the broker stamps the authoritative- // values.- for _, n := range discovered {- p.codec.Notify("node/discovered", n.withPrimaryIP())- if err := p.codec.Notify("errors:clear", errors.ClearParams{ID: upstreamUnreachableID(n.ID)}); err != nil {- slog.Debug("failed to send errors:clear", "node", n.ID, "err", err)- }- }- for _, n := range updated {- p.codec.Notify("node/updated", n.withPrimaryIP())- }- for _, n := range removed {- p.codec.Notify("node/removed", n.withPrimaryIP())- if err := p.codec.Notify("errors:report", errors.ServiceError{- ID: upstreamUnreachableID(n.ID),- Message: fmt.Sprintf("Upstream node %q is no longer reachable (dropped from discovery)", n.Host),- Severity: "warning",- Action: "none",- }); err != nil {- slog.Debug("failed to send errors:report", "node", n.ID, "err", err)- }- }- p.clearSelectionIfNotPresent(present)-}--// upstreamUnreachableID is the canonical ServiceError id for an upstream the-// proxy no longer sees in discovery. Kept as a single function so the report and-// clear can't drift (nvpair-errors matches by literal id).-func upstreamUnreachableID(nodeID string) string {- return "lmstudio-proxy:upstream-unreachable:" + nodeID-}--// subscribedToNode projects a relay DirectoryNode onto the proxy's routable Node-// for the lm service, returning false when the node doesn't advertise lm or has-// no dialable address. The engine port comes from the lm service key (the real-// LM Studio port the broker's engine poller registered, not the proxy's listen-// port).-func subscribedToNode(n noderec.DirectoryNode) (Node, bool) {- svc, ok := n.Services[noderec.ServiceLMStudio]- if !ok || n.IP == "" {- return Node{}, false- }- // Key routing by the stable per-host UUID, not the hostname: candidate ids,- // scheduledOn, node selection, and the scheduler's priority list are all this- // value, so routing survives a PC rename and never conflates two same-named- // machines. Host stays the hostname (display / dial name). A relay- // DirectoryNode always carries a hostUuid (the scanner guarantees it at the- // browse boundary), so there is no name fallback here.- return Node{- ID: n.HostUUID,- Host: n.Name,- Port: svc.Port,- // The node's whole ranked address list, not just its canonical one: a- // multi-homed peer's best address from its own vantage point may be a- // direct-connect link this host cannot reach, and routing needs somewhere- // to fail over to when that happens.- Addresses: n.CandidateIPs(),- TXT: n.AddressTXT(),- IP: n.IP,- ClusterUUID: n.ClusterUUID,- // Filter on this node's LM Studio models only, not the cross-engine union,- // so a model a dual-engine node serves solely via Ollama isn't accepted as- // an LM Studio owner here (falls back to the union for a peer that sends- // no attribution — see DirectoryNode.EngineModels).- Models: append([]string(nil), n.EngineModels("lmstudio")...),- }, true-}--func (p *Proxy) readLoop(ctx context.Context) error {- for {- msg, err := p.codec.Read()- if err != nil {- if err == io.EOF || ctx.Err() != nil {- return nil- }- log.Printf("JSON-RPC read error: %v", err)- continue- }- p.handleMessage(msg)- }-}--func (p *Proxy) handleMessage(msg *Message) {- if msg.Method == applog.SetLevelMethod {- resolved, err := applog.HandleSetLevelParams(msg.Params)- if msg.IsRequest() {- if err != nil {- p.codec.RespondError(msg.ID, -32602, err.Error())- return- }- p.codec.Respond(msg.ID, map[string]string{"level": resolved})- }- if err != nil {- slog.Warn("log/set-level rejected", "err", err)- } else {- slog.Info("log level changed", "level", resolved)- }- return- }-- switch msg.Method {- case noderec.NotifyNodes:- p.replaceSubscribed(msg.Params)- return- }-- if !msg.IsRequest() {- if msg.IsNotification() {- log.Printf("ignoring incoming notification: %s", msg.Method)- }- return- }-- switch msg.Method {- case "nodes/list":- nodes := p.discovery.Nodes()- if err := p.codec.Respond(msg.ID, NodesResult{Nodes: nodes}); err != nil {- log.Printf("failed to respond to nodes/list: %v", err)- }-- case "node/select":- var params SelectParams- if err := json.Unmarshal(msg.Params, ¶ms); err != nil {- p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"id\": \"...\"}")- return- }- if params.ID != "" {- found := false- for _, n := range p.discovery.Nodes() {- if n.ID == params.ID {- found = true- break- }- }- if !found {- p.codec.RespondError(msg.ID, -32602, fmt.Sprintf("node %q not found", params.ID))- return- }- }- p.SetSelected(params.ID)- log.Printf("node selection changed to %q", params.ID)- if err := p.codec.Respond(msg.ID, SelectedResult{ID: params.ID}); err != nil {- log.Printf("failed to respond to node/select: %v", err)- }- p.codec.Notify("node/selection-changed", SelectedResult{ID: params.ID})-- case "node/selected":- if err := p.codec.Respond(msg.ID, SelectedResult{ID: p.SelectedID()}); err != nil {- log.Printf("failed to respond to node/selected: %v", err)- }-- case "node/set-priority":- var params schedulerwire.Priority- if err := json.Unmarshal(msg.Params, ¶ms); err != nil {- p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"nodes\": [\"id\", ...], \"ranks\": [...]}")- return- }- count := p.SetPrioritySnapshot(params)- log.Printf("priority snapshot set (%d nodes, %d ranks): %v", count, len(params.Ranks), params.Nodes)- if err := p.codec.Respond(msg.ID, map[string]int{"count": count}); err != nil {- log.Printf("failed to respond to node/set-priority: %v", err)- }-- case "set-port":- var params struct {- Port int `json:"port"`- }- if err := json.Unmarshal(msg.Params, ¶ms); err != nil {- p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"port\": <int>}")- return- }- if params.Port < 1 || params.Port > 65535 {- p.codec.RespondError(msg.ID, -32602, "port must be between 1 and 65535")- return- }- if err := p.setPort(params.Port); err != nil {- p.codec.RespondError(msg.ID, -32000, err.Error())- return- }- if err := p.codec.Respond(msg.ID, ReadyParams{Version: Version, Port: params.Port}); err != nil {- log.Printf("failed to respond to set-port: %v", err)- }-- case "node/add-manual":- var node Node- if err := json.Unmarshal(msg.Params, &node); err != nil {- p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"id\",\"host\",\"port\",\"addresses\"}")- return- }- if node.ID == "" || node.Port == 0 || len(node.Addresses) == 0 {- p.codec.RespondError(msg.ID, -32602, "id, port, and at least one address are required")- return- }- added := p.discovery.AddManual(node)- if err := p.codec.Respond(msg.ID, map[string]bool{"added": added}); err != nil {- log.Printf("failed to respond to node/add-manual: %v", err)- }- if added {- log.Printf("manual node added: %s (%s:%d)", node.ID, node.Addresses[0], node.Port)- p.codec.Notify("node/discovered", node.withPrimaryIP())- } else {- log.Printf("manual node updated: %s (%s:%d)", node.ID, node.Addresses[0], node.Port)- p.codec.Notify("node/updated", node.withPrimaryIP())- }-- case "node/remove-manual":- var params SelectParams- if err := json.Unmarshal(msg.Params, ¶ms); err != nil {- p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"id\": \"...\"}")- return- }- removed := p.discovery.RemoveManual(params.ID)- if err := p.codec.Respond(msg.ID, map[string]bool{"removed": removed}); err != nil {- log.Printf("failed to respond to node/remove-manual: %v", err)- }- if removed {- log.Printf("manual node removed: %s", params.ID)- p.selectedMu.Lock()- if p.selectedID == params.ID {- p.selectedID = ""- p.selectedMu.Unlock()- p.codec.Notify("node/selection-changed", SelectedResult{ID: ""})- } else {- p.selectedMu.Unlock()- }- p.codec.Notify("node/removed", Node{ID: params.ID})- }-- case "node/set-local-backend":- var b localBackend- if err := json.Unmarshal(msg.Params, &b); err != nil {- p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"engine\",\"host\",\"port\",\"healthy\"}")- return- }- p.setLocalBackend(b)- slog.Info("local backend updated", "engine", b.Engine, "host", b.Host, "port", b.Port, "healthy", b.Healthy)- if err := p.codec.Respond(msg.ID, map[string]bool{"ok": true}); err != nil {- log.Printf("failed to respond to node/set-local-backend: %v", err)- }-- case "shutdown":- if err := p.codec.Respond(msg.ID, nil); err != nil {- log.Printf("failed to respond to shutdown: %v", err)- }- log.Println("shutdown requested via JSON-RPC")- p.cancel()-- default:- if err := p.codec.RespondError(msg.ID, -32601, fmt.Sprintf("method not found: %s", msg.Method)); err != nil {- log.Printf("failed to send error response: %v", err)- }- }-}diff --git a/services/lmstudio-proxy/proxy_test.go b/services/lmstudio-proxy/proxy_test.godeleted file mode 100644index d0a029c..0000000--- a/services/lmstudio-proxy/proxy_test.go+++ /dev/null@@ -1,63 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import "testing"--// TestNodeURL covers the URL-construction part of nodeURL — specifically-// that IPv6 literals are bracket-wrapped and IPv4/hostnames remain-// byte-identical to the pre-JoinHostPort implementation. The-// local-address shortcut is not exercised here because it depends on-// the host's network interfaces (see init() in proxy.go) and would be-// flaky across environments.-func TestNodeURL(t *testing.T) {- // Pick addresses that are unlikely to appear on any local interface.- tests := []struct {- name string- node Node- wantHost string- wantURL string- }{- {- name: "ipv4",- node: Node{Addresses: []string{"192.0.2.10"}, Port: 11434},- wantHost: "192.0.2.10:11434",- wantURL: "http://192.0.2.10:11434",- },- {- name: "ipv6",- node: Node{Addresses: []string{"2001:db8::1"}, Port: 11434},- wantHost: "[2001:db8::1]:11434",- wantURL: "http://[2001:db8::1]:11434",- },- {- name: "hostname",- node: Node{Addresses: []string{"gpu-host.lan"}, Port: 11434},- wantHost: "gpu-host.lan:11434",- wantURL: "http://gpu-host.lan:11434",- },- {- // Empty Addresses slice — nodeURL should fall back to Host.- name: "fallback to Host",- node: Node{Host: "gpu-host.lan", Port: 11434},- wantHost: "gpu-host.lan:11434",- wantURL: "http://gpu-host.lan:11434",- },- }-- for _, tc := range tests {- t.Run(tc.name, func(t *testing.T) {- u := nodeURL(tc.node)- if u == nil {- t.Fatalf("nodeURL returned nil")- }- if u.Host != tc.wantHost {- t.Errorf("Host = %q, want %q", u.Host, tc.wantHost)- }- if got := u.String(); got != tc.wantURL {- t.Errorf("String() = %q, want %q", got, tc.wantURL)- }- })- }-}diff --git a/services/lmstudio-proxy/reservation_test.go b/services/lmstudio-proxy/reservation_test.godeleted file mode 100644index 97049eb..0000000--- a/services/lmstudio-proxy/reservation_test.go+++ /dev/null@@ -1,235 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "sync"- "testing"-- "nvpair-shared/schedulerwire"-)--func reservationCandidates(ids ...string) []candidate {- out := make([]candidate, 0, len(ids))- for _, id := range ids {- out = append(out, candidate{id: id})- }- return out-}--func reservedID(p *Proxy, candidates []candidate) string {- candidates = append([]candidate(nil), candidates...)- return p.reserveCandidate(candidates)[0].id-}--func TestReserveCandidate_ConcurrentEqualLoadHasAtMostOneSkew(t *testing.T) {- p := prProxy(t)- ids := []string{"a", "b", "c", "d"}- ranks := make([]schedulerwire.NodeRank, 0, len(ids))- for i, id := range ids {- ranks = append(ranks, schedulerwire.NodeRank{ID: id, Rank: i})- }- p.SetPrioritySnapshot(schedulerwire.Priority{Nodes: ids, Ranks: ranks})- candidates := reservationCandidates(ids...)-- const requests = 100- chosen := make(chan string, requests)- var wg sync.WaitGroup- for range requests {- wg.Add(1)- go func() {- defer wg.Done()- chosen <- reservedID(p, candidates)- }()- }- wg.Wait()- close(chosen)-- counts := make(map[string]int, len(ids))- for id := range chosen {- counts[id]++- }- min, max := requests, 0- for _, id := range ids {- if counts[id] < min {- min = counts[id]- }- if counts[id] > max {- max = counts[id]- }- }- if max-min > 1 {- t.Fatalf("100 equal-load reservations are imbalanced: %v", counts)- }-}--func TestReserveCandidate_ConvergesUnequalPendingDepths(t *testing.T) {- p := prProxy(t)- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"a", "b", "c"},- Ranks: []schedulerwire.NodeRank{- {ID: "a", Pending: 0, Rank: 0},- {ID: "b", Pending: 2, Rank: 1},- {ID: "c", Pending: 4, Rank: 2},- },- })- candidates := reservationCandidates("a", "b", "c")- assigned := map[string]int{}- for range 6 {- assigned[reservedID(p, candidates)]++- }-- total := map[string]int{- "a": assigned["a"],- "b": 2 + assigned["b"],- "c": 4 + assigned["c"],- }- if total["a"] != 4 || total["b"] != 4 || total["c"] != 4 {- t.Fatalf("unequal depths did not converge: assigned=%v total=%v", assigned, total)- }-}--func TestReserveCandidate_CombinesPendingPressureAndReservations(t *testing.T) {- p := prProxy(t)- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"a", "b", "c"},- Ranks: []schedulerwire.NodeRank{- {ID: "a", Pending: 0, GPUPressure: 3},- {ID: "b", Pending: 1, GPUPressure: 0},- {ID: "c", Pending: 0, GPUPressure: 2},- },- })- candidates := reservationCandidates("a", "b", "c")- got := []string{- reservedID(p, candidates),- reservedID(p, candidates),- reservedID(p, candidates),- }- want := []string{"b", "b", "c"}- for i := range want {- if got[i] != want[i] {- t.Fatalf("GPU-aware reservations = %v, want %v", got, want)- }- }-}--func TestSetPrioritySnapshotClampsGPUPressure(t *testing.T) {- p := prProxy(t)- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"low", "high"},- Ranks: []schedulerwire.NodeRank{- {ID: "low", GPUPressure: -1},- {ID: "high", GPUPressure: schedulerwire.MaxGPUPressure + 1},- },- })- p.priorityMu.RLock()- low := p.priorityGPUPressure["low"]- high := p.priorityGPUPressure["high"]- p.priorityMu.RUnlock()- if low != 0 || high != schedulerwire.MaxGPUPressure {- t.Fatalf("clamped GPU pressure = low:%d high:%d", low, high)- }-}--func TestReserveCandidate_LegacyNodesOnlyUsesZeroBaseline(t *testing.T) {- p := prProxy(t)- p.SetPriority([]string{"a", "b", "c"})- candidates := reservationCandidates("a", "b", "c")- counts := map[string]int{}- for range 5 {- counts[reservedID(p, candidates)]++- }- want := map[string]int{"a": 2, "b": 2, "c": 1}- for id, n := range want {- if counts[id] != n {- t.Fatalf("legacy nodes-only assignments = %v, want %v", counts, want)- }- }-}--func TestReserveCandidate_UsesEligibleCandidates(t *testing.T) {- p := prProxy(t)- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"missing", "owner-a", "owner-b", "unknown"},- Ranks: []schedulerwire.NodeRank{- {ID: "missing", Pending: 0},- {ID: "owner-a", Pending: 4},- {ID: "owner-b", Pending: 5},- {ID: "unknown", Pending: 0},- },- })- candidates := reservationCandidates("owner-a", "owner-b")-- for range 8 {- got := reservedID(p, candidates)- if got != "owner-a" && got != "owner-b" {- t.Fatalf("reservation escaped eligible candidates to %q", got)- }- }-}--func TestReserveCandidate_ManualPinBypassesReservations(t *testing.T) {- p := prProxy(t)- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"a", "b"},- Ranks: []schedulerwire.NodeRank{{ID: "a"}, {ID: "b"}},- })- p.SetSelected("b")- candidates := reservationCandidates("b", "a") // resolveCandidates puts the pin first- if got := reservedID(p, candidates); got != "b" {- t.Fatalf("manual pin resolved to %q, want b", got)- }- p.priorityMu.RLock()- defer p.priorityMu.RUnlock()- if len(p.priorityReservations) != 0 {- t.Fatalf("manual pin created optimistic reservations: %v", p.priorityReservations)- }-}--func TestReserveCandidate_IneligibleManualPinDoesNotBypassReservations(t *testing.T) {- p := prProxy(t)- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"owner-b", "owner-a"},- Ranks: []schedulerwire.NodeRank{{ID: "owner-b"}, {ID: "owner-a"}},- })- p.SetSelected("missing")- if got := reservedID(p, reservationCandidates("owner-a", "owner-b")); got != "owner-b" {- t.Fatalf("reservation with ineligible pin = %q, want owner-b", got)- }-}--func TestReserveCandidate_PreservesFailoverAndSnapshotReset(t *testing.T) {- p := prProxy(t)- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"b", "a", "c"},- Ranks: []schedulerwire.NodeRank{- {ID: "b", Pending: 0},- {ID: "a", Pending: 5},- {ID: "c", Pending: 6},- },- })- got := p.reserveCandidate(reservationCandidates("a", "b", "c"))- want := []string{"b", "a", "c"}- for i, id := range want {- if got[i].id != id {- t.Fatalf("reserved failover order = %v, want %v", candidateIDsFrom(got), want)- }- }-- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"a", "b", "c"},- Ranks: []schedulerwire.NodeRank{{ID: "a"}, {ID: "b"}, {ID: "c"}},- })- if next := reservedID(p, reservationCandidates("a", "b", "c")); next != "a" {- t.Fatalf("new snapshot did not reset reservations: next = %q, want a", next)- }-}--func candidateIDsFrom(candidates []candidate) []string {- out := make([]string, 0, len(candidates))- for _, candidate := range candidates {- out = append(out, candidate.id)- }- return out-}diff --git a/services/lmstudio-proxy/subscribed_test.go b/services/lmstudio-proxy/subscribed_test.godeleted file mode 100644index 6022782..0000000--- a/services/lmstudio-proxy/subscribed_test.go+++ /dev/null@@ -1,98 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "testing"-- "nvpair-shared/noderec"-)--// TestSubscribedToNode covers the DirectoryNode -> routable Node projection for-// the lm service, including per-engine model attribution: the proxy ranks on the-// node's LM Studio models only, never the cross-engine union.-func TestSubscribedToNode(t *testing.T) {- withLM := noderec.DirectoryNode{- HostUUID: "uuid-a",- Name: "host-a",- IP: "10.0.0.5",- Models: []string{"gguf"},- Services: map[noderec.ServiceKey]noderec.ServiceStatus{- noderec.ServiceLMStudio: {Port: 1234},- },- }- got, ok := subscribedToNode(withLM)- if !ok {- t.Fatal("node with lm + IP should project")- }- // No attribution -> fall back to the flat union (single-engine peer).- if got.ID != "uuid-a" || got.Port != 1234 || got.IP != "10.0.0.5" ||- len(got.Models) != 1 || got.Models[0] != "gguf" {- t.Fatalf("unexpected projection: %+v", got)- }-- noIP := withLM- noIP.IP = ""- if _, ok := subscribedToNode(noIP); ok {- t.Fatal("node without IP should not project")- }-- // A node advertising only a non-lm service must not be an lm routing target.- olOnly := noderec.DirectoryNode{- Name: "host-b",- IP: "10.0.0.6",- Services: map[noderec.ServiceKey]noderec.ServiceStatus{noderec.ServiceOllama: {Port: 11434}},- }- if _, ok := subscribedToNode(olOnly); ok {- t.Fatal("node without lm should not project")- }-- // Per-engine attribution: a dual-engine node projects ONLY its LM Studio- // models, never the union — so an Ollama-only model isn't ranked as an- // LM Studio owner.- dual := noderec.DirectoryNode{- HostUUID: "uuid-d",- Name: "host-d",- IP: "10.0.0.7",- Models: []string{"ollama-model", "lmstudio-model"},- ModelsByEngine: map[string][]string{- "ollama": {"ollama-model"},- "lmstudio": {"lmstudio-model"},- },- Services: map[noderec.ServiceKey]noderec.ServiceStatus{- noderec.ServiceLMStudio: {Port: 1234},- },- }- got, ok = subscribedToNode(dual)- if !ok {- t.Fatal("dual-engine node with lm should project")- }- if len(got.Models) != 1 || got.Models[0] != "lmstudio-model" {- t.Fatalf("dual-engine projection Models = %v, want [lmstudio-model] only", got.Models)- }-}--// TestSubscribedToNodeKeysByHostUUID: the routable Node keys on the stable-// hostUuid, not the hostname, so routing/scheduledOn/selection survive a PC-// rename and never conflate same-named machines. Host stays the hostname for-// display.-func TestSubscribedToNodeKeysByHostUUID(t *testing.T) {- const uuid = "22222222-2222-2222-2222-222222222222"- n := noderec.DirectoryNode{- HostUUID: uuid,- Name: "host-a",- IP: "10.0.0.5",- Services: map[noderec.ServiceKey]noderec.ServiceStatus{noderec.ServiceLMStudio: {Port: 1234}},- }- got, ok := subscribedToNode(n)- if !ok {- t.Fatal("node with lm + IP should project")- }- if got.ID != uuid {- t.Fatalf("ID = %q, want hostUuid %q", got.ID, uuid)- }- if got.Host != "host-a" {- t.Fatalf("Host = %q, want hostname for display", got.Host)- }-}diff --git a/services/lmstudio-proxy/transport_pool_test.go b/services/lmstudio-proxy/transport_pool_test.godeleted file mode 100644index 2dc50f0..0000000--- a/services/lmstudio-proxy/transport_pool_test.go+++ /dev/null@@ -1,76 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "path/filepath"- "testing"-- "nvpair-shared/clustertrust"- "nvpair-shared/clustertrusttest"-)--func TestCandidateTransportReusesPlainTransport(t *testing.T) {- p := testProxy(NewDiscovery(), 1235)- a := p.candidateTransport(candidate{})- b := p.candidateTransport(candidate{id: "manual"})- if a != b {- t.Fatalf("plain candidates returned distinct Transports")- }- if a == nil {- t.Fatal("plain Transport is nil")- }-}--func TestCandidateTransportReusesPeerTransport(t *testing.T) {- const peerUUID = "principal-peer"- clusterDir := filepath.Join(t.TempDir(), "cluster")- clustertrusttest.Join(t, clusterDir, "cluster-xyz", "principal-self", peerUUID)-- p := testProxy(NewDiscovery(), 1235)- p.mesh = clustertrust.Open(clusterDir)-- a := p.candidateTransport(candidate{peerUUID: peerUUID})- b := p.candidateTransport(candidate{peerUUID: peerUUID})- if a != b {- t.Fatalf("same peerUUID returned distinct Transports")- }- if a.TLSClientConfig == nil {- t.Fatal("peer Transport missing TLSClientConfig")- }-- other := p.candidateTransport(candidate{peerUUID: "principal-other"})- if other == a {- t.Fatal("unpinned peer reused pinned peer Transport")- }-}--func TestDropUnpinnedPeerTransportsRemovesEntry(t *testing.T) {- const peerUUID = "principal-peer"- clusterDir := filepath.Join(t.TempDir(), "cluster")- clustertrusttest.Join(t, clusterDir, "cluster-xyz", "principal-self", peerUUID)-- p := testProxy(NewDiscovery(), 1235)- p.mesh = clustertrust.Open(clusterDir)-- tr := p.candidateTransport(candidate{peerUUID: peerUUID})- p.transportMu.Lock()- if _, ok := p.peerTransports[peerUUID]; !ok {- p.transportMu.Unlock()- t.Fatal("peer Transport was not cached")- }- p.transportMu.Unlock()-- clustertrusttest.RemovePeerPin(t, clusterDir, peerUUID)- p.mesh.Refresh()- p.dropUnpinnedPeerTransports()-- p.transportMu.Lock()- _, still := p.peerTransports[peerUUID]- p.transportMu.Unlock()- if still {- t.Fatal("peer Transport remained after pin removal")- }- _ = tr-}diff --git a/services/lmstudio-proxy/zombie_test.go b/services/lmstudio-proxy/zombie_test.godeleted file mode 100644index 29fde4f..0000000--- a/services/lmstudio-proxy/zombie_test.go+++ /dev/null@@ -1,494 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "bufio"- "bytes"- "context"- "errors"- "fmt"- "io"- "net"- "net/http"- "net/http/httptest"- "strings"- "sync"- "testing"- "time"-)--// count reports how many times s appears across the recorded codec frames.-// Method names appear once per emitted notification, so this counts emissions.-func (r *prRec) count(s string) int {- r.mu.Lock()- defer r.mu.Unlock()- hay := string(r.b)- n := 0- for i := 0; i+len(s) <= len(hay); i++ {- if hay[i:i+len(s)] == s {- n++- }- }- return n-}--// clientGoneWriter is an http.ResponseWriter whose body Write fails after the-// status line is sent, standing in for a client that vanished mid-stream: the-// idle write deadline trips (statusCapture.Write) and the underlying-// connection write returns an error. It records the status and supports Flush-// so the reverse proxy streams through it. This is the shape of the zombie-job-// bug — the response has committed (200), so without the wroteErr check the-// terminal would be misreported as completed.-type clientGoneWriter struct {- header http.Header- status int- err error- wrote bool-}--func (c *clientGoneWriter) Header() http.Header {- if c.header == nil {- c.header = make(http.Header)- }- return c.header-}--func (c *clientGoneWriter) WriteHeader(code int) { c.status = code }--func (c *clientGoneWriter) Write(b []byte) (int, error) {- c.wrote = true- return 0, c.err-}--func (c *clientGoneWriter) Flush() {}--// TestHandleHTTP_ClientWriteError_MarksFailed is the zombie-job regression: a-// streaming inference response that has committed (200 headers sent) but whose-// body write to the client fails — the signature of a killed / half-open client-// whose write deadline tripped — must terminate the workload as FAILED, not be-// silently reported completed.-func TestHandleHTTP_ClientWriteError_MarksFailed(t *testing.T) {- upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.Header().Set("Content-Type", "application/json")- w.WriteHeader(http.StatusOK)- io.WriteString(w, `{"choices":[{"delta":{"content":"partial tokens"}}]}`)- }))- defer upstream.Close()-- rec := &prRec{}- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "node-a", upstream.URL, "llama"))- p := NewProxy(NewCodec(rec), disc, 1235)-- cw := &clientGoneWriter{err: errors.New("write tcp: connection reset by peer")}-- done := make(chan struct{})- go func() {- defer close(done)- p.handleHTTP(cw, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`)))- }()-- select {- case <-done:- case <-time.After(5 * time.Second):- t.Fatal("handleHTTP did not return after the client write failed (zombie: handler blocked)")- }-- if !cw.wrote {- t.Fatal("reverse proxy never attempted a body write to the client; test did not exercise the streaming path")- }- if got := rec.count("workload:errored"); got != 1 {- t.Fatalf("workload:errored emitted %d times, want exactly 1", got)- }- if rec.has("workload:completed") {- t.Fatal("workload:completed emitted for a request whose client write failed (should be failed)")- }-}--// TestHandleHTTP_ClientDisconnect_TerminalOnce covers the disconnect watcher: a-// request whose context is cancelled mid-flight must emit exactly one terminal-// (errored) — the watcher and post-handler path are guarded by terminalOnce —-// and handleHTTP must return promptly rather than hang.-func TestHandleHTTP_ClientDisconnect_TerminalOnce(t *testing.T) {- received := make(chan struct{}, 1)- release := make(chan struct{})- var releaseOnce sync.Once- doRelease := func() { releaseOnce.Do(func() { close(release) }) }-- upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.Header().Set("Content-Type", "application/json")- w.WriteHeader(http.StatusOK)- if f, ok := w.(http.Flusher); ok {- io.WriteString(w, `{"choices":[{"delta":{"content":"first chunk"}}]}`+"\n")- f.Flush()- }- select {- case received <- struct{}{}:- default:- }- <-release- }))- defer upstream.Close()- defer doRelease()-- rec := &prRec{}- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "node-a", upstream.URL, "llama"))- p := NewProxy(NewCodec(rec), disc, 1235)-- ctx, cancel := context.WithCancel(context.Background())- req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`)).WithContext(ctx)-- done := make(chan struct{})- go func() {- defer close(done)- p.handleHTTP(httptest.NewRecorder(), req)- }()-- select {- case <-received:- case <-time.After(5 * time.Second):- t.Fatal("upstream never started streaming")- }- cancel()-- select {- case <-done:- case <-time.After(5 * time.Second):- t.Fatal("handleHTTP did not return after client disconnect (zombie: handler blocked)")- }-- if got := rec.count("workload:errored"); got != 1 {- t.Fatalf("workload:errored emitted %d times, want exactly 1 (terminalOnce guard)", got)- }- if rec.has("workload:completed") {- t.Fatal("workload:completed emitted for a cancelled request")- }-}--// deadlineRW records SetWriteDeadline calls and can force a Write error, so the-// statusCapture write-deadline mechanics can be tested without a real socket.-type deadlineRW struct {- *httptest.ResponseRecorder- deadlines []time.Time- writeErr error- flushErr error- flushed int-}--func (d *deadlineRW) SetWriteDeadline(t time.Time) error {- d.deadlines = append(d.deadlines, t)- return nil-}--func (d *deadlineRW) Write(b []byte) (int, error) {- if d.writeErr != nil {- return 0, d.writeErr- }- return d.ResponseRecorder.Write(b)-}--// FlushError lets a test drive statusCapture.FlushError against a controllable-// flush outcome (recorded so the deadline arm/clear can be asserted).-func (d *deadlineRW) FlushError() error {- d.flushed++- return d.flushErr-}--// TestStatusCapture_WriteDeadline verifies statusCapture arms a write deadline-// around each streamed write and clears it after a successful one, and that the-// first write error is retained for the caller to classify the workload failed.-func TestStatusCapture_WriteDeadline(t *testing.T) {- t.Run("armed then cleared on success", func(t *testing.T) {- d := &deadlineRW{ResponseRecorder: httptest.NewRecorder()}- sc := &statusCapture{ResponseWriter: d, status: http.StatusOK, idle: 50 * time.Millisecond}- if _, err := sc.Write([]byte("tokens")); err != nil {- t.Fatalf("Write returned error: %v", err)- }- if len(d.deadlines) != 2 {- t.Fatalf("SetWriteDeadline called %d times, want 2 (arm + clear)", len(d.deadlines))- }- if d.deadlines[0].IsZero() {- t.Fatal("first SetWriteDeadline should arm a future deadline, got zero")- }- if !d.deadlines[1].IsZero() {- t.Fatal("second SetWriteDeadline should clear the deadline (zero time)")- }- if sc.wroteErr != nil {- t.Fatalf("wroteErr set after a successful write: %v", sc.wroteErr)- }- })-- t.Run("write error retained, deadline not cleared", func(t *testing.T) {- boom := errors.New("i/o timeout")- d := &deadlineRW{ResponseRecorder: httptest.NewRecorder(), writeErr: boom}- sc := &statusCapture{ResponseWriter: d, status: http.StatusOK, idle: 50 * time.Millisecond}- if _, err := sc.Write([]byte("tokens")); !errors.Is(err, boom) {- t.Fatalf("Write err = %v, want %v", err, boom)- }- if !errors.Is(sc.wroteErr, boom) {- t.Fatalf("wroteErr = %v, want %v", sc.wroteErr, boom)- }- if len(d.deadlines) != 1 {- t.Fatalf("SetWriteDeadline called %d times, want 1 (arm only; not cleared on error)", len(d.deadlines))- }- })-- t.Run("no deadline when idle is zero", func(t *testing.T) {- d := &deadlineRW{ResponseRecorder: httptest.NewRecorder()}- sc := &statusCapture{ResponseWriter: d, status: http.StatusOK}- if _, err := sc.Write([]byte("tokens")); err != nil {- t.Fatalf("Write returned error: %v", err)- }- if len(d.deadlines) != 0 {- t.Fatalf("SetWriteDeadline called %d times with idle=0, want 0", len(d.deadlines))- }- })-}--// TestStatusCapture_FlushDeadline verifies the flush path is deadline-aware:-// statusCapture.FlushError arms the idle deadline around the underlying flush,-// clears it after a successful flush, and retains a real flush error (but not an-// unsupported-flush) so a stalled client's blocked flush is classified failed-// rather than hanging forever.-func TestStatusCapture_FlushDeadline(t *testing.T) {- t.Run("armed then cleared on success", func(t *testing.T) {- d := &deadlineRW{ResponseRecorder: httptest.NewRecorder()}- sc := &statusCapture{ResponseWriter: d, status: http.StatusOK, idle: 50 * time.Millisecond}- if err := sc.FlushError(); err != nil {- t.Fatalf("FlushError returned error: %v", err)- }- if d.flushed != 1 {- t.Fatalf("underlying flushed %d times, want 1", d.flushed)- }- if len(d.deadlines) != 2 {- t.Fatalf("SetWriteDeadline called %d times, want 2 (arm + clear)", len(d.deadlines))- }- if d.deadlines[0].IsZero() {- t.Fatal("flush should arm a future deadline, got zero")- }- if !d.deadlines[1].IsZero() {- t.Fatal("flush should clear the deadline on success (zero time)")- }- if sc.wroteErr != nil {- t.Fatalf("wroteErr set after a successful flush: %v", sc.wroteErr)- }- })-- t.Run("flush error retained, deadline not cleared", func(t *testing.T) {- boom := errors.New("i/o timeout")- d := &deadlineRW{ResponseRecorder: httptest.NewRecorder(), flushErr: boom}- sc := &statusCapture{ResponseWriter: d, status: http.StatusOK, idle: 50 * time.Millisecond}- if err := sc.FlushError(); !errors.Is(err, boom) {- t.Fatalf("FlushError = %v, want %v", err, boom)- }- if !errors.Is(sc.wroteErr, boom) {- t.Fatalf("wroteErr = %v, want %v", sc.wroteErr, boom)- }- if len(d.deadlines) != 1 {- t.Fatalf("SetWriteDeadline called %d times, want 1 (arm only; not cleared on error)", len(d.deadlines))- }- })-- t.Run("unsupported flush is not a client failure", func(t *testing.T) {- d := &deadlineRW{ResponseRecorder: httptest.NewRecorder(), flushErr: http.ErrNotSupported}- sc := &statusCapture{ResponseWriter: d, status: http.StatusOK, idle: 50 * time.Millisecond}- _ = sc.FlushError()- if sc.wroteErr != nil {- t.Fatalf("ErrNotSupported must not be retained as wroteErr, got %v", sc.wroteErr)- }- })-}--// TestHandleHTTP_RealSocketWriteDeadline is the end-to-end, OS-level proof of-// the zombie-job fix. It drives handleHTTP over a REAL TCP socket with a client-// that reads the response headers and then stops reading — the shape of a-// killed / half-open client whose receive window closes without a FIN/RST. The-// upstream streams without end, so the proxy's kernel send buffer to the client-// fills and its next write blocks. Before the fix that write blocks ~forever-// (no terminal event; the zombie job), so r.Context() never fires and the-// handler never returns. With the fix, statusCapture arms a real-// SetWriteDeadline that the Go runtime's netpoller enforces on every platform-// (IOCP on Windows, epoll/kqueue elsewhere) regardless of the peer's TCP state,-// so the stuck write fails and the workload terminates as failed. This is the-// piece the in-process tests stub out — here the deadline is genuinely enforced-// by the OS/runtime.-func TestHandleHTTP_RealSocketWriteDeadline(t *testing.T) {- // Shorten the idle write deadline so a stuck write trips quickly; restore- // the production default for any test that runs after this one.- orig := idleClientWriteTimeout- idleClientWriteTimeout = 300 * time.Millisecond- defer func() { idleClientWriteTimeout = orig }()-- // Upstream streams 64 KiB chunks endlessly. Once the proxy stops reading- // from it (because the proxy is itself blocked writing to the stalled- // client), the upstream's own writes block too — no busy loop — and it- // unwinds when the proxy tears the connection down (write error or context- // cancel).- upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.Header().Set("Content-Type", "application/json")- w.WriteHeader(http.StatusOK)- flusher, _ := w.(http.Flusher)- chunk := bytes.Repeat([]byte("x"), 64*1024)- for {- if _, err := w.Write(chunk); err != nil {- return- }- if flusher != nil {- flusher.Flush()- }- select {- case <-r.Context().Done():- return- default:- }- }- }))- defer upstream.Close()-- rec := &prRec{}- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "node-a", upstream.URL, "llama"))- p := NewProxy(NewCodec(rec), disc, 1235)-- ln, err := net.Listen("tcp", "127.0.0.1:0")- if err != nil {- t.Fatalf("listen: %v", err)- }- srv := &http.Server{Handler: http.HandlerFunc(p.handleHTTP)}- go func() { _ = srv.Serve(ln) }()- defer srv.Close()-- conn, err := net.Dial("tcp", ln.Addr().String())- if err != nil {- t.Fatalf("dial proxy: %v", err)- }- defer conn.Close()-- body := `{"model":"llama"}`- reqText := "POST /v1/chat/completions HTTP/1.1\r\n" +- "Host: localhost\r\n" +- "Content-Type: application/json\r\n" +- fmt.Sprintf("Content-Length: %d\r\n", len(body)) +- "\r\n" + body- if _, err := conn.Write([]byte(reqText)); err != nil {- t.Fatalf("write request: %v", err)- }-- // Read the status line only — enough to confirm the response committed and- // started streaming — then STOP reading so the proxy's send buffer backs- // up. A read deadline guards against a hang if the proxy never responds.- _ = conn.SetReadDeadline(time.Now().Add(5 * time.Second))- statusLine, err := bufio.NewReader(conn).ReadString('\n')- if err != nil {- t.Fatalf("read status line: %v", err)- }- if !strings.Contains(statusLine, "200") {- t.Fatalf("unexpected status line: %q", statusLine)- }-- // The stuck write must trip the deadline and terminate the workload as- // failed within a few multiples of the deadline — never completed.- deadline := time.Now().Add(5 * time.Second)- for time.Now().Before(deadline) && !rec.has("workload:errored") {- time.Sleep(10 * time.Millisecond)- }- if !rec.has("workload:errored") {- t.Fatal("workload never terminated after the client stopped reading (zombie: write deadline did not trip / no terminal emitted)")- }- if rec.has("workload:completed") {- t.Fatal("workload:completed emitted for a client that stopped reading (should be failed)")- }-}--// TestHandleHTTP_RealSocketFlushDeadline is the flush-path counterpart to the-// write-deadline test. A streaming response is flushed after every chunk, so a-// small chunk buffers on a successful Write (no network I/O) and the actual-// network write happens in a separate Flush. If only Write is deadline-aware, a-// stalled client makes that Flush block unbounded and the handler never returns-// — a zombie the 64 KiB Write-blocking test does not catch. This drives real,-// paced small flushed chunks over a real socket and asserts the flush deadline-// terminates the workload as failed.-func TestHandleHTTP_RealSocketFlushDeadline(t *testing.T) {- orig := idleClientWriteTimeout- idleClientWriteTimeout = 300 * time.Millisecond- defer func() { idleClientWriteTimeout = orig }()-- upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.Header().Set("Content-Type", "application/x-ndjson")- w.WriteHeader(http.StatusOK)- flusher, ok := w.(http.Flusher)- if !ok {- t.Errorf("upstream ResponseWriter is not a Flusher")- return- }- chunk := bytes.Repeat([]byte("x"), 1500)- for {- if _, err := w.Write(chunk); err != nil {- return- }- flusher.Flush()- // Pace so the reverse proxy's 32 KiB copy read returns one small- // chunk per iteration rather than coalescing many into a >2 KiB- // write (which would block inside Write, not Flush).- time.Sleep(2 * time.Millisecond)- select {- case <-r.Context().Done():- return- default:- }- }- }))- defer upstream.Close()-- rec := &prRec{}- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "node-a", upstream.URL, "llama"))- p := NewProxy(NewCodec(rec), disc, 1235)-- ln, err := net.Listen("tcp", "127.0.0.1:0")- if err != nil {- t.Fatalf("listen: %v", err)- }- srv := &http.Server{Handler: http.HandlerFunc(p.handleHTTP)}- go func() { _ = srv.Serve(ln) }()- defer srv.Close()-- conn, err := net.Dial("tcp", ln.Addr().String())- if err != nil {- t.Fatalf("dial proxy: %v", err)- }- defer conn.Close()-- body := `{"model":"llama"}`- reqText := "POST /v1/chat/completions HTTP/1.1\r\n" +- "Host: localhost\r\n" +- "Content-Type: application/json\r\n" +- fmt.Sprintf("Content-Length: %d\r\n", len(body)) +- "\r\n" + body- if _, err := conn.Write([]byte(reqText)); err != nil {- t.Fatalf("write request: %v", err)- }-- _ = conn.SetReadDeadline(time.Now().Add(5 * time.Second))- statusLine, err := bufio.NewReader(conn).ReadString('\n')- if err != nil {- t.Fatalf("read status line: %v", err)- }- if !strings.Contains(statusLine, "200") {- t.Fatalf("unexpected status line: %q", statusLine)- }-- deadline := time.Now().Add(5 * time.Second)- for time.Now().Before(deadline) && !rec.has("workload:errored") {- time.Sleep(10 * time.Millisecond)- }- if !rec.has("workload:errored") {- t.Fatal("workload never terminated after the client stopped reading (flush path not deadline-aware)")- }- if rec.has("workload:completed") {- t.Fatal("workload:completed emitted for a stalled client (should be failed)")- }-}diff --git a/services/nvpair-engine-manager/manifestset_test.go b/services/nvpair-engine-manager/manifestset_test.gonew file mode 100644index 0000000..2239a69--- /dev/null+++ b/services/nvpair-engine-manager/manifestset_test.go@@ -0,0 +1,45 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++import (+ "io/fs"+ "strings"+ "testing"++ "nvpair-shared/engines"+)++// nvpair-shared/engines states that an engine's Name must match its manifest+// basename, and argues that declaring the set once turns a mismatch into a+// build error. For the broker, proxy and scheduler it does, because they all+// read the table. This component does not: it embeds manifests/*.json and+// never imports engines, so adding an engine to the table without a manifest+// (or the reverse) still compiles.+//+// The failure that produces is the one the package comment describes — an+// engine that discovers and advertises but never resolves a model owner —+// which is invisible until inference is attempted. The existing manifest tests+// validate content, not the set, so this closes the last gap.+func TestBundledManifestSetMatchesEngineTable(t *testing.T) {+ entries, err := fs.ReadDir(bundledManifests, "manifests")+ if err != nil {+ t.Fatal(err)+ }++ have := map[string]bool{}+ for _, e := range entries {+ have[strings.TrimSuffix(e.Name(), ".json")] = true+ }++ for _, engine := range engines.All() {+ if !have[engine.Name] {+ t.Errorf("no manifests/%s.json for engine %q", engine.Name, engine.Name)+ }+ delete(have, engine.Name)+ }+ for name := range have {+ t.Errorf("manifests/%s.json has no entry in nvpair-shared/engines", name)+ }+}diff --git a/services/nvpair-engine-manager/spec.md b/services/nvpair-engine-manager/spec.mdindex 820089b..3150aaa 100644--- a/services/nvpair-engine-manager/spec.md+++ b/services/nvpair-engine-manager/spec.md@@ -18,7 +18,7 @@ A declarative, config-driven control plane for **local inference engines** (Olla- Expose all of the above over the `engine:*` JSON-RPC surface to whatever orchestrates the service, plus an optional plain-HTTP LAN endpoint (`--http-port`, `GET /v1/models`) that serves the model list to a peer's discovery daemon (the list moved off the size-limited mDNS TXT onto HTTP).**Out of scope**-- **Inference traffic** — stays with `ollama-proxy`; this service never proxies `/api/chat` etc.+- **Inference traffic** — stays with `nvpair-proxy`; this service never proxies `/api/chat` etc.- **Multi-instance per engine and an MCP server** — future-additive, not v1.- **The node's error list** — owned by `nvpair-errors`, which holds it as in-memory session state; this service only emits `errors:report` / `errors:clear`.diff --git a/services/nvpair-job-scheduler/README.md b/services/nvpair-job-scheduler/README.mdindex 8e81046..68090b0 100644--- a/services/nvpair-job-scheduler/README.md+++ b/services/nvpair-job-scheduler/README.md@@ -124,9 +124,8 @@ go test ./...- [`../nvpair-ui-broker/README.md`](../nvpair-ui-broker/README.md) — thesupervisor that feeds this component and relays its output-- [`../ollama-proxy/README.md`](../ollama-proxy/README.md) and- [`../lmstudio-proxy/README.md`](../lmstudio-proxy/README.md) — how the- ordering is consumed at request time+- [`../nvpair-proxy/README.md`](../nvpair-proxy/README.md) — how the ordering+ is consumed at request time- [`../nvpair-workload-manager/README.md`](../nvpair-workload-manager/README.md)— where workload state originates- [`../VERSIONING.md`](../VERSIONING.md) — SemVer bump rulesdiff --git a/services/nvpair-job-scheduler/schedule.go b/services/nvpair-job-scheduler/schedule.goindex 1afe2fe..2bfbf7f 100644--- a/services/nvpair-job-scheduler/schedule.go+++ b/services/nvpair-job-scheduler/schedule.go@@ -9,12 +9,13 @@ import ("sort""time"+ "nvpair-shared/engines""nvpair-shared/schedulerwire")-// schedulerEngines is the fixed set of engine-specific output contracts. Both-// receive the same node-wide ranking because their work shares node resources.-var schedulerEngines = []string{"ollama", "lmstudio"}+// schedulerEngines is the set of engine-specific output contracts. Every engine+// receives the same node-wide ranking because their work shares node resources.+var schedulerEngines = engines.Names()// NodeRank is retained as the scheduler's public status type while the wire// definition is shared with the broker and proxies.diff --git a/services/nvpair-job-scheduler/spec.md b/services/nvpair-job-scheduler/spec.mdindex ee31151..c461e92 100644--- a/services/nvpair-job-scheduler/spec.md+++ b/services/nvpair-job-scheduler/spec.md@@ -14,7 +14,7 @@ Decides **where inference should go**. On every accepted workload, discovery, oreffective GPU-pressure change, with a periodic timer as reconciliation, the JobScheduler ranks the cluster's nodes by total pending work plus GPU pressure. Itemits that node-wide order, pending count, and pressure so the Broker can push the-snapshot to the matching local proxy (`ollama-proxy`, `lmstudio-proxy`). Before+snapshot to the matching local `nvpair-proxy` instance. Beforeforwarding, a proxy atomically adds its own not-yet-observed reservations andchooses the least-busy capable node; its existing failover walks the remainingcandidates.@@ -235,16 +235,32 @@ Notifications:The new proxy method the *Broker* calls, carrying the scheduler's order (definedproxy-side, §4). The scheduler never calls it — it only emits `schedule:priority`.-- **Params**: `{ nodes: [string], ranks?: [NodeRank] }` — ordered node ids plus- their authoritative pending counts and GPU pressure.+- **Params**: `{ generation: uint64, nodes: [string], ranks?: [NodeRank] }` —+ a monotonic delivery generation, ordered node ids, and their authoritative+ pending counts and GPU pressure. The Broker mints `generation` (always ≥ 1);+ the scheduler does not send it. **Required**: omitted or `0` is rejected with+ `-32602`, because a snapshot that cleared the proxy's reservations without+ advancing its applied epoch would let one taken before it release one taken+ after — the undercount this field exists to prevent.- **Semantics** (proxy): for each auto-routed model-bearing inference request,atomically choose the minimum authoritative pending count plus GPU pressure- plus local reservations, then increment that node's reservation before- forwarding.+ plus local reservations, then reserve that node before forwarding.Consider only scheduler-listed nodes in the best available model tier; usescheduler order as the tie-break. **Unknown ids are ignored.** A **known nodenot in the list** remains a lowest-priority fallback, and an **empty list**reverts to the proxy's default auto heuristic.+- **Semantics** (proxy, generations): a snapshot is applied **at most once**. A+ `generation` less than or equal to the newest applied one is ignored whole.+ Applying a snapshot supersedes the reservations taken against earlier ones, so+ a redelivery must not clear reservations a second time — and a reservation+ released after a newer snapshot arrived is dropped rather than decremented,+ since that snapshot's pending counts already account for the completed work.+ Without both rules a node's estimated load can fall below zero, which reads as+ permanently idle and attracts every subsequent dispatch.+- **Semantics** (proxy, reservations): reservations are **process-wide**, shared+ by every engine facade in the proxy process — two facades bursting at once+ compete for the same node's GPU. A reservation is released when its request+ ends and moved to the node a failover actually lands on.- **Result**: the reply shape is defined proxy-side, not here.- **Precedence**: a user `node/select` pin overrides the list (§7.3).@@ -292,11 +308,17 @@ consumes the proxy's `workload:*` and routes it to the workload-manager(`routeProxyWorkload`); here it consumes `schedule:priority` and routes it to theproxies. **Not** a new "originated request" direction. Broker responsibilities(implemented in `nvpair-ui-broker`, §15):-- **Fan-out**: on `schedule:priority`, call `node/set-priority` on the matching- supervised proxy; an absent proxy → logged no-op.-- **Cache + restart-resync**: keep the last complete snapshot per engine and- re-push it when a proxy (re)spawns (the Broker owns the proxy lifecycle +- `ready`, so it knows when one came back empty).+- **Fan-out**: on `schedule:priority`, mint a generation and call+ `node/set-priority` once per **distinct live proxy handle** — every engine+ resolves to the same process, so a per-engine call would deliver the same+ snapshot twice. An absent proxy → logged no-op.+- **Ingestion dedupe**: a snapshot whose ranking matches the cached one is+ coalesced without minting a generation, so a redundant emission does not+ cause a redundant delivery.+- **Cache + restart-resync**: keep the last complete snapshot **process-wide**+ (the ranking is node-global and not engine-filtered — see §7.2) and re-push it+ when the proxy (re)spawns, from the supervisor's spawn hook rather than from+ `ready`, so the replay cannot race the handle's publication.- **Feed and seed the streams**: on scheduler spawn, replay active workloads,replay cached telemetry, then send discovery so the first non-empty ranking hasall baselines. Resume live workload, telemetry, and discovery fanout afterward.@@ -326,7 +348,7 @@ side.- **Upstream**: the supervising parent — `nvpair-ui-broker` in practice(supervisor-agnostic: any parent that speaks §7.0, feeds the three streams, anddelivers `schedule:priority` to the proxies).-- **Downstream (via the Broker)**: `ollama-proxy`/`lmstudio-proxy` (receive+- **Downstream (via the Broker)**: each `nvpair-proxy` instance (receives`node/set-priority`); `nvpair-errors` (receives `errors:*`). No direct link to any.- **Data sources (via the Broker)**: `nvpair-workload-manager` + proxies(workloads), `nvpair-node-scanner` / manual nodes (discovery and GPU telemetry).@@ -336,7 +358,8 @@ side.## 9. Data Ownership- **Owned**: transient in-memory state only — the accumulated workload catalog,discovered-node view, smoothed telemetry state, and last-emitted rank snapshot- per engine. Each proxy owns only its short-lived optimistic reservation deltas.+ per engine. The proxy process owns only its short-lived optimistic reservation+ deltas, shared across the engine facades it hosts.- **Source of truth**: no. Workloads → proxies/WM; discovery and telemetry →scanner/manual-nodes; the active routing decision → the proxy; delivery/caching→ the Broker. The scheduler owns only the policy computation.@@ -393,11 +416,13 @@ The Broker spawns `nvpair-job-scheduler`, replays active workloads and telemetrythen sends discovery and resumes all three live streams. Discovery seeds`GPU-RIG`, `MY-PC`, `LAB-DESK-B`. Pending counts are `3`, `0`, `1` and GPUpressures are `3`, `0`, `1`, so combined loads are `6`, `0`, `2`. The scheduler-emits `["MY-PC","LAB-DESK-B","GPU-RIG"]` for both engines. The Broker delivers-each via `node/set-priority`; each proxy applies the subset it can route to.+emits `["MY-PC","LAB-DESK-B","GPU-RIG"]` for both engines. The ranking is+node-global, so the Broker coalesces the duplicate and delivers it once per+distinct proxy process via `node/set-priority`; each facade applies the subset it+can route to.As load and smoothed pressure change, only a new pressure band, pending count, or-order triggers another snapshot. If `ollama-proxy` is not supervised, the Broker+order triggers another snapshot. If the Ollama proxy is not supervised, the Brokerdrops the push and re-delivers once it is back — the scheduler is unaware.## 15. Process model, CLI, and build wiring@@ -424,10 +449,10 @@ No flag carries node/cluster identity — the scheduler holds none.spawned + supervised (auto-restart with crash surfacing);- replay active workloads and cached telemetry on scheduler spawn, send thediscovery baseline, then fan all three live streams to the child;-- consume its `schedule:priority` and fan out to the proxies via `node/set-priority`,- caching the last snapshot per engine and re-pushing on proxy (re)spawn (§7.4) —- reuses the proxy `workload:*` → workload-manager routing pattern, not a new- direction;+- consume its `schedule:priority` and fan out via `node/set-priority` to each+ distinct live proxy handle, caching the last snapshot process-wide and+ re-pushing on proxy (re)spawn (§7.4) — reuses the proxy `workload:*` →+ workload-manager routing pattern, not a new direction;- forward its `errors:*` to `nvpair-errors`.**Build wiring**: `nvpair-job-scheduler` is one of the Go binaries in the productdiff --git a/services/nvpair-manual-nodes/README.md b/services/nvpair-manual-nodes/README.mdindex 5fb648a..52c5de7 100644--- a/services/nvpair-manual-nodes/README.md+++ b/services/nvpair-manual-nodes/README.md@@ -60,7 +60,7 @@ Emitted when a manually added node has been probed and its initial status determ}```-Each node is probed for both inference engines: Ollama on its default `:11434` (`GET /` + `/api/tags`) and LM Studio on its default `:1234` (`GET /v1/models`, which doubles as the liveness check and the model list). `lmstudio_up` / `lmstudio_port` / `lmstudio_models` mirror the `ollama_*` fields and let a supervising broker bridge the node into `lmstudio-proxy` the same way it bridges Ollama into `ollama-proxy`. A node can run either engine, both, or neither.+Each node is probed for both inference engines: Ollama on its default `:11434` (`GET /` + `/api/tags`) and LM Studio on its default `:1234` (`GET /v1/models`, which doubles as the liveness check and the model list). `lmstudio_up` / `lmstudio_port` / `lmstudio_models` mirror the `ollama_*` fields and let a supervising broker bridge the node into that engine's `nvpair-proxy` instance the same way it bridges Ollama into its own. A node can run either engine, both, or neither.### `node/updated`diff --git a/services/nvpair-proxy/README.md b/services/nvpair-proxy/README.mdnew file mode 100644index 0000000..731e6b9--- /dev/null+++ b/services/nvpair-proxy/README.md@@ -0,0 +1,251 @@+<!--+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+SPDX-License-Identifier: Apache-2.0+-->++# Engine Proxy++A discovery-aware HTTP reverse proxy that fronts every enabled inference engine.+It runs no mDNS browse of its own: routing targets come from the broker's+discovery relay (each facade sends `discovery:subscribe` for its engine's+service key and replaces its routing overlay from each pushed `discovery:nodes`+snapshot) plus user-added manual nodes. It forwards HTTP requests to the+selected node, aggregates the model-list routes across candidate nodes, and+exposes a bidirectional JSON-RPC 2.0 control channel over stdio (or an IPC+socket).++**One process, one facade per engine.** The process starts with no engine and no+listener; the broker asks for each engine's facade with `facade/enable`, which+carries that engine's port and alias addresses. A flag cannot express this,+because the broker plans a different port for each engine.++They share a process on purpose. Between scheduler snapshots a facade takes+short-lived reservations for work it has dispatched, and those live in the+process — a process per engine split that picture, so simultaneous bursts on+both engines could pick the same node believing it idle.++The cost is shared fate for the **process**: a crash takes every facade down and+the supervisor restarts them together. Smaller failures are contained to one+engine — a lost bind race, a port that cannot be re-listened, or a panic while+handling a request. Each of those used to end the process, which was fine when+the process was one engine.++`spec.md` is normative; where it and this file disagree, it wins. Everything+below is shared behavior unless a section says otherwise.++## Build++```bash+go build -o nvpair-proxy .+```++## Usage++```+nvpair-proxy [flags]+```++Then, over the control channel, for each engine to front:++```json+{"jsonrpc":"2.0","id":1,"method":"facade/enable",+ "params":{"engine":"ollama","port":11434,"ignorePersistedPort":true}}+```++The response is `{"engine":"ollama","port":11434}`, reporting the port actually+bound — enable is a request, and the child's persisted-port restore can override+the port asked for. Enabling an engine that is already up is an idempotent+success, so a redelivered enable never tears down a working listener.++### Flags++Only process-scoped settings are flags. Anything per-engine is a `facade/enable`+parameter, because one flag cannot carry two engines' plans.++| Flag | Default | Description |+|------|---------|-------------|+| `--ipc` | *(empty — use stdio)* | Path to a Unix domain socket or Windows named pipe for IPC |+| `--cluster-dir` | *(empty)* | Cluster trust directory (`node.crt`/`node.key` plus trusted pins). Enables the LAN mTLS inference ingress while this node is a cluster member; empty means no ingress and no peer candidates. |+| `--log-level` | *(`$NVPAIR_LOG_LEVEL`, else `info`)* | Initial log level: `debug`, `info`, `warn`, or `error`. Changeable at runtime with `log/set-level`. |+| `--version` | | Print version and exit |++### `facade/enable` parameters++| Field | Default | Description |+|-------|---------|-------------|+| `engine` | *(required)* | Which engine to front: `ollama` or `lmstudio`. An unknown name is rejected with the accepted values. |+| `port` | per engine, see below | HTTP listen port for request forwarding. Must be 1–65535, or omitted for the engine's standalone default. `0` means "the default" rather than "pick an ephemeral port", and any other out-of-range value is rejected, because the facade announces the requested port in its `ready` notification and the broker would be told `0`. |+| `aliasAddresses` | *(empty)* | Optional secondary `host:port` values for the same routing handler, one per loopback family so `localhost` resolves either way. Only literal loopback addresses are accepted; the broker uses this for a safe inherited local `OLLAMA_HOST`, and the aliases are not advertised to peers. Accepted only for an engine with an inherited host variable — today Ollama alone — and rejected for any other. |+| `ignorePersistedPort` | `false` | Use `port` even when a saved port exists (used by broker-managed startup) |++### What differs per engine++Everything engine-specific is one entry in `engines.go`, plus the shared+identity in `nvpair-shared/engines`.++| | `"engine":"ollama"` | `"engine":"lmstudio"` |+|---|---|---|+| Facade id — error-ID prefix, broker relay namespace, TUI proxies-view tab | `ollama-proxy` | `lmstudio-proxy` |+| Discovery service key | `ol` | `lm` |++The **log component, supervisor label, and TUI health crash key are not in that+table**: they name the process (`nvpair-proxy`), not a facade, because one+process hosts every engine and there is no single engine to attribute them to.+Facade-scoped log records carry an `engine` field instead. The supervisor label+and the health crash key are matched against each other, so they move together+— see `nvpair-shared/engines` and `spec.md` §9.++| | `"engine":"ollama"` | `"engine":"lmstudio"` |+|---|---|---|+| Engine's own client-facing port | 11434 | 1234 |+| Where PAIR relocates the engine | 11435 | 1235 |+| Standalone port, used when `port` is omitted | 11435 | 1234 |+| Persisted-port file (declared, not derived) | `proxy-port.json` | `lmstudio-proxy-port.json` |+| Model-list routes | `GET /api/tags` (native), `GET /v1/models` (OpenAI) | `GET /v1/models` (OpenAI) |+| Inference routes | `/api/generate`, `/api/chat`, `/api/embeddings`, `/api/embed`, plus the OpenAI set | `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings` |+| Model naming | untagged means `:latest`, so `llama3` and `llama3:latest` are one model | identifiers compared byte for byte |++The route table is a **classifier, not an allowlist**. An unlisted path is+forwarded verbatim, which is how `/api/show`, `/api/pull`, `/api/ps`,+`/api/version` and `OPTIONS` preflights keep working.++### HTTP Reverse Proxy++A facade listens on its enabled port and forwards incoming requests to the+currently active node — except the model-list routes, which are queried across+every candidate node concurrently and merged into one de-duplicated inventory.+Point your client at the proxy and it handles routing.++When the broker supplies `aliasAddresses`, the facade reserves that+loopback-only endpoint before reporting ready and serves it through the same+routing and workload-lifecycle handler. A `localhost` alias reserves `127.0.0.1`+and `::1` atomically so client resolution cannot bypass the router. An occupied+alias is non-fatal: its existing owner is untouched and the proxy reports an+actionable warning while the primary listener stays available.++**Cluster ingress.** The listener carries two personalities, demultiplexed by+each connection's first byte. Plaintext HTTP is accepted only from loopback; a+LAN caller is refused. When `--cluster-dir` shows this node is a cluster member,+the same listener also terminates cluster mTLS: a peer whose client certificate+matches one of this node's pins is forwarded straight to the local engine+reported by `node/set-local-backend`, and is never re-routed onward to another+node. Membership and pins are re-derived per request, so joining or leaving a+cluster needs no restart.++**Persisted port.** A port chosen at runtime via `set-port` is saved to the+per-user data dir (`%LocalAppData%\Nvidia Corporation\Personal AI Router` on+Windows, `~/.config/Nvidia Corporation/Personal AI Router` on Linux) and+**restored when the facade is enabled**, taking precedence over the requested+`port`/the default — so the proxy comes back up where it was last put. Each+engine has its own file, so moving one proxy never moves another.+`ignorePersistedPort` deliberately bypasses that restoration for a+broker-coordinated start. Delete the file (or+`set-port` back to the default) to revert.++LM Studio additionally refuses to restore port 1235 even if it is stored: that+is where engine-manager runs a managed LM Studio, so a proxy restoring it would+sit on the engine's own port. The stored value predates the current default.++**Browser clients (CORS).** The proxy is usable from a web front end. When an+engine is available, the proxy forwards an `OPTIONS` preflight so an+engine-declared exact origin and credentials policy reaches the browser+unchanged. If no engine is available, the engine returns no CORS policy, or a+non-loopback caller must be refused before routing, the proxy answers with its+own permissive `204` fallback. It also labels every response it generates —+including rejections such as the `502` when no node is available and the `403`+refusing a non-loopback plaintext caller — with `Access-Control-Allow-Origin: *`.+That matters as much as the success path: a response without those headers+reaches the browser as a generic "CORS error" with the real status and reason+stripped out, so the caller cannot tell what went wrong. A locally answered+preflight grants no access, since the request that follows still faces the same+gate. The policy comes from `nvpair-shared/cors`, so every engine answers+identically.++A response *forwarded from an engine* is different: if the engine set its own+`Access-Control-Allow-Origin` (Ollama with `OLLAMA_ORIGINS`), that header is+passed through untouched rather than replaced with the wildcard, so a+deliberately narrow engine policy is never silently widened and a credentialed+response is not broken. An engine that sends no CORS header has expressed no+policy to keep, so the proxy supplies its own — and drops any+`Access-Control-Allow-Credentials` that arrived without an origin, because a+browser rejects that header alongside a wildcard origin and would discard the+response the fallback exists to make readable.++One limit is outside the proxy's control: current Chromium-based browsers gate a+request from a public origin to a local or loopback address behind the user's+[Local Network Access](https://chromestatus.com/feature/5152728072060928)+permission, which replaced the old server-side opt-in header. No header the+proxy sends can grant that. A hosted page needs the permission plus a+`fetch(url, { targetAddressSpace: 'loopback' })` annotation; a page served from+the local machine is unaffected.++### Node selection++- **Eligibility**: Before routing model-bearing inference, the proxy keeps only+ nodes whose current inventory for this engine advertises the requested model,+ compared using the engine's naming rule. An empty or non-matching inventory is+ excluded until a later discovery update; if no advertised owner is routable,+ the proxy returns a local `502`.+- **Auto**: When no eligible node is explicitly selected, the proxy follows+ `node/set-priority`, then discovered nodes in stable ID order.+- **Priority (scheduler-driven)**: The Job Scheduler ranks the cluster+ least-loaded-first by pending workload plus smoothed GPU pressure and, via+ `nvpair-ui-broker`, pushes the ordered node list with those per-node counts to+ this proxy with `node/set-priority`. Auto routing sends the request to the+ listed node carrying the least estimated load. See+ [`nvpair-job-scheduler`](../nvpair-job-scheduler/README.md).+- **Manual**: `node/select` pins traffic to a specific node. A manual pin+ **overrides the priority list only when that node is eligible** for the+ requested model.+- **Failover**: If the selected node disappears from the discovery set, the+ proxy falls back to auto-select and emits `node/selection-changed`. A+ transport error or retryable status, including a model `404` from an+ advertised owner with stale inventory, steps to the next eligible owner.++### IPC Transport++By default the proxy communicates over **stdin/stdout** using newline-delimited+JSON-RPC 2.0 (one message per line). All diagnostic logging goes to **stderr**.++For environments where stdout may conflict with the host process (e.g. Electron),+pass `--ipc` to redirect the JSON-RPC channel to a named socket or pipe. The+parent process should create and listen on the endpoint before spawning the+proxy.++```bash+# Default: stdin/stdout+nvpair-proxy++# Unix domain socket+nvpair-proxy --ipc /tmp/nvpair-proxy.sock++# Windows named pipe+nvpair-proxy --ipc \\.\pipe\nvpair-proxy+```++## JSON-RPC 2.0 Protocol++The control channel is identical for every engine; only the broker's relay+prefix differs. The generated method surface — every request and notification,+with payload shapes — is in+[`desktop/docs/services-api.md`](../../desktop/docs/services-api.md), produced by+`npm --prefix desktop run service-contracts:write`. Read that rather than a+hand-maintained copy, which is how the two per-engine READMEs drifted.++The proxy emits `ready` on startup carrying its bound port, which is the+authoritative source of where it is listening — do not assume a port before it+reports one.++## Shutdown++Closing stdin (observed as EOF) drains the HTTP server and exits. The broker+owns the lifecycle: `shutdown` relayed from a client is refused.++## Adding an engine++Add one entry to `nvpair-shared/engines` for the cross-process identity, one to+`engines.go` here for the proxy-only values, and one to the broker's+`engineproxy.go` for its ownership. See the "Adding a new engine" section of the+proxy-unification plan for the full checklist, including the one-liners that are+still per-engine (a `noderec.ServiceKey`, the TUI health label, the desktop+engine-type constants, and an engine-manager manifest).diff --git a/services/nvpair-proxy/activity_test.go b/services/nvpair-proxy/activity_test.gonew file mode 100644index 0000000..cb575b3--- /dev/null+++ b/services/nvpair-proxy/activity_test.go@@ -0,0 +1,149 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++import (+ "fmt"+ "net/http"+ "net/http/httptest"+ "strings"+ "sync"+ "testing"+ "time"+)++// Response bytes arriving from a node's engine are the only liveness evidence+// that gets stronger as the node gets busier, which is exactly when discovery's+// probes start timing out and evicting it. These tests pin that the proxy+// actually raises it.++func waitFor(t *testing.T, rec *recRW, frame string) bool {+ t.Helper()+ deadline := time.Now().Add(2 * time.Second)+ for time.Now().Before(deadline) {+ if rec.has(frame) {+ return true+ }+ time.Sleep(5 * time.Millisecond)+ }+ return false+}++// TestStreamedBytesReportNodeActivity is the point of the feature: the node that+// served the response is named to the broker so discovery can keep it.+func TestStreamedBytesReportNodeActivity(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.WriteHeader(http.StatusOK)+ _, _ = w.Write([]byte(`{"done":true}`))+ }))+ defer upstream.Close()++ rec := &recRW{}+ disc := NewDiscovery()+ disc.AddManual(nodeForModel(t, "serving-node", upstream.URL, tc.advertisedModel))+ p := newTestProxy(tc.profile, NewCodec(rec), disc, tc.profile.FacadePort)++ p.soleFacade().handleHTTP(httptest.NewRecorder(), httptest.NewRequest(+ http.MethodPost, tc.inferencePath,+ strings.NewReader(fmt.Sprintf(`{"model":%q}`, tc.requestedModel))))++ if !waitFor(t, rec, `"method":"node/activity"`) {+ t.Fatal("no node/activity was reported after the upstream streamed a response")+ }+ if !waitFor(t, rec, `"hostUuid":"serving-node"`) {+ t.Fatal("node/activity did not name the node that served the request")+ }+ })+}++// A node that accepted the connection but never wrote a byte has proved nothing+// about being able to do work, so it must not be vouched for. This is the case+// that matters: an accept can succeed on a machine whose user space is wedged.+func TestNoActivityReportedWithoutUpstreamBytes(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ release := make(chan struct{})+ var releaseOnce sync.Once+ doRelease := func() { releaseOnce.Do(func() { close(release) }) }++ received := make(chan struct{}, 1)+ silent := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ select {+ case received <- struct{}{}:+ default:+ }+ <-release+ w.WriteHeader(http.StatusOK)+ }))+ // LIFO: unblock the handler before tearing the server down, or Close+ // hangs on the live connection.+ defer silent.Close()+ defer doRelease()++ rec := &recRW{}+ disc := NewDiscovery()+ disc.AddManual(nodeForModel(t, "silent-node", silent.URL, tc.advertisedModel))+ p := newTestProxy(tc.profile, NewCodec(rec), disc, tc.profile.FacadePort)++ done := make(chan struct{})+ go func() {+ defer close(done)+ p.soleFacade().handleHTTP(httptest.NewRecorder(), httptest.NewRequest(+ http.MethodPost, tc.inferencePath,+ strings.NewReader(fmt.Sprintf(`{"model":%q}`, tc.requestedModel))))+ }()++ select {+ case <-received:+ case <-time.After(5 * time.Second):+ t.Fatal("upstream never received the forwarded request")+ }++ // Give the proxy the same window the positive test uses, so a report+ // would have landed by now if one were going to.+ time.Sleep(200 * time.Millisecond)+ if rec.has(`"method":"node/activity"`) {+ t.Fatal("activity was reported for a node that had not sent a single response byte")+ }++ doRelease()+ <-done+ })+}++// A streaming generation writes hundreds of chunks. Reporting each one would put+// thousands of frames a minute on the broker pipe for no gain, since the scanner+// treats one report as good for a minute.+func TestRepeatedChunksAreCoalescedIntoOneReport(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ chunks := 40+ upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.Header().Set("Content-Type", "application/x-ndjson")+ w.WriteHeader(http.StatusOK)+ for i := 0; i < chunks; i++ {+ _, _ = w.Write([]byte(`{"response":"x"}` + "\n"))+ if f, ok := w.(http.Flusher); ok {+ f.Flush()+ }+ }+ }))+ defer upstream.Close()++ rec := &recRW{}+ disc := NewDiscovery()+ disc.AddManual(nodeForModel(t, "streaming-node", upstream.URL, tc.advertisedModel))+ p := newTestProxy(tc.profile, NewCodec(rec), disc, tc.profile.FacadePort)++ p.soleFacade().handleHTTP(httptest.NewRecorder(), httptest.NewRequest(+ http.MethodPost, tc.inferencePath,+ strings.NewReader(fmt.Sprintf(`{"model":%q,"stream":true}`, tc.requestedModel))))++ if !waitFor(t, rec, `"method":"node/activity"`) {+ t.Fatal("a streamed response reported no activity at all")+ }+ if got := rec.count(`"method":"node/activity"`); got != 1 {+ t.Fatalf("%d activity reports for %d chunks; want 1 within the throttle interval", got, chunks)+ }+ })+}diff --git a/services/ollama-proxy/alias_test.go b/services/nvpair-proxy/alias_test.gosimilarity index 73%rename from services/ollama-proxy/alias_test.gorename to services/nvpair-proxy/alias_test.goindex 9b15732..9d3e675 100644--- a/services/ollama-proxy/alias_test.go+++ b/services/nvpair-proxy/alias_test.go@@ -32,11 +32,11 @@ func TestSetLoopbackAliasValidation(t *testing.T) {{"localhost:0", true, ""},{"not-an-address", true, ""},} {- p := NewProxy(NewCodec(rwNop{}), NewDiscovery(), 11434)- if err := p.setLoopbackAlias(tc.address); (err != nil) != tc.wantErr {+ p := newTestProxy(ollamaOnlyProfile(t), NewCodec(rwNop{}), NewDiscovery(), 11434)+ if err := p.soleFacade().setLoopbackAlias(tc.address); (err != nil) != tc.wantErr {t.Errorf("setLoopbackAlias(%q) error = %v, wantErr %v", tc.address, err, tc.wantErr)- } else if err == nil && p.aliasAddr != tc.wantAddr {- t.Errorf("setLoopbackAlias(%q) address = %q, want %q", tc.address, p.aliasAddr, tc.wantAddr)+ } else if err == nil && p.soleFacade().aliasAddr != tc.wantAddr {+ t.Errorf("setLoopbackAlias(%q) address = %q, want %q", tc.address, p.soleFacade().aliasAddr, tc.wantAddr)}}}@@ -54,19 +54,19 @@ func TestLoopbackAliasUsesPrimaryRouterAndSurvivesPrimaryRebind(t *testing.T) {disc.AddManual(nodeForModel(t, "upstream", upstream.URL, "test"))primaryPort := freeTCPPort(t)aliasPort := freeTCPPort(t)- p := NewProxy(NewCodec(rec), disc, primaryPort)- if err := p.setLoopbackAlias(fmt.Sprintf("127.0.0.1:%d", aliasPort)); err != nil {+ p := newTestProxy(ollamaOnlyProfile(t), NewCodec(rec), disc, primaryPort)+ if err := p.soleFacade().setLoopbackAlias(fmt.Sprintf("127.0.0.1:%d", aliasPort)); err != nil {t.Fatal(err)}- p.bindLoopbackAlias()+ p.soleFacade().bindLoopbackAlias()primary, err := net.Listen("tcp", fmt.Sprintf(":%d", primaryPort))if err != nil {t.Fatal(err)}ctx, cancel := context.WithCancel(context.Background())defer cancel()- p.serveHTTP(ctx, primary)- p.serveLoopbackAlias()+ p.soleFacade().serveHTTP(ctx, primary)+ p.soleFacade().serveLoopbackAlias()defer p.shutdown(context.Background())assertRouted := func(port int) {@@ -88,15 +88,15 @@ func TestLoopbackAliasUsesPrimaryRouterAndSurvivesPrimaryRebind(t *testing.T) {}newPrimary := freeTCPPort(t)- if err := p.setPort(newPrimary); err != nil {+ if err := p.soleFacade().setPort(newPrimary); err != nil {t.Fatalf("rebind primary: %v", err)}assertRouted(newPrimary)assertRouted(aliasPort)- p.httpMu.Lock()- bound := p.aliasLn.Addr().(*net.TCPAddr).IP- p.httpMu.Unlock()+ p.soleFacade().httpMu.Lock()+ bound := p.soleFacade().aliasLn.Addr().(*net.TCPAddr).IP+ p.soleFacade().httpMu.Unlock()if !bound.IsLoopback() {t.Fatalf("alias bound non-loopback address %s", bound)}@@ -113,19 +113,19 @@ func TestOccupiedLoopbackAliasLeavesOwnerAndPrimaryRunning(t *testing.T) {aliasPort := owner.Listener.Addr().(*net.TCPAddr).PortprimaryPort := freeTCPPort(t)- p := NewProxy(NewCodec(rec), NewDiscovery(), primaryPort)- if err := p.setLoopbackAlias(fmt.Sprintf("127.0.0.1:%d", aliasPort)); err != nil {+ p := newTestProxy(ollamaOnlyProfile(t), NewCodec(rec), NewDiscovery(), primaryPort)+ if err := p.soleFacade().setLoopbackAlias(fmt.Sprintf("127.0.0.1:%d", aliasPort)); err != nil {t.Fatal(err)}- p.bindLoopbackAlias()+ p.soleFacade().bindLoopbackAlias()primary, err := net.Listen("tcp", fmt.Sprintf(":%d", primaryPort))if err != nil {t.Fatal(err)}ctx, cancel := context.WithCancel(context.Background())defer cancel()- p.serveHTTP(ctx, primary)- p.serveLoopbackAlias()+ p.soleFacade().serveHTTP(ctx, primary)+ p.soleFacade().serveLoopbackAlias()defer p.shutdown(context.Background())resp, err := http.Get(owner.URL)@@ -159,13 +159,13 @@ func TestAliasPortIsAProxySelfTarget(t *testing.T) {disc := NewDiscovery()disc.AddManual(Node{ID: "alias-self", Addresses: []string{"127.0.0.1"}, Port: aliasPort})disc.AddManual(Node{ID: "real", Addresses: []string{"192.0.2.10"}, Port: primaryPort})- p := NewProxy(NewCodec(rwNop{}), disc, primaryPort)- if err := p.setLoopbackAlias(fmt.Sprintf("127.0.0.1:%d", aliasPort)); err != nil {+ p := newTestProxy(ollamaOnlyProfile(t), NewCodec(rwNop{}), disc, primaryPort)+ if err := p.soleFacade().setLoopbackAlias(fmt.Sprintf("127.0.0.1:%d", aliasPort)); err != nil {t.Fatal(err)}- p.bindLoopbackAlias()- defer p.closeLoopbackAlias()- candidates := p.resolveCandidates("")+ p.soleFacade().bindLoopbackAlias()+ defer p.soleFacade().closeLoopbackAlias()+ candidates := p.soleFacade().resolveCandidates("")for _, candidate := range candidates {if candidate.id == "alias-self" {t.Fatalf("alias endpoint survived the proxy self-target guard: %+v", candidates)@@ -177,9 +177,14 @@ func TestAliasPortIsAProxySelfTarget(t *testing.T) {}func TestAliasSelfTargetMatchesBoundLoopbackAddressNotPortAlone(t *testing.T) {+ // Proving the guard matches the bound address rather than the port alone+ // needs a second loopback address. Linux aliases all of 127.0.0.0/8 to lo+ // so this runs there and in CI; macOS binds only 127.0.0.1 unless an alias+ // was added by hand, and there is no way to make the distinction without+ // one. Skip rather than fail, so the suite is green on a Mac.probe, err := net.Listen("tcp", "127.0.0.2:0")if err != nil {- t.Fatalf("reserve 127.0.0.2 test port: %v", err)+ t.Skipf("no second loopback address available on this host: %v", err)}aliasPort := probe.Addr().(*net.TCPAddr).Port_ = probe.Close()@@ -191,14 +196,14 @@ func TestAliasSelfTargetMatchesBoundLoopbackAddressNotPortAlone(t *testing.T) {for primaryPort == aliasPort {primaryPort = freeTCPPort(t)}- p := NewProxy(NewCodec(rwNop{}), disc, primaryPort)- if err := p.setLoopbackAlias(fmt.Sprintf("127.0.0.2:%d", aliasPort)); err != nil {+ p := newTestProxy(ollamaOnlyProfile(t), NewCodec(rwNop{}), disc, primaryPort)+ if err := p.soleFacade().setLoopbackAlias(fmt.Sprintf("127.0.0.2:%d", aliasPort)); err != nil {t.Fatal(err)}- p.bindLoopbackAlias()- defer p.closeLoopbackAlias()+ p.soleFacade().bindLoopbackAlias()+ defer p.soleFacade().closeLoopbackAlias()- candidates := p.resolveCandidates("")+ candidates := p.soleFacade().resolveCandidates("")if len(candidates) != 1 || candidates[0].id != "same-port-other-address" {t.Fatalf("candidates = %+v, want only same-port-other-address", candidates)}@@ -215,16 +220,16 @@ func TestFailedAliasBindDoesNotClaimOwnersEndpointAsSelf(t *testing.T) {for primaryPort == aliasPort {primaryPort = freeTCPPort(t)}- p := NewProxy(NewCodec(rwNop{}), disc, primaryPort)- if err := p.setLoopbackAlias(fmt.Sprintf("127.0.0.1:%d", aliasPort)); err != nil {+ p := newTestProxy(ollamaOnlyProfile(t), NewCodec(rwNop{}), disc, primaryPort)+ if err := p.soleFacade().setLoopbackAlias(fmt.Sprintf("127.0.0.1:%d", aliasPort)); err != nil {t.Fatal(err)}- p.bindLoopbackAlias()- if p.aliasLn != nil {+ p.soleFacade().bindLoopbackAlias()+ if p.soleFacade().aliasLn != nil {t.Fatal("alias unexpectedly bound over the existing owner")}- candidates := p.resolveCandidates("")+ candidates := p.soleFacade().resolveCandidates("")if len(candidates) != 1 || candidates[0].id != "owner" {t.Fatalf("candidates = %+v, want existing owner retained", candidates)}@@ -279,16 +284,16 @@ func TestLoopbackAliasOwnsBothLocalhostFamilies(t *testing.T) {t.Fatal("could not find a port free on both loopback families")}- p := NewProxy(NewCodec(rwNop{}), NewDiscovery(), freeTCPPort(t))- if err := p.setLoopbackAlias(fmt.Sprintf("127.0.0.1:%d", port)); err != nil {+ p := newTestProxy(ollamaOnlyProfile(t), NewCodec(rwNop{}), NewDiscovery(), freeTCPPort(t))+ if err := p.soleFacade().setLoopbackAlias(fmt.Sprintf("127.0.0.1:%d", port)); err != nil {t.Fatal(err)}- if err := p.setLoopbackAlias(fmt.Sprintf("[::1]:%d", port)); err != nil {+ if err := p.soleFacade().setLoopbackAlias(fmt.Sprintf("[::1]:%d", port)); err != nil {t.Fatal(err)}- p.bindLoopbackAlias()- defer p.closeLoopbackAlias()- if p.aliasLn == nil || p.aliasAltLn == nil {+ p.soleFacade().bindLoopbackAlias()+ defer p.soleFacade().closeLoopbackAlias()+ if p.soleFacade().aliasLn == nil || p.soleFacade().aliasAltLn == nil {t.Fatal("localhost alias did not reserve both IPv4 and IPv6 loopback")}for _, address := range []string{@@ -316,15 +321,15 @@ func TestDualAliasBindIsAtomic(t *testing.T) {}_ = probe.Close()- p := NewProxy(NewCodec(rwNop{}), NewDiscovery(), freeTCPPort(t))- if err := p.setLoopbackAlias(fmt.Sprintf("127.0.0.1:%d", port)); err != nil {+ p := newTestProxy(ollamaOnlyProfile(t), NewCodec(rwNop{}), NewDiscovery(), freeTCPPort(t))+ if err := p.soleFacade().setLoopbackAlias(fmt.Sprintf("127.0.0.1:%d", port)); err != nil {t.Fatal(err)}- if err := p.setLoopbackAlias(fmt.Sprintf("[::1]:%d", port)); err != nil {+ if err := p.soleFacade().setLoopbackAlias(fmt.Sprintf("[::1]:%d", port)); err != nil {t.Fatal(err)}- p.bindLoopbackAlias()- if p.aliasLn != nil || p.aliasAltLn != nil {+ p.soleFacade().bindLoopbackAlias()+ if p.soleFacade().aliasLn != nil || p.soleFacade().aliasAltLn != nil {t.Fatal("partial localhost ownership survived an alternate-family bind failure")}rebound, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", port))diff --git a/services/ollama-proxy/choose_reachable_test.go b/services/nvpair-proxy/choose_reachable_test.gosimilarity index 90%rename from services/ollama-proxy/choose_reachable_test.gorename to services/nvpair-proxy/choose_reachable_test.goindex 2542719..535e323 100644--- a/services/ollama-proxy/choose_reachable_test.go+++ b/services/nvpair-proxy/choose_reachable_test.go@@ -29,7 +29,7 @@ func waitForTarget(t *testing.T, p *Proxy, n Node, want string) {t.Helper()deadline := time.Now().Add(2 * time.Second)for {- u := p.targetURL(n)+ u := p.soleFacade().targetURL(n)if u != nil && u.Host == want {return}@@ -49,7 +49,7 @@ func waitForTarget(t *testing.T, p *Proxy, n Node, want string) {// confirmation behaviour without opening sockets.func countingChooser(p *Proxy, accept bool) *atomic.Int32 {var dials atomic.Int32- p.targets = reach.NewChooser(reach.WithDial(+ p.soleFacade().targets = reach.NewChooser(reach.WithDial(func(_, _ string, _ time.Duration) (net.Conn, error) {dials.Add(1)if !accept {@@ -68,10 +68,10 @@ func TestChooseReachableFailsOverForPinnedPeer(t *testing.T) {clusterDir := filepath.Join(t.TempDir(), "cluster")clustertrusttest.Join(t, clusterDir, "cluster-xyz", "principal-self", peerUUID)- p := testProxy(NewDiscovery(), 11435)+ p := testProxy(anyProfile(t), NewDiscovery(), 11435)p.mesh = clustertrust.Open(clusterDir)var dials atomic.Int32- p.targets = reach.NewChooser(reach.WithDial(+ p.soleFacade().targets = reach.NewChooser(reach.WithDial(func(_, address string, _ time.Duration) (net.Conn, error) {dials.Add(1)host, _, _ := net.SplitHostPort(address)@@ -91,7 +91,7 @@ func TestChooseReachableFailsOverForPinnedPeer(t *testing.T) {}// The first request is not made to wait for the confirmation, so it uses the// node's own top-ranked address; the ones behind it use the one that answers.- if u := p.targetURL(n); u == nil || u.Host != net.JoinHostPort("192.0.2.10", "11434") {+ if u := p.soleFacade().targetURL(n); u == nil || u.Host != net.JoinHostPort("192.0.2.10", "11434") {t.Fatalf("first selection = %v, want the published ranking without waiting", u)}waitForTarget(t, p, n, net.JoinHostPort(reachable, "11434"))@@ -101,14 +101,14 @@ func TestChooseReachableFailsOverForPinnedPeer(t *testing.T) {}func TestChooseReachableProbesPlainMultiHomed(t *testing.T) {- p := testProxy(NewDiscovery(), 11435)+ p := testProxy(anyProfile(t), NewDiscovery(), 11435)dials := countingChooser(p, true)n := Node{ID: "manual-a",Port: 11434,Addresses: []string{"192.0.2.10", "192.0.2.11"},}- if u := p.targetURL(n); u == nil {+ if u := p.soleFacade().targetURL(n); u == nil {t.Fatal("targetURL returned nil")}deadline := time.Now().Add(2 * time.Second)@@ -125,8 +125,8 @@ func TestChooseReachableProbesPlainMultiHomed(t *testing.T) {// reach must still be routed to, at the address that answers.func TestTargetURLFailsOverToAReachableAddress(t *testing.T) {const reachable = "10.172.55.129"- p := testProxy(NewDiscovery(), 11435)- p.targets = reach.NewChooser(reach.WithDial(+ p := testProxy(anyProfile(t), NewDiscovery(), 11435)+ p.soleFacade().targets = reach.NewChooser(reach.WithDial(func(_, address string, _ time.Duration) (net.Conn, error) {host, _, _ := net.SplitHostPort(address)if host != reachable {@@ -184,7 +184,7 @@ func (f *fakeNetwork) accept(address string) {}func (f *fakeNetwork) install(p *Proxy) {- p.targets = reach.NewChooser(reach.WithDial(+ p.soleFacade().targets = reach.NewChooser(reach.WithDial(func(_, address string, _ time.Duration) (net.Conn, error) {f.dials.Add(1)host, _, _ := net.SplitHostPort(address)@@ -214,7 +214,7 @@ func confirmedDeadPeer(t *testing.T, replacement string) (*Proxy, Node, *fakeNetdisc := NewDiscovery()disc.AddManual(n)- p := testProxy(disc, 11435)+ p := testProxy(anyProfile(t), disc, 11435)fake := &fakeNetwork{accepting: confirmed}fake.install(p)@@ -246,7 +246,7 @@ func TestUpstreamTransportFailureReprobesTheNextSelection(t *testing.T) {p, n, fake := confirmedDeadPeer(t, replacement)rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)))+ p.soleFacade().handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)))if rec.Code != http.StatusBadGateway {t.Fatalf("status = %d, want 502 from the only, unreachable candidate", rec.Code)}@@ -262,7 +262,7 @@ func TestModelListTransportFailureReprobesTheNextSelection(t *testing.T) {p, n, fake := confirmedDeadPeer(t, replacement)rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/tags", nil))+ p.soleFacade().handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/tags", nil))if rec.Code != http.StatusServiceUnavailable {t.Fatalf("status = %d, want 503 when the only inventory source is unreachable", rec.Code)}diff --git a/services/ollama-proxy/cluster_trust_test.go b/services/nvpair-proxy/cluster_trust_test.gosimilarity index 90%rename from services/ollama-proxy/cluster_trust_test.gorename to services/nvpair-proxy/cluster_trust_test.goindex b17c423..3835204 100644--- a/services/ollama-proxy/cluster_trust_test.go+++ b/services/nvpair-proxy/cluster_trust_test.go@@ -29,11 +29,11 @@ func TestResolveCandidatesFollowsLivePinSet(t *testing.T) {IP: "192.0.2.10",ClusterUUID: peerUUID,}})- p := testProxy(disc, 11435)+ p := testProxy(anyProfile(t), disc, 11435)p.mesh = clustertrust.Open(clusterDir)// Pre-join: no identity, no pins, so the peer is not a routable target.- if got := p.resolveCandidates(""); len(got) != 0 {+ if got := p.soleFacade().resolveCandidates(""); len(got) != 0 {t.Fatalf("pre-join candidates = %+v, want none", got)}@@ -41,7 +41,7 @@ func TestResolveCandidatesFollowsLivePinSet(t *testing.T) {// new discovery snapshot arrives — the peer's mDNS record has not changed.clustertrusttest.Join(t, clusterDir, "cluster-xyz", "principal-self", peerUUID)- cands := p.resolveCandidates("")+ cands := p.soleFacade().resolveCandidates("")if len(cands) != 1 {t.Fatalf("post-join candidates = %+v, want the peer", cands)}@@ -52,7 +52,7 @@ func TestResolveCandidatesFollowsLivePinSet(t *testing.T) {// Removing the peer from the cluster retires it as a target just as promptly,// again with no discovery event involved.clustertrusttest.RemovePeerPin(t, clusterDir, peerUUID)- if got := p.resolveCandidates(""); len(got) != 0 {+ if got := p.soleFacade().resolveCandidates(""); len(got) != 0 {t.Fatalf("post-removal candidates = %+v, want none", got)}}@@ -72,10 +72,10 @@ func TestResolveCandidatesRejectsUnpinnedClusteredPeer(t *testing.T) {IP: "192.0.2.30",ClusterUUID: "principal-stranger",}})- p := testProxy(disc, 11435)+ p := testProxy(anyProfile(t), disc, 11435)p.mesh = clustertrust.Open(clusterDir)- if got := p.resolveCandidates(""); len(got) != 0 {+ if got := p.soleFacade().resolveCandidates(""); len(got) != 0 {t.Fatalf("candidates = %+v, want none for a peer in another cluster", got)}}diff --git a/services/lmstudio-proxy/codec.go b/services/nvpair-proxy/codec.gosimilarity index 100%rename from services/lmstudio-proxy/codec.gorename to services/nvpair-proxy/codec.godiff --git a/services/ollama-proxy/discovery.go b/services/nvpair-proxy/discovery.gosimilarity index 100%rename from services/ollama-proxy/discovery.gorename to services/nvpair-proxy/discovery.godiff --git a/services/nvpair-proxy/e2e_test.go b/services/nvpair-proxy/e2e_test.gonew file mode 100644index 0000000..27579a9--- /dev/null+++ b/services/nvpair-proxy/e2e_test.go@@ -0,0 +1,325 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++import (+ "bufio"+ "encoding/json"+ "fmt"+ "io"+ "net"+ "net/http"+ "net/http/httptest"+ "os"+ "os/exec"+ "path/filepath"+ "runtime"+ "strconv"+ "strings"+ "sync"+ "testing"+ "time"++ "nvpair-shared/engines"+)++// The real binary, built on first use rather than in TestMain.+//+// A TestMain that builds unconditionally charges the go build to all ~130+// tests in this package, and moving the whole file behind a build tag — the+// obvious alternative — would take the only end-to-end test of the shipped+// artifact out of the CI gate entirely, in the change that replaces that+// artifact. A sync.Once pays the cost once, only when an e2e case runs.+var (+ proxyBinOnce sync.Once+ proxyBinPath string+ proxyBinErr error+)++func proxyBinary(t *testing.T) string {+ t.Helper()+ proxyBinOnce.Do(func() {+ tmp, err := os.MkdirTemp("", "nvpair-proxy-e2e-*")+ if err != nil {+ proxyBinErr = err+ return+ }+ suffix := ""+ if runtime.GOOS == "windows" {+ suffix = ".exe"+ }+ proxyBinPath = filepath.Join(tmp, "nvpair-proxy"+suffix)+ if out, err := exec.Command("go", "build", "-o", proxyBinPath, ".").CombinedOutput(); err != nil {+ proxyBinErr = fmt.Errorf("build nvpair-proxy: %w\n%s", err, out)+ }+ })+ if proxyBinErr != nil {+ t.Fatal(proxyBinErr)+ }+ return proxyBinPath+}++type e2eFrame struct {+ ID json.RawMessage `json:"id"`+ Method string `json:"method"`+ Params json.RawMessage `json:"params"`+ Result json.RawMessage `json:"result"`+ Error json.RawMessage `json:"error"`+}++func e2eReadFrames(r io.Reader, out chan<- e2eFrame) {+ sc := bufio.NewScanner(r)+ sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)+ for sc.Scan() {+ var f e2eFrame+ if err := json.Unmarshal(sc.Bytes(), &f); err != nil {+ continue+ }+ out <- f+ }+}++func e2eSend(t *testing.T, w io.Writer, id int, method string, params any) {+ t.Helper()+ msg := map[string]any{"jsonrpc": "2.0", "id": id, "method": method}+ if params != nil {+ msg["params"] = params+ }+ data, _ := json.Marshal(msg)+ if _, err := w.Write(append(data, '\n')); err != nil {+ t.Fatalf("send %s: %v", method, err)+ }+}++func e2eWaitResult(t *testing.T, frames <-chan e2eFrame, id string, timeout time.Duration) {+ t.Helper()+ deadline := time.After(timeout)+ for {+ select {+ case f := <-frames:+ if string(f.ID) != id {+ continue+ }+ if len(f.Error) > 0 && string(f.Error) != "null" {+ t.Fatalf("rpc id %s returned error: %s", id, f.Error)+ }+ return+ case <-deadline:+ t.Fatalf("timed out waiting for response id %s", id)+ }+ }+}++// e2eEnableWithRetry enables a facade, trying a fresh port whenever the child+// reports a lost bind race, and returns the port it actually bound.+func e2eEnableWithRetry(t *testing.T, stdin io.Writer, frames <-chan e2eFrame, engine string, port int) int {+ t.Helper()+ for attempt := 0; attempt < 8; attempt++ {+ id := 100 + attempt+ e2eSend(t, stdin, id, "facade/enable", map[string]any{+ "engine": engine,+ "port": port,+ "ignorePersistedPort": true,+ })+ bound, bindRace := e2eEnabledPort(t, frames, fmt.Sprint(id), 10*time.Second)+ if bindRace {+ port = e2eFreePort(t)+ continue+ }+ if bound != port {+ t.Fatalf("enabled port = %d, want %d", bound, port)+ }+ return port+ }+ t.Fatalf("enable %s: every probed port was taken before the child could bind", engine)+ return 0+}++// e2eEnabledPort reads the port a facade/enable bound, reporting bindRace when+// the child refused because the port was taken. Only a bind race is retryable;+// any other rejection is a real failure.+func e2eEnabledPort(t *testing.T, frames <-chan e2eFrame, id string, timeout time.Duration) (port int, bindRace bool) {+ t.Helper()+ deadline := time.After(timeout)+ for {+ select {+ case f := <-frames:+ if string(f.ID) != id {+ continue+ }+ if len(f.Error) > 0 && string(f.Error) != "null" {+ var rpcErr struct {+ Code int `json:"code"`+ }+ if json.Unmarshal(f.Error, &rpcErr) == nil && rpcErr.Code == codeFacadeBindFailed {+ return 0, true+ }+ t.Fatalf("facade/enable returned error: %s", f.Error)+ }+ var res struct {+ Port int `json:"port"`+ }+ if err := json.Unmarshal(f.Result, &res); err != nil {+ t.Fatalf("parse facade/enable result: %v", err)+ }+ return res.Port, false+ case <-deadline:+ t.Fatalf("timed out waiting for facade/enable response id %s", id)+ }+ }+}++func e2eWaitReadyPort(t *testing.T, frames <-chan e2eFrame, timeout time.Duration) int {+ t.Helper()+ deadline := time.After(timeout)+ for {+ select {+ case f := <-frames:+ if f.Method != "ready" {+ continue+ }+ var p struct {+ Port int `json:"port"`+ }+ if err := json.Unmarshal(f.Params, &p); err != nil {+ t.Fatalf("parse ready params: %v", err)+ }+ return p.Port+ case <-deadline:+ t.Fatalf("timed out waiting for ready notification")+ return 0+ }+ }+}++// e2eFreePort returns a port that was free a moment ago. A process spawn sits+// between this probe and the child's bind, so the caller retries — see+// e2eEnableWithRetry.+func e2eFreePort(t *testing.T) int {+ t.Helper()+ l, err := net.Listen("tcp", "127.0.0.1:0")+ if err != nil {+ t.Fatal(err)+ }+ defer l.Close()+ return l.Addr().(*net.TCPAddr).Port+}++func e2eSplitHostPort(t *testing.T, serverURL string) (string, int) {+ t.Helper()+ host, portStr, err := net.SplitHostPort(strings.TrimPrefix(serverURL, "http://"))+ if err != nil {+ t.Fatalf("split %q: %v", serverURL, err)+ }+ port, err := strconv.Atoi(portStr)+ if err != nil {+ t.Fatalf("port %q: %v", portStr, err)+ }+ return host, port+}++// TestE2EFailoverOverRealBinary spawns the real binary and drives it the way+// the broker and UI do: register a busy (503) and a healthy (200) upstream as+// manual nodes over JSON-RPC stdio, then send a genuine inference POST to the+// proxy's real HTTP port. It asserts the request fails over from the busy node+// to the healthy one, the original body is replayed, and CORS headers are+// present — the whole shipped path end-to-end, no mocks.+//+// Only LM Studio had this. Running it per engine also makes it the one place+// Ollama's implied-:latest naming is proven end-to-end: the nodes advertise+// the tagged spelling while the client asks for the untagged one, so a+// regression in normalizeModel shows up as an unroutable request rather than+// a unit-test diff.+func TestE2EFailoverOverRealBinary(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ bin := proxyBinary(t)++ var gotBody string+ busy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.WriteHeader(http.StatusServiceUnavailable)+ }))+ defer busy.Close()+ good := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ b, _ := io.ReadAll(r.Body)+ gotBody = string(b)+ w.WriteHeader(http.StatusOK)+ io.WriteString(w, `{"ok":true}`)+ }))+ defer good.Close()++ port := e2eFreePort(t)+ // No engine on argv: the binary starts with no facade and no listener,+ // and the broker chooses what it fronts over facade/enable. This is the+ // bring-up sequence exercised end to end.+ cmd := exec.Command(bin)+ stdin, err := cmd.StdinPipe()+ if err != nil {+ t.Fatal(err)+ }+ stdout, err := cmd.StdoutPipe()+ if err != nil {+ t.Fatal(err)+ }+ if err := cmd.Start(); err != nil {+ t.Fatal(err)+ }+ defer func() {+ _ = stdin.Close()+ _ = cmd.Process.Kill()+ _ = cmd.Wait()+ }()++ frames := make(chan e2eFrame, 256)+ go e2eReadFrames(stdout, frames)++ // Retried on a bind race: e2eFreePort can only probe-then-close, and a+ // process spawn sits between the probe and the child's bind, so the+ // port can be taken in between. Treating the probe as authoritative is+ // what made this flake on "address already in use".+ port = e2eEnableWithRetry(t, stdin, frames, tc.profile.Name, port)++ // Facade-scoped methods are addressed to their engine, the same way the+ // broker addresses them: an unaddressed one resolves to no facade+ // rather than to "the only one", so it cannot be misrouted later.+ addressed := func(method string) string {+ return engines.AddressMethod(tc.profile.Name, method)+ }+ busyHost, busyPort := e2eSplitHostPort(t, busy.URL)+ goodHost, goodPort := e2eSplitHostPort(t, good.URL)+ e2eSend(t, stdin, 1, addressed("node/add-manual"), map[string]any{+ "id": "busy", "host": busyHost, "port": busyPort,+ "addresses": []string{busyHost}, "models": []string{tc.advertisedModel},+ })+ e2eWaitResult(t, frames, "1", 5*time.Second)+ e2eSend(t, stdin, 2, addressed("node/add-manual"), map[string]any{+ "id": "good", "host": goodHost, "port": goodPort,+ "addresses": []string{goodHost}, "models": []string{tc.advertisedModel},+ })+ e2eWaitResult(t, frames, "2", 5*time.Second)+ // Select the busy node so the failover path is deterministic.+ e2eSend(t, stdin, 3, addressed("node/select"), map[string]any{"id": "busy"})+ e2eWaitResult(t, frames, "3", 5*time.Second)++ body := fmt.Sprintf(`{"model":%q}`, tc.requestedModel)+ resp, err := http.Post(+ fmt.Sprintf("http://127.0.0.1:%d%s", port, tc.inferencePath),+ "application/json", strings.NewReader(body))+ if err != nil {+ t.Fatalf("inference POST: %v", err)+ }+ defer resp.Body.Close()+ if resp.StatusCode != http.StatusOK {+ t.Fatalf("status = %d, want 200 (should fail over from the 503 node)", resp.StatusCode)+ }+ if got := resp.Header.Get("Access-Control-Allow-Origin"); got != "*" {+ t.Errorf("Access-Control-Allow-Origin = %q, want *", got)+ }+ if gotBody != body {+ t.Errorf("healthy upstream got body %q, want the original request body %q", gotBody, body)+ }++ e2eSend(t, stdin, 9, "shutdown", nil)+ e2eWaitResult(t, frames, "9", 5*time.Second)+ })+}diff --git a/services/nvpair-proxy/enginecase_test.go b/services/nvpair-proxy/enginecase_test.gonew file mode 100644index 0000000..1a4b541--- /dev/null+++ b/services/nvpair-proxy/enginecase_test.go@@ -0,0 +1,165 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++// The harness the merged suite runs each engine through.+//+// The two proxies previously had parallel suites that drifted: same-named+// tests with different bodies, and behaviors covered on one side only. A+// single table means a case is either declared to apply to both engines or+// explicitly scoped to one, so a gap has to be a decision rather than an+// oversight.+//+// engineCase carries the fixtures a body cannot derive from the profile — the+// paths a client of that engine actually calls, and the shapes its upstream+// returns.++import (+ "fmt"+ "net/http"+ "net/http/httptest"+ "strings"+ "testing"+)++type engineCase struct {+ // profile is the engine under test.+ profile engineProfile++ // inferencePath is a path this engine treats as inference, used by bodies+ // that need to drive a routed request.+ inferencePath string++ // nonInferencePath is a real path this engine's clients call that is+ // neither inference nor a model list, so the proxy forwards it verbatim.+ // Bodies use it to prove that model-eligibility filtering and 404+ // failover apply to inference alone. It must stay absent from Routes.+ nonInferencePath string++ // modelListPath is the path whose response the federated list serves.+ modelListPath string++ // emptyModelList is the dialect's empty envelope. It is both what an+ // upstream with no models returns and what the federated list re-emits,+ // which is the assertion: an engine's client must see its own dialect's+ // empty shape rather than a bare 200 or the other engine's.+ emptyModelList string++ // advertisedModel is how a node's discovery inventory spells the model a+ // client reaches with requestedModel. For Ollama the two differ, because+ // an untagged request means the :latest tag.+ advertisedModel string++ // requestedModel is what a client asks for.+ requestedModel string+}++func ollamaCase(t *testing.T) engineCase {+ t.Helper()+ p, ok := profileFor("ollama")+ if !ok {+ t.Fatal("ollama profile missing")+ }+ return engineCase{+ profile: p,+ inferencePath: "/api/chat",+ nonInferencePath: "/api/version",+ modelListPath: "/api/tags",+ emptyModelList: `{"models":[]}`,+ advertisedModel: "qwen3:latest",+ requestedModel: "qwen3",+ }+}++func lmstudioCase(t *testing.T) engineCase {+ t.Helper()+ p, ok := profileFor("lmstudio")+ if !ok {+ t.Fatal("lmstudio profile missing")+ }+ return engineCase{+ profile: p,+ inferencePath: "/v1/chat/completions",+ // LM Studio's own REST surface, unlisted in Routes and so forwarded+ // verbatim.+ nonInferencePath: "/api/v0/models",+ modelListPath: "/v1/models",+ emptyModelList: `{"object":"list","data":[]}`,+ advertisedModel: "qwen3-8b",+ requestedModel: "qwen3-8b",+ }+}++// inferenceBody is the request body a client sends for requestedModel. It is+// the one shape both dialects share: a top-level "model" field, which is all+// bufferBodyAndModel reads.+func (tc engineCase) inferenceBody() string {+ return fmt.Sprintf(`{"model":%q}`, tc.requestedModel)+}++// inferenceRequest is a routed inference call for requestedModel, the request+// most bodies in the failover suite need.+func (tc engineCase) inferenceRequest() *http.Request {+ return httptest.NewRequest(http.MethodPost, tc.inferencePath, strings.NewReader(tc.inferenceBody()))+}++// engineCases is every engine the merged suite exercises. A body that runs+// under this is asserting behavior both proxies must share.+func engineCases(t *testing.T) []engineCase {+ t.Helper()+ return []engineCase{ollamaCase(t), lmstudioCase(t)}+}++// forEachEngine runs body as a subtest per engine.+func forEachEngine(t *testing.T, body func(t *testing.T, tc engineCase)) {+ t.Helper()+ for _, tc := range engineCases(t) {+ t.Run(tc.profile.Name, func(t *testing.T) { body(t, tc) })+ }+}++// anyProfile is for tests whose subject has no engine input at all — transport+// pooling, cluster trust, server timeouts, priority ordering. They need a+// profile only because NewProxy takes one, and running them per engine would+// double the runtime while proving nothing, since there is no branch for the+// second pass to take. Naming that explicitly keeps them from being mistaken+// for coverage gaps.+func anyProfile(t *testing.T) engineProfile {+ t.Helper()+ return ollamaCase(t).profile+}++// anyCase is anyProfile for a body that still needs an engine's fixtures as a+// vehicle — a routed inference request to reach the streaming path — while its+// subject remains engine-free. The zombie suite is the case in point: it tests+// statusCapture's write and flush deadlines and the terminalOnce guard, none of+// which consult the profile. That the path classifies as inference on both+// engines is already covered by the parameterized workload tests, so a second+// pass here would add real-socket seconds and no coverage.+func anyCase(t *testing.T) engineCase {+ t.Helper()+ return ollamaCase(t)+}++// otherEngine returns an engine that is not tc's, for negative fixtures that+// need to prove this proxy ignores a node running only a sibling engine.+func otherEngine(t *testing.T, tc engineCase) engineProfile {+ t.Helper()+ for _, candidate := range engineCases(t) {+ if candidate.profile.Name != tc.profile.Name {+ return candidate.profile+ }+ }+ t.Fatal("no sibling engine to contrast against")+ return engineProfile{}+}++// ollamaOnlyProfile is for bodies that are genuinely Ollama-scoped — the+// OLLAMA_HOST alias, and the native /api/tags dialect LM Studio does not+// serve. Using it rather than engineCases documents that the scoping is+// deliberate.+func ollamaOnlyProfile(t *testing.T) engineProfile {+ t.Helper()+ return ollamaCase(t).profile+}diff --git a/services/nvpair-proxy/engines.go b/services/nvpair-proxy/engines.gonew file mode 100644index 0000000..06d1a54--- /dev/null+++ b/services/nvpair-proxy/engines.go@@ -0,0 +1,224 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++// This is the only file in the proxy that carries table-driven engine values.+// Everything else reads them from the profile it is handed.+//+// Cross-process identity — Name, DisplayName, DiscoveryService, Namespace,+// FacadePort, EnginePortBase — lives in nvpair-shared/engines, because the+// broker, the scheduler and the TUI need the same values and a second+// declaration here could drift from theirs without failing to compile. What+// remains below is what only the proxy needs to know.+//+// Adding an engine is one entry here plus one in the broker's table. The+// values that are derivable are derived: the applog component and the error-ID+// prefix both come from Name via ComponentName, so they cannot disagree with+// each other. The persisted-port filename is declared instead, because+// Ollama's predates the multi-engine world — see PortFile in+// nvpair-shared/engines.++import (+ "strings"++ "nvpair-shared/engines"+)++// routeRole classifies an inbound request path. It carries the HTTP method+// because the method is part of the classification, not a separate axis: a+// POST to /v1/models is not a model list, and a GET to /api/chat is not+// inference. Folding them into one constant keeps the two from disagreeing.+//+// The dialect distinction is meaningful only for model-list roles. All seven+// of Ollama's inference paths are handled identically — no envelope, identity+// field or response shape is selected — so there is deliberately no+// per-dialect inference role.+type routeRole int++const (+ // roleInferencePOST is a POST that counts as a cluster workload: it is+ // routed to an owner of the requested model, emits workload lifecycle+ // notifications, and retries a 404 on another owner.+ roleInferencePOST routeRole = iota++ // roleModelListNativeGET is Ollama's /api/tags dialect: the upstream+ // envelope carries a "models" array, each record's identity is its+ // "model" field falling back to "name", and the federated response is+ // re-emitted as {"models": [...]}.+ roleModelListNativeGET++ // roleModelListOpenAIGET is the OpenAI /v1/models dialect: the upstream+ // envelope carries a "data" array, each record's identity is its "id",+ // and the federated response is re-emitted as+ // {"object": "list", "data": [...]}.+ roleModelListOpenAIGET+)++// modelNaming is how an engine spells a model identifier. It governs the+// native model list's dedupe key and nodeAdvertisesModel's match against a+// node's advertised inventory; those two must agree there, or the proxy can+// list a model it then refuses to route.+//+// The OpenAI list path (roleModelListOpenAIGET) is deliberately outside that+// pairing: it dedupes on the upstream identity verbatim, as the pre-unification+// proxies both did, while nodeAdvertisesModel still normalizes. Widening the+// claim to "both list paths" would describe code that does not exist.+type modelNaming int++const (+ // impliedLatestTag is Ollama's convention: an untagged name means the+ // ":latest" tag, so "llama3" and "llama3:latest" are one model.+ impliedLatestTag modelNaming = iota++ // exactID compares identifiers byte for byte.+ exactID+)++// route is one classified request path.+type route struct {+ Path string+ Role routeRole+}++// engineProfile is everything the proxy needs to front one engine.+type engineProfile struct {+ engines.Engine++ // StandalonePort is the port the proxy binds when nothing passes --port+ // and no port has been persisted. It must be neither FacadePort (which+ // the engine itself owns) nor a port an engine is relocated onto.+ //+ // Ollama's 11435 currently equals its EnginePortBase, which violates that+ // second half. It is left as-is because it is the value on disk in every+ // existing install; see the plan's port-field note.+ StandalonePort int++ // Routes classifies the paths this engine's clients call. It is NOT an+ // allowlist: an unlisted path is forwarded verbatim, which is how+ // /api/show, /api/pull, /api/ps, /api/version and OPTIONS preflights keep+ // working.+ Routes []route++ // ModelNaming is how this engine spells model identifiers.+ ModelNaming modelNaming++ // ReservedPersistedPort is a port that must never be restored from the+ // persisted-port file even if it is stored there, because it belongs to+ // the engine rather than the proxy. Zero means no port is reserved.+ ReservedPersistedPort int++ // SupportsHostAlias reports whether facade/enable's aliasAddresses apply to+ // this engine. Only Ollama has an inherited host variable (OLLAMA_HOST) for the+ // alias to stand in for, and the warning the proxy raises when it cannot+ // claim one names that variable. Gating on this makes the scoping+ // enforced rather than left to the broker's restraint in passing the flag.+ SupportsHostAlias bool+}++// openAIInferenceRoutes is the inference surface every OpenAI-compatible+// engine exposes. Ollama serves these alongside its native routes.+var openAIInferenceRoutes = []route{+ {Path: "/v1/chat/completions", Role: roleInferencePOST},+ {Path: "/v1/completions", Role: roleInferencePOST},+ {Path: "/v1/embeddings", Role: roleInferencePOST},+}++var profiles = buildProfiles()++func buildProfiles() []engineProfile {+ ollama, _ := engines.ByName("ollama")+ lmstudio, _ := engines.ByName("lmstudio")++ return []engineProfile{+ {+ Engine: ollama,+ StandalonePort: 11435,+ Routes: append([]route{+ {Path: "/api/generate", Role: roleInferencePOST},+ {Path: "/api/chat", Role: roleInferencePOST},+ {Path: "/api/embeddings", Role: roleInferencePOST},+ {Path: "/api/embed", Role: roleInferencePOST},+ {Path: "/api/tags", Role: roleModelListNativeGET},+ {Path: "/v1/models", Role: roleModelListOpenAIGET},+ }, openAIInferenceRoutes...),+ ModelNaming: impliedLatestTag,+ ReservedPersistedPort: 0,+ SupportsHostAlias: true,+ },+ {+ Engine: lmstudio,+ StandalonePort: 1234,+ Routes: append([]route{+ {Path: "/v1/models", Role: roleModelListOpenAIGET},+ }, openAIInferenceRoutes...),+ ModelNaming: exactID,+ // 1235 is where engine-manager runs a managed LM Studio, so a+ // proxy that restored it would sit on the engine's own port. The+ // stored value predates the current default of 1234.+ ReservedPersistedPort: 1235,+ },+ }+}++// profileFor resolves the engine named in a facade/enable request.+func profileFor(name string) (engineProfile, bool) {+ for _, p := range profiles {+ if p.Name == name {+ return p, true+ }+ }+ return engineProfile{}, false+}++// engineNames lists the accepted engine ids for a facade/enable rejection.+func engineNames() string {+ names := make([]string, 0, len(profiles))+ for _, p := range profiles {+ names = append(names, p.Name)+ }+ return strings.Join(names, ", ")+}++// roleFor classifies a request. The bool reports whether the path is one this+// engine handles specially; false means forward it verbatim.+func (p engineProfile) roleFor(method, path string) (routeRole, bool) {+ for _, r := range p.Routes {+ // Keep scanning on a method mismatch rather than bailing: a path may+ // legitimately appear twice under different methods, and returning+ // here would make the second entry permanently unreachable — silently+ // forwarded verbatim instead of classified.+ if r.Path != path || r.Role.method() != method {+ continue+ }+ return r.Role, true+ }+ return 0, false+}++// method is the HTTP method a role applies to.+func (r routeRole) method() string {+ if r == roleInferencePOST {+ return "POST"+ }+ return "GET"+}++// isModelList reports whether a role serves the federated model list.+func (r routeRole) isModelList() bool {+ return r == roleModelListNativeGET || r == roleModelListOpenAIGET+}++// normalizeModel applies the engine's naming convention to a model+// identifier, so the federated list's dedupe key and the routing gate agree.+func (p engineProfile) normalizeModel(model string) string {+ model = strings.TrimSpace(model)+ if p.ModelNaming != impliedLatestTag || model == "" {+ return model+ }+ name := model[strings.LastIndex(model, "/")+1:]+ if name != "" && !strings.ContainsAny(name, ":@") {+ return model + ":latest"+ }+ return model+}diff --git a/services/nvpair-proxy/engines_test.go b/services/nvpair-proxy/engines_test.gonew file mode 100644index 0000000..4b149e3--- /dev/null+++ b/services/nvpair-proxy/engines_test.go@@ -0,0 +1,211 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++import "testing"++// Routes is a classifier, not an allowlist. handlePlain forwards every+// loopback path into handleHTTP with no filtering, so a path the table does+// not mention must still reach the upstream verbatim — that is how /api/show,+// /api/pull, /api/ps, /api/version and OPTIONS preflights keep working.+func TestRoleForClassifiesOnlyDeclaredRoutes(t *testing.T) {+ ollama, ok := profileFor("ollama")+ if !ok {+ t.Fatal("ollama profile missing")+ }+ lmstudio, ok := profileFor("lmstudio")+ if !ok {+ t.Fatal("lmstudio profile missing")+ }++ for _, tc := range []struct {+ name string+ profile engineProfile+ method string+ path string+ wantRole routeRole+ wantOK bool+ }{+ {"ollama native chat", ollama, "POST", "/api/chat", roleInferencePOST, true},+ {"ollama openai chat", ollama, "POST", "/v1/chat/completions", roleInferencePOST, true},+ {"ollama native list", ollama, "GET", "/api/tags", roleModelListNativeGET, true},+ {"ollama openai list", ollama, "GET", "/v1/models", roleModelListOpenAIGET, true},+ {"ollama passthrough", ollama, "POST", "/api/pull", 0, false},+ {"ollama version passthrough", ollama, "GET", "/api/version", 0, false},++ {"lmstudio chat", lmstudio, "POST", "/v1/chat/completions", roleInferencePOST, true},+ {"lmstudio list", lmstudio, "GET", "/v1/models", roleModelListOpenAIGET, true},+ // LM Studio serves no native Ollama routes, so /api/chat is not+ // inference for it — it is forwarded verbatim like any other path.+ {"lmstudio has no native routes", lmstudio, "POST", "/api/chat", 0, false},++ // The method is part of the classification. Without it a POST to the+ // model-list path would be served as a list, and a GET to an+ // inference path would emit a workload.+ {"wrong method on list", ollama, "POST", "/v1/models", 0, false},+ {"wrong method on inference", ollama, "GET", "/api/chat", 0, false},+ } {+ t.Run(tc.name, func(t *testing.T) {+ role, ok := tc.profile.roleFor(tc.method, tc.path)+ if ok != tc.wantOK {+ t.Fatalf("roleFor(%s %s) ok = %v, want %v", tc.method, tc.path, ok, tc.wantOK)+ }+ if ok && role != tc.wantRole {+ t.Fatalf("roleFor(%s %s) role = %v, want %v", tc.method, tc.path, role, tc.wantRole)+ }+ })+ }+}++func TestIsInferenceRequestFollowsTheProfile(t *testing.T) {+ ollama, _ := profileFor("ollama")+ lmstudio, _ := profileFor("lmstudio")++ if !isInferenceRequest(ollama, "POST", "/api/generate") {+ t.Error("ollama /api/generate must be inference")+ }+ if isInferenceRequest(lmstudio, "POST", "/api/generate") {+ t.Error("lmstudio serves no native routes, so /api/generate is not inference for it")+ }+ if isInferenceRequest(ollama, "GET", "/api/tags") {+ t.Error("a model list is not inference")+ }+}++// One naming convention governs both the federated list's dedupe key and the+// routing gate. If they disagreed the proxy could advertise a model it then+// refuses to route.+func TestModelNaming(t *testing.T) {+ ollama, _ := profileFor("ollama")+ lmstudio, _ := profileFor("lmstudio")++ for _, tc := range []struct {+ profile engineProfile+ in string+ want string+ }{+ {ollama, "llama3", "llama3:latest"},+ {ollama, "llama3:latest", "llama3:latest"},+ {ollama, "llama3:8b", "llama3:8b"},+ {ollama, "registry.example/llama3", "registry.example/llama3:latest"},+ {ollama, "llama3@sha256:abc", "llama3@sha256:abc"},+ {ollama, "", ""},++ {lmstudio, "qwen3-8b", "qwen3-8b"},+ {lmstudio, "qwen3-8b:latest", "qwen3-8b:latest"},+ {lmstudio, "", ""},+ } {+ if got := tc.profile.normalizeModel(tc.in); got != tc.want {+ t.Errorf("%s normalizeModel(%q) = %q, want %q", tc.profile.Name, tc.in, got, tc.want)+ }+ }+}++func TestNodeAdvertisesModelUsesTheProfilesNaming(t *testing.T) {+ ollama, _ := profileFor("ollama")+ lmstudio, _ := profileFor("lmstudio")++ tagged := Node{Models: []string{"llama3:latest"}}+ if !nodeAdvertisesModel(ollama, tagged, "llama3") {+ t.Error("ollama must treat an untagged request as the :latest tag")+ }+ if nodeAdvertisesModel(lmstudio, tagged, "llama3") {+ t.Error("lmstudio matches identifiers exactly, so llama3 is not llama3:latest")+ }+ if nodeAdvertisesModel(ollama, Node{Models: []string{"llama3:latest"}}, "") {+ t.Error("an empty request model advertises nothing")+ }+}++// A path may legitimately be declared twice under different methods. roleFor+// used to return on the first path match, which made the second entry+// unreachable — it would be forwarded verbatim instead of classified, and+// nothing would say so.+func TestRoleForFindsAPathDeclaredUnderTwoMethods(t *testing.T) {+ p := engineProfile{Routes: []route{+ {Path: "/v1/models", Role: roleInferencePOST},+ {Path: "/v1/models", Role: roleModelListOpenAIGET},+ }}++ if role, ok := p.roleFor("GET", "/v1/models"); !ok || role != roleModelListOpenAIGET {+ t.Errorf("GET /v1/models = (%v, %v), want the model-list role", role, ok)+ }+ if role, ok := p.roleFor("POST", "/v1/models"); !ok || role != roleInferencePOST {+ t.Errorf("POST /v1/models = (%v, %v), want the inference role", role, ok)+ }+ if _, ok := p.roleFor("DELETE", "/v1/models"); ok {+ t.Error("DELETE /v1/models classified; an undeclared method must forward verbatim")+ }+}++// No shipped engine declares the same (path, method) twice; a duplicate would+// make whichever entry came second dead.+func TestNoDuplicateRoutePerMethod(t *testing.T) {+ for _, p := range profiles {+ seen := map[string]bool{}+ for _, r := range p.Routes {+ key := r.Role.method() + " " + r.Path+ if seen[key] {+ t.Errorf("%s declares %q twice; the second entry is unreachable", p.Name, key)+ }+ seen[key] = true+ }+ }+}++// The error-ID prefix derives from Name so it cannot disagree with the+// broker's expectations. The persisted-port filename deliberately does NOT+// derive: Ollama's predates the multi-engine world, and deriving it would+// rename the file under every existing install and orphan the port a user set.+func TestDerivedIdentifiers(t *testing.T) {+ ollama, _ := profileFor("ollama")+ lmstudio, _ := profileFor("lmstudio")++ if got := ollama.PortFile; got != "proxy-port.json" {+ t.Errorf("ollama PortFile = %q, want the pre-unification name", got)+ }+ if got := lmstudio.PortFile; got != "lmstudio-proxy-port.json" {+ t.Errorf("lmstudio PortFile = %q", got)+ }+ if got := upstreamUnreachableID(ollama, "peer-A"); got != "ollama-proxy:upstream-unreachable:peer-A" {+ t.Errorf("ollama upstreamUnreachableID = %q", got)+ }+ if got := upstreamUnreachableID(lmstudio, "peer-A"); got != "lmstudio-proxy:upstream-unreachable:peer-A" {+ t.Errorf("lmstudio upstreamUnreachableID = %q", got)+ }+}++// chooseStartupPort restores a previously chosen port, except when the broker+// forces one or when the stored value belongs to the engine itself.+func TestChooseStartupPort(t *testing.T) {+ ollama, _ := profileFor("ollama")+ lmstudio, _ := profileFor("lmstudio")++ for _, tc := range []struct {+ name string+ profile engineProfile+ flagPort int+ ignorePersisted bool+ persisted int+ hasPersisted bool+ want int+ }{+ {"restores persisted", ollama, 11435, false, 11500, true, 11500},+ {"broker override wins", ollama, 11434, true, 11500, true, 11434},+ {"no persisted value", ollama, 11435, false, 0, false, 11435},+ {"lmstudio restores persisted", lmstudio, 1234, false, 1300, true, 1300},+ // 1235 is where engine-manager runs a managed LM Studio, so restoring+ // it would put the proxy on the engine's own port.+ {"lmstudio refuses the engine's port", lmstudio, 1234, false, 1235, true, 1234},+ // Ollama reserves nothing, so the equivalent value is honoured.+ {"ollama reserves nothing", ollama, 11434, false, 11435, true, 11435},+ } {+ t.Run(tc.name, func(t *testing.T) {+ got := chooseStartupPort(tc.profile, tc.flagPort, tc.ignorePersisted, tc.persisted, tc.hasPersisted)+ if got != tc.want {+ t.Fatalf("chooseStartupPort = %d, want %d", got, tc.want)+ }+ })+ }+}diff --git a/services/nvpair-proxy/facade.go b/services/nvpair-proxy/facade.gonew file mode 100644index 0000000..1b70c91--- /dev/null+++ b/services/nvpair-proxy/facade.go@@ -0,0 +1,800 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++import (+ "context"+ "crypto/tls"+ "encoding/json"+ stderrors "errors"+ "fmt"+ "log/slog"+ "net"+ "net/http"+ "net/url"+ "strconv"+ "strings"+ "sync"+ "sync/atomic"+ "time"++ "nvpair-shared/engines"+ "nvpair-shared/errors"+ "nvpair-shared/noderec"+ "nvpair-shared/reach"+ "nvpair-shared/splitlisten"+)++// facade is one engine's presence on this node: the ports it listens on, the+// dialect it speaks, and the routing state that answers for it. Proxy is the+// process that hosts facades and owns everything they share — the JSON-RPC+// codec, the cluster mesh, the outbound transport pool, the scheduler baseline.+//+// One Proxy hosts a facade per enabled engine, keyed by engine id, and they run+// concurrently — see spec.md §3. That is what makes the ownership question+// below a correctness question rather than a stylistic one: a field on the+// wrong side is either duplicated state two engines will disagree about, or+// shared state one engine can corrupt for the other.+//+// Ownership rule for anything added here: a field belongs to the facade if two+// engines would each need their own, and to Proxy if a second copy would be+// wasteful or actively wrong. Listeners are per facade because the ports+// differ. The transport pool is per process because it is keyed by peer, not+// by engine, and duplicating it would double the connections to every peer.+type facade struct {+ // host is the process hosting this facade. Reached for shared state only.+ host *Proxy++ // profile is the engine this facade fronts. Set once at construction and+ // never mutated, so every read is safe without a lock.+ profile engineProfile++ // httpMu guards port, the servers, the split listener, and the listeners+ // across a live set-port rebind. The HTTP handlers never read port (they+ // route by upstream node), so the only contention is set-port vs set-port+ // (serialized) and the initial serveHTTP store vs a later rebind.+ httpMu sync.Mutex+ port int+ aliasAddr string+ aliasAltAddr string+ plainSrv *http.Server+ tlsSrv *http.Server+ split *splitlisten.Splitter+ aliasLn net.Listener+ aliasAltLn net.Listener++ // discovery is this facade's routing overlay: the relay-fed set of peers+ // advertising this engine, plus the manual nodes the broker bridged in.+ // Per facade because the relay filters snapshots by discovery service key,+ // so each engine sees a different set of nodes.+ discovery *Discovery++ // selectedMu guards selectedID, the user's manual route pin. Per facade+ // because pinning a node for Ollama says nothing about LM Studio.+ selectedMu sync.RWMutex+ selectedID string++ // backendMu guards backend, the explicit loopback engine this facade's+ // cluster mTLS ingress forwards to. The broker sets/clears it via+ // node/set-local-backend; it is never sourced from discovery, so an ingress+ // request can only ever reach this node's own local engine and can never be+ // re-routed to a peer.+ backendMu sync.RWMutex+ backend localBackend++ // targets remembers, per node, which of its published addresses accepted a+ // connection, so a repeated forward costs no confirmation. An entry is+ // re-confirmed when the node's candidate list changes and forgotten on an+ // upstream error, so the next request fails over to another address.+ //+ // Per facade even though it is keyed by node: the candidate addresses carry+ // each engine's own port, so a shared chooser would have two engines+ // overwriting each other's confirmed endpoint for the same peer.+ targets *reach.Chooser++ // nextRequestID tags this facade's RequestStarted / RequestEvent pairs.+ // Atomic add returns the new value, so ids start at 1 per facade.+ //+ // Per facade, and load-bearing: TestWorkloadCrossEngineIdentityDistinct in+ // services/tests requires both engines to mint id "1" concurrently, which+ // is what proves the (origin, engine, runId, id) store key actually+ // separates them. A shared counter would hand the second facade "2" and+ // leave that guarantee untested.+ nextRequestID atomic.Uint64+}++func newFacade(host *Proxy, profile engineProfile, discovery *Discovery, port int) *facade {+ return &facade{+ host: host,+ profile: profile,+ discovery: discovery,+ port: port,+ targets: reach.NewChooser(),+ }+}++// enableFacadeParams is the broker's request to bring one engine's facade up.+//+// The port arrives here rather than as a process flag because the broker plans+// a different port for each engine, and a single-valued flag cannot carry more+// than one plan. IgnorePersistedPort travels with it for the same reason: it+// qualifies this engine's port, not the process.+type enableFacadeParams struct {+ Engine string `json:"engine"`+ // Port is the port the broker wants this facade on. Zero means the+ // engine's standalone default.+ Port int `json:"port,omitempty"`+ // AliasAddresses are extra loopback-only addresses to answer on, for an+ // engine whose clients read an inherited host variable.+ AliasAddresses []string `json:"aliasAddresses,omitempty"`+ // IgnorePersistedPort uses Port even when this engine has a persisted one.+ IgnorePersistedPort bool `json:"ignorePersistedPort,omitempty"`+}++// facadeWithdrawGrace bounds the teardown of a facade whose bring-up failed.+// Short on purpose: nothing has been announced as serving, so there is no+// legitimate long-lived request to drain.+const facadeWithdrawGrace = 2 * time.Second++// errFacadeBindFailed marks the one enable failure a different port could fix.+// Every other rejection — unknown engine, unsupported alias, port out of range+// — is a caller bug that retrying elsewhere would not help, so the broker must+// be able to tell them apart. It reaches the broker as codeFacadeBindFailed.+var errFacadeBindFailed = stderrors.New("facade bind failed")++// enableFacadeResult reports where the facade actually landed.+//+// The bound port is returned rather than only announced, because the broker+// asks for a port and the persisted-port restore may override it: enable is a+// request, not a command. Reading the outcome from the response lets the caller+// finish its port transaction without waiting for a notification.+type enableFacadeResult struct {+ Engine string `json:"engine"`+ Port int `json:"port"`+}++// enableFacade brings up one engine's facade and reports the port it bound.+//+// Enabling an engine that is already up is an idempotent success, so a+// redelivered enable after a timeout does not tear down a working listener.+//+// It deliberately takes no context. The facade's HTTP servers outlive this+// call, and their BaseContext has to be the process lifetime: handing them the+// enable request's context would cancel every request the facade later serves+// the moment this returns.+func (p *Proxy) enableFacade(params enableFacadeParams) (enableFacadeResult, error) {+ profile, ok := profileFor(params.Engine)+ if !ok {+ return enableFacadeResult{}, fmt.Errorf("unknown engine %q; want one of %s", params.Engine, engineNames())+ }+ if params.Port != 0 && (params.Port < 1 || params.Port > 65535) {+ // Zero means "the standalone default", so it is the one out-of-range+ // value with a meaning. Anything else out of range is a caller bug: the+ // listener would bind on an ephemeral port but the facade announces the+ // requested one, leaving a proxy the broker cannot locate.+ return enableFacadeResult{}, fmt.Errorf("port must be between 1 and 65535 (got %d)", params.Port)+ }+ if len(params.AliasAddresses) > 0 && !profile.SupportsHostAlias {+ return enableFacadeResult{}, fmt.Errorf(+ "alias addresses are not supported for engine %q: it has no inherited host variable for the alias to stand in for",+ profile.Name)+ }++ p.facadeMu.Lock()+ defer p.facadeMu.Unlock()++ if existing := p.facades[profile.Name]; existing != nil {+ return enableFacadeResult{Engine: profile.Name, Port: existing.port}, nil+ }++ requested := params.Port+ if requested == 0 {+ requested = profile.StandalonePort+ }+ // Restore a port the user previously chose via set-port, so the facade+ // comes back where they left it rather than where the broker last planned.+ persisted, hasPersisted := loadPersistedPort(profile)+ port := chooseStartupPort(profile, requested, params.IgnorePersistedPort, persisted, hasPersisted)+ if port == persisted {+ slog.Info("restored persisted proxy port", "engine", profile.Name, "port", persisted)+ }++ f := newFacade(p, profile, NewDiscovery(), port)+ for _, address := range params.AliasAddresses {+ if err := f.setLoopbackAlias(address); err != nil {+ return enableFacadeResult{}, fmt.Errorf("invalid alias address %q: %w", address, err)+ }+ }+ // Published before start so a request arriving on this facade's listener,+ // which start opens, finds it.+ if p.facades == nil {+ p.facades = make(map[string]*facade, len(engines.Names()))+ }+ p.facades[profile.Name] = f++ // Withdrawal is deferred rather than done on the error return, because the+ // dispatch recovers panics instead of ending the process: an unwinding+ // bring-up would otherwise leave a published facade that may hold a bound+ // port with no serving loop, and the idempotent branch above would then+ // report that dead port as success to every later enable.+ //+ // Only this engine is withdrawn. Every other facade keeps serving: a lost+ // bind race is one engine's problem, and taking working listeners down with+ // it is exactly what moving off argv was meant to stop.+ started := false+ defer func() {+ if started {+ return+ }+ delete(p.facades, profile.Name)+ // Dispatched to a goroutine, and that is what takes it off the lock:+ // this defer is registered after the facadeMu unlock, so LIFO ordering+ // runs it *before* the unlock, with the mutex still held. Shutting down+ // inline would hold facadeMu for as long as one in-flight request took,+ // blocking every facade lookup and process teardown behind it.+ //+ // Bounded for the same reason it is asynchronous. start releases its+ // own listener when it does not reach serving, so this covers only the+ // window after the split was published.+ go func() {+ ctx, cancel := context.WithTimeout(context.Background(), facadeWithdrawGrace)+ defer cancel()+ f.stopServing(ctx)+ }()+ }()++ if err := f.start(p.serveCtx); err != nil {+ return enableFacadeResult{}, err+ }+ started = true+ return enableFacadeResult{Engine: profile.Name, Port: f.port}, nil+}++func (f *facade) SelectedID() string {+ f.selectedMu.RLock()+ defer f.selectedMu.RUnlock()+ return f.selectedID+}++func (f *facade) SetSelected(id string) {+ f.selectedMu.Lock()+ defer f.selectedMu.Unlock()+ f.selectedID = id+}++// clearSelectionOf drops the route pin when it points at the given node,+// reporting whether it did so the caller can announce the change.+//+// The lock is released by defer rather than on each branch, so a panic in this+// critical section cannot leave the pin locked. That matters now that a panic+// is recovered instead of ending the process: a stranded mutex would turn a+// contained crash into a hang for every facade.+func (f *facade) clearSelectionOf(id string) bool {+ f.selectedMu.Lock()+ defer f.selectedMu.Unlock()+ if f.selectedID != id {+ return false+ }+ f.selectedID = ""+ return true+}++// clearSelectionIfNotPresent resets the user-selected node (and notifies the+// client) when it's neither in the relay-fed set nor a manual node, so a stale+// selection can't pin routing to a departed target.+func (f *facade) clearSelectionIfNotPresent(present map[string]bool) {+ sel := f.SelectedID()+ if sel == "" || present[sel] || f.discovery.IsManual(sel) {+ return+ }+ // Re-checked under the lock rather than cleared outright: the selection can+ // change while IsManual runs, and clearing a newer pin would drop a choice+ // the user just made.+ if f.clearSelectionOf(sel) {+ _ = f.notify("node/selection-changed", SelectedResult{ID: ""})+ }+}++// targetURL resolves a node to the address this facade should dial.+//+// reach.Prefer, not a blocking confirmation: this runs once per discovered node+// per request, including nodes this request will not be routed to, so a+// handshake here would charge every request for every node's connectivity. An+// address that is wrong is caught by the reverse proxy's ErrorHandler, which+// forgets it and fails over.+func (f *facade) targetURL(n Node) *url.URL {+ candidates := nodeCandidates(n)+ if len(candidates) == 0 {+ return nil+ }+ host := f.targets.Prefer(n.ID, candidates)+ return &url.URL{Scheme: "http", Host: host}+}++// replaceSubscribed replaces this facade's relay-fed routing overlay from a+// discovery:nodes snapshot: it projects every node advertising this engine with+// a dialable IP into the overlay (dropping the rest) and clears a user selection+// pinned to a node that's no longer routable. The broker sends the full filtered+// set on every change, so this is a wholesale replace, not a per-node apply — a+// departed node is simply absent from the next snapshot.+func (f *facade) replaceSubscribed(params json.RawMessage) {+ var res noderec.GetNodesResult+ if err := json.Unmarshal(params, &res); err != nil {+ slog.Warn("invalid discovery:nodes snapshot", "err", err)+ return+ }+ nodes := make([]Node, 0, len(res.Nodes))+ present := make(map[string]bool, len(res.Nodes))+ for _, dn := range res.Nodes {+ n, ok := subscribedToNode(f.profile, dn)+ if !ok {+ continue+ }+ nodes = append(nodes, n)+ present[n.ID] = true+ }+ discovered, updated, removed := f.discovery.SetSubscribed(nodes)+ // Surface the relay-fed set to the client as node/* events — the signal a+ // consumer (the UI) uses to show which peers run this engine — mirroring how+ // manual nodes are announced. Without this the routing overlay updates+ // silently and peers appear engine-less. A node dropping out is also the+ // proxy's "this upstream is gone" signal, surfaced through the errors+ // pipeline (the broker forwards these to nvpair-errors); a re-appearance clears+ // it. NodeID/Timestamp are left unset so the broker stamps the authoritative+ // values.+ for _, n := range discovered {+ _ = f.notify("node/discovered", n.withPrimaryIP())+ if err := f.notify("errors:clear", errors.ClearParams{ID: upstreamUnreachableID(f.profile, n.ID)}); err != nil {+ slog.Debug("failed to send errors:clear", "node", n.ID, "err", err)+ }+ }+ for _, n := range updated {+ _ = f.notify("node/updated", n.withPrimaryIP())+ }+ for _, n := range removed {+ _ = f.notify("node/removed", n.withPrimaryIP())+ if err := f.notify("errors:report", errors.ServiceError{+ ID: upstreamUnreachableID(f.profile, n.ID),+ Message: fmt.Sprintf("Upstream node %q is no longer reachable (dropped from discovery)", n.Host),+ Severity: "warning",+ Action: "none",+ }); err != nil {+ slog.Debug("failed to send errors:report", "node", n.ID, "err", err)+ }+ }+ f.clearSelectionIfNotPresent(present)+}++// notify emits a facade-scoped notification, addressed to this engine.+//+// The address is what lets the broker attribute a "ready" to a facade rather+// than to whichever process the message arrived on. Every facade-scoped+// notification goes through here, so that attribution cannot be forgotten at+// one call site.+//+// Process-scoped notifications — anything the broker attributes to the process+// rather than an engine — go to host.codec directly and must not come here.+func (f *facade) notify(method string, params any) error {+ return f.host.codec.Notify(engines.AddressMethod(f.profile.Name, method), params)+}++// listen binds this facade's TCP listener synchronously so bind failures+// (EADDRINUSE and friends) can be reported through a structured error+// notification before the process exits. The caller is responsible for+// closing the returned listener if it doesn't hand it to serveHTTP.+func (f *facade) listen() (net.Listener, error) {+ return net.Listen("tcp", fmt.Sprintf(":%d", f.port))+}++// start brings this facade up: bind, reserve the alias, announce readiness,+// subscribe for routing targets, and serve. It is the single bring-up path, so+// there is one place that decides what "the facade is up" means.+//+// The bind is synchronous and precedes the ready announcement. If the port is+// taken we want the reason surfaced, rather than the UI sitting on "proxy+// running" while a background ListenAndServe fails silently.+func (f *facade) start(ctx context.Context) error {+ ln, err := f.listen()+ if err != nil {+ // Best-effort: the caller decides what to do about this, so a failed+ // notify here is not worth surfacing separately.+ _ = f.notify("error", ErrorParams{+ Code: "bind-failed",+ Message: fmt.Sprintf("failed to bind port %d: %v", f.port, err),+ Port: f.port,+ })+ // Tagged so the broker can retry this on another port and leave every+ // other enable rejection alone. The cause is kept for the log line.+ return fmt.Errorf("%w: port %d: %w", errFacadeBindFailed, f.port, err)+ }++ // The listener is released unless bring-up completes, and by defer so an+ // unwinding panic releases it too.+ //+ // start owns the port from listen() until serveHTTP hands it to the split.+ // Leaving it held by nothing would be worse than not binding at all: a+ // later enable for this engine gets EADDRINUSE from its own process, the+ // broker reads that as a lost bind race, and the facade spends the rest of+ // the process on a fallback port.+ serving := false+ defer func() {+ if serving {+ return+ }+ _ = ln.Close()+ f.closeLoopbackAlias()+ }()++ // Reserve the inherited OLLAMA_HOST alias before announcing readiness, so a+ // successful ready event is truthful for clients that start the moment+ // NVPAIR comes up. Alias conflicts are deliberately non-fatal: the primary+ // facade stays available and the existing owner is untouched.+ f.bindLoopbackAlias()++ if err := f.notify("ready", ReadyParams{Version: Version, Port: f.port}); err != nil {+ return fmt.Errorf("failed to send ready notification: %w", err)+ }++ // Routing targets come from the broker's discovery relay. Snapshots arrive+ // as discovery:nodes and each replaces the subscribed overlay. Non-fatal: a+ // parent that is not a relay-aware broker still leaves manual-node routing.+ // Addressed like every other facade-scoped notification, because each+ // facade subscribes to its own engine's discovery service. An unaddressed+ // subscribe would leave the broker holding one subscription id per process,+ // so a second facade's subscribe would silently replace the first's.+ slog.Debug("subscribing to discovery relay for routing targets",+ "service", string(f.profile.DiscoveryService))+ if err := f.notify(noderec.MethodSubscribe, noderec.SubscribeParams{+ Services: []noderec.ServiceKey{f.profile.DiscoveryService},+ }); err != nil {+ slog.Warn("failed to subscribe to discovery relay", "err", err)+ }++ // serveHTTP hands the listener to the split, which stopServing owns from+ // here on, so the deferred release above must stand down.+ f.serveHTTP(ctx, ln)+ serving = true+ f.serveLoopbackAlias()+ return nil+}++// serveHTTP takes the already-bound base listener and drives the two proxy+// personalities over it: a plaintext HTTP server (loopback-only, full local+// router) and a LAN mTLS ingress (pin-gated, forwards to the local engine),+// split by the connection's first byte via nvpair-shared/splitlisten. The two+// http.Servers are recorded so set-port can rebind both onto a fresh split+// without tearing the servers down.+func (f *facade) serveHTTP(ctx context.Context, ln net.Listener) {+ base := func(_ net.Listener) context.Context { return ctx }+ plainSrv := &http.Server{+ Handler: http.HandlerFunc(f.handlePlain),+ BaseContext: base,+ ReadHeaderTimeout: proxyReadHeaderTimeout,+ IdleTimeout: proxyServerIdleTimeout,+ }+ tlsSrv := &http.Server{+ Handler: http.HandlerFunc(f.handleClusterIngress),+ BaseContext: base,+ ReadHeaderTimeout: proxyReadHeaderTimeout,+ IdleTimeout: proxyServerIdleTimeout,+ }++ f.publishServers(ln, plainSrv, tlsSrv)++ slog.Info("proxy timeouts configured",+ "dial_timeout", proxyDialTimeout,+ "keep_alive", proxyKeepAlive,+ "response_header_timeout", proxyResponseTimeout,+ "max_idle_conns", proxyMaxIdleConns,+ "idle_conn_timeout", proxyIdleConnTimeout,+ )+ slog.Info("HTTP proxy listening", "port", f.port, "addr", ln.Addr().String(),+ "cluster_ingress", f.host.mesh.Clustered())+}++// publishServers records this facade's servers and starts the split listener.+//+// Every httpMu section in this file releases by defer, and that is load-bearing+// rather than stylistic: bring-up is reachable from the JSON-RPC dispatch,+// which recovers panics instead of ending the process. resolveCandidates takes+// httpMu on every request and stopServing takes it during teardown, so a+// stranded httpMu would turn a contained panic into a hung facade plus a hung+// process exit — worse than the crash the recover replaced.+func (f *facade) publishServers(ln net.Listener, plainSrv, tlsSrv *http.Server) {+ f.httpMu.Lock()+ defer f.httpMu.Unlock()+ f.plainSrv = plainSrv+ f.tlsSrv = tlsSrv+ // The base listener is not retained: the splitter owns it from here, and+ // stopServing closes the splitter. A second copy of the reference was+ // written and never read, which read like teardown released the base+ // listener when in fact only the split ever did.+ f.startSplitLocked(ln)+}++// selfAddresses snapshots this facade's own bound port and alias addresses, so+// the self-forward guard can recognize its own listeners.+func (f *facade) selfAddresses() (selfPort int, aliasBound []string) {+ f.httpMu.Lock()+ defer f.httpMu.Unlock()+ if f.aliasLn != nil {+ aliasBound = append(aliasBound, f.aliasLn.Addr().String())+ }+ if f.aliasAltLn != nil {+ aliasBound = append(aliasBound, f.aliasAltLn.Addr().String())+ }+ return f.port, aliasBound+}++// servingState snapshots what stopServing has to shut down, so the shutdowns+// themselves run outside httpMu.+func (f *facade) servingState() (plainSrv, tlsSrv *http.Server, split *splitlisten.Splitter, aliasLn, aliasAltLn net.Listener) {+ f.httpMu.Lock()+ defer f.httpMu.Unlock()+ return f.plainSrv, f.tlsSrv, f.split, f.aliasLn, f.aliasAltLn+}++// aliasAddresses snapshots the configured alias addresses. Index 0 is empty+// when no alias was requested.+func (f *facade) aliasAddresses() []string {+ f.httpMu.Lock()+ defer f.httpMu.Unlock()+ return []string{f.aliasAddr, f.aliasAltAddr}+}++// setAliasListeners records the reserved alias listeners in order.+func (f *facade) setAliasListeners(listeners []net.Listener) {+ f.httpMu.Lock()+ defer f.httpMu.Unlock()+ f.aliasLn = listeners[0]+ if len(listeners) > 1 {+ f.aliasAltLn = listeners[1]+ }+}++// aliasServing snapshots what serveLoopbackAlias needs to start its goroutines+// without holding httpMu across the Serve calls.+func (f *facade) aliasServing() (addresses []string, listeners []net.Listener, plainSrv *http.Server) {+ f.httpMu.Lock()+ defer f.httpMu.Unlock()+ return []string{f.aliasAddr, f.aliasAltAddr},+ []net.Listener{f.aliasLn, f.aliasAltLn},+ f.plainSrv+}++// takeAliasListeners clears and returns the alias listeners so the caller can+// close them outside the lock.+func (f *facade) takeAliasListeners() []net.Listener {+ f.httpMu.Lock()+ defer f.httpMu.Unlock()+ listeners := []net.Listener{f.aliasLn, f.aliasAltLn}+ f.aliasLn = nil+ f.aliasAltLn = nil+ return listeners+}++// setLoopbackAlias validates and records the optional secondary plaintext+// listener. The broker supplies this only for an inherited local HTTP+// OLLAMA_HOST. Validate again here so a direct invocation can never turn the+// alias flag into a LAN plaintext listener.+func (f *facade) setLoopbackAlias(address string) error {+ host, portText, err := net.SplitHostPort(address)+ if err != nil {+ return fmt.Errorf("expected host:port: %w", err)+ }+ port, err := strconv.Atoi(portText)+ if err != nil || port < 1 || port > 65535 {+ return fmt.Errorf("port must be between 1 and 65535")+ }+ ip := net.ParseIP(host)+ if strings.EqualFold(strings.TrimSuffix(host, "."), "localhost") {+ host = "127.0.0.1"+ address = net.JoinHostPort(host, portText)+ } else if ip == nil || !ip.IsLoopback() {+ return fmt.Errorf("host %q is not loopback", host)+ }+ if port == f.port {+ return fmt.Errorf("alias port %d is already the primary listener", port)+ }+ switch {+ case f.aliasAddr == "":+ f.aliasAddr = address+ case f.aliasAddr == address || f.aliasAltAddr == address:+ return nil+ case f.aliasAltAddr == "":+ _, firstPort, _ := net.SplitHostPort(f.aliasAddr)+ if firstPort != portText {+ return fmt.Errorf("alias addresses must use the same port")+ }+ f.aliasAltAddr = address+ default:+ return fmt.Errorf("at most two alias addresses are supported")+ }+ return nil+}++// bindLoopbackAlias reserves the configured alias before the proxy announces+// ready. A conflict is non-fatal: the primary stays healthy, the existing owner+// is untouched, and a sticky warning tells the user how to recover.+func (f *facade) bindLoopbackAlias() {+ addresses := f.aliasAddresses()+ if addresses[0] == "" {+ return+ }++ listeners := make([]net.Listener, 0, 2)+ for _, address := range addresses {+ if address == "" {+ continue+ }+ ln, err := net.Listen("tcp", address)+ if err != nil {+ for _, opened := range listeners {+ _ = opened.Close()+ }+ f.reportLoopbackAliasBlocked(address, err)+ return+ }+ listeners = append(listeners, ln)+ }+ f.setAliasListeners(listeners)+ if err := f.notify("errors:clear", errors.ClearParams{ID: ollamaHostAliasBlockedID}); err != nil {+ slog.Debug("failed to clear OLLAMA_HOST alias warning", "err", err)+ }+ slog.Info("OLLAMA_HOST loopback alias reserved", "addresses", addresses)+}++func (f *facade) serveLoopbackAlias() {+ addresses, listeners, plainSrv := f.aliasServing()+ if listeners[0] == nil || plainSrv == nil {+ return+ }+ for i, ln := range listeners {+ if ln == nil {+ continue+ }+ address := addresses[i]+ go func() {+ if err := plainSrv.Serve(ln); err != nil && err != http.ErrServerClosed && !stderrors.Is(err, net.ErrClosed) {+ slog.Error("OLLAMA_HOST alias server exited", "address", address, "err", err)+ }+ }()+ }+ slog.Info("OLLAMA_HOST loopback alias listening", "addresses", addresses)+}++func (f *facade) closeLoopbackAlias() {+ for _, ln := range f.takeAliasListeners() {+ if ln != nil {+ _ = ln.Close()+ }+ }+}++// The alias is gated to engines with SupportsHostAlias, which today is Ollama+// alone, so this prefix is correct by construction rather than by convention —+// enableFacade refuses alias addresses for any other engine. The broker matches+// this exact id.+const ollamaHostAliasBlockedID = "ollama-proxy:ollama-host-alias-blocked"++func (f *facade) reportLoopbackAliasBlocked(address string, bindErr error) {+ message := fmt.Sprintf(+ "NVPAIR kept its primary Ollama compatibility proxy separate, but could not claim the local OLLAMA_HOST alias %s: %v. Stop or reconfigure the application using that port, or change or unset OLLAMA_HOST, then restart NVPAIR. No process was stopped.",+ address, bindErr)+ if err := f.notify("errors:report", errors.ServiceError{+ ID: ollamaHostAliasBlockedID,+ Message: message,+ Severity: "warning",+ Action: "none",+ }); err != nil {+ slog.Debug("failed to report OLLAMA_HOST alias warning", "err", err)+ }+ slog.Warn("OLLAMA_HOST alias unavailable", "address", address, "err", bindErr)+}++// startSplitLocked wraps base in a first-byte splitter and starts both servers+// on its sub-listeners. Caller holds httpMu. Reuses the persistent plainSrv /+// tlsSrv so set-port can call it repeatedly on fresh listeners.+func (f *facade) startSplitLocked(base net.Listener) {+ split := splitlisten.New(base)+ f.split = split+ go func() {+ if err := f.plainSrv.Serve(split.Plain()); err != nil && err != http.ErrServerClosed {+ slog.Error("plaintext HTTP server exited", "err", err)+ }+ }()+ go f.serveTLS(split.TLS())+}++// serveTLS terminates cluster mTLS on the split's TLS sub-listener. The server+// certificate is resolved per handshake from the live mesh, so this one+// sub-listener covers both states: while this node is unclustered there is no+// leaf to present and the handshake is refused (it exposes no LAN inference+// surface), and the moment the node becomes a member the same sub-listener+// serves the pin-gated ingress — no rebind, and no process restart to pick up a+// freshly-minted identity.+func (f *facade) serveTLS(l net.Listener) {+ if err := f.tlsSrv.Serve(tls.NewListener(l, f.host.mesh.ServerTLSConfig())); err != nil && err != http.ErrServerClosed {+ slog.Error("cluster mTLS ingress exited", "err", err)+ }+}++// stopServing gracefully stops this facade's two personalities and closes the+// listeners it owns. It deliberately does not touch the outbound transport+// pool: that is shared across facades, so closing it here would drop another+// engine's pooled connections to every peer. Process teardown owns it.+func (f *facade) stopServing(ctx context.Context) {+ plainSrv, tlsSrv, split, aliasLn, aliasAltLn := f.servingState()+ if plainSrv != nil {+ _ = plainSrv.Shutdown(ctx)+ }+ if tlsSrv != nil {+ _ = tlsSrv.Shutdown(ctx)+ }+ if split != nil {+ _ = split.Close()+ }+ if aliasLn != nil {+ _ = aliasLn.Close()+ }+ if aliasAltLn != nil {+ _ = aliasAltLn.Close()+ }+}++// setPort live-rebinds this facade's HTTP listener onto newPort and persists+// the choice so it survives a restart. It binds the new listener first (so a+// bind failure leaves the current one serving), starts the same server on it,+// then closes the old listener — in-flight connections on the old port drain+// naturally. A fresh `ready` notification announces the new port so the+// orchestrator/UI learn where the proxy is now listening.+// Only the listener swap runs under httpMu. Persisting the choice and+// announcing it happen after the unlock, because every request takes this+// mutex through resolveCandidates: holding it across slow storage or a+// backpressured stdout would stall all inference through this facade, and delay+// teardown with it.+func (f *facade) setPort(newPort int) error {+ swapped, err := f.swapListener(newPort)+ if err != nil || !swapped {+ return err+ }++ if err := savePersistedPort(f.profile, newPort); err != nil {+ slog.Warn("failed to persist proxy port", "port", newPort, "err", err)+ }+ if err := f.notify("ready", ReadyParams{Version: Version, Port: newPort}); err != nil {+ slog.Warn("failed to emit ready after rebind", "err", err)+ }+ return nil+}++// swapListener binds newPort and moves both personalities onto it, reporting+// false when the facade was already there. The new listener is bound before the+// old split closes, so a bind failure leaves the current one serving.+func (f *facade) swapListener(newPort int) (bool, error) {+ f.httpMu.Lock()+ defer f.httpMu.Unlock()++ if newPort == f.port {+ return false, nil+ }+ newLn, err := net.Listen("tcp", fmt.Sprintf(":%d", newPort))+ if err != nil {+ return false, fmt.Errorf("failed to bind port %d: %w", newPort, err)+ }+ oldSplit := f.split+ f.port = newPort++ // Re-serve both personalities on a fresh split over the new listener, then+ // close the old split (and its base listener) so in-flight connections on+ // the old port drain naturally. The plaintext and mTLS personalities always+ // move together as one unit.+ slog.Info("HTTP proxy listening", "port", newPort, "addr", newLn.Addr().String(),+ "cluster_ingress", f.host.mesh.Clustered())+ f.startSplitLocked(newLn)+ if oldSplit != nil {+ _ = oldSplit.Close()+ }+ return true, nil+}diff --git a/services/nvpair-proxy/failover_test.go b/services/nvpair-proxy/failover_test.gonew file mode 100644index 0000000..47c0693--- /dev/null+++ b/services/nvpair-proxy/failover_test.go@@ -0,0 +1,727 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++import (+ "encoding/json"+ "fmt"+ "io"+ "net"+ "net/http"+ "net/http/httptest"+ "net/url"+ "strconv"+ "strings"+ "testing"+ "time"+)++// rwNop is a no-op io.ReadWriter so a Codec can be constructed in tests+// without a real transport: reads hit EOF immediately and writes are+// discarded. handleHTTP only ever writes (notifications), so this is enough.+type rwNop struct{}++func (rwNop) Read([]byte) (int, error) { return 0, io.EOF }+func (rwNop) Write(p []byte) (int, error) { return len(p), nil }++// newTestProxy builds a proxy with one facade attached but not started, which+// is what nearly every test wants: routing, dispatch, and handler behavior+// without binding a port.+//+// Production brings a facade up through facade/enable, which also binds,+// announces ready, and subscribes for routing targets. A test that needs any of+// that calls facade.start itself — see the alias and port-store tests.+func newTestProxy(profile engineProfile, codec *Codec, disc *Discovery, port int) *Proxy {+ p := NewProxy(codec)+ p.facades = map[string]*facade{+ profile.Name: newFacade(p, profile, disc, port),+ }+ return p+}++// soleFacade returns the one facade a single-engine test proxy was built with.+//+// Production resolves facades by engine, because a process hosts several. A+// test that builds exactly one has nothing to disambiguate, so naming the+// engine at every call site would be noise; tests that do host two address them+// explicitly instead.+func (p *Proxy) soleFacade() *facade {+ p.facadeMu.Lock()+ defer p.facadeMu.Unlock()+ if len(p.facades) != 1 {+ panic(fmt.Sprintf("soleFacade: proxy has %d facades, want exactly 1", len(p.facades)))+ }+ for _, f := range p.facades {+ return f+ }+ return nil+}++func testProxy(p engineProfile, disc *Discovery, port int) *Proxy {+ return newTestProxy(p, NewCodec(rwNop{}), disc, port)+}++// nodeFor turns an httptest server URL into a discovery Node pointing at it.+func nodeFor(t *testing.T, id, serverURL string) Node {+ t.Helper()+ u, err := url.Parse(serverURL)+ if err != nil {+ t.Fatalf("parse %q: %v", serverURL, err)+ }+ host, portStr, err := net.SplitHostPort(u.Host)+ if err != nil {+ t.Fatalf("split %q: %v", u.Host, err)+ }+ port, err := strconv.Atoi(portStr)+ if err != nil {+ t.Fatalf("port %q: %v", portStr, err)+ }+ return Node{ID: id, Addresses: []string{host}, Port: port}+}++func nodeForModel(t *testing.T, id, serverURL, model string) Node {+ t.Helper()+ node := nodeFor(t, id, serverURL)+ node.Models = []string{model}+ return node+}++// TestHandlePlain_OptionsPreflight: a CORS preflight is answered locally with+// 204 + permissive headers and never forwarded.+func TestHandlePlain_OptionsPreflight(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ p := testProxy(tc.profile, NewDiscovery(), tc.profile.FacadePort)+ rec := httptest.NewRecorder()+ req := httptest.NewRequest(http.MethodOptions, tc.inferencePath, nil)+ req.RemoteAddr = "127.0.0.1:40000"+ req.Header.Set("Access-Control-Request-Headers", "X-Custom-Token")+ p.soleFacade().handlePlain(rec, req)++ if rec.Code != http.StatusNoContent {+ t.Fatalf("status = %d, want 204", rec.Code)+ }+ if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {+ t.Errorf("Access-Control-Allow-Origin = %q, want *", got)+ }+ if rec.Header().Get("Access-Control-Allow-Methods") == "" {+ t.Errorf("missing Access-Control-Allow-Methods")+ }+ if got := rec.Header().Get("Access-Control-Expose-Headers"); got != "*" {+ t.Errorf("Access-Control-Expose-Headers = %q, want *", got)+ }+ // The browser's requested headers are echoed so an arbitrary header clears preflight.+ if got := rec.Header().Get("Access-Control-Allow-Headers"); got != "X-Custom-Token" {+ t.Errorf("Access-Control-Allow-Headers = %q, want echoed X-Custom-Token", got)+ }+ })+}++// TestHandlePlain_EngineCredentialedPreflightPreserved: when an engine opts an+// exact origin into credentialed CORS, its preflight policy reaches the browser+// instead of being replaced by the proxy's uncredentialed wildcard fallback.+func TestHandlePlain_EngineCredentialedPreflightPreserved(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ preflightSeen := make(chan struct{}, 1)+ engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ if r.Method != http.MethodOptions {+ t.Errorf("engine method = %s, want OPTIONS", r.Method)+ }+ preflightSeen <- struct{}{}+ w.Header().Set("Access-Control-Allow-Origin", "https://app.example")+ w.Header().Set("Access-Control-Allow-Credentials", "true")+ w.Header().Set("Access-Control-Allow-Methods", "POST")+ w.Header().Set("Access-Control-Allow-Headers", "Content-Type")+ w.WriteHeader(http.StatusNoContent)+ }))+ defer engine.Close()++ disc := NewDiscovery()+ disc.AddManual(nodeFor(t, "engine", engine.URL))+ p := testProxy(tc.profile, disc, tc.profile.FacadePort)+ req := httptest.NewRequest(http.MethodOptions, tc.inferencePath, nil)+ req.RemoteAddr = "127.0.0.1:40000"+ req.Header.Set("Origin", "https://app.example")+ req.Header.Set("Access-Control-Request-Method", http.MethodPost)+ req.Header.Set("Access-Control-Request-Headers", "Content-Type")+ rec := httptest.NewRecorder()++ p.soleFacade().handlePlain(rec, req)++ select {+ case <-preflightSeen:+ default:+ t.Fatal("engine did not receive the credentialed preflight")+ }+ if rec.Code != http.StatusNoContent {+ t.Errorf("status = %d, want %d", rec.Code, http.StatusNoContent)+ }+ if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "https://app.example" {+ t.Errorf("Access-Control-Allow-Origin = %q, want the engine's exact origin", got)+ }+ if got := rec.Header().Get("Access-Control-Allow-Credentials"); got != "true" {+ t.Errorf("Access-Control-Allow-Credentials = %q, want the engine's true", got)+ }+ })+}++// TestHandleHTTP_EngineCORSPolicyPreserved: an engine that declares its own+// origin policy keeps it. Replacing it with the proxy's wildcard would widen+// what the user configured, and would break a credentialed response outright.+func TestHandleHTTP_EngineCORSPolicyPreserved(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {+ w.Header().Set("Access-Control-Allow-Origin", "https://app.example")+ w.Header().Set("Access-Control-Allow-Credentials", "true")+ w.WriteHeader(http.StatusOK)+ io.WriteString(w, `{"done":true}`)+ }))+ defer engine.Close()++ disc := NewDiscovery()+ disc.AddManual(nodeForModel(t, "engine", engine.URL, tc.advertisedModel))+ p := testProxy(tc.profile, disc, tc.profile.FacadePort)++ rec := httptest.NewRecorder()+ p.soleFacade().handleHTTP(rec, tc.inferenceRequest())++ if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "https://app.example" {+ t.Errorf("Access-Control-Allow-Origin = %q, want the engine's own origin", got)+ }+ if got := rec.Header().Get("Access-Control-Allow-Credentials"); got != "true" {+ t.Errorf("Access-Control-Allow-Credentials = %q, want the engine's true", got)+ }+ })+}++// TestHandleHTTP_EngineCredentialsWithoutOriginDropped: an engine (or an+// intermediary in front of it) that sends Allow-Credentials but no origin has+// declared no policy to keep, so the proxy supplies its own. The wildcard it+// writes is invalid next to Allow-Credentials: true, and a browser rejects that+// pair, so the inherited header must not survive the forward.+func TestHandleHTTP_EngineCredentialsWithoutOriginDropped(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {+ w.Header().Set("Access-Control-Allow-Credentials", "true")+ w.WriteHeader(http.StatusOK)+ io.WriteString(w, `{"done":true}`)+ }))+ defer engine.Close()++ disc := NewDiscovery()+ disc.AddManual(nodeForModel(t, "engine", engine.URL, tc.advertisedModel))+ p := testProxy(tc.profile, disc, tc.profile.FacadePort)++ rec := httptest.NewRecorder()+ p.soleFacade().handleHTTP(rec, tc.inferenceRequest())++ if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {+ t.Errorf("Access-Control-Allow-Origin = %q, want the proxy's wildcard", got)+ }+ if got := rec.Header().Get("Access-Control-Allow-Credentials"); got != "" {+ t.Errorf("Access-Control-Allow-Credentials = %q, want cleared alongside the wildcard origin", got)+ }+ })+}++// TestHandleHTTP_HappyPathSingleNode: the common case — one healthy node+// answers directly, body forwarded, CORS present on the success response.+func TestHandleHTTP_HappyPathSingleNode(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ var gotBody string+ good := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ b, _ := io.ReadAll(r.Body)+ gotBody = string(b)+ w.WriteHeader(http.StatusOK)+ io.WriteString(w, `{"done":true}`)+ }))+ defer good.Close()++ disc := NewDiscovery()+ disc.AddManual(nodeForModel(t, "good", good.URL, tc.advertisedModel))+ p := testProxy(tc.profile, disc, tc.profile.FacadePort)++ rec := httptest.NewRecorder()+ p.soleFacade().handleHTTP(rec, tc.inferenceRequest())++ if rec.Code != http.StatusOK {+ t.Fatalf("status = %d, want 200", rec.Code)+ }+ if gotBody != tc.inferenceBody() {+ t.Errorf("node got body %q, want the original request body", gotBody)+ }+ if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {+ t.Errorf("Access-Control-Allow-Origin = %q, want * on success", got)+ }+ })+}++// TestHandleHTTP_NoRetryOn400: a client error (400) is returned as-is and not+// failed over — retrying elsewhere would return the same error and mask it.+func TestHandleHTTP_NoRetryOn400(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ hits := 0+ bad := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ hits+++ w.WriteHeader(http.StatusBadRequest)+ io.WriteString(w, `{"error":"bad request"}`)+ }))+ defer bad.Close()+ other := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.WriteHeader(http.StatusOK)+ }))+ defer other.Close()++ disc := NewDiscovery()+ disc.AddManual(nodeForModel(t, "bad", bad.URL, tc.advertisedModel))+ disc.AddManual(nodeForModel(t, "other", other.URL, tc.advertisedModel))+ p := testProxy(tc.profile, disc, tc.profile.FacadePort)+ p.soleFacade().SetSelected("bad")++ rec := httptest.NewRecorder()+ p.soleFacade().handleHTTP(rec, tc.inferenceRequest())++ if rec.Code != http.StatusBadRequest {+ t.Fatalf("status = %d, want 400 (client errors must not fail over)", rec.Code)+ }+ if hits != 1 {+ t.Errorf("bad node hit %d times, want exactly 1 (no retry on 400)", hits)+ }+ })+}++// TestHandleHTTP_RejectionHasCORS: even the no-node rejection carries CORS so a+// browser sees the real 502 instead of an opaque CORS error.+func TestHandleHTTP_RejectionHasCORS(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ p := testProxy(tc.profile, NewDiscovery(), tc.profile.FacadePort)+ rec := httptest.NewRecorder()+ p.soleFacade().handleHTTP(rec, tc.inferenceRequest())++ if rec.Code != http.StatusBadGateway {+ t.Fatalf("status = %d, want 502", rec.Code)+ }+ if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {+ t.Errorf("Access-Control-Allow-Origin = %q, want * on rejection", got)+ }+ })+}++// TestHandleHTTP_FailoverOn503: a busy first node (503) is skipped and the+// request is filled by the next node, with the original body replayed.+func TestHandleHTTP_FailoverOn503(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ busy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.WriteHeader(http.StatusServiceUnavailable)+ io.WriteString(w, `{"error":"loading model"}`)+ }))+ defer busy.Close()++ var gotBody string+ good := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ b, _ := io.ReadAll(r.Body)+ gotBody = string(b)+ w.WriteHeader(http.StatusOK)+ io.WriteString(w, `{"done":true}`)+ }))+ defer good.Close()++ disc := NewDiscovery()+ disc.AddManual(nodeForModel(t, "busy", busy.URL, tc.advertisedModel))+ disc.AddManual(nodeForModel(t, "good", good.URL, tc.advertisedModel))+ p := testProxy(tc.profile, disc, tc.profile.FacadePort)+ p.soleFacade().SetSelected("busy") // deterministic: busy is tried first++ rec := httptest.NewRecorder()+ p.soleFacade().handleHTTP(rec, tc.inferenceRequest())++ if rec.Code != http.StatusOK {+ t.Fatalf("status = %d, want 200 (should have failed over past the 503)", rec.Code)+ }+ if gotBody != tc.inferenceBody() {+ t.Errorf("failover node got body %q, want the original request body", gotBody)+ }+ if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {+ t.Errorf("Access-Control-Allow-Origin = %q, want * on proxied success", got)+ }+ })+}++// TestHandleHTTP_AllNodesDownReturnsError: when every candidate fails at the+// transport, the client gets one clean 502 (not a hang), still with CORS.+func TestHandleHTTP_AllNodesDownReturnsError(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ // Two servers we immediately close so dials fail.+ a := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))+ b := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))+ na := nodeForModel(t, "a", a.URL, tc.advertisedModel)+ nb := nodeForModel(t, "b", b.URL, tc.advertisedModel)+ a.Close()+ b.Close()++ disc := NewDiscovery()+ disc.AddManual(na)+ disc.AddManual(nb)+ p := testProxy(tc.profile, disc, tc.profile.FacadePort)++ rec := httptest.NewRecorder()+ p.soleFacade().handleHTTP(rec, tc.inferenceRequest())++ if rec.Code != http.StatusBadGateway {+ t.Fatalf("status = %d, want 502 when all nodes are down", rec.Code)+ }+ if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {+ t.Errorf("Access-Control-Allow-Origin = %q, want * on exhausted error", got)+ }+ })+}++// TestHandleHTTP_404FailoverInferenceOnly: a 404 (model-not-found) on an+// inference call fails over to the next advertised owner, but a 404 on a+// non-inference path is returned as-is.+func TestHandleHTTP_404FailoverInferenceOnly(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ missing := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.WriteHeader(http.StatusNotFound)+ io.WriteString(w, `{"error":"model not found"}`)+ }))+ defer missing.Close()+ has := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.WriteHeader(http.StatusOK)+ io.WriteString(w, `{"done":true}`)+ }))+ defer has.Close()++ newProxy := func() *Proxy {+ disc := NewDiscovery()+ disc.AddManual(nodeForModel(t, "missing", missing.URL, tc.advertisedModel))+ disc.AddManual(nodeForModel(t, "has", has.URL, tc.advertisedModel))+ p := testProxy(tc.profile, disc, tc.profile.FacadePort)+ p.soleFacade().SetSelected("missing")+ return p+ }++ // Inference POST: 404 on first → fail over → 200.+ rec := httptest.NewRecorder()+ newProxy().soleFacade().handleHTTP(rec, tc.inferenceRequest())+ if rec.Code != http.StatusOK {+ t.Fatalf("inference 404: status = %d, want 200 (should fail over)", rec.Code)+ }++ // An ordinary non-inference GET still returns the first node's 404.+ rec = httptest.NewRecorder()+ newProxy().soleFacade().handleHTTP(rec, httptest.NewRequest(http.MethodGet, tc.nonInferencePath, nil))+ if rec.Code != http.StatusNotFound {+ t.Fatalf("non-inference 404: status = %d, want 404 (must NOT fail over)", rec.Code)+ }+ })+}++// TestHandleHTTP_AggregatesNativeModelList is Ollama-scoped because the+// native "models" dialect is Ollama's alone — LM Studio serves no /api/tags.+// The OpenAI dialect both engines share is covered by the sibling test below.+func TestHandleHTTP_AggregatesNativeModelList(t *testing.T) {+ entered := make(chan struct{}, 2)+ release := make(chan struct{})+ server := func(body string) *httptest.Server {+ return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ if r.Method != http.MethodGet || r.URL.Path != "/api/tags" {+ t.Errorf("upstream request = %s %s, want GET /api/tags", r.Method, r.URL.Path)+ }+ if r.Header.Get("Authorization") != "" || r.Header.Get("Cookie") != "" {+ t.Errorf("client credentials leaked to fan-out target")+ }+ entered <- struct{}{}+ <-release+ _, _ = io.WriteString(w, body)+ }))+ }+ a := server(`{"models":[{"name":"a","model":"a","digest":"a-only"},{"name":"shared","digest":"first"}]}`)+ defer a.Close()+ b := server(`{"models":[{"name":"shared:latest","model":"shared:latest","digest":"second"},{"name":"c","model":"c","digest":"c-only"}]}`)+ defer b.Close()+ malformed := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {+ _, _ = io.WriteString(w, `{"models":null}`)+ }))+ defer malformed.Close()+ down := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))+ downNode := nodeFor(t, "down", down.URL)+ down.Close()++ disc := NewDiscovery()+ disc.AddManual(nodeFor(t, "a", a.URL))+ disc.AddManual(nodeFor(t, "b", b.URL))+ disc.AddManual(downNode)+ disc.AddManual(nodeFor(t, "malformed", malformed.URL))+ events := &recRW{}+ p := newTestProxy(ollamaOnlyProfile(t), NewCodec(events), disc, 11434)+ p.soleFacade().SetSelected("a")+ rec := httptest.NewRecorder()+ req := httptest.NewRequest(http.MethodGet, "/api/tags", nil)+ req.Header.Set("Authorization", "Bearer client-secret")+ req.Header.Set("Cookie", "session=client-secret")+ done := make(chan struct{})+ go func() {+ p.soleFacade().handleHTTP(rec, req)+ close(done)+ }()++ for range 2 {+ select {+ case <-entered:+ case <-time.After(5 * time.Second):+ close(release)+ t.Fatal("model-list requests were not issued concurrently")+ }+ }+ close(release)+ select {+ case <-done:+ case <-time.After(5 * time.Second):+ t.Fatal("aggregate request did not finish")+ }++ if rec.Code != http.StatusOK {+ t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())+ }+ var got struct {+ Models []struct {+ Name string `json:"name"`+ Model string `json:"model"`+ Digest string `json:"digest"`+ } `json:"models"`+ }+ if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {+ t.Fatal(err)+ }+ if len(got.Models) != 3 || got.Models[0].Model != "a" || got.Models[1].Name != "shared" || got.Models[1].Model != "" || got.Models[2].Model != "c" {+ t.Fatalf("models = %+v, want a, shared, c", got.Models)+ }+ if got.Models[1].Digest != "first" {+ t.Errorf("duplicate metadata = %q, want deterministic first candidate", got.Models[1].Digest)+ }+ if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {+ t.Errorf("Access-Control-Allow-Origin = %q, want *", got)+ }+ // Addressed to the engine, like every facade-scoped notification: the+ // broker's process-scoped router claims only workload and node-activity+ // methods, so an unaddressed request event is dropped rather than relayed.+ if !events.has(`"method":"ollama:proxy/request-started"`) ||+ !events.has(`"method":"ollama:proxy/request"`) ||+ !events.has(`"target":"cluster"`) {+ t.Errorf("aggregate telemetry missing paired addressed cluster events: %s", events.b)+ }+}++// TestHandleHTTP_AggregatesOpenAIModelList runs for both engines: /v1/models+// is the one model-list route they share, and the dedupe must keep the first+// candidate's metadata on either.+func TestHandleHTTP_AggregatesOpenAIModelList(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ serve := func(body string) *httptest.Server {+ return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ if r.Method != http.MethodGet || r.URL.Path != "/v1/models" {+ t.Errorf("upstream request = %s %s, want GET /v1/models", r.Method, r.URL.Path)+ }+ _, _ = io.WriteString(w, body)+ }))+ }+ a := serve(`{"object":"list","data":[{"id":"a","owned_by":"a"},{"id":"shared","owned_by":"first"}]}`)+ defer a.Close()+ b := serve(`{"object":"list","data":[{"id":"shared","owned_by":"second"},{"id":"c","owned_by":"b"}]}`)+ defer b.Close()++ disc := NewDiscovery()+ disc.AddManual(nodeFor(t, "a", a.URL))+ disc.AddManual(nodeFor(t, "b", b.URL))+ p := testProxy(tc.profile, disc, tc.profile.FacadePort)+ p.soleFacade().SetSelected("a")+ rec := httptest.NewRecorder()+ p.soleFacade().handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/v1/models", nil))++ var got struct {+ Object string `json:"object"`+ Data []struct {+ ID string `json:"id"`+ OwnedBy string `json:"owned_by"`+ } `json:"data"`+ }+ if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {+ t.Fatal(err)+ }+ if rec.Code != http.StatusOK || got.Object != "list" || len(got.Data) != 3 {+ t.Fatalf("response = %d %+v", rec.Code, got)+ }+ if got.Data[0].ID != "a" || got.Data[1].ID != "shared" || got.Data[1].OwnedBy != "first" || got.Data[2].ID != "c" {+ t.Fatalf("models = %+v, want a, shared(first), c", got.Data)+ }+ })+}++func TestHandleHTTP_ModelListEmptyAndUnavailable(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ serveEmpty := func() *httptest.Server {+ return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {+ _, _ = io.WriteString(w, tc.emptyModelList)+ }))+ }++ // An unreachable sole node is 503, not an empty list: the client must+ // be able to tell "nothing is answering" from "nothing is loaded".+ down := serveEmpty()+ downNode := nodeFor(t, "empty", down.URL)+ down.Close()++ disc := NewDiscovery()+ disc.AddManual(downNode)+ rec := httptest.NewRecorder()+ testProxy(tc.profile, disc, tc.profile.FacadePort).soleFacade().+ handleHTTP(rec, httptest.NewRequest(http.MethodGet, tc.modelListPath, nil))+ if rec.Code != http.StatusServiceUnavailable {+ t.Fatalf("unavailable status = %d, want 503", rec.Code)+ }++ // A reachable node with no models is 200 and this engine's own empty+ // envelope, so its client can parse the response.+ empty := serveEmpty()+ defer empty.Close()+ disc = NewDiscovery()+ disc.AddManual(nodeFor(t, "empty", empty.URL))+ rec = httptest.NewRecorder()+ testProxy(tc.profile, disc, tc.profile.FacadePort).soleFacade().+ handleHTTP(rec, httptest.NewRequest(http.MethodGet, tc.modelListPath, nil))+ if rec.Code != http.StatusOK || rec.Body.String() != tc.emptyModelList {+ t.Fatalf("empty response = %d %s, want 200 %s", rec.Code, rec.Body.String(), tc.emptyModelList)+ }+ })+}++// TestHandleHTTP_StrictModelRouting proves capability is a gate before+// selection and priority, on both engines. The matching node advertises the+// model in the engine's own spelling — which for Ollama differs from what the+// client requests, so this also pins that normalizeModel is applied on the+// gate rather than only on the model list.+func TestHandleHTTP_StrictModelRouting(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ missHits, unknownHits, matchHits := 0, 0, 0+ miss := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {+ missHits+++ w.WriteHeader(http.StatusOK)+ }))+ defer miss.Close()+ unknown := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {+ unknownHits+++ w.WriteHeader(http.StatusOK)+ }))+ defer unknown.Close()+ match := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ matchHits+++ body, _ := io.ReadAll(r.Body)+ if string(body) != tc.inferenceBody() {+ t.Errorf("matching node got body %q", body)+ }+ w.WriteHeader(http.StatusOK)+ }))+ defer match.Close()++ disc := NewDiscovery()+ missNode := nodeFor(t, "selected-miss", miss.URL)+ missNode.Models = []string{"a-model-this-engine-does-not-have"}+ unknownNode := nodeFor(t, "a-unknown", unknown.URL)+ matchNode := nodeFor(t, "z-match", match.URL)+ matchNode.Models = []string{tc.advertisedModel}+ disc.AddManual(missNode)+ disc.AddManual(unknownNode)+ disc.AddManual(matchNode)+ p := testProxy(tc.profile, disc, tc.profile.FacadePort)+ p.soleFacade().SetSelected("selected-miss")+ p.SetPriority([]string{"a-unknown", "selected-miss", "z-match"})+ candidates := p.soleFacade().resolveCandidates(tc.requestedModel)+ if len(candidates) != 1 || candidates[0].id != "z-match" {+ t.Fatalf("model candidates = %v, want only z-match", candidates)+ }++ rec := httptest.NewRecorder()+ p.soleFacade().handleHTTP(rec, tc.inferenceRequest())++ if rec.Code != http.StatusOK {+ t.Fatalf("status = %d, want 200", rec.Code)+ }+ if missHits != 0 || unknownHits != 0 || matchHits != 1 {+ t.Fatalf("hits miss=%d unknown=%d match=%d, want 0/0/1", missHits, unknownHits, matchHits)+ }++ // Capability filtering applies only to model-bearing inference.+ rec = httptest.NewRecorder()+ p.soleFacade().handleHTTP(rec, httptest.NewRequest(http.MethodGet, tc.nonInferencePath, nil))+ if rec.Code != http.StatusOK {+ t.Fatalf("non-inference status = %d, want 200", rec.Code)+ }+ if missHits != 1 || unknownHits != 0 || matchHits != 1 {+ t.Fatalf("non-inference hits miss=%d unknown=%d match=%d, want 1/0/1", missHits, unknownHits, matchHits)+ }+ })+}++func TestHandleHTTP_NoAdvertisedModelRejectsLocally(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ hits := 0+ upstream := func() *httptest.Server {+ return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {+ hits+++ w.WriteHeader(http.StatusOK)+ }))+ }+ missing := upstream()+ defer missing.Close()+ unknown := upstream()+ defer unknown.Close()++ disc := NewDiscovery()+ disc.AddManual(nodeForModel(t, "missing", missing.URL, "a-model-this-engine-does-not-have"))+ disc.AddManual(nodeFor(t, "unknown", unknown.URL))+ events := &recRW{}+ p := newTestProxy(tc.profile, NewCodec(events), disc, tc.profile.FacadePort)+ p.soleFacade().SetSelected("missing")++ rec := httptest.NewRecorder()+ p.soleFacade().handleHTTP(rec, tc.inferenceRequest())++ if rec.Code != http.StatusBadGateway ||+ !strings.Contains(rec.Body.String(), "no available node advertises the requested model") {+ t.Fatalf("response = %d %s, want actionable local 502", rec.Code, rec.Body.String())+ }+ if hits != 0 {+ t.Fatalf("ineligible upstreams received %d requests, want 0", hits)+ }+ if !events.has("no node advertises requested model") {+ t.Fatalf("missing rejected request event: %s", events.b)+ }+ })+}++// TestResolveCandidates_SelfGuard: a node resolving to the proxy's own+// listen address is dropped so we never self-forward.+func TestResolveCandidates_SelfGuard(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ port := tc.profile.FacadePort+ disc := NewDiscovery()+ disc.AddManual(Node{ID: "self", Addresses: []string{"127.0.0.1"}, Port: port})+ disc.AddManual(Node{ID: "real", Addresses: []string{"192.0.2.10"}, Port: port})+ p := testProxy(tc.profile, disc, port)++ cands := p.soleFacade().resolveCandidates("")+ var haveReal bool+ for _, c := range cands {+ if c.id == "self" {+ t.Errorf("self-target node must be excluded, got candidate %+v", c)+ }+ if c.id == "real" {+ haveReal = true+ }+ }+ if !haveReal {+ t.Errorf("expected the real node to survive the self-guard, candidates = %+v", cands)+ }+ })+}diff --git a/services/ollama-proxy/go.mod b/services/nvpair-proxy/go.modsimilarity index 70%rename from services/ollama-proxy/go.modrename to services/nvpair-proxy/go.modindex 73e423e..b3e5457 100644--- a/services/ollama-proxy/go.mod+++ b/services/nvpair-proxy/go.mod@@ -1,4 +1,4 @@-module ollama-proxy+module nvpair-proxygo 1.25.0@@ -11,9 +11,8 @@ require (github.com/cenkalti/backoff v2.2.1+incompatible // indirectgithub.com/grandcat/zeroconf v1.0.0 // indirectgithub.com/miekg/dns v1.1.55 // indirect- golang.org/x/mod v0.17.0 // indirect+ golang.org/x/mod v0.12.0 // indirectgolang.org/x/net v0.58.0 // indirect- golang.org/x/sync v0.10.0 // indirectgolang.org/x/sys v0.47.0 // indirect- golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect+ golang.org/x/tools v0.11.0 // indirect)diff --git a/services/lmstudio-proxy/go.sum b/services/nvpair-proxy/go.sumsimilarity index 83%rename from services/lmstudio-proxy/go.sumrename to services/nvpair-proxy/go.sumindex fe333ff..c65e083 100644--- a/services/lmstudio-proxy/go.sum+++ b/services/nvpair-proxy/go.sum@@ -12,8 +12,8 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEgolang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=-golang.org/x/mod v0.17.0 h1:zY54UmvipHiNd+pm+m0x9KhZ9hl1/7QNMyxXbc6ICqA=-golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=+golang.org/x/mod v0.12.0 h1:rmsUpXtvNzj340zd98LZ4KntptpfRHwpFOHG188oHXc=+golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=@@ -21,8 +21,8 @@ golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLgolang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=-golang.org/x/sync v0.10.0 h1:3NQrjDixjgGwUOCaF8w2+VYHv0Ve/vGYSbdkTa98gmQ=-golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=+golang.org/x/sync v0.3.0 h1:ftCYgMx6zT/asHUrPw8BLLscYtGznsLAnjq5RH9P66E=+golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=@@ -30,6 +30,6 @@ golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=-golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d h1:vU5i/LfpvrRCpgM/VPfJLg5KjxD3E+hfT1SH+d9zLwg=-golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=+golang.org/x/tools v0.11.0 h1:EMCa6U9S2LtZXLAMoWiR/R8dAQFRqbAitmbJ2UKhoi8=+golang.org/x/tools v0.11.0/go.mod h1:anzJrxPjNtfgiYQYirP2CPGzGLxrH2u2QBhn6Bf3qY8=golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=diff --git a/services/ollama-proxy/ingress.go b/services/nvpair-proxy/ingress.gosimilarity index 75%rename from services/ollama-proxy/ingress.gorename to services/nvpair-proxy/ingress.goindex 984b7b2..828c98e 100644--- a/services/ollama-proxy/ingress.go+++ b/services/nvpair-proxy/ingress.go@@ -5,6 +5,7 @@ package mainimport ("encoding/json"+ "fmt""log/slog""net""net/http"@@ -30,21 +31,38 @@ type localBackend struct {Healthy bool `json:"healthy"`}+// currentLocalBackend snapshots the configured local engine.+func (f *facade) currentLocalBackend() localBackend {+ f.backendMu.RLock()+ defer f.backendMu.RUnlock()+ return f.backend+}+// setLocalBackend records (or, with a zero port / unhealthy flag, effectively-// clears) the local engine the ingress serves.-func (p *Proxy) setLocalBackend(b localBackend) {- p.backendMu.Lock()- p.backend = b- p.backendMu.Unlock()+// clears) the local engine this facade's ingress serves.+//+// A non-loopback host is rejected rather than stored. The ingress forwards a+// pin-authenticated peer's request straight here without consulting discovery,+// so an off-box host would turn this node into a relay to an address chosen by+// whoever can reach the control channel. The broker only ever sends 127.0.0.1;+// this is the same defence-in-depth re-validation setLoopbackAlias performs on+// the alias the broker sends it.+func (f *facade) setLocalBackend(b localBackend) error {+ if b.Host != "" && !isLoopbackHost(b.Host) {+ return fmt.Errorf("local backend host %q is not loopback", b.Host)+ }+ f.backendMu.Lock()+ defer f.backendMu.Unlock()+ f.backend = b+ return nil}// localBackendTarget returns the loopback URL of the current local engine, and// false when none is set/healthy (the ingress then answers 503 rather than-// forwarding). The host defaults to 127.0.0.1 and is always loopback.-func (p *Proxy) localBackendTarget() (*url.URL, bool) {- p.backendMu.RLock()- b := p.backend- p.backendMu.RUnlock()+// forwarding). The host defaults to 127.0.0.1, and setLocalBackend refuses to+// store anything that is not loopback, so this is always a loopback target.+func (f *facade) localBackendTarget() (*url.URL, bool) {+ b := f.currentLocalBackend()if b.Port <= 0 || !b.Healthy {return nil, false}@@ -60,7 +78,7 @@ func (p *Proxy) localBackendTarget() (*url.URL, bool) {// caller — any LAN peer — is refused; peers must use the mTLS ingress. This is// what closes the former open-relay exposure (the listener still binds all// interfaces for the TLS personality, but plaintext is loopback-only).-func (p *Proxy) handlePlain(w http.ResponseWriter, r *http.Request) {+func (f *facade) handlePlain(w http.ResponseWriter, r *http.Request) {if !isLoopbackRemote(r.RemoteAddr) {// Answer a non-loopback preflight ahead of the gate. It grants no access// on its own; the request that follows still receives the real 403. A@@ -78,10 +96,11 @@ func (p *Proxy) handlePlain(w http.ResponseWriter, r *http.Request) {// Engine-manager marks its private identity/action requests so this// compatibility facade can never be mistaken for the local Ollama backend.if r.Header.Get(engineIdentityProbeHeader) == "1" {- writeIngressError(w, http.StatusConflict, "proxy-facade", "the compatibility facade is not an Ollama engine")+ writeIngressError(w, http.StatusConflict, "proxy-facade",+ "the compatibility facade is not the "+f.profile.DisplayName+" engine")return}- p.handleHTTP(w, r)+ f.handleHTTP(w, r)}// handleClusterIngress is the LAN mTLS personality: it authenticates the caller@@ -91,19 +110,19 @@ func (p *Proxy) handlePlain(w http.ResponseWriter, r *http.Request) {// sole authorization boundary (a trusted peer is treated like a local client),// so the two personalities stay behaviorally identical toward the engine. It// never calls resolveCandidates, so a peer request cannot be re-routed onward.-func (p *Proxy) handleClusterIngress(w http.ResponseWriter, r *http.Request) {+func (f *facade) handleClusterIngress(w http.ResponseWriter, r *http.Request) {// Re-derive membership and pins per request so a cluster left, or a peer// paired or removed, after startup is reflected immediately without a proxy// restart — a removed peer must stop being accepted right away, which is the// whole point of the gate.- p.mesh.Refresh()- peer, ok := p.mesh.VerifyClientPin(r)+ f.host.mesh.Refresh()+ peer, ok := f.host.mesh.VerifyClientPin(r)if !ok {writeIngressError(w, http.StatusForbidden, "cluster-auth","client certificate is not a pinned member of this node's cluster")return}- target, ok := p.localBackendTarget()+ target, ok := f.localBackendTarget()if !ok {writeIngressError(w, http.StatusServiceUnavailable, "no-local-backend","no local inference backend is available on this node")@@ -111,24 +130,24 @@ func (p *Proxy) handleClusterIngress(w http.ResponseWriter, r *http.Request) {}slog.Debug("cluster ingress forwarding to local backend","peer", peer, "method", r.Method, "path", r.URL.Path, "target", target.Host)- p.reverseProxyToLocal(w, r, target)+ f.reverseProxyToLocal(w, r, target)}// reverseProxyToLocal streams the request to the local engine, preserving// cancellation (the request context is the proxy's root context, so a client// disconnect or shutdown tears down the upstream call and stops generation).-func (p *Proxy) reverseProxyToLocal(w http.ResponseWriter, r *http.Request, target *url.URL) {- p.newLocalReverseProxy(target).ServeHTTP(w, r)+func (f *facade) reverseProxyToLocal(w http.ResponseWriter, r *http.Request, target *url.URL) {+ f.newLocalReverseProxy(target).ServeHTTP(w, r)}-func (p *Proxy) newLocalReverseProxy(target *url.URL) *httputil.ReverseProxy {+func (f *facade) newLocalReverseProxy(target *url.URL) *httputil.ReverseProxy {return &httputil.ReverseProxy{Director: func(req *http.Request) {req.URL.Scheme = target.Schemereq.URL.Host = target.Hostreq.Host = target.Host},- Transport: p.plainHTTPTransport(),+ Transport: f.host.plainHTTPTransport(),ErrorHandler: func(ew http.ResponseWriter, _ *http.Request, err error) {slog.Warn("cluster ingress upstream error", "target", target.Host, "err", err)writeIngressError(ew, http.StatusBadGateway, "backend-error", "local inference backend error")diff --git a/services/ollama-proxy/ingress_test.go b/services/nvpair-proxy/ingress_test.gosimilarity index 74%rename from services/ollama-proxy/ingress_test.gorename to services/nvpair-proxy/ingress_test.goindex 34934a1..7d5cc8a 100644--- a/services/ollama-proxy/ingress_test.go+++ b/services/nvpair-proxy/ingress_test.go@@ -7,6 +7,7 @@ import ("net/http""net/http/httptest""net/url"+ "strings""testing")@@ -27,9 +28,9 @@ func TestResolveCandidatesUnclusteredDropsRelayPeers(t *testing.T) {ID: "manual-x", Host: "manual-x", Port: 11434,Addresses: []string{"192.0.2.20"}, IP: "192.0.2.20",})- p := testProxy(disc, 11435) // mesh nil => unclustered+ p := testProxy(anyProfile(t), disc, 11435) // mesh nil => unclustered- cands := p.resolveCandidates("")+ cands := p.soleFacade().resolveCandidates("")if len(cands) != 1 {t.Fatalf("unclustered candidate set = %+v, want exactly the manual node", cands)}@@ -42,12 +43,12 @@ func TestResolveCandidatesUnclusteredDropsRelayPeers(t *testing.T) {// loopback-only: a LAN caller is refused (closing the former open-relay), so// peers cannot use the plaintext path at all.func TestHandlePlainRejectsNonLoopback(t *testing.T) {- p := testProxy(NewDiscovery(), 11435)+ p := testProxy(anyProfile(t), NewDiscovery(), 11435)req := httptest.NewRequest(http.MethodPost, "/api/generate", nil)req.RemoteAddr = "192.0.2.50:40000"rec := httptest.NewRecorder()- p.handlePlain(rec, req)+ p.soleFacade().handlePlain(rec, req)if rec.Code != http.StatusForbidden {t.Fatalf("non-loopback plaintext status = %d, want %d", rec.Code, http.StatusForbidden)}@@ -63,12 +64,12 @@ func TestHandlePlainRejectsNonLoopback(t *testing.T) {// that follows is still rejected — but without it the browser never sends that// request and reports the refusal as a generic CORS failure.func TestHandlePlainAnswersPreflightBeforeLoopbackGate(t *testing.T) {- p := testProxy(NewDiscovery(), 11435)+ p := testProxy(anyProfile(t), NewDiscovery(), 11435)req := httptest.NewRequest(http.MethodOptions, "/api/generate", nil)req.RemoteAddr = "192.0.2.50:40000"rec := httptest.NewRecorder()- p.handlePlain(rec, req)+ p.soleFacade().handlePlain(rec, req)if rec.Code != http.StatusNoContent {t.Fatalf("preflight status = %d, want %d", rec.Code, http.StatusNoContent)}@@ -77,28 +78,37 @@ func TestHandlePlainAnswersPreflightBeforeLoopbackGate(t *testing.T) {}}+// engine-manager marks its own identity probes so the compatibility facade can+// never be adopted as the engine itself. The rejection has to name the engine+// this proxy fronts, or the operator reading it is sent to the wrong process —+// which is why this runs per engine rather than asserting only the status.func TestHandlePlainRejectsEngineIdentityProbe(t *testing.T) {- p := testProxy(NewDiscovery(), 11435)- req := httptest.NewRequest(http.MethodGet, "/api/version", nil)- req.RemoteAddr = "127.0.0.1:40000"- req.Header.Set(engineIdentityProbeHeader, "1")- rec := httptest.NewRecorder()+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ p := testProxy(tc.profile, NewDiscovery(), tc.profile.StandalonePort)+ req := httptest.NewRequest(http.MethodGet, "/api/version", nil)+ req.RemoteAddr = "127.0.0.1:40000"+ req.Header.Set(engineIdentityProbeHeader, "1")+ rec := httptest.NewRecorder()- p.handlePlain(rec, req)- if rec.Code != http.StatusConflict {- t.Fatalf("identity probe status = %d, want %d", rec.Code, http.StatusConflict)- }+ p.soleFacade().handlePlain(rec, req)+ if rec.Code != http.StatusConflict {+ t.Fatalf("identity probe status = %d, want %d", rec.Code, http.StatusConflict)+ }+ if body := rec.Body.String(); !strings.Contains(body, tc.profile.DisplayName) {+ t.Fatalf("rejection does not name %s: %s", tc.profile.DisplayName, body)+ }+ })}// TestHandleClusterIngressUnclusteredForbids proves the mTLS ingress fails// closed on an unclustered node (nil mesh): with no cluster identity there are// no pins, so every caller is rejected.func TestHandleClusterIngressUnclusteredForbids(t *testing.T) {- p := testProxy(NewDiscovery(), 11435)+ p := testProxy(anyProfile(t), NewDiscovery(), 11435)req := httptest.NewRequest(http.MethodPost, "/api/generate", nil) // no client certrec := httptest.NewRecorder()- p.handleClusterIngress(rec, req)+ p.soleFacade().handleClusterIngress(rec, req)if rec.Code != http.StatusForbidden {t.Fatalf("unclustered ingress status = %d, want %d", rec.Code, http.StatusForbidden)}@@ -123,10 +133,10 @@ func TestIsLoopbackRemote(t *testing.T) {}func TestLocalReverseProxyUsesSharedPlainTransport(t *testing.T) {- p := testProxy(NewDiscovery(), 11435)+ p := testProxy(anyProfile(t), NewDiscovery(), 11435)shared := p.plainHTTPTransport()target := &url.URL{Scheme: "http", Host: "127.0.0.1:1"}- rp := p.newLocalReverseProxy(target)+ rp := p.soleFacade().newLocalReverseProxy(target)tr, ok := rp.Transport.(*http.Transport)if !ok {t.Fatalf("Transport type = %T, want *http.Transport", rp.Transport)diff --git a/services/lmstudio-proxy/ipc.go b/services/nvpair-proxy/ipc.gosimilarity index 100%rename from services/lmstudio-proxy/ipc.gorename to services/nvpair-proxy/ipc.godiff --git a/services/lmstudio-proxy/main.go b/services/nvpair-proxy/main.gosimilarity index 77%rename from services/lmstudio-proxy/main.gorename to services/nvpair-proxy/main.goindex 846ac9c..ebfa902 100644--- a/services/lmstudio-proxy/main.go+++ b/services/nvpair-proxy/main.go@@ -16,11 +16,10 @@ import ("nvpair-shared/applog""nvpair-shared/clustertrust"+ "nvpair-shared/engines")func main() {- port := flag.Int("port", defaultProxyPort, "HTTP listen port")- ignorePersistedPort := flag.Bool("ignore-persisted-port", false, "use --port even when a persisted port exists")ipcPath := flag.String("ipc", "", "IPC endpoint: Unix domain socket path or Windows named pipe (default: stdin/stdout)")clusterDir := flag.String("cluster-dir", "", "cluster trust directory (node.crt/key + trusted pins); enables the LAN mTLS inference ingress when this node is clustered")showVersion := flag.Bool("version", false, "print version and exit")@@ -32,7 +31,15 @@ func main() {os.Exit(0)}- applog.Init("lmstudio-proxy", resolveLevel())+ // The process starts with no engine facade and no listener. Which engines+ // it fronts, on which ports, and with which alias addresses all arrive over+ // facade/enable, because the broker computes a different port plan per+ // engine and a flag cannot express more than one of them.+ //+ // The log component is therefore the process, not an engine: there is no+ // profile to name here. Facade-scoped records carry an engine field so a+ // line is still attributable once several facades share this sink.+ applog.Init(engines.ProxyComponent, resolveLevel())var transport io.ReadWriteCloserif *ipcPath != "" {@@ -62,17 +69,8 @@ func main() {}}()- // Restore a previously chosen port (set via set-port) over the- // --port/default, so the proxy comes back up where the user last put it.- persisted, hasPersisted := loadPersistedPort()- effectivePort := chooseStartupPort(*port, *ignorePersistedPort, persisted, hasPersisted)- if hasPersisted && !*ignorePersistedPort && effectivePort == persisted {- log.Printf("restored persisted proxy port %d", persisted)- }-codec := NewCodec(transport)- disc := NewDiscovery()- proxy := NewProxy(codec, disc, effectivePort)+ proxy := NewProxy(codec)// Open a live view of this node's cluster mTLS trust fabric. While unclustered// the proxy serves only the loopback plaintext personality; once this node is// a member the same listener also serves the pin-gated LAN mTLS ingress, and@@ -83,6 +81,7 @@ func main() {// presence: a left/removed node keeps its keypair by design, and would// otherwise keep logging cluster_ingress with no cluster peers to serve.proxy.mesh = clustertrust.Open(*clusterDir)+go proxy.mesh.Watch(ctx, func(clustered bool) {slog.Info("cluster inference ingress switched personality", "cluster_ingress", clustered)proxy.dropUnpinnedPeerTransports()diff --git a/services/lmstudio-proxy/portstore.go b/services/nvpair-proxy/portstore.gosimilarity index 60%rename from services/lmstudio-proxy/portstore.gorename to services/nvpair-proxy/portstore.goindex 98ff450..357f165 100644--- a/services/lmstudio-proxy/portstore.go+++ b/services/nvpair-proxy/portstore.go@@ -11,33 +11,34 @@ import ("nvpair-shared/appdir")-const proxyPortFile = "lmstudio-proxy-port.json"--const (- defaultProxyPort = 1234- legacyDefaultProxyPort = 1235-)-type persistedPort struct {Port int `json:"port"`}-func chooseStartupPort(flagPort int, ignorePersisted bool, persisted int, hasPersisted bool) int {- if ignorePersisted || !hasPersisted || persisted == legacyDefaultProxyPort {+// chooseStartupPort resolves the port to bind. A previously chosen port wins+// over the flag so the proxy comes back where the user last put it, except+// when the broker forces an explicit port or when the stored value is one the+// engine itself owns — restoring that would put the proxy on the engine's+// port and guarantee a bind conflict.+func chooseStartupPort(p engineProfile, flagPort int, ignorePersisted bool, persisted int, hasPersisted bool) int {+ if ignorePersisted || !hasPersisted {+ return flagPort+ }+ if p.ReservedPersistedPort != 0 && persisted == p.ReservedPersistedPort {return flagPort}return persisted}-func proxyPortPath() (string, error) {- return appdir.Path(proxyPortFile)+func proxyPortPath(p engineProfile) (string, error) {+ return appdir.Path(p.PortFile)}// loadPersistedPort returns the previously chosen proxy port, if a valid one// was saved. Any error (no file, bad JSON, out-of-range) reports "none" so// startup falls back to the --port flag / default.-func loadPersistedPort() (int, bool) {- path, err := proxyPortPath()+func loadPersistedPort(p engineProfile) (int, bool) {+ path, err := proxyPortPath(p)if err != nil {return 0, false}@@ -57,8 +58,8 @@ func loadPersistedPort() (int, bool) {// savePersistedPort atomically writes the chosen port (tmp + rename) so a// crash mid-write can't leave a truncated file behind.-func savePersistedPort(port int) error {- path, err := proxyPortPath()+func savePersistedPort(p engineProfile, port int) error {+ path, err := proxyPortPath(p)if err != nil {return err}diff --git a/services/nvpair-proxy/portstore_test.go b/services/nvpair-proxy/portstore_test.gonew file mode 100644index 0000000..e63c260--- /dev/null+++ b/services/nvpair-proxy/portstore_test.go@@ -0,0 +1,153 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++import (+ "bytes"+ "context"+ "fmt"+ "net"+ "os"+ "strings"+ "testing"+ "time"+)++// redirectConfigDir points os.UserConfigDir() at a temp dir for the test, so+// the persisted-port file doesn't touch the real per-user config. Sets all+// four env vars os.UserConfigDir() consults across platforms: XDG_CONFIG_HOME+// on Linux, $HOME/Library on macOS, and APPDATA/LOCALAPPDATA on Windows.+// Missing any one of them means the test reads and writes the developer's+// real file, clobbering their saved port and failing on the next run.+func redirectConfigDir(t *testing.T) {+ t.Helper()+ dir := t.TempDir()+ t.Setenv("XDG_CONFIG_HOME", dir)+ t.Setenv("HOME", dir)+ t.Setenv("APPDATA", dir)+ t.Setenv("LOCALAPPDATA", dir)+}++// freeTCPPort returns a port that was free a moment ago.+//+// A bind probe cannot be held open and handed over, so the port is genuinely+// free when checked and may not be by the time a facade binds it. Callers that+// can tolerate a retry should use one — see enableOnFreePort in+// twofacade_test.go — because this helper cannot close that window.+func freeTCPPort(t *testing.T) int {+ t.Helper()+ l, err := net.Listen("tcp", "127.0.0.1:0")+ if err != nil {+ t.Fatalf("reserve free port: %v", err)+ }+ defer l.Close()+ return l.Addr().(*net.TCPAddr).Port+}++// Each engine persists to its own file, so one engine's saved port can never+// be read back as the other's.+func TestPersistedPortRoundTrip(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ redirectConfigDir(t)++ if _, ok := loadPersistedPort(tc.profile); ok {+ t.Fatal("expected no persisted port before any save")+ }+ if err := savePersistedPort(tc.profile, 11500); err != nil {+ t.Fatalf("savePersistedPort: %v", err)+ }+ if p, ok := loadPersistedPort(tc.profile); !ok || p != 11500 {+ t.Errorf("round-trip: got %d ok=%v, want 11500", p, ok)+ }++ // An out-of-range stored value is treated as "none" so startup falls+ // back to the flag/default rather than trying to bind port 0.+ path, err := proxyPortPath(tc.profile)+ if err != nil {+ t.Fatalf("proxyPortPath: %v", err)+ }+ if err := os.WriteFile(path, []byte(`{"port":0}`), 0o644); err != nil {+ t.Fatal(err)+ }+ if _, ok := loadPersistedPort(tc.profile); ok {+ t.Error("port 0 should be treated as none")+ }+ })+}++// The two engines must not share a persisted-port file, or moving one proxy+// would silently move the other on its next start.+func TestPersistedPortIsPerEngine(t *testing.T) {+ redirectConfigDir(t)++ ollama := ollamaCase(t).profile+ lmstudio := lmstudioCase(t).profile++ if err := savePersistedPort(ollama, 11500); err != nil {+ t.Fatal(err)+ }+ if _, ok := loadPersistedPort(lmstudio); ok {+ t.Fatal("LM Studio read a port only Ollama saved")+ }+ if err := savePersistedPort(lmstudio, 1300); err != nil {+ t.Fatal(err)+ }+ if p, ok := loadPersistedPort(ollama); !ok || p != 11500 {+ t.Errorf("Ollama's port changed when LM Studio saved: got %d ok=%v", p, ok)+ }+}++// TestSetPortRebinds drives a live rebind: the proxy starts serving on one+// port, set-port moves it to another, and afterward the new port accepts+// connections, the old one doesn't, the choice is persisted, and a fresh+// ready notification carries the new port.+func TestSetPortRebinds(t *testing.T) {+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ redirectConfigDir(t)++ buf := &bytes.Buffer{}+ codec := NewCodec(buf)+ disc := NewDiscovery()++ portA := freeTCPPort(t)+ proxy := newTestProxy(tc.profile, codec, disc, portA)++ lnA, err := net.Listen("tcp", fmt.Sprintf(":%d", portA))+ if err != nil {+ t.Fatalf("listen on port A: %v", err)+ }+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ proxy.soleFacade().serveHTTP(ctx, lnA)+ defer proxy.shutdown(context.Background())++ portB := freeTCPPort(t)+ if err := proxy.soleFacade().setPort(portB); err != nil {+ t.Fatalf("setPort: %v", err)+ }++ // New port is now serving.+ conn, err := net.DialTimeout("tcp", fmt.Sprintf("127.0.0.1:%d", portB), 2*time.Second)+ if err != nil {+ t.Fatalf("new port %d not listening after rebind: %v", portB, err)+ }+ conn.Close()++ // Old port stopped accepting.+ if c, err := net.DialTimeout("tcp", fmt.Sprintf("127.0.0.1:%d", portA), 500*time.Millisecond); err == nil {+ c.Close()+ t.Errorf("old port %d should be closed after rebind", portA)+ }++ // Persisted for next startup.+ if p, ok := loadPersistedPort(tc.profile); !ok || p != portB {+ t.Errorf("persisted port: got %d ok=%v, want %d", p, ok, portB)+ }++ // A fresh ready notification announced the new port.+ if !strings.Contains(buf.String(), fmt.Sprintf("\"port\":%d", portB)) {+ t.Errorf("expected ready notification carrying port %d, got %q", portB, buf.String())+ }+ })+}diff --git a/services/ollama-proxy/priority_test.go b/services/nvpair-proxy/priority_test.gosimilarity index 70%rename from services/ollama-proxy/priority_test.gorename to services/nvpair-proxy/priority_test.goindex df6b4d5..8d47ea2 100644--- a/services/ollama-proxy/priority_test.go+++ b/services/nvpair-proxy/priority_test.go@@ -7,6 +7,8 @@ import ("encoding/json""strconv""testing"++ "nvpair-shared/schedulerwire")// prNode builds a discovery Node with a single non-local address so@@ -24,7 +26,7 @@ func prProxy(t *testing.T, ids ...string) *Proxy {for _, id := range ids {disc.AddManual(prNode(id))}- return testProxy(disc, 11435)+ return testProxy(anyProfile(t), disc, 11435)}func candidateIDs(p *Proxy) []string {@@ -32,7 +34,7 @@ func candidateIDs(p *Proxy) []string {}func candidateIDsForModel(p *Proxy, model string) []string {- cands := p.resolveCandidates(model)+ cands := p.soleFacade().resolveCandidates(model)out := make([]string, 0, len(cands))for _, c := range cands {out = append(out, c.id)@@ -79,7 +81,7 @@ func TestResolveCandidates_UnknownIgnored(t *testing.T) {func TestResolveCandidates_ManualPinOverridesPriority(t *testing.T) {p := prProxy(t, "a", "b", "c")p.SetPriority([]string{"a", "b", "c"})- p.SetSelected("b")+ p.soleFacade().SetSelected("b")assertOrder(t, candidateIDs(p), []string{"b", "a", "c"})}@@ -90,12 +92,12 @@ func TestResolveCandidates_FiltersBeforeSelectionAndPriority(t *testing.T) {for _, n := range []Node{a, prNode("b"), c, d} {disc.AddManual(n)}- p := testProxy(disc, 11435)- p.SetSelected("d")+ p := testProxy(anyProfile(t), disc, 11435)+ p.soleFacade().SetSelected("d")p.SetPriority([]string{"d", "b", "c", "a"})assertOrder(t, candidateIDsForModel(p, "llama"), []string{"c", "a"})- p.SetSelected("a")+ p.soleFacade().SetSelected("a")assertOrder(t, candidateIDsForModel(p, "llama"), []string{"a", "c"})}@@ -125,14 +127,14 @@ func TestSetPriority_CountAndCopy(t *testing.T) {// TestHandleSetPriority_Response: the node/set-priority request returns {count}.func TestHandleSetPriority_Response(t *testing.T) {rec := &recRW{}- p := NewProxy(NewCodec(rec), NewDiscovery(), 11435)+ p := newTestProxy(anyProfile(t), NewCodec(rec), NewDiscovery(), 11435)id := json.RawMessage(`7`)p.handleMessage(&Message{JSONRPC: "2.0",ID: &id,Method: "node/set-priority",- Params: json.RawMessage(`{"nodes":["x","y"]}`),+ Params: json.RawMessage(`{"generation":1,"nodes":["x","y"]}`),})if !rec.has(`"count":2`) {@@ -142,3 +144,39 @@ func TestHandleSetPriority_Response(t *testing.T) {t.Fatalf("stored priority = %v, want [x y]", got)}}++// A generation is required, because this method is relayed verbatim from any+// client. An unversioned snapshot would clear the reservations without+// advancing the epoch, so a reservation taken before it would still match and+// release one taken after — leaving the node reading idle and attracting every+// later dispatch.+func TestHandleSetPriority_RejectsUnversionedSnapshot(t *testing.T) {+ rec := &recRW{}+ p := newTestProxy(anyProfile(t), NewCodec(rec), NewDiscovery(), 11435)++ // Establish a baseline plus a live reservation against it.+ applySnapshot(p, schedulerwire.Priority{Nodes: []string{"x", "y"}})+ _, held := p.reserveCandidate(p.soleFacade(), reservationCandidates("x", "y"))+ if !held.held {+ t.Fatal("no reservation was taken from the baseline")+ }++ id := json.RawMessage(`8`)+ p.handleMessage(&Message{+ JSONRPC: "2.0",+ ID: &id,+ Method: "node/set-priority",+ Params: json.RawMessage(`{"nodes":["y","x"]}`),+ })++ if !rec.has(`"code":-32602`) {+ t.Fatalf("unversioned snapshot was not rejected: %s", rec.b)+ }+ // The baseline and the reservation both survive the rejection.+ if got := p.PriorityList(); len(got) != 2 || got[0] != "x" {+ t.Errorf("a rejected snapshot changed the baseline: %v", got)+ }+ if got := reservationCount(p, held.nodeID); got != 1 {+ t.Errorf("a rejected snapshot cleared the reservation on %q: %d", held.nodeID, got)+ }+}diff --git a/services/ollama-proxy/proxy.go b/services/nvpair-proxy/proxy.gosimilarity index 71%rename from services/ollama-proxy/proxy.gorename to services/nvpair-proxy/proxy.goindex ad4ac7a..33a32bd 100644--- a/services/ollama-proxy/proxy.go+++ b/services/nvpair-proxy/proxy.go@@ -19,24 +19,22 @@ import ("net/http""net/http/httputil""net/url"+ "runtime/debug""sort""strconv""strings""sync"- "sync/atomic""time""nvpair-shared/applog""nvpair-shared/clustertrust""nvpair-shared/cors"- "nvpair-shared/errors"+ "nvpair-shared/engines""nvpair-shared/netmon""nvpair-shared/netpick""nvpair-shared/nodeactivity""nvpair-shared/noderec"- "nvpair-shared/reach""nvpair-shared/schedulerwire"- "nvpair-shared/splitlisten")// Version is stamped at build time via -ldflags "-X main.Version=...".@@ -48,17 +46,27 @@ type ReadyParams struct {Port int `json:"port"`}-// ErrorParams is sent as a JSON-RPC "error" notification when the-// proxy encounters a fatal startup-time condition it wants to surface-// to the orchestrator before exiting. Code is a short machine-readable-// tag ("bind-failed" today); Message is a human-friendly string suitable-// for an error bar.+// ErrorParams is sent as a JSON-RPC "error" notification when a facade hits a+// bring-up condition it wants to surface to the orchestrator. Code is a short+// machine-readable tag ("bind-failed" today); Message is a human-friendly+// string suitable for an error bar.+//+// It is no longer a pre-exit notice: a failed bind ends one facade's enable,+// not the process. The broker also learns this from the enable response, which+// carries codeFacadeBindFailed; this notification is what lets it choose the+// fallback port before that response is dispatched.type ErrorParams struct {Code string `json:"code"`Message string `json:"message"`Port int `json:"port,omitempty"`}+// codeFacadeBindFailed is the JSON-RPC error code for a facade/enable that+// failed only because the port was taken. The broker matches on it to retry on+// a fallback port; see enableProxyFacadeWithFallback in nvpair-ui-broker. It+// sits in the implementation-defined -32000..-32099 server range.+const codeFacadeBindFailed = -32010+type NodesResult struct {Nodes []Node `json:"nodes"`}@@ -71,6 +79,13 @@ type SelectedResult struct {ID string `json:"id"`}+// Request events are facade-scoped: they name an engine's method, path and+// target, so they go out addressed like every other facade notification. They+// were unaddressed until the process began hosting several facades, at which+// point the broker's process-scoped router — which claims only workload and+// node-activity methods — silently dropped them instead of relaying them to+// <engine>-proxy: subscribers.+//// RequestStartedEvent is emitted as a `proxy/request-started`// notification the moment we've resolved a target node and are about// to forward the request to it. Pairs with the existing@@ -138,31 +153,18 @@ const (workloadStartedMethod = "workload:started"workloadCompletedMethod = "workload:completed"workloadErroredMethod = "workload:errored"-- // workloadEngine is the opaque engine identifier carried in every- // workload this proxy produces. The proxy only ever fronts Ollama.- workloadEngine = "ollama")-// inferenceEndpoints is the set of request paths that count as cluster-// workloads. Health checks, model listings (/api/tags), and other control-// traffic are deliberately excluded so we don't flood the cluster with-// non-inference noise. Both the native Ollama and OpenAI-compatible-// inference routes are included.-var inferenceEndpoints = map[string]bool{- "/api/generate": true,- "/api/chat": true,- "/api/embeddings": true,- "/api/embed": true,- "/v1/chat/completions": true,- "/v1/completions": true,- "/v1/embeddings": true,-}--// isInferenceRequest reports whether a request should be tracked as a-// workload — a POST to one of the known inference endpoints.-func isInferenceRequest(method, path string) bool {- return method == http.MethodPost && inferenceEndpoints[path]+// isInferenceRequest reports whether a request should be tracked as a cluster+// workload. Health checks, model listings and other control traffic are+// deliberately excluded so we don't flood the cluster with non-inference+// noise; the engine's profile declares which paths qualify.+//+// Kept a free function taking the profile rather than a method, so the+// classification is testable without constructing a Proxy.+func isInferenceRequest(p engineProfile, method, path string) bool {+ role, ok := p.roleFor(method, path)+ return ok && role == roleInferencePOST}// Workload mirrors the workload-manager spec 6 object. The proxy populates@@ -320,40 +322,30 @@ func (sc *statusCapture) FlushError() error {}type Proxy struct {- codec *Codec- discovery *Discovery- cancel context.CancelFunc-- // httpMu guards port, the servers, the split listener, and ln across a- // live set-port rebind. The HTTP handlers never read port (they route by- // upstream node), so the only contention is set-port vs set-port- // (serialized) and the initial serveHTTP store vs a later rebind.- httpMu sync.Mutex- port int- aliasAddr string- aliasAltAddr string- plainSrv *http.Server- tlsSrv *http.Server- split *splitlisten.Splitter- ln net.Listener- aliasLn net.Listener- aliasAltLn net.Listener+ // facadeMu guards facades, which is empty until the broker enables one. The+ // process starts with no engine presence at all, so every reader has to+ // tolerate a miss rather than assume a facade exists.+ //+ // Keyed by engine because one process hosts every enabled engine's facade.+ // Each owns its own listeners, ports and dialect; what they share is this+ // host's transports, workload stream, and reservation map. See facade.go.+ facadeMu sync.Mutex+ facades map[string]*facade++ // serveCtx is the process lifetime, and is what a facade's HTTP servers use+ // as their BaseContext. It must not be an enable request's context: that one+ // is cancelled when the request returns, which would cancel the context of+ // every request the facade later serves.+ serveCtx context.Context++ codec *Codec+ cancel context.CancelFunc// mesh is this node's cluster mTLS trust fabric, loaded from --cluster-dir.// nil = unclustered: the LAN TLS ingress accepts nothing and the node does// only loopback-plaintext local routing. Read-only after startup.mesh *clustertrust.Mesh- // backendMu guards backend, the explicit loopback engine the cluster mTLS- // ingress forwards to. The broker sets/clears it via node/set-local-backend;- // it is never sourced from discovery, so an ingress request can only ever- // reach this node's own local engine and can never be re-routed to a peer.- backendMu sync.RWMutex- backend localBackend-- selectedMu sync.RWMutex- selectedID string-// activity coalesces the liveness reports raised when a peer's engine streams// response bytes back through us (see reportActivity).activity *nodeactivity.Reporter@@ -369,47 +361,75 @@ type Proxy struct {priorityGPUPressure map[string]intpriorityReservations map[string]int- // targets remembers, per node, which of its published addresses accepted a- // connection, so a repeated forward costs no confirmation. An entry is- // re-confirmed when the node's candidate list changes and forgotten on an- // upstream error, so the next request fails over to another address.- targets *reach.Chooser+ // appliedPriorityGeneration is the newest snapshot generation applied, so a+ // redelivered or superseded one cannot clear reservations twice. See+ // SetPrioritySnapshot.+ appliedPriorityGeneration uint64// transportMu guards the long-lived HTTP transports reused across forwards// and model-list fetches. Allocating a new http.Transport per request// defeats connection pooling and leaks idle sockets until GC.+ //+ // Shared across facades: the pool is keyed by peer, not by engine, so a+ // second copy would double the connections this node opens to every peer.transportMu sync.MutexplainTransport *http.TransportpeerTransports map[string]*http.Transport- // nextRequestID is a monotonic counter for tagging RequestStarted /- // RequestEvent pairs. Atomic add returns the new value, so request- // IDs start at 1 and never collide within a single proxy lifetime.- // IDs deliberately reset across restarts — they're only meaningful- // while the orchestrator's in-flight map is also alive, and a- // fresh proxy session always starts that map empty on the- // orchestrator side via the proxy:stopped → proxy:ready event- // pair.- nextRequestID atomic.Uint64-// runID is a per-process nonce minted at startup and stamped on every- // workload this proxy emits. It makes a workload's identity- // (originatedFrom, engine, runId, id) globally unique even though- // nextRequestID resets to 1 on restart and the LM Studio proxy also counts- // from 1 — without it, two concurrent cross-engine jobs, or a reused id- // after a restart, would collide in the broker's store.+ // workload this proxy emits. Shared across facades is correct: a workload's+ // identity is (originatedFrom, engine, runId, id), so engine already+ // separates two facades' records and only the process needs the nonce. It+ // exists because each facade's request counter restarts at 1, so without it+ // a reused id after a restart would collide in the broker's store.runID string}-func NewProxy(codec *Codec, discovery *Discovery, port int) *Proxy {+// NewProxy builds a facade-less process host. Facades arrive via+// enableFacade, so nothing is listening when this returns.+func NewProxy(codec *Codec) *Proxy {return &Proxy{- codec: codec,- discovery: discovery,- port: port,- targets: reach.NewChooser(),- runID: newRunID(),- activity: nodeactivity.NewReporter(activityReportInterval),+ codec: codec,+ runID: newRunID(),+ activity: nodeactivity.NewReporter(activityReportInterval),+ }+}++// facadeFor resolves the facade an addressed message names, or nil when that+// engine has no facade enabled here.+//+// An unaddressed facade-scoped message resolves to nothing: an empty engine is+// not a key. Falling back to "the only facade" would misroute as soon as a+// second one exists, which is the bug addressing is here to prevent.+func (p *Proxy) facadeFor(engine string) *facade {+ p.facadeMu.Lock()+ defer p.facadeMu.Unlock()+ return p.facades[engine]+}++// enabledFacades snapshots the live facades so a caller can iterate without+// holding facadeMu across work that might take it again.+func (p *Proxy) enabledFacades() []*facade {+ p.facadeMu.Lock()+ defer p.facadeMu.Unlock()+ out := make([]*facade, 0, len(p.facades))+ for _, f := range p.facades {+ out = append(out, f)}+ return out+}++// requireFacade resolves the facade for an engine-scoped request, answering the+// caller itself when that engine has none. Every engine-scoped method needs it+// now that the process starts with no facade at all: the alternative is a nil+// dereference on a request that simply arrived early.+func (p *Proxy) requireFacade(msg *Message, engine string) (*facade, bool) {+ if f := p.facadeFor(engine); f != nil {+ return f, true+ }+ p.codec.RespondError(msg.ID, -32002,+ fmt.Sprintf("no facade is enabled for engine %q on this proxy", engine))+ return nil, false}// activityReportInterval is how often a single node's streaming may raise a@@ -449,6 +469,7 @@ func newRunID() string {func (p *Proxy) Run(ctx context.Context) error {ctx, cancel := context.WithCancel(ctx)p.cancel = cancel+ p.serveCtx = ctxdefer cancel()// Keep the "is this address local?" set fresh as interfaces come and go,@@ -456,53 +477,12 @@ func (p *Proxy) Run(ctx context.Context) error {// or a sleep/wake IP reassignment.startLocalAddrWatch(ctx)- // Bind synchronously before announcing "ready": if the port is- // already in use, we want the UI to see a real error reason- // instead of being stuck on "Proxy running" while ListenAndServe- // silently fails in a goroutine.- ln, err := p.listen()- if err != nil {- // Best-effort: notify the orchestrator with a structured- // reason. The process will exit non-zero regardless (main.go- // log.Fatalf's on a non-nil Run error), so a failed Notify- // here is not worth surfacing separately.- _ = p.codec.Notify("error", ErrorParams{- Code: "bind-failed",- Message: fmt.Sprintf("failed to bind port %d: %v", p.port, err),- Port: p.port,- })- return fmt.Errorf("failed to bind port %d: %w", p.port, err)- }- // Reserve the inherited OLLAMA_HOST alias before announcing readiness.- // That keeps a successful ready event truthful for clients that start as- // soon as NVPAIR comes up. Alias conflicts are deliberately non-fatal: the- // primary facade remains available and the existing owner is untouched.- p.bindLoopbackAlias()-- if err := p.codec.Notify("ready", ReadyParams{- Version: Version,- Port: p.port,- }); err != nil {- // Don't leave a dangling listener holding the port if we- // couldn't even tell the orchestrator about it.- _ = ln.Close()- p.closeLoopbackAlias()- return fmt.Errorf("failed to send ready notification: %w", err)- }-- // Routing targets come from the broker's discovery relay. Subscribe- // for ol nodes; they arrive as discovery:nodes snapshots (handled in- // handleMessage), each replacing the subscribed overlay. Non-fatal: if the- // parent isn't a relay-aware broker the proxy still routes to manual nodes.- slog.Debug("subscribing to discovery relay for routing targets", "service", string(noderec.ServiceOllama))- if err := p.codec.Notify(noderec.MethodSubscribe, noderec.SubscribeParams{Services: []noderec.ServiceKey{noderec.ServiceOllama}}); err != nil {- slog.Warn("failed to subscribe to discovery relay", "err", err)- }-- p.serveHTTP(ctx, ln)- p.serveLoopbackAlias()-- err = p.readLoop(ctx)+ // No facade is brought up here: the process serves the JSON-RPC channel+ // first and binds only when the broker enables an engine. A bind failure is+ // therefore an enable failure, reported in that call's response, rather+ // than a process exit — which is what lets one engine lose a bind race+ // without taking the others down.+ err := p.readLoop(ctx)// The app is going away (stdin closed or ctx cancelled). Stop any// inference requests still in flight rather than letting them run to@@ -565,276 +545,20 @@ var modelListClient = &http.Client{Timeout: proxyDialTimeout,}-// listen binds the proxy's TCP listener synchronously so bind failures-// (EADDRINUSE and friends) can be reported through a structured error-// notification before the process exits. The caller is responsible for-// closing the returned listener if it doesn't hand it to serveHTTP.-func (p *Proxy) listen() (net.Listener, error) {- return net.Listen("tcp", fmt.Sprintf(":%d", p.port))-}--// serveHTTP takes the already-bound base listener and drives the two proxy-// personalities over it: a plaintext HTTP server (loopback-only, full local-// router) and a LAN mTLS ingress (pin-gated, forwards to the local engine),-// split by the connection's first byte via nvpair-shared/splitlisten. The two-// http.Servers are recorded so set-port can rebind both onto a fresh split-// without tearing the servers down.-func (p *Proxy) serveHTTP(ctx context.Context, ln net.Listener) {- base := func(_ net.Listener) context.Context { return ctx }- plainSrv := &http.Server{- Handler: http.HandlerFunc(p.handlePlain),- BaseContext: base,- ReadHeaderTimeout: proxyReadHeaderTimeout,- IdleTimeout: proxyServerIdleTimeout,- }- tlsSrv := &http.Server{- Handler: http.HandlerFunc(p.handleClusterIngress),- BaseContext: base,- ReadHeaderTimeout: proxyReadHeaderTimeout,- IdleTimeout: proxyServerIdleTimeout,- }-- p.httpMu.Lock()- p.plainSrv = plainSrv- p.tlsSrv = tlsSrv- p.ln = ln- p.startSplitLocked(ln)- p.httpMu.Unlock()-- slog.Info("proxy timeouts configured",- "dial_timeout", proxyDialTimeout,- "keep_alive", proxyKeepAlive,- "response_header_timeout", proxyResponseTimeout,- "max_idle_conns", proxyMaxIdleConns,- "idle_conn_timeout", proxyIdleConnTimeout,- )- slog.Info("HTTP proxy listening", "port", p.port, "addr", ln.Addr().String(),- "cluster_ingress", p.mesh.Clustered())-}--// setLoopbackAlias validates and records the optional secondary plaintext-// listener. The broker supplies this only for an inherited local HTTP-// OLLAMA_HOST. Validate again here so a direct invocation can never turn the-// alias flag into a LAN plaintext listener.-func (p *Proxy) setLoopbackAlias(address string) error {- host, portText, err := net.SplitHostPort(address)- if err != nil {- return fmt.Errorf("expected host:port: %w", err)- }- port, err := strconv.Atoi(portText)- if err != nil || port < 1 || port > 65535 {- return fmt.Errorf("port must be between 1 and 65535")- }- ip := net.ParseIP(host)- if strings.EqualFold(strings.TrimSuffix(host, "."), "localhost") {- host = "127.0.0.1"- address = net.JoinHostPort(host, portText)- } else if ip == nil || !ip.IsLoopback() {- return fmt.Errorf("host %q is not loopback", host)- }- if port == p.port {- return fmt.Errorf("alias port %d is already the primary listener", port)- }- switch {- case p.aliasAddr == "":- p.aliasAddr = address- case p.aliasAddr == address || p.aliasAltAddr == address:- return nil- case p.aliasAltAddr == "":- _, firstPort, _ := net.SplitHostPort(p.aliasAddr)- if firstPort != portText {- return fmt.Errorf("alias addresses must use the same port")- }- p.aliasAltAddr = address- default:- return fmt.Errorf("at most two alias addresses are supported")- }- return nil-}--// bindLoopbackAlias reserves the configured alias before the proxy announces-// ready. A conflict is non-fatal: the primary stays healthy, the existing owner-// is untouched, and a sticky warning tells the user how to recover.-func (p *Proxy) bindLoopbackAlias() {- p.httpMu.Lock()- addresses := []string{p.aliasAddr, p.aliasAltAddr}- p.httpMu.Unlock()- if addresses[0] == "" {- return- }-- listeners := make([]net.Listener, 0, 2)- for _, address := range addresses {- if address == "" {- continue- }- ln, err := net.Listen("tcp", address)- if err != nil {- for _, opened := range listeners {- _ = opened.Close()- }- p.reportLoopbackAliasBlocked(address, err)- return- }- listeners = append(listeners, ln)- }- p.httpMu.Lock()- p.aliasLn = listeners[0]- if len(listeners) > 1 {- p.aliasAltLn = listeners[1]- }- p.httpMu.Unlock()- if err := p.codec.Notify("errors:clear", errors.ClearParams{ID: ollamaHostAliasBlockedID}); err != nil {- slog.Debug("failed to clear OLLAMA_HOST alias warning", "err", err)- }- slog.Info("OLLAMA_HOST loopback alias reserved", "addresses", addresses)-}--func (p *Proxy) serveLoopbackAlias() {- p.httpMu.Lock()- addresses := []string{p.aliasAddr, p.aliasAltAddr}- listeners := []net.Listener{p.aliasLn, p.aliasAltLn}- plainSrv := p.plainSrv- p.httpMu.Unlock()- if listeners[0] == nil || plainSrv == nil {- return- }- for i, ln := range listeners {- if ln == nil {- continue- }- address := addresses[i]- go func() {- if err := plainSrv.Serve(ln); err != nil && err != http.ErrServerClosed && !stderrors.Is(err, net.ErrClosed) {- slog.Error("OLLAMA_HOST alias server exited", "address", address, "err", err)- }- }()- }- slog.Info("OLLAMA_HOST loopback alias listening", "addresses", addresses)-}--func (p *Proxy) closeLoopbackAlias() {- p.httpMu.Lock()- listeners := []net.Listener{p.aliasLn, p.aliasAltLn}- p.aliasLn = nil- p.aliasAltLn = nil- p.httpMu.Unlock()- for _, ln := range listeners {- if ln != nil {- _ = ln.Close()- }- }-}--const ollamaHostAliasBlockedID = "ollama-proxy:ollama-host-alias-blocked"--func (p *Proxy) reportLoopbackAliasBlocked(address string, bindErr error) {- message := fmt.Sprintf(- "NVPAIR kept its primary Ollama compatibility proxy separate, but could not claim the local OLLAMA_HOST alias %s: %v. Stop or reconfigure the application using that port, or change or unset OLLAMA_HOST, then restart NVPAIR. No process was stopped.",- address, bindErr)- if err := p.codec.Notify("errors:report", errors.ServiceError{- ID: ollamaHostAliasBlockedID,- Message: message,- Severity: "warning",- Action: "none",- }); err != nil {- slog.Debug("failed to report OLLAMA_HOST alias warning", "err", err)- }- slog.Warn("OLLAMA_HOST alias unavailable", "address", address, "err", bindErr)-}--// startSplitLocked wraps base in a first-byte splitter and starts both servers-// on its sub-listeners. Caller holds httpMu. Reuses the persistent plainSrv /-// tlsSrv so set-port can call it repeatedly on fresh listeners.-func (p *Proxy) startSplitLocked(base net.Listener) {- split := splitlisten.New(base)- p.split = split- go func() {- if err := p.plainSrv.Serve(split.Plain()); err != nil && err != http.ErrServerClosed {- slog.Error("plaintext HTTP server exited", "err", err)- }- }()- go p.serveTLS(split.TLS())-}--// serveTLS terminates cluster mTLS on the split's TLS sub-listener. The server-// certificate is resolved per handshake from the live mesh, so this one-// sub-listener covers both states: while this node is unclustered there is no-// leaf to present and the handshake is refused (it exposes no LAN inference-// surface), and the moment the node becomes a member the same sub-listener-// serves the pin-gated ingress — no rebind, and no process restart to pick up a-// freshly-minted identity.-func (p *Proxy) serveTLS(l net.Listener) {- if err := p.tlsSrv.Serve(tls.NewListener(l, p.mesh.ServerTLSConfig())); err != nil && err != http.ErrServerClosed {- slog.Error("cluster mTLS ingress exited", "err", err)- }-}--// shutdown gracefully stops both personalities and closes the split listener.+// shutdown stops every facade and then releases the process-wide outbound+// transport pool. The pool is closed here rather than in facade.stopServing+// because it is shared: closing it per facade would drop another engine's+// pooled connections to every peer.func (p *Proxy) shutdown(ctx context.Context) {- p.httpMu.Lock()- plainSrv, tlsSrv, split := p.plainSrv, p.tlsSrv, p.split- aliasLn, aliasAltLn := p.aliasLn, p.aliasAltLn- p.httpMu.Unlock()- if plainSrv != nil {- _ = plainSrv.Shutdown(ctx)- }- if tlsSrv != nil {- _ = tlsSrv.Shutdown(ctx)- }- if split != nil {- _ = split.Close()- }- if aliasLn != nil {- _ = aliasLn.Close()- }- if aliasAltLn != nil {- _ = aliasAltLn.Close()+ // Every facade, then the shared transports: the pool is process-wide, so+ // closing it before a facade has drained would cut that facade's own+ // in-flight upstream connections.+ for _, f := range p.enabledFacades() {+ f.stopServing(ctx)}p.closeIdleTransports()}-// setPort live-rebinds the HTTP listener onto newPort and persists the choice-// so it survives a restart. It binds the new listener first (so a bind-// failure leaves the current one serving), starts the same server on it, then-// closes the old listener — in-flight connections on the old port drain-// naturally. A fresh `ready` notification announces the new port so the-// orchestrator/UI learn where the proxy is now listening.-func (p *Proxy) setPort(newPort int) error {- p.httpMu.Lock()- defer p.httpMu.Unlock()-- if newPort == p.port {- return nil- }- newLn, err := net.Listen("tcp", fmt.Sprintf(":%d", newPort))- if err != nil {- return fmt.Errorf("failed to bind port %d: %w", newPort, err)- }- oldSplit := p.split- p.ln = newLn- p.port = newPort-- // Re-serve both personalities on a fresh split over the new listener, then- // close the old split (and its base listener) so in-flight connections on- // the old port drain naturally. The plaintext and mTLS personalities always- // move together as one unit.- slog.Info("HTTP proxy listening", "port", newPort, "addr", newLn.Addr().String(),- "cluster_ingress", p.mesh.Clustered())- p.startSplitLocked(newLn)- if oldSplit != nil {- _ = oldSplit.Close()- }-- if err := savePersistedPort(newPort); err != nil {- slog.Warn("failed to persist proxy port", "port", newPort, "err", err)- }- if err := p.codec.Notify("ready", ReadyParams{Version: Version, Port: newPort}); err != nil {- slog.Warn("failed to emit ready after rebind", "err", err)- }- return nil-}-// emitWorkload sends a workload:* lifecycle notification to the// orchestrator. The broker stamps the origin (originatedFrom) and forwards it to the// workload-manager; a failed write is logged but never blocks the request.@@ -950,7 +674,7 @@ func (p *Proxy) closeIdleTransports() {// because this package aliases nvpair-shared/errors as `errors` (which has no New).type retrySignal struct{}-func (retrySignal) Error() string { return "ollama-proxy: retry next candidate" }+func (retrySignal) Error() string { return "nvpair-proxy: retry next candidate" }type modelListItem struct {key string@@ -964,22 +688,16 @@ type modelListResult struct {err error}-func ollamaModelKey(model string) string {- model = strings.TrimSpace(model)- name := model[strings.LastIndex(model, "/")+1:]- if name != "" && !strings.ContainsAny(name, ":@") {- return model + ":latest"- }- return model-}--// serveModelList queries every Ollama candidate concurrently and returns the-// native /api/tags or OpenAI /v1/models envelope with duplicate model records-// removed. Results are merged in candidate order, not completion order, so-// duplicate metadata is deterministic while an unavailable peer cannot hide-// healthy inventories.-func (p *Proxy) serveModelList(w http.ResponseWriter, r *http.Request, candidates []candidate) (int, error) {- openAI := r.URL.Path == "/v1/models"+// serveModelList queries every candidate concurrently and returns the engine's+// model-list envelope with duplicate model records removed. Results are merged+// in candidate order, not completion order, so duplicate metadata is+// deterministic while an unavailable peer cannot hide healthy inventories.+//+// role selects the wire dialect: which array the upstream envelope carries,+// which field identifies a record, and how the federated response is shaped.+func (f *facade) serveModelList(w http.ResponseWriter, r *http.Request, role routeRole, candidates []candidate) (int, error) {+ p := f.host+ openAI := role == roleModelListOpenAIGETwriteJSON := func(status int, body []byte) {cors.Apply(w.Header())w.Header().Set("Content-Type", "application/json")@@ -1013,7 +731,7 @@ func (p *Proxy) serveModelList(w http.ResponseWriter, r *http.Request, candidatedefer wg.Done()resp, err := client.Do(req)if err != nil {- p.targets.Forget(cand.id)+ f.targets.Forget(cand.id)results[i].err = errreturn}@@ -1065,7 +783,7 @@ func (p *Proxy) serveModelList(w http.ResponseWriter, r *http.Request, candidateif key == "" {key = identity.Name}- key = ollamaModelKey(key)+ key = f.profile.normalizeModel(key)}if key == "" {results[i].err = fmt.Errorf("model record has no identity")@@ -1126,7 +844,8 @@ func (p *Proxy) serveModelList(w http.ResponseWriter, r *http.Request, candidatereturn http.StatusOK, nil}-func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) {+func (f *facade) handleHTTP(w http.ResponseWriter, r *http.Request) {+ p := f.hoststart := time.Now()// Allocate the request ID up front so both code paths (rejection@@ -1135,33 +854,60 @@ func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) {// appear in any orchestrator in-flight map — that's fine; the// completion event still bumps the failed counter regardless of// whether a matching Started was seen.- reqID := strconv.FormatUint(p.nextRequestID.Add(1), 10)+ reqID := strconv.FormatUint(f.nextRequestID.Add(1), 10)// Parse the request's model before choosing a node. Model eligibility only// applies to inference routes; control endpoints retain their existing// routing behavior even when their JSON happens to contain a model field.bodyBytes, model := bufferBodyAndModel(r)- isInf := isInferenceRequest(r.Method, r.URL.Path)+ isInf := isInferenceRequest(f.profile, r.Method, r.URL.Path)routingModel := ""if isInf {routingModel = model}- candidates := p.resolveCandidates(routingModel)+ candidates := f.resolveCandidates(routingModel)+ // held is this request's claim on a node's capacity while it is in flight.+ // resMu guards it because failover re-points it from the ReverseProxy's+ // callbacks while the disconnect watcher may be releasing it.+ var (+ resMu sync.Mutex+ held reservation+ )if isInf && model != "" {- candidates = p.reserveCandidate(candidates)- }- if r.Method == http.MethodGet && (r.URL.Path == "/api/tags" || r.URL.Path == "/v1/models") {+ candidates, held = p.reserveCandidate(f, candidates)+ }+ // Released exactly once, whichever way the request ends: normal unwind,+ // client disconnect, or an early return below. Without this a node stays+ // counted as loaded until the next scheduler snapshot, so a steady trickle+ // of short requests makes it look busier than it is and drives dispatch+ // away from it.+ //+ // Both accessors release resMu by defer. The release itself runs from a+ // defer, so a resMu stranded by a panic in the move would deadlock this+ // request against its own cleanup.+ moveHeld := func(nodeID string) {+ resMu.Lock()+ defer resMu.Unlock()+ held = p.moveReservation(held, nodeID)+ }+ defer func() {+ resMu.Lock()+ defer resMu.Unlock()+ p.releaseReservation(held)+ held = reservation{}+ }()+ if role, ok := f.profile.roleFor(r.Method, r.URL.Path); ok && role.isModelList() {if len(candidates) > 0 {- p.codec.Notify("proxy/request-started", RequestStartedEvent{+ _ = f.notify("proxy/request-started", RequestStartedEvent{ID: reqID, Method: r.Method, Path: r.URL.Path, Target: "cluster",})}- status, err := p.serveModelList(w, r, candidates)+ status, err := f.serveModelList(w, r, role, candidates)errText := ""if err != nil {errText = err.Error()}- p.codec.Notify("proxy/request", RequestEvent{+ _ = f.notify("proxy/request", RequestEvent{ID: reqID, Method: r.Method, Path: r.URL.Path, Target: "cluster",Status: status, Duration: time.Since(start).Milliseconds(), Error: errText,})@@ -1184,7 +930,7 @@ func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) {"id", reqID, "method", r.Method, "path", r.URL.Path,"remote", r.RemoteAddr, "reason", rejectionError)http.Error(w, rejectionBody, http.StatusBadGateway)- p.codec.Notify("proxy/request", RequestEvent{+ _ = f.notify("proxy/request", RequestEvent{ID: reqID,Method: r.Method,Path: r.URL.Path,@@ -1237,7 +983,7 @@ func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) {wl = &Workload{ID: reqID,Model: model,- Engine: workloadEngine,+ Engine: f.profile.Name,RunID: p.runID,State: "running",ScheduledOn: candidates[0].id,@@ -1372,7 +1118,7 @@ func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) {}if !started {started = true- p.codec.Notify("proxy/request-started", RequestStartedEvent{+ _ = f.notify("proxy/request-started", RequestStartedEvent{ID: reqID,NodeID: cand.id,Method: r.Method,@@ -1386,6 +1132,10 @@ func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) {// the node that actually served. Guarded by wlMu against the// disconnect watcher, and skipped once terminated so a late// re-point can't resurrect a workload we've already failed.+ // Move the capacity claim to the node that actually served,+ // for the same reason: the node that refused the request is+ // not doing the work and must not keep counting as loaded.+ moveHeld(cand.id)if wl != nil {wlMu.Lock()if !terminated && wl.ScheduledOn != cand.id {@@ -1409,7 +1159,7 @@ func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) {// can fail over to another of its published addresses// (multi-homed peer). The in-request failover below moves on to// the next node.- p.targets.Forget(cand.id)+ f.targets.Forget(cand.id)if !last {// Transport/dial error with candidates left: fail over.retry = true@@ -1465,7 +1215,7 @@ func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) {"err", proxyErr,)- p.codec.Notify("proxy/request", RequestEvent{+ _ = f.notify("proxy/request", RequestEvent{ID: reqID,NodeID: servedNodeID,Method: r.Method,@@ -1524,23 +1274,13 @@ func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) {//// Returns an empty slice when no forwarding target is available; the caller// treats that as the rejection path.-func (p *Proxy) resolveCandidates(model string) []candidate {- p.selectedMu.RLock()- id := p.selectedID- p.selectedMu.RUnlock()+func (f *facade) resolveCandidates(model string) []candidate {+ p := f.host+ id := f.SelectedID()priority := p.PriorityList()- p.httpMu.Lock()- selfPort := p.port- var aliasBoundAddresses []string- if p.aliasLn != nil {- aliasBoundAddresses = append(aliasBoundAddresses, p.aliasLn.Addr().String())- }- if p.aliasAltLn != nil {- aliasBoundAddresses = append(aliasBoundAddresses, p.aliasAltLn.Addr().String())- }- p.httpMu.Unlock()+ selfPort, aliasBoundAddresses := f.selfAddresses()// Re-derive membership and pins before resolving so a cluster joined or left,// and a peer paired or removed, since the last request is reflected without a@@ -1548,12 +1288,12 @@ func (p *Proxy) resolveCandidates(model string) []candidate {// freshly-paired one becomes one.p.mesh.Refresh()- nodes := p.discovery.Nodes()+ nodes := f.discovery.Nodes()known := len(nodes)if model != "" {owners := make([]Node, 0, len(nodes))for _, node := range nodes {- if nodeAdvertisesModel(node, model) {+ if nodeAdvertisesModel(f.profile, node, model) {owners = append(owners, node)}}@@ -1582,7 +1322,7 @@ func (p *Proxy) resolveCandidates(model string) []candidate {// targetURL picks a reachable address for a multi-homed node (cached,// TCP-probed), falling back to the first candidate; nil only when the// node advertises no usable address.- u := p.targetURL(n)+ u := f.targetURL(n)if u == nil {return}@@ -1593,7 +1333,7 @@ func (p *Proxy) resolveCandidates(model string) []candidate {// it from the explicit local backend — the loopback engine — rather// than dialing our own mTLS ingress, which would recurse. Ranking// still used this node's real (discovered) model list above.- lb, ok := p.localBackendTarget()+ lb, ok := f.localBackendTarget()if !ok {slog.Debug("resolveCandidates: no local backend for self", "node_id", n.ID)return@@ -1611,7 +1351,7 @@ func (p *Proxy) resolveCandidates(model string) []candidate {// the relayed value can only ever disagree by being stale.u.Scheme = "https"peerUUID = n.ClusterUUID- case p.discovery.IsManual(n.ID):+ case f.discovery.IsManual(n.ID):// An explicit user-added manual node: dialed plain to the address// the user supplied (a deliberate, separately-labeled bypass).default:@@ -1663,23 +1403,48 @@ func (p *Proxy) resolveCandidates(model string) []candidate {return out}+// reservation is one in-flight dispatch this proxy has made since the last+// scheduler snapshot, held so it can be released when the request ends.+//+// The generation is the snapshot the reservation was counted against. A+// snapshot supersedes every reservation taken before it — its pending counts+// already include that work — so a release carrying an older generation is+// dropped. Without that check a long request finishing after a snapshot would+// decrement a counter that had already been reset, pushing a node's estimated+// load below zero and making it look permanently idle.+type reservation struct {+ nodeID string+ generation uint64+ // held distinguishes "no reservation was taken" from "a reservation on the+ // zero-value node", so release on the paths that never reserved is a no-op.+ held bool+}+// reserveCandidate atomically moves the least estimated loaded scheduler-listed-// candidate to the front of this request's failover list. The scheduler's-// pending count and GPU pressure form the authoritative baseline; reservations-// are local dispatches made since that snapshot arrived and have not necessarily-// completed the proxy→broker→scheduler→proxy feedback loop yet.+// candidate to the front of this request's failover list, and returns the+// reservation it took so the caller can release it when the request ends.+//+// The scheduler's pending count and GPU pressure form the authoritative+// baseline; reservations are local dispatches made since that snapshot arrived+// and have not necessarily completed the proxy→broker→scheduler→proxy feedback+// loop yet.//// Model eligibility was enforced before this function receives the list. An// explicit node/select pin bypasses reservations, and unlisted/manual owners// retain their existing fallback position.-func (p *Proxy) reserveCandidate(candidates []candidate) []candidate {+//+// The reservation map is process-wide while the route pin is per facade, so+// this stays a host method and takes the facade whose request it is serving.+// Process-wide is the point: two facades bursting at once are competing for the+// same GPU, so a dispatch through either has to be visible to the other.+func (p *Proxy) reserveCandidate(f *facade, candidates []candidate) ([]candidate, reservation) {if len(candidates) == 0 {- return candidates+ return candidates, reservation{}}- if selectedID := p.SelectedID(); selectedID != "" {+ if selectedID := f.SelectedID(); selectedID != "" {for _, cand := range candidates {if cand.id == selectedID {- return candidates+ return candidates, reservation{}}}}@@ -1692,7 +1457,7 @@ func (p *Proxy) reserveCandidate(candidates []candidate) []candidate {p.priorityMu.Lock()defer p.priorityMu.Unlock()if len(p.priority) == 0 {- return candidates+ return candidates, reservation{}}if p.priorityReservations == nil {p.priorityReservations = make(map[string]int)@@ -1716,7 +1481,7 @@ func (p *Proxy) reserveCandidate(candidates []candidate) []candidate {}}if bestIndex < 0 {- return candidates+ return candidates, reservation{}}chosen := candidates[bestIndex]@@ -1725,16 +1490,84 @@ func (p *Proxy) reserveCandidate(candidates []candidate) []candidate {copy(candidates[1:bestIndex+1], candidates[:bestIndex])candidates[0] = chosen}- return candidates+ return candidates, reservation{+ nodeID: chosen.id,+ generation: p.appliedPriorityGeneration,+ held: true,+ }+}++// releaseReservation drops an in-flight dispatch's reservation once its request+// has ended, so the node stops counting as loaded before the next snapshot+// arrives. Safe to call on a request that never reserved.+//+// A release from before the current snapshot is ignored: that snapshot already+// accounts for the work, and decrementing here would double-count the+// completion and leave the node looking idle.+func (p *Proxy) releaseReservation(r reservation) {+ if !r.held {+ return+ }+ p.priorityMu.Lock()+ defer p.priorityMu.Unlock()+ if r.generation != p.appliedPriorityGeneration {+ return+ }+ if p.priorityReservations[r.nodeID] <= 1 {+ // Deleted rather than left at zero so the map does not accumulate an+ // entry per node ever dispatched to.+ delete(p.priorityReservations, r.nodeID)+ return+ }+ p.priorityReservations[r.nodeID]--+}++// moveReservation transfers a reservation to the node failover actually landed+// on, so the node that refused the request stops carrying its load and the one+// now serving it starts.+//+// It returns the new reservation; the old one must not be released separately.+// A move across a snapshot boundary takes a fresh reservation rather than+// carrying the stale one forward, because the new snapshot has already+// superseded it.+func (p *Proxy) moveReservation(r reservation, nodeID string) reservation {+ if !r.held || r.nodeID == nodeID {+ return r+ }+ p.priorityMu.Lock()+ defer p.priorityMu.Unlock()+ if r.generation == p.appliedPriorityGeneration {+ if p.priorityReservations[r.nodeID] <= 1 {+ delete(p.priorityReservations, r.nodeID)+ } else {+ p.priorityReservations[r.nodeID]--+ }+ }+ if p.priorityReservations == nil {+ p.priorityReservations = make(map[string]int)+ }+ p.priorityReservations[nodeID]+++ return reservation{+ nodeID: nodeID,+ generation: p.appliedPriorityGeneration,+ held: true,+ }}-func nodeAdvertisesModel(n Node, model string) bool {- requested := ollamaModelKey(model)+// nodeAdvertisesModel reports whether a node's advertised inventory contains+// the requested model, under the engine's naming convention. Both sides are+// normalized so the answer agrees with the federated model list's dedupe key;+// if they disagreed the proxy could advertise a model it then refuses to route.+//+// Kept a free function taking the profile so the match is testable without a+// Proxy.+func nodeAdvertisesModel(p engineProfile, n Node, model string) bool {+ requested := p.normalizeModel(model)if requested == "" {return false}for _, available := range n.Models {- if ollamaModelKey(available) == requested {+ if p.normalizeModel(available) == requested {return true}}@@ -1825,27 +1658,6 @@ func nodeURL(n Node) *url.URL {return &url.URL{Scheme: "http", Host: candidates[0]}}-// targetURL picks the forward URL for a node, preferring an address we can-// actually reach. With a single candidate it's just that candidate; with-// several (a multi-homed peer) the shared reach.Chooser returns the confirmed-// last-good address, or the node's own top-ranked one while it confirms in the-// background. The confirmation is transport-neutral: a TCP accept proves the-// address is reachable, while the real pinned mTLS request still authenticates-// which peer answered there.-//-// reach.Prefer, not a blocking confirmation: this runs once per discovered node per-// request, including nodes this request will not be routed to, so a handshake here-// would charge every request for every node's connectivity. An address that is-// wrong is caught by the ErrorHandler below, which forgets it and fails over.-func (p *Proxy) targetURL(n Node) *url.URL {- candidates := nodeCandidates(n)- if len(candidates) == 0 {- return nil- }- host := p.targets.Prefer(n.ID, candidates)- return &url.URL{Scheme: "http", Host: host}-}-// nodeCandidates returns the ordered, de-duplicated host:port targets for a node.//// Order comes from the node itself: netpick.Candidates keeps the node's published@@ -1936,77 +1748,11 @@ func startLocalAddrWatch(ctx context.Context) {}()}-func (p *Proxy) SelectedID() string {- p.selectedMu.RLock()- defer p.selectedMu.RUnlock()- return p.selectedID-}--func (p *Proxy) SetSelected(id string) {- p.selectedMu.Lock()- p.selectedID = id- p.selectedMu.Unlock()-}--// replaceSubscribed replaces the proxy's relay-fed routing overlay from a-// discovery:nodes snapshot: it projects every node advertising ol with a dialable-// IP into the overlay (dropping the rest) and clears a user selection pinned to a-// node that's no longer routable. The broker sends the full filtered set on every-// change, so this is a wholesale replace, not a per-node apply — a departed node-// is simply absent from the next snapshot.-func (p *Proxy) replaceSubscribed(params json.RawMessage) {- var res noderec.GetNodesResult- if err := json.Unmarshal(params, &res); err != nil {- slog.Warn("invalid discovery:nodes snapshot", "err", err)- return- }- nodes := make([]Node, 0, len(res.Nodes))- present := make(map[string]bool, len(res.Nodes))- for _, dn := range res.Nodes {- n, ok := subscribedToNode(dn)- if !ok {- continue- }- nodes = append(nodes, n)- present[n.ID] = true- }- discovered, updated, removed := p.discovery.SetSubscribed(nodes)- // Surface the relay-fed set to the client as node/* events — the signal a- // consumer (the UI) uses to show which peers run this engine — mirroring how- // manual nodes are announced. Without this the routing overlay updates- // silently and peers appear engine-less. A node dropping out is also the- // proxy's "this upstream is gone" signal, surfaced through the errors- // pipeline (the broker forwards these to nvpair-errors); a re-appearance clears- // it. NodeID/Timestamp are left unset so the broker stamps the authoritative- // values.- for _, n := range discovered {- p.codec.Notify("node/discovered", n.withPrimaryIP())- if err := p.codec.Notify("errors:clear", errors.ClearParams{ID: upstreamUnreachableID(n.ID)}); err != nil {- slog.Debug("failed to send errors:clear", "node", n.ID, "err", err)- }- }- for _, n := range updated {- p.codec.Notify("node/updated", n.withPrimaryIP())- }- for _, n := range removed {- p.codec.Notify("node/removed", n.withPrimaryIP())- if err := p.codec.Notify("errors:report", errors.ServiceError{- ID: upstreamUnreachableID(n.ID),- Message: fmt.Sprintf("Upstream node %q is no longer reachable (dropped from discovery)", n.Host),- Severity: "warning",- Action: "none",- }); err != nil {- slog.Debug("failed to send errors:report", "node", n.ID, "err", err)- }- }- p.clearSelectionIfNotPresent(present)-}-// upstreamUnreachableID is the canonical ServiceError id for an upstream the// proxy no longer sees in discovery. Kept as a single function so the report and// clear can't drift (nvpair-errors matches by literal id).-func upstreamUnreachableID(nodeID string) string {- return "ollama-proxy:upstream-unreachable:" + nodeID+func upstreamUnreachableID(p engineProfile, nodeID string) string {+ return p.ComponentName() + ":upstream-unreachable:" + nodeID}// PriorityList returns a copy of the current scheduler-supplied priority order.@@ -2020,14 +1766,49 @@ func (p *Proxy) PriorityList() []string {// the number of ids stored. The list is kept verbatim — unknown ids are retained// (a node may appear in discovery later) and only consulted at request time. An// empty list clears the scheduler's influence.+//+// It stamps its own generation rather than sending zero. An unversioned+// snapshot would clear the reservations without advancing the epoch, so a+// reservation taken before it would still match and release one taken after —+// the undercount the stamp exists to prevent.+//+// The stamp and the apply are one critical section. Reading the epoch, dropping+// the lock, and then applying would let two callers mint the same generation+// and have the second silently ignored by the guard.func (p *Proxy) SetPriority(nodes []string) int {- return p.SetPrioritySnapshot(schedulerwire.Priority{Nodes: nodes})+ p.priorityMu.Lock()+ defer p.priorityMu.Unlock()+ return p.setPriorityLocked(schedulerwire.Priority{+ Generation: p.appliedPriorityGeneration + 1,+ Nodes: nodes,+ })}// SetPrioritySnapshot replaces the scheduler baseline and clears optimistic// reservations made against the previous snapshot. Nodes-only callers remain// valid: a missing rank supplies zero pending and GPU-pressure baselines.+//+// Applying is idempotent per generation, and that is load-bearing rather than+// tidy. Clearing the reservations is the whole point of applying a snapshot —+// the new pending counts already account for those dispatches — so replaying a+// generation the proxy has already applied would discard reservations taken+// since, which are exactly the dispatches the scheduler cannot see yet. The+// broker can and does replay: a node/set-priority that times out may already+// have been applied.+//+// A generation is mandatory: zero is rejected at the RPC boundary and ignored+// here, because a snapshot that cleared the reservations without advancing the+// epoch would let one taken before it release one taken after.func (p *Proxy) SetPrioritySnapshot(priority schedulerwire.Priority) int {+ p.priorityMu.Lock()+ defer p.priorityMu.Unlock()+ return p.setPriorityLocked(priority)+}++// setPriorityLocked is the body of SetPrioritySnapshot. Split out so a caller+// that has to mint a generation can do so in the same critical section that+// applies it. Caller holds priorityMu.+func (p *Proxy) setPriorityLocked(priority schedulerwire.Priority) int {cleaned := append([]string(nil), priority.Nodes...)pending := make(map[string]int, len(priority.Ranks))gpuPressure := make(map[string]int, len(priority.Ranks))@@ -2047,43 +1828,32 @@ func (p *Proxy) SetPrioritySnapshot(priority schedulerwire.Priority) int {gpuPressure[rank.ID] = rank.GPUPressure}- p.priorityMu.Lock()+ // Unconditional, with no escape hatch for an unversioned snapshot. One used+ // to apply while leaving the epoch untouched, which reintroduced exactly+ // the undercount the stamp prevents: clear the map at epoch N, reserve+ // again, and a reservation from before the clear still matches N and+ // releases the new one. A generation of 0 now fails this comparison and is+ // ignored; both callers stamp, so it cannot arrive.+ if priority.Generation <= p.appliedPriorityGeneration {+ // Already applied, or superseded. Leave the baseline and the+ // reservations exactly as they are.+ return len(p.priority)+ }+ p.appliedPriorityGeneration = priority.Generationp.priority = cleanedp.priorityPending = pendingp.priorityGPUPressure = gpuPressurep.priorityReservations = make(map[string]int)- p.priorityMu.Unlock()return len(cleaned)}-// clearSelectionIfNotPresent resets the user-selected node (and notifies the-// client) when it's neither in the relay-fed set nor a manual node, so a stale-// selection can't pin routing to a departed target.-func (p *Proxy) clearSelectionIfNotPresent(present map[string]bool) {- p.selectedMu.Lock()- sel := p.selectedID- p.selectedMu.Unlock()- if sel == "" || present[sel] || p.discovery.IsManual(sel) {- return- }- p.selectedMu.Lock()- cleared := p.selectedID == sel- if cleared {- p.selectedID = ""- }- p.selectedMu.Unlock()- if cleared {- p.codec.Notify("node/selection-changed", SelectedResult{ID: ""})- }-}-// subscribedToNode projects a relay DirectoryNode onto the proxy's routable Node// for the ol service, returning false when the node doesn't advertise ol or has// no dialable address. The engine port comes from the ol service key (the real// Ollama port the broker's engine poller registered, not the proxy's listen// port).-func subscribedToNode(n noderec.DirectoryNode) (Node, bool) {- svc, ok := n.Services[noderec.ServiceOllama]+func subscribedToNode(p engineProfile, n noderec.DirectoryNode) (Node, bool) {+ svc, ok := n.Services[p.DiscoveryService]if !ok || n.IP == "" {return Node{}, false}@@ -2109,7 +1879,7 @@ func subscribedToNode(n noderec.DirectoryNode) (Node, bool) {// a model that a dual-engine node serves solely via LM Studio isn't// accepted as an Ollama owner here (falls back to the union for a peer// that sends no attribution — see DirectoryNode.EngineModels).- Models: append([]string(nil), n.EngineModels("ollama")...),+ Models: append([]string(nil), n.EngineModels(p.Name)...),}, true}@@ -2127,7 +1897,38 @@ func (p *Proxy) readLoop(ctx context.Context) error {}}+// handleMessage dispatches one control-plane message.+//+// A panic here is contained rather than fatal. It used to be reasonable to let+// one crash the process: the process was one engine, so it took down only the+// engine whose message was being handled. Now it hosts every facade, so an+// unhandled panic while parsing one engine's request would drop every other+// engine's listener and the inference streaming through it.+//+// Contained, not swallowed: the panic is logged at error level with its stack+// and the method that caused it, and the caller is answered with an internal+// error rather than left waiting.+//+// Every critical section reachable from here releases its lock by defer, which+// unwinding runs. That is a precondition, not a nicety: a mutex stranded by a+// recovered panic would trade a contained crash for a deadlock across every+// facade — strictly worse than the crash it replaced. resolveCandidates takes+// httpMu on every request and stopServing takes it during teardown, so the+// facade bring-up path in facade.go is held to the same rule.func (p *Proxy) handleMessage(msg *Message) {+ defer func() {+ recovered := recover()+ if recovered == nil {+ return+ }+ slog.Error("recovered from a panic while handling a proxy request",+ "method", msg.Method, "panic", fmt.Sprint(recovered),+ "stack", string(debug.Stack()))+ if msg.IsRequest() {+ p.codec.RespondError(msg.ID, -32603, "internal error handling "+msg.Method)+ }+ }()+if msg.Method == applog.SetLevelMethod {resolved, err := applog.HandleSetLevelParams(msg.Params)if msg.IsRequest() {@@ -2145,9 +1946,19 @@ func (p *Proxy) handleMessage(msg *Message) {return}- switch msg.Method {+ // Facade-scoped traffic arrives addressed to its engine, because the broker+ // cannot know which facade a bare method is for. Split once here so every+ // case below dispatches on the bare method.+ engine, method := engines.SplitAddressedMethod(msg.Method)++ switch method {case noderec.NotifyNodes:- p.replaceSubscribed(msg.Params)+ // Dropped rather than queued when the addressed facade is not enabled:+ // the relay re-sends the full set on every change, and enabling+ // subscribes, so the next snapshot supersedes anything missed here.+ if f := p.facadeFor(engine); f != nil {+ f.replaceSubscribed(msg.Params)+ }return}@@ -2158,14 +1969,47 @@ func (p *Proxy) handleMessage(msg *Message) {return}- switch msg.Method {+ switch method {+ case "facade/enable":+ var params enableFacadeParams+ if err := json.Unmarshal(msg.Params, ¶ms); err != nil {+ p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"engine\",\"port\",\"aliasAddresses\",\"ignorePersistedPort\"}")+ return+ }+ result, err := p.enableFacade(params)+ if err != nil {+ // A bind failure is a normal outcome the broker acts on, not a+ // process-fatal one: it picks another port and asks again. Nothing+ // else on this process is disturbed, which is the whole reason+ // bring-up moved off argv.+ code := -32000+ if stderrors.Is(err, errFacadeBindFailed) {+ code = codeFacadeBindFailed+ }+ slog.Warn("facade/enable failed", "engine", params.Engine, "port", params.Port, "err", err)+ p.codec.RespondError(msg.ID, code, err.Error())+ return+ }+ slog.Info("facade enabled", "engine", result.Engine, "port", result.Port)+ if err := p.codec.Respond(msg.ID, result); err != nil {+ log.Printf("failed to respond to facade/enable: %v", err)+ }+case "nodes/list":- nodes := p.discovery.Nodes()+ f, ok := p.requireFacade(msg, engine)+ if !ok {+ return+ }+ nodes := f.discovery.Nodes()if err := p.codec.Respond(msg.ID, NodesResult{Nodes: nodes}); err != nil {log.Printf("failed to respond to nodes/list: %v", err)}case "node/select":+ f, ok := p.requireFacade(msg, engine)+ if !ok {+ return+ }var params SelectParamsif err := json.Unmarshal(msg.Params, ¶ms); err != nil {p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"id\": \"...\"}")@@ -2173,7 +2017,7 @@ func (p *Proxy) handleMessage(msg *Message) {}if params.ID != "" {found := false- for _, n := range p.discovery.Nodes() {+ for _, n := range f.discovery.Nodes() {if n.ID == params.ID {found = truebreak@@ -2184,22 +2028,35 @@ func (p *Proxy) handleMessage(msg *Message) {return}}- p.SetSelected(params.ID)+ f.SetSelected(params.ID)log.Printf("node selection changed to %q", params.ID)if err := p.codec.Respond(msg.ID, SelectedResult{ID: params.ID}); err != nil {log.Printf("failed to respond to node/select: %v", err)}- p.codec.Notify("node/selection-changed", SelectedResult{ID: params.ID})+ _ = f.notify("node/selection-changed", SelectedResult{ID: params.ID})case "node/selected":- if err := p.codec.Respond(msg.ID, SelectedResult{ID: p.SelectedID()}); err != nil {+ f, ok := p.requireFacade(msg, engine)+ if !ok {+ return+ }+ if err := p.codec.Respond(msg.ID, SelectedResult{ID: f.SelectedID()}); err != nil {log.Printf("failed to respond to node/selected: %v", err)}case "node/set-priority":var params schedulerwire.Priorityif err := json.Unmarshal(msg.Params, ¶ms); err != nil {- p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"nodes\": [\"id\", ...], \"ranks\": [...]}")+ p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"generation\": <uint>, \"nodes\": [\"id\", ...], \"ranks\": [...]}")+ return+ }+ // A generation is required, not optional. This method is relayed+ // verbatim from any client, and an unversioned snapshot would clear the+ // reservations without advancing the epoch — letting a reservation+ // taken before it release one taken after, which reads as an idle node+ // and attracts every later dispatch.+ if params.Generation == 0 {+ p.codec.RespondError(msg.ID, -32602, "generation is required and must be greater than 0")return}count := p.SetPrioritySnapshot(params)@@ -2209,6 +2066,10 @@ func (p *Proxy) handleMessage(msg *Message) {}case "set-port":+ f, ok := p.requireFacade(msg, engine)+ if !ok {+ return+ }var params struct {Port int `json:"port"`}@@ -2220,7 +2081,7 @@ func (p *Proxy) handleMessage(msg *Message) {p.codec.RespondError(msg.ID, -32602, "port must be between 1 and 65535")return}- if err := p.setPort(params.Port); err != nil {+ if err := f.setPort(params.Port); err != nil {p.codec.RespondError(msg.ID, -32000, err.Error())return}@@ -2229,6 +2090,10 @@ func (p *Proxy) handleMessage(msg *Message) {}case "node/add-manual":+ f, ok := p.requireFacade(msg, engine)+ if !ok {+ return+ }var node Nodeif err := json.Unmarshal(msg.Params, &node); err != nil {p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"id\",\"host\",\"port\",\"addresses\"}")@@ -2238,49 +2103,66 @@ func (p *Proxy) handleMessage(msg *Message) {p.codec.RespondError(msg.ID, -32602, "id, port, and at least one address are required")return}- added := p.discovery.AddManual(node)+ added := f.discovery.AddManual(node)if err := p.codec.Respond(msg.ID, map[string]bool{"added": added}); err != nil {log.Printf("failed to respond to node/add-manual: %v", err)}if added {log.Printf("manual node added: %s (%s:%d)", node.ID, node.Addresses[0], node.Port)- p.codec.Notify("node/discovered", node.withPrimaryIP())+ _ = f.notify("node/discovered", node.withPrimaryIP())} else {log.Printf("manual node updated: %s (%s:%d)", node.ID, node.Addresses[0], node.Port)- p.codec.Notify("node/updated", node.withPrimaryIP())+ _ = f.notify("node/updated", node.withPrimaryIP())}case "node/remove-manual":+ f, ok := p.requireFacade(msg, engine)+ if !ok {+ return+ }var params SelectParamsif err := json.Unmarshal(msg.Params, ¶ms); err != nil {p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"id\": \"...\"}")return}- removed := p.discovery.RemoveManual(params.ID)+ removed := f.discovery.RemoveManual(params.ID)if err := p.codec.Respond(msg.ID, map[string]bool{"removed": removed}); err != nil {log.Printf("failed to respond to node/remove-manual: %v", err)}if removed {log.Printf("manual node removed: %s", params.ID)- p.selectedMu.Lock()- if p.selectedID == params.ID {- p.selectedID = ""- p.selectedMu.Unlock()- p.codec.Notify("node/selection-changed", SelectedResult{ID: ""})- } else {- p.selectedMu.Unlock()+ if f.clearSelectionOf(params.ID) {+ _ = f.notify("node/selection-changed", SelectedResult{ID: ""})}- p.codec.Notify("node/removed", Node{ID: params.ID})+ _ = f.notify("node/removed", Node{ID: params.ID})}case "node/set-local-backend":+ f, ok := p.requireFacade(msg, engine)+ if !ok {+ return+ }var b localBackendif err := json.Unmarshal(msg.Params, &b); err != nil {p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"engine\",\"host\",\"port\",\"healthy\"}")return}- p.setLocalBackend(b)- slog.Info("local backend updated", "engine", b.Engine, "host", b.Host, "port", b.Port, "healthy", b.Healthy)+ // The address decides which facade this applies to, so a payload naming+ // a different engine is a caller bug, not a preference. Accepting it+ // would point one engine's ingress and self-candidate at the other+ // engine's port — and the old log line, which echoed the payload,+ // would have named the wrong engine and hidden the cross-wire.+ if b.Engine != "" && b.Engine != engine {+ p.codec.RespondError(msg.ID, -32602, fmt.Sprintf(+ "params name engine %q but the request is addressed to %q", b.Engine, engine))+ return+ }+ if err := f.setLocalBackend(b); err != nil {+ p.codec.RespondError(msg.ID, -32602, err.Error())+ return+ }+ slog.Info("local backend updated",+ "engine", f.profile.Name, "host", b.Host, "port", b.Port, "healthy", b.Healthy)if err := p.codec.Respond(msg.ID, map[string]bool{"ok": true}); err != nil {log.Printf("failed to respond to node/set-local-backend: %v", err)}diff --git a/services/ollama-proxy/proxy_test.go b/services/nvpair-proxy/proxy_test.gosimilarity index 100%rename from services/ollama-proxy/proxy_test.gorename to services/nvpair-proxy/proxy_test.godiff --git a/services/nvpair-proxy/reservation_test.go b/services/nvpair-proxy/reservation_test.gonew file mode 100644index 0000000..3a844c7--- /dev/null+++ b/services/nvpair-proxy/reservation_test.go@@ -0,0 +1,442 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++import (+ "sync"+ "testing"++ "nvpair-shared/schedulerwire"+)++// applySnapshot delivers a ranking the way the broker does: with a fresh+// generation, always greater than the last applied.+//+// Generation 0 is rejected rather than defaulted, because an unversioned+// snapshot that cleared reservations without advancing the epoch would let a+// reservation taken before it release one taken after. Tests that care about a+// specific generation set it themselves and this leaves it alone.+func applySnapshot(p *Proxy, priority schedulerwire.Priority) int {+ if priority.Generation == 0 {+ p.priorityMu.RLock()+ priority.Generation = p.appliedPriorityGeneration + 1+ p.priorityMu.RUnlock()+ }+ return p.SetPrioritySnapshot(priority)+}++func reservationCandidates(ids ...string) []candidate {+ out := make([]candidate, 0, len(ids))+ for _, id := range ids {+ out = append(out, candidate{id: id})+ }+ return out+}++func reservedID(p *Proxy, candidates []candidate) string {+ candidates = append([]candidate(nil), candidates...)+ ordered, _ := p.reserveCandidate(p.soleFacade(), candidates)+ return ordered[0].id+}++func TestReserveCandidate_ConcurrentEqualLoadHasAtMostOneSkew(t *testing.T) {+ p := prProxy(t)+ ids := []string{"a", "b", "c", "d"}+ ranks := make([]schedulerwire.NodeRank, 0, len(ids))+ for i, id := range ids {+ ranks = append(ranks, schedulerwire.NodeRank{ID: id, Rank: i})+ }+ applySnapshot(p, schedulerwire.Priority{Nodes: ids, Ranks: ranks})+ candidates := reservationCandidates(ids...)++ const requests = 100+ chosen := make(chan string, requests)+ var wg sync.WaitGroup+ for range requests {+ wg.Add(1)+ go func() {+ defer wg.Done()+ chosen <- reservedID(p, candidates)+ }()+ }+ wg.Wait()+ close(chosen)++ counts := make(map[string]int, len(ids))+ for id := range chosen {+ counts[id]+++ }+ min, max := requests, 0+ for _, id := range ids {+ if counts[id] < min {+ min = counts[id]+ }+ if counts[id] > max {+ max = counts[id]+ }+ }+ if max-min > 1 {+ t.Fatalf("100 equal-load reservations are imbalanced: %v", counts)+ }+}++func TestReserveCandidate_ConvergesUnequalPendingDepths(t *testing.T) {+ p := prProxy(t)+ applySnapshot(p, schedulerwire.Priority{+ Nodes: []string{"a", "b", "c"},+ Ranks: []schedulerwire.NodeRank{+ {ID: "a", Pending: 0, Rank: 0},+ {ID: "b", Pending: 2, Rank: 1},+ {ID: "c", Pending: 4, Rank: 2},+ },+ })+ candidates := reservationCandidates("a", "b", "c")+ assigned := map[string]int{}+ for range 6 {+ assigned[reservedID(p, candidates)]+++ }++ total := map[string]int{+ "a": assigned["a"],+ "b": 2 + assigned["b"],+ "c": 4 + assigned["c"],+ }+ if total["a"] != 4 || total["b"] != 4 || total["c"] != 4 {+ t.Fatalf("unequal depths did not converge: assigned=%v total=%v", assigned, total)+ }+}++func TestReserveCandidate_CombinesPendingPressureAndReservations(t *testing.T) {+ p := prProxy(t)+ applySnapshot(p, schedulerwire.Priority{+ Nodes: []string{"a", "b", "c"},+ Ranks: []schedulerwire.NodeRank{+ {ID: "a", Pending: 0, GPUPressure: 3},+ {ID: "b", Pending: 1, GPUPressure: 0},+ {ID: "c", Pending: 0, GPUPressure: 2},+ },+ })+ candidates := reservationCandidates("a", "b", "c")+ got := []string{+ reservedID(p, candidates),+ reservedID(p, candidates),+ reservedID(p, candidates),+ }+ want := []string{"b", "b", "c"}+ for i := range want {+ if got[i] != want[i] {+ t.Fatalf("GPU-aware reservations = %v, want %v", got, want)+ }+ }+}++func TestSetPrioritySnapshotClampsGPUPressure(t *testing.T) {+ p := prProxy(t)+ applySnapshot(p, schedulerwire.Priority{+ Nodes: []string{"low", "high"},+ Ranks: []schedulerwire.NodeRank{+ {ID: "low", GPUPressure: -1},+ {ID: "high", GPUPressure: schedulerwire.MaxGPUPressure + 1},+ },+ })+ p.priorityMu.RLock()+ low := p.priorityGPUPressure["low"]+ high := p.priorityGPUPressure["high"]+ p.priorityMu.RUnlock()+ if low != 0 || high != schedulerwire.MaxGPUPressure {+ t.Fatalf("clamped GPU pressure = low:%d high:%d", low, high)+ }+}++// A node/set-priority call can time out after the proxy already applied it, so+// the broker may redeliver a generation. Applying a snapshot clears the+// optimistic reservations — that is its purpose, since the new pending counts+// already include those dispatches — so a redelivery must not clear them a+// second time, or it discards precisely the dispatches the scheduler has not+// seen yet and two concurrent bursts converge on one node.+func TestSetPrioritySnapshot_RedeliveredGenerationKeepsReservations(t *testing.T) {+ p := prProxy(t)+ snapshot := schedulerwire.Priority{+ Generation: 7,+ Nodes: []string{"a", "b"},+ Ranks: []schedulerwire.NodeRank{+ {ID: "a", Pending: 0},+ {ID: "b", Pending: 0},+ },+ }+ p.SetPrioritySnapshot(snapshot)++ candidates := reservationCandidates("a", "b")+ first := reservedID(p, candidates)++ // The same generation arriving again is an acknowledged no-op.+ p.SetPrioritySnapshot(snapshot)+ p.priorityMu.RLock()+ kept := p.priorityReservations[first]+ p.priorityMu.RUnlock()+ if kept != 1 {+ t.Fatalf("reservation on %q after redelivery = %d, want 1 (redelivery cleared it)", first, kept)+ }++ // An older generation is stale and must not roll the baseline back either.+ applySnapshot(p, schedulerwire.Priority{Generation: 6, Nodes: []string{"only-stale"}})+ p.priorityMu.RLock()+ order := append([]string(nil), p.priority...)+ p.priorityMu.RUnlock()+ if len(order) != 2 || order[0] != "a" {+ t.Fatalf("priority after a stale generation = %v, want the generation-7 order", order)+ }++ // A newer generation applies and clears, which is the behavior the+ // idempotency must not have broken.+ applySnapshot(p, schedulerwire.Priority{+ Generation: 8,+ Nodes: []string{"a", "b"},+ Ranks: []schedulerwire.NodeRank{{ID: "a", Pending: 0}, {ID: "b", Pending: 0}},+ })+ p.priorityMu.RLock()+ cleared := len(p.priorityReservations)+ p.priorityMu.RUnlock()+ if cleared != 0 {+ t.Fatalf("reservations after a newer generation = %d, want 0", cleared)+ }+}++func TestReserveCandidate_LegacyNodesOnlyUsesZeroBaseline(t *testing.T) {+ p := prProxy(t)+ p.SetPriority([]string{"a", "b", "c"})+ candidates := reservationCandidates("a", "b", "c")+ counts := map[string]int{}+ for range 5 {+ counts[reservedID(p, candidates)]+++ }+ want := map[string]int{"a": 2, "b": 2, "c": 1}+ for id, n := range want {+ if counts[id] != n {+ t.Fatalf("legacy nodes-only assignments = %v, want %v", counts, want)+ }+ }+}++func TestReserveCandidate_UsesEligibleCandidates(t *testing.T) {+ p := prProxy(t)+ applySnapshot(p, schedulerwire.Priority{+ Nodes: []string{"missing", "owner-a", "owner-b", "unknown"},+ Ranks: []schedulerwire.NodeRank{+ {ID: "missing", Pending: 0},+ {ID: "owner-a", Pending: 4},+ {ID: "owner-b", Pending: 5},+ {ID: "unknown", Pending: 0},+ },+ })+ candidates := reservationCandidates("owner-a", "owner-b")++ for range 8 {+ got := reservedID(p, candidates)+ if got != "owner-a" && got != "owner-b" {+ t.Fatalf("reservation escaped eligible candidates to %q", got)+ }+ }+}++func TestReserveCandidate_ManualPinBypassesReservations(t *testing.T) {+ p := prProxy(t)+ applySnapshot(p, schedulerwire.Priority{+ Nodes: []string{"a", "b"},+ Ranks: []schedulerwire.NodeRank{{ID: "a"}, {ID: "b"}},+ })+ p.soleFacade().SetSelected("b")+ candidates := reservationCandidates("b", "a") // resolveCandidates puts the pin first+ if got := reservedID(p, candidates); got != "b" {+ t.Fatalf("manual pin resolved to %q, want b", got)+ }+ p.priorityMu.RLock()+ defer p.priorityMu.RUnlock()+ if len(p.priorityReservations) != 0 {+ t.Fatalf("manual pin created optimistic reservations: %v", p.priorityReservations)+ }+}++func TestReserveCandidate_IneligibleManualPinDoesNotBypassReservations(t *testing.T) {+ p := prProxy(t)+ applySnapshot(p, schedulerwire.Priority{+ Nodes: []string{"owner-b", "owner-a"},+ Ranks: []schedulerwire.NodeRank{{ID: "owner-b"}, {ID: "owner-a"}},+ })+ p.soleFacade().SetSelected("missing")+ if got := reservedID(p, reservationCandidates("owner-a", "owner-b")); got != "owner-b" {+ t.Fatalf("reservation with ineligible pin = %q, want owner-b", got)+ }+}++func TestReserveCandidate_PreservesFailoverAndSnapshotReset(t *testing.T) {+ p := prProxy(t)+ applySnapshot(p, schedulerwire.Priority{+ Nodes: []string{"b", "a", "c"},+ Ranks: []schedulerwire.NodeRank{+ {ID: "b", Pending: 0},+ {ID: "a", Pending: 5},+ {ID: "c", Pending: 6},+ },+ })+ got, _ := p.reserveCandidate(p.soleFacade(), reservationCandidates("a", "b", "c"))+ want := []string{"b", "a", "c"}+ for i, id := range want {+ if got[i].id != id {+ t.Fatalf("reserved failover order = %v, want %v", candidateIDsFrom(got), want)+ }+ }++ applySnapshot(p, schedulerwire.Priority{+ Nodes: []string{"a", "b", "c"},+ Ranks: []schedulerwire.NodeRank{{ID: "a"}, {ID: "b"}, {ID: "c"}},+ })+ if next := reservedID(p, reservationCandidates("a", "b", "c")); next != "a" {+ t.Fatalf("new snapshot did not reset reservations: next = %q, want a", next)+ }+}++func candidateIDsFrom(candidates []candidate) []string {+ out := make([]string, 0, len(candidates))+ for _, candidate := range candidates {+ out = append(out, candidate.id)+ }+ return out+}++// flatPriority makes every listed node equally loaded, so reservations are the+// only thing that can break the tie and dispatch is observable.+func flatPriority(p *Proxy, generation uint64, ids ...string) {+ ranks := make([]schedulerwire.NodeRank, 0, len(ids))+ for _, id := range ids {+ ranks = append(ranks, schedulerwire.NodeRank{ID: id})+ }+ applySnapshot(p, schedulerwire.Priority{+ Generation: generation,+ Nodes: ids,+ Ranks: ranks,+ })+}++func reservationCount(p *Proxy, id string) int {+ p.priorityMu.RLock()+ defer p.priorityMu.RUnlock()+ return p.priorityReservations[id]+}++// A reservation is a claim on capacity for one in-flight request. Releasing it+// when the request ends is what keeps a node from looking loaded until the next+// snapshot: without it a steady trickle of short requests drives dispatch away+// from a node that is actually idle.+func TestReleasedReservationStopsCountingAsLoad(t *testing.T) {+ p := prProxy(t)+ flatPriority(p, 1, "a", "b")++ _, held := p.reserveCandidate(p.soleFacade(), reservationCandidates("a", "b"))+ if !held.held {+ t.Fatal("no reservation was taken on a flat snapshot")+ }+ if got := reservationCount(p, held.nodeID); got != 1 {+ t.Fatalf("reservation count for %q = %d, want 1", held.nodeID, got)+ }++ p.releaseReservation(held)+ if got := reservationCount(p, held.nodeID); got != 0 {+ t.Fatalf("reservation count for %q after release = %d, want 0", held.nodeID, got)+ }+ // The entry is deleted rather than left at zero, so the map cannot grow one+ // key per node ever dispatched to.+ p.priorityMu.RLock()+ _, present := p.priorityReservations[held.nodeID]+ p.priorityMu.RUnlock()+ if present {+ t.Errorf("released reservation left a zero entry for %q", held.nodeID)+ }+}++// A snapshot supersedes every reservation taken before it, because its pending+// counts already include that work. A late release must therefore be dropped:+// applying it would double-count the completion and leave the node reading as+// permanently idle.+func TestReleaseFromBeforeASnapshotIsIgnored(t *testing.T) {+ p := prProxy(t)+ flatPriority(p, 1, "a", "b")++ _, stale := p.reserveCandidate(p.soleFacade(), reservationCandidates("a", "b"))+ if !stale.held {+ t.Fatal("no reservation was taken")+ }++ // A new snapshot arrives, resetting reservations, and another request+ // reserves against it.+ flatPriority(p, 2, "a", "b")+ _, fresh := p.reserveCandidate(p.soleFacade(), reservationCandidates("a", "b"))+ freshCount := reservationCount(p, fresh.nodeID)++ // The first request now finishes.+ p.releaseReservation(stale)++ if got := reservationCount(p, fresh.nodeID); got != freshCount {+ t.Fatalf("a release from generation %d changed the generation-%d count for %q: %d, want %d",+ stale.generation, fresh.generation, fresh.nodeID, got, freshCount)+ }+ for _, id := range []string{"a", "b"} {+ if got := reservationCount(p, id); got < 0 {+ t.Fatalf("reservation count for %q went negative: %d", id, got)+ }+ }+}++// Failover moves the claim to the node that actually served. The node that+// refused the request is not doing the work, so leaving its reservation in+// place would steer later dispatch away from it for no reason.+func TestFailoverMovesTheReservationToTheServingNode(t *testing.T) {+ p := prProxy(t)+ flatPriority(p, 1, "a", "b")++ _, held := p.reserveCandidate(p.soleFacade(), reservationCandidates("a", "b"))+ from := held.nodeID+ to := "a"+ if from == to {+ to = "b"+ }++ moved := p.moveReservation(held, to)+ if moved.nodeID != to {+ t.Fatalf("moved reservation node = %q, want %q", moved.nodeID, to)+ }+ if got := reservationCount(p, from); got != 0 {+ t.Errorf("refusing node %q still holds %d reservations", from, got)+ }+ if got := reservationCount(p, to); got != 1 {+ t.Errorf("serving node %q holds %d reservations, want 1", to, got)+ }++ // Releasing the moved token clears the serving node, not the original.+ p.releaseReservation(moved)+ if got := reservationCount(p, to); got != 0 {+ t.Errorf("serving node %q still holds %d reservations after release", to, got)+ }+}++// Two facades in one process compete for the same GPU, so a dispatch through+// either has to be visible to the other. That is the whole reason the map is+// process-wide rather than per facade.+func TestReservationsAreSharedAcrossFacades(t *testing.T) {+ p := prProxy(t)+ flatPriority(p, 1, "a", "b")++ // Same host, so this stands in for a second facade's request.+ _, first := p.reserveCandidate(p.soleFacade(), reservationCandidates("a", "b"))+ _, second := p.reserveCandidate(p.soleFacade(), reservationCandidates("a", "b"))++ if first.nodeID == second.nodeID {+ t.Fatalf("both dispatches chose %q; the second did not see the first's reservation", first.nodeID)+ }+ for _, r := range []reservation{first, second} {+ if got := reservationCount(p, r.nodeID); got != 1 {+ t.Errorf("node %q holds %d reservations, want 1", r.nodeID, got)+ }+ }+}diff --git a/services/lmstudio-proxy/server_timeout_test.go b/services/nvpair-proxy/server_timeout_test.gosimilarity index 74%rename from services/lmstudio-proxy/server_timeout_test.gorename to services/nvpair-proxy/server_timeout_test.goindex 39d9b88..bb42a7c 100644--- a/services/lmstudio-proxy/server_timeout_test.go+++ b/services/nvpair-proxy/server_timeout_test.go@@ -12,23 +12,23 @@ import ()func TestHTTPServersConfigureIdleTimeouts(t *testing.T) {- p := testProxy(NewDiscovery(), 1235)+ p := testProxy(anyProfile(t), NewDiscovery(), 11435)p.mesh = clustertrust.Open(t.TempDir())ln, err := net.Listen("tcp", "127.0.0.1:0")if err != nil {t.Fatalf("listen: %v", err)}- p.serveHTTP(context.Background(), ln)+ p.soleFacade().serveHTTP(context.Background(), ln)defer p.shutdown(context.Background())- if p.plainSrv == nil || p.tlsSrv == nil {+ if p.soleFacade().plainSrv == nil || p.soleFacade().tlsSrv == nil {t.Fatal("servers not recorded")}for name, srv := range map[string]struct {readHeader, idle interface{}}{- "plain": {p.plainSrv.ReadHeaderTimeout, p.plainSrv.IdleTimeout},- "tls": {p.tlsSrv.ReadHeaderTimeout, p.tlsSrv.IdleTimeout},+ "plain": {p.soleFacade().plainSrv.ReadHeaderTimeout, p.soleFacade().plainSrv.IdleTimeout},+ "tls": {p.soleFacade().tlsSrv.ReadHeaderTimeout, p.soleFacade().tlsSrv.IdleTimeout},} {if srv.readHeader != proxyReadHeaderTimeout {t.Errorf("%s ReadHeaderTimeout = %v, want %v", name, srv.readHeader, proxyReadHeaderTimeout)diff --git a/services/nvpair-proxy/spec.md b/services/nvpair-proxy/spec.mdnew file mode 100644index 0000000..69d486f--- /dev/null+++ b/services/nvpair-proxy/spec.md@@ -0,0 +1,210 @@+<!--+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+SPDX-License-Identifier: Apache-2.0+-->++# Microservice: Engine Proxy (`nvpair-proxy`)++> **Status: implemented.** House spec style (`nvpair-cluster-manager`,+> `nvpair-workload-manager`, `nvpair-engine-manager`, `nvpair-job-scheduler`).+> Where this file and `README.md` disagree, this file wins.++## 1. Purpose++Carries inference **to** the node that should serve it. One process hosts a+**facade** per enabled engine: each facade is an HTTP reverse proxy speaking its+engine's own dialect, filtering to nodes that advertise the requested model,+choosing among them, and failing over when one refuses.++It is a data plane, not a policy component. Ranking belongs to+`nvpair-job-scheduler`, membership and trust to `nvpair-cluster-manager`,+lifecycle to `nvpair-engine-manager`. What this process owns is the choice of+*which advertised owner receives this request*, and the local accounting that+makes that choice good under bursts.++## 2. Scope++**In scope**++- One listener per enabled engine, serving loopback plaintext for local clients+ and pin-gated cluster mTLS for peers on the same port.+- Model-owner filtering, target precedence, and retryable failover.+- Per-request workload lifecycle events.+- The process-wide reservation map that makes concurrent dispatch spread.+- The persisted per-engine port a user chose through `set-port`.++**Out of scope**++- Deciding node rank (scheduler), membership or pinning (cluster-manager),+ engine install/start/stop (engine-manager).+- Any cryptography of its own: it consumes the trust fabric, never builds it.+- Model search or catalog browsing.++## 3. Process model++The process starts with **no engine and no listener**. The broker then sends one+`facade/enable` per engine, carrying that engine's port and any alias addresses.++A flag cannot express this. The broker plans a different port for each engine —+Ollama's managed facade wants `:11434` while LM Studio's wants `:1234`, and+either may be absent so the child keeps its own persisted port — and a+single-valued flag carries only one plan.++### 3.1 Why one process++Between scheduler snapshots a facade takes short-lived **reservations** for work+it has dispatched but that the scheduler has not yet observed. Those live in the+process. With a process per engine each held half the picture, so simultaneous+Ollama and LM Studio bursts could both pick the same node believing it idle.+Sharing the map is the reason the engines share a process.++### 3.2 What sharing costs++**Shared fate on the process.** A crash takes every facade down; the supervisor+restarts them together and the crash is reported once, against the process.++Inside the process the boundaries are finer, and deliberately so:++| Failure | Blast radius |+| --- | --- |+| Bind race lost on enable | That engine only; reported so the broker can retry elsewhere |+| Port cannot be re-listened | That engine only; its ownership gate is released |+| Panic handling a request | That request only; logged with stack, answered as an internal error |+| Process crash or exit | Every facade |++The first three used to be process-fatal, which was acceptable when the process+was one engine. They are contained now because they are not.++## 4. Addressing++Every facade-scoped message carries the engine it concerns, in both directions.+Without it the engine would be implied by which process a message travelled+through, which is precisely the property one process destroys.++The address is a bare engine id (`ollama:ready`), deliberately **not** the+component name (`ollama-proxy:ready`). The component form is how a client+addresses the component through the broker; the two meet in the broker's relay,+which takes one off and puts the other on.++| Class | Addressed? | Examples |+| --- | --- | --- |+| Facade-scoped | yes | `ready`, `error`, `node/*`, `errors:*`, `set-port`, `discovery:subscribe`, `discovery:nodes` |+| Process-scoped | no | `log/set-level`, `node/set-priority`, `workload:*`, `discovery:node-activity` |+| Bootstrap | names its engine in the payload | `facade/enable` |++`facade/enable` is unaddressed because it runs *before* the facade it names+exists. Only a known engine id counts as an address: plenty of unaddressed+methods contain a colon, so treating `errors:report` as engine `errors` would+strip a real method down to `report`.++An unaddressed facade-scoped message resolves to **no** facade. Falling back to+"the only facade" would work with one engine enabled and misroute silently with+two.++## 5. Request path++For a model-bearing inference request:++1. Filter a request-local discovery snapshot to nodes whose per-engine inventory+ advertises the requested model. Ollama normalizes the implicit `:latest` tag;+ LM Studio ids match exactly. An empty owner set returns a local `502` without+ contacting an engine.+2. Order the eligible owners: explicit `node/select` pin, then the scheduler's+ priority list, then deterministic default ordering.+3. Reserve the least estimated-loaded scheduler-listed candidate and move it to+ the front (§6).+4. Forward, failing over through the remaining candidates on a retryable status+ or transport error. An upstream model `404` counts as retryable, because+ positive inventory can be stale.++An ineligible manual selection cannot override the capability gate, and failover+never broadens to an excluded node.++## 6. Reservations++A reservation is one in-flight dispatch this process has made since the last+scheduler snapshot. Estimated load for a node is+`pending + gpuPressure + reservations`, the first two from the scheduler.++- **Taken** when a candidate is chosen, unless an eligible manual pin applies.+- **Moved** to the node a failover actually lands on, because the node that+ refused the request is not doing the work.+- **Released** when the request ends, so a node stops counting as loaded as soon+ as it stops working. Without release a trickle of short requests makes an idle+ node read as busy for the rest of the snapshot interval.++Each is stamped with the snapshot **generation** it was counted against. A+snapshot supersedes every reservation taken before it — its pending counts+already include that work — so a release naming an older generation is dropped.+Applying it would double-count the completion and drive the node's estimated+load below zero, which reads as permanently idle and attracts every subsequent+dispatch.++`node/set-priority` therefore carries a generation and is applied at most once;+a redelivered snapshot must not clear live reservations.++## 7. Listener personalities++One port per facade, demultiplexed on the connection's first byte:++- **Loopback plaintext** for local clients. A LAN caller is refused.+- **Cluster mTLS** when `--cluster-dir` shows this node is a member: a peer+ whose client certificate matches a local pin is forwarded straight to the+ local engine reported by `node/set-local-backend`, never re-routed onward.++Membership and pins are re-derived per request and on a watch, so joining or+leaving a cluster needs no restart.++An engine with an inherited host variable (today Ollama alone) may also be given+loopback-only **alias** addresses, so clients already using that variable enter+the same routing path. An occupied alias is non-fatal: the existing owner is+untouched, the primary listener stays up, and a warning is reported.++## 8. Ports++| | Ollama | LM Studio |+| --- | --- | --- |+| Engine's own client-facing port | 11434 | 1234 |+| Where PAIR relocates the engine | 11435 | 1235 |+| Standalone port, when `port` is omitted | 11435 | 1234 |+| Persisted-port file | `proxy-port.json` | `lmstudio-proxy-port.json` |++A port chosen at runtime via `set-port` is persisted per engine and restored+when that facade is enabled, taking precedence over the requested port, so the+facade returns where the user left it. `ignorePersistedPort` bypasses that for a+broker-coordinated start.++`port` 0 means "the engine's standalone default". Any other out-of-range value+is rejected rather than honoured as ephemeral: the facade announces the+requested port in `ready`, so binding an ephemeral one would leave the broker+unable to locate it.++## 9. Logging constraints++Never log prompts, chat messages, request bodies, stream chunks, full responses,+credentials, or pairing data. Operational metadata only — engine, model, job id,+node id, path, stream flag, normalized error text — and keep per-request routing+detail below info level.++The log component is the **process** (`nvpair-proxy`), because at process init+there is no engine to name. Facade-scoped records carry an `engine` field, which+is what keeps a line attributable when several facades share the sink.++## 10. Adding an engine++Everything engine-specific is one entry in `engines.go` plus the shared identity+in `nvpair-shared/engines`. A new engine needs: its identity and discovery+service key, its ports and persisted-port filename, its route table and model+normalization, and its empty model-list envelope. It needs no new process, no+new supervisor, and no new relay wiring.++## 11. Failure modes++| Mode | Behavior |+| --- | --- |+| Requested port taken at enable | Tagged bind failure; broker retries on a fallback port |+| No advertised owner for the model | Local `502`, no engine contacted |+| All owners refuse retryably | Last upstream status surfaced |+| Transport error with candidates left | Forget the node's confirmed address, fail over |+| Client disconnects mid-stream | Terminal workload event emitted at once; upstream cancelled |+| Broker link closed | Facades stop serving, then the shared transport pool closes |diff --git a/services/ollama-proxy/subscribed_test.go b/services/nvpair-proxy/subscribed_test.gosimilarity index 50%rename from services/ollama-proxy/subscribed_test.gorename to services/nvpair-proxy/subscribed_test.goindex 7546441..7066160 100644--- a/services/ollama-proxy/subscribed_test.go+++ b/services/nvpair-proxy/subscribed_test.go@@ -86,67 +86,90 @@ func TestSubscribedDiff(t *testing.T) {}}-// TestSubscribedToNode covers the DirectoryNode -> routable Node projection: the-// ol engine port is read from the service key, and a node without ol (or without-// an IP) is rejected.+// TestSubscribedToNode covers the DirectoryNode -> routable Node projection:+// this engine's port is read from its own service key, and a node lacking that+// key, or an IP, is rejected.+//+// The two negative fixtures matter for different reasons and the old suites+// carried one each — Ollama rejected a non-engine service, LM Studio rejected+// the other engine. Both run for both engines here, because the second is what+// regresses if the projection ever loses its service-key filter.func TestSubscribedToNode(t *testing.T) {- withOl := noderec.DirectoryNode{- HostUUID: "uuid-a",- Name: "host-a",- IP: "10.0.0.5",- Models: []string{"llama"},- Services: map[noderec.ServiceKey]noderec.ServiceStatus{- noderec.ServiceOllama: {Port: 11434},- },- }- got, ok := subscribedToNode(withOl)- if !ok {- t.Fatal("node with ol + IP should project")- }- if got.ID != "uuid-a" || got.Port != 11434 || got.IP != "10.0.0.5" || len(got.Models) != 1 || got.Models[0] != "llama" {- t.Fatalf("unexpected projection: %+v", got)- }+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ other := otherEngine(t, tc)++ withService := noderec.DirectoryNode{+ HostUUID: "uuid-a",+ Name: "host-a",+ IP: "10.0.0.5",+ Models: []string{"llama"},+ Services: map[noderec.ServiceKey]noderec.ServiceStatus{+ tc.profile.DiscoveryService: {Port: tc.profile.FacadePort},+ },+ }+ got, ok := subscribedToNode(tc.profile, withService)+ if !ok {+ t.Fatal("node advertising this engine + IP should project")+ }+ if got.ID != "uuid-a" || got.Port != tc.profile.FacadePort || got.IP != "10.0.0.5" ||+ len(got.Models) != 1 || got.Models[0] != "llama" {+ t.Fatalf("unexpected projection: %+v", got)+ }- noIP := withOl- noIP.IP = ""- if _, ok := subscribedToNode(noIP); ok {- t.Fatal("node without IP should not project")- }+ noIP := withService+ noIP.IP = ""+ if _, ok := subscribedToNode(tc.profile, noIP); ok {+ t.Fatal("node without IP should not project")+ }- // A node that advertises only a non-ol service (e.g. node-info) must not be- // treated as an ollama routing target.- niOnly := noderec.DirectoryNode{- Name: "host-c",- IP: "10.0.0.6",- Services: map[noderec.ServiceKey]noderec.ServiceStatus{noderec.ServiceNodeInfo: {Port: 14318}},- }- if _, ok := subscribedToNode(niOnly); ok {- t.Fatal("node without ol should not project")- }+ // A node advertising only a non-engine service (node-info) is not a+ // routing target.+ niOnly := noderec.DirectoryNode{+ Name: "host-c",+ IP: "10.0.0.6",+ Services: map[noderec.ServiceKey]noderec.ServiceStatus{noderec.ServiceNodeInfo: {Port: 14318}},+ }+ if _, ok := subscribedToNode(tc.profile, niOnly); ok {+ t.Fatal("node without this engine's service should not project")+ }- // Per-engine attribution: a dual-engine node projects ONLY its Ollama models- // into the ol proxy's routing inventory, never the cross-engine union — so a- // model served solely via LM Studio isn't ranked as an Ollama owner.- dual := noderec.DirectoryNode{- HostUUID: "uuid-d",- Name: "host-d",- IP: "10.0.0.7",- Models: []string{"ollama-model", "lmstudio-model"},- ModelsByEngine: map[string][]string{- "ollama": {"ollama-model"},- "lmstudio": {"lmstudio-model"},- },- Services: map[noderec.ServiceKey]noderec.ServiceStatus{- noderec.ServiceOllama: {Port: 11434},- },- }- got, ok = subscribedToNode(dual)- if !ok {- t.Fatal("dual-engine node with ol should project")- }- if len(got.Models) != 1 || got.Models[0] != "ollama-model" {- t.Fatalf("dual-engine projection Models = %v, want [ollama-model] only", got.Models)- }+ // Nor is a node running only the *other* engine.+ otherOnly := noderec.DirectoryNode{+ Name: "host-e",+ IP: "10.0.0.8",+ Services: map[noderec.ServiceKey]noderec.ServiceStatus{+ other.DiscoveryService: {Port: other.FacadePort},+ },+ }+ if _, ok := subscribedToNode(tc.profile, otherOnly); ok {+ t.Fatalf("a node running only %s should not project as a %s target",+ other.DisplayName, tc.profile.DisplayName)+ }++ // Per-engine attribution: a dual-engine node projects ONLY this+ // engine's models, never the cross-engine union, so a model served+ // solely via the other engine is not ranked as an owner here.+ dual := noderec.DirectoryNode{+ HostUUID: "uuid-d",+ Name: "host-d",+ IP: "10.0.0.7",+ Models: []string{"mine", "theirs"},+ ModelsByEngine: map[string][]string{+ tc.profile.Name: {"mine"},+ other.Name: {"theirs"},+ },+ Services: map[noderec.ServiceKey]noderec.ServiceStatus{+ tc.profile.DiscoveryService: {Port: tc.profile.FacadePort},+ },+ }+ got, ok = subscribedToNode(tc.profile, dual)+ if !ok {+ t.Fatal("dual-engine node advertising this engine should project")+ }+ if len(got.Models) != 1 || got.Models[0] != "mine" {+ t.Fatalf("dual-engine projection Models = %v, want [mine] only", got.Models)+ }+ })}// TestSubscribedToNodeKeysByHostUUID: the routable Node keys on the stable@@ -154,21 +177,25 @@ func TestSubscribedToNode(t *testing.T) {// scheduler's priority list all survive a PC rename and never conflate two// same-named machines. Host stays the hostname for display.func TestSubscribedToNodeKeysByHostUUID(t *testing.T) {- const uuid = "11111111-1111-1111-1111-111111111111"- n := noderec.DirectoryNode{- HostUUID: uuid,- Name: "host-a",- IP: "10.0.0.5",- Services: map[noderec.ServiceKey]noderec.ServiceStatus{noderec.ServiceOllama: {Port: 11434}},- }- got, ok := subscribedToNode(n)- if !ok {- t.Fatal("node with ol + IP should project")- }- if got.ID != uuid {- t.Fatalf("ID = %q, want hostUuid %q", got.ID, uuid)- }- if got.Host != "host-a" {- t.Fatalf("Host = %q, want hostname for display", got.Host)- }+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ const uuid = "11111111-1111-1111-1111-111111111111"+ n := noderec.DirectoryNode{+ HostUUID: uuid,+ Name: "host-a",+ IP: "10.0.0.5",+ Services: map[noderec.ServiceKey]noderec.ServiceStatus{+ tc.profile.DiscoveryService: {Port: tc.profile.FacadePort},+ },+ }+ got, ok := subscribedToNode(tc.profile, n)+ if !ok {+ t.Fatal("node advertising this engine + IP should project")+ }+ if got.ID != uuid {+ t.Fatalf("ID = %q, want hostUuid %q", got.ID, uuid)+ }+ if got.Host != "host-a" {+ t.Fatalf("Host = %q, want hostname for display", got.Host)+ }+ })}diff --git a/services/lmstudio-proxy/transport.go b/services/nvpair-proxy/transport.gosimilarity index 100%rename from services/lmstudio-proxy/transport.gorename to services/nvpair-proxy/transport.godiff --git a/services/ollama-proxy/transport_pool_test.go b/services/nvpair-proxy/transport_pool_test.gosimilarity index 92%rename from services/ollama-proxy/transport_pool_test.gorename to services/nvpair-proxy/transport_pool_test.goindex 6c5741c..1c4d374 100644--- a/services/ollama-proxy/transport_pool_test.go+++ b/services/nvpair-proxy/transport_pool_test.go@@ -12,7 +12,7 @@ import ()func TestCandidateTransportReusesPlainTransport(t *testing.T) {- p := testProxy(NewDiscovery(), 11435)+ p := testProxy(anyProfile(t), NewDiscovery(), 11435)a := p.candidateTransport(candidate{})b := p.candidateTransport(candidate{id: "manual"})if a != b {@@ -28,7 +28,7 @@ func TestCandidateTransportReusesPeerTransport(t *testing.T) {clusterDir := filepath.Join(t.TempDir(), "cluster")clustertrusttest.Join(t, clusterDir, "cluster-xyz", "principal-self", peerUUID)- p := testProxy(NewDiscovery(), 11435)+ p := testProxy(anyProfile(t), NewDiscovery(), 11435)p.mesh = clustertrust.Open(clusterDir)a := p.candidateTransport(candidate{peerUUID: peerUUID})@@ -51,7 +51,7 @@ func TestDropUnpinnedPeerTransportsRemovesEntry(t *testing.T) {clusterDir := filepath.Join(t.TempDir(), "cluster")clustertrusttest.Join(t, clusterDir, "cluster-xyz", "principal-self", peerUUID)- p := testProxy(NewDiscovery(), 11435)+ p := testProxy(anyProfile(t), NewDiscovery(), 11435)p.mesh = clustertrust.Open(clusterDir)tr := p.candidateTransport(candidate{peerUUID: peerUUID})diff --git a/services/nvpair-proxy/twofacade_test.go b/services/nvpair-proxy/twofacade_test.gonew file mode 100644index 0000000..9136f6a--- /dev/null+++ b/services/nvpair-proxy/twofacade_test.go@@ -0,0 +1,479 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++// Coverage for one process hosting several engine facades at once.+//+// Every other test in this package builds a single-facade proxy, because one+// engine is enough to exercise routing. These are the ones that would catch a+// misroute between facades, because they are the only place two facades exist+// to be confused with each other.++import (+ "bytes"+ "encoding/json"+ stderrors "errors"+ "net"+ "sync"+ "testing"++ "nvpair-shared/engines"+ "nvpair-shared/schedulerwire"+)++// recordingWriter captures the newline-delimited JSON-RPC frames a proxy writes+// upward, so a test can inspect the notifications its facades emit.+type recordingWriter struct {+ mu sync.Mutex+ buf bytes.Buffer+}++func (w *recordingWriter) Read([]byte) (int, error) { return 0, stderrors.New("no input") }++func (w *recordingWriter) Write(p []byte) (int, error) {+ w.mu.Lock()+ defer w.mu.Unlock()+ return w.buf.Write(p)+}++func (w *recordingWriter) lines() [][]byte {+ w.mu.Lock()+ defer w.mu.Unlock()+ var out [][]byte+ for _, line := range bytes.Split(w.buf.Bytes(), []byte("\n")) {+ if len(bytes.TrimSpace(line)) > 0 {+ out = append(out, append([]byte(nil), line...))+ }+ }+ return out+}++// enableOnFreePort enables one engine's facade on an ephemeral port, retrying+// if the port was taken between the probe and the bind.+//+// freeTCPPort can only probe-then-close — a bind probe cannot be held open and+// handed over — so the port is genuinely free when checked and may not be a+// moment later. Retrying is the only honest fix; treating the probe as+// authoritative is what made these tests flake on "address already in use".+func enableOnFreePort(t *testing.T, p *Proxy, engine string) int {+ t.Helper()+ var lastErr error+ for attempt := 0; attempt < 8; attempt++ {+ port := freeTCPPort(t)+ _, err := p.enableFacade(enableFacadeParams{+ Engine: engine,+ Port: port,+ IgnorePersistedPort: true,+ })+ if err == nil {+ return port+ }+ if !stderrors.Is(err, errFacadeBindFailed) {+ t.Fatalf("enable %s facade on :%d: %v", engine, port, err)+ }+ lastErr = err+ }+ t.Fatalf("enable %s facade: every probed port was taken before the bind: %v", engine, lastErr)+ return 0+}++// twoFacadeProxy enables a facade for every engine on one host, each on its own+// ephemeral port so the test does not depend on a real engine's port being free.+func twoFacadeProxy(t *testing.T) *Proxy {+ t.Helper()+ redirectConfigDir(t)++ p := NewProxy(NewCodec(rwNop{}))+ p.serveCtx = t.Context()+ for _, e := range engines.All() {+ enableOnFreePort(t, p, e.Name)+ }+ t.Cleanup(func() { p.shutdown(t.Context()) })++ if got := len(p.enabledFacades()); got != len(engines.All()) {+ t.Fatalf("enabled %d facades, want %d", got, len(engines.All()))+ }+ return p+}++// Each facade binds its own listener and keeps its own dialect. A shared port+// or a shared profile would mean one engine's clients reaching the other's+// translation.+func TestTwoFacadesKeepSeparatePortsAndProfiles(t *testing.T) {+ p := twoFacadeProxy(t)++ ports := map[int]string{}+ for _, e := range engines.All() {+ f := p.facadeFor(e.Name)+ if f == nil {+ t.Fatalf("no facade enabled for %s", e.Name)+ }+ if f.profile.Name != e.Name {+ t.Errorf("facade for %s carries profile %q", e.Name, f.profile.Name)+ }+ if owner, clash := ports[f.port]; clash {+ t.Errorf("facades for %s and %s both bound port %d", e.Name, owner, f.port)+ }+ ports[f.port] = e.Name+ }+}++// Enabling an engine twice is an idempotent success on the existing listener,+// so a redelivered enable cannot tear down a working facade or displace its+// sibling.+func TestReEnablingOneFacadeLeavesBothIntact(t *testing.T) {+ p := twoFacadeProxy(t)++ first := engines.All()[0]+ before := p.facadeFor(first.Name)++ result, err := p.enableFacade(enableFacadeParams{+ Engine: first.Name,+ Port: before.port,+ })+ if err != nil {+ t.Fatalf("re-enable %s: %v", first.Name, err)+ }+ if result.Port != before.port {+ t.Errorf("re-enable reported port %d, want the bound %d", result.Port, before.port)+ }+ if p.facadeFor(first.Name) != before {+ t.Error("re-enable replaced the running facade")+ }+ if got := len(p.enabledFacades()); got != len(engines.All()) {+ t.Errorf("after re-enable there are %d facades, want %d", got, len(engines.All()))+ }+}++// An enable that loses a bind race withdraws only its own engine. This is the+// property that made the collapse acceptable: before, a lost race ended the+// process, which with several facades would take working listeners down.+func TestOneFacadeFailingToBindLeavesTheOthersServing(t *testing.T) {+ redirectConfigDir(t)++ all := engines.All()+ if len(all) < 2 {+ t.Skip("needs at least two engines")+ }+ surviving, failing := all[0], all[1]++ p := NewProxy(NewCodec(rwNop{}))+ p.serveCtx = t.Context()+ t.Cleanup(func() { p.shutdown(t.Context()) })++ enableOnFreePort(t, p, surviving.Name)++ // Hold a port so the second facade cannot have it.+ squatter, err := net.Listen("tcp", ":0")+ if err != nil {+ t.Fatalf("listen: %v", err)+ }+ defer squatter.Close()+ taken := squatter.Addr().(*net.TCPAddr).Port++ _, err = p.enableFacade(enableFacadeParams{+ Engine: failing.Name, Port: taken, IgnorePersistedPort: true,+ })+ if err == nil {+ t.Fatalf("enabling %s on the occupied port %d succeeded", failing.Name, taken)+ }+ if !stderrors.Is(err, errFacadeBindFailed) {+ t.Errorf("bind failure was not tagged retryable: %v", err)+ }++ if p.facadeFor(failing.Name) != nil {+ t.Errorf("%s facade was published despite failing to bind", failing.Name)+ }+ if p.facadeFor(surviving.Name) == nil {+ t.Errorf("%s facade was taken down by %s's bind failure", surviving.Name, failing.Name)+ }+ if got := len(p.enabledFacades()); got != 1 {+ t.Errorf("enabled facades = %d, want just the surviving one", got)+ }+}++// A facade-scoped request reaches only the engine it names, and an engine with+// no facade is refused rather than served by whichever one happens to exist.+func TestAddressedRequestReachesOnlyItsOwnFacade(t *testing.T) {+ p := twoFacadeProxy(t)++ all := engines.All()+ first, second := all[0], all[1]++ // Give each facade a distinct manual node, then confirm nodes/list on one+ // engine never reports the other's.+ for i, e := range []engines.Engine{first, second} {+ node := Node{+ ID: e.Name + "-node",+ Port: 9000 + i,+ Addresses: []string{"127.0.0.1"},+ }+ p.facadeFor(e.Name).discovery.AddManual(node)+ }++ for _, e := range []engines.Engine{first, second} {+ f := p.facadeFor(e.Name)+ nodes := f.discovery.Nodes()+ if len(nodes) != 1 {+ t.Fatalf("%s facade sees %d nodes, want 1: %+v", e.Name, len(nodes), nodes)+ }+ if want := e.Name + "-node"; nodes[0].ID != want {+ t.Errorf("%s facade sees node %q, want %q", e.Name, nodes[0].ID, want)+ }+ }++ // An engine with no facade resolves to nothing rather than to a sibling.+ if f := p.facadeFor("vllm"); f != nil {+ t.Errorf("unknown engine resolved to the %s facade", f.profile.Name)+ }+ if f := p.facadeFor(""); f != nil {+ t.Errorf("an unaddressed message resolved to the %s facade", f.profile.Name)+ }+}++// The scheduler baseline and the reservation map are process-wide, so a+// dispatch through one facade is visible to the other. Two facades bursting at+// once are competing for the same node's GPU; per-facade state would let both+// pick the same idle node in the same instant, which is the whole reason for+// hosting them together.+func TestFacadesShareSchedulerStateAndReservations(t *testing.T) {+ p := twoFacadeProxy(t)++ all := engines.All()+ first, second := all[0], all[1]++ p.SetPrioritySnapshot(schedulerwire.Priority{+ Generation: 1,+ Nodes: []string{"x", "y"},+ Ranks: []schedulerwire.NodeRank{{ID: "x"}, {ID: "y"}},+ })++ // One dispatch through each facade. The second must see the first's claim.+ _, firstRes := p.reserveCandidate(p.facadeFor(first.Name), reservationCandidates("x", "y"))+ _, secondRes := p.reserveCandidate(p.facadeFor(second.Name), reservationCandidates("x", "y"))++ if !firstRes.held || !secondRes.held {+ t.Fatal("a facade failed to take a reservation from the shared snapshot")+ }+ if firstRes.nodeID == secondRes.nodeID {+ t.Fatalf("both facades dispatched to %q: the reservation map is not shared",+ firstRes.nodeID)+ }++ // Releasing through one facade's request does not disturb the other's.+ p.releaseReservation(firstRes)+ if got := reservationCount(p, secondRes.nodeID); got != 1 {+ t.Errorf("%s's reservation on %q = %d after the other facade released, want 1",+ second.Name, secondRes.nodeID, got)+ }+}++// A panic while handling one engine's request must not end the process, because+// the process now holds every engine's listener and the inference streaming+// through it.+//+// Driven through the real dispatch with a payload that panics inside a handler:+// node/select reads the facade's discovery, so a nil discovery panics there+// rather than at the boundary, which is what makes this a test of containment+// and not of argument validation.+func TestPanicHandlingOneEngineLeavesTheOtherServing(t *testing.T) {+ p := twoFacadeProxy(t)++ all := engines.All()+ victim, bystander := all[0], all[1]++ // Take the victim facade's discovery out from under its handler.+ p.facadeFor(victim.Name).discovery = nil++ params, err := json.Marshal(map[string]string{"id": "some-node"})+ if err != nil {+ t.Fatal(err)+ }+ id := json.RawMessage(`7`)+ msg := &Message{+ Method: engines.AddressMethod(victim.Name, "node/select"),+ Params: params,+ ID: &id,+ }++ // The bare call: if the panic is not contained this test binary dies here,+ // which is the failure mode being guarded against.+ p.handleMessage(msg)++ // The bystander facade is untouched and still answers.+ if f := p.facadeFor(bystander.Name); f == nil {+ t.Fatalf("%s facade disappeared after %s panicked", bystander.Name, victim.Name)+ }+ bystanderMsg := &Message{+ Method: engines.AddressMethod(bystander.Name, "nodes/list"),+ ID: &id,+ }+ p.handleMessage(bystanderMsg)++ if got := len(p.enabledFacades()); got != len(all) {+ t.Errorf("enabled facades = %d after a panic, want %d", got, len(all))+ }+}++// A panic during bring-up withdraws that facade and releases its port, while+// every other facade keeps serving.+//+// This path deliberately does not run on the error return — it is a deferred+// rollback, because a panic skips the error return entirely. Without a test,+// dropping the defer (or the listener close inside start) leaves a published+// facade holding a port with no serving loop, and CI stays green.+func TestPanicDuringEnableWithdrawsOnlyThatFacade(t *testing.T) {+ redirectConfigDir(t)++ all := engines.All()+ if len(all) < 2 {+ t.Skip("needs at least two engines")+ }+ surviving, panicking := all[0], all[1]++ p := NewProxy(NewCodec(rwNop{}))+ p.serveCtx = t.Context()+ t.Cleanup(func() { p.shutdown(t.Context()) })++ enableOnFreePort(t, p, surviving.Name)++ // Panic inside bring-up, after the facade is published and its listener is+ // bound: nil the host codec so the ready notification dereferences it.+ port := freeTCPPort(t)+ codec := p.codec+ p.codec = nil+ func() {+ defer func() {+ if recover() == nil {+ t.Error("enable did not panic, so the rollback path was not exercised")+ }+ p.codec = codec+ }()+ _, _ = p.enableFacade(enableFacadeParams{+ Engine: panicking.Name, Port: port, IgnorePersistedPort: true,+ })+ }()++ if f := p.facadeFor(panicking.Name); f != nil {+ t.Errorf("%s facade survived a panic during its bring-up", panicking.Name)+ }+ if p.facadeFor(surviving.Name) == nil {+ t.Errorf("%s facade was taken down by %s's panic", surviving.Name, panicking.Name)+ }++ // The port is bindable again, which is the part the deferred rollback is+ // for: a listener left holding it would make the next enable read as a lost+ // bind race and strand that engine on a fallback for the process life.+ //+ // Re-enabling is the check rather than a bare net.Listen, because it+ // distinguishes the two reasons a bind can fail here. If this process still+ // held the port the rollback leaked it; if something else took it in the+ // meantime that is an unrelated race, and skipping says so instead of+ // blaming the rollback.+ if _, err := p.enableFacade(enableFacadeParams{+ Engine: panicking.Name, Port: port, IgnorePersistedPort: true,+ }); err != nil {+ if stderrors.Is(err, errFacadeBindFailed) {+ t.Skipf("port %d was taken by something else between the rollback and the re-enable: %v", port, err)+ }+ t.Fatalf("re-enable after the rollback failed for a non-bind reason: %v", err)+ }+ if p.facadeFor(panicking.Name) == nil {+ t.Errorf("%s did not come back up on the released port %d", panicking.Name, port)+ }+}++// Alias validation runs on the surface the broker actually uses. The addresses+// arrive in facade/enable now rather than on argv, and the alias listener is a+// plaintext one, so a weakened check here would put a LAN plaintext listener in+// front of the router.+func TestEnableFacadeRejectsUnsafeAliasAddresses(t *testing.T) {+ redirectConfigDir(t)++ var aliasEngine, plainEngine engines.Engine+ for _, e := range engines.All() {+ p, ok := profileFor(e.Name)+ if !ok {+ t.Fatalf("no profile for %s", e.Name)+ }+ if p.SupportsHostAlias {+ aliasEngine = e+ } else {+ plainEngine = e+ }+ }+ if aliasEngine.Name == "" || plainEngine.Name == "" {+ t.Skip("needs one alias-capable and one alias-incapable engine")+ }++ for _, tc := range []struct {+ name string+ engine string+ alias string+ }{+ {"a LAN address", aliasEngine.Name, "192.168.1.20:11433"},+ {"the wildcard address", aliasEngine.Name, "0.0.0.0:11433"},+ {"an engine with no inherited host variable", plainEngine.Name, "127.0.0.1:11433"},+ } {+ t.Run(tc.name, func(t *testing.T) {+ p := NewProxy(NewCodec(rwNop{}))+ p.serveCtx = t.Context()+ t.Cleanup(func() { p.shutdown(t.Context()) })++ if _, err := p.enableFacade(enableFacadeParams{+ Engine: tc.engine,+ Port: freeTCPPort(t),+ AliasAddresses: []string{tc.alias},+ IgnorePersistedPort: true,+ }); err == nil {+ t.Fatalf("enable accepted alias %q for %s", tc.alias, tc.engine)+ }+ // Rejected outright, not enabled-then-partially-configured.+ if f := p.facadeFor(tc.engine); f != nil {+ t.Errorf("a rejected alias still left a %s facade enabled", tc.engine)+ }+ })+ }+}++// Facade notifications are addressed, so the broker can attribute a bare+// "ready" to an engine when several share the stream. Without the address the+// two facades' events are indistinguishable on the wire.+func TestBothFacadesAddressTheirNotifications(t *testing.T) {+ redirectConfigDir(t)++ recorded := &recordingWriter{}+ p := NewProxy(NewCodec(recorded))+ p.serveCtx = t.Context()+ t.Cleanup(func() { p.shutdown(t.Context()) })++ for _, e := range engines.All() {+ enableOnFreePort(t, p, e.Name)+ }++ readyFor := map[string]bool{}+ for _, line := range recorded.lines() {+ var msg struct {+ Method string `json:"method"`+ }+ if json.Unmarshal(line, &msg) != nil {+ continue+ }+ engine, bare := engines.SplitAddressedMethod(msg.Method)+ if bare != "ready" {+ continue+ }+ if engine == "" {+ t.Errorf("a facade announced an unaddressed %q", msg.Method)+ continue+ }+ readyFor[engine] = true+ }++ for _, e := range engines.All() {+ if !readyFor[e.Name] {+ t.Errorf("no addressed ready notification for %s; saw %v", e.Name, readyFor)+ }+ }+}diff --git a/services/ollama-proxy/workload_test.go b/services/nvpair-proxy/workload_test.gosimilarity index 50%rename from services/ollama-proxy/workload_test.gorename to services/nvpair-proxy/workload_test.goindex 64fac63..07acdd8 100644--- a/services/ollama-proxy/workload_test.go+++ b/services/nvpair-proxy/workload_test.go@@ -4,6 +4,7 @@package mainimport (+ "fmt""io""net/http""net/http/httptest"@@ -49,55 +50,55 @@ func (r *recRW) has(s string) bool {// the node that actually served after any failover); only workload:started// moves to forward time, so this test asserts only that.func TestHandleHTTP_WorkloadStartedAtForward(t *testing.T) {- received := make(chan struct{}, 1)- release := make(chan struct{})- var releaseOnce sync.Once- doRelease := func() { releaseOnce.Do(func() { close(release) }) }+ forEachEngine(t, func(t *testing.T, tc engineCase) {+ received := make(chan struct{}, 1)+ release := make(chan struct{})+ var releaseOnce sync.Once+ doRelease := func() { releaseOnce.Do(func() { close(release) }) }- slow := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- select {- case received <- struct{}{}:- default:- }- <-release // no response byte until the test releases us- w.WriteHeader(http.StatusOK)- io.WriteString(w, `{"done":true}`)- }))- // Deferred order matters (LIFO): doRelease runs BEFORE slow.Close so the- // blocked upstream handler is unblocked before the server is torn down —- // otherwise Close hangs on the active connection, including on t.Fatal.- defer slow.Close()- defer doRelease()+ slow := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ select {+ case received <- struct{}{}:+ default:+ }+ <-release // no response byte until the test releases us+ w.WriteHeader(http.StatusOK)+ io.WriteString(w, `{"done":true}`)+ }))+ // Deferred order matters (LIFO): doRelease runs BEFORE slow.Close so+ // the blocked upstream handler is unblocked before the server is torn+ // down — otherwise Close hangs on the active connection, including on+ // t.Fatal.+ defer slow.Close()+ defer doRelease()- rec := &recRW{}- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "busy-node", slow.URL, "llama"))- p := NewProxy(NewCodec(rec), disc, 11434)+ rec := &recRW{}+ disc := NewDiscovery()+ disc.AddManual(nodeForModel(t, "busy-node", slow.URL, tc.advertisedModel))+ p := newTestProxy(tc.profile, NewCodec(rec), disc, tc.profile.FacadePort)- done := make(chan struct{})- go func() {- defer close(done)- p.handleHTTP(httptest.NewRecorder(),- httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)))- }()+ done := make(chan struct{})+ go func() {+ defer close(done)+ p.soleFacade().handleHTTP(httptest.NewRecorder(), httptest.NewRequest(+ http.MethodPost, tc.inferencePath,+ strings.NewReader(fmt.Sprintf(`{"model":%q}`, tc.requestedModel))))+ }()- select {- case <-received:- case <-time.After(5 * time.Second):- t.Fatal("upstream never received the forwarded request")- }+ select {+ case <-received:+ case <-time.After(5 * time.Second):+ t.Fatal("upstream never received the forwarded request")+ }- // The upstream has the request but has sent no response byte, so a- // first-byte emission could not have fired. workload:started must still be- // on the wire — that is the forward-time emission.- deadline := time.Now().Add(2 * time.Second)- for time.Now().Before(deadline) && !rec.has("workload:started") {- time.Sleep(5 * time.Millisecond)- }- if !rec.has("workload:started") {- t.Fatal("workload:started not emitted at forward time")- }+ // The upstream has the request but has sent no response byte, so a+ // first-byte emission could not have fired. workload:started must+ // still be on the wire — that is the forward-time emission.+ if !waitFor(t, rec, "workload:started") {+ t.Fatal("workload:started not emitted at forward time")+ }- doRelease()- <-done+ doRelease()+ <-done+ })}diff --git a/services/ollama-proxy/zombie_test.go b/services/nvpair-proxy/zombie_test.gosimilarity index 91%rename from services/ollama-proxy/zombie_test.gorename to services/nvpair-proxy/zombie_test.goindex b146760..372d8f3 100644--- a/services/ollama-proxy/zombie_test.go+++ b/services/nvpair-proxy/zombie_test.go@@ -19,6 +19,12 @@ import ("time")+// This file runs on one arbitrary engine (anyCase), not both. Its subject is+// statusCapture's write and flush deadlines and the terminalOnce guard, which+// never consult the profile; the inference request is only the vehicle that+// reaches the streaming path. Two of these drive real sockets against+// multi-second deadlines, so a second pass would cost real time for no branch.+// count reports how many newline-delimited frames the codec wrote that mention// s. Method names appear once per emitted notification, so counting the// substring counts emissions.@@ -73,17 +79,18 @@ func TestHandleHTTP_ClientWriteError_MarksFailed(t *testing.T) {}))defer upstream.Close()+ tc := anyCase(t)rec := &recRW{}disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "node-a", upstream.URL, "llama"))- p := NewProxy(NewCodec(rec), disc, 11434)+ disc.AddManual(nodeForModel(t, "node-a", upstream.URL, tc.advertisedModel))+ p := newTestProxy(tc.profile, NewCodec(rec), disc, tc.profile.FacadePort)cw := &clientGoneWriter{err: errors.New("write tcp: connection reset by peer")}done := make(chan struct{})go func() {defer close(done)- p.handleHTTP(cw, httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)))+ p.soleFacade().handleHTTP(cw, tc.inferenceRequest())}()select {@@ -131,18 +138,19 @@ func TestHandleHTTP_ClientDisconnect_TerminalOnce(t *testing.T) {defer upstream.Close()defer doRelease()+ tc := anyCase(t)rec := &recRW{}disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "node-a", upstream.URL, "llama"))- p := NewProxy(NewCodec(rec), disc, 11434)+ disc.AddManual(nodeForModel(t, "node-a", upstream.URL, tc.advertisedModel))+ p := newTestProxy(tc.profile, NewCodec(rec), disc, tc.profile.FacadePort)ctx, cancel := context.WithCancel(context.Background())- req := httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)).WithContext(ctx)+ req := tc.inferenceRequest().WithContext(ctx)done := make(chan struct{})go func() {defer close(done)- p.handleHTTP(httptest.NewRecorder(), req)+ p.soleFacade().handleHTTP(httptest.NewRecorder(), req)}()// Wait until the upstream is actively streaming, then simulate the client@@ -349,16 +357,17 @@ func TestHandleHTTP_RealSocketWriteDeadline(t *testing.T) {}))defer upstream.Close()+ tc := anyCase(t)rec := &recRW{}disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "node-a", upstream.URL, "llama"))- p := NewProxy(NewCodec(rec), disc, 11434)+ disc.AddManual(nodeForModel(t, "node-a", upstream.URL, tc.advertisedModel))+ p := newTestProxy(tc.profile, NewCodec(rec), disc, tc.profile.FacadePort)ln, err := net.Listen("tcp", "127.0.0.1:0")if err != nil {t.Fatalf("listen: %v", err)}- srv := &http.Server{Handler: http.HandlerFunc(p.handleHTTP)}+ srv := &http.Server{Handler: http.HandlerFunc(p.soleFacade().handleHTTP)}go func() { _ = srv.Serve(ln) }()defer srv.Close()@@ -368,8 +377,8 @@ func TestHandleHTTP_RealSocketWriteDeadline(t *testing.T) {}defer conn.Close()- body := `{"model":"llama"}`- reqText := "POST /api/chat HTTP/1.1\r\n" ++ body := tc.inferenceBody()+ reqText := fmt.Sprintf("POST %s HTTP/1.1\r\n", tc.inferencePath) +"Host: localhost\r\n" +"Content-Type: application/json\r\n" +fmt.Sprintf("Content-Length: %d\r\n", len(body)) +@@ -449,16 +458,17 @@ func TestHandleHTTP_RealSocketFlushDeadline(t *testing.T) {}))defer upstream.Close()+ tc := anyCase(t)rec := &recRW{}disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "node-a", upstream.URL, "llama"))- p := NewProxy(NewCodec(rec), disc, 11434)+ disc.AddManual(nodeForModel(t, "node-a", upstream.URL, tc.advertisedModel))+ p := newTestProxy(tc.profile, NewCodec(rec), disc, tc.profile.FacadePort)ln, err := net.Listen("tcp", "127.0.0.1:0")if err != nil {t.Fatalf("listen: %v", err)}- srv := &http.Server{Handler: http.HandlerFunc(p.handleHTTP)}+ srv := &http.Server{Handler: http.HandlerFunc(p.soleFacade().handleHTTP)}go func() { _ = srv.Serve(ln) }()defer srv.Close()@@ -468,8 +478,8 @@ func TestHandleHTTP_RealSocketFlushDeadline(t *testing.T) {}defer conn.Close()- body := `{"model":"llama"}`- reqText := "POST /api/chat HTTP/1.1\r\n" ++ body := tc.inferenceBody()+ reqText := fmt.Sprintf("POST %s HTTP/1.1\r\n", tc.inferencePath) +"Host: localhost\r\n" +"Content-Type: application/json\r\n" +fmt.Sprintf("Content-Length: %d\r\n", len(body)) +diff --git a/services/nvpair-tui/README.md b/services/nvpair-tui/README.mdindex e5b92ca..3963d1a 100644--- a/services/nvpair-tui/README.md+++ b/services/nvpair-tui/README.md@@ -71,7 +71,7 @@ nvpair-tui (this process)└── ui/ Bubble Tea root model + one file per tab│ stdio (newline-delimited JSON-RPC 2.0)▼- nvpair-ui-broker ──► nvpair-node-scanner, ollama-proxy, nvpair-errors, ... (workers)+ nvpair-ui-broker ──► nvpair-node-scanner, nvpair-proxy, nvpair-errors, ... (workers)```The supervisor sends `shutdown` and closes the broker's stdin on exit; thediff --git a/services/nvpair-tui/ui/health.go b/services/nvpair-tui/ui/health.goindex c1863ea..587ecf7 100644--- a/services/nvpair-tui/ui/health.go+++ b/services/nvpair-tui/ui/health.go@@ -8,6 +8,7 @@ import ("strings""time"+ "nvpair-shared/engines"svcerrors "nvpair-shared/errors""nvpair-tui/rpc"@@ -30,16 +31,28 @@ const healthPollInterval = 5 * time.Second// healthWorkers are the workers the broker supervises and reports crashes// for. nvpair-errors is deliberately absent: it is the error sink itself and// cannot report its own death, so it has no crash entry to key on.+//+// The proxy appears once, as engines.ProxyComponent, because one nvpair-proxy+// process hosts every engine's facade under one supervisor — so the broker+// reports one crash for the process rather than one per engine. Keying this on+// the per-engine ComponentName instead would be silent in both directions: the+// real crash entry would match no row, and the per-engine rows could never+// leave "ok". See the identity split in nvpair-shared/engines.var healthWorkers = []string{"scanner","node-info",- "proxy",- "lmstudio-proxy",+ engines.ProxyComponent,"workload-manager","engine-manager","manual-nodes","settings","cluster-manager",+ // The scheduler produces the rankings every facade routes on. Without it+ // each facade holds an empty ranking, takes no reservations, and dispatch+ // stops spreading — so its death is exactly the kind an operator would come+ // to this screen to find. Its crash key is "scheduler", from+ // startOptionalWorker in the broker.+ "scheduler",}// healthView is the Overview tab: broker liveness/version/uptime fromdiff --git a/services/nvpair-tui/ui/health_test.go b/services/nvpair-tui/ui/health_test.goindex 2e03b0b..f674f5d 100644--- a/services/nvpair-tui/ui/health_test.go+++ b/services/nvpair-tui/ui/health_test.go@@ -6,9 +6,52 @@ package uiimport ("testing"+ "nvpair-shared/engines"svcerrors "nvpair-shared/errors")+// The Overview's proxy row must key on the identity the broker actually stamps+// on a proxy crash.+//+// One nvpair-proxy process hosts every engine's facade under one supervisor, so+// the broker reports one crash for the process. Keying on the per-engine+// ComponentName was silent in both directions: the real crash entry matched no+// row, so a dead proxy was invisible in the one view meant to show worker+// liveness, and the per-engine rows could never leave "ok". Nothing else in the+// build catches that — the mismatch compiles and every other test passes — so+// this assertion is what makes the identity rule in nvpair-shared/engines+// enforceable rather than advisory.+func TestHealthProxyRowMatchesTheBrokerCrashIdentity(t *testing.T) {+ found := false+ for _, w := range healthWorkers {+ if w == engines.ProxyComponent {+ found = true+ }+ for _, e := range engines.All() {+ if w == e.ComponentName() {+ t.Errorf("health row %q keys on a per-facade identity; the broker reports proxy crashes against %q",+ w, engines.ProxyComponent)+ }+ }+ }+ if !found {+ t.Fatalf("no health row keys on %q, so a proxy crash would have no row at all: %v",+ engines.ProxyComponent, healthWorkers)+ }++ // End to end through the real matcher, with the id the broker builds.+ v := newHealthView(nil)+ v.localNodeUUID = "self-uuid"+ v.rebuildCrashes([]svcerrors.ServiceError{{+ ID: crashPrefix + engines.ProxyComponent,+ Message: "proxy crashed",+ NodeID: "self-uuid",+ }})+ if _, down := v.crashed[engines.ProxyComponent]; !down {+ t.Fatalf("a proxy crash did not register: %v", v.crashed)+ }+}+// TestHealthRebuildCrashesFiltersByUUID: the Overview// keeps only local-origin crashes, keyed on this host's stable UUID (the value// the broker stamps on local reports). A peer's crash must be dropped, and a@@ -21,14 +64,14 @@ func TestHealthRebuildCrashesFiltersByUUID(t *testing.T) {return svcerrors.ServiceError{ID: crashPrefix + worker, Message: worker + " crashed", NodeID: nodeID}}v.rebuildCrashes([]svcerrors.ServiceError{- crash("scanner", "self-uuid"), // local crash — keep- crash("proxy", "peer-uuid"), // a peer's crash — drop+ crash("scanner", "self-uuid"), // local crash — keep+ crash("ollama-proxy", "peer-uuid"), // a peer's crash — drop})if _, down := v.crashed["scanner"]; !down {t.Fatal("local UUID-stamped crash should be surfaced, not filtered as remote")}- if _, down := v.crashed["proxy"]; down {+ if _, down := v.crashed["ollama-proxy"]; down {t.Fatal("a peer's crash must be filtered out of the local health view")}}diff --git a/services/nvpair-tui/ui/proxies.go b/services/nvpair-tui/ui/proxies.goindex 072c1a8..42c36dc 100644--- a/services/nvpair-tui/ui/proxies.go+++ b/services/nvpair-tui/ui/proxies.go@@ -8,6 +8,7 @@ import ("strconv""strings"+ "nvpair-shared/engines""nvpair-tui/rpc""github.com/charmbracelet/bubbles/key"@@ -17,19 +18,30 @@ import ()// proxyNode mirrors a discovered upstream as reported by the proxy's-// nodes/list (ollama-proxy/discovery.go Node).+// nodes/list (nvpair-proxy/discovery.go Node).type proxyNode struct {ID string `json:"id"`Host string `json:"host"`Port int `json:"port"`}-// proxyEngine is one of the two reverse proxies the broker fronts. Both-// speak the same routing/failover contract; only the JSON-RPC prefix and-// label differ.+// buildProxyEngines makes one tab per engine, in the shared table's order, so+// an engine added there appears here rather than being silently absent from+// this view.+func buildProxyEngines() []*proxyEngine {+ all := engines.All()+ out := make([]*proxyEngine, 0, len(all))+ for _, e := range all {+ out = append(out, &proxyEngine{label: e.DisplayName, prefix: e.ComponentName(), table: newTable(nil)})+ }+ return out+}++// proxyEngine is one reverse proxy the broker fronts. They all speak the same+// routing/failover contract; only the JSON-RPC prefix and label differ.type proxyEngine struct {label string // "Ollama" / "LM Studio"- prefix string // "proxy" / "lmstudio-proxy"+ prefix string // "ollama-proxy" / "lmstudio-proxy"ready boolport intselected string@@ -89,10 +101,7 @@ func newProxiesView(client *rpc.Client) *proxiesView {v := &proxiesView{client: client,portInput: ti,- engines: []*proxyEngine{- {label: "Ollama", prefix: "proxy", table: newTable(nil)},- {label: "LM Studio", prefix: "lmstudio-proxy", table: newTable(nil)},- },+ engines: buildProxyEngines(),}return v}@@ -211,13 +220,17 @@ func (v *proxiesView) Update(msg tea.Msg) tea.Cmd {}func (v *proxiesView) handleNotification(msg *rpc.Message) tea.Cmd {+ // Match against the prefixes the tabs were built from rather than a fixed+ // switch, so this follows the engine table instead of hardcoding both the+ // names and their positions.idx := -1- switch {- case strings.HasPrefix(msg.Method, "lmstudio-proxy:"):- idx = 1- case strings.HasPrefix(msg.Method, "proxy:"):- idx = 0- default:+ for i, e := range v.engines {+ if strings.HasPrefix(msg.Method, e.prefix+":") {+ idx = i+ break+ }+ }+ if idx < 0 {return nil}if strings.HasSuffix(msg.Method, ":ready") {diff --git a/services/nvpair-ui-broker/README.md b/services/nvpair-ui-broker/README.mdindex d2b0e17..6eeee29 100644--- a/services/nvpair-ui-broker/README.md+++ b/services/nvpair-ui-broker/README.md@@ -20,7 +20,7 @@ namespace:| --- | --- | --- || `nvpair-node-scanner` | Discovery daemon: advertises this host's one `_nvpair-node._tcp` record and browses the LAN | `discovery:*` || `nvpair-node-info` | Local GPU / CPU / memory inventory over HTTP at `/v1/node-info` | — (HTTP only) |-| `ollama-proxy` | Ollama-compatible inference proxy and router | `proxy:*` |+| `ollama-proxy` | Ollama-compatible inference proxy and router | `ollama-proxy:*` || `lmstudio-proxy` | The LM Studio counterpart, supervised identically | `lmstudio-proxy:*` || `nvpair-engine-manager` | Local engine and model control plane; also serves `GET /v1/models` to peers | `engine:*` || `nvpair-cluster-manager` | Node identity, trusted-node store, PIN pairing | `cluster:*`, `nodes:*` |@@ -57,6 +57,8 @@ Two responsibilities live in the broker itself rather than in a worker:Every worker runs under a supervisor that auto-restarts it on an unexpected exit: exponential backoff (~1 s → 16 s), a budget of 5 attempts per unhealthy streak, and a 60 s "healthy reset" window after which a restarted worker is considered recovered. A crash is surfaced as a sticky `supervisor:subprocess-crashed:<name>` entry in the error registry (so connected tools can show "this helper is down"); recovery clears it, and a worker that exhausts its restart budget leaves the entry up. The scanner is the one worker whose **initial** spawn is fatal (the broker has no job without discovery); every other worker is optional and a failed first spawn just runs the broker without it. `nvpair-errors` is a special case: it can't report its own death into itself, so its crash logs to stderr (rather than the pipeline) while still being auto-restarted.+There is **one supervisor per process, not per engine**. `nvpair-proxy` hosts a facade for every enabled engine, so its crash entry names the process (`nvpair-proxy`) and its restart brings every facade back together; a terminal failure releases each engine's port-ownership gate individually, so a caller waiting on one engine is never held by another's state.+## CommunicationBidirectional newline-delimited JSON-RPC 2.0 — same conventions as every other NVPAIR subprocess. By default the broker reads stdin and writes stdout; with `--ipc <path>` it dials a Unix domain socket or Windows named pipe instead. The parent UI process owns the endpoint in both cases: it either spawns the broker with piped stdio, or sets up the pipe / socket beforehand and points the broker at it via `--ipc`. The broker always talks to exactly one peer.@@ -68,8 +70,8 @@ Bidirectional newline-delimited JSON-RPC 2.0 — same conventions as every other| `--ipc <path>` | _(stdio)_ | IPC endpoint to dial: Unix socket path or Windows named pipe (e.g. `\\.\pipe\nvpair-ui-broker`) || `--scanner-path <path>` | `./nvpair-node-scanner[.exe]` in the CWD | Explicit path to the `nvpair-node-scanner` binary the broker should spawn || `--node-info-path <path>` | `./nvpair-node-info[.exe]` in the CWD | Explicit path to the `nvpair-node-info` binary the broker should spawn. When omitted and no default sibling exists, the broker runs without the local inventory server (non-fatal); when set to an invalid path, the broker exits with an error |-| `--proxy-path <path>` | `./ollama-proxy[.exe]` in the CWD | Explicit path to the `ollama-proxy` binary the broker spawns for the local Ollama reverse proxy. Same optional semantics as `--node-info-path`: an absent default sibling means no local proxy (non-fatal); an invalid explicit path exits with an error |-| `--lmstudio-proxy-path <path>` | `./lmstudio-proxy[.exe]` in the CWD | Explicit path to the `lmstudio-proxy` binary the broker spawns for the local LM Studio reverse proxy. Same optional semantics as `--proxy-path` |+| `--proxy-path <path>` | `./nvpair-proxy[.exe]` in the CWD | Explicit path to the `nvpair-proxy` binary. One process fronts every engine: the broker spawns it once and then sends a `facade/enable` per entry in `--proxy-engines`. Same optional semantics as `--node-info-path`: an absent default sibling means no local proxies (non-fatal); an invalid explicit path exits with an error |+| `--proxy-engines <csv>` | every engine in `nvpair-shared/engines` (currently `ollama,lmstudio`) | Which engines to front with a proxy. An unrecognized name exits with an error rather than being skipped, so a typo cannot look like it worked. An engine left out is not started **and not prepared** — the broker will not relocate an engine whose facade nothing is going to claim || `--workload-manager-path <path>` | `./nvpair-workload-manager[.exe]` in the CWD | Explicit path to the `nvpair-workload-manager` binary the broker spawns for the cluster workload relay. Same optional semantics as `--node-info-path`: an absent default sibling means no workload relay (non-fatal); an invalid explicit path exits with an error || `--errors-path <path>` | `./nvpair-errors[.exe]` in the CWD | Explicit path to the `nvpair-errors` binary the broker spawns (with `--peer-sync`) for the service-error pipeline. Same optional semantics as `--node-info-path`: an absent default sibling means the error pipeline is disabled — producers' errors are dropped (non-fatal); an invalid explicit path exits with an error || `--engine-manager-path <path>` | `./nvpair-engine-manager[.exe]` in the CWD | Explicit path to the `nvpair-engine-manager` binary the broker spawns for engine management. Same optional semantics as `--node-info-path` |@@ -81,11 +83,11 @@ Bidirectional newline-delimited JSON-RPC 2.0 — same conventions as every other| `--log-level <lvl>` | _(env `NVPAIR_LOG_LEVEL` or `info`)_ | `debug` \| `info` \| `warn` \| `error` || `--version` | | Print version and exit |-Logs go to **stderr** (shared `applog` format, same as every other NVPAIR binary). stdout is reserved for JSON-RPC frames in stdio mode, so logging there would corrupt the protocol. Every spawned worker's stderr is forwarded to the broker's stderr unmodified, so `[nvpair-node-scanner]`, `[nvpair-node-info]`, `[ollama-proxy]`, `[nvpair-workload-manager]`, and `[nvpair-cluster-manager]` lines interleave with the broker's `[nvpair-ui-broker]` lines on a single stream.+Logs go to **stderr** (shared `applog` format, same as every other NVPAIR binary). stdout is reserved for JSON-RPC frames in stdio mode, so logging there would corrupt the protocol. Every spawned worker's stderr is forwarded to the broker's stderr unmodified, so `[nvpair-node-scanner]`, `[nvpair-node-info]`, `[nvpair-proxy]`, `[nvpair-workload-manager]`, and `[nvpair-cluster-manager]` lines interleave with the broker's `[nvpair-ui-broker]` lines on a single stream. The proxy logs under the process name because that is what it is; a line about one engine's facade carries an `engine` field instead of a distinct prefix.## Worker subprocesses-On startup — **before** emitting `app:ready` — the broker spawns the scanner and (when available) node-info, ollama-proxy, the workload-manager, and the cluster-manager as child processes over stdio. The proxy is spawned up front but doesn't gate `app:ready` — it announces its listen port asynchronously (see below). None of the auxiliary workers gate `app:ready`.+On startup — **before** emitting `app:ready` — the broker spawns the scanner and (when available) node-info, `nvpair-proxy`, the workload-manager, and the cluster-manager as child processes over stdio. The proxy is spawned up front but doesn't gate `app:ready` — each of its facades announces its listen port asynchronously (see below). None of the auxiliary workers gate `app:ready`.**`nvpair-node-scanner`** (the consolidated discovery daemon) is spawned first. It pushes `discovery:node-discovered`, `discovery:node-updated`, and `discovery:node-removed` notifications into the broker, which maintains them in an in-memory map keyed by `id`. Clients query that map via `discovery:get-nodes` and — once they've opted in via `discovery:subscribe` — receive a `discovery:nodes-changed` notification on every store mutation. The raw `discovery:node-*` notifications are never forwarded as-is. The scanner polls healthy node-info endpoints on a staggered two-second cadence, backs consecutive remote failures off to a 30-second cap, and emits compact `discovery:node-telemetry` observations containing maximum GPU utilization, validity, and age; these remain internal to broker scheduling. The broker registers this node's local service ports (`ni`/`er`/`wl`/`cl`/`em`, plus `ol`/`lm` from the engine poller) with the daemon over the same link, so the daemon can advertise them all in one `_nvpair-node` record.@@ -98,13 +100,21 @@ On startup — **before** emitting `app:ready` — the broker spawns the scannerThe daemon folds those registrations into this host's single `_nvpair-node` record, so a peer discovers the engine through the shared channel. The model list is not part of that registration — it's served over HTTP by `nvpair-engine-manager` (the `em` service, `GET /v1/models`) and enriched onto each node by the peer's daemon. There is no separate advertiser subprocess and no manual-advertise RPC.-**`ollama-proxy`** is a local Ollama reverse proxy: it forwards inference requests to an Ollama node it discovers on the network. Its standalone default is `:11435`; with managed port ownership enabled (the default), the broker starts settings and engine-manager first, claims `:11434` with the proxy, and only then moves a stopped default-port Ollama backend to a free port. Custom backend ports are preserved. When the inherited `OLLAMA_HOST` names a distinct local plaintext port, the broker also gives the proxy that normalized loopback-only alias so clients already using the variable enter the same routing path; `localhost` reserves both canonical loopback families atomically, while remote and HTTPS targets are ignored. The alias port is reserved against every configured engine, local or remote engine start override, both proxy control planes, and the managed Ollama and LM Studio backend port plans, so a backend that has to move can never land on the alias. A running Ollama or unknown owner on either requested port is never stopped or moved: the primary uses a safe fallback when needed, and an occupied alias remains with its owner while the broker reports a warning. For automatic model-bearing inference, both engine proxies combine scheduler pending counts and GPU pressure with short-lived local reservations under one lock before forwarding, so concurrent requests distribute without an artificial delay or a round trip through the scheduler. Manual pins, model-owner tiers, and the complete failover list keep their existing precedence. The broker otherwise treats the proxy as **optional and non-fatal**.+**`nvpair-proxy`** is one process that fronts every enabled engine. It starts with no engine and no listener; the broker then sends it a `facade/enable` per engine, carrying that engine's port and any alias addresses. A flag could not express this, because the broker plans a different port for each engine. Each facade forwards inference to a node it discovers on the network and speaks its own engine's dialect.++One process for all of them is deliberate. Between scheduler snapshots a facade takes short-lived reservations for work it has dispatched, and those live in the process — two processes each held half that picture, so simultaneous Ollama and LM Studio bursts could both pick the same node believing it idle. The cost is **shared fate**: a crash takes every facade down and the supervisor brings them all back together, reported once as `supervisor:subprocess-crashed:nvpair-proxy` rather than against one engine. Within the process the boundaries are finer — a facade that loses its bind race, cannot be moved to a free port, or panics while handling a request is withdrawn or answered with an error on its own, leaving the others serving.++Ollama's standalone default is `:11435`; with managed port ownership enabled (the default), the broker starts settings and engine-manager first, claims `:11434` with that facade, and only then moves a stopped default-port Ollama backend to a free port. Custom backend ports are preserved. When the inherited `OLLAMA_HOST` names a distinct local plaintext port, the broker also gives the facade that normalized loopback-only alias so clients already using the variable enter the same routing path; `localhost` reserves both canonical loopback families atomically, while remote and HTTPS targets are ignored. The alias port is reserved against every configured engine, local or remote engine start override, every facade's control plane, and the managed Ollama and LM Studio backend port plans, so a backend that has to move can never land on the alias. A running Ollama or unknown owner on either requested port is never stopped or moved: the primary uses a safe fallback when needed, and an occupied alias remains with its owner while the broker reports a warning.++For automatic model-bearing inference, every facade combines scheduler pending counts and GPU pressure with the process-wide reservation map under one lock before forwarding, so concurrent requests distribute without an artificial delay or a round trip through the scheduler. A reservation is released when its request ends and moves with a failover, so a node stops counting as loaded as soon as it stops working. Manual pins, model-owner tiers, and the complete failover list keep their existing precedence. The broker otherwise treats the proxy as **optional and non-fatal**.++Each facade announces its bound port **asynchronously**, via an engine-addressed `ready` notification emitted once its HTTP listener is up. The broker records readiness per engine and exposes it through that engine's `<engine>-proxy:get-status` request — per engine, because the facades bind different ports and a single port for the process would be whichever readied last. Because `ready` arrives after `app:ready` (and the proxy is optional), clients learn a port by **polling** `<engine>-proxy:get-status` rather than assuming it from `app:ready`.-The proxy is the one worker that announces its bound port **asynchronously**, via a `ready` notification it emits once its HTTP listener is up. The broker captures that port and exposes it through the `proxy:get-status` request. Because `ready` arrives after `app:ready` (and the proxy is optional), clients learn the proxy's port by **polling** `proxy:get-status` rather than assuming it from `app:ready`.+The proxy is a full bidirectional JSON-RPC peer with a control plane (node selection, manual nodes, ...) and an event stream. The broker acts as a **generic relay** in both directions: any request a client sends under the `ollama-proxy:` namespace (other than the reserved broker-local ones) is forwarded to that engine's facade and the response relayed straight back (see `ollama-proxy:<method>` below), and every notification the facade emits is re-emitted to subscribed clients as `ollama-proxy:<method>` (see `ollama-proxy:<event>` below).-The proxy is a full bidirectional JSON-RPC peer with a control plane (node selection, manual nodes, ...) and an event stream. The broker acts as a **generic relay** in both directions: any request a client sends under the `proxy:` namespace (other than the reserved broker-local ones) is forwarded to the proxy with the `proxy:` prefix stripped and the response relayed straight back (see `proxy:<method>` below), and every notification the proxy emits is re-emitted to subscribed clients as `proxy:<method>` (see `proxy:<event>` below). The `proxy:` prefix is the only translation in either direction — everything after it is the proxy's own native method name.+Client namespaces are unchanged by the process collapse, but the wire inside is not. Because one process holds every facade, a message on that link carries the engine it concerns: the client's `ollama-proxy:` prefix comes off and a bare `ollama:` facade address goes on. The two are not interchangeable — `ollama-proxy:` is how a client addresses the component, `ollama:` is how a message addresses a facade inside the process — so the relay is a translation between them rather than a strip. Process-scoped methods (`log/set-level`, the scheduler's `node/set-priority`) carry no address, and `facade/enable` names its engine in the payload because it runs before that facade exists.-Two classes of proxy notification are **not** re-emitted under the `proxy:` namespace, because neither is a proxy control-plane event:+Two classes of proxy notification are **not** re-emitted under the `ollama-proxy:` namespace, because neither is a proxy control-plane event:- The `workload:*` lifecycle events the proxy fires per inference request. Those are workload-manager traffic — see the workload-manager paragraph below for how they're routed.- `node/activity`, which a proxy raises while a peer's engine is streaming response bytes back through it. That is discovery input: the broker forwards it to `nvpair-node-scanner` as a `discovery:node-activity` notification, where it counts as proof the peer is alive and cancels the eviction it would otherwise face for failing a liveness probe it had no spare CPU to answer. No client has any use for a per-request liveness frame. The handoff is a bounded queue drained by one goroutine — reports arrive for as long as inference streams, so a wedged scanner must not be able to stall the proxy reader, and a dropped report only means the scanner falls back to probing a node that will very likely answer.@@ -123,7 +133,7 @@ Two classes of proxy notification are **not** re-emitted under the `proxy:` nameShared lifecycle for all workers:-- Path resolution: the explicit `--scanner-path` / `--node-info-path` / `--proxy-path` / `--workload-manager-path` / `--cluster-manager-path` (and the analogous flags for lmstudio-proxy / errors / engine-manager / manual-nodes / settings) if set, otherwise the same-named `./<binary>` (with `.exe` on Windows) in the broker's current working directory. No PATH fallback — a clear "not found" beats a surprising stale binary. A missing scanner is fatal (it's the broker's core job); every other worker is optional.+- Path resolution: the explicit `--scanner-path` / `--node-info-path` / `--proxy-path` / `--workload-manager-path` / `--cluster-manager-path` (and the analogous flags for errors / engine-manager / manual-nodes / settings; every engine's facade lives in the single process at `--proxy-path`, with `--proxy-engines` choosing which are enabled) if set, otherwise the same-named `./<binary>` (with `.exe` on Windows) in the broker's current working directory. No PATH fallback — a clear "not found" beats a surprising stale binary. A missing scanner is fatal (it's the broker's core job); every other worker is optional.- Log level: the broker's currently resolved level is forwarded at spawn time via `--log-level <lvl>`. Runtime `log/set-level` requests are fanned out to every running child over its stdin (see `log/set-level` below).- Hidden console window on Windows (`CREATE_NO_WINDOW`).- Shutdown: when the broker exits (signal, peer EOF, or `shutdown` RPC), it first asks engine-manager to stop its engines (`engine:prepare-shutdown`) while everything is still up, then closes each running worker's stdin. The worker sees EOF and exits cleanly, and the broker waits for it to exit — no timeout and no force-kill. Each worker owns its own bounded shutdown (engine-manager bounds its engine stop internally; the HTTP workers cancel their context and drain their server on EOF), so the broker never SIGKILLs a worker mid-teardown. A grace-then-kill teardown would orphan engine processes by killing engine-manager partway through stopping them, which is why there is no timeout here.@@ -135,7 +145,7 @@ Shared lifecycle for all workers:#### `app:ready`-Emitted once on startup, after the scanner (and, when available, node-info and ollama-proxy) have been spawned and the transport is connected. Seeing `app:ready` means the broker is accepting requests **and** discovery events are flowing into its store (even if no nodes are known yet — mDNS browsing takes a moment). It does not imply node-info came up — that worker is optional and a failure to spawn it is logged but does not block `app:ready`. Nor does it imply the proxy is serving yet: the proxy reports its bound port asynchronously, so `proxy:get-status` may briefly show `ready:false` right after `app:ready`.+Emitted once on startup, after the scanner (and, when available, node-info and ollama-proxy) have been spawned and the transport is connected. Seeing `app:ready` means the broker is accepting requests **and** discovery events are flowing into its store (even if no nodes are known yet — mDNS browsing takes a moment). It does not imply node-info came up — that worker is optional and a failure to spawn it is logged but does not block `app:ready`. Nor does it imply the proxy is serving yet: the proxy reports its bound port asynchronously, so `ollama-proxy:get-status` may briefly show `ready:false` right after `app:ready`.```json{"jsonrpc":"2.0","method":"app:ready","params":{"version":"0.5.0"}}@@ -169,25 +179,25 @@ A few semantics worth knowing:- **Empty array is a valid payload.** If every previously-known node ages out, you'll get a `discovery:nodes-changed` with `"params": []`.- **Best-effort delivery.** The broker tries the notification once per mutation; transient write errors are logged but not retried.-#### `proxy:<event>`+#### `ollama-proxy:<event>`-**Opt-in.** Only delivered to a peer that has called `proxy:subscribe`; silent otherwise. Once subscribed, every notification `ollama-proxy` emits is re-emitted to the client as `proxy:<method>` — the proxy's native method name with the `proxy:` prefix, and `params` passed through verbatim (the proxy's own snake_case schema). The proxy's current notifications are:+**Opt-in.** Only delivered to a peer that has called `ollama-proxy:subscribe`; silent otherwise. Once subscribed, every notification `ollama-proxy` emits is re-emitted to the client as `ollama-proxy:<method>` — the proxy's native method name with the `ollama-proxy:` prefix, and `params` passed through verbatim (the proxy's own snake_case schema). The proxy's current notifications are:| Forwarded as | Proxy emits | Meaning ||---|---|---|-| `proxy:ready` | `ready` | The proxy bound its HTTP port (`{version, port}`). Also replayed as the baseline on a fresh subscribe. |-| `proxy:node/discovered` | `node/discovered` | A manual node was added to the proxy (relay-sourced routing targets don't emit these). |-| `proxy:node/updated` | `node/updated` | A manual node's fields changed. |-| `proxy:node/removed` | `node/removed` | A manual node was removed. |-| `proxy:node/selection-changed` | `node/selection-changed` | The active upstream changed (`{id}`; empty `id` = auto-select). |-| `proxy:proxy/request-started` | `proxy/request-started` | An HTTP request began being forwarded. |-| `proxy:proxy/request` | `proxy/request` | An HTTP request completed (status, durations). |+| `ollama-proxy:ready` | `ready` | The proxy bound its HTTP port (`{version, port}`). Also replayed as the baseline on a fresh subscribe. |+| `ollama-proxy:node/discovered` | `node/discovered` | A manual node was added to the proxy (relay-sourced routing targets don't emit these). |+| `ollama-proxy:node/updated` | `node/updated` | A manual node's fields changed. |+| `ollama-proxy:node/removed` | `node/removed` | A manual node was removed. |+| `ollama-proxy:node/selection-changed` | `node/selection-changed` | The active upstream changed (`{id}`; empty `id` = auto-select). |+| `ollama-proxy:proxy/request-started` | `proxy/request-started` | An HTTP request began being forwarded. |+| `ollama-proxy:proxy/request` | `proxy/request` | An HTTP request completed (status, durations). |```json-{"jsonrpc":"2.0","method":"proxy:node/discovered","params":{"id":"MY-GPU-BOX","host":"my-gpu-box.local.","port":11434,"addresses":["192.168.1.10"],"txt":["models=llama3"]}}+{"jsonrpc":"2.0","method":"ollama-proxy:node/discovered","params":{"id":"MY-GPU-BOX","host":"my-gpu-box.local.","port":11434,"addresses":["192.168.1.10"],"txt":["models=llama3"]}}```-Two classes of proxy notification are **not** re-emitted under the `proxy:` namespace: `errors:report` / `errors:clear` are routed into the `nvpair-errors` pipeline and surface on the `errors:update` stream (see [`errors:update`](#errorsupdate)); the proxy's `workload:*` events go to the workload-manager and surface on the separate `workloads:*` stream below.+Two classes of proxy notification are **not** re-emitted under the `ollama-proxy:` namespace: `errors:report` / `errors:clear` are routed into the `nvpair-errors` pipeline and surface on the `errors:update` stream (see [`errors:update`](#errorsupdate)); the proxy's `workload:*` events go to the workload-manager and surface on the separate `workloads:*` stream below.#### `workloads:upsert` / `workloads:remove`@@ -337,12 +347,12 @@ Response:{"jsonrpc":"2.0","id":3,"result":{"version":"0.5.0"}}```-#### `proxy:get-status`+#### `ollama-proxy:get-status`Reports whether the supervised `ollama-proxy` is up and, if so, the HTTP port it bound. `ready` is `false` and `port` is `0` until the proxy emits its `ready` notification — and stays that way if no proxy is being supervised or the proxy has crashed — so the call is always safe and never errors. Answered locally from the broker's captured state (no round-trip to the proxy).```json-{"jsonrpc":"2.0","id":8,"method":"proxy:get-status"}+{"jsonrpc":"2.0","id":8,"method":"ollama-proxy:get-status"}```Response (once the proxy is serving):@@ -353,44 +363,44 @@ Response (once the proxy is serving):`port` is the proxy's HTTP reverse-proxy listener — point an Ollama client at `http://localhost:<port>` to route inference traffic through it. The proxy comes up asynchronously, so a client typically polls this shortly after `app:ready` until `ready` is `true`.-#### `proxy:<method>` (generic relay)+#### `ollama-proxy:<method>` (generic relay)-Any request whose method starts with `proxy:` — except the reserved broker-local methods (`proxy:get-status`, `proxy:set-port`, and the subscription methods below) — is **forwarded to `ollama-proxy`** with the `proxy:` prefix stripped, and the proxy's result or error is relayed straight back to the caller unchanged. The `proxy:` prefix is the only translation: everything after it is the proxy's own native method name, and `params` pass through verbatim. This means the proxy's whole control plane is reachable through the broker without the broker enumerating individual methods. Current proxy methods include `nodes/list`, `node/select`, `node/selected`, `node/add-manual`, and `node/remove-manual` (see the `ollama-proxy` README for their params/results).+Any request whose method starts with `ollama-proxy:` — except the reserved broker-local methods (`ollama-proxy:get-status`, `ollama-proxy:set-port`, and the subscription methods below) — is **forwarded to `ollama-proxy`** with the `ollama-proxy:` prefix stripped, and the proxy's result or error is relayed straight back to the caller unchanged. The `ollama-proxy:` prefix is the only translation: everything after it is the proxy's own native method name, and `params` pass through verbatim. This means the proxy's whole control plane is reachable through the broker without the broker enumerating individual methods. Current proxy methods include `nodes/list`, `node/select`, `node/selected`, `node/add-manual`, and `node/remove-manual` (see the `ollama-proxy` README for their params/results).For example, to list the Ollama nodes the proxy currently knows and pin traffic to one:```json-{"jsonrpc":"2.0","id":10,"method":"proxy:nodes/list"}-{"jsonrpc":"2.0","id":11,"method":"proxy:node/select","params":{"id":"MY-GPU-BOX"}}+{"jsonrpc":"2.0","id":10,"method":"ollama-proxy:nodes/list"}+{"jsonrpc":"2.0","id":11,"method":"ollama-proxy:node/select","params":{"id":"MY-GPU-BOX"}}```-The responses are exactly what the proxy returns (its payloads use the proxy's own snake_case `Node` schema — `id` / `host` / `port` / `addresses` / `txt` — not the camelCase `discovery:*` shape; the `proxy:` namespace is a deliberate exception kept thin to avoid coupling).+The responses are exactly what the proxy returns (its payloads use the proxy's own snake_case `Node` schema — `id` / `host` / `port` / `addresses` / `txt` — not the camelCase `discovery:*` shape; the `ollama-proxy:` namespace is a deliberate exception kept thin to avoid coupling).Two relay-specific error cases:- If no proxy is being supervised (or it has exited), the broker replies with error `-32000` `"ollama-proxy not available"`.-- `proxy:shutdown` is **refused** with error `-32601` — the broker owns the proxy's lifecycle, so a client can't terminate it independently. Shut the broker down instead (which tears the proxy down with it).+- `ollama-proxy:shutdown` is **refused** with error `-32601` — the broker owns the proxy's lifecycle, so a client can't terminate it independently. Shut the broker down instead (which tears the proxy down with it).-#### `proxy:set-port`+#### `ollama-proxy:set-port`-**Intercepted, not relayed verbatim** — this is where the broker coordinates the engine ↔ proxy port conflict, because it's the only component that sees both. On `proxy:set-port {port}` the broker asks `nvpair-engine-manager` (`engine:get-installed`) which ports its **running** engines hold. If the requested port is free, it relays `set-port {port}` to the proxy unchanged. If a running engine already holds it, **the engine wins**: the broker bumps the proxy to the next free port, relays `set-port {effectivePort}`, and surfaces a sticky `warning` into the errors pipeline (id `ollama-proxy:port-bumped`, `action:"none"`) explaining the move; a later non-colliding `proxy:set-port` clears that warning. The conflict path **never changes an engine's port** — only the proxy is moved. The response is the proxy's own `set-port` result (`{version, port}` with the actually-bound port).+**Intercepted, not relayed verbatim** — this is where the broker coordinates the engine ↔ proxy port conflict, because it's the only component that sees both. On `ollama-proxy:set-port {port}` the broker asks `nvpair-engine-manager` (`engine:get-installed`) which ports its **running** engines hold. If the requested port is free, it relays `set-port {port}` to the proxy unchanged. If a running engine already holds it, **the engine wins**: the broker bumps the proxy to the next free port, relays `set-port {effectivePort}`, and surfaces a sticky `warning` into the errors pipeline (id `ollama-proxy:port-bumped`, `action:"none"`) explaining the move; a later non-colliding `ollama-proxy:set-port` clears that warning. The conflict path **never changes an engine's port** — only the proxy is moved. The response is the proxy's own `set-port` result (`{version, port}` with the actually-bound port).```json-{"jsonrpc":"2.0","id":9,"method":"proxy:set-port","params":{"port":11500}}+{"jsonrpc":"2.0","id":9,"method":"ollama-proxy:set-port","params":{"port":11500}}```The same check runs whenever the proxy announces a (re)bound port (its restored port on startup), so a proxy that comes back up on a port a running engine has since taken is steered to a free one automatically. Error `-32000 "ollama-proxy not available"` when no proxy is supervised.#### `lmstudio-proxy:get-status` / `lmstudio-proxy:subscribe` / `lmstudio-proxy:unsubscribe` / `lmstudio-proxy:<method>` (generic relay)-The LM Studio counterpart of the `proxy:*` surface runs the supervised `lmstudio-proxy` on compatibility port `:1234` and tracks the managed LM Studio backend on `:1235`. With managed port ownership enabled (the default), the broker identifies and moves an existing LM Studio server through engine-manager before allowing the proxy to claim `1234`; unknown owners are left untouched and force a warned proxy fallback. Disabling managed ownership preserves explicit custom backend and proxy ports. `lmstudio-proxy:get-status` reports the actual bound port; `lmstudio-proxy:subscribe` / `lmstudio-proxy:unsubscribe` opt into / out of its `lmstudio-proxy:<event>` stream; and any other `lmstudio-proxy:<method>` is relayed verbatim with the prefix stripped (`nodes/list`, `node/select`, `node/add-manual`, `node/remove-manual`, ...). `lmstudio-proxy:shutdown` is refused because the broker owns lifecycle ordering. Workload and error events feed the shared streams exactly as Ollama's do.+The LM Studio counterpart of the `ollama-proxy:*` surface runs the supervised `lmstudio-proxy` on compatibility port `:1234` and tracks the managed LM Studio backend on `:1235`. With managed port ownership enabled (the default), the broker identifies and moves an existing LM Studio server through engine-manager before allowing the proxy to claim `1234`; unknown owners are left untouched and force a warned proxy fallback. Disabling managed ownership preserves explicit custom backend and proxy ports. `lmstudio-proxy:get-status` reports the actual bound port; `lmstudio-proxy:subscribe` / `lmstudio-proxy:unsubscribe` opt into / out of its `lmstudio-proxy:<event>` stream; and any other `lmstudio-proxy:<method>` is relayed verbatim with the prefix stripped (`nodes/list`, `node/select`, `node/add-manual`, `node/remove-manual`, ...). `lmstudio-proxy:shutdown` is refused because the broker owns lifecycle ordering. Workload and error events feed the shared streams exactly as Ollama's do.-#### `proxy:subscribe`+#### `ollama-proxy:subscribe`-Opts the peer into the `proxy:<event>` stream (off by default). On a fresh subscription the broker immediately replays the proxy's last `ready` payload as a baseline `proxy:ready` (if the proxy has come up), so a subscriber learns the port without a separate `proxy:get-status`.+Opts the peer into the `ollama-proxy:<event>` stream (off by default). On a fresh subscription the broker immediately replays the proxy's last `ready` payload as a baseline `ollama-proxy:ready` (if the proxy has come up), so a subscriber learns the port without a separate `ollama-proxy:get-status`.```json-{"jsonrpc":"2.0","id":12,"method":"proxy:subscribe"}+{"jsonrpc":"2.0","id":12,"method":"ollama-proxy:subscribe"}```Response:@@ -401,12 +411,12 @@ Response:Idempotent: re-subscribing returns `{"subscribed":true}` and does not re-send a baseline. This is a reserved broker-local method — it is not forwarded to the proxy.-#### `proxy:unsubscribe`+#### `ollama-proxy:unsubscribe`-Stops the `proxy:<event>` stream for this peer. Reserved broker-local; not forwarded. Idempotent.+Stops the `ollama-proxy:<event>` stream for this peer. Reserved broker-local; not forwarded. Idempotent.```json-{"jsonrpc":"2.0","id":13,"method":"proxy:unsubscribe"}+{"jsonrpc":"2.0","id":13,"method":"ollama-proxy:unsubscribe"}```Response:@@ -477,13 +487,13 @@ Any `settings/*` request is forwarded to `nvpair-node-settings` and its responseRelayed to `nvpair-manual-nodes`. `node/add` (`{ address, name?, tls_port?, mtls? }`) registers a user-added node and probes it; `node/remove` (`{ id }`) drops it; `nodes/list` returns the tracked manual nodes. Manually added nodes also surface in the shared `discovery:get-nodes` / `discovery:nodes-changed` snapshot — the broker merges `nvpair-manual-nodes`' `node/discovered|updated|removed` into the same store the scanner feeds. A `nvpair-manual-nodes` restart loses the in-memory entries because neither that worker nor the broker persists an authoritative copy, so clients must re-add manual nodes after a restart. Error `-32000 "manual-nodes not available"` when no manual-nodes worker is supervised.-**Manual → proxy bridge.** When the broker supervises both `nvpair-manual-nodes` and a proxy, it also bridges a manual node whose engine is reachable into that proxy via `node/add-manual` (host/port from the node's per-engine status), so inference can route to it through `proxy:node/select` / `lmstudio-proxy:node/select` just like a relay-discovered node. This is per-engine: a node whose `ollama_*` status is up is bridged into `ollama-proxy` (host/port from `ollama_port`), and one whose `lmstudio_*` status is up into `lmstudio-proxy` (from `lmstudio_port`) — a node running both is bridged into both. Manual nodes are by definition the ones that never appear via the daemon's `_nvpair-node` discovery, so this explicit add is what makes them routable. The bridge tracks reachability: an engine that goes down (or a node that is removed, or whose prober crashes) is pulled back out with `node/remove-manual`. A proxy that isn't supervised → that leg is a no-op; manual nodes still appear in the discovery snapshot as before.+**Manual → proxy bridge.** When the broker supervises both `nvpair-manual-nodes` and a proxy, it also bridges a manual node whose engine is reachable into that proxy via `node/add-manual` (host/port from the node's per-engine status), so inference can route to it through `ollama-proxy:node/select` / `lmstudio-proxy:node/select` just like a relay-discovered node. This is per-engine: a node whose `ollama_*` status is up is bridged into `ollama-proxy` (host/port from `ollama_port`), and one whose `lmstudio_*` status is up into `lmstudio-proxy` (from `lmstudio_port`) — a node running both is bridged into both. Manual nodes are by definition the ones that never appear via the daemon's `_nvpair-node` discovery, so this explicit add is what makes them routable. The bridge tracks reachability: an engine that goes down (or a node that is removed, or whose prober crashes) is pulled back out with `node/remove-manual`. A proxy that isn't supervised → that leg is a no-op; manual nodes still appear in the discovery snapshot as before.#### `cluster:<method>` / `nodes:<method>` (generic relay)Any request whose method starts with `cluster:` or `nodes:` is **forwarded to `nvpair-cluster-manager`** verbatim (no prefix stripping — these are the manager's own native namespaces), and the manager's result or JSON-RPC error is relayed straight back to the caller unchanged. As with the proxy relay this means the cluster-manager's whole surface is reachable through the broker without enumerating individual methods. Current methods include `cluster:get-node-id`, `cluster:set-identity`, `cluster:create`, `cluster:invite-node`, `cluster:respond-to-invite`, `cluster:cancel-invite`, `cluster:invite-status`, `nodes:get-initial`, and `nodes:remove` (see the [`nvpair-cluster-manager` README](../nvpair-cluster-manager/README.md) for their params/results and app-defined error codes).-The cluster-manager also pushes notifications (`cluster:invite-received`, `cluster:invite-canceled`, `cluster:identity-changed`, `nodes:changed`, ...); the broker forwards them to the caller verbatim. Unlike `proxy:<event>` / `workloads:*` these are **not opt-in** — there is no `cluster:subscribe`; the events always flow.+The cluster-manager also pushes notifications (`cluster:invite-received`, `cluster:invite-canceled`, `cluster:identity-changed`, `nodes:changed`, ...); the broker forwards them to the caller verbatim. Unlike `ollama-proxy:<event>` / `workloads:*` these are **not opt-in** — there is no `cluster:subscribe`; the events always flow.If no cluster-manager is being supervised (or it has exited), the broker replies with error `-32000` `"nvpair-cluster-manager not available"`. Because pairing calls drive multi-round-trip inter-node network exchanges, the relay's per-call timeout is 30 s (vs. the proxy's 5 s).@@ -519,9 +529,9 @@ The broker shuts down on:## Running-The broker needs `nvpair-node-scanner` reachable as a sibling file in its working directory (or via `--scanner-path`); it also looks for `nvpair-node-info` (or via `--node-info-path`), `ollama-proxy` (or via `--proxy-path`), `nvpair-workload-manager` (or via `--workload-manager-path`), and `nvpair-cluster-manager` (or via `--cluster-manager-path`) the same way (plus lmstudio-proxy / errors / engine-manager / manual-nodes / settings), though all but the scanner are optional. The simplest local layout is to put all the built `.exe`s next to each other — which is exactly how the installer lays them out.+The broker needs `nvpair-node-scanner` reachable as a sibling file in its working directory (or via `--scanner-path`); it also looks for `nvpair-node-info` (or via `--node-info-path`), `nvpair-proxy` (or via `--proxy-path`), `nvpair-workload-manager` (or via `--workload-manager-path`), and `nvpair-cluster-manager` (or via `--cluster-manager-path`) the same way (plus errors / engine-manager / manual-nodes / settings), though all but the scanner are optional. The simplest local layout is to put all the built `.exe`s next to each other — which is exactly how the installer lays them out.-Quick smoke test in stdio mode (PowerShell, run from a directory that has the worker binaries — at minimum `nvpair-node-scanner`, plus `nvpair-node-info` if you want the local inventory server, `ollama-proxy` if you want the local Ollama proxy, `nvpair-workload-manager` if you want the cluster workload relay, and `nvpair-cluster-manager` if you want cluster pairing):+Quick smoke test in stdio mode (PowerShell, run from a directory that has the worker binaries — at minimum `nvpair-node-scanner`, plus `nvpair-node-info` if you want the local inventory server, `nvpair-proxy` if you want the local engine proxies, `nvpair-workload-manager` if you want the cluster workload relay, and `nvpair-cluster-manager` if you want cluster pairing):```powershell'{"jsonrpc":"2.0","id":1,"method":"discovery:get-nodes"}' | .\nvpair-ui-broker.exe@@ -536,7 +546,7 @@ You should see an `app:ready` notification followed by a `discovery:get-nodes` rPass explicit worker paths:```powershell-.\nvpair-ui-broker.exe --scanner-path C:\path\to\nvpair-node-scanner.exe --node-info-path C:\path\to\nvpair-node-info.exe --proxy-path C:\path\to\ollama-proxy.exe --workload-manager-path C:\path\to\nvpair-workload-manager.exe --errors-path C:\path\to\nvpair-errors.exe --engine-manager-path C:\path\to\nvpair-engine-manager.exe --manual-nodes-path C:\path\to\nvpair-manual-nodes.exe --settings-path C:\path\to\nvpair-node-settings.exe --cluster-manager-path C:\path\to\nvpair-cluster-manager.exe+.\nvpair-ui-broker.exe --scanner-path C:\path\to\nvpair-node-scanner.exe --node-info-path C:\path\to\nvpair-node-info.exe --proxy-path C:\path\to\nvpair-proxy.exe --workload-manager-path C:\path\to\nvpair-workload-manager.exe --errors-path C:\path\to\nvpair-errors.exe --engine-manager-path C:\path\to\nvpair-engine-manager.exe --manual-nodes-path C:\path\to\nvpair-manual-nodes.exe --settings-path C:\path\to\nvpair-node-settings.exe --cluster-manager-path C:\path\to\nvpair-cluster-manager.exe```Attach to a pre-existing endpoint:diff --git a/services/nvpair-ui-broker/advertiser.go b/services/nvpair-ui-broker/advertiser.goindex c3f2e7d..0a908ce 100644--- a/services/nvpair-ui-broker/advertiser.go+++ b/services/nvpair-ui-broker/advertiser.go@@ -13,16 +13,18 @@ import ("nvpair-shared/noderec")-const (- // defaultOllamaPort / defaultLMStudioPort are the engines' stock ports,- // used ONLY as a fallback when engine-manager can't report the real one.- // Never hardcode the advertise/health port: the product-default proxy takes- // Ollama's :11434, so a fixed :11434 would advertise the proxy- // as Ollama and make the proxy (and peers) self-forward into a loop. The- // real port is resolved per poll via localEnginePort.- defaultOllamaPort = 11434- defaultLMStudioPort = 1234+// An engine's stock port is its FacadePort in the engine table, used ONLY as a+// fallback when engine-manager cannot report the real one. Never hardcode the+// advertise or health port: the product-default proxy takes Ollama's 11434, so+// a fixed 11434 would advertise the proxy as Ollama and make the proxy — and+// peers — self-forward into a loop. The real port is resolved per poll via+// localEnginePort.+var (+ defaultOllamaPort = ollamaProxyProfile.FacadePort+ defaultLMStudioPort = lmstudioProxyProfile.FacadePort+)+const (// engineManagerHTTPPort is the fixed LAN port the broker tells// nvpair-engine-manager to serve its HTTP surface (/v1/models) on, and the port// it registers as the em service so peers' daemons can fetch this node's@@ -99,7 +101,7 @@ func (b *Broker) reconcileAdvertise(client *http.Client) {// ports differ. Equal ports mean we can't tell the engine from the proxy// (or there is no separate engine), and setting the local backend to the// proxy's own port would make the ingress forward to itself.- up := probe && proxyPort != 0 && enginePort != proxyPort && checkOllamaHealth(client, enginePort)+ up := probe && proxyPort != 0 && enginePort != proxyPort && checkEngineHealth(ollamaProxyProfile, client, enginePort)if up {b.registerService(noderec.RegisterParams{Service: noderec.ServiceOllama, Port: proxyPort})b.setProxyLocalBackend(b.getProxy(), "ollama", enginePort, true)@@ -113,13 +115,13 @@ func (b *Broker) ollamaFacadeIsPendingBackend() bool {if b.managedOllamaBackend.Load() != 0 || b.ollamaMoveInFlight.Load() {return true}- if b.ollamaBackendPort.Load() != managedOllamaFacadePort {+ if int(b.ollamaState().backendPort.Load()) != managedOllamaFacadePort {return false}// Recovery flips managed mode off before it live-rebinds the proxy away// from :11434. Keep probes gated through that interval (and indefinitely if// the rebind fails) so the proxy can never be adopted as Ollama.- return b.managedOllamaFacade.Load() || b.proxyListenPort() == managedOllamaFacadePort+ return b.ollamaState().managedFacade.Load() || b.proxyListenPort() == managedOllamaFacadePort}// runAutoAdvertiseLMStudio is the LM Studio sibling of runAutoAdvertise: it@@ -156,7 +158,7 @@ func (b *Broker) reconcileAdvertiseLMStudio(client *http.Client) {// engine-manager may be temporarily unavailable after managed setup.// Prefer the last confirmed backend, but never hand the proxy its own// listener as a local destination.- if cached := int(b.lmstudioBackendPort.Load()); cached > 0 && cached != proxyPort {+ if cached := int(b.lmstudioState().backendPort.Load()); cached > 0 && cached != proxyPort {enginePort = cached} else {enginePort = 0@@ -171,7 +173,7 @@ func (b *Broker) reconcileAdvertiseLMStudio(client *http.Client) {// cache while the proxy and engine restart together (as on the first invite),// which later makes the compatibility proxy on the facade port look like the// backend and wrongly disables managed mode.- up := probe && proxyPort != 0 && enginePort != proxyPort && checkLMStudioHealth(client, enginePort)+ up := probe && proxyPort != 0 && enginePort != proxyPort && checkEngineHealth(lmstudioProxyProfile, client, enginePort)if up {b.registerService(noderec.RegisterParams{Service: noderec.ServiceLMStudio, Port: proxyPort})b.setProxyLocalBackend(b.getLMStudioProxy(), "lmstudio", enginePort, true)@@ -237,50 +239,34 @@ func runningEnginePort(result json.RawMessage) (int, bool) {return st.Port, true}-// proxyListenPort returns the ollama-proxy's current listen port, or 0 if no-// proxy is supervised or it hasn't reported ready yet. Used to refuse-// advertising Ollama at the proxy's own port and creating a self-forward loop.-func (b *Broker) proxyListenPort() int {- if p := b.getProxy(); p != nil {- if ready, port := p.Status(); ready {+// engineProxyListenPort returns an engine proxy's current listen port, or 0 if+// it is not supervised or has not reported ready. Used to refuse advertising an+// engine at its own proxy's port, which would be a self-forward loop, and to+// keep the compatibility fallback from mistaking a proxy that moved onto the+// facade port for the engine itself.+func (b *Broker) engineProxyListenPort(profile engineProxyProfile) int {+ if p := b.engineProxyHandle(profile); p != nil {+ if ready, port := p.Status(profile.Name); ready {return port}}return 0}-// lmstudioProxyListenPort is the LM Studio sibling of proxyListenPort. It-// prevents the compatibility fallback from mistaking a proxy moved onto :1234-// for the actual engine.-func (b *Broker) lmstudioProxyListenPort() int {- if p := b.getLMStudioProxy(); p != nil {- if ready, port := p.Status(); ready {- return port- }- }- return 0+func (b *Broker) proxyListenPort() int {+ return b.engineProxyListenPort(ollamaProxyProfile)}-// checkOllamaHealth reports whether a local ollama server is answering on the-// given port. A plain GET of the root that returns 200 is ollama's liveness-// convention. The port is resolved per poll (see-// localEnginePort), not hardcoded, so the proxy is never mistaken for Ollama.-func checkOllamaHealth(client *http.Client, port int) bool {- resp, err := client.Get(fmt.Sprintf("http://localhost:%d/", port))- if err != nil {- return false- }- resp.Body.Close()- return resp.StatusCode == http.StatusOK+func (b *Broker) lmstudioProxyListenPort() int {+ return b.engineProxyListenPort(lmstudioProxyProfile)}-// checkLMStudioHealth reports whether a local LM Studio server is answering on-// the given port. LM Studio serves the OpenAI-compatible API, so a 200 from-// /v1/models is its liveness signal (the Ollama equivalent of GET /). The port-// is resolved per poll (see localEnginePort), not hardcoded, so the proxy is-// never mistaken for LM Studio.-func checkLMStudioHealth(client *http.Client, port int) bool {- resp, err := client.Get(fmt.Sprintf("http://localhost:%d/v1/models", port))+// checkEngineHealth reports whether a local engine is answering on the given+// port, by probing the path its own liveness convention uses. The port is+// resolved per poll (see localEnginePort) rather than hardcoded, so the proxy+// is never mistaken for the engine it fronts.+func checkEngineHealth(profile engineProxyProfile, client *http.Client, port int) bool {+ resp, err := client.Get(fmt.Sprintf("http://localhost:%d%s", port, profile.HealthProbePath))if err != nil {return false}diff --git a/services/nvpair-ui-broker/advertiser_test.go b/services/nvpair-ui-broker/advertiser_test.goindex e6e6a2b..a32c2cb 100644--- a/services/nvpair-ui-broker/advertiser_test.go+++ b/services/nvpair-ui-broker/advertiser_test.go@@ -51,9 +51,8 @@ func TestLMStudioFallbackNeverAdvertisesItsProxy(t *testing.T) {defer proxyClient.Close()defer proxyServer.Close()proxy := &proxyProcess{- peer: NewPeer(NewCodec(proxyClient)),- ready: true,- port: defaultLMStudioPort,+ peer: NewPeer(NewCodec(proxyClient)),+ facadeState: readyFacade(lmstudioProxyProfile.Name, defaultLMStudioPort),}go proxy.peer.Serve(nil, nil)@@ -97,13 +96,13 @@ func TestLMStudioFallbackNeverAdvertisesItsProxy(t *testing.T) {// proxy on :1234 for the backend and disables managed mode.func TestLMStudioFallbackDoesNotOverwriteKnownBackend(t *testing.T) {b := &Broker{regCache: relay.NewRegistrationCache()}- b.lmstudioBackendPort.Store(managedLMStudioBackendStart)+ b.lmstudioState().backendPort.Store(managedLMStudioBackendStart)// No engine-manager and no proxy: the fallback path that used to poison// the cache with defaultLMStudioPort.b.reconcileAdvertiseLMStudio(nil)- if got := int(b.lmstudioBackendPort.Load()); got != managedLMStudioBackendStart {+ if got := int(b.lmstudioState().backendPort.Load()); got != managedLMStudioBackendStart {t.Fatalf("backend cache = %d, want %d (fallback must not overwrite the confirmed backend)", got, managedLMStudioBackendStart)}}@@ -119,12 +118,12 @@ func TestProxyListenPortNoProxy(t *testing.T) {func TestOllamaFacadeIsPendingBackend(t *testing.T) {b := &Broker{}- b.managedOllamaFacade.Store(true)- b.ollamaBackendPort.Store(managedOllamaFacadePort)+ b.ollamaState().managedFacade.Store(true)+ b.ollamaState().backendPort.Store(managedOllamaFacadePort)if !b.ollamaFacadeIsPendingBackend() {t.Fatal("managed facade must block liveness probes while the backend still points at 11434")}- b.ollamaBackendPort.Store(11435)+ b.ollamaState().backendPort.Store(11435)if b.ollamaFacadeIsPendingBackend() {t.Fatal("liveness probes should resume after the backend moves off 11434")}@@ -139,9 +138,11 @@ func TestOllamaFacadeIsPendingBackend(t *testing.T) {}b.ollamaMoveInFlight.Store(false)- b.managedOllamaFacade.Store(false)- b.ollamaBackendPort.Store(managedOllamaFacadePort)- b.setProxy(&proxyProcess{ready: true, port: managedOllamaFacadePort})+ b.ollamaState().managedFacade.Store(false)+ b.ollamaState().backendPort.Store(managedOllamaFacadePort)+ b.setProxy(&proxyProcess{+ facadeState: readyFacade(ollamaProxyProfile.Name, managedOllamaFacadePort),+ })if !b.ollamaFacadeIsPendingBackend() {t.Fatal("recovery must keep probes blocked until the proxy vacates 11434")}diff --git a/services/nvpair-ui-broker/broker.go b/services/nvpair-ui-broker/broker.goindex 0d18957..cf43461 100644--- a/services/nvpair-ui-broker/broker.go+++ b/services/nvpair-ui-broker/broker.go@@ -6,6 +6,7 @@ package mainimport ("context""encoding/json"+ stderrors "errors""fmt""io""log"@@ -20,6 +21,7 @@ import ("nvpair-shared/appdir""nvpair-shared/applog""nvpair-shared/clustertrust"+ "nvpair-shared/engines""nvpair-shared/errors""nvpair-shared/nodeid""nvpair-shared/noderec"@@ -131,7 +133,7 @@ type SubscriptionResult struct {Subscribed bool `json:"subscribed"`}-// ProxyStatusResult is the response to "proxy:get-status". Ready is false+// ProxyStatusResult is the response to "ollama-proxy:get-status". Ready is false// (and Port 0) until the supervised ollama-proxy has emitted its "ready"// notification — or always, if no proxy is being supervised. Clients poll// this to learn where the local proxy is listening, since the proxy is@@ -145,31 +147,32 @@ type ProxyStatusResult struct {// client connection in listen mode so future per-session caches (auth// tokens, watched-resource cursors, etc.) don't bleed across clients.type Broker struct {- codec *Codec- cancel context.CancelFunc- startedAt time.Time- nodeID string- scannerPath string- nodeInfoPath string- proxyPath string- lmstudioProxyPath string- workloadMgrPath string- errorsPath string- engineMgrPath string- manualNodesPath string- settingsPath string- clusterMgrPath string- schedulerPath string- clusterDir string+ codec *Codec+ cancel context.CancelFunc+ startedAt time.Time+ nodeID string+ scannerPath string+ nodeInfoPath string+ proxyPath string+ proxyEngines []string+ workloadMgrPath string+ errorsPath string+ engineMgrPath string+ manualNodesPath string+ settingsPath string+ clusterMgrPath string+ schedulerPath string+ clusterDir string// Managed-port state is prepared before proxy startup and read by the proxy// supervisor/reader goroutines. Ollama commits its pending backend move after// its proxy reserves :11434; LM Studio moves through engine-manager first,// then releases its gate after the proxy owns :1234 and has its backend.- managedOllamaFacade atomic.Bool- managedOllamaBackend atomic.Int32- ollamaMoveInFlight atomic.Bool- ollamaBackendPort atomic.Int32- ollamaProxyStartupPort atomic.Int32+ //+ // managedFacade, backendPort and startupPort now live per engine on+ // engineProxyRuntime. What remains here is Ollama's deferred-move state,+ // which only an adopted engine has.+ managedOllamaBackend atomic.Int32+ ollamaMoveInFlight atomic.Bool// ollamaHostAlias is one synchronized snapshot because engine-manager and// proxy supervisors can respawn while startup is still committing (or// rolling back) the inherited, local-http OLLAMA_HOST alias.@@ -181,9 +184,6 @@ type Broker struct {ollamaHostAliasError *errors.ServiceErrorollamaPortReady chan struct{}ollamaPortReadyOnce sync.Once- managedLMStudioFacade atomic.Bool- lmstudioBackendPort atomic.Int32- lmstudioProxyStartupPort atomic.Int32lmstudioProxyGeneration atomic.Uint64lmstudioProxyPublishedGeneration atomic.Uint64lmstudioPortReady chan struct{}@@ -208,33 +208,38 @@ type Broker struct {// goroutine) while request handlers and the log-level fan-out read// them (on the read-loop / reader goroutines). Every access therefore// goes through the get*/set* helpers under this lock.- workersMu sync.Mutex- scanner *scannerProcess- nodeInfo *nodeInfoProcess- proxy *proxyProcess- lmstudioProxy *proxyProcess- workloadMgr *workloadManagerProcess- errorsProc *errorsProcess- engineMgr *rpcWorker- manualNodes *rpcWorker- settings *rpcWorker- clusterMgr *clusterManagerProcess- scheduler *rpcWorker+ workersMu sync.Mutex+ // serveCtx is the broker's own lifetime, recorded so work started on a+ // spawn goroutine can be abandoned when the broker is shutting down.+ // Guarded by workersMu because a respawn reads it off the monitor+ // goroutine. See proxyBringUpContext.+ serveCtx context.Context+ scanner *scannerProcess+ nodeInfo *nodeInfoProcess+ workloadMgr *workloadManagerProcess+ errorsProc *errorsProcess+ engineMgr *rpcWorker+ manualNodes *rpcWorker+ settings *rpcWorker+ clusterMgr *clusterManagerProcess+ scheduler *rpcWorker// Per-worker supervisors own the (re)spawn + crash-detect + restart// lifecycle of each worker. nil when the worker's binary wasn't resolved// (or its first spawn failed).- scannerSup *supervisor- nodeInfoSup *supervisor- proxySup *supervisor- lmstudioProxySup *supervisor- workloadMgrSup *supervisor- errorsSup *supervisor- engineMgrSup *supervisor- manualNodesSup *supervisor- settingsSup *supervisor- clusterMgrSup *supervisor- schedulerSup *supervisor+ //+ // proxySup covers every engine: one nvpair-proxy process hosts a facade per+ // enabled engine, so there is no per-engine proxy supervisor.+ scannerSup *supervisor+ nodeInfoSup *supervisor+ proxySup *supervisor+ workloadMgrSup *supervisor+ errorsSup *supervisor+ engineMgrSup *supervisor+ manualNodesSup *supervisor+ settingsSup *supervisor+ clusterMgrSup *supervisor+ schedulerSup *supervisor// subMu guards subscribed. The discovery:nodes-changed stream is// opt-in: emitNodesChanged (called on the scanner-event goroutine)@@ -244,14 +249,16 @@ type Broker struct {subMu sync.Mutexsubscribed bool- // proxyMu guards proxySubscribed and lmstudioProxySubscribed. The- // proxy:<event> / lmstudio-proxy:<event> streams are opt-in like- // discovery's: the forward*Notification hooks (on each proxy's reader- // goroutine) read the flags while the *:subscribe / *:unsubscribe- // handlers (on the read-loop goroutine) flip them.- proxyMu sync.Mutex- proxySubscribed bool- lmstudioProxySubscribed bool+ // proxyMu guards every engineProxyRuntime.subscribed. The+ // <namespace>:<event> streams are opt-in like discovery's: the forward+ // hooks (on each proxy's reader goroutine) read the flag while the+ // subscribe / unsubscribe handlers (on the read-loop goroutine) flip it.+ proxyMu sync.Mutex++ // engineProxies holds per-engine proxy state, one entry per engine in the+ // descriptor table. Built on first use; see engineProxy.+ engineProxiesOnce sync.Once+ engineProxies map[string]*engineProxyRuntime// workloadsMu guards workloadsSubscribed. The workloads:* stream is// opt-in too: emitWorkloadEvent (called on the proxy reader goroutine@@ -306,13 +313,21 @@ type Broker struct {manualNodeKeys map[string]stringmanualNodeStatuses map[string]manualNodeStatusEntry- // schedMu guards each engine's cached priority and generation. Per-engine- // delivery locks serialize asynchronous node/set-priority calls; a stale- // generation is skipped before it can overwrite a newer proxy order.+ // schedMu guards the cached ranking and its generation.+ //+ // One cache, not one per engine: the scheduler computes a single node-wide+ // ranking and emits it once per engine, so per-engine copies would hold+ // identical content and prevent the duplicate from being recognized as one.+ // havePriority distinguishes "no ranking yet" from a cached empty one, which+ // is a legitimate state meaning the scheduler has no influence.+ //+ // priorityApplyMu serializes asynchronous node/set-priority deliveries; a+ // stale generation is skipped before it can overwrite a newer proxy order.schedMu sync.Mutex- lastPriority map[string]schedulerwire.Priority- priorityGeneration map[string]uint64- priorityApplyMu map[string]*sync.Mutex+ lastPriority schedulerwire.Priority+ havePriority bool+ priorityGeneration uint64+ priorityApplyMu sync.Mutex// schedulerFeedMu serializes scheduler initialization with live discovery// and workload fanout. A restarted scheduler receives its active workload@@ -326,17 +341,19 @@ type Broker struct {// required; every other field is optional (an empty string means the// broker won't spawn that worker because its binary couldn't be resolved).type workerPaths struct {- scanner string- nodeInfo string- proxy string- lmstudioProxy string- workloadMgr string- errors string- engineMgr string- manualNodes string- settings string- clusterMgr string- scheduler string+ scanner string+ nodeInfo string+ // proxy is the single nvpair-proxy binary, started once per entry in+ // proxyEngines and then asked for that engine's facade via facade/enable.+ proxy string+ proxyEngines []string+ workloadMgr string+ errors string+ engineMgr string+ manualNodes string+ settings string+ clusterMgr string+ scheduler string// clusterDir is the cluster-manager config dir (node.crt/node.key +// trusted/). Threaded to every worker that does cluster-scoped inter-node// mTLS so they serve/dial pinned peers once this node joins a cluster.@@ -368,7 +385,7 @@ func NewBroker(codec *Codec, paths workerPaths) *Broker {scannerPath: paths.scanner,nodeInfoPath: paths.nodeInfo,proxyPath: paths.proxy,- lmstudioProxyPath: paths.lmstudioProxy,+ proxyEngines: paths.proxyEngines,workloadMgrPath: paths.workloadMgr,errorsPath: paths.errors,engineMgrPath: paths.engineMgr,@@ -441,15 +458,11 @@ func (b *Broker) getNodeInfo() *nodeInfoProcess {}func (b *Broker) setProxy(p *proxyProcess) {- b.workersMu.Lock()- b.proxy = p- b.workersMu.Unlock()+ b.setEngineProxyHandle(ollamaProxyProfile, p)}func (b *Broker) getProxy() *proxyProcess {- b.workersMu.Lock()- defer b.workersMu.Unlock()- return b.proxy+ return b.engineProxyHandle(ollamaProxyProfile)}func (b *Broker) setWorkloadMgr(m *workloadManagerProcess) {@@ -634,43 +647,328 @@ func (b *Broker) spawnNodeInfo() (supervisedHandle, error) {return np, nil}-func (b *Broker) spawnProxy() (supervisedHandle, error) {- var args []string- if port := int(b.ollamaProxyStartupPort.Load()); port != 0 {- args = []string{"--port", fmt.Sprintf("%d", port), "--ignore-persisted-port"}+// proxyArgs builds the child's argv. Only process-scoped flags remain: which+// engines it fronts, on which ports, and with which alias addresses all travel+// in facade/enable, because the broker plans a different port per engine and a+// single-valued flag cannot carry two plans.+func (b *Broker) proxyArgs() []string {+ // Thread the cluster dir so the proxy can bring up its pin-gated LAN mTLS+ // ingress (and dial peers over mTLS) once this node is clustered; empty/+ // absent certs leave it loopback-plaintext only.+ return b.clusterDirArgs()+}++// ollamaFacadeSpec is the enable request for Ollama's facade under one alias+// snapshot. Taken as a parameter rather than read here so the caller keeps the+// generation bump — beginOllamaProxyGeneration has a side effect — and so the+// spec is assertable on its own.+func (b *Broker) ollamaFacadeSpec(alias ollamaHostAlias) enableFacadeRequest {+ spec := enableFacadeRequest{Engine: ollamaProxyProfile.Name}+ if port := int(b.ollamaState().startupPort.Load()); port != 0 {+ spec.Port = port+ spec.IgnorePersistedPort = true}- generation, alias := b.beginOllamaProxyGeneration()if alias.Address != "" {- args = append(args, "--alias-address", alias.Address)+ spec.AliasAddresses = append(spec.AliasAddresses, alias.Address)}if alias.AlternateAddress != "" {- args = append(args, "--alias-address", alias.AlternateAddress)+ spec.AliasAddresses = append(spec.AliasAddresses, alias.AlternateAddress)}- // Thread the cluster dir so the proxy can bring up its pin-gated LAN mTLS- // ingress (and dial peers over mTLS) once this node is clustered; empty/- // absent certs leave it loopback-plaintext only.- args = append(args, b.clusterDirArgs()...)- pp, err := startProxy("proxy", b.proxyPath, applog.LevelString(), b.relayDir,+ return spec+}++// setServeCtx records the broker's own lifetime so work started on a spawn+// goroutine can be abandoned when the broker is shutting down.+func (b *Broker) setServeCtx(ctx context.Context) {+ b.workersMu.Lock()+ defer b.workersMu.Unlock()+ b.serveCtx = ctx+}++func (b *Broker) getServeCtx() context.Context {+ b.workersMu.Lock()+ defer b.workersMu.Unlock()+ return b.serveCtx+}++// proxyBringUpContext bounds facade bring-up and abandons it the moment either+// the broker or the proxy supervisor starts shutting down.+//+// This exists because spawn stopped being instantaneous. It used to fork a+// process and return; it now issues facade/enable RPCs — up to two per engine,+// each bounded by proxyCallTimeout — and supervisor.Stop blocks on the spawn+// returning with no timeout of its own. A crash-looping proxy whose fresh child+// never answers could therefore spend the whole teardownBudget inside the+// proxy's join, and graceFor would clip every worker behind it, including+// engine-manager's stop grace. stopwait.go records that cutting that grace+// short is what orphaned engines.+//+// Both signals are needed, not either one. Stopping() covers a respawn, but the+// *first* spawn runs synchronously inside Start on the Serve goroutine — Serve+// cannot reach shutdownInferenceStack to call Stop until that spawn returns, so+// Stopping() can never close during it. The Serve context is what interrupts+// that case.+//+// The caller must cancel, as with any context: shutdown is one way out, a+// completed bring-up is the other.+func (b *Broker) proxyBringUpContext() (context.Context, context.CancelFunc) {+ parent := b.getServeCtx()+ if parent == nil {+ parent = context.Background()+ }+ ctx, cancel := context.WithCancel(parent)++ sup := b.proxySup+ if sup == nil {+ return ctx, cancel+ }+ go func() {+ select {+ case <-sup.Stopping():+ cancel()+ case <-ctx.Done():+ }+ }()+ return ctx, cancel+}++// enableEngineFacade brings up one engine's facade with that engine's port plan+// and fallback policy. The alias snapshot is only meaningful to an engine with+// an inherited host variable, so the others ignore it.+func (b *Broker) enableEngineFacade(+ ctx context.Context, pp *proxyProcess, profile engineProxyProfile, alias ollamaHostAlias,+) error {+ switch profile.Name {+ case ollamaProxyProfile.Name:+ return b.enableProxyFacadeWithFallback(ctx, pp, b.ollamaFacadeSpec(alias), b.ollamaFallbackPort)+ case lmstudioProxyProfile.Name:+ return b.enableProxyFacadeWithFallback(ctx, pp, b.lmstudioFacadeSpec(), b.lmstudioFallbackPort)+ default:+ return fmt.Errorf("no facade spec for engine %q", profile.Name)+ }+}++// blockAndFinishEngineProxy gives up one engine's managed claim, reports it, and+// releases its startup gate — the treatment for an engine whose facade will not+// come up in a process that is otherwise staying.+//+// The block and the finish are both needed. Blocking gives up the compatibility+// port and plans a fallback but deliberately leaves the OLLAMA_HOST alias+// reserved, because most of its callers still have a live facade serving that+// alias. Finishing is what returns the alias, which is only correct once this+// engine is known not to be coming up.+func (b *Broker) blockAndFinishEngineProxy(profile engineProxyProfile) {+ switch profile.Name {+ case ollamaProxyProfile.Name:+ if b.ollamaState().managedFacade.Load() {+ b.blockManagedOllamaFacade("the Ollama proxy facade could not be brought up")+ }+ b.finishOllamaProxyTerminal()+ case lmstudioProxyProfile.Name:+ if b.lmstudioState().managedFacade.Load() {+ _, _ = b.blockManagedLMStudioFacade("the LM Studio proxy facade could not be brought up", nil)+ }+ b.finishLMStudioProxyTerminal()+ default:+ slog.Warn("no terminal handling for engine", "engine", profile.Name)+ }+}++// facadeCameUpAnyway reports whether a facade that failed to answer its enable+// nevertheless announced itself ready.+//+// Only for **transport-level ambiguity** — a timeout or a lost response, where+// the child does the bind, two notification writes and the serve inside that+// call and may genuinely be serving. Disowning it then would strand a listener+// on a port the broker reports as unavailable, with no facade/disable to+// withdraw it, so the facade's own ready notification gets the deciding vote.+//+// Deliberately not applied to an explicit rejection. A rejection means the+// child decided against the facade and withdrew it, and a recorded ready from+// earlier in the same attempt is then stale: honouring it would publish a dead+// port and handle, and the broker would never retry because it believes the+// facade is up. errFacadeBindFailed is the clearest case — the child deleted+// the facade before answering — but the rule holds for any answered error.+func facadeCameUpAnyway(pp *proxyProcess, engine string, enableErr error) bool {+ if !isTransportAmbiguous(enableErr) {+ return false+ }+ ready, port := pp.Status(engine)+ if !ready || port <= 0 {+ return false+ }+ slog.Warn("facade enable did not answer, but the facade reported ready; keeping it",+ "engine", engine, "port", port, "err", enableErr)+ return true+}++// isTransportAmbiguous reports whether an enable failure leaves the child's+// actual state unknown, as opposed to the child having answered.+//+// errRPCAnswered wraps every case where a response arrived, so the absence of+// it is what identifies a timeout, a closed pipe, or a cancelled call.+func isTransportAmbiguous(err error) bool {+ return err != nil && !stderrors.Is(err, errRPCAnswered)+}++// ollamaFallbackPort is the port to retry an Ollama facade on after a lost bind+// race.+//+// The child's bind-failed notification precedes the enable response on the same+// stream, and forwardProxyNotification handles it inline — so by the time the+// response is dispatched, that path has usually already chosen a fallback and+// reported the ownership block to the UI. Prefer its choice: it also decides+// whether the managed facade is off, which a port picked here would not.+//+// Recomputed only when that did not happen, because the notification is+// best-effort on the child side.+func (b *Broker) ollamaFallbackPort(failed int) int {+ if planned := int(b.ollamaState().startupPort.Load()); planned != 0 && planned != failed {+ return planned+ }+ return b.setOllamaProxyFallback(failed)+}++// spawnProxy starts the one nvpair-proxy process and brings up a facade in it+// for every enabled engine.+//+// One process for all of them is what makes the scheduler's picture whole: the+// estimated-work reservations a proxy takes between scheduler snapshots live in+// the process, so two processes each kept half the story and could send+// simultaneous bursts to the same node believing it idle.+//+// A facade that fails to come up is logged and skipped, not fatal. The spawn+// only fails if no engine came up at all, because a process with no listener+// serves nobody and should be retried by the supervisor rather than left+// running.+func (b *Broker) spawnProxy() (supervisedHandle, error) {+ // Both generations are captured before the spawn and advance together: they+ // are per-incarnation stale-notification filters, and with one process the+ // facades are re-created as a set. Each engine keeps its own counter so its+ // existing port choreography is untouched.+ ollamaGeneration, alias := b.beginOllamaProxyGeneration()+ // Bumped without lmstudioReadyMu. Reconciliation holds that mutex across+ // its set-port and engine:status round trips, so taking it here would make+ // a replacement spawn wait out the dead process's RPC timeouts before it+ // could even start. The generation itself is atomic, and+ // lmstudioProxyGenerationIsCurrent is what drops the stale run.+ lmstudioGeneration := b.lmstudioProxyGeneration.Add(1)++ pp, err := startProxy(engines.ProxyComponent, b.proxyPath, applog.LevelString(), b.relayDir,func(method string, params json.RawMessage) {- b.forwardProxyNotificationForGeneration(generation, method, params)- }, args...)+ b.forwardProxyProcessNotification(ollamaGeneration, lmstudioGeneration, method, params)+ }, b.proxyArgs()...)if err != nil {return nil, err}- b.setProxy(pp)- // The child can emit ready before startProxy returns and before setProxy- // publishes the handle. Recover that narrow race after publication; the- // pending backend port is consumed atomically, so a concurrent notification- // can safely perform the same reconciliation.- if b.managedOllamaFacade.Load() {- if ready, port := pp.Status(); ready && port > 0 {- go b.reconcileProxyPortOnReady(port)+ // Published for every engine before any facade is enabled, so a facade's+ // own ready notification finds the handle it belongs to.+ for _, profile := range engineProxyProfiles {+ if b.proxyEnabled(profile) {+ b.setEngineProxyHandle(profile, pp)}}+ b.lmstudioProxyPublishedGeneration.Store(lmstudioGeneration)slog.Info("proxy started", "path", b.proxyPath, "pid", pp.cmd.Process.Pid)++ // The child is serving its JSON-RPC channel but has no listeners yet, so+ // ask for each facade before reporting the worker up. Synchronous, on the+ // spawn goroutine, so a caller that treats spawn success as "the engines+ // are being brought up" stays correct.+ bringUp, cancelBringUp := b.proxyBringUpContext()+ defer cancelBringUp()++ enabled := 0+ var failed []engineProxyProfile+ for _, profile := range engineProxyProfiles {+ if !b.proxyEnabled(profile) {+ continue+ }+ err := b.enableEngineFacade(bringUp, pp, profile, alias)+ switch {+ case err == nil:+ enabled+++ case facadeCameUpAnyway(pp, profile.Name, err):+ enabled+++ default:+ slog.Warn("facade enable failed; continuing without a local proxy for it",+ "engine", profile.Name, "err", err)+ failed = append(failed, profile)+ }+ }+ if enabled == 0 {+ // The whole spawn failed, so the supervisor will retry it. Leave every+ // engine's managed claim and its OLLAMA_HOST alias reservation exactly+ // as they are: the retry needs them, and releasing the alias here would+ // let engine-manager take that port in between, so a retry that would+ // otherwise have succeeded comes up without the inherited endpoint.+ //+ // A first-spawn failure and an exhausted restart budget are the paths+ // that give up for good, and the supervisor owns both.+ pp.Stop()+ for _, profile := range engineProxyProfiles {+ b.setEngineProxyHandle(profile, nil)+ }+ // The startup gates open anyway, because they are a separate concern+ // from the claim: the alias reservation has to survive a retry, the+ // gates must not wait for one. waitForManagedPortOwnership is unbounded+ // apart from the Serve context and the gates also hold client+ // requests, so keeping them shut across a 1→16s backoff budget defers+ // desired-state restore and makes every engine:status wait out its call+ // timeout and answer "retry" — the exact symptom finishOllamaProxyTerminal+ // exists to prevent. A later successful spawn re-derives the real state.+ b.markOllamaPortReady()+ b.markLMStudioPortReady()+ return nil, fmt.Errorf("no engine facade could be brought up in %s", b.proxyPath)+ }++ // At least one facade is serving, so this process incarnation is staying+ // and will not be retried. Only now is a failed engine terminal for it.+ for _, profile := range failed {+ b.setEngineProxyHandle(profile, nil)+ b.blockAndFinishEngineProxy(profile)+ }++ // A facade can announce ready before its handle was published, so replay+ // the reconciliation now. Both gates are sync.Once-guarded, which makes+ // this safe when the notification goroutine already handled it.+ if b.ollamaState().managedFacade.Load() {+ if ready, port := pp.Status(ollamaProxyProfile.Name); ready && port > 0 {+ go b.reconcileProxyPortOnReady(port)+ }+ }+ if ready, port := pp.Status(lmstudioProxyProfile.Name); ready && port > 0 {+ go b.reconcileLMStudioProxyPortOnReadyForGeneration(lmstudioGeneration, port)+ }return pp, nil}+// forwardProxyProcessNotification routes one proxy process's notifications to+// the engine each belongs to.+//+// A facade-scoped notification is addressed, so it goes to that engine's+// handler with that engine's generation. A process-scoped one is unaddressed+// and must be handled exactly once, not once per engine — routing a workload+// record through both handlers would record every job twice.+func (b *Broker) forwardProxyProcessNotification(+ ollamaGeneration, lmstudioGeneration uint64, method string, params json.RawMessage,+) {+ engine, bare := engines.SplitAddressedMethod(method)+ switch engine {+ case ollamaProxyProfile.Name:+ b.forwardProxyNotificationForGeneration(ollamaGeneration, method, params)+ case lmstudioProxyProfile.Name:+ b.forwardLMStudioProxyNotificationForGeneration(lmstudioGeneration, method, params)+ case "":+ if !b.routeProcessScopedProxyNotification(bare, params) {+ slog.Debug("ignoring unaddressed proxy notification", "method", bare)+ }+ default:+ slog.Warn("proxy addressed a notification to an unknown engine",+ "engine", engine, "method", bare)+ }+}+func (b *Broker) spawnWorkloadManager() (supervisedHandle, error) {wm, err := startWorkloadManager(b.workloadMgrPath, applog.LevelString(), b.relayDir, b.forwardWorkloadManagerNotification, b.clusterDirArgs()...)if err != nil {@@ -1406,11 +1704,16 @@ func (b *Broker) forwardSchedulerNotification(method string, params json.RawMessslog.Warn("bad schedule:priority payload", "err", err)return}- generation := b.cachePrioritySnapshot(p.Engine, p.Snapshot())+ generation, fresh := b.cachePrioritySnapshot(p.Snapshot())+ if !fresh {+ // The scheduler's duplicate emission of one recompute. Dropped here,+ // before it mints a generation, so nothing downstream sees it at all.+ return+ }// Dispatch off the reader goroutine: the node/set-priority round-trip// blocks on the proxy's response, so calling it inline would stall the// scheduler's notification stream.- go b.applyPriorityToProxy(p.Engine, generation)+ go b.applyPriority(generation)return}slog.Debug("ignoring scheduler notification", "method", method)@@ -1440,99 +1743,174 @@ func (b *Broker) forwardNodeInfoNotification(method string, params json.RawMessago sp.pushObservedAddresses(p.Addresses)}-// cachePrioritySnapshot records the newest pending-aware snapshot and returns-// its monotonically increasing per-engine generation.-func (b *Broker) cachePrioritySnapshot(engine string, priority schedulerwire.Priority) uint64 {+// cachePrioritySnapshot records a new ranking and returns its generation, or+// reports fresh=false when the ranking is byte-identical to the cached one.+//+// The cache is process-wide, not per engine, and that is what makes the dedupe+// work: one scheduler recompute emits the same ranking once per engine+// (schedule.go computes one order and loops the engine list), so the second+// frame is a duplicate that differs only in a routing key the proxy never sees.+// Comparing per engine would never match those two against each other.+//+// Dropping the duplicate here, before it mints a generation, is deliberate.+// The alternative — comparing content at delivery — silently breaks+// repushPriority, whose whole job is to re-send an unchanged ranking to a proxy+// that lost its state, and which is therefore always byte-identical to the last+// delivery.+func (b *Broker) cachePrioritySnapshot(priority schedulerwire.Priority) (generation uint64, fresh bool) {b.schedMu.Lock()defer b.schedMu.Unlock()- if b.lastPriority == nil {- b.lastPriority = make(map[string]schedulerwire.Priority)+ if b.havePriority && b.lastPriority.SameRanking(priority) {+ return b.priorityGeneration, false}- if b.priorityGeneration == nil {- b.priorityGeneration = make(map[string]uint64)- }- b.lastPriority[engine] = priority.Clone()- b.priorityGeneration[engine]++- return b.priorityGeneration[engine]+ b.lastPriority = priority.Clone()+ b.havePriority = true+ b.priorityGeneration+++ b.lastPriority.Generation = b.priorityGeneration+ return b.priorityGeneration, true}-// proxyForEngine returns the supervised proxy that serves an engine, or nil.-func (b *Broker) proxyForEngine(engine string) *proxyProcess {- switch engine {- case "ollama":- return b.getProxy()- case "lmstudio":- return b.getLMStudioProxy()- default:- return nil+// applyPriority delivers one cached generation to every live proxy. A stale+// generation is silently skipped; an engine with no proxy is a logged no-op and+// picks the snapshot up from the replay on its next spawn.+//+// It fans out rather than targeting one engine because the ranking is node-wide+// and the cache is now process-wide: with the duplicate emission dropped at+// ingestion, the surviving frame is whichever engine the scheduler emitted+// first, so delivering only to that engine's proxy would starve the other.+func (b *Broker) applyPriority(generation uint64) {+ b.deliverPrioritySnapshot(generation, func(priority schedulerwire.Priority) {+ b.sendPriority(context.Background(), priority)+ })+}++// sendPriority delivers one ranking to every live proxy. Shared by the+// scheduler's gated fan-out and the ungated spawn replay, so both agree on how+// a delivery is made and only differ on when one is allowed.+func (b *Broker) sendPriority(ctx context.Context, priority schedulerwire.Priority) {+ params, err := json.Marshal(priority)+ if err != nil {+ slog.Warn("marshal node/set-priority failed", "err", err)+ return}+ for _, target := range b.livePriorityTargets() {+ if _, rpcErr, err := target.proxy.Call(ctx, "node/set-priority", params); err != nil {+ slog.Warn("node/set-priority call failed", "engine", target.engine, "err", err)+ } else if rpcErr != nil {+ slog.Warn("node/set-priority rejected", "engine", target.engine,+ "code", rpcErr.Code, "msg", rpcErr.Message)+ }+ }+}++// priorityTarget is one proxy to deliver a ranking to, with the engine name+// kept for log context.+type priorityTarget struct {+ engine string+ proxy *proxyProcess}-// applyPriorityToProxy sends one cached generation to the engine's proxy. An-// absent proxy is a logged no-op; a stale generation is silently skipped.-func (b *Broker) applyPriorityToProxy(engine string, generation uint64) {- b.deliverPrioritySnapshot(engine, generation, func(priority schedulerwire.Priority) {- p := b.proxyForEngine(engine)+// livePriorityTargets returns the distinct live proxies to deliver a ranking to.+//+// Distinct by handle, not by engine: once one process hosts every facade, two+// engine entries resolve to the same *proxyProcess, and calling it once per+// engine would deliver the same generation twice. The child treats the repeat as+// an acknowledged no-op, but sending it is still pointless work on the request+// path's critical peer.+func (b *Broker) livePriorityTargets() []priorityTarget {+ var out []priorityTarget+ seen := make(map[*proxyProcess]bool)+ for _, profile := range engineProxyProfiles {+ p := b.engineProxyHandle(profile)if p == nil {- slog.Debug("no proxy for engine; priority snapshot cached for later", "engine", engine)- return- }- params, err := json.Marshal(priority)- if err != nil {- slog.Warn("marshal node/set-priority failed", "engine", engine, "err", err)- return+ slog.Debug("no proxy for engine; ranking arrives on its next spawn", "engine", profile.Name)+ continue}- if _, rpcErr, err := p.Call(context.Background(), "node/set-priority", params); err != nil {- slog.Warn("node/set-priority call failed", "engine", engine, "err", err)- } else if rpcErr != nil {- slog.Warn("node/set-priority rejected", "engine", engine, "code", rpcErr.Code, "msg", rpcErr.Message)+ if seen[p] {+ continue}- })+ seen[p] = true+ out = append(out, priorityTarget{engine: profile.Name, proxy: p})+ }+ return out}-// deliverPrioritySnapshot serializes one engine's deliveries and invokes-// deliver only if generation is still current. It is split from the proxy call-// so ordering can be verified without a subprocess.-func (b *Broker) deliverPrioritySnapshot(engine string, generation uint64, deliver func(schedulerwire.Priority)) {- applyMu := b.priorityApplyMutex(engine)- applyMu.Lock()- defer applyMu.Unlock()+// deliverPrioritySnapshot serializes deliveries and invokes deliver only if+// generation is still current. It is split from the proxy call so ordering can+// be verified without a subprocess.+func (b *Broker) deliverPrioritySnapshot(generation uint64, deliver func(schedulerwire.Priority)) {+ b.priorityApplyMu.Lock()+ defer b.priorityApplyMu.Unlock()b.schedMu.Lock()- if b.priorityGeneration[engine] != generation {+ if b.priorityGeneration != generation {b.schedMu.Unlock()return}- priority := b.lastPriority[engine].Clone()+ priority := b.lastPriority.Clone()b.schedMu.Unlock()deliver(priority)}-func (b *Broker) priorityApplyMutex(engine string) *sync.Mutex {- b.schedMu.Lock()- defer b.schedMu.Unlock()- if b.priorityApplyMu == nil {- b.priorityApplyMu = make(map[string]*sync.Mutex)- }- if b.priorityApplyMu[engine] == nil {- b.priorityApplyMu[engine] = &sync.Mutex{}- }- return b.priorityApplyMu[engine]+// replayProxyStateAfterSpawn restores the broker-pushed state a fresh proxy+// process does not have. It is wired to every proxy supervisor's onSpawned,+// which is the only event that loses that state — a set-port rebind keeps the+// child's in-process state, so it needs no replay.+//+// The spawn hook rather than the proxy's own "ready" notification, for a+// specific reason: spawn publishes the handle before returning, so a replay+// from here is guaranteed to find it. A replay triggered by "ready" races that+// publication, because the child can emit ready before the broker has stored+// the handle — and a replay that misses the handle is dropped by the generation+// check and never retried, leaving the new child with no ranking at all.+func (b *Broker) replayProxyStateAfterSpawn() {+ // Abandoned once teardown starts, which the onSpawned field comment+ // requires of any callback that talks to the worker: this one calls the+ // proxy, and a teardown is about to close that worker's stdin. It runs on+ // its own goroutine so it cannot stall Stop, but there is no reason to+ // spend a call timeout re-seeding a child that is going away.+ ctx, cancel := b.proxyBringUpContext()+ defer cancel()+ b.repushPriority(ctx)}-// repushPriority re-sends the cached priority snapshot for an engine after its-// proxy (re)spawns. A no-op when nothing has been cached yet.-func (b *Broker) repushPriority(engine string) {+// repushPriority re-sends the cached ranking after a proxy (re)spawns, because+// a fresh child starts with no scheduler state. A no-op when nothing has been+// cached yet.+//+// It reuses the current generation rather than minting one, so the child's+// idempotency check makes a repush to a proxy that never lost its state+// harmless — which is what lets several spawn paths call this without+// coordinating.+func (b *Broker) repushPriority(ctx context.Context) {+ // Serialized against the scheduler's own deliveries, but deliberately not+ // gated on a generation.+ //+ // The gate stops an older generation from overwriting a newer delivery,+ // which is right for the scheduler's fan-out and wrong here. A replay races+ // a snapshot that arrived while no handle was published: that snapshot found+ // no targets, and gating the replay on the generation it read then discards+ // it as stale, so neither is delivered. The proxy is left with an empty+ // ranking, reserveCandidate returns early, and no reservations are taken at+ // all — the behaviour the shared map exists to provide — until the next+ // genuine ranking change.+ //+ // Unconditional is cheap: the child applies at most one snapshot per+ // generation, so a repush to a proxy that never lost its state costs one+ // ignored call.+ b.priorityApplyMu.Lock()+ defer b.priorityApplyMu.Unlock()+b.schedMu.Lock()- _, ok := b.lastPriority[engine]- generation := b.priorityGeneration[engine]+ have := b.havePriority+ priority := b.lastPriority.Clone()b.schedMu.Unlock()- if !ok {+ if !have {return}- b.applyPriorityToProxy(engine, generation)+ b.sendPriority(ctx, priority)}// onErrorsUpdate relays an errors:update snapshot from nvpair-errors straight@@ -1617,6 +1995,7 @@ func (b *Broker) runWorkloadHistoryFlusher(ctx context.Context) func() {func (b *Broker) Serve(ctx context.Context) error {ctx, cancel := context.WithCancel(ctx)b.cancel = cancel+ b.setServeCtx(ctx)defer cancel()// Enable durable workload history: load the prior session's terminal@@ -1695,8 +2074,7 @@ func (b *Broker) Serve(ctx context.Context) error {if b.engineMgrSup != nil {defer b.engineMgrSup.Stop()}- b.prepareManagedOllamaFacade()- b.prepareManagedLMStudioFacade()+ b.prepareEnabledFacades()// node-info is an auxiliary worker: spawning it lets the broker's own// host advertise its hardware inventory on the network, but it is NOT@@ -1717,62 +2095,56 @@ func (b *Broker) Serve(ctx context.Context) error {slog.Info("node-info path not resolved; running without local node advertisement")}- // ollama-proxy is another auxiliary worker: it runs a local Ollama- // reverse proxy and exposes a control plane the broker relays to- // clients. Like node-info, a spawn failure or never-resolved binary- // is non-fatal. It does NOT gate app:ready: the proxy announces its- // listen port asynchronously via a "ready" notification (after its- // HTTP bind), so clients learn the port by polling proxy:get-status+ // nvpair-proxy is another auxiliary worker: one process that runs a local+ // reverse proxy per enabled engine and exposes a control plane the broker+ // relays to clients. Like node-info, a spawn failure or never-resolved+ // binary is non-fatal. It does NOT gate app:ready: each facade announces+ // its listen port asynchronously via a "ready" notification (after its HTTP+ // bind), so clients learn the port by polling <engine>-proxy:get-status// rather than by the presence of app:ready.- if b.proxyPath != "" {- b.proxySup = newSupervisor("proxy", defaultRestartPolicy(), b.spawnProxy)- b.proxySup.onCrash, b.proxySup.onRecovered = b.supervisedWorkerCallbacks("proxy", func() { b.setProxy(nil) })+ //+ // One supervisor for every engine. That is what puts the estimated-work+ // reservations of all facades in one place, which is the point of the+ // unified proxy; the cost is shared fate, so a crash takes every engine's+ // facade down and the supervisor brings them all back together.+ anyProxyEnabled := false+ for _, profile := range engineProxyProfiles {+ if b.proxyEnabled(profile) {+ anyProxyEnabled = true+ continue+ }+ reason, report := proxyDisabledReason(b.proxyPath)+ slog.Info("engine proxy not enabled; running without one",+ "engine", profile.Name, "reason", reason)+ // Preparation was skipped for a disabled engine, so managedFacade is+ // necessarily false and there is no facade to block. Surface the+ // unavailability directly instead, when it is worth surfacing.+ if report {+ b.reportProxyUnavailable(profile, reason)+ }+ b.finishEngineProxyStartup(profile)+ }+ if anyProxyEnabled {+ b.proxySup = newSupervisor(engines.ProxyComponent, defaultRestartPolicy(), b.spawnProxy)+ b.configureProxySupervisorCallbacks(b.proxySup)if err := b.proxySup.Start(); err != nil {- slog.Warn("proxy failed to start; continuing without local Ollama proxy", "path", b.proxyPath, "err", err)+ slog.Warn("proxy failed to start; continuing without local engine proxies",+ "path", b.proxyPath, "err", err)b.proxySup = nil- b.disableOllamaHostAliasReservation()- if b.managedOllamaFacade.Load() {- b.blockManagedOllamaFacade("the Ollama proxy could not be started")- } else {- b.markOllamaPortReady()+ // No facade will come up, so both engines take the terminal path:+ // report the blocked claim where one was made, then finish, which+ // is what returns the OLLAMA_HOST alias reservation.+ if b.ollamaState().managedFacade.Load() {+ b.blockManagedOllamaFacade("the proxy could not be started")}- } else {- defer b.proxySup.Stop()- }- } else {- slog.Info("proxy path not resolved; running without local Ollama proxy")- b.disableOllamaHostAliasReservation()- if b.managedOllamaFacade.Load() {- b.blockManagedOllamaFacade("the Ollama proxy is unavailable")- } else {- b.markOllamaPortReady()- }- }-- // lmstudio-proxy is the LM Studio counterpart of ollama-proxy and is- // supervised identically (non-fatal, port learned via its "ready"- // notification, control plane relayed under lmstudio-proxy:). Having the- // broker own it is what lets it bridge a reachable manual LM Studio node- // into routing, the same way it does for Ollama.- if b.lmstudioProxyPath != "" {- b.lmstudioProxySup = newSupervisor("lmstudio-proxy", defaultRestartPolicy(), b.spawnLMStudioProxy)- b.configureLMStudioProxySupervisorCallbacks(b.lmstudioProxySup)- if err := b.lmstudioProxySup.Start(); err != nil {- slog.Warn("lmstudio-proxy failed to start; continuing without local LM Studio proxy", "path", b.lmstudioProxyPath, "err", err)- b.lmstudioProxySup = nil- if b.managedLMStudioFacade.Load() {- _, _ = b.blockManagedLMStudioFacade("the LM Studio proxy could not be started", nil)+ b.finishOllamaProxyTerminal()+ if b.lmstudioState().managedFacade.Load() {+ _, _ = b.blockManagedLMStudioFacade("the proxy could not be started", nil)}b.finishLMStudioProxyTerminal()} else {- defer b.lmstudioProxySup.Stop()- }- } else {- slog.Info("lmstudio-proxy path not resolved; running without local LM Studio proxy")- if b.managedLMStudioFacade.Load() {- _, _ = b.blockManagedLMStudioFacade("the LM Studio proxy is unavailable", nil)+ defer b.proxySup.Stop()}- b.finishLMStudioProxyTerminal()}// Restore engines and begin both advertising loops only after both proxy@@ -1854,15 +2226,15 @@ func (b *Broker) shutdownInferenceStack() {beginTeardown()// Stop ingress first so no new inference can arrive while engine-manager is- // draining engines. supervisor.Stop uses each proxy's stdin-close/join path;- // it never adds a parent-side kill timeout.- if b.lmstudioProxySup != nil {- b.lmstudioProxySup.Stop()- b.setLMStudioProxy(nil)- }+ // draining engines. supervisor.Stop uses the proxy's stdin-close/join path;+ // it never adds a parent-side kill timeout. One process holds every+ // engine's facade, so this closes all of them at once — the child drains+ // each facade before releasing the shared transport pool.if b.proxySup != nil {b.proxySup.Stop()- b.setProxy(nil)+ for _, profile := range engineProxyProfiles {+ b.setEngineProxyHandle(profile, nil)+ }}b.prepareEngineManagerShutdown()}@@ -1959,6 +2331,12 @@ func (b *Broker) forwardProxyNotificationForGeneration(generation uint64, methodif generation != b.currentOllamaProxyGeneration() {return}+ // Strip the facade address before anything dispatches on the method: every+ // comparison below, and the errors relay in particular, matches bare names.+ method, addressed := facadeMethodFor(ollamaProxyProfile, method)+ if !addressed {+ return+ }// The alias warning is also the proxy's authoritative "bind did not win"// receipt. Release the candidate reservation before forwarding the sticky// warning so an existing Ollama owner can be adopted normally.@@ -1970,33 +2348,37 @@ func (b *Broker) forwardProxyNotificationForGeneration(generation uint64, method}}}- if b.dispatchErrorsNotif("proxy", method, params) {+ if b.dispatchErrorsNotif(ollamaProxyProfile.ComponentName(), method, params) {return}- // A process can win :11434 after startup preflight but before the proxy- // binds. Flip the supervisor's next spawn to an explicit safe fallback;- // the backend move is still pending, so no engine state needs rollback.- if method == "error" && b.managedOllamaFacade.Load() {+ // A process can win the port after startup preflight but before the facade+ // binds. Choose a safe fallback here, without calling back into this reader+ // goroutine; the backend move is still pending, so no engine state needs+ // rollback.+ //+ // This notification precedes the failing enable's response on the same+ // stream, so the port chosen here is what ollamaFallbackPort finds when the+ // enable retries. Deciding it here rather than there is deliberate: only+ // this path knows whether the managed facade has to be given up, and it+ // owns telling the user.+ if method == "error" {var ep struct {Code string `json:"code"`Port int `json:"port"`}- if json.Unmarshal(params, &ep) == nil && ep.Code == "bind-failed" && ep.Port == managedOllamaFacadePort {- b.blockManagedOllamaFacade("another process acquired the compatibility port during startup")+ if json.Unmarshal(params, &ep) == nil && ep.Code == "bind-failed" {+ switch {+ case b.ollamaState().managedFacade.Load() && ep.Port == managedOllamaFacadePort:+ b.blockManagedOllamaFacade("another process acquired the compatibility port during startup")+ default:+ // Without this the retry asks for the same port and the enable+ // fails identically, leaving the facade down for the life of+ // the process.+ fallback := b.setOllamaProxyFallback(ep.Port)+ slog.Warn("Ollama proxy bind failed; retrying on fallback", "port", ep.Port, "fallback", fallback)+ }}}- // Workload lifecycle / removal events are not proxy control-plane- // events: instead of being re-emitted to proxy:subscribe clients they- // are stamped with the local node id and forwarded to the- // workload-manager for cluster broadcast.- if proxyWorkloadMethods[method] {- b.routeProxyWorkload(method, params)- return- }- if method == noderec.NotifyNodeActivity {- b.routeNodeActivity(params)- return- }// When the proxy announces a (re)bound port — notably its restored port// on startup — steer it clear of any running engine's port. Dispatched// on a separate goroutine: the corrective set-port round-trips through@@ -2006,19 +2388,8 @@ func (b *Broker) forwardProxyNotificationForGeneration(generation uint64, methodif err := json.Unmarshal(params, &rp); err == nil && rp.Port > 0 {go b.reconcileProxyPortOnReady(rp.Port)}- // A restarted proxy comes up with no priority list; re-push the cached- // one so scheduler-driven ordering survives a proxy respawn.- go b.repushPriority("ollama")- }- b.proxyMu.Lock()- subscribed := b.proxySubscribed- b.proxyMu.Unlock()- if !subscribed {- return- }- if err := b.codec.Notify("proxy:"+method, params); err != nil {- slog.Warn("forward proxy notification failed", "method", method, "err", err)}+ b.forwardEngineProxyNotification(ollamaProxyProfile, method, params)}// routeNodeActivity relays a proxy's liveness report about a peer down to the@@ -2231,16 +2602,15 @@ func (b *Broker) forwardLogLevel(level string) {slog.Warn("failed to forward log/set-level to node-info", "err", err)}}- if p := b.getProxy(); p != nil {- if err := p.SetLogLevel(level); err != nil {+ // One iteration over distinct handles, not one per engine: every engine's+ // facade lives in the same process now, so both engine slots resolve to the+ // same handle and a per-engine loop would send this process-scoped+ // notification twice.+ for _, target := range b.livePriorityTargets() {+ if err := target.proxy.SetLogLevel(level); err != nil {slog.Warn("failed to forward log/set-level to proxy", "err", err)}}- if p := b.getLMStudioProxy(); p != nil {- if err := p.SetLogLevel(level); err != nil {- slog.Warn("failed to forward log/set-level to lmstudio-proxy", "err", err)- }- }if wm := b.getWorkloadMgr(); wm != nil {if err := wm.SetLogLevel(level); err != nil {slog.Warn("failed to forward log/set-level to workload-manager", "err", err)@@ -2768,14 +3138,14 @@ func (b *Broker) handleMessage(msg *Message) {log.Printf("failed to respond to discovery:unsubscribe: %v", err)}- case "proxy:get-status":+ case "ollama-proxy:get-status":// Answered locally from the proxy handle's captured state — no// round-trip to the proxy. When no proxy is supervised the// zero value ({ready:false, port:0}) is a valid "not available"// answer, so the method never errors.var result ProxyStatusResultif p := b.getProxy(); p != nil {- ready, port := p.Status()+ ready, port := p.Status(ollamaProxyProfile.Name)result.Ready = readyresult.Port = port}@@ -2783,10 +3153,9 @@ func (b *Broker) handleMessage(msg *Message) {log.Printf("failed to respond to proxy:get-status: %v", err)}- case "proxy:subscribe":+ case "ollama-proxy:subscribe":b.proxyMu.Lock()- wasSubscribed := b.proxySubscribed- b.proxySubscribed = true+ wasSubscribed := b.setEngineProxySubscribed(ollamaProxyProfile, true)b.proxyMu.Unlock()if err := b.codec.Respond(msg.ID, SubscriptionResult{Subscribed: true}); err != nil {log.Printf("failed to respond to proxy:subscribe: %v", err)@@ -2797,23 +3166,23 @@ func (b *Broker) handleMessage(msg *Message) {// the ack. A redundant re-subscribe doesn't re-emit.if !wasSubscribed {if p := b.getProxy(); p != nil {- if rp := p.ReadyParams(); rp != nil {- if err := b.codec.Notify("proxy:ready", rp); err != nil {+ if rp := p.ReadyParams(ollamaProxyProfile.Name); rp != nil {+ if err := b.codec.Notify("ollama-proxy:ready", rp); err != nil {slog.Warn("emit baseline proxy:ready failed", "err", err)}}}}- case "proxy:unsubscribe":+ case "ollama-proxy:unsubscribe":b.proxyMu.Lock()- b.proxySubscribed = false+ b.setEngineProxySubscribed(ollamaProxyProfile, false)b.proxyMu.Unlock()if err := b.codec.Respond(msg.ID, SubscriptionResult{Subscribed: false}); err != nil {log.Printf("failed to respond to proxy:unsubscribe: %v", err)}- case "proxy:set-port":+ case "ollama-proxy:set-port":// Intercepted rather than relayed verbatim: the broker resolves a// port conflict against the running engines (engines win, the proxy// is bumped) before handing the proxy the port to bind.@@ -2827,7 +3196,7 @@ func (b *Broker) handleMessage(msg *Message) {// mirroring proxy:get-status. Zero value when none is supervised.var result ProxyStatusResultif p := b.getLMStudioProxy(); p != nil {- ready, port := p.Status()+ ready, port := p.Status(lmstudioProxyProfile.Name)result.Ready = readyresult.Port = port}@@ -2837,15 +3206,14 @@ func (b *Broker) handleMessage(msg *Message) {case "lmstudio-proxy:subscribe":b.proxyMu.Lock()- wasSubscribed := b.lmstudioProxySubscribed- b.lmstudioProxySubscribed = true+ wasSubscribed := b.setEngineProxySubscribed(lmstudioProxyProfile, true)b.proxyMu.Unlock()if err := b.codec.Respond(msg.ID, SubscriptionResult{Subscribed: true}); err != nil {log.Printf("failed to respond to lmstudio-proxy:subscribe: %v", err)}if !wasSubscribed {if p := b.getLMStudioProxy(); p != nil {- if rp := p.ReadyParams(); rp != nil {+ if rp := p.ReadyParams(lmstudioProxyProfile.Name); rp != nil {if err := b.codec.Notify("lmstudio-proxy:ready", rp); err != nil {slog.Warn("emit baseline lmstudio-proxy:ready failed", "err", err)}@@ -2855,7 +3223,7 @@ func (b *Broker) handleMessage(msg *Message) {case "lmstudio-proxy:unsubscribe":b.proxyMu.Lock()- b.lmstudioProxySubscribed = false+ b.setEngineProxySubscribed(lmstudioProxyProfile, false)b.proxyMu.Unlock()if err := b.codec.Respond(msg.ID, SubscriptionResult{Subscribed: false}); err != nil {log.Printf("failed to respond to lmstudio-proxy:unsubscribe: %v", err)@@ -2935,20 +3303,16 @@ func (b *Broker) handleMessage(msg *Message) {b.cancel()default:- // Any remaining method under the proxy: namespace is relayed- // verbatim to ollama-proxy (the reserved broker-local ones —- // proxy:get-status, and the subscription methods — are handled by+ // Any remaining method under an engine's <component>: prefix is+ // relayed verbatim to that engine's proxy (the reserved broker-local+ // ones — get-status and the subscription methods — are handled by// their own cases above). This makes the broker a thin pass-through- // for the proxy's whole control plane without enumerating methods.- // lmstudio-proxy:* is checked before proxy:* — though the prefixes- // don't actually overlap (lmstudio-proxy: vs proxy:), keeping it- // first makes the LM Studio namespace explicit.- if strings.HasPrefix(msg.Method, "lmstudio-proxy:") {- b.relayToLMStudioProxy(msg)- return- }- if strings.HasPrefix(msg.Method, "proxy:") {- b.relayToProxy(msg)+ // for each proxy's whole control plane without enumerating methods.+ //+ // The prefixes are the engines' ComponentName values, so this loop+ // gains a new engine for free rather than needing another branch.+ if profile, ok := engineProxyProfileForMethod(msg.Method); ok {+ b.relayToEngineProxy(profile, msg)return}if strings.HasPrefix(msg.Method, "engine:") {@@ -2971,46 +3335,6 @@ func (b *Broker) handleMessage(msg *Message) {}}-// relayToProxy forwards a proxy:<method> request to ollama-proxy as-// <method> (prefix stripped) and maps the proxy's response straight back-// to the client. proxy:shutdown is refused — the broker owns the proxy's-// lifecycle, so a client must not be able to kill it out from under us. If-// no proxy is supervised (or it's gone), the client gets a clear error-// rather than a silent hang.-func (b *Broker) relayToProxy(msg *Message) {- proxyMethod := strings.TrimPrefix(msg.Method, "proxy:")- if proxyMethod == "shutdown" {- if err := b.codec.RespondError(msg.ID, -32601, "proxy:shutdown is not allowed; the broker owns the proxy lifecycle"); err != nil {- log.Printf("failed to respond to proxy:shutdown: %v", err)- }- return- }-- p := b.getProxy()- if p == nil {- if err := b.codec.RespondError(msg.ID, -32000, "ollama-proxy not available"); err != nil {- log.Printf("failed to respond to %s: %v", msg.Method, err)- }- return- }-- result, rpcErr, err := p.Call(context.Background(), proxyMethod, msg.Params)- switch {- case err != nil:- if err := b.codec.RespondError(msg.ID, -32000, fmt.Sprintf("proxy call failed: %v", err)); err != nil {- log.Printf("failed to respond to %s: %v", msg.Method, err)- }- case rpcErr != nil:- if err := b.codec.RespondError(msg.ID, rpcErr.Code, rpcErr.Message); err != nil {- log.Printf("failed to relay proxy error for %s: %v", msg.Method, err)- }- default:- if err := b.codec.Respond(msg.ID, result); err != nil {- log.Printf("failed to relay proxy result for %s: %v", msg.Method, err)- }- }-}-// relayToEngine forwards an engine:<method> request to nvpair-engine-manager// and relays its eventual response to the client. engine-manager's methods// are already engine:-prefixed, so the method is forwarded verbatim (no@@ -3054,14 +3378,14 @@ func (b *Broker) relayToEngine(msg *Message) {if isEnginePortAssignment && b.rejectOllamaHostAliasPort(msg, requestedPort, requestedEngine) {return}- if isOllamaPortAssignment && requestedPort == managedOllamaFacadePort && b.managedOllamaFacade.Load() {+ if isOllamaPortAssignment && requestedPort == managedOllamaFacadePort && b.ollamaState().managedFacade.Load() {if err := b.codec.RespondError(msg.ID, -32000, "port 11434 is reserved by the managed Ollama proxy; choose another backend port or disable managed port ownership and restart NVPAIR"); err != nil {log.Printf("failed to reject conflicting Ollama port assignment: %v", err)}return}requestedLMStudioPort, isLMStudioSetPort := lmstudioSetPortRequest(msg.Method, msg.Params)- if isLMStudioSetPort && requestedLMStudioPort == managedLMStudioFacadePort && b.managedLMStudioFacade.Load() {+ if isLMStudioSetPort && requestedLMStudioPort == managedLMStudioFacadePort && b.lmstudioState().managedFacade.Load() {if err := b.codec.RespondError(msg.ID, -32000, "port 1234 is reserved by the managed LM Studio proxy; choose another backend port or disable managed port ownership and restart NVPAIR"); err != nil {log.Printf("failed to reject conflicting LM Studio engine:set-port: %v", err)}@@ -3095,11 +3419,11 @@ func (b *Broker) relayToEngine(msg *Message) {Port int `json:"port"`}if json.Unmarshal(result, &status) == nil && status.Port > 0 {- b.ollamaBackendPort.Store(int32(status.Port))+ b.ollamaState().backendPort.Store(int32(status.Port))}}if isLMStudioSetPort {- b.lmstudioBackendPort.Store(int32(requestedLMStudioPort))+ b.lmstudioState().backendPort.Store(int32(requestedLMStudioPort))}if e := b.codec.Respond(id, result); e != nil {log.Printf("failed to relay engine result for %s: %v", method, e)diff --git a/services/nvpair-ui-broker/broker_lifecycle_test.go b/services/nvpair-ui-broker/broker_lifecycle_test.goindex 51cc091..f60fcbe 100644--- a/services/nvpair-ui-broker/broker_lifecycle_test.go+++ b/services/nvpair-ui-broker/broker_lifecycle_test.go@@ -5,12 +5,13 @@ package mainimport ("context"- "encoding/json""net""path/filepath""sync""testing""time"++ "nvpair-shared/engines")// TestClusterManagerConfigDirTracksBrokerClusterDir pins the invariant that keeps@@ -123,18 +124,18 @@ func (h *orderedLifecycleHandle) Stop() {})}-func TestInferenceShutdownStopsProxiesBeforeEngines(t *testing.T) {- order := make(chan string, 3)- ollama := newOrderedLifecycleHandle("ollama-proxy", order)- lmstudio := newOrderedLifecycleHandle("lmstudio-proxy", order)- proxySup := newSupervisor("proxy", noRestartPolicy(), func() (supervisedHandle, error) { return ollama, nil })- lmstudioSup := newSupervisor("lmstudio-proxy", noRestartPolicy(), func() (supervisedHandle, error) { return lmstudio, nil })+// Ingress stops before engines drain, so no new inference can arrive while+// engine-manager is stopping engines. One proxy process holds every engine's+// facade now, so there is one proxy stop rather than two — but it must still+// precede engine-manager.+func TestInferenceShutdownStopsTheProxyBeforeEngines(t *testing.T) {+ order := make(chan string, 2)+ proxy := newOrderedLifecycleHandle(engines.ProxyComponent, order)+ proxySup := newSupervisor(engines.ProxyComponent, noRestartPolicy(),+ func() (supervisedHandle, error) { return proxy, nil })if err := proxySup.Start(); err != nil {t.Fatal(err)}- if err := lmstudioSup.Start(); err != nil {- t.Fatal(err)- }engineClient, engineServer := net.Pipe()defer engineClient.Close()@@ -151,48 +152,64 @@ func TestInferenceShutdownStopsProxiesBeforeEngines(t *testing.T) {_ = codec.Respond(msg.ID, nil)}()- b := &Broker{proxySup: proxySup, lmstudioProxySup: lmstudioSup}+ b := &Broker{proxySup: proxySup}b.setEngineMgr(engine)b.shutdownInferenceStack()- got := []string{<-order, <-order, <-order}- if got[0] != "lmstudio-proxy" || got[1] != "ollama-proxy" || got[2] != "engine-manager" {- t.Fatalf("shutdown order = %v, want [lmstudio-proxy ollama-proxy engine-manager]", got)+ got := []string{<-order, <-order}+ if got[0] != engines.ProxyComponent || got[1] != "engine-manager" {+ t.Fatalf("shutdown order = %v, want [%s engine-manager]", got, engines.ProxyComponent)}}-func TestLMStudioSupervisorTerminalFailureOpensGate(t *testing.T) {+// A terminally failed proxy releases every engine's ownership gate. Shared fate+// is the accepted cost of one process, so a caller waiting on LM Studio's gate+// must not be left waiting because the failure was reported for the process.+func TestTerminalProxyFailureOpensEveryEnginesGate(t *testing.T) {spawned := make(chan *fakeHandle, 1)- sup := newSupervisor("lmstudio-proxy", noRestartPolicy(), func() (supervisedHandle, error) {+ sup := newSupervisor(engines.ProxyComponent, noRestartPolicy(), func() (supervisedHandle, error) {h := newFakeHandle()spawned <- hreturn h, nil})- b := &Broker{lmstudioPortReady: make(chan struct{})}- b.managedLMStudioFacade.Store(true)- b.configureLMStudioProxySupervisorCallbacks(sup)+ b := &Broker{+ ollamaPortReady: make(chan struct{}),+ lmstudioPortReady: make(chan struct{}),+ }+ b.ollamaState().managedFacade.Store(true)+ b.lmstudioState().managedFacade.Store(true)+ b.configureProxySupervisorCallbacks(sup)if err := sup.Start(); err != nil {t.Fatal(err)}defer sup.Stop()mustSpawn(t, spawned).crash()- select {- case <-b.lmstudioPortReady:- case <-time.After(2 * time.Second):- t.Fatal("terminal LM Studio proxy failure left ownership pending")+ for name, gate := range map[string]<-chan struct{}{+ ollamaProxyProfile.Name: b.ollamaPortReady,+ lmstudioProxyProfile.Name: b.lmstudioPortReady,+ } {+ select {+ case <-gate:+ case <-time.After(2 * time.Second):+ t.Fatalf("terminal proxy failure left %s ownership pending", name)+ }}- if b.managedLMStudioFacade.Load() {- t.Fatal("terminal LM Studio proxy failure left managed ownership enabled")+ if b.ollamaState().managedFacade.Load() || b.lmstudioState().managedFacade.Load() {+ t.Fatal("terminal proxy failure left managed ownership enabled")}}-// TestLMStudioProxyRestartInvalidatesPendingGeneration: a requested LM Studio-// proxy restart must invalidate the current logical generation and handle before-// the restart is observable, so a ready notification already queued by the-// outgoing process cannot open the port-ownership gate — and the replacement-// generation's ready still does.-func TestLMStudioProxyRestartInvalidatesPendingGeneration(t *testing.T) {+// An unbindable LM Studio facade is given up on by itself: its ownership gate+// is released so callers stop waiting, without restarting the process and+// without clearing the shared proxy handle.+//+// This used to restart the process, which was fine when a process hosted one+// engine. It is not fine now: the same process hosts Ollama's facade, so a+// restart to fix LM Studio's port would drop Ollama's listener and the+// inference in flight on it. The handle assertion is the load-bearing one —+// clearing it would strand every other facade's traffic.+func TestUnbindableLMStudioFacadeFinishesWithoutRestartingTheProcess(t *testing.T) {proxyClient, proxyServer := net.Pipe()t.Cleanup(func() {_ = proxyClient.Close()@@ -202,59 +219,56 @@ func TestLMStudioProxyRestartInvalidatesPendingGeneration(t *testing.T) {go proxy.peer.Serve(nil, nil)b := &Broker{clusterDir: "cluster", lmstudioPortReady: make(chan struct{})}- b.lmstudioBackendPort.Store(managedLMStudioBackendStart)+ b.lmstudioState().backendPort.Store(managedLMStudioBackendStart)b.lmstudioProxyGeneration.Store(1)b.lmstudioProxyPublishedGeneration.Store(1)b.setLMStudioProxy(proxy)- b.lmstudioProxySup = newSupervisor("lmstudio-proxy", noRestartPolicy(), nil)+ // Both engines resolve to this one process, which is what the assertions+ // below rely on: Ollama's handle has to be observable to show it survived.+ b.setProxy(proxy)+ b.proxySup = newSupervisor(engines.ProxyComponent, noRestartPolicy(), nil)+ // Answer every set-port with a port other than the one asked for, which is+ // how the child reports it could not take it.go func() {codec := NewCodec(proxyServer)- msg, err := codec.Read()- if err == nil {- _ = codec.Respond(msg.ID, map[string]bool{"ok": true})+ for {+ msg, err := codec.Read()+ if err != nil {+ return+ }+ _ = codec.Respond(msg.ID, proxyReadyParams{Port: 1})}}()b.lmstudioReadyMu.Lock()- b.restartLMStudioProxyOrFinish(1)+ port, ok := b.rebindLMStudioFacadeOrFinish(proxy, 1, managedLMStudioFacadePort)b.lmstudioReadyMu.Unlock()- select {- case <-b.lmstudioProxySup.restartCh:- case <-time.After(2 * time.Second):- t.Fatal("restart request did not reach the LM Studio proxy supervisor")++ if ok || port != 0 {+ t.Fatalf("rebind reported success (port %d) though every set-port was refused", port)}- b.forwardLMStudioProxyNotificationForGeneration(1, "ready", json.RawMessage(`{"port":1236}`))+ // The gate has to open, or every engine:status for LM Studio waits out the+ // call timeout and answers "retry" for the life of the process.select {case <-b.lmstudioPortReady:- t.Fatal("old-generation ready opened gate after restart")- case <-time.After(100 * time.Millisecond):+ case <-time.After(2 * time.Second):+ t.Fatal("an unbindable facade left its ownership gate closed")}- if b.lmstudioProxyGeneration.Load() == 1 || b.getLMStudioProxy() != nil {- t.Fatal("restart did not invalidate LM Studio generation and handle")+ if b.lmstudioState().managedFacade.Load() {+ t.Error("managed LM Studio mode survived an unbindable facade")}-- replacementClient, replacementServer := net.Pipe()- t.Cleanup(func() {- _ = replacementClient.Close()- _ = replacementServer.Close()- })- replacement := &proxyProcess{peer: NewPeer(NewCodec(replacementClient))}- go replacement.peer.Serve(nil, nil)- replacementGeneration := b.lmstudioProxyGeneration.Add(1)- b.setLMStudioProxy(replacement)- b.lmstudioProxyPublishedGeneration.Store(replacementGeneration)- go func() {- codec := NewCodec(replacementServer)- msg, err := codec.Read()- if err == nil {- _ = codec.Respond(msg.ID, map[string]bool{"ok": true})- }- }()- b.forwardLMStudioProxyNotificationForGeneration(replacementGeneration, "ready", json.RawMessage(`{"port":1236}`))- select {- case <-b.lmstudioPortReady:- case <-time.After(2 * time.Second):- t.Fatal("replacement generation did not open gate after the restart")+ // The process is untouched: the same handle is still published for BOTH+ // engines, so Ollama's facade in that same process keeps serving. Handle+ // identity is the whole property — a process restart would replace it,+ // which is what would take Ollama's listener down to fix LM Studio's port.+ //+ // The Ollama handle is published in the setup precisely so this can fail:+ // asserting on a slot that was never populated would pass no matter what.+ if b.getLMStudioProxy() != proxy {+ t.Error("a facade port failure replaced the shared proxy handle")+ }+ if b.getProxy() != proxy {+ t.Error("a facade port failure disturbed the other engine's handle on the same process")}}diff --git a/services/nvpair-ui-broker/brokertest_test.go b/services/nvpair-ui-broker/brokertest_test.gonew file mode 100644index 0000000..262ceeb--- /dev/null+++ b/services/nvpair-ui-broker/brokertest_test.go@@ -0,0 +1,162 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++// Fixtures shared by more than one test file in this package. Go test files+// share a package, so a helper defined next to the tests that first needed it+// is reachable from every other file — which is how newTestRPCWorkerPipe came+// to live in lmstudioport_test.go while ollamahost_test.go depends on it. That+// works until the defining file moves or is consolidated, at which point the+// whole test binary stops compiling and the cause is invisible in the diff.+// Anything used across files belongs here instead.++import (+ "encoding/json"+ "net"+ "testing"+ "time"+)++// Both engines' port tests assert on an ownership gate opening or staying+// shut, and did it by hand with an inline select. Three shapes recurred, and+// the difference between two of them is easy to get wrong: waiting on+// time.After proves the gate stayed shut through a window, while a default arm+// only proves it was shut at that instant. Naming the three keeps a test from+// silently asserting the weaker one.+const (+ // gateSettleWindow is how long a gate is given to open before the test+ // calls it stuck. Generous, because opening one can involve a real RPC+ // round-trip over a pipe.+ gateSettleWindow = 2 * time.Second++ // gateQuietWindow is how long a gate must stay shut to count as shut.+ gateQuietWindow = 100 * time.Millisecond+)++// requireGateOpens fails unless the ownership gate opens. reason names what+// should have opened it.+func requireGateOpens(t *testing.T, gate <-chan struct{}, reason string) {+ t.Helper()+ select {+ case <-gate:+ case <-time.After(gateSettleWindow):+ t.Fatalf("ownership gate never opened: %s", reason)+ }+}++// requireGateStaysShut fails if the gate opens during the quiet window. Use it+// after something that must not settle ownership; the wait is what makes it+// meaningful, since the work under test may be on another goroutine.+func requireGateStaysShut(t *testing.T, gate <-chan struct{}, reason string) {+ t.Helper()+ select {+ case <-gate:+ t.Fatalf("ownership gate opened when it should not have: %s", reason)+ case <-time.After(gateQuietWindow):+ }+}++// requireGateShutNow fails if the gate is already open, without waiting. Use it+// only when nothing asynchronous is in flight; otherwise prefer+// requireGateStaysShut.+func requireGateShutNow(t *testing.T, gate <-chan struct{}, reason string) {+ t.Helper()+ select {+ case <-gate:+ t.Fatalf("ownership gate was already open: %s", reason)+ default:+ }+}++// requireGateOpenNow fails unless the gate is already open, without waiting.+// The synchronous counterpart to requireGateOpens, for a caller that settled+// ownership on this goroutine and should not need to wait for it.+func requireGateOpenNow(t *testing.T, gate <-chan struct{}, reason string) {+ t.Helper()+ select {+ case <-gate:+ default:+ t.Fatalf("ownership gate was not open: %s", reason)+ }+}++// newTestRPCWorkerPipe returns a worker whose peer is served over an in-memory+// pipe, plus the codec for the far end so a test can act as the worker.+func newTestRPCWorkerPipe(t *testing.T) (*rpcWorker, *Codec) {+ t.Helper()+ client, server := net.Pipe()+ t.Cleanup(func() {+ _ = client.Close()+ _ = server.Close()+ })+ worker := &rpcWorker{peer: NewPeer(NewCodec(client))}+ go worker.peer.Serve(nil, nil)+ return worker, NewCodec(server)+}++// brokerWithEngineStatus is brokerWithEngineInventory for a single engine.+func brokerWithEngineStatus(t *testing.T, engine string, port int) *Broker {+ return brokerWithEngineInventory(t, map[string]int{engine: port})+}++// brokerWithEngineInventory returns a broker whose engine-manager answers one+// engine:status or engine:get-installed request from a fixed port inventory.+func brokerWithEngineInventory(t *testing.T, ports map[string]int) *Broker {+ t.Helper()+ client, server := net.Pipe()+ worker := &rpcWorker{peer: NewPeer(NewCodec(client))}+ go worker.peer.Serve(nil, nil)+ b := &Broker{}+ b.setEngineMgr(worker)+ go func() {+ codec := NewCodec(server)+ request, err := codec.Read()+ if err != nil {+ return+ }+ switch request.Method {+ case "engine:status":+ var params struct {+ Engine string `json:"engine"`+ }+ if json.Unmarshal(request.Params, ¶ms) != nil {+ _ = codec.RespondError(request.ID, -32602, "invalid engine status request")+ return+ }+ _ = codec.Respond(request.ID, map[string]any{"port": ports[params.Engine]})+ case "engine:get-installed":+ engines := make([]map[string]any, 0, len(ports))+ for engine, port := range ports {+ engines = append(engines, map[string]any{"engine": engine, "port": port})+ }+ _ = codec.Respond(request.ID, map[string]any{"engines": engines})+ default:+ _ = codec.RespondError(request.ID, -32602, "unexpected engine status request")+ }+ }()+ t.Cleanup(func() {+ _ = client.Close()+ _ = server.Close()+ })+ return b+}++// isolateOllamaHostTestConfig points the per-user config dir at a temp dir so a+// test's persisted state cannot touch the developer's real configuration.+//+// All four variables are required. appdir consults LOCALAPPDATA only on+// Windows and otherwise falls through to os.UserConfigDir, which reads HOME on+// macOS and XDG_CONFIG_HOME (then HOME) on Linux. Setting only the first two+// left macOS unisolated, so TestConfiguredLMStudioProxyPort wrote its fixture+// into the developer's real config dir and then failed on every subsequent run+// against the value it had planted. CI never saw it because its config dir+// starts empty.+func isolateOllamaHostTestConfig(t *testing.T) {+ t.Helper()+ dir := t.TempDir()+ t.Setenv("LOCALAPPDATA", dir)+ t.Setenv("APPDATA", dir)+ t.Setenv("XDG_CONFIG_HOME", dir)+ t.Setenv("HOME", dir)+}diff --git a/services/nvpair-ui-broker/engineproxy.go b/services/nvpair-ui-broker/engineproxy.gonew file mode 100644index 0000000..17045bd--- /dev/null+++ b/services/nvpair-ui-broker/engineproxy.go@@ -0,0 +1,624 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++// engineproxy.go is the broker's per-engine proxy table. Cross-process+// identity — name, namespace, facade and backend ports — comes from+// nvpair-shared/engines so the proxy, scheduler and TUI cannot disagree with+// the broker about it. What this file adds is the one thing only the broker+// needs: whether it may reposition the engine's process while it is running.+//+// That single question decides every place the two engines' port choreography+// diverges, which is why it is a named enum rather than a set of booleans or a+// bag of function pointers. A hook would only move the divergent bodies into+// this file; naming the reason keeps them where they belong and makes the+// difference reviewable.++import (+ "context"+ "encoding/json"+ stderrors "errors"+ "fmt"+ "log"+ "log/slog"+ "strings"+ "sync/atomic"++ "nvpair-shared/engines"+ "nvpair-shared/errors"+ "nvpair-shared/noderec"+)++// engineOwnership answers a single question: may the broker reposition this+// engine's process while it is running?+type engineOwnership int++const (+ // adoptedEngine — the engine may have been started outside PAIR, so the+ // broker has no takeover authority over it. It is repositioned only while+ // stopped, and even then the move is deferred until the proxy has proved+ // it holds the facade port. Ollama.+ adoptedEngine engineOwnership = iota++ // managedEngine — engine-manager launched it in identified command mode+ // and has an official stop command for it, so it may be stopped and+ // repositioned before the proxy starts. LM Studio.+ managedEngine+)++// engineProxyProfile is everything the broker needs to supervise one engine's+// proxy.+type engineProxyProfile struct {+ engines.Engine++ // Ownership decides the port choreography: plan-then-commit for an+ // adopted engine, move-then-verify for a managed one. It is the only+ // judgment call in adding an engine.+ Ownership engineOwnership++ // HealthProbePath is the path whose 200 means "this engine is answering".+ // Ollama's liveness convention is a plain GET of the root; an+ // OpenAI-compatible engine answers on its model list.+ HealthProbePath string+}++// mayMoveRunningEngine reports whether engine-manager can be asked to+// reposition this engine while it is running. When it cannot, a facade port+// that is already occupied is a hard block: the broker has no way to tell its+// own engine from a stranger, and no authority to stop either.+func (p engineProxyProfile) mayMoveRunningEngine() bool {+ return p.Ownership == managedEngine+}++// blocksOnOccupiedFacade reports whether planning must refuse up front when the+// facade port is unavailable.+//+// Only an adopted engine can answer this before the move. For a managed engine+// the occupant is usually the engine itself, and availability cannot+// distinguish that from a stranger — so it moves the backend first and+// re-checks the facade afterwards, when a still-busy port genuinely means+// someone else. Applying the adopted rule to a managed engine would block the+// most common install, where LM Studio is already running on its facade port.+func (p engineProxyProfile) blocksOnOccupiedFacade() bool {+ return p.Ownership == adoptedEngine+}++// engineProxyRuntime is one engine's mutable proxy state. The broker holds one+// per engine, so adding an engine adds a table entry rather than another block+// of parallel fields on Broker.+//+// Locking is inherited from the Broker fields each of these replaced and is+// noted per field. The map itself is built once and never mutated afterwards,+// so resolving a runtime needs no lock — only its fields do.+type engineProxyRuntime struct {+ // profile is the descriptor this runtime belongs to.+ profile engineProxyProfile++ // proxy is the live child process, or nil. Guarded by Broker.workersMu: a+ // supervisor swaps it on restart from the monitor goroutine while request+ // handlers read it from the read loop.+ proxy *proxyProcess++ // subscribed reports whether a client asked for this engine's relay+ // stream. Guarded by Broker.proxyMu.+ subscribed bool++ // managedFacade reports that the broker is holding this engine's facade+ // port for its proxy.+ managedFacade atomic.Bool++ // backendPort is engine-manager's configured port for the engine itself,+ // as last observed. Zero means not yet known, which several paths treat+ // as a reason to fail closed rather than assume a default.+ backendPort atomic.Int32++ // startupPort is the explicit port the next spawn should bind. Zero lets+ // the proxy restore its own persisted port.+ startupPort atomic.Int32+}++// Deliberately not here yet: the ownership gate channel and its sync.Once, the+// respawn generations, and the reconcile mutex.+//+// The gate channel's nil-ness is load-bearing — a nil channel means "no gate+// configured", which is how a test that constructs a bare &Broker{} gets+// ungated relays. Moving it behind a lazily-built runtime would quietly make+// every such broker gated instead. The generations are worse: Ollama's lives+// under ollamaHostAliasMu because it is bumped together with the alias+// snapshot, while LM Studio's is a plain atomic, and collapsing that+// difference is what deadlocks the JSON-RPC read pump.++// engineProxy resolves an engine's runtime state.+//+// The map is initialized on first use rather than in NewBroker because a large+// body of tests constructs a bare &Broker{} and drives one behavior directly;+// requiring a constructor would mean touching all of them to change nothing.+// A zero sync.Once works on such a value, and Do gives every later reader a+// happens-before edge on the map.+func (b *Broker) engineProxy(p engineProxyProfile) *engineProxyRuntime {+ b.engineProxiesOnce.Do(func() {+ b.engineProxies = make(map[string]*engineProxyRuntime, len(engineProxyProfiles))+ for _, profile := range engineProxyProfiles {+ b.engineProxies[profile.Name] = &engineProxyRuntime{profile: profile}+ }+ })+ return b.engineProxies[p.Name]+}++// ollamaState and lmstudioState are shorthand for the two engines this build+// ships, for code that is inherently about one of them. Profile-generic code+// should take an engineProxyProfile and call engineProxy instead.+func (b *Broker) ollamaState() *engineProxyRuntime { return b.engineProxy(ollamaProxyProfile) }+func (b *Broker) lmstudioState() *engineProxyRuntime { return b.engineProxy(lmstudioProxyProfile) }++// engineProxyProfiles is the broker's engine set, in preparation order.+// Ollama is first because its preparation reserves any inherited OLLAMA_HOST+// alias that later engines must route around.+var engineProxyProfiles = buildEngineProxyProfiles()++func buildEngineProxyProfiles() []engineProxyProfile {+ brokerOnly := map[string]engineProxyProfile{+ "ollama": {Ownership: adoptedEngine, HealthProbePath: "/"},+ // LM Studio is the one engine engine-manager may move while running:+ // its identified command-mode runtime has an official stop command.+ "lmstudio": {Ownership: managedEngine, HealthProbePath: "/v1/models"},+ }+ out := make([]engineProxyProfile, 0, len(engines.All()))+ for _, e := range engines.All() {+ p := brokerOnly[e.Name]+ p.Engine = e+ out = append(out, p)+ }+ return out+}++// enableFacadeRequest is the broker's facade/enable payload. It mirrors the+// child's parameter struct; the port and alias addresses travel here rather+// than on argv because the broker plans a different port for each engine.+type enableFacadeRequest struct {+ Engine string `json:"engine"`+ Port int `json:"port,omitempty"`+ AliasAddresses []string `json:"aliasAddresses,omitempty"`+ IgnorePersistedPort bool `json:"ignorePersistedPort,omitempty"`+}++// enableFacadeReply is where the facade actually landed. The bound port is read+// from here rather than from the ready notification because enable is a+// request: the child's persisted-port restore can override the asked-for port.+type enableFacadeReply struct {+ Engine string `json:"engine"`+ Port int `json:"port"`+}++// codeFacadeBindFailed mirrors the constant of the same name in nvpair-proxy:+// the JSON-RPC error code for an enable that failed only because the port was+// taken. Matching on it is what separates a retryable bind race from a+// rejection no other port would fix.+const codeFacadeBindFailed = -32010++// errFacadeBindFailed tags a bind race locally so callers match on intent+// rather than on a wire code.+var errFacadeBindFailed = stderrors.New("facade bind failed")++// errRPCAnswered tags an enable failure where the child actually replied, as+// opposed to one where the call timed out or the pipe closed.+//+// The distinction decides whether a recorded "ready" from earlier in the same+// attempt can still be trusted: an answered rejection means the child withdrew+// the facade, so that ready is stale. See facadeCameUpAnyway.+var errRPCAnswered = stderrors.New("proxy answered")++// enableProxyFacade asks a proxy process to bring one engine's facade up.+//+// Bounded twice over. proxyCallTimeout is a wedge detector — the child's own+// work is a listen plus two notifications — and parent carries teardown, so+// bring-up is abandoned rather than run to completion while the broker is+// trying to exit. See proxyBringUpContext for why that matters.+func (b *Broker) enableProxyFacade(parent context.Context, p *proxyProcess, spec enableFacadeRequest) error {+ params, err := json.Marshal(spec)+ if err != nil {+ return fmt.Errorf("marshal facade/enable for %s: %w", spec.Engine, err)+ }+ ctx, cancel := context.WithTimeout(parent, proxyCallTimeout)+ defer cancel()++ raw, rpcErr, err := p.Call(ctx, "facade/enable", params)+ if err != nil {+ return fmt.Errorf("facade/enable %s: %w", spec.Engine, err)+ }+ if rpcErr != nil {+ if rpcErr.Code == codeFacadeBindFailed {+ return fmt.Errorf("%w: %w: %s: %s",+ errRPCAnswered, errFacadeBindFailed, spec.Engine, rpcErr.Message)+ }+ return fmt.Errorf("%w: facade/enable %s rejected: %d %s",+ errRPCAnswered, spec.Engine, rpcErr.Code, rpcErr.Message)+ }+ var reply enableFacadeReply+ if err := json.Unmarshal(raw, &reply); err != nil {+ // A malformed reply is still a reply: the child answered and its facade+ // state is whatever it decided, not unknown.+ return fmt.Errorf("%w: decode facade/enable reply for %s: %w",+ errRPCAnswered, spec.Engine, err)+ }+ slog.Info("facade enabled", "engine", reply.Engine, "port", reply.Port,+ "requested", spec.Port)+ return nil+}++// enableProxyFacadeWithFallback enables a facade, retrying once on a fallback+// port if the first attempt lost a bind race.+//+// Retrying in-process is the point of moving bring-up off argv. A bind failure+// used to end the child so the supervisor could respawn it on a corrected port,+// which is only acceptable while a process hosts one engine: once it hosts+// several, one engine losing a race would take working listeners down with it.+// The supervisor stays the backstop for a second failure.+//+// nextPort receives the port that failed and returns the one to try, or 0 to+// give up. It is a callback because each engine excludes a different set of+// ports — its own engine's port, its alias, its siblings' proxies.+func (b *Broker) enableProxyFacadeWithFallback(+ parent context.Context, p *proxyProcess, spec enableFacadeRequest, nextPort func(failed int) int,+) error {+ err := b.enableProxyFacade(parent, p, spec)+ if err == nil || !stderrors.Is(err, errFacadeBindFailed) {+ return err+ }+ // Teardown began between the attempt and the retry: stop here rather than+ // spending another proxyCallTimeout of the shared budget on a port the+ // process is about to stop serving anyway.+ if parent.Err() != nil {+ return err+ }+ failed := spec.Port+ fallback := nextPort(failed)+ if fallback == 0 || fallback == failed {+ return err+ }+ slog.Warn("facade bind failed; retrying on a fallback port",+ "engine", spec.Engine, "port", failed, "fallback", fallback, "err", err)++ spec.Port = fallback+ // The persisted port is what the child would otherwise restore, and it is+ // the port that just failed to bind.+ spec.IgnorePersistedPort = true+ return b.enableProxyFacade(parent, p, spec)+}++// facadeMethodFor strips a facade-scoped notification's engine address and+// confirms it belongs to the engine this reader speaks for.+//+// A process-scoped notification arrives unaddressed and passes through+// unchanged. An addressed one naming a different engine is dropped rather than+// forwarded: forwardProxyProcessNotification already routes by engine, so+// reaching this reader with another engine's address means addressing broke,+// and forwarding it would file one engine's event under another — a wrong port+// on a status, or a cleared error on the wrong tab.+//+// Callers must strip before every dispatch, including the errors relay: a+// prefixed "ollama:errors:report" does not match methodErrorsReport, so an+// unstripped error would fall through to the client stream as an unknown event.+func facadeMethodFor(profile engineProxyProfile, method string) (string, bool) {+ engine, bare := engines.SplitAddressedMethod(method)+ if engine == "" {+ return method, true+ }+ if engine != profile.Name {+ slog.Warn("dropping proxy notification addressed to another engine",+ "reader", profile.Name, "addressed", engine, "method", bare)+ return "", false+ }+ return bare, true+}++// engineProxyProfileForMethod resolves a relayed client method such as+// "ollama-proxy:nodes/list" to the engine whose proxy should receive it.+func engineProxyProfileForMethod(method string) (engineProxyProfile, bool) {+ for _, p := range engineProxyProfiles {+ if strings.HasPrefix(method, p.ComponentName()+":") {+ return p, true+ }+ }+ return engineProxyProfile{}, false+}++// engineProxyProfileFor resolves an engine id.+func engineProxyProfileFor(name string) (engineProxyProfile, bool) {+ for _, p := range engineProxyProfiles {+ if p.Name == name {+ return p, true+ }+ }+ return engineProxyProfile{}, false+}++// proxyEnabled reports whether the broker should front this engine: the+// unified binary has to be resolvable, and the engine has to be one the+// operator asked for via --proxy-engines.+//+// A disabled engine still runs the not-resolved branch at its call site, which+// is what settles its ownership gate. Skipping the branch entirely would leave+// the gate closed and strand every engine request behind it.+func (b *Broker) proxyEnabled(p engineProxyProfile) bool {+ if b.proxyPath == "" {+ return false+ }+ for _, name := range b.proxyEngines {+ if name == p.Name {+ return true+ }+ }+ return false+}++// prepareEnabledFacades prepares managed port ownership for the engines the+// broker is actually going to front, in the table's order — Ollama first,+// because its preparation reserves any inherited OLLAMA_HOST alias that later+// engines must route around.+//+// The enablement check belongs here and not downstream, because preparation is+// not read-only: for a managed engine the backend move runs inside it, so+// preparing an engine whose proxy is never started relocates that engine off+// its own stock port and leaves nothing serving it. Ollama cannot show the+// symptom, since its move is deferred until its proxy proves it holds the+// facade — which is exactly why this cannot be left to the callee.+func (b *Broker) prepareEnabledFacades() {+ if b.proxyEnabled(ollamaProxyProfile) {+ b.prepareManagedOllamaFacade()+ }+ if b.proxyEnabled(lmstudioProxyProfile) {+ b.prepareManagedLMStudioFacade()+ }+}++// proxyDisabledReason explains why an engine has no proxy, and reports whether+// that is worth surfacing to the user rather than only logging it.+//+// The two causes need different treatment: an unresolved binary is a packaging+// or --proxy-path problem someone has to act on, while an engine deliberately+// left out of --proxy-engines is the operator's own choice and an error about+// it would be noise.+func proxyDisabledReason(proxyPath string) (reason string, report bool) {+ if proxyPath == "" {+ return "proxy binary not resolved", true+ }+ return "engine not listed in --proxy-engines", false+}++// addressed prefixes a facade-scoped method with this engine, so a process+// hosting several facades routes it to the right one.+//+// Only facade-scoped methods take an address. facade/enable names its engine in+// the payload, and log/set-level and node/set-priority concern the whole+// process, so addressing any of those would make the child look for a facade+// that the message was never about.+func (p engineProxyProfile) addressed(method string) string {+ return engines.AddressMethod(p.Name, method)+}++// portBumpedID is the sticky warning surfaced when the broker moves this+// engine's proxy off a port a running engine holds. Sticky (no timestamp+// suffix) so repeated bumps upsert one entry.+func (p engineProxyProfile) portBumpedID() string {+ return p.ComponentName() + ":port-bumped"+}++// ownershipBlockedID matches the per-engine constants; see+// TestBrokerConstantsMatchTheEngineTable.+func (p engineProxyProfile) ownershipBlockedID() string {+ return p.ComponentName() + ":port-ownership-blocked"+}++// reportProxyUnavailable surfaces an engine the broker is not fronting at all+// because its binary did not resolve.+//+// This shares the ownership-blocked ID so the desktop keeps one entry per+// engine, but it must not reuse that message: no facade was prepared, so+// nothing was reserved and no process was inspected, let alone left running.+// Claiming a failed reservation sends the reader looking for a port conflict+// that does not exist.+func (b *Broker) reportProxyUnavailable(p engineProxyProfile, reason string) {+ b.forwardErrorsReport(errors.ServiceError{+ ID: p.ownershipBlockedID(),+ Message: fmt.Sprintf("NVPAIR is not fronting %s on port %d: %s.", p.DisplayName, p.FacadePort, reason),+ Timestamp: nowMillis(),+ NodeID: b.nodeID,+ Severity: "warning",+ Action: "none",+ })+}++// managedFacade reports whether the broker currently holds this engine's+// facade port for its proxy.+func (b *Broker) managedFacade(p engineProxyProfile) bool {+ return b.engineProxy(p).managedFacade.Load()+}++// engineProxyHandle returns the live proxy process for an engine, or nil.+func (b *Broker) engineProxyHandle(p engineProxyProfile) *proxyProcess {+ b.workersMu.Lock()+ defer b.workersMu.Unlock()+ return b.engineProxy(p).proxy+}++// setEngineProxyHandle publishes (or clears, with nil) an engine's live proxy.+func (b *Broker) setEngineProxyHandle(p engineProxyProfile, proxy *proxyProcess) {+ b.workersMu.Lock()+ b.engineProxy(p).proxy = proxy+ b.workersMu.Unlock()+}++// forwardEngineProxyNotification is the tail every proxy notification takes+// once the engine-specific handling is done: workload lifecycle and node+// activity are routed into the backend rather than re-emitted, and anything+// else reaches <namespace>:subscribe'd clients as <namespace>:<method>.+//+// This is the end of the line for a notification — every path consumes it, so+// there is nothing for a caller to do afterwards and nothing to report back.+// The heads of the two callers stay separate: the bind-failure and readiness+// handling genuinely differ by ownership, and folding them in behind a+// callback would move those bodies into this file without making them any more+// shared.+func (b *Broker) forwardEngineProxyNotification(profile engineProxyProfile, method string, params json.RawMessage) {+ if b.routeProcessScopedProxyNotification(method, params) {+ return+ }++ b.proxyMu.Lock()+ subscribed := b.engineProxySubscribed(profile)+ b.proxyMu.Unlock()+ if !subscribed {+ return+ }+ if err := b.codec.Notify(profile.ComponentName()+":"+method, params); err != nil {+ slog.Warn("forward proxy notification failed", "engine", profile.Name, "method", method, "err", err)+ }+}++// routeProcessScopedProxyNotification handles the notifications that belong to+// the proxy process rather than to one of its facades, and reports whether it+// took the method.+//+// Both routes here read the engine out of the payload — a workload record+// carries its own Engine, and a node-activity report is about a peer — so+// neither needs to know which facade emitted it. That is exactly why they must+// be handled once per process: dispatching them through each engine's handler+// in turn would file every workload twice.+func (b *Broker) routeProcessScopedProxyNotification(method string, params json.RawMessage) bool {+ switch {+ case proxyWorkloadMethods[method]:+ b.routeProxyWorkload(method, params)+ return true+ case method == noderec.NotifyNodeActivity:+ b.routeNodeActivity(params)+ return true+ }+ return false+}++// engineProxySubscribed reports whether a client has subscribed to this+// engine's relay stream. Caller holds proxyMu.+func (b *Broker) engineProxySubscribed(p engineProxyProfile) bool {+ return b.engineProxy(p).subscribed+}++// setEngineProxySubscribed records a subscribe or unsubscribe and reports the+// previous value, which callers use to decide whether to replay a baseline+// snapshot. Caller holds proxyMu.+func (b *Broker) setEngineProxySubscribed(p engineProxyProfile, subscribed bool) bool {+ rt := b.engineProxy(p)+ was := rt.subscribed+ rt.subscribed = subscribed+ return was+}++// relayToEngineProxy forwards an <engine>-proxy:<method> client request to that+// engine's facade and maps the response straight back.+//+// The client's component prefix comes off and the facade address goes on. Both+// name the engine, but they are not the same string and not interchangeable:+// "ollama-proxy:" is how a client addresses the component, "ollama:" is how a+// message addresses a facade inside the process.+//+// <engine>-proxy:shutdown is refused: the broker owns the proxy lifecycle, so a+// client must not be able to kill it out from under us. A missing proxy gets a+// clear error rather than a silent hang.+func (b *Broker) relayToEngineProxy(profile engineProxyProfile, msg *Message) {+ respondErr := func(code int, format string, args ...any) {+ if err := b.codec.RespondError(msg.ID, code, fmt.Sprintf(format, args...)); err != nil {+ log.Printf("failed to respond to %s: %v", msg.Method, err)+ }+ }++ method := strings.TrimPrefix(msg.Method, profile.ComponentName()+":")+ if method == "shutdown" {+ respondErr(-32601, "%s:shutdown is not allowed; the broker owns the proxy lifecycle", profile.ComponentName())+ return+ }++ p := b.engineProxyHandle(profile)+ if p == nil {+ respondErr(-32000, "%s not available", profile.ComponentName())+ return+ }++ result, rpcErr, err := p.Call(context.Background(), profile.addressed(method), msg.Params)+ switch {+ case err != nil:+ respondErr(-32000, "proxy call failed: %v", err)+ case rpcErr != nil:+ if err := b.codec.RespondError(msg.ID, rpcErr.Code, rpcErr.Message); err != nil {+ log.Printf("failed to relay %s error for %s: %v", profile.ComponentName(), msg.Method, err)+ }+ default:+ if err := b.codec.Respond(msg.ID, result); err != nil {+ log.Printf("failed to relay %s result for %s: %v", profile.ComponentName(), msg.Method, err)+ }+ }+}++// mustEngineProxyProfile resolves an engine id the broker itself names. A miss+// means the shared table and this one disagree, which is a build-time mistake+// rather than a runtime condition, so it panics at init rather than degrading+// into an engine with a zero-valued facade port.+func mustEngineProxyProfile(name string) engineProxyProfile {+ p, ok := engineProxyProfileFor(name)+ if !ok {+ panic("nvpair-ui-broker: no engine proxy profile for " + name)+ }+ return p+}++// planManagedEnginePorts is the ownership policy core, kept separate from RPC+// so every safety branch is deterministic in tests. It never plans a move the+// broker lacks the authority to make: a running engine is repositioned only+// when engine-manager owns it, and an unknown occupant of the facade port is+// never displaced.+//+// The two facade-availability checks below look redundant and are not. An+// adopted engine has to know before it plans anything, because it cannot move+// a running occupant; a managed engine cannot answer the question yet, because+// the likeliest occupant is its own engine, which the relocation branch is+// about to move. The second check is therefore reached only by managed+// engines — for an adopted one the first check has already returned.+func planManagedEnginePorts(p engineProxyProfile, enabled bool, st ollamaPortStatus, available func(int) bool) managedPortPlan {+ if !enabled {+ return managedPortPlan{}+ }+ facade, backendStart := p.FacadePort, p.EnginePortBase++ // An engine already running on the facade that the broker may not move is+ // the end of the story: it owns the port and there is nothing to plan.+ if !p.mayMoveRunningEngine() && st.Running && (st.Port == facade || st.Port == 0) {+ return managedPortPlan{Blocked: fmt.Sprintf("%s is already running on the compatibility port", p.DisplayName)}+ }+ if p.blocksOnOccupiedFacade() && !available(facade) {+ return managedPortPlan{Blocked: "the compatibility port is already in use"}+ }++ if st.Port > 0 && st.Port != facade {+ // The engine is not on the facade, so an occupant here is a stranger+ // under either ownership.+ if !available(facade) {+ return managedPortPlan{Blocked: "the compatibility port is already in use"}+ }+ // A stopped engine whose configured port has been taken has to be+ // advanced; a running one keeps the port it is serving on.+ if !st.Running && !available(st.Port) {+ backend := nextAvailablePort(backendStart, available)+ if backend == 0 {+ return managedPortPlan{Blocked: "no free backend port is available"}+ }+ return managedPortPlan{Enabled: true, BackendPort: backend}+ }+ return managedPortPlan{Enabled: true}+ }++ backend := nextAvailablePort(backendStart, available)+ if backend == 0 {+ return managedPortPlan{Blocked: "no free backend port is available"}+ }+ return managedPortPlan{Enabled: true, BackendPort: backend}+}diff --git a/services/nvpair-ui-broker/engineproxy_test.go b/services/nvpair-ui-broker/engineproxy_test.gonew file mode 100644index 0000000..7319e55--- /dev/null+++ b/services/nvpair-ui-broker/engineproxy_test.go@@ -0,0 +1,301 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++import (+ "encoding/json"+ "testing"+ "time"+)++// The broker's table must cover exactly the shared engine set, in the same+// order. Preparation order is load-bearing: Ollama goes first because its+// preparation reserves any inherited OLLAMA_HOST alias that later engines have+// to route around.+func TestEngineProxyTableMatchesSharedEngines(t *testing.T) {+ if len(engineProxyProfiles) == 0 {+ t.Fatal("no engine proxy profiles")+ }+ if got := engineProxyProfiles[0].Name; got != "ollama" {+ t.Fatalf("first profile = %q, want ollama prepared first", got)+ }+ for _, p := range engineProxyProfiles {+ if p.FacadePort == p.EnginePortBase {+ t.Errorf("%s: facade and backend base are both %d; the proxy and engine would collide",+ p.Name, p.FacadePort)+ }+ if p.ComponentName() == "" || p.DisplayName == "" {+ t.Errorf("%s: incomplete identity %+v", p.Name, p.Engine)+ }+ // An empty probe path would silently become a GET of the root, which+ // is Ollama's convention and wrong for anything OpenAI-compatible —+ // the engine would read as down whenever it is actually up.+ if p.HealthProbePath == "" {+ t.Errorf("%s: no health probe path", p.Name)+ }+ }+}++// The probe path is the one advertiser value that is per-engine, and getting it+// wrong makes a healthy engine look permanently down rather than failing loudly.+func TestEngineHealthProbePaths(t *testing.T) {+ for _, tc := range []struct {+ engine string+ want string+ }{+ {"ollama", "/"},+ {"lmstudio", "/v1/models"},+ } {+ p, ok := engineProxyProfileFor(tc.engine)+ if !ok {+ t.Fatalf("no profile for %s", tc.engine)+ }+ if p.HealthProbePath != tc.want {+ t.Errorf("%s health probe = %q, want %q", tc.engine, p.HealthProbePath, tc.want)+ }+ }+}++// proxyport.go and lmstudioport.go restate ports and error-ID prefixes the+// engine table already carries, because they feed int32 atomics throughout the+// package. They are deleted when those files collapse; until then this is what+// stops them drifting from the table.+func TestBrokerConstantsMatchTheEngineTable(t *testing.T) {+ for _, tc := range []struct {+ engine string+ facade int+ backendStart int+ blockedID string+ }{+ {"ollama", managedOllamaFacadePort, managedOllamaBackendStart, portOwnershipBlockedID},+ {"lmstudio", managedLMStudioFacadePort, managedLMStudioBackendStart, lmstudioPortOwnershipBlockedID},+ } {+ t.Run(tc.engine, func(t *testing.T) {+ p, ok := engineProxyProfileFor(tc.engine)+ if !ok {+ t.Fatalf("no profile for %s", tc.engine)+ }+ if tc.facade != p.FacadePort {+ t.Errorf("facade constant = %d, table says %d", tc.facade, p.FacadePort)+ }+ if tc.backendStart != p.EnginePortBase {+ t.Errorf("backend-start constant = %d, table says %d", tc.backendStart, p.EnginePortBase)+ }+ if want := p.ComponentName() + ":port-ownership-blocked"; tc.blockedID != want {+ t.Errorf("blocked error id = %q, want %q", tc.blockedID, want)+ }+ })+ }+ if want := ollamaProxyProfile.ComponentName() + ":port-bumped"; proxyPortBumpedID != want {+ t.Errorf("bumped error id = %q, want %q", proxyPortBumpedID, want)+ }+}++// Ownership is the one judgment call in adding an engine, so the two values in+// the table today are pinned explicitly. Getting these backwards does not fail+// to compile — it silently changes which engine the broker believes it may stop.+func TestEngineOwnershipAssignments(t *testing.T) {+ for _, tc := range []struct {+ engine string+ want engineOwnership+ }{+ {"ollama", adoptedEngine},+ {"lmstudio", managedEngine},+ } {+ p, ok := engineProxyProfileFor(tc.engine)+ if !ok {+ t.Fatalf("no profile for %s", tc.engine)+ }+ if p.Ownership != tc.want {+ t.Errorf("%s ownership = %v, want %v", tc.engine, p.Ownership, tc.want)+ }+ }+}++// The two engines' port choreography diverges on exactly one input, and this is+// it: the engine is running on its own facade port, so the port is unavailable.+//+// An adopted engine must block — the broker cannot tell its own engine from a+// stranger there, and has authority over neither. A managed engine must plan+// the move, because engine-manager can stop and reposition it; blocking instead+// would break the most common LM Studio install, where it is already running on+// 1234.+//+// This is the regression guard for collapsing the two planners into one. A+// change that makes both engines agree here has broken one of them.+func TestOwnershipDecidesTheOccupiedFacadeOutcome(t *testing.T) {+ for _, tc := range []struct {+ engine string+ wantMove bool+ wantBlock string+ }{+ {engine: "ollama", wantBlock: "Ollama is already running on the compatibility port"},+ {engine: "lmstudio", wantMove: true},+ } {+ t.Run(tc.engine, func(t *testing.T) {+ p, ok := engineProxyProfileFor(tc.engine)+ if !ok {+ t.Fatalf("no profile for %s", tc.engine)+ }+ // Running on the facade, which is therefore taken; the backend base+ // is free.+ status := ollamaPortStatus{Running: true, Port: p.FacadePort}+ available := func(port int) bool { return port == p.EnginePortBase }++ got := planManagedEnginePorts(p, true, status, available)++ if tc.wantMove {+ want := managedPortPlan{Enabled: true, BackendPort: p.EnginePortBase}+ if got != want {+ t.Fatalf("plan = %+v, want %+v (a managed engine must be moved, not refused)", got, want)+ }+ return+ }+ if got.Blocked != tc.wantBlock {+ t.Fatalf("plan = %+v, want blocked with %q (an adopted engine must not be displaced)", got, tc.wantBlock)+ }+ })+ }+}++// --proxy-engines selects which engines the one binary is started for.+func TestParseProxyEngines(t *testing.T) {+ for _, tc := range []struct {+ name string+ csv string+ want []string+ wantErr bool+ }{+ {name: "default is every engine", csv: "ollama,lmstudio", want: []string{"ollama", "lmstudio"}},+ {name: "single engine", csv: "lmstudio", want: []string{"lmstudio"}},+ {name: "whitespace and blanks are tolerated", csv: " ollama , , lmstudio ", want: []string{"ollama", "lmstudio"}},+ {name: "duplicates collapse", csv: "ollama,ollama", want: []string{"ollama"}},+ {name: "empty selects nothing", csv: "", want: nil},+ // Silently fronting the engines it did recognize would look like the+ // flag worked.+ {name: "unknown engine fails", csv: "ollama,vllm", wantErr: true},+ } {+ t.Run(tc.name, func(t *testing.T) {+ got, err := parseProxyEngines(tc.csv)+ if tc.wantErr {+ if err == nil {+ t.Fatalf("parseProxyEngines(%q) = %v, want an error", tc.csv, got)+ }+ return+ }+ if err != nil {+ t.Fatalf("parseProxyEngines(%q): %v", tc.csv, err)+ }+ if len(got) != len(tc.want) {+ t.Fatalf("parseProxyEngines(%q) = %v, want %v", tc.csv, got, tc.want)+ }+ for i := range tc.want {+ if got[i] != tc.want[i] {+ t.Fatalf("parseProxyEngines(%q) = %v, want %v", tc.csv, got, tc.want)+ }+ }+ })+ }+}++// An engine left out of --proxy-engines is not started, and neither is any+// engine when the binary could not be resolved.+func TestProxyEnabledHonorsSelectionAndBinary(t *testing.T) {+ b := &Broker{proxyPath: "/path/to/nvpair-proxy", proxyEngines: []string{"ollama"}}+ if !b.proxyEnabled(ollamaProxyProfile) {+ t.Error("ollama was selected but is not enabled")+ }+ if b.proxyEnabled(lmstudioProxyProfile) {+ t.Error("lmstudio was not selected but is enabled")+ }++ b = &Broker{proxyEngines: []string{"ollama", "lmstudio"}}+ if b.proxyEnabled(ollamaProxyProfile) {+ t.Error("no proxy binary resolved, but the engine is enabled")+ }+}++// Preparing a facade is not read-only. For a managed engine the backend move+// runs inside preparation, so preparing an engine whose proxy is then never+// started relocates the engine off its own stock port and leaves nothing+// serving it — every existing client on that port breaks.+//+// Ollama cannot show the symptom, because its move is deferred until its proxy+// proves it holds the facade. That asymmetry is exactly why the guard has to be+// at the preparation call rather than relied on further down.+func TestDeselectedEngineIsNotPrepared(t *testing.T) {+ settings, settingsCodec := newTestRPCWorkerPipe(t)+ engine, engineCodec := newTestRPCWorkerPipe(t)++ // Every preparation begins by reading the managed-port policy, so counting+ // those reads is what proves how many engines were prepared — one for+ // Ollama and no more. Asserting on engine:set-port instead would depend on+ // whether 1234/1235 happen to be free on this machine, and would pass for+ // the wrong reason whenever they are not.+ policyReads := make(chan struct{}, 4)+ go func() {+ for {+ msg, err := settingsCodec.Read()+ if err != nil {+ return+ }+ if msg.Method == "settings/get-force-ports" {+ policyReads <- struct{}{}+ }+ _ = settingsCodec.Respond(msg.ID, map[string]bool{"value": true})+ }+ }()+ go func() {+ for {+ msg, err := engineCodec.Read()+ if err != nil {+ return+ }+ _ = engineCodec.Respond(msg.ID, json.RawMessage(`{}`))+ }+ }()++ b := &Broker{+ nodeID: "local-node",+ proxyPath: "/path/to/nvpair-proxy",+ proxyEngines: []string{"ollama"},+ ollamaPortReady: make(chan struct{}),+ lmstudioPortReady: make(chan struct{}),+ }+ b.setSettings(settings)+ b.setEngineMgr(engine)++ b.prepareEnabledFacades()++ select {+ case <-policyReads:+ case <-time.After(2 * time.Second):+ t.Fatal("the selected engine was never prepared; the fixture is not exercising preparation")+ }+ select {+ case <-policyReads:+ t.Fatal("a deselected engine was prepared; its backend would be relocated with no proxy to claim the port")+ case <-time.After(300 * time.Millisecond):+ }+}++// A stranger on the facade blocks both engines. This is the companion to the+// test above: the divergence is about who the occupant is, not about whether an+// occupied facade matters.+func TestAStrangerOnTheFacadeBlocksEveryEngine(t *testing.T) {+ for _, p := range engineProxyProfiles {+ t.Run(p.Name, func(t *testing.T) {+ // The engine is stopped somewhere else entirely, so whoever holds+ // the facade is not it.+ status := ollamaPortStatus{Port: p.EnginePortBase + 100}+ available := func(port int) bool { return port != p.FacadePort }++ got := planManagedEnginePorts(p, true, status, available)++ if got.Blocked != "the compatibility port is already in use" {+ t.Fatalf("plan = %+v, want blocked on the occupied facade", got)+ }+ })+ }+}diff --git a/services/nvpair-ui-broker/lmstudioport.go b/services/nvpair-ui-broker/lmstudioport.goindex a43509c..783e21f 100644--- a/services/nvpair-ui-broker/lmstudioport.go+++ b/services/nvpair-ui-broker/lmstudioport.go@@ -12,38 +12,24 @@ import ("nvpair-shared/errors")+// Restates descriptor values; see the equivalent block in proxyport.go.const (managedLMStudioFacadePort = 1234managedLMStudioBackendStart = 1235lmstudioPortOwnershipBlockedID = "lmstudio-proxy:port-ownership-blocked")-// planManagedLMStudioPorts keeps the ownership policy deterministic. LM Studio-// is the one engine that engine-manager may move while running: its identified-// command-mode runtime has an official stop command. Process-mode and unknown+// lmstudioProxyProfile is the descriptor entry this file's callers plan+// against. See ollamaProxyProfile.+var lmstudioProxyProfile = mustEngineProxyProfile("lmstudio")++// planManagedLMStudioPorts plans LM Studio's managed ports. LM Studio is a+// managed engine — engine-manager may stop and reposition its identified+// command-mode runtime while it is running — so the shared policy moves the+// backend first and judges the facade afterwards. Process-mode and unknown// owners are still refused by engine:set-port.func planManagedLMStudioPorts(enabled bool, st ollamaPortStatus, available func(int) bool) managedPortPlan {- if !enabled {- return managedPortPlan{}- }- if st.Port > 0 && st.Port != managedLMStudioFacadePort {- if !available(managedLMStudioFacadePort) {- return managedPortPlan{Blocked: "the compatibility port is already in use"}- }- if !st.Running && !available(st.Port) {- backend := nextAvailablePort(managedLMStudioBackendStart, available)- if backend == 0 {- return managedPortPlan{Blocked: "no free backend port is available"}- }- return managedPortPlan{Enabled: true, BackendPort: backend}- }- return managedPortPlan{Enabled: true}- }- backend := nextAvailablePort(managedLMStudioBackendStart, available)- if backend == 0 {- return managedPortPlan{Blocked: "no free backend port is available"}- }- return managedPortPlan{Enabled: true, BackendPort: backend}+ return planManagedEnginePorts(lmstudioProxyProfile, enabled, st, available)}func (b *Broker) markLMStudioPortReady() {@@ -86,11 +72,16 @@ func (b *Broker) lmstudioPortOwnershipPending() bool {}}+// needsLMStudioPortGate mirrors needsOllamaPortGate: these are the client+// requests that can probe and adopt the configured port, so they must wait+// while :1234 is changing hands. engine:install is gated for the same reason+// as the rest — an install that starts the engine assigns it a port, and doing+// that mid-transition can land on the port the proxy is about to take.func needsLMStudioPortGate(method string, params json.RawMessage) bool {if method == "engine:get-installed" {return true}- if method != "engine:status" && method != "engine:start" && method != "engine:restart" {+ if method != "engine:status" && method != "engine:install" && method != "engine:start" && method != "engine:restart" {return false}var request struct {@@ -128,7 +119,10 @@ func (b *Broker) setLMStudioProxyFallback(excludedPorts ...int) int {if aliasPort := b.currentOllamaHostAlias().Port; aliasPort > 0 {excludedPorts = append(excludedPorts, aliasPort)}- if backend := int(b.lmstudioBackendPort.Load()); backend > 0 {+ for port := range b.siblingEngineProxyPorts(lmstudioProxyProfile) {+ excludedPorts = append(excludedPorts, port)+ }+ if backend := int(b.lmstudioState().backendPort.Load()); backend > 0 {excludedPorts = append(excludedPorts, backend)} else {// With no authoritative backend yet, neither LM Studio's compatibility@@ -137,7 +131,7 @@ func (b *Broker) setLMStudioProxyFallback(excludedPorts ...int) int {excludedPorts = append(excludedPorts, managedLMStudioFacadePort, managedLMStudioBackendStart)}fallback := nextAvailablePortExcluding(managedLMStudioBackendStart, excludedPorts, tcpPortAvailable)- b.lmstudioProxyStartupPort.Store(int32(fallback))+ b.lmstudioState().startupPort.Store(int32(fallback))return fallback}@@ -146,7 +140,7 @@ func (b *Broker) rebindLMStudioProxy(p *proxyProcess, port int) bool {return false}body, _ := json.Marshal(map[string]int{"port": port})- result, rpcErr, err := p.Call(context.Background(), "set-port", body)+ result, rpcErr, err := p.Call(context.Background(), lmstudioProxyProfile.addressed("set-port"), body)if err != nil || rpcErr != nil {slog.Warn("failed to rebind LM Studio proxy", "port", port, "err", err, "rpcErr", rpcErr)return false@@ -159,7 +153,7 @@ func (b *Broker) rebindLMStudioProxy(p *proxyProcess, port int) bool {// rebinds a live proxy. It does not open the ownership gate: only a confirmed// bound proxy generation or an exhausted supervisor may do that.func (b *Broker) blockManagedLMStudioFacade(reason string, p *proxyProcess, excludedPorts ...int) (int, bool) {- b.managedLMStudioFacade.Store(false)+ b.lmstudioState().managedFacade.Store(false)fallback := b.setLMStudioProxyFallback(excludedPorts...)b.reportLMStudioPortOwnershipBlocked(reason)return fallback, b.rebindLMStudioProxy(p, fallback)@@ -182,13 +176,13 @@ func (b *Broker) cacheLMStudioPortStatus() (ollamaPortStatus, bool) {if st.Port <= 0 {return st, false}- b.lmstudioBackendPort.Store(int32(st.Port))+ b.lmstudioState().backendPort.Store(int32(st.Port))return st, true}func (b *Broker) configureUnmanagedLMStudioFacade() {- b.managedLMStudioFacade.Store(false)- b.lmstudioProxyStartupPort.Store(0)+ b.lmstudioState().managedFacade.Store(false)+ b.lmstudioState().startupPort.Store(0)b.forwardErrorsClear(lmstudioPortOwnershipBlockedID)}@@ -257,7 +251,7 @@ func (b *Broker) prepareManagedLMStudioFacadeWithPortCheck(portAvailable func(inreturn}if st.Port > 0 {- b.lmstudioBackendPort.Store(int32(st.Port))+ b.lmstudioState().backendPort.Store(int32(st.Port))}if !policy.Value {b.configureUnmanagedLMStudioFacade()@@ -277,10 +271,10 @@ func (b *Broker) prepareManagedLMStudioFacadeWithPortCheck(portAvailable func(in_, _ = b.blockManagedLMStudioFacade("the LM Studio backend could not be moved", nil, st.Port, plan.BackendPort)return}- b.lmstudioBackendPort.Store(int32(plan.BackendPort))+ b.lmstudioState().backendPort.Store(int32(plan.BackendPort))var moved ollamaPortStatusif json.Unmarshal(result, &moved) == nil && moved.Port > 0 {- b.lmstudioBackendPort.Store(int32(moved.Port))+ b.lmstudioState().backendPort.Store(int32(moved.Port))}}if !portAvailable(managedLMStudioFacadePort) {@@ -288,8 +282,8 @@ func (b *Broker) prepareManagedLMStudioFacadeWithPortCheck(portAvailable func(inreturn}- b.managedLMStudioFacade.Store(plan.Enabled)- b.lmstudioProxyStartupPort.Store(managedLMStudioFacadePort)+ b.lmstudioState().managedFacade.Store(plan.Enabled)+ b.lmstudioState().startupPort.Store(int32(managedLMStudioFacadePort))b.forwardErrorsClear(lmstudioPortOwnershipBlockedID)}@@ -299,30 +293,37 @@ func (b *Broker) lmstudioProxyGenerationIsCurrent(generation uint64, p *proxyProb.getLMStudioProxy() == p}-// invalidateLMStudioProxyForRestartLocked invalidates the current logical-// generation and broker-visible handle. Caller holds lmstudioReadyMu.-func (b *Broker) invalidateLMStudioProxyForRestartLocked(generation uint64) bool {+// rebindLMStudioFacadeOrFinish moves LM Studio's facade to another port when+// the port just asked for could not be bound, and gives up on that engine alone+// if the second attempt fails too.+//+// This replaces a process restart. What the restart actually bought was a fresh+// port plan — it re-ran startup planning and so picked a different fallback —+// and the process death was incidental. It is also unusable once one process+// hosts every facade: restarting to fix LM Studio's port would drop Ollama's+// listener and the inference in flight on it.+//+// Simply skipping the restart is not enough either. That would leave the facade+// parked on a port it cannot serve with its ownership gate never opening, so+// every engine:status for LM Studio would wait out the call timeout and answer+// "retry" for the life of the process. Hence: try elsewhere, then finish.+//+// Caller holds lmstudioReadyMu. Reports whether the facade is now serving a+// port the caller may keep reconciling against.+func (b *Broker) rebindLMStudioFacadeOrFinish(p *proxyProcess, generation uint64, avoid ...int) (int, bool) {if b.lmstudioProxyGeneration.Load() != generation {- return false- }- b.lmstudioProxyGeneration.Add(1)- b.lmstudioProxyPublishedGeneration.Store(0)- b.setLMStudioProxy(nil)- return true-}--func (b *Broker) restartLMStudioProxyOrFinish(generation uint64) {- // Caller holds lmstudioReadyMu. Invalidate before the restart request- // becomes observable so a ready notification already queued by the failed- // process cannot complete the gate.- if !b.invalidateLMStudioProxyForRestartLocked(generation) {- return+ return 0, false}- if b.lmstudioProxySup != nil {- b.lmstudioProxySup.Restart()- return+ fallback := b.setLMStudioProxyFallback(avoid...)+ if fallback != 0 && b.rebindLMStudioProxy(p, fallback) {+ return fallback, true}+ // Nothing left to try for this engine. Release its gate so callers stop+ // waiting on it; every other facade keeps serving.+ slog.Warn("LM Studio facade could not be rebound; releasing its ownership gate",+ "attempted", fallback, "avoided", avoid)b.finishLMStudioProxyTerminal()+ return 0, false}func (b *Broker) reconcileLMStudioProxyAfterEngineManagerReady() {@@ -333,7 +334,7 @@ func (b *Broker) reconcileLMStudioProxyAfterEngineManagerReady() {if p == nil {return}- ready, port := p.Status()+ ready, port := p.Status(lmstudioProxyProfile.Name)generation := b.lmstudioProxyPublishedGeneration.Load()if !ready || port <= 0 || generation != b.lmstudioProxyGeneration.Load() {return@@ -362,20 +363,23 @@ func (b *Broker) reconcileLMStudioProxyPortOnReadyForGeneration(generation uint6if p == nil || !b.lmstudioProxyGenerationIsCurrent(generation, p) {return}- if b.managedLMStudioFacade.Load() && boundPort != managedLMStudioFacadePort {+ if b.lmstudioState().managedFacade.Load() && boundPort != managedLMStudioFacadePort {if b.rebindLMStudioProxy(p, managedLMStudioFacadePort) {boundPort = managedLMStudioFacadePort} else {fallback, rebound := b.blockManagedLMStudioFacade("the proxy could not bind the compatibility port", p, boundPort)if !rebound {- b.restartLMStudioProxyOrFinish(generation)- return+ next, ok := b.rebindLMStudioFacadeOrFinish(p, generation, boundPort, fallback)+ if !ok {+ return+ }+ fallback = next}boundPort = fallback}}- backend := int(b.lmstudioBackendPort.Load())+ backend := int(b.lmstudioState().backendPort.Load())if backend == 0 {// Fail closed on the bundled backend default before asking// engine-manager for status. Its identity probe is also rejected by the@@ -383,8 +387,11 @@ func (b *Broker) reconcileLMStudioProxyPortOnReadyForGeneration(generation uint6if boundPort == managedLMStudioBackendStart {fallback := b.setLMStudioProxyFallback(boundPort)if !b.rebindLMStudioProxy(p, fallback) {- b.restartLMStudioProxyOrFinish(generation)- return+ next, ok := b.rebindLMStudioFacadeOrFinish(p, generation, boundPort, fallback)+ if !ok {+ return+ }+ fallback = next}boundPort = fallback}@@ -396,15 +403,18 @@ func (b *Broker) reconcileLMStudioProxyPortOnReadyForGeneration(generation uint6}var fallback intvar rebound bool- if b.managedLMStudioFacade.Load() {+ if b.lmstudioState().managedFacade.Load() {fallback, rebound = b.blockManagedLMStudioFacade("the proxy bound the configured LM Studio backend port", p, boundPort)} else {fallback = b.setLMStudioProxyFallback(boundPort)rebound = b.rebindLMStudioProxy(p, fallback)}if !rebound {- b.restartLMStudioProxyOrFinish(generation)- return false+ next, ok := b.rebindLMStudioFacadeOrFinish(p, generation, boundPort, fallback, backend)+ if !ok {+ return false+ }+ fallback = next}boundPort = fallbackreturn true@@ -420,7 +430,7 @@ func (b *Broker) reconcileLMStudioProxyPortOnReadyForGeneration(generation uint6if !b.lmstudioProxyGenerationIsCurrent(generation, p) {return}- cached := int(b.lmstudioBackendPort.Load())+ cached := int(b.lmstudioState().backendPort.Load())if cached <= 0 {// A bound proxy plus an unknown configured backend is not a terminal// ownership result. Keep restoration gated until engine-manager (or a@@ -441,5 +451,4 @@ func (b *Broker) reconcileLMStudioProxyPortOnReadyForGeneration(generation uint6return}b.markLMStudioPortReady()- b.repushPriority("lmstudio")}diff --git a/services/nvpair-ui-broker/lmstudioport_test.go b/services/nvpair-ui-broker/lmstudioport_test.goindex 7d19c6d..f33251e 100644--- a/services/nvpair-ui-broker/lmstudioport_test.go+++ b/services/nvpair-ui-broker/lmstudioport_test.go@@ -11,6 +11,20 @@ import ("reflect""testing""time"++ "nvpair-shared/engines"+)++// The wire names of the facade-scoped methods the broker sends LM Studio's+// proxy. Asserted in their addressed form on purpose: a regression that drops+// the address would otherwise reach a process hosting several facades and be+// applied to whichever one happened to be first.+//+// engine:status and the engine-manager restore method are not here — they go to+// engine-manager, which has no facades and takes no address.+var (+ lmstudioSetPort = lmstudioProxyProfile.addressed("set-port")+ lmstudioSetLocalBackend = lmstudioProxyProfile.addressed("node/set-local-backend"))func TestPlanManagedLMStudioPorts(t *testing.T) {@@ -45,17 +59,34 @@ func TestPlanManagedLMStudioPorts(t *testing.T) {}}-func TestManagedLMStudioProxyStartupArgs(t *testing.T) {+// The engine, its port, and the ignore-persisted decision travel in+// facade/enable rather than on argv, because the broker plans a different port+// per engine and a single-valued flag cannot carry two plans.+func TestManagedLMStudioFacadeSpec(t *testing.T) {b := &Broker{}- b.lmstudioProxyStartupPort.Store(managedLMStudioFacadePort)- want := []string{"--port", "1234", "--ignore-persisted-port"}- if got := b.lmstudioProxyArgs(); !reflect.DeepEqual(got, want) {- t.Fatalf("managed startup args = %v, want %v", got, want)+ b.lmstudioState().startupPort.Store(managedLMStudioFacadePort)+ want := enableFacadeRequest{+ Engine: "lmstudio",+ Port: managedLMStudioFacadePort,+ IgnorePersistedPort: true,+ }+ if got := b.lmstudioFacadeSpec(); !reflect.DeepEqual(got, want) {+ t.Fatalf("managed facade spec = %+v, want %+v", got, want)+ }++ // No startup port means the child decides: its own persisted port, else the+ // engine's standalone default. Naming a port here would override the port a+ // user chose through set-port.+ b.lmstudioState().startupPort.Store(0)+ want = enableFacadeRequest{Engine: "lmstudio"}+ if got := b.lmstudioFacadeSpec(); !reflect.DeepEqual(got, want) {+ t.Fatalf("opt-out facade spec = %+v, want %+v (persisted-port behavior)", got, want)}- b.lmstudioProxyStartupPort.Store(0)- if got := b.lmstudioProxyArgs(); len(got) != 0 {- t.Fatalf("opt-out startup args = %v, want persisted-port behavior", got)+ // The alias stands in for an inherited host variable, which LM Studio does+ // not have; the child rejects alias addresses for it outright.+ if len(b.lmstudioFacadeSpec().AliasAddresses) != 0 {+ t.Fatal("LM Studio facade spec carried alias addresses")}}@@ -75,8 +106,8 @@ func TestManagedLMStudioReadyOpensGateAndPushesBackend(t *testing.T) {go proxy.peer.Serve(nil, nil)b := &Broker{lmstudioPortReady: make(chan struct{})}- b.managedLMStudioFacade.Store(true)- b.lmstudioBackendPort.Store(managedLMStudioBackendStart)+ b.lmstudioState().managedFacade.Store(true)+ b.lmstudioState().backendPort.Store(managedLMStudioBackendStart)b.setEngineMgr(engine)b.setLMStudioProxy(proxy)@@ -107,11 +138,7 @@ func TestManagedLMStudioReadyOpensGateAndPushesBackend(t *testing.T) {b.forwardLMStudioProxyNotification("ready", json.RawMessage(`{"version":"test","port":1234}`))- select {- case <-b.lmstudioPortReady:- case <-time.After(2 * time.Second):- t.Fatal("LM Studio ownership gate did not open")- }+ requireGateOpens(t, b.lmstudioPortReady, "LM Studio preparation")select {case got := <-engineMethod:if got != "engine:status" {@@ -144,8 +171,8 @@ func TestManagedLMStudioWrongReadyEntersFallbackAndWarns(t *testing.T) {go proxy.peer.Serve(nil, nil)errs := &errorsProcess{peer: NewPeer(NewCodec(errorsClient))}b := &Broker{lmstudioPortReady: make(chan struct{})}- b.managedLMStudioFacade.Store(true)- b.lmstudioBackendPort.Store(managedLMStudioBackendStart)+ b.lmstudioState().managedFacade.Store(true)+ b.lmstudioState().backendPort.Store(managedLMStudioBackendStart)b.setLMStudioProxy(proxy)b.setErrors(errs)@@ -158,7 +185,7 @@ func TestManagedLMStudioWrongReadyEntersFallbackAndWarns(t *testing.T) {if err != nil {return}- if msg.Method == "node/set-local-backend" {+ if msg.Method == lmstudioSetLocalBackend {_ = codec.Respond(msg.ID, map[string]bool{"ok": true})return}@@ -194,15 +221,11 @@ func TestManagedLMStudioWrongReadyEntersFallbackAndWarns(t *testing.T) {b.forwardLMStudioProxyNotification("ready", json.RawMessage(`{"version":"test","port":1235}`))- select {- case <-b.lmstudioPortReady:- case <-time.After(2 * time.Second):- t.Fatal("LM Studio fallback did not open the ownership gate")- }- if b.managedLMStudioFacade.Load() {+ requireGateOpens(t, b.lmstudioPortReady, "the LM Studio fallback")+ if b.lmstudioState().managedFacade.Load() {t.Fatal("managed LM Studio mode remained enabled after compatibility-port failure")}- fallback := int(b.lmstudioProxyStartupPort.Load())+ fallback := int(b.lmstudioState().startupPort.Load())if fallback == 0 || fallback == managedLMStudioFacadePort || fallback == managedLMStudioBackendStart {t.Fatalf("unsafe LM Studio fallback port %d", fallback)}@@ -254,7 +277,7 @@ func TestManagedLMStudioRequestsWaitForPortGate(t *testing.T) {codec: NewCodec(brokerClient),lmstudioPortReady: make(chan struct{}),}- b.managedLMStudioFacade.Store(true)+ b.lmstudioState().managedFacade.Store(true)b.setEngineMgr(engine)method := make(chan string, 1)@@ -317,7 +340,11 @@ func TestManagedLMStudioPortGateRequestMatcher(t *testing.T) {{"engine:status", `{"engine":"lmstudio"}`, true},{"engine:start", `{"engine":"lmstudio"}`, true},{"engine:restart", `{"engine":"lmstudio"}`, true},- {"engine:install", `{"engine":"lmstudio"}`, false},+ // Gated for the same reason Ollama gates it: an install that starts+ // the engine assigns a port, and mid-transition that can be the port+ // the proxy is taking.+ {"engine:install", `{"engine":"lmstudio"}`, true},+ {"engine:install", `{"engine":"ollama"}`, false},{"engine:status", `{"engine":"ollama"}`, false},{"engine:models", `{"engine":"lmstudio"}`, false},{"engine:status", `{`, false},@@ -342,7 +369,7 @@ func TestManagedLMStudioPortGateHonorsCancellation(t *testing.T) {func TestManagedLMStudioFallbackRemainsPendingUntilGateCloses(t *testing.T) {b := &Broker{lmstudioPortReady: make(chan struct{})}- b.managedLMStudioFacade.Store(false) // fallback disables managed mode first+ b.lmstudioState().managedFacade.Store(false) // fallback disables managed mode firstif !b.lmstudioPortOwnershipPending() {t.Fatal("fallback became observable before its proxy rebind completed")}@@ -365,8 +392,8 @@ func TestManagedLMStudioConcurrentReadyWaitsForFallbackRebind(t *testing.T) {proxy := &proxyProcess{peer: NewPeer(NewCodec(proxyClient))}go proxy.peer.Serve(nil, nil)b := &Broker{lmstudioPortReady: make(chan struct{})}- b.managedLMStudioFacade.Store(true)- b.lmstudioBackendPort.Store(managedLMStudioBackendStart)+ b.lmstudioState().managedFacade.Store(true)+ b.lmstudioState().backendPort.Store(managedLMStudioBackendStart)b.setLMStudioProxy(proxy)b.setErrors(&errorsProcess{peer: NewPeer(NewCodec(errorsClient))})@@ -381,7 +408,7 @@ func TestManagedLMStudioConcurrentReadyWaitsForFallbackRebind(t *testing.T) {}count++requests <- msg.Method- if msg.Method == "set-port" && count == 1 {+ if msg.Method == lmstudioSetPort && count == 1 {_ = codec.RespondError(msg.ID, -32000, "occupied")} else {_ = codec.Respond(msg.ID, proxyReadyParams{Port: managedLMStudioBackendStart + 1})@@ -392,14 +419,14 @@ func TestManagedLMStudioConcurrentReadyWaitsForFallbackRebind(t *testing.T) {b.forwardLMStudioProxyNotification("ready", json.RawMessage(`{"port":1235}`))select {case method := <-requests:- if method != "set-port" {- t.Fatalf("first proxy request = %q, want set-port", method)+ if method != lmstudioSetPort {+ t.Fatalf("first proxy request = %q, want %q", method, lmstudioSetPort)}case <-time.After(2 * time.Second):t.Fatal("missing compatibility-port rebind")}deadline := time.After(2 * time.Second)- for b.managedLMStudioFacade.Load() {+ for b.lmstudioState().managedFacade.Load() {select {case <-deadline:t.Fatal("managed mode was not disabled after rebind failure")@@ -411,23 +438,22 @@ func TestManagedLMStudioConcurrentReadyWaitsForFallbackRebind(t *testing.T) {// The first reconciliation is now blocked writing its warning. A duplicate// ready must not overtake it and release the gate before the fallback move.b.forwardLMStudioProxyNotification("ready", json.RawMessage(`{"port":1235}`))- select {- case <-b.lmstudioPortReady:- t.Fatal("duplicate ready opened the gate before the fallback rebind")- case <-time.After(100 * time.Millisecond):- }+ requireGateStaysShut(t, b.lmstudioPortReady, "a duplicate ready before the fallback rebind")if _, err := NewCodec(errorsServer).Read(); err != nil {t.Fatalf("read fallback warning: %v", err)}- select {- case <-b.lmstudioPortReady:- case <-time.After(2 * time.Second):- t.Fatal("fallback rebind did not open the gate")- }+ requireGateOpens(t, b.lmstudioPortReady, "the fallback rebind")}-func TestManagedLMStudioFailedRebindInvalidatesQueuedReady(t *testing.T) {+// The whole reconciliation path, driven by a ready notification, when no port+// the broker offers can be bound: it exhausts its fallbacks and then gives up+// on LM Studio alone.+//+// This used to end in a supervised restart of the process. It cannot now — the+// same process hosts Ollama's facade — so the assertions are inverted: the gate+// must open anyway, and the handle and supervisor must be left alone.+func TestManagedLMStudioExhaustedFallbacksFinishOnlyThatEngine(t *testing.T) {proxyClient, proxyServer := net.Pipe()t.Cleanup(func() {_ = proxyClient.Close()@@ -437,89 +463,44 @@ func TestManagedLMStudioFailedRebindInvalidatesQueuedReady(t *testing.T) {go proxy.peer.Serve(nil, nil)b := &Broker{lmstudioPortReady: make(chan struct{})}- b.managedLMStudioFacade.Store(true)- b.lmstudioBackendPort.Store(managedLMStudioBackendStart)+ b.lmstudioState().managedFacade.Store(true)+ b.lmstudioState().backendPort.Store(managedLMStudioBackendStart)b.lmstudioProxyGeneration.Store(1)b.lmstudioProxyPublishedGeneration.Store(1)b.setLMStudioProxy(proxy)- b.lmstudioProxySup = newSupervisor("lmstudio-proxy", noRestartPolicy(), nil)+ b.proxySup = newSupervisor(engines.ProxyComponent, noRestartPolicy(), nil)- requests := make(chan string, 3)+ // Refuse every port offered, so the fallback chain runs to exhaustion.+ attempts := make(chan string, 8)go func() {codec := NewCodec(proxyServer)- for i := 0; i < 3; i++ {+ for {msg, err := codec.Read()if err != nil {return}- requests <- msg.Method- switch i {- case 0:- _ = codec.RespondError(msg.ID, -32000, "primary occupied")- case 1:- var request struct {- Port int `json:"port"`- }- _ = json.Unmarshal(msg.Params, &request)- b.forwardLMStudioProxyNotificationForGeneration(- 1,- "ready",- json.RawMessage(fmt.Sprintf(`{"port":%d}`, request.Port)),- )- // Model a lost/malformed response after the proxy emitted ready.- _ = codec.Respond(msg.ID, proxyReadyParams{})- default:- _ = codec.Respond(msg.ID, map[string]bool{"ok": true})- }+ attempts <- msg.Method+ _ = codec.RespondError(msg.ID, -32000, "occupied")}}()b.forwardLMStudioProxyNotificationForGeneration(1, "ready", json.RawMessage(`{"port":1235}`))- select {- case <-b.lmstudioProxySup.restartCh:- case <-time.After(2 * time.Second):- t.Fatal("failed fallback did not request a supervised restart")- }- select {- case <-b.lmstudioPortReady:- t.Fatal("queued old-generation ready opened the gate before replacement readiness")- case <-time.After(100 * time.Millisecond):- }- if b.lmstudioProxyGeneration.Load() == 1 {- t.Fatal("failed fallback did not invalidate the current generation")- }- if got := b.getLMStudioProxy(); got != nil {- t.Fatal("failed fallback left the invalidated proxy handle published")- }- replacementClient, replacementServer := net.Pipe()- t.Cleanup(func() {- _ = replacementClient.Close()- _ = replacementServer.Close()- })- replacement := &proxyProcess{peer: NewPeer(NewCodec(replacementClient))}- go replacement.peer.Serve(nil, nil)- replacementGeneration := b.lmstudioProxyGeneration.Add(1)- b.setLMStudioProxy(replacement)- b.lmstudioProxyPublishedGeneration.Store(replacementGeneration)- go func() {- codec := NewCodec(replacementServer)- msg, err := codec.Read()- if err == nil {- _ = codec.Respond(msg.ID, map[string]bool{"ok": true})- }- }()+ // Releasing the gate is what stops every engine:status for LM Studio from+ // waiting out the call timeout and answering "retry" forever.+ requireGateOpens(t, b.lmstudioPortReady, "exhausted LM Studio fallbacks")- fallback := b.lmstudioProxyStartupPort.Load()- b.forwardLMStudioProxyNotificationForGeneration(- replacementGeneration,- "ready",- json.RawMessage(fmt.Sprintf(`{"port":%d}`, fallback)),- )- select {- case <-b.lmstudioPortReady:- case <-time.After(2 * time.Second):- t.Fatal("replacement-generation readiness did not open the gate")+ if got := <-attempts; got != lmstudioSetPort {+ t.Errorf("first attempt was %q, want %q", got, lmstudioSetPort)+ }+ // Same handle still published: the process was not replaced, so every other+ // facade in it keeps serving. Handle identity is the observable form of+ // "no process restart".+ if b.getLMStudioProxy() != proxy {+ t.Error("exhausting one facade's ports replaced the shared proxy handle")+ }+ if b.lmstudioState().managedFacade.Load() {+ t.Error("managed mode survived exhausted fallbacks")}}@@ -584,20 +565,16 @@ func TestPrepareManagedLMStudioFacadeMovesDefaultBackend(t *testing.T) {t.Fatalf("missing preparation call %d (%s)", i, want)}}- if !b.managedLMStudioFacade.Load() {+ if !b.lmstudioState().managedFacade.Load() {t.Fatal("managed LM Studio ownership was not enabled")}- if got := b.lmstudioBackendPort.Load(); got != managedLMStudioBackendStart {+ if got := b.lmstudioState().backendPort.Load(); got != managedLMStudioBackendStart {t.Fatalf("backend port = %d, want %d", got, managedLMStudioBackendStart)}- if got := b.lmstudioProxyStartupPort.Load(); got != managedLMStudioFacadePort {+ if got := b.lmstudioState().startupPort.Load(); got != managedLMStudioFacadePort {t.Fatalf("proxy startup port = %d, want %d", got, managedLMStudioFacadePort)}- select {- case <-b.lmstudioPortReady:- t.Fatal("preparation opened the gate before proxy readiness")- default:- }+ requireGateShutNow(t, b.lmstudioPortReady, "preparation before proxy readiness")}func TestPrepareUnmanagedLMStudioPreservesPortsAndWaitsForProxy(t *testing.T) {@@ -629,34 +606,26 @@ func TestPrepareUnmanagedLMStudioPreservesPortsAndWaitsForProxy(t *testing.T) {if portChecked {t.Fatal("opt-out preparation unexpectedly probed compatibility ports")}- if b.managedLMStudioFacade.Load() {+ if b.lmstudioState().managedFacade.Load() {t.Fatal("opt-out preparation enabled managed ownership")}- if got := b.lmstudioBackendPort.Load(); got != 12400 {+ if got := b.lmstudioState().backendPort.Load(); got != 12400 {t.Fatalf("custom backend = %d, want 12400", got)}- if got := b.lmstudioProxyStartupPort.Load(); got != 0 {+ if got := b.lmstudioState().startupPort.Load(); got != 0 {t.Fatalf("opt-out forced proxy startup port %d", got)}- select {- case <-b.lmstudioPortReady:- t.Fatal("opt-out opened the gate before the persisted proxy port was known")- default:- }+ requireGateShutNow(t, b.lmstudioPortReady, "opt-out before the persisted proxy port was known")}func TestManagedLMStudioBindFailureWaitsForFallbackReady(t *testing.T) {b := &Broker{lmstudioPortReady: make(chan struct{})}- b.managedLMStudioFacade.Store(true)- b.lmstudioBackendPort.Store(managedLMStudioBackendStart)+ b.lmstudioState().managedFacade.Store(true)+ b.lmstudioState().backendPort.Store(managedLMStudioBackendStart)b.forwardLMStudioProxyNotification("error", json.RawMessage(`{"code":"bind-failed","port":1234}`))- select {- case <-b.lmstudioPortReady:- t.Fatal("bind failure opened the gate before a fallback proxy bound")- case <-time.After(100 * time.Millisecond):- }- fallback := int(b.lmstudioProxyStartupPort.Load())+ requireGateStaysShut(t, b.lmstudioPortReady, "a bind failure before a fallback proxy bound")+ fallback := int(b.lmstudioState().startupPort.Load())if fallback == 0 || fallback == managedLMStudioFacadePort || fallback == managedLMStudioBackendStart {t.Fatalf("unsafe fallback port %d", fallback)}@@ -678,24 +647,20 @@ func TestManagedLMStudioBindFailureWaitsForFallbackReady(t *testing.T) {}()b.forwardLMStudioProxyNotification("ready", json.RawMessage(fmt.Sprintf(`{"port":%d}`, fallback)))- select {- case <-b.lmstudioPortReady:- case <-time.After(2 * time.Second):- t.Fatal("fallback readiness did not open the gate")- }+ requireGateOpens(t, b.lmstudioPortReady, "fallback readiness")}func TestManagedLMStudioRestartBindFailureChoosesFallback(t *testing.T) {b := &Broker{lmstudioPortReady: make(chan struct{})}- b.managedLMStudioFacade.Store(true)- b.lmstudioBackendPort.Store(managedLMStudioBackendStart)+ b.lmstudioState().managedFacade.Store(true)+ b.lmstudioState().backendPort.Store(managedLMStudioBackendStart)b.markLMStudioPortReady() // a later supervised proxy generation failedb.forwardLMStudioProxyNotification("error", json.RawMessage(`{"code":"bind-failed","port":1234}`))- if b.managedLMStudioFacade.Load() {+ if b.lmstudioState().managedFacade.Load() {t.Fatal("restart bind failure left managed ownership enabled")}- fallback := int(b.lmstudioProxyStartupPort.Load())+ fallback := int(b.lmstudioState().startupPort.Load())if fallback == 0 || fallback == managedLMStudioFacadePort || fallback == managedLMStudioBackendStart {t.Fatalf("restart bind failure selected unsafe fallback %d", fallback)}@@ -705,11 +670,7 @@ func TestManagedLMStudioStaleReadyDoesNotOpenCurrentGate(t *testing.T) {b := &Broker{lmstudioPortReady: make(chan struct{})}b.lmstudioProxyGeneration.Store(2)b.forwardLMStudioProxyNotificationForGeneration(1, "ready", json.RawMessage(`{"port":1234}`))- select {- case <-b.lmstudioPortReady:- t.Fatal("stale proxy generation opened the current ownership gate")- case <-time.After(100 * time.Millisecond):- }+ requireGateStaysShut(t, b.lmstudioPortReady, "a stale proxy generation's ready")}func TestUnmanagedLMStudioProxyCollisionRebindsBeforeGate(t *testing.T) {@@ -722,7 +683,7 @@ func TestUnmanagedLMStudioProxyCollisionRebindsBeforeGate(t *testing.T) {go proxy.peer.Serve(nil, nil)b := &Broker{lmstudioPortReady: make(chan struct{})}- b.lmstudioBackendPort.Store(12400)+ b.lmstudioState().backendPort.Store(12400)b.setLMStudioProxy(proxy)methods := make(chan string, 2)@@ -737,14 +698,14 @@ func TestUnmanagedLMStudioProxyCollisionRebindsBeforeGate(t *testing.T) {}methods <- msg.Methodswitch msg.Method {- case "set-port":+ case lmstudioSetPort:var request struct {Port int `json:"port"`}_ = json.Unmarshal(msg.Params, &request)requestedFallback <- request.Port_ = codec.Respond(msg.ID, proxyReadyParams{Port: request.Port})- case "node/set-local-backend":+ case lmstudioSetLocalBackend:var backend proxyLocalBackend_ = json.Unmarshal(msg.Params, &backend)localBackend <- backend@@ -756,13 +717,9 @@ func TestUnmanagedLMStudioProxyCollisionRebindsBeforeGate(t *testing.T) {}()b.forwardLMStudioProxyNotification("ready", json.RawMessage(`{"port":12400}`))- select {- case <-b.lmstudioPortReady:- case <-time.After(2 * time.Second):- t.Fatal("opt-out collision did not finish proxy fallback")- }+ requireGateOpens(t, b.lmstudioPortReady, "the opt-out collision's proxy fallback")- for i, want := range []string{"set-port", "node/set-local-backend"} {+ for i, want := range []string{lmstudioSetPort, lmstudioSetLocalBackend} {select {case got := <-methods:if got != want {@@ -776,10 +733,10 @@ func TestUnmanagedLMStudioProxyCollisionRebindsBeforeGate(t *testing.T) {if fallback == 0 || fallback == 12400 {t.Fatalf("collision fallback = %d", fallback)}- if got := int(b.lmstudioProxyStartupPort.Load()); got != fallback {+ if got := int(b.lmstudioState().startupPort.Load()); got != fallback {t.Fatalf("cached proxy fallback = %d, want %d", got, fallback)}- if got := int(b.lmstudioBackendPort.Load()); got != 12400 {+ if got := int(b.lmstudioState().backendPort.Load()); got != 12400 {t.Fatalf("custom backend changed to %d", got)}if got := <-localBackend; got.Port != 12400 {@@ -798,7 +755,7 @@ func TestUnmanagedLMStudioCollisionRebindsBeforeStatusProbe(t *testing.T) {engine, engineCodec := newTestRPCWorkerPipe(t)b := &Broker{lmstudioPortReady: make(chan struct{})}- b.lmstudioBackendPort.Store(12400)+ b.lmstudioState().backendPort.Store(12400)b.setLMStudioProxy(proxy)b.setEngineMgr(engine)@@ -811,7 +768,7 @@ func TestUnmanagedLMStudioCollisionRebindsBeforeStatusProbe(t *testing.T) {return}order <- msg.Method- if msg.Method == "set-port" {+ if msg.Method == lmstudioSetPort {var request struct {Port int `json:"port"`}@@ -832,13 +789,9 @@ func TestUnmanagedLMStudioCollisionRebindsBeforeStatusProbe(t *testing.T) {}()b.forwardLMStudioProxyNotification("ready", json.RawMessage(`{"port":12400}`))- select {- case <-b.lmstudioPortReady:- case <-time.After(2 * time.Second):- t.Fatal("opt-out collision did not finish")- }+ requireGateOpens(t, b.lmstudioPortReady, "the opt-out collision")got := []string{<-order, <-order, <-order}- if got[0] != "set-port" || got[1] != "engine:status" || got[2] != "node/set-local-backend" {+ if got[0] != lmstudioSetPort || got[1] != "engine:status" || got[2] != lmstudioSetLocalBackend {t.Fatalf("collision reconciliation order = %v", got)}}@@ -875,7 +828,7 @@ func TestUnknownLMStudioBackendMovesProxyBeforeStatusProbe(t *testing.T) {return}order <- msg.Method- if msg.Method == "set-port" {+ if msg.Method == lmstudioSetPort {var request struct {Port int `json:"port"`}@@ -897,13 +850,9 @@ func TestUnknownLMStudioBackendMovesProxyBeforeStatusProbe(t *testing.T) {}()b.forwardLMStudioProxyNotification("ready", json.RawMessage(`{"port":1235}`))- select {- case <-b.lmstudioPortReady:- case <-time.After(2 * time.Second):- t.Fatal("unknown-backend fallback did not finish")- }+ requireGateOpens(t, b.lmstudioPortReady, "the unknown-backend fallback")got := []string{<-order, <-order, <-order}- if got[0] != "set-port" || got[1] != "engine:status" || got[2] != "node/set-local-backend" {+ if got[0] != lmstudioSetPort || got[1] != "engine:status" || got[2] != lmstudioSetLocalBackend {t.Fatalf("unknown-backend reconciliation order = %v", got)}fallback := <-fallbackPort@@ -957,11 +906,8 @@ func TestUnknownCustomBackendStatusFailuresKeepRestoreGated(t *testing.T) {case <-time.After(2 * time.Second):t.Fatalf("missing status attempt %d", i+1)}- select {- case <-b.lmstudioPortReady:- t.Fatalf("status failure %d released restore with unknown custom backend", i+1)- case <-time.After(100 * time.Millisecond):- }+ requireGateStaysShut(t, b.lmstudioPortReady,+ fmt.Sprintf("status failure %d with an unknown custom backend", i+1))}}@@ -972,9 +918,8 @@ func TestEngineManagerRespawnReconcilesUnknownCustomBackendBeforeRestore(t *test_ = proxyServer.Close()})proxy := &proxyProcess{- peer: NewPeer(NewCodec(proxyClient)),- ready: true,- port: 12400,+ peer: NewPeer(NewCodec(proxyClient)),+ facadeState: readyFacade(lmstudioProxyProfile.Name, 12400),}go proxy.peer.Serve(nil, nil)oldEngine, oldEngineCodec := newTestRPCWorkerPipe(t)@@ -1004,11 +949,7 @@ func TestEngineManagerRespawnReconcilesUnknownCustomBackendBeforeRestore(t *testcase <-time.After(2 * time.Second):t.Fatal("initial status failure was not exercised")}- select {- case <-b.lmstudioPortReady:- t.Fatal("initial status failure released restore")- case <-time.After(100 * time.Millisecond):- }+ requireGateStaysShut(t, b.lmstudioPortReady, "the initial status failure")newEngine, newEngineCodec := newTestRPCWorkerPipe(t)b.setEngineMgr(newEngine)@@ -1034,7 +975,7 @@ func TestEngineManagerRespawnReconcilesUnknownCustomBackendBeforeRestore(t *testreturn}order <- msg.Method- if msg.Method == "set-port" {+ if msg.Method == lmstudioSetPort {var request struct {Port int `json:"port"`}@@ -1047,31 +988,15 @@ func TestEngineManagerRespawnReconcilesUnknownCustomBackendBeforeRestore(t *test}()b.forwardEngineNotification("engine:ready", nil)- select {- case <-b.lmstudioPortReady:- case <-time.After(2 * time.Second):- t.Fatal("engine-manager respawn did not complete ownership")- }+ requireGateOpens(t, b.lmstudioPortReady, "the engine-manager respawn")if !<-restoreDone {t.Fatal("restore waiter reported cancellation")}got := []string{<-order, <-order, <-order, <-order}- want := []string{"engine:status", "set-port", "node/set-local-backend", restoreEnabledEnginesMethod}+ want := []string{"engine:status", lmstudioSetPort, lmstudioSetLocalBackend, restoreEnabledEnginesMethod}for i := range want {if got[i] != want[i] {t.Fatalf("respawn reconciliation order = %v, want %v", got, want)}}}--func newTestRPCWorkerPipe(t *testing.T) (*rpcWorker, *Codec) {- t.Helper()- client, server := net.Pipe()- t.Cleanup(func() {- _ = client.Close()- _ = server.Close()- })- worker := &rpcWorker{peer: NewPeer(NewCodec(client))}- go worker.peer.Serve(nil, nil)- return worker, NewCodec(server)-}diff --git a/services/nvpair-ui-broker/lmstudioproxy.go b/services/nvpair-ui-broker/lmstudioproxy.goindex 09903d3..2af00f7 100644--- a/services/nvpair-ui-broker/lmstudioproxy.go+++ b/services/nvpair-ui-broker/lmstudioproxy.go@@ -4,15 +4,8 @@package mainimport (- "context""encoding/json"- "fmt"- "log""log/slog"- "strings"-- "nvpair-shared/applog"- "nvpair-shared/noderec")// lmstudioproxy.go is the broker's LM Studio counterpart to its ollama-proxy@@ -25,70 +18,38 @@ import (// manual LM Studio node into routing, the same way it does for Ollama.func (b *Broker) setLMStudioProxy(p *proxyProcess) {- b.workersMu.Lock()- b.lmstudioProxy = p- b.workersMu.Unlock()+ b.setEngineProxyHandle(lmstudioProxyProfile, p)}func (b *Broker) getLMStudioProxy() *proxyProcess {- b.workersMu.Lock()- defer b.workersMu.Unlock()- return b.lmstudioProxy+ return b.engineProxyHandle(lmstudioProxyProfile)}func (b *Broker) finishLMStudioProxyTerminal() {- b.managedLMStudioFacade.Store(false)+ b.lmstudioState().managedFacade.Store(false)b.markLMStudioPortReady()}-func (b *Broker) configureLMStudioProxySupervisorCallbacks(sup *supervisor) {- sup.onCrash, sup.onRecovered = b.supervisedWorkerCallbacks("lmstudio-proxy", func() { b.setLMStudioProxy(nil) })- sup.onExhausted = func(attempt int) {- slog.Warn("lmstudio-proxy is terminally unavailable; releasing ownership gate", "attempt", attempt)- b.finishLMStudioProxyTerminal()+// lmstudioFacadeSpec is the enable request for LM Studio's facade, mirroring+// ollamaFacadeSpec. It has no alias addresses: the alias stands in for an+// inherited host variable, which only Ollama has.+func (b *Broker) lmstudioFacadeSpec() enableFacadeRequest {+ spec := enableFacadeRequest{Engine: lmstudioProxyProfile.Name}+ if port := int(b.lmstudioState().startupPort.Load()); port != 0 {+ spec.Port = port+ spec.IgnorePersistedPort = true}+ return spec}-func (b *Broker) lmstudioProxyArgs() []string {- var args []string- if port := int(b.lmstudioProxyStartupPort.Load()); port != 0 {- args = []string{"--port", fmt.Sprintf("%d", port), "--ignore-persisted-port"}+// lmstudioFallbackPort mirrors ollamaFallbackPort: prefer the port the+// bind-failed notification already chose, since that path also decides whether+// the managed facade stays in play, and recompute only if it did not run.+func (b *Broker) lmstudioFallbackPort(failed int) int {+ if planned := int(b.lmstudioState().startupPort.Load()); planned != 0 && planned != failed {+ return planned}- return append(args, b.clusterDirArgs()...)-}--// spawnLMStudioProxy is the lmstudio-proxy supervisor's spawn closure,-// mirroring spawnProxy. It reuses startProxy because the two proxies share a-// binary protocol.-func (b *Broker) spawnLMStudioProxy() (supervisedHandle, error) {- b.lmstudioReadyMu.Lock()- generation := b.lmstudioProxyGeneration.Add(1)- b.lmstudioReadyMu.Unlock()- // Thread the cluster dir so the LM Studio proxy brings up its pin-gated LAN- // mTLS ingress (and dials peers over mTLS) once this node is clustered.- pp, err := startProxy(- "lmstudio-proxy",- b.lmstudioProxyPath,- applog.LevelString(),- b.relayDir,- func(method string, params json.RawMessage) {- b.forwardLMStudioProxyNotificationForGeneration(generation, method, params)- },- b.lmstudioProxyArgs()...,- )- if err != nil {- return nil, err- }- b.setLMStudioProxy(pp)- b.lmstudioProxyPublishedGeneration.Store(generation)- // A fast child can announce ready before its handle is published. Replay- // reconciliation after publication; the gate's sync.Once makes this safe- // when the notification goroutine already handled it.- if ready, port := pp.Status(); ready && port > 0 {- go b.reconcileLMStudioProxyPortOnReadyForGeneration(generation, port)- }- slog.Info("lmstudio-proxy started", "path", b.lmstudioProxyPath, "pid", pp.cmd.Process.Pid)- return pp, nil+ return b.setLMStudioProxyFallback(failed)}// forwardLMStudioProxyNotification is the hook startProxy invokes on the@@ -107,19 +68,29 @@ func (b *Broker) forwardLMStudioProxyNotificationForGeneration(generation uint64if b.lmstudioProxyGeneration.Load() != generation {return}- if b.dispatchErrorsNotif("lmstudio-proxy", method, params) {+ // Strip the facade address before anything dispatches on the method,+ // starting with the errors relay directly below: it matches bare names.+ method, addressed := facadeMethodFor(lmstudioProxyProfile, method)+ if !addressed {+ return+ }+ if b.dispatchErrorsNotif(lmstudioProxyProfile.ComponentName(), method, params) {return}// A process can win :1234 after preparation's free-port check but before- // the proxy binds. The failed process is exiting, so set the next spawn to- // an explicit fallback without calling back into this reader goroutine.+ // the facade binds. Choose an explicit fallback here, without calling back+ // into this reader goroutine.+ //+ // The failed process is not exiting any more: this notification precedes+ // the failing enable's response, so the port chosen here is what+ // lmstudioFallbackPort finds when that enable retries in-process.if method == "error" {var ep struct {Code string `json:"code"`Port int `json:"port"`}if json.Unmarshal(params, &ep) == nil && ep.Code == "bind-failed" {- if b.managedLMStudioFacade.Load() && ep.Port == managedLMStudioFacadePort {+ if b.lmstudioState().managedFacade.Load() && ep.Port == managedLMStudioFacadePort {_, _ = b.blockManagedLMStudioFacade("another process acquired the compatibility port during startup", nil)} else {fallback := b.setLMStudioProxyFallback(ep.Port)@@ -127,65 +98,11 @@ func (b *Broker) forwardLMStudioProxyNotificationForGeneration(generation uint64}}}- if proxyWorkloadMethods[method] {- b.routeProxyWorkload(method, params)- return- }- if method == noderec.NotifyNodeActivity {- b.routeNodeActivity(params)- return- }if method == "ready" {var rp proxyReadyParamsif err := json.Unmarshal(params, &rp); err == nil && rp.Port > 0 {go b.reconcileLMStudioProxyPortOnReadyForGeneration(generation, rp.Port)}}- b.proxyMu.Lock()- subscribed := b.lmstudioProxySubscribed- b.proxyMu.Unlock()- if !subscribed {- return- }- if err := b.codec.Notify("lmstudio-proxy:"+method, params); err != nil {- slog.Warn("forward lmstudio-proxy notification failed", "method", method, "err", err)- }-}--// relayToLMStudioProxy forwards an lmstudio-proxy:<method> request to-// lmstudio-proxy as <method> (prefix stripped) and maps its response straight-// back, mirroring relayToProxy. lmstudio-proxy:shutdown is refused — the broker-// owns the proxy's lifecycle.-func (b *Broker) relayToLMStudioProxy(msg *Message) {- method := strings.TrimPrefix(msg.Method, "lmstudio-proxy:")- if method == "shutdown" {- if err := b.codec.RespondError(msg.ID, -32601, "lmstudio-proxy:shutdown is not allowed; the broker owns the proxy lifecycle"); err != nil {- log.Printf("failed to respond to lmstudio-proxy:shutdown: %v", err)- }- return- }-- p := b.getLMStudioProxy()- if p == nil {- if err := b.codec.RespondError(msg.ID, -32000, "lmstudio-proxy not available"); err != nil {- log.Printf("failed to respond to %s: %v", msg.Method, err)- }- return- }-- result, rpcErr, err := p.Call(context.Background(), method, msg.Params)- switch {- case err != nil:- if err := b.codec.RespondError(msg.ID, -32000, fmt.Sprintf("lmstudio-proxy call failed: %v", err)); err != nil {- log.Printf("failed to respond to %s: %v", msg.Method, err)- }- case rpcErr != nil:- if err := b.codec.RespondError(msg.ID, rpcErr.Code, rpcErr.Message); err != nil {- log.Printf("failed to relay lmstudio-proxy error for %s: %v", msg.Method, err)- }- default:- if err := b.codec.Respond(msg.ID, result); err != nil {- log.Printf("failed to relay lmstudio-proxy result for %s: %v", msg.Method, err)- }- }+ b.forwardEngineProxyNotification(lmstudioProxyProfile, method, params)}diff --git a/services/nvpair-ui-broker/main.go b/services/nvpair-ui-broker/main.goindex ced0d78..65e254c 100644--- a/services/nvpair-ui-broker/main.go+++ b/services/nvpair-ui-broker/main.go@@ -14,18 +14,20 @@ import ("os/signal""path/filepath""runtime"+ "strings""syscall""nvpair-shared/appdir""nvpair-shared/applog"+ "nvpair-shared/engines")func main() {ipcPath := flag.String("ipc", "", "IPC endpoint: Unix domain socket path or Windows named pipe (default: stdin/stdout)")scannerPath := flag.String("scanner-path", "", "path to nvpair-node-scanner binary (default: ./nvpair-node-scanner in the current working directory)")nodeInfoPath := flag.String("node-info-path", "", "path to nvpair-node-info binary (default: ./nvpair-node-info in the current working directory)")- proxyPath := flag.String("proxy-path", "", "path to ollama-proxy binary (default: ./ollama-proxy in the current working directory)")- lmstudioProxyPath := flag.String("lmstudio-proxy-path", "", "path to lmstudio-proxy binary (default: ./lmstudio-proxy in the current working directory)")+ proxyPath := flag.String("proxy-path", "", "path to nvpair-proxy binary (default: ./nvpair-proxy in the current working directory)")+ proxyEngines := flag.String("proxy-engines", strings.Join(engines.Names(), ","), "comma-separated engines to front with a proxy; one nvpair-proxy process hosts a facade for each entry")workloadMgrPath := flag.String("workload-manager-path", "", "path to nvpair-workload-manager binary (default: ./nvpair-workload-manager in the current working directory)")errorsPath := flag.String("errors-path", "", "path to nvpair-errors binary (default: ./nvpair-errors in the current working directory)")engineMgrPath := flag.String("engine-manager-path", "", "path to nvpair-engine-manager binary (default: ./nvpair-engine-manager in the current working directory)")@@ -110,30 +112,25 @@ func main() {resolvedNodeInfo = ""}- // ollama-proxy is auxiliary too, resolved with the same rules: an+ // nvpair-proxy is auxiliary too, resolved with the same rules: an// explicit --proxy-path that doesn't exist is a loud operator mistake// (fatal), but an absent default sibling just means the broker runs- // without a local Ollama reverse proxy.+ // without local reverse proxies.resolvedProxy, err := resolveProxyPath(*proxyPath)if err != nil {if *proxyPath != "" {fatalf("proxy binary: %v", err)}- slog.Warn("proxy binary not found; broker will run without local Ollama proxy", "err", err)+ slog.Warn("proxy binary not found; broker will run without local engine proxies", "err", err)resolvedProxy = ""}- // lmstudio-proxy is auxiliary too, resolved with the same rules: an- // explicit --lmstudio-proxy-path that doesn't exist is a loud operator- // mistake (fatal), but an absent default sibling just means the broker- // runs without a local LM Studio reverse proxy.- resolvedLMStudioProxy, err := resolveLMStudioProxyPath(*lmstudioProxyPath)+ // One process, a facade per selected engine. An unknown name is an operator+ // mistake worth failing on rather than silently fronting fewer engines+ // than asked for.+ selectedEngines, err := parseProxyEngines(*proxyEngines)if err != nil {- if *lmstudioProxyPath != "" {- fatalf("lmstudio-proxy binary: %v", err)- }- slog.Warn("lmstudio-proxy binary not found; broker will run without local LM Studio proxy", "err", err)- resolvedLMStudioProxy = ""+ fatalf("proxy engines: %v", err)}// nvpair-workload-manager is auxiliary too, resolved with the same rules:@@ -249,18 +246,18 @@ func main() {codec := NewCodec(transport)paths := workerPaths{- scanner: resolvedScanner,- nodeInfo: resolvedNodeInfo,- proxy: resolvedProxy,- lmstudioProxy: resolvedLMStudioProxy,- workloadMgr: resolvedWorkloadMgr,- errors: resolvedErrors,- engineMgr: resolvedEngineMgr,- manualNodes: resolvedManualNodes,- settings: resolvedSettings,- clusterMgr: resolvedClusterMgr,- scheduler: resolvedScheduler,- clusterDir: clusterDir,+ scanner: resolvedScanner,+ nodeInfo: resolvedNodeInfo,+ proxy: resolvedProxy,+ proxyEngines: selectedEngines,+ workloadMgr: resolvedWorkloadMgr,+ errors: resolvedErrors,+ engineMgr: resolvedEngineMgr,+ manualNodes: resolvedManualNodes,+ settings: resolvedSettings,+ clusterMgr: resolvedClusterMgr,+ scheduler: resolvedScheduler,+ clusterDir: clusterDir,}if err := NewBroker(codec, paths).Serve(ctx); err != nil && ctx.Err() == nil {fatalf("broker error: %v", err)@@ -304,20 +301,36 @@ func resolveNodeInfoPath(override string) (string, error) {return resolveSiblingBinary(override, "nvpair-node-info", "--node-info-path")}-// resolveProxyPath mirrors resolveNodeInfoPath for the ollama-proxy binary-// the broker supervises. Like node-info its result is optional at the call-// site: a not-found default sibling degrades to "no local Ollama proxy"-// rather than aborting the broker.+// resolveProxyPath mirrors resolveNodeInfoPath for the nvpair-proxy binary the+// broker supervises. One process fronts every engine: it is spawned once and+// then asked for a facade per enabled engine over facade/enable. Like node-info+// its result is optional at the call site: a not-found default sibling degrades+// to "no local engine proxies" rather than aborting the broker.func resolveProxyPath(override string) (string, error) {- return resolveSiblingBinary(override, "ollama-proxy", "--proxy-path")+ return resolveSiblingBinary(override, "nvpair-proxy", "--proxy-path")}-// resolveLMStudioProxyPath mirrors resolveProxyPath for the lmstudio-proxy-// binary the broker supervises. Like the Ollama proxy its result is optional-// at the call site: a not-found default sibling degrades to "no local LM-// Studio proxy" rather than aborting the broker.-func resolveLMStudioProxyPath(override string) (string, error) {- return resolveSiblingBinary(override, "lmstudio-proxy", "--lmstudio-proxy-path")+// parseProxyEngines narrows the --proxy-engines list against the shared engine+// table. An unknown name fails rather than being skipped: the operator asked+// for an engine that does not exist, and quietly fronting the others would+// look like the flag worked.+func parseProxyEngines(csv string) ([]string, error) {+ seen := map[string]bool{}+ var out []string+ for _, raw := range strings.Split(csv, ",") {+ name := strings.TrimSpace(raw)+ if name == "" {+ continue+ }+ if _, ok := engines.ByName(name); !ok {+ return nil, fmt.Errorf("unknown engine %q; known engines are %s", name, strings.Join(engines.Names(), ", "))+ }+ if !seen[name] {+ seen[name] = true+ out = append(out, name)+ }+ }+ return out, nil}// resolveWorkloadManagerPath mirrors resolveProxyPath for thediff --git a/services/nvpair-ui-broker/manualnodes.go b/services/nvpair-ui-broker/manualnodes.goindex 47fe0ff..e078687 100644--- a/services/nvpair-ui-broker/manualnodes.go+++ b/services/nvpair-ui-broker/manualnodes.go@@ -9,6 +9,7 @@ import ("log/slog""time"+ "nvpair-shared/engines""nvpair-shared/noderec")@@ -198,14 +199,18 @@ func (b *Broker) removeManualNodeFromProxies(id string) {b.callProxyManual(b.getLMStudioProxy(), "lmstudio", "node/remove-manual", map[string]string{"id": id}, id)}-// callProxyManual issues a best-effort node/add-manual|remove-manual to a-// proxy. The bridge is advisory plumbing the broker owns, not part of the-// manual-node request's own response, so a failure is logged and swallowed+// callProxyManual issues a best-effort node/add-manual|remove-manual to one+// engine's facade. The bridge is advisory plumbing the broker owns, not part of+// the manual-node request's own response, so a failure is logged and swallowed// rather than surfaced to the client. A nil proxy (not supervised) is a no-op.// Runs synchronously on the manual-nodes reader goroutine; a proxy answers// these control-plane calls locally in well under proxyCallTimeout, and// manual-node events are infrequent (one per 10s probe), so the brief inline// call keeps add/remove strictly ordered without a queue.+//+// Every method routed here is facade-scoped — a manual node belongs to one+// engine's routing set — so the engine is addressed on the wire rather than+// left implicit in which process was picked.func (b *Broker) callProxyManual(p *proxyProcess, engine, method string, params any, id string) {if p == nil {return@@ -215,7 +220,7 @@ func (b *Broker) callProxyManual(p *proxyProcess, engine, method string, paramsslog.Warn("manual->proxy bridge: marshal failed", "engine", engine, "method", method, "id", id, "err", err)return}- if _, rpcErr, err := p.Call(context.Background(), method, raw); err != nil {+ if _, rpcErr, err := p.Call(context.Background(), engines.AddressMethod(engine, method), raw); err != nil {slog.Warn("manual->proxy bridge failed", "engine", engine, "method", method, "id", id, "err", err)} else if rpcErr != nil {slog.Warn("manual->proxy bridge rejected", "engine", engine, "method", method, "id", id, "code", rpcErr.Code, "msg", rpcErr.Message)diff --git a/services/nvpair-ui-broker/nodeactivity_test.go b/services/nvpair-ui-broker/nodeactivity_test.goindex dbd3f95..f1e0bd3 100644--- a/services/nvpair-ui-broker/nodeactivity_test.go+++ b/services/nvpair-ui-broker/nodeactivity_test.go@@ -219,7 +219,7 @@ func TestActivityIsNotForwardedToClients(t *testing.T) {b := &Broker{codec: NewCodec(clientSide)}b.setScanner(sp)b.proxyMu.Lock()- b.proxySubscribed = true+ b.setEngineProxySubscribed(ollamaProxyProfile, true)b.proxyMu.Unlock()toClient := make(chan *Message, 1)diff --git a/services/nvpair-ui-broker/ollamahost.go b/services/nvpair-ui-broker/ollamahost.goindex 68a5d4c..ecb9f70 100644--- a/services/nvpair-ui-broker/ollamahost.go+++ b/services/nvpair-ui-broker/ollamahost.go@@ -27,8 +27,6 @@ const (errorsHTTPPort = 14319workloadHTTPPort = 14320clusterManagerHTTPPort = 14321-- lmstudioProxyPortFile = "lmstudio-proxy-port.json")type ollamaHostAlias struct {@@ -186,7 +184,7 @@ func (b *Broker) prepareOllamaHostAlias(enabled bool, backendPort int) {if backendPort > 0 {enginePorts[backendPort] = "ollama"}- if reason := reservedOllamaHostAliasPort(alias.Port, enginePorts, configuredLMStudioProxyPort()); reason != "" {+ if reason := reservedOllamaHostAliasPort(alias.Port, enginePorts, b.siblingEngineProxyPorts(ollamaProxyProfile)); reason != "" {b.reportOllamaHostAliasBlocked(alias.displayAddress(), reason)return}@@ -323,43 +321,79 @@ func (b *Broker) configuredEnginePorts() (map[int]string, error) {return ports, nil}-// configuredLMStudioProxyPort mirrors lmstudio-proxy's small persisted-port-// contract so the Ollama alias cannot win its bind race. Invalid/missing state-// means the proxy will use its default.-func configuredLMStudioProxyPort() int {- path, err := appdir.Path(lmstudioProxyPortFile)+// configuredEngineProxyPort mirrors a proxy's small persisted-port contract so+// the Ollama alias cannot win its bind race. Invalid or missing state means+// that proxy will come up on its facade port.+func configuredEngineProxyPort(p engineProxyProfile) int {+ path, err := appdir.Path(p.PortFile)if err != nil {- return managedLMStudioFacadePort+ return p.FacadePort}data, err := os.ReadFile(path)if err != nil {- return managedLMStudioFacadePort+ return p.FacadePort}var stored struct {Port int `json:"port"`}if json.Unmarshal(data, &stored) != nil || stored.Port < 1 || stored.Port > 65535 {- return managedLMStudioFacadePort+ return p.FacadePort}return stored.Port}-func reservedOllamaHostAliasPort(port int, enginePorts map[int]string, lmstudioProxy int) string {+// siblingEngineProxyPorts is every port an engine other than self is going to+// want, with the reason to show a user who collided with one: the facade it+// claims, the backend base a managed engine gets relocated onto, and its+// persisted port if it has one.+//+// Two callers need this set. The OLLAMA_HOST alias must not squat a sibling's+// port, and neither must a proxy's own bind-failure fallback: a fallback that+// lands on a stopped engine's configured port leaves that engine unable to+// start on the next restore, with nothing pointing at the cause.+//+// Building this from the engine table rather than naming LM Studio is what+// keeps a third engine from silently losing the check — its ports would+// otherwise be invisible here and either caller could take one.+//+// Ports are reserved even for an engine deselected via --proxy-engines. The+// engine's persisted configuration outlives this process, so handing its port+// away is not made safe by the proxy being off right now; the reason text says+// so rather than the set quietly shrinking.+func (b *Broker) siblingEngineProxyPorts(self engineProxyProfile) map[int]string {+ reserved := map[int]string{}+ for _, p := range engineProxyProfiles {+ if p.Name == self.Name {+ continue+ }+ suffix := ""+ if !b.proxyEnabled(p) {+ suffix = " (its proxy is not enabled, but the port stays reserved)"+ }+ reserved[p.FacadePort] = fmt.Sprintf("the %s compatibility proxy uses port %d%s", p.DisplayName, p.FacadePort, suffix)+ // Managed ownership is prepared after this check and moves a colliding+ // backend here, so nothing else may be sitting on it.+ reserved[p.EnginePortBase] = fmt.Sprintf("the managed %s backend uses port %d%s", p.DisplayName, p.EnginePortBase, suffix)+ // Written last so a persisted port outranks the defaults it may equal,+ // matching the precedence this check has always had.+ if persisted := configuredEngineProxyPort(p); persisted > 0 {+ reserved[persisted] = fmt.Sprintf("the %s proxy is configured on port %d%s", p.DisplayName, persisted, suffix)+ }+ }+ return reserved+}++func reservedOllamaHostAliasPort(port int, enginePorts map[int]string, siblingProxyPorts map[int]string) string {if engine, ok := enginePorts[port]; ok {if engine == "" {engine = "an engine"}return fmt.Sprintf("%s backend is configured on port %d", engine, port)}+ if reason, ok := siblingProxyPorts[port]; ok {+ return reason+ }switch port {- case lmstudioProxy:- return fmt.Sprintf("the LM Studio proxy is configured on port %d", port)- case managedLMStudioFacadePort:- return fmt.Sprintf("the LM Studio compatibility proxy uses port %d", port)- case managedLMStudioBackendStart:- // Managed LM Studio ownership is prepared after this check and moves a- // colliding backend here, so the alias must not be sitting on it.- return fmt.Sprintf("the managed LM Studio backend uses port %d", port)case nodeInfoHTTPPort:return fmt.Sprintf("the node-info service uses port %d", port)case errorsHTTPPort:diff --git a/services/nvpair-ui-broker/ollamahost_test.go b/services/nvpair-ui-broker/ollamahost_test.goindex c9a732a..c9abe6c 100644--- a/services/nvpair-ui-broker/ollamahost_test.go+++ b/services/nvpair-ui-broker/ollamahost_test.go@@ -5,6 +5,7 @@ package mainimport ("encoding/json"+ "fmt""net""os""path/filepath"@@ -114,7 +115,17 @@ func TestReservedOllamaHostAliasPort(t *testing.T) {if tc.lmstudioBackend > 0 {enginePorts[tc.lmstudioBackend] = "lmstudio"}- reason := reservedOllamaHostAliasPort(tc.port, enginePorts, tc.lmstudioProxy)+ // The sibling-proxy set is built from the engine table in+ // production; here it is spelled out so a case can vary the+ // persisted port independently.+ siblings := map[int]string{+ managedLMStudioFacadePort: fmt.Sprintf("the LM Studio compatibility proxy uses port %d", managedLMStudioFacadePort),+ managedLMStudioBackendStart: fmt.Sprintf("the managed LM Studio backend uses port %d", managedLMStudioBackendStart),+ }+ if tc.lmstudioProxy > 0 {+ siblings[tc.lmstudioProxy] = fmt.Sprintf("the LM Studio proxy is configured on port %d", tc.lmstudioProxy)+ }+ reason := reservedOllamaHostAliasPort(tc.port, enginePorts, siblings)if tc.wantReasonSubstr == "" && reason != "" {t.Fatalf("reason = %q, want none", reason)}@@ -125,23 +136,98 @@ func TestReservedOllamaHostAliasPort(t *testing.T) {}}-func TestConfiguredLMStudioProxyPort(t *testing.T) {+// The alias reads each sibling proxy's persisted port from its own file, so a+// third engine is covered by adding a table entry rather than another reader.+func TestConfiguredEngineProxyPort(t *testing.T) {isolateOllamaHostTestConfig(t)- if got := configuredLMStudioProxyPort(); got != managedLMStudioFacadePort {- t.Fatalf("missing persisted port = %d, want default %d", got, managedLMStudioFacadePort)++ for _, p := range engineProxyProfiles {+ if got := configuredEngineProxyPort(p); got != p.FacadePort {+ t.Fatalf("%s: missing persisted port = %d, want its facade %d", p.Name, got, p.FacadePort)+ }}- path, err := appdir.Path(lmstudioProxyPortFile)++ // Persisting one engine's port must not move another's.+ stored := lmstudioProxyProfile.FacadePort + 6+ path, err := appdir.Path(lmstudioProxyProfile.PortFile)if err != nil {t.Fatal(err)}if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {t.Fatal(err)}- if err := os.WriteFile(path, []byte(`{"port":1240}`), 0o600); err != nil {+ if err := os.WriteFile(path, fmt.Appendf(nil, `{"port":%d}`, stored), 0o600); err != nil {t.Fatal(err)}- if got := configuredLMStudioProxyPort(); got != 1240 {- t.Fatalf("persisted port = %d, want 1240", got)+ if got := configuredEngineProxyPort(lmstudioProxyProfile); got != stored {+ t.Fatalf("persisted port = %d, want %d", got, stored)+ }+ if got := configuredEngineProxyPort(ollamaProxyProfile); got != ollamaProxyProfile.FacadePort {+ t.Fatalf("ollama port = %d, want its own facade %d; it read LM Studio's file",+ got, ollamaProxyProfile.FacadePort)+ }++ // And the reserved set the alias checks picks that persisted port up.+ b := &Broker{}+ if reason := b.siblingEngineProxyPorts(ollamaProxyProfile)[stored]; reason == "" {+ t.Fatalf("persisted sibling port %d is not reserved against the alias", stored)+ }+}++// A proxy that bind-fails picks a fallback port, and that search must not land+// on a port a *different* engine is configured to use. Nothing enforced this+// before: the fallback excluded the alias and its own backend, so a stopped+// sibling configured just above the search base was taken, and that engine then+// silently failed to start on the next desired-state restore.+//+// Both directions are checked because the two engines have separate fallback+// functions and fixing one is not evidence about the other.+func TestProxyFallbackSkipsASiblingEnginesConfiguredPort(t *testing.T) {+ for _, tc := range []struct {+ name string+ self engineProxyProfile+ sibling engineProxyProfile+ fallback func(*Broker, ...int) int+ }{+ {+ name: "ollama fallback avoids LM Studio",+ self: ollamaProxyProfile,+ sibling: lmstudioProxyProfile,+ fallback: (*Broker).setOllamaProxyFallback,+ },+ {+ name: "lmstudio fallback avoids Ollama",+ self: lmstudioProxyProfile,+ sibling: ollamaProxyProfile,+ fallback: (*Broker).setLMStudioProxyFallback,+ },+ } {+ t.Run(tc.name, func(t *testing.T) {+ isolateOllamaHostTestConfig(t)++ // Put the sibling's persisted port inside the search range so a+ // fallback that ignores siblings would hand it out.+ stored := tc.self.EnginePortBase + 1+ path, err := appdir.Path(tc.sibling.PortFile)+ if err != nil {+ t.Fatal(err)+ }+ if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {+ t.Fatal(err)+ }+ if err := os.WriteFile(path, fmt.Appendf(nil, `{"port":%d}`, stored), 0o600); err != nil {+ t.Fatal(err)+ }++ b := &Broker{}+ got := tc.fallback(b, tc.self.EnginePortBase)+ if got == stored {+ t.Fatalf("fallback chose %d, the port %s is configured on", got, tc.sibling.DisplayName)+ }+ if got == tc.sibling.FacadePort || got == tc.sibling.EnginePortBase {+ t.Fatalf("fallback chose %d, one of %s's default ports", got, tc.sibling.DisplayName)+ }+ })}}@@ -165,57 +251,6 @@ func TestPrepareOllamaHostAliasHonorsOptOutAndBackendOwnership(t *testing.T) {}}-func brokerWithEngineStatus(t *testing.T, engine string, port int) *Broker {- return brokerWithEngineInventory(t, map[string]int{engine: port})-}--func brokerWithEngineInventory(t *testing.T, ports map[string]int) *Broker {- t.Helper()- client, server := net.Pipe()- worker := &rpcWorker{peer: NewPeer(NewCodec(client))}- go worker.peer.Serve(nil, nil)- b := &Broker{}- b.setEngineMgr(worker)- go func() {- codec := NewCodec(server)- request, err := codec.Read()- if err != nil {- return- }- switch request.Method {- case "engine:status":- var params struct {- Engine string `json:"engine"`- }- if json.Unmarshal(request.Params, ¶ms) != nil {- _ = codec.RespondError(request.ID, -32602, "invalid engine status request")- return- }- _ = codec.Respond(request.ID, map[string]any{"port": ports[params.Engine]})- case "engine:get-installed":- engines := make([]map[string]any, 0, len(ports))- for engine, port := range ports {- engines = append(engines, map[string]any{"engine": engine, "port": port})- }- _ = codec.Respond(request.ID, map[string]any{"engines": engines})- default:- _ = codec.RespondError(request.ID, -32602, "unexpected engine status request")- }- }()- t.Cleanup(func() {- _ = client.Close()- _ = server.Close()- })- return b-}--func isolateOllamaHostTestConfig(t *testing.T) {- t.Helper()- dir := t.TempDir()- t.Setenv("LOCALAPPDATA", dir)- t.Setenv("XDG_CONFIG_HOME", dir)-}-func TestAliasWarningReplaysAfterErrorsProcessRecovery(t *testing.T) {b := &Broker{nodeID: "local-node"}b.forwardErrorsReport(errors.ServiceError{diff --git a/services/nvpair-ui-broker/proxy.go b/services/nvpair-ui-broker/proxy.goindex 57b5305..1d9a27d 100644--- a/services/nvpair-ui-broker/proxy.go+++ b/services/nvpair-ui-broker/proxy.go@@ -14,6 +14,7 @@ import ("time""nvpair-shared/applog"+ "nvpair-shared/engines""nvpair-shared/noderec""nvpair-ui-broker/relay")@@ -22,49 +23,66 @@ import (// relayed control-plane request before giving up.const proxyCallTimeout = 5 * time.Second-// proxyProcess is the broker's handle on its child ollama-proxy: a full-// bidirectional JSON-RPC peer that runs an HTTP reverse proxy (default :11435),-// emits a "ready" notification carrying the bound port, and accepts id-bearing-// control-plane requests. The id-correlation + read pump live in the shared-// jsonrpc.Peer; this handle owns the OS process and the readiness/port state.+// proxyProcess is the broker's handle on its child nvpair-proxy: a full+// bidirectional JSON-RPC peer hosting a facade per enabled engine, each of+// which emits an engine-addressed "ready" notification carrying its bound port+// and accepts id-bearing control-plane requests. The id-correlation + read pump+// live in the shared jsonrpc.Peer; this handle owns the OS process and the+// per-engine readiness/port state.type proxyProcess struct {- // name identifies which proxy this is ("proxy" / "lmstudio-proxy") in- // shutdown diagnostics; both engines share this handle type.+ // name identifies the process in shutdown diagnostics. It is+ // engines.ProxyComponent: one process covers every engine, so there is no+ // per-engine handle to name.name stringcmd *exec.Cmdstdin io.WriteCloserpeer *Peerdone chan struct{}- // readyMu guards ready, port, and readyParams, written by the read pump- // when the proxy's "ready" notification arrives and read by the broker to- // answer proxy:get-status / replay the baseline on a fresh subscribe.+ // readyMu guards facadeState, written by the read pump when a facade's+ // addressed "ready" notification arrives and read by the broker to answer+ // <engine>-proxy:get-status / replay the baseline on a fresh subscribe.+ //+ // Keyed by engine because one process hosts several facades on different+ // ports. A single port for the process would be whichever facade readied+ // last, so get-status would hand clients another engine's port.readyMu sync.Mutex- ready bool- port int- readyParams json.RawMessage+ facadeState map[string]proxyFacadeState// onNotify, if non-nil, is invoked for every notification the proxy emits// (including "ready") so the broker can forward its event stream.onNotify func(method string, params json.RawMessage)- // relayDir is the broker's LAN directory. When the proxy sends a+ // relayDir is the broker's LAN directory. When a facade sends a// discovery:subscribe, the broker subscribes it here and pushes- // discovery:nodes snapshots back down for the proxy's routing set. nil when the- // relay isn't wired (older call paths / tests).+ // discovery:nodes snapshots back down for that facade's routing set. nil+ // when the relay isn't wired (older call paths / tests).+ //+ // subIDs is keyed by engine because each facade subscribes for its own+ // engine's discovery service: one id per process would let a second+ // facade's subscribe replace the first's. Nil until the first subscribe.relayDir *relay.DirectorysubMu sync.Mutex- subID int+ subIDs map[string]int}-// proxyReadyParams mirrors ollama-proxy's "ready" notification payload: the-// proxy binds its port synchronously and echoes it here, the authoritative-// source of the proxy's listen port.+// proxyReadyParams mirrors the proxy's "ready" notification payload: a facade+// binds its port synchronously and echoes it here, the authoritative source of+// that facade's listen port.type proxyReadyParams struct {Version string `json:"version"`Port int `json:"port"`}+// proxyFacadeState is what one facade has told the broker about itself.+type proxyFacadeState struct {+ ready bool+ port int+ // params is the verbatim ready payload, replayed to a client that+ // subscribes after the fact so it does not have to wait for the next one.+ params json.RawMessage+}+// SetLogLevel forwards an already-validated log level as a log/set-level// notification (through the peer's codec).func (p *proxyProcess) SetLogLevel(level string) error {@@ -75,25 +93,27 @@ func (p *proxyProcess) SetLogLevel(level string) error {// process has exited (cmd.Wait returned).func (p *proxyProcess) Done() <-chan struct{} { return p.done }-// Status reports whether the proxy has announced itself ready and, if so, the-// HTTP port it bound (0 until "ready" arrives).-func (p *proxyProcess) Status() (bool, int) {+// Status reports whether one engine's facade has announced itself ready and, if+// so, the HTTP port it bound (0 until its "ready" arrives).+func (p *proxyProcess) Status(engine string) (bool, int) {p.readyMu.Lock()defer p.readyMu.Unlock()- return p.ready, p.port+ state := p.facadeState[engine]+ return state.ready, state.port}-// ReadyParams returns the verbatim payload of the proxy's "ready" notification,-// or nil if it hasn't announced itself yet.-func (p *proxyProcess) ReadyParams() json.RawMessage {+// ReadyParams returns the verbatim payload of one facade's "ready"+// notification, or nil if that engine hasn't announced itself yet.+func (p *proxyProcess) ReadyParams(engine string) json.RawMessage {p.readyMu.Lock()defer p.readyMu.Unlock()- return p.readyParams+ return p.facadeState[engine].params}-// startProxy spawns ollama-proxy with the broker's current log level, hides the-// console window on Windows, and runs the peer read pump to capture the "ready"-// notification (and thus the bound port). onNotify (may be nil) is invoked for+// startProxy spawns nvpair-proxy with the broker's current log level, hides the+// console window on Windows, and runs the peer read pump to capture each+// facade's "ready" notification (and thus its bound port). The process starts+// with no facade; the caller enables them. onNotify (may be nil) is invoked for// every notification the proxy emits so the broker can forward its event// stream. Proxy stderr goes to the broker's non-blocking sink (see// stderrsink.go), so a stalled reader cannot block the proxy's exit.@@ -132,10 +152,14 @@ func startProxy(name, binaryPath, logLevel string, relayDir *relay.Directory, ongo func() {_ = cmd.Wait()pp.peer.Close()+ // Every facade's subscription goes, not just one: the process is gone,+ // so a surviving registration would push snapshots at a closed peer.pp.subMu.Lock()- if pp.subID != 0 && pp.relayDir != nil {- pp.relayDir.Unsubscribe(pp.subID)- pp.subID = 0+ for engine, id := range pp.subIDs {+ if id != 0 && pp.relayDir != nil {+ pp.relayDir.Unsubscribe(id)+ }+ delete(pp.subIDs, engine)}pp.subMu.Unlock()close(pp.done)@@ -147,25 +171,36 @@ func startProxy(name, binaryPath, logLevel string, relayDir *relay.Directory, on// handleNotify records the proxy's "ready" port and forwards every notification// to the broker. A malformed "ready" is logged and not forwarded (matching the// pre-refactor behavior).+//+// Facade-scoped notifications arrive addressed to their engine, so the address+// is split off before dispatching on the method. The addressed form is what+// goes to onNotify: each consumer re-splits and checks the engine itself,+// rather than trusting a bare method to have come from the engine it expected.func (p *proxyProcess) handleNotify(method string, params json.RawMessage) {- if method == "ready" {+ engine, bare := engines.SplitAddressedMethod(method)+ if bare == "ready" {var rp proxyReadyParamsif err := json.Unmarshal(params, &rp); err != nil {- slog.Warn("proxy emitted invalid ready payload", "err", err)+ slog.Warn("proxy emitted invalid ready payload", "engine", engine, "err", err)return}p.readyMu.Lock()- p.ready = true- p.port = rp.Port- p.readyParams = append(json.RawMessage(nil), params...)+ if p.facadeState == nil {+ p.facadeState = make(map[string]proxyFacadeState, len(engines.Names()))+ }+ p.facadeState[engine] = proxyFacadeState{+ ready: true,+ port: rp.Port,+ params: append(json.RawMessage(nil), params...),+ }p.readyMu.Unlock()- slog.Info("proxy reported ready", "version", rp.Version, "port", rp.Port)+ slog.Info("proxy reported ready", "engine", engine, "version", rp.Version, "port", rp.Port)}// The proxy subscribes upward for its routing targets: wire it to// the relay directory and push events down, rather than forwarding this as a// client-facing event.- if method == noderec.MethodSubscribe {- p.handleSubscribe(params)+ if bare == noderec.MethodSubscribe {+ p.handleSubscribe(engine, params)return}if p.onNotify != nil {@@ -173,29 +208,42 @@ func (p *proxyProcess) handleNotify(method string, params json.RawMessage) {}}-// handleSubscribe wires the proxy's discovery:subscribe into the relay+// handleSubscribe wires one facade's discovery:subscribe into the relay// directory: it registers a subscriber whose Send pushes a discovery:nodes-// snapshot down the proxy's peer, then sends the initial snapshot so the proxy's-// routing set is populated immediately. A re-subscribe (e.g. the proxy resends)-// drops the prior registration first so it isn't double-fed.-func (p *proxyProcess) handleSubscribe(params json.RawMessage) {+// snapshot down the proxy's peer, then sends the initial snapshot so the+// facade's routing set is populated immediately. A re-subscribe (e.g. the+// facade resends) drops that facade's prior registration so it isn't+// double-fed.+//+// Subscriptions are tracked per engine, not per process. Each facade filters on+// its own engine's discovery service, so a single id per process would make a+// second facade's subscribe silently replace the first's — leaving that engine+// with a routing set that never updates again.+//+// Snapshots are pushed back addressed to the requesting engine, so the child+// hands each one to the facade that asked for it.+func (p *proxyProcess) handleSubscribe(engine string, params json.RawMessage) {if p.relayDir == nil {return}send := func(nodes []noderec.DirectoryNode) {- if err := p.peer.Notify(noderec.NotifyNodes, noderec.GetNodesResult{Nodes: nodes}); err != nil {- slog.Debug("failed to push node snapshot to proxy", "err", err)+ method := engines.AddressMethod(engine, noderec.NotifyNodes)+ if err := p.peer.Notify(method, noderec.GetNodesResult{Nodes: nodes}); err != nil {+ slog.Debug("failed to push node snapshot to proxy", "engine", engine, "err", err)}}p.subMu.Lock()- if p.subID != 0 {- p.relayDir.Unsubscribe(p.subID)+ if p.subIDs == nil {+ p.subIDs = make(map[string]int, len(engines.Names()))+ }+ if prior := p.subIDs[engine]; prior != 0 {+ p.relayDir.Unsubscribe(prior)}id, sub, err := subscribeRelay(p.relayDir, params, send)- p.subID = id+ p.subIDs[engine] = idp.subMu.Unlock()if err != nil {- slog.Warn("proxy sent invalid discovery:subscribe", "err", err)+ slog.Warn("proxy sent invalid discovery:subscribe", "engine", engine, "err", err)return}p.relayDir.Deliver(sub)diff --git a/services/nvpair-ui-broker/proxyaddressing_test.go b/services/nvpair-ui-broker/proxyaddressing_test.gonew file mode 100644index 0000000..533b176--- /dev/null+++ b/services/nvpair-ui-broker/proxyaddressing_test.go@@ -0,0 +1,414 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++// Coverage for the engine addressing on the broker↔proxy wire.+//+// One process per engine made the engine implicit: whichever read pump+// delivered a notification decided which engine it belonged to, and whichever+// process a call was sent to decided which engine applied it. A process hosting+// several facades has neither property, so both directions carry an explicit+// engine and these tests pin what happens when it is absent, present, or wrong.++import (+ "context"+ "encoding/json"+ "fmt"+ "io"+ "net"+ "testing"+ "time"++ "nvpair-shared/engines"+ "nvpair-shared/noderec"+ "nvpair-ui-broker/relay"+)++// The relay must address what it sends downward, not just strip the client's+// prefix.+//+// Nothing else catches this: dropping profile.addressed() in relayToEngineProxy+// compiles, and the child answers -32002 because an unaddressed facade-scoped+// method resolves to no facade — which looks like an unavailable proxy rather+// than a broker bug. The two prefixes are deliberately different strings, so+// the relay is a translation and this asserts both halves of it.+func TestRelayAddressesTheMethodItSendsDownward(t *testing.T) {+ for _, profile := range engineProxyProfiles {+ t.Run(profile.Name, func(t *testing.T) {+ client, server := net.Pipe()+ t.Cleanup(func() {+ _ = client.Close()+ _ = server.Close()+ })+ proxy := &proxyProcess{peer: NewPeer(NewCodec(client))}+ go proxy.peer.Serve(nil, nil)++ seen := make(chan string, 1)+ go func() {+ codec := NewCodec(server)+ msg, err := codec.Read()+ if err != nil {+ return+ }+ seen <- msg.Method+ _ = codec.Respond(msg.ID, map[string][]string{"nodes": {}})+ }()++ b := &Broker{codec: NewCodec(rwDiscard{})}+ b.setEngineProxyHandle(profile, proxy)++ id := json.RawMessage(`1`)+ b.relayToEngineProxy(profile, &Message{+ Method: profile.ComponentName() + ":nodes/list",+ ID: &id,+ })++ select {+ case got := <-seen:+ want := profile.addressed("nodes/list")+ if got != want {+ t.Fatalf("relayed method = %q, want %q", got, want)+ }+ case <-time.After(2 * time.Second):+ t.Fatal("relay sent nothing downward")+ }+ })+ }+}++// rwDiscard is a client transport that reads nothing and swallows writes, for a+// relay whose reply nobody inspects.+type rwDiscard struct{}++func (rwDiscard) Read([]byte) (int, error) { return 0, io.EOF }+func (rwDiscard) Write(p []byte) (int, error) { return len(p), nil }+func (rwDiscard) Close() error { return nil }++// Giving up the managed claim keeps the OLLAMA_HOST alias reservation, so a+// spawn whose every facade failed still has it when the supervisor retries.+//+// This is the half of the block/finish split that the enabled==0 path depends+// on: it may run the block, and must not run the finish. Releasing the alias+// there was a regression, because the retry still needs it and engine-manager+// could take that port in between, so an attempt that would otherwise have+// succeeded comes up without the inherited endpoint.+//+// The call-site ordering itself is not reachable from a unit test — spawnProxy+// launches a real binary — so this pins the property that makes the ordering+// matter, and the sibling test below pins the other side of it.+func TestBlockingAManagedFacadeKeepsTheAliasForTheRetry(t *testing.T) {+ isolateOllamaHostTestConfig(t)+ b := &Broker{+ nodeID: "local-node",+ ollamaPortReady: make(chan struct{}),+ lmstudioPortReady: make(chan struct{}),+ }+ alias := ollamaHostAlias{Address: "127.0.0.1:11433", Port: 11433}+ b.setOllamaHostAlias(alias)+ b.ollamaState().managedFacade.Store(true)++ b.blockManagedOllamaFacade("the Ollama proxy facade could not be brought up")++ if got := b.currentOllamaHostAlias(); got.Port != alias.Port {+ t.Errorf("blocking released the alias a retry still needs: %+v", got)+ }+}++// An engine that failed inside a process that is staying gets terminal+// treatment, which is what finally returns the alias.+//+// Without this the broker would hold a reservation for a facade that is never+// coming back, and engine-manager could never be given that port.+func TestSurvivingProcessReleasesTheAliasForAFailedFacade(t *testing.T) {+ b := &Broker{+ ollamaPortReady: make(chan struct{}),+ lmstudioPortReady: make(chan struct{}),+ }+ alias := ollamaHostAlias{Address: "127.0.0.1:11433", Port: 11433}+ b.setOllamaHostAlias(alias)+ if got := b.currentOllamaHostAlias(); got.Port != alias.Port {+ t.Fatalf("alias not established for the test: %+v", got)+ }++ b.blockAndFinishEngineProxy(ollamaProxyProfile)+ if got := b.currentOllamaHostAlias(); got.Port != 0 {+ t.Errorf("terminal treatment did not release the alias: %+v", got)+ }+}++// A facade that reported ready is kept when its enable call went unanswered,+// and disowned when the child answered with a rejection.+//+// Both halves matter and neither runs on the ordinary error return, so without+// this test a later edit can either strand a live listener the broker reports+// as unavailable, or publish a dead port the broker will never retry.+func TestFacadeCameUpAnywayOnlyTrustsAnUnansweredEnable(t *testing.T) {+ engine := ollamaProxyProfile.Name+ ready := &proxyProcess{facadeState: readyFacade(engine, 11434)}+ notReady := &proxyProcess{}++ for _, tc := range []struct {+ name string+ pp *proxyProcess+ err error+ want bool+ }{+ {+ // A timeout or closed pipe: the child does the bind, two notifies+ // and the serve inside the call, so it may genuinely be serving.+ name: "unanswered enable, facade reported ready",+ pp: ready, err: context.DeadlineExceeded, want: true,+ },+ {+ name: "unanswered enable, facade never reported ready",+ pp: notReady, err: context.DeadlineExceeded, want: false,+ },+ {+ // The child answered, so it decided against the facade and+ // withdrew it: an earlier ready from the same attempt is stale.+ name: "answered rejection, stale ready ignored",+ pp: ready, err: fmt.Errorf("%w: rejected", errRPCAnswered), want: false,+ },+ {+ name: "answered bind failure, stale ready ignored",+ pp: ready,+ err: fmt.Errorf("%w: %w: taken", errRPCAnswered, errFacadeBindFailed),+ want: false,+ },+ } {+ t.Run(tc.name, func(t *testing.T) {+ if got := facadeCameUpAnyway(tc.pp, engine, tc.err); got != tc.want {+ t.Fatalf("facadeCameUpAnyway = %v, want %v", got, tc.want)+ }+ })+ }+}++// readyFacade is the handle state one engine's facade would leave behind after+// announcing itself ready on a port.+func readyFacade(engine string, port int) map[string]proxyFacadeState {+ return map[string]proxyFacadeState{+ engine: {ready: true, port: port},+ }+}++// Readiness is tracked per engine because one process hosts several facades on+// different ports. A single port for the process would be whichever facade+// readied last, so get-status would hand a client the other engine's port —+// and its inference traffic with it.+func TestReadinessIsTrackedPerEngine(t *testing.T) {+ all := engines.All()+ if len(all) < 2 {+ t.Skip("needs at least two engines")+ }+ first, second := all[0], all[1]++ p := &proxyProcess{}+ p.handleNotify(+ engines.AddressMethod(first.Name, "ready"),+ json.RawMessage(`{"version":"test","port":11434}`),+ )+ p.handleNotify(+ engines.AddressMethod(second.Name, "ready"),+ json.RawMessage(`{"version":"test","port":1234}`),+ )++ for engine, wantPort := range map[string]int{first.Name: 11434, second.Name: 1234} {+ ready, port := p.Status(engine)+ if !ready {+ t.Errorf("%s facade is not ready", engine)+ }+ if port != wantPort {+ t.Errorf("%s facade port = %d, want %d", engine, port, wantPort)+ }+ if p.ReadyParams(engine) == nil {+ t.Errorf("%s facade has no replayable ready payload", engine)+ }+ }++ // An engine that never announced itself is not ready, rather than+ // inheriting a sibling's port.+ if ready, port := p.Status("vllm"); ready || port != 0 {+ t.Errorf("an unannounced engine reported ready=%v port=%d", ready, port)+ }+}++// Each facade subscribes for its own engine's discovery service, so the broker+// has to track a subscription per engine. A single id per process let the second+// facade's subscribe replace the first's, which unsubscribed a live facade and+// left that engine with a routing set that never updated again.+func TestSubscriptionsAreTrackedPerEngine(t *testing.T) {+ client, server := net.Pipe()+ t.Cleanup(func() {+ _ = client.Close()+ _ = server.Close()+ })+ // Drain whatever the directory pushes, so Deliver cannot block on the pipe.+ go func() {+ codec := NewCodec(server)+ for {+ if _, err := codec.Read(); err != nil {+ return+ }+ }+ }()++ dir := relay.NewDirectory()+ p := &proxyProcess{peer: NewPeer(NewCodec(client)), relayDir: dir}++ subscribe := func(engine string, service noderec.ServiceKey) {+ params, err := json.Marshal(noderec.SubscribeParams{+ Services: []noderec.ServiceKey{service},+ })+ if err != nil {+ t.Fatalf("marshal subscribe for %s: %v", engine, err)+ }+ p.handleSubscribe(engine, params)+ }++ for _, e := range engines.All() {+ subscribe(e.Name, e.DiscoveryService)+ }++ p.subMu.Lock()+ defer p.subMu.Unlock()+ if len(p.subIDs) != len(engines.All()) {+ t.Fatalf("tracked %d subscriptions for %d engines: %v",+ len(p.subIDs), len(engines.All()), p.subIDs)+ }+ seen := map[int]string{}+ for engine, id := range p.subIDs {+ if id == 0 {+ t.Errorf("engine %q has subscription id 0", engine)+ }+ if other, dup := seen[id]; dup {+ t.Errorf("engines %q and %q share subscription id %d", engine, other, id)+ }+ seen[id] = engine+ }+}++// A facade re-subscribing must drop only its own prior registration. Dropping+// every one would silence the other engines, and dropping none would double-feed+// this one.+func TestResubscribeReplacesOnlyThatEnginesSubscription(t *testing.T) {+ client, server := net.Pipe()+ t.Cleanup(func() {+ _ = client.Close()+ _ = server.Close()+ })+ go func() {+ codec := NewCodec(server)+ for {+ if _, err := codec.Read(); err != nil {+ return+ }+ }+ }()++ dir := relay.NewDirectory()+ p := &proxyProcess{peer: NewPeer(NewCodec(client)), relayDir: dir}++ all := engines.All()+ if len(all) < 2 {+ t.Skip("needs at least two engines to tell the subscriptions apart")+ }+ first, second := all[0], all[1]++ params := func(e engines.Engine) json.RawMessage {+ raw, err := json.Marshal(noderec.SubscribeParams{+ Services: []noderec.ServiceKey{e.DiscoveryService},+ })+ if err != nil {+ t.Fatalf("marshal subscribe: %v", err)+ }+ return raw+ }++ p.handleSubscribe(first.Name, params(first))+ p.handleSubscribe(second.Name, params(second))++ p.subMu.Lock()+ firstID, secondID := p.subIDs[first.Name], p.subIDs[second.Name]+ p.subMu.Unlock()++ p.handleSubscribe(first.Name, params(first))++ p.subMu.Lock()+ defer p.subMu.Unlock()+ if p.subIDs[first.Name] == firstID {+ t.Errorf("re-subscribe kept %s's original id %d, so it is now double-fed",+ first.Name, firstID)+ }+ if p.subIDs[second.Name] != secondID {+ t.Errorf("re-subscribing %s changed %s's id from %d to %d",+ first.Name, second.Name, secondID, p.subIDs[second.Name])+ }+}++// The read pump reports ready and wires subscriptions by matching bare method+// names, so it has to see through the address. An unaddressed notification is+// process-scoped and passes through untouched.+func TestFacadeMethodForStripsOnlyItsOwnEngine(t *testing.T) {+ for _, tc := range []struct {+ name string+ profile engineProxyProfile+ method string+ want string+ ok bool+ }{+ {+ name: "own address is stripped",+ profile: ollamaProxyProfile,+ method: ollamaProxyProfile.addressed("ready"),+ want: "ready",+ ok: true,+ },+ {+ // forwardProxyProcessNotification routes by engine before this+ // reader sees anything, so reaching it with another engine's+ // address means addressing broke. Forwarding it would file one+ // engine's event under another.+ name: "another engine's address is dropped",+ profile: ollamaProxyProfile,+ method: lmstudioProxyProfile.addressed("ready"),+ ok: false,+ },+ {+ name: "process-scoped method passes through",+ profile: ollamaProxyProfile,+ method: "workload:started",+ want: "workload:started",+ ok: true,+ },+ {+ // The errors relay matches bare names, so an addressed error has to+ // come out as errors:report and not as ollama:errors:report.+ name: "addressed error becomes a bare errors method",+ profile: ollamaProxyProfile,+ method: ollamaProxyProfile.addressed(methodErrorsReport),+ want: methodErrorsReport,+ ok: true,+ },+ {+ name: "unaddressed error is untouched",+ profile: lmstudioProxyProfile,+ method: methodErrorsReport,+ want: methodErrorsReport,+ ok: true,+ },+ } {+ t.Run(tc.name, func(t *testing.T) {+ got, ok := facadeMethodFor(tc.profile, tc.method)+ if ok != tc.ok {+ t.Fatalf("facadeMethodFor(%s, %q) ok = %v, want %v",+ tc.profile.Name, tc.method, ok, tc.ok)+ }+ if ok && got != tc.want {+ t.Fatalf("facadeMethodFor(%s, %q) = %q, want %q",+ tc.profile.Name, tc.method, got, tc.want)+ }+ })+ }+}diff --git a/services/nvpair-ui-broker/proxyport.go b/services/nvpair-ui-broker/proxyport.goindex 63eddc7..c23f970 100644--- a/services/nvpair-ui-broker/proxyport.go+++ b/services/nvpair-ui-broker/proxyport.go@@ -11,6 +11,7 @@ import ("log/slog""net"+ "nvpair-shared/engines""nvpair-shared/errors")@@ -23,6 +24,12 @@ import (// the proxy off a port a running engine holds. Sticky (no timestamp suffix)// so repeated bumps upsert one entry; cleared when the user later picks a// proxy port that doesn't collide.+// These restate values the engine table already carries. They stay untyped+// constants because they feed int32 atomics throughout the package, where+// converting a table field at every use would be pure noise;+// TestBrokerConstantsMatchTheEngineTable fails if they ever disagree with the+// table. The prefix is ComponentName, which is also the relay prefix and the+// supervisor label.const (proxyPortBumpedID = "ollama-proxy:port-bumped"portOwnershipBlockedID = "ollama-proxy:port-ownership-blocked"@@ -41,35 +48,16 @@ type managedPortPlan struct {Blocked string}-// planManagedOllamaPorts is the policy core kept separate from RPC so every-// safety branch is deterministic in tests. It never plans a move for a-// running engine; a stopped backend whose configured port is occupied advances-// to the next available managed backend port.+// ollamaProxyProfile is the descriptor entry this file's Ollama-specific+// callers plan against. It is resolved once; a missing entry is a programming+// error in engineproxy.go, not a runtime condition.+var ollamaProxyProfile = mustEngineProxyProfile("ollama")++// planManagedOllamaPorts plans Ollama's managed ports. Ollama is an adopted+// engine, so the shared policy refuses to move it while it is running and+// blocks up front on an occupied facade port.func planManagedOllamaPorts(enabled bool, st ollamaPortStatus, available func(int) bool) managedPortPlan {- if !enabled {- return managedPortPlan{}- }- if st.Running && (st.Port == managedOllamaFacadePort || st.Port == 0) {- return managedPortPlan{Blocked: "Ollama is already running on the compatibility port"}- }- if !available(managedOllamaFacadePort) {- return managedPortPlan{Blocked: "the compatibility port is already in use"}- }- if st.Port > 0 && st.Port != managedOllamaFacadePort {- if !st.Running && !available(st.Port) {- backend := nextAvailablePort(managedOllamaBackendStart, available)- if backend == 0 {- return managedPortPlan{Blocked: "no free backend port is available"}- }- return managedPortPlan{Enabled: true, BackendPort: backend}- }- return managedPortPlan{Enabled: true}- }- backend := nextAvailablePort(managedOllamaBackendStart, available)- if backend == 0 {- return managedPortPlan{Blocked: "no free backend port is available"}- }- return managedPortPlan{Enabled: true, BackendPort: backend}+ return planManagedEnginePorts(ollamaProxyProfile, enabled, st, available)}func nextAvailablePort(start int, available func(int) bool) int {@@ -119,11 +107,8 @@ func (b *Broker) reportPortOwnershipBlocked(reason string) {// persisted :11434 must not make a blocked proxy crash-loop behind an existing// owner. It returns the fallback so a live proxy can rebind there too.func (b *Broker) blockManagedOllamaFacade(reason string, excludedPorts ...int) int {- b.managedOllamaFacade.Store(false)+ b.ollamaState().managedFacade.Store(false)b.managedOllamaBackend.Store(0)- if backend := int(b.ollamaBackendPort.Load()); backend > 0 {- excludedPorts = append(excludedPorts, backend)- }fallback := b.setOllamaProxyFallback(excludedPorts...)b.reportPortOwnershipBlocked(reason)b.markOllamaPortReady()@@ -136,12 +121,123 @@ func (b *Broker) markOllamaPortReady() {}}+// finishOllamaProxyTerminal releases the ownership gate when ollama-proxy will+// never come back, mirroring finishLMStudioProxyTerminal. Without it a+// terminally-failed proxy leaves the gate closed for the life of the process,+// so every engine:status / engine:get-installed for Ollama waits out+// rpcWorkerCallTimeout and answers "retry" forever.+//+// Ollama needs more than closing the channel. Its gate re-checks+// ollamaFacadeIsPendingBackend after the wait, so a pending move left behind by+// the dead proxy would keep rejecting requests even with the channel closed.+// Clearing the move state is what actually reopens the path; LM Studio has no+// equivalent second check.+func (b *Broker) finishOllamaProxyTerminal() {+ b.ollamaState().managedFacade.Store(false)+ b.managedOllamaBackend.Store(0)+ b.ollamaMoveInFlight.Store(false)+ // The inherited OLLAMA_HOST alias is reserved with engine-manager during+ // preparation, before enablement is known, so every terminal path has to+ // give it back. Releasing it here rather than at each call site is what+ // makes that true: it was previously released on the two live-failure paths+ // and missed on the disabled-engine one, which left the alias port refusing+ // engine:set-port for the life of the process, citing a proxy alias that+ // did not exist.+ b.disableOllamaHostAliasReservation()+ b.markOllamaPortReady()+}++// configureProxySupervisorCallbacks wires the one proxy supervisor.+//+// Every engine's facade lives in the supervised process, so a crash is reported+// once for the process and clears every engine's handle — shared fate is the+// accepted cost of putting all the facades' reservations in one place. Each+// engine's ownership gate is still released individually, because a caller+// waiting on LM Studio's gate must not be held by Ollama's state.+func (b *Broker) configureProxySupervisorCallbacks(sup *supervisor) {+ sup.onCrash, sup.onRecovered = b.supervisedWorkerCallbacks(engines.ProxyComponent, func() {+ for _, profile := range engineProxyProfiles {+ b.setEngineProxyHandle(profile, nil)+ }+ })+ sup.onExhausted = func(attempt int) {+ slog.Warn("the proxy is terminally unavailable; releasing every engine's ownership gate",+ "attempt", attempt)+ for _, profile := range engineProxyProfiles {+ b.finishEngineProxyStartup(profile)+ }+ }+ sup.onSpawned = b.replayProxyStateAfterSpawn+}++// finishEngineProxyStartup releases one engine's port-ownership gate when its+// facade will not be coming up, so callers stop waiting on it.+//+// Ollama needs more than closing a channel, which is why this dispatches rather+// than looping over a single helper: its gate re-checks whether a backend move+// is pending, so clearing that move state is what actually reopens the path.+func (b *Broker) finishEngineProxyStartup(profile engineProxyProfile) {+ switch profile.Name {+ case ollamaProxyProfile.Name:+ b.finishOllamaProxyTerminal()+ case lmstudioProxyProfile.Name:+ b.finishLMStudioProxyTerminal()+ default:+ slog.Warn("no startup-gate finisher for engine", "engine", profile.Name)+ }+}++// rebindOllamaProxy moves the live Ollama facade onto a port, mirroring+// rebindLMStudioProxy. Best-effort: the caller has already given up the managed+// claim and released the gate, so a failure here leaves the facade where it is+// rather than blocking startup.+//+// No process restart. Every engine's facade shares this process, so restarting+// to move Ollama would drop LM Studio's listener and the inference on it.+func (b *Broker) rebindOllamaProxy(p *proxyProcess, port int, from int) {+ if p == nil || port == 0 || port == from {+ return+ }+ body, err := json.Marshal(map[string]int{"port": port})+ if err != nil {+ slog.Warn("failed to encode Ollama facade rebind", "port", port, "err", err)+ return+ }+ result, rpcErr, callErr := p.Call(context.Background(), ollamaProxyProfile.addressed("set-port"), body)+ if callErr != nil || rpcErr != nil {+ slog.Warn("failed to move the Ollama facade off the compatibility port",+ "from", from, "to", port, "err", callErr, "rpcErr", rpcErr)+ return+ }+ var ready proxyReadyParams+ if json.Unmarshal(result, &ready) != nil || ready.Port != port {+ slog.Warn("Ollama facade did not confirm the rebind", "from", from, "to", port)+ return+ }+ slog.Info("moved the Ollama facade off the compatibility port", "from", from, "to", port)+}++// setOllamaProxyFallback picks an explicit port for the proxy to start on when+// the managed facade is not in play. The engine's own configured port is+// excluded here rather than at each call site, so no caller can forget it.func (b *Broker) setOllamaProxyFallback(excludedPorts ...int) int {if aliasPort := b.currentOllamaHostAlias().Port; aliasPort > 0 {excludedPorts = append(excludedPorts, aliasPort)}+ for port := range b.siblingEngineProxyPorts(ollamaProxyProfile) {+ excludedPorts = append(excludedPorts, port)+ }+ if backend := int(b.ollamaState().backendPort.Load()); backend > 0 {+ excludedPorts = append(excludedPorts, backend)+ } else {+ // No authoritative backend port yet, so a free port is not evidence+ // the engine will not claim it. The stock backend port in particular+ // is where a managed Ollama gets placed, and taking it now means the+ // proxy is squatting the engine's port when it starts.+ excludedPorts = append(excludedPorts, managedOllamaFacadePort, managedOllamaBackendStart)+ }fallback := nextAvailablePortExcluding(managedOllamaBackendStart, excludedPorts, tcpPortAvailable)- b.ollamaProxyStartupPort.Store(int32(fallback))+ b.ollamaState().startupPort.Store(int32(fallback))return fallback}@@ -215,13 +311,13 @@ func (b *Broker) prepareManagedOllamaFacadeWithPortCheck(portAvailable func(int)b.blockManagedOllamaFacade("Ollama status could not be decoded")return}- b.ollamaBackendPort.Store(int32(st.Port))+ b.ollamaState().backendPort.Store(int32(st.Port))b.prepareOllamaHostAlias(policy.Value, st.Port)if !policy.Value {// Opting out after a managed run commonly leaves stopped Ollama on// :11435. Start the proxy explicitly elsewhere so it cannot squat that// configured backend port before Ollama starts.- b.managedOllamaFacade.Store(false)+ b.ollamaState().managedFacade.Store(false)b.managedOllamaBackend.Store(0)b.setOllamaProxyFallback(st.Port)b.forwardErrorsClear(portOwnershipBlockedID)@@ -233,9 +329,9 @@ func (b *Broker) prepareManagedOllamaFacadeWithPortCheck(portAvailable func(int)b.blockManagedOllamaFacade(plan.Blocked, st.Port)return}- b.managedOllamaFacade.Store(plan.Enabled)+ b.ollamaState().managedFacade.Store(plan.Enabled)b.managedOllamaBackend.Store(int32(plan.BackendPort))- b.ollamaProxyStartupPort.Store(managedOllamaFacadePort)+ b.ollamaState().startupPort.Store(int32(managedOllamaFacadePort))if plan.BackendPort == 0 {slog.Info("preserving custom Ollama backend port", "port", st.Port)b.markOllamaPortReady()@@ -257,7 +353,7 @@ func (b *Broker) cacheOllamaPortStatus() {}var st ollamaPortStatusif json.Unmarshal(result, &st) == nil && st.Port > 0 {- b.ollamaBackendPort.Store(int32(st.Port))+ b.ollamaState().backendPort.Store(int32(st.Port))}}@@ -293,15 +389,11 @@ func (b *Broker) rejectOllamaHostAliasPort(msg *Message, port int, owner string)return true}+// handleLMStudioProxySetPort used to relay the request verbatim, so a user+// could park the LM Studio proxy on a port a running engine already held and+// get no warning. It now takes the same path Ollama's does.func (b *Broker) handleLMStudioProxySetPort(msg *Message) {- var params struct {- Port int `json:"port"`- }- if json.Unmarshal(msg.Params, ¶ms) == nil &&- b.rejectOllamaHostAliasPort(msg, params.Port, "the LM Studio proxy") {- return- }- b.relayToLMStudioProxy(msg)+ b.handleEngineProxySetPort(lmstudioProxyProfile, msg)}// needsOllamaPortGate reports the client requests that can probe and adopt the@@ -326,44 +418,55 @@ func needsOllamaPortGate(method string, params json.RawMessage) bool {// bind. It responds with the proxy's own set-port result (which echoes the// actually-bound port).func (b *Broker) handleProxySetPort(msg *Message) {- p := b.getProxy()- if p == nil {- if err := b.codec.RespondError(msg.ID, -32000, "ollama-proxy not available"); err != nil {- log.Printf("failed to respond to proxy:set-port: %v", err)+ b.handleEngineProxySetPort(ollamaProxyProfile, msg)+}++// handleEngineProxySetPort is the engine-agnostic form. Both proxies get the+// same treatment: validate the port, resolve it against the running engines,+// then hand the proxy the port to bind and echo back what it actually bound.+func (b *Broker) handleEngineProxySetPort(profile engineProxyProfile, msg *Message) {+ respondErr := func(code int, format string, args ...any) {+ if err := b.codec.RespondError(msg.ID, code, fmt.Sprintf(format, args...)); err != nil {+ log.Printf("failed to respond to %s: %v", msg.Method, err)}- return}++ // Validate the request before reporting on the proxy: a port that is+ // reserved for the inherited OLLAMA_HOST alias is refused with an+ // actionable message whether or not the proxy happens to be up, which is+ // more use than "not available".var params struct {Port int `json:"port"`}if err := json.Unmarshal(msg.Params, ¶ms); err != nil {- if err := b.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"port\": <int>}"); err != nil {- log.Printf("failed to respond to proxy:set-port: %v", err)- }+ respondErr(-32602, "invalid params: expected {\"port\": <int>}")return}if params.Port < 1 || params.Port > 65535 {- if err := b.codec.RespondError(msg.ID, -32602, "port must be between 1 and 65535"); err != nil {- log.Printf("failed to respond to proxy:set-port: %v", err)- }+ respondErr(-32602, "port must be between 1 and 65535")+ return+ }+ if b.rejectOllamaHostAliasPort(msg, params.Port, "the "+profile.DisplayName+" proxy") {return}- effective := b.resolveProxyPort(params.Port)+ p := b.engineProxyHandle(profile)+ if p == nil {+ respondErr(-32000, "%s not available", profile.ComponentName())+ return+ }++ effective := b.resolveEngineProxyPort(profile, params.Port)body, err := json.Marshal(map[string]int{"port": effective})if err != nil {- if err := b.codec.RespondError(msg.ID, -32000, fmt.Sprintf("encode set-port: %v", err)); err != nil {- log.Printf("failed to respond to proxy:set-port: %v", err)- }+ respondErr(-32000, "encode set-port: %v", err)return}- result, rpcErr, err := p.Call(context.Background(), "set-port", body)+ result, rpcErr, err := p.Call(context.Background(), profile.addressed("set-port"), body)switch {case err != nil:- if err := b.codec.RespondError(msg.ID, -32000, fmt.Sprintf("proxy set-port failed: %v", err)); err != nil {- log.Printf("failed to respond to proxy:set-port: %v", err)- }+ respondErr(-32000, "proxy set-port failed: %v", err)case rpcErr != nil:if err := b.codec.RespondError(msg.ID, rpcErr.Code, rpcErr.Message); err != nil {log.Printf("failed to relay proxy set-port error: %v", err)@@ -381,9 +484,18 @@ func (b *Broker) handleProxySetPort(msg *Message) {// a sticky warning into the errors pipeline; a clean request clears any stale// one so the notice doesn't outlive the conflict.func (b *Broker) resolveProxyPort(requested int) int {- if b.managedOllamaFacade.Load() {- b.forwardErrorsClear(proxyPortBumpedID)- return managedOllamaFacadePort+ return b.resolveEngineProxyPort(ollamaProxyProfile, requested)+}++// resolveEngineProxyPort is the engine-agnostic form. A managed facade is not+// negotiable, so the request is answered with the facade port; otherwise a port+// a running engine already holds is bumped past, because engines take+// precedence over their proxy.+func (b *Broker) resolveEngineProxyPort(p engineProxyProfile, requested int) int {+ bumpedID := p.portBumpedID()+ if b.managedFacade(p) {+ b.forwardErrorsClear(bumpedID)+ return p.FacadePort}taken := b.runningEnginePorts()if aliasPort := b.currentOllamaHostAlias().Port; aliasPort > 0 {@@ -392,15 +504,15 @@ func (b *Broker) resolveProxyPort(requested int) int {effective := nextFreeProxyPort(requested, taken)if effective != requested {b.forwardErrorsReport(errors.ServiceError{- ID: proxyPortBumpedID,- Message: fmt.Sprintf("Port %d is in use by a running engine; the proxy was moved to %d.", requested, effective),+ ID: bumpedID,+ Message: fmt.Sprintf("Port %d is in use by a running engine; the %s proxy was moved to %d.", requested, p.DisplayName, effective),Timestamp: nowMillis(),NodeID: b.nodeID,Severity: "warning",Action: "none",})} else {- b.forwardErrorsClear(proxyPortBumpedID)+ b.forwardErrorsClear(bumpedID)}return effective}@@ -412,16 +524,24 @@ func (b *Broker) resolveProxyPort(requested int) int {// goroutine that calls forwardProxyNotification), so the p.Call round-trip// can't deadlock the very reader that would deliver its response.func (b *Broker) reconcileProxyPortOnReady(boundPort int) {- if b.managedOllamaFacade.Load() {+ if b.ollamaState().managedFacade.Load() {p := b.getProxy()if p == nil {return}if boundPort != managedOllamaFacadePort {body, _ := json.Marshal(map[string]int{"port": managedOllamaFacadePort})- if _, rpcErr, err := p.Call(context.Background(), "set-port", body); err != nil || rpcErr != nil {- b.blockManagedOllamaFacade("the proxy could not bind the compatibility port")+ if _, rpcErr, err := p.Call(context.Background(), ollamaProxyProfile.addressed("set-port"), body); err != nil || rpcErr != nil {slog.Warn("failed to bind managed Ollama facade", "err", err, "rpcErr", rpcErr)+ // Give up the managed claim, then actually move the live facade+ // off the port it is on. blockManagedOllamaFacade only plans a+ // fallback for the next spawn, so without this the facade stays+ // where it bound — possibly on the engine's own backend port —+ // and nothing corrects it until a restart. LM Studio has had+ // this since its port recovery became facade-scoped; Ollama's+ // half was missing.+ fallback := b.blockManagedOllamaFacade("the proxy could not bind the compatibility port")+ b.rebindOllamaProxy(p, fallback, boundPort)}return}@@ -447,7 +567,7 @@ func (b *Broker) reconcileProxyPortOnReady(boundPort int) {vacated := falseif fallback != 0 {body, _ := json.Marshal(map[string]int{"port": fallback})- if _, moveErr, callErr := p.Call(context.Background(), "set-port", body); callErr != nil || moveErr != nil {+ if _, moveErr, callErr := p.Call(context.Background(), ollamaProxyProfile.addressed("set-port"), body); callErr != nil || moveErr != nil {slog.Warn("failed to vacate managed Ollama facade", "err", callErr, "rpcErr", moveErr)} else {vacated = true@@ -463,7 +583,7 @@ func (b *Broker) reconcileProxyPortOnReady(boundPort int) {return}b.managedOllamaBackend.Store(0)- b.ollamaBackendPort.Store(int32(backend))+ b.ollamaState().backendPort.Store(int32(backend))slog.Info("configured managed Ollama backend port", "port", backend)b.ollamaMoveInFlight.Store(false)}@@ -502,13 +622,13 @@ func (b *Broker) reconcileProxyPortOnReady(boundPort int) {slog.Warn("failed to encode corrective proxy set-port", "err", err)return}- if _, rpcErr, err := p.Call(context.Background(), "set-port", body); err != nil || rpcErr != nil {+ if _, rpcErr, err := p.Call(context.Background(), ollamaProxyProfile.addressed("set-port"), body); err != nil || rpcErr != nil {slog.Warn("failed to reconcile proxy port on ready", "err", err, "rpcErr", rpcErr)}}func (b *Broker) takePendingManagedOllamaBackend(boundPort int) int {- if !b.managedOllamaFacade.Load() || boundPort != managedOllamaFacadePort {+ if !b.ollamaState().managedFacade.Load() || boundPort != managedOllamaFacadePort {return 0}if !b.ollamaMoveInFlight.CompareAndSwap(false, true) {@@ -522,7 +642,7 @@ func (b *Broker) takePendingManagedOllamaBackend(boundPort int) int {}func (b *Broker) ollamaBackendSourcePort() int {- if port := int(b.ollamaBackendPort.Load()); port > 0 {+ if port := int(b.ollamaState().backendPort.Load()); port > 0 {return port}return managedOllamaFacadePortdiff --git a/services/nvpair-ui-broker/proxyport_test.go b/services/nvpair-ui-broker/proxyport_test.goindex 61debdf..a101c38 100644--- a/services/nvpair-ui-broker/proxyport_test.go+++ b/services/nvpair-ui-broker/proxyport_test.go@@ -148,7 +148,7 @@ func TestNextAvailablePortExcludingCustomBackend(t *testing.T) {func TestTakePendingManagedOllamaBackendOnce(t *testing.T) {b := &Broker{}- b.managedOllamaFacade.Store(true)+ b.ollamaState().managedFacade.Store(true)b.managedOllamaBackend.Store(11435)if got := b.takePendingManagedOllamaBackend(11436); got != 0 {@@ -167,7 +167,7 @@ func TestOllamaBackendSourcePort(t *testing.T) {if got := b.ollamaBackendSourcePort(); got != managedOllamaFacadePort {t.Fatalf("unset source = %d, want %d", got, managedOllamaFacadePort)}- b.ollamaBackendPort.Store(managedOllamaBackendStart)+ b.ollamaState().backendPort.Store(managedOllamaBackendStart)if got := b.ollamaBackendSourcePort(); got != managedOllamaBackendStart {t.Fatalf("configured source = %d, want %d", got, managedOllamaBackendStart)}@@ -175,7 +175,7 @@ func TestOllamaBackendSourcePort(t *testing.T) {func TestDuplicateOllamaReadyDoesNotOpenGateDuringMove(t *testing.T) {b := &Broker{ollamaPortReady: make(chan struct{})}- b.managedOllamaFacade.Store(true)+ b.ollamaState().managedFacade.Store(true)b.managedOllamaBackend.Store(managedOllamaBackendStart + 1)b.setProxy(&proxyProcess{})if got := b.takePendingManagedOllamaBackend(managedOllamaFacadePort); got != managedOllamaBackendStart+1 {@@ -183,11 +183,7 @@ func TestDuplicateOllamaReadyDoesNotOpenGateDuringMove(t *testing.T) {}b.reconcileProxyPortOnReady(managedOllamaFacadePort)- select {- case <-b.ollamaPortReady:- t.Fatal("duplicate ready opened the gate while the backend move was in flight")- default:- }+ requireGateShutNow(t, b.ollamaPortReady, "a duplicate ready while the backend move was in flight")}func TestOwningOllamaReadyOpensGateAfterMove(t *testing.T) {@@ -198,9 +194,9 @@ func TestOwningOllamaReadyOpensGateAfterMove(t *testing.T) {go worker.peer.Serve(nil, nil)b := &Broker{ollamaPortReady: make(chan struct{})}- b.managedOllamaFacade.Store(true)+ b.ollamaState().managedFacade.Store(true)b.managedOllamaBackend.Store(managedOllamaBackendStart + 1)- b.ollamaBackendPort.Store(managedOllamaBackendStart)+ b.ollamaState().backendPort.Store(managedOllamaBackendStart)b.setEngineMgr(worker)b.setProxy(&proxyProcess{})@@ -213,11 +209,7 @@ func TestOwningOllamaReadyOpensGateAfterMove(t *testing.T) {}()b.reconcileProxyPortOnReady(managedOllamaFacadePort)- select {- case <-b.ollamaPortReady:- default:- t.Fatal("owning reconciler did not open the gate after a successful move")- }+ requireGateOpenNow(t, b.ollamaPortReady, "a successful backend move")}func TestEnginePortAssignmentRequest(t *testing.T) {@@ -267,7 +259,7 @@ func TestManagedLMStudioRejectsFacadeBackendPort(t *testing.T) {defer brokerClient.Close()defer brokerServer.Close()b := &Broker{codec: NewCodec(brokerClient)}- b.managedLMStudioFacade.Store(true)+ b.lmstudioState().managedFacade.Store(true)response := make(chan *Message, 1)go func() {@@ -348,7 +340,7 @@ func TestLMStudioSetPortUpdatesBackendCache(t *testing.T) {case <-time.After(2 * time.Second):t.Fatal("missing LM Studio set-port response")}- if got := b.lmstudioBackendPort.Load(); got != 12400 {+ if got := b.lmstudioState().backendPort.Load(); got != 12400 {t.Fatalf("cached LM Studio backend = %d, want 12400", got)}}@@ -486,7 +478,7 @@ func TestRelayCachesActualOllamaPortFromResponse(t *testing.T) {case <-time.After(2 * time.Second):t.Fatal("timed out waiting for engine:start response")}- if got := b.ollamaBackendPort.Load(); got != 11435 {+ if got := b.ollamaState().backendPort.Load(); got != 11435 {t.Fatalf("cached Ollama backend port = %d, want returned port 11435", got)}}@@ -553,8 +545,8 @@ func TestOllamaPresenceRequestWaitsForPortGate(t *testing.T) {defer brokerServer.Close()b := &Broker{codec: NewCodec(brokerClient), ollamaPortReady: make(chan struct{})}b.setEngineMgr(worker)- b.managedOllamaFacade.Store(true)- b.ollamaBackendPort.Store(managedOllamaFacadePort)+ b.ollamaState().managedFacade.Store(true)+ b.ollamaState().backendPort.Store(managedOllamaFacadePort)method := make(chan string, 1)workerErr := make(chan error, 1)@@ -591,7 +583,7 @@ func TestOllamaPresenceRequestWaitsForPortGate(t *testing.T) {}// Model the successful backend move before releasing the existing gate.- b.ollamaBackendPort.Store(managedOllamaBackendStart)+ b.ollamaState().backendPort.Store(managedOllamaBackendStart)close(b.ollamaPortReady)select {case got := <-method:diff --git a/services/nvpair-ui-broker/proxyportbump_test.go b/services/nvpair-ui-broker/proxyportbump_test.gonew file mode 100644index 0000000..e22ae3b--- /dev/null+++ b/services/nvpair-ui-broker/proxyportbump_test.go@@ -0,0 +1,693 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package main++// Baseline coverage for the Ollama proxy-port paths that had none.+//+// resolveProxyPort, handleProxySetPort and the unmanaged branch of+// reconcileProxyPortOnReady were reachable only through code no test drove:+// the two existing reconcile tests store managedOllamaFacade=true first, so+// they return before the unmanaged branch, and nothing referenced the set-port+// handler at all. Those are precisely the paths the proxy unification+// generalizes onto LM Studio, so they need assertions that fail if the+// generalization changes them.++import (+ "context"+ "encoding/json"+ "fmt"+ "net"+ "reflect"+ "strconv"+ "strings"+ "testing"+ "time"++ "nvpair-shared/engines"+ "nvpair-shared/errors"+)++// brokerWithRunningEngines answers one engine:get-installed with the given+// ports marked running, which is what runningEnginePorts treats as taken.+// brokerWithEngineInventory reports no running flag, so it cannot drive these+// paths.+func brokerWithRunningEngines(t *testing.T, ports ...int) *Broker {+ t.Helper()+ client, server := net.Pipe()+ worker := &rpcWorker{peer: NewPeer(NewCodec(client))}+ go worker.peer.Serve(nil, nil)+ b := &Broker{nodeID: "local-node"}+ b.setEngineMgr(worker)+ go func() {+ codec := NewCodec(server)+ request, err := codec.Read()+ if err != nil {+ return+ }+ engines := make([]map[string]any, 0, len(ports))+ for _, port := range ports {+ engines = append(engines, map[string]any{"running": true, "port": port})+ }+ _ = codec.Respond(request.ID, map[string]any{"engines": engines})+ }()+ t.Cleanup(func() {+ _ = client.Close()+ _ = server.Close()+ })+ return b+}++// observeErrors attaches an errors process and returns a channel of the+// notifications the broker pushes into it.+func observeErrors(t *testing.T, b *Broker) <-chan *Message {+ t.Helper()+ client, server := net.Pipe()+ t.Cleanup(func() {+ _ = client.Close()+ _ = server.Close()+ })+ b.setErrors(&errorsProcess{peer: NewPeer(NewCodec(client))})++ seen := make(chan *Message, 4)+ go func() {+ codec := NewCodec(server)+ for {+ msg, err := codec.Read()+ if err != nil {+ return+ }+ select {+ case seen <- msg:+ default:+ }+ }+ }()+ return seen+}++func awaitErrorsNotification(t *testing.T, seen <-chan *Message, method string) *Message {+ t.Helper()+ deadline := time.After(2 * time.Second)+ for {+ select {+ case msg := <-seen:+ if msg != nil && msg.Method == method {+ return msg+ }+ case <-deadline:+ t.Fatalf("no %s notification arrived", method)+ return nil+ }+ }+}++// A running engine owns its port, so a proxy asked for that port is moved to+// the next free one and the move is surfaced as a sticky warning.+func TestResolveProxyPortBumpsPastRunningEngine(t *testing.T) {+ b := brokerWithRunningEngines(t, 11434)+ seen := observeErrors(t, b)++ if got := b.resolveProxyPort(11434); got != 11435 {+ t.Fatalf("resolveProxyPort(11434) = %d, want 11435", got)+ }++ msg := awaitErrorsNotification(t, seen, methodErrorsReport)+ if !strings.Contains(string(msg.Params), proxyPortBumpedID) {+ t.Fatalf("warning did not carry %s: %s", proxyPortBumpedID, msg.Params)+ }+ var reported struct {+ Severity string `json:"severity"`+ Action string `json:"action"`+ }+ if err := json.Unmarshal(msg.Params, &reported); err != nil {+ t.Fatal(err)+ }+ if reported.Severity != "warning" || reported.Action != "none" {+ t.Fatalf("severity/action = %q/%q, want warning/none", reported.Severity, reported.Action)+ }+}++// A request that collides with nothing clears any warning a previous bump+// left, so the notice cannot outlive the conflict that produced it.+func TestResolveProxyPortClearsStaleBumpWarning(t *testing.T) {+ b := brokerWithRunningEngines(t)+ seen := observeErrors(t, b)++ if got := b.resolveProxyPort(11500); got != 11500 {+ t.Fatalf("resolveProxyPort(11500) = %d, want it unchanged", got)+ }++ msg := awaitErrorsNotification(t, seen, methodErrorsClear)+ var cleared errors.ClearParams+ if err := json.Unmarshal(msg.Params, &cleared); err != nil {+ t.Fatal(err)+ }+ if cleared.ID != proxyPortBumpedID {+ t.Fatalf("cleared id = %q, want %q", cleared.ID, proxyPortBumpedID)+ }+}++// Managed mode overrides the request entirely: the proxy owns the facade port,+// so no bump warning applies.+func TestResolveProxyPortForcesFacadeWhenManaged(t *testing.T) {+ b := brokerWithRunningEngines(t)+ b.ollamaState().managedFacade.Store(true)++ if got := b.resolveProxyPort(9999); got != managedOllamaFacadePort {+ t.Fatalf("resolveProxyPort under managed mode = %d, want %d", got, managedOllamaFacadePort)+ }+}++// The unmanaged reconcile branch: a proxy that came up on a port a running+// engine holds is corrected onto the next free port, and the correction is+// announced. Both existing reconcile tests set managedOllamaFacade first and+// return before reaching this code.+func TestReconcileUnmanagedProxyPortBumpsOffRunningEngine(t *testing.T) {+ b := brokerWithRunningEngines(t, 11435)+ seen := observeErrors(t, b)++ proxyClient, proxyServer := net.Pipe()+ defer proxyClient.Close()+ defer proxyServer.Close()+ proxy := &proxyProcess{peer: NewPeer(NewCodec(proxyClient))}+ go proxy.peer.Serve(nil, nil)+ b.setProxy(proxy)++ setPort := make(chan int, 1)+ go func() {+ codec := NewCodec(proxyServer)+ msg, err := codec.Read()+ if err != nil {+ return+ }+ var params struct {+ Port int `json:"port"`+ }+ _ = json.Unmarshal(msg.Params, ¶ms)+ select {+ case setPort <- params.Port:+ default:+ }+ _ = codec.Respond(msg.ID, map[string]any{"port": params.Port})+ }()++ b.reconcileProxyPortOnReady(11435)++ select {+ case got := <-setPort:+ if got != 11436 {+ t.Fatalf("corrective set-port asked for %d, want 11436", got)+ }+ case <-time.After(2 * time.Second):+ t.Fatal("no corrective set-port reached the proxy")+ }++ msg := awaitErrorsNotification(t, seen, methodErrorsReport)+ if !strings.Contains(string(msg.Params), proxyPortBumpedID) {+ t.Fatalf("bump was not announced: %s", msg.Params)+ }+}++// A proxy on a port nothing holds is left alone — no corrective call, so the+// reconcile must not disturb a healthy unmanaged proxy.+func TestReconcileUnmanagedProxyPortLeavesFreePortAlone(t *testing.T) {+ b := brokerWithRunningEngines(t)++ proxyClient, proxyServer := net.Pipe()+ defer proxyClient.Close()+ defer proxyServer.Close()+ proxy := &proxyProcess{peer: NewPeer(NewCodec(proxyClient))}+ go proxy.peer.Serve(nil, nil)+ b.setProxy(proxy)++ called := make(chan struct{}, 1)+ go func() {+ codec := NewCodec(proxyServer)+ if _, err := codec.Read(); err == nil {+ select {+ case called <- struct{}{}:+ default:+ }+ }+ }()++ b.reconcileProxyPortOnReady(11435)++ select {+ case <-called:+ t.Fatal("reconcile issued a corrective set-port for a port no engine holds")+ case <-time.After(250 * time.Millisecond):+ }+}++// The facade/enable request the broker hands an Ollama proxy. This is where the+// port decision and the inherited OLLAMA_HOST alias travel now; the child's+// argv carries only process-scoped flags, because a single-valued flag cannot+// express a different port plan per engine.+func TestOllamaFacadeSpec(t *testing.T) {+ for _, tc := range []struct {+ name string+ startupPort int+ alias ollamaHostAlias+ want enableFacadeRequest+ }{+ {+ name: "managed facade",+ startupPort: managedOllamaFacadePort,+ want: enableFacadeRequest{+ Engine: "ollama",+ Port: managedOllamaFacadePort,+ IgnorePersistedPort: true,+ },+ },+ {+ // No port named, so the child keeps its persisted one. Naming a port+ // here would override whatever the user last chose via set-port.+ name: "no startup port leaves the proxy on its persisted or default port",+ startupPort: 0,+ want: enableFacadeRequest{Engine: "ollama"},+ },+ {+ name: "inherited OLLAMA_HOST alias is threaded through",+ startupPort: managedOllamaFacadePort,+ alias: ollamaHostAlias{Address: "127.0.0.1:11433", AlternateAddress: "[::1]:11433"},+ want: enableFacadeRequest{+ Engine: "ollama",+ Port: managedOllamaFacadePort,+ IgnorePersistedPort: true,+ AliasAddresses: []string{"127.0.0.1:11433", "[::1]:11433"},+ },+ },+ } {+ t.Run(tc.name, func(t *testing.T) {+ b := &Broker{}+ b.ollamaState().startupPort.Store(int32(tc.startupPort))++ if got := b.ollamaFacadeSpec(tc.alias); !reflect.DeepEqual(got, tc.want) {+ t.Fatalf("facade spec = %+v, want %+v", got, tc.want)+ }+ })+ }+}++// serveFacadeEnable answers facade/enable frames on a pipe, recording the port+// each attempt asked for and rejecting the ones in reject as bind races.+//+// It replies with codeFacadeBindFailed rather than a generic error because that+// code is the entire basis on which the broker decides a retry is worthwhile.+func serveFacadeEnable(t *testing.T, conn net.Conn, reject map[int]bool, attempts chan<- enableFacadeRequest) {+ t.Helper()+ go func() {+ codec := NewCodec(conn)+ for {+ msg, err := codec.Read()+ if err != nil {+ return+ }+ if msg.Method != "facade/enable" {+ continue+ }+ var spec enableFacadeRequest+ if err := json.Unmarshal(msg.Params, &spec); err != nil {+ return+ }+ attempts <- spec+ if reject[spec.Port] {+ _ = codec.RespondError(msg.ID, codeFacadeBindFailed,+ fmt.Sprintf("facade bind failed: port %d: address already in use", spec.Port))+ continue+ }+ _ = codec.Respond(msg.ID, enableFacadeReply{Engine: spec.Engine, Port: spec.Port})+ }+ }()+}++// A facade that loses a bind race is retried on a fallback port inside the same+// process. It used to be a process exit: the child log.Fatalf'd and the+// supervisor respawned it on a corrected port. That is only acceptable while a+// process hosts one engine — once it hosts several, one engine losing a race+// would take working listeners down with it.+func TestFacadeBindRaceRetriesInProcess(t *testing.T) {+ proxyClient, proxyServer := net.Pipe()+ t.Cleanup(func() {+ _ = proxyClient.Close()+ _ = proxyServer.Close()+ })+ proxy := &proxyProcess{peer: NewPeer(NewCodec(proxyClient))}+ go proxy.peer.Serve(nil, nil)++ attempts := make(chan enableFacadeRequest, 4)+ serveFacadeEnable(t, proxyServer, map[int]bool{managedOllamaFacadePort: true}, attempts)++ b := &Broker{}+ b.ollamaState().managedFacade.Store(true)+ b.ollamaState().backendPort.Store(managedOllamaBackendStart)++ const fallback = 12000+ err := b.enableProxyFacadeWithFallback(context.Background(), proxy,+ enableFacadeRequest{+ Engine: ollamaProxyProfile.Name,+ Port: managedOllamaFacadePort,+ IgnorePersistedPort: true,+ },+ func(failed int) int {+ if failed != managedOllamaFacadePort {+ t.Errorf("fallback asked for port %d, want the port that failed (%d)", failed, managedOllamaFacadePort)+ }+ return fallback+ })+ if err != nil {+ t.Fatalf("enable with fallback = %v, want success on the fallback port", err)+ }++ first := <-attempts+ if first.Port != managedOllamaFacadePort {+ t.Fatalf("first attempt port = %d, want %d", first.Port, managedOllamaFacadePort)+ }+ second := <-attempts+ if second.Port != fallback {+ t.Fatalf("retry port = %d, want the fallback %d", second.Port, fallback)+ }+ // The persisted port is the one that just failed to bind, so a retry that+ // let the child restore it would land straight back on the taken port.+ if !second.IgnorePersistedPort {+ t.Error("retry did not set ignorePersistedPort, so the child could restore the port that just failed")+ }+ if second.Engine != first.Engine {+ t.Errorf("retry engine = %q, want %q", second.Engine, first.Engine)+ }+}++// A rejection no other port would fix must not be retried. Retrying an+// unknown-engine or unsupported-alias enable would burn a port change and hide+// the real cause behind a second identical failure.+func TestFacadeEnableRejectionIsNotRetried(t *testing.T) {+ proxyClient, proxyServer := net.Pipe()+ t.Cleanup(func() {+ _ = proxyClient.Close()+ _ = proxyServer.Close()+ })+ proxy := &proxyProcess{peer: NewPeer(NewCodec(proxyClient))}+ go proxy.peer.Serve(nil, nil)++ attempts := make(chan enableFacadeRequest, 4)+ go func() {+ codec := NewCodec(proxyServer)+ for {+ msg, err := codec.Read()+ if err != nil {+ return+ }+ var spec enableFacadeRequest+ if json.Unmarshal(msg.Params, &spec) != nil {+ return+ }+ attempts <- spec+ // -32602 is what an unknown engine or an unsupported alias returns.+ _ = codec.RespondError(msg.ID, -32602, "unknown engine \"nope\"")+ }+ }()++ b := &Broker{}+ fallbackCalls := 0+ err := b.enableProxyFacadeWithFallback(context.Background(), proxy,+ enableFacadeRequest{Engine: "nope", Port: 11434},+ func(int) int {+ fallbackCalls+++ return 12000+ })+ if err == nil {+ t.Fatal("enable with fallback = nil, want the rejection surfaced")+ }+ if fallbackCalls != 0 {+ t.Errorf("fallback consulted %d times, want 0: only a bind race is retryable", fallbackCalls)+ }+ <-attempts+ select {+ case extra := <-attempts:+ t.Fatalf("a second enable was sent for %+v; a rejection must not be retried", extra)+ case <-time.After(200 * time.Millisecond):+ }+}++// LM Studio's set-port used to relay verbatim, so a user could park its proxy+// on a port a running engine already held and hear nothing about it. Both+// engines now resolve the request the same way, and the warning names the+// engine so two proxies bumping cannot produce one indistinguishable notice.+func TestBothProxiesResolveSetPortAgainstRunningEngines(t *testing.T) {+ for _, tc := range []struct {+ engine string+ requested int+ want int+ }{+ {engine: "ollama", requested: 11434, want: 11435},+ {engine: "lmstudio", requested: 1234, want: 1235},+ } {+ t.Run(tc.engine, func(t *testing.T) {+ profile, ok := engineProxyProfileFor(tc.engine)+ if !ok {+ t.Fatalf("no profile for %s", tc.engine)+ }+ b := brokerWithRunningEngines(t, tc.requested)+ seen := observeErrors(t, b)++ if got := b.resolveEngineProxyPort(profile, tc.requested); got != tc.want {+ t.Fatalf("resolveEngineProxyPort(%d) = %d, want %d", tc.requested, got, tc.want)+ }++ msg := awaitErrorsNotification(t, seen, methodErrorsReport)+ if !strings.Contains(string(msg.Params), profile.portBumpedID()) {+ t.Fatalf("warning did not carry %s: %s", profile.portBumpedID(), msg.Params)+ }+ if !strings.Contains(string(msg.Params), profile.DisplayName) {+ t.Fatalf("warning does not name the engine: %s", msg.Params)+ }+ })+ }+}++// A managed facade is not negotiable: the request is answered with the facade+// port and any stale bump warning is cleared, for either engine.+func TestManagedFacadeOverridesRequestedProxyPort(t *testing.T) {+ b := brokerWithRunningEngines(t)+ b.lmstudioState().managedFacade.Store(true)++ if got := b.resolveEngineProxyPort(lmstudioProxyProfile, 4321); got != managedLMStudioFacadePort {+ t.Fatalf("resolveEngineProxyPort = %d, want the managed facade %d", got, managedLMStudioFacadePort)+ }+}++// The proxy's explicit fallback port must never be one the engine is going to+// want. When the configured backend is unknown, a free port is not evidence the+// engine will not claim it, and the stock backend port is exactly where a+// managed Ollama gets placed — so both it and the facade are excluded until an+// authoritative port is known.+func TestOllamaProxyFallbackAvoidsTheEnginesPorts(t *testing.T) {+ t.Run("backend unknown excludes both stock ports", func(t *testing.T) {+ b := &Broker{}+ fallback := b.setOllamaProxyFallback()++ if fallback == managedOllamaFacadePort || fallback == managedOllamaBackendStart {+ t.Fatalf("fallback = %d, want neither the facade (%d) nor the stock backend (%d) while the backend is unknown",+ fallback, managedOllamaFacadePort, managedOllamaBackendStart)+ }+ if got := int(b.ollamaState().startupPort.Load()); got != fallback {+ t.Fatalf("startup port = %d, want the chosen fallback %d", got, fallback)+ }+ })++ t.Run("known backend is excluded", func(t *testing.T) {+ b := &Broker{}+ b.ollamaState().backendPort.Store(managedOllamaBackendStart)++ if fallback := b.setOllamaProxyFallback(); fallback == managedOllamaBackendStart {+ t.Fatalf("fallback = %d, want the configured backend port excluded", fallback)+ }+ })++ t.Run("callers need not pass the backend", func(t *testing.T) {+ // blockManagedOllamaFacade used to append the backend itself. The+ // exclusion moved into setOllamaProxyFallback so no call site can omit+ // it; this pins that the blocked path still avoids the engine's port.+ b := &Broker{nodeID: "local-node", ollamaPortReady: make(chan struct{})}+ b.ollamaState().backendPort.Store(managedOllamaBackendStart)++ if fallback := b.blockManagedOllamaFacade("test"); fallback == managedOllamaBackendStart {+ t.Fatalf("blocked fallback = %d, want the configured backend port excluded", fallback)+ }+ })+}++// The two ways an engine ends up without a proxy need different treatment. An+// unresolved binary is a packaging or --proxy-path problem the user should see;+// an engine deliberately left out of --proxy-engines is not, and reporting it+// would be noise for something they asked for.+//+// Gating preparation on proxyEnabled made this worth pinning: the disabled+// branches used to reach the blocked-ownership report through managedFacade,+// which is now necessarily false there, so the report had to move.+func TestUnavailableProxyIsReportedButDeselectionIsNot(t *testing.T) {+ reason, report := proxyDisabledReason("")+ if !report {+ t.Error("an unresolved proxy binary is not reported; a packaging problem would be log-only")+ }+ if !strings.Contains(reason, "not resolved") {+ t.Errorf("reason for a missing binary = %q", reason)+ }++ reason, report = proxyDisabledReason("/path/to/nvpair-proxy")+ if report {+ t.Error("a deliberately deselected engine is reported as an error; that is noise for a requested state")+ }+ if !strings.Contains(reason, "--proxy-engines") {+ t.Errorf("reason for a deselected engine = %q", reason)+ }+}++// A bind failure has to move the next spawn somewhere else. The supervisor+// restarts the proxy with whatever startup port is currently stored, so leaving+// it unchanged means respawning onto the port that just rejected the bind, over+// and over until the restart budget runs out. LM Studio handled this; Ollama+// only handled the managed-facade case and ignored every other bind failure.+func TestBindFailureMovesTheNextSpawnOffTheContestedPort(t *testing.T) {+ t.Run("ollama", func(t *testing.T) {+ b := &Broker{nodeID: "local-node", ollamaPortReady: make(chan struct{})}+ // Unmanaged: no facade claim, so this is the path that used to be+ // ignored entirely.+ b.ollamaState().backendPort.Store(managedOllamaBackendStart)+ contested := managedOllamaBackendStart + 7+ b.ollamaState().startupPort.Store(int32(contested))++ b.forwardProxyNotification("error", json.RawMessage(+ `{"code":"bind-failed","port":`+strconv.Itoa(contested)+`}`))++ if got := int(b.ollamaState().startupPort.Load()); got == contested {+ t.Fatalf("startup port still %d; the next spawn would hit the same bind failure", got)+ }+ })++ t.Run("lmstudio", func(t *testing.T) {+ b := &Broker{nodeID: "local-node", lmstudioPortReady: make(chan struct{})}+ b.lmstudioState().backendPort.Store(managedLMStudioBackendStart)+ contested := managedLMStudioBackendStart + 7+ b.lmstudioState().startupPort.Store(int32(contested))++ b.forwardLMStudioProxyNotification("error", json.RawMessage(+ `{"code":"bind-failed","port":`+strconv.Itoa(contested)+`}`))++ if got := int(b.lmstudioState().startupPort.Load()); got == contested {+ t.Fatalf("startup port still %d; the next spawn would hit the same bind failure", got)+ }+ })+}++// A "ready" from a proxy generation that has already been replaced must not+// drive the current generation's reconciliation. The dead process announced a+// port it no longer holds, and acting on it opens the ownership gate — and so+// releases every request waiting behind it — while the replacement is still+// deciding where to bind.+//+// LM Studio pinned this; Ollama's only stale-generation test covered the alias+// release, leaving the readiness path uncovered on the engine whose gate also+// gates engine:status and engine:get-installed.+func TestStaleProxyReadyDoesNotOpenTheOwnershipGate(t *testing.T) {+ newBroker := func(t *testing.T) *Broker {+ t.Helper()+ b := &Broker{nodeID: "local-node", ollamaPortReady: make(chan struct{})}+ observeErrors(t, b)+ // Managed with nothing pending is the state where a ready would+ // otherwise settle ownership immediately, so the guard is the only+ // thing keeping the gate shut.+ b.ollamaState().managedFacade.Store(true)+ // Reconciliation returns early without a live proxy, which would make+ // the stale case below pass for the wrong reason.+ client, server := net.Pipe()+ t.Cleanup(func() {+ _ = client.Close()+ _ = server.Close()+ })+ proxy := &proxyProcess{peer: NewPeer(NewCodec(client))}+ go proxy.peer.Serve(nil, nil)+ b.setProxy(proxy)+ return b+ }+ ready := json.RawMessage(fmt.Sprintf(`{"port":%d}`, managedOllamaFacadePort))++ t.Run("stale generation is ignored", func(t *testing.T) {+ b := newBroker(t)+ stale, _ := b.beginOllamaProxyGeneration()+ b.beginOllamaProxyGeneration() // the replacement++ b.forwardProxyNotificationForGeneration(stale, "ready", ready)++ requireGateStaysShut(t, b.ollamaPortReady, "a replaced proxy's ready")+ })++ // Without this the test above would pass even if the guard swallowed every+ // ready, stale or not.+ t.Run("current generation still settles ownership", func(t *testing.T) {+ b := newBroker(t)+ current, _ := b.beginOllamaProxyGeneration()++ b.forwardProxyNotificationForGeneration(current, "ready", ready)++ requireGateOpens(t, b.ollamaPortReady, "the current proxy's ready")+ })+}++// A proxy the supervisor has given up on must release its ownership gate.+// Otherwise every gated engine request waits out rpcWorkerCallTimeout and+// answers "retry" for the life of the broker, because nothing else ever closes+// the channel. LM Studio wired onExhausted to do this; Ollama did not.+func TestTerminalProxyReleasesOwnershipGate(t *testing.T) {+ t.Run("ollama", func(t *testing.T) {+ b := &Broker{+ nodeID: "local-node",+ ollamaPortReady: make(chan struct{}),+ lmstudioPortReady: make(chan struct{}),+ }+ // Mid-move state left behind by the proxy that died: the facade is+ // claimed and a backend move is pending.+ b.ollamaState().managedFacade.Store(true)+ b.managedOllamaBackend.Store(managedOllamaBackendStart)+ b.ollamaMoveInFlight.Store(true)++ sup := newSupervisor(engines.ProxyComponent, defaultRestartPolicy(), nil)+ b.configureProxySupervisorCallbacks(sup)+ if sup.onExhausted == nil {+ t.Fatal("proxy supervisor has no onExhausted; a terminal proxy would strand the gate")+ }+ sup.onExhausted(3)++ requireGateOpenNow(t, b.ollamaPortReady, "the proxy going terminal")+ // Closing the channel is not enough for Ollama: relayToEngine re-checks+ // this predicate after the wait and rejects while it holds.+ if b.ollamaFacadeIsPendingBackend() {+ t.Fatal("gate reopened but the pending-move re-check still rejects; requests would keep failing")+ }+ })++ t.Run("lmstudio", func(t *testing.T) {+ b := &Broker{+ nodeID: "local-node",+ ollamaPortReady: make(chan struct{}),+ lmstudioPortReady: make(chan struct{}),+ }+ b.lmstudioState().managedFacade.Store(true)++ sup := newSupervisor(engines.ProxyComponent, defaultRestartPolicy(), nil)+ b.configureProxySupervisorCallbacks(sup)+ if sup.onExhausted == nil {+ t.Fatal("proxy supervisor has no onExhausted")+ }+ sup.onExhausted(3)++ if b.lmstudioPortOwnershipPending() {+ t.Fatal("ownership gate still pending after the proxy went terminal")+ }+ if b.lmstudioState().managedFacade.Load() {+ t.Fatal("managed facade still claimed by a proxy that will not return")+ }+ })+}diff --git a/services/nvpair-ui-broker/scheduler_test.go b/services/nvpair-ui-broker/scheduler_test.goindex f74dcd4..30ced48 100644--- a/services/nvpair-ui-broker/scheduler_test.go+++ b/services/nvpair-ui-broker/scheduler_test.go@@ -4,9 +4,9 @@package mainimport (+ "context""encoding/json""net"- "reflect""sync""testing""time"@@ -131,7 +131,7 @@ func TestDeliverPrioritySkipsStaleGenerationsAndPreservesNewest(t *testing.T) {Nodes: []string{"old"},Ranks: []schedulerwire.NodeRank{{ID: "old", Pending: 1}},}- oldGeneration := b.cachePrioritySnapshot("ollama", oldInput)+ oldGeneration, _ := b.cachePrioritySnapshot(oldInput)oldInput.Nodes[0] = "mutated-after-cache"oldInput.Ranks[0].Pending = 99@@ -147,7 +147,7 @@ func TestDeliverPrioritySkipsStaleGenerationsAndPreservesNewest(t *testing.T) {releaseOld := make(chan struct{})oldDone := make(chan struct{})go func() {- b.deliverPrioritySnapshot("ollama", oldGeneration, func(priority schedulerwire.Priority) {+ b.deliverPrioritySnapshot(oldGeneration, func(priority schedulerwire.Priority) {record(priority)close(oldEntered)<-releaseOld@@ -156,23 +156,23 @@ func TestDeliverPrioritySkipsStaleGenerationsAndPreservesNewest(t *testing.T) {}()waitSchedulerTestChannel(t, oldEntered, "old delivery did not start")- middleGeneration := b.cachePrioritySnapshot("ollama", schedulerwire.Priority{+ middleGeneration, _ := b.cachePrioritySnapshot(schedulerwire.Priority{Nodes: []string{"middle"},Ranks: []schedulerwire.NodeRank{{ID: "middle", Pending: 2}},})middleDone := make(chan struct{})go func() {- b.deliverPrioritySnapshot("ollama", middleGeneration, record)+ b.deliverPrioritySnapshot(middleGeneration, record)close(middleDone)}()newPriority := schedulerwire.Priority{Nodes: []string{"new"},Ranks: []schedulerwire.NodeRank{{ID: "new", Pending: 3}},}- newGeneration := b.cachePrioritySnapshot("ollama", newPriority)+ newGeneration, _ := b.cachePrioritySnapshot(newPriority)newDone := make(chan struct{})go func() {- b.deliverPrioritySnapshot("ollama", newGeneration, record)+ b.deliverPrioritySnapshot(newGeneration, record)close(newDone)}()@@ -190,12 +190,73 @@ func TestDeliverPrioritySkipsStaleGenerationsAndPreservesNewest(t *testing.T) {Nodes: []string{"old"},Ranks: []schedulerwire.NodeRank{{ID: "old", Pending: 1}},}- if !reflect.DeepEqual(applied[0], wantOld) {+ // Ranking rather than DeepEqual: a cached snapshot also carries the+ // generation the broker stamped on it, which is not part of what the+ // scheduler produced.+ if !applied[0].SameRanking(wantOld) {t.Fatalf("first applied snapshot = %#v, want cached copy %#v", applied[0], wantOld)}- if !reflect.DeepEqual(applied[1], newPriority) {+ if applied[0].Generation != oldGeneration {+ t.Fatalf("first applied generation = %d, want %d", applied[0].Generation, oldGeneration)+ }+ if !applied[1].SameRanking(newPriority) {t.Fatalf("last applied snapshot = %#v, want newest %#v", applied[1], newPriority)}+ if applied[1].Generation != newGeneration {+ t.Fatalf("last applied generation = %d, want %d", applied[1].Generation, newGeneration)+ }+}++// The scheduler computes one node-wide ranking and emits it once per engine, so+// the broker sees the same content twice per recompute. The duplicate has to be+// dropped before it mints a generation: a second delivery of one recompute+// clears the proxy's optimistic reservations a second time, discarding the+// dispatches the first delivery's pending counts do not yet include.+func TestDuplicatePerEngineEmissionDoesNotMintAGeneration(t *testing.T) {+ b := &Broker{}+ ranking := schedulerwire.Priority{+ Nodes: []string{"b", "a"},+ Ranks: []schedulerwire.NodeRank{{ID: "b", Pending: 1}, {ID: "a", Pending: 4}},+ }++ first, fresh := b.cachePrioritySnapshot(ranking)+ if !fresh {+ t.Fatal("first emission of a ranking was treated as a duplicate")+ }++ second, fresh := b.cachePrioritySnapshot(ranking)+ if fresh {+ t.Fatal("the sibling engine's identical emission minted a second generation")+ }+ if second != first {+ t.Fatalf("duplicate emission moved the generation from %d to %d", first, second)+ }++ // A genuinely changed ranking still advances, so the dedupe is not just+ // swallowing everything after the first.+ changed, fresh := b.cachePrioritySnapshot(schedulerwire.Priority{+ Nodes: []string{"a", "b"},+ Ranks: []schedulerwire.NodeRank{{ID: "a", Pending: 0}, {ID: "b", Pending: 5}},+ })+ if !fresh || changed <= first {+ t.Fatalf("changed ranking = generation %d fresh %v, want a newer generation", changed, fresh)+ }+}++// An empty ranking is a legitimate state — it means the scheduler has no+// influence — and must be distinguishable from having no ranking at all, or a+// repush would either send nothing or send an uncached zero value.+func TestEmptyRankingIsCachedNotTreatedAsAbsent(t *testing.T) {+ b := &Broker{}+ if _, fresh := b.cachePrioritySnapshot(schedulerwire.Priority{}); !fresh {+ t.Fatal("first empty ranking was treated as a duplicate")+ }+ b.schedMu.Lock()+ have := b.havePriority+ b.schedMu.Unlock()+ if !have {+ t.Fatal("an empty ranking did not mark a ranking as cached, so repush would skip it")+ }}func TestRepushPriorityDeliversCompleteSnapshotToReplacementProxy(t *testing.T) {@@ -215,12 +276,12 @@ func TestRepushPriorityDeliversCompleteSnapshotToReplacementProxy(t *testing.T){ID: "a", Pending: 4, Rank: 1},},}- b.cachePrioritySnapshot("ollama", want)+ b.cachePrioritySnapshot(want)b.setProxy(proxy) // replacement proxy starts with no scheduler statereplayed := make(chan struct{})go func() {- b.repushPriority("ollama")+ b.repushPriority(context.Background())close(replayed)}()@@ -236,9 +297,15 @@ func TestRepushPriorityDeliversCompleteSnapshotToReplacementProxy(t *testing.T)if err := json.Unmarshal(request.Params, &got); err != nil {t.Fatalf("decode replacement priority: %v", err)}- if !reflect.DeepEqual(got, want) {+ if !got.SameRanking(want) {t.Fatalf("replacement priority = %#v, want %#v", got, want)}+ // The replay reuses the cached generation rather than minting one, so the+ // child can recognize a redelivery it has already applied. A repush that+ // arrived unversioned would clear reservations on every proxy ready.+ if got.Generation == 0 {+ t.Fatal("replayed snapshot carried no generation; the child cannot detect a redelivery")+ }if err := codec.Respond(request.ID, map[string]int{"count": len(got.Nodes)}); err != nil {t.Fatalf("respond to replacement priority: %v", err)}diff --git a/services/nvpair-ui-broker/stopwait.go b/services/nvpair-ui-broker/stopwait.goindex 99fd5d1..0f0d6d6 100644--- a/services/nvpair-ui-broker/stopwait.go+++ b/services/nvpair-ui-broker/stopwait.go@@ -23,8 +23,8 @@ const (// engine processes it launched (StopAll) on its way out. Cutting that short is// exactly what the old 2s-grace-then-kill did, and it orphaned engines.// graceFor reserves this out of the shared budget so the workers joined ahead- // of it cannot spend it — it is joined eighth of eleven, so join order alone- // would otherwise decide how much it gets.+ // of it cannot spend it — it is joined late in the sequence, so join order+ // alone would otherwise decide how much it gets.engineManagerStopGrace = 6 * time.Second// engineStopAllBudget bounds the broker's wait for engine-manager to stop the@@ -39,9 +39,9 @@ const (// teardownBudget bounds everything the broker does between being told to shut// down and its last worker join. The joins run one after another, so- // per-worker graces alone add up: a tree where each of the eleven workers- // hangs for its own grace outlives the parent's shutdown grace several times- // over, and being killed from outside mid-teardown skips the rest of shutdown.+ // per-worker graces alone add up: a tree where every worker hangs for its+ // own grace outlives the parent's shutdown grace several times over, and+ // being killed from outside mid-teardown skips the rest of shutdown.//// The accounting the parent's grace has to cover, in order://@@ -49,8 +49,8 @@ const (// workload history shutdown flush <= workloadHistoryFlushJoinTimeout (5s)//// which is the 15s the desktop parent and nvpair-tui both allow. Everything- // inside the first line — the two proxy joins, the engine StopAll wait, and- // all eleven joins — draws on this one budget, so no step can be lengthened+ // inside the first line — the proxy join, the engine StopAll wait, and every+ // other worker join — draws on this one budget, so no step can be lengthened// without the others giving way. It must also exceed the largest single worker// grace, or that worker could never be granted it.teardownBudget = 10 * time.Second@@ -67,10 +67,10 @@ const (// teardownClock records when shutdown began.//// It is armed explicitly rather than by the first join, because a join also-// happens at ordinary runtime: supervisor.Restart stops the current worker before-// spawning a replacement, which LM Studio port reconciliation does on a rebind.-// Arming the clock there would leave the budget permanently spent for the life of-// the process, so every later join — including engine-manager's at the real+// happens at ordinary runtime: spawnProxy stops the child it just started when+// no engine facade came up, so the supervisor can retry a clean one. Arming the+// clock there would leave the budget permanently spent for the life of the+// process, so every later join — including engine-manager's at the real// shutdown — would fall to minWorkerGrace and be signalled almost immediately.var teardownClock struct {mu sync.Mutex@@ -87,9 +87,10 @@ func beginTeardown() {}// teardownRemaining reports what is left of the budget, and whether the budget-// applies at all. A Stop outside teardown (supervisor.Restart) is not clipped:-// there is no sequence of joins for it to add up with, and a restarting worker-// still deserves its full grace to drain.+// applies at all. A Stop outside teardown — spawnProxy discarding a child whose+// facades all failed — is not clipped: there is no sequence of joins for it to+// add up with, and a worker being replaced still deserves its full grace to+// drain.func teardownRemaining() (time.Duration, bool) {teardownClock.mu.Lock()defer teardownClock.mu.Unlock()diff --git a/services/nvpair-ui-broker/supervisor.go b/services/nvpair-ui-broker/supervisor.goindex 9bb9386..795851d 100644--- a/services/nvpair-ui-broker/supervisor.go+++ b/services/nvpair-ui-broker/supervisor.go@@ -106,9 +106,6 @@ const (// watchExited means the worker process exited on its own (an// unexpected death the supervisor should react to).watchExited- // watchRestart means Restart was called: stop the current (healthy)- // worker and respawn it promptly, without counting it as a crash.- watchRestart)// supervisor owns the lifecycle of a single broker worker: it spawns the@@ -146,28 +143,34 @@ type supervisor struct {// before it waits for Stop.onExhausted func(attempt int)+ // onSpawned, if set, is invoked after every successful spawn — the first+ // from Start and each respawn from monitor — and always after spawn has+ // published the handle. It exists because a fresh worker process starts+ // with none of the state the broker pushed into its predecessor, and spawn+ // is the only event that loses that state.+ //+ // Invoked on its own goroutine, deliberately. The respawn sites run on the+ // monitor goroutine, which is the only thing that reaches h.Stop() when+ // stopCh closes, so a callback that blocks there delays crash detection and+ // makes Stop wait it out inside the shared teardown budget. A callback that+ // talks to the worker must give up when Stopping() closes, or it races a+ // teardown that is about to close the worker's stdin.+ onSpawned func()+stopCh chan struct{}stopOnce sync.OncedoneCh chan struct{}-- // restartCh requests a graceful restart of the current worker process so- // it re-reads external state it only loads at startup — e.g. cluster- // identity/pins that appear under --cluster-dir when this node joins a- // cluster, flipping the worker from plain HTTP to pin-gated mTLS (a- // bind-time decision). Buffered + coalesced via Restart.- restartCh chan struct{}}// newSupervisor constructs a supervisor for one worker. onCrash and// onRecovered are set by the caller as needed before Start.func newSupervisor(name string, policy restartPolicy, spawn func() (supervisedHandle, error)) *supervisor {return &supervisor{- name: name,- policy: policy,- spawn: spawn,- stopCh: make(chan struct{}),- doneCh: make(chan struct{}),- restartCh: make(chan struct{}, 1),+ name: name,+ policy: policy,+ spawn: spawn,+ stopCh: make(chan struct{}),+ doneCh: make(chan struct{}),}}@@ -181,10 +184,23 @@ func (s *supervisor) Start() error {if err != nil {return err}+ s.notifySpawned()go s.monitor(h)return nil}+// Stopping reports teardown to an onSpawned callback, so work it started+// against the worker can be abandoned rather than racing a closing stdin.+func (s *supervisor) Stopping() <-chan struct{} { return s.stopCh }++// notifySpawned runs onSpawned off the calling goroutine. See the field comment+// for why that is required rather than tidy.+func (s *supervisor) notifySpawned() {+ if s.onSpawned != nil {+ go s.onSpawned()+ }+}+// workerStopReportAfter is how long a worker may take to tear down before this// starts naming it. The whole tree normally stops in well under a second, so a// worker past this is either doing real work or wedged.@@ -215,19 +231,6 @@ func (s *supervisor) Stop() {"worker", s.name, "waitedMs", time.Since(started).Milliseconds())}-// Restart asks the supervisor to stop the current worker process and spawn a-// fresh one, without counting it as a crash. Used to re-point a worker at-// external state it only reads at startup — e.g. cluster identity/pins that-// appear under --cluster-dir when this node joins a cluster, which flip the-// worker from plain HTTP to pin-gated mTLS (a bind-time decision). Non-blocking-// and coalescing: a burst collapses to one restart. A no-op once Stopped.-func (s *supervisor) Restart() {- select {- case s.restartCh <- struct{}{}:- default:- }-}-// monitor is the supervisor's control loop. It watches the current handle;// on an unexpected exit it surfaces the crash, then backs off and respawns// per the policy, until either the worker recovers (resetting the budget),@@ -255,26 +258,7 @@ func (s *supervisor) monitor(h supervisedHandle) {h.Stop()return}- if res == watchRestart {- // Graceful, operator-driven restart (e.g. cluster certs appeared on- // join): stop the running worker and respawn it promptly. Not a- // crash — it surfaces no error and doesn't spend the restart budget.- slog.Info("supervisor: restarting worker on request", "worker", s.name)- h.Stop()- if nh, err := s.spawn(); err == nil {- attempts = 0- h = nh- continue- } else {- // A requested respawn failed (rare — the same command was just- // running). Fall through to the crash/backoff path so it still- // recovers rather than wedging the worker down.- slog.Warn("supervisor: requested restart respawn failed; falling back to crash recovery", "worker", s.name, "err", err)- }- }-- // Unexpected exit (crash) — or a failed requested respawn above. Surface- // it and decide whether to restart.+ // Unexpected exit (crash). Surface it and decide whether to restart.attempts++slog.Warn("supervisor: worker exited unexpectedly", "worker", s.name, "attempt", attempts)if s.onCrash != nil {@@ -308,6 +292,7 @@ func (s *supervisor) monitor(h supervisedHandle) {}slog.Info("supervisor: worker restarted", "worker", s.name, "attempt", attempts)h = next+ s.notifySpawned()}}@@ -350,8 +335,6 @@ func (s *supervisor) watch(h supervisedHandle, onStable func()) watchResult {select {case <-s.stopCh:return watchStopped- case <-s.restartCh:- return watchRestartcase <-healthyC:// Fire once: nil-out the channel so this arm blocks forever// after the first tick (the worker is now considered stable).diff --git a/services/nvpair-ui-broker/supervisor_test.go b/services/nvpair-ui-broker/supervisor_test.goindex 217c05e..605e46d 100644--- a/services/nvpair-ui-broker/supervisor_test.go+++ b/services/nvpair-ui-broker/supervisor_test.go@@ -29,6 +29,78 @@ func (h *fakeHandle) Stop() { h.stopOnce.Do(func() { close(h.stoppedC) }) }// crash closes the handle's done channel, simulating an unexpected exit.func (h *fakeHandle) crash() { close(h.done) }+// A fresh worker process has none of the state the broker pushed into its+// predecessor, so onSpawned has to fire for the first spawn and every respawn.+// Missing the respawn is the interesting failure: the worker comes back and+// silently serves without the ranking the broker believes it has.+func TestOnSpawnedFiresForFirstSpawnAndEveryRespawn(t *testing.T) {+ handles := make(chan *fakeHandle, 4)+ spawned := make(chan struct{}, 4)++ sup := newSupervisor("replay-probe", defaultRestartPolicy(), func() (supervisedHandle, error) {+ h := newFakeHandle()+ handles <- h+ return h, nil+ })+ sup.onSpawned = func() { spawned <- struct{}{} }+ sup.policy.baseDelay = time.Millisecond+ sup.policy.maxDelay = time.Millisecond++ if err := sup.Start(); err != nil {+ t.Fatalf("Start: %v", err)+ }+ t.Cleanup(sup.Stop)++ first := <-handles+ waitSupervisorSignal(t, spawned, "onSpawned did not fire for the first spawn")++ first.crash()+ <-handles+ waitSupervisorSignal(t, spawned, "onSpawned did not fire for the respawn")+}++// onSpawned must not run on the monitor goroutine: that goroutine is the only+// thing that reaches h.Stop() when stopCh closes, so a callback blocking there+// makes Stop wait it out inside the shared teardown budget.+func TestOnSpawnedDoesNotBlockStop(t *testing.T) {+ release := make(chan struct{})+ t.Cleanup(func() { close(release) })+ entered := make(chan struct{})++ sup := newSupervisor("blocking-replay", noRestartPolicy(), func() (supervisedHandle, error) {+ return newFakeHandle(), nil+ })+ var once sync.Once+ sup.onSpawned = func() {+ once.Do(func() { close(entered) })+ <-release+ }+ if err := sup.Start(); err != nil {+ t.Fatalf("Start: %v", err)+ }+ waitSupervisorSignal(t, entered, "onSpawned never ran")++ stopped := make(chan struct{})+ go func() {+ sup.Stop()+ close(stopped)+ }()+ select {+ case <-stopped:+ case <-time.After(2 * time.Second):+ t.Fatal("Stop blocked behind a long-running onSpawned callback")+ }+}++func waitSupervisorSignal(t *testing.T, ch <-chan struct{}, failure string) {+ t.Helper()+ select {+ case <-ch:+ case <-time.After(2 * time.Second):+ t.Fatal(failure)+ }+}+func TestRestartPolicyBackoff(t *testing.T) {p := restartPolicy{baseDelay: time.Second, maxDelay: 16 * time.Second}cases := []struct {@@ -195,11 +267,16 @@ func TestSupervisorStopTearsDownCurrentHandle(t *testing.T) {}}-// TestSupervisorRestartRespawnsWithoutCrash proves the graceful restart used to-// re-point a worker at changed cluster certs on join: the running worker is-// torn down and a fresh one is spawned, but it is NOT counted as a crash (no-// onCrash, so no spurious "subprocess-crashed" error and no budget spent).-func TestSupervisorRestartRespawnsWithoutCrash(t *testing.T) {+// A healthy worker is never torn down and respawned on its own.+//+// This replaces a test of the graceful-restart mechanism, which had no+// production caller and is now deleted: its purpose — re-pointing a worker at+// cluster certs it only read at startup — went away when the proxy gained a+// live trust watch, and the last caller went with the LM Studio port restarts+// that became facade-scoped. What remains worth asserting is the invariant+// that replaced it: nothing short of an actual exit replaces a running worker,+// so a facade-level failure cannot cost every engine its listener.+func TestSupervisorLeavesAHealthyWorkerAlone(t *testing.T) {spawned := make(chan *fakeHandle, 8)spawn := func() (supervisedHandle, error) {h := newFakeHandle()@@ -217,24 +294,23 @@ func TestSupervisorRestartRespawnsWithoutCrash(t *testing.T) {h0 := mustSpawn(t, spawned)- // Request a graceful restart (the broker does this to re-point a worker at- // external state it only reads at startup, e.g. an LM Studio proxy port change).- sup.Restart()-- // The running worker is torn down...select {case <-h0.stoppedC:- case <-time.After(2 * time.Second):- t.Fatal("Restart did not stop the running worker")+ t.Fatal("a healthy worker was stopped without exiting")+ case h := <-spawned:+ t.Fatalf("a second worker was spawned alongside a healthy one: %v", h)+ case a := <-crashes:+ t.Fatalf("a healthy worker surfaced a crash (attempt %d)", a)+ case <-time.After(300 * time.Millisecond):}- // ...and a fresh one is spawned in its place.- _ = mustSpawn(t, spawned)- // A requested restart must not look like a crash.+ // A real exit still recovers, so the above is not simply a dead supervisor.+ h0.crash()+ _ = mustSpawn(t, spawned)select {- case a := <-crashes:- t.Fatalf("Restart surfaced a crash (attempt %d); it must be graceful", a)- case <-time.After(200 * time.Millisecond):+ case <-crashes:+ case <-time.After(2 * time.Second):+ t.Fatal("an actual exit did not surface a crash")}}diff --git a/services/ollama-proxy/.gitignore b/services/ollama-proxy/.gitignoredeleted file mode 100644index 2e63b18..0000000--- a/services/ollama-proxy/.gitignore+++ /dev/null@@ -1,21 +0,0 @@-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0--# Build output-ollama-proxy-ollama-proxy.exe--# Go-*.test-*.out--# IDE-.idea/-.vscode/-*.swp-*.swo-*~--# OS-.DS_Store-Thumbs.dbdiff --git a/services/ollama-proxy/README.md b/services/ollama-proxy/README.mddeleted file mode 100644index 35f959b..0000000--- a/services/ollama-proxy/README.md+++ /dev/null@@ -1,454 +0,0 @@-<!---SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-SPDX-License-Identifier: Apache-2.0--->--# Ollama Proxy--A discovery-aware HTTP reverse proxy for Ollama nodes on the local network. It runs no mDNS browse of its own: its routing targets come from the broker's discovery relay (it sends `discovery:subscribe {services:[ol]}` and replaces its routing overlay from each pushed `discovery:nodes` snapshot) plus user-added manual nodes. It forwards HTTP requests to the selected node, aggregates the model-list routes across candidate nodes, and exposes a bidirectional JSON-RPC 2.0 control channel over stdio (or an IPC socket).--## Build--```bash-go build -o ollama-proxy .-```--## Usage--```-ollama-proxy [flags]-```--### Flags--| Flag | Default | Description |-|------|---------|-------------|-| `--port` | `11435` | HTTP listen port for request forwarding |-| `--alias-address` | *(empty)* | Optional secondary `host:port` for the same routing handler. Repeat the flag to reserve both loopback families for `localhost`. Only literal loopback addresses are accepted; the broker uses this for a safe inherited local `OLLAMA_HOST`, and the aliases are not advertised to peers. |-| `--ignore-persisted-port` | `false` | Use `--port` even when `proxy-port.json` contains a saved port (used by broker-managed startup) |-| `--ipc` | *(empty — use stdio)* | Path to a Unix domain socket or Windows named pipe for IPC |-| `--cluster-dir` | *(empty)* | Cluster trust directory (`node.crt`/`node.key` plus trusted pins). Enables the LAN mTLS inference ingress while this node is a cluster member; empty means no ingress and no peer candidates. |-| `--log-level` | *(`$NVPAIR_LOG_LEVEL`, else `info`)* | Initial log level: `debug`, `info`, `warn`, or `error`. Changeable at runtime with `log/set-level`. |-| `--version` | | Print version and exit |--### HTTP Reverse Proxy--The proxy listens on `--port` (default 11435) and forwards incoming HTTP requests to the currently active Ollama node — except the model-list routes `GET /api/tags` and `GET /v1/models`, which are queried across every candidate node concurrently and merged into one de-duplicated inventory. Point your Ollama client at `http://localhost:11435` and the proxy handles routing. When the broker supplies `--alias-address`, the proxy reserves that loopback-only endpoint before reporting ready and serves it through the same routing and workload-lifecycle handler. A `localhost` alias reserves `127.0.0.1` and `::1` atomically so client resolution cannot bypass the router. An occupied alias is non-fatal: its existing owner is untouched and the proxy reports an actionable warning while the primary listener stays available.--**Cluster ingress.** The listener carries two personalities, demultiplexed by each connection's first byte. Plaintext HTTP is accepted only from loopback; a LAN caller is refused. When `--cluster-dir` shows this node is a cluster member, the same listener also terminates cluster mTLS: a peer whose client certificate matches one of this node's pins is forwarded straight to the local engine reported by `node/set-local-backend`, and is never re-routed onward to another node. Membership and pins are re-derived per request, so joining or leaving a cluster needs no restart.--**Persisted port.** A port chosen at runtime via the `set-port` request (see below) is saved as `proxy-port.json` in the per-user data dir (`%LocalAppData%\Nvidia Corporation\Personal AI Router` on Windows, `~/.config/Nvidia Corporation/Personal AI Router` on Linux) and **restored on startup**, taking precedence over `--port`/the default — so the proxy comes back up where it was last put. `--ignore-persisted-port` deliberately bypasses that restoration for a broker-coordinated start. Delete the file (or `set-port` back to the default) to revert.--**Browser clients (CORS).** The proxy is usable from a web front end. When an engine is available, the proxy forwards an `OPTIONS` preflight so an engine-declared exact origin and credentials policy reaches the browser unchanged. If no engine is available, the engine returns no CORS policy, or a non-loopback caller must be refused before routing, the proxy answers with its own permissive `204` fallback. It also labels every response it generates — including rejections such as the `502` when no node is available and the `403` refusing a non-loopback plaintext caller — with `Access-Control-Allow-Origin: *`. That matters as much as the success path: a response without those headers reaches the browser as a generic "CORS error" with the real status and reason stripped out, so the caller cannot tell what went wrong. A locally answered preflight grants no access, since the request that follows still faces the same gate. The policy is shared with [`lmstudio-proxy`](../lmstudio-proxy/README.md) through `nvpair-shared/cors`, so both proxies answer identically.--A response *forwarded from an engine* is different: if the engine set its own `Access-Control-Allow-Origin` (Ollama with `OLLAMA_ORIGINS`), that header is passed through untouched rather than replaced with the wildcard, so a deliberately narrow engine policy is never silently widened and a credentialed response is not broken. An engine that sends no CORS header has expressed no policy to keep, so the proxy supplies its own — and drops any `Access-Control-Allow-Credentials` that arrived without an origin, because a browser rejects that header alongside a wildcard origin and would discard the response the fallback exists to make readable.--One limit is outside the proxy's control: current Chromium-based browsers gate a request from a public origin to a local or loopback address behind the user's [Local Network Access](https://chromestatus.com/feature/5152728072060928) permission, which replaced the old server-side opt-in header. No header the proxy sends can grant that. A hosted page needs the permission plus a `fetch(url, { targetAddressSpace: 'loopback' })` annotation; a page served from the local machine is unaffected.--Node selection:-- **Eligibility**: Before routing model-bearing inference, the proxy keeps only nodes whose current Ollama inventory advertises the requested model. Ollama's implicit `:latest` tag is normalized. An empty or non-matching inventory is excluded until a later discovery update; if no advertised owner is routable, the proxy returns a local `502`.-- **Auto**: When no eligible node is explicitly selected, the proxy follows `node/set-priority` (see below), then discovered nodes in stable ID order.-- **Priority (scheduler-driven)**: The Job Scheduler ranks the cluster least-loaded-first by pending workload plus smoothed GPU pressure and, via `nvpair-ui-broker`, pushes the ordered node list with those per-node counts to this proxy with `node/set-priority`. Auto routing sends the request to the listed node carrying the least estimated load. See [`nvpair-job-scheduler`](../nvpair-job-scheduler/README.md).-- **Manual**: Use the `node/select` JSON-RPC method to pin traffic to a specific node. A manual pin **overrides the priority list only when that node is eligible** for the requested model.-- **Failover**: If the selected node disappears from the discovery set, the proxy falls back to auto-select and emits a `node/selection-changed` notification. A transport error or retryable status, including a model `404` from an advertised owner with stale inventory, steps to the next eligible owner.--### IPC Transport--By default the proxy communicates over **stdin/stdout** using newline-delimited JSON-RPC 2.0 (one message per line). All diagnostic logging goes to **stderr**.--For environments where stdout may conflict with the host process (e.g. Electron), pass `--ipc` to redirect the JSON-RPC channel to a named socket or pipe. The parent process should create and listen on the endpoint before spawning the proxy.--```bash-# Default: stdin/stdout-ollama-proxy--# Unix domain socket-ollama-proxy --ipc /tmp/ollama-proxy.sock--# Windows named pipe-ollama-proxy --ipc \\.\pipe\ollama-proxy-```--## JSON-RPC 2.0 Protocol--All messages conform to the [JSON-RPC 2.0 specification](https://www.jsonrpc.org/specification). Messages are newline-delimited (one JSON object per `\n`).--### Node Object--Nodes are represented throughout the protocol with this shape:--```json-{- "id": "22222222-2222-2222-2222-222222222222",- "host": "my-workstation",- "port": 11434,- "addresses": ["192.168.1.50"],- "txt": ["uuid=22222222-2222-2222-2222-222222222222", "ol=11434"],- "models": ["llama3.2:latest", "gemma3:4b"],- "ip": "192.168.1.50"-}-```--| Field | Type | Description |-|-------|------|-------------|-| `id` | string | Stable per-host UUID from the discovery record (the ID you supply, for a manual node) |-| `host` | string | Hostname, for display — routing never keys on it |-| `port` | int | Ollama port from the discovery record's `ol` service entry |-| `addresses` | string[] | Addresses to dial. A node fed by the discovery relay always carries exactly one canonical address; several only ever appear on a manual node |-| `txt` | string[] | The discovery record's TXT pairs, carried verbatim |-| `models` | string[] | The node's Ollama model inventory from the discovery snapshot. Model-bearing inference is eligible only when this list advertises the requested model. An omitted or empty list excludes the node from that request until inventory updates; it remains available for non-inference routes and model-list aggregation |-| `ip` | string | The single canonical LAN address to dial or display, resolved from the node's `ip=` TXT if present and otherwise the best-scored advertised IPv4. Stamped onto outbound `node/*` notifications so consumers agree with the address the proxy routes to |-------### Notifications (proxy → client)--Notifications have no `id` field and do not expect a response.--#### `ready`--Sent after startup, before discovery begins, and again after every successful `set-port` rebind — `port` carries the port now bound.--```json-{"jsonrpc":"2.0","method":"ready","params":{"version":"0.1.0","port":11435}}-```--| Param | Type | Description |-|-------|------|-------------|-| `version` | string | Proxy version |-| `port` | int | HTTP listen port |--#### `error`--Sent when a fatal startup condition stops the proxy from serving — currently only a failed bind — immediately before the process exits non-zero.--```json-{"jsonrpc":"2.0","method":"error","params":{"code":"bind-failed","message":"failed to bind port 11435: ...","port":11435}}-```--#### `node/discovered`--A new Ollama node appeared on the network.--```json-{"jsonrpc":"2.0","method":"node/discovered","params":{"id":"22222222-2222-2222-2222-222222222222","host":"my-workstation","port":11434,"addresses":["192.168.1.50"]}}-```--#### `node/updated`--A previously discovered node changed its host, port, or addresses.--```json-{"jsonrpc":"2.0","method":"node/updated","params":{"id":"22222222-2222-2222-2222-222222222222","host":"my-workstation","port":11434,"addresses":["192.168.1.51"]}}-```--#### `node/removed`--A node is no longer present in the discovery set (it left the relay's `ol` nodes, or a manual node was removed).--```json-{"jsonrpc":"2.0","method":"node/removed","params":{"id":"22222222-2222-2222-2222-222222222222","host":"my-workstation","port":11434,"addresses":["192.168.1.50"]}}-```--#### `node/selection-changed`--The active node selection changed (either explicitly via `node/select` or because the selected node was removed).--```json-{"jsonrpc":"2.0","method":"node/selection-changed","params":{"id":"22222222-2222-2222-2222-222222222222"}}-```--An empty `id` means the proxy has reverted to auto-select mode.--#### `proxy/request-started`--A request has been committed to a target and its response body is about to stream. It pairs by `id` with the matching `proxy/request`, so a consumer can keep an in-flight count per node. Model-list aggregation has no single target, so it reports `"target":"cluster"` with no `node_id`; a request rejected before forwarding was never in flight and gets no started event.--```json-{"jsonrpc":"2.0","method":"proxy/request-started","params":{"id":"17","node_id":"22222222-2222-2222-2222-222222222222","method":"POST","path":"/api/chat","target":"192.168.1.50:11434"}}-```--#### `proxy/request`--A proxied request finished, or was rejected before forwarding. `duration_ms` covers the whole request; `ttfb_ms` is the time to the upstream's status line and is omitted where no response header arrived (rejection and transport-error paths). `error` carries normalized error text. This is operational metadata only — request and response bodies are never reported.--```json-{"jsonrpc":"2.0","method":"proxy/request","params":{"id":"17","node_id":"22222222-2222-2222-2222-222222222222","method":"POST","path":"/api/chat","target":"192.168.1.50:11434","status":200,"duration_ms":6120,"ttfb_ms":95}}-```--#### `workload:started` / `workload:completed` / `workload:errored`--One lifecycle transition per forwarded inference request, carrying a single `workloadInfo`. `engine` is always `ollama`; `originatedFrom` is left empty for the broker to stamp, and `scheduledOn` names the node that actually served (re-pointed if failover moved the request). The broker relays these to `nvpair-workload-manager`. The proxy never emits `workload:submitted` — it forwards immediately rather than queueing.--```json-{"jsonrpc":"2.0","method":"workload:started","params":{"workloadInfo":{"id":"17","model":"llama3:latest","engine":"ollama","runId":"3ce8a1740b62df95","state":"running","originatedFrom":"","scheduledOn":"22222222-2222-2222-2222-222222222222","createdAt":1716998400000,"startedAt":1716998400000,"completedAt":null,"error":null,"requesterId":null}}}-```--#### `node/activity`--Raised while a node's engine is streaming a response back through the proxy: every successful write of upstream body bytes reports the node that produced them. The broker relays it to `nvpair-node-scanner`, which treats it as proof of life and cancels that node's eviction — a node saturated by inference cannot answer a liveness probe, but it is demonstrably alive precisely because it is streaming. Coalesced to one report per node per 2s (`nvpair-shared/nodeactivity`), since a generation writes hundreds of chunks and the scanner treats one report as good for a minute. `msSince` is the age of the observation; the broker adds its own relay delay before passing it on.--Only bytes that came from the upstream count. The proxy's own error bodies travel through the same writer and are never reported, because they say nothing about the node.--```json-{"jsonrpc":"2.0","method":"node/activity","params":{"hostUuid":"22222222-2222-2222-2222-222222222222","msSince":0}}-```--#### `errors:report` / `errors:clear`--Entries for the `nvpair-errors` pipeline, keyed by a stable `id` so a report and its clear cannot drift. A node dropping out of the discovery set raises `ollama-proxy:upstream-unreachable:<nodeId>`; its reappearance clears the same id. An `--alias-address` the proxy could not claim raises `ollama-proxy:ollama-host-alias-blocked`. `nodeId` and `timestamp` are left unset for the broker to stamp.--```json-{"jsonrpc":"2.0","method":"errors:report","params":{"id":"ollama-proxy:upstream-unreachable:22222222-2222-2222-2222-222222222222","message":"Upstream node \"my-workstation\" is no longer reachable (dropped from discovery)","severity":"warning","action":"none"}}-```-------### Requests (client → proxy)--Requests carry an `id` and receive a response.--#### `nodes/list`--Returns all currently discovered nodes.--**Request:**-```json-{"jsonrpc":"2.0","id":1,"method":"nodes/list"}-```--**Response:**-```json-{"jsonrpc":"2.0","id":1,"result":{"nodes":[{"id":"22222222-2222-2222-2222-222222222222","host":"my-workstation","port":11434,"addresses":["192.168.1.50"]}]}}-```--#### `node/select`--Pin the proxy to route HTTP traffic to a specific node. Pass an empty `id` to return to auto-select.--**Request:**-```json-{"jsonrpc":"2.0","id":2,"method":"node/select","params":{"id":"22222222-2222-2222-2222-222222222222"}}-```--**Response:**-```json-{"jsonrpc":"2.0","id":2,"result":{"id":"22222222-2222-2222-2222-222222222222"}}-```--**Error** (node not found):-```json-{"jsonrpc":"2.0","id":2,"error":{"code":-32602,"message":"node \"xyz\" not found"}}-```--#### `node/selected`--Query the currently selected node.--**Request:**-```json-{"jsonrpc":"2.0","id":3,"method":"node/selected"}-```--**Response:**-```json-{"jsonrpc":"2.0","id":3,"result":{"id":"22222222-2222-2222-2222-222222222222"}}-```--An empty `id` means auto-select is active.--#### `node/set-priority`--Set the **auto-routing priority order** — an ordered list of node IDs, highest-priority first, optionally with each node's pending-work count and GPU pressure-in `ranks`. Delivered by `nvpair-ui-broker` on behalf of `nvpair-job-scheduler`,-which ranks the cluster least-loaded-first by total pending workload across-engines plus smoothed GPU pressure (see-[`nvpair-job-scheduler`](../nvpair-job-scheduler/README.md)). The snapshot is-stored verbatim and applied at request time. A `nodes`-only payload is valid and-supplies zero pending and GPU-pressure baselines.--**Request:**-```json-{"jsonrpc":"2.0","id":8,"method":"node/set-priority","params":{"nodes":["MY-PC","LAB-DESK-B","GPU-RIG"],"ranks":[{"id":"MY-PC","pending":0,"gpuPressure":0,"rank":0},{"id":"LAB-DESK-B","pending":1,"gpuPressure":1,"rank":1},{"id":"GPU-RIG","pending":3,"gpuPressure":3,"rank":2}]}}-```--**Response:**-```json-{"jsonrpc":"2.0","id":8,"result":{"count":3}}-```--| Field | Type | Description |-|-------|------|-------------|-| `count` | int | Number of node IDs stored (the length of the accepted list) |--Semantics:-- **Capability gate.** A model-bearing inference request first intersects the- discovery snapshot with nodes advertising that Ollama model. Selection,- priority, reservations, and failover operate only on that request-local owner- set. If it is empty, the proxy returns `502` without contacting an engine.-- **Auto ordering.** Within the eligible owner set, the proxy picks the listed- node carrying the least estimated load —- `pending + gpuPressure` from the last snapshot plus the proxy's own- reservations for requests it has already dispatched but whose workload feedback- has not come back yet — breaking ties by position in `nodes`. It increments the- chosen node's reservation before forwarding, so a concurrent burst spreads- instead of repeatedly choosing from the same stale snapshot. That node moves to- the front of this request's failover list and the rest keeps its order, so a- transport error or retryable status (the existing failover trigger) steps to- the next candidate.-- **Snapshot reset.** Each new snapshot replaces the pending and GPU-pressure- baselines and clears the reservations. GPU pressure is clamped to the- scheduler's 0–3 range.-- **Eligible manual pin wins.** An active `node/select` pin takes precedence when- it is in the request's owner set. An ineligible pin is ignored for that request,- so automatic reservations still apply among eligible owners. Clearing the pin- (`node/select` with an empty `id`) activates the most-recently-set list. Setting- a priority list does **not** emit `node/selection-changed` (the manual selection- is unchanged).-- **Unknown IDs are ignored.** IDs not currently in discovery are kept in the- stored list (a node may appear later) but contribute nothing until discovered.-- **Eligible unlisted nodes are a lowest-priority fallback.** An advertised owner- absent from the list stays routable, but only after every listed owner —- ordered among themselves by the default stable ID sort. This ensures an- eligible manually-added node the scheduler never saw is never stranded.-- **Empty list reverts to default.** `{"nodes":[]}` clears the scheduler's- influence and returns the proxy to its default auto ordering (eligible- discovered nodes by stable ID).--The list persists only in memory for the proxy's lifetime; it is not saved across-restarts. On restart the proxy comes back with an empty list, and the broker-re-pushes the last order once the proxy re-announces `ready`.--> **`lmstudio-proxy` parity.** [`lmstudio-proxy`](../lmstudio-proxy/README.md) is a-> deliberate clone of this proxy (identical routing/failover/selection). It-> implements `node/set-priority` with the exact same contract; the only differences-> are engine-specific — it subscribes to the discovery relay for `lm` nodes and-> receives the `lmstudio` engine's list from the broker.--#### `set-port`--Change the HTTP listen port at runtime and persist the choice. The proxy-binds the new port first (so a bind failure leaves the current listener-serving), starts serving on it, then closes the old listener — in-flight-connections on the old port drain naturally. The new port is saved to-`proxy-port.json` and a fresh `ready` notification announces it.--**Request:**-```json-{"jsonrpc":"2.0","id":7,"method":"set-port","params":{"port":11500}}-```--**Response:**-```json-{"jsonrpc":"2.0","id":7,"result":{"version":"0.9.0","port":11500}}-```--**Error** (port in use / out of range):-```json-{"jsonrpc":"2.0","id":7,"error":{"code":-32000,"message":"failed to bind port 11500: ..."}}-```--When supervised by `nvpair-ui-broker`, callers reach this as `proxy:set-port`,-and the broker first steers the port clear of any running engine's port-(engines take precedence) before handing it down — see the broker README.--#### `node/add-manual`--Add a node manually (for networks where mDNS is blocked). If the node ID already exists as a manual node, it is updated.--**Request:**-```json-{"jsonrpc":"2.0","id":5,"method":"node/add-manual","params":{"id":"remote-server","host":"remote-server","port":11434,"addresses":["10.0.1.50"]}}-```--**Response:**-```json-{"jsonrpc":"2.0","id":5,"result":{"added":true}}-```--The proxy emits a `node/discovered` notification (or `node/updated` if the node was already registered). Manual nodes are a separate overlay that discovery snapshots never touch — they persist until explicitly removed.--#### `node/remove-manual`--Remove a previously added manual node.--**Request:**-```json-{"jsonrpc":"2.0","id":6,"method":"node/remove-manual","params":{"id":"remote-server"}}-```--**Response:**-```json-{"jsonrpc":"2.0","id":6,"result":{"removed":true}}-```--The proxy emits a `node/removed` notification and clears the active selection if it pointed to this node.--#### `node/set-local-backend`--Tell the proxy which loopback engine this node's own traffic terminates on. The broker sends it once the local Ollama address and health are known. It is the target the cluster mTLS ingress forwards to, and the substitute used when discovery advertises this node's own proxy endpoint as a candidate. A zero `port` or `"healthy":false` effectively clears it, and the ingress then answers `503`.--**Request:**-```json-{"jsonrpc":"2.0","id":9,"method":"node/set-local-backend","params":{"engine":"ollama","host":"127.0.0.1","port":11434,"healthy":true}}-```--**Response:**-```json-{"jsonrpc":"2.0","id":9,"result":{"ok":true}}-```--#### `log/set-level`--Change the active log level at runtime (`debug`, `info`, `warn`, `error`). Accepted as a request or a notification; as a request it responds with the resolved level and rejects an unknown one with `-32602`.--**Request:**-```json-{"jsonrpc":"2.0","id":10,"method":"log/set-level","params":{"level":"debug"}}-```--**Response:**-```json-{"jsonrpc":"2.0","id":10,"result":{"level":"debug"}}-```--#### `shutdown`--Gracefully shuts down the proxy.--**Request:**-```json-{"jsonrpc":"2.0","id":4,"method":"shutdown"}-```--**Response:**-```json-{"jsonrpc":"2.0","id":4,"result":null}-```-------### Error Codes--Standard JSON-RPC 2.0 error codes apply:--| Code | Meaning |-|------|---------|-| `-32601` | Method not found |-| `-32602` | Invalid params |-| `-32000` | The request was well-formed but could not be carried out — returned by `set-port` when the new port cannot be bound |-------## Shutdown--The proxy shuts down gracefully on any of:--1. **stdin EOF** — parent closes stdin (stdio mode only)-2. **`shutdown` JSON-RPC request** — programmatic shutdown-3. **SIGINT / SIGTERM** — standard OS signals--## Discovery--The proxy does not browse mDNS. On startup it subscribes to the broker's discovery relay for `ol` (Ollama) nodes (`discovery:subscribe {services:[ol]}`). Targets then arrive as `discovery:nodes` notifications carrying the relay's full filtered node set, and each snapshot replaces the routing overlay wholesale — a departed node is simply absent from the next one — while the diff against the previous overlay is what produces the `node/discovered`, `node/updated`, and `node/removed` notifications. User-added manual nodes are merged on top. Nodes are keyed by the discovery record's stable per-host UUID, so routing survives a machine being renamed. The single `_nvpair-node` browse that feeds the relay lives in the `nvpair-node-scanner` daemon (see its README) — this proxy is a pure consumer of the resulting routing set.diff --git a/services/ollama-proxy/activity_test.go b/services/ollama-proxy/activity_test.godeleted file mode 100644index ae94940..0000000--- a/services/ollama-proxy/activity_test.go+++ /dev/null@@ -1,139 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "net/http"- "net/http/httptest"- "strings"- "sync"- "testing"- "time"-)--// Response bytes arriving from a node's engine are the only liveness evidence-// that gets stronger as the node gets busier, which is exactly when discovery's-// probes start timing out and evicting it. These tests pin that the proxy-// actually raises it.--func waitFor(t *testing.T, rec *recRW, frame string) bool {- t.Helper()- deadline := time.Now().Add(2 * time.Second)- for time.Now().Before(deadline) {- if rec.has(frame) {- return true- }- time.Sleep(5 * time.Millisecond)- }- return false-}--// TestStreamedBytesReportNodeActivity is the point of the feature: the node that-// served the response is named to the broker so discovery can keep it.-func TestStreamedBytesReportNodeActivity(t *testing.T) {- upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.WriteHeader(http.StatusOK)- _, _ = w.Write([]byte(`{"done":true}`))- }))- defer upstream.Close()-- rec := &recRW{}- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "serving-node", upstream.URL, "llama"))- p := NewProxy(NewCodec(rec), disc, 11434)-- p.handleHTTP(httptest.NewRecorder(),- httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)))-- if !waitFor(t, rec, `"method":"node/activity"`) {- t.Fatal("no node/activity was reported after the upstream streamed a response")- }- if !waitFor(t, rec, `"hostUuid":"serving-node"`) {- t.Fatal("node/activity did not name the node that served the request")- }-}--// A node that accepted the connection but never wrote a byte has proved nothing-// about being able to do work, so it must not be vouched for. This is the case-// that matters: an accept can succeed on a machine whose user space is wedged.-func TestNoActivityReportedWithoutUpstreamBytes(t *testing.T) {- release := make(chan struct{})- var releaseOnce sync.Once- doRelease := func() { releaseOnce.Do(func() { close(release) }) }-- received := make(chan struct{}, 1)- silent := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- select {- case received <- struct{}{}:- default:- }- <-release- w.WriteHeader(http.StatusOK)- }))- // LIFO: unblock the handler before tearing the server down, or Close hangs- // on the live connection.- defer silent.Close()- defer doRelease()-- rec := &recRW{}- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "silent-node", silent.URL, "llama"))- p := NewProxy(NewCodec(rec), disc, 11434)-- done := make(chan struct{})- go func() {- defer close(done)- p.handleHTTP(httptest.NewRecorder(),- httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)))- }()-- select {- case <-received:- case <-time.After(5 * time.Second):- t.Fatal("upstream never received the forwarded request")- }-- // Give the proxy the same window the positive test uses, so a report would- // have landed by now if one were going to.- time.Sleep(200 * time.Millisecond)- if rec.has(`"method":"node/activity"`) {- t.Fatal("activity was reported for a node that had not sent a single response byte")- }-- doRelease()- <-done-}--// A streaming generation writes hundreds of chunks. Reporting each one would put-// thousands of frames a minute on the broker pipe for no gain, since the scanner-// treats one report as good for a minute.-func TestRepeatedChunksAreCoalescedIntoOneReport(t *testing.T) {- chunks := 40- upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.Header().Set("Content-Type", "application/x-ndjson")- w.WriteHeader(http.StatusOK)- for i := 0; i < chunks; i++ {- _, _ = w.Write([]byte(`{"response":"x"}` + "\n"))- if f, ok := w.(http.Flusher); ok {- f.Flush()- }- }- }))- defer upstream.Close()-- rec := &recRW{}- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "streaming-node", upstream.URL, "llama"))- p := NewProxy(NewCodec(rec), disc, 11434)-- p.handleHTTP(httptest.NewRecorder(),- httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama","stream":true}`)))-- if !waitFor(t, rec, `"method":"node/activity"`) {- t.Fatal("a streamed response reported no activity at all")- }- if got := rec.count(`"method":"node/activity"`); got != 1 {- t.Fatalf("%d activity reports for %d chunks; want 1 within the throttle interval", got, chunks)- }-}diff --git a/services/ollama-proxy/codec.go b/services/ollama-proxy/codec.godeleted file mode 100644index 5d6fd17..0000000--- a/services/ollama-proxy/codec.go+++ /dev/null@@ -1,18 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--// The newline-delimited JSON-RPC 2.0 codec is single-sourced in-// nvpair-shared/jsonrpc. These local aliases keep this package's call sites and-// tests unchanged after removing the copy-pasted per-service codec.--import "nvpair-shared/jsonrpc"--type (- Message = jsonrpc.Message- RPCError = jsonrpc.RPCError- Codec = jsonrpc.Codec-)--var NewCodec = jsonrpc.NewCodecdiff --git a/services/ollama-proxy/failover_test.go b/services/ollama-proxy/failover_test.godeleted file mode 100644index 4878471..0000000--- a/services/ollama-proxy/failover_test.go+++ /dev/null@@ -1,644 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "encoding/json"- "io"- "net"- "net/http"- "net/http/httptest"- "net/url"- "strconv"- "strings"- "testing"- "time"-)--// rwNop is a no-op io.ReadWriter so a Codec can be constructed in tests-// without a real transport: reads hit EOF immediately and writes are-// discarded. handleHTTP only ever writes (notifications), so this is enough.-type rwNop struct{}--func (rwNop) Read([]byte) (int, error) { return 0, io.EOF }-func (rwNop) Write(p []byte) (int, error) { return len(p), nil }--func testProxy(disc *Discovery, port int) *Proxy {- return NewProxy(NewCodec(rwNop{}), disc, port)-}--// nodeFor turns an httptest server URL into a discovery Node pointing at it.-func nodeFor(t *testing.T, id, serverURL string) Node {- t.Helper()- u, err := url.Parse(serverURL)- if err != nil {- t.Fatalf("parse %q: %v", serverURL, err)- }- host, portStr, err := net.SplitHostPort(u.Host)- if err != nil {- t.Fatalf("split %q: %v", u.Host, err)- }- port, err := strconv.Atoi(portStr)- if err != nil {- t.Fatalf("port %q: %v", portStr, err)- }- return Node{ID: id, Addresses: []string{host}, Port: port}-}--func nodeForModel(t *testing.T, id, serverURL, model string) Node {- t.Helper()- node := nodeFor(t, id, serverURL)- node.Models = []string{model}- return node-}--// TestHandlePlain_OptionsPreflight: a CORS preflight is answered locally with-// 204 + permissive headers and never forwarded.-func TestHandlePlain_OptionsPreflight(t *testing.T) {- p := testProxy(NewDiscovery(), 11434)- rec := httptest.NewRecorder()- req := httptest.NewRequest(http.MethodOptions, "/api/chat", nil)- req.RemoteAddr = "127.0.0.1:40000"- req.Header.Set("Access-Control-Request-Headers", "X-Custom-Token")- p.handlePlain(rec, req)-- if rec.Code != http.StatusNoContent {- t.Fatalf("status = %d, want 204", rec.Code)- }- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want *", got)- }- if rec.Header().Get("Access-Control-Allow-Methods") == "" {- t.Errorf("missing Access-Control-Allow-Methods")- }- if got := rec.Header().Get("Access-Control-Expose-Headers"); got != "*" {- t.Errorf("Access-Control-Expose-Headers = %q, want *", got)- }- // The browser's requested headers are echoed so an arbitrary header clears preflight.- if got := rec.Header().Get("Access-Control-Allow-Headers"); got != "X-Custom-Token" {- t.Errorf("Access-Control-Allow-Headers = %q, want echoed X-Custom-Token", got)- }-}--// TestHandlePlain_EngineCredentialedPreflightPreserved: when an engine opts an-// exact origin into credentialed CORS, its preflight policy reaches the browser-// instead of being replaced by the proxy's uncredentialed wildcard fallback.-func TestHandlePlain_EngineCredentialedPreflightPreserved(t *testing.T) {- preflightSeen := make(chan struct{}, 1)- engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- if r.Method != http.MethodOptions {- t.Errorf("engine method = %s, want OPTIONS", r.Method)- }- preflightSeen <- struct{}{}- w.Header().Set("Access-Control-Allow-Origin", "https://app.example")- w.Header().Set("Access-Control-Allow-Credentials", "true")- w.Header().Set("Access-Control-Allow-Methods", "POST")- w.Header().Set("Access-Control-Allow-Headers", "Content-Type")- w.WriteHeader(http.StatusNoContent)- }))- defer engine.Close()-- disc := NewDiscovery()- disc.AddManual(nodeFor(t, "engine", engine.URL))- p := testProxy(disc, 11434)- req := httptest.NewRequest(http.MethodOptions, "/api/chat", nil)- req.RemoteAddr = "127.0.0.1:40000"- req.Header.Set("Origin", "https://app.example")- req.Header.Set("Access-Control-Request-Method", http.MethodPost)- req.Header.Set("Access-Control-Request-Headers", "Content-Type")- rec := httptest.NewRecorder()-- p.handlePlain(rec, req)-- select {- case <-preflightSeen:- default:- t.Fatal("engine did not receive the credentialed preflight")- }- if rec.Code != http.StatusNoContent {- t.Errorf("status = %d, want %d", rec.Code, http.StatusNoContent)- }- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "https://app.example" {- t.Errorf("Access-Control-Allow-Origin = %q, want the engine's exact origin", got)- }- if got := rec.Header().Get("Access-Control-Allow-Credentials"); got != "true" {- t.Errorf("Access-Control-Allow-Credentials = %q, want the engine's true", got)- }-}--// TestHandleHTTP_EngineCORSPolicyPreserved: an engine that declares its own-// origin policy keeps it. Replacing it with the proxy's wildcard would widen-// what the user configured, and would break a credentialed response outright.-func TestHandleHTTP_EngineCORSPolicyPreserved(t *testing.T) {- engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- w.Header().Set("Access-Control-Allow-Origin", "https://app.example")- w.Header().Set("Access-Control-Allow-Credentials", "true")- w.WriteHeader(http.StatusOK)- io.WriteString(w, `{"done":true}`)- }))- defer engine.Close()-- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "engine", engine.URL, "llama"))- p := testProxy(disc, 11434)-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)))-- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "https://app.example" {- t.Errorf("Access-Control-Allow-Origin = %q, want the engine's own origin", got)- }- if got := rec.Header().Get("Access-Control-Allow-Credentials"); got != "true" {- t.Errorf("Access-Control-Allow-Credentials = %q, want the engine's true", got)- }-}--// TestHandleHTTP_EngineCredentialsWithoutOriginDropped: an engine (or an-// intermediary in front of it) that sends Allow-Credentials but no origin has-// declared no policy to keep, so the proxy supplies its own. The wildcard it-// writes is invalid next to Allow-Credentials: true, and a browser rejects that-// pair, so the inherited header must not survive the forward.-func TestHandleHTTP_EngineCredentialsWithoutOriginDropped(t *testing.T) {- engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- w.Header().Set("Access-Control-Allow-Credentials", "true")- w.WriteHeader(http.StatusOK)- io.WriteString(w, `{"done":true}`)- }))- defer engine.Close()-- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "engine", engine.URL, "llama"))- p := testProxy(disc, 11434)-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)))-- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want the proxy's wildcard", got)- }- if got := rec.Header().Get("Access-Control-Allow-Credentials"); got != "" {- t.Errorf("Access-Control-Allow-Credentials = %q, want cleared alongside the wildcard origin", got)- }-}--// TestHandleHTTP_HappyPathSingleNode: the common case — one healthy node-// answers directly, body forwarded, CORS present on the success response.-func TestHandleHTTP_HappyPathSingleNode(t *testing.T) {- var gotBody string- good := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- b, _ := io.ReadAll(r.Body)- gotBody = string(b)- w.WriteHeader(http.StatusOK)- io.WriteString(w, `{"done":true}`)- }))- defer good.Close()-- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "good", good.URL, "llama"))- p := testProxy(disc, 11434)-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)))-- if rec.Code != http.StatusOK {- t.Fatalf("status = %d, want 200", rec.Code)- }- if gotBody != `{"model":"llama"}` {- t.Errorf("node got body %q, want the original request body", gotBody)- }- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want * on success", got)- }-}--// TestHandleHTTP_NoRetryOn400: a client error (400) is returned as-is and not-// failed over — retrying elsewhere would return the same error and mask it.-func TestHandleHTTP_NoRetryOn400(t *testing.T) {- hits := 0- bad := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- hits++- w.WriteHeader(http.StatusBadRequest)- io.WriteString(w, `{"error":"bad request"}`)- }))- defer bad.Close()- other := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.WriteHeader(http.StatusOK)- }))- defer other.Close()-- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "bad", bad.URL, "llama"))- disc.AddManual(nodeForModel(t, "other", other.URL, "llama"))- p := testProxy(disc, 11434)- p.SetSelected("bad")-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)))-- if rec.Code != http.StatusBadRequest {- t.Fatalf("status = %d, want 400 (client errors must not fail over)", rec.Code)- }- if hits != 1 {- t.Errorf("bad node hit %d times, want exactly 1 (no retry on 400)", hits)- }-}--// TestHandleHTTP_RejectionHasCORS: even the no-node rejection carries CORS so a-// browser sees the real 502 instead of an opaque CORS error.-func TestHandleHTTP_RejectionHasCORS(t *testing.T) {- p := testProxy(NewDiscovery(), 11434)- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"x"}`)))-- if rec.Code != http.StatusBadGateway {- t.Fatalf("status = %d, want 502", rec.Code)- }- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want * on rejection", got)- }-}--// TestHandleHTTP_FailoverOn503: a busy first node (503) is skipped and the-// request is filled by the next node, with the original body replayed.-func TestHandleHTTP_FailoverOn503(t *testing.T) {- busy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.WriteHeader(http.StatusServiceUnavailable)- io.WriteString(w, `{"error":"loading model"}`)- }))- defer busy.Close()-- var gotBody string- good := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- b, _ := io.ReadAll(r.Body)- gotBody = string(b)- w.WriteHeader(http.StatusOK)- io.WriteString(w, `{"done":true}`)- }))- defer good.Close()-- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "busy", busy.URL, "llama"))- disc.AddManual(nodeForModel(t, "good", good.URL, "llama"))- p := testProxy(disc, 11434)- p.SetSelected("busy") // deterministic: busy is tried first-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)))-- if rec.Code != http.StatusOK {- t.Fatalf("status = %d, want 200 (should have failed over past the 503)", rec.Code)- }- if gotBody != `{"model":"llama"}` {- t.Errorf("failover node got body %q, want the original request body", gotBody)- }- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want * on proxied success", got)- }-}--// TestHandleHTTP_AllNodesDownReturnsError: when every candidate fails at the-// transport, the client gets one clean 502 (not a hang), still with CORS.-func TestHandleHTTP_AllNodesDownReturnsError(t *testing.T) {- // Two servers we immediately close so dials fail.- a := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))- b := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))- na := nodeForModel(t, "a", a.URL, "llama")- nb := nodeForModel(t, "b", b.URL, "llama")- a.Close()- b.Close()-- disc := NewDiscovery()- disc.AddManual(na)- disc.AddManual(nb)- p := testProxy(disc, 11434)-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)))-- if rec.Code != http.StatusBadGateway {- t.Fatalf("status = %d, want 502 when all nodes are down", rec.Code)- }- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want * on exhausted error", got)- }-}--// TestHandleHTTP_404FailoverInferenceOnly: a 404 (model-not-found) on an-// inference call fails over to the next advertised owner, but a 404 on a-// non-inference path is returned as-is.-func TestHandleHTTP_404FailoverInferenceOnly(t *testing.T) {- missing := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.WriteHeader(http.StatusNotFound)- io.WriteString(w, `{"error":"model not found"}`)- }))- defer missing.Close()- has := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- w.WriteHeader(http.StatusOK)- io.WriteString(w, `{"done":true}`)- }))- defer has.Close()-- newProxy := func() *Proxy {- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "missing", missing.URL, "llama"))- disc.AddManual(nodeForModel(t, "has", has.URL, "llama"))- p := testProxy(disc, 11434)- p.SetSelected("missing")- return p- }-- // Inference POST: 404 on first → fail over → 200.- rec := httptest.NewRecorder()- newProxy().handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)))- if rec.Code != http.StatusOK {- t.Fatalf("inference 404: status = %d, want 200 (should fail over)", rec.Code)- }-- // An ordinary non-inference GET still returns the first node's 404.- rec = httptest.NewRecorder()- newProxy().handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/version", nil))- if rec.Code != http.StatusNotFound {- t.Fatalf("non-inference 404: status = %d, want 404 (must NOT fail over)", rec.Code)- }-}--func TestHandleHTTP_AggregatesModelList(t *testing.T) {- entered := make(chan struct{}, 2)- release := make(chan struct{})- server := func(body string) *httptest.Server {- return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- if r.Method != http.MethodGet || r.URL.Path != "/api/tags" {- t.Errorf("upstream request = %s %s, want GET /api/tags", r.Method, r.URL.Path)- }- if r.Header.Get("Authorization") != "" || r.Header.Get("Cookie") != "" {- t.Errorf("client credentials leaked to fan-out target")- }- entered <- struct{}{}- <-release- _, _ = io.WriteString(w, body)- }))- }- a := server(`{"models":[{"name":"a","model":"a","digest":"a-only"},{"name":"shared","digest":"first"}]}`)- defer a.Close()- b := server(`{"models":[{"name":"shared:latest","model":"shared:latest","digest":"second"},{"name":"c","model":"c","digest":"c-only"}]}`)- defer b.Close()- malformed := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- _, _ = io.WriteString(w, `{"models":null}`)- }))- defer malformed.Close()- down := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))- downNode := nodeFor(t, "down", down.URL)- down.Close()-- disc := NewDiscovery()- disc.AddManual(nodeFor(t, "a", a.URL))- disc.AddManual(nodeFor(t, "b", b.URL))- disc.AddManual(downNode)- disc.AddManual(nodeFor(t, "malformed", malformed.URL))- events := &recRW{}- p := NewProxy(NewCodec(events), disc, 11434)- p.SetSelected("a")- rec := httptest.NewRecorder()- req := httptest.NewRequest(http.MethodGet, "/api/tags", nil)- req.Header.Set("Authorization", "Bearer client-secret")- req.Header.Set("Cookie", "session=client-secret")- done := make(chan struct{})- go func() {- p.handleHTTP(rec, req)- close(done)- }()-- for range 2 {- select {- case <-entered:- case <-time.After(5 * time.Second):- close(release)- t.Fatal("model-list requests were not issued concurrently")- }- }- close(release)- select {- case <-done:- case <-time.After(5 * time.Second):- t.Fatal("aggregate request did not finish")- }-- if rec.Code != http.StatusOK {- t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())- }- var got struct {- Models []struct {- Name string `json:"name"`- Model string `json:"model"`- Digest string `json:"digest"`- } `json:"models"`- }- if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {- t.Fatal(err)- }- if len(got.Models) != 3 || got.Models[0].Model != "a" || got.Models[1].Name != "shared" || got.Models[1].Model != "" || got.Models[2].Model != "c" {- t.Fatalf("models = %+v, want a, shared, c", got.Models)- }- if got.Models[1].Digest != "first" {- t.Errorf("duplicate metadata = %q, want deterministic first candidate", got.Models[1].Digest)- }- if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {- t.Errorf("Access-Control-Allow-Origin = %q, want *", got)- }- if !events.has(`"method":"proxy/request-started"`) || !events.has(`"method":"proxy/request"`) || !events.has(`"target":"cluster"`) {- t.Errorf("aggregate telemetry missing paired cluster events: %s", events.b)- }-}--func TestHandleHTTP_AggregatesOpenAIModelList(t *testing.T) {- a := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- if r.Method != http.MethodGet || r.URL.Path != "/v1/models" {- t.Errorf("upstream request = %s %s, want GET /v1/models", r.Method, r.URL.Path)- }- _, _ = io.WriteString(w, `{"object":"list","data":[{"id":"a","owned_by":"a"},{"id":"shared","owned_by":"first"}]}`)- }))- defer a.Close()- b := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- if r.Method != http.MethodGet || r.URL.Path != "/v1/models" {- t.Errorf("upstream request = %s %s, want GET /v1/models", r.Method, r.URL.Path)- }- _, _ = io.WriteString(w, `{"object":"list","data":[{"id":"shared","owned_by":"second"},{"id":"c","owned_by":"b"}]}`)- }))- defer b.Close()-- disc := NewDiscovery()- disc.AddManual(nodeFor(t, "a", a.URL))- disc.AddManual(nodeFor(t, "b", b.URL))- p := testProxy(disc, 11434)- p.SetSelected("a")- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/v1/models", nil))-- var got struct {- Object string `json:"object"`- Data []struct {- ID string `json:"id"`- OwnedBy string `json:"owned_by"`- } `json:"data"`- }- if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {- t.Fatal(err)- }- if rec.Code != http.StatusOK || got.Object != "list" || len(got.Data) != 3 {- t.Fatalf("response = %d %+v", rec.Code, got)- }- if got.Data[0].ID != "a" || got.Data[1].ID != "shared" || got.Data[1].OwnedBy != "first" || got.Data[2].ID != "c" {- t.Fatalf("models = %+v, want a, shared(first), c", got.Data)- }-}--func TestHandleHTTP_ModelListEmptyAndUnavailable(t *testing.T) {- empty := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- _, _ = io.WriteString(w, `{"models":[]}`)- }))- emptyNode := nodeFor(t, "empty", empty.URL)- empty.Close()-- disc := NewDiscovery()- disc.AddManual(emptyNode)- rec := httptest.NewRecorder()- testProxy(disc, 11434).handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/tags", nil))- if rec.Code != http.StatusServiceUnavailable {- t.Fatalf("unavailable status = %d, want 503", rec.Code)- }-- empty = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- _, _ = io.WriteString(w, `{"models":[]}`)- }))- defer empty.Close()- disc = NewDiscovery()- disc.AddManual(nodeFor(t, "empty", empty.URL))- rec = httptest.NewRecorder()- testProxy(disc, 11434).handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/tags", nil))- if rec.Code != http.StatusOK || rec.Body.String() != `{"models":[]}` {- t.Fatalf("empty response = %d %s, want 200 native empty list", rec.Code, rec.Body.String())- }-}--// TestHandleHTTP_StrictModelRouting proves capability is a gate before-// selection and priority. Ollama's implicit :latest alias is normalized, while-// unknown and known-missing nodes are excluded from inference but remain-// available for non-inference routes.-func TestHandleHTTP_StrictModelRouting(t *testing.T) {- missHits, unknownHits, matchHits := 0, 0, 0- miss := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- missHits++- w.WriteHeader(http.StatusOK)- }))- defer miss.Close()- unknown := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- unknownHits++- w.WriteHeader(http.StatusOK)- }))- defer unknown.Close()- match := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- matchHits++- body, _ := io.ReadAll(r.Body)- if string(body) != `{"model":"llama"}` {- t.Errorf("matching node got body %q", body)- }- w.WriteHeader(http.StatusOK)- }))- defer match.Close()-- disc := NewDiscovery()- missNode := nodeFor(t, "selected-miss", miss.URL)- missNode.Models = []string{"mistral"}- unknownNode := nodeFor(t, "a-unknown", unknown.URL)- matchNode := nodeFor(t, "z-match", match.URL)- matchNode.Models = []string{"llama:latest"}- disc.AddManual(missNode)- disc.AddManual(unknownNode)- disc.AddManual(matchNode)- p := testProxy(disc, 11434)- p.SetSelected("selected-miss")- p.SetPriority([]string{"a-unknown", "selected-miss", "z-match"})- candidates := p.resolveCandidates("llama")- if len(candidates) != 1 || candidates[0].id != "z-match" {- t.Fatalf("model candidates = %v, want only z-match", candidates)- }-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)))-- if rec.Code != http.StatusOK {- t.Fatalf("status = %d, want 200", rec.Code)- }- if missHits != 0 || unknownHits != 0 || matchHits != 1 {- t.Fatalf("hits miss=%d unknown=%d match=%d, want 0/0/1", missHits, unknownHits, matchHits)- }-- // Capability filtering applies only to model-bearing inference.- rec = httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/version", nil))- if rec.Code != http.StatusOK {- t.Fatalf("non-inference status = %d, want 200", rec.Code)- }- if missHits != 1 || unknownHits != 0 || matchHits != 1 {- t.Fatalf("non-inference hits miss=%d unknown=%d match=%d, want 1/0/1", missHits, unknownHits, matchHits)- }-}--func TestHandleHTTP_NoAdvertisedModelRejectsLocally(t *testing.T) {- hits := 0- upstream := func() *httptest.Server {- return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {- hits++- w.WriteHeader(http.StatusOK)- }))- }- missing := upstream()- defer missing.Close()- unknown := upstream()- defer unknown.Close()-- disc := NewDiscovery()- disc.AddManual(nodeForModel(t, "missing", missing.URL, "mistral"))- disc.AddManual(nodeFor(t, "unknown", unknown.URL))- events := &recRW{}- p := NewProxy(NewCodec(events), disc, 11434)- p.SetSelected("missing")-- rec := httptest.NewRecorder()- p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/api/chat", strings.NewReader(`{"model":"llama"}`)))-- if rec.Code != http.StatusBadGateway ||- !strings.Contains(rec.Body.String(), "no available node advertises the requested model") {- t.Fatalf("response = %d %s, want actionable local 502", rec.Code, rec.Body.String())- }- if hits != 0 {- t.Fatalf("ineligible upstreams received %d requests, want 0", hits)- }- if !events.has("no node advertises requested model") {- t.Fatalf("missing rejected request event: %s", events.b)- }-}--// TestResolveCandidates_SelfGuard: a node resolving to the proxy's own-// listen address is dropped so we never self-forward.-func TestResolveCandidates_SelfGuard(t *testing.T) {- disc := NewDiscovery()- disc.AddManual(Node{ID: "self", Addresses: []string{"127.0.0.1"}, Port: 11434})- disc.AddManual(Node{ID: "real", Addresses: []string{"192.0.2.10"}, Port: 11434})- p := testProxy(disc, 11434)-- cands := p.resolveCandidates("")- var haveReal bool- for _, c := range cands {- if c.id == "self" {- t.Errorf("self-target node must be excluded, got candidate %+v", c)- }- if c.id == "real" {- haveReal = true- }- }- if !haveReal {- t.Errorf("expected the real node to survive the self-guard, candidates = %+v", cands)- }-}diff --git a/services/ollama-proxy/go.sum b/services/ollama-proxy/go.sumdeleted file mode 100644index fe333ff..0000000--- a/services/ollama-proxy/go.sum+++ /dev/null@@ -1,35 +0,0 @@-github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=-github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=-github.com/cenkalti/backoff v2.2.1+incompatible h1:tNowT99t7UNflLxfYYSlKYsBpXdEet03Pg2g16Swow4=-github.com/cenkalti/backoff v2.2.1+incompatible/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM=-github.com/grandcat/zeroconf v1.0.0 h1:uHhahLBKqwWBV6WZUDAT71044vwOTL+McW0mBJvo6kE=-github.com/grandcat/zeroconf v1.0.0/go.mod h1:lTKmG1zh86XyCoUeIHSA4FJMBwCJiQmGfcP2PdzytEs=-github.com/miekg/dns v1.1.27/go.mod h1:KNUDUusw/aVsxyTYZM1oqvCicbwhgbNgztCETuNZ7xM=-github.com/miekg/dns v1.1.55 h1:GoQ4hpsj0nFLYe+bWiCToyrBEJXkQfOOIvFGFy0lEgo=-github.com/miekg/dns v1.1.55/go.mod h1:uInx36IzPl7FYnDcMeVWxj9byh7DutNykX4G9Sj60FY=-github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=-github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=-golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=-golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=-golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=-golang.org/x/mod v0.17.0 h1:zY54UmvipHiNd+pm+m0x9KhZ9hl1/7QNMyxXbc6ICqA=-golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=-golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=-golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=-golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=-golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=-golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=-golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=-golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=-golang.org/x/sync v0.10.0 h1:3NQrjDixjgGwUOCaF8w2+VYHv0Ve/vGYSbdkTa98gmQ=-golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=-golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=-golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=-golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=-golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=-golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=-golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=-golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=-golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d h1:vU5i/LfpvrRCpgM/VPfJLg5KjxD3E+hfT1SH+d9zLwg=-golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=-golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=diff --git a/services/ollama-proxy/ipc.go b/services/ollama-proxy/ipc.godeleted file mode 100644index 63c92ff..0000000--- a/services/ollama-proxy/ipc.go+++ /dev/null@@ -1,12 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--// The IPC transport (Unix socket / Windows named pipe) is single-sourced in-// nvpair-shared/ipc; the platform split lives there. dialIPC aliases it so call-// sites are unchanged.--import "nvpair-shared/ipc"--var dialIPC = ipc.Dialdiff --git a/services/ollama-proxy/main.go b/services/ollama-proxy/main.godeleted file mode 100644index afd147d..0000000--- a/services/ollama-proxy/main.go+++ /dev/null@@ -1,112 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "context"- "flag"- "fmt"- "io"- "log"- "log/slog"- "os"- "os/signal"- "strings"- "syscall"-- "nvpair-shared/applog"- "nvpair-shared/clustertrust"-)--type aliasAddressFlags []string--func (a *aliasAddressFlags) String() string { return strings.Join(*a, ",") }-func (a *aliasAddressFlags) Set(value string) error {- *a = append(*a, value)- return nil-}--func main() {- port := flag.Int("port", 11435, "HTTP listen port")- var aliasAddresses aliasAddressFlags- flag.Var(&aliasAddresses, "alias-address", "optional loopback-only HTTP alias address (repeat for another loopback family)")- ignorePersistedPort := flag.Bool("ignore-persisted-port", false, "use --port even when a persisted port exists")- ipcPath := flag.String("ipc", "", "IPC endpoint: Unix domain socket path or Windows named pipe (default: stdin/stdout)")- clusterDir := flag.String("cluster-dir", "", "cluster trust directory (node.crt/key + trusted pins); enables the LAN mTLS inference ingress when this node is clustered")- showVersion := flag.Bool("version", false, "print version and exit")- resolveLevel := applog.RegisterFlag(nil, slog.LevelInfo)- flag.Parse()-- if *showVersion {- fmt.Println(Version)- os.Exit(0)- }-- applog.Init("ollama-proxy", resolveLevel())-- var transport io.ReadWriteCloser- if *ipcPath != "" {- conn, err := dialIPC(*ipcPath)- if err != nil {- log.Fatalf("failed to connect to IPC endpoint %q: %v", *ipcPath, err)- }- transport = conn- log.Printf("using IPC transport: %s", *ipcPath)- } else {- transport = newStdioTransport()- log.Print("using stdio transport")- }- defer transport.Close()-- ctx, cancel := context.WithCancel(context.Background())- defer cancel()-- sigCh := make(chan os.Signal, 1)- signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)- go func() {- select {- case sig := <-sigCh:- log.Printf("received %s, shutting down", sig)- cancel()- case <-ctx.Done():- }- }()-- // Restore a previously chosen port (set via set-port) over the- // --port/default, so the proxy comes back up where the user last put it.- persisted, hasPersisted := loadPersistedPort()- effectivePort := chooseStartupPort(*port, *ignorePersistedPort, persisted, hasPersisted)- if hasPersisted && !*ignorePersistedPort {- log.Printf("restored persisted proxy port %d", persisted)- }-- codec := NewCodec(transport)- disc := NewDiscovery()- proxy := NewProxy(codec, disc, effectivePort)- for _, aliasAddress := range aliasAddresses {- if err := proxy.setLoopbackAlias(aliasAddress); err != nil {- log.Fatalf("invalid alias address: %v", err)- }- }- // Open a live view of this node's cluster mTLS trust fabric. While unclustered- // the proxy serves only the loopback plaintext personality; once this node is- // a member the same listener also serves the pin-gated LAN mTLS ingress, and- // peers become routable candidates. The proxy needs no restart to notice- // either transition — it re-derives membership per request and on a watch.- //- // Membership is gated on an active admission or a pin, never on keypair- // presence: a left/removed node keeps its keypair by design, and would- // otherwise keep logging cluster_ingress with no cluster peers to serve.- proxy.mesh = clustertrust.Open(*clusterDir)-- go proxy.mesh.Watch(ctx, func(clustered bool) {- slog.Info("cluster inference ingress switched personality", "cluster_ingress", clustered)- proxy.dropUnpinnedPeerTransports()- })-- if err := proxy.Run(ctx); err != nil && ctx.Err() == nil {- log.Fatalf("proxy error: %v", err)- }- log.Print("shutdown complete")-}diff --git a/services/ollama-proxy/portstore.go b/services/ollama-proxy/portstore.godeleted file mode 100644index 5774a1c..0000000--- a/services/ollama-proxy/portstore.go+++ /dev/null@@ -1,76 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "encoding/json"- "os"- "path/filepath"-- "nvpair-shared/appdir"-)--const proxyPortFile = "proxy-port.json"--type persistedPort struct {- Port int `json:"port"`-}--func chooseStartupPort(flagPort int, ignorePersisted bool, persisted int, hasPersisted bool) int {- if hasPersisted && !ignorePersisted {- return persisted- }- return flagPort-}--func proxyPortPath() (string, error) {- return appdir.Path(proxyPortFile)-}--// loadPersistedPort returns the previously chosen proxy port, if a valid one-// was saved. Any error (no file, bad JSON, out-of-range) reports "none" so-// startup falls back to the --port flag / default.-func loadPersistedPort() (int, bool) {- path, err := proxyPortPath()- if err != nil {- return 0, false- }- data, err := os.ReadFile(path)- if err != nil {- return 0, false- }- var pp persistedPort- if err := json.Unmarshal(data, &pp); err != nil {- return 0, false- }- if pp.Port < 1 || pp.Port > 65535 {- return 0, false- }- return pp.Port, true-}--// savePersistedPort atomically writes the chosen port (tmp + rename) so a-// crash mid-write can't leave a truncated file behind.-func savePersistedPort(port int) error {- path, err := proxyPortPath()- if err != nil {- return err- }- if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {- return err- }- data, err := json.Marshal(persistedPort{Port: port})- if err != nil {- return err- }- tmp := path + ".tmp"- if err := os.WriteFile(tmp, data, 0o644); err != nil {- return err- }- if err := os.Rename(tmp, path); err != nil {- _ = os.Remove(tmp)- return err- }- return nil-}diff --git a/services/ollama-proxy/portstore_test.go b/services/ollama-proxy/portstore_test.godeleted file mode 100644index 49c359d..0000000--- a/services/ollama-proxy/portstore_test.go+++ /dev/null@@ -1,140 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "bytes"- "context"- "fmt"- "net"- "os"- "strings"- "testing"- "time"-)--func TestChooseStartupPort(t *testing.T) {- for _, tc := range []struct {- name string- flagPort int- ignore bool- persisted int- has bool- want int- }{- {"restores persisted by default", 11434, false, 11435, true, 11435},- {"managed startup overrides persisted facade", 11435, true, 11434, true, 11435},- {"flag used without persisted value", 11434, false, 0, false, 11434},- } {- t.Run(tc.name, func(t *testing.T) {- if got := chooseStartupPort(tc.flagPort, tc.ignore, tc.persisted, tc.has); got != tc.want {- t.Fatalf("chooseStartupPort() = %d, want %d", got, tc.want)- }- })- }-}--// redirectConfigDir points os.UserConfigDir() at a temp dir for the test, so-// proxy-port.json reads/writes don't touch the real per-user config. Sets all-// three env vars os.UserConfigDir() consults across platforms: XDG_CONFIG_HOME-// on Linux, $HOME/Library on macOS, and APPDATA on Windows. Missing APPDATA-// meant the Windows-first-class path read/wrote the real %AppData% file —-// clobbering the user's saved port and making the test fail on repeat runs.-func redirectConfigDir(t *testing.T) {- t.Helper()- dir := t.TempDir()- t.Setenv("XDG_CONFIG_HOME", dir)- t.Setenv("HOME", dir)- t.Setenv("APPDATA", dir)- t.Setenv("LOCALAPPDATA", dir)-}--func freeTCPPort(t *testing.T) int {- t.Helper()- l, err := net.Listen("tcp", "127.0.0.1:0")- if err != nil {- t.Fatalf("reserve free port: %v", err)- }- defer l.Close()- return l.Addr().(*net.TCPAddr).Port-}--func TestPersistedPortRoundTrip(t *testing.T) {- redirectConfigDir(t)-- if _, ok := loadPersistedPort(); ok {- t.Fatal("expected no persisted port before any save")- }- if err := savePersistedPort(11500); err != nil {- t.Fatalf("savePersistedPort: %v", err)- }- if p, ok := loadPersistedPort(); !ok || p != 11500 {- t.Errorf("round-trip: got %d ok=%v, want 11500", p, ok)- }-- // An out-of-range stored value is treated as "none" so startup falls- // back to the flag/default rather than trying to bind port 0.- path, err := proxyPortPath()- if err != nil {- t.Fatalf("proxyPortPath: %v", err)- }- if err := os.WriteFile(path, []byte(`{"port":0}`), 0o644); err != nil {- t.Fatal(err)- }- if _, ok := loadPersistedPort(); ok {- t.Error("port 0 should be treated as none")- }-}--// TestSetPortRebinds drives a live rebind: the proxy starts serving on one-// port, set-port moves it to another, and afterward the new port accepts-// connections, the old one doesn't, the choice is persisted, and a fresh-// ready notification carries the new port.-func TestSetPortRebinds(t *testing.T) {- redirectConfigDir(t)-- buf := &bytes.Buffer{}- codec := NewCodec(buf)- disc := NewDiscovery()-- portA := freeTCPPort(t)- proxy := NewProxy(codec, disc, portA)-- lnA, err := net.Listen("tcp", fmt.Sprintf(":%d", portA))- if err != nil {- t.Fatalf("listen on port A: %v", err)- }- ctx, cancel := context.WithCancel(context.Background())- defer cancel()- proxy.serveHTTP(ctx, lnA)- defer proxy.shutdown(context.Background())-- portB := freeTCPPort(t)- if err := proxy.setPort(portB); err != nil {- t.Fatalf("setPort: %v", err)- }-- // New port is now serving.- conn, err := net.DialTimeout("tcp", fmt.Sprintf("127.0.0.1:%d", portB), 2*time.Second)- if err != nil {- t.Fatalf("new port %d not listening after rebind: %v", portB, err)- }- conn.Close()-- // Old port stopped accepting.- if c, err := net.DialTimeout("tcp", fmt.Sprintf("127.0.0.1:%d", portA), 500*time.Millisecond); err == nil {- c.Close()- t.Errorf("old port %d should be closed after rebind", portA)- }-- // Persisted for next startup.- if p, ok := loadPersistedPort(); !ok || p != portB {- t.Errorf("persisted port: got %d ok=%v, want %d", p, ok, portB)- }-- // A fresh ready notification announced the new port.- if !strings.Contains(buf.String(), fmt.Sprintf("\"port\":%d", portB)) {- t.Errorf("expected ready notification carrying port %d, got %q", portB, buf.String())- }-}diff --git a/services/ollama-proxy/reservation_test.go b/services/ollama-proxy/reservation_test.godeleted file mode 100644index 97049eb..0000000--- a/services/ollama-proxy/reservation_test.go+++ /dev/null@@ -1,235 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "sync"- "testing"-- "nvpair-shared/schedulerwire"-)--func reservationCandidates(ids ...string) []candidate {- out := make([]candidate, 0, len(ids))- for _, id := range ids {- out = append(out, candidate{id: id})- }- return out-}--func reservedID(p *Proxy, candidates []candidate) string {- candidates = append([]candidate(nil), candidates...)- return p.reserveCandidate(candidates)[0].id-}--func TestReserveCandidate_ConcurrentEqualLoadHasAtMostOneSkew(t *testing.T) {- p := prProxy(t)- ids := []string{"a", "b", "c", "d"}- ranks := make([]schedulerwire.NodeRank, 0, len(ids))- for i, id := range ids {- ranks = append(ranks, schedulerwire.NodeRank{ID: id, Rank: i})- }- p.SetPrioritySnapshot(schedulerwire.Priority{Nodes: ids, Ranks: ranks})- candidates := reservationCandidates(ids...)-- const requests = 100- chosen := make(chan string, requests)- var wg sync.WaitGroup- for range requests {- wg.Add(1)- go func() {- defer wg.Done()- chosen <- reservedID(p, candidates)- }()- }- wg.Wait()- close(chosen)-- counts := make(map[string]int, len(ids))- for id := range chosen {- counts[id]++- }- min, max := requests, 0- for _, id := range ids {- if counts[id] < min {- min = counts[id]- }- if counts[id] > max {- max = counts[id]- }- }- if max-min > 1 {- t.Fatalf("100 equal-load reservations are imbalanced: %v", counts)- }-}--func TestReserveCandidate_ConvergesUnequalPendingDepths(t *testing.T) {- p := prProxy(t)- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"a", "b", "c"},- Ranks: []schedulerwire.NodeRank{- {ID: "a", Pending: 0, Rank: 0},- {ID: "b", Pending: 2, Rank: 1},- {ID: "c", Pending: 4, Rank: 2},- },- })- candidates := reservationCandidates("a", "b", "c")- assigned := map[string]int{}- for range 6 {- assigned[reservedID(p, candidates)]++- }-- total := map[string]int{- "a": assigned["a"],- "b": 2 + assigned["b"],- "c": 4 + assigned["c"],- }- if total["a"] != 4 || total["b"] != 4 || total["c"] != 4 {- t.Fatalf("unequal depths did not converge: assigned=%v total=%v", assigned, total)- }-}--func TestReserveCandidate_CombinesPendingPressureAndReservations(t *testing.T) {- p := prProxy(t)- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"a", "b", "c"},- Ranks: []schedulerwire.NodeRank{- {ID: "a", Pending: 0, GPUPressure: 3},- {ID: "b", Pending: 1, GPUPressure: 0},- {ID: "c", Pending: 0, GPUPressure: 2},- },- })- candidates := reservationCandidates("a", "b", "c")- got := []string{- reservedID(p, candidates),- reservedID(p, candidates),- reservedID(p, candidates),- }- want := []string{"b", "b", "c"}- for i := range want {- if got[i] != want[i] {- t.Fatalf("GPU-aware reservations = %v, want %v", got, want)- }- }-}--func TestSetPrioritySnapshotClampsGPUPressure(t *testing.T) {- p := prProxy(t)- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"low", "high"},- Ranks: []schedulerwire.NodeRank{- {ID: "low", GPUPressure: -1},- {ID: "high", GPUPressure: schedulerwire.MaxGPUPressure + 1},- },- })- p.priorityMu.RLock()- low := p.priorityGPUPressure["low"]- high := p.priorityGPUPressure["high"]- p.priorityMu.RUnlock()- if low != 0 || high != schedulerwire.MaxGPUPressure {- t.Fatalf("clamped GPU pressure = low:%d high:%d", low, high)- }-}--func TestReserveCandidate_LegacyNodesOnlyUsesZeroBaseline(t *testing.T) {- p := prProxy(t)- p.SetPriority([]string{"a", "b", "c"})- candidates := reservationCandidates("a", "b", "c")- counts := map[string]int{}- for range 5 {- counts[reservedID(p, candidates)]++- }- want := map[string]int{"a": 2, "b": 2, "c": 1}- for id, n := range want {- if counts[id] != n {- t.Fatalf("legacy nodes-only assignments = %v, want %v", counts, want)- }- }-}--func TestReserveCandidate_UsesEligibleCandidates(t *testing.T) {- p := prProxy(t)- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"missing", "owner-a", "owner-b", "unknown"},- Ranks: []schedulerwire.NodeRank{- {ID: "missing", Pending: 0},- {ID: "owner-a", Pending: 4},- {ID: "owner-b", Pending: 5},- {ID: "unknown", Pending: 0},- },- })- candidates := reservationCandidates("owner-a", "owner-b")-- for range 8 {- got := reservedID(p, candidates)- if got != "owner-a" && got != "owner-b" {- t.Fatalf("reservation escaped eligible candidates to %q", got)- }- }-}--func TestReserveCandidate_ManualPinBypassesReservations(t *testing.T) {- p := prProxy(t)- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"a", "b"},- Ranks: []schedulerwire.NodeRank{{ID: "a"}, {ID: "b"}},- })- p.SetSelected("b")- candidates := reservationCandidates("b", "a") // resolveCandidates puts the pin first- if got := reservedID(p, candidates); got != "b" {- t.Fatalf("manual pin resolved to %q, want b", got)- }- p.priorityMu.RLock()- defer p.priorityMu.RUnlock()- if len(p.priorityReservations) != 0 {- t.Fatalf("manual pin created optimistic reservations: %v", p.priorityReservations)- }-}--func TestReserveCandidate_IneligibleManualPinDoesNotBypassReservations(t *testing.T) {- p := prProxy(t)- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"owner-b", "owner-a"},- Ranks: []schedulerwire.NodeRank{{ID: "owner-b"}, {ID: "owner-a"}},- })- p.SetSelected("missing")- if got := reservedID(p, reservationCandidates("owner-a", "owner-b")); got != "owner-b" {- t.Fatalf("reservation with ineligible pin = %q, want owner-b", got)- }-}--func TestReserveCandidate_PreservesFailoverAndSnapshotReset(t *testing.T) {- p := prProxy(t)- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"b", "a", "c"},- Ranks: []schedulerwire.NodeRank{- {ID: "b", Pending: 0},- {ID: "a", Pending: 5},- {ID: "c", Pending: 6},- },- })- got := p.reserveCandidate(reservationCandidates("a", "b", "c"))- want := []string{"b", "a", "c"}- for i, id := range want {- if got[i].id != id {- t.Fatalf("reserved failover order = %v, want %v", candidateIDsFrom(got), want)- }- }-- p.SetPrioritySnapshot(schedulerwire.Priority{- Nodes: []string{"a", "b", "c"},- Ranks: []schedulerwire.NodeRank{{ID: "a"}, {ID: "b"}, {ID: "c"}},- })- if next := reservedID(p, reservationCandidates("a", "b", "c")); next != "a" {- t.Fatalf("new snapshot did not reset reservations: next = %q, want a", next)- }-}--func candidateIDsFrom(candidates []candidate) []string {- out := make([]string, 0, len(candidates))- for _, candidate := range candidates {- out = append(out, candidate.id)- }- return out-}diff --git a/services/ollama-proxy/server_timeout_test.go b/services/ollama-proxy/server_timeout_test.godeleted file mode 100644index 30384b5..0000000--- a/services/ollama-proxy/server_timeout_test.go+++ /dev/null@@ -1,43 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "context"- "net"- "testing"-- "nvpair-shared/clustertrust"-)--func TestHTTPServersConfigureIdleTimeouts(t *testing.T) {- p := testProxy(NewDiscovery(), 11435)- p.mesh = clustertrust.Open(t.TempDir())- ln, err := net.Listen("tcp", "127.0.0.1:0")- if err != nil {- t.Fatalf("listen: %v", err)- }- p.serveHTTP(context.Background(), ln)- defer p.shutdown(context.Background())-- if p.plainSrv == nil || p.tlsSrv == nil {- t.Fatal("servers not recorded")- }- for name, srv := range map[string]struct {- readHeader, idle interface{}- }{- "plain": {p.plainSrv.ReadHeaderTimeout, p.plainSrv.IdleTimeout},- "tls": {p.tlsSrv.ReadHeaderTimeout, p.tlsSrv.IdleTimeout},- } {- if srv.readHeader != proxyReadHeaderTimeout {- t.Errorf("%s ReadHeaderTimeout = %v, want %v", name, srv.readHeader, proxyReadHeaderTimeout)- }- if srv.idle != proxyServerIdleTimeout {- t.Errorf("%s IdleTimeout = %v, want %v", name, srv.idle, proxyServerIdleTimeout)- }- }- if proxyServerIdleTimeout != proxyIdleConnTimeout {- t.Fatalf("server IdleTimeout %v != client IdleConnTimeout %v", proxyServerIdleTimeout, proxyIdleConnTimeout)- }-}diff --git a/services/ollama-proxy/transport.go b/services/ollama-proxy/transport.godeleted file mode 100644index b62d627..0000000--- a/services/ollama-proxy/transport.go+++ /dev/null@@ -1,25 +0,0 @@-// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-// SPDX-License-Identifier: Apache-2.0--package main--import (- "io"- "os"-)--type stdioTransport struct {- io.Reader- io.Writer-}--func newStdioTransport() io.ReadWriteCloser {- return &stdioTransport{- Reader: os.Stdin,- Writer: os.Stdout,- }-}--func (s *stdioTransport) Close() error {- return nil-}diff --git a/services/readme.md b/services/readme.mdindex ee3523d..84a9421 100644--- a/services/readme.md+++ b/services/readme.md@@ -35,13 +35,12 @@ see the [root README](../README.md#what-is-supported).## Architecture-This tree builds thirteen Go binaries. `nvpair-ui-broker` is the parent service and supervises the eleven workers, all spawned at startup — only the scanner is required, and a missing binary for any other leaves the broker running without that capability. `nvpair-tui` is the thirteenth: a terminal client that launches and supervises its own broker rather than being supervised. Processes communicate via newline-delimited JSON-RPC 2.0 over stdio or, optionally, a Unix socket / Windows named pipe.+This tree builds twelve Go binaries. `nvpair-ui-broker` is the parent service and supervises the workers, all spawned at startup — only the scanner is required, and a missing binary for any other leaves the broker running without that capability. `nvpair-proxy` is one worker process that fronts every enabled engine, hosting a facade for each. `nvpair-tui` is a terminal client that launches and supervises its own broker rather than being supervised. Processes communicate via newline-delimited JSON-RPC 2.0 over stdio or, optionally, a Unix socket / Windows named pipe.| Binary | Role || --- | --- || `nvpair-ui-broker` | Parent service and JSON-RPC API surface used by the bundled UI and other clients. Supervises workers, relays consolidated discovery, and coordinates routing and scheduling. |-| `ollama-proxy` | Ollama-compatible HTTP reverse proxy. Routes only to advertised model owners, with owner failover and scheduler priorities. |-| `lmstudio-proxy` | LM Studio counterpart to `ollama-proxy`, forwarding OpenAI-compatible inference routes with equivalent owner-only routing and failover behavior. |+| `nvpair-proxy` | The engine HTTP reverse proxy: one process hosting a facade per enabled engine, each enabled over `facade/enable` after spawn. Serves each engine's own dialect (Ollama-native and OpenAI-compatible), routes only to advertised model owners, and applies owner failover and scheduler priorities. || `nvpair-node-info` | Local HTTP service on `:14318` exposing GPU, CPU, and memory inventory at `/v1/node-info`. || `nvpair-node-scanner` | Consolidated discovery daemon. Advertises and browses `_nvpair-node._tcp`, maintains the node directory, and enriches peers with hardware and model information over HTTP. || `nvpair-manual-nodes` | Manages user-added nodes that don't appear via mDNS; probes them every 10 s. |@@ -68,8 +67,7 @@ so bursts spread without waiting for workload feedback.```nvpair-ui-broker/ Parent service / JSON-RPC API surface-ollama-proxy/ Ollama-compatible routing proxy-lmstudio-proxy/ OpenAI-compatible routing proxy for LM Studio+nvpair-proxy/ Engine routing proxy, one process hosting a facade per enginenvpair-node-info/ Local GPU-inventory HTTP servicenvpair-node-scanner/ Consolidated _nvpair-node._tcp discovery daemonnvpair-manual-nodes/ Manual-node manager@@ -84,8 +82,8 @@ shared/ Shared Go module (nvpair-shared/…)eap-noob/ EAP-NOOB implementation used by cluster pairingtests/ Cross-process integration tests (separate go.mod)versions.json Single source of truth for every component version-build.bat Builds all thirteen binaries (Windows)-build.sh Builds all thirteen binaries (Linux)+build.bat Builds all twelve binaries (Windows)+build.sh Builds all twelve binaries (Linux)VERSIONING.md SemVer rules and version-bump workflow```@@ -115,7 +113,7 @@ On Linux and macOS:./build.sh```-Both scripts read `versions.json`, build all thirteen Go binaries with `-X main.Version=…` ldflags, and stage them together in `services/build/bin/`.+Both scripts read `versions.json`, build all twelve Go binaries with `-X main.Version=…` ldflags, and stage them together in `services/build/bin/`.Do **not** build individual components by hand without also copying their binaries into `build/bin/`: the broker will silently keep using the older binary there.@@ -190,7 +188,7 @@ cd sharedgo test ./...```-**Every one of the thirteen binaries has tests**, as do `shared/` and+**Every one of the twelve binaries has tests**, as do `shared/` and`eap-noob/`. Depth varies with how much behaviour a component carries:`nvpair-engine-manager` and `nvpair-cluster-manager` have the largest suites,while a component with one test file may still hold twenty test functions in it.@@ -237,11 +235,11 @@ A skip is not a pass. If you are relying on a test, check it actually ran.You can verify a built binary's stamped version at any time:```powershell-.\build\bin\ollama-proxy.exe --version # Windows+.\build\bin\nvpair-proxy.exe --version # Windows``````bash-./build/bin/ollama-proxy --version # Linux+./build/bin/nvpair-proxy --version # Linux```## Further readingdiff --git a/services/shared/discovery/discovery.go b/services/shared/discovery/discovery.goindex 973ac4b..143a0af 100644--- a/services/shared/discovery/discovery.go+++ b/services/shared/discovery/discovery.go@@ -1,10 +1,10 @@// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.// SPDX-License-Identifier: Apache-2.0-// Package discovery provides a single mDNS service browser that six services-// (nvpair-node-scanner, nvpair-errors, ollama-proxy, lmstudio-proxy,-// nvpair-workload-manager, nvpair-cluster-manager) each carried a near-identical copy-// of (the mDNS dedup).+// Package discovery provides a single mDNS service browser that five services+// (nvpair-node-scanner, nvpair-errors, nvpair-proxy, nvpair-workload-manager,+// nvpair-cluster-manager) each carried a near-identical copy of (the mDNS+// dedup).//// The core is a scan-and-diff state machine: each scan browses a service type// over grandcat/zeroconf, re-sends the PTR query from a per-interface unicastdiff --git a/services/shared/engines/engines.go b/services/shared/engines/engines.gonew file mode 100644index 0000000..b7f1e8c--- /dev/null+++ b/services/shared/engines/engines.go@@ -0,0 +1,203 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++// Package engines is the single source of truth for the identity of every+// inference engine PAIR fronts, shared by the proxy, the broker, the scheduler,+// and the TUI.+//+// These values are cross-process contracts, not labels. Name is the engine id+// carried in every engine-manager JSON-RPC call, in the per-engine model+// attribution on a discovery record, on a workload's Engine field, and as the+// basename of the engine-manager manifest that describes how to install and run+// the engine. A mismatch between any two of those does not fail to compile in a+// world where each component spells the id itself — it produces an engine that+// discovers, advertises, and never resolves a model owner. Declaring the set+// once, here, is what turns that class of bug into a build error.+//+// Design decisions this package encodes:+//+// - Name is lower-case and unpunctuated ("lmstudio", not "lm-studio"). The+// desktop UI uses its own hyphenated spelling for display types and bridges+// to this one at the process boundary; that bridge is the only place the two+// vocabularies are allowed to meet.+//+// - There are two proxy identities, and which one a caller wants depends on+// whether it is naming a *facade* or the *process*.+//+// ComponentName ("<Name>-proxy") is the per-engine facade identity: the+// JSON-RPC relay prefix, the error-ID prefix, and the TUI proxies-view tab.+// These are per-engine because a client addressing "ollama-proxy:nodes/list"+// or reading an "ollama-proxy:upstream-unreachable" error means that engine,+// not whichever process happens to serve it.+//+// ProxyComponent ("nvpair-proxy") is the process identity: the applog+// component, the subprocess diagnostics name, the supervisor label, and the+// crash key the TUI health view matches. These belong to the process+// because one OS process serves every facade, so there is no single engine+// to attribute them to; logs carry an engine field instead.+//+// The errors-pipeline *source* deliberately stays per-facade. It is only+// log context on the broker's side, and knowing which engine reported an+// error is more useful there than knowing it came from the proxy process.+//+// The supervisor label and the TUI crash key are matched against each+// other, so they must move together — the broker stamps its crash id from+// the label and the health view looks that id up by name. Moving one and+// not the other is silent in both directions: the crash entry matches no+// row, and the stale row can never leave "ok". It compiles, and every other+// test passes. TestHealthProxyRowMatchesTheBrokerCrashIdentity in+// nvpair-tui/ui is what turns this paragraph into something enforced.+//+// These were one value until the proxy binary began hosting more than one+// engine. Ollama's relay prefix was itself once the bare "proxy" while its+// error IDs were already "ollama-proxy" — two near-identical identities+// that coincided for LM Studio and diverged for Ollama, which is the trap+// this package exists to prevent. Keep the distinction explicit: a new+// identity belongs to the facade or to the process, never to both.+//+// - FacadePort is the engine's own client-facing default — the port a stock+// install listens on, and therefore the port PAIR's compatibility proxy+// claims so existing clients keep working. EnginePortBase is where PAIR+// relocates the engine to free that port, and the base of the next-free-port+// search. The two must differ.+//+// - All is ordered, and Ollama is first. The broker prepares managed ports in+// this order, and Ollama's preparation reserves any inherited OLLAMA_HOST+// alias that later engines must route around. Iterating a map here would+// make that a coin flip and the test that guards it a flake.+package engines++import (+ "strings"++ "nvpair-shared/noderec"+)++// Engine is the cross-process identity of one inference engine.+type Engine struct {+ // Name is the canonical engine id used by engine-manager RPCs, discovery+ // model attribution, workload records, the scheduler, and the manifest+ // basename. It must match services/nvpair-engine-manager/manifests/<Name>.json.+ Name string++ // DisplayName is the engine's name in user-facing text. Not derivable from+ // Name: it carries capitalization and, for LM Studio, a space.+ DisplayName string++ // DiscoveryService is the compact TXT key under which a node advertises this+ // engine's port on the consolidated _nvpair-node record.+ DiscoveryService noderec.ServiceKey++ // FacadePort is the engine's stock client-facing port, which PAIR's proxy+ // claims in managed mode.+ FacadePort int++ // EnginePortBase is where PAIR relocates the engine so the proxy can take+ // FacadePort, and the base of the next-free-port search.+ EnginePortBase int++ // PortFile is the per-user file this engine's proxy persists its chosen+ // port to. Declared rather than derived from Name, because Ollama's+ // predates the multi-engine world and is the bare "proxy-port.json";+ // deriving it would rename the file under every existing install and+ // silently orphan the port a user had set. The proxy writes it and the+ // broker reads it when reserving ports away from the OLLAMA_HOST alias,+ // so the two must agree — which is why it lives here.+ PortFile string+}++// ProxyComponent is the proxy *process* identity, as distinct from the+// per-engine facade identity ComponentName returns. It matches the binary name+// in versions.json and the desktop binary inventory, so a log prefix, a support+// bundle, and a process listing all agree.+//+// It is what the applog component, supervisor label, subprocess diagnostics+// name, and TUI health crash key all spell. The last two are matched against+// each other; see the package comment.+const ProxyComponent = "nvpair-proxy"++// ComponentName is the engine's *facade* identity: the JSON-RPC relay prefix,+// the error-ID prefix, the errors-pipeline log source, and the TUI+// proxies-view tab. Always "<Name>-proxy".+//+// Not the supervisor label or the health crash key — those name the process,+// which hosts every facade. The persisted-port filename is never derived from+// it either; see PortFile.+func (e Engine) ComponentName() string { return e.Name + "-proxy" }++// all is the ordered engine set. Ollama is first; see the package comment.+var all = []Engine{+ {+ Name: "ollama",+ DisplayName: "Ollama",+ DiscoveryService: noderec.ServiceOllama,+ FacadePort: 11434,+ EnginePortBase: 11435,+ PortFile: "proxy-port.json",+ },+ {+ Name: "lmstudio",+ DisplayName: "LM Studio",+ DiscoveryService: noderec.ServiceLMStudio,+ FacadePort: 1234,+ EnginePortBase: 1235,+ PortFile: "lmstudio-proxy-port.json",+ },+}++// All returns the engine set in preparation order. The result is a copy, so a+// caller cannot reorder the shared table.+func All() []Engine {+ out := make([]Engine, len(all))+ copy(out, all)+ return out+}++// Names returns every engine id in preparation order.+func Names() []string {+ out := make([]string, len(all))+ for i, e := range all {+ out[i] = e.Name+ }+ return out+}++// ByName resolves an engine id. Callers narrowing an id that arrived from a+// flag, a config file, or the wire must check ok rather than assuming.+func ByName(name string) (Engine, bool) {+ for _, e := range all {+ if e.Name == name {+ return e, true+ }+ }+ return Engine{}, false+}++// AddressMethod prefixes a facade-scoped JSON-RPC method with its engine, so a+// process hosting several facades can tell which one a message concerns.+//+// This is the proxy's *internal* wire addressing, deliberately not+// ComponentName: "ollama-proxy:" is what a client addresses the component by,+// and reusing it here would leave two different meanings on one string. The+// prefix matches the engine ids in facade/enable, so one message and its enable+// name the facade the same way.+func AddressMethod(engine, method string) string { return engine + ":" + method }++// SplitAddressedMethod separates a facade-scoped method's engine prefix from the+// method itself.+//+// A process-scoped method comes back with an empty engine and unchanged, so a+// caller distinguishes the two by the engine rather than by guessing from the+// method name. That matters because plenty of unaddressed methods contain a+// colon already — "errors:report", "discovery:subscribe", "workload:started" —+// and only a known engine id counts as an address.+func SplitAddressedMethod(method string) (engine, bare string) {+ name, rest, found := strings.Cut(method, ":")+ if !found {+ return "", method+ }+ if _, ok := ByName(name); !ok {+ return "", method+ }+ return name, rest+}diff --git a/services/shared/engines/engines_test.go b/services/shared/engines/engines_test.gonew file mode 100644index 0000000..aa609be--- /dev/null+++ b/services/shared/engines/engines_test.go@@ -0,0 +1,210 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++package engines++import (+ "strings"+ "testing"++ "nvpair-shared/noderec"+)++// TestOllamaIsPreparedFirst pins the ordering the broker's managed-port+// preparation depends on: Ollama's preparation reserves any inherited+// OLLAMA_HOST alias, and every later engine's port planning must route around+// it. Reordering the table silently breaks that on roughly half of all runs.+func TestOllamaIsPreparedFirst(t *testing.T) {+ got := Names()+ if len(got) == 0 || got[0] != "ollama" {+ t.Fatalf("Names()[0] = %q, want \"ollama\" — the alias-owning engine must be prepared first", got)+ }+}++func TestNames(t *testing.T) {+ got := Names()+ want := []string{"ollama", "lmstudio"}+ if len(got) != len(want) {+ t.Fatalf("Names() = %v, want %v", got, want)+ }+ for i := range want {+ if got[i] != want[i] {+ t.Fatalf("Names() = %v, want %v", got, want)+ }+ }+}++// There are two proxy identities: ComponentName per facade, ProxyComponent per+// process. Ollama's relay prefix was once the bare "proxy" while its error IDs+// were already "ollama-proxy" — two near-identical values that coincided for LM+// Studio, which made them easy to mistake for one field. This pins the shape of+// both so a new engine cannot reintroduce that ambiguity.+func TestProxyIdentities(t *testing.T) {+ for _, e := range All() {+ want := e.Name + "-proxy"+ if e.ComponentName() != want {+ t.Errorf("%s ComponentName = %q, want %q", e.Name, e.ComponentName(), want)+ }+ // The bare prefix Ollama used to relay under must not come back.+ if e.ComponentName() == "proxy" {+ t.Errorf("%s still uses the bare %q relay prefix", e.Name, "proxy")+ }+ // A facade identity that equals the process identity would make the+ // relay prefix and the process name indistinguishable.+ if e.ComponentName() == ProxyComponent {+ t.Errorf("%s facade identity collides with the process identity %q", e.Name, ProxyComponent)+ }+ }++ // The process identity must match the binary name, so a log prefix, a+ // support bundle, and a process listing agree. services/versions.json and+ // the desktop binary inventory both spell it this way.+ if ProxyComponent != "nvpair-proxy" {+ t.Errorf("ProxyComponent = %q, want the binary name %q", ProxyComponent, "nvpair-proxy")+ }+}++// TestFacadeAndEnginePortsDiffer guards the invariant that makes managed mode+// coherent: the proxy claims the engine's stock port, so the engine has to move+// somewhere else.+func TestFacadeAndEnginePortsDiffer(t *testing.T) {+ for _, e := range All() {+ if e.FacadePort == e.EnginePortBase {+ t.Errorf("%s: FacadePort and EnginePortBase are both %d; the engine has nowhere to move",+ e.Name, e.FacadePort)+ }+ if e.FacadePort <= 0 || e.EnginePortBase <= 0 {+ t.Errorf("%s: ports must be positive, got facade=%d base=%d",+ e.Name, e.FacadePort, e.EnginePortBase)+ }+ }+}++// PortFile is declared, not derived. Ollama's is the bare "proxy-port.json"+// from before there was more than one engine; deriving it from Name would+// rename the file under every existing install and orphan whatever port the+// user had chosen. This test exists so a future tidy-up cannot quietly make+// that trade.+func TestPortFilesAreDeclaredNotDerived(t *testing.T) {+ ollama, ok := ByName("ollama")+ if !ok {+ t.Fatal("no ollama engine")+ }+ if ollama.PortFile != "proxy-port.json" {+ t.Errorf("ollama PortFile = %q, want the pre-unification proxy-port.json", ollama.PortFile)+ }+ if derived := ollama.ComponentName() + "-port.json"; ollama.PortFile == derived {+ t.Errorf("ollama PortFile now matches the derived name %q; existing installs would be orphaned", derived)+ }+ for _, e := range All() {+ if e.PortFile == "" {+ t.Errorf("%s has no PortFile; its proxy would persist nothing", e.Name)+ }+ }+}++func TestIdentitiesAreUnique(t *testing.T) {+ names := map[string]bool{}+ components := map[string]bool{}+ services := map[noderec.ServiceKey]bool{}+ facades := map[int]bool{}++ for _, e := range All() {+ if e.Name == "" || e.DisplayName == "" || e.PortFile == "" || e.DiscoveryService == "" {+ t.Errorf("%+v: every identity field must be set", e)+ }+ if names[e.Name] {+ t.Errorf("duplicate Name %q", e.Name)+ }+ if components[e.ComponentName()] {+ t.Errorf("duplicate ComponentName %q", e.ComponentName())+ }+ if services[e.DiscoveryService] {+ t.Errorf("duplicate DiscoveryService %q", e.DiscoveryService)+ }+ if facades[e.FacadePort] {+ t.Errorf("duplicate FacadePort %d", e.FacadePort)+ }+ names[e.Name] = true+ components[e.ComponentName()] = true+ services[e.DiscoveryService] = true+ facades[e.FacadePort] = true+ }+}++// TestAllReturnsACopy keeps a caller from reordering the shared table, which+// would defeat TestOllamaIsPreparedFirst at a distance.+func TestAllReturnsACopy(t *testing.T) {+ first := All()+ if len(first) < 2 {+ t.Fatalf("expected at least two engines, got %d", len(first))+ }+ first[0], first[1] = first[1], first[0]++ if Names()[0] != "ollama" {+ t.Fatal("mutating the result of All() reordered the shared table")+ }+}++func TestLookups(t *testing.T) {+ for _, e := range All() {+ byName, ok := ByName(e.Name)+ if !ok || byName.ComponentName() != e.ComponentName() {+ t.Errorf("ByName(%q) = %+v, %v", e.Name, byName, ok)+ }+ }++ if _, ok := ByName("vllm"); ok {+ t.Error("ByName should report ok=false for an unknown engine")+ }+}++func TestAddressedMethodRoundTrip(t *testing.T) {+ for _, e := range All() {+ for _, method := range []string{"ready", "nodes/list", "errors:report"} {+ addressed := AddressMethod(e.Name, method)+ engine, bare := SplitAddressedMethod(addressed)+ if engine != e.Name || bare != method {+ t.Errorf("round trip of %q gave (%q, %q), want (%q, %q)",+ addressed, engine, bare, e.Name, method)+ }+ }+ }+}++// Plenty of unaddressed methods contain a colon, so only a known engine id may+// count as an address. Treating "errors:report" as engine "errors" would strip+// a real method down to "report" and route it nowhere.+func TestUnaddressedMethodsAreNotMistakenForAddressed(t *testing.T) {+ for _, method := range []string{+ "ready",+ "errors:report",+ "errors:clear",+ "discovery:subscribe",+ "discovery:nodes",+ "discovery:node-activity",+ "workload:started",+ "proxy/request",+ "node/selection-changed",+ "log/set-level",+ } {+ engine, bare := SplitAddressedMethod(method)+ if engine != "" {+ t.Errorf("SplitAddressedMethod(%q) found engine %q; nothing but an engine id is an address", method, engine)+ }+ if bare != method {+ t.Errorf("SplitAddressedMethod(%q) rewrote the method to %q", method, bare)+ }+ }+}++// An engine id that is a prefix of another would make addresses ambiguous:+// splitting on the first colon cannot tell "lm:x" from "lmstudio:x" if one id+// is a prefix of the other and the separator is ever omitted.+func TestNoEngineIDContainsTheAddressSeparator(t *testing.T) {+ for _, e := range All() {+ if strings.Contains(e.Name, ":") {+ t.Errorf("engine id %q contains the address separator", e.Name)+ }+ }+}diff --git a/services/shared/netmon/netmon.go b/services/shared/netmon/netmon.goindex ecf7fb4..f736678 100644--- a/services/shared/netmon/netmon.go+++ b/services/shared/netmon/netmon.go@@ -6,7 +6,7 @@// addresses) changes.//// It exists because several nvpair subprocesses snapshot the interface list once-// at startup and never refresh it: ollama-proxy's "is this address local?"+// at startup and never refresh it: nvpair-proxy's "is this address local?"// table and every mDNS responder's per-interface address map. On a multi-homed// host, or after a sleep/wake that reassigns an IP, those snapshots go stale// and the process keeps using (or advertising) an address that no longerdiff --git a/services/shared/schedulerwire/schedulerwire.go b/services/shared/schedulerwire/schedulerwire.goindex bcedca5..f49677a 100644--- a/services/shared/schedulerwire/schedulerwire.go+++ b/services/shared/schedulerwire/schedulerwire.go@@ -5,6 +5,8 @@// scheduler rankings through the broker to the inference proxies.package schedulerwire+import "slices"+const (MethodTelemetry = "scheduler:telemetry"MaxGPUPressure = 3@@ -23,18 +25,50 @@ type NodeRank struct {// Ranks is optional so a newer broker can still drive an older nodes-only// producer or consumer during a rolling upgrade.type Priority struct {- Nodes []string `json:"nodes"`- Ranks []NodeRank `json:"ranks,omitempty"`+ // Generation makes applying a snapshot idempotent, which matters because+ // applying one also clears the proxy's optimistic reservations.+ //+ // A node/set-priority call can time out after the proxy already applied it,+ // so the broker cannot distinguish "never arrived" from "already done" and+ // may redeliver. Without a version the redelivery would clear reservations+ // taken since the first apply — discarding exactly the dispatches the+ // scheduler has not yet seen. The proxy records the last generation it+ // applied and treats an equal one as an acknowledged no-op and an older one+ // as stale.+ //+ // Required, and greater than zero. A producer must mint a monotonic+ // generation per delivery; the proxy rejects zero at the RPC boundary and+ // ignores it thereafter.+ //+ // Zero used to mean "apply unconditionally", which reintroduced the very+ // undercount this field prevents: it cleared the reservations while+ // leaving the applied epoch untouched, so a reservation taken before the+ // snapshot still matched and released one taken after it.+ Generation uint64 `json:"generation,omitempty"`+ Nodes []string `json:"nodes"`+ Ranks []NodeRank `json:"ranks,omitempty"`}// Clone returns an independently owned snapshot safe to cache across goroutines.func (p Priority) Clone() Priority {return Priority{- Nodes: append([]string(nil), p.Nodes...),- Ranks: append([]NodeRank(nil), p.Ranks...),+ Generation: p.Generation,+ Nodes: append([]string(nil), p.Nodes...),+ Ranks: append([]NodeRank(nil), p.Ranks...),}}+// SameRanking reports whether two snapshots carry the same node-wide ranking,+// ignoring Generation. The broker uses it to drop the scheduler's duplicate+// per-engine emission of one recompute before it mints a generation.+//+// Not reflect.DeepEqual: Snapshot and Clone normalize an empty slice to nil, so+// the two agree on every reachable value here, but DeepEqual would separate+// nil from empty if a producer ever sent one.+func (p Priority) SameRanking(other Priority) bool {+ return slices.Equal(p.Nodes, other.Nodes) && slices.Equal(p.Ranks, other.Ranks)+}+// EnginePriority is the schedule:priority notification emitted by the scheduler.type EnginePriority struct {Engine string `json:"engine"`diff --git a/services/tests/broker_management_test.go b/services/tests/broker_management_test.goindex 25a4afe..d09a675 100644--- a/services/tests/broker_management_test.go+++ b/services/tests/broker_management_test.go@@ -140,7 +140,7 @@ func proxyNodesHas(t *testing.T, raw json.RawMessage, id string) bool {// TestBrokerBridgesManualNodeIntoProxy is the leg-A repro: with the broker// supervising both nvpair-manual-nodes and ollama-proxy, a manual node whose// Ollama is reachable must be bridged into the proxy (node/add-manual) so it-// shows up in proxy:nodes/list and can be routed to — even though it never+// shows up in ollama-proxy:nodes/list and can be routed to — even though it never// appears over mDNS.func TestBrokerBridgesManualNodeIntoProxy(t *testing.T) {if portBusy(11435) {@@ -151,7 +151,7 @@ func TestBrokerBridgesManualNodeIntoProxy(t *testing.T) {stdin, msgs, _, cleanup := startBrokerWith(t,"--manual-nodes-path", manualNodesBin,- "--proxy-path", proxyBin,+ "--proxy-path", proxyBin, "--proxy-engines", "ollama",)t.Cleanup(cleanup)@@ -163,14 +163,14 @@ func TestBrokerBridgesManualNodeIntoProxy(t *testing.T) {t.Fatalf("write node/add: %v", err)}- // Poll the proxy's node set (via the broker's proxy: relay) until the+ // Poll the proxy's node set (via the broker's ollama-proxy: relay) until the// bridged manual node appears. The first probe fires immediately on// node/add; re-probes every 10s re-bridge if the proxy wasn't ready yet.deadline := time.After(25 * time.Second)ticker := time.NewTicker(1 * time.Second)defer ticker.Stop()reqID := 901- sendReq(t, stdin, reqID, "proxy:nodes/list")+ sendReq(t, stdin, reqID, "ollama-proxy:nodes/list")for {select {case msg, ok := <-msgs:@@ -183,9 +183,9 @@ func TestBrokerBridgesManualNodeIntoProxy(t *testing.T) {}case <-ticker.C:reqID++- sendReq(t, stdin, reqID, "proxy:nodes/list")+ sendReq(t, stdin, reqID, "ollama-proxy:nodes/list")case <-deadline:- t.Fatalf("timed out waiting for manual node %q in proxy:nodes/list", nodeName)+ t.Fatalf("timed out waiting for manual node %q in ollama-proxy:nodes/list", nodeName)}}}@@ -194,7 +194,7 @@ func TestBrokerBridgesManualNodeIntoProxy(t *testing.T) {// manual node's node-info reports its stable hostUuid, the proxy candidate must// be keyed by that UUID — the same operational key the discovery store and// scheduler use — so scheduler priority (node/set-priority) and scheduledOn-// resolve to it. The candidate must therefore appear in proxy:nodes/list under+// resolve to it. The candidate must therefore appear in ollama-proxy:nodes/list under// the learned hostUuid, not the user-supplied manual name.func TestBrokerBridgesManualNodeUnderLearnedUUID(t *testing.T) {if portBusy(11435) {@@ -207,7 +207,7 @@ func TestBrokerBridgesManualNodeUnderLearnedUUID(t *testing.T) {stdin, msgs, _, cleanup := startBrokerWith(t,"--manual-nodes-path", manualNodesBin,- "--proxy-path", proxyBin,+ "--proxy-path", proxyBin, "--proxy-engines", "ollama",)t.Cleanup(cleanup)@@ -223,7 +223,7 @@ func TestBrokerBridgesManualNodeUnderLearnedUUID(t *testing.T) {ticker := time.NewTicker(1 * time.Second)defer ticker.Stop()reqID := 941- sendReq(t, stdin, reqID, "proxy:nodes/list")+ sendReq(t, stdin, reqID, "ollama-proxy:nodes/list")for {select {case msg, ok := <-msgs:@@ -241,9 +241,9 @@ func TestBrokerBridgesManualNodeUnderLearnedUUID(t *testing.T) {}case <-ticker.C:reqID++- sendReq(t, stdin, reqID, "proxy:nodes/list")+ sendReq(t, stdin, reqID, "ollama-proxy:nodes/list")case <-deadline:- t.Fatalf("timed out waiting for manual node under learned hostUuid in proxy:nodes/list")+ t.Fatalf("timed out waiting for manual node under learned hostUuid in ollama-proxy:nodes/list")}}}@@ -260,7 +260,7 @@ func TestBrokerBridgesManualNodeIntoLMStudioProxy(t *testing.T) {stdin, msgs, _, cleanup := startBrokerWithConfigDir(t, configDir,"--manual-nodes-path", manualNodesBin,- "--lmstudio-proxy-path", lmstudioProxyBin,+ "--proxy-path", proxyBin, "--proxy-engines", "lmstudio",)t.Cleanup(cleanup)diff --git a/services/tests/broker_supervision_test.go b/services/tests/broker_supervision_test.goindex de917bc..7eb37e6 100644--- a/services/tests/broker_supervision_test.go+++ b/services/tests/broker_supervision_test.go@@ -22,6 +22,7 @@ import ("testing""time"+ "nvpair-shared/engines""nvpair-shared/errors""nvpair-shared/jsonrpc")@@ -205,10 +206,15 @@ func portBusy(port int) bool {return false}-// TestBrokerCrashSurfacingAndRestart kills the broker's supervised proxy-// and asserts the crash is surfaced as supervisor:subprocess-crashed:proxy-// through the nvpair-errors pipeline (errors:update relay + errors:get-initial),-// and that the supervisor restarts the proxy.+// TestBrokerCrashSurfacingAndRestart kills the broker's supervised proxy and+// asserts the crash is surfaced through the nvpair-errors pipeline+// (errors:update relay + errors:get-initial), and that the supervisor restarts+// the proxy.+//+// The crash is reported for the process, not an engine: one nvpair-proxy hosts+// every facade, so there is one supervisor and one name to report. A crash-id+// naming a single engine would claim that only that engine went down when in+// fact all of them did.func TestBrokerCrashSurfacingAndRestart(t *testing.T) {if portBusy(14319) {t.Skip("nvpair-errors --peer-sync port 14319 already in use; skipping")@@ -219,7 +225,7 @@ func TestBrokerCrashSurfacingAndRestart(t *testing.T) {stdin, msgs, stderr, cleanup := startBrokerWith(t,"--errors-path", errorsBin,- "--proxy-path", proxyBin,+ "--proxy-path", proxyBin, "--proxy-engines", "ollama",)t.Cleanup(cleanup)@@ -235,13 +241,14 @@ func TestBrokerCrashSurfacingAndRestart(t *testing.T) {}// The crash must surface as a sticky supervisor error via errors:update.- waitForErrorsUpdateContaining(t, msgs, "supervisor:subprocess-crashed:proxy", 15*time.Second)+ crashID := "supervisor:subprocess-crashed:" + engines.ProxyComponent+ waitForErrorsUpdateContaining(t, msgs, crashID, 15*time.Second)t.Log("crash surfaced via errors:update")// errors:get-initial must relay the same entry from nvpair-errors.sendReq(t, stdin, 900, "errors:get-initial")resp := waitForResponse(t, msgs, 5*time.Second)- if !errorsListHasID(t, resp.Result, "supervisor:subprocess-crashed:proxy") {+ if !errorsListHasID(t, resp.Result, crashID) {t.Fatalf("errors:get-initial missing the crash entry: %s", resp.Result)}t.Log("errors:get-initial carries the crash entry")@@ -269,6 +276,66 @@ func mustPid(t *testing.T, line string) int {return pid}+// Both engines are fronted by ONE process. This is the property the whole+// unification exists for: the estimated-work reservations a proxy takes+// between scheduler snapshots live in the process, so two processes each held+// half the picture and could dispatch simultaneous bursts to the same node+// each believing it idle.+//+// Asserted on pids rather than on the code shape, because "one supervisor" is+// an implementation detail and "one OS process serving both engines" is the+// thing that actually makes the reservation map whole.+func TestBothEnginesAreServedByOneProxyProcess(t *testing.T) {+ if portBusy(11435) || portBusy(1234) {+ t.Skip("ollama-proxy (11435) or lmstudio-proxy (1234) default port already in use; skipping")+ }++ stdin, msgs, stderr, cleanup := startBrokerWith(t,+ // No --proxy-engines: both engines, which is the default.+ "--proxy-path", proxyBin,+ )+ t.Cleanup(cleanup)++ pids := make(chan int, 8)+ go func() {+ for line := range stderr {+ if m := proxyPidRe.FindStringSubmatch(line); m != nil {+ if pid, err := strconv.Atoi(m[1]); err == nil {+ pids <- pid+ }+ }+ }+ }()++ waitForMethod(t, msgs, "app:ready", 10*time.Second)++ // Both facades must actually come up, or "one process" would be trivially+ // true by one of them having failed.+ ollamaPort := waitProxyReady(t, stdin, msgs, 15*time.Second)+ lmstudioPort := waitLMStudioProxyReady(t, stdin, msgs, 15*time.Second)+ if ollamaPort == lmstudioPort {+ t.Fatalf("both facades report port %d; they must bind separately", ollamaPort)+ }+ t.Logf("ollama facade on :%d, lmstudio facade on :%d", ollamaPort, lmstudioPort)++ // Collect every spawn announced up to this point plus a settling window, so+ // a second process starting late still shows up.+ seen := map[int]bool{}+ deadline := time.After(2 * time.Second)+ for collecting := true; collecting; {+ select {+ case pid := <-pids:+ seen[pid] = true+ case <-deadline:+ collecting = false+ }+ }+ if len(seen) != 1 {+ t.Fatalf("saw %d proxy processes %v, want exactly 1 hosting both engines", len(seen), seen)+ }+ t.Logf("both engines served by pid %v", seen)+}+// TestBrokerShutsDownOnSignal verifies the broker exits promptly when it// receives SIGINT/SIGTERM, even though its read loop is parked on stdin// (no input arriving) — and that the supervisors do NOT resurrect the@@ -286,7 +353,7 @@ func TestBrokerShutsDownOnSignal(t *testing.T) {}cmd := exec.Command(brokerBin,"--scanner-path", scannerBin,- "--proxy-path", proxyBin,+ "--proxy-path", proxyBin, "--proxy-engines", "ollama","--cluster-dir", t.TempDir(),)cmd.Stderr = os.Stderr@@ -332,7 +399,7 @@ func TestBrokerSpawnsAllModules(t *testing.T) {stdin, msgs, stderr, cleanup := startBrokerWith(t,"--errors-path", errorsBin,"--node-info-path", nodeInfoBin,- "--proxy-path", proxyBin,+ "--proxy-path", proxyBin, "--proxy-engines", "ollama","--workload-manager-path", workloadMgrBin,"--engine-manager-path", engineMgrBin,"--manual-nodes-path", manualNodesBin,@@ -387,29 +454,29 @@ func TestBrokerEngineRelay(t *testing.T) {t.Logf("engine:get-installed returned %d engine(s)", len(result.Engines))}-// proxyStatus queries proxy:get-status through the broker and returns the+// proxyStatus queries ollama-proxy:get-status through the broker and returns the// proxy's readiness + bound port (the broker's captured view of the proxy's// last "ready").func proxyStatus(t *testing.T, stdin io.Writer, msgs <-chan jsonrpc.Message, id int) (bool, int) {t.Helper()- sendReq(t, stdin, id, "proxy:get-status")+ sendReq(t, stdin, id, "ollama-proxy:get-status")resp := waitForResponse(t, msgs, 5*time.Second)if resp.Error != nil {- t.Fatalf("proxy:get-status errored: %d %s", resp.Error.Code, resp.Error.Message)+ t.Fatalf("ollama-proxy:get-status errored: %d %s", resp.Error.Code, resp.Error.Message)}var s struct {Ready bool `json:"ready"`Port int `json:"port"`}if err := json.Unmarshal(resp.Result, &s); err != nil {- t.Fatalf("parse proxy:get-status: %v\nraw: %s", err, resp.Result)+ t.Fatalf("parse ollama-proxy:get-status: %v\nraw: %s", err, resp.Result)}return s.Ready, s.Port}-// TestBrokerProxySetPortRebinds drives proxy:set-port through the broker and-// asserts the proxy live-rebinds onto the new port (reflected by-// proxy:get-status). The broker is given a private config dir so the proxy's+// TestBrokerProxySetPortRebinds drives ollama-proxy:set-port through the broker+// and asserts the proxy live-rebinds onto the new port (reflected by+// ollama-proxy:get-status). The broker is given a private config dir so the proxy's// persisted port doesn't touch the developer's real config. With no engine// running, the requested port is free, so the broker relays it unchanged (no// bump) — exercising the interception + relay + rebind wiring end to end.@@ -420,7 +487,7 @@ func TestBrokerProxySetPortRebinds(t *testing.T) {cfg := t.TempDir()cmd := exec.Command(brokerBin,"--scanner-path", scannerBin,- "--proxy-path", proxyBin,+ "--proxy-path", proxyBin, "--proxy-engines", "ollama","--engine-manager-path", engineMgrBin,)// Isolate per-user state (proxy-port.json, any engine override) to a@@ -473,25 +540,25 @@ func TestBrokerProxySetPortRebinds(t *testing.T) {}target := freePort(t)- req := fmt.Sprintf(`{"jsonrpc":"2.0","id":860,"method":"proxy:set-port","params":{"port":%d}}`, target) + "\n"+ req := fmt.Sprintf(`{"jsonrpc":"2.0","id":860,"method":"ollama-proxy:set-port","params":{"port":%d}}`, target) + "\n"if _, err := stdin.Write([]byte(req)); err != nil {- t.Fatalf("write proxy:set-port: %v", err)+ t.Fatalf("write ollama-proxy:set-port: %v", err)}resp := waitForResponse(t, msgs, 10*time.Second)if resp.Error != nil {- t.Fatalf("proxy:set-port errored: %d %s", resp.Error.Code, resp.Error.Message)+ t.Fatalf("ollama-proxy:set-port errored: %d %s", resp.Error.Code, resp.Error.Message)}var sp struct {Port int `json:"port"`}if err := json.Unmarshal(resp.Result, &sp); err != nil {- t.Fatalf("parse proxy:set-port result: %v\nraw: %s", err, resp.Result)+ t.Fatalf("parse ollama-proxy:set-port result: %v\nraw: %s", err, resp.Result)}if sp.Port != target {t.Fatalf("set-port result port = %d, want %d (no engine running, so no bump expected)", sp.Port, target)}- // proxy:get-status must now reflect the rebound port.+ // ollama-proxy:get-status must now reflect the rebound port.id = 870deadline = time.Now().Add(10 * time.Second)for {@@ -501,7 +568,7 @@ func TestBrokerProxySetPortRebinds(t *testing.T) {}id++if time.Now().After(deadline) {- t.Fatalf("proxy:get-status never reported the rebound port %d", target)+ t.Fatalf("ollama-proxy:get-status never reported the rebound port %d", target)}time.Sleep(300 * time.Millisecond)}diff --git a/services/tests/log_level_test.go b/services/tests/log_level_test.goindex c1a1dfb..a99b697 100644--- a/services/tests/log_level_test.go+++ b/services/tests/log_level_test.go@@ -7,6 +7,7 @@ import ("bufio""bytes""encoding/json"+ "fmt""io""os/exec""strings"@@ -14,6 +15,7 @@ import ("testing""time"+ "nvpair-shared/engines""nvpair-shared/jsonrpc")@@ -47,6 +49,9 @@ func (b *safeBuffer) Reset() {func startProxyWithLog(t *testing.T, level string) (stdin io.WriteCloser, msgs <-chan jsonrpc.Message, stderr *safeBuffer, cleanup func()) {t.Helper()+ // No engine on argv: the binary starts with no facade and no listener, and+ // takes engines over facade/enable. Log level is process-scoped, so this+ // needs no facade at all.cmd := exec.Command(proxyBin, "--log-level", level)stderrBuf := &safeBuffer{}cmd.Stderr = stderrBuf@@ -66,6 +71,43 @@ func startProxyWithLog(t *testing.T, level string) (stdin io.WriteCloser, msgs <ch := startMsgReader(stdoutPipe)+ // Enable a facade so the process behaves like one the broker brought up:+ // it binds, announces ready, and starts its background work, which is what+ // produces the startup log lines this test counts. Which engine is+ // immaterial to log-level plumbing.+ //+ // The response is consumed here rather than left in the stream, so this+ // returns a proxy that is already up and the caller's first waitForResponse+ // still belongs to the caller's own request.+ const enableID = 900+ sendLine(t, stdinPipe, map[string]any{+ "jsonrpc": "2.0",+ "id": enableID,+ "method": "facade/enable",+ "params": map[string]any{+ "engine": "ollama",+ "port": freePort(t),+ "ignorePersistedPort": true,+ },+ })+ deadline := time.After(15 * time.Second)+ for enabled := false; !enabled; {+ select {+ case m, ok := <-ch:+ if !ok {+ t.Fatal("proxy stdout closed before facade/enable answered")+ }+ if m.Method == "" && m.ID != nil && string(*m.ID) == fmt.Sprint(enableID) {+ if m.Error != nil {+ t.Fatalf("facade/enable failed: %v", m.Error)+ }+ enabled = true+ }+ case <-deadline:+ t.Fatal("timed out waiting for facade/enable")+ }+ }+return stdinPipe, ch, stderrBuf, func() {stdinPipe.Close()done := make(chan error, 1)@@ -113,10 +155,10 @@ func TestLogSetLevelViaRPC(t *testing.T) {stdin, msgs, stderr, cleanup := startProxyWithLog(t, "debug")t.Cleanup(cleanup)- // Wait for ready (emitted at startup at info; also discovery/startup- // emits debug lines because we booted at debug).- waitForMethod(t, msgs, "ready", 5*time.Second)-+ // startProxyWithLog already waited out facade/enable, so the proxy is up+ // and has emitted its startup lines at info (plus debug ones, because we+ // booted at debug).+ //// Give the proxy a moment to emit a few debug lines (the first mDNS// scan runs immediately in Discovery.Run).time.Sleep(2 * time.Second)@@ -185,6 +227,8 @@ func TestLogSetLevelViaRPC(t *testing.T) {// TestLogLevelEnvFallback verifies NVPAIR_LOG_LEVEL is honoured when --log-level// is not passed.func TestLogLevelEnvFallback(t *testing.T) {+ // --log-level deliberately omitted, so the env fallback is what this+ // exercises. No engine needed: log level is process-scoped.cmd := exec.Command(proxyBin)cmd.Env = append(cmd.Environ(), "NVPAIR_LOG_LEVEL=debug")stderrBuf := &safeBuffer{}@@ -214,7 +258,23 @@ func TestLogLevelEnvFallback(t *testing.T) {}})- waitForMethod(t, msgs, "ready", 5*time.Second)+ // Enable a facade so the process reaches ready, the same way the broker+ // brings one up. The env-derived level applies from process init, before+ // any facade exists.+ sendLine(t, stdinPipe, map[string]any{+ "jsonrpc": "2.0",+ "id": 901,+ "method": "facade/enable",+ "params": map[string]any{+ "engine": "ollama",+ "port": freePort(t),+ "ignorePersistedPort": true,+ },+ })++ // A facade announces itself addressed to its engine, so the broker can tell+ // which one is ready when a process hosts several.+ waitForMethod(t, msgs, engines.AddressMethod("ollama", "ready"), 5*time.Second)// The first mDNS scan browse runs with a 3s timeout before emitting// its DEBUG summary; wait longer than that to guarantee we see at// least one DEBUG line.diff --git a/services/tests/main_test.go b/services/tests/main_test.goindex c5d63c9..a30b119 100644--- a/services/tests/main_test.go+++ b/services/tests/main_test.go@@ -19,18 +19,17 @@ import ()var (- proxyBin string- lmstudioProxyBin string- errorsBin string- nodeInfoBin string- scannerBin string- nodeSettingsBin string- brokerBin string- workloadMgrBin string- engineMgrBin string- manualNodesBin string- clusterMgrBin string- schedulerBin string+ proxyBin string+ errorsBin string+ nodeInfoBin string+ scannerBin string+ nodeSettingsBin string+ brokerBin string+ workloadMgrBin string+ engineMgrBin string+ manualNodesBin string+ clusterMgrBin string+ schedulerBin string)func TestMain(m *testing.M) {@@ -44,8 +43,7 @@ func TestMain(m *testing.M) {ext = ".exe"}- proxyBin = filepath.Join(tmpDir, "ollama-proxy"+ext)- lmstudioProxyBin = filepath.Join(tmpDir, "lmstudio-proxy"+ext)+ proxyBin = filepath.Join(tmpDir, "nvpair-proxy"+ext)errorsBin = filepath.Join(tmpDir, "nvpair-errors"+ext)nodeInfoBin = filepath.Join(tmpDir, "nvpair-node-info"+ext)scannerBin = filepath.Join(tmpDir, "nvpair-node-scanner"+ext)@@ -57,18 +55,12 @@ func TestMain(m *testing.M) {clusterMgrBin = filepath.Join(tmpDir, "nvpair-cluster-manager"+ext)schedulerBin = filepath.Join(tmpDir, "nvpair-job-scheduler"+ext)- log.Println("building ollama-proxy...")- if err := goBuild(filepath.Join("..", "ollama-proxy"), proxyBin); err != nil {+ // One process fronts every engine, with a facade enabled per engine, so+ // both the Ollama and LM Studio bridge tests run against this build.+ log.Println("building nvpair-proxy...")+ if err := goBuild(filepath.Join("..", "nvpair-proxy"), proxyBin); err != nil {os.RemoveAll(tmpDir)- log.Fatalf("build ollama-proxy: %v", err)- }-- // The broker supervises lmstudio-proxy too, so the LM Studio bridge test- // needs its binary (pointed at via --lmstudio-proxy-path).- log.Println("building lmstudio-proxy...")- if err := goBuild(filepath.Join("..", "lmstudio-proxy"), lmstudioProxyBin); err != nil {- os.RemoveAll(tmpDir)- log.Fatalf("build lmstudio-proxy: %v", err)+ log.Fatalf("build nvpair-proxy: %v", err)}log.Println("building nvpair-errors...")diff --git a/services/tests/model_routing_interop_test.go b/services/tests/model_routing_interop_test.goindex b6c13dd..8c81a27 100644--- a/services/tests/model_routing_interop_test.go+++ b/services/tests/model_routing_interop_test.go@@ -48,8 +48,9 @@ func TestStrictModelRoutingAcrossProcesses(t *testing.T) {ineligible := newRoutingUpstream(t, http.StatusOK)stdin, msgs, stderr, cleanup := startBrokerWith(t,+ // No --proxy-engines: this test wants both engines fronted, which is+ // the default."--proxy-path", proxyBin,- "--lmstudio-proxy-path", lmstudioProxyBin,)t.Cleanup(cleanup)go func() {@@ -68,7 +69,7 @@ func TestStrictModelRoutingAcrossProcesses(t *testing.T) {port int}cases := []proxyCase{- {name: "ollama", rpcPrefix: "proxy", path: "/api/chat", port: ollamaPort},+ {name: "ollama", rpcPrefix: "ollama-proxy", path: "/api/chat", port: ollamaPort},{name: "lmstudio", rpcPrefix: "lmstudio-proxy", path: "/v1/chat/completions", port: lmstudioPort},}client := &http.Client{Timeout: 5 * time.Second}@@ -110,7 +111,8 @@ func TestStrictModelRoutingAcrossProcesses(t *testing.T) {requestID++}callBrokerRPC(t, stdin, msgs, requestID, tc.rpcPrefix+":node/set-priority", map[string]any{- "nodes": []string{missingID, unknownID, owner404ID, ownerOKID},+ "generation": 1,+ "nodes": []string{missingID, unknownID, owner404ID, ownerOKID},})before := snapshot()diff --git a/services/tests/ollama_host_alias_test.go b/services/tests/ollama_host_alias_test.goindex 1892e34..75bda00 100644--- a/services/tests/ollama_host_alias_test.go+++ b/services/tests/ollama_host_alias_test.go@@ -40,7 +40,7 @@ func TestInheritedOllamaHostAliasEndToEnd(t *testing.T) {[]string{fmt.Sprintf("OLLAMA_HOST=localhost:%d", aliasPort)},"--settings-path", nodeSettingsBin,"--engine-manager-path", engineMgrBin,- "--proxy-path", proxyBin,+ "--proxy-path", proxyBin, "--proxy-engines", "ollama",)t.Cleanup(cleanup)@@ -51,9 +51,9 @@ func TestInheritedOllamaHostAliasEndToEnd(t *testing.T) {writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":1,"method":"workloads:subscribe"}`)waitForResponse(t, msgs, 5*time.Second)- writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":2,"method":"proxy:node/add-manual","params":{"id":"alias-upstream","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["alias-e2e-model"]}}`, upstreamPort))+ writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":2,"method":"ollama-proxy:node/add-manual","params":{"id":"alias-upstream","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["alias-e2e-model"]}}`, upstreamPort))waitForResponse(t, msgs, 5*time.Second)- writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":3,"method":"proxy:node/select","params":{"id":"alias-upstream"}}`)+ writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":3,"method":"ollama-proxy:node/select","params":{"id":"alias-upstream"}}`)waitForResponse(t, msgs, 5*time.Second)client := &http.Client{Timeout: 5 * time.Second}diff --git a/services/tests/scheduler_interop_test.go b/services/tests/scheduler_interop_test.goindex 102f44c..d494fce 100644--- a/services/tests/scheduler_interop_test.go+++ b/services/tests/scheduler_interop_test.go@@ -7,7 +7,7 @@// - Scheduler process tests drive workload, telemetry, staleness, and restart// baselines over stdin and inspect complete schedule:priority snapshots.// - TestProxySetPriorityViaBroker exercises the proxy's node/set-priority-// through the broker's proxy:<method> relay end-to-end.+// through the broker's ollama-proxy:<method> relay end-to-end.// - TestProxyConcurrentBurstDistribution holds real upstream requests open and// verifies the proxy balances before scheduler feedback can arrive.// - TestBrokerSpawnsScheduler confirms the broker comes up healthy with the@@ -352,7 +352,7 @@ func runBlockedProxyBurst(t *testing.T, pending, pressure []int, requests int) (}))}- stdin, msgs, stderr, cleanup := startBrokerWith(t, "--proxy-path", proxyBin)+ stdin, msgs, stderr, cleanup := startBrokerWith(t, "--proxy-path", proxyBin, "--proxy-engines", "ollama")t.Cleanup(cleanup)t.Cleanup(func() {unblock()@@ -370,7 +370,7 @@ func runBlockedProxyBurst(t *testing.T, pending, pressure []int, requests int) (requestID := 100for i, server := range servers {- callBrokerRPC(t, stdin, msgs, requestID, "proxy:node/add-manual", map[string]any{+ callBrokerRPC(t, stdin, msgs, requestID, "ollama-proxy:node/add-manual", map[string]any{"id": ids[i],"host": "127.0.0.1","port": portOfURL(t, server.URL),@@ -407,8 +407,8 @@ func runBlockedProxyBurst(t *testing.T, pending, pressure []int, requests int) (Rank: rank,})}- callBrokerRPC(t, stdin, msgs, requestID, "proxy:node/set-priority",- schedulerwire.Priority{Nodes: order, Ranks: ranks})+ callBrokerRPC(t, stdin, msgs, requestID, "ollama-proxy:node/set-priority",+ schedulerwire.Priority{Generation: 1, Nodes: order, Ranks: ranks})// Workload and request notifications can exceed the reader's buffer while// the upstreams are blocked. Drain them after setup so proxy writes never@@ -523,18 +523,20 @@ func assertBurstTotalsSkew(}// TestProxySetPriorityViaBroker: the proxy's node/set-priority is reachable-// through the broker's proxy:<method> relay and returns {count}.+// through the broker's ollama-proxy:<method> relay and returns {count}.func TestProxySetPriorityViaBroker(t *testing.T) {stdin, msgs, cleanup := startBrokerProc(t,"--scanner-path", scannerBin,- "--proxy-path", proxyBin,+ "--proxy-path", proxyBin, "--proxy-engines", "ollama",)t.Cleanup(cleanup)waitForMethod(t, msgs, "app:ready", 10*time.Second)waitProxyReady(t, stdin, msgs, 15*time.Second)- writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":70,"method":"proxy:node/set-priority","params":{"nodes":["alpha","beta","gamma"]}}`)+ // A generation is required: an unversioned snapshot would clear the+ // reservations without advancing the epoch, so the proxy rejects one.+ writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":70,"method":"ollama-proxy:node/set-priority","params":{"generation":1,"nodes":["alpha","beta","gamma"]}}`)resp := waitForResponse(t, msgs, 5*time.Second)var r struct {Count int `json:"count"`@@ -571,7 +573,7 @@ func TestLMStudioProxyIgnoresPriorityNodesAbsentFromDiscovery(t *testing.T) {realPort := portOfURL(t, realEngine.URL)stdin, msgs, stderr, cleanup := startBrokerWith(t,- "--lmstudio-proxy-path", lmstudioProxyBin,+ "--proxy-path", proxyBin, "--proxy-engines", "lmstudio",)t.Cleanup(cleanup)go func() {@@ -592,7 +594,7 @@ func TestLMStudioProxyIgnoresPriorityNodesAbsentFromDiscovery(t *testing.T) {// Priority puts a peer that never advertised LM Studio first. The proxy must// skip it and route to the discovered real-lm node.- writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":91,"method":"lmstudio-proxy:node/set-priority","params":{"nodes":["no-lm-peer","real-lm"]}}`)+ writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":91,"method":"lmstudio-proxy:node/set-priority","params":{"generation":1,"nodes":["no-lm-peer","real-lm"]}}`)if resp := waitForResponse(t, msgs, 5*time.Second); resp.Error != nil {t.Fatalf("lmstudio-proxy:node/set-priority rejected: %d %s", resp.Error.Code, resp.Error.Message)}diff --git a/services/tests/secure_inference_test.go b/services/tests/secure_inference_test.goindex cf4853a..a889839 100644--- a/services/tests/secure_inference_test.go+++ b/services/tests/secure_inference_test.go@@ -20,6 +20,7 @@ import ("testing""time"+ "nvpair-shared/engines""nvpair-shared/jsonrpc")@@ -60,11 +61,9 @@ type proxyProc struct {func startProxyProc(t *testing.T, clusterDir string, listenPort int) *proxyProc {t.Helper()cfg := t.TempDir()- cmd := exec.Command(proxyBin,- "--cluster-dir", clusterDir,- "--port", strconv.Itoa(listenPort),- "--ignore-persisted-port",- )+ // The engine and its port arrive over facade/enable below; only the cluster+ // dir is process-scoped enough to stay on argv.+ cmd := exec.Command(proxyBin, "--cluster-dir", clusterDir)cmd.Env = append(os.Environ(),"HOME="+cfg, "XDG_CONFIG_HOME="+cfg, "APPDATA="+cfg, "LOCALAPPDATA="+cfg,)@@ -82,14 +81,22 @@ func startProxyProc(t *testing.T, clusterDir string, listenPort int) *proxyProc}p := &proxyProc{t: t, cmd: cmd, stdin: stdin, msgs: startMsgReader(stdout), nextID: 1}- ready := p.pump(func(m jsonrpc.Message) bool { return m.Method == "ready" }, 15*time.Second)- var rp struct {- Port int `json:"port"`+ // The mTLS ingress under test is engine-agnostic; Ollama is an arbitrary+ // pick. The bound port comes from the enable response rather than the ready+ // notification, which is where the broker reads it too.+ resp := p.call("facade/enable", map[string]any{+ "engine": "ollama",+ "port": listenPort,+ "ignorePersistedPort": true,+ })+ var enabled struct {+ Engine string `json:"engine"`+ Port int `json:"port"`}- if err := json.Unmarshal(ready.Params, &rp); err != nil || rp.Port == 0 {- t.Fatalf("proxy ready params = %s (err %v)", ready.Params, err)+ if err := json.Unmarshal(resp.Result, &enabled); err != nil || enabled.Port == 0 {+ t.Fatalf("facade/enable result = %s (err %v)", resp.Result, err)}- p.port = rp.Port+ p.port = enabled.Portreturn p}@@ -163,10 +170,10 @@ func (p *proxyProc) notify(method string, params any) {}}-// setLocalBackend points the proxy's cluster ingress + local self candidate at a-// loopback engine.+// setLocalBackend points one facade's cluster ingress + local self candidate at+// a loopback engine. Addressed to that engine, the way the broker sends it.func (p *proxyProc) setLocalBackend(engine, host string, port int, healthy bool) {- p.call("node/set-local-backend", map[string]any{+ p.call(engines.AddressMethod(engine, "node/set-local-backend"), map[string]any{"engine": engine, "host": host, "port": port, "healthy": healthy,})}@@ -175,7 +182,7 @@ func (p *proxyProc) setLocalBackend(engine, host string, port int, healthy bool)// remote peer that runs ollama at proxyPort, tagged as a trusted cluster member// keyed by clusterUUID (the peer's cluster cert principal).func (p *proxyProc) pushOllamaPeer(name, ip, clusterUUID string, proxyPort int, models []string) {- p.notify("discovery:nodes", map[string]any{+ p.notify(engines.AddressMethod("ollama", "discovery:nodes"), map[string]any{"nodes": []map[string]any{{"hostUuid": name,"name": name,@@ -196,7 +203,7 @@ func (p *proxyProc) waitForRoutableNode(t *testing.T) {t.Helper()deadline := time.Now().Add(5 * time.Second)for time.Now().Before(deadline) {- resp := p.call("nodes/list", nil)+ resp := p.call(engines.AddressMethod("ollama", "nodes/list"), nil)var r struct {Nodes []json.RawMessage `json:"nodes"`}diff --git a/services/tests/workload_identity_interop_test.go b/services/tests/workload_identity_interop_test.goindex f3879a7..c89eebe 100644--- a/services/tests/workload_identity_interop_test.go+++ b/services/tests/workload_identity_interop_test.go@@ -66,8 +66,8 @@ func TestWorkloadCrossEngineIdentityDistinct(t *testing.T) {lmstudioPort := portOfURL(t, lmstudio.URL)stdin, msgs, _, cleanup := startBrokerWith(t,+ // Both engines fronted, which is the default."--proxy-path", proxyBin,- "--lmstudio-proxy-path", lmstudioProxyBin,"--workload-manager-path", workloadMgrBin,)t.Cleanup(cleanup)@@ -89,9 +89,9 @@ func TestWorkloadCrossEngineIdentityDistinct(t *testing.T) {// Pin each fake engine into its proxy so inference routes deterministically.// These are stdio control notifications, not HTTP hits on the proxy port,// so the first HTTP request each proxy serves is the inference below → id "1".- writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":50,"method":"proxy:node/add-manual","params":{"id":"fake-ollama","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["crossengine-model"]}}`, ollamaPort))+ writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":50,"method":"ollama-proxy:node/add-manual","params":{"id":"fake-ollama","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["crossengine-model"]}}`, ollamaPort))waitForResponse(t, msgs, 5*time.Second)- writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":51,"method":"proxy:node/select","params":{"id":"fake-ollama"}}`)+ writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":51,"method":"ollama-proxy:node/select","params":{"id":"fake-ollama"}}`)waitForResponse(t, msgs, 5*time.Second)writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":52,"method":"lmstudio-proxy:node/add-manual","params":{"id":"fake-lmstudio","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["crossengine-model"]}}`, lmstudioPort))waitForResponse(t, msgs, 5*time.Second)@@ -192,7 +192,7 @@ func TestWorkloadManagerRehydratesActiveWorkloadOnRestart(t *testing.T) {received := fx.startStubClusterPeer(t, "rehydrate-wm-peer", "rehydrate-peer-uuid")stdin, msgs, stderr, cleanup := startBrokerWithDirs(t, t.TempDir(), fx.nodeDir,- "--proxy-path", proxyBin,+ "--proxy-path", proxyBin, "--proxy-engines", "ollama","--workload-manager-path", workloadMgrBin,)t.Cleanup(cleanup)@@ -223,9 +223,9 @@ func TestWorkloadManagerRehydratesActiveWorkloadOnRestart(t *testing.T) {// Pin the fake ollama, then fire one inference in the background — the// backend blocks, so the workload stays running until teardown.- writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":50,"method":"proxy:node/add-manual","params":{"id":"fake-ollama","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["rehydrate-model"]}}`, ollamaPort))+ writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":50,"method":"ollama-proxy:node/add-manual","params":{"id":"fake-ollama","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["rehydrate-model"]}}`, ollamaPort))waitForResponse(t, msgs, 5*time.Second)- writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":51,"method":"proxy:node/select","params":{"id":"fake-ollama"}}`)+ writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":51,"method":"ollama-proxy:node/select","params":{"id":"fake-ollama"}}`)waitForResponse(t, msgs, 5*time.Second)go func() {c := &http.Client{Timeout: 120 * time.Second}@@ -293,7 +293,7 @@ func TestWorkloadManagerRehydratesRecentTerminalOnRestart(t *testing.T) {received := fx.startStubClusterPeer(t, "rehydrate-term-peer", "rehydrate-term-peer-uuid")stdin, msgs, stderr, cleanup := startBrokerWithDirs(t, t.TempDir(), fx.nodeDir,- "--proxy-path", proxyBin,+ "--proxy-path", proxyBin, "--proxy-engines", "ollama","--workload-manager-path", workloadMgrBin,)t.Cleanup(cleanup)@@ -316,9 +316,9 @@ func TestWorkloadManagerRehydratesRecentTerminalOnRestart(t *testing.T) {proxyPort := waitProxyReady(t, stdin, msgs, 15*time.Second)pid1 := awaitInt(t, wmPids, 10*time.Second, "initial workload-manager start")- writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":50,"method":"proxy:node/add-manual","params":{"id":"fake-ollama","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["rehydrate-term-model"]}}`, ollamaPort))+ writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":50,"method":"ollama-proxy:node/add-manual","params":{"id":"fake-ollama","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["rehydrate-term-model"]}}`, ollamaPort))waitForResponse(t, msgs, 5*time.Second)- writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":51,"method":"proxy:node/select","params":{"id":"fake-ollama"}}`)+ writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":51,"method":"ollama-proxy:node/select","params":{"id":"fake-ollama"}}`)waitForResponse(t, msgs, 5*time.Second)// Fire one inference; it completes promptly → terminal workload.@@ -360,39 +360,6 @@ func TestWorkloadManagerRehydratesRecentTerminalOnRestart(t *testing.T) {var wmStartedPidRe = regexp.MustCompile(`workload-manager started.*\bpid=(\d+)`)-// waitLMStudioProxyReady polls lmstudio-proxy:get-status until the LM Studio-// proxy reports ready and returns its bound port (mirrors waitProxyReady).-func waitLMStudioProxyReady(t *testing.T, stdin io.Writer, msgs <-chan jsonrpc.Message, timeout time.Duration) int {- t.Helper()- id := 9500- writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":%d,"method":"lmstudio-proxy:get-status"}`, id))- deadline := time.After(timeout)- tick := time.NewTicker(500 * time.Millisecond)- defer tick.Stop()- for {- select {- case msg, ok := <-msgs:- if !ok {- t.Fatal("broker stream closed waiting for lmstudio-proxy:get-status")- }- if msg.ID != nil && msg.Method == "" {- var st struct {- Ready bool `json:"ready"`- Port int `json:"port"`- }- if json.Unmarshal(msg.Result, &st) == nil && st.Ready {- return st.Port- }- }- case <-tick.C:- id++- writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":%d,"method":"lmstudio-proxy:get-status"}`, id))- case <-deadline:- t.Fatalf("timed out (%s) waiting for lmstudio-proxy to become ready", timeout)- }- }-}-// waitStubPeerWorkload blocks until the stub peer records a workload:* frame// for the given model in the given state.func waitStubPeerWorkload(t *testing.T, received <-chan jsonrpc.Message, model, state string, timeout time.Duration) {diff --git a/services/tests/workload_interop_test.go b/services/tests/workload_interop_test.goindex b1593d5..1e72686 100644--- a/services/tests/workload_interop_test.go+++ b/services/tests/workload_interop_test.go@@ -32,6 +32,7 @@ import ("os/exec""strconv""strings"+ "sync/atomic""testing""time"@@ -367,7 +368,7 @@ func TestWorkloadManagerOutboundBroadcast(t *testing.T) {stdin, msgs, cleanup := startBrokerProcInCluster(t, fx.nodeDir,"--scanner-path", scannerBin,- "--proxy-path", proxyBin,+ "--proxy-path", proxyBin, "--proxy-engines", "ollama","--workload-manager-path", workloadMgrBin,)t.Cleanup(cleanup)@@ -375,15 +376,15 @@ func TestWorkloadManagerOutboundBroadcast(t *testing.T) {waitForMethod(t, msgs, "app:ready", 10*time.Second)t.Log("broker ready")- // The proxy reports its bound port asynchronously; poll proxy:get-status.+ // The proxy reports its bound port asynchronously; poll ollama-proxy:get-status.proxyPort := waitProxyReady(t, stdin, msgs, 15*time.Second)t.Logf("proxy ready on port %d", proxyPort)// Register the fake Ollama as a manual node and pin it as the target so// the proxy forwards there deterministically.- writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":50,"method":"proxy:node/add-manual","params":{"id":"fake-ollama","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["test-model"]}}`, ollamaPort))+ writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":50,"method":"ollama-proxy:node/add-manual","params":{"id":"fake-ollama","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["test-model"]}}`, ollamaPort))waitForResponse(t, msgs, 5*time.Second)- writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":51,"method":"proxy:node/select","params":{"id":"fake-ollama"}}`)+ writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":51,"method":"ollama-proxy:node/select","params":{"id":"fake-ollama"}}`)waitForResponse(t, msgs, 5*time.Second)t.Log("fake ollama pinned as proxy target")@@ -465,12 +466,42 @@ func postPeerEvent(t *testing.T, client *http.Client, endpoint, body string, tim}}-// waitProxyReady polls proxy:get-status until the proxy reports ready and-// returns its bound port.+// waitProxyReady polls ollama-proxy:get-status until the Ollama proxy reports ready+// and returns its bound port.func waitProxyReady(t *testing.T, stdin io.Writer, msgs <-chan jsonrpc.Message, timeout time.Duration) int {t.Helper()- id := 9000- writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":%d,"method":"proxy:get-status"}`, id))+ return waitEngineProxyReady(t, "ollama-proxy", stdin, msgs, timeout)+}++// waitLMStudioProxyReady is waitProxyReady for the LM Studio relay namespace.+func waitLMStudioProxyReady(t *testing.T, stdin io.Writer, msgs <-chan jsonrpc.Message, timeout time.Duration) int {+ t.Helper()+ return waitEngineProxyReady(t, "lmstudio-proxy", stdin, msgs, timeout)+}++// engineReadyPollSeq hands every readiness poll a distinct JSON-RPC id. Based+// at 9000 to stay clear of the fixed ids other helpers on this stream use.+var engineReadyPollSeq atomic.Int64++// waitEngineProxyReady polls <namespace>:get-status until that proxy reports+// ready and returns its bound port. The two engines differ only by relay+// namespace, so they share one implementation.+func waitEngineProxyReady(t *testing.T, namespace string, stdin io.Writer, msgs <-chan jsonrpc.Message, timeout time.Duration) int {+ t.Helper()+ method := namespace + ":get-status"+ // Accept only replies to this invocation's own polls. One process hosts+ // both facades and answers get-status for each on this one stream, so a+ // poll's reply can still be in flight when the next wait begins — and+ // every one of them looks alike: a response frame carrying ready and port.+ // Matching on shape alone returns the other engine's port, which is+ // exactly what the tests waiting on both facades assert is different.+ pending := map[int64]bool{}+ poll := func() {+ id := 9000 + engineReadyPollSeq.Add(1)+ pending[id] = true+ writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":%d,"method":%q}`, id, method))+ }+ poll()deadline := time.After(timeout)tick := time.NewTicker(500 * time.Millisecond)defer tick.Stop()@@ -478,22 +509,27 @@ func waitProxyReady(t *testing.T, stdin io.Writer, msgs <-chan jsonrpc.Message,select {case msg, ok := <-msgs:if !ok {- t.Fatal("broker stream closed waiting for proxy:get-status")+ t.Fatalf("broker stream closed waiting for %s", method)+ }+ if msg.ID == nil || msg.Method != "" {+ continue+ }+ var id int64+ if json.Unmarshal(*msg.ID, &id) != nil || !pending[id] {+ continue+ }+ delete(pending, id)+ var st struct {+ Ready bool `json:"ready"`+ Port int `json:"port"`}- if msg.ID != nil && msg.Method == "" {- var st struct {- Ready bool `json:"ready"`- Port int `json:"port"`- }- if json.Unmarshal(msg.Result, &st) == nil && st.Ready {- return st.Port- }+ if json.Unmarshal(msg.Result, &st) == nil && st.Ready {+ return st.Port}case <-tick.C:- id++- writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":%d,"method":"proxy:get-status"}`, id))+ poll()case <-deadline:- t.Fatalf("timed out (%s) waiting for proxy to become ready", timeout)+ t.Fatalf("timed out (%s) waiting for %s proxy to become ready", timeout, namespace)}}}diff --git a/services/versions.json b/services/versions.jsonindex 29d8c23..0a130f7 100644--- a/services/versions.json+++ b/services/versions.json@@ -1,20 +1,19 @@{"$comment": "Single source of truth for all version numbers. See VERSIONING.md for bump rules.",- "product": "0.91.7",- "installer": "0.91.7",+ "product": "1.0.0",+ "installer": "1.0.0","components": {- "ollama-proxy": "0.26.2",- "lmstudio-proxy": "0.16.2",+ "nvpair-proxy": "1.0.0","nvpair-node-info": "0.13.3","nvpair-node-scanner": "0.20.3","nvpair-manual-nodes": "0.11.1","nvpair-workload-manager": "0.13.3","nvpair-errors": "0.7.4","nvpair-node-settings": "1.0.4",- "nvpair-ui-broker": "0.40.2",+ "nvpair-ui-broker": "1.0.0","nvpair-engine-manager": "0.17.4","nvpair-cluster-manager": "1.1.4",- "nvpair-job-scheduler": "0.4.1",- "nvpair-tui": "0.7.2"+ "nvpair-job-scheduler": "0.4.2",+ "nvpair-tui": "0.7.3"}}