Skip to content

Router that virtually distributes inference across connected devices in the home.

Go
5.4 KB
Raw Blame
---
description: Backend-owned engine model catalogue (Ollama committed list + LM Studio and llama.cpp Hugging Face catalogs)
alwaysApply: true
---
<!--
SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
-->

# Model Registry — engine model catalogue

- The "Add Model" browse is served by **`nvpair-engine-manager`** over the
  **`engine:catalog`** JSON-RPC method (`services/nvpair-engine-manager/catalog.go`).
  It lives in the backend because both front ends need it: the desktop app's
  Add Model modal and the terminal interface's model browser. There is **no
  shared GGUF registry** — each enabled engine has exactly one curated source,
  and only llama.cpp's can be searched.
- **Ollama**: a **locked, committed list**, not a live scrape.
  `services/nvpair-engine-manager/catalog/ollama-models.json` (wire shape
  `{ scrapedAt, source, count, models: [...] }`) is compiled in with `go:embed`,
  so the catalogue is served with **no network access at runtime**. Devs
  regenerate it with `npm run scrape:ollama-models`
  (`desktop/scripts/scrape-ollama-models.ts`) when Ollama's catalog changes,
  then review the diff and commit. The scraper is the **only** place Ollama
  scraping lives: two phases — (A) parse the `https://ollama.com/library` index
  (anchored on each card's `<a class="group w-full">` link, capability chips are
  `text-indigo-600`, size chips `text-blue-600`, updated timestamp in the stats
  span `title`), then (B) fetch each `https://ollama.com/library/<base>/tags`
  page (concurrency 4, anchored on the self-contained mobile card
  `class="md:hidden"`) for per-tag digest + binary byte size + context window;
  falls back to `https://ollama.com/api/tags` only if Phase A fails. Cloud-only
  variants are filtered. Names are used as-is for `ollama pull`. **When Ollama
  changes their markup, update the parser in the script and re-run it** — the
  running app is unaffected.
- **LM Studio**: a **live fetch**. GETs
  `https://huggingface.co/api/models?author=lmstudio-community&sort=downloads&direction=-1&limit=500`
  and normalizes each repo to a pull-ready id (e.g.
  `lmstudio-community/Qwen3-8B-GGUF`, accepted by `lms get`).
- **llama.cpp**: a **live fetch** of the top 50 GGUF repositories for each
  approved publisher (`ggml-org`, `bartowski`, and `unsloth`). It is the one
  **searchable** source: `engine:catalog` takes an optional `query`, which
  searches up to 50 matches across all public publishers, each query cached on
  its own (the 20 most recently used are kept). Both paths reject unsafe,
  private, gated, non-generative, and non-pull-ready repos. Every result is an
  exact `owner/repository:Q4_K_M` pull id; other quantizations are not listed.
  The reply says `searchable: true`; other sources ignore a query and return
  their whole list. **Never log the user's search text.**
- The LM Studio list, the llama.cpp list, and each llama.cpp query share one
  cache: 6-hour TTL, a coalescing guard so concurrent callers share one request,
  and a failure backoff so a dead upstream is not re-dialled on every call.
- All sources normalize to `CatalogModel`, which the desktop maps to the shared
  `EngineHubModel` (`desktop/src/shared/types/engine-api.ts`).
- **The catalogue is filtered for the machine the models will install on**, not
  the one serving it. MLX quantizations only install on Apple Silicon — not on
  an Intel Mac — so `engine:catalog` takes an optional `platform` and `arch`
  (GOOS and GOARCH; omitting both means the server's own), marks Apple-only rows
  with `appleOnly`, keeps them only for `darwin`/`arm64`, and echoes the target
  it filtered for. A named platform without an arch leaves the arch unknown
  rather than borrowing the server's. A client driving a peer should say which
  peer; a client that cannot determine it must tell the operator which machine
  the list applies to rather than presenting it as universal.
- **Frame size matters here.** The Ollama reply is a single ~1.9 MiB JSON-RPC
  line. Every hop on its path shares `jsonrpc.WorkerFrameBytes`, because an
  over-long line is a terminal read error that silently closes the peer while
  the child keeps running. A test in `catalog_test.go` fails if the marshalled
  catalogue outgrows the frame; **filter or paginate rather than raising the cap
  again**.
- The desktop relays through
  `desktop/src/electron/service-bridge/model-catalog.ts`, which exposes
  `getEngineHubModels(engineType, query?)` and `warmEngineHubs()` (warms the
  LM Studio and llama.cpp catalogs — Ollama needs none). The `engine:search-hub`
  handler in `empty-handlers.ts` calls the former; the `overview:ready` handler
  in `window.ipc.ts` calls the latter. Warm on renderer-ready, **not** on
  service connect: a catalog fetch started before the window has painted
  competes with the renderer's own load, and a hanging one leaves an unpainted
  window behind.
- Both front ends fetch an engine's full catalog once (cached per-engine) and
  filter **locally** as the user types. The desktop renderer
  (`src/ui/components/ModelHub/`) sorts client-side by Updated / Name / Size;
  for llama.cpp, Enter/Search sends an explicit query and merges the response
  with matching listed rows. The terminal interface's browser sends `/` as a
  query for a searchable source and shows its results. Pulling goes through the
  engine manager (`engine:pull-model` → `pull_model`); the catalogue only
  produces the pull-ready id.