Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript
16.6 KB
Raw Blame

Adding an LLM provider

This guide explains how to add a new LLM provider to Clerum. There are two paths:

  • Data-only (OpenAI-compatible) — the provider exposes an OpenAI-compatible /chat/completions endpoint reachable with a base URL + Authorization: Bearer API key. No driver code. Most of the industry (Groq, DeepSeek, Mistral, xAI, Together, Fireworks, OpenRouter, Perplexity, Moonshot, Cerebras, DeepInfra, Nebius, Novita, Google Gemini via its compat endpoint, …) falls here.
  • Driver-based — a bespoke wire protocol (Anthropic, Vertex, Bedrock) or non-vanilla auth/host (Azure OpenAI: per-resource host + api-key header + deployment-name-as-model). Needs a makeProvider arm and possibly an SDK.

The single source of truth

The canonical provider set lives in packages/llm-providers (index.cjs + index.d.ts). It is a DATA-ONLY leaf consumed by mcp-host, control-api, WRC, HCC and control-ui — so the provider id, its credential slots, its display label and its non-secret env are defined once here and everyone inherits them. Never reintroduce a hardcoded provider list anywhere else.

Runtime-only fields (base URL, default model, tokenizer, how to build the client) stay local to mcp-host/src/llm/registryCore.ts (RUNTIME_FIELDS) and mcp-host/src/llm/registry.ts (makeProvider).


Path A — OpenAI-compatible provider (data-only)

Example: adding groq.

1. Shared package — packages/llm-providers/

  • index.cjs: add the id to PROVIDER_IDS, a credential slot to PROVIDER_CREDENTIAL_SLOTS (use the apiKeySlot('<id>-api-key', '<ID>_API_KEY') helper), a brand label to PROVIDER_DISPLAY_LABELS, and an empty PROVIDER_NON_SECRET_ENV entry (fixed base URL → no per-Host config).
  • index.d.ts: append the id to the PROVIDER_IDS readonly tuple (the LlmProviderId union derives from it).

2. mcp-host runtime row — registryCore.ts RUNTIME_FIELDS

Add { baseURL, defaultModel, tokenizer: 'fallback' }. The data-driven arm in makeProvider picks up anything carrying a baseURL — no driver needed.

  • baseURL must be exact so ${baseURL}/chat/completions resolves. Watch the path: some include /v1 (https://api.mistral.ai/v1), some don't (https://api.perplexity.ai), Fireworks needs an id prefix, DeepInfra is /v1/openai. See the gotchas table below.
  • tokenizer: 'fallback' for non-OpenAI models ('openai' only for GPT-family).

3. CRD enums — charts/clerum-crds/crds/

  • host.yaml: add the id to the spec.model.provider enum and the spec.llmPolicy.fallbacks[].provider enum.
  • workflowrecipe.yaml: add it to both provider enums iff the provider is single-credential with no required non-secret env (all OpenAI-compatible api-key providers qualify). These enums are additive — old resources stay valid.

4. Allowlist seed — control-api migration (control-api/src/db.ts)

Append a new migration (next NNNN_… version, append-only + idempotent) that INSERT … ON CONFLICT DO NOTHINGs one sensible default (provider, model, vendor) into llm_allowed_models. Operators curate the rest from /llm-models.

5. Image-input capability — llm_allowed_models.image_input (#654)

Declare image support per exact (provider, model) pair, never from the model name, family, or an OpenAI-compatible request shape. Do it in one of two ways:

  • Seed it in the same migration as step 4 for ids the provider's own documentation covers. Keep it to exact ids (no family extrapolation), reference the public https:// doc page, and set checkedAt to the date the doc was read — never the deploy date, so re-running the migration cannot rejuvenate evidence.
  • Let the operator curate it. The catalog sync fills the column from models.dev first; the operator overrides only where the vendor documentation disagrees, through the admin API — POST /admin/llm-models or PUT /admin/llm-models/:id with the image_input field. The Control UI /llm-models page has no editor for this field. A row with no metadata reads as unknown: text keeps working, images are refused with the model named.

The seed is one idempotent statement. Curated evidence replaces discovery evidence; once curated evidence is in place a re-run changes nothing:

UPDATE llm_allowed_models
   SET image_input = $1::jsonb
 WHERE provider = $2 AND model = $3
   AND (image_input IS NULL OR image_input->'evidence'->>'source' = 'discovery')

Shape (validated by @clerum/llm-providers parseImageInputCapability; GET /admin/llm-models always returns it normalized):

{
  "state": "supported",
  "evidence": {
    "source": "curated",
    "reference": "https://docs.example.com/models/vision-flash",
    "checkedAt": "2026-09-16T00:00:00.000Z"
  }
}

A supported/unsupported claim REQUIRES evidence; a discovery-sourced claim also requires validUntil. null clears (and normalizes to unknown), and renaming a row's provider/model clears it automatically. The catalog sync derives supported / unsupported from models.dev modalities.input (image present / absent) and records unknown when the entry has no usable modalities; attachment is not consulted. Discovery evidence never overwrites curated evidence and never overwrites newer discovery evidence — see discoveryImageInput in control-api/src/services/llmCatalogSync.ts.

What operators still curate:

  1. ids models.dev does not list — 15 of the 44 pairs the allowlist seeds today (for example zai:glm-5, zai:glm-5.1, the Bedrock Claude ids, openai:gpt-5.1-codex, openrouter:anthropic/claude-sonnet-latest). After a deploy, the sync that runs shortly after control-api starts fills image_input for the 29 listed pairs; the unlisted ones stay unknown (images refused, text unaffected) until an operator curates them with PUT /admin/llm-models/:id and an image_input field;
  2. models where the vendor documentation disagrees with models.dev (curated evidence wins and is never overwritten);
  3. nothing on installations that run the cron — but without it and without a sync inside the TTL the evidence expires, and running a sync restores it.

Transport registration (mcp-host)

Evidence is necessary, not sufficient. mcp-host/src/llm/imageInput.ts maps every provider id to a wire family (EXPLICIT_DRIVER_FAMILY, imageWireFamilyFor), and TRANSPORT_SUPPORT declares, per family and per dispatch method, whether the serializer carries image parts. A provider whose family is not registered (or registered false) resolves to transport_unsupported even with curated supported evidence. Path A providers get the openai-compatible family automatically when their descriptor carries a baseURL; without one they resolve to unregistered. Path B drivers must add an EXPLICIT_DRIVER_FAMILY entry and an imageInputTransport.test.ts case for each method they implement.

6. control-ui — control-ui/lib/llm.ts

Add the provider's default model to LLM_DEFAULT_MODEL_BY_PROVIDER (a Record<LlmProvider, …>, so it must be exhaustive — tsc enforces it). The provider dropdown (LLM_PROVIDER_OPTIONS), the multi-slot secrets form (LLM_CREDENTIAL_GROUPS) and the field label all auto-derive from the package — no other UI change. A nicer SECRET_FIELD_HINTS placeholder is optional.

7. Param compatibility (important)

The OpenAI-compatible driver (mcp-host/src/llm/openaiCompatible.ts) shares the openai provider's request shape. Some providers reject params: Groq (n>1, logprobs, logit_bias), Cerebras (frequency_penalty/presence_penalty), Moonshot (tool_choice: 'required', temperature outside [0,1]), DeepSeek reasoner (sampling params). Keep the default request to a conservative, portable set (model + messages + tools). If a provider needs something special, gate it per-provider — don't broaden the shared default.

8. Tests

mcp-host registry tests (provider present + constructs), the control-api migration/enum test, and confirm the credential-exclusion leak test still passes with the expanded slot set (it derives from ALL_PROVIDERS, so it should).


Path B — driver-based provider

Everything in Path A, plus:

makeProvider arm — registry.ts

Add an explicit case '<id>': that builds the provider. Load any SDK lazily with a synchronous require() inside the arm (mcp-host is CommonJS) so the SDK is only parsed when the provider is actually used, and a broken dependency can't take down startup. Never route a driver provider through the baseURL arm. Register the driver's image wire family (step 5, Transport registration) or images are refused for it.

Credentials & non-secret env

  • Multi-credential providers (Bedrock: key pair; Vertex: SA JSON) list every slot in PROVIDER_CREDENTIAL_SLOTS. The credential-exclusion boundary (userEnvSnapshot) derives from the full slot set — never primarySlot().
  • Non-secret, per-Host config (region, project, endpoint) goes in PROVIDER_NON_SECRET_ENV, surfaced in the UI as a "configure in Host → Environment" hint, never a secret field. It is not in the pod env: mcp-host resolves it from the watched per-Host host-<ref>-env / -env-secret and passes the driver a lookup (ProviderEnvLookup, src/llm/providerEnv.ts) limited to the provider's declared names (process.env outside a Host pod, e.g. dev mode). Read these values through that lookup, never process.env directly.
  • Exclude such providers from workflowrecipe.yaml — the WRC configure transport is single-credential and doesn't carry non-secret env, so a multi-credential / endpoint-requiring provider must fail at admission, not mid-workflow (this is why bedrock and azure are host-only).

Azure OpenAI — the reference non-vanilla case

Azure is OpenAI-compatible in shape but not a baseURL + Bearer row:

  • No fixed base URL — the operator supplies AZURE_OPENAI_ENDPOINT (non-secret env). The driver builds ${endpoint}/openai/v1/ (v1 GA path).
  • Auth is the api-key header, not Authorization: Bearer.
  • model is the deployment name, not a catalog id. Its RUNTIME_FIELDS.azure carries a defaultModel + tokenizer: 'openai' but no static baseURL, so it takes the explicit case 'azure': arm and reads the endpoint from the non-secret env lookup at construction. Fail-closed if the endpoint is absent.

Gotchas encoded in the registry (2026)

provider base_url quirk param / behavior gotcha
fireworks model id needs prefix accounts/fireworks/models/<name> —
deepinfra base path is /v1/openai (not /v1) tool-calling only on ~most models
perplexity no /v1 in base URL every call is web-grounded (returns citations)
deepseek accepts with or without /v1 reasoner mode restricts sampling params
moonshot .ai (global) vs .cn (China) host no tool_choice: 'required'; temperature [0,1]
groq — rejects n>1, logprobs, logit_bias
cerebras — rejects frequency_penalty/presence_penalty
openrouter — router: same slug may hit different upstreams; tool support per-model
nebius two brands mid-rename (tokenfactory / studio) org/model namespaced ids
together / novita / nebius — org/model namespaced ids
gemini (compat) …/v1beta/openai/ beta: unsupported params silently ignored; distinct from vertex
minimax .io (international) vs api.minimaxi.com (China) host JWT-shaped api key; portable [0,2] temperature (no gating)
azure per-resource endpoint (env) api-key header; model = deployment name

Hand-maintained mirrors — not type-checked

Several scripts and docs enumerate the provider list by hand. tsc does not guard them, so run these two greps and update every hit:

# 1. every surface that names the current last OpenAI-compatible provider must also name yours
grep -rn --include='*.sh' --include='*.md' --include='*.ts' --include='*.tsx' \
  --include=Makefile -e 'novita' evenfire/ | grep -vi node_modules
# 2. the old provider count in prose — substitute the PRE-bump number for <N>
#    (this repo went 21 -> 22; the next add greps for 22, and so on). `the other
#    <N>` catches the placeholder-fallback sub-count in apply-llm-secret.sh.
grep -rn --include='*.sh' --include='*.md' --include='*.ts' --include='*.tsx' \
  --include=Makefile -e '<N> providers' -e 'one of <N>' -e 'all <N>' -e 'the other <N>' evenfire/

Surfaces to update:

  • scripts/minikube/apply-llm-secret.sh — the SLOTS array (<id>-api-key|<ID>_API_KEY|).
  • scripts/check-prereqs.sh — the PROVIDER_KEYS list.
  • scripts/e2e/e2e-plugin-workload-sdk.sh, scripts/e2e/seed-e2e-data.sh — the provider→credential-slot case arms (a <id>) arm per single-key provider).
  • README.md (repo root) — the provider table.
  • docs/meta/claims-guardrails.md — the provider table.
  • docs/crds/workflowrecipe.md — the documented provider enum.
  • docs/llms.txt, Makefile — provider-count comments.

Checklist

Type-checked source of truth (tsc fails if missed):

  • packages/llm-providers/index.cjs — id, slot(s), label, non-secret env
  • packages/llm-providers/index.d.ts — id in the PROVIDER_IDS tuple
  • mcp-host/src/llm/registryCore.ts — RUNTIME_FIELDS row (baseURL/defaultModel/tokenizer)
  • mcp-host/src/llm/registry.ts — makeProvider arm (driver path only)
  • control-api/src/services/modelsDevClient.ts — PROVIDER_KEY_MAP entry (our id → models.dev key; verify against live api.json)
  • control-ui/lib/llm.ts — LLM_DEFAULT_MODEL_BY_PROVIDER entry

Data / assets (not type-checked, but required for the provider to work end-to-end):

  • control-api/src/db.ts — allowlist seed migration (append-only, idempotent)
  • control-api/src/db.ts — curated image_input evidence only for ids that models.dev does not list, or where the vendor document contradicts models.dev (exact ids, checkedAt = the date the doc was read)
  • control-api/src/data/modelsDevSnapshot.ts — vendored offline snapshot block, derived from live models.dev (the offline catalog-sync fallback; keys must match PROVIDER_KEY_MAP)
  • control-api/src/data/modelsDevSnapshot.ts — bump VENDORED_MODELS_DEV_SNAPSHOT_CAPTURED_AT to the regeneration commit's date (git log -1 --format=%cI -- <file>); discovery provenance is stamped with it, so leaving it stale misdates evidence
  • charts/clerum-crds/crds/host.yaml — both provider enums
  • charts/clerum-crds/crds/workflowrecipe.yaml — enums (single-credential only)
  • control-ui/public/provider-icons/<id>.svg — brand icon (the picker renders /provider-icons/${provider}.svg; degrades to the label initial if absent). Add a theme-knockout rule in control-ui/app/globals.css only if the mark is monochrome.

Hand-maintained mirrors (run both greps above, then):

  • scripts/minikube/apply-llm-secret.sh · scripts/check-prereqs.sh · both scripts/e2e/*.sh case arms
  • README.md · docs/meta/claims-guardrails.md · docs/crds/workflowrecipe.md · docs/llms.txt · Makefile
  • operator doc updated (docs/deploy/llm-providers.md) and overview (docs/llm-providers/README.md) — counts + tables

Verify: param-compat reviewed; tests updated (every count-bearing assertion); tsc --noEmit clean across services; the two drift greps return only intended hits.