Skip to content

OpenRAG is a comprehensive, single package Retrieval-Augmented Generation platform built on Langflow, Docling, and Opensearch.

Python
9.6 KB
Raw Blame
# Which model providers OpenRAG offers, per run mode.
#
# This file is the single source of truth for provider visibility in the
# Settings provider cards, the Onboarding tabs, and the model catalogue that
# feeds every agent/ingest picker. The backend filters on it; the UI renders
# whatever the API returns, so there is no second denylist in the frontend.
#
# Keys under `modes` match OPENRAG_RUN_MODE: oss | on_prem | saas.
#   true  -> the provider is shown in that mode
#   false -> the provider is hidden
# A missing mode key means hidden, so adding a new run mode can never expose a
# provider by accident.
#
# A provider that is not listed here is not offered anywhere in the UI. A model
# id for an unlisted provider can still be typed by hand and will be routed if
# the gateway already supports it.
#
# Set OPENRAG_MODEL_PROVIDERS_CONFIG to another YAML file to override this list
# for a single deployment; this file is the default.
#
# A row may also declare `models` / `embedding_models`. LiteLLM's bundled table
# already knows the public vendors, so those rows need nothing. A self-hosted
# OpenAI-compatible gateway serves whatever its operator deployed, and those ids
# can only come from here — see the commented example at the end of this file.
#
# `exclude_models` keeps ids out of the pickers. `*` and `?` wildcards work, and
# matching is against the id as the picker shows it (`gpt-3.5-turbo`, not
# `openai/gpt-3.5-turbo`). Exclusion is applied last, so it also overrides a
# `models` row on the same provider. Models the provider has already retired are
# dropped without being listed here, and ones retiring within 90 days are hidden
# behind a "Show N retiring" row in the picker — use this for the ones a
# deployment does not want offered at all.

providers:
  - name: openai
    display_name: OpenAI
    modes:
      oss: true
      on_prem: true
      saas: true
    exclude_models:
      - gpt-3.5-* # a whole generation, superseded and near retirement
      # Refuses function tools on /v1/chat/completions. Both generations, and
      # the trailing `*` for the dated snapshots (`gpt-5.6-luna-2026-07-09`):
      # an exact pattern matches the undated id only, so a snapshot released
      # later reappears in the picker on its own.
      #
      # LiteLLM's table reports `supports_function_calling: true` for every
      # luna row, including the 5.6 that does not — which is why the exclusion
      # is hand-maintained here rather than read off the catalogue. Treat a new
      # luna generation as defective until someone proves otherwise with a real
      # tool call: an id excluded here can still be typed by hand, so hiding a
      # working model costs less than offering a broken one.
      - gpt-5.6-luna*
      - gpt-6-luna*

  - name: ollama
    display_name: Ollama
    modes:
      oss: true
      on_prem: false
      saas: false # local inference is not offered in SaaS

  - name: watsonx
    display_name: IBM watsonx.ai
    modes:
      oss: true
      on_prem: true
      saas: true
    exclude_models:
      # LiteLLM ids that do not match the API ids in IBM's model documentation.
      - bigscience/mt0-xxl-13b
      - google/flan-t5-xl-3b
      - ibm/granite-guardian-3-2-2b
      - ibm/granite-guardian-3-3-8b
      - meta-llama/llama-4-maverick-17b
      - mistralai/mistral-small-2503
      - mistralai/pixtral-12b-2409
      # Models reported unusable against watsonx.ai in tracker issue 82237.
      - mistralai/mistral-medium-2505 # fails during streamed chat responses
      - ibm/granite-3-3-8b-instruct # unsupported or removed by watsonx.ai
      - ibm/granite-3-8b-instruct
      - meta-llama/llama-3-2-90b-vision-instruct
      - meta-llama/llama-3-2-11b-vision-instruct
      - meta-llama/llama-3-2-3b-instruct
      - meta-llama/llama-3-2-1b-instruct
      - mistralai/mistral-large
      - meta-llama/llama-guard-3-11b-vision
      - ibm/granite-vision-3-2-2b
      - ibm/granite-ttm-1536-96-r2
      - ibm/granite-ttm-1024-96-r2
      - ibm/granite-ttm-512-96-r2
      - core42/jais-13b-chat
      - ibm/granite-13b-chat-v2
      - ibm/granite-13b-instruct-v2
      - sdaia/allam-1-13b-instruct
      - bigscience/mt0-xxl
    # Accept images on watsonx.ai (`image_chat` in its foundation_model_specs)
    # but LiteLLM does not flag them `supports_vision`. Every watsonx id it does
    # flag is withdrawn and excluded above, so without these the VLM picker
    # offers no watsonx model at all.
    vision_models:
      - meta-llama/llama-4-maverick-17b-128e-instruct-fp8
      - mistralai/mistral-small-3-1-24b-instruct-2503

  # watsonx.ai running on a Cloud Pak for Data / IBM Software Hub cluster. It
  # is a second front door onto the same `/ml/v1` API as the row above, kept
  # separate because the two need their own credentials: the cluster
  # authenticates with a ZenApiKey built from a CPD username and API key, not
  # with an IBM Cloud IAM key, and it is reached at a cluster URL rather than a
  # region endpoint. OpenRAG routes it through LiteLLM's `watsonx` provider —
  # see `enhancements/providers/watsonx/onprem.py`.
  #
  # The two lists below are a *fallback*, not the catalogue. OpenRAG asks the
  # cluster itself what it serves (`GET /ml/v1/foundation_model_specs`) and
  # shows that in the pickers, so a model the cluster does not have is never
  # offered. These rows are what it falls back to while the cluster is
  # unreachable or before any credentials are entered — keep them plausible for
  # your fleet, but there is no need to keep them exact.
  - name: watsonx_onprem
    display_name: IBM watsonx.ai
    badge: On-prem
    modes:
      oss: true
      on_prem: true
      saas: false # a cluster-hosted deployment has no place in multi-tenant SaaS
    # Checked against a Cloud Pak for Data 5.x cluster, which serves exactly
    # these two. Note the embedding id has no `-v2` suffix: `slate-30m-
    # english-rtrvr-v2` is an IBM Cloud id and 404s here, which is the kind of
    # near-miss that makes a guessed fallback worse than none.
    models:
      - openai/gpt-oss-120b
    embedding_models:
      - ibm/slate-30m-english-rtrvr

  # Red Hat OpenShift AI: models served by KServe + vLLM inside the customer's
  # own OpenShift cluster. vLLM speaks the OpenAI API, so OpenRAG routes this
  # through LiteLLM's `hosted_vllm` provider — see
  # `enhancements/providers/redhat/openshift_ai.py`, which also explains why it
  # is not `openai_like` (that key silently drops the per-provider TLS setting a
  # cluster CA needs).
  #
  # On in on_prem only as shipped: the cluster is the customer's own, and its two
  # endpoint URLs and token are entered in Settings. Elsewhere it is enabled per
  # deployment with OPENRAG_MODEL_PROVIDERS_CONFIG rather than offered to everyone.
  #
  # No `models:` / `embedding_models:` rows are shipped. Model ids are the
  # InferenceServices' `--served-model-name` values, which only the deployment
  # knows, so OpenRAG asks each endpoint what it serves (`GET /v1/models`):
  # with the saved credentials for the catalogue, and with the typed ones
  # (`POST /models/rhoai/discover`) during onboarding. A model the cluster does
  # not serve is never offered; while it cannot be reached, the pickers accept
  # a typed model id.
  #
  # A deployment's own config may still add rows as a fallback for while the
  # endpoints are unreachable, e.g. `models: [gpt-oss-120b]`. Use the
  # served-model-name, not the image tag.
  - name: rhoai
    display_name: Red Hat OpenShift AI
    modes:
      oss: false
      on_prem: true
      saas: false

  - name: anthropic
    display_name: Anthropic
    modes:
      oss: true
      on_prem: true
      saas: true
  - name: azure_ai
    display_name: Azure AI Foundry
    modes:
      oss: false
      on_prem: true
      saas: true

  # Azure OpenAI Service, which LiteLLM keys separately from Foundry above:
  # `azure_ai` is the Foundry catalogue (Llama, Phi, Mistral, Cohere, and the
  # gpt-5.x entries Foundry serves), `azure` is the OpenAI Service deployment
  # where the gpt-4.1 family lives. Both can be offered at once.
  - name: azure
    display_name: Azure OpenAI
    modes:
      oss: true
      on_prem: true
      saas: true
    # Azure serves the same OpenAI models, so it needs the same exclusions.
    # The plain name covers the regional ids (`eu/`, `us/`, `global/`) too;
    # see the openai row above for why both luna generations are wildcarded.
    exclude_models:
      - gpt-3.5-*
      - gpt-5.6-luna*
      - gpt-6-luna*
    # Azure lets you name a deployment anything, so LiteLLM's table cannot know
    # yours. Declare them here and they appear in the pickers for everyone;
    # otherwise each user has to type the name into the model box by hand.
    # models:
    #   - prod-gpt41-eastus
    # embedding_models:
    #   - prod-embed-3-large

  # A self-hosted OpenAI-compatible endpoint (vLLM, LM Studio, an internal
  # gateway). Use one of LiteLLM's passthrough provider keys — `openai_like`,
  # `hosted_vllm`, `lm_studio`, `llamafile`, `custom_openai` — rather than
  # inventing a name: the key has to be one the gateway can route, or the
  # `provider:model` ids it publishes resolve to the default provider and get
  # called with the wrong credentials. (A name of OpenRAG's own works only if
  # it is registered in `PROVIDER_ROUTE_ALIASES` in `services/model_catalog.py`
  # alongside the LiteLLM key it routes as, the way `watsonx_onprem` is.) The
  # endpoint and key are entered in Settings; only the model ids belong here.
  #
  # - name: openai_like
  #   display_name: Internal LLM Gateway
  #   modes:
  #     oss: true
  #     on_prem: true
  #     saas: false
  #   models:
  #     - llama-3.3-70b-instruct
  #     - qwen2.5-coder-32b
  #   embedding_models:
  #     - bge-m3