Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
docs(otel): clarify per-candidate fallback call metrics (#839)
Graham King08b075dd8667
diff --git a/docs/reference/opentelemetry.md b/docs/reference/opentelemetry.mdindex dae067c06..3e4c4657e 100644--- a/docs/reference/opentelemetry.md+++ b/docs/reference/opentelemetry.md@@ -83,8 +83,8 @@ Metrics use the `switchyard` meter scope. The tables use OTel instrument names.| `switchyard.run_duration_ms` | Histogram | `algorithm`, `outcome` | Algorithm-task duration in milliseconds. || `switchyard.algorithms_in_flight` | UpDownCounter | `algorithm` | Active algorithm tasks; exported as a Prometheus gauge. || `switchyard.decisions` | Counter | `algorithm`, `selected_model` | Published routing decisions. |-| `switchyard.llm_calls` | Counter | `algorithm`, `selected_model`, `outcome` | Logical offloaded and terminal model calls. |-| `switchyard.llm_call_duration_ms` | Histogram | `algorithm`, `selected_model`, `outcome` | Logical call duration in milliseconds; see streaming limits below. |+| `switchyard.llm_calls` | Counter | `algorithm`, `selected_model`, `outcome` | Routing-time offloaded calls and each terminal answer candidate, including failures. |+| `switchyard.llm_call_duration_ms` | Histogram | `algorithm`, `selected_model`, `outcome` | One duration sample in milliseconds per offloaded call or terminal answer candidate, including failures; see streaming limits below. || `switchyard.total_requests` | ObservableGauge | none | Process-wide total of successful and failed answer candidates after routing, including reused routing responses. || `switchyard.total_errors` | ObservableGauge | none | Process-wide total of failures after routing, including failed answer candidates and reused routing responses. || `switchyard.requests` | Counter | `model` | Successful answer candidates or reused routing responses, by model ID. |@@ -101,9 +101,20 @@ Algorithm/call `outcome` is `ok` or `error`. HTTP attempt `outcome` is `ok` foror `none`. Classifier `reason` is `timeout`, `transport`, `upstream_5xx`,`upstream_non_5xx`, `invalid_response`, `parse_error`, `client_error`, or `call_error`.-Logical calls, candidate calls, and HTTP attempts are different counts. Candidate-fallbacks and HTTP retries remain within one logical call. A response produced-during routing is not counted again as a new terminal model call.+Client requests, routing decisions, model calls, and HTTP attempts have separate+counts. For `switchyard.llm_calls` and `switchyard.llm_call_duration_ms`, each+routing-time offloaded call and each terminal answer candidate contributes one+observation. HTTP retries stay within the same candidate's count and duration.+Reusing a response produced during routing keeps only its routing-time call observation.++For example, one buffered client request with no judge calls or retries, a failed+primary, and a successful fallback produces one client response, one routing decision,+two `llm_calls`, and two `llm_call_duration_ms` samples. The primary has+`selected_model=<primary>` and `outcome=error`; the fallback has+`selected_model=<fallback>` and `outcome=ok`. For terminal answer-candidate observations,+`selected_model` identifies the model called. For routing-time observations, it+identifies the initial routing candidate. For `switchyard.decisions`, it identifies+the initial routing selection.The five request/error/latency instruments above record each answer candidate afterrouting, not each HTTP attempt. A failed candidate followed by a successful fallback@@ -112,9 +123,10 @@ model, and one to `requests` for the successful model. Routing-time calls and fabefore a routing outcome do not contribute. A response reused from routing contributesone to `total_requests` and one to either `requests` or `errors` (also `total_errors`on failure), but no `model_call_latency_ms` sample. The `model` label is the-candidate's configured model ID (the serving model for a reused response), not the-`selected_model` label used by logical-call metrics. The two total gauges are cumulative-process-wide compatibility values, not counts of requests currently in flight.+candidate's configured model ID (the serving model for a reused response).+For terminal answer candidates, it matches `selected_model` on `llm_calls` and+`llm_call_duration_ms`. The two total gauges are cumulative process-wide+compatibility values.For streams, these five instruments record when the response stream ends or is dropped.Stream errors count as failures. Dropping an unfinished stream also counts as a failure;@@ -153,7 +165,8 @@ Advisor Gate instruments use the prefix `switchyard.advisor_gate.`:### Timing and streaming- `libsy.run` may finish before the answer call. Nested algorithms have separate run spans.-- `libsy.llm_call` and logical call-duration metrics end at response-handle availability, not stream completion. The span's `input_tokens`, `output_tokens`, `total_tokens`, and `reasoning_tokens` fields are buffered-response only.+- `libsy.llm_call` and routing-time call metrics end when the host fulfills the offloaded call. They include host queueing. The LLM client driver buffers routing streams before fulfilling the call. The span's `input_tokens`, `output_tokens`, `total_tokens`, and `reasoning_tokens` fields are buffered-response only.+- For terminal answer candidates, `switchyard.llm_calls` and `switchyard.llm_call_duration_ms` record when the response stream ends or is dropped. Duration includes that candidate's retries and stream consumption. Stream errors and unfinished drops record `outcome=error`; a terminal message permits a successful drop.- `libsy.client_call` remains open while its stream is consumed. IDs, usage, and finish reasons update from normalized events. An unfinished stream dropped by its consumer records `cancelled`; a stream error records `error`.### Data exposure