Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++
15.5 KB
Raw Blame

Terminal telemetry field contract (schema v1)

This is the source map for the current strata status, strata monitor, and strata logs surfaces. The authoritative read-only response is GET /v1/telemetry; strata_cli/client.py validates and bounds it before display. The server builds v1 in serve/server.py from its current metrics, status, scheduler, engine samples and GPU sampler. A snapshot is a momentary observation, not a transaction across all subsystems. sampled_at is server wall-clock seconds for telemetry snapshots; the status-only fallback uses local receipt time because that older endpoint supplies no current sample clock. Age and rate labels must be tied to the relevant interval.

-- or JSON null means unavailable. Zero is used only when a source reports zero. The client accepts at most 2 MiB of JSON, 16 devices, 64 slots, 50 active and 50 recent requests, 32 activity events and 16 stages. Text is shortened and stripped of control characters. An instance change resets client-derived rates, selected-row identity and display histories. Monitor history retains at most 32 samples for each of at most 16 current selected cards, server output, processed batch rows, pre-admission queue count and each of at most 16 configured replica batch series. Absent or duplicate current sensors and failed collection append a gap instead of connecting unrelated samples. A missing newest value hides its output or work trend. The Overview also treats an engine batch or stage sample older than five seconds as unavailable; Serving and Engine detail retain its last value with the reported age.

Display / JSON field Server source Unit and scope Sensitivity / freshness Missing or older-server behavior
Model and loaded state (model.id, model.loaded) /v1/status as normalized into v1 Model identifier and Boolean for this server instance Identifier can reveal the served model; current snapshot -- / null; never infer load from a config file
Context (model.context) Status cache limit Tokens, current model Operational metadata; current snapshot -- / null
Instance ID/state (instance.id, instance.state) Server-assigned boot identity and drain state One running server; ready or draining ID is operational metadata; current snapshot Legacy has boot time but no reliable restart identity; client-derived rate stays unavailable
Requested/effective slots (work.slots_requested, slots_effective) Requested serving configuration; actual engine/status concurrency Slot counts for this instance Operational; after load, may differ because engine reduced concurrency Requested is unavailable in legacy; effective may be reported by status
In flight, occupied, waiting, pre-admission (work.active, occupied_slots, waiting, queued) Live engine/status counters and slot states Request/slot counts at snapshot time Operational; state transitions can race a poll; these sets overlap The bounded client derives occupied slots from validated slot rows; it is -- unless every effective slot has a unique index and known state; do not add counts as disjoint populations
Admission (admission_reserved, admission_limit, max_queue) Server admission state and configured cap Accepted/reserved call count, total limit and queue cap for this instance Operational; current snapshot Legacy adapter leaves these unavailable
Rejected, timeouts, disconnect cancellations (admission_rejected, queue_timeouts, client_disconnect_cancels) Server counters Calls since server start Operational; reset on restart Legacy unavailable; a reported zero is real zero
Oldest queue wait and deadline (oldest_queue_s, queue_timeout_s) Active queued/waiting rows and configured deadline Seconds; oldest live wait and configured limit Current snapshot; wait changes as requests move Legacy unavailable; deadline absent unless configured
Slot/group and request progress (work.slots[]) Engine live slots plus engine slot_group/slot_live map Slot index, pipeline group, state, held/prompt/read/generated tokens and per-slot tokens/s Pseudonymous request ID; at most 64 rows; current snapshot Group/slot values remain unavailable if engine does not report them; group is never a GPU assignment
Wide Overview active context and slot rate Complete work.slots[] joined with model.context and work.requests[] Largest active slot's held tokens divided by per-slot model capacity; separate active-slot held/capacity rows; a request's slot tokens/s only when its slot and pseudonymous ID match Display-only derivation from one snapshot; no summing independent contexts or treating idle retained cache as active Incomplete slot report, missing held tokens/capacity or unmatched request ID yields --; no inferred expert or MTP activity
Overview slot phases (read/decode/idle) Count of work.slots[] states Slot counts at one snapshot; these exclude queued/admitted requests outside slots Derived only when every effective slot has a unique reported index and a known state Hidden for partial, duplicate or unknown slot reports; never infer phase counts from active request totals
Pipeline group occupancy (Gx busy/total) Count of complete work.slots[] by reported group and non-idle state Occupied/effective slots per pipeline group at one snapshot; not rows per batch window or a GPU lane Exact counts require every effective slot to have a unique index, a known state and a valid group Shows unavailable when the slot report is partial, duplicate or ambiguous; it never treats a partial group list as a total
Replica and group identity (work.slots[].replica, work.requests[].replica) The opt-in replica router and each child engine's slot map Zero-based replica ID; group IDs are offset across replica engines so G0 in two engines is never collapsed into one chart Instance-scoped, at most 64 slots and 50 live requests; a solo request may have a replica while its slot is null No replica mode: null; a child slot without an ID retains null request correlation. Server-wide rows/window and stage timing remain unavailable in replica mode; separate child batch, placement and host-span samples are contracted; the first child is never presented as the whole server
Replica card assignment (topology.replica_devices[]) Validated configured GPU groups used to start each replica Replica number and selected physical GPU indices; the Overview labels each selected card R0, R1 and so on Configuration claim, not a measured routing trace; at most 16 replicas and 16 devices Missing, duplicate or ambiguous assignment is omitted; card sensors remain independent of the configured mapping
Replica layer placement (topology.replica_stages[]) Each live child engine's bounded startup layer-split log plus that child's launch visibility order Local stage number, contiguous layer range and physical card when it belongs to the configured replica group; the Overview labels a card Rn/Sm only when exactly one reported local stage maps to it Engine-reported at startup, at most 16 replicas with 16 stages each; not a live utilization trace Missing/oversized log leaves ranges unavailable; missing or mismatched visibility leaves card index --; stopped children do not retain a stale plan; server-wide topology.stages stays empty in replica mode
Active requests (work.requests[]) Server terminal-request registry enriched with live slots At most 50 request states, queue/admit time, slot/group, prompt-read/generated counts Pseudonymous instance-scoped ID; no prompt/answer/client address; requests_omitted counts overflow Legacy has no correlated active request list; show unavailable, not synthetic IDs
Recent completed requests (recent[]) Existing /metrics recent-request ring At most 50 outcome, token and timing summaries No prompt/answer body; entries can disappear; current ring Empty list or unavailable fields remain empty/--
First engine token (first_engine_token_s) Engine queue-entry to first generated token measurement Seconds per request, includes queue/load/prefill after queue entry Current live or recent request; missing until observed --; this is not client-observed TTFT or HTTP ingress/network time
Answer/prefill rates (rates.answer_tok_s, prefill_tok_s) Existing engine metrics Tokens/s over engine-reported windows or mean Operational; answer_window_s identifies answer window -- if no engine sample; do not equate with aggregate multi-request output
Wide Overview rate histories (client only) Recent displayed prompt/prefill and answer rates Relative eight-cell trends; the aggregate output history also includes its displayed range At most 32 polls per series; reset on server instance change and append gaps on outage Hidden when fewer than two valid samples or newest sample missing; not a benchmark or per-card throughput claim
Aggregate output (rates.output_tokens_server_counter -> client aggregate_output_tok_s) Completed server output tokens plus current in-flight generated tokens Tokens/s for all requests over one common pair of polls Derived by TUI; valid only for same instance, positive interval ≤30 s and monotonic counter First poll, restart, invalid interval or counter reset: --; the Overview trend uses only recent valid display samples and labels their min/max; not a per-GPU rate
Batch rows/window (effective_rows_window, batch_active_rows_window, batch_sample_*) Opt-in engine BSTAT processed-row and active-slot counters Average rows and active slots per batch window; count, elapsed span and age in seconds Low-rate opt-in sample; age shown; rows can include padding or MTP work Default and unsupported engines: --; slot occupancy is not batch rows/window
Per-replica batch rows/window (work.replica_batches[]) Each live child engine's opt-in BSTAT, kept in its own row Processed/active rows per window and sample age for that replica only; R0 and R1 are never summed or averaged across asynchronous sample windows At most 16 replica rows; missing or dead child has ID with no sample; bounded counts are revalidated in the terminal -- per replica without a valid sample; no server-wide rows/window claim in replica mode; native multi-GPU qualification remains open
Overview work trends (client only) Recent displayed effective_rows_window, queued and per-replica effective_rows_window values Small relative graph with the visible min/max in rows/window or queued requests; independent polling samples, not another engine batch statistic At most 32 samples per series; a server instance change resets the series; no cross-replica average and no addition of queue populations Hidden when fewer than two valid samples exist, the newest sample is missing, the batch sample is more than five seconds old or the connection is stale; compact 80-column Overview keeps current values without these extra graphs
Per-replica stage host span (work.replica_stage_samples[]) Each live child engine's opt-in PSTAT, kept in its own row Local stage in-flight percent, milliseconds, sample span and sample age for each replica; no cross-replica average At most 16 child rows and 16 stages per child; stage count must match that child's reported plan when available; values include host polling delay -- per replica without a valid sample; server-wide stage_in_flight[] stays empty in replica mode; not GPU kernel occupancy or utilization; native multi-GPU qualification remains open
Pipeline stage in-flight (stage_in_flight[], stage_sample_*) Opt-in engine PSTAT host spans Percent of sample span and milliseconds per stage Low-rate opt-in sample; includes host polling delay -- without sample; never label it GPU kernel occupancy/utilization
Layer stages and device mapping (topology.stages[]) Bounded current engine startup layer-split line, mapped through unambiguous numeric child visibility order Inclusive layer ranges by stage and optionally physical device index Reported allocation for loaded model, not measured runtime placement No current log: unavailable; ambiguous visibility: retain ranges with no claimed card mapping
GPU sensors (devices[]) Per-device NVML/AMD sampler, selected device indices Bytes VRAM used/total, utilization %, temperature °C, power W, per card Host telemetry at sampler cadence; no inferred device values Missing sensor fields --; Overview rows follow declared selected-device order and show a blank selected card if its sensor is absent or ambiguous; unrelated host cards stay in Hardware detail; legacy aggregate becomes one card only if sampler reports exactly one GPU
Host RAM (ram.used, ram.total) Existing hardware sampler Bytes for host RAM Operational; sampler cadence -- if not reported
Redacted activity (activity.events[]) Server in-memory terminal transition ring Instance-scoped sequence, time, pseudonymous request ID, event/outcome Server retains 128 and sends newest 32; gaps possible; no prompt, answer, tool text or credential Legacy reports activity unavailable; empty supported feed says no transitions retained
Lifetime totals (totals.*) Server metrics totals Requests and tokens since server start Operational; reset with instance -- if omitted; do not use across restarts

The v1 quality object marks telemetry, request correlation, batch sample, stage host span and topology availability. quality.replica_stage_in_flight=measured_host_span means at least one child has a valid local PSTAT sample; it does not imply that every child has a sample. quality.stage_in_flight remains unavailable for the server as a whole in replica mode. A consumer should inspect it before drawing a chart or making a claim. quality.telemetry=unavailable means only /v1/status was reachable; live work, rates and sensors remain null or empty. work.requests[] and activity.events[] are monitoring data, not durable audit records. The monitor key authorizes /v1/telemetry only. An inference API key also works but grants broader legacy access; keep the keys distinct and pass them through environment references. Remote monitoring requires HTTPS or a local SSH tunnel; the client refuses redirects.

For an older server, the client only falls back after a v1 404, reading /metrics and /v1/status and normalizing into the same bounded shape with source=legacy-adapter. If /metrics also returns 404 but /v1/status answers, it produces a source=status-only snapshot with model and reported effective concurrency, labels telemetry unavailable, and leaves live work and rates null. It never manufactures active request correlation, batch rows, stage spans, admission counters or four per-card readings. A schema mismatch or auth failure does not trigger fallback. Fixture screenshots are illustrative and cannot establish RTX 6000 throughput or native backend support.

Source and qualification map

  • Contract and sanitization: strata_cli/client.py; TUI aggregate rate and polling: strata_cli/monitor.py; labels and fallbacks: strata_cli/render.py.
  • Server response: serve/server.py /v1/telemetry; opt-in engine batch and pipeline samples are described in docs/TERMINAL.md.
  • Before a four-card release, capture server schema, source commit, engine/backend build, model hash/revision, exact RTX 6000 generation and VRAM, driver, selected device IDs, layer ranges and visibility order. Run the load/parallel/cancel and overhead matrix on that host. This file documents meanings; it is not hardware qualification evidence.