Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Observability
What switch-core reports about itself, where it goes, and how to turn it on.
Structured logging came first and is older than the rest of this
(logging_config.py, logging_context.py). Metrics, readiness and the export
path are CHOO-2807.
The short version
Nothing leaves the deployment until OTLP_ENDPOINT names a collector. Set it,
set DEPLOYMENT_ID to a UUID, and the server reports metrics. Logs are a
second switch because they cost more.
OTLP_ENDPOINT=https://telemetry.example.com
DEPLOYMENT_ID=0e5d1b3a-6c1f-4c22-9a4c-3a9f5a2b7d10 # `uuidgen`, once, then leave it
OTLP_LOGS_ENABLED=true # optional
In Helm the same settings are switchCore.observability.*, with one
exception: OTLP_HEADERS is secrets.otlpHeaders, because the documented use
for it is a collector's API key and every other credential in that chart comes
from the Secret rather than the pod spec. Dashboards and alerts are checked in
under deploy/observability/.
Checking it actually works
You do not need a Datadog account, or the relay, or a cluster. Run a collector on your laptop and read what the server sends it.
The self-contained way
This touches nothing you already have: its own database on its own port, its
own settings, no .env, and it cleans up after itself. Worth preferring even
if you do have a working local stack, because nothing it does can disturb one.
# A throwaway database.
docker run -d --name switch-obs-check \
-e POSTGRES_PASSWORD=check -e POSTGRES_DB=switch \
-p 55432:5432 postgres:16-alpine
# One terminal: a stand-in collector that prints what arrives.
python scripts/otlp_sink.py
# Another: the server, reporting to it.
DB_HOST=localhost DB_PORT=55432 DB_USER=postgres DB_PASSWORD=check DB_NAME=switch \
DB_REQUIRE_RESTRICTED_ROLE=false \
ID_SERVER_NAME=switch.local \
AGENT_REGISTRATION_TOKEN=check SECRET_KEYS=check:check-secret-key-long-enough-for-32 \
[email protected] GATEWAY_ADMIN_PASSWORD=check \
SERVER_PORT=8099 LOG_FORMAT=json ENVIRONMENT=local \
OTLP_ENDPOINT=http://localhost:4318 \
DEPLOYMENT_ID=$(uuidgen | tr 'A-Z' 'a-z') \
OTLP_EXPORT_INTERVAL_SECONDS=5 \
OTLP_LOGS_ENABLED=true \
uv run --project core python -m switch_core.main
DB_REQUIRE_RESTRICTED_ROLE=false is what lets this run as the superuser
against a throwaway database. It logs an error saying tenant isolation is not
in force, which is correct and is why it is not a default — never set it on
anything real.
uv run directly rather than just run, because just loads .env from the
working directory and this recipe deliberately has none. Clean up with
docker rm -f switch-obs-check.
Using your own local stack instead
just run reads .env from the directory you run it in. A git worktree has
no .env — it is untracked, so it does not come across with the branch — and
without one every database and secret setting is missing and SwitchConfig
refuses to start, listing all ten. Copy one in (cp ../switch/.env .), and
check it against .env.example first: an older one predates the
runtime/owner database roles and just up will refuse it. Add
SERVER_PORT=8099 if your usual server is already on 8000.
Reading what comes out
Within a few seconds the sink prints each interval. Make some requests
(curl localhost:8099/health) and the next one carries them:
[15:37:39] POST /v1/metrics
resource: service=switch-core version=0.26.0 env=local deployment=0e5d1b3a-…
switch.http.requests +9 {method=GET, route=/health, status_class=2xx}
switch.http.requests +4 {method=GET, route=unmatched, status_class=4xx}
switch.http.request.duration n=9 mean=0.3ms {method=GET, route=/health}
switch.health.check = 1 {check=database}
switch.db.pool.in_use = 0
Four things in that output are worth checking deliberately, because each is a way this could be quietly wrong rather than visibly broken:
- Routes are templates. Requests to several unknown paths collapse into one
unmatchedseries rather than one series each. - Counters are deltas. An interval with no traffic carries no
switch.http.requestsline at all, rather than repeating the last total. switch.db.pool.overflowreads 0 on an idle pool, not a negative number.- Log records carry
tenant_idandrequest_id, which is the whole reason for shipping them rather than counting them.
Two things that look like faults and are not. Off Linux,
switch.runtime.memory_rss and switch.runtime.open_fds are absent, and a
warning at startup says so. And for the first few intervals after boot,
switch.health.check{check:message_listener} reads 0 — the checks begin before
the notification listener has finished connecting, and it reports what is true
at the time rather than assuming the best. It goes to 1 on its own, and
readiness never gated on it.
A steady switch.runtime.event_loop_lag of one or two milliseconds is the
floor, not a stall: it is how far past its deadline a timer normally wakes.
The alert on it fires at a thousand times that. The reading comes from a probe
that wakes every 100 ms, so a stall longer than that is always caught and read
to within 100 ms. switch.runtime.event_loop_stalls counts each stall of 50 ms
or more by duration, which says how often the loop blocks, not only how badly.
What this does not exercise. A bare server has no rooms, no agents and no
bridges, so nine of the twenty-five metrics never appear — everything under
switch.messages.*, switch.bridge.* and switch.agent.*. Their absence
here is correct and
says nothing about whether they work; what covers them is
TestWhatIsMeasured in core/tests/switch_core/transport/test_postgres_transport.py
and test_bridge_metrics.py, which record against a real database and a real
dispatch path. To see them live you need an agent in a room actually talking.
Checking readiness for real
Stop the database out from under a running server:
docker stop switch-obs-check
curl -s localhost:8099/health/ready # 503, naming the database
curl -so /dev/null -w '%{http_code}' localhost:8099/health # still 200
docker start switch-obs-check
curl -s localhost:8099/health/ready # back to 200 within ~15s
Both halves matter. Readiness failing is what takes the pod out of service; liveness staying up is what stops Kubernetes restarting a server whose only problem is a database it does not own. A restart would not fix it and would drop every live session to find that out.
The three things it emits
Logs go to the container's output, with tenant_id, request_id,
agent_id, room_id and user_id stamped on every line by a filter on the
handler — so records from libraries carry them too. A room id is a log field
and deliberately not a metric attribute: it is unbounded and belongs to one
tenant, which rules it out of a dashboard label for the reasons under "Why
there is a catalogue" — and following one room through a failure is the single
most common thing anyone asks these logs for. A request from Switch Console also
carries console_id and console_name: on a server a Console runs for its
user, everyone with access to the host signs in as the one seeded account, so
user_id cannot tell them apart and these can. They are attribution the caller
supplies, not authentication. Set LOG_FORMAT=json to get them as
fields rather than inside the message text; the default is text, which is for
reading in a terminal, so anywhere the logs are actually collected wants the
JSON form. With OTLP_LOGS_ENABLED the same records are also posted to the
collector. The stderr copy is never replaced: kubectl logs keeps working, and
a collector outage costs a copy rather than the record.
Metrics are declared in
core/switch_core/observability/catalogue.py,
which is the reference for what exists. Nothing not declared there can be
recorded — the registry rejects an unknown name and an attribute the spec does
not list.
Traces are not implemented. See "What is missing" below.
The agent event streams are counted but not timed. A long poll is held open until something happens or the caller's own timeout expires, so its duration measures a parameter the client chose rather than anything this server did. In a latency histogram that is worse than useless: it would make those routes' percentiles meaningless and, sharing an axis, flatten every other route to the floor. They are counted like everything else.
What is timed, as against counted. Four things: HTTP requests, message
delivery lag, database statements and outbound calls to a collaboration
platform. That set is chosen to answer "what is slow" without guessing — on
this server a slow request is nearly always a slow query or a slow platform,
and those are the two the process cannot see from a count alone. A database
statement is timed around the driver call by SQLAlchemy's cursor events
(observability/query.py), so what is measured is the round trip rather than
the Python either side of it; the statement text never becomes an attribute,
only its leading keyword mapped through a fixed table — with the per-transaction
tenant set_config given a label of its own, because it is a third of all
statements and always trivial, so counted as a select it drags the
percentiles toward bookkeeping. A statement that raised is not timed, because a
query that failed in four milliseconds is not evidence the database is fast.
It is not every statement the process runs. Alembic builds its own engine, the
message listener talks to asyncpg directly, BEGIN/COMMIT come from the
dialect, and an executemany is one measurement for the whole batch because it
is one round trip. Read the panel as "the application's queries", not "all
database work".
Not every HTTP surface is counted. switch.http.* comes from middleware on
the FastAPI app, which is the agent bridge, the MCP mount and the gateway
beneath it. Two listeners sit outside it — the Teams bridge and the
collaboration callback ingress each run their own aiohttp server on their own
port — so their traffic appears in no request metric. Read the HTTP panels as
"the main API", not "everything this process serves".
Why there is a catalogue
A metric's cost is the number of distinct attribute combinations it produces, and the values that feel most natural to attach are unbounded: a room id, an agent id, the raw request path. One of those reaching a call site is not a noisier dashboard, it is a bill and a collector that starts dropping.
On a multi-tenant server it is also a disclosure question. Dashboards are read by whoever can see them, which is not the same set of people as those entitled to a given tenant's rows, so tenant attribution stays out of metrics. Product events that are tenant-attributed are CHOO-2806's concern and travel as log records, where they are scoped separately.
So every attribute value must come from a fixed set the code controls — a
platform name, a route template, a status class. Where that cannot be
guaranteed at the call site, it does not belong in a metric. Two examples in
the tree: the HTTP middleware labels by route template and folds everything
unmatched into one bucket, because a 404 path is attacker-chosen; and the
transport classifies send_event's arbitrary event type into four values
rather than passing it through.
What is counted because it is otherwise invisible
Most of the catalogue is volume and state — useful, and you would miss it, but its absence would not mislead anyone. A handful of counters exist for a different reason: each one measures something that is already handled, and handled in a way that leaves no trace.
switch.messages.delivery_failures— the delivery loop swallows one room's exception so it cannot stop the others. Correct, and it means a room that stopped delivering leaves only a log line.switch.messages.backlog_skipped: a consumer resuming from its stored cursor steps over rows older than its replay cap rather than answer them late. The rows stay in the room; what they would have caused (a command's reply, a platform post) never happens, and only this says so.switch.messages.send_failures— a send that raises before the commit writes nothing, so the symptom is an absence, and an absence is what a quiet room looks like too.switch.messages.observer_failures— reporting a message for product telemetry raised, after the transport committed it or after the router decided an agent was asked something in it. Caught, because the message itself is fine: a send that raised after its commit would be retried and posted twice. So the only trace is a rate-limited log line.switch.messages.routing_failures: the router could not store one agent's own event for a message it was writing, could not post a reply that message owed (an offline notice, a refusal, a command's answer), or could not decide a multi-file post it had given up waiting for. Each is caught so that one agent cannot cost the room its message, sostagesays which it was.switch.agent.events_dropped— an agent's buffer discards events when it overflows or when they age out. The agent is told it missed them, so nothing is hidden from the agent; nothing told anybody else.switch.agent.connections_expired— a lapsed heartbeat closes a connection and the agent reconnects. A gauge of live connections stays perfectly flat while that happens as fast as it can.switch.bridge.errors— an inbound bridge failure is a message a person sent that nobody received; from the platform it is indistinguishable from being ignored.switch.bridge.throttle.held— how long a platform rate limit held a publication back. On a shared app bot one tenant's burst holds back every tenant, so a risingdelivery=sharedreading is the signal that per-tenant fairness is needed.switch.messaging.events_ignored— events from chats nobody claimed, on a platform where the app routinely sits in them (a shared Telegram bot with Group Privacy off). Counted instead of logged, so the drops that are real losses — a bridge down, a migration's window — keep their own errors.
If you add another place that catches an exception to keep something alive, this is the list it belongs on.
Health: two routes, on purpose
| Route | Answers | Used by |
|---|---|---|
/health |
Always ok, checks nothing |
Liveness probe; the gateway Deployment and setup Job wait on it at boot |
/health/ready |
Each dependency, and 503 when a gating one fails | Readiness probe |
/health could not simply be made stricter. Two other workloads block on it
during a deploy, so anything it checked would become a boot-ordering
dependency for them.
What gates readiness is deliberately narrow. switch-core is pinned to one
replica with a Recreate strategy, because it holds live sessions in memory.
A failing readiness probe therefore does not move traffic to a healthy pod —
there is no other pod. It empties the Service. So only the database gates:
without it every request is an error anyway.
Everything else is reported and alertable but never fatal. A crashed Slack adapter is a real fault; taking Switch off the air over it would turn one broken bridge into every broken bridge.
The checks run on a timer rather than per request, and the kubelet and the metrics exporter read the same cached answer. The cache carries its own age, and one nobody is refreshing reports itself as a failure — which is also how a blocked event loop shows up.
What the process reports about itself
There is no Datadog agent in the cluster, so switch-core reports its own
memory, CPU, descriptors and garbage collections, with no extra dependency.
CPU and collection counts come from the standard library and work anywhere;
memory and open descriptors are read from /proc/self, so off Linux those two
are absent rather than fabricated, and a warning at startup names them.
Event-loop lag is the one no external agent could produce. The server is single-threaded and cooperative: one blocking call stalls every room, every bridge and every heartbeat at once, and from the outside that looks like an unrelated timeout somewhere else. The connection sweep already measured its own oversleep to avoid expiring connections it had simply failed to hear from; that number is now reported instead of discarded.
What a Datadog agent would add
There is no Datadog agent in this cluster, and nothing above needs one — an application reporting over OTLP covers application health on its own. What an agent adds is everything the application is not in a position to know, and the list is worth writing down because the gaps are not obvious from a dashboard that looks full.
It can report the process's death. This is the structural one. Everything here is reported by switch-core, so a process that is OOM-killed or segfaults takes its queued logs and its last metric interval with it. What survives is an alert saying reporting stopped — which says something happened and nothing about what. An agent watches from outside and reports the restart, the exit code and the kill reason.
Limits, not just usage. The process reads its own resident memory from
/proc/self, which is the numerator. The cgroup limit it is measured against,
and CPU throttling when it exceeds its quota, are container facts it cannot
see. A server being throttled looks from the inside like a server that is
mysteriously slow — and the event-loop lag metric would show the symptom while
naming nothing.
Kubernetes state. Pod phase, restart counts, evictions, pending pods that never scheduled, node pressure, PersistentVolume usage. The last matters more than it sounds: Postgres and Mattermost hold this deployment's data on volumes, and nothing currently reports how full they are.
The database from the database's side. Switch reports its own connection pool, which answers "are we holding too many connections" and not "is the database struggling". The agent's Postgres integration gives query performance, locks, replication lag and server-side connection counts. Given that the database is the one dependency that gates readiness, seeing only the client's half of it is a real gap.
Log collection without an in-process shipper. An agent tailing container
output would make OTLP_LOGS_ENABLED unnecessary and would capture the lines
a dying process cannot flush.
Trace intake. An agent accepts OTLP traces directly, which is one of the two ways the gap below gets closed.
None of this blocks the work here, and none of it should be added to the application — it is infrastructure, tracked separately.
What is missing
Tracing. The export path is signal-agnostic, but nothing produces spans,
and the relay Switch reports to does not serve /v1/traces — a POST there
returns 404. Two things have to happen: the collector must accept the signal,
and the server must produce spans.
Tracing is the one observability item not built, and it was always scoped as optional. The order matters: spans built against a collector that 404s cannot be turned on, cannot be verified, and would be reviewed against nothing. The collector side is tracked with the infrastructure work (an agent that accepts OTLP traces directly is one of the two ways it closes); the server side is a day's work once there is somewhere to send them, and the log records already carry the fields to correlate against.
There is deliberately no OTLP_TRACES_ENABLED setting in the meantime. A
flag a deployment can turn on and see no difference from is a configuration
surface that lies about what it controls; it arrives when there is something
for it to switch off.
When it does, the log records already carry traceId and spanId fields
wherever they are set, so log-to-trace correlation needs no further change to
the log path.
Per-route trace sampling will matter when it arrives. Switch's agent
polling endpoints (/events, /room-history, per-agent reads) are very high
volume and are already filtered out of the uvicorn access log for that reason;
tracing them at full rate would swamp APM and cost more than it tells anyone.
Failure behaviour
Everything here follows the repository's rule that a visible failure beats a silent fallback.
- A collector that rejects a payload raises rather than being ignored,
including a partial rejection inside a
200— the collector answers200for a payload it drops, so the body is read. - Repeated export failures escalate from a warning to an error naming how many intervals were lost, because by then the dashboards are stale and somebody is about to be misled by them.
- The log queue is bounded and drops the oldest, and says how many it dropped. A gap that announces itself beats one nobody can see.
DEPLOYMENT_IDis required whenever an endpoint is set, and the server refuses to start without it. The collector drops unattributed payloads silently, with a200, so the alternative is a deployment that looks configured and appears on no dashboard.- An export failure never takes the server down. A metrics exporter that can kill the thing it measures is worse than no metrics exporter.