Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
{
"title": "Switch \u2014 core service health",
"description": "What switch-core reports about itself. Metric definitions live in core/switch_core/observability/catalogue.py; this dashboard is checked into deploy/observability/ in the switch repository, so edit it there rather than here.",
"layout_type": "ordered",
"reflow_type": "fixed",
"template_variables": [
{
"name": "env",
"prefix": "env",
"default": "*"
},
{
"name": "service",
"prefix": "service",
"default": "switch-core"
}
],
"widgets": [
{
"definition": {
"type": "note",
"content": "**Is it working?** Each dependency reports 1 when its last check passed and 0 when it failed. Only the database takes a pod out of service. With one replica that empties the service; with several, traffic moves to the pods that still pass. A crashed bridge is a real fault that is deliberately not fatal.",
"background_color": "blue",
"font_size": "14",
"text_align": "left",
"vertical_align": "top",
"show_tick": false,
"tick_pos": "50%",
"tick_edge": "left",
"has_padding": true
},
"layout": {
"x": 0,
"y": 0,
"width": 12,
"height": 2
}
},
{
"definition": {
"type": "timeseries",
"title": "Dependency checks (1 = passing)",
"show_legend": true,
"legend_layout": "horizontal",
"requests": [
{
"q": "min:switch.health.check{$env,$service} by {check}",
"display_type": "line",
"style": {
"palette": "cool",
"line_type": "solid",
"line_width": "normal"
}
}
],
"yaxis": {
"min": "0",
"max": "1.2"
}
},
"layout": {
"x": 0,
"y": 2,
"width": 8,
"height": 4
}
},
{
"definition": {
"type": "query_value",
"title": "Agents connected",
"requests": [
{
"q": "sum:switch.agents.connected{$env,$service}",
"aggregator": "last"
}
],
"autoscale": true,
"precision": 0
},
"layout": {
"x": 8,
"y": 2,
"width": 2,
"height": 2
}
},
{
"definition": {
"type": "query_value",
"title": "Bridges running",
"requests": [
{
"q": "sum:switch.bridges.running{$env,$service}",
"aggregator": "last"
}
],
"autoscale": true,
"precision": 0
},
"layout": {
"x": 10,
"y": 2,
"width": 2,
"height": 2
}
},
{
"definition": {
"type": "query_value",
"title": "Consumers running (read loops)",
"requests": [
{
"q": "sum:switch.consumers.running{$env,$service}",
"aggregator": "last"
}
],
"autoscale": true,
"precision": 0
},
"layout": {
"x": 8,
"y": 6,
"width": 2,
"height": 2
}
},
{
"definition": {
"type": "query_value",
"title": "Event-loop lag (ms, worst)",
"requests": [
{
"q": "max:switch.runtime.event_loop_lag{$env,$service}",
"aggregator": "max",
"conditional_formats": [
{
"comparator": "<",
"value": 100,
"palette": "white_on_green"
},
{
"comparator": "<",
"value": 1000,
"palette": "white_on_yellow"
},
{
"comparator": ">=",
"value": 1000,
"palette": "white_on_red"
}
]
}
],
"autoscale": true,
"precision": 0
},
"layout": {
"x": 10,
"y": 6,
"width": 2,
"height": 2
}
},
{
"definition": {
"type": "note",
"content": "**HTTP.** Routes are templates, never resolved paths \u2014 a path carries an id per request and would be a new series each time. Anything the router could not place is folded into `unmatched`.",
"background_color": "gray",
"font_size": "14",
"text_align": "left",
"vertical_align": "top",
"show_tick": false,
"tick_pos": "50%",
"tick_edge": "left",
"has_padding": true
},
"layout": {
"x": 0,
"y": 8,
"width": 12,
"height": 1
}
},
{
"definition": {
"type": "timeseries",
"title": "Requests per second, by outcome (none = no response sent)",
"show_legend": true,
"requests": [
{
"q": "sum:switch.http.requests{$env,$service} by {status_class}.as_rate()",
"display_type": "bars"
}
]
},
"layout": {
"x": 0,
"y": 9,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Server errors per second, by route",
"show_legend": true,
"requests": [
{
"q": "sum:switch.http.requests{$env,$service,status_class:5xx} by {route}.as_rate()",
"display_type": "bars",
"style": {
"palette": "warm"
}
}
]
},
"layout": {
"x": 6,
"y": 9,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Request duration p95 by route (ms) \u2014 needs distribution mapping, see README",
"show_legend": true,
"requests": [
{
"q": "p95:switch.http.request.duration{$env,$service} by {route}",
"display_type": "line"
}
]
},
"layout": {
"x": 0,
"y": 13,
"width": 12,
"height": 4
}
},
{
"definition": {
"type": "note",
"content": "**Rooms.** Delivery lag is the age of a message when it reached a recipient, measured per delivery. Delivery failures are the exceptions the delivery loop swallows on purpose so one bad room cannot stop the others \u2014 which is also what makes them invisible without this.\n\n**Sent and delivered are not the same unit.** A send is one write; a delivery is one recipient, so a room with five clients delivers five times per send. Delivered sitting above sent is normal. What matters is the shape of each line, not the gap between them.",
"background_color": "gray",
"font_size": "14",
"text_align": "left",
"vertical_align": "top",
"show_tick": false,
"tick_pos": "50%",
"tick_edge": "left",
"has_padding": true
},
"layout": {
"x": 0,
"y": 17,
"width": 12,
"height": 2
}
},
{
"definition": {
"type": "timeseries",
"title": "Messages per second: sent (writes) vs delivered (per recipient)",
"show_legend": true,
"requests": [
{
"q": "sum:switch.messages.sent{$env,$service} by {actor}.as_rate()",
"display_type": "line"
},
{
"q": "sum:switch.messages.delivered{$env,$service} by {actor}.as_rate()",
"display_type": "line"
}
]
},
"layout": {
"x": 0,
"y": 19,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Delivery, routing and message-observer failures, and rows too old to replay, per second",
"show_legend": true,
"requests": [
{
"q": "sum:switch.messages.delivery_failures{$env,$service}.as_rate()",
"display_type": "bars",
"style": {
"palette": "warm"
}
},
{
"q": "sum:switch.messages.routing_failures{$env,$service} by {stage}.as_rate()",
"display_type": "bars",
"style": {
"palette": "purple"
}
},
{
"q": "sum:switch.messages.observer_failures{$env,$service} by {actor}.as_rate()",
"display_type": "bars",
"style": {
"palette": "orange"
}
},
{
"q": "sum:switch.messages.backlog_skipped{$env,$service} by {actor}.as_rate()",
"display_type": "bars",
"style": {
"palette": "cool"
}
}
]
},
"layout": {
"x": 6,
"y": 19,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Delivery lag p95 / p99 (ms) \u2014 needs distribution mapping, see README",
"show_legend": true,
"requests": [
{
"q": "p95:switch.messages.delivery_lag{$env,$service}",
"display_type": "line"
},
{
"q": "p99:switch.messages.delivery_lag{$env,$service}",
"display_type": "line"
}
]
},
"layout": {
"x": 0,
"y": 23,
"width": 12,
"height": 4
}
},
{
"definition": {
"type": "note",
"content": "**Bridges.** Inbound is counted where every platform event already funnels through; outbound at the relay, so a human actor's own echo and a room with no channel are not counted as traffic nobody sent.",
"background_color": "gray",
"font_size": "14",
"text_align": "left",
"vertical_align": "top",
"show_tick": false,
"tick_pos": "50%",
"tick_edge": "left",
"has_padding": true
},
"layout": {
"x": 0,
"y": 27,
"width": 12,
"height": 1
}
},
{
"definition": {
"type": "timeseries",
"title": "Bridge events in per second, by bridge and platform",
"show_legend": true,
"requests": [
{
"q": "sum:switch.bridge.events_in{$env,$service} by {bridge,platform}.as_rate()",
"display_type": "area"
}
]
},
"layout": {
"x": 0,
"y": 28,
"width": 4,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Bridge events out per second, by bridge and platform",
"show_legend": true,
"requests": [
{
"q": "sum:switch.bridge.events_out{$env,$service} by {bridge,platform}.as_rate()",
"display_type": "area"
}
]
},
"layout": {
"x": 4,
"y": 28,
"width": 4,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Bridge errors per second, by bridge, platform and direction",
"show_legend": true,
"requests": [
{
"q": "sum:switch.bridge.errors{$env,$service} by {bridge,platform,direction}.as_rate()",
"display_type": "bars",
"style": {
"palette": "warm"
}
}
]
},
"layout": {
"x": 8,
"y": 28,
"width": 4,
"height": 4
}
},
{
"definition": {
"type": "note",
"content": "**Resources.** Reported by the process about itself \u2014 there is no infrastructure agent in this cluster. Event-loop lag is the one no agent could produce: the server is single-threaded, so one blocking call stalls every room and bridge at once, and from outside that looks like an unrelated timeout somewhere else.\n\nPool overflow is zero until the pool is full, then counts the connections opened beyond it \u2014 saturation shows as in-use approaching size.",
"background_color": "gray",
"font_size": "14",
"text_align": "left",
"vertical_align": "top",
"show_tick": false,
"tick_pos": "50%",
"tick_edge": "left",
"has_padding": true
},
"layout": {
"x": 0,
"y": 32,
"width": 12,
"height": 2
}
},
{
"definition": {
"type": "timeseries",
"title": "Database pool: peak in use vs size",
"show_legend": true,
"requests": [
{
"q": "sum:switch.db.pool.in_use{$env,$service}",
"display_type": "area"
},
{
"q": "sum:switch.db.pool.size{$env,$service}",
"display_type": "line"
},
{
"q": "sum:switch.db.pool.overflow{$env,$service}",
"display_type": "line"
}
]
},
"layout": {
"x": 0,
"y": 34,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Event-loop lag (ms) and stalls of 50 ms or more",
"show_legend": true,
"requests": [
{
"q": "max:switch.runtime.event_loop_lag{$env,$service}",
"display_type": "bars"
},
{
"q": "count:switch.runtime.event_loop_stalls{$env,$service}.as_count()",
"display_type": "line",
"on_right_yaxis": true
}
]
},
"layout": {
"x": 6,
"y": 34,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Process memory (bytes)",
"show_legend": false,
"requests": [
{
"q": "sum:switch.runtime.memory_rss{$env,$service}",
"display_type": "area"
}
]
},
"layout": {
"x": 0,
"y": 38,
"width": 3,
"height": 3
}
},
{
"definition": {
"type": "timeseries",
"title": "CPU seconds per second, by mode",
"show_legend": true,
"requests": [
{
"q": "sum:switch.runtime.cpu_seconds{$env,$service} by {mode}.as_rate()",
"display_type": "area"
}
]
},
"layout": {
"x": 3,
"y": 38,
"width": 3,
"height": 3
}
},
{
"definition": {
"type": "timeseries",
"title": "Open file descriptors",
"show_legend": false,
"requests": [
{
"q": "sum:switch.runtime.open_fds{$env,$service}",
"display_type": "line"
}
]
},
"layout": {
"x": 6,
"y": 38,
"width": 3,
"height": 3
}
},
{
"definition": {
"type": "timeseries",
"title": "Garbage-collection pause p99 / avg (ms) by generation. Needs distribution mapping, see README",
"show_legend": true,
"requests": [
{
"q": "p99:switch.runtime.gc_pause{$env,$service} by {generation}",
"display_type": "line"
},
{
"q": "avg:switch.runtime.gc_pause{$env,$service} by {generation}",
"display_type": "bars"
}
]
},
"layout": {
"x": 9,
"y": 38,
"width": 3,
"height": 3
}
},
{
"definition": {
"type": "note",
"content": "**Things that are lost.** Each of these is a message or an event that existed and did not arrive. They are counted rather than only logged because every one of them is invisible from the outside: a send that never persisted looks like a quiet room, and an agent told it missed events has no way to tell anyone else.",
"background_color": "gray",
"font_size": "14",
"text_align": "left",
"vertical_align": "top",
"show_tick": false,
"tick_pos": "50%",
"tick_edge": "left",
"has_padding": true
},
"layout": {
"x": 0,
"y": 41,
"width": 12,
"height": 2
}
},
{
"definition": {
"type": "timeseries",
"title": "Sends that never persisted, per second",
"show_legend": true,
"requests": [
{
"q": "sum:switch.messages.send_failures{$env,$service} by {kind}.as_rate()",
"display_type": "bars",
"style": {
"palette": "warm"
}
}
]
},
"layout": {
"x": 0,
"y": 43,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Agent events dropped per second, by cause",
"show_legend": true,
"requests": [
{
"q": "sum:switch.agent.events_dropped{$env,$service} by {reason}.as_rate()",
"display_type": "bars",
"style": {
"palette": "warm"
}
}
]
},
"layout": {
"x": 6,
"y": 43,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Agent connections expired per second (heartbeat lapsed)",
"show_legend": true,
"requests": [
{
"q": "sum:switch.agent.connections_expired{$env,$service}.as_rate()",
"display_type": "bars"
}
]
},
"layout": {
"x": 0,
"y": 47,
"width": 12,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Database query duration p95 / p99 (ms) by operation \u2014 needs distribution mapping, see README",
"show_legend": true,
"requests": [
{
"q": "p95:switch.db.query.duration{$env,$service} by {operation}",
"display_type": "line"
},
{
"q": "p99:switch.db.query.duration{$env,$service} by {operation}",
"display_type": "line"
}
]
},
"layout": {
"x": 0,
"y": 51,
"width": 12,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Bridge call duration p95 (ms) by bridge and platform \u2014 needs distribution mapping, see README",
"show_legend": true,
"requests": [
{
"q": "p95:switch.bridge.call.duration{$env,$service} by {bridge,platform}",
"display_type": "line"
}
]
},
"layout": {
"x": 0,
"y": 55,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Publications held back by a rate limit p95 (s) by platform and delivery \u2014 needs distribution mapping, see README",
"show_legend": true,
"requests": [
{
"q": "p95:switch.bridge.throttle.held{$env,$service} by {platform,delivery}",
"display_type": "line"
}
]
},
"layout": {
"x": 6,
"y": 55,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Bridges running by platform",
"show_legend": true,
"requests": [
{
"q": "sum:switch.bridges.running{$env,$service} by {platform}",
"display_type": "line"
}
]
},
"layout": {
"x": 0,
"y": 59,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "toplist",
"title": "Crashed bridges, by name (empty while all run)",
"requests": [
{
"q": "max:switch.bridges.crashed{$env,$service} by {platform,bridge_name}",
"aggregator": "last"
}
]
},
"layout": {
"x": 0,
"y": 101,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Messaging events from unclaimed chats, by platform",
"show_legend": true,
"requests": [
{
"q": "sum:switch.messaging.events_ignored{$env,$service} by {platform,reason}.as_count()",
"display_type": "bars"
}
]
},
"layout": {
"x": 6,
"y": 59,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Database pool: checkout timeouts",
"show_legend": false,
"requests": [
{
"q": "sum:switch.db.pool.timeouts{$env,$service}.as_count()",
"display_type": "bars"
}
]
},
"layout": {
"x": 0,
"y": 63,
"width": 12,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Agents connected, by transport and client",
"show_legend": true,
"requests": [
{
"q": "sum:switch.agents.connected{$env,$service} by {transport,client}",
"display_type": "line"
}
]
},
"layout": {
"x": 0,
"y": 67,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Agent connections refused per second, by reason",
"show_legend": true,
"requests": [
{
"q": "sum:switch.agent.connections_refused{$env,$service} by {reason}.as_rate()",
"display_type": "bars",
"style": {
"palette": "warm"
}
}
]
},
"layout": {
"x": 6,
"y": 67,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Repeated refusals answered 429 per second, by the route that refused them",
"show_legend": true,
"requests": [
{
"q": "sum:switch.http.refusals_throttled{$env,$service} by {route}.as_rate()",
"display_type": "bars",
"style": {
"palette": "warm"
}
}
]
},
"layout": {
"x": 0,
"y": 71,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Connection opens told to come back later per second, by route (a reconnect storm being paced)",
"show_legend": true,
"requests": [
{
"q": "sum:switch.agent.opens_deferred{$env,$service} by {route}.as_rate()",
"display_type": "bars",
"style": {
"palette": "warm"
}
}
]
},
"layout": {
"x": 6,
"y": 71,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Database sessions for this server's role, by state (the database's view)",
"show_legend": true,
"requests": [
{
"q": "max:switch.db.server.connections{$env,$service} by {state}",
"display_type": "area"
}
]
},
"layout": {
"x": 0,
"y": 75,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Database transactions per second (the database's view)",
"show_legend": false,
"requests": [
{
"q": "avg:switch.db.server.transactions{$env,$service}.as_rate()",
"display_type": "line"
}
]
},
"layout": {
"x": 6,
"y": 75,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Agent buffer events walked per scan, p50 / p95 by operation (cap 2000), needs distribution mapping, see README",
"show_legend": true,
"requests": [
{
"q": "p50:switch.agent.buffer.scanned{$env,$service} by {operation}",
"display_type": "line"
},
{
"q": "p95:switch.agent.buffer.scanned{$env,$service} by {operation}",
"display_type": "line"
}
]
},
"layout": {
"x": 0,
"y": 79,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Agent buffer scan duration p95 / p99 (ms) by operation, needs distribution mapping, see README",
"show_legend": true,
"requests": [
{
"q": "p95:switch.agent.buffer.scan.duration{$env,$service} by {operation}",
"display_type": "line"
},
{
"q": "p99:switch.agent.buffer.scan.duration{$env,$service} by {operation}",
"display_type": "line"
}
]
},
"layout": {
"x": 6,
"y": 79,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Database pool: connection hold p95 / p99 (ms) by caller. Needs distribution mapping, see README",
"show_legend": true,
"requests": [
{
"q": "p95:switch.db.pool.hold.duration{$env,$service} by {caller}",
"display_type": "line"
},
{
"q": "p99:switch.db.pool.hold.duration{$env,$service} by {caller}",
"display_type": "line"
}
]
},
"layout": {
"x": 0,
"y": 83,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Database pool: wait for a connection p95 / p99 (ms). Needs distribution mapping, see README",
"show_legend": true,
"requests": [
{
"q": "p95:switch.db.pool.wait.duration{$env,$service}",
"display_type": "line"
},
{
"q": "p99:switch.db.pool.wait.duration{$env,$service}",
"display_type": "line"
}
]
},
"layout": {
"x": 6,
"y": 83,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Agent connections opened per second, by kind (after_lapse = returning from a lapse)",
"show_legend": true,
"requests": [
{
"q": "sum:switch.agent.connections_opened{$env,$service} by {kind}.as_rate()",
"display_type": "bars"
}
]
},
"layout": {
"x": 0,
"y": 87,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Process starts (deploys and restarts)",
"show_legend": false,
"requests": [
{
"q": "sum:switch.runtime.starts{$env,$service}.as_count()",
"display_type": "bars"
}
]
},
"layout": {
"x": 6,
"y": 87,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "note",
"content": "**Shared room reads.** Each fill is one database read standing in for one per member of the room, so fills against deliveries is the saving. A fallback (behind, evicted, unwatched, gave_up) is a transport reading the database itself: it costs a read, never a message.",
"background_color": "gray",
"font_size": "14",
"text_align": "left",
"vertical_align": "top",
"show_tick": false,
"tick_pos": "50%",
"tick_edge": "left",
"has_padding": true
},
"layout": {
"x": 0,
"y": 91,
"width": 12,
"height": 2
}
},
{
"definition": {
"type": "timeseries",
"title": "Room cache reads per second, by outcome (hit and filled are served from memory)",
"show_legend": true,
"requests": [
{
"q": "sum:switch.delivery_cache.reads{$env,$service} by {outcome}.as_rate()",
"display_type": "bars"
}
]
},
"layout": {
"x": 0,
"y": 93,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Room cache fills and rows read per second, against messages delivered",
"show_legend": true,
"requests": [
{
"q": "sum:switch.delivery_cache.fills{$env,$service} by {outcome}.as_rate()",
"display_type": "line"
},
{
"q": "sum:switch.delivery_cache.rows_read{$env,$service}.as_rate()",
"display_type": "line"
},
{
"q": "sum:switch.messages.delivered{$env,$service}.as_rate()",
"display_type": "line"
}
]
},
"layout": {
"x": 6,
"y": 93,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Room cache memory (bytes) and rooms held",
"show_legend": true,
"requests": [
{
"q": "sum:switch.delivery_cache.bytes{$env,$service}",
"display_type": "line"
},
{
"q": "sum:switch.delivery_cache.rooms{$env,$service}",
"display_type": "line"
}
]
},
"layout": {
"x": 0,
"y": 97,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Room cache evictions per second, by reason",
"show_legend": true,
"requests": [
{
"q": "sum:switch.delivery_cache.evictions{$env,$service} by {reason}.as_rate()",
"display_type": "bars"
}
]
},
"layout": {
"x": 6,
"y": 97,
"width": 6,
"height": 4
}
},
{
"definition": {
"type": "timeseries",
"title": "Service tokens per second, by connector and outcome (issued, or the refusal's reason code)",
"show_legend": true,
"requests": [
{
"q": "sum:switch.service_tokens.requests{$env,$service} by {connector,outcome}.as_rate()",
"display_type": "bars"
}
]
},
"layout": {
"x": 0,
"y": 97,
"width": 12,
"height": 4
}
}
]
}